2024-01-16T14:44:51.514 INFO:root:teuthology version: 0.0.1.dev249+gcd45576 2024-01-16T14:44:51.514 DEBUG:teuthology.run:Teuthology command: teuthology --description fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} --owner scheduled_rishabh@teuthology --name rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi --archive /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206 --verbose -- /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/orig.config.yaml 2024-01-16T14:44:51.569 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T14:44:51.659 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206 branch: rishabh-clone-stats description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} email: null first_in_suite: false job_id: '7518206' kernel: flavor: default kdb: true sha1: 2668949e0971849893a3fb7f177ce9032fc65780 last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-clone-stats ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-clone-stats sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 owner: scheduled_rishabh@teuthology priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 3695 sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 sleep_before_teardown: 0 suite: fs:volumes suite_branch: rishabh-clone-stats suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi053.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM0JQjhuNggf5ic6Z+v+ATyjbZxtPtAlHaYJNWISkWvf8Z0gynhR96Yu8TtxOs6iLtho5IdeDI+TpDb1zD7vs7E= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPgRJ4UNIuRtkJXGmoH7Fyp95mWSrJnXr95VQfYW7AHroNiCVmwc+foJoouNVGIELbaIXhrMCms6FZ2ILBwlnIQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestCloneProgress teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: cd45576300487d997e5a85abed65500b9f5d143b timestamp: 2024-01-16_14:39:36 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-01-16T14:44:51.659 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa; will attempt to use it 2024-01-16T14:44:51.660 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks 2024-01-16T14:44:51.660 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-16T14:44:51.663 INFO:teuthology.task.internal:Checking packages... 2024-01-16T14:44:51.686 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '73a4532fc4d97adee0b985d4d42874ea351a1c86' 2024-01-16T14:44:51.686 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-16T14:44:51.686 INFO:teuthology.packaging:ref: None 2024-01-16T14:44:51.686 INFO:teuthology.packaging:tag: None 2024-01-16T14:44:51.686 INFO:teuthology.packaging:branch: rishabh-clone-stats 2024-01-16T14:44:51.686 INFO:teuthology.packaging:sha1: 73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:44:51.687 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-clone-stats 2024-01-16T14:44:52.026 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-7629.g73a4532f 2024-01-16T14:44:52.027 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-16T14:44:52.034 INFO:teuthology.task.internal:no buildpackages task found 2024-01-16T14:44:52.034 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-16T14:44:52.040 INFO:teuthology.task.internal:Saving configuration 2024-01-16T14:44:52.056 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-16T14:44:52.062 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T14:44:52.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:39:43.438575', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM0JQjhuNggf5ic6Z+v+ATyjbZxtPtAlHaYJNWISkWvf8Z0gynhR96Yu8TtxOs6iLtho5IdeDI+TpDb1zD7vs7E='} 2024-01-16T14:44:52.108 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:39:43.436837', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPgRJ4UNIuRtkJXGmoH7Fyp95mWSrJnXr95VQfYW7AHroNiCVmwc+foJoouNVGIELbaIXhrMCms6FZ2ILBwlnIQ='} 2024-01-16T14:44:52.109 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-16T14:44:52.115 INFO:teuthology.task.internal:roles: ubuntu@smithi053.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-01-16T14:44:52.115 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-01-16T14:44:52.115 INFO:teuthology.run_tasks:Running task console_log... 2024-01-16T14:44:52.179 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ffacc37b160>, signals=[15]) 2024-01-16T14:44:52.180 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-16T14:44:52.185 INFO:teuthology.task.internal:Opening connections... 2024-01-16T14:44:52.186 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T14:44:52.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:52.290 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T14:44:52.291 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:44:52.376 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-16T14:44:52.383 DEBUG:teuthology.orchestra.run.smithi053:> uname -m 2024-01-16T14:44:52.400 INFO:teuthology.orchestra.run.smithi053.stdout:x86_64 2024-01-16T14:44:52.401 DEBUG:teuthology.orchestra.run.smithi053:> cat /etc/os-release 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:NAME="CentOS Stream" 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION="9" 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:ID="centos" 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:VERSION_ID="9" 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:44:52.457 INFO:teuthology.orchestra.run.smithi053.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:ANSI_COLOR="0;31" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:44:52.458 INFO:teuthology.orchestra.run.smithi053.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:44:52.458 INFO:teuthology.lock.ops:Updating smithi053.front.sepia.ceph.com on lock server 2024-01-16T14:44:52.484 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-01-16T14:44:52.502 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-01-16T14:44:52.503 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="CentOS Stream" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="9" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:ID="centos" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE="rhel fedora" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="9" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:PLATFORM_ID="platform:el9" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="CentOS Stream 9" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:ANSI_COLOR="0;31" 2024-01-16T14:44:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:LOGO="fedora-logo-icon" 2024-01-16T14:44:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-01-16T14:44:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://centos.org/" 2024-01-16T14:44:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-16T14:44:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-01-16T14:44:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-16T14:44:52.561 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-01-16T14:44:52.584 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-16T14:44:52.592 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-16T14:44:52.597 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-16T14:44:52.597 DEBUG:teuthology.orchestra.run.smithi053:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:44:52.600 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-01-16T14:44:52.616 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-16T14:44:52.623 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-16T14:44:52.623 DEBUG:teuthology.orchestra.run.smithi053:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:44:52.657 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-01-16T14:44:52.683 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-16T14:44:52.735 INFO:teuthology.run_tasks:Running task kernel... 2024-01-16T14:44:52.749 INFO:teuthology.task.kernel:normalize config orig: {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'} 2024-01-16T14:44:52.749 INFO:teuthology.task.kernel:config {'mon.a': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'mon.b': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, timeout 300 2024-01-16T14:44:52.749 DEBUG:teuthology.orchestra.run.smithi053:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:44:52.750 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-16T14:44:52.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:44:52.766 DEBUG:teuthology.task.kernel:sha1 for mon.b is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:52.766 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:44:52.766 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-01-16T14:44:52.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:44:52.767 DEBUG:teuthology.task.kernel:sha1 for mon.a is 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:52.767 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "2668949e0971849893a3fb7f177ce9032fc65780"... 2024-01-16T14:44:52.768 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-01-16T14:44:52.824 INFO:teuthology.orchestra.run.smithi146.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:44:52.824 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:52.824 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:44:52.824 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:52.828 INFO:teuthology.orchestra.run.smithi053.stdout:5.14.0-361.el9.x86_64 2024-01-16T14:44:52.828 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:52.828 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-01-16T14:44:52.828 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:44:52.975 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum upgrade -y linux-firmware 2024-01-16T14:44:53.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum upgrade -y linux-firmware 2024-01-16T14:44:55.569 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 6.7 MB/s | 6.4 MB 00:00 2024-01-16T14:44:56.684 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 3.4 MB/s | 6.4 MB 00:01 2024-01-16T14:44:57.285 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-01-16T14:44:58.826 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 1.7 MB/s | 2.0 MB 00:01 2024-01-16T14:45:06.108 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 961 kB/s | 8.0 MB 00:08 2024-01-16T14:45:06.255 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 1.1 MB/s | 8.0 MB 00:07 2024-01-16T14:45:08.109 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-01-16T14:45:08.506 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-01-16T14:45:13.334 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 5.9 MB 00:01 2024-01-16T14:45:15.102 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 15 kB 00:00 2024-01-16T14:45:16.315 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 1.4 MB/s | 5.9 MB 00:04 2024-01-16T14:45:17.824 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 15 kB 00:00 2024-01-16T14:45:19.858 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-01-16T14:45:22.836 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 2.6 MB/s | 20 MB 00:07 2024-01-16T14:45:24.633 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 24 kB/s | 1.7 kB 00:00 2024-01-16T14:45:26.913 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-16T14:45:26.925 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:================================================================================ 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 1 Package 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 375 M 2024-01-16T14:45:26.926 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-16T14:45:27.794 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-01-16T14:45:30.207 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout: linux-firmware noarch 20240109-142.el9 CentOS-BaseOS 375 M 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout:================================================================================ 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 1 Package 2024-01-16T14:45:30.222 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:45:30.223 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 375 M 2024-01-16T14:45:30.223 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-16T14:45:40.166 INFO:teuthology.orchestra.run.smithi146.stdout:linux-firmware-20240109-142.el9.noarch.rpm 28 MB/s | 375 MB 00:13 2024-01-16T14:45:40.167 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:45:40.167 INFO:teuthology.orchestra.run.smithi146.stdout:Total 28 MB/s | 375 MB 00:13 2024-01-16T14:45:40.168 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-16T14:45:40.263 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-16T14:45:40.263 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-16T14:45:42.719 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-16T14:45:42.719 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-16T14:45:44.227 INFO:teuthology.orchestra.run.smithi053.stdout:linux-firmware-20240109-142.el9.noarch.rpm 27 MB/s | 375 MB 00:14 2024-01-16T14:45:44.228 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:45:44.229 INFO:teuthology.orchestra.run.smithi053.stdout:Total 27 MB/s | 375 MB 00:14 2024-01-16T14:45:44.229 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-16T14:45:44.324 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-16T14:45:44.324 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-16T14:45:46.808 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-16T14:45:46.808 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-16T14:45:48.997 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-16T14:45:49.249 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:45:49.335 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:45:50.497 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:45:50.497 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:45:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:45:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:45:50.868 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-01-16T14:45:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:45:50.868 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:45:50.868 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-16T14:45:51.169 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:45:51.169 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:45:51.311 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:45:51.311 DEBUG:teuthology.orchestra.run.smithi146:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:45:51.314 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi146.front.sepia.ceph.com to complete... 2024-01-16T14:45:53.231 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-16T14:45:53.464 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:45:53.545 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:45:54.705 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:45:54.705 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : linux-firmware-20240109-142.el9.noarch 1/2 2024-01-16T14:45:55.089 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-01-16T14:45:55.089 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:45:55.089 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-01-16T14:45:55.089 INFO:teuthology.orchestra.run.smithi053.stdout: linux-firmware-20240109-142.el9.noarch 2024-01-16T14:45:55.089 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:45:55.089 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-16T14:45:55.374 INFO:teuthology.task.kernel:Downloading kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:45:55.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2668949e0971849893a3fb7f177ce9032fc65780 2024-01-16T14:45:55.656 INFO:teuthology.task.kernel:fetching, builder baseurl is https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ 2024-01-16T14:45:55.657 DEBUG:teuthology.orchestra.run.smithi053:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.7.0_g2668949e0971-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/ --input-file=- 2024-01-16T14:45:55.659 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi053.front.sepia.ceph.com to complete... 2024-01-16T14:46:05.568 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16 14:46:05 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:46:05.569 INFO:teuthology.orchestra.run.smithi146.stderr:FINISHED --2024-01-16 14:46:05-- 2024-01-16T14:46:05.569 INFO:teuthology.orchestra.run.smithi146.stderr:Total wall clock time: 14s 2024-01-16T14:46:05.569 INFO:teuthology.orchestra.run.smithi146.stderr:Downloaded: 1 files, 315M in 14s (22.6 MB/s) 2024-01-16T14:46:05.571 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.b... 2024-01-16T14:46:05.571 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:46:07.465 INFO:teuthology.orchestra.run.smithi146.stdout:Verifying... ######################################## 2024-01-16T14:46:07.527 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing... ######################################## 2024-01-16T14:46:07.606 INFO:teuthology.orchestra.run.smithi146.stdout:Updating / installing... 2024-01-16T14:46:14.968 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:46:30.333 INFO:teuthology.orchestra.run.smithi146.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:46:52.478 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:46:52.478 DEBUG:teuthology.orchestra.run.smithi146:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:46:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:46:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:46:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:46:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:46:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:46:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:46:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:46:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:46:52.560 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:46:52.561 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:46:52.562 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:46:52.563 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:46:52.564 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:46:52.565 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:46:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:46:52.567 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:46:52.568 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:46:52.569 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:46:52.570 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:46:52.571 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:46:52.572 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:46:52.573 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:46:52.574 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:46:52.575 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:46:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:46:52.577 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:46:52.578 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:46:52.579 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:46:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:46:52.581 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:46:52.582 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:46:52.583 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:46:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:46:52.585 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:46:52.586 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:46:52.587 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:46:52.588 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:46:52.589 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:46:52.590 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:46:52.591 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:46:52.592 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:46:52.593 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:46:52.594 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:46:52.595 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:46:52.596 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:46:52.597 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:46:52.598 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:46:52.599 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:46:52.600 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:46:52.601 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:46:52.602 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:46:52.603 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:46:52.604 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:46:52.605 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:46:52.606 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:46:52.607 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:46:52.608 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:46:52.609 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:46:52.610 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:46:52.611 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:46:52.612 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:46:52.613 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:46:52.614 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:46:52.615 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:46:52.616 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:46:52.617 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:46:52.618 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:46:52.619 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:46:52.620 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:46:52.621 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:46:52.622 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:46:52.623 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:46:52.624 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:46:52.625 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:46:52.626 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:46:52.627 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:46:52.628 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:46:52.629 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:46:52.630 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:46:52.631 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:46:52.632 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:46:52.633 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:46:52.634 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:46:52.635 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:46:52.636 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:46:52.637 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:46:52.638 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:46:52.639 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:46:52.640 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:46:52.641 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:46:52.642 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:46:52.643 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:46:52.644 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:46:52.645 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:46:52.646 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:46:52.647 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:46:52.648 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:46:52.649 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:46:52.650 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:46:52.651 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:46:52.652 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:46:52.653 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:46:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:46:52.655 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:46:52.656 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:46:52.657 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:46:52.658 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:46:52.659 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:46:52.660 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:46:52.661 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:46:52.662 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:46:52.663 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:46:52.664 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:46:52.665 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:46:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:46:52.667 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:46:52.668 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:46:52.669 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:46:52.670 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:46:52.671 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:46:52.672 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:46:52.673 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:46:52.674 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:46:52.675 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:46:52.676 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:46:52.677 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:46:52.678 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:46:52.679 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:46:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:46:52.681 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:46:52.682 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:46:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:46:52.684 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:46:52.685 INFO:teuthology.orchestra.run.smithi146.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:46:52.685 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:46:52.686 DEBUG:teuthology.orchestra.run.smithi146:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:46:52.707 INFO:teuthology.orchestra.run.smithi146.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:46:52.707 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:46:52.707 INFO:teuthology.orchestra.run.smithi146.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:46:52.708 INFO:teuthology.orchestra.run.smithi146.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:46:52.709 INFO:teuthology.orchestra.run.smithi146.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:46:52.709 INFO:teuthology.orchestra.run.smithi146.stdout:fi 2024-01-16T14:46:52.709 DEBUG:teuthology.orchestra.run.smithi146:> sudo rpm -qi grub2-tools 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Name : grub2-tools 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Epoch : 1 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Version : 2.06 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Release : 68.el9 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: x86_64 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Group : Unspecified 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Size : 8264254 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:License : GPLv3+ 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:46:52.785 INFO:teuthology.orchestra.run.smithi146.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:Packager : builder@centos.org 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:Vendor : CentOS 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:Summary : Support tools for GRUB. 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:Description : 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:hardware devices. 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:46:52.786 INFO:teuthology.orchestra.run.smithi146.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:46:52.787 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:46:52.787 INFO:teuthology.task.kernel:Updating grub on smithi146 to boot 6.7.0-g2668949e0971 2024-01-16T14:46:52.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:46:53.399 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-01-16T14:46:54.767 INFO:teuthology.orchestra.run.smithi146.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:46:54.799 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-01-16T14:46:54.807 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-01-16T14:46:54.824 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.al0ENWACQU 2024-01-16T14:46:54.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.al0ENWACQU 2024-01-16T14:46:54.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.al0ENWACQU 2024-01-16T14:46:55.017 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.al0ENWACQU is 6KB 2024-01-16T14:46:55.066 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.al0ENWACQU 2024-01-16T14:46:55.081 DEBUG:teuthology.orchestra.run.smithi146:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:46:55.146 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:46:55.146 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:46:55.146 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:46:55.146 INFO:teuthology.orchestra.run.smithi146.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:46:55.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:46:55.309 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-01-16T14:47:25.311 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-01-16T14:47:25.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:47:25.312 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:43.676 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-01-16T14:47:52.677 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:47:52.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:47:55.740 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-01-16T14:48:07.741 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:48:07.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:48:31.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16 14:48:31 URL:https://3.chacra.ceph.com/r/kernel/testing/2668949e0971849893a3fb7f177ce9032fc65780/centos/9/flavors/default/x86_64/kernel-6.7.0_g2668949e0971-1.x86_64.rpm [330271041/330271041] -> "/tmp/kernel.x86_64.rpm" [1] 2024-01-16T14:48:31.309 INFO:teuthology.orchestra.run.smithi053.stderr:FINISHED --2024-01-16 14:48:31-- 2024-01-16T14:48:31.309 INFO:teuthology.orchestra.run.smithi053.stderr:Total wall clock time: 2m 36s 2024-01-16T14:48:31.309 INFO:teuthology.orchestra.run.smithi053.stderr:Downloaded: 1 files, 315M in 2m 35s (2.03 MB/s) 2024-01-16T14:48:31.312 INFO:teuthology.task.kernel:Installing kernel 2668949e0971849893a3fb7f177ce9032fc65780 on mon.a... 2024-01-16T14:48:31.313 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-01-16T14:48:33.312 INFO:teuthology.orchestra.run.smithi053.stdout:Verifying... ######################################## 2024-01-16T14:48:33.388 INFO:teuthology.orchestra.run.smithi053.stdout:Preparing... ######################################## 2024-01-16T14:48:33.472 INFO:teuthology.orchestra.run.smithi053.stdout:Updating / installing... 2024-01-16T14:48:41.236 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-6.7.0_g2668949e0971-1 ######################################## 2024-01-16T14:48:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-01-16T14:49:07.743 DEBUG:teuthology.orchestra.remote:timed out 2024-01-16T14:49:17.800 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi053.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-01-16T14:49:17.800 DEBUG:teuthology.orchestra.run.smithi053:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/boot/config-6.7.0-g2668949e0971 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/boot/vmlinuz-6.7.0-g2668949e0971 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch 2024-01-16T14:49:17.877 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crc32c-intel.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-01-16T14:49:17.878 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-avx2.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-01-16T14:49:17.879 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-amd.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm-intel.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/arch/x86/kvm/kvm.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ansi_cprng.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/anubis.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/arc4.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_memcpy.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_pq.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_raid6_recov.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_tx.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/async_xor.ko 2024-01-16T14:49:17.880 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/async_tx/raid6test.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authenc.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/authencesn.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blake2b_generic.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_common.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/blowfish_generic.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/camellia_generic.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast5_generic.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast6_generic.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cast_common.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/ccm.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cmac.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crc32_generic.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_engine.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/crypto_user.ko 2024-01-16T14:49:17.881 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/cts.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/des_generic.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/dh_generic.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/echainiv.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/essiv.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/fcrypt.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/khazad.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/lz4hc.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/md4.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/michael_mic.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcbc.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/pcrypt.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/rmd160.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/seed.ko 2024-01-16T14:49:17.882 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/serpent_generic.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tcrypt.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/tea.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_common.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/twofish_generic.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/vmac.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/wp512.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xcbc.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xor.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/crypto/xxhash_generic.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_ipmi.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/acpi_pad.ko 2024-01-16T14:49:17.883 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/custom_method.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/ec_sys.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/platform_profile.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbs.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/sbshc.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/acpi/video.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/acard-ahci.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ahci_platform.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/ata_generic.ko 2024-01-16T14:49:17.884 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/libahci_platform.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_acpi.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ali.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_amd.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_artop.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atiixp.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_atp867x.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd640.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cmd64x.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_cypress.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_efar.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt366.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt37x.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x2n.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_hpt3x3.ko 2024-01-16T14:49:17.885 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it8213.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_it821x.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_jmicron.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_marvell.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_mpiix.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_netcell.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ninja32.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87410.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_ns87415.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_oldpiix.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_opti.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_optidma.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pcmcia.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc2027x.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-01-16T14:49:17.886 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_piccolo.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_rdc.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sch.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_serverworks.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sil680.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sis.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_sl82c105.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_triflex.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pata_via.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/pdc_adma.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_inic162x.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_mv.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_nv.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_promise.ko 2024-01-16T14:49:17.887 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_qstor.ko 2024-01-16T14:49:17.888 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil.ko 2024-01-16T14:49:17.888 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sil24.ko 2024-01-16T14:49:17.888 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sis.ko 2024-01-16T14:49:17.888 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_svw.ko 2024-01-16T14:49:17.888 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_sx4.ko 2024-01-16T14:49:17.888 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_uli.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_via.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ata/sata_vsc.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/auxdisplay/ks0108.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-i2c.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/base/regmap/regmap-mmio.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/bcma/bcma.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/aoe/aoe.ko 2024-01-16T14:49:17.889 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/brd.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/drbd/drbd.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/floppy.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/loop.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/nbd.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/null_blk/null_blk.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/pktcdvd.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/rbd.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/virtio_blk.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-01-16T14:49:17.890 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/xen-blkfront.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/block/zram/zram.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hangcheck-timer.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/amd-rng.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/intel-rng.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/via-rng.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/hw_random/virtio-rng.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-01-16T14:49:17.891 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_si.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/lp.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/mwave/mwave.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/ppdev.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tlclk.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_atmel.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_infineon.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_nsc.ko 2024-01-16T14:49:17.892 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/uv_mmtimer.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/char/virtio_console.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/p4-clockmod.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/powernow-k8.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/cpufreq/speedstep-lib.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/ccp/ccp.ko 2024-01-16T14:49:17.893 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-aes.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/padlock-sha.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dca/dca.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw 2024-01-16T14:49:17.894 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/dw/dw_dmac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/dma/ioat/ioatdma.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/amd64_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/e752x_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/edac_mce_amd.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3000_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i3200_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5100_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i5400_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7300_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i7core_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/i82975x_edac.ko 2024-01-16T14:49:17.895 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/ie31200_edac.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/sb_edac.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/edac/x38_edac.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/edd.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/firmware/iscsi_ibft.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-ich.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/gpio/gpio-viperboard.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-appleir.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-aureal.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-axff.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-dr.ko 2024-01-16T14:49:17.896 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elecom.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-elo.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-emsff.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gaff.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gt683r.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-gyration.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-kbd.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtek-mouse.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-holtekff.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-hyperv.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-icade.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-keytouch.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-kye.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lcpower.ko 2024-01-16T14:49:17.897 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-led.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-lenovo.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-dj.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-multitouch.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-ortek.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-penmount.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-petalynx.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-picolcd.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-pl.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-plantronics.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-primax.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-rmi.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-arvo.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-common.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-isku.ko 2024-01-16T14:49:17.898 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kone.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-konepure.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-lua.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-pyra.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-ryos.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat-savu.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-roccat.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-saitek.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-samsung.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sensor-hub.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sjoy.ko 2024-01-16T14:49:17.899 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sony.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-speedlink.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-steelseries.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-sunplus.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-thrustmaster.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tivo.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-tmff.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-topseed.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-twinhan.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-uclogic.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-waltop.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-wiimote.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-xinmo.ko 2024-01-16T14:49:17.900 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zpff.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/hid-zydacron.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/uhid.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hid/wacom.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_balloon.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_utils.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hv/hv_vmbus.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/abituguru3.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/acpi_power_meter.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7414.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ad7418.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adc128d818.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1025.ko 2024-01-16T14:49:17.901 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1026.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1029.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm1031.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adm9240.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ads7828.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7410.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7411.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7462.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7470.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7475.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/adt7x10.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/amc6821.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/applesmc.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asb100.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asc7621.ko 2024-01-16T14:49:17.902 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/asus_atk0110.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/atxp1.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/coretemp.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/dme1737.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds1621.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ds620.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc1403.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/emc6w201.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71805f.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f71882fg.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/f75375s.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fam15h_power.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/fschmd.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g760a.ko 2024-01-16T14:49:17.903 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/g762.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl518sm.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/gl520sm.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/hwmon-vid.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5500_temp.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/i5k_amb.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmaem.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ibmpex.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina209.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ina2xx.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/it87.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k10temp.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/k8temp.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lineage-pem.ko 2024-01-16T14:49:17.904 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm63.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm73.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm75.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm77.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm78.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm80.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm83.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm85.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm87.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm90.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm92.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm93.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95234.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95241.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/lm95245.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc2945.ko 2024-01-16T14:49:17.905 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4151.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4215.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4222.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4245.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4260.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/ltc4261.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max16065.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1619.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max1668.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max197.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6639.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6650.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/max6697.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/mcp3021.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6683.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775-core.ko 2024-01-16T14:49:17.906 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct6775.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/nct7802.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87360.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pc87427.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pcf8591.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max16064.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max34440.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/max8688.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-01-16T14:49:17.907 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/powr1220.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5627.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch5636.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sch56xx-common.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht15.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sht21.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/shtc1.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/sis5595.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47b397.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m1.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/smsc47m192.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/thmc50.ko 2024-01-16T14:49:17.908 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp102.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp103.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp401.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/tmp421.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via-cputemp.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/via686a.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt1211.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/vt8231.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627ehf.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83627hf.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83781d.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83791d.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83792d.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83793.ko 2024-01-16T14:49:17.909 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83795.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l785ts.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/hwmon/w83l786ng.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-01-16T14:49:17.910 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-i801.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-isch.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-parport.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-via.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-01-16T14:49:17.911 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-dev.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-mux.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-smbus.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/i2c/i2c-stub.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_cm.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_core.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_umad.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/iw_cm.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_cm.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-01-16T14:49:17.912 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-01-16T14:49:17.913 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/fm801-gp.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/gameport/gameport.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/joydev.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/keyboard/gpio_keys.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/appletouch.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/bcm5974.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/cyapatp.ko 2024-01-16T14:49:17.914 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/elan_i2c.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/sermouse.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/synaptics_usb.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/mouse/vsxxxaa.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/rmi4/rmi_core.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/altera_ps2.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/arc_ps2.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/serio/serio_raw.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/input/sparse-keymap.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-blinkm.ko 2024-01-16T14:49:17.915 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lm3530.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-lp3944.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/leds-ss4200.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md 2024-01-16T14:49:17.916 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/bcache/bcache.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-bio-prison.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache-smq.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-cache.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-crypt.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-delay.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-flakey.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-log-userspace.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-multipath.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-queue-length.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-raid.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-round-robin.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-service-time.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-switch.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-thin-pool.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/dm-verity.ko 2024-01-16T14:49:17.917 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/faulty.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/linear.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/multipath.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid0.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid1.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid10.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/md/raid456.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptbase.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptctl.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptfc.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptlan.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptsas.ko 2024-01-16T14:49:17.918 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptscsih.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/message/fusion/mptspi.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_ich.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/lpc_sch.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/mfd-core.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/sm501.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/viperboard.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/vx855.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/mfd/wl1273-core.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds9802als.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/apds990x.ko 2024-01-16T14:49:17.919 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/bh1770glc.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/cb710/cb710.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/at24.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/eeprom/max6875.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/enclosure.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/hpilo.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29003.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/isl29020.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-01-16T14:49:17.920 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-me.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei-txe.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/mei/mei.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-gru/gru.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xp.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpc.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_7xx1.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tifm_core.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/tsl2550.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_balloon.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net 2024-01-16T14:49:17.921 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/bonding/bonding.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/b53/b53_common.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/bcm-sf2.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6060.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/dummy.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/eql.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-01-16T14:49:17.922 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/8390.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/agere/et131x.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-01-16T14:49:17.923 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-01-16T14:49:17.924 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-01-16T14:49:17.925 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-01-16T14:49:17.926 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/dnet.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ethoc.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/fealnx.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e100.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k 2024-01-16T14:49:17.927 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/jme.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-01-16T14:49:17.928 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/skge.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-01-16T14:49:17.929 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc 2024-01-16T14:49:17.930 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/atp.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis190.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sis/sis900.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-01-16T14:49:17.931 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/cassini.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/niu.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sungem.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti 2024-01-16T14:49:17.932 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/ti/tlan.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/geneve.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ieee802154/fakelb.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ifb.ko 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan 2024-01-16T14:49:17.933 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ipvlan/ipvlan.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvlan.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/macvtap.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mdio/mdio-regmap.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/mii.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/net_failover.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/netconsole.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/nlmon.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs-lynx.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/amd.ko 2024-01-16T14:49:17.934 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm7xxx.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/bcm87xx.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/broadcom.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/cicada.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/davicom.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/dp83640.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/et1011c.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/icplus.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/lxt.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/marvell.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/micrel.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/national.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/phylink.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/qsemi.ko 2024-01-16T14:49:17.935 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/realtek.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/smsc.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/ste10Xp.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/phy/vitesse.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/bsd_comp.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_async.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_deflate.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_generic.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_mppe.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/ppp_synctty.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppoe.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pppox.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/ppp/pptp.ko 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip 2024-01-16T14:49:17.936 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slhc.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/slip/slip.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/sungem_phy.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tap.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_activebackup.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_broadcast.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_random.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/tun.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/veth.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/virtio_net.ko 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3 2024-01-16T14:49:17.937 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/vxlan/vxlan.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netback/xen-netback.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/net/xen-netfront.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ntb/ntb.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme-core.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/nvme/host/nvme.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_cs.ko 2024-01-16T14:49:17.938 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_pc.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/parport/parport_serial.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/pcie/aer_inject.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pci/xen-pcifront.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/i82092.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/pd6729.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pcmcia/yenta_socket.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-01-16T14:49:17.939 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acer-wmi.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/acerhdf.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/apple-gmux.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-laptop.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/asus-wmi.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/classmate-laptop.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-01-16T14:49:17.940 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/hdaps.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/intel_ips.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/msi-wmi.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/mxm-wmi.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-laptop.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/samsung-q10.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/topstar-laptop.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/toshiba_haps.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi-bmof.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/platform/x86/wmi.ko 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap 2024-01-16T14:49:17.941 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_common.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/powercap/intel_rapl_msr.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-gpio.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps-ldisc.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/clients/pps_parport.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/pps/pps_core.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ptp/ptp_kvm.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-bq32k.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1286.ko 2024-01-16T14:49:17.942 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1307.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1374.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1511.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1553.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1672.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds1742.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds2404.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-ds3232.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-em3027.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-fm3130.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl12022.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-isl1208.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m41t80.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t35.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-m48t59.ko 2024-01-16T14:49:17.943 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-max6900.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-msm6242.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf2127.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf85063.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8523.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8563.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-pcf8583.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rp5c01.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rs5c372.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8025.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-rx8581.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/rtc/rtc-x1205.ko 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi 2024-01-16T14:49:17.944 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-9xxx.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-sas.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/3w-xxxx.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/BusLogic.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/a100u2w.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aacraid/aacraid.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/advansys.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/am53c974.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/atp870u.ko 2024-01-16T14:49:17.945 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bfa/bfa.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ch.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/csiostor/csiostor.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-01-16T14:49:17.946 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dc395x.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/dmx3191d.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esas2r/esas2r.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/esp_scsi.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/fcoe.ko 2024-01-16T14:49:17.947 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/fnic/fnic.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hpsa.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hptiop.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/hv_storvsc.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/initio.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ipr.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ips.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/isci/isci.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/iscsi_tcp.ko 2024-01-16T14:49:17.948 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libfc/libfc.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libiscsi_tcp.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/libsas/libsas.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/lpfc/lpfc.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-01-16T14:49:17.949 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvsas/mvsas.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/mvumi.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/pmcraid.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla1280.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/raid_class.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_debug.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_fc.ko 2024-01-16T14:49:17.950 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_sas.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_spi.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/scsi_transport_srp.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/ses.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/st.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/stex.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/virtio_scsi.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/vmw_pvscsi.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/scsi/wd719x.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ssb/ssb.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-01-16T14:49:17.951 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/loopback/tcm_loop.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_file.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_iblock.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_mod.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_pscsi.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/target_core_user.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-01-16T14:49:17.952 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-01-16T14:49:17.953 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_gsm.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/n_hdlc.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/nozomi.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/arc_uart.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/tty/serial/jsm/jsm.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/core/ufshcd-core.ko 2024-01-16T14:49:17.954 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_aec.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_cif.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_pci_generic.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/uio/uio_sercos3.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-acm.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/cdc-wdm.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usblp.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/class/usbtmc.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/host/sl811-hcd.ko 2024-01-16T14:49:17.955 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/mdc800.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/image/microtek.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/adutux.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/appledisplay.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi26.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/emi62.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ezusb.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/idmouse.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/iowarrior.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/isight_firmware.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/ldusb.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/legousbtower.ko 2024-01-16T14:49:17.956 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/trancevibrator.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usb3503.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usblcd.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/usbsevseg.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/uss720.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/misc/yurex.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/phy/phy-generic.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/aircable.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ark3116.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/belkin_sa.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ch341.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cp210x.ko 2024-01-16T14:49:17.957 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cyberjack.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/cypress_m8.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/digi_acceleport.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/empeg.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ftdi_sio.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/garmin_gps.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_edgeport.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/io_ti.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipaq.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ipw.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ir-usb.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/keyspan_pda.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kl5kusb105.ko 2024-01-16T14:49:17.958 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/kobil_sct.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mct_u232.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7720.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/mos7840.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/navman.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/omninet.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/opticon.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/option.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/oti6858.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/pl2303.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcaux.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/qcserial.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/quatech2.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/safe_serial.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/sierra.ko 2024-01-16T14:49:17.959 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/spcp8x5.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ssu100.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/symbolserial.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_debug.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/usb_wwan.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/visor.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/whiteheat.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/serial/xsens_mt.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/uas.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-alauda.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-cypress.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-datafab.ko 2024-01-16T14:49:17.960 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-freecom.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-isd200.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-karma.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-onetouch.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-realtek.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr09.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-sddr55.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/ums-usbat.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/storage/usb-storage.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-core.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/usbip-host.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-01-16T14:49:17.961 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/pci/vfio-pci.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_iotlb.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_net.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/vhost/vhost_scsi.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/apple_bl.ko 2024-01-16T14:49:17.962 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/lcd.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/video/backlight/platform_lcd.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_balloon.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_mmio.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/virtio/virtio_ring.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2482.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/masters/ds2490.ko 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves 2024-01-16T14:49:17.963 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_smem.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/slaves/w1_therm.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/w1/wire.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim1535_wdt.ko 2024-01-16T14:49:17.964 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/alim7101_wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/f71808e_wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/hpwdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/i6300esb.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/iTCO_wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ib700wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ibmasr.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it8712f_wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/it87_wdt.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/machzwd.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/nv_tco.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_pci.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/pcwd_usb.ko 2024-01-16T14:49:17.965 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sch311x_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/softdog.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/sp5100_tco.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/via_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83877f_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/w83977f_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/wdt_pci.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/watchdog/xen_wdt.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-acpi-processor.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-evtchn.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntalloc.ko 2024-01-16T14:49:17.966 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-gntdev.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xen-privcmd.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/drivers/xen/xenfs/xenfs.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/9p/9p.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/affs/affs.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/befs/befs.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/binfmt_misc.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/btrfs/btrfs.ko 2024-01-16T14:49:17.967 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cachefiles/cachefiles.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ceph/ceph.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/coda/coda.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/cramfs/cramfs.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/dlm/dlm.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ecryptfs/ecryptfs.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/f2fs/f2fs.ko 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat 2024-01-16T14:49:17.968 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/fat.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/msdos.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fat/vfat.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fscache/fscache.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/cuse.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/fuse/fuse.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/gfs2/gfs2.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfs/hfs.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/hfsplus/hfsplus.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/isofs/isofs.ko 2024-01-16T14:49:17.969 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/jfs/jfs.ko 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/lockd/lockd.ko 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/minix/minix.ko 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/netfs/netfs.ko 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout 2024-01-16T14:49:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfs.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv3.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs/nfsv4.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/grace.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfs_common/nfs_acl.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nfsd/nfsd.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nilfs2/nilfs2.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-celtic.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-centeuro.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-croatian.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-cyrillic.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-gaelic.ko 2024-01-16T14:49:17.971 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-greek.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-iceland.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-inuit.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-roman.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-romanian.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/mac-turkish.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1250.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1251.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp1255.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp737.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp775.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp850.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp852.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp855.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp857.ko 2024-01-16T14:49:17.972 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp860.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp861.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp862.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp863.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp864.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp865.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp866.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp869.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp874.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp932.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp936.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp949.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_cp950.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_euc-jp.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-1.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-13.ko 2024-01-16T14:49:17.973 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-14.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-15.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-2.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-3.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-4.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-5.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-6.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-7.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_iso8859-9.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-r.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-ru.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_koi8-u.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_ucs2_utils.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/nls/nls_utf8.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-01-16T14:49:17.974 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/overlayfs/overlay.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/pstore/ramoops.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/reiserfs/reiserfs.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/romfs/romfs.ko 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb 2024-01-16T14:49:17.975 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/client/cifs.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_arc4.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/smb/common/cifs_md4.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/squashfs/squashfs.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/sysv/sysv.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/udf/udf.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/ufs/ufs.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/fs/xfs/xfs.ko 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu 2024-01-16T14:49:17.976 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/rcu/rcutorture.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/torture.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/asn1_encoder.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc-itu-t.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crc8.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libarc4.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/crypto/libdes.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/libcrc32c.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lru_cache.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4_compress.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/lz4/lz4hc_compress.ko 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math 2024-01-16T14:49:17.977 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/math/cordic.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/raid6/raid6_pq.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/reed_solomon/reed_solomon.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_bm.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_fsm.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/ts_kmp.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/lib/zstd/zstd_compress.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/mm/hwpoison-inject.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/6lowpan.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_dest.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_fragment.ko 2024-01-16T14:49:17.978 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_hop.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_ipv6.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_mobility.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_routing.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/6lowpan/nhc_udp.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/garp.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/mrp.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/p8022.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/psnap.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/802/stp.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/8021q/8021q.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet.ko 2024-01-16T14:49:17.979 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_fd.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_rdma.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/9p/9pnet_virtio.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/batman-adv/batman-adv.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/br_netfilter.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/bridge.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_among.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arp.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip.ko 2024-01-16T14:49:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_limit.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_log.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_snat.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_stp.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/ebtables.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-01-16T14:49:17.981 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ceph/libceph.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/failover.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/core/pktgen.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_diag.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv4.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dccp/dccp_ipv6.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dns_resolver/dns_resolver.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/dsa_core.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_brcm.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_dsa.ko 2024-01-16T14:49:17.982 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_none.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/dsa/tag_trailer.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ieee802154/ieee802154_socket.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ah4.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/esp4.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/fou.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/gre.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/inet_diag.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_gre.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_tunnel.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ip_vti.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipcomp.ko 2024-01-16T14:49:17.983 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/ipip.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arp_tables.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/iptable_security.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-01-16T14:49:17.984 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_bic.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_dctcp.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_diag.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_highspeed.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_htcp.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_hybla.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_illinois.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_lp.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_scalable.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_vegas.ko 2024-01-16T14:49:17.985 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_veno.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_westwood.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tcp_yeah.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/tunnel4.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_diag.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/udp_tunnel.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv4/xfrm4_tunnel.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ah6.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/esp6.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/fou6.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_tunnel.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ip6_vti.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/ipcomp6.ko 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter 2024-01-16T14:49:17.986 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-01-16T14:49:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/sit.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/tunnel6.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/ipv6/xfrm6_tunnel.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/key/af_key.ko 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp 2024-01-16T14:49:17.988 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_core.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_debugfs.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_eth.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ip6.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_netlink.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/l2tp/l2tp_ppp.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/llc/llc.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mac802154/mac802154.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/mpls/mpls_gso.ko 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset 2024-01-16T14:49:17.989 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-01-16T14:49:17.990 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-01-16T14:49:17.991 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conncount.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_h323.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_irc.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sane.ko 2024-01-16T14:49:17.992 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_sip.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_dup_netdev.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_flow_table_inet.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_log_syslog.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_amanda.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_ftp.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_irc.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_sip.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_nat_tftp.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_synproxy_core.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nf_tables.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink.ko 2024-01-16T14:49:17.993 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_acct.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_log.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_osf.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nfnetlink_queue.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_chain_nat.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_compat.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_ct.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_dup_netdev.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fib.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_fwd_netdev.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_hash.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_limit.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_log.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_masq.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_nat.ko 2024-01-16T14:49:17.994 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_numgen.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_queue.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_quota.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_redir.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_reject_inet.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_socket.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/nft_tunnel.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_AUDIT.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CHECKSUM.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CLASSIFY.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_CT.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_DSCP.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HL.ko 2024-01-16T14:49:17.995 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_HMARK.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_IDLETIMER.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LED.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_LOG.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_MASQUERADE.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NETMAP.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFLOG.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_NFQUEUE.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_RATEEST.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_REDIRECT.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_SECMARK.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPMSS.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TEE.ko 2024-01-16T14:49:17.996 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TPROXY.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_TRACE.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_addrtype.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_bpf.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cgroup.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cluster.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_comment.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connbytes.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlabel.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connlimit.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_connmark.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_conntrack.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_cpu.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dccp.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_devgroup.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_dscp.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ecn.ko 2024-01-16T14:49:17.997 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_esp.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hashlimit.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_helper.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_hl.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipcomp.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_iprange.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_ipvs.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_l2tp.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_length.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_limit.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mac.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_mark.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_multiport.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nat.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_nfacct.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_osf.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_owner.ko 2024-01-16T14:49:17.998 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_physdev.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_pkttype.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_policy.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_quota.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_rateest.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_realm.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_recent.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_sctp.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_set.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_socket.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_state.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_statistic.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_string.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_tcpmss.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_time.ko 2024-01-16T14:49:17.999 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netfilter/xt_u32.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/netlink/netlink_diag.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/nsh/nsh.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/openvswitch.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-geneve.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-gre.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/openvswitch/vport-vxlan.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/packet/af_packet_diag.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_rdma.ko 2024-01-16T14:49:18.000 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/rds/rds_tcp.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_csum.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_gact.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_ipt.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_mirred.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_nat.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_pedit.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_police.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_simple.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_skbedit.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/act_vlan.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_basic.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_bpf.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_flow.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_fw.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_route.ko 2024-01-16T14:49:18.001 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/cls_u32.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_cmp.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_ipset.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_meta.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_nbyte.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_text.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/em_u32.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_choke.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_codel.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_drr.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_fq.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_gred.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hfsc.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_hhf.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_htb.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_ingress.ko 2024-01-16T14:49:18.002 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_mqprio_lib.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_multiq.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_netem.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_pie.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_plug.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_prio.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_qfq.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_red.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfb.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_sfq.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_tbf.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sched/sch_teql.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sctp/sctp_diag.ko 2024-01-16T14:49:18.003 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/sunrpc.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/diag.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/tipc/tipc.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/unix/unix_diag.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock.ko 2024-01-16T14:49:18.004 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_diag.ko 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/vmw_vsock/vsock_loopback.ko 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/net/xfrm/xfrm_ipcomp.ko 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/security/keys/trusted-keys/trusted.ko 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/kernel/virt/lib/irqbypass.ko 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.alias.bin 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin 2024-01-16T14:49:18.005 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.alias.bin 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.bin 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.builtin.modinfo 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.dep.bin 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.devname 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.order 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.softdep 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols 2024-01-16T14:49:18.006 INFO:teuthology.orchestra.run.smithi053.stdout:/lib/modules/6.7.0-g2668949e0971/modules.symbols.bin 2024-01-16T14:49:18.007 DEBUG:teuthology.task.kernel:get_image_version: 6.7.0-g2668949e0971 2024-01-16T14:49:18.007 DEBUG:teuthology.orchestra.run.smithi053:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-01-16T14:49:18.030 INFO:teuthology.orchestra.run.smithi053.stdout:postinstall scriptlet (using /bin/sh): 2024-01-16T14:49:18.030 INFO:teuthology.orchestra.run.smithi053.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.7.0-g2668949e0971 -a -r /boot/System.map-6.7.0-g2668949e0971 ]; then 2024-01-16T14:49:18.030 INFO:teuthology.orchestra.run.smithi053.stdout:cp /boot/vmlinuz-6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:cp /boot/System.map-6.7.0-g2668949e0971 /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:rm -f /boot/vmlinuz-6.7.0-g2668949e0971 /boot/System.map-6.7.0-g2668949e0971 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/installkernel 6.7.0-g2668949e0971 /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:rm -f /boot/.vmlinuz-6.7.0-g2668949e0971-rpm /boot/.System.map-6.7.0-g2668949e0971-rpm 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:fi 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:preuninstall scriptlet (using /bin/sh): 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:new-kernel-pkg --remove 6.7.0-g2668949e0971 --rminitrd --initrdfile=/boot/initramfs-6.7.0-g2668949e0971.img 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:kernel-install remove 6.7.0-g2668949e0971 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:fi 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:postuninstall scriptlet (using /bin/sh): 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:if [ -x /sbin/update-bootloader ]; then 2024-01-16T14:49:18.031 INFO:teuthology.orchestra.run.smithi053.stdout:/sbin/update-bootloader --remove 6.7.0-g2668949e0971 2024-01-16T14:49:18.032 INFO:teuthology.orchestra.run.smithi053.stdout:fi 2024-01-16T14:49:18.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo rpm -qi grub2-tools 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Name : grub2-tools 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Epoch : 1 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Version : 2.06 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Release : 68.el9 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Architecture: x86_64 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Group : Unspecified 2024-01-16T14:49:18.111 INFO:teuthology.orchestra.run.smithi053.stdout:Size : 8264254 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:License : GPLv3+ 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Packager : builder@centos.org 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Vendor : CentOS 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Summary : Support tools for GRUB. 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:Description : 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout:hardware devices. 2024-01-16T14:49:18.112 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:49:18.113 INFO:teuthology.orchestra.run.smithi053.stdout:This subpackage provides tools for support of all platforms. 2024-01-16T14:49:18.113 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-16T14:49:18.113 INFO:teuthology.task.kernel:Updating grub on smithi053 to boot 6.7.0-g2668949e0971 2024-01-16T14:49:18.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-16T14:49:18.680 INFO:teuthology.orchestra.run.smithi053.stderr:Generating grub configuration file ... 2024-01-16T14:49:20.070 INFO:teuthology.orchestra.run.smithi053.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-01-16T14:49:20.128 INFO:teuthology.orchestra.run.smithi053.stderr:done 2024-01-16T14:49:20.130 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-01-16T14:49:20.145 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.18YtPPtqrl 2024-01-16T14:49:20.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.18YtPPtqrl 2024-01-16T14:49:20.217 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.18YtPPtqrl 2024-01-16T14:49:20.358 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.18YtPPtqrl is 6KB 2024-01-16T14:49:20.406 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.18YtPPtqrl 2024-01-16T14:49:20.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo /bin/ls /boot/loader/entries || true 2024-01-16T14:49:20.489 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-01-16T14:49:20.489 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-01-16T14:49:20.489 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-01-16T14:49:20.489 INFO:teuthology.orchestra.run.smithi053.stdout:15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971.conf 2024-01-16T14:49:20.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.7.0-g2668949e0971 2024-01-16T14:49:20.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo shutdown -r now 2024-01-16T14:49:22.747 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:49:22.748 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:49:23.233 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-01-16T14:49:23.935 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:49:23.936 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "6.7.0-g2668949e0971"... 2024-01-16T14:49:23.936 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-01-16T14:49:23.988 INFO:teuthology.orchestra.run.smithi146.stdout:6.7.0-g2668949e0971 2024-01-16T14:49:23.989 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:49:23.989 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:49:23.989 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:49:24.989 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-01-16T14:49:24.990 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:49:25.124 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-01-16T14:49:25.171 DEBUG:teuthology.parallel:result is None 2024-01-16T14:49:50.635 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-01-16T14:49:50.635 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-16T14:49:50.636 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:50:09.084 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-01-16T14:50:18.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-16T14:50:18.086 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:50:36.476 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.53 2024-01-16T14:50:48.477 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-16T14:50:48.478 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:50:50.825 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-01-16T14:50:51.689 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-16T14:50:51.689 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "6.7.0-g2668949e0971"... 2024-01-16T14:50:51.690 DEBUG:teuthology.orchestra.run.smithi053:> uname -r 2024-01-16T14:50:51.746 INFO:teuthology.orchestra.run.smithi053.stdout:6.7.0-g2668949e0971 2024-01-16T14:50:51.746 DEBUG:teuthology.task.kernel:current kernel version is 6.7.0-g2668949e0971 vs 6.7.0-g2668949e0971 2024-01-16T14:50:51.746 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-16T14:50:51.746 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-16T14:50:52.747 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-01-16T14:50:52.747 DEBUG:teuthology.orchestra.run.smithi053:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-16T14:50:52.863 INFO:teuthology.orchestra.run.smithi053.stdout:ttyS1 2024-01-16T14:50:52.909 DEBUG:teuthology.parallel:result is None 2024-01-16T14:50:52.910 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-16T14:50:52.917 INFO:teuthology.task.internal:Creating test directory... 2024-01-16T14:50:52.917 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:50:52.922 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-16T14:50:52.978 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-16T14:50:52.988 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-16T14:50:52.994 INFO:teuthology.task.internal:Creating archive directory... 2024-01-16T14:50:52.995 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:50:53.002 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-16T14:50:53.073 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-16T14:50:53.079 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-16T14:50:53.079 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:50:53.084 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-16T14:50:53.160 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:53.195 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:53.206 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:53.251 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-16T14:50:53.256 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-16T14:50:53.262 INFO:teuthology.task.internal:Configuring sudo... 2024-01-16T14:50:53.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:50:53.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-16T14:50:53.360 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-16T14:50:53.369 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-16T14:50:53.369 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:50:53.376 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-16T14:50:53.431 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:53.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:53.607 DEBUG:teuthology.orchestra.run.smithi053:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:53.674 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:53.762 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:50:53.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:50:53.977 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:54.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-16T14:50:54.171 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:54.256 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-16T14:50:54.350 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:50:54.350 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-16T14:50:54.451 DEBUG:teuthology.orchestra.run.smithi053:> sudo service rsyslog restart 2024-01-16T14:50:54.459 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-01-16T14:50:54.573 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:50:54.616 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T14:50:54.996 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-16T14:50:55.004 INFO:teuthology.task.internal:Starting timer... 2024-01-16T14:50:55.004 INFO:teuthology.run_tasks:Running task pcp... 2024-01-16T14:50:55.014 INFO:teuthology.run_tasks:Running task selinux... 2024-01-16T14:50:55.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo service auditd rotate 2024-01-16T14:50:55.140 INFO:teuthology.orchestra.run.smithi053.stdout:Rotating logs: 2024-01-16T14:50:55.145 DEBUG:teuthology.orchestra.run.smithi146:> sudo service auditd rotate 2024-01-16T14:50:55.270 INFO:teuthology.orchestra.run.smithi146.stdout:Rotating logs: 2024-01-16T14:50:55.276 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-16T14:50:55.276 DEBUG:teuthology.orchestra.run.smithi053:> /usr/sbin/getenforce 2024-01-16T14:50:55.337 INFO:teuthology.orchestra.run.smithi053.stdout:Permissive 2024-01-16T14:50:55.338 DEBUG:teuthology.orchestra.run.smithi146:> /usr/sbin/getenforce 2024-01-16T14:50:55.402 INFO:teuthology.orchestra.run.smithi146.stdout:Permissive 2024-01-16T14:50:55.403 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi053.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi146.front.sepia.ceph.com': 'permissive'} 2024-01-16T14:50:55.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:50:55.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:50:55.500 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T14:50:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T14:50:55.597 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-16T14:50:55.597 DEBUG:teuthology.orchestra.run.smithi053:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:50:55.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo /usr/sbin/setenforce permissive 2024-01-16T14:50:55.784 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-16T14:50:55.793 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-16T14:50:55.807 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-16T14:50:55.808 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi053.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-16T14:56:43.662 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi053.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-01-16T14:56:43.664 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-16T14:56:43.665 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi053.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:56:43.809 DEBUG:teuthology.orchestra.run.smithi053:> true 2024-01-16T14:56:43.967 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi053.front.sepia.ceph.com' 2024-01-16T14:56:43.968 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:56:43.968 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-16T14:56:44.112 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-01-16T14:56:44.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-01-16T14:56:44.260 INFO:teuthology.run_tasks:Running task clock... 2024-01-16T14:56:44.270 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-16T14:56:44.270 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:56:44.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:56:44.278 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-16T14:56:44.278 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T14:56:44.378 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:56:44.381 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-16T14:56:44.431 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:56:44.435 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-16T14:56:44.521 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: ntpd: command not found 2024-01-16T14:56:44.541 INFO:teuthology.orchestra.run.smithi053.stderr:sudo: ntpd: command not found 2024-01-16T14:56:44.562 INFO:teuthology.orchestra.run.smithi146.stdout:506 Cannot talk to daemon 2024-01-16T14:56:44.589 INFO:teuthology.orchestra.run.smithi053.stdout:506 Cannot talk to daemon 2024-01-16T14:56:44.614 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:56:44.649 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-16T14:56:44.665 INFO:teuthology.orchestra.run.smithi146.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:56:44.707 INFO:teuthology.orchestra.run.smithi053.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-16T14:56:44.791 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:56:44.800 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:56:44.800 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-01-16T14:56:44.800 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.800 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.800 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.800 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.835 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-01-16T14:56:44.844 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T14:56:44.844 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-01-16T14:56:44.844 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.844 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.844 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.844 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T14:56:44.846 INFO:teuthology.run_tasks:Running task install... 2024-01-16T14:56:44.854 DEBUG:teuthology.task.install:project ceph 2024-01-16T14:56:44.854 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}} 2024-01-16T14:56:44.854 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'} 2024-01-16T14:56:44.855 INFO:teuthology.task.install:Using flavor: default 2024-01-16T14:56:44.860 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-16T14:56:44.860 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-01-16T14:56:44.861 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:56:44.861 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:56:44.863 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'tag': None, 'wait_for_package': False} 2024-01-16T14:56:44.863 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:56:45.021 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:56:45.022 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:56:45.095 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/ 2024-01-16T14:56:45.096 INFO:teuthology.task.install.rpm:Package version is 18.0.0-7629.g73a4532f 2024-01-16T14:56:45.133 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:56:45.133 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:56:45.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:56:45.238 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:56:45.238 DEBUG:teuthology.orchestra.run.smithi053:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:56:45.271 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-clone-stats/73a4532fc4d97adee0b985d4d42874ea351a1c86/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-16T14:56:45.271 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:56:45.271 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-16T14:56:45.370 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-01-16T14:56:45.370 DEBUG:teuthology.orchestra.run.smithi146:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/73a4532fc4d97adee0b985d4d42874ea351a1c86/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-16T14:56:45.382 DEBUG:teuthology.orchestra.run.smithi053:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:56:45.508 DEBUG:teuthology.orchestra.run.smithi146:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-16T14:56:45.522 DEBUG:teuthology.orchestra.run.smithi053:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:56:45.618 INFO:teuthology.orchestra.run.smithi053.stdout:check_obsoletes = 1 2024-01-16T14:56:45.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum clean all 2024-01-16T14:56:45.638 DEBUG:teuthology.orchestra.run.smithi146:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-16T14:56:45.722 INFO:teuthology.orchestra.run.smithi146.stdout:check_obsoletes = 1 2024-01-16T14:56:45.726 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum clean all 2024-01-16T14:56:45.983 INFO:teuthology.orchestra.run.smithi053.stdout:73 files removed 2024-01-16T14:56:46.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:56:46.091 INFO:teuthology.orchestra.run.smithi146.stdout:73 files removed 2024-01-16T14:56:46.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-01-16T14:56:47.112 INFO:teuthology.orchestra.run.smithi146.stdout:ceph packages for x86_64 165 kB/s | 78 kB 00:00 2024-01-16T14:56:47.234 INFO:teuthology.orchestra.run.smithi053.stdout:ceph packages for x86_64 114 kB/s | 78 kB 00:00 2024-01-16T14:56:47.696 INFO:teuthology.orchestra.run.smithi146.stdout:ceph noarch packages 19 kB/s | 9.5 kB 00:00 2024-01-16T14:56:47.785 INFO:teuthology.orchestra.run.smithi053.stdout:ceph noarch packages 21 kB/s | 9.5 kB 00:00 2024-01-16T14:56:48.138 INFO:teuthology.orchestra.run.smithi146.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-01-16T14:56:48.310 INFO:teuthology.orchestra.run.smithi053.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2024-01-16T14:56:48.950 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - AppStream 8.6 MB/s | 6.4 MB 00:00 2024-01-16T14:56:49.206 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - AppStream 7.8 MB/s | 6.4 MB 00:00 2024-01-16T14:56:51.123 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS-9 - BaseOS 2.9 MB/s | 2.0 MB 00:00 2024-01-16T14:56:51.250 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS-9 - BaseOS 1.9 MB/s | 2.0 MB 00:01 2024-01-16T14:56:53.412 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - BaseOS 4.5 MB/s | 8.0 MB 00:01 2024-01-16T14:56:53.827 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - BaseOS 3.5 MB/s | 8.0 MB 00:02 2024-01-16T14:56:55.787 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-01-16T14:56:55.963 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - AppStream 14 MB/s | 19 MB 00:01 2024-01-16T14:57:00.596 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - CRB 6.5 MB/s | 5.9 MB 00:00 2024-01-16T14:57:02.106 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 5.9 MB 00:02 2024-01-16T14:57:02.305 INFO:teuthology.orchestra.run.smithi053.stdout:CentOS Stream 9 - Extras packages 40 kB/s | 15 kB 00:00 2024-01-16T14:57:04.373 INFO:teuthology.orchestra.run.smithi146.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 15 kB 00:00 2024-01-16T14:57:04.465 INFO:teuthology.orchestra.run.smithi053.stdout:Extra Packages for Enterprise Linux 10 MB/s | 20 MB 00:01 2024-01-16T14:57:09.263 INFO:teuthology.orchestra.run.smithi146.stdout:Extra Packages for Enterprise Linux 4.3 MB/s | 20 MB 00:04 2024-01-16T14:57:10.365 INFO:teuthology.orchestra.run.smithi053.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-01-16T14:57:13.204 INFO:teuthology.orchestra.run.smithi053.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:57:13.206 INFO:teuthology.orchestra.run.smithi053.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:57:13.224 INFO:teuthology.orchestra.run.smithi053.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:57:13.225 INFO:teuthology.orchestra.run.smithi053.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:57:13.226 INFO:teuthology.orchestra.run.smithi053.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:57:13.228 INFO:teuthology.orchestra.run.smithi053.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:57:13.230 INFO:teuthology.orchestra.run.smithi053.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:57:13.234 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:57:13.235 INFO:teuthology.orchestra.run.smithi053.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:57:13.321 INFO:teuthology.orchestra.run.smithi053.stdout:Dependencies resolved. 2024-01-16T14:57:13.340 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-01-16T14:57:13.340 INFO:teuthology.orchestra.run.smithi053.stdout: Package Arch Version Repository Size 2024-01-16T14:57:13.340 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-01-16T14:57:13.340 INFO:teuthology.orchestra.run.smithi053.stdout:Installing: 2024-01-16T14:57:13.340 INFO:teuthology.orchestra.run.smithi053.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:57:13.340 INFO:teuthology.orchestra.run.smithi053.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:57:13.341 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:57:13.342 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrading: 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:57:13.343 INFO:teuthology.orchestra.run.smithi053.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout:Installing dependencies: 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:57:13.344 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:57:13.345 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:57:13.346 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:57:13.347 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:57:13.348 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:57:13.349 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:57:13.350 INFO:teuthology.orchestra.run.smithi053.stdout:Installing weak dependencies: 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction Summary 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout:============================================================================================ 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout:Install 135 Packages 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout:Upgrade 16 Packages 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout:Total download size: 204 M 2024-01-16T14:57:13.351 INFO:teuthology.orchestra.run.smithi053.stdout:Downloading Packages: 2024-01-16T14:57:14.058 INFO:teuthology.orchestra.run.smithi053.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 22 kB/s | 6.6 kB 00:00 2024-01-16T14:57:14.725 INFO:teuthology.orchestra.run.smithi053.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 1.3 MB/s | 861 kB 00:00 2024-01-16T14:57:14.974 INFO:teuthology.orchestra.run.smithi146.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-01-16T14:57:15.134 INFO:teuthology.orchestra.run.smithi053.stdout:(3/151): ceph-common-18.0.0-7629.g73a4532f.el9. 14 MB/s | 19 MB 00:01 2024-01-16T14:57:15.151 INFO:teuthology.orchestra.run.smithi053.stdout:(4/151): ceph-immutable-object-cache-18.0.0-762 332 kB/s | 141 kB 00:00 2024-01-16T14:57:15.293 INFO:teuthology.orchestra.run.smithi053.stdout:(5/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 13 MB/s | 2.1 MB 00:00 2024-01-16T14:57:15.401 INFO:teuthology.orchestra.run.smithi053.stdout:(6/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 3.2 MB/s | 5.2 MB 00:01 2024-01-16T14:57:15.568 INFO:teuthology.orchestra.run.smithi053.stdout:(7/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 16 MB/s | 4.4 MB 00:00 2024-01-16T14:57:16.143 INFO:teuthology.orchestra.run.smithi053.stdout:(8/151): ceph-radosgw-18.0.0-7629.g73a4532f.el9 15 MB/s | 8.6 MB 00:00 2024-01-16T14:57:16.210 INFO:teuthology.orchestra.run.smithi053.stdout:(9/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 1.4 MB/s | 1.5 MB 00:01 2024-01-16T14:57:16.235 INFO:teuthology.orchestra.run.smithi053.stdout:(10/151): ceph-selinux-18.0.0-7629.g73a4532f.el 276 kB/s | 25 kB 00:00 2024-01-16T14:57:16.302 INFO:teuthology.orchestra.run.smithi053.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 3.2 MB/s | 217 kB 00:00 2024-01-16T14:57:16.411 INFO:teuthology.orchestra.run.smithi053.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 289 kB/s | 31 kB 00:00 2024-01-16T14:57:16.903 INFO:teuthology.orchestra.run.smithi053.stdout:(13/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 1.4 MB/s | 719 kB 00:00 2024-01-16T14:57:16.987 INFO:teuthology.orchestra.run.smithi053.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 1.9 MB/s | 163 kB 00:00 2024-01-16T14:57:17.062 INFO:teuthology.orchestra.run.smithi053.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 1.7 MB/s | 127 kB 00:00 2024-01-16T14:57:17.154 INFO:teuthology.orchestra.run.smithi053.stdout:(16/151): libradosstriper1-18.0.0-7629.g73a4532 5.1 MB/s | 476 kB 00:00 2024-01-16T14:57:17.588 INFO:teuthology.orchestra.run.smithi053.stdout:(17/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 8.1 MB/s | 18 MB 00:02 2024-01-16T14:57:17.676 INFO:teuthology.orchestra.run.smithi146.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:57:17.677 INFO:teuthology.orchestra.run.smithi146.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-01-16T14:57:17.695 INFO:teuthology.orchestra.run.smithi146.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-01-16T14:57:17.696 INFO:teuthology.orchestra.run.smithi146.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-01-16T14:57:17.697 INFO:teuthology.orchestra.run.smithi146.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-01-16T14:57:17.698 INFO:teuthology.orchestra.run.smithi146.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-01-16T14:57:17.701 INFO:teuthology.orchestra.run.smithi146.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-01-16T14:57:17.701 INFO:teuthology.orchestra.run.smithi053.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 402 kB/s | 45 kB 00:00 2024-01-16T14:57:17.704 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-01-16T14:57:17.705 INFO:teuthology.orchestra.run.smithi146.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-01-16T14:57:17.787 INFO:teuthology.orchestra.run.smithi146.stdout:Dependencies resolved. 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout: Package Arch Version Repository Size 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout:Installing: 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout: ceph x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 6.6 k 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 5.2 M 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 861 k 2024-01-16T14:57:17.807 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 141 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 1.5 M 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 133 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 1.8 M 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 7.4 M 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 50 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 8.6 M 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 41 M 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 262 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 778 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 217 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 18 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-01-16T14:57:17.808 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 31 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 719 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 127 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel x86_64 2.37.4-15.el9 CentOS-AppStream 24 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 161 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 321 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 298 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 99 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 85 k 2024-01-16T14:57:17.809 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 171 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrading: 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager x86_64 1:1.45.9-1.el9 CentOS-BaseOS 2.2 M 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm x86_64 1:1.45.9-1.el9 CentOS-BaseOS 1.8 M 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team x86_64 1:1.45.9-1.el9 CentOS-BaseOS 42 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui x86_64 1:1.45.9-1.el9 CentOS-BaseOS 246 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: librados2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.3 M 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 3.0 M 2024-01-16T14:57:17.810 INFO:teuthology.orchestra.run.smithi146.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout:Installing dependencies: 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 19 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 28 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 2.1 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 246 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.4 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 18 M 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts noarch 2:18.0.0-7629.g73a4532f.el9 ceph-noarch 15 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 25 k 2024-01-16T14:57:17.811 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 163 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 476 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-01-16T14:57:17.812 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2 x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 4.8 M 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-01-16T14:57:17.813 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 45 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common x86_64 2:18.0.0-7629.g73a4532f.el9 ceph 129 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth noarch 1:2.26.0-1.el9 epel 202 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-01-16T14:57:17.814 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-01-16T14:57:17.815 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-01-16T14:57:17.816 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout:Installing weak dependencies: 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction Summary 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================================ 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout:Install 135 Packages 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout:Upgrade 16 Packages 2024-01-16T14:57:17.817 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:57:17.818 INFO:teuthology.orchestra.run.smithi053.stdout:(19/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 7.3 MB/s | 4.8 MB 00:00 2024-01-16T14:57:17.818 INFO:teuthology.orchestra.run.smithi146.stdout:Total download size: 204 M 2024-01-16T14:57:17.818 INFO:teuthology.orchestra.run.smithi146.stdout:Downloading Packages: 2024-01-16T14:57:17.846 INFO:teuthology.orchestra.run.smithi053.stdout:(20/151): python3-ceph-common-18.0.0-7629.g73a4 893 kB/s | 129 kB 00:00 2024-01-16T14:57:17.980 INFO:teuthology.orchestra.run.smithi053.stdout:(21/151): python3-cephfs-18.0.0-7629.g73a4532f. 973 kB/s | 161 kB 00:00 2024-01-16T14:57:18.014 INFO:teuthology.orchestra.run.smithi053.stdout:(22/151): python3-rados-18.0.0-7629.g73a4532f.e 1.9 MB/s | 321 kB 00:00 2024-01-16T14:57:18.047 INFO:teuthology.orchestra.run.smithi053.stdout:(23/151): python3-rbd-18.0.0-7629.g73a4532f.el9 4.4 MB/s | 298 kB 00:00 2024-01-16T14:57:18.098 INFO:teuthology.orchestra.run.smithi053.stdout:(24/151): python3-rgw-18.0.0-7629.g73a4532f.el9 1.2 MB/s | 99 kB 00:00 2024-01-16T14:57:18.146 INFO:teuthology.orchestra.run.smithi053.stdout:(25/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 876 kB/s | 85 kB 00:00 2024-01-16T14:57:18.215 INFO:teuthology.orchestra.run.smithi053.stdout:(26/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 2.5 MB/s | 171 kB 00:00 2024-01-16T14:57:18.273 INFO:teuthology.orchestra.run.smithi053.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 488 kB/s | 28 kB 00:00 2024-01-16T14:57:18.340 INFO:teuthology.orchestra.run.smithi053.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 2.0 MB/s | 133 kB 00:00 2024-01-16T14:57:18.524 INFO:teuthology.orchestra.run.smithi053.stdout:(29/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 7.1 MB/s | 3.0 MB 00:00 2024-01-16T14:57:18.650 INFO:teuthology.orchestra.run.smithi053.stdout:(30/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 5.7 MB/s | 1.8 MB 00:00 2024-01-16T14:57:18.717 INFO:teuthology.orchestra.run.smithi053.stdout:(31/151): ceph-mgr-modules-core-18.0.0-7629.g73 3.7 MB/s | 246 kB 00:00 2024-01-16T14:57:18.784 INFO:teuthology.orchestra.run.smithi053.stdout:(32/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 764 kB/s | 50 kB 00:00 2024-01-16T14:57:18.816 INFO:teuthology.orchestra.run.smithi146.stdout:(1/151): ceph-18.0.0-7629.g73a4532f.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-01-16T14:57:18.851 INFO:teuthology.orchestra.run.smithi053.stdout:(33/151): ceph-prometheus-alerts-18.0.0-7629.g7 230 kB/s | 15 kB 00:00 2024-01-16T14:57:19.069 INFO:teuthology.orchestra.run.smithi053.stdout:(34/151): ceph-volume-18.0.0-7629.g73a4532f.el9 1.2 MB/s | 262 kB 00:00 2024-01-16T14:57:19.368 INFO:teuthology.orchestra.run.smithi053.stdout:(35/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 2.6 MB/s | 778 kB 00:00 2024-01-16T14:57:19.526 INFO:teuthology.orchestra.run.smithi053.stdout:(36/151): ceph-mgr-diskprediction-local-18.0.0- 7.4 MB/s | 7.4 MB 00:01 2024-01-16T14:57:19.552 INFO:teuthology.orchestra.run.smithi053.stdout:(37/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 100 kB/s | 18 kB 00:00 2024-01-16T14:57:19.608 INFO:teuthology.orchestra.run.smithi146.stdout:(2/151): ceph-fuse-18.0.0-7629.g73a4532f.el9.x8 1.1 MB/s | 861 kB 00:00 2024-01-16T14:57:19.709 INFO:teuthology.orchestra.run.smithi146.stdout:(3/151): ceph-immutable-object-cache-18.0.0-762 1.4 MB/s | 141 kB 00:00 2024-01-16T14:57:19.786 INFO:teuthology.orchestra.run.smithi053.stdout:(38/151): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 662 kB 00:00 2024-01-16T14:57:19.835 INFO:teuthology.orchestra.run.smithi053.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 3.2 MB/s | 920 kB 00:00 2024-01-16T14:57:19.861 INFO:teuthology.orchestra.run.smithi053.stdout:(40/151): boost-program-options-1.75.0-8.el9.x8 1.5 MB/s | 106 kB 00:00 2024-01-16T14:57:19.886 INFO:teuthology.orchestra.run.smithi053.stdout:(41/151): elfutils-libelf-devel-0.190-2.el9.x86 416 kB/s | 20 kB 00:00 2024-01-16T14:57:19.951 INFO:teuthology.orchestra.run.smithi053.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 3.4 MB/s | 307 kB 00:00 2024-01-16T14:57:19.979 INFO:teuthology.orchestra.run.smithi053.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 358 kB/s | 32 kB 00:00 2024-01-16T14:57:20.221 INFO:teuthology.orchestra.run.smithi053.stdout:(44/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 11 MB/s | 3.0 MB 00:00 2024-01-16T14:57:20.246 INFO:teuthology.orchestra.run.smithi053.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 67 kB/s | 17 kB 00:00 2024-01-16T14:57:20.293 INFO:teuthology.orchestra.run.smithi053.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 218 kB/s | 10 kB 00:00 2024-01-16T14:57:20.321 INFO:teuthology.orchestra.run.smithi053.stdout:(47/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-01-16T14:57:20.346 INFO:teuthology.orchestra.run.smithi053.stdout:(48/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 871 kB/s | 44 kB 00:00 2024-01-16T14:57:20.375 INFO:teuthology.orchestra.run.smithi146.stdout:(4/151): ceph-mds-18.0.0-7629.g73a4532f.el9.x86 3.1 MB/s | 2.1 MB 00:00 2024-01-16T14:57:20.405 INFO:teuthology.orchestra.run.smithi053.stdout:(49/151): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-01-16T14:57:20.455 INFO:teuthology.orchestra.run.smithi053.stdout:(50/151): libtool-2.4.6-45.el9.x86_64.rpm 5.5 MB/s | 578 kB 00:00 2024-01-16T14:57:20.481 INFO:teuthology.orchestra.run.smithi053.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 353 kB/s | 24 kB 00:00 2024-01-16T14:57:20.506 INFO:teuthology.orchestra.run.smithi053.stdout:(52/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 942 kB/s | 46 kB 00:00 2024-01-16T14:57:20.540 INFO:teuthology.orchestra.run.smithi053.stdout:(53/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-01-16T14:57:20.573 INFO:teuthology.orchestra.run.smithi053.stdout:(54/151): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-01-16T14:57:20.592 INFO:teuthology.orchestra.run.smithi146.stdout:(5/151): ceph-base-18.0.0-7629.g73a4532f.el9.x8 2.5 MB/s | 5.2 MB 00:02 2024-01-16T14:57:20.599 INFO:teuthology.orchestra.run.smithi053.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 610 kB/s | 34 kB 00:00 2024-01-16T14:57:20.774 INFO:teuthology.orchestra.run.smithi053.stdout:(56/151): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-01-16T14:57:20.801 INFO:teuthology.orchestra.run.smithi146.stdout:(6/151): ceph-mgr-18.0.0-7629.g73a4532f.el9.x86 3.4 MB/s | 1.5 MB 00:00 2024-01-16T14:57:20.882 INFO:teuthology.orchestra.run.smithi053.stdout:(57/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-01-16T14:57:20.941 INFO:teuthology.orchestra.run.smithi053.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 9.9 MB/s | 1.5 MB 00:00 2024-01-16T14:57:20.983 INFO:teuthology.orchestra.run.smithi053.stdout:(59/151): postgresql-contrib-13.11-1.el9.x86_64 9.1 MB/s | 811 kB 00:00 2024-01-16T14:57:21.000 INFO:teuthology.orchestra.run.smithi053.stdout:(60/151): postgresql-private-libs-13.11-1.el9.x 2.6 MB/s | 134 kB 00:00 2024-01-16T14:57:21.267 INFO:teuthology.orchestra.run.smithi053.stdout:(61/151): postgresql-server-13.11-1.el9.x86_64. 20 MB/s | 5.7 MB 00:00 2024-01-16T14:57:21.418 INFO:teuthology.orchestra.run.smithi053.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2024-01-16T14:57:21.443 INFO:teuthology.orchestra.run.smithi053.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 1.4 MB/s | 228 kB 00:00 2024-01-16T14:57:21.476 INFO:teuthology.orchestra.run.smithi053.stdout:(64/151): python3-jsonpatch-1.21-16.el9.noarch. 761 kB/s | 24 kB 00:00 2024-01-16T14:57:21.502 INFO:teuthology.orchestra.run.smithi053.stdout:(65/151): python3-jsonpointer-2.0-4.el9.noarch. 317 kB/s | 18 kB 00:00 2024-01-16T14:57:21.535 INFO:teuthology.orchestra.run.smithi053.stdout:(66/151): python3-markupsafe-1.1.1-12.el9.x86_6 988 kB/s | 32 kB 00:00 2024-01-16T14:57:21.561 INFO:teuthology.orchestra.run.smithi053.stdout:(67/151): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-01-16T14:57:21.751 INFO:teuthology.orchestra.run.smithi146.stdout:(7/151): ceph-mon-18.0.0-7629.g73a4532f.el9.x86 3.8 MB/s | 4.4 MB 00:01 2024-01-16T14:57:21.819 INFO:teuthology.orchestra.run.smithi053.stdout:(68/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-01-16T14:57:21.845 INFO:teuthology.orchestra.run.smithi053.stdout:(69/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 780 kB/s | 217 kB 00:00 2024-01-16T14:57:21.886 INFO:teuthology.orchestra.run.smithi053.stdout:(70/151): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2024-01-16T14:57:21.911 INFO:teuthology.orchestra.run.smithi053.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 2.6 MB/s | 174 kB 00:00 2024-01-16T14:57:21.945 INFO:teuthology.orchestra.run.smithi053.stdout:(72/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-01-16T14:57:21.979 INFO:teuthology.orchestra.run.smithi053.stdout:(73/151): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-01-16T14:57:22.012 INFO:teuthology.orchestra.run.smithi053.stdout:(74/151): python3-pytz-2021.1-5.el9.noarch.rpm 735 kB/s | 47 kB 00:00 2024-01-16T14:57:22.054 INFO:teuthology.orchestra.run.smithi053.stdout:(75/151): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-01-16T14:57:22.846 INFO:teuthology.orchestra.run.smithi053.stdout:(76/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-01-16T14:57:22.871 INFO:teuthology.orchestra.run.smithi053.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 48 kB/s | 39 kB 00:00 2024-01-16T14:57:22.905 INFO:teuthology.orchestra.run.smithi053.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-01-16T14:57:22.930 INFO:teuthology.orchestra.run.smithi053.stdout:(79/151): uuid-1.6.2-55.el9.x86_64.rpm 1.0 MB/s | 56 kB 00:00 2024-01-16T14:57:22.955 INFO:teuthology.orchestra.run.smithi053.stdout:(80/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2024-01-16T14:57:23.015 INFO:teuthology.orchestra.run.smithi053.stdout:(81/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 764 kB/s | 63 kB 00:00 2024-01-16T14:57:23.040 INFO:teuthology.orchestra.run.smithi053.stdout:(82/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 536 kB/s | 44 kB 00:00 2024-01-16T14:57:23.065 INFO:teuthology.orchestra.run.smithi053.stdout:(83/151): fuse-2.9.9-15.el9.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2024-01-16T14:57:23.098 INFO:teuthology.orchestra.run.smithi053.stdout:(84/151): jq-1.6-16.el9.x86_64.rpm 3.2 MB/s | 186 kB 00:00 2024-01-16T14:57:23.109 INFO:teuthology.orchestra.run.smithi146.stdout:(8/151): ceph-common-18.0.0-7629.g73a4532f.el9. 4.1 MB/s | 19 MB 00:04 2024-01-16T14:57:23.157 INFO:teuthology.orchestra.run.smithi053.stdout:(85/151): libgfortran-11.4.1-3.el9.x86_64.rpm 8.6 MB/s | 802 kB 00:00 2024-01-16T14:57:23.191 INFO:teuthology.orchestra.run.smithi053.stdout:(86/151): libquadmath-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 194 kB 00:00 2024-01-16T14:57:23.219 INFO:teuthology.orchestra.run.smithi053.stdout:(87/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.6 MB/s | 217 kB 00:00 2024-01-16T14:57:23.252 INFO:teuthology.orchestra.run.smithi053.stdout:(88/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.1 MB/s | 241 kB 00:00 2024-01-16T14:57:23.259 INFO:teuthology.orchestra.run.smithi146.stdout:(9/151): ceph-radosgw-18.0.0-7629.g73a4532f.el9 5.7 MB/s | 8.6 MB 00:01 2024-01-16T14:57:23.285 INFO:teuthology.orchestra.run.smithi146.stdout:(10/151): ceph-selinux-18.0.0-7629.g73a4532f.el 144 kB/s | 25 kB 00:00 2024-01-16T14:57:23.344 INFO:teuthology.orchestra.run.smithi053.stdout:(89/151): python3-cryptography-36.0.1-4.el9.x86 9.4 MB/s | 1.1 MB 00:00 2024-01-16T14:57:23.360 INFO:teuthology.orchestra.run.smithi146.stdout:(11/151): cephfs-mirror-18.0.0-7629.g73a4532f.e 2.9 MB/s | 217 kB 00:00 2024-01-16T14:57:23.361 INFO:teuthology.orchestra.run.smithi053.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 977 kB/s | 103 kB 00:00 2024-01-16T14:57:23.402 INFO:teuthology.orchestra.run.smithi146.stdout:(12/151): libcephfs-devel-18.0.0-7629.g73a4532f 761 kB/s | 31 kB 00:00 2024-01-16T14:57:23.429 INFO:teuthology.orchestra.run.smithi053.stdout:(91/151): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-01-16T14:57:23.461 INFO:teuthology.orchestra.run.smithi053.stdout:(92/151): python3-requests-2.25.1-7.el9.noarch. 1.1 MB/s | 113 kB 00:00 2024-01-16T14:57:23.487 INFO:teuthology.orchestra.run.smithi053.stdout:(93/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.3 MB/s | 187 kB 00:00 2024-01-16T14:57:23.519 INFO:teuthology.orchestra.run.smithi146.stdout:(13/151): libcephfs2-18.0.0-7629.g73a4532f.el9. 6.1 MB/s | 719 kB 00:00 2024-01-16T14:57:23.561 INFO:teuthology.orchestra.run.smithi146.stdout:(14/151): libcephsqlite-18.0.0-7629.g73a4532f.e 3.9 MB/s | 163 kB 00:00 2024-01-16T14:57:23.611 INFO:teuthology.orchestra.run.smithi146.stdout:(15/151): librados-devel-18.0.0-7629.g73a4532f. 2.5 MB/s | 127 kB 00:00 2024-01-16T14:57:23.712 INFO:teuthology.orchestra.run.smithi146.stdout:(16/151): libradosstriper1-18.0.0-7629.g73a4532 4.7 MB/s | 476 kB 00:00 2024-01-16T14:57:23.987 INFO:teuthology.orchestra.run.smithi053.stdout:(94/151): lua-devel-5.4.4-4.el9.x86_64.rpm 45 kB/s | 22 kB 00:00 2024-01-16T14:57:24.301 INFO:teuthology.orchestra.run.smithi053.stdout:(95/151): python3-devel-3.9.17-2.el9.x86_64.rpm 294 kB/s | 245 kB 00:00 2024-01-16T14:57:24.337 INFO:teuthology.orchestra.run.smithi146.stdout:(17/151): librgw2-18.0.0-7629.g73a4532f.el9.x86 7.7 MB/s | 4.8 MB 00:00 2024-01-16T14:57:24.412 INFO:teuthology.orchestra.run.smithi146.stdout:(18/151): python3-ceph-argparse-18.0.0-7629.g73 610 kB/s | 45 kB 00:00 2024-01-16T14:57:24.454 INFO:teuthology.orchestra.run.smithi146.stdout:(19/151): python3-ceph-common-18.0.0-7629.g73a4 3.1 MB/s | 129 kB 00:00 2024-01-16T14:57:24.504 INFO:teuthology.orchestra.run.smithi146.stdout:(20/151): python3-cephfs-18.0.0-7629.g73a4532f. 3.2 MB/s | 161 kB 00:00 2024-01-16T14:57:24.521 INFO:teuthology.orchestra.run.smithi053.stdout:(96/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 579 kB/s | 309 kB 00:00 2024-01-16T14:57:24.596 INFO:teuthology.orchestra.run.smithi053.stdout:(97/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 351 kB/s | 26 kB 00:00 2024-01-16T14:57:24.596 INFO:teuthology.orchestra.run.smithi146.stdout:(21/151): python3-rados-18.0.0-7629.g73a4532f.e 3.4 MB/s | 321 kB 00:00 2024-01-16T14:57:24.672 INFO:teuthology.orchestra.run.smithi146.stdout:(22/151): python3-rbd-18.0.0-7629.g73a4532f.el9 3.9 MB/s | 298 kB 00:00 2024-01-16T14:57:24.672 INFO:teuthology.orchestra.run.smithi053.stdout:(98/151): liboath-2.6.7-2.el9.x86_64.rpm 654 kB/s | 49 kB 00:00 2024-01-16T14:57:24.722 INFO:teuthology.orchestra.run.smithi146.stdout:(23/151): python3-rgw-18.0.0-7629.g73a4532f.el9 2.0 MB/s | 99 kB 00:00 2024-01-16T14:57:24.755 INFO:teuthology.orchestra.run.smithi053.stdout:(99/151): libunwind-1.6.2-1.el9.x86_64.rpm 816 kB/s | 67 kB 00:00 2024-01-16T14:57:24.764 INFO:teuthology.orchestra.run.smithi146.stdout:(24/151): rbd-fuse-18.0.0-7629.g73a4532f.el9.x8 2.0 MB/s | 85 kB 00:00 2024-01-16T14:57:24.847 INFO:teuthology.orchestra.run.smithi053.stdout:(100/151): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-01-16T14:57:25.164 INFO:teuthology.orchestra.run.smithi053.stdout:(101/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.6 MB/s | 838 kB 00:00 2024-01-16T14:57:25.223 INFO:teuthology.orchestra.run.smithi146.stdout:(25/151): rbd-mirror-18.0.0-7629.g73a4532f.el9. 6.6 MB/s | 3.0 MB 00:00 2024-01-16T14:57:25.281 INFO:teuthology.orchestra.run.smithi146.stdout:(26/151): rbd-nbd-18.0.0-7629.g73a4532f.el9.x86 2.9 MB/s | 171 kB 00:00 2024-01-16T14:57:25.331 INFO:teuthology.orchestra.run.smithi146.stdout:(27/151): ceph-grafana-dashboards-18.0.0-7629.g 571 kB/s | 28 kB 00:00 2024-01-16T14:57:25.373 INFO:teuthology.orchestra.run.smithi053.stdout:(102/151): python3-asyncssh-2.13.2-3.el9.noarch 2.6 MB/s | 548 kB 00:00 2024-01-16T14:57:25.390 INFO:teuthology.orchestra.run.smithi146.stdout:(28/151): ceph-mgr-cephadm-18.0.0-7629.g73a4532 2.3 MB/s | 133 kB 00:00 2024-01-16T14:57:25.423 INFO:teuthology.orchestra.run.smithi053.stdout:(103/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 885 kB/s | 43 kB 00:00 2024-01-16T14:57:25.465 INFO:teuthology.orchestra.run.smithi053.stdout:(104/151): python3-cachetools-4.2.4-1.el9.noarc 791 kB/s | 32 kB 00:00 2024-01-16T14:57:25.507 INFO:teuthology.orchestra.run.smithi053.stdout:(105/151): python3-certifi-2021.10.8-2.el9.noar 368 kB/s | 15 kB 00:00 2024-01-16T14:57:25.607 INFO:teuthology.orchestra.run.smithi053.stdout:(106/151): python3-cheroot-8.6.0-4.el9.noarch.r 1.7 MB/s | 172 kB 00:00 2024-01-16T14:57:25.682 INFO:teuthology.orchestra.run.smithi146.stdout:(29/151): ceph-mgr-dashboard-18.0.0-7629.g73a45 6.0 MB/s | 1.8 MB 00:00 2024-01-16T14:57:25.749 INFO:teuthology.orchestra.run.smithi053.stdout:(107/151): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-01-16T14:57:25.825 INFO:teuthology.orchestra.run.smithi053.stdout:(108/151): python3-google-auth-2.26.0-1.el9.noa 2.7 MB/s | 202 kB 00:00 2024-01-16T14:57:25.875 INFO:teuthology.orchestra.run.smithi053.stdout:(109/151): python3-jaraco-8.2.1-3.el9.noarch.rp 217 kB/s | 11 kB 00:00 2024-01-16T14:57:25.925 INFO:teuthology.orchestra.run.smithi053.stdout:(110/151): python3-jaraco-classes-3.2.1-5.el9.n 360 kB/s | 18 kB 00:00 2024-01-16T14:57:25.975 INFO:teuthology.orchestra.run.smithi053.stdout:(111/151): python3-jaraco-collections-3.0.0-8.e 472 kB/s | 23 kB 00:00 2024-01-16T14:57:26.042 INFO:teuthology.orchestra.run.smithi053.stdout:(112/151): python3-jaraco-functools-3.5.0-2.el9 295 kB/s | 19 kB 00:00 2024-01-16T14:57:26.085 INFO:teuthology.orchestra.run.smithi053.stdout:(113/151): python3-jaraco-text-3.2.0-6.el9.noar 479 kB/s | 20 kB 00:00 2024-01-16T14:57:26.127 INFO:teuthology.orchestra.run.smithi053.stdout:(114/151): python3-jwt+crypto-2.4.0-1.el9.noarc 221 kB/s | 9.0 kB 00:00 2024-01-16T14:57:26.244 INFO:teuthology.orchestra.run.smithi053.stdout:(115/151): libarrow-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 4.4 MB 00:01 2024-01-16T14:57:26.269 INFO:teuthology.orchestra.run.smithi053.stdout:(116/151): python3-jwt-2.4.0-1.el9.noarch.rpm 290 kB/s | 41 kB 00:00 2024-01-16T14:57:26.311 INFO:teuthology.orchestra.run.smithi053.stdout:(117/151): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-01-16T14:57:26.378 INFO:teuthology.orchestra.run.smithi053.stdout:(118/151): python3-more-itertools-8.12.0-2.el9. 1.2 MB/s | 79 kB 00:00 2024-01-16T14:57:26.420 INFO:teuthology.orchestra.run.smithi053.stdout:(119/151): python3-natsort-7.1.1-5.el9.noarch.r 1.4 MB/s | 58 kB 00:00 2024-01-16T14:57:26.604 INFO:teuthology.orchestra.run.smithi053.stdout:(120/151): python3-pecan-1.4.2-3.el9.noarch.rpm 1.5 MB/s | 272 kB 00:00 2024-01-16T14:57:26.662 INFO:teuthology.orchestra.run.smithi053.stdout:(121/151): python3-kubernetes-26.1.0-2.el9.noar 2.4 MB/s | 1.0 MB 00:00 2024-01-16T14:57:26.788 INFO:teuthology.orchestra.run.smithi053.stdout:(122/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.6 MB/s | 90 kB 00:00 2024-01-16T14:57:26.824 INFO:teuthology.orchestra.run.smithi146.stdout:(30/151): ceph-osd-18.0.0-7629.g73a4532f.el9.x8 2.9 MB/s | 18 MB 00:06 2024-01-16T14:57:26.830 INFO:teuthology.orchestra.run.smithi053.stdout:(123/151): python3-repoze-lru-0.7-16.el9.noarch 755 kB/s | 31 kB 00:00 2024-01-16T14:57:26.855 INFO:teuthology.orchestra.run.smithi053.stdout:(124/151): python3-portend-3.1.0-2.el9.noarch.r 66 kB/s | 16 kB 00:00 2024-01-16T14:57:26.922 INFO:teuthology.orchestra.run.smithi053.stdout:(125/151): python3-rsa-4.9-2.el9.noarch.rpm 897 kB/s | 59 kB 00:00 2024-01-16T14:57:26.955 INFO:teuthology.orchestra.run.smithi053.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-01-16T14:57:26.989 INFO:teuthology.orchestra.run.smithi053.stdout:(127/151): python3-tempora-5.0.0-2.el9.noarch.r 542 kB/s | 36 kB 00:00 2024-01-16T14:57:26.999 INFO:teuthology.orchestra.run.smithi146.stdout:(31/151): ceph-mgr-diskprediction-local-18.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-01-16T14:57:27.014 INFO:teuthology.orchestra.run.smithi053.stdout:(128/151): python3-typing-extensions-4.4.0-2.el 882 kB/s | 51 kB 00:00 2024-01-16T14:57:27.032 INFO:teuthology.orchestra.run.smithi146.stdout:(32/151): ceph-mgr-modules-core-18.0.0-7629.g73 1.2 MB/s | 246 kB 00:00 2024-01-16T14:57:27.058 INFO:teuthology.orchestra.run.smithi146.stdout:(33/151): ceph-mgr-rook-18.0.0-7629.g73a4532f.e 873 kB/s | 50 kB 00:00 2024-01-16T14:57:27.065 INFO:teuthology.orchestra.run.smithi053.stdout:(129/151): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-01-16T14:57:27.098 INFO:teuthology.orchestra.run.smithi053.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-01-16T14:57:27.100 INFO:teuthology.orchestra.run.smithi146.stdout:(34/151): ceph-prometheus-alerts-18.0.0-7629.g7 229 kB/s | 15 kB 00:00 2024-01-16T14:57:27.125 INFO:teuthology.orchestra.run.smithi146.stdout:(35/151): ceph-volume-18.0.0-7629.g73a4532f.el9 3.9 MB/s | 262 kB 00:00 2024-01-16T14:57:27.140 INFO:teuthology.orchestra.run.smithi053.stdout:(131/151): python3-zc-lockfile-2.0-10.el9.noarc 490 kB/s | 20 kB 00:00 2024-01-16T14:57:27.175 INFO:teuthology.orchestra.run.smithi146.stdout:(36/151): cephfs-top-18.0.0-7629.g73a4532f.el9. 368 kB/s | 18 kB 00:00 2024-01-16T14:57:27.232 INFO:teuthology.orchestra.run.smithi053.stdout:(132/151): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-01-16T14:57:27.283 INFO:teuthology.orchestra.run.smithi053.stdout:(133/151): python3-werkzeug-2.0.3-3.el9.1.noarc 1.9 MB/s | 427 kB 00:00 2024-01-16T14:57:27.359 INFO:teuthology.orchestra.run.smithi146.stdout:(37/151): cephadm-18.0.0-7629.g73a4532f.el9.noa 2.9 MB/s | 778 kB 00:00 2024-01-16T14:57:27.401 INFO:teuthology.orchestra.run.smithi146.stdout:(38/151): automake-1.16.2-8.el9.noarch.rpm 2.9 MB/s | 662 kB 00:00 2024-01-16T14:57:27.451 INFO:teuthology.orchestra.run.smithi146.stdout:(39/151): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 920 kB 00:00 2024-01-16T14:57:27.485 INFO:teuthology.orchestra.run.smithi146.stdout:(40/151): boost-program-options-1.75.0-8.el9.x8 1.3 MB/s | 106 kB 00:00 2024-01-16T14:57:27.502 INFO:teuthology.orchestra.run.smithi146.stdout:(41/151): elfutils-libelf-devel-0.190-2.el9.x86 417 kB/s | 20 kB 00:00 2024-01-16T14:57:27.536 INFO:teuthology.orchestra.run.smithi146.stdout:(42/151): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 307 kB 00:00 2024-01-16T14:57:27.553 INFO:teuthology.orchestra.run.smithi146.stdout:(43/151): flexiblas-3.0.4-8.el9.x86_64.rpm 672 kB/s | 32 kB 00:00 2024-01-16T14:57:27.678 INFO:teuthology.orchestra.run.smithi146.stdout:(44/151): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-01-16T14:57:27.703 INFO:teuthology.orchestra.run.smithi146.stdout:(45/151): flexiblas-openblas-openmp-3.0.4-8.el9 118 kB/s | 17 kB 00:00 2024-01-16T14:57:27.716 INFO:teuthology.orchestra.run.smithi053.stdout:(134/151): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-01-16T14:57:27.728 INFO:teuthology.orchestra.run.smithi146.stdout:(46/151): libaio-devel-0.3.111-13.el9.x86_64.rp 422 kB/s | 10 kB 00:00 2024-01-16T14:57:27.753 INFO:teuthology.orchestra.run.smithi146.stdout:(47/151): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 159 kB 00:00 2024-01-16T14:57:27.779 INFO:teuthology.orchestra.run.smithi146.stdout:(48/151): librabbitmq-0.11.0-7.el9.x86_64.rpm 917 kB/s | 44 kB 00:00 2024-01-16T14:57:27.821 INFO:teuthology.orchestra.run.smithi146.stdout:(49/151): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-01-16T14:57:27.824 INFO:teuthology.orchestra.run.smithi053.stdout:(135/151): librados2-18.0.0-7629.g73a4532f.el9. 6.1 MB/s | 3.3 MB 00:00 2024-01-16T14:57:27.858 INFO:teuthology.orchestra.run.smithi053.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-01-16T14:57:27.863 INFO:teuthology.orchestra.run.smithi146.stdout:(50/151): libtool-2.4.6-45.el9.x86_64.rpm 7.3 MB/s | 578 kB 00:00 2024-01-16T14:57:27.888 INFO:teuthology.orchestra.run.smithi146.stdout:(51/151): libuuid-devel-2.37.4-15.el9.x86_64.rp 399 kB/s | 24 kB 00:00 2024-01-16T14:57:27.913 INFO:teuthology.orchestra.run.smithi146.stdout:(52/151): libzstd-devel-1.5.1-2.el9.x86_64.rpm 944 kB/s | 46 kB 00:00 2024-01-16T14:57:27.947 INFO:teuthology.orchestra.run.smithi146.stdout:(53/151): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-01-16T14:57:27.980 INFO:teuthology.orchestra.run.smithi146.stdout:(54/151): lua-5.4.4-4.el9.x86_64.rpm 2.9 MB/s | 187 kB 00:00 2024-01-16T14:57:28.006 INFO:teuthology.orchestra.run.smithi146.stdout:(55/151): openblas-0.3.21-2.el9.x86_64.rpm 607 kB/s | 34 kB 00:00 2024-01-16T14:57:28.009 INFO:teuthology.orchestra.run.smithi053.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 15 MB/s | 2.2 MB 00:00 2024-01-16T14:57:28.143 INFO:teuthology.orchestra.run.smithi053.stdout:(138/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 7.1 MB/s | 3.0 MB 00:00 2024-01-16T14:57:28.148 INFO:teuthology.orchestra.run.smithi146.stdout:(56/151): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-01-16T14:57:28.218 INFO:teuthology.orchestra.run.smithi053.stdout:(139/151): NetworkManager-libnm-1.45.9-1.el9.x8 8.5 MB/s | 1.8 MB 00:00 2024-01-16T14:57:28.231 INFO:teuthology.orchestra.run.smithi146.stdout:(57/151): openssl-devel-3.0.7-25.el9.x86_64.rpm 14 MB/s | 3.0 MB 00:00 2024-01-16T14:57:28.243 INFO:teuthology.orchestra.run.smithi053.stdout:(140/151): NetworkManager-team-1.45.9-1.el9.x86 463 kB/s | 42 kB 00:00 2024-01-16T14:57:28.277 INFO:teuthology.orchestra.run.smithi053.stdout:(141/151): NetworkManager-tui-1.45.9-1.el9.x86_ 4.2 MB/s | 246 kB 00:00 2024-01-16T14:57:28.290 INFO:teuthology.orchestra.run.smithi146.stdout:(58/151): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-01-16T14:57:28.302 INFO:teuthology.orchestra.run.smithi053.stdout:(142/151): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 69 kB 00:00 2024-01-16T14:57:28.327 INFO:teuthology.orchestra.run.smithi053.stdout:(143/151): elfutils-debuginfod-client-0.190-2.e 727 kB/s | 35 kB 00:00 2024-01-16T14:57:28.331 INFO:teuthology.orchestra.run.smithi146.stdout:(59/151): postgresql-contrib-13.11-1.el9.x86_64 9.1 MB/s | 811 kB 00:00 2024-01-16T14:57:28.361 INFO:teuthology.orchestra.run.smithi053.stdout:(144/151): elfutils-libelf-0.190-2.el9.x86_64.r 3.3 MB/s | 191 kB 00:00 2024-01-16T14:57:28.365 INFO:teuthology.orchestra.run.smithi146.stdout:(60/151): postgresql-private-libs-13.11-1.el9.x 1.9 MB/s | 134 kB 00:00 2024-01-16T14:57:28.394 INFO:teuthology.orchestra.run.smithi053.stdout:(145/151): elfutils-libs-0.190-2.el9.x86_64.rpm 3.9 MB/s | 254 kB 00:00 2024-01-16T14:57:28.420 INFO:teuthology.orchestra.run.smithi053.stdout:(146/151): libacl-2.3.1-4.el9.x86_64.rpm 396 kB/s | 22 kB 00:00 2024-01-16T14:57:28.512 INFO:teuthology.orchestra.run.smithi053.stdout:(147/151): openssl-3.0.7-25.el9.x86_64.rpm 9.9 MB/s | 1.2 MB 00:00 2024-01-16T14:57:28.549 INFO:teuthology.orchestra.run.smithi146.stdout:(61/151): postgresql-server-13.11-1.el9.x86_64. 26 MB/s | 5.7 MB 00:00 2024-01-16T14:57:28.617 INFO:teuthology.orchestra.run.smithi053.stdout:(148/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 11 MB/s | 2.1 MB 00:00 2024-01-16T14:57:28.645 INFO:teuthology.orchestra.run.smithi053.stdout:(149/151): xfsdump-3.1.12-4.el9.x86_64.rpm 2.6 MB/s | 321 kB 00:00 2024-01-16T14:57:28.682 INFO:teuthology.orchestra.run.smithi146.stdout:(62/151): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-01-16T14:57:28.704 INFO:teuthology.orchestra.run.smithi053.stdout:(150/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 13 MB/s | 1.1 MB 00:00 2024-01-16T14:57:28.708 INFO:teuthology.orchestra.run.smithi146.stdout:(63/151): python3-jinja2-2.11.3-4.el9.noarch.rp 1.5 MB/s | 228 kB 00:00 2024-01-16T14:57:28.756 INFO:teuthology.orchestra.run.smithi146.stdout:(64/151): python3-jsonpatch-1.21-16.el9.noarch. 341 kB/s | 24 kB 00:00 2024-01-16T14:57:28.783 INFO:teuthology.orchestra.run.smithi146.stdout:(65/151): python3-jsonpointer-2.0-4.el9.noarch. 243 kB/s | 18 kB 00:00 2024-01-16T14:57:28.809 INFO:teuthology.orchestra.run.smithi146.stdout:(66/151): python3-mako-1.1.4-6.el9.noarch.rpm 2.7 MB/s | 144 kB 00:00 2024-01-16T14:57:28.834 INFO:teuthology.orchestra.run.smithi146.stdout:(67/151): python3-markupsafe-1.1.1-12.el9.x86_6 655 kB/s | 32 kB 00:00 2024-01-16T14:57:29.301 INFO:teuthology.orchestra.run.smithi146.stdout:(68/151): ceph-test-18.0.0-7629.g73a4532f.el9.x 6.8 MB/s | 41 MB 00:06 2024-01-16T14:57:29.427 INFO:teuthology.orchestra.run.smithi146.stdout:(69/151): python3-numpy-1.20.1-5.el9.x86_64.rpm 7.8 MB/s | 4.8 MB 00:00 2024-01-16T14:57:29.460 INFO:teuthology.orchestra.run.smithi146.stdout:(70/151): python3-numpy-f2py-1.20.1-5.el9.x86_6 348 kB/s | 217 kB 00:00 2024-01-16T14:57:29.477 INFO:teuthology.orchestra.run.smithi146.stdout:(71/151): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-01-16T14:57:29.527 INFO:teuthology.orchestra.run.smithi146.stdout:(72/151): python3-prettytable-0.7.2-27.el9.noar 634 kB/s | 41 kB 00:00 2024-01-16T14:57:29.561 INFO:teuthology.orchestra.run.smithi146.stdout:(73/151): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 132 kB 00:00 2024-01-16T14:57:29.586 INFO:teuthology.orchestra.run.smithi146.stdout:(74/151): python3-pyasn1-modules-0.4.8-6.el9.no 1.9 MB/s | 211 kB 00:00 2024-01-16T14:57:29.611 INFO:teuthology.orchestra.run.smithi146.stdout:(75/151): python3-pytz-2021.1-5.el9.noarch.rpm 591 kB/s | 47 kB 00:00 2024-01-16T14:57:29.645 INFO:teuthology.orchestra.run.smithi146.stdout:(76/151): python3-requests-oauthlib-1.3.0-12.el 749 kB/s | 43 kB 00:00 2024-01-16T14:57:30.078 INFO:teuthology.orchestra.run.smithi146.stdout:(77/151): python3-toml-0.10.2-6.el9.noarch.rpm 84 kB/s | 39 kB 00:00 2024-01-16T14:57:30.112 INFO:teuthology.orchestra.run.smithi146.stdout:(78/151): socat-1.7.4.1-5.el9.x86_64.rpm 656 kB/s | 300 kB 00:00 2024-01-16T14:57:30.312 INFO:teuthology.orchestra.run.smithi146.stdout:(79/151): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-01-16T14:57:30.337 INFO:teuthology.orchestra.run.smithi146.stdout:(80/151): uuid-1.6.2-55.el9.x86_64.rpm 220 kB/s | 56 kB 00:00 2024-01-16T14:57:30.354 INFO:teuthology.orchestra.run.smithi146.stdout:(81/151): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 371 kB/s | 87 kB 00:00 2024-01-16T14:57:30.379 INFO:teuthology.orchestra.run.smithi146.stdout:(82/151): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-01-16T14:57:30.502 INFO:teuthology.orchestra.run.smithi146.stdout:(83/151): xmlstarlet-1.6.1-20.el9.x86_64.rpm 384 kB/s | 63 kB 00:00 2024-01-16T14:57:30.555 INFO:teuthology.orchestra.run.smithi146.stdout:(84/151): fuse-2.9.9-15.el9.x86_64.rpm 394 kB/s | 79 kB 00:00 2024-01-16T14:57:30.580 INFO:teuthology.orchestra.run.smithi146.stdout:(85/151): jq-1.6-16.el9.x86_64.rpm 937 kB/s | 186 kB 00:00 2024-01-16T14:57:30.605 INFO:teuthology.orchestra.run.smithi146.stdout:(86/151): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 194 kB 00:00 2024-01-16T14:57:30.647 INFO:teuthology.orchestra.run.smithi146.stdout:(87/151): libgfortran-11.4.1-3.el9.x86_64.rpm 8.7 MB/s | 802 kB 00:00 2024-01-16T14:57:30.672 INFO:teuthology.orchestra.run.smithi146.stdout:(88/151): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.4 MB/s | 217 kB 00:00 2024-01-16T14:57:30.698 INFO:teuthology.orchestra.run.smithi146.stdout:(89/151): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-01-16T14:57:30.723 INFO:teuthology.orchestra.run.smithi146.stdout:(90/151): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 103 kB 00:00 2024-01-16T14:57:30.782 INFO:teuthology.orchestra.run.smithi146.stdout:(91/151): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-01-16T14:57:30.798 INFO:teuthology.orchestra.run.smithi146.stdout:(92/151): python3-pycparser-2.20-6.el9.noarch.r 1.2 MB/s | 124 kB 00:00 2024-01-16T14:57:30.824 INFO:teuthology.orchestra.run.smithi146.stdout:(93/151): python3-requests-2.25.1-7.el9.noarch. 1.2 MB/s | 113 kB 00:00 2024-01-16T14:57:30.857 INFO:teuthology.orchestra.run.smithi146.stdout:(94/151): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-01-16T14:57:31.058 INFO:teuthology.orchestra.run.smithi146.stdout:(95/151): lua-devel-5.4.4-4.el9.x86_64.rpm 96 kB/s | 22 kB 00:00 2024-01-16T14:57:31.091 INFO:teuthology.orchestra.run.smithi146.stdout:(96/151): python3-devel-3.9.17-2.el9.x86_64.rpm 846 kB/s | 245 kB 00:00 2024-01-16T14:57:31.175 INFO:teuthology.orchestra.run.smithi146.stdout:(97/151): libarrow-doc-9.0.0-10.el9.noarch.rpm 315 kB/s | 26 kB 00:00 2024-01-16T14:57:31.208 INFO:teuthology.orchestra.run.smithi146.stdout:(98/151): gperftools-libs-2.9.1-2.el9.x86_64.rp 882 kB/s | 309 kB 00:00 2024-01-16T14:57:31.259 INFO:teuthology.orchestra.run.smithi146.stdout:(99/151): liboath-2.6.7-2.el9.x86_64.rpm 595 kB/s | 49 kB 00:00 2024-01-16T14:57:31.626 INFO:teuthology.orchestra.run.smithi146.stdout:(100/151): libarrow-9.0.0-10.el9.x86_64.rpm 7.8 MB/s | 4.4 MB 00:00 2024-01-16T14:57:31.651 INFO:teuthology.orchestra.run.smithi146.stdout:(101/151): libunwind-1.6.2-1.el9.x86_64.rpm 153 kB/s | 67 kB 00:00 2024-01-16T14:57:31.676 INFO:teuthology.orchestra.run.smithi146.stdout:(102/151): luarocks-3.9.2-1.el9.noarch.rpm 365 kB/s | 151 kB 00:00 2024-01-16T14:57:31.752 INFO:teuthology.orchestra.run.smithi146.stdout:(103/151): parquet-libs-9.0.0-10.el9.x86_64.rpm 11 MB/s | 838 kB 00:00 2024-01-16T14:57:31.793 INFO:teuthology.orchestra.run.smithi146.stdout:(104/151): python3-asyncssh-2.13.2-3.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-01-16T14:57:31.819 INFO:teuthology.orchestra.run.smithi146.stdout:(105/151): python3-bcrypt-3.2.2-1.el9.x86_64.rp 309 kB/s | 43 kB 00:00 2024-01-16T14:57:31.844 INFO:teuthology.orchestra.run.smithi146.stdout:(106/151): python3-cachetools-4.2.4-1.el9.noarc 353 kB/s | 32 kB 00:00 2024-01-16T14:57:31.877 INFO:teuthology.orchestra.run.smithi146.stdout:(107/151): python3-certifi-2021.10.8-2.el9.noar 181 kB/s | 15 kB 00:00 2024-01-16T14:57:31.911 INFO:teuthology.orchestra.run.smithi146.stdout:(108/151): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-01-16T14:57:31.945 INFO:teuthology.orchestra.run.smithi146.stdout:(109/151): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-01-16T14:57:31.978 INFO:teuthology.orchestra.run.smithi146.stdout:(110/151): python3-google-auth-2.26.0-1.el9.noa 2.0 MB/s | 202 kB 00:00 2024-01-16T14:57:32.021 INFO:teuthology.orchestra.run.smithi146.stdout:(111/151): python3-jaraco-8.2.1-3.el9.noarch.rp 101 kB/s | 11 kB 00:00 2024-01-16T14:57:32.053 INFO:teuthology.orchestra.run.smithi146.stdout:(112/151): python3-jaraco-classes-3.2.1-5.el9.n 166 kB/s | 18 kB 00:00 2024-01-16T14:57:32.079 INFO:teuthology.orchestra.run.smithi146.stdout:(113/151): python3-jaraco-collections-3.0.0-8.e 408 kB/s | 23 kB 00:00 2024-01-16T14:57:32.104 INFO:teuthology.orchestra.run.smithi146.stdout:(114/151): python3-jaraco-functools-3.5.0-2.el9 238 kB/s | 19 kB 00:00 2024-01-16T14:57:32.129 INFO:teuthology.orchestra.run.smithi146.stdout:(115/151): python3-jaraco-text-3.2.0-6.el9.noar 265 kB/s | 20 kB 00:00 2024-01-16T14:57:32.155 INFO:teuthology.orchestra.run.smithi146.stdout:(116/151): python3-jwt+crypto-2.4.0-1.el9.noarc 185 kB/s | 9.0 kB 00:00 2024-01-16T14:57:32.180 INFO:teuthology.orchestra.run.smithi146.stdout:(117/151): python3-jwt-2.4.0-1.el9.noarch.rpm 553 kB/s | 41 kB 00:00 2024-01-16T14:57:32.388 INFO:teuthology.orchestra.run.smithi146.stdout:(118/151): python3-kubernetes-26.1.0-2.el9.noar 4.0 MB/s | 1.0 MB 00:00 2024-01-16T14:57:32.414 INFO:teuthology.orchestra.run.smithi146.stdout:(119/151): python3-logutils-0.3.5-21.el9.noarch 180 kB/s | 46 kB 00:00 2024-01-16T14:57:32.439 INFO:teuthology.orchestra.run.smithi146.stdout:(120/151): python3-more-itertools-8.12.0-2.el9. 305 kB/s | 79 kB 00:00 2024-01-16T14:57:32.472 INFO:teuthology.orchestra.run.smithi146.stdout:(121/151): python3-natsort-7.1.1-5.el9.noarch.r 710 kB/s | 58 kB 00:00 2024-01-16T14:57:32.506 INFO:teuthology.orchestra.run.smithi146.stdout:(122/151): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-01-16T14:57:32.531 INFO:teuthology.orchestra.run.smithi146.stdout:(123/151): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-01-16T14:57:32.556 INFO:teuthology.orchestra.run.smithi146.stdout:(124/151): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-01-16T14:57:32.582 INFO:teuthology.orchestra.run.smithi146.stdout:(125/151): python3-repoze-lru-0.7-16.el9.noarch 412 kB/s | 31 kB 00:00 2024-01-16T14:57:32.607 INFO:teuthology.orchestra.run.smithi146.stdout:(126/151): python3-routes-2.5.1-5.el9.noarch.rp 2.5 MB/s | 188 kB 00:00 2024-01-16T14:57:32.674 INFO:teuthology.orchestra.run.smithi146.stdout:(127/151): python3-rsa-4.9-2.el9.noarch.rpm 511 kB/s | 59 kB 00:00 2024-01-16T14:57:32.699 INFO:teuthology.orchestra.run.smithi146.stdout:(128/151): python3-tempora-5.0.0-2.el9.noarch.r 310 kB/s | 36 kB 00:00 2024-01-16T14:57:32.724 INFO:teuthology.orchestra.run.smithi146.stdout:(129/151): python3-typing-extensions-4.4.0-2.el 438 kB/s | 51 kB 00:00 2024-01-16T14:57:32.758 INFO:teuthology.orchestra.run.smithi146.stdout:(130/151): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-01-16T14:57:32.800 INFO:teuthology.orchestra.run.smithi146.stdout:(131/151): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-01-16T14:57:32.833 INFO:teuthology.orchestra.run.smithi146.stdout:(132/151): python3-werkzeug-2.0.3-3.el9.1.noarc 4.0 MB/s | 427 kB 00:00 2024-01-16T14:57:32.858 INFO:teuthology.orchestra.run.smithi146.stdout:(133/151): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2024-01-16T14:57:32.892 INFO:teuthology.orchestra.run.smithi146.stdout:(134/151): re2-20211101-3.el9.x86_64.rpm 2.1 MB/s | 192 kB 00:00 2024-01-16T14:57:32.959 INFO:teuthology.orchestra.run.smithi146.stdout:(135/151): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-01-16T14:57:33.010 INFO:teuthology.orchestra.run.smithi146.stdout:(136/151): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.3 MB/s | 68 kB 00:00 2024-01-16T14:57:33.194 INFO:teuthology.orchestra.run.smithi146.stdout:(137/151): NetworkManager-1.45.9-1.el9.x86_64.r 12 MB/s | 2.2 MB 00:00 2024-01-16T14:57:33.310 INFO:teuthology.orchestra.run.smithi146.stdout:(138/151): NetworkManager-libnm-1.45.9-1.el9.x8 15 MB/s | 1.8 MB 00:00 2024-01-16T14:57:33.352 INFO:teuthology.orchestra.run.smithi146.stdout:(139/151): NetworkManager-team-1.45.9-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-01-16T14:57:33.402 INFO:teuthology.orchestra.run.smithi146.stdout:(140/151): NetworkManager-tui-1.45.9-1.el9.x86_ 4.9 MB/s | 246 kB 00:00 2024-01-16T14:57:33.444 INFO:teuthology.orchestra.run.smithi146.stdout:(141/151): acl-2.3.1-4.el9.x86_64.rpm 1.7 MB/s | 69 kB 00:00 2024-01-16T14:57:33.478 INFO:teuthology.orchestra.run.smithi146.stdout:(142/151): elfutils-debuginfod-client-0.190-2.e 1.1 MB/s | 35 kB 00:00 2024-01-16T14:57:33.520 INFO:teuthology.orchestra.run.smithi146.stdout:(143/151): elfutils-libelf-0.190-2.el9.x86_64.r 4.6 MB/s | 191 kB 00:00 2024-01-16T14:57:33.595 INFO:teuthology.orchestra.run.smithi146.stdout:(144/151): librbd1-18.0.0-7629.g73a4532f.el9.x8 4.4 MB/s | 3.0 MB 00:00 2024-01-16T14:57:33.629 INFO:teuthology.orchestra.run.smithi146.stdout:(145/151): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 254 kB 00:00 2024-01-16T14:57:33.670 INFO:teuthology.orchestra.run.smithi146.stdout:(146/151): libacl-2.3.1-4.el9.x86_64.rpm 303 kB/s | 22 kB 00:00 2024-01-16T14:57:33.754 INFO:teuthology.orchestra.run.smithi146.stdout:(147/151): openssl-3.0.7-25.el9.x86_64.rpm 9.3 MB/s | 1.2 MB 00:00 2024-01-16T14:57:33.888 INFO:teuthology.orchestra.run.smithi146.stdout:(148/151): openssl-libs-3.0.7-25.el9.x86_64.rpm 10 MB/s | 2.1 MB 00:00 2024-01-16T14:57:33.922 INFO:teuthology.orchestra.run.smithi146.stdout:(149/151): xfsdump-3.1.12-4.el9.x86_64.rpm 1.9 MB/s | 321 kB 00:00 2024-01-16T14:57:34.005 INFO:teuthology.orchestra.run.smithi146.stdout:(150/151): xfsprogs-6.3.0-1.el9.x86_64.rpm 9.1 MB/s | 1.1 MB 00:00 2024-01-16T14:57:35.847 INFO:teuthology.orchestra.run.smithi146.stdout:(151/151): librados2-18.0.0-7629.g73a4532f.el9. 1.1 MB/s | 3.3 MB 00:02 2024-01-16T14:57:35.857 INFO:teuthology.orchestra.run.smithi146.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:57:35.857 INFO:teuthology.orchestra.run.smithi146.stdout:Total 11 MB/s | 204 MB 00:18 2024-01-16T14:57:35.947 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction check 2024-01-16T14:57:36.291 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction check succeeded. 2024-01-16T14:57:36.291 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction test 2024-01-16T14:57:38.392 INFO:teuthology.orchestra.run.smithi146.stdout:Transaction test succeeded. 2024-01-16T14:57:38.393 INFO:teuthology.orchestra.run.smithi146.stdout:Running transaction 2024-01-16T14:57:39.054 INFO:teuthology.orchestra.run.smithi053.stdout:(151/151): ceph-test-18.0.0-7629.g73a4532f.el9. 1.8 MB/s | 41 MB 00:22 2024-01-16T14:57:39.069 INFO:teuthology.orchestra.run.smithi053.stdout:-------------------------------------------------------------------------------- 2024-01-16T14:57:39.069 INFO:teuthology.orchestra.run.smithi053.stdout:Total 7.9 MB/s | 204 MB 00:25 2024-01-16T14:57:39.165 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction check 2024-01-16T14:57:39.526 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction check succeeded. 2024-01-16T14:57:39.526 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction test 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi053.stdout:Transaction test succeeded. 2024-01-16T14:57:41.789 INFO:teuthology.orchestra.run.smithi053.stdout:Running transaction 2024-01-16T14:57:44.575 INFO:teuthology.orchestra.run.smithi146.stdout: Preparing : 1/1 2024-01-16T14:57:44.765 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:57:44.803 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:57:44.877 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:57:44.934 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:57:44.965 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:57:45.061 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:57:45.488 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:57:45.746 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:45.916 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:45.947 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:46.071 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:46.113 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:57:46.270 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:57:46.383 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:57:46.684 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:57:46.745 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:57:46.771 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:46.910 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:46.938 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:47.022 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:47.346 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:57:47.543 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:47.613 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:47.805 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:47.862 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:48.289 INFO:teuthology.orchestra.run.smithi053.stdout: Preparing : 1/1 2024-01-16T14:57:48.302 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:48.368 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:57:48.482 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/167 2024-01-16T14:57:48.490 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:57:48.521 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : thrift-0.15.0-2.el9.x86_64 2/167 2024-01-16T14:57:48.553 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:57:48.597 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:57:48.609 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/167 2024-01-16T14:57:48.667 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/167 2024-01-16T14:57:48.670 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:57:48.723 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/167 2024-01-16T14:57:48.727 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:57:48.832 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/167 2024-01-16T14:57:48.842 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:57:48.891 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:57:48.937 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:57:48.984 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:57:49.103 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:57:49.137 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:57:49.173 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:57:49.202 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:57:49.249 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/167 2024-01-16T14:57:49.294 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:49.313 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:57:49.415 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:57:49.461 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:57:49.495 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:49.592 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:57:49.629 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:57:49.658 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:57:49.678 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:57:49.847 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 11/167 2024-01-16T14:57:49.943 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 12/167 2024-01-16T14:57:50.251 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 13/167 2024-01-16T14:57:50.302 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-13.11-1.el9.x86_64 14/167 2024-01-16T14:57:50.327 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:50.345 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:57:50.445 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:57:50.487 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:57:50.511 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:50.519 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:57:50.577 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:57:50.604 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 16/167 2024-01-16T14:57:50.622 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:57:50.697 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:57:50.893 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 17/167 2024-01-16T14:57:50.916 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:57:50.964 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:57:51.084 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:51.113 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:57:51.151 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:57:51.155 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 18/167 2024-01-16T14:57:51.213 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:57:51.300 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:57:51.342 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:57:51.348 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:51.390 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:57:51.428 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:51.457 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:57:51.517 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:57:51.576 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:57:51.785 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:57:51.833 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:57:51.870 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.45.9-1.el9.x86_64 19/167 2024-01-16T14:57:51.879 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:57:51.909 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 20/167 2024-01-16T14:57:52.006 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:57:52.028 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/167 2024-01-16T14:57:52.049 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:57:52.095 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 22/167 2024-01-16T14:57:52.096 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:57:52.130 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 23/167 2024-01-16T14:57:52.131 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:57:52.188 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 24/167 2024-01-16T14:57:52.251 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 25/167 2024-01-16T14:57:52.284 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:57:52.345 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:57:52.399 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 26/167 2024-01-16T14:57:52.412 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:57:52.441 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 27/167 2024-01-16T14:57:52.485 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:57:52.487 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/167 2024-01-16T14:57:52.526 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/167 2024-01-16T14:57:52.649 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/167 2024-01-16T14:57:52.696 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 31/167 2024-01-16T14:57:52.752 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 32/167 2024-01-16T14:57:52.758 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:57:52.787 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 33/167 2024-01-16T14:57:52.823 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:57:52.912 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/167 2024-01-16T14:57:52.992 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:57:53.012 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/167 2024-01-16T14:57:53.042 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:57:53.067 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:57:53.106 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:57:53.150 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:57:53.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:57:53.219 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 36/167 2024-01-16T14:57:53.238 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:57:53.280 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:57:53.284 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 37/167 2024-01-16T14:57:53.323 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:57:53.388 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:57:53.455 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:57:53.506 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:57:53.648 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:57:53.692 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:57:53.735 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:57:53.825 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:57:53.872 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:57:54.003 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rsa-4.9-2.el9.noarch 38/167 2024-01-16T14:57:54.019 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:57:54.061 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:57:54.103 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 39/167 2024-01-16T14:57:54.150 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 40/167 2024-01-16T14:57:54.198 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 41/167 2024-01-16T14:57:54.202 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:57:54.246 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 42/167 2024-01-16T14:57:54.258 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:57:54.309 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 43/167 2024-01-16T14:57:54.511 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 44/167 2024-01-16T14:57:54.558 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 45/167 2024-01-16T14:57:54.698 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : re2-1:20211101-3.el9.x86_64 46/167 2024-01-16T14:57:54.736 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 47/167 2024-01-16T14:57:54.799 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/167 2024-01-16T14:57:54.887 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 49/167 2024-01-16T14:57:54.929 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 50/167 2024-01-16T14:57:54.968 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 51/167 2024-01-16T14:57:55.035 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 52/167 2024-01-16T14:57:55.093 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 53/167 2024-01-16T14:57:55.154 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/167 2024-01-16T14:57:55.235 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:57:55.282 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:57:55.315 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:57:55.349 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:57:55.350 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/167 2024-01-16T14:57:55.403 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/167 2024-01-16T14:57:55.449 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 57/167 2024-01-16T14:57:55.571 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 58/167 2024-01-16T14:57:55.596 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:57:55.610 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-google-auth-1:2.26.0-1.el9.noarch 59/167 2024-01-16T14:57:55.628 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:57:55.658 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 60/167 2024-01-16T14:57:55.710 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 61/167 2024-01-16T14:57:55.719 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:57:55.871 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 62/167 2024-01-16T14:57:55.933 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 63/167 2024-01-16T14:57:56.006 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-ply-3.11-14.el9.noarch 64/167 2024-01-16T14:57:56.097 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 65/167 2024-01-16T14:57:56.384 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 66/167 2024-01-16T14:57:56.445 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 67/167 2024-01-16T14:57:56.522 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:57:56.619 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 68/167 2024-01-16T14:57:56.664 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:57:56.673 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 69/167 2024-01-16T14:57:56.706 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 70/167 2024-01-16T14:57:56.747 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 71/167 2024-01-16T14:57:56.791 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 72/167 2024-01-16T14:57:56.840 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 73/167 2024-01-16T14:57:56.898 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : jq-1.6-16.el9.x86_64 74/167 2024-01-16T14:57:56.946 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : fuse-2.9.9-15.el9.x86_64 75/167 2024-01-16T14:57:57.023 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 76/167 2024-01-16T14:57:57.079 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 77/167 2024-01-16T14:57:57.146 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : uuid-1.6.2-55.el9.x86_64 78/167 2024-01-16T14:57:57.198 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 79/167 2024-01-16T14:57:57.342 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 80/167 2024-01-16T14:57:57.382 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 81/167 2024-01-16T14:57:57.437 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 82/167 2024-01-16T14:57:57.502 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 83/167 2024-01-16T14:57:57.548 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 84/167 2024-01-16T14:57:57.701 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 85/167 2024-01-16T14:57:57.737 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 86/167 2024-01-16T14:57:57.886 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 87/167 2024-01-16T14:57:57.951 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 88/167 2024-01-16T14:57:58.189 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:57:58.282 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:57:58.319 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:57:58.354 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:57:58.700 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:57:58.729 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:57:58.848 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:57:58.876 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:57:58.929 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:57:58.973 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 89/167 2024-01-16T14:57:59.026 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 90/167 2024-01-16T14:57:59.059 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 91/167 2024-01-16T14:57:59.092 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 92/167 2024-01-16T14:57:59.391 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/167 2024-01-16T14:57:59.402 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:57:59.422 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/167 2024-01-16T14:57:59.431 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:57:59.517 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/167 2024-01-16T14:57:59.522 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:57:59.550 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:58:00.372 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/167 2024-01-16T14:58:00.532 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/167 2024-01-16T14:58:01.995 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:58:02.046 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:58:02.133 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/167 2024-01-16T14:58:02.237 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/167 2024-01-16T14:58:02.273 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/167 2024-01-16T14:58:02.316 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 101/167 2024-01-16T14:58:02.675 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:58:02.680 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/167 2024-01-16T14:58:02.717 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:58:02.717 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:58:02.839 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 103/167 2024-01-16T14:58:02.873 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 104/167 2024-01-16T14:58:02.892 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:58:02.892 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:58:02.892 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:58:02.892 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:02.928 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/167 2024-01-16T14:58:02.942 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:58:03.429 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 106/167 2024-01-16T14:58:03.453 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:58:03.553 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 107/167 2024-01-16T14:58:03.571 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 108/167 2024-01-16T14:58:06.117 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:58:06.160 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:58:06.830 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 109/167 2024-01-16T14:58:06.872 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:58:07.067 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 110/167 2024-01-16T14:58:07.067 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-01-16T14:58:07.067 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-01-16T14:58:07.067 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:07.106 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:58:15.177 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:58:15.177 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /sys 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /proc 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /mnt 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /var/tmp 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /home 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /root 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout:skipping the directory /tmp 2024-01-16T14:58:15.178 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:15.381 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:58:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:58:15.428 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:15.428 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:58:15.428 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:17.091 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:58:17.170 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:58:17.171 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:17.171 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:58:17.171 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:58:17.171 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:58:17.171 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:17.383 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:58:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:58:17.453 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:17.453 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:58:17.453 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:58:17.453 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:58:17.453 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:17.950 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:58:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:58:18.022 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:18.022 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:58:18.022 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:58:18.022 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:58:18.022 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:18.152 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:58:18.782 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:58:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:58:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:58:18.952 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:58:19.004 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:58:19.318 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:58:19.357 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 111/167 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /sys 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /proc 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /mnt 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /var/tmp 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /home 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /root 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout:skipping the directory /tmp 2024-01-16T14:58:20.380 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:20.475 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:58:20.531 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:58:20.614 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:58:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 112/167 2024-01-16T14:58:20.666 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:20.666 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-01-16T14:58:20.666 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:20.676 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:58:20.838 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:58:20.875 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:58:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:58:20.949 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:20.949 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:58:20.949 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:58:20.949 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:58:20.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:21.001 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:58:21.037 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:58:21.093 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:58:21.943 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:58:21.977 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:58:22.047 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:58:22.047 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:22.047 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:58:22.047 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:58:22.047 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:58:22.048 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:22.103 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:58:22.174 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:58:22.175 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:22.175 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:58:22.175 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:22.275 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:58:22.351 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:58:22.352 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:22.352 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:58:22.352 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:58:22.352 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:58:22.352 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:22.444 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:58:22.527 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 113/167 2024-01-16T14:58:22.528 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:22.528 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-01-16T14:58:22.528 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:58:22.528 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-01-16T14:58:22.528 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:22.695 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:58:22.752 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:58:22.770 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:58:22.771 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:22.839 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 114/167 2024-01-16T14:58:22.839 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:22.839 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-01-16T14:58:22.840 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:58:22.840 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-01-16T14:58:22.840 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:23.436 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:58:23.526 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 115/167 2024-01-16T14:58:23.527 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:23.527 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-01-16T14:58:23.527 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:58:23.527 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-01-16T14:58:23.527 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:23.671 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : automake-1.16.2-8.el9.noarch 116/167 2024-01-16T14:58:24.434 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:58:24.464 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 117/167 2024-01-16T14:58:24.502 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 118/167 2024-01-16T14:58:24.587 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 119/167 2024-01-16T14:58:25.036 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:58:25.072 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 120/167 2024-01-16T14:58:25.110 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:58:26.286 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 121/167 2024-01-16T14:58:26.318 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:58:26.468 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 122/167 2024-01-16T14:58:26.643 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 123/167 2024-01-16T14:58:26.679 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:58:26.754 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 124/167 2024-01-16T14:58:26.754 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:26.754 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-01-16T14:58:26.754 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:58:26.754 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-01-16T14:58:26.755 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:26.804 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:58:26.841 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:58:26.843 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 125/167 2024-01-16T14:58:26.900 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:58:26.929 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 126/167 2024-01-16T14:58:26.948 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:58:26.982 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:58:27.241 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:58:27.357 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:58:27.387 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:27.420 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:58:27.459 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:58:27.495 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:58:27.609 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:27.664 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:58:27.722 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:58:27.840 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libtool-2.4.6-45.el9.x86_64 127/167 2024-01-16T14:58:27.898 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:58:27.975 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 128/167 2024-01-16T14:58:27.975 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:27.975 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-01-16T14:58:27.975 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:58:27.975 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-01-16T14:58:27.975 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:28.032 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-7629.g73a45 129/167 2024-01-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-01-16T14:58:28.107 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:28.183 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:58:28.274 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 130/167 2024-01-16T14:58:28.274 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:28.274 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-01-16T14:58:28.274 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:58:28.274 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-01-16T14:58:28.274 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:28.600 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:28.636 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:28.675 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:58:28.722 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:28.749 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 131/167 2024-01-16T14:58:28.749 INFO:teuthology.orchestra.run.smithi053.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-01-16T14:58:28.749 INFO:teuthology.orchestra.run.smithi053.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-01-16T14:58:28.750 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:58:28.750 INFO:teuthology.orchestra.run.smithi053.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-01-16T14:58:28.750 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:28.808 INFO:teuthology.orchestra.run.smithi146.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:58:30.161 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:58:30.241 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:58:30.289 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:58:30.427 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:58:30.469 INFO:teuthology.orchestra.run.smithi146.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:58:30.502 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:58:30.536 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:30.577 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:58:30.578 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:30.711 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:30.762 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:58:30.830 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:58:31.300 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:58:31.332 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:31.332 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:31.469 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:31.500 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:58:31.539 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:31.540 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:31.883 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:32.016 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:32.052 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:58:32.092 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:58:32.137 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:58:32.291 INFO:teuthology.orchestra.run.smithi146.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:32.549 INFO:teuthology.orchestra.run.smithi146.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:32.550 INFO:teuthology.orchestra.run.smithi146.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:58:32.550 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:33.122 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 132/167 2024-01-16T14:58:33.167 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 133/167 2024-01-16T14:58:33.224 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 134/167 2024-01-16T14:58:33.262 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 135/167 2024-01-16T14:58:33.552 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 136/167 2024-01-16T14:58:33.683 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 137/167 2024-01-16T14:58:33.716 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:33.750 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 139/167 2024-01-16T14:58:33.789 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 140/167 2024-01-16T14:58:33.843 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-team-1:1.45.9-1.el9.x86_64 141/167 2024-01-16T14:58:33.954 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : NetworkManager-tui-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:34.046 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 143/167 2024-01-16T14:58:34.102 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 144/167 2024-01-16T14:58:35.073 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:35.116 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:35.195 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 145/167 2024-01-16T14:58:35.294 INFO:teuthology.orchestra.run.smithi053.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 146/167 2024-01-16T14:58:36.700 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 147/167 2024-01-16T14:58:36.780 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 148/167 2024-01-16T14:58:36.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:58:36.831 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:58:36.832 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:58:36.833 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:58:36.834 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:58:36.835 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:58:36.836 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:58:36.837 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:58:36.838 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:58:36.839 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:58:36.840 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:58:36.841 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:36.842 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:58:36.843 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:58:36.844 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:58:36.845 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:58:36.845 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:58:36.845 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 149/167 2024-01-16T14:58:36.969 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : flex-2.6.4-9.el9.x86_64 150/167 2024-01-16T14:58:37.007 INFO:teuthology.orchestra.run.smithi053.stdout: Installing : bison-3.7.4-5.el9.x86_64 151/167 2024-01-16T14:58:37.040 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 152/167 2024-01-16T14:58:37.075 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:37.116 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 154/167 2024-01-16T14:58:37.116 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:37.266 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 155/167 2024-01-16T14:58:37.318 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 156/167 2024-01-16T14:58:37.367 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 157/167 2024-01-16T14:58:37.421 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 158/167 2024-01-16T14:58:37.453 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:37.454 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:37.603 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/167 2024-01-16T14:58:37.638 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 161/167 2024-01-16T14:58:37.686 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:37.687 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:38.030 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 162/167 2024-01-16T14:58:38.030 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:38.172 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 163/167 2024-01-16T14:58:38.207 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 164/167 2024-01-16T14:58:38.278 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 165/167 2024-01-16T14:58:38.326 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 166/167 2024-01-16T14:58:38.449 INFO:teuthology.orchestra.run.smithi053.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:38.703 INFO:teuthology.orchestra.run.smithi053.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 167/167 2024-01-16T14:58:38.703 INFO:teuthology.orchestra.run.smithi053.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-01-16T14:58:38.703 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:38.985 INFO:teuthology.orchestra.run.smithi146.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:58:38.985 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout:Upgraded: 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:58:38.986 INFO:teuthology.orchestra.run.smithi146.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout:Installed: 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.987 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.988 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:58:38.989 INFO:teuthology.orchestra.run.smithi146.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:58:38.990 INFO:teuthology.orchestra.run.smithi146.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:38.991 INFO:teuthology.orchestra.run.smithi146.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.992 INFO:teuthology.orchestra.run.smithi146.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:58:38.993 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:58:38.994 INFO:teuthology.orchestra.run.smithi146.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:58:38.995 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:58:38.996 INFO:teuthology.orchestra.run.smithi146.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:38.997 INFO:teuthology.orchestra.run.smithi146.stdout:Complete! 2024-01-16T14:58:39.550 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:43.337 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 1/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 3/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 4/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-7629.g73a45 5/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 6/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 7/167 2024-01-16T14:58:43.338 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 8/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 9/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 10/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 11/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 12/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 13/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_ 14/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 15/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 16/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_6 17/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86 18/167 2024-01-16T14:58:43.339 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 19/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el 20/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ceph-common-2:18.0.0-7629.g73a4532f.el9. 21/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_6 22/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 23/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 24/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 25/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 26/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 27/167 2024-01-16T14:58:43.340 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 28/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f. 29/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noa 30/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.n 31/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a 32/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el 33/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 34/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.e 35/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 36/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 37/167 2024-01-16T14:58:43.341 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 38/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : automake-1.16.2-8.el9.noarch 39/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : bison-3.7.4-5.el9.x86_64 40/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 41/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 42/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/167 2024-01-16T14:58:43.342 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 52/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 53/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 54/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-5.4.4-4.el9.x86_64 55/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/167 2024-01-16T14:58:43.343 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 58/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-13.11-1.el9.x86_64 59/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 60/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 61/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 62/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 63/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 64/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/167 2024-01-16T14:58:43.344 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 72/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 73/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 74/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 75/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 76/167 2024-01-16T14:58:43.345 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 77/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 78/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 79/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 80/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 81/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 82/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 83/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 84/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : jq-1.6-16.el9.x86_64 85/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 86/167 2024-01-16T14:58:43.346 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 87/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 88/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 89/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 90/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-ply-3.11-14.el9.noarch 91/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 92/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 93/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 94/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 95/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 96/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 97/167 2024-01-16T14:58:43.347 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 98/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 99/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 100/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 101/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 102/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 103/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 104/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 105/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 106/167 2024-01-16T14:58:43.348 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 107/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 108/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 109/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-google-auth-1:2.26.0-1.el9.noarch 110/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 111/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 112/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 113/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 114/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 115/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 116/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 117/167 2024-01-16T14:58:43.349 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 118/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 119/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 120/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 121/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 122/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 123/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 124/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 125/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 126/167 2024-01-16T14:58:43.350 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 127/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 128/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 129/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 130/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 131/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 132/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 133/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : re2-1:20211101-3.el9.x86_64 134/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 135/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 136/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 137/167 2024-01-16T14:58:43.351 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 138/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 139/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 140/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 141/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-1:1.45.9-1.el9.x86_64 142/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 143/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-libnm-1:1.45.9-1.el9.x86_64 144/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 145/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-team-1:1.45.9-1.el9.x86_64 146/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 147/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-tui-1:1.45.9-1.el9.x86_64 148/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 149/167 2024-01-16T14:58:43.352 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : acl-2.3.1-4.el9.x86_64 150/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : acl-2.3.1-3.el9.x86_64 151/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 152/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 153/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 154/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 155/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 156/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 157/167 2024-01-16T14:58:43.353 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 158/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 159/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 160/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 161/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 162/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 163/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 164/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 165/167 2024-01-16T14:58:43.354 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 166/167 2024-01-16T14:58:45.586 INFO:teuthology.orchestra.run.smithi053.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 167/167 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout:Upgraded: 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-libnm-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-team-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: NetworkManager-tui-1:1.45.9-1.el9.x86_64 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: acl-2.3.1-4.el9.x86_64 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-01-16T14:58:45.587 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: libacl-2.3.1-4.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: librados2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: librbd1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout:Installed: 2024-01-16T14:58:45.588 INFO:teuthology.orchestra.run.smithi053.stdout: automake-1.16.2-8.el9.noarch 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: bison-3.7.4-5.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-base-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-grafana-dashboards-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-immutable-object-cache-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mds-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-dashboard-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.589 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-diskprediction-local-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-modules-core-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mgr-rook-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-mon-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-osd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-prometheus-alerts-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-radosgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-selinux-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-test-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: ceph-volume-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephadm-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: cephfs-top-2:18.0.0-7629.g73a4532f.el9.noarch 2024-01-16T14:58:45.590 INFO:teuthology.orchestra.run.smithi053.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: flex-2.6.4-9.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: fuse-2.9.9-15.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: jq-1.6-16.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-9.0.0-10.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.591 INFO:teuthology.orchestra.run.smithi053.stdout: libcephfs2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libcephsqlite-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: liboath-2.6.7-2.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: librados-devel-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libradosstriper1-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: librgw2-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libtool-2.4.6-45.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libunwind-1.6.2-1.el9.x86_64 2024-01-16T14:58:45.592 INFO:teuthology.orchestra.run.smithi053.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: lua-5.4.4-4.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: luarocks-3.9.2-1.el9.noarch 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-0.3.21-2.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-13.11-1.el9.x86_64 2024-01-16T14:58:45.593 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: postgresql-server-13.11-1.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-babel-2.9.1-2.el9.noarch 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-argparse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ceph-common-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cephfs-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-01-16T14:58:45.594 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-google-auth-1:2.26.0-1.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jinja2-2.11.3-4.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-01-16T14:58:45.595 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-01-16T14:58:45.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-01-16T14:58:45.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-01-16T14:58:45.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-01-16T14:58:45.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-01-16T14:58:45.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-mako-1.1.4-6.el9.noarch 2024-01-16T14:58:45.596 INFO:teuthology.orchestra.run.smithi053.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-ply-3.11-14.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-portend-3.1.0-2.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-01-16T14:58:45.597 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pycparser-2.20-6.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-pytz-2021.1-5.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rados-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-2.25.1-7.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-01-16T14:58:45.598 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rgw-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-routes-2.5.1-5.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-rsa-4.9-2.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-toml-0.10.2-6.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-webob-1.8.7-6.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-01-16T14:58:45.599 INFO:teuthology.orchestra.run.smithi053.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-fuse-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-mirror-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: rbd-nbd-2:18.0.0-7629.g73a4532f.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: re2-1:20211101-3.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: socat-1.7.4.1-5.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: thrift-0.15.0-2.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: uuid-1.6.2-55.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-01-16T14:58:45.600 INFO:teuthology.orchestra.run.smithi053.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-01-16T14:58:45.601 INFO:teuthology.orchestra.run.smithi053.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-01-16T14:58:45.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:45.601 INFO:teuthology.orchestra.run.smithi053.stdout:Complete! 2024-01-16T14:58:46.171 DEBUG:teuthology.parallel:result is None 2024-01-16T14:58:46.172 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:58:46.338 DEBUG:teuthology.orchestra.run.smithi053:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:58:46.402 INFO:teuthology.orchestra.run.smithi053.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:46.402 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:46.403 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:58:46.405 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=73a4532fc4d97adee0b985d4d42874ea351a1c86 2024-01-16T14:58:46.547 DEBUG:teuthology.orchestra.run.smithi146:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-16T14:58:46.617 INFO:teuthology.orchestra.run.smithi146.stdout:18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:46.618 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-7629.g73a4532f.el9 2024-01-16T14:58:46.618 INFO:teuthology.task.install:The correct ceph version 18.0.0-7629.g73a4532f is installed. 2024-01-16T14:58:46.620 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-16T14:58:46.620 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:46.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:58:46.700 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:46.701 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-16T14:58:46.801 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-16T14:58:46.801 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:46.801 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:58:46.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:58:47.021 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:47.022 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-01-16T14:58:47.115 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-16T14:58:47.268 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-16T14:58:47.268 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:47.268 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:58:47.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:58:47.473 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:47.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-16T14:58:47.555 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-16T14:58:47.670 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-16T14:58:47.671 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:47.671 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:58:47.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:58:47.871 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:47.871 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-01-16T14:58:47.956 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-16T14:58:48.075 INFO:teuthology.run_tasks:Running task ceph... 2024-01-16T14:58:48.160 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-01-16T14:58:48.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 777 /var/log/ceph 2024-01-16T14:58:48.165 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-01-16T14:58:48.244 INFO:tasks.ceph:Disabling ceph logrotate... 2024-01-16T14:58:48.245 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:58:48.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-01-16T14:58:48.323 INFO:tasks.ceph:Creating extra log directories... 2024-01-16T14:58:48.324 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:58:48.328 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-01-16T14:58:48.435 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-01-16T14:58:48.436 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-01-16T14:58:48.436 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-01-16T14:58:48.436 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:48.436 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:58:48.489 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:48.689 DEBUG:teuthology.orchestra.run.smithi053:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:48.757 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:48.757 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-01-16T14:58:48.812 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:49.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-01-16T14:58:49.175 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-01-16T14:58:49.175 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-01-16T14:58:49.175 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206', 'branch': 'rishabh-clone-stats', 'description': 'fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}}', 'email': None, 'first_in_suite': False, 'job_id': '7518206', 'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'rishabh-clone-stats'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'rishabh-clone-stats', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 50, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 3695, 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'sleep_before_teardown': 0, 'suite': 'fs:volumes', 'suite_branch': 'rishabh-clone-stats', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi053.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM0JQjhuNggf5ic6Z+v+ATyjbZxtPtAlHaYJNWISkWvf8Z0gynhR96Yu8TtxOs6iLtho5IdeDI+TpDb1zD7vs7E=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPgRJ4UNIuRtkJXGmoH7Fyp95mWSrJnXr95VQfYW7AHroNiCVmwc+foJoouNVGIELbaIXhrMCms6FZ2ILBwlnIQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'flavor': 'default', 'kdb': True, 'sha1': '2668949e0971849893a3fb7f177ce9032fc65780'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '73a4532fc4d97adee0b985d4d42874ea351a1c86', 'cluster': 'ceph'}}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestCloneProgress']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'cd45576300487d997e5a85abed65500b9f5d143b', 'timestamp': '2024-01-16_14:39:36', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-01-16T14:58:49.176 DEBUG:teuthology.orchestra.run.smithi053:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:58:49.182 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-01-16T14:58:49.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:58:49.264 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-01-16T14:58:49.362 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:49.362 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:58:49.402 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:58:49.402 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_1 2024-01-16T14:58:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:58:49.487 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-01-16 14:58:39.168244099 +0000 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-01-16 14:56:11.447909569 +0000 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-01-16 14:56:11.447909569 +0000 2024-01-16T14:58:49.488 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-01-16 14:56:11.447909569 +0000 2024-01-16T14:58:49.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:58:49.598 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-01-16T14:58:49.598 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-01-16T14:58:49.598 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000230169 s, 2.2 MB/s 2024-01-16T14:58:49.601 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:58:49.645 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_2 2024-01-16T14:58:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:58:49.725 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:49.725 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-01-16T14:58:49.725 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:49.725 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:49.725 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-01-16 14:58:39.168244099 +0000 2024-01-16T14:58:49.726 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-01-16 14:56:12.017897831 +0000 2024-01-16T14:58:49.726 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-01-16 14:56:12.017897831 +0000 2024-01-16T14:58:49.726 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-01-16 14:56:12.017897831 +0000 2024-01-16T14:58:49.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:58:49.835 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-01-16T14:58:49.835 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-01-16T14:58:49.836 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000205167 s, 2.5 MB/s 2024-01-16T14:58:49.838 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:58:49.878 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_3 2024-01-16T14:58:49.958 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-01-16 14:58:39.169244079 +0000 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-01-16 14:56:12.616885496 +0000 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-01-16 14:56:12.616885496 +0000 2024-01-16T14:58:49.959 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-01-16 14:56:12.616885496 +0000 2024-01-16T14:58:49.959 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:58:50.062 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-01-16T14:58:50.062 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-01-16T14:58:50.062 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.00018439 s, 2.8 MB/s 2024-01-16T14:58:50.065 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:58:50.105 DEBUG:teuthology.orchestra.run.smithi053:> stat /dev/vg_nvme/lv_4 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout:Access: 2024-01-16 14:58:39.170244058 +0000 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout:Modify: 2024-01-16 14:56:13.188873716 +0000 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout:Change: 2024-01-16 14:56:13.188873716 +0000 2024-01-16T14:58:50.186 INFO:teuthology.orchestra.run.smithi053.stdout: Birth: 2024-01-16 14:56:13.188873716 +0000 2024-01-16T14:58:50.187 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:58:50.291 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records in 2024-01-16T14:58:50.292 INFO:teuthology.orchestra.run.smithi053.stderr:1+0 records out 2024-01-16T14:58:50.292 INFO:teuthology.orchestra.run.smithi053.stderr:512 bytes copied, 0.000191252 s, 2.7 MB/s 2024-01-16T14:58:50.294 DEBUG:teuthology.orchestra.run.smithi053:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:58:50.335 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:50.335 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:50.335 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-01-16T14:58:50.393 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-01-16T14:58:50.393 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-01-16T14:58:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-01-16T14:58:50.447 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-01-16T14:58:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:50.447 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-01-16 14:58:33.000267884 +0000 2024-01-16T14:58:50.448 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-01-16 14:56:02.926459605 +0000 2024-01-16T14:58:50.448 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-01-16 14:56:02.926459605 +0000 2024-01-16T14:58:50.448 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-01-16 14:56:02.926459605 +0000 2024-01-16T14:58:50.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-01-16T14:58:50.535 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-01-16T14:58:50.535 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-01-16T14:58:50.535 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00030831 s, 1.7 MB/s 2024-01-16T14:58:50.539 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-01-16T14:58:50.592 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-01-16 14:58:33.001267868 +0000 2024-01-16T14:58:50.635 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-01-16 14:56:03.485450842 +0000 2024-01-16T14:58:50.636 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-01-16 14:56:03.485450842 +0000 2024-01-16T14:58:50.636 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-01-16 14:56:03.485450842 +0000 2024-01-16T14:58:50.636 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-01-16T14:58:50.769 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-01-16T14:58:50.770 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-01-16T14:58:50.770 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000323127 s, 1.6 MB/s 2024-01-16T14:58:50.774 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-01-16T14:58:50.829 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-01-16 14:58:33.002267853 +0000 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-01-16 14:56:04.088441389 +0000 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-01-16 14:56:04.088441389 +0000 2024-01-16T14:58:50.878 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-01-16 14:56:04.088441389 +0000 2024-01-16T14:58:50.879 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-01-16T14:58:50.961 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-01-16T14:58:50.961 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-01-16T14:58:50.961 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00031711 s, 1.6 MB/s 2024-01-16T14:58:50.965 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-01-16T14:58:51.019 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-01-16T14:58:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-01-16T14:58:51.066 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-01-16T14:58:51.066 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 691 Links: 1 2024-01-16T14:58:51.066 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-01-16T14:58:51.066 INFO:teuthology.orchestra.run.smithi146.stdout:Context: system_u:object_r:device_t:s0 2024-01-16T14:58:51.067 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-01-16 14:58:33.003267837 +0000 2024-01-16T14:58:51.067 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-01-16 14:56:04.663432375 +0000 2024-01-16T14:58:51.067 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-01-16 14:56:04.663432375 +0000 2024-01-16T14:58:51.067 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: 2024-01-16 14:56:04.663432375 +0000 2024-01-16T14:58:51.067 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-01-16T14:58:51.199 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-01-16T14:58:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-01-16T14:58:51.200 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000341508 s, 1.5 MB/s 2024-01-16T14:58:51.204 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-01-16T14:58:51.257 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:51.257 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:58:51.257 INFO:tasks.ceph:Generating config... 2024-01-16T14:58:51.259 INFO:tasks.ceph:[client] client mount timeout = 600 2024-01-16T14:58:51.259 INFO:tasks.ceph:[client] debug client = 20 2024-01-16T14:58:51.259 INFO:tasks.ceph:[client] debug ms = 1 2024-01-16T14:58:51.259 INFO:tasks.ceph:[client] pid file = 2024-01-16T14:58:51.259 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-01-16T14:58:51.259 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] debug mds = 20 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] debug ms = 1 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] mds debug frag = True 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-01-16T14:58:51.259 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mgr] debug client = 20 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mgr] debug finisher = 20 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mgr] debug ms = 1 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mon] debug mon = 20 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mon] debug ms = 1 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mon] debug paxos = 20 2024-01-16T14:58:51.260 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-01-16T14:58:51.260 INFO:tasks.ceph:[osd] bdev async discard = True 2024-01-16T14:58:51.260 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-01-16T14:58:51.260 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-01-16T14:58:51.260 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-01-16T14:58:51.260 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-01-16T14:58:51.260 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] debug ms = 1 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] debug osd = 20 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-01-16T14:58:51.261 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-01-16T14:58:51.261 INFO:tasks.ceph:Setting up mon.a... 2024-01-16T14:58:51.263 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-01-16T14:58:51.374 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.keyring 2024-01-16T14:58:51.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-01-16T14:58:51.487 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:58:51.555 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.53'), ('mon.b', '172.21.15.146'), ('mon.c', '[v2:172.21.15.146:3301,v1:172.21.15.146:6790]')] 2024-01-16T14:58:51.556 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.53,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-01-16T14:58:51.557 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:51.557 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:58:51.593 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.53 --add b 172.21.15.146 --addv c '[v2:172.21.15.146:3301,v1:172.21.15.146:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:51.718 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:51.718 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: generated fsid c2138e65-ef75-44e8-8327-90d7ebaa10cd 2024-01-16T14:58:51.718 INFO:teuthology.orchestra.run.smithi053.stdout:setting min_mon_release = pacific 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:epoch 0 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:fsid c2138e65-ef75-44e8-8327-90d7ebaa10cd 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:last_changed 2024-01-16T14:58:51.738318+0000 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:created 2024-01-16T14:58:51.738318+0000 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:min_mon_release 16 (pacific) 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:election_strategy: 1 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:0: [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] mon.a 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:1: [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] mon.b 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:2: [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] mon.c 2024-01-16T14:58:51.719 INFO:teuthology.orchestra.run.smithi053.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-01-16T14:58:51.721 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-01-16T14:58:51.769 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c2138e65-ef75-44e8-8327-90d7ebaa10cd... 2024-01-16T14:58:51.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:58:51.774 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-01-16T14:58:51.929 INFO:teuthology.orchestra.run.smithi053.stdout:[global] 2024-01-16T14:58:51.929 INFO:teuthology.orchestra.run.smithi053.stdout: chdir = "" 2024-01-16T14:58:51.929 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: auth supported = cephx 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: filestore xattr use omap = true 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: mon clock drift allowed = 1.000 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: osd crush chooseleaf type = 0 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: auth debug = true 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on old message = true 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: ms die on bug = true 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:58:51.930 INFO:teuthology.orchestra.run.smithi053.stdout: mon pg warn max object skew = 0 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default size = 2 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow primary affinity = true 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd allow pg remap = true 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on no sortbitwise = false 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon warn on too few osds = false 2024-01-16T14:58:51.931 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: mon_allow_pool_size_one = true 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: osd default data pool replay window = 5 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: mon allow pool delete = true 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: mon cluster log file level = debug 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: debug asserts on shutdown = true 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: mon health detail to clog = false 2024-01-16T14:58:51.932 INFO:teuthology.orchestra.run.smithi053.stdout: mon host = "172.21.15.53,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: fsid = c2138e65-ef75-44e8-8327-90d7ebaa10cd 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout:[osd] 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd journal size = 100 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub load threshold = 5.0 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd scrub max interval = 600 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd recover clone overlap = true 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd recovery max chunk = 1048576 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug shutdown = true 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug op order = true 2024-01-16T14:58:51.933 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug verify stray on activate = true 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug trim objects = true 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: osd open classes on start = true 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug pg log writeout = true 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: osd map max advance = 10 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: journal zero on create = true 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: filestore apply finisher threads = 3 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.934 INFO:teuthology.orchestra.run.smithi053.stdout: bdev debug aio = true 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: osd debug misdirected ops = true 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: bdev async discard = True 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: bdev enable discard = True 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore allocator = bitmap 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore block size = 96636764160 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: bluestore fsck on mount = True 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluefs = 1/20 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: debug bluestore = 1/20 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: debug osd = 20 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: debug rocksdb = 4/10 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd full ratio = 0.9 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:58:51.935 INFO:teuthology.orchestra.run.smithi053.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: osd objectstore = bluestore 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr] 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug mgr = 20 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: mgr/telemetry/nag = false 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: debug finisher = 20 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.936 INFO:teuthology.orchestra.run.smithi053.stdout:[mon] 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: debug mon = 20 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: debug paxos = 20 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: debug auth = 20 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: mon data avail warn = 5 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: mon mgr mkfs grace = 240 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd reporter subtree level = osd 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: mon osd prime pg temp = true 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:58:51.937 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: # 1m isn't quite enough 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: mon op complaint time = 120 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout:[client] 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: rgw cache enabled = true 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable ops log = true 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: rgw enable usage log = true 2024-01-16T14:58:51.938 INFO:teuthology.orchestra.run.smithi053.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: client mount timeout = 600 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug client = 20 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: pid file = "" 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.a] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.b] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mon.c] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-01-16T14:58:51.939 INFO:teuthology.orchestra.run.smithi053.stdout:[mds] 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds = 20 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug mds balancer = 20 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: debug ms = 1 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug frag = True 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: mds debug scatterstat = True 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: mds op complaint time = 180 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: mds verify scatter = True 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: osd op complaint time = 180 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: rados mon op timeout = 900 2024-01-16T14:58:51.940 INFO:teuthology.orchestra.run.smithi053.stdout: rados osd op timeout = 900 2024-01-16T14:58:51.946 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-01-16T14:58:51.947 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-01-16T14:58:51.948 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-01-16T14:58:51.949 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "172.21.15.53,172.21.15.146,[v2:172.21.15.146:3301,v1:172.21.15.146:6790]" 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = c2138e65-ef75-44e8-8327-90d7ebaa10cd 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-01-16T14:58:51.950 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.951 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: bdev async discard = True 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: bdev enable discard = True 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore allocator = bitmap 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 1/20 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 1/20 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 4/10 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-01-16T14:58:51.952 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug finisher = 20 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-01-16T14:58:51.953 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-01-16T14:58:51.954 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: mon op complaint time = 120 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: client mount timeout = 600 2024-01-16T14:58:51.955 INFO:teuthology.orchestra.run.smithi146.stdout: debug client = 20 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = "" 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout:[mds] 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds = 20 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: debug mds balancer = 20 2024-01-16T14:58:51.956 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug frag = True 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: mds debug scatterstat = True 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: mds op complaint time = 180 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: mds verify scatter = True 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: osd op complaint time = 180 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: rados mon op timeout = 900 2024-01-16T14:58:51.957 INFO:teuthology.orchestra.run.smithi146.stdout: rados osd op timeout = 900 2024-01-16T14:58:51.989 INFO:tasks.ceph:Creating admin key on mon.a... 2024-01-16T14:58:51.989 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-01-16T14:58:52.149 INFO:tasks.ceph:Copying monmap to all nodes... 2024-01-16T14:58:52.150 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:52.150 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-01-16T14:58:52.202 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:52.202 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-01-16T14:58:52.254 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T14:58:52.254 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:52.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:58:52.254 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:58:52.407 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:58:52.407 DEBUG:teuthology.orchestra.run.smithi053:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:52.507 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T14:58:52.507 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:52.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-01-16T14:58:52.507 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-01-16T14:58:52.659 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:58:52.659 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:52.756 INFO:tasks.ceph:Setting up mon nodes... 2024-01-16T14:58:52.756 INFO:tasks.ceph:Setting up mgr nodes... 2024-01-16T14:58:52.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:58:52.980 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-01-16T14:58:52.990 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:58:53.223 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-01-16T14:58:53.234 INFO:tasks.ceph:Setting up mds nodes... 2024-01-16T14:58:53.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:58:53.438 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-01-16T14:58:53.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-01-16T14:58:53.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:58:53.736 INFO:teuthology.orchestra.run.smithi053.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-01-16T14:58:53.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-01-16T14:58:53.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:58:54.032 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-01-16T14:58:54.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-01-16T14:58:54.126 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:58:54.302 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-01-16T14:58:54.311 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-01-16T14:58:54.401 INFO:tasks.ceph_client:Setting up client nodes... 2024-01-16T14:58:54.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-01-16T14:58:54.539 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-01-16T14:58:54.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-01-16T14:58:54.734 INFO:teuthology.orchestra.run.smithi053.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-01-16T14:58:54.795 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-01-16T14:58:54.926 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-01-16T14:58:54.981 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-01-16T14:58:55.103 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-01-16T14:58:55.152 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-01-16T14:58:55.152 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi053.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-01-16T14:58:55.152 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:55.260 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:55.260 INFO:tasks.ceph:role: osd.0 2024-01-16T14:58:55.260 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T14:58:55.260 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:55.360 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:55.361 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:55.393 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-01-16T14:58:55.400 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-01-16T14:58:55.401 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:55.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-01-16T14:58:55.612 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:55.704 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:55.705 INFO:tasks.ceph:role: osd.1 2024-01-16T14:58:55.705 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T14:58:55.705 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:58:55.799 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:55.799 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:55.799 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:55.799 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:55.799 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:55.800 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:55.800 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:55.800 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:55.800 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:55.800 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:55.834 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-01-16T14:58:55.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-01-16T14:58:55.840 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:55.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-01-16T14:58:56.037 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:56.124 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:56.125 INFO:tasks.ceph:role: osd.2 2024-01-16T14:58:56.125 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T14:58:56.125 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:56.220 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:56.254 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-01-16T14:58:56.259 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-01-16T14:58:56.260 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:56.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-01-16T14:58:56.469 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:56.561 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-01-16T14:58:56.561 INFO:tasks.ceph:role: osd.3 2024-01-16T14:58:56.561 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T14:58:56.561 DEBUG:teuthology.orchestra.run.smithi053:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:58:56.660 INFO:teuthology.orchestra.run.smithi053.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:58:56.661 INFO:teuthology.orchestra.run.smithi053.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:58:56.694 INFO:teuthology.orchestra.run.smithi053.stdout:Discarding blocks...Done. 2024-01-16T14:58:56.699 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi053.front.sepia.ceph.com -o noatime 2024-01-16T14:58:56.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:56.818 DEBUG:teuthology.orchestra.run.smithi053:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-01-16T14:58:56.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:58:57.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:58:57.091+0000 7f230ab37500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-01-16T14:58:57.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:58:57.092+0000 7f230ab37500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-01-16T14:58:57.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:58:57.092+0000 7f230ab37500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:58:57.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:58:57.092+0000 7f230ab37500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:58:57.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:58:57.092+0000 7f230ab37500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-01-16T14:58:57.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:58:57.096+0000 7f230ab37500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-01-16T14:59:00.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-01-16T14:59:00.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:00.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:00.531+0000 7fb9509ad500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-01-16T14:59:00.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:00.532+0000 7fb9509ad500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-01-16T14:59:00.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:00.532+0000 7fb9509ad500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:59:00.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:00.532+0000 7fb9509ad500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:59:00.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:00.532+0000 7fb9509ad500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-01-16T14:59:00.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:00.534+0000 7fb9509ad500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-01-16T14:59:03.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-01-16T14:59:03.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:03.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:03.952+0000 7f546d60e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-01-16T14:59:03.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:03.952+0000 7f546d60e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-01-16T14:59:03.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:03.952+0000 7f546d60e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:59:03.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:03.952+0000 7f546d60e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:59:03.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:03.953+0000 7f546d60e500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-01-16T14:59:03.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:03.955+0000 7f546d60e500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-01-16T14:59:06.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-01-16T14:59:06.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:07.124+0000 7f6388c1c500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-01-16T14:59:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:07.124+0000 7f6388c1c500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-01-16T14:59:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:07.124+0000 7f6388c1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:59:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:07.124+0000 7f6388c1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:59:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:07.124+0000 7f6388c1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-01-16T14:59:07.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:07.126+0000 7f6388c1c500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-01-16T14:59:10.291 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-01-16T14:59:10.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-01-16T14:59:10.492 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:59:10.492 INFO:tasks.ceph:role: osd.4 2024-01-16T14:59:10.492 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T14:59:10.492 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-01-16T14:59:10.589 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:59:10.590 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:59:11.112 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-01-16T14:59:11.120 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-01-16T14:59:11.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-01-16T14:59:11.300 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-01-16T14:59:11.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-01-16T14:59:11.494 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:59:11.494 INFO:tasks.ceph:role: osd.5 2024-01-16T14:59:11.494 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T14:59:11.494 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:59:11.593 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:59:12.081 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-01-16T14:59:12.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-01-16T14:59:12.087 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-01-16T14:59:12.233 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-01-16T14:59:12.322 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-01-16T14:59:12.408 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:59:12.408 INFO:tasks.ceph:role: osd.6 2024-01-16T14:59:12.409 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T14:59:12.409 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-01-16T14:59:12.500 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:59:12.500 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:59:12.500 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:59:12.500 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:59:12.500 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:59:12.501 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:59:12.501 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:59:12.501 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:59:12.501 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:59:12.501 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:59:13.019 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-01-16T14:59:13.026 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-01-16T14:59:13.026 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-01-16T14:59:13.231 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-01-16T14:59:13.333 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-01-16T14:59:13.423 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-01-16T14:59:13.423 INFO:tasks.ceph:role: osd.7 2024-01-16T14:59:13.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T14:59:13.423 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-01-16T14:59:13.524 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-01-16T14:59:13.986 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-01-16T14:59:13.991 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-01-16T14:59:13.992 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-01-16T14:59:14.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-01-16T14:59:14.226 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:14.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:14.383+0000 7f922e0f6500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-01-16T14:59:14.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:14.388+0000 7f922e0f6500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-01-16T14:59:14.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:14.388+0000 7f922e0f6500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:59:14.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:14.388+0000 7f922e0f6500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:59:14.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:14.388+0000 7f922e0f6500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-01-16T14:59:14.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:14.391+0000 7f922e0f6500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-01-16T14:59:17.545 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-01-16T14:59:17.644 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:17.782 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:17.801+0000 7f1d85e21500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-01-16T14:59:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:17.801+0000 7f1d85e21500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-01-16T14:59:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:17.801+0000 7f1d85e21500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:59:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:17.801+0000 7f1d85e21500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:59:17.783 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:17.801+0000 7f1d85e21500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-01-16T14:59:17.784 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:17.803+0000 7f1d85e21500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-01-16T14:59:19.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:19.186 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:20.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-01-16T14:59:20.824 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:20.961 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:20.980+0000 7fa7c5a0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:20.981+0000 7fa7c5a0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:20.981+0000 7fa7c5a0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:20.981+0000 7fa7c5a0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:59:20.962 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:20.981+0000 7fa7c5a0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-01-16T14:59:20.964 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:20.983+0000 7fa7c5a0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-01-16T14:59:23.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-01-16T14:59:23.987 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:24.120 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:24.139+0000 7fcafc327500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-01-16T14:59:24.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:24.139+0000 7fcafc327500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-01-16T14:59:24.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:24.140+0000 7fcafc327500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:59:24.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:24.140+0000 7fcafc327500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:59:24.121 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:24.140+0000 7fcafc327500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-01-16T14:59:24.122 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:24.142+0000 7fcafc327500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-01-16T14:59:27.049 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-01-16T14:59:27.137 INFO:tasks.ceph:Reading keys from all nodes... 2024-01-16T14:59:27.137 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-01-16T14:59:27.245 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.246 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-01-16T14:59:27.348 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-01-16T14:59:27.447 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-01-16T14:59:27.552 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-01-16T14:59:27.657 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.657 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-01-16T14:59:27.758 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:27.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-01-16T14:59:27.857 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:27.857 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-01-16T14:59:27.952 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:27.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-01-16T14:59:28.043 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.043 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-01-16T14:59:28.127 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.128 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-01-16T14:59:28.267 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-01-16T14:59:28.361 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.361 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-01-16T14:59:28.450 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.450 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-01-16T14:59:28.536 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:28.537 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-01-16T14:59:28.595 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:28.595 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-01-16T14:59:28.649 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.649 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-01-16T14:59:28.706 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:28.706 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-01-16T14:59:28.757 INFO:tasks.ceph:Adding keys to all mons... 2024-01-16T14:59:28.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:59:28.765 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-01-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.y] 2024-01-16T14:59:28.845 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.a] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.c] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww== 2024-01-16T14:59:28.846 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.b] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[mds.d] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.6] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.7] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout:[client.2] 2024-01-16T14:59:28.847 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw== 2024-01-16T14:59:28.848 INFO:teuthology.orchestra.run.smithi146.stdout:[client.1] 2024-01-16T14:59:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA== 2024-01-16T14:59:28.848 INFO:teuthology.orchestra.run.smithi146.stdout:[client.3] 2024-01-16T14:59:28.848 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw== 2024-01-16T14:59:28.853 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.y] 2024-01-16T14:59:28.853 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ== 2024-01-16T14:59:28.853 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.a] 2024-01-16T14:59:28.853 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ== 2024-01-16T14:59:28.853 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.c] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.0] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.1] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.2] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.3] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[mgr.x] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.b] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[mds.d] 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA== 2024-01-16T14:59:28.854 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.4] 2024-01-16T14:59:28.855 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA== 2024-01-16T14:59:28.855 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.5] 2024-01-16T14:59:28.855 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg== 2024-01-16T14:59:28.855 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.6] 2024-01-16T14:59:28.855 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg== 2024-01-16T14:59:28.855 INFO:teuthology.orchestra.run.smithi053.stdout:[osd.7] 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ== 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout:[client.2] 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw== 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout:[client.1] 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA== 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout:[client.3] 2024-01-16T14:59:28.856 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw== 2024-01-16T14:59:28.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:28.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:29.032 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:29.041 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:29.190 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:29.206 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:29.358 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.365 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.528 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.675 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.688 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.850 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:29.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:30.001 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-01-16T14:59:30.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:30.154 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:30.304 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:30.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-01-16T14:59:30.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.610 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.615 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.756 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:30.914 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-01-16T14:59:31.059 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.065 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.221 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.366 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.373 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.525 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.530 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-01-16T14:59:31.680 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-01-16T14:59:31.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-01-16T14:59:31.774 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:59:31.935 DEBUG:teuthology.orchestra.run.smithi053:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-01-16T14:59:32.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-01-16T14:59:32.116 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:59:32.276 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-01-16T14:59:32.364 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-01-16T14:59:32.457 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-01-16T14:59:32.611 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-01-16T14:59:32.696 DEBUG:teuthology.orchestra.run.smithi053:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:32.704 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-01-16T14:59:32.756 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-01-16T14:59:32.756 INFO:tasks.ceph.mon.a:Restarting daemon 2024-01-16T14:59:32.756 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-01-16T14:59:32.762 INFO:tasks.ceph.mon.a:Started 2024-01-16T14:59:32.762 INFO:tasks.ceph.mon.b:Restarting daemon 2024-01-16T14:59:32.762 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-01-16T14:59:32.767 INFO:tasks.ceph.mon.b:Started 2024-01-16T14:59:32.767 INFO:tasks.ceph.mon.c:Restarting daemon 2024-01-16T14:59:32.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-01-16T14:59:32.772 INFO:tasks.ceph.mon.c:Started 2024-01-16T14:59:32.772 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-01-16T14:59:32.772 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-01-16T14:59:32.773 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-01-16T14:59:32.779 INFO:tasks.ceph.mgr.y:Started 2024-01-16T14:59:32.779 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-01-16T14:59:32.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-01-16T14:59:32.784 INFO:tasks.ceph.mgr.x:Started 2024-01-16T14:59:32.784 INFO:tasks.ceph:Setting crush tunables to default 2024-01-16T14:59:32.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph osd crush tunables default 2024-01-16T14:59:32.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.984+0000 7f6bed600640 1 Processor -- start 2024-01-16T14:59:32.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.984+0000 7f6bed600640 1 -- start start 2024-01-16T14:59:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6bed600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6bed600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6bed600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6bed600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be80cdf90 con 0x7f6be80b4280 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6bed600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be80b4ba0 con 0x7f6be806e7e0 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6bed600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8072000 con 0x7f6be80b3c10 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:35572/0 (socket says 172.21.15.53:35572) 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 -- 172.21.15.53:0/1482543789 learned_addr learned my addr 172.21.15.53:0/1482543789 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6a00640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6000640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 -- 172.21.15.53:0/1482543789 >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 msgr2=0x7f6be806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 -- 172.21.15.53:0/1482543789 >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 msgr2=0x7f6be806ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be7400640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:59:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6000640 1 -- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 msgr2=0x7f6be80713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6000640 1 -- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 msgr2=0x7f6be80713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6000640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6a00640 1 -- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 msgr2=0x7f6be80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6a00640 1 -- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 msgr2=0x7f6be80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6a00640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6000640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:59:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:32.986+0000 7f6be6a00640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-01-16T14:59:33.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.187+0000 7f6be6a00640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.187+0000 7f6be7400640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.146:3300/0 conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.187+0000 7f6be6000640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.188+0000 7f6be7400640 1 -- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 msgr2=0x7f6be80713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.188+0000 7f6be7400640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.188+0000 7f6be7400640 1 -- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 msgr2=0x7f6be80b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.188+0000 7f6be7400640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.188+0000 7f6be7400640 1 -- 172.21.15.53:0/1482543789 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be8071d20 con 0x7f6be806e7e0 2024-01-16T14:59:33.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.189+0000 7f6be7400640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be806ec90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6be8074960 tx=0x7f6be002c080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=824bf56208886586 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.189+0000 7f6be5600640 1 -- 172.21.15.53:0/1482543789 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be002ce30 con 0x7f6be806e7e0 2024-01-16T14:59:33.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.190+0000 7f6be5600640 1 -- 172.21.15.53:0/1482543789 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6be0030070 con 0x7f6be806e7e0 2024-01-16T14:59:33.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.190+0000 7f6be5600640 1 -- 172.21.15.53:0/1482543789 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6be0036620 con 0x7f6be806e7e0 2024-01-16T14:59:33.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 -- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 msgr2=0x7f6be806ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be806ec90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6be8074960 tx=0x7f6be002c080 comp rx=0 tx=0).stop 2024-01-16T14:59:33.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 -- 172.21.15.53:0/1482543789 shutdown_connections 2024-01-16T14:59:33.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be806ec90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 --2- 172.21.15.53:0/1482543789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b3c10 0x7f6be80713d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 --2- 172.21.15.53:0/1482543789 >> v2:172.21.15.53:3300/0 conn(0x7f6be80b4280 0x7f6be80b4660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 -- 172.21.15.53:0/1482543789 >> 172.21.15.53:0/1482543789 conn(0x7f6be80cfef0 msgr2=0x7f6be80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 -- 172.21.15.53:0/1482543789 shutdown_connections 2024-01-16T14:59:33.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.191+0000 7f6bed600640 1 -- 172.21.15.53:0/1482543789 wait complete. 2024-01-16T14:59:33.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.193+0000 7f6bed600640 1 Processor -- start 2024-01-16T14:59:33.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.193+0000 7f6bed600640 1 -- start start 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.194+0000 7f6bed600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.194+0000 7f6bed600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 0x7f6be806d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.194+0000 7f6bed600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b4280 0x7f6be8136570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.194+0000 7f6bed600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be81393f0 con 0x7f6be80b3c10 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.194+0000 7f6bed600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8139810 con 0x7f6be806e7e0 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.194+0000 7f6bed600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be8138770 con 0x7f6be80b4280 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 0x7f6be806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 0x7f6be806d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49478/0 (socket says 172.21.15.53:49478) 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 -- 172.21.15.53:0/3816327222 learned_addr learned my addr 172.21.15.53:0/3816327222 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:33.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be814c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:35600/0 (socket says 172.21.15.53:35600) 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be7400640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b4280 0x7f6be8136570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 -- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b4280 msgr2=0x7f6be8136570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b4280 0x7f6be8136570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 -- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 msgr2=0x7f6be814c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be814c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 -- 172.21.15.53:0/3816327222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be00097c0 con 0x7f6be80b3c10 2024-01-16T14:59:33.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.195+0000 7f6be6000640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 0x7f6be806d800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000d8f0 tx=0x7f6bd000ddb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0012e50 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd001b440 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be8138a50 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd001b7f0 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be8139010 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6bd001bc50 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.196+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6bd001a720 con 0x7f6be80b3c10 2024-01-16T14:59:33.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.197+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be80675d0 con 0x7f6be80b3c10 2024-01-16T14:59:33.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.201+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f6bd0024d50 con 0x7f6be80b3c10 2024-01-16T14:59:33.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.313+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6be81401e0 con 0x7f6be80b3c10 2024-01-16T14:59:33.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.317+0000 7f6bdea00640 1 -- 172.21.15.53:0/3816327222 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f6bd0024440 con 0x7f6be80b3c10 2024-01-16T14:59:33.298 INFO:teuthology.orchestra.run.smithi053.stderr:adjusted tunables profile to default 2024-01-16T14:59:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.318+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 msgr2=0x7f6be806d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.318+0000 7f6bed600640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 0x7f6be806d800 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6bd000d8f0 tx=0x7f6bd000ddb0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.318+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 shutdown_connections 2024-01-16T14:59:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.318+0000 7f6bed600640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6be80b4280 0x7f6be8136570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.319+0000 7f6bed600640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6be80b3c10 0x7f6be806d800 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.319+0000 7f6bed600640 1 --2- 172.21.15.53:0/3816327222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6be806e7e0 0x7f6be814c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.319+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 >> 172.21.15.53:0/3816327222 conn(0x7f6be80cfef0 msgr2=0x7f6be80ce420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.319+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 shutdown_connections 2024-01-16T14:59:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:33.319+0000 7f6bed600640 1 -- 172.21.15.53:0/3816327222 wait complete. 2024-01-16T14:59:33.316 INFO:tasks.ceph:check_enable_crimson: False 2024-01-16T14:59:33.316 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-01-16T14:59:33.317 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:33.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-01-16T14:59:33.402 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:33.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-01-16T14:59:33.537 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:33.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-01-16T14:59:33.630 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T14:59:33.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-01-16T14:59:33.703 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:33.722+0000 7f61be83b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:59:33.713 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:33.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-01-16T14:59:33.765 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:33.784+0000 7f61be83b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:59:33.775 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:33.794+0000 7fed924fa040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-01-16T14:59:33.807 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:33.807 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-01-16T14:59:33.839 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:33.858+0000 7fed924fa040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-01-16T14:59:33.935 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:33.935 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-01-16T14:59:34.013 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T14:59:34.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-01-16T14:59:34.140 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new bc153f72-2c5f-47f1-b716-49b22c3153a1 0 2024-01-16T14:59:34.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.384+0000 7f2378400640 1 Processor -- start 2024-01-16T14:59:34.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.385+0000 7f2378400640 1 -- start start 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.386+0000 7f2378400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f2370161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.386+0000 7f2378400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f23700ced50 0x7f23700d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.386+0000 7f2378400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23700cf1b0 con 0x7f23700ce8f0 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.386+0000 7f2378400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23701618a0 con 0x7f23700ced50 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.386+0000 7f2378400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23700d0de0 con 0x7f2370160f80 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2375a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f2370161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2376400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f23700ced50 0x7f23700d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2375a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f2370161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:39816/0 (socket says 172.21.15.146:39816) 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2376400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f23700ced50 0x7f23700d02f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59048/0 (socket says 172.21.15.146:59048) 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2375a00640 1 -- 172.21.15.146:0/796876658 learned_addr learned my addr 172.21.15.146:0/796876658 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:34.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2375000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f23700ce8f0 0x7f23700cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47498/0 (socket says 172.21.15.146:47498) 2024-01-16T14:59:34.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2376400640 1 -- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 msgr2=0x7f2370161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2376400640 1 --2- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f2370161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2376400640 1 -- 172.21.15.146:0/796876658 >> v1:172.21.15.146:6789/0 conn(0x7f23700ce8f0 legacy=0x7f23700cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.387+0000 7f2376400640 1 -- 172.21.15.146:0/796876658 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23700d10c0 con 0x7f23700ced50 2024-01-16T14:59:34.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.388+0000 7f2376400640 1 --2- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ced50 0x7f23700d02f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2368009d90 tx=0x7f236802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c005257023b12fe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.388+0000 7f236fe00640 1 -- 172.21.15.146:0/796876658 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236802cc60 con 0x7f23700ced50 2024-01-16T14:59:34.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.388+0000 7f236fe00640 1 -- 172.21.15.146:0/796876658 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236802cdc0 con 0x7f23700ced50 2024-01-16T14:59:34.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.388+0000 7f236fe00640 1 -- 172.21.15.146:0/796876658 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368034660 con 0x7f23700ced50 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 -- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ced50 msgr2=0x7f23700d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 --2- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ced50 0x7f23700d02f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2368009d90 tx=0x7f236802beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 -- 172.21.15.146:0/796876658 shutdown_connections 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 --2- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ced50 0x7f23700d02f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 --2- 172.21.15.146:0/796876658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f2370161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 -- 172.21.15.146:0/796876658 >> 172.21.15.146:0/796876658 conn(0x7f237009c550 msgr2=0x7f23700d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 -- 172.21.15.146:0/796876658 shutdown_connections 2024-01-16T14:59:34.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.389+0000 7f2378400640 1 -- 172.21.15.146:0/796876658 wait complete. 2024-01-16T14:59:34.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.391+0000 7f2378400640 1 Processor -- start 2024-01-16T14:59:34.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.391+0000 7f2378400640 1 -- start start 2024-01-16T14:59:34.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2378400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ce8f0 0x7f23700b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2378400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23700ced50 0x7f23701524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2378400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f23701529e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2378400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700ccba0 con 0x7f23700ced50 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2378400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cbf90 con 0x7f23700ce8f0 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2378400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23700cc270 con 0x7f2370160f80 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2375a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ce8f0 0x7f23700b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2376400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f23701529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2375a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ce8f0 0x7f23700b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59062/0 (socket says 172.21.15.146:59062) 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2376400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f23701529e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:39832/0 (socket says 172.21.15.146:39832) 2024-01-16T14:59:34.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.392+0000 7f2375a00640 1 -- 172.21.15.146:0/2876160918 learned_addr learned my addr 172.21.15.146:0/2876160918 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:34.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2376400640 1 -- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ce8f0 msgr2=0x7f23700b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2376400640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ce8f0 0x7f23700b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2376400640 1 -- 172.21.15.146:0/2876160918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23700ced50 msgr2=0x7f23701524a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2375000640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23700ced50 0x7f23701524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2376400640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23700ced50 0x7f23701524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2376400640 1 -- 172.21.15.146:0/2876160918 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2368009a70 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2376400640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f23701529e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2368033000 tx=0x7f2368004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368040650 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23680407b0 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2368040b60 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.393+0000 7f2378400640 1 -- 172.21.15.146:0/2876160918 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23700cc550 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.394+0000 7f2378400640 1 -- 172.21.15.146:0/2876160918 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23700cd030 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.394+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2368049400 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.394+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f236803e400 con 0x7f2370160f80 2024-01-16T14:59:34.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.395+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2338005250 con 0x7f2370160f80 2024-01-16T14:59:34.379 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.398+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f2368067ce0 con 0x7f2370160f80 2024-01-16T14:59:34.394 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.413+0000 7f61be83b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:59:34.457 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.476+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bc153f72-2c5f-47f1-b716-49b22c3153a1", "id": 0} v 0) v1 -- 0x7f23380055a0 con 0x7f2370160f80 2024-01-16T14:59:34.461 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.480+0000 7f61be83b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:59:34.464 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.483+0000 7f236d600640 1 -- 172.21.15.146:0/2876160918 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bc153f72-2c5f-47f1-b716-49b22c3153a1", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2368056030 con 0x7f2370160f80 2024-01-16T14:59:34.465 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-01-16T14:59:34.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 msgr2=0x7f23701529e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f23701529e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2368033000 tx=0x7f2368004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:34.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 shutdown_connections 2024-01-16T14:59:34.466 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2370160f80 0x7f23701529e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f23700ced50 0x7f23701524a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 --2- 172.21.15.146:0/2876160918 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f23700ce8f0 0x7f23700b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 >> 172.21.15.146:0/2876160918 conn(0x7f237009c550 msgr2=0x7f23700b4520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 shutdown_connections 2024-01-16T14:59:34.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.485+0000 7f2366a00640 1 -- 172.21.15.146:0/2876160918 wait complete. 2024-01-16T14:59:34.480 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.500+0000 7fed924fa040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-01-16T14:59:34.484 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 61465415-c9f2-4095-9f00-2721503d20f9 1 2024-01-16T14:59:34.520 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.539+0000 7f61be83b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:59:34.544 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.564+0000 7fed924fa040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-01-16T14:59:34.590 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.609+0000 7f61be83b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:59:34.598 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.617+0000 7fed924fa040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-01-16T14:59:34.641 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.660+0000 7f61be83b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:59:34.662 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.682+0000 7fed924fa040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.686+0000 7f3a44e00640 1 Processor -- start 2024-01-16T14:59:34.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.686+0000 7f3a44e00640 1 -- start start 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a44e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3a400cf9a0 0x7f3a400cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a400b31d0 0x7f3a400b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a44e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a400d0320 con 0x7f3a4015d4b0 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a44e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a400d1c60 con 0x7f3a400cf9a0 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a44e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400b2730 con 0x7f3a400b31d0 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a400b31d0 0x7f3a400b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a3ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3a400cf9a0 0x7f3a400cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a3f400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f3a4015d4b0 0x7f3a400d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47506/0 (socket says 172.21.15.146:47506) 2024-01-16T14:59:34.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.688+0000 7f3a3f400640 1 -- 172.21.15.146:0/296707205 learned_addr learned my addr 172.21.15.146:0/296707205 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-01-16T14:59:34.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.689+0000 7f3a3fe00640 1 -- 172.21.15.146:0/296707205 >> v2:172.21.15.146:3300/0 conn(0x7f3a400cf9a0 msgr2=0x7f3a400cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.689+0000 7f3a3fe00640 1 --2- 172.21.15.146:0/296707205 >> v2:172.21.15.146:3300/0 conn(0x7f3a400cf9a0 0x7f3a400cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.689+0000 7f3a3fe00640 1 -- 172.21.15.146:0/296707205 >> v1:172.21.15.146:6789/0 conn(0x7f3a4015d4b0 legacy=0x7f3a400d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.689+0000 7f3a3fe00640 1 -- 172.21.15.146:0/296707205 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a400d1ec0 con 0x7f3a400b31d0 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.689+0000 7f3a3e000640 1 -- 172.21.15.146:0/296707205 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333864908 0 0) 0x7f3a400d0320 con 0x7f3a4015d4b0 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.689+0000 7f3a3ea00640 1 --2- 172.21.15.146:0/296707205 >> v2:172.21.15.146:3300/0 conn(0x7f3a400cf9a0 0x7f3a400cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:34.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.690+0000 7f3a3fe00640 1 --2- 172.21.15.146:0/296707205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a400b31d0 0x7f3a400b3660 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009d90 tx=0x7f3a3802beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=808526e049642c1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.690+0000 7f3a3e000640 1 -- 172.21.15.146:0/296707205 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a3802cc60 con 0x7f3a400b31d0 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.690+0000 7f3a3e000640 1 -- 172.21.15.146:0/296707205 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a3802cdc0 con 0x7f3a400b31d0 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.690+0000 7f3a3e000640 1 -- 172.21.15.146:0/296707205 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38034660 con 0x7f3a400b31d0 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.690+0000 7f3a44e00640 1 -- 172.21.15.146:0/296707205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a400b31d0 msgr2=0x7f3a400b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.690+0000 7f3a44e00640 1 --2- 172.21.15.146:0/296707205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a400b31d0 0x7f3a400b3660 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a38009d90 tx=0x7f3a3802beb0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.691+0000 7f3a44e00640 1 -- 172.21.15.146:0/296707205 shutdown_connections 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.691+0000 7f3a44e00640 1 --2- 172.21.15.146:0/296707205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a400b31d0 0x7f3a400b3660 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.691+0000 7f3a44e00640 1 --2- 172.21.15.146:0/296707205 >> v2:172.21.15.146:3300/0 conn(0x7f3a400cf9a0 0x7f3a400cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.691+0000 7f3a44e00640 1 -- 172.21.15.146:0/296707205 >> 172.21.15.146:0/296707205 conn(0x7f3a400d3000 msgr2=0x7f3a400d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.691+0000 7f3a44e00640 1 -- 172.21.15.146:0/296707205 shutdown_connections 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.691+0000 7f3a44e00640 1 -- 172.21.15.146:0/296707205 wait complete. 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.692+0000 7f3a44e00640 1 Processor -- start 2024-01-16T14:59:34.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.692+0000 7f3a44e00640 1 -- start start 2024-01-16T14:59:34.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.693+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a400b31d0 0x7f3a401545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.693+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a400cf9a0 0x7f3a4009fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.693+0000 7f3a44e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a4015d4b0 0x7f3a400cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:34.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.693+0000 7f3a44e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400a0da0 con 0x7f3a400cf9a0 2024-01-16T14:59:34.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.693+0000 7f3a44e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400cb6a0 con 0x7f3a400b31d0 2024-01-16T14:59:34.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.693+0000 7f3a44e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a40154af0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a400b31d0 0x7f3a401545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a4015d4b0 0x7f3a400cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a400b31d0 0x7f3a401545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59074/0 (socket says 172.21.15.146:59074) 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3f400640 1 -- 172.21.15.146:0/3319676552 learned_addr learned my addr 172.21.15.146:0/3319676552 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3ea00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a400cf9a0 0x7f3a4009fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 -- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a400b31d0 msgr2=0x7f3a401545b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a400b31d0 0x7f3a401545b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 -- 172.21.15.146:0/3319676552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a400cf9a0 msgr2=0x7f3a4009fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a400cf9a0 0x7f3a4009fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 -- 172.21.15.146:0/3319676552 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a28009400 con 0x7f3a4015d4b0 2024-01-16T14:59:34.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.694+0000 7f3a3fe00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a4015d4b0 0x7f3a400cddf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a38033000 tx=0x7f3a38004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38040650 con 0x7f3a4015d4b0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a380407f0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a38040ba0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a38009a70 con 0x7f3a4015d4b0 2024-01-16T14:59:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a400a0ac0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3a38049400 con 0x7f3a4015d4b0 2024-01-16T14:59:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.695+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f3a3803e7b0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.696+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a40067450 con 0x7f3a4015d4b0 2024-01-16T14:59:34.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.698+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f3a38067ce0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.714 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.734+0000 7fed924fa040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-01-16T14:59:34.757 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.776+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "61465415-c9f2-4095-9f00-2721503d20f9", "id": 1} v 0) v1 -- 0x7f3a400cb7e0 con 0x7f3a4015d4b0 2024-01-16T14:59:34.763 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.782+0000 7f61be83b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:59:34.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.784+0000 7f3a37400640 1 -- 172.21.15.146:0/3319676552 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "61465415-c9f2-4095-9f00-2721503d20f9", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3a38056030 con 0x7f3a4015d4b0 2024-01-16T14:59:34.765 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T14:59:34.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a4015d4b0 msgr2=0x7f3a400cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a4015d4b0 0x7f3a400cddf0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a38033000 tx=0x7f3a38004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 shutdown_connections 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a4015d4b0 0x7f3a400cddf0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a400cf9a0 0x7f3a4009fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 --2- 172.21.15.146:0/3319676552 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a400b31d0 0x7f3a401545b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.785+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 >> 172.21.15.146:0/3319676552 conn(0x7f3a400d3000 msgr2=0x7f3a4006ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.786+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 shutdown_connections 2024-01-16T14:59:34.767 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:34.786+0000 7f3a44e00640 1 -- 172.21.15.146:0/3319676552 wait complete. 2024-01-16T14:59:34.783 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 2b673652-9d47-429f-aa72-16eca4b02335 2 2024-01-16T14:59:34.833 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.853+0000 7fed924fa040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-01-16T14:59:34.849 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.868+0000 7f61be83b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:34.906 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.925+0000 7f61be83b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:59:34.917 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.936+0000 7fed924fa040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:34.970 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:34.989+0000 7f61be83b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:59:34.972 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:34.992+0000 7fed924fa040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.031+0000 7fe1ad000640 1 Processor -- start 2024-01-16T14:59:35.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.031+0000 7fe1ad000640 1 -- start start 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1ad000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe1a80ced30 0x7fe1a80b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1ad000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80cf1c0 con 0x7fe1a8160f80 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1ad000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1a80d0de0 con 0x7fe1a80ce8f0 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1ad000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1a80d10c0 con 0x7fe1a80ced30 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe19fe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe1a80ce8f0 0x7fe1a80b2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47508/0 (socket says 172.21.15.146:47508) 2024-01-16T14:59:35.014 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe19fe00640 1 -- 172.21.15.146:0/2841607645 learned_addr learned my addr 172.21.15.146:0/2841607645 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1a6800640 1 --2- 172.21.15.146:0/2841607645 >> v2:172.21.15.53:3300/0 conn(0x7fe1a80ced30 0x7fe1a80b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685320067 0 0) 0x7fe1a80d0de0 con 0x7fe1a80ce8f0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.033+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe188003520 con 0x7fe1a80ce8f0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 162746356 0 0) 0x7fe188003520 con 0x7fe1a80ce8f0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 >> v2:172.21.15.53:3300/0 conn(0x7fe1a80ced30 msgr2=0x7fe1a80b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 --2- 172.21.15.146:0/2841607645 >> v2:172.21.15.53:3300/0 conn(0x7fe1a80ced30 0x7fe1a80b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 >> v1:172.21.15.53:6789/0 conn(0x7fe1a8160f80 legacy=0x7fe1a8161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80d13a0 con 0x7fe1a80ce8f0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe190001670 con 0x7fe1a80ce8f0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 >> v1:172.21.15.146:6789/0 conn(0x7fe1a80ce8f0 legacy=0x7fe1a80b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1880039c0 0x7fe188003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880047c0 0x7fe188004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe188005180 0x7fe18800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80d10c0 con 0x7fe1880047c0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18800c080 con 0x7fe1880039c0 2024-01-16T14:59:35.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe18800c240 con 0x7fe188005180 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1880039c0 0x7fe188003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.034+0000 7fe1a6800640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe188005180 0x7fe18800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe19fe00640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880047c0 0x7fe188004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe1a5e00640 1 -- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe188005180 msgr2=0x7fe18800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe188005180 0x7fe18800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe1a5e00640 1 -- 172.21.15.146:0/2841607645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880047c0 msgr2=0x7fe188004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1880047c0 0x7fe188004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe1a5e00640 1 -- 172.21.15.146:0/2841607645 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe188003520 con 0x7fe1880039c0 2024-01-16T14:59:35.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.035+0000 7fe1a6800640 1 -- 172.21.15.146:0/2841607645 reap_dead start 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.036+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1880039c0 0x7fe188003de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe1980015b0 tx=0x7fe1880057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c35f4dbebe584d75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.036+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980052d0 con 0x7fe1880039c0 2024-01-16T14:59:35.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.036+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19802e050 con 0x7fe1880039c0 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.036+0000 7fe1a5400640 1 -- 172.21.15.146:0/2841607645 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198003830 con 0x7fe1880039c0 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 -- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1880039c0 msgr2=0x7fe188003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1880039c0 0x7fe188003de0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe1980015b0 tx=0x7fe1880057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 -- 172.21.15.146:0/2841607645 shutdown_connections 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 --2- 172.21.15.146:0/2841607645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1880039c0 0x7fe188003de0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 -- 172.21.15.146:0/2841607645 >> 172.21.15.146:0/2841607645 conn(0x7fe1a809c550 msgr2=0x7fe1a80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 -- 172.21.15.146:0/2841607645 shutdown_connections 2024-01-16T14:59:35.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.037+0000 7fe1ad000640 1 -- 172.21.15.146:0/2841607645 wait complete. 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.039+0000 7fe1ad000640 1 Processor -- start 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.039+0000 7fe1ad000640 1 -- start start 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1ad000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 0x7fe1a80b8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1ad000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a815b610 0x7fe1a815ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1ad000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a81434f0 0x7fe1a8143970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1ad000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a815bfd0 con 0x7fe1a815b610 2024-01-16T14:59:35.021 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1ad000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80b79c0 con 0x7fe1a81434f0 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1ad000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a80b7bf0 con 0x7fe1880039c0 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1a5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 0x7fe1a80b8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1a6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a81434f0 0x7fe1a8143970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1a5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 0x7fe1a80b8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:39844/0 (socket says 172.21.15.146:39844) 2024-01-16T14:59:35.022 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe1a5e00640 1 -- 172.21.15.146:0/1804822699 learned_addr learned my addr 172.21.15.146:0/1804822699 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.040+0000 7fe19fe00640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a815b610 0x7fe1a815ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1a5e00640 1 -- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a81434f0 msgr2=0x7fe1a8143970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a81434f0 0x7fe1a8143970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1a5e00640 1 -- 172.21.15.146:0/1804822699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a815b610 msgr2=0x7fe1a815ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a815b610 0x7fe1a815ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1a5e00640 1 -- 172.21.15.146:0/1804822699 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a80b61d0 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe19fe00640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a815b610 0x7fe1a815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1a5e00640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 0x7fe1a80b8690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe19802ba70 tx=0x7fe198001120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1980076c0 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1980069e0 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe198006d50 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a80b7e90 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.041+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a80b8bd0 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.042+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe1980038f0 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.042+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fe19803d470 con 0x7fe1880039c0 2024-01-16T14:59:35.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.043+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a80675d0 con 0x7fe1880039c0 2024-01-16T14:59:35.028 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.047+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe198064ce0 con 0x7fe1880039c0 2024-01-16T14:59:35.032 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:35.051+0000 7f61be83b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:59:35.035 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:35.054+0000 7fed924fa040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-01-16T14:59:35.092 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:35.111+0000 7fed924fa040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-01-16T14:59:35.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.125+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2b673652-9d47-429f-aa72-16eca4b02335", "id": 2} v 0) v1 -- 0x7fe1a80d3920 con 0x7fe1880039c0 2024-01-16T14:59:35.113 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.132+0000 7fe19e000640 1 -- 172.21.15.146:0/1804822699 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2b673652-9d47-429f-aa72-16eca4b02335", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe198053030 con 0x7fe1880039c0 2024-01-16T14:59:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 msgr2=0x7fe1a80b8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 0x7fe1a80b8690 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe19802ba70 tx=0x7fe198001120 comp rx=0 tx=0).stop 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 shutdown_connections 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe1a81434f0 0x7fe1a8143970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe1a815b610 0x7fe1a815ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 --2- 172.21.15.146:0/1804822699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1880039c0 0x7fe1a80b8690 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 >> 172.21.15.146:0/1804822699 conn(0x7fe1a809c550 msgr2=0x7fe1a80cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 shutdown_connections 2024-01-16T14:59:35.115 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.133+0000 7fe1ad000640 1 -- 172.21.15.146:0/1804822699 wait complete. 2024-01-16T14:59:35.130 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 256f1c58-ed5e-4b01-9237-2b77f0bea192 3 2024-01-16T14:59:35.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.381+0000 7f8f3a800640 1 Processor -- start 2024-01-16T14:59:35.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.381+0000 7f8f3a800640 1 -- start start 2024-01-16T14:59:35.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f3a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f340ce8f0 0x7f8f340ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f3a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f340b2a00 0x7f8f340b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f3a800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f340d0de0 con 0x7f8f34160f80 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f3a800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340d10c0 con 0x7f8f340ce8f0 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f3a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f340d13a0 con 0x7f8f340b2a00 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f33e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f340b2a00 0x7f8f340b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f33e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f340b2a00 0x7f8f340b2e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59098/0 (socket says 172.21.15.146:59098) 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f33e00640 1 -- 172.21.15.146:0/164935654 learned_addr learned my addr 172.21.15.146:0/164935654 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.383+0000 7f8f2be00640 1 --2- 172.21.15.146:0/164935654 >> v2:172.21.15.53:3300/0 conn(0x7f8f340ce8f0 0x7f8f340ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.384+0000 7f8f33e00640 1 -- 172.21.15.146:0/164935654 >> v2:172.21.15.53:3300/0 conn(0x7f8f340ce8f0 msgr2=0x7f8f340ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.384+0000 7f8f33e00640 1 --2- 172.21.15.146:0/164935654 >> v2:172.21.15.53:3300/0 conn(0x7f8f340ce8f0 0x7f8f340ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.384+0000 7f8f33e00640 1 -- 172.21.15.146:0/164935654 >> v1:172.21.15.53:6789/0 conn(0x7f8f34160f80 legacy=0x7f8f34161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.384+0000 7f8f33e00640 1 -- 172.21.15.146:0/164935654 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f340d1680 con 0x7f8f340b2a00 2024-01-16T14:59:35.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.385+0000 7f8f33e00640 1 --2- 172.21.15.146:0/164935654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340b2a00 0x7f8f340b2e90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8f340b4cb0 tx=0x7f8f2c02bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa9329221c9905fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.385+0000 7f8f32a00640 1 -- 172.21.15.146:0/164935654 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c02ca80 con 0x7f8f340b2a00 2024-01-16T14:59:35.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.385+0000 7f8f32a00640 1 -- 172.21.15.146:0/164935654 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f2c02cbe0 con 0x7f8f340b2a00 2024-01-16T14:59:35.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.385+0000 7f8f32a00640 1 -- 172.21.15.146:0/164935654 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f2c0343f0 con 0x7f8f340b2a00 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 -- 172.21.15.146:0/164935654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340b2a00 msgr2=0x7f8f340b2e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 --2- 172.21.15.146:0/164935654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340b2a00 0x7f8f340b2e90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8f340b4cb0 tx=0x7f8f2c02bee0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 -- 172.21.15.146:0/164935654 shutdown_connections 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 --2- 172.21.15.146:0/164935654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340b2a00 0x7f8f340b2e90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 --2- 172.21.15.146:0/164935654 >> v2:172.21.15.53:3300/0 conn(0x7f8f340ce8f0 0x7f8f340ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 -- 172.21.15.146:0/164935654 >> 172.21.15.146:0/164935654 conn(0x7f8f3409c550 msgr2=0x7f8f340d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 -- 172.21.15.146:0/164935654 shutdown_connections 2024-01-16T14:59:35.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.386+0000 7f8f3a800640 1 -- 172.21.15.146:0/164935654 wait complete. 2024-01-16T14:59:35.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.388+0000 7f8f3a800640 1 Processor -- start 2024-01-16T14:59:35.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.388+0000 7f8f3a800640 1 -- start start 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f3a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f340b2a00 0x7f8f340b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f3a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 0x7f8f340b8a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f3a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 0x7f8f340b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f3a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b7c40 con 0x7f8f340b2a00 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f3a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b7f20 con 0x7f8f340ce8f0 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f3a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f340b8200 con 0x7f8f34160f80 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 0x7f8f340b8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f33e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 0x7f8f340b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f33e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 0x7f8f340b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:39860/0 (socket says 172.21.15.146:39860) 2024-01-16T14:59:35.370 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f33e00640 1 -- 172.21.15.146:0/1011266975 learned_addr learned my addr 172.21.15.146:0/1011266975 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:35.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 0x7f8f340b8a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59114/0 (socket says 172.21.15.146:59114) 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 -- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 msgr2=0x7f8f340b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 0x7f8f340b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 -- 172.21.15.146:0/1011266975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f340b2a00 msgr2=0x7f8f340b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f340b2a00 0x7f8f340b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f2be00640 1 -- 172.21.15.146:0/1011266975 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f2c009b30 con 0x7f8f340ce8f0 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.389+0000 7f8f33e00640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 0x7f8f340b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:35.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f2be00640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 0x7f8f340b8a00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00d5f0 tx=0x7f8f1c00dab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c0125f0 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f1c012750 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f340b84e0 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c012b00 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f340b8f40 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.390+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8f1c01e460 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.391+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f8f1c01eba0 con 0x7f8f340ce8f0 2024-01-16T14:59:35.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.391+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f340675d0 con 0x7f8f340ce8f0 2024-01-16T14:59:35.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.394+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f8f1c01e710 con 0x7f8f340ce8f0 2024-01-16T14:59:35.453 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.472+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "256f1c58-ed5e-4b01-9237-2b77f0bea192", "id": 3} v 0) v1 -- 0x7f8f340d3920 con 0x7f8f340ce8f0 2024-01-16T14:59:35.460 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.479+0000 7f8f2b400640 1 -- 172.21.15.146:0/1011266975 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "256f1c58-ed5e-4b01-9237-2b77f0bea192", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8f1c014480 con 0x7f8f340ce8f0 2024-01-16T14:59:35.460 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 msgr2=0x7f8f340b8a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 0x7f8f340b8a00 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c00d5f0 tx=0x7f8f1c00dab0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 shutdown_connections 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f34160f80 0x7f8f340b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f340ce8f0 0x7f8f340b8a00 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 --2- 172.21.15.146:0/1011266975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f340b2a00 0x7f8f340b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 >> 172.21.15.146:0/1011266975 conn(0x7f8f3409c550 msgr2=0x7f8f3415c8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 shutdown_connections 2024-01-16T14:59:35.462 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.481+0000 7f8f3a800640 1 -- 172.21.15.146:0/1011266975 wait complete. 2024-01-16T14:59:35.477 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 003cd61b-e9e3-4139-8f3f-1348822ea61f 4 2024-01-16T14:59:35.519 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:35.539+0000 7f61be83b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:59:35.563 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:35.582+0000 7fed924fa040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-01-16T14:59:35.667 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.686+0000 7fd7d5000640 1 Processor -- start 2024-01-16T14:59:35.668 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.686+0000 7fd7d5000640 1 -- start start 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7d5000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7d0160f80 0x7fd7d0161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7d5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d01618a0 con 0x7fd7d00ced50 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7d5000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7d00d0a00 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7d5000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7d00cf1e0 con 0x7fd7d0160f80 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7cde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7d0160f80 0x7fd7d0161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7cde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7d0160f80 0x7fd7d0161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59116/0 (socket says 172.21.15.146:59116) 2024-01-16T14:59:35.669 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7cde00640 1 -- 172.21.15.146:0/2841369476 learned_addr learned my addr 172.21.15.146:0/2841369476 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:35.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.688+0000 7fd7c7e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd7d00ce8f0 0x7fd7d00cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:47512/0 (socket says 172.21.15.146:47512) 2024-01-16T14:59:35.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.689+0000 7fd7cde00640 1 -- 172.21.15.146:0/2841369476 >> v1:172.21.15.146:6789/0 conn(0x7fd7d00ce8f0 legacy=0x7fd7d00cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.689+0000 7fd7cde00640 1 -- 172.21.15.146:0/2841369476 >> v1:172.21.15.53:6789/0 conn(0x7fd7d00ced50 legacy=0x7fd7d00d02f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:35.670 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.689+0000 7fd7cde00640 1 -- 172.21.15.146:0/2841369476 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7d00d0de0 con 0x7fd7d0160f80 2024-01-16T14:59:35.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.690+0000 7fd7cde00640 1 --2- 172.21.15.146:0/2841369476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d0160f80 0x7fd7d0161360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009870 tx=0x7fd7c002b990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=836bffbfc1193e34 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.690+0000 7fd7cd400640 1 -- 172.21.15.146:0/2841369476 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0030850 con 0x7fd7d0160f80 2024-01-16T14:59:35.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.690+0000 7fd7cd400640 1 -- 172.21.15.146:0/2841369476 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c00309b0 con 0x7fd7d0160f80 2024-01-16T14:59:35.671 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.690+0000 7fd7cd400640 1 -- 172.21.15.146:0/2841369476 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0030d40 con 0x7fd7d0160f80 2024-01-16T14:59:35.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 -- 172.21.15.146:0/2841369476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d0160f80 msgr2=0x7fd7d0161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 --2- 172.21.15.146:0/2841369476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d0160f80 0x7fd7d0161360 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd7c0009870 tx=0x7fd7c002b990 comp rx=0 tx=0).stop 2024-01-16T14:59:35.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 -- 172.21.15.146:0/2841369476 shutdown_connections 2024-01-16T14:59:35.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 --2- 172.21.15.146:0/2841369476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d0160f80 0x7fd7d0161360 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 -- 172.21.15.146:0/2841369476 >> 172.21.15.146:0/2841369476 conn(0x7fd7d009c550 msgr2=0x7fd7d00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.672 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 -- 172.21.15.146:0/2841369476 shutdown_connections 2024-01-16T14:59:35.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.691+0000 7fd7d5000640 1 -- 172.21.15.146:0/2841369476 wait complete. 2024-01-16T14:59:35.673 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.693+0000 7fd7d5000640 1 Processor -- start 2024-01-16T14:59:35.674 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.693+0000 7fd7d5000640 1 -- start start 2024-01-16T14:59:35.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7d5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 0x7fd7d01601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.675 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7d5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7d00ced50 0x7fd7d0156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7d5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7d0160f80 0x7fd7d015d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7d5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d00cd190 con 0x7fd7d00ced50 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7d5000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d00c93a0 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7d5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7d00c9680 con 0x7fd7d0160f80 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 0x7fd7d01601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 0x7fd7d01601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59118/0 (socket says 172.21.15.146:59118) 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 -- 172.21.15.146:0/737167054 learned_addr learned my addr 172.21.15.146:0/737167054 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7ce800640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7d0160f80 0x7fd7d015d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7c7e00640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7d00ced50 0x7fd7d0156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 -- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7d0160f80 msgr2=0x7fd7d015d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7d0160f80 0x7fd7d015d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 -- 172.21.15.146:0/737167054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7d00ced50 msgr2=0x7fd7d0156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7d00ced50 0x7fd7d0156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.676 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7cde00640 1 -- 172.21.15.146:0/737167054 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c00094d0 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.694+0000 7fd7c7e00640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7d00ced50 0x7fd7d0156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:35.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7cde00640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 0x7fd7d01601d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd7c002b9d0 tx=0x7fd7c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:35.677 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0030820 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c0044480 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c0044850 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7d00ca380 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7d00ca940 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.695+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd7c002e070 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.696+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fd7c003e070 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.678 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.696+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7d00675d0 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.679 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.699+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fd7c0049070 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.689 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:35.708+0000 7f61be83b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:59:35.730 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:35.749+0000 7fed924fa040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-01-16T14:59:35.758 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.777+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "003cd61b-e9e3-4139-8f3f-1348822ea61f", "id": 4} v 0) v1 -- 0x7fd7d00d3920 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.764 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.783+0000 7fd7c6000640 1 -- 172.21.15.146:0/737167054 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "003cd61b-e9e3-4139-8f3f-1348822ea61f", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd7c0034070 con 0x7fd7d00ce8f0 2024-01-16T14:59:35.764 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-01-16T14:59:35.765 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 msgr2=0x7fd7d01601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 0x7fd7d01601d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fd7c002b9d0 tx=0x7fd7c0004730 comp rx=0 tx=0).stop 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 shutdown_connections 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7d0160f80 0x7fd7d015d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7d00ced50 0x7fd7d0156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 --2- 172.21.15.146:0/737167054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7d00ce8f0 0x7fd7d01601d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.784+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 >> 172.21.15.146:0/737167054 conn(0x7fd7d009c550 msgr2=0x7fd7d00cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.785+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 shutdown_connections 2024-01-16T14:59:35.766 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:35.785+0000 7fd7d5000640 1 -- 172.21.15.146:0/737167054 wait complete. 2024-01-16T14:59:35.781 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 4a258471-1806-49be-91f6-98d1a0c01f95 5 2024-01-16T14:59:35.828 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:35.846+0000 7f61be83b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:59:35.879 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:35.899+0000 7fed924fa040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-01-16T14:59:35.883 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:35.902+0000 7f61be83b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:59:35.934 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:35.954+0000 7fed924fa040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-01-16T14:59:36.000 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:36.018+0000 7f61be83b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:36.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.031+0000 7f4e2d200640 1 Processor -- start 2024-01-16T14:59:36.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.031+0000 7f4e2d200640 1 -- start start 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.033+0000 7f4e2d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e28160a20 0x7f4e280cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.033+0000 7f4e2d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e280cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.033+0000 7f4e2d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4e280b3b20 0x7f4e280b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.033+0000 7f4e2d200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e280d3c40 con 0x7f4e280b3b20 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.033+0000 7f4e2d200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e280d1e00 con 0x7f4e28160a20 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.033+0000 7f4e2d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280d3e50 con 0x7f4e28160e00 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e28160a20 0x7f4e280cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.015 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e1e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e280cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e1e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e280cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:39876/0 (socket says 172.21.15.146:39876) 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e28160a20 0x7f4e280cf850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59124/0 (socket says 172.21.15.146:59124) 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e1e800640 1 -- 172.21.15.146:0/1347546654 learned_addr learned my addr 172.21.15.146:0/1347546654 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27e00640 1 --2- 172.21.15.146:0/1347546654 >> v2:172.21.15.53:3300/0 conn(0x7f4e280b3b20 0x7f4e280b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 -- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 msgr2=0x7f4e280cfd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 --2- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e280cfd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 -- 172.21.15.146:0/1347546654 >> v2:172.21.15.53:3300/0 conn(0x7f4e280b3b20 msgr2=0x7f4e280b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 --2- 172.21.15.146:0/1347546654 >> v2:172.21.15.53:3300/0 conn(0x7f4e280b3b20 0x7f4e280b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e27400640 1 -- 172.21.15.146:0/1347546654 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e280d1a90 con 0x7f4e28160a20 2024-01-16T14:59:36.016 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.034+0000 7f4e1e800640 1 --2- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e280cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:36.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.035+0000 7f4e27400640 1 --2- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e28160a20 0x7f4e280cf850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e2808d140 tx=0x7f4e1802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36ad3f38f56f2cdc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.036+0000 7f4e26a00640 1 -- 172.21.15.146:0/1347546654 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e1802e070 con 0x7f4e28160a20 2024-01-16T14:59:36.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.036+0000 7f4e26a00640 1 -- 172.21.15.146:0/1347546654 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e180315a0 con 0x7f4e28160a20 2024-01-16T14:59:36.017 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.036+0000 7f4e26a00640 1 -- 172.21.15.146:0/1347546654 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18031930 con 0x7f4e28160a20 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.036+0000 7f4e2d200640 1 -- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e28160a20 msgr2=0x7f4e280cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.036+0000 7f4e2d200640 1 --2- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e28160a20 0x7f4e280cf850 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4e2808d140 tx=0x7f4e1802b840 comp rx=0 tx=0).stop 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 -- 172.21.15.146:0/1347546654 shutdown_connections 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 --2- 172.21.15.146:0/1347546654 >> v2:172.21.15.53:3300/0 conn(0x7f4e280b3b20 0x7f4e280b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 --2- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e280cfd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 --2- 172.21.15.146:0/1347546654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e28160a20 0x7f4e280cf850 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 -- 172.21.15.146:0/1347546654 >> 172.21.15.146:0/1347546654 conn(0x7f4e280d3120 msgr2=0x7f4e280d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.018 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 -- 172.21.15.146:0/1347546654 shutdown_connections 2024-01-16T14:59:36.019 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.037+0000 7f4e2d200640 1 -- 172.21.15.146:0/1347546654 wait complete. 2024-01-16T14:59:36.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.039+0000 7f4e2d200640 1 Processor -- start 2024-01-16T14:59:36.020 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.040+0000 7f4e2d200640 1 -- start start 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e2d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 0x7f4e280b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e2d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e28160a20 0x7f4e280b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e2d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e2815c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e2d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280b8790 con 0x7f4e28160a20 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e2d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280b6c60 con 0x7f4e280b3b20 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e2d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e280b7b80 con 0x7f4e28160e00 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e27400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 0x7f4e280b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e27400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 0x7f4e280b75a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59126/0 (socket says 172.21.15.146:59126) 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e27400640 1 -- 172.21.15.146:0/337795575 learned_addr learned my addr 172.21.15.146:0/337795575 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:36.023 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.041+0000 7f4e27e00640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e2815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e1e800640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e28160a20 0x7f4e280b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e27400640 1 -- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 msgr2=0x7f4e2815c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e27400640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e2815c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e27400640 1 -- 172.21.15.146:0/337795575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e28160a20 msgr2=0x7f4e280b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e27400640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e28160a20 0x7f4e280b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e27400640 1 -- 172.21.15.146:0/337795575 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e100094d0 con 0x7f4e280b3b20 2024-01-16T14:59:36.024 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e27400640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 0x7f4e280b75a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009890 tx=0x7f4e180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18032840 con 0x7f4e280b3b20 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e18031f00 con 0x7f4e280b3b20 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e18032be0 con 0x7f4e280b3b20 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.042+0000 7f4e2d200640 1 -- 172.21.15.146:0/337795575 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e18009400 con 0x7f4e280b3b20 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.043+0000 7f4e2d200640 1 -- 172.21.15.146:0/337795575 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e280b8140 con 0x7f4e280b3b20 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.043+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4e18005c90 con 0x7f4e280b3b20 2024-01-16T14:59:36.025 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.043+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f4e18032d40 con 0x7f4e280b3b20 2024-01-16T14:59:36.026 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.044+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df0005250 con 0x7f4e280b3b20 2024-01-16T14:59:36.027 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.046+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f4e18006750 con 0x7f4e280b3b20 2024-01-16T14:59:36.049 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:36.069+0000 7fed924fa040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:36.106 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.124+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4a258471-1806-49be-91f6-98d1a0c01f95", "id": 5} v 0) v1 -- 0x7f4df00055a0 con 0x7f4e280b3b20 2024-01-16T14:59:36.112 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.131+0000 7f4e1fe00640 1 -- 172.21.15.146:0/337795575 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a258471-1806-49be-91f6-98d1a0c01f95", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4e18005f40 con 0x7f4e280b3b20 2024-01-16T14:59:36.112 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 msgr2=0x7f4e280b75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 0x7f4e280b75a0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4e18009890 tx=0x7f4e180046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 shutdown_connections 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e28160e00 0x7f4e2815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e28160a20 0x7f4e280b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 --2- 172.21.15.146:0/337795575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e280b3b20 0x7f4e280b75a0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.132+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 >> 172.21.15.146:0/337795575 conn(0x7f4e280d3120 msgr2=0x7f4e2815f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.133+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 shutdown_connections 2024-01-16T14:59:36.114 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.133+0000 7f4e1ca00640 1 -- 172.21.15.146:0/337795575 wait complete. 2024-01-16T14:59:36.123 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:36.143+0000 7f61be83b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:59:36.131 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new fabb3d13-1a7f-4e95-8058-1ccc16400482 6 2024-01-16T14:59:36.165 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:36.185+0000 7fed924fa040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-01-16T14:59:36.293 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:36.311+0000 7f61be83b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:36.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.337+0000 7fea65c00640 1 Processor -- start 2024-01-16T14:59:36.318 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.337+0000 7fea65c00640 1 -- start start 2024-01-16T14:59:36.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea65c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea60160a20 0x7fea600cf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea65c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea600d1e00 con 0x7fea60160e00 2024-01-16T14:59:36.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea65c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea600b38a0 con 0x7fea600b3b20 2024-01-16T14:59:36.320 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea65c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea600d3c40 con 0x7fea60160a20 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea5f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fea60160e00 0x7fea600b2a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.146:50654/0 (socket says 172.21.15.146:50654) 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea5f400640 1 -- 172.21.15.146:0/584460770 learned_addr learned my addr 172.21.15.146:0/584460770 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.339+0000 7fea5fe00640 1 --2- 172.21.15.146:0/584460770 >> v2:172.21.15.146:3300/0 conn(0x7fea60160a20 0x7fea600cf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.340+0000 7fea5fe00640 1 -- 172.21.15.146:0/584460770 >> v1:172.21.15.146:6789/0 conn(0x7fea600b3b20 legacy=0x7fea600b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.340+0000 7fea5ea00640 1 -- 172.21.15.146:0/584460770 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750100269 0 0) 0x7fea600d1e00 con 0x7fea60160e00 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.340+0000 7fea5fe00640 1 -- 172.21.15.146:0/584460770 >> v1:172.21.15.53:6789/0 conn(0x7fea60160e00 legacy=0x7fea600b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.321 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.340+0000 7fea5fe00640 1 -- 172.21.15.146:0/584460770 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea600d3e50 con 0x7fea60160a20 2024-01-16T14:59:36.322 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.341+0000 7fea5fe00640 1 --2- 172.21.15.146:0/584460770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea60160a20 0x7fea600cf850 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea50009720 tx=0x7fea5002b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=738d7e884ea82bbb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.342+0000 7fea5ea00640 1 -- 172.21.15.146:0/584460770 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50030710 con 0x7fea60160a20 2024-01-16T14:59:36.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.342+0000 7fea5ea00640 1 -- 172.21.15.146:0/584460770 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5002e050 con 0x7fea60160a20 2024-01-16T14:59:36.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.342+0000 7fea5ea00640 1 -- 172.21.15.146:0/584460770 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea50030a90 con 0x7fea60160a20 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.342+0000 7fea65c00640 1 -- 172.21.15.146:0/584460770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea60160a20 msgr2=0x7fea600cf850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.342+0000 7fea65c00640 1 --2- 172.21.15.146:0/584460770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea60160a20 0x7fea600cf850 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fea50009720 tx=0x7fea5002b730 comp rx=0 tx=0).stop 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.343+0000 7fea65c00640 1 -- 172.21.15.146:0/584460770 shutdown_connections 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.343+0000 7fea65c00640 1 --2- 172.21.15.146:0/584460770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea60160a20 0x7fea600cf850 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.343+0000 7fea65c00640 1 -- 172.21.15.146:0/584460770 >> 172.21.15.146:0/584460770 conn(0x7fea600d3120 msgr2=0x7fea600d3530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.343+0000 7fea65c00640 1 -- 172.21.15.146:0/584460770 shutdown_connections 2024-01-16T14:59:36.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.343+0000 7fea65c00640 1 -- 172.21.15.146:0/584460770 wait complete. 2024-01-16T14:59:36.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.344+0000 7fea65c00640 1 Processor -- start 2024-01-16T14:59:36.325 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.344+0000 7fea65c00640 1 -- start start 2024-01-16T14:59:36.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.345+0000 7fea65c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 0x7fea600cb850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.345+0000 7fea65c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea60160a20 0x7fea601f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.345+0000 7fea65c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea60160e00 0x7fea601f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.345+0000 7fea65c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea600cb430 con 0x7fea60160e00 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.345+0000 7fea65c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea600cbd90 con 0x7fea600b3b20 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.345+0000 7fea65c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea600ca7b0 con 0x7fea60160a20 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 0x7fea600cb850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 0x7fea600cb850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59146/0 (socket says 172.21.15.146:59146) 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 -- 172.21.15.146:0/3967975759 learned_addr learned my addr 172.21.15.146:0/3967975759 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5f400640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea60160a20 0x7fea601f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea65200640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea60160e00 0x7fea601f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 -- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea60160a20 msgr2=0x7fea601f37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea60160a20 0x7fea601f37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 -- 172.21.15.146:0/3967975759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea60160e00 msgr2=0x7fea601f6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.328 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea60160e00 0x7fea601f6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea5fe00640 1 -- 172.21.15.146:0/3967975759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea50009400 con 0x7fea600b3b20 2024-01-16T14:59:36.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.346+0000 7fea65200640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea60160e00 0x7fea601f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:36.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea5fe00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 0x7fea600cb850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fea50000c00 tx=0x7fea500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea500329d0 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea50032b50 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5003c470 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea600caa90 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea600cb050 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.347+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fea50039040 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.348+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fea50048590 con 0x7fea600b3b20 2024-01-16T14:59:36.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.348+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea600cb570 con 0x7fea600b3b20 2024-01-16T14:59:36.332 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:36.351+0000 7fed924fa040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-01-16T14:59:36.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.351+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fea600cb570 con 0x7fea600b3b20 2024-01-16T14:59:36.347 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:36.366+0000 7f61be83b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:36.383 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:36.403+0000 7fed924fa040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-01-16T14:59:36.410 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.429+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fabb3d13-1a7f-4e95-8058-1ccc16400482", "id": 6} v 0) v1 -- 0x7fea60067b60 con 0x7fea600b3b20 2024-01-16T14:59:36.417 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.436+0000 7fea57e00640 1 -- 172.21.15.146:0/3967975759 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fabb3d13-1a7f-4e95-8058-1ccc16400482", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fea50043020 con 0x7fea600b3b20 2024-01-16T14:59:36.417 INFO:teuthology.orchestra.run.smithi146.stdout:6 2024-01-16T14:59:36.418 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 msgr2=0x7fea600cb850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 0x7fea600cb850 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fea50000c00 tx=0x7fea500046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 shutdown_connections 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea60160e00 0x7fea601f6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea60160a20 0x7fea601f37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 --2- 172.21.15.146:0/3967975759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea600b3b20 0x7fea600cb850 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.437+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 >> 172.21.15.146:0/3967975759 conn(0x7fea600d3120 msgr2=0x7fea6006c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.438+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 shutdown_connections 2024-01-16T14:59:36.419 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.438+0000 7fea65c00640 1 -- 172.21.15.146:0/3967975759 wait complete. 2024-01-16T14:59:36.435 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5 7 2024-01-16T14:59:36.518 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:36.536+0000 7f61be83b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:59:36.552 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:36.572+0000 7fed924fa040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-01-16T14:59:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.630+0000 7f27f3800640 1 Processor -- start 2024-01-16T14:59:36.611 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.630+0000 7f27f3800640 1 -- start start 2024-01-16T14:59:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f3800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27ec143ae0 0x7f27ec0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f3800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 0x7f27ec15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f3800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27ec15e1e0 0x7f27ec15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.613 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f3800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ec0cfc30 con 0x7f27ec143ae0 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f3800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27ec0b1ee0 con 0x7f27ec15e1e0 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f3800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec0b2780 con 0x7f27ec0cf7d0 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 0x7f27ec15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27ec15e1e0 0x7f27ec15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 0x7f27ec15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.146:39894/0 (socket says 172.21.15.146:39894) 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 -- 172.21.15.146:0/4172724739 learned_addr learned my addr 172.21.15.146:0/4172724739 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27f0e00640 1 --2- 172.21.15.146:0/4172724739 >> v2:172.21.15.53:3300/0 conn(0x7f27ec143ae0 0x7f27ec0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 -- 172.21.15.146:0/4172724739 >> v2:172.21.15.146:3300/0 conn(0x7f27ec15e1e0 msgr2=0x7f27ec15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 --2- 172.21.15.146:0/4172724739 >> v2:172.21.15.146:3300/0 conn(0x7f27ec15e1e0 0x7f27ec15a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 -- 172.21.15.146:0/4172724739 >> v2:172.21.15.53:3300/0 conn(0x7f27ec143ae0 msgr2=0x7f27ec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 --2- 172.21.15.146:0/4172724739 >> v2:172.21.15.53:3300/0 conn(0x7f27ec143ae0 0x7f27ec0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.614 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.632+0000 7f27e3e00640 1 -- 172.21.15.146:0/4172724739 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec15e990 con 0x7f27ec0cf7d0 2024-01-16T14:59:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.633+0000 7f27e3e00640 1 --2- 172.21.15.146:0/4172724739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 0x7f27ec15dca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27ec072160 tx=0x7f27d402e0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4636b91c4c8c24b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27ebe00640 1 -- 172.21.15.146:0/4172724739 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d402ee90 con 0x7f27ec0cf7d0 2024-01-16T14:59:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27ebe00640 1 -- 172.21.15.146:0/4172724739 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d4036440 con 0x7f27ec0cf7d0 2024-01-16T14:59:36.615 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27ebe00640 1 -- 172.21.15.146:0/4172724739 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d40367d0 con 0x7f27ec0cf7d0 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 -- 172.21.15.146:0/4172724739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 msgr2=0x7f27ec15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 --2- 172.21.15.146:0/4172724739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 0x7f27ec15dca0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f27ec072160 tx=0x7f27d402e0e0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 -- 172.21.15.146:0/4172724739 shutdown_connections 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 --2- 172.21.15.146:0/4172724739 >> v2:172.21.15.146:3300/0 conn(0x7f27ec15e1e0 0x7f27ec15a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 --2- 172.21.15.146:0/4172724739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec0cf7d0 0x7f27ec15dca0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 --2- 172.21.15.146:0/4172724739 >> v2:172.21.15.53:3300/0 conn(0x7f27ec143ae0 0x7f27ec0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.634+0000 7f27f3800640 1 -- 172.21.15.146:0/4172724739 >> 172.21.15.146:0/4172724739 conn(0x7f27ec06a800 msgr2=0x7f27ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.616 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.635+0000 7f27f3800640 1 -- 172.21.15.146:0/4172724739 shutdown_connections 2024-01-16T14:59:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.635+0000 7f27f3800640 1 -- 172.21.15.146:0/4172724739 wait complete. 2024-01-16T14:59:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.636+0000 7f27f3800640 1 Processor -- start 2024-01-16T14:59:36.617 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.636+0000 7f27f3800640 1 -- start start 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.637+0000 7f27f3800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27ec0cf7d0 0x7f27ec06c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.637+0000 7f27f3800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 0x7f27ec06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.637+0000 7f27f3800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec15e1e0 0x7f27ec1363e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.637+0000 7f27f3800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec1373b0 con 0x7f27ec0cf7d0 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.637+0000 7f27f3800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec1370d0 con 0x7f27ec143ae0 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.637+0000 7f27f3800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27ec1344d0 con 0x7f27ec15e1e0 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 0x7f27ec06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 0x7f27ec06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.146:59150/0 (socket says 172.21.15.146:59150) 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 -- 172.21.15.146:0/1141650390 learned_addr learned my addr 172.21.15.146:0/1141650390 (peer_addr_for_me v2:172.21.15.146:0/0) 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27f1800640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec15e1e0 0x7f27ec1363e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27f0e00640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27ec0cf7d0 0x7f27ec06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:36.620 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 -- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec15e1e0 msgr2=0x7f27ec1363e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec15e1e0 0x7f27ec1363e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 -- 172.21.15.146:0/1141650390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27ec0cf7d0 msgr2=0x7f27ec06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27ec0cf7d0 0x7f27ec06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 -- 172.21.15.146:0/1141650390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d400b770 con 0x7f27ec143ae0 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27f0e00640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27ec0cf7d0 0x7f27ec06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.638+0000 7f27e3e00640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 0x7f27ec06c9b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f27d402e120 tx=0x7f27d40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d40347d0 con 0x7f27ec143ae0 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27d4034930 con 0x7f27ec143ae0 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27d4034ce0 con 0x7f27ec143ae0 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27ec1347b0 con 0x7f27ec143ae0 2024-01-16T14:59:36.621 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27ec13a350 con 0x7f27ec143ae0 2024-01-16T14:59:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f27d4041740 con 0x7f27ec143ae0 2024-01-16T14:59:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.639+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f27d4034e40 con 0x7f27ec143ae0 2024-01-16T14:59:36.622 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.640+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27ec134cd0 con 0x7f27ec143ae0 2024-01-16T14:59:36.623 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.642+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f27ec134cd0 con 0x7f27ec143ae0 2024-01-16T14:59:36.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.720+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5", "id": 7} v 0) v1 -- 0x7f27ec0675d0 con 0x7f27ec143ae0 2024-01-16T14:59:36.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.729+0000 7f27e9600640 1 -- 172.21.15.146:0/1141650390 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f27d4046070 con 0x7f27ec143ae0 2024-01-16T14:59:36.710 INFO:teuthology.orchestra.run.smithi146.stdout:7 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.730+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 msgr2=0x7f27ec06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.730+0000 7f27f3800640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 0x7f27ec06c9b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f27d402e120 tx=0x7f27d40046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.730+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 shutdown_connections 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.730+0000 7f27f3800640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27ec15e1e0 0x7f27ec1363e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.730+0000 7f27f3800640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27ec143ae0 0x7f27ec06c9b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.730+0000 7f27f3800640 1 --2- 172.21.15.146:0/1141650390 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27ec0cf7d0 0x7f27ec06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.731+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 >> 172.21.15.146:0/1141650390 conn(0x7f27ec06a800 msgr2=0x7f27ec15a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.731+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 shutdown_connections 2024-01-16T14:59:36.712 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T14:59:36.731+0000 7f27f3800640 1 -- 172.21.15.146:0/1141650390 wait complete. 2024-01-16T14:59:36.727 INFO:tasks.ceph.osd.0:Restarting daemon 2024-01-16T14:59:36.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-01-16T14:59:36.735 INFO:tasks.ceph.osd.0:Started 2024-01-16T14:59:36.735 INFO:tasks.ceph.osd.1:Restarting daemon 2024-01-16T14:59:36.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-01-16T14:59:36.742 INFO:tasks.ceph.osd.1:Started 2024-01-16T14:59:36.742 INFO:tasks.ceph.osd.2:Restarting daemon 2024-01-16T14:59:36.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-01-16T14:59:36.748 INFO:tasks.ceph.osd.2:Started 2024-01-16T14:59:36.748 INFO:tasks.ceph.osd.3:Restarting daemon 2024-01-16T14:59:36.748 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-01-16T14:59:36.753 INFO:tasks.ceph.osd.3:Started 2024-01-16T14:59:36.753 INFO:tasks.ceph.osd.4:Restarting daemon 2024-01-16T14:59:36.753 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-01-16T14:59:36.760 INFO:tasks.ceph.osd.4:Started 2024-01-16T14:59:36.760 INFO:tasks.ceph.osd.5:Restarting daemon 2024-01-16T14:59:36.760 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-01-16T14:59:36.767 INFO:tasks.ceph.osd.5:Started 2024-01-16T14:59:36.767 INFO:tasks.ceph.osd.6:Restarting daemon 2024-01-16T14:59:36.767 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-01-16T14:59:36.772 INFO:tasks.ceph.osd.6:Started 2024-01-16T14:59:36.772 INFO:tasks.ceph.osd.7:Restarting daemon 2024-01-16T14:59:36.772 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-01-16T14:59:36.777 INFO:tasks.ceph.osd.7:Started 2024-01-16T14:59:36.778 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.023+0000 7f0bfb400640 1 Processor -- start 2024-01-16T14:59:37.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.024+0000 7f0bfb400640 1 -- start start 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bfb400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 0x7f0bf40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bfb400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0bf415f9c0 0x7f0bf40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bfb400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bf415c5f0 con 0x7f0bf413df80 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bfb400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bf406c470 con 0x7f0bf415f9c0 2024-01-16T14:59:37.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bfb400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf406c750 con 0x7f0bf413e360 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 0x7f0bf40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 0x7f0bf40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:54994/0 (socket says 172.21.15.53:54994) 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bebe00640 1 -- 172.21.15.53:0/2429766699 learned_addr learned my addr 172.21.15.53:0/2429766699 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.025+0000 7f0bf9400640 1 --2- 172.21.15.53:0/2429766699 >> v2:172.21.15.146:3300/0 conn(0x7f0bf415f9c0 0x7f0bf40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.026+0000 7f0bebe00640 1 -- 172.21.15.53:0/2429766699 >> v2:172.21.15.146:3300/0 conn(0x7f0bf415f9c0 msgr2=0x7f0bf40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.026+0000 7f0bebe00640 1 --2- 172.21.15.53:0/2429766699 >> v2:172.21.15.146:3300/0 conn(0x7f0bf415f9c0 0x7f0bf40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.026+0000 7f0bebe00640 1 -- 172.21.15.53:0/2429766699 >> v1:172.21.15.146:6789/0 conn(0x7f0bf413df80 legacy=0x7f0bf415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.026+0000 7f0bebe00640 1 -- 172.21.15.53:0/2429766699 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf406ca30 con 0x7f0bf413e360 2024-01-16T14:59:37.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.027+0000 7f0bebe00640 1 --2- 172.21.15.53:0/2429766699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 0x7f0bf40cf290 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009870 tx=0x7f0bdc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac7f4832febfb5df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.027+0000 7f0bf3e00640 1 -- 172.21.15.53:0/2429766699 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc030840 con 0x7f0bf413e360 2024-01-16T14:59:37.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.027+0000 7f0bf3e00640 1 -- 172.21.15.53:0/2429766699 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bdc0309a0 con 0x7f0bf413e360 2024-01-16T14:59:37.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.027+0000 7f0bf3e00640 1 -- 172.21.15.53:0/2429766699 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bdc030d30 con 0x7f0bf413e360 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.027+0000 7f0bfb400640 1 -- 172.21.15.53:0/2429766699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 msgr2=0x7f0bf40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.027+0000 7f0bfb400640 1 --2- 172.21.15.53:0/2429766699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 0x7f0bf40cf290 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0bdc009870 tx=0x7f0bdc02b980 comp rx=0 tx=0).stop 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.028+0000 7f0bfb400640 1 -- 172.21.15.53:0/2429766699 shutdown_connections 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.028+0000 7f0bfb400640 1 --2- 172.21.15.53:0/2429766699 >> v2:172.21.15.146:3300/0 conn(0x7f0bf415f9c0 0x7f0bf40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.028+0000 7f0bfb400640 1 --2- 172.21.15.53:0/2429766699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413e360 0x7f0bf40cf290 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.028+0000 7f0bfb400640 1 -- 172.21.15.53:0/2429766699 >> 172.21.15.53:0/2429766699 conn(0x7f0bf406a800 msgr2=0x7f0bf406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.028+0000 7f0bfb400640 1 -- 172.21.15.53:0/2429766699 shutdown_connections 2024-01-16T14:59:37.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.028+0000 7f0bfb400640 1 -- 172.21.15.53:0/2429766699 wait complete. 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.029+0000 7f0bfb400640 1 Processor -- start 2024-01-16T14:59:37.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.029+0000 7f0bfb400640 1 -- start start 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.030+0000 7f0bfb400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413df80 0x7f0bf413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.030+0000 7f0bfb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf413e360 0x7f0bf40adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.030+0000 7f0bfb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 0x7f0bf40b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.030+0000 7f0bfb400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40a89c0 con 0x7f0bf415f9c0 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.030+0000 7f0bfb400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40a8ca0 con 0x7f0bf413e360 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.030+0000 7f0bfb400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bf40a8f80 con 0x7f0bf413df80 2024-01-16T14:59:37.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 0x7f0bf40b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bebe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf413e360 0x7f0bf40adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413df80 0x7f0bf413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:37.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 0x7f0bf40b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49506/0 (socket says 172.21.15.53:49506) 2024-01-16T14:59:37.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 -- 172.21.15.53:0/1934032584 learned_addr learned my addr 172.21.15.53:0/1934032584 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:37.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 -- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413df80 msgr2=0x7f0bf413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413df80 0x7f0bf413d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 -- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf413e360 msgr2=0x7f0bf40adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf413e360 0x7f0bf40adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 -- 172.21.15.53:0/1934032584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0094d0 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bebe00640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf413e360 0x7f0bf40adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.031+0000 7f0bf9400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 0x7f0bf40b2cf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00efc0 tx=0x7f0bec00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec014480 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bec00f040 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bec014820 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bf40a9260 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bf413dcf0 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0bec014c80 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.032+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f0bec0133e0 con 0x7f0bf415f9c0 2024-01-16T14:59:37.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.033+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bf40a9780 con 0x7f0bf415f9c0 2024-01-16T14:59:37.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.036+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7f0bf40a9780 con 0x7f0bf415f9c0 2024-01-16T14:59:37.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.100+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bf40675d0 con 0x7f0bf415f9c0 2024-01-16T14:59:37.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.101+0000 7f0bf1600640 1 -- 172.21.15.53:0/1934032584 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f0bec018020 con 0x7f0bf415f9c0 2024-01-16T14:59:37.081 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:37.081 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":10,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:36.723268+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 msgr2=0x7f0bf40b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 0x7f0bf40b2cf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0bec00efc0 tx=0x7f0bec00c490 comp rx=0 tx=0).stop 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 shutdown_connections 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bf415f9c0 0x7f0bf40b2cf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bf413e360 0x7f0bf40adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 --2- 172.21.15.53:0/1934032584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bf413df80 0x7f0bf413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 >> 172.21.15.53:0/1934032584 conn(0x7f0bf406a800 msgr2=0x7f0bf40707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 shutdown_connections 2024-01-16T14:59:37.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:37.102+0000 7f0bfb400640 1 -- 172.21.15.53:0/1934032584 wait complete. 2024-01-16T14:59:37.097 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-01-16T14:59:37.098 INFO:tasks.ceph:Waiting for OSDs to come up 2024-01-16T14:59:37.434 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-01-16T14:59:37.454+0000 7f74bd238500 -1 Falling back to public interface 2024-01-16T14:59:37.447 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-01-16T14:59:37.467+0000 7f22f228a500 -1 Falling back to public interface 2024-01-16T14:59:37.457 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-01-16T14:59:37.477+0000 7f0f2f80e500 -1 Falling back to public interface 2024-01-16T14:59:37.465 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-01-16T14:59:37.485+0000 7f036a421500 -1 Falling back to public interface 2024-01-16T14:59:37.473 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-01-16T14:59:37.492+0000 7fb333012500 -1 Falling back to public interface 2024-01-16T14:59:37.474 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-01-16T14:59:37.492+0000 7fef79849500 -1 Falling back to public interface 2024-01-16T14:59:37.491 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-01-16T14:59:37.510+0000 7f0345abe500 -1 Falling back to public interface 2024-01-16T14:59:37.493 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-01-16T14:59:37.513+0000 7f4973420500 -1 Falling back to public interface 2024-01-16T14:59:37.631 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:37.650+0000 7f61be83b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:59:37.714 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:37.733+0000 7fed924fa040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-01-16T14:59:37.903 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:37.938 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:37.957+0000 7f61be83b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:59:38.001 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:38.019+0000 7f61be83b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:59:38.036 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:38.055+0000 7fed924fa040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-01-16T14:59:38.070 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.089+0000 7fe477a00640 1 Processor -- start 2024-01-16T14:59:38.070 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.090+0000 7fe477a00640 1 -- start start 2024-01-16T14:59:38.071 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe477a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe47015d550 0x7fe4700d11d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.072 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe477a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4700ce0a0 con 0x7fe4700d38e0 2024-01-16T14:59:38.072 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe477a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4700ce340 con 0x7fe4700b2640 2024-01-16T14:59:38.072 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe477a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4700ce620 con 0x7fe47015d550 2024-01-16T14:59:38.072 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe475a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe4700d38e0 0x7fe4700b2fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36718/0 (socket says 172.21.15.53:36718) 2024-01-16T14:59:38.072 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe475a00640 1 -- 172.21.15.53:0/2688711380 learned_addr learned my addr 172.21.15.53:0/2688711380 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:38.072 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.091+0000 7fe475000640 1 --2- 172.21.15.53:0/2688711380 >> v2:172.21.15.146:3300/0 conn(0x7fe47015d550 0x7fe4700d11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.073 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.092+0000 7fe46fe00640 1 -- 172.21.15.53:0/2688711380 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4239372792 0 0) 0x7fe4700ce0a0 con 0x7fe4700d38e0 2024-01-16T14:59:38.073 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.092+0000 7fe46fe00640 1 -- 172.21.15.53:0/2688711380 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe450003520 con 0x7fe4700d38e0 2024-01-16T14:59:38.073 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.092+0000 7fe475000640 1 -- 172.21.15.53:0/2688711380 >> v1:172.21.15.146:6789/0 conn(0x7fe4700b2640 legacy=0x7fe4700d31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.073 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.092+0000 7fe46fe00640 1 -- 172.21.15.53:0/2688711380 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612866904 0 0) 0x7fe4700ce340 con 0x7fe4700b2640 2024-01-16T14:59:38.073 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.092+0000 7fe475000640 1 -- 172.21.15.53:0/2688711380 >> v1:172.21.15.53:6789/0 conn(0x7fe4700d38e0 legacy=0x7fe4700b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.073 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.092+0000 7fe475000640 1 -- 172.21.15.53:0/2688711380 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700ce900 con 0x7fe47015d550 2024-01-16T14:59:38.074 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.093+0000 7fe475000640 1 --2- 172.21.15.53:0/2688711380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe47015d550 0x7fe4700d11d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe4600097a0 tx=0x7fe46002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1923c59c92a956a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.074 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.093+0000 7fe46fe00640 1 -- 172.21.15.53:0/2688711380 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460030610 con 0x7fe47015d550 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.093+0000 7fe46fe00640 1 -- 172.21.15.53:0/2688711380 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460030770 con 0x7fe47015d550 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.093+0000 7fe46fe00640 1 -- 172.21.15.53:0/2688711380 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe460030b00 con 0x7fe47015d550 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 -- 172.21.15.53:0/2688711380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe47015d550 msgr2=0x7fe4700d11d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 --2- 172.21.15.53:0/2688711380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe47015d550 0x7fe4700d11d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe4600097a0 tx=0x7fe46002b960 comp rx=0 tx=0).stop 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 -- 172.21.15.53:0/2688711380 shutdown_connections 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 --2- 172.21.15.53:0/2688711380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe47015d550 0x7fe4700d11d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 -- 172.21.15.53:0/2688711380 >> 172.21.15.53:0/2688711380 conn(0x7fe47008cc50 msgr2=0x7fe47009c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 -- 172.21.15.53:0/2688711380 shutdown_connections 2024-01-16T14:59:38.075 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.094+0000 7fe477a00640 1 -- 172.21.15.53:0/2688711380 wait complete. 2024-01-16T14:59:38.076 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.096+0000 7fe477a00640 1 Processor -- start 2024-01-16T14:59:38.076 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.096+0000 7fe477a00640 1 -- start start 2024-01-16T14:59:38.077 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe477a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 0x7fe470159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe477a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4700d38e0 0x7fe47014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe477a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 0x7fe47015f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe477a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47014f210 con 0x7fe47015d550 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe477a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4700b6270 con 0x7fe4700d38e0 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe477a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4700b66f0 con 0x7fe4700b2640 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe475a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 0x7fe47015f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe475000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 0x7fe470159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe475000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 0x7fe470159d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:55002/0 (socket says 172.21.15.53:55002) 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe475000640 1 -- 172.21.15.53:0/2759763797 learned_addr learned my addr 172.21.15.53:0/2759763797 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe475a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 0x7fe47015f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49520/0 (socket says 172.21.15.53:49520) 2024-01-16T14:59:38.078 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.097+0000 7fe467c00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4700d38e0 0x7fe47014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:38.079 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475000640 1 -- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4700d38e0 msgr2=0x7fe47014d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475000640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4700d38e0 0x7fe47014d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475000640 1 -- 172.21.15.53:0/2759763797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 msgr2=0x7fe47015f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475000640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 0x7fe47015f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475000640 1 -- 172.21.15.53:0/2759763797 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe460009400 con 0x7fe4700b2640 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe467c00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4700d38e0 0x7fe47014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475a00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 0x7fe47015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe475000640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 0x7fe470159d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe4600098c0 tx=0x7fe4600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46003e070 con 0x7fe4700b2640 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe460030e30 con 0x7fe4700b2640 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4600399a0 con 0x7fe4700b2640 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4700b69d0 con 0x7fe4700b2640 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.098+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4701607e0 con 0x7fe4700b2640 2024-01-16T14:59:38.080 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.099+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe460042070 con 0x7fe4700b2640 2024-01-16T14:59:38.081 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.099+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7fe460039e00 con 0x7fe4700b2640 2024-01-16T14:59:38.081 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.099+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4700b5b30 con 0x7fe4700b2640 2024-01-16T14:59:38.083 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.103+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72898 (secure 0 0 0) 0x7fe460049070 con 0x7fe4700b2640 2024-01-16T14:59:38.100 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:38.120+0000 7fed924fa040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-01-16T14:59:38.120 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:38.139+0000 7f61be83b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:59:38.136 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.156+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mgrmap(e 2) v1 ==== 44721+0+0 (secure 0 0 0) 0x7fe460044420 con 0x7fe4700b2640 2024-01-16T14:59:38.154 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.173+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4700675d0 con 0x7fe4700b2640 2024-01-16T14:59:38.155 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.174+0000 7fe46d600640 1 -- 172.21.15.53:0/2759763797 <== mon.2 v2:172.21.15.146:3301/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7fe460034020 con 0x7fe4700b2640 2024-01-16T14:59:38.155 INFO:teuthology.misc.health.smithi053.stdout: 2024-01-16T14:59:38.155 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":10,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:36.723268+0000","last_up_change":"0.000000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 msgr2=0x7fe470159d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 0x7fe470159d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe4600098c0 tx=0x7fe4600046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 shutdown_connections 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47015d550 0x7fe47015f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4700d38e0 0x7fe47014d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 --2- 172.21.15.53:0/2759763797 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4700b2640 0x7fe470159d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 >> 172.21.15.53:0/2759763797 conn(0x7fe47008cc50 msgr2=0x7fe47006bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:38.156 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.175+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 shutdown_connections 2024-01-16T14:59:38.157 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:38.176+0000 7fe477a00640 1 -- 172.21.15.53:0/2759763797 wait complete. 2024-01-16T14:59:38.169 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-01-16T14:59:38.225 INFO:tasks.ceph.mgr.y.smithi053.stderr:2024-01-16T14:59:38.245+0000 7fed924fa040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-01-16T14:59:39.154 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:39.173+0000 7f615f800640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-01-16T14:59:39.154 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T14:59:39.174+0000 7f615f800640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-01-16T14:59:40.962 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-01-16T14:59:40.982+0000 7f74bd238500 -1 osd.0 0 log_to_monitors true 2024-01-16T14:59:41.029 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-01-16T14:59:41.047+0000 7f036a421500 -1 osd.1 0 log_to_monitors true 2024-01-16T14:59:41.037 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-01-16T14:59:41.056+0000 7fef79849500 -1 osd.7 0 log_to_monitors true 2024-01-16T14:59:41.052 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-01-16T14:59:41.071+0000 7f4973420500 -1 osd.2 0 log_to_monitors true 2024-01-16T14:59:41.083 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-01-16T14:59:41.102+0000 7f0345abe500 -1 osd.5 0 log_to_monitors true 2024-01-16T14:59:41.192 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-01-16T14:59:41.211+0000 7f0f2f80e500 -1 osd.6 0 log_to_monitors true 2024-01-16T14:59:41.267 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-01-16T14:59:41.287+0000 7f22f228a500 -1 osd.3 0 log_to_monitors true 2024-01-16T14:59:41.285 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-01-16T14:59:41.304+0000 7fb333012500 -1 osd.4 0 log_to_monitors true 2024-01-16T14:59:43.713 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-01-16T14:59:43.732+0000 7f74b7600640 -1 osd.0 0 waiting for initial osdmap 2024-01-16T14:59:43.718 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-01-16T14:59:43.737+0000 7f0364200640 -1 osd.1 0 waiting for initial osdmap 2024-01-16T14:59:43.721 INFO:tasks.ceph.osd.0.smithi053.stderr:2024-01-16T14:59:43.740+0000 7f74ae000640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:43.737 INFO:tasks.ceph.osd.1.smithi053.stderr:2024-01-16T14:59:43.756+0000 7f035ac00640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:43.738 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-01-16T14:59:43.757+0000 7f496d200640 -1 osd.2 0 waiting for initial osdmap 2024-01-16T14:59:43.750 INFO:tasks.ceph.osd.2.smithi053.stderr:2024-01-16T14:59:43.769+0000 7f4963c00640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:43.917 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-01-16T14:59:43.933+0000 7fef73c00640 -1 osd.7 0 waiting for initial osdmap 2024-01-16T14:59:43.931 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-01-16T14:59:43.951+0000 7f0341e00640 -1 osd.5 0 waiting for initial osdmap 2024-01-16T14:59:43.937 INFO:tasks.ceph.osd.7.smithi146.stderr:2024-01-16T14:59:43.956+0000 7fef6a600640 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:43.949 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-01-16T14:59:43.968+0000 7f0336a00640 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:44.440 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-01-16T14:59:44.459+0000 7f22ec800640 -1 osd.3 0 waiting for initial osdmap 2024-01-16T14:59:44.451 INFO:tasks.ceph.osd.3.smithi053.stderr:2024-01-16T14:59:44.470+0000 7f22e3200640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:44.725 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-01-16T14:59:44.744+0000 7fb32d000640 -1 osd.4 0 waiting for initial osdmap 2024-01-16T14:59:44.748 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-01-16T14:59:44.767+0000 7fb323a00640 -1 osd.4 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:44.766 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-01-16T14:59:44.786+0000 7f0f2b400640 -1 osd.6 0 waiting for initial osdmap 2024-01-16T14:59:44.772 INFO:tasks.ceph.osd.6.smithi146.stderr:2024-01-16T14:59:44.792+0000 7f0f20000640 -1 osd.6 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-01-16T14:59:44.975 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:45.170 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.188+0000 7f9efb200640 1 Processor -- start 2024-01-16T14:59:45.170 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.189+0000 7f9efb200640 1 -- start start 2024-01-16T14:59:45.172 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.190+0000 7f9efb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9efc15d3a0 0x7f9efc0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.172 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.190+0000 7f9efb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc0caa70 0x7f9efc06d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.172 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.190+0000 7f9efb200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9efc0ce5b0 con 0x7f9efc0cf930 2024-01-16T14:59:45.172 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.190+0000 7f9efb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9efc06fb50 con 0x7f9efc15d3a0 2024-01-16T14:59:45.172 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.190+0000 7f9efb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0cdc80 con 0x7f9efc0caa70 2024-01-16T14:59:45.173 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9efc15d3a0 0x7f9efc0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.173 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9efc15d3a0 0x7f9efc0cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46968/0 (socket says 172.21.15.53:46968) 2024-01-16T14:59:45.173 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 -- 172.21.15.53:0/2411385919 learned_addr learned my addr 172.21.15.53:0/2411385919 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:45.173 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9efa800640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc0caa70 0x7f9efc06d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.173 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 -- 172.21.15.53:0/2411385919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc0caa70 msgr2=0x7f9efc06d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:45.173 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc0caa70 0x7f9efc06d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.174 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 -- 172.21.15.53:0/2411385919 >> v1:172.21.15.146:6789/0 conn(0x7f9efc0cf930 legacy=0x7f9efc0ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.174 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9ef9e00640 1 -- 172.21.15.53:0/2411385919 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9efc0cdf60 con 0x7f9efc15d3a0 2024-01-16T14:59:45.174 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.191+0000 7f9efa800640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc0caa70 0x7f9efc06d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:45.175 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.193+0000 7f9ef9e00640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc15d3a0 0x7f9efc0cea40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009870 tx=0x7f9eec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da2cb3f871079f9d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.175 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.194+0000 7f9ef8a00640 1 -- 172.21.15.53:0/2411385919 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec031640 con 0x7f9efc15d3a0 2024-01-16T14:59:45.175 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.194+0000 7f9ef8a00640 1 -- 172.21.15.53:0/2411385919 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec0317a0 con 0x7f9efc15d3a0 2024-01-16T14:59:45.175 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.194+0000 7f9ef8a00640 1 -- 172.21.15.53:0/2411385919 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec031b30 con 0x7f9efc15d3a0 2024-01-16T14:59:45.176 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.194+0000 7f9efb200640 1 -- 172.21.15.53:0/2411385919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc15d3a0 msgr2=0x7f9efc0cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.176 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.194+0000 7f9efb200640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc15d3a0 0x7f9efc0cea40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009870 tx=0x7f9eec02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.176 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.195+0000 7f9efb200640 1 -- 172.21.15.53:0/2411385919 shutdown_connections 2024-01-16T14:59:45.177 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.195+0000 7f9efb200640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc0caa70 0x7f9efc06d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.177 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.195+0000 7f9efb200640 1 --2- 172.21.15.53:0/2411385919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc15d3a0 0x7f9efc0cea40 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.177 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.195+0000 7f9efb200640 1 -- 172.21.15.53:0/2411385919 >> 172.21.15.53:0/2411385919 conn(0x7f9efc0cc790 msgr2=0x7f9efc0ccba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.177 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.195+0000 7f9efb200640 1 -- 172.21.15.53:0/2411385919 shutdown_connections 2024-01-16T14:59:45.177 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.195+0000 7f9efb200640 1 -- 172.21.15.53:0/2411385919 wait complete. 2024-01-16T14:59:45.178 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.196+0000 7f9efb200640 1 Processor -- start 2024-01-16T14:59:45.178 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.196+0000 7f9efb200640 1 -- start start 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.197+0000 7f9efb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 0x7f9efc1f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.197+0000 7f9efb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9efc0cf930 0x7f9efc0aedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.197+0000 7f9efb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc15d3a0 0x7f9efc14feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.197+0000 7f9efb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0b00c0 con 0x7f9efc0caa70 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.197+0000 7f9efb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0af4b0 con 0x7f9efc0cf930 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.197+0000 7f9efb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc0af790 con 0x7f9efc15d3a0 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 0x7f9efc1f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 0x7f9efc1f23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46980/0 (socket says 172.21.15.53:46980) 2024-01-16T14:59:45.179 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 -- 172.21.15.53:0/4237681110 learned_addr learned my addr 172.21.15.53:0/4237681110 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9efa800640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc15d3a0 0x7f9efc14feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9400640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9efc0cf930 0x7f9efc0aedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 -- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc15d3a0 msgr2=0x7f9efc14feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc15d3a0 0x7f9efc14feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 -- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9efc0cf930 msgr2=0x7f9efc0aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9efc0cf930 0x7f9efc0aedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 -- 172.21.15.53:0/4237681110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec0094d0 con 0x7f9efc0caa70 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9efa800640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc15d3a0 0x7f9efc14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.198+0000 7f9ef9e00640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 0x7f9efc1f23c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9eec000c00 tx=0x7f9eec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.199+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec03ac60 con 0x7f9efc0caa70 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.199+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec03adc0 con 0x7f9efc0caa70 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.199+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec044630 con 0x7f9efc0caa70 2024-01-16T14:59:45.181 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.199+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc0afa70 con 0x7f9efc0caa70 2024-01-16T14:59:45.182 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.199+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc152c80 con 0x7f9efc0caa70 2024-01-16T14:59:45.182 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.200+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc067b60 con 0x7f9efc0caa70 2024-01-16T14:59:45.182 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.201+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 4) v1 ==== 89483+0+0 (secure 0 0 0) 0x7f9eec02f050 con 0x7f9efc0caa70 2024-01-16T14:59:45.183 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.202+0000 7f9ef2000640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ed406b230 0x7f9ed406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:45.183 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.202+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 3911+0+0 (secure 0 0 0) 0x7f9eec0ab1d0 con 0x7f9efc0caa70 2024-01-16T14:59:45.184 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.203+0000 7f9ef9400640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ed406b230 0x7f9ed406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:45.186 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.204+0000 7f9ef9400640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ed406b230 0x7f9ed406d6f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0c9db0 tx=0x7f9ee4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:45.190 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.209+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9eec0768b0 con 0x7f9efc0caa70 2024-01-16T14:59:45.386 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.405+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9efc0cf5d0 con 0x7f9efc0caa70 2024-01-16T14:59:45.387 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.406+0000 7f9ef2000640 1 -- 172.21.15.53:0/4237681110 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+9912 (secure 0 0 0) 0x7f9eec031e60 con 0x7f9efc0caa70 2024-01-16T14:59:45.387 INFO:teuthology.misc.health.smithi053.stdout: 2024-01-16T14:59:45.388 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":14,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:44.741954+0000","last_up_change":"2024-01-16T14:59:44.741954+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:45.390 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ed406b230 msgr2=0x7f9ed406d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.390 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ed406b230 0x7f9ed406d6f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9efc0c9db0 tx=0x7f9ee4009040 comp rx=0 tx=0).stop 2024-01-16T14:59:45.390 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 msgr2=0x7f9efc1f23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 0x7f9efc1f23c0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9eec000c00 tx=0x7f9eec0046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 shutdown_connections 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ed406b230 0x7f9ed406d6f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9efc15d3a0 0x7f9efc14feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9efc0cf930 0x7f9efc0aedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 --2- 172.21.15.53:0/4237681110 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9efc0caa70 0x7f9efc1f23c0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.408+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 >> 172.21.15.53:0/4237681110 conn(0x7f9efc0cc790 msgr2=0x7f9efc15f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.409+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 shutdown_connections 2024-01-16T14:59:45.391 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:45.409+0000 7f9efb200640 1 -- 172.21.15.53:0/4237681110 wait complete. 2024-01-16T14:59:45.405 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-01-16T14:59:49.618 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:49.626 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T14:59:52.211 DEBUG:teuthology.orchestra.run.smithi053:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:52.390 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.408+0000 7f7560000640 1 Processor -- start 2024-01-16T14:59:52.390 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.409+0000 7f7560000640 1 -- start start 2024-01-16T14:59:52.392 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.410+0000 7f7560000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 0x7f755806a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.392 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.410+0000 7f7560000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75580b5740 0x7f755806a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.392 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.410+0000 7f7560000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7558146fc0 con 0x7f755815e8c0 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.410+0000 7f7560000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755806fa30 con 0x7f75580b5740 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.410+0000 7f7560000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f755815e430 con 0x7f7558146be0 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.411+0000 7f755cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f755815e8c0 0x7f75580b5030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42324/0 (socket says 172.21.15.53:42324) 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.411+0000 7f755cc00640 1 -- 172.21.15.53:0/4082746038 learned_addr learned my addr 172.21.15.53:0/4082746038 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.411+0000 7f755d600640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 0x7f755806a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.411+0000 7f755e000640 1 --2- 172.21.15.53:0/4082746038 >> v2:172.21.15.146:3300/0 conn(0x7f75580b5740 0x7f755806a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.411+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520687979 0 0) 0x7f7558146fc0 con 0x7f755815e8c0 2024-01-16T14:59:52.393 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.411+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7538003520 con 0x7f755815e8c0 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1050441761 0 0) 0x7f7538003520 con 0x7f755815e8c0 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 msgr2=0x7f755806a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 0x7f755806a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 >> v2:172.21.15.146:3300/0 conn(0x7f75580b5740 msgr2=0x7f755806a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 --2- 172.21.15.53:0/4082746038 >> v2:172.21.15.146:3300/0 conn(0x7f75580b5740 0x7f755806a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f755815db00 con 0x7f755815e8c0 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7540001670 con 0x7f755815e8c0 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 >> v1:172.21.15.53:6789/0 conn(0x7f755815e8c0 legacy=0x7f75580b5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 0x7f7538003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7538003ef0 0x7f75380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.395 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7538004d00 0x7f7538005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755806fa30 con 0x7f7538004d00 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f755815e430 con 0x7f7538003ef0 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753800b980 con 0x7f7558146be0 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755e000640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7538004d00 0x7f7538005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755cc00640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7538003ef0 0x7f75380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755d600640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 0x7f7538003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755e000640 1 -- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 msgr2=0x7f7538003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755e000640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7558146be0 0x7f7538003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755e000640 1 -- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7538003ef0 msgr2=0x7f75380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755e000640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7538003ef0 0x7f75380047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.412+0000 7f755e000640 1 -- 172.21.15.53:0/4082746038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7538003520 con 0x7f7538004d00 2024-01-16T14:59:52.396 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f755e000640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7538004d00 0x7f7538005180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7550009f90 tx=0x7f75380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=410065b5fe2ad1e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f755002cc00 con 0x7f7538004d00 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f755000f040 con 0x7f7538004d00 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f7557e00640 1 -- 172.21.15.53:0/4082746038 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550034440 con 0x7f7538004d00 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f7560000640 1 -- 172.21.15.53:0/4082746038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7538004d00 msgr2=0x7f7538005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f7560000640 1 --2- 172.21.15.53:0/4082746038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7538004d00 0x7f7538005180 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7550009f90 tx=0x7f75380057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.414+0000 7f755e000640 1 -- 172.21.15.53:0/4082746038 reap_dead start 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.415+0000 7f7560000640 1 -- 172.21.15.53:0/4082746038 shutdown_connections 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.415+0000 7f7560000640 1 -- 172.21.15.53:0/4082746038 >> 172.21.15.53:0/4082746038 conn(0x7f755806d430 msgr2=0x7f755806d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.415+0000 7f7560000640 1 -- 172.21.15.53:0/4082746038 shutdown_connections 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.415+0000 7f7560000640 1 -- 172.21.15.53:0/4082746038 wait complete. 2024-01-16T14:59:52.398 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.417+0000 7f7560000640 1 Processor -- start 2024-01-16T14:59:52.399 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.417+0000 7f7560000640 1 -- start start 2024-01-16T14:59:52.399 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f7560000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7538004d00 0x7f7558145410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f7560000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75580cbba0 0x7f7558145950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f7560000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 0x7f75581605d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f7560000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7558131800 con 0x7f7558160140 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f7560000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7558160e70 con 0x7f75580cbba0 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f7560000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7558161150 con 0x7f7538004d00 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f755e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 0x7f75581605d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f755e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 0x7f75581605d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48822/0 (socket says 172.21.15.53:48822) 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f755e000640 1 -- 172.21.15.53:0/2468329613 learned_addr learned my addr 172.21.15.53:0/2468329613 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f755cc00640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75580cbba0 0x7f7558145950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.400 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.418+0000 7f755d600640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7538004d00 0x7f7558145410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.401 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f755e000640 1 -- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7538004d00 msgr2=0x7f7558145410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.401 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f755e000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7538004d00 0x7f7558145410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.401 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f755e000640 1 -- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75580cbba0 msgr2=0x7f7558145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.401 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f755e000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75580cbba0 0x7f7558145950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.401 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f755e000640 1 -- 172.21.15.53:0/2468329613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7558161430 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f755e000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 0x7f75581605d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7550002230 tx=0x7f7550009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550046070 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7550035ae0 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550035e70 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7558161710 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.419+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75581336a0 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.420+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7558131940 con 0x7f7558160140 2024-01-16T14:59:52.402 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.421+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f755003b050 con 0x7f7558160140 2024-01-16T14:59:52.403 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.422+0000 7f7555600640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f753006b2c0 0x7f753006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.404 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.422+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f75500aaec0 con 0x7f7558160140 2024-01-16T14:59:52.404 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.423+0000 7f755d600640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f753006b2c0 0x7f753006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.405 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.424+0000 7f755d600640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f753006b2c0 0x7f753006d780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75581429c0 tx=0x7f7548008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.409 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.427+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7558131940 con 0x7f7558160140 2024-01-16T14:59:52.606 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.624+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7558131940 con 0x7f7558160140 2024-01-16T14:59:52.607 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.625+0000 7f7555600640 1 -- 172.21.15.53:0/2468329613 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12814 (secure 0 0 0) 0x7f7558131940 con 0x7f7558160140 2024-01-16T14:59:52.607 INFO:teuthology.misc.health.smithi053.stdout: 2024-01-16T14:59:52.607 INFO:teuthology.misc.health.smithi053.stdout:{"epoch":18,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:48.969744+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:52.609 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f753006b2c0 msgr2=0x7f753006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.609 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f753006b2c0 0x7f753006d780 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f75581429c0 tx=0x7f7548008040 comp rx=0 tx=0).stop 2024-01-16T14:59:52.609 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 msgr2=0x7f75581605d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.610 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 0x7f75581605d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7550002230 tx=0x7f7550009750 comp rx=0 tx=0).stop 2024-01-16T14:59:52.610 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 shutdown_connections 2024-01-16T14:59:52.610 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f753006b2c0 0x7f753006d780 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.610 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7558160140 0x7f75581605d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.610 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75580cbba0 0x7f7558145950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.610 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 --2- 172.21.15.53:0/2468329613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7538004d00 0x7f7558145410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.611 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 >> 172.21.15.53:0/2468329613 conn(0x7f755806d430 msgr2=0x7f75580b5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.611 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 shutdown_connections 2024-01-16T14:59:52.611 INFO:teuthology.misc.health.smithi053.stderr:2024-01-16T14:59:52.628+0000 7f7560000640 1 -- 172.21.15.53:0/2468329613 wait complete. 2024-01-16T14:59:52.623 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-01-16T14:59:52.623 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-01-16T14:59:52.624 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T14:59:52.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T14:59:52.628 INFO:tasks.ceph.mds.a:Started 2024-01-16T14:59:52.628 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T14:59:52.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T14:59:52.633 INFO:tasks.ceph.mds.c:Started 2024-01-16T14:59:52.633 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T14:59:52.633 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T14:59:52.641 INFO:tasks.ceph.mds.b:Started 2024-01-16T14:59:52.641 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T14:59:52.641 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T14:59:52.648 INFO:tasks.ceph.mds.d:Started 2024-01-16T14:59:52.648 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-01-16T14:59:52.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:52.784 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T14:59:52.787 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T14:59:52.804 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T14:59:52.804 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T14:59:52.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.846+0000 7f2def000640 1 Processor -- start 2024-01-16T14:59:52.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.846+0000 7f2def000640 1 -- start start 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.847+0000 7f2def000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de80ce8f0 0x7f2de80ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.847+0000 7f2def000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2de80b2a00 0x7f2de80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.847+0000 7f2def000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de80d0de0 con 0x7f2de8160f80 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.847+0000 7f2def000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de80d10c0 con 0x7f2de80b2a00 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.847+0000 7f2def000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80d13a0 con 0x7f2de80ce8f0 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2de7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2de8160f80 0x7f2de8161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42340/0 (socket says 172.21.15.53:42340) 2024-01-16T14:59:52.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2de7e00640 1 -- 172.21.15.53:0/4217850819 learned_addr learned my addr 172.21.15.53:0/4217850819 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:52.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2ded000640 1 --2- 172.21.15.53:0/4217850819 >> v2:172.21.15.146:3300/0 conn(0x7f2de80b2a00 0x7f2de80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2ddfe00640 1 --2- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de80ce8f0 0x7f2de80ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2de7400640 1 -- 172.21.15.53:0/4217850819 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106853397 0 0) 0x7f2de80d0de0 con 0x7f2de8160f80 2024-01-16T14:59:52.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2de7400640 1 -- 172.21.15.53:0/4217850819 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dc8003520 con 0x7f2de8160f80 2024-01-16T14:59:52.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2ded000640 1 -- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de80ce8f0 msgr2=0x7f2de80ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:52.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2ded000640 1 --2- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de80ce8f0 0x7f2de80ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2ded000640 1 -- 172.21.15.53:0/4217850819 >> v1:172.21.15.53:6789/0 conn(0x7f2de8160f80 legacy=0x7f2de8161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.848+0000 7f2ded000640 1 -- 172.21.15.53:0/4217850819 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de80d1680 con 0x7f2de80b2a00 2024-01-16T14:59:52.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.849+0000 7f2ded000640 1 --2- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80b2e60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009d90 tx=0x7f2de002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c8dc68fad992fb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2de7400640 1 -- 172.21.15.53:0/4217850819 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de002cc50 con 0x7f2de80b2a00 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2de7400640 1 -- 172.21.15.53:0/4217850819 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de002cdb0 con 0x7f2de80b2a00 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2de7400640 1 -- 172.21.15.53:0/4217850819 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2de0034660 con 0x7f2de80b2a00 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 -- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 msgr2=0x7f2de80b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 --2- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80b2e60 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f2de0009d90 tx=0x7f2de002bea0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 -- 172.21.15.53:0/4217850819 shutdown_connections 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 --2- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80b2e60 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 --2- 172.21.15.53:0/4217850819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de80ce8f0 0x7f2de80ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 -- 172.21.15.53:0/4217850819 >> 172.21.15.53:0/4217850819 conn(0x7f2de809c550 msgr2=0x7f2de80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 -- 172.21.15.53:0/4217850819 shutdown_connections 2024-01-16T14:59:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.850+0000 7f2def000640 1 -- 172.21.15.53:0/4217850819 wait complete. 2024-01-16T14:59:52.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.852+0000 7f2def000640 1 Processor -- start 2024-01-16T14:59:52.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.852+0000 7f2def000640 1 -- start start 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2def000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80d3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2def000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 0x7f2de814db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2def000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de8160f80 0x7f2de814cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2def000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de814e0d0 con 0x7f2de80ce8f0 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2def000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de80ccea0 con 0x7f2de80b2a00 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2def000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de814d240 con 0x7f2de8160f80 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2ddfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 0x7f2de814db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2ddfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 0x7f2de814db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48868/0 (socket says 172.21.15.53:48868) 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.853+0000 7f2ddfe00640 1 -- 172.21.15.53:0/2086054076 learned_addr learned my addr 172.21.15.53:0/2086054076 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ded000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de8160f80 0x7f2de814cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2de7e00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80d3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ddfe00640 1 -- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de8160f80 msgr2=0x7f2de814cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ddfe00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de8160f80 0x7f2de814cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ddfe00640 1 -- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 msgr2=0x7f2de80d3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ddfe00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80d3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ddfe00640 1 -- 172.21.15.53:0/2086054076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de0009a70 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2ddfe00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 0x7f2de814db90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e9b0 tx=0x7f2dd000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0009670 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd000c9a0 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd000cd50 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de80cc570 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.854+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de80ccb30 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.855+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de80c6bb0 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.856+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2dd0013470 con 0x7f2de80ce8f0 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.857+0000 7f2de4c00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2dc006b2c0 0x7f2dc006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:52.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.857+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2dd0089b40 con 0x7f2de80ce8f0 2024-01-16T14:59:52.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.857+0000 7f2de7e00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2dc006b2c0 0x7f2dc006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:52.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.858+0000 7f2de7e00640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2dc006b2c0 0x7f2dc006d780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2dd80011e0 tx=0x7f2dd8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:52.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:52.862+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2dd0054cf0 con 0x7f2de80ce8f0 2024-01-16T14:59:53.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.057+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2de80675d0 con 0x7f2de80ce8f0 2024-01-16T14:59:53.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.059+0000 7f2de4c00640 1 -- 172.21.15.53:0/2086054076 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12814 (secure 0 0 0) 0x7f2dd0017090 con 0x7f2de80ce8f0 2024-01-16T14:59:53.040 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:53.040 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":18,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:48.969744+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:53.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.061+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2dc006b2c0 msgr2=0x7f2dc006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.061+0000 7f2def000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2dc006b2c0 0x7f2dc006d780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2dd80011e0 tx=0x7f2dd8009040 comp rx=0 tx=0).stop 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 msgr2=0x7f2de814db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 0x7f2de814db90 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2dd000e9b0 tx=0x7f2dd000ee70 comp rx=0 tx=0).stop 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 shutdown_connections 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2dc006b2c0 0x7f2dc006d780 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2de8160f80 0x7f2de814cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2de80ce8f0 0x7f2de814db90 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 --2- 172.21.15.53:0/2086054076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2de80b2a00 0x7f2de80d3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 >> 172.21.15.53:0/2086054076 conn(0x7f2de809c550 msgr2=0x7f2de8161360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 shutdown_connections 2024-01-16T14:59:53.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.062+0000 7f2def000640 1 -- 172.21.15.53:0/2086054076 wait complete. 2024-01-16T14:59:53.061 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:53.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:53.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.304+0000 7f4cfd400640 1 Processor -- start 2024-01-16T14:59:53.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.304+0000 7f4cfd400640 1 -- start start 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cf8070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cfd400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4cf806cb10 0x7f4cf806cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cfd400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf80706a0 con 0x7f4cf806c730 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cfd400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cf80b3290 con 0x7f4cf806cb10 2024-01-16T14:59:53.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cfd400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80708d0 con 0x7f4cf80b2e00 2024-01-16T14:59:53.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cf7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4cf806c730 0x7f4cf8071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42350/0 (socket says 172.21.15.53:42350) 2024-01-16T14:59:53.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cf7e00640 1 -- 172.21.15.53:0/2325983411 learned_addr learned my addr 172.21.15.53:0/2325983411 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:53.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cf7400640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cf8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cfca00640 1 --2- 172.21.15.53:0/2325983411 >> v2:172.21.15.53:3300/0 conn(0x7f4cf806cb10 0x7f4cf806cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.306+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709390982 0 0) 0x7f4cf80706a0 con 0x7f4cf806c730 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd8003520 con 0x7f4cf806c730 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2991857111 0 0) 0x7f4cd8003520 con 0x7f4cf806c730 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 msgr2=0x7f4cf8070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cf8070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 >> v2:172.21.15.53:3300/0 conn(0x7f4cf806cb10 msgr2=0x7f4cf806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 --2- 172.21.15.53:0/2325983411 >> v2:172.21.15.53:3300/0 conn(0x7f4cf806cb10 0x7f4cf806cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf8070bb0 con 0x7f4cf806c730 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4ce8001670 con 0x7f4cf806c730 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 >> v1:172.21.15.53:6789/0 conn(0x7f4cf806c730 legacy=0x7f4cf8071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cd8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80708d0 con 0x7f4cd8003970 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800b980 con 0x7f4cd80047c0 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bb40 con 0x7f4cf80b2e00 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cfca00640 1 --2- 172.21.15.53:0/2325983411 >> v2:172.21.15.53:3300/0 conn(0x7f4cf806cb10 0x7f4cf806cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.307+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cd8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.308+0000 7f4cf7400640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.308+0000 7f4cf7e00640 1 -- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 msgr2=0x7f4cd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.308+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.308+0000 7f4cf7e00640 1 -- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 msgr2=0x7f4cd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.308+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.308+0000 7f4cf7e00640 1 -- 172.21.15.53:0/2325983411 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8003520 con 0x7f4cd8003970 2024-01-16T14:59:53.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.309+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cd8003d90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c110 tx=0x7f4cd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9546a94a4f1073b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.309+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8005ad0 con 0x7f4cd8003970 2024-01-16T14:59:53.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.309+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce8005c50 con 0x7f4cd8003970 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.309+0000 7f4cf6a00640 1 -- 172.21.15.53:0/2325983411 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce802c830 con 0x7f4cd8003970 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfd400640 1 -- 172.21.15.53:0/2325983411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 msgr2=0x7f4cd8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfd400640 1 --2- 172.21.15.53:0/2325983411 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cd8003d90 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c110 tx=0x7f4cd80057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfca00640 1 -- 172.21.15.53:0/2325983411 reap_dead start 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfd400640 1 -- 172.21.15.53:0/2325983411 shutdown_connections 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfd400640 1 -- 172.21.15.53:0/2325983411 >> 172.21.15.53:0/2325983411 conn(0x7f4cf806a7d0 msgr2=0x7f4cf806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfd400640 1 -- 172.21.15.53:0/2325983411 shutdown_connections 2024-01-16T14:59:53.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.310+0000 7f4cfd400640 1 -- 172.21.15.53:0/2325983411 wait complete. 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.311+0000 7f4cfd400640 1 Processor -- start 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.312+0000 7f4cfd400640 1 -- start start 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.312+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cf81e6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.312+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cf80ac130 0x7f4cf81608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cfd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf81f6f30 0x7f4cf80ad970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cfd400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf81fa2a0 con 0x7f4cd8003970 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cfd400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80ac8d0 con 0x7f4cf80ac130 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cfd400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80acbb0 con 0x7f4cf81f6f30 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cf81e6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cf81e6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48890/0 (socket says 172.21.15.53:48890) 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 -- 172.21.15.53:0/381708242 learned_addr learned my addr 172.21.15.53:0/381708242 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 -- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf81f6f30 msgr2=0x7f4cf80ad970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf81f6f30 0x7f4cf80ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 -- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cf80ac130 msgr2=0x7f4cf81608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cf80ac130 0x7f4cf81608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cf80ac130 0x7f4cf81608b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.313+0000 7f4cf7e00640 1 -- 172.21.15.53:0/381708242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80ace90 con 0x7f4cd8003970 2024-01-16T14:59:53.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.314+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cf81e6930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ce802c3d0 tx=0x7f4ce8001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.314+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce803c040 con 0x7f4cd8003970 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.314+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce802dcd0 con 0x7f4cd8003970 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.314+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce80072b0 con 0x7f4cd8003970 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.314+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf80ad170 con 0x7f4cd8003970 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.314+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf80ac5b0 con 0x7f4cd8003970 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.315+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf8067450 con 0x7f4cd8003970 2024-01-16T14:59:53.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.316+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4ce8032070 con 0x7f4cd8003970 2024-01-16T14:59:53.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.317+0000 7f4cefe00640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.317+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4ce80ab2b0 con 0x7f4cd8003970 2024-01-16T14:59:53.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.317+0000 7f4cf7400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.318+0000 7f4cf7400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cf806d230 tx=0x7f4ce0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.322+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ce8076460 con 0x7f4cd8003970 2024-01-16T14:59:53.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.508+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4cf80b23b0 con 0x7f4cd8003970 2024-01-16T14:59:53.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.508+0000 7f4cefe00640 1 -- 172.21.15.53:0/381708242 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4ce8079a80 con 0x7f4cd8003970 2024-01-16T14:59:53.490 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T14:59:53.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 msgr2=0x7f4cd006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4cf806d230 tx=0x7f4ce0008040 comp rx=0 tx=0).stop 2024-01-16T14:59:53.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 msgr2=0x7f4cf81e6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cf81e6930 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4ce802c3d0 tx=0x7f4ce8001930 comp rx=0 tx=0).stop 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 shutdown_connections 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf81f6f30 0x7f4cf80ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cf80ac130 0x7f4cf81608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 --2- 172.21.15.53:0/381708242 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd8003970 0x7f4cf81e6930 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 >> 172.21.15.53:0/381708242 conn(0x7f4cf806a7d0 msgr2=0x7f4cf80ae7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 shutdown_connections 2024-01-16T14:59:53.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.511+0000 7f4cfd400640 1 -- 172.21.15.53:0/381708242 wait complete. 2024-01-16T14:59:53.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T14:59:53.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.749+0000 7f6382200640 1 Processor -- start 2024-01-16T14:59:53.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.750+0000 7f6382200640 1 -- start start 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.751+0000 7f6382200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f637c13df80 0x7f637c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.751+0000 7f6382200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c0cf9a0 0x7f637c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.751+0000 7f6382200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c06c470 con 0x7f637c13e360 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.751+0000 7f6382200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c06c750 con 0x7f637c13df80 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.751+0000 7f6382200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c06ca30 con 0x7f637c0cf9a0 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f637c13df80 0x7f637c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f637c13df80 0x7f637c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48904/0 (socket says 172.21.15.53:48904) 2024-01-16T14:59:53.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 -- 172.21.15.53:0/2445016059 learned_addr learned my addr 172.21.15.53:0/2445016059 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:53.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637be00640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c0cf9a0 0x7f637c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 -- 172.21.15.53:0/2445016059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c0cf9a0 msgr2=0x7f637c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c0cf9a0 0x7f637c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 -- 172.21.15.53:0/2445016059 >> v1:172.21.15.146:6789/0 conn(0x7f637c13e360 legacy=0x7f637c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:53.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637b400640 1 -- 172.21.15.53:0/2445016059 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c06cd10 con 0x7f637c13df80 2024-01-16T14:59:53.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.752+0000 7f637be00640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c0cf9a0 0x7f637c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:53.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.753+0000 7f637b400640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c13df80 0x7f637c15bee0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52dd767ac7a68c27 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f637a000640 1 -- 172.21.15.53:0/2445016059 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c031640 con 0x7f637c13df80 2024-01-16T14:59:53.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f637a000640 1 -- 172.21.15.53:0/2445016059 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c0317a0 con 0x7f637c13df80 2024-01-16T14:59:53.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f637a000640 1 -- 172.21.15.53:0/2445016059 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c031b30 con 0x7f637c13df80 2024-01-16T14:59:53.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f6382200640 1 -- 172.21.15.53:0/2445016059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c13df80 msgr2=0x7f637c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f6382200640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c13df80 0x7f637c15bee0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c02b8f0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f6382200640 1 -- 172.21.15.53:0/2445016059 shutdown_connections 2024-01-16T14:59:53.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f6382200640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c0cf9a0 0x7f637c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f6382200640 1 --2- 172.21.15.53:0/2445016059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c13df80 0x7f637c15bee0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.754+0000 7f6382200640 1 -- 172.21.15.53:0/2445016059 >> 172.21.15.53:0/2445016059 conn(0x7f637c06a800 msgr2=0x7f637c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.755+0000 7f6382200640 1 -- 172.21.15.53:0/2445016059 shutdown_connections 2024-01-16T14:59:53.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.755+0000 7f6382200640 1 -- 172.21.15.53:0/2445016059 wait complete. 2024-01-16T14:59:53.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.756+0000 7f6382200640 1 Processor -- start 2024-01-16T14:59:53.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.756+0000 7f6382200640 1 -- start start 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.757+0000 7f6382200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 0x7f637c0709d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.757+0000 7f6382200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c13df80 0x7f637c0a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.757+0000 7f6382200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f637c13e360 0x7f637c1531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.757+0000 7f6382200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c071d80 con 0x7f637c0cf9a0 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.757+0000 7f6382200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c070f10 con 0x7f637c13e360 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.757+0000 7f6382200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0a9fb0 con 0x7f637c13df80 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 0x7f637c0709d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 0x7f637c0709d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48906/0 (socket says 172.21.15.53:48906) 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 -- 172.21.15.53:0/4263218731 learned_addr learned my addr 172.21.15.53:0/4263218731 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637aa00640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c13df80 0x7f637c0a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637be00640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f637c13e360 0x7f637c1531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 -- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c13df80 msgr2=0x7f637c0a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c13df80 0x7f637c0a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 -- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f637c13e360 msgr2=0x7f637c1531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f637c13e360 0x7f637c1531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 -- 172.21.15.53:0/4263218731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0094d0 con 0x7f637c0cf9a0 2024-01-16T14:59:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637be00640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f637c13e360 0x7f637c1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:53.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.758+0000 7f637b400640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 0x7f637c0709d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.759+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c03ac60 con 0x7f637c0cf9a0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.759+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c03ade0 con 0x7f637c0cf9a0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.759+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c044630 con 0x7f637c0cf9a0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.759+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c071100 con 0x7f637c0cf9a0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.759+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c0716c0 con 0x7f637c0cf9a0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.760+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c0675d0 con 0x7f637c0cf9a0 2024-01-16T14:59:53.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.761+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f636c02f050 con 0x7f637c0cf9a0 2024-01-16T14:59:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.762+0000 7f6373400640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f635406b2f0 0x7f635406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.762+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f636c0abae0 con 0x7f637c0cf9a0 2024-01-16T14:59:53.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.762+0000 7f637aa00640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f635406b2f0 0x7f635406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.764+0000 7f637aa00640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f635406b2f0 0x7f635406d7b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f637c1555b0 tx=0x7f6364009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:53.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:53.767+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f636c076c00 con 0x7f637c0cf9a0 2024-01-16T14:59:53.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.002+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f637c15b0f0 con 0x7f637c0cf9a0 2024-01-16T14:59:53.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.003+0000 7f6373400640 1 -- 172.21.15.53:0/4263218731 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3697 (secure 0 0 0) 0x7f636c07a220 con 0x7f637c0cf9a0 2024-01-16T14:59:53.984 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:53.985 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4245,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-01-16T14:59:53.985 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 3 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f635406b2f0 msgr2=0x7f635406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f635406b2f0 0x7f635406d7b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f637c1555b0 tx=0x7f6364009040 comp rx=0 tx=0).stop 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 msgr2=0x7f637c0709d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 0x7f637c0709d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f636c009870 tx=0x7f636c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 shutdown_connections 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f635406b2f0 0x7f635406d7b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f637c13e360 0x7f637c1531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f637c13df80 0x7f637c0a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 --2- 172.21.15.53:0/4263218731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f637c0cf9a0 0x7f637c0709d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 >> 172.21.15.53:0/4263218731 conn(0x7f637c06a800 msgr2=0x7f637c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 shutdown_connections 2024-01-16T14:59:53.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.006+0000 7f6382200640 1 -- 172.21.15.53:0/4263218731 wait complete. 2024-01-16T14:59:54.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:54.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.237+0000 7f4e7d800640 1 Processor -- start 2024-01-16T14:59:54.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.237+0000 7f4e7d800640 1 -- start start 2024-01-16T14:59:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e7d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e78143c40 0x7f4e78071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e7d800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4e780717d0 0x7f4e780cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e7d800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e78070e50 con 0x7f4e780cca40 2024-01-16T14:59:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e7d800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e780723c0 con 0x7f4e780717d0 2024-01-16T14:59:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e7d800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e780cce40 con 0x7f4e78143c40 2024-01-16T14:59:54.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4e780717d0 0x7f4e780cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4e780717d0 0x7f4e780cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48916/0 (socket says 172.21.15.53:48916) 2024-01-16T14:59:54.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 -- 172.21.15.53:0/1936319970 learned_addr learned my addr 172.21.15.53:0/1936319970 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:54.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77e00640 1 --2- 172.21.15.53:0/1936319970 >> v2:172.21.15.146:3300/0 conn(0x7f4e78143c40 0x7f4e78071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 -- 172.21.15.53:0/1936319970 >> v2:172.21.15.146:3300/0 conn(0x7f4e78143c40 msgr2=0x7f4e78071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 --2- 172.21.15.53:0/1936319970 >> v2:172.21.15.146:3300/0 conn(0x7f4e78143c40 0x7f4e78071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 -- 172.21.15.53:0/1936319970 >> v1:172.21.15.146:6789/0 conn(0x7f4e780cca40 legacy=0x7f4e7815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77400640 1 -- 172.21.15.53:0/1936319970 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7806e170 con 0x7f4e780717d0 2024-01-16T14:59:54.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.239+0000 7f4e77e00640 1 --2- 172.21.15.53:0/1936319970 >> v2:172.21.15.146:3300/0 conn(0x7f4e78143c40 0x7f4e78071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:54.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.241+0000 7f4e77400640 1 --2- 172.21.15.53:0/1936319970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780717d0 0x7f4e780cc500 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e7806e410 tx=0x7f4e6002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754d6d46874e6a72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.241+0000 7f4e76a00640 1 -- 172.21.15.53:0/1936319970 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600317e0 con 0x7f4e780717d0 2024-01-16T14:59:54.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.241+0000 7f4e76a00640 1 -- 172.21.15.53:0/1936319970 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e60031940 con 0x7f4e780717d0 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.241+0000 7f4e76a00640 1 -- 172.21.15.53:0/1936319970 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e60031cd0 con 0x7f4e780717d0 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 -- 172.21.15.53:0/1936319970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780717d0 msgr2=0x7f4e780cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 --2- 172.21.15.53:0/1936319970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780717d0 0x7f4e780cc500 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4e7806e410 tx=0x7f4e6002b880 comp rx=0 tx=0).stop 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 -- 172.21.15.53:0/1936319970 shutdown_connections 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 --2- 172.21.15.53:0/1936319970 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780717d0 0x7f4e780cc500 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 --2- 172.21.15.53:0/1936319970 >> v2:172.21.15.146:3300/0 conn(0x7f4e78143c40 0x7f4e78071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 -- 172.21.15.53:0/1936319970 >> 172.21.15.53:0/1936319970 conn(0x7f4e7806a800 msgr2=0x7f4e7806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 -- 172.21.15.53:0/1936319970 shutdown_connections 2024-01-16T14:59:54.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.242+0000 7f4e7d800640 1 -- 172.21.15.53:0/1936319970 wait complete. 2024-01-16T14:59:54.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.244+0000 7f4e7d800640 1 Processor -- start 2024-01-16T14:59:54.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.244+0000 7f4e7d800640 1 -- start start 2024-01-16T14:59:54.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780717d0 0x7f4e78139570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 0x7f4e781e0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e78143c40 0x7f4e780aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7d800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781e2190 con 0x7f4e780cca40 2024-01-16T14:59:54.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781e1510 con 0x7f4e78143c40 2024-01-16T14:59:54.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e781e17f0 con 0x7f4e780717d0 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e77400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 0x7f4e781e0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e77400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 0x7f4e781e0a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48932/0 (socket says 172.21.15.53:48932) 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e77400640 1 -- 172.21.15.53:0/2777375883 learned_addr learned my addr 172.21.15.53:0/2777375883 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e77e00640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780717d0 0x7f4e78139570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.245+0000 7f4e7ce00640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e78143c40 0x7f4e780aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e77400640 1 -- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780717d0 msgr2=0x7f4e78139570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e77400640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780717d0 0x7f4e78139570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e77400640 1 -- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e78143c40 msgr2=0x7f4e780aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e77400640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e78143c40 0x7f4e780aee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e77400640 1 -- 172.21.15.53:0/2777375883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e600094d0 con 0x7f4e780cca40 2024-01-16T14:59:54.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e77400640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 0x7f4e781e0a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e6002b8c0 tx=0x7f4e600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600329c0 con 0x7f4e780cca40 2024-01-16T14:59:54.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e60032b40 con 0x7f4e780cca40 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600453f0 con 0x7f4e780cca40 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.246+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e781e1ad0 con 0x7f4e780cca40 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.247+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e78139ab0 con 0x7f4e780cca40 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.248+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e781f08e0 con 0x7f4e780cca40 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.248+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4e6004d3f0 con 0x7f4e780cca40 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.249+0000 7f4e6fe00640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.249+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f4e600ab7b0 con 0x7f4e780cca40 2024-01-16T14:59:54.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.250+0000 7f4e77e00640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.251+0000 7f4e77e00640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800a950 tx=0x7f4e68008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.254+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e60076a10 con 0x7f4e780cca40 2024-01-16T14:59:54.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.448+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e78067450 con 0x7f4e780cca40 2024-01-16T14:59:54.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.449+0000 7f4e6fe00640 1 -- 172.21.15.53:0/2777375883 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+12814 (secure 0 0 0) 0x7f4e6007a030 con 0x7f4e780cca40 2024-01-16T14:59:54.431 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:54.431 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":18,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:48.969744+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:54.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 msgr2=0x7f4e5006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800a950 tx=0x7f4e68008040 comp rx=0 tx=0).stop 2024-01-16T14:59:54.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 msgr2=0x7f4e781e0a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 0x7f4e781e0a00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4e6002b8c0 tx=0x7f4e600046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 shutdown_connections 2024-01-16T14:59:54.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e78143c40 0x7f4e780aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e780cca40 0x7f4e781e0a00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 --2- 172.21.15.53:0/2777375883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780717d0 0x7f4e78139570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 >> 172.21.15.53:0/2777375883 conn(0x7f4e7806a800 msgr2=0x7f4e78141230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 shutdown_connections 2024-01-16T14:59:54.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.452+0000 7f4e7d800640 1 -- 172.21.15.53:0/2777375883 wait complete. 2024-01-16T14:59:54.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T14:59:54.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T14:59:54.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.693+0000 7fb5e5000640 1 Processor -- start 2024-01-16T14:59:54.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.694+0000 7fb5e5000640 1 -- start start 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.695+0000 7fb5e5000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb5e00cf9a0 0x7fb5e00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.695+0000 7fb5e5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00d0320 con 0x7fb5e00b31d0 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.695+0000 7fb5e5000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00d1c60 con 0x7fb5e015d4b0 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.695+0000 7fb5e5000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5e00b2730 con 0x7fb5e00cf9a0 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5dfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb5e00b31d0 0x7fb5e00b3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42362/0 (socket says 172.21.15.53:42362) 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5dfe00640 1 -- 172.21.15.53:0/4181226454 learned_addr learned my addr 172.21.15.53:0/4181226454 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:54.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5dea00640 1 --2- 172.21.15.53:0/4181226454 >> v2:172.21.15.146:3300/0 conn(0x7fb5e00cf9a0 0x7fb5e00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429828273 0 0) 0x7fb5e00d0320 con 0x7fb5e00b31d0 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c0003520 con 0x7fb5e00b31d0 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927271682 0 0) 0x7fb5c0003520 con 0x7fb5e00b31d0 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 >> v2:172.21.15.146:3300/0 conn(0x7fb5e00cf9a0 msgr2=0x7fb5e00cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 --2- 172.21.15.53:0/4181226454 >> v2:172.21.15.146:3300/0 conn(0x7fb5e00cf9a0 0x7fb5e00cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 >> v1:172.21.15.146:6789/0 conn(0x7fb5e015d4b0 legacy=0x7fb5e00d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.696+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00d1ec0 con 0x7fb5e00b31d0 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb5d8001670 con 0x7fb5e00b31d0 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 >> v1:172.21.15.53:6789/0 conn(0x7fb5e00b31d0 legacy=0x7fb5e00b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c0003970 0x7fb5c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5c00047c0 0x7fb5c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5c0005180 0x7fb5c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00b2730 con 0x7fb5c00047c0 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c000c020 con 0x7fb5c0005180 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c000c1e0 con 0x7fb5c0003970 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dea00640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5c00047c0 0x7fb5c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dfe00640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5c0005180 0x7fb5c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5df400640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c0003970 0x7fb5c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dea00640 1 -- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c0003970 msgr2=0x7fb5c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dea00640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c0003970 0x7fb5c0003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dea00640 1 -- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5c0005180 msgr2=0x7fb5c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:54.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dea00640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5c0005180 0x7fb5c000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dea00640 1 -- 172.21.15.53:0/4181226454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c0003520 con 0x7fb5c00047c0 2024-01-16T14:59:54.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.697+0000 7fb5dfe00640 1 -- 172.21.15.53:0/4181226454 reap_dead start 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.698+0000 7fb5dea00640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5c00047c0 0x7fb5c0004c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009fc0 tx=0x7fb5c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d490bbcd852002 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c802cc20 con 0x7fb5c00047c0 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c800f040 con 0x7fb5c00047c0 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5de000640 1 -- 172.21.15.53:0/4181226454 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8034440 con 0x7fb5c00047c0 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5e5000640 1 -- 172.21.15.53:0/4181226454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5c00047c0 msgr2=0x7fb5c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5e5000640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5c00047c0 0x7fb5c0004c40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009fc0 tx=0x7fb5c00057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5e5000640 1 -- 172.21.15.53:0/4181226454 shutdown_connections 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5e5000640 1 --2- 172.21.15.53:0/4181226454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5c00047c0 0x7fb5c0004c40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5e5000640 1 -- 172.21.15.53:0/4181226454 >> 172.21.15.53:0/4181226454 conn(0x7fb5e00d3000 msgr2=0x7fb5e00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.699+0000 7fb5e5000640 1 -- 172.21.15.53:0/4181226454 shutdown_connections 2024-01-16T14:59:54.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.700+0000 7fb5e5000640 1 -- 172.21.15.53:0/4181226454 wait complete. 2024-01-16T14:59:54.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.701+0000 7fb5e5000640 1 Processor -- start 2024-01-16T14:59:54.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.701+0000 7fb5e5000640 1 -- start start 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.702+0000 7fb5e5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c00047c0 0x7fb5e0158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.702+0000 7fb5e5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e00b0fa0 0x7fb5e00ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.702+0000 7fb5e5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 0x7fb5e0151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.702+0000 7fb5e5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0158fd0 con 0x7fb5e0160610 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.702+0000 7fb5e5000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01592b0 con 0x7fb5e00b0fa0 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.702+0000 7fb5e5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0159590 con 0x7fb5c00047c0 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 0x7fb5e0151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 0x7fb5e0151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48954/0 (socket says 172.21.15.53:48954) 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 -- 172.21.15.53:0/1312083402 learned_addr learned my addr 172.21.15.53:0/1312083402 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5df400640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c00047c0 0x7fb5e0158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dea00640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e00b0fa0 0x7fb5e00ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 -- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c00047c0 msgr2=0x7fb5e0158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c00047c0 0x7fb5e0158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 -- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e00b0fa0 msgr2=0x7fb5e00ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e00b0fa0 0x7fb5e00ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.703+0000 7fb5dfe00640 1 -- 172.21.15.53:0/1312083402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e0159870 con 0x7fb5e0160610 2024-01-16T14:59:54.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.704+0000 7fb5dfe00640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 0x7fb5e0151050 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8001050 tx=0x7fb5d800e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.704+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8010510 con 0x7fb5e0160610 2024-01-16T14:59:54.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.704+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d8010690 con 0x7fb5e0160610 2024-01-16T14:59:54.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.704+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8010a40 con 0x7fb5e0160610 2024-01-16T14:59:54.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.704+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e0159b50 con 0x7fb5e0160610 2024-01-16T14:59:54.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.704+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e00ca610 con 0x7fb5e0160610 2024-01-16T14:59:54.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.705+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e0159d40 con 0x7fb5e0160610 2024-01-16T14:59:54.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.706+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb5d8010ba0 con 0x7fb5e0160610 2024-01-16T14:59:54.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.707+0000 7fb5d7400640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:54.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.707+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb5d8088ff0 con 0x7fb5e0160610 2024-01-16T14:59:54.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.708+0000 7fb5df400640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:54.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.709+0000 7fb5df400640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0001710 tx=0x7fb5d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:54.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.711+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5d80bbe90 con 0x7fb5e0160610 2024-01-16T14:59:54.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.903+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb5e0067450 con 0x7fb5e0160610 2024-01-16T14:59:54.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.904+0000 7fb5d7400640 1 -- 172.21.15.53:0/1312083402 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb5d8019020 con 0x7fb5e0160610 2024-01-16T14:59:54.886 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T14:59:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 msgr2=0x7fb5b806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0001710 tx=0x7fb5d0009040 comp rx=0 tx=0).stop 2024-01-16T14:59:54.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 msgr2=0x7fb5e0151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 0x7fb5e0151050 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb5d8001050 tx=0x7fb5d800e960 comp rx=0 tx=0).stop 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 shutdown_connections 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e0160610 0x7fb5e0151050 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e00b0fa0 0x7fb5e00ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 --2- 172.21.15.53:0/1312083402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5c00047c0 0x7fb5e0158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 >> 172.21.15.53:0/1312083402 conn(0x7fb5e00d3000 msgr2=0x7fb5e00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 shutdown_connections 2024-01-16T14:59:54.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:54.907+0000 7fb5e5000640 1 -- 172.21.15.53:0/1312083402 wait complete. 2024-01-16T14:59:54.908 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T14:59:55.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.146+0000 7efff2c00640 1 Processor -- start 2024-01-16T14:59:55.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.146+0000 7efff2c00640 1 -- start start 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.148+0000 7efff2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec06d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.148+0000 7efff2c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec070e50 con 0x7effec0cc500 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.148+0000 7efff2c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effec144020 con 0x7effec143c40 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.148+0000 7efff2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec0723c0 con 0x7effec0cc990 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7efff0e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7effec0cc500 0x7effec06d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42372/0 (socket says 172.21.15.53:42372) 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7efff0e00640 1 -- 172.21.15.53:0/3705734939 learned_addr learned my addr 172.21.15.53:0/3705734939 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7efff2200640 1 --2- 172.21.15.53:0/3705734939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054445418 0 0) 0x7effec070e50 con 0x7effec0cc500 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effcc003520 con 0x7effec0cc500 2024-01-16T14:59:55.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2318104170 0 0) 0x7effec144020 con 0x7effec143c40 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effec070e50 con 0x7effec143c40 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531041581 0 0) 0x7effcc003520 con 0x7effec0cc500 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7efff2200640 1 -- 172.21.15.53:0/3705734939 >> v1:172.21.15.146:6789/0 conn(0x7effec143c40 legacy=0x7effec071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7efff2200640 1 -- 172.21.15.53:0/3705734939 >> v1:172.21.15.53:6789/0 conn(0x7effec0cc500 legacy=0x7effec06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.149+0000 7efff2200640 1 -- 172.21.15.53:0/3705734939 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effec071e00 con 0x7effec0cc990 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.151+0000 7efff2200640 1 --2- 172.21.15.53:0/3705734939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec06d870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effe4009dc0 tx=0x7effe402bd50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dccba2ff87494f53 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.151+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4001690 con 0x7effec0cc990 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.151+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe402c880 con 0x7effec0cc990 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.151+0000 7effebe00640 1 -- 172.21.15.53:0/3705734939 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe402cc10 con 0x7effec0cc990 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 -- 172.21.15.53:0/3705734939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 msgr2=0x7effec06d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 --2- 172.21.15.53:0/3705734939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec06d870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7effe4009dc0 tx=0x7effe402bd50 comp rx=0 tx=0).stop 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 -- 172.21.15.53:0/3705734939 shutdown_connections 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 --2- 172.21.15.53:0/3705734939 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec06d870 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 -- 172.21.15.53:0/3705734939 >> 172.21.15.53:0/3705734939 conn(0x7effec06a800 msgr2=0x7effec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 -- 172.21.15.53:0/3705734939 shutdown_connections 2024-01-16T14:59:55.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.152+0000 7efff2c00640 1 -- 172.21.15.53:0/3705734939 wait complete. 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.154+0000 7efff2c00640 1 Processor -- start 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.154+0000 7efff2c00640 1 -- start start 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effec0cc500 0x7effec159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec13e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 0x7effec13e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec1f20a0 con 0x7effec143c40 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effec15a520 con 0x7effec0cc500 2024-01-16T14:59:55.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effec1f22b0 con 0x7effec0cc990 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 0x7effec13e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 0x7effec13e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48956/0 (socket says 172.21.15.53:48956) 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.155+0000 7efff2200640 1 -- 172.21.15.53:0/88637104 learned_addr learned my addr 172.21.15.53:0/88637104 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff0e00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec13e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff1800640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effec0cc500 0x7effec159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff2200640 1 -- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 msgr2=0x7effec13e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff2200640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec13e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff2200640 1 -- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effec0cc500 msgr2=0x7effec159fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff2200640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effec0cc500 0x7effec159fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff2200640 1 -- 172.21.15.53:0/88637104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe4009a70 con 0x7effec143c40 2024-01-16T14:59:55.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.156+0000 7efff2200640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 0x7effec13e5e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7effe402c360 tx=0x7effe4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.157+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe402ce70 con 0x7effec143c40 2024-01-16T14:59:55.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.157+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effe4034bd0 con 0x7effec143c40 2024-01-16T14:59:55.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.157+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effec1f2590 con 0x7effec143c40 2024-01-16T14:59:55.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.157+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe4003990 con 0x7effec143c40 2024-01-16T14:59:55.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.157+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec1f2b50 con 0x7effec143c40 2024-01-16T14:59:55.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.158+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effec1f23f0 con 0x7effec143c40 2024-01-16T14:59:55.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.159+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7effe403f420 con 0x7effec143c40 2024-01-16T14:59:55.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.160+0000 7effe9600640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effc406b2c0 0x7effc406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.161+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7effe40ac180 con 0x7effec143c40 2024-01-16T14:59:55.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.161+0000 7efff1800640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effc406b2c0 0x7effc406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.162+0000 7efff1800640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effc406b2c0 0x7effc406d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7effdc0011e0 tx=0x7effdc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.165+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effec1f23f0 con 0x7effec143c40 2024-01-16T14:59:55.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.361+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7effec1f23f0 con 0x7effec143c40 2024-01-16T14:59:55.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.362+0000 7effe9600640 1 -- 172.21.15.53:0/88637104 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+65 (secure 0 0 0) 0x7effec1f23f0 con 0x7effec143c40 2024-01-16T14:59:55.343 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:55.343 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T14:59:55.343 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T14:59:55.343 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T14:59:55.343 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T14:59:55.344 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T14:59:55.344 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T14:59:55.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effc406b2c0 msgr2=0x7effc406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effc406b2c0 0x7effc406d780 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7effdc0011e0 tx=0x7effdc009040 comp rx=0 tx=0).stop 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 msgr2=0x7effec13e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 0x7effec13e5e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7effe402c360 tx=0x7effe4004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 shutdown_connections 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effc406b2c0 0x7effc406d780 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effec143c40 0x7effec13e5e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effec0cc990 0x7effec13e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 --2- 172.21.15.53:0/88637104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effec0cc500 0x7effec159fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.364+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 >> 172.21.15.53:0/88637104 conn(0x7effec06a800 msgr2=0x7effec071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.365+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 shutdown_connections 2024-01-16T14:59:55.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.365+0000 7efff2c00640 1 -- 172.21.15.53:0/88637104 wait complete. 2024-01-16T14:59:55.363 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T14:59:55.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T14:59:55.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.598+0000 7f8a2be00640 1 Processor -- start 2024-01-16T14:59:55.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.599+0000 7f8a2be00640 1 -- start start 2024-01-16T14:59:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.600+0000 7f8a2be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a2c160920 0x7f8a2c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.600+0000 7f8a2be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8a2c0cfea0 0x7f8a2c161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.600+0000 7f8a2be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c0d0a10 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.600+0000 7f8a2be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2c0d1f50 con 0x7f8a2c160920 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.600+0000 7f8a2be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a2c0cf080 con 0x7f8a2c0cfea0 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a2c160920 0x7f8a2c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8a2c160920 0x7f8a2c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48970/0 (socket says 172.21.15.53:48970) 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 -- 172.21.15.53:0/3657668 learned_addr learned my addr 172.21.15.53:0/3657668 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 -- 172.21.15.53:0/3657668 >> v2:172.21.15.146:3300/0 conn(0x7f8a2c0cfea0 msgr2=0x7f8a2c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 --2- 172.21.15.53:0/3657668 >> v2:172.21.15.146:3300/0 conn(0x7f8a2c0cfea0 0x7f8a2c161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 -- 172.21.15.53:0/3657668 >> v1:172.21.15.146:6789/0 conn(0x7f8a2c0cf9f0 legacy=0x7f8a2c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.601+0000 7f8a2aa00640 1 -- 172.21.15.53:0/3657668 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c0d0310 con 0x7f8a2c160920 2024-01-16T14:59:55.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.602+0000 7f8a2aa00640 1 --2- 172.21.15.53:0/3657668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c160920 0x7f8a2c0b3f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c08c2c0 tx=0x7f8a1c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b8d650a919454eb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.602+0000 7f8a29600640 1 -- 172.21.15.53:0/3657668 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c031870 con 0x7f8a2c160920 2024-01-16T14:59:55.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.602+0000 7f8a29600640 1 -- 172.21.15.53:0/3657668 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c0319d0 con 0x7f8a2c160920 2024-01-16T14:59:55.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.602+0000 7f8a29600640 1 -- 172.21.15.53:0/3657668 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c031d60 con 0x7f8a2c160920 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 -- 172.21.15.53:0/3657668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c160920 msgr2=0x7f8a2c0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 --2- 172.21.15.53:0/3657668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c160920 0x7f8a2c0b3f30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f8a2c08c2c0 tx=0x7f8a1c02b910 comp rx=0 tx=0).stop 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 -- 172.21.15.53:0/3657668 shutdown_connections 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 --2- 172.21.15.53:0/3657668 >> v2:172.21.15.146:3300/0 conn(0x7f8a2c0cfea0 0x7f8a2c161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 --2- 172.21.15.53:0/3657668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c160920 0x7f8a2c0b3f30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 -- 172.21.15.53:0/3657668 >> 172.21.15.53:0/3657668 conn(0x7f8a2c0d3090 msgr2=0x7f8a2c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 -- 172.21.15.53:0/3657668 shutdown_connections 2024-01-16T14:59:55.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.603+0000 7f8a2be00640 1 -- 172.21.15.53:0/3657668 wait complete. 2024-01-16T14:59:55.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.605+0000 7f8a2be00640 1 Processor -- start 2024-01-16T14:59:55.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.605+0000 7f8a2be00640 1 -- start start 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 0x7f8a2c150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a2c0cfea0 0x7f8a2c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a2c160920 0x7f8a2c1610e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0b70e0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0b7330 con 0x7f8a2c160920 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a2c0b7610 con 0x7f8a2c0cfea0 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.606+0000 7f8a2aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 0x7f8a2c150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 0x7f8a2c150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48982/0 (socket says 172.21.15.53:48982) 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 -- 172.21.15.53:0/1925029731 learned_addr learned my addr 172.21.15.53:0/1925029731 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2b400640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a2c160920 0x7f8a2c1610e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2a000640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a2c0cfea0 0x7f8a2c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 -- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a2c0cfea0 msgr2=0x7f8a2c0ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a2c0cfea0 0x7f8a2c0ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 -- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a2c160920 msgr2=0x7f8a2c1610e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a2c160920 0x7f8a2c1610e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 -- 172.21.15.53:0/1925029731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a1c0094d0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2a000640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a2c0cfea0 0x7f8a2c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:55.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.607+0000 7f8a2aa00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 0x7f8a2c150820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c000c00 tx=0x7f8a1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.608+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c03ad10 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.608+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1c03ae70 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.608+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a1c044610 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.608+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c0b78f0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.608+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c0ca6a0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.609+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c067450 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.610+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8a1c04d450 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.611+0000 7f8a23400640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a0406b2c0 0x7f8a0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:55.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.611+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f8a1c02d040 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.611+0000 7f8a2a000640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a0406b2c0 0x7f8a0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:55.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.612+0000 7f8a2a000640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a0406b2c0 0x7f8a0406d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400a810 tx=0x7f8a14008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:55.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.615+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a1c0771a0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.808+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8a2c08c7e0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.915+0000 7f8a23400640 1 -- 172.21.15.53:0/1925029731 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v19) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8a1c07a7c0 con 0x7f8a2c0cf9f0 2024-01-16T14:59:55.897 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a0406b2c0 msgr2=0x7f8a0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a0406b2c0 0x7f8a0406d780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a1400a810 tx=0x7f8a14008040 comp rx=0 tx=0).stop 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 msgr2=0x7f8a2c150820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 0x7f8a2c150820 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8a1c000c00 tx=0x7f8a1c0046c0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 shutdown_connections 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a0406b2c0 0x7f8a0406d780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a2c160920 0x7f8a2c1610e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a2c0cfea0 0x7f8a2c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 --2- 172.21.15.53:0/1925029731 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a2c0cf9f0 0x7f8a2c150820 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:55.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.918+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 >> 172.21.15.53:0/1925029731 conn(0x7f8a2c0d3090 msgr2=0x7f8a2c15b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:55.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.919+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 shutdown_connections 2024-01-16T14:59:55.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:55.919+0000 7f8a2be00640 1 -- 172.21.15.53:0/1925029731 wait complete. 2024-01-16T14:59:55.928 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T14:59:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.154+0000 7efc0b600640 1 Processor -- start 2024-01-16T14:59:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.155+0000 7efc0b600640 1 -- start start 2024-01-16T14:59:56.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.157+0000 7efc0b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.157+0000 7efc0b600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7efc040cdcc0 0x7efc0406c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.157+0000 7efc0b600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc0413e440 con 0x7efc0415ed20 2024-01-16T14:59:56.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.157+0000 7efc0b600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc041417a0 con 0x7efc040cdcc0 2024-01-16T14:59:56.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.157+0000 7efc0b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc040ce1d0 con 0x7efc04141360 2024-01-16T14:59:56.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.157+0000 7efbfbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc08c00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7efc0415ed20 0x7efc0415f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:58078/0 (socket says 172.21.15.53:58078) 2024-01-16T14:59:56.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc08c00640 1 -- 172.21.15.53:0/704247751 learned_addr learned my addr 172.21.15.53:0/704247751 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:56.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc09600640 1 --2- 172.21.15.53:0/704247751 >> v2:172.21.15.53:3300/0 conn(0x7efc040cdcc0 0x7efc0406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc09600640 1 -- 172.21.15.53:0/704247751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 msgr2=0x7efc040cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc09600640 1 --2- 172.21.15.53:0/704247751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc09600640 1 -- 172.21.15.53:0/704247751 >> v1:172.21.15.146:6789/0 conn(0x7efc0415ed20 legacy=0x7efc0415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.158+0000 7efc09600640 1 -- 172.21.15.53:0/704247751 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc040ce4b0 con 0x7efc040cdcc0 2024-01-16T14:59:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.159+0000 7efc09600640 1 --2- 172.21.15.53:0/704247751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc040cdcc0 0x7efc0406c850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009d90 tx=0x7efbfc02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c1232dd8d4bc3a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.160+0000 7efc03e00640 1 -- 172.21.15.53:0/704247751 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc02cbc0 con 0x7efc040cdcc0 2024-01-16T14:59:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.160+0000 7efc03e00640 1 -- 172.21.15.53:0/704247751 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc02cd20 con 0x7efc040cdcc0 2024-01-16T14:59:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.160+0000 7efc03e00640 1 -- 172.21.15.53:0/704247751 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc034660 con 0x7efc040cdcc0 2024-01-16T14:59:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.160+0000 7efc0b600640 1 -- 172.21.15.53:0/704247751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc040cdcc0 msgr2=0x7efc0406c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.160+0000 7efc0b600640 1 --2- 172.21.15.53:0/704247751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc040cdcc0 0x7efc0406c850 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009d90 tx=0x7efbfc02be10 comp rx=0 tx=0).stop 2024-01-16T14:59:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.161+0000 7efc0b600640 1 -- 172.21.15.53:0/704247751 shutdown_connections 2024-01-16T14:59:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.161+0000 7efc0b600640 1 --2- 172.21.15.53:0/704247751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc040cdcc0 0x7efc0406c850 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.161+0000 7efc0b600640 1 --2- 172.21.15.53:0/704247751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.161+0000 7efc0b600640 1 -- 172.21.15.53:0/704247751 >> 172.21.15.53:0/704247751 conn(0x7efc040b0be0 msgr2=0x7efc040b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:56.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.161+0000 7efc0b600640 1 -- 172.21.15.53:0/704247751 shutdown_connections 2024-01-16T14:59:56.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.161+0000 7efc0b600640 1 -- 172.21.15.53:0/704247751 wait complete. 2024-01-16T14:59:56.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.162+0000 7efc0b600640 1 Processor -- start 2024-01-16T14:59:56.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.162+0000 7efc0b600640 1 -- start start 2024-01-16T14:59:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.163+0000 7efc0b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc040cdcc0 0x7efc040aae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.163+0000 7efc0b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040ad6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.163+0000 7efc0b600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 0x7efc04071120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.163+0000 7efc0b600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04070d00 con 0x7efc0415ed20 2024-01-16T14:59:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.163+0000 7efc0b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc04151ba0 con 0x7efc040cdcc0 2024-01-16T14:59:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.163+0000 7efc0b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc040ae3b0 con 0x7efc04141360 2024-01-16T14:59:56.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 0x7efc04071120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 0x7efc04071120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49004/0 (socket says 172.21.15.53:49004) 2024-01-16T14:59:56.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 -- 172.21.15.53:0/2622094766 learned_addr learned my addr 172.21.15.53:0/2622094766 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:56.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efbfbe00640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040ad6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc08c00640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc040cdcc0 0x7efc040aae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 -- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 msgr2=0x7efc040ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040ad6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 -- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc040cdcc0 msgr2=0x7efc040aae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc040cdcc0 0x7efc040aae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc09600640 1 -- 172.21.15.53:0/2622094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbfc009a70 con 0x7efc0415ed20 2024-01-16T14:59:56.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.164+0000 7efc08c00640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc040cdcc0 0x7efc040aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:56.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.165+0000 7efc09600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 0x7efc04071120 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009eb0 tx=0x7efbfc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:56.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.165+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc03e610 con 0x7efc0415ed20 2024-01-16T14:59:56.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.165+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbfc02ce90 con 0x7efc0415ed20 2024-01-16T14:59:56.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.165+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc040ae0d0 con 0x7efc0415ed20 2024-01-16T14:59:56.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.165+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbfc03e9b0 con 0x7efc0415ed20 2024-01-16T14:59:56.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.165+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc04070920 con 0x7efc0415ed20 2024-01-16T14:59:56.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.166+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc04070e40 con 0x7efc0415ed20 2024-01-16T14:59:56.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.167+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7efbfc03b050 con 0x7efc0415ed20 2024-01-16T14:59:56.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.168+0000 7efc01600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbdc06b2c0 0x7efbdc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:56.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.168+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5227+0+0 (secure 0 0 0) 0x7efbfc0ac180 con 0x7efc0415ed20 2024-01-16T14:59:56.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.169+0000 7efc08c00640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbdc06b2c0 0x7efbdc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:56.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.170+0000 7efc08c00640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbdc06b2c0 0x7efbdc06d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efbf4000f60 tx=0x7efbf4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:56.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.173+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efc04070e40 con 0x7efc0415ed20 2024-01-16T14:59:56.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.368+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7efc04070e40 con 0x7efc0415ed20 2024-01-16T14:59:56.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.926+0000 7efc01600640 1 -- 172.21.15.53:0/2622094766 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v20) v1 ==== 170+0+0 (secure 0 0 0) 0x7efc04070e40 con 0x7efc0415ed20 2024-01-16T14:59:56.908 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T14:59:56.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbdc06b2c0 msgr2=0x7efbdc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbdc06b2c0 0x7efbdc06d780 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efbf4000f60 tx=0x7efbf4009040 comp rx=0 tx=0).stop 2024-01-16T14:59:56.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 msgr2=0x7efc04071120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:56.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 0x7efc04071120 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efbfc009eb0 tx=0x7efbfc004a40 comp rx=0 tx=0).stop 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 shutdown_connections 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbdc06b2c0 0x7efbdc06d780 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc0415ed20 0x7efc04071120 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc04141360 0x7efc040ad6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 --2- 172.21.15.53:0/2622094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc040cdcc0 0x7efc040aae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.930+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 >> 172.21.15.53:0/2622094766 conn(0x7efc040b0be0 msgr2=0x7efc0413c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:56.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.931+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 shutdown_connections 2024-01-16T14:59:56.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:56.931+0000 7efc0b600640 1 -- 172.21.15.53:0/2622094766 wait complete. 2024-01-16T14:59:56.953 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T14:59:57.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.158+0000 7fa0ae600640 1 Processor -- start 2024-01-16T14:59:57.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.158+0000 7fa0ae600640 1 -- start start 2024-01-16T14:59:57.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.160+0000 7fa0ae600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0a80cf9a0 0x7fa0a8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.160+0000 7fa0ae600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a8071690 con 0x7fa0a8070ea0 2024-01-16T14:59:57.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.160+0000 7fa0ae600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0a8071970 con 0x7fa0a814b2c0 2024-01-16T14:59:57.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.160+0000 7fa0ae600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0a814b6a0 con 0x7fa0a80cf9a0 2024-01-16T14:59:57.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.160+0000 7fa0a7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa0a8070ea0 0x7fa0a80cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42384/0 (socket says 172.21.15.53:42384) 2024-01-16T14:59:57.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.160+0000 7fa0a7e00640 1 -- 172.21.15.53:0/2341770785 learned_addr learned my addr 172.21.15.53:0/2341770785 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6a00640 1 --2- 172.21.15.53:0/2341770785 >> v2:172.21.15.146:3300/0 conn(0x7fa0a80cf9a0 0x7fa0a8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307805236 0 0) 0x7fa0a8071690 con 0x7fa0a8070ea0 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa088003520 con 0x7fa0a8070ea0 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 897299138 0 0) 0x7fa088003520 con 0x7fa0a8070ea0 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 >> v2:172.21.15.146:3300/0 conn(0x7fa0a80cf9a0 msgr2=0x7fa0a8070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 --2- 172.21.15.53:0/2341770785 >> v2:172.21.15.146:3300/0 conn(0x7fa0a80cf9a0 0x7fa0a8070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 >> v1:172.21.15.146:6789/0 conn(0x7fa0a814b2c0 legacy=0x7fa0a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a8071c50 con 0x7fa0a8070ea0 2024-01-16T14:59:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fa0a0001670 con 0x7fa0a8070ea0 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 >> v1:172.21.15.53:6789/0 conn(0x7fa0a8070ea0 legacy=0x7fa0a80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa088003970 0x7fa088003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0880047c0 0x7fa088004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa088005180 0x7fa08800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a814b6a0 con 0x7fa088003970 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08800c020 con 0x7fa0880047c0 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.161+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa08800c1e0 con 0x7fa088005180 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7400640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa088003970 0x7fa088003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7e00640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa088005180 0x7fa08800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a6a00640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0880047c0 0x7fa088004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7400640 1 -- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa088005180 msgr2=0x7fa08800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7400640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa088005180 0x7fa08800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7400640 1 -- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0880047c0 msgr2=0x7fa088004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:57.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7400640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0880047c0 0x7fa088004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7400640 1 -- 172.21.15.53:0/2341770785 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa088003520 con 0x7fa088003970 2024-01-16T14:59:57.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7e00640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa088005180 0x7fa08800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:57.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.162+0000 7fa0a7e00640 1 -- 172.21.15.53:0/2341770785 reap_dead start 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.164+0000 7fa0a7400640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa088003970 0x7fa088003d90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa098001750 tx=0x7fa0880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c40f60868c988a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.164+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0980052d0 con 0x7fa088003970 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.164+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa09802e050 con 0x7fa088003970 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.164+0000 7fa0a6000640 1 -- 172.21.15.53:0/2341770785 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa098003830 con 0x7fa088003970 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 -- 172.21.15.53:0/2341770785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa088003970 msgr2=0x7fa088003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa088003970 0x7fa088003d90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa098001750 tx=0x7fa0880057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 -- 172.21.15.53:0/2341770785 shutdown_connections 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 --2- 172.21.15.53:0/2341770785 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa088003970 0x7fa088003d90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 -- 172.21.15.53:0/2341770785 >> 172.21.15.53:0/2341770785 conn(0x7fa0a806a800 msgr2=0x7fa0a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 -- 172.21.15.53:0/2341770785 shutdown_connections 2024-01-16T14:59:57.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.165+0000 7fa0ae600640 1 -- 172.21.15.53:0/2341770785 wait complete. 2024-01-16T14:59:57.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.167+0000 7fa0ae600640 1 Processor -- start 2024-01-16T14:59:57.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.167+0000 7fa0ae600640 1 -- start start 2024-01-16T14:59:57.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0ae600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa088003970 0x7fa0a80a8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0ae600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 0x7fa0a8160f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0ae600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80abb20 0x7fa0a80abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0ae600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a8161800 con 0x7fa0a8160af0 2024-01-16T14:59:57.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0ae600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80a94b0 con 0x7fa088003970 2024-01-16T14:59:57.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0ae600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0a80a9790 con 0x7fa0a80abb20 2024-01-16T14:59:57.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 0x7fa0a8160f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 0x7fa0a8160f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49008/0 (socket says 172.21.15.53:49008) 2024-01-16T14:59:57.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 -- 172.21.15.53:0/419150892 learned_addr learned my addr 172.21.15.53:0/419150892 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a7400640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa088003970 0x7fa0a80a8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a7e00640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80abb20 0x7fa0a80abfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 -- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80abb20 msgr2=0x7fa0a80abfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80abb20 0x7fa0a80abfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 -- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa088003970 msgr2=0x7fa0a80a8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.168+0000 7fa0a6a00640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa088003970 0x7fa0a80a8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa0a6a00640 1 -- 172.21.15.53:0/419150892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a80a9a70 con 0x7fa0a8160af0 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa0a7e00640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80abb20 0x7fa0a80abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa0a7400640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa088003970 0x7fa0a80a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa0a6a00640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 0x7fa0a8160f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa090007f30 tx=0x7fa090011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa09000e030 con 0x7fa0a8160af0 2024-01-16T14:59:57.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa090016470 con 0x7fa0a8160af0 2024-01-16T14:59:57.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0a80a9d50 con 0x7fa0a8160af0 2024-01-16T14:59:57.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa090016820 con 0x7fa0a8160af0 2024-01-16T14:59:57.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.169+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0a80aa550 con 0x7fa0a8160af0 2024-01-16T14:59:57.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.170+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0a80675d0 con 0x7fa0a8160af0 2024-01-16T14:59:57.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.171+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa090016980 con 0x7fa0a8160af0 2024-01-16T14:59:57.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.172+0000 7fa09f400640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa08006b2f0 0x7fa08006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:57.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.172+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fa09000f070 con 0x7fa0a8160af0 2024-01-16T14:59:57.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.173+0000 7fa0a7400640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa08006b2f0 0x7fa08006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:57.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.174+0000 7fa0a7400640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa08006b2f0 0x7fa08006d7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa098033000 tx=0x7fa098009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:57.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.177+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa090055950 con 0x7fa0a8160af0 2024-01-16T14:59:57.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.405+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa0a806dff0 con 0x7fa0a8160af0 2024-01-16T14:59:57.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.927+0000 7fa09f400640 1 -- 172.21.15.53:0/419150892 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa090058f70 con 0x7fa0a8160af0 2024-01-16T14:59:57.909 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 2 and data pool 3 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa08006b2f0 msgr2=0x7fa08006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa08006b2f0 0x7fa08006d7b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa098033000 tx=0x7fa098009030 comp rx=0 tx=0).stop 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 msgr2=0x7fa0a8160f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 0x7fa0a8160f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa090007f30 tx=0x7fa090011660 comp rx=0 tx=0).stop 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 shutdown_connections 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa08006b2f0 0x7fa08006d7b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0a80abb20 0x7fa0a80abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0a8160af0 0x7fa0a8160f70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 --2- 172.21.15.53:0/419150892 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa088003970 0x7fa0a80a8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:57.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 >> 172.21.15.53:0/419150892 conn(0x7fa0a806a800 msgr2=0x7fa0a8148010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:57.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.930+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 shutdown_connections 2024-01-16T14:59:57.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:57.931+0000 7fa0ae600640 1 -- 172.21.15.53:0/419150892 wait complete. 2024-01-16T14:59:57.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:58.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.133+0000 7fac68a00640 1 Processor -- start 2024-01-16T14:59:58.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.134+0000 7fac68a00640 1 -- start start 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.135+0000 7fac68a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fac640cc500 0x7fac640cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.135+0000 7fac68a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac64070e50 con 0x7fac64143c40 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.135+0000 7fac68a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac64144020 con 0x7fac6406d160 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.135+0000 7fac68a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac640723c0 con 0x7fac640cc500 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac63400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fac64143c40 0x7fac64071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42394/0 (socket says 172.21.15.53:42394) 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac63400640 1 -- 172.21.15.53:0/3718910034 learned_addr learned my addr 172.21.15.53:0/3718910034 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:58.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac5a800640 1 --2- 172.21.15.53:0/3718910034 >> v2:172.21.15.53:3300/0 conn(0x7fac640cc500 0x7fac640cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac5a800640 1 -- 172.21.15.53:0/3718910034 >> v1:172.21.15.146:6789/0 conn(0x7fac6406d160 legacy=0x7fac6406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac5a800640 1 -- 172.21.15.53:0/3718910034 >> v1:172.21.15.53:6789/0 conn(0x7fac64143c40 legacy=0x7fac64071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac5a800640 1 -- 172.21.15.53:0/3718910034 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64071e00 con 0x7fac640cc500 2024-01-16T14:59:58.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.136+0000 7fac62a00640 1 -- 172.21.15.53:0/3718910034 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4256411918 0 0) 0x7fac64070e50 con 0x7fac64143c40 2024-01-16T14:59:58.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.137+0000 7fac5a800640 1 --2- 172.21.15.53:0/3718910034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac640cc500 0x7fac640cc990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097a0 tx=0x7fac4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71e046eeaf22497e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac62a00640 1 -- 172.21.15.53:0/3718910034 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c02e070 con 0x7fac640cc500 2024-01-16T14:59:58.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac62a00640 1 -- 172.21.15.53:0/3718910034 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c0315a0 con 0x7fac640cc500 2024-01-16T14:59:58.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac62a00640 1 -- 172.21.15.53:0/3718910034 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4c031930 con 0x7fac640cc500 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac68a00640 1 -- 172.21.15.53:0/3718910034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac640cc500 msgr2=0x7fac640cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac68a00640 1 --2- 172.21.15.53:0/3718910034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac640cc500 0x7fac640cc990 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fac4c0097a0 tx=0x7fac4c02b840 comp rx=0 tx=0).stop 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac68a00640 1 -- 172.21.15.53:0/3718910034 shutdown_connections 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac68a00640 1 --2- 172.21.15.53:0/3718910034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac640cc500 0x7fac640cc990 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac68a00640 1 -- 172.21.15.53:0/3718910034 >> 172.21.15.53:0/3718910034 conn(0x7fac6406a800 msgr2=0x7fac6406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.138+0000 7fac68a00640 1 -- 172.21.15.53:0/3718910034 shutdown_connections 2024-01-16T14:59:58.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.139+0000 7fac68a00640 1 -- 172.21.15.53:0/3718910034 wait complete. 2024-01-16T14:59:58.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.140+0000 7fac68a00640 1 Processor -- start 2024-01-16T14:59:58.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.140+0000 7fac68a00640 1 -- start start 2024-01-16T14:59:58.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac68a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 0x7fac64149240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac68a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac640cc500 0x7fac640c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac68a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac64143c40 0x7fac640b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac68a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac64147420 con 0x7fac6406d160 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac68a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6414a1b0 con 0x7fac640cc500 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac68a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac64149910 con 0x7fac64143c40 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac63400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 0x7fac64149240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac63400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 0x7fac64149240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49022/0 (socket says 172.21.15.53:49022) 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.141+0000 7fac63400640 1 -- 172.21.15.53:0/4029618770 learned_addr learned my addr 172.21.15.53:0/4029618770 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac5a800640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac640cc500 0x7fac640c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63400640 1 -- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac64143c40 msgr2=0x7fac640b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63e00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac64143c40 0x7fac640b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63400640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac64143c40 0x7fac640b2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63400640 1 -- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac640cc500 msgr2=0x7fac640c4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63400640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac640cc500 0x7fac640c4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63400640 1 -- 172.21.15.53:0/4029618770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c009400 con 0x7fac6406d160 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac5a800640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac640cc500 0x7fac640c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:58.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac63400640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 0x7fac64149240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c8c0 tx=0x7fac5400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.142+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5400a640 con 0x7fac6406d160 2024-01-16T14:59:58.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.143+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5400a7c0 con 0x7fac6406d160 2024-01-16T14:59:58.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.143+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5400ab70 con 0x7fac6406d160 2024-01-16T14:59:58.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.143+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac64149bf0 con 0x7fac6406d160 2024-01-16T14:59:58.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.143+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6415aae0 con 0x7fac6406d160 2024-01-16T14:59:58.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.144+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac64147560 con 0x7fac6406d160 2024-01-16T14:59:58.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.145+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fac5400acd0 con 0x7fac6406d160 2024-01-16T14:59:58.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.146+0000 7fac5be00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac3c06b2c0 0x7fac3c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.146+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fac540891f0 con 0x7fac6406d160 2024-01-16T14:59:58.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.146+0000 7fac5a800640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac3c06b2c0 0x7fac3c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.147+0000 7fac5a800640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac3c06b2c0 0x7fac3c06d780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac4c036000 tx=0x7fac4c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.150+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac64147560 con 0x7fac6406d160 2024-01-16T14:59:58.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.346+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac64147560 con 0x7fac6406d160 2024-01-16T14:59:58.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.348+0000 7fac5be00640 1 -- 172.21.15.53:0/4029618770 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16482 (secure 0 0 0) 0x7fac64147560 con 0x7fac6406d160 2024-01-16T14:59:58.330 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:58.330 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":21,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:57.407354+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:58.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac3c06b2c0 msgr2=0x7fac3c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac3c06b2c0 0x7fac3c06d780 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fac4c036000 tx=0x7fac4c03d040 comp rx=0 tx=0).stop 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 msgr2=0x7fac64149240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 0x7fac64149240 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fac5400c8c0 tx=0x7fac5400cd80 comp rx=0 tx=0).stop 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 shutdown_connections 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac3c06b2c0 0x7fac3c06d780 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac64143c40 0x7fac640b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac640cc500 0x7fac640c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 --2- 172.21.15.53:0/4029618770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac6406d160 0x7fac64149240 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 >> 172.21.15.53:0/4029618770 conn(0x7fac6406a800 msgr2=0x7fac6406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 shutdown_connections 2024-01-16T14:59:58.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.351+0000 7fac68a00640 1 -- 172.21.15.53:0/4029618770 wait complete. 2024-01-16T14:59:58.351 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T14:59:58.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.575+0000 7fb42b000640 1 Processor -- start 2024-01-16T14:59:58.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.575+0000 7fb42b000640 1 -- start start 2024-01-16T14:59:58.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb42b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 0x7fb4240b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb42b000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4240d0a10 con 0x7fb4240cf9f0 2024-01-16T14:59:58.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb42b000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb424161e30 con 0x7fb4240cfe70 2024-01-16T14:59:58.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb42b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240d1f50 con 0x7fb424160920 2024-01-16T14:59:58.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb429200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb4240cf9f0 0x7fb424161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42398/0 (socket says 172.21.15.53:42398) 2024-01-16T14:59:58.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb429200640 1 -- 172.21.15.53:0/1321371630 learned_addr learned my addr 172.21.15.53:0/1321371630 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T14:59:58.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.577+0000 7fb429c00640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 0x7fb4240b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097718678 0 0) 0x7fb4240d0a10 con 0x7fb4240cf9f0 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb404003520 con 0x7fb4240cf9f0 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2275014577 0 0) 0x7fb404003520 con 0x7fb4240cf9f0 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 msgr2=0x7fb4240b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 0x7fb4240b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 >> v1:172.21.15.146:6789/0 conn(0x7fb4240cfe70 legacy=0x7fb424161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240cf080 con 0x7fb4240cf9f0 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb40c001670 con 0x7fb4240cf9f0 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 >> v1:172.21.15.53:6789/0 conn(0x7fb4240cf9f0 legacy=0x7fb424161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 0x7fb4040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb404003f70 0x7fb4040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404004d00 0x7fb404005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240d1f50 con 0x7fb404003f70 2024-01-16T14:59:58.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40400bac0 con 0x7fb404004d00 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40400bc80 con 0x7fb424160920 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.578+0000 7fb429200640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb404003f70 0x7fb4040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb42a600640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404004d00 0x7fb404005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb429200640 1 -- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 msgr2=0x7fb4040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb429200640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb424160920 0x7fb4040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb429200640 1 -- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404004d00 msgr2=0x7fb404005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb429200640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404004d00 0x7fb404005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb429200640 1 -- 172.21.15.53:0/1321371630 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404003520 con 0x7fb404003f70 2024-01-16T14:59:58.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.579+0000 7fb42a600640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404004d00 0x7fb404005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:58.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.580+0000 7fb429200640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb404003f70 0x7fb4040047c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c0e0 tx=0x7fb4040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a43162613ba0c8a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.580+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c005ad0 con 0x7fb404003f70 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.580+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c005c30 con 0x7fb404003f70 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.580+0000 7fb423e00640 1 -- 172.21.15.53:0/1321371630 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0072b0 con 0x7fb404003f70 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42b000640 1 -- 172.21.15.53:0/1321371630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb404003f70 msgr2=0x7fb4040047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42b000640 1 --2- 172.21.15.53:0/1321371630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb404003f70 0x7fb4040047c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb40c00c0e0 tx=0x7fb4040057f0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42a600640 1 -- 172.21.15.53:0/1321371630 reap_dead start 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42b000640 1 -- 172.21.15.53:0/1321371630 shutdown_connections 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42b000640 1 -- 172.21.15.53:0/1321371630 >> 172.21.15.53:0/1321371630 conn(0x7fb4240d3090 msgr2=0x7fb4240d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42b000640 1 -- 172.21.15.53:0/1321371630 shutdown_connections 2024-01-16T14:59:58.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.581+0000 7fb42b000640 1 -- 172.21.15.53:0/1321371630 wait complete. 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.583+0000 7fb42b000640 1 Processor -- start 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.583+0000 7fb42b000640 1 -- start start 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42b000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404003f70 0x7fb4241514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42b000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4240cb050 0x7fb4240c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42b000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 0x7fb4241f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42b000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240c9500 con 0x7fb4240c9070 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42b000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4241f74a0 con 0x7fb404003f70 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42b000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4240ca290 con 0x7fb4240cb050 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 0x7fb4241f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 0x7fb4241f6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49046/0 (socket says 172.21.15.53:49046) 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 -- 172.21.15.53:0/1533981053 learned_addr learned my addr 172.21.15.53:0/1533981053 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb429200640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4240cb050 0x7fb4240c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb429c00640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404003f70 0x7fb4241514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 -- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4240cb050 msgr2=0x7fb4240c8b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4240cb050 0x7fb4240c8b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 -- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404003f70 msgr2=0x7fb4241514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.584+0000 7fb42a600640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404003f70 0x7fb4241514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb42a600640 1 -- 172.21.15.53:0/1533981053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4240ca570 con 0x7fb4240c9070 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb429c00640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404003f70 0x7fb4241514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb42a600640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 0x7fb4241f6720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb42406a740 tx=0x7fb41c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c006d50 con 0x7fb4240c9070 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb41c003970 con 0x7fb4240c9070 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4240ca850 con 0x7fb4240c9070 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb41c003d20 con 0x7fb4240c9070 2024-01-16T14:59:58.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.585+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4241519e0 con 0x7fb4240c9070 2024-01-16T14:59:58.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.586+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4240cad70 con 0x7fb4240c9070 2024-01-16T14:59:58.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.587+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb41c005070 con 0x7fb4240c9070 2024-01-16T14:59:58.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.588+0000 7fb421600640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3fc06b2c0 0x7fb3fc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:58.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.588+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb41c089980 con 0x7fb4240c9070 2024-01-16T14:59:58.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.589+0000 7fb429c00640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3fc06b2c0 0x7fb3fc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:58.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.590+0000 7fb429c00640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3fc06b2c0 0x7fb3fc06d780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb4140099e0 tx=0x7fb414008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:58.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.593+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4240cad70 con 0x7fb4240c9070 2024-01-16T14:59:58.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.781+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4240cad70 con 0x7fb4240c9070 2024-01-16T14:59:58.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.783+0000 7fb421600640 1 -- 172.21.15.53:0/1533981053 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16482 (secure 0 0 0) 0x7fb4240cad70 con 0x7fb4240c9070 2024-01-16T14:59:58.765 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T14:59:58.765 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":21,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:57.407354+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T14:59:58.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3fc06b2c0 msgr2=0x7fb3fc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3fc06b2c0 0x7fb3fc06d780 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb4140099e0 tx=0x7fb414008040 comp rx=0 tx=0).stop 2024-01-16T14:59:58.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 msgr2=0x7fb4241f6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:58.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 0x7fb4241f6720 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb42406a740 tx=0x7fb41c00dc00 comp rx=0 tx=0).stop 2024-01-16T14:59:58.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 shutdown_connections 2024-01-16T14:59:58.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3fc06b2c0 0x7fb3fc06d780 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4240c9070 0x7fb4241f6720 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4240cb050 0x7fb4240c8b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.785+0000 7fb42b000640 1 --2- 172.21.15.53:0/1533981053 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb404003f70 0x7fb4241514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.786+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 >> 172.21.15.53:0/1533981053 conn(0x7fb4240d3090 msgr2=0x7fb4240d36e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.786+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 shutdown_connections 2024-01-16T14:59:58.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:58.786+0000 7fb42b000640 1 -- 172.21.15.53:0/1533981053 wait complete. 2024-01-16T14:59:58.785 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T14:59:58.810 INFO:tasks.ceph.mds.b.smithi146.stderr:2024-01-16T14:59:58.829+0000 7f1d13c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T14:59:59.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.020+0000 7f2a20600640 1 Processor -- start 2024-01-16T14:59:59.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.021+0000 7f2a20600640 1 -- start start 2024-01-16T14:59:59.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.022+0000 7f2a20600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180cfcf0 0x7f2a18160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.022+0000 7f2a20600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2a180d3910 0x7f2a18160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.022+0000 7f2a20600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a180d4160 con 0x7f2a180d00d0 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.022+0000 7f2a20600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a180ceff0 con 0x7f2a180d3910 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.022+0000 7f2a20600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180cf410 con 0x7f2a180cfcf0 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2a180d3910 0x7f2a18160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2a180d3910 0x7f2a18160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49060/0 (socket says 172.21.15.53:49060) 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 -- 172.21.15.53:0/1784099678 learned_addr learned my addr 172.21.15.53:0/1784099678 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1dc00640 1 --2- 172.21.15.53:0/1784099678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180cfcf0 0x7f2a18160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 -- 172.21.15.53:0/1784099678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180cfcf0 msgr2=0x7f2a18160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 --2- 172.21.15.53:0/1784099678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180cfcf0 0x7f2a18160700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 -- 172.21.15.53:0/1784099678 >> v1:172.21.15.146:6789/0 conn(0x7f2a180d00d0 legacy=0x7f2a180b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T14:59:59.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.023+0000 7f2a1d200640 1 -- 172.21.15.53:0/1784099678 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a180b2420 con 0x7f2a180d3910 2024-01-16T14:59:59.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.024+0000 7f2a1d200640 1 --2- 172.21.15.53:0/1784099678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a18160c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a000098a0 tx=0x7f2a0002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c32b7ce4ab7e603c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a17e00640 1 -- 172.21.15.53:0/1784099678 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a000317c0 con 0x7f2a180d3910 2024-01-16T14:59:59.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a17e00640 1 -- 172.21.15.53:0/1784099678 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a00031920 con 0x7f2a180d3910 2024-01-16T14:59:59.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a17e00640 1 -- 172.21.15.53:0/1784099678 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a00031cb0 con 0x7f2a180d3910 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a20600640 1 -- 172.21.15.53:0/1784099678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 msgr2=0x7f2a18160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a20600640 1 --2- 172.21.15.53:0/1784099678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a18160c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2a000098a0 tx=0x7f2a0002b860 comp rx=0 tx=0).stop 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a20600640 1 -- 172.21.15.53:0/1784099678 shutdown_connections 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.025+0000 7f2a20600640 1 --2- 172.21.15.53:0/1784099678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a18160c40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.026+0000 7f2a20600640 1 --2- 172.21.15.53:0/1784099678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180cfcf0 0x7f2a18160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.026+0000 7f2a20600640 1 -- 172.21.15.53:0/1784099678 >> 172.21.15.53:0/1784099678 conn(0x7f2a1809c530 msgr2=0x7f2a180d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.026+0000 7f2a20600640 1 -- 172.21.15.53:0/1784099678 shutdown_connections 2024-01-16T14:59:59.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.026+0000 7f2a20600640 1 -- 172.21.15.53:0/1784099678 wait complete. 2024-01-16T14:59:59.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.027+0000 7f2a20600640 1 Processor -- start 2024-01-16T14:59:59.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.028+0000 7f2a20600640 1 -- start start 2024-01-16T14:59:59.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a20600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180cfcf0 0x7f2a1814e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a20600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180d00d0 0x7f2a1815d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a20600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a1815d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a20600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180b7c10 con 0x7f2a180d3910 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a20600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180c8830 con 0x7f2a180cfcf0 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a20600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a180c8b10 con 0x7f2a180d00d0 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a1815d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a1815d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49062/0 (socket says 172.21.15.53:49062) 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 -- 172.21.15.53:0/3247069141 learned_addr learned my addr 172.21.15.53:0/3247069141 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1dc00640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180cfcf0 0x7f2a1814e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1d200640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180d00d0 0x7f2a1815d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 -- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180d00d0 msgr2=0x7f2a1815d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180d00d0 0x7f2a1815d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 -- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180cfcf0 msgr2=0x7f2a1814e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180cfcf0 0x7f2a1814e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1e600640 1 -- 172.21.15.53:0/3247069141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a000094d0 con 0x7f2a180d3910 2024-01-16T14:59:59.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1dc00640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180cfcf0 0x7f2a1814e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:59.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.029+0000 7f2a1d200640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180d00d0 0x7f2a1815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.030+0000 7f2a1e600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a1815d9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000c780 tx=0x7f2a1000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.030+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a10004b60 con 0x7f2a180d3910 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.030+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a1000ce90 con 0x7f2a180d3910 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.030+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a1000a850 con 0x7f2a180d3910 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.030+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a180c8df0 con 0x7f2a180d3910 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.030+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a180c93b0 con 0x7f2a180d3910 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.031+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a180c95a0 con 0x7f2a180d3910 2024-01-16T14:59:59.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.032+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2a1000a9b0 con 0x7f2a180d3910 2024-01-16T14:59:59.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.034+0000 7f2a15600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f29f006b2c0 0x7f29f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T14:59:59.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.034+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2a10089e30 con 0x7f2a180d3910 2024-01-16T14:59:59.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.034+0000 7f2a1dc00640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f29f006b2c0 0x7f29f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T14:59:59.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.035+0000 7f2a1dc00640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f29f006b2c0 0x7f29f006d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2a08006fd0 tx=0x7f2a08008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T14:59:59.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.038+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a10054c00 con 0x7f2a180d3910 2024-01-16T14:59:59.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.273+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2a180675d0 con 0x7f2a180d3910 2024-01-16T14:59:59.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.982+0000 7f2a15600640 1 -- 172.21.15.53:0/3247069141 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v6) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2a10058220 con 0x7f2a180d3910 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.985+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f29f006b2c0 msgr2=0x7f29f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f29f006b2c0 0x7f29f006d780 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2a08006fd0 tx=0x7f2a08008040 comp rx=0 tx=0).stop 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 msgr2=0x7f2a1815d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a1815d9f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2a1000c780 tx=0x7f2a1000cc40 comp rx=0 tx=0).stop 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 shutdown_connections 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f29f006b2c0 0x7f29f006d780 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2a180d3910 0x7f2a1815d9f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2a180d00d0 0x7f2a1815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 --2- 172.21.15.53:0/3247069141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2a180cfcf0 0x7f2a1814e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 >> 172.21.15.53:0/3247069141 conn(0x7f2a1809c530 msgr2=0x7f2a1815f2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T14:59:59.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 shutdown_connections 2024-01-16T14:59:59.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T14:59:59.986+0000 7f2a20600640 1 -- 172.21.15.53:0/3247069141 wait complete. 2024-01-16T14:59:59.990 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T14:59:59.990 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:00:00.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.237+0000 7fbac4c00640 1 Processor -- start 2024-01-16T15:00:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.237+0000 7fbac4c00640 1 -- start start 2024-01-16T15:00:00.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbac4c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbac013fcb0 0x7fbac00cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbac4c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00ce440 con 0x7fbac00cdc20 2024-01-16T15:00:00.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbac4c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac00ce720 con 0x7fbac013cf10 2024-01-16T15:00:00.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbac4c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac00cea00 con 0x7fbac013fcb0 2024-01-16T15:00:00.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbabfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbac00cdc20 0x7fbac00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42404/0 (socket says 172.21.15.53:42404) 2024-01-16T15:00:00.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbabfe00640 1 -- 172.21.15.53:0/689652296 learned_addr learned my addr 172.21.15.53:0/689652296 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:00.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.239+0000 7fbabea00640 1 --2- 172.21.15.53:0/689652296 >> v2:172.21.15.146:3300/0 conn(0x7fbac013fcb0 0x7fbac00cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.240+0000 7fbabe000640 1 -- 172.21.15.53:0/689652296 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501515022 0 0) 0x7fbac00ce440 con 0x7fbac00cdc20 2024-01-16T15:00:00.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.240+0000 7fbabe000640 1 -- 172.21.15.53:0/689652296 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaa0003520 con 0x7fbac00cdc20 2024-01-16T15:00:00.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.240+0000 7fbabe000640 1 -- 172.21.15.53:0/689652296 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63579238 0 0) 0x7fbaa0003520 con 0x7fbac00cdc20 2024-01-16T15:00:00.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.240+0000 7fbabea00640 1 -- 172.21.15.53:0/689652296 >> v1:172.21.15.146:6789/0 conn(0x7fbac013cf10 legacy=0x7fbac013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.240+0000 7fbabea00640 1 -- 172.21.15.53:0/689652296 >> v1:172.21.15.53:6789/0 conn(0x7fbac00cdc20 legacy=0x7fbac00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.240+0000 7fbabea00640 1 -- 172.21.15.53:0/689652296 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac00b1d90 con 0x7fbac013fcb0 2024-01-16T15:00:00.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.241+0000 7fbabea00640 1 --2- 172.21.15.53:0/689652296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013fcb0 0x7fbac00cd6e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80097a0 tx=0x7fbaa802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1c47c4bce694372 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.242+0000 7fbabe000640 1 -- 172.21.15.53:0/689652296 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaa802f050 con 0x7fbac013fcb0 2024-01-16T15:00:00.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.242+0000 7fbabe000640 1 -- 172.21.15.53:0/689652296 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbaa8031500 con 0x7fbac013fcb0 2024-01-16T15:00:00.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.242+0000 7fbabe000640 1 -- 172.21.15.53:0/689652296 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab80019f0 con 0x7fbac013fcb0 2024-01-16T15:00:00.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 -- 172.21.15.53:0/689652296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013fcb0 msgr2=0x7fbac00cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 --2- 172.21.15.53:0/689652296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013fcb0 0x7fbac00cd6e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbaa80097a0 tx=0x7fbaa802b820 comp rx=0 tx=0).stop 2024-01-16T15:00:00.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 -- 172.21.15.53:0/689652296 shutdown_connections 2024-01-16T15:00:00.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 --2- 172.21.15.53:0/689652296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013fcb0 0x7fbac00cd6e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 -- 172.21.15.53:0/689652296 >> 172.21.15.53:0/689652296 conn(0x7fbac006a7d0 msgr2=0x7fbac006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 -- 172.21.15.53:0/689652296 shutdown_connections 2024-01-16T15:00:00.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.243+0000 7fbac4c00640 1 -- 172.21.15.53:0/689652296 wait complete. 2024-01-16T15:00:00.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.245+0000 7fbac4c00640 1 Processor -- start 2024-01-16T15:00:00.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.245+0000 7fbac4c00640 1 -- start start 2024-01-16T15:00:00.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbac4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 0x7fbac015ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbac4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013cf10 0x7fbac01f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbac4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac013fcb0 0x7fbac00ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbac4c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac013c610 con 0x7fbac00cdc20 2024-01-16T15:00:00.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbac4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac013ba90 con 0x7fbac013cf10 2024-01-16T15:00:00.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbac4c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac013bd70 con 0x7fbac013fcb0 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbabf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 0x7fbac015ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbabf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 0x7fbac015ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49068/0 (socket says 172.21.15.53:49068) 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.246+0000 7fbabf400640 1 -- 172.21.15.53:0/3887834004 learned_addr learned my addr 172.21.15.53:0/3887834004 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabea00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013cf10 0x7fbac01f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabfe00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac013fcb0 0x7fbac00ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabf400640 1 -- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac013fcb0 msgr2=0x7fbac00ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabf400640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac013fcb0 0x7fbac00ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabf400640 1 -- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013cf10 msgr2=0x7fbac01f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabf400640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013cf10 0x7fbac01f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabf400640 1 -- 172.21.15.53:0/3887834004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa8009400 con 0x7fbac00cdc20 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabfe00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac013fcb0 0x7fbac00ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:00.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbabf400640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 0x7fbac015ce60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c450 tx=0x7fbab000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab00052d0 con 0x7fbac00cdc20 2024-01-16T15:00:00.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab0005430 con 0x7fbac00cdc20 2024-01-16T15:00:00.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab0012870 con 0x7fbac00cdc20 2024-01-16T15:00:00.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.247+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac013c050 con 0x7fbac00cdc20 2024-01-16T15:00:00.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.248+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac015d3a0 con 0x7fbac00cdc20 2024-01-16T15:00:00.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.249+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac0067450 con 0x7fbac00cdc20 2024-01-16T15:00:00.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.250+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbab00129d0 con 0x7fbac00cdc20 2024-01-16T15:00:00.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.250+0000 7fbab7400640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba9806b2c0 0x7fba9806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:00.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.250+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbab0089770 con 0x7fbac00cdc20 2024-01-16T15:00:00.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.251+0000 7fbabea00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba9806b2c0 0x7fba9806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:00.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.252+0000 7fbabea00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba9806b2c0 0x7fba9806d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8035000 tx=0x7fbaa8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:00.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.255+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbab0054540 con 0x7fbac00cdc20 2024-01-16T15:00:00.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.490+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbac00cbf10 con 0x7fbac00cdc20 2024-01-16T15:00:00.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.992+0000 7fbab7400640 1 -- 172.21.15.53:0/3887834004 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbab0057b60 con 0x7fbac00cdc20 2024-01-16T15:00:00.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba9806b2c0 msgr2=0x7fba9806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba9806b2c0 0x7fba9806d780 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbaa8035000 tx=0x7fbaa8008030 comp rx=0 tx=0).stop 2024-01-16T15:00:00.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 msgr2=0x7fbac015ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:00.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 0x7fbac015ce60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbab000c450 tx=0x7fbab000c910 comp rx=0 tx=0).stop 2024-01-16T15:00:00.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 shutdown_connections 2024-01-16T15:00:00.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba9806b2c0 0x7fba9806d780 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbac013fcb0 0x7fbac00ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbac013cf10 0x7fbac01f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 --2- 172.21.15.53:0/3887834004 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbac00cdc20 0x7fbac015ce60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:00.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.996+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 >> 172.21.15.53:0/3887834004 conn(0x7fbac006a7d0 msgr2=0x7fbac00b37f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:00.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.997+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 shutdown_connections 2024-01-16T15:00:00.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:00.997+0000 7fbac4c00640 1 -- 172.21.15.53:0/3887834004 wait complete. 2024-01-16T15:00:01.002 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:00:01.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:01.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.247+0000 7f1c09600640 1 Processor -- start 2024-01-16T15:00:01.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.247+0000 7f1c09600640 1 -- start start 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1c09600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1c04071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1c09600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c040ccc10 0x7f1c0406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1c09600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c04070e50 con 0x7f1c040717d0 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1c09600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c040723c0 con 0x7f1c040ccc10 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1c09600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04144020 con 0x7f1c04143c40 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1bfbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1c040717d0 0x7f1c040cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42414/0 (socket says 172.21.15.53:42414) 2024-01-16T15:00:01.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.249+0000 7f1bfbe00640 1 -- 172.21.15.53:0/1713895270 learned_addr learned my addr 172.21.15.53:0/1713895270 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03e00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1c04071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c08c00640 1 --2- 172.21.15.53:0/1713895270 >> v2:172.21.15.146:3300/0 conn(0x7f1c040ccc10 0x7f1c0406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368565202 0 0) 0x7f1c04070e50 con 0x7f1c040717d0 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be4003520 con 0x7f1c040717d0 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310373113 0 0) 0x7f1be4003520 con 0x7f1c040717d0 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 msgr2=0x7f1c04071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1c04071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 >> v2:172.21.15.146:3300/0 conn(0x7f1c040ccc10 msgr2=0x7f1c0406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 --2- 172.21.15.53:0/1713895270 >> v2:172.21.15.146:3300/0 conn(0x7f1c040ccc10 0x7f1c0406d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c04071e00 con 0x7f1c040717d0 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1bec001670 con 0x7f1c040717d0 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 >> v1:172.21.15.53:6789/0 conn(0x7f1c040717d0 legacy=0x7f1c040cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.250+0000 7f1c03400640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1be4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03400640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1be40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03400640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be4004d00 0x7f1be4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c040723c0 con 0x7f1be4003ef0 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04144020 con 0x7f1be4004d00 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1be400b9d0 con 0x7f1c04143c40 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1bfbe00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1be40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03e00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1be4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c08c00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be4004d00 0x7f1be4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1bfbe00640 1 -- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 msgr2=0x7f1be4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1bfbe00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1be4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1bfbe00640 1 -- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be4004d00 msgr2=0x7f1be4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1bfbe00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be4004d00 0x7f1be4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1bfbe00640 1 -- 172.21.15.53:0/1713895270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1be4003520 con 0x7f1be4003ef0 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c08c00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1be4004d00 0x7f1be4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.251+0000 7f1c03e00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04143c40 0x7f1be4003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.252+0000 7f1bfbe00640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1be40047c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00c270 tx=0x7f1be40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66ee9a77da3093d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec02cae0 con 0x7f1be4003ef0 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bec02cc40 con 0x7f1be4003ef0 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c03400640 1 -- 172.21.15.53:0/1713895270 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bec005af0 con 0x7f1be4003ef0 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c09600640 1 -- 172.21.15.53:0/1713895270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 msgr2=0x7f1be40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c09600640 1 --2- 172.21.15.53:0/1713895270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1be40047c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1bec00c270 tx=0x7f1be40057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c08c00640 1 -- 172.21.15.53:0/1713895270 reap_dead start 2024-01-16T15:00:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c09600640 1 -- 172.21.15.53:0/1713895270 shutdown_connections 2024-01-16T15:00:01.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c09600640 1 -- 172.21.15.53:0/1713895270 >> 172.21.15.53:0/1713895270 conn(0x7f1c0406a800 msgr2=0x7f1c0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c09600640 1 -- 172.21.15.53:0/1713895270 shutdown_connections 2024-01-16T15:00:01.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.253+0000 7f1c09600640 1 -- 172.21.15.53:0/1713895270 wait complete. 2024-01-16T15:00:01.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.255+0000 7f1c09600640 1 Processor -- start 2024-01-16T15:00:01.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.255+0000 7f1c09600640 1 -- start start 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.256+0000 7f1c09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1c0413b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.256+0000 7f1c09600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c04159970 0x7f1c04159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.256+0000 7f1c09600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04142770 0x7f1c04142bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c09600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041f4a70 con 0x7f1be4003ef0 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c09600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041f4e90 con 0x7f1c04159970 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c09600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c041434a0 con 0x7f1c04142770 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1c0413b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1c0413b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49082/0 (socket says 172.21.15.53:49082) 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 -- 172.21.15.53:0/779136349 learned_addr learned my addr 172.21.15.53:0/779136349 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 -- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04142770 msgr2=0x7f1c04142bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04142770 0x7f1c04142bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 -- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c04159970 msgr2=0x7f1c04159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c04159970 0x7f1c04159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 -- 172.21.15.53:0/779136349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0415a330 con 0x7f1be4003ef0 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.257+0000 7f1c03e00640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1c0413b5e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40108e0 tx=0x7f1bf4010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.258+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf400eba0 con 0x7f1be4003ef0 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.258+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bf400ed00 con 0x7f1be4003ef0 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.258+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf401c600 con 0x7f1be4003ef0 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.258+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c041f40d0 con 0x7f1be4003ef0 2024-01-16T15:00:01.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.258+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c041f4690 con 0x7f1be4003ef0 2024-01-16T15:00:01.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.259+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c04067450 con 0x7f1be4003ef0 2024-01-16T15:00:01.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.260+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1bf401c760 con 0x7f1be4003ef0 2024-01-16T15:00:01.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.261+0000 7f1c00c00640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bdc06b2c0 0x7f1bdc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.261+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1bf408a0a0 con 0x7f1be4003ef0 2024-01-16T15:00:01.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.261+0000 7f1bfbe00640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bdc06b2c0 0x7f1bdc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.263+0000 7f1bfbe00640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bdc06b2c0 0x7f1bdc06d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1bec034000 tx=0x7f1bec02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.265+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1bf4055050 con 0x7f1be4003ef0 2024-01-16T15:00:01.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.500+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c0415ae40 con 0x7f1be4003ef0 2024-01-16T15:00:01.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.501+0000 7f1c00c00640 1 -- 172.21.15.53:0/779136349 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f1bf4058670 con 0x7f1be4003ef0 2024-01-16T15:00:01.483 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:01.483 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:01.483 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bdc06b2c0 msgr2=0x7f1bdc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bdc06b2c0 0x7f1bdc06d780 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1bec034000 tx=0x7f1bec02e000 comp rx=0 tx=0).stop 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 msgr2=0x7f1c0413b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1c0413b5e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1bf40108e0 tx=0x7f1bf4010da0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 shutdown_connections 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bdc06b2c0 0x7f1bdc06d780 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c04142770 0x7f1c04142bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c04159970 0x7f1c04159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 --2- 172.21.15.53:0/779136349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1be4003ef0 0x7f1c0413b5e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 >> 172.21.15.53:0/779136349 conn(0x7f1c0406a800 msgr2=0x7f1c0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.504+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 shutdown_connections 2024-01-16T15:00:01.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.505+0000 7f1c09600640 1 -- 172.21.15.53:0/779136349 wait complete. 2024-01-16T15:00:01.506 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:01.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.747+0000 7f19df200640 1 Processor -- start 2024-01-16T15:00:01.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.748+0000 7f19df200640 1 -- start start 2024-01-16T15:00:01.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.749+0000 7f19df200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19e015d440 0x7f19e00d1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19df200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e00cfd00 con 0x7f19e006c220 2024-01-16T15:00:01.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19df200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e00cffe0 con 0x7f19e006b700 2024-01-16T15:00:01.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19df200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e00cfaf0 con 0x7f19e015d440 2024-01-16T15:00:01.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19de800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f19e006c220 0x7f19e00cb950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42420/0 (socket says 172.21.15.53:42420) 2024-01-16T15:00:01.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19de800640 1 -- 172.21.15.53:0/983988992 learned_addr learned my addr 172.21.15.53:0/983988992 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:01.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19dde00640 1 --2- 172.21.15.53:0/983988992 >> v2:172.21.15.146:3300/0 conn(0x7f19e015d440 0x7f19e00d1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632440905 0 0) 0x7f19e00cfd00 con 0x7f19e006c220 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.750+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19c0003520 con 0x7f19e006c220 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 632188250 0 0) 0x7f19c0003520 con 0x7f19e006c220 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 >> v2:172.21.15.146:3300/0 conn(0x7f19e015d440 msgr2=0x7f19e00d1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 --2- 172.21.15.53:0/983988992 >> v2:172.21.15.146:3300/0 conn(0x7f19e015d440 0x7f19e00d1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 >> v1:172.21.15.146:6789/0 conn(0x7f19e006b700 legacy=0x7f19e006bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00d02c0 con 0x7f19e006c220 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f19d8001670 con 0x7f19e006c220 2024-01-16T15:00:01.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 >> v1:172.21.15.53:6789/0 conn(0x7f19e006c220 legacy=0x7f19e00cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c00039c0 0x7f19c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c00047c0 0x7f19c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c0005180 0x7f19c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00cfaf0 con 0x7f19c0005180 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c000c080 con 0x7f19c00039c0 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c000c240 con 0x7f19c00047c0 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19de800640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c0005180 0x7f19c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dde00640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c00039c0 0x7f19c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.751+0000 7f19dd400640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c00047c0 0x7f19c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19de800640 1 -- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c00047c0 msgr2=0x7f19c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19de800640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c00047c0 0x7f19c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19de800640 1 -- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c00039c0 msgr2=0x7f19c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:01.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19de800640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19c00039c0 0x7f19c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19de800640 1 -- 172.21.15.53:0/983988992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c0003520 con 0x7f19c0005180 2024-01-16T15:00:01.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19dd400640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c00047c0 0x7f19c0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.752+0000 7f19de800640 1 -- 172.21.15.53:0/983988992 reap_dead start 2024-01-16T15:00:01.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.753+0000 7f19de800640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c0005180 0x7f19c000b980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c0e0 tx=0x7f19c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd9e5ec4da4a0a3b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.753+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8005ad0 con 0x7f19c0005180 2024-01-16T15:00:01.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.753+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8005c70 con 0x7f19c0005180 2024-01-16T15:00:01.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.753+0000 7f19dca00640 1 -- 172.21.15.53:0/983988992 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d802c7f0 con 0x7f19c0005180 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 -- 172.21.15.53:0/983988992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c0005180 msgr2=0x7f19c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c0005180 0x7f19c000b980 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c0e0 tx=0x7f19c00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 -- 172.21.15.53:0/983988992 shutdown_connections 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 --2- 172.21.15.53:0/983988992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19c0005180 0x7f19c000b980 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 -- 172.21.15.53:0/983988992 >> 172.21.15.53:0/983988992 conn(0x7f19e00d0de0 msgr2=0x7f19e00d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 -- 172.21.15.53:0/983988992 shutdown_connections 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.754+0000 7f19df200640 1 -- 172.21.15.53:0/983988992 wait complete. 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.756+0000 7f19df200640 1 Processor -- start 2024-01-16T15:00:01.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.756+0000 7f19df200640 1 -- start start 2024-01-16T15:00:01.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19df200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c0005180 0x7f19e015b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19df200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 0x7f19e00b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19df200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19e00b5d90 0x7f19e00ca080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19df200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00c6ee0 con 0x7f19e00c7300 2024-01-16T15:00:01.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19df200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00b6340 con 0x7f19e00b5d90 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19df200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e00b6620 con 0x7f19c0005180 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19dd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 0x7f19e00b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19dd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 0x7f19e00b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49100/0 (socket says 172.21.15.53:49100) 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.757+0000 7f19dd400640 1 -- 172.21.15.53:0/1724201831 learned_addr learned my addr 172.21.15.53:0/1724201831 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dde00640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c0005180 0x7f19e015b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19de800640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19e00b5d90 0x7f19e00ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dd400640 1 -- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c0005180 msgr2=0x7f19e015b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dd400640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c0005180 0x7f19e015b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dd400640 1 -- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19e00b5d90 msgr2=0x7f19e00ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dd400640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19e00b5d90 0x7f19e00ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dd400640 1 -- 172.21.15.53:0/1724201831 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00c6b00 con 0x7f19e00c7300 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dde00640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c0005180 0x7f19e015b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19de800640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19e00b5d90 0x7f19e00ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19dd400640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 0x7f19e00b5850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f19c800db10 tx=0x7f19c80020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c8003da0 con 0x7f19e00c7300 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c8006b50 con 0x7f19e00c7300 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19c800b740 con 0x7f19e00c7300 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.758+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e00c7780 con 0x7f19e00c7300 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.759+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e00ca5c0 con 0x7f19e00c7300 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.760+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e00c7020 con 0x7f19e00c7300 2024-01-16T15:00:01.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.761+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f19c800b8a0 con 0x7f19e00c7300 2024-01-16T15:00:01.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.762+0000 7f19d6000640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19b806b2f0 0x7f19b806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:01.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.762+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f19c80890d0 con 0x7f19e00c7300 2024-01-16T15:00:01.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.762+0000 7f19dde00640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19b806b2f0 0x7f19b806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:01.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.764+0000 7f19dde00640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19b806b2f0 0x7f19b806d7b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009970 tx=0x7f19d0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:01.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.766+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19e00c7020 con 0x7f19e00c7300 2024-01-16T15:00:01.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.964+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19e00c7020 con 0x7f19e00c7300 2024-01-16T15:00:01.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.966+0000 7f19d6000640 1 -- 172.21.15.53:0/1724201831 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f19e00c7020 con 0x7f19e00c7300 2024-01-16T15:00:01.948 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:01.948 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:01.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19b806b2f0 msgr2=0x7f19b806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19b806b2f0 0x7f19b806d7b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f19d0009970 tx=0x7f19d0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 msgr2=0x7f19e00b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 0x7f19e00b5850 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f19c800db10 tx=0x7f19c80020c0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 shutdown_connections 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19b806b2f0 0x7f19b806d7b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19e00b5d90 0x7f19e00ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19e00c7300 0x7f19e00b5850 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 --2- 172.21.15.53:0/1724201831 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19c0005180 0x7f19e015b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 >> 172.21.15.53:0/1724201831 conn(0x7f19e00d0de0 msgr2=0x7f19e00cb7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 shutdown_connections 2024-01-16T15:00:01.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:01.969+0000 7f19df200640 1 -- 172.21.15.53:0/1724201831 wait complete. 2024-01-16T15:00:01.970 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:00:01.970 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:02.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.206+0000 7f7727e00640 1 Processor -- start 2024-01-16T15:00:02.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.206+0000 7f7727e00640 1 -- start start 2024-01-16T15:00:02.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.208+0000 7f7727e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f77280b3f30 0x7f77280d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.208+0000 7f7727e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280d0a10 con 0x7f7728160920 2024-01-16T15:00:02.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.208+0000 7f7727e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77280cf080 con 0x7f77280d0100 2024-01-16T15:00:02.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.208+0000 7f7727e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77280c6be0 con 0x7f77280b3f30 2024-01-16T15:00:02.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.208+0000 7f7726a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7728160920 0x7f77280cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49672/0 (socket says 172.21.15.53:49672) 2024-01-16T15:00:02.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.208+0000 7f7726a00640 1 -- 172.21.15.53:0/3406684470 learned_addr learned my addr 172.21.15.53:0/3406684470 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:02.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7727400640 1 --2- 172.21.15.53:0/3406684470 >> v2:172.21.15.146:3300/0 conn(0x7f77280b3f30 0x7f77280d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482020776 0 0) 0x7f77280d0a10 con 0x7f7728160920 2024-01-16T15:00:02.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7708003520 con 0x7f7728160920 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949593918 0 0) 0x7f77280cf080 con 0x7f77280d0100 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77280d0a10 con 0x7f77280d0100 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781950978 0 0) 0x7f7708003520 con 0x7f7728160920 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.209+0000 7f7727400640 1 -- 172.21.15.53:0/3406684470 >> v1:172.21.15.146:6789/0 conn(0x7f77280d0100 legacy=0x7f77280d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.210+0000 7f7727400640 1 -- 172.21.15.53:0/3406684470 >> v1:172.21.15.53:6789/0 conn(0x7f7728160920 legacy=0x7f77280cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.210+0000 7f7727400640 1 -- 172.21.15.53:0/3406684470 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77280ce7e0 con 0x7f77280b3f30 2024-01-16T15:00:02.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.211+0000 7f7727400640 1 --2- 172.21.15.53:0/3406684470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280b3f30 0x7f77280d1650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7720009d90 tx=0x7f772002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4421672a49739c9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.212+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718001670 con 0x7f77280b3f30 2024-01-16T15:00:02.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.212+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772002c940 con 0x7f77280b3f30 2024-01-16T15:00:02.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.212+0000 7f7726000640 1 -- 172.21.15.53:0/3406684470 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772002ccd0 con 0x7f77280b3f30 2024-01-16T15:00:02.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 -- 172.21.15.53:0/3406684470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280b3f30 msgr2=0x7f77280d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 --2- 172.21.15.53:0/3406684470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280b3f30 0x7f77280d1650 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7720009d90 tx=0x7f772002be10 comp rx=0 tx=0).stop 2024-01-16T15:00:02.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 -- 172.21.15.53:0/3406684470 shutdown_connections 2024-01-16T15:00:02.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 --2- 172.21.15.53:0/3406684470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280b3f30 0x7f77280d1650 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 -- 172.21.15.53:0/3406684470 >> 172.21.15.53:0/3406684470 conn(0x7f77280d3090 msgr2=0x7f77280d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 -- 172.21.15.53:0/3406684470 shutdown_connections 2024-01-16T15:00:02.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.213+0000 7f7727e00640 1 -- 172.21.15.53:0/3406684470 wait complete. 2024-01-16T15:00:02.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.215+0000 7f7727e00640 1 Processor -- start 2024-01-16T15:00:02.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.215+0000 7f7727e00640 1 -- start start 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7727e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 0x7f772815b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7727e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0100 0x7f77281f4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7727e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7728160920 0x7f772806be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7727e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280b6e20 con 0x7f77280b3f30 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7727e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77280b7100 con 0x7f77280d0100 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7727e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77280b73e0 con 0x7f7728160920 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.216+0000 7f7726a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 0x7f772815b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 0x7f772815b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34740/0 (socket says 172.21.15.53:34740) 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 -- 172.21.15.53:0/3325903957 learned_addr learned my addr 172.21.15.53:0/3325903957 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f771de00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0100 0x7f77281f4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7727400640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7728160920 0x7f772806be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 -- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7728160920 msgr2=0x7f772806be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7728160920 0x7f772806be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 -- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0100 msgr2=0x7f77281f4000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0100 0x7f77281f4000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 -- 172.21.15.53:0/3325903957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7720009a70 con 0x7f77280b3f30 2024-01-16T15:00:02.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.217+0000 7f7726a00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 0x7f772815b160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f771800c750 tx=0x7f771800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.218+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718005ad0 con 0x7f77280b3f30 2024-01-16T15:00:02.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.218+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7718005c30 con 0x7f77280b3f30 2024-01-16T15:00:02.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.218+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7718007070 con 0x7f77280b3f30 2024-01-16T15:00:02.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.218+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77280b76c0 con 0x7f77280b3f30 2024-01-16T15:00:02.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.218+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772806c3c0 con 0x7f77280b3f30 2024-01-16T15:00:02.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.219+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77281f2df0 con 0x7f77280b3f30 2024-01-16T15:00:02.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.220+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f77180071d0 con 0x7f77280b3f30 2024-01-16T15:00:02.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.221+0000 7f771f400640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f770006b310 0x7f770006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.221+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7718089860 con 0x7f77280b3f30 2024-01-16T15:00:02.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.222+0000 7f771de00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f770006b310 0x7f770006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.223+0000 7f771de00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f770006b310 0x7f770006d7d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f772815b6a0 tx=0x7f7710009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.226+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7718054740 con 0x7f77280b3f30 2024-01-16T15:00:02.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.418+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f77280b7270 con 0x7f770006b310 2024-01-16T15:00:02.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.442+0000 7f771f400640 1 -- 172.21.15.53:0/3325903957 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264419 (secure 0 0 0) 0x7f77280b7270 con 0x7f770006b310 2024-01-16T15:00:02.428 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:02.429 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f770006b310 msgr2=0x7f770006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f770006b310 0x7f770006d7d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f772815b6a0 tx=0x7f7710009040 comp rx=0 tx=0).stop 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 msgr2=0x7f772815b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 0x7f772815b160 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f771800c750 tx=0x7f771800cc10 comp rx=0 tx=0).stop 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 shutdown_connections 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f770006b310 0x7f770006d7d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7728160920 0x7f772806be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77280d0100 0x7f77281f4000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 --2- 172.21.15.53:0/3325903957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77280b3f30 0x7f772815b160 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 >> 172.21.15.53:0/3325903957 conn(0x7f77280d3090 msgr2=0x7f772815d330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 shutdown_connections 2024-01-16T15:00:02.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.448+0000 7f7727e00640 1 -- 172.21.15.53:0/3325903957 wait complete. 2024-01-16T15:00:02.448 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T15:00:02.170734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235776,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738944,"statfs":{"total":773094113280,"available":772852678656,"internally_reserved":0,"allocated":3022848,"data_stored":1810900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001892"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882738+0000","last_change":"2024-01-16T14:59:57.913241+0000","last_active":"2024-01-16T14:59:58.882738+0000","last_peered":"2024-01-16T14:59:58.882738+0000","last_clean":"2024-01-16T14:59:58.882738+0000","last_became_active":"2024-01-16T14:59:57.912720+0000","last_became_peered":"2024-01-16T14:59:57.912720+0000","last_unstale":"2024-01-16T14:59:58.882738+0000","last_undegraded":"2024-01-16T14:59:58.882738+0000","last_fullsized":"2024-01-16T14:59:58.882738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964756+0000","last_change":"2024-01-16T14:59:57.910357+0000","last_active":"2024-01-16T14:59:58.964756+0000","last_peered":"2024-01-16T14:59:58.964756+0000","last_clean":"2024-01-16T14:59:58.964756+0000","last_became_active":"2024-01-16T14:59:57.909933+0000","last_became_peered":"2024-01-16T14:59:57.909933+0000","last_unstale":"2024-01-16T14:59:58.964756+0000","last_undegraded":"2024-01-16T14:59:58.964756+0000","last_fullsized":"2024-01-16T14:59:58.964756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974477+0000","last_change":"2024-01-16T14:59:57.909709+0000","last_active":"2024-01-16T14:59:58.974477+0000","last_peered":"2024-01-16T14:59:58.974477+0000","last_clean":"2024-01-16T14:59:58.974477+0000","last_became_active":"2024-01-16T14:59:57.909405+0000","last_became_peered":"2024-01-16T14:59:57.909405+0000","last_unstale":"2024-01-16T14:59:58.974477+0000","last_undegraded":"2024-01-16T14:59:58.974477+0000","last_fullsized":"2024-01-16T14:59:58.974477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:51.411241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975424+0000","last_change":"2024-01-16T14:59:57.920032+0000","last_active":"2024-01-16T14:59:58.975424+0000","last_peered":"2024-01-16T14:59:58.975424+0000","last_clean":"2024-01-16T14:59:58.975424+0000","last_became_active":"2024-01-16T14:59:57.919630+0000","last_became_peered":"2024-01-16T14:59:57.919630+0000","last_unstale":"2024-01-16T14:59:58.975424+0000","last_undegraded":"2024-01-16T14:59:58.975424+0000","last_fullsized":"2024-01-16T14:59:58.975424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882328+0000","last_change":"2024-01-16T14:59:57.917909+0000","last_active":"2024-01-16T14:59:58.882328+0000","last_peered":"2024-01-16T14:59:58.882328+0000","last_clean":"2024-01-16T14:59:58.882328+0000","last_became_active":"2024-01-16T14:59:57.914603+0000","last_became_peered":"2024-01-16T14:59:57.914603+0000","last_unstale":"2024-01-16T14:59:58.882328+0000","last_undegraded":"2024-01-16T14:59:58.882328+0000","last_fullsized":"2024-01-16T14:59:58.882328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976981+0000","last_change":"2024-01-16T14:59:57.909424+0000","last_active":"2024-01-16T14:59:58.976981+0000","last_peered":"2024-01-16T14:59:58.976981+0000","last_clean":"2024-01-16T14:59:58.976981+0000","last_became_active":"2024-01-16T14:59:57.909160+0000","last_became_peered":"2024-01-16T14:59:57.909160+0000","last_unstale":"2024-01-16T14:59:58.976981+0000","last_undegraded":"2024-01-16T14:59:58.976981+0000","last_fullsized":"2024-01-16T14:59:58.976981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880812+0000","last_change":"2024-01-16T14:59:57.917053+0000","last_active":"2024-01-16T14:59:58.880812+0000","last_peered":"2024-01-16T14:59:58.880812+0000","last_clean":"2024-01-16T14:59:58.880812+0000","last_became_active":"2024-01-16T14:59:57.914775+0000","last_became_peered":"2024-01-16T14:59:57.914775+0000","last_unstale":"2024-01-16T14:59:58.880812+0000","last_undegraded":"2024-01-16T14:59:58.880812+0000","last_fullsized":"2024-01-16T14:59:58.880812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976696+0000","last_change":"2024-01-16T14:59:57.909184+0000","last_active":"2024-01-16T14:59:58.976696+0000","last_peered":"2024-01-16T14:59:58.976696+0000","last_clean":"2024-01-16T14:59:58.976696+0000","last_became_active":"2024-01-16T14:59:57.908905+0000","last_became_peered":"2024-01-16T14:59:57.908905+0000","last_unstale":"2024-01-16T14:59:58.976696+0000","last_undegraded":"2024-01-16T14:59:58.976696+0000","last_fullsized":"2024-01-16T14:59:58.976696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883086+0000","last_change":"2024-01-16T14:59:57.921823+0000","last_active":"2024-01-16T14:59:58.883086+0000","last_peered":"2024-01-16T14:59:58.883086+0000","last_clean":"2024-01-16T14:59:58.883086+0000","last_became_active":"2024-01-16T14:59:57.921476+0000","last_became_peered":"2024-01-16T14:59:57.921476+0000","last_unstale":"2024-01-16T14:59:58.883086+0000","last_undegraded":"2024-01-16T14:59:58.883086+0000","last_fullsized":"2024-01-16T14:59:58.883086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975563+0000","last_change":"2024-01-16T14:59:57.909965+0000","last_active":"2024-01-16T14:59:58.975563+0000","last_peered":"2024-01-16T14:59:58.975563+0000","last_clean":"2024-01-16T14:59:58.975563+0000","last_became_active":"2024-01-16T14:59:57.909707+0000","last_became_peered":"2024-01-16T14:59:57.909707+0000","last_unstale":"2024-01-16T14:59:58.975563+0000","last_undegraded":"2024-01-16T14:59:58.975563+0000","last_fullsized":"2024-01-16T14:59:58.975563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975772+0000","last_change":"2024-01-16T14:59:57.908484+0000","last_active":"2024-01-16T14:59:58.975772+0000","last_peered":"2024-01-16T14:59:58.975772+0000","last_clean":"2024-01-16T14:59:58.975772+0000","last_became_active":"2024-01-16T14:59:57.908164+0000","last_became_peered":"2024-01-16T14:59:57.908164+0000","last_unstale":"2024-01-16T14:59:58.975772+0000","last_undegraded":"2024-01-16T14:59:58.975772+0000","last_fullsized":"2024-01-16T14:59:58.975772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885578+0000","last_change":"2024-01-16T14:59:57.914372+0000","last_active":"2024-01-16T14:59:58.885578+0000","last_peered":"2024-01-16T14:59:58.885578+0000","last_clean":"2024-01-16T14:59:58.885578+0000","last_became_active":"2024-01-16T14:59:57.913918+0000","last_became_peered":"2024-01-16T14:59:57.913918+0000","last_unstale":"2024-01-16T14:59:58.885578+0000","last_undegraded":"2024-01-16T14:59:58.885578+0000","last_fullsized":"2024-01-16T14:59:58.885578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:08.622882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977380+0000","last_change":"2024-01-16T14:59:57.907940+0000","last_active":"2024-01-16T14:59:58.977380+0000","last_peered":"2024-01-16T14:59:58.977380+0000","last_clean":"2024-01-16T14:59:58.977380+0000","last_became_active":"2024-01-16T14:59:57.906517+0000","last_became_peered":"2024-01-16T14:59:57.906517+0000","last_unstale":"2024-01-16T14:59:58.977380+0000","last_undegraded":"2024-01-16T14:59:58.977380+0000","last_fullsized":"2024-01-16T14:59:58.977380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974085+0000","last_change":"2024-01-16T14:59:57.907053+0000","last_active":"2024-01-16T14:59:58.974085+0000","last_peered":"2024-01-16T14:59:58.974085+0000","last_clean":"2024-01-16T14:59:58.974085+0000","last_became_active":"2024-01-16T14:59:57.906443+0000","last_became_peered":"2024-01-16T14:59:57.906443+0000","last_unstale":"2024-01-16T14:59:58.974085+0000","last_undegraded":"2024-01-16T14:59:58.974085+0000","last_fullsized":"2024-01-16T14:59:58.974085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:01.194359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964073+0000","last_change":"2024-01-16T14:59:57.893114+0000","last_active":"2024-01-16T14:59:58.964073+0000","last_peered":"2024-01-16T14:59:58.964073+0000","last_clean":"2024-01-16T14:59:58.964073+0000","last_became_active":"2024-01-16T14:59:57.891027+0000","last_became_peered":"2024-01-16T14:59:57.891027+0000","last_unstale":"2024-01-16T14:59:58.964073+0000","last_undegraded":"2024-01-16T14:59:58.964073+0000","last_fullsized":"2024-01-16T14:59:58.964073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974627+0000","last_change":"2024-01-16T14:59:57.911794+0000","last_active":"2024-01-16T14:59:58.974627+0000","last_peered":"2024-01-16T14:59:58.974627+0000","last_clean":"2024-01-16T14:59:58.974627+0000","last_became_active":"2024-01-16T14:59:57.911528+0000","last_became_peered":"2024-01-16T14:59:57.911528+0000","last_unstale":"2024-01-16T14:59:58.974627+0000","last_undegraded":"2024-01-16T14:59:58.974627+0000","last_fullsized":"2024-01-16T14:59:58.974627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976633+0000","last_change":"2024-01-16T14:59:57.917524+0000","last_active":"2024-01-16T14:59:58.976633+0000","last_peered":"2024-01-16T14:59:58.976633+0000","last_clean":"2024-01-16T14:59:58.976633+0000","last_became_active":"2024-01-16T14:59:57.916983+0000","last_became_peered":"2024-01-16T14:59:57.916983+0000","last_unstale":"2024-01-16T14:59:58.976633+0000","last_undegraded":"2024-01-16T14:59:58.976633+0000","last_fullsized":"2024-01-16T14:59:58.976633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974145+0000","last_change":"2024-01-16T14:59:57.914130+0000","last_active":"2024-01-16T14:59:58.974145+0000","last_peered":"2024-01-16T14:59:58.974145+0000","last_clean":"2024-01-16T14:59:58.974145+0000","last_became_active":"2024-01-16T14:59:57.913722+0000","last_became_peered":"2024-01-16T14:59:57.913722+0000","last_unstale":"2024-01-16T14:59:58.974145+0000","last_undegraded":"2024-01-16T14:59:58.974145+0000","last_fullsized":"2024-01-16T14:59:58.974145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:37.662076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882022+0000","last_change":"2024-01-16T14:59:57.914510+0000","last_active":"2024-01-16T14:59:58.882022+0000","last_peered":"2024-01-16T14:59:58.882022+0000","last_clean":"2024-01-16T14:59:58.882022+0000","last_became_active":"2024-01-16T14:59:57.912912+0000","last_became_peered":"2024-01-16T14:59:57.912912+0000","last_unstale":"2024-01-16T14:59:58.882022+0000","last_undegraded":"2024-01-16T14:59:58.882022+0000","last_fullsized":"2024-01-16T14:59:58.882022+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975057+0000","last_change":"2024-01-16T14:59:57.911759+0000","last_active":"2024-01-16T14:59:58.975057+0000","last_peered":"2024-01-16T14:59:58.975057+0000","last_clean":"2024-01-16T14:59:58.975057+0000","last_became_active":"2024-01-16T14:59:57.911474+0000","last_became_peered":"2024-01-16T14:59:57.911474+0000","last_unstale":"2024-01-16T14:59:58.975057+0000","last_undegraded":"2024-01-16T14:59:58.975057+0000","last_fullsized":"2024-01-16T14:59:58.975057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975453+0000","last_change":"2024-01-16T14:59:57.923664+0000","last_active":"2024-01-16T14:59:58.975453+0000","last_peered":"2024-01-16T14:59:58.975453+0000","last_clean":"2024-01-16T14:59:58.975453+0000","last_became_active":"2024-01-16T14:59:57.923402+0000","last_became_peered":"2024-01-16T14:59:57.923402+0000","last_unstale":"2024-01-16T14:59:58.975453+0000","last_undegraded":"2024-01-16T14:59:58.975453+0000","last_fullsized":"2024-01-16T14:59:58.975453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977267+0000","last_change":"2024-01-16T14:59:57.923807+0000","last_active":"2024-01-16T14:59:58.977267+0000","last_peered":"2024-01-16T14:59:58.977267+0000","last_clean":"2024-01-16T14:59:58.977267+0000","last_became_active":"2024-01-16T14:59:57.923493+0000","last_became_peered":"2024-01-16T14:59:57.923493+0000","last_unstale":"2024-01-16T14:59:58.977267+0000","last_undegraded":"2024-01-16T14:59:58.977267+0000","last_fullsized":"2024-01-16T14:59:58.977267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977086+0000","last_change":"2024-01-16T14:59:57.912156+0000","last_active":"2024-01-16T14:59:58.977086+0000","last_peered":"2024-01-16T14:59:58.977086+0000","last_clean":"2024-01-16T14:59:58.977086+0000","last_became_active":"2024-01-16T14:59:57.911497+0000","last_became_peered":"2024-01-16T14:59:57.911497+0000","last_unstale":"2024-01-16T14:59:58.977086+0000","last_undegraded":"2024-01-16T14:59:58.977086+0000","last_fullsized":"2024-01-16T14:59:58.977086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974975+0000","last_change":"2024-01-16T14:59:57.918668+0000","last_active":"2024-01-16T14:59:58.974975+0000","last_peered":"2024-01-16T14:59:58.974975+0000","last_clean":"2024-01-16T14:59:58.974975+0000","last_became_active":"2024-01-16T14:59:57.918314+0000","last_became_peered":"2024-01-16T14:59:57.918314+0000","last_unstale":"2024-01-16T14:59:58.974975+0000","last_undegraded":"2024-01-16T14:59:58.974975+0000","last_fullsized":"2024-01-16T14:59:58.974975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978927+0000","last_change":"2024-01-16T14:59:57.912774+0000","last_active":"2024-01-16T14:59:58.978927+0000","last_peered":"2024-01-16T14:59:58.978927+0000","last_clean":"2024-01-16T14:59:58.978927+0000","last_became_active":"2024-01-16T14:59:57.910934+0000","last_became_peered":"2024-01-16T14:59:57.910934+0000","last_unstale":"2024-01-16T14:59:58.978927+0000","last_undegraded":"2024-01-16T14:59:58.978927+0000","last_fullsized":"2024-01-16T14:59:58.978927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:47.822358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975717+0000","last_change":"2024-01-16T14:59:57.921118+0000","last_active":"2024-01-16T14:59:58.975717+0000","last_peered":"2024-01-16T14:59:58.975717+0000","last_clean":"2024-01-16T14:59:58.975717+0000","last_became_active":"2024-01-16T14:59:57.920835+0000","last_became_peered":"2024-01-16T14:59:57.920835+0000","last_unstale":"2024-01-16T14:59:58.975717+0000","last_undegraded":"2024-01-16T14:59:58.975717+0000","last_fullsized":"2024-01-16T14:59:58.975717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964405+0000","last_change":"2024-01-16T14:59:57.898548+0000","last_active":"2024-01-16T14:59:58.964405+0000","last_peered":"2024-01-16T14:59:58.964405+0000","last_clean":"2024-01-16T14:59:58.964405+0000","last_became_active":"2024-01-16T14:59:57.897874+0000","last_became_peered":"2024-01-16T14:59:57.897874+0000","last_unstale":"2024-01-16T14:59:58.964405+0000","last_undegraded":"2024-01-16T14:59:58.964405+0000","last_fullsized":"2024-01-16T14:59:58.964405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978167+0000","last_change":"2024-01-16T14:59:57.914061+0000","last_active":"2024-01-16T14:59:58.978167+0000","last_peered":"2024-01-16T14:59:58.978167+0000","last_clean":"2024-01-16T14:59:58.978167+0000","last_became_active":"2024-01-16T14:59:57.913718+0000","last_became_peered":"2024-01-16T14:59:57.913718+0000","last_unstale":"2024-01-16T14:59:58.978167+0000","last_undegraded":"2024-01-16T14:59:58.978167+0000","last_fullsized":"2024-01-16T14:59:58.978167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:46.106265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974760+0000","last_change":"2024-01-16T14:59:57.917278+0000","last_active":"2024-01-16T14:59:58.974760+0000","last_peered":"2024-01-16T14:59:58.974760+0000","last_clean":"2024-01-16T14:59:58.974760+0000","last_became_active":"2024-01-16T14:59:57.916768+0000","last_became_peered":"2024-01-16T14:59:57.916768+0000","last_unstale":"2024-01-16T14:59:58.974760+0000","last_undegraded":"2024-01-16T14:59:58.974760+0000","last_fullsized":"2024-01-16T14:59:58.974760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:12.721250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977355+0000","last_change":"2024-01-16T14:59:57.915403+0000","last_active":"2024-01-16T14:59:58.977355+0000","last_peered":"2024-01-16T14:59:58.977355+0000","last_clean":"2024-01-16T14:59:58.977355+0000","last_became_active":"2024-01-16T14:59:57.915134+0000","last_became_peered":"2024-01-16T14:59:57.915134+0000","last_unstale":"2024-01-16T14:59:58.977355+0000","last_undegraded":"2024-01-16T14:59:58.977355+0000","last_fullsized":"2024-01-16T14:59:58.977355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973385+0000","last_change":"2024-01-16T14:59:57.910767+0000","last_active":"2024-01-16T14:59:58.973385+0000","last_peered":"2024-01-16T14:59:58.973385+0000","last_clean":"2024-01-16T14:59:58.973385+0000","last_became_active":"2024-01-16T14:59:57.908877+0000","last_became_peered":"2024-01-16T14:59:57.908877+0000","last_unstale":"2024-01-16T14:59:58.973385+0000","last_undegraded":"2024-01-16T14:59:58.973385+0000","last_fullsized":"2024-01-16T14:59:58.973385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:58.524840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974275+0000","last_change":"2024-01-16T14:59:57.912035+0000","last_active":"2024-01-16T14:59:58.974275+0000","last_peered":"2024-01-16T14:59:58.974275+0000","last_clean":"2024-01-16T14:59:58.974275+0000","last_became_active":"2024-01-16T14:59:57.911641+0000","last_became_peered":"2024-01-16T14:59:57.911641+0000","last_unstale":"2024-01-16T14:59:58.974275+0000","last_undegraded":"2024-01-16T14:59:58.974275+0000","last_fullsized":"2024-01-16T14:59:58.974275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884477+0000","last_change":"2024-01-16T14:59:57.898975+0000","last_active":"2024-01-16T14:59:58.884477+0000","last_peered":"2024-01-16T14:59:58.884477+0000","last_clean":"2024-01-16T14:59:58.884477+0000","last_became_active":"2024-01-16T14:59:57.894960+0000","last_became_peered":"2024-01-16T14:59:57.894960+0000","last_unstale":"2024-01-16T14:59:58.884477+0000","last_undegraded":"2024-01-16T14:59:58.884477+0000","last_fullsized":"2024-01-16T14:59:58.884477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:50.235874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977646+0000","last_change":"2024-01-16T14:59:56.961128+0000","last_active":"2024-01-16T14:59:58.977646+0000","last_peered":"2024-01-16T14:59:58.977646+0000","last_clean":"2024-01-16T14:59:58.977646+0000","last_became_active":"2024-01-16T14:59:56.960829+0000","last_became_peered":"2024-01-16T14:59:56.960829+0000","last_unstale":"2024-01-16T14:59:58.977646+0000","last_undegraded":"2024-01-16T14:59:58.977646+0000","last_fullsized":"2024-01-16T14:59:58.977646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975040+0000","last_change":"2024-01-16T14:59:57.907318+0000","last_active":"2024-01-16T14:59:58.975040+0000","last_peered":"2024-01-16T14:59:58.975040+0000","last_clean":"2024-01-16T14:59:58.975040+0000","last_became_active":"2024-01-16T14:59:57.907025+0000","last_became_peered":"2024-01-16T14:59:57.907025+0000","last_unstale":"2024-01-16T14:59:58.975040+0000","last_undegraded":"2024-01-16T14:59:58.975040+0000","last_fullsized":"2024-01-16T14:59:58.975040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.937959+0000","last_change":"2024-01-16T14:59:56.968922+0000","last_active":"2024-01-16T14:59:57.937959+0000","last_peered":"2024-01-16T14:59:57.937959+0000","last_clean":"2024-01-16T14:59:57.937959+0000","last_became_active":"2024-01-16T14:59:56.968639+0000","last_became_peered":"2024-01-16T14:59:56.968639+0000","last_unstale":"2024-01-16T14:59:57.937959+0000","last_undegraded":"2024-01-16T14:59:57.937959+0000","last_fullsized":"2024-01-16T14:59:57.937959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:29.672100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977060+0000","last_change":"2024-01-16T14:59:56.966987+0000","last_active":"2024-01-16T14:59:58.977060+0000","last_peered":"2024-01-16T14:59:58.977060+0000","last_clean":"2024-01-16T14:59:58.977060+0000","last_became_active":"2024-01-16T14:59:56.966640+0000","last_became_peered":"2024-01-16T14:59:56.966640+0000","last_unstale":"2024-01-16T14:59:58.977060+0000","last_undegraded":"2024-01-16T14:59:58.977060+0000","last_fullsized":"2024-01-16T14:59:58.977060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:19.845423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976503+0000","last_change":"2024-01-16T14:59:57.914604+0000","last_active":"2024-01-16T14:59:58.976503+0000","last_peered":"2024-01-16T14:59:58.976503+0000","last_clean":"2024-01-16T14:59:58.976503+0000","last_became_active":"2024-01-16T14:59:57.914326+0000","last_became_peered":"2024-01-16T14:59:57.914326+0000","last_unstale":"2024-01-16T14:59:58.976503+0000","last_undegraded":"2024-01-16T14:59:58.976503+0000","last_fullsized":"2024-01-16T14:59:58.976503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975252+0000","last_change":"2024-01-16T14:59:56.967597+0000","last_active":"2024-01-16T14:59:58.975252+0000","last_peered":"2024-01-16T14:59:58.975252+0000","last_clean":"2024-01-16T14:59:58.975252+0000","last_became_active":"2024-01-16T14:59:56.967030+0000","last_became_peered":"2024-01-16T14:59:56.967030+0000","last_unstale":"2024-01-16T14:59:58.975252+0000","last_undegraded":"2024-01-16T14:59:58.975252+0000","last_fullsized":"2024-01-16T14:59:58.975252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964053+0000","last_change":"2024-01-16T14:59:57.905418+0000","last_active":"2024-01-16T14:59:58.964053+0000","last_peered":"2024-01-16T14:59:58.964053+0000","last_clean":"2024-01-16T14:59:58.964053+0000","last_became_active":"2024-01-16T14:59:57.905057+0000","last_became_peered":"2024-01-16T14:59:57.905057+0000","last_unstale":"2024-01-16T14:59:58.964053+0000","last_undegraded":"2024-01-16T14:59:58.964053+0000","last_fullsized":"2024-01-16T14:59:58.964053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.903408+0000","last_change":"2024-01-16T14:59:56.966647+0000","last_active":"2024-01-16T14:59:57.903408+0000","last_peered":"2024-01-16T14:59:57.903408+0000","last_clean":"2024-01-16T14:59:57.903408+0000","last_became_active":"2024-01-16T14:59:56.966276+0000","last_became_peered":"2024-01-16T14:59:56.966276+0000","last_unstale":"2024-01-16T14:59:57.903408+0000","last_undegraded":"2024-01-16T14:59:57.903408+0000","last_fullsized":"2024-01-16T14:59:57.903408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:05.820349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976343+0000","last_change":"2024-01-16T14:59:57.923839+0000","last_active":"2024-01-16T14:59:58.976343+0000","last_peered":"2024-01-16T14:59:58.976343+0000","last_clean":"2024-01-16T14:59:58.976343+0000","last_became_active":"2024-01-16T14:59:57.923483+0000","last_became_peered":"2024-01-16T14:59:57.923483+0000","last_unstale":"2024-01-16T14:59:58.976343+0000","last_undegraded":"2024-01-16T14:59:58.976343+0000","last_fullsized":"2024-01-16T14:59:58.976343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976571+0000","last_change":"2024-01-16T14:59:56.969754+0000","last_active":"2024-01-16T14:59:58.976571+0000","last_peered":"2024-01-16T14:59:58.976571+0000","last_clean":"2024-01-16T14:59:58.976571+0000","last_became_active":"2024-01-16T14:59:56.969381+0000","last_became_peered":"2024-01-16T14:59:56.969381+0000","last_unstale":"2024-01-16T14:59:58.976571+0000","last_undegraded":"2024-01-16T14:59:58.976571+0000","last_fullsized":"2024-01-16T14:59:58.976571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974983+0000","last_change":"2024-01-16T14:59:57.914892+0000","last_active":"2024-01-16T14:59:58.974983+0000","last_peered":"2024-01-16T14:59:58.974983+0000","last_clean":"2024-01-16T14:59:58.974983+0000","last_became_active":"2024-01-16T14:59:57.913384+0000","last_became_peered":"2024-01-16T14:59:57.913384+0000","last_unstale":"2024-01-16T14:59:58.974983+0000","last_undegraded":"2024-01-16T14:59:58.974983+0000","last_fullsized":"2024-01-16T14:59:58.974983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.998301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964546+0000","last_change":"2024-01-16T14:59:56.960948+0000","last_active":"2024-01-16T14:59:58.964546+0000","last_peered":"2024-01-16T14:59:58.964546+0000","last_clean":"2024-01-16T14:59:58.964546+0000","last_became_active":"2024-01-16T14:59:56.960588+0000","last_became_peered":"2024-01-16T14:59:56.960588+0000","last_unstale":"2024-01-16T14:59:58.964546+0000","last_undegraded":"2024-01-16T14:59:58.964546+0000","last_fullsized":"2024-01-16T14:59:58.964546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885497+0000","last_change":"2024-01-16T14:59:57.911677+0000","last_active":"2024-01-16T14:59:58.885497+0000","last_peered":"2024-01-16T14:59:58.885497+0000","last_clean":"2024-01-16T14:59:58.885497+0000","last_became_active":"2024-01-16T14:59:57.911160+0000","last_became_peered":"2024-01-16T14:59:57.911160+0000","last_unstale":"2024-01-16T14:59:58.885497+0000","last_undegraded":"2024-01-16T14:59:58.885497+0000","last_fullsized":"2024-01-16T14:59:58.885497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:43.320590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.945676+0000","last_change":"2024-01-16T14:59:56.963872+0000","last_active":"2024-01-16T14:59:57.945676+0000","last_peered":"2024-01-16T14:59:57.945676+0000","last_clean":"2024-01-16T14:59:57.945676+0000","last_became_active":"2024-01-16T14:59:56.963545+0000","last_became_peered":"2024-01-16T14:59:56.963545+0000","last_unstale":"2024-01-16T14:59:57.945676+0000","last_undegraded":"2024-01-16T14:59:57.945676+0000","last_fullsized":"2024-01-16T14:59:57.945676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881456+0000","last_change":"2024-01-16T14:59:57.918001+0000","last_active":"2024-01-16T14:59:58.881456+0000","last_peered":"2024-01-16T14:59:58.881456+0000","last_clean":"2024-01-16T14:59:58.881456+0000","last_became_active":"2024-01-16T14:59:57.917646+0000","last_became_peered":"2024-01-16T14:59:57.917646+0000","last_unstale":"2024-01-16T14:59:58.881456+0000","last_undegraded":"2024-01-16T14:59:58.881456+0000","last_fullsized":"2024-01-16T14:59:58.881456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975183+0000","last_change":"2024-01-16T14:59:56.965453+0000","last_active":"2024-01-16T14:59:58.975183+0000","last_peered":"2024-01-16T14:59:58.975183+0000","last_clean":"2024-01-16T14:59:58.975183+0000","last_became_active":"2024-01-16T14:59:56.965153+0000","last_became_peered":"2024-01-16T14:59:56.965153+0000","last_unstale":"2024-01-16T14:59:58.975183+0000","last_undegraded":"2024-01-16T14:59:58.975183+0000","last_fullsized":"2024-01-16T14:59:58.975183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974193+0000","last_change":"2024-01-16T14:59:57.921295+0000","last_active":"2024-01-16T14:59:58.974193+0000","last_peered":"2024-01-16T14:59:58.974193+0000","last_clean":"2024-01-16T14:59:58.974193+0000","last_became_active":"2024-01-16T14:59:57.920961+0000","last_became_peered":"2024-01-16T14:59:57.920961+0000","last_unstale":"2024-01-16T14:59:58.974193+0000","last_undegraded":"2024-01-16T14:59:58.974193+0000","last_fullsized":"2024-01-16T14:59:58.974193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:09.224854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977499+0000","last_change":"2024-01-16T14:59:56.964539+0000","last_active":"2024-01-16T14:59:58.977499+0000","last_peered":"2024-01-16T14:59:58.977499+0000","last_clean":"2024-01-16T14:59:58.977499+0000","last_became_active":"2024-01-16T14:59:56.964337+0000","last_became_peered":"2024-01-16T14:59:56.964337+0000","last_unstale":"2024-01-16T14:59:58.977499+0000","last_undegraded":"2024-01-16T14:59:58.977499+0000","last_fullsized":"2024-01-16T14:59:58.977499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881748+0000","last_change":"2024-01-16T14:59:57.918457+0000","last_active":"2024-01-16T14:59:58.881748+0000","last_peered":"2024-01-16T14:59:58.881748+0000","last_clean":"2024-01-16T14:59:58.881748+0000","last_became_active":"2024-01-16T14:59:57.918278+0000","last_became_peered":"2024-01-16T14:59:57.918278+0000","last_unstale":"2024-01-16T14:59:58.881748+0000","last_undegraded":"2024-01-16T14:59:58.881748+0000","last_fullsized":"2024-01-16T14:59:58.881748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.900831+0000","last_change":"2024-01-16T14:59:56.969818+0000","last_active":"2024-01-16T14:59:57.900831+0000","last_peered":"2024-01-16T14:59:57.900831+0000","last_clean":"2024-01-16T14:59:57.900831+0000","last_became_active":"2024-01-16T14:59:56.969530+0000","last_became_peered":"2024-01-16T14:59:56.969530+0000","last_unstale":"2024-01-16T14:59:57.900831+0000","last_undegraded":"2024-01-16T14:59:57.900831+0000","last_fullsized":"2024-01-16T14:59:57.900831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.795006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974601+0000","last_change":"2024-01-16T14:59:57.907805+0000","last_active":"2024-01-16T14:59:58.974601+0000","last_peered":"2024-01-16T14:59:58.974601+0000","last_clean":"2024-01-16T14:59:58.974601+0000","last_became_active":"2024-01-16T14:59:57.907499+0000","last_became_peered":"2024-01-16T14:59:57.907499+0000","last_unstale":"2024-01-16T14:59:58.974601+0000","last_undegraded":"2024-01-16T14:59:58.974601+0000","last_fullsized":"2024-01-16T14:59:58.974601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973072+0000","last_change":"2024-01-16T14:59:56.950566+0000","last_active":"2024-01-16T14:59:58.973072+0000","last_peered":"2024-01-16T14:59:58.973072+0000","last_clean":"2024-01-16T14:59:58.973072+0000","last_became_active":"2024-01-16T14:59:56.949990+0000","last_became_peered":"2024-01-16T14:59:56.949990+0000","last_unstale":"2024-01-16T14:59:58.973072+0000","last_undegraded":"2024-01-16T14:59:58.973072+0000","last_fullsized":"2024-01-16T14:59:58.973072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:14.695598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880960+0000","last_change":"2024-01-16T14:59:57.918222+0000","last_active":"2024-01-16T14:59:58.880960+0000","last_peered":"2024-01-16T14:59:58.880960+0000","last_clean":"2024-01-16T14:59:58.880960+0000","last_became_active":"2024-01-16T14:59:57.917997+0000","last_became_peered":"2024-01-16T14:59:57.917997+0000","last_unstale":"2024-01-16T14:59:58.880960+0000","last_undegraded":"2024-01-16T14:59:58.880960+0000","last_fullsized":"2024-01-16T14:59:58.880960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.942520+0000","last_change":"2024-01-16T14:59:56.958392+0000","last_active":"2024-01-16T14:59:57.942520+0000","last_peered":"2024-01-16T14:59:57.942520+0000","last_clean":"2024-01-16T14:59:57.942520+0000","last_became_active":"2024-01-16T14:59:56.955933+0000","last_became_peered":"2024-01-16T14:59:56.955933+0000","last_unstale":"2024-01-16T14:59:57.942520+0000","last_undegraded":"2024-01-16T14:59:57.942520+0000","last_fullsized":"2024-01-16T14:59:57.942520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:50.131057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883242+0000","last_change":"2024-01-16T14:59:57.917801+0000","last_active":"2024-01-16T14:59:58.883242+0000","last_peered":"2024-01-16T14:59:58.883242+0000","last_clean":"2024-01-16T14:59:58.883242+0000","last_became_active":"2024-01-16T14:59:57.917470+0000","last_became_peered":"2024-01-16T14:59:57.917470+0000","last_unstale":"2024-01-16T14:59:58.883242+0000","last_undegraded":"2024-01-16T14:59:58.883242+0000","last_fullsized":"2024-01-16T14:59:58.883242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:34.142895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974235+0000","last_change":"2024-01-16T14:59:56.954099+0000","last_active":"2024-01-16T14:59:58.974235+0000","last_peered":"2024-01-16T14:59:58.974235+0000","last_clean":"2024-01-16T14:59:58.974235+0000","last_became_active":"2024-01-16T14:59:56.953811+0000","last_became_peered":"2024-01-16T14:59:56.953811+0000","last_unstale":"2024-01-16T14:59:58.974235+0000","last_undegraded":"2024-01-16T14:59:58.974235+0000","last_fullsized":"2024-01-16T14:59:58.974235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:08.095887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883459+0000","last_change":"2024-01-16T14:59:57.921075+0000","last_active":"2024-01-16T14:59:58.883459+0000","last_peered":"2024-01-16T14:59:58.883459+0000","last_clean":"2024-01-16T14:59:58.883459+0000","last_became_active":"2024-01-16T14:59:57.920649+0000","last_became_peered":"2024-01-16T14:59:57.920649+0000","last_unstale":"2024-01-16T14:59:58.883459+0000","last_undegraded":"2024-01-16T14:59:58.883459+0000","last_fullsized":"2024-01-16T14:59:58.883459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:30.432751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964505+0000","last_change":"2024-01-16T14:59:56.965927+0000","last_active":"2024-01-16T14:59:58.964505+0000","last_peered":"2024-01-16T14:59:58.964505+0000","last_clean":"2024-01-16T14:59:58.964505+0000","last_became_active":"2024-01-16T14:59:56.965627+0000","last_became_peered":"2024-01-16T14:59:56.965627+0000","last_unstale":"2024-01-16T14:59:58.964505+0000","last_undegraded":"2024-01-16T14:59:58.964505+0000","last_fullsized":"2024-01-16T14:59:58.964505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:11.715798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975760+0000","last_change":"2024-01-16T14:59:57.919926+0000","last_active":"2024-01-16T14:59:58.975760+0000","last_peered":"2024-01-16T14:59:58.975760+0000","last_clean":"2024-01-16T14:59:58.975760+0000","last_became_active":"2024-01-16T14:59:57.919380+0000","last_became_peered":"2024-01-16T14:59:57.919380+0000","last_unstale":"2024-01-16T14:59:58.975760+0000","last_undegraded":"2024-01-16T14:59:58.975760+0000","last_fullsized":"2024-01-16T14:59:58.975760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973702+0000","last_change":"2024-01-16T14:59:56.961433+0000","last_active":"2024-01-16T14:59:58.973702+0000","last_peered":"2024-01-16T14:59:58.973702+0000","last_clean":"2024-01-16T14:59:58.973702+0000","last_became_active":"2024-01-16T14:59:56.961175+0000","last_became_peered":"2024-01-16T14:59:56.961175+0000","last_unstale":"2024-01-16T14:59:58.973702+0000","last_undegraded":"2024-01-16T14:59:58.973702+0000","last_fullsized":"2024-01-16T14:59:58.973702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:34.084632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882600+0000","last_change":"2024-01-16T14:59:57.912741+0000","last_active":"2024-01-16T14:59:58.882600+0000","last_peered":"2024-01-16T14:59:58.882600+0000","last_clean":"2024-01-16T14:59:58.882600+0000","last_became_active":"2024-01-16T14:59:57.908529+0000","last_became_peered":"2024-01-16T14:59:57.908529+0000","last_unstale":"2024-01-16T14:59:58.882600+0000","last_undegraded":"2024-01-16T14:59:58.882600+0000","last_fullsized":"2024-01-16T14:59:58.882600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.907222+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T14:59:57.907222+0000","last_peered":"2024-01-16T14:59:57.907222+0000","last_clean":"2024-01-16T14:59:57.907222+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T14:59:57.907222+0000","last_undegraded":"2024-01-16T14:59:57.907222+0000","last_fullsized":"2024-01-16T14:59:57.907222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974661+0000","last_change":"2024-01-16T14:59:56.964503+0000","last_active":"2024-01-16T14:59:58.974661+0000","last_peered":"2024-01-16T14:59:58.974661+0000","last_clean":"2024-01-16T14:59:58.974661+0000","last_became_active":"2024-01-16T14:59:56.964165+0000","last_became_peered":"2024-01-16T14:59:56.964165+0000","last_unstale":"2024-01-16T14:59:58.974661+0000","last_undegraded":"2024-01-16T14:59:58.974661+0000","last_fullsized":"2024-01-16T14:59:58.974661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:39.083282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977186+0000","last_change":"2024-01-16T14:59:57.916509+0000","last_active":"2024-01-16T14:59:58.977186+0000","last_peered":"2024-01-16T14:59:58.977186+0000","last_clean":"2024-01-16T14:59:58.977186+0000","last_became_active":"2024-01-16T14:59:57.915770+0000","last_became_peered":"2024-01-16T14:59:57.915770+0000","last_unstale":"2024-01-16T14:59:58.977186+0000","last_undegraded":"2024-01-16T14:59:58.977186+0000","last_fullsized":"2024-01-16T14:59:58.977186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880642+0000","last_change":"2024-01-16T14:59:56.965724+0000","last_active":"2024-01-16T14:59:58.880642+0000","last_peered":"2024-01-16T14:59:58.880642+0000","last_clean":"2024-01-16T14:59:58.880642+0000","last_became_active":"2024-01-16T14:59:56.965458+0000","last_became_peered":"2024-01-16T14:59:56.965458+0000","last_unstale":"2024-01-16T14:59:58.880642+0000","last_undegraded":"2024-01-16T14:59:58.880642+0000","last_fullsized":"2024-01-16T14:59:58.880642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973635+0000","last_change":"2024-01-16T14:59:57.912107+0000","last_active":"2024-01-16T14:59:58.973635+0000","last_peered":"2024-01-16T14:59:58.973635+0000","last_clean":"2024-01-16T14:59:58.973635+0000","last_became_active":"2024-01-16T14:59:57.911776+0000","last_became_peered":"2024-01-16T14:59:57.911776+0000","last_unstale":"2024-01-16T14:59:58.973635+0000","last_undegraded":"2024-01-16T14:59:58.973635+0000","last_fullsized":"2024-01-16T14:59:58.973635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:33.290097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973813+0000","last_change":"2024-01-16T14:59:56.956041+0000","last_active":"2024-01-16T14:59:58.973813+0000","last_peered":"2024-01-16T14:59:58.973813+0000","last_clean":"2024-01-16T14:59:58.973813+0000","last_became_active":"2024-01-16T14:59:56.955669+0000","last_became_peered":"2024-01-16T14:59:56.955669+0000","last_unstale":"2024-01-16T14:59:58.973813+0000","last_undegraded":"2024-01-16T14:59:58.973813+0000","last_fullsized":"2024-01-16T14:59:58.973813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964006+0000","last_change":"2024-01-16T14:59:57.905614+0000","last_active":"2024-01-16T14:59:58.964006+0000","last_peered":"2024-01-16T14:59:58.964006+0000","last_clean":"2024-01-16T14:59:58.964006+0000","last_became_active":"2024-01-16T14:59:57.905277+0000","last_became_peered":"2024-01-16T14:59:57.905277+0000","last_unstale":"2024-01-16T14:59:58.964006+0000","last_undegraded":"2024-01-16T14:59:58.964006+0000","last_fullsized":"2024-01-16T14:59:58.964006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977517+0000","last_change":"2024-01-16T14:59:56.956330+0000","last_active":"2024-01-16T14:59:58.977517+0000","last_peered":"2024-01-16T14:59:58.977517+0000","last_clean":"2024-01-16T14:59:58.977517+0000","last_became_active":"2024-01-16T14:59:56.955936+0000","last_became_peered":"2024-01-16T14:59:56.955936+0000","last_unstale":"2024-01-16T14:59:58.977517+0000","last_undegraded":"2024-01-16T14:59:58.977517+0000","last_fullsized":"2024-01-16T14:59:58.977517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:53.098539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884894+0000","last_change":"2024-01-16T14:59:57.911172+0000","last_active":"2024-01-16T14:59:58.884894+0000","last_peered":"2024-01-16T14:59:58.884894+0000","last_clean":"2024-01-16T14:59:58.884894+0000","last_became_active":"2024-01-16T14:59:57.910403+0000","last_became_peered":"2024-01-16T14:59:57.910403+0000","last_unstale":"2024-01-16T14:59:58.884894+0000","last_undegraded":"2024-01-16T14:59:58.884894+0000","last_fullsized":"2024-01-16T14:59:58.884894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:23.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964455+0000","last_change":"2024-01-16T14:59:56.955294+0000","last_active":"2024-01-16T14:59:58.964455+0000","last_peered":"2024-01-16T14:59:58.964455+0000","last_clean":"2024-01-16T14:59:58.964455+0000","last_became_active":"2024-01-16T14:59:56.953316+0000","last_became_peered":"2024-01-16T14:59:56.953316+0000","last_unstale":"2024-01-16T14:59:58.964455+0000","last_undegraded":"2024-01-16T14:59:58.964455+0000","last_fullsized":"2024-01-16T14:59:58.964455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976348+0000","last_change":"2024-01-16T14:59:57.917538+0000","last_active":"2024-01-16T14:59:58.976348+0000","last_peered":"2024-01-16T14:59:58.976348+0000","last_clean":"2024-01-16T14:59:58.976348+0000","last_became_active":"2024-01-16T14:59:57.917208+0000","last_became_peered":"2024-01-16T14:59:57.917208+0000","last_unstale":"2024-01-16T14:59:58.976348+0000","last_undegraded":"2024-01-16T14:59:58.976348+0000","last_fullsized":"2024-01-16T14:59:58.976348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882243+0000","last_change":"2024-01-16T14:59:56.963301+0000","last_active":"2024-01-16T14:59:58.882243+0000","last_peered":"2024-01-16T14:59:58.882243+0000","last_clean":"2024-01-16T14:59:58.882243+0000","last_became_active":"2024-01-16T14:59:56.962999+0000","last_became_peered":"2024-01-16T14:59:56.962999+0000","last_unstale":"2024-01-16T14:59:58.882243+0000","last_undegraded":"2024-01-16T14:59:58.882243+0000","last_fullsized":"2024-01-16T14:59:58.882243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:34.876313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964280+0000","last_change":"2024-01-16T14:59:57.901887+0000","last_active":"2024-01-16T14:59:58.964280+0000","last_peered":"2024-01-16T14:59:58.964280+0000","last_clean":"2024-01-16T14:59:58.964280+0000","last_became_active":"2024-01-16T14:59:57.901474+0000","last_became_peered":"2024-01-16T14:59:57.901474+0000","last_unstale":"2024-01-16T14:59:58.964280+0000","last_undegraded":"2024-01-16T14:59:58.964280+0000","last_fullsized":"2024-01-16T14:59:58.964280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964917+0000","last_change":"2024-01-16T14:59:56.967570+0000","last_active":"2024-01-16T14:59:58.964917+0000","last_peered":"2024-01-16T14:59:58.964917+0000","last_clean":"2024-01-16T14:59:58.964917+0000","last_became_active":"2024-01-16T14:59:56.967245+0000","last_became_peered":"2024-01-16T14:59:56.967245+0000","last_unstale":"2024-01-16T14:59:58.964917+0000","last_undegraded":"2024-01-16T14:59:58.964917+0000","last_fullsized":"2024-01-16T14:59:58.964917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:50.405789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.979264+0000","last_change":"2024-01-16T14:59:57.914280+0000","last_active":"2024-01-16T14:59:58.979264+0000","last_peered":"2024-01-16T14:59:58.979264+0000","last_clean":"2024-01-16T14:59:58.979264+0000","last_became_active":"2024-01-16T14:59:57.909069+0000","last_became_peered":"2024-01-16T14:59:57.909069+0000","last_unstale":"2024-01-16T14:59:58.979264+0000","last_undegraded":"2024-01-16T14:59:58.979264+0000","last_fullsized":"2024-01-16T14:59:58.979264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:45:32.241461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.943628+0000","last_change":"2024-01-16T14:59:56.963783+0000","last_active":"2024-01-16T14:59:57.943628+0000","last_peered":"2024-01-16T14:59:57.943628+0000","last_clean":"2024-01-16T14:59:57.943628+0000","last_became_active":"2024-01-16T14:59:56.963167+0000","last_became_peered":"2024-01-16T14:59:56.963167+0000","last_unstale":"2024-01-16T14:59:57.943628+0000","last_undegraded":"2024-01-16T14:59:57.943628+0000","last_fullsized":"2024-01-16T14:59:57.943628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974544+0000","last_change":"2024-01-16T14:59:57.907693+0000","last_active":"2024-01-16T14:59:58.974544+0000","last_peered":"2024-01-16T14:59:58.974544+0000","last_clean":"2024-01-16T14:59:58.974544+0000","last_became_active":"2024-01-16T14:59:57.907365+0000","last_became_peered":"2024-01-16T14:59:57.907365+0000","last_unstale":"2024-01-16T14:59:58.974544+0000","last_undegraded":"2024-01-16T14:59:58.974544+0000","last_fullsized":"2024-01-16T14:59:58.974544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:31.975241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881061+0000","last_change":"2024-01-16T14:59:56.964456+0000","last_active":"2024-01-16T14:59:58.881061+0000","last_peered":"2024-01-16T14:59:58.881061+0000","last_clean":"2024-01-16T14:59:58.881061+0000","last_became_active":"2024-01-16T14:59:56.964105+0000","last_became_peered":"2024-01-16T14:59:56.964105+0000","last_unstale":"2024-01-16T14:59:58.881061+0000","last_undegraded":"2024-01-16T14:59:58.881061+0000","last_fullsized":"2024-01-16T14:59:58.881061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884695+0000","last_change":"2024-01-16T14:59:57.909622+0000","last_active":"2024-01-16T14:59:58.884695+0000","last_peered":"2024-01-16T14:59:58.884695+0000","last_clean":"2024-01-16T14:59:58.884695+0000","last_became_active":"2024-01-16T14:59:57.905809+0000","last_became_peered":"2024-01-16T14:59:57.905809+0000","last_unstale":"2024-01-16T14:59:58.884695+0000","last_undegraded":"2024-01-16T14:59:58.884695+0000","last_fullsized":"2024-01-16T14:59:58.884695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:30.677552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964306+0000","last_change":"2024-01-16T14:59:56.961402+0000","last_active":"2024-01-16T14:59:58.964306+0000","last_peered":"2024-01-16T14:59:58.964306+0000","last_clean":"2024-01-16T14:59:58.964306+0000","last_became_active":"2024-01-16T14:59:56.959909+0000","last_became_peered":"2024-01-16T14:59:56.959909+0000","last_unstale":"2024-01-16T14:59:58.964306+0000","last_undegraded":"2024-01-16T14:59:58.964306+0000","last_fullsized":"2024-01-16T14:59:58.964306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:00.188907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976651+0000","last_change":"2024-01-16T14:59:57.902379+0000","last_active":"2024-01-16T14:59:58.976651+0000","last_peered":"2024-01-16T14:59:58.976651+0000","last_clean":"2024-01-16T14:59:58.976651+0000","last_became_active":"2024-01-16T14:59:57.902002+0000","last_became_peered":"2024-01-16T14:59:57.902002+0000","last_unstale":"2024-01-16T14:59:58.976651+0000","last_undegraded":"2024-01-16T14:59:58.976651+0000","last_fullsized":"2024-01-16T14:59:58.976651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.963988+0000","last_change":"2024-01-16T14:59:56.956035+0000","last_active":"2024-01-16T14:59:58.963988+0000","last_peered":"2024-01-16T14:59:58.963988+0000","last_clean":"2024-01-16T14:59:58.963988+0000","last_became_active":"2024-01-16T14:59:56.955666+0000","last_became_peered":"2024-01-16T14:59:56.955666+0000","last_unstale":"2024-01-16T14:59:58.963988+0000","last_undegraded":"2024-01-16T14:59:58.963988+0000","last_fullsized":"2024-01-16T14:59:58.963988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977506+0000","last_change":"2024-01-16T14:59:57.913992+0000","last_active":"2024-01-16T14:59:58.977506+0000","last_peered":"2024-01-16T14:59:58.977506+0000","last_clean":"2024-01-16T14:59:58.977506+0000","last_became_active":"2024-01-16T14:59:57.913639+0000","last_became_peered":"2024-01-16T14:59:57.913639+0000","last_unstale":"2024-01-16T14:59:58.977506+0000","last_undegraded":"2024-01-16T14:59:58.977506+0000","last_fullsized":"2024-01-16T14:59:58.977506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882012+0000","last_change":"2024-01-16T14:59:56.970741+0000","last_active":"2024-01-16T14:59:58.882012+0000","last_peered":"2024-01-16T14:59:58.882012+0000","last_clean":"2024-01-16T14:59:58.882012+0000","last_became_active":"2024-01-16T14:59:56.970075+0000","last_became_peered":"2024-01-16T14:59:56.970075+0000","last_unstale":"2024-01-16T14:59:58.882012+0000","last_undegraded":"2024-01-16T14:59:58.882012+0000","last_fullsized":"2024-01-16T14:59:58.882012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:21.992849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974326+0000","last_change":"2024-01-16T14:59:57.906046+0000","last_active":"2024-01-16T14:59:58.974326+0000","last_peered":"2024-01-16T14:59:58.974326+0000","last_clean":"2024-01-16T14:59:58.974326+0000","last_became_active":"2024-01-16T14:59:57.905751+0000","last_became_peered":"2024-01-16T14:59:57.905751+0000","last_unstale":"2024-01-16T14:59:58.974326+0000","last_undegraded":"2024-01-16T14:59:58.974326+0000","last_fullsized":"2024-01-16T14:59:58.974326+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976133+0000","last_change":"2024-01-16T14:59:57.920773+0000","last_active":"2024-01-16T14:59:58.976133+0000","last_peered":"2024-01-16T14:59:58.976133+0000","last_clean":"2024-01-16T14:59:58.976133+0000","last_became_active":"2024-01-16T14:59:57.920488+0000","last_became_peered":"2024-01-16T14:59:57.920488+0000","last_unstale":"2024-01-16T14:59:58.976133+0000","last_undegraded":"2024-01-16T14:59:58.976133+0000","last_fullsized":"2024-01-16T14:59:58.976133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881475+0000","last_change":"2024-01-16T14:59:56.968586+0000","last_active":"2024-01-16T14:59:58.881475+0000","last_peered":"2024-01-16T14:59:58.881475+0000","last_clean":"2024-01-16T14:59:58.881475+0000","last_became_active":"2024-01-16T14:59:56.967923+0000","last_became_peered":"2024-01-16T14:59:56.967923+0000","last_unstale":"2024-01-16T14:59:58.881475+0000","last_undegraded":"2024-01-16T14:59:58.881475+0000","last_fullsized":"2024-01-16T14:59:58.881475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975571+0000","last_change":"2024-01-16T14:59:57.919788+0000","last_active":"2024-01-16T14:59:58.975571+0000","last_peered":"2024-01-16T14:59:58.975571+0000","last_clean":"2024-01-16T14:59:58.975571+0000","last_became_active":"2024-01-16T14:59:57.918939+0000","last_became_peered":"2024-01-16T14:59:57.918939+0000","last_unstale":"2024-01-16T14:59:58.975571+0000","last_undegraded":"2024-01-16T14:59:58.975571+0000","last_fullsized":"2024-01-16T14:59:58.975571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881694+0000","last_change":"2024-01-16T14:59:56.969250+0000","last_active":"2024-01-16T14:59:58.881694+0000","last_peered":"2024-01-16T14:59:58.881694+0000","last_clean":"2024-01-16T14:59:58.881694+0000","last_became_active":"2024-01-16T14:59:56.968877+0000","last_became_peered":"2024-01-16T14:59:56.968877+0000","last_unstale":"2024-01-16T14:59:58.881694+0000","last_undegraded":"2024-01-16T14:59:58.881694+0000","last_fullsized":"2024-01-16T14:59:58.881694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975133+0000","last_change":"2024-01-16T14:59:57.909922+0000","last_active":"2024-01-16T14:59:58.975133+0000","last_peered":"2024-01-16T14:59:58.975133+0000","last_clean":"2024-01-16T14:59:58.975133+0000","last_became_active":"2024-01-16T14:59:57.908518+0000","last_became_peered":"2024-01-16T14:59:57.908518+0000","last_unstale":"2024-01-16T14:59:58.975133+0000","last_undegraded":"2024-01-16T14:59:58.975133+0000","last_fullsized":"2024-01-16T14:59:58.975133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978143+0000","last_change":"2024-01-16T14:59:56.956482+0000","last_active":"2024-01-16T14:59:58.978143+0000","last_peered":"2024-01-16T14:59:58.978143+0000","last_clean":"2024-01-16T14:59:58.978143+0000","last_became_active":"2024-01-16T14:59:56.955091+0000","last_became_peered":"2024-01-16T14:59:56.955091+0000","last_unstale":"2024-01-16T14:59:58.978143+0000","last_undegraded":"2024-01-16T14:59:58.978143+0000","last_fullsized":"2024-01-16T14:59:58.978143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977730+0000","last_change":"2024-01-16T14:59:57.913479+0000","last_active":"2024-01-16T14:59:58.977730+0000","last_peered":"2024-01-16T14:59:58.977730+0000","last_clean":"2024-01-16T14:59:58.977730+0000","last_became_active":"2024-01-16T14:59:57.911939+0000","last_became_peered":"2024-01-16T14:59:57.911939+0000","last_unstale":"2024-01-16T14:59:58.977730+0000","last_undegraded":"2024-01-16T14:59:58.977730+0000","last_fullsized":"2024-01-16T14:59:58.977730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:26.642001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975233+0000","last_change":"2024-01-16T14:59:56.957873+0000","last_active":"2024-01-16T14:59:58.975233+0000","last_peered":"2024-01-16T14:59:58.975233+0000","last_clean":"2024-01-16T14:59:58.975233+0000","last_became_active":"2024-01-16T14:59:56.957287+0000","last_became_peered":"2024-01-16T14:59:56.957287+0000","last_unstale":"2024-01-16T14:59:58.975233+0000","last_undegraded":"2024-01-16T14:59:58.975233+0000","last_fullsized":"2024-01-16T14:59:58.975233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509444,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.675}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4770000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":669118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":78168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9670000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":78328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.101}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":669174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:02.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:02.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.726+0000 7ff10e400640 1 Processor -- start 2024-01-16T15:00:02.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.726+0000 7ff10e400640 1 -- start start 2024-01-16T15:00:02.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.728+0000 7ff10e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10813df80 0x7ff10815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.728+0000 7ff10e400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff10813e360 0x7ff1080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.728+0000 7ff10e400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff10806c470 con 0x7ff10815f9c0 2024-01-16T15:00:02.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.728+0000 7ff10e400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff10806c750 con 0x7ff10813e360 2024-01-16T15:00:02.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.728+0000 7ff10e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff10806ca30 con 0x7ff10813df80 2024-01-16T15:00:02.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.728+0000 7ff106a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff10813e360 0x7ff1080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff107400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10813df80 0x7ff10815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff106a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff10813e360 0x7ff1080cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47494/0 (socket says 172.21.15.53:47494) 2024-01-16T15:00:02.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff106a00640 1 -- 172.21.15.53:0/1700256224 learned_addr learned my addr 172.21.15.53:0/1700256224 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:02.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff107e00640 1 --1- 172.21.15.53:0/1700256224 >> v1:172.21.15.146:6789/0 conn(0x7ff10815f9c0 0x7ff1081408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38324/0 (socket says 172.21.15.53:38324) 2024-01-16T15:00:02.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff106a00640 1 -- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10813df80 msgr2=0x7ff10815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:02.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff106a00640 1 --2- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10813df80 0x7ff10815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff106a00640 1 -- 172.21.15.53:0/1700256224 >> v1:172.21.15.146:6789/0 conn(0x7ff10815f9c0 legacy=0x7ff1081408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.729+0000 7ff106a00640 1 -- 172.21.15.53:0/1700256224 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff10806cd10 con 0x7ff10813e360 2024-01-16T15:00:02.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.731+0000 7ff106a00640 1 --2- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813e360 0x7ff1080cf290 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009870 tx=0x7ff0f002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a53645d2876d9d01 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.731+0000 7ff106000640 1 -- 172.21.15.53:0/1700256224 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0030840 con 0x7ff10813e360 2024-01-16T15:00:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff106000640 1 -- 172.21.15.53:0/1700256224 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f00309a0 con 0x7ff10813e360 2024-01-16T15:00:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff106000640 1 -- 172.21.15.53:0/1700256224 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0030d30 con 0x7ff10813e360 2024-01-16T15:00:02.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 -- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813e360 msgr2=0x7ff1080cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 --2- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813e360 0x7ff1080cf290 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0009870 tx=0x7ff0f002b980 comp rx=0 tx=0).stop 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 -- 172.21.15.53:0/1700256224 shutdown_connections 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 --2- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813e360 0x7ff1080cf290 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 --2- 172.21.15.53:0/1700256224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10813df80 0x7ff10815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 -- 172.21.15.53:0/1700256224 >> 172.21.15.53:0/1700256224 conn(0x7ff10806a800 msgr2=0x7ff10806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 -- 172.21.15.53:0/1700256224 shutdown_connections 2024-01-16T15:00:02.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.732+0000 7ff10e400640 1 -- 172.21.15.53:0/1700256224 wait complete. 2024-01-16T15:00:02.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.734+0000 7ff10e400640 1 Processor -- start 2024-01-16T15:00:02.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.734+0000 7ff10e400640 1 -- start start 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff10e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813df80 0x7ff1081f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff10e400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 0x7ff10814d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff10e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10815f9c0 0x7ff1081524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff10e400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10815dce0 con 0x7ff10813e360 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff10e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff10814c340 con 0x7ff10813df80 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff10e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1081509e0 con 0x7ff10815f9c0 2024-01-16T15:00:02.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.735+0000 7ff106a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 0x7ff10814d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 0x7ff10814d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34752/0 (socket says 172.21.15.53:34752) 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 -- 172.21.15.53:0/797051617 learned_addr learned my addr 172.21.15.53:0/797051617 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff107e00640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10815f9c0 0x7ff1081524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff107400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813df80 0x7ff1081f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 -- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10815f9c0 msgr2=0x7ff1081524b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10815f9c0 0x7ff1081524b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 -- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813df80 msgr2=0x7ff1081f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813df80 0x7ff1081f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 -- 172.21.15.53:0/797051617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f00094d0 con 0x7ff10813e360 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.736+0000 7ff106a00640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 0x7ff10814d4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0000c00 tx=0x7ff0f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.737+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0039c90 con 0x7ff10813e360 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.737+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f0039df0 con 0x7ff10813e360 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.737+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0043640 con 0x7ff10813e360 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.737+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff108150c80 con 0x7ff10813e360 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.737+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff108150f60 con 0x7ff10813e360 2024-01-16T15:00:02.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.738+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1080675d0 con 0x7ff10813e360 2024-01-16T15:00:02.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.739+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff0f00437a0 con 0x7ff10813e360 2024-01-16T15:00:02.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.740+0000 7ff0ff400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0e006b2c0 0x7ff0e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:02.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.740+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff0f003e070 con 0x7ff10813e360 2024-01-16T15:00:02.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.741+0000 7ff107400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0e006b2c0 0x7ff0e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:02.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.742+0000 7ff107400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0e006b2c0 0x7ff0e006d780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009a30 tx=0x7ff0f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:02.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.745+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0f00aee60 con 0x7ff10813e360 2024-01-16T15:00:02.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.939+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff1080cd140 con 0x7ff0e006b2c0 2024-01-16T15:00:02.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.960+0000 7ff0ff400640 1 -- 172.21.15.53:0/797051617 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264419 (secure 0 0 0) 0x7ff1080cd140 con 0x7ff0e006b2c0 2024-01-16T15:00:02.946 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:02.947 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:00:02.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0e006b2c0 msgr2=0x7ff0e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0e006b2c0 0x7ff0e006d780 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8009a30 tx=0x7ff0f8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:02.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 msgr2=0x7ff10814d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:02.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 0x7ff10814d4f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff0f0000c00 tx=0x7ff0f0004730 comp rx=0 tx=0).stop 2024-01-16T15:00:02.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 shutdown_connections 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0e006b2c0 0x7ff0e006d780 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff10815f9c0 0x7ff1081524b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff10813e360 0x7ff10814d4f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 --2- 172.21.15.53:0/797051617 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff10813df80 0x7ff1081f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 >> 172.21.15.53:0/797051617 conn(0x7ff10806a800 msgr2=0x7ff1080b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 shutdown_connections 2024-01-16T15:00:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:02.967+0000 7ff10e400640 1 -- 172.21.15.53:0/797051617 wait complete. 2024-01-16T15:00:02.968 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-01-16T15:00:02.170734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":166,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235776,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738944,"statfs":{"total":773094113280,"available":772852678656,"internally_reserved":0,"allocated":3022848,"data_stored":1810900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001892"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882738+0000","last_change":"2024-01-16T14:59:57.913241+0000","last_active":"2024-01-16T14:59:58.882738+0000","last_peered":"2024-01-16T14:59:58.882738+0000","last_clean":"2024-01-16T14:59:58.882738+0000","last_became_active":"2024-01-16T14:59:57.912720+0000","last_became_peered":"2024-01-16T14:59:57.912720+0000","last_unstale":"2024-01-16T14:59:58.882738+0000","last_undegraded":"2024-01-16T14:59:58.882738+0000","last_fullsized":"2024-01-16T14:59:58.882738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964756+0000","last_change":"2024-01-16T14:59:57.910357+0000","last_active":"2024-01-16T14:59:58.964756+0000","last_peered":"2024-01-16T14:59:58.964756+0000","last_clean":"2024-01-16T14:59:58.964756+0000","last_became_active":"2024-01-16T14:59:57.909933+0000","last_became_peered":"2024-01-16T14:59:57.909933+0000","last_unstale":"2024-01-16T14:59:58.964756+0000","last_undegraded":"2024-01-16T14:59:58.964756+0000","last_fullsized":"2024-01-16T14:59:58.964756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974477+0000","last_change":"2024-01-16T14:59:57.909709+0000","last_active":"2024-01-16T14:59:58.974477+0000","last_peered":"2024-01-16T14:59:58.974477+0000","last_clean":"2024-01-16T14:59:58.974477+0000","last_became_active":"2024-01-16T14:59:57.909405+0000","last_became_peered":"2024-01-16T14:59:57.909405+0000","last_unstale":"2024-01-16T14:59:58.974477+0000","last_undegraded":"2024-01-16T14:59:58.974477+0000","last_fullsized":"2024-01-16T14:59:58.974477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:51.411241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975424+0000","last_change":"2024-01-16T14:59:57.920032+0000","last_active":"2024-01-16T14:59:58.975424+0000","last_peered":"2024-01-16T14:59:58.975424+0000","last_clean":"2024-01-16T14:59:58.975424+0000","last_became_active":"2024-01-16T14:59:57.919630+0000","last_became_peered":"2024-01-16T14:59:57.919630+0000","last_unstale":"2024-01-16T14:59:58.975424+0000","last_undegraded":"2024-01-16T14:59:58.975424+0000","last_fullsized":"2024-01-16T14:59:58.975424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882328+0000","last_change":"2024-01-16T14:59:57.917909+0000","last_active":"2024-01-16T14:59:58.882328+0000","last_peered":"2024-01-16T14:59:58.882328+0000","last_clean":"2024-01-16T14:59:58.882328+0000","last_became_active":"2024-01-16T14:59:57.914603+0000","last_became_peered":"2024-01-16T14:59:57.914603+0000","last_unstale":"2024-01-16T14:59:58.882328+0000","last_undegraded":"2024-01-16T14:59:58.882328+0000","last_fullsized":"2024-01-16T14:59:58.882328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976981+0000","last_change":"2024-01-16T14:59:57.909424+0000","last_active":"2024-01-16T14:59:58.976981+0000","last_peered":"2024-01-16T14:59:58.976981+0000","last_clean":"2024-01-16T14:59:58.976981+0000","last_became_active":"2024-01-16T14:59:57.909160+0000","last_became_peered":"2024-01-16T14:59:57.909160+0000","last_unstale":"2024-01-16T14:59:58.976981+0000","last_undegraded":"2024-01-16T14:59:58.976981+0000","last_fullsized":"2024-01-16T14:59:58.976981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880812+0000","last_change":"2024-01-16T14:59:57.917053+0000","last_active":"2024-01-16T14:59:58.880812+0000","last_peered":"2024-01-16T14:59:58.880812+0000","last_clean":"2024-01-16T14:59:58.880812+0000","last_became_active":"2024-01-16T14:59:57.914775+0000","last_became_peered":"2024-01-16T14:59:57.914775+0000","last_unstale":"2024-01-16T14:59:58.880812+0000","last_undegraded":"2024-01-16T14:59:58.880812+0000","last_fullsized":"2024-01-16T14:59:58.880812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976696+0000","last_change":"2024-01-16T14:59:57.909184+0000","last_active":"2024-01-16T14:59:58.976696+0000","last_peered":"2024-01-16T14:59:58.976696+0000","last_clean":"2024-01-16T14:59:58.976696+0000","last_became_active":"2024-01-16T14:59:57.908905+0000","last_became_peered":"2024-01-16T14:59:57.908905+0000","last_unstale":"2024-01-16T14:59:58.976696+0000","last_undegraded":"2024-01-16T14:59:58.976696+0000","last_fullsized":"2024-01-16T14:59:58.976696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883086+0000","last_change":"2024-01-16T14:59:57.921823+0000","last_active":"2024-01-16T14:59:58.883086+0000","last_peered":"2024-01-16T14:59:58.883086+0000","last_clean":"2024-01-16T14:59:58.883086+0000","last_became_active":"2024-01-16T14:59:57.921476+0000","last_became_peered":"2024-01-16T14:59:57.921476+0000","last_unstale":"2024-01-16T14:59:58.883086+0000","last_undegraded":"2024-01-16T14:59:58.883086+0000","last_fullsized":"2024-01-16T14:59:58.883086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975563+0000","last_change":"2024-01-16T14:59:57.909965+0000","last_active":"2024-01-16T14:59:58.975563+0000","last_peered":"2024-01-16T14:59:58.975563+0000","last_clean":"2024-01-16T14:59:58.975563+0000","last_became_active":"2024-01-16T14:59:57.909707+0000","last_became_peered":"2024-01-16T14:59:57.909707+0000","last_unstale":"2024-01-16T14:59:58.975563+0000","last_undegraded":"2024-01-16T14:59:58.975563+0000","last_fullsized":"2024-01-16T14:59:58.975563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975772+0000","last_change":"2024-01-16T14:59:57.908484+0000","last_active":"2024-01-16T14:59:58.975772+0000","last_peered":"2024-01-16T14:59:58.975772+0000","last_clean":"2024-01-16T14:59:58.975772+0000","last_became_active":"2024-01-16T14:59:57.908164+0000","last_became_peered":"2024-01-16T14:59:57.908164+0000","last_unstale":"2024-01-16T14:59:58.975772+0000","last_undegraded":"2024-01-16T14:59:58.975772+0000","last_fullsized":"2024-01-16T14:59:58.975772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885578+0000","last_change":"2024-01-16T14:59:57.914372+0000","last_active":"2024-01-16T14:59:58.885578+0000","last_peered":"2024-01-16T14:59:58.885578+0000","last_clean":"2024-01-16T14:59:58.885578+0000","last_became_active":"2024-01-16T14:59:57.913918+0000","last_became_peered":"2024-01-16T14:59:57.913918+0000","last_unstale":"2024-01-16T14:59:58.885578+0000","last_undegraded":"2024-01-16T14:59:58.885578+0000","last_fullsized":"2024-01-16T14:59:58.885578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:08.622882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977380+0000","last_change":"2024-01-16T14:59:57.907940+0000","last_active":"2024-01-16T14:59:58.977380+0000","last_peered":"2024-01-16T14:59:58.977380+0000","last_clean":"2024-01-16T14:59:58.977380+0000","last_became_active":"2024-01-16T14:59:57.906517+0000","last_became_peered":"2024-01-16T14:59:57.906517+0000","last_unstale":"2024-01-16T14:59:58.977380+0000","last_undegraded":"2024-01-16T14:59:58.977380+0000","last_fullsized":"2024-01-16T14:59:58.977380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974085+0000","last_change":"2024-01-16T14:59:57.907053+0000","last_active":"2024-01-16T14:59:58.974085+0000","last_peered":"2024-01-16T14:59:58.974085+0000","last_clean":"2024-01-16T14:59:58.974085+0000","last_became_active":"2024-01-16T14:59:57.906443+0000","last_became_peered":"2024-01-16T14:59:57.906443+0000","last_unstale":"2024-01-16T14:59:58.974085+0000","last_undegraded":"2024-01-16T14:59:58.974085+0000","last_fullsized":"2024-01-16T14:59:58.974085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:01.194359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964073+0000","last_change":"2024-01-16T14:59:57.893114+0000","last_active":"2024-01-16T14:59:58.964073+0000","last_peered":"2024-01-16T14:59:58.964073+0000","last_clean":"2024-01-16T14:59:58.964073+0000","last_became_active":"2024-01-16T14:59:57.891027+0000","last_became_peered":"2024-01-16T14:59:57.891027+0000","last_unstale":"2024-01-16T14:59:58.964073+0000","last_undegraded":"2024-01-16T14:59:58.964073+0000","last_fullsized":"2024-01-16T14:59:58.964073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974627+0000","last_change":"2024-01-16T14:59:57.911794+0000","last_active":"2024-01-16T14:59:58.974627+0000","last_peered":"2024-01-16T14:59:58.974627+0000","last_clean":"2024-01-16T14:59:58.974627+0000","last_became_active":"2024-01-16T14:59:57.911528+0000","last_became_peered":"2024-01-16T14:59:57.911528+0000","last_unstale":"2024-01-16T14:59:58.974627+0000","last_undegraded":"2024-01-16T14:59:58.974627+0000","last_fullsized":"2024-01-16T14:59:58.974627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976633+0000","last_change":"2024-01-16T14:59:57.917524+0000","last_active":"2024-01-16T14:59:58.976633+0000","last_peered":"2024-01-16T14:59:58.976633+0000","last_clean":"2024-01-16T14:59:58.976633+0000","last_became_active":"2024-01-16T14:59:57.916983+0000","last_became_peered":"2024-01-16T14:59:57.916983+0000","last_unstale":"2024-01-16T14:59:58.976633+0000","last_undegraded":"2024-01-16T14:59:58.976633+0000","last_fullsized":"2024-01-16T14:59:58.976633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974145+0000","last_change":"2024-01-16T14:59:57.914130+0000","last_active":"2024-01-16T14:59:58.974145+0000","last_peered":"2024-01-16T14:59:58.974145+0000","last_clean":"2024-01-16T14:59:58.974145+0000","last_became_active":"2024-01-16T14:59:57.913722+0000","last_became_peered":"2024-01-16T14:59:57.913722+0000","last_unstale":"2024-01-16T14:59:58.974145+0000","last_undegraded":"2024-01-16T14:59:58.974145+0000","last_fullsized":"2024-01-16T14:59:58.974145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:37.662076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882022+0000","last_change":"2024-01-16T14:59:57.914510+0000","last_active":"2024-01-16T14:59:58.882022+0000","last_peered":"2024-01-16T14:59:58.882022+0000","last_clean":"2024-01-16T14:59:58.882022+0000","last_became_active":"2024-01-16T14:59:57.912912+0000","last_became_peered":"2024-01-16T14:59:57.912912+0000","last_unstale":"2024-01-16T14:59:58.882022+0000","last_undegraded":"2024-01-16T14:59:58.882022+0000","last_fullsized":"2024-01-16T14:59:58.882022+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975057+0000","last_change":"2024-01-16T14:59:57.911759+0000","last_active":"2024-01-16T14:59:58.975057+0000","last_peered":"2024-01-16T14:59:58.975057+0000","last_clean":"2024-01-16T14:59:58.975057+0000","last_became_active":"2024-01-16T14:59:57.911474+0000","last_became_peered":"2024-01-16T14:59:57.911474+0000","last_unstale":"2024-01-16T14:59:58.975057+0000","last_undegraded":"2024-01-16T14:59:58.975057+0000","last_fullsized":"2024-01-16T14:59:58.975057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975453+0000","last_change":"2024-01-16T14:59:57.923664+0000","last_active":"2024-01-16T14:59:58.975453+0000","last_peered":"2024-01-16T14:59:58.975453+0000","last_clean":"2024-01-16T14:59:58.975453+0000","last_became_active":"2024-01-16T14:59:57.923402+0000","last_became_peered":"2024-01-16T14:59:57.923402+0000","last_unstale":"2024-01-16T14:59:58.975453+0000","last_undegraded":"2024-01-16T14:59:58.975453+0000","last_fullsized":"2024-01-16T14:59:58.975453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977267+0000","last_change":"2024-01-16T14:59:57.923807+0000","last_active":"2024-01-16T14:59:58.977267+0000","last_peered":"2024-01-16T14:59:58.977267+0000","last_clean":"2024-01-16T14:59:58.977267+0000","last_became_active":"2024-01-16T14:59:57.923493+0000","last_became_peered":"2024-01-16T14:59:57.923493+0000","last_unstale":"2024-01-16T14:59:58.977267+0000","last_undegraded":"2024-01-16T14:59:58.977267+0000","last_fullsized":"2024-01-16T14:59:58.977267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977086+0000","last_change":"2024-01-16T14:59:57.912156+0000","last_active":"2024-01-16T14:59:58.977086+0000","last_peered":"2024-01-16T14:59:58.977086+0000","last_clean":"2024-01-16T14:59:58.977086+0000","last_became_active":"2024-01-16T14:59:57.911497+0000","last_became_peered":"2024-01-16T14:59:57.911497+0000","last_unstale":"2024-01-16T14:59:58.977086+0000","last_undegraded":"2024-01-16T14:59:58.977086+0000","last_fullsized":"2024-01-16T14:59:58.977086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974975+0000","last_change":"2024-01-16T14:59:57.918668+0000","last_active":"2024-01-16T14:59:58.974975+0000","last_peered":"2024-01-16T14:59:58.974975+0000","last_clean":"2024-01-16T14:59:58.974975+0000","last_became_active":"2024-01-16T14:59:57.918314+0000","last_became_peered":"2024-01-16T14:59:57.918314+0000","last_unstale":"2024-01-16T14:59:58.974975+0000","last_undegraded":"2024-01-16T14:59:58.974975+0000","last_fullsized":"2024-01-16T14:59:58.974975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978927+0000","last_change":"2024-01-16T14:59:57.912774+0000","last_active":"2024-01-16T14:59:58.978927+0000","last_peered":"2024-01-16T14:59:58.978927+0000","last_clean":"2024-01-16T14:59:58.978927+0000","last_became_active":"2024-01-16T14:59:57.910934+0000","last_became_peered":"2024-01-16T14:59:57.910934+0000","last_unstale":"2024-01-16T14:59:58.978927+0000","last_undegraded":"2024-01-16T14:59:58.978927+0000","last_fullsized":"2024-01-16T14:59:58.978927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:47.822358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975717+0000","last_change":"2024-01-16T14:59:57.921118+0000","last_active":"2024-01-16T14:59:58.975717+0000","last_peered":"2024-01-16T14:59:58.975717+0000","last_clean":"2024-01-16T14:59:58.975717+0000","last_became_active":"2024-01-16T14:59:57.920835+0000","last_became_peered":"2024-01-16T14:59:57.920835+0000","last_unstale":"2024-01-16T14:59:58.975717+0000","last_undegraded":"2024-01-16T14:59:58.975717+0000","last_fullsized":"2024-01-16T14:59:58.975717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964405+0000","last_change":"2024-01-16T14:59:57.898548+0000","last_active":"2024-01-16T14:59:58.964405+0000","last_peered":"2024-01-16T14:59:58.964405+0000","last_clean":"2024-01-16T14:59:58.964405+0000","last_became_active":"2024-01-16T14:59:57.897874+0000","last_became_peered":"2024-01-16T14:59:57.897874+0000","last_unstale":"2024-01-16T14:59:58.964405+0000","last_undegraded":"2024-01-16T14:59:58.964405+0000","last_fullsized":"2024-01-16T14:59:58.964405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978167+0000","last_change":"2024-01-16T14:59:57.914061+0000","last_active":"2024-01-16T14:59:58.978167+0000","last_peered":"2024-01-16T14:59:58.978167+0000","last_clean":"2024-01-16T14:59:58.978167+0000","last_became_active":"2024-01-16T14:59:57.913718+0000","last_became_peered":"2024-01-16T14:59:57.913718+0000","last_unstale":"2024-01-16T14:59:58.978167+0000","last_undegraded":"2024-01-16T14:59:58.978167+0000","last_fullsized":"2024-01-16T14:59:58.978167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:46.106265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974760+0000","last_change":"2024-01-16T14:59:57.917278+0000","last_active":"2024-01-16T14:59:58.974760+0000","last_peered":"2024-01-16T14:59:58.974760+0000","last_clean":"2024-01-16T14:59:58.974760+0000","last_became_active":"2024-01-16T14:59:57.916768+0000","last_became_peered":"2024-01-16T14:59:57.916768+0000","last_unstale":"2024-01-16T14:59:58.974760+0000","last_undegraded":"2024-01-16T14:59:58.974760+0000","last_fullsized":"2024-01-16T14:59:58.974760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:12.721250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977355+0000","last_change":"2024-01-16T14:59:57.915403+0000","last_active":"2024-01-16T14:59:58.977355+0000","last_peered":"2024-01-16T14:59:58.977355+0000","last_clean":"2024-01-16T14:59:58.977355+0000","last_became_active":"2024-01-16T14:59:57.915134+0000","last_became_peered":"2024-01-16T14:59:57.915134+0000","last_unstale":"2024-01-16T14:59:58.977355+0000","last_undegraded":"2024-01-16T14:59:58.977355+0000","last_fullsized":"2024-01-16T14:59:58.977355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973385+0000","last_change":"2024-01-16T14:59:57.910767+0000","last_active":"2024-01-16T14:59:58.973385+0000","last_peered":"2024-01-16T14:59:58.973385+0000","last_clean":"2024-01-16T14:59:58.973385+0000","last_became_active":"2024-01-16T14:59:57.908877+0000","last_became_peered":"2024-01-16T14:59:57.908877+0000","last_unstale":"2024-01-16T14:59:58.973385+0000","last_undegraded":"2024-01-16T14:59:58.973385+0000","last_fullsized":"2024-01-16T14:59:58.973385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:58.524840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974275+0000","last_change":"2024-01-16T14:59:57.912035+0000","last_active":"2024-01-16T14:59:58.974275+0000","last_peered":"2024-01-16T14:59:58.974275+0000","last_clean":"2024-01-16T14:59:58.974275+0000","last_became_active":"2024-01-16T14:59:57.911641+0000","last_became_peered":"2024-01-16T14:59:57.911641+0000","last_unstale":"2024-01-16T14:59:58.974275+0000","last_undegraded":"2024-01-16T14:59:58.974275+0000","last_fullsized":"2024-01-16T14:59:58.974275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884477+0000","last_change":"2024-01-16T14:59:57.898975+0000","last_active":"2024-01-16T14:59:58.884477+0000","last_peered":"2024-01-16T14:59:58.884477+0000","last_clean":"2024-01-16T14:59:58.884477+0000","last_became_active":"2024-01-16T14:59:57.894960+0000","last_became_peered":"2024-01-16T14:59:57.894960+0000","last_unstale":"2024-01-16T14:59:58.884477+0000","last_undegraded":"2024-01-16T14:59:58.884477+0000","last_fullsized":"2024-01-16T14:59:58.884477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:50.235874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977646+0000","last_change":"2024-01-16T14:59:56.961128+0000","last_active":"2024-01-16T14:59:58.977646+0000","last_peered":"2024-01-16T14:59:58.977646+0000","last_clean":"2024-01-16T14:59:58.977646+0000","last_became_active":"2024-01-16T14:59:56.960829+0000","last_became_peered":"2024-01-16T14:59:56.960829+0000","last_unstale":"2024-01-16T14:59:58.977646+0000","last_undegraded":"2024-01-16T14:59:58.977646+0000","last_fullsized":"2024-01-16T14:59:58.977646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975040+0000","last_change":"2024-01-16T14:59:57.907318+0000","last_active":"2024-01-16T14:59:58.975040+0000","last_peered":"2024-01-16T14:59:58.975040+0000","last_clean":"2024-01-16T14:59:58.975040+0000","last_became_active":"2024-01-16T14:59:57.907025+0000","last_became_peered":"2024-01-16T14:59:57.907025+0000","last_unstale":"2024-01-16T14:59:58.975040+0000","last_undegraded":"2024-01-16T14:59:58.975040+0000","last_fullsized":"2024-01-16T14:59:58.975040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.937959+0000","last_change":"2024-01-16T14:59:56.968922+0000","last_active":"2024-01-16T14:59:57.937959+0000","last_peered":"2024-01-16T14:59:57.937959+0000","last_clean":"2024-01-16T14:59:57.937959+0000","last_became_active":"2024-01-16T14:59:56.968639+0000","last_became_peered":"2024-01-16T14:59:56.968639+0000","last_unstale":"2024-01-16T14:59:57.937959+0000","last_undegraded":"2024-01-16T14:59:57.937959+0000","last_fullsized":"2024-01-16T14:59:57.937959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:29.672100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977060+0000","last_change":"2024-01-16T14:59:56.966987+0000","last_active":"2024-01-16T14:59:58.977060+0000","last_peered":"2024-01-16T14:59:58.977060+0000","last_clean":"2024-01-16T14:59:58.977060+0000","last_became_active":"2024-01-16T14:59:56.966640+0000","last_became_peered":"2024-01-16T14:59:56.966640+0000","last_unstale":"2024-01-16T14:59:58.977060+0000","last_undegraded":"2024-01-16T14:59:58.977060+0000","last_fullsized":"2024-01-16T14:59:58.977060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:19.845423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976503+0000","last_change":"2024-01-16T14:59:57.914604+0000","last_active":"2024-01-16T14:59:58.976503+0000","last_peered":"2024-01-16T14:59:58.976503+0000","last_clean":"2024-01-16T14:59:58.976503+0000","last_became_active":"2024-01-16T14:59:57.914326+0000","last_became_peered":"2024-01-16T14:59:57.914326+0000","last_unstale":"2024-01-16T14:59:58.976503+0000","last_undegraded":"2024-01-16T14:59:58.976503+0000","last_fullsized":"2024-01-16T14:59:58.976503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975252+0000","last_change":"2024-01-16T14:59:56.967597+0000","last_active":"2024-01-16T14:59:58.975252+0000","last_peered":"2024-01-16T14:59:58.975252+0000","last_clean":"2024-01-16T14:59:58.975252+0000","last_became_active":"2024-01-16T14:59:56.967030+0000","last_became_peered":"2024-01-16T14:59:56.967030+0000","last_unstale":"2024-01-16T14:59:58.975252+0000","last_undegraded":"2024-01-16T14:59:58.975252+0000","last_fullsized":"2024-01-16T14:59:58.975252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964053+0000","last_change":"2024-01-16T14:59:57.905418+0000","last_active":"2024-01-16T14:59:58.964053+0000","last_peered":"2024-01-16T14:59:58.964053+0000","last_clean":"2024-01-16T14:59:58.964053+0000","last_became_active":"2024-01-16T14:59:57.905057+0000","last_became_peered":"2024-01-16T14:59:57.905057+0000","last_unstale":"2024-01-16T14:59:58.964053+0000","last_undegraded":"2024-01-16T14:59:58.964053+0000","last_fullsized":"2024-01-16T14:59:58.964053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.903408+0000","last_change":"2024-01-16T14:59:56.966647+0000","last_active":"2024-01-16T14:59:57.903408+0000","last_peered":"2024-01-16T14:59:57.903408+0000","last_clean":"2024-01-16T14:59:57.903408+0000","last_became_active":"2024-01-16T14:59:56.966276+0000","last_became_peered":"2024-01-16T14:59:56.966276+0000","last_unstale":"2024-01-16T14:59:57.903408+0000","last_undegraded":"2024-01-16T14:59:57.903408+0000","last_fullsized":"2024-01-16T14:59:57.903408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:05.820349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976343+0000","last_change":"2024-01-16T14:59:57.923839+0000","last_active":"2024-01-16T14:59:58.976343+0000","last_peered":"2024-01-16T14:59:58.976343+0000","last_clean":"2024-01-16T14:59:58.976343+0000","last_became_active":"2024-01-16T14:59:57.923483+0000","last_became_peered":"2024-01-16T14:59:57.923483+0000","last_unstale":"2024-01-16T14:59:58.976343+0000","last_undegraded":"2024-01-16T14:59:58.976343+0000","last_fullsized":"2024-01-16T14:59:58.976343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976571+0000","last_change":"2024-01-16T14:59:56.969754+0000","last_active":"2024-01-16T14:59:58.976571+0000","last_peered":"2024-01-16T14:59:58.976571+0000","last_clean":"2024-01-16T14:59:58.976571+0000","last_became_active":"2024-01-16T14:59:56.969381+0000","last_became_peered":"2024-01-16T14:59:56.969381+0000","last_unstale":"2024-01-16T14:59:58.976571+0000","last_undegraded":"2024-01-16T14:59:58.976571+0000","last_fullsized":"2024-01-16T14:59:58.976571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974983+0000","last_change":"2024-01-16T14:59:57.914892+0000","last_active":"2024-01-16T14:59:58.974983+0000","last_peered":"2024-01-16T14:59:58.974983+0000","last_clean":"2024-01-16T14:59:58.974983+0000","last_became_active":"2024-01-16T14:59:57.913384+0000","last_became_peered":"2024-01-16T14:59:57.913384+0000","last_unstale":"2024-01-16T14:59:58.974983+0000","last_undegraded":"2024-01-16T14:59:58.974983+0000","last_fullsized":"2024-01-16T14:59:58.974983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.998301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964546+0000","last_change":"2024-01-16T14:59:56.960948+0000","last_active":"2024-01-16T14:59:58.964546+0000","last_peered":"2024-01-16T14:59:58.964546+0000","last_clean":"2024-01-16T14:59:58.964546+0000","last_became_active":"2024-01-16T14:59:56.960588+0000","last_became_peered":"2024-01-16T14:59:56.960588+0000","last_unstale":"2024-01-16T14:59:58.964546+0000","last_undegraded":"2024-01-16T14:59:58.964546+0000","last_fullsized":"2024-01-16T14:59:58.964546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885497+0000","last_change":"2024-01-16T14:59:57.911677+0000","last_active":"2024-01-16T14:59:58.885497+0000","last_peered":"2024-01-16T14:59:58.885497+0000","last_clean":"2024-01-16T14:59:58.885497+0000","last_became_active":"2024-01-16T14:59:57.911160+0000","last_became_peered":"2024-01-16T14:59:57.911160+0000","last_unstale":"2024-01-16T14:59:58.885497+0000","last_undegraded":"2024-01-16T14:59:58.885497+0000","last_fullsized":"2024-01-16T14:59:58.885497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:43.320590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.945676+0000","last_change":"2024-01-16T14:59:56.963872+0000","last_active":"2024-01-16T14:59:57.945676+0000","last_peered":"2024-01-16T14:59:57.945676+0000","last_clean":"2024-01-16T14:59:57.945676+0000","last_became_active":"2024-01-16T14:59:56.963545+0000","last_became_peered":"2024-01-16T14:59:56.963545+0000","last_unstale":"2024-01-16T14:59:57.945676+0000","last_undegraded":"2024-01-16T14:59:57.945676+0000","last_fullsized":"2024-01-16T14:59:57.945676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881456+0000","last_change":"2024-01-16T14:59:57.918001+0000","last_active":"2024-01-16T14:59:58.881456+0000","last_peered":"2024-01-16T14:59:58.881456+0000","last_clean":"2024-01-16T14:59:58.881456+0000","last_became_active":"2024-01-16T14:59:57.917646+0000","last_became_peered":"2024-01-16T14:59:57.917646+0000","last_unstale":"2024-01-16T14:59:58.881456+0000","last_undegraded":"2024-01-16T14:59:58.881456+0000","last_fullsized":"2024-01-16T14:59:58.881456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975183+0000","last_change":"2024-01-16T14:59:56.965453+0000","last_active":"2024-01-16T14:59:58.975183+0000","last_peered":"2024-01-16T14:59:58.975183+0000","last_clean":"2024-01-16T14:59:58.975183+0000","last_became_active":"2024-01-16T14:59:56.965153+0000","last_became_peered":"2024-01-16T14:59:56.965153+0000","last_unstale":"2024-01-16T14:59:58.975183+0000","last_undegraded":"2024-01-16T14:59:58.975183+0000","last_fullsized":"2024-01-16T14:59:58.975183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974193+0000","last_change":"2024-01-16T14:59:57.921295+0000","last_active":"2024-01-16T14:59:58.974193+0000","last_peered":"2024-01-16T14:59:58.974193+0000","last_clean":"2024-01-16T14:59:58.974193+0000","last_became_active":"2024-01-16T14:59:57.920961+0000","last_became_peered":"2024-01-16T14:59:57.920961+0000","last_unstale":"2024-01-16T14:59:58.974193+0000","last_undegraded":"2024-01-16T14:59:58.974193+0000","last_fullsized":"2024-01-16T14:59:58.974193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:09.224854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977499+0000","last_change":"2024-01-16T14:59:56.964539+0000","last_active":"2024-01-16T14:59:58.977499+0000","last_peered":"2024-01-16T14:59:58.977499+0000","last_clean":"2024-01-16T14:59:58.977499+0000","last_became_active":"2024-01-16T14:59:56.964337+0000","last_became_peered":"2024-01-16T14:59:56.964337+0000","last_unstale":"2024-01-16T14:59:58.977499+0000","last_undegraded":"2024-01-16T14:59:58.977499+0000","last_fullsized":"2024-01-16T14:59:58.977499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881748+0000","last_change":"2024-01-16T14:59:57.918457+0000","last_active":"2024-01-16T14:59:58.881748+0000","last_peered":"2024-01-16T14:59:58.881748+0000","last_clean":"2024-01-16T14:59:58.881748+0000","last_became_active":"2024-01-16T14:59:57.918278+0000","last_became_peered":"2024-01-16T14:59:57.918278+0000","last_unstale":"2024-01-16T14:59:58.881748+0000","last_undegraded":"2024-01-16T14:59:58.881748+0000","last_fullsized":"2024-01-16T14:59:58.881748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.900831+0000","last_change":"2024-01-16T14:59:56.969818+0000","last_active":"2024-01-16T14:59:57.900831+0000","last_peered":"2024-01-16T14:59:57.900831+0000","last_clean":"2024-01-16T14:59:57.900831+0000","last_became_active":"2024-01-16T14:59:56.969530+0000","last_became_peered":"2024-01-16T14:59:56.969530+0000","last_unstale":"2024-01-16T14:59:57.900831+0000","last_undegraded":"2024-01-16T14:59:57.900831+0000","last_fullsized":"2024-01-16T14:59:57.900831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.795006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974601+0000","last_change":"2024-01-16T14:59:57.907805+0000","last_active":"2024-01-16T14:59:58.974601+0000","last_peered":"2024-01-16T14:59:58.974601+0000","last_clean":"2024-01-16T14:59:58.974601+0000","last_became_active":"2024-01-16T14:59:57.907499+0000","last_became_peered":"2024-01-16T14:59:57.907499+0000","last_unstale":"2024-01-16T14:59:58.974601+0000","last_undegraded":"2024-01-16T14:59:58.974601+0000","last_fullsized":"2024-01-16T14:59:58.974601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973072+0000","last_change":"2024-01-16T14:59:56.950566+0000","last_active":"2024-01-16T14:59:58.973072+0000","last_peered":"2024-01-16T14:59:58.973072+0000","last_clean":"2024-01-16T14:59:58.973072+0000","last_became_active":"2024-01-16T14:59:56.949990+0000","last_became_peered":"2024-01-16T14:59:56.949990+0000","last_unstale":"2024-01-16T14:59:58.973072+0000","last_undegraded":"2024-01-16T14:59:58.973072+0000","last_fullsized":"2024-01-16T14:59:58.973072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:14.695598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880960+0000","last_change":"2024-01-16T14:59:57.918222+0000","last_active":"2024-01-16T14:59:58.880960+0000","last_peered":"2024-01-16T14:59:58.880960+0000","last_clean":"2024-01-16T14:59:58.880960+0000","last_became_active":"2024-01-16T14:59:57.917997+0000","last_became_peered":"2024-01-16T14:59:57.917997+0000","last_unstale":"2024-01-16T14:59:58.880960+0000","last_undegraded":"2024-01-16T14:59:58.880960+0000","last_fullsized":"2024-01-16T14:59:58.880960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.942520+0000","last_change":"2024-01-16T14:59:56.958392+0000","last_active":"2024-01-16T14:59:57.942520+0000","last_peered":"2024-01-16T14:59:57.942520+0000","last_clean":"2024-01-16T14:59:57.942520+0000","last_became_active":"2024-01-16T14:59:56.955933+0000","last_became_peered":"2024-01-16T14:59:56.955933+0000","last_unstale":"2024-01-16T14:59:57.942520+0000","last_undegraded":"2024-01-16T14:59:57.942520+0000","last_fullsized":"2024-01-16T14:59:57.942520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:50.131057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883242+0000","last_change":"2024-01-16T14:59:57.917801+0000","last_active":"2024-01-16T14:59:58.883242+0000","last_peered":"2024-01-16T14:59:58.883242+0000","last_clean":"2024-01-16T14:59:58.883242+0000","last_became_active":"2024-01-16T14:59:57.917470+0000","last_became_peered":"2024-01-16T14:59:57.917470+0000","last_unstale":"2024-01-16T14:59:58.883242+0000","last_undegraded":"2024-01-16T14:59:58.883242+0000","last_fullsized":"2024-01-16T14:59:58.883242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:34.142895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974235+0000","last_change":"2024-01-16T14:59:56.954099+0000","last_active":"2024-01-16T14:59:58.974235+0000","last_peered":"2024-01-16T14:59:58.974235+0000","last_clean":"2024-01-16T14:59:58.974235+0000","last_became_active":"2024-01-16T14:59:56.953811+0000","last_became_peered":"2024-01-16T14:59:56.953811+0000","last_unstale":"2024-01-16T14:59:58.974235+0000","last_undegraded":"2024-01-16T14:59:58.974235+0000","last_fullsized":"2024-01-16T14:59:58.974235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:08.095887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883459+0000","last_change":"2024-01-16T14:59:57.921075+0000","last_active":"2024-01-16T14:59:58.883459+0000","last_peered":"2024-01-16T14:59:58.883459+0000","last_clean":"2024-01-16T14:59:58.883459+0000","last_became_active":"2024-01-16T14:59:57.920649+0000","last_became_peered":"2024-01-16T14:59:57.920649+0000","last_unstale":"2024-01-16T14:59:58.883459+0000","last_undegraded":"2024-01-16T14:59:58.883459+0000","last_fullsized":"2024-01-16T14:59:58.883459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:30.432751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964505+0000","last_change":"2024-01-16T14:59:56.965927+0000","last_active":"2024-01-16T14:59:58.964505+0000","last_peered":"2024-01-16T14:59:58.964505+0000","last_clean":"2024-01-16T14:59:58.964505+0000","last_became_active":"2024-01-16T14:59:56.965627+0000","last_became_peered":"2024-01-16T14:59:56.965627+0000","last_unstale":"2024-01-16T14:59:58.964505+0000","last_undegraded":"2024-01-16T14:59:58.964505+0000","last_fullsized":"2024-01-16T14:59:58.964505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:11.715798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975760+0000","last_change":"2024-01-16T14:59:57.919926+0000","last_active":"2024-01-16T14:59:58.975760+0000","last_peered":"2024-01-16T14:59:58.975760+0000","last_clean":"2024-01-16T14:59:58.975760+0000","last_became_active":"2024-01-16T14:59:57.919380+0000","last_became_peered":"2024-01-16T14:59:57.919380+0000","last_unstale":"2024-01-16T14:59:58.975760+0000","last_undegraded":"2024-01-16T14:59:58.975760+0000","last_fullsized":"2024-01-16T14:59:58.975760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973702+0000","last_change":"2024-01-16T14:59:56.961433+0000","last_active":"2024-01-16T14:59:58.973702+0000","last_peered":"2024-01-16T14:59:58.973702+0000","last_clean":"2024-01-16T14:59:58.973702+0000","last_became_active":"2024-01-16T14:59:56.961175+0000","last_became_peered":"2024-01-16T14:59:56.961175+0000","last_unstale":"2024-01-16T14:59:58.973702+0000","last_undegraded":"2024-01-16T14:59:58.973702+0000","last_fullsized":"2024-01-16T14:59:58.973702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:34.084632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882600+0000","last_change":"2024-01-16T14:59:57.912741+0000","last_active":"2024-01-16T14:59:58.882600+0000","last_peered":"2024-01-16T14:59:58.882600+0000","last_clean":"2024-01-16T14:59:58.882600+0000","last_became_active":"2024-01-16T14:59:57.908529+0000","last_became_peered":"2024-01-16T14:59:57.908529+0000","last_unstale":"2024-01-16T14:59:58.882600+0000","last_undegraded":"2024-01-16T14:59:58.882600+0000","last_fullsized":"2024-01-16T14:59:58.882600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.907222+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T14:59:57.907222+0000","last_peered":"2024-01-16T14:59:57.907222+0000","last_clean":"2024-01-16T14:59:57.907222+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T14:59:57.907222+0000","last_undegraded":"2024-01-16T14:59:57.907222+0000","last_fullsized":"2024-01-16T14:59:57.907222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974661+0000","last_change":"2024-01-16T14:59:56.964503+0000","last_active":"2024-01-16T14:59:58.974661+0000","last_peered":"2024-01-16T14:59:58.974661+0000","last_clean":"2024-01-16T14:59:58.974661+0000","last_became_active":"2024-01-16T14:59:56.964165+0000","last_became_peered":"2024-01-16T14:59:56.964165+0000","last_unstale":"2024-01-16T14:59:58.974661+0000","last_undegraded":"2024-01-16T14:59:58.974661+0000","last_fullsized":"2024-01-16T14:59:58.974661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:39.083282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977186+0000","last_change":"2024-01-16T14:59:57.916509+0000","last_active":"2024-01-16T14:59:58.977186+0000","last_peered":"2024-01-16T14:59:58.977186+0000","last_clean":"2024-01-16T14:59:58.977186+0000","last_became_active":"2024-01-16T14:59:57.915770+0000","last_became_peered":"2024-01-16T14:59:57.915770+0000","last_unstale":"2024-01-16T14:59:58.977186+0000","last_undegraded":"2024-01-16T14:59:58.977186+0000","last_fullsized":"2024-01-16T14:59:58.977186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880642+0000","last_change":"2024-01-16T14:59:56.965724+0000","last_active":"2024-01-16T14:59:58.880642+0000","last_peered":"2024-01-16T14:59:58.880642+0000","last_clean":"2024-01-16T14:59:58.880642+0000","last_became_active":"2024-01-16T14:59:56.965458+0000","last_became_peered":"2024-01-16T14:59:56.965458+0000","last_unstale":"2024-01-16T14:59:58.880642+0000","last_undegraded":"2024-01-16T14:59:58.880642+0000","last_fullsized":"2024-01-16T14:59:58.880642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973635+0000","last_change":"2024-01-16T14:59:57.912107+0000","last_active":"2024-01-16T14:59:58.973635+0000","last_peered":"2024-01-16T14:59:58.973635+0000","last_clean":"2024-01-16T14:59:58.973635+0000","last_became_active":"2024-01-16T14:59:57.911776+0000","last_became_peered":"2024-01-16T14:59:57.911776+0000","last_unstale":"2024-01-16T14:59:58.973635+0000","last_undegraded":"2024-01-16T14:59:58.973635+0000","last_fullsized":"2024-01-16T14:59:58.973635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:33.290097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973813+0000","last_change":"2024-01-16T14:59:56.956041+0000","last_active":"2024-01-16T14:59:58.973813+0000","last_peered":"2024-01-16T14:59:58.973813+0000","last_clean":"2024-01-16T14:59:58.973813+0000","last_became_active":"2024-01-16T14:59:56.955669+0000","last_became_peered":"2024-01-16T14:59:56.955669+0000","last_unstale":"2024-01-16T14:59:58.973813+0000","last_undegraded":"2024-01-16T14:59:58.973813+0000","last_fullsized":"2024-01-16T14:59:58.973813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964006+0000","last_change":"2024-01-16T14:59:57.905614+0000","last_active":"2024-01-16T14:59:58.964006+0000","last_peered":"2024-01-16T14:59:58.964006+0000","last_clean":"2024-01-16T14:59:58.964006+0000","last_became_active":"2024-01-16T14:59:57.905277+0000","last_became_peered":"2024-01-16T14:59:57.905277+0000","last_unstale":"2024-01-16T14:59:58.964006+0000","last_undegraded":"2024-01-16T14:59:58.964006+0000","last_fullsized":"2024-01-16T14:59:58.964006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977517+0000","last_change":"2024-01-16T14:59:56.956330+0000","last_active":"2024-01-16T14:59:58.977517+0000","last_peered":"2024-01-16T14:59:58.977517+0000","last_clean":"2024-01-16T14:59:58.977517+0000","last_became_active":"2024-01-16T14:59:56.955936+0000","last_became_peered":"2024-01-16T14:59:56.955936+0000","last_unstale":"2024-01-16T14:59:58.977517+0000","last_undegraded":"2024-01-16T14:59:58.977517+0000","last_fullsized":"2024-01-16T14:59:58.977517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:53.098539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884894+0000","last_change":"2024-01-16T14:59:57.911172+0000","last_active":"2024-01-16T14:59:58.884894+0000","last_peered":"2024-01-16T14:59:58.884894+0000","last_clean":"2024-01-16T14:59:58.884894+0000","last_became_active":"2024-01-16T14:59:57.910403+0000","last_became_peered":"2024-01-16T14:59:57.910403+0000","last_unstale":"2024-01-16T14:59:58.884894+0000","last_undegraded":"2024-01-16T14:59:58.884894+0000","last_fullsized":"2024-01-16T14:59:58.884894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:23.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964455+0000","last_change":"2024-01-16T14:59:56.955294+0000","last_active":"2024-01-16T14:59:58.964455+0000","last_peered":"2024-01-16T14:59:58.964455+0000","last_clean":"2024-01-16T14:59:58.964455+0000","last_became_active":"2024-01-16T14:59:56.953316+0000","last_became_peered":"2024-01-16T14:59:56.953316+0000","last_unstale":"2024-01-16T14:59:58.964455+0000","last_undegraded":"2024-01-16T14:59:58.964455+0000","last_fullsized":"2024-01-16T14:59:58.964455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976348+0000","last_change":"2024-01-16T14:59:57.917538+0000","last_active":"2024-01-16T14:59:58.976348+0000","last_peered":"2024-01-16T14:59:58.976348+0000","last_clean":"2024-01-16T14:59:58.976348+0000","last_became_active":"2024-01-16T14:59:57.917208+0000","last_became_peered":"2024-01-16T14:59:57.917208+0000","last_unstale":"2024-01-16T14:59:58.976348+0000","last_undegraded":"2024-01-16T14:59:58.976348+0000","last_fullsized":"2024-01-16T14:59:58.976348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882243+0000","last_change":"2024-01-16T14:59:56.963301+0000","last_active":"2024-01-16T14:59:58.882243+0000","last_peered":"2024-01-16T14:59:58.882243+0000","last_clean":"2024-01-16T14:59:58.882243+0000","last_became_active":"2024-01-16T14:59:56.962999+0000","last_became_peered":"2024-01-16T14:59:56.962999+0000","last_unstale":"2024-01-16T14:59:58.882243+0000","last_undegraded":"2024-01-16T14:59:58.882243+0000","last_fullsized":"2024-01-16T14:59:58.882243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:34.876313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964280+0000","last_change":"2024-01-16T14:59:57.901887+0000","last_active":"2024-01-16T14:59:58.964280+0000","last_peered":"2024-01-16T14:59:58.964280+0000","last_clean":"2024-01-16T14:59:58.964280+0000","last_became_active":"2024-01-16T14:59:57.901474+0000","last_became_peered":"2024-01-16T14:59:57.901474+0000","last_unstale":"2024-01-16T14:59:58.964280+0000","last_undegraded":"2024-01-16T14:59:58.964280+0000","last_fullsized":"2024-01-16T14:59:58.964280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964917+0000","last_change":"2024-01-16T14:59:56.967570+0000","last_active":"2024-01-16T14:59:58.964917+0000","last_peered":"2024-01-16T14:59:58.964917+0000","last_clean":"2024-01-16T14:59:58.964917+0000","last_became_active":"2024-01-16T14:59:56.967245+0000","last_became_peered":"2024-01-16T14:59:56.967245+0000","last_unstale":"2024-01-16T14:59:58.964917+0000","last_undegraded":"2024-01-16T14:59:58.964917+0000","last_fullsized":"2024-01-16T14:59:58.964917+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:50.405789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.979264+0000","last_change":"2024-01-16T14:59:57.914280+0000","last_active":"2024-01-16T14:59:58.979264+0000","last_peered":"2024-01-16T14:59:58.979264+0000","last_clean":"2024-01-16T14:59:58.979264+0000","last_became_active":"2024-01-16T14:59:57.909069+0000","last_became_peered":"2024-01-16T14:59:57.909069+0000","last_unstale":"2024-01-16T14:59:58.979264+0000","last_undegraded":"2024-01-16T14:59:58.979264+0000","last_fullsized":"2024-01-16T14:59:58.979264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:45:32.241461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.943628+0000","last_change":"2024-01-16T14:59:56.963783+0000","last_active":"2024-01-16T14:59:57.943628+0000","last_peered":"2024-01-16T14:59:57.943628+0000","last_clean":"2024-01-16T14:59:57.943628+0000","last_became_active":"2024-01-16T14:59:56.963167+0000","last_became_peered":"2024-01-16T14:59:56.963167+0000","last_unstale":"2024-01-16T14:59:57.943628+0000","last_undegraded":"2024-01-16T14:59:57.943628+0000","last_fullsized":"2024-01-16T14:59:57.943628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974544+0000","last_change":"2024-01-16T14:59:57.907693+0000","last_active":"2024-01-16T14:59:58.974544+0000","last_peered":"2024-01-16T14:59:58.974544+0000","last_clean":"2024-01-16T14:59:58.974544+0000","last_became_active":"2024-01-16T14:59:57.907365+0000","last_became_peered":"2024-01-16T14:59:57.907365+0000","last_unstale":"2024-01-16T14:59:58.974544+0000","last_undegraded":"2024-01-16T14:59:58.974544+0000","last_fullsized":"2024-01-16T14:59:58.974544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:31.975241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881061+0000","last_change":"2024-01-16T14:59:56.964456+0000","last_active":"2024-01-16T14:59:58.881061+0000","last_peered":"2024-01-16T14:59:58.881061+0000","last_clean":"2024-01-16T14:59:58.881061+0000","last_became_active":"2024-01-16T14:59:56.964105+0000","last_became_peered":"2024-01-16T14:59:56.964105+0000","last_unstale":"2024-01-16T14:59:58.881061+0000","last_undegraded":"2024-01-16T14:59:58.881061+0000","last_fullsized":"2024-01-16T14:59:58.881061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884695+0000","last_change":"2024-01-16T14:59:57.909622+0000","last_active":"2024-01-16T14:59:58.884695+0000","last_peered":"2024-01-16T14:59:58.884695+0000","last_clean":"2024-01-16T14:59:58.884695+0000","last_became_active":"2024-01-16T14:59:57.905809+0000","last_became_peered":"2024-01-16T14:59:57.905809+0000","last_unstale":"2024-01-16T14:59:58.884695+0000","last_undegraded":"2024-01-16T14:59:58.884695+0000","last_fullsized":"2024-01-16T14:59:58.884695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:30.677552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964306+0000","last_change":"2024-01-16T14:59:56.961402+0000","last_active":"2024-01-16T14:59:58.964306+0000","last_peered":"2024-01-16T14:59:58.964306+0000","last_clean":"2024-01-16T14:59:58.964306+0000","last_became_active":"2024-01-16T14:59:56.959909+0000","last_became_peered":"2024-01-16T14:59:56.959909+0000","last_unstale":"2024-01-16T14:59:58.964306+0000","last_undegraded":"2024-01-16T14:59:58.964306+0000","last_fullsized":"2024-01-16T14:59:58.964306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:00.188907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976651+0000","last_change":"2024-01-16T14:59:57.902379+0000","last_active":"2024-01-16T14:59:58.976651+0000","last_peered":"2024-01-16T14:59:58.976651+0000","last_clean":"2024-01-16T14:59:58.976651+0000","last_became_active":"2024-01-16T14:59:57.902002+0000","last_became_peered":"2024-01-16T14:59:57.902002+0000","last_unstale":"2024-01-16T14:59:58.976651+0000","last_undegraded":"2024-01-16T14:59:58.976651+0000","last_fullsized":"2024-01-16T14:59:58.976651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.963988+0000","last_change":"2024-01-16T14:59:56.956035+0000","last_active":"2024-01-16T14:59:58.963988+0000","last_peered":"2024-01-16T14:59:58.963988+0000","last_clean":"2024-01-16T14:59:58.963988+0000","last_became_active":"2024-01-16T14:59:56.955666+0000","last_became_peered":"2024-01-16T14:59:56.955666+0000","last_unstale":"2024-01-16T14:59:58.963988+0000","last_undegraded":"2024-01-16T14:59:58.963988+0000","last_fullsized":"2024-01-16T14:59:58.963988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977506+0000","last_change":"2024-01-16T14:59:57.913992+0000","last_active":"2024-01-16T14:59:58.977506+0000","last_peered":"2024-01-16T14:59:58.977506+0000","last_clean":"2024-01-16T14:59:58.977506+0000","last_became_active":"2024-01-16T14:59:57.913639+0000","last_became_peered":"2024-01-16T14:59:57.913639+0000","last_unstale":"2024-01-16T14:59:58.977506+0000","last_undegraded":"2024-01-16T14:59:58.977506+0000","last_fullsized":"2024-01-16T14:59:58.977506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882012+0000","last_change":"2024-01-16T14:59:56.970741+0000","last_active":"2024-01-16T14:59:58.882012+0000","last_peered":"2024-01-16T14:59:58.882012+0000","last_clean":"2024-01-16T14:59:58.882012+0000","last_became_active":"2024-01-16T14:59:56.970075+0000","last_became_peered":"2024-01-16T14:59:56.970075+0000","last_unstale":"2024-01-16T14:59:58.882012+0000","last_undegraded":"2024-01-16T14:59:58.882012+0000","last_fullsized":"2024-01-16T14:59:58.882012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:21.992849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974326+0000","last_change":"2024-01-16T14:59:57.906046+0000","last_active":"2024-01-16T14:59:58.974326+0000","last_peered":"2024-01-16T14:59:58.974326+0000","last_clean":"2024-01-16T14:59:58.974326+0000","last_became_active":"2024-01-16T14:59:57.905751+0000","last_became_peered":"2024-01-16T14:59:57.905751+0000","last_unstale":"2024-01-16T14:59:58.974326+0000","last_undegraded":"2024-01-16T14:59:58.974326+0000","last_fullsized":"2024-01-16T14:59:58.974326+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976133+0000","last_change":"2024-01-16T14:59:57.920773+0000","last_active":"2024-01-16T14:59:58.976133+0000","last_peered":"2024-01-16T14:59:58.976133+0000","last_clean":"2024-01-16T14:59:58.976133+0000","last_became_active":"2024-01-16T14:59:57.920488+0000","last_became_peered":"2024-01-16T14:59:57.920488+0000","last_unstale":"2024-01-16T14:59:58.976133+0000","last_undegraded":"2024-01-16T14:59:58.976133+0000","last_fullsized":"2024-01-16T14:59:58.976133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881475+0000","last_change":"2024-01-16T14:59:56.968586+0000","last_active":"2024-01-16T14:59:58.881475+0000","last_peered":"2024-01-16T14:59:58.881475+0000","last_clean":"2024-01-16T14:59:58.881475+0000","last_became_active":"2024-01-16T14:59:56.967923+0000","last_became_peered":"2024-01-16T14:59:56.967923+0000","last_unstale":"2024-01-16T14:59:58.881475+0000","last_undegraded":"2024-01-16T14:59:58.881475+0000","last_fullsized":"2024-01-16T14:59:58.881475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975571+0000","last_change":"2024-01-16T14:59:57.919788+0000","last_active":"2024-01-16T14:59:58.975571+0000","last_peered":"2024-01-16T14:59:58.975571+0000","last_clean":"2024-01-16T14:59:58.975571+0000","last_became_active":"2024-01-16T14:59:57.918939+0000","last_became_peered":"2024-01-16T14:59:57.918939+0000","last_unstale":"2024-01-16T14:59:58.975571+0000","last_undegraded":"2024-01-16T14:59:58.975571+0000","last_fullsized":"2024-01-16T14:59:58.975571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881694+0000","last_change":"2024-01-16T14:59:56.969250+0000","last_active":"2024-01-16T14:59:58.881694+0000","last_peered":"2024-01-16T14:59:58.881694+0000","last_clean":"2024-01-16T14:59:58.881694+0000","last_became_active":"2024-01-16T14:59:56.968877+0000","last_became_peered":"2024-01-16T14:59:56.968877+0000","last_unstale":"2024-01-16T14:59:58.881694+0000","last_undegraded":"2024-01-16T14:59:58.881694+0000","last_fullsized":"2024-01-16T14:59:58.881694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975133+0000","last_change":"2024-01-16T14:59:57.909922+0000","last_active":"2024-01-16T14:59:58.975133+0000","last_peered":"2024-01-16T14:59:58.975133+0000","last_clean":"2024-01-16T14:59:58.975133+0000","last_became_active":"2024-01-16T14:59:57.908518+0000","last_became_peered":"2024-01-16T14:59:57.908518+0000","last_unstale":"2024-01-16T14:59:58.975133+0000","last_undegraded":"2024-01-16T14:59:58.975133+0000","last_fullsized":"2024-01-16T14:59:58.975133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978143+0000","last_change":"2024-01-16T14:59:56.956482+0000","last_active":"2024-01-16T14:59:58.978143+0000","last_peered":"2024-01-16T14:59:58.978143+0000","last_clean":"2024-01-16T14:59:58.978143+0000","last_became_active":"2024-01-16T14:59:56.955091+0000","last_became_peered":"2024-01-16T14:59:56.955091+0000","last_unstale":"2024-01-16T14:59:58.978143+0000","last_undegraded":"2024-01-16T14:59:58.978143+0000","last_fullsized":"2024-01-16T14:59:58.978143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977730+0000","last_change":"2024-01-16T14:59:57.913479+0000","last_active":"2024-01-16T14:59:58.977730+0000","last_peered":"2024-01-16T14:59:58.977730+0000","last_clean":"2024-01-16T14:59:58.977730+0000","last_became_active":"2024-01-16T14:59:57.911939+0000","last_became_peered":"2024-01-16T14:59:57.911939+0000","last_unstale":"2024-01-16T14:59:58.977730+0000","last_undegraded":"2024-01-16T14:59:58.977730+0000","last_fullsized":"2024-01-16T14:59:58.977730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:26.642001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975233+0000","last_change":"2024-01-16T14:59:56.957873+0000","last_active":"2024-01-16T14:59:58.975233+0000","last_peered":"2024-01-16T14:59:58.975233+0000","last_clean":"2024-01-16T14:59:58.975233+0000","last_became_active":"2024-01-16T14:59:56.957287+0000","last_became_peered":"2024-01-16T14:59:56.957287+0000","last_unstale":"2024-01-16T14:59:58.975233+0000","last_undegraded":"2024-01-16T14:59:58.975233+0000","last_fullsized":"2024-01-16T14:59:58.975233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509444,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.675}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.141}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2629999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4770000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.746}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9359999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.028}]}]},{"osd":4,"up_from":15,"seq":64424509446,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92300000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2010000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.429}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6659999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542148,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":669118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.496}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2829999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6299999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542149,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.395}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.125}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2350000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542149,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1970000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0920000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.579}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.397}]}]},{"osd":2,"up_from":14,"seq":60129542148,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":78168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.095}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2110000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.256}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8029999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.9670000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.774}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":78328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95599999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.101}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4870000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.389}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.897}]}]},{"osd":0,"up_from":14,"seq":60129542148,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":669174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.153}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:02.973 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:00:02.973 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-01-16T15:00:02.973 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-01-16T15:00:02.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-01-16T15:00:02.973 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-01-16T15:00:03.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.261+0000 7fce85a00640 1 Processor -- start 2024-01-16T15:00:03.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.261+0000 7fce85a00640 1 -- start start 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.262+0000 7fce85a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fce8013df80 0x7fce8015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.262+0000 7fce85a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.262+0000 7fce85a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce8015f9c0 0x7fce800cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce85a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8006c470 con 0x7fce8013df80 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce85a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce8006c750 con 0x7fce8015f9c0 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce85a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce8006ca30 con 0x7fce8013e360 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fce8013df80 0x7fce8015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fce8013df80 0x7fce8015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34754/0 (socket says 172.21.15.53:34754) 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 -- 172.21.15.53:0/4232258305 learned_addr learned my addr 172.21.15.53:0/4232258305 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:03.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7e000640 1 --2- 172.21.15.53:0/4232258305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7f400640 1 --2- 172.21.15.53:0/4232258305 >> v2:172.21.15.146:3300/0 conn(0x7fce8015f9c0 0x7fce800cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 -- 172.21.15.53:0/4232258305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 msgr2=0x7fce800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4232258305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce800cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 -- 172.21.15.53:0/4232258305 >> v2:172.21.15.146:3300/0 conn(0x7fce8015f9c0 msgr2=0x7fce800cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.263+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4232258305 >> v2:172.21.15.146:3300/0 conn(0x7fce8015f9c0 0x7fce800cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.264+0000 7fce7ea00640 1 -- 172.21.15.53:0/4232258305 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8006cd10 con 0x7fce8013df80 2024-01-16T15:00:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.265+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4232258305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8013df80 0x7fce8015bee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce8006e6d0 tx=0x7fce7002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f92380b72f4c6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.265+0000 7fce7d600640 1 -- 172.21.15.53:0/4232258305 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7002ca80 con 0x7fce8013df80 2024-01-16T15:00:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.265+0000 7fce7d600640 1 -- 172.21.15.53:0/4232258305 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7002cbe0 con 0x7fce8013df80 2024-01-16T15:00:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.265+0000 7fce7d600640 1 -- 172.21.15.53:0/4232258305 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce700343f0 con 0x7fce8013df80 2024-01-16T15:00:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.265+0000 7fce85a00640 1 -- 172.21.15.53:0/4232258305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8013df80 msgr2=0x7fce8015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.265+0000 7fce85a00640 1 --2- 172.21.15.53:0/4232258305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8013df80 0x7fce8015bee0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce8006e6d0 tx=0x7fce7002bee0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 -- 172.21.15.53:0/4232258305 shutdown_connections 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 --2- 172.21.15.53:0/4232258305 >> v2:172.21.15.146:3300/0 conn(0x7fce8015f9c0 0x7fce800cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 --2- 172.21.15.53:0/4232258305 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce800cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 --2- 172.21.15.53:0/4232258305 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8013df80 0x7fce8015bee0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 -- 172.21.15.53:0/4232258305 >> 172.21.15.53:0/4232258305 conn(0x7fce8006a800 msgr2=0x7fce8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 -- 172.21.15.53:0/4232258305 shutdown_connections 2024-01-16T15:00:03.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.266+0000 7fce85a00640 1 -- 172.21.15.53:0/4232258305 wait complete. 2024-01-16T15:00:03.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.268+0000 7fce85a00640 1 Processor -- start 2024-01-16T15:00:03.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.268+0000 7fce85a00640 1 -- start start 2024-01-16T15:00:03.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce85a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce8013df80 0x7fce801516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce85a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce80151c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce85a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 0x7fce801f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce85a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800af300 con 0x7fce8015f9c0 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce85a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce800aea60 con 0x7fce8013df80 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce85a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce80154890 con 0x7fce8013e360 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce7f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 0x7fce801f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce7f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 0x7fce801f2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34758/0 (socket says 172.21.15.53:34758) 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce7f400640 1 -- 172.21.15.53:0/4070156346 learned_addr learned my addr 172.21.15.53:0/4070156346 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.269+0000 7fce7e000640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce80151c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce8013df80 0x7fce801516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7f400640 1 -- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 msgr2=0x7fce80151c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7f400640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce80151c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7f400640 1 -- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce8013df80 msgr2=0x7fce801516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7f400640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce8013df80 0x7fce801516d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7f400640 1 -- 172.21.15.53:0/4070156346 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce70009b30 con 0x7fce8015f9c0 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce8013df80 0x7fce801516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce7f400640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 0x7fce801f2430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce7800d800 tx=0x7fce7800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7800b6c0 con 0x7fce8015f9c0 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7800b840 con 0x7fce8015f9c0 2024-01-16T15:00:03.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7800bbf0 con 0x7fce8015f9c0 2024-01-16T15:00:03.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce800aecb0 con 0x7fce8015f9c0 2024-01-16T15:00:03.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.270+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce800703c0 con 0x7fce8015f9c0 2024-01-16T15:00:03.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.272+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce800af4b0 con 0x7fce8015f9c0 2024-01-16T15:00:03.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.272+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fce7800bd50 con 0x7fce8015f9c0 2024-01-16T15:00:03.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.273+0000 7fce76a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce5806b2c0 0x7fce5806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.273+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce7808a1f0 con 0x7fce8015f9c0 2024-01-16T15:00:03.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.274+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce5806b2c0 0x7fce5806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.275+0000 7fce7ea00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce5806b2c0 0x7fce5806d780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce70035000 tx=0x7fce70030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.278+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce800af4b0 con 0x7fce8015f9c0 2024-01-16T15:00:03.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.524+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fce800af4b0 con 0x7fce8015f9c0 2024-01-16T15:00:03.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.529+0000 7fce76a00640 1 -- 172.21.15.53:0/4070156346 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171564 (secure 0 0 0) 0x7fce800af4b0 con 0x7fce8015f9c0 2024-01-16T15:00:03.512 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce5806b2c0 msgr2=0x7fce5806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce5806b2c0 0x7fce5806d780 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fce70035000 tx=0x7fce70030000 comp rx=0 tx=0).stop 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 msgr2=0x7fce801f2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 0x7fce801f2430 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fce7800d800 tx=0x7fce7800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 shutdown_connections 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce5806b2c0 0x7fce5806d780 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce8015f9c0 0x7fce801f2430 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce8013e360 0x7fce80151c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 --2- 172.21.15.53:0/4070156346 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce8013df80 0x7fce801516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 >> 172.21.15.53:0/4070156346 conn(0x7fce8006a800 msgr2=0x7fce8013f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 shutdown_connections 2024-01-16T15:00:03.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.534+0000 7fce85a00640 1 -- 172.21.15.53:0/4070156346 wait complete. 2024-01-16T15:00:03.535 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":5,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6832","nonce":2200557793},{"type":"v1","addr":"172.21.15.146:6833","nonce":2200557793}]},"active_addr":"172.21.15.146:6833/2200557793","active_change":"2024-01-16T14:59:38.145349+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4107,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"clone_delay_every_regfile":{"name":"clone_delay_every_regfile","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Sleep for specified seconds after copying a regfile","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":1878088862}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2974908892}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":702425156}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":3529409252}]}]} 2024-01-16T15:00:03.538 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-01-16T15:00:03.538 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-01-16T15:00:03.539 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:03.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.777+0000 7f9c2b200640 1 Processor -- start 2024-01-16T15:00:03.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.777+0000 7f9c2b200640 1 -- start start 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c2b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c2c0cc500 0x7f9c2c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c2b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9c2c06d160 0x7f9c2c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c2b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c2c070e50 con 0x7f9c2c143c40 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c2b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c0723c0 con 0x7f9c2c0cc500 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c2b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c2c0cced0 con 0x7f9c2c06d160 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c21200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c2c0cc500 0x7f9c2c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c21200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c2c0cc500 0x7f9c2c0cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34762/0 (socket says 172.21.15.53:34762) 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.779+0000 7f9c21200640 1 -- 172.21.15.53:0/2040039241 learned_addr learned my addr 172.21.15.53:0/2040039241 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:03.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.780+0000 7f9c21200640 1 -- 172.21.15.53:0/2040039241 >> v2:172.21.15.146:3300/0 conn(0x7f9c2c06d160 msgr2=0x7f9c2c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.780+0000 7f9c21200640 1 --2- 172.21.15.53:0/2040039241 >> v2:172.21.15.146:3300/0 conn(0x7f9c2c06d160 0x7f9c2c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.780+0000 7f9c21200640 1 -- 172.21.15.53:0/2040039241 >> v1:172.21.15.146:6789/0 conn(0x7f9c2c143c40 legacy=0x7f9c2c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:03.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.780+0000 7f9c21200640 1 -- 172.21.15.53:0/2040039241 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c144020 con 0x7f9c2c0cc500 2024-01-16T15:00:03.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.781+0000 7f9c21200640 1 --2- 172.21.15.53:0/2040039241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c0cc500 0x7f9c2c0cc990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c070690 tx=0x7f9c1402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a2beea3cbd2acf8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.781+0000 7f9c29400640 1 -- 172.21.15.53:0/2040039241 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c140317e0 con 0x7f9c2c0cc500 2024-01-16T15:00:03.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.781+0000 7f9c29400640 1 -- 172.21.15.53:0/2040039241 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14031940 con 0x7f9c2c0cc500 2024-01-16T15:00:03.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.781+0000 7f9c29400640 1 -- 172.21.15.53:0/2040039241 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14031cd0 con 0x7f9c2c0cc500 2024-01-16T15:00:03.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 -- 172.21.15.53:0/2040039241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c0cc500 msgr2=0x7f9c2c0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 --2- 172.21.15.53:0/2040039241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c0cc500 0x7f9c2c0cc990 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c070690 tx=0x7f9c1402b880 comp rx=0 tx=0).stop 2024-01-16T15:00:03.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 -- 172.21.15.53:0/2040039241 shutdown_connections 2024-01-16T15:00:03.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 --2- 172.21.15.53:0/2040039241 >> v2:172.21.15.146:3300/0 conn(0x7f9c2c06d160 0x7f9c2c06d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 --2- 172.21.15.53:0/2040039241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c0cc500 0x7f9c2c0cc990 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 -- 172.21.15.53:0/2040039241 >> 172.21.15.53:0/2040039241 conn(0x7f9c2c06a800 msgr2=0x7f9c2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 -- 172.21.15.53:0/2040039241 shutdown_connections 2024-01-16T15:00:03.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.782+0000 7f9c2b200640 1 -- 172.21.15.53:0/2040039241 wait complete. 2024-01-16T15:00:03.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.784+0000 7f9c2b200640 1 Processor -- start 2024-01-16T15:00:03.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.785+0000 7f9c2b200640 1 -- start start 2024-01-16T15:00:03.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c2c06d160 0x7f9c2c13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2c0cc500 0x7f9c2c158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 0x7f9c2c158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c141230 con 0x7f9c2c143c40 2024-01-16T15:00:03.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c13e3a0 con 0x7f9c2c06d160 2024-01-16T15:00:03.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c2c13e680 con 0x7f9c2c0cc500 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 0x7f9c2c158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 0x7f9c2c158ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34778/0 (socket says 172.21.15.53:34778) 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.786+0000 7f9c2a800640 1 -- 172.21.15.53:0/875833014 learned_addr learned my addr 172.21.15.53:0/875833014 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c21200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2c0cc500 0x7f9c2c158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c29e00640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c2c06d160 0x7f9c2c13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2a800640 1 -- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2c0cc500 msgr2=0x7f9c2c158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2a800640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2c0cc500 0x7f9c2c158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2a800640 1 -- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c2c06d160 msgr2=0x7f9c2c13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2a800640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c2c06d160 0x7f9c2c13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2a800640 1 -- 172.21.15.53:0/875833014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140094d0 con 0x7f9c2c143c40 2024-01-16T15:00:03.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2a800640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 0x7f9c2c158ff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009fd0 tx=0x7f9c2400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24011830 con 0x7f9c2c143c40 2024-01-16T15:00:03.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c24011990 con 0x7f9c2c143c40 2024-01-16T15:00:03.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24011d20 con 0x7f9c2c143c40 2024-01-16T15:00:03.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.787+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c2c13e960 con 0x7f9c2c143c40 2024-01-16T15:00:03.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.788+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c2c13cb30 con 0x7f9c2c143c40 2024-01-16T15:00:03.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.789+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c2c1413a0 con 0x7f9c2c143c40 2024-01-16T15:00:03.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.790+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9c2401b460 con 0x7f9c2c143c40 2024-01-16T15:00:03.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.790+0000 7f9c22a00640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c0406b2c0 0x7f9c0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:03.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.791+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9c24089960 con 0x7f9c2c143c40 2024-01-16T15:00:03.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.791+0000 7f9c29e00640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c0406b2c0 0x7f9c0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:03.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.793+0000 7f9c29e00640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c0406b2c0 0x7f9c0406d780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c15ac20 tx=0x7f9c1c0094d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:03.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.796+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c240582f0 con 0x7f9c2c143c40 2024-01-16T15:00:03.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.993+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9c2c067450 con 0x7f9c2c143c40 2024-01-16T15:00:03.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.995+0000 7f9c22a00640 1 -- 172.21.15.53:0/875833014 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f9c24057e80 con 0x7f9c2c143c40 2024-01-16T15:00:03.977 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:03.977 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.998+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c0406b2c0 msgr2=0x7f9c0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.998+0000 7f9c2b200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c0406b2c0 0x7f9c0406d780 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c15ac20 tx=0x7f9c1c0094d0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.998+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 msgr2=0x7f9c2c158ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.998+0000 7f9c2b200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 0x7f9c2c158ff0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009fd0 tx=0x7f9c2400b8e0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 shutdown_connections 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c0406b2c0 0x7f9c0406d780 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c2c143c40 0x7f9c2c158ff0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c2c0cc500 0x7f9c2c158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 --2- 172.21.15.53:0/875833014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c2c06d160 0x7f9c2c13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:03.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 >> 172.21.15.53:0/875833014 conn(0x7f9c2c06a800 msgr2=0x7f9c2c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:03.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 shutdown_connections 2024-01-16T15:00:03.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:03.999+0000 7f9c2b200640 1 -- 172.21.15.53:0/875833014 wait complete. 2024-01-16T15:00:04.000 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-01-16T15:00:04.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:04.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.241+0000 7f3d79400640 1 Processor -- start 2024-01-16T15:00:04.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.241+0000 7f3d79400640 1 -- start start 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d79400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3d740d0100 0x7f3d740b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d79400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 0x7f3d740d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d79400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d740d0a10 con 0x7f3d74160920 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d79400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d740cf080 con 0x7f3d740d0100 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d79400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740c6be0 con 0x7f3d740d0f40 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d73e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3d74160920 0x7f3d740cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49686/0 (socket says 172.21.15.53:49686) 2024-01-16T15:00:04.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d73e00640 1 -- 172.21.15.53:0/4242539964 learned_addr learned my addr 172.21.15.53:0/4242539964 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d73400640 1 --2- 172.21.15.53:0/4242539964 >> v2:172.21.15.146:3300/0 conn(0x7f3d740d0100 0x7f3d740b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d78a00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 0x7f3d740d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094958426 0 0) 0x7f3d740d0a10 con 0x7f3d74160920 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d54003520 con 0x7f3d74160920 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.243+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552936893 0 0) 0x7f3d54003520 con 0x7f3d74160920 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 msgr2=0x7f3d740d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 0x7f3d740d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 >> v2:172.21.15.146:3300/0 conn(0x7f3d740d0100 msgr2=0x7f3d740b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 --2- 172.21.15.53:0/4242539964 >> v2:172.21.15.146:3300/0 conn(0x7f3d740d0100 0x7f3d740b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d740ce7e0 con 0x7f3d74160920 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3d64001670 con 0x7f3d74160920 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 >> v1:172.21.15.53:6789/0 conn(0x7f3d74160920 legacy=0x7f3d740cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 0x7f3d54003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d54003f40 0x7f3d540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d54004d00 0x7f3d54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740cf080 con 0x7f3d54003f40 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d740c6be0 con 0x7f3d54004d00 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5400b980 con 0x7f3d740d0f40 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73400640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d54003f40 0x7f3d540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73e00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 0x7f3d54003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d78a00640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d54004d00 0x7f3d54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73400640 1 -- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 msgr2=0x7f3d54003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73400640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d740d0f40 0x7f3d54003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73400640 1 -- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d54004d00 msgr2=0x7f3d54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73400640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d54004d00 0x7f3d54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.244+0000 7f3d73400640 1 -- 172.21.15.53:0/4242539964 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d54003520 con 0x7f3d54003f40 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d73400640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d54003f40 0x7f3d540047c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009af0 tx=0x7f3d540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a00e0ede75ae9789 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c02c710 con 0x7f3d54003f40 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c02c870 con 0x7f3d54003f40 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d72a00640 1 -- 172.21.15.53:0/4242539964 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c02cc00 con 0x7f3d54003f40 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d79400640 1 -- 172.21.15.53:0/4242539964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d54003f40 msgr2=0x7f3d540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d79400640 1 --2- 172.21.15.53:0/4242539964 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d54003f40 0x7f3d540047c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c009af0 tx=0x7f3d540057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d78a00640 1 -- 172.21.15.53:0/4242539964 reap_dead start 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d79400640 1 -- 172.21.15.53:0/4242539964 shutdown_connections 2024-01-16T15:00:04.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.246+0000 7f3d79400640 1 -- 172.21.15.53:0/4242539964 >> 172.21.15.53:0/4242539964 conn(0x7f3d740d3090 msgr2=0x7f3d740d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.247+0000 7f3d79400640 1 -- 172.21.15.53:0/4242539964 shutdown_connections 2024-01-16T15:00:04.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.247+0000 7f3d79400640 1 -- 172.21.15.53:0/4242539964 wait complete. 2024-01-16T15:00:04.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.248+0000 7f3d79400640 1 Processor -- start 2024-01-16T15:00:04.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.249+0000 7f3d79400640 1 -- start start 2024-01-16T15:00:04.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.249+0000 7f3d79400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d54003f40 0x7f3d740b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.249+0000 7f3d79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 0x7f3d740cb080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d79400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d74151860 0x7f3d740b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d79400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7415dc80 con 0x7f3d741513e0 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d79400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7415df20 con 0x7f3d74151860 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d79400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7415e200 con 0x7f3d54003f40 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 0x7f3d740cb080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 0x7f3d740cb080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34808/0 (socket says 172.21.15.53:34808) 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 -- 172.21.15.53:0/3946639028 learned_addr learned my addr 172.21.15.53:0/3946639028 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73e00640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d54003f40 0x7f3d740b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d78a00640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d74151860 0x7f3d740b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 -- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d54003f40 msgr2=0x7f3d740b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d54003f40 0x7f3d740b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 -- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d74151860 msgr2=0x7f3d740b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d74151860 0x7f3d740b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.250+0000 7f3d73400640 1 -- 172.21.15.53:0/3946639028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7415e4e0 con 0x7f3d741513e0 2024-01-16T15:00:04.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.251+0000 7f3d73400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 0x7f3d740cb080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c037040 tx=0x7f3d5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.251+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c003810 con 0x7f3d741513e0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.251+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c02de90 con 0x7f3d741513e0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.251+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5c003b70 con 0x7f3d741513e0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.251+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d740b6190 con 0x7f3d741513e0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.251+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7415d3b0 con 0x7f3d741513e0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.252+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7415e670 con 0x7f3d741513e0 2024-01-16T15:00:04.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.253+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3d5c003cd0 con 0x7f3d741513e0 2024-01-16T15:00:04.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.254+0000 7f3d6be00640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d4c06b2c0 0x7f3d4c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.254+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3d5c0abca0 con 0x7f3d741513e0 2024-01-16T15:00:04.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.255+0000 7f3d73e00640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d4c06b2c0 0x7f3d4c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.257+0000 7f3d73e00640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d4c06b2c0 0x7f3d4c06d780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d64001050 tx=0x7f3d64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.259+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d5c076b90 con 0x7f3d741513e0 2024-01-16T15:00:04.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.454+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d74067450 con 0x7f3d741513e0 2024-01-16T15:00:04.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.456+0000 7f3d6be00640 1 -- 172.21.15.53:0/3946639028 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f3d5c07a1b0 con 0x7f3d741513e0 2024-01-16T15:00:04.437 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:04.438 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d4c06b2c0 msgr2=0x7f3d4c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d4c06b2c0 0x7f3d4c06d780 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d64001050 tx=0x7f3d64009040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 msgr2=0x7f3d740cb080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 0x7f3d740cb080 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c037040 tx=0x7f3d5c004730 comp rx=0 tx=0).stop 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 shutdown_connections 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d4c06b2c0 0x7f3d4c06d780 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d74151860 0x7f3d740b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d741513e0 0x7f3d740cb080 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 --2- 172.21.15.53:0/3946639028 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d54003f40 0x7f3d740b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.458+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 >> 172.21.15.53:0/3946639028 conn(0x7f3d740d3090 msgr2=0x7f3d7406aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.459+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 shutdown_connections 2024-01-16T15:00:04.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.459+0000 7f3d79400640 1 -- 172.21.15.53:0/3946639028 wait complete. 2024-01-16T15:00:04.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-01-16T15:00:04.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-01-16T15:00:04.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-01-16T15:00:04.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-01-16T15:00:04.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-01-16T15:00:04.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-01-16T15:00:04.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-01-16T15:00:04.464 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-01-16T15:00:04.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.782+0000 7fb42ea00640 1 Processor -- start 2024-01-16T15:00:04.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.782+0000 7fb42ea00640 1 -- start start 2024-01-16T15:00:04.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.784+0000 7fb42ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb4280d0190 0x7fb4281683f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.784+0000 7fb42ea00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4280cf140 con 0x7fb4280cdf40 2024-01-16T15:00:04.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.784+0000 7fb42ea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4280d2430 con 0x7fb42815d760 2024-01-16T15:00:04.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.784+0000 7fb42ea00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4280ce530 con 0x7fb4280d0190 2024-01-16T15:00:04.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.784+0000 7fb427400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb4280cdf40 0x7fb4280cfa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49694/0 (socket says 172.21.15.53:49694) 2024-01-16T15:00:04.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.784+0000 7fb427400640 1 -- 172.21.15.53:0/2742582424 learned_addr learned my addr 172.21.15.53:0/2742582424 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:04.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.785+0000 7fb42ca00640 1 --2- 172.21.15.53:0/2742582424 >> v2:172.21.15.146:3300/0 conn(0x7fb4280d0190 0x7fb4281683f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.786+0000 7fb42ca00640 1 -- 172.21.15.53:0/2742582424 >> v1:172.21.15.146:6789/0 conn(0x7fb42815d760 legacy=0x7fb4280d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.786+0000 7fb42ca00640 1 -- 172.21.15.53:0/2742582424 >> v1:172.21.15.53:6789/0 conn(0x7fb4280cdf40 legacy=0x7fb4280cfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.786+0000 7fb42ca00640 1 -- 172.21.15.53:0/2742582424 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4280ce810 con 0x7fb4280d0190 2024-01-16T15:00:04.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.786+0000 7f58f6000640 1 Processor -- start 2024-01-16T15:00:04.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.787+0000 7f58f6000640 1 -- start start 2024-01-16T15:00:04.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.788+0000 7fb42ca00640 1 --2- 172.21.15.53:0/2742582424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4281683f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4180097a0 tx=0x7fb41802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84010cc731d9d978 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.788+0000 7fb426a00640 1 -- 172.21.15.53:0/2742582424 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418030610 con 0x7fb4280d0190 2024-01-16T15:00:04.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.789+0000 7fb426a00640 1 -- 172.21.15.53:0/2742582424 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb418030770 con 0x7fb4280d0190 2024-01-16T15:00:04.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.789+0000 7fb426a00640 1 -- 172.21.15.53:0/2742582424 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418030610 con 0x7fb4280d0190 2024-01-16T15:00:04.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7f58f6000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58f0160920 0x7f58f00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7f58f6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cfea0 0x7f58f0161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7f58f6000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58f00d0a10 con 0x7f58f00cf9f0 2024-01-16T15:00:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7f58f6000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58f00d0330 con 0x7f58f0160920 2024-01-16T15:00:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7f58f6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00d1f50 con 0x7f58f00cfea0 2024-01-16T15:00:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7fb42ea00640 1 -- 172.21.15.53:0/2742582424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 msgr2=0x7fb4281683f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.790+0000 7fb42ea00640 1 --2- 172.21.15.53:0/2742582424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4281683f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb4180097a0 tx=0x7fb41802b960 comp rx=0 tx=0).stop 2024-01-16T15:00:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7f58f4c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58f0160920 0x7f58f00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7f58efe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f58f00cf9f0 0x7f58f0161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38346/0 (socket says 172.21.15.53:38346) 2024-01-16T15:00:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7f58efe00640 1 -- 172.21.15.53:0/1174065892 learned_addr learned my addr 172.21.15.53:0/1174065892 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7fb42ea00640 1 -- 172.21.15.53:0/2742582424 shutdown_connections 2024-01-16T15:00:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7fb42ea00640 1 --2- 172.21.15.53:0/2742582424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4281683f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7fb42ea00640 1 -- 172.21.15.53:0/2742582424 >> 172.21.15.53:0/2742582424 conn(0x7fb42809c440 msgr2=0x7fb4280d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7f58f5600640 1 --2- 172.21.15.53:0/1174065892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cfea0 0x7f58f0161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7fb42ea00640 1 -- 172.21.15.53:0/2742582424 shutdown_connections 2024-01-16T15:00:04.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.791+0000 7fb42ea00640 1 -- 172.21.15.53:0/2742582424 wait complete. 2024-01-16T15:00:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.792+0000 7f58f5600640 1 -- 172.21.15.53:0/1174065892 >> v2:172.21.15.146:3300/0 conn(0x7f58f0160920 msgr2=0x7f58f00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.792+0000 7f58f5600640 1 --2- 172.21.15.53:0/1174065892 >> v2:172.21.15.146:3300/0 conn(0x7f58f0160920 0x7f58f00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.792+0000 7f58f5600640 1 -- 172.21.15.53:0/1174065892 >> v1:172.21.15.146:6789/0 conn(0x7f58f00cf9f0 legacy=0x7f58f0161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.792+0000 7f58f5600640 1 -- 172.21.15.53:0/1174065892 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58f00cf080 con 0x7f58f00cfea0 2024-01-16T15:00:04.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.793+0000 7f58f5600640 1 --2- 172.21.15.53:0/1174065892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cfea0 0x7f58f0161720 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009d90 tx=0x7f58e802bee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7953ade9e430c856 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.794+0000 7fb42ea00640 1 Processor -- start 2024-01-16T15:00:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.794+0000 7f58ef400640 1 -- 172.21.15.53:0/1174065892 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e802cc90 con 0x7f58f00cfea0 2024-01-16T15:00:04.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.794+0000 7fb42ea00640 1 -- start start 2024-01-16T15:00:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.795+0000 7f58ef400640 1 -- 172.21.15.53:0/1174065892 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e802cdf0 con 0x7f58f00cfea0 2024-01-16T15:00:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.795+0000 7f58ef400640 1 -- 172.21.15.53:0/1174065892 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8034640 con 0x7f58f00cfea0 2024-01-16T15:00:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.795+0000 7f888be00640 1 Processor -- start 2024-01-16T15:00:04.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.795+0000 7f888be00640 1 -- start start 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7f58f6000640 1 -- 172.21.15.53:0/1174065892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cfea0 msgr2=0x7f58f0161720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7f58f6000640 1 --2- 172.21.15.53:0/1174065892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cfea0 0x7f58f0161720 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58e8009d90 tx=0x7f58e802bee0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7fb42ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4280cdf40 0x7fb4280c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7fb42ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4280b4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7fb42ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4280c95f0 0x7fb4280b50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7fb42ea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280c6c60 con 0x7fb4280cdf40 2024-01-16T15:00:04.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7fb42ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280c6f40 con 0x7fb4280d0190 2024-01-16T15:00:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.796+0000 7fb42ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4280c7220 con 0x7fb4280c95f0 2024-01-16T15:00:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f58f6000640 1 -- 172.21.15.53:0/1174065892 shutdown_connections 2024-01-16T15:00:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f58f6000640 1 --2- 172.21.15.53:0/1174065892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cfea0 0x7f58f0161720 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f58f6000640 1 --2- 172.21.15.53:0/1174065892 >> v2:172.21.15.146:3300/0 conn(0x7f58f0160920 0x7f58f00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f58f6000640 1 -- 172.21.15.53:0/1174065892 >> 172.21.15.53:0/1174065892 conn(0x7f58f00d3090 msgr2=0x7f58f00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f58f6000640 1 -- 172.21.15.53:0/1174065892 shutdown_connections 2024-01-16T15:00:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7fb42ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4280c95f0 0x7fb4280b50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7fb42ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4280c95f0 0x7fb4280b50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49404/0 (socket says 172.21.15.53:49404) 2024-01-16T15:00:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7fb427e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4280cdf40 0x7fb4280c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f888be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7fb427400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4280b4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7fb427400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4280b4b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47576/0 (socket says 172.21.15.53:47576) 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7fb427400640 1 -- 172.21.15.53:0/3608373581 learned_addr learned my addr 172.21.15.53:0/3608373581 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7f888be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8884007b40 con 0x7f8884053c80 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7f888be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88840587e0 con 0x7f888403c070 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7f888be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8884058ac0 con 0x7f88840e3130 2024-01-16T15:00:04.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7f888aa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.797+0000 7f58f6000640 1 -- 172.21.15.53:0/1174065892 wait complete. 2024-01-16T15:00:04.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7f888aa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49410/0 (socket says 172.21.15.53:49410) 2024-01-16T15:00:04.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.798+0000 7f888aa00640 1 -- 172.21.15.53:0/3079821242 learned_addr learned my addr 172.21.15.53:0/3079821242 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.799+0000 7fb427400640 1 -- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4280c95f0 msgr2=0x7fb4280b50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.799+0000 7fb427400640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4280c95f0 0x7fb4280b50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.799+0000 7fb427400640 1 -- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4280cdf40 msgr2=0x7fb4280c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.799+0000 7f888aa00640 1 -- 172.21.15.53:0/3079821242 >> v1:172.21.15.146:6789/0 conn(0x7f888403c070 legacy=0x7f8884058020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.800+0000 7f888aa00640 1 -- 172.21.15.53:0/3079821242 >> v1:172.21.15.53:6789/0 conn(0x7f8884053c80 legacy=0x7f88840540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.800+0000 7f888aa00640 1 -- 172.21.15.53:0/3079821242 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884006330 con 0x7f88840e3130 2024-01-16T15:00:04.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.800+0000 7fb427400640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4280cdf40 0x7fb4280c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.800+0000 7fb427400640 1 -- 172.21.15.53:0/3608373581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb418009400 con 0x7fb4280d0190 2024-01-16T15:00:04.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.800+0000 7fb42ca00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4280c95f0 0x7fb4280b50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:04.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.800+0000 7fb427e00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4280cdf40 0x7fb4280c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:04.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.801+0000 7f58f6000640 1 Processor -- start 2024-01-16T15:00:04.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.801+0000 7fb427400640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4280b4b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c480 tx=0x7fb41000c940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.801+0000 7f58f6000640 1 -- start start 2024-01-16T15:00:04.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.802+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb410005ce0 con 0x7fb4280d0190 2024-01-16T15:00:04.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.802+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb410005e40 con 0x7fb4280d0190 2024-01-16T15:00:04.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.802+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4100072f0 con 0x7fb4280d0190 2024-01-16T15:00:04.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.802+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4280c7500 con 0x7fb4280d0190 2024-01-16T15:00:04.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.802+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4280c8e50 con 0x7fb4280d0190 2024-01-16T15:00:04.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f25ce800640 1 Processor -- start 2024-01-16T15:00:04.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f25ce800640 1 -- start start 2024-01-16T15:00:04.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb4280c70b0 con 0x7fb4280d0190 2024-01-16T15:00:04.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f888aa00640 1 --2- 172.21.15.53:0/3079821242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e7750 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f887c009820 tx=0x7f887c02b7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7a7b5708856614e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f58f6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cf9f0 0x7f58f00b4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f58f6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f00cfea0 0x7f58f01499e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f58f6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58f0160920 0x7f58f00cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f58f6000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f014a3c0 con 0x7f58f00cfea0 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f58f6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f00b4ed0 con 0x7f58f0160920 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.803+0000 7f58f6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58f015a6c0 con 0x7f58f00cf9f0 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.804+0000 7f58f5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58f0160920 0x7f58f00cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.804+0000 7f58f5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58f0160920 0x7f58f00cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47588/0 (socket says 172.21.15.53:47588) 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.804+0000 7f58f5600640 1 -- 172.21.15.53:0/1819463601 learned_addr learned my addr 172.21.15.53:0/1819463601 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.804+0000 7f58f4c00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cf9f0 0x7f58f00b4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.804+0000 7f58efe00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f00cfea0 0x7f58f01499e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f6c49000640 1 Processor -- start 2024-01-16T15:00:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f58f5600640 1 -- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cf9f0 msgr2=0x7f58f00b4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f58f5600640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58f00cf9f0 0x7f58f00b4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f58f5600640 1 -- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f00cfea0 msgr2=0x7f58f01499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f58f5600640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f00cfea0 0x7f58f01499e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f58f5600640 1 -- 172.21.15.53:0/1819463601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e8009a70 con 0x7f58f0160920 2024-01-16T15:00:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f6c49000640 1 -- start start 2024-01-16T15:00:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7fa6baa00640 1 Processor -- start 2024-01-16T15:00:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f58efe00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58f00cfea0 0x7f58f01499e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f8889600640 1 -- 172.21.15.53:0/3079821242 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c031520 con 0x7f88840e3130 2024-01-16T15:00:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f8889600640 1 -- 172.21.15.53:0/3079821242 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c031680 con 0x7f88840e3130 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.805+0000 7f8889600640 1 -- 172.21.15.53:0/3079821242 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c031a10 con 0x7f88840e3130 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7f25ce800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25c815d460 0x7f25c80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7f25ce800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25c80b30b0 0x7f25c806d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7f25ce800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25c815d840 con 0x7f25c806e380 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7f25ce800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c8071490 con 0x7f25c815d460 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7f25ce800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25c80b3790 con 0x7f25c80b30b0 2024-01-16T15:00:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f58f5600640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58f0160920 0x7f58f00cdbf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58e802bf20 tx=0x7f58e8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f888be00640 1 -- 172.21.15.53:0/3079821242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 msgr2=0x7f88840e7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f888be00640 1 --2- 172.21.15.53:0/3079821242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e7750 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f887c009820 tx=0x7f887c02b7c0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7fa6baa00640 1 -- start start 2024-01-16T15:00:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.806+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb410007510 con 0x7fb4280d0190 2024-01-16T15:00:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25c815d460 0x7f25c80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25c815d460 0x7f25c80ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34832/0 (socket says 172.21.15.53:34832) 2024-01-16T15:00:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7400640 1 -- 172.21.15.53:0/2900857258 learned_addr learned my addr 172.21.15.53:0/2900857258 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7e00640 1 --2- 172.21.15.53:0/2900857258 >> v2:172.21.15.146:3300/0 conn(0x7f25c80b30b0 0x7f25c806d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8004780 con 0x7f58f0160920 2024-01-16T15:00:04.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f014c8a0 con 0x7f58f0160920 2024-01-16T15:00:04.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7400640 1 -- 172.21.15.53:0/2900857258 >> v2:172.21.15.146:3300/0 conn(0x7f25c80b30b0 msgr2=0x7f25c806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7400640 1 --2- 172.21.15.53:0/2900857258 >> v2:172.21.15.146:3300/0 conn(0x7f25c80b30b0 0x7f25c806d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.807+0000 7f25c7400640 1 -- 172.21.15.53:0/2900857258 >> v1:172.21.15.146:6789/0 conn(0x7f25c806e380 legacy=0x7f25c80b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7fde12a00640 1 Processor -- start 2024-01-16T15:00:04.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f014ce60 con 0x7f58f0160920 2024-01-16T15:00:04.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c49000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c4413df80 0x7f6c4415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e8034440 con 0x7f58f0160920 2024-01-16T15:00:04.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f888be00640 1 -- 172.21.15.53:0/3079821242 shutdown_connections 2024-01-16T15:00:04.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f888be00640 1 --2- 172.21.15.53:0/3079821242 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e7750 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f888be00640 1 -- 172.21.15.53:0/3079821242 >> 172.21.15.53:0/3079821242 conn(0x7f8884058eb0 msgr2=0x7f8884059290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f25c7400640 1 -- 172.21.15.53:0/2900857258 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c8072190 con 0x7f25c815d460 2024-01-16T15:00:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7fde12a00640 1 -- start start 2024-01-16T15:00:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c49000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6c440cf9a0 0x7f6c440cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c49000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c4406c470 con 0x7f6c4413e360 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c49000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4406c750 con 0x7f6c4413df80 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c49000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c4406ca30 con 0x7f6c440cf9a0 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c41e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c4413df80 0x7f6c4415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c41e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c4413df80 0x7f6c4415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34848/0 (socket says 172.21.15.53:34848) 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f6c41e00640 1 -- 172.21.15.53:0/2395563448 learned_addr learned my addr 172.21.15.53:0/2395563448 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.808+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e8004780 con 0x7f58f0160920 2024-01-16T15:00:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6baa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b40ced30 0x7fa6b40b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6baa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40cf1c0 con 0x7fa6b40ce8f0 2024-01-16T15:00:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6baa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b40d0de0 con 0x7fa6b4160f80 2024-01-16T15:00:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6baa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b40d10c0 con 0x7fa6b40ced30 2024-01-16T15:00:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7f6c42800640 1 --2- 172.21.15.53:0/2395563448 >> v2:172.21.15.146:3300/0 conn(0x7f6c440cf9a0 0x7f6c440cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7f888be00640 1 -- 172.21.15.53:0/3079821242 shutdown_connections 2024-01-16T15:00:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6b3e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa6b4160f80 0x7fa6b4161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38378/0 (socket says 172.21.15.53:38378) 2024-01-16T15:00:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6b3e00640 1 -- 172.21.15.53:0/3021473544 learned_addr learned my addr 172.21.15.53:0/3021473544 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fa6b8a00640 1 --2- 172.21.15.53:0/3021473544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b40ced30 0x7fa6b40b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fb41fe00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb40006b2c0 0x7fb40006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb410089060 con 0x7fb4280d0190 2024-01-16T15:00:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fb41fe00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7fb400073460 0x7fb400075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 --> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb400075f30 con 0x7fb400073460 2024-01-16T15:00:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb41008b720 con 0x7fb4280d0190 2024-01-16T15:00:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fa6b8a00640 1 -- 172.21.15.53:0/3021473544 >> v1:172.21.15.146:6789/0 conn(0x7fa6b4160f80 legacy=0x7fa6b4161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fde12a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 0x7fde0c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fa6b8a00640 1 -- 172.21.15.53:0/3021473544 >> v1:172.21.15.53:6789/0 conn(0x7fa6b40ce8f0 legacy=0x7fa6b40b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fb427e00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb40006b2c0 0x7fb40006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fb42ca00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7fb400073460 0x7fb400075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fde12a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c06cea0 con 0x7fde0c06c6b0 2024-01-16T15:00:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fde12a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c06cbc0 con 0x7fde0c0cd240 2024-01-16T15:00:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fde12a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c06f350 con 0x7fde0c09a170 2024-01-16T15:00:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58c0000fc0 con 0x7f58f0160920 2024-01-16T15:00:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.810+0000 7fa6b8a00640 1 -- 172.21.15.53:0/3021473544 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b40d13a0 con 0x7fa6b40ced30 2024-01-16T15:00:04.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7f6c42800640 1 -- 172.21.15.53:0/2395563448 >> v2:172.21.15.53:3300/0 conn(0x7f6c4413df80 msgr2=0x7f6c4415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7f6c42800640 1 --2- 172.21.15.53:0/2395563448 >> v2:172.21.15.53:3300/0 conn(0x7f6c4413df80 0x7f6c4415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7f6c42800640 1 -- 172.21.15.53:0/2395563448 >> v1:172.21.15.146:6789/0 conn(0x7f6c4413e360 legacy=0x7f6c440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7fb427e00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb40006b2c0 0x7fb40006d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4280c86b0 tx=0x7fb420009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7f6c42800640 1 -- 172.21.15.53:0/2395563448 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4406cd10 con 0x7f6c440cf9a0 2024-01-16T15:00:04.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7fde0b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 0x7fde0c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7fde0b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 0x7fde0c0b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49442/0 (socket says 172.21.15.53:49442) 2024-01-16T15:00:04.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.811+0000 7fde0b400640 1 -- 172.21.15.53:0/2467955471 learned_addr learned my addr 172.21.15.53:0/2467955471 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.812+0000 7fb42ca00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7fb400073460 0x7fb400075880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.812+0000 7f25c7400640 1 --2- 172.21.15.53:0/2900857258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c815d460 0x7f25c80ac8d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25c80aeb00 tx=0x7f25b802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc383703ac8ff71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.809+0000 7f888be00640 1 -- 172.21.15.53:0/3079821242 wait complete. 2024-01-16T15:00:04.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.812+0000 7f888be00640 1 Processor -- start 2024-01-16T15:00:04.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.812+0000 7f888be00640 1 -- start start 2024-01-16T15:00:04.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7fde0aa00640 1 -- 172.21.15.53:0/2467955471 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99375798 0 0) 0x7fde0c06cbc0 con 0x7fde0c0cd240 2024-01-16T15:00:04.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7fde0b400640 1 -- 172.21.15.53:0/2467955471 >> v1:172.21.15.146:6789/0 conn(0x7fde0c0cd240 legacy=0x7fde0c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7fde0b400640 1 -- 172.21.15.53:0/2467955471 >> v1:172.21.15.53:6789/0 conn(0x7fde0c06c6b0 legacy=0x7fde0c0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7fde0b400640 1 -- 172.21.15.53:0/2467955471 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c06f630 con 0x7fde0c09a170 2024-01-16T15:00:04.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7f25c6a00640 1 -- 172.21.15.53:0/2900857258 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8030860 con 0x7f25c815d460 2024-01-16T15:00:04.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7fa6b8a00640 1 --2- 172.21.15.53:0/3021473544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b40ced30 0x7fa6b40b3110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009dc0 tx=0x7fa6ac02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f34aa566063acd2d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7f6c42800640 1 --2- 172.21.15.53:0/2395563448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c440cf9a0 0x7f6c440cd840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c440ccb10 tx=0x7f6c3c02b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca6998d18bbcc12c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.813+0000 7fa6b2a00640 1 -- 172.21.15.53:0/3021473544 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac02cc50 con 0x7fa6b40ced30 2024-01-16T15:00:04.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7fa6b2a00640 1 -- 172.21.15.53:0/3021473544 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac02cdb0 con 0x7fa6b40ced30 2024-01-16T15:00:04.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7fa6b2a00640 1 -- 172.21.15.53:0/3021473544 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ac034660 con 0x7fa6b40ced30 2024-01-16T15:00:04.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f25c6a00640 1 -- 172.21.15.53:0/2900857258 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b80309c0 con 0x7f25c815d460 2024-01-16T15:00:04.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f25c6a00640 1 -- 172.21.15.53:0/2900857258 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b8030d50 con 0x7f25c815d460 2024-01-16T15:00:04.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== osd.1 v2:172.21.15.53:6816/3560880694 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb400075f30 con 0x7fb400073460 2024-01-16T15:00:04.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f6c41400640 1 -- 172.21.15.53:0/2395563448 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c030710 con 0x7f6c440cf9a0 2024-01-16T15:00:04.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f6c41400640 1 -- 172.21.15.53:0/2395563448 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3c02e050 con 0x7f6c440cf9a0 2024-01-16T15:00:04.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f6c41400640 1 -- 172.21.15.53:0/2395563448 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c030a90 con 0x7f6c440cf9a0 2024-01-16T15:00:04.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f888be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888403c070 0x7f8884188d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f888be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8884053c80 0x7f888404e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f888be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f25ce800640 1 -- 172.21.15.53:0/2900857258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c815d460 msgr2=0x7f25c80ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f25ce800640 1 --2- 172.21.15.53:0/2900857258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c815d460 0x7f25c80ac8d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f25c80aeb00 tx=0x7f25b802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7fde0b400640 1 --2- 172.21.15.53:0/2467955471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 0x7fde0c0b3820 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fddf40098a0 tx=0x7fddf402ba00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53caaa956de12c43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f6c49000640 1 -- 172.21.15.53:0/2395563448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c440cf9a0 msgr2=0x7f6c440cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f6c49000640 1 --2- 172.21.15.53:0/2395563448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c440cf9a0 0x7f6c440cd840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f6c440ccb10 tx=0x7f6c3c02b710 comp rx=0 tx=0).stop 2024-01-16T15:00:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 -- 172.21.15.53:0/3021473544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b40ced30 msgr2=0x7fa6b40b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 --2- 172.21.15.53:0/3021473544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b40ced30 0x7fa6b40b3110 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa6ac009dc0 tx=0x7fa6ac02bea0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.814+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f58e80345b0 con 0x7f58f0160920 2024-01-16T15:00:04.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f6c49000640 1 -- 172.21.15.53:0/2395563448 shutdown_connections 2024-01-16T15:00:04.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f6c49000640 1 --2- 172.21.15.53:0/2395563448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c440cf9a0 0x7f6c440cd840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f6c49000640 1 --2- 172.21.15.53:0/2395563448 >> v2:172.21.15.53:3300/0 conn(0x7f6c4413df80 0x7f6c4415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f6c49000640 1 -- 172.21.15.53:0/2395563448 >> 172.21.15.53:0/2395563448 conn(0x7f6c4406a800 msgr2=0x7f6c4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 -- 172.21.15.53:0/3021473544 shutdown_connections 2024-01-16T15:00:04.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 --2- 172.21.15.53:0/3021473544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b40ced30 0x7fa6b40b3110 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 -- 172.21.15.53:0/3021473544 >> 172.21.15.53:0/3021473544 conn(0x7fa6b409c550 msgr2=0x7fa6b40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f888a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8884053c80 0x7f888404e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f25ce800640 1 -- 172.21.15.53:0/2900857258 shutdown_connections 2024-01-16T15:00:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fde0aa00640 1 -- 172.21.15.53:0/2467955471 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf402f820 con 0x7fde0c09a170 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fde0aa00640 1 -- 172.21.15.53:0/2467955471 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf402f980 con 0x7fde0c09a170 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fde0aa00640 1 -- 172.21.15.53:0/2467955471 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf402fd10 con 0x7fde0c09a170 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f888a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8884053c80 0x7f888404e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34854/0 (socket says 172.21.15.53:34854) 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f888aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888403c070 0x7f8884188d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f25ce800640 1 --2- 172.21.15.53:0/2900857258 >> v2:172.21.15.146:3300/0 conn(0x7f25c80b30b0 0x7f25c806d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f25ce800640 1 --2- 172.21.15.53:0/2900857258 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c815d460 0x7f25c80ac8d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7f25ce800640 1 -- 172.21.15.53:0/2900857258 >> 172.21.15.53:0/2900857258 conn(0x7f25c80b0be0 msgr2=0x7f25c80b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f888aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888403c070 0x7f8884188d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47604/0 (socket says 172.21.15.53:47604) 2024-01-16T15:00:04.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f25ce800640 1 -- 172.21.15.53:0/2900857258 shutdown_connections 2024-01-16T15:00:04.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888404c300 con 0x7f8884053c80 2024-01-16T15:00:04.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f888a000640 1 -- 172.21.15.53:0/1235404705 learned_addr learned my addr 172.21.15.53:0/1235404705 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88840dac90 con 0x7f888403c070 2024-01-16T15:00:04.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888404d520 con 0x7f88840e3130 2024-01-16T15:00:04.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f6c49000640 1 -- 172.21.15.53:0/2395563448 shutdown_connections 2024-01-16T15:00:04.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f6c49000640 1 -- 172.21.15.53:0/2395563448 wait complete. 2024-01-16T15:00:04.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f25ce800640 1 -- 172.21.15.53:0/2900857258 wait complete. 2024-01-16T15:00:04.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f888b400640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7fde12a00640 1 -- 172.21.15.53:0/2467955471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 msgr2=0x7fde0c0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7fde12a00640 1 --2- 172.21.15.53:0/2467955471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 0x7fde0c0b3820 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fddf40098a0 tx=0x7fddf402ba00 comp rx=0 tx=0).stop 2024-01-16T15:00:04.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7f58d4c00640 1 Processor -- start 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.816+0000 7fde12a00640 1 -- 172.21.15.53:0/2467955471 shutdown_connections 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7fde12a00640 1 --2- 172.21.15.53:0/2467955471 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c09a170 0x7fde0c0b3820 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7fde12a00640 1 -- 172.21.15.53:0/2467955471 >> 172.21.15.53:0/2467955471 conn(0x7fde0c0b2d00 msgr2=0x7fde0c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f58d4c00640 1 -- start start 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f888b400640 1 -- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888403c070 msgr2=0x7f8884188d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f888b400640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888403c070 0x7f8884188d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f888b400640 1 -- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8884053c80 msgr2=0x7f888404e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f888b400640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8884053c80 0x7f888404e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f888b400640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0094d0 con 0x7f88840e3130 2024-01-16T15:00:04.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 -- 172.21.15.53:0/3021473544 shutdown_connections 2024-01-16T15:00:04.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f888b400640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e4af0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f887400c3f0 tx=0x7f887400c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f58ecc00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58c806b380 0x7f58c806d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58e80ac8a0 con 0x7f58f0160920 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.817+0000 7f58ecc00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f58c8073520 0x7f58c8075940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 --> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58c8075ff0 con 0x7f58c8073520 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58e8004780 con 0x7f58f0160920 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7fde12a00640 1 -- 172.21.15.53:0/2467955471 shutdown_connections 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.815+0000 7fa6baa00640 1 -- 172.21.15.53:0/3021473544 wait complete. 2024-01-16T15:00:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88740052d0 con 0x7f88840e3130 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874005450 con 0x7f88840e3130 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874003830 con 0x7f88840e3130 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f6c49000640 1 Processor -- start 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7fde12a00640 1 -- 172.21.15.53:0/2467955471 wait complete. 2024-01-16T15:00:04.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f6c49000640 1 -- start start 2024-01-16T15:00:04.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7fa6baa00640 1 Processor -- start 2024-01-16T15:00:04.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58f4c00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58c806b380 0x7f58c806d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888404d800 con 0x7f88840e3130 2024-01-16T15:00:04.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58d4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c8057d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58d4c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58c80542d0 0x7f58c80068a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58d4c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c80582a0 con 0x7f58c803e810 2024-01-16T15:00:04.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58d4c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58c8006170 con 0x7f58c80542d0 2024-01-16T15:00:04.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7f58d4c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c8058540 con 0x7f58c80e3090 2024-01-16T15:00:04.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f58efe00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f58c8073520 0x7f58c8075940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888404ddc0 con 0x7f88840e3130 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.819+0000 7fa6baa00640 1 -- start start 2024-01-16T15:00:04.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f58cf400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c8057d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f6c49000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c440cf9a0 0x7f6c4413ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f6c49000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4413df80 0x7f6c44150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f6c49000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4413e360 0x7f6c44152b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f6c49000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44151130 con 0x7f6c4413e360 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f6c49000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44150b70 con 0x7f6c4413df80 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f6c49000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44150e50 con 0x7f6c440cf9a0 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f58cfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58c80542d0 0x7f58c80068a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f58cfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58c80542d0 0x7f58c80068a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47612/0 (socket says 172.21.15.53:47612) 2024-01-16T15:00:04.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7f58cfe00640 1 -- 172.21.15.53:0/976314236 learned_addr learned my addr 172.21.15.53:0/976314236 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.820+0000 7fde12a00640 1 Processor -- start 2024-01-16T15:00:04.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f25ce800640 1 Processor -- start 2024-01-16T15:00:04.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7fde12a00640 1 -- start start 2024-01-16T15:00:04.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f6c42800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4413e360 0x7f6c44152b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f25ce800640 1 -- start start 2024-01-16T15:00:04.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f6c42800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4413e360 0x7f6c44152b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34870/0 (socket says 172.21.15.53:34870) 2024-01-16T15:00:04.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f6c42800640 1 -- 172.21.15.53:0/4053326520 learned_addr learned my addr 172.21.15.53:0/4053326520 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f6c39200640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4413df80 0x7f6c44150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f6c41e00640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c440cf9a0 0x7f6c4413ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f58ce000640 1 -- 172.21.15.53:0/976314236 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1257624779 0 0) 0x7f58c80582a0 con 0x7f58c803e810 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f58ce000640 1 -- 172.21.15.53:0/976314236 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b0003520 con 0x7f58c803e810 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f58cfe00640 1 -- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 msgr2=0x7f58c8057d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f58cfe00640 1 --2- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c8057d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f58cfe00640 1 -- 172.21.15.53:0/976314236 >> v1:172.21.15.146:6789/0 conn(0x7f58c803e810 legacy=0x7f58c8053bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f58cfe00640 1 -- 172.21.15.53:0/976314236 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c8058820 con 0x7f58c80542d0 2024-01-16T15:00:04.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.821+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8884005a60 con 0x7f88840e3130 2024-01-16T15:00:04.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fa6baa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b40ce8f0 0x7fa6b40cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fa6baa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b40ced30 0x7fa6b4158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fa6baa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b4160f80 0x7fa6b415f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fa6baa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41f89a0 con 0x7fa6b40ced30 2024-01-16T15:00:04.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fa6baa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b40cbde0 con 0x7fa6b40ce8f0 2024-01-16T15:00:04.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fa6baa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b41f7e20 con 0x7fa6b4160f80 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c42800640 1 -- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c440cf9a0 msgr2=0x7f6c4413ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c42800640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c440cf9a0 0x7f6c4413ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c42800640 1 -- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4413df80 msgr2=0x7f6c44150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c42800640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4413df80 0x7f6c44150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c42800640 1 -- 172.21.15.53:0/4053326520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c3c009400 con 0x7f6c4413e360 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f58f4c00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58c806b380 0x7f58c806d840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58d8006fd0 tx=0x7f58d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c39200640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c4413df80 0x7f6c44150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f6c41e00640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c440cf9a0 0x7f6c4413ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7f58efe00640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f58c8073520 0x7f58c8075940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fde12a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c06c6b0 0x7fde0c0cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fde12a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c09a170 0x7fde0c0c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.822+0000 7fde12a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cd240 0x7fde0c0b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f58cfe00640 1 --2- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c80068a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58d00695b0 tx=0x7f58d00a5f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=21664a38ff9058e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f6c42800640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4413e360 0x7f6c44152b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c02b750 tx=0x7f6c3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b40ced30 0x7fa6b4158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b40ced30 0x7fa6b4158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34872/0 (socket says 172.21.15.53:34872) 2024-01-16T15:00:04.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 -- 172.21.15.53:0/920522417 learned_addr learned my addr 172.21.15.53:0/920522417 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde0b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c09a170 0x7fde0c0c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde0b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c09a170 0x7fde0c0c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34886/0 (socket says 172.21.15.53:34886) 2024-01-16T15:00:04.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde0b400640 1 -- 172.21.15.53:0/1060578075 learned_addr learned my addr 172.21.15.53:0/1060578075 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c03b810 con 0x7f6c4413e360 2024-01-16T15:00:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde10a00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cd240 0x7fde0c0b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f58ce000640 1 -- 172.21.15.53:0/976314236 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00a6e00 con 0x7f58c80542d0 2024-01-16T15:00:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f58ce000640 1 -- 172.21.15.53:0/976314236 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d00ae440 con 0x7f58c80542d0 2024-01-16T15:00:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f58ce000640 1 -- 172.21.15.53:0/976314236 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d00a6e00 con 0x7f58c80542d0 2024-01-16T15:00:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde0be00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c06c6b0 0x7fde0c0cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3e00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b40ce8f0 0x7fa6b40cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 -- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b4160f80 msgr2=0x7fa6b415f8b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-16T15:00:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25ce800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c806e380 0x7f25c8159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0c5d70 con 0x7fde0c09a170 2024-01-16T15:00:04.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1f69c0 con 0x7fde0c0cd240 2024-01-16T15:00:04.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c3c03b990 con 0x7f6c4413e360 2024-01-16T15:00:04.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3c03bd40 con 0x7f6c4413e360 2024-01-16T15:00:04.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c1f6ca0 con 0x7fde0c06c6b0 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25ce800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c80b30b0 0x7f25c80cb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25ce800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c8144250 0x7f25c80cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25ce800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c80ccc40 con 0x7f25c80b30b0 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25ce800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25c8144d30 con 0x7f25c806e380 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25ce800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25c815b180 con 0x7f25c8144250 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== osd.0 v2:172.21.15.53:6800/2075393413 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f58c8075ff0 con 0x7f58c8073520 2024-01-16T15:00:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25bfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c80b30b0 0x7f25c80cb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25bfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c80b30b0 0x7f25c80cb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34902/0 (socket says 172.21.15.53:34902) 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f25bfe00640 1 -- 172.21.15.53:0/3353682231 learned_addr learned my addr 172.21.15.53:0/3353682231 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8874003990 con 0x7f88840e3130 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fde0b400640 1 -- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c06c6b0 msgr2=0x7fde0c0cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fde0b400640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c06c6b0 0x7fde0c0cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fde0b400640 1 -- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cd240 msgr2=0x7fde0c0b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fde0b400640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cd240 0x7fde0c0b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fde0b400640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40094d0 con 0x7fde0c09a170 2024-01-16T15:00:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6b4160f80 0x7fa6b415f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 -- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b40ce8f0 msgr2=0x7fa6b40cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6b40ce8f0 0x7fa6b40cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.824+0000 7fa6b3400640 1 -- 172.21.15.53:0/920522417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ac009a70 con 0x7fa6b40ced30 2024-01-16T15:00:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fa6b3400640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b40ced30 0x7fa6b4158790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400c450 tx=0x7fa6a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25c7e00640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c8144250 0x7f25c80cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25c7400640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c806e380 0x7f25c8159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25bfe00640 1 -- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c8144250 msgr2=0x7f25c80cd060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25bfe00640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25c8144250 0x7f25c80cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25bfe00640 1 -- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c806e380 msgr2=0x7f25c8159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25bfe00640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25c806e380 0x7f25c8159910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f25bfe00640 1 -- 172.21.15.53:0/3353682231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b80094d0 con 0x7f25c80b30b0 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fde10a00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cd240 0x7fde0c0b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:04.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4414b1a0 con 0x7f6c4413e360 2024-01-16T15:00:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4413d900 con 0x7f6c4413e360 2024-01-16T15:00:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.825+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a40052d0 con 0x7fa6b40ced30 2024-01-16T15:00:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f58d4c00640 1 -- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 msgr2=0x7f58c80068a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f58d4c00640 1 --2- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c80068a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f58d00695b0 tx=0x7f58d00a5f90 comp rx=0 tx=0).stop 2024-01-16T15:00:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f8882a00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406b310 0x7f886406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88740890a0 con 0x7f88840e3130 2024-01-16T15:00:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f25bfe00640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c80b30b0 0x7f25c80cb8c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c400 tx=0x7f25b000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b41f8100 con 0x7fa6b40ced30 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b41f86c0 con 0x7fa6b40ced30 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a4005430 con 0x7fa6b40ced30 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4012870 con 0x7fa6b40ced30 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f8882a00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f8864073530 0x7f8864075950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8864076000 con 0x7f8864073530 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8874089540 con 0x7f88840e3130 2024-01-16T15:00:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.827+0000 7fde0b400640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c09a170 0x7fde0c0c7480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fddf40098a0 tx=0x7fddf40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c4415f9c0 con 0x7f6c4413e360 2024-01-16T15:00:04.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f888aa00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406b310 0x7f886406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f888a000640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f8864073530 0x7f8864075950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf4046070 con 0x7fde0c09a170 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f58d4c00640 1 -- 172.21.15.53:0/976314236 shutdown_connections 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f58d4c00640 1 --2- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c80068a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f58d4c00640 1 --2- 172.21.15.53:0/976314236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c8057d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f58d4c00640 1 -- 172.21.15.53:0/976314236 >> 172.21.15.53:0/976314236 conn(0x7f58c80559b0 msgr2=0x7f58c8055dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b00052d0 con 0x7f25c80b30b0 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0005430 con 0x7f25c80b30b0 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0003830 con 0x7f25c80b30b0 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25c80cbfc0 con 0x7f25c80b30b0 2024-01-16T15:00:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25c80cc580 con 0x7f25c80b30b0 2024-01-16T15:00:04.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf4037bd0 con 0x7fde0c09a170 2024-01-16T15:00:04.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf403b040 con 0x7fde0c09a170 2024-01-16T15:00:04.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c1f6f80 con 0x7fde0c09a170 2024-01-16T15:00:04.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c0cc0b0 con 0x7fde0c09a170 2024-01-16T15:00:04.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f58d4c00640 1 -- 172.21.15.53:0/976314236 shutdown_connections 2024-01-16T15:00:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f58d4c00640 1 -- 172.21.15.53:0/976314236 wait complete. 2024-01-16T15:00:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.828+0000 7f888a000640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f8864073530 0x7f8864075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.830+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2598000fc0 con 0x7f25c80b30b0 2024-01-16T15:00:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.830+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa684000fc0 con 0x7fa6b40ced30 2024-01-16T15:00:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.830+0000 7f888aa00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406b310 0x7f886406d7d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f887c035000 tx=0x7f887c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.830+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== osd.2 v2:172.21.15.53:6824/16573798 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f8864076000 con 0x7f8864073530 2024-01-16T15:00:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.830+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6c3c039040 con 0x7f6c4413e360 2024-01-16T15:00:04.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.829+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fde0c0cd620 con 0x7fde0c09a170 2024-01-16T15:00:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.831+0000 7f58d4c00640 1 Processor -- start 2024-01-16T15:00:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.831+0000 7f6c3aa00640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6c1c06b340 0x7f6c1c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.831+0000 7f58d4c00640 1 -- start start 2024-01-16T15:00:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.831+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6c3c0ab7e0 con 0x7f6c4413e360 2024-01-16T15:00:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.832+0000 7f6c3aa00640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7f6c1c073560 0x7f6c1c075980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.832+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 --> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6c1c076030 con 0x7f6c1c073560 2024-01-16T15:00:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.832+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c3c0abc80 con 0x7f6c4413e360 2024-01-16T15:00:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.832+0000 7f6c41e00640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6c1c06b340 0x7f6c1c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.832+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa6a40129d0 con 0x7fa6b40ced30 2024-01-16T15:00:04.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.833+0000 7f6c39200640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7f6c1c073560 0x7f6c1c075980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.833+0000 7f58d4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58c803e810 0x7f58c80cb7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.833+0000 7f58d4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c804ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.833+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fddf402d040 con 0x7fde0c09a170 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58d4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c817ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58d4c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c80504b0 con 0x7f58c803e810 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58cf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58c803e810 0x7f58c80cb7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58cf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58c803e810 0x7f58c80cb7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34912/0 (socket says 172.21.15.53:34912) 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58cf400640 1 -- 172.21.15.53:0/2792425572 learned_addr learned my addr 172.21.15.53:0/2792425572 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58c80dec10 con 0x7f58c80542d0 2024-01-16T15:00:04.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58c817cfb0 con 0x7f58c80e3090 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f6c41e00640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6c1c06b340 0x7f6c1c06d800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009ac0 tx=0x7f6c34008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f6c39200640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7f6c1c073560 0x7f6c1c075980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7fa6abe00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa68c06b2c0 0x7fa68c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6a4089580 con 0x7fa6b40ced30 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7fa6abe00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fa68c073460 0x7fa68c075880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 --> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa68c075f30 con 0x7fa68c073460 2024-01-16T15:00:04.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a40011e0 con 0x7fa6b40ced30 2024-01-16T15:00:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58cea00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c804ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.834+0000 7f58cfe00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c817ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7fa6b3e00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa68c06b2c0 0x7fa68c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cf400640 1 -- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 msgr2=0x7f58c817ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cf400640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58c80e3090 0x7f58c817ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cf400640 1 -- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 msgr2=0x7f58c804ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cf400640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c804ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cf400640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d004ee20 con 0x7f58c803e810 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cea00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58c80542d0 0x7f58c804ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f58cf400640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58c803e810 0x7f58c80cb7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58b800e9e0 tx=0x7f58b800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.833+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25b0003990 con 0x7f25c80b30b0 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f25bf400640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a006b390 0x7f25a006d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25b00898d0 con 0x7f25c80b30b0 2024-01-16T15:00:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f25bf400640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7f25a00735b0 0x7f25a00759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 --> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25a0076080 con 0x7f25a00735b0 2024-01-16T15:00:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.835+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25b0089c80 con 0x7f25c80b30b0 2024-01-16T15:00:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b8009670 con 0x7f58c803e810 2024-01-16T15:00:04.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f25c7e00640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7f25a00735b0 0x7f25a00759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58b800c9d0 con 0x7f58c803e810 2024-01-16T15:00:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7fde03e00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406b390 0x7fdde406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58b800cd80 con 0x7f58c803e810 2024-01-16T15:00:04.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c8052790 con 0x7f58c803e810 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f25c7400640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a006b390 0x7f25a006d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fddf40acdc0 con 0x7fde0c09a170 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7fde03e00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7fdde40735b0 0x7fdde40759d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fdde4076080 con 0x7fdde40735b0 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddf40ad170 con 0x7fde0c09a170 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7fde0be00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406b390 0x7fdde406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7fde10a00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7fdde40735b0 0x7fdde40759d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7fa6b8a00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fa68c073460 0x7fa68c075880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7fa6b3e00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa68c06b2c0 0x7fa68c06d780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa69c00b830 tx=0x7fa69c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.836+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c8052cf0 con 0x7f58c803e810 2024-01-16T15:00:04.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.837+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58c803bf90 con 0x7f58c803e810 2024-01-16T15:00:04.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.838+0000 7f25c7400640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a006b390 0x7f25a006d850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f25b8036000 tx=0x7f25b803b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.838+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== osd.5 v2:172.21.15.146:6824/2159577511 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f6c1c076030 con 0x7f6c1c073560 2024-01-16T15:00:04.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.838+0000 7fa6b8a00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fa68c073460 0x7fa68c075880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.838+0000 7f25c7e00640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7f25a00735b0 0x7f25a00759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.840+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== osd.3 v2:172.21.15.53:6808/2110883972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f25a0076080 con 0x7f25a00735b0 2024-01-16T15:00:04.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.840+0000 7fde10a00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7fdde40735b0 0x7fdde40759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.840+0000 7fde0be00640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406b390 0x7fdde406d850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddfc000f60 tx=0x7fddfc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.841+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== osd.6 v2:172.21.15.146:6800/2492658325 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fa68c075f30 con 0x7fa68c073460 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.839+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f58b8013470 con 0x7f58c803e810 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.841+0000 7f58c7400640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58a806b310 0x7f58a806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.841+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58b8089d40 con 0x7f58c803e810 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.841+0000 7f58c7400640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7f58a80734b0 0x7f58a80758d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.841+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58a8075f80 con 0x7f58a80734b0 2024-01-16T15:00:04.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.841+0000 7f58cea00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58a806b310 0x7f58a806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.842+0000 7f58cfe00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7f58a80734b0 0x7f58a80758d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:04.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.842+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== osd.4 v2:172.21.15.146:6808/981755469 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fdde4076080 con 0x7fdde40735b0 2024-01-16T15:00:04.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.843+0000 7f58cea00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58a806b310 0x7f58a806d7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58c80dd0b0 tx=0x7f58c0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.843+0000 7f58cfe00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7f58a80734b0 0x7f58a80758d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:04.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.844+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=22) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58b800cee0 con 0x7f58c803e810 2024-01-16T15:00:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.846+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 --> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb4280cedd0 con 0x7fb400073460 2024-01-16T15:00:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.846+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== osd.7 v2:172.21.15.146:6809/902907051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f58a8075f80 con 0x7f58a80734b0 2024-01-16T15:00:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.846+0000 7fb41fe00640 1 -- 172.21.15.53:0/3608373581 <== osd.1 v2:172.21.15.53:6816/3560880694 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb418040040 con 0x7fb400073460 2024-01-16T15:00:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.846+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7fb400073460 msgr2=0x7fb400075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.846+0000 7fb42ea00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7fb400073460 0x7fb400075880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb40006b2c0 msgr2=0x7fb40006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ea00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb40006b2c0 0x7fb40006d780 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4280c86b0 tx=0x7fb420009040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 msgr2=0x7fb4280b4b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ea00640 1 --2- 172.21.15.53:0/3608373581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4280d0190 0x7fb4280b4b80 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb41000c480 tx=0x7fb41000c940 comp rx=0 tx=0).stop 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ca00640 1 -- 172.21.15.53:0/3608373581 reap_dead start 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 shutdown_connections 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.849+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 >> 172.21.15.53:0/3608373581 conn(0x7fb42809c440 msgr2=0x7fb4280d2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.851+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 shutdown_connections 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.851+0000 7fb42ea00640 1 -- 172.21.15.53:0/3608373581 wait complete. 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.863+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 --> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8884005360 con 0x7f8864073530 2024-01-16T15:00:04.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.863+0000 7f8882a00640 1 -- 172.21.15.53:0/1235404705 <== osd.2 v2:172.21.15.53:6824/16573798 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8884005360 con 0x7f8864073530 2024-01-16T15:00:04.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.864+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f8864073530 msgr2=0x7f8864075950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.863+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 --> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f58c0002cb0 con 0x7f58c8073520 2024-01-16T15:00:04.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.864+0000 7f888be00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f8864073530 0x7f8864075950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f58ecc00640 1 -- 172.21.15.53:0/1819463601 <== osd.0 v2:172.21.15.53:6800/2075393413 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f58c0002cb0 con 0x7f58c8073520 2024-01-16T15:00:04.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406b310 msgr2=0x7f886406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406b310 0x7f886406d7d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f887c035000 tx=0x7f887c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 msgr2=0x7f88840e4af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 --2- 172.21.15.53:0/1235404705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88840e3130 0x7f88840e4af0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f887400c3f0 tx=0x7f887400c8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888b400640 1 -- 172.21.15.53:0/1235404705 reap_dead start 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 shutdown_connections 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 >> 172.21.15.53:0/1235404705 conn(0x7f8884058eb0 msgr2=0x7f88840de8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.865+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 shutdown_connections 2024-01-16T15:00:04.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.866+0000 7f888be00640 1 -- 172.21.15.53:0/1235404705 wait complete. 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f58c8073520 msgr2=0x7f58c8075940 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f58c8073520 0x7f58c8075940 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58c806b380 msgr2=0x7f58c806d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58c806b380 0x7f58c806d840 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58d8006fd0 tx=0x7f58d8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58f0160920 msgr2=0x7f58f00cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 --2- 172.21.15.53:0/1819463601 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58f0160920 0x7f58f00cdbf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f58e802bf20 tx=0x7f58e8004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f5600640 1 -- 172.21.15.53:0/1819463601 reap_dead start 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 shutdown_connections 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.868+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 >> 172.21.15.53:0/1819463601 conn(0x7f58f00d3090 msgr2=0x7f58f006b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.869+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 shutdown_connections 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.869+0000 7f58f6000640 1 -- 172.21.15.53:0/1819463601 wait complete. 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.870+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 --> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6c4414b6c0 con 0x7f6c1c073560 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.872+0000 7f6c3aa00640 1 -- 172.21.15.53:0/4053326520 <== osd.5 v2:172.21.15.146:6824/2159577511 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6c4414b6c0 con 0x7f6c1c073560 2024-01-16T15:00:04.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.872+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 --> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2598002cb0 con 0x7f25a00735b0 2024-01-16T15:00:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.873+0000 7f25bf400640 1 -- 172.21.15.53:0/3353682231 <== osd.3 v2:172.21.15.53:6808/2110883972 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2598002cb0 con 0x7f25a00735b0 2024-01-16T15:00:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7f6c1c073560 msgr2=0x7f6c1c075980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7f6c1c073560 0x7f6c1c075980 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6c1c06b340 msgr2=0x7f6c1c06d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6c1c06b340 0x7f6c1c06d800 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009ac0 tx=0x7f6c34008040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4413e360 msgr2=0x7f6c44152b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 --2- 172.21.15.53:0/4053326520 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c4413e360 0x7f6c44152b30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c02b750 tx=0x7f6c3c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c42800640 1 -- 172.21.15.53:0/4053326520 reap_dead start 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7f25a00735b0 msgr2=0x7f25a00759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7f25a00735b0 0x7f25a00759d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 shutdown_connections 2024-01-16T15:00:04.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.874+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 >> 172.21.15.53:0/4053326520 conn(0x7f6c4406a800 msgr2=0x7f6c4415d8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a006b390 msgr2=0x7f25a006d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a006b390 0x7f25a006d850 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f25b8036000 tx=0x7f25b803b040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c80b30b0 msgr2=0x7f25c80cb8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 --2- 172.21.15.53:0/3353682231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25c80b30b0 0x7f25c80cb8c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f25b000c400 tx=0x7f25b000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25c7e00640 1 -- 172.21.15.53:0/3353682231 reap_dead start 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 shutdown_connections 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 >> 172.21.15.53:0/3353682231 conn(0x7f25c80b0be0 msgr2=0x7f25c806d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 shutdown_connections 2024-01-16T15:00:04.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7fde12a00640 1 -- 172.21.15.53:0/1060578075 --> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fde0c0c5eb0 con 0x7fdde40735b0 2024-01-16T15:00:04.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.875+0000 7f25ce800640 1 -- 172.21.15.53:0/3353682231 wait complete. 2024-01-16T15:00:04.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.877+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 shutdown_connections 2024-01-16T15:00:04.875 INFO:teuthology.orchestra.run.smithi053.stdout:60129542151 2024-01-16T15:00:04.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.877+0000 7f6c49000640 1 -- 172.21.15.53:0/4053326520 wait complete. 2024-01-16T15:00:04.876 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T15:00:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.877+0000 7fde03e00640 1 -- 172.21.15.53:0/1060578075 <== osd.4 v2:172.21.15.146:6808/981755469 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fde0c0c5eb0 con 0x7fdde40735b0 2024-01-16T15:00:04.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.877+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 --> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f58c8006380 con 0x7f58a80734b0 2024-01-16T15:00:04.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.878+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 --> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa684002cb0 con 0x7fa68c073460 2024-01-16T15:00:04.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.878+0000 7f58c7400640 1 -- 172.21.15.53:0/2792425572 <== osd.7 v2:172.21.15.146:6809/902907051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f58d00ae5b0 con 0x7f58a80734b0 2024-01-16T15:00:04.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7f58a80734b0 msgr2=0x7f58a80758d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7f58d4c00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7f58a80734b0 0x7f58a80758d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7fa6abe00640 1 -- 172.21.15.53:0/920522417 <== osd.6 v2:172.21.15.146:6800/2492658325 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa684002cb0 con 0x7fa68c073460 2024-01-16T15:00:04.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7fdde40735b0 msgr2=0x7fdde40759d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7fde01600640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7fdde40735b0 0x7fdde40759d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58a806b310 msgr2=0x7f58a806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7f58d4c00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f58a806b310 0x7f58a806d7d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f58c80dd0b0 tx=0x7f58c0009040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58c803e810 msgr2=0x7f58c80cb7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.880+0000 7f58d4c00640 1 --2- 172.21.15.53:0/2792425572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58c803e810 0x7f58c80cb7b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f58b800e9e0 tx=0x7f58b800eea0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fa68c073460 msgr2=0x7fa68c075880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fa68c073460 0x7fa68c075880 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7f58cfe00640 1 -- 172.21.15.53:0/2792425572 reap_dead start 2024-01-16T15:00:04.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 shutdown_connections 2024-01-16T15:00:04.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 >> 172.21.15.53:0/2792425572 conn(0x7f58c80559b0 msgr2=0x7f58c80e0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa68c06b2c0 msgr2=0x7fa68c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa68c06b2c0 0x7fa68c06d780 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa69c00b830 tx=0x7fa69c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406b390 msgr2=0x7fdde406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406b390 0x7fdde406d850 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fddfc000f60 tx=0x7fddfc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c09a170 msgr2=0x7fde0c0c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 --2- 172.21.15.53:0/1060578075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c09a170 0x7fde0c0c7480 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fddf40098a0 tx=0x7fddf40046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde10a00640 1 -- 172.21.15.53:0/1060578075 reap_dead start 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 shutdown_connections 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b40ced30 msgr2=0x7fa6b4158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:04.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 --2- 172.21.15.53:0/920522417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6b40ced30 0x7fa6b4158790 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400c450 tx=0x7fa6a400c910 comp rx=0 tx=0).stop 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6b8a00640 1 -- 172.21.15.53:0/920522417 reap_dead start 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 shutdown_connections 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 >> 172.21.15.53:0/920522417 conn(0x7fa6b409c550 msgr2=0x7fa6b4161360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 shutdown_connections 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 >> 172.21.15.53:0/1060578075 conn(0x7fde0c0b2d00 msgr2=0x7fde0c06bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 shutdown_connections 2024-01-16T15:00:04.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fa6baa00640 1 -- 172.21.15.53:0/920522417 wait complete. 2024-01-16T15:00:04.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7f58d4c00640 1 -- 172.21.15.53:0/2792425572 wait complete. 2024-01-16T15:00:04.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 shutdown_connections 2024-01-16T15:00:04.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:04.881+0000 7fde01600640 1 -- 172.21.15.53:0/1060578075 wait complete. 2024-01-16T15:00:04.884 INFO:teuthology.orchestra.run.smithi053.stdout:60129542149 2024-01-16T15:00:04.884 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T15:00:04.885 INFO:teuthology.orchestra.run.smithi053.stdout:60129542149 2024-01-16T15:00:04.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T15:00:04.886 INFO:teuthology.orchestra.run.smithi053.stdout:60129542150 2024-01-16T15:00:04.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T15:00:04.887 INFO:teuthology.orchestra.run.smithi053.stdout:60129542150 2024-01-16T15:00:04.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T15:00:04.888 INFO:teuthology.orchestra.run.smithi053.stdout:64424509445 2024-01-16T15:00:04.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T15:00:04.890 INFO:teuthology.orchestra.run.smithi053.stdout:64424509447 2024-01-16T15:00:04.890 INFO:teuthology.orchestra.run.smithi053.stdout:60129542149 2024-01-16T15:00:04.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T15:00:04.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T15:00:05.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.168+0000 7fbbc6c00640 1 Processor -- start 2024-01-16T15:00:05.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.168+0000 7fa2f9a00640 1 Processor -- start 2024-01-16T15:00:05.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.168+0000 7fbbc6c00640 1 -- start start 2024-01-16T15:00:05.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.168+0000 7fa2f9a00640 1 -- start start 2024-01-16T15:00:05.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.170+0000 7fa2f9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f40ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fbbbfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbbc015a320 0x7fbbc006c340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49740/0 (socket says 172.21.15.53:49740) 2024-01-16T15:00:05.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fbbbfe00640 1 -- 172.21.15.53:0/3444877728 learned_addr learned my addr 172.21.15.53:0/3444877728 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:05.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fa2f9a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f415a010 con 0x7fa2f4072290 2024-01-16T15:00:05.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fa2f9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f415a2f0 con 0x7fa2f40b15f0 2024-01-16T15:00:05.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fa2f9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f415a5d0 con 0x7fa2f40cc7f0 2024-01-16T15:00:05.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fbbc6c00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbbc00b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fbbc6c00640 1 --2- 172.21.15.53:0/3444877728 >> v2:172.21.15.146:3300/0 conn(0x7fbbc00cc7b0 0x7fbbc00ccc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00b4870 con 0x7fbbc015a320 2024-01-16T15:00:05.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc015a700 con 0x7fbbc00cc7b0 2024-01-16T15:00:05.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc006cdc0 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fbbc4c00640 1 --2- 172.21.15.53:0/3444877728 >> v2:172.21.15.146:3300/0 conn(0x7fbbc00cc7b0 0x7fbbc00ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.173+0000 7fbbbf400640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbbc00b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fa2f3e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa2f40b15f0 0x7fa2f4159750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38408/0 (socket says 172.21.15.53:38408) 2024-01-16T15:00:05.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fa2f3e00640 1 -- 172.21.15.53:0/3949193122 learned_addr learned my addr 172.21.15.53:0/3949193122 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:05.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.171+0000 7fa2f9000640 1 --2- 172.21.15.53:0/3949193122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f40ccc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fa2f9000640 1 -- 172.21.15.53:0/3949193122 >> v1:172.21.15.146:6789/0 conn(0x7fa2f40b15f0 legacy=0x7fa2f4159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fa2f9000640 1 -- 172.21.15.53:0/3949193122 >> v1:172.21.15.53:6789/0 conn(0x7fa2f4072290 legacy=0x7fa2f4153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fa2f9000640 1 -- 172.21.15.53:0/3949193122 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4070400 con 0x7fa2f40cc7f0 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.172+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213134198 0 0) 0x7fbbb0001110 con 0x7fbbc015a320 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.173+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba0003520 con 0x7fbbc015a320 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2997275541 0 0) 0x7fbba0003520 con 0x7fbbc015a320 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 msgr2=0x7fbbc00b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbbc00b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 >> v2:172.21.15.146:3300/0 conn(0x7fbbc00cc7b0 msgr2=0x7fbbc00ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 --2- 172.21.15.53:0/3444877728 >> v2:172.21.15.146:3300/0 conn(0x7fbbc00cc7b0 0x7fbbc00ccc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc006d0a0 con 0x7fbbc015a320 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fbbb0001800 con 0x7fbbc015a320 2024-01-16T15:00:05.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 >> v1:172.21.15.53:6789/0 conn(0x7fbbc015a320 legacy=0x7fbbc006c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.174+0000 7fa2f9000640 1 --2- 172.21.15.53:0/3949193122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f40ccc40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0097d0 tx=0x7fa2dc02b9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=834f38847ed47609 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbea00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbba0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbea00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba0003eb0 0x7fbba00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbea00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba0004d00 0x7fbba0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc015a700 con 0x7fbba0003eb0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc006cdc0 con 0x7fbba0004d00 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba000b980 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbba0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fbbbf400640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba0003eb0 0x7fbba00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fa2f2a00640 1 -- 172.21.15.53:0/3949193122 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc030890 con 0x7fa2f40cc7f0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.175+0000 7fa2f2a00640 1 -- 172.21.15.53:0/3949193122 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2dc0309f0 con 0x7fa2f40cc7f0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f2a00640 1 -- 172.21.15.53:0/3949193122 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2dc030d80 con 0x7fa2f40cc7f0 2024-01-16T15:00:05.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 -- 172.21.15.53:0/3949193122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 msgr2=0x7fa2f40ccc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/3949193122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f40ccc40 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa2dc0097d0 tx=0x7fa2dc02b9d0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 -- 172.21.15.53:0/3949193122 shutdown_connections 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/3949193122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f40ccc40 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 -- 172.21.15.53:0/3949193122 >> 172.21.15.53:0/3949193122 conn(0x7fa2f40b0b00 msgr2=0x7fa2f40b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 -- 172.21.15.53:0/3949193122 shutdown_connections 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fbbc4c00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba0004d00 0x7fbba0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.176+0000 7fa2f9a00640 1 -- 172.21.15.53:0/3949193122 wait complete. 2024-01-16T15:00:05.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.177+0000 7fbbbfe00640 1 -- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba0004d00 msgr2=0x7fbba0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.177+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbba0004d00 0x7fbba0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.177+0000 7fbbbfe00640 1 -- 172.21.15.53:0/3444877728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba0003eb0 msgr2=0x7fbba00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.177+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba0003eb0 0x7fbba00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.177+0000 7fbbbfe00640 1 -- 172.21.15.53:0/3444877728 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba0003520 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.179+0000 7fbbbf400640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbba0003eb0 0x7fbba00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:05.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.179+0000 7fa2f9a00640 1 Processor -- start 2024-01-16T15:00:05.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.180+0000 7fa2f9a00640 1 -- start start 2024-01-16T15:00:05.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.180+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbba0003970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004fc0 tx=0x7fbba00057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3758929a52d3b7a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.181+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb0005ad0 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.182+0000 7fa2f9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2f4072290 0x7fa2f4202fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.181+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb002f070 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.181+0000 7fbbbea00640 1 -- 172.21.15.53:0/3444877728 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb00080d0 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.182+0000 7fa2f9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 0x7fa2f4160000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.182+0000 7fa2f9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f4160540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.182+0000 7fa2f3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 0x7fa2f4160000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 0x7fa2f4160000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34932/0 (socket says 172.21.15.53:34932) 2024-01-16T15:00:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f3400640 1 -- 172.21.15.53:0/53986302 learned_addr learned my addr 172.21.15.53:0/53986302 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 msgr2=0x7fbba0003970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4157530 con 0x7fa2f40b15f0 2024-01-16T15:00:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f9000640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f4160540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f42034f0 con 0x7fa2f4072290 2024-01-16T15:00:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f3e00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2f4072290 0x7fa2f4202fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f4155e60 con 0x7fa2f40cc7f0 2024-01-16T15:00:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.183+0000 7fbbc6c00640 1 --2- 172.21.15.53:0/3444877728 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00b3eb0 0x7fbba0003970 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004fc0 tx=0x7fbba00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.184+0000 7fbbc4c00640 1 -- 172.21.15.53:0/3444877728 reap_dead start 2024-01-16T15:00:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.184+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 shutdown_connections 2024-01-16T15:00:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.184+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 >> 172.21.15.53:0/3444877728 conn(0x7fbbc00aea00 msgr2=0x7fbbc00aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.184+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 shutdown_connections 2024-01-16T15:00:05.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.184+0000 7fa2f3400640 1 -- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 msgr2=0x7fa2f4160540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fbbc6c00640 1 -- 172.21.15.53:0/3444877728 wait complete. 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fa2f3400640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f4160540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fa2f3400640 1 -- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2f4072290 msgr2=0x7fa2f4202fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fa2f3400640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2f4072290 0x7fa2f4202fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fa2f3e00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2f4072290 0x7fa2f4202fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fa2f9000640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f4160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.185+0000 7fa2f3400640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2dc009400 con 0x7fa2f40b15f0 2024-01-16T15:00:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.186+0000 7fa2f3400640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 0x7fa2f4160000 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c790 tx=0x7fa2ec00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.187+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00cea0 con 0x7fa2f40b15f0 2024-01-16T15:00:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.187+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec003b90 con 0x7fa2f40b15f0 2024-01-16T15:00:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.187+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f4156140 con 0x7fa2f40b15f0 2024-01-16T15:00:05.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.187+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f4156700 con 0x7fa2f40b15f0 2024-01-16T15:00:05.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.187+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00a7f0 con 0x7fa2f40b15f0 2024-01-16T15:00:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.189+0000 7fbbc6c00640 1 Processor -- start 2024-01-16T15:00:05.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.190+0000 7fbbc6c00640 1 -- start start 2024-01-16T15:00:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fbbc6c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00b3eb0 0x7fbbc0155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fbbc6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 0x7fbbc0132590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fbbc6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc0132ad0 0x7fbbc00711e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fbbc6c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0155f50 con 0x7fbbc00b3eb0 2024-01-16T15:00:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fbbc6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc0133cf0 con 0x7fbbc0132ad0 2024-01-16T15:00:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa2ec00a950 con 0x7fa2f40b15f0 2024-01-16T15:00:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.192+0000 7fbbc6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc01418f0 con 0x7fbbc0134180 2024-01-16T15:00:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.194+0000 7fbbc4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc0132ad0 0x7fbbc00711e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.194+0000 7fa2ebe00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2cc06b390 0x7fa2cc06d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.194+0000 7fbbbf400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 0x7fbbc0132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.194+0000 7fbbbf400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 0x7fbbc0132590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49524/0 (socket says 172.21.15.53:49524) 2024-01-16T15:00:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.194+0000 7fbbbf400640 1 -- 172.21.15.53:0/2692631404 learned_addr learned my addr 172.21.15.53:0/2692631404 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.194+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa2ec0897c0 con 0x7fa2f40b15f0 2024-01-16T15:00:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.195+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00b3eb0 0x7fbbc0155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.195+0000 7fa2f3e00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2cc06b390 0x7fa2cc06d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.197+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c4005250 con 0x7fa2f40b15f0 2024-01-16T15:00:05.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.197+0000 7fbbbf400640 1 -- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc0132ad0 msgr2=0x7fbbc00711e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.197+0000 7fbbbf400640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc0132ad0 0x7fbbc00711e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.197+0000 7fbbbf400640 1 -- 172.21.15.53:0/2692631404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00b3eb0 msgr2=0x7fbbc0155a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.197+0000 7fbbbf400640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00b3eb0 0x7fbbc0155a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.198+0000 7fbbbf400640 1 -- 172.21.15.53:0/2692631404 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc01330e0 con 0x7fbbc0134180 2024-01-16T15:00:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.198+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00b3eb0 0x7fbbc0155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:05.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.199+0000 7fbbbf400640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 0x7fbbc0132590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbba80108c0 tx=0x7fbba8010d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.199+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba800eb60 con 0x7fbbc0134180 2024-01-16T15:00:05.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.200+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbba800ecc0 con 0x7fbbc0134180 2024-01-16T15:00:05.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.200+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbba801c400 con 0x7fbbc0134180 2024-01-16T15:00:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.201+0000 7fbbc6c00640 1 -- 172.21.15.53:0/2692631404 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc01333c0 con 0x7fbbc0134180 2024-01-16T15:00:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.202+0000 7fbbc6c00640 1 -- 172.21.15.53:0/2692631404 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc0133980 con 0x7fbbc0134180 2024-01-16T15:00:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.203+0000 7fa2f3e00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2cc06b390 0x7fa2cc06d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4005880 tx=0x7fa2e4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.205+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb88005250 con 0x7fbbc0134180 2024-01-16T15:00:05.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.207+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbba801c560 con 0x7fbbc0134180 2024-01-16T15:00:05.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.209+0000 7fbbb7e00640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806b2f0 0x7fbb9806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.209+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa2ec054750 con 0x7fa2f40b15f0 2024-01-16T15:00:05.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.209+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbba8058c20 con 0x7fbbc0134180 2024-01-16T15:00:05.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.210+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806b2f0 0x7fbb9806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.211+0000 7fbbbfe00640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806b2f0 0x7fbb9806d7b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004f90 tx=0x7fbbb000a070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.227+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbba80556f0 con 0x7fbbc0134180 2024-01-16T15:00:05.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.239+0000 7fa7d1000640 1 Processor -- start 2024-01-16T15:00:05.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.239+0000 7fa7d1000640 1 -- start start 2024-01-16T15:00:05.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.241+0000 7fa7d1000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa7cc0d1c40 0x7fa7cc0b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.242+0000 7fa7d1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0b3400 con 0x7fa7cc0d17e0 2024-01-16T15:00:05.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.242+0000 7fa7c9e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa7cc15d610 0x7fa7cc0ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38420/0 (socket says 172.21.15.53:38420) 2024-01-16T15:00:05.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.242+0000 7fa7c9e00640 1 -- 172.21.15.53:0/845626961 learned_addr learned my addr 172.21.15.53:0/845626961 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:05.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.242+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7cc0b27f0 con 0x7fa7cc15d610 2024-01-16T15:00:05.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.242+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7cc0d1390 con 0x7fa7cc0d1c40 2024-01-16T15:00:05.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.243+0000 7fa7ca800640 1 --2- 172.21.15.53:0/845626961 >> v2:172.21.15.53:3300/0 conn(0x7fa7cc0d1c40 0x7fa7cc0b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.243+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870516529 0 0) 0x7fa7cc0b27f0 con 0x7fa7cc15d610 2024-01-16T15:00:05.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.243+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7ac003520 con 0x7fa7cc15d610 2024-01-16T15:00:05.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2040586136 0 0) 0x7fa7cc0b3400 con 0x7fa7cc0d17e0 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7cc0b27f0 con 0x7fa7cc0d17e0 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1932759420 0 0) 0x7fa7ac003520 con 0x7fa7cc15d610 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 >> v2:172.21.15.53:3300/0 conn(0x7fa7cc0d1c40 msgr2=0x7fa7cc0b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 --2- 172.21.15.53:0/845626961 >> v2:172.21.15.53:3300/0 conn(0x7fa7cc0d1c40 0x7fa7cc0b3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 >> v1:172.21.15.53:6789/0 conn(0x7fa7cc0d17e0 legacy=0x7fa7cc0b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0b2ad0 con 0x7fa7cc15d610 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fa7bc001670 con 0x7fa7cc15d610 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 >> v1:172.21.15.146:6789/0 conn(0x7fa7cc15d610 legacy=0x7fa7cc0ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.244+0000 7fa7c8a00640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac003b00 0x7fa7ac003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.245+0000 7fa7c8a00640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7ac0047c0 0x7fa7ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.245+0000 7fa7c8a00640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7ac005180 0x7fa7ac014190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.245+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0d1390 con 0x7fa7ac003b00 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.245+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac0147b0 con 0x7fa7ac005180 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.245+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7ac014970 con 0x7fa7ac0047c0 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.246+0000 7fa7c9400640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7ac0047c0 0x7fa7ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.246+0000 7fa7ca800640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7ac005180 0x7fa7ac014190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.246+0000 7fa7c9e00640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac003b00 0x7fa7ac003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.247+0000 7fa7c9400640 1 -- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7ac005180 msgr2=0x7fa7ac014190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.247+0000 7fa7c9400640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7ac005180 0x7fa7ac014190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.247+0000 7fa7c9400640 1 -- 172.21.15.53:0/845626961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac003b00 msgr2=0x7fa7ac003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.247+0000 7fa7c9400640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac003b00 0x7fa7ac003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.247+0000 7fa7c9400640 1 -- 172.21.15.53:0/845626961 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc0b2ad0 con 0x7fa7ac0047c0 2024-01-16T15:00:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.248+0000 7fa7ca800640 1 -- 172.21.15.53:0/845626961 reap_dead start 2024-01-16T15:00:05.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.249+0000 7fa7c9400640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7ac0047c0 0x7fa7ac004c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4002010 tx=0x7fa7ac0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=101f2e309da67db5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.250+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4006d50 con 0x7fa7ac0047c0 2024-01-16T15:00:05.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.250+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c402c550 con 0x7fa7ac0047c0 2024-01-16T15:00:05.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.250+0000 7fa7c8a00640 1 -- 172.21.15.53:0/845626961 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c402c980 con 0x7fa7ac0047c0 2024-01-16T15:00:05.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7ac0047c0 msgr2=0x7fa7ac004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fa7d1000640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7ac0047c0 0x7fa7ac004c40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa7c4002010 tx=0x7fa7ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 shutdown_connections 2024-01-16T15:00:05.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fa7d1000640 1 --2- 172.21.15.53:0/845626961 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7ac0047c0 0x7fa7ac004c40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 >> 172.21.15.53:0/845626961 conn(0x7fa7cc09c540 msgr2=0x7fa7cc0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fb11be00640 1 Processor -- start 2024-01-16T15:00:05.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.251+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 shutdown_connections 2024-01-16T15:00:05.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.252+0000 7fb11be00640 1 -- start start 2024-01-16T15:00:05.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.252+0000 7fa7d1000640 1 -- 172.21.15.53:0/845626961 wait complete. 2024-01-16T15:00:05.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.254+0000 7fa7d1000640 1 Processor -- start 2024-01-16T15:00:05.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.255+0000 7fb11be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c0adca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.255+0000 7fb11be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb11c0d0900 0x7fb11c0d17d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.255+0000 7fb11be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb11c0b2170 con 0x7fb11c15bbd0 2024-01-16T15:00:05.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.255+0000 7fb11be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb11c0b2450 con 0x7fb11c0d0900 2024-01-16T15:00:05.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.255+0000 7fb11be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c0b2730 con 0x7fb11c15b7f0 2024-01-16T15:00:05.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fb11b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb11c15bbd0 0x7fb11c15c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38428/0 (socket says 172.21.15.53:38428) 2024-01-16T15:00:05.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fb11b400640 1 -- 172.21.15.53:0/3923629926 learned_addr learned my addr 172.21.15.53:0/3923629926 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:05.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.254+0000 7fa7d1000640 1 -- start start 2024-01-16T15:00:05.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fb11aa00640 1 --2- 172.21.15.53:0/3923629926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c0adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac0047c0 0x7fa7cc0b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc151000 0x7fa7cc0b1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fa7d1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc15fe00 0x7fa7cc160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fa7d1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc155700 con 0x7fa7ac0047c0 2024-01-16T15:00:05.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fa7d1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b15e0 con 0x7fa7cc15fe00 2024-01-16T15:00:05.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fb11a000640 1 --2- 172.21.15.53:0/3923629926 >> v2:172.21.15.146:3300/0 conn(0x7fb11c0d0900 0x7fb11c0d17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.256+0000 7fa7d1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7cc0b1300 con 0x7fa7cc151000 2024-01-16T15:00:05.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7f3ae5000640 1 Processor -- start 2024-01-16T15:00:05.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7f3ae5000640 1 -- start start 2024-01-16T15:00:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fa7c9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc151000 0x7fa7cc0b1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fb11aa00640 1 -- 172.21.15.53:0/3923629926 >> v2:172.21.15.146:3300/0 conn(0x7fb11c0d0900 msgr2=0x7fb11c0d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fb11aa00640 1 --2- 172.21.15.53:0/3923629926 >> v2:172.21.15.146:3300/0 conn(0x7fb11c0d0900 0x7fb11c0d17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fa7c9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc151000 0x7fa7cc0b1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49556/0 (socket says 172.21.15.53:49556) 2024-01-16T15:00:05.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fa7c9400640 1 -- 172.21.15.53:0/1743759211 learned_addr learned my addr 172.21.15.53:0/1743759211 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fde3c000640 1 Processor -- start 2024-01-16T15:00:05.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fa7ca800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc15fe00 0x7fa7cc160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.257+0000 7fa7c9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac0047c0 0x7fa7cc0b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fb119600640 1 -- 172.21.15.53:0/3923629926 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 43779615 0 0) 0x7fb11c0b2170 con 0x7fb11c15bbd0 2024-01-16T15:00:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fb11aa00640 1 -- 172.21.15.53:0/3923629926 >> v1:172.21.15.146:6789/0 conn(0x7fb11c15bbd0 legacy=0x7fb11c15c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fb11aa00640 1 -- 172.21.15.53:0/3923629926 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb11c0b08d0 con 0x7fb11c15b7f0 2024-01-16T15:00:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fde3c000640 1 -- start start 2024-01-16T15:00:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fa7ca800640 1 -- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc151000 msgr2=0x7fa7cc0b1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fa7ca800640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc151000 0x7fa7cc0b1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fa7ca800640 1 -- 172.21.15.53:0/1743759211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac0047c0 msgr2=0x7fa7cc0b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.258+0000 7fa7ca800640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac0047c0 0x7fa7cc0b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7fa7ca800640 1 -- 172.21.15.53:0/1743759211 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7cc1550b0 con 0x7fa7cc15fe00 2024-01-16T15:00:05.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ae5000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae0143ae0 0x7f3ae00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ae5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae00cf7d0 0x7f3ae015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ae5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ae015e1e0 0x7f3ae015a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ae5000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae00cfc30 con 0x7f3ae015e1e0 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ae5000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae00b1ee0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ae5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00b2780 con 0x7f3ae00cf7d0 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3adde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae0143ae0 0x7f3ae00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.259+0000 7f3ade800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ae015e1e0 0x7f3ae015a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7f3ade800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ae015e1e0 0x7f3ae015a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34982/0 (socket says 172.21.15.53:34982) 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7f3adde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae0143ae0 0x7f3ae00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47740/0 (socket says 172.21.15.53:47740) 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7f3ade800640 1 -- 172.21.15.53:0/3503389742 learned_addr learned my addr 172.21.15.53:0/3503389742 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7fde3c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde3415d460 0x7fde340ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7fde3c000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde3406e380 0x7fde340b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7fde3c000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3415d840 con 0x7fde340b1ee0 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7fde3c000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde34071490 con 0x7fde3406e380 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7fde3c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde340b3790 con 0x7fde3415d460 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.260+0000 7f3add400640 1 --2- 172.21.15.53:0/3503389742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae00cf7d0 0x7f3ae015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7f3ade800640 1 -- 172.21.15.53:0/3503389742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae00cf7d0 msgr2=0x7f3ae015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7f3ade800640 1 --2- 172.21.15.53:0/3503389742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae00cf7d0 0x7f3ae015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7f3ade800640 1 -- 172.21.15.53:0/3503389742 >> v2:172.21.15.146:3300/0 conn(0x7f3ae0143ae0 msgr2=0x7f3ae00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7f3ade800640 1 --2- 172.21.15.53:0/3503389742 >> v2:172.21.15.146:3300/0 conn(0x7f3ae0143ae0 0x7f3ae00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7fde3a000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fde340b1ee0 0x7fde3406d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38432/0 (socket says 172.21.15.53:38432) 2024-01-16T15:00:05.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7fde3a000640 1 -- 172.21.15.53:0/59149268 learned_addr learned my addr 172.21.15.53:0/59149268 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:05.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7f3ade800640 1 -- 172.21.15.53:0/3503389742 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae015e990 con 0x7f3ae015e1e0 2024-01-16T15:00:05.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7fa7ca800640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc15fe00 0x7fa7cc160280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc08baf0 tx=0x7fa7b4009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7fde39600640 1 --2- 172.21.15.53:0/59149268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde3415d460 0x7fde340ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7fde38c00640 1 --2- 172.21.15.53:0/59149268 >> v2:172.21.15.53:3300/0 conn(0x7fde3406e380 0x7fde340b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.261+0000 7fb11aa00640 1 --2- 172.21.15.53:0/3923629926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c0adca0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1140097a0 tx=0x7fb11402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85ded51fb97b0e77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fde39600640 1 -- 172.21.15.53:0/59149268 >> v2:172.21.15.53:3300/0 conn(0x7fde3406e380 msgr2=0x7fde340b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fb119600640 1 -- 172.21.15.53:0/3923629926 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114031780 con 0x7fb11c15b7f0 2024-01-16T15:00:05.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fde33e00640 1 -- 172.21.15.53:0/59149268 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801661283 0 0) 0x7fde3415d840 con 0x7fde340b1ee0 2024-01-16T15:00:05.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fb119600640 1 -- 172.21.15.53:0/3923629926 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1140318e0 con 0x7fb11c15b7f0 2024-01-16T15:00:05.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4004440 con 0x7fa7cc15fe00 2024-01-16T15:00:05.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b4003610 con 0x7fa7cc15fe00 2024-01-16T15:00:05.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4003a00 con 0x7fa7cc15fe00 2024-01-16T15:00:05.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7cc155390 con 0x7fa7cc15fe00 2024-01-16T15:00:05.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.262+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7cc151480 con 0x7fa7cc15fe00 2024-01-16T15:00:05.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.263+0000 7f3ade800640 1 --2- 172.21.15.53:0/3503389742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae015e1e0 0x7f3ae015a670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ae00cfea0 tx=0x7f3ac802e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be246d7f1a92b596 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.263+0000 7fde39600640 1 --2- 172.21.15.53:0/59149268 >> v2:172.21.15.53:3300/0 conn(0x7fde3406e380 0x7fde340b29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.263+0000 7fde39600640 1 -- 172.21.15.53:0/59149268 >> v1:172.21.15.146:6789/0 conn(0x7fde340b1ee0 legacy=0x7fde3406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.263+0000 7fb119600640 1 -- 172.21.15.53:0/3923629926 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb114031c70 con 0x7fb11c15b7f0 2024-01-16T15:00:05.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7f3adca00640 1 -- 172.21.15.53:0/3503389742 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac802ee80 con 0x7f3ae015e1e0 2024-01-16T15:00:05.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7f3adca00640 1 -- 172.21.15.53:0/3503389742 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac8036440 con 0x7f3ae015e1e0 2024-01-16T15:00:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7fb11be00640 1 -- 172.21.15.53:0/3923629926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 msgr2=0x7fb11c0adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7fb11be00640 1 --2- 172.21.15.53:0/3923629926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c0adca0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb1140097a0 tx=0x7fb11402b820 comp rx=0 tx=0).stop 2024-01-16T15:00:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7fb11be00640 1 -- 172.21.15.53:0/3923629926 shutdown_connections 2024-01-16T15:00:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7fb11be00640 1 --2- 172.21.15.53:0/3923629926 >> v2:172.21.15.146:3300/0 conn(0x7fb11c0d0900 0x7fb11c0d17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.264+0000 7fb11be00640 1 --2- 172.21.15.53:0/3923629926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c0adca0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7fb11be00640 1 -- 172.21.15.53:0/3923629926 >> 172.21.15.53:0/3923629926 conn(0x7fb11c099a60 msgr2=0x7fb11c099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7f3ae5000640 1 -- 172.21.15.53:0/3503389742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae015e1e0 msgr2=0x7f3ae015a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3503389742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae015e1e0 0x7f3ae015a670 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ae00cfea0 tx=0x7f3ac802e0d0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.263+0000 7fde39600640 1 -- 172.21.15.53:0/59149268 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde34072190 con 0x7fde3415d460 2024-01-16T15:00:05.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7fde38c00640 1 --2- 172.21.15.53:0/59149268 >> v2:172.21.15.53:3300/0 conn(0x7fde3406e380 0x7fde340b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7fb11be00640 1 -- 172.21.15.53:0/3923629926 shutdown_connections 2024-01-16T15:00:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7fde39600640 1 --2- 172.21.15.53:0/59149268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde3415d460 0x7fde340ac8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009d90 tx=0x7fde2c02bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61b27e6214465ac9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 -- 172.21.15.53:0/3503389742 shutdown_connections 2024-01-16T15:00:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3503389742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae015e1e0 0x7f3ae015a670 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3503389742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae00cf7d0 0x7f3ae015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3503389742 >> v2:172.21.15.146:3300/0 conn(0x7f3ae0143ae0 0x7f3ae00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 -- 172.21.15.53:0/3503389742 >> 172.21.15.53:0/3503389742 conn(0x7f3ae006a800 msgr2=0x7f3ae006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7fde33e00640 1 -- 172.21.15.53:0/59149268 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c02cc80 con 0x7fde3415d460 2024-01-16T15:00:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7fde33e00640 1 -- 172.21.15.53:0/59149268 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde2c02cde0 con 0x7fde3415d460 2024-01-16T15:00:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7fde33e00640 1 -- 172.21.15.53:0/59149268 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde2c02cc80 con 0x7fde3415d460 2024-01-16T15:00:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7cc155520 con 0x7fa7cc15fe00 2024-01-16T15:00:05.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 -- 172.21.15.53:0/3503389742 shutdown_connections 2024-01-16T15:00:05.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.266+0000 7f3ae5000640 1 -- 172.21.15.53:0/3503389742 wait complete. 2024-01-16T15:00:05.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.265+0000 7fb11be00640 1 -- 172.21.15.53:0/3923629926 wait complete. 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.267+0000 7fde3c000640 1 -- 172.21.15.53:0/59149268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde3415d460 msgr2=0x7fde340ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.267+0000 7fde3c000640 1 --2- 172.21.15.53:0/59149268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde3415d460 0x7fde340ac8d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009d90 tx=0x7fde2c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.267+0000 7fde3c000640 1 -- 172.21.15.53:0/59149268 shutdown_connections 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.267+0000 7fde3c000640 1 --2- 172.21.15.53:0/59149268 >> v2:172.21.15.53:3300/0 conn(0x7fde3406e380 0x7fde340b29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.268+0000 7fde3c000640 1 --2- 172.21.15.53:0/59149268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde3415d460 0x7fde340ac8d0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009d90 tx=0x7fde2c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.268+0000 7fde3c000640 1 -- 172.21.15.53:0/59149268 >> 172.21.15.53:0/59149268 conn(0x7fde340b0be0 msgr2=0x7fde340b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.268+0000 7fde3c000640 1 -- 172.21.15.53:0/59149268 shutdown_connections 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.269+0000 7f3ae5000640 1 Processor -- start 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.268+0000 7fde3c000640 1 -- 172.21.15.53:0/59149268 wait complete. 2024-01-16T15:00:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.269+0000 7fb11be00640 1 Processor -- start 2024-01-16T15:00:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.269+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa7b4003b60 con 0x7fa7cc15fe00 2024-01-16T15:00:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.270+0000 7fb11be00640 1 -- start start 2024-01-16T15:00:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.269+0000 7f3ae5000640 1 -- start start 2024-01-16T15:00:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7fa7c2000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7a406b390 0x7fa7a406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.270+0000 7f3ae5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae00cf7d0 0x7f3ae00ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f3ae5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 0x7f3ae013e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f3ae5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae015e1e0 0x7f3ae0140cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa7b4089e50 con 0x7fa7cc15fe00 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f3adde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae00cf7d0 0x7f3ae00ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f3adde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae00cf7d0 0x7f3ae00ac170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47754/0 (socket says 172.21.15.53:47754) 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f3add400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 0x7f3ae013e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f3add400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 0x7f3ae013e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34998/0 (socket says 172.21.15.53:34998) 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7f3adde00640 1 -- 172.21.15.53:0/3246712605 learned_addr learned my addr 172.21.15.53:0/3246712605 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fb11be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0d0900 0x7fb11c156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fb11be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c144830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fb11be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bbd0 0x7fb11c1e4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fb11be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c1f5380 con 0x7fb11c15bbd0 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fb11be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c155560 con 0x7fb11c0d0900 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fb11be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb11c155840 con 0x7fb11c15b7f0 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae00ac6b0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae013fc30 con 0x7f3ae00cf7d0 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae013ff10 con 0x7f3ae015e1e0 2024-01-16T15:00:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fde3c000640 1 Processor -- start 2024-01-16T15:00:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7f3ade800640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae015e1e0 0x7f3ae0140cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fa7c9e00640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7a406b390 0x7fa7a406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f4226200640 1 Processor -- start 2024-01-16T15:00:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.271+0000 7f4226200640 1 -- start start 2024-01-16T15:00:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7fb11a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7fb11aa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0d0900 0x7fb11c156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f3add400640 1 -- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae015e1e0 msgr2=0x7f3ae0140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f3add400640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae015e1e0 0x7f3ae0140cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f3add400640 1 -- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae00cf7d0 msgr2=0x7f3ae00ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f3add400640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae00cf7d0 0x7f3ae00ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f3add400640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac800b770 con 0x7f3ae0143ae0 2024-01-16T15:00:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7fb11a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c144830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49594/0 (socket says 172.21.15.53:49594) 2024-01-16T15:00:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7fb11a000640 1 -- 172.21.15.53:0/305991540 learned_addr learned my addr 172.21.15.53:0/305991540 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f4226200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f422014b2c0 0x7f4220070750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f4226200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42200ae5b0 0x7f42200cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f4226200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42200cf770 con 0x7f42200cf340 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f4226200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f422014b6a0 con 0x7f42200ae5b0 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7f4226200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4220070fc0 con 0x7f422014b2c0 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7f421ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42200ae5b0 0x7f42200cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7f421ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42200ae5b0 0x7f42200cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35020/0 (socket says 172.21.15.53:35020) 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7f421ea00640 1 -- 172.21.15.53:0/811623845 learned_addr learned my addr 172.21.15.53:0/811623845 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7f421f400640 1 --2- 172.21.15.53:0/811623845 >> v2:172.21.15.146:3300/0 conn(0x7f422014b2c0 0x7f4220070750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.272+0000 7fde3c000640 1 -- start start 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7f3add400640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 0x7f3ae013e6d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000d850 tx=0x7f3ad000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fde3c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde3406e380 0x7fde34160690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fde3c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde340b1ee0 0x7fde34159470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fde3c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 0x7fde341f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fde3c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde3415ab20 con 0x7fde340b1ee0 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fde3c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde34159f10 con 0x7fde3406e380 2024-01-16T15:00:05.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fde3c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3415a1f0 con 0x7fde341599b0 2024-01-16T15:00:05.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.273+0000 7fb11b400640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bbd0 0x7fb11c1e4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.274+0000 7fa7c9e00640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7a406b390 0x7fa7a406d850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000fd0 tx=0x7fa7bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fb11a000640 1 -- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0d0900 msgr2=0x7fb11c156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fb11a000640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0d0900 0x7fb11c156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fb11a000640 1 -- 172.21.15.53:0/305991540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bbd0 msgr2=0x7fb11c1e4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fde3a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 0x7fde341f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f421f400640 1 -- 172.21.15.53:0/811623845 >> v2:172.21.15.53:3300/0 conn(0x7f42200ae5b0 msgr2=0x7f42200cee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f421f400640 1 --2- 172.21.15.53:0/811623845 >> v2:172.21.15.53:3300/0 conn(0x7f42200ae5b0 0x7f42200cee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f421f400640 1 -- 172.21.15.53:0/811623845 >> v1:172.21.15.53:6789/0 conn(0x7f42200cf340 legacy=0x7f422006c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fde3a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 0x7fde341f64a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49604/0 (socket says 172.21.15.53:49604) 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fde3a000640 1 -- 172.21.15.53:0/4185735645 learned_addr learned my addr 172.21.15.53:0/4185735645 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad00096a0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fde38c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde340b1ee0 0x7fde34159470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fb11a000640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bbd0 0x7fb11c1e4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fb11a000640 1 -- 172.21.15.53:0/305991540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb114009400 con 0x7fb11c15b7f0 2024-01-16T15:00:05.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fb11b400640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bbd0 0x7fb11c1e4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:05.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae01401f0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad0009820 con 0x7f3ae0143ae0 2024-01-16T15:00:05.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae01407b0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7fde39600640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde3406e380 0x7fde34160690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.275+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0009bd0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7fb11a000640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c144830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb10400e980 tx=0x7fb10400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7fde3a000640 1 -- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde3406e380 msgr2=0x7fde34160690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7fde3a000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde3406e380 0x7fde34160690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7fde3a000640 1 -- 172.21.15.53:0/4185735645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde340b1ee0 msgr2=0x7fde34159470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7fde3a000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde340b1ee0 0x7fde34159470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7fde3a000640 1 -- 172.21.15.53:0/4185735645 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c009400 con 0x7fde341599b0 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7f421f400640 1 -- 172.21.15.53:0/811623845 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42200712a0 con 0x7f422014b2c0 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.277+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104009670 con 0x7fb11c15b7f0 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.277+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10400ca00 con 0x7fb11c15b7f0 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.277+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10400cd90 con 0x7fb11c15b7f0 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.277+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb11c155b20 con 0x7fb11c15b7f0 2024-01-16T15:00:05.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.277+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb11c1e47d0 con 0x7fb11c15b7f0 2024-01-16T15:00:05.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7f078da00640 1 Processor -- start 2024-01-16T15:00:05.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.276+0000 7f078da00640 1 -- start start 2024-01-16T15:00:05.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.278+0000 7f421f400640 1 --2- 172.21.15.53:0/811623845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f4220070750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42200b2100 tx=0x7f421802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a69423b5f51e408 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.278+0000 7fde3a000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 0x7fde341f64a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c450 tx=0x7fde2400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.278+0000 7f078da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07880d1540 0x7f07880cfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f078da00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07880c6d80 0x7f07880d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f078da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07880cdd40 con 0x7f07880d1e60 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f078da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07880ce000 con 0x7f07880d1540 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f078da00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07880d0ab0 con 0x7f07880c6d80 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f0786a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07880d1540 0x7f07880cfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.278+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae0140970 con 0x7f3ae0143ae0 2024-01-16T15:00:05.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f0786a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07880d1540 0x7f07880cfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35040/0 (socket says 172.21.15.53:35040) 2024-01-16T15:00:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f0786a00640 1 -- 172.21.15.53:0/3242833240 learned_addr learned my addr 172.21.15.53:0/3242833240 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde240074e0 con 0x7fde341599b0 2024-01-16T15:00:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde24007660 con 0x7fde341599b0 2024-01-16T15:00:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde24004a20 con 0x7fde341599b0 2024-01-16T15:00:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c009a70 con 0x7fde341599b0 2024-01-16T15:00:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3ad0009d30 con 0x7f3ae0143ae0 2024-01-16T15:00:05.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3415a7b0 con 0x7fde341599b0 2024-01-16T15:00:05.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.279+0000 7f0786000640 1 --2- 172.21.15.53:0/3242833240 >> v2:172.21.15.146:3300/0 conn(0x7f07880c6d80 0x7f07880d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.280+0000 7f0786a00640 1 -- 172.21.15.53:0/3242833240 >> v2:172.21.15.146:3300/0 conn(0x7f07880c6d80 msgr2=0x7f07880d1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.280+0000 7f0786a00640 1 --2- 172.21.15.53:0/3242833240 >> v2:172.21.15.146:3300/0 conn(0x7f07880c6d80 0x7f07880d1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.280+0000 7f0786a00640 1 -- 172.21.15.53:0/3242833240 >> v1:172.21.15.53:6789/0 conn(0x7f07880d1e60 legacy=0x7f07880d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:05.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.280+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb11c067450 con 0x7fb11c15b7f0 2024-01-16T15:00:05.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.281+0000 7f3ad6000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ab806b390 0x7f3ab806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.280+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde340710c0 con 0x7fde341599b0 2024-01-16T15:00:05.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.281+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ad0089ed0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.281+0000 7f3adde00640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ab806b390 0x7f3ab806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.282+0000 7f0786a00640 1 -- 172.21.15.53:0/3242833240 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07880cff40 con 0x7f07880d1540 2024-01-16T15:00:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.282+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb104013470 con 0x7fb11c15b7f0 2024-01-16T15:00:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.282+0000 7f3adde00640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ab806b390 0x7f3ab806d850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009aa0 tx=0x7f3ad8003a50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.283+0000 7f0786a00640 1 --2- 172.21.15.53:0/3242833240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880d1540 0x7f07880cfa00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0788165510 tx=0x7f077802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45759fa0c4b093bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.284+0000 7fb112a00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb0f406b310 0x7fb0f406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.284+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb104089dc0 con 0x7fb11c15b7f0 2024-01-16T15:00:05.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.284+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fde2401b360 con 0x7fde341599b0 2024-01-16T15:00:05.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.285+0000 7fde31600640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde0c06b2c0 0x7fde0c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.285+0000 7fb11aa00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb0f406b310 0x7fb0f406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.285+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde24088f90 con 0x7fde341599b0 2024-01-16T15:00:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.286+0000 7fde39600640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde0c06b2c0 0x7fde0c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.287+0000 7fb11aa00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb0f406b310 0x7fb0f406d7d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb114036000 tx=0x7fb11403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.287+0000 7f421e000640 1 -- 172.21.15.53:0/811623845 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f421802e070 con 0x7f422014b2c0 2024-01-16T15:00:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.287+0000 7fde39600640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde0c06b2c0 0x7fde0c06d780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde2c036000 tx=0x7fde2c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.288+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa7b4054d50 con 0x7fa7cc15fe00 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.288+0000 7f421e000640 1 -- 172.21.15.53:0/811623845 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42180315a0 con 0x7f422014b2c0 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.288+0000 7f421e000640 1 -- 172.21.15.53:0/811623845 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218031930 con 0x7f422014b2c0 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f0785600640 1 -- 172.21.15.53:0/3242833240 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778030860 con 0x7f07880d1540 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f0785600640 1 -- 172.21.15.53:0/3242833240 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07780309c0 con 0x7f07880d1540 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f0785600640 1 -- 172.21.15.53:0/3242833240 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778030860 con 0x7f07880d1540 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 -- 172.21.15.53:0/811623845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 msgr2=0x7f4220070750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 --2- 172.21.15.53:0/811623845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f4220070750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f42200b2100 tx=0x7f421802b840 comp rx=0 tx=0).stop 2024-01-16T15:00:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ad0054d20 con 0x7f3ae0143ae0 2024-01-16T15:00:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 -- 172.21.15.53:0/811623845 shutdown_connections 2024-01-16T15:00:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 --2- 172.21.15.53:0/811623845 >> v2:172.21.15.53:3300/0 conn(0x7f42200ae5b0 0x7f42200cee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 --2- 172.21.15.53:0/811623845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f4220070750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 -- 172.21.15.53:0/811623845 >> 172.21.15.53:0/811623845 conn(0x7f422006a800 msgr2=0x7f422006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 -- 172.21.15.53:0/811623845 shutdown_connections 2024-01-16T15:00:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.289+0000 7f4226200640 1 -- 172.21.15.53:0/811623845 wait complete. 2024-01-16T15:00:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 -- 172.21.15.53:0/3242833240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880d1540 msgr2=0x7f07880cfa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 --2- 172.21.15.53:0/3242833240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880d1540 0x7f07880cfa00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0788165510 tx=0x7f077802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 -- 172.21.15.53:0/3242833240 shutdown_connections 2024-01-16T15:00:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 --2- 172.21.15.53:0/3242833240 >> v2:172.21.15.146:3300/0 conn(0x7f07880c6d80 0x7f07880d1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 --2- 172.21.15.53:0/3242833240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07880d1540 0x7f07880cfa00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 -- 172.21.15.53:0/3242833240 >> 172.21.15.53:0/3242833240 conn(0x7f078808cc50 msgr2=0x7f078809c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 -- 172.21.15.53:0/3242833240 shutdown_connections 2024-01-16T15:00:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.290+0000 7f078da00640 1 -- 172.21.15.53:0/3242833240 wait complete. 2024-01-16T15:00:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.292+0000 7f4226200640 1 Processor -- start 2024-01-16T15:00:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.292+0000 7f4226200640 1 -- start start 2024-01-16T15:00:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.293+0000 7f078da00640 1 Processor -- start 2024-01-16T15:00:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.293+0000 7f078da00640 1 -- start start 2024-01-16T15:00:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.294+0000 7f4226200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 0x7f4220144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.294+0000 7f4226200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42200cf340 0x7f422015ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.294+0000 7f4226200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f42200a8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.294+0000 7f4226200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f422015f390 con 0x7f42200ae5b0 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.294+0000 7f4226200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42200a9410 con 0x7f422014b2c0 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.294+0000 7f4226200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42200a96b0 con 0x7f42200cf340 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 0x7f4220144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 0x7f4220144de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35052/0 (socket says 172.21.15.53:35052) 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 -- 172.21.15.53:0/439077397 learned_addr learned my addr 172.21.15.53:0/439077397 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421fe00640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f42200a8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421ea00640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42200cf340 0x7f422015ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f078da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 0x7f078815f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f078da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07880b6820 0x7f07880ce390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 -- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42200cf340 msgr2=0x7f422015ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42200cf340 0x7f422015ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 -- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 msgr2=0x7f42200a8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f42200a8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.295+0000 7f421f400640 1 -- 172.21.15.53:0/439077397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4218009400 con 0x7f42200ae5b0 2024-01-16T15:00:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f078da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0788164a50 0x7f07880ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f421f400640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 0x7f4220144de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4218009890 tx=0x7f42180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f0786a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 0x7f078815f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f0786a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 0x7f078815f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47812/0 (socket says 172.21.15.53:47812) 2024-01-16T15:00:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f078da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07881f7e80 con 0x7f0788164a50 2024-01-16T15:00:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f0786a00640 1 -- 172.21.15.53:0/3867368255 learned_addr learned my addr 172.21.15.53:0/3867368255 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07881630d0 con 0x7f07880c6d80 2024-01-16T15:00:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218032840 con 0x7f42200ae5b0 2024-01-16T15:00:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.297+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42180329a0 con 0x7f42200ae5b0 2024-01-16T15:00:05.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07881f8310 con 0x7f07880b6820 2024-01-16T15:00:05.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.296+0000 7f0786000640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07880b6820 0x7f07880ce390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.297+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42200a9990 con 0x7f42200ae5b0 2024-01-16T15:00:05.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.297+0000 7f0787400640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0788164a50 0x7f07880ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.297+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4218032d50 con 0x7f42200ae5b0 2024-01-16T15:00:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.297+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42200a9f50 con 0x7f42200ae5b0 2024-01-16T15:00:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.298+0000 7f0786a00640 1 -- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07880b6820 msgr2=0x7f07880ce390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.298+0000 7f0786a00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07880b6820 0x7f07880ce390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.298+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42200675d0 con 0x7f42200ae5b0 2024-01-16T15:00:05.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.299+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde24053f90 con 0x7fde341599b0 2024-01-16T15:00:05.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.298+0000 7f0786a00640 1 -- 172.21.15.53:0/3867368255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0788164a50 msgr2=0x7f07880ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.298+0000 7f0786a00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0788164a50 0x7f07880ce8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.299+0000 7f0786a00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07780094d0 con 0x7f07880c6d80 2024-01-16T15:00:05.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.300+0000 7f0786a00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 0x7f078815f800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f077802b9e0 tx=0x7f07780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.300+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f42180453f0 con 0x7f42200ae5b0 2024-01-16T15:00:05.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077803e070 con 0x7f07880c6d80 2024-01-16T15:00:05.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f4217400640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41f806b310 0x7f41f806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f42180abec0 con 0x7f42200ae5b0 2024-01-16T15:00:05.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07780044c0 con 0x7f07880c6d80 2024-01-16T15:00:05.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07881f7550 con 0x7f07880c6d80 2024-01-16T15:00:05.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07881f7b10 con 0x7f07880c6d80 2024-01-16T15:00:05.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.301+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0778003850 con 0x7f07880c6d80 2024-01-16T15:00:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.302+0000 7f421ea00640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41f806b310 0x7f41f806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.304+0000 7f421ea00640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41f806b310 0x7f41f806d7d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4210009440 tx=0x7f4210009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.303+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07880675d0 con 0x7f07880c6d80 2024-01-16T15:00:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.304+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0778042070 con 0x7f07880c6d80 2024-01-16T15:00:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.306+0000 7f077ea00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f076006b340 0x7f076006d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.306+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f07780aba60 con 0x7f07880c6d80 2024-01-16T15:00:05.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.307+0000 7f0786000640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f076006b340 0x7f076006d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:05.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.309+0000 7f0786000640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f076006b340 0x7f076006d800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0770006fd0 tx=0x7f0770008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:05.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.310+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4218076e30 con 0x7f42200ae5b0 2024-01-16T15:00:05.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.315+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb104054cc0 con 0x7fb11c15b7f0 2024-01-16T15:00:05.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.318+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0778076af0 con 0x7f07880c6d80 2024-01-16T15:00:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.617+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fbb880057d0 con 0x7fbbc0134180 2024-01-16T15:00:05.600 INFO:teuthology.orchestra.run.smithi053.stdout:60129542150 2024-01-16T15:00:05.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.619+0000 7fbbb7e00640 1 -- 172.21.15.53:0/2692631404 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbba8055890 con 0x7fbbc0134180 2024-01-16T15:00:05.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.622+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806b2f0 msgr2=0x7fbb9806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.622+0000 7fbbb5600640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806b2f0 0x7fbb9806d7b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0004f90 tx=0x7fbbb000a070 comp rx=0 tx=0).stop 2024-01-16T15:00:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 msgr2=0x7fbbc0132590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 0x7fbbc0132590 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbba80108c0 tx=0x7fbba8010d80 comp rx=0 tx=0).stop 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 shutdown_connections 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806b2f0 0x7fbb9806d7b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc0132ad0 0x7fbbc00711e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0134180 0x7fbbc0132590 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 --2- 172.21.15.53:0/2692631404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00b3eb0 0x7fbbc0155a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.623+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 >> 172.21.15.53:0/2692631404 conn(0x7fbbc00aea00 msgr2=0x7fbbc01562b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.624+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 shutdown_connections 2024-01-16T15:00:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.624+0000 7fbbb5600640 1 -- 172.21.15.53:0/2692631404 wait complete. 2024-01-16T15:00:05.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.625+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fa2c4005500 con 0x7fa2f40b15f0 2024-01-16T15:00:05.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.625+0000 7fa2ebe00640 1 -- 172.21.15.53:0/53986302 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa2ec054750 con 0x7fa2f40b15f0 2024-01-16T15:00:05.606 INFO:teuthology.orchestra.run.smithi053.stdout:60129542148 2024-01-16T15:00:05.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.629+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2cc06b390 msgr2=0x7fa2cc06d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.629+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2cc06b390 0x7fa2cc06d850 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4005880 tx=0x7fa2e4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.629+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 msgr2=0x7fa2f4160000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.629+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 0x7fa2f4160000 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c790 tx=0x7fa2ec00cc50 comp rx=0 tx=0).stop 2024-01-16T15:00:05.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.630+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 shutdown_connections 2024-01-16T15:00:05.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.630+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2cc06b390 0x7fa2cc06d850 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.630+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa2f40cc7f0 0x7fa2f4160540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.630+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa2f40b15f0 0x7fa2f4160000 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00c790 tx=0x7fa2ec00cc50 comp rx=0 tx=0).stop 2024-01-16T15:00:05.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.630+0000 7fa2f9a00640 1 --2- 172.21.15.53:0/53986302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa2f4072290 0x7fa2f4202fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.630+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 >> 172.21.15.53:0/53986302 conn(0x7fa2f40b0b00 msgr2=0x7fa2f4153250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.631+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 shutdown_connections 2024-01-16T15:00:05.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.631+0000 7fa2f9a00640 1 -- 172.21.15.53:0/53986302 wait complete. 2024-01-16T15:00:05.632 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542150 for osd.1 2024-01-16T15:00:05.636 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.2 2024-01-16T15:00:05.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.685+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f07880d4140 con 0x7f07880c6d80 2024-01-16T15:00:05.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.686+0000 7f077ea00640 1 -- 172.21.15.53:0/3867368255 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f077807a110 con 0x7f07880c6d80 2024-01-16T15:00:05.667 INFO:teuthology.orchestra.run.smithi053.stdout:60129542148 2024-01-16T15:00:05.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f076006b340 msgr2=0x7f076006d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f076006b340 0x7f076006d800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0770006fd0 tx=0x7f0770008040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 msgr2=0x7f078815f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 0x7f078815f800 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f077802b9e0 tx=0x7f07780046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 shutdown_connections 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f076006b340 0x7f076006d800 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0788164a50 0x7f07880ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07880b6820 0x7f07880ce390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 --2- 172.21.15.53:0/3867368255 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07880c6d80 0x7f078815f800 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 >> 172.21.15.53:0/3867368255 conn(0x7f078808cc50 msgr2=0x7f07880c8760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 shutdown_connections 2024-01-16T15:00:05.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.689+0000 7f078da00640 1 -- 172.21.15.53:0/3867368255 wait complete. 2024-01-16T15:00:05.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.692+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fb11c155980 con 0x7fb11c15b7f0 2024-01-16T15:00:05.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.693+0000 7fb112a00640 1 -- 172.21.15.53:0/305991540 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb11c155980 con 0x7fb11c15b7f0 2024-01-16T15:00:05.673 INFO:teuthology.orchestra.run.smithi053.stdout:64424509444 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb0f406b310 msgr2=0x7fb0f406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb0f406b310 0x7fb0f406d7d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb114036000 tx=0x7fb11403d040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 msgr2=0x7fb11c144830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c144830 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb10400e980 tx=0x7fb10400ee40 comp rx=0 tx=0).stop 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 shutdown_connections 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb0f406b310 0x7fb0f406d7d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb11c15bbd0 0x7fb11c1e4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb11c15b7f0 0x7fb11c144830 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 --2- 172.21.15.53:0/305991540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb11c0d0900 0x7fb11c156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 >> 172.21.15.53:0/305991540 conn(0x7fb11c099a60 msgr2=0x7fb11c0ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.695+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 shutdown_connections 2024-01-16T15:00:05.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.696+0000 7fb11be00640 1 -- 172.21.15.53:0/305991540 wait complete. 2024-01-16T15:00:05.686 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.7 2024-01-16T15:00:05.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.711+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fa7cc155870 con 0x7fa7cc15fe00 2024-01-16T15:00:05.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.712+0000 7fa7c2000640 1 -- 172.21.15.53:0/1743759211 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7b4058370 con 0x7fa7cc15fe00 2024-01-16T15:00:05.693 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509444 for osd.6 2024-01-16T15:00:05.693 INFO:teuthology.orchestra.run.smithi053.stdout:60129542148 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7a406b390 msgr2=0x7fa7a406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7a406b390 0x7fa7a406d850 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000fd0 tx=0x7fa7bc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc15fe00 msgr2=0x7fa7cc160280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc15fe00 0x7fa7cc160280 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa7cc08baf0 tx=0x7fa7b4009f90 comp rx=0 tx=0).stop 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 shutdown_connections 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7a406b390 0x7fa7a406d850 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa7cc15fe00 0x7fa7cc160280 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa7cc151000 0x7fa7cc0b1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 --2- 172.21.15.53:0/1743759211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa7ac0047c0 0x7fa7cc0b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.714+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 >> 172.21.15.53:0/1743759211 conn(0x7fa7cc09c540 msgr2=0x7fa7cc15b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.715+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 shutdown_connections 2024-01-16T15:00:05.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.715+0000 7fa7d1000640 1 -- 172.21.15.53:0/1743759211 wait complete. 2024-01-16T15:00:05.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.718+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fde340675d0 con 0x7fde341599b0 2024-01-16T15:00:05.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.719+0000 7fde31600640 1 -- 172.21.15.53:0/4185735645 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde24018020 con 0x7fde341599b0 2024-01-16T15:00:05.699 INFO:teuthology.orchestra.run.smithi053.stdout:60129542149 2024-01-16T15:00:05.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.721+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde0c06b2c0 msgr2=0x7fde0c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.721+0000 7fde3c000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde0c06b2c0 0x7fde0c06d780 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fde2c036000 tx=0x7fde2c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:05.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.721+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 msgr2=0x7fde341f64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.721+0000 7fde3c000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 0x7fde341f64a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fde2400c450 tx=0x7fde2400c910 comp rx=0 tx=0).stop 2024-01-16T15:00:05.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.721+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 shutdown_connections 2024-01-16T15:00:05.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde0c06b2c0 0x7fde0c06d780 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde341599b0 0x7fde341f64a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde340b1ee0 0x7fde34159470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 --2- 172.21.15.53:0/4185735645 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde3406e380 0x7fde34160690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 >> 172.21.15.53:0/4185735645 conn(0x7fde340b0be0 msgr2=0x7fde340ad740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 shutdown_connections 2024-01-16T15:00:05.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.722+0000 7fde3c000640 1 -- 172.21.15.53:0/4185735645 wait complete. 2024-01-16T15:00:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.725+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f3ae00675d0 con 0x7f3ae0143ae0 2024-01-16T15:00:05.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.725+0000 7f3ad6000640 1 -- 172.21.15.53:0/3246712605 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3ad0018090 con 0x7f3ae0143ae0 2024-01-16T15:00:05.706 INFO:teuthology.orchestra.run.smithi053.stdout:60129542149 2024-01-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ab806b390 msgr2=0x7f3ab806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ab806b390 0x7f3ab806d850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009aa0 tx=0x7f3ad8003a50 comp rx=0 tx=0).stop 2024-01-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 msgr2=0x7f3ae013e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 0x7f3ae013e6d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000d850 tx=0x7f3ad000dd10 comp rx=0 tx=0).stop 2024-01-16T15:00:05.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 shutdown_connections 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ab806b390 0x7f3ab806d850 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae015e1e0 0x7f3ae0140cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae0143ae0 0x7f3ae013e6d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 --2- 172.21.15.53:0/3246712605 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae00cf7d0 0x7f3ae00ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 >> 172.21.15.53:0/3246712605 conn(0x7f3ae006a800 msgr2=0x7f3ae015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 shutdown_connections 2024-01-16T15:00:05.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.729+0000 7f3ae5000640 1 -- 172.21.15.53:0/3246712605 wait complete. 2024-01-16T15:00:05.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542148 for osd.0 2024-01-16T15:00:05.718 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.5 2024-01-16T15:00:05.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.739+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4220161fd0 con 0x7f42200ae5b0 2024-01-16T15:00:05.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.740+0000 7f4217400640 1 -- 172.21.15.53:0/439077397 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f421807a450 con 0x7f42200ae5b0 2024-01-16T15:00:05.720 INFO:teuthology.orchestra.run.smithi053.stdout:64424509446 2024-01-16T15:00:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41f806b310 msgr2=0x7f41f806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41f806b310 0x7f41f806d7d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4210009440 tx=0x7f4210009040 comp rx=0 tx=0).stop 2024-01-16T15:00:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 msgr2=0x7f4220144de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 0x7f4220144de0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4218009890 tx=0x7f42180046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 shutdown_connections 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41f806b310 0x7f41f806d7d0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f422014b2c0 0x7f42200a8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42200cf340 0x7f422015ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 --2- 172.21.15.53:0/439077397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42200ae5b0 0x7f4220144de0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.742+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 >> 172.21.15.53:0/439077397 conn(0x7f422006a800 msgr2=0x7f42200ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.743+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 shutdown_connections 2024-01-16T15:00:05.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:05.743+0000 7f4226200640 1 -- 172.21.15.53:0/439077397 wait complete. 2024-01-16T15:00:05.730 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.3 2024-01-16T15:00:05.738 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509446 for osd.4 2024-01-16T15:00:06.633 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-01-16T15:00:06.637 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-01-16T15:00:06.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-01-16T15:00:06.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-01-16T15:00:06.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-01-16T15:00:06.719 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-01-16T15:00:06.731 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-01-16T15:00:06.739 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-01-16T15:00:06.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.931+0000 7f4e7f200640 1 Processor -- start 2024-01-16T15:00:06.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.931+0000 7f4e7f200640 1 -- start start 2024-01-16T15:00:06.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.934+0000 7f4e7f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e7815c2e0 0x7f4e780adca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.934+0000 7f4e7f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e780b2170 con 0x7f4e780d0900 2024-01-16T15:00:06.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.934+0000 7f4e7f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e780b2450 con 0x7f4e7815b7f0 2024-01-16T15:00:06.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.934+0000 7f4e7f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e780b2730 con 0x7f4e7815c2e0 2024-01-16T15:00:06.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.935+0000 7f4e7e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e7815c2e0 0x7f4e780adca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.935+0000 7f4e7e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e7815c2e0 0x7f4e780adca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47828/0 (socket says 172.21.15.53:47828) 2024-01-16T15:00:06.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.935+0000 7f4e7e800640 1 -- 172.21.15.53:0/3336556773 learned_addr learned my addr 172.21.15.53:0/3336556773 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:06.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.937+0000 7f4e7ca00640 1 -- 172.21.15.53:0/3336556773 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283145829 0 0) 0x7f4e780b2170 con 0x7f4e780d0900 2024-01-16T15:00:06.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.937+0000 7f4e7ca00640 1 -- 172.21.15.53:0/3336556773 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e58003520 con 0x7f4e780d0900 2024-01-16T15:00:06.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.937+0000 7f4e7e800640 1 -- 172.21.15.53:0/3336556773 >> v1:172.21.15.146:6789/0 conn(0x7f4e7815b7f0 legacy=0x7f4e780d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.937+0000 7f4e7e800640 1 -- 172.21.15.53:0/3336556773 >> v1:172.21.15.53:6789/0 conn(0x7f4e780d0900 legacy=0x7f4e7815bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.937+0000 7f4e7e800640 1 -- 172.21.15.53:0/3336556773 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e780b08d0 con 0x7f4e7815c2e0 2024-01-16T15:00:06.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.940+0000 7f4e7e800640 1 --2- 172.21.15.53:0/3336556773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e780adca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009e50 tx=0x7f4e7002bf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8edba55031181734 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.941+0000 7f4e7ca00640 1 -- 172.21.15.53:0/3336556773 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7002cb10 con 0x7f4e7815c2e0 2024-01-16T15:00:06.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.941+0000 7f4e7ca00640 1 -- 172.21.15.53:0/3336556773 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7002cc70 con 0x7f4e7815c2e0 2024-01-16T15:00:06.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.941+0000 7f4e7ca00640 1 -- 172.21.15.53:0/3336556773 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e700343f0 con 0x7f4e7815c2e0 2024-01-16T15:00:06.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.942+0000 7f4e7f200640 1 -- 172.21.15.53:0/3336556773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 msgr2=0x7f4e780adca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.942+0000 7f4e7f200640 1 --2- 172.21.15.53:0/3336556773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e780adca0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e70009e50 tx=0x7f4e7002bf70 comp rx=0 tx=0).stop 2024-01-16T15:00:06.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.943+0000 7f4e7f200640 1 -- 172.21.15.53:0/3336556773 shutdown_connections 2024-01-16T15:00:06.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.943+0000 7f4e7f200640 1 --2- 172.21.15.53:0/3336556773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e780adca0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.943+0000 7f4e7f200640 1 -- 172.21.15.53:0/3336556773 >> 172.21.15.53:0/3336556773 conn(0x7f4e78099a60 msgr2=0x7f4e78099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.943+0000 7f4e7f200640 1 -- 172.21.15.53:0/3336556773 shutdown_connections 2024-01-16T15:00:06.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.945+0000 7f4e7f200640 1 -- 172.21.15.53:0/3336556773 wait complete. 2024-01-16T15:00:06.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.948+0000 7f4e7f200640 1 Processor -- start 2024-01-16T15:00:06.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.949+0000 7f4e7f200640 1 -- start start 2024-01-16T15:00:06.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.951+0000 7f4e7f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780d0900 0x7f4e781594c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.951+0000 7f4e7f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 0x7f4e780b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.951+0000 7f4e7f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e7813f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.952+0000 7f4e7f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78156630 con 0x7f4e7815b7f0 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.952+0000 7f4e7d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 0x7f4e780b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.952+0000 7f4e7d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 0x7f4e780b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35060/0 (socket says 172.21.15.53:35060) 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.952+0000 7f4e7d400640 1 -- 172.21.15.53:0/2205176572 learned_addr learned my addr 172.21.15.53:0/2205176572 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.952+0000 7f4e7e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e7813f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.952+0000 7f4e7f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e78156020 con 0x7f4e7815c2e0 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.953+0000 7f4e7f200640 1 -- 172.21.15.53:0/2205176572 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e780b5b60 con 0x7f4e780d0900 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.953+0000 7f4e7e800640 1 -- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780d0900 msgr2=0x7f4e781594c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.953+0000 7f4e7e800640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780d0900 0x7f4e781594c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.953+0000 7f4e7e800640 1 -- 172.21.15.53:0/2205176572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 msgr2=0x7f4e780b4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.954+0000 7f4e7e800640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 0x7f4e780b4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.954+0000 7f4e7e800640 1 -- 172.21.15.53:0/2205176572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e70009b30 con 0x7f4e7815c2e0 2024-01-16T15:00:06.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.954+0000 7f4e7d400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 0x7f4e780b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:06.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.955+0000 7f4e7e800640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e7813f6b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e70032000 tx=0x7f4e70004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.955+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70007920 con 0x7f4e7815c2e0 2024-01-16T15:00:06.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.956+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e70007ac0 con 0x7f4e7815c2e0 2024-01-16T15:00:06.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.956+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e70007e70 con 0x7f4e7815c2e0 2024-01-16T15:00:06.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.956+0000 7f4e7f200640 1 -- 172.21.15.53:0/2205176572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e780b55a0 con 0x7f4e7815c2e0 2024-01-16T15:00:06.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.956+0000 7f4e7f200640 1 -- 172.21.15.53:0/2205176572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e78156250 con 0x7f4e7815c2e0 2024-01-16T15:00:06.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.959+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e40005250 con 0x7f4e7815c2e0 2024-01-16T15:00:06.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.962+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4e70049070 con 0x7f4e7815c2e0 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.964+0000 7f4e76000640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.964+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e700b41b0 con 0x7f4e7815c2e0 2024-01-16T15:00:06.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.965+0000 7f4e7de00640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.967+0000 7f4e7de00640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e60000f60 tx=0x7f4e600079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.974+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e7007f1d0 con 0x7f4e7815c2e0 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.983+0000 7f0a14400640 1 Processor -- start 2024-01-16T15:00:06.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.983+0000 7f0a14400640 1 -- start start 2024-01-16T15:00:06.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.986+0000 7f0a14400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c0b3eb0 0x7f0a0c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.987+0000 7f0a14400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c0b4870 con 0x7f0a0c15a320 2024-01-16T15:00:06.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.987+0000 7f0a14400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0c06cdc0 con 0x7f0a0c0cc7b0 2024-01-16T15:00:06.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.987+0000 7f0a11000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c0b3eb0 0x7f0a0c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.987+0000 7f0a14400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c15a700 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.987+0000 7f0a12400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0a0c0cc7b0 0x7f0a0c0ccc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38450/0 (socket says 172.21.15.53:38450) 2024-01-16T15:00:06.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.987+0000 7f0a12400640 1 -- 172.21.15.53:0/1553085210 learned_addr learned my addr 172.21.15.53:0/1553085210 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:06.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.989+0000 7f0a0be00640 1 -- 172.21.15.53:0/1553085210 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199230654 0 0) 0x7f0a0c06cdc0 con 0x7f0a0c0cc7b0 2024-01-16T15:00:06.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.989+0000 7f0a0be00640 1 -- 172.21.15.53:0/1553085210 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09ec003520 con 0x7f0a0c0cc7b0 2024-01-16T15:00:06.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.989+0000 7f0a11000640 1 -- 172.21.15.53:0/1553085210 >> v1:172.21.15.146:6789/0 conn(0x7f0a0c0cc7b0 legacy=0x7f0a0c0ccc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.989+0000 7f0a11000640 1 -- 172.21.15.53:0/1553085210 >> v1:172.21.15.53:6789/0 conn(0x7f0a0c15a320 legacy=0x7f0a0c06c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:06.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.989+0000 7f0a11000640 1 -- 172.21.15.53:0/1553085210 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0c06d0a0 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.991+0000 7f0a11000640 1 --2- 172.21.15.53:0/1553085210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c0b3eb0 0x7f0a0c0b4330 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09f40097d0 tx=0x7f09f402b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d9b09ed84e6193b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.992+0000 7f0a0be00640 1 -- 172.21.15.53:0/1553085210 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4031800 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.992+0000 7f0a0be00640 1 -- 172.21.15.53:0/1553085210 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f4031960 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.992+0000 7f0a0be00640 1 -- 172.21.15.53:0/1553085210 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f4031cf0 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.993+0000 7f0a14400640 1 -- 172.21.15.53:0/1553085210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c0b3eb0 msgr2=0x7f0a0c0b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.993+0000 7f0a14400640 1 --2- 172.21.15.53:0/1553085210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c0b3eb0 0x7f0a0c0b4330 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f09f40097d0 tx=0x7f09f402b8a0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.994+0000 7f0a14400640 1 -- 172.21.15.53:0/1553085210 shutdown_connections 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.994+0000 7f0a14400640 1 --2- 172.21.15.53:0/1553085210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c0b3eb0 0x7f0a0c0b4330 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.994+0000 7f0a14400640 1 -- 172.21.15.53:0/1553085210 >> 172.21.15.53:0/1553085210 conn(0x7f0a0c0aea00 msgr2=0x7f0a0c0aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:06.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.994+0000 7f0a14400640 1 -- 172.21.15.53:0/1553085210 shutdown_connections 2024-01-16T15:00:06.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.994+0000 7f0a14400640 1 -- 172.21.15.53:0/1553085210 wait complete. 2024-01-16T15:00:06.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.997+0000 7f0a14400640 1 Processor -- start 2024-01-16T15:00:06.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:06.997+0000 7f0a14400640 1 -- start start 2024-01-16T15:00:06.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.000+0000 7f0a14400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 0x7f0a0c152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.000+0000 7f0a11a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 0x7f0a0c152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.001+0000 7f0a11a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 0x7f0a0c152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35062/0 (socket says 172.21.15.53:35062) 2024-01-16T15:00:06.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.001+0000 7f0a11a00640 1 -- 172.21.15.53:0/286823298 learned_addr learned my addr 172.21.15.53:0/286823298 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:06.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.001+0000 7f0a14400640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a0c0cc7b0 0x7f0a0c0b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.001+0000 7f0a14400640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c15a320 0x7f0a0c1ff730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.002+0000 7f0a14400640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c1ffc70 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.002+0000 7f0a11000640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a0c0cc7b0 0x7f0a0c0b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.002+0000 7f0a14400640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c0ae490 con 0x7f0a0c0cc7b0 2024-01-16T15:00:06.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.002+0000 7f0a12400640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c15a320 0x7f0a0c1ff730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:06.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.002+0000 7f0a14400640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0c14fff0 con 0x7f0a0c15a320 2024-01-16T15:00:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.004+0000 7f0a11a00640 1 -- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c15a320 msgr2=0x7f0a0c1ff730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.004+0000 7f0a11a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c15a320 0x7f0a0c1ff730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.004+0000 7f0a11a00640 1 -- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a0c0cc7b0 msgr2=0x7f0a0c0b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.004+0000 7f0a11a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a0c0cc7b0 0x7f0a0c0b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:06.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.004+0000 7f0a11a00640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4009400 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.005+0000 7f0a11a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 0x7f0a0c152230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00b780 tx=0x7f09fc00bc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:06.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.006+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00c720 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.006+0000 7f0a14400640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0c150b70 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.006+0000 7f0a14400640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0c150520 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.008+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc00c880 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.009+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00c720 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.012+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09dc005250 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.014+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09fc00c9f0 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.016+0000 7f0a09600640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09e406b350 0x7f09e406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:06.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.016+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f09fc088e40 con 0x7f0a0c0b3eb0 2024-01-16T15:00:06.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.017+0000 7f0a11000640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09e406b350 0x7f09e406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.022+0000 7f0a11000640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09e406b350 0x7f09e406d810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036000 tx=0x7f09f403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.027+0000 7f848f200640 1 Processor -- start 2024-01-16T15:00:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.027+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09fc053cd0 con 0x7f0a0c0b3eb0 2024-01-16T15:00:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.027+0000 7f848f200640 1 -- start start 2024-01-16T15:00:07.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.029+0000 7f848f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f84880e3130 0x7f84880e7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.029+0000 7f848f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8488007b40 con 0x7f8488053c80 2024-01-16T15:00:07.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.029+0000 7f848f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84880587e0 con 0x7f848803c070 2024-01-16T15:00:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.029+0000 7f848f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8488058ac0 con 0x7f84880e3130 2024-01-16T15:00:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.030+0000 7f848e800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8488053c80 0x7f84880540d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49810/0 (socket says 172.21.15.53:49810) 2024-01-16T15:00:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.030+0000 7f848e800640 1 -- 172.21.15.53:0/2531178016 learned_addr learned my addr 172.21.15.53:0/2531178016 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:07.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.030+0000 7f848de00640 1 --2- 172.21.15.53:0/2531178016 >> v2:172.21.15.146:3300/0 conn(0x7f84880e3130 0x7f84880e7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.031+0000 7f848ca00640 1 -- 172.21.15.53:0/2531178016 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587015196 0 0) 0x7f8488007b40 con 0x7f8488053c80 2024-01-16T15:00:07.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.031+0000 7f848ca00640 1 -- 172.21.15.53:0/2531178016 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8470003520 con 0x7f8488053c80 2024-01-16T15:00:07.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.032+0000 7f848de00640 1 -- 172.21.15.53:0/2531178016 >> v1:172.21.15.146:6789/0 conn(0x7f848803c070 legacy=0x7f8488058020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.032+0000 7f848de00640 1 -- 172.21.15.53:0/2531178016 >> v1:172.21.15.53:6789/0 conn(0x7f8488053c80 legacy=0x7f84880540d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.032+0000 7f848de00640 1 -- 172.21.15.53:0/2531178016 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8488006330 con 0x7f84880e3130 2024-01-16T15:00:07.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.036+0000 7f848de00640 1 --2- 172.21.15.53:0/2531178016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880e7750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f84800097a0 tx=0x7f848002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8677983c117b0ad1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.037+0000 7f848ca00640 1 -- 172.21.15.53:0/2531178016 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8480030860 con 0x7f84880e3130 2024-01-16T15:00:07.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.037+0000 7f848ca00640 1 -- 172.21.15.53:0/2531178016 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84800309c0 con 0x7f84880e3130 2024-01-16T15:00:07.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.037+0000 7f848ca00640 1 -- 172.21.15.53:0/2531178016 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8480030d90 con 0x7f84880e3130 2024-01-16T15:00:07.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.038+0000 7f848f200640 1 -- 172.21.15.53:0/2531178016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 msgr2=0x7f84880e7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.038+0000 7f848f200640 1 --2- 172.21.15.53:0/2531178016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880e7750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f84800097a0 tx=0x7f848002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.039+0000 7f848f200640 1 -- 172.21.15.53:0/2531178016 shutdown_connections 2024-01-16T15:00:07.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.039+0000 7f848f200640 1 --2- 172.21.15.53:0/2531178016 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880e7750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.039+0000 7f848f200640 1 -- 172.21.15.53:0/2531178016 >> 172.21.15.53:0/2531178016 conn(0x7f8488058eb0 msgr2=0x7f8488059290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.039+0000 7f848f200640 1 -- 172.21.15.53:0/2531178016 shutdown_connections 2024-01-16T15:00:07.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.039+0000 7f3492c00640 1 Processor -- start 2024-01-16T15:00:07.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.040+0000 7f848f200640 1 -- 172.21.15.53:0/2531178016 wait complete. 2024-01-16T15:00:07.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.040+0000 7f3492c00640 1 -- start start 2024-01-16T15:00:07.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.040+0000 7facfc000640 1 Processor -- start 2024-01-16T15:00:07.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.040+0000 7facfc000640 1 -- start start 2024-01-16T15:00:07.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.042+0000 7facfc000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf40c6d80 0x7facf40d1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7f848f200640 1 Processor -- start 2024-01-16T15:00:07.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.042+0000 7facfc000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40cdd40 con 0x7facf40d1e60 2024-01-16T15:00:07.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.042+0000 7facfc000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facf40ce000 con 0x7facf40d1540 2024-01-16T15:00:07.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.042+0000 7facfc000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf40d0ab0 con 0x7facf40c6d80 2024-01-16T15:00:07.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7f3492c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f348c0d0100 0x7f348c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7f848f200640 1 -- start start 2024-01-16T15:00:07.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7facf9600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7facf40d1540 0x7facf40cfa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38468/0 (socket says 172.21.15.53:38468) 2024-01-16T15:00:07.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7facf9600640 1 -- 172.21.15.53:0/2514591482 learned_addr learned my addr 172.21.15.53:0/2514591482 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:07.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7facf8c00640 1 --2- 172.21.15.53:0/2514591482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf40c6d80 0x7facf40d1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7facf8c00640 1 -- 172.21.15.53:0/2514591482 >> v1:172.21.15.146:6789/0 conn(0x7facf40d1540 legacy=0x7facf40cfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7facf8c00640 1 -- 172.21.15.53:0/2514591482 >> v1:172.21.15.53:6789/0 conn(0x7facf40d1e60 legacy=0x7facf40d2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f848f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f848803c070 0x7f84880d46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f848f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8488053c80 0x7f84880dcff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7f3492c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f348c0d0f40 0x7f348c0d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7f3492200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f348c0d0f40 0x7f348c0d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7f3492200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f348c0d0f40 0x7f348c0d1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35078/0 (socket says 172.21.15.53:35078) 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7f3492200640 1 -- 172.21.15.53:0/1488621918 learned_addr learned my addr 172.21.15.53:0/1488621918 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.044+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348c0d0a10 con 0x7f348c160920 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348c0cf080 con 0x7f348c0d0f40 2024-01-16T15:00:07.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348c0c6be0 con 0x7f348c0d0100 2024-01-16T15:00:07.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f3490e00640 1 --2- 172.21.15.53:0/1488621918 >> v2:172.21.15.146:3300/0 conn(0x7f348c0d0100 0x7f348c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f3492200640 1 -- 172.21.15.53:0/1488621918 >> v2:172.21.15.146:3300/0 conn(0x7f348c0d0100 msgr2=0x7f348c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.043+0000 7f3491800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f348c160920 0x7f348c0cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:38484/0 (socket says 172.21.15.53:38484) 2024-01-16T15:00:07.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f3492200640 1 --2- 172.21.15.53:0/1488621918 >> v2:172.21.15.146:3300/0 conn(0x7f348c0d0100 0x7f348c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7f3492200640 1 -- 172.21.15.53:0/1488621918 >> v1:172.21.15.146:6789/0 conn(0x7f348c160920 legacy=0x7f348c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f3492200640 1 -- 172.21.15.53:0/1488621918 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348c0ce7e0 con 0x7f348c0d0f40 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880d39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f848803c070 0x7f84880d46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f848803c070 0x7f84880d46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49666/0 (socket says 172.21.15.53:49666) 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848de00640 1 -- 172.21.15.53:0/3820444356 learned_addr learned my addr 172.21.15.53:0/3820444356 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848804ff80 con 0x7f8488053c80 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84880dd820 con 0x7f84880e3130 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84880ddb00 con 0x7f848803c070 2024-01-16T15:00:07.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.047+0000 7f848e800640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880d39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.045+0000 7facf8c00640 1 -- 172.21.15.53:0/2514591482 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf40d0210 con 0x7facf40c6d80 2024-01-16T15:00:07.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.046+0000 7f848d400640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8488053c80 0x7f84880dcff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f3492200640 1 --2- 172.21.15.53:0/1488621918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c0d0f40 0x7f348c0d1390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f348c08c1f0 tx=0x7f3474034390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a187880d0bd972e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f848d400640 1 -- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f848803c070 msgr2=0x7f84880d46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f848d400640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f848803c070 0x7f84880d46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f848d400640 1 -- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 msgr2=0x7f84880d39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f848d400640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880d39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f848d400640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480009400 con 0x7f8488053c80 2024-01-16T15:00:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7facf8c00640 1 --2- 172.21.15.53:0/2514591482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf40c6d80 0x7facf40d1920 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7face40097d0 tx=0x7face402b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29106ec5c76fc302 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.048+0000 7f348be00640 1 -- 172.21.15.53:0/1488621918 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347403b600 con 0x7f348c0d0f40 2024-01-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.049+0000 7f348be00640 1 -- 172.21.15.53:0/1488621918 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3474037050 con 0x7f348c0d0f40 2024-01-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.049+0000 7f348be00640 1 -- 172.21.15.53:0/1488621918 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347403b980 con 0x7f348c0d0f40 2024-01-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.049+0000 7facf3e00640 1 -- 172.21.15.53:0/2514591482 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face4030800 con 0x7facf40c6d80 2024-01-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.049+0000 7facf3e00640 1 -- 172.21.15.53:0/2514591482 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face4030960 con 0x7facf40c6d80 2024-01-16T15:00:07.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.049+0000 7facf3e00640 1 -- 172.21.15.53:0/2514591482 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face4030cf0 con 0x7facf40c6d80 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c0d0f40 msgr2=0x7f348c0d1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f3492c00640 1 --2- 172.21.15.53:0/1488621918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c0d0f40 0x7f348c0d1390 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f348c08c1f0 tx=0x7f3474034390 comp rx=0 tx=0).stop 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 shutdown_connections 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f3492c00640 1 --2- 172.21.15.53:0/1488621918 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c0d0f40 0x7f348c0d1390 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f3492c00640 1 --2- 172.21.15.53:0/1488621918 >> v2:172.21.15.146:3300/0 conn(0x7f348c0d0100 0x7f348c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 >> 172.21.15.53:0/1488621918 conn(0x7f348c0d3090 msgr2=0x7f348c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f848d400640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8488053c80 0x7f84880dcff0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f847800c390 tx=0x7f847800c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7facfc000640 1 -- 172.21.15.53:0/2514591482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf40c6d80 msgr2=0x7facf40d1920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7facfc000640 1 --2- 172.21.15.53:0/2514591482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf40c6d80 0x7facf40d1920 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7face40097d0 tx=0x7face402b940 comp rx=0 tx=0).stop 2024-01-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.050+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84780052d0 con 0x7f8488053c80 2024-01-16T15:00:07.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84880ddde0 con 0x7f8488053c80 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7facfc000640 1 -- 172.21.15.53:0/2514591482 shutdown_connections 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7facfc000640 1 --2- 172.21.15.53:0/2514591482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf40c6d80 0x7facf40d1920 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7facfc000640 1 -- 172.21.15.53:0/2514591482 >> 172.21.15.53:0/2514591482 conn(0x7facf408cc50 msgr2=0x7facf409c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7facfc000640 1 -- 172.21.15.53:0/2514591482 shutdown_connections 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7facfc000640 1 -- 172.21.15.53:0/2514591482 wait complete. 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478005430 con 0x7f8488053c80 2024-01-16T15:00:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84780052d0 con 0x7f8488053c80 2024-01-16T15:00:07.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.051+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84880dd530 con 0x7f8488053c80 2024-01-16T15:00:07.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.053+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 shutdown_connections 2024-01-16T15:00:07.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.053+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8488050410 con 0x7f8488053c80 2024-01-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.053+0000 7f3492c00640 1 -- 172.21.15.53:0/1488621918 wait complete. 2024-01-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7f3492c00640 1 Processor -- start 2024-01-16T15:00:07.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7f3492c00640 1 -- start start 2024-01-16T15:00:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.055+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f84780035f0 con 0x7f8488053c80 2024-01-16T15:00:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7f8486000640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f846806b350 0x7f846806d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8478088c20 con 0x7f8488053c80 2024-01-16T15:00:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.058+0000 7f2e9a800640 1 Processor -- start 2024-01-16T15:00:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.058+0000 7f848de00640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f846806b350 0x7f846806d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.058+0000 7f2e9a800640 1 -- start start 2024-01-16T15:00:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7f3492c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f348c0d0100 0x7f348c1f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.054+0000 7facfc000640 1 Processor -- start 2024-01-16T15:00:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.054+0000 7facfc000640 1 -- start start 2024-01-16T15:00:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7facfc000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf40c6d80 0x7facf41fb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7facfc000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40d1540 0x7facf41f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7facfc000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 0x7facf40c81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7facfc000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf4158600 con 0x7facf40d1540 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7facfc000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40ca300 con 0x7facf40c6d80 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.056+0000 7facfc000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facf41fbb10 con 0x7facf41f7570 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 0x7facf40c81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 0x7facf40c81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49676/0 (socket says 172.21.15.53:49676) 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 -- 172.21.15.53:0/1457454481 learned_addr learned my addr 172.21.15.53:0/1457454481 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facf9600640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf40c6d80 0x7facf41fb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 -- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf40c6d80 msgr2=0x7facf41fb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf40c6d80 0x7facf41fb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 -- 172.21.15.53:0/1457454481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40d1540 msgr2=0x7facf41f49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.057+0000 7facfa000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40d1540 0x7facf41f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.058+0000 7facfa000640 1 -- 172.21.15.53:0/1457454481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face4009400 con 0x7facf41f7570 2024-01-16T15:00:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7facfa000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 0x7facf40c81b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c3c0 tx=0x7facdc00c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7f3492c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f348c0d0f40 0x7f348c159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7f3492c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 0x7f348c159bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc0052d0 con 0x7facf41f7570 2024-01-16T15:00:07.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf41f7b00 con 0x7facf41f7570 2024-01-16T15:00:07.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc005450 con 0x7facf41f7570 2024-01-16T15:00:07.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.059+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf41f80c0 con 0x7facf41f7570 2024-01-16T15:00:07.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facdc003830 con 0x7facf41f7570 2024-01-16T15:00:07.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3491800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f348c0d0100 0x7f348c1f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3490e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f348c0d0f40 0x7f348c159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3491800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f348c0d0100 0x7f348c1f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47896/0 (socket says 172.21.15.53:47896) 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3490e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f348c0d0f40 0x7f348c159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49690/0 (socket says 172.21.15.53:49690) 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3492200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 0x7f348c159bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3492200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 0x7f348c159bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35098/0 (socket says 172.21.15.53:35098) 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3491800640 1 -- 172.21.15.53:0/3624396375 learned_addr learned my addr 172.21.15.53:0/3624396375 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c159360 con 0x7f348c160920 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348c1f6c50 con 0x7f348c0d0100 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.060+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348c158ac0 con 0x7f348c0d0f40 2024-01-16T15:00:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.061+0000 7f2e9a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e94153950 0x7f2e940cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.061+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facc4005250 con 0x7facf41f7570 2024-01-16T15:00:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e9a800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e94153d30 con 0x7f2e9415ce90 2024-01-16T15:00:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f3492200640 1 -- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f348c0d0f40 msgr2=0x7f348c159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f3492200640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f348c0d0f40 0x7f348c159690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e9a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e940b2b70 con 0x7f2e9406d480 2024-01-16T15:00:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f3492200640 1 -- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f348c0d0100 msgr2=0x7f348c1f6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e9a800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e940b2e50 con 0x7f2e94153950 2024-01-16T15:00:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f3492200640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f348c0d0100 0x7f348c1f6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e93400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e94153950 0x7f2e940cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e93400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2e94153950 0x7f2e940cb0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35112/0 (socket says 172.21.15.53:35112) 2024-01-16T15:00:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e93400640 1 -- 172.21.15.53:0/3614082625 learned_addr learned my addr 172.21.15.53:0/3614082625 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f3492200640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474013a10 con 0x7f348c160920 2024-01-16T15:00:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.063+0000 7f3492200640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 0x7f348c159bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f347403b000 tx=0x7f34740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.064+0000 7f2e93400640 1 -- 172.21.15.53:0/3614082625 >> v1:172.21.15.146:6789/0 conn(0x7f2e9406d480 legacy=0x7f2e94159fb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:00:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.064+0000 7f2e93400640 1 -- 172.21.15.53:0/3614082625 >> v1:172.21.15.53:6789/0 conn(0x7f2e9415ce90 legacy=0x7f2e941598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.062+0000 7f2e92a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2e9415ce90 0x7f2e941598a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49818/0 (socket says 172.21.15.53:49818) 2024-01-16T15:00:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.063+0000 7f848de00640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f846806b350 0x7f846806d810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f848002b9e0 tx=0x7f848003b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.064+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7facdc003990 con 0x7facf41f7570 2024-01-16T15:00:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.064+0000 7f2e93400640 1 -- 172.21.15.53:0/3614082625 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e940b3130 con 0x7f2e94153950 2024-01-16T15:00:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.066+0000 7facf1600640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faccc06b2c0 0x7faccc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.066+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7facdc088fc0 con 0x7facf41f7570 2024-01-16T15:00:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.066+0000 7f2e93400640 1 --2- 172.21.15.53:0/3614082625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 0x7f2e940cb0d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009870 tx=0x7f2e7c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d06ecee0f3ffee0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.067+0000 7facf9600640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faccc06b2c0 0x7faccc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7facf9600640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faccc06b2c0 0x7faccc06d780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7facec005c00 tx=0x7facec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f2e92000640 1 -- 172.21.15.53:0/3614082625 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c030860 con 0x7f2e94153950 2024-01-16T15:00:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f2e92000640 1 -- 172.21.15.53:0/3614082625 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c0309c0 con 0x7f2e94153950 2024-01-16T15:00:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f2e92000640 1 -- 172.21.15.53:0/3614082625 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c030d50 con 0x7f2e94153950 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474045460 con 0x7f348c160920 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347400c050 con 0x7f348c160920 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3474045800 con 0x7f348c160920 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348c158da0 con 0x7f348c160920 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.068+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f348c0ca290 con 0x7f348c160920 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.069+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8488050410 con 0x7f8488053c80 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.070+0000 7f2e9a800640 1 -- 172.21.15.53:0/3614082625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 msgr2=0x7f2e940cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.070+0000 7f2e9a800640 1 --2- 172.21.15.53:0/3614082625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 0x7f2e940cb0d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009870 tx=0x7f2e7c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.070+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f348c15c110 con 0x7f348c160920 2024-01-16T15:00:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.072+0000 7f2e9a800640 1 -- 172.21.15.53:0/3614082625 shutdown_connections 2024-01-16T15:00:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.072+0000 7f2e9a800640 1 --2- 172.21.15.53:0/3614082625 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 0x7f2e940cb0d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.072+0000 7f2e9a800640 1 -- 172.21.15.53:0/3614082625 >> 172.21.15.53:0/3614082625 conn(0x7f2e940b0be0 msgr2=0x7f2e940b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.073+0000 7f2e9a800640 1 -- 172.21.15.53:0/3614082625 shutdown_connections 2024-01-16T15:00:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.073+0000 7f2e9a800640 1 -- 172.21.15.53:0/3614082625 wait complete. 2024-01-16T15:00:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.072+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3474045960 con 0x7f348c160920 2024-01-16T15:00:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.075+0000 7f3489600640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f346406b390 0x7f346406d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.075+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34740ac620 con 0x7f348c160920 2024-01-16T15:00:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.075+0000 7f3491800640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f346406b390 0x7f346406d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.076+0000 7f2e9a800640 1 Processor -- start 2024-01-16T15:00:07.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.077+0000 7f3491800640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f346406b390 0x7f346406d850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f348400e250 tx=0x7f348400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.077+0000 7f2e9a800640 1 -- start start 2024-01-16T15:00:07.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.078+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facdc053ea0 con 0x7facf41f7570 2024-01-16T15:00:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e9a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 0x7f2e940b1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e9a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 0x7f2e94141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e9a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e9415ce90 0x7f2e94141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e9a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94151e40 con 0x7f2e94153950 2024-01-16T15:00:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e9a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94152120 con 0x7f2e9415ce90 2024-01-16T15:00:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e9a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e94152400 con 0x7f2e9406d480 2024-01-16T15:00:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e93400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 0x7f2e940b1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e93400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 0x7f2e940b1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49696/0 (socket says 172.21.15.53:49696) 2024-01-16T15:00:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.080+0000 7f2e93400640 1 -- 172.21.15.53:0/2176172300 learned_addr learned my addr 172.21.15.53:0/2176172300 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.082+0000 7f2e93400640 1 -- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e9415ce90 msgr2=0x7f2e94141e90 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:00:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.082+0000 7f2e93400640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e9415ce90 0x7f2e94141e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.082+0000 7f2e93400640 1 -- 172.21.15.53:0/2176172300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 msgr2=0x7f2e94141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.082+0000 7f2e93400640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 0x7f2e94141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.082+0000 7f2e93400640 1 -- 172.21.15.53:0/2176172300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7c0094d0 con 0x7f2e9406d480 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.082+0000 7f2e93400640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 0x7f2e940b1230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009870 tx=0x7f2e7c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.083+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c039e20 con 0x7f2e9406d480 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.083+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7c043400 con 0x7f2e9406d480 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.083+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e7c0437b0 con 0x7f2e9406d480 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.083+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e941574c0 con 0x7f2e9406d480 2024-01-16T15:00:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.083+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3474077500 con 0x7f348c160920 2024-01-16T15:00:07.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.085+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e940b1770 con 0x7f2e9406d480 2024-01-16T15:00:07.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.086+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e941579e0 con 0x7f2e9406d480 2024-01-16T15:00:07.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.088+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2e7c043910 con 0x7f2e9406d480 2024-01-16T15:00:07.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.092+0000 7f2e8b400640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e6c06b310 0x7f2e6c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.092+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e7c03e070 con 0x7f2e9406d480 2024-01-16T15:00:07.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.093+0000 7f2e92a00640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e6c06b310 0x7f2e6c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.094+0000 7f2e92a00640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e6c06b310 0x7f2e6c06d7d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000fb0 tx=0x7f2e84007a00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.101+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e7c0781a0 con 0x7f2e9406d480 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.107+0000 7fd2a9a00640 1 Processor -- start 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.107+0000 7fd2a9a00640 1 -- start start 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.109+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.109+0000 7fd2a9a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2a4074520 0x7fd2a40c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.109+0000 7fd2a9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2a40cdc50 con 0x7fd2a406d190 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.109+0000 7fd2a9a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a40cdf10 con 0x7fd2a4074520 2024-01-16T15:00:07.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.109+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a406dac0 con 0x7fd2a4161bf0 2024-01-16T15:00:07.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.110+0000 7fd2a2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2a4074520 0x7fd2a40c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.110+0000 7fd2a2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2a4074520 0x7fd2a40c21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47906/0 (socket says 172.21.15.53:47906) 2024-01-16T15:00:07.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.110+0000 7fd2a2000640 1 -- 172.21.15.53:0/3256569009 learned_addr learned my addr 172.21.15.53:0/3256569009 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.110+0000 7fd2a2a00640 1 --2- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.111+0000 7fd2a2000640 1 -- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 msgr2=0x7fd2a40ac050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.111+0000 7fd2a2000640 1 --2- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40ac050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.111+0000 7fd2a2000640 1 -- 172.21.15.53:0/3256569009 >> v1:172.21.15.146:6789/0 conn(0x7fd2a406d190 legacy=0x7fd2a40ce360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.111+0000 7fd2a2000640 1 -- 172.21.15.53:0/3256569009 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a406dda0 con 0x7fd2a4074520 2024-01-16T15:00:07.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.113+0000 7fd2a2000640 1 --2- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a40c21f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd294009e50 tx=0x7fd29402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d58636bce6d1f525 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.113+0000 7fd2a1600640 1 -- 172.21.15.53:0/3256569009 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29402cb20 con 0x7fd2a4074520 2024-01-16T15:00:07.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.113+0000 7fd2a1600640 1 -- 172.21.15.53:0/3256569009 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29402cc80 con 0x7fd2a4074520 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.114+0000 7fd2a1600640 1 -- 172.21.15.53:0/3256569009 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294034610 con 0x7fd2a4074520 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.115+0000 7fd2a9a00640 1 -- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 msgr2=0x7fd2a40c21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.115+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a40c21f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd294009e50 tx=0x7fd29402bf80 comp rx=0 tx=0).stop 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.116+0000 7fd2a9a00640 1 -- 172.21.15.53:0/3256569009 shutdown_connections 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.116+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a40c21f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.116+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/3256569009 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40ac050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.116+0000 7fd2a9a00640 1 -- 172.21.15.53:0/3256569009 >> 172.21.15.53:0/3256569009 conn(0x7fd2a40b0d70 msgr2=0x7fd2a40b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.117+0000 7fd2a9a00640 1 -- 172.21.15.53:0/3256569009 shutdown_connections 2024-01-16T15:00:07.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.117+0000 7fd2a9a00640 1 -- 172.21.15.53:0/3256569009 wait complete. 2024-01-16T15:00:07.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.117+0000 7f9b50000640 1 Processor -- start 2024-01-16T15:00:07.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.119+0000 7fd2a9a00640 1 Processor -- start 2024-01-16T15:00:07.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.120+0000 7fd2a9a00640 1 -- start start 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.119+0000 7f9b50000640 1 -- start start 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.121+0000 7f9b50000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9b4814b2c0 0x7f9b480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.121+0000 7f9b50000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9b48071070 0x7f9b480cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.121+0000 7f9b50000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b480cfbf0 con 0x7f9b480cf7d0 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.121+0000 7f9b50000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b4814b6a0 con 0x7f9b4814b2c0 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.121+0000 7f9b50000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b48071690 con 0x7f9b48071070 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.122+0000 7f9b4d600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9b4814b2c0 0x7f9b480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.122+0000 7f9b4cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9b480cf7d0 0x7f9b48070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49834/0 (socket says 172.21.15.53:49834) 2024-01-16T15:00:07.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.122+0000 7f9b4cc00640 1 -- 172.21.15.53:0/3323378462 learned_addr learned my addr 172.21.15.53:0/3323378462 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:07.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.122+0000 7f9b4e000640 1 --2- 172.21.15.53:0/3323378462 >> v2:172.21.15.146:3300/0 conn(0x7f9b48071070 0x7f9b480cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.123+0000 7f9b4e000640 1 -- 172.21.15.53:0/3323378462 >> v2:172.21.15.53:3300/0 conn(0x7f9b4814b2c0 msgr2=0x7f9b480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.123+0000 7f9b4e000640 1 --2- 172.21.15.53:0/3323378462 >> v2:172.21.15.53:3300/0 conn(0x7f9b4814b2c0 0x7f9b480cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.123+0000 7f9b4e000640 1 -- 172.21.15.53:0/3323378462 >> v1:172.21.15.53:6789/0 conn(0x7f9b480cf7d0 legacy=0x7f9b48070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.123+0000 7f9b4e000640 1 -- 172.21.15.53:0/3323378462 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b48071970 con 0x7f9b48071070 2024-01-16T15:00:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.126+0000 7f9b4e000640 1 --2- 172.21.15.53:0/3323378462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b48071070 0x7f9b480cc160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b48165330 tx=0x7f9b4002bd00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10398e6d248544cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.126+0000 7f9b47e00640 1 -- 172.21.15.53:0/3323378462 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4002ca30 con 0x7f9b48071070 2024-01-16T15:00:07.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.126+0000 7f9b47e00640 1 -- 172.21.15.53:0/3323378462 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4002cb90 con 0x7f9b48071070 2024-01-16T15:00:07.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.126+0000 7f9b47e00640 1 -- 172.21.15.53:0/3323378462 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40034440 con 0x7f9b48071070 2024-01-16T15:00:07.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.126+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 0x7fd2a40cc590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.127+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a415d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.127+0000 7fd2a9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 -- 172.21.15.53:0/3323378462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b48071070 msgr2=0x7f9b480cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.127+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a415d7f0 con 0x7fd2a406d190 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 --2- 172.21.15.53:0/3323378462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b48071070 0x7f9b480cc160 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9b48165330 tx=0x7f9b4002bd00 comp rx=0 tx=0).stop 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 -- 172.21.15.53:0/3323378462 shutdown_connections 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 --2- 172.21.15.53:0/3323378462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b48071070 0x7f9b480cc160 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 --2- 172.21.15.53:0/3323378462 >> v2:172.21.15.53:3300/0 conn(0x7f9b4814b2c0 0x7f9b480cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 -- 172.21.15.53:0/3323378462 >> 172.21.15.53:0/3323378462 conn(0x7f9b4806a800 msgr2=0x7f9b4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 -- 172.21.15.53:0/3323378462 shutdown_connections 2024-01-16T15:00:07.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.128+0000 7f9b50000640 1 -- 172.21.15.53:0/3323378462 wait complete. 2024-01-16T15:00:07.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.127+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40c9810 con 0x7fd2a4074520 2024-01-16T15:00:07.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.127+0000 7fd2a9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a40c4170 con 0x7fd2a4161bf0 2024-01-16T15:00:07.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.129+0000 7fd2a2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 0x7fd2a40cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.129+0000 7fd2a2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a415d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.129+0000 7fd2a2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 0x7fd2a40cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35134/0 (socket says 172.21.15.53:35134) 2024-01-16T15:00:07.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.129+0000 7fd2a2a00640 1 -- 172.21.15.53:0/4212772856 learned_addr learned my addr 172.21.15.53:0/4212772856 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.130+0000 7fd2a2a00640 1 -- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 msgr2=0x7fd2a40c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.130+0000 7fd2a2a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.130+0000 7fd2a2a00640 1 -- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 msgr2=0x7fd2a415d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.130+0000 7fd2a2a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a415d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.130+0000 7fd2a2a00640 1 -- 172.21.15.53:0/4212772856 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009b30 con 0x7fd2a406d190 2024-01-16T15:00:07.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.131+0000 7fd2a2000640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a415d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:07.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.132+0000 7fd2a2a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 0x7fd2a40cc590 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00d840 tx=0x7fd28c00dd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.132+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00b6c0 con 0x7fd2a406d190 2024-01-16T15:00:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.132+0000 7f9b50000640 1 Processor -- start 2024-01-16T15:00:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.132+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c00b820 con 0x7fd2a406d190 2024-01-16T15:00:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.132+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00bbd0 con 0x7fd2a406d190 2024-01-16T15:00:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.132+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a40c9a60 con 0x7fd2a406d190 2024-01-16T15:00:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.133+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a40ca020 con 0x7fd2a406d190 2024-01-16T15:00:07.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.133+0000 7f9b50000640 1 -- start start 2024-01-16T15:00:07.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.134+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a40675d0 con 0x7fd2a406d190 2024-01-16T15:00:07.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.135+0000 7f9b50000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 0x7f9b4814a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.135+0000 7f9b50000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b480cf7d0 0x7f9b4815e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b50000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4814b2c0 0x7f9b48161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b50000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48147480 con 0x7f9b48071070 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b50000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48164bf0 con 0x7f9b4814b2c0 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b50000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b48164350 con 0x7f9b480cf7d0 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b4d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 0x7f9b4814a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b4d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 0x7f9b4814a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35146/0 (socket says 172.21.15.53:35146) 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b4d600640 1 -- 172.21.15.53:0/2719373021 learned_addr learned my addr 172.21.15.53:0/2719373021 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b4e000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4814b2c0 0x7f9b48161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.136+0000 7f9b4cc00640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b480cf7d0 0x7f9b4815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.137+0000 7f9b4e000640 1 -- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b480cf7d0 msgr2=0x7f9b4815e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.137+0000 7f9b4e000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b480cf7d0 0x7f9b4815e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.137+0000 7f9b4e000640 1 -- 172.21.15.53:0/2719373021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 msgr2=0x7f9b4814a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.141+0000 7f9b4e000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 0x7f9b4814a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.141+0000 7f9b4e000640 1 -- 172.21.15.53:0/2719373021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b40009a70 con 0x7f9b4814b2c0 2024-01-16T15:00:07.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.141+0000 7f9b4d600640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 0x7f9b4814a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.142+0000 7f9b4e000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4814b2c0 0x7f9b48161e70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.142+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4003e610 con 0x7f9b4814b2c0 2024-01-16T15:00:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.142+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b48164630 con 0x7f9b4814b2c0 2024-01-16T15:00:07.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.143+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b481476b0 con 0x7f9b4814b2c0 2024-01-16T15:00:07.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.138+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd28c00bd30 con 0x7fd2a406d190 2024-01-16T15:00:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.143+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4003e770 con 0x7f9b4814b2c0 2024-01-16T15:00:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.143+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4003e610 con 0x7f9b4814b2c0 2024-01-16T15:00:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.139+0000 7fd29aa00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd27c06b310 0x7fd27c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.141+0000 7fd2a2000640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd27c06b310 0x7fd27c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.145+0000 7fd2a2000640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd27c06b310 0x7fd27c06d7d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd294035000 tx=0x7fd294030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.146+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b18005250 con 0x7f9b4814b2c0 2024-01-16T15:00:07.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.148+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9b4003b050 con 0x7f9b4814b2c0 2024-01-16T15:00:07.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.148+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd28c059e10 con 0x7fd2a406d190 2024-01-16T15:00:07.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.148+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd28c055a50 con 0x7fd2a406d190 2024-01-16T15:00:07.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.149+0000 7f9b45600640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9b2006b320 0x7f9b2006d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.149+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9b400ac130 con 0x7f9b4814b2c0 2024-01-16T15:00:07.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.150+0000 7f9b4d600640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9b2006b320 0x7f9b2006d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.152+0000 7f9b4d600640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9b2006b320 0x7f9b2006d7e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9b38007e50 tx=0x7f9b38008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.163+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9b400770d0 con 0x7f9b4814b2c0 2024-01-16T15:00:07.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.383+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f4e400057d0 con 0x7f4e7815c2e0 2024-01-16T15:00:07.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.384+0000 7f4e76000640 1 -- 172.21.15.53:0/2205176572 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e70009510 con 0x7f4e7815c2e0 2024-01-16T15:00:07.366 INFO:teuthology.orchestra.run.smithi053.stdout:60129542152 2024-01-16T15:00:07.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.388+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 msgr2=0x7f4e5006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.388+0000 7f4e6f400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e60000f60 tx=0x7f4e600079b0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.388+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 msgr2=0x7f4e7813f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.388+0000 7f4e6f400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e7813f6b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e70032000 tx=0x7f4e70004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:07.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.389+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 shutdown_connections 2024-01-16T15:00:07.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.389+0000 7f4e6f400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e5006b2c0 0x7f4e5006d780 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.389+0000 7f4e6f400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e7815c2e0 0x7f4e7813f6b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.389+0000 7f4e6f400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e7815b7f0 0x7f4e780b4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.389+0000 7f4e6f400640 1 --2- 172.21.15.53:0/2205176572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e780d0900 0x7f4e781594c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.389+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 >> 172.21.15.53:0/2205176572 conn(0x7f4e78099a60 msgr2=0x7f4e7815ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.390+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 shutdown_connections 2024-01-16T15:00:07.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.390+0000 7f4e6f400640 1 -- 172.21.15.53:0/2205176572 wait complete. 2024-01-16T15:00:07.395 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542152 for osd.1 2024-01-16T15:00:07.395 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.450+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f348c067450 con 0x7f348c160920 2024-01-16T15:00:07.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.451+0000 7f3489600640 1 -- 172.21.15.53:0/3624396375 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f347407ab20 con 0x7f348c160920 2024-01-16T15:00:07.431 INFO:teuthology.orchestra.run.smithi053.stdout:64424509445 2024-01-16T15:00:07.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.453+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f346406b390 msgr2=0x7f346406d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.453+0000 7f3492c00640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f346406b390 0x7f346406d850 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f348400e250 tx=0x7f348400b040 comp rx=0 tx=0).stop 2024-01-16T15:00:07.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.453+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 msgr2=0x7f348c159bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.453+0000 7f3492c00640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 0x7f348c159bd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f347403b000 tx=0x7f34740046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 shutdown_connections 2024-01-16T15:00:07.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f346406b390 0x7f346406d850 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f348c160920 0x7f348c159bd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f348c0d0f40 0x7f348c159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 --2- 172.21.15.53:0/3624396375 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f348c0d0100 0x7f348c1f6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 >> 172.21.15.53:0/3624396375 conn(0x7f348c0d3090 msgr2=0x7f348c15b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 shutdown_connections 2024-01-16T15:00:07.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.454+0000 7f3492c00640 1 -- 172.21.15.53:0/3624396375 wait complete. 2024-01-16T15:00:07.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.455+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f09dc005500 con 0x7f0a0c0b3eb0 2024-01-16T15:00:07.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.455+0000 7f0a09600640 1 -- 172.21.15.53:0/286823298 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f09fc019090 con 0x7f0a0c0b3eb0 2024-01-16T15:00:07.436 INFO:teuthology.orchestra.run.smithi053.stdout:60129542150 2024-01-16T15:00:07.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09e406b350 msgr2=0x7f09e406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09e406b350 0x7f09e406d810 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036000 tx=0x7f09f403c040 comp rx=0 tx=0).stop 2024-01-16T15:00:07.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 msgr2=0x7f0a0c152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 0x7f0a0c152230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00b780 tx=0x7f09fc00bc40 comp rx=0 tx=0).stop 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 shutdown_connections 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09e406b350 0x7f09e406d810 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a0c15a320 0x7f0a0c1ff730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a0c0cc7b0 0x7f0a0c0b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 --2- 172.21.15.53:0/286823298 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a0c0b3eb0 0x7f0a0c152230 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.459+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 >> 172.21.15.53:0/286823298 conn(0x7f0a0c0aea00 msgr2=0x7f0a0c0afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.460+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 shutdown_connections 2024-01-16T15:00:07.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.460+0000 7f0a02a00640 1 -- 172.21.15.53:0/286823298 wait complete. 2024-01-16T15:00:07.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.466+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f8488050410 con 0x7f8488053c80 2024-01-16T15:00:07.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.466+0000 7f8486000640 1 -- 172.21.15.53:0/3820444356 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8488050410 con 0x7f8488053c80 2024-01-16T15:00:07.447 INFO:teuthology.orchestra.run.smithi053.stdout:60129542150 2024-01-16T15:00:07.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.469+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f846806b350 msgr2=0x7f846806d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f846806b350 0x7f846806d810 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f848002b9e0 tx=0x7f848003b040 comp rx=0 tx=0).stop 2024-01-16T15:00:07.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8488053c80 msgr2=0x7f84880dcff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8488053c80 0x7f84880dcff0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f847800c390 tx=0x7f847800c790 comp rx=0 tx=0).stop 2024-01-16T15:00:07.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 shutdown_connections 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f846806b350 0x7f846806d810 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84880e3130 0x7f84880d39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8488053c80 0x7f84880dcff0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 --2- 172.21.15.53:0/3820444356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f848803c070 0x7f84880d46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.470+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 >> 172.21.15.53:0/3820444356 conn(0x7f8488058eb0 msgr2=0x7f8488039b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.471+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 shutdown_connections 2024-01-16T15:00:07.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.471+0000 7f848f200640 1 -- 172.21.15.53:0/3820444356 wait complete. 2024-01-16T15:00:07.457 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509445 got 64424509445 for osd.6 2024-01-16T15:00:07.457 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.466 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.2 2024-01-16T15:00:07.466 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.468 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.7 2024-01-16T15:00:07.469 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.498+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7facc40057d0 con 0x7facf41f7570 2024-01-16T15:00:07.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.499+0000 7facf1600640 1 -- 172.21.15.53:0/1457454481 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7facdc017090 con 0x7facf41f7570 2024-01-16T15:00:07.479 INFO:teuthology.orchestra.run.smithi053.stdout:60129542150 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faccc06b2c0 msgr2=0x7faccc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faccc06b2c0 0x7faccc06d780 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7facec005c00 tx=0x7facec009040 comp rx=0 tx=0).stop 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 msgr2=0x7facf40c81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 0x7facf40c81b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7facdc00c3c0 tx=0x7facdc00c880 comp rx=0 tx=0).stop 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 shutdown_connections 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faccc06b2c0 0x7faccc06d780 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facf41f7570 0x7facf40c81b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facf40d1540 0x7facf41f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 --2- 172.21.15.53:0/1457454481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facf40c6d80 0x7facf41fb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 >> 172.21.15.53:0/1457454481 conn(0x7facf408cc50 msgr2=0x7facf4160a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.501+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 shutdown_connections 2024-01-16T15:00:07.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.502+0000 7facfc000640 1 -- 172.21.15.53:0/1457454481 wait complete. 2024-01-16T15:00:07.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.515+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f2e941579e0 con 0x7f2e9406d480 2024-01-16T15:00:07.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.515+0000 7f2e8b400640 1 -- 172.21.15.53:0/2176172300 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2e941579e0 con 0x7f2e9406d480 2024-01-16T15:00:07.496 INFO:teuthology.orchestra.run.smithi053.stdout:60129542151 2024-01-16T15:00:07.497 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542149 got 60129542150 for osd.0 2024-01-16T15:00:07.498 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e6c06b310 msgr2=0x7f2e6c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e6c06b310 0x7f2e6c06d7d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2e84000fb0 tx=0x7f2e84007a00 comp rx=0 tx=0).stop 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 msgr2=0x7f2e940b1230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 0x7f2e940b1230 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e7c009870 tx=0x7f2e7c004730 comp rx=0 tx=0).stop 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 shutdown_connections 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e6c06b310 0x7f2e6c06d7d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e9415ce90 0x7f2e94141e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e94153950 0x7f2e94141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 --2- 172.21.15.53:0/2176172300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e9406d480 0x7f2e940b1230 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 >> 172.21.15.53:0/2176172300 conn(0x7f2e940b0be0 msgr2=0x7f2e9415bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 shutdown_connections 2024-01-16T15:00:07.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.518+0000 7f2e9a800640 1 -- 172.21.15.53:0/2176172300 wait complete. 2024-01-16T15:00:07.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.522+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd2a40b51a0 con 0x7fd2a406d190 2024-01-16T15:00:07.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.522+0000 7fd29aa00640 1 -- 172.21.15.53:0/4212772856 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd28c059070 con 0x7fd2a406d190 2024-01-16T15:00:07.503 INFO:teuthology.orchestra.run.smithi053.stdout:64424509447 2024-01-16T15:00:07.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd27c06b310 msgr2=0x7fd27c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd27c06b310 0x7fd27c06d7d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd294035000 tx=0x7fd294030000 comp rx=0 tx=0).stop 2024-01-16T15:00:07.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 msgr2=0x7fd2a40cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 0x7fd2a40cc590 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00d840 tx=0x7fd28c00dd00 comp rx=0 tx=0).stop 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 shutdown_connections 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd27c06b310 0x7fd27c06d7d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2a4161bf0 0x7fd2a40c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2a4074520 0x7fd2a415d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 --2- 172.21.15.53:0/4212772856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2a406d190 0x7fd2a40cc590 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 >> 172.21.15.53:0/4212772856 conn(0x7fd2a40b0d70 msgr2=0x7fd2a40cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 shutdown_connections 2024-01-16T15:00:07.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.525+0000 7fd2a9a00640 1 -- 172.21.15.53:0/4212772856 wait complete. 2024-01-16T15:00:07.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.5 2024-01-16T15:00:07.522 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.522 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.4 2024-01-16T15:00:07.522 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.554+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f9b18005500 con 0x7f9b4814b2c0 2024-01-16T15:00:07.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.555+0000 7f9b45600640 1 -- 172.21.15.53:0/2719373021 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9b4007a6f0 con 0x7f9b4814b2c0 2024-01-16T15:00:07.535 INFO:teuthology.orchestra.run.smithi053.stdout:60129542151 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9b2006b320 msgr2=0x7f9b2006d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9b2006b320 0x7f9b2006d7e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f9b38007e50 tx=0x7f9b38008040 comp rx=0 tx=0).stop 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4814b2c0 msgr2=0x7f9b48161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4814b2c0 0x7f9b48161e70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9b40033000 tx=0x7f9b40004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 shutdown_connections 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9b2006b320 0x7f9b2006d7e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9b4814b2c0 0x7f9b48161e70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9b480cf7d0 0x7f9b4815e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 --2- 172.21.15.53:0/2719373021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9b48071070 0x7f9b4814a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 >> 172.21.15.53:0/2719373021 conn(0x7f9b4806a800 msgr2=0x7f9b480aeb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 shutdown_connections 2024-01-16T15:00:07.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.557+0000 7f9b50000640 1 -- 172.21.15.53:0/2719373021 wait complete. 2024-01-16T15:00:07.554 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.3 2024-01-16T15:00:07.554 DEBUG:teuthology.parallel:result is None 2024-01-16T15:00:07.554 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T15:00:07.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.826+0000 7fe986c00640 1 Processor -- start 2024-01-16T15:00:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.827+0000 7fe986c00640 1 -- start start 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe986c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe9800cc500 0x7fe9800cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe986c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe98006d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe986c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe980070e50 con 0x7fe980143c40 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe986c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9800723c0 con 0x7fe9800cc500 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe986c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800cced0 con 0x7fe98006d160 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe984e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe9800cc500 0x7fe9800cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe984e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe9800cc500 0x7fe9800cc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35162/0 (socket says 172.21.15.53:35162) 2024-01-16T15:00:07.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.828+0000 7fe984e00640 1 -- 172.21.15.53:0/1677149493 learned_addr learned my addr 172.21.15.53:0/1677149493 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.829+0000 7fe986200640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe98006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.829+0000 7fe984e00640 1 -- 172.21.15.53:0/1677149493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 msgr2=0x7fe98006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.829+0000 7fe984e00640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe98006d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.829+0000 7fe984e00640 1 -- 172.21.15.53:0/1677149493 >> v1:172.21.15.146:6789/0 conn(0x7fe980143c40 legacy=0x7fe980071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.829+0000 7fe984e00640 1 -- 172.21.15.53:0/1677149493 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe980144020 con 0x7fe9800cc500 2024-01-16T15:00:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.829+0000 7fe986200640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe98006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.830+0000 7fe984e00640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800cc990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9800b44a0 tx=0x7fe96802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=343a35f462706f0c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.830+0000 7fe97fe00640 1 -- 172.21.15.53:0/1677149493 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe96802e070 con 0x7fe9800cc500 2024-01-16T15:00:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.830+0000 7fe97fe00640 1 -- 172.21.15.53:0/1677149493 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9680315a0 con 0x7fe9800cc500 2024-01-16T15:00:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.830+0000 7fe97fe00640 1 -- 172.21.15.53:0/1677149493 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968031930 con 0x7fe9800cc500 2024-01-16T15:00:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.830+0000 7fe986c00640 1 -- 172.21.15.53:0/1677149493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 msgr2=0x7fe9800cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.830+0000 7fe986c00640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800cc990 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe9800b44a0 tx=0x7fe96802b840 comp rx=0 tx=0).stop 2024-01-16T15:00:07.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.831+0000 7fe986c00640 1 -- 172.21.15.53:0/1677149493 shutdown_connections 2024-01-16T15:00:07.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.831+0000 7fe986c00640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe98006d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.831+0000 7fe986c00640 1 --2- 172.21.15.53:0/1677149493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800cc990 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.831+0000 7fe986c00640 1 -- 172.21.15.53:0/1677149493 >> 172.21.15.53:0/1677149493 conn(0x7fe98006a800 msgr2=0x7fe98006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:07.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.831+0000 7fe986c00640 1 -- 172.21.15.53:0/1677149493 shutdown_connections 2024-01-16T15:00:07.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.831+0000 7fe986c00640 1 -- 172.21.15.53:0/1677149493 wait complete. 2024-01-16T15:00:07.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.833+0000 7fe986c00640 1 Processor -- start 2024-01-16T15:00:07.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.833+0000 7fe986c00640 1 -- start start 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe986c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe9801f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe986c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800b2b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe986c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe980143c40 0x7fe9801f2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe986c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800b3060 con 0x7fe9800cc500 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe986c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800c4d20 con 0x7fe980143c40 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe986c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9800ac8d0 con 0x7fe98006d160 2024-01-16T15:00:07.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800b2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800b2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35168/0 (socket says 172.21.15.53:35168) 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 -- 172.21.15.53:0/3636595618 learned_addr learned my addr 172.21.15.53:0/3636595618 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 -- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 msgr2=0x7fe9801f5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe9801f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 -- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe980143c40 msgr2=0x7fe9801f2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe980143c40 0x7fe9801f2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.834+0000 7fe984e00640 1 -- 172.21.15.53:0/3636595618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9680094d0 con 0x7fe9800cc500 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.835+0000 7fe984e00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800b2b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe96802b880 tx=0x7fe9680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.835+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968032980 con 0x7fe9800cc500 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.835+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968032ae0 con 0x7fe9800cc500 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.835+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9800acbb0 con 0x7fe9800cc500 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.835+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe968032e90 con 0x7fe9800cc500 2024-01-16T15:00:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.835+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9800ad170 con 0x7fe9800cc500 2024-01-16T15:00:07.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.836+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9800b31d0 con 0x7fe9800cc500 2024-01-16T15:00:07.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.837+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe96804d3f0 con 0x7fe9800cc500 2024-01-16T15:00:07.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.838+0000 7fe97d600640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe95806b2c0 0x7fe95806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.839+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe9680ab850 con 0x7fe9800cc500 2024-01-16T15:00:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.839+0000 7fe985800640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe95806b2c0 0x7fe95806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:07.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.840+0000 7fe985800640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe95806b2c0 0x7fe95806d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe97000ab90 tx=0x7fe970009230 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:07.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:07.843+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9680766c0 con 0x7fe9800cc500 2024-01-16T15:00:08.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.024+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe98006c960 con 0x7fe95806b2c0 2024-01-16T15:00:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.048+0000 7fe97d600640 1 -- 172.21.15.53:0/3636595618 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264324 (secure 0 0 0) 0x7fe97000a830 con 0x7fe95806b2c0 2024-01-16T15:00:08.032 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:08.033 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe95806b2c0 msgr2=0x7fe95806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe95806b2c0 0x7fe95806d780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe97000ab90 tx=0x7fe970009230 comp rx=0 tx=0).stop 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 msgr2=0x7fe9800b2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800b2b20 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fe96802b880 tx=0x7fe9680046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 shutdown_connections 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe95806b2c0 0x7fe95806d780 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe980143c40 0x7fe9801f2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9800cc500 0x7fe9800b2b20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 --2- 172.21.15.53:0/3636595618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe98006d160 0x7fe9801f5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 >> 172.21.15.53:0/3636595618 conn(0x7fe98006a800 msgr2=0x7fe98006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 shutdown_connections 2024-01-16T15:00:08.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.053+0000 7fe986c00640 1 -- 172.21.15.53:0/3636595618 wait complete. 2024-01-16T15:00:08.051 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-01-16T15:00:06.172292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235776,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738944,"statfs":{"total":773094113280,"available":772852678656,"internally_reserved":0,"allocated":3022848,"data_stored":1810900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.296883"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882738+0000","last_change":"2024-01-16T14:59:57.913241+0000","last_active":"2024-01-16T14:59:58.882738+0000","last_peered":"2024-01-16T14:59:58.882738+0000","last_clean":"2024-01-16T14:59:58.882738+0000","last_became_active":"2024-01-16T14:59:57.912720+0000","last_became_peered":"2024-01-16T14:59:57.912720+0000","last_unstale":"2024-01-16T14:59:58.882738+0000","last_undegraded":"2024-01-16T14:59:58.882738+0000","last_fullsized":"2024-01-16T14:59:58.882738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964756+0000","last_change":"2024-01-16T14:59:57.910357+0000","last_active":"2024-01-16T14:59:58.964756+0000","last_peered":"2024-01-16T14:59:58.964756+0000","last_clean":"2024-01-16T14:59:58.964756+0000","last_became_active":"2024-01-16T14:59:57.909933+0000","last_became_peered":"2024-01-16T14:59:57.909933+0000","last_unstale":"2024-01-16T14:59:58.964756+0000","last_undegraded":"2024-01-16T14:59:58.964756+0000","last_fullsized":"2024-01-16T14:59:58.964756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974477+0000","last_change":"2024-01-16T14:59:57.909709+0000","last_active":"2024-01-16T14:59:58.974477+0000","last_peered":"2024-01-16T14:59:58.974477+0000","last_clean":"2024-01-16T14:59:58.974477+0000","last_became_active":"2024-01-16T14:59:57.909405+0000","last_became_peered":"2024-01-16T14:59:57.909405+0000","last_unstale":"2024-01-16T14:59:58.974477+0000","last_undegraded":"2024-01-16T14:59:58.974477+0000","last_fullsized":"2024-01-16T14:59:58.974477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:51.411241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975424+0000","last_change":"2024-01-16T14:59:57.920032+0000","last_active":"2024-01-16T14:59:58.975424+0000","last_peered":"2024-01-16T14:59:58.975424+0000","last_clean":"2024-01-16T14:59:58.975424+0000","last_became_active":"2024-01-16T14:59:57.919630+0000","last_became_peered":"2024-01-16T14:59:57.919630+0000","last_unstale":"2024-01-16T14:59:58.975424+0000","last_undegraded":"2024-01-16T14:59:58.975424+0000","last_fullsized":"2024-01-16T14:59:58.975424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882328+0000","last_change":"2024-01-16T14:59:57.917909+0000","last_active":"2024-01-16T14:59:58.882328+0000","last_peered":"2024-01-16T14:59:58.882328+0000","last_clean":"2024-01-16T14:59:58.882328+0000","last_became_active":"2024-01-16T14:59:57.914603+0000","last_became_peered":"2024-01-16T14:59:57.914603+0000","last_unstale":"2024-01-16T14:59:58.882328+0000","last_undegraded":"2024-01-16T14:59:58.882328+0000","last_fullsized":"2024-01-16T14:59:58.882328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976981+0000","last_change":"2024-01-16T14:59:57.909424+0000","last_active":"2024-01-16T14:59:58.976981+0000","last_peered":"2024-01-16T14:59:58.976981+0000","last_clean":"2024-01-16T14:59:58.976981+0000","last_became_active":"2024-01-16T14:59:57.909160+0000","last_became_peered":"2024-01-16T14:59:57.909160+0000","last_unstale":"2024-01-16T14:59:58.976981+0000","last_undegraded":"2024-01-16T14:59:58.976981+0000","last_fullsized":"2024-01-16T14:59:58.976981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880812+0000","last_change":"2024-01-16T14:59:57.917053+0000","last_active":"2024-01-16T14:59:58.880812+0000","last_peered":"2024-01-16T14:59:58.880812+0000","last_clean":"2024-01-16T14:59:58.880812+0000","last_became_active":"2024-01-16T14:59:57.914775+0000","last_became_peered":"2024-01-16T14:59:57.914775+0000","last_unstale":"2024-01-16T14:59:58.880812+0000","last_undegraded":"2024-01-16T14:59:58.880812+0000","last_fullsized":"2024-01-16T14:59:58.880812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976696+0000","last_change":"2024-01-16T14:59:57.909184+0000","last_active":"2024-01-16T14:59:58.976696+0000","last_peered":"2024-01-16T14:59:58.976696+0000","last_clean":"2024-01-16T14:59:58.976696+0000","last_became_active":"2024-01-16T14:59:57.908905+0000","last_became_peered":"2024-01-16T14:59:57.908905+0000","last_unstale":"2024-01-16T14:59:58.976696+0000","last_undegraded":"2024-01-16T14:59:58.976696+0000","last_fullsized":"2024-01-16T14:59:58.976696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883086+0000","last_change":"2024-01-16T14:59:57.921823+0000","last_active":"2024-01-16T14:59:58.883086+0000","last_peered":"2024-01-16T14:59:58.883086+0000","last_clean":"2024-01-16T14:59:58.883086+0000","last_became_active":"2024-01-16T14:59:57.921476+0000","last_became_peered":"2024-01-16T14:59:57.921476+0000","last_unstale":"2024-01-16T14:59:58.883086+0000","last_undegraded":"2024-01-16T14:59:58.883086+0000","last_fullsized":"2024-01-16T14:59:58.883086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975563+0000","last_change":"2024-01-16T14:59:57.909965+0000","last_active":"2024-01-16T14:59:58.975563+0000","last_peered":"2024-01-16T14:59:58.975563+0000","last_clean":"2024-01-16T14:59:58.975563+0000","last_became_active":"2024-01-16T14:59:57.909707+0000","last_became_peered":"2024-01-16T14:59:57.909707+0000","last_unstale":"2024-01-16T14:59:58.975563+0000","last_undegraded":"2024-01-16T14:59:58.975563+0000","last_fullsized":"2024-01-16T14:59:58.975563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975772+0000","last_change":"2024-01-16T14:59:57.908484+0000","last_active":"2024-01-16T14:59:58.975772+0000","last_peered":"2024-01-16T14:59:58.975772+0000","last_clean":"2024-01-16T14:59:58.975772+0000","last_became_active":"2024-01-16T14:59:57.908164+0000","last_became_peered":"2024-01-16T14:59:57.908164+0000","last_unstale":"2024-01-16T14:59:58.975772+0000","last_undegraded":"2024-01-16T14:59:58.975772+0000","last_fullsized":"2024-01-16T14:59:58.975772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885578+0000","last_change":"2024-01-16T14:59:57.914372+0000","last_active":"2024-01-16T14:59:58.885578+0000","last_peered":"2024-01-16T14:59:58.885578+0000","last_clean":"2024-01-16T14:59:58.885578+0000","last_became_active":"2024-01-16T14:59:57.913918+0000","last_became_peered":"2024-01-16T14:59:57.913918+0000","last_unstale":"2024-01-16T14:59:58.885578+0000","last_undegraded":"2024-01-16T14:59:58.885578+0000","last_fullsized":"2024-01-16T14:59:58.885578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:08.622882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977380+0000","last_change":"2024-01-16T14:59:57.907940+0000","last_active":"2024-01-16T14:59:58.977380+0000","last_peered":"2024-01-16T14:59:58.977380+0000","last_clean":"2024-01-16T14:59:58.977380+0000","last_became_active":"2024-01-16T14:59:57.906517+0000","last_became_peered":"2024-01-16T14:59:57.906517+0000","last_unstale":"2024-01-16T14:59:58.977380+0000","last_undegraded":"2024-01-16T14:59:58.977380+0000","last_fullsized":"2024-01-16T14:59:58.977380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974085+0000","last_change":"2024-01-16T14:59:57.907053+0000","last_active":"2024-01-16T14:59:58.974085+0000","last_peered":"2024-01-16T14:59:58.974085+0000","last_clean":"2024-01-16T14:59:58.974085+0000","last_became_active":"2024-01-16T14:59:57.906443+0000","last_became_peered":"2024-01-16T14:59:57.906443+0000","last_unstale":"2024-01-16T14:59:58.974085+0000","last_undegraded":"2024-01-16T14:59:58.974085+0000","last_fullsized":"2024-01-16T14:59:58.974085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:01.194359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964073+0000","last_change":"2024-01-16T14:59:57.893114+0000","last_active":"2024-01-16T14:59:58.964073+0000","last_peered":"2024-01-16T14:59:58.964073+0000","last_clean":"2024-01-16T14:59:58.964073+0000","last_became_active":"2024-01-16T14:59:57.891027+0000","last_became_peered":"2024-01-16T14:59:57.891027+0000","last_unstale":"2024-01-16T14:59:58.964073+0000","last_undegraded":"2024-01-16T14:59:58.964073+0000","last_fullsized":"2024-01-16T14:59:58.964073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974627+0000","last_change":"2024-01-16T14:59:57.911794+0000","last_active":"2024-01-16T14:59:58.974627+0000","last_peered":"2024-01-16T14:59:58.974627+0000","last_clean":"2024-01-16T14:59:58.974627+0000","last_became_active":"2024-01-16T14:59:57.911528+0000","last_became_peered":"2024-01-16T14:59:57.911528+0000","last_unstale":"2024-01-16T14:59:58.974627+0000","last_undegraded":"2024-01-16T14:59:58.974627+0000","last_fullsized":"2024-01-16T14:59:58.974627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976633+0000","last_change":"2024-01-16T14:59:57.917524+0000","last_active":"2024-01-16T14:59:58.976633+0000","last_peered":"2024-01-16T14:59:58.976633+0000","last_clean":"2024-01-16T14:59:58.976633+0000","last_became_active":"2024-01-16T14:59:57.916983+0000","last_became_peered":"2024-01-16T14:59:57.916983+0000","last_unstale":"2024-01-16T14:59:58.976633+0000","last_undegraded":"2024-01-16T14:59:58.976633+0000","last_fullsized":"2024-01-16T14:59:58.976633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974145+0000","last_change":"2024-01-16T14:59:57.914130+0000","last_active":"2024-01-16T14:59:58.974145+0000","last_peered":"2024-01-16T14:59:58.974145+0000","last_clean":"2024-01-16T14:59:58.974145+0000","last_became_active":"2024-01-16T14:59:57.913722+0000","last_became_peered":"2024-01-16T14:59:57.913722+0000","last_unstale":"2024-01-16T14:59:58.974145+0000","last_undegraded":"2024-01-16T14:59:58.974145+0000","last_fullsized":"2024-01-16T14:59:58.974145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:37.662076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882022+0000","last_change":"2024-01-16T14:59:57.914510+0000","last_active":"2024-01-16T14:59:58.882022+0000","last_peered":"2024-01-16T14:59:58.882022+0000","last_clean":"2024-01-16T14:59:58.882022+0000","last_became_active":"2024-01-16T14:59:57.912912+0000","last_became_peered":"2024-01-16T14:59:57.912912+0000","last_unstale":"2024-01-16T14:59:58.882022+0000","last_undegraded":"2024-01-16T14:59:58.882022+0000","last_fullsized":"2024-01-16T14:59:58.882022+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975057+0000","last_change":"2024-01-16T14:59:57.911759+0000","last_active":"2024-01-16T14:59:58.975057+0000","last_peered":"2024-01-16T14:59:58.975057+0000","last_clean":"2024-01-16T14:59:58.975057+0000","last_became_active":"2024-01-16T14:59:57.911474+0000","last_became_peered":"2024-01-16T14:59:57.911474+0000","last_unstale":"2024-01-16T14:59:58.975057+0000","last_undegraded":"2024-01-16T14:59:58.975057+0000","last_fullsized":"2024-01-16T14:59:58.975057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975453+0000","last_change":"2024-01-16T14:59:57.923664+0000","last_active":"2024-01-16T14:59:58.975453+0000","last_peered":"2024-01-16T14:59:58.975453+0000","last_clean":"2024-01-16T14:59:58.975453+0000","last_became_active":"2024-01-16T14:59:57.923402+0000","last_became_peered":"2024-01-16T14:59:57.923402+0000","last_unstale":"2024-01-16T14:59:58.975453+0000","last_undegraded":"2024-01-16T14:59:58.975453+0000","last_fullsized":"2024-01-16T14:59:58.975453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977267+0000","last_change":"2024-01-16T14:59:57.923807+0000","last_active":"2024-01-16T14:59:58.977267+0000","last_peered":"2024-01-16T14:59:58.977267+0000","last_clean":"2024-01-16T14:59:58.977267+0000","last_became_active":"2024-01-16T14:59:57.923493+0000","last_became_peered":"2024-01-16T14:59:57.923493+0000","last_unstale":"2024-01-16T14:59:58.977267+0000","last_undegraded":"2024-01-16T14:59:58.977267+0000","last_fullsized":"2024-01-16T14:59:58.977267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977086+0000","last_change":"2024-01-16T14:59:57.912156+0000","last_active":"2024-01-16T14:59:58.977086+0000","last_peered":"2024-01-16T14:59:58.977086+0000","last_clean":"2024-01-16T14:59:58.977086+0000","last_became_active":"2024-01-16T14:59:57.911497+0000","last_became_peered":"2024-01-16T14:59:57.911497+0000","last_unstale":"2024-01-16T14:59:58.977086+0000","last_undegraded":"2024-01-16T14:59:58.977086+0000","last_fullsized":"2024-01-16T14:59:58.977086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974975+0000","last_change":"2024-01-16T14:59:57.918668+0000","last_active":"2024-01-16T14:59:58.974975+0000","last_peered":"2024-01-16T14:59:58.974975+0000","last_clean":"2024-01-16T14:59:58.974975+0000","last_became_active":"2024-01-16T14:59:57.918314+0000","last_became_peered":"2024-01-16T14:59:57.918314+0000","last_unstale":"2024-01-16T14:59:58.974975+0000","last_undegraded":"2024-01-16T14:59:58.974975+0000","last_fullsized":"2024-01-16T14:59:58.974975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978927+0000","last_change":"2024-01-16T14:59:57.912774+0000","last_active":"2024-01-16T14:59:58.978927+0000","last_peered":"2024-01-16T14:59:58.978927+0000","last_clean":"2024-01-16T14:59:58.978927+0000","last_became_active":"2024-01-16T14:59:57.910934+0000","last_became_peered":"2024-01-16T14:59:57.910934+0000","last_unstale":"2024-01-16T14:59:58.978927+0000","last_undegraded":"2024-01-16T14:59:58.978927+0000","last_fullsized":"2024-01-16T14:59:58.978927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:47.822358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975717+0000","last_change":"2024-01-16T14:59:57.921118+0000","last_active":"2024-01-16T14:59:58.975717+0000","last_peered":"2024-01-16T14:59:58.975717+0000","last_clean":"2024-01-16T14:59:58.975717+0000","last_became_active":"2024-01-16T14:59:57.920835+0000","last_became_peered":"2024-01-16T14:59:57.920835+0000","last_unstale":"2024-01-16T14:59:58.975717+0000","last_undegraded":"2024-01-16T14:59:58.975717+0000","last_fullsized":"2024-01-16T14:59:58.975717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964405+0000","last_change":"2024-01-16T14:59:57.898548+0000","last_active":"2024-01-16T14:59:58.964405+0000","last_peered":"2024-01-16T14:59:58.964405+0000","last_clean":"2024-01-16T14:59:58.964405+0000","last_became_active":"2024-01-16T14:59:57.897874+0000","last_became_peered":"2024-01-16T14:59:57.897874+0000","last_unstale":"2024-01-16T14:59:58.964405+0000","last_undegraded":"2024-01-16T14:59:58.964405+0000","last_fullsized":"2024-01-16T14:59:58.964405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978167+0000","last_change":"2024-01-16T14:59:57.914061+0000","last_active":"2024-01-16T14:59:58.978167+0000","last_peered":"2024-01-16T14:59:58.978167+0000","last_clean":"2024-01-16T14:59:58.978167+0000","last_became_active":"2024-01-16T14:59:57.913718+0000","last_became_peered":"2024-01-16T14:59:57.913718+0000","last_unstale":"2024-01-16T14:59:58.978167+0000","last_undegraded":"2024-01-16T14:59:58.978167+0000","last_fullsized":"2024-01-16T14:59:58.978167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:46.106265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974760+0000","last_change":"2024-01-16T14:59:57.917278+0000","last_active":"2024-01-16T14:59:58.974760+0000","last_peered":"2024-01-16T14:59:58.974760+0000","last_clean":"2024-01-16T14:59:58.974760+0000","last_became_active":"2024-01-16T14:59:57.916768+0000","last_became_peered":"2024-01-16T14:59:57.916768+0000","last_unstale":"2024-01-16T14:59:58.974760+0000","last_undegraded":"2024-01-16T14:59:58.974760+0000","last_fullsized":"2024-01-16T14:59:58.974760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:12.721250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977355+0000","last_change":"2024-01-16T14:59:57.915403+0000","last_active":"2024-01-16T14:59:58.977355+0000","last_peered":"2024-01-16T14:59:58.977355+0000","last_clean":"2024-01-16T14:59:58.977355+0000","last_became_active":"2024-01-16T14:59:57.915134+0000","last_became_peered":"2024-01-16T14:59:57.915134+0000","last_unstale":"2024-01-16T14:59:58.977355+0000","last_undegraded":"2024-01-16T14:59:58.977355+0000","last_fullsized":"2024-01-16T14:59:58.977355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973385+0000","last_change":"2024-01-16T14:59:57.910767+0000","last_active":"2024-01-16T14:59:58.973385+0000","last_peered":"2024-01-16T14:59:58.973385+0000","last_clean":"2024-01-16T14:59:58.973385+0000","last_became_active":"2024-01-16T14:59:57.908877+0000","last_became_peered":"2024-01-16T14:59:57.908877+0000","last_unstale":"2024-01-16T14:59:58.973385+0000","last_undegraded":"2024-01-16T14:59:58.973385+0000","last_fullsized":"2024-01-16T14:59:58.973385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:58.524840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974275+0000","last_change":"2024-01-16T14:59:57.912035+0000","last_active":"2024-01-16T14:59:58.974275+0000","last_peered":"2024-01-16T14:59:58.974275+0000","last_clean":"2024-01-16T14:59:58.974275+0000","last_became_active":"2024-01-16T14:59:57.911641+0000","last_became_peered":"2024-01-16T14:59:57.911641+0000","last_unstale":"2024-01-16T14:59:58.974275+0000","last_undegraded":"2024-01-16T14:59:58.974275+0000","last_fullsized":"2024-01-16T14:59:58.974275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884477+0000","last_change":"2024-01-16T14:59:57.898975+0000","last_active":"2024-01-16T14:59:58.884477+0000","last_peered":"2024-01-16T14:59:58.884477+0000","last_clean":"2024-01-16T14:59:58.884477+0000","last_became_active":"2024-01-16T14:59:57.894960+0000","last_became_peered":"2024-01-16T14:59:57.894960+0000","last_unstale":"2024-01-16T14:59:58.884477+0000","last_undegraded":"2024-01-16T14:59:58.884477+0000","last_fullsized":"2024-01-16T14:59:58.884477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:50.235874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977646+0000","last_change":"2024-01-16T14:59:56.961128+0000","last_active":"2024-01-16T14:59:58.977646+0000","last_peered":"2024-01-16T14:59:58.977646+0000","last_clean":"2024-01-16T14:59:58.977646+0000","last_became_active":"2024-01-16T14:59:56.960829+0000","last_became_peered":"2024-01-16T14:59:56.960829+0000","last_unstale":"2024-01-16T14:59:58.977646+0000","last_undegraded":"2024-01-16T14:59:58.977646+0000","last_fullsized":"2024-01-16T14:59:58.977646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975040+0000","last_change":"2024-01-16T14:59:57.907318+0000","last_active":"2024-01-16T14:59:58.975040+0000","last_peered":"2024-01-16T14:59:58.975040+0000","last_clean":"2024-01-16T14:59:58.975040+0000","last_became_active":"2024-01-16T14:59:57.907025+0000","last_became_peered":"2024-01-16T14:59:57.907025+0000","last_unstale":"2024-01-16T14:59:58.975040+0000","last_undegraded":"2024-01-16T14:59:58.975040+0000","last_fullsized":"2024-01-16T14:59:58.975040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.937959+0000","last_change":"2024-01-16T14:59:56.968922+0000","last_active":"2024-01-16T14:59:57.937959+0000","last_peered":"2024-01-16T14:59:57.937959+0000","last_clean":"2024-01-16T14:59:57.937959+0000","last_became_active":"2024-01-16T14:59:56.968639+0000","last_became_peered":"2024-01-16T14:59:56.968639+0000","last_unstale":"2024-01-16T14:59:57.937959+0000","last_undegraded":"2024-01-16T14:59:57.937959+0000","last_fullsized":"2024-01-16T14:59:57.937959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:29.672100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977060+0000","last_change":"2024-01-16T14:59:56.966987+0000","last_active":"2024-01-16T14:59:58.977060+0000","last_peered":"2024-01-16T14:59:58.977060+0000","last_clean":"2024-01-16T14:59:58.977060+0000","last_became_active":"2024-01-16T14:59:56.966640+0000","last_became_peered":"2024-01-16T14:59:56.966640+0000","last_unstale":"2024-01-16T14:59:58.977060+0000","last_undegraded":"2024-01-16T14:59:58.977060+0000","last_fullsized":"2024-01-16T14:59:58.977060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:19.845423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976503+0000","last_change":"2024-01-16T14:59:57.914604+0000","last_active":"2024-01-16T14:59:58.976503+0000","last_peered":"2024-01-16T14:59:58.976503+0000","last_clean":"2024-01-16T14:59:58.976503+0000","last_became_active":"2024-01-16T14:59:57.914326+0000","last_became_peered":"2024-01-16T14:59:57.914326+0000","last_unstale":"2024-01-16T14:59:58.976503+0000","last_undegraded":"2024-01-16T14:59:58.976503+0000","last_fullsized":"2024-01-16T14:59:58.976503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975252+0000","last_change":"2024-01-16T14:59:56.967597+0000","last_active":"2024-01-16T14:59:58.975252+0000","last_peered":"2024-01-16T14:59:58.975252+0000","last_clean":"2024-01-16T14:59:58.975252+0000","last_became_active":"2024-01-16T14:59:56.967030+0000","last_became_peered":"2024-01-16T14:59:56.967030+0000","last_unstale":"2024-01-16T14:59:58.975252+0000","last_undegraded":"2024-01-16T14:59:58.975252+0000","last_fullsized":"2024-01-16T14:59:58.975252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964053+0000","last_change":"2024-01-16T14:59:57.905418+0000","last_active":"2024-01-16T14:59:58.964053+0000","last_peered":"2024-01-16T14:59:58.964053+0000","last_clean":"2024-01-16T14:59:58.964053+0000","last_became_active":"2024-01-16T14:59:57.905057+0000","last_became_peered":"2024-01-16T14:59:57.905057+0000","last_unstale":"2024-01-16T14:59:58.964053+0000","last_undegraded":"2024-01-16T14:59:58.964053+0000","last_fullsized":"2024-01-16T14:59:58.964053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.903408+0000","last_change":"2024-01-16T14:59:56.966647+0000","last_active":"2024-01-16T14:59:57.903408+0000","last_peered":"2024-01-16T14:59:57.903408+0000","last_clean":"2024-01-16T14:59:57.903408+0000","last_became_active":"2024-01-16T14:59:56.966276+0000","last_became_peered":"2024-01-16T14:59:56.966276+0000","last_unstale":"2024-01-16T14:59:57.903408+0000","last_undegraded":"2024-01-16T14:59:57.903408+0000","last_fullsized":"2024-01-16T14:59:57.903408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:05.820349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976343+0000","last_change":"2024-01-16T14:59:57.923839+0000","last_active":"2024-01-16T14:59:58.976343+0000","last_peered":"2024-01-16T14:59:58.976343+0000","last_clean":"2024-01-16T14:59:58.976343+0000","last_became_active":"2024-01-16T14:59:57.923483+0000","last_became_peered":"2024-01-16T14:59:57.923483+0000","last_unstale":"2024-01-16T14:59:58.976343+0000","last_undegraded":"2024-01-16T14:59:58.976343+0000","last_fullsized":"2024-01-16T14:59:58.976343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976571+0000","last_change":"2024-01-16T14:59:56.969754+0000","last_active":"2024-01-16T14:59:58.976571+0000","last_peered":"2024-01-16T14:59:58.976571+0000","last_clean":"2024-01-16T14:59:58.976571+0000","last_became_active":"2024-01-16T14:59:56.969381+0000","last_became_peered":"2024-01-16T14:59:56.969381+0000","last_unstale":"2024-01-16T14:59:58.976571+0000","last_undegraded":"2024-01-16T14:59:58.976571+0000","last_fullsized":"2024-01-16T14:59:58.976571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974983+0000","last_change":"2024-01-16T14:59:57.914892+0000","last_active":"2024-01-16T14:59:58.974983+0000","last_peered":"2024-01-16T14:59:58.974983+0000","last_clean":"2024-01-16T14:59:58.974983+0000","last_became_active":"2024-01-16T14:59:57.913384+0000","last_became_peered":"2024-01-16T14:59:57.913384+0000","last_unstale":"2024-01-16T14:59:58.974983+0000","last_undegraded":"2024-01-16T14:59:58.974983+0000","last_fullsized":"2024-01-16T14:59:58.974983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.998301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964546+0000","last_change":"2024-01-16T14:59:56.960948+0000","last_active":"2024-01-16T14:59:58.964546+0000","last_peered":"2024-01-16T14:59:58.964546+0000","last_clean":"2024-01-16T14:59:58.964546+0000","last_became_active":"2024-01-16T14:59:56.960588+0000","last_became_peered":"2024-01-16T14:59:56.960588+0000","last_unstale":"2024-01-16T14:59:58.964546+0000","last_undegraded":"2024-01-16T14:59:58.964546+0000","last_fullsized":"2024-01-16T14:59:58.964546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885497+0000","last_change":"2024-01-16T14:59:57.911677+0000","last_active":"2024-01-16T14:59:58.885497+0000","last_peered":"2024-01-16T14:59:58.885497+0000","last_clean":"2024-01-16T14:59:58.885497+0000","last_became_active":"2024-01-16T14:59:57.911160+0000","last_became_peered":"2024-01-16T14:59:57.911160+0000","last_unstale":"2024-01-16T14:59:58.885497+0000","last_undegraded":"2024-01-16T14:59:58.885497+0000","last_fullsized":"2024-01-16T14:59:58.885497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:43.320590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.945676+0000","last_change":"2024-01-16T14:59:56.963872+0000","last_active":"2024-01-16T14:59:57.945676+0000","last_peered":"2024-01-16T14:59:57.945676+0000","last_clean":"2024-01-16T14:59:57.945676+0000","last_became_active":"2024-01-16T14:59:56.963545+0000","last_became_peered":"2024-01-16T14:59:56.963545+0000","last_unstale":"2024-01-16T14:59:57.945676+0000","last_undegraded":"2024-01-16T14:59:57.945676+0000","last_fullsized":"2024-01-16T14:59:57.945676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881456+0000","last_change":"2024-01-16T14:59:57.918001+0000","last_active":"2024-01-16T14:59:58.881456+0000","last_peered":"2024-01-16T14:59:58.881456+0000","last_clean":"2024-01-16T14:59:58.881456+0000","last_became_active":"2024-01-16T14:59:57.917646+0000","last_became_peered":"2024-01-16T14:59:57.917646+0000","last_unstale":"2024-01-16T14:59:58.881456+0000","last_undegraded":"2024-01-16T14:59:58.881456+0000","last_fullsized":"2024-01-16T14:59:58.881456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975183+0000","last_change":"2024-01-16T14:59:56.965453+0000","last_active":"2024-01-16T14:59:58.975183+0000","last_peered":"2024-01-16T14:59:58.975183+0000","last_clean":"2024-01-16T14:59:58.975183+0000","last_became_active":"2024-01-16T14:59:56.965153+0000","last_became_peered":"2024-01-16T14:59:56.965153+0000","last_unstale":"2024-01-16T14:59:58.975183+0000","last_undegraded":"2024-01-16T14:59:58.975183+0000","last_fullsized":"2024-01-16T14:59:58.975183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974193+0000","last_change":"2024-01-16T14:59:57.921295+0000","last_active":"2024-01-16T14:59:58.974193+0000","last_peered":"2024-01-16T14:59:58.974193+0000","last_clean":"2024-01-16T14:59:58.974193+0000","last_became_active":"2024-01-16T14:59:57.920961+0000","last_became_peered":"2024-01-16T14:59:57.920961+0000","last_unstale":"2024-01-16T14:59:58.974193+0000","last_undegraded":"2024-01-16T14:59:58.974193+0000","last_fullsized":"2024-01-16T14:59:58.974193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:09.224854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977499+0000","last_change":"2024-01-16T14:59:56.964539+0000","last_active":"2024-01-16T14:59:58.977499+0000","last_peered":"2024-01-16T14:59:58.977499+0000","last_clean":"2024-01-16T14:59:58.977499+0000","last_became_active":"2024-01-16T14:59:56.964337+0000","last_became_peered":"2024-01-16T14:59:56.964337+0000","last_unstale":"2024-01-16T14:59:58.977499+0000","last_undegraded":"2024-01-16T14:59:58.977499+0000","last_fullsized":"2024-01-16T14:59:58.977499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881748+0000","last_change":"2024-01-16T14:59:57.918457+0000","last_active":"2024-01-16T14:59:58.881748+0000","last_peered":"2024-01-16T14:59:58.881748+0000","last_clean":"2024-01-16T14:59:58.881748+0000","last_became_active":"2024-01-16T14:59:57.918278+0000","last_became_peered":"2024-01-16T14:59:57.918278+0000","last_unstale":"2024-01-16T14:59:58.881748+0000","last_undegraded":"2024-01-16T14:59:58.881748+0000","last_fullsized":"2024-01-16T14:59:58.881748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.900831+0000","last_change":"2024-01-16T14:59:56.969818+0000","last_active":"2024-01-16T14:59:57.900831+0000","last_peered":"2024-01-16T14:59:57.900831+0000","last_clean":"2024-01-16T14:59:57.900831+0000","last_became_active":"2024-01-16T14:59:56.969530+0000","last_became_peered":"2024-01-16T14:59:56.969530+0000","last_unstale":"2024-01-16T14:59:57.900831+0000","last_undegraded":"2024-01-16T14:59:57.900831+0000","last_fullsized":"2024-01-16T14:59:57.900831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.795006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974601+0000","last_change":"2024-01-16T14:59:57.907805+0000","last_active":"2024-01-16T14:59:58.974601+0000","last_peered":"2024-01-16T14:59:58.974601+0000","last_clean":"2024-01-16T14:59:58.974601+0000","last_became_active":"2024-01-16T14:59:57.907499+0000","last_became_peered":"2024-01-16T14:59:57.907499+0000","last_unstale":"2024-01-16T14:59:58.974601+0000","last_undegraded":"2024-01-16T14:59:58.974601+0000","last_fullsized":"2024-01-16T14:59:58.974601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973072+0000","last_change":"2024-01-16T14:59:56.950566+0000","last_active":"2024-01-16T14:59:58.973072+0000","last_peered":"2024-01-16T14:59:58.973072+0000","last_clean":"2024-01-16T14:59:58.973072+0000","last_became_active":"2024-01-16T14:59:56.949990+0000","last_became_peered":"2024-01-16T14:59:56.949990+0000","last_unstale":"2024-01-16T14:59:58.973072+0000","last_undegraded":"2024-01-16T14:59:58.973072+0000","last_fullsized":"2024-01-16T14:59:58.973072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:14.695598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880960+0000","last_change":"2024-01-16T14:59:57.918222+0000","last_active":"2024-01-16T14:59:58.880960+0000","last_peered":"2024-01-16T14:59:58.880960+0000","last_clean":"2024-01-16T14:59:58.880960+0000","last_became_active":"2024-01-16T14:59:57.917997+0000","last_became_peered":"2024-01-16T14:59:57.917997+0000","last_unstale":"2024-01-16T14:59:58.880960+0000","last_undegraded":"2024-01-16T14:59:58.880960+0000","last_fullsized":"2024-01-16T14:59:58.880960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.942520+0000","last_change":"2024-01-16T14:59:56.958392+0000","last_active":"2024-01-16T14:59:57.942520+0000","last_peered":"2024-01-16T14:59:57.942520+0000","last_clean":"2024-01-16T14:59:57.942520+0000","last_became_active":"2024-01-16T14:59:56.955933+0000","last_became_peered":"2024-01-16T14:59:56.955933+0000","last_unstale":"2024-01-16T14:59:57.942520+0000","last_undegraded":"2024-01-16T14:59:57.942520+0000","last_fullsized":"2024-01-16T14:59:57.942520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:50.131057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883242+0000","last_change":"2024-01-16T14:59:57.917801+0000","last_active":"2024-01-16T14:59:58.883242+0000","last_peered":"2024-01-16T14:59:58.883242+0000","last_clean":"2024-01-16T14:59:58.883242+0000","last_became_active":"2024-01-16T14:59:57.917470+0000","last_became_peered":"2024-01-16T14:59:57.917470+0000","last_unstale":"2024-01-16T14:59:58.883242+0000","last_undegraded":"2024-01-16T14:59:58.883242+0000","last_fullsized":"2024-01-16T14:59:58.883242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:34.142895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974235+0000","last_change":"2024-01-16T14:59:56.954099+0000","last_active":"2024-01-16T14:59:58.974235+0000","last_peered":"2024-01-16T14:59:58.974235+0000","last_clean":"2024-01-16T14:59:58.974235+0000","last_became_active":"2024-01-16T14:59:56.953811+0000","last_became_peered":"2024-01-16T14:59:56.953811+0000","last_unstale":"2024-01-16T14:59:58.974235+0000","last_undegraded":"2024-01-16T14:59:58.974235+0000","last_fullsized":"2024-01-16T14:59:58.974235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:08.095887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883459+0000","last_change":"2024-01-16T14:59:57.921075+0000","last_active":"2024-01-16T14:59:58.883459+0000","last_peered":"2024-01-16T14:59:58.883459+0000","last_clean":"2024-01-16T14:59:58.883459+0000","last_became_active":"2024-01-16T14:59:57.920649+0000","last_became_peered":"2024-01-16T14:59:57.920649+0000","last_unstale":"2024-01-16T14:59:58.883459+0000","last_undegraded":"2024-01-16T14:59:58.883459+0000","last_fullsized":"2024-01-16T14:59:58.883459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:30.432751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964505+0000","last_change":"2024-01-16T14:59:56.965927+0000","last_active":"2024-01-16T14:59:58.964505+0000","last_peered":"2024-01-16T14:59:58.964505+0000","last_clean":"2024-01-16T14:59:58.964505+0000","last_became_active":"2024-01-16T14:59:56.965627+0000","last_became_peered":"2024-01-16T14:59:56.965627+0000","last_unstale":"2024-01-16T14:59:58.964505+0000","last_undegraded":"2024-01-16T14:59:58.964505+0000","last_fullsized":"2024-01-16T14:59:58.964505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:11.715798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975760+0000","last_change":"2024-01-16T14:59:57.919926+0000","last_active":"2024-01-16T14:59:58.975760+0000","last_peered":"2024-01-16T14:59:58.975760+0000","last_clean":"2024-01-16T14:59:58.975760+0000","last_became_active":"2024-01-16T14:59:57.919380+0000","last_became_peered":"2024-01-16T14:59:57.919380+0000","last_unstale":"2024-01-16T14:59:58.975760+0000","last_undegraded":"2024-01-16T14:59:58.975760+0000","last_fullsized":"2024-01-16T14:59:58.975760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973702+0000","last_change":"2024-01-16T14:59:56.961433+0000","last_active":"2024-01-16T14:59:58.973702+0000","last_peered":"2024-01-16T14:59:58.973702+0000","last_clean":"2024-01-16T14:59:58.973702+0000","last_became_active":"2024-01-16T14:59:56.961175+0000","last_became_peered":"2024-01-16T14:59:56.961175+0000","last_unstale":"2024-01-16T14:59:58.973702+0000","last_undegraded":"2024-01-16T14:59:58.973702+0000","last_fullsized":"2024-01-16T14:59:58.973702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:34.084632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882600+0000","last_change":"2024-01-16T14:59:57.912741+0000","last_active":"2024-01-16T14:59:58.882600+0000","last_peered":"2024-01-16T14:59:58.882600+0000","last_clean":"2024-01-16T14:59:58.882600+0000","last_became_active":"2024-01-16T14:59:57.908529+0000","last_became_peered":"2024-01-16T14:59:57.908529+0000","last_unstale":"2024-01-16T14:59:58.882600+0000","last_undegraded":"2024-01-16T14:59:58.882600+0000","last_fullsized":"2024-01-16T14:59:58.882600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.907222+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T14:59:57.907222+0000","last_peered":"2024-01-16T14:59:57.907222+0000","last_clean":"2024-01-16T14:59:57.907222+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T14:59:57.907222+0000","last_undegraded":"2024-01-16T14:59:57.907222+0000","last_fullsized":"2024-01-16T14:59:57.907222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974661+0000","last_change":"2024-01-16T14:59:56.964503+0000","last_active":"2024-01-16T14:59:58.974661+0000","last_peered":"2024-01-16T14:59:58.974661+0000","last_clean":"2024-01-16T14:59:58.974661+0000","last_became_active":"2024-01-16T14:59:56.964165+0000","last_became_peered":"2024-01-16T14:59:56.964165+0000","last_unstale":"2024-01-16T14:59:58.974661+0000","last_undegraded":"2024-01-16T14:59:58.974661+0000","last_fullsized":"2024-01-16T14:59:58.974661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:39.083282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977186+0000","last_change":"2024-01-16T14:59:57.916509+0000","last_active":"2024-01-16T14:59:58.977186+0000","last_peered":"2024-01-16T14:59:58.977186+0000","last_clean":"2024-01-16T14:59:58.977186+0000","last_became_active":"2024-01-16T14:59:57.915770+0000","last_became_peered":"2024-01-16T14:59:57.915770+0000","last_unstale":"2024-01-16T14:59:58.977186+0000","last_undegraded":"2024-01-16T14:59:58.977186+0000","last_fullsized":"2024-01-16T14:59:58.977186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880642+0000","last_change":"2024-01-16T14:59:56.965724+0000","last_active":"2024-01-16T14:59:58.880642+0000","last_peered":"2024-01-16T14:59:58.880642+0000","last_clean":"2024-01-16T14:59:58.880642+0000","last_became_active":"2024-01-16T14:59:56.965458+0000","last_became_peered":"2024-01-16T14:59:56.965458+0000","last_unstale":"2024-01-16T14:59:58.880642+0000","last_undegraded":"2024-01-16T14:59:58.880642+0000","last_fullsized":"2024-01-16T14:59:58.880642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973635+0000","last_change":"2024-01-16T14:59:57.912107+0000","last_active":"2024-01-16T14:59:58.973635+0000","last_peered":"2024-01-16T14:59:58.973635+0000","last_clean":"2024-01-16T14:59:58.973635+0000","last_became_active":"2024-01-16T14:59:57.911776+0000","last_became_peered":"2024-01-16T14:59:57.911776+0000","last_unstale":"2024-01-16T14:59:58.973635+0000","last_undegraded":"2024-01-16T14:59:58.973635+0000","last_fullsized":"2024-01-16T14:59:58.973635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:33.290097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973813+0000","last_change":"2024-01-16T14:59:56.956041+0000","last_active":"2024-01-16T14:59:58.973813+0000","last_peered":"2024-01-16T14:59:58.973813+0000","last_clean":"2024-01-16T14:59:58.973813+0000","last_became_active":"2024-01-16T14:59:56.955669+0000","last_became_peered":"2024-01-16T14:59:56.955669+0000","last_unstale":"2024-01-16T14:59:58.973813+0000","last_undegraded":"2024-01-16T14:59:58.973813+0000","last_fullsized":"2024-01-16T14:59:58.973813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964006+0000","last_change":"2024-01-16T14:59:57.905614+0000","last_active":"2024-01-16T14:59:58.964006+0000","last_peered":"2024-01-16T14:59:58.964006+0000","last_clean":"2024-01-16T14:59:58.964006+0000","last_became_active":"2024-01-16T14:59:57.905277+0000","last_became_peered":"2024-01-16T14:59:57.905277+0000","last_unstale":"2024-01-16T14:59:58.964006+0000","last_undegraded":"2024-01-16T14:59:58.964006+0000","last_fullsized":"2024-01-16T14:59:58.964006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977517+0000","last_change":"2024-01-16T14:59:56.956330+0000","last_active":"2024-01-16T14:59:58.977517+0000","last_peered":"2024-01-16T14:59:58.977517+0000","last_clean":"2024-01-16T14:59:58.977517+0000","last_became_active":"2024-01-16T14:59:56.955936+0000","last_became_peered":"2024-01-16T14:59:56.955936+0000","last_unstale":"2024-01-16T14:59:58.977517+0000","last_undegraded":"2024-01-16T14:59:58.977517+0000","last_fullsized":"2024-01-16T14:59:58.977517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:53.098539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884894+0000","last_change":"2024-01-16T14:59:57.911172+0000","last_active":"2024-01-16T14:59:58.884894+0000","last_peered":"2024-01-16T14:59:58.884894+0000","last_clean":"2024-01-16T14:59:58.884894+0000","last_became_active":"2024-01-16T14:59:57.910403+0000","last_became_peered":"2024-01-16T14:59:57.910403+0000","last_unstale":"2024-01-16T14:59:58.884894+0000","last_undegraded":"2024-01-16T14:59:58.884894+0000","last_fullsized":"2024-01-16T14:59:58.884894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:23.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964455+0000","last_change":"2024-01-16T14:59:56.955294+0000","last_active":"2024-01-16T14:59:58.964455+0000","last_peered":"2024-01-16T14:59:58.964455+0000","last_clean":"2024-01-16T14:59:58.964455+0000","last_became_active":"2024-01-16T14:59:56.953316+0000","last_became_peered":"2024-01-16T14:59:56.953316+0000","last_unstale":"2024-01-16T14:59:58.964455+0000","last_undegraded":"2024-01-16T14:59:58.964455+0000","last_fullsized":"2024-01-16T14:59:58.964455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976348+0000","last_change":"2024-01-16T14:59:57.917538+0000","last_active":"2024-01-16T14:59:58.976348+0000","last_peered":"2024-01-16T14:59:58.976348+0000","last_clean":"2024-01-16T14:59:58.976348+0000","last_became_active":"2024-01-16T14:59:57.917208+0000","last_became_peered":"2024-01-16T14:59:57.917208+0000","last_unstale":"2024-01-16T14:59:58.976348+0000","last_undegraded":"2024-01-16T14:59:58.976348+0000","last_fullsized":"2024-01-16T14:59:58.976348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882243+0000","last_change":"2024-01-16T14:59:56.963301+0000","last_active":"2024-01-16T14:59:58.882243+0000","last_peered":"2024-01-16T14:59:58.882243+0000","last_clean":"2024-01-16T14:59:58.882243+0000","last_became_active":"2024-01-16T14:59:56.962999+0000","last_became_peered":"2024-01-16T14:59:56.962999+0000","last_unstale":"2024-01-16T14:59:58.882243+0000","last_undegraded":"2024-01-16T14:59:58.882243+0000","last_fullsized":"2024-01-16T14:59:58.882243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:34.876313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964280+0000","last_change":"2024-01-16T14:59:57.901887+0000","last_active":"2024-01-16T14:59:58.964280+0000","last_peered":"2024-01-16T14:59:58.964280+0000","last_clean":"2024-01-16T14:59:58.964280+0000","last_became_active":"2024-01-16T14:59:57.901474+0000","last_became_peered":"2024-01-16T14:59:57.901474+0000","last_unstale":"2024-01-16T14:59:58.964280+0000","last_undegraded":"2024-01-16T14:59:58.964280+0000","last_fullsized":"2024-01-16T14:59:58.964280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T15:00:00.206379+0000","last_change":"2024-01-16T14:59:56.967570+0000","last_active":"2024-01-16T15:00:00.206379+0000","last_peered":"2024-01-16T15:00:00.206379+0000","last_clean":"2024-01-16T15:00:00.206379+0000","last_became_active":"2024-01-16T14:59:56.967245+0000","last_became_peered":"2024-01-16T14:59:56.967245+0000","last_unstale":"2024-01-16T15:00:00.206379+0000","last_undegraded":"2024-01-16T15:00:00.206379+0000","last_fullsized":"2024-01-16T15:00:00.206379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:50.405789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.979264+0000","last_change":"2024-01-16T14:59:57.914280+0000","last_active":"2024-01-16T14:59:58.979264+0000","last_peered":"2024-01-16T14:59:58.979264+0000","last_clean":"2024-01-16T14:59:58.979264+0000","last_became_active":"2024-01-16T14:59:57.909069+0000","last_became_peered":"2024-01-16T14:59:57.909069+0000","last_unstale":"2024-01-16T14:59:58.979264+0000","last_undegraded":"2024-01-16T14:59:58.979264+0000","last_fullsized":"2024-01-16T14:59:58.979264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:45:32.241461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.943628+0000","last_change":"2024-01-16T14:59:56.963783+0000","last_active":"2024-01-16T14:59:57.943628+0000","last_peered":"2024-01-16T14:59:57.943628+0000","last_clean":"2024-01-16T14:59:57.943628+0000","last_became_active":"2024-01-16T14:59:56.963167+0000","last_became_peered":"2024-01-16T14:59:56.963167+0000","last_unstale":"2024-01-16T14:59:57.943628+0000","last_undegraded":"2024-01-16T14:59:57.943628+0000","last_fullsized":"2024-01-16T14:59:57.943628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974544+0000","last_change":"2024-01-16T14:59:57.907693+0000","last_active":"2024-01-16T14:59:58.974544+0000","last_peered":"2024-01-16T14:59:58.974544+0000","last_clean":"2024-01-16T14:59:58.974544+0000","last_became_active":"2024-01-16T14:59:57.907365+0000","last_became_peered":"2024-01-16T14:59:57.907365+0000","last_unstale":"2024-01-16T14:59:58.974544+0000","last_undegraded":"2024-01-16T14:59:58.974544+0000","last_fullsized":"2024-01-16T14:59:58.974544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:31.975241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881061+0000","last_change":"2024-01-16T14:59:56.964456+0000","last_active":"2024-01-16T14:59:58.881061+0000","last_peered":"2024-01-16T14:59:58.881061+0000","last_clean":"2024-01-16T14:59:58.881061+0000","last_became_active":"2024-01-16T14:59:56.964105+0000","last_became_peered":"2024-01-16T14:59:56.964105+0000","last_unstale":"2024-01-16T14:59:58.881061+0000","last_undegraded":"2024-01-16T14:59:58.881061+0000","last_fullsized":"2024-01-16T14:59:58.881061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884695+0000","last_change":"2024-01-16T14:59:57.909622+0000","last_active":"2024-01-16T14:59:58.884695+0000","last_peered":"2024-01-16T14:59:58.884695+0000","last_clean":"2024-01-16T14:59:58.884695+0000","last_became_active":"2024-01-16T14:59:57.905809+0000","last_became_peered":"2024-01-16T14:59:57.905809+0000","last_unstale":"2024-01-16T14:59:58.884695+0000","last_undegraded":"2024-01-16T14:59:58.884695+0000","last_fullsized":"2024-01-16T14:59:58.884695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:30.677552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964306+0000","last_change":"2024-01-16T14:59:56.961402+0000","last_active":"2024-01-16T14:59:58.964306+0000","last_peered":"2024-01-16T14:59:58.964306+0000","last_clean":"2024-01-16T14:59:58.964306+0000","last_became_active":"2024-01-16T14:59:56.959909+0000","last_became_peered":"2024-01-16T14:59:56.959909+0000","last_unstale":"2024-01-16T14:59:58.964306+0000","last_undegraded":"2024-01-16T14:59:58.964306+0000","last_fullsized":"2024-01-16T14:59:58.964306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:00.188907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976651+0000","last_change":"2024-01-16T14:59:57.902379+0000","last_active":"2024-01-16T14:59:58.976651+0000","last_peered":"2024-01-16T14:59:58.976651+0000","last_clean":"2024-01-16T14:59:58.976651+0000","last_became_active":"2024-01-16T14:59:57.902002+0000","last_became_peered":"2024-01-16T14:59:57.902002+0000","last_unstale":"2024-01-16T14:59:58.976651+0000","last_undegraded":"2024-01-16T14:59:58.976651+0000","last_fullsized":"2024-01-16T14:59:58.976651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.963988+0000","last_change":"2024-01-16T14:59:56.956035+0000","last_active":"2024-01-16T14:59:58.963988+0000","last_peered":"2024-01-16T14:59:58.963988+0000","last_clean":"2024-01-16T14:59:58.963988+0000","last_became_active":"2024-01-16T14:59:56.955666+0000","last_became_peered":"2024-01-16T14:59:56.955666+0000","last_unstale":"2024-01-16T14:59:58.963988+0000","last_undegraded":"2024-01-16T14:59:58.963988+0000","last_fullsized":"2024-01-16T14:59:58.963988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977506+0000","last_change":"2024-01-16T14:59:57.913992+0000","last_active":"2024-01-16T14:59:58.977506+0000","last_peered":"2024-01-16T14:59:58.977506+0000","last_clean":"2024-01-16T14:59:58.977506+0000","last_became_active":"2024-01-16T14:59:57.913639+0000","last_became_peered":"2024-01-16T14:59:57.913639+0000","last_unstale":"2024-01-16T14:59:58.977506+0000","last_undegraded":"2024-01-16T14:59:58.977506+0000","last_fullsized":"2024-01-16T14:59:58.977506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882012+0000","last_change":"2024-01-16T14:59:56.970741+0000","last_active":"2024-01-16T14:59:58.882012+0000","last_peered":"2024-01-16T14:59:58.882012+0000","last_clean":"2024-01-16T14:59:58.882012+0000","last_became_active":"2024-01-16T14:59:56.970075+0000","last_became_peered":"2024-01-16T14:59:56.970075+0000","last_unstale":"2024-01-16T14:59:58.882012+0000","last_undegraded":"2024-01-16T14:59:58.882012+0000","last_fullsized":"2024-01-16T14:59:58.882012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:21.992849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974326+0000","last_change":"2024-01-16T14:59:57.906046+0000","last_active":"2024-01-16T14:59:58.974326+0000","last_peered":"2024-01-16T14:59:58.974326+0000","last_clean":"2024-01-16T14:59:58.974326+0000","last_became_active":"2024-01-16T14:59:57.905751+0000","last_became_peered":"2024-01-16T14:59:57.905751+0000","last_unstale":"2024-01-16T14:59:58.974326+0000","last_undegraded":"2024-01-16T14:59:58.974326+0000","last_fullsized":"2024-01-16T14:59:58.974326+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976133+0000","last_change":"2024-01-16T14:59:57.920773+0000","last_active":"2024-01-16T14:59:58.976133+0000","last_peered":"2024-01-16T14:59:58.976133+0000","last_clean":"2024-01-16T14:59:58.976133+0000","last_became_active":"2024-01-16T14:59:57.920488+0000","last_became_peered":"2024-01-16T14:59:57.920488+0000","last_unstale":"2024-01-16T14:59:58.976133+0000","last_undegraded":"2024-01-16T14:59:58.976133+0000","last_fullsized":"2024-01-16T14:59:58.976133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881475+0000","last_change":"2024-01-16T14:59:56.968586+0000","last_active":"2024-01-16T14:59:58.881475+0000","last_peered":"2024-01-16T14:59:58.881475+0000","last_clean":"2024-01-16T14:59:58.881475+0000","last_became_active":"2024-01-16T14:59:56.967923+0000","last_became_peered":"2024-01-16T14:59:56.967923+0000","last_unstale":"2024-01-16T14:59:58.881475+0000","last_undegraded":"2024-01-16T14:59:58.881475+0000","last_fullsized":"2024-01-16T14:59:58.881475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975571+0000","last_change":"2024-01-16T14:59:57.919788+0000","last_active":"2024-01-16T14:59:58.975571+0000","last_peered":"2024-01-16T14:59:58.975571+0000","last_clean":"2024-01-16T14:59:58.975571+0000","last_became_active":"2024-01-16T14:59:57.918939+0000","last_became_peered":"2024-01-16T14:59:57.918939+0000","last_unstale":"2024-01-16T14:59:58.975571+0000","last_undegraded":"2024-01-16T14:59:58.975571+0000","last_fullsized":"2024-01-16T14:59:58.975571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881694+0000","last_change":"2024-01-16T14:59:56.969250+0000","last_active":"2024-01-16T14:59:58.881694+0000","last_peered":"2024-01-16T14:59:58.881694+0000","last_clean":"2024-01-16T14:59:58.881694+0000","last_became_active":"2024-01-16T14:59:56.968877+0000","last_became_peered":"2024-01-16T14:59:56.968877+0000","last_unstale":"2024-01-16T14:59:58.881694+0000","last_undegraded":"2024-01-16T14:59:58.881694+0000","last_fullsized":"2024-01-16T14:59:58.881694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975133+0000","last_change":"2024-01-16T14:59:57.909922+0000","last_active":"2024-01-16T14:59:58.975133+0000","last_peered":"2024-01-16T14:59:58.975133+0000","last_clean":"2024-01-16T14:59:58.975133+0000","last_became_active":"2024-01-16T14:59:57.908518+0000","last_became_peered":"2024-01-16T14:59:57.908518+0000","last_unstale":"2024-01-16T14:59:58.975133+0000","last_undegraded":"2024-01-16T14:59:58.975133+0000","last_fullsized":"2024-01-16T14:59:58.975133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978143+0000","last_change":"2024-01-16T14:59:56.956482+0000","last_active":"2024-01-16T14:59:58.978143+0000","last_peered":"2024-01-16T14:59:58.978143+0000","last_clean":"2024-01-16T14:59:58.978143+0000","last_became_active":"2024-01-16T14:59:56.955091+0000","last_became_peered":"2024-01-16T14:59:56.955091+0000","last_unstale":"2024-01-16T14:59:58.978143+0000","last_undegraded":"2024-01-16T14:59:58.978143+0000","last_fullsized":"2024-01-16T14:59:58.978143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977730+0000","last_change":"2024-01-16T14:59:57.913479+0000","last_active":"2024-01-16T14:59:58.977730+0000","last_peered":"2024-01-16T14:59:58.977730+0000","last_clean":"2024-01-16T14:59:58.977730+0000","last_became_active":"2024-01-16T14:59:57.911939+0000","last_became_peered":"2024-01-16T14:59:57.911939+0000","last_unstale":"2024-01-16T14:59:58.977730+0000","last_undegraded":"2024-01-16T14:59:58.977730+0000","last_fullsized":"2024-01-16T14:59:58.977730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:26.642001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975233+0000","last_change":"2024-01-16T14:59:56.957873+0000","last_active":"2024-01-16T14:59:58.975233+0000","last_peered":"2024-01-16T14:59:58.975233+0000","last_clean":"2024-01-16T14:59:58.975233+0000","last_became_active":"2024-01-16T14:59:56.957287+0000","last_became_peered":"2024-01-16T14:59:56.957287+0000","last_unstale":"2024-01-16T14:59:58.975233+0000","last_undegraded":"2024-01-16T14:59:58.975233+0000","last_fullsized":"2024-01-16T14:59:58.975233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509445,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1739999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]}]},{"osd":4,"up_from":15,"seq":64424509447,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":669118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":78168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":78328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.907}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.036}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":669174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0449999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:08.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:00:08.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.332+0000 7fb562600640 1 Processor -- start 2024-01-16T15:00:08.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.332+0000 7fb562600640 1 -- start start 2024-01-16T15:00:08.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb562600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb55c143c40 0x7fb55c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb562600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c0ccc10 0x7fb55c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb562600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55c070e50 con 0x7fb55c0717d0 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb562600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55c0723c0 con 0x7fb55c143c40 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb562600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c144020 con 0x7fb55c0ccc10 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb561200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb55c143c40 0x7fb55c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb561200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb55c143c40 0x7fb55c071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35184/0 (socket says 172.21.15.53:35184) 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.334+0000 7fb561200640 1 -- 172.21.15.53:0/3899203749 learned_addr learned my addr 172.21.15.53:0/3899203749 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.335+0000 7fb561c00640 1 --2- 172.21.15.53:0/3899203749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c0ccc10 0x7fb55c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.335+0000 7fb561200640 1 -- 172.21.15.53:0/3899203749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c0ccc10 msgr2=0x7fb55c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.335+0000 7fb561200640 1 --2- 172.21.15.53:0/3899203749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c0ccc10 0x7fb55c06d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.335+0000 7fb561200640 1 -- 172.21.15.53:0/3899203749 >> v1:172.21.15.146:6789/0 conn(0x7fb55c0717d0 legacy=0x7fb55c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.335+0000 7fb561200640 1 -- 172.21.15.53:0/3899203749 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c071e00 con 0x7fb55c143c40 2024-01-16T15:00:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.336+0000 7fb561200640 1 --2- 172.21.15.53:0/3899203749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c143c40 0x7fb55c071290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009870 tx=0x7fb54c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bde0cdd8ec76b8a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.337+0000 7fb55b400640 1 -- 172.21.15.53:0/3899203749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c031640 con 0x7fb55c143c40 2024-01-16T15:00:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.337+0000 7fb55b400640 1 -- 172.21.15.53:0/3899203749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c0317a0 con 0x7fb55c143c40 2024-01-16T15:00:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.337+0000 7fb55b400640 1 -- 172.21.15.53:0/3899203749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c031b30 con 0x7fb55c143c40 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.337+0000 7fb562600640 1 -- 172.21.15.53:0/3899203749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c143c40 msgr2=0x7fb55c071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.337+0000 7fb562600640 1 --2- 172.21.15.53:0/3899203749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c143c40 0x7fb55c071290 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb54c009870 tx=0x7fb54c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.338+0000 7fb562600640 1 -- 172.21.15.53:0/3899203749 shutdown_connections 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.338+0000 7fb562600640 1 --2- 172.21.15.53:0/3899203749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c0ccc10 0x7fb55c06d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.338+0000 7fb562600640 1 --2- 172.21.15.53:0/3899203749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c143c40 0x7fb55c071290 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.338+0000 7fb562600640 1 -- 172.21.15.53:0/3899203749 >> 172.21.15.53:0/3899203749 conn(0x7fb55c06a800 msgr2=0x7fb55c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.338+0000 7fb562600640 1 -- 172.21.15.53:0/3899203749 shutdown_connections 2024-01-16T15:00:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.338+0000 7fb562600640 1 -- 172.21.15.53:0/3899203749 wait complete. 2024-01-16T15:00:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.339+0000 7fb562600640 1 Processor -- start 2024-01-16T15:00:08.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 -- start start 2024-01-16T15:00:08.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 0x7fb55c0ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55c0ccc10 0x7fb55c1f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c143c40 0x7fb55c141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c0b1f00 con 0x7fb55c0717d0 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c0b3830 con 0x7fb55c0ccc10 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.340+0000 7fb562600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c0b3a80 con 0x7fb55c143c40 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 0x7fb55c0ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 0x7fb55c0ad2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35190/0 (socket says 172.21.15.53:35190) 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 -- 172.21.15.53:0/48069042 learned_addr learned my addr 172.21.15.53:0/48069042 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb55be00640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55c0ccc10 0x7fb55c1f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561c00640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c143c40 0x7fb55c141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 -- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c143c40 msgr2=0x7fb55c141c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c143c40 0x7fb55c141c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 -- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55c0ccc10 msgr2=0x7fb55c1f67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55c0ccc10 0x7fb55c1f67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.341+0000 7fb561200640 1 -- 172.21.15.53:0/48069042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c0094d0 con 0x7fb55c0717d0 2024-01-16T15:00:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.342+0000 7fb561200640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 0x7fb55c0ad2e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb54c000c00 tx=0x7fb54c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.342+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c03ac60 con 0x7fb55c0717d0 2024-01-16T15:00:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.342+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54c03ade0 con 0x7fb55c0717d0 2024-01-16T15:00:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.342+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c0b3d60 con 0x7fb55c0717d0 2024-01-16T15:00:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.342+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54c044630 con 0x7fb55c0717d0 2024-01-16T15:00:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.342+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c0b23f0 con 0x7fb55c0717d0 2024-01-16T15:00:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.343+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c067450 con 0x7fb55c0717d0 2024-01-16T15:00:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.344+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb54c02f050 con 0x7fb55c0717d0 2024-01-16T15:00:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.345+0000 7fb558c00640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53406b310 0x7fb53406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.345+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb54c0aba20 con 0x7fb55c0717d0 2024-01-16T15:00:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.345+0000 7fb55be00640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53406b310 0x7fb53406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.346+0000 7fb55be00640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53406b310 0x7fb53406d7d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb544007fd0 tx=0x7fb544009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.350+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb54c076870 con 0x7fb55c0717d0 2024-01-16T15:00:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.535+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb55c06cc40 con 0x7fb53406b310 2024-01-16T15:00:08.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.556+0000 7fb558c00640 1 -- 172.21.15.53:0/48069042 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264301 (secure 0 0 0) 0x7fb5440050d0 con 0x7fb53406b310 2024-01-16T15:00:08.540 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:08.541 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53406b310 msgr2=0x7fb53406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53406b310 0x7fb53406d7d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb544007fd0 tx=0x7fb544009040 comp rx=0 tx=0).stop 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 msgr2=0x7fb55c0ad2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 0x7fb55c0ad2e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb54c000c00 tx=0x7fb54c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 shutdown_connections 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53406b310 0x7fb53406d7d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55c143c40 0x7fb55c141c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55c0ccc10 0x7fb55c1f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 --2- 172.21.15.53:0/48069042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb55c0717d0 0x7fb55c0ad2e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 >> 172.21.15.53:0/48069042 conn(0x7fb55c06a800 msgr2=0x7fb55c06d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 shutdown_connections 2024-01-16T15:00:08.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.562+0000 7fb562600640 1 -- 172.21.15.53:0/48069042 wait complete. 2024-01-16T15:00:08.561 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2024-01-16T15:00:08.173225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235776,"kb_used_data":2952,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738944,"statfs":{"total":773094113280,"available":772852678656,"internally_reserved":0,"allocated":3022848,"data_stored":1810900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003739"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882738+0000","last_change":"2024-01-16T14:59:57.913241+0000","last_active":"2024-01-16T14:59:58.882738+0000","last_peered":"2024-01-16T14:59:58.882738+0000","last_clean":"2024-01-16T14:59:58.882738+0000","last_became_active":"2024-01-16T14:59:57.912720+0000","last_became_peered":"2024-01-16T14:59:57.912720+0000","last_unstale":"2024-01-16T14:59:58.882738+0000","last_undegraded":"2024-01-16T14:59:58.882738+0000","last_fullsized":"2024-01-16T14:59:58.882738+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964756+0000","last_change":"2024-01-16T14:59:57.910357+0000","last_active":"2024-01-16T14:59:58.964756+0000","last_peered":"2024-01-16T14:59:58.964756+0000","last_clean":"2024-01-16T14:59:58.964756+0000","last_became_active":"2024-01-16T14:59:57.909933+0000","last_became_peered":"2024-01-16T14:59:57.909933+0000","last_unstale":"2024-01-16T14:59:58.964756+0000","last_undegraded":"2024-01-16T14:59:58.964756+0000","last_fullsized":"2024-01-16T14:59:58.964756+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974477+0000","last_change":"2024-01-16T14:59:57.909709+0000","last_active":"2024-01-16T14:59:58.974477+0000","last_peered":"2024-01-16T14:59:58.974477+0000","last_clean":"2024-01-16T14:59:58.974477+0000","last_became_active":"2024-01-16T14:59:57.909405+0000","last_became_peered":"2024-01-16T14:59:57.909405+0000","last_unstale":"2024-01-16T14:59:58.974477+0000","last_undegraded":"2024-01-16T14:59:58.974477+0000","last_fullsized":"2024-01-16T14:59:58.974477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:51.411241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975424+0000","last_change":"2024-01-16T14:59:57.920032+0000","last_active":"2024-01-16T14:59:58.975424+0000","last_peered":"2024-01-16T14:59:58.975424+0000","last_clean":"2024-01-16T14:59:58.975424+0000","last_became_active":"2024-01-16T14:59:57.919630+0000","last_became_peered":"2024-01-16T14:59:57.919630+0000","last_unstale":"2024-01-16T14:59:58.975424+0000","last_undegraded":"2024-01-16T14:59:58.975424+0000","last_fullsized":"2024-01-16T14:59:58.975424+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:09.502162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882328+0000","last_change":"2024-01-16T14:59:57.917909+0000","last_active":"2024-01-16T14:59:58.882328+0000","last_peered":"2024-01-16T14:59:58.882328+0000","last_clean":"2024-01-16T14:59:58.882328+0000","last_became_active":"2024-01-16T14:59:57.914603+0000","last_became_peered":"2024-01-16T14:59:57.914603+0000","last_unstale":"2024-01-16T14:59:58.882328+0000","last_undegraded":"2024-01-16T14:59:58.882328+0000","last_fullsized":"2024-01-16T14:59:58.882328+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976981+0000","last_change":"2024-01-16T14:59:57.909424+0000","last_active":"2024-01-16T14:59:58.976981+0000","last_peered":"2024-01-16T14:59:58.976981+0000","last_clean":"2024-01-16T14:59:58.976981+0000","last_became_active":"2024-01-16T14:59:57.909160+0000","last_became_peered":"2024-01-16T14:59:57.909160+0000","last_unstale":"2024-01-16T14:59:58.976981+0000","last_undegraded":"2024-01-16T14:59:58.976981+0000","last_fullsized":"2024-01-16T14:59:58.976981+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880812+0000","last_change":"2024-01-16T14:59:57.917053+0000","last_active":"2024-01-16T14:59:58.880812+0000","last_peered":"2024-01-16T14:59:58.880812+0000","last_clean":"2024-01-16T14:59:58.880812+0000","last_became_active":"2024-01-16T14:59:57.914775+0000","last_became_peered":"2024-01-16T14:59:57.914775+0000","last_unstale":"2024-01-16T14:59:58.880812+0000","last_undegraded":"2024-01-16T14:59:58.880812+0000","last_fullsized":"2024-01-16T14:59:58.880812+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976696+0000","last_change":"2024-01-16T14:59:57.909184+0000","last_active":"2024-01-16T14:59:58.976696+0000","last_peered":"2024-01-16T14:59:58.976696+0000","last_clean":"2024-01-16T14:59:58.976696+0000","last_became_active":"2024-01-16T14:59:57.908905+0000","last_became_peered":"2024-01-16T14:59:57.908905+0000","last_unstale":"2024-01-16T14:59:58.976696+0000","last_undegraded":"2024-01-16T14:59:58.976696+0000","last_fullsized":"2024-01-16T14:59:58.976696+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883086+0000","last_change":"2024-01-16T14:59:57.921823+0000","last_active":"2024-01-16T14:59:58.883086+0000","last_peered":"2024-01-16T14:59:58.883086+0000","last_clean":"2024-01-16T14:59:58.883086+0000","last_became_active":"2024-01-16T14:59:57.921476+0000","last_became_peered":"2024-01-16T14:59:57.921476+0000","last_unstale":"2024-01-16T14:59:58.883086+0000","last_undegraded":"2024-01-16T14:59:58.883086+0000","last_fullsized":"2024-01-16T14:59:58.883086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975563+0000","last_change":"2024-01-16T14:59:57.909965+0000","last_active":"2024-01-16T14:59:58.975563+0000","last_peered":"2024-01-16T14:59:58.975563+0000","last_clean":"2024-01-16T14:59:58.975563+0000","last_became_active":"2024-01-16T14:59:57.909707+0000","last_became_peered":"2024-01-16T14:59:57.909707+0000","last_unstale":"2024-01-16T14:59:58.975563+0000","last_undegraded":"2024-01-16T14:59:58.975563+0000","last_fullsized":"2024-01-16T14:59:58.975563+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975772+0000","last_change":"2024-01-16T14:59:57.908484+0000","last_active":"2024-01-16T14:59:58.975772+0000","last_peered":"2024-01-16T14:59:58.975772+0000","last_clean":"2024-01-16T14:59:58.975772+0000","last_became_active":"2024-01-16T14:59:57.908164+0000","last_became_peered":"2024-01-16T14:59:57.908164+0000","last_unstale":"2024-01-16T14:59:58.975772+0000","last_undegraded":"2024-01-16T14:59:58.975772+0000","last_fullsized":"2024-01-16T14:59:58.975772+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885578+0000","last_change":"2024-01-16T14:59:57.914372+0000","last_active":"2024-01-16T14:59:58.885578+0000","last_peered":"2024-01-16T14:59:58.885578+0000","last_clean":"2024-01-16T14:59:58.885578+0000","last_became_active":"2024-01-16T14:59:57.913918+0000","last_became_peered":"2024-01-16T14:59:57.913918+0000","last_unstale":"2024-01-16T14:59:58.885578+0000","last_undegraded":"2024-01-16T14:59:58.885578+0000","last_fullsized":"2024-01-16T14:59:58.885578+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:08.622882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977380+0000","last_change":"2024-01-16T14:59:57.907940+0000","last_active":"2024-01-16T14:59:58.977380+0000","last_peered":"2024-01-16T14:59:58.977380+0000","last_clean":"2024-01-16T14:59:58.977380+0000","last_became_active":"2024-01-16T14:59:57.906517+0000","last_became_peered":"2024-01-16T14:59:57.906517+0000","last_unstale":"2024-01-16T14:59:58.977380+0000","last_undegraded":"2024-01-16T14:59:58.977380+0000","last_fullsized":"2024-01-16T14:59:58.977380+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974085+0000","last_change":"2024-01-16T14:59:57.907053+0000","last_active":"2024-01-16T14:59:58.974085+0000","last_peered":"2024-01-16T14:59:58.974085+0000","last_clean":"2024-01-16T14:59:58.974085+0000","last_became_active":"2024-01-16T14:59:57.906443+0000","last_became_peered":"2024-01-16T14:59:57.906443+0000","last_unstale":"2024-01-16T14:59:58.974085+0000","last_undegraded":"2024-01-16T14:59:58.974085+0000","last_fullsized":"2024-01-16T14:59:58.974085+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:01.194359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964073+0000","last_change":"2024-01-16T14:59:57.893114+0000","last_active":"2024-01-16T14:59:58.964073+0000","last_peered":"2024-01-16T14:59:58.964073+0000","last_clean":"2024-01-16T14:59:58.964073+0000","last_became_active":"2024-01-16T14:59:57.891027+0000","last_became_peered":"2024-01-16T14:59:57.891027+0000","last_unstale":"2024-01-16T14:59:58.964073+0000","last_undegraded":"2024-01-16T14:59:58.964073+0000","last_fullsized":"2024-01-16T14:59:58.964073+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974627+0000","last_change":"2024-01-16T14:59:57.911794+0000","last_active":"2024-01-16T14:59:58.974627+0000","last_peered":"2024-01-16T14:59:58.974627+0000","last_clean":"2024-01-16T14:59:58.974627+0000","last_became_active":"2024-01-16T14:59:57.911528+0000","last_became_peered":"2024-01-16T14:59:57.911528+0000","last_unstale":"2024-01-16T14:59:58.974627+0000","last_undegraded":"2024-01-16T14:59:58.974627+0000","last_fullsized":"2024-01-16T14:59:58.974627+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976633+0000","last_change":"2024-01-16T14:59:57.917524+0000","last_active":"2024-01-16T14:59:58.976633+0000","last_peered":"2024-01-16T14:59:58.976633+0000","last_clean":"2024-01-16T14:59:58.976633+0000","last_became_active":"2024-01-16T14:59:57.916983+0000","last_became_peered":"2024-01-16T14:59:57.916983+0000","last_unstale":"2024-01-16T14:59:58.976633+0000","last_undegraded":"2024-01-16T14:59:58.976633+0000","last_fullsized":"2024-01-16T14:59:58.976633+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:46.051135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974145+0000","last_change":"2024-01-16T14:59:57.914130+0000","last_active":"2024-01-16T14:59:58.974145+0000","last_peered":"2024-01-16T14:59:58.974145+0000","last_clean":"2024-01-16T14:59:58.974145+0000","last_became_active":"2024-01-16T14:59:57.913722+0000","last_became_peered":"2024-01-16T14:59:57.913722+0000","last_unstale":"2024-01-16T14:59:58.974145+0000","last_undegraded":"2024-01-16T14:59:58.974145+0000","last_fullsized":"2024-01-16T14:59:58.974145+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:37.662076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882022+0000","last_change":"2024-01-16T14:59:57.914510+0000","last_active":"2024-01-16T14:59:58.882022+0000","last_peered":"2024-01-16T14:59:58.882022+0000","last_clean":"2024-01-16T14:59:58.882022+0000","last_became_active":"2024-01-16T14:59:57.912912+0000","last_became_peered":"2024-01-16T14:59:57.912912+0000","last_unstale":"2024-01-16T14:59:58.882022+0000","last_undegraded":"2024-01-16T14:59:58.882022+0000","last_fullsized":"2024-01-16T14:59:58.882022+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975057+0000","last_change":"2024-01-16T14:59:57.911759+0000","last_active":"2024-01-16T14:59:58.975057+0000","last_peered":"2024-01-16T14:59:58.975057+0000","last_clean":"2024-01-16T14:59:58.975057+0000","last_became_active":"2024-01-16T14:59:57.911474+0000","last_became_peered":"2024-01-16T14:59:57.911474+0000","last_unstale":"2024-01-16T14:59:58.975057+0000","last_undegraded":"2024-01-16T14:59:58.975057+0000","last_fullsized":"2024-01-16T14:59:58.975057+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975453+0000","last_change":"2024-01-16T14:59:57.923664+0000","last_active":"2024-01-16T14:59:58.975453+0000","last_peered":"2024-01-16T14:59:58.975453+0000","last_clean":"2024-01-16T14:59:58.975453+0000","last_became_active":"2024-01-16T14:59:57.923402+0000","last_became_peered":"2024-01-16T14:59:57.923402+0000","last_unstale":"2024-01-16T14:59:58.975453+0000","last_undegraded":"2024-01-16T14:59:58.975453+0000","last_fullsized":"2024-01-16T14:59:58.975453+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:34:28.532818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977267+0000","last_change":"2024-01-16T14:59:57.923807+0000","last_active":"2024-01-16T14:59:58.977267+0000","last_peered":"2024-01-16T14:59:58.977267+0000","last_clean":"2024-01-16T14:59:58.977267+0000","last_became_active":"2024-01-16T14:59:57.923493+0000","last_became_peered":"2024-01-16T14:59:57.923493+0000","last_unstale":"2024-01-16T14:59:58.977267+0000","last_undegraded":"2024-01-16T14:59:58.977267+0000","last_fullsized":"2024-01-16T14:59:58.977267+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:42:18.248488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977086+0000","last_change":"2024-01-16T14:59:57.912156+0000","last_active":"2024-01-16T14:59:58.977086+0000","last_peered":"2024-01-16T14:59:58.977086+0000","last_clean":"2024-01-16T14:59:58.977086+0000","last_became_active":"2024-01-16T14:59:57.911497+0000","last_became_peered":"2024-01-16T14:59:57.911497+0000","last_unstale":"2024-01-16T14:59:58.977086+0000","last_undegraded":"2024-01-16T14:59:58.977086+0000","last_fullsized":"2024-01-16T14:59:58.977086+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:51:12.814322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974975+0000","last_change":"2024-01-16T14:59:57.918668+0000","last_active":"2024-01-16T14:59:58.974975+0000","last_peered":"2024-01-16T14:59:58.974975+0000","last_clean":"2024-01-16T14:59:58.974975+0000","last_became_active":"2024-01-16T14:59:57.918314+0000","last_became_peered":"2024-01-16T14:59:57.918314+0000","last_unstale":"2024-01-16T14:59:58.974975+0000","last_undegraded":"2024-01-16T14:59:58.974975+0000","last_fullsized":"2024-01-16T14:59:58.974975+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978927+0000","last_change":"2024-01-16T14:59:57.912774+0000","last_active":"2024-01-16T14:59:58.978927+0000","last_peered":"2024-01-16T14:59:58.978927+0000","last_clean":"2024-01-16T14:59:58.978927+0000","last_became_active":"2024-01-16T14:59:57.910934+0000","last_became_peered":"2024-01-16T14:59:57.910934+0000","last_unstale":"2024-01-16T14:59:58.978927+0000","last_undegraded":"2024-01-16T14:59:58.978927+0000","last_fullsized":"2024-01-16T14:59:58.978927+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:47.822358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975717+0000","last_change":"2024-01-16T14:59:57.921118+0000","last_active":"2024-01-16T14:59:58.975717+0000","last_peered":"2024-01-16T14:59:58.975717+0000","last_clean":"2024-01-16T14:59:58.975717+0000","last_became_active":"2024-01-16T14:59:57.920835+0000","last_became_peered":"2024-01-16T14:59:57.920835+0000","last_unstale":"2024-01-16T14:59:58.975717+0000","last_undegraded":"2024-01-16T14:59:58.975717+0000","last_fullsized":"2024-01-16T14:59:58.975717+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:26.232093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964405+0000","last_change":"2024-01-16T14:59:57.898548+0000","last_active":"2024-01-16T14:59:58.964405+0000","last_peered":"2024-01-16T14:59:58.964405+0000","last_clean":"2024-01-16T14:59:58.964405+0000","last_became_active":"2024-01-16T14:59:57.897874+0000","last_became_peered":"2024-01-16T14:59:57.897874+0000","last_unstale":"2024-01-16T14:59:58.964405+0000","last_undegraded":"2024-01-16T14:59:58.964405+0000","last_fullsized":"2024-01-16T14:59:58.964405+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:35.881765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978167+0000","last_change":"2024-01-16T14:59:57.914061+0000","last_active":"2024-01-16T14:59:58.978167+0000","last_peered":"2024-01-16T14:59:58.978167+0000","last_clean":"2024-01-16T14:59:58.978167+0000","last_became_active":"2024-01-16T14:59:57.913718+0000","last_became_peered":"2024-01-16T14:59:57.913718+0000","last_unstale":"2024-01-16T14:59:58.978167+0000","last_undegraded":"2024-01-16T14:59:58.978167+0000","last_fullsized":"2024-01-16T14:59:58.978167+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:46.106265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974760+0000","last_change":"2024-01-16T14:59:57.917278+0000","last_active":"2024-01-16T14:59:58.974760+0000","last_peered":"2024-01-16T14:59:58.974760+0000","last_clean":"2024-01-16T14:59:58.974760+0000","last_became_active":"2024-01-16T14:59:57.916768+0000","last_became_peered":"2024-01-16T14:59:57.916768+0000","last_unstale":"2024-01-16T14:59:58.974760+0000","last_undegraded":"2024-01-16T14:59:58.974760+0000","last_fullsized":"2024-01-16T14:59:58.974760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:12.721250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977355+0000","last_change":"2024-01-16T14:59:57.915403+0000","last_active":"2024-01-16T14:59:58.977355+0000","last_peered":"2024-01-16T14:59:58.977355+0000","last_clean":"2024-01-16T14:59:58.977355+0000","last_became_active":"2024-01-16T14:59:57.915134+0000","last_became_peered":"2024-01-16T14:59:57.915134+0000","last_unstale":"2024-01-16T14:59:58.977355+0000","last_undegraded":"2024-01-16T14:59:58.977355+0000","last_fullsized":"2024-01-16T14:59:58.977355+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:11:24.224104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973385+0000","last_change":"2024-01-16T14:59:57.910767+0000","last_active":"2024-01-16T14:59:58.973385+0000","last_peered":"2024-01-16T14:59:58.973385+0000","last_clean":"2024-01-16T14:59:58.973385+0000","last_became_active":"2024-01-16T14:59:57.908877+0000","last_became_peered":"2024-01-16T14:59:57.908877+0000","last_unstale":"2024-01-16T14:59:58.973385+0000","last_undegraded":"2024-01-16T14:59:58.973385+0000","last_fullsized":"2024-01-16T14:59:58.973385+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:58.524840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974275+0000","last_change":"2024-01-16T14:59:57.912035+0000","last_active":"2024-01-16T14:59:58.974275+0000","last_peered":"2024-01-16T14:59:58.974275+0000","last_clean":"2024-01-16T14:59:58.974275+0000","last_became_active":"2024-01-16T14:59:57.911641+0000","last_became_peered":"2024-01-16T14:59:57.911641+0000","last_unstale":"2024-01-16T14:59:58.974275+0000","last_undegraded":"2024-01-16T14:59:58.974275+0000","last_fullsized":"2024-01-16T14:59:58.974275+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884477+0000","last_change":"2024-01-16T14:59:57.898975+0000","last_active":"2024-01-16T14:59:58.884477+0000","last_peered":"2024-01-16T14:59:58.884477+0000","last_clean":"2024-01-16T14:59:58.884477+0000","last_became_active":"2024-01-16T14:59:57.894960+0000","last_became_peered":"2024-01-16T14:59:57.894960+0000","last_unstale":"2024-01-16T14:59:58.884477+0000","last_undegraded":"2024-01-16T14:59:58.884477+0000","last_fullsized":"2024-01-16T14:59:58.884477+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:50.235874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977646+0000","last_change":"2024-01-16T14:59:56.961128+0000","last_active":"2024-01-16T14:59:58.977646+0000","last_peered":"2024-01-16T14:59:58.977646+0000","last_clean":"2024-01-16T14:59:58.977646+0000","last_became_active":"2024-01-16T14:59:56.960829+0000","last_became_peered":"2024-01-16T14:59:56.960829+0000","last_unstale":"2024-01-16T14:59:58.977646+0000","last_undegraded":"2024-01-16T14:59:58.977646+0000","last_fullsized":"2024-01-16T14:59:58.977646+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975040+0000","last_change":"2024-01-16T14:59:57.907318+0000","last_active":"2024-01-16T14:59:58.975040+0000","last_peered":"2024-01-16T14:59:58.975040+0000","last_clean":"2024-01-16T14:59:58.975040+0000","last_became_active":"2024-01-16T14:59:57.907025+0000","last_became_peered":"2024-01-16T14:59:57.907025+0000","last_unstale":"2024-01-16T14:59:58.975040+0000","last_undegraded":"2024-01-16T14:59:58.975040+0000","last_fullsized":"2024-01-16T14:59:58.975040+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.937959+0000","last_change":"2024-01-16T14:59:56.968922+0000","last_active":"2024-01-16T14:59:57.937959+0000","last_peered":"2024-01-16T14:59:57.937959+0000","last_clean":"2024-01-16T14:59:57.937959+0000","last_became_active":"2024-01-16T14:59:56.968639+0000","last_became_peered":"2024-01-16T14:59:56.968639+0000","last_unstale":"2024-01-16T14:59:57.937959+0000","last_undegraded":"2024-01-16T14:59:57.937959+0000","last_fullsized":"2024-01-16T14:59:57.937959+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:29.672100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977060+0000","last_change":"2024-01-16T14:59:56.966987+0000","last_active":"2024-01-16T14:59:58.977060+0000","last_peered":"2024-01-16T14:59:58.977060+0000","last_clean":"2024-01-16T14:59:58.977060+0000","last_became_active":"2024-01-16T14:59:56.966640+0000","last_became_peered":"2024-01-16T14:59:56.966640+0000","last_unstale":"2024-01-16T14:59:58.977060+0000","last_undegraded":"2024-01-16T14:59:58.977060+0000","last_fullsized":"2024-01-16T14:59:58.977060+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:12:19.845423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976503+0000","last_change":"2024-01-16T14:59:57.914604+0000","last_active":"2024-01-16T14:59:58.976503+0000","last_peered":"2024-01-16T14:59:58.976503+0000","last_clean":"2024-01-16T14:59:58.976503+0000","last_became_active":"2024-01-16T14:59:57.914326+0000","last_became_peered":"2024-01-16T14:59:57.914326+0000","last_unstale":"2024-01-16T14:59:58.976503+0000","last_undegraded":"2024-01-16T14:59:58.976503+0000","last_fullsized":"2024-01-16T14:59:58.976503+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:03.085045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975252+0000","last_change":"2024-01-16T14:59:56.967597+0000","last_active":"2024-01-16T14:59:58.975252+0000","last_peered":"2024-01-16T14:59:58.975252+0000","last_clean":"2024-01-16T14:59:58.975252+0000","last_became_active":"2024-01-16T14:59:56.967030+0000","last_became_peered":"2024-01-16T14:59:56.967030+0000","last_unstale":"2024-01-16T14:59:58.975252+0000","last_undegraded":"2024-01-16T14:59:58.975252+0000","last_fullsized":"2024-01-16T14:59:58.975252+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964053+0000","last_change":"2024-01-16T14:59:57.905418+0000","last_active":"2024-01-16T14:59:58.964053+0000","last_peered":"2024-01-16T14:59:58.964053+0000","last_clean":"2024-01-16T14:59:58.964053+0000","last_became_active":"2024-01-16T14:59:57.905057+0000","last_became_peered":"2024-01-16T14:59:57.905057+0000","last_unstale":"2024-01-16T14:59:58.964053+0000","last_undegraded":"2024-01-16T14:59:58.964053+0000","last_fullsized":"2024-01-16T14:59:58.964053+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.903408+0000","last_change":"2024-01-16T14:59:56.966647+0000","last_active":"2024-01-16T14:59:57.903408+0000","last_peered":"2024-01-16T14:59:57.903408+0000","last_clean":"2024-01-16T14:59:57.903408+0000","last_became_active":"2024-01-16T14:59:56.966276+0000","last_became_peered":"2024-01-16T14:59:56.966276+0000","last_unstale":"2024-01-16T14:59:57.903408+0000","last_undegraded":"2024-01-16T14:59:57.903408+0000","last_fullsized":"2024-01-16T14:59:57.903408+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:05.820349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976343+0000","last_change":"2024-01-16T14:59:57.923839+0000","last_active":"2024-01-16T14:59:58.976343+0000","last_peered":"2024-01-16T14:59:58.976343+0000","last_clean":"2024-01-16T14:59:58.976343+0000","last_became_active":"2024-01-16T14:59:57.923483+0000","last_became_peered":"2024-01-16T14:59:57.923483+0000","last_unstale":"2024-01-16T14:59:58.976343+0000","last_undegraded":"2024-01-16T14:59:58.976343+0000","last_fullsized":"2024-01-16T14:59:58.976343+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976571+0000","last_change":"2024-01-16T14:59:56.969754+0000","last_active":"2024-01-16T14:59:58.976571+0000","last_peered":"2024-01-16T14:59:58.976571+0000","last_clean":"2024-01-16T14:59:58.976571+0000","last_became_active":"2024-01-16T14:59:56.969381+0000","last_became_peered":"2024-01-16T14:59:56.969381+0000","last_unstale":"2024-01-16T14:59:58.976571+0000","last_undegraded":"2024-01-16T14:59:58.976571+0000","last_fullsized":"2024-01-16T14:59:58.976571+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974983+0000","last_change":"2024-01-16T14:59:57.914892+0000","last_active":"2024-01-16T14:59:58.974983+0000","last_peered":"2024-01-16T14:59:58.974983+0000","last_clean":"2024-01-16T14:59:58.974983+0000","last_became_active":"2024-01-16T14:59:57.913384+0000","last_became_peered":"2024-01-16T14:59:57.913384+0000","last_unstale":"2024-01-16T14:59:58.974983+0000","last_undegraded":"2024-01-16T14:59:58.974983+0000","last_fullsized":"2024-01-16T14:59:58.974983+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:22.998301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964546+0000","last_change":"2024-01-16T14:59:56.960948+0000","last_active":"2024-01-16T14:59:58.964546+0000","last_peered":"2024-01-16T14:59:58.964546+0000","last_clean":"2024-01-16T14:59:58.964546+0000","last_became_active":"2024-01-16T14:59:56.960588+0000","last_became_peered":"2024-01-16T14:59:56.960588+0000","last_unstale":"2024-01-16T14:59:58.964546+0000","last_undegraded":"2024-01-16T14:59:58.964546+0000","last_fullsized":"2024-01-16T14:59:58.964546+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:30.969788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.885497+0000","last_change":"2024-01-16T14:59:57.911677+0000","last_active":"2024-01-16T14:59:58.885497+0000","last_peered":"2024-01-16T14:59:58.885497+0000","last_clean":"2024-01-16T14:59:58.885497+0000","last_became_active":"2024-01-16T14:59:57.911160+0000","last_became_peered":"2024-01-16T14:59:57.911160+0000","last_unstale":"2024-01-16T14:59:58.885497+0000","last_undegraded":"2024-01-16T14:59:58.885497+0000","last_fullsized":"2024-01-16T14:59:58.885497+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:43.320590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.945676+0000","last_change":"2024-01-16T14:59:56.963872+0000","last_active":"2024-01-16T14:59:57.945676+0000","last_peered":"2024-01-16T14:59:57.945676+0000","last_clean":"2024-01-16T14:59:57.945676+0000","last_became_active":"2024-01-16T14:59:56.963545+0000","last_became_peered":"2024-01-16T14:59:56.963545+0000","last_unstale":"2024-01-16T14:59:57.945676+0000","last_undegraded":"2024-01-16T14:59:57.945676+0000","last_fullsized":"2024-01-16T14:59:57.945676+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881456+0000","last_change":"2024-01-16T14:59:57.918001+0000","last_active":"2024-01-16T14:59:58.881456+0000","last_peered":"2024-01-16T14:59:58.881456+0000","last_clean":"2024-01-16T14:59:58.881456+0000","last_became_active":"2024-01-16T14:59:57.917646+0000","last_became_peered":"2024-01-16T14:59:57.917646+0000","last_unstale":"2024-01-16T14:59:58.881456+0000","last_undegraded":"2024-01-16T14:59:58.881456+0000","last_fullsized":"2024-01-16T14:59:58.881456+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975183+0000","last_change":"2024-01-16T14:59:56.965453+0000","last_active":"2024-01-16T14:59:58.975183+0000","last_peered":"2024-01-16T14:59:58.975183+0000","last_clean":"2024-01-16T14:59:58.975183+0000","last_became_active":"2024-01-16T14:59:56.965153+0000","last_became_peered":"2024-01-16T14:59:56.965153+0000","last_unstale":"2024-01-16T14:59:58.975183+0000","last_undegraded":"2024-01-16T14:59:58.975183+0000","last_fullsized":"2024-01-16T14:59:58.975183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974193+0000","last_change":"2024-01-16T14:59:57.921295+0000","last_active":"2024-01-16T14:59:58.974193+0000","last_peered":"2024-01-16T14:59:58.974193+0000","last_clean":"2024-01-16T14:59:58.974193+0000","last_became_active":"2024-01-16T14:59:57.920961+0000","last_became_peered":"2024-01-16T14:59:57.920961+0000","last_unstale":"2024-01-16T14:59:58.974193+0000","last_undegraded":"2024-01-16T14:59:58.974193+0000","last_fullsized":"2024-01-16T14:59:58.974193+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:37:09.224854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977499+0000","last_change":"2024-01-16T14:59:56.964539+0000","last_active":"2024-01-16T14:59:58.977499+0000","last_peered":"2024-01-16T14:59:58.977499+0000","last_clean":"2024-01-16T14:59:58.977499+0000","last_became_active":"2024-01-16T14:59:56.964337+0000","last_became_peered":"2024-01-16T14:59:56.964337+0000","last_unstale":"2024-01-16T14:59:58.977499+0000","last_undegraded":"2024-01-16T14:59:58.977499+0000","last_fullsized":"2024-01-16T14:59:58.977499+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881748+0000","last_change":"2024-01-16T14:59:57.918457+0000","last_active":"2024-01-16T14:59:58.881748+0000","last_peered":"2024-01-16T14:59:58.881748+0000","last_clean":"2024-01-16T14:59:58.881748+0000","last_became_active":"2024-01-16T14:59:57.918278+0000","last_became_peered":"2024-01-16T14:59:57.918278+0000","last_unstale":"2024-01-16T14:59:58.881748+0000","last_undegraded":"2024-01-16T14:59:58.881748+0000","last_fullsized":"2024-01-16T14:59:58.881748+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:46.758507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.900831+0000","last_change":"2024-01-16T14:59:56.969818+0000","last_active":"2024-01-16T14:59:57.900831+0000","last_peered":"2024-01-16T14:59:57.900831+0000","last_clean":"2024-01-16T14:59:57.900831+0000","last_became_active":"2024-01-16T14:59:56.969530+0000","last_became_peered":"2024-01-16T14:59:56.969530+0000","last_unstale":"2024-01-16T14:59:57.900831+0000","last_undegraded":"2024-01-16T14:59:57.900831+0000","last_fullsized":"2024-01-16T14:59:57.900831+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:20:24.795006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974601+0000","last_change":"2024-01-16T14:59:57.907805+0000","last_active":"2024-01-16T14:59:58.974601+0000","last_peered":"2024-01-16T14:59:58.974601+0000","last_clean":"2024-01-16T14:59:58.974601+0000","last_became_active":"2024-01-16T14:59:57.907499+0000","last_became_peered":"2024-01-16T14:59:57.907499+0000","last_unstale":"2024-01-16T14:59:58.974601+0000","last_undegraded":"2024-01-16T14:59:58.974601+0000","last_fullsized":"2024-01-16T14:59:58.974601+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:07:25.657510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":20,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973072+0000","last_change":"2024-01-16T14:59:56.950566+0000","last_active":"2024-01-16T14:59:58.973072+0000","last_peered":"2024-01-16T14:59:58.973072+0000","last_clean":"2024-01-16T14:59:58.973072+0000","last_became_active":"2024-01-16T14:59:56.949990+0000","last_became_peered":"2024-01-16T14:59:56.949990+0000","last_unstale":"2024-01-16T14:59:58.973072+0000","last_undegraded":"2024-01-16T14:59:58.973072+0000","last_fullsized":"2024-01-16T14:59:58.973072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:14.695598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880960+0000","last_change":"2024-01-16T14:59:57.918222+0000","last_active":"2024-01-16T14:59:58.880960+0000","last_peered":"2024-01-16T14:59:58.880960+0000","last_clean":"2024-01-16T14:59:58.880960+0000","last_became_active":"2024-01-16T14:59:57.917997+0000","last_became_peered":"2024-01-16T14:59:57.917997+0000","last_unstale":"2024-01-16T14:59:58.880960+0000","last_undegraded":"2024-01-16T14:59:58.880960+0000","last_fullsized":"2024-01-16T14:59:58.880960+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.942520+0000","last_change":"2024-01-16T14:59:56.958392+0000","last_active":"2024-01-16T14:59:57.942520+0000","last_peered":"2024-01-16T14:59:57.942520+0000","last_clean":"2024-01-16T14:59:57.942520+0000","last_became_active":"2024-01-16T14:59:56.955933+0000","last_became_peered":"2024-01-16T14:59:56.955933+0000","last_unstale":"2024-01-16T14:59:57.942520+0000","last_undegraded":"2024-01-16T14:59:57.942520+0000","last_fullsized":"2024-01-16T14:59:57.942520+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:50.131057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883242+0000","last_change":"2024-01-16T14:59:57.917801+0000","last_active":"2024-01-16T14:59:58.883242+0000","last_peered":"2024-01-16T14:59:58.883242+0000","last_clean":"2024-01-16T14:59:58.883242+0000","last_became_active":"2024-01-16T14:59:57.917470+0000","last_became_peered":"2024-01-16T14:59:57.917470+0000","last_unstale":"2024-01-16T14:59:58.883242+0000","last_undegraded":"2024-01-16T14:59:58.883242+0000","last_fullsized":"2024-01-16T14:59:58.883242+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:34.142895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974235+0000","last_change":"2024-01-16T14:59:56.954099+0000","last_active":"2024-01-16T14:59:58.974235+0000","last_peered":"2024-01-16T14:59:58.974235+0000","last_clean":"2024-01-16T14:59:58.974235+0000","last_became_active":"2024-01-16T14:59:56.953811+0000","last_became_peered":"2024-01-16T14:59:56.953811+0000","last_unstale":"2024-01-16T14:59:58.974235+0000","last_undegraded":"2024-01-16T14:59:58.974235+0000","last_fullsized":"2024-01-16T14:59:58.974235+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:08.095887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.883459+0000","last_change":"2024-01-16T14:59:57.921075+0000","last_active":"2024-01-16T14:59:58.883459+0000","last_peered":"2024-01-16T14:59:58.883459+0000","last_clean":"2024-01-16T14:59:58.883459+0000","last_became_active":"2024-01-16T14:59:57.920649+0000","last_became_peered":"2024-01-16T14:59:57.920649+0000","last_unstale":"2024-01-16T14:59:58.883459+0000","last_undegraded":"2024-01-16T14:59:58.883459+0000","last_fullsized":"2024-01-16T14:59:58.883459+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:15:30.432751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964505+0000","last_change":"2024-01-16T14:59:56.965927+0000","last_active":"2024-01-16T14:59:58.964505+0000","last_peered":"2024-01-16T14:59:58.964505+0000","last_clean":"2024-01-16T14:59:58.964505+0000","last_became_active":"2024-01-16T14:59:56.965627+0000","last_became_peered":"2024-01-16T14:59:56.965627+0000","last_unstale":"2024-01-16T14:59:58.964505+0000","last_undegraded":"2024-01-16T14:59:58.964505+0000","last_fullsized":"2024-01-16T14:59:58.964505+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:11.715798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975760+0000","last_change":"2024-01-16T14:59:57.919926+0000","last_active":"2024-01-16T14:59:58.975760+0000","last_peered":"2024-01-16T14:59:58.975760+0000","last_clean":"2024-01-16T14:59:58.975760+0000","last_became_active":"2024-01-16T14:59:57.919380+0000","last_became_peered":"2024-01-16T14:59:57.919380+0000","last_unstale":"2024-01-16T14:59:58.975760+0000","last_undegraded":"2024-01-16T14:59:58.975760+0000","last_fullsized":"2024-01-16T14:59:58.975760+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:58:16.489732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973702+0000","last_change":"2024-01-16T14:59:56.961433+0000","last_active":"2024-01-16T14:59:58.973702+0000","last_peered":"2024-01-16T14:59:58.973702+0000","last_clean":"2024-01-16T14:59:58.973702+0000","last_became_active":"2024-01-16T14:59:56.961175+0000","last_became_peered":"2024-01-16T14:59:56.961175+0000","last_unstale":"2024-01-16T14:59:58.973702+0000","last_undegraded":"2024-01-16T14:59:58.973702+0000","last_fullsized":"2024-01-16T14:59:58.973702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:34.084632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882600+0000","last_change":"2024-01-16T14:59:57.912741+0000","last_active":"2024-01-16T14:59:58.882600+0000","last_peered":"2024-01-16T14:59:58.882600+0000","last_clean":"2024-01-16T14:59:58.882600+0000","last_became_active":"2024-01-16T14:59:57.908529+0000","last_became_peered":"2024-01-16T14:59:57.908529+0000","last_unstale":"2024-01-16T14:59:58.882600+0000","last_undegraded":"2024-01-16T14:59:58.882600+0000","last_fullsized":"2024-01-16T14:59:58.882600+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:15.358921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":165,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.907222+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T14:59:57.907222+0000","last_peered":"2024-01-16T14:59:57.907222+0000","last_clean":"2024-01-16T14:59:57.907222+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T14:59:57.907222+0000","last_undegraded":"2024-01-16T14:59:57.907222+0000","last_fullsized":"2024-01-16T14:59:57.907222+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974661+0000","last_change":"2024-01-16T14:59:56.964503+0000","last_active":"2024-01-16T14:59:58.974661+0000","last_peered":"2024-01-16T14:59:58.974661+0000","last_clean":"2024-01-16T14:59:58.974661+0000","last_became_active":"2024-01-16T14:59:56.964165+0000","last_became_peered":"2024-01-16T14:59:56.964165+0000","last_unstale":"2024-01-16T14:59:58.974661+0000","last_undegraded":"2024-01-16T14:59:58.974661+0000","last_fullsized":"2024-01-16T14:59:58.974661+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:39.083282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977186+0000","last_change":"2024-01-16T14:59:57.916509+0000","last_active":"2024-01-16T14:59:58.977186+0000","last_peered":"2024-01-16T14:59:58.977186+0000","last_clean":"2024-01-16T14:59:58.977186+0000","last_became_active":"2024-01-16T14:59:57.915770+0000","last_became_peered":"2024-01-16T14:59:57.915770+0000","last_unstale":"2024-01-16T14:59:58.977186+0000","last_undegraded":"2024-01-16T14:59:58.977186+0000","last_fullsized":"2024-01-16T14:59:58.977186+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:27.306592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.880642+0000","last_change":"2024-01-16T14:59:56.965724+0000","last_active":"2024-01-16T14:59:58.880642+0000","last_peered":"2024-01-16T14:59:58.880642+0000","last_clean":"2024-01-16T14:59:58.880642+0000","last_became_active":"2024-01-16T14:59:56.965458+0000","last_became_peered":"2024-01-16T14:59:56.965458+0000","last_unstale":"2024-01-16T14:59:58.880642+0000","last_undegraded":"2024-01-16T14:59:58.880642+0000","last_fullsized":"2024-01-16T14:59:58.880642+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973635+0000","last_change":"2024-01-16T14:59:57.912107+0000","last_active":"2024-01-16T14:59:58.973635+0000","last_peered":"2024-01-16T14:59:58.973635+0000","last_clean":"2024-01-16T14:59:58.973635+0000","last_became_active":"2024-01-16T14:59:57.911776+0000","last_became_peered":"2024-01-16T14:59:57.911776+0000","last_unstale":"2024-01-16T14:59:58.973635+0000","last_undegraded":"2024-01-16T14:59:58.973635+0000","last_fullsized":"2024-01-16T14:59:58.973635+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:33.290097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.973813+0000","last_change":"2024-01-16T14:59:56.956041+0000","last_active":"2024-01-16T14:59:58.973813+0000","last_peered":"2024-01-16T14:59:58.973813+0000","last_clean":"2024-01-16T14:59:58.973813+0000","last_became_active":"2024-01-16T14:59:56.955669+0000","last_became_peered":"2024-01-16T14:59:56.955669+0000","last_unstale":"2024-01-16T14:59:58.973813+0000","last_undegraded":"2024-01-16T14:59:58.973813+0000","last_fullsized":"2024-01-16T14:59:58.973813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964006+0000","last_change":"2024-01-16T14:59:57.905614+0000","last_active":"2024-01-16T14:59:58.964006+0000","last_peered":"2024-01-16T14:59:58.964006+0000","last_clean":"2024-01-16T14:59:58.964006+0000","last_became_active":"2024-01-16T14:59:57.905277+0000","last_became_peered":"2024-01-16T14:59:57.905277+0000","last_unstale":"2024-01-16T14:59:58.964006+0000","last_undegraded":"2024-01-16T14:59:58.964006+0000","last_fullsized":"2024-01-16T14:59:58.964006+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:05.697394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977517+0000","last_change":"2024-01-16T14:59:56.956330+0000","last_active":"2024-01-16T14:59:58.977517+0000","last_peered":"2024-01-16T14:59:58.977517+0000","last_clean":"2024-01-16T14:59:58.977517+0000","last_became_active":"2024-01-16T14:59:56.955936+0000","last_became_peered":"2024-01-16T14:59:56.955936+0000","last_unstale":"2024-01-16T14:59:58.977517+0000","last_undegraded":"2024-01-16T14:59:58.977517+0000","last_fullsized":"2024-01-16T14:59:58.977517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:53.098539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884894+0000","last_change":"2024-01-16T14:59:57.911172+0000","last_active":"2024-01-16T14:59:58.884894+0000","last_peered":"2024-01-16T14:59:58.884894+0000","last_clean":"2024-01-16T14:59:58.884894+0000","last_became_active":"2024-01-16T14:59:57.910403+0000","last_became_peered":"2024-01-16T14:59:57.910403+0000","last_unstale":"2024-01-16T14:59:58.884894+0000","last_undegraded":"2024-01-16T14:59:58.884894+0000","last_fullsized":"2024-01-16T14:59:58.884894+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:56:23.023236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964455+0000","last_change":"2024-01-16T14:59:56.955294+0000","last_active":"2024-01-16T14:59:58.964455+0000","last_peered":"2024-01-16T14:59:58.964455+0000","last_clean":"2024-01-16T14:59:58.964455+0000","last_became_active":"2024-01-16T14:59:56.953316+0000","last_became_peered":"2024-01-16T14:59:56.953316+0000","last_unstale":"2024-01-16T14:59:58.964455+0000","last_undegraded":"2024-01-16T14:59:58.964455+0000","last_fullsized":"2024-01-16T14:59:58.964455+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:57:49.095044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976348+0000","last_change":"2024-01-16T14:59:57.917538+0000","last_active":"2024-01-16T14:59:58.976348+0000","last_peered":"2024-01-16T14:59:58.976348+0000","last_clean":"2024-01-16T14:59:58.976348+0000","last_became_active":"2024-01-16T14:59:57.917208+0000","last_became_peered":"2024-01-16T14:59:57.917208+0000","last_unstale":"2024-01-16T14:59:58.976348+0000","last_undegraded":"2024-01-16T14:59:58.976348+0000","last_fullsized":"2024-01-16T14:59:58.976348+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:51.476292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882243+0000","last_change":"2024-01-16T14:59:56.963301+0000","last_active":"2024-01-16T14:59:58.882243+0000","last_peered":"2024-01-16T14:59:58.882243+0000","last_clean":"2024-01-16T14:59:58.882243+0000","last_became_active":"2024-01-16T14:59:56.962999+0000","last_became_peered":"2024-01-16T14:59:56.962999+0000","last_unstale":"2024-01-16T14:59:58.882243+0000","last_undegraded":"2024-01-16T14:59:58.882243+0000","last_fullsized":"2024-01-16T14:59:58.882243+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:21:34.876313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964280+0000","last_change":"2024-01-16T14:59:57.901887+0000","last_active":"2024-01-16T14:59:58.964280+0000","last_peered":"2024-01-16T14:59:58.964280+0000","last_clean":"2024-01-16T14:59:58.964280+0000","last_became_active":"2024-01-16T14:59:57.901474+0000","last_became_peered":"2024-01-16T14:59:57.901474+0000","last_unstale":"2024-01-16T14:59:58.964280+0000","last_undegraded":"2024-01-16T14:59:58.964280+0000","last_fullsized":"2024-01-16T14:59:58.964280+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:38.483259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"22'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T15:00:00.206379+0000","last_change":"2024-01-16T14:59:56.967570+0000","last_active":"2024-01-16T15:00:00.206379+0000","last_peered":"2024-01-16T15:00:00.206379+0000","last_clean":"2024-01-16T15:00:00.206379+0000","last_became_active":"2024-01-16T14:59:56.967245+0000","last_became_peered":"2024-01-16T14:59:56.967245+0000","last_unstale":"2024-01-16T15:00:00.206379+0000","last_undegraded":"2024-01-16T15:00:00.206379+0000","last_fullsized":"2024-01-16T15:00:00.206379+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:50.405789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.979264+0000","last_change":"2024-01-16T14:59:57.914280+0000","last_active":"2024-01-16T14:59:58.979264+0000","last_peered":"2024-01-16T14:59:58.979264+0000","last_clean":"2024-01-16T14:59:58.979264+0000","last_became_active":"2024-01-16T14:59:57.909069+0000","last_became_peered":"2024-01-16T14:59:57.909069+0000","last_unstale":"2024-01-16T14:59:58.979264+0000","last_undegraded":"2024-01-16T14:59:58.979264+0000","last_fullsized":"2024-01-16T14:59:58.979264+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:45:32.241461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-01-16T14:59:57.943628+0000","last_change":"2024-01-16T14:59:56.963783+0000","last_active":"2024-01-16T14:59:57.943628+0000","last_peered":"2024-01-16T14:59:57.943628+0000","last_clean":"2024-01-16T14:59:57.943628+0000","last_became_active":"2024-01-16T14:59:56.963167+0000","last_became_peered":"2024-01-16T14:59:56.963167+0000","last_unstale":"2024-01-16T14:59:57.943628+0000","last_undegraded":"2024-01-16T14:59:57.943628+0000","last_fullsized":"2024-01-16T14:59:57.943628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:39.450616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974544+0000","last_change":"2024-01-16T14:59:57.907693+0000","last_active":"2024-01-16T14:59:58.974544+0000","last_peered":"2024-01-16T14:59:58.974544+0000","last_clean":"2024-01-16T14:59:58.974544+0000","last_became_active":"2024-01-16T14:59:57.907365+0000","last_became_peered":"2024-01-16T14:59:57.907365+0000","last_unstale":"2024-01-16T14:59:58.974544+0000","last_undegraded":"2024-01-16T14:59:58.974544+0000","last_fullsized":"2024-01-16T14:59:58.974544+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:31.975241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881061+0000","last_change":"2024-01-16T14:59:56.964456+0000","last_active":"2024-01-16T14:59:58.881061+0000","last_peered":"2024-01-16T14:59:58.881061+0000","last_clean":"2024-01-16T14:59:58.881061+0000","last_became_active":"2024-01-16T14:59:56.964105+0000","last_became_peered":"2024-01-16T14:59:56.964105+0000","last_unstale":"2024-01-16T14:59:58.881061+0000","last_undegraded":"2024-01-16T14:59:58.881061+0000","last_fullsized":"2024-01-16T14:59:58.881061+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:57.519388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.884695+0000","last_change":"2024-01-16T14:59:57.909622+0000","last_active":"2024-01-16T14:59:58.884695+0000","last_peered":"2024-01-16T14:59:58.884695+0000","last_clean":"2024-01-16T14:59:58.884695+0000","last_became_active":"2024-01-16T14:59:57.905809+0000","last_became_peered":"2024-01-16T14:59:57.905809+0000","last_unstale":"2024-01-16T14:59:58.884695+0000","last_undegraded":"2024-01-16T14:59:58.884695+0000","last_fullsized":"2024-01-16T14:59:58.884695+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:30.677552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.964306+0000","last_change":"2024-01-16T14:59:56.961402+0000","last_active":"2024-01-16T14:59:58.964306+0000","last_peered":"2024-01-16T14:59:58.964306+0000","last_clean":"2024-01-16T14:59:58.964306+0000","last_became_active":"2024-01-16T14:59:56.959909+0000","last_became_peered":"2024-01-16T14:59:56.959909+0000","last_unstale":"2024-01-16T14:59:58.964306+0000","last_undegraded":"2024-01-16T14:59:58.964306+0000","last_fullsized":"2024-01-16T14:59:58.964306+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:53:00.188907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976651+0000","last_change":"2024-01-16T14:59:57.902379+0000","last_active":"2024-01-16T14:59:58.976651+0000","last_peered":"2024-01-16T14:59:58.976651+0000","last_clean":"2024-01-16T14:59:58.976651+0000","last_became_active":"2024-01-16T14:59:57.902002+0000","last_became_peered":"2024-01-16T14:59:57.902002+0000","last_unstale":"2024-01-16T14:59:58.976651+0000","last_undegraded":"2024-01-16T14:59:58.976651+0000","last_fullsized":"2024-01-16T14:59:58.976651+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.963988+0000","last_change":"2024-01-16T14:59:56.956035+0000","last_active":"2024-01-16T14:59:58.963988+0000","last_peered":"2024-01-16T14:59:58.963988+0000","last_clean":"2024-01-16T14:59:58.963988+0000","last_became_active":"2024-01-16T14:59:56.955666+0000","last_became_peered":"2024-01-16T14:59:56.955666+0000","last_unstale":"2024-01-16T14:59:58.963988+0000","last_undegraded":"2024-01-16T14:59:58.963988+0000","last_fullsized":"2024-01-16T14:59:58.963988+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:30:00.038659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977506+0000","last_change":"2024-01-16T14:59:57.913992+0000","last_active":"2024-01-16T14:59:58.977506+0000","last_peered":"2024-01-16T14:59:58.977506+0000","last_clean":"2024-01-16T14:59:58.977506+0000","last_became_active":"2024-01-16T14:59:57.913639+0000","last_became_peered":"2024-01-16T14:59:57.913639+0000","last_unstale":"2024-01-16T14:59:58.977506+0000","last_undegraded":"2024-01-16T14:59:58.977506+0000","last_fullsized":"2024-01-16T14:59:58.977506+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:26.399672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.882012+0000","last_change":"2024-01-16T14:59:56.970741+0000","last_active":"2024-01-16T14:59:58.882012+0000","last_peered":"2024-01-16T14:59:58.882012+0000","last_clean":"2024-01-16T14:59:58.882012+0000","last_became_active":"2024-01-16T14:59:56.970075+0000","last_became_peered":"2024-01-16T14:59:56.970075+0000","last_unstale":"2024-01-16T14:59:58.882012+0000","last_undegraded":"2024-01-16T14:59:58.882012+0000","last_fullsized":"2024-01-16T14:59:58.882012+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:21.992849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.974326+0000","last_change":"2024-01-16T14:59:57.906046+0000","last_active":"2024-01-16T14:59:58.974326+0000","last_peered":"2024-01-16T14:59:58.974326+0000","last_clean":"2024-01-16T14:59:58.974326+0000","last_became_active":"2024-01-16T14:59:57.905751+0000","last_became_peered":"2024-01-16T14:59:57.905751+0000","last_unstale":"2024-01-16T14:59:58.974326+0000","last_undegraded":"2024-01-16T14:59:58.974326+0000","last_fullsized":"2024-01-16T14:59:58.974326+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:10.998325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.976133+0000","last_change":"2024-01-16T14:59:57.920773+0000","last_active":"2024-01-16T14:59:58.976133+0000","last_peered":"2024-01-16T14:59:58.976133+0000","last_clean":"2024-01-16T14:59:58.976133+0000","last_became_active":"2024-01-16T14:59:57.920488+0000","last_became_peered":"2024-01-16T14:59:57.920488+0000","last_unstale":"2024-01-16T14:59:58.976133+0000","last_undegraded":"2024-01-16T14:59:58.976133+0000","last_fullsized":"2024-01-16T14:59:58.976133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:48:44.184020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881475+0000","last_change":"2024-01-16T14:59:56.968586+0000","last_active":"2024-01-16T14:59:58.881475+0000","last_peered":"2024-01-16T14:59:58.881475+0000","last_clean":"2024-01-16T14:59:58.881475+0000","last_became_active":"2024-01-16T14:59:56.967923+0000","last_became_peered":"2024-01-16T14:59:56.967923+0000","last_unstale":"2024-01-16T14:59:58.881475+0000","last_undegraded":"2024-01-16T14:59:58.881475+0000","last_fullsized":"2024-01-16T14:59:58.881475+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975571+0000","last_change":"2024-01-16T14:59:57.919788+0000","last_active":"2024-01-16T14:59:58.975571+0000","last_peered":"2024-01-16T14:59:58.975571+0000","last_clean":"2024-01-16T14:59:58.975571+0000","last_became_active":"2024-01-16T14:59:57.918939+0000","last_became_peered":"2024-01-16T14:59:57.918939+0000","last_unstale":"2024-01-16T14:59:58.975571+0000","last_undegraded":"2024-01-16T14:59:58.975571+0000","last_fullsized":"2024-01-16T14:59:58.975571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:13:06.992632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.881694+0000","last_change":"2024-01-16T14:59:56.969250+0000","last_active":"2024-01-16T14:59:58.881694+0000","last_peered":"2024-01-16T14:59:58.881694+0000","last_clean":"2024-01-16T14:59:58.881694+0000","last_became_active":"2024-01-16T14:59:56.968877+0000","last_became_peered":"2024-01-16T14:59:56.968877+0000","last_unstale":"2024-01-16T14:59:58.881694+0000","last_undegraded":"2024-01-16T14:59:58.881694+0000","last_fullsized":"2024-01-16T14:59:58.881694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:16:04.691942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975133+0000","last_change":"2024-01-16T14:59:57.909922+0000","last_active":"2024-01-16T14:59:58.975133+0000","last_peered":"2024-01-16T14:59:58.975133+0000","last_clean":"2024-01-16T14:59:58.975133+0000","last_became_active":"2024-01-16T14:59:57.908518+0000","last_became_peered":"2024-01-16T14:59:57.908518+0000","last_unstale":"2024-01-16T14:59:58.975133+0000","last_undegraded":"2024-01-16T14:59:58.975133+0000","last_fullsized":"2024-01-16T14:59:58.975133+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:36:54.103991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.978143+0000","last_change":"2024-01-16T14:59:56.956482+0000","last_active":"2024-01-16T14:59:58.978143+0000","last_peered":"2024-01-16T14:59:58.978143+0000","last_clean":"2024-01-16T14:59:58.978143+0000","last_became_active":"2024-01-16T14:59:56.955091+0000","last_became_peered":"2024-01-16T14:59:56.955091+0000","last_unstale":"2024-01-16T14:59:58.978143+0000","last_undegraded":"2024-01-16T14:59:58.978143+0000","last_fullsized":"2024-01-16T14:59:58.978143+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:50:27.880529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.977730+0000","last_change":"2024-01-16T14:59:57.913479+0000","last_active":"2024-01-16T14:59:58.977730+0000","last_peered":"2024-01-16T14:59:58.977730+0000","last_clean":"2024-01-16T14:59:58.977730+0000","last_became_active":"2024-01-16T14:59:57.911939+0000","last_became_peered":"2024-01-16T14:59:57.911939+0000","last_unstale":"2024-01-16T14:59:58.977730+0000","last_undegraded":"2024-01-16T14:59:58.977730+0000","last_fullsized":"2024-01-16T14:59:58.977730+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:56.917096+0000","last_clean_scrub_stamp":"2024-01-16T14:59:56.917096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:26.642001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2024-01-16T14:59:58.975233+0000","last_change":"2024-01-16T14:59:56.957873+0000","last_active":"2024-01-16T14:59:58.975233+0000","last_peered":"2024-01-16T14:59:58.975233+0000","last_clean":"2024-01-16T14:59:58.975233+0000","last_became_active":"2024-01-16T14:59:56.957287+0000","last_became_peered":"2024-01-16T14:59:56.957287+0000","last_unstale":"2024-01-16T14:59:58.975233+0000","last_undegraded":"2024-01-16T14:59:58.975233+0000","last_fullsized":"2024-01-16T14:59:58.975233+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:55.911644+0000","last_clean_scrub_stamp":"2024-01-16T14:59:55.911644+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:36.656624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509446,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1579999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.913}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1240000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1739999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.597}]}]},{"osd":4,"up_from":15,"seq":64424509448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":78294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2490000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.607}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5089999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]}]},{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":819200,"data_stored":669118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.294}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.552}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.349}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.494}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":225280,"data_stored":78258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.399}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0529999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.874}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.173}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.532}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4330000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542151,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31416,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340424,"statfs":{"total":96636764160,"available":96604594176,"internally_reserved":0,"allocated":233472,"data_stored":79780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94199999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.359}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3069999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.498}]}]},{"osd":2,"up_from":14,"seq":60129542150,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31396,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340444,"statfs":{"total":96636764160,"available":96604614656,"internally_reserved":0,"allocated":221184,"data_stored":78168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.365}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5069999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93500000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0229999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5880000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31404,"kb_used_data":224,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340436,"statfs":{"total":96636764160,"available":96604606464,"internally_reserved":0,"allocated":229376,"data_stored":78328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.95399999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0740000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4339999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.823}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.907}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.736}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.036}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31988,"kb_used_data":808,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339852,"statfs":{"total":96636764160,"available":96604008448,"internally_reserved":0,"allocated":827392,"data_stored":669174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4359999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.593}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.383}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.3700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5600000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0619999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0449999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:00:08.565 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T15:00:08.565 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-01-16T15:00:08.565 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-01-16T15:00:08.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-01-16T15:00:08.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.833+0000 7f1823600640 1 Processor -- start 2024-01-16T15:00:08.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.833+0000 7f1823600640 1 -- start start 2024-01-16T15:00:08.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1823600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f181c13df80 0x7f181c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1823600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0cf9a0 0x7f181c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1823600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f181c06c470 con 0x7f181c13e360 2024-01-16T15:00:08.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1823600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f181c06c750 con 0x7f181c13df80 2024-01-16T15:00:08.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1823600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c06ca30 con 0x7f181c0cf9a0 2024-01-16T15:00:08.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f181be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f181c13e360 0x7f181c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49842/0 (socket says 172.21.15.53:49842) 2024-01-16T15:00:08.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f181be00640 1 -- 172.21.15.53:0/2182773270 learned_addr learned my addr 172.21.15.53:0/2182773270 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:08.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1820c00640 1 --2- 172.21.15.53:0/2182773270 >> v2:172.21.15.146:3300/0 conn(0x7f181c13df80 0x7f181c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f1821600640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0cf9a0 0x7f181c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900174373 0 0) 0x7f181c06c470 con 0x7f181c13e360 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.835+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17fc003520 con 0x7f181c13e360 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3029338194 0 0) 0x7f17fc003520 con 0x7f181c13e360 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0cf9a0 msgr2=0x7f181c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0cf9a0 0x7f181c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 >> v2:172.21.15.146:3300/0 conn(0x7f181c13df80 msgr2=0x7f181c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 --2- 172.21.15.53:0/2182773270 >> v2:172.21.15.146:3300/0 conn(0x7f181c13df80 0x7f181c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c06cd10 con 0x7f181c13e360 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1804001670 con 0x7f181c13e360 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 >> v1:172.21.15.53:6789/0 conn(0x7f181c13e360 legacy=0x7f181c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc003970 0x7f17fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17fc0047c0 0x7f17fc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17fc005180 0x7f17fc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c06c750 con 0x7f17fc0047c0 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c06ca30 con 0x7f17fc003970 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17fc00bef0 con 0x7f17fc005180 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f1821600640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0cf9a0 0x7f181c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.836+0000 7f181be00640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17fc0047c0 0x7f17fc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f1820c00640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc003970 0x7f17fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f181be00640 1 -- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17fc005180 msgr2=0x7f17fc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f1821600640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17fc005180 0x7f17fc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f181be00640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f17fc005180 0x7f17fc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f181be00640 1 -- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc003970 msgr2=0x7f17fc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f181be00640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc003970 0x7f17fc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f181be00640 1 -- 172.21.15.53:0/2182773270 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17fc003520 con 0x7f17fc0047c0 2024-01-16T15:00:08.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.837+0000 7f1820c00640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc003970 0x7f17fc003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:08.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.838+0000 7f181be00640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17fc0047c0 0x7f17fc004c40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f180400c210 tx=0x7f17fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826c3098902666e8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.838+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804005ad0 con 0x7f17fc0047c0 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.838+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1804005c30 con 0x7f17fc0047c0 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.838+0000 7f181b400640 1 -- 172.21.15.53:0/2182773270 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180402c8c0 con 0x7f17fc0047c0 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1823600640 1 -- 172.21.15.53:0/2182773270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17fc0047c0 msgr2=0x7f17fc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1823600640 1 --2- 172.21.15.53:0/2182773270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f17fc0047c0 0x7f17fc004c40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f180400c210 tx=0x7f17fc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1821600640 1 -- 172.21.15.53:0/2182773270 reap_dead start 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1823600640 1 -- 172.21.15.53:0/2182773270 shutdown_connections 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1823600640 1 -- 172.21.15.53:0/2182773270 >> 172.21.15.53:0/2182773270 conn(0x7f181c06a800 msgr2=0x7f181c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1823600640 1 -- 172.21.15.53:0/2182773270 shutdown_connections 2024-01-16T15:00:08.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.839+0000 7f1823600640 1 -- 172.21.15.53:0/2182773270 wait complete. 2024-01-16T15:00:08.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.841+0000 7f1823600640 1 Processor -- start 2024-01-16T15:00:08.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.841+0000 7f1823600640 1 -- start start 2024-01-16T15:00:08.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f1823600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc0047c0 0x7f181c1f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f1823600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 0x7f181c1efd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f1823600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0b33a0 0x7f181c1f0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f1823600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c14d4c0 con 0x7f181c0b2f20 2024-01-16T15:00:08.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f1823600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f181c1f07c0 con 0x7f17fc0047c0 2024-01-16T15:00:08.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f1823600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f181c14c180 con 0x7f181c0b33a0 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.842+0000 7f181be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 0x7f181c1efd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 0x7f181c1efd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35210/0 (socket says 172.21.15.53:35210) 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 -- 172.21.15.53:0/1337831693 learned_addr learned my addr 172.21.15.53:0/1337831693 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f1821600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0b33a0 0x7f181c1f0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f1820c00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc0047c0 0x7f181c1f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 -- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0b33a0 msgr2=0x7f181c1f0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0b33a0 0x7f181c1f0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 -- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc0047c0 msgr2=0x7f181c1f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:08.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc0047c0 0x7f181c1f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 -- 172.21.15.53:0/1337831693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c1f1880 con 0x7f181c0b2f20 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.843+0000 7f181be00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 0x7f181c1efd40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f180402c4c0 tx=0x7f1804001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.844+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804039070 con 0x7f181c0b2f20 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.844+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180402ccd0 con 0x7f181c0b2f20 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.844+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f180403c040 con 0x7f181c0b2f20 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.844+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f181c14c420 con 0x7f181c0b2f20 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.844+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f181c14c9e0 con 0x7f181c0b2f20 2024-01-16T15:00:08.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.845+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f181c0675d0 con 0x7f181c0b2f20 2024-01-16T15:00:08.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.846+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1804032070 con 0x7f181c0b2f20 2024-01-16T15:00:08.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.847+0000 7f1818c00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f17f406b2c0 0x7f17f406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:08.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.847+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18040aafe0 con 0x7f181c0b2f20 2024-01-16T15:00:08.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.848+0000 7f1820c00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f17f406b2c0 0x7f17f406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:08.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.849+0000 7f1820c00640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f17f406b2c0 0x7f17f406d780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f181c0cc4a0 tx=0x7f180c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:08.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:08.852+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1804079970 con 0x7f181c0b2f20 2024-01-16T15:00:09.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.087+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f181c06df00 con 0x7f181c0b2f20 2024-01-16T15:00:09.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.088+0000 7f1818c00640 1 -- 172.21.15.53:0/1337831693 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f1804079500 con 0x7f181c0b2f20 2024-01-16T15:00:09.068 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:09.069 INFO:teuthology.orchestra.run.smithi053.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-01-16T15:00:09.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f17f406b2c0 msgr2=0x7f17f406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f17f406b2c0 0x7f17f406d780 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f181c0cc4a0 tx=0x7f180c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:09.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 msgr2=0x7f181c1efd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 0x7f181c1efd40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f180402c4c0 tx=0x7f1804001a00 comp rx=0 tx=0).stop 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 shutdown_connections 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f17f406b2c0 0x7f17f406d780 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f181c0b33a0 0x7f181c1f0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f181c0b2f20 0x7f181c1efd40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 --2- 172.21.15.53:0/1337831693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f17fc0047c0 0x7f181c1f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 >> 172.21.15.53:0/1337831693 conn(0x7f181c06a800 msgr2=0x7f181c154910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 shutdown_connections 2024-01-16T15:00:09.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.091+0000 7f1823600640 1 -- 172.21.15.53:0/1337831693 wait complete. 2024-01-16T15:00:09.090 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-01-16T15:00:09.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:09.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.322+0000 7f12ef400640 1 Processor -- start 2024-01-16T15:00:09.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.322+0000 7f12ef400640 1 -- start start 2024-01-16T15:00:09.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ef400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12e80d0100 0x7f12e80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ef400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 0x7f12e80d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ef400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12e80d0a10 con 0x7f12e8160920 2024-01-16T15:00:09.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ef400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12e80cf080 con 0x7f12e80d0100 2024-01-16T15:00:09.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ef400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e80c6be0 con 0x7f12e80d0f40 2024-01-16T15:00:09.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ee000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f12e8160920 0x7f12e80cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49844/0 (socket says 172.21.15.53:49844) 2024-01-16T15:00:09.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ee000640 1 -- 172.21.15.53:0/3078356688 learned_addr learned my addr 172.21.15.53:0/3078356688 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:09.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12ed600640 1 --2- 172.21.15.53:0/3078356688 >> v2:172.21.15.146:3300/0 conn(0x7f12e80d0100 0x7f12e80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.324+0000 7f12eea00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 0x7f12e80d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1818357509 0 0) 0x7f12e80d0a10 con 0x7f12e8160920 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c8003520 con 0x7f12e8160920 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 101436423 0 0) 0x7f12c8003520 con 0x7f12e8160920 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 msgr2=0x7f12e80d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 0x7f12e80d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 >> v2:172.21.15.146:3300/0 conn(0x7f12e80d0100 msgr2=0x7f12e80b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 --2- 172.21.15.53:0/3078356688 >> v2:172.21.15.146:3300/0 conn(0x7f12e80d0100 0x7f12e80b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e80ce7e0 con 0x7f12e8160920 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f12d8001670 con 0x7f12e8160920 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 >> v1:172.21.15.53:6789/0 conn(0x7f12e8160920 legacy=0x7f12e80cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c8003970 0x7f12c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 0x7f12c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c8004d00 0x7f12c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e80cf080 con 0x7f12c8004d00 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e80c6be0 con 0x7f12c8003970 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12c800b980 con 0x7f12e80d0f40 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.325+0000 7f12eea00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c8004d00 0x7f12c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12ee000640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c8003970 0x7f12c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12ed600640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 0x7f12c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12eea00640 1 -- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 msgr2=0x7f12c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12eea00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12e80d0f40 0x7f12c80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12eea00640 1 -- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c8003970 msgr2=0x7f12c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12eea00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12c8003970 0x7f12c8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.326+0000 7f12eea00640 1 -- 172.21.15.53:0/3078356688 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c8003520 con 0x7f12c8004d00 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.327+0000 7f12eea00640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c8004d00 0x7f12c8005180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009fc0 tx=0x7f12c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5cfcd42e41d660d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.327+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e002cc20 con 0x7f12c8004d00 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.327+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e000f040 con 0x7f12c8004d00 2024-01-16T15:00:09.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.327+0000 7f12ecc00640 1 -- 172.21.15.53:0/3078356688 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0034440 con 0x7f12c8004d00 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12ef400640 1 -- 172.21.15.53:0/3078356688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c8004d00 msgr2=0x7f12c8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12ef400640 1 --2- 172.21.15.53:0/3078356688 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12c8004d00 0x7f12c8005180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f12e0009fc0 tx=0x7f12c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12eea00640 1 -- 172.21.15.53:0/3078356688 reap_dead start 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12ef400640 1 -- 172.21.15.53:0/3078356688 shutdown_connections 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12ef400640 1 -- 172.21.15.53:0/3078356688 >> 172.21.15.53:0/3078356688 conn(0x7f12e80d3090 msgr2=0x7f12e80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12ef400640 1 -- 172.21.15.53:0/3078356688 shutdown_connections 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.328+0000 7f12ef400640 1 -- 172.21.15.53:0/3078356688 wait complete. 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.330+0000 7f12ef400640 1 Processor -- start 2024-01-16T15:00:09.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.330+0000 7f12ef400640 1 -- start start 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ef400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12c8004d00 0x7f12e815c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ef400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e80b5c50 0x7f12e8157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 0x7f12e80c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ef400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e80cd2c0 con 0x7f12e80c9670 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ef400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e815cc70 con 0x7f12e80b5c50 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ef400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12e815cf50 con 0x7f12c8004d00 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 0x7f12e80c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 0x7f12e80c8a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35236/0 (socket says 172.21.15.53:35236) 2024-01-16T15:00:09.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12eea00640 1 -- 172.21.15.53:0/1623104291 learned_addr learned my addr 172.21.15.53:0/1623104291 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ee000640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12c8004d00 0x7f12e815c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.331+0000 7f12ed600640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e80b5c50 0x7f12e8157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12eea00640 1 -- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12c8004d00 msgr2=0x7f12e815c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12eea00640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12c8004d00 0x7f12e815c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12eea00640 1 -- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e80b5c50 msgr2=0x7f12e8157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12eea00640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e80b5c50 0x7f12e8157320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12eea00640 1 -- 172.21.15.53:0/1623104291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e815d230 con 0x7f12e80c9670 2024-01-16T15:00:09.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12eea00640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 0x7f12e80c8a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12e0002230 tx=0x7f12e0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0034ad0 con 0x7f12e80c9670 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0034c30 con 0x7f12e80c9670 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e815d510 con 0x7f12e80c9670 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.332+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0003990 con 0x7f12e80c9670 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.333+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e8157860 con 0x7f12e80c9670 2024-01-16T15:00:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.334+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e815da30 con 0x7f12e80c9670 2024-01-16T15:00:09.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.335+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f12e003e440 con 0x7f12e80c9670 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.335+0000 7f12e6000640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f12c006b2f0 0x7f12c006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.336+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f12e00ab970 con 0x7f12e80c9670 2024-01-16T15:00:09.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.336+0000 7f12ee000640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f12c006b2f0 0x7f12c006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.337+0000 7f12ee000640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f12c006b2f0 0x7f12c006d7b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f12d8001050 tx=0x7f12d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.340+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12e815da30 con 0x7f12e80c9670 2024-01-16T15:00:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.537+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12e80cf1c0 con 0x7f12e80c9670 2024-01-16T15:00:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.539+0000 7f12e6000640 1 -- 172.21.15.53:0/1623104291 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f12e0039090 con 0x7f12e80c9670 2024-01-16T15:00:09.520 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:09.520 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f12c006b2f0 msgr2=0x7f12c006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f12c006b2f0 0x7f12c006d7b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f12d8001050 tx=0x7f12d8009040 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 msgr2=0x7f12e80c8a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 0x7f12e80c8a20 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f12e0002230 tx=0x7f12e0009750 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 shutdown_connections 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f12c006b2f0 0x7f12c006d7b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12e80c9670 0x7f12e80c8a20 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12e80b5c50 0x7f12e8157320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 --2- 172.21.15.53:0/1623104291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12c8004d00 0x7f12e815c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 >> 172.21.15.53:0/1623104291 conn(0x7f12e80d3090 msgr2=0x7f12e806c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.542+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 shutdown_connections 2024-01-16T15:00:09.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.543+0000 7f12ef400640 1 -- 172.21.15.53:0/1623104291 wait complete. 2024-01-16T15:00:09.543 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:09.543 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.789+0000 7f78be000640 1 Processor -- start 2024-01-16T15:00:09.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.789+0000 7f78be000640 1 -- start start 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.791+0000 7f78be000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f78b813df80 0x7f78b815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.791+0000 7f78be000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78b80cf9a0 0x7f78b80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.791+0000 7f78be000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78b806c470 con 0x7f78b813e360 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.791+0000 7f78be000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b806c750 con 0x7f78b80cf9a0 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.791+0000 7f78be000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78b806ca30 con 0x7f78b813df80 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78b80cf9a0 0x7f78b80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f78b80cf9a0 0x7f78b80cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35246/0 (socket says 172.21.15.53:35246) 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 -- 172.21.15.53:0/3214912399 learned_addr learned my addr 172.21.15.53:0/3214912399 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78afe00640 1 --1- 172.21.15.53:0/3214912399 >> v1:172.21.15.53:6789/0 conn(0x7f78b813e360 0x7f78b80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49848/0 (socket says 172.21.15.53:49848) 2024-01-16T15:00:09.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7400640 1 --2- 172.21.15.53:0/3214912399 >> v2:172.21.15.146:3300/0 conn(0x7f78b813df80 0x7f78b815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 -- 172.21.15.53:0/3214912399 >> v2:172.21.15.146:3300/0 conn(0x7f78b813df80 msgr2=0x7f78b815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:09.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 --2- 172.21.15.53:0/3214912399 >> v2:172.21.15.146:3300/0 conn(0x7f78b813df80 0x7f78b815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 -- 172.21.15.53:0/3214912399 >> v1:172.21.15.53:6789/0 conn(0x7f78b813e360 legacy=0x7f78b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.792+0000 7f78b7e00640 1 -- 172.21.15.53:0/3214912399 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b806cd10 con 0x7f78b80cf9a0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.793+0000 7f78b7e00640 1 --2- 172.21.15.53:0/3214912399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b80cd840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f78b8072770 tx=0x7f78b002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5a155e2af9957e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78b6a00640 1 -- 172.21.15.53:0/3214912399 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b002cb10 con 0x7f78b80cf9a0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78b6a00640 1 -- 172.21.15.53:0/3214912399 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b002cc70 con 0x7f78b80cf9a0 2024-01-16T15:00:09.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78b6a00640 1 -- 172.21.15.53:0/3214912399 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b00343f0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 -- 172.21.15.53:0/3214912399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 msgr2=0x7f78b80cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 --2- 172.21.15.53:0/3214912399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b80cd840 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f78b8072770 tx=0x7f78b002bf70 comp rx=0 tx=0).stop 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 -- 172.21.15.53:0/3214912399 shutdown_connections 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 --2- 172.21.15.53:0/3214912399 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b80cd840 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 --2- 172.21.15.53:0/3214912399 >> v2:172.21.15.146:3300/0 conn(0x7f78b813df80 0x7f78b815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 -- 172.21.15.53:0/3214912399 >> 172.21.15.53:0/3214912399 conn(0x7f78b806a800 msgr2=0x7f78b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 -- 172.21.15.53:0/3214912399 shutdown_connections 2024-01-16T15:00:09.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.794+0000 7f78be000640 1 -- 172.21.15.53:0/3214912399 wait complete. 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.796+0000 7f78be000640 1 Processor -- start 2024-01-16T15:00:09.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.796+0000 7f78be000640 1 -- start start 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78be000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b8152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78be000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78b813df80 0x7f78b81531a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78be000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b813e360 0x7f78b814c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78be000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b813dbc0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78be000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b814afe0 con 0x7f78b813e360 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78be000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78b814b2c0 con 0x7f78b813df80 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b8152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b8152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35250/0 (socket says 172.21.15.53:35250) 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.797+0000 7f78b7400640 1 -- 172.21.15.53:0/3358646336 learned_addr learned my addr 172.21.15.53:0/3358646336 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78afe00640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78b813df80 0x7f78b81531a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7e00640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b813e360 0x7f78b814c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7400640 1 -- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78b813df80 msgr2=0x7f78b81531a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7400640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78b813df80 0x7f78b81531a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7400640 1 -- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b813e360 msgr2=0x7f78b814c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7400640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b813e360 0x7f78b814c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7400640 1 -- 172.21.15.53:0/3358646336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b0009b30 con 0x7f78b80cf9a0 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7e00640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b813e360 0x7f78b814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78b7400640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b8152c60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f78a800d800 tx=0x7f78a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a800b6c0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78a800b820 con 0x7f78b80cf9a0 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a800bbb0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.798+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b814b5a0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.799+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b814bb60 con 0x7f78b80cf9a0 2024-01-16T15:00:09.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.800+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78b80675d0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.801+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f78a800bd10 con 0x7f78b80cf9a0 2024-01-16T15:00:09.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.801+0000 7f78af400640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f789006b2c0 0x7f789006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:09.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.802+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f78a808a2a0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.802+0000 7f78afe00640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f789006b2c0 0x7f789006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:09.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.803+0000 7f78afe00640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f789006b2c0 0x7f789006d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f78b813d7b0 tx=0x7f78a0007c80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:09.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:09.806+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78a8054fe0 con 0x7f78b80cf9a0 2024-01-16T15:00:09.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.000+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f78b806df00 con 0x7f78b80cf9a0 2024-01-16T15:00:09.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.001+0000 7f78af400640 1 -- 172.21.15.53:0/3358646336 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f78a8058600 con 0x7f78b80cf9a0 2024-01-16T15:00:09.982 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f789006b2c0 msgr2=0x7f789006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f789006b2c0 0x7f789006d780 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f78b813d7b0 tx=0x7f78a0007c80 comp rx=0 tx=0).stop 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 msgr2=0x7f78b8152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b8152c60 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f78a800d800 tx=0x7f78a800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 shutdown_connections 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f789006b2c0 0x7f789006d780 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78b813e360 0x7f78b814c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78b813df80 0x7f78b81531a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 --2- 172.21.15.53:0/3358646336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78b80cf9a0 0x7f78b8152c60 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 >> 172.21.15.53:0/3358646336 conn(0x7f78b806a800 msgr2=0x7f78b815d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 shutdown_connections 2024-01-16T15:00:09.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.004+0000 7f78be000640 1 -- 172.21.15.53:0/3358646336 wait complete. 2024-01-16T15:00:10.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:10.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.248+0000 7f8912200640 1 Processor -- start 2024-01-16T15:00:10.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.248+0000 7f8912200640 1 -- start start 2024-01-16T15:00:10.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f8912200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f890c13df80 0x7f890c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f8912200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f890c0cf9a0 0x7f890c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f8912200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f890c06c470 con 0x7f890c13e360 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f8912200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c06c750 con 0x7f890c13df80 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f8912200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f890c06ca30 con 0x7f890c0cf9a0 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f890b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f890c13df80 0x7f890c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f890b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f890c13df80 0x7f890c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35258/0 (socket says 172.21.15.53:35258) 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f890b400640 1 -- 172.21.15.53:0/2777636860 learned_addr learned my addr 172.21.15.53:0/2777636860 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.250+0000 7f8902800640 1 --1- 172.21.15.53:0/2777636860 >> v1:172.21.15.53:6789/0 conn(0x7f890c13e360 0x7f890c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49862/0 (socket says 172.21.15.53:49862) 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.251+0000 7f890be00640 1 --2- 172.21.15.53:0/2777636860 >> v2:172.21.15.146:3300/0 conn(0x7f890c0cf9a0 0x7f890c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.251+0000 7f890b400640 1 -- 172.21.15.53:0/2777636860 >> v2:172.21.15.146:3300/0 conn(0x7f890c0cf9a0 msgr2=0x7f890c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.251+0000 7f890b400640 1 --2- 172.21.15.53:0/2777636860 >> v2:172.21.15.146:3300/0 conn(0x7f890c0cf9a0 0x7f890c0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.251+0000 7f890b400640 1 -- 172.21.15.53:0/2777636860 >> v1:172.21.15.53:6789/0 conn(0x7f890c13e360 legacy=0x7f890c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.251+0000 7f890b400640 1 -- 172.21.15.53:0/2777636860 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f890c06cd10 con 0x7f890c13df80 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.251+0000 7f890aa00640 1 -- 172.21.15.53:0/2777636860 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242368055 0 0) 0x7f890c06c470 con 0x7f890c13e360 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.252+0000 7f890b400640 1 --2- 172.21.15.53:0/2777636860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c13df80 0x7f890c15bee0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f890c0ccb10 tx=0x7f88fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=228e9ab2f662f77d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f890aa00640 1 -- 172.21.15.53:0/2777636860 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc030860 con 0x7f890c13df80 2024-01-16T15:00:10.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f890aa00640 1 -- 172.21.15.53:0/2777636860 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc0309c0 con 0x7f890c13df80 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f890aa00640 1 -- 172.21.15.53:0/2777636860 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc030d50 con 0x7f890c13df80 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f8912200640 1 -- 172.21.15.53:0/2777636860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c13df80 msgr2=0x7f890c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f8912200640 1 --2- 172.21.15.53:0/2777636860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c13df80 0x7f890c15bee0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f890c0ccb10 tx=0x7f88fc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f8912200640 1 -- 172.21.15.53:0/2777636860 shutdown_connections 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f8912200640 1 --2- 172.21.15.53:0/2777636860 >> v2:172.21.15.146:3300/0 conn(0x7f890c0cf9a0 0x7f890c0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f8912200640 1 --2- 172.21.15.53:0/2777636860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c13df80 0x7f890c15bee0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.253+0000 7f8912200640 1 -- 172.21.15.53:0/2777636860 >> 172.21.15.53:0/2777636860 conn(0x7f890c06a800 msgr2=0x7f890c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.254+0000 7f8912200640 1 -- 172.21.15.53:0/2777636860 shutdown_connections 2024-01-16T15:00:10.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.254+0000 7f8912200640 1 -- 172.21.15.53:0/2777636860 wait complete. 2024-01-16T15:00:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.255+0000 7f8912200640 1 Processor -- start 2024-01-16T15:00:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.255+0000 7f8912200640 1 -- start start 2024-01-16T15:00:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.256+0000 7f8912200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 0x7f890c159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.256+0000 7f8912200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f890c13df80 0x7f890c159f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.256+0000 7f8912200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f890c13e360 0x7f890c14bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f8912200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c14aa20 con 0x7f890c0cf9a0 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f8912200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890c1f2f00 con 0x7f890c13e360 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f8912200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f890c14ac30 con 0x7f890c13df80 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 0x7f890c159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 0x7f890c159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35272/0 (socket says 172.21.15.53:35272) 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 -- 172.21.15.53:0/444573297 learned_addr learned my addr 172.21.15.53:0/444573297 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890be00640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f890c13e360 0x7f890c14bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f8902800640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f890c13df80 0x7f890c159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 -- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f890c13df80 msgr2=0x7f890c159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f890c13df80 0x7f890c159f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 -- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f890c13e360 msgr2=0x7f890c14bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f890c13e360 0x7f890c14bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890b400640 1 -- 172.21.15.53:0/444573297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88fc0094d0 con 0x7f890c0cf9a0 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.257+0000 7f890be00640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f890c13e360 0x7f890c14bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.258+0000 7f890b400640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 0x7f890c159a30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f88fc000c00 tx=0x7f88fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.258+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc039c90 con 0x7f890c0cf9a0 2024-01-16T15:00:10.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.258+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88fc039df0 con 0x7f890c0cf9a0 2024-01-16T15:00:10.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.258+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88fc043620 con 0x7f890c0cf9a0 2024-01-16T15:00:10.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.258+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f890c14af10 con 0x7f890c0cf9a0 2024-01-16T15:00:10.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.258+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890c14b4d0 con 0x7f890c0cf9a0 2024-01-16T15:00:10.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.259+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f890c0675d0 con 0x7f890c0cf9a0 2024-01-16T15:00:10.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.260+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f88fc043780 con 0x7f890c0cf9a0 2024-01-16T15:00:10.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.261+0000 7f8903e00640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88e406b2c0 0x7f88e406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.261+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f88fc03e070 con 0x7f890c0cf9a0 2024-01-16T15:00:10.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.262+0000 7f8902800640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88e406b2c0 0x7f88e406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.263+0000 7f8902800640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88e406b2c0 0x7f88e406d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88f4007fd0 tx=0x7f88f4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.266+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88fc077bd0 con 0x7f890c0cf9a0 2024-01-16T15:00:10.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.460+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f890c14bb50 con 0x7f890c0cf9a0 2024-01-16T15:00:10.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.461+0000 7f8903e00640 1 -- 172.21.15.53:0/444573297 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f88fc0340a0 con 0x7f890c0cf9a0 2024-01-16T15:00:10.441 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:10.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88e406b2c0 msgr2=0x7f88e406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88e406b2c0 0x7f88e406d780 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88f4007fd0 tx=0x7f88f4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:10.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 msgr2=0x7f890c159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 0x7f890c159a30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f88fc000c00 tx=0x7f88fc004730 comp rx=0 tx=0).stop 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 shutdown_connections 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88e406b2c0 0x7f88e406d780 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f890c13e360 0x7f890c14bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f890c13df80 0x7f890c159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 --2- 172.21.15.53:0/444573297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f890c0cf9a0 0x7f890c159a30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.463+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 >> 172.21.15.53:0/444573297 conn(0x7f890c06a800 msgr2=0x7f890c15d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.464+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 shutdown_connections 2024-01-16T15:00:10.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.464+0000 7f8912200640 1 -- 172.21.15.53:0/444573297 wait complete. 2024-01-16T15:00:10.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:10.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.700+0000 7fa62f000640 1 Processor -- start 2024-01-16T15:00:10.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.700+0000 7fa62f000640 1 -- start start 2024-01-16T15:00:10.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa6280ccb50 0x7fa628070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62f000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa62815bf10 0x7fa628143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62f000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62815f4e0 con 0x7fa6280adf20 2024-01-16T15:00:10.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62f000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa62815f970 con 0x7fa62815bf10 2024-01-16T15:00:10.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa628071130 con 0x7fa6280ccb50 2024-01-16T15:00:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62e600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa62815bf10 0x7fa628143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62dc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa6280adf20 0x7fa6280cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49866/0 (socket says 172.21.15.53:49866) 2024-01-16T15:00:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.702+0000 7fa62dc00640 1 -- 172.21.15.53:0/2933461255 learned_addr learned my addr 172.21.15.53:0/2933461255 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.703+0000 7fa62d200640 1 --2- 172.21.15.53:0/2933461255 >> v2:172.21.15.146:3300/0 conn(0x7fa6280ccb50 0x7fa628070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.703+0000 7fa62e600640 1 -- 172.21.15.53:0/2933461255 >> v2:172.21.15.146:3300/0 conn(0x7fa6280ccb50 msgr2=0x7fa628070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:10.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.703+0000 7fa62e600640 1 --2- 172.21.15.53:0/2933461255 >> v2:172.21.15.146:3300/0 conn(0x7fa6280ccb50 0x7fa628070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.703+0000 7fa62e600640 1 -- 172.21.15.53:0/2933461255 >> v1:172.21.15.53:6789/0 conn(0x7fa6280adf20 legacy=0x7fa6280cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.703+0000 7fa62e600640 1 -- 172.21.15.53:0/2933461255 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa62815ebb0 con 0x7fa62815bf10 2024-01-16T15:00:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.703+0000 7fa627e00640 1 -- 172.21.15.53:0/2933461255 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154425771 0 0) 0x7fa62815f4e0 con 0x7fa6280adf20 2024-01-16T15:00:10.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.704+0000 7fa62e600640 1 --2- 172.21.15.53:0/2933461255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa628143d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa62815e9e0 tx=0x7fa62002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3507d3725579bf3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.704+0000 7fa627e00640 1 -- 172.21.15.53:0/2933461255 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62002e070 con 0x7fa62815bf10 2024-01-16T15:00:10.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.704+0000 7fa627e00640 1 -- 172.21.15.53:0/2933461255 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6200315a0 con 0x7fa62815bf10 2024-01-16T15:00:10.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.704+0000 7fa627e00640 1 -- 172.21.15.53:0/2933461255 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa620031930 con 0x7fa62815bf10 2024-01-16T15:00:10.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 -- 172.21.15.53:0/2933461255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 msgr2=0x7fa628143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 --2- 172.21.15.53:0/2933461255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa628143d30 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa62815e9e0 tx=0x7fa62002b840 comp rx=0 tx=0).stop 2024-01-16T15:00:10.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 -- 172.21.15.53:0/2933461255 shutdown_connections 2024-01-16T15:00:10.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 --2- 172.21.15.53:0/2933461255 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa628143d30 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 --2- 172.21.15.53:0/2933461255 >> v2:172.21.15.146:3300/0 conn(0x7fa6280ccb50 0x7fa628070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 -- 172.21.15.53:0/2933461255 >> 172.21.15.53:0/2933461255 conn(0x7fa62806a7d0 msgr2=0x7fa62806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 -- 172.21.15.53:0/2933461255 shutdown_connections 2024-01-16T15:00:10.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.705+0000 7fa62f000640 1 -- 172.21.15.53:0/2933461255 wait complete. 2024-01-16T15:00:10.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.707+0000 7fa62f000640 1 Processor -- start 2024-01-16T15:00:10.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.707+0000 7fa62f000640 1 -- start start 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6280adf20 0x7fa6280c5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6280ccb50 0x7fa6280c6470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62f000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa6280ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62f000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6280b2dd0 con 0x7fa62815bf10 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6280b21c0 con 0x7fa6280ccb50 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6280b24a0 con 0x7fa6280adf20 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa6280ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa6280ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35290/0 (socket says 172.21.15.53:35290) 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62e600640 1 -- 172.21.15.53:0/3047844049 learned_addr learned my addr 172.21.15.53:0/3047844049 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:10.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.708+0000 7fa62dc00640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6280adf20 0x7fa6280c5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62d200640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6280ccb50 0x7fa6280c6470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62e600640 1 -- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6280adf20 msgr2=0x7fa6280c5f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62e600640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6280adf20 0x7fa6280c5f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62e600640 1 -- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6280ccb50 msgr2=0x7fa6280c6470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62e600640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6280ccb50 0x7fa6280c6470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62e600640 1 -- 172.21.15.53:0/3047844049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620009400 con 0x7fa62815bf10 2024-01-16T15:00:10.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62e600640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa6280ac8d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa620034040 tx=0x7fa6200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa620032840 con 0x7fa62815bf10 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6200329c0 con 0x7fa62815bf10 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa620032d70 con 0x7fa62815bf10 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.709+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6280b2780 con 0x7fa62815bf10 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.710+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6280b1e90 con 0x7fa62815bf10 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.711+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6280b2f40 con 0x7fa62815bf10 2024-01-16T15:00:10.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.711+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa6200453f0 con 0x7fa62815bf10 2024-01-16T15:00:10.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.712+0000 7fa625600640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa60006b2c0 0x7fa60006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:10.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.712+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6200abdc0 con 0x7fa62815bf10 2024-01-16T15:00:10.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.713+0000 7fa62dc00640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa60006b2c0 0x7fa60006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:10.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.714+0000 7fa62dc00640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa60006b2c0 0x7fa60006d780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6180012e0 tx=0x7fa618007ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:10.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.717+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa620076cb0 con 0x7fa62815bf10 2024-01-16T15:00:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.908+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa628067450 con 0x7fa62815bf10 2024-01-16T15:00:10.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.909+0000 7fa625600640 1 -- 172.21.15.53:0/3047844049 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa62007a2d0 con 0x7fa62815bf10 2024-01-16T15:00:10.890 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa60006b2c0 msgr2=0x7fa60006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa60006b2c0 0x7fa60006d780 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa6180012e0 tx=0x7fa618007ed0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 msgr2=0x7fa6280ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa6280ac8d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa620034040 tx=0x7fa6200046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 shutdown_connections 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa60006b2c0 0x7fa60006d780 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa62815bf10 0x7fa6280ac8d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6280ccb50 0x7fa6280c6470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 --2- 172.21.15.53:0/3047844049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6280adf20 0x7fa6280c5f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 >> 172.21.15.53:0/3047844049 conn(0x7fa62806a7d0 msgr2=0x7fa62806d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 shutdown_connections 2024-01-16T15:00:10.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:10.913+0000 7fa62f000640 1 -- 172.21.15.53:0/3047844049 wait complete. 2024-01-16T15:00:10.910 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:11.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.129+0000 7f32f7400640 1 Processor -- start 2024-01-16T15:00:11.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.130+0000 7f32f7400640 1 -- start start 2024-01-16T15:00:11.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.131+0000 7f32f7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f32f0160920 0x7f32f00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.131+0000 7f32f7400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f00d0a10 con 0x7f32f00cfea0 2024-01-16T15:00:11.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.131+0000 7f32f7400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f0161e30 con 0x7f32f00cf9f0 2024-01-16T15:00:11.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.131+0000 7f32f7400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f00d1f50 con 0x7f32f0160920 2024-01-16T15:00:11.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.132+0000 7f32f6a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f32f00cfea0 0x7f32f0161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49868/0 (socket says 172.21.15.53:49868) 2024-01-16T15:00:11.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.132+0000 7f32f6a00640 1 -- 172.21.15.53:0/2525060772 learned_addr learned my addr 172.21.15.53:0/2525060772 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:11.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.132+0000 7f32f6000640 1 --2- 172.21.15.53:0/2525060772 >> v2:172.21.15.146:3300/0 conn(0x7f32f0160920 0x7f32f00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.132+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012869052 0 0) 0x7f32f00d0a10 con 0x7f32f00cfea0 2024-01-16T15:00:11.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.132+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32d0003520 con 0x7f32f00cfea0 2024-01-16T15:00:11.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3203658226 0 0) 0x7f32d0003520 con 0x7f32f00cfea0 2024-01-16T15:00:11.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 >> v2:172.21.15.146:3300/0 conn(0x7f32f0160920 msgr2=0x7f32f00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 --2- 172.21.15.53:0/2525060772 >> v2:172.21.15.146:3300/0 conn(0x7f32f0160920 0x7f32f00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 >> v1:172.21.15.146:6789/0 conn(0x7f32f00cf9f0 legacy=0x7f32f0161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f00cf080 con 0x7f32f00cfea0 2024-01-16T15:00:11.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f32e80019f0 con 0x7f32f00cfea0 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 >> v1:172.21.15.53:6789/0 conn(0x7f32f00cfea0 legacy=0x7f32f0161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0003970 0x7f32d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32d00047c0 0x7f32d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32d0005180 0x7f32d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00d1f50 con 0x7f32d0005180 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d000c020 con 0x7f32d00047c0 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d000c1e0 con 0x7f32d0003970 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f6a00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32d0005180 0x7f32d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f5600640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32d00047c0 0x7f32d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.133+0000 7f32f6000640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0003970 0x7f32d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.134+0000 7f32f6a00640 1 -- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0003970 msgr2=0x7f32d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.134+0000 7f32f6a00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0003970 0x7f32d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.134+0000 7f32f6a00640 1 -- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32d00047c0 msgr2=0x7f32d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.134+0000 7f32f6a00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32d00047c0 0x7f32d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.134+0000 7f32f6a00640 1 -- 172.21.15.53:0/2525060772 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d0003520 con 0x7f32d0005180 2024-01-16T15:00:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.135+0000 7f32f6a00640 1 -- 172.21.15.53:0/2525060772 reap_dead start 2024-01-16T15:00:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.135+0000 7f32f6a00640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32d0005180 0x7f32d000b980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f32e800a830 tx=0x7f32d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64662a824d4a7ece server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.135+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80072d0 con 0x7f32d0005180 2024-01-16T15:00:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.135+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e802f070 con 0x7f32d0005180 2024-01-16T15:00:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.135+0000 7f32f4c00640 1 -- 172.21.15.53:0/2525060772 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80076b0 con 0x7f32d0005180 2024-01-16T15:00:11.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 -- 172.21.15.53:0/2525060772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32d0005180 msgr2=0x7f32d000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32d0005180 0x7f32d000b980 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f32e800a830 tx=0x7f32d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 -- 172.21.15.53:0/2525060772 shutdown_connections 2024-01-16T15:00:11.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 --2- 172.21.15.53:0/2525060772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32d0005180 0x7f32d000b980 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 -- 172.21.15.53:0/2525060772 >> 172.21.15.53:0/2525060772 conn(0x7f32f00d3090 msgr2=0x7f32f00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 -- 172.21.15.53:0/2525060772 shutdown_connections 2024-01-16T15:00:11.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.136+0000 7f32f7400640 1 -- 172.21.15.53:0/2525060772 wait complete. 2024-01-16T15:00:11.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.138+0000 7f32f7400640 1 Processor -- start 2024-01-16T15:00:11.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.138+0000 7f32f7400640 1 -- start start 2024-01-16T15:00:11.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0005180 0x7f32f015fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015d1a0 0x7f32f00b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 0x7f32f01f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f7400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00b8680 con 0x7f32f00cc330 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f7400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00b3a00 con 0x7f32f015d1a0 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f7400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f015d720 con 0x7f32d0005180 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 0x7f32f01f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 0x7f32f01f6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35322/0 (socket says 172.21.15.53:35322) 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f6a00640 1 -- 172.21.15.53:0/3506939835 learned_addr learned my addr 172.21.15.53:0/3506939835 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f5600640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015d1a0 0x7f32f00b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.139+0000 7f32f6000640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0005180 0x7f32f015fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6a00640 1 -- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0005180 msgr2=0x7f32f015fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6a00640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0005180 0x7f32f015fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6a00640 1 -- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015d1a0 msgr2=0x7f32f00b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6a00640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015d1a0 0x7f32f00b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6a00640 1 -- 172.21.15.53:0/3506939835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f015ee80 con 0x7f32f00cc330 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6000640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0005180 0x7f32f015fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:11.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f6a00640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 0x7f32f01f6490 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f32e802cc40 tx=0x7f32e8001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80019b0 con 0x7f32f00cc330 2024-01-16T15:00:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e8004b60 con 0x7f32f00cc330 2024-01-16T15:00:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8007450 con 0x7f32f00cc330 2024-01-16T15:00:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f015f160 con 0x7f32f00cc330 2024-01-16T15:00:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.140+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f015f720 con 0x7f32f00cc330 2024-01-16T15:00:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.142+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f015f940 con 0x7f32f00cc330 2024-01-16T15:00:11.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.143+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f32e8031030 con 0x7f32f00cc330 2024-01-16T15:00:11.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.144+0000 7f32ee000640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806b2c0 0x7f32c806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.144+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f32e80ab080 con 0x7f32f00cc330 2024-01-16T15:00:11.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.144+0000 7f32f6000640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806b2c0 0x7f32c806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.145+0000 7f32f6000640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806b2c0 0x7f32c806d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009980 tx=0x7f32d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.148+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32e8075f00 con 0x7f32f00cc330 2024-01-16T15:00:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.382+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32f0067450 con 0x7f32f00cc330 2024-01-16T15:00:11.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.384+0000 7f32ee000640 1 -- 172.21.15.53:0/3506939835 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f32e8079520 con 0x7f32f00cc330 2024-01-16T15:00:11.364 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:11.364 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:11.364 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806b2c0 msgr2=0x7f32c806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806b2c0 0x7f32c806d780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009980 tx=0x7f32d8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 msgr2=0x7f32f01f6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 0x7f32f01f6490 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f32e802cc40 tx=0x7f32e8001d80 comp rx=0 tx=0).stop 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 shutdown_connections 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806b2c0 0x7f32c806d780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00cc330 0x7f32f01f6490 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015d1a0 0x7f32f00b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 --2- 172.21.15.53:0/3506939835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32d0005180 0x7f32f015fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.386+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 >> 172.21.15.53:0/3506939835 conn(0x7f32f00d3090 msgr2=0x7f32f00c9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.387+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 shutdown_connections 2024-01-16T15:00:11.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.387+0000 7f32f7400640 1 -- 172.21.15.53:0/3506939835 wait complete. 2024-01-16T15:00:11.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:11.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.605+0000 7f9f0ee00640 1 Processor -- start 2024-01-16T15:00:11.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.605+0000 7f9f0ee00640 1 -- start start 2024-01-16T15:00:11.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0ee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f08146770 0x7f9f0806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0ee00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f080b2e10 con 0x7f9f08071410 2024-01-16T15:00:11.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0ee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f08146bc0 con 0x7f9f08071f00 2024-01-16T15:00:11.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0ee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f0806cec0 con 0x7f9f08146770 2024-01-16T15:00:11.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9f08071410 0x7f9f080717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49878/0 (socket says 172.21.15.53:49878) 2024-01-16T15:00:11.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0da00640 1 -- 172.21.15.53:0/2840567540 learned_addr learned my addr 172.21.15.53:0/2840567540 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:11.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.607+0000 7f9f0e400640 1 --2- 172.21.15.53:0/2840567540 >> v2:172.21.15.146:3300/0 conn(0x7f9f08146770 0x7f9f0806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275489157 0 0) 0x7f9f080b2e10 con 0x7f9f08071410 2024-01-16T15:00:11.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee8003520 con 0x7f9f08071410 2024-01-16T15:00:11.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509800403 0 0) 0x7f9ee8003520 con 0x7f9f08071410 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 >> v2:172.21.15.146:3300/0 conn(0x7f9f08146770 msgr2=0x7f9f0806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 --2- 172.21.15.53:0/2840567540 >> v2:172.21.15.146:3300/0 conn(0x7f9f08146770 0x7f9f0806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 >> v1:172.21.15.146:6789/0 conn(0x7f9f08071f00 legacy=0x7f9f08164ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0806d1a0 con 0x7f9f08071410 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9ef8001670 con 0x7f9f08071410 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 >> v1:172.21.15.53:6789/0 conn(0x7f9f08071410 legacy=0x7f9f080717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee80039c0 0x7f9ee8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ee80047c0 0x7f9ee8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80051d0 0x7f9ee800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f0806cec0 con 0x7f9ee80039c0 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee800c120 con 0x7f9ee80047c0 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee800c2e0 con 0x7f9ee80051d0 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.608+0000 7f9f0da00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee80039c0 0x7f9ee8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0d000640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ee80047c0 0x7f9ee8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0e400640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80051d0 0x7f9ee800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0da00640 1 -- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80051d0 msgr2=0x7f9ee800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0da00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80051d0 0x7f9ee800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0da00640 1 -- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ee80047c0 msgr2=0x7f9ee8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0da00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ee80047c0 0x7f9ee8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0da00640 1 -- 172.21.15.53:0/2840567540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee8003520 con 0x7f9ee80039c0 2024-01-16T15:00:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0d000640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ee80047c0 0x7f9ee8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:11.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.609+0000 7f9f0e400640 1 -- 172.21.15.53:0/2840567540 reap_dead start 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.610+0000 7f9f0da00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee80039c0 0x7f9ee8003de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8001eb0 tx=0x7f9ee80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fa58d301d45b62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.610+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8005ad0 con 0x7f9ee80039c0 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.610+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef8005c30 con 0x7f9ee80039c0 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.610+0000 7f9f07e00640 1 -- 172.21.15.53:0/2840567540 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef8007070 con 0x7f9ee80039c0 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 -- 172.21.15.53:0/2840567540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee80039c0 msgr2=0x7f9ee8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee80039c0 0x7f9ee8003de0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8001eb0 tx=0x7f9ee80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 -- 172.21.15.53:0/2840567540 shutdown_connections 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/2840567540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ee80039c0 0x7f9ee8003de0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 -- 172.21.15.53:0/2840567540 >> 172.21.15.53:0/2840567540 conn(0x7f9f0806a800 msgr2=0x7f9f0806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 -- 172.21.15.53:0/2840567540 shutdown_connections 2024-01-16T15:00:11.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.611+0000 7f9f0ee00640 1 -- 172.21.15.53:0/2840567540 wait complete. 2024-01-16T15:00:11.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.613+0000 7f9f0ee00640 1 Processor -- start 2024-01-16T15:00:11.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.613+0000 7f9f0ee00640 1 -- start start 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80039c0 0x7f9f081f25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 0x7f9f0813f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f080c6710 0x7f9f080c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f08144fb0 con 0x7f9f08143c30 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f080ce760 con 0x7f9f080c6710 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f080c70d0 con 0x7f9ee80039c0 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 0x7f9f0813f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 0x7f9f0813f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35340/0 (socket says 172.21.15.53:35340) 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0d000640 1 -- 172.21.15.53:0/3650675868 learned_addr learned my addr 172.21.15.53:0/3650675868 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.614+0000 7f9f0e400640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f080c6710 0x7f9f080c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0da00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80039c0 0x7f9f081f25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0d000640 1 -- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80039c0 msgr2=0x7f9f081f25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0d000640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80039c0 0x7f9f081f25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0d000640 1 -- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f080c6710 msgr2=0x7f9f080c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0d000640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f080c6710 0x7f9f080c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0d000640 1 -- 172.21.15.53:0/3650675868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f0813fea0 con 0x7f9f08143c30 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0e400640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f080c6710 0x7f9f080c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0da00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80039c0 0x7f9f081f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:11.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0d000640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 0x7f9f0813f960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000e390 tx=0x7f9ef000e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef00052e0 con 0x7f9f08143c30 2024-01-16T15:00:11.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef0005440 con 0x7f9f08143c30 2024-01-16T15:00:11.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ef0003810 con 0x7f9f08143c30 2024-01-16T15:00:11.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.615+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f081440b0 con 0x7f9f08143c30 2024-01-16T15:00:11.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.616+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f08144330 con 0x7f9f08143c30 2024-01-16T15:00:11.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.617+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f081450f0 con 0x7f9f08143c30 2024-01-16T15:00:11.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.618+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9ef0003970 con 0x7f9f08143c30 2024-01-16T15:00:11.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.618+0000 7f9f05600640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ee006b2f0 0x7f9ee006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:11.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.619+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9ef0089110 con 0x7f9f08143c30 2024-01-16T15:00:11.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.619+0000 7f9f0da00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ee006b2f0 0x7f9ee006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:11.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.620+0000 7f9f0da00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ee006b2f0 0x7f9ee006d7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8034000 tx=0x7f9ef8007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.623+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f081450f0 con 0x7f9f08143c30 2024-01-16T15:00:11.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.817+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f0813fea0 con 0x7f9f08143c30 2024-01-16T15:00:11.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.818+0000 7f9f05600640 1 -- 172.21.15.53:0/3650675868 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f9f0813fea0 con 0x7f9f08143c30 2024-01-16T15:00:11.799 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:11.799 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:11.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ee006b2f0 msgr2=0x7f9ee006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ee006b2f0 0x7f9ee006d7b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9ef8034000 tx=0x7f9ef8007000 comp rx=0 tx=0).stop 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 msgr2=0x7f9f0813f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 0x7f9f0813f960 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9ef000e390 tx=0x7f9ef000e850 comp rx=0 tx=0).stop 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 shutdown_connections 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ee006b2f0 0x7f9ee006d7b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f080c6710 0x7f9f080c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f08143c30 0x7f9f0813f960 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 --2- 172.21.15.53:0/3650675868 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ee80039c0 0x7f9f081f25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 >> 172.21.15.53:0/3650675868 conn(0x7f9f0806a800 msgr2=0x7f9f081600b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 shutdown_connections 2024-01-16T15:00:11.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:11.821+0000 7f9f0ee00640 1 -- 172.21.15.53:0/3650675868 wait complete. 2024-01-16T15:00:11.821 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:11.822 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.061+0000 7f754ca00640 1 Processor -- start 2024-01-16T15:00:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.061+0000 7f754ca00640 1 -- start start 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f754ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75480adf20 0x7f7548070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f754ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f754815bf10 0x7f75480cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f754ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f754815f4e0 con 0x7f75480cc980 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f754ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f754815f970 con 0x7f754815bf10 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f754ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75480ccdd0 con 0x7f75480adf20 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f7546a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f754815bf10 0x7f75480cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f7546a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f754815bf10 0x7f75480cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52774/0 (socket says 172.21.15.53:52774) 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f7546a00640 1 -- 172.21.15.53:0/571698213 learned_addr learned my addr 172.21.15.53:0/571698213 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.063+0000 7f7547400640 1 --2- 172.21.15.53:0/571698213 >> v2:172.21.15.146:3300/0 conn(0x7f75480adf20 0x7f7548070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.064+0000 7f7546a00640 1 -- 172.21.15.53:0/571698213 >> v2:172.21.15.146:3300/0 conn(0x7f75480adf20 msgr2=0x7f7548070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.064+0000 7f7546a00640 1 --2- 172.21.15.53:0/571698213 >> v2:172.21.15.146:3300/0 conn(0x7f75480adf20 0x7f7548070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.064+0000 7f7546a00640 1 -- 172.21.15.53:0/571698213 >> v1:172.21.15.146:6789/0 conn(0x7f75480cc980 legacy=0x7f7548143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.064+0000 7f7546a00640 1 -- 172.21.15.53:0/571698213 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7548071130 con 0x7f754815bf10 2024-01-16T15:00:12.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.065+0000 7f7546a00640 1 --2- 172.21.15.53:0/571698213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f754815bf10 0x7f75480cc440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f75480b4150 tx=0x7f753002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03cd819142f0ac9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.065+0000 7f7546000640 1 -- 172.21.15.53:0/571698213 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75300317e0 con 0x7f754815bf10 2024-01-16T15:00:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.066+0000 7f7546000640 1 -- 172.21.15.53:0/571698213 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7530031940 con 0x7f754815bf10 2024-01-16T15:00:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.066+0000 7f7546000640 1 -- 172.21.15.53:0/571698213 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7530031cd0 con 0x7f754815bf10 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.066+0000 7f754ca00640 1 -- 172.21.15.53:0/571698213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f754815bf10 msgr2=0x7f75480cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.066+0000 7f754ca00640 1 --2- 172.21.15.53:0/571698213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f754815bf10 0x7f75480cc440 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f75480b4150 tx=0x7f753002b880 comp rx=0 tx=0).stop 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.066+0000 7f754ca00640 1 -- 172.21.15.53:0/571698213 shutdown_connections 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.066+0000 7f754ca00640 1 --2- 172.21.15.53:0/571698213 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f754815bf10 0x7f75480cc440 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.067+0000 7f754ca00640 1 --2- 172.21.15.53:0/571698213 >> v2:172.21.15.146:3300/0 conn(0x7f75480adf20 0x7f7548070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.067+0000 7f754ca00640 1 -- 172.21.15.53:0/571698213 >> 172.21.15.53:0/571698213 conn(0x7f754806a7d0 msgr2=0x7f754806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.067+0000 7f754ca00640 1 -- 172.21.15.53:0/571698213 shutdown_connections 2024-01-16T15:00:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.067+0000 7f754ca00640 1 -- 172.21.15.53:0/571698213 wait complete. 2024-01-16T15:00:12.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.069+0000 7f754ca00640 1 Processor -- start 2024-01-16T15:00:12.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.069+0000 7f754ca00640 1 -- start start 2024-01-16T15:00:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.070+0000 7f754ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 0x7f754813ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.070+0000 7f754ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75480cc980 0x7f7548142680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.070+0000 7f754ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754815bf10 0x7f754806d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.070+0000 7f754ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754814f580 con 0x7f75480adf20 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.070+0000 7f754ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f754813c100 con 0x7f754815bf10 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.070+0000 7f754ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7548140510 con 0x7f75480cc980 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 0x7f754813ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 0x7f754813ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52776/0 (socket says 172.21.15.53:52776) 2024-01-16T15:00:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 -- 172.21.15.53:0/4125271183 learned_addr learned my addr 172.21.15.53:0/4125271183 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547e00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754815bf10 0x7f754806d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7546a00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75480cc980 0x7f7548142680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 -- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75480cc980 msgr2=0x7f7548142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75480cc980 0x7f7548142680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 -- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754815bf10 msgr2=0x7f754806d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754815bf10 0x7f754806d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 -- 172.21.15.53:0/4125271183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75300094d0 con 0x7f75480adf20 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7546a00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75480cc980 0x7f7548142680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.071+0000 7f7547400640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 0x7f754813ffd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f753800d5f0 tx=0x7f753800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.072+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75380125f0 con 0x7f75480adf20 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.072+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7538012750 con 0x7f75480adf20 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.072+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7548142bc0 con 0x7f75480adf20 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.072+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7538012b00 con 0x7f75480adf20 2024-01-16T15:00:12.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.072+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7548141ba0 con 0x7f75480adf20 2024-01-16T15:00:12.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.073+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7548067450 con 0x7f75480adf20 2024-01-16T15:00:12.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.074+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7538012c60 con 0x7f75480adf20 2024-01-16T15:00:12.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.075+0000 7f753f400640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f752006b2c0 0x7f752006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.075+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f753808a110 con 0x7f75480adf20 2024-01-16T15:00:12.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.076+0000 7f7546a00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f752006b2c0 0x7f752006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.077+0000 7f7546a00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f752006b2c0 0x7f752006d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7530036000 tx=0x7f753003d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.080+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7538054ef0 con 0x7f75480adf20 2024-01-16T15:00:12.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.280+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75480cc220 con 0x7f75480adf20 2024-01-16T15:00:12.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.280+0000 7f753f400640 1 -- 172.21.15.53:0/4125271183 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7538058510 con 0x7f75480adf20 2024-01-16T15:00:12.261 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:12.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f752006b2c0 msgr2=0x7f752006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f752006b2c0 0x7f752006d780 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7530036000 tx=0x7f753003d040 comp rx=0 tx=0).stop 2024-01-16T15:00:12.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 msgr2=0x7f754813ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 0x7f754813ffd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f753800d5f0 tx=0x7f753800dab0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 shutdown_connections 2024-01-16T15:00:12.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f752006b2c0 0x7f752006d780 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f754815bf10 0x7f754806d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75480cc980 0x7f7548142680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 --2- 172.21.15.53:0/4125271183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75480adf20 0x7f754813ffd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.283+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 >> 172.21.15.53:0/4125271183 conn(0x7f754806a7d0 msgr2=0x7f7548142fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.284+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 shutdown_connections 2024-01-16T15:00:12.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.284+0000 7f754ca00640 1 -- 172.21.15.53:0/4125271183 wait complete. 2024-01-16T15:00:12.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:12.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.524+0000 7f2001000640 1 Processor -- start 2024-01-16T15:00:12.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.524+0000 7f2001000640 1 -- start start 2024-01-16T15:00:12.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f2001000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ffc0ccb50 0x7f1ffc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f2001000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1ffc15bf10 0x7f1ffc143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f2001000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc15f4e0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f2001000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc15f970 con 0x7f1ffc0ccb50 2024-01-16T15:00:12.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f2001000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc071130 con 0x7f1ffc15bf10 2024-01-16T15:00:12.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f1ffb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1ffc0adf20 0x7f1ffc0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39042/0 (socket says 172.21.15.53:39042) 2024-01-16T15:00:12.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f1ffb400640 1 -- 172.21.15.53:0/3255623555 learned_addr learned my addr 172.21.15.53:0/3255623555 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:12.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f1ff3e00640 1 --2- 172.21.15.53:0/3255623555 >> v2:172.21.15.53:3300/0 conn(0x7f1ffc0ccb50 0x7f1ffc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.526+0000 7f1ffbe00640 1 --2- 172.21.15.53:0/3255623555 >> v2:172.21.15.146:3300/0 conn(0x7f1ffc15bf10 0x7f1ffc143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ffaa00640 1 -- 172.21.15.53:0/3255623555 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867387434 0 0) 0x7f1ffc15f4e0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ffaa00640 1 -- 172.21.15.53:0/3255623555 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fdc003520 con 0x7f1ffc0adf20 2024-01-16T15:00:12.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ff3e00640 1 -- 172.21.15.53:0/3255623555 >> v2:172.21.15.146:3300/0 conn(0x7f1ffc15bf10 msgr2=0x7f1ffc143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ff3e00640 1 --2- 172.21.15.53:0/3255623555 >> v2:172.21.15.146:3300/0 conn(0x7f1ffc15bf10 0x7f1ffc143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ff3e00640 1 -- 172.21.15.53:0/3255623555 >> v1:172.21.15.53:6789/0 conn(0x7f1ffc0adf20 legacy=0x7f1ffc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ff3e00640 1 -- 172.21.15.53:0/3255623555 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc15ebb0 con 0x7f1ffc0ccb50 2024-01-16T15:00:12.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.527+0000 7f1ffbe00640 1 --2- 172.21.15.53:0/3255623555 >> v2:172.21.15.146:3300/0 conn(0x7f1ffc15bf10 0x7f1ffc143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:12.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.528+0000 7f1ff3e00640 1 --2- 172.21.15.53:0/3255623555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0ccb50 0x7f1ffc070a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc06ff20 tx=0x7f1fe402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95ad6c1e0276ffbf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f1ffaa00640 1 -- 172.21.15.53:0/3255623555 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe402e070 con 0x7f1ffc0ccb50 2024-01-16T15:00:12.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f1ffaa00640 1 -- 172.21.15.53:0/3255623555 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe40315a0 con 0x7f1ffc0ccb50 2024-01-16T15:00:12.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f1ffaa00640 1 -- 172.21.15.53:0/3255623555 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4031910 con 0x7f1ffc0ccb50 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 -- 172.21.15.53:0/3255623555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0ccb50 msgr2=0x7f1ffc070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 --2- 172.21.15.53:0/3255623555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0ccb50 0x7f1ffc070a30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc06ff20 tx=0x7f1fe402b840 comp rx=0 tx=0).stop 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 -- 172.21.15.53:0/3255623555 shutdown_connections 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 --2- 172.21.15.53:0/3255623555 >> v2:172.21.15.146:3300/0 conn(0x7f1ffc15bf10 0x7f1ffc143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 --2- 172.21.15.53:0/3255623555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0ccb50 0x7f1ffc070a30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 -- 172.21.15.53:0/3255623555 >> 172.21.15.53:0/3255623555 conn(0x7f1ffc06a7d0 msgr2=0x7f1ffc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 -- 172.21.15.53:0/3255623555 shutdown_connections 2024-01-16T15:00:12.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.529+0000 7f2001000640 1 -- 172.21.15.53:0/3255623555 wait complete. 2024-01-16T15:00:12.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.531+0000 7f2001000640 1 Processor -- start 2024-01-16T15:00:12.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.531+0000 7f2001000640 1 -- start start 2024-01-16T15:00:12.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f2001000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 0x7f1ffc15a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f2001000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ffc0ccb50 0x7f1ffc0b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f2001000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ffc15bf10 0x7f1ffc0c46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f2001000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c3d00 con 0x7f1ffc0adf20 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f2001000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c61e0 con 0x7f1ffc15bf10 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f2001000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc0c42c0 con 0x7f1ffc0ccb50 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.532+0000 7f1ffb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 0x7f1ffc15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 0x7f1ffc15a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52802/0 (socket says 172.21.15.53:52802) 2024-01-16T15:00:12.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 -- 172.21.15.53:0/1905751396 learned_addr learned my addr 172.21.15.53:0/1905751396 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:12.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ff3e00640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ffc0ccb50 0x7f1ffc0b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffbe00640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ffc15bf10 0x7f1ffc0c46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 -- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ffc0ccb50 msgr2=0x7f1ffc0b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ffc0ccb50 0x7f1ffc0b1e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 -- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ffc15bf10 msgr2=0x7f1ffc0c46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ffc15bf10 0x7f1ffc0c46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 -- 172.21.15.53:0/1905751396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe4009400 con 0x7f1ffc0adf20 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffbe00640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ffc15bf10 0x7f1ffc0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.533+0000 7f1ffb400640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 0x7f1ffc15a970 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c430 tx=0x7f1fec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.534+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec005ce0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.534+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec005e40 con 0x7f1ffc0adf20 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.534+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc15aeb0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.534+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec0072f0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.534+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc0b2bd0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.535+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc0c4400 con 0x7f1ffc0adf20 2024-01-16T15:00:12.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.536+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1fec007450 con 0x7f1ffc0adf20 2024-01-16T15:00:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.537+0000 7f1ff3400640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd406b2c0 0x7f1fd406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.537+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fec089270 con 0x7f1ffc0adf20 2024-01-16T15:00:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.537+0000 7f1ff3e00640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd406b2c0 0x7f1fd406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.538+0000 7f1ff3e00640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd406b2c0 0x7f1fd406d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4036000 tx=0x7f1fe403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.541+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ffc0c4400 con 0x7f1ffc0adf20 2024-01-16T15:00:12.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.736+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ffc0c4400 con 0x7f1ffc0adf20 2024-01-16T15:00:12.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.737+0000 7f1ff3400640 1 -- 172.21.15.53:0/1905751396 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1fec0540d0 con 0x7f1ffc0adf20 2024-01-16T15:00:12.718 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:12.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd406b2c0 msgr2=0x7f1fd406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd406b2c0 0x7f1fd406d780 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4036000 tx=0x7f1fe403d040 comp rx=0 tx=0).stop 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 msgr2=0x7f1ffc15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 0x7f1ffc15a970 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00c430 tx=0x7f1fec00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 shutdown_connections 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd406b2c0 0x7f1fd406d780 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ffc15bf10 0x7f1ffc0c46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ffc0ccb50 0x7f1ffc0b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 --2- 172.21.15.53:0/1905751396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ffc0adf20 0x7f1ffc15a970 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 >> 172.21.15.53:0/1905751396 conn(0x7f1ffc06a7d0 msgr2=0x7f1ffc0708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 shutdown_connections 2024-01-16T15:00:12.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.740+0000 7f2001000640 1 -- 172.21.15.53:0/1905751396 wait complete. 2024-01-16T15:00:12.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:12.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.991+0000 7f72fd400640 1 Processor -- start 2024-01-16T15:00:12.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.992+0000 7f72fd400640 1 -- start start 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72fd400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72f80cf9a0 0x7f72f80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72fd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 0x7f72f80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72fd400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72f80d0320 con 0x7f72f815d4b0 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72fd400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72f80d1c60 con 0x7f72f80cf9a0 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72fd400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80b2730 con 0x7f72f80b31d0 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72f7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f72f815d4b0 0x7f72f80d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39052/0 (socket says 172.21.15.53:39052) 2024-01-16T15:00:12.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72f7e00640 1 -- 172.21.15.53:0/3407535588 learned_addr learned my addr 172.21.15.53:0/3407535588 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:12.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72efe00640 1 --2- 172.21.15.53:0/3407535588 >> v2:172.21.15.146:3300/0 conn(0x7f72f80cf9a0 0x7f72f80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1399582059 0 0) 0x7f72f80d0320 con 0x7f72f815d4b0 2024-01-16T15:00:12.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d8003520 con 0x7f72f815d4b0 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.994+0000 7f72fca00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 0x7f72f80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2246410126 0 0) 0x7f72d8003520 con 0x7f72f815d4b0 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 msgr2=0x7f72f80b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 0x7f72f80b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 >> v2:172.21.15.146:3300/0 conn(0x7f72f80cf9a0 msgr2=0x7f72f80cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 --2- 172.21.15.53:0/3407535588 >> v2:172.21.15.146:3300/0 conn(0x7f72f80cf9a0 0x7f72f80cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80d1ec0 con 0x7f72f815d4b0 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f72e8001670 con 0x7f72f815d4b0 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 >> v1:172.21.15.53:6789/0 conn(0x7f72f815d4b0 legacy=0x7f72f80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d8003970 0x7f72d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 0x7f72d80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80d1c60 con 0x7f72d8004d00 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80b2730 con 0x7f72d8003970 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72d800b980 con 0x7f72f80b31d0 2024-01-16T15:00:12.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.995+0000 7f72fca00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72f7e00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d8003970 0x7f72d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72efe00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 0x7f72d80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72fca00640 1 -- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 msgr2=0x7f72d80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72fca00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80b31d0 0x7f72d80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72fca00640 1 -- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d8003970 msgr2=0x7f72d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72fca00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d8003970 0x7f72d8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72fca00640 1 -- 172.21.15.53:0/3407535588 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72d8003520 con 0x7f72d8004d00 2024-01-16T15:00:12.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.996+0000 7f72f7e00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72d8003970 0x7f72d8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.997+0000 7f72fca00640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72d8005180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f72f000e170 tx=0x7f72d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b70f4482ba7b9fd5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.997+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f002cbf0 con 0x7f72d8004d00 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.997+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f000f040 con 0x7f72d8004d00 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.997+0000 7f72f7400640 1 -- 172.21.15.53:0/3407535588 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0034440 con 0x7f72d8004d00 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fd400640 1 -- 172.21.15.53:0/3407535588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 msgr2=0x7f72d8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fd400640 1 --2- 172.21.15.53:0/3407535588 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72d8005180 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f72f000e170 tx=0x7f72d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fca00640 1 -- 172.21.15.53:0/3407535588 reap_dead start 2024-01-16T15:00:12.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fd400640 1 -- 172.21.15.53:0/3407535588 shutdown_connections 2024-01-16T15:00:12.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fd400640 1 -- 172.21.15.53:0/3407535588 >> 172.21.15.53:0/3407535588 conn(0x7f72f80d3000 msgr2=0x7f72f80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:12.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fd400640 1 -- 172.21.15.53:0/3407535588 shutdown_connections 2024-01-16T15:00:12.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:12.998+0000 7f72fd400640 1 -- 172.21.15.53:0/3407535588 wait complete. 2024-01-16T15:00:12.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.000+0000 7f72fd400640 1 Processor -- start 2024-01-16T15:00:12.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.000+0000 7f72fd400640 1 -- start start 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72fd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72f80ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72fd400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72f80c9300 0x7f72f80c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72fd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80c7c90 0x7f72f80c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72fd400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80cc5b0 con 0x7f72d8004d00 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72fd400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80c9870 con 0x7f72f80c9300 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72fd400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f80cc850 con 0x7f72f80c7c90 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72f7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72f80ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72f7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72f80ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52826/0 (socket says 172.21.15.53:52826) 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.001+0000 7f72f7e00640 1 -- 172.21.15.53:0/3853698347 learned_addr learned my addr 172.21.15.53:0/3853698347 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72efe00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72f80c9300 0x7f72f80c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72fca00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80c7c90 0x7f72f80c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f7e00640 1 -- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80c7c90 msgr2=0x7f72f80c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f7e00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80c7c90 0x7f72f80c8110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f7e00640 1 -- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72f80c9300 msgr2=0x7f72f80c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:12.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f7e00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72f80c9300 0x7f72f80c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:12.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f7e00640 1 -- 172.21.15.53:0/3853698347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f80ccb30 con 0x7f72d8004d00 2024-01-16T15:00:12.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f7e00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72f80ca0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72e8001b40 tx=0x7f72e800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e800c8d0 con 0x7f72d8004d00 2024-01-16T15:00:12.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e800ca30 con 0x7f72d8004d00 2024-01-16T15:00:12.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f80cce10 con 0x7f72d8004d00 2024-01-16T15:00:12.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.002+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e800cdc0 con 0x7f72d8004d00 2024-01-16T15:00:12.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.003+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f80ca610 con 0x7f72d8004d00 2024-01-16T15:00:12.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.004+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8067450 con 0x7f72d8004d00 2024-01-16T15:00:12.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.005+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f72e801d460 con 0x7f72d8004d00 2024-01-16T15:00:12.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.006+0000 7f72f4c00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72d006b2f0 0x7f72d006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:12.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.006+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f72e8089cc0 con 0x7f72d8004d00 2024-01-16T15:00:12.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.007+0000 7f72efe00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72d006b2f0 0x7f72d006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:12.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.008+0000 7f72efe00640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72d006b2f0 0x7f72d006d7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f72f80c6700 tx=0x7f72e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:12.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.011+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72e8090080 con 0x7f72d8004d00 2024-01-16T15:00:13.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.209+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f72f80d24d0 con 0x7f72d8004d00 2024-01-16T15:00:13.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.210+0000 7f72f4c00640 1 -- 172.21.15.53:0/3853698347 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f72e8054a90 con 0x7f72d8004d00 2024-01-16T15:00:13.190 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72d006b2f0 msgr2=0x7f72d006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72d006b2f0 0x7f72d006d7b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f72f80c6700 tx=0x7f72e0008040 comp rx=0 tx=0).stop 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 msgr2=0x7f72f80ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72f80ca0d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f72e8001b40 tx=0x7f72e800ec80 comp rx=0 tx=0).stop 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 shutdown_connections 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72d006b2f0 0x7f72d006d7b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72f80c7c90 0x7f72f80c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72f80c9300 0x7f72f80c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.212+0000 7f72fd400640 1 --2- 172.21.15.53:0/3853698347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72d8004d00 0x7f72f80ca0d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.213+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 >> 172.21.15.53:0/3853698347 conn(0x7f72f80d3000 msgr2=0x7f72f806c440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.213+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 shutdown_connections 2024-01-16T15:00:13.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.213+0000 7f72fd400640 1 -- 172.21.15.53:0/3853698347 wait complete. 2024-01-16T15:00:13.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:13.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.435+0000 7fe1a1c00640 1 Processor -- start 2024-01-16T15:00:13.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.435+0000 7fe1a1c00640 1 -- start start 2024-01-16T15:00:13.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.437+0000 7fe1a1c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 0x7fe19c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.437+0000 7fe1a1c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c06c470 con 0x7fe19c13e360 2024-01-16T15:00:13.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.437+0000 7fe1a1c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c06c750 con 0x7fe19c0cf9a0 2024-01-16T15:00:13.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.437+0000 7fe1a1c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c06ca30 con 0x7fe19c13df80 2024-01-16T15:00:13.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.437+0000 7fe191e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe19c13e360 0x7fe19c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39062/0 (socket says 172.21.15.53:39062) 2024-01-16T15:00:13.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.437+0000 7fe191e00640 1 -- 172.21.15.53:0/3970841184 learned_addr learned my addr 172.21.15.53:0/3970841184 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:13.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19aa00640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 0x7fe19c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259298694 0 0) 0x7fe19c06c470 con 0x7fe19c13e360 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe17c003520 con 0x7fe19c13e360 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 714097803 0 0) 0x7fe17c003520 con 0x7fe19c13e360 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 msgr2=0x7fe19c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 0x7fe19c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 >> v1:172.21.15.146:6789/0 conn(0x7fe19c0cf9a0 legacy=0x7fe19c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c06cd10 con 0x7fe19c13e360 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe184001670 con 0x7fe19c13e360 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 >> v1:172.21.15.53:6789/0 conn(0x7fe19c13e360 legacy=0x7fe19c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 0x7fe17c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c003f20 0x7fe17c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe17c004d00 0x7fe17c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c06ca30 con 0x7fe17c004d00 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c00ba60 con 0x7fe17c003f20 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.438+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe17c00bc20 con 0x7fe19c13df80 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19b400640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe17c004d00 0x7fe17c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe191e00640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c003f20 0x7fe17c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19aa00640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 0x7fe17c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19b400640 1 -- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 msgr2=0x7fe17c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19b400640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c13df80 0x7fe17c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19b400640 1 -- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c003f20 msgr2=0x7fe17c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19b400640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c003f20 0x7fe17c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.439+0000 7fe19b400640 1 -- 172.21.15.53:0/3970841184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe17c003520 con 0x7fe17c004d00 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.440+0000 7fe19b400640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe17c004d00 0x7fe17c005180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe194001750 tx=0x7fe17c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e85d894a884e8f85 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.440+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1940052d0 con 0x7fe17c004d00 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.440+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe19402e050 con 0x7fe17c004d00 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.440+0000 7fe19a000640 1 -- 172.21.15.53:0/3970841184 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194003830 con 0x7fe17c004d00 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe1a1c00640 1 -- 172.21.15.53:0/3970841184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe17c004d00 msgr2=0x7fe17c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/3970841184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe17c004d00 0x7fe17c005180 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe194001750 tx=0x7fe17c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe19b400640 1 -- 172.21.15.53:0/3970841184 reap_dead start 2024-01-16T15:00:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe1a1c00640 1 -- 172.21.15.53:0/3970841184 shutdown_connections 2024-01-16T15:00:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe1a1c00640 1 -- 172.21.15.53:0/3970841184 >> 172.21.15.53:0/3970841184 conn(0x7fe19c06a800 msgr2=0x7fe19c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe1a1c00640 1 -- 172.21.15.53:0/3970841184 shutdown_connections 2024-01-16T15:00:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.441+0000 7fe1a1c00640 1 -- 172.21.15.53:0/3970841184 wait complete. 2024-01-16T15:00:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.443+0000 7fe1a1c00640 1 Processor -- start 2024-01-16T15:00:13.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.443+0000 7fe1a1c00640 1 -- start start 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe1a1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c004d00 0x7fe19c1ee020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe1a1c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c1524a0 0x7fe19c152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe1a1c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 0x7fe19c154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe1a1c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0b2220 con 0x7fe19c1540f0 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe1a1c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c152160 con 0x7fe17c004d00 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe1a1c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c159ff0 con 0x7fe19c1524a0 2024-01-16T15:00:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe19b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 0x7fe19c154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe19b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 0x7fe19c154570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52842/0 (socket says 172.21.15.53:52842) 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.444+0000 7fe19b400640 1 -- 172.21.15.53:0/4197925200 learned_addr learned my addr 172.21.15.53:0/4197925200 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19aa00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c004d00 0x7fe19c1ee020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe191e00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c1524a0 0x7fe19c152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19b400640 1 -- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c1524a0 msgr2=0x7fe19c152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19b400640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c1524a0 0x7fe19c152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19b400640 1 -- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c004d00 msgr2=0x7fe19c1ee020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19b400640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c004d00 0x7fe19c1ee020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19b400640 1 -- 172.21.15.53:0/4197925200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c15a2d0 con 0x7fe19c1540f0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19aa00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c004d00 0x7fe19c1ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe19b400640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 0x7fe19c154570 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe194034040 tx=0x7fe194001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194007a30 con 0x7fe19c1540f0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1940067d0 con 0x7fe19c1540f0 2024-01-16T15:00:13.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c15a5b0 con 0x7fe19c1540f0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.445+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe194006b80 con 0x7fe19c1540f0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.446+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c154ab0 con 0x7fe19c1540f0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.447+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c0675d0 con 0x7fe19c1540f0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.448+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe194006ce0 con 0x7fe19c1540f0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.449+0000 7fe193400640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe17406b2f0 0x7fe17406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.449+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1940ab1c0 con 0x7fe19c1540f0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.449+0000 7fe19aa00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe17406b2f0 0x7fe17406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.450+0000 7fe19aa00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe17406b2f0 0x7fe17406d7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0074d0 tx=0x7fe18c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.453+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe194076020 con 0x7fe19c1540f0 2024-01-16T15:00:13.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.692+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe19c06df00 con 0x7fe19c1540f0 2024-01-16T15:00:13.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.693+0000 7fe193400640 1 -- 172.21.15.53:0/4197925200 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7fe194079640 con 0x7fe19c1540f0 2024-01-16T15:00:13.674 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:13.674 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:13.674 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.696+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe17406b2f0 msgr2=0x7fe17406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.696+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe17406b2f0 0x7fe17406d7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0074d0 tx=0x7fe18c00b040 comp rx=0 tx=0).stop 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.696+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 msgr2=0x7fe19c154570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.696+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 0x7fe19c154570 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe194034040 tx=0x7fe194001120 comp rx=0 tx=0).stop 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 shutdown_connections 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe17406b2f0 0x7fe17406d7b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe19c1540f0 0x7fe19c154570 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe19c1524a0 0x7fe19c152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 --2- 172.21.15.53:0/4197925200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe17c004d00 0x7fe19c1ee020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 >> 172.21.15.53:0/4197925200 conn(0x7fe19c06a800 msgr2=0x7fe19c0b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 shutdown_connections 2024-01-16T15:00:13.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.697+0000 7fe1a1c00640 1 -- 172.21.15.53:0/4197925200 wait complete. 2024-01-16T15:00:13.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:13.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.927+0000 7f077a000640 1 Processor -- start 2024-01-16T15:00:13.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.928+0000 7f077a000640 1 -- start start 2024-01-16T15:00:13.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f077a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f077413e360 0x7f07740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f077a000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077415c5f0 con 0x7f077415f9c0 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f077a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f077406c470 con 0x7f077413df80 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f077a000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f077406c750 con 0x7f077413e360 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f076a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f077413e360 0x7f07740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f076a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f077413e360 0x7f07740cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52858/0 (socket says 172.21.15.53:52858) 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f0773e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f077415f9c0 0x7f07740cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39074/0 (socket says 172.21.15.53:39074) 2024-01-16T15:00:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.929+0000 7f076a800640 1 -- 172.21.15.53:0/1633957540 learned_addr learned my addr 172.21.15.53:0/1633957540 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.930+0000 7f076a800640 1 -- 172.21.15.53:0/1633957540 >> v1:172.21.15.146:6789/0 conn(0x7f077413df80 legacy=0x7f077415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.930+0000 7f076a800640 1 -- 172.21.15.53:0/1633957540 >> v1:172.21.15.53:6789/0 conn(0x7f077415f9c0 legacy=0x7f07740cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.930+0000 7f0772a00640 1 -- 172.21.15.53:0/1633957540 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409357572 0 0) 0x7f077415c5f0 con 0x7f077415f9c0 2024-01-16T15:00:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.930+0000 7f076a800640 1 -- 172.21.15.53:0/1633957540 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f077406ca30 con 0x7f077413e360 2024-01-16T15:00:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.931+0000 7f076a800640 1 --2- 172.21.15.53:0/1633957540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413e360 0x7f07740cf290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f075c009870 tx=0x7f075c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a84f358baf4e96a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.931+0000 7f0772a00640 1 -- 172.21.15.53:0/1633957540 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c02f840 con 0x7f077413e360 2024-01-16T15:00:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.931+0000 7f0772a00640 1 -- 172.21.15.53:0/1633957540 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075c02f9a0 con 0x7f077413e360 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.931+0000 7f0772a00640 1 -- 172.21.15.53:0/1633957540 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075c02fd30 con 0x7f077413e360 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 -- 172.21.15.53:0/1633957540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413e360 msgr2=0x7f07740cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 --2- 172.21.15.53:0/1633957540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413e360 0x7f07740cf290 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f075c009870 tx=0x7f075c02ba20 comp rx=0 tx=0).stop 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 -- 172.21.15.53:0/1633957540 shutdown_connections 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 --2- 172.21.15.53:0/1633957540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413e360 0x7f07740cf290 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 -- 172.21.15.53:0/1633957540 >> 172.21.15.53:0/1633957540 conn(0x7f077406a800 msgr2=0x7f077406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 -- 172.21.15.53:0/1633957540 shutdown_connections 2024-01-16T15:00:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.932+0000 7f077a000640 1 -- 172.21.15.53:0/1633957540 wait complete. 2024-01-16T15:00:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.934+0000 7f077a000640 1 Processor -- start 2024-01-16T15:00:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.934+0000 7f077a000640 1 -- start start 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f077a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 0x7f077413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f077a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f077413e360 0x7f0774152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f077a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f077415f9c0 0x7f077415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f077a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f077413dd80 con 0x7f077413df80 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f077a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0774153ea0 con 0x7f077415f9c0 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f077a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0774152e00 con 0x7f077413e360 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f0773400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 0x7f077413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f0773400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 0x7f077413d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52874/0 (socket says 172.21.15.53:52874) 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.935+0000 7f0773400640 1 -- 172.21.15.53:0/1999461808 learned_addr learned my addr 172.21.15.53:0/1999461808 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773e00640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f077415f9c0 0x7f077415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773400640 1 -- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f077413e360 msgr2=0x7f0774152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773400640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f077413e360 0x7f0774152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773400640 1 -- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f077415f9c0 msgr2=0x7f077415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773400640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f077415f9c0 0x7f077415d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773400640 1 -- 172.21.15.53:0/1999461808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c0094d0 con 0x7f077413df80 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f0773400640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 0x7f077413d840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f076400c730 tx=0x7f076400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764003b70 con 0x7f077413df80 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0764003cf0 con 0x7f077413df80 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07741530e0 con 0x7f077413df80 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0764003ea0 con 0x7f077413df80 2024-01-16T15:00:13.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.936+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07741536a0 con 0x7f077413df80 2024-01-16T15:00:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.937+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07740675d0 con 0x7f077413df80 2024-01-16T15:00:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.938+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f076400aa90 con 0x7f077413df80 2024-01-16T15:00:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.939+0000 7f076be00640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f074c06b2c0 0x7f074c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:13.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.939+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0764089ab0 con 0x7f077413df80 2024-01-16T15:00:13.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.940+0000 7f076a800640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f074c06b2c0 0x7f074c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:13.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.941+0000 7f076a800640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f074c06b2c0 0x7f074c06d780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f075c036000 tx=0x7f075c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:13.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:13.944+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0764054880 con 0x7f077413df80 2024-01-16T15:00:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.141+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f077406df00 con 0x7f077413df80 2024-01-16T15:00:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.144+0000 7f076be00640 1 -- 172.21.15.53:0/1999461808 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f0764057ea0 con 0x7f077413df80 2024-01-16T15:00:14.124 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:14.124 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f074c06b2c0 msgr2=0x7f074c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f074c06b2c0 0x7f074c06d780 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f075c036000 tx=0x7f075c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 msgr2=0x7f077413d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 0x7f077413d840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f076400c730 tx=0x7f076400cbf0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 shutdown_connections 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f074c06b2c0 0x7f074c06d780 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f077415f9c0 0x7f077415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f077413e360 0x7f0774152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 --2- 172.21.15.53:0/1999461808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f077413df80 0x7f077413d840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.146+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 >> 172.21.15.53:0/1999461808 conn(0x7f077406a800 msgr2=0x7f07740b00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.147+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 shutdown_connections 2024-01-16T15:00:14.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.147+0000 7f077a000640 1 -- 172.21.15.53:0/1999461808 wait complete. 2024-01-16T15:00:14.145 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:00:14.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.383+0000 7f1dd8000640 1 Processor -- start 2024-01-16T15:00:14.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.383+0000 7f1dd8000640 1 -- start start 2024-01-16T15:00:14.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd8000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1dd013df80 0x7f1dd015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd8000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1dd00cf9a0 0x7f1dd00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd8000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1dd006c470 con 0x7f1dd013e360 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd8000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd006c750 con 0x7f1dd00cf9a0 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd8000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1dd006ca30 con 0x7f1dd013df80 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1dd00cf9a0 0x7f1dd00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1dd00cf9a0 0x7f1dd00cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52876/0 (socket says 172.21.15.53:52876) 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd6000640 1 -- 172.21.15.53:0/3678882780 learned_addr learned my addr 172.21.15.53:0/3678882780 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.385+0000 7f1dd5600640 1 --2- 172.21.15.53:0/3678882780 >> v2:172.21.15.146:3300/0 conn(0x7f1dd013df80 0x7f1dd015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.386+0000 7f1dd6000640 1 -- 172.21.15.53:0/3678882780 >> v2:172.21.15.146:3300/0 conn(0x7f1dd013df80 msgr2=0x7f1dd015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.386+0000 7f1dd6000640 1 --2- 172.21.15.53:0/3678882780 >> v2:172.21.15.146:3300/0 conn(0x7f1dd013df80 0x7f1dd015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.386+0000 7f1dd6000640 1 -- 172.21.15.53:0/3678882780 >> v1:172.21.15.146:6789/0 conn(0x7f1dd013e360 legacy=0x7f1dd00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:14.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.386+0000 7f1dd6000640 1 -- 172.21.15.53:0/3678882780 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd006cd10 con 0x7f1dd00cf9a0 2024-01-16T15:00:14.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.386+0000 7f1dd5600640 1 --2- 172.21.15.53:0/3678882780 >> v2:172.21.15.146:3300/0 conn(0x7f1dd013df80 0x7f1dd015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:14.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.387+0000 7f1dd6000640 1 --2- 172.21.15.53:0/3678882780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd00cf9a0 0x7f1dd00cd840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0072770 tx=0x7f1dc802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70aa2247a0d5ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.388+0000 7f1dcfe00640 1 -- 172.21.15.53:0/3678882780 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8031870 con 0x7f1dd00cf9a0 2024-01-16T15:00:14.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.388+0000 7f1dcfe00640 1 -- 172.21.15.53:0/3678882780 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dc80319d0 con 0x7f1dd00cf9a0 2024-01-16T15:00:14.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.388+0000 7f1dcfe00640 1 -- 172.21.15.53:0/3678882780 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dc8031d60 con 0x7f1dd00cf9a0 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.388+0000 7f1dd8000640 1 -- 172.21.15.53:0/3678882780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd00cf9a0 msgr2=0x7f1dd00cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.388+0000 7f1dd8000640 1 --2- 172.21.15.53:0/3678882780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd00cf9a0 0x7f1dd00cd840 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0072770 tx=0x7f1dc802b910 comp rx=0 tx=0).stop 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.389+0000 7f1dd8000640 1 -- 172.21.15.53:0/3678882780 shutdown_connections 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.389+0000 7f1dd8000640 1 --2- 172.21.15.53:0/3678882780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd00cf9a0 0x7f1dd00cd840 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.389+0000 7f1dd8000640 1 --2- 172.21.15.53:0/3678882780 >> v2:172.21.15.146:3300/0 conn(0x7f1dd013df80 0x7f1dd015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.389+0000 7f1dd8000640 1 -- 172.21.15.53:0/3678882780 >> 172.21.15.53:0/3678882780 conn(0x7f1dd006a800 msgr2=0x7f1dd006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.389+0000 7f1dd8000640 1 -- 172.21.15.53:0/3678882780 shutdown_connections 2024-01-16T15:00:14.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.389+0000 7f1dd8000640 1 -- 172.21.15.53:0/3678882780 wait complete. 2024-01-16T15:00:14.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.390+0000 7f1dd8000640 1 Processor -- start 2024-01-16T15:00:14.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.391+0000 7f1dd8000640 1 -- start start 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.391+0000 7f1dd8000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dd00cf9a0 0x7f1dd013d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd8000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 0x7f1dd013ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd8000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd013e360 0x7f1dd0071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd8000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0153260 con 0x7f1dd013df80 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd8000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd01f2f70 con 0x7f1dd013e360 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd8000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1dd0153500 con 0x7f1dd00cf9a0 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 0x7f1dd013ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 0x7f1dd013ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52886/0 (socket says 172.21.15.53:52886) 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 -- 172.21.15.53:0/2694698920 learned_addr learned my addr 172.21.15.53:0/2694698920 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd5600640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dd00cf9a0 0x7f1dd013d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd6000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd013e360 0x7f1dd0071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 -- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dd00cf9a0 msgr2=0x7f1dd013d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dd00cf9a0 0x7f1dd013d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 -- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd013e360 msgr2=0x7f1dd0071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.392+0000 7f1dd4c00640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd013e360 0x7f1dd0071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dd4c00640 1 -- 172.21.15.53:0/2694698920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dc80094d0 con 0x7f1dd013df80 2024-01-16T15:00:14.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dd4c00640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 0x7f1dd013ce80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1db800c430 tx=0x7f1db800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8011790 con 0x7f1dd013df80 2024-01-16T15:00:14.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db80118f0 con 0x7f1dd013df80 2024-01-16T15:00:14.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db8011ca0 con 0x7f1dd013df80 2024-01-16T15:00:14.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dd014e700 con 0x7f1dd013df80 2024-01-16T15:00:14.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.393+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dd014ecc0 con 0x7f1dd013df80 2024-01-16T15:00:14.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.394+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1dd014ee80 con 0x7f1dd013df80 2024-01-16T15:00:14.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.396+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1db801c460 con 0x7f1dd013df80 2024-01-16T15:00:14.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.396+0000 7f1dcd600640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1da806b2c0 0x7f1da806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:14.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.397+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1db8089b70 con 0x7f1dd013df80 2024-01-16T15:00:14.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.397+0000 7f1dd5600640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1da806b2c0 0x7f1da806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:14.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.398+0000 7f1dd5600640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1da806b2c0 0x7f1da806d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0003fd0 tx=0x7f1dc0003990 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:14.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.401+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1db8054940 con 0x7f1dd013df80 2024-01-16T15:00:14.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.637+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1dd0153670 con 0x7f1dd013df80 2024-01-16T15:00:14.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.639+0000 7f1dcd600640 1 -- 172.21.15.53:0/2694698920 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f1db8057f60 con 0x7f1dd013df80 2024-01-16T15:00:14.619 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:14.619 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:00:14.620 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:00:14.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.641+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1da806b2c0 msgr2=0x7f1da806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.641+0000 7f1dd8000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1da806b2c0 0x7f1da806d780 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1dc0003fd0 tx=0x7f1dc0003990 comp rx=0 tx=0).stop 2024-01-16T15:00:14.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 msgr2=0x7f1dd013ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:14.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 0x7f1dd013ce80 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1db800c430 tx=0x7f1db800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 shutdown_connections 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1da806b2c0 0x7f1da806d780 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1dd013e360 0x7f1dd0071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1dd013df80 0x7f1dd013ce80 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 --2- 172.21.15.53:0/2694698920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1dd00cf9a0 0x7f1dd013d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 >> 172.21.15.53:0/2694698920 conn(0x7f1dd006a800 msgr2=0x7f1dd015a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 shutdown_connections 2024-01-16T15:00:14.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:14.642+0000 7f1dd8000640 1 -- 172.21.15.53:0/2694698920 wait complete. 2024-01-16T15:00:15.045 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T14:59:57.407704+0000', 'modified': '2024-01-16T15:00:00.492292+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4245}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4245': {'gid': 4245, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6837/3309916828', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 3309916828}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 3309916828}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:00:15.046 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:00:15.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-01-16T15:00:15.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.301+0000 7f84c8a00640 1 Processor -- start 2024-01-16T15:00:15.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.302+0000 7f84c8a00640 1 -- start start 2024-01-16T15:00:15.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.303+0000 7f84c8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 0x7f84c40d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.303+0000 7f84c8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c40cf6a0 con 0x7f84c4160890 2024-01-16T15:00:15.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.303+0000 7f84c8a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84c40d1430 con 0x7f84c4160c70 2024-01-16T15:00:15.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.303+0000 7f84c8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b4370 con 0x7f84c40c6be0 2024-01-16T15:00:15.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.303+0000 7f84c2800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f84c4160890 0x7f84c40d1670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39076/0 (socket says 172.21.15.53:39076) 2024-01-16T15:00:15.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c2800640 1 -- 172.21.15.53:0/4013928776 learned_addr learned my addr 172.21.15.53:0/4013928776 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:15.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c3200640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 0x7f84c40d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164720703 0 0) 0x7f84c40cf6a0 con 0x7f84c4160890 2024-01-16T15:00:15.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a4003520 con 0x7f84c4160890 2024-01-16T15:00:15.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2923728354 0 0) 0x7f84a4003520 con 0x7f84c4160890 2024-01-16T15:00:15.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 msgr2=0x7f84c40d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 0x7f84c40d3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 >> v1:172.21.15.146:6789/0 conn(0x7f84c4160c70 legacy=0x7f84c40cfb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c40ce7c0 con 0x7f84c4160890 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.304+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f84b4001670 con 0x7f84c4160890 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 >> v1:172.21.15.53:6789/0 conn(0x7f84c4160890 legacy=0x7f84c40d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 0x7f84a40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a4003f70 0x7f84a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84a4005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b4370 con 0x7f84a4004d00 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a400bc10 con 0x7f84a4003f70 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a400bdd0 con 0x7f84c40c6be0 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c3200640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84a4005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c2800640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 0x7f84a40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c1e00640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a4003f70 0x7f84a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c3200640 1 -- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 msgr2=0x7f84a40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c3200640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c40c6be0 0x7f84a40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c3200640 1 -- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a4003f70 msgr2=0x7f84a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c3200640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84a4003f70 0x7f84a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.305+0000 7f84c3200640 1 -- 172.21.15.53:0/4013928776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4003520 con 0x7f84a4004d00 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.306+0000 7f84c3200640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84a4005160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009f90 tx=0x7f84a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38fed3b2dffe8881 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc02cbd0 con 0x7f84a4004d00 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84bc00f040 con 0x7f84a4004d00 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c1400640 1 -- 172.21.15.53:0/4013928776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84bc034440 con 0x7f84a4004d00 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c8a00640 1 -- 172.21.15.53:0/4013928776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 msgr2=0x7f84a4005160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c8a00640 1 --2- 172.21.15.53:0/4013928776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84a4005160 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f84bc009f90 tx=0x7f84a40057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c3200640 1 -- 172.21.15.53:0/4013928776 reap_dead start 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c8a00640 1 -- 172.21.15.53:0/4013928776 shutdown_connections 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.307+0000 7f84c8a00640 1 -- 172.21.15.53:0/4013928776 >> 172.21.15.53:0/4013928776 conn(0x7f84c40d3090 msgr2=0x7f84c40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.308+0000 7f84c8a00640 1 -- 172.21.15.53:0/4013928776 shutdown_connections 2024-01-16T15:00:15.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.308+0000 7f84c8a00640 1 -- 172.21.15.53:0/4013928776 wait complete. 2024-01-16T15:00:15.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.309+0000 7f84c8a00640 1 Processor -- start 2024-01-16T15:00:15.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.310+0000 7f84c8a00640 1 -- start start 2024-01-16T15:00:15.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.310+0000 7f84c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84c414bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.310+0000 7f84c8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c414df90 0x7f84c40b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.310+0000 7f84c8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c40b4ef0 0x7f84c41efe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.310+0000 7f84c8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40a0d30 con 0x7f84a4004d00 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.310+0000 7f84c8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84c40b5370 con 0x7f84c40b4ef0 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84c415f050 con 0x7f84c414df90 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84c414bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84c414bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52902/0 (socket says 172.21.15.53:52902) 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 -- 172.21.15.53:0/3517724684 learned_addr learned my addr 172.21.15.53:0/3517724684 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 -- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c414df90 msgr2=0x7f84c40b49b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c414df90 0x7f84c40b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 -- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c40b4ef0 msgr2=0x7f84c41efe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c40b4ef0 0x7f84c41efe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 -- 172.21.15.53:0/3517724684 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c414e4c0 con 0x7f84a4004d00 2024-01-16T15:00:15.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.311+0000 7f84c2800640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84c414bf00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84b400e7e0 tx=0x7f84b400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.312+0000 7f84baa00640 1 -- 172.21.15.53:0/3517724684 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b400ee90 con 0x7f84a4004d00 2024-01-16T15:00:15.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.312+0000 7f84baa00640 1 -- 172.21.15.53:0/3517724684 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b400c7f0 con 0x7f84a4004d00 2024-01-16T15:00:15.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.312+0000 7f84baa00640 1 -- 172.21.15.53:0/3517724684 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b400cba0 con 0x7f84a4004d00 2024-01-16T15:00:15.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.312+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84c414e7a0 con 0x7f84a4004d00 2024-01-16T15:00:15.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.312+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84c414ed60 con 0x7f84a4004d00 2024-01-16T15:00:15.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.314+0000 7f84baa00640 1 -- 172.21.15.53:0/3517724684 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f84b400cd00 con 0x7f84a4004d00 2024-01-16T15:00:15.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.315+0000 7f84baa00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f849c06b2c0 0x7f849c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.315+0000 7f84baa00640 1 -- 172.21.15.53:0/3517724684 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84b4089bc0 con 0x7f84a4004d00 2024-01-16T15:00:15.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.316+0000 7f84c1e00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f849c06b2c0 0x7f849c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.316+0000 7f84cc1ff740 1 Processor -- start 2024-01-16T15:00:15.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.316+0000 7f84cc1ff740 1 -- start start 2024-01-16T15:00:15.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84c1e00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f849c06b2c0 0x7f849c06d780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84ac007fd0 tx=0x7f84ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d7dd450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d7dc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d820730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d73b880 con 0x558b1d7d7d70 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d74aae0 con 0x558b1d7dd070 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.317+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b1d7481d0 con 0x558b1d7dc7b0 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d7dc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d7dc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52910/0 (socket says 172.21.15.53:52910) 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 -- 172.21.15.53:0/1594770020 learned_addr learned my addr 172.21.15.53:0/1594770020 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c2800640 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d7dd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c3200640 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d820730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 -- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 msgr2=0x558b1d820730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d820730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 -- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 msgr2=0x558b1d7dd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d7dd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 -- 172.21.15.53:0/1594770020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b1d70b860 con 0x558b1d7d7d70 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c3200640 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d820730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.318+0000 7f84c1e00640 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d7dc270 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005340 tx=0x7f84ac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d743593d3025b193 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84b3400640 1 -- 172.21.15.53:0/1594770020 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac021450 con 0x558b1d7d7d70 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84b3400640 1 -- 172.21.15.53:0/1594770020 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac0215b0 con 0x558b1d7d7d70 2024-01-16T15:00:15.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84b3400640 1 -- 172.21.15.53:0/1594770020 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac021920 con 0x558b1d7d7d70 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 -- 172.21.15.53:0/1594770020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 msgr2=0x558b1d7dc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d7dc270 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005340 tx=0x7f84ac011000 comp rx=0 tx=0).stop 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 -- 172.21.15.53:0/1594770020 shutdown_connections 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d820730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d7dc270 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/1594770020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d7dd450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 -- 172.21.15.53:0/1594770020 >> 172.21.15.53:0/1594770020 conn(0x558b1d79ec70 msgr2=0x558b1d822b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 -- 172.21.15.53:0/1594770020 shutdown_connections 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.319+0000 7f84cc1ff740 1 -- 172.21.15.53:0/1594770020 wait complete. 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.320+0000 7f84cc1ff740 1 -- start start 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.320+0000 7f84cc1ff740 1 Processor -- start 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d82aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d827230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d827770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d743ac0 con 0x558b1d7d7d70 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d6e02f0 con 0x558b1d7dd070 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b1d6fdc70 con 0x558b1d7dc7b0 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.321+0000 7f84cc1ff740 10 client.0 _ll_register_callbacks cb 0x558b1d7023b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d82aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d82aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52914/0 (socket says 172.21.15.53:52914) 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 -- 172.21.15.53:0/709491281 learned_addr learned my addr 172.21.15.53:0/709491281 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 -- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 msgr2=0x558b1d827230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d827230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 -- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 msgr2=0x558b1d827770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d827770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 -- 172.21.15.53:0/709491281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ac00f040 con 0x558b1d7d7d70 2024-01-16T15:00:15.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84c2800640 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d82aa40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84b400e7b0 tx=0x7f84b405c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cdf71a3afc031dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84abe00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:00:15.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84abe00640 1 -- 172.21.15.53:0/709491281 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4057d40 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84abe00640 1 -- 172.21.15.53:0/709491281 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84b4056ab0 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.322+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b1d73caa0 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84abe00640 1 -- 172.21.15.53:0/709491281 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84b40604c0 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 10 client.4608 fetch_fsmap learned FSMap version 7 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x558b1d742140 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84abe00640 1 -- 172.21.15.53:0/709491281 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558b1d73caa0 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84abe00640 1 -- 172.21.15.53:0/709491281 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f84b4078070 con 0x558b1d7d7d70 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 10 client.4608 fetch_fsmap finished waiting for FSMap version 7 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 10 client.4608 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 20 client.4608 populate_metadata read hostname 'smithi053' 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d81b370 0x558b1d802d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 4 client.4608 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.323+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 --> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b1d73c7c0 con 0x558b1d81b370 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.324+0000 7f84c1e00640 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d81b370 0x558b1d802d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.325+0000 7f84c1e00640 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d81b370 0x558b1d802d10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19403f49acf584f2 server_cookie=cf9afbe5d1b6e7ed in_seq=0 out_seq=0 2024-01-16T15:00:15.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.325+0000 7f84abe00640 10 client.4608 ms_handle_connect on v2:172.21.15.146:6835/3309916828 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84abe00640 1 -- 172.21.15.53:0/709491281 <== mds.0 v2:172.21.15.146:6835/3309916828 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558b1d73c7c0 con 0x558b1d81b370 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84abe00640 10 client.4608 handle_command_reply: tid=0 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 client.4608 shutdown 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 msgr2=0x558b1d82aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d82aa40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f84b400e7b0 tx=0x7f84b405c860 comp rx=0 tx=0).stop 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 shutdown_connections 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d81b370 0x558b1d802d10 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d827770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dc7b0 0x558b1d827230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/709491281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7d7d70 0x558b1d82aa40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84abe00640 0 client.4608 ms_handle_reset on v2:172.21.15.146:6835/3309916828 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.327+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 >> 172.21.15.53:0/709491281 conn(0x558b1d79ec70 msgr2=0x558b1d7dd450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.328+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 shutdown_connections 2024-01-16T15:00:15.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.328+0000 7f84cc1ff740 1 -- 172.21.15.53:0/709491281 wait complete. 2024-01-16T15:00:15.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.328+0000 7f84cc1ff740 20 client.4608 trim_cache size 0 max 16384 2024-01-16T15:00:15.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.345+0000 7f84cc1ff740 1 Processor -- start 2024-01-16T15:00:15.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.345+0000 7f84cc1ff740 1 -- start start 2024-01-16T15:00:15.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7d7d70 0x558b1d802d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d828f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dd070 0x558b1d8273d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d4a6f20 con 0x558b1d7dc7b0 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d74c2f0 con 0x558b1d7d7d70 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b1d7d3950 con 0x558b1d7dd070 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d828f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d828f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52918/0 (socket says 172.21.15.53:52918) 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.346+0000 7f84c1e00640 1 -- 172.21.15.53:0/898761933 learned_addr learned my addr 172.21.15.53:0/898761933 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c2800640 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7d7d70 0x558b1d802d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c3200640 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dd070 0x558b1d8273d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c1e00640 1 -- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dd070 msgr2=0x558b1d8273d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c1e00640 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dd070 0x558b1d8273d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c1e00640 1 -- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7d7d70 msgr2=0x558b1d802d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c1e00640 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7d7d70 0x558b1d802d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c1e00640 1 -- 172.21.15.53:0/898761933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b1d6dfb80 con 0x558b1d7dc7b0 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c2800640 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7d7d70 0x558b1d802d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84c1e00640 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d828f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005310 tx=0x7f84ac005000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c143d48f96b443 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.347+0000 7f84b3e00640 1 -- 172.21.15.53:0/898761933 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac028070 con 0x558b1d7dc7b0 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84b3e00640 1 -- 172.21.15.53:0/898761933 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac029040 con 0x558b1d7dc7b0 2024-01-16T15:00:15.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84b3e00640 1 -- 172.21.15.53:0/898761933 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac02ca90 con 0x558b1d7dc7b0 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 -- 172.21.15.53:0/898761933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 msgr2=0x558b1d828f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d828f80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f84ac005310 tx=0x7f84ac005000 comp rx=0 tx=0).stop 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 -- 172.21.15.53:0/898761933 shutdown_connections 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7dd070 0x558b1d8273d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d828f80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/898761933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7d7d70 0x558b1d802d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 -- 172.21.15.53:0/898761933 >> 172.21.15.53:0/898761933 conn(0x558b1d79ec70 msgr2=0x558b1d7db960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 -- 172.21.15.53:0/898761933 shutdown_connections 2024-01-16T15:00:15.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.348+0000 7f84cc1ff740 1 -- 172.21.15.53:0/898761933 wait complete. 2024-01-16T15:00:15.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.349+0000 7f84cc1ff740 1 -- start start 2024-01-16T15:00:15.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.349+0000 7f84cc1ff740 1 Processor -- start 2024-01-16T15:00:15.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7d7d70 0x558b1d823100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d81eb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d81f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d88e250 con 0x558b1d7dc7b0 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b1d7d74b0 con 0x558b1d7dd070 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558b1d83b560 con 0x558b1d7d7d70 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84cc1ff740 10 client.0 _ll_register_callbacks cb 0x7f84b4058190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.350+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d81eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d81eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52926/0 (socket says 172.21.15.53:52926) 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 -- 172.21.15.53:0/869822457 learned_addr learned my addr 172.21.15.53:0/869822457 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c2800640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7d7d70 0x558b1d823100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c3200640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d81f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 -- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7d7d70 msgr2=0x558b1d823100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7d7d70 0x558b1d823100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 -- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 msgr2=0x558b1d81f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d81f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 -- 172.21.15.53:0/869822457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558b1d70b860 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84c1e00640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d81eb60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84ac023040 tx=0x7f84ac007a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f32c174fdacf81 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.351+0000 7f84abe00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84abe00640 1 -- 172.21.15.53:0/869822457 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac028040 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84abe00640 1 -- 172.21.15.53:0/869822457 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f84ac02c2a0 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f84ac0215b0 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84abe00640 1 -- 172.21.15.53:0/869822457 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84ac021e60 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 10 client.4614 fetch_fsmap learned FSMap version 7 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x558b1d705170 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84abe00640 1 -- 172.21.15.53:0/869822457 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac0215b0 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84abe00640 1 -- 172.21.15.53:0/869822457 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 7) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f84ac02c460 con 0x558b1d7dc7b0 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 10 client.4614 fetch_fsmap finished waiting for FSMap version 7 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 10 client.4614 resolve_mds: resolved name 'b' to daemon mds.b 2024-01-16T15:00:15.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 20 client.4614 populate_metadata read hostname 'smithi053' 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d88db50 0x558b1d825ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 4 client.4614 mds_command: new command op to 4245 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.352+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 --> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558b1d4446c0 con 0x558b1d88db50 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.353+0000 7f84c2800640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d88db50 0x558b1d825ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84c2800640 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d88db50 0x558b1d825ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f09117f9c95fdad server_cookie=ed7c9309654422fc in_seq=0 out_seq=0 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84abe00640 10 client.4614 ms_handle_connect on v2:172.21.15.146:6835/3309916828 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84abe00640 1 -- 172.21.15.53:0/869822457 <== mds.0 v2:172.21.15.146:6835/3309916828 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+318 (crc 0 0 0) 0x558b1d4446c0 con 0x558b1d88db50 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84abe00640 10 client.4614 handle_command_reply: tid=0 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84cc1ff740 1 client.4614 shutdown 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 msgr2=0x558b1d81eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.354+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d81eb60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84ac023040 tx=0x7f84ac007a30 comp rx=0 tx=0).stop 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 shutdown_connections 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:6835/3309916828,v1:172.21.15.146:6837/3309916828] conn(0x558b1d88db50 0x558b1d825ca0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x558b1d7dd070 0x558b1d81f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558b1d7dc7b0 0x558b1d81eb60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 --2- 172.21.15.53:0/869822457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558b1d7d7d70 0x558b1d823100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84abe00640 0 client.4614 ms_handle_reset on v2:172.21.15.146:6835/3309916828 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 >> 172.21.15.53:0/869822457 conn(0x558b1d79ec70 msgr2=0x558b1d844480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 shutdown_connections 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 1 -- 172.21.15.53:0/869822457 wait complete. 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84cc1ff740 20 client.4614 trim_cache size 0 max 16384 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:00:15.339 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4245, 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 17.425041512, 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 7, 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 22, 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 22, 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 22.525247856 2024-01-16T15:00:15.340 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f849c06b2c0 msgr2=0x7f849c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84c8a00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f849c06b2c0 0x7f849c06d780 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f84ac007fd0 tx=0x7f84ac009040 comp rx=0 tx=0).stop 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 msgr2=0x7f84c414bf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.355+0000 7f84c8a00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84c414bf00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84b400e7e0 tx=0x7f84b400eca0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 shutdown_connections 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f849c06b2c0 0x7f849c06d780 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84c40b4ef0 0x7f84c41efe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84c414df90 0x7f84c40b49b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 --2- 172.21.15.53:0/3517724684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84a4004d00 0x7f84c414bf00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 >> 172.21.15.53:0/3517724684 conn(0x7f84c40d3090 msgr2=0x7f84c40c7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 shutdown_connections 2024-01-16T15:00:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.356+0000 7f84c8a00640 1 -- 172.21.15.53:0/3517724684 wait complete. 2024-01-16T15:00:15.353 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-01-16T15:00:15.376 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-01-16T15:00:15.377 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-01-16T15:00:15.377 INFO:tasks.ceph_fuse:client.0 config is {} 2024-01-16T15:00:15.377 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T15:00:15.377 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:15.377 INFO:tasks.ceph_fuse:client.2 config is {} 2024-01-16T15:00:15.377 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T15:00:15.377 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:15.377 INFO:tasks.ceph_fuse:client.1 config is {} 2024-01-16T15:00:15.377 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T15:00:15.377 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:15.378 INFO:tasks.ceph_fuse:client.3 config is {} 2024-01-16T15:00:15.378 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-01-16T15:00:15.378 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:15.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.378 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:00:15.439 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-01-16T15:00:15.545 INFO:teuthology.orchestra.run.smithi053.stderr:Cannot find device "ceph-brx" 2024-01-16T15:00:15.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:00:15.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.550 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:00:15.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:15.609 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-01-16T15:00:15.707 INFO:teuthology.orchestra.run.smithi146.stderr:Cannot find device "ceph-brx" 2024-01-16T15:00:15.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:00:15.712 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-01-16T15:00:15.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:15.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.964+0000 7fef48a00640 1 Processor -- start 2024-01-16T15:00:15.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.964+0000 7fef48a00640 1 -- start start 2024-01-16T15:00:15.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef48a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 0x7fef440cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef48a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef44070e50 con 0x7fef4406d160 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef48a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef440ccea0 con 0x7fef44143c40 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef48a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef440723c0 con 0x7fef440cc500 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef41e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 0x7fef440cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef41e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 0x7fef440cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:52450/0 (socket says 172.21.15.53:52450) 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef41e00640 1 -- 172.21.15.53:0/300556843 learned_addr learned my addr 172.21.15.53:0/300556843 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.966+0000 7fef43200640 1 --1- 172.21.15.53:0/300556843 >> v1:172.21.15.53:6789/0 conn(0x7fef4406d160 0x7fef4406d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39082/0 (socket says 172.21.15.53:39082) 2024-01-16T15:00:15.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.967+0000 7fef41e00640 1 -- 172.21.15.53:0/300556843 >> v1:172.21.15.146:6789/0 conn(0x7fef44143c40 legacy=0x7fef44071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.967+0000 7fef41e00640 1 -- 172.21.15.53:0/300556843 >> v1:172.21.15.53:6789/0 conn(0x7fef4406d160 legacy=0x7fef4406d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.967+0000 7fef41e00640 1 -- 172.21.15.53:0/300556843 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef44144020 con 0x7fef440cc500 2024-01-16T15:00:15.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.967+0000 7fef41400640 1 -- 172.21.15.53:0/300556843 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224408670 0 0) 0x7fef44070e50 con 0x7fef4406d160 2024-01-16T15:00:15.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.968+0000 7fef41e00640 1 --2- 172.21.15.53:0/300556843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 0x7fef440cc960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0098a0 tx=0x7fef2c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d93b6a39b28bd215 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.968+0000 7fef41400640 1 -- 172.21.15.53:0/300556843 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c030840 con 0x7fef440cc500 2024-01-16T15:00:15.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.968+0000 7fef41400640 1 -- 172.21.15.53:0/300556843 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c0309a0 con 0x7fef440cc500 2024-01-16T15:00:15.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.968+0000 7fef41400640 1 -- 172.21.15.53:0/300556843 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c030d30 con 0x7fef440cc500 2024-01-16T15:00:15.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 -- 172.21.15.53:0/300556843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 msgr2=0x7fef440cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 --2- 172.21.15.53:0/300556843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 0x7fef440cc960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fef2c0098a0 tx=0x7fef2c02b980 comp rx=0 tx=0).stop 2024-01-16T15:00:15.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 -- 172.21.15.53:0/300556843 shutdown_connections 2024-01-16T15:00:15.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 --2- 172.21.15.53:0/300556843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef440cc500 0x7fef440cc960 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 -- 172.21.15.53:0/300556843 >> 172.21.15.53:0/300556843 conn(0x7fef4406a800 msgr2=0x7fef4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:15.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 -- 172.21.15.53:0/300556843 shutdown_connections 2024-01-16T15:00:15.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.969+0000 7fef48a00640 1 -- 172.21.15.53:0/300556843 wait complete. 2024-01-16T15:00:15.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.971+0000 7fef48a00640 1 Processor -- start 2024-01-16T15:00:15.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.971+0000 7fef48a00640 1 -- start start 2024-01-16T15:00:15.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef48a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef4406d160 0x7fef44139d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef48a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 0x7fef441418d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef48a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef44143c40 0x7fef44141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef48a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef440b3b50 con 0x7fef440cc500 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef48a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef440b2ed0 con 0x7fef44143c40 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef48a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef440b31b0 con 0x7fef4406d160 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef41e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 0x7fef441418d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.972+0000 7fef41e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 0x7fef441418d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52928/0 (socket says 172.21.15.53:52928) 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 -- 172.21.15.53:0/2942198268 learned_addr learned my addr 172.21.15.53:0/2942198268 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef42800640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef4406d160 0x7fef44139d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef43200640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef44143c40 0x7fef44141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 -- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef4406d160 msgr2=0x7fef44139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef4406d160 0x7fef44139d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 -- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef44143c40 msgr2=0x7fef44141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef44143c40 0x7fef44141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 -- 172.21.15.53:0/2942198268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0094d0 con 0x7fef440cc500 2024-01-16T15:00:15.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.973+0000 7fef41e00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 0x7fef441418d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fef2c000c00 tx=0x7fef2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.974+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0307f0 con 0x7fef440cc500 2024-01-16T15:00:15.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.974+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef2c043400 con 0x7fef440cc500 2024-01-16T15:00:15.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.974+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef2c0437b0 con 0x7fef440cc500 2024-01-16T15:00:15.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.974+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef440b3490 con 0x7fef440cc500 2024-01-16T15:00:15.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.974+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef4413a240 con 0x7fef440cc500 2024-01-16T15:00:15.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.975+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef4413b5b0 con 0x7fef440cc500 2024-01-16T15:00:15.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.976+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fef2c043910 con 0x7fef440cc500 2024-01-16T15:00:15.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.977+0000 7fef3aa00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef1c06b2c0 0x7fef1c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:15.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.977+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef2c03e070 con 0x7fef440cc500 2024-01-16T15:00:15.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.977+0000 7fef42800640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef1c06b2c0 0x7fef1c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:15.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.978+0000 7fef42800640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef1c06b2c0 0x7fef1c06d780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fef340012e0 tx=0x7fef34009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:15.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:15.981+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef2c077cf0 con 0x7fef440cc500 2024-01-16T15:00:16.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.223+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fef44067450 con 0x7fef440cc500 2024-01-16T15:00:16.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.224+0000 7fef3aa00640 1 -- 172.21.15.53:0/2942198268 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fef2c07b310 con 0x7fef440cc500 2024-01-16T15:00:16.204 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:16.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef1c06b2c0 msgr2=0x7fef1c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef1c06b2c0 0x7fef1c06d780 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fef340012e0 tx=0x7fef34009040 comp rx=0 tx=0).stop 2024-01-16T15:00:16.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 msgr2=0x7fef441418d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 0x7fef441418d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fef2c000c00 tx=0x7fef2c004730 comp rx=0 tx=0).stop 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 shutdown_connections 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef1c06b2c0 0x7fef1c06d780 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef44143c40 0x7fef44141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef440cc500 0x7fef441418d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 --2- 172.21.15.53:0/2942198268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef4406d160 0x7fef44139d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 >> 172.21.15.53:0/2942198268 conn(0x7fef4406a800 msgr2=0x7fef4406d590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 shutdown_connections 2024-01-16T15:00:16.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:16.227+0000 7fef48a00640 1 -- 172.21.15.53:0/2942198268 wait complete. 2024-01-16T15:00:16.228 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:16.228 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:16.228 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:00:16.229 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:00:16.229 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:16.229 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:16.229 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:16.229 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:16.229 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:16.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.229 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:00:16.284 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:16.284 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:16.284 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:16.284 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:16.284 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42634sec preferred_lft 42634sec 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:16.285 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:16.286 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:16.286 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:00:16.287 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T15:00:16.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add name ceph-brx type bridge 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr flush dev ceph-brx 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set ceph-brx up 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T15:00:16.287 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:16.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.749 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:16.834 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:00:16.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.837 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:00:16.892 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:00:16.892 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:16.893 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:00:16.893 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:00:16.893 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:16.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:16.893 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:16.893 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:16.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:00:16.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:00:16.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:00:16.894 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:17.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.612 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:00:17.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.660 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-01-16T15:00:17.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.751 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:17.751 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:17.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:17.751 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-01-16T15:00:17.751 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:17.886 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-01-16T15:00:17.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-01-16T15:00:17.886 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:18.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.146 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:18.146 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:18.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:00:18.146 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T15:00:18.146 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:18.306 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:00:18.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:18.306 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:18.351 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:00:18.352 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:18.352 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:18.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:00:18.525 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:18.572 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:00:18.572 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:00:18.572 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:00:18.572 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:00:18.572 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:00:18.572 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:18.573 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:18.574 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:18.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.575 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:00:18.664 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:18.664 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:00:18.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:00:18.744 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:00:18.792 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:18.793 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:18.794 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:18.794 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:18.794 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:00:18.794 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:18.794 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:18.795 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:18.795 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:00:18.859 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:00:18.879+0000 7f207ba39480 -1 init, newargv = 0x55f18fef3bc0 newargc=15 2024-01-16T15:00:18.860 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[42894]: starting ceph client 2024-01-16T15:00:18.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[42894]: starting fuse 2024-01-16T15:00:18.880 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:00:18.880 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:00:18.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:18.880 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:18.881 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:00:18.882 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:00:19.083 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.42894.asok 2024-01-16T15:00:19.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:19.102 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42894.asok 2024-01-16T15:00:19.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:19.102 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42894.asok status 2024-01-16T15:00:19.337 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:00:19.337 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:00:19.337 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:19.337 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "42894", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4629, 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 4629 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:00:19.338 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 922280206 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 922280206 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.4629 192.168.0.1:0/922280206", 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/922280206", 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 7, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 22, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:00:19.339 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:00:19.340 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:00:19.352 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:19.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.583+0000 7f6553e00640 1 Processor -- start 2024-01-16T15:00:19.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.583+0000 7f6553e00640 1 -- start start 2024-01-16T15:00:19.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.585+0000 7f6553e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.585+0000 7f6553e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f655406d160 0x7f655406d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.585+0000 7f6553e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6554070e50 con 0x7f6554143c40 2024-01-16T15:00:19.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.585+0000 7f6553e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65540723c0 con 0x7f655406d160 2024-01-16T15:00:19.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.585+0000 7f6553e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6554144020 con 0x7f65540cc500 2024-01-16T15:00:19.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:52460/0 (socket says 172.21.15.53:52460) 2024-01-16T15:00:19.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 -- 172.21.15.53:0/1456889875 learned_addr learned my addr 172.21.15.53:0/1456889875 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:19.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f6553400640 1 --2- 172.21.15.53:0/1456889875 >> v2:172.21.15.146:3300/0 conn(0x7f655406d160 0x7f655406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 -- 172.21.15.53:0/1456889875 >> v2:172.21.15.146:3300/0 conn(0x7f655406d160 msgr2=0x7f655406d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:19.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 --2- 172.21.15.53:0/1456889875 >> v2:172.21.15.146:3300/0 conn(0x7f655406d160 0x7f655406d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 -- 172.21.15.53:0/1456889875 >> v1:172.21.15.146:6789/0 conn(0x7f6554143c40 legacy=0x7f6554071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.586+0000 7f654be00640 1 -- 172.21.15.53:0/1456889875 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65540ccea0 con 0x7f65540cc500 2024-01-16T15:00:19.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.588+0000 7f654be00640 1 --2- 172.21.15.53:0/1456889875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540cc960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d46a5f11d5691b94 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.588+0000 7f6552000640 1 -- 172.21.15.53:0/1456889875 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c031640 con 0x7f65540cc500 2024-01-16T15:00:19.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6552000640 1 -- 172.21.15.53:0/1456889875 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f653c0317a0 con 0x7f65540cc500 2024-01-16T15:00:19.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6552000640 1 -- 172.21.15.53:0/1456889875 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c031b30 con 0x7f65540cc500 2024-01-16T15:00:19.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6553e00640 1 -- 172.21.15.53:0/1456889875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 msgr2=0x7f65540cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6553e00640 1 --2- 172.21.15.53:0/1456889875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540cc960 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f653c009870 tx=0x7f653c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6553e00640 1 -- 172.21.15.53:0/1456889875 shutdown_connections 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6553e00640 1 --2- 172.21.15.53:0/1456889875 >> v2:172.21.15.146:3300/0 conn(0x7f655406d160 0x7f655406d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6553e00640 1 --2- 172.21.15.53:0/1456889875 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540cc960 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.589+0000 7f6553e00640 1 -- 172.21.15.53:0/1456889875 >> 172.21.15.53:0/1456889875 conn(0x7f655406a800 msgr2=0x7f655406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.590+0000 7f6553e00640 1 -- 172.21.15.53:0/1456889875 shutdown_connections 2024-01-16T15:00:19.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.590+0000 7f6553e00640 1 -- 172.21.15.53:0/1456889875 wait complete. 2024-01-16T15:00:19.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.591+0000 7f6553e00640 1 Processor -- start 2024-01-16T15:00:19.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.591+0000 7f6553e00640 1 -- start start 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.592+0000 7f6553e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 0x7f65540b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.592+0000 7f6553e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.592+0000 7f6553e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6554143c40 0x7f655413c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.592+0000 7f6553e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65541e2780 con 0x7f655406d160 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.592+0000 7f6553e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65541e1b70 con 0x7f6554143c40 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.592+0000 7f6553e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65541e1e50 con 0x7f65540cc500 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 0x7f65540b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 0x7f65540b29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52944/0 (socket says 172.21.15.53:52944) 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 -- 172.21.15.53:0/1816778610 learned_addr learned my addr 172.21.15.53:0/1816778610 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:19.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6553400640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6554143c40 0x7f655413c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f654be00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 -- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 msgr2=0x7f65540adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 -- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6554143c40 msgr2=0x7f655413c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6554143c40 0x7f655413c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f6552a00640 1 -- 172.21.15.53:0/1816778610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f653c0094d0 con 0x7f655406d160 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.593+0000 7f654be00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.594+0000 7f6552a00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 0x7f65540b29c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f654400c3c0 tx=0x7f654400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.594+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65440052d0 con 0x7f655406d160 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.594+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6544005450 con 0x7f655406d160 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.594+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6544007260 con 0x7f655406d160 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.594+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65541e2130 con 0x7f655406d160 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.594+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65540b2f00 con 0x7f655406d160 2024-01-16T15:00:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.595+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65540b30c0 con 0x7f655406d160 2024-01-16T15:00:19.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.596+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f65440073c0 con 0x7f655406d160 2024-01-16T15:00:19.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.598+0000 7f654aa00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f652c06b2c0 0x7f652c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:19.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.598+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6544088ff0 con 0x7f655406d160 2024-01-16T15:00:19.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.598+0000 7f654be00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f652c06b2c0 0x7f652c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:19.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.599+0000 7f654be00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f652c06b2c0 0x7f652c06d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f653c036000 tx=0x7f653c03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:19.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.603+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65440bbe90 con 0x7f655406d160 2024-01-16T15:00:19.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.834+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f65540cdf30 con 0x7f655406d160 2024-01-16T15:00:19.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.835+0000 7f654aa00640 1 -- 172.21.15.53:0/1816778610 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f65540b30c0 con 0x7f655406d160 2024-01-16T15:00:19.815 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f652c06b2c0 msgr2=0x7f652c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f652c06b2c0 0x7f652c06d780 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f653c036000 tx=0x7f653c03c040 comp rx=0 tx=0).stop 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 msgr2=0x7f65540b29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 0x7f65540b29c0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f654400c3c0 tx=0x7f654400c880 comp rx=0 tx=0).stop 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 shutdown_connections 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f652c06b2c0 0x7f652c06d780 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.837+0000 7f6553e00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6554143c40 0x7f655413c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.838+0000 7f6553e00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65540cc500 0x7f65540adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.838+0000 7f6553e00640 1 --2- 172.21.15.53:0/1816778610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f655406d160 0x7f65540b29c0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.838+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 >> 172.21.15.53:0/1816778610 conn(0x7f655406a800 msgr2=0x7f65540aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.838+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 shutdown_connections 2024-01-16T15:00:19.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:19.838+0000 7f6553e00640 1 -- 172.21.15.53:0/1816778610 wait complete. 2024-01-16T15:00:19.836 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:19.836 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:19.836 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:00:19.836 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-01-16T15:00:19.836 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:19.836 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:19.837 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:19.837 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:19.837 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-01-16T15:00:19.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:19.837 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:00:19.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:19.887 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:19.890 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:19.890 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:19.890 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:19.890 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:19.890 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:19.890 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42630sec preferred_lft 42630sec 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:19.891 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:00:19.892 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:19.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:19.893 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:20.017 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:00:20.020 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.021 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:00:20.121 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:00:20.121 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:20.122 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:00:20.122 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:00:20.122 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:20.122 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.123 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:20.123 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:20.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:00:20.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:00:20.123 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:00:20.123 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:20.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.319 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:00:20.372 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:00:20.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.374 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list-id 2024-01-16T15:00:20.428 INFO:teuthology.orchestra.run.smithi053.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-01-16T15:00:20.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.429 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:20.429 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:20.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:20.429 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-01-16T15:00:20.429 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:20.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.577 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T15:00:20.674 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:20.674 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 7e:29:66:a7:fc:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c29:66ff:fea7:fcbe/64 scope link 2024-01-16T15:00:20.675 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.680 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-01-16T15:00:20.777 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 7e:29:66:a7:fc:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c29:66ff:fea7:fcbe/64 scope link 2024-01-16T15:00:20.778 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:20.783 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-01-16T15:00:20.783 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:20.783 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:20.783 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:20.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-01-16T15:00:20.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T15:00:20.783 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-01-16T15:00:20.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-01-16T15:00:20.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-01-16T15:00:20.784 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:21.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:21.071 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:00:21.071 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:00:21.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.1 up 2024-01-16T15:00:21.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T15:00:21.071 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:00:21.252 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-01-16T15:00:21.252 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:21.252 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:21.302 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-01-16T15:00:21.302 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:21.302 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:21.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:00:21.459 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:21.519 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:00:21.519 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:00:21.519 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:00:21.519 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:00:21.520 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:21.521 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:21.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:21.522 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:00:21.576 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:00:21.576 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [39] 2024-01-16T15:00:21.576 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-01-16T15:00:21.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:00:21.666 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:00:21.711 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:21.712 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:00:21.713 INFO:teuthology.orchestra.run.smithi053.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:21.713 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:21.713 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:00:21.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:21.713 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:00:21.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi053.stderr:2024-01-16T15:00:21.800+0000 7fca25e35480 -1 init, newargv = 0x555d53a4a8e0 newargc=15 2024-01-16T15:00:21.780 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi053.stderr:ceph-fuse[43406]: starting ceph client 2024-01-16T15:00:21.790 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi053.stderr:ceph-fuse[43406]: starting fuse 2024-01-16T15:00:21.799 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:00:21.799 INFO:teuthology.orchestra.run.smithi053.stdout:42 2024-01-16T15:00:21.799 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39, 42] 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:00:21.800 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.2")) 2024-01-16T15:00:21.801 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:00:22.017 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.2.43406.asok 2024-01-16T15:00:22.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:22.034 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.43406.asok 2024-01-16T15:00:22.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:22.035 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.43406.asok status 2024-01-16T15:00:22.275 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:00:22.276 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:00:22.276 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:22.276 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "2", 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "43406", 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:22.277 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4641, 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:00:22.278 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 4641 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.2:0", 2024-01-16T15:00:22.279 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 3688245748 2024-01-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:00:22.280 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:00:22.281 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.2:0", 2024-01-16T15:00:22.282 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 3688245748 2024-01-16T15:00:22.282 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:00:22.282 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.4641 192.168.0.2:0/3688245748", 2024-01-16T15:00:22.282 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.2:0/3688245748", 2024-01-16T15:00:22.283 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:00:22.283 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 7, 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 22, 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:00:22.284 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:00:22.290 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:22.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.538+0000 7fe67d600640 1 Processor -- start 2024-01-16T15:00:22.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.539+0000 7fe67d600640 1 -- start start 2024-01-16T15:00:22.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.540+0000 7fe67d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6780cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.540+0000 7fe67d600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe678070e50 con 0x7fe67806d160 2024-01-16T15:00:22.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.540+0000 7fe67d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780ccea0 con 0x7fe678143c40 2024-01-16T15:00:22.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.540+0000 7fe67d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780723c0 con 0x7fe6780cc500 2024-01-16T15:00:22.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.541+0000 7fe67cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe67806d160 0x7fe67806d590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33438/0 (socket says 172.21.15.53:33438) 2024-01-16T15:00:22.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.541+0000 7fe67cc00640 1 -- 172.21.15.53:0/4062750080 learned_addr learned my addr 172.21.15.53:0/4062750080 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.541+0000 7fe66fe00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6780cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.541+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2793676808 0 0) 0x7fe678070e50 con 0x7fe67806d160 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.541+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658003520 con 0x7fe67806d160 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4146431005 0 0) 0x7fe658003520 con 0x7fe67806d160 2024-01-16T15:00:22.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 msgr2=0x7fe6780cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6780cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 >> v1:172.21.15.146:6789/0 conn(0x7fe678143c40 legacy=0x7fe678071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe678144020 con 0x7fe67806d160 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe6700019f0 con 0x7fe67806d160 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 >> v1:172.21.15.53:6789/0 conn(0x7fe67806d160 legacy=0x7fe67806d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe658005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780723c0 con 0x7fe6580039a0 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65800bae0 con 0x7fe658004d00 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65800bca0 con 0x7fe6780cc500 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.542+0000 7fe677e00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe67cc00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe658005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe66fe00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe677e00640 1 -- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 msgr2=0x7fe6580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe677e00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe677e00640 1 -- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 msgr2=0x7fe658005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe677e00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe658005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe677e00640 1 -- 172.21.15.53:0/4062750080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658003520 con 0x7fe6580039a0 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe67cc00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe658005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.543+0000 7fe66fe00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cc500 0x7fe6580047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.544+0000 7fe677e00640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe668001750 tx=0x7fe6580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6b85c0748658b59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.544+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66802bea0 con 0x7fe6580039a0 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.544+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66802e050 con 0x7fe6580039a0 2024-01-16T15:00:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.544+0000 7fe677400640 1 -- 172.21.15.53:0/4062750080 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6680035f0 con 0x7fe6580039a0 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67d600640 1 -- 172.21.15.53:0/4062750080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580039a0 msgr2=0x7fe658003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67d600640 1 --2- 172.21.15.53:0/4062750080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe668001750 tx=0x7fe6580057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67cc00640 1 -- 172.21.15.53:0/4062750080 reap_dead start 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67d600640 1 -- 172.21.15.53:0/4062750080 shutdown_connections 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67d600640 1 -- 172.21.15.53:0/4062750080 >> 172.21.15.53:0/4062750080 conn(0x7fe67806a800 msgr2=0x7fe67806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67d600640 1 -- 172.21.15.53:0/4062750080 shutdown_connections 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.545+0000 7fe67d600640 1 -- 172.21.15.53:0/4062750080 wait complete. 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.547+0000 7fe67d600640 1 Processor -- start 2024-01-16T15:00:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.547+0000 7fe67d600640 1 -- start start 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe67d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe6780aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe67d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 0x7fe6781f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe67d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f67d0 0x7fe6780b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe67d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780acfc0 con 0x7fe67815a4c0 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe67d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6781f6c50 con 0x7fe6580039a0 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe67d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780ac340 con 0x7fe6781f67d0 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 0x7fe6781f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe66fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 0x7fe6781f6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49380/0 (socket says 172.21.15.53:49380) 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe66fe00640 1 -- 172.21.15.53:0/2520878946 learned_addr learned my addr 172.21.15.53:0/2520878946 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:22.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.548+0000 7fe677e00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe6780aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe67cc00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f67d0 0x7fe6780b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe66fe00640 1 -- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f67d0 msgr2=0x7fe6780b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe66fe00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f67d0 0x7fe6780b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe66fe00640 1 -- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 msgr2=0x7fe6780aa4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe66fe00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe6780aa4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe66fe00640 1 -- 172.21.15.53:0/2520878946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780ac620 con 0x7fe67815a4c0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe67cc00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f67d0 0x7fe6780b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe66fe00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 0x7fe6781f6290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe6600108b0 tx=0x7fe660010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66000eb60 con 0x7fe67815a4c0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66000ecc0 con 0x7fe67815a4c0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe66001c400 con 0x7fe67815a4c0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6780ac900 con 0x7fe67815a4c0 2024-01-16T15:00:22.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.549+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6780aaa10 con 0x7fe67815a4c0 2024-01-16T15:00:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.551+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6780ad100 con 0x7fe67815a4c0 2024-01-16T15:00:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.552+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe66001c560 con 0x7fe67815a4c0 2024-01-16T15:00:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.553+0000 7fe674c00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006b2c0 0x7fe65006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.553+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe66008a1a0 con 0x7fe67815a4c0 2024-01-16T15:00:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.553+0000 7fe677e00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006b2c0 0x7fe65006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:22.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.554+0000 7fe677e00640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006b2c0 0x7fe65006d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe668033000 tx=0x7fe668009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.557+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe6780ad100 con 0x7fe67815a4c0 2024-01-16T15:00:22.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.792+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe6780ad100 con 0x7fe67815a4c0 2024-01-16T15:00:22.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.793+0000 7fe674c00640 1 -- 172.21.15.53:0/2520878946 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe6780ad100 con 0x7fe67815a4c0 2024-01-16T15:00:22.774 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.795+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006b2c0 msgr2=0x7fe65006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006b2c0 0x7fe65006d780 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe668033000 tx=0x7fe668009030 comp rx=0 tx=0).stop 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 msgr2=0x7fe6781f6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 0x7fe6781f6290 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe6600108b0 tx=0x7fe660010d70 comp rx=0 tx=0).stop 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 shutdown_connections 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006b2c0 0x7fe65006d780 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f67d0 0x7fe6780b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe67815a4c0 0x7fe6781f6290 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 --2- 172.21.15.53:0/2520878946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe6780aa4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 >> 172.21.15.53:0/2520878946 conn(0x7fe67806a800 msgr2=0x7fe6781404a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:22.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 shutdown_connections 2024-01-16T15:00:22.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:22.796+0000 7fe67d600640 1 -- 172.21.15.53:0/2520878946 wait complete. 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:22.797 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:22.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:22.798 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:00:22.856 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:22.856 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:22.857 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42507sec preferred_lft 42507sec 2024-01-16T15:00:22.858 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:00:22.858 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:22.858 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-01-16T15:00:22.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:22.858 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:22.858 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:22.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add name ceph-brx type bridge 2024-01-16T15:00:22.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr flush dev ceph-brx 2024-01-16T15:00:22.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set ceph-brx up 2024-01-16T15:00:22.858 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-01-16T15:00:22.859 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:23.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:23.293 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:23.373 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:00:23.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:23.377 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:00:23.436 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:00:23.436 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:23.436 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:23.437 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:23.437 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:23.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:23.437 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:23.437 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:23.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:23.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:23.437 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:23.438 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:24.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:24.146 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:00:24.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:24.197 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-01-16T15:00:24.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:24.249 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:24.249 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:24.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:24.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-01-16T15:00:24.249 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:24.390 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-01-16T15:00:24.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:24.391 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:24.391 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:24.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-01-16T15:00:24.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-01-16T15:00:24.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-01-16T15:00:24.391 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-01-16T15:00:24.392 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-01-16T15:00:24.392 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:24.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:24.666 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:24.666 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:24.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:00:24.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.0 master ceph-brx 2024-01-16T15:00:24.666 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:24.827 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:00:24.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:24.827 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:24.871 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:00:24.871 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:24.871 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:24.967 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:00:25.059 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:00:25.114 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:25.115 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:25.116 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:25.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:25.117 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:00:25.160 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:00:25.161 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:00:25.208 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:00:25.284 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:00:25.325 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:25.326 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:25.327 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:25.327 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:25.327 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:00:25.327 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:25.327 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:25.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:25.328 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:00:25.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:00:25.428+0000 7fba5da3a480 -1 init, newargv = 0x5625eac51aa0 newargc=15 2024-01-16T15:00:25.409 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[41177]: starting ceph client 2024-01-16T15:00:25.419 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[41177]: starting fuse 2024-01-16T15:00:26.514 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:00:26.615 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:26.671 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:00:26.671 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:00:26.671 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:00:26.672 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:26.673 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:26.674 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:26.674 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:26.674 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:00:26.728 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:00:26.728 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:00:26.728 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:26.728 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:00:26.728 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:00:26.728 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:00:26.728 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:00:26.728 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:00:26.729 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:00:26.730 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:26.730 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:26.730 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:00:26.730 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:00:26.902 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.41177.asok 2024-01-16T15:00:26.906 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:00:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:26.920 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.41177.asok 2024-01-16T15:00:26.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:26.921 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.41177.asok status 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "41177", 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:00:27.157 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4478, 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4478 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3847744315 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:00:27.158 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3847744315 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4478 192.168.0.1:0/3847744315", 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/3847744315", 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 7, 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 22, 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:00:27.159 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:00:27.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:00:27.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.421+0000 7f9974a00640 1 Processor -- start 2024-01-16T15:00:27.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.421+0000 7f9974a00640 1 -- start start 2024-01-16T15:00:27.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.423+0000 7f9974a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99700cc500 0x7f99700cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.423+0000 7f9974a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 0x7f997006d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.423+0000 7f9974a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9970070e50 con 0x7f9970143c40 2024-01-16T15:00:27.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.423+0000 7f9974a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99700723c0 con 0x7f99700cc500 2024-01-16T15:00:27.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.423+0000 7f9974a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99700cced0 con 0x7f997006d160 2024-01-16T15:00:27.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99700cc500 0x7f99700cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 0x7f997006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 0x7f997006d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:37306/0 (socket says 172.21.15.53:37306) 2024-01-16T15:00:27.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996f200640 1 -- 172.21.15.53:0/3528466675 learned_addr learned my addr 172.21.15.53:0/3528466675 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:27.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996f200640 1 -- 172.21.15.53:0/3528466675 >> v2:172.21.15.146:3300/0 conn(0x7f99700cc500 msgr2=0x7f99700cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:27.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996f200640 1 --2- 172.21.15.53:0/3528466675 >> v2:172.21.15.146:3300/0 conn(0x7f99700cc500 0x7f99700cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.424+0000 7f996f200640 1 -- 172.21.15.53:0/3528466675 >> v1:172.21.15.146:6789/0 conn(0x7f9970143c40 legacy=0x7f9970071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.425+0000 7f996f200640 1 -- 172.21.15.53:0/3528466675 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9970144020 con 0x7f997006d160 2024-01-16T15:00:27.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.426+0000 7f996f200640 1 --2- 172.21.15.53:0/3528466675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 0x7f997006d560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9968009e50 tx=0x7f996802bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623d89dae35655ac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.426+0000 7f996d400640 1 -- 172.21.15.53:0/3528466675 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996802ca70 con 0x7f997006d160 2024-01-16T15:00:27.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.426+0000 7f996d400640 1 -- 172.21.15.53:0/3528466675 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f996802cbd0 con 0x7f997006d160 2024-01-16T15:00:27.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f996d400640 1 -- 172.21.15.53:0/3528466675 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680343f0 con 0x7f997006d160 2024-01-16T15:00:27.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 -- 172.21.15.53:0/3528466675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 msgr2=0x7f997006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 --2- 172.21.15.53:0/3528466675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 0x7f997006d560 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9968009e50 tx=0x7f996802bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 -- 172.21.15.53:0/3528466675 shutdown_connections 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 --2- 172.21.15.53:0/3528466675 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f997006d160 0x7f997006d560 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 --2- 172.21.15.53:0/3528466675 >> v2:172.21.15.146:3300/0 conn(0x7f99700cc500 0x7f99700cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 -- 172.21.15.53:0/3528466675 >> 172.21.15.53:0/3528466675 conn(0x7f997006a800 msgr2=0x7f997006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 -- 172.21.15.53:0/3528466675 shutdown_connections 2024-01-16T15:00:27.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.427+0000 7f9974a00640 1 -- 172.21.15.53:0/3528466675 wait complete. 2024-01-16T15:00:27.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.429+0000 7f9974a00640 1 Processor -- start 2024-01-16T15:00:27.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.429+0000 7f9974a00640 1 -- start start 2024-01-16T15:00:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.430+0000 7f9974a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f997006d160 0x7f99700b2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.430+0000 7f9974a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 0x7f99701e0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.430+0000 7f9974a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9970143c40 0x7f99701e13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.430+0000 7f9974a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99700b28d0 con 0x7f99700cc500 2024-01-16T15:00:27.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.430+0000 7f9974a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9970130b30 con 0x7f997006d160 2024-01-16T15:00:27.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.430+0000 7f9974a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99700b1cc0 con 0x7f9970143c40 2024-01-16T15:00:27.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 0x7f99701e0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 0x7f99701e0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49396/0 (socket says 172.21.15.53:49396) 2024-01-16T15:00:27.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 -- 172.21.15.53:0/3711125415 learned_addr learned my addr 172.21.15.53:0/3711125415 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996e800640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f997006d160 0x7f99700b2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996f200640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9970143c40 0x7f99701e13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 -- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9970143c40 msgr2=0x7f99701e13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9970143c40 0x7f99701e13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 -- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f997006d160 msgr2=0x7f99700b2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f997006d160 0x7f99700b2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.431+0000 7f996de00640 1 -- 172.21.15.53:0/3711125415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968009b30 con 0x7f99700cc500 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.432+0000 7f996de00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 0x7f99701e0e90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f995800e9b0 tx=0x7f995800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.432+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9958009670 con 0x7f99700cc500 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.432+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995800c9a0 con 0x7f99700cc500 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.432+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f995800cd50 con 0x7f99700cc500 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.432+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99700b1fa0 con 0x7f99700cc500 2024-01-16T15:00:27.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.432+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99700b2560 con 0x7f99700cc500 2024-01-16T15:00:27.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.433+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99700b2a40 con 0x7f99700cc500 2024-01-16T15:00:27.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.434+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f9958013470 con 0x7f99700cc500 2024-01-16T15:00:27.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.435+0000 7f9966a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f994806b2c0 0x7f994806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:27.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.435+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9958089de0 con 0x7f99700cc500 2024-01-16T15:00:27.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.436+0000 7f996e800640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f994806b2c0 0x7f994806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:27.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.437+0000 7f996e800640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f994806b2c0 0x7f994806d780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9960000f60 tx=0x7f9960009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:27.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.440+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9958054cf0 con 0x7f99700cc500 2024-01-16T15:00:27.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.670+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f9970067450 con 0x7f99700cc500 2024-01-16T15:00:27.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.670+0000 7f9966a00640 1 -- 172.21.15.53:0/3711125415 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+73 (secure 0 0 0) 0x7f9958017070 con 0x7f99700cc500 2024-01-16T15:00:27.651 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:00:27.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f994806b2c0 msgr2=0x7f994806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f994806b2c0 0x7f994806d780 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9960000f60 tx=0x7f9960009040 comp rx=0 tx=0).stop 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 msgr2=0x7f99701e0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 0x7f99701e0e90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f995800e9b0 tx=0x7f995800ee70 comp rx=0 tx=0).stop 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 shutdown_connections 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f994806b2c0 0x7f994806d780 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9970143c40 0x7f99701e13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99700cc500 0x7f99701e0e90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 --2- 172.21.15.53:0/3711125415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f997006d160 0x7f99700b2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 >> 172.21.15.53:0/3711125415 conn(0x7f997006a800 msgr2=0x7f9970071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.673+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 shutdown_connections 2024-01-16T15:00:27.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:27.674+0000 7f9974a00640 1 -- 172.21.15.53:0/3711125415 wait complete. 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:00:27.673 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-01-16T15:00:27.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:27.674 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:27.731 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42502sec preferred_lft 42502sec 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:00:27.732 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:27.733 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:00:27.733 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:27.733 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:00:27.733 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:27.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:27.734 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:00:27.818 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:00:27.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:27.822 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:00:27.873 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:00:27.873 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:00:27.873 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:00:27.873 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:00:27.873 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:00:27.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:27.874 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:27.874 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:27.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:00:27.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:00:27.874 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:00:27.874 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:28.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.050 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:00:28.100 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:00:28.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.101 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list-id 2024-01-16T15:00:28.150 INFO:teuthology.orchestra.run.smithi146.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-01-16T15:00:28.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.151 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:28.151 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:28.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:28.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-01-16T15:00:28.151 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:28.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.284 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:1b:4d:e3:50:1b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.369 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::901b:4dff:fee3:501b/64 scope link 2024-01-16T15:00:28.370 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.374 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 92:1b:4d:e3:50:1b brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::901b:4dff:fee3:501b/64 scope link 2024-01-16T15:00:28.469 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:00:28.473 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-01-16T15:00:28.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-01-16T15:00:28.474 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:28.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:28.719 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:00:28.719 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:00:28.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.1 up 2024-01-16T15:00:28.720 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set dev brx.1 master ceph-brx 2024-01-16T15:00:28.720 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:00:28.879 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-01-16T15:00:28.879 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:28.879 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:28.924 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-01-16T15:00:28.924 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:00:28.925 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:29.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:00:29.103 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:00:29.160 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:29.161 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:29.162 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:29.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:29.163 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:00:29.215 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:00:29.215 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-01-16T15:00:29.215 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-01-16T15:00:29.222 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:00:29.302 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:00:29.344 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:00:29.344 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:00:29.344 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:00:29.345 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:29.346 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:00:29.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:29.347 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:00:29.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi146.stderr:2024-01-16T15:00:29.430+0000 7f4e09c41480 -1 init, newargv = 0x5567fd2c66f0 newargc=15 2024-01-16T15:00:29.410 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi146.stderr:ceph-fuse[41697]: starting ceph client 2024-01-16T15:00:29.420 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi146.stderr:ceph-fuse[41697]: starting fuse 2024-01-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:00:29.432 INFO:teuthology.orchestra.run.smithi146.stdout:42 2024-01-16T15:00:29.433 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 42] 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-01-16T15:00:29.433 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.3")) 2024-01-16T15:00:29.434 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:00:29.629 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.3.41697.asok 2024-01-16T15:00:29.633 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:00:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:00:29.645 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.41697.asok 2024-01-16T15:00:29.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:29.646 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.41697.asok status 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "3", 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "41697", 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:29.882 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 4684, 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 4684 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3532585731 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:29.883 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.2:0", 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3532585731 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.4684 192.168.0.2:0/3532585731", 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.2:0/3532585731", 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 7, 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 22, 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:00:29.884 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:00:29.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:29.899 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:00:29.899 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:29.959 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:00:29.960 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:29.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:29.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:30.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:30.067 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:00:30.067 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:30.127 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:00:30.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:30.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:30.127 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:30.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:30.235 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:00:30.235 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:30.293 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:00:30.293 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:30.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:30.293 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:30.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:30.390 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:00:30.390 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:30.445 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:00:30.445 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:30.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:30.445 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-01-16T15:00:30.540 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-01-16T15:00:30.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:30.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.803+0000 7fd77fe00640 1 Processor -- start 2024-01-16T15:00:30.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.803+0000 7fd77fe00640 1 -- start start 2024-01-16T15:00:30.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.805+0000 7fd77fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 0x7fd77815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.805+0000 7fd77fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77806c470 con 0x7fd77813e360 2024-01-16T15:00:30.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.805+0000 7fd77fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd77806c750 con 0x7fd7780cf9a0 2024-01-16T15:00:30.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.805+0000 7fd77fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77806ca30 con 0x7fd77813df80 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.806+0000 7fd77ca00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd77813e360 0x7fd7780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33452/0 (socket says 172.21.15.53:33452) 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.806+0000 7fd77ca00640 1 -- 172.21.15.53:0/3990485432 learned_addr learned my addr 172.21.15.53:0/3990485432 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.806+0000 7fd77d400640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 0x7fd77815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.806+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517283910 0 0) 0x7fd77806c470 con 0x7fd77813e360 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.806+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd758003520 con 0x7fd77813e360 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.806+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3994125327 0 0) 0x7fd758003520 con 0x7fd77813e360 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 msgr2=0x7fd77815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 0x7fd77815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 >> v1:172.21.15.146:6789/0 conn(0x7fd7780cf9a0 legacy=0x7fd7780cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77806cd10 con 0x7fd77813e360 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd760001670 con 0x7fd77813e360 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 >> v1:172.21.15.53:6789/0 conn(0x7fd77813e360 legacy=0x7fd7780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7580039a0 0x7fd758003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7580047c0 0x7fd758004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 0x7fd758005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77806ca30 con 0x7fd7580039a0 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd75800ba30 con 0x7fd7580047c0 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd75800bbf0 con 0x7fd77813df80 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77d400640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7580039a0 0x7fd758003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77ca00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7580047c0 0x7fd758004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77d400640 1 -- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 msgr2=0x7fd758005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:30.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77d400640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd77813df80 0x7fd758005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77d400640 1 -- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7580047c0 msgr2=0x7fd758004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:30.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77d400640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7580047c0 0x7fd758004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77d400640 1 -- 172.21.15.53:0/3990485432 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd758003520 con 0x7fd7580039a0 2024-01-16T15:00:30.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.807+0000 7fd77ca00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7580047c0 0x7fd758004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd77d400640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7580039a0 0x7fd758003dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd768009a90 tx=0x7fd7580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223794b1d5287634 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76802c6b0 con 0x7fd7580039a0 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76802c810 con 0x7fd7580039a0 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd777e00640 1 -- 172.21.15.53:0/3990485432 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76802cba0 con 0x7fd7580039a0 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd77fe00640 1 -- 172.21.15.53:0/3990485432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7580039a0 msgr2=0x7fd758003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3990485432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7580039a0 0x7fd758003dc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd768009a90 tx=0x7fd7580057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd77de00640 1 -- 172.21.15.53:0/3990485432 reap_dead start 2024-01-16T15:00:30.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd77fe00640 1 -- 172.21.15.53:0/3990485432 shutdown_connections 2024-01-16T15:00:30.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.809+0000 7fd77fe00640 1 -- 172.21.15.53:0/3990485432 >> 172.21.15.53:0/3990485432 conn(0x7fd77806a800 msgr2=0x7fd77806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:30.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.810+0000 7fd77fe00640 1 -- 172.21.15.53:0/3990485432 shutdown_connections 2024-01-16T15:00:30.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.810+0000 7fd77fe00640 1 -- 172.21.15.53:0/3990485432 wait complete. 2024-01-16T15:00:30.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.811+0000 7fd77fe00640 1 Processor -- start 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.811+0000 7fd77fe00640 1 -- start start 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.812+0000 7fd77fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7580039a0 0x7fd778153dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.812+0000 7fd77fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 0x7fd7781f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.812+0000 7fd77fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7780cb180 0x7fd7780cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.812+0000 7fd77fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd778154eb0 con 0x7fd778156440 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.812+0000 7fd77fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7781546e0 con 0x7fd7780cb180 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.812+0000 7fd77fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd778156990 con 0x7fd7580039a0 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 0x7fd7781f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 0x7fd7781f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49414/0 (socket says 172.21.15.53:49414) 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 -- 172.21.15.53:0/3303957429 learned_addr learned my addr 172.21.15.53:0/3303957429 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77de00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7780cb180 0x7fd7780cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77d400640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7580039a0 0x7fd778153dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 -- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7580039a0 msgr2=0x7fd778153dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7580039a0 0x7fd778153dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 -- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7780cb180 msgr2=0x7fd7780cb5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:30.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7780cb180 0x7fd7780cb5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:30.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77ca00640 1 -- 172.21.15.53:0/3303957429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7781548f0 con 0x7fd778156440 2024-01-16T15:00:30.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.813+0000 7fd77d400640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7580039a0 0x7fd778153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:30.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.814+0000 7fd77ca00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 0x7fd7781f6750 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd76000d8b0 tx=0x7fd76000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.814+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600074e0 con 0x7fd778156440 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.814+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760007640 con 0x7fd778156440 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.814+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600049e0 con 0x7fd778156440 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.814+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd778154bd0 con 0x7fd778156440 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.814+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7780cbb20 con 0x7fd778156440 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.815+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7780675d0 con 0x7fd778156440 2024-01-16T15:00:30.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.816+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd760004b40 con 0x7fd778156440 2024-01-16T15:00:30.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.817+0000 7fd775600640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd75006b2c0 0x7fd75006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:30.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.817+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd760089700 con 0x7fd778156440 2024-01-16T15:00:30.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.818+0000 7fd77d400640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd75006b2c0 0x7fd75006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:30.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.819+0000 7fd77d400640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd75006b2c0 0x7fd75006d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd768036000 tx=0x7fd768030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:30.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:30.822+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd760058020 con 0x7fd778156440 2024-01-16T15:00:30.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.015+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7780702c0 con 0x7fd778156440 2024-01-16T15:00:30.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.019+0000 7fd775600640 1 -- 172.21.15.53:0/3303957429 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7fd760057bb0 con 0x7fd778156440 2024-01-16T15:00:30.999 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:30.999 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd75006b2c0 msgr2=0x7fd75006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd75006b2c0 0x7fd75006d780 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd768036000 tx=0x7fd768030000 comp rx=0 tx=0).stop 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 msgr2=0x7fd7781f6750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 0x7fd7781f6750 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd76000d8b0 tx=0x7fd76000dd70 comp rx=0 tx=0).stop 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 shutdown_connections 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd75006b2c0 0x7fd75006d780 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd7780cb180 0x7fd7780cb5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd778156440 0x7fd7781f6750 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 --2- 172.21.15.53:0/3303957429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7580039a0 0x7fd778153dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.021+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 >> 172.21.15.53:0/3303957429 conn(0x7fd77806a800 msgr2=0x7fd7780b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.022+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 shutdown_connections 2024-01-16T15:00:31.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.022+0000 7fd77fe00640 1 -- 172.21.15.53:0/3303957429 wait complete. 2024-01-16T15:00:31.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:31.021 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:31.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.272+0000 7eff4be00640 1 Processor -- start 2024-01-16T15:00:31.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.273+0000 7eff4be00640 1 -- start start 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.274+0000 7eff4be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff4c13cf10 0x7eff4c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.274+0000 7eff4be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7eff4c0cddf0 0x7eff4c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.274+0000 7eff4be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0ce440 con 0x7eff4c13fae0 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.274+0000 7eff4be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4c13ff90 con 0x7eff4c13cf10 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.274+0000 7eff4be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4c0ce220 con 0x7eff4c0cddf0 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.275+0000 7eff4aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff4c13cf10 0x7eff4c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.275+0000 7eff4aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff4c13cf10 0x7eff4c13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49422/0 (socket says 172.21.15.53:49422) 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.275+0000 7eff4aa00640 1 -- 172.21.15.53:0/875935025 learned_addr learned my addr 172.21.15.53:0/875935025 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.275+0000 7eff4b400640 1 --2- 172.21.15.53:0/875935025 >> v2:172.21.15.146:3300/0 conn(0x7eff4c0cddf0 0x7eff4c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.275+0000 7eff4aa00640 1 -- 172.21.15.53:0/875935025 >> v2:172.21.15.146:3300/0 conn(0x7eff4c0cddf0 msgr2=0x7eff4c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.275+0000 7eff4aa00640 1 --2- 172.21.15.53:0/875935025 >> v2:172.21.15.146:3300/0 conn(0x7eff4c0cddf0 0x7eff4c0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.276+0000 7eff4aa00640 1 -- 172.21.15.53:0/875935025 >> v1:172.21.15.146:6789/0 conn(0x7eff4c13fae0 legacy=0x7eff4c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.276+0000 7eff4aa00640 1 -- 172.21.15.53:0/875935025 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c0ce720 con 0x7eff4c13cf10 2024-01-16T15:00:31.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.276+0000 7eff4b400640 1 --2- 172.21.15.53:0/875935025 >> v2:172.21.15.146:3300/0 conn(0x7eff4c0cddf0 0x7eff4c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:31.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.277+0000 7eff4aa00640 1 --2- 172.21.15.53:0/875935025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c13f5a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff4c157f90 tx=0x7eff3c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c903569cd057010 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.277+0000 7eff49600640 1 -- 172.21.15.53:0/875935025 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c02cb20 con 0x7eff4c13cf10 2024-01-16T15:00:31.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff49600640 1 -- 172.21.15.53:0/875935025 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3c02cc80 con 0x7eff4c13cf10 2024-01-16T15:00:31.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff49600640 1 -- 172.21.15.53:0/875935025 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3c0343f0 con 0x7eff4c13cf10 2024-01-16T15:00:31.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 -- 172.21.15.53:0/875935025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 msgr2=0x7eff4c13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 --2- 172.21.15.53:0/875935025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c13f5a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7eff4c157f90 tx=0x7eff3c02bf80 comp rx=0 tx=0).stop 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 -- 172.21.15.53:0/875935025 shutdown_connections 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 --2- 172.21.15.53:0/875935025 >> v2:172.21.15.146:3300/0 conn(0x7eff4c0cddf0 0x7eff4c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 --2- 172.21.15.53:0/875935025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c13f5a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 -- 172.21.15.53:0/875935025 >> 172.21.15.53:0/875935025 conn(0x7eff4c06a7d0 msgr2=0x7eff4c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.278+0000 7eff4be00640 1 -- 172.21.15.53:0/875935025 shutdown_connections 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.279+0000 7eff4be00640 1 -- 172.21.15.53:0/875935025 wait complete. 2024-01-16T15:00:31.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.280+0000 7eff4be00640 1 Processor -- start 2024-01-16T15:00:31.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.280+0000 7eff4be00640 1 -- start start 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.281+0000 7eff4be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4c0cddf0 0x7eff4c0711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.281+0000 7eff4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c0716f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.281+0000 7eff4be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff4c13fae0 0x7eff4c1569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c1578a0 con 0x7eff4c13cf10 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c151a80 con 0x7eff4c13fae0 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4c1f19e0 con 0x7eff4c0cddf0 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c0716f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c0716f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49438/0 (socket says 172.21.15.53:49438) 2024-01-16T15:00:31.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 -- 172.21.15.53:0/2643522872 learned_addr learned my addr 172.21.15.53:0/2643522872 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4b400640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff4c13fae0 0x7eff4c1569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4aa00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4c0cddf0 0x7eff4c0711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 -- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4c0cddf0 msgr2=0x7eff4c0711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4c0cddf0 0x7eff4c0711b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 -- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff4c13fae0 msgr2=0x7eff4c1569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff4c13fae0 0x7eff4c1569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.282+0000 7eff4a000640 1 -- 172.21.15.53:0/2643522872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3c009b30 con 0x7eff4c13cf10 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.283+0000 7eff4a000640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c0716f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7eff3400c430 tx=0x7eff3400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.283+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff340052d0 con 0x7eff4c13cf10 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.283+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff34005430 con 0x7eff4c13cf10 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.283+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff34003830 con 0x7eff4c13cf10 2024-01-16T15:00:31.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.283+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4c1539e0 con 0x7eff4c13cf10 2024-01-16T15:00:31.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.283+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4c1575c0 con 0x7eff4c13cf10 2024-01-16T15:00:31.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.284+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4c1f1b50 con 0x7eff4c13cf10 2024-01-16T15:00:31.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.286+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff34003990 con 0x7eff4c13cf10 2024-01-16T15:00:31.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.287+0000 7eff43400640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2406b2f0 0x7eff2406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.287+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff34089140 con 0x7eff4c13cf10 2024-01-16T15:00:31.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.287+0000 7eff4aa00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2406b2f0 0x7eff2406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.288+0000 7eff4aa00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2406b2f0 0x7eff2406d7b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7eff3c035000 tx=0x7eff3c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.291+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff34053fa0 con 0x7eff4c13cf10 2024-01-16T15:00:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.481+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff4c140350 con 0x7eff4c13cf10 2024-01-16T15:00:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.482+0000 7eff43400640 1 -- 172.21.15.53:0/2643522872 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff4c140350 con 0x7eff4c13cf10 2024-01-16T15:00:31.463 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2406b2f0 msgr2=0x7eff2406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2406b2f0 0x7eff2406d7b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7eff3c035000 tx=0x7eff3c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 msgr2=0x7eff4c0716f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c0716f0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7eff3400c430 tx=0x7eff3400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 shutdown_connections 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2406b2f0 0x7eff2406d7b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff4c13fae0 0x7eff4c1569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4c13cf10 0x7eff4c0716f0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.485+0000 7eff4be00640 1 --2- 172.21.15.53:0/2643522872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4c0cddf0 0x7eff4c0711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.486+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 >> 172.21.15.53:0/2643522872 conn(0x7eff4c06a7d0 msgr2=0x7eff4c06cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.486+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 shutdown_connections 2024-01-16T15:00:31.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.486+0000 7eff4be00640 1 -- 172.21.15.53:0/2643522872 wait complete. 2024-01-16T15:00:31.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:31.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.731+0000 7f2b2b200640 1 Processor -- start 2024-01-16T15:00:31.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.731+0000 7f2b2b200640 1 -- start start 2024-01-16T15:00:31.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.732+0000 7f2b2b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b2c15d580 0x7f2b2c0cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c0cfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b2b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b2c0b20d0 0x7f2b2c0d00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b2b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c0d0a00 con 0x7f2b2c15d580 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b2b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b2c0ce760 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0cf250 con 0x7f2b2c15d960 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b2c15d580 0x7f2b2c0cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b2c15d580 0x7f2b2c0cf650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49450/0 (socket says 172.21.15.53:49450) 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 -- 172.21.15.53:0/658327377 learned_addr learned my addr 172.21.15.53:0/658327377 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 -- 172.21.15.53:0/658327377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 msgr2=0x7f2b2c0cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 --2- 172.21.15.53:0/658327377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c0cfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 -- 172.21.15.53:0/658327377 >> v2:172.21.15.146:3300/0 conn(0x7f2b2c0b20d0 msgr2=0x7f2b2c0d00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 --2- 172.21.15.53:0/658327377 >> v2:172.21.15.146:3300/0 conn(0x7f2b2c0b20d0 0x7f2b2c0d00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.733+0000 7f2b29e00640 1 -- 172.21.15.53:0/658327377 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c0ce9b0 con 0x7f2b2c15d580 2024-01-16T15:00:31.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.735+0000 7f2b29e00640 1 --2- 172.21.15.53:0/658327377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c15d580 0x7f2b2c0cf650 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0b4b00 tx=0x7f2b1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fbb85a4a1d160e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.735+0000 7f2b28a00640 1 -- 172.21.15.53:0/658327377 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c0317e0 con 0x7f2b2c15d580 2024-01-16T15:00:31.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.735+0000 7f2b28a00640 1 -- 172.21.15.53:0/658327377 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c031940 con 0x7f2b2c15d580 2024-01-16T15:00:31.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.735+0000 7f2b28a00640 1 -- 172.21.15.53:0/658327377 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c031cd0 con 0x7f2b2c15d580 2024-01-16T15:00:31.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 -- 172.21.15.53:0/658327377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c15d580 msgr2=0x7f2b2c0cf650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 --2- 172.21.15.53:0/658327377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c15d580 0x7f2b2c0cf650 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c0b4b00 tx=0x7f2b1c02b880 comp rx=0 tx=0).stop 2024-01-16T15:00:31.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 -- 172.21.15.53:0/658327377 shutdown_connections 2024-01-16T15:00:31.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 --2- 172.21.15.53:0/658327377 >> v2:172.21.15.146:3300/0 conn(0x7f2b2c0b20d0 0x7f2b2c0d00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 --2- 172.21.15.53:0/658327377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c0cfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 --2- 172.21.15.53:0/658327377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c15d580 0x7f2b2c0cf650 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 -- 172.21.15.53:0/658327377 >> 172.21.15.53:0/658327377 conn(0x7f2b2c0d3040 msgr2=0x7f2b2c0d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 -- 172.21.15.53:0/658327377 shutdown_connections 2024-01-16T15:00:31.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.736+0000 7f2b2b200640 1 -- 172.21.15.53:0/658327377 wait complete. 2024-01-16T15:00:31.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.738+0000 7f2b2b200640 1 Processor -- start 2024-01-16T15:00:31.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.738+0000 7f2b2b200640 1 -- start start 2024-01-16T15:00:31.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 0x7f2b2c0b2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b2c15d580 0x7f2b2c1e7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b2b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0b2a40 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b2b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0caac0 con 0x7f2b2c15d580 2024-01-16T15:00:31.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2c0cada0 con 0x7f2b2c15d960 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 0x7f2b2c0b2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 0x7f2b2c0b2500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49458/0 (socket says 172.21.15.53:49458) 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.739+0000 7f2b29e00640 1 -- 172.21.15.53:0/2736461478 learned_addr learned my addr 172.21.15.53:0/2736461478 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b2a800640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29400640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b2c15d580 0x7f2b2c1e7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29e00640 1 -- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 msgr2=0x7f2b2c159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29e00640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c159010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29e00640 1 -- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b2c15d580 msgr2=0x7f2b2c1e7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29e00640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b2c15d580 0x7f2b2c1e7310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29e00640 1 -- 172.21.15.53:0/2736461478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0094d0 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b29e00640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 0x7f2b2c0b2500 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c000c00 tx=0x7f2b1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c032880 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c0329e0 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c032d70 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.740+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2c15eef0 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.741+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b2c15cc50 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.742+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b2c15f410 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.742+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2b1c04d3f0 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.743+0000 7f2b22000640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b0406b2c0 0x7f2b0406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:31.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.744+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2b1c0abb80 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.744+0000 7f2b29400640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b0406b2c0 0x7f2b0406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:31.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.745+0000 7f2b29400640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b0406b2c0 0x7f2b0406d780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2b14006fd0 tx=0x7f2b14008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:31.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.748+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b2c15f410 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.943+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2b2c15f410 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.944+0000 7f2b22000640 1 -- 172.21.15.53:0/2736461478 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2b2c15f410 con 0x7f2b2c0b20d0 2024-01-16T15:00:31.924 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.946+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b0406b2c0 msgr2=0x7f2b0406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.946+0000 7f2b2b200640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b0406b2c0 0x7f2b0406d780 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f2b14006fd0 tx=0x7f2b14008040 comp rx=0 tx=0).stop 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.946+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 msgr2=0x7f2b2c0b2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.946+0000 7f2b2b200640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 0x7f2b2c0b2500 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c000c00 tx=0x7f2b1c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 shutdown_connections 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b0406b2c0 0x7f2b0406d780 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b2c15d960 0x7f2b2c159010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b2c15d580 0x7f2b2c1e7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 --2- 172.21.15.53:0/2736461478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b2c0b20d0 0x7f2b2c0b2500 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:31.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 >> 172.21.15.53:0/2736461478 conn(0x7f2b2c0d3040 msgr2=0x7f2b2c06be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:31.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 shutdown_connections 2024-01-16T15:00:31.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:31.947+0000 7f2b2b200640 1 -- 172.21.15.53:0/2736461478 wait complete. 2024-01-16T15:00:31.947 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:32.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.192+0000 7fed7ca00640 1 Processor -- start 2024-01-16T15:00:32.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.193+0000 7fed7ca00640 1 -- start start 2024-01-16T15:00:32.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed7ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed74143ae0 0x7fed740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed7ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fed740cf7d0 0x7fed7415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed7ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740cfc30 con 0x7fed7415e1e0 2024-01-16T15:00:32.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed7ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740b1ee0 con 0x7fed740cf7d0 2024-01-16T15:00:32.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed7ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740b2780 con 0x7fed74143ae0 2024-01-16T15:00:32.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed79600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fed740cf7d0 0x7fed7415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed79600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fed740cf7d0 0x7fed7415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48484/0 (socket says 172.21.15.53:48484) 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.194+0000 7fed79600640 1 -- 172.21.15.53:0/1574596755 learned_addr learned my addr 172.21.15.53:0/1574596755 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.195+0000 7fed7a000640 1 --2- 172.21.15.53:0/1574596755 >> v2:172.21.15.146:3300/0 conn(0x7fed74143ae0 0x7fed740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.195+0000 7fed79600640 1 -- 172.21.15.53:0/1574596755 >> v2:172.21.15.146:3300/0 conn(0x7fed74143ae0 msgr2=0x7fed740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.195+0000 7fed79600640 1 --2- 172.21.15.53:0/1574596755 >> v2:172.21.15.146:3300/0 conn(0x7fed74143ae0 0x7fed740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.195+0000 7fed79600640 1 -- 172.21.15.53:0/1574596755 >> v1:172.21.15.146:6789/0 conn(0x7fed7415e1e0 legacy=0x7fed7415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:32.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.195+0000 7fed79600640 1 -- 172.21.15.53:0/1574596755 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7415e990 con 0x7fed740cf7d0 2024-01-16T15:00:32.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.196+0000 7fed79600640 1 --2- 172.21.15.53:0/1574596755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed7415dca0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed740cfea0 tx=0x7fed6402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a845f51670cac19 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.197+0000 7fed78c00640 1 -- 172.21.15.53:0/1574596755 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64030860 con 0x7fed740cf7d0 2024-01-16T15:00:32.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.197+0000 7fed78c00640 1 -- 172.21.15.53:0/1574596755 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed640309c0 con 0x7fed740cf7d0 2024-01-16T15:00:32.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.197+0000 7fed78c00640 1 -- 172.21.15.53:0/1574596755 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64030d50 con 0x7fed740cf7d0 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.197+0000 7fed7ca00640 1 -- 172.21.15.53:0/1574596755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 msgr2=0x7fed7415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.197+0000 7fed7ca00640 1 --2- 172.21.15.53:0/1574596755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed7415dca0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed740cfea0 tx=0x7fed6402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.197+0000 7fed7ca00640 1 -- 172.21.15.53:0/1574596755 shutdown_connections 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.198+0000 7fed7ca00640 1 --2- 172.21.15.53:0/1574596755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed7415dca0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.198+0000 7fed7ca00640 1 --2- 172.21.15.53:0/1574596755 >> v2:172.21.15.146:3300/0 conn(0x7fed74143ae0 0x7fed740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.198+0000 7fed7ca00640 1 -- 172.21.15.53:0/1574596755 >> 172.21.15.53:0/1574596755 conn(0x7fed7406a800 msgr2=0x7fed7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.198+0000 7fed7ca00640 1 -- 172.21.15.53:0/1574596755 shutdown_connections 2024-01-16T15:00:32.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.198+0000 7fed7ca00640 1 -- 172.21.15.53:0/1574596755 wait complete. 2024-01-16T15:00:32.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.199+0000 7fed7ca00640 1 Processor -- start 2024-01-16T15:00:32.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.199+0000 7fed7ca00640 1 -- start start 2024-01-16T15:00:32.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.200+0000 7fed7ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed740c4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.200+0000 7fed7ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed74143ae0 0x7fed74133d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.200+0000 7fed7ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed7415e1e0 0x7fed741412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.200+0000 7fed7ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed741346c0 con 0x7fed740cf7d0 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.200+0000 7fed7ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed741349a0 con 0x7fed7415e1e0 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.200+0000 7fed7ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed74134c80 con 0x7fed74143ae0 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed740c4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed740c4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48498/0 (socket says 172.21.15.53:48498) 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 -- 172.21.15.53:0/2594176776 learned_addr learned my addr 172.21.15.53:0/2594176776 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 -- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed74143ae0 msgr2=0x7fed74133d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed74143ae0 0x7fed74133d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 -- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed7415e1e0 msgr2=0x7fed741412f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed7415e1e0 0x7fed741412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 -- 172.21.15.53:0/2594176776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed640094d0 con 0x7fed740cf7d0 2024-01-16T15:00:32.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.201+0000 7fed7a000640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed740c4510 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00d800 tx=0x7fed5c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.202+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00b6c0 con 0x7fed740cf7d0 2024-01-16T15:00:32.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.202+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c00b820 con 0x7fed740cf7d0 2024-01-16T15:00:32.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.202+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed74131dd0 con 0x7fed740cf7d0 2024-01-16T15:00:32.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.202+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00bbb0 con 0x7fed740cf7d0 2024-01-16T15:00:32.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.202+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed740c4a50 con 0x7fed740cf7d0 2024-01-16T15:00:32.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.203+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed740675d0 con 0x7fed740cf7d0 2024-01-16T15:00:32.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.204+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fed5c00bd10 con 0x7fed740cf7d0 2024-01-16T15:00:32.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.205+0000 7fed72000640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed4c06b310 0x7fed4c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.205+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed5c08a180 con 0x7fed740cf7d0 2024-01-16T15:00:32.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.206+0000 7fed79600640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed4c06b310 0x7fed4c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.207+0000 7fed79600640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed4c06b310 0x7fed4c06d7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fed64036000 tx=0x7fed6403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.209+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed5c054fd0 con 0x7fed740cf7d0 2024-01-16T15:00:32.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.402+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed74071cb0 con 0x7fed740cf7d0 2024-01-16T15:00:32.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.403+0000 7fed72000640 1 -- 172.21.15.53:0/2594176776 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed5c0585f0 con 0x7fed740cf7d0 2024-01-16T15:00:32.384 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:32.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed4c06b310 msgr2=0x7fed4c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed4c06b310 0x7fed4c06d7d0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fed64036000 tx=0x7fed6403b040 comp rx=0 tx=0).stop 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 msgr2=0x7fed740c4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed740c4510 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00d800 tx=0x7fed5c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 shutdown_connections 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed4c06b310 0x7fed4c06d7d0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed7415e1e0 0x7fed741412f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed74143ae0 0x7fed74133d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 --2- 172.21.15.53:0/2594176776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed740cf7d0 0x7fed740c4510 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 >> 172.21.15.53:0/2594176776 conn(0x7fed7406a800 msgr2=0x7fed7406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 shutdown_connections 2024-01-16T15:00:32.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.406+0000 7fed7ca00640 1 -- 172.21.15.53:0/2594176776 wait complete. 2024-01-16T15:00:32.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:32.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.637+0000 7feb04e00640 1 Processor -- start 2024-01-16T15:00:32.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.638+0000 7feb04e00640 1 -- start start 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feb04e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feb0013e360 0x7feb000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feb04e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0015c5f0 con 0x7feb0013df80 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feb04e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb0006c470 con 0x7feb0015f9c0 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feb04e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb0006c750 con 0x7feb0013e360 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feafd400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feb0013e360 0x7feb000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feafde00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7feb0013df80 0x7feb0015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47080/0 (socket says 172.21.15.53:47080) 2024-01-16T15:00:32.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.640+0000 7feafde00640 1 -- 172.21.15.53:0/2159165694 learned_addr learned my addr 172.21.15.53:0/2159165694 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:32.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.641+0000 7feafd400640 1 -- 172.21.15.53:0/2159165694 >> v1:172.21.15.146:6789/0 conn(0x7feb0015f9c0 legacy=0x7feb000cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:32.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.641+0000 7feafd400640 1 -- 172.21.15.53:0/2159165694 >> v1:172.21.15.53:6789/0 conn(0x7feb0013df80 legacy=0x7feb0015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.641+0000 7feafd400640 1 -- 172.21.15.53:0/2159165694 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0006ca30 con 0x7feb0013e360 2024-01-16T15:00:32.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.641+0000 7feafca00640 1 -- 172.21.15.53:0/2159165694 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737127738 0 0) 0x7feb0015c5f0 con 0x7feb0013df80 2024-01-16T15:00:32.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.642+0000 7feafd400640 1 --2- 172.21.15.53:0/2159165694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013e360 0x7feb000cf290 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feae8009d90 tx=0x7feae802bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f31fb367801dfae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.642+0000 7feafca00640 1 -- 172.21.15.53:0/2159165694 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae802ccf0 con 0x7feb0013e360 2024-01-16T15:00:32.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.642+0000 7feafca00640 1 -- 172.21.15.53:0/2159165694 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae802ce50 con 0x7feb0013e360 2024-01-16T15:00:32.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.642+0000 7feafca00640 1 -- 172.21.15.53:0/2159165694 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae8034660 con 0x7feb0013e360 2024-01-16T15:00:32.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 -- 172.21.15.53:0/2159165694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013e360 msgr2=0x7feb000cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 --2- 172.21.15.53:0/2159165694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013e360 0x7feb000cf290 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feae8009d90 tx=0x7feae802bf40 comp rx=0 tx=0).stop 2024-01-16T15:00:32.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 -- 172.21.15.53:0/2159165694 shutdown_connections 2024-01-16T15:00:32.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 --2- 172.21.15.53:0/2159165694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013e360 0x7feb000cf290 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 -- 172.21.15.53:0/2159165694 >> 172.21.15.53:0/2159165694 conn(0x7feb0006a800 msgr2=0x7feb0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 -- 172.21.15.53:0/2159165694 shutdown_connections 2024-01-16T15:00:32.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.643+0000 7feb04e00640 1 -- 172.21.15.53:0/2159165694 wait complete. 2024-01-16T15:00:32.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.645+0000 7feb04e00640 1 Processor -- start 2024-01-16T15:00:32.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.645+0000 7feb04e00640 1 -- start start 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feb04e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 0x7feb0013d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feb04e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb0013e360 0x7feb0014ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feb04e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb0015f9c0 0x7feb0015dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feb04e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0014b040 con 0x7feb0013df80 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feb04e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb0014aa80 con 0x7feb0013e360 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feb04e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb0014ad60 con 0x7feb0015f9c0 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feafde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 0x7feb0013d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feafde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 0x7feb0013d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48518/0 (socket says 172.21.15.53:48518) 2024-01-16T15:00:32.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feafde00640 1 -- 172.21.15.53:0/759566230 learned_addr learned my addr 172.21.15.53:0/759566230 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.646+0000 7feafd400640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb0013e360 0x7feb0014ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafe800640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb0015f9c0 0x7feb0015dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafde00640 1 -- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb0015f9c0 msgr2=0x7feb0015dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafde00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb0015f9c0 0x7feb0015dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafde00640 1 -- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb0013e360 msgr2=0x7feb0014ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafde00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb0013e360 0x7feb0014ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafde00640 1 -- 172.21.15.53:0/759566230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae8009a70 con 0x7feb0013df80 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafd400640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb0013e360 0x7feb0014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafe800640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb0015f9c0 0x7feb0015dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:32.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feafde00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 0x7feb0013d8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feaf000c400 tx=0x7feaf000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0005ce0 con 0x7feb0013df80 2024-01-16T15:00:32.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaf0005e40 con 0x7feb0013df80 2024-01-16T15:00:32.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf0007310 con 0x7feb0013df80 2024-01-16T15:00:32.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.647+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb0014d390 con 0x7feb0013df80 2024-01-16T15:00:32.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.648+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb0014b480 con 0x7feb0013df80 2024-01-16T15:00:32.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.649+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb0014d780 con 0x7feb0013df80 2024-01-16T15:00:32.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.650+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feaf0007530 con 0x7feb0013df80 2024-01-16T15:00:32.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.651+0000 7feaf6000640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fead806b2c0 0x7fead806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:32.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.651+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7feaf0089160 con 0x7feb0013df80 2024-01-16T15:00:32.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.651+0000 7feafd400640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fead806b2c0 0x7fead806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:32.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.652+0000 7feafd400640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fead806b2c0 0x7fead806d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feae8036000 tx=0x7feae8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:32.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.655+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feaf0054060 con 0x7feb0013df80 2024-01-16T15:00:32.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.852+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb000675d0 con 0x7feb0013df80 2024-01-16T15:00:32.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.854+0000 7feaf6000640 1 -- 172.21.15.53:0/759566230 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7feaf0057680 con 0x7feb0013df80 2024-01-16T15:00:32.834 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:32.835 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fead806b2c0 msgr2=0x7fead806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fead806b2c0 0x7fead806d780 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7feae8036000 tx=0x7feae8030000 comp rx=0 tx=0).stop 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 msgr2=0x7feb0013d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 0x7feb0013d8f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7feaf000c400 tx=0x7feaf000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 shutdown_connections 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fead806b2c0 0x7fead806d780 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb0015f9c0 0x7feb0015dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb0013e360 0x7feb0014ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 --2- 172.21.15.53:0/759566230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb0013df80 0x7feb0013d8f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:32.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 >> 172.21.15.53:0/759566230 conn(0x7feb0006a800 msgr2=0x7feb001540e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:32.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 shutdown_connections 2024-01-16T15:00:32.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:32.857+0000 7feb04e00640 1 -- 172.21.15.53:0/759566230 wait complete. 2024-01-16T15:00:32.855 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:32.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:33.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.104+0000 7fa045e00640 1 Processor -- start 2024-01-16T15:00:33.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.104+0000 7fa045e00640 1 -- start start 2024-01-16T15:00:33.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.105+0000 7fa045e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa0400cf9a0 0x7fa040070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.105+0000 7fa045e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 0x7fa0400cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.105+0000 7fa045e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa040071690 con 0x7fa04014b2c0 2024-01-16T15:00:33.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.105+0000 7fa045e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa04014b6a0 con 0x7fa0400cf9a0 2024-01-16T15:00:33.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.105+0000 7fa045e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa040071970 con 0x7fa040070ea0 2024-01-16T15:00:33.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.106+0000 7fa03ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa04014b2c0 0x7fa0400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47090/0 (socket says 172.21.15.53:47090) 2024-01-16T15:00:33.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.106+0000 7fa03ea00640 1 -- 172.21.15.53:0/2710282222 learned_addr learned my addr 172.21.15.53:0/2710282222 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:33.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.106+0000 7fa03f400640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 0x7fa0400cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa037e00640 1 --2- 172.21.15.53:0/2710282222 >> v2:172.21.15.146:3300/0 conn(0x7fa0400cf9a0 0x7fa040070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.106+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670442444 0 0) 0x7fa040071690 con 0x7fa04014b2c0 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa020003520 con 0x7fa04014b2c0 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045682127 0 0) 0x7fa020003520 con 0x7fa04014b2c0 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 msgr2=0x7fa0400cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 0x7fa0400cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 >> v2:172.21.15.146:3300/0 conn(0x7fa0400cf9a0 msgr2=0x7fa040070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 --2- 172.21.15.53:0/2710282222 >> v2:172.21.15.146:3300/0 conn(0x7fa0400cf9a0 0x7fa040070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa040071c50 con 0x7fa04014b2c0 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fa030001670 con 0x7fa04014b2c0 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 >> v1:172.21.15.53:6789/0 conn(0x7fa04014b2c0 legacy=0x7fa0400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa020003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0200047c0 0x7fa020004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 0x7fa020005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04014b6a0 con 0x7fa020003970 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040071970 con 0x7fa0200047c0 2024-01-16T15:00:33.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.107+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02000b980 con 0x7fa040070ea0 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03ea00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa020003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa037e00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0200047c0 0x7fa020004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03f400640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 0x7fa020005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03ea00640 1 -- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 msgr2=0x7fa020005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03ea00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040070ea0 0x7fa020005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03ea00640 1 -- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0200047c0 msgr2=0x7fa020004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03ea00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0200047c0 0x7fa020004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa03ea00640 1 -- 172.21.15.53:0/2710282222 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa020003520 con 0x7fa020003970 2024-01-16T15:00:33.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.108+0000 7fa037e00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0200047c0 0x7fa020004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.109+0000 7fa03ea00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa020003d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c110 tx=0x7fa0200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15640232dc091356 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030005ad0 con 0x7fa020003970 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa030005c50 con 0x7fa020003970 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa03e000640 1 -- 172.21.15.53:0/2710282222 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03002c810 con 0x7fa020003970 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa045e00640 1 -- 172.21.15.53:0/2710282222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 msgr2=0x7fa020003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa045e00640 1 --2- 172.21.15.53:0/2710282222 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa020003d90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c110 tx=0x7fa0200057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa03f400640 1 -- 172.21.15.53:0/2710282222 reap_dead start 2024-01-16T15:00:33.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa045e00640 1 -- 172.21.15.53:0/2710282222 shutdown_connections 2024-01-16T15:00:33.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa045e00640 1 -- 172.21.15.53:0/2710282222 >> 172.21.15.53:0/2710282222 conn(0x7fa04006a800 msgr2=0x7fa04006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa045e00640 1 -- 172.21.15.53:0/2710282222 shutdown_connections 2024-01-16T15:00:33.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.110+0000 7fa045e00640 1 -- 172.21.15.53:0/2710282222 wait complete. 2024-01-16T15:00:33.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.112+0000 7fa045e00640 1 Processor -- start 2024-01-16T15:00:33.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.112+0000 7fa045e00640 1 -- start start 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.113+0000 7fa045e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa0401fb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.113+0000 7fa045e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040163030 0x7fa0401421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.113+0000 7fa045e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040147310 0x7fa040147790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.113+0000 7fa045e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040156d60 con 0x7fa020003970 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.113+0000 7fa045e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa040164370 con 0x7fa040163030 2024-01-16T15:00:33.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.113+0000 7fa045e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa040147cd0 con 0x7fa040147310 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa0401fb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa0401fb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48544/0 (socket says 172.21.15.53:48544) 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 -- 172.21.15.53:0/177317575 learned_addr learned my addr 172.21.15.53:0/177317575 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03f400640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040147310 0x7fa040147790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa037e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040163030 0x7fa0401421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 -- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040147310 msgr2=0x7fa040147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040147310 0x7fa040147790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 -- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040163030 msgr2=0x7fa0401421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040163030 0x7fa0401421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.114+0000 7fa03ea00640 1 -- 172.21.15.53:0/177317575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0401634b0 con 0x7fa020003970 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.115+0000 7fa03ea00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa0401fb0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c230 tx=0x7fa030001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.115+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030039070 con 0x7fa020003970 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.115+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03002cc20 con 0x7fa020003970 2024-01-16T15:00:33.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.115+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa040163760 con 0x7fa020003970 2024-01-16T15:00:33.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.115+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03003c040 con 0x7fa020003970 2024-01-16T15:00:33.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.115+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa040163d20 con 0x7fa020003970 2024-01-16T15:00:33.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.117+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0400675d0 con 0x7fa020003970 2024-01-16T15:00:33.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.117+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa030032070 con 0x7fa020003970 2024-01-16T15:00:33.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.118+0000 7fa037400640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa01806b2c0 0x7fa01806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.119+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa0300aaf50 con 0x7fa020003970 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.119+0000 7fa037e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa01806b2c0 0x7fa01806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.120+0000 7fa037e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa01806b2c0 0x7fa01806d780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0400afb30 tx=0x7fa028008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.123+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa030075d20 con 0x7fa020003970 2024-01-16T15:00:33.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.315+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0400cf5e0 con 0x7fa020003970 2024-01-16T15:00:33.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.316+0000 7fa037400640 1 -- 172.21.15.53:0/177317575 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa030079340 con 0x7fa020003970 2024-01-16T15:00:33.299 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa01806b2c0 msgr2=0x7fa01806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa01806b2c0 0x7fa01806d780 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fa0400afb30 tx=0x7fa028008040 comp rx=0 tx=0).stop 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 msgr2=0x7fa0401fb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa0401fb0f0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa03000c230 tx=0x7fa030001930 comp rx=0 tx=0).stop 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 shutdown_connections 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa01806b2c0 0x7fa01806d780 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa040147310 0x7fa040147790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa040163030 0x7fa0401421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 --2- 172.21.15.53:0/177317575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa020003970 0x7fa0401fb0f0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.318+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 >> 172.21.15.53:0/177317575 conn(0x7fa04006a800 msgr2=0x7fa0400cb0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.319+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 shutdown_connections 2024-01-16T15:00:33.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.319+0000 7fa045e00640 1 -- 172.21.15.53:0/177317575 wait complete. 2024-01-16T15:00:33.318 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:33.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.563+0000 7fec22e00640 1 Processor -- start 2024-01-16T15:00:33.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.563+0000 7fec22e00640 1 -- start start 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.564+0000 7fec22e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fec1c13cf10 0x7fec1c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec22e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 0x7fec1c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec22e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec1c0ce440 con 0x7fec1c13fae0 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec22e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1c13ff90 con 0x7fec1c13cf10 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec22e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0ce260 con 0x7fec1c0cddf0 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec21000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fec1c13fae0 0x7fec1c0cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47098/0 (socket says 172.21.15.53:47098) 2024-01-16T15:00:33.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec21000640 1 -- 172.21.15.53:0/3605994694 learned_addr learned my addr 172.21.15.53:0/3605994694 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:33.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2669872981 0 0) 0x7fec1c0ce440 con 0x7fec1c13fae0 2024-01-16T15:00:33.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec21a00640 1 --2- 172.21.15.53:0/3605994694 >> v2:172.21.15.146:3300/0 conn(0x7fec1c13cf10 0x7fec1c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.565+0000 7fec22400640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 0x7fec1c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febfc003520 con 0x7fec1c13fae0 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2689878791 0 0) 0x7febfc003520 con 0x7fec1c13fae0 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 msgr2=0x7fec1c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 0x7fec1c0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 >> v2:172.21.15.146:3300/0 conn(0x7fec1c13cf10 msgr2=0x7fec1c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 --2- 172.21.15.53:0/3605994694 >> v2:172.21.15.146:3300/0 conn(0x7fec1c13cf10 0x7fec1c13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c0ce720 con 0x7fec1c13fae0 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fec04001670 con 0x7fec1c13fae0 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 >> v1:172.21.15.53:6789/0 conn(0x7fec1c13fae0 legacy=0x7fec1c0cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 0x7febfc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febfc003f40 0x7febfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febfc004d00 0x7febfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c13ff90 con 0x7febfc004d00 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c0ce260 con 0x7febfc003f40 2024-01-16T15:00:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc00b980 con 0x7fec1c0cddf0 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.566+0000 7fec22400640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febfc004d00 0x7febfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec22400640 1 -- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 msgr2=0x7febfc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec22400640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c0cddf0 0x7febfc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec21000640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febfc003f40 0x7febfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec22400640 1 -- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febfc003f40 msgr2=0x7febfc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec22400640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febfc003f40 0x7febfc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec22400640 1 -- 172.21.15.53:0/3605994694 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc003520 con 0x7febfc004d00 2024-01-16T15:00:33.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.567+0000 7fec21000640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febfc003f40 0x7febfc0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.568+0000 7fec22400640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febfc004d00 0x7febfc005180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e170 tx=0x7febfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ba621834134ae2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.568+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1402cc20 con 0x7febfc004d00 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1400f040 con 0x7febfc004d00 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec1be00640 1 -- 172.21.15.53:0/3605994694 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec14034440 con 0x7febfc004d00 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22e00640 1 -- 172.21.15.53:0/3605994694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febfc004d00 msgr2=0x7febfc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22e00640 1 --2- 172.21.15.53:0/3605994694 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febfc004d00 0x7febfc005180 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fec1400e170 tx=0x7febfc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22400640 1 -- 172.21.15.53:0/3605994694 reap_dead start 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22e00640 1 -- 172.21.15.53:0/3605994694 shutdown_connections 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22e00640 1 -- 172.21.15.53:0/3605994694 >> 172.21.15.53:0/3605994694 conn(0x7fec1c06a7d0 msgr2=0x7fec1c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22e00640 1 -- 172.21.15.53:0/3605994694 shutdown_connections 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.569+0000 7fec22e00640 1 -- 172.21.15.53:0/3605994694 wait complete. 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.571+0000 7fec22e00640 1 Processor -- start 2024-01-16T15:00:33.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.571+0000 7fec22e00640 1 -- start start 2024-01-16T15:00:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec22e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 0x7fec1c139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec22e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c13e0f0 0x7fec1c155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec22e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c1561f0 0x7fec1c154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec22e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c13f210 con 0x7fec1c1511f0 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec22e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c156670 con 0x7fec1c13e0f0 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec22e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1c135930 con 0x7fec1c1561f0 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.572+0000 7fec21a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 0x7fec1c139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 0x7fec1c139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48564/0 (socket says 172.21.15.53:48564) 2024-01-16T15:00:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 -- 172.21.15.53:0/653292441 learned_addr learned my addr 172.21.15.53:0/653292441 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 -- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c1561f0 msgr2=0x7fec1c154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c1561f0 0x7fec1c154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 -- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c13e0f0 msgr2=0x7fec1c155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c13e0f0 0x7fec1c155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 -- 172.21.15.53:0/653292441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c1515d0 con 0x7fec1c1511f0 2024-01-16T15:00:33.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.573+0000 7fec21a00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 0x7fec1c139b10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00bcf0 tx=0x7fec0c00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.574+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c004440 con 0x7fec1c1511f0 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.574+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c011040 con 0x7fec1c1511f0 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.574+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c00e8a0 con 0x7fec1c1511f0 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.574+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1c13e690 con 0x7fec1c1511f0 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.574+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec1c13ec50 con 0x7fec1c1511f0 2024-01-16T15:00:33.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.575+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec1c067450 con 0x7fec1c1511f0 2024-01-16T15:00:33.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.576+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fec0c00ea00 con 0x7fec1c1511f0 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.577+0000 7fec19600640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febf406b2c0 0x7febf406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:33.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.577+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fec0c08bcd0 con 0x7fec1c1511f0 2024-01-16T15:00:33.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.578+0000 7fec21000640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febf406b2c0 0x7febf406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:33.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.579+0000 7fec21000640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febf406b2c0 0x7febf406d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fec040017a0 tx=0x7fec04009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:33.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.582+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fec0c056b40 con 0x7fec1c1511f0 2024-01-16T15:00:33.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.776+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec1c0cbf10 con 0x7fec1c1511f0 2024-01-16T15:00:33.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.777+0000 7fec19600640 1 -- 172.21.15.53:0/653292441 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec0c05a160 con 0x7fec1c1511f0 2024-01-16T15:00:33.759 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:33.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febf406b2c0 msgr2=0x7febf406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febf406b2c0 0x7febf406d780 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fec040017a0 tx=0x7fec04009040 comp rx=0 tx=0).stop 2024-01-16T15:00:33.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 msgr2=0x7fec1c139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:33.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 0x7fec1c139b10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00bcf0 tx=0x7fec0c00bde0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 shutdown_connections 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febf406b2c0 0x7febf406d780 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fec1c1561f0 0x7fec1c154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fec1c13e0f0 0x7fec1c155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 --2- 172.21.15.53:0/653292441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fec1c1511f0 0x7fec1c139b10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 >> 172.21.15.53:0/653292441 conn(0x7fec1c06a7d0 msgr2=0x7fec1c1529b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 shutdown_connections 2024-01-16T15:00:33.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:33.780+0000 7fec22e00640 1 -- 172.21.15.53:0/653292441 wait complete. 2024-01-16T15:00:33.780 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.025+0000 7f943dc00640 1 Processor -- start 2024-01-16T15:00:34.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.025+0000 7f943dc00640 1 -- start start 2024-01-16T15:00:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.026+0000 7f943dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f943815d4b0 0x7f94380d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.027+0000 7f943dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94380b2730 con 0x7f94380b41e0 2024-01-16T15:00:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.027+0000 7f943dc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94380d1ec0 con 0x7f94380cf9a0 2024-01-16T15:00:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.027+0000 7f943dc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94380b29d0 con 0x7f943815d4b0 2024-01-16T15:00:34.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.027+0000 7f9437e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f943815d4b0 0x7f94380d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.027+0000 7f9437e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f943815d4b0 0x7f94380d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48578/0 (socket says 172.21.15.53:48578) 2024-01-16T15:00:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.027+0000 7f9437e00640 1 -- 172.21.15.53:0/3589880121 learned_addr learned my addr 172.21.15.53:0/3589880121 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.028+0000 7f9437e00640 1 -- 172.21.15.53:0/3589880121 >> v1:172.21.15.146:6789/0 conn(0x7f94380cf9a0 legacy=0x7f94380cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.028+0000 7f9437e00640 1 -- 172.21.15.53:0/3589880121 >> v1:172.21.15.53:6789/0 conn(0x7f94380b41e0 legacy=0x7f94380ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.028+0000 7f9437e00640 1 -- 172.21.15.53:0/3589880121 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94380b2cb0 con 0x7f943815d4b0 2024-01-16T15:00:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.029+0000 7f9437e00640 1 --2- 172.21.15.53:0/3589880121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f943815d4b0 0x7f94380d1550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9428009870 tx=0x7f942802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4b593f7275b5b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.029+0000 7f9436a00640 1 -- 172.21.15.53:0/3589880121 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428030860 con 0x7f943815d4b0 2024-01-16T15:00:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.029+0000 7f9436a00640 1 -- 172.21.15.53:0/3589880121 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94280309c0 con 0x7f943815d4b0 2024-01-16T15:00:34.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.029+0000 7f9436a00640 1 -- 172.21.15.53:0/3589880121 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428030d50 con 0x7f943815d4b0 2024-01-16T15:00:34.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 -- 172.21.15.53:0/3589880121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f943815d4b0 msgr2=0x7f94380d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 --2- 172.21.15.53:0/3589880121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f943815d4b0 0x7f94380d1550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9428009870 tx=0x7f942802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 -- 172.21.15.53:0/3589880121 shutdown_connections 2024-01-16T15:00:34.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 --2- 172.21.15.53:0/3589880121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f943815d4b0 0x7f94380d1550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 -- 172.21.15.53:0/3589880121 >> 172.21.15.53:0/3589880121 conn(0x7f94380d3000 msgr2=0x7f94380d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 -- 172.21.15.53:0/3589880121 shutdown_connections 2024-01-16T15:00:34.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.030+0000 7f943dc00640 1 -- 172.21.15.53:0/3589880121 wait complete. 2024-01-16T15:00:34.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.032+0000 7f943dc00640 1 Processor -- start 2024-01-16T15:00:34.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.032+0000 7f943dc00640 1 -- start start 2024-01-16T15:00:34.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f943dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 0x7f94380cb450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f943dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94380cf9a0 0x7f94380b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f943dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943815d4b0 0x7f94380b8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f943dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94380cafc0 con 0x7f94380b41e0 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f943dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94380ca0d0 con 0x7f943815d4b0 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f943dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94380ca3b0 con 0x7f94380cf9a0 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f9437e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 0x7f94380cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f9437e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 0x7f94380cb450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48580/0 (socket says 172.21.15.53:48580) 2024-01-16T15:00:34.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.033+0000 7f9437e00640 1 -- 172.21.15.53:0/4061482908 learned_addr learned my addr 172.21.15.53:0/4061482908 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:34.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f943d200640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943815d4b0 0x7f94380b8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437400640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94380cf9a0 0x7f94380b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437e00640 1 -- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94380cf9a0 msgr2=0x7f94380b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437e00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94380cf9a0 0x7f94380b7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437e00640 1 -- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943815d4b0 msgr2=0x7f94380b8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437e00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943815d4b0 0x7f94380b8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437e00640 1 -- 172.21.15.53:0/4061482908 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94280094d0 con 0x7f94380b41e0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f943d200640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943815d4b0 0x7f94380b8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.034+0000 7f9437e00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 0x7f94380cb450 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9428009870 tx=0x7f9428004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.035+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428030810 con 0x7f94380b41e0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.035+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9428039d90 con 0x7f94380b41e0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.035+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943815da10 con 0x7f94380b41e0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.035+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9428043630 con 0x7f94380b41e0 2024-01-16T15:00:34.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.035+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94380ca970 con 0x7f94380b41e0 2024-01-16T15:00:34.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.036+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9438067450 con 0x7f94380b41e0 2024-01-16T15:00:34.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.037+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f942804c3b0 con 0x7f94380b41e0 2024-01-16T15:00:34.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.038+0000 7f942fe00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f941006b2c0 0x7f941006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.038+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f94280ac290 con 0x7f94380b41e0 2024-01-16T15:00:34.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.039+0000 7f9437400640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f941006b2c0 0x7f941006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.040+0000 7f9437400640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f941006b2c0 0x7f941006d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9420007fd0 tx=0x7f9420009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.043+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9428077100 con 0x7f94380b41e0 2024-01-16T15:00:34.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.239+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f943806be90 con 0x7f94380b41e0 2024-01-16T15:00:34.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.240+0000 7f942fe00640 1 -- 172.21.15.53:0/4061482908 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f942807a720 con 0x7f94380b41e0 2024-01-16T15:00:34.221 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:34.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f941006b2c0 msgr2=0x7f941006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f941006b2c0 0x7f941006d780 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9420007fd0 tx=0x7f9420009040 comp rx=0 tx=0).stop 2024-01-16T15:00:34.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 msgr2=0x7f94380cb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 0x7f94380cb450 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9428009870 tx=0x7f9428004730 comp rx=0 tx=0).stop 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 shutdown_connections 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f941006b2c0 0x7f941006d780 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f943815d4b0 0x7f94380b8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94380cf9a0 0x7f94380b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 --2- 172.21.15.53:0/4061482908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94380b41e0 0x7f94380cb450 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.242+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 >> 172.21.15.53:0/4061482908 conn(0x7f94380d3000 msgr2=0x7f94380ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.243+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 shutdown_connections 2024-01-16T15:00:34.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.243+0000 7f943dc00640 1 -- 172.21.15.53:0/4061482908 wait complete. 2024-01-16T15:00:34.244 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:34.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.483+0000 7f602e600640 1 Processor -- start 2024-01-16T15:00:34.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.484+0000 7f602e600640 1 -- start start 2024-01-16T15:00:34.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.485+0000 7f602e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6028160920 0x7f60280b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.485+0000 7f602e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60280cf9f0 0x7f60280cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.485+0000 7f602e600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60280d0a10 con 0x7f6028161010 2024-01-16T15:00:34.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.485+0000 7f602e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60280d03b0 con 0x7f6028160920 2024-01-16T15:00:34.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.485+0000 7f602e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60280d1f50 con 0x7f60280cf9f0 2024-01-16T15:00:34.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.486+0000 7f602dc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6028161010 0x7f60281614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47116/0 (socket says 172.21.15.53:47116) 2024-01-16T15:00:34.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.486+0000 7f602dc00640 1 -- 172.21.15.53:0/1605939211 learned_addr learned my addr 172.21.15.53:0/1605939211 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:34.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.486+0000 7f602d200640 1 --2- 172.21.15.53:0/1605939211 >> v2:172.21.15.146:3300/0 conn(0x7f6028160920 0x7f60280b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.486+0000 7f6027e00640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60280cf9f0 0x7f60280cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.486+0000 7f6027400640 1 -- 172.21.15.53:0/1605939211 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773719861 0 0) 0x7f60280d0a10 con 0x7f6028161010 2024-01-16T15:00:34.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.486+0000 7f6027400640 1 -- 172.21.15.53:0/1605939211 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6008003520 con 0x7f6028161010 2024-01-16T15:00:34.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.487+0000 7f602d200640 1 -- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60280cf9f0 msgr2=0x7f60280cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.487+0000 7f602d200640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60280cf9f0 0x7f60280cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.487+0000 7f602d200640 1 -- 172.21.15.53:0/1605939211 >> v1:172.21.15.53:6789/0 conn(0x7f6028161010 legacy=0x7f60281614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.487+0000 7f6027400640 1 -- 172.21.15.53:0/1605939211 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1066944313 0 0) 0x7f6008003520 con 0x7f6028161010 2024-01-16T15:00:34.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.487+0000 7f602d200640 1 -- 172.21.15.53:0/1605939211 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280cf080 con 0x7f6028160920 2024-01-16T15:00:34.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.487+0000 7f6027e00640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60280cf9f0 0x7f60280cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:00:34.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.488+0000 7f602d200640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028160920 0x7f60280b3f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f60100097a0 tx=0x7f601002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e098a9810af80a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f6027400640 1 -- 172.21.15.53:0/1605939211 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6020001670 con 0x7f6028160920 2024-01-16T15:00:34.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f6027400640 1 -- 172.21.15.53:0/1605939211 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601002e050 con 0x7f6028160920 2024-01-16T15:00:34.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f6027400640 1 -- 172.21.15.53:0/1605939211 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010030710 con 0x7f6028160920 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f602e600640 1 -- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028160920 msgr2=0x7f60280b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f602e600640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028160920 0x7f60280b3f30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f60100097a0 tx=0x7f601002b790 comp rx=0 tx=0).stop 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f602e600640 1 -- 172.21.15.53:0/1605939211 shutdown_connections 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f602e600640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60280cf9f0 0x7f60280cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f602e600640 1 --2- 172.21.15.53:0/1605939211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028160920 0x7f60280b3f30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.489+0000 7f602e600640 1 -- 172.21.15.53:0/1605939211 >> 172.21.15.53:0/1605939211 conn(0x7f60280d3090 msgr2=0x7f60280d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.490+0000 7f602e600640 1 -- 172.21.15.53:0/1605939211 shutdown_connections 2024-01-16T15:00:34.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.490+0000 7f602e600640 1 -- 172.21.15.53:0/1605939211 wait complete. 2024-01-16T15:00:34.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.492+0000 7f602e600640 1 Processor -- start 2024-01-16T15:00:34.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.492+0000 7f602e600640 1 -- start start 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 0x7f60280cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6028160920 0x7f602814e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028161010 0x7f60280b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602e600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60280ca950 con 0x7f60280cf9f0 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60280cc7c0 con 0x7f6028161010 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60280cc4e0 con 0x7f6028160920 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.493+0000 7f602d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 0x7f60280cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 0x7f60280cade0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48594/0 (socket says 172.21.15.53:48594) 2024-01-16T15:00:34.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 -- 172.21.15.53:0/3096847101 learned_addr learned my addr 172.21.15.53:0/3096847101 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f6027e00640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6028160920 0x7f602814e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602dc00640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028161010 0x7f60280b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 -- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6028160920 msgr2=0x7f602814e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6028160920 0x7f602814e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 -- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028161010 msgr2=0x7f60280b7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028161010 0x7f60280b7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 -- 172.21.15.53:0/3096847101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6018009a70 con 0x7f60280cf9f0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602dc00640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028161010 0x7f60280b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.494+0000 7f602d200640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 0x7f60280cade0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6010000c00 tx=0x7f60100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.495+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601003c620 con 0x7f60280cf9f0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.495+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601003c780 con 0x7f60280cf9f0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.495+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601003cb30 con 0x7f60280cf9f0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.495+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6010009400 con 0x7f60280cf9f0 2024-01-16T15:00:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.495+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60280ca5e0 con 0x7f60280cf9f0 2024-01-16T15:00:34.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.496+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60280ca140 con 0x7f60280cf9f0 2024-01-16T15:00:34.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.497+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6010039070 con 0x7f60280cf9f0 2024-01-16T15:00:34.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.498+0000 7f6024c00640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f600006b2c0 0x7f600006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.498+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f60100ab390 con 0x7f60280cf9f0 2024-01-16T15:00:34.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.499+0000 7f6027e00640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f600006b2c0 0x7f600006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.500+0000 7f6027e00640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f600006b2c0 0x7f600006d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6028158fd0 tx=0x7f601800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.503+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6010076160 con 0x7f60280cf9f0 2024-01-16T15:00:34.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.698+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6028067450 con 0x7f60280cf9f0 2024-01-16T15:00:34.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.701+0000 7f6024c00640 1 -- 172.21.15.53:0/3096847101 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f6010079780 con 0x7f60280cf9f0 2024-01-16T15:00:34.683 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:34.683 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f600006b2c0 msgr2=0x7f600006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f600006b2c0 0x7f600006d780 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f6028158fd0 tx=0x7f601800b040 comp rx=0 tx=0).stop 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 msgr2=0x7f60280cade0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 0x7f60280cade0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6010000c00 tx=0x7f60100046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 shutdown_connections 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f600006b2c0 0x7f600006d780 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6028161010 0x7f60280b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6028160920 0x7f602814e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 --2- 172.21.15.53:0/3096847101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60280cf9f0 0x7f60280cade0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 >> 172.21.15.53:0/3096847101 conn(0x7f60280d3090 msgr2=0x7f602815bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.704+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 shutdown_connections 2024-01-16T15:00:34.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.705+0000 7f602e600640 1 -- 172.21.15.53:0/3096847101 wait complete. 2024-01-16T15:00:34.706 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:34.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:34.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.949+0000 7f5ff2400640 1 Processor -- start 2024-01-16T15:00:34.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.949+0000 7f5ff2400640 1 -- start start 2024-01-16T15:00:34.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.950+0000 7f5ff2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5fec0cf9a0 0x7f5fec166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.950+0000 7f5ff2400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec15c5f0 con 0x7f5fec13df80 2024-01-16T15:00:34.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.950+0000 7f5ff2400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fec06c470 con 0x7f5fec13e360 2024-01-16T15:00:34.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.950+0000 7f5ff2400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fec06c750 con 0x7f5fec0cf9a0 2024-01-16T15:00:34.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.951+0000 7f5feb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5fec13df80 0x7f5fec15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47126/0 (socket says 172.21.15.53:47126) 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.951+0000 7f5feb400640 1 -- 172.21.15.53:0/2207659861 learned_addr learned my addr 172.21.15.53:0/2207659861 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.951+0000 7f5febe00640 1 --2- 172.21.15.53:0/2207659861 >> v2:172.21.15.146:3300/0 conn(0x7f5fec0cf9a0 0x7f5fec166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.951+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142544288 0 0) 0x7f5fec15c5f0 con 0x7f5fec13df80 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.951+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fcc003520 con 0x7f5fec13df80 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3904650172 0 0) 0x7f5fcc003520 con 0x7f5fec13df80 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 >> v2:172.21.15.146:3300/0 conn(0x7f5fec0cf9a0 msgr2=0x7f5fec166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 --2- 172.21.15.53:0/2207659861 >> v2:172.21.15.146:3300/0 conn(0x7f5fec0cf9a0 0x7f5fec166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 >> v1:172.21.15.146:6789/0 conn(0x7f5fec13e360 legacy=0x7f5fec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec06ca30 con 0x7f5fec13df80 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5fdc001670 con 0x7f5fec13df80 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 >> v1:172.21.15.53:6789/0 conn(0x7f5fec13df80 legacy=0x7f5fec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc003970 0x7f5fcc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fcc0047c0 0x7f5fcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc005180 0x7f5fcc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec06c750 con 0x7f5fcc003970 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc00c020 con 0x7f5fcc0047c0 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fcc00c1e0 con 0x7f5fcc005180 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5feb400640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc003970 0x7f5fcc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.952+0000 7f5fe3e00640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fcc0047c0 0x7f5fcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5febe00640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc005180 0x7f5fcc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5feb400640 1 -- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc005180 msgr2=0x7f5fcc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5feb400640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc005180 0x7f5fcc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5feb400640 1 -- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fcc0047c0 msgr2=0x7f5fcc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5feb400640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fcc0047c0 0x7f5fcc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5feb400640 1 -- 172.21.15.53:0/2207659861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc003520 con 0x7f5fcc003970 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5febe00640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc005180 0x7f5fcc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.953+0000 7f5febe00640 1 -- 172.21.15.53:0/2207659861 reap_dead start 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.954+0000 7f5feb400640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc003970 0x7f5fcc003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00c240 tx=0x7f5fcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7126088b2b9801d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.954+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc005ad0 con 0x7f5fcc003970 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.954+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fdc005c30 con 0x7f5fcc003970 2024-01-16T15:00:34.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.954+0000 7f5feaa00640 1 -- 172.21.15.53:0/2207659861 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc02ca00 con 0x7f5fcc003970 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 -- 172.21.15.53:0/2207659861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc003970 msgr2=0x7f5fcc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc003970 0x7f5fcc003d90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc00c240 tx=0x7f5fcc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 -- 172.21.15.53:0/2207659861 shutdown_connections 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2207659861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fcc003970 0x7f5fcc003d90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 -- 172.21.15.53:0/2207659861 >> 172.21.15.53:0/2207659861 conn(0x7f5fec06a800 msgr2=0x7f5fec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 -- 172.21.15.53:0/2207659861 shutdown_connections 2024-01-16T15:00:34.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.955+0000 7f5ff2400640 1 -- 172.21.15.53:0/2207659861 wait complete. 2024-01-16T15:00:34.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.957+0000 7f5ff2400640 1 Processor -- start 2024-01-16T15:00:34.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.957+0000 7f5ff2400640 1 -- start start 2024-01-16T15:00:34.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5ff2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc003970 0x7f5fec0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5ff2400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fec0a84c0 0x7f5fec0a8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5ff2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 0x7f5fec13d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5ff2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0a94d0 con 0x7f5fec13d2c0 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5ff2400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec0a91f0 con 0x7f5fec0a84c0 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5ff2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fec1f4250 con 0x7f5fcc003970 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5febe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 0x7f5fec13d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5febe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 0x7f5fec13d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48618/0 (socket says 172.21.15.53:48618) 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.958+0000 7f5febe00640 1 -- 172.21.15.53:0/2221175273 learned_addr learned my addr 172.21.15.53:0/2221175273 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5feb400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc003970 0x7f5fec0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5fe3e00640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fec0a84c0 0x7f5fec0a8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5febe00640 1 -- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc003970 msgr2=0x7f5fec0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5febe00640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc003970 0x7f5fec0b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5febe00640 1 -- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fec0a84c0 msgr2=0x7f5fec0a8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5febe00640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fec0a84c0 0x7f5fec0a8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5febe00640 1 -- 172.21.15.53:0/2221175273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fec1f4530 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.959+0000 7f5febe00640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 0x7f5fec13d750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0ccae0 tx=0x7f5fe400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.960+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4016620 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.960+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe4016780 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.960+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fec1f4810 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.960+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4016af0 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.960+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fec13dc90 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.961+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fec1f4a00 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.962+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5fe4016c50 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.963+0000 7f5fe3400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5fc406b2f0 0x7f5fc406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.963+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fe408a8b0 con 0x7f5fec13d2c0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.963+0000 7f5feb400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5fc406b2f0 0x7f5fc406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:34.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.964+0000 7f5feb400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5fc406b2f0 0x7f5fc406d7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc034000 tx=0x7f5fdc009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:34.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:34.967+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5fe4055720 con 0x7f5fec13d2c0 2024-01-16T15:00:35.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.159+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fec0675d0 con 0x7f5fec13d2c0 2024-01-16T15:00:35.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.160+0000 7f5fe3400640 1 -- 172.21.15.53:0/2221175273 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fe4058d40 con 0x7f5fec13d2c0 2024-01-16T15:00:35.142 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:35.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5fc406b2f0 msgr2=0x7f5fc406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5fc406b2f0 0x7f5fc406d7b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc034000 tx=0x7f5fdc009030 comp rx=0 tx=0).stop 2024-01-16T15:00:35.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 msgr2=0x7f5fec13d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 0x7f5fec13d750 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fec0ccae0 tx=0x7f5fe400f520 comp rx=0 tx=0).stop 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 shutdown_connections 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5fc406b2f0 0x7f5fc406d7b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5fec13d2c0 0x7f5fec13d750 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5fec0a84c0 0x7f5fec0a8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 --2- 172.21.15.53:0/2221175273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5fcc003970 0x7f5fec0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.163+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 >> 172.21.15.53:0/2221175273 conn(0x7f5fec06a800 msgr2=0x7f5fec0af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.164+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 shutdown_connections 2024-01-16T15:00:35.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.164+0000 7f5ff2400640 1 -- 172.21.15.53:0/2221175273 wait complete. 2024-01-16T15:00:35.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:35.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.400+0000 7f0a82e00640 1 Processor -- start 2024-01-16T15:00:35.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.400+0000 7f0a82e00640 1 -- start start 2024-01-16T15:00:35.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.402+0000 7f0a82e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a7c143ae0 0x7f0a7c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.402+0000 7f0a82e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c0cfc30 con 0x7f0a7c15e3b0 2024-01-16T15:00:35.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.402+0000 7f0a82e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a7c0b1ee0 con 0x7f0a7c0cf7d0 2024-01-16T15:00:35.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.402+0000 7f0a82e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a7c0b2780 con 0x7f0a7c143ae0 2024-01-16T15:00:35.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.402+0000 7f0a80e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0a7c15e3b0 0x7f0a7c0702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47142/0 (socket says 172.21.15.53:47142) 2024-01-16T15:00:35.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.402+0000 7f0a80e00640 1 -- 172.21.15.53:0/1769066636 learned_addr learned my addr 172.21.15.53:0/1769066636 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7be00640 1 --2- 172.21.15.53:0/1769066636 >> v2:172.21.15.146:3300/0 conn(0x7f0a7c143ae0 0x7f0a7c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272594288 0 0) 0x7f0a7c0cfc30 con 0x7f0a7c15e3b0 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a5c003520 con 0x7f0a7c15e3b0 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228647419 0 0) 0x7f0a5c003520 con 0x7f0a7c15e3b0 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 >> v2:172.21.15.146:3300/0 conn(0x7f0a7c143ae0 msgr2=0x7f0a7c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7b400640 1 --2- 172.21.15.53:0/1769066636 >> v2:172.21.15.146:3300/0 conn(0x7f0a7c143ae0 0x7f0a7c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.403+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 >> v1:172.21.15.146:6789/0 conn(0x7f0a7c0cf7d0 legacy=0x7f0a7c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c15e990 con 0x7f0a7c15e3b0 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0a74001670 con 0x7f0a7c15e3b0 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 >> v1:172.21.15.53:6789/0 conn(0x7f0a7c15e3b0 legacy=0x7f0a7c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5c0039c0 0x7f0a5c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5c0047c0 0x7f0a5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a5c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c0b2780 con 0x7f0a5c005180 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c00c080 con 0x7f0a5c0039c0 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a5c00c240 con 0x7f0a5c0047c0 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a80e00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a5c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a7be00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5c0039c0 0x7f0a5c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a80e00640 1 -- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5c0047c0 msgr2=0x7f0a5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.404+0000 7f0a80e00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5c0047c0 0x7f0a5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.405+0000 7f0a80e00640 1 -- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5c0039c0 msgr2=0x7f0a5c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.405+0000 7f0a80e00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a5c0039c0 0x7f0a5c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.405+0000 7f0a80e00640 1 -- 172.21.15.53:0/1769066636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c003520 con 0x7f0a5c005180 2024-01-16T15:00:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.405+0000 7f0a80e00640 1 -- 172.21.15.53:0/1769066636 reap_dead start 2024-01-16T15:00:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.406+0000 7f0a80e00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a5c00b980 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400c0e0 tx=0x7f0a5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8869c2f18d8f33c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.406+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a74005ad0 con 0x7f0a5c005180 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.406+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a74005c70 con 0x7f0a5c005180 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.406+0000 7f0a7b400640 1 -- 172.21.15.53:0/1769066636 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a7402c7d0 con 0x7f0a5c005180 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 -- 172.21.15.53:0/1769066636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 msgr2=0x7f0a5c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a5c00b980 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0a7400c0e0 tx=0x7f0a5c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 -- 172.21.15.53:0/1769066636 shutdown_connections 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 --2- 172.21.15.53:0/1769066636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a5c00b980 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 -- 172.21.15.53:0/1769066636 >> 172.21.15.53:0/1769066636 conn(0x7f0a7c06a800 msgr2=0x7f0a7c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 -- 172.21.15.53:0/1769066636 shutdown_connections 2024-01-16T15:00:35.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.407+0000 7f0a82e00640 1 -- 172.21.15.53:0/1769066636 wait complete. 2024-01-16T15:00:35.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.409+0000 7f0a82e00640 1 Processor -- start 2024-01-16T15:00:35.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.409+0000 7f0a82e00640 1 -- start start 2024-01-16T15:00:35.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a82e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a7c159a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a82e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a7c1366d0 0x7f0a7c136b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a82e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a7c0b0bf0 0x7f0a7c0b1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a82e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c0b15b0 con 0x7f0a5c005180 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a82e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c134b70 con 0x7f0a7c0b0bf0 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a82e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c137160 con 0x7f0a7c1366d0 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a7c159a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a7be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a7c159a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48634/0 (socket says 172.21.15.53:48634) 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a7be00640 1 -- 172.21.15.53:0/311779705 learned_addr learned my addr 172.21.15.53:0/311779705 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a80e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a7c0b0bf0 0x7f0a7c0b1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a7be00640 1 -- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a7c1366d0 msgr2=0x7f0a7c136b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.410+0000 7f0a73200640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a7c1366d0 0x7f0a7c136b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a7be00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a7c1366d0 0x7f0a7c136b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a7be00640 1 -- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a7c0b0bf0 msgr2=0x7f0a7c0b1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a7be00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a7c0b0bf0 0x7f0a7c0b1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a7be00640 1 -- 172.21.15.53:0/311779705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c137440 con 0x7f0a5c005180 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a80e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a7c0b0bf0 0x7f0a7c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a73200640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a7c1366d0 0x7f0a7c136b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a7be00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a7c159a90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009fb0 tx=0x7f0a6c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c00e030 con 0x7f0a5c005180 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a6c016470 con 0x7f0a5c005180 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a6c016820 con 0x7f0a5c005180 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c134800 con 0x7f0a5c005180 2024-01-16T15:00:35.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.411+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c1418e0 con 0x7f0a5c005180 2024-01-16T15:00:35.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.412+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c137680 con 0x7f0a5c005180 2024-01-16T15:00:35.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.413+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0a6c016980 con 0x7f0a5c005180 2024-01-16T15:00:35.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.414+0000 7f0a78c00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a5406b2c0 0x7f0a5406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.414+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0a6c00f070 con 0x7f0a5c005180 2024-01-16T15:00:35.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.415+0000 7f0a73200640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a5406b2c0 0x7f0a5406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.416+0000 7f0a73200640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a5406b2c0 0x7f0a5406d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c142c80 tx=0x7f0a64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.419+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a6c0559d0 con 0x7f0a5c005180 2024-01-16T15:00:35.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.616+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a7c0675d0 con 0x7f0a5c005180 2024-01-16T15:00:35.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.618+0000 7f0a78c00640 1 -- 172.21.15.53:0/311779705 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0a6c058ff0 con 0x7f0a5c005180 2024-01-16T15:00:35.599 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.620+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a5406b2c0 msgr2=0x7f0a5406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.620+0000 7f0a82e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a5406b2c0 0x7f0a5406d780 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c142c80 tx=0x7f0a64009040 comp rx=0 tx=0).stop 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.620+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 msgr2=0x7f0a7c159a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.620+0000 7f0a82e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a7c159a90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c009fb0 tx=0x7f0a6c0115a0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 shutdown_connections 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a5406b2c0 0x7f0a5406d780 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a7c0b0bf0 0x7f0a7c0b1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a7c1366d0 0x7f0a7c136b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 --2- 172.21.15.53:0/311779705 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5c005180 0x7f0a7c159a90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 >> 172.21.15.53:0/311779705 conn(0x7f0a7c06a800 msgr2=0x7f0a7c142d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 shutdown_connections 2024-01-16T15:00:35.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.621+0000 7f0a82e00640 1 -- 172.21.15.53:0/311779705 wait complete. 2024-01-16T15:00:35.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:35.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.859+0000 7f904e000640 1 Processor -- start 2024-01-16T15:00:35.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.859+0000 7f904e000640 1 -- start start 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f904806e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9048157a20 0x7f90480cc390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f90480cc8d0 0x7f904806a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904e000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90480ccd60 con 0x7f9048157a20 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9048161c70 con 0x7f90480cc8d0 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904806de90 con 0x7f9048157640 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f9047e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9048157a20 0x7f90480cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f9047e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9048157a20 0x7f90480cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48648/0 (socket says 172.21.15.53:48648) 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f9047e00640 1 -- 172.21.15.53:0/4028633692 learned_addr learned my addr 172.21.15.53:0/4028633692 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904d600640 1 --2- 172.21.15.53:0/4028633692 >> v2:172.21.15.146:3300/0 conn(0x7f90480cc8d0 0x7f904806a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.861+0000 7f904cc00640 1 --2- 172.21.15.53:0/4028633692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f904806e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.862+0000 7f9047e00640 1 -- 172.21.15.53:0/4028633692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 msgr2=0x7f904806e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.862+0000 7f9047e00640 1 --2- 172.21.15.53:0/4028633692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f904806e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.862+0000 7f9047e00640 1 -- 172.21.15.53:0/4028633692 >> v2:172.21.15.146:3300/0 conn(0x7f90480cc8d0 msgr2=0x7f904806a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:35.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.862+0000 7f9047e00640 1 --2- 172.21.15.53:0/4028633692 >> v2:172.21.15.146:3300/0 conn(0x7f90480cc8d0 0x7f904806a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.862+0000 7f9047e00640 1 -- 172.21.15.53:0/4028633692 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904806e170 con 0x7f9048157a20 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.863+0000 7f9047e00640 1 --2- 172.21.15.53:0/4028633692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9048157a20 0x7f90480cc390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f904815d830 tx=0x7f903002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d07bf5a3b28b121a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.863+0000 7f9047400640 1 -- 172.21.15.53:0/4028633692 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9030031870 con 0x7f9048157a20 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.863+0000 7f9047400640 1 -- 172.21.15.53:0/4028633692 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90300319d0 con 0x7f9048157a20 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.863+0000 7f9047400640 1 -- 172.21.15.53:0/4028633692 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9030031d60 con 0x7f9048157a20 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 -- 172.21.15.53:0/4028633692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9048157a20 msgr2=0x7f90480cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 --2- 172.21.15.53:0/4028633692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9048157a20 0x7f90480cc390 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f904815d830 tx=0x7f903002b910 comp rx=0 tx=0).stop 2024-01-16T15:00:35.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 -- 172.21.15.53:0/4028633692 shutdown_connections 2024-01-16T15:00:35.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 --2- 172.21.15.53:0/4028633692 >> v2:172.21.15.146:3300/0 conn(0x7f90480cc8d0 0x7f904806a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 --2- 172.21.15.53:0/4028633692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9048157a20 0x7f90480cc390 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 --2- 172.21.15.53:0/4028633692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f904806e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 -- 172.21.15.53:0/4028633692 >> 172.21.15.53:0/4028633692 conn(0x7f90480b1ba0 msgr2=0x7f90480b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:35.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 -- 172.21.15.53:0/4028633692 shutdown_connections 2024-01-16T15:00:35.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.864+0000 7f904e000640 1 -- 172.21.15.53:0/4028633692 wait complete. 2024-01-16T15:00:35.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.866+0000 7f904e000640 1 Processor -- start 2024-01-16T15:00:35.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.867+0000 7f904e000640 1 -- start start 2024-01-16T15:00:35.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 0x7f90481f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f90481f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9048157a20 0x7f904815ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904e000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90481f2b60 con 0x7f90480cc8d0 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90481f68d0 con 0x7f9048157a20 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90481f6bb0 con 0x7f9048157640 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 0x7f90481f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 0x7f90481f62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48650/0 (socket says 172.21.15.53:48650) 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.868+0000 7f904cc00640 1 -- 172.21.15.53:0/1106698014 learned_addr learned my addr 172.21.15.53:0/1106698014 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904d600640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9048157a20 0x7f904815ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f9047e00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f90481f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904cc00640 1 -- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 msgr2=0x7f90481f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904cc00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f90481f2620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904cc00640 1 -- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9048157a20 msgr2=0x7f904815ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:35.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904cc00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9048157a20 0x7f904815ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:35.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904cc00640 1 -- 172.21.15.53:0/1106698014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90300094d0 con 0x7f90480cc8d0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904d600640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9048157a20 0x7f904815ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.869+0000 7f904cc00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 0x7f90481f62d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f903800d800 tx=0x7f903800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.870+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903800b6c0 con 0x7f90480cc8d0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.870+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903800b820 con 0x7f90480cc8d0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.870+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90481f6e90 con 0x7f90480cc8d0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.870+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903800bbb0 con 0x7f90480cc8d0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.870+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90480af9c0 con 0x7f90480cc8d0 2024-01-16T15:00:35.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.871+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90480cd940 con 0x7f90480cc8d0 2024-01-16T15:00:35.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.872+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f903800bd10 con 0x7f90480cc8d0 2024-01-16T15:00:35.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.873+0000 7f9044c00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f902006b2c0 0x7f902006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:35.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.873+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f903808a040 con 0x7f90480cc8d0 2024-01-16T15:00:35.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.874+0000 7f9047e00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f902006b2c0 0x7f902006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:35.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.875+0000 7f9047e00640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f902006b2c0 0x7f902006d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9030036000 tx=0x7f903003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:35.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:35.878+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9038054f40 con 0x7f90480cc8d0 2024-01-16T15:00:36.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.071+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9048067b60 con 0x7f90480cc8d0 2024-01-16T15:00:36.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.072+0000 7f9044c00640 1 -- 172.21.15.53:0/1106698014 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9038058560 con 0x7f90480cc8d0 2024-01-16T15:00:36.053 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f902006b2c0 msgr2=0x7f902006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f902006b2c0 0x7f902006d780 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9030036000 tx=0x7f903003c040 comp rx=0 tx=0).stop 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 msgr2=0x7f90481f62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 0x7f90481f62d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f903800d800 tx=0x7f903800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 shutdown_connections 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f902006b2c0 0x7f902006d780 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9048157a20 0x7f904815ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9048157640 0x7f90481f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 --2- 172.21.15.53:0/1106698014 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90480cc8d0 0x7f90481f62d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 >> 172.21.15.53:0/1106698014 conn(0x7f90480b1ba0 msgr2=0x7f90480b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 shutdown_connections 2024-01-16T15:00:36.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.075+0000 7f904e000640 1 -- 172.21.15.53:0/1106698014 wait complete. 2024-01-16T15:00:36.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:36.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.317+0000 7f2800200640 1 Processor -- start 2024-01-16T15:00:36.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.318+0000 7f2800200640 1 -- start start 2024-01-16T15:00:36.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.319+0000 7f2800200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27f80cd240 0x7f27f80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.319+0000 7f2800200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f809a170 0x7f27f80cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.319+0000 7f2800200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f806cea0 con 0x7f27f806c6b0 2024-01-16T15:00:36.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.319+0000 7f2800200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f806cbc0 con 0x7f27f80cd240 2024-01-16T15:00:36.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.319+0000 7f2800200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f806f350 con 0x7f27f809a170 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fd800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27f80cd240 0x7f27f80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fce00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f27f806c6b0 0x7f27f8099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47148/0 (socket says 172.21.15.53:47148) 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fce00640 1 -- 172.21.15.53:0/1685281976 learned_addr learned my addr 172.21.15.53:0/1685281976 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fe200640 1 --2- 172.21.15.53:0/1685281976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f809a170 0x7f27f80cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fd800640 1 -- 172.21.15.53:0/1685281976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f809a170 msgr2=0x7f27f80cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fd800640 1 --2- 172.21.15.53:0/1685281976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f809a170 0x7f27f80cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fd800640 1 -- 172.21.15.53:0/1685281976 >> v1:172.21.15.53:6789/0 conn(0x7f27f806c6b0 legacy=0x7f27f8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27fd800640 1 -- 172.21.15.53:0/1685281976 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f806f630 con 0x7f27f80cd240 2024-01-16T15:00:36.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.320+0000 7f27f7e00640 1 -- 172.21.15.53:0/1685281976 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349818396 0 0) 0x7f27f806cea0 con 0x7f27f806c6b0 2024-01-16T15:00:36.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.322+0000 7f27fd800640 1 --2- 172.21.15.53:0/1685281976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f80cd240 0x7f27f80b3820 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f27e80097a0 tx=0x7f27e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a25462341c4828 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.322+0000 7f27f7e00640 1 -- 172.21.15.53:0/1685281976 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8031830 con 0x7f27f80cd240 2024-01-16T15:00:36.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.322+0000 7f27f7e00640 1 -- 172.21.15.53:0/1685281976 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e8031990 con 0x7f27f80cd240 2024-01-16T15:00:36.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.322+0000 7f27f7e00640 1 -- 172.21.15.53:0/1685281976 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8031d20 con 0x7f27f80cd240 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.322+0000 7f2800200640 1 -- 172.21.15.53:0/1685281976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f80cd240 msgr2=0x7f27f80b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.322+0000 7f2800200640 1 --2- 172.21.15.53:0/1685281976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f80cd240 0x7f27f80b3820 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f27e80097a0 tx=0x7f27e802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.323+0000 7f2800200640 1 -- 172.21.15.53:0/1685281976 shutdown_connections 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.323+0000 7f2800200640 1 --2- 172.21.15.53:0/1685281976 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f809a170 0x7f27f80cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.323+0000 7f2800200640 1 --2- 172.21.15.53:0/1685281976 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f80cd240 0x7f27f80b3820 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.323+0000 7f2800200640 1 -- 172.21.15.53:0/1685281976 >> 172.21.15.53:0/1685281976 conn(0x7f27f80b2d00 msgr2=0x7f27f80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.323+0000 7f2800200640 1 -- 172.21.15.53:0/1685281976 shutdown_connections 2024-01-16T15:00:36.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.323+0000 7f2800200640 1 -- 172.21.15.53:0/1685281976 wait complete. 2024-01-16T15:00:36.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.324+0000 7f2800200640 1 Processor -- start 2024-01-16T15:00:36.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 -- start start 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f806c6b0 0x7f27f8155760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 0x7f27f80c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f80cd240 0x7f27f80cf640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f81f3a40 con 0x7f27f809a170 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f8155ca0 con 0x7f27f806c6b0 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.325+0000 7f2800200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f81f29c0 con 0x7f27f80cd240 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 0x7f27f80c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 0x7f27f80c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48662/0 (socket says 172.21.15.53:48662) 2024-01-16T15:00:36.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 -- 172.21.15.53:0/1378884370 learned_addr learned my addr 172.21.15.53:0/1378884370 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fe200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f80cd240 0x7f27f80cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 -- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f80cd240 msgr2=0x7f27f80cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f80cd240 0x7f27f80cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fd800640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f806c6b0 0x7f27f8155760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 -- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f806c6b0 msgr2=0x7f27f8155760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f806c6b0 0x7f27f8155760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.326+0000 7f27fce00640 1 -- 172.21.15.53:0/1378884370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e8009400 con 0x7f27f809a170 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.327+0000 7f27fce00640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 0x7f27f80c7480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c410 tx=0x7f27e000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.327+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e00074e0 con 0x7f27f809a170 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.327+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e0007660 con 0x7f27f809a170 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.327+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e0004a20 con 0x7f27f809a170 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.327+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f81f2c80 con 0x7f27f809a170 2024-01-16T15:00:36.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.327+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f81f3240 con 0x7f27f809a170 2024-01-16T15:00:36.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.328+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f80675d0 con 0x7f27f809a170 2024-01-16T15:00:36.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.329+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f27e0004b80 con 0x7f27f809a170 2024-01-16T15:00:36.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.330+0000 7f27f5600640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27d006b310 0x7f27d006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.330+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f27e0089a10 con 0x7f27f809a170 2024-01-16T15:00:36.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.331+0000 7f27fd800640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27d006b310 0x7f27d006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.332+0000 7f27fd800640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27d006b310 0x7f27d006d7d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27e8036000 tx=0x7f27e8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.334+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27e00582f0 con 0x7f27f809a170 2024-01-16T15:00:36.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.529+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27f8073d00 con 0x7f27f809a170 2024-01-16T15:00:36.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.530+0000 7f27f5600640 1 -- 172.21.15.53:0/1378884370 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f27e0057e80 con 0x7f27f809a170 2024-01-16T15:00:36.512 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:36.512 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:36.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27d006b310 msgr2=0x7f27d006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27d006b310 0x7f27d006d7d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f27e8036000 tx=0x7f27e8008030 comp rx=0 tx=0).stop 2024-01-16T15:00:36.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 msgr2=0x7f27f80c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 0x7f27f80c7480 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f27e000c410 tx=0x7f27e000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 shutdown_connections 2024-01-16T15:00:36.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27d006b310 0x7f27d006d7d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27f80cd240 0x7f27f80cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27f809a170 0x7f27f80c7480 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 --2- 172.21.15.53:0/1378884370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27f806c6b0 0x7f27f8155760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 >> 172.21.15.53:0/1378884370 conn(0x7f27f80b2d00 msgr2=0x7f27f81607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 shutdown_connections 2024-01-16T15:00:36.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.534+0000 7f2800200640 1 -- 172.21.15.53:0/1378884370 wait complete. 2024-01-16T15:00:36.533 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:36.534 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.782+0000 7fb34e800640 1 Processor -- start 2024-01-16T15:00:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.782+0000 7fb34e800640 1 -- start start 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb3480cf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3480b20d0 0x7fb3480b0000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34e800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480ce760 con 0x7fb34815d960 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480d0a00 con 0x7fb3480b20d0 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480cf250 con 0x7fb34815d580 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34ca00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb34815d960 0x7fb3480cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47154/0 (socket says 172.21.15.53:47154) 2024-01-16T15:00:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34ca00640 1 -- 172.21.15.53:0/1593810818 learned_addr learned my addr 172.21.15.53:0/1593810818 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34de00640 1 --2- 172.21.15.53:0/1593810818 >> v2:172.21.15.146:3300/0 conn(0x7fb3480b20d0 0x7fb3480b0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb34d400640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb3480cf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.784+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362106307 0 0) 0x7fb3480ce760 con 0x7fb34815d960 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328003520 con 0x7fb34815d960 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2088679971 0 0) 0x7fb328003520 con 0x7fb34815d960 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 msgr2=0x7fb3480cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb3480cf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 >> v2:172.21.15.146:3300/0 conn(0x7fb3480b20d0 msgr2=0x7fb3480b0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 --2- 172.21.15.53:0/1593810818 >> v2:172.21.15.146:3300/0 conn(0x7fb3480b20d0 0x7fb3480b0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480d1f50 con 0x7fb34815d960 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb330001670 con 0x7fb34815d960 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 >> v1:172.21.15.53:6789/0 conn(0x7fb34815d960 legacy=0x7fb3480cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb328003970 0x7fb328003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb328004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb328005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480d0a00 con 0x7fb3280047c0 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480cf250 con 0x7fb328003970 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb32800b9d0 con 0x7fb34815d580 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.785+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb328004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34d400640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb328003970 0x7fb328003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34de00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb328005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34ca00640 1 -- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 msgr2=0x7fb328005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb328005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34ca00640 1 -- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb328003970 msgr2=0x7fb328003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb328003970 0x7fb328003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34ca00640 1 -- 172.21.15.53:0/1593810818 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328003520 con 0x7fb3280047c0 2024-01-16T15:00:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.786+0000 7fb34de00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb34815d580 0x7fb328005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.787+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb328004c40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c140 tx=0x7fb3280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6539846db19d645 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.787+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb330005ad0 con 0x7fb3280047c0 2024-01-16T15:00:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb330005c50 con 0x7fb3280047c0 2024-01-16T15:00:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb347e00640 1 -- 172.21.15.53:0/1593810818 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33002c810 con 0x7fb3280047c0 2024-01-16T15:00:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34e800640 1 -- 172.21.15.53:0/1593810818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 msgr2=0x7fb328004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34e800640 1 --2- 172.21.15.53:0/1593810818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb328004c40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb33000c140 tx=0x7fb3280057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34de00640 1 -- 172.21.15.53:0/1593810818 reap_dead start 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34e800640 1 -- 172.21.15.53:0/1593810818 shutdown_connections 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34e800640 1 -- 172.21.15.53:0/1593810818 >> 172.21.15.53:0/1593810818 conn(0x7fb3480d3040 msgr2=0x7fb3480d3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34e800640 1 -- 172.21.15.53:0/1593810818 shutdown_connections 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.788+0000 7fb34e800640 1 -- 172.21.15.53:0/1593810818 wait complete. 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.790+0000 7fb34e800640 1 Processor -- start 2024-01-16T15:00:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.790+0000 7fb34e800640 1 -- start start 2024-01-16T15:00:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb3480b86e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3480cb190 0x7fb3480b6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3481588c0 0x7fb3480b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34e800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480b6c60 con 0x7fb3280047c0 2024-01-16T15:00:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480b6290 con 0x7fb3480cb190 2024-01-16T15:00:36.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3480cb610 con 0x7fb3481588c0 2024-01-16T15:00:36.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb3480b86e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb3480b86e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48678/0 (socket says 172.21.15.53:48678) 2024-01-16T15:00:36.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.791+0000 7fb34d400640 1 -- 172.21.15.53:0/1295040051 learned_addr learned my addr 172.21.15.53:0/1295040051 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:36.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3480cb190 0x7fb3480b6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34de00640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3481588c0 0x7fb3480b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34d400640 1 -- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3481588c0 msgr2=0x7fb3480b4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34d400640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3481588c0 0x7fb3480b4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34d400640 1 -- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3480cb190 msgr2=0x7fb3480b6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34d400640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3480cb190 0x7fb3480b6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34d400640 1 -- 172.21.15.53:0/1295040051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480cb8c0 con 0x7fb3280047c0 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34de00640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3481588c0 0x7fb3480b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34d400640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb3480b86e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb3380115c0 tx=0x7fb338011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338015870 con 0x7fb3280047c0 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3380159d0 con 0x7fb3280047c0 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338015d80 con 0x7fb3280047c0 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.792+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3480cbba0 con 0x7fb3280047c0 2024-01-16T15:00:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.793+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3480b5f20 con 0x7fb3280047c0 2024-01-16T15:00:36.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.794+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb348067450 con 0x7fb3280047c0 2024-01-16T15:00:36.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.795+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb33800f480 con 0x7fb3280047c0 2024-01-16T15:00:36.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.796+0000 7fb345600640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb32006b2f0 0x7fb32006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:36.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.796+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb33808a3c0 con 0x7fb3280047c0 2024-01-16T15:00:36.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.796+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb32006b2f0 0x7fb32006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:36.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.797+0000 7fb34ca00640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb32006b2f0 0x7fb32006d7b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb330034000 tx=0x7fb330007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:36.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.800+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb338055190 con 0x7fb3280047c0 2024-01-16T15:00:36.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.995+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3480ce4e0 con 0x7fb3280047c0 2024-01-16T15:00:36.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.996+0000 7fb345600640 1 -- 172.21.15.53:0/1295040051 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb3380587b0 con 0x7fb3280047c0 2024-01-16T15:00:36.978 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb32006b2f0 msgr2=0x7fb32006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb32006b2f0 0x7fb32006d7b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb330034000 tx=0x7fb330007000 comp rx=0 tx=0).stop 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 msgr2=0x7fb3480b86e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb3480b86e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb3380115c0 tx=0x7fb338011a80 comp rx=0 tx=0).stop 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 shutdown_connections 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb32006b2f0 0x7fb32006d7b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3481588c0 0x7fb3480b4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3480cb190 0x7fb3480b6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 --2- 172.21.15.53:0/1295040051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3280047c0 0x7fb3480b86e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:36.999+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 >> 172.21.15.53:0/1295040051 conn(0x7fb3480d3040 msgr2=0x7fb3480c92f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.000+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 shutdown_connections 2024-01-16T15:00:36.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.000+0000 7fb34e800640 1 -- 172.21.15.53:0/1295040051 wait complete. 2024-01-16T15:00:36.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:37.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.245+0000 7f7887600640 1 Processor -- start 2024-01-16T15:00:37.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.245+0000 7f7887600640 1 -- start start 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7887600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 0x7f788015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7887600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f788006f550 0x7f788006f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7887600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f788006eb00 con 0x7f788006a3e0 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7887600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f788006f0c0 con 0x7f788006f550 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7887600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f788006ab80 con 0x7f78800cd960 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f787fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f788006a3e0 0x7f78800cdd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47170/0 (socket says 172.21.15.53:47170) 2024-01-16T15:00:37.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f787fe00640 1 -- 172.21.15.53:0/2980492775 learned_addr learned my addr 172.21.15.53:0/2980492775 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:37.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7885600640 1 --2- 172.21.15.53:0/2980492775 >> v2:172.21.15.146:3300/0 conn(0x7f788006f550 0x7f788006f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f7884c00640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 0x7f788015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.247+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886635226 0 0) 0x7f788006eb00 con 0x7f788006a3e0 2024-01-16T15:00:37.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7860003520 con 0x7f788006a3e0 2024-01-16T15:00:37.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 234710160 0 0) 0x7f7860003520 con 0x7f788006a3e0 2024-01-16T15:00:37.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 msgr2=0x7f788015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 0x7f788015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 >> v2:172.21.15.146:3300/0 conn(0x7f788006f550 msgr2=0x7f788006f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 --2- 172.21.15.53:0/2980492775 >> v2:172.21.15.146:3300/0 conn(0x7f788006f550 0x7f788006f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788006ae60 con 0x7f788006a3e0 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7868001670 con 0x7f788006a3e0 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 >> v1:172.21.15.53:6789/0 conn(0x7f788006a3e0 legacy=0x7f78800cdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7860003970 0x7f7860003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 0x7f78600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f7860005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788006f0c0 con 0x7f7860004d00 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f788006ab80 con 0x7f7860003970 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786000b9b0 con 0x7f78800cd960 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.248+0000 7f7885600640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f7860005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.249+0000 7f7884c00640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7860003970 0x7f7860003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.249+0000 7f7885600640 1 -- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 msgr2=0x7f78600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:37.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.249+0000 7f7885600640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800cd960 0x7f78600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.249+0000 7f7885600640 1 -- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7860003970 msgr2=0x7f7860003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:37.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.249+0000 7f7885600640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7860003970 0x7f7860003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.249+0000 7f7885600640 1 -- 172.21.15.53:0/2980492775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7860003520 con 0x7f7860004d00 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.250+0000 7f7885600640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f7860005180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7878009fc0 tx=0x7f78600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e03213c06ea1240 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.250+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787802cc60 con 0x7f7860004d00 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.250+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787800f040 con 0x7f7860004d00 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.250+0000 7f787f400640 1 -- 172.21.15.53:0/2980492775 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7878034440 con 0x7f7860004d00 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7887600640 1 -- 172.21.15.53:0/2980492775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 msgr2=0x7f7860005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7887600640 1 --2- 172.21.15.53:0/2980492775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f7860005180 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7878009fc0 tx=0x7f78600057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7885600640 1 -- 172.21.15.53:0/2980492775 reap_dead start 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7887600640 1 -- 172.21.15.53:0/2980492775 shutdown_connections 2024-01-16T15:00:37.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7887600640 1 -- 172.21.15.53:0/2980492775 >> 172.21.15.53:0/2980492775 conn(0x7f78800b1dd0 msgr2=0x7f78800b21b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:37.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7887600640 1 -- 172.21.15.53:0/2980492775 shutdown_connections 2024-01-16T15:00:37.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.251+0000 7f7887600640 1 -- 172.21.15.53:0/2980492775 wait complete. 2024-01-16T15:00:37.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.253+0000 7f7887600640 1 Processor -- start 2024-01-16T15:00:37.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.253+0000 7f7887600640 1 -- start start 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7887600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f78800ae960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7887600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800b3470 0x7f78800aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7887600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78801f60b0 0x7f78801f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7887600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7880158440 con 0x7f7860004d00 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7887600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78800b3930 con 0x7f78801f60b0 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7887600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78800b3c10 con 0x7f78800b3470 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7884c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f78800ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7884c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f78800ae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48694/0 (socket says 172.21.15.53:48694) 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.254+0000 7f7884c00640 1 -- 172.21.15.53:0/2763870967 learned_addr learned my addr 172.21.15.53:0/2763870967 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f787fe00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800b3470 0x7f78800aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7884c00640 1 -- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800b3470 msgr2=0x7f78800aeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7885600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78801f60b0 0x7f78801f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7884c00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800b3470 0x7f78800aeea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7884c00640 1 -- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78801f60b0 msgr2=0x7f78801f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7884c00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78801f60b0 0x7f78801f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7884c00640 1 -- 172.21.15.53:0/2763870967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78800b3ef0 con 0x7f7860004d00 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f787fe00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800b3470 0x7f78800aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.255+0000 7f7884c00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f78800ae960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f78700116a0 tx=0x7f7870011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.256+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000f9c0 con 0x7f7860004d00 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.256+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787000fb20 con 0x7f7860004d00 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.256+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78800b41d0 con 0x7f7860004d00 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.256+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787000fed0 con 0x7f7860004d00 2024-01-16T15:00:37.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.256+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78800b49d0 con 0x7f7860004d00 2024-01-16T15:00:37.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.257+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78800675d0 con 0x7f7860004d00 2024-01-16T15:00:37.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.258+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7870015460 con 0x7f7860004d00 2024-01-16T15:00:37.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.259+0000 7f787cc00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f785806b2c0 0x7f785806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.259+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f787008a220 con 0x7f7860004d00 2024-01-16T15:00:37.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.260+0000 7f787fe00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f785806b2c0 0x7f785806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.261+0000 7f787fe00640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f785806b2c0 0x7f785806d780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78680017a0 tx=0x7f7868009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.264+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7870054ff0 con 0x7f7860004d00 2024-01-16T15:00:37.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.460+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78800b1680 con 0x7f7860004d00 2024-01-16T15:00:37.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.461+0000 7f787cc00640 1 -- 172.21.15.53:0/2763870967 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7870058610 con 0x7f7860004d00 2024-01-16T15:00:37.443 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f785806b2c0 msgr2=0x7f785806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f785806b2c0 0x7f785806d780 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f78680017a0 tx=0x7f7868009040 comp rx=0 tx=0).stop 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 msgr2=0x7f78800ae960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f78800ae960 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f78700116a0 tx=0x7f7870011b60 comp rx=0 tx=0).stop 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 shutdown_connections 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f785806b2c0 0x7f785806d780 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78801f60b0 0x7f78801f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78800b3470 0x7f78800aeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 --2- 172.21.15.53:0/2763870967 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7860004d00 0x7f78800ae960 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.464+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 >> 172.21.15.53:0/2763870967 conn(0x7f78800b1dd0 msgr2=0x7f7880159eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.465+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 shutdown_connections 2024-01-16T15:00:37.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.465+0000 7f7887600640 1 -- 172.21.15.53:0/2763870967 wait complete. 2024-01-16T15:00:37.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:37.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.714+0000 7f4347a00640 1 Processor -- start 2024-01-16T15:00:37.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.714+0000 7f4347a00640 1 -- start start 2024-01-16T15:00:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.716+0000 7f4347a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f434006d7d0 0x7f43400aedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.716+0000 7f4347a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43400b27c0 0x7f43400af2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.716+0000 7f4347a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43400b3670 con 0x7f434015d240 2024-01-16T15:00:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.716+0000 7f4347a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43400b1d20 con 0x7f434006d7d0 2024-01-16T15:00:37.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.716+0000 7f4347a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43400b38d0 con 0x7f43400b27c0 2024-01-16T15:00:37.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.717+0000 7f4345000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f434015d240 0x7f43400b2f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:40250/0 (socket says 172.21.15.53:40250) 2024-01-16T15:00:37.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.717+0000 7f4345000640 1 -- 172.21.15.53:0/172316662 learned_addr learned my addr 172.21.15.53:0/172316662 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:37.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.717+0000 7f4345a00640 1 --2- 172.21.15.53:0/172316662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43400b27c0 0x7f43400af2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.717+0000 7f433fe00640 1 --2- 172.21.15.53:0/172316662 >> v2:172.21.15.146:3300/0 conn(0x7f434006d7d0 0x7f43400aedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.718+0000 7f433f400640 1 -- 172.21.15.53:0/172316662 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254547251 0 0) 0x7f43400b3670 con 0x7f434015d240 2024-01-16T15:00:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.718+0000 7f433f400640 1 -- 172.21.15.53:0/172316662 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4320003520 con 0x7f434015d240 2024-01-16T15:00:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.718+0000 7f4345a00640 1 -- 172.21.15.53:0/172316662 >> v2:172.21.15.146:3300/0 conn(0x7f434006d7d0 msgr2=0x7f43400aedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.718+0000 7f4345a00640 1 --2- 172.21.15.53:0/172316662 >> v2:172.21.15.146:3300/0 conn(0x7f434006d7d0 0x7f43400aedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.718+0000 7f4345a00640 1 -- 172.21.15.53:0/172316662 >> v1:172.21.15.146:6789/0 conn(0x7f434015d240 legacy=0x7f43400b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.718+0000 7f4345a00640 1 -- 172.21.15.53:0/172316662 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43400b1fc0 con 0x7f43400b27c0 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.719+0000 7f4345a00640 1 --2- 172.21.15.53:0/172316662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43400b27c0 0x7f43400af2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4338009d90 tx=0x7f433802be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c57f1dcf230c214 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.719+0000 7f433f400640 1 -- 172.21.15.53:0/172316662 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433802cbc0 con 0x7f43400b27c0 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.719+0000 7f433f400640 1 -- 172.21.15.53:0/172316662 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433802cd20 con 0x7f43400b27c0 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f433f400640 1 -- 172.21.15.53:0/172316662 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4338034660 con 0x7f43400b27c0 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 -- 172.21.15.53:0/172316662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43400b27c0 msgr2=0x7f43400af2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 --2- 172.21.15.53:0/172316662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43400b27c0 0x7f43400af2f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4338009d90 tx=0x7f433802be10 comp rx=0 tx=0).stop 2024-01-16T15:00:37.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 -- 172.21.15.53:0/172316662 shutdown_connections 2024-01-16T15:00:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 --2- 172.21.15.53:0/172316662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43400b27c0 0x7f43400af2f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 --2- 172.21.15.53:0/172316662 >> v2:172.21.15.146:3300/0 conn(0x7f434006d7d0 0x7f43400aedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 -- 172.21.15.53:0/172316662 >> 172.21.15.53:0/172316662 conn(0x7f434006b4f0 msgr2=0x7f434006b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 -- 172.21.15.53:0/172316662 shutdown_connections 2024-01-16T15:00:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.720+0000 7f4347a00640 1 -- 172.21.15.53:0/172316662 wait complete. 2024-01-16T15:00:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.722+0000 7f4347a00640 1 Processor -- start 2024-01-16T15:00:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.722+0000 7f4347a00640 1 -- start start 2024-01-16T15:00:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.723+0000 7f4347a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 0x7f4340156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.723+0000 7f4347a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43400b27c0 0x7f43400c5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.723+0000 7f4347a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f434015d240 0x7f43400c4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.723+0000 7f4347a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4340159160 con 0x7f434006d7d0 2024-01-16T15:00:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.723+0000 7f4347a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4340159440 con 0x7f43400b27c0 2024-01-16T15:00:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.723+0000 7f4347a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4340159720 con 0x7f434015d240 2024-01-16T15:00:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 0x7f4340156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 0x7f4340156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48708/0 (socket says 172.21.15.53:48708) 2024-01-16T15:00:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 -- 172.21.15.53:0/856571090 learned_addr learned my addr 172.21.15.53:0/856571090 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f433fe00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43400b27c0 0x7f43400c5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f434015d240 0x7f43400c4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 -- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f434015d240 msgr2=0x7f43400c4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f434015d240 0x7f43400c4ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 -- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43400b27c0 msgr2=0x7f43400c5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43400b27c0 0x7f43400c5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 -- 172.21.15.53:0/856571090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4338009a70 con 0x7f434006d7d0 2024-01-16T15:00:37.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.724+0000 7f4345000640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 0x7f4340156260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f43300010e0 tx=0x7f433000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.725+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4330005ce0 con 0x7f434006d7d0 2024-01-16T15:00:37.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.725+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4330005e60 con 0x7f434006d7d0 2024-01-16T15:00:37.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.725+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43300072f0 con 0x7f434006d7d0 2024-01-16T15:00:37.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.725+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4340159a00 con 0x7f434006d7d0 2024-01-16T15:00:37.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.725+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43401567a0 con 0x7f434006d7d0 2024-01-16T15:00:37.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.726+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43400675d0 con 0x7f434006d7d0 2024-01-16T15:00:37.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.727+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4330007450 con 0x7f434006d7d0 2024-01-16T15:00:37.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.728+0000 7f433cc00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431806b2c0 0x7f431806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:37.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.728+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f43300891e0 con 0x7f434006d7d0 2024-01-16T15:00:37.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.729+0000 7f433fe00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431806b2c0 0x7f431806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:37.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.730+0000 7f433fe00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431806b2c0 0x7f431806d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f43400cf040 tx=0x7f4328008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:37.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.733+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4330054050 con 0x7f434006d7d0 2024-01-16T15:00:37.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.931+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43400725d0 con 0x7f434006d7d0 2024-01-16T15:00:37.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.932+0000 7f433cc00640 1 -- 172.21.15.53:0/856571090 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4330057670 con 0x7f434006d7d0 2024-01-16T15:00:37.913 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:37.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431806b2c0 msgr2=0x7f431806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431806b2c0 0x7f431806d780 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f43400cf040 tx=0x7f4328008040 comp rx=0 tx=0).stop 2024-01-16T15:00:37.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 msgr2=0x7f4340156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 0x7f4340156260 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f43300010e0 tx=0x7f433000c890 comp rx=0 tx=0).stop 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 shutdown_connections 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431806b2c0 0x7f431806d780 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f434015d240 0x7f43400c4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43400b27c0 0x7f43400c5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 --2- 172.21.15.53:0/856571090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f434006d7d0 0x7f4340156260 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 >> 172.21.15.53:0/856571090 conn(0x7f434006b4f0 msgr2=0x7f434006b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 shutdown_connections 2024-01-16T15:00:37.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:37.935+0000 7f4347a00640 1 -- 172.21.15.53:0/856571090 wait complete. 2024-01-16T15:00:38.006 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) Test that when one clone is in progress, one progress bar is printed 2024-01-16T15:00:38.006 INFO:tasks.cephfs_test_runner:test_progress_bar_for_1_clone (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:00:38.007 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:38.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.242+0000 7f3bbb200640 1 Processor -- start 2024-01-16T15:00:38.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.242+0000 7f3bbb200640 1 -- start start 2024-01-16T15:00:38.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.243+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc143c40 0x7f3bbc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.243+0000 7f3bbb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3bbc0717d0 0x7f3bbc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.243+0000 7f3bbb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bbc070e50 con 0x7f3bbc0cca40 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.243+0000 7f3bbb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bbc0723c0 con 0x7f3bbc0717d0 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.243+0000 7f3bbb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc0cce70 con 0x7f3bbc143c40 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bb3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3bbc0717d0 0x7f3bbc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bba800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3bbc0cca40 0x7f3bbc15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47174/0 (socket says 172.21.15.53:47174) 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bba800640 1 -- 172.21.15.53:0/1642034005 learned_addr learned my addr 172.21.15.53:0/1642034005 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bb9e00640 1 --2- 172.21.15.53:0/1642034005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc143c40 0x7f3bbc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bb3e00640 1 -- 172.21.15.53:0/1642034005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc143c40 msgr2=0x7f3bbc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:38.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bb3e00640 1 --2- 172.21.15.53:0/1642034005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc143c40 0x7f3bbc071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bb3e00640 1 -- 172.21.15.53:0/1642034005 >> v1:172.21.15.53:6789/0 conn(0x7f3bbc0cca40 legacy=0x7f3bbc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.244+0000 7f3bb3e00640 1 -- 172.21.15.53:0/1642034005 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc06e170 con 0x7f3bbc0717d0 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.245+0000 7f3bb3e00640 1 --2- 172.21.15.53:0/1642034005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0717d0 0x7f3bbc0cc500 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40097a0 tx=0x7f3ba402b660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92697490c92446a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bb9400640 1 -- 172.21.15.53:0/1642034005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4030710 con 0x7f3bbc0717d0 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bb9400640 1 -- 172.21.15.53:0/1642034005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba402e050 con 0x7f3bbc0717d0 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bb9400640 1 -- 172.21.15.53:0/1642034005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4030a90 con 0x7f3bbc0717d0 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bbb200640 1 -- 172.21.15.53:0/1642034005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0717d0 msgr2=0x7f3bbc0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bbb200640 1 --2- 172.21.15.53:0/1642034005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0717d0 0x7f3bbc0cc500 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3ba40097a0 tx=0x7f3ba402b660 comp rx=0 tx=0).stop 2024-01-16T15:00:38.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bbb200640 1 -- 172.21.15.53:0/1642034005 shutdown_connections 2024-01-16T15:00:38.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bbb200640 1 --2- 172.21.15.53:0/1642034005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0717d0 0x7f3bbc0cc500 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bbb200640 1 --2- 172.21.15.53:0/1642034005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc143c40 0x7f3bbc071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.246+0000 7f3bbb200640 1 -- 172.21.15.53:0/1642034005 >> 172.21.15.53:0/1642034005 conn(0x7f3bbc06a800 msgr2=0x7f3bbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:38.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.247+0000 7f3bbb200640 1 -- 172.21.15.53:0/1642034005 shutdown_connections 2024-01-16T15:00:38.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.247+0000 7f3bbb200640 1 -- 172.21.15.53:0/1642034005 wait complete. 2024-01-16T15:00:38.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.249+0000 7f3bbb200640 1 Processor -- start 2024-01-16T15:00:38.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.249+0000 7f3bbb200640 1 -- start start 2024-01-16T15:00:38.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.250+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc0717d0 0x7f3bbc15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.250+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 0x7f3bbc142520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.250+0000 7f3bbb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bbc143c40 0x7f3bbc142a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.250+0000 7f3bbb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc15ad30 con 0x7f3bbc0cca40 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.250+0000 7f3bbb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc138600 con 0x7f3bbc143c40 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.250+0000 7f3bbb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bbc137cf0 con 0x7f3bbc0717d0 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 0x7f3bbc142520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 0x7f3bbc142520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48722/0 (socket says 172.21.15.53:48722) 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 -- 172.21.15.53:0/244788547 learned_addr learned my addr 172.21.15.53:0/244788547 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb9e00640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc0717d0 0x7f3bbc15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 -- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc0717d0 msgr2=0x7f3bbc15a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc0717d0 0x7f3bbc15a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 -- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bbc143c40 msgr2=0x7f3bbc142a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bbc143c40 0x7f3bbc142a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 -- 172.21.15.53:0/244788547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba4009400 con 0x7f3bbc0cca40 2024-01-16T15:00:38.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.251+0000 7f3bb3e00640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 0x7f3bbc142520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4035040 tx=0x7f3ba40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:38.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.252+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba403b8d0 con 0x7f3bbc0cca40 2024-01-16T15:00:38.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.252+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba403ba50 con 0x7f3bbc0cca40 2024-01-16T15:00:38.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.252+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bbc1f3350 con 0x7f3bbc0cca40 2024-01-16T15:00:38.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.252+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba403be00 con 0x7f3bbc0cca40 2024-01-16T15:00:38.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.252+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bbc138290 con 0x7f3bbc0cca40 2024-01-16T15:00:38.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.253+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bbc067450 con 0x7f3bbc0cca40 2024-01-16T15:00:38.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.254+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3ba4039040 con 0x7f3bbc0cca40 2024-01-16T15:00:38.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.255+0000 7f3bb2000640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3b9406b2f0 0x7f3b9406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.255+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3ba40aae30 con 0x7f3bbc0cca40 2024-01-16T15:00:38.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.256+0000 7f3bb9e00640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3b9406b2f0 0x7f3b9406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.257+0000 7f3bb9e00640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3b9406b2f0 0x7f3b9406d7b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3bac006fd0 tx=0x7f3bac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:38.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.260+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ba4075d30 con 0x7f3bbc0cca40 2024-01-16T15:00:38.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.456+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bbc138740 con 0x7f3bbc0cca40 2024-01-16T15:00:38.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.458+0000 7f3bb2000640 1 -- 172.21.15.53:0/244788547 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16469 (secure 0 0 0) 0x7f3ba404a090 con 0x7f3bbc0cca40 2024-01-16T15:00:38.440 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:38.440 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":22,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T14:59:58.872348+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3b9406b2f0 msgr2=0x7f3b9406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3b9406b2f0 0x7f3b9406d7b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f3bac006fd0 tx=0x7f3bac008040 comp rx=0 tx=0).stop 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 msgr2=0x7f3bbc142520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 0x7f3bbc142520 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4035040 tx=0x7f3ba40046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 shutdown_connections 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3b9406b2f0 0x7f3b9406d7b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bbc143c40 0x7f3bbc142a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bbc0cca40 0x7f3bbc142520 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 --2- 172.21.15.53:0/244788547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bbc0717d0 0x7f3bbc15a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 >> 172.21.15.53:0/244788547 conn(0x7f3bbc06a800 msgr2=0x7f3bbc1374b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.461+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 shutdown_connections 2024-01-16T15:00:38.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.462+0000 7f3bbb200640 1 -- 172.21.15.53:0/244788547 wait complete. 2024-01-16T15:00:38.463 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:38.463 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:38.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.713+0000 7f0746c00640 1 Processor -- start 2024-01-16T15:00:38.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.714+0000 7f0746c00640 1 -- start start 2024-01-16T15:00:38.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0746c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f074015d4b0 0x7f07400d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0746c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f07400b41e0 0x7f07400cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0746c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07400d0320 con 0x7f07400cf9a0 2024-01-16T15:00:38.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0746c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400b2730 con 0x7f074015d4b0 2024-01-16T15:00:38.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0746c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07400d1ec0 con 0x7f07400b41e0 2024-01-16T15:00:38.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0745800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f074015d4b0 0x7f07400d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0745800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f074015d4b0 0x7f07400d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48736/0 (socket says 172.21.15.53:48736) 2024-01-16T15:00:38.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.715+0000 7f0745800640 1 -- 172.21.15.53:0/737549848 learned_addr learned my addr 172.21.15.53:0/737549848 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:38.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.716+0000 7f0746200640 1 --2- 172.21.15.53:0/737549848 >> v2:172.21.15.146:3300/0 conn(0x7f07400b41e0 0x7f07400cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.716+0000 7f0745800640 1 -- 172.21.15.53:0/737549848 >> v2:172.21.15.146:3300/0 conn(0x7f07400b41e0 msgr2=0x7f07400cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:38.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.716+0000 7f0745800640 1 --2- 172.21.15.53:0/737549848 >> v2:172.21.15.146:3300/0 conn(0x7f07400b41e0 0x7f07400cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.716+0000 7f0745800640 1 -- 172.21.15.53:0/737549848 >> v1:172.21.15.146:6789/0 conn(0x7f07400cf9a0 legacy=0x7f07400cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:38.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.716+0000 7f0745800640 1 -- 172.21.15.53:0/737549848 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07400b29d0 con 0x7f074015d4b0 2024-01-16T15:00:38.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.716+0000 7f0746200640 1 --2- 172.21.15.53:0/737549848 >> v2:172.21.15.146:3300/0 conn(0x7f07400b41e0 0x7f07400cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.717+0000 7f0745800640 1 --2- 172.21.15.53:0/737549848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f07400d1550 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f07400d0680 tx=0x7f073002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f81165df1c2c600d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.717+0000 7f073fe00640 1 -- 172.21.15.53:0/737549848 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730031870 con 0x7f074015d4b0 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.717+0000 7f073fe00640 1 -- 172.21.15.53:0/737549848 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07300319d0 con 0x7f074015d4b0 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.717+0000 7f073fe00640 1 -- 172.21.15.53:0/737549848 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0730031d60 con 0x7f074015d4b0 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 -- 172.21.15.53:0/737549848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 msgr2=0x7f07400d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 --2- 172.21.15.53:0/737549848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f07400d1550 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f07400d0680 tx=0x7f073002b910 comp rx=0 tx=0).stop 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 -- 172.21.15.53:0/737549848 shutdown_connections 2024-01-16T15:00:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 --2- 172.21.15.53:0/737549848 >> v2:172.21.15.146:3300/0 conn(0x7f07400b41e0 0x7f07400cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 --2- 172.21.15.53:0/737549848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f07400d1550 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 -- 172.21.15.53:0/737549848 >> 172.21.15.53:0/737549848 conn(0x7f07400d3000 msgr2=0x7f07400d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:38.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 -- 172.21.15.53:0/737549848 shutdown_connections 2024-01-16T15:00:38.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.718+0000 7f0746c00640 1 -- 172.21.15.53:0/737549848 wait complete. 2024-01-16T15:00:38.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.720+0000 7f0746c00640 1 Processor -- start 2024-01-16T15:00:38.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.720+0000 7f0746c00640 1 -- start start 2024-01-16T15:00:38.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07400b41e0 0x7f07400ccd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07400cf9a0 0x7f074009e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f0740148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074009cce0 con 0x7f074015d4b0 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074009cf80 con 0x7f07400b41e0 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074009d260 con 0x7f07400cf9a0 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f0740148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f0740148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48752/0 (socket says 172.21.15.53:48752) 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0746200640 1 -- 172.21.15.53:0/3768643046 learned_addr learned my addr 172.21.15.53:0/3768643046 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0745800640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07400b41e0 0x7f07400ccd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.721+0000 7f0744e00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07400cf9a0 0x7f074009e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746200640 1 -- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07400cf9a0 msgr2=0x7f074009e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746200640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07400cf9a0 0x7f074009e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746200640 1 -- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07400b41e0 msgr2=0x7f07400ccd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746200640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07400b41e0 0x7f07400ccd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746200640 1 -- 172.21.15.53:0/3768643046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300094d0 con 0x7f074015d4b0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0744e00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07400cf9a0 0x7f074009e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746200640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f0740148e40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f073800cea0 tx=0x7f073800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738003b90 con 0x7f074015d4b0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0738003d10 con 0x7f074015d4b0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0738004260 con 0x7f074015d4b0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074015da10 con 0x7f074015d4b0 2024-01-16T15:00:38.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.722+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074009d820 con 0x7f074015d4b0 2024-01-16T15:00:38.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.724+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0740067450 con 0x7f074015d4b0 2024-01-16T15:00:38.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.725+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f07380043c0 con 0x7f074015d4b0 2024-01-16T15:00:38.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.726+0000 7f073d600640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f071806b2c0 0x7f071806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:38.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.726+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f073808a5a0 con 0x7f074015d4b0 2024-01-16T15:00:38.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.727+0000 7f0745800640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f071806b2c0 0x7f071806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:38.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.728+0000 7f0745800640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f071806b2c0 0x7f071806d780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0730036000 tx=0x7f073003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:38.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.731+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0738055370 con 0x7f074015d4b0 2024-01-16T15:00:38.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.924+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07400d26d0 con 0x7f074015d4b0 2024-01-16T15:00:38.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.925+0000 7f073d600640 1 -- 172.21.15.53:0/3768643046 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0738058990 con 0x7f074015d4b0 2024-01-16T15:00:38.906 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:38.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f071806b2c0 msgr2=0x7f071806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f071806b2c0 0x7f071806d780 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f0730036000 tx=0x7f073003c040 comp rx=0 tx=0).stop 2024-01-16T15:00:38.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 msgr2=0x7f0740148e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f0740148e40 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f073800cea0 tx=0x7f073800cf90 comp rx=0 tx=0).stop 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 shutdown_connections 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f071806b2c0 0x7f071806d780 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f074015d4b0 0x7f0740148e40 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07400cf9a0 0x7f074009e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 --2- 172.21.15.53:0/3768643046 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07400b41e0 0x7f07400ccd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 >> 172.21.15.53:0/3768643046 conn(0x7f07400d3000 msgr2=0x7f07400cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 shutdown_connections 2024-01-16T15:00:38.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:38.928+0000 7f0746c00640 1 -- 172.21.15.53:0/3768643046 wait complete. 2024-01-16T15:00:38.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:39.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.179+0000 7f4083000640 1 Processor -- start 2024-01-16T15:00:39.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.179+0000 7f4083000640 1 -- start start 2024-01-16T15:00:39.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.181+0000 7f4083000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f407c071290 0x7f407c071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.181+0000 7f4083000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c071bb0 con 0x7f407c15f010 2024-01-16T15:00:39.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.181+0000 7f4083000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f407c15f620 con 0x7f407c15eb60 2024-01-16T15:00:39.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.181+0000 7f4083000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f407c0ccb10 con 0x7f407c071290 2024-01-16T15:00:39.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.181+0000 7f4082600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f407c15f010 0x7f407c0b2ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47184/0 (socket says 172.21.15.53:47184) 2024-01-16T15:00:39.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.181+0000 7f4082600640 1 -- 172.21.15.53:0/463679128 learned_addr learned my addr 172.21.15.53:0/463679128 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:39.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f4081c00640 1 --2- 172.21.15.53:0/463679128 >> v2:172.21.15.146:3300/0 conn(0x7f407c071290 0x7f407c071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f407be00640 1 -- 172.21.15.53:0/463679128 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984502928 0 0) 0x7f407c071bb0 con 0x7f407c15f010 2024-01-16T15:00:39.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f407be00640 1 -- 172.21.15.53:0/463679128 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f405c003520 con 0x7f407c15f010 2024-01-16T15:00:39.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f407be00640 1 -- 172.21.15.53:0/463679128 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3529731809 0 0) 0x7f405c003520 con 0x7f407c15f010 2024-01-16T15:00:39.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f4081c00640 1 -- 172.21.15.53:0/463679128 >> v1:172.21.15.146:6789/0 conn(0x7f407c15eb60 legacy=0x7f407c0cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f4081c00640 1 -- 172.21.15.53:0/463679128 >> v1:172.21.15.53:6789/0 conn(0x7f407c15f010 legacy=0x7f407c0b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.182+0000 7f4081c00640 1 -- 172.21.15.53:0/463679128 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f407c15f900 con 0x7f407c071290 2024-01-16T15:00:39.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.184+0000 7f4081c00640 1 --2- 172.21.15.53:0/463679128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c071670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f406c0097a0 tx=0x7f406c02b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4703ea1807ed4261 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.184+0000 7f407be00640 1 -- 172.21.15.53:0/463679128 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40740019f0 con 0x7f407c071290 2024-01-16T15:00:39.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.184+0000 7f407be00640 1 -- 172.21.15.53:0/463679128 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f406c02e050 con 0x7f407c071290 2024-01-16T15:00:39.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.184+0000 7f407be00640 1 -- 172.21.15.53:0/463679128 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f406c030730 con 0x7f407c071290 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 -- 172.21.15.53:0/463679128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 msgr2=0x7f407c071670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 --2- 172.21.15.53:0/463679128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c071670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f406c0097a0 tx=0x7f406c02b790 comp rx=0 tx=0).stop 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 -- 172.21.15.53:0/463679128 shutdown_connections 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 --2- 172.21.15.53:0/463679128 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c071670 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 -- 172.21.15.53:0/463679128 >> 172.21.15.53:0/463679128 conn(0x7f407c0b0ed0 msgr2=0x7f407c0cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 -- 172.21.15.53:0/463679128 shutdown_connections 2024-01-16T15:00:39.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.185+0000 7f4083000640 1 -- 172.21.15.53:0/463679128 wait complete. 2024-01-16T15:00:39.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.187+0000 7f4083000640 1 Processor -- start 2024-01-16T15:00:39.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.187+0000 7f4083000640 1 -- start start 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4083000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c1e6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4083000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 0x7f407c0c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4083000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f407c15f010 0x7f407c0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4083000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c130400 con 0x7f407c15eb60 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4083000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f407c1e6480 con 0x7f407c071290 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4083000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f407c1e6760 con 0x7f407c15f010 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4081200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 0x7f407c0c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4081200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 0x7f407c0c3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48762/0 (socket says 172.21.15.53:48762) 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4081200640 1 -- 172.21.15.53:0/2998620344 learned_addr learned my addr 172.21.15.53:0/2998620344 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.188+0000 7f4081c00640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c1e6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4082600640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f407c15f010 0x7f407c0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4081200640 1 -- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f407c15f010 msgr2=0x7f407c0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4081200640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f407c15f010 0x7f407c0ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4081200640 1 -- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 msgr2=0x7f407c1e6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4081200640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c1e6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4081200640 1 -- 172.21.15.53:0/2998620344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c009400 con 0x7f407c15eb60 2024-01-16T15:00:39.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.189+0000 7f4081200640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 0x7f407c0c3b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f406400c400 tx=0x7f406400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.190+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40640052d0 con 0x7f407c15eb60 2024-01-16T15:00:39.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.190+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4064005450 con 0x7f407c15eb60 2024-01-16T15:00:39.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.190+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4064007260 con 0x7f407c15eb60 2024-01-16T15:00:39.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.190+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f407c130650 con 0x7f407c15eb60 2024-01-16T15:00:39.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.190+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407c0c4050 con 0x7f407c15eb60 2024-01-16T15:00:39.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.191+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f407c15e950 con 0x7f407c15eb60 2024-01-16T15:00:39.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.193+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f40640073c0 con 0x7f407c15eb60 2024-01-16T15:00:39.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.194+0000 7f4079600640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f405406b310 0x7f405406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.194+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4064088fa0 con 0x7f407c15eb60 2024-01-16T15:00:39.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.194+0000 7f4081c00640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f405406b310 0x7f405406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.195+0000 7f4081c00640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f405406b310 0x7f405406d7d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f406c034000 tx=0x7f406c02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.200+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4064053e70 con 0x7f407c15eb60 2024-01-16T15:00:39.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.411+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f407c067450 con 0x7f407c15eb60 2024-01-16T15:00:39.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.412+0000 7f4079600640 1 -- 172.21.15.53:0/2998620344 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4064057490 con 0x7f407c15eb60 2024-01-16T15:00:39.393 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.414+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f405406b310 msgr2=0x7f405406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.414+0000 7f4083000640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f405406b310 0x7f405406d7d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f406c034000 tx=0x7f406c02f000 comp rx=0 tx=0).stop 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.414+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 msgr2=0x7f407c0c3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.414+0000 7f4083000640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 0x7f407c0c3b10 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f406400c400 tx=0x7f406400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 shutdown_connections 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f405406b310 0x7f405406d7d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f407c15f010 0x7f407c0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f407c15eb60 0x7f407c0c3b10 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 --2- 172.21.15.53:0/2998620344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f407c071290 0x7f407c1e6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 >> 172.21.15.53:0/2998620344 conn(0x7f407c0b0ed0 msgr2=0x7f407c0b2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:39.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 shutdown_connections 2024-01-16T15:00:39.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.415+0000 7f4083000640 1 -- 172.21.15.53:0/2998620344 wait complete. 2024-01-16T15:00:39.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:39.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.657+0000 7fe79c400640 1 Processor -- start 2024-01-16T15:00:39.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.657+0000 7fe79c400640 1 -- start start 2024-01-16T15:00:39.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.659+0000 7fe79c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe794164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.659+0000 7fe79c400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940b1ee0 con 0x7fe794143ae0 2024-01-16T15:00:39.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.659+0000 7fe79c400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7940b2780 con 0x7fe7940cf9a0 2024-01-16T15:00:39.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.659+0000 7fe79c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79415e990 con 0x7fe79415e3b0 2024-01-16T15:00:39.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.660+0000 7fe799a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe794143ae0 0x7fe7940cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47196/0 (socket says 172.21.15.53:47196) 2024-01-16T15:00:39.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.660+0000 7fe799a00640 1 -- 172.21.15.53:0/3374313649 learned_addr learned my addr 172.21.15.53:0/3374313649 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.660+0000 7fe79a400640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe794164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.660+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362744680 0 0) 0x7fe7940b1ee0 con 0x7fe794143ae0 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.660+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe774003520 con 0x7fe794143ae0 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2310701460 0 0) 0x7fe774003520 con 0x7fe794143ae0 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 msgr2=0x7fe794164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe794164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 >> v1:172.21.15.146:6789/0 conn(0x7fe7940cf9a0 legacy=0x7fe79415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940b2130 con 0x7fe794143ae0 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe784001670 con 0x7fe794143ae0 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 >> v1:172.21.15.53:6789/0 conn(0x7fe794143ae0 legacy=0x7fe7940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7740039a0 0x7fe774003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7740047c0 0x7fe774004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe774005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79415e990 con 0x7fe7740047c0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400ba90 con 0x7fe7740039a0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe77400bc50 con 0x7fe79415e3b0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.661+0000 7fe799000640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7740047c0 0x7fe774004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe799a00640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7740039a0 0x7fe774003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe79a400640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe774005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe799000640 1 -- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 msgr2=0x7fe774005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe799000640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe774005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe799000640 1 -- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7740039a0 msgr2=0x7fe774003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe799000640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7740039a0 0x7fe774003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe799000640 1 -- 172.21.15.53:0/3374313649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe774003520 con 0x7fe7740047c0 2024-01-16T15:00:39.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.662+0000 7fe79a400640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe79415e3b0 0x7fe774005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:39.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.663+0000 7fe799000640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7740047c0 0x7fe774004c40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe77c001720 tx=0x7fe7740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d2771f69cd52cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.663+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c0052d0 con 0x7fe7740047c0 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.663+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77c02e050 con 0x7fe7740047c0 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.663+0000 7fe793e00640 1 -- 172.21.15.53:0/3374313649 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c0035f0 con 0x7fe7740047c0 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79c400640 1 -- 172.21.15.53:0/3374313649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7740047c0 msgr2=0x7fe774004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79c400640 1 --2- 172.21.15.53:0/3374313649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7740047c0 0x7fe774004c40 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fe77c001720 tx=0x7fe7740057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79a400640 1 -- 172.21.15.53:0/3374313649 reap_dead start 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79c400640 1 -- 172.21.15.53:0/3374313649 shutdown_connections 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79c400640 1 -- 172.21.15.53:0/3374313649 >> 172.21.15.53:0/3374313649 conn(0x7fe79406a800 msgr2=0x7fe79406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79c400640 1 -- 172.21.15.53:0/3374313649 shutdown_connections 2024-01-16T15:00:39.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.664+0000 7fe79c400640 1 -- 172.21.15.53:0/3374313649 wait complete. 2024-01-16T15:00:39.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.666+0000 7fe79c400640 1 Processor -- start 2024-01-16T15:00:39.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.666+0000 7fe79c400640 1 -- start start 2024-01-16T15:00:39.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.667+0000 7fe79c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7740047c0 0x7fe79413b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.667+0000 7fe79c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 0x7fe7941561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.667+0000 7fe79c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe794156730 0x7fe794156bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.667+0000 7fe79c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940ad650 con 0x7fe794159750 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.667+0000 7fe79c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940ad930 con 0x7fe794156730 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.667+0000 7fe79c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7940adc10 con 0x7fe7740047c0 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 0x7fe7941561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 0x7fe7941561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48786/0 (socket says 172.21.15.53:48786) 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 -- 172.21.15.53:0/1300652308 learned_addr learned my addr 172.21.15.53:0/1300652308 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799a00640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7740047c0 0x7fe79413b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 -- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7740047c0 msgr2=0x7fe79413b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7740047c0 0x7fe79413b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 -- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe794156730 msgr2=0x7fe794156bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe79a400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe794156730 0x7fe794156bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe794156730 0x7fe794156bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.668+0000 7fe799000640 1 -- 172.21.15.53:0/1300652308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7940adef0 con 0x7fe794159750 2024-01-16T15:00:39.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.669+0000 7fe799000640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 0x7fe7941561f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe77c008fd0 tx=0x7fe77c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.669+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c0076c0 con 0x7fe794159750 2024-01-16T15:00:39.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.669+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77c007a30 con 0x7fe794159750 2024-01-16T15:00:39.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.669+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77c006c00 con 0x7fe794159750 2024-01-16T15:00:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.669+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7940ae1d0 con 0x7fe794159750 2024-01-16T15:00:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.669+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79413c620 con 0x7fe794159750 2024-01-16T15:00:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.670+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79406c3f0 con 0x7fe794159750 2024-01-16T15:00:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.671+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe77c0035f0 con 0x7fe794159750 2024-01-16T15:00:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.672+0000 7fe791600640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe76c06b2f0 0x7fe76c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.672+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe77c0abda0 con 0x7fe794159750 2024-01-16T15:00:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.673+0000 7fe799a00640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe76c06b2f0 0x7fe76c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:39.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.674+0000 7fe799a00640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe76c06b2f0 0x7fe76c06d7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe784001050 tx=0x7fe784000ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:39.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.677+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe77c07a610 con 0x7fe794159750 2024-01-16T15:00:39.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.868+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe7940675d0 con 0x7fe794159750 2024-01-16T15:00:39.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.869+0000 7fe791600640 1 -- 172.21.15.53:0/1300652308 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe77c04b0a0 con 0x7fe794159750 2024-01-16T15:00:39.851 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe76c06b2f0 msgr2=0x7fe76c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe76c06b2f0 0x7fe76c06d7b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe784001050 tx=0x7fe784000ef0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 msgr2=0x7fe7941561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 0x7fe7941561f0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fe77c008fd0 tx=0x7fe77c001120 comp rx=0 tx=0).stop 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 shutdown_connections 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe76c06b2f0 0x7fe76c06d7b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe794156730 0x7fe794156bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe794159750 0x7fe7941561f0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 --2- 172.21.15.53:0/1300652308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7740047c0 0x7fe79413b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.872+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 >> 172.21.15.53:0/1300652308 conn(0x7fe79406a800 msgr2=0x7fe7940cb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.873+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 shutdown_connections 2024-01-16T15:00:39.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:39.873+0000 7fe79c400640 1 -- 172.21.15.53:0/1300652308 wait complete. 2024-01-16T15:00:39.873 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T15:00:40.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.112+0000 7fdad3200640 1 Processor -- start 2024-01-16T15:00:40.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.112+0000 7fdad3200640 1 -- start start 2024-01-16T15:00:40.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdad40ce6a0 0x7fdad40cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415ea50 0x7fdad40cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad3200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdad40cca90 con 0x7fdad415e670 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdad40cf170 con 0x7fdad40ce6a0 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40ccd70 con 0x7fdad415ea50 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdad40ce6a0 0x7fdad40cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdad40ce6a0 0x7fdad40cf590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48800/0 (socket says 172.21.15.53:48800) 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad1e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdad415e670 0x7fdad40b3d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47210/0 (socket says 172.21.15.53:47210) 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.113+0000 7fdad1400640 1 -- 172.21.15.53:0/1521841066 learned_addr learned my addr 172.21.15.53:0/1521841066 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.114+0000 7fdad1400640 1 -- 172.21.15.53:0/1521841066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415ea50 msgr2=0x7fdad40cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:40.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.114+0000 7fdad1400640 1 --2- 172.21.15.53:0/1521841066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415ea50 0x7fdad40cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.114+0000 7fdad1400640 1 -- 172.21.15.53:0/1521841066 >> v1:172.21.15.53:6789/0 conn(0x7fdad415e670 legacy=0x7fdad40b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.114+0000 7fdad1400640 1 -- 172.21.15.53:0/1521841066 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad40ceab0 con 0x7fdad40ce6a0 2024-01-16T15:00:40.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.115+0000 7fdad1400640 1 --2- 172.21.15.53:0/1521841066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad40ce6a0 0x7fdad40cf590 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009870 tx=0x7fdabc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47d0ee6eac68c7b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:40.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.115+0000 7fdad0a00640 1 -- 172.21.15.53:0/1521841066 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc030860 con 0x7fdad40ce6a0 2024-01-16T15:00:40.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad0a00640 1 -- 172.21.15.53:0/1521841066 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdabc0309c0 con 0x7fdad40ce6a0 2024-01-16T15:00:40.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad0a00640 1 -- 172.21.15.53:0/1521841066 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc030d50 con 0x7fdad40ce6a0 2024-01-16T15:00:40.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 -- 172.21.15.53:0/1521841066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad40ce6a0 msgr2=0x7fdad40cf590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 --2- 172.21.15.53:0/1521841066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad40ce6a0 0x7fdad40cf590 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdabc009870 tx=0x7fdabc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 -- 172.21.15.53:0/1521841066 shutdown_connections 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 --2- 172.21.15.53:0/1521841066 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415ea50 0x7fdad40cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 --2- 172.21.15.53:0/1521841066 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad40ce6a0 0x7fdad40cf590 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 -- 172.21.15.53:0/1521841066 >> 172.21.15.53:0/1521841066 conn(0x7fdad40b2a30 msgr2=0x7fdad40b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 -- 172.21.15.53:0/1521841066 shutdown_connections 2024-01-16T15:00:40.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.116+0000 7fdad3200640 1 -- 172.21.15.53:0/1521841066 wait complete. 2024-01-16T15:00:40.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.118+0000 7fdad3200640 1 Processor -- start 2024-01-16T15:00:40.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.118+0000 7fdad3200640 1 -- start start 2024-01-16T15:00:40.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.119+0000 7fdad3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415e670 0x7fdad4158f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.119+0000 7fdad3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 0x7fdad40cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.119+0000 7fdad3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad40c71b0 0x7fdad40c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.119+0000 7fdad3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40c6110 con 0x7fdad415ea50 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.119+0000 7fdad3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40c63f0 con 0x7fdad40c71b0 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.119+0000 7fdad3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdad40c66d0 con 0x7fdad415e670 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 0x7fdad40cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 0x7fdad40cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48808/0 (socket says 172.21.15.53:48808) 2024-01-16T15:00:40.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 -- 172.21.15.53:0/3549298863 learned_addr learned my addr 172.21.15.53:0/3549298863 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:40.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad2800640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad40c71b0 0x7fdad40c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1e00640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415e670 0x7fdad4158f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 -- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415e670 msgr2=0x7fdad4158f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415e670 0x7fdad4158f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 -- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad40c71b0 msgr2=0x7fdad40c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad40c71b0 0x7fdad40c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 -- 172.21.15.53:0/3549298863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdabc0094d0 con 0x7fdad415ea50 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad2800640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad40c71b0 0x7fdad40c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.120+0000 7fdad1400640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 0x7fdad40cace0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdabc000c00 tx=0x7fdabc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.121+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc0307f0 con 0x7fdad415ea50 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.121+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdabc043400 con 0x7fdad415ea50 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.121+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdabc043790 con 0x7fdad415ea50 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.121+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdad40b45c0 con 0x7fdad415ea50 2024-01-16T15:00:40.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.121+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdad40c6c90 con 0x7fdad415ea50 2024-01-16T15:00:40.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.122+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdad40c6e80 con 0x7fdad415ea50 2024-01-16T15:00:40.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.123+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdabc0438f0 con 0x7fdad415ea50 2024-01-16T15:00:40.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.125+0000 7fdaca000640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdaac06b2c0 0x7fdaac06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:40.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.125+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdabc03e070 con 0x7fdad415ea50 2024-01-16T15:00:40.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.125+0000 7fdad1e00640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdaac06b2c0 0x7fdaac06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:40.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.126+0000 7fdad1e00640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdaac06b2c0 0x7fdaac06d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdac40012e0 tx=0x7fdac4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:40.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.129+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdabc077fb0 con 0x7fdad415ea50 2024-01-16T15:00:40.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:40.407+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7fdad4067b60 con 0x7fdad415ea50 2024-01-16T15:00:41.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.205+0000 7fdaca000640 1 -- 172.21.15.53:0/3549298863 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v61) v1 ==== 155+0+0 (secure 0 0 0) 0x7fdabc0340e0 con 0x7fdad415ea50 2024-01-16T15:00:41.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdaac06b2c0 msgr2=0x7fdaac06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdaac06b2c0 0x7fdaac06d780 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fdac40012e0 tx=0x7fdac4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:41.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 msgr2=0x7fdad40cace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 0x7fdad40cace0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdabc000c00 tx=0x7fdabc004730 comp rx=0 tx=0).stop 2024-01-16T15:00:41.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 shutdown_connections 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdaac06b2c0 0x7fdaac06d780 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdad40c71b0 0x7fdad40c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdad415ea50 0x7fdad40cace0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 --2- 172.21.15.53:0/3549298863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdad415e670 0x7fdad4158f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 >> 172.21.15.53:0/3549298863 conn(0x7fdad40b2a30 msgr2=0x7fdad406bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 shutdown_connections 2024-01-16T15:00:41.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.210+0000 7fdad3200640 1 -- 172.21.15.53:0/3549298863 wait complete. 2024-01-16T15:00:41.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:00:41.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.457+0000 7fc5f3e00640 1 Processor -- start 2024-01-16T15:00:41.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.457+0000 7fc5f3e00640 1 -- start start 2024-01-16T15:00:41.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5ec13df80 0x7fc5ec15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc5ec13e360 0x7fc5ec0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec06c470 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec06c750 con 0x7fc5ec13e360 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5ec06ca30 con 0x7fc5ec13df80 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f0a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc5ec13e360 0x7fc5ec0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f1e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc5ec15f9c0 0x7fc5ec1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47218/0 (socket says 172.21.15.53:47218) 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f1e00640 1 -- 172.21.15.53:0/3247612401 learned_addr learned my addr 172.21.15.53:0/3247612401 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f0a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc5ec13e360 0x7fc5ec0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48820/0 (socket says 172.21.15.53:48820) 2024-01-16T15:00:41.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.459+0000 7fc5f1400640 1 --2- 172.21.15.53:0/3247612401 >> v2:172.21.15.146:3300/0 conn(0x7fc5ec13df80 0x7fc5ec15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.460+0000 7fc5f0a00640 1 -- 172.21.15.53:0/3247612401 >> v2:172.21.15.146:3300/0 conn(0x7fc5ec13df80 msgr2=0x7fc5ec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:41.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.460+0000 7fc5f0a00640 1 --2- 172.21.15.53:0/3247612401 >> v2:172.21.15.146:3300/0 conn(0x7fc5ec13df80 0x7fc5ec15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.460+0000 7fc5f0a00640 1 -- 172.21.15.53:0/3247612401 >> v1:172.21.15.53:6789/0 conn(0x7fc5ec15f9c0 legacy=0x7fc5ec1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.460+0000 7fc5f0a00640 1 -- 172.21.15.53:0/3247612401 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec06cd10 con 0x7fc5ec13e360 2024-01-16T15:00:41.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.460+0000 7fc5ebe00640 1 -- 172.21.15.53:0/3247612401 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1204537646 0 0) 0x7fc5ec06c470 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.461+0000 7fc5f0a00640 1 --2- 172.21.15.53:0/3247612401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec13e360 0x7fc5ec0cf290 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec072770 tx=0x7fc5d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58487f301e0bb507 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.461+0000 7fc5ebe00640 1 -- 172.21.15.53:0/3247612401 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4031830 con 0x7fc5ec13e360 2024-01-16T15:00:41.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.461+0000 7fc5ebe00640 1 -- 172.21.15.53:0/3247612401 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d4031990 con 0x7fc5ec13e360 2024-01-16T15:00:41.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.461+0000 7fc5ebe00640 1 -- 172.21.15.53:0/3247612401 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4031d20 con 0x7fc5ec13e360 2024-01-16T15:00:41.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 -- 172.21.15.53:0/3247612401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec13e360 msgr2=0x7fc5ec0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/3247612401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec13e360 0x7fc5ec0cf290 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec072770 tx=0x7fc5d402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 -- 172.21.15.53:0/3247612401 shutdown_connections 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/3247612401 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec13e360 0x7fc5ec0cf290 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/3247612401 >> v2:172.21.15.146:3300/0 conn(0x7fc5ec13df80 0x7fc5ec15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 -- 172.21.15.53:0/3247612401 >> 172.21.15.53:0/3247612401 conn(0x7fc5ec06a800 msgr2=0x7fc5ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 -- 172.21.15.53:0/3247612401 shutdown_connections 2024-01-16T15:00:41.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.462+0000 7fc5f3e00640 1 -- 172.21.15.53:0/3247612401 wait complete. 2024-01-16T15:00:41.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.464+0000 7fc5f3e00640 1 Processor -- start 2024-01-16T15:00:41.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.464+0000 7fc5f3e00640 1 -- start start 2024-01-16T15:00:41.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5ec13df80 0x7fc5ec14c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5ec13e360 0x7fc5ec150fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 0x7fc5ec14d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec14c610 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec14c8f0 con 0x7fc5ec13df80 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5ec14cbd0 con 0x7fc5ec13e360 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 0x7fc5ec14d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 0x7fc5ec14d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48826/0 (socket says 172.21.15.53:48826) 2024-01-16T15:00:41.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.465+0000 7fc5f1e00640 1 -- 172.21.15.53:0/763160900 learned_addr learned my addr 172.21.15.53:0/763160900 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1400640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5ec13df80 0x7fc5ec14c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f0a00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5ec13e360 0x7fc5ec150fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1e00640 1 -- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5ec13e360 msgr2=0x7fc5ec150fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5ec13e360 0x7fc5ec150fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1e00640 1 -- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5ec13df80 msgr2=0x7fc5ec14c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5ec13df80 0x7fc5ec14c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1e00640 1 -- 172.21.15.53:0/763160900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d4009400 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1400640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5ec13df80 0x7fc5ec14c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5f1e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 0x7fc5ec14d6b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4001560 tx=0x7fc5e400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.466+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400a640 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.467+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5e400a7a0 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.467+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec14ceb0 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.467+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e400ab30 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.467+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec13dc70 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.468+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5ec14d3d0 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.469+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc5e400ac90 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.470+0000 7fc5e9600640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5c406b2c0 0x7fc5c406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:41.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.470+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc5e4089280 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.471+0000 7fc5f1400640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5c406b2c0 0x7fc5c406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:41.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.472+0000 7fc5f1400640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5c406b2c0 0x7fc5c406d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc006fd0 tx=0x7fc5dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:41.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.474+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5e4053fe0 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.677+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc5ec0675d0 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.682+0000 7fc5e9600640 1 -- 172.21.15.53:0/763160900 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc5e4018020 con 0x7fc5ec15f9c0 2024-01-16T15:00:41.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5c406b2c0 msgr2=0x7fc5c406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5c406b2c0 0x7fc5c406d780 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc006fd0 tx=0x7fc5dc008040 comp rx=0 tx=0).stop 2024-01-16T15:00:41.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 msgr2=0x7fc5ec14d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:41.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 0x7fc5ec14d6b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4001560 tx=0x7fc5e400cc70 comp rx=0 tx=0).stop 2024-01-16T15:00:41.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 shutdown_connections 2024-01-16T15:00:41.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5c406b2c0 0x7fc5c406d780 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5ec15f9c0 0x7fc5ec14d6b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5ec13e360 0x7fc5ec150fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 --2- 172.21.15.53:0/763160900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5ec13df80 0x7fc5ec14c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:41.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 >> 172.21.15.53:0/763160900 conn(0x7fc5ec06a800 msgr2=0x7fc5ec15c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:41.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 shutdown_connections 2024-01-16T15:00:41.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:41.685+0000 7fc5f3e00640 1 -- 172.21.15.53:0/763160900 wait complete. 2024-01-16T15:00:41.684 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:00:41.684 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:00:41.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:00:41.932 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:00:41.932 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:00:41.932 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:41.994 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:00:41.994 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:42.048 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:42.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.287+0000 7ff27a200640 1 Processor -- start 2024-01-16T15:00:42.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.287+0000 7ff27a200640 1 -- start start 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.288+0000 7ff27a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 0x7ff2740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff27a200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff2740cf7d0 0x7ff27415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff27a200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2740cfc30 con 0x7ff27415e1e0 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff27a200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2740b1ee0 con 0x7ff2740cf7d0 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff27a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740b2780 con 0x7ff274143ae0 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff273e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff27415e1e0 0x7ff27415a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57832/0 (socket says 172.21.15.53:57832) 2024-01-16T15:00:42.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff273e00640 1 -- 172.21.15.53:0/688806737 learned_addr learned my addr 172.21.15.53:0/688806737 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff26be00640 1 --2- 172.21.15.53:0/688806737 >> v2:172.21.15.146:3300/0 conn(0x7ff2740cf7d0 0x7ff27415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.289+0000 7ff273400640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 0x7ff2740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722717577 0 0) 0x7ff2740cfc30 con 0x7ff27415e1e0 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff254003520 con 0x7ff27415e1e0 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716711566 0 0) 0x7ff254003520 con 0x7ff27415e1e0 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 msgr2=0x7ff2740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 0x7ff2740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 >> v2:172.21.15.146:3300/0 conn(0x7ff2740cf7d0 msgr2=0x7ff27415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 --2- 172.21.15.53:0/688806737 >> v2:172.21.15.146:3300/0 conn(0x7ff2740cf7d0 0x7ff27415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27415e990 con 0x7ff27415e1e0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7ff26c0019f0 con 0x7ff27415e1e0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 >> v1:172.21.15.53:6789/0 conn(0x7ff27415e1e0 legacy=0x7ff27415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 0x7ff254003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff254003ef0 0x7ff2540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff254004d00 0x7ff254005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740b1ee0 con 0x7ff254004d00 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2740b2780 con 0x7ff254003ef0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.290+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff25400b980 con 0x7ff274143ae0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273e00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff254004d00 0x7ff254005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273400640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 0x7ff254003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff26be00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff254003ef0 0x7ff2540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273e00640 1 -- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 msgr2=0x7ff254003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273e00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274143ae0 0x7ff254003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273e00640 1 -- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff254003ef0 msgr2=0x7ff2540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273e00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff254003ef0 0x7ff2540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff273e00640 1 -- 172.21.15.53:0/688806737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff254003520 con 0x7ff254004d00 2024-01-16T15:00:42.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.291+0000 7ff26be00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff254003ef0 0x7ff2540047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:42.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.292+0000 7ff273e00640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff254004d00 0x7ff254005180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00a830 tx=0x7ff2540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b3bf95aeff1ffab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.292+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c034630 con 0x7ff254004d00 2024-01-16T15:00:42.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.292+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c02f070 con 0x7ff254004d00 2024-01-16T15:00:42.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff272a00640 1 -- 172.21.15.53:0/688806737 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c034a30 con 0x7ff254004d00 2024-01-16T15:00:42.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff27a200640 1 -- 172.21.15.53:0/688806737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff254004d00 msgr2=0x7ff254005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff27a200640 1 --2- 172.21.15.53:0/688806737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff254004d00 0x7ff254005180 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00a830 tx=0x7ff2540057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff273e00640 1 -- 172.21.15.53:0/688806737 reap_dead start 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff27a200640 1 -- 172.21.15.53:0/688806737 shutdown_connections 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff27a200640 1 -- 172.21.15.53:0/688806737 >> 172.21.15.53:0/688806737 conn(0x7ff27406a800 msgr2=0x7ff27406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff27a200640 1 -- 172.21.15.53:0/688806737 shutdown_connections 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.293+0000 7ff27a200640 1 -- 172.21.15.53:0/688806737 wait complete. 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.295+0000 7ff27a200640 1 Processor -- start 2024-01-16T15:00:42.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.295+0000 7ff27a200640 1 -- start start 2024-01-16T15:00:42.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff27a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274154d70 0x7ff27406c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff27a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff274155150 0x7ff27406c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff27a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 0x7ff274138600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff27a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2741393a0 con 0x7ff27415c640 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff27a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff27415b8c0 con 0x7ff274155150 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff27a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff27415bb60 con 0x7ff274154d70 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff273e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 0x7ff274138600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff273e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 0x7ff274138600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60742/0 (socket says 172.21.15.53:60742) 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.296+0000 7ff273e00640 1 -- 172.21.15.53:0/1471862098 learned_addr learned my addr 172.21.15.53:0/1471862098 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273400640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274154d70 0x7ff27406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273e00640 1 -- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274154d70 msgr2=0x7ff27406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273e00640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274154d70 0x7ff27406c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273e00640 1 -- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff274155150 msgr2=0x7ff27406c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273e00640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff274155150 0x7ff27406c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273e00640 1 -- 172.21.15.53:0/1471862098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff27415be40 con 0x7ff27415c640 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff273e00640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 0x7ff274138600 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00a830 tx=0x7ff2740b26f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.297+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c034840 con 0x7ff27415c640 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.298+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff26c0349a0 con 0x7ff27415c640 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.298+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff26c034d50 con 0x7ff27415c640 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.298+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2741393a0 con 0x7ff27415c640 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.298+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff274138b40 con 0x7ff27415c640 2024-01-16T15:00:42.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.299+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff27415bca0 con 0x7ff27415c640 2024-01-16T15:00:42.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.300+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff26c031030 con 0x7ff27415c640 2024-01-16T15:00:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.301+0000 7ff26b400640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff24c06b2c0 0x7ff24c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.301+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff26c0ac040 con 0x7ff27415c640 2024-01-16T15:00:42.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.302+0000 7ff273400640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff24c06b2c0 0x7ff24c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.303+0000 7ff273400640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff24c06b2c0 0x7ff24c06d780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2740b0b40 tx=0x7ff264008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.306+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff27415bca0 con 0x7ff27415c640 2024-01-16T15:00:42.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.501+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff27415bca0 con 0x7ff27415c640 2024-01-16T15:00:42.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.504+0000 7ff26b400640 1 -- 172.21.15.53:0/1471862098 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7ff27415bca0 con 0x7ff27415c640 2024-01-16T15:00:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:42.486 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff24c06b2c0 msgr2=0x7ff24c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff24c06b2c0 0x7ff24c06d780 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2740b0b40 tx=0x7ff264008040 comp rx=0 tx=0).stop 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 msgr2=0x7ff274138600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 0x7ff274138600 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7ff26c00a830 tx=0x7ff2740b26f0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 shutdown_connections 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff24c06b2c0 0x7ff24c06d780 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff27415c640 0x7ff274138600 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff274155150 0x7ff27406c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 --2- 172.21.15.53:0/1471862098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff274154d70 0x7ff27406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 >> 172.21.15.53:0/1471862098 conn(0x7ff27406a800 msgr2=0x7ff274139df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 shutdown_connections 2024-01-16T15:00:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.508+0000 7ff27a200640 1 -- 172.21.15.53:0/1471862098 wait complete. 2024-01-16T15:00:42.509 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:42.510 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:42.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.752+0000 7faa3ec00640 1 Processor -- start 2024-01-16T15:00:42.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.752+0000 7faa3ec00640 1 -- start start 2024-01-16T15:00:42.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.754+0000 7faa3ec00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa380ccb50 0x7faa38070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.754+0000 7faa3ec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa3815f4e0 con 0x7faa380adf20 2024-01-16T15:00:42.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.754+0000 7faa3ec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa38071130 con 0x7faa3815bf10 2024-01-16T15:00:42.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.754+0000 7faa3ec00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa3815f970 con 0x7faa380ccb50 2024-01-16T15:00:42.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.754+0000 7faa3d800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faa380adf20 0x7faa380cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57840/0 (socket says 172.21.15.53:57840) 2024-01-16T15:00:42.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.754+0000 7faa3d800640 1 -- 172.21.15.53:0/3987017424 learned_addr learned my addr 172.21.15.53:0/3987017424 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa3ce00640 1 --2- 172.21.15.53:0/3987017424 >> v2:172.21.15.146:3300/0 conn(0x7faa380ccb50 0x7faa38070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527127583 0 0) 0x7faa3815f4e0 con 0x7faa380adf20 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa18003520 con 0x7faa380adf20 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899393063 0 0) 0x7faa18003520 con 0x7faa380adf20 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 >> v2:172.21.15.146:3300/0 conn(0x7faa380ccb50 msgr2=0x7faa38070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 --2- 172.21.15.53:0/3987017424 >> v2:172.21.15.146:3300/0 conn(0x7faa380ccb50 0x7faa38070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 >> v1:172.21.15.146:6789/0 conn(0x7faa3815bf10 legacy=0x7faa38143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa3815ebb0 con 0x7faa380adf20 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.755+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7faa28001670 con 0x7faa380adf20 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 >> v1:172.21.15.53:6789/0 conn(0x7faa380adf20 legacy=0x7faa380cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa18003970 0x7faa18003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa180047c0 0x7faa18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa18005180 0x7faa1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa3815f970 con 0x7faa180047c0 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1800c120 con 0x7faa18003970 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1800c2e0 con 0x7faa18005180 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3ce00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa180047c0 0x7faa18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3d800640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa18003970 0x7faa18003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3e200640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa18005180 0x7faa1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3ce00640 1 -- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa18005180 msgr2=0x7faa1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3ce00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa18005180 0x7faa1800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3ce00640 1 -- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa18003970 msgr2=0x7faa18003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3ce00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa18003970 0x7faa18003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.756+0000 7faa3ce00640 1 -- 172.21.15.53:0/3987017424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa18003520 con 0x7faa180047c0 2024-01-16T15:00:42.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.757+0000 7faa3e200640 1 -- 172.21.15.53:0/3987017424 reap_dead start 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.758+0000 7faa3ce00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa180047c0 0x7faa18004c40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa20009a90 tx=0x7faa180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318dfbaffad68701 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.758+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2002c680 con 0x7faa180047c0 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.758+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa2002c7e0 con 0x7faa180047c0 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.758+0000 7faa37e00640 1 -- 172.21.15.53:0/3987017424 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa2002cb70 con 0x7faa180047c0 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.758+0000 7faa3ec00640 1 -- 172.21.15.53:0/3987017424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa180047c0 msgr2=0x7faa18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.758+0000 7faa3ec00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa180047c0 0x7faa18004c40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7faa20009a90 tx=0x7faa180057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.759+0000 7faa3ec00640 1 -- 172.21.15.53:0/3987017424 shutdown_connections 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.759+0000 7faa3ec00640 1 --2- 172.21.15.53:0/3987017424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa180047c0 0x7faa18004c40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.759+0000 7faa3ec00640 1 -- 172.21.15.53:0/3987017424 >> 172.21.15.53:0/3987017424 conn(0x7faa3806a7d0 msgr2=0x7faa3806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.759+0000 7faa3ec00640 1 -- 172.21.15.53:0/3987017424 shutdown_connections 2024-01-16T15:00:42.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.759+0000 7faa3ec00640 1 -- 172.21.15.53:0/3987017424 wait complete. 2024-01-16T15:00:42.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.761+0000 7faa3ec00640 1 Processor -- start 2024-01-16T15:00:42.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.761+0000 7faa3ec00640 1 -- start start 2024-01-16T15:00:42.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa180047c0 0x7faa380ae900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa3815a330 0x7faa380aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 0x7faa380b37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380b1f30 con 0x7faa380b3340 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa380b2210 con 0x7faa180047c0 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa380b24f0 con 0x7faa3815a330 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 0x7faa380b37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 0x7faa380b37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60754/0 (socket says 172.21.15.53:60754) 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3e200640 1 -- 172.21.15.53:0/1089106622 learned_addr learned my addr 172.21.15.53:0/1089106622 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3d800640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa180047c0 0x7faa380ae900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.762+0000 7faa3ce00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa3815a330 0x7faa380aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3e200640 1 -- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa3815a330 msgr2=0x7faa380aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3e200640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa3815a330 0x7faa380aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3e200640 1 -- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa180047c0 msgr2=0x7faa380ae900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3e200640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa180047c0 0x7faa380ae900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3e200640 1 -- 172.21.15.53:0/1089106622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa380b27d0 con 0x7faa380b3340 2024-01-16T15:00:42.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3e200640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 0x7faa380b37c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faa380b1e20 tx=0x7faa3000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30003b70 con 0x7faa380b3340 2024-01-16T15:00:42.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa30003cd0 con 0x7faa380b3340 2024-01-16T15:00:42.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa380b2ab0 con 0x7faa380b3340 2024-01-16T15:00:42.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa30005290 con 0x7faa380b3340 2024-01-16T15:00:42.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.763+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa381f6530 con 0x7faa380b3340 2024-01-16T15:00:42.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.765+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa38066eb0 con 0x7faa380b3340 2024-01-16T15:00:42.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.766+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faa300053f0 con 0x7faa380b3340 2024-01-16T15:00:42.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.767+0000 7faa35600640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa1006b2c0 0x7faa1006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:42.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.767+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faa300895b0 con 0x7faa380b3340 2024-01-16T15:00:42.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.767+0000 7faa3d800640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa1006b2c0 0x7faa1006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:42.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.769+0000 7faa3d800640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa1006b2c0 0x7faa1006d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa28001b40 tx=0x7faa28009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:42.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.772+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa30054300 con 0x7faa380b3340 2024-01-16T15:00:42.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.966+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa38067450 con 0x7faa380b3340 2024-01-16T15:00:42.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.967+0000 7faa35600640 1 -- 172.21.15.53:0/1089106622 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa30057920 con 0x7faa380b3340 2024-01-16T15:00:42.948 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.969+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa1006b2c0 msgr2=0x7faa1006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa1006b2c0 0x7faa1006d780 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7faa28001b40 tx=0x7faa28009040 comp rx=0 tx=0).stop 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 msgr2=0x7faa380b37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 0x7faa380b37c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7faa380b1e20 tx=0x7faa3000dc00 comp rx=0 tx=0).stop 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 shutdown_connections 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa1006b2c0 0x7faa1006d780 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa380b3340 0x7faa380b37c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa3815a330 0x7faa380aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 --2- 172.21.15.53:0/1089106622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa180047c0 0x7faa380ae900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 >> 172.21.15.53:0/1089106622 conn(0x7faa3806a7d0 msgr2=0x7faa3815db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 shutdown_connections 2024-01-16T15:00:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:42.970+0000 7faa3ec00640 1 -- 172.21.15.53:0/1089106622 wait complete. 2024-01-16T15:00:42.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:43.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.216+0000 7f5d53a00640 1 Processor -- start 2024-01-16T15:00:43.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.216+0000 7f5d53a00640 1 -- start start 2024-01-16T15:00:43.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.218+0000 7f5d53a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 0x7f5d4c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.218+0000 7f5d53a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c070c40 con 0x7f5d4c0cf290 2024-01-16T15:00:43.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.218+0000 7f5d53a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4c06c470 con 0x7f5d4c0ae5a0 2024-01-16T15:00:43.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.218+0000 7f5d53a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c071450 con 0x7f5d4c0b3ae0 2024-01-16T15:00:43.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d51a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5d4c0cf290 0x7f5d4c0b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57850/0 (socket says 172.21.15.53:57850) 2024-01-16T15:00:43.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d51a00640 1 -- 172.21.15.53:0/4183494189 learned_addr learned my addr 172.21.15.53:0/4183494189 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:43.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4be00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 0x7f5d4c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867785992 0 0) 0x7f5d4c070c40 con 0x7f5d4c0cf290 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c003520 con 0x7f5d4c0cf290 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 237028300 0 0) 0x7f5d2c003520 con 0x7f5d4c0cf290 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 msgr2=0x7f5d4c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 0x7f5d4c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 >> v1:172.21.15.146:6789/0 conn(0x7f5d4c0ae5a0 legacy=0x7f5d4c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c06c750 con 0x7f5d4c0cf290 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.219+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5d440019f0 con 0x7f5d4c0cf290 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 >> v1:172.21.15.53:6789/0 conn(0x7f5d4c0cf290 legacy=0x7f5d4c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 0x7f5d2c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d2c003f20 0x7f5d2c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c004d00 0x7f5d2c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c071450 con 0x7f5d2c004d00 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c00baf0 con 0x7f5d2c003f20 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d2c00bcb0 con 0x7f5d4c0b3ae0 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d51a00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c004d00 0x7f5d2c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d4be00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d2c003f20 0x7f5d2c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d51a00640 1 -- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 msgr2=0x7f5d2c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d51a00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d4c0b3ae0 0x7f5d2c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d51a00640 1 -- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d2c003f20 msgr2=0x7f5d2c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d51a00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d2c003f20 0x7f5d2c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.220+0000 7f5d51a00640 1 -- 172.21.15.53:0/4183494189 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c003520 con 0x7f5d2c004d00 2024-01-16T15:00:43.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.221+0000 7f5d51a00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c004d00 0x7f5d2c005180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400a800 tx=0x7f5d2c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3147b1b7eeab3d09 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44034630 con 0x7f5d2c004d00 2024-01-16T15:00:43.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4402f070 con 0x7f5d2c004d00 2024-01-16T15:00:43.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d4b400640 1 -- 172.21.15.53:0/4183494189 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d44034a90 con 0x7f5d2c004d00 2024-01-16T15:00:43.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d53a00640 1 -- 172.21.15.53:0/4183494189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c004d00 msgr2=0x7f5d2c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d53a00640 1 --2- 172.21.15.53:0/4183494189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d2c004d00 0x7f5d2c005180 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400a800 tx=0x7f5d2c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d51a00640 1 -- 172.21.15.53:0/4183494189 reap_dead start 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d53a00640 1 -- 172.21.15.53:0/4183494189 shutdown_connections 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.222+0000 7f5d53a00640 1 -- 172.21.15.53:0/4183494189 >> 172.21.15.53:0/4183494189 conn(0x7f5d4c06a800 msgr2=0x7f5d4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.223+0000 7f5d53a00640 1 -- 172.21.15.53:0/4183494189 shutdown_connections 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.223+0000 7f5d53a00640 1 -- 172.21.15.53:0/4183494189 wait complete. 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.224+0000 7f5d53a00640 1 Processor -- start 2024-01-16T15:00:43.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.225+0000 7f5d53a00640 1 -- start start 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d53a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d2c004d00 0x7f5d4c14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d53a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 0x7f5d4c158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d53a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d4c0ccf00 0x7f5d4c0b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d53a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c14ce00 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d53a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c159060 con 0x7f5d4c0ccf00 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d53a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4c0cbe60 con 0x7f5d2c004d00 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 0x7f5d4c158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 0x7f5d4c158b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60774/0 (socket says 172.21.15.53:60774) 2024-01-16T15:00:43.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 -- 172.21.15.53:0/3249422725 learned_addr learned my addr 172.21.15.53:0/3249422725 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 -- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d2c004d00 msgr2=0x7f5d4c14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d2c004d00 0x7f5d4c14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 -- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d4c0ccf00 msgr2=0x7f5d4c0b2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d4c0ccf00 0x7f5d4c0b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.226+0000 7f5d4be00640 1 -- 172.21.15.53:0/3249422725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c0cc140 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.227+0000 7f5d4be00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 0x7f5d4c158b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d340116a0 tx=0x7f5d34011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.227+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d3400f9c0 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.227+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d3400fb20 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.227+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d4c0cc420 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.227+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d34015460 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.227+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4c0cc9e0 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.228+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d4c0ccba0 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.229+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5d340155c0 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.230+0000 7f5d48c00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d2406b2f0 0x7f5d2406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.230+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5d3408a590 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.231+0000 7f5d51000640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d2406b2f0 0x7f5d2406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.232+0000 7f5d51000640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d2406b2f0 0x7f5d2406d7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c001590 tx=0x7f5d3c007990 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.235+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d34055300 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.433+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5d4c0675d0 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.433+0000 7f5d48c00640 1 -- 172.21.15.53:0/3249422725 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5d34058920 con 0x7f5d4c0cb8a0 2024-01-16T15:00:43.415 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:43.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d2406b2f0 msgr2=0x7f5d2406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d2406b2f0 0x7f5d2406d7b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5d3c001590 tx=0x7f5d3c007990 comp rx=0 tx=0).stop 2024-01-16T15:00:43.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 msgr2=0x7f5d4c158b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 0x7f5d4c158b20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5d340116a0 tx=0x7f5d34011b60 comp rx=0 tx=0).stop 2024-01-16T15:00:43.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 shutdown_connections 2024-01-16T15:00:43.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d2406b2f0 0x7f5d2406d7b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d4c0ccf00 0x7f5d4c0b2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d4c0cb8a0 0x7f5d4c158b20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 --2- 172.21.15.53:0/3249422725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d2c004d00 0x7f5d4c14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 >> 172.21.15.53:0/3249422725 conn(0x7f5d4c06a800 msgr2=0x7f5d4c0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 shutdown_connections 2024-01-16T15:00:43.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.436+0000 7f5d53a00640 1 -- 172.21.15.53:0/3249422725 wait complete. 2024-01-16T15:00:43.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:43.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.682+0000 7f25e7e00640 1 Processor -- start 2024-01-16T15:00:43.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.682+0000 7f25e7e00640 1 -- start start 2024-01-16T15:00:43.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.683+0000 7f25e7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25e815d4b0 0x7f25e80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.683+0000 7f25e7e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e80b2730 con 0x7f25e80cf9a0 2024-01-16T15:00:43.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.683+0000 7f25e7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25e80d1ec0 con 0x7f25e80b41e0 2024-01-16T15:00:43.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.683+0000 7f25e7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25e80b29d0 con 0x7f25e815d4b0 2024-01-16T15:00:43.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.684+0000 7f25e6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25e815d4b0 0x7f25e80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.684+0000 7f25e6000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f25e80cf9a0 0x7f25e80cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57864/0 (socket says 172.21.15.53:57864) 2024-01-16T15:00:43.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.684+0000 7f25e6000640 1 -- 172.21.15.53:0/1019533691 learned_addr learned my addr 172.21.15.53:0/1019533691 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:43.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.684+0000 7f25e6a00640 1 -- 172.21.15.53:0/1019533691 >> v1:172.21.15.146:6789/0 conn(0x7f25e80b41e0 legacy=0x7f25e80ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:43.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.684+0000 7f25e6a00640 1 -- 172.21.15.53:0/1019533691 >> v1:172.21.15.53:6789/0 conn(0x7f25e80cf9a0 legacy=0x7f25e80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.684+0000 7f25e6a00640 1 -- 172.21.15.53:0/1019533691 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e80b2cb0 con 0x7f25e815d4b0 2024-01-16T15:00:43.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.686+0000 7f25e6a00640 1 --2- 172.21.15.53:0/1019533691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80d1550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097a0 tx=0x7f25d802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4dc4bdba2beddc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.686+0000 7f25e5600640 1 -- 172.21.15.53:0/1019533691 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8031830 con 0x7f25e815d4b0 2024-01-16T15:00:43.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.686+0000 7f25e5600640 1 -- 172.21.15.53:0/1019533691 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25d8031990 con 0x7f25e815d4b0 2024-01-16T15:00:43.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.686+0000 7f25e5600640 1 -- 172.21.15.53:0/1019533691 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25d8031d20 con 0x7f25e815d4b0 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 -- 172.21.15.53:0/1019533691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 msgr2=0x7f25e80d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 --2- 172.21.15.53:0/1019533691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80d1550 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f25d80097a0 tx=0x7f25d802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 -- 172.21.15.53:0/1019533691 shutdown_connections 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 --2- 172.21.15.53:0/1019533691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80d1550 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 -- 172.21.15.53:0/1019533691 >> 172.21.15.53:0/1019533691 conn(0x7f25e80d3000 msgr2=0x7f25e80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 -- 172.21.15.53:0/1019533691 shutdown_connections 2024-01-16T15:00:43.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.687+0000 7f25e7e00640 1 -- 172.21.15.53:0/1019533691 wait complete. 2024-01-16T15:00:43.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.689+0000 7f25e7e00640 1 Processor -- start 2024-01-16T15:00:43.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.689+0000 7f25e7e00640 1 -- start start 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25e80b41e0 0x7f25e809f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e80cf9a0 0x7f25e809cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80b7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e80a0ce0 con 0x7f25e815d4b0 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e815ae30 con 0x7f25e80cf9a0 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25e809d4f0 con 0x7f25e80b41e0 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80b7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80b7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60786/0 (socket says 172.21.15.53:60786) 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.690+0000 7f25e7400640 1 -- 172.21.15.53:0/3742348069 learned_addr learned my addr 172.21.15.53:0/3742348069 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e6000640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e80cf9a0 0x7f25e809cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e6a00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25e80b41e0 0x7f25e809f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e7400640 1 -- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25e80b41e0 msgr2=0x7f25e809f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e7400640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25e80b41e0 0x7f25e809f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e7400640 1 -- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e80cf9a0 msgr2=0x7f25e809cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e7400640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e80cf9a0 0x7f25e809cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e7400640 1 -- 172.21.15.53:0/3742348069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d8009400 con 0x7f25e815d4b0 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e6000640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e80cf9a0 0x7f25e809cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:43.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25e7400640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80b7ea0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c700 tx=0x7f25e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000aa50 con 0x7f25e815d4b0 2024-01-16T15:00:43.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.691+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25e000abb0 con 0x7f25e815d4b0 2024-01-16T15:00:43.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.692+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25e000ce10 con 0x7f25e815d4b0 2024-01-16T15:00:43.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.692+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e815da10 con 0x7f25e815d4b0 2024-01-16T15:00:43.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.692+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e80a0620 con 0x7f25e815d4b0 2024-01-16T15:00:43.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.693+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e8067450 con 0x7f25e815d4b0 2024-01-16T15:00:43.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.694+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f25e0012420 con 0x7f25e815d4b0 2024-01-16T15:00:43.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.695+0000 7f25dea00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25c006b2c0 0x7f25c006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:43.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.695+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f25e008a110 con 0x7f25e815d4b0 2024-01-16T15:00:43.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.696+0000 7f25e6a00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25c006b2c0 0x7f25c006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:43.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.697+0000 7f25e6a00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25c006b2c0 0x7f25c006d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f25d8036000 tx=0x7f25d803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:43.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.700+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25e0054ef0 con 0x7f25e815d4b0 2024-01-16T15:00:43.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.894+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25e806b690 con 0x7f25e815d4b0 2024-01-16T15:00:43.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.895+0000 7f25dea00640 1 -- 172.21.15.53:0/3742348069 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f25e00160a0 con 0x7f25e815d4b0 2024-01-16T15:00:43.876 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:43.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25c006b2c0 msgr2=0x7f25c006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25c006b2c0 0x7f25c006d780 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f25d8036000 tx=0x7f25d803c040 comp rx=0 tx=0).stop 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 msgr2=0x7f25e80b7ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80b7ea0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f25e000c700 tx=0x7f25e000cbc0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 shutdown_connections 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25c006b2c0 0x7f25c006d780 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25e815d4b0 0x7f25e80b7ea0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25e80cf9a0 0x7f25e809cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 --2- 172.21.15.53:0/3742348069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25e80b41e0 0x7f25e809f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 >> 172.21.15.53:0/3742348069 conn(0x7f25e80d3000 msgr2=0x7f25e80d33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 shutdown_connections 2024-01-16T15:00:43.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:43.898+0000 7f25e7e00640 1 -- 172.21.15.53:0/3742348069 wait complete. 2024-01-16T15:00:43.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:44.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.138+0000 7f8f85c00640 1 Processor -- start 2024-01-16T15:00:44.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.139+0000 7f8f85c00640 1 -- start start 2024-01-16T15:00:44.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.140+0000 7f8f85c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f80146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.140+0000 7f8f85c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f8006c7c0 0x7f8f800717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.140+0000 7f8f85c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8006cc20 con 0x7f8f80071d30 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.140+0000 7f8f85c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8006cec0 con 0x7f8f8006c7c0 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.140+0000 7f8f85c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8006d1a0 con 0x7f8f80071410 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f8006c7c0 0x7f8f800717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f8006c7c0 0x7f8f800717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60798/0 (socket says 172.21.15.53:60798) 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 -- 172.21.15.53:0/2583178509 learned_addr learned my addr 172.21.15.53:0/2583178509 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f7fe00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f80146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 -- 172.21.15.53:0/2583178509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 msgr2=0x7f8f80146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f80146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 -- 172.21.15.53:0/2583178509 >> v1:172.21.15.53:6789/0 conn(0x7f8f80071d30 legacy=0x7f8f800b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f77e00640 1 -- 172.21.15.53:0/2583178509 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8006d480 con 0x7f8f8006c7c0 2024-01-16T15:00:44.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.141+0000 7f8f7fe00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f80146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:44.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.143+0000 7f8f77e00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f800717f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009870 tx=0x7f8f6802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f5d4a6fce14159 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.143+0000 7f8f7f400640 1 -- 172.21.15.53:0/2583178509 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68030840 con 0x7f8f8006c7c0 2024-01-16T15:00:44.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.143+0000 7f8f7f400640 1 -- 172.21.15.53:0/2583178509 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f680309a0 con 0x7f8f8006c7c0 2024-01-16T15:00:44.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.143+0000 7f8f7f400640 1 -- 172.21.15.53:0/2583178509 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f68030d30 con 0x7f8f8006c7c0 2024-01-16T15:00:44.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.143+0000 7f8f85c00640 1 -- 172.21.15.53:0/2583178509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 msgr2=0x7f8f800717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.143+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f800717f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f68009870 tx=0x7f8f6802b980 comp rx=0 tx=0).stop 2024-01-16T15:00:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.144+0000 7f8f85c00640 1 -- 172.21.15.53:0/2583178509 shutdown_connections 2024-01-16T15:00:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.144+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f800717f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.144+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2583178509 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f80146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.144+0000 7f8f85c00640 1 -- 172.21.15.53:0/2583178509 >> 172.21.15.53:0/2583178509 conn(0x7f8f8006a800 msgr2=0x7f8f8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.144+0000 7f8f85c00640 1 -- 172.21.15.53:0/2583178509 shutdown_connections 2024-01-16T15:00:44.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.144+0000 7f8f85c00640 1 -- 172.21.15.53:0/2583178509 wait complete. 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.145+0000 7f8f85c00640 1 Processor -- start 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.146+0000 7f8f85c00640 1 -- start start 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f8013f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f801e35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f80071d30 0x7f8f801e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f801e3140 con 0x7f8f8006c7c0 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800cdbc0 con 0x7f8f80071d30 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f801e2530 con 0x7f8f80071410 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f8013f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f8013f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60810/0 (socket says 172.21.15.53:60810) 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 -- 172.21.15.53:0/2409426613 learned_addr learned my addr 172.21.15.53:0/2409426613 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:44.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f77e00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f801e35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 -- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 msgr2=0x7f8f801e35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f801e35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85200640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f80071d30 0x7f8f801e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 -- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f80071d30 msgr2=0x7f8f801e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f80071d30 0x7f8f801e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f7fe00640 1 -- 172.21.15.53:0/2409426613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f680094d0 con 0x7f8f8006c7c0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f77e00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f801e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.147+0000 7f8f85200640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f80071d30 0x7f8f801e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.148+0000 7f8f7fe00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f8013f3f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c7f0 tx=0x7f8f7000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.148+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70007a30 con 0x7f8f8006c7c0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.148+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f70007b90 con 0x7f8f8006c7c0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.148+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f801e2810 con 0x7f8f8006c7c0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.148+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7000ce40 con 0x7f8f8006c7c0 2024-01-16T15:00:44.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.148+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f801e2dd0 con 0x7f8f8006c7c0 2024-01-16T15:00:44.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.150+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f80072150 con 0x7f8f8006c7c0 2024-01-16T15:00:44.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.151+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8f700113f0 con 0x7f8f8006c7c0 2024-01-16T15:00:44.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.152+0000 7f8f7cc00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f5806b2c0 0x7f8f5806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.152+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8f70089e40 con 0x7f8f8006c7c0 2024-01-16T15:00:44.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.152+0000 7f8f77e00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f5806b2c0 0x7f8f5806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.153+0000 7f8f77e00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f5806b2c0 0x7f8f5806d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f68036000 tx=0x7f8f6803b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.156+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f70054c30 con 0x7f8f8006c7c0 2024-01-16T15:00:44.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.365+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8f80067450 con 0x7f8f8006c7c0 2024-01-16T15:00:44.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.366+0000 7f8f7cc00640 1 -- 172.21.15.53:0/2409426613 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f8f7001f080 con 0x7f8f8006c7c0 2024-01-16T15:00:44.347 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:44.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f5806b2c0 msgr2=0x7f8f5806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f5806b2c0 0x7f8f5806d780 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f8f68036000 tx=0x7f8f6803b040 comp rx=0 tx=0).stop 2024-01-16T15:00:44.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 msgr2=0x7f8f8013f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f8013f3f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f7000c7f0 tx=0x7f8f7000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 shutdown_connections 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f5806b2c0 0x7f8f5806d780 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f80071d30 0x7f8f801e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f80071410 0x7f8f801e35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 --2- 172.21.15.53:0/2409426613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8006c7c0 0x7f8f8013f3f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 >> 172.21.15.53:0/2409426613 conn(0x7f8f8006a800 msgr2=0x7f8f80140290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 shutdown_connections 2024-01-16T15:00:44.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.369+0000 7f8f85c00640 1 -- 172.21.15.53:0/2409426613 wait complete. 2024-01-16T15:00:44.369 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:00:44.369 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:44.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:44.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.701+0000 7f0a57e00640 1 Processor -- start 2024-01-16T15:00:44.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.702+0000 7f0a57e00640 1 -- start start 2024-01-16T15:00:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.703+0000 7f0a57e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a58160920 0x7f0a580b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.703+0000 7f0a57e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a580d0a10 con 0x7f0a580cf9f0 2024-01-16T15:00:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.703+0000 7f0a57e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a58161e30 con 0x7f0a580cfea0 2024-01-16T15:00:44.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.703+0000 7f0a57e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a580d1f50 con 0x7f0a58160920 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a56000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0a580cf9f0 0x7f0a58161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57882/0 (socket says 172.21.15.53:57882) 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a56000640 1 -- 172.21.15.53:0/1702193802 learned_addr learned my addr 172.21.15.53:0/1702193802 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438971952 0 0) 0x7f0a580d0a10 con 0x7f0a580cf9f0 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a56a00640 1 --2- 172.21.15.53:0/1702193802 >> v2:172.21.15.146:3300/0 conn(0x7f0a58160920 0x7f0a580b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a38003520 con 0x7f0a580cf9f0 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950334319 0 0) 0x7f0a38003520 con 0x7f0a580cf9f0 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 >> v2:172.21.15.146:3300/0 conn(0x7f0a58160920 msgr2=0x7f0a580b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 --2- 172.21.15.53:0/1702193802 >> v2:172.21.15.146:3300/0 conn(0x7f0a58160920 0x7f0a580b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 >> v1:172.21.15.146:6789/0 conn(0x7f0a580cfea0 legacy=0x7f0a58161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a580cf080 con 0x7f0a580cf9f0 2024-01-16T15:00:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0a40001670 con 0x7f0a580cf9f0 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.704+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 >> v1:172.21.15.53:6789/0 conn(0x7f0a580cf9f0 legacy=0x7f0a58161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a55600640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a38003970 0x7f0a38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a55600640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a380047c0 0x7f0a38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a55600640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a38005180 0x7f0a3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a580d1f50 con 0x7f0a380047c0 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3800c020 con 0x7f0a38005180 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3800c1e0 con 0x7f0a38003970 2024-01-16T15:00:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a56000640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a380047c0 0x7f0a38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a57400640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a38005180 0x7f0a3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a56000640 1 -- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a38003970 msgr2=0x7f0a38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a56000640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a38003970 0x7f0a38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a56000640 1 -- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a38005180 msgr2=0x7f0a3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a56000640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a38005180 0x7f0a3800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a56000640 1 -- 172.21.15.53:0/1702193802 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a38003520 con 0x7f0a380047c0 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a57400640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a38005180 0x7f0a3800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.705+0000 7f0a57400640 1 -- 172.21.15.53:0/1702193802 reap_dead start 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.706+0000 7f0a56000640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a380047c0 0x7f0a38004c40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000c240 tx=0x7f0a380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da02dfc5fd5a7159 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.707+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40005ad0 con 0x7f0a380047c0 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.707+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a40005c30 con 0x7f0a380047c0 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.707+0000 7f0a55600640 1 -- 172.21.15.53:0/1702193802 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4002caf0 con 0x7f0a380047c0 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.707+0000 7f0a57e00640 1 -- 172.21.15.53:0/1702193802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a380047c0 msgr2=0x7f0a38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.707+0000 7f0a57e00640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a380047c0 0x7f0a38004c40 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0a4000c240 tx=0x7f0a380057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.708+0000 7f0a57e00640 1 -- 172.21.15.53:0/1702193802 shutdown_connections 2024-01-16T15:00:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.708+0000 7f0a57e00640 1 --2- 172.21.15.53:0/1702193802 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a380047c0 0x7f0a38004c40 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.708+0000 7f0a57e00640 1 -- 172.21.15.53:0/1702193802 >> 172.21.15.53:0/1702193802 conn(0x7f0a580d3090 msgr2=0x7f0a580d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.708+0000 7f0a57e00640 1 -- 172.21.15.53:0/1702193802 shutdown_connections 2024-01-16T15:00:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.708+0000 7f0a57e00640 1 -- 172.21.15.53:0/1702193802 wait complete. 2024-01-16T15:00:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.709+0000 7f0a57e00640 1 Processor -- start 2024-01-16T15:00:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.710+0000 7f0a57e00640 1 -- start start 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.710+0000 7f0a57e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a380047c0 0x7f0a581538c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a57e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 0x7f0a58155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a57e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cbfd0 0x7f0a580ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a57e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a580b6720 con 0x7f0a58154010 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a57e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58153e00 con 0x7f0a580cbfd0 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a57e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a580b5d80 con 0x7f0a380047c0 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 0x7f0a58155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 0x7f0a58155440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60826/0 (socket says 172.21.15.53:60826) 2024-01-16T15:00:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 -- 172.21.15.53:0/223305226 learned_addr learned my addr 172.21.15.53:0/223305226 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56a00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a380047c0 0x7f0a581538c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 -- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a380047c0 msgr2=0x7f0a581538c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a380047c0 0x7f0a581538c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 -- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cbfd0 msgr2=0x7f0a580ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cbfd0 0x7f0a580ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56000640 1 -- 172.21.15.53:0/223305226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a580b6060 con 0x7f0a58154010 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.711+0000 7f0a56a00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a380047c0 0x7f0a581538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.712+0000 7f0a56000640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 0x7f0a58155440 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a40001930 tx=0x7f0a4002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.712+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a4003c040 con 0x7f0a58154010 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.712+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a4002cd80 con 0x7f0a58154010 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.712+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a40007290 con 0x7f0a58154010 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.712+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a580b6340 con 0x7f0a58154010 2024-01-16T15:00:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.712+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a58155980 con 0x7f0a58154010 2024-01-16T15:00:44.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.713+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a580b5ec0 con 0x7f0a58154010 2024-01-16T15:00:44.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.714+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0a40032070 con 0x7f0a58154010 2024-01-16T15:00:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.715+0000 7f0a4ea00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.715+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0a400ab190 con 0x7f0a58154010 2024-01-16T15:00:44.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.716+0000 7f0a56a00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.717+0000 7f0a56a00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0a58067b00 tx=0x7f0a48008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:44.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.720+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a580b5ec0 con 0x7f0a58154010 2024-01-16T15:00:44.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.915+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a580b5ec0 con 0x7f0a58154010 2024-01-16T15:00:44.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.919+0000 7f0a4ea00640 1 -- 172.21.15.53:0/223305226 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f0a580b5ec0 con 0x7f0a58154010 2024-01-16T15:00:44.900 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:44.900 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 msgr2=0x7f0a3006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0a58067b00 tx=0x7f0a48008040 comp rx=0 tx=0).stop 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 msgr2=0x7f0a58155440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 0x7f0a58155440 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0a40001930 tx=0x7f0a4002e070 comp rx=0 tx=0).stop 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 shutdown_connections 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cbfd0 0x7f0a580ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58154010 0x7f0a58155440 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 --2- 172.21.15.53:0/223305226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a380047c0 0x7f0a581538c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.921+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 >> 172.21.15.53:0/223305226 conn(0x7f0a580d3090 msgr2=0x7f0a58160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:44.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.922+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 shutdown_connections 2024-01-16T15:00:44.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:44.922+0000 7f0a57e00640 1 -- 172.21.15.53:0/223305226 wait complete. 2024-01-16T15:00:44.923 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:44.923 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:45.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.155+0000 7f331f200640 1 Processor -- start 2024-01-16T15:00:45.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.155+0000 7f331f200640 1 -- start start 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.156+0000 7f331f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f332015d4b0 0x7f33200d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331f200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33200b41e0 0x7f33200cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33200d0320 con 0x7f33200cf9a0 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331f200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33200b2730 con 0x7f33200b41e0 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33200d1ec0 con 0x7f332015d4b0 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331e800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33200b41e0 0x7f33200cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331d400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f33200cf9a0 0x7f33200cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57894/0 (socket says 172.21.15.53:57894) 2024-01-16T15:00:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331d400640 1 -- 172.21.15.53:0/1261426455 learned_addr learned my addr 172.21.15.53:0/1261426455 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:45.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.157+0000 7f331de00640 1 --2- 172.21.15.53:0/1261426455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f332015d4b0 0x7f33200d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.158+0000 7f331e800640 1 -- 172.21.15.53:0/1261426455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f332015d4b0 msgr2=0x7f33200d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.158+0000 7f331e800640 1 --2- 172.21.15.53:0/1261426455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f332015d4b0 0x7f33200d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.158+0000 7f331e800640 1 -- 172.21.15.53:0/1261426455 >> v1:172.21.15.53:6789/0 conn(0x7f33200cf9a0 legacy=0x7f33200cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.158+0000 7f331e800640 1 -- 172.21.15.53:0/1261426455 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33200b29d0 con 0x7f33200b41e0 2024-01-16T15:00:45.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.159+0000 7f331e800640 1 --2- 172.21.15.53:0/1261426455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200b41e0 0x7f33200cfde0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33180097d0 tx=0x7f331802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902467f9cbbd5baa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.159+0000 7f331ca00640 1 -- 172.21.15.53:0/1261426455 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331802d040 con 0x7f33200b41e0 2024-01-16T15:00:45.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.159+0000 7f331ca00640 1 -- 172.21.15.53:0/1261426455 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3318031810 con 0x7f33200b41e0 2024-01-16T15:00:45.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.159+0000 7f331ca00640 1 -- 172.21.15.53:0/1261426455 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3318031ba0 con 0x7f33200b41e0 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 -- 172.21.15.53:0/1261426455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200b41e0 msgr2=0x7f33200cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 --2- 172.21.15.53:0/1261426455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200b41e0 0x7f33200cfde0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f33180097d0 tx=0x7f331802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 -- 172.21.15.53:0/1261426455 shutdown_connections 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 --2- 172.21.15.53:0/1261426455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200b41e0 0x7f33200cfde0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 --2- 172.21.15.53:0/1261426455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f332015d4b0 0x7f33200d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 -- 172.21.15.53:0/1261426455 >> 172.21.15.53:0/1261426455 conn(0x7f33200d3000 msgr2=0x7f33200d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 -- 172.21.15.53:0/1261426455 shutdown_connections 2024-01-16T15:00:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.160+0000 7f331f200640 1 -- 172.21.15.53:0/1261426455 wait complete. 2024-01-16T15:00:45.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.162+0000 7f331f200640 1 Processor -- start 2024-01-16T15:00:45.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.162+0000 7f331f200640 1 -- start start 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33200b41e0 0x7f33200cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 0x7f33200b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f332015d4b0 0x7f33200b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200ca270 con 0x7f33200cf9a0 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33200ca550 con 0x7f332015d4b0 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33200ca830 con 0x7f33200b41e0 2024-01-16T15:00:45.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 0x7f33200b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 0x7f33200b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60848/0 (socket says 172.21.15.53:60848) 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331d400640 1 -- 172.21.15.53:0/1509685106 learned_addr learned my addr 172.21.15.53:0/1509685106 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331e800640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f332015d4b0 0x7f33200b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331de00640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33200b41e0 0x7f33200cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331d400640 1 -- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33200b41e0 msgr2=0x7f33200cb310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331d400640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33200b41e0 0x7f33200cb310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.164+0000 7f331d400640 1 -- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f332015d4b0 msgr2=0x7f33200b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f331d400640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f332015d4b0 0x7f33200b6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f331d400640 1 -- 172.21.15.53:0/1509685106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3318009400 con 0x7f33200cf9a0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f331de00640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33200b41e0 0x7f33200cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f331d400640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 0x7f33200b6630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f330800cba0 tx=0x7f330800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330800ad40 con 0x7f33200cf9a0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308012400 con 0x7f33200cf9a0 2024-01-16T15:00:45.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f330800aea0 con 0x7f33200cf9a0 2024-01-16T15:00:45.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332015da10 con 0x7f33200cf9a0 2024-01-16T15:00:45.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.165+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33200cadf0 con 0x7f33200cf9a0 2024-01-16T15:00:45.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.167+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33200cafe0 con 0x7f33200cf9a0 2024-01-16T15:00:45.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.168+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3308007070 con 0x7f33200cf9a0 2024-01-16T15:00:45.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.169+0000 7f3316000640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32f806b2c0 0x7f32f806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.169+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3308089bd0 con 0x7f33200cf9a0 2024-01-16T15:00:45.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.170+0000 7f331de00640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32f806b2c0 0x7f32f806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.171+0000 7f331de00640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32f806b2c0 0x7f32f806d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3310006fd0 tx=0x7f3310008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.173+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33080549d0 con 0x7f33200cf9a0 2024-01-16T15:00:45.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.367+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3320067450 con 0x7f33200cf9a0 2024-01-16T15:00:45.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.368+0000 7f3316000640 1 -- 172.21.15.53:0/1509685106 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3308017020 con 0x7f33200cf9a0 2024-01-16T15:00:45.352 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:45.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32f806b2c0 msgr2=0x7f32f806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32f806b2c0 0x7f32f806d780 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3310006fd0 tx=0x7f3310008040 comp rx=0 tx=0).stop 2024-01-16T15:00:45.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 msgr2=0x7f33200b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 0x7f33200b6630 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f330800cba0 tx=0x7f330800a430 comp rx=0 tx=0).stop 2024-01-16T15:00:45.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 shutdown_connections 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32f806b2c0 0x7f32f806d780 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f332015d4b0 0x7f33200b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33200cf9a0 0x7f33200b6630 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 --2- 172.21.15.53:0/1509685106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33200b41e0 0x7f33200cb310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 >> 172.21.15.53:0/1509685106 conn(0x7f33200d3000 msgr2=0x7f33200b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.370+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 shutdown_connections 2024-01-16T15:00:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.371+0000 7f331f200640 1 -- 172.21.15.53:0/1509685106 wait complete. 2024-01-16T15:00:45.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:45.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.617+0000 7f506fe00640 1 Processor -- start 2024-01-16T15:00:45.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.618+0000 7f506fe00640 1 -- start start 2024-01-16T15:00:45.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.619+0000 7f506fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f507015ae40 0x7f5070161610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.619+0000 7f506fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5070071f80 0x7f507015d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.619+0000 7f506fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f507015e690 con 0x7f507015dc30 2024-01-16T15:00:45.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.619+0000 7f506fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507015e0d0 con 0x7f507015ae40 2024-01-16T15:00:45.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.619+0000 7f506fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507015e3b0 con 0x7f5070071f80 2024-01-16T15:00:45.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f507015ae40 0x7f5070161610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f507015ae40 0x7f5070161610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60856/0 (socket says 172.21.15.53:60856) 2024-01-16T15:00:45.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 -- 172.21.15.53:0/1661785575 learned_addr learned my addr 172.21.15.53:0/1661785575 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506e000640 1 --2- 172.21.15.53:0/1661785575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5070071f80 0x7f507015d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 -- 172.21.15.53:0/1661785575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5070071f80 msgr2=0x7f507015d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 --2- 172.21.15.53:0/1661785575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5070071f80 0x7f507015d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 -- 172.21.15.53:0/1661785575 >> v1:172.21.15.146:6789/0 conn(0x7f507015dc30 legacy=0x7f507006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.620+0000 7f506ea00640 1 -- 172.21.15.53:0/1661785575 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5070160e10 con 0x7f507015ae40 2024-01-16T15:00:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.621+0000 7f506ea00640 1 --2- 172.21.15.53:0/1661785575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507015ae40 0x7f5070161610 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5060009870 tx=0x7f506002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29ae682ccbb9906a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506d600640 1 -- 172.21.15.53:0/1661785575 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600317c0 con 0x7f507015ae40 2024-01-16T15:00:45.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506d600640 1 -- 172.21.15.53:0/1661785575 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5060031920 con 0x7f507015ae40 2024-01-16T15:00:45.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506d600640 1 -- 172.21.15.53:0/1661785575 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060031cb0 con 0x7f507015ae40 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506fe00640 1 -- 172.21.15.53:0/1661785575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507015ae40 msgr2=0x7f5070161610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506fe00640 1 --2- 172.21.15.53:0/1661785575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507015ae40 0x7f5070161610 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5060009870 tx=0x7f506002b860 comp rx=0 tx=0).stop 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506fe00640 1 -- 172.21.15.53:0/1661785575 shutdown_connections 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506fe00640 1 --2- 172.21.15.53:0/1661785575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5070071f80 0x7f507015d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506fe00640 1 --2- 172.21.15.53:0/1661785575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507015ae40 0x7f5070161610 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.622+0000 7f506fe00640 1 -- 172.21.15.53:0/1661785575 >> 172.21.15.53:0/1661785575 conn(0x7f507006bb20 msgr2=0x7f50700cffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.623+0000 7f506fe00640 1 -- 172.21.15.53:0/1661785575 shutdown_connections 2024-01-16T15:00:45.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.623+0000 7f506fe00640 1 -- 172.21.15.53:0/1661785575 wait complete. 2024-01-16T15:00:45.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.624+0000 7f506fe00640 1 Processor -- start 2024-01-16T15:00:45.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.624+0000 7f506fe00640 1 -- start start 2024-01-16T15:00:45.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.625+0000 7f506fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 0x7f5070145910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.625+0000 7f506fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507015ae40 0x7f50701550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.625+0000 7f506fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015dc30 0x7f507015c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.625+0000 7f506fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50701537b0 con 0x7f5070071f80 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.625+0000 7f506fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070153a90 con 0x7f507015ae40 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.625+0000 7f506fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5070153d70 con 0x7f507015dc30 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 0x7f5070145910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 0x7f5070145910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60864/0 (socket says 172.21.15.53:60864) 2024-01-16T15:00:45.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 -- 172.21.15.53:0/999675771 learned_addr learned my addr 172.21.15.53:0/999675771 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 -- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015dc30 msgr2=0x7f507015c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015dc30 0x7f507015c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 -- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507015ae40 msgr2=0x7f50701550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507015ae40 0x7f50701550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 -- 172.21.15.53:0/999675771 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600094d0 con 0x7f5070071f80 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.626+0000 7f506ea00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 0x7f5070145910 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f50600090b0 tx=0x7f50600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.627+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060032860 con 0x7f5070071f80 2024-01-16T15:00:45.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.627+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50600329c0 con 0x7f5070071f80 2024-01-16T15:00:45.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.627+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060032d70 con 0x7f5070071f80 2024-01-16T15:00:45.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.627+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5070154050 con 0x7f5070071f80 2024-01-16T15:00:45.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.627+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5070153000 con 0x7f5070071f80 2024-01-16T15:00:45.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.628+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5070154570 con 0x7f5070071f80 2024-01-16T15:00:45.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.629+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f506004d3f0 con 0x7f5070071f80 2024-01-16T15:00:45.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.630+0000 7f5066a00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b2c0 0x7f504806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.630+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f50600abaa0 con 0x7f5070071f80 2024-01-16T15:00:45.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.630+0000 7f506e000640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b2c0 0x7f504806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:45.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.631+0000 7f506e000640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b2c0 0x7f504806d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f50700ce290 tx=0x7f5058008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:45.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.634+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5060076880 con 0x7f5070071f80 2024-01-16T15:00:45.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.828+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5070067b60 con 0x7f5070071f80 2024-01-16T15:00:45.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.828+0000 7f5066a00640 1 -- 172.21.15.53:0/999675771 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5060079ea0 con 0x7f5070071f80 2024-01-16T15:00:45.810 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b2c0 msgr2=0x7f504806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b2c0 0x7f504806d780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f50700ce290 tx=0x7f5058008040 comp rx=0 tx=0).stop 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 msgr2=0x7f5070145910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 0x7f5070145910 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f50600090b0 tx=0x7f50600046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 shutdown_connections 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b2c0 0x7f504806d780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015dc30 0x7f507015c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507015ae40 0x7f50701550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 --2- 172.21.15.53:0/999675771 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5070071f80 0x7f5070145910 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.831+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 >> 172.21.15.53:0/999675771 conn(0x7f507006bb20 msgr2=0x7f5070158bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.832+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 shutdown_connections 2024-01-16T15:00:45.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:45.832+0000 7f506fe00640 1 -- 172.21.15.53:0/999675771 wait complete. 2024-01-16T15:00:45.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:46.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.080+0000 7f58b9400640 1 Processor -- start 2024-01-16T15:00:46.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.080+0000 7f58b9400640 1 -- start start 2024-01-16T15:00:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.082+0000 7f58b9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f58b40ccb50 0x7f58b4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.082+0000 7f58b9400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b415f4e0 con 0x7f58b415bf10 2024-01-16T15:00:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.082+0000 7f58b9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58b4071130 con 0x7f58b40adf20 2024-01-16T15:00:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.082+0000 7f58b9400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58b415f970 con 0x7f58b40ccb50 2024-01-16T15:00:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.082+0000 7f58b8a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f58b415bf10 0x7f58b4143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57908/0 (socket says 172.21.15.53:57908) 2024-01-16T15:00:46.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.082+0000 7f58b8a00640 1 -- 172.21.15.53:0/4254819768 learned_addr learned my addr 172.21.15.53:0/4254819768 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:46.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b3400640 1 --2- 172.21.15.53:0/4254819768 >> v2:172.21.15.146:3300/0 conn(0x7f58b40ccb50 0x7f58b4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061449215 0 0) 0x7f58b415f4e0 con 0x7f58b415bf10 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5894003520 con 0x7f58b415bf10 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1333713196 0 0) 0x7f5894003520 con 0x7f58b415bf10 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 >> v2:172.21.15.146:3300/0 conn(0x7f58b40ccb50 msgr2=0x7f58b4070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 --2- 172.21.15.53:0/4254819768 >> v2:172.21.15.146:3300/0 conn(0x7f58b40ccb50 0x7f58b4070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 >> v1:172.21.15.146:6789/0 conn(0x7f58b40adf20 legacy=0x7f58b40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b415ebb0 con 0x7f58b415bf10 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f58ac0019f0 con 0x7f58b415bf10 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 >> v1:172.21.15.53:6789/0 conn(0x7f58b415bf10 legacy=0x7f58b4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58940039c0 0x7f5894003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940047c0 0x7f5894004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.083+0000 7f58b2a00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58940052a0 0x7f589400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b415f970 con 0x7f58940039c0 2024-01-16T15:00:46.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f589400c1c0 con 0x7f58940047c0 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589400c380 con 0x7f58940052a0 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3e00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58940039c0 0x7f5894003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3400640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940047c0 0x7f5894004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b8a00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58940052a0 0x7f589400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3e00640 1 -- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58940052a0 msgr2=0x7f589400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3e00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58940052a0 0x7f589400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3e00640 1 -- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940047c0 msgr2=0x7f5894004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3e00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940047c0 0x7f5894004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3e00640 1 -- 172.21.15.53:0/4254819768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5894003520 con 0x7f58940039c0 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b8a00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58940052a0 0x7f589400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b3400640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940047c0 0x7f5894004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:46.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.084+0000 7f58b8a00640 1 -- 172.21.15.53:0/4254819768 reap_dead start 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.086+0000 7f58b3e00640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58940039c0 0x7f5894003de0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f58a4001750 tx=0x7f58940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7252cdf76ebed3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.086+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a40052d0 con 0x7f58940039c0 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.086+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58a402e050 con 0x7f58940039c0 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.086+0000 7f58b2a00640 1 -- 172.21.15.53:0/4254819768 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a4003830 con 0x7f58940039c0 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.086+0000 7f58b9400640 1 -- 172.21.15.53:0/4254819768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58940039c0 msgr2=0x7f5894003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.086+0000 7f58b9400640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58940039c0 0x7f5894003de0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f58a4001750 tx=0x7f58940057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.087+0000 7f58b9400640 1 -- 172.21.15.53:0/4254819768 shutdown_connections 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.087+0000 7f58b9400640 1 --2- 172.21.15.53:0/4254819768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58940039c0 0x7f5894003de0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.087+0000 7f58b9400640 1 -- 172.21.15.53:0/4254819768 >> 172.21.15.53:0/4254819768 conn(0x7f58b406a7d0 msgr2=0x7f58b406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.087+0000 7f58b9400640 1 -- 172.21.15.53:0/4254819768 shutdown_connections 2024-01-16T15:00:46.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.087+0000 7f58b9400640 1 -- 172.21.15.53:0/4254819768 wait complete. 2024-01-16T15:00:46.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.088+0000 7f58b9400640 1 Processor -- start 2024-01-16T15:00:46.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.089+0000 7f58b9400640 1 -- start start 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940039c0 0x7f58b4143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 0x7f58b41428b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58b4157ff0 0x7f58b41584a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b4142010 con 0x7f58b4142430 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b415a860 con 0x7f58940039c0 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58b41f7030 con 0x7f58b4157ff0 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 0x7f58b41428b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 0x7f58b41428b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60872/0 (socket says 172.21.15.53:60872) 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b3400640 1 -- 172.21.15.53:0/476545666 learned_addr learned my addr 172.21.15.53:0/476545666 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b3e00640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940039c0 0x7f58b4143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.090+0000 7f58b8a00640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58b4157ff0 0x7f58b41584a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b3400640 1 -- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58b4157ff0 msgr2=0x7f58b41584a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b3400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58b4157ff0 0x7f58b41584a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b3400640 1 -- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940039c0 msgr2=0x7f58b4143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b3400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940039c0 0x7f58b4143090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b3400640 1 -- 172.21.15.53:0/476545666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4141950 con 0x7f58b4142430 2024-01-16T15:00:46.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b3400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 0x7f58b41428b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f589c00e810 tx=0x7f589c00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00c6d0 con 0x7f58b4142430 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c00c830 con 0x7f58b4142430 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00cbc0 con 0x7f58b4142430 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.091+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b4141c30 con 0x7f58b4142430 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.092+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b41435d0 con 0x7f58b4142430 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.093+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b41417b0 con 0x7f58b4142430 2024-01-16T15:00:46.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.094+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f589c00cd20 con 0x7f58b4142430 2024-01-16T15:00:46.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.094+0000 7f58abe00640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f588c06b310 0x7f588c06d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.095+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f589c089920 con 0x7f58b4142430 2024-01-16T15:00:46.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.095+0000 7f58b3e00640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f588c06b310 0x7f588c06d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.096+0000 7f58b3e00640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f588c06b310 0x7f588c06d7d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f58a4033000 tx=0x7f58a4009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.099+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58b41417b0 con 0x7f58b4142430 2024-01-16T15:00:46.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.296+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f58b41417b0 con 0x7f58b4142430 2024-01-16T15:00:46.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.297+0000 7f58abe00640 1 -- 172.21.15.53:0/476545666 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f58b41417b0 con 0x7f58b4142430 2024-01-16T15:00:46.279 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:46.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f588c06b310 msgr2=0x7f588c06d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f588c06b310 0x7f588c06d7d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f58a4033000 tx=0x7f58a4009030 comp rx=0 tx=0).stop 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 msgr2=0x7f58b41428b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 0x7f58b41428b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f589c00e810 tx=0x7f589c00ecd0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 shutdown_connections 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f588c06b310 0x7f588c06d7d0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58b4157ff0 0x7f58b41584a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58b4142430 0x7f58b41428b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 --2- 172.21.15.53:0/476545666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58940039c0 0x7f58b4143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 >> 172.21.15.53:0/476545666 conn(0x7f58b406a7d0 msgr2=0x7f58b406d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 shutdown_connections 2024-01-16T15:00:46.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.300+0000 7f58b9400640 1 -- 172.21.15.53:0/476545666 wait complete. 2024-01-16T15:00:46.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.543+0000 7f376ea00640 1 Processor -- start 2024-01-16T15:00:46.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.543+0000 7f376ea00640 1 -- start start 2024-01-16T15:00:46.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.545+0000 7f376ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f376813df80 0x7f376815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.545+0000 7f376ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f37680cf9a0 0x7f37680cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.545+0000 7f376ea00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376806c470 con 0x7f376813e360 2024-01-16T15:00:46.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.545+0000 7f376ea00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376806c750 con 0x7f376813df80 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.545+0000 7f376ea00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376806ca30 con 0x7f37680cf9a0 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.545+0000 7f3767e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f376813df80 0x7f376815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f376813df80 0x7f376815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60886/0 (socket says 172.21.15.53:60886) 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f375fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f376813e360 0x7f37680cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57912/0 (socket says 172.21.15.53:57912) 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767e00640 1 -- 172.21.15.53:0/2965270708 learned_addr learned my addr 172.21.15.53:0/2965270708 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f376ca00640 1 --2- 172.21.15.53:0/2965270708 >> v2:172.21.15.146:3300/0 conn(0x7f37680cf9a0 0x7f37680cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767e00640 1 -- 172.21.15.53:0/2965270708 >> v2:172.21.15.146:3300/0 conn(0x7f37680cf9a0 msgr2=0x7f37680cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:46.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767e00640 1 --2- 172.21.15.53:0/2965270708 >> v2:172.21.15.146:3300/0 conn(0x7f37680cf9a0 0x7f37680cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767e00640 1 -- 172.21.15.53:0/2965270708 >> v1:172.21.15.53:6789/0 conn(0x7f376813e360 legacy=0x7f37680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767400640 1 -- 172.21.15.53:0/2965270708 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727752655 0 0) 0x7f376806c470 con 0x7f376813e360 2024-01-16T15:00:46.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.546+0000 7f3767e00640 1 -- 172.21.15.53:0/2965270708 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376806cd10 con 0x7f376813df80 2024-01-16T15:00:46.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.547+0000 7f3767e00640 1 --2- 172.21.15.53:0/2965270708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813df80 0x7f376815bee0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f37680ccb10 tx=0x7f375802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b271914bb3a609 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f3767400640 1 -- 172.21.15.53:0/2965270708 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758030860 con 0x7f376813df80 2024-01-16T15:00:46.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f3767400640 1 -- 172.21.15.53:0/2965270708 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37580309c0 con 0x7f376813df80 2024-01-16T15:00:46.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f3767400640 1 -- 172.21.15.53:0/2965270708 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758030d50 con 0x7f376813df80 2024-01-16T15:00:46.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 -- 172.21.15.53:0/2965270708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813df80 msgr2=0x7f376815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 --2- 172.21.15.53:0/2965270708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813df80 0x7f376815bee0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f37680ccb10 tx=0x7f375802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 -- 172.21.15.53:0/2965270708 shutdown_connections 2024-01-16T15:00:46.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 --2- 172.21.15.53:0/2965270708 >> v2:172.21.15.146:3300/0 conn(0x7f37680cf9a0 0x7f37680cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 --2- 172.21.15.53:0/2965270708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813df80 0x7f376815bee0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 -- 172.21.15.53:0/2965270708 >> 172.21.15.53:0/2965270708 conn(0x7f376806a800 msgr2=0x7f376806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 -- 172.21.15.53:0/2965270708 shutdown_connections 2024-01-16T15:00:46.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.548+0000 7f376ea00640 1 -- 172.21.15.53:0/2965270708 wait complete. 2024-01-16T15:00:46.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.550+0000 7f376ea00640 1 Processor -- start 2024-01-16T15:00:46.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.550+0000 7f376ea00640 1 -- start start 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.551+0000 7f376ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37680cf9a0 0x7f37680b2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.551+0000 7f376ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376813df80 0x7f37681f6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.551+0000 7f376ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 0x7f37681f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.551+0000 7f376ea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37680b2f30 con 0x7f376813e360 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.551+0000 7f376ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3768071760 con 0x7f376813df80 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.551+0000 7f376ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3768070bf0 con 0x7f37680cf9a0 2024-01-16T15:00:46.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 0x7f37681f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 0x7f37681f6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60892/0 (socket says 172.21.15.53:60892) 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 -- 172.21.15.53:0/2486315921 learned_addr learned my addr 172.21.15.53:0/2486315921 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f375fe00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376813df80 0x7f37681f6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f3767e00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37680cf9a0 0x7f37680b2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 -- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37680cf9a0 msgr2=0x7f37680b2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37680cf9a0 0x7f37680b2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 -- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376813df80 msgr2=0x7f37681f6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376813df80 0x7f37681f6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.552+0000 7f376ca00640 1 -- 172.21.15.53:0/2486315921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580094d0 con 0x7f376813e360 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.553+0000 7f376ca00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 0x7f37681f6d60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3760007d80 tx=0x7f376000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.553+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760014420 con 0x7f376813e360 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.553+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3760014580 con 0x7f376813e360 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.553+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3760014930 con 0x7f376813e360 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.553+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3768070e70 con 0x7f376813e360 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.553+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3768071bf0 con 0x7f376813e360 2024-01-16T15:00:46.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.554+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3768071910 con 0x7f376813e360 2024-01-16T15:00:46.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.555+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3760014a90 con 0x7f376813e360 2024-01-16T15:00:46.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.556+0000 7f3764c00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f374006b2c0 0x7f374006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:46.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.556+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f376008a5d0 con 0x7f376813e360 2024-01-16T15:00:46.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.557+0000 7f3767e00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f374006b2c0 0x7f374006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:46.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.558+0000 7f3767e00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f374006b2c0 0x7f374006d780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3758036000 tx=0x7f3758008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:46.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.561+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3768071910 con 0x7f376813e360 2024-01-16T15:00:46.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.767+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3768071910 con 0x7f376813e360 2024-01-16T15:00:46.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.768+0000 7f3764c00640 1 -- 172.21.15.53:0/2486315921 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3768071910 con 0x7f376813e360 2024-01-16T15:00:46.749 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:46.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f374006b2c0 msgr2=0x7f374006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f374006b2c0 0x7f374006d780 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3758036000 tx=0x7f3758008030 comp rx=0 tx=0).stop 2024-01-16T15:00:46.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 msgr2=0x7f37681f6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:46.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 0x7f37681f6d60 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3760007d80 tx=0x7f376000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 shutdown_connections 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f374006b2c0 0x7f374006d780 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f376813e360 0x7f37681f6d60 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f376813df80 0x7f37681f6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 --2- 172.21.15.53:0/2486315921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37680cf9a0 0x7f37680b2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 >> 172.21.15.53:0/2486315921 conn(0x7f376806a800 msgr2=0x7f376815d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 shutdown_connections 2024-01-16T15:00:46.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:46.770+0000 7f376ea00640 1 -- 172.21.15.53:0/2486315921 wait complete. 2024-01-16T15:00:46.771 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:00:46.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:46.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:00:46.866 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:46.866 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:00:46.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:46.867 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:00:46.924 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:00:46.924 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:46.984 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:00:46.984 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:47.041 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:47.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.289+0000 7f87bfe00640 1 Processor -- start 2024-01-16T15:00:47.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.289+0000 7f87bfe00640 1 -- start start 2024-01-16T15:00:47.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.291+0000 7f87bfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f87c00cc500 0x7f87c00cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.291+0000 7f87bfe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c0070e50 con 0x7f87c006d160 2024-01-16T15:00:47.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.291+0000 7f87bfe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87c0144020 con 0x7f87c0143c40 2024-01-16T15:00:47.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.291+0000 7f87bfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87c00723c0 con 0x7f87c00cc500 2024-01-16T15:00:47.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.291+0000 7f87bf400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f87c006d160 0x7f87c006d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57918/0 (socket says 172.21.15.53:57918) 2024-01-16T15:00:47.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.291+0000 7f87bf400640 1 -- 172.21.15.53:0/3269468660 learned_addr learned my addr 172.21.15.53:0/3269468660 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:47.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87b5e00640 1 --2- 172.21.15.53:0/3269468660 >> v2:172.21.15.146:3300/0 conn(0x7f87c00cc500 0x7f87c00cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2306869623 0 0) 0x7f87c0070e50 con 0x7f87c006d160 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87a0003520 con 0x7f87c006d160 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1788979234 0 0) 0x7f87a0003520 con 0x7f87c006d160 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 >> v2:172.21.15.146:3300/0 conn(0x7f87c00cc500 msgr2=0x7f87c00cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 --2- 172.21.15.53:0/3269468660 >> v2:172.21.15.146:3300/0 conn(0x7f87c00cc500 0x7f87c00cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 >> v1:172.21.15.146:6789/0 conn(0x7f87c0143c40 legacy=0x7f87c0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0071e00 con 0x7f87c006d160 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f87b8001670 con 0x7f87c006d160 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 >> v1:172.21.15.53:6789/0 conn(0x7f87c006d160 legacy=0x7f87c006d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.292+0000 7f87be000640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87a00039c0 0x7f87a0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87be000640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87a00047c0 0x7f87a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87be000640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87a000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c00723c0 con 0x7f87a00051d0 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87a000c120 con 0x7f87a00039c0 2024-01-16T15:00:47.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87a000c2e0 con 0x7f87a00047c0 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bf400640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87a000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87b5e00640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87a00047c0 0x7f87a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bea00640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87a00039c0 0x7f87a0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bf400640 1 -- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87a00047c0 msgr2=0x7f87a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bf400640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87a00047c0 0x7f87a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bf400640 1 -- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87a00039c0 msgr2=0x7f87a0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bf400640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87a00039c0 0x7f87a0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bf400640 1 -- 172.21.15.53:0/3269468660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87a0003520 con 0x7f87a00051d0 2024-01-16T15:00:47.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.293+0000 7f87bea00640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87a00039c0 0x7f87a0003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.294+0000 7f87bf400640 1 -- 172.21.15.53:0/3269468660 reap_dead start 2024-01-16T15:00:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.294+0000 7f87bf400640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87a000b980 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f87b8001eb0 tx=0x7f87a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0845b3320f02795 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b8005ad0 con 0x7f87a00051d0 2024-01-16T15:00:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8005c30 con 0x7f87a00051d0 2024-01-16T15:00:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87be000640 1 -- 172.21.15.53:0/3269468660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b802c7c0 con 0x7f87a00051d0 2024-01-16T15:00:47.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87bfe00640 1 -- 172.21.15.53:0/3269468660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 msgr2=0x7f87a000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87bfe00640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87a000b980 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f87b8001eb0 tx=0x7f87a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87bfe00640 1 -- 172.21.15.53:0/3269468660 shutdown_connections 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87bfe00640 1 --2- 172.21.15.53:0/3269468660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87a000b980 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.295+0000 7f87bfe00640 1 -- 172.21.15.53:0/3269468660 >> 172.21.15.53:0/3269468660 conn(0x7f87c006a800 msgr2=0x7f87c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.296+0000 7f87bfe00640 1 -- 172.21.15.53:0/3269468660 shutdown_connections 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.296+0000 7f87bfe00640 1 -- 172.21.15.53:0/3269468660 wait complete. 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.297+0000 7f87bfe00640 1 Processor -- start 2024-01-16T15:00:47.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.297+0000 7f87bfe00640 1 -- start start 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.298+0000 7f87bfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87c015b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.298+0000 7f87bfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87c0143050 0x7f87c015a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.298+0000 7f87bfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87c015cf90 0x7f87c015dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.298+0000 7f87bfe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c01434d0 con 0x7f87a00051d0 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.298+0000 7f87bfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c015d410 con 0x7f87c015cf90 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.298+0000 7f87bfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87c015bef0 con 0x7f87c0143050 2024-01-16T15:00:47.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87c015b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87c015b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60922/0 (socket says 172.21.15.53:60922) 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 -- 172.21.15.53:0/4201822966 learned_addr learned my addr 172.21.15.53:0/4201822966 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87b5e00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87c0143050 0x7f87c015a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bf400640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87c015cf90 0x7f87c015dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 -- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87c0143050 msgr2=0x7f87c015a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87c0143050 0x7f87c015a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 -- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87c015cf90 msgr2=0x7f87c015dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87c015cf90 0x7f87c015dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87bea00640 1 -- 172.21.15.53:0/4201822966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c015c1d0 con 0x7f87a00051d0 2024-01-16T15:00:47.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.299+0000 7f87b5e00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87c0143050 0x7f87c015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.300+0000 7f87bea00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87c015b860 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87b000dae0 tx=0x7f87b00020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.300+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b0003da0 con 0x7f87a00051d0 2024-01-16T15:00:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.300+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b000b710 con 0x7f87a00051d0 2024-01-16T15:00:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.300+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c015c4b0 con 0x7f87a00051d0 2024-01-16T15:00:47.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.300+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87b000bac0 con 0x7f87a00051d0 2024-01-16T15:00:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.300+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c015ca70 con 0x7f87a00051d0 2024-01-16T15:00:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.301+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c00cbb60 con 0x7f87a00051d0 2024-01-16T15:00:47.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.303+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f87b000bc20 con 0x7f87a00051d0 2024-01-16T15:00:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.303+0000 7f87b7400640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f879806b310 0x7f879806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.304+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f87b0089f70 con 0x7f87a00051d0 2024-01-16T15:00:47.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.304+0000 7f87b5e00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f879806b310 0x7f879806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.305+0000 7f87b5e00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f879806b310 0x7f879806d7d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87c0135eb0 tx=0x7f87a8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.308+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87b0054d40 con 0x7f87a00051d0 2024-01-16T15:00:47.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.503+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87c0067450 con 0x7f87a00051d0 2024-01-16T15:00:47.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.506+0000 7f87b7400640 1 -- 172.21.15.53:0/4201822966 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f87b0058360 con 0x7f87a00051d0 2024-01-16T15:00:47.488 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:47.488 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:47.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f879806b310 msgr2=0x7f879806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f879806b310 0x7f879806d7d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f87c0135eb0 tx=0x7f87a8008040 comp rx=0 tx=0).stop 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 msgr2=0x7f87c015b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87c015b860 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f87b000dae0 tx=0x7f87b00020a0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 shutdown_connections 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f879806b310 0x7f879806d7d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87c015cf90 0x7f87c015dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87c0143050 0x7f87c015a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 --2- 172.21.15.53:0/4201822966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87a00051d0 0x7f87c015b860 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 >> 172.21.15.53:0/4201822966 conn(0x7f87c006a800 msgr2=0x7f87c00aeb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 shutdown_connections 2024-01-16T15:00:47.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.509+0000 7f87bfe00640 1 -- 172.21.15.53:0/4201822966 wait complete. 2024-01-16T15:00:47.511 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:47.512 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:47.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.757+0000 7f27e9400640 1 Processor -- start 2024-01-16T15:00:47.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.758+0000 7f27e9400640 1 -- start start 2024-01-16T15:00:47.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.759+0000 7f27e9400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27e4071e40 0x7f27e40ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.759+0000 7f27e9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f27e415d770 0x7f27e415dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.759+0000 7f27e9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e406c730 con 0x7f27e4143810 2024-01-16T15:00:47.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.759+0000 7f27e9400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e4143bf0 con 0x7f27e4071e40 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.759+0000 7f27e9400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27e40722f0 con 0x7f27e415d770 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27e4071e40 0x7f27e40ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f27e4071e40 0x7f27e40ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60930/0 (socket says 172.21.15.53:60930) 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 -- 172.21.15.53:0/1966384638 learned_addr learned my addr 172.21.15.53:0/1966384638 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27e8a00640 1 --2- 172.21.15.53:0/1966384638 >> v2:172.21.15.146:3300/0 conn(0x7f27e415d770 0x7f27e415dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 -- 172.21.15.53:0/1966384638 >> v2:172.21.15.146:3300/0 conn(0x7f27e415d770 msgr2=0x7f27e415dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 --2- 172.21.15.53:0/1966384638 >> v2:172.21.15.146:3300/0 conn(0x7f27e415d770 0x7f27e415dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 -- 172.21.15.53:0/1966384638 >> v1:172.21.15.146:6789/0 conn(0x7f27e4143810 legacy=0x7f27e415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:47.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.760+0000 7f27dbe00640 1 -- 172.21.15.53:0/1966384638 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40724d0 con 0x7f27e4071e40 2024-01-16T15:00:47.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.761+0000 7f27dbe00640 1 --2- 172.21.15.53:0/1966384638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e4071e40 0x7f27e40ae170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f27e40b4440 tx=0x7f27cc02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce58989bfbb7553e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.762+0000 7f27e3400640 1 -- 172.21.15.53:0/1966384638 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc031750 con 0x7f27e4071e40 2024-01-16T15:00:47.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.762+0000 7f27e3400640 1 -- 172.21.15.53:0/1966384638 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27cc0318b0 con 0x7f27e4071e40 2024-01-16T15:00:47.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.762+0000 7f27e3400640 1 -- 172.21.15.53:0/1966384638 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27cc031c40 con 0x7f27e4071e40 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.762+0000 7f27e9400640 1 -- 172.21.15.53:0/1966384638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e4071e40 msgr2=0x7f27e40ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.762+0000 7f27e9400640 1 --2- 172.21.15.53:0/1966384638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e4071e40 0x7f27e40ae170 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f27e40b4440 tx=0x7f27cc02b7f0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.763+0000 7f27e9400640 1 -- 172.21.15.53:0/1966384638 shutdown_connections 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.763+0000 7f27e9400640 1 --2- 172.21.15.53:0/1966384638 >> v2:172.21.15.146:3300/0 conn(0x7f27e415d770 0x7f27e415dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.763+0000 7f27e9400640 1 --2- 172.21.15.53:0/1966384638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e4071e40 0x7f27e40ae170 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.763+0000 7f27e9400640 1 -- 172.21.15.53:0/1966384638 >> 172.21.15.53:0/1966384638 conn(0x7f27e406a7d0 msgr2=0x7f27e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.763+0000 7f27e9400640 1 -- 172.21.15.53:0/1966384638 shutdown_connections 2024-01-16T15:00:47.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.763+0000 7f27e9400640 1 -- 172.21.15.53:0/1966384638 wait complete. 2024-01-16T15:00:47.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.764+0000 7f27e9400640 1 Processor -- start 2024-01-16T15:00:47.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.765+0000 7f27e9400640 1 -- start start 2024-01-16T15:00:47.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27e4071e40 0x7f27e40b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e4143810 0x7f27e41440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 0x7f27e415f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40b1940 con 0x7f27e415d770 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27e41e6dd0 con 0x7f27e4143810 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27e40b10c0 con 0x7f27e4071e40 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 0x7f27e415f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 0x7f27e415f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60932/0 (socket says 172.21.15.53:60932) 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 -- 172.21.15.53:0/4186124362 learned_addr learned my addr 172.21.15.53:0/4186124362 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e3e00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27e4071e40 0x7f27e40b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 -- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27e4071e40 msgr2=0x7f27e40b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27e4071e40 0x7f27e40b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 -- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e4143810 msgr2=0x7f27e41440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27dbe00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e4143810 0x7f27e41440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e4143810 0x7f27e41440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.766+0000 7f27e8a00640 1 -- 172.21.15.53:0/4186124362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0094d0 con 0x7f27e415d770 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e3e00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27e4071e40 0x7f27e40b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e8a00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 0x7f27e415f040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f27dc007e60 tx=0x7f27dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc014420 con 0x7f27e415d770 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27dc0145a0 con 0x7f27e415d770 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27dc014950 con 0x7f27e415d770 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e40b1380 con 0x7f27e415d770 2024-01-16T15:00:47.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.767+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e40b0de0 con 0x7f27e415d770 2024-01-16T15:00:47.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.768+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27e41e6ba0 con 0x7f27e415d770 2024-01-16T15:00:47.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.770+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f27dc014ab0 con 0x7f27e415d770 2024-01-16T15:00:47.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.771+0000 7f27e0c00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27bc06b2f0 0x7f27bc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:47.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.771+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f27dc08a5f0 con 0x7f27e415d770 2024-01-16T15:00:47.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.772+0000 7f27e3e00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27bc06b2f0 0x7f27bc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:47.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.773+0000 7f27e3e00640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27bc06b2f0 0x7f27bc06d7b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f27d4003eb0 tx=0x7f27d4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:47.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.776+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27e41e6ba0 con 0x7f27e415d770 2024-01-16T15:00:47.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.969+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f27e4067450 con 0x7f27e415d770 2024-01-16T15:00:47.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.970+0000 7f27e0c00640 1 -- 172.21.15.53:0/4186124362 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f27dc058dd0 con 0x7f27e415d770 2024-01-16T15:00:47.952 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:47.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27bc06b2f0 msgr2=0x7f27bc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27bc06b2f0 0x7f27bc06d7b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f27d4003eb0 tx=0x7f27d4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:47.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 msgr2=0x7f27e415f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:47.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 0x7f27e415f040 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f27dc007e60 tx=0x7f27dc00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 shutdown_connections 2024-01-16T15:00:47.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27bc06b2f0 0x7f27bc06d7b0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.973+0000 7f27e9400640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f27e415d770 0x7f27e415f040 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.974+0000 7f27e9400640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f27e4143810 0x7f27e41440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.974+0000 7f27e9400640 1 --2- 172.21.15.53:0/4186124362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f27e4071e40 0x7f27e40b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:47.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.974+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 >> 172.21.15.53:0/4186124362 conn(0x7f27e406a7d0 msgr2=0x7f27e40b24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:47.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.974+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 shutdown_connections 2024-01-16T15:00:47.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:47.974+0000 7f27e9400640 1 -- 172.21.15.53:0/4186124362 wait complete. 2024-01-16T15:00:47.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:48.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.215+0000 7faea7e00640 1 Processor -- start 2024-01-16T15:00:48.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.216+0000 7faea7e00640 1 -- start start 2024-01-16T15:00:48.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.217+0000 7faea7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faea80cf9a0 0x7faea80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.217+0000 7faea7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea80b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.217+0000 7faea7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea80d0320 con 0x7faea815d4b0 2024-01-16T15:00:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.217+0000 7faea7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea80d1c60 con 0x7faea80cf9a0 2024-01-16T15:00:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.217+0000 7faea7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea80b2730 con 0x7faea80b31d0 2024-01-16T15:00:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.218+0000 7faea7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea80b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.218+0000 7faea6a00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7faea815d4b0 0x7faea80d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:58488/0 (socket says 172.21.15.53:58488) 2024-01-16T15:00:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.218+0000 7faea6a00640 1 -- 172.21.15.53:0/2117678223 learned_addr learned my addr 172.21.15.53:0/2117678223 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.219+0000 7faea6000640 1 --2- 172.21.15.53:0/2117678223 >> v2:172.21.15.146:3300/0 conn(0x7faea80cf9a0 0x7faea80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.219+0000 7faea7400640 1 -- 172.21.15.53:0/2117678223 >> v2:172.21.15.146:3300/0 conn(0x7faea80cf9a0 msgr2=0x7faea80cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.219+0000 7faea7400640 1 --2- 172.21.15.53:0/2117678223 >> v2:172.21.15.146:3300/0 conn(0x7faea80cf9a0 0x7faea80cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.219+0000 7faea7400640 1 -- 172.21.15.53:0/2117678223 >> v1:172.21.15.146:6789/0 conn(0x7faea815d4b0 legacy=0x7faea80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.219+0000 7faea7400640 1 -- 172.21.15.53:0/2117678223 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea80d1ec0 con 0x7faea80b31d0 2024-01-16T15:00:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.219+0000 7faea5600640 1 -- 172.21.15.53:0/2117678223 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443842672 0 0) 0x7faea80d0320 con 0x7faea815d4b0 2024-01-16T15:00:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.221+0000 7faea7400640 1 --2- 172.21.15.53:0/2117678223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea80b3660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faea00097a0 tx=0x7faea002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=820e83e785b3e85e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.221+0000 7faea5600640 1 -- 172.21.15.53:0/2117678223 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0031830 con 0x7faea80b31d0 2024-01-16T15:00:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.221+0000 7faea5600640 1 -- 172.21.15.53:0/2117678223 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea0031990 con 0x7faea80b31d0 2024-01-16T15:00:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.221+0000 7faea5600640 1 -- 172.21.15.53:0/2117678223 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0031d20 con 0x7faea80b31d0 2024-01-16T15:00:48.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 -- 172.21.15.53:0/2117678223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 msgr2=0x7faea80b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 --2- 172.21.15.53:0/2117678223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea80b3660 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faea00097a0 tx=0x7faea002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 -- 172.21.15.53:0/2117678223 shutdown_connections 2024-01-16T15:00:48.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 --2- 172.21.15.53:0/2117678223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea80b3660 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 --2- 172.21.15.53:0/2117678223 >> v2:172.21.15.146:3300/0 conn(0x7faea80cf9a0 0x7faea80cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 -- 172.21.15.53:0/2117678223 >> 172.21.15.53:0/2117678223 conn(0x7faea80d3000 msgr2=0x7faea80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:48.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 -- 172.21.15.53:0/2117678223 shutdown_connections 2024-01-16T15:00:48.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.222+0000 7faea7e00640 1 -- 172.21.15.53:0/2117678223 wait complete. 2024-01-16T15:00:48.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.224+0000 7faea7e00640 1 Processor -- start 2024-01-16T15:00:48.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.224+0000 7faea7e00640 1 -- start start 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea814fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 0x7faea814f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea815d4b0 0x7faea809eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea809cd80 con 0x7faea80cf9a0 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea809d020 con 0x7faea815d4b0 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea809d300 con 0x7faea80b31d0 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.225+0000 7faea6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 0x7faea814f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 0x7faea814f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60944/0 (socket says 172.21.15.53:60944) 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 -- 172.21.15.53:0/3615636369 learned_addr learned my addr 172.21.15.53:0/3615636369 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6a00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea814fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea7400640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea815d4b0 0x7faea809eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 -- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 msgr2=0x7faea814fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea814fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 -- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea815d4b0 msgr2=0x7faea809eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea815d4b0 0x7faea809eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 -- 172.21.15.53:0/3615636369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea0009400 con 0x7faea80cf9a0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.226+0000 7faea6000640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 0x7faea814f360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae90007e40 tx=0x7fae9000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.227+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90014620 con 0x7faea80cf9a0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.227+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9000ee90 con 0x7faea80cf9a0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.227+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae900149c0 con 0x7faea80cf9a0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.227+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea809d5e0 con 0x7faea80cf9a0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.227+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea814f8a0 con 0x7faea80cf9a0 2024-01-16T15:00:48.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.228+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea8067450 con 0x7faea80cf9a0 2024-01-16T15:00:48.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.229+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fae90014b20 con 0x7faea80cf9a0 2024-01-16T15:00:48.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.230+0000 7fae9ea00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae8006b2c0 0x7fae8006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.230+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae9008a630 con 0x7faea80cf9a0 2024-01-16T15:00:48.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.231+0000 7faea6a00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae8006b2c0 0x7fae8006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.232+0000 7faea6a00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae8006b2c0 0x7fae8006d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae98000f60 tx=0x7fae98007b90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.235+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae90055380 con 0x7faea80cf9a0 2024-01-16T15:00:48.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.431+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faea80b1750 con 0x7faea80cf9a0 2024-01-16T15:00:48.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.432+0000 7fae9ea00640 1 -- 172.21.15.53:0/3615636369 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae900589a0 con 0x7faea80cf9a0 2024-01-16T15:00:48.413 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:48.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae8006b2c0 msgr2=0x7fae8006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae8006b2c0 0x7fae8006d780 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae98000f60 tx=0x7fae98007b90 comp rx=0 tx=0).stop 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 msgr2=0x7faea814f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 0x7faea814f360 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fae90007e40 tx=0x7fae9000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 shutdown_connections 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae8006b2c0 0x7fae8006d780 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea815d4b0 0x7faea809eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea80cf9a0 0x7faea814f360 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 --2- 172.21.15.53:0/3615636369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea80b31d0 0x7faea814fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 >> 172.21.15.53:0/3615636369 conn(0x7faea80d3000 msgr2=0x7faea80cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 shutdown_connections 2024-01-16T15:00:48.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.435+0000 7faea7e00640 1 -- 172.21.15.53:0/3615636369 wait complete. 2024-01-16T15:00:48.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:48.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.673+0000 7f2b42400640 1 Processor -- start 2024-01-16T15:00:48.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.673+0000 7f2b42400640 1 -- start start 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.675+0000 7f2b42400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b3c14b2c0 0x7f2b3c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.675+0000 7f2b42400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0cfbf0 con 0x7f2b3c071070 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.675+0000 7f2b42400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c071690 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.675+0000 7f2b42400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c14b6a0 con 0x7f2b3c14b2c0 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b3c14b2c0 0x7f2b3c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b3c14b2c0 0x7f2b3c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60946/0 (socket says 172.21.15.53:60946) 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3b400640 1 -- 172.21.15.53:0/859435799 learned_addr learned my addr 172.21.15.53:0/859435799 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:48.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3be00640 1 --1- 172.21.15.53:0/859435799 >> v1:172.21.15.53:6789/0 conn(0x7f2b3c071070 0x7f2b3c0cc160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57932/0 (socket says 172.21.15.53:57932) 2024-01-16T15:00:48.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3b400640 1 -- 172.21.15.53:0/859435799 >> v1:172.21.15.146:6789/0 conn(0x7f2b3c0cf7d0 legacy=0x7f2b3c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:48.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3b400640 1 -- 172.21.15.53:0/859435799 >> v1:172.21.15.53:6789/0 conn(0x7f2b3c071070 legacy=0x7f2b3c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3b400640 1 -- 172.21.15.53:0/859435799 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c071970 con 0x7f2b3c14b2c0 2024-01-16T15:00:48.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.676+0000 7f2b3a000640 1 -- 172.21.15.53:0/859435799 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940537684 0 0) 0x7f2b3c0cfbf0 con 0x7f2b3c071070 2024-01-16T15:00:48.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.678+0000 7f2b3b400640 1 --2- 172.21.15.53:0/859435799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c14b2c0 0x7f2b3c0cf290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009e50 tx=0x7f2b2c02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8976b4eb7ae7271f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.678+0000 7f2b3a000640 1 -- 172.21.15.53:0/859435799 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c02cb20 con 0x7f2b3c14b2c0 2024-01-16T15:00:48.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.678+0000 7f2b3a000640 1 -- 172.21.15.53:0/859435799 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b2c02cc80 con 0x7f2b3c14b2c0 2024-01-16T15:00:48.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.678+0000 7f2b3a000640 1 -- 172.21.15.53:0/859435799 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b2c0343f0 con 0x7f2b3c14b2c0 2024-01-16T15:00:48.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 -- 172.21.15.53:0/859435799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c14b2c0 msgr2=0x7f2b3c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 --2- 172.21.15.53:0/859435799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c14b2c0 0x7f2b3c0cf290 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c009e50 tx=0x7f2b2c02bf80 comp rx=0 tx=0).stop 2024-01-16T15:00:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 -- 172.21.15.53:0/859435799 shutdown_connections 2024-01-16T15:00:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 --2- 172.21.15.53:0/859435799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c14b2c0 0x7f2b3c0cf290 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 -- 172.21.15.53:0/859435799 >> 172.21.15.53:0/859435799 conn(0x7f2b3c06a800 msgr2=0x7f2b3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 -- 172.21.15.53:0/859435799 shutdown_connections 2024-01-16T15:00:48.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.679+0000 7f2b42400640 1 -- 172.21.15.53:0/859435799 wait complete. 2024-01-16T15:00:48.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.681+0000 7f2b42400640 1 Processor -- start 2024-01-16T15:00:48.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.681+0000 7f2b42400640 1 -- start start 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b42400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b3c071070 0x7f2b3c155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b42400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 0x7f2b3c143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b42400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b3c14b2c0 0x7f2b3c1441d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b42400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c161ee0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b42400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c144850 con 0x7f2b3c071070 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b42400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c162180 con 0x7f2b3c14b2c0 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.682+0000 7f2b3aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 0x7f2b3c143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 0x7f2b3c143c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60956/0 (socket says 172.21.15.53:60956) 2024-01-16T15:00:48.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 -- 172.21.15.53:0/495781628 learned_addr learned my addr 172.21.15.53:0/495781628 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3b400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b3c071070 0x7f2b3c155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3be00640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b3c14b2c0 0x7f2b3c1441d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 -- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b3c14b2c0 msgr2=0x7f2b3c1441d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b3c14b2c0 0x7f2b3c1441d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 -- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b3c071070 msgr2=0x7f2b3c155eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b3c071070 0x7f2b3c155eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 -- 172.21.15.53:0/495781628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b2c009b30 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3be00640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b3c14b2c0 0x7f2b3c1441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.683+0000 7f2b3aa00640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 0x7f2b3c143c90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c430 tx=0x7f2b2400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.684+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24011790 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.684+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b240118f0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.684+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b24011ce0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.684+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c162460 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.684+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c144b30 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.685+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c162650 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.686+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2b2401c460 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.687+0000 7f2b33400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b1406b2c0 0x7f2b1406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.687+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2b24089bb0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.687+0000 7f2b3b400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b1406b2c0 0x7f2b1406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:48.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.689+0000 7f2b3b400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b1406b2c0 0x7f2b1406d780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c035000 tx=0x7f2b2c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.691+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b240549a0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.886+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b3c0675d0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.887+0000 7f2b33400640 1 -- 172.21.15.53:0/495781628 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2b24057fc0 con 0x7f2b3c0cf7d0 2024-01-16T15:00:48.868 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:48.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b1406b2c0 msgr2=0x7f2b1406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b1406b2c0 0x7f2b1406d780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2b2c035000 tx=0x7f2b2c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:48.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 msgr2=0x7f2b3c143c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:48.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 0x7f2b3c143c90 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2b2400c430 tx=0x7f2b2400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 shutdown_connections 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b1406b2c0 0x7f2b1406d780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b3c14b2c0 0x7f2b3c1441d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b3c0cf7d0 0x7f2b3c143c90 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 --2- 172.21.15.53:0/495781628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b3c071070 0x7f2b3c155eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 >> 172.21.15.53:0/495781628 conn(0x7f2b3c06a800 msgr2=0x7f2b3c0b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 shutdown_connections 2024-01-16T15:00:48.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:48.890+0000 7f2b42400640 1 -- 172.21.15.53:0/495781628 wait complete. 2024-01-16T15:00:48.889 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:49.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.130+0000 7fe4c7200640 1 Processor -- start 2024-01-16T15:00:49.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.131+0000 7fe4c7200640 1 -- start start 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.132+0000 7fe4c7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4c00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.132+0000 7fe4c7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4c00cfe70 0x7fe4c0161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.132+0000 7fe4c7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c00d0a10 con 0x7fe4c00cf9f0 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.132+0000 7fe4c7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c00d1f50 con 0x7fe4c00cfe70 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.132+0000 7fe4c7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00cf080 con 0x7fe4c0160920 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c5400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe4c00cf9f0 0x7fe4c0161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57936/0 (socket says 172.21.15.53:57936) 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c5400640 1 -- 172.21.15.53:0/2624181300 learned_addr learned my addr 172.21.15.53:0/2624181300 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:49.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4c00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c6800640 1 --2- 172.21.15.53:0/2624181300 >> v2:172.21.15.146:3300/0 conn(0x7fe4c00cfe70 0x7fe4c0161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261700481 0 0) 0x7fe4c00d0a10 con 0x7fe4c00cf9f0 2024-01-16T15:00:49.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a0003520 con 0x7fe4c00cf9f0 2024-01-16T15:00:49.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.133+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3766931770 0 0) 0x7fe4a0003520 con 0x7fe4c00cf9f0 2024-01-16T15:00:49.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 msgr2=0x7fe4c00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4c00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 >> v2:172.21.15.146:3300/0 conn(0x7fe4c00cfe70 msgr2=0x7fe4c0161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 --2- 172.21.15.53:0/2624181300 >> v2:172.21.15.146:3300/0 conn(0x7fe4c00cfe70 0x7fe4c0161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c00c6be0 con 0x7fe4c00cf9f0 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe4b00019f0 con 0x7fe4c00cf9f0 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 >> v1:172.21.15.53:6789/0 conn(0x7fe4c00cf9f0 legacy=0x7fe4c0161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4a0003970 0x7fe4a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0004d00 0x7fe4a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00d1f50 con 0x7fe4a0003970 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00cf080 con 0x7fe4a0004d00 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a000b980 con 0x7fe4c0160920 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4a0003970 0x7fe4a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5400640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5e00640 1 -- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 msgr2=0x7fe4a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5e00640 1 -- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0004d00 msgr2=0x7fe4a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0004d00 0x7fe4a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.134+0000 7fe4c5e00640 1 -- 172.21.15.53:0/2624181300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a0003520 con 0x7fe4a0003970 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.135+0000 7fe4c5400640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c0160920 0x7fe4a00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:49.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.136+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4a0003970 0x7fe4a0003d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009ac0 tx=0x7fe4a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db36038e0175b98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.136+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a802c6b0 con 0x7fe4a0003970 2024-01-16T15:00:49.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.136+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4a802c810 con 0x7fe4a0003970 2024-01-16T15:00:49.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.136+0000 7fe4c4a00640 1 -- 172.21.15.53:0/2624181300 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a802cba0 con 0x7fe4a0003970 2024-01-16T15:00:49.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c7200640 1 -- 172.21.15.53:0/2624181300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4a0003970 msgr2=0x7fe4a0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2624181300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4a0003970 0x7fe4a0003d90 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8009ac0 tx=0x7fe4a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c6800640 1 -- 172.21.15.53:0/2624181300 reap_dead start 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c7200640 1 -- 172.21.15.53:0/2624181300 shutdown_connections 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c7200640 1 -- 172.21.15.53:0/2624181300 >> 172.21.15.53:0/2624181300 conn(0x7fe4c00d3090 msgr2=0x7fe4c00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c7200640 1 -- 172.21.15.53:0/2624181300 shutdown_connections 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.137+0000 7fe4c7200640 1 -- 172.21.15.53:0/2624181300 wait complete. 2024-01-16T15:00:49.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.139+0000 7fe4c7200640 1 Processor -- start 2024-01-16T15:00:49.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.139+0000 7fe4c7200640 1 -- start start 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0003970 0x7fe4c0151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 0x7fe4c00c8a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c00c8f80 0x7fe4c01f6720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c0156430 con 0x7fe4c0153b10 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c00c9400 con 0x7fe4a0003970 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c0151b50 con 0x7fe4c00c8f80 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 0x7fe4c00c8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 0x7fe4c00c8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60978/0 (socket says 172.21.15.53:60978) 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 -- 172.21.15.53:0/2996692941 learned_addr learned my addr 172.21.15.53:0/2996692941 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 -- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c00c8f80 msgr2=0x7fe4c01f6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c00c8f80 0x7fe4c01f6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 -- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0003970 msgr2=0x7fe4c0151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0003970 0x7fe4c0151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.140+0000 7fe4c5400640 1 -- 172.21.15.53:0/2996692941 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c0151f50 con 0x7fe4c0153b10 2024-01-16T15:00:49.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.141+0000 7fe4c5400640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 0x7fe4c00c8a40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b0d0 tx=0x7fe4b000b590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.141+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0004d60 con 0x7fe4c0153b10 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.141+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b00070d0 con 0x7fe4c0153b10 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.141+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0007440 con 0x7fe4c0153b10 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.141+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c01554d0 con 0x7fe4c0153b10 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.141+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c0155a90 con 0x7fe4c0153b10 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.142+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c0067450 con 0x7fe4c0153b10 2024-01-16T15:00:49.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.143+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe4b00075a0 con 0x7fe4c0153b10 2024-01-16T15:00:49.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.144+0000 7fe4be000640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe49806b340 0x7fe49806d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.144+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe4b0089f30 con 0x7fe4c0153b10 2024-01-16T15:00:49.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.145+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe49806b340 0x7fe49806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.146+0000 7fe4c5e00640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe49806b340 0x7fe49806d800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8036000 tx=0x7fe4a802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.149+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4b0054e40 con 0x7fe4c0153b10 2024-01-16T15:00:49.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.361+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe4c015fff0 con 0x7fe4c0153b10 2024-01-16T15:00:49.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.362+0000 7fe4be000640 1 -- 172.21.15.53:0/2996692941 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe4b0019020 con 0x7fe4c0153b10 2024-01-16T15:00:49.343 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:49.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe49806b340 msgr2=0x7fe49806d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe49806b340 0x7fe49806d800 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe4a8036000 tx=0x7fe4a802e000 comp rx=0 tx=0).stop 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 msgr2=0x7fe4c00c8a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 0x7fe4c00c8a40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000b0d0 tx=0x7fe4b000b590 comp rx=0 tx=0).stop 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 shutdown_connections 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe49806b340 0x7fe49806d800 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4c00c8f80 0x7fe4c01f6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4c0153b10 0x7fe4c00c8a40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 --2- 172.21.15.53:0/2996692941 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4a0003970 0x7fe4c0151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 >> 172.21.15.53:0/2996692941 conn(0x7fe4c00d3090 msgr2=0x7fe4c00cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 shutdown_connections 2024-01-16T15:00:49.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.365+0000 7fe4c7200640 1 -- 172.21.15.53:0/2996692941 wait complete. 2024-01-16T15:00:49.365 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:00:49.365 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:49.422 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.665+0000 7ff7cac00640 1 Processor -- start 2024-01-16T15:00:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.666+0000 7ff7cac00640 1 -- start start 2024-01-16T15:00:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7cac00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff7c40ce8f0 0x7ff7c40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7cac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7c40b2a00 0x7ff7c40b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7cac00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7c40d0de0 con 0x7ff7c4160f80 2024-01-16T15:00:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7cac00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40d10c0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7cac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7c40d13a0 con 0x7ff7c40b2a00 2024-01-16T15:00:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7c3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff7c40ce8f0 0x7ff7c40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7c3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff7c40ce8f0 0x7ff7c40ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60984/0 (socket says 172.21.15.53:60984) 2024-01-16T15:00:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7c3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff7c4160f80 0x7ff7c4161360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57948/0 (socket says 172.21.15.53:57948) 2024-01-16T15:00:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.667+0000 7ff7c3400640 1 -- 172.21.15.53:0/2403576081 learned_addr learned my addr 172.21.15.53:0/2403576081 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c3400640 1 -- 172.21.15.53:0/2403576081 >> v2:172.21.15.146:3300/0 conn(0x7ff7c40b2a00 msgr2=0x7ff7c40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c8c00640 1 --2- 172.21.15.53:0/2403576081 >> v2:172.21.15.146:3300/0 conn(0x7ff7c40b2a00 0x7ff7c40b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c3400640 1 --2- 172.21.15.53:0/2403576081 >> v2:172.21.15.146:3300/0 conn(0x7ff7c40b2a00 0x7ff7c40b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c3400640 1 -- 172.21.15.53:0/2403576081 >> v1:172.21.15.53:6789/0 conn(0x7ff7c4160f80 legacy=0x7ff7c4161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c2a00640 1 -- 172.21.15.53:0/2403576081 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1658021317 0 0) 0x7ff7c40d0de0 con 0x7ff7c4160f80 2024-01-16T15:00:49.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c3400640 1 -- 172.21.15.53:0/2403576081 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7c40d1680 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.668+0000 7ff7c8c00640 1 --2- 172.21.15.53:0/2403576081 >> v2:172.21.15.146:3300/0 conn(0x7ff7c40b2a00 0x7ff7c40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.669+0000 7ff7c3400640 1 --2- 172.21.15.53:0/2403576081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40ced30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40b4cb0 tx=0x7ff7ac02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd3cce73d13cdf6d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.669+0000 7ff7c2a00640 1 -- 172.21.15.53:0/2403576081 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac030870 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.669+0000 7ff7c2a00640 1 -- 172.21.15.53:0/2403576081 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac0309d0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.669+0000 7ff7c2a00640 1 -- 172.21.15.53:0/2403576081 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac030d60 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 -- 172.21.15.53:0/2403576081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 msgr2=0x7ff7c40ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 --2- 172.21.15.53:0/2403576081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40ced30 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff7c40b4cb0 tx=0x7ff7ac02b9b0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 -- 172.21.15.53:0/2403576081 shutdown_connections 2024-01-16T15:00:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 --2- 172.21.15.53:0/2403576081 >> v2:172.21.15.146:3300/0 conn(0x7ff7c40b2a00 0x7ff7c40b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 --2- 172.21.15.53:0/2403576081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40ced30 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 -- 172.21.15.53:0/2403576081 >> 172.21.15.53:0/2403576081 conn(0x7ff7c409c550 msgr2=0x7ff7c40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 -- 172.21.15.53:0/2403576081 shutdown_connections 2024-01-16T15:00:49.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.670+0000 7ff7cac00640 1 -- 172.21.15.53:0/2403576081 wait complete. 2024-01-16T15:00:49.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.672+0000 7ff7cac00640 1 Processor -- start 2024-01-16T15:00:49.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.672+0000 7ff7cac00640 1 -- start start 2024-01-16T15:00:49.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7cac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7c40b2a00 0x7ff7c40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7cac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7cac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7c4160f80 0x7ff7c415ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7cac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c4160470 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7cac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40cd100 con 0x7ff7c40b2a00 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7cac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c40c7dc0 con 0x7ff7c4160f80 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7c3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7c3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40c9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60988/0 (socket says 172.21.15.53:60988) 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.673+0000 7ff7c3400640 1 -- 172.21.15.53:0/1177308657 learned_addr learned my addr 172.21.15.53:0/1177308657 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3e00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7c40b2a00 0x7ff7c40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c8c00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7c4160f80 0x7ff7c415ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3400640 1 -- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7c4160f80 msgr2=0x7ff7c415ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3400640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7c4160f80 0x7ff7c415ff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3400640 1 -- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7c40b2a00 msgr2=0x7ff7c40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3400640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7c40b2a00 0x7ff7c40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3400640 1 -- 172.21.15.53:0/1177308657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7ac0094d0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3e00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7c40b2a00 0x7ff7c40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7c3400640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40c9410 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac02b9f0 tx=0x7ff7ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.674+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac030860 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.675+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7ac043400 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.675+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7ac0437b0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.675+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c40ca2e0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.675+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c40ca8a0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.676+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c40675d0 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.677+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff7ac043910 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.677+0000 7ff7bbe00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff79c06b2c0 0x7ff79c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.678+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff7ac03e070 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.678+0000 7ff7c3e00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff79c06b2c0 0x7ff79c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:49.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.679+0000 7ff7c3e00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff79c06b2c0 0x7ff79c06d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4000f60 tx=0x7ff7b4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:49.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.682+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff7ac077f10 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.873+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c40d3920 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.876+0000 7ff7bbe00640 1 -- 172.21.15.53:0/1177308657 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7ff7ac07b530 con 0x7ff7c40ce8f0 2024-01-16T15:00:49.857 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:49.858 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff79c06b2c0 msgr2=0x7ff79c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff79c06b2c0 0x7ff79c06d780 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff7b4000f60 tx=0x7ff7b4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 msgr2=0x7ff7c40c9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40c9410 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff7ac02b9f0 tx=0x7ff7ac004730 comp rx=0 tx=0).stop 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 shutdown_connections 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff79c06b2c0 0x7ff79c06d780 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff7c4160f80 0x7ff7c415ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff7c40ce8f0 0x7ff7c40c9410 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 --2- 172.21.15.53:0/1177308657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7c40b2a00 0x7ff7c40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:49.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 >> 172.21.15.53:0/1177308657 conn(0x7ff7c409c550 msgr2=0x7ff7c40afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:49.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 shutdown_connections 2024-01-16T15:00:49.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:49.879+0000 7ff7cac00640 1 -- 172.21.15.53:0/1177308657 wait complete. 2024-01-16T15:00:49.877 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:49.877 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:50.023 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:50.030 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:00:50.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.112+0000 7fc1d0c00640 1 Processor -- start 2024-01-16T15:00:50.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.113+0000 7fc1d0c00640 1 -- start start 2024-01-16T15:00:50.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.114+0000 7fc1d0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1cc0b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.114+0000 7fc1d0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc0d1c60 con 0x7fc1cc15d4b0 2024-01-16T15:00:50.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.114+0000 7fc1d0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1cc0b2730 con 0x7fc1cc0cf9a0 2024-01-16T15:00:50.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.114+0000 7fc1d0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0d1ec0 con 0x7fc1cc0cfde0 2024-01-16T15:00:50.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.114+0000 7fc1cb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc1cc15d4b0 0x7fc1cc0d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57960/0 (socket says 172.21.15.53:57960) 2024-01-16T15:00:50.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.114+0000 7fc1cb400640 1 -- 172.21.15.53:0/822589744 learned_addr learned my addr 172.21.15.53:0/822589744 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:50.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1cc0b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212734163 0 0) 0x7fc1cc0d1c60 con 0x7fc1cc15d4b0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1ac003520 con 0x7fc1cc15d4b0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363655702 0 0) 0x7fc1cc0b2730 con 0x7fc1cc0cf9a0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1cc0d1c60 con 0x7fc1cc0cf9a0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2381479113 0 0) 0x7fc1ac003520 con 0x7fc1cc15d4b0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 msgr2=0x7fc1cc0b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1cc0b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 >> v1:172.21.15.146:6789/0 conn(0x7fc1cc0cf9a0 legacy=0x7fc1cc0b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0b29d0 con 0x7fc1cc15d4b0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fc1bc001670 con 0x7fc1cc15d4b0 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.115+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 >> v1:172.21.15.53:6789/0 conn(0x7fc1cc15d4b0 legacy=0x7fc1cc0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1ca000640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1ac003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1ca000640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1ac0040e0 0x7fc1ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1ca000640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1ac004d00 0x7fc1ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0d1ec0 con 0x7fc1ac004d00 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac00bac0 con 0x7fc1ac0040e0 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac00bc80 con 0x7fc1cc0cfde0 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1ac004d00 0x7fc1ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cb400640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1ac003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1caa00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1ac0040e0 0x7fc1ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cbe00640 1 -- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 msgr2=0x7fc1ac003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1ac003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cbe00640 1 -- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1ac0040e0 msgr2=0x7fc1ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1ac0040e0 0x7fc1ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cbe00640 1 -- 172.21.15.53:0/822589744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ac003520 con 0x7fc1ac004d00 2024-01-16T15:00:50.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.116+0000 7fc1cb400640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1cc0cfde0 0x7fc1ac003b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:50.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.117+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1ac004d00 0x7fc1ac005180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009ac0 tx=0x7fc1ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de6ba236df227d89 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c402c6b0 con 0x7fc1ac004d00 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c402c810 con 0x7fc1ac004d00 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1ca000640 1 -- 172.21.15.53:0/822589744 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c402cba0 con 0x7fc1ac004d00 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1d0c00640 1 -- 172.21.15.53:0/822589744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1ac004d00 msgr2=0x7fc1ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/822589744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1ac004d00 0x7fc1ac005180 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4009ac0 tx=0x7fc1ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1cbe00640 1 -- 172.21.15.53:0/822589744 reap_dead start 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1d0c00640 1 -- 172.21.15.53:0/822589744 shutdown_connections 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1d0c00640 1 -- 172.21.15.53:0/822589744 >> 172.21.15.53:0/822589744 conn(0x7fc1cc0d3000 msgr2=0x7fc1cc0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.118+0000 7fc1d0c00640 1 -- 172.21.15.53:0/822589744 shutdown_connections 2024-01-16T15:00:50.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.119+0000 7fc1d0c00640 1 -- 172.21.15.53:0/822589744 wait complete. 2024-01-16T15:00:50.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.120+0000 7fc1d0c00640 1 Processor -- start 2024-01-16T15:00:50.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.120+0000 7fc1d0c00640 1 -- start start 2024-01-16T15:00:50.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.121+0000 7fc1d0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1ac004d00 0x7fc1cc0b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.121+0000 7fc1d0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1cc154640 0x7fc1cc09d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1d0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 0x7fc1cc0b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1d0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc09e140 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1d0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0b6bc0 con 0x7fc1cc154640 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1d0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1cc0b5a70 con 0x7fc1ac004d00 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 0x7fc1cc0b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 0x7fc1cc0b7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32774/0 (socket says 172.21.15.53:32774) 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 -- 172.21.15.53:0/1129973039 learned_addr learned my addr 172.21.15.53:0/1129973039 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cb400640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1ac004d00 0x7fc1cc0b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1caa00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1cc154640 0x7fc1cc09d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 -- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1ac004d00 msgr2=0x7fc1cc0b53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1ac004d00 0x7fc1cc0b53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 -- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1cc154640 msgr2=0x7fc1cc09d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1cc154640 0x7fc1cc09d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cbe00640 1 -- 172.21.15.53:0/1129973039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1cc0b5d50 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.122+0000 7fc1cb400640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1ac004d00 0x7fc1cc0b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:50.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.123+0000 7fc1cbe00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 0x7fc1cc0b7050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4037040 tx=0x7fc1c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.123+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40044e0 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.123+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1c4003610 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.123+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1c40039a0 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.123+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1cc0b6030 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.123+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1cc154ac0 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.125+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1cc0b6d50 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.125+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc1c4003b00 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.126+0000 7fc1c3400640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1a406b2c0 0x7fc1a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.126+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc1c40ab8f0 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.127+0000 7fc1cb400640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1a406b2c0 0x7fc1a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.128+0000 7fc1cb400640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1a406b2c0 0x7fc1a406d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc000ff0 tx=0x7fc1bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.130+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1c4076770 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.326+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc1cc067450 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.326+0000 7fc1c3400640 1 -- 172.21.15.53:0/1129973039 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc1c40420a0 con 0x7fc1cc09dcc0 2024-01-16T15:00:50.307 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:50.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1a406b2c0 msgr2=0x7fc1a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1a406b2c0 0x7fc1a406d780 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc000ff0 tx=0x7fc1bc009040 comp rx=0 tx=0).stop 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 msgr2=0x7fc1cc0b7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 0x7fc1cc0b7050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fc1c4037040 tx=0x7fc1c4004730 comp rx=0 tx=0).stop 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 shutdown_connections 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1a406b2c0 0x7fc1a406d780 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1cc09dcc0 0x7fc1cc0b7050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1cc154640 0x7fc1cc09d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 --2- 172.21.15.53:0/1129973039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1ac004d00 0x7fc1cc0b53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 >> 172.21.15.53:0/1129973039 conn(0x7fc1cc0d3000 msgr2=0x7fc1cc0b0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.329+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 shutdown_connections 2024-01-16T15:00:50.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.330+0000 7fc1d0c00640 1 -- 172.21.15.53:0/1129973039 wait complete. 2024-01-16T15:00:50.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:50.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.558+0000 7f60c2000640 1 Processor -- start 2024-01-16T15:00:50.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.559+0000 7f60c2000640 1 -- start start 2024-01-16T15:00:50.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c2000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f60bc0ccbc0 0x7f60bc15b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc15bc40 0x7f60bc0ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c2000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60bc0cc730 con 0x7f60bc15b320 2024-01-16T15:00:50.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c2000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60bc0cbe00 con 0x7f60bc0ccbc0 2024-01-16T15:00:50.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc0cc0e0 con 0x7f60bc15bc40 2024-01-16T15:00:50.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c0c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f60bc15b320 0x7f60bc0ce060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57974/0 (socket says 172.21.15.53:57974) 2024-01-16T15:00:50.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.560+0000 7f60c0c00640 1 -- 172.21.15.53:0/3512287176 learned_addr learned my addr 172.21.15.53:0/3512287176 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:50.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60b3e00640 1 --2- 172.21.15.53:0/3512287176 >> v2:172.21.15.53:3300/0 conn(0x7f60bc0ccbc0 0x7f60bc15b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60c1600640 1 --2- 172.21.15.53:0/3512287176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc15bc40 0x7f60bc0ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60bbe00640 1 -- 172.21.15.53:0/3512287176 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865760372 0 0) 0x7f60bc0cc730 con 0x7f60bc15b320 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60bbe00640 1 -- 172.21.15.53:0/3512287176 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f609c003520 con 0x7f60bc15b320 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60b3e00640 1 -- 172.21.15.53:0/3512287176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc15bc40 msgr2=0x7f60bc0ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60b3e00640 1 --2- 172.21.15.53:0/3512287176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc15bc40 0x7f60bc0ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60b3e00640 1 -- 172.21.15.53:0/3512287176 >> v1:172.21.15.53:6789/0 conn(0x7f60bc15b320 legacy=0x7f60bc0ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.561+0000 7f60b3e00640 1 -- 172.21.15.53:0/3512287176 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0cc3c0 con 0x7f60bc0ccbc0 2024-01-16T15:00:50.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.562+0000 7f60b3e00640 1 --2- 172.21.15.53:0/3512287176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc0ccbc0 0x7f60bc15b700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a5e885cff431e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60bbe00640 1 -- 172.21.15.53:0/3512287176 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4031830 con 0x7f60bc0ccbc0 2024-01-16T15:00:50.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60bbe00640 1 -- 172.21.15.53:0/3512287176 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a4031990 con 0x7f60bc0ccbc0 2024-01-16T15:00:50.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60bbe00640 1 -- 172.21.15.53:0/3512287176 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4031d20 con 0x7f60bc0ccbc0 2024-01-16T15:00:50.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 -- 172.21.15.53:0/3512287176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc0ccbc0 msgr2=0x7f60bc15b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 --2- 172.21.15.53:0/3512287176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc0ccbc0 0x7f60bc15b700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60a40097a0 tx=0x7f60a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 -- 172.21.15.53:0/3512287176 shutdown_connections 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 --2- 172.21.15.53:0/3512287176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc15bc40 0x7f60bc0ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 --2- 172.21.15.53:0/3512287176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc0ccbc0 0x7f60bc15b700 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 -- 172.21.15.53:0/3512287176 >> 172.21.15.53:0/3512287176 conn(0x7f60bc0b2a60 msgr2=0x7f60bc0b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 -- 172.21.15.53:0/3512287176 shutdown_connections 2024-01-16T15:00:50.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.563+0000 7f60c2000640 1 -- 172.21.15.53:0/3512287176 wait complete. 2024-01-16T15:00:50.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.565+0000 7f60c2000640 1 Processor -- start 2024-01-16T15:00:50.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.565+0000 7f60c2000640 1 -- start start 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60c2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc0ccbc0 0x7f60bc1f2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60c2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 0x7f60bc156630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60c2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60bc15bc40 0x7f60bc156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60c2000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc06cbb0 con 0x7f60bc15b320 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60c2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc1570b0 con 0x7f60bc15bc40 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60c2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc06bfa0 con 0x7f60bc0ccbc0 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.566+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 0x7f60bc156630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 0x7f60bc156630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32780/0 (socket says 172.21.15.53:32780) 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 -- 172.21.15.53:0/2485514049 learned_addr learned my addr 172.21.15.53:0/2485514049 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:50.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60c0c00640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc0ccbc0 0x7f60bc1f2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60c1600640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60bc15bc40 0x7f60bc156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 -- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc0ccbc0 msgr2=0x7f60bc1f2950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc0ccbc0 0x7f60bc1f2950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 -- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60bc15bc40 msgr2=0x7f60bc156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60bc15bc40 0x7f60bc156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 -- 172.21.15.53:0/2485514049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a4009400 con 0x7f60bc15b320 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.567+0000 7f60b3e00640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 0x7f60bc156630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009890 tx=0x7f60a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.568+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a403b980 con 0x7f60bc15b320 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.568+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a403bb00 con 0x7f60bc15b320 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.568+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc06c280 con 0x7f60bc15b320 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.568+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a4045400 con 0x7f60bc15b320 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.568+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc06c840 con 0x7f60bc15b320 2024-01-16T15:00:50.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.569+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc06cd60 con 0x7f60bc15b320 2024-01-16T15:00:50.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.570+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f60a404e430 con 0x7f60bc15b320 2024-01-16T15:00:50.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.571+0000 7f60b9600640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f609406b2c0 0x7f609406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:50.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.571+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f60a40abf00 con 0x7f60bc15b320 2024-01-16T15:00:50.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.572+0000 7f60c0c00640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f609406b2c0 0x7f609406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:50.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.573+0000 7f60c0c00640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f609406b2c0 0x7f609406d780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0011e0 tx=0x7f60ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:50.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.576+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60a4076ce0 con 0x7f60bc15b320 2024-01-16T15:00:50.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.771+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f60bc067b60 con 0x7f60bc15b320 2024-01-16T15:00:50.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.772+0000 7f60b9600640 1 -- 172.21.15.53:0/2485514049 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f60a407a300 con 0x7f60bc15b320 2024-01-16T15:00:50.753 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.774+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f609406b2c0 msgr2=0x7f609406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f609406b2c0 0x7f609406d780 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0011e0 tx=0x7f60ac009040 comp rx=0 tx=0).stop 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 msgr2=0x7f60bc156630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 0x7f60bc156630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f60a4009890 tx=0x7f60a40046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 shutdown_connections 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f609406b2c0 0x7f609406d780 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60bc15bc40 0x7f60bc156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60bc15b320 0x7f60bc156630 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 --2- 172.21.15.53:0/2485514049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60bc0ccbc0 0x7f60bc1f2950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 >> 172.21.15.53:0/2485514049 conn(0x7f60bc0b2a60 msgr2=0x7f60bc0b5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 shutdown_connections 2024-01-16T15:00:50.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:50.775+0000 7f60c2000640 1 -- 172.21.15.53:0/2485514049 wait complete. 2024-01-16T15:00:50.776 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:51.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.019+0000 7fb20b400640 1 Processor -- start 2024-01-16T15:00:51.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.020+0000 7fb20b400640 1 -- start start 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.021+0000 7fb20b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb20409a170 0x7fb2040b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.021+0000 7fb20b400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20406cea0 con 0x7fb2040cd240 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.021+0000 7fb20b400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20406cbc0 con 0x7fb20406c6b0 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.021+0000 7fb20b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20406f350 con 0x7fb20409a170 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb208a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb2040cd240 0x7fb204099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57980/0 (socket says 172.21.15.53:57980) 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb208a00640 1 -- 172.21.15.53:0/2056772300 learned_addr learned my addr 172.21.15.53:0/2056772300 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203e00640 1 --2- 172.21.15.53:0/2056772300 >> v2:172.21.15.53:3300/0 conn(0x7fb20409a170 0x7fb2040b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203400640 1 -- 172.21.15.53:0/2056772300 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278708421 0 0) 0x7fb20406cea0 con 0x7fb2040cd240 2024-01-16T15:00:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203400640 1 -- 172.21.15.53:0/2056772300 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e4003520 con 0x7fb2040cd240 2024-01-16T15:00:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203400640 1 -- 172.21.15.53:0/2056772300 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1081637643 0 0) 0x7fb1e4003520 con 0x7fb2040cd240 2024-01-16T15:00:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203e00640 1 -- 172.21.15.53:0/2056772300 >> v1:172.21.15.146:6789/0 conn(0x7fb20406c6b0 legacy=0x7fb2040cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203e00640 1 -- 172.21.15.53:0/2056772300 >> v1:172.21.15.53:6789/0 conn(0x7fb2040cd240 legacy=0x7fb204099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.022+0000 7fb203e00640 1 -- 172.21.15.53:0/2056772300 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20406f630 con 0x7fb20409a170 2024-01-16T15:00:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.024+0000 7fb203e00640 1 --2- 172.21.15.53:0/2056772300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040b3820 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0097a0 tx=0x7fb1ec02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21314124cc932765 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.024+0000 7fb203400640 1 -- 172.21.15.53:0/2056772300 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec02f050 con 0x7fb20409a170 2024-01-16T15:00:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.024+0000 7fb203400640 1 -- 172.21.15.53:0/2056772300 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec031500 con 0x7fb20409a170 2024-01-16T15:00:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.024+0000 7fb203400640 1 -- 172.21.15.53:0/2056772300 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4001670 con 0x7fb20409a170 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 -- 172.21.15.53:0/2056772300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 msgr2=0x7fb2040b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 --2- 172.21.15.53:0/2056772300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040b3820 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec0097a0 tx=0x7fb1ec02b820 comp rx=0 tx=0).stop 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 -- 172.21.15.53:0/2056772300 shutdown_connections 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 --2- 172.21.15.53:0/2056772300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040b3820 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 -- 172.21.15.53:0/2056772300 >> 172.21.15.53:0/2056772300 conn(0x7fb2040b2d00 msgr2=0x7fb2040b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 -- 172.21.15.53:0/2056772300 shutdown_connections 2024-01-16T15:00:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.025+0000 7fb20b400640 1 -- 172.21.15.53:0/2056772300 wait complete. 2024-01-16T15:00:51.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.027+0000 7fb20b400640 1 Processor -- start 2024-01-16T15:00:51.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.027+0000 7fb20b400640 1 -- start start 2024-01-16T15:00:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb20b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20406c6b0 0x7fb2041f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb20b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb20b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2040cd240 0x7fb20415f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb20b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2040cb5d0 con 0x7fb20409a170 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb20b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2040cad30 con 0x7fb20406c6b0 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb20b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2040cb010 con 0x7fb2040cd240 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040ca500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32808/0 (socket says 172.21.15.53:32808) 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 -- 172.21.15.53:0/2387273819 learned_addr learned my addr 172.21.15.53:0/2387273819 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb208a00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20406c6b0 0x7fb2041f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb209400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2040cd240 0x7fb20415f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 -- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2040cd240 msgr2=0x7fb20415f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2040cd240 0x7fb20415f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 -- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20406c6b0 msgr2=0x7fb2041f6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20406c6b0 0x7fb2041f6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.028+0000 7fb203e00640 1 -- 172.21.15.53:0/2387273819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ec009400 con 0x7fb20409a170 2024-01-16T15:00:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb209400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2040cd240 0x7fb20415f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb208a00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20406c6b0 0x7fb2041f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb203e00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040ca500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec036040 tx=0x7fb1ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec03d940 con 0x7fb20409a170 2024-01-16T15:00:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1ec03daa0 con 0x7fb20409a170 2024-01-16T15:00:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1ec03de10 con 0x7fb20409a170 2024-01-16T15:00:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2040cb2f0 con 0x7fb20409a170 2024-01-16T15:00:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.029+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2040caa40 con 0x7fb20409a170 2024-01-16T15:00:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.031+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2040c6050 con 0x7fb20409a170 2024-01-16T15:00:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.031+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb1ec03a040 con 0x7fb20409a170 2024-01-16T15:00:51.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.032+0000 7fb200c00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1dc06b2c0 0x7fb1dc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.032+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb1ec0ab980 con 0x7fb20409a170 2024-01-16T15:00:51.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.033+0000 7fb208a00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1dc06b2c0 0x7fb1dc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.034+0000 7fb208a00640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1dc06b2c0 0x7fb1dc06d780 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000f60 tx=0x7fb1f4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.037+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2040c6050 con 0x7fb20409a170 2024-01-16T15:00:51.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.230+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb2040c6050 con 0x7fb20409a170 2024-01-16T15:00:51.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.231+0000 7fb200c00640 1 -- 172.21.15.53:0/2387273819 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb2040c6050 con 0x7fb20409a170 2024-01-16T15:00:51.212 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:51.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1dc06b2c0 msgr2=0x7fb1dc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1dc06b2c0 0x7fb1dc06d780 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4000f60 tx=0x7fb1f4009040 comp rx=0 tx=0).stop 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 msgr2=0x7fb2040ca500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040ca500 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fb1ec036040 tx=0x7fb1ec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 shutdown_connections 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1dc06b2c0 0x7fb1dc06d780 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2040cd240 0x7fb20415f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20409a170 0x7fb2040ca500 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 --2- 172.21.15.53:0/2387273819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb20406c6b0 0x7fb2041f6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.233+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 >> 172.21.15.53:0/2387273819 conn(0x7fb2040b2d00 msgr2=0x7fb2040ce3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.234+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 shutdown_connections 2024-01-16T15:00:51.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.234+0000 7fb20b400640 1 -- 172.21.15.53:0/2387273819 wait complete. 2024-01-16T15:00:51.235 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:51.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.479+0000 7f99be200640 1 Processor -- start 2024-01-16T15:00:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.480+0000 7f99be200640 1 -- start start 2024-01-16T15:00:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.481+0000 7f99be200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 0x7f99b80ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.481+0000 7f99be200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99b80d0b60 0x7f99b80aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.481+0000 7f99be200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b8071de0 con 0x7f99b806a7c0 2024-01-16T15:00:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.481+0000 7f99be200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b80af320 con 0x7f99b80d0b60 2024-01-16T15:00:51.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.481+0000 7f99be200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b8072030 con 0x7f99b806a3e0 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.482+0000 7f99b7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 0x7f99b80ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.482+0000 7f99b7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 0x7f99b80ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:55218/0 (socket says 172.21.15.53:55218) 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.482+0000 7f99b7400640 1 -- 172.21.15.53:0/3870592313 learned_addr learned my addr 172.21.15.53:0/3870592313 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.482+0000 7f99b6a00640 1 --2- 172.21.15.53:0/3870592313 >> v2:172.21.15.146:3300/0 conn(0x7f99b80d0b60 0x7f99b80aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.483+0000 7f99b7400640 1 -- 172.21.15.53:0/3870592313 >> v2:172.21.15.146:3300/0 conn(0x7f99b80d0b60 msgr2=0x7f99b80aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.483+0000 7f99b7400640 1 --2- 172.21.15.53:0/3870592313 >> v2:172.21.15.146:3300/0 conn(0x7f99b80d0b60 0x7f99b80aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.483+0000 7f99b7400640 1 -- 172.21.15.53:0/3870592313 >> v1:172.21.15.146:6789/0 conn(0x7f99b806a7c0 legacy=0x7f99b806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.483+0000 7f99b7400640 1 -- 172.21.15.53:0/3870592313 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b8072310 con 0x7f99b806a3e0 2024-01-16T15:00:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.483+0000 7f99b6000640 1 -- 172.21.15.53:0/3870592313 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974411090 0 0) 0x7f99b8071de0 con 0x7f99b806a7c0 2024-01-16T15:00:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.485+0000 7f99b7400640 1 --2- 172.21.15.53:0/3870592313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 0x7f99b80ac8d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009e50 tx=0x7f99a802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0de661e4e8d90df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.485+0000 7f99b6000640 1 -- 172.21.15.53:0/3870592313 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a802cb00 con 0x7f99b806a3e0 2024-01-16T15:00:51.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99b6000640 1 -- 172.21.15.53:0/3870592313 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a802cc60 con 0x7f99b806a3e0 2024-01-16T15:00:51.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99b6000640 1 -- 172.21.15.53:0/3870592313 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a80343f0 con 0x7f99b806a3e0 2024-01-16T15:00:51.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 -- 172.21.15.53:0/3870592313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 msgr2=0x7f99b80ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 --2- 172.21.15.53:0/3870592313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 0x7f99b80ac8d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009e50 tx=0x7f99a802bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 -- 172.21.15.53:0/3870592313 shutdown_connections 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 --2- 172.21.15.53:0/3870592313 >> v2:172.21.15.146:3300/0 conn(0x7f99b80d0b60 0x7f99b80aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 --2- 172.21.15.53:0/3870592313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a3e0 0x7f99b80ac8d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 -- 172.21.15.53:0/3870592313 >> 172.21.15.53:0/3870592313 conn(0x7f99b80cfef0 msgr2=0x7f99b80d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 -- 172.21.15.53:0/3870592313 shutdown_connections 2024-01-16T15:00:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.486+0000 7f99be200640 1 -- 172.21.15.53:0/3870592313 wait complete. 2024-01-16T15:00:51.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.488+0000 7f99be200640 1 Processor -- start 2024-01-16T15:00:51.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.488+0000 7f99be200640 1 -- start start 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.489+0000 7f99be200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 0x7f99b8154c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.489+0000 7f99be200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a7c0 0x7f99b80ce8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.489+0000 7f99be200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99b80d0b60 0x7f99b8153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.489+0000 7f99be200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b81562a0 con 0x7f99b806a3e0 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.489+0000 7f99be200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b8155200 con 0x7f99b80d0b60 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.489+0000 7f99be200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b81554e0 con 0x7f99b806a7c0 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 0x7f99b8154c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 0x7f99b8154c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32816/0 (socket says 172.21.15.53:32816) 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 -- 172.21.15.53:0/1417980750 learned_addr learned my addr 172.21.15.53:0/1417980750 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:51.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b6a00640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a7c0 0x7f99b80ce8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7e00640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99b80d0b60 0x7f99b8153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 -- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a7c0 msgr2=0x7f99b80ce8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a7c0 0x7f99b80ce8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 -- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99b80d0b60 msgr2=0x7f99b8153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99b80d0b60 0x7f99b8153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7400640 1 -- 172.21.15.53:0/1417980750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a8009b30 con 0x7f99b806a3e0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.490+0000 7f99b7e00640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99b80d0b60 0x7f99b8153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.491+0000 7f99b7400640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 0x7f99b8154c20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f99a8000f80 tx=0x7f99a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.491+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a803e450 con 0x7f99b806a3e0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.491+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a803e5f0 con 0x7f99b806a3e0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.491+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99a803e9a0 con 0x7f99b806a3e0 2024-01-16T15:00:51.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.491+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b81557c0 con 0x7f99b806a3e0 2024-01-16T15:00:51.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.491+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b8155d80 con 0x7f99b806a3e0 2024-01-16T15:00:51.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.493+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b80675d0 con 0x7f99b806a3e0 2024-01-16T15:00:51.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.494+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f99a803eb00 con 0x7f99b806a3e0 2024-01-16T15:00:51.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.495+0000 7f99af400640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f999006b2c0 0x7f999006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:51.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.495+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f99a80ac880 con 0x7f99b806a3e0 2024-01-16T15:00:51.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.495+0000 7f99b6a00640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f999006b2c0 0x7f999006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:51.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.496+0000 7f99b6a00640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f999006b2c0 0x7f999006d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99b806b310 tx=0x7f99a0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:51.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.499+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99a8077780 con 0x7f99b806a3e0 2024-01-16T15:00:51.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.708+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f99b815cda0 con 0x7f99b806a3e0 2024-01-16T15:00:51.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.709+0000 7f99af400640 1 -- 172.21.15.53:0/1417980750 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f99a807ada0 con 0x7f99b806a3e0 2024-01-16T15:00:51.690 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:51.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f999006b2c0 msgr2=0x7f999006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f999006b2c0 0x7f999006d780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f99b806b310 tx=0x7f99a0009190 comp rx=0 tx=0).stop 2024-01-16T15:00:51.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 msgr2=0x7f99b8154c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:51.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 0x7f99b8154c20 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f99a8000f80 tx=0x7f99a8004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:51.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 shutdown_connections 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f999006b2c0 0x7f999006d780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99b80d0b60 0x7f99b8153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99b806a7c0 0x7f99b80ce8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 --2- 172.21.15.53:0/1417980750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99b806a3e0 0x7f99b8154c20 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.711+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 >> 172.21.15.53:0/1417980750 conn(0x7f99b80cfef0 msgr2=0x7f99b80ada70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.712+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 shutdown_connections 2024-01-16T15:00:51.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:51.712+0000 7f99be200640 1 -- 172.21.15.53:0/1417980750 wait complete. 2024-01-16T15:00:51.713 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:00:51.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:51.713 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:00:51.805 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:51.805 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:00:51.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:51.806 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:00:51.857 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:00:51.858 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:51.914 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:00:51.914 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:51.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:52.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.199+0000 7f1061a00640 1 Processor -- start 2024-01-16T15:00:52.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.200+0000 7f1061a00640 1 -- start start 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f1061a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c0cd240 0x7f105c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f1061a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f105c06c6b0 0x7f105c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f1061a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f105c099fa0 0x7f105c06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f1061a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105c09a430 con 0x7f105c06c6b0 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f1061a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105c06cea0 con 0x7f105c099fa0 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f1061a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105c06cbc0 con 0x7f105c0cd240 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f105a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f105c06c6b0 0x7f105c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.201+0000 7f105a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f105c06c6b0 0x7f105c099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45412/0 (socket says 172.21.15.53:45412) 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105a000640 1 -- 172.21.15.53:0/1512597628 learned_addr learned my addr 172.21.15.53:0/1512597628 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105aa00640 1 --2- 172.21.15.53:0/1512597628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c0cd240 0x7f105c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105b400640 1 --2- 172.21.15.53:0/1512597628 >> v2:172.21.15.146:3300/0 conn(0x7f105c099fa0 0x7f105c06b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105a000640 1 -- 172.21.15.53:0/1512597628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c0cd240 msgr2=0x7f105c0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105a000640 1 --2- 172.21.15.53:0/1512597628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c0cd240 0x7f105c0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105a000640 1 -- 172.21.15.53:0/1512597628 >> v2:172.21.15.146:3300/0 conn(0x7f105c099fa0 msgr2=0x7f105c06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105a000640 1 --2- 172.21.15.53:0/1512597628 >> v2:172.21.15.146:3300/0 conn(0x7f105c099fa0 0x7f105c06b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.202+0000 7f105a000640 1 -- 172.21.15.53:0/1512597628 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105c06f350 con 0x7f105c06c6b0 2024-01-16T15:00:52.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.203+0000 7f105a000640 1 --2- 172.21.15.53:0/1512597628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c06c6b0 0x7f105c099a60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f105c0722b0 tx=0x7f104402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4785b42e180c2bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.203+0000 7f1059600640 1 -- 172.21.15.53:0/1512597628 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044031870 con 0x7f105c06c6b0 2024-01-16T15:00:52.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.203+0000 7f1059600640 1 -- 172.21.15.53:0/1512597628 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10440319d0 con 0x7f105c06c6b0 2024-01-16T15:00:52.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.203+0000 7f1059600640 1 -- 172.21.15.53:0/1512597628 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044031d60 con 0x7f105c06c6b0 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 -- 172.21.15.53:0/1512597628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c06c6b0 msgr2=0x7f105c099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 --2- 172.21.15.53:0/1512597628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c06c6b0 0x7f105c099a60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f105c0722b0 tx=0x7f104402b910 comp rx=0 tx=0).stop 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 -- 172.21.15.53:0/1512597628 shutdown_connections 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 --2- 172.21.15.53:0/1512597628 >> v2:172.21.15.146:3300/0 conn(0x7f105c099fa0 0x7f105c06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 --2- 172.21.15.53:0/1512597628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c06c6b0 0x7f105c099a60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 --2- 172.21.15.53:0/1512597628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c0cd240 0x7f105c0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 -- 172.21.15.53:0/1512597628 >> 172.21.15.53:0/1512597628 conn(0x7f105c0b2d00 msgr2=0x7f105c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 -- 172.21.15.53:0/1512597628 shutdown_connections 2024-01-16T15:00:52.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.204+0000 7f1061a00640 1 -- 172.21.15.53:0/1512597628 wait complete. 2024-01-16T15:00:52.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.206+0000 7f1061a00640 1 Processor -- start 2024-01-16T15:00:52.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.206+0000 7f1061a00640 1 -- start start 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.207+0000 7f1061a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c06c6b0 0x7f105c14e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.207+0000 7f1061a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 0x7f105c0d0050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.207+0000 7f1061a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f105c0cd240 0x7f105c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.207+0000 7f1061a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0b6ee0 con 0x7f105c099fa0 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.207+0000 7f1061a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0cafd0 con 0x7f105c0cd240 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.207+0000 7f1061a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105c0caa10 con 0x7f105c06c6b0 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 0x7f105c0d0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 0x7f105c0d0050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45418/0 (socket says 172.21.15.53:45418) 2024-01-16T15:00:52.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 -- 172.21.15.53:0/1074450045 learned_addr learned my addr 172.21.15.53:0/1074450045 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105aa00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c06c6b0 0x7f105c14e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 -- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c06c6b0 msgr2=0x7f105c14e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c06c6b0 0x7f105c14e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 -- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f105c0cd240 msgr2=0x7f105c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105b400640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f105c0cd240 0x7f105c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f105c0cd240 0x7f105c0d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105a000640 1 -- 172.21.15.53:0/1074450045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440094d0 con 0x7f105c099fa0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.208+0000 7f105b400640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f105c0cd240 0x7f105c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.209+0000 7f105a000640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 0x7f105c0d0050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1044000c00 tx=0x7f10440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.209+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104403abe0 con 0x7f105c099fa0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.209+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104403ad40 con 0x7f105c099fa0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.209+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044044610 con 0x7f105c099fa0 2024-01-16T15:00:52.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.209+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105c0cacf0 con 0x7f105c099fa0 2024-01-16T15:00:52.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.209+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f105c0c50c0 con 0x7f105c099fa0 2024-01-16T15:00:52.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.210+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105c0675d0 con 0x7f105c099fa0 2024-01-16T15:00:52.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.211+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f104404d450 con 0x7f105c099fa0 2024-01-16T15:00:52.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.212+0000 7f1052a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f103406b2c0 0x7f103406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.212+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f104402d040 con 0x7f105c099fa0 2024-01-16T15:00:52.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.213+0000 7f105aa00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f103406b2c0 0x7f103406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.214+0000 7f105aa00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f103406b2c0 0x7f103406d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f104c006fd0 tx=0x7f104c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.217+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1044076ff0 con 0x7f105c099fa0 2024-01-16T15:00:52.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.413+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f105c0cfb30 con 0x7f105c099fa0 2024-01-16T15:00:52.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.417+0000 7f1052a00640 1 -- 172.21.15.53:0/1074450045 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f104407a610 con 0x7f105c099fa0 2024-01-16T15:00:52.398 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:52.398 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f103406b2c0 msgr2=0x7f103406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f103406b2c0 0x7f103406d780 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f104c006fd0 tx=0x7f104c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 msgr2=0x7f105c0d0050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 0x7f105c0d0050 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1044000c00 tx=0x7f10440046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 shutdown_connections 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f103406b2c0 0x7f103406d780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f105c0cd240 0x7f105c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f105c099fa0 0x7f105c0d0050 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.419+0000 7f1061a00640 1 --2- 172.21.15.53:0/1074450045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f105c06c6b0 0x7f105c14e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.420+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 >> 172.21.15.53:0/1074450045 conn(0x7f105c0b2d00 msgr2=0x7f105c0cc6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.420+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 shutdown_connections 2024-01-16T15:00:52.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.420+0000 7f1061a00640 1 -- 172.21.15.53:0/1074450045 wait complete. 2024-01-16T15:00:52.419 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:52.419 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:52.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.665+0000 7f3e5be00640 1 Processor -- start 2024-01-16T15:00:52.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.666+0000 7f3e5be00640 1 -- start start 2024-01-16T15:00:52.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3e5c15d4b0 0x7f3e5c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c0b2730 con 0x7f3e5c0b41e0 2024-01-16T15:00:52.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c0d1ec0 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c0b29d0 con 0x7f3e5c15d4b0 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3e5c15d4b0 0x7f3e5c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3e5c0b41e0 0x7f3e5c0ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36154/0 (socket says 172.21.15.53:36154) 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.667+0000 7f3e5b400640 1 -- 172.21.15.53:0/2848694713 learned_addr learned my addr 172.21.15.53:0/2848694713 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.668+0000 7f3e5aa00640 1 -- 172.21.15.53:0/2848694713 >> v1:172.21.15.146:6789/0 conn(0x7f3e5c0cf9a0 legacy=0x7f3e5c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.668+0000 7f3e5aa00640 1 -- 172.21.15.53:0/2848694713 >> v1:172.21.15.53:6789/0 conn(0x7f3e5c0b41e0 legacy=0x7f3e5c0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.668+0000 7f3e5aa00640 1 -- 172.21.15.53:0/2848694713 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c0b2cb0 con 0x7f3e5c15d4b0 2024-01-16T15:00:52.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.668+0000 7f3e59600640 1 -- 172.21.15.53:0/2848694713 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043148117 0 0) 0x7f3e5c0b2730 con 0x7f3e5c0b41e0 2024-01-16T15:00:52.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.669+0000 7f3e5aa00640 1 --2- 172.21.15.53:0/2848694713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c15d4b0 0x7f3e5c0d1550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009d90 tx=0x7f3e4c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336e55edd3de58da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.670+0000 7f3e59600640 1 -- 172.21.15.53:0/2848694713 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c02cc60 con 0x7f3e5c15d4b0 2024-01-16T15:00:52.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.670+0000 7f3e59600640 1 -- 172.21.15.53:0/2848694713 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c02cdc0 con 0x7f3e5c15d4b0 2024-01-16T15:00:52.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.670+0000 7f3e59600640 1 -- 172.21.15.53:0/2848694713 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c034660 con 0x7f3e5c15d4b0 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.670+0000 7f3e5be00640 1 -- 172.21.15.53:0/2848694713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c15d4b0 msgr2=0x7f3e5c0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.670+0000 7f3e5be00640 1 --2- 172.21.15.53:0/2848694713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c15d4b0 0x7f3e5c0d1550 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c009d90 tx=0x7f3e4c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.671+0000 7f3e5be00640 1 -- 172.21.15.53:0/2848694713 shutdown_connections 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.671+0000 7f3e5be00640 1 --2- 172.21.15.53:0/2848694713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c15d4b0 0x7f3e5c0d1550 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.671+0000 7f3e5be00640 1 -- 172.21.15.53:0/2848694713 >> 172.21.15.53:0/2848694713 conn(0x7f3e5c0d3000 msgr2=0x7f3e5c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.671+0000 7f3e5be00640 1 -- 172.21.15.53:0/2848694713 shutdown_connections 2024-01-16T15:00:52.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.671+0000 7f3e5be00640 1 -- 172.21.15.53:0/2848694713 wait complete. 2024-01-16T15:00:52.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.672+0000 7f3e5be00640 1 Processor -- start 2024-01-16T15:00:52.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.672+0000 7f3e5be00640 1 -- start start 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.673+0000 7f3e5be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e5c0b41e0 0x7f3e5c09d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.673+0000 7f3e5be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 0x7f3e5c1f65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.673+0000 7f3e5be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e5c15d4b0 0x7f3e5c1f6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.673+0000 7f3e5be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c15a9e0 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.673+0000 7f3e5be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0b5f90 con 0x7f3e5c0b41e0 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.673+0000 7f3e5be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0b6270 con 0x7f3e5c15d4b0 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 0x7f3e5c1f65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 0x7f3e5c1f65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45436/0 (socket says 172.21.15.53:45436) 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 -- 172.21.15.53:0/825468755 learned_addr learned my addr 172.21.15.53:0/825468755 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:52.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5aa00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e5c0b41e0 0x7f3e5c09d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5b400640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e5c15d4b0 0x7f3e5c1f6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 -- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e5c15d4b0 msgr2=0x7f3e5c1f6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e5c15d4b0 0x7f3e5c1f6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 -- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e5c0b41e0 msgr2=0x7f3e5c09d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e5c0b41e0 0x7f3e5c09d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5a000640 1 -- 172.21.15.53:0/825468755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c009a70 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.674+0000 7f3e5aa00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e5c0b41e0 0x7f3e5c09d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.675+0000 7f3e5a000640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 0x7f3e5c1f65d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c430 tx=0x7f3e4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.675+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44011790 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.675+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e440118f0 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.675+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e44011ca0 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.675+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5c15da10 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.675+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c0b6b10 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.676+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c0c9f40 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.677+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3e4401c460 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.678+0000 7f3e52a00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e3406b2c0 0x7f3e3406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:52.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.678+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3e44089c00 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.679+0000 7f3e5aa00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e3406b2c0 0x7f3e3406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:52.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.680+0000 7f3e5aa00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e3406b2c0 0x7f3e3406d780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c036000 tx=0x7f3e4c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:52.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.683+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3e44090080 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.876+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e5c0b6ca0 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.877+0000 7f3e52a00640 1 -- 172.21.15.53:0/825468755 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e440549f0 con 0x7f3e5c0cf9a0 2024-01-16T15:00:52.858 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e3406b2c0 msgr2=0x7f3e3406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e3406b2c0 0x7f3e3406d780 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c036000 tx=0x7f3e4c030000 comp rx=0 tx=0).stop 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 msgr2=0x7f3e5c1f65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 0x7f3e5c1f65d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400c430 tx=0x7f3e4400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 shutdown_connections 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e3406b2c0 0x7f3e3406d780 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3e5c15d4b0 0x7f3e5c1f6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3e5c0cf9a0 0x7f3e5c1f65d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 --2- 172.21.15.53:0/825468755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3e5c0b41e0 0x7f3e5c09d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:52.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 >> 172.21.15.53:0/825468755 conn(0x7f3e5c0d3000 msgr2=0x7f3e5c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:52.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 shutdown_connections 2024-01-16T15:00:52.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:52.880+0000 7f3e5be00640 1 -- 172.21.15.53:0/825468755 wait complete. 2024-01-16T15:00:52.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:53.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.118+0000 7f18af200640 1 Processor -- start 2024-01-16T15:00:53.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.119+0000 7f18af200640 1 -- start start 2024-01-16T15:00:53.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.120+0000 7f18af200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 0x7f18b00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.120+0000 7f18af200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b00d0320 con 0x7f18b015d4b0 2024-01-16T15:00:53.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.120+0000 7f18af200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18b00d1c60 con 0x7f18b00b31d0 2024-01-16T15:00:53.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.120+0000 7f18af200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00b2730 con 0x7f18b00cf9a0 2024-01-16T15:00:53.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.121+0000 7f18ade00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f18b015d4b0 0x7f18b00d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36164/0 (socket says 172.21.15.53:36164) 2024-01-16T15:00:53.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.121+0000 7f18ade00640 1 -- 172.21.15.53:0/3369700646 learned_addr learned my addr 172.21.15.53:0/3369700646 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:53.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.121+0000 7f18a7e00640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 0x7f18b00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.121+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2752580779 0 0) 0x7f18b00d0320 con 0x7f18b015d4b0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.121+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1890003520 con 0x7f18b015d4b0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2123383283 0 0) 0x7f1890003520 con 0x7f18b015d4b0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 msgr2=0x7f18b00cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 0x7f18b00cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 >> v1:172.21.15.146:6789/0 conn(0x7f18b00b31d0 legacy=0x7f18b00b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b00d1ec0 con 0x7f18b015d4b0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f18a0001670 con 0x7f18b015d4b0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 >> v1:172.21.15.53:6789/0 conn(0x7f18b015d4b0 legacy=0x7f18b00d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900039a0 0x7f1890003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18900047c0 0x7f1890004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 0x7f18900051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00b2730 con 0x7f18900047c0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f189000bab0 con 0x7f18900039a0 2024-01-16T15:00:53.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f189000bc70 con 0x7f18b00cf9a0 2024-01-16T15:00:53.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.122+0000 7f18a7e00640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18900047c0 0x7f1890004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18ade00640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900039a0 0x7f1890003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18ae800640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 0x7f18900051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18a7e00640 1 -- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 msgr2=0x7f18900051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18a7e00640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00cf9a0 0x7f18900051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18a7e00640 1 -- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900039a0 msgr2=0x7f1890003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18a7e00640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900039a0 0x7f1890003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.123+0000 7f18a7e00640 1 -- 172.21.15.53:0/3369700646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1890003520 con 0x7f18900047c0 2024-01-16T15:00:53.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.124+0000 7f18a7e00640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18900047c0 0x7f1890004c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f189800ba30 tx=0x7f18900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1de6c321bab8fae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.124+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898032640 con 0x7f18900047c0 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.124+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18980327a0 con 0x7f18900047c0 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.124+0000 7f18ad400640 1 -- 172.21.15.53:0/3369700646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898032b30 con 0x7f18900047c0 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18af200640 1 -- 172.21.15.53:0/3369700646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18900047c0 msgr2=0x7f1890004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18af200640 1 --2- 172.21.15.53:0/3369700646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18900047c0 0x7f1890004c40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f189800ba30 tx=0x7f18900057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18ae800640 1 -- 172.21.15.53:0/3369700646 reap_dead start 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18af200640 1 -- 172.21.15.53:0/3369700646 shutdown_connections 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18af200640 1 -- 172.21.15.53:0/3369700646 >> 172.21.15.53:0/3369700646 conn(0x7f18b00d3000 msgr2=0x7f18b00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18af200640 1 -- 172.21.15.53:0/3369700646 shutdown_connections 2024-01-16T15:00:53.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.125+0000 7f18af200640 1 -- 172.21.15.53:0/3369700646 wait complete. 2024-01-16T15:00:53.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.127+0000 7f18af200640 1 Processor -- start 2024-01-16T15:00:53.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.127+0000 7f18af200640 1 -- start start 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18af200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900047c0 0x7f18b014ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18af200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 0x7f18b01f2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18af200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00b7750 0x7f18b00b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18af200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00c83d0 con 0x7f18b00ca0d0 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18af200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00cd450 con 0x7f18900047c0 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18af200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b00c8c00 con 0x7f18b00b7750 2024-01-16T15:00:53.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18a7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 0x7f18b01f2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18a7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 0x7f18b01f2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45452/0 (socket says 172.21.15.53:45452) 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.128+0000 7f18a7e00640 1 -- 172.21.15.53:0/21233280 learned_addr learned my addr 172.21.15.53:0/21233280 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18ade00640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900047c0 0x7f18b014ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18ae800640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00b7750 0x7f18b00b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18a7e00640 1 -- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00b7750 msgr2=0x7f18b00b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18a7e00640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00b7750 0x7f18b00b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18a7e00640 1 -- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900047c0 msgr2=0x7f18b014ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18a7e00640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900047c0 0x7f18b014ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18a7e00640 1 -- 172.21.15.53:0/21233280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b00b8110 con 0x7f18b00ca0d0 2024-01-16T15:00:53.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.129+0000 7f18a7e00640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 0x7f18b01f2870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1898035040 tx=0x7f1898004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.130+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1898004220 con 0x7f18b00ca0d0 2024-01-16T15:00:53.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.130+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1898004380 con 0x7f18b00ca0d0 2024-01-16T15:00:53.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.130+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189803ba90 con 0x7f18b00ca0d0 2024-01-16T15:00:53.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.130+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b00cd170 con 0x7f18b00ca0d0 2024-01-16T15:00:53.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.130+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b00c8890 con 0x7f18b00ca0d0 2024-01-16T15:00:53.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.131+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b00c8d70 con 0x7f18b00ca0d0 2024-01-16T15:00:53.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.132+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f189803bbf0 con 0x7f18b00ca0d0 2024-01-16T15:00:53.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.133+0000 7f18a6000640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f188806b2c0 0x7f188806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.133+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f18980ac530 con 0x7f18b00ca0d0 2024-01-16T15:00:53.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.133+0000 7f18ade00640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f188806b2c0 0x7f188806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.135+0000 7f18ade00640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f188806b2c0 0x7f188806d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f18a0001c90 tx=0x7f18a0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.137+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1898077310 con 0x7f18b00ca0d0 2024-01-16T15:00:53.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.331+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f18b00c8d70 con 0x7f18b00ca0d0 2024-01-16T15:00:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.332+0000 7f18a6000640 1 -- 172.21.15.53:0/21233280 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f189807fe60 con 0x7f18b00ca0d0 2024-01-16T15:00:53.314 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:53.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f188806b2c0 msgr2=0x7f188806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f188806b2c0 0x7f188806d780 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f18a0001c90 tx=0x7f18a0009040 comp rx=0 tx=0).stop 2024-01-16T15:00:53.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 msgr2=0x7f18b01f2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 0x7f18b01f2870 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1898035040 tx=0x7f1898004730 comp rx=0 tx=0).stop 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 shutdown_connections 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f188806b2c0 0x7f188806d780 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18b00b7750 0x7f18b00b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18b00ca0d0 0x7f18b01f2870 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 --2- 172.21.15.53:0/21233280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18900047c0 0x7f18b014ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 >> 172.21.15.53:0/21233280 conn(0x7f18b00d3000 msgr2=0x7f18b00b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 shutdown_connections 2024-01-16T15:00:53.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.335+0000 7f18af200640 1 -- 172.21.15.53:0/21233280 wait complete. 2024-01-16T15:00:53.335 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:53.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.582+0000 7fcb4da00640 1 Processor -- start 2024-01-16T15:00:53.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.583+0000 7fcb4da00640 1 -- start start 2024-01-16T15:00:53.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.584+0000 7fcb4da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcb4815e3b0 0x7fcb48164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.584+0000 7fcb4da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480b1ee0 con 0x7fcb480cf9a0 2024-01-16T15:00:53.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb4da00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb480b2780 con 0x7fcb48143ae0 2024-01-16T15:00:53.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb4da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb4815e990 con 0x7fcb4815e3b0 2024-01-16T15:00:53.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb47400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcb4815e3b0 0x7fcb48164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb47400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcb4815e3b0 0x7fcb48164de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45458/0 (socket says 172.21.15.53:45458) 2024-01-16T15:00:53.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb47400640 1 -- 172.21.15.53:0/2596188947 learned_addr learned my addr 172.21.15.53:0/2596188947 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:53.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb3fe00640 1 --1- 172.21.15.53:0/2596188947 >> v1:172.21.15.53:6789/0 conn(0x7fcb480cf9a0 0x7fcb4815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36176/0 (socket says 172.21.15.53:36176) 2024-01-16T15:00:53.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb47400640 1 -- 172.21.15.53:0/2596188947 >> v1:172.21.15.146:6789/0 conn(0x7fcb48143ae0 legacy=0x7fcb480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:53.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb47400640 1 -- 172.21.15.53:0/2596188947 >> v1:172.21.15.53:6789/0 conn(0x7fcb480cf9a0 legacy=0x7fcb4815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.585+0000 7fcb47400640 1 -- 172.21.15.53:0/2596188947 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb480b2130 con 0x7fcb4815e3b0 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.587+0000 7fcb47400640 1 --2- 172.21.15.53:0/2596188947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb4815e3b0 0x7fcb48164de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009e50 tx=0x7fcb4002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa912e7b98c35e59 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.587+0000 7fcb46000640 1 -- 172.21.15.53:0/2596188947 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4002cb20 con 0x7fcb4815e3b0 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.587+0000 7fcb46000640 1 -- 172.21.15.53:0/2596188947 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4002cc80 con 0x7fcb4815e3b0 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.587+0000 7fcb46000640 1 -- 172.21.15.53:0/2596188947 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400343f0 con 0x7fcb4815e3b0 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 -- 172.21.15.53:0/2596188947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb4815e3b0 msgr2=0x7fcb48164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2596188947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb4815e3b0 0x7fcb48164de0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009e50 tx=0x7fcb4002bf80 comp rx=0 tx=0).stop 2024-01-16T15:00:53.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 -- 172.21.15.53:0/2596188947 shutdown_connections 2024-01-16T15:00:53.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2596188947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb4815e3b0 0x7fcb48164de0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 -- 172.21.15.53:0/2596188947 >> 172.21.15.53:0/2596188947 conn(0x7fcb4806a800 msgr2=0x7fcb4806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 -- 172.21.15.53:0/2596188947 shutdown_connections 2024-01-16T15:00:53.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.588+0000 7fcb4da00640 1 -- 172.21.15.53:0/2596188947 wait complete. 2024-01-16T15:00:53.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.590+0000 7fcb4da00640 1 Processor -- start 2024-01-16T15:00:53.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.590+0000 7fcb4da00640 1 -- start start 2024-01-16T15:00:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.591+0000 7fcb4da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb480cf9a0 0x7fcb48141b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb4da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 0x7fcb480af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb4da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb4815e3b0 0x7fcb481f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb4da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480b0440 con 0x7fcb48143ae0 2024-01-16T15:00:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb4da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb480af970 con 0x7fcb480cf9a0 2024-01-16T15:00:53.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb4da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4813f320 con 0x7fcb4815e3b0 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 0x7fcb480af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 0x7fcb480af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45468/0 (socket says 172.21.15.53:45468) 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 -- 172.21.15.53:0/2670368235 learned_addr learned my addr 172.21.15.53:0/2670368235 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb46a00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb480cf9a0 0x7fcb48141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 -- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb4815e3b0 msgr2=0x7fcb481f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb47400640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb4815e3b0 0x7fcb481f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb4815e3b0 0x7fcb481f2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 -- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb480cf9a0 msgr2=0x7fcb48141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb480cf9a0 0x7fcb48141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb3fe00640 1 -- 172.21.15.53:0/2670368235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40009b30 con 0x7fcb48143ae0 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.592+0000 7fcb46a00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb480cf9a0 0x7fcb48141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.593+0000 7fcb3fe00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 0x7fcb480af430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000c4d0 tx=0x7fcb3000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.593+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb30007480 con 0x7fcb48143ae0 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.593+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb300075e0 con 0x7fcb48143ae0 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.593+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb300049e0 con 0x7fcb48143ae0 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.593+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb480cb670 con 0x7fcb48143ae0 2024-01-16T15:00:53.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.593+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb480afe80 con 0x7fcb48143ae0 2024-01-16T15:00:53.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.595+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4813f0f0 con 0x7fcb48143ae0 2024-01-16T15:00:53.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.595+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fcb30004b40 con 0x7fcb48143ae0 2024-01-16T15:00:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.596+0000 7fcb3ea00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2006b2c0 0x7fcb2006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.597+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcb30088ca0 con 0x7fcb48143ae0 2024-01-16T15:00:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.597+0000 7fcb46a00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2006b2c0 0x7fcb2006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:53.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.599+0000 7fcb46a00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2006b2c0 0x7fcb2006d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcb480b1bf0 tx=0x7fcb38009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:53.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.603+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb4813f0f0 con 0x7fcb48143ae0 2024-01-16T15:00:53.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.799+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb4813f0f0 con 0x7fcb48143ae0 2024-01-16T15:00:53.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.800+0000 7fcb3ea00640 1 -- 172.21.15.53:0/2670368235 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb4813f0f0 con 0x7fcb48143ae0 2024-01-16T15:00:53.781 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:53.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2006b2c0 msgr2=0x7fcb2006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2006b2c0 0x7fcb2006d780 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fcb480b1bf0 tx=0x7fcb38009040 comp rx=0 tx=0).stop 2024-01-16T15:00:53.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 msgr2=0x7fcb480af430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:53.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 0x7fcb480af430 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fcb3000c4d0 tx=0x7fcb3000c990 comp rx=0 tx=0).stop 2024-01-16T15:00:53.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 shutdown_connections 2024-01-16T15:00:53.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2006b2c0 0x7fcb2006d780 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb4815e3b0 0x7fcb481f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb48143ae0 0x7fcb480af430 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 --2- 172.21.15.53:0/2670368235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb480cf9a0 0x7fcb48141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:53.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 >> 172.21.15.53:0/2670368235 conn(0x7fcb4806a800 msgr2=0x7fcb4815c880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:53.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 shutdown_connections 2024-01-16T15:00:53.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:53.803+0000 7fcb4da00640 1 -- 172.21.15.53:0/2670368235 wait complete. 2024-01-16T15:00:53.804 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:54.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.049+0000 7f94f7200640 1 Processor -- start 2024-01-16T15:00:54.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.050+0000 7f94f7200640 1 -- start start 2024-01-16T15:00:54.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.051+0000 7f94f7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f94f80d0100 0x7f94f80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.051+0000 7f94f7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f80d0a10 con 0x7f94f80d0f40 2024-01-16T15:00:54.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.051+0000 7f94f7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94f80cf080 con 0x7f94f8160920 2024-01-16T15:00:54.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.051+0000 7f94f7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94f80c6be0 con 0x7f94f80d0100 2024-01-16T15:00:54.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.051+0000 7f94f6800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f94f80d0f40 0x7f94f80d1390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36182/0 (socket says 172.21.15.53:36182) 2024-01-16T15:00:54.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.051+0000 7f94f6800640 1 -- 172.21.15.53:0/774097648 learned_addr learned my addr 172.21.15.53:0/774097648 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94efe00640 1 --2- 172.21.15.53:0/774097648 >> v2:172.21.15.146:3300/0 conn(0x7f94f80d0100 0x7f94f80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1301023132 0 0) 0x7f94f80d0a10 con 0x7f94f80d0f40 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94d8003520 con 0x7f94f80d0f40 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2041560043 0 0) 0x7f94d8003520 con 0x7f94f80d0f40 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 >> v2:172.21.15.146:3300/0 conn(0x7f94f80d0100 msgr2=0x7f94f80b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 --2- 172.21.15.53:0/774097648 >> v2:172.21.15.146:3300/0 conn(0x7f94f80d0100 0x7f94f80b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 >> v1:172.21.15.146:6789/0 conn(0x7f94f8160920 legacy=0x7f94f80cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80ce7e0 con 0x7f94f80d0f40 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f94f00019f0 con 0x7f94f80d0f40 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 >> v1:172.21.15.53:6789/0 conn(0x7f94f80d0f40 legacy=0x7f94f80d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80039c0 0x7f94d8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.052+0000 7f94f5400640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d80047c0 0x7f94d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94f5400640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d8005180 0x7f94d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80c6be0 con 0x7f94d80047c0 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d800c080 con 0x7f94d80039c0 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d800c240 con 0x7f94d8005180 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94efe00640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d80047c0 0x7f94d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94f6800640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d8005180 0x7f94d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94efe00640 1 -- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d8005180 msgr2=0x7f94d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94efe00640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94d8005180 0x7f94d800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94efe00640 1 -- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80039c0 msgr2=0x7f94d8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94efe00640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80039c0 0x7f94d8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94efe00640 1 -- 172.21.15.53:0/774097648 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d8003520 con 0x7f94d80047c0 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.053+0000 7f94f6800640 1 -- 172.21.15.53:0/774097648 reap_dead start 2024-01-16T15:00:54.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.054+0000 7f94efe00640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d80047c0 0x7f94d8004c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009ac0 tx=0x7f94d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb0745d4a8d9b024 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e002c6b0 con 0x7f94d80047c0 2024-01-16T15:00:54.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e002c810 con 0x7f94d80047c0 2024-01-16T15:00:54.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f5400640 1 -- 172.21.15.53:0/774097648 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e002cba0 con 0x7f94d80047c0 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 -- 172.21.15.53:0/774097648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d80047c0 msgr2=0x7f94d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d80047c0 0x7f94d8004c40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f94e0009ac0 tx=0x7f94d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 -- 172.21.15.53:0/774097648 shutdown_connections 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 --2- 172.21.15.53:0/774097648 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94d80047c0 0x7f94d8004c40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 -- 172.21.15.53:0/774097648 >> 172.21.15.53:0/774097648 conn(0x7f94f80d3090 msgr2=0x7f94f80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 -- 172.21.15.53:0/774097648 shutdown_connections 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.055+0000 7f94f7200640 1 -- 172.21.15.53:0/774097648 wait complete. 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.057+0000 7f94f7200640 1 Processor -- start 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.057+0000 7f94f7200640 1 -- start start 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.058+0000 7f94f7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80047c0 0x7f94f8150570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.058+0000 7f94f7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 0x7f94f815fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.058+0000 7f94f7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94f80b79a0 0x7f94f80b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.058+0000 7f94f7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f80b8360 con 0x7f94f815f850 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.058+0000 7f94f7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94f815a8e0 con 0x7f94d80047c0 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.058+0000 7f94f7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94f8150ab0 con 0x7f94f80b79a0 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 0x7f94f815fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 0x7f94f815fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45482/0 (socket says 172.21.15.53:45482) 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 -- 172.21.15.53:0/4186370787 learned_addr learned my addr 172.21.15.53:0/4186370787 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94f6800640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94f80b79a0 0x7f94f80b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94f5e00640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80047c0 0x7f94f8150570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 -- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94f80b79a0 msgr2=0x7f94f80b7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94f80b79a0 0x7f94f80b7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 -- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80047c0 msgr2=0x7f94f8150570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80047c0 0x7f94f8150570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 -- 172.21.15.53:0/4186370787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94f80ca160 con 0x7f94f815f850 2024-01-16T15:00:54.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.059+0000 7f94efe00640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 0x7f94f815fcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f94e0037040 tx=0x7f94e0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.060+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0003810 con 0x7f94f815f850 2024-01-16T15:00:54.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.060+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94e0003970 con 0x7f94f815f850 2024-01-16T15:00:54.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.060+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94e0003d20 con 0x7f94f815f850 2024-01-16T15:00:54.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.060+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94f80ca440 con 0x7f94f815f850 2024-01-16T15:00:54.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.060+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94f80caa00 con 0x7f94f815f850 2024-01-16T15:00:54.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.061+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94f8067450 con 0x7f94f815f850 2024-01-16T15:00:54.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.062+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f94e0034440 con 0x7f94f815f850 2024-01-16T15:00:54.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.063+0000 7f94ee000640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f94d006b2f0 0x7f94d006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.063+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f94e00abc10 con 0x7f94f815f850 2024-01-16T15:00:54.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.064+0000 7f94f5e00640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f94d006b2f0 0x7f94d006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.065+0000 7f94f5e00640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f94d006b2f0 0x7f94d006d7b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94f8067b00 tx=0x7f94e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.068+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f94e0076b30 con 0x7f94f815f850 2024-01-16T15:00:54.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.276+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f94f808c7e0 con 0x7f94f815f850 2024-01-16T15:00:54.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.277+0000 7f94ee000640 1 -- 172.21.15.53:0/4186370787 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f94e007a150 con 0x7f94f815f850 2024-01-16T15:00:54.258 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.279+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f94d006b2f0 msgr2=0x7f94d006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.279+0000 7f94f7200640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f94d006b2f0 0x7f94d006d7b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94f8067b00 tx=0x7f94e8009040 comp rx=0 tx=0).stop 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.279+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 msgr2=0x7f94f815fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.279+0000 7f94f7200640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 0x7f94f815fcd0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f94e0037040 tx=0x7f94e0004730 comp rx=0 tx=0).stop 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 shutdown_connections 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f94d006b2f0 0x7f94d006d7b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94f80b79a0 0x7f94f80b7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94f815f850 0x7f94f815fcd0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 --2- 172.21.15.53:0/4186370787 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f94d80047c0 0x7f94f8150570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 >> 172.21.15.53:0/4186370787 conn(0x7f94f80d3090 msgr2=0x7f94f80c94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 shutdown_connections 2024-01-16T15:00:54.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.280+0000 7f94f7200640 1 -- 172.21.15.53:0/4186370787 wait complete. 2024-01-16T15:00:54.280 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:00:54.280 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:54.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:54.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.568+0000 7f67cbe00640 1 Processor -- start 2024-01-16T15:00:54.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.568+0000 7f67cbe00640 1 -- start start 2024-01-16T15:00:54.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67cbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f67cc160920 0x7f67cc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67cbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc0cfea0 0x7f67cc161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67cbe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc0d0a10 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67cbe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67cc0d0330 con 0x7f67cc160920 2024-01-16T15:00:54.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67cbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc0d1f50 con 0x7f67cc0cfea0 2024-01-16T15:00:54.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67caa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f67cc160920 0x7f67cc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67caa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f67cc160920 0x7f67cc0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45492/0 (socket says 172.21.15.53:45492) 2024-01-16T15:00:54.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.570+0000 7f67caa00640 1 -- 172.21.15.53:0/967416000 learned_addr learned my addr 172.21.15.53:0/967416000 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:54.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.571+0000 7f67caa00640 1 -- 172.21.15.53:0/967416000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc0cfea0 msgr2=0x7f67cc161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.571+0000 7f67caa00640 1 --2- 172.21.15.53:0/967416000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc0cfea0 0x7f67cc161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.571+0000 7f67caa00640 1 -- 172.21.15.53:0/967416000 >> v1:172.21.15.146:6789/0 conn(0x7f67cc0cf9f0 legacy=0x7f67cc161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:54.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.571+0000 7f67caa00640 1 -- 172.21.15.53:0/967416000 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0cf080 con 0x7f67cc160920 2024-01-16T15:00:54.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.572+0000 7f67caa00640 1 --2- 172.21.15.53:0/967416000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc160920 0x7f67cc0b3f30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009e50 tx=0x7f67bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abd78982f7ae8653 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.572+0000 7f67ca000640 1 -- 172.21.15.53:0/967416000 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc02ca70 con 0x7f67cc160920 2024-01-16T15:00:54.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.572+0000 7f67ca000640 1 -- 172.21.15.53:0/967416000 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc02cbd0 con 0x7f67cc160920 2024-01-16T15:00:54.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.572+0000 7f67ca000640 1 -- 172.21.15.53:0/967416000 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc0343f0 con 0x7f67cc160920 2024-01-16T15:00:54.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 -- 172.21.15.53:0/967416000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc160920 msgr2=0x7f67cc0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 --2- 172.21.15.53:0/967416000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc160920 0x7f67cc0b3f30 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009e50 tx=0x7f67bc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 -- 172.21.15.53:0/967416000 shutdown_connections 2024-01-16T15:00:54.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 --2- 172.21.15.53:0/967416000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc0cfea0 0x7f67cc161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 --2- 172.21.15.53:0/967416000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc160920 0x7f67cc0b3f30 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 -- 172.21.15.53:0/967416000 >> 172.21.15.53:0/967416000 conn(0x7f67cc0d3090 msgr2=0x7f67cc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 -- 172.21.15.53:0/967416000 shutdown_connections 2024-01-16T15:00:54.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.573+0000 7f67cbe00640 1 -- 172.21.15.53:0/967416000 wait complete. 2024-01-16T15:00:54.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.575+0000 7f67cbe00640 1 Processor -- start 2024-01-16T15:00:54.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.575+0000 7f67cbe00640 1 -- start start 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 0x7f67cc151910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc0cfea0 0x7f67cc0c8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc160920 0x7f67cc0c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc0cb020 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc0cb300 con 0x7f67cc0cfea0 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc0cb5e0 con 0x7f67cc160920 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67caa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 0x7f67cc151910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67caa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 0x7f67cc151910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45494/0 (socket says 172.21.15.53:45494) 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67caa00640 1 -- 172.21.15.53:0/4005958136 learned_addr learned my addr 172.21.15.53:0/4005958136 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67cb400640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc160920 0x7f67cc0c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.576+0000 7f67c1e00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc0cfea0 0x7f67cc0c8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67caa00640 1 -- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc160920 msgr2=0x7f67cc0c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67caa00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc160920 0x7f67cc0c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67caa00640 1 -- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc0cfea0 msgr2=0x7f67cc0c8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67caa00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc0cfea0 0x7f67cc0c8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67caa00640 1 -- 172.21.15.53:0/4005958136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67bc009b30 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67c1e00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc0cfea0 0x7f67cc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67cb400640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc160920 0x7f67cc0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:54.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67caa00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 0x7f67cc151910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009e50 tx=0x7f67bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc03e450 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc03e5f0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc03e9a0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.577+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc0cb8c0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.578+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc0cc0c0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.579+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc067450 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.579+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f67bc03eb00 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.580+0000 7f67c3400640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406b2c0 0x7f67a406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:54.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.580+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f67bc0acea0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.581+0000 7f67c1e00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406b2c0 0x7f67a406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:54.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.582+0000 7f67c1e00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406b2c0 0x7f67a406d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67cc0b79b0 tx=0x7f67b4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:54.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.585+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67bc077d10 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.780+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67cc08c7e0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.782+0000 7f67c3400640 1 -- 172.21.15.53:0/4005958136 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f67bc03edb0 con 0x7f67cc0cf9f0 2024-01-16T15:00:54.764 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:54.764 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406b2c0 msgr2=0x7f67a406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406b2c0 0x7f67a406d780 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f67cc0b79b0 tx=0x7f67b4008040 comp rx=0 tx=0).stop 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 msgr2=0x7f67cc151910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 0x7f67cc151910 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f67bc009e50 tx=0x7f67bc004a40 comp rx=0 tx=0).stop 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 shutdown_connections 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406b2c0 0x7f67a406d780 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc160920 0x7f67cc0c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc0cfea0 0x7f67cc0c8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 --2- 172.21.15.53:0/4005958136 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc0cf9f0 0x7f67cc151910 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.785+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 >> 172.21.15.53:0/4005958136 conn(0x7f67cc0d3090 msgr2=0x7f67cc06aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.786+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 shutdown_connections 2024-01-16T15:00:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:54.786+0000 7f67cbe00640 1 -- 172.21.15.53:0/4005958136 wait complete. 2024-01-16T15:00:54.786 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:54.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.035+0000 7f3766e00640 1 Processor -- start 2024-01-16T15:00:55.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.035+0000 7f3766e00640 1 -- start start 2024-01-16T15:00:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37600adf20 0x7f3760070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f37600ccb50 0x7f3760143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376015f4e0 con 0x7f376015bf10 2024-01-16T15:00:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f376015f970 con 0x7f37600ccb50 2024-01-16T15:00:55.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3760071130 con 0x7f37600adf20 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f37600ccb50 0x7f3760143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3765000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f376015bf10 0x7f37600cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36196/0 (socket says 172.21.15.53:36196) 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3765000640 1 -- 172.21.15.53:0/372837844 learned_addr learned my addr 172.21.15.53:0/372837844 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3765a00640 1 --2- 172.21.15.53:0/372837844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37600adf20 0x7f3760070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766400640 1 -- 172.21.15.53:0/372837844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37600adf20 msgr2=0x7f3760070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766400640 1 --2- 172.21.15.53:0/372837844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37600adf20 0x7f3760070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766400640 1 -- 172.21.15.53:0/372837844 >> v1:172.21.15.53:6789/0 conn(0x7f376015bf10 legacy=0x7f37600cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f375fe00640 1 -- 172.21.15.53:0/372837844 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804243911 0 0) 0x7f376015f4e0 con 0x7f376015bf10 2024-01-16T15:00:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.037+0000 7f3766400640 1 -- 172.21.15.53:0/372837844 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376015c390 con 0x7f37600ccb50 2024-01-16T15:00:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.038+0000 7f3766400640 1 --2- 172.21.15.53:0/372837844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600ccb50 0x7f3760143d30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f37580097a0 tx=0x7f375802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85dafcddc0b4c55f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.038+0000 7f375fe00640 1 -- 172.21.15.53:0/372837844 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375802e070 con 0x7f37600ccb50 2024-01-16T15:00:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.038+0000 7f375fe00640 1 -- 172.21.15.53:0/372837844 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37580315a0 con 0x7f37600ccb50 2024-01-16T15:00:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.038+0000 7f375fe00640 1 -- 172.21.15.53:0/372837844 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3758031930 con 0x7f37600ccb50 2024-01-16T15:00:55.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 -- 172.21.15.53:0/372837844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600ccb50 msgr2=0x7f3760143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 --2- 172.21.15.53:0/372837844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600ccb50 0x7f3760143d30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f37580097a0 tx=0x7f375802b840 comp rx=0 tx=0).stop 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 -- 172.21.15.53:0/372837844 shutdown_connections 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 --2- 172.21.15.53:0/372837844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600ccb50 0x7f3760143d30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 --2- 172.21.15.53:0/372837844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37600adf20 0x7f3760070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 -- 172.21.15.53:0/372837844 >> 172.21.15.53:0/372837844 conn(0x7f376006a7d0 msgr2=0x7f376006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.039+0000 7f3766e00640 1 -- 172.21.15.53:0/372837844 shutdown_connections 2024-01-16T15:00:55.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.040+0000 7f3766e00640 1 -- 172.21.15.53:0/372837844 wait complete. 2024-01-16T15:00:55.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.041+0000 7f3766e00640 1 Processor -- start 2024-01-16T15:00:55.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.041+0000 7f3766e00640 1 -- start start 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.042+0000 7f3766e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 0x7f37600acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.042+0000 7f3766e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37600ccb50 0x7f376013ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.042+0000 7f3766e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f376015bf10 0x7f376013f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.042+0000 7f3766e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376013ea10 con 0x7f37600adf20 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.042+0000 7f3766e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37600b1fc0 con 0x7f37600ccb50 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.042+0000 7f3766e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376013fa80 con 0x7f376015bf10 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 0x7f37600acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 0x7f37600acb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45522/0 (socket says 172.21.15.53:45522) 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 -- 172.21.15.53:0/3668953490 learned_addr learned my addr 172.21.15.53:0/3668953490 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3766400640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f376015bf10 0x7f376013f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765000640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37600ccb50 0x7f376013ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 -- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f376015bf10 msgr2=0x7f376013f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f376015bf10 0x7f376013f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 -- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37600ccb50 msgr2=0x7f376013ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37600ccb50 0x7f376013ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.043+0000 7f3765a00640 1 -- 172.21.15.53:0/3668953490 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3758009400 con 0x7f37600adf20 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.044+0000 7f3765a00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 0x7f37600acb40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3750007ef0 tx=0x7f375000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.044+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750014620 con 0x7f37600adf20 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.044+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3750014780 con 0x7f37600adf20 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.044+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f376013fd60 con 0x7f37600adf20 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.044+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3750014b30 con 0x7f37600adf20 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.044+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3760140320 con 0x7f37600adf20 2024-01-16T15:00:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.045+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3760067450 con 0x7f37600adf20 2024-01-16T15:00:55.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.047+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3750014c90 con 0x7f37600adf20 2024-01-16T15:00:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.047+0000 7f375d600640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f373806b2f0 0x7f373806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.048+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f375008a6d0 con 0x7f37600adf20 2024-01-16T15:00:55.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.048+0000 7f3765000640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f373806b2f0 0x7f373806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.049+0000 7f3765000640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f373806b2f0 0x7f373806d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f376015deb0 tx=0x7f3748007c00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.052+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3750055420 con 0x7f37600adf20 2024-01-16T15:00:55.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.245+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3760143610 con 0x7f37600adf20 2024-01-16T15:00:55.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.245+0000 7f375d600640 1 -- 172.21.15.53:0/3668953490 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3750058a40 con 0x7f37600adf20 2024-01-16T15:00:55.226 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f373806b2f0 msgr2=0x7f373806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f373806b2f0 0x7f373806d7b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f376015deb0 tx=0x7f3748007c00 comp rx=0 tx=0).stop 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 msgr2=0x7f37600acb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 0x7f37600acb40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3750007ef0 tx=0x7f375000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 shutdown_connections 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f373806b2f0 0x7f373806d7b0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f376015bf10 0x7f376013f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f37600ccb50 0x7f376013ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 --2- 172.21.15.53:0/3668953490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37600adf20 0x7f37600acb40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 >> 172.21.15.53:0/3668953490 conn(0x7f376006a7d0 msgr2=0x7f376006d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 shutdown_connections 2024-01-16T15:00:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.248+0000 7f3766e00640 1 -- 172.21.15.53:0/3668953490 wait complete. 2024-01-16T15:00:55.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.493+0000 7fdbe1000640 1 Processor -- start 2024-01-16T15:00:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.493+0000 7fdbe1000640 1 -- start start 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.494+0000 7fdbe1000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdbdc15d4b0 0x7fdbdc0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.494+0000 7fdbe1000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdbdc0b41e0 0x7fdbdc0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.494+0000 7fdbe1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbdc0d0320 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.494+0000 7fdbe1000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc0b2730 con 0x7fdbdc0b41e0 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.494+0000 7fdbe1000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbdc0d1ec0 con 0x7fdbdc15d4b0 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdbdc0b41e0 0x7fdbdc0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdbdc0b41e0 0x7fdbdc0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45528/0 (socket says 172.21.15.53:45528) 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdaa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdbdc0cf9a0 0x7fdbdc0cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36206/0 (socket says 172.21.15.53:36206) 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 -- 172.21.15.53:0/2146614906 learned_addr learned my addr 172.21.15.53:0/2146614906 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdb400640 1 --2- 172.21.15.53:0/2146614906 >> v2:172.21.15.146:3300/0 conn(0x7fdbdc15d4b0 0x7fdbdc0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 -- 172.21.15.53:0/2146614906 >> v2:172.21.15.146:3300/0 conn(0x7fdbdc15d4b0 msgr2=0x7fdbdc0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:55.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 --2- 172.21.15.53:0/2146614906 >> v2:172.21.15.146:3300/0 conn(0x7fdbdc15d4b0 0x7fdbdc0d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 -- 172.21.15.53:0/2146614906 >> v1:172.21.15.53:6789/0 conn(0x7fdbdc0cf9a0 legacy=0x7fdbdc0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.495+0000 7fdbdbe00640 1 -- 172.21.15.53:0/2146614906 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbdc0b29d0 con 0x7fdbdc0b41e0 2024-01-16T15:00:55.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.497+0000 7fdbdbe00640 1 --2- 172.21.15.53:0/2146614906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc0cfde0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc08b5b0 tx=0x7fdbd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42bbe4f270794f00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.497+0000 7fdbda000640 1 -- 172.21.15.53:0/2146614906 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4030860 con 0x7fdbdc0b41e0 2024-01-16T15:00:55.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.497+0000 7fdbda000640 1 -- 172.21.15.53:0/2146614906 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd40309c0 con 0x7fdbdc0b41e0 2024-01-16T15:00:55.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.497+0000 7fdbda000640 1 -- 172.21.15.53:0/2146614906 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd4030d50 con 0x7fdbdc0b41e0 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.497+0000 7fdbe1000640 1 -- 172.21.15.53:0/2146614906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0b41e0 msgr2=0x7fdbdc0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.497+0000 7fdbe1000640 1 --2- 172.21.15.53:0/2146614906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc0cfde0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc08b5b0 tx=0x7fdbd402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.498+0000 7fdbe1000640 1 -- 172.21.15.53:0/2146614906 shutdown_connections 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.498+0000 7fdbe1000640 1 --2- 172.21.15.53:0/2146614906 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc0cfde0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.498+0000 7fdbe1000640 1 --2- 172.21.15.53:0/2146614906 >> v2:172.21.15.146:3300/0 conn(0x7fdbdc15d4b0 0x7fdbdc0d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.498+0000 7fdbe1000640 1 -- 172.21.15.53:0/2146614906 >> 172.21.15.53:0/2146614906 conn(0x7fdbdc0d3000 msgr2=0x7fdbdc0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.498+0000 7fdbe1000640 1 -- 172.21.15.53:0/2146614906 shutdown_connections 2024-01-16T15:00:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.498+0000 7fdbe1000640 1 -- 172.21.15.53:0/2146614906 wait complete. 2024-01-16T15:00:55.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.500+0000 7fdbe1000640 1 Processor -- start 2024-01-16T15:00:55.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.500+0000 7fdbe1000640 1 -- start start 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc1f5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 0x7fdbdc09e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbdc15d4b0 0x7fdbdc09eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc09deb0 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc09d2a0 con 0x7fdbdc0b41e0 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc09d580 con 0x7fdbdc15d4b0 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 0x7fdbdc09e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 0x7fdbdc09e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45544/0 (socket says 172.21.15.53:45544) 2024-01-16T15:00:55.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 -- 172.21.15.53:0/355581032 learned_addr learned my addr 172.21.15.53:0/355581032 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdb400640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc1f5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdbe00640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbdc15d4b0 0x7fdbdc09eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 -- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbdc15d4b0 msgr2=0x7fdbdc09eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbdc15d4b0 0x7fdbdc09eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 -- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbdc0b41e0 msgr2=0x7fdbdc1f5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc1f5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 -- 172.21.15.53:0/355581032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd40094d0 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdb400640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc1f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdbe00640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbdc15d4b0 0x7fdbdc09eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbdaa00640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 0x7fdbdc09e9b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c800 tx=0x7fdbc400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400a640 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbc400a7a0 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbc400ab50 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc15da10 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.502+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc09db40 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.503+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc067450 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.504+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fdbc400acb0 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.507+0000 7fdbd3400640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdbb406b2c0 0x7fdbb406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.507+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdbc4089400 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.507+0000 7fdbdb400640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdbb406b2c0 0x7fdbb406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.509+0000 7fdbdb400640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdbb406b2c0 0x7fdbb406d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009a20 tx=0x7fdbcc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.511+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdbc4057d30 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.707+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdbdc073e10 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.708+0000 7fdbd3400640 1 -- 172.21.15.53:0/355581032 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdbc40018c0 con 0x7fdbdc0cf9a0 2024-01-16T15:00:55.689 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdbb406b2c0 msgr2=0x7fdbb406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdbb406b2c0 0x7fdbb406d780 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc009a20 tx=0x7fdbcc008040 comp rx=0 tx=0).stop 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 msgr2=0x7fdbdc09e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 0x7fdbdc09e9b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fdbc400c800 tx=0x7fdbc400ccc0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 shutdown_connections 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdbb406b2c0 0x7fdbb406d780 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdbdc15d4b0 0x7fdbdc09eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdbdc0cf9a0 0x7fdbdc09e9b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 --2- 172.21.15.53:0/355581032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdbdc0b41e0 0x7fdbdc1f5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 >> 172.21.15.53:0/355581032 conn(0x7fdbdc0d3000 msgr2=0x7fdbdc06ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 shutdown_connections 2024-01-16T15:00:55.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.712+0000 7fdbe1000640 1 -- 172.21.15.53:0/355581032 wait complete. 2024-01-16T15:00:55.712 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:55.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.964+0000 7feecee00640 1 Processor -- start 2024-01-16T15:00:55.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.964+0000 7feecee00640 1 -- start start 2024-01-16T15:00:55.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feecee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feec815e880 0x7feec806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feecee00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80cfa70 con 0x7feec8070490 2024-01-16T15:00:55.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feecee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80cff00 con 0x7feec809a170 2024-01-16T15:00:55.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feecee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec806fc70 con 0x7feec815e880 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feec815e880 0x7feec806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7feec8070490 0x7feec8099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36216/0 (socket says 172.21.15.53:36216) 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7feec815e880 0x7feec806ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45546/0 (socket says 172.21.15.53:45546) 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7400640 1 -- 172.21.15.53:0/1148135005 learned_addr learned my addr 172.21.15.53:0/1148135005 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7e00640 1 -- 172.21.15.53:0/1148135005 >> v1:172.21.15.146:6789/0 conn(0x7feec809a170 legacy=0x7feec806e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7e00640 1 -- 172.21.15.53:0/1148135005 >> v1:172.21.15.53:6789/0 conn(0x7feec8070490 legacy=0x7feec8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.966+0000 7feec7e00640 1 -- 172.21.15.53:0/1148135005 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80cf420 con 0x7feec815e880 2024-01-16T15:00:55.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.969+0000 7feec7e00640 1 --2- 172.21.15.53:0/1148135005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec815e880 0x7feec806ff50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7feeb80097a0 tx=0x7feeb802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=205e450e98970631 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.969+0000 7feec6a00640 1 -- 172.21.15.53:0/1148135005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb802f810 con 0x7feec815e880 2024-01-16T15:00:55.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.969+0000 7feec6a00640 1 -- 172.21.15.53:0/1148135005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb802f970 con 0x7feec815e880 2024-01-16T15:00:55.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.969+0000 7feec6a00640 1 -- 172.21.15.53:0/1148135005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb802fd00 con 0x7feec815e880 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 -- 172.21.15.53:0/1148135005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec815e880 msgr2=0x7feec806ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 --2- 172.21.15.53:0/1148135005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec815e880 0x7feec806ff50 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7feeb80097a0 tx=0x7feeb802b9f0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 -- 172.21.15.53:0/1148135005 shutdown_connections 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 --2- 172.21.15.53:0/1148135005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec815e880 0x7feec806ff50 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 -- 172.21.15.53:0/1148135005 >> 172.21.15.53:0/1148135005 conn(0x7feec80b2d00 msgr2=0x7feec80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 -- 172.21.15.53:0/1148135005 shutdown_connections 2024-01-16T15:00:55.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.970+0000 7feecee00640 1 -- 172.21.15.53:0/1148135005 wait complete. 2024-01-16T15:00:55.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.972+0000 7feecee00640 1 Processor -- start 2024-01-16T15:00:55.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.972+0000 7feecee00640 1 -- start start 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feec8070490 0x7feec80b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 0x7feec8157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feec815e880 0x7feec81586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec81f2c70 con 0x7feec809a170 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec81f66b0 con 0x7feec8070490 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec81f6990 con 0x7feec815e880 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 0x7feec8157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 0x7feec8157780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45552/0 (socket says 172.21.15.53:45552) 2024-01-16T15:00:55.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 -- 172.21.15.53:0/1858025286 learned_addr learned my addr 172.21.15.53:0/1858025286 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecce00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feec815e880 0x7feec81586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7e00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feec8070490 0x7feec80b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 -- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feec815e880 msgr2=0x7feec81586f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feec815e880 0x7feec81586f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 -- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feec8070490 msgr2=0x7feec80b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feec8070490 0x7feec80b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 -- 172.21.15.53:0/1858025286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb8009400 con 0x7feec809a170 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7e00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feec8070490 0x7feec80b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecce00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feec815e880 0x7feec81586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feec7400640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 0x7feec8157780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c4a0 tx=0x7feeb000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0005ce0 con 0x7feec809a170 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb0005e40 con 0x7feec809a170 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0007270 con 0x7feec809a170 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec81f6c70 con 0x7feec809a170 2024-01-16T15:00:55.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.973+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec80b47b0 con 0x7feec809a170 2024-01-16T15:00:55.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.975+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec81f2de0 con 0x7feec809a170 2024-01-16T15:00:55.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.976+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7feeb0007490 con 0x7feec809a170 2024-01-16T15:00:55.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.976+0000 7feebfe00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feea006b2c0 0x7feea006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:55.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.977+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feeb0089100 con 0x7feec809a170 2024-01-16T15:00:55.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.977+0000 7feec7e00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feea006b2c0 0x7feea006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:55.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.978+0000 7feec7e00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feea006b2c0 0x7feea006d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feeb8036000 tx=0x7feeb8039040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:55.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:55.981+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feeb0053f00 con 0x7feec809a170 2024-01-16T15:00:56.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.175+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7feec80675d0 con 0x7feec809a170 2024-01-16T15:00:56.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.176+0000 7feebfe00640 1 -- 172.21.15.53:0/1858025286 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7feeb00170a0 con 0x7feec809a170 2024-01-16T15:00:56.156 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:56.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feea006b2c0 msgr2=0x7feea006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feea006b2c0 0x7feea006d780 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feeb8036000 tx=0x7feeb8039040 comp rx=0 tx=0).stop 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 msgr2=0x7feec8157780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 0x7feec8157780 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7feeb000c4a0 tx=0x7feeb000c960 comp rx=0 tx=0).stop 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 shutdown_connections 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feea006b2c0 0x7feea006d780 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feec815e880 0x7feec81586f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feec809a170 0x7feec8157780 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 --2- 172.21.15.53:0/1858025286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feec8070490 0x7feec80b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 >> 172.21.15.53:0/1858025286 conn(0x7feec80b2d00 msgr2=0x7feec815db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 shutdown_connections 2024-01-16T15:00:56.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.178+0000 7feecee00640 1 -- 172.21.15.53:0/1858025286 wait complete. 2024-01-16T15:00:56.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:56.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.416+0000 7f296b200640 1 Processor -- start 2024-01-16T15:00:56.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.416+0000 7f296b200640 1 -- start start 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f296b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f29640cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f296b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f296406d160 0x7f296406d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f296b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2964070e50 con 0x7f2964143c40 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f296b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29640723c0 con 0x7f296406d160 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f296b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2964144020 con 0x7f29640cc500 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f2969e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2964143c40 0x7f2964071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36222/0 (socket says 172.21.15.53:36222) 2024-01-16T15:00:56.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f2969e00640 1 -- 172.21.15.53:0/1330108659 learned_addr learned my addr 172.21.15.53:0/1330108659 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:56.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.418+0000 7f296a800640 1 --2- 172.21.15.53:0/1330108659 >> v2:172.21.15.53:3300/0 conn(0x7f296406d160 0x7f296406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f2969400640 1 --2- 172.21.15.53:0/1330108659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f29640cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f2968a00640 1 -- 172.21.15.53:0/1330108659 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854982145 0 0) 0x7f2964070e50 con 0x7f2964143c40 2024-01-16T15:00:56.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f2968a00640 1 -- 172.21.15.53:0/1330108659 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2944003520 con 0x7f2964143c40 2024-01-16T15:00:56.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f296a800640 1 -- 172.21.15.53:0/1330108659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 msgr2=0x7f29640cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:56.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f296a800640 1 --2- 172.21.15.53:0/1330108659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f29640cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f296a800640 1 -- 172.21.15.53:0/1330108659 >> v1:172.21.15.53:6789/0 conn(0x7f2964143c40 legacy=0x7f2964071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.419+0000 7f296a800640 1 -- 172.21.15.53:0/1330108659 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29640ccea0 con 0x7f296406d160 2024-01-16T15:00:56.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.420+0000 7f296a800640 1 --2- 172.21.15.53:0/1330108659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f296406d590 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f295c009d90 tx=0x7f295c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8865ddb28de6c48d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:56.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.420+0000 7f2968a00640 1 -- 172.21.15.53:0/1330108659 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c02cbc0 con 0x7f296406d160 2024-01-16T15:00:56.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f2968a00640 1 -- 172.21.15.53:0/1330108659 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f295c02cd20 con 0x7f296406d160 2024-01-16T15:00:56.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f2968a00640 1 -- 172.21.15.53:0/1330108659 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f295c034660 con 0x7f296406d160 2024-01-16T15:00:56.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 -- 172.21.15.53:0/1330108659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 msgr2=0x7f296406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 --2- 172.21.15.53:0/1330108659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f296406d590 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f295c009d90 tx=0x7f295c02be10 comp rx=0 tx=0).stop 2024-01-16T15:00:56.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 -- 172.21.15.53:0/1330108659 shutdown_connections 2024-01-16T15:00:56.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 --2- 172.21.15.53:0/1330108659 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f296406d590 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 --2- 172.21.15.53:0/1330108659 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f29640cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 -- 172.21.15.53:0/1330108659 >> 172.21.15.53:0/1330108659 conn(0x7f296406a800 msgr2=0x7f296406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:56.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 -- 172.21.15.53:0/1330108659 shutdown_connections 2024-01-16T15:00:56.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.421+0000 7f296b200640 1 -- 172.21.15.53:0/1330108659 wait complete. 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.423+0000 7f296b200640 1 Processor -- start 2024-01-16T15:00:56.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.423+0000 7f296b200640 1 -- start start 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.424+0000 7f296b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f2964159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.424+0000 7f296b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f296413e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.424+0000 7f296b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2964143c40 0x7f296413e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.424+0000 7f296b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296413ab50 con 0x7f296406d160 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.424+0000 7f296b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2964139f40 con 0x7f2964143c40 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.424+0000 7f296b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f296413a220 con 0x7f29640cc500 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f2964159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f2964159f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45566/0 (socket says 172.21.15.53:45566) 2024-01-16T15:00:56.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 -- 172.21.15.53:0/3229855676 learned_addr learned my addr 172.21.15.53:0/3229855676 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969400640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f296413e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f296a800640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2964143c40 0x7f296413e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 -- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 msgr2=0x7f296413e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f296413e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 -- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2964143c40 msgr2=0x7f296413e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2964143c40 0x7f296413e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 -- 172.21.15.53:0/3229855676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c009a70 con 0x7f296406d160 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f296a800640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2964143c40 0x7f296413e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.425+0000 7f2969e00640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f2964159f10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f295400c410 tx=0x7f295400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.426+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2954005ce0 con 0x7f296406d160 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.426+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2954005e40 con 0x7f296406d160 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.426+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29540072f0 con 0x7f296406d160 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.426+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296413a500 con 0x7f296406d160 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.426+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296415a450 con 0x7f296406d160 2024-01-16T15:00:56.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.427+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296415a610 con 0x7f296406d160 2024-01-16T15:00:56.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.428+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2954007510 con 0x7f296406d160 2024-01-16T15:00:56.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.429+0000 7f2962000640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f293c06b2c0 0x7f293c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:56.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.429+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2954089370 con 0x7f296406d160 2024-01-16T15:00:56.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.430+0000 7f2969400640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f293c06b2c0 0x7f293c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:56.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.431+0000 7f2969400640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f293c06b2c0 0x7f293c06d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f29641391a0 tx=0x7f294c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:56.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.433+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2954054180 con 0x7f296406d160 2024-01-16T15:00:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.639+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2964067450 con 0x7f296406d160 2024-01-16T15:00:56.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.640+0000 7f2962000640 1 -- 172.21.15.53:0/3229855676 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f29540577a0 con 0x7f296406d160 2024-01-16T15:00:56.620 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f293c06b2c0 msgr2=0x7f293c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f293c06b2c0 0x7f293c06d780 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f29641391a0 tx=0x7f294c008040 comp rx=0 tx=0).stop 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 msgr2=0x7f2964159f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f2964159f10 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f295400c410 tx=0x7f295400c8d0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 shutdown_connections 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f293c06b2c0 0x7f293c06d780 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2964143c40 0x7f296413e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29640cc500 0x7f296413e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 --2- 172.21.15.53:0/3229855676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f296406d160 0x7f2964159f10 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 >> 172.21.15.53:0/3229855676 conn(0x7f296406a800 msgr2=0x7f296406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 shutdown_connections 2024-01-16T15:00:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:56.643+0000 7f296b200640 1 -- 172.21.15.53:0/3229855676 wait complete. 2024-01-16T15:00:56.643 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:00:56.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-01-16T15:00:56.735 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:00:56.735 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:00:56.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:00:56.735 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:00:56.788 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:00:56.788 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:56.846 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:00:56.846 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:56.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:57.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.145+0000 7f098d000640 1 Processor -- start 2024-01-16T15:00:57.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.145+0000 7f098d000640 1 -- start start 2024-01-16T15:00:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.147+0000 7f098d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f09880cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.147+0000 7f098d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098815deb0 con 0x7f09881618f0 2024-01-16T15:00:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.147+0000 7f098d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f098815e190 con 0x7f0988071200 2024-01-16T15:00:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.147+0000 7f098d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098815ea30 con 0x7f0988157c30 2024-01-16T15:00:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.148+0000 7f0985400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f09881618f0 0x7f0988070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36236/0 (socket says 172.21.15.53:36236) 2024-01-16T15:00:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.148+0000 7f0985400640 1 -- 172.21.15.53:0/1717526283 learned_addr learned my addr 172.21.15.53:0/1717526283 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.148+0000 7f0985e00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f09880cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.148+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123045831 0 0) 0x7f098815deb0 con 0x7f09881618f0 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.148+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968003520 con 0x7f09881618f0 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2760036237 0 0) 0x7f0968003520 con 0x7f09881618f0 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 msgr2=0x7f09880cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f09880cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 >> v1:172.21.15.146:6789/0 conn(0x7f0988071200 legacy=0x7f098806dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f098815e470 con 0x7f09881618f0 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0978001a10 con 0x7f09881618f0 2024-01-16T15:00:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 >> v1:172.21.15.53:6789/0 conn(0x7f09881618f0 legacy=0x7f0988070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09680039a0 0x7f0968003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09680047c0 0x7f0968004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f0968005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098815ea30 con 0x7f09680047c0 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f096800ba60 con 0x7f09680039a0 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f096800bc20 con 0x7f0988157c30 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0985400640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09680047c0 0x7f0968004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0986800640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f0968005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0985e00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09680039a0 0x7f0968003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.149+0000 7f0985400640 1 -- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 msgr2=0x7f0968005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.150+0000 7f0985400640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f0968005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.150+0000 7f0985400640 1 -- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09680039a0 msgr2=0x7f0968003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.150+0000 7f0985400640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09680039a0 0x7f0968003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.150+0000 7f0985400640 1 -- 172.21.15.53:0/1717526283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0968003520 con 0x7f09680047c0 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.150+0000 7f0985e00640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09680039a0 0x7f0968003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.150+0000 7f0986800640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0988157c30 0x7f0968005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.151+0000 7f0985400640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09680047c0 0x7f0968004c40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f09880cbb90 tx=0x7f09680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb38a8531c862fdf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.151+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978034630 con 0x7f09680047c0 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.151+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f097802f070 con 0x7f09680047c0 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.151+0000 7f0984a00640 1 -- 172.21.15.53:0/1717526283 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978034a10 con 0x7f09680047c0 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f098d000640 1 -- 172.21.15.53:0/1717526283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09680047c0 msgr2=0x7f0968004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f098d000640 1 --2- 172.21.15.53:0/1717526283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09680047c0 0x7f0968004c40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f09880cbb90 tx=0x7f09680057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f0986800640 1 -- 172.21.15.53:0/1717526283 reap_dead start 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f098d000640 1 -- 172.21.15.53:0/1717526283 shutdown_connections 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f098d000640 1 -- 172.21.15.53:0/1717526283 >> 172.21.15.53:0/1717526283 conn(0x7f09880cf700 msgr2=0x7f09880cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f098d000640 1 -- 172.21.15.53:0/1717526283 shutdown_connections 2024-01-16T15:00:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.152+0000 7f098d000640 1 -- 172.21.15.53:0/1717526283 wait complete. 2024-01-16T15:00:57.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.154+0000 7f098d000640 1 Processor -- start 2024-01-16T15:00:57.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.154+0000 7f098d000640 1 -- start start 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f098d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09680047c0 0x7f0988156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f098d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 0x7f0988158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f098d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09880b3400 0x7f09880b3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f098d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0988158830 con 0x7f09881e8190 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f098d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09881e8610 con 0x7f09880b3400 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f098d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098815b590 con 0x7f09680047c0 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f0985400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 0x7f0988158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f0985400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 0x7f0988158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45596/0 (socket says 172.21.15.53:45596) 2024-01-16T15:00:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f0985400640 1 -- 172.21.15.53:0/1908253111 learned_addr learned my addr 172.21.15.53:0/1908253111 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:57.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f0986800640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09880b3400 0x7f09880b3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.155+0000 7f0985400640 1 -- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09680047c0 msgr2=0x7f0988156e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f0985400640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09680047c0 0x7f0988156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f0985400640 1 -- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09880b3400 msgr2=0x7f09880b3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f0985400640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09880b3400 0x7f09880b3880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f0985400640 1 -- 172.21.15.53:0/1908253111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09881585b0 con 0x7f09881e8190 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f0985400640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 0x7f0988158070 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0978001a00 tx=0x7f0978001cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09780356c0 con 0x7f09881e8190 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0978035820 con 0x7f09881e8190 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0978035b90 con 0x7f09881e8190 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f098815a950 con 0x7f09881e8190 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.156+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f098815aed0 con 0x7f09881e8190 2024-01-16T15:00:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.158+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09880675d0 con 0x7f09881e8190 2024-01-16T15:00:57.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.158+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0978031030 con 0x7f09881e8190 2024-01-16T15:00:57.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.160+0000 7f097e000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f096006b2f0 0x7f096006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.160+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f09780abfe0 con 0x7f09881e8190 2024-01-16T15:00:57.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.161+0000 7f0985e00640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f096006b2f0 0x7f096006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.162+0000 7f0985e00640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f096006b2f0 0x7f096006d7b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09700099e0 tx=0x7f0970008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:57.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.164+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0978076ec0 con 0x7f09881e8190 2024-01-16T15:00:57.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.358+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f098815b6d0 con 0x7f09881e8190 2024-01-16T15:00:57.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.360+0000 7f097e000640 1 -- 172.21.15.53:0/1908253111 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f098815b6d0 con 0x7f09881e8190 2024-01-16T15:00:57.341 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:57.341 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f096006b2f0 msgr2=0x7f096006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f096006b2f0 0x7f096006d7b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f09700099e0 tx=0x7f0970008040 comp rx=0 tx=0).stop 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 msgr2=0x7f0988158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 0x7f0988158070 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0978001a00 tx=0x7f0978001cd0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 shutdown_connections 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f096006b2f0 0x7f096006d7b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09880b3400 0x7f09880b3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09881e8190 0x7f0988158070 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 --2- 172.21.15.53:0/1908253111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09680047c0 0x7f0988156e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 >> 172.21.15.53:0/1908253111 conn(0x7f09880cf700 msgr2=0x7f0988160410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 shutdown_connections 2024-01-16T15:00:57.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.363+0000 7f098d000640 1 -- 172.21.15.53:0/1908253111 wait complete. 2024-01-16T15:00:57.361 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:57.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:57.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.609+0000 7fa961600640 1 Processor -- start 2024-01-16T15:00:57.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.609+0000 7fa961600640 1 -- start start 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa961600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa95c143ae0 0x7fa95c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa961600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 0x7fa95c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa961600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa95c0cfc30 con 0x7fa95c15e1e0 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa961600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa95c0b1ee0 con 0x7fa95c143ae0 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa961600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b2780 con 0x7fa95c0cf7d0 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa95b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa95c15e1e0 0x7fa95c15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36238/0 (socket says 172.21.15.53:36238) 2024-01-16T15:00:57.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa95b400640 1 -- 172.21.15.53:0/2048287808 learned_addr learned my addr 172.21.15.53:0/2048287808 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:57.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa95a000640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 0x7fa95c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.611+0000 7fa95aa00640 1 --2- 172.21.15.53:0/2048287808 >> v2:172.21.15.146:3300/0 conn(0x7fa95c143ae0 0x7fa95c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637342858 0 0) 0x7fa95c0cfc30 con 0x7fa95c15e1e0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93c003420 con 0x7fa95c15e1e0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1458691244 0 0) 0x7fa93c003420 con 0x7fa95c15e1e0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 msgr2=0x7fa95c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 0x7fa95c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 >> v2:172.21.15.146:3300/0 conn(0x7fa95c143ae0 msgr2=0x7fa95c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 --2- 172.21.15.53:0/2048287808 >> v2:172.21.15.146:3300/0 conn(0x7fa95c143ae0 0x7fa95c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c15e990 con 0x7fa95c15e1e0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fa954001670 con 0x7fa95c15e1e0 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 >> v1:172.21.15.53:6789/0 conn(0x7fa95c15e1e0 legacy=0x7fa95c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 0x7fa93c004a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa93c003660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa93c003ba0 0x7fa93c004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b1ee0 con 0x7fa93c005020 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0b2780 con 0x7fa93c003ba0 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.612+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa93c00fdb0 con 0x7fa95c0cf7d0 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95a000640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa93c003660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95b400640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa93c003ba0 0x7fa93c004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95aa00640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 0x7fa93c004a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95a000640 1 -- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 msgr2=0x7fa93c004a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95a000640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c0cf7d0 0x7fa93c004a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95a000640 1 -- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa93c003ba0 msgr2=0x7fa93c004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95a000640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa93c003ba0 0x7fa93c004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95a000640 1 -- 172.21.15.53:0/2048287808 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c003420 con 0x7fa93c005020 2024-01-16T15:00:57.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.613+0000 7fa95b400640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa93c003ba0 0x7fa93c004020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:57.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.615+0000 7fa95a000640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa93c003660 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa944009ac0 tx=0x7fa93c0046e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae89f08289f50d72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:57.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.615+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94402c6b0 con 0x7fa93c005020 2024-01-16T15:00:57.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.615+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94402c810 con 0x7fa93c005020 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.615+0000 7fa959600640 1 -- 172.21.15.53:0/2048287808 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94402cba0 con 0x7fa93c005020 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa961600640 1 -- 172.21.15.53:0/2048287808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 msgr2=0x7fa93c003660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa961600640 1 --2- 172.21.15.53:0/2048287808 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa93c003660 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa944009ac0 tx=0x7fa93c0046e0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa95b400640 1 -- 172.21.15.53:0/2048287808 reap_dead start 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa961600640 1 -- 172.21.15.53:0/2048287808 shutdown_connections 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa961600640 1 -- 172.21.15.53:0/2048287808 >> 172.21.15.53:0/2048287808 conn(0x7fa95c06a800 msgr2=0x7fa95c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa961600640 1 -- 172.21.15.53:0/2048287808 shutdown_connections 2024-01-16T15:00:57.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.616+0000 7fa961600640 1 -- 172.21.15.53:0/2048287808 wait complete. 2024-01-16T15:00:57.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.618+0000 7fa961600640 1 Processor -- start 2024-01-16T15:00:57.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.618+0000 7fa961600640 1 -- start start 2024-01-16T15:00:57.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.619+0000 7fa961600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa95c0cda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.619+0000 7fa961600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa95c137180 0x7fa95c13a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.619+0000 7fa961600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c13a5c0 0x7fa95c1375f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.619+0000 7fa961600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c15d5e0 con 0x7fa93c005020 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.619+0000 7fa961600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0cdfb0 con 0x7fa95c137180 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.619+0000 7fa961600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c0cc9d0 con 0x7fa95c13a5c0 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa95c0cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa95c0cda70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45618/0 (socket says 172.21.15.53:45618) 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 -- 172.21.15.53:0/1815015378 learned_addr learned my addr 172.21.15.53:0/1815015378 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95b400640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c13a5c0 0x7fa95c1375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95a000640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa95c137180 0x7fa95c13a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 -- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c13a5c0 msgr2=0x7fa95c1375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c13a5c0 0x7fa95c1375f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 -- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa95c137180 msgr2=0x7fa95c13a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa95c137180 0x7fa95c13a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 -- 172.21.15.53:0/1815015378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa95c0cccb0 con 0x7fa93c005020 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.620+0000 7fa95aa00640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa95c0cda70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa94c007f30 tx=0x7fa94c011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:57.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.621+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c00e030 con 0x7fa93c005020 2024-01-16T15:00:57.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.621+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa94c016470 con 0x7fa93c005020 2024-01-16T15:00:57.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.621+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa94c016820 con 0x7fa93c005020 2024-01-16T15:00:57.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.621+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c0ccf90 con 0x7fa93c005020 2024-01-16T15:00:57.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.621+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c0cd550 con 0x7fa93c005020 2024-01-16T15:00:57.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.622+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c0675d0 con 0x7fa93c005020 2024-01-16T15:00:57.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.623+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa94c016980 con 0x7fa93c005020 2024-01-16T15:00:57.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.624+0000 7fa952a00640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa93406b2c0 0x7fa93406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:57.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.624+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa94c00f070 con 0x7fa93c005020 2024-01-16T15:00:57.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.625+0000 7fa95a000640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa93406b2c0 0x7fa93406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:57.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.626+0000 7fa95a000640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa93406b2c0 0x7fa93406d780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa944036000 tx=0x7fa944030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:57.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.628+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa94c08ce60 con 0x7fa93c005020 2024-01-16T15:00:57.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.822+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa95c141310 con 0x7fa93c005020 2024-01-16T15:00:57.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.823+0000 7fa952a00640 1 -- 172.21.15.53:0/1815015378 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa94c055aa0 con 0x7fa93c005020 2024-01-16T15:00:57.804 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.825+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa93406b2c0 msgr2=0x7fa93406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.825+0000 7fa961600640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa93406b2c0 0x7fa93406d780 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa944036000 tx=0x7fa944030000 comp rx=0 tx=0).stop 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.825+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 msgr2=0x7fa95c0cda70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.825+0000 7fa961600640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa95c0cda70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa94c007f30 tx=0x7fa94c011660 comp rx=0 tx=0).stop 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 shutdown_connections 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa93406b2c0 0x7fa93406d780 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa95c13a5c0 0x7fa95c1375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa95c137180 0x7fa95c13a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 --2- 172.21.15.53:0/1815015378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa93c005020 0x7fa95c0cda70 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:57.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 >> 172.21.15.53:0/1815015378 conn(0x7fa95c06a800 msgr2=0x7fa95c06c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:57.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 shutdown_connections 2024-01-16T15:00:57.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:57.826+0000 7fa961600640 1 -- 172.21.15.53:0/1815015378 wait complete. 2024-01-16T15:00:57.827 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:00:58.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.062+0000 7f3bccc00640 1 Processor -- start 2024-01-16T15:00:58.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.062+0000 7f3bccc00640 1 -- start start 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bccc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3bc80cf9a0 0x7f3bc8166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bccc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc815c5f0 con 0x7f3bc813e360 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bccc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bc806c470 con 0x7f3bc813df80 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bccc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bc806c750 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bc6800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3bc80cf9a0 0x7f3bc8166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bbfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3bc813e360 0x7f3bc80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36254/0 (socket says 172.21.15.53:36254) 2024-01-16T15:00:58.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.064+0000 7f3bbfe00640 1 -- 172.21.15.53:0/3709620602 learned_addr learned my addr 172.21.15.53:0/3709620602 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:58.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.065+0000 7f3bc6800640 1 -- 172.21.15.53:0/3709620602 >> v1:172.21.15.146:6789/0 conn(0x7f3bc813df80 legacy=0x7f3bc815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:58.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.065+0000 7f3bc6800640 1 -- 172.21.15.53:0/3709620602 >> v1:172.21.15.53:6789/0 conn(0x7f3bc813e360 legacy=0x7f3bc80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.065+0000 7f3bc6800640 1 -- 172.21.15.53:0/3709620602 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc806ca30 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.065+0000 7f3bc5400640 1 -- 172.21.15.53:0/3709620602 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689414179 0 0) 0x7f3bc815c5f0 con 0x7f3bc813e360 2024-01-16T15:00:58.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.066+0000 7f3bc6800640 1 --2- 172.21.15.53:0/3709620602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc8166030 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009d90 tx=0x7f3bc002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb6841c886089a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.066+0000 7f3bc5400640 1 -- 172.21.15.53:0/3709620602 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc002cc60 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.066+0000 7f3bc5400640 1 -- 172.21.15.53:0/3709620602 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc002cdc0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.066+0000 7f3bc5400640 1 -- 172.21.15.53:0/3709620602 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0034660 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 -- 172.21.15.53:0/3709620602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 msgr2=0x7f3bc8166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 --2- 172.21.15.53:0/3709620602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc8166030 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3bc0009d90 tx=0x7f3bc002beb0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 -- 172.21.15.53:0/3709620602 shutdown_connections 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 --2- 172.21.15.53:0/3709620602 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc8166030 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 -- 172.21.15.53:0/3709620602 >> 172.21.15.53:0/3709620602 conn(0x7f3bc806a800 msgr2=0x7f3bc806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 -- 172.21.15.53:0/3709620602 shutdown_connections 2024-01-16T15:00:58.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.067+0000 7f3bccc00640 1 -- 172.21.15.53:0/3709620602 wait complete. 2024-01-16T15:00:58.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.069+0000 7f3bccc00640 1 Processor -- start 2024-01-16T15:00:58.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.069+0000 7f3bccc00640 1 -- start start 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bccc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc80add20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bccc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bc813df80 0x7f3bc81e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bccc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bc813e360 0x7f3bc80b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bccc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc81e7ca0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bccc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc81e7020 con 0x7f3bc813df80 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bccc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc81e7300 con 0x7f3bc813e360 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc80add20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc80add20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45630/0 (socket says 172.21.15.53:45630) 2024-01-16T15:00:58.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 -- 172.21.15.53:0/1994426388 learned_addr learned my addr 172.21.15.53:0/1994426388 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:58.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 -- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bc813e360 msgr2=0x7f3bc80b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bc813e360 0x7f3bc80b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 -- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bc813df80 msgr2=0x7f3bc81e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bc813df80 0x7f3bc81e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.070+0000 7f3bc5e00640 1 -- 172.21.15.53:0/1994426388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc0009a70 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.071+0000 7f3bc5e00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc80add20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b3f0 tx=0x7f3bb800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.071+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8011830 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.071+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb8011990 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.071+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc81e75e0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.071+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bb8011d40 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.071+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc80ae260 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.073+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc81e7de0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.073+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f3bb801a400 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.074+0000 7f3bbe000640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ba006b310 0x7f3ba006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.074+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3bb8089a40 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.075+0000 7f3bbfe00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ba006b310 0x7f3ba006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.076+0000 7f3bbfe00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ba006b310 0x7f3ba006d7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0007fd0 tx=0x7f3bb0007920 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.079+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3bc81e7de0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.274+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3bc81e7de0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.275+0000 7f3bbe000640 1 -- 172.21.15.53:0/1994426388 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3bc81e7de0 con 0x7f3bc80cf9a0 2024-01-16T15:00:58.255 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:00:58.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ba006b310 msgr2=0x7f3ba006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ba006b310 0x7f3ba006d7d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f3bb0007fd0 tx=0x7f3bb0007920 comp rx=0 tx=0).stop 2024-01-16T15:00:58.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 msgr2=0x7f3bc80add20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc80add20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3bb800b3f0 tx=0x7f3bb800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 shutdown_connections 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ba006b310 0x7f3ba006d7d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3bc813e360 0x7f3bc80b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3bc813df80 0x7f3bc81e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 --2- 172.21.15.53:0/1994426388 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3bc80cf9a0 0x7f3bc80add20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 >> 172.21.15.53:0/1994426388 conn(0x7f3bc806a800 msgr2=0x7f3bc80af4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 shutdown_connections 2024-01-16T15:00:58.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.278+0000 7f3bccc00640 1 -- 172.21.15.53:0/1994426388 wait complete. 2024-01-16T15:00:58.278 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:00:58.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.519+0000 7f8da5a00640 1 Processor -- start 2024-01-16T15:00:58.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.519+0000 7f8da5a00640 1 -- start start 2024-01-16T15:00:58.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8da5a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8da00ae5a0 0x7f8da015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8da5a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8da00b3ae0 0x7f8da006cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8da5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da006d120 0x7f8da00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8da5a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da0071450 con 0x7f8da00ae5a0 2024-01-16T15:00:58.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8da5a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8da0070c40 con 0x7f8da00b3ae0 2024-01-16T15:00:58.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8da5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da006c470 con 0x7f8da006d120 2024-01-16T15:00:58.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.521+0000 7f8d9ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8da00ae5a0 0x7f8da015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8da00ae5a0 0x7f8da015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45644/0 (socket says 172.21.15.53:45644) 2024-01-16T15:00:58.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 -- 172.21.15.53:0/3339029759 learned_addr learned my addr 172.21.15.53:0/3339029759 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9e000640 1 --2- 172.21.15.53:0/3339029759 >> v2:172.21.15.146:3300/0 conn(0x7f8da00b3ae0 0x7f8da006cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9f400640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da006d120 0x7f8da00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 -- 172.21.15.53:0/3339029759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da006d120 msgr2=0x7f8da00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da006d120 0x7f8da00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 -- 172.21.15.53:0/3339029759 >> v2:172.21.15.146:3300/0 conn(0x7f8da00b3ae0 msgr2=0x7f8da006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 --2- 172.21.15.53:0/3339029759 >> v2:172.21.15.146:3300/0 conn(0x7f8da00b3ae0 0x7f8da006cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9ea00640 1 -- 172.21.15.53:0/3339029759 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da006c750 con 0x7f8da00ae5a0 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9f400640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da006d120 0x7f8da00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:58.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.522+0000 7f8d9e000640 1 --2- 172.21.15.53:0/3339029759 >> v2:172.21.15.146:3300/0 conn(0x7f8da00b3ae0 0x7f8da006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:00:58.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.524+0000 7f8d9ea00640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da00ae5a0 0x7f8da015e430 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8da00cfea0 tx=0x7f8d9002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=425f81e18f133a5d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.524+0000 7f8d9d600640 1 -- 172.21.15.53:0/3339029759 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90031860 con 0x7f8da00ae5a0 2024-01-16T15:00:58.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.524+0000 7f8d9d600640 1 -- 172.21.15.53:0/3339029759 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d900319c0 con 0x7f8da00ae5a0 2024-01-16T15:00:58.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.524+0000 7f8d9d600640 1 -- 172.21.15.53:0/3339029759 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90031d50 con 0x7f8da00ae5a0 2024-01-16T15:00:58.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 -- 172.21.15.53:0/3339029759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da00ae5a0 msgr2=0x7f8da015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da00ae5a0 0x7f8da015e430 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8da00cfea0 tx=0x7f8d9002b900 comp rx=0 tx=0).stop 2024-01-16T15:00:58.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 -- 172.21.15.53:0/3339029759 shutdown_connections 2024-01-16T15:00:58.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da006d120 0x7f8da00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 --2- 172.21.15.53:0/3339029759 >> v2:172.21.15.146:3300/0 conn(0x7f8da00b3ae0 0x7f8da006cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 --2- 172.21.15.53:0/3339029759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da00ae5a0 0x7f8da015e430 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 -- 172.21.15.53:0/3339029759 >> 172.21.15.53:0/3339029759 conn(0x7f8da006a800 msgr2=0x7f8da006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 -- 172.21.15.53:0/3339029759 shutdown_connections 2024-01-16T15:00:58.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.525+0000 7f8da5a00640 1 -- 172.21.15.53:0/3339029759 wait complete. 2024-01-16T15:00:58.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.527+0000 7f8da5a00640 1 Processor -- start 2024-01-16T15:00:58.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.527+0000 7f8da5a00640 1 -- start start 2024-01-16T15:00:58.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8da5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 0x7f8da01600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8da5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da00ae5a0 0x7f8da00cc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8da5a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8da00b3ae0 0x7f8da014af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8da5a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0160600 con 0x7f8da006d120 2024-01-16T15:00:58.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8da5a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da015f020 con 0x7f8da00b3ae0 2024-01-16T15:00:58.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8da5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da015f300 con 0x7f8da00ae5a0 2024-01-16T15:00:58.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8d9ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 0x7f8da01600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8d9ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 0x7f8da01600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45648/0 (socket says 172.21.15.53:45648) 2024-01-16T15:00:58.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.528+0000 7f8d9ea00640 1 -- 172.21.15.53:0/669898775 learned_addr learned my addr 172.21.15.53:0/669898775 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:58.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9ea00640 1 -- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da00ae5a0 msgr2=0x7f8da00cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9ea00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da00ae5a0 0x7f8da00cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9ea00640 1 -- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8da00b3ae0 msgr2=0x7f8da014af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9f400640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8da00b3ae0 0x7f8da014af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9ea00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8da00b3ae0 0x7f8da014af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9ea00640 1 -- 172.21.15.53:0/669898775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d900094d0 con 0x7f8da006d120 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d9ea00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 0x7f8da01600c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8d90000c00 tx=0x7f8d900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9003adf0 con 0x7f8da006d120 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d900443f0 con 0x7f8da006d120 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.529+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900447a0 con 0x7f8da006d120 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.530+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da015f5e0 con 0x7f8da006d120 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.530+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da015fba0 con 0x7f8da006d120 2024-01-16T15:00:58.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.531+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8da00675d0 con 0x7f8da006d120 2024-01-16T15:00:58.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.532+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8d90044900 con 0x7f8da006d120 2024-01-16T15:00:58.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.533+0000 7f8d96a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d7806b310 0x7f8d7806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.533+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8d900ac280 con 0x7f8da006d120 2024-01-16T15:00:58.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.533+0000 7f8d9e000640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d7806b310 0x7f8d7806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.534+0000 7f8d9e000640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d7806b310 0x7f8d7806d7d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8da00cea00 tx=0x7f8d88008330 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.537+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d900770e0 con 0x7f8da006d120 2024-01-16T15:00:58.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.730+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8da00b1900 con 0x7f8da006d120 2024-01-16T15:00:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.731+0000 7f8d96a00640 1 -- 172.21.15.53:0/669898775 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8d9007a700 con 0x7f8da006d120 2024-01-16T15:00:58.711 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.733+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d7806b310 msgr2=0x7f8d7806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.733+0000 7f8da5a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d7806b310 0x7f8d7806d7d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8da00cea00 tx=0x7f8d88008330 comp rx=0 tx=0).stop 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.733+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 msgr2=0x7f8da01600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.733+0000 7f8da5a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 0x7f8da01600c0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8d90000c00 tx=0x7f8d900046c0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 shutdown_connections 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d7806b310 0x7f8d7806d7d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8da00b3ae0 0x7f8da014af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8da00ae5a0 0x7f8da00cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 --2- 172.21.15.53:0/669898775 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8da006d120 0x7f8da01600c0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 >> 172.21.15.53:0/669898775 conn(0x7f8da006a800 msgr2=0x7f8da015d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 shutdown_connections 2024-01-16T15:00:58.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.734+0000 7f8da5a00640 1 -- 172.21.15.53:0/669898775 wait complete. 2024-01-16T15:00:58.732 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:00:58.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.975+0000 7eff8ae00640 1 Processor -- start 2024-01-16T15:00:58.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.975+0000 7eff8ae00640 1 -- start start 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.976+0000 7eff8ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413df80 0x7eff8415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff8ae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff840cf9a0 0x7eff840cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff8ae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff8406c470 con 0x7eff8413e360 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff8ae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff8406c750 con 0x7eff840cf9a0 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff8ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff8406ca30 con 0x7eff8413df80 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff88e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff840cf9a0 0x7eff840cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff88e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff840cf9a0 0x7eff840cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45660/0 (socket says 172.21.15.53:45660) 2024-01-16T15:00:58.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff88e00640 1 -- 172.21.15.53:0/3620245840 learned_addr learned my addr 172.21.15.53:0/3620245840 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:58.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.977+0000 7eff83e00640 1 --2- 172.21.15.53:0/3620245840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413df80 0x7eff8415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.978+0000 7eff88e00640 1 -- 172.21.15.53:0/3620245840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413df80 msgr2=0x7eff8415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:58.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.978+0000 7eff88e00640 1 --2- 172.21.15.53:0/3620245840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413df80 0x7eff8415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.978+0000 7eff88e00640 1 -- 172.21.15.53:0/3620245840 >> v1:172.21.15.146:6789/0 conn(0x7eff8413e360 legacy=0x7eff840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.978+0000 7eff88e00640 1 -- 172.21.15.53:0/3620245840 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff8406cd10 con 0x7eff840cf9a0 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.979+0000 7eff88e00640 1 --2- 172.21.15.53:0/3620245840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff840cf9a0 0x7eff840cd840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009e50 tx=0x7eff7c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1eb3f36af91337 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.979+0000 7eff82a00640 1 -- 172.21.15.53:0/3620245840 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c02cb00 con 0x7eff840cf9a0 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.979+0000 7eff82a00640 1 -- 172.21.15.53:0/3620245840 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff7c02cc60 con 0x7eff840cf9a0 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.979+0000 7eff82a00640 1 -- 172.21.15.53:0/3620245840 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0343f0 con 0x7eff840cf9a0 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 -- 172.21.15.53:0/3620245840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff840cf9a0 msgr2=0x7eff840cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 --2- 172.21.15.53:0/3620245840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff840cf9a0 0x7eff840cd840 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7eff7c009e50 tx=0x7eff7c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 -- 172.21.15.53:0/3620245840 shutdown_connections 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 --2- 172.21.15.53:0/3620245840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff840cf9a0 0x7eff840cd840 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 --2- 172.21.15.53:0/3620245840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413df80 0x7eff8415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 -- 172.21.15.53:0/3620245840 >> 172.21.15.53:0/3620245840 conn(0x7eff8406a800 msgr2=0x7eff8406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 -- 172.21.15.53:0/3620245840 shutdown_connections 2024-01-16T15:00:58.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.980+0000 7eff8ae00640 1 -- 172.21.15.53:0/3620245840 wait complete. 2024-01-16T15:00:58.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.982+0000 7eff8ae00640 1 Processor -- start 2024-01-16T15:00:58.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.982+0000 7eff8ae00640 1 -- start start 2024-01-16T15:00:58.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff8ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff840cf9a0 0x7eff84137850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff8ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 0x7eff841387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff8ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413e360 0x7eff84153a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff8ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8413b090 con 0x7eff8413df80 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff8ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff841392d0 con 0x7eff840cf9a0 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff8ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff840722d0 con 0x7eff8413e360 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 0x7eff841387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 0x7eff841387a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45672/0 (socket says 172.21.15.53:45672) 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 -- 172.21.15.53:0/2344220521 learned_addr learned my addr 172.21.15.53:0/2344220521 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83e00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff840cf9a0 0x7eff84137850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff88e00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413e360 0x7eff84153a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 -- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413e360 msgr2=0x7eff84153a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413e360 0x7eff84153a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 -- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff840cf9a0 msgr2=0x7eff84137850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff840cf9a0 0x7eff84137850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.983+0000 7eff83400640 1 -- 172.21.15.53:0/2344220521 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c009b30 con 0x7eff8413df80 2024-01-16T15:00:58.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.984+0000 7eff83400640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 0x7eff841387a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c480 tx=0x7eff6c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.984+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c0052d0 con 0x7eff8413df80 2024-01-16T15:00:58.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.984+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff6c005430 con 0x7eff8413df80 2024-01-16T15:00:58.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.984+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff6c012870 con 0x7eff8413df80 2024-01-16T15:00:58.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.984+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8413a730 con 0x7eff8413df80 2024-01-16T15:00:58.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.984+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff84138ff0 con 0x7eff8413df80 2024-01-16T15:00:58.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.985+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8413b1d0 con 0x7eff8413df80 2024-01-16T15:00:58.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.986+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7eff6c0129d0 con 0x7eff8413df80 2024-01-16T15:00:58.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.987+0000 7eff7be00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff5c06b2f0 0x7eff5c06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:58.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.987+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7eff6c0898f0 con 0x7eff8413df80 2024-01-16T15:00:58.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.988+0000 7eff83e00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff5c06b2f0 0x7eff5c06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:58.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.989+0000 7eff83e00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff5c06b2f0 0x7eff5c06d7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff74006fd0 tx=0x7eff74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:58.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:58.992+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff8413b1d0 con 0x7eff8413df80 2024-01-16T15:00:59.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.209+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7eff8413b1d0 con 0x7eff8413df80 2024-01-16T15:00:59.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.210+0000 7eff7be00640 1 -- 172.21.15.53:0/2344220521 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7eff8413b1d0 con 0x7eff8413df80 2024-01-16T15:00:59.190 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:00:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff5c06b2f0 msgr2=0x7eff5c06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff5c06b2f0 0x7eff5c06d7b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff74006fd0 tx=0x7eff74008040 comp rx=0 tx=0).stop 2024-01-16T15:00:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 msgr2=0x7eff841387a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 0x7eff841387a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7eff6c00c480 tx=0x7eff6c00c940 comp rx=0 tx=0).stop 2024-01-16T15:00:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 shutdown_connections 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff5c06b2f0 0x7eff5c06d7b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff8413e360 0x7eff84153a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff8413df80 0x7eff841387a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 --2- 172.21.15.53:0/2344220521 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff840cf9a0 0x7eff84137850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 >> 172.21.15.53:0/2344220521 conn(0x7eff8406a800 msgr2=0x7eff8415d440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 shutdown_connections 2024-01-16T15:00:59.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.213+0000 7eff8ae00640 1 -- 172.21.15.53:0/2344220521 wait complete. 2024-01-16T15:00:59.213 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:00:59.213 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:00:59.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:00:59.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.518+0000 7fa350200640 1 Processor -- start 2024-01-16T15:00:59.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.518+0000 7fa350200640 1 -- start start 2024-01-16T15:00:59.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.520+0000 7fa350200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.520+0000 7fa350200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3480cf9a0 0x7fa3480cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.520+0000 7fa350200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa34806c470 con 0x7fa34813e360 2024-01-16T15:00:59.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.520+0000 7fa350200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34806c750 con 0x7fa3480cf9a0 2024-01-16T15:00:59.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.520+0000 7fa350200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa34806ca30 con 0x7fa34813df80 2024-01-16T15:00:59.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3480cf9a0 0x7fa3480cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3480cf9a0 0x7fa3480cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:54860/0 (socket says 172.21.15.53:54860) 2024-01-16T15:00:59.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34e200640 1 -- 172.21.15.53:0/3424030625 learned_addr learned my addr 172.21.15.53:0/3424030625 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:59.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34e200640 1 -- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813df80 msgr2=0x7fa34815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34e200640 1 --2- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.521+0000 7fa34e200640 1 -- 172.21.15.53:0/3424030625 >> v1:172.21.15.146:6789/0 conn(0x7fa34813e360 legacy=0x7fa3480cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.522+0000 7fa34e200640 1 -- 172.21.15.53:0/3424030625 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34806cd10 con 0x7fa3480cf9a0 2024-01-16T15:00:59.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.523+0000 7fa34e200640 1 --2- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480cd840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa340009e50 tx=0x7fa34002bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed5e9de408291586 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa347e00640 1 -- 172.21.15.53:0/3424030625 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34002cb00 con 0x7fa3480cf9a0 2024-01-16T15:00:59.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa347e00640 1 -- 172.21.15.53:0/3424030625 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa34002cc60 con 0x7fa3480cf9a0 2024-01-16T15:00:59.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa347e00640 1 -- 172.21.15.53:0/3424030625 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3400343f0 con 0x7fa3480cf9a0 2024-01-16T15:00:59.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 -- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 msgr2=0x7fa3480cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 --2- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480cd840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa340009e50 tx=0x7fa34002bf60 comp rx=0 tx=0).stop 2024-01-16T15:00:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 -- 172.21.15.53:0/3424030625 shutdown_connections 2024-01-16T15:00:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 --2- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480cd840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 --2- 172.21.15.53:0/3424030625 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813df80 0x7fa34815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 -- 172.21.15.53:0/3424030625 >> 172.21.15.53:0/3424030625 conn(0x7fa34806a800 msgr2=0x7fa34806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 -- 172.21.15.53:0/3424030625 shutdown_connections 2024-01-16T15:00:59.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.524+0000 7fa350200640 1 -- 172.21.15.53:0/3424030625 wait complete. 2024-01-16T15:00:59.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.526+0000 7fa350200640 1 Processor -- start 2024-01-16T15:00:59.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.526+0000 7fa350200640 1 -- start start 2024-01-16T15:00:59.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.527+0000 7fa350200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.527+0000 7fa350200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 0x7fa3481e7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.527+0000 7fa350200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813e360 0x7fa3481e8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.527+0000 7fa350200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3481e7880 con 0x7fa34813df80 2024-01-16T15:00:59.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.527+0000 7fa350200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3481e7ab0 con 0x7fa3480cf9a0 2024-01-16T15:00:59.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.527+0000 7fa350200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3481e7d90 con 0x7fa34813e360 2024-01-16T15:00:59.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 0x7fa3481e7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 0x7fa3481e7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45686/0 (socket says 172.21.15.53:45686) 2024-01-16T15:00:59.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 -- 172.21.15.53:0/2308875768 learned_addr learned my addr 172.21.15.53:0/2308875768 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34e200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813e360 0x7fa3481e8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34d800640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 -- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813e360 msgr2=0x7fa3481e8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813e360 0x7fa3481e8b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 -- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 msgr2=0x7fa3480a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.528+0000 7fa34ce00640 1 -- 172.21.15.53:0/2308875768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa340009b30 con 0x7fa34813df80 2024-01-16T15:00:59.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.529+0000 7fa34ce00640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 0x7fa3481e7340 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa33000c450 tx=0x7fa33000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.529+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3300052d0 con 0x7fa34813df80 2024-01-16T15:00:59.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.529+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa330005430 con 0x7fa34813df80 2024-01-16T15:00:59.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.529+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa330012870 con 0x7fa34813df80 2024-01-16T15:00:59.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.529+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3481e8070 con 0x7fa34813df80 2024-01-16T15:00:59.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.529+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3481e8630 con 0x7fa34813df80 2024-01-16T15:00:59.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.530+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3480675d0 con 0x7fa34813df80 2024-01-16T15:00:59.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.531+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa3300129d0 con 0x7fa34813df80 2024-01-16T15:00:59.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.532+0000 7fa345600640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa32006b2c0 0x7fa32006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.532+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa3300897a0 con 0x7fa34813df80 2024-01-16T15:00:59.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.533+0000 7fa34d800640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa32006b2c0 0x7fa32006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.534+0000 7fa34d800640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa32006b2c0 0x7fa32006d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa338009ad0 tx=0x7fa338008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.538+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa330054580 con 0x7fa34813df80 2024-01-16T15:00:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.738+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa34815aed0 con 0x7fa34813df80 2024-01-16T15:00:59.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.740+0000 7fa345600640 1 -- 172.21.15.53:0/2308875768 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7fa330057ba0 con 0x7fa34813df80 2024-01-16T15:00:59.721 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:00:59.721 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:00:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa32006b2c0 msgr2=0x7fa32006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa32006b2c0 0x7fa32006d780 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa338009ad0 tx=0x7fa338008040 comp rx=0 tx=0).stop 2024-01-16T15:00:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 msgr2=0x7fa3481e7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 0x7fa3481e7340 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa33000c450 tx=0x7fa33000c910 comp rx=0 tx=0).stop 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 shutdown_connections 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa32006b2c0 0x7fa32006d780 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa34813e360 0x7fa3481e8b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa34813df80 0x7fa3481e7340 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 --2- 172.21.15.53:0/2308875768 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3480cf9a0 0x7fa3480a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 >> 172.21.15.53:0/2308875768 conn(0x7fa34806a800 msgr2=0x7fa3480ce6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 shutdown_connections 2024-01-16T15:00:59.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.743+0000 7fa350200640 1 -- 172.21.15.53:0/2308875768 wait complete. 2024-01-16T15:00:59.743 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:00:59.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:00:59.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.995+0000 7f34f5a00640 1 Processor -- start 2024-01-16T15:00:59.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.996+0000 7f34f5a00640 1 -- start start 2024-01-16T15:00:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.997+0000 7f34f5a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34f00cfde0 0x7f34f00b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.997+0000 7f34f5a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f00d1c60 con 0x7f34f015d4b0 2024-01-16T15:00:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.997+0000 7f34f5a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34f00b2730 con 0x7f34f00cf9a0 2024-01-16T15:00:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.997+0000 7f34f5a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34f00d1ec0 con 0x7f34f00cfde0 2024-01-16T15:00:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.998+0000 7f34efe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f34f015d4b0 0x7f34f00d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36268/0 (socket says 172.21.15.53:36268) 2024-01-16T15:00:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.998+0000 7f34efe00640 1 -- 172.21.15.53:0/3966907370 learned_addr learned my addr 172.21.15.53:0/3966907370 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:00:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.998+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 204377938 0 0) 0x7f34f00d1c60 con 0x7f34f015d4b0 2024-01-16T15:00:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.998+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d0003520 con 0x7f34f015d4b0 2024-01-16T15:00:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.998+0000 7f34f5000640 1 --2- 172.21.15.53:0/3966907370 >> v2:172.21.15.146:3300/0 conn(0x7f34f00cfde0 0x7f34f00b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3542357674 0 0) 0x7f34d0003520 con 0x7f34f015d4b0 2024-01-16T15:00:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 >> v2:172.21.15.146:3300/0 conn(0x7f34f00cfde0 msgr2=0x7f34f00b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 --2- 172.21.15.53:0/3966907370 >> v2:172.21.15.146:3300/0 conn(0x7f34f00cfde0 0x7f34f00b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 >> v1:172.21.15.146:6789/0 conn(0x7f34f00cf9a0 legacy=0x7f34f00b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f00b29d0 con 0x7f34f015d4b0 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f34e0001670 con 0x7f34f015d4b0 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 >> v1:172.21.15.53:6789/0 conn(0x7f34f015d4b0 legacy=0x7f34f00d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34d0003970 0x7f34d0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34d0005180 0x7f34d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00d1ec0 con 0x7f34d00047c0 2024-01-16T15:00:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d000c0d0 con 0x7f34d0003970 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d000c290 con 0x7f34d0005180 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34f5000640 1 --2- 172.21.15.53:0/3966907370 >> v2:172.21.15.146:3300/0 conn(0x7f34f00cfde0 0x7f34f00b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34e7e00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:00:59.999+0000 7f34efe00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34d0003970 0x7f34d0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34f5000640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34d0005180 0x7f34d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34e7e00640 1 -- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34d0005180 msgr2=0x7f34d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34e7e00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34d0005180 0x7f34d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34e7e00640 1 -- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34d0003970 msgr2=0x7f34d0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34e7e00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34d0003970 0x7f34d0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34e7e00640 1 -- 172.21.15.53:0/3966907370 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d0003520 con 0x7f34d00047c0 2024-01-16T15:00:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.000+0000 7f34f5000640 1 -- 172.21.15.53:0/3966907370 reap_dead start 2024-01-16T15:00:59.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.001+0000 7f34e7e00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34d0004c40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f34d8001750 tx=0x7f34d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f72f141a8ae92ed8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.001+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d80052d0 con 0x7f34d00047c0 2024-01-16T15:00:59.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.001+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34d802e050 con 0x7f34d00047c0 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.001+0000 7f34ef400640 1 -- 172.21.15.53:0/3966907370 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34d8003830 con 0x7f34d00047c0 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 -- 172.21.15.53:0/3966907370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 msgr2=0x7f34d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34d0004c40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f34d8001750 tx=0x7f34d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 -- 172.21.15.53:0/3966907370 shutdown_connections 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 --2- 172.21.15.53:0/3966907370 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34d0004c40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 -- 172.21.15.53:0/3966907370 >> 172.21.15.53:0/3966907370 conn(0x7f34f00d3000 msgr2=0x7f34f00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 -- 172.21.15.53:0/3966907370 shutdown_connections 2024-01-16T15:00:59.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.002+0000 7f34f5a00640 1 -- 172.21.15.53:0/3966907370 wait complete. 2024-01-16T15:00:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.004+0000 7f34f5a00640 1 Processor -- start 2024-01-16T15:00:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.004+0000 7f34f5a00640 1 -- start start 2024-01-16T15:00:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34f5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34f014bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34f5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34f00b3020 0x7f34f00b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34f5a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34f014e820 0x7f34f014ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34f5a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00b8860 con 0x7f34d00047c0 2024-01-16T15:00:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34f5a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f014c520 con 0x7f34f014e820 2024-01-16T15:00:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34f5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34f00b39e0 con 0x7f34f00b3020 2024-01-16T15:00:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.005+0000 7f34efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34f014bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34f014bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45706/0 (socket says 172.21.15.53:45706) 2024-01-16T15:00:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 -- 172.21.15.53:0/2840553745 learned_addr learned my addr 172.21.15.53:0/2840553745 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34e7e00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34f00b3020 0x7f34f00b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 -- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34f00b3020 msgr2=0x7f34f00b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34f00b3020 0x7f34f00b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34f5000640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34f014e820 0x7f34f014ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 -- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34f014e820 msgr2=0x7f34f014ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:00:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34f014e820 0x7f34f014ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 -- 172.21.15.53:0/2840553745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f00ccb50 con 0x7f34d00047c0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.006+0000 7f34efe00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34f014bfe0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f34e00011e0 tx=0x7f34e0007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.007+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e0001650 con 0x7f34d00047c0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.007+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34e000f940 con 0x7f34d00047c0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.007+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34e000fcd0 con 0x7f34d00047c0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.007+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f014f400 con 0x7f34d00047c0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.007+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f00b5f40 con 0x7f34d00047c0 2024-01-16T15:00:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.008+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f0067450 con 0x7f34d00047c0 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.009+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f34e001d460 con 0x7f34d00047c0 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.010+0000 7f34ecc00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f34c806b2f0 0x7f34c806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.010+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f34e0089d70 con 0x7f34d00047c0 2024-01-16T15:00:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.011+0000 7f34e7e00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f34c806b2f0 0x7f34c806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:00:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.012+0000 7f34e7e00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f34c806b2f0 0x7f34c806d7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34d8033000 tx=0x7f34d8009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:00:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.015+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34e0054b60 con 0x7f34d00047c0 2024-01-16T15:01:00.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.210+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f34f00afd00 con 0x7f34d00047c0 2024-01-16T15:01:00.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.211+0000 7f34ecc00640 1 -- 172.21.15.53:0/2840553745 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f34e0058180 con 0x7f34d00047c0 2024-01-16T15:01:00.191 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f34c806b2f0 msgr2=0x7f34c806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f34c806b2f0 0x7f34c806d7b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f34d8033000 tx=0x7f34d8009030 comp rx=0 tx=0).stop 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 msgr2=0x7f34f014bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34f014bfe0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f34e00011e0 tx=0x7f34e0007720 comp rx=0 tx=0).stop 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 shutdown_connections 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f34c806b2f0 0x7f34c806d7b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34f014e820 0x7f34f014ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34f00b3020 0x7f34f00b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 --2- 172.21.15.53:0/2840553745 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34d00047c0 0x7f34f014bfe0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 >> 172.21.15.53:0/2840553745 conn(0x7f34f00d3000 msgr2=0x7f34f00b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.213+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 shutdown_connections 2024-01-16T15:01:00.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.214+0000 7f34f5a00640 1 -- 172.21.15.53:0/2840553745 wait complete. 2024-01-16T15:01:00.214 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:01:00.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.449+0000 7f7fe2400640 1 Processor -- start 2024-01-16T15:01:00.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.450+0000 7f7fe2400640 1 -- start start 2024-01-16T15:01:00.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.451+0000 7f7fe2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7fdc15d1d0 0x7f7fdc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.451+0000 7f7fe2400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc0d0090 con 0x7f7fdc0b1f70 2024-01-16T15:01:00.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.451+0000 7f7fe2400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fdc0d02e0 con 0x7f7fdc0b11d0 2024-01-16T15:01:00.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.451+0000 7f7fe2400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fdc0d05c0 con 0x7f7fdc15d1d0 2024-01-16T15:01:00.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.452+0000 7f7fd3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7fdc0b1f70 0x7f7fdc0cdb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36270/0 (socket says 172.21.15.53:36270) 2024-01-16T15:01:00.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.452+0000 7f7fd3e00640 1 -- 172.21.15.53:0/3843398788 learned_addr learned my addr 172.21.15.53:0/3843398788 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:00.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.452+0000 7f7fe1000640 1 --2- 172.21.15.53:0/3843398788 >> v2:172.21.15.146:3300/0 conn(0x7f7fdc15d1d0 0x7f7fdc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.452+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023059562 0 0) 0x7f7fdc0d0090 con 0x7f7fdc0b1f70 2024-01-16T15:01:00.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.452+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fbc003520 con 0x7f7fdc0b1f70 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2933533058 0 0) 0x7f7fbc003520 con 0x7f7fdc0b1f70 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 >> v2:172.21.15.146:3300/0 conn(0x7f7fdc15d1d0 msgr2=0x7f7fdc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 --2- 172.21.15.53:0/3843398788 >> v2:172.21.15.146:3300/0 conn(0x7f7fdc15d1d0 0x7f7fdc0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 >> v1:172.21.15.146:6789/0 conn(0x7f7fdc0b11d0 legacy=0x7f7fdc0ce270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0d08a0 con 0x7f7fdc0b1f70 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7fc4001670 con 0x7f7fdc0b1f70 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 >> v1:172.21.15.53:6789/0 conn(0x7f7fdc0b1f70 legacy=0x7f7fdc0cdb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fbc003970 0x7f7fbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fbc005180 0x7f7fbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0d05c0 con 0x7f7fbc0047c0 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc00c020 con 0x7f7fbc005180 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fbc00c1e0 con 0x7f7fbc003970 2024-01-16T15:01:00.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fd3e00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.453+0000 7f7fe1a00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fbc005180 0x7f7fbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.454+0000 7f7fd3e00640 1 -- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fbc003970 msgr2=0x7f7fbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.454+0000 7f7fd3e00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fbc003970 0x7f7fbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.454+0000 7f7fd3e00640 1 -- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fbc005180 msgr2=0x7f7fbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.454+0000 7f7fd3e00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fbc005180 0x7f7fbc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.454+0000 7f7fd3e00640 1 -- 172.21.15.53:0/3843398788 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fbc003520 con 0x7f7fbc0047c0 2024-01-16T15:01:00.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.454+0000 7f7fe1a00640 1 -- 172.21.15.53:0/3843398788 reap_dead start 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.455+0000 7f7fd3e00640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fbc004c40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c240 tx=0x7f7fbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63797b736b4b3685 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.455+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc4005ad0 con 0x7f7fbc0047c0 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.455+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fc4005c30 con 0x7f7fbc0047c0 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.455+0000 7f7fdbe00640 1 -- 172.21.15.53:0/3843398788 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fc402caf0 con 0x7f7fbc0047c0 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 -- 172.21.15.53:0/3843398788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 msgr2=0x7f7fbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fbc004c40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7fc400c240 tx=0x7f7fbc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 -- 172.21.15.53:0/3843398788 shutdown_connections 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 --2- 172.21.15.53:0/3843398788 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fbc004c40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 -- 172.21.15.53:0/3843398788 >> 172.21.15.53:0/3843398788 conn(0x7f7fdc0d3050 msgr2=0x7f7fdc0d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 -- 172.21.15.53:0/3843398788 shutdown_connections 2024-01-16T15:01:00.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.456+0000 7f7fe2400640 1 -- 172.21.15.53:0/3843398788 wait complete. 2024-01-16T15:01:00.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.458+0000 7f7fe2400640 1 Processor -- start 2024-01-16T15:01:00.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.458+0000 7f7fe2400640 1 -- start start 2024-01-16T15:01:00.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fdc15a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fdc0a0600 0x7f7fdc0b7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe2400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fdc0a0a80 0x7f7fdc0b84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc1f32c0 con 0x7f7fbc0047c0 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe2400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0b24b0 con 0x7f7fdc0a0a80 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fdc0b2790 con 0x7f7fdc0a0600 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fdc15a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fdc15a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45718/0 (socket says 172.21.15.53:45718) 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe1000640 1 -- 172.21.15.53:0/566772992 learned_addr learned my addr 172.21.15.53:0/566772992 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe1000640 1 -- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fdc0a0600 msgr2=0x7f7fdc0b7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe1000640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fdc0a0600 0x7f7fdc0b7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.459+0000 7f7fe1a00640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fdc0a0a80 0x7f7fdc0b84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fe1000640 1 -- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fdc0a0a80 msgr2=0x7f7fdc0b84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fe1000640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fdc0a0a80 0x7f7fdc0b84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fe1000640 1 -- 172.21.15.53:0/566772992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fdc0b2a70 con 0x7f7fbc0047c0 2024-01-16T15:01:00.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fe1000640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fdc15a1f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00e770 tx=0x7f7fcc00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc00c730 con 0x7f7fbc0047c0 2024-01-16T15:01:00.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc00c890 con 0x7f7fbc0047c0 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fcc00cc40 con 0x7f7fbc0047c0 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fdc0b2d50 con 0x7f7fbc0047c0 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.460+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fdc15a730 con 0x7f7fbc0047c0 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.461+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fdc0a0e60 con 0x7f7fbc0047c0 2024-01-16T15:01:00.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.462+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7fcc013470 con 0x7f7fbc0047c0 2024-01-16T15:01:00.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.463+0000 7f7fd9600640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7fb406b2f0 0x7f7fb406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.463+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7fcc089fb0 con 0x7f7fbc0047c0 2024-01-16T15:01:00.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.464+0000 7f7fd3e00640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7fb406b2f0 0x7f7fb406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.465+0000 7f7fd3e00640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7fb406b2f0 0x7f7fb406d7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4034000 tx=0x7f7fc4039040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.468+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fcc054c70 con 0x7f7fbc0047c0 2024-01-16T15:01:00.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.662+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7fdc067450 con 0x7f7fbc0047c0 2024-01-16T15:01:00.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.663+0000 7f7fd9600640 1 -- 172.21.15.53:0/566772992 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7fcc021020 con 0x7f7fbc0047c0 2024-01-16T15:01:00.643 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:01:00.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.665+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7fb406b2f0 msgr2=0x7f7fb406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.665+0000 7f7fe2400640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7fb406b2f0 0x7f7fb406d7b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4034000 tx=0x7f7fc4039040 comp rx=0 tx=0).stop 2024-01-16T15:01:00.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.665+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 msgr2=0x7f7fdc15a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.665+0000 7f7fe2400640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fdc15a1f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f7fcc00e770 tx=0x7f7fcc00ec30 comp rx=0 tx=0).stop 2024-01-16T15:01:00.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 shutdown_connections 2024-01-16T15:01:00.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7fb406b2f0 0x7f7fb406d7b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fdc0a0a80 0x7f7fdc0b84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7fdc0a0600 0x7f7fdc0b7fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 --2- 172.21.15.53:0/566772992 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fbc0047c0 0x7f7fdc15a1f0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 >> 172.21.15.53:0/566772992 conn(0x7f7fdc0d3050 msgr2=0x7f7fdc15dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 shutdown_connections 2024-01-16T15:01:00.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.666+0000 7f7fe2400640 1 -- 172.21.15.53:0/566772992 wait complete. 2024-01-16T15:01:00.664 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:01:00.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.907+0000 7f4cfec00640 1 Processor -- start 2024-01-16T15:01:00.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.907+0000 7f4cfec00640 1 -- start start 2024-01-16T15:01:00.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.909+0000 7f4cfec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cf8070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.909+0000 7f4cfec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf80706a0 con 0x7f4cf806c730 2024-01-16T15:01:00.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.909+0000 7f4cfec00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cf80708d0 con 0x7f4cf806cb10 2024-01-16T15:01:00.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.909+0000 7f4cfec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80b3290 con 0x7f4cf80b2e00 2024-01-16T15:01:00.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.909+0000 7f4cfd800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4cf806c730 0x7f4cf8071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36274/0 (socket says 172.21.15.53:36274) 2024-01-16T15:01:00.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.909+0000 7f4cfd800640 1 -- 172.21.15.53:0/1999957912 learned_addr learned my addr 172.21.15.53:0/1999957912 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cfce00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cf8070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1035767607 0 0) 0x7f4cf80706a0 con 0x7f4cf806c730 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd8003520 con 0x7f4cf806c730 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377926264 0 0) 0x7f4cd8003520 con 0x7f4cf806c730 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 msgr2=0x7f4cf8070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cf8070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 >> v1:172.21.15.146:6789/0 conn(0x7f4cf806cb10 legacy=0x7f4cf806cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf8070bb0 con 0x7f4cf806c730 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4ce80019f0 con 0x7f4cf806c730 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 >> v1:172.21.15.53:6789/0 conn(0x7f4cf806c730 legacy=0x7f4cf8071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80039a0 0x7f4cd8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cd80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bab0 con 0x7f4cd80047c0 2024-01-16T15:01:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bc20 con 0x7f4cd80039a0 2024-01-16T15:01:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.910+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bde0 con 0x7f4cf80b2e00 2024-01-16T15:01:00.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfce00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfd800640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80039a0 0x7f4cd8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfce00640 1 -- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 msgr2=0x7f4cd80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfe200640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cd80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfce00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf80b2e00 0x7f4cd80051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfce00640 1 -- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80039a0 msgr2=0x7f4cd8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfce00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80039a0 0x7f4cd8003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.911+0000 7f4cfce00640 1 -- 172.21.15.53:0/1999957912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8003520 con 0x7f4cd80047c0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.912+0000 7f4cfce00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009ac0 tx=0x7f4cd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49f5876705442b3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.913+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce002c6b0 con 0x7f4cd80047c0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.913+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce002c810 con 0x7f4cd80047c0 2024-01-16T15:01:00.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.913+0000 7f4cf7e00640 1 -- 172.21.15.53:0/1999957912 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce002cba0 con 0x7f4cd80047c0 2024-01-16T15:01:00.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.913+0000 7f4cfec00640 1 -- 172.21.15.53:0/1999957912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd80047c0 msgr2=0x7f4cd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.913+0000 7f4cfec00640 1 --2- 172.21.15.53:0/1999957912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0009ac0 tx=0x7f4cd80057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.913+0000 7f4cfe200640 1 -- 172.21.15.53:0/1999957912 reap_dead start 2024-01-16T15:01:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.914+0000 7f4cfec00640 1 -- 172.21.15.53:0/1999957912 shutdown_connections 2024-01-16T15:01:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.914+0000 7f4cfec00640 1 -- 172.21.15.53:0/1999957912 >> 172.21.15.53:0/1999957912 conn(0x7f4cf806a7d0 msgr2=0x7f4cf806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.914+0000 7f4cfec00640 1 -- 172.21.15.53:0/1999957912 shutdown_connections 2024-01-16T15:01:00.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.914+0000 7f4cfec00640 1 -- 172.21.15.53:0/1999957912 wait complete. 2024-01-16T15:01:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.915+0000 7f4cfec00640 1 Processor -- start 2024-01-16T15:01:00.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.916+0000 7f4cfec00640 1 -- start start 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.916+0000 7f4cfec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cf8144080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf8144df0 0x7f4cf80ad820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 0x7f4cf80add60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf81445c0 con 0x7f4cf80aae00 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80ab300 con 0x7f4cd80047c0 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80ab830 con 0x7f4cf8144df0 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 0x7f4cf80add60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 0x7f4cf80add60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45740/0 (socket says 172.21.15.53:45740) 2024-01-16T15:01:00.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 -- 172.21.15.53:0/835753101 learned_addr learned my addr 172.21.15.53:0/835753101 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:00.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfd800640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cf8144080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfce00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf8144df0 0x7f4cf80ad820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 -- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf8144df0 msgr2=0x7f4cf80ad820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf8144df0 0x7f4cf80ad820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 -- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 msgr2=0x7f4cf8144080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cf8144080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfe200640 1 -- 172.21.15.53:0/835753101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80abb10 con 0x7f4cf80aae00 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfd800640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cf8144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.917+0000 7f4cfce00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf8144df0 0x7f4cf80ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.918+0000 7f4cfe200640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 0x7f4cf80add60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80b1e20 tx=0x7f4cf000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.918+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf000b070 con 0x7f4cf80aae00 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.918+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf00054e0 con 0x7f4cf80aae00 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.918+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf0003810 con 0x7f4cf80aae00 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.918+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf80abdf0 con 0x7f4cf80aae00 2024-01-16T15:01:00.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.918+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf80c3d00 con 0x7f4cf80aae00 2024-01-16T15:01:00.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.920+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf80ac310 con 0x7f4cf80aae00 2024-01-16T15:01:00.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.921+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4cf0003970 con 0x7f4cf80aae00 2024-01-16T15:01:00.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.922+0000 7f4cf5600640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:00.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.922+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4cf00893a0 con 0x7f4cf80aae00 2024-01-16T15:01:00.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.923+0000 7f4cfd800640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:00.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.924+0000 7f4cfd800640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8007c40 tx=0x7f4ce8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:00.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:00.927+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cf00540f0 con 0x7f4cf80aae00 2024-01-16T15:01:01.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.124+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4cf8067450 con 0x7f4cf80aae00 2024-01-16T15:01:01.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.126+0000 7f4cf5600640 1 -- 172.21.15.53:0/835753101 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4cf0057710 con 0x7f4cf80aae00 2024-01-16T15:01:01.106 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.128+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 msgr2=0x7f4cd006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.128+0000 7f4cfec00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4ce8007c40 tx=0x7f4ce8009040 comp rx=0 tx=0).stop 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.128+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 msgr2=0x7f4cf80add60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.128+0000 7f4cfec00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 0x7f4cf80add60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80b1e20 tx=0x7f4cf000d890 comp rx=0 tx=0).stop 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 shutdown_connections 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4cd006b2c0 0x7f4cd006d780 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cf80aae00 0x7f4cf80add60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4cf8144df0 0x7f4cf80ad820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 --2- 172.21.15.53:0/835753101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cd80047c0 0x7f4cf8144080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 >> 172.21.15.53:0/835753101 conn(0x7f4cf806a7d0 msgr2=0x7f4cf80cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 shutdown_connections 2024-01-16T15:01:01.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.129+0000 7f4cfec00640 1 -- 172.21.15.53:0/835753101 wait complete. 2024-01-16T15:01:01.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:01:01.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.373+0000 7f1c91000640 1 Processor -- start 2024-01-16T15:01:01.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.374+0000 7f1c91000640 1 -- start start 2024-01-16T15:01:01.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.375+0000 7f1c91000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c8c0cc500 0x7f1c8c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.375+0000 7f1c91000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c06d160 0x7f1c8c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.375+0000 7f1c91000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c8c070e50 con 0x7f1c8c143c40 2024-01-16T15:01:01.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.375+0000 7f1c91000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c0723c0 con 0x7f1c8c0cc500 2024-01-16T15:01:01.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.375+0000 7f1c91000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0cced0 con 0x7f1c8c06d160 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.376+0000 7f1c8b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1c8c143c40 0x7f1c8c071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36286/0 (socket says 172.21.15.53:36286) 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.376+0000 7f1c8b400640 1 -- 172.21.15.53:0/1476640094 learned_addr learned my addr 172.21.15.53:0/1476640094 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.376+0000 7f1c8be00640 1 --2- 172.21.15.53:0/1476640094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c06d160 0x7f1c8c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.376+0000 7f1c83e00640 1 --2- 172.21.15.53:0/1476640094 >> v2:172.21.15.146:3300/0 conn(0x7f1c8c0cc500 0x7f1c8c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.376+0000 7f1c8aa00640 1 -- 172.21.15.53:0/1476640094 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1724546457 0 0) 0x7f1c8c070e50 con 0x7f1c8c143c40 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.377+0000 7f1c8aa00640 1 -- 172.21.15.53:0/1476640094 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c6c003520 con 0x7f1c8c143c40 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.377+0000 7f1c8be00640 1 -- 172.21.15.53:0/1476640094 >> v2:172.21.15.146:3300/0 conn(0x7f1c8c0cc500 msgr2=0x7f1c8c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.377+0000 7f1c8be00640 1 --2- 172.21.15.53:0/1476640094 >> v2:172.21.15.146:3300/0 conn(0x7f1c8c0cc500 0x7f1c8c0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.377+0000 7f1c8be00640 1 -- 172.21.15.53:0/1476640094 >> v1:172.21.15.53:6789/0 conn(0x7f1c8c143c40 legacy=0x7f1c8c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.377+0000 7f1c8be00640 1 -- 172.21.15.53:0/1476640094 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c144020 con 0x7f1c8c06d160 2024-01-16T15:01:01.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.379+0000 7f1c8be00640 1 --2- 172.21.15.53:0/1476640094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c06d160 0x7f1c8c06d560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009d90 tx=0x7f1c8402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e01cb9777b3b050 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.379+0000 7f1c8aa00640 1 -- 172.21.15.53:0/1476640094 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8402cbc0 con 0x7f1c8c06d160 2024-01-16T15:01:01.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.379+0000 7f1c8aa00640 1 -- 172.21.15.53:0/1476640094 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c8402cd20 con 0x7f1c8c06d160 2024-01-16T15:01:01.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.379+0000 7f1c8aa00640 1 -- 172.21.15.53:0/1476640094 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84034660 con 0x7f1c8c06d160 2024-01-16T15:01:01.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 -- 172.21.15.53:0/1476640094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c06d160 msgr2=0x7f1c8c06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 --2- 172.21.15.53:0/1476640094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c06d160 0x7f1c8c06d560 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009d90 tx=0x7f1c8402be10 comp rx=0 tx=0).stop 2024-01-16T15:01:01.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 -- 172.21.15.53:0/1476640094 shutdown_connections 2024-01-16T15:01:01.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 --2- 172.21.15.53:0/1476640094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c06d160 0x7f1c8c06d560 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 --2- 172.21.15.53:0/1476640094 >> v2:172.21.15.146:3300/0 conn(0x7f1c8c0cc500 0x7f1c8c0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 -- 172.21.15.53:0/1476640094 >> 172.21.15.53:0/1476640094 conn(0x7f1c8c06a800 msgr2=0x7f1c8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 -- 172.21.15.53:0/1476640094 shutdown_connections 2024-01-16T15:01:01.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.380+0000 7f1c91000640 1 -- 172.21.15.53:0/1476640094 wait complete. 2024-01-16T15:01:01.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.383+0000 7f1c91000640 1 Processor -- start 2024-01-16T15:01:01.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.383+0000 7f1c91000640 1 -- start start 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c91000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 0x7f1c8c1f2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c91000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0cc500 0x7f1c8c142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c91000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c143c40 0x7f1c8c154900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c91000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c1555d0 con 0x7f1c8c06d160 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c91000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c1558b0 con 0x7f1c8c0cc500 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c91000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c155b90 con 0x7f1c8c143c40 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c8b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 0x7f1c8c1f2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c8b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 0x7f1c8c1f2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45746/0 (socket says 172.21.15.53:45746) 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.384+0000 7f1c8b400640 1 -- 172.21.15.53:0/3085764208 learned_addr learned my addr 172.21.15.53:0/3085764208 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c83e00640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0cc500 0x7f1c8c142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8be00640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c143c40 0x7f1c8c154900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8b400640 1 -- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c143c40 msgr2=0x7f1c8c154900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8b400640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c143c40 0x7f1c8c154900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8b400640 1 -- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0cc500 msgr2=0x7f1c8c142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8b400640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0cc500 0x7f1c8c142a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8b400640 1 -- 172.21.15.53:0/3085764208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c84009a70 con 0x7f1c8c06d160 2024-01-16T15:01:01.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c83e00640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0cc500 0x7f1c8c142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:01.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c8b400640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 0x7f1c8c1f2750 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0010e0 tx=0x7f1c7c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c005ce0 con 0x7f1c8c06d160 2024-01-16T15:01:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7c005e60 con 0x7f1c8c06d160 2024-01-16T15:01:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.385+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0072f0 con 0x7f1c8c06d160 2024-01-16T15:01:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.386+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c155e70 con 0x7f1c8c06d160 2024-01-16T15:01:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.386+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c1f2c90 con 0x7f1c8c06d160 2024-01-16T15:01:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.387+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c067450 con 0x7f1c8c06d160 2024-01-16T15:01:01.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.388+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1c7c007450 con 0x7f1c8c06d160 2024-01-16T15:01:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.389+0000 7f1c83400640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c6406b2c0 0x7f1c6406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.389+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1c7c0892b0 con 0x7f1c8c06d160 2024-01-16T15:01:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.389+0000 7f1c83e00640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c6406b2c0 0x7f1c6406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.391+0000 7f1c83e00640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c6406b2c0 0x7f1c6406d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0708b0 tx=0x7f1c74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.393+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c7c054000 con 0x7f1c8c06d160 2024-01-16T15:01:01.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.603+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1c8c134fe0 con 0x7f1c8c06d160 2024-01-16T15:01:01.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.604+0000 7f1c83400640 1 -- 172.21.15.53:0/3085764208 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v24) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1c7c057620 con 0x7f1c8c06d160 2024-01-16T15:01:01.584 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c6406b2c0 msgr2=0x7f1c6406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c6406b2c0 0x7f1c6406d780 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0708b0 tx=0x7f1c74008040 comp rx=0 tx=0).stop 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 msgr2=0x7f1c8c1f2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 0x7f1c8c1f2750 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0010e0 tx=0x7f1c7c00c890 comp rx=0 tx=0).stop 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 shutdown_connections 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c6406b2c0 0x7f1c6406d780 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c8c143c40 0x7f1c8c154900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c8c0cc500 0x7f1c8c142a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 --2- 172.21.15.53:0/3085764208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c8c06d160 0x7f1c8c1f2750 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 >> 172.21.15.53:0/3085764208 conn(0x7f1c8c06a800 msgr2=0x7f1c8c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 shutdown_connections 2024-01-16T15:01:01.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.607+0000 7f1c91000640 1 -- 172.21.15.53:0/3085764208 wait complete. 2024-01-16T15:01:01.606 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:01:01.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:01.606 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-01-16T15:01:01.703 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:01.704 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:01:01.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:01.704 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:01:01.757 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:01:01.758 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:01.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.995+0000 7f84b9800640 1 Processor -- start 2024-01-16T15:01:01.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.995+0000 7f84b9800640 1 -- start start 2024-01-16T15:01:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.997+0000 7f84b9800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f84b413df80 0x7f84b415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.997+0000 7f84b9800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 0x7f84b40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.997+0000 7f84b9800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b406c470 con 0x7f84b413e360 2024-01-16T15:01:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.997+0000 7f84b9800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b406c750 con 0x7f84b413df80 2024-01-16T15:01:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.997+0000 7f84b9800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b406ca30 con 0x7f84b40cf9a0 2024-01-16T15:01:01.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.998+0000 7f84b3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 0x7f84b40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.998+0000 7f84b2a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f84b413df80 0x7f84b415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.998+0000 7f84b3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 0x7f84b40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:55726/0 (socket says 172.21.15.53:55726) 2024-01-16T15:01:01.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.998+0000 7f84b3400640 1 -- 172.21.15.53:0/3217424015 learned_addr learned my addr 172.21.15.53:0/3217424015 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:01.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.999+0000 7f84b2a00640 1 -- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 msgr2=0x7f84b40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:01.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.999+0000 7f84b2a00640 1 --2- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 0x7f84b40cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.999+0000 7f84b2a00640 1 -- 172.21.15.53:0/3217424015 >> v1:172.21.15.146:6789/0 conn(0x7f84b413e360 legacy=0x7f84b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.999+0000 7f84b2a00640 1 -- 172.21.15.53:0/3217424015 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b406cd10 con 0x7f84b413df80 2024-01-16T15:01:01.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:01.999+0000 7f84b3400640 1 --2- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 0x7f84b40cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:01.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.000+0000 7f84b2a00640 1 --2- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413df80 0x7f84b415bee0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84a40097a0 tx=0x7f84a402b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9db04ff14de21d9a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.001+0000 7f84b2000640 1 -- 172.21.15.53:0/3217424015 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a402d040 con 0x7f84b413df80 2024-01-16T15:01:01.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.001+0000 7f84b2000640 1 -- 172.21.15.53:0/3217424015 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a4031810 con 0x7f84b413df80 2024-01-16T15:01:01.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.001+0000 7f84b2000640 1 -- 172.21.15.53:0/3217424015 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4031ba0 con 0x7f84b413df80 2024-01-16T15:01:01.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.001+0000 7f84b9800640 1 -- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413df80 msgr2=0x7f84b415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.001+0000 7f84b9800640 1 --2- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413df80 0x7f84b415bee0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f84a40097a0 tx=0x7f84a402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.002+0000 7f84b9800640 1 -- 172.21.15.53:0/3217424015 shutdown_connections 2024-01-16T15:01:01.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.002+0000 7f84b9800640 1 --2- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b40cf9a0 0x7f84b40cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.002+0000 7f84b9800640 1 --2- 172.21.15.53:0/3217424015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413df80 0x7f84b415bee0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.002+0000 7f84b9800640 1 -- 172.21.15.53:0/3217424015 >> 172.21.15.53:0/3217424015 conn(0x7f84b406a800 msgr2=0x7f84b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:01.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.002+0000 7f84b9800640 1 -- 172.21.15.53:0/3217424015 shutdown_connections 2024-01-16T15:01:01.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.002+0000 7f84b9800640 1 -- 172.21.15.53:0/3217424015 wait complete. 2024-01-16T15:01:01.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.004+0000 7f84b9800640 1 Processor -- start 2024-01-16T15:01:01.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.004+0000 7f84b9800640 1 -- start start 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b9800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 0x7f84b40adf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b9800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b413df80 0x7f84b413b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b9800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413e360 0x7f84b413c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b9800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b4153330 con 0x7f84b40cf9a0 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b9800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b4153750 con 0x7f84b413e360 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b9800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b41526b0 con 0x7f84b413df80 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 0x7f84b40adf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 0x7f84b40adf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33444/0 (socket says 172.21.15.53:33444) 2024-01-16T15:01:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b2a00640 1 -- 172.21.15.53:0/2583154749 learned_addr learned my addr 172.21.15.53:0/2583154749 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84a9e00640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b413df80 0x7f84b413b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b3400640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413e360 0x7f84b413c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.005+0000 7f84b2a00640 1 -- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b413df80 msgr2=0x7f84b413b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b2a00640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b413df80 0x7f84b413b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b2a00640 1 -- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413e360 msgr2=0x7f84b413c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b2a00640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413e360 0x7f84b413c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b2a00640 1 -- 172.21.15.53:0/2583154749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac009b30 con 0x7f84b40cf9a0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b2a00640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 0x7f84b40adf20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f84a40098c0 tx=0x7f84a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4006e10 con 0x7f84b40cf9a0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a403ad60 con 0x7f84b40cf9a0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a4009400 con 0x7f84b40cf9a0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84a4044620 con 0x7f84b40cf9a0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.006+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b4152c70 con 0x7f84b40cf9a0 2024-01-16T15:01:01.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.008+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b40675d0 con 0x7f84b40cf9a0 2024-01-16T15:01:01.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.008+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f84a404d440 con 0x7f84b40cf9a0 2024-01-16T15:01:01.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.010+0000 7f84ab400640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f848c06b2c0 0x7f848c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:01.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.010+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f84a40ac060 con 0x7f84b40cf9a0 2024-01-16T15:01:01.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.010+0000 7f84a9e00640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f848c06b2c0 0x7f848c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:01.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.012+0000 7f84a9e00640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f848c06b2c0 0x7f848c06d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f84b4072440 tx=0x7f849c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:01.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.014+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84a40dee90 con 0x7f84b40cf9a0 2024-01-16T15:01:02.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.246+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84b413a310 con 0x7f84b40cf9a0 2024-01-16T15:01:02.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.247+0000 7f84ab400640 1 -- 172.21.15.53:0/2583154749 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f84a4035090 con 0x7f84b40cf9a0 2024-01-16T15:01:02.228 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:02.228 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:01:02.228 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f848c06b2c0 msgr2=0x7f848c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f848c06b2c0 0x7f848c06d780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f84b4072440 tx=0x7f849c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 msgr2=0x7f84b40adf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 0x7f84b40adf20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f84a40098c0 tx=0x7f84a40046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 shutdown_connections 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f848c06b2c0 0x7f848c06d780 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84b413e360 0x7f84b413c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84b413df80 0x7f84b413b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 --2- 172.21.15.53:0/2583154749 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84b40cf9a0 0x7f84b40adf20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 >> 172.21.15.53:0/2583154749 conn(0x7f84b406a800 msgr2=0x7f84b40cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 shutdown_connections 2024-01-16T15:01:02.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.250+0000 7f84b9800640 1 -- 172.21.15.53:0/2583154749 wait complete. 2024-01-16T15:01:02.248 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:02.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.488+0000 7f34b7e00640 1 Processor -- start 2024-01-16T15:01:02.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.488+0000 7f34b7e00640 1 -- start start 2024-01-16T15:01:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.490+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.490+0000 7f34b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34b80ccc10 0x7f34b806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.490+0000 7f34b7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34b8070e50 con 0x7f34b80717d0 2024-01-16T15:01:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.490+0000 7f34b7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34b80723c0 con 0x7f34b80ccc10 2024-01-16T15:01:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.490+0000 7f34b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b8144020 con 0x7f34b8143c40 2024-01-16T15:01:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34b80ccc10 0x7f34b806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b8071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:55742/0 (socket says 172.21.15.53:55742) 2024-01-16T15:01:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 -- 172.21.15.53:0/775364606 learned_addr learned my addr 172.21.15.53:0/775364606 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 -- 172.21.15.53:0/775364606 >> v2:172.21.15.146:3300/0 conn(0x7f34b80ccc10 msgr2=0x7f34b806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 --2- 172.21.15.53:0/775364606 >> v2:172.21.15.146:3300/0 conn(0x7f34b80ccc10 0x7f34b806d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 -- 172.21.15.53:0/775364606 >> v1:172.21.15.146:6789/0 conn(0x7f34b80717d0 legacy=0x7f34b80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.491+0000 7f34b6a00640 1 -- 172.21.15.53:0/775364606 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b8071e00 con 0x7f34b8143c40 2024-01-16T15:01:02.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.493+0000 7f34b6a00640 1 --2- 172.21.15.53:0/775364606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b8071290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009870 tx=0x7f34a802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e56539fb1bfcf7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.493+0000 7f34b6000640 1 -- 172.21.15.53:0/775364606 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8031640 con 0x7f34b8143c40 2024-01-16T15:01:02.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.493+0000 7f34b6000640 1 -- 172.21.15.53:0/775364606 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a80317a0 con 0x7f34b8143c40 2024-01-16T15:01:02.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b6000640 1 -- 172.21.15.53:0/775364606 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8031b30 con 0x7f34b8143c40 2024-01-16T15:01:02.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 -- 172.21.15.53:0/775364606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 msgr2=0x7f34b8071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 --2- 172.21.15.53:0/775364606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b8071290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009870 tx=0x7f34a802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 -- 172.21.15.53:0/775364606 shutdown_connections 2024-01-16T15:01:02.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 --2- 172.21.15.53:0/775364606 >> v2:172.21.15.146:3300/0 conn(0x7f34b80ccc10 0x7f34b806d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 --2- 172.21.15.53:0/775364606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b8071290 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 -- 172.21.15.53:0/775364606 >> 172.21.15.53:0/775364606 conn(0x7f34b806a800 msgr2=0x7f34b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 -- 172.21.15.53:0/775364606 shutdown_connections 2024-01-16T15:01:02.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.494+0000 7f34b7e00640 1 -- 172.21.15.53:0/775364606 wait complete. 2024-01-16T15:01:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.497+0000 7f34b7e00640 1 Processor -- start 2024-01-16T15:01:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.497+0000 7f34b7e00640 1 -- start start 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b80717d0 0x7f34b80c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 0x7f34b813ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b81421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34b7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b80c56a0 con 0x7f34b80ccc10 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34b7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b8142700 con 0x7f34b80717d0 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34b80c4a90 con 0x7f34b8143c40 2024-01-16T15:01:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 0x7f34b813ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 0x7f34b813ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33448/0 (socket says 172.21.15.53:33448) 2024-01-16T15:01:02.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.498+0000 7f34afe00640 1 -- 172.21.15.53:0/3058111955 learned_addr learned my addr 172.21.15.53:0/3058111955 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34b6a00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b80717d0 0x7f34b80c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34b7400640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b81421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34afe00640 1 -- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 msgr2=0x7f34b81421c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34afe00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b81421c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34afe00640 1 -- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b80717d0 msgr2=0x7f34b80c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34afe00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b80717d0 0x7f34b80c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34afe00640 1 -- 172.21.15.53:0/3058111955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a80094d0 con 0x7f34b80ccc10 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34b6a00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b80717d0 0x7f34b80c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34b7400640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b81421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34afe00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 0x7f34b813ca20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c400 tx=0x7f34a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a00052d0 con 0x7f34b80ccc10 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.499+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34a0005450 con 0x7f34b80ccc10 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.500+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0007260 con 0x7f34b80ccc10 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.500+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b80c4d70 con 0x7f34b80ccc10 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.500+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b80c5330 con 0x7f34b80ccc10 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.501+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b80c54f0 con 0x7f34b80ccc10 2024-01-16T15:01:02.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.502+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f34a00073c0 con 0x7f34b80ccc10 2024-01-16T15:01:02.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.503+0000 7f34aea00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f349006b2f0 0x7f349006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.503+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f34a00890e0 con 0x7f34b80ccc10 2024-01-16T15:01:02.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.504+0000 7f34b6a00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f349006b2f0 0x7f349006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.505+0000 7f34b6a00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f349006b2f0 0x7f349006d7b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f34a8036000 tx=0x7f34a803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.510+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f34a0053e30 con 0x7f34b80ccc10 2024-01-16T15:01:02.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.708+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34b8067450 con 0x7f34b80ccc10 2024-01-16T15:01:02.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.710+0000 7f34aea00640 1 -- 172.21.15.53:0/3058111955 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f34a0057450 con 0x7f34b80ccc10 2024-01-16T15:01:02.690 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:02.690 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f349006b2f0 msgr2=0x7f349006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f349006b2f0 0x7f349006d7b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f34a8036000 tx=0x7f34a803c040 comp rx=0 tx=0).stop 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 msgr2=0x7f34b813ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 0x7f34b813ca20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f34a000c400 tx=0x7f34a000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 shutdown_connections 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f349006b2f0 0x7f349006d7b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34b8143c40 0x7f34b81421c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34b80ccc10 0x7f34b813ca20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 --2- 172.21.15.53:0/3058111955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34b80717d0 0x7f34b80c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 >> 172.21.15.53:0/3058111955 conn(0x7f34b806a800 msgr2=0x7f34b813e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 shutdown_connections 2024-01-16T15:01:02.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.713+0000 7f34b7e00640 1 -- 172.21.15.53:0/3058111955 wait complete. 2024-01-16T15:01:02.715 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T14:59:55.861367+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T14:59:56.423962+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:01:02.715 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:01:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.952+0000 7f601dc00640 1 Processor -- start 2024-01-16T15:01:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.953+0000 7f601dc00640 1 -- start start 2024-01-16T15:01:02.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.954+0000 7f601dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 0x7f60180d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.954+0000 7f601dc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f60180b41e0 0x7f60180cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.954+0000 7f601dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60180d0320 con 0x7f60180cf9a0 2024-01-16T15:01:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.954+0000 7f601dc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60180b2730 con 0x7f60180b41e0 2024-01-16T15:01:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.954+0000 7f601dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60180d1ec0 con 0x7f601815d4b0 2024-01-16T15:01:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.955+0000 7f6017400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f60180cf9a0 0x7f60180cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56028/0 (socket says 172.21.15.53:56028) 2024-01-16T15:01:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.955+0000 7f6017400640 1 -- 172.21.15.53:0/161376293 learned_addr learned my addr 172.21.15.53:0/161376293 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.955+0000 7f6017e00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 0x7f60180d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.955+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604865476 0 0) 0x7f60180d0320 con 0x7f60180cf9a0 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.955+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8003520 con 0x7f60180cf9a0 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.955+0000 7f601d200640 1 --2- 172.21.15.53:0/161376293 >> v2:172.21.15.146:3300/0 conn(0x7f60180b41e0 0x7f60180cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2022552039 0 0) 0x7f5ff8003520 con 0x7f60180cf9a0 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 msgr2=0x7f60180d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 0x7f60180d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 >> v2:172.21.15.146:3300/0 conn(0x7f60180b41e0 msgr2=0x7f60180cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 --2- 172.21.15.53:0/161376293 >> v2:172.21.15.146:3300/0 conn(0x7f60180b41e0 0x7f60180cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60180b29d0 con 0x7f60180cf9a0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f60000019f0 con 0x7f60180cf9a0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 >> v1:172.21.15.53:6789/0 conn(0x7f60180cf9a0 legacy=0x7f60180cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 0x7f5ff8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8003ef0 0x7f5ff80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff8004d00 0x7f5ff8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180b2730 con 0x7f5ff8004d00 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180d1ec0 con 0x7f5ff8003ef0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff800b9d0 con 0x7f601815d4b0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f601d200640 1 --2- 172.21.15.53:0/161376293 >> v2:172.21.15.146:3300/0 conn(0x7f60180b41e0 0x7f60180cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.956+0000 7f601d200640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff8004d00 0x7f5ff8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f6017400640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8003ef0 0x7f5ff80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f6017e00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 0x7f5ff8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f601d200640 1 -- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 msgr2=0x7f5ff8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f601d200640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f601815d4b0 0x7f5ff8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f601d200640 1 -- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8003ef0 msgr2=0x7f5ff80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:02.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f601d200640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8003ef0 0x7f5ff80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.957+0000 7f601d200640 1 -- 172.21.15.53:0/161376293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff8003520 con 0x7f5ff8004d00 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.958+0000 7f601d200640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff8004d00 0x7f5ff8005180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f60180b4670 tx=0x7f5ff80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e849ded27263e0c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.958+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010004230 con 0x7f5ff8004d00 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.958+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6010004390 con 0x7f5ff8004d00 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.958+0000 7f6016a00640 1 -- 172.21.15.53:0/161376293 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6010003850 con 0x7f5ff8004d00 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601dc00640 1 -- 172.21.15.53:0/161376293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff8004d00 msgr2=0x7f5ff8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601dc00640 1 --2- 172.21.15.53:0/161376293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ff8004d00 0x7f5ff8005180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f60180b4670 tx=0x7f5ff80057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601d200640 1 -- 172.21.15.53:0/161376293 reap_dead start 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601dc00640 1 -- 172.21.15.53:0/161376293 shutdown_connections 2024-01-16T15:01:02.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601dc00640 1 -- 172.21.15.53:0/161376293 >> 172.21.15.53:0/161376293 conn(0x7f60180d3000 msgr2=0x7f60180d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:02.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601dc00640 1 -- 172.21.15.53:0/161376293 shutdown_connections 2024-01-16T15:01:02.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.959+0000 7f601dc00640 1 -- 172.21.15.53:0/161376293 wait complete. 2024-01-16T15:01:02.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.961+0000 7f601dc00640 1 Processor -- start 2024-01-16T15:01:02.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.961+0000 7f601dc00640 1 -- start start 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f601dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8004d00 0x7f6018159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f601dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 0x7f60180cafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f601dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180cb4e0 0x7f60180cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f601dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60181e7580 con 0x7f60181e7100 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f601dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180cd040 con 0x7f5ff8004d00 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f601dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60181e5d80 con 0x7f60180cb4e0 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f6017400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 0x7f60180cafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f6017400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 0x7f60180cafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33470/0 (socket says 172.21.15.53:33470) 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.962+0000 7f6017400640 1 -- 172.21.15.53:0/2242050026 learned_addr learned my addr 172.21.15.53:0/2242050026 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017e00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8004d00 0x7f6018159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f601d200640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180cb4e0 0x7f60180cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017400640 1 -- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180cb4e0 msgr2=0x7f60180cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017400640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180cb4e0 0x7f60180cddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017400640 1 -- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8004d00 msgr2=0x7f6018159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:02.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017400640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8004d00 0x7f6018159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:02.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017400640 1 -- 172.21.15.53:0/2242050026 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601815da10 con 0x7f60181e7100 2024-01-16T15:01:02.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.963+0000 7f6017400640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 0x7f60180cafa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6000007b30 tx=0x7f600000e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.964+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000013420 con 0x7f60181e7100 2024-01-16T15:01:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.964+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60000135a0 con 0x7f60181e7100 2024-01-16T15:01:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.964+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6000013950 con 0x7f60181e7100 2024-01-16T15:01:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.964+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60181e6060 con 0x7f60181e7100 2024-01-16T15:01:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.964+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60181e6620 con 0x7f60181e7100 2024-01-16T15:01:02.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.965+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6018067450 con 0x7f60181e7100 2024-01-16T15:01:02.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.966+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f6000013ab0 con 0x7f60181e7100 2024-01-16T15:01:02.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.967+0000 7f600fe00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ff006b2f0 0x7f5ff006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:02.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.967+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f600008a410 con 0x7f60181e7100 2024-01-16T15:01:02.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.968+0000 7f6017e00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ff006b2f0 0x7f5ff006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:02.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.969+0000 7f6017e00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ff006b2f0 0x7f5ff006d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6008009a10 tx=0x7f6008008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:02.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:02.972+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f60000551f0 con 0x7f60181e7100 2024-01-16T15:01:03.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.167+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f60181e8c00 con 0x7f60181e7100 2024-01-16T15:01:03.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.168+0000 7f600fe00640 1 -- 172.21.15.53:0/2242050026 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v24) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6000058810 con 0x7f60181e7100 2024-01-16T15:01:03.148 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:01:03.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ff006b2f0 msgr2=0x7f5ff006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ff006b2f0 0x7f5ff006d7b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6008009a10 tx=0x7f6008008040 comp rx=0 tx=0).stop 2024-01-16T15:01:03.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 msgr2=0x7f60180cafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 0x7f60180cafa0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6000007b30 tx=0x7f600000e530 comp rx=0 tx=0).stop 2024-01-16T15:01:03.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 shutdown_connections 2024-01-16T15:01:03.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ff006b2f0 0x7f5ff006d7b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60180cb4e0 0x7f60180cddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60181e7100 0x7f60180cafa0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 --2- 172.21.15.53:0/2242050026 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ff8004d00 0x7f6018159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 >> 172.21.15.53:0/2242050026 conn(0x7f60180d3000 msgr2=0x7f601815c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 shutdown_connections 2024-01-16T15:01:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.171+0000 7f601dc00640 1 -- 172.21.15.53:0/2242050026 wait complete. 2024-01-16T15:01:03.169 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:01:03.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.412+0000 7f191de00640 1 Processor -- start 2024-01-16T15:01:03.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.412+0000 7f191de00640 1 -- start start 2024-01-16T15:01:03.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f191814ae80 0x7f1918071130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191de00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1918071670 con 0x7f191806d7a0 2024-01-16T15:01:03.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191de00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1918071910 con 0x7f19180b1ba0 2024-01-16T15:01:03.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191de00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19180ceab0 con 0x7f191814ae80 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191d400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f191806d7a0 0x7f191806bde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56038/0 (socket says 172.21.15.53:56038) 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191d400640 1 -- 172.21.15.53:0/2622827305 learned_addr learned my addr 172.21.15.53:0/2622827305 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.414+0000 7f191ca00640 1 --2- 172.21.15.53:0/2622827305 >> v2:172.21.15.146:3300/0 conn(0x7f191814ae80 0x7f1918071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.415+0000 7f1917e00640 1 -- 172.21.15.53:0/2622827305 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902030606 0 0) 0x7f1918071670 con 0x7f191806d7a0 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.415+0000 7f1917e00640 1 -- 172.21.15.53:0/2622827305 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f8003520 con 0x7f191806d7a0 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.415+0000 7f1917e00640 1 -- 172.21.15.53:0/2622827305 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 17023503 0 0) 0x7f18f8003520 con 0x7f191806d7a0 2024-01-16T15:01:03.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.415+0000 7f191ca00640 1 -- 172.21.15.53:0/2622827305 >> v1:172.21.15.146:6789/0 conn(0x7f19180b1ba0 legacy=0x7f191806d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.415+0000 7f191ca00640 1 -- 172.21.15.53:0/2622827305 >> v1:172.21.15.53:6789/0 conn(0x7f191806d7a0 legacy=0x7f191806bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.415+0000 7f191ca00640 1 -- 172.21.15.53:0/2622827305 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1918071bf0 con 0x7f191814ae80 2024-01-16T15:01:03.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.417+0000 7f191ca00640 1 --2- 172.21.15.53:0/2622827305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191814ae80 0x7f1918071130 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19080097a0 tx=0x7f190802b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c68d440faeea42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.417+0000 7f1917e00640 1 -- 172.21.15.53:0/2622827305 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f190802f050 con 0x7f191814ae80 2024-01-16T15:01:03.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.417+0000 7f1917e00640 1 -- 172.21.15.53:0/2622827305 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1908031500 con 0x7f191814ae80 2024-01-16T15:01:03.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.417+0000 7f1917e00640 1 -- 172.21.15.53:0/2622827305 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19100019f0 con 0x7f191814ae80 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 -- 172.21.15.53:0/2622827305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191814ae80 msgr2=0x7f1918071130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 --2- 172.21.15.53:0/2622827305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191814ae80 0x7f1918071130 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f19080097a0 tx=0x7f190802b820 comp rx=0 tx=0).stop 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 -- 172.21.15.53:0/2622827305 shutdown_connections 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 --2- 172.21.15.53:0/2622827305 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191814ae80 0x7f1918071130 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 -- 172.21.15.53:0/2622827305 >> 172.21.15.53:0/2622827305 conn(0x7f19180b0ed0 msgr2=0x7f19180cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 -- 172.21.15.53:0/2622827305 shutdown_connections 2024-01-16T15:01:03.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.418+0000 7f191de00640 1 -- 172.21.15.53:0/2622827305 wait complete. 2024-01-16T15:01:03.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.420+0000 7f191de00640 1 Processor -- start 2024-01-16T15:01:03.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.420+0000 7f191de00640 1 -- start start 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.421+0000 7f191de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191806d7a0 0x7f19181546f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.421+0000 7f191de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19180b1ba0 0x7f1918161bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.421+0000 7f191de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 0x7f1918164570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.421+0000 7f191de00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19181632f0 con 0x7f191814ae80 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.421+0000 7f191de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1918154c30 con 0x7f191806d7a0 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.421+0000 7f191de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1918162100 con 0x7f19180b1ba0 2024-01-16T15:01:03.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 0x7f1918164570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 0x7f1918164570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33482/0 (socket says 172.21.15.53:33482) 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 -- 172.21.15.53:0/3067724416 learned_addr learned my addr 172.21.15.53:0/3067724416 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191ca00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191806d7a0 0x7f19181546f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f190fe00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19180b1ba0 0x7f1918161bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 -- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19180b1ba0 msgr2=0x7f1918161bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19180b1ba0 0x7f1918161bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 -- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191806d7a0 msgr2=0x7f19181546f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191806d7a0 0x7f19181546f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 -- 172.21.15.53:0/3067724416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1908009400 con 0x7f191814ae80 2024-01-16T15:01:03.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.422+0000 7f191d400640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 0x7f1918164570 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f191000ccc0 tx=0x7f191000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.423+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1910012400 con 0x7f191814ae80 2024-01-16T15:01:03.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.423+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1910012560 con 0x7f191814ae80 2024-01-16T15:01:03.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.423+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19181626b0 con 0x7f191814ae80 2024-01-16T15:01:03.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.423+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19100128d0 con 0x7f191814ae80 2024-01-16T15:01:03.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.423+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1918162c30 con 0x7f191814ae80 2024-01-16T15:01:03.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.424+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1918163430 con 0x7f191814ae80 2024-01-16T15:01:03.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.425+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1910007070 con 0x7f191814ae80 2024-01-16T15:01:03.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.426+0000 7f1915600640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18f006b2c0 0x7f18f006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.426+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1910089e10 con 0x7f191814ae80 2024-01-16T15:01:03.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.427+0000 7f191ca00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18f006b2c0 0x7f18f006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.428+0000 7f191ca00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18f006b2c0 0x7f18f006d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1908035000 tx=0x7f1908008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.431+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1918163430 con 0x7f191814ae80 2024-01-16T15:01:03.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.623+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1918163430 con 0x7f191814ae80 2024-01-16T15:01:03.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.624+0000 7f1915600640 1 -- 172.21.15.53:0/3067724416 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v24) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1918163430 con 0x7f191814ae80 2024-01-16T15:01:03.604 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:01:03.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18f006b2c0 msgr2=0x7f18f006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18f006b2c0 0x7f18f006d780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1908035000 tx=0x7f1908008030 comp rx=0 tx=0).stop 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 msgr2=0x7f1918164570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 0x7f1918164570 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f191000ccc0 tx=0x7f191000a430 comp rx=0 tx=0).stop 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 shutdown_connections 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18f006b2c0 0x7f18f006d780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f191814ae80 0x7f1918164570 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19180b1ba0 0x7f1918161bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 --2- 172.21.15.53:0/3067724416 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f191806d7a0 0x7f19181546f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.626+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 >> 172.21.15.53:0/3067724416 conn(0x7f19180b0ed0 msgr2=0x7f19180b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.627+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 shutdown_connections 2024-01-16T15:01:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.627+0000 7f191de00640 1 -- 172.21.15.53:0/3067724416 wait complete. 2024-01-16T15:01:03.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:01:03.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.866+0000 7f7d6ee00640 1 Processor -- start 2024-01-16T15:01:03.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.866+0000 7f7d6ee00640 1 -- start start 2024-01-16T15:01:03.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 0x7f7d68070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6ee00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6815f4e0 con 0x7f7d680adf20 2024-01-16T15:01:03.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6ee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6815c3c0 con 0x7f7d6815bf10 2024-01-16T15:01:03.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6815f970 con 0x7f7d680ccb50 2024-01-16T15:01:03.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7d680adf20 0x7f7d680cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56052/0 (socket says 172.21.15.53:56052) 2024-01-16T15:01:03.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6da00640 1 -- 172.21.15.53:0/1850664716 learned_addr learned my addr 172.21.15.53:0/1850664716 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.868+0000 7f7d6d000640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 0x7f7d68070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319168346 0 0) 0x7f7d6815f4e0 con 0x7f7d680adf20 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d48003520 con 0x7f7d680adf20 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453461870 0 0) 0x7f7d48003520 con 0x7f7d680adf20 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 msgr2=0x7f7d68070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 0x7f7d68070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 >> v1:172.21.15.146:6789/0 conn(0x7f7d6815bf10 legacy=0x7f7d68143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68071130 con 0x7f7d680adf20 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7d58001670 con 0x7f7d680adf20 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 >> v1:172.21.15.53:6789/0 conn(0x7f7d680adf20 legacy=0x7f7d680cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d480039a0 0x7f7d48003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 0x7f7d480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6815f970 con 0x7f7d48004d00 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4800bb10 con 0x7f7d480039a0 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.869+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4800bcd0 con 0x7f7d680ccb50 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6da00640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d480039a0 0x7f7d48003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6d000640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 0x7f7d480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6e400640 1 -- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 msgr2=0x7f7d480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d680ccb50 0x7f7d480047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6e400640 1 -- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d480039a0 msgr2=0x7f7d48003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:03.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d480039a0 0x7f7d48003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.870+0000 7f7d6e400640 1 -- 172.21.15.53:0/1850664716 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48003520 con 0x7f7d48004d00 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.871+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7d60001e50 tx=0x7f7d480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d632ff478362cd5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.871+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60003b70 con 0x7f7d48004d00 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.871+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d60003cd0 con 0x7f7d48004d00 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.871+0000 7f7d67e00640 1 -- 172.21.15.53:0/1850664716 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d60005310 con 0x7f7d48004d00 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1850664716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d48004d00 msgr2=0x7f7d48005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1850664716 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d48004d00 0x7f7d48005180 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f7d60001e50 tx=0x7f7d480057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6e400640 1 -- 172.21.15.53:0/1850664716 reap_dead start 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1850664716 shutdown_connections 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1850664716 >> 172.21.15.53:0/1850664716 conn(0x7f7d6806a7d0 msgr2=0x7f7d6806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1850664716 shutdown_connections 2024-01-16T15:01:03.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.872+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1850664716 wait complete. 2024-01-16T15:01:03.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.874+0000 7f7d6ee00640 1 Processor -- start 2024-01-16T15:01:03.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.874+0000 7f7d6ee00640 1 -- start start 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d48004d00 0x7f7d6814aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d680c3d00 0x7f7d680c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 0x7f7d680c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d680c57f0 con 0x7f7d680c4180 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d680c4750 con 0x7f7d680c3d00 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d680c4a30 con 0x7f7d48004d00 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6e400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 0x7f7d680c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6e400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 0x7f7d680c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33502/0 (socket says 172.21.15.53:33502) 2024-01-16T15:01:03.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.875+0000 7f7d6e400640 1 -- 172.21.15.53:0/1592023469 learned_addr learned my addr 172.21.15.53:0/1592023469 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:03.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6d000640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d680c3d00 0x7f7d680c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6da00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d48004d00 0x7f7d6814aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6e400640 1 -- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d48004d00 msgr2=0x7f7d6814aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d48004d00 0x7f7d6814aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6e400640 1 -- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d680c3d00 msgr2=0x7f7d680c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d680c3d00 0x7f7d680c87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6e400640 1 -- 172.21.15.53:0/1592023469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d680c4d10 con 0x7f7d680c4180 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6e400640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 0x7f7d680c8cf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7d60001e50 tx=0x7f7d600014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6003e070 con 0x7f7d680c4180 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d60003e40 con 0x7f7d680c4180 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6002f660 con 0x7f7d680c4180 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.876+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d680c4ff0 con 0x7f7d680c4180 2024-01-16T15:01:03.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.877+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6814afe0 con 0x7f7d680c4180 2024-01-16T15:01:03.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.878+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d680c5510 con 0x7f7d680c4180 2024-01-16T15:01:03.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.879+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7d6002f7c0 con 0x7f7d680c4180 2024-01-16T15:01:03.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.879+0000 7f7d65600640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d4006b2f0 0x7f7d4006d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:03.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.880+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7d600aaf10 con 0x7f7d680c4180 2024-01-16T15:01:03.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.880+0000 7f7d6da00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d4006b2f0 0x7f7d4006d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:03.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.881+0000 7f7d6da00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d4006b2f0 0x7f7d4006d7b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d680ca610 tx=0x7f7d58009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:03.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:03.884+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d680c5510 con 0x7f7d680c4180 2024-01-16T15:01:04.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.076+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7d680c5510 con 0x7f7d680c4180 2024-01-16T15:01:04.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.077+0000 7f7d65600640 1 -- 172.21.15.53:0/1592023469 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v24) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7d680c5510 con 0x7f7d680c4180 2024-01-16T15:01:04.057 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.079+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d4006b2f0 msgr2=0x7f7d4006d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d4006b2f0 0x7f7d4006d7b0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7d680ca610 tx=0x7f7d58009040 comp rx=0 tx=0).stop 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 msgr2=0x7f7d680c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 0x7f7d680c8cf0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7d60001e50 tx=0x7f7d600014a0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 shutdown_connections 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d4006b2f0 0x7f7d4006d7b0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d680c4180 0x7f7d680c8cf0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d680c3d00 0x7f7d680c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 --2- 172.21.15.53:0/1592023469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d48004d00 0x7f7d6814aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 >> 172.21.15.53:0/1592023469 conn(0x7f7d6806a7d0 msgr2=0x7f7d6815cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 shutdown_connections 2024-01-16T15:01:04.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.080+0000 7f7d6ee00640 1 -- 172.21.15.53:0/1592023469 wait complete. 2024-01-16T15:01:04.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:04.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.307+0000 7effe3200640 1 Processor -- start 2024-01-16T15:01:04.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.307+0000 7effe3200640 1 -- start start 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.308+0000 7effe3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7effdc14b2c0 0x7effdc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.308+0000 7effe3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7effdc0cf7d0 0x7effdc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effe3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effdc0cfbf0 con 0x7effdc070ea0 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effe3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effdc14b6a0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effe3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effdc071690 con 0x7effdc14b2c0 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effd3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7effdc0cf7d0 0x7effdc070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effd3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7effdc0cf7d0 0x7effdc070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33514/0 (socket says 172.21.15.53:33514) 2024-01-16T15:01:04.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effd3200640 1 -- 172.21.15.53:0/1127261502 learned_addr learned my addr 172.21.15.53:0/1127261502 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:04.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.309+0000 7effdbe00640 1 --2- 172.21.15.53:0/1127261502 >> v2:172.21.15.146:3300/0 conn(0x7effdc14b2c0 0x7effdc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.310+0000 7effd3200640 1 -- 172.21.15.53:0/1127261502 >> v2:172.21.15.146:3300/0 conn(0x7effdc14b2c0 msgr2=0x7effdc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.310+0000 7effd3200640 1 --2- 172.21.15.53:0/1127261502 >> v2:172.21.15.146:3300/0 conn(0x7effdc14b2c0 0x7effdc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.310+0000 7effd3200640 1 -- 172.21.15.53:0/1127261502 >> v1:172.21.15.146:6789/0 conn(0x7effdc070ea0 legacy=0x7effdc0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.310+0000 7effd3200640 1 -- 172.21.15.53:0/1127261502 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effdc071970 con 0x7effdc0cf7d0 2024-01-16T15:01:04.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.310+0000 7effdbe00640 1 --2- 172.21.15.53:0/1127261502 >> v2:172.21.15.146:3300/0 conn(0x7effdc14b2c0 0x7effdc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.311+0000 7effd3200640 1 --2- 172.21.15.53:0/1127261502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc070960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7effdc06e650 tx=0x7effc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5153ba515a8cea14 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effdb400640 1 -- 172.21.15.53:0/1127261502 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4031870 con 0x7effdc0cf7d0 2024-01-16T15:01:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effdb400640 1 -- 172.21.15.53:0/1127261502 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc40319d0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effdb400640 1 -- 172.21.15.53:0/1127261502 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4031d60 con 0x7effdc0cf7d0 2024-01-16T15:01:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 -- 172.21.15.53:0/1127261502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 msgr2=0x7effdc070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 --2- 172.21.15.53:0/1127261502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc070960 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7effdc06e650 tx=0x7effc402b910 comp rx=0 tx=0).stop 2024-01-16T15:01:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 -- 172.21.15.53:0/1127261502 shutdown_connections 2024-01-16T15:01:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 --2- 172.21.15.53:0/1127261502 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc070960 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 --2- 172.21.15.53:0/1127261502 >> v2:172.21.15.146:3300/0 conn(0x7effdc14b2c0 0x7effdc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 -- 172.21.15.53:0/1127261502 >> 172.21.15.53:0/1127261502 conn(0x7effdc06a800 msgr2=0x7effdc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 -- 172.21.15.53:0/1127261502 shutdown_connections 2024-01-16T15:01:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.312+0000 7effe3200640 1 -- 172.21.15.53:0/1127261502 wait complete. 2024-01-16T15:01:04.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.314+0000 7effe3200640 1 Processor -- start 2024-01-16T15:01:04.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.314+0000 7effe3200640 1 -- start start 2024-01-16T15:01:04.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effe3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effdc070ea0 0x7effdc1613b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effe3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc144820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effe3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effdc14b2c0 0x7effdc142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effe3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc162280 con 0x7effdc0cf7d0 2024-01-16T15:01:04.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effe3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effdc142b70 con 0x7effdc070ea0 2024-01-16T15:01:04.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effe3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effdc144eb0 con 0x7effdc14b2c0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effd3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc144820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effd3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc144820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33520/0 (socket says 172.21.15.53:33520) 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.315+0000 7effd3200640 1 -- 172.21.15.53:0/1848368972 learned_addr learned my addr 172.21.15.53:0/1848368972 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effdbe00640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effdc070ea0 0x7effdc1613b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effe1200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effdc14b2c0 0x7effdc142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effd3200640 1 -- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effdc14b2c0 msgr2=0x7effdc142630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effd3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effdc14b2c0 0x7effdc142630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effd3200640 1 -- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effdc070ea0 msgr2=0x7effdc1613b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effd3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effdc070ea0 0x7effdc1613b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effd3200640 1 -- 172.21.15.53:0/1848368972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc40094d0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.316+0000 7effd3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc144820 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7effc4000c00 tx=0x7effc40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.317+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc403abe0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.317+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effc403ad40 con 0x7effdc0cf7d0 2024-01-16T15:01:04.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.317+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc4044610 con 0x7effdc0cf7d0 2024-01-16T15:01:04.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.317+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effdc145190 con 0x7effdc0cf7d0 2024-01-16T15:01:04.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.317+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effdc145470 con 0x7effdc0cf7d0 2024-01-16T15:01:04.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.318+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effdc1623f0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.319+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7effc404d450 con 0x7effdc0cf7d0 2024-01-16T15:01:04.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.320+0000 7effd8c00640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effb406b2c0 0x7effb406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.320+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7effc402d040 con 0x7effdc0cf7d0 2024-01-16T15:01:04.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.321+0000 7effdbe00640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effb406b2c0 0x7effb406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.322+0000 7effdbe00640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effb406b2c0 0x7effb406d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7effdc149ac0 tx=0x7effcc003a10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.324+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7effc4077080 con 0x7effdc0cf7d0 2024-01-16T15:01:04.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.562+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effdc0675d0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.563+0000 7effd8c00640 1 -- 172.21.15.53:0/1848368972 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7effc407a6a0 con 0x7effdc0cf7d0 2024-01-16T15:01:04.543 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:04.543 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:01:04.544 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:01:04.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effb406b2c0 msgr2=0x7effb406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effb406b2c0 0x7effb406d780 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7effdc149ac0 tx=0x7effcc003a10 comp rx=0 tx=0).stop 2024-01-16T15:01:04.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 msgr2=0x7effdc144820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc144820 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7effc4000c00 tx=0x7effc40046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 shutdown_connections 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7effb406b2c0 0x7effb406d780 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7effdc14b2c0 0x7effdc142630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7effdc0cf7d0 0x7effdc144820 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 --2- 172.21.15.53:0/1848368972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7effdc070ea0 0x7effdc1613b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 >> 172.21.15.53:0/1848368972 conn(0x7effdc06a800 msgr2=0x7effdc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 shutdown_connections 2024-01-16T15:01:04.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.566+0000 7effe3200640 1 -- 172.21.15.53:0/1848368972 wait complete. 2024-01-16T15:01:04.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:04.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.806+0000 7fe07e200640 1 Processor -- start 2024-01-16T15:01:04.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.807+0000 7fe07e200640 1 -- start start 2024-01-16T15:01:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.808+0000 7fe07e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0780b3f30 0x7fe0780d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.808+0000 7fe07e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0780d0a10 con 0x7fe078160920 2024-01-16T15:01:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.808+0000 7fe07e200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0780cf080 con 0x7fe0780d0100 2024-01-16T15:01:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.808+0000 7fe07e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0780c6be0 con 0x7fe0780b3f30 2024-01-16T15:01:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe07ce00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe078160920 0x7fe0780cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56066/0 (socket says 172.21.15.53:56066) 2024-01-16T15:01:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe07ce00640 1 -- 172.21.15.53:0/3541290923 learned_addr learned my addr 172.21.15.53:0/3541290923 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:04.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe07d800640 1 --2- 172.21.15.53:0/3541290923 >> v2:172.21.15.53:3300/0 conn(0x7fe0780b3f30 0x7fe0780d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe07d800640 1 -- 172.21.15.53:0/3541290923 >> v1:172.21.15.146:6789/0 conn(0x7fe0780d0100 legacy=0x7fe0780d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe07d800640 1 -- 172.21.15.53:0/3541290923 >> v1:172.21.15.53:6789/0 conn(0x7fe078160920 legacy=0x7fe0780cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe07d800640 1 -- 172.21.15.53:0/3541290923 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0780ce7e0 con 0x7fe0780b3f30 2024-01-16T15:01:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.809+0000 7fe077400640 1 -- 172.21.15.53:0/3541290923 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593379558 0 0) 0x7fe0780d0a10 con 0x7fe078160920 2024-01-16T15:01:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.810+0000 7fe07d800640 1 --2- 172.21.15.53:0/3541290923 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780b3f30 0x7fe0780d1650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe0700097a0 tx=0x7fe07002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=471344b4e7bbcad3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe077400640 1 -- 172.21.15.53:0/3541290923 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070031830 con 0x7fe0780b3f30 2024-01-16T15:01:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe077400640 1 -- 172.21.15.53:0/3541290923 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe070031990 con 0x7fe0780b3f30 2024-01-16T15:01:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe077400640 1 -- 172.21.15.53:0/3541290923 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070031d20 con 0x7fe0780b3f30 2024-01-16T15:01:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 -- 172.21.15.53:0/3541290923 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780b3f30 msgr2=0x7fe0780d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 --2- 172.21.15.53:0/3541290923 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780b3f30 0x7fe0780d1650 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe0700097a0 tx=0x7fe07002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 -- 172.21.15.53:0/3541290923 shutdown_connections 2024-01-16T15:01:04.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 --2- 172.21.15.53:0/3541290923 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780b3f30 0x7fe0780d1650 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 -- 172.21.15.53:0/3541290923 >> 172.21.15.53:0/3541290923 conn(0x7fe0780d3090 msgr2=0x7fe0780d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:04.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 -- 172.21.15.53:0/3541290923 shutdown_connections 2024-01-16T15:01:04.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.811+0000 7fe07e200640 1 -- 172.21.15.53:0/3541290923 wait complete. 2024-01-16T15:01:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.814+0000 7fe07e200640 1 Processor -- start 2024-01-16T15:01:04.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.814+0000 7fe07e200640 1 -- start start 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe07e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0780b3f30 0x7fe0780cb880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe07e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 0x7fe07814d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe07e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe078160920 0x7fe0780b7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe07e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07814db40 con 0x7fe0780d0100 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe07e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780cab30 con 0x7fe078160920 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe07e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0780b7980 con 0x7fe0780b3f30 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe077e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 0x7fe07814d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe077e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 0x7fe07814d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33544/0 (socket says 172.21.15.53:33544) 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.815+0000 7fe077e00640 1 -- 172.21.15.53:0/1798174638 learned_addr learned my addr 172.21.15.53:0/1798174638 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe07ce00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0780b3f30 0x7fe0780cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe07d800640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe078160920 0x7fe0780b7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe077e00640 1 -- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0780b3f30 msgr2=0x7fe0780cb880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe077e00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0780b3f30 0x7fe0780cb880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe077e00640 1 -- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe078160920 msgr2=0x7fe0780b7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe077e00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe078160920 0x7fe0780b7de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe077e00640 1 -- 172.21.15.53:0/1798174638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe070009400 con 0x7fe0780d0100 2024-01-16T15:01:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe07d800640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe078160920 0x7fe0780b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:04.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe077e00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 0x7fe07814d600 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe06800c7c0 tx=0x7fe06800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe06800aa50 con 0x7fe0780d0100 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe06800abb0 con 0x7fe0780d0100 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe068012420 con 0x7fe0780d0100 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.816+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0780cadc0 con 0x7fe0780d0100 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.817+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0780cb380 con 0x7fe0780d0100 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.818+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe078067450 con 0x7fe0780d0100 2024-01-16T15:01:04.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.819+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe068012580 con 0x7fe0780d0100 2024-01-16T15:01:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.820+0000 7fe074c00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe05006b2c0 0x7fe05006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.820+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe068089bf0 con 0x7fe0780d0100 2024-01-16T15:01:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.820+0000 7fe07ce00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe05006b2c0 0x7fe05006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.822+0000 7fe07ce00640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe05006b2c0 0x7fe05006d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe060000f60 tx=0x7fe060007c00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:04.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:04.825+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0680549e0 con 0x7fe0780d0100 2024-01-16T15:01:04.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.019+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe07808c7e0 con 0x7fe0780d0100 2024-01-16T15:01:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.021+0000 7fe074c00640 1 -- 172.21.15.53:0/1798174638 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7fe068016090 con 0x7fe0780d0100 2024-01-16T15:01:05.002 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:05.002 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe05006b2c0 msgr2=0x7fe05006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe05006b2c0 0x7fe05006d780 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe060000f60 tx=0x7fe060007c00 comp rx=0 tx=0).stop 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 msgr2=0x7fe07814d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 0x7fe07814d600 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe06800c7c0 tx=0x7fe06800cc80 comp rx=0 tx=0).stop 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 shutdown_connections 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe05006b2c0 0x7fe05006d780 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe078160920 0x7fe0780b7de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0780d0100 0x7fe07814d600 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 --2- 172.21.15.53:0/1798174638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0780b3f30 0x7fe0780cb880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 >> 172.21.15.53:0/1798174638 conn(0x7fe0780d3090 msgr2=0x7fe07806bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 shutdown_connections 2024-01-16T15:01:05.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.024+0000 7fe07e200640 1 -- 172.21.15.53:0/1798174638 wait complete. 2024-01-16T15:01:05.024 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T15:01:05.024 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:05.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.270+0000 7f7b29800640 1 Processor -- start 2024-01-16T15:01:05.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.271+0000 7f7b29800640 1 -- start start 2024-01-16T15:01:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.272+0000 7f7b29800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 0x7f7b2415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.272+0000 7f7b29800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240cdec0 con 0x7f7b2413fcb0 2024-01-16T15:01:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.272+0000 7f7b29800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b240ce1a0 con 0x7f7b240cd530 2024-01-16T15:01:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.272+0000 7f7b29800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240ce480 con 0x7f7b2413cfa0 2024-01-16T15:01:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.273+0000 7f7b28e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7b2413fcb0 0x7f7b240b2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56080/0 (socket says 172.21.15.53:56080) 2024-01-16T15:01:05.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.273+0000 7f7b28e00640 1 -- 172.21.15.53:0/3267600283 learned_addr learned my addr 172.21.15.53:0/3267600283 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:05.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.273+0000 7f7b23e00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 0x7f7b2415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.273+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462345950 0 0) 0x7f7b240cdec0 con 0x7f7b2413fcb0 2024-01-16T15:01:05.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.273+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b04003520 con 0x7f7b2413fcb0 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1498741137 0 0) 0x7f7b04003520 con 0x7f7b2413fcb0 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 msgr2=0x7f7b2415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 0x7f7b2415afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 >> v1:172.21.15.146:6789/0 conn(0x7f7b240cd530 legacy=0x7f7b2413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b240ce760 con 0x7f7b2413fcb0 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7b1c0019f0 con 0x7f7b2413fcb0 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 >> v1:172.21.15.53:6789/0 conn(0x7f7b2413fcb0 legacy=0x7f7b240b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b040039a0 0x7f7b04003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 0x7f7b040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b04004d00 0x7f7b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240ce480 con 0x7f7b04004d00 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0400dd20 con 0x7f7b040039a0 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0400dee0 con 0x7f7b2413cfa0 2024-01-16T15:01:05.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.274+0000 7f7b28e00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b04004d00 0x7f7b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b23e00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b040039a0 0x7f7b04003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b1be00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 0x7f7b040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b28e00640 1 -- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 msgr2=0x7f7b040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b28e00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b2413cfa0 0x7f7b040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b28e00640 1 -- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b040039a0 msgr2=0x7f7b04003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b28e00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b040039a0 0x7f7b04003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.275+0000 7f7b28e00640 1 -- 172.21.15.53:0/3267600283 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b04003520 con 0x7f7b04004d00 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.276+0000 7f7b28e00640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b04004d00 0x7f7b04005180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00a760 tx=0x7f7b040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f8ba14eb1fe8e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.276+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c034630 con 0x7f7b04004d00 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.276+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b1c02f070 con 0x7f7b04004d00 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.276+0000 7f7b23400640 1 -- 172.21.15.53:0/3267600283 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b1c0349b0 con 0x7f7b04004d00 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b29800640 1 -- 172.21.15.53:0/3267600283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b04004d00 msgr2=0x7f7b04005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b29800640 1 --2- 172.21.15.53:0/3267600283 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b04004d00 0x7f7b04005180 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7b1c00a760 tx=0x7f7b040057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b28e00640 1 -- 172.21.15.53:0/3267600283 reap_dead start 2024-01-16T15:01:05.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b29800640 1 -- 172.21.15.53:0/3267600283 shutdown_connections 2024-01-16T15:01:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b29800640 1 -- 172.21.15.53:0/3267600283 >> 172.21.15.53:0/3267600283 conn(0x7f7b2406a800 msgr2=0x7f7b2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b29800640 1 -- 172.21.15.53:0/3267600283 shutdown_connections 2024-01-16T15:01:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.277+0000 7f7b29800640 1 -- 172.21.15.53:0/3267600283 wait complete. 2024-01-16T15:01:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.279+0000 7f7b29800640 1 Processor -- start 2024-01-16T15:01:05.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.279+0000 7f7b29800640 1 -- start start 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b29800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b04004d00 0x7f7b240ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b29800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 0x7f7b2413c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b29800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b2415c570 0x7f7b24159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b29800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2415c150 con 0x7f7b2415d0d0 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b29800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2415b5a0 con 0x7f7b2415c570 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b29800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b240ae620 con 0x7f7b04004d00 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 0x7f7b2413c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 0x7f7b2413c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33570/0 (socket says 172.21.15.53:33570) 2024-01-16T15:01:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 -- 172.21.15.53:0/278284054 learned_addr learned my addr 172.21.15.53:0/278284054 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 -- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b04004d00 msgr2=0x7f7b240ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b04004d00 0x7f7b240ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 -- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b2415c570 msgr2=0x7f7b24159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b2415c570 0x7f7b24159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.280+0000 7f7b1be00640 1 -- 172.21.15.53:0/278284054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2415b7b0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.281+0000 7f7b1be00640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 0x7f7b2413c2e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00e4d0 tx=0x7f7b0c00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.281+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c0053a0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.281+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0c0035f0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.281+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b2415ba90 con 0x7f7b2415d0d0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.281+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0c003980 con 0x7f7b2415d0d0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.281+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b2413c820 con 0x7f7b2415d0d0 2024-01-16T15:01:05.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.282+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2413c9e0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.283+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7b0c003ae0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.284+0000 7f7b20c00640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7afc06b2c0 0x7f7afc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.284+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7b0c089970 con 0x7f7b2415d0d0 2024-01-16T15:01:05.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.285+0000 7f7b23e00640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7afc06b2c0 0x7f7afc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.286+0000 7f7b23e00640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7afc06b2c0 0x7f7afc06d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009910 tx=0x7f7b14008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.289+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b2413c9e0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.519+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7b2413c9e0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.521+0000 7f7b20c00640 1 -- 172.21.15.53:0/278284054 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f7b2413c9e0 con 0x7f7b2415d0d0 2024-01-16T15:01:05.501 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:05.501 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:01:05.501 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.523+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7afc06b2c0 msgr2=0x7f7afc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.523+0000 7f7b29800640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7afc06b2c0 0x7f7afc06d780 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7b14009910 tx=0x7f7b14008040 comp rx=0 tx=0).stop 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.523+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 msgr2=0x7f7b2413c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.523+0000 7f7b29800640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 0x7f7b2413c2e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c00e4d0 tx=0x7f7b0c00e990 comp rx=0 tx=0).stop 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 shutdown_connections 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7afc06b2c0 0x7f7afc06d780 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b2415c570 0x7f7b24159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b2415d0d0 0x7f7b2413c2e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 --2- 172.21.15.53:0/278284054 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b04004d00 0x7f7b240ae0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 >> 172.21.15.53:0/278284054 conn(0x7f7b2406a800 msgr2=0x7f7b240b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 shutdown_connections 2024-01-16T15:01:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.524+0000 7f7b29800640 1 -- 172.21.15.53:0/278284054 wait complete. 2024-01-16T15:01:05.523 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:05.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.753+0000 7f40ab200640 1 Processor -- start 2024-01-16T15:01:05.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.753+0000 7f40ab200640 1 -- start start 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40ab200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40ac071410 0x7f40ac146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40ab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 0x7f40ac0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40ab200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ac06cc20 con 0x7f40ac071d30 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40ab200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ac06cec0 con 0x7f40ac071410 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40ab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac06d1a0 con 0x7f40ac06c7c0 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40aa800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f40ac071d30 0x7f40ac0b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56090/0 (socket says 172.21.15.53:56090) 2024-01-16T15:01:05.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.755+0000 7f40aa800640 1 -- 172.21.15.53:0/2945636931 learned_addr learned my addr 172.21.15.53:0/2945636931 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:05.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a3e00640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 0x7f40ac0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9e00640 1 --2- 172.21.15.53:0/2945636931 >> v2:172.21.15.146:3300/0 conn(0x7f40ac071410 0x7f40ac146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056684452 0 0) 0x7f40ac06cc20 con 0x7f40ac071d30 2024-01-16T15:01:05.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c003520 con 0x7f40ac071d30 2024-01-16T15:01:05.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1395410692 0 0) 0x7f408c003520 con 0x7f40ac071d30 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 msgr2=0x7f40ac0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 0x7f40ac0717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 >> v2:172.21.15.146:3300/0 conn(0x7f40ac071410 msgr2=0x7f40ac146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 --2- 172.21.15.53:0/2945636931 >> v2:172.21.15.146:3300/0 conn(0x7f40ac071410 0x7f40ac146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac06d480 con 0x7f40ac071d30 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f40a40019f0 con 0x7f40ac071d30 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 >> v1:172.21.15.53:6789/0 conn(0x7f40ac071d30 legacy=0x7f40ac0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.756+0000 7f40a9400640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f408c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9400640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 0x7f408c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9400640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c004d00 0x7f408c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac06cec0 con 0x7f408c003970 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac06d1a0 con 0x7f408c004d00 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f408c00b980 con 0x7f40ac06c7c0 2024-01-16T15:01:05.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9e00640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f408c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9e00640 1 -- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 msgr2=0x7f408c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9e00640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac06c7c0 0x7f408c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9e00640 1 -- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c004d00 msgr2=0x7f408c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:05.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9e00640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f408c004d00 0x7f408c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.757+0000 7f40a9e00640 1 -- 172.21.15.53:0/2945636931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408c003520 con 0x7f408c003970 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.758+0000 7f40a9e00640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f408c003d90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f409c009af0 tx=0x7f408c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c056f8449a322715 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c02c710 con 0x7f408c003970 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c02c870 con 0x7f408c003970 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40a9400640 1 -- 172.21.15.53:0/2945636931 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c02cc00 con 0x7f408c003970 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40ab200640 1 -- 172.21.15.53:0/2945636931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 msgr2=0x7f408c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40ab200640 1 --2- 172.21.15.53:0/2945636931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f408c003d90 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f409c009af0 tx=0x7f408c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40aa800640 1 -- 172.21.15.53:0/2945636931 reap_dead start 2024-01-16T15:01:05.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40ab200640 1 -- 172.21.15.53:0/2945636931 shutdown_connections 2024-01-16T15:01:05.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40ab200640 1 -- 172.21.15.53:0/2945636931 >> 172.21.15.53:0/2945636931 conn(0x7f40ac06a800 msgr2=0x7f40ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40ab200640 1 -- 172.21.15.53:0/2945636931 shutdown_connections 2024-01-16T15:01:05.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.759+0000 7f40ab200640 1 -- 172.21.15.53:0/2945636931 wait complete. 2024-01-16T15:01:05.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.761+0000 7f40ab200640 1 Processor -- start 2024-01-16T15:01:05.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.761+0000 7f40ab200640 1 -- start start 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40ab200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f40ac13eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40ab200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40ac1f2d00 0x7f40ac14f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40ab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac1f3140 0x7f40ac14f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40ab200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac14fce0 con 0x7f408c003970 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40ab200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac13d240 con 0x7f40ac1f2d00 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40ab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ac13c630 con 0x7f40ac1f3140 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.762+0000 7f40a9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f40ac13eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f40ac13eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33600/0 (socket says 172.21.15.53:33600) 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 -- 172.21.15.53:0/1635932940 learned_addr learned my addr 172.21.15.53:0/1635932940 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:05.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a3e00640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40ac1f2d00 0x7f40ac14f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 -- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac1f3140 msgr2=0x7f40ac14f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac1f3140 0x7f40ac14f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 -- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40ac1f2d00 msgr2=0x7f40ac14f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40ac1f2d00 0x7f40ac14f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 -- 172.21.15.53:0/1635932940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ac13c910 con 0x7f408c003970 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.763+0000 7f40a9e00640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f40ac13eba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f409c037040 tx=0x7f409c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.764+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c003810 con 0x7f408c003970 2024-01-16T15:01:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.764+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c02de90 con 0x7f408c003970 2024-01-16T15:01:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.764+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c003b70 con 0x7f408c003970 2024-01-16T15:01:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.764+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ac13cbf0 con 0x7f408c003970 2024-01-16T15:01:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.764+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ac13f0e0 con 0x7f408c003970 2024-01-16T15:01:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.765+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ac13f330 con 0x7f408c003970 2024-01-16T15:01:05.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.766+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f409c003cd0 con 0x7f408c003970 2024-01-16T15:01:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.767+0000 7f40a2000640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f408406b2c0 0x7f408406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:05.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.767+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f409c0abbb0 con 0x7f408c003970 2024-01-16T15:01:05.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.768+0000 7f40a3e00640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f408406b2c0 0x7f408406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:05.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.769+0000 7f40a3e00640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f408406b2c0 0x7f408406d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0b01c0 tx=0x7f4094008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:05.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:05.772+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f409c076910 con 0x7f408c003970 2024-01-16T15:01:05.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.001+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40ac067450 con 0x7f408c003970 2024-01-16T15:01:05.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.003+0000 7f40a2000640 1 -- 172.21.15.53:0/1635932940 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4910 (secure 0 0 0) 0x7f409c039090 con 0x7f408c003970 2024-01-16T15:01:05.983 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:05.984 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4230,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1510361901","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1510361901},{"type":"v1","addr":"172.21.15.53:6833","nonce":1510361901}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/606495327","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":606495327},{"type":"v1","addr":"172.21.15.53:6835","nonce":606495327}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4242,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2276587918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2276587918},{"type":"v1","addr":"172.21.15.146:6836","nonce":2276587918}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T14:59:57.407704+0000","modified":"2024-01-16T15:00:00.492292+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4245},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4245":{"gid":4245,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3309916828","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3309916828},{"type":"v1","addr":"172.21.15.146:6837","nonce":3309916828}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-01-16T15:01:05.984 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 7 2024-01-16T15:01:05.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f408406b2c0 msgr2=0x7f408406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f408406b2c0 0x7f408406d780 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f40ac0b01c0 tx=0x7f4094008040 comp rx=0 tx=0).stop 2024-01-16T15:01:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 msgr2=0x7f40ac13eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f40ac13eba0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f409c037040 tx=0x7f409c004730 comp rx=0 tx=0).stop 2024-01-16T15:01:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 shutdown_connections 2024-01-16T15:01:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f408406b2c0 0x7f408406d780 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40ac1f3140 0x7f40ac14f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40ac1f2d00 0x7f40ac14f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 --2- 172.21.15.53:0/1635932940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f408c003970 0x7f40ac13eba0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:05.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 >> 172.21.15.53:0/1635932940 conn(0x7f40ac06a800 msgr2=0x7f40ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:05.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 shutdown_connections 2024-01-16T15:01:05.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.006+0000 7f40ab200640 1 -- 172.21.15.53:0/1635932940 wait complete. 2024-01-16T15:01:06.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.245+0000 7f925a200640 1 Processor -- start 2024-01-16T15:01:06.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.245+0000 7f925a200640 1 -- start start 2024-01-16T15:01:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.247+0000 7f925a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 0x7f92540b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.247+0000 7f925a200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92540d0f40 0x7f92540d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.247+0000 7f925a200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92540d0a10 con 0x7f9254160920 2024-01-16T15:01:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.247+0000 7f925a200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92540cf080 con 0x7f92540d0f40 2024-01-16T15:01:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.247+0000 7f925a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92540c6be0 con 0x7f92540d0100 2024-01-16T15:01:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9258e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9254160920 0x7f92540cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56096/0 (socket says 172.21.15.53:56096) 2024-01-16T15:01:06.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9258e00640 1 -- 172.21.15.53:0/3264456933 learned_addr learned my addr 172.21.15.53:0/3264456933 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596823729 0 0) 0x7f92540d0a10 con 0x7f9254160920 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9234003520 con 0x7f9254160920 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f924bc00640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 0x7f92540b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9259800640 1 --2- 172.21.15.53:0/3264456933 >> v2:172.21.15.146:3300/0 conn(0x7f92540d0f40 0x7f92540d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2068438558 0 0) 0x7f9234003520 con 0x7f9254160920 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 msgr2=0x7f92540b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 0x7f92540b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 >> v2:172.21.15.146:3300/0 conn(0x7f92540d0f40 msgr2=0x7f92540d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 --2- 172.21.15.53:0/3264456933 >> v2:172.21.15.146:3300/0 conn(0x7f92540d0f40 0x7f92540d13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92540ce7e0 con 0x7f9254160920 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.248+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9244001670 con 0x7f9254160920 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 >> v1:172.21.15.53:6789/0 conn(0x7f9254160920 legacy=0x7f92540cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 0x7f9234003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9234003fa0 0x7f92340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9234004d00 0x7f9234005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92540cf080 con 0x7f9234004d00 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92540c6be0 con 0x7f9234003fa0 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f923400ba80 con 0x7f92540d0100 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9259800640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9234004d00 0x7f9234005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9259800640 1 -- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 msgr2=0x7f9234003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f924bc00640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9234003fa0 0x7f92340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9259800640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f92540d0100 0x7f9234003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9259800640 1 -- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9234003fa0 msgr2=0x7f92340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9259800640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9234003fa0 0x7f92340047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.249+0000 7f9259800640 1 -- 172.21.15.53:0/3264456933 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9234003520 con 0x7f9234004d00 2024-01-16T15:01:06.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f9259800640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9234004d00 0x7f9234005180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f924c007360 tx=0x7f92340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114233cb5da46eae server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:06.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c02ccf0 con 0x7f9234004d00 2024-01-16T15:01:06.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c02ce50 con 0x7f9234004d00 2024-01-16T15:01:06.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f9253e00640 1 -- 172.21.15.53:0/3264456933 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c034660 con 0x7f9234004d00 2024-01-16T15:01:06.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f925a200640 1 -- 172.21.15.53:0/3264456933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9234004d00 msgr2=0x7f9234005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f925a200640 1 --2- 172.21.15.53:0/3264456933 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9234004d00 0x7f9234005180 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f924c007360 tx=0x7f92340057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f9259800640 1 -- 172.21.15.53:0/3264456933 reap_dead start 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f925a200640 1 -- 172.21.15.53:0/3264456933 shutdown_connections 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.251+0000 7f925a200640 1 -- 172.21.15.53:0/3264456933 >> 172.21.15.53:0/3264456933 conn(0x7f92540d3090 msgr2=0x7f92540d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.252+0000 7f925a200640 1 -- 172.21.15.53:0/3264456933 shutdown_connections 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.252+0000 7f925a200640 1 -- 172.21.15.53:0/3264456933 wait complete. 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.253+0000 7f925a200640 1 Processor -- start 2024-01-16T15:01:06.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.253+0000 7f925a200640 1 -- start start 2024-01-16T15:01:06.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.254+0000 7f925a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9234004d00 0x7f925414e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.254+0000 7f925a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540b76b0 0x7f925415c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.254+0000 7f925a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 0x7f925415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.254+0000 7f925a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92540cabf0 con 0x7f92540b7b30 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.254+0000 7f925a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f925415e610 con 0x7f92540b76b0 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.254+0000 7f925a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92540b8630 con 0x7f9234004d00 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 0x7f925415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 0x7f925415cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33622/0 (socket says 172.21.15.53:33622) 2024-01-16T15:01:06.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 -- 172.21.15.53:0/3497936708 learned_addr learned my addr 172.21.15.53:0/3497936708 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:06.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9258e00640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9234004d00 0x7f925414e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 -- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9234004d00 msgr2=0x7f925414e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9234004d00 0x7f925414e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 -- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540b76b0 msgr2=0x7f925415c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540b76b0 0x7f925415c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.255+0000 7f9259800640 1 -- 172.21.15.53:0/3497936708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f925415ebb0 con 0x7f92540b7b30 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.256+0000 7f9259800640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 0x7f925415cb30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f924c02dfd0 tx=0x7f924c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.256+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c045070 con 0x7f92540b7b30 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.256+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f924c003990 con 0x7f92540b7b30 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.256+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92540b8070 con 0x7f92540b7b30 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.256+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f924c034ea0 con 0x7f92540b7b30 2024-01-16T15:01:06.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.256+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f925415e840 con 0x7f92540b7b30 2024-01-16T15:01:06.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.257+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f925415ed40 con 0x7f92540b7b30 2024-01-16T15:01:06.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.258+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f924c03b040 con 0x7f92540b7b30 2024-01-16T15:01:06.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.259+0000 7f9251600640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b2c0 0x7f922c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.259+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f924c0ab440 con 0x7f92540b7b30 2024-01-16T15:01:06.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.260+0000 7f9258e00640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b2c0 0x7f922c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.261+0000 7f9258e00640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b2c0 0x7f922c06d780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9244007ab0 tx=0x7f9244009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:06.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.263+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f924c076190 con 0x7f92540b7b30 2024-01-16T15:01:06.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.463+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9254067450 con 0x7f92540b7b30 2024-01-16T15:01:06.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.465+0000 7f9251600640 1 -- 172.21.15.53:0/3497936708 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+16719 (secure 0 0 0) 0x7f924c0797b0 con 0x7f92540b7b30 2024-01-16T15:01:06.445 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:06.445 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":24,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:00:40.194304+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-01-16T14:59:55.861367+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-01-16T14:59:56.423962+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.12","mappings":[{"from":0,"to":2}]},{"pgid":"3.19","mappings":[{"from":3,"to":2}]},{"pgid":"3.1f","mappings":[{"from":5,"to":1}]},{"pgid":"3.2a","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:06.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.467+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b2c0 msgr2=0x7f922c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.467+0000 7f925a200640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b2c0 0x7f922c06d780 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f9244007ab0 tx=0x7f9244009040 comp rx=0 tx=0).stop 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.467+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 msgr2=0x7f925415cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.467+0000 7f925a200640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 0x7f925415cb30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f924c02dfd0 tx=0x7f924c004ab0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.467+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 shutdown_connections 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b2c0 0x7f922c06d780 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540b7b30 0x7f925415cb30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92540b76b0 0x7f925415c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 --2- 172.21.15.53:0/3497936708 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9234004d00 0x7f925414e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 >> 172.21.15.53:0/3497936708 conn(0x7f92540d3090 msgr2=0x7f925406c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 shutdown_connections 2024-01-16T15:01:06.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.468+0000 7f925a200640 1 -- 172.21.15.53:0/3497936708 wait complete. 2024-01-16T15:01:06.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-01-16T15:01:06.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.706+0000 7fe777400640 1 Processor -- start 2024-01-16T15:01:06.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.706+0000 7fe777400640 1 -- start start 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe777400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7700cc960 0x7fe770168080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe777400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700d1d10 con 0x7fe7700d1900 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe777400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe77006c520 con 0x7fe77015d440 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe777400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7700cfaf0 con 0x7fe7700cc960 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe776a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7700cc960 0x7fe770168080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe776a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7700cc960 0x7fe770168080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33624/0 (socket says 172.21.15.53:33624) 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe775600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe7700d1900 0x7fe77006be10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56108/0 (socket says 172.21.15.53:56108) 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.708+0000 7fe776a00640 1 -- 172.21.15.53:0/3258261432 learned_addr learned my addr 172.21.15.53:0/3258261432 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.709+0000 7fe776a00640 1 -- 172.21.15.53:0/3258261432 >> v1:172.21.15.146:6789/0 conn(0x7fe77015d440 legacy=0x7fe77006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.709+0000 7fe776a00640 1 -- 172.21.15.53:0/3258261432 >> v1:172.21.15.53:6789/0 conn(0x7fe7700d1900 legacy=0x7fe77006be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.709+0000 7fe776a00640 1 -- 172.21.15.53:0/3258261432 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7700cfd00 con 0x7fe7700cc960 2024-01-16T15:01:06.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.710+0000 7fe776a00640 1 --2- 172.21.15.53:0/3258261432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700cc960 0x7fe770168080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe768009870 tx=0x7fe76802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab66b6191ae22b3b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:06.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.710+0000 7fe774c00640 1 -- 172.21.15.53:0/3258261432 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768030860 con 0x7fe7700cc960 2024-01-16T15:01:06.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.710+0000 7fe774c00640 1 -- 172.21.15.53:0/3258261432 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7680309c0 con 0x7fe7700cc960 2024-01-16T15:01:06.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.710+0000 7fe774c00640 1 -- 172.21.15.53:0/3258261432 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768030d50 con 0x7fe7700cc960 2024-01-16T15:01:06.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 -- 172.21.15.53:0/3258261432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700cc960 msgr2=0x7fe770168080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 --2- 172.21.15.53:0/3258261432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700cc960 0x7fe770168080 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe768009870 tx=0x7fe76802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 -- 172.21.15.53:0/3258261432 shutdown_connections 2024-01-16T15:01:06.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 --2- 172.21.15.53:0/3258261432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700cc960 0x7fe770168080 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 -- 172.21.15.53:0/3258261432 >> 172.21.15.53:0/3258261432 conn(0x7fe7700d0de0 msgr2=0x7fe7700d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:06.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 -- 172.21.15.53:0/3258261432 shutdown_connections 2024-01-16T15:01:06.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.711+0000 7fe777400640 1 -- 172.21.15.53:0/3258261432 wait complete. 2024-01-16T15:01:06.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.713+0000 7fe777400640 1 Processor -- start 2024-01-16T15:01:06.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.713+0000 7fe777400640 1 -- start start 2024-01-16T15:01:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe777400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7700cc960 0x7fe77015b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe777400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 0x7fe770150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe777400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe77015d440 0x7fe770151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe777400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700c62e0 con 0x7fe7700d1900 2024-01-16T15:01:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe777400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700c6510 con 0x7fe77015d440 2024-01-16T15:01:06.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe777400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7700c67f0 con 0x7fe7700cc960 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.714+0000 7fe775600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 0x7fe770150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 0x7fe770150ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33640/0 (socket says 172.21.15.53:33640) 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 -- 172.21.15.53:0/1167234324 learned_addr learned my addr 172.21.15.53:0/1167234324 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe776000640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7700cc960 0x7fe77015b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe776a00640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe77015d440 0x7fe770151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 -- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7700cc960 msgr2=0x7fe77015b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7700cc960 0x7fe77015b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 -- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe77015d440 msgr2=0x7fe770151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe77015d440 0x7fe770151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 -- 172.21.15.53:0/1167234324 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680094d0 con 0x7fe7700d1900 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.715+0000 7fe775600640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 0x7fe770150ee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c4d0 tx=0x7fe75800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:06.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.716+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758007480 con 0x7fe7700d1900 2024-01-16T15:01:06.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.716+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7580075e0 con 0x7fe7700d1900 2024-01-16T15:01:06.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.716+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7580049e0 con 0x7fe7700d1900 2024-01-16T15:01:06.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.716+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7700c6ad0 con 0x7fe7700d1900 2024-01-16T15:01:06.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.716+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7700c7090 con 0x7fe7700d1900 2024-01-16T15:01:06.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.717+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7700c72a0 con 0x7fe7700d1900 2024-01-16T15:01:06.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.718+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fe758004b40 con 0x7fe7700d1900 2024-01-16T15:01:06.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.719+0000 7fe76e000640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe74806b2c0 0x7fe74806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:06.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.719+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe758088ce0 con 0x7fe7700d1900 2024-01-16T15:01:06.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.719+0000 7fe776000640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe74806b2c0 0x7fe74806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:06.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.720+0000 7fe776000640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe74806b2c0 0x7fe74806d780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe760005660 tx=0x7fe760003d90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:06.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.723+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe758053b70 con 0x7fe7700d1900 2024-01-16T15:01:06.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.953+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe770067450 con 0x7fe7700d1900 2024-01-16T15:01:06.934 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:01:06.954+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:01:06.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.958+0000 7fe76e000640 1 -- 172.21.15.53:0/1167234324 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe758018020 con 0x7fe7700d1900 2024-01-16T15:01:06.938 INFO:teuthology.orchestra.run.smithi053.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe74806b2c0 msgr2=0x7fe74806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe74806b2c0 0x7fe74806d780 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe760005660 tx=0x7fe760003d90 comp rx=0 tx=0).stop 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 msgr2=0x7fe770150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 0x7fe770150ee0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe75800c4d0 tx=0x7fe75800c990 comp rx=0 tx=0).stop 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 shutdown_connections 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe74806b2c0 0x7fe74806d780 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe77015d440 0x7fe770151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7700d1900 0x7fe770150ee0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 --2- 172.21.15.53:0/1167234324 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7700cc960 0x7fe77015b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:06.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 >> 172.21.15.53:0/1167234324 conn(0x7fe7700d0de0 msgr2=0x7fe77006fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:06.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 shutdown_connections 2024-01-16T15:01:06.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:06.961+0000 7fe777400640 1 -- 172.21.15.53:0/1167234324 wait complete. 2024-01-16T15:01:06.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-01-16T15:01:06.965 INFO:tasks.ceph.mds.b.smithi146.stderr: -4016> 2024-01-16T14:59:58.829+0000 7f1d13c00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:01:07.040 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:01:07.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.180+0000 7f2f00c00640 1 Processor -- start 2024-01-16T15:01:07.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.180+0000 7f2f00c00640 1 -- start start 2024-01-16T15:01:07.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.181+0000 7f2f00c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2efc160c90 0x7f2efc0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:07.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2f00c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2efc0c6c00 0x7f2efc0cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:07.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2f00c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2efc0cf6c0 con 0x7f2efc1608b0 2024-01-16T15:01:07.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2f00c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc0d1450 con 0x7f2efc160c90 2024-01-16T15:01:07.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2f00c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2efc0b4390 con 0x7f2efc0c6c00 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2ef3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2efc160c90 0x7f2efc0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2ef3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2efc160c90 0x7f2efc0d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33652/0 (socket says 172.21.15.53:33652) 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2ef3e00640 1 -- 172.21.15.53:0/3841520279 learned_addr learned my addr 172.21.15.53:0/3841520279 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2efbe00640 1 --2- 172.21.15.53:0/3841520279 >> v2:172.21.15.146:3300/0 conn(0x7f2efc0c6c00 0x7f2efc0cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2ef3e00640 1 -- 172.21.15.53:0/3841520279 >> v2:172.21.15.146:3300/0 conn(0x7f2efc0c6c00 msgr2=0x7f2efc0cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2ef3e00640 1 --2- 172.21.15.53:0/3841520279 >> v2:172.21.15.146:3300/0 conn(0x7f2efc0c6c00 0x7f2efc0cfb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.182+0000 7f2ef3e00640 1 -- 172.21.15.53:0/3841520279 >> v1:172.21.15.146:6789/0 conn(0x7f2efc1608b0 legacy=0x7f2efc0d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.183+0000 7f2ef3e00640 1 -- 172.21.15.53:0/3841520279 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0ce7e0 con 0x7f2efc160c90 2024-01-16T15:01:07.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.184+0000 7f2ef3e00640 1 --2- 172.21.15.53:0/3841520279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc0d3bd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2efc161070 tx=0x7f2ee402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f51c6c7a65d98a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:07.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.184+0000 7f2efaa00640 1 -- 172.21.15.53:0/3841520279 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4031750 con 0x7f2efc160c90 2024-01-16T15:01:07.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.184+0000 7f2efaa00640 1 -- 172.21.15.53:0/3841520279 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee40318b0 con 0x7f2efc160c90 2024-01-16T15:01:07.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.184+0000 7f2efaa00640 1 -- 172.21.15.53:0/3841520279 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee4031c40 con 0x7f2efc160c90 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 -- 172.21.15.53:0/3841520279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 msgr2=0x7f2efc0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 --2- 172.21.15.53:0/3841520279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc0d3bd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f2efc161070 tx=0x7f2ee402b7f0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 -- 172.21.15.53:0/3841520279 shutdown_connections 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 --2- 172.21.15.53:0/3841520279 >> v2:172.21.15.146:3300/0 conn(0x7f2efc0c6c00 0x7f2efc0cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 --2- 172.21.15.53:0/3841520279 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc0d3bd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 -- 172.21.15.53:0/3841520279 >> 172.21.15.53:0/3841520279 conn(0x7f2efc0d30b0 msgr2=0x7f2efc0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 -- 172.21.15.53:0/3841520279 shutdown_connections 2024-01-16T15:01:07.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.185+0000 7f2f00c00640 1 -- 172.21.15.53:0/3841520279 wait complete. 2024-01-16T15:01:07.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.187+0000 7f2f00c00640 1 Processor -- start 2024-01-16T15:01:07.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.187+0000 7f2f00c00640 1 -- start start 2024-01-16T15:01:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2f00c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2efc0c6c00 0x7f2efc0b7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2f00c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2efc1608b0 0x7f2efc1f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:07.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2f00c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2f00c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0cb220 con 0x7f2efc160c90 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2f00c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc1f2b90 con 0x7f2efc1608b0 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2f00c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2efc0b7de0 con 0x7f2efc0c6c00 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2efbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2efbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc153200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33666/0 (socket says 172.21.15.53:33666) 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2efbe00640 1 -- 172.21.15.53:0/2383980981 learned_addr learned my addr 172.21.15.53:0/2383980981 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2efb400640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2efc0c6c00 0x7f2efc0b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2efbe00640 1 -- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2efc0c6c00 msgr2=0x7f2efc0b7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2ef3e00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2efc1608b0 0x7f2efc1f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.188+0000 7f2efbe00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2efc0c6c00 0x7f2efc0b7790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2efbe00640 1 -- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2efc1608b0 msgr2=0x7f2efc1f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2efbe00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2efc1608b0 0x7f2efc1f2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2efbe00640 1 -- 172.21.15.53:0/2383980981 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee40094d0 con 0x7f2efc160c90 2024-01-16T15:01:07.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2efbe00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc153200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007e60 tx=0x7f2ef400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:07.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4014420 con 0x7f2efc160c90 2024-01-16T15:01:07.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef40145a0 con 0x7f2efc160c90 2024-01-16T15:01:07.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4014950 con 0x7f2efc160c90 2024-01-16T15:01:07.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2efc0b80c0 con 0x7f2efc160c90 2024-01-16T15:01:07.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.189+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2efc0cb6e0 con 0x7f2efc160c90 2024-01-16T15:01:07.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.190+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2efc0cb8d0 con 0x7f2efc160c90 2024-01-16T15:01:07.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.191+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ef4014ab0 con 0x7f2efc160c90 2024-01-16T15:01:07.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.192+0000 7f2ef3400640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed406b2f0 0x7f2ed406d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:07.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.192+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f2ef408a640 con 0x7f2efc160c90 2024-01-16T15:01:07.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.193+0000 7f2efb400640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed406b2f0 0x7f2ed406d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:07.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.194+0000 7f2efb400640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed406b2f0 0x7f2ed406d7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006fd0 tx=0x7f2eec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.196+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ef4055340 con 0x7f2efc160c90 2024-01-16T15:01:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.423+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2efc0b82b0 con 0x7f2efc160c90 2024-01-16T15:01:07.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.961+0000 7f2ef3400640 1 -- 172.21.15.53:0/2383980981 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2ef4058960 con 0x7f2efc160c90 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed406b2f0 msgr2=0x7f2ed406d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed406b2f0 0x7f2ed406d7b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006fd0 tx=0x7f2eec008040 comp rx=0 tx=0).stop 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 msgr2=0x7f2efc153200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc153200 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f2ef4007e60 tx=0x7f2ef400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 shutdown_connections 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed406b2f0 0x7f2ed406d7b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2efc160c90 0x7f2efc153200 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2efc1608b0 0x7f2efc1f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 --2- 172.21.15.53:0/2383980981 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2efc0c6c00 0x7f2efc0b7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.965+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 >> 172.21.15.53:0/2383980981 conn(0x7f2efc0d30b0 msgr2=0x7f2efc0c7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.966+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 shutdown_connections 2024-01-16T15:01:07.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:07.966+0000 7f2f00c00640 1 -- 172.21.15.53:0/2383980981 wait complete. 2024-01-16T15:01:07.967 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-01-16T15:01:08.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.215+0000 7f9643400640 1 Processor -- start 2024-01-16T15:01:08.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.215+0000 7f9643400640 1 -- start start 2024-01-16T15:01:08.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.216+0000 7f9643400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f963c13df80 0x7f963c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.216+0000 7f9643400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06c470 con 0x7f963c13e360 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.216+0000 7f9643400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963c06c750 con 0x7f963c0cf9a0 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.216+0000 7f9643400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963c06ca30 con 0x7f963c13df80 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.217+0000 7f9640a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f963c13df80 0x7f963c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.217+0000 7f9633c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f963c13e360 0x7f963c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56110/0 (socket says 172.21.15.53:56110) 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.217+0000 7f9633c00640 1 -- 172.21.15.53:0/1999831951 learned_addr learned my addr 172.21.15.53:0/1999831951 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.217+0000 7f9640a00640 1 -- 172.21.15.53:0/1999831951 >> v1:172.21.15.146:6789/0 conn(0x7f963c0cf9a0 legacy=0x7f963c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.217+0000 7f9640a00640 1 -- 172.21.15.53:0/1999831951 >> v1:172.21.15.53:6789/0 conn(0x7f963c13e360 legacy=0x7f963c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.217+0000 7f9640a00640 1 -- 172.21.15.53:0/1999831951 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963c06cd10 con 0x7f963c13df80 2024-01-16T15:01:08.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.218+0000 7f9640a00640 1 --2- 172.21.15.53:0/1999831951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13df80 0x7f963c15bee0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f962c0097a0 tx=0x7f962c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa36dc6f46b3f8f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.219+0000 7f963be00640 1 -- 172.21.15.53:0/1999831951 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c031830 con 0x7f963c13df80 2024-01-16T15:01:08.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.219+0000 7f963be00640 1 -- 172.21.15.53:0/1999831951 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962c031990 con 0x7f963c13df80 2024-01-16T15:01:08.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.219+0000 7f963be00640 1 -- 172.21.15.53:0/1999831951 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f962c031d20 con 0x7f963c13df80 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.219+0000 7f9643400640 1 -- 172.21.15.53:0/1999831951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13df80 msgr2=0x7f963c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.219+0000 7f9643400640 1 --2- 172.21.15.53:0/1999831951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13df80 0x7f963c15bee0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f962c0097a0 tx=0x7f962c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.220+0000 7f9643400640 1 -- 172.21.15.53:0/1999831951 shutdown_connections 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.220+0000 7f9643400640 1 --2- 172.21.15.53:0/1999831951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13df80 0x7f963c15bee0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.220+0000 7f9643400640 1 -- 172.21.15.53:0/1999831951 >> 172.21.15.53:0/1999831951 conn(0x7f963c06a800 msgr2=0x7f963c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.220+0000 7f9643400640 1 -- 172.21.15.53:0/1999831951 shutdown_connections 2024-01-16T15:01:08.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.220+0000 7f9643400640 1 -- 172.21.15.53:0/1999831951 wait complete. 2024-01-16T15:01:08.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.221+0000 7f9643400640 1 Processor -- start 2024-01-16T15:01:08.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.222+0000 7f9643400640 1 -- start start 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.222+0000 7f9643400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f963c0cf9a0 0x7f963c13ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9643400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f963c13df80 0x7f963c0b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9643400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 0x7f963c145a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9643400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c13c320 con 0x7f963c13e360 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9643400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f963c13b6a0 con 0x7f963c13df80 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9643400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963c13b980 con 0x7f963c0cf9a0 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 0x7f963c145a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 0x7f963c145a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33684/0 (socket says 172.21.15.53:33684) 2024-01-16T15:01:08.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 -- 172.21.15.53:0/2346775274 learned_addr learned my addr 172.21.15.53:0/2346775274 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:08.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9640a00640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f963c0cf9a0 0x7f963c13ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9633c00640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f963c13df80 0x7f963c0b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 -- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f963c0cf9a0 msgr2=0x7f963c13ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f963c0cf9a0 0x7f963c13ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 -- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f963c13df80 msgr2=0x7f963c0b3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f963c13df80 0x7f963c0b3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9641400640 1 -- 172.21.15.53:0/2346775274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962c009400 con 0x7f963c13e360 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.223+0000 7f9640a00640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f963c0cf9a0 0x7f963c13ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.224+0000 7f9641400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 0x7f963c145a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f963400b9e0 tx=0x7f963400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.224+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f963400cb20 con 0x7f963c13e360 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.224+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f963400cc80 con 0x7f963c13e360 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.224+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9634014680 con 0x7f963c13e360 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.224+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f963c13bc60 con 0x7f963c13e360 2024-01-16T15:01:08.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.224+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f963c13d390 con 0x7f963c13e360 2024-01-16T15:01:08.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.225+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963c13c460 con 0x7f963c13e360 2024-01-16T15:01:08.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.226+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f963401d3b0 con 0x7f963c13e360 2024-01-16T15:01:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.227+0000 7f9639600640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f961406b310 0x7f961406d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.227+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f9634089c30 con 0x7f963c13e360 2024-01-16T15:01:08.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.228+0000 7f9640a00640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f961406b310 0x7f961406d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:08.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.229+0000 7f9640a00640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f961406b310 0x7f961406d7d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f962c036000 tx=0x7f962c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:08.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.232+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f963c13c460 con 0x7f963c13e360 2024-01-16T15:01:08.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:08.427+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f963c13c460 con 0x7f963c13e360 2024-01-16T15:01:09.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.371+0000 7f9639600640 1 -- 172.21.15.53:0/2346775274 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v26) v1 ==== 181+0+0 (secure 0 0 0) 0x7f963c13c460 con 0x7f963c13e360 2024-01-16T15:01:09.352 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' removed 2024-01-16T15:01:09.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f961406b310 msgr2=0x7f961406d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f961406b310 0x7f961406d7d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f962c036000 tx=0x7f962c008030 comp rx=0 tx=0).stop 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 msgr2=0x7f963c145a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 0x7f963c145a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f963400b9e0 tx=0x7f963400bea0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 shutdown_connections 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f961406b310 0x7f961406d7d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f963c13e360 0x7f963c145a50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f963c13df80 0x7f963c0b3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 --2- 172.21.15.53:0/2346775274 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f963c0cf9a0 0x7f963c13ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.375+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 >> 172.21.15.53:0/2346775274 conn(0x7f963c06a800 msgr2=0x7f963c15a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.376+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 shutdown_connections 2024-01-16T15:01:09.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.376+0000 7f9643400640 1 -- 172.21.15.53:0/2346775274 wait complete. 2024-01-16T15:01:09.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-01-16T15:01:09.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.622+0000 7f2ef5200640 1 Processor -- start 2024-01-16T15:01:09.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.622+0000 7f2ef5200640 1 -- start start 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2ef5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 0x7f2ef00ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2ef5200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ef00ce310 0x7f2ef0166f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2ef5200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef00b1430 con 0x7f2ef00d0640 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2ef5200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef00d0c40 con 0x7f2ef00ce310 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2ef5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00d0f20 con 0x7f2ef00cd820 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2eeea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2ef00d0640 0x7f2ef00cdc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56112/0 (socket says 172.21.15.53:56112) 2024-01-16T15:01:09.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2eeea00640 1 -- 172.21.15.53:0/3829403816 learned_addr learned my addr 172.21.15.53:0/3829403816 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2eefe00640 1 --2- 172.21.15.53:0/3829403816 >> v2:172.21.15.146:3300/0 conn(0x7f2ef00ce310 0x7f2ef0166f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.624+0000 7f2eef400640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 0x7f2ef00ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673639316 0 0) 0x7f2ef00b1430 con 0x7f2ef00d0640 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ed0003520 con 0x7f2ef00d0640 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4129733920 0 0) 0x7f2ed0003520 con 0x7f2ef00d0640 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 msgr2=0x7f2ef00ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 0x7f2ef00ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 >> v2:172.21.15.146:3300/0 conn(0x7f2ef00ce310 msgr2=0x7f2ef0166f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 --2- 172.21.15.53:0/3829403816 >> v2:172.21.15.146:3300/0 conn(0x7f2ef00ce310 0x7f2ef0166f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef00d1200 con 0x7f2ef00d0640 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2ed80019f0 con 0x7f2ef00d0640 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 >> v1:172.21.15.53:6789/0 conn(0x7f2ef00d0640 legacy=0x7f2ef00cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed0003970 0x7f2ed0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 0x7f2ed00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0004d00 0x7f2ed0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00d0c40 con 0x7f2ed0003970 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef00d0f20 con 0x7f2ed0004d00 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed000b980 con 0x7f2ef00cd820 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.625+0000 7f2eef400640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed0003970 0x7f2ed0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.626+0000 7f2eeea00640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 0x7f2ed00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.626+0000 7f2eef400640 1 -- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 msgr2=0x7f2ed00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.626+0000 7f2eef400640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef00cd820 0x7f2ed00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.626+0000 7f2eef400640 1 -- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0004d00 msgr2=0x7f2ed0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.626+0000 7f2eef400640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0004d00 0x7f2ed0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.626+0000 7f2eef400640 1 -- 172.21.15.53:0/3829403816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed0003520 con 0x7f2ed0003970 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.627+0000 7f2eef400640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed0003970 0x7f2ed0003d90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009ac0 tx=0x7f2ed00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27cb25f7f5278dce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.627+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee002c6b0 con 0x7f2ed0003970 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.627+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee002c810 con 0x7f2ed0003970 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.627+0000 7f2eee000640 1 -- 172.21.15.53:0/3829403816 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee002cba0 con 0x7f2ed0003970 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2ef5200640 1 -- 172.21.15.53:0/3829403816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed0003970 msgr2=0x7f2ed0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3829403816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ed0003970 0x7f2ed0003d90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0009ac0 tx=0x7f2ed00057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2eefe00640 1 -- 172.21.15.53:0/3829403816 reap_dead start 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2ef5200640 1 -- 172.21.15.53:0/3829403816 shutdown_connections 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2ef5200640 1 -- 172.21.15.53:0/3829403816 >> 172.21.15.53:0/3829403816 conn(0x7f2ef0099a60 msgr2=0x7f2ef0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2ef5200640 1 -- 172.21.15.53:0/3829403816 shutdown_connections 2024-01-16T15:01:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.628+0000 7f2ef5200640 1 -- 172.21.15.53:0/3829403816 wait complete. 2024-01-16T15:01:09.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.630+0000 7f2ef5200640 1 Processor -- start 2024-01-16T15:01:09.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.630+0000 7f2ef5200640 1 -- start start 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2ef5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0003970 0x7f2ef01f6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2ef5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 0x7f2ef006bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2ef5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef01537a0 0x7f2ef0153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2ef5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef006c450 con 0x7f2ef006ba90 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2ef5200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef0143360 con 0x7f2ed0003970 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2ef5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef01456a0 con 0x7f2ef01537a0 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2eeea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 0x7f2ef006bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.631+0000 7f2eeea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 0x7f2ef006bf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33700/0 (socket says 172.21.15.53:33700) 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 -- 172.21.15.53:0/3538014777 learned_addr learned my addr 172.21.15.53:0/3538014777 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eef400640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0003970 0x7f2ef01f6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eefe00640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef01537a0 0x7f2ef0153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 -- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef01537a0 msgr2=0x7f2ef0153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef01537a0 0x7f2ef0153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 -- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0003970 msgr2=0x7f2ef01f6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0003970 0x7f2ef01f6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 -- 172.21.15.53:0/3538014777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef0154160 con 0x7f2ef006ba90 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.632+0000 7f2eeea00640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 0x7f2ef006bf10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8002180 tx=0x7f2ed800b560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.633+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8004d60 con 0x7f2ef006ba90 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.633+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed80070d0 con 0x7f2ef006ba90 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.633+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed8007480 con 0x7f2ef006ba90 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.633+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef01458f0 con 0x7f2ef006ba90 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.633+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef0145eb0 con 0x7f2ef006ba90 2024-01-16T15:01:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.634+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef0067450 con 0x7f2ef006ba90 2024-01-16T15:01:09.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.635+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ed80075e0 con 0x7f2ef006ba90 2024-01-16T15:01:09.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.636+0000 7f2ee7400640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ec806b2c0 0x7f2ec806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:09.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.636+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 5473+0+0 (secure 0 0 0) 0x7f2ed8089bb0 con 0x7f2ef006ba90 2024-01-16T15:01:09.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.637+0000 7f2eef400640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ec806b2c0 0x7f2ec806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:09.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.638+0000 7f2eef400640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ec806b2c0 0x7f2ec806d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0036000 tx=0x7f2ee0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.641+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ed8054a60 con 0x7f2ef006ba90 2024-01-16T15:01:09.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:09.836+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2ef0066640 con 0x7f2ef006ba90 2024-01-16T15:01:10.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.375+0000 7f2ee7400640 1 -- 172.21.15.53:0/3538014777 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v27) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2ed8019020 con 0x7f2ef006ba90 2024-01-16T15:01:10.355 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' removed 2024-01-16T15:01:10.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.378+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ec806b2c0 msgr2=0x7f2ec806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.378+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ec806b2c0 0x7f2ec806d780 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0036000 tx=0x7f2ee0030000 comp rx=0 tx=0).stop 2024-01-16T15:01:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.378+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 msgr2=0x7f2ef006bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.378+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 0x7f2ef006bf10 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8002180 tx=0x7f2ed800b560 comp rx=0 tx=0).stop 2024-01-16T15:01:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.379+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 shutdown_connections 2024-01-16T15:01:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.379+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ec806b2c0 0x7f2ec806d780 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.379+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef01537a0 0x7f2ef0153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.379+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef006ba90 0x7f2ef006bf10 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.379+0000 7f2ef5200640 1 --2- 172.21.15.53:0/3538014777 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ed0003970 0x7f2ef01f6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.379+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 >> 172.21.15.53:0/3538014777 conn(0x7f2ef0099a60 msgr2=0x7f2ef006f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.380+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 shutdown_connections 2024-01-16T15:01:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.381+0000 7f2ef5200640 1 -- 172.21.15.53:0/3538014777 wait complete. 2024-01-16T15:01:10.387 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:01:10.387 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:01:10.387 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:01:10.387 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:10.387 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:01:10.387 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:01:10.388 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:01:10.388 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:10.388 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:01:10.388 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:01:10.388 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:01:10.388 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:10.388 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:01:10.388 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:01:10.389 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:01:10.389 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:01:10.406 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:01:10.406 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:01:10.412 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:01:10.412 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:01:10.412 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:01:10.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:01:10.414 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:01:10.414 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:01:10.460 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:01:10.460 DEBUG:teuthology.parallel:result is None 2024-01-16T15:01:10.461 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:01:10.461 DEBUG:teuthology.parallel:result is None 2024-01-16T15:01:10.465 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:01:10.466 DEBUG:teuthology.parallel:result is None 2024-01-16T15:01:10.467 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:01:10.467 DEBUG:teuthology.parallel:result is None 2024-01-16T15:01:10.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:01:10.619 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:01:10.619 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:01:10.620 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:01:10.620 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:01:10.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.678+0000 7f280d800640 1 Processor -- start 2024-01-16T15:01:10.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.678+0000 7f280d800640 1 -- start start 2024-01-16T15:01:10.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.679+0000 7f280d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f280806c6b0 0x7f28080b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.679+0000 7f280d800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280806cea0 con 0x7f28080cd240 2024-01-16T15:01:10.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.679+0000 7f280d800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280806cbc0 con 0x7f280809a170 2024-01-16T15:01:10.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.679+0000 7f280d800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280806f350 con 0x7f280806c6b0 2024-01-16T15:01:10.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.679+0000 7f2806a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f28080cd240 0x7f2808099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56122/0 (socket says 172.21.15.53:56122) 2024-01-16T15:01:10.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.679+0000 7f2806a00640 1 -- 172.21.15.53:0/924924578 learned_addr learned my addr 172.21.15.53:0/924924578 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:10.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2807400640 1 --2- 172.21.15.53:0/924924578 >> v2:172.21.15.146:3300/0 conn(0x7f280806c6b0 0x7f28080b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2806000640 1 -- 172.21.15.53:0/924924578 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2090165539 0 0) 0x7f280806cea0 con 0x7f28080cd240 2024-01-16T15:01:10.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2806000640 1 -- 172.21.15.53:0/924924578 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27e8003520 con 0x7f28080cd240 2024-01-16T15:01:10.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2806000640 1 -- 172.21.15.53:0/924924578 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094305420 0 0) 0x7f27e8003520 con 0x7f28080cd240 2024-01-16T15:01:10.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2807400640 1 -- 172.21.15.53:0/924924578 >> v1:172.21.15.146:6789/0 conn(0x7f280809a170 legacy=0x7f28080cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2807400640 1 -- 172.21.15.53:0/924924578 >> v1:172.21.15.53:6789/0 conn(0x7f28080cd240 legacy=0x7f2808099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.680+0000 7f2807400640 1 -- 172.21.15.53:0/924924578 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280806f630 con 0x7f280806c6b0 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.681+0000 7f2807400640 1 --2- 172.21.15.53:0/924924578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080b3820 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2800009d90 tx=0x7f280002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5e1481abb3440a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f2806000640 1 -- 172.21.15.53:0/924924578 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8001670 con 0x7f280806c6b0 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f2806000640 1 -- 172.21.15.53:0/924924578 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f280002c940 con 0x7f280806c6b0 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f2806000640 1 -- 172.21.15.53:0/924924578 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f280002ccd0 con 0x7f280806c6b0 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f280d800640 1 -- 172.21.15.53:0/924924578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 msgr2=0x7f28080b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f280d800640 1 --2- 172.21.15.53:0/924924578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080b3820 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2800009d90 tx=0x7f280002be10 comp rx=0 tx=0).stop 2024-01-16T15:01:10.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f280d800640 1 -- 172.21.15.53:0/924924578 shutdown_connections 2024-01-16T15:01:10.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f280d800640 1 --2- 172.21.15.53:0/924924578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080b3820 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f280d800640 1 -- 172.21.15.53:0/924924578 >> 172.21.15.53:0/924924578 conn(0x7f28080b2d00 msgr2=0x7f28080b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:10.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.682+0000 7f280d800640 1 -- 172.21.15.53:0/924924578 shutdown_connections 2024-01-16T15:01:10.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.683+0000 7f280d800640 1 -- 172.21.15.53:0/924924578 wait complete. 2024-01-16T15:01:10.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.684+0000 7f280d800640 1 Processor -- start 2024-01-16T15:01:10.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.684+0000 7f280d800640 1 -- start start 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.685+0000 7f280d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.685+0000 7f280d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280809a170 0x7f28080b58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.685+0000 7f280d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28080cd240 0x7f28080b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.685+0000 7f280d800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28081f6800 con 0x7f280809a170 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.685+0000 7f280d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28081f6ae0 con 0x7f280806c6b0 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.685+0000 7f280d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28081f6dc0 con 0x7f28080cd240 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f27fde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280809a170 0x7f28080b58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:33160/0 (socket says 172.21.15.53:33160) 2024-01-16T15:01:10.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 -- 172.21.15.53:0/3375501230 learned_addr learned my addr 172.21.15.53:0/3375501230 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2807400640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28080cd240 0x7f28080b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 -- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28080cd240 msgr2=0x7f28080b5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28080cd240 0x7f28080b5e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 -- 172.21.15.53:0/3375501230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280809a170 msgr2=0x7f28080b58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280809a170 0x7f28080b58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 -- 172.21.15.53:0/3375501230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2800009a70 con 0x7f280806c6b0 2024-01-16T15:01:10.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.686+0000 7f2806a00640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080cdd90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f27f800c730 tx=0x7f27f800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:10.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.687+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8005ad0 con 0x7f280806c6b0 2024-01-16T15:01:10.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.687+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f8005c50 con 0x7f280806c6b0 2024-01-16T15:01:10.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.687+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28081f70a0 con 0x7f280806c6b0 2024-01-16T15:01:10.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.687+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8007070 con 0x7f280806c6b0 2024-01-16T15:01:10.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.687+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28080ce2d0 con 0x7f280806c6b0 2024-01-16T15:01:10.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.688+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280806d0d0 con 0x7f280806c6b0 2024-01-16T15:01:10.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.689+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f27f80071d0 con 0x7f280806c6b0 2024-01-16T15:01:10.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.690+0000 7f27ff400640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b2c0 0x7f27e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:10.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.690+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f27f8088ff0 con 0x7f280806c6b0 2024-01-16T15:01:10.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.691+0000 7f27fde00640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b2c0 0x7f27e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:10.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.691+0000 7f27fde00640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b2c0 0x7f27e006d780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f28080ce9a0 tx=0x7f27f0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:10.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.697+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27f80541d0 con 0x7f280806c6b0 2024-01-16T15:01:10.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:10.918+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f28080675d0 con 0x7f280806c6b0 2024-01-16T15:01:11.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.422+0000 7f27ff400640 1 -- 172.21.15.53:0/3375501230 <== mon.1 v2:172.21.15.146:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f27f80577f0 con 0x7f280806c6b0 2024-01-16T15:01:11.402 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b2c0 msgr2=0x7f27e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b2c0 0x7f27e006d780 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f28080ce9a0 tx=0x7f27f0009040 comp rx=0 tx=0).stop 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 msgr2=0x7f28080cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080cdd90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f27f800c730 tx=0x7f27f800cbf0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 shutdown_connections 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b2c0 0x7f27e006d780 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28080cd240 0x7f28080b5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280809a170 0x7f28080b58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 --2- 172.21.15.53:0/3375501230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280806c6b0 0x7f28080cdd90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 >> 172.21.15.53:0/3375501230 conn(0x7f28080b2d00 msgr2=0x7f2808099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:11.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 shutdown_connections 2024-01-16T15:01:11.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.427+0000 7f280d800640 1 -- 172.21.15.53:0/3375501230 wait complete. 2024-01-16T15:01:11.433 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:01:11.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.658+0000 7fbfa5400640 1 Processor -- start 2024-01-16T15:01:11.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.659+0000 7fbfa5400640 1 -- start start 2024-01-16T15:01:11.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa5400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbfa00cf9f0 0x7fbfa00cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa5400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbfa0161010 0x7fbfa01614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa5400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa00d03b0 con 0x7fbfa0161010 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa5400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfa00d0a10 con 0x7fbfa00cf9f0 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa5400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa00cf080 con 0x7fbfa0160920 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbfa0161010 0x7fbfa01614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.660+0000 7fbfa4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbfa0161010 0x7fbfa01614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33746/0 (socket says 172.21.15.53:33746) 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbfa4a00640 1 -- 172.21.15.53:0/1075050744 learned_addr learned my addr 172.21.15.53:0/1075050744 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbf9fe00640 1 --2- 172.21.15.53:0/1075050744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbf9f400640 1 --2- 172.21.15.53:0/1075050744 >> v2:172.21.15.146:3300/0 conn(0x7fbfa00cf9f0 0x7fbfa00cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbfa4a00640 1 -- 172.21.15.53:0/1075050744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 msgr2=0x7fbfa00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbfa4a00640 1 --2- 172.21.15.53:0/1075050744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa00b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbfa4a00640 1 -- 172.21.15.53:0/1075050744 >> v2:172.21.15.146:3300/0 conn(0x7fbfa00cf9f0 msgr2=0x7fbfa00cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbfa4a00640 1 --2- 172.21.15.53:0/1075050744 >> v2:172.21.15.146:3300/0 conn(0x7fbfa00cf9f0 0x7fbfa00cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbfa4a00640 1 -- 172.21.15.53:0/1075050744 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfa00c6be0 con 0x7fbfa0161010 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.661+0000 7fbf9f400640 1 --2- 172.21.15.53:0/1075050744 >> v2:172.21.15.146:3300/0 conn(0x7fbfa00cf9f0 0x7fbfa00cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:11.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.662+0000 7fbfa4a00640 1 --2- 172.21.15.53:0/1075050744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa01614a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbfa008d300 tx=0x7fbf9802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6654d8d0ffa5a0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:11.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.662+0000 7fbf9ea00640 1 -- 172.21.15.53:0/1075050744 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf98031880 con 0x7fbfa0161010 2024-01-16T15:01:11.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.662+0000 7fbf9ea00640 1 -- 172.21.15.53:0/1075050744 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf980319e0 con 0x7fbfa0161010 2024-01-16T15:01:11.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.662+0000 7fbf9ea00640 1 -- 172.21.15.53:0/1075050744 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf98031d70 con 0x7fbfa0161010 2024-01-16T15:01:11.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 -- 172.21.15.53:0/1075050744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 msgr2=0x7fbfa01614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 --2- 172.21.15.53:0/1075050744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa01614a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fbfa008d300 tx=0x7fbf9802b920 comp rx=0 tx=0).stop 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 -- 172.21.15.53:0/1075050744 shutdown_connections 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 --2- 172.21.15.53:0/1075050744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa01614a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 --2- 172.21.15.53:0/1075050744 >> v2:172.21.15.146:3300/0 conn(0x7fbfa00cf9f0 0x7fbfa00cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 --2- 172.21.15.53:0/1075050744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 -- 172.21.15.53:0/1075050744 >> 172.21.15.53:0/1075050744 conn(0x7fbfa00d3090 msgr2=0x7fbfa00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.663+0000 7fbfa5400640 1 -- 172.21.15.53:0/1075050744 shutdown_connections 2024-01-16T15:01:11.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.664+0000 7fbfa5400640 1 -- 172.21.15.53:0/1075050744 wait complete. 2024-01-16T15:01:11.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.665+0000 7fbfa5400640 1 Processor -- start 2024-01-16T15:01:11.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.665+0000 7fbfa5400640 1 -- start start 2024-01-16T15:01:11.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.666+0000 7fbfa5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa00cf9f0 0x7fbfa00cb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.666+0000 7fbfa5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa015e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.666+0000 7fbfa5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa015e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.666+0000 7fbfa5400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa015f1c0 con 0x7fbfa0161010 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.666+0000 7fbfa5400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa015f4a0 con 0x7fbfa00cf9f0 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.666+0000 7fbfa5400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfa015f780 con 0x7fbfa0160920 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa015e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa015e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33752/0 (socket says 172.21.15.53:33752) 2024-01-16T15:01:11.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 -- 172.21.15.53:0/992245575 learned_addr learned my addr 172.21.15.53:0/992245575 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbf9fe00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa00cf9f0 0x7fbfa00cb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbf9f400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa015e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 -- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 msgr2=0x7fbfa015e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa015e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 -- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa00cf9f0 msgr2=0x7fbfa00cb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa00cf9f0 0x7fbfa00cb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.667+0000 7fbfa4a00640 1 -- 172.21.15.53:0/992245575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf980094d0 con 0x7fbfa0161010 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.668+0000 7fbfa4a00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa015e940 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbf980090b0 tx=0x7fbf980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.668+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf980443f0 con 0x7fbfa0161010 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.668+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf98044570 con 0x7fbfa0161010 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.668+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf98044920 con 0x7fbfa0161010 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.668+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfa015fa60 con 0x7fbfa0161010 2024-01-16T15:01:11.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.668+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfa015ee80 con 0x7fbfa0161010 2024-01-16T15:01:11.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.670+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfa015ff80 con 0x7fbfa0161010 2024-01-16T15:01:11.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.671+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fbf9804d450 con 0x7fbfa0161010 2024-01-16T15:01:11.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.671+0000 7fbf97e00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf7806b2c0 0x7fbf7806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:11.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.672+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbf980ac0b0 con 0x7fbfa0161010 2024-01-16T15:01:11.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.672+0000 7fbf9fe00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf7806b2c0 0x7fbf7806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:11.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.673+0000 7fbf9fe00640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf7806b2c0 0x7fbf7806d780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbfa014c2f0 tx=0x7fbf90008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:11.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.676+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbfa015ff80 con 0x7fbfa0161010 2024-01-16T15:01:11.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.944+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fbfa015ff80 con 0x7fbfa0161010 2024-01-16T15:01:11.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.945+0000 7fbf97e00640 1 -- 172.21.15.53:0/992245575 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fbfa015ff80 con 0x7fbfa0161010 2024-01-16T15:01:11.925 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:11.925 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:01:11.925 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:01:11.925 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.926 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.927 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:01:11.928 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.929 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.930 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:01:11.931 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.932 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.933 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.934 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.935 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.936 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.bootstrap-mds", 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDUmaZlKdrHOhAAuPvfrAjVrPdS1ilKDQlNOA==", 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile bootstrap-mds" 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.bootstrap-mgr", 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDUmaZlAOTHOhAADIzrwUQ5+3VWF74whjM4iQ==", 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile bootstrap-mgr" 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.937 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.bootstrap-osd", 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDUmaZlaOzHOhAAFguOBnuI3mAotveCFAiWcg==", 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile bootstrap-osd" 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.bootstrap-rbd", 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDUmaZlNfXHOhAACSLDNbwDdo/JsioDVlmvVg==", 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile bootstrap-rbd" 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.938 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDUmaZlDP7HOhAAtDKSQWh7gLWCe38BzcPBAw==", 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.bootstrap-rgw", 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDUmaZlDQfIOhAA+yTU8VLSaETUx0wcSbXjow==", 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile bootstrap-rgw" 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.939 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.940 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:11.941 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:01:11.941 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.948+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf7806b2c0 msgr2=0x7fbf7806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.948+0000 7fbfa5400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf7806b2c0 0x7fbf7806d780 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbfa014c2f0 tx=0x7fbf90008040 comp rx=0 tx=0).stop 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.948+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 msgr2=0x7fbfa015e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.948+0000 7fbfa5400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa015e940 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fbf980090b0 tx=0x7fbf980046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 shutdown_connections 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf7806b2c0 0x7fbf7806d780 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbfa0161010 0x7fbfa015e940 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbfa0160920 0x7fbfa015e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 --2- 172.21.15.53:0/992245575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbfa00cf9f0 0x7fbfa00cb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 >> 172.21.15.53:0/992245575 conn(0x7fbfa00d3090 msgr2=0x7fbfa006bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 shutdown_connections 2024-01-16T15:01:11.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:11.949+0000 7fbfa5400640 1 -- 172.21.15.53:0/992245575 wait complete. 2024-01-16T15:01:11.946 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-01-16T15:01:12.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.189+0000 7f83a4000640 1 Processor -- start 2024-01-16T15:01:12.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.190+0000 7f83a4000640 1 -- start start 2024-01-16T15:01:12.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.191+0000 7f83a4000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c0cf9a0 0x7f839c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.191+0000 7f83a4000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f839c15e1e0 0x7f839c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.191+0000 7f83a4000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c0b1ee0 con 0x7f839c143ae0 2024-01-16T15:01:12.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.191+0000 7f83a4000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f839c0b2780 con 0x7f839c15e1e0 2024-01-16T15:01:12.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.191+0000 7f83a4000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c15e990 con 0x7f839c0cf9a0 2024-01-16T15:01:12.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.192+0000 7f83a2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f839c15e1e0 0x7f839c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.192+0000 7f83a2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f839c15e1e0 0x7f839c0702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:56254/0 (socket says 172.21.15.53:56254) 2024-01-16T15:01:12.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.192+0000 7f83a2000640 1 -- 172.21.15.53:0/1183584293 learned_addr learned my addr 172.21.15.53:0/1183584293 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:12.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.192+0000 7f83a0c00640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c0cf9a0 0x7f839c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.193+0000 7f83a2000640 1 -- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c0cf9a0 msgr2=0x7f839c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:12.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.193+0000 7f83a2000640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c0cf9a0 0x7f839c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.193+0000 7f83a2000640 1 -- 172.21.15.53:0/1183584293 >> v1:172.21.15.146:6789/0 conn(0x7f839c143ae0 legacy=0x7f839c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.193+0000 7f83a2000640 1 -- 172.21.15.53:0/1183584293 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c0b2130 con 0x7f839c15e1e0 2024-01-16T15:01:12.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.193+0000 7f83a0c00640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c0cf9a0 0x7f839c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:12.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.195+0000 7f83a2000640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c15e1e0 0x7f839c0702c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8394009870 tx=0x7f839402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebe673089d7e5e11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:12.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.195+0000 7f839be00640 1 -- 172.21.15.53:0/1183584293 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394030840 con 0x7f839c15e1e0 2024-01-16T15:01:12.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.195+0000 7f839be00640 1 -- 172.21.15.53:0/1183584293 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83940309a0 con 0x7f839c15e1e0 2024-01-16T15:01:12.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.195+0000 7f839be00640 1 -- 172.21.15.53:0/1183584293 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8394030d30 con 0x7f839c15e1e0 2024-01-16T15:01:12.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 -- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c15e1e0 msgr2=0x7f839c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c15e1e0 0x7f839c0702c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8394009870 tx=0x7f839402b980 comp rx=0 tx=0).stop 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 -- 172.21.15.53:0/1183584293 shutdown_connections 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c15e1e0 0x7f839c0702c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 --2- 172.21.15.53:0/1183584293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c0cf9a0 0x7f839c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 -- 172.21.15.53:0/1183584293 >> 172.21.15.53:0/1183584293 conn(0x7f839c06a800 msgr2=0x7f839c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 -- 172.21.15.53:0/1183584293 shutdown_connections 2024-01-16T15:01:12.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.196+0000 7f83a4000640 1 -- 172.21.15.53:0/1183584293 wait complete. 2024-01-16T15:01:12.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.198+0000 7f83a4000640 1 Processor -- start 2024-01-16T15:01:12.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.198+0000 7f83a4000640 1 -- start start 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a4000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c0cf9a0 0x7f839c159c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 0x7f839c1347f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a4000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c15e1e0 0x7f839c15b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c135240 con 0x7f839c143ae0 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a4000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c135520 con 0x7f839c0cf9a0 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a4000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c135800 con 0x7f839c15e1e0 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 0x7f839c1347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 0x7f839c1347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38560/0 (socket says 172.21.15.53:38560) 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.199+0000 7f83a0c00640 1 -- 172.21.15.53:0/3748619686 learned_addr learned my addr 172.21.15.53:0/3748619686 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a1600640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c0cf9a0 0x7f839c159c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a0c00640 1 -- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c15e1e0 msgr2=0x7f839c15b8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a0c00640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c15e1e0 0x7f839c15b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a0c00640 1 -- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c0cf9a0 msgr2=0x7f839c159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a0c00640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c0cf9a0 0x7f839c159c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a0c00640 1 -- 172.21.15.53:0/3748619686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8384009a70 con 0x7f839c143ae0 2024-01-16T15:01:12.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a1600640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c0cf9a0 0x7f839c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:12.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a0c00640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 0x7f839c1347f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f838400efc0 tx=0x7f838400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:12.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384014680 con 0x7f839c143ae0 2024-01-16T15:01:12.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838400f040 con 0x7f839c143ae0 2024-01-16T15:01:12.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8384014a00 con 0x7f839c143ae0 2024-01-16T15:01:12.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.200+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83940094d0 con 0x7f839c143ae0 2024-01-16T15:01:12.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.201+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c135dc0 con 0x7f839c143ae0 2024-01-16T15:01:12.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.202+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c135f80 con 0x7f839c143ae0 2024-01-16T15:01:12.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.203+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f8384014b60 con 0x7f839c143ae0 2024-01-16T15:01:12.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.204+0000 7f8399600640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f837406b2c0 0x7f837406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.204+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f838408a910 con 0x7f839c143ae0 2024-01-16T15:01:12.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.205+0000 7f83a1600640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f837406b2c0 0x7f837406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.206+0000 7f83a1600640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f837406b2c0 0x7f837406d780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f838c001680 tx=0x7f838c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:12.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.209+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8384055ab0 con 0x7f839c143ae0 2024-01-16T15:01:12.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.475+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f839c0675d0 con 0x7f839c143ae0 2024-01-16T15:01:12.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.480+0000 7f8399600640 1 -- 172.21.15.53:0/3748619686 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f83840590d0 con 0x7f839c143ae0 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f837406b2c0 msgr2=0x7f837406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f837406b2c0 0x7f837406d780 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f838c001680 tx=0x7f838c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 msgr2=0x7f839c1347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 0x7f839c1347f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f838400efc0 tx=0x7f838400c490 comp rx=0 tx=0).stop 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 shutdown_connections 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f837406b2c0 0x7f837406d780 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f839c15e1e0 0x7f839c15b8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f839c143ae0 0x7f839c1347f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 --2- 172.21.15.53:0/3748619686 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f839c0cf9a0 0x7f839c159c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 >> 172.21.15.53:0/3748619686 conn(0x7f839c06a800 msgr2=0x7f839c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 shutdown_connections 2024-01-16T15:01:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.483+0000 7f83a4000640 1 -- 172.21.15.53:0/3748619686 wait complete. 2024-01-16T15:01:12.484 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-01-16T15:01:12.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.722+0000 7fc277e00640 1 Processor -- start 2024-01-16T15:01:12.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.722+0000 7fc277e00640 1 -- start start 2024-01-16T15:01:12.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.723+0000 7fc277e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc278160920 0x7fc2780b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.723+0000 7fc277e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc2780cf9f0 0x7fc2780cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.723+0000 7fc277e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2780d0a10 con 0x7fc278161010 2024-01-16T15:01:12.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.724+0000 7fc277e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2780d03b0 con 0x7fc2780cf9f0 2024-01-16T15:01:12.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.724+0000 7fc277e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2780d1f50 con 0x7fc278160920 2024-01-16T15:01:12.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.724+0000 7fc277400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc278161010 0x7fc2781614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33308/0 (socket says 172.21.15.53:33308) 2024-01-16T15:01:12.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.724+0000 7fc277400640 1 -- 172.21.15.53:0/1283803673 learned_addr learned my addr 172.21.15.53:0/1283803673 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.724+0000 7fc276a00640 1 --2- 172.21.15.53:0/1283803673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc278160920 0x7fc2780b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.724+0000 7fc276000640 1 --2- 172.21.15.53:0/1283803673 >> v2:172.21.15.146:3300/0 conn(0x7fc2780cf9f0 0x7fc2780cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc275600640 1 -- 172.21.15.53:0/1283803673 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388896131 0 0) 0x7fc2780d0a10 con 0x7fc278161010 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc275600640 1 -- 172.21.15.53:0/1283803673 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc258003520 con 0x7fc278161010 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc276a00640 1 -- 172.21.15.53:0/1283803673 >> v2:172.21.15.146:3300/0 conn(0x7fc2780cf9f0 msgr2=0x7fc2780cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc276a00640 1 --2- 172.21.15.53:0/1283803673 >> v2:172.21.15.146:3300/0 conn(0x7fc2780cf9f0 0x7fc2780cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc275600640 1 -- 172.21.15.53:0/1283803673 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1224445553 0 0) 0x7fc258003520 con 0x7fc278161010 2024-01-16T15:01:12.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc276a00640 1 -- 172.21.15.53:0/1283803673 >> v1:172.21.15.53:6789/0 conn(0x7fc278161010 legacy=0x7fc2781614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.725+0000 7fc276a00640 1 -- 172.21.15.53:0/1283803673 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2780cf080 con 0x7fc278160920 2024-01-16T15:01:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.727+0000 7fc276a00640 1 --2- 172.21.15.53:0/1283803673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc278160920 0x7fc2780b3f30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc2680097a0 tx=0x7fc26802b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=847cde0a31bae8a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.727+0000 7fc275600640 1 -- 172.21.15.53:0/1283803673 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2700019f0 con 0x7fc278160920 2024-01-16T15:01:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.727+0000 7fc275600640 1 -- 172.21.15.53:0/1283803673 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26802e050 con 0x7fc278160920 2024-01-16T15:01:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.727+0000 7fc275600640 1 -- 172.21.15.53:0/1283803673 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc268030730 con 0x7fc278160920 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 -- 172.21.15.53:0/1283803673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc278160920 msgr2=0x7fc2780b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 --2- 172.21.15.53:0/1283803673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc278160920 0x7fc2780b3f30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc2680097a0 tx=0x7fc26802b790 comp rx=0 tx=0).stop 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 -- 172.21.15.53:0/1283803673 shutdown_connections 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 --2- 172.21.15.53:0/1283803673 >> v2:172.21.15.146:3300/0 conn(0x7fc2780cf9f0 0x7fc2780cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 --2- 172.21.15.53:0/1283803673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc278160920 0x7fc2780b3f30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 -- 172.21.15.53:0/1283803673 >> 172.21.15.53:0/1283803673 conn(0x7fc2780d3090 msgr2=0x7fc2780d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 -- 172.21.15.53:0/1283803673 shutdown_connections 2024-01-16T15:01:12.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.728+0000 7fc277e00640 1 -- 172.21.15.53:0/1283803673 wait complete. 2024-01-16T15:01:12.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.730+0000 7fc277e00640 1 Processor -- start 2024-01-16T15:01:12.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.730+0000 7fc277e00640 1 -- start start 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2780cf9f0 0x7fc2781f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc278160920 0x7fc2781f2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 0x7fc27815ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781f2da0 con 0x7fc278161010 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781f3e00 con 0x7fc278160920 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2781f3040 con 0x7fc2780cf9f0 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 0x7fc27815ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 0x7fc27815ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38566/0 (socket says 172.21.15.53:38566) 2024-01-16T15:01:12.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.731+0000 7fc277400640 1 -- 172.21.15.53:0/3111490622 learned_addr learned my addr 172.21.15.53:0/3111490622 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:12.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc276a00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2780cf9f0 0x7fc2781f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc276000640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc278160920 0x7fc2781f2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277400640 1 -- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2780cf9f0 msgr2=0x7fc2781f21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277400640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2780cf9f0 0x7fc2781f21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277400640 1 -- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc278160920 msgr2=0x7fc2781f2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277400640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc278160920 0x7fc2781f2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277400640 1 -- 172.21.15.53:0/3111490622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc268009400 con 0x7fc278161010 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277400640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 0x7fc27815ad30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc27000cc70 tx=0x7fc27000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27000ce80 con 0x7fc278161010 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc270012400 con 0x7fc278161010 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2700127d0 con 0x7fc278161010 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2781f3320 con 0x7fc278161010 2024-01-16T15:01:12.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.732+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2781f38e0 con 0x7fc278161010 2024-01-16T15:01:12.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.734+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2781f3ad0 con 0x7fc278161010 2024-01-16T15:01:12.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.735+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fc270007070 con 0x7fc278161010 2024-01-16T15:01:12.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.735+0000 7fc26ea00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc25006b2c0 0x7fc25006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:12.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.736+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fc270089980 con 0x7fc278161010 2024-01-16T15:01:12.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.736+0000 7fc276a00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc25006b2c0 0x7fc25006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:12.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.737+0000 7fc276a00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc25006b2c0 0x7fc25006d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc268034000 tx=0x7fc26802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:12.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:12.740+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc270054bd0 con 0x7fc278161010 2024-01-16T15:01:12.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.006+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7fc278067450 con 0x7fc278161010 2024-01-16T15:01:12.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.011+0000 7fc26ea00640 1 -- 172.21.15.53:0/3111490622 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7fc270017020 con 0x7fc278161010 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc25006b2c0 msgr2=0x7fc25006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc25006b2c0 0x7fc25006d780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc268034000 tx=0x7fc26802f000 comp rx=0 tx=0).stop 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 msgr2=0x7fc27815ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 0x7fc27815ad30 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc27000cc70 tx=0x7fc27000a430 comp rx=0 tx=0).stop 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 shutdown_connections 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc25006b2c0 0x7fc25006d780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc278161010 0x7fc27815ad30 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc278160920 0x7fc2781f2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.013+0000 7fc277e00640 1 --2- 172.21.15.53:0/3111490622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2780cf9f0 0x7fc2781f21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.014+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 >> 172.21.15.53:0/3111490622 conn(0x7fc2780d3090 msgr2=0x7fc27806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.014+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 shutdown_connections 2024-01-16T15:01:12.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.014+0000 7fc277e00640 1 -- 172.21.15.53:0/3111490622 wait complete. 2024-01-16T15:01:13.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-01-16T15:01:13.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.250+0000 7f1d03800640 1 Processor -- start 2024-01-16T15:01:13.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.250+0000 7f1d03800640 1 -- start start 2024-01-16T15:01:13.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1d03800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1d03800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1cfc0b3ae0 0x7f1cfc06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1d03800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cfc070c40 con 0x7f1cfc06d120 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1d03800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cfc071450 con 0x7f1cfc0b3ae0 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1d03800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc06c470 con 0x7f1cfc0ae5a0 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1cfbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1cfc0b3ae0 0x7f1cfc06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1cfbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1cfc0b3ae0 0x7f1cfc06cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38580/0 (socket says 172.21.15.53:38580) 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.252+0000 7f1cfbe00640 1 -- 172.21.15.53:0/2333297927 learned_addr learned my addr 172.21.15.53:0/2333297927 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.253+0000 7f1d00e00640 1 --2- 172.21.15.53:0/2333297927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.253+0000 7f1cfbe00640 1 -- 172.21.15.53:0/2333297927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 msgr2=0x7f1cfc15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.253+0000 7f1cfbe00640 1 --2- 172.21.15.53:0/2333297927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.253+0000 7f1cfbe00640 1 -- 172.21.15.53:0/2333297927 >> v1:172.21.15.53:6789/0 conn(0x7f1cfc06d120 legacy=0x7f1cfc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.253+0000 7f1cfbe00640 1 -- 172.21.15.53:0/2333297927 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cfc06c750 con 0x7f1cfc0b3ae0 2024-01-16T15:01:13.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.254+0000 7f1cfbe00640 1 --2- 172.21.15.53:0/2333297927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc06cbe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009870 tx=0x7f1ce402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ce76129e1cf55 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.254+0000 7f1cfb400640 1 -- 172.21.15.53:0/2333297927 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4030840 con 0x7f1cfc0b3ae0 2024-01-16T15:01:13.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.254+0000 7f1cfb400640 1 -- 172.21.15.53:0/2333297927 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ce40309a0 con 0x7f1cfc0b3ae0 2024-01-16T15:01:13.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.254+0000 7f1cfb400640 1 -- 172.21.15.53:0/2333297927 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ce4030d30 con 0x7f1cfc0b3ae0 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 -- 172.21.15.53:0/2333297927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0b3ae0 msgr2=0x7f1cfc06cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 --2- 172.21.15.53:0/2333297927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc06cbe0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4009870 tx=0x7f1ce402b980 comp rx=0 tx=0).stop 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 -- 172.21.15.53:0/2333297927 shutdown_connections 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 --2- 172.21.15.53:0/2333297927 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc06cbe0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 --2- 172.21.15.53:0/2333297927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 -- 172.21.15.53:0/2333297927 >> 172.21.15.53:0/2333297927 conn(0x7f1cfc06a800 msgr2=0x7f1cfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 -- 172.21.15.53:0/2333297927 shutdown_connections 2024-01-16T15:01:13.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.255+0000 7f1d03800640 1 -- 172.21.15.53:0/2333297927 wait complete. 2024-01-16T15:01:13.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.257+0000 7f1d03800640 1 Processor -- start 2024-01-16T15:01:13.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.257+0000 7f1d03800640 1 -- start start 2024-01-16T15:01:13.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d03800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 0x7f1cfc15c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d03800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc0cbc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d03800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d03800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc15ce10 con 0x7f1cfc06d120 2024-01-16T15:01:13.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d03800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc15b830 con 0x7f1cfc0b3ae0 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d03800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cfc15bb10 con 0x7f1cfc0ae5a0 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d00e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 0x7f1cfc15c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d00e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 0x7f1cfc15c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38590/0 (socket says 172.21.15.53:38590) 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.258+0000 7f1d00e00640 1 -- 172.21.15.53:0/4075839408 learned_addr learned my addr 172.21.15.53:0/4075839408 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1cfbe00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc0cbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d00e00640 1 -- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 msgr2=0x7f1cfc0cbc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d00e00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc0cbc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d00e00640 1 -- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cfc0b3ae0 msgr2=0x7f1cfc144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d01800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d00e00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc144cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d00e00640 1 -- 172.21.15.53:0/4075839408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40094d0 con 0x7f1cfc06d120 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1cfbe00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d00e00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 0x7f1cfc15c8d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00d800 tx=0x7f1cec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00b6c0 con 0x7f1cfc06d120 2024-01-16T15:01:13.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec00b820 con 0x7f1cfc06d120 2024-01-16T15:01:13.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00bbb0 con 0x7f1cfc06d120 2024-01-16T15:01:13.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.259+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cfc15bdf0 con 0x7f1cfc06d120 2024-01-16T15:01:13.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.260+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cfc15c3b0 con 0x7f1cfc06d120 2024-01-16T15:01:13.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.261+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cfc15c540 con 0x7f1cfc06d120 2024-01-16T15:01:13.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.262+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1cec00bd10 con 0x7f1cfc06d120 2024-01-16T15:01:13.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.262+0000 7f1cf8c00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cd406b2c0 0x7f1cd406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.263+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1cec089d90 con 0x7f1cfc06d120 2024-01-16T15:01:13.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.263+0000 7f1cfbe00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cd406b2c0 0x7f1cd406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.264+0000 7f1cfbe00640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cd406b2c0 0x7f1cd406d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4036000 tx=0x7f1ce403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.267+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cec054fe0 con 0x7f1cfc06d120 2024-01-16T15:01:13.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.538+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f1cfc0675d0 con 0x7f1cfc06d120 2024-01-16T15:01:13.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.542+0000 7f1cf8c00640 1 -- 172.21.15.53:0/4075839408 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f1cec058600 con 0x7f1cfc06d120 2024-01-16T15:01:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cd406b2c0 msgr2=0x7f1cd406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cd406b2c0 0x7f1cd406d780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1ce4036000 tx=0x7f1ce403b040 comp rx=0 tx=0).stop 2024-01-16T15:01:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 msgr2=0x7f1cfc15c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 0x7f1cfc15c8d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f1cec00d800 tx=0x7f1cec00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 shutdown_connections 2024-01-16T15:01:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cd406b2c0 0x7f1cd406d780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cfc0b3ae0 0x7f1cfc144cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cfc0ae5a0 0x7f1cfc0cbc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 --2- 172.21.15.53:0/4075839408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cfc06d120 0x7f1cfc15c8d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 >> 172.21.15.53:0/4075839408 conn(0x7f1cfc06a800 msgr2=0x7f1cfc0cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:13.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 shutdown_connections 2024-01-16T15:01:13.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.546+0000 7f1d03800640 1 -- 172.21.15.53:0/4075839408 wait complete. 2024-01-16T15:01:13.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-01-16T15:01:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.785+0000 7f4832200640 1 Processor -- start 2024-01-16T15:01:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.785+0000 7f4832200640 1 -- start start 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f4832200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f482c1447f0 0x7f482c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f4832200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f482c06c470 0x7f482c0cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f4832200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f482c15c5c0 con 0x7f482c06dab0 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f4832200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c06d690 con 0x7f482c06c470 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f4832200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f482c0cebb0 con 0x7f482c1447f0 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f482be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f482c06c470 0x7f482c0cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f482be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f482c06c470 0x7f482c0cc5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38594/0 (socket says 172.21.15.53:38594) 2024-01-16T15:01:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.787+0000 7f482be00640 1 -- 172.21.15.53:0/395326891 learned_addr learned my addr 172.21.15.53:0/395326891 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:13.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.788+0000 7f4830e00640 1 --2- 172.21.15.53:0/395326891 >> v2:172.21.15.146:3300/0 conn(0x7f482c1447f0 0x7f482c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.788+0000 7f482be00640 1 -- 172.21.15.53:0/395326891 >> v2:172.21.15.146:3300/0 conn(0x7f482c1447f0 msgr2=0x7f482c15c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:13.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.788+0000 7f482be00640 1 --2- 172.21.15.53:0/395326891 >> v2:172.21.15.146:3300/0 conn(0x7f482c1447f0 0x7f482c15c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.788+0000 7f482be00640 1 -- 172.21.15.53:0/395326891 >> v1:172.21.15.146:6789/0 conn(0x7f482c06dab0 legacy=0x7f482c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:13.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.788+0000 7f482be00640 1 -- 172.21.15.53:0/395326891 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f482c06ca10 con 0x7f482c06c470 2024-01-16T15:01:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.789+0000 7f482be00640 1 --2- 172.21.15.53:0/395326891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06c470 0x7f482c0cc5e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f482c0cecf0 tx=0x7f481402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db2991b69f983532 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.789+0000 7f482b400640 1 -- 172.21.15.53:0/395326891 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48140317e0 con 0x7f482c06c470 2024-01-16T15:01:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.789+0000 7f482b400640 1 -- 172.21.15.53:0/395326891 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4814031940 con 0x7f482c06c470 2024-01-16T15:01:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.789+0000 7f482b400640 1 -- 172.21.15.53:0/395326891 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814031cd0 con 0x7f482c06c470 2024-01-16T15:01:13.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 -- 172.21.15.53:0/395326891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06c470 msgr2=0x7f482c0cc5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 --2- 172.21.15.53:0/395326891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06c470 0x7f482c0cc5e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f482c0cecf0 tx=0x7f481402b880 comp rx=0 tx=0).stop 2024-01-16T15:01:13.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 -- 172.21.15.53:0/395326891 shutdown_connections 2024-01-16T15:01:13.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 --2- 172.21.15.53:0/395326891 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06c470 0x7f482c0cc5e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 --2- 172.21.15.53:0/395326891 >> v2:172.21.15.146:3300/0 conn(0x7f482c1447f0 0x7f482c15c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 -- 172.21.15.53:0/395326891 >> 172.21.15.53:0/395326891 conn(0x7f482c06a3e0 msgr2=0x7f482c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:13.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 -- 172.21.15.53:0/395326891 shutdown_connections 2024-01-16T15:01:13.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.790+0000 7f4832200640 1 -- 172.21.15.53:0/395326891 wait complete. 2024-01-16T15:01:13.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.792+0000 7f4832200640 1 Processor -- start 2024-01-16T15:01:13.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.792+0000 7f4832200640 1 -- start start 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f4832200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f482c06c470 0x7f482c13c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f4832200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 0x7f482c1f3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f4832200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f482c1447f0 0x7f482c15b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f4832200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c0afb00 con 0x7f482c06dab0 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f4832200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f482c13c5d0 con 0x7f482c1447f0 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f4832200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f482c1f3d90 con 0x7f482c06c470 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f482be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 0x7f482c1f3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f482be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 0x7f482c1f3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38606/0 (socket says 172.21.15.53:38606) 2024-01-16T15:01:13.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.793+0000 7f482be00640 1 -- 172.21.15.53:0/1244681126 learned_addr learned my addr 172.21.15.53:0/1244681126 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f4830e00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f482c06c470 0x7f482c13c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f482be00640 1 -- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f482c06c470 msgr2=0x7f482c13c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f482be00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f482c06c470 0x7f482c13c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f4831800640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f482c1447f0 0x7f482c15b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f482be00640 1 -- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f482c1447f0 msgr2=0x7f482c15b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f482be00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f482c1447f0 0x7f482c15b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f482be00640 1 -- 172.21.15.53:0/1244681126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48140094d0 con 0x7f482c06dab0 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f4831800640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f482c1447f0 0x7f482c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f482be00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 0x7f482c1f3850 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4814000c00 tx=0x7f48140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.794+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814032880 con 0x7f482c06dab0 2024-01-16T15:01:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.795+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48140329e0 con 0x7f482c06dab0 2024-01-16T15:01:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.795+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4814032d90 con 0x7f482c06dab0 2024-01-16T15:01:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.795+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f482c0ace70 con 0x7f482c06dab0 2024-01-16T15:01:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.795+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f482c0ad430 con 0x7f482c06dab0 2024-01-16T15:01:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.796+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f482c0afc70 con 0x7f482c06dab0 2024-01-16T15:01:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.797+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f48140453f0 con 0x7f482c06dab0 2024-01-16T15:01:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.798+0000 7f4828c00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f480406b2c0 0x7f480406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:13.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.798+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f48140ab9e0 con 0x7f482c06dab0 2024-01-16T15:01:13.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.798+0000 7f4830e00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f480406b2c0 0x7f480406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:13.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.800+0000 7f4830e00640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f480406b2c0 0x7f480406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f481c006fd0 tx=0x7f481c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:13.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:13.802+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f482c0afc70 con 0x7f482c06dab0 2024-01-16T15:01:14.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.065+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f482c0afc70 con 0x7f482c06dab0 2024-01-16T15:01:14.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.069+0000 7f4828c00640 1 -- 172.21.15.53:0/1244681126 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f482c0afc70 con 0x7f482c06dab0 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.071+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f480406b2c0 msgr2=0x7f480406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.071+0000 7f4832200640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f480406b2c0 0x7f480406d780 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f481c006fd0 tx=0x7f481c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.071+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 msgr2=0x7f482c1f3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.071+0000 7f4832200640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 0x7f482c1f3850 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f4814000c00 tx=0x7f48140046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 shutdown_connections 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f480406b2c0 0x7f480406d780 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f482c1447f0 0x7f482c15b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f482c06dab0 0x7f482c1f3850 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 --2- 172.21.15.53:0/1244681126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f482c06c470 0x7f482c13c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 >> 172.21.15.53:0/1244681126 conn(0x7f482c06a3e0 msgr2=0x7f482c159a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 shutdown_connections 2024-01-16T15:01:14.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.072+0000 7f4832200640 1 -- 172.21.15.53:0/1244681126 wait complete. 2024-01-16T15:01:14.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-01-16T15:01:14.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.310+0000 7f0268000640 1 Processor -- start 2024-01-16T15:01:14.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.311+0000 7f0268000640 1 -- start start 2024-01-16T15:01:14.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.312+0000 7f0268000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02600d3910 0x7f02600d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.312+0000 7f0268000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0260160e10 0x7f02600d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.312+0000 7f0268000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02600d4160 con 0x7f02600cfcf0 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.312+0000 7f0268000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02600ceff0 con 0x7f0260160e10 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.312+0000 7f0268000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02600cf410 con 0x7f02600d3910 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0260160e10 0x7f02600d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0260160e10 0x7f02600d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38616/0 (socket says 172.21.15.53:38616) 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 -- 172.21.15.53:0/1798887144 learned_addr learned my addr 172.21.15.53:0/1798887144 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0264c00640 1 --2- 172.21.15.53:0/1798887144 >> v2:172.21.15.146:3300/0 conn(0x7f02600d3910 0x7f02600d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 -- 172.21.15.53:0/1798887144 >> v2:172.21.15.146:3300/0 conn(0x7f02600d3910 msgr2=0x7f02600d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:14.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 --2- 172.21.15.53:0/1798887144 >> v2:172.21.15.146:3300/0 conn(0x7f02600d3910 0x7f02600d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 -- 172.21.15.53:0/1798887144 >> v1:172.21.15.146:6789/0 conn(0x7f02600cfcf0 legacy=0x7f0260160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:14.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.313+0000 7f0266000640 1 -- 172.21.15.53:0/1798887144 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02600b2420 con 0x7f0260160e10 2024-01-16T15:01:14.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.314+0000 7f0266000640 1 --2- 172.21.15.53:0/1798887144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f02600d1640 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f026008b640 tx=0x7f025802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2870494b26214bd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:14.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.315+0000 7f025fe00640 1 -- 172.21.15.53:0/1798887144 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025802c9f0 con 0x7f0260160e10 2024-01-16T15:01:14.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.315+0000 7f025fe00640 1 -- 172.21.15.53:0/1798887144 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025802cb50 con 0x7f0260160e10 2024-01-16T15:01:14.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.315+0000 7f025fe00640 1 -- 172.21.15.53:0/1798887144 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02580343f0 con 0x7f0260160e10 2024-01-16T15:01:14.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 -- 172.21.15.53:0/1798887144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 msgr2=0x7f02600d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 --2- 172.21.15.53:0/1798887144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f02600d1640 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f026008b640 tx=0x7f025802be50 comp rx=0 tx=0).stop 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 -- 172.21.15.53:0/1798887144 shutdown_connections 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 --2- 172.21.15.53:0/1798887144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f02600d1640 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 --2- 172.21.15.53:0/1798887144 >> v2:172.21.15.146:3300/0 conn(0x7f02600d3910 0x7f02600d1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 -- 172.21.15.53:0/1798887144 >> 172.21.15.53:0/1798887144 conn(0x7f026009c530 msgr2=0x7f02600d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 -- 172.21.15.53:0/1798887144 shutdown_connections 2024-01-16T15:01:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.316+0000 7f0268000640 1 -- 172.21.15.53:0/1798887144 wait complete. 2024-01-16T15:01:14.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.318+0000 7f0268000640 1 Processor -- start 2024-01-16T15:01:14.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.318+0000 7f0268000640 1 -- start start 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0268000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02600cfcf0 0x7f0260154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0268000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02600d3910 0x7f02600b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0268000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f026006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0268000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02600c6ca0 con 0x7f0260160e10 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0268000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02600c7260 con 0x7f02600cfcf0 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0268000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02600b6b10 con 0x7f02600d3910 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0266000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f026006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0266000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f026006bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38624/0 (socket says 172.21.15.53:38624) 2024-01-16T15:01:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.319+0000 7f0266000640 1 -- 172.21.15.53:0/3223354072 learned_addr learned my addr 172.21.15.53:0/3223354072 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0265600640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02600cfcf0 0x7f0260154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0264c00640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02600d3910 0x7f02600b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0266000640 1 -- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02600d3910 msgr2=0x7f02600b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0266000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02600d3910 0x7f02600b6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0266000640 1 -- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02600cfcf0 msgr2=0x7f0260154960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0266000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02600cfcf0 0x7f0260154960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0266000640 1 -- 172.21.15.53:0/3223354072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0258009b30 con 0x7f0260160e10 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0265600640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02600cfcf0 0x7f0260154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0266000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f026006bb60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0258009e50 tx=0x7f0258004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0258009510 con 0x7f0260160e10 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025803e450 con 0x7f0260160e10 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f025803e7e0 con 0x7f0260160e10 2024-01-16T15:01:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.320+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02600b6df0 con 0x7f0260160e10 2024-01-16T15:01:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.321+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0260154ea0 con 0x7f0260160e10 2024-01-16T15:01:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.322+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02600675d0 con 0x7f0260160e10 2024-01-16T15:01:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.323+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f025803e940 con 0x7f0260160e10 2024-01-16T15:01:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.324+0000 7f025d600640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f023806b2c0 0x7f023806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.324+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f02580ac660 con 0x7f0260160e10 2024-01-16T15:01:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.324+0000 7f0265600640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f023806b2c0 0x7f023806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.325+0000 7f0265600640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f023806b2c0 0x7f023806d780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f025000a830 tx=0x7f0250008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:14.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.328+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0258077810 con 0x7f0260160e10 2024-01-16T15:01:14.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.593+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f02600c73a0 con 0x7f0260160e10 2024-01-16T15:01:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.598+0000 7f025d600640 1 -- 172.21.15.53:0/3223354072 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f025807ae30 con 0x7f0260160e10 2024-01-16T15:01:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f023806b2c0 msgr2=0x7f023806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f023806b2c0 0x7f023806d780 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f025000a830 tx=0x7f0250008040 comp rx=0 tx=0).stop 2024-01-16T15:01:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 msgr2=0x7f026006bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f026006bb60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f0258009e50 tx=0x7f0258004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 shutdown_connections 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f023806b2c0 0x7f023806d780 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0260160e10 0x7f026006bb60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02600d3910 0x7f02600b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 --2- 172.21.15.53:0/3223354072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02600cfcf0 0x7f0260154960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 >> 172.21.15.53:0/3223354072 conn(0x7f026009c530 msgr2=0x7f02600d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 shutdown_connections 2024-01-16T15:01:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.601+0000 7f0268000640 1 -- 172.21.15.53:0/3223354072 wait complete. 2024-01-16T15:01:14.602 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-01-16T15:01:14.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.838+0000 7f0a01400640 1 Processor -- start 2024-01-16T15:01:14.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.838+0000 7f0a01400640 1 -- start start 2024-01-16T15:01:14.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a01400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f09fc1447f0 0x7f09fc15c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a01400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06c470 0x7f09fc0cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a01400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09fc06dab0 0x7f09fc0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a01400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc06d690 con 0x7f09fc06dab0 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a01400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09fc0cebb0 con 0x7f09fc1447f0 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a01400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc15c5c0 con 0x7f09fc06c470 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a00a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09fc06dab0 0x7f09fc0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a00a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f09fc06dab0 0x7f09fc0ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38626/0 (socket says 172.21.15.53:38626) 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f0a00a00640 1 -- 172.21.15.53:0/2561687040 learned_addr learned my addr 172.21.15.53:0/2561687040 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.840+0000 7f09fbe00640 1 --2- 172.21.15.53:0/2561687040 >> v2:172.21.15.146:3300/0 conn(0x7f09fc1447f0 0x7f09fc15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.841+0000 7f09fb400640 1 --2- 172.21.15.53:0/2561687040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06c470 0x7f09fc0cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.841+0000 7f0a00a00640 1 -- 172.21.15.53:0/2561687040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06c470 msgr2=0x7f09fc0cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.841+0000 7f0a00a00640 1 --2- 172.21.15.53:0/2561687040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06c470 0x7f09fc0cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.841+0000 7f0a00a00640 1 -- 172.21.15.53:0/2561687040 >> v2:172.21.15.146:3300/0 conn(0x7f09fc1447f0 msgr2=0x7f09fc15c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.841+0000 7f0a00a00640 1 --2- 172.21.15.53:0/2561687040 >> v2:172.21.15.146:3300/0 conn(0x7f09fc1447f0 0x7f09fc15c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.841+0000 7f0a00a00640 1 -- 172.21.15.53:0/2561687040 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc06ca10 con 0x7f09fc06dab0 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.842+0000 7f0a00a00640 1 --2- 172.21.15.53:0/2561687040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc06dab0 0x7f09fc0ccb20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f09fc072520 tx=0x7f09f402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=565c0ddd11b57810 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.842+0000 7f09faa00640 1 -- 172.21.15.53:0/2561687040 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f402c9e0 con 0x7f09fc06dab0 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.842+0000 7f09faa00640 1 -- 172.21.15.53:0/2561687040 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f402cb40 con 0x7f09fc06dab0 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.842+0000 7f09faa00640 1 -- 172.21.15.53:0/2561687040 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f40343f0 con 0x7f09fc06dab0 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 -- 172.21.15.53:0/2561687040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc06dab0 msgr2=0x7f09fc0ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 --2- 172.21.15.53:0/2561687040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc06dab0 0x7f09fc0ccb20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f09fc072520 tx=0x7f09f402be40 comp rx=0 tx=0).stop 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 -- 172.21.15.53:0/2561687040 shutdown_connections 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 --2- 172.21.15.53:0/2561687040 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc06dab0 0x7f09fc0ccb20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 --2- 172.21.15.53:0/2561687040 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06c470 0x7f09fc0cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 --2- 172.21.15.53:0/2561687040 >> v2:172.21.15.146:3300/0 conn(0x7f09fc1447f0 0x7f09fc15c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 -- 172.21.15.53:0/2561687040 >> 172.21.15.53:0/2561687040 conn(0x7f09fc06a3e0 msgr2=0x7f09fc06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 -- 172.21.15.53:0/2561687040 shutdown_connections 2024-01-16T15:01:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.843+0000 7f0a01400640 1 -- 172.21.15.53:0/2561687040 wait complete. 2024-01-16T15:01:14.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.845+0000 7f0a01400640 1 Processor -- start 2024-01-16T15:01:14.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.845+0000 7f0a01400640 1 -- start start 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.846+0000 7f0a01400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09fc06c470 0x7f09fc071510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.846+0000 7f0a01400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06dab0 0x7f09fc13be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.846+0000 7f0a01400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 0x7f09fc150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.846+0000 7f0a01400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc13cf90 con 0x7f09fc1447f0 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.846+0000 7f0a01400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc070b00 con 0x7f09fc06c470 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.846+0000 7f0a01400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09fc071a50 con 0x7f09fc06dab0 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 0x7f09fc150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 0x7f09fc150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38640/0 (socket says 172.21.15.53:38640) 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 -- 172.21.15.53:0/1430197805 learned_addr learned my addr 172.21.15.53:0/1430197805 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f09fbe00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09fc06c470 0x7f09fc071510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f09fb400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06dab0 0x7f09fc13be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 -- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06dab0 msgr2=0x7f09fc13be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06dab0 0x7f09fc13be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 -- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09fc06c470 msgr2=0x7f09fc071510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09fc06c470 0x7f09fc071510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 -- 172.21.15.53:0/1430197805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f4009b30 con 0x7f09fc1447f0 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f09fb400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06dab0 0x7f09fc13be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f09fbe00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09fc06c470 0x7f09fc071510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:01:14.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.847+0000 7f0a00a00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 0x7f09fc150650 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036040 tx=0x7f09f4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.848+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f403e450 con 0x7f09fc1447f0 2024-01-16T15:01:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.848+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09f403e5b0 con 0x7f09fc1447f0 2024-01-16T15:01:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.848+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09f403e960 con 0x7f09fc1447f0 2024-01-16T15:01:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.848+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09fc13c9d0 con 0x7f09fc1447f0 2024-01-16T15:01:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.848+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09fc070d10 con 0x7f09fc1447f0 2024-01-16T15:01:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.849+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09fc067450 con 0x7f09fc1447f0 2024-01-16T15:01:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.850+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f09f403eac0 con 0x7f09fc1447f0 2024-01-16T15:01:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.851+0000 7f09f3e00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09d406b2c0 0x7f09d406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.851+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f09f40ac5e0 con 0x7f09fc1447f0 2024-01-16T15:01:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.852+0000 7f09fbe00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09d406b2c0 0x7f09d406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:14.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.853+0000 7f09fbe00640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09d406b2c0 0x7f09d406d780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006fd0 tx=0x7f09ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:14.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:14.856+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09f40778c0 con 0x7f09fc1447f0 2024-01-16T15:01:15.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.122+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f09fc15df00 con 0x7f09fc1447f0 2024-01-16T15:01:15.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.126+0000 7f09f3e00640 1 -- 172.21.15.53:0/1430197805 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f09f407aee0 con 0x7f09fc1447f0 2024-01-16T15:01:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09d406b2c0 msgr2=0x7f09d406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09d406b2c0 0x7f09d406d780 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006fd0 tx=0x7f09ec008040 comp rx=0 tx=0).stop 2024-01-16T15:01:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 msgr2=0x7f09fc150650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 0x7f09fc150650 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f09f4036040 tx=0x7f09f4004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 shutdown_connections 2024-01-16T15:01:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09d406b2c0 0x7f09d406d780 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09fc1447f0 0x7f09fc150650 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09fc06dab0 0x7f09fc13be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 --2- 172.21.15.53:0/1430197805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09fc06c470 0x7f09fc071510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 >> 172.21.15.53:0/1430197805 conn(0x7f09fc06a3e0 msgr2=0x7f09fc0af500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:15.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 shutdown_connections 2024-01-16T15:01:15.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.129+0000 7f0a01400640 1 -- 172.21.15.53:0/1430197805 wait complete. 2024-01-16T15:01:15.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.358+0000 7fd31c400640 1 Processor -- start 2024-01-16T15:01:15.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.358+0000 7fd31c400640 1 -- start start 2024-01-16T15:01:15.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.360+0000 7fd31c400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3140cd600 0x7fd314070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.360+0000 7fd31c400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd314071450 con 0x7fd314157c30 2024-01-16T15:01:15.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.360+0000 7fd31c400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd31415deb0 con 0x7fd314071030 2024-01-16T15:01:15.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.360+0000 7fd31c400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd31415ea30 con 0x7fd3140cd600 2024-01-16T15:01:15.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.360+0000 7fd319a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd314157c30 0x7fd3141618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33334/0 (socket says 172.21.15.53:33334) 2024-01-16T15:01:15.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.360+0000 7fd319a00640 1 -- 172.21.15.53:0/3088172816 learned_addr learned my addr 172.21.15.53:0/3088172816 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638748302 0 0) 0x7fd314071450 con 0x7fd314157c30 2024-01-16T15:01:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd319000640 1 --2- 172.21.15.53:0/3088172816 >> v2:172.21.15.146:3300/0 conn(0x7fd3140cd600 0x7fd314070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2f4003520 con 0x7fd314157c30 2024-01-16T15:01:15.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3502873560 0 0) 0x7fd2f4003520 con 0x7fd314157c30 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 >> v2:172.21.15.146:3300/0 conn(0x7fd3140cd600 msgr2=0x7fd314070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 --2- 172.21.15.53:0/3088172816 >> v2:172.21.15.146:3300/0 conn(0x7fd3140cd600 0x7fd314070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 >> v1:172.21.15.146:6789/0 conn(0x7fd314071030 legacy=0x7fd31406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31415e190 con 0x7fd314157c30 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd304001670 con 0x7fd314157c30 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 >> v1:172.21.15.53:6789/0 conn(0x7fd314157c30 legacy=0x7fd3141618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.361+0000 7fd313e00640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd313e00640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31415ea30 con 0x7fd2f4005180 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f400c080 con 0x7fd2f40047c0 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f400c240 con 0x7fd2f40039c0 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd31a400640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd319000640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd319a00640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd31a400640 1 -- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f40039c0 msgr2=0x7fd2f4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:15.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd31a400640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f40039c0 0x7fd2f4003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd31a400640 1 -- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f40047c0 msgr2=0x7fd2f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:15.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd31a400640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2f40047c0 0x7fd2f4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.362+0000 7fd31a400640 1 -- 172.21.15.53:0/3088172816 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f4003520 con 0x7fd2f4005180 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.363+0000 7fd31a400640 1 -- 172.21.15.53:0/3088172816 reap_dead start 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.363+0000 7fd31a400640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd30c001e50 tx=0x7fd2f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e064826f4bd03964 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c003b70 con 0x7fd2f4005180 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30c003d10 con 0x7fd2f4005180 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd313e00640 1 -- 172.21.15.53:0/3088172816 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30c005310 con 0x7fd2f4005180 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd31c400640 1 -- 172.21.15.53:0/3088172816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2f4005180 msgr2=0x7fd2f400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd31c400640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd30c001e50 tx=0x7fd2f40057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd31c400640 1 -- 172.21.15.53:0/3088172816 shutdown_connections 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd31c400640 1 --2- 172.21.15.53:0/3088172816 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2f4005180 0x7fd2f400b980 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.364+0000 7fd31c400640 1 -- 172.21.15.53:0/3088172816 >> 172.21.15.53:0/3088172816 conn(0x7fd3140cf700 msgr2=0x7fd3140cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:15.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.365+0000 7fd31c400640 1 -- 172.21.15.53:0/3088172816 shutdown_connections 2024-01-16T15:01:15.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.365+0000 7fd31c400640 1 -- 172.21.15.53:0/3088172816 wait complete. 2024-01-16T15:01:15.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.366+0000 7fd31c400640 1 Processor -- start 2024-01-16T15:01:15.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.366+0000 7fd31c400640 1 -- start start 2024-01-16T15:01:15.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.367+0000 7fd31c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f4005180 0x7fd314145af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.367+0000 7fd31c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 0x7fd31415fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.367+0000 7fd31c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd314160020 0x7fd3140ad920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.367+0000 7fd31c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141604a0 con 0x7fd31415cb80 2024-01-16T15:01:15.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.367+0000 7fd31c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141f38a0 con 0x7fd314160020 2024-01-16T15:01:15.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.367+0000 7fd31c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3141f3b80 con 0x7fd2f4005180 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 0x7fd31415fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 0x7fd31415fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38660/0 (socket says 172.21.15.53:38660) 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 -- 172.21.15.53:0/3302696522 learned_addr learned my addr 172.21.15.53:0/3302696522 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319a00640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f4005180 0x7fd314145af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd31a400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd314160020 0x7fd3140ad920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 -- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f4005180 msgr2=0x7fd314145af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f4005180 0x7fd314145af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 -- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd314160020 msgr2=0x7fd3140ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd314160020 0x7fd3140ad920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.368+0000 7fd319000640 1 -- 172.21.15.53:0/3302696522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3141f3e60 con 0x7fd31415cb80 2024-01-16T15:01:15.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.369+0000 7fd319000640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 0x7fd31415fae0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc011540 tx=0x7fd2fc011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:15.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.369+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc015930 con 0x7fd31415cb80 2024-01-16T15:01:15.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.369+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2fc015a90 con 0x7fd31415cb80 2024-01-16T15:01:15.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.369+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2fc015e60 con 0x7fd31415cb80 2024-01-16T15:01:15.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.369+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3141f4140 con 0x7fd31415cb80 2024-01-16T15:01:15.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.369+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3141f3550 con 0x7fd31415cb80 2024-01-16T15:01:15.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.370+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3140675d0 con 0x7fd31415cb80 2024-01-16T15:01:15.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.371+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd2fc00f460 con 0x7fd31415cb80 2024-01-16T15:01:15.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.372+0000 7fd311600640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd2ec06b2f0 0x7fd2ec06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.372+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd2fc08a0f0 con 0x7fd31415cb80 2024-01-16T15:01:15.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.373+0000 7fd319a00640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd2ec06b2f0 0x7fd2ec06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.374+0000 7fd319a00640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd2ec06b2f0 0x7fd2ec06d7b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd304005990 tx=0x7fd304009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.377+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd2fc055210 con 0x7fd31415cb80 2024-01-16T15:01:15.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.574+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd314160dc0 con 0x7fd31415cb80 2024-01-16T15:01:15.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.576+0000 7fd311600640 1 -- 172.21.15.53:0/3302696522 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v31) v1 ==== 74+0+12821 (secure 0 0 0) 0x7fd2fc058830 con 0x7fd31415cb80 2024-01-16T15:01:15.556 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:15.556 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":31,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:01:11.428659+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:15.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd2ec06b2f0 msgr2=0x7fd2ec06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd2ec06b2f0 0x7fd2ec06d7b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd304005990 tx=0x7fd304009040 comp rx=0 tx=0).stop 2024-01-16T15:01:15.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 msgr2=0x7fd31415fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 0x7fd31415fae0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fd2fc011540 tx=0x7fd2fc011a00 comp rx=0 tx=0).stop 2024-01-16T15:01:15.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 shutdown_connections 2024-01-16T15:01:15.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd2ec06b2f0 0x7fd2ec06d7b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd314160020 0x7fd3140ad920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd31415cb80 0x7fd31415fae0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 --2- 172.21.15.53:0/3302696522 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2f4005180 0x7fd314145af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 >> 172.21.15.53:0/3302696522 conn(0x7fd3140cf700 msgr2=0x7fd31406e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:15.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 shutdown_connections 2024-01-16T15:01:15.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.579+0000 7fd31c400640 1 -- 172.21.15.53:0/3302696522 wait complete. 2024-01-16T15:01:15.579 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:01:15.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:01:15.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.808+0000 7f490fe00640 1 Processor -- start 2024-01-16T15:01:15.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.808+0000 7f490fe00640 1 -- start start 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49100d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49100cf9a0 0x7f49100cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49100d0320 con 0x7f49100cddf0 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49100b2730 con 0x7f49100cf9a0 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49100d1ec0 con 0x7f491015d4b0 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f49100cddf0 0x7f49100ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33348/0 (socket says 172.21.15.53:33348) 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490f400640 1 -- 172.21.15.53:0/3606608699 learned_addr learned my addr 172.21.15.53:0/3606608699 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:15.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490e000640 1 --2- 172.21.15.53:0/3606608699 >> v2:172.21.15.53:3300/0 conn(0x7f49100cf9a0 0x7f49100cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.810+0000 7f490ea00640 1 --2- 172.21.15.53:0/3606608699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49100d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.811+0000 7f490e000640 1 -- 172.21.15.53:0/3606608699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 msgr2=0x7f49100d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:15.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.811+0000 7f490e000640 1 --2- 172.21.15.53:0/3606608699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49100d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.811+0000 7f490e000640 1 -- 172.21.15.53:0/3606608699 >> v1:172.21.15.53:6789/0 conn(0x7f49100cddf0 legacy=0x7f49100ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.811+0000 7f490e000640 1 -- 172.21.15.53:0/3606608699 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49100b29d0 con 0x7f49100cf9a0 2024-01-16T15:01:15.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.812+0000 7f490e000640 1 --2- 172.21.15.53:0/3606608699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cf9a0 0x7f49100cfde0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f48f80097d0 tx=0x7f48f802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dadc1bac65ccb659 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:15.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.812+0000 7f490d600640 1 -- 172.21.15.53:0/3606608699 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f802d040 con 0x7f49100cf9a0 2024-01-16T15:01:15.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.812+0000 7f490d600640 1 -- 172.21.15.53:0/3606608699 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48f8031810 con 0x7f49100cf9a0 2024-01-16T15:01:15.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.812+0000 7f490d600640 1 -- 172.21.15.53:0/3606608699 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8031ba0 con 0x7f49100cf9a0 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 -- 172.21.15.53:0/3606608699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cf9a0 msgr2=0x7f49100cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 --2- 172.21.15.53:0/3606608699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cf9a0 0x7f49100cfde0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f48f80097d0 tx=0x7f48f802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 -- 172.21.15.53:0/3606608699 shutdown_connections 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 --2- 172.21.15.53:0/3606608699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cf9a0 0x7f49100cfde0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 --2- 172.21.15.53:0/3606608699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49100d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 -- 172.21.15.53:0/3606608699 >> 172.21.15.53:0/3606608699 conn(0x7f49100d3000 msgr2=0x7f49100d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 -- 172.21.15.53:0/3606608699 shutdown_connections 2024-01-16T15:01:15.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.813+0000 7f490fe00640 1 -- 172.21.15.53:0/3606608699 wait complete. 2024-01-16T15:01:15.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.815+0000 7f490fe00640 1 Processor -- start 2024-01-16T15:01:15.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.815+0000 7f490fe00640 1 -- start start 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 0x7f4910160960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49100cf9a0 0x7f49100a05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49101f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49101f4020 con 0x7f49100cddf0 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4910160030 con 0x7f49100cf9a0 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4910160ea0 con 0x7f491015d4b0 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 0x7f4910160960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 0x7f4910160960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38686/0 (socket says 172.21.15.53:38686) 2024-01-16T15:01:15.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 -- 172.21.15.53:0/3661942330 learned_addr learned my addr 172.21.15.53:0/3661942330 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:15.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 -- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 msgr2=0x7f49101f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49101f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 -- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49100cf9a0 msgr2=0x7f49100a05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:15.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49100cf9a0 0x7f49100a05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.816+0000 7f490ea00640 1 -- 172.21.15.53:0/3661942330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f8009400 con 0x7f49100cddf0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.817+0000 7f490ea00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 0x7f4910160960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4900007e40 tx=0x7f490000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.817+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900014620 con 0x7f49100cddf0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.817+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490000ee90 con 0x7f49100cddf0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.817+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49000149c0 con 0x7f49100cddf0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.817+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491015da10 con 0x7f49100cddf0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.817+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49101f3130 con 0x7f49100cddf0 2024-01-16T15:01:15.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.819+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49101f4290 con 0x7f49100cddf0 2024-01-16T15:01:15.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.819+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f4900014b20 con 0x7f49100cddf0 2024-01-16T15:01:15.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.820+0000 7f4906a00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e806b2c0 0x7f48e806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:15.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.820+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f490008a200 con 0x7f49100cddf0 2024-01-16T15:01:15.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.821+0000 7f490e000640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e806b2c0 0x7f48e806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:15.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.822+0000 7f490e000640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e806b2c0 0x7f48e806d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f48f8036000 tx=0x7f48f8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:15.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:15.824+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49000553c0 con 0x7f49100cddf0 2024-01-16T15:01:15.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.019+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4910067450 con 0x7f49100cddf0 2024-01-16T15:01:16.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.020+0000 7f4906a00640 1 -- 172.21.15.53:0/3661942330 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v31) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49000589e0 con 0x7f49100cddf0 2024-01-16T15:01:16.000 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:01:16.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e806b2c0 msgr2=0x7f48e806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e806b2c0 0x7f48e806d780 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f48f8036000 tx=0x7f48f8030000 comp rx=0 tx=0).stop 2024-01-16T15:01:16.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 msgr2=0x7f4910160960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 0x7f4910160960 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f4900007e40 tx=0x7f490000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 shutdown_connections 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e806b2c0 0x7f48e806d780 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f491015d4b0 0x7f49101f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49100cf9a0 0x7f49100a05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 --2- 172.21.15.53:0/3661942330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49100cddf0 0x7f4910160960 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 >> 172.21.15.53:0/3661942330 conn(0x7f49100d3000 msgr2=0x7f49100b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 shutdown_connections 2024-01-16T15:01:16.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.023+0000 7f490fe00640 1 -- 172.21.15.53:0/3661942330 wait complete. 2024-01-16T15:01:16.020 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:01:16.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.256+0000 7f0a5da00640 1 Processor -- start 2024-01-16T15:01:16.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.256+0000 7f0a5da00640 1 -- start start 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a5da00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a580cc500 0x7f0a580cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a5da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0a5806d160 0x7f0a5806d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a5da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a58070e50 con 0x7f0a58143c40 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a5da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a580723c0 con 0x7f0a5806d160 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a5da00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a580cced0 con 0x7f0a580cc500 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a57e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0a58143c40 0x7f0a58071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33352/0 (socket says 172.21.15.53:33352) 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a57e00640 1 -- 172.21.15.53:0/2181731302 learned_addr learned my addr 172.21.15.53:0/2181731302 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.258+0000 7f0a5d000640 1 --2- 172.21.15.53:0/2181731302 >> v2:172.21.15.53:3300/0 conn(0x7f0a5806d160 0x7f0a5806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.259+0000 7f0a57400640 1 --2- 172.21.15.53:0/2181731302 >> v2:172.21.15.146:3300/0 conn(0x7f0a580cc500 0x7f0a580cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.259+0000 7f0a5d000640 1 -- 172.21.15.53:0/2181731302 >> v2:172.21.15.146:3300/0 conn(0x7f0a580cc500 msgr2=0x7f0a580cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:16.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.259+0000 7f0a5d000640 1 --2- 172.21.15.53:0/2181731302 >> v2:172.21.15.146:3300/0 conn(0x7f0a580cc500 0x7f0a580cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.259+0000 7f0a5d000640 1 -- 172.21.15.53:0/2181731302 >> v1:172.21.15.53:6789/0 conn(0x7f0a58143c40 legacy=0x7f0a58071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.259+0000 7f0a5d000640 1 -- 172.21.15.53:0/2181731302 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a58144020 con 0x7f0a5806d160 2024-01-16T15:01:16.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.260+0000 7f0a5d000640 1 --2- 172.21.15.53:0/2181731302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5806d160 0x7f0a5806d560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0a580b44a0 tx=0x7f0a5002be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b31116ed81c2ad0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.260+0000 7f0a56a00640 1 -- 172.21.15.53:0/2181731302 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a5002cbe0 con 0x7f0a5806d160 2024-01-16T15:01:16.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.260+0000 7f0a56a00640 1 -- 172.21.15.53:0/2181731302 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a5002cd40 con 0x7f0a5806d160 2024-01-16T15:01:16.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.260+0000 7f0a56a00640 1 -- 172.21.15.53:0/2181731302 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50034660 con 0x7f0a5806d160 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 -- 172.21.15.53:0/2181731302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5806d160 msgr2=0x7f0a5806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 --2- 172.21.15.53:0/2181731302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5806d160 0x7f0a5806d560 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0a580b44a0 tx=0x7f0a5002be30 comp rx=0 tx=0).stop 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 -- 172.21.15.53:0/2181731302 shutdown_connections 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 --2- 172.21.15.53:0/2181731302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a5806d160 0x7f0a5806d560 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 --2- 172.21.15.53:0/2181731302 >> v2:172.21.15.146:3300/0 conn(0x7f0a580cc500 0x7f0a580cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 -- 172.21.15.53:0/2181731302 >> 172.21.15.53:0/2181731302 conn(0x7f0a5806a800 msgr2=0x7f0a5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 -- 172.21.15.53:0/2181731302 shutdown_connections 2024-01-16T15:01:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.261+0000 7f0a5da00640 1 -- 172.21.15.53:0/2181731302 wait complete. 2024-01-16T15:01:16.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.263+0000 7f0a5da00640 1 Processor -- start 2024-01-16T15:01:16.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.263+0000 7f0a5da00640 1 -- start start 2024-01-16T15:01:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.264+0000 7f0a5da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5806d160 0x7f0a5815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.264+0000 7f0a5da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cc500 0x7f0a58141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.264+0000 7f0a5da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 0x7f0a58141770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.264+0000 7f0a5da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58135dd0 con 0x7f0a58143c40 2024-01-16T15:01:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.264+0000 7f0a5da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58136040 con 0x7f0a580cc500 2024-01-16T15:01:16.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.264+0000 7f0a5da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a58136320 con 0x7f0a5806d160 2024-01-16T15:01:16.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 0x7f0a58141770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 0x7f0a58141770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38702/0 (socket says 172.21.15.53:38702) 2024-01-16T15:01:16.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 -- 172.21.15.53:0/1840538252 learned_addr learned my addr 172.21.15.53:0/1840538252 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a57400640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cc500 0x7f0a58141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a57e00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5806d160 0x7f0a5815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 -- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5806d160 msgr2=0x7f0a5815a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5806d160 0x7f0a5815a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 -- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cc500 msgr2=0x7f0a58141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cc500 0x7f0a58141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 -- 172.21.15.53:0/1840538252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a50009a70 con 0x7f0a58143c40 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a57e00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5806d160 0x7f0a5815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a57400640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cc500 0x7f0a58141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.265+0000 7f0a5d000640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 0x7f0a58141770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009e80 tx=0x7f0a50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.266+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50040650 con 0x7f0a58143c40 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.266+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a500407d0 con 0x7f0a58143c40 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.266+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a50040b80 con 0x7f0a58143c40 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.266+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a58136600 con 0x7f0a58143c40 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.266+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a58136b20 con 0x7f0a58143c40 2024-01-16T15:01:16.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.267+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a58136820 con 0x7f0a58143c40 2024-01-16T15:01:16.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.268+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f0a50040ce0 con 0x7f0a58143c40 2024-01-16T15:01:16.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.269+0000 7f0a4fe00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.269+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f0a500ac750 con 0x7f0a58143c40 2024-01-16T15:01:16.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.270+0000 7f0a57e00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.271+0000 7f0a57e00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0a48007fd0 tx=0x7f0a48009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.274+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a50077870 con 0x7f0a58143c40 2024-01-16T15:01:16.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.469+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0a5813fbf0 con 0x7f0a58143c40 2024-01-16T15:01:16.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.470+0000 7f0a4fe00640 1 -- 172.21.15.53:0/1840538252 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v31) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0a5007ae90 con 0x7f0a58143c40 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:16.450 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.472+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 msgr2=0x7f0a3006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.472+0000 7f0a5da00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0a48007fd0 tx=0x7f0a48009040 comp rx=0 tx=0).stop 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.472+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 msgr2=0x7f0a58141770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.472+0000 7f0a5da00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 0x7f0a58141770 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0a50009e80 tx=0x7f0a50004a40 comp rx=0 tx=0).stop 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 shutdown_connections 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a3006b2c0 0x7f0a3006d780 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a58143c40 0x7f0a58141770 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a580cc500 0x7f0a58141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 --2- 172.21.15.53:0/1840538252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a5806d160 0x7f0a5815a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 >> 172.21.15.53:0/1840538252 conn(0x7f0a5806a800 msgr2=0x7f0a580aec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 shutdown_connections 2024-01-16T15:01:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.473+0000 7f0a5da00640 1 -- 172.21.15.53:0/1840538252 wait complete. 2024-01-16T15:01:16.472 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:01:16.472 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:01:16.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.713+0000 7f5077800640 1 Processor -- start 2024-01-16T15:01:16.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.713+0000 7f5077800640 1 -- start start 2024-01-16T15:01:16.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.715+0000 7f5077800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f507013df80 0x7f507015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.715+0000 7f5077800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507013e360 0x7f50700cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.715+0000 7f5077800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f507015f9c0 0x7f50700cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.715+0000 7f5077800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507006c470 con 0x7f507013df80 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.715+0000 7f5077800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f507006c750 con 0x7f507015f9c0 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.715+0000 7f5077800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f507006ca30 con 0x7f507013e360 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f507013df80 0x7f507015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f507013df80 0x7f507015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38714/0 (socket says 172.21.15.53:38714) 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 -- 172.21.15.53:0/1284861295 learned_addr learned my addr 172.21.15.53:0/1284861295 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f506fe00640 1 --2- 172.21.15.53:0/1284861295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507013e360 0x7f50700cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5075800640 1 --2- 172.21.15.53:0/1284861295 >> v2:172.21.15.146:3300/0 conn(0x7f507015f9c0 0x7f50700cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 -- 172.21.15.53:0/1284861295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507013e360 msgr2=0x7f50700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:16.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 --2- 172.21.15.53:0/1284861295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507013e360 0x7f50700cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 -- 172.21.15.53:0/1284861295 >> v2:172.21.15.146:3300/0 conn(0x7f507015f9c0 msgr2=0x7f50700cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:16.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 --2- 172.21.15.53:0/1284861295 >> v2:172.21.15.146:3300/0 conn(0x7f507015f9c0 0x7f50700cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.716+0000 7f5074e00640 1 -- 172.21.15.53:0/1284861295 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f507006cd10 con 0x7f507013df80 2024-01-16T15:01:16.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.718+0000 7f5074e00640 1 --2- 172.21.15.53:0/1284861295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f507015bee0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f507006e6d0 tx=0x7f506002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f6b5897435eed3d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f506f400640 1 -- 172.21.15.53:0/1284861295 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060031870 con 0x7f507013df80 2024-01-16T15:01:16.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f506f400640 1 -- 172.21.15.53:0/1284861295 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50600319d0 con 0x7f507013df80 2024-01-16T15:01:16.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f506f400640 1 -- 172.21.15.53:0/1284861295 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5060031d60 con 0x7f507013df80 2024-01-16T15:01:16.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 -- 172.21.15.53:0/1284861295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 msgr2=0x7f507015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 --2- 172.21.15.53:0/1284861295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f507015bee0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f507006e6d0 tx=0x7f506002b910 comp rx=0 tx=0).stop 2024-01-16T15:01:16.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 -- 172.21.15.53:0/1284861295 shutdown_connections 2024-01-16T15:01:16.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 --2- 172.21.15.53:0/1284861295 >> v2:172.21.15.146:3300/0 conn(0x7f507015f9c0 0x7f50700cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 --2- 172.21.15.53:0/1284861295 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507013e360 0x7f50700cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 --2- 172.21.15.53:0/1284861295 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f507015bee0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.719+0000 7f5077800640 1 -- 172.21.15.53:0/1284861295 >> 172.21.15.53:0/1284861295 conn(0x7f507006a800 msgr2=0x7f507006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:16.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.720+0000 7f5077800640 1 -- 172.21.15.53:0/1284861295 shutdown_connections 2024-01-16T15:01:16.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.720+0000 7f5077800640 1 -- 172.21.15.53:0/1284861295 wait complete. 2024-01-16T15:01:16.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.721+0000 7f5077800640 1 Processor -- start 2024-01-16T15:01:16.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.721+0000 7f5077800640 1 -- start start 2024-01-16T15:01:16.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.722+0000 7f5077800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f5070154a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.722+0000 7f5077800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507013e360 0x7f507015d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.722+0000 7f5077800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015f9c0 0x7f50701f2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.722+0000 7f5077800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070154f90 con 0x7f507013df80 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.722+0000 7f5077800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5070070810 con 0x7f507013e360 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.722+0000 7f5077800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5070070af0 con 0x7f507015f9c0 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f5070154a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f5070154a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38718/0 (socket says 172.21.15.53:38718) 2024-01-16T15:01:16.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 -- 172.21.15.53:0/2686009635 learned_addr learned my addr 172.21.15.53:0/2686009635 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:16.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 -- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015f9c0 msgr2=0x7f50701f2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5075800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015f9c0 0x7f50701f2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f506fe00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507013e360 0x7f507015d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015f9c0 0x7f50701f2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 -- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507013e360 msgr2=0x7f507015d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507013e360 0x7f507015d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.723+0000 7f5074e00640 1 -- 172.21.15.53:0/2686009635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50600094d0 con 0x7f507013df80 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.724+0000 7f5074e00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f5070154a50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f506002b950 tx=0x7f50600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.724+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f506003ad30 con 0x7f507013df80 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.724+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50600443f0 con 0x7f507013df80 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.724+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50600447a0 con 0x7f507013df80 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.724+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5070070dd0 con 0x7f507013df80 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.724+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5070071390 con 0x7f507013df80 2024-01-16T15:01:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.725+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50700675d0 con 0x7f507013df80 2024-01-16T15:01:16.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.726+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f5060044900 con 0x7f507013df80 2024-01-16T15:01:16.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.727+0000 7f506cc00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b310 0x7f504806d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:16.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.727+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f506002d040 con 0x7f507013df80 2024-01-16T15:01:16.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.728+0000 7f506fe00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b310 0x7f504806d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.729+0000 7f506fe00640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b310 0x7f504806d7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f507013d9e0 tx=0x7f5058008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:16.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.732+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5060077040 con 0x7f507013df80 2024-01-16T15:01:16.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:16.925+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f507006df00 con 0x7f507013df80 2024-01-16T15:01:17.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.443+0000 7f506cc00640 1 -- 172.21.15.53:0/2686009635 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v32) v1 ==== 138+0+0 (secure 0 0 0) 0x7f5060035020 con 0x7f507013df80 2024-01-16T15:01:17.423 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:01:17.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b310 msgr2=0x7f504806d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:17.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b310 0x7f504806d7d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f507013d9e0 tx=0x7f5058008040 comp rx=0 tx=0).stop 2024-01-16T15:01:17.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 msgr2=0x7f5070154a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:17.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f5070154a50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f506002b950 tx=0x7f50600046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 shutdown_connections 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f504806b310 0x7f504806d7d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f507015f9c0 0x7f50701f2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f507013e360 0x7f507015d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.449+0000 7f5077800640 1 --2- 172.21.15.53:0/2686009635 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f507013df80 0x7f5070154a50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.450+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 >> 172.21.15.53:0/2686009635 conn(0x7f507006a800 msgr2=0x7f507015e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.450+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 shutdown_connections 2024-01-16T15:01:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.450+0000 7f5077800640 1 -- 172.21.15.53:0/2686009635 wait complete. 2024-01-16T15:01:17.456 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:01:17.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.689+0000 7f2f0ca00640 1 Processor -- start 2024-01-16T15:01:17.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.689+0000 7f2f0ca00640 1 -- start start 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415dd40 0x7f2f0415e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2f0415bed0 0x7f2f0415c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0415e6e0 con 0x7f2f04157dc0 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f041581a0 con 0x7f2f0415bed0 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0415ba40 con 0x7f2f0415dd40 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2f0415bed0 0x7f2f0415c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2f0415bed0 0x7f2f0415c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38724/0 (socket says 172.21.15.53:38724) 2024-01-16T15:01:17.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.691+0000 7f2f0aa00640 1 -- 172.21.15.53:0/1171930572 learned_addr learned my addr 172.21.15.53:0/1171930572 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:17.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.692+0000 7f2f09600640 1 --2- 172.21.15.53:0/1171930572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415dd40 0x7f2f0415e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:17.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.692+0000 7f2f0aa00640 1 -- 172.21.15.53:0/1171930572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415dd40 msgr2=0x7f2f0415e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:17.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.692+0000 7f2f0aa00640 1 --2- 172.21.15.53:0/1171930572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415dd40 0x7f2f0415e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.692+0000 7f2f0aa00640 1 -- 172.21.15.53:0/1171930572 >> v1:172.21.15.146:6789/0 conn(0x7f2f04157dc0 legacy=0x7f2f04070cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:17.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.692+0000 7f2f0aa00640 1 -- 172.21.15.53:0/1171930572 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0415eb10 con 0x7f2f0415bed0 2024-01-16T15:01:17.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.693+0000 7f2f0aa00640 1 --2- 172.21.15.53:0/1171930572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f0415bed0 0x7f2f0415c360 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0098a0 tx=0x7f2efc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2cd77553378b859 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:17.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.693+0000 7f2f08c00640 1 -- 172.21.15.53:0/1171930572 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc031640 con 0x7f2f0415bed0 2024-01-16T15:01:17.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.693+0000 7f2f08c00640 1 -- 172.21.15.53:0/1171930572 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2efc0317a0 con 0x7f2f0415bed0 2024-01-16T15:01:17.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.693+0000 7f2f08c00640 1 -- 172.21.15.53:0/1171930572 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2efc031b30 con 0x7f2f0415bed0 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 -- 172.21.15.53:0/1171930572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f0415bed0 msgr2=0x7f2f0415c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/1171930572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f0415bed0 0x7f2f0415c360 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f2efc0098a0 tx=0x7f2efc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 -- 172.21.15.53:0/1171930572 shutdown_connections 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/1171930572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f0415bed0 0x7f2f0415c360 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/1171930572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415dd40 0x7f2f0415e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 -- 172.21.15.53:0/1171930572 >> 172.21.15.53:0/1171930572 conn(0x7f2f040b0be0 msgr2=0x7f2f040b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 -- 172.21.15.53:0/1171930572 shutdown_connections 2024-01-16T15:01:17.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.694+0000 7f2f0ca00640 1 -- 172.21.15.53:0/1171930572 wait complete. 2024-01-16T15:01:17.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.696+0000 7f2f0ca00640 1 Processor -- start 2024-01-16T15:01:17.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.696+0000 7f2f0ca00640 1 -- start start 2024-01-16T15:01:17.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 0x7f2f04142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:17.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415bed0 0x7f2f040afba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:17.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f0415dd40 0x7f2f040b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04142f80 con 0x7f2f04157dc0 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0413d090 con 0x7f2f0415dd40 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f04154ff0 con 0x7f2f0415bed0 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 0x7f2f04142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 0x7f2f04142a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38732/0 (socket says 172.21.15.53:38732) 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.697+0000 7f2f0a000640 1 -- 172.21.15.53:0/3477779094 learned_addr learned my addr 172.21.15.53:0/3477779094 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f09600640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415bed0 0x7f2f040afba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0a000640 1 -- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415bed0 msgr2=0x7f2f040afba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0a000640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415bed0 0x7f2f040afba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0aa00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f0415dd40 0x7f2f040b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0a000640 1 -- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f0415dd40 msgr2=0x7f2f040b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0a000640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f0415dd40 0x7f2f040b00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:17.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0a000640 1 -- 172.21.15.53:0/3477779094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2efc0094d0 con 0x7f2f04157dc0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.698+0000 7f2f0a000640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 0x7f2f04142a40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c7c0 tx=0x7f2ef400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.699+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef400aa50 con 0x7f2f04157dc0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.699+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ef400abb0 con 0x7f2f04157dc0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.699+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ef4012420 con 0x7f2f04157dc0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.699+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f041552d0 con 0x7f2f04157dc0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.699+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f04155890 con 0x7f2f04157dc0 2024-01-16T15:01:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.700+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f0413d240 con 0x7f2f04157dc0 2024-01-16T15:01:17.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.701+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2ef4006b50 con 0x7f2f04157dc0 2024-01-16T15:01:17.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.702+0000 7f2f02000640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2edc06b2c0 0x7f2edc06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:17.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.702+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2ef4089fe0 con 0x7f2f04157dc0 2024-01-16T15:01:17.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.703+0000 7f2f09600640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2edc06b2c0 0x7f2edc06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.704+0000 7f2f09600640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2edc06b2c0 0x7f2edc06d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006fd0 tx=0x7f2eec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.706+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ef4055010 con 0x7f2f04157dc0 2024-01-16T15:01:17.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:17.900+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2f0413d200 con 0x7f2f04157dc0 2024-01-16T15:01:18.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.449+0000 7f2f02000640 1 -- 172.21.15.53:0/3477779094 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v33) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2ef4022300 con 0x7f2f04157dc0 2024-01-16T15:01:18.431 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:01:18.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.452+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2edc06b2c0 msgr2=0x7f2edc06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.452+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2edc06b2c0 0x7f2edc06d780 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2eec006fd0 tx=0x7f2eec008040 comp rx=0 tx=0).stop 2024-01-16T15:01:18.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.452+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 msgr2=0x7f2f04142a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.452+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 0x7f2f04142a40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f2ef400c7c0 tx=0x7f2ef400cc80 comp rx=0 tx=0).stop 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 shutdown_connections 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2edc06b2c0 0x7f2edc06d780 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f0415dd40 0x7f2f040b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f0415bed0 0x7f2f040afba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 --2- 172.21.15.53:0/3477779094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f04157dc0 0x7f2f04142a40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 >> 172.21.15.53:0/3477779094 conn(0x7f2f040b0be0 msgr2=0x7f2f04158610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:18.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.453+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 shutdown_connections 2024-01-16T15:01:18.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.454+0000 7f2f0ca00640 1 -- 172.21.15.53:0/3477779094 wait complete. 2024-01-16T15:01:18.466 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:01:18.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.673+0000 7ff3e9000640 1 Processor -- start 2024-01-16T15:01:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.674+0000 7ff3e9000640 1 -- start start 2024-01-16T15:01:18.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.675+0000 7ff3e9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e414b2c0 0x7ff3e40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.675+0000 7ff3e9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3e40cf7d0 0x7ff3e4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.675+0000 7ff3e9000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3e4070ea0 0x7ff3e40b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.675+0000 7ff3e9000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e414b6a0 con 0x7ff3e40cf7d0 2024-01-16T15:01:18.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.675+0000 7ff3e9000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3e40cfbf0 con 0x7ff3e4070ea0 2024-01-16T15:01:18.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.675+0000 7ff3e9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e4071690 con 0x7ff3e414b2c0 2024-01-16T15:01:18.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3e40cf7d0 0x7ff3e4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3e40cf7d0 0x7ff3e4070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38742/0 (socket says 172.21.15.53:38742) 2024-01-16T15:01:18.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 -- 172.21.15.53:0/3066295935 learned_addr learned my addr 172.21.15.53:0/3066295935 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3e1e00640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e414b2c0 0x7ff3e40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3e2800640 1 --2- 172.21.15.53:0/3066295935 >> v2:172.21.15.146:3300/0 conn(0x7ff3e4070ea0 0x7ff3e40b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 -- 172.21.15.53:0/3066295935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e414b2c0 msgr2=0x7ff3e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e414b2c0 0x7ff3e40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 -- 172.21.15.53:0/3066295935 >> v2:172.21.15.146:3300/0 conn(0x7ff3e4070ea0 msgr2=0x7ff3e40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 --2- 172.21.15.53:0/3066295935 >> v2:172.21.15.146:3300/0 conn(0x7ff3e4070ea0 0x7ff3e40b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3dbe00640 1 -- 172.21.15.53:0/3066295935 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e4071970 con 0x7ff3e40cf7d0 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.676+0000 7ff3e1e00640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e414b2c0 0x7ff3e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.677+0000 7ff3dbe00640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e40cf7d0 0x7ff3e4070960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff3e406e2a0 tx=0x7ff3cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9d392737b43b99 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.677+0000 7ff3e1400640 1 -- 172.21.15.53:0/3066295935 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031870 con 0x7ff3e40cf7d0 2024-01-16T15:01:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e1400640 1 -- 172.21.15.53:0/3066295935 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3cc0319d0 con 0x7ff3e40cf7d0 2024-01-16T15:01:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e1400640 1 -- 172.21.15.53:0/3066295935 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3cc031d60 con 0x7ff3e40cf7d0 2024-01-16T15:01:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 -- 172.21.15.53:0/3066295935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e40cf7d0 msgr2=0x7ff3e4070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e40cf7d0 0x7ff3e4070960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7ff3e406e2a0 tx=0x7ff3cc02b910 comp rx=0 tx=0).stop 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 -- 172.21.15.53:0/3066295935 shutdown_connections 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3066295935 >> v2:172.21.15.146:3300/0 conn(0x7ff3e4070ea0 0x7ff3e40b1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e40cf7d0 0x7ff3e4070960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3066295935 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e414b2c0 0x7ff3e40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.678+0000 7ff3e9000640 1 -- 172.21.15.53:0/3066295935 >> 172.21.15.53:0/3066295935 conn(0x7ff3e406a800 msgr2=0x7ff3e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.679+0000 7ff3e9000640 1 -- 172.21.15.53:0/3066295935 shutdown_connections 2024-01-16T15:01:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.679+0000 7ff3e9000640 1 -- 172.21.15.53:0/3066295935 wait complete. 2024-01-16T15:01:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.680+0000 7ff3e9000640 1 Processor -- start 2024-01-16T15:01:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.680+0000 7ff3e9000640 1 -- start start 2024-01-16T15:01:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.681+0000 7ff3e9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e4070ea0 0x7ff3e40a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.681+0000 7ff3e9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e40cf7d0 0x7ff3e40a9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.681+0000 7ff3e9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 0x7ff3e4144cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e9000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40aa900 con 0x7ff3e414b2c0 2024-01-16T15:01:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40a8a00 con 0x7ff3e4070ea0 2024-01-16T15:01:18.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e40a9870 con 0x7ff3e40cf7d0 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 0x7ff3e4144cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 0x7ff3e4144cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38752/0 (socket says 172.21.15.53:38752) 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 -- 172.21.15.53:0/3530849012 learned_addr learned my addr 172.21.15.53:0/3530849012 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3dbe00640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e40cf7d0 0x7ff3e40a9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e1e00640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e4070ea0 0x7ff3e40a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 -- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e40cf7d0 msgr2=0x7ff3e40a9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e40cf7d0 0x7ff3e40a9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 -- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e4070ea0 msgr2=0x7ff3e40a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e4070ea0 0x7ff3e40a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.682+0000 7ff3e2800640 1 -- 172.21.15.53:0/3530849012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3cc0094d0 con 0x7ff3e414b2c0 2024-01-16T15:01:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.683+0000 7ff3e2800640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 0x7ff3e4144cc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00cbb0 tx=0x7ff3dc00a4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.683+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc00cea0 con 0x7ff3e414b2c0 2024-01-16T15:01:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.683+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3dc00abd0 con 0x7ff3e414b2c0 2024-01-16T15:01:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.683+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3dc011410 con 0x7ff3e414b2c0 2024-01-16T15:01:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.683+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e40a8c60 con 0x7ff3e414b2c0 2024-01-16T15:01:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.683+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e40a9220 con 0x7ff3e414b2c0 2024-01-16T15:01:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.685+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e40675d0 con 0x7ff3e414b2c0 2024-01-16T15:01:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.685+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff3dc011570 con 0x7ff3e414b2c0 2024-01-16T15:01:18.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.687+0000 7ff3da000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff3bc06b2f0 0x7ff3bc06d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:18.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.687+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5609+0+0 (secure 0 0 0) 0x7ff3dc089960 con 0x7ff3e414b2c0 2024-01-16T15:01:18.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.687+0000 7ff3e1e00640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff3bc06b2f0 0x7ff3bc06d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:18.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.688+0000 7ff3e1e00640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff3bc06b2f0 0x7ff3bc06d7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400a990 tx=0x7ff3d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:18.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.691+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3dc054820 con 0x7ff3e414b2c0 2024-01-16T15:01:18.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:18.923+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff3e4161e10 con 0x7ff3e414b2c0 2024-01-16T15:01:19.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.464+0000 7ff3da000640 1 -- 172.21.15.53:0/3530849012 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7ff3dc01f0c0 con 0x7ff3e414b2c0 2024-01-16T15:01:19.445 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 4 and data pool 5 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff3bc06b2f0 msgr2=0x7ff3bc06d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff3bc06b2f0 0x7ff3bc06d7b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400a990 tx=0x7ff3d4008040 comp rx=0 tx=0).stop 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 msgr2=0x7ff3e4144cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 0x7ff3e4144cc0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff3dc00cbb0 tx=0x7ff3dc00a4c0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 shutdown_connections 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff3bc06b2f0 0x7ff3bc06d7b0 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff3d400a990 tx=0x7ff3d4008040 comp rx=0 tx=0).stop 2024-01-16T15:01:19.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3e414b2c0 0x7ff3e4144cc0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3e40cf7d0 0x7ff3e40a9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 --2- 172.21.15.53:0/3530849012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3e4070ea0 0x7ff3e40a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 >> 172.21.15.53:0/3530849012 conn(0x7ff3e406a800 msgr2=0x7ff3e406c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 shutdown_connections 2024-01-16T15:01:19.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.469+0000 7ff3e9000640 1 -- 172.21.15.53:0/3530849012 wait complete. 2024-01-16T15:01:19.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:19.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.679+0000 7f1df6000640 1 Processor -- start 2024-01-16T15:01:19.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.679+0000 7f1df6000640 1 -- start start 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1df6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df013cfa0 0x7f1df015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1df6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1df00cd530 0x7f1df013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1df6000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df00cdec0 con 0x7f1df00cd9b0 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1df6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df00ce1a0 con 0x7f1df00cd530 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1df6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df00ce480 con 0x7f1df013cfa0 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1defe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1df00cd530 0x7f1df013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1defe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1df00cd530 0x7f1df013f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38754/0 (socket says 172.21.15.53:38754) 2024-01-16T15:01:19.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.681+0000 7f1defe00640 1 -- 172.21.15.53:0/744074003 learned_addr learned my addr 172.21.15.53:0/744074003 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:19.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.682+0000 7f1defe00640 1 -- 172.21.15.53:0/744074003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df013cfa0 msgr2=0x7f1df015afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:19.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.682+0000 7f1defe00640 1 --2- 172.21.15.53:0/744074003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df013cfa0 0x7f1df015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.682+0000 7f1defe00640 1 -- 172.21.15.53:0/744074003 >> v1:172.21.15.146:6789/0 conn(0x7f1df00cd9b0 legacy=0x7f1df015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:19.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.682+0000 7f1defe00640 1 -- 172.21.15.53:0/744074003 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df00ce760 con 0x7f1df00cd530 2024-01-16T15:01:19.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.683+0000 7f1defe00640 1 --2- 172.21.15.53:0/744074003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd530 0x7f1df013f5a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1df006e2a0 tx=0x7f1de002da70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6d6d0d7471cd11 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.683+0000 7f1def400640 1 -- 172.21.15.53:0/744074003 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de002f830 con 0x7f1df00cd530 2024-01-16T15:01:19.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.683+0000 7f1def400640 1 -- 172.21.15.53:0/744074003 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de002f990 con 0x7f1df00cd530 2024-01-16T15:01:19.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.683+0000 7f1def400640 1 -- 172.21.15.53:0/744074003 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de002fd20 con 0x7f1df00cd530 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 -- 172.21.15.53:0/744074003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd530 msgr2=0x7f1df013f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 --2- 172.21.15.53:0/744074003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd530 0x7f1df013f5a0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f1df006e2a0 tx=0x7f1de002da70 comp rx=0 tx=0).stop 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 -- 172.21.15.53:0/744074003 shutdown_connections 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 --2- 172.21.15.53:0/744074003 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd530 0x7f1df013f5a0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 --2- 172.21.15.53:0/744074003 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df013cfa0 0x7f1df015afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 -- 172.21.15.53:0/744074003 >> 172.21.15.53:0/744074003 conn(0x7f1df006a800 msgr2=0x7f1df006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 -- 172.21.15.53:0/744074003 shutdown_connections 2024-01-16T15:01:19.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.684+0000 7f1df6000640 1 -- 172.21.15.53:0/744074003 wait complete. 2024-01-16T15:01:19.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.686+0000 7f1df6000640 1 Processor -- start 2024-01-16T15:01:19.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.686+0000 7f1df6000640 1 -- start start 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1df6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df00cd530 0x7f1df0152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1df6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 0x7f1df0152ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1df6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df013cfa0 0x7f1df01533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1df6000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df0159a10 con 0x7f1df00cd9b0 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1df6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df00af860 con 0x7f1df013cfa0 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1df6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df01f3b90 con 0x7f1df00cd530 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1defe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 0x7f1df0152ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1defe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 0x7f1df0152ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38770/0 (socket says 172.21.15.53:38770) 2024-01-16T15:01:19.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.687+0000 7f1defe00640 1 -- 172.21.15.53:0/2465492703 learned_addr learned my addr 172.21.15.53:0/2465492703 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:19.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1df4c00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df00cd530 0x7f1df0152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1defe00640 1 -- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df00cd530 msgr2=0x7f1df0152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1df5600640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df013cfa0 0x7f1df01533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1defe00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df00cd530 0x7f1df0152960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1defe00640 1 -- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df013cfa0 msgr2=0x7f1df01533e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1defe00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df013cfa0 0x7f1df01533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1defe00640 1 -- 172.21.15.53:0/2465492703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de00094d0 con 0x7f1df00cd9b0 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1df4c00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df00cd530 0x7f1df0152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1defe00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 0x7f1df0152ea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009960 tx=0x7f1de00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.688+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de003d070 con 0x7f1df00cd9b0 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.689+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de002f7e0 con 0x7f1df00cd9b0 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.689+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df0159730 con 0x7f1df00cd9b0 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.689+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0039830 con 0x7f1df00cd9b0 2024-01-16T15:01:19.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.689+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df0152160 con 0x7f1df00cd9b0 2024-01-16T15:01:19.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.690+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df0067450 con 0x7f1df00cd9b0 2024-01-16T15:01:19.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.691+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1de0039990 con 0x7f1df00cd9b0 2024-01-16T15:01:19.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.692+0000 7f1decc00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dc806b2c0 0x7f1dc806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:19.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.692+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1de0034070 con 0x7f1df00cd9b0 2024-01-16T15:01:19.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.693+0000 7f1df4c00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dc806b2c0 0x7f1dc806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:19.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.694+0000 7f1df4c00640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dc806b2c0 0x7f1dc806d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800a870 tx=0x7f1dd8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:19.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.697+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1de0076ab0 con 0x7f1df00cd9b0 2024-01-16T15:01:19.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.893+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1df00cbff0 con 0x7f1df00cd9b0 2024-01-16T15:01:19.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.895+0000 7f1decc00640 1 -- 172.21.15.53:0/2465492703 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16546 (secure 0 0 0) 0x7f1de007a0d0 con 0x7f1df00cd9b0 2024-01-16T15:01:19.875 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:19.876 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":34,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:01:18.925470+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:01:16.979092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T15:01:17.951950+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dc806b2c0 msgr2=0x7f1dc806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dc806b2c0 0x7f1dc806d780 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1dd800a870 tx=0x7f1dd8008040 comp rx=0 tx=0).stop 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 msgr2=0x7f1df0152ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 0x7f1df0152ea0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009960 tx=0x7f1de00046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 shutdown_connections 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dc806b2c0 0x7f1dc806d780 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df013cfa0 0x7f1df01533e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df00cd9b0 0x7f1df0152ea0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 --2- 172.21.15.53:0/2465492703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df00cd530 0x7f1df0152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:19.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 >> 172.21.15.53:0/2465492703 conn(0x7f1df006a800 msgr2=0x7f1df00aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:19.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 shutdown_connections 2024-01-16T15:01:19.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:19.898+0000 7f1df6000640 1 -- 172.21.15.53:0/2465492703 wait complete. 2024-01-16T15:01:19.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:20.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.135+0000 7f2439200640 1 Processor -- start 2024-01-16T15:01:20.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.135+0000 7f2439200640 1 -- start start 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.136+0000 7f2439200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f2434161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.136+0000 7f2439200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24340ce8f0 0x7f24340ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.136+0000 7f2439200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24340cfbe0 0x7f24340d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2439200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24341618a0 con 0x7f24340ce8f0 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2439200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24340d0de0 con 0x7f24340cfbe0 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2439200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340d10c0 con 0x7f2434160f80 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24340ce8f0 0x7f24340ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24340ce8f0 0x7f24340ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38780/0 (socket says 172.21.15.53:38780) 2024-01-16T15:01:20.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 -- 172.21.15.53:0/874916268 learned_addr learned my addr 172.21.15.53:0/874916268 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2432800640 1 --2- 172.21.15.53:0/874916268 >> v2:172.21.15.146:3300/0 conn(0x7f24340cfbe0 0x7f24340d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 -- 172.21.15.53:0/874916268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 msgr2=0x7f2434161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 --2- 172.21.15.53:0/874916268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f2434161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 -- 172.21.15.53:0/874916268 >> v2:172.21.15.146:3300/0 conn(0x7f24340cfbe0 msgr2=0x7f24340d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 --2- 172.21.15.53:0/874916268 >> v2:172.21.15.146:3300/0 conn(0x7f24340cfbe0 0x7f24340d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.137+0000 7f2431400640 1 -- 172.21.15.53:0/874916268 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24340d13a0 con 0x7f24340ce8f0 2024-01-16T15:01:20.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.138+0000 7f2432800640 1 --2- 172.21.15.53:0/874916268 >> v2:172.21.15.146:3300/0 conn(0x7f24340cfbe0 0x7f24340d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.139+0000 7f2431400640 1 --2- 172.21.15.53:0/874916268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f24340ced50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f24340c6a40 tx=0x7f241c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc69e76be80c9bcc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.139+0000 7f2430a00640 1 -- 172.21.15.53:0/874916268 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c02ca80 con 0x7f24340ce8f0 2024-01-16T15:01:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.139+0000 7f2430a00640 1 -- 172.21.15.53:0/874916268 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241c02cbe0 con 0x7f24340ce8f0 2024-01-16T15:01:20.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.139+0000 7f2430a00640 1 -- 172.21.15.53:0/874916268 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241c0343f0 con 0x7f24340ce8f0 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 -- 172.21.15.53:0/874916268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 msgr2=0x7f24340ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 --2- 172.21.15.53:0/874916268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f24340ced50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f24340c6a40 tx=0x7f241c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 -- 172.21.15.53:0/874916268 shutdown_connections 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 --2- 172.21.15.53:0/874916268 >> v2:172.21.15.146:3300/0 conn(0x7f24340cfbe0 0x7f24340d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 --2- 172.21.15.53:0/874916268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f24340ced50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 --2- 172.21.15.53:0/874916268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f2434161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 -- 172.21.15.53:0/874916268 >> 172.21.15.53:0/874916268 conn(0x7f243409c550 msgr2=0x7f24340d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:20.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 -- 172.21.15.53:0/874916268 shutdown_connections 2024-01-16T15:01:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.140+0000 7f2439200640 1 -- 172.21.15.53:0/874916268 wait complete. 2024-01-16T15:01:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.142+0000 7f2439200640 1 Processor -- start 2024-01-16T15:01:20.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.142+0000 7f2439200640 1 -- start start 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2439200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f2434158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2439200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24340cfbe0 0x7f24340cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2439200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f24340cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2439200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340cb860 con 0x7f24340ce8f0 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2439200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24340cbb40 con 0x7f24340cfbe0 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2439200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24340cbe20 con 0x7f2434160f80 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2431e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f2434158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2431e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f2434158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38792/0 (socket says 172.21.15.53:38792) 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2431e00640 1 -- 172.21.15.53:0/4289258273 learned_addr learned my addr 172.21.15.53:0/4289258273 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.143+0000 7f2432800640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f24340cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2431e00640 1 -- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 msgr2=0x7f24340cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2431e00640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f24340cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2431e00640 1 -- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24340cfbe0 msgr2=0x7f24340cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2431e00640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24340cfbe0 0x7f24340cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2431e00640 1 -- 172.21.15.53:0/4289258273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c009b30 con 0x7f24340ce8f0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2432800640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f24340cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2431e00640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f2434158a20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f242400b410 tx=0x7f242400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424011830 con 0x7f24340ce8f0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24240119b0 con 0x7f24340ce8f0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2424011d60 con 0x7f24340ce8f0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24340cc100 con 0x7f24340ce8f0 2024-01-16T15:01:20.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.144+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24340cd080 con 0x7f24340ce8f0 2024-01-16T15:01:20.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.146+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24340cc620 con 0x7f24340ce8f0 2024-01-16T15:01:20.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.146+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f242401a400 con 0x7f24340ce8f0 2024-01-16T15:01:20.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.148+0000 7f242a000640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240c06b2c0 0x7f240c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.148+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2424089c90 con 0x7f24340ce8f0 2024-01-16T15:01:20.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.148+0000 7f2431400640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240c06b2c0 0x7f240c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.150+0000 7f2431400640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240c06b2c0 0x7f240c06d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f241c035000 tx=0x7f241c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.152+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24340cc620 con 0x7f24340ce8f0 2024-01-16T15:01:20.164 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:20.172 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:20.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.349+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24340cc620 con 0x7f24340ce8f0 2024-01-16T15:01:20.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.353+0000 7f242a000640 1 -- 172.21.15.53:0/4289258273 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f24340cc620 con 0x7f24340ce8f0 2024-01-16T15:01:20.333 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:20.333 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":35,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:01:20.006052+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:01:16.979092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T15:01:17.951950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:20.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240c06b2c0 msgr2=0x7f240c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240c06b2c0 0x7f240c06d780 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f241c035000 tx=0x7f241c02e000 comp rx=0 tx=0).stop 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 msgr2=0x7f2434158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f2434158a20 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f242400b410 tx=0x7f242400b8d0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 shutdown_connections 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240c06b2c0 0x7f240c06d780 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2434160f80 0x7f24340cb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24340cfbe0 0x7f24340cacc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 --2- 172.21.15.53:0/4289258273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24340ce8f0 0x7f2434158a20 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 >> 172.21.15.53:0/4289258273 conn(0x7f243409c550 msgr2=0x7f24340d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 shutdown_connections 2024-01-16T15:01:20.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.356+0000 7f2439200640 1 -- 172.21.15.53:0/4289258273 wait complete. 2024-01-16T15:01:20.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:01:20.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.603+0000 7f7537200640 1 Processor -- start 2024-01-16T15:01:20.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.604+0000 7f7537200640 1 -- start start 2024-01-16T15:01:20.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7537200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f753815d4b0 0x7f75380d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7537200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75380cf9a0 0x7f75380cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7537200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75380cddf0 0x7f75380ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7537200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75380d0320 con 0x7f75380cf9a0 2024-01-16T15:01:20.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7537200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75380b2730 con 0x7f753815d4b0 2024-01-16T15:01:20.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7537200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75380d1ec0 con 0x7f75380cddf0 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.605+0000 7f7535400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75380cf9a0 0x7f75380cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f75380cf9a0 0x7f75380cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38798/0 (socket says 172.21.15.53:38798) 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 -- 172.21.15.53:0/2448029536 learned_addr learned my addr 172.21.15.53:0/2448029536 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7536800640 1 --2- 172.21.15.53:0/2448029536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75380cddf0 0x7f75380ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535e00640 1 --2- 172.21.15.53:0/2448029536 >> v2:172.21.15.146:3300/0 conn(0x7f753815d4b0 0x7f75380d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 -- 172.21.15.53:0/2448029536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75380cddf0 msgr2=0x7f75380ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 --2- 172.21.15.53:0/2448029536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75380cddf0 0x7f75380ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 -- 172.21.15.53:0/2448029536 >> v2:172.21.15.146:3300/0 conn(0x7f753815d4b0 msgr2=0x7f75380d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 --2- 172.21.15.53:0/2448029536 >> v2:172.21.15.146:3300/0 conn(0x7f753815d4b0 0x7f75380d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.606+0000 7f7535400640 1 -- 172.21.15.53:0/2448029536 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75380b29d0 con 0x7f75380cf9a0 2024-01-16T15:01:20.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.607+0000 7f7535400640 1 --2- 172.21.15.53:0/2448029536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cfde0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f753808b5b0 tx=0x7f752002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f094667b7ba4fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.607+0000 7f7534a00640 1 -- 172.21.15.53:0/2448029536 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520031860 con 0x7f75380cf9a0 2024-01-16T15:01:20.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7534a00640 1 -- 172.21.15.53:0/2448029536 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75200319c0 con 0x7f75380cf9a0 2024-01-16T15:01:20.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7534a00640 1 -- 172.21.15.53:0/2448029536 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520031d50 con 0x7f75380cf9a0 2024-01-16T15:01:20.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 -- 172.21.15.53:0/2448029536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 msgr2=0x7f75380cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 --2- 172.21.15.53:0/2448029536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cfde0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f753808b5b0 tx=0x7f752002b900 comp rx=0 tx=0).stop 2024-01-16T15:01:20.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 -- 172.21.15.53:0/2448029536 shutdown_connections 2024-01-16T15:01:20.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 --2- 172.21.15.53:0/2448029536 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75380cddf0 0x7f75380ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 --2- 172.21.15.53:0/2448029536 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cfde0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 --2- 172.21.15.53:0/2448029536 >> v2:172.21.15.146:3300/0 conn(0x7f753815d4b0 0x7f75380d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 -- 172.21.15.53:0/2448029536 >> 172.21.15.53:0/2448029536 conn(0x7f75380d3000 msgr2=0x7f75380d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:20.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 -- 172.21.15.53:0/2448029536 shutdown_connections 2024-01-16T15:01:20.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.608+0000 7f7537200640 1 -- 172.21.15.53:0/2448029536 wait complete. 2024-01-16T15:01:20.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.610+0000 7f7537200640 1 Processor -- start 2024-01-16T15:01:20.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.610+0000 7f7537200640 1 -- start start 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.611+0000 7f7537200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75380cddf0 0x7f753815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.611+0000 7f7537200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cb310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.611+0000 7f7537200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f753815d4b0 0x7f753809ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.611+0000 7f7537200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753815c470 con 0x7f75380cf9a0 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.611+0000 7f7537200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75380cae80 con 0x7f75380cddf0 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.611+0000 7f7537200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75380ca270 con 0x7f753815d4b0 2024-01-16T15:01:20.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cb310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cb310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38806/0 (socket says 172.21.15.53:38806) 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 -- 172.21.15.53:0/2630611271 learned_addr learned my addr 172.21.15.53:0/2630611271 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535e00640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75380cddf0 0x7f753815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7536800640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f753815d4b0 0x7f753809ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 -- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f753815d4b0 msgr2=0x7f753809ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f753815d4b0 0x7f753809ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 -- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75380cddf0 msgr2=0x7f753815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75380cddf0 0x7f753815b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7535400640 1 -- 172.21.15.53:0/2630611271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75200094d0 con 0x7f75380cf9a0 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.612+0000 7f7536800640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f753815d4b0 0x7f753809ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:20.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.613+0000 7f7535400640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cb310 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f75200097c0 tx=0x7f75200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.613+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f752003ac60 con 0x7f75380cf9a0 2024-01-16T15:01:20.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.613+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f752003adc0 con 0x7f75380cf9a0 2024-01-16T15:01:20.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.613+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7520044610 con 0x7f75380cf9a0 2024-01-16T15:01:20.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.613+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753815da10 con 0x7f75380cf9a0 2024-01-16T15:01:20.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.613+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75380ca830 con 0x7f75380cf9a0 2024-01-16T15:01:20.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.614+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75380cb010 con 0x7f75380cf9a0 2024-01-16T15:01:20.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.615+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f752004d450 con 0x7f75380cf9a0 2024-01-16T15:01:20.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.616+0000 7f752e000640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f751006b2c0 0x7f751006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:20.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.616+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f75200ac150 con 0x7f75380cf9a0 2024-01-16T15:01:20.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.617+0000 7f7535e00640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f751006b2c0 0x7f751006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:20.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.618+0000 7f7535e00640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f751006b2c0 0x7f751006d780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7528006fd0 tx=0x7f7528008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:20.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.620+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7520076fb0 con 0x7f75380cf9a0 2024-01-16T15:01:20.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:20.856+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7538067450 con 0x7f75380cf9a0 2024-01-16T15:01:21.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.496+0000 7f752e000640 1 -- 172.21.15.53:0/2630611271 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f752007a5d0 con 0x7f75380cf9a0 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f751006b2c0 msgr2=0x7f751006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f751006b2c0 0x7f751006d780 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f7528006fd0 tx=0x7f7528008040 comp rx=0 tx=0).stop 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 msgr2=0x7f75380cb310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cb310 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f75200097c0 tx=0x7f75200046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 shutdown_connections 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f751006b2c0 0x7f751006d780 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f753815d4b0 0x7f753809ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75380cf9a0 0x7f75380cb310 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 --2- 172.21.15.53:0/2630611271 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75380cddf0 0x7f753815b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 >> 172.21.15.53:0/2630611271 conn(0x7f75380d3000 msgr2=0x7f753806b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:21.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 shutdown_connections 2024-01-16T15:01:21.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.500+0000 7f7537200640 1 -- 172.21.15.53:0/2630611271 wait complete. 2024-01-16T15:01:21.501 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:01:21.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:01:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.742+0000 7ff30d400640 1 Processor -- start 2024-01-16T15:01:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.742+0000 7ff30d400640 1 -- start start 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.743+0000 7ff30d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff30813df80 0x7ff30815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.743+0000 7ff30d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff30813e360 0x7ff3080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.743+0000 7ff30d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff30806c470 con 0x7ff30815f9c0 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.743+0000 7ff30d400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30806c750 con 0x7ff30813df80 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.743+0000 7ff30d400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff30806ca30 con 0x7ff30813e360 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff30813df80 0x7ff30815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff30813df80 0x7ff30815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38812/0 (socket says 172.21.15.53:38812) 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 -- 172.21.15.53:0/3139320685 learned_addr learned my addr 172.21.15.53:0/3139320685 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff305e00640 1 --2- 172.21.15.53:0/3139320685 >> v2:172.21.15.146:3300/0 conn(0x7ff30813e360 0x7ff3080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 -- 172.21.15.53:0/3139320685 >> v2:172.21.15.146:3300/0 conn(0x7ff30813e360 msgr2=0x7ff3080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 --2- 172.21.15.53:0/3139320685 >> v2:172.21.15.146:3300/0 conn(0x7ff30813e360 0x7ff3080cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 -- 172.21.15.53:0/3139320685 >> v1:172.21.15.146:6789/0 conn(0x7ff30815f9c0 legacy=0x7ff3081408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.744+0000 7ff306800640 1 -- 172.21.15.53:0/3139320685 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30806cd10 con 0x7ff30813df80 2024-01-16T15:01:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.746+0000 7ff306800640 1 --2- 172.21.15.53:0/3139320685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813df80 0x7ff30815bee0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff30806ded0 tx=0x7ff2f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cb45a2416431b26 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.746+0000 7ff305400640 1 -- 172.21.15.53:0/3139320685 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8031870 con 0x7ff30813df80 2024-01-16T15:01:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.746+0000 7ff305400640 1 -- 172.21.15.53:0/3139320685 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f80319d0 con 0x7ff30813df80 2024-01-16T15:01:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.746+0000 7ff305400640 1 -- 172.21.15.53:0/3139320685 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f8031d60 con 0x7ff30813df80 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 -- 172.21.15.53:0/3139320685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813df80 msgr2=0x7ff30815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 --2- 172.21.15.53:0/3139320685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813df80 0x7ff30815bee0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7ff30806ded0 tx=0x7ff2f802b910 comp rx=0 tx=0).stop 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 -- 172.21.15.53:0/3139320685 shutdown_connections 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 --2- 172.21.15.53:0/3139320685 >> v2:172.21.15.146:3300/0 conn(0x7ff30813e360 0x7ff3080cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 --2- 172.21.15.53:0/3139320685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813df80 0x7ff30815bee0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 -- 172.21.15.53:0/3139320685 >> 172.21.15.53:0/3139320685 conn(0x7ff30806a800 msgr2=0x7ff30806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 -- 172.21.15.53:0/3139320685 shutdown_connections 2024-01-16T15:01:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.747+0000 7ff30d400640 1 -- 172.21.15.53:0/3139320685 wait complete. 2024-01-16T15:01:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.749+0000 7ff30d400640 1 Processor -- start 2024-01-16T15:01:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.750+0000 7ff30d400640 1 -- start start 2024-01-16T15:01:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff30d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30813df80 0x7ff30815dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff30d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 0x7ff30814fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff30d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30815f9c0 0x7ff30814ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff30d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff308151150 con 0x7ff30813e360 2024-01-16T15:01:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff30d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30815e1b0 con 0x7ff30813df80 2024-01-16T15:01:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff30d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff308153500 con 0x7ff30815f9c0 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff305e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 0x7ff30814fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff305e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 0x7ff30814fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38826/0 (socket says 172.21.15.53:38826) 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff305e00640 1 -- 172.21.15.53:0/2998190942 learned_addr learned my addr 172.21.15.53:0/2998190942 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff306800640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30813df80 0x7ff30815dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.751+0000 7ff307200640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30815f9c0 0x7ff30814ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff305e00640 1 -- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30815f9c0 msgr2=0x7ff30814ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff305e00640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30815f9c0 0x7ff30814ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff305e00640 1 -- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30813df80 msgr2=0x7ff30815dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff305e00640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30813df80 0x7ff30815dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff305e00640 1 -- 172.21.15.53:0/2998190942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2f80094d0 con 0x7ff30813e360 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff305e00640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 0x7ff30814fad0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000d7d0 tx=0x7ff2f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f0018070 con 0x7ff30813e360 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2f0004020 con 0x7ff30813e360 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2f00041b0 con 0x7ff30813e360 2024-01-16T15:01:21.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.752+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff30814e0d0 con 0x7ff30813e360 2024-01-16T15:01:21.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.753+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff30814e690 con 0x7ff30813e360 2024-01-16T15:01:21.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.754+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3080675d0 con 0x7ff30813e360 2024-01-16T15:01:21.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.755+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7ff2f0004310 con 0x7ff30813e360 2024-01-16T15:01:21.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.755+0000 7ff2fea00640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff2e006b2c0 0x7ff2e006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:21.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.756+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff2f0089a60 con 0x7ff30813e360 2024-01-16T15:01:21.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.756+0000 7ff306800640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff2e006b2c0 0x7ff2e006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:21.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.757+0000 7ff306800640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff2e006b2c0 0x7ff2e006d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8036000 tx=0x7ff2f803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.760+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2f0054960 con 0x7ff30813e360 2024-01-16T15:01:21.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:21.996+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7ff3080cec50 con 0x7ff30813e360 2024-01-16T15:01:22.403 INFO:tasks.ceph.mds.c.smithi053.stderr:2024-01-16T15:01:22.423+0000 7f0974000640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:01:22.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.506+0000 7ff2fea00640 1 -- 172.21.15.53:0/2998190942 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff2f0057f80 con 0x7ff30813e360 2024-01-16T15:01:22.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff2e006b2c0 msgr2=0x7ff2e006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff2e006b2c0 0x7ff2e006d780 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff2f8036000 tx=0x7ff2f803c040 comp rx=0 tx=0).stop 2024-01-16T15:01:22.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 msgr2=0x7ff30814fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 0x7ff30814fad0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7ff2f000d7d0 tx=0x7ff2f000dc90 comp rx=0 tx=0).stop 2024-01-16T15:01:22.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 shutdown_connections 2024-01-16T15:01:22.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff2e006b2c0 0x7ff2e006d780 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff30815f9c0 0x7ff30814ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff30813e360 0x7ff30814fad0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 --2- 172.21.15.53:0/2998190942 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff30813df80 0x7ff30815dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.511+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 >> 172.21.15.53:0/2998190942 conn(0x7ff30806a800 msgr2=0x7ff3080af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.512+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 shutdown_connections 2024-01-16T15:01:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.512+0000 7ff30d400640 1 -- 172.21.15.53:0/2998190942 wait complete. 2024-01-16T15:01:22.514 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:01:22.514 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:22.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.757+0000 7fd67e200640 1 Processor -- start 2024-01-16T15:01:22.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.757+0000 7fd67e200640 1 -- start start 2024-01-16T15:01:22.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.758+0000 7fd67e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd67813e360 0x7fd6780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.758+0000 7fd67e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd6780cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.758+0000 7fd67e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd67815c5f0 con 0x7fd67813df80 2024-01-16T15:01:22.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd67e200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd67806c470 con 0x7fd67813e360 2024-01-16T15:01:22.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd67e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67806c750 con 0x7fd67815f9c0 2024-01-16T15:01:22.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd677400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd67813df80 0x7fd67815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44562/0 (socket says 172.21.15.53:44562) 2024-01-16T15:01:22.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd677400640 1 -- 172.21.15.53:0/129800130 learned_addr learned my addr 172.21.15.53:0/129800130 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:22.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd677e00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd6780cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd66fe00640 1 --2- 172.21.15.53:0/129800130 >> v2:172.21.15.146:3300/0 conn(0x7fd67813e360 0x7fd6780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701117350 0 0) 0x7fd67815c5f0 con 0x7fd67813df80 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.759+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd658003520 con 0x7fd67813df80 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943113215 0 0) 0x7fd658003520 con 0x7fd67813df80 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 msgr2=0x7fd6780cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd6780cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 >> v2:172.21.15.146:3300/0 conn(0x7fd67813e360 msgr2=0x7fd6780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 --2- 172.21.15.53:0/129800130 >> v2:172.21.15.146:3300/0 conn(0x7fd67813e360 0x7fd6780cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67806ca30 con 0x7fd67813df80 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd668001690 con 0x7fd67813df80 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 >> v1:172.21.15.53:6789/0 conn(0x7fd67813df80 legacy=0x7fd67815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd658003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd658003f40 0x7fd6580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658004d00 0x7fd658005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67806c470 con 0x7fd658003f40 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67806c750 con 0x7fd658004d00 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd65800b980 con 0x7fd67815f9c0 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.760+0000 7fd66fe00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd658003f40 0x7fd6580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd677e00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658004d00 0x7fd658005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd677400640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd658003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd66fe00640 1 -- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 msgr2=0x7fd658003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd66fe00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd658003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd66fe00640 1 -- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658004d00 msgr2=0x7fd658005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd66fe00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658004d00 0x7fd658005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd66fe00640 1 -- 172.21.15.53:0/129800130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd658003520 con 0x7fd658003f40 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd677e00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658004d00 0x7fd658005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.761+0000 7fd677400640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67815f9c0 0x7fd658003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:22.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.762+0000 7fd66fe00640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd658003f40 0x7fd6580047c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd660009ac0 tx=0x7fd6580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=325e8104eb58981 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.762+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66002c6b0 con 0x7fd658003f40 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.762+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd66002c810 con 0x7fd658003f40 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.762+0000 7fd676a00640 1 -- 172.21.15.53:0/129800130 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd66002cbc0 con 0x7fd658003f40 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd67e200640 1 -- 172.21.15.53:0/129800130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd658003f40 msgr2=0x7fd6580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd67e200640 1 --2- 172.21.15.53:0/129800130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd658003f40 0x7fd6580047c0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fd660009ac0 tx=0x7fd6580057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd677e00640 1 -- 172.21.15.53:0/129800130 reap_dead start 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd67e200640 1 -- 172.21.15.53:0/129800130 shutdown_connections 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd67e200640 1 -- 172.21.15.53:0/129800130 >> 172.21.15.53:0/129800130 conn(0x7fd67806a800 msgr2=0x7fd67806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd67e200640 1 -- 172.21.15.53:0/129800130 shutdown_connections 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.763+0000 7fd67e200640 1 -- 172.21.15.53:0/129800130 wait complete. 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.765+0000 7fd67e200640 1 Processor -- start 2024-01-16T15:01:22.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.765+0000 7fd67e200640 1 -- start start 2024-01-16T15:01:22.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd67e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658003f40 0x7fd678153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd67e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67814f530 0x7fd67814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd67e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 0x7fd678150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd67e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd678150640 con 0x7fd678150a60 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd67e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd67814f9c0 con 0x7fd658003f40 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd67e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd67814fca0 con 0x7fd67814f530 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd677e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 0x7fd678150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd677e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 0x7fd678150ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47726/0 (socket says 172.21.15.53:47726) 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.766+0000 7fd677e00640 1 -- 172.21.15.53:0/2778847882 learned_addr learned my addr 172.21.15.53:0/2778847882 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677400640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658003f40 0x7fd678153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd66fe00640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67814f530 0x7fd67814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677e00640 1 -- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67814f530 msgr2=0x7fd67814d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677e00640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67814f530 0x7fd67814d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677e00640 1 -- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658003f40 msgr2=0x7fd678153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677e00640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658003f40 0x7fd678153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677e00640 1 -- 172.21.15.53:0/2778847882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd67814ff80 con 0x7fd678150a60 2024-01-16T15:01:22.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd677e00640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 0x7fd678150ef0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd6780cfea0 tx=0x7fd67000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd670017400 con 0x7fd678150a60 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd670011040 con 0x7fd678150a60 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6700177c0 con 0x7fd678150a60 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.767+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd678150260 con 0x7fd678150a60 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.768+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd678154270 con 0x7fd678150a60 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.769+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd678150780 con 0x7fd678150a60 2024-01-16T15:01:22.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.770+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd6700203b0 con 0x7fd678150a60 2024-01-16T15:01:22.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.771+0000 7fd66f400640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd65006b2c0 0x7fd65006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:22.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.771+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd670089c50 con 0x7fd678150a60 2024-01-16T15:01:22.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.771+0000 7fd677400640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd65006b2c0 0x7fd65006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:22.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.772+0000 7fd677400640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd65006b2c0 0x7fd65006d780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd668001880 tx=0x7fd668009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:22.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:22.775+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd678150780 con 0x7fd678150a60 2024-01-16T15:01:22.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.009+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd678150780 con 0x7fd678150a60 2024-01-16T15:01:22.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.010+0000 7fd66f400640 1 -- 172.21.15.53:0/2778847882 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fd678150780 con 0x7fd678150a60 2024-01-16T15:01:22.990 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:22.990 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3324487785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3324487785},{"type":"v1","addr":"172.21.15.146:6836","nonce":3324487785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3124134221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3124134221},{"type":"v1","addr":"172.21.15.146:6837","nonce":3124134221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1994154705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1994154705},{"type":"v1","addr":"172.21.15.53:6834","nonce":1994154705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:01:18.926439+0000","modified":"2024-01-16T15:01:21.997432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5190},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5190":{"gid":5190,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/2729092299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2729092299},{"type":"v1","addr":"172.21.15.53:6835","nonce":2729092299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T15:01:22.991 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 16 2024-01-16T15:01:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd65006b2c0 msgr2=0x7fd65006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd65006b2c0 0x7fd65006d780 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd668001880 tx=0x7fd668009040 comp rx=0 tx=0).stop 2024-01-16T15:01:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 msgr2=0x7fd678150ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 0x7fd678150ef0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fd6780cfea0 tx=0x7fd67000f4a0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 shutdown_connections 2024-01-16T15:01:22.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd65006b2c0 0x7fd65006d780 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd678150a60 0x7fd678150ef0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd67814f530 0x7fd67814d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 --2- 172.21.15.53:0/2778847882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd658003f40 0x7fd678153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:22.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 >> 172.21.15.53:0/2778847882 conn(0x7fd67806a800 msgr2=0x7fd67815e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:22.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 shutdown_connections 2024-01-16T15:01:22.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.013+0000 7fd67e200640 1 -- 172.21.15.53:0/2778847882 wait complete. 2024-01-16T15:01:23.013 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:01:23.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.258+0000 7f1a87e00640 1 Processor -- start 2024-01-16T15:01:23.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.258+0000 7f1a87e00640 1 -- start start 2024-01-16T15:01:23.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a87e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a8813cfa0 0x7f1a8815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a87e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a880cdec0 con 0x7f1a880cd530 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a87e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a880ce1a0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a87e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a880ce480 con 0x7f1a8813cfa0 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a86a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a8813cfa0 0x7f1a8815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a86a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a8813cfa0 0x7f1a8815afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47736/0 (socket says 172.21.15.53:47736) 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.260+0000 7f1a86a00640 1 -- 172.21.15.53:0/2610603120 learned_addr learned my addr 172.21.15.53:0/2610603120 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.261+0000 7f1a86a00640 1 -- 172.21.15.53:0/2610603120 >> v1:172.21.15.146:6789/0 conn(0x7f1a8813fcb0 legacy=0x7f1a880b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.261+0000 7f1a86a00640 1 -- 172.21.15.53:0/2610603120 >> v1:172.21.15.53:6789/0 conn(0x7f1a880cd530 legacy=0x7f1a8813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.261+0000 7f1a86a00640 1 -- 172.21.15.53:0/2610603120 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a880ce760 con 0x7f1a8813cfa0 2024-01-16T15:01:23.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.262+0000 7f1a86a00640 1 --2- 172.21.15.53:0/2610603120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813cfa0 0x7f1a8815afb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800bb10 tx=0x7f1a7802e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=853130a8ac209e5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:23.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a85600640 1 -- 172.21.15.53:0/2610603120 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7802ee80 con 0x7f1a8813cfa0 2024-01-16T15:01:23.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a85600640 1 -- 172.21.15.53:0/2610603120 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a78036440 con 0x7f1a8813cfa0 2024-01-16T15:01:23.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a85600640 1 -- 172.21.15.53:0/2610603120 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a780367b0 con 0x7f1a8813cfa0 2024-01-16T15:01:23.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a87e00640 1 -- 172.21.15.53:0/2610603120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813cfa0 msgr2=0x7f1a8815afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a87e00640 1 --2- 172.21.15.53:0/2610603120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813cfa0 0x7f1a8815afb0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800bb10 tx=0x7f1a7802e0d0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a87e00640 1 -- 172.21.15.53:0/2610603120 shutdown_connections 2024-01-16T15:01:23.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a87e00640 1 --2- 172.21.15.53:0/2610603120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813cfa0 0x7f1a8815afb0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a87e00640 1 -- 172.21.15.53:0/2610603120 >> 172.21.15.53:0/2610603120 conn(0x7f1a8806a800 msgr2=0x7f1a8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:23.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.263+0000 7f1a87e00640 1 -- 172.21.15.53:0/2610603120 shutdown_connections 2024-01-16T15:01:23.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.264+0000 7f1a87e00640 1 -- 172.21.15.53:0/2610603120 wait complete. 2024-01-16T15:01:23.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.265+0000 7f1a87e00640 1 Processor -- start 2024-01-16T15:01:23.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.266+0000 7f1a87e00640 1 -- start start 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.266+0000 7f1a87e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a880cd530 0x7f1a881fb670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.266+0000 7f1a87e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8813cfa0 0x7f1a881f7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.266+0000 7f1a87e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 0x7f1a881555e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88156320 con 0x7f1a8813fcb0 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a881f8530 con 0x7f1a880cd530 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a88155b20 con 0x7f1a8813cfa0 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 0x7f1a881555e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 0x7f1a881555e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47748/0 (socket says 172.21.15.53:47748) 2024-01-16T15:01:23.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 -- 172.21.15.53:0/690892173 learned_addr learned my addr 172.21.15.53:0/690892173 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a86a00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a880cd530 0x7f1a881fb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a86000640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8813cfa0 0x7f1a881f7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 -- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8813cfa0 msgr2=0x7f1a881f7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8813cfa0 0x7f1a881f7ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 -- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a880cd530 msgr2=0x7f1a881fb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a880cd530 0x7f1a881fb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a87400640 1 -- 172.21.15.53:0/690892173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7800b770 con 0x7f1a8813fcb0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.267+0000 7f1a86a00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a880cd530 0x7f1a881fb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.268+0000 7f1a87400640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 0x7f1a881555e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000b9a0 tx=0x7f1a8000be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.268+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000cb40 con 0x7f1a8813fcb0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.268+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8000cca0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.268+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80014680 con 0x7f1a8813fcb0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.268+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a88155d60 con 0x7f1a8813fcb0 2024-01-16T15:01:23.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.268+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a881f77f0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.269+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a88067450 con 0x7f1a8813fcb0 2024-01-16T15:01:23.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.270+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1a8001d3b0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.271+0000 7f1a7ea00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a6006b2c0 0x7f1a6006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.271+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a800898d0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.272+0000 7f1a86a00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a6006b2c0 0x7f1a6006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.273+0000 7f1a86a00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a6006b2c0 0x7f1a6006d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a78038000 tx=0x7f1a78008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:23.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.276+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a800547d0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.475+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a880cbff0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.476+0000 7f1a7ea00640 1 -- 172.21.15.53:0/690892173 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) v1 ==== 74+0+16533 (secure 0 0 0) 0x7f1a80057df0 con 0x7f1a8813fcb0 2024-01-16T15:01:23.456 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:23.457 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":35,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:01:20.006052+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:01:16.979092+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-01-16T15:01:17.951950+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:01:23.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a6006b2c0 msgr2=0x7f1a6006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a6006b2c0 0x7f1a6006d780 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1a78038000 tx=0x7f1a78008040 comp rx=0 tx=0).stop 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 msgr2=0x7f1a881555e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 0x7f1a881555e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f1a8000b9a0 tx=0x7f1a8000be60 comp rx=0 tx=0).stop 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 shutdown_connections 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a6006b2c0 0x7f1a6006d780 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a8813fcb0 0x7f1a881555e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a8813cfa0 0x7f1a881f7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 --2- 172.21.15.53:0/690892173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a880cd530 0x7f1a881fb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.479+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 >> 172.21.15.53:0/690892173 conn(0x7f1a8806a800 msgr2=0x7f1a880ade30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.480+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 shutdown_connections 2024-01-16T15:01:23.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.480+0000 7f1a87e00640 1 -- 172.21.15.53:0/690892173 wait complete. 2024-01-16T15:01:23.480 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:01:23.481 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:01:23.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.720+0000 7fcd12800640 1 Processor -- start 2024-01-16T15:01:23.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.720+0000 7fcd12800640 1 -- start start 2024-01-16T15:01:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.722+0000 7fcd12800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd0c13cfa0 0x7fcd0c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.722+0000 7fcd12800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd0c0cd530 0x7fcd0c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.722+0000 7fcd12800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd0c0cdec0 con 0x7fcd0c0cd9e0 2024-01-16T15:01:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.722+0000 7fcd12800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0c0ce1a0 con 0x7fcd0c13cfa0 2024-01-16T15:01:23.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.722+0000 7fcd12800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd0c0ce480 con 0x7fcd0c0cd530 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd0c13cfa0 0x7fcd0c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd0c13cfa0 0x7fcd0c15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47760/0 (socket says 172.21.15.53:47760) 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 -- 172.21.15.53:0/1911278117 learned_addr learned my addr 172.21.15.53:0/1911278117 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd10a00640 1 --2- 172.21.15.53:0/1911278117 >> v2:172.21.15.146:3300/0 conn(0x7fcd0c0cd530 0x7fcd0c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 -- 172.21.15.53:0/1911278117 >> v2:172.21.15.146:3300/0 conn(0x7fcd0c0cd530 msgr2=0x7fcd0c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 --2- 172.21.15.53:0/1911278117 >> v2:172.21.15.146:3300/0 conn(0x7fcd0c0cd530 0x7fcd0c13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 -- 172.21.15.53:0/1911278117 >> v1:172.21.15.53:6789/0 conn(0x7fcd0c0cd9e0 legacy=0x7fcd0c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd11400640 1 -- 172.21.15.53:0/1911278117 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c0ce760 con 0x7fcd0c13cfa0 2024-01-16T15:01:23.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.723+0000 7fcd10a00640 1 --2- 172.21.15.53:0/1911278117 >> v2:172.21.15.146:3300/0 conn(0x7fcd0c0cd530 0x7fcd0c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:23.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.724+0000 7fcd11400640 1 --2- 172.21.15.53:0/1911278117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c15afb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c15b4f0 tx=0x7fccfc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e0580d01e825ea1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:23.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd0be00640 1 -- 172.21.15.53:0/1911278117 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc031880 con 0x7fcd0c13cfa0 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd0be00640 1 -- 172.21.15.53:0/1911278117 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc0319e0 con 0x7fcd0c13cfa0 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd0be00640 1 -- 172.21.15.53:0/1911278117 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc031d70 con 0x7fcd0c13cfa0 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd12800640 1 -- 172.21.15.53:0/1911278117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c13cfa0 msgr2=0x7fcd0c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd12800640 1 --2- 172.21.15.53:0/1911278117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c15afb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c15b4f0 tx=0x7fccfc02b920 comp rx=0 tx=0).stop 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd12800640 1 -- 172.21.15.53:0/1911278117 shutdown_connections 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd12800640 1 --2- 172.21.15.53:0/1911278117 >> v2:172.21.15.146:3300/0 conn(0x7fcd0c0cd530 0x7fcd0c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd12800640 1 --2- 172.21.15.53:0/1911278117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c15afb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.725+0000 7fcd12800640 1 -- 172.21.15.53:0/1911278117 >> 172.21.15.53:0/1911278117 conn(0x7fcd0c06a800 msgr2=0x7fcd0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.726+0000 7fcd12800640 1 -- 172.21.15.53:0/1911278117 shutdown_connections 2024-01-16T15:01:23.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.726+0000 7fcd12800640 1 -- 172.21.15.53:0/1911278117 wait complete. 2024-01-16T15:01:23.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.727+0000 7fcd12800640 1 Processor -- start 2024-01-16T15:01:23.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.728+0000 7fcd12800640 1 -- start start 2024-01-16T15:01:23.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.728+0000 7fcd12800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 0x7fcd0c12ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.728+0000 7fcd12800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0c0cd9e0 0x7fcd0c12f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.728+0000 7fcd12800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd12800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c155a60 con 0x7fcd0c0cd530 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd12800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c1578c0 con 0x7fcd0c13cfa0 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd12800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0c0ad5c0 con 0x7fcd0c0cd9e0 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 0x7fcd0c12ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 0x7fcd0c12ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47764/0 (socket says 172.21.15.53:47764) 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 -- 172.21.15.53:0/1550743114 learned_addr learned my addr 172.21.15.53:0/1550743114 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 -- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0c0cd9e0 msgr2=0x7fcd0c12f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd10a00640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0c0cd9e0 0x7fcd0c12f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11e00640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0c0cd9e0 0x7fcd0c12f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 -- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd0c13cfa0 msgr2=0x7fcd0c0ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c0ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11400640 1 -- 172.21.15.53:0/1550743114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccfc0094d0 con 0x7fcd0c0cd530 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.729+0000 7fcd11e00640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:23.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.730+0000 7fcd11400640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 0x7fcd0c12ec50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fccfc02b960 tx=0x7fccfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:23.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.730+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc031870 con 0x7fcd0c0cd530 2024-01-16T15:01:23.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.730+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccfc0443f0 con 0x7fcd0c0cd530 2024-01-16T15:01:23.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.730+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccfc0447a0 con 0x7fcd0c0cd530 2024-01-16T15:01:23.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.730+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0c0ad8a0 con 0x7fcd0c0cd530 2024-01-16T15:01:23.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.730+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c157520 con 0x7fcd0c0cd530 2024-01-16T15:01:23.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.731+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0c067450 con 0x7fcd0c0cd530 2024-01-16T15:01:23.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.732+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fccfc04d450 con 0x7fcd0c0cd530 2024-01-16T15:01:23.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.733+0000 7fcd09600640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcce406b2c0 0x7fcce406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:23.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.733+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fccfc0ac420 con 0x7fcd0c0cd530 2024-01-16T15:01:23.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.734+0000 7fcd10a00640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcce406b2c0 0x7fcce406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:23.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.735+0000 7fcd10a00640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcce406b2c0 0x7fcce406d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c15d560 tx=0x7fccf4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:23.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.738+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccfc0771f0 con 0x7fcd0c0cd530 2024-01-16T15:01:23.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.933+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd0c0b4000 con 0x7fcce406b2c0 2024-01-16T15:01:23.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.961+0000 7fcd09600640 1 -- 172.21.15.53:0/1550743114 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286684 (secure 0 0 0) 0x7fcd0c0b4000 con 0x7fcce406b2c0 2024-01-16T15:01:23.945 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:23.947 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcce406b2c0 msgr2=0x7fcce406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcce406b2c0 0x7fcce406d780 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c15d560 tx=0x7fccf4008040 comp rx=0 tx=0).stop 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 msgr2=0x7fcd0c12ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 0x7fcd0c12ec50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fccfc02b960 tx=0x7fccfc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 shutdown_connections 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcce406b2c0 0x7fcce406d780 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd0c13cfa0 0x7fcd0c0ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0c0cd9e0 0x7fcd0c12f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 --2- 172.21.15.53:0/1550743114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd0c0cd530 0x7fcd0c12ec50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:23.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 >> 172.21.15.53:0/1550743114 conn(0x7fcd0c06a800 msgr2=0x7fcd0c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:23.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 shutdown_connections 2024-01-16T15:01:23.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:23.967+0000 7fcd12800640 1 -- 172.21.15.53:0/1550743114 wait complete. 2024-01-16T15:01:23.968 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-01-16T15:01:22.200843+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236984,"kb_used_data":4188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737736,"statfs":{"total":773094113280,"available":772851441664,"internally_reserved":0,"allocated":4288512,"data_stored":2383222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.755005"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754626+0000","last_change":"2024-01-16T15:01:19.047111+0000","last_active":"2024-01-16T15:01:20.754626+0000","last_peered":"2024-01-16T15:01:20.754626+0000","last_clean":"2024-01-16T15:01:20.754626+0000","last_became_active":"2024-01-16T15:01:19.046844+0000","last_became_peered":"2024-01-16T15:01:19.046844+0000","last_unstale":"2024-01-16T15:01:20.754626+0000","last_undegraded":"2024-01-16T15:01:20.754626+0000","last_fullsized":"2024-01-16T15:01:20.754626+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.533005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.039308+0000","last_change":"2024-01-16T15:01:19.035158+0000","last_active":"2024-01-16T15:01:19.039308+0000","last_peered":"2024-01-16T15:01:19.039308+0000","last_clean":"2024-01-16T15:01:19.039308+0000","last_became_active":"2024-01-16T15:01:19.034893+0000","last_became_peered":"2024-01-16T15:01:19.034893+0000","last_unstale":"2024-01-16T15:01:19.039308+0000","last_undegraded":"2024-01-16T15:01:19.039308+0000","last_fullsized":"2024-01-16T15:01:19.039308+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:27:26.968382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019029+0000","last_change":"2024-01-16T15:01:19.048055+0000","last_active":"2024-01-16T15:01:20.019029+0000","last_peered":"2024-01-16T15:01:20.019029+0000","last_clean":"2024-01-16T15:01:20.019029+0000","last_became_active":"2024-01-16T15:01:19.046966+0000","last_became_peered":"2024-01-16T15:01:19.046966+0000","last_unstale":"2024-01-16T15:01:20.019029+0000","last_undegraded":"2024-01-16T15:01:20.019029+0000","last_fullsized":"2024-01-16T15:01:20.019029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:18.453597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049185+0000","last_change":"2024-01-16T15:01:19.044420+0000","last_active":"2024-01-16T15:01:19.049185+0000","last_peered":"2024-01-16T15:01:19.049185+0000","last_clean":"2024-01-16T15:01:19.049185+0000","last_became_active":"2024-01-16T15:01:19.043953+0000","last_became_peered":"2024-01-16T15:01:19.043953+0000","last_unstale":"2024-01-16T15:01:19.049185+0000","last_undegraded":"2024-01-16T15:01:19.049185+0000","last_fullsized":"2024-01-16T15:01:19.049185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:08.957102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.046920+0000","last_change":"2024-01-16T15:01:19.037628+0000","last_active":"2024-01-16T15:01:19.046920+0000","last_peered":"2024-01-16T15:01:19.046920+0000","last_clean":"2024-01-16T15:01:19.046920+0000","last_became_active":"2024-01-16T15:01:19.037334+0000","last_became_peered":"2024-01-16T15:01:19.037334+0000","last_unstale":"2024-01-16T15:01:19.046920+0000","last_undegraded":"2024-01-16T15:01:19.046920+0000","last_fullsized":"2024-01-16T15:01:19.046920+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:32:40.541057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.041145+0000","last_change":"2024-01-16T15:01:19.037650+0000","last_active":"2024-01-16T15:01:19.041145+0000","last_peered":"2024-01-16T15:01:19.041145+0000","last_clean":"2024-01-16T15:01:19.041145+0000","last_became_active":"2024-01-16T15:01:19.036793+0000","last_became_peered":"2024-01-16T15:01:19.036793+0000","last_unstale":"2024-01-16T15:01:19.041145+0000","last_undegraded":"2024-01-16T15:01:19.041145+0000","last_fullsized":"2024-01-16T15:01:19.041145+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:25.010165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015600+0000","last_change":"2024-01-16T15:01:19.041618+0000","last_active":"2024-01-16T15:01:20.015600+0000","last_peered":"2024-01-16T15:01:20.015600+0000","last_clean":"2024-01-16T15:01:20.015600+0000","last_became_active":"2024-01-16T15:01:19.041264+0000","last_became_peered":"2024-01-16T15:01:19.041264+0000","last_unstale":"2024-01-16T15:01:20.015600+0000","last_undegraded":"2024-01-16T15:01:20.015600+0000","last_fullsized":"2024-01-16T15:01:20.015600+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:18.453597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015002+0000","last_change":"2024-01-16T15:01:19.041987+0000","last_active":"2024-01-16T15:01:20.015002+0000","last_peered":"2024-01-16T15:01:20.015002+0000","last_clean":"2024-01-16T15:01:20.015002+0000","last_became_active":"2024-01-16T15:01:19.039018+0000","last_became_peered":"2024-01-16T15:01:19.039018+0000","last_unstale":"2024-01-16T15:01:20.015002+0000","last_undegraded":"2024-01-16T15:01:20.015002+0000","last_fullsized":"2024-01-16T15:01:20.015002+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:29.247514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.143062+0000","last_change":"2024-01-16T15:01:19.043165+0000","last_active":"2024-01-16T15:01:20.143062+0000","last_peered":"2024-01-16T15:01:20.143062+0000","last_clean":"2024-01-16T15:01:20.143062+0000","last_became_active":"2024-01-16T15:01:19.040497+0000","last_became_peered":"2024-01-16T15:01:19.040497+0000","last_unstale":"2024-01-16T15:01:20.143062+0000","last_undegraded":"2024-01-16T15:01:20.143062+0000","last_fullsized":"2024-01-16T15:01:20.143062+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:17.993305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048939+0000","last_change":"2024-01-16T15:01:19.043101+0000","last_active":"2024-01-16T15:01:19.048939+0000","last_peered":"2024-01-16T15:01:19.048939+0000","last_clean":"2024-01-16T15:01:19.048939+0000","last_became_active":"2024-01-16T15:01:19.042761+0000","last_became_peered":"2024-01-16T15:01:19.042761+0000","last_unstale":"2024-01-16T15:01:19.048939+0000","last_undegraded":"2024-01-16T15:01:19.048939+0000","last_fullsized":"2024-01-16T15:01:19.048939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:53.580359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016576+0000","last_change":"2024-01-16T15:01:19.047751+0000","last_active":"2024-01-16T15:01:20.016576+0000","last_peered":"2024-01-16T15:01:20.016576+0000","last_clean":"2024-01-16T15:01:20.016576+0000","last_became_active":"2024-01-16T15:01:19.047275+0000","last_became_peered":"2024-01-16T15:01:19.047275+0000","last_unstale":"2024-01-16T15:01:20.016576+0000","last_undegraded":"2024-01-16T15:01:20.016576+0000","last_fullsized":"2024-01-16T15:01:20.016576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:19:58.174525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018831+0000","last_change":"2024-01-16T15:01:19.042987+0000","last_active":"2024-01-16T15:01:20.018831+0000","last_peered":"2024-01-16T15:01:20.018831+0000","last_clean":"2024-01-16T15:01:20.018831+0000","last_became_active":"2024-01-16T15:01:19.041451+0000","last_became_peered":"2024-01-16T15:01:19.041451+0000","last_unstale":"2024-01-16T15:01:20.018831+0000","last_undegraded":"2024-01-16T15:01:20.018831+0000","last_fullsized":"2024-01-16T15:01:20.018831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:29.247514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048186+0000","last_change":"2024-01-16T15:01:19.045254+0000","last_active":"2024-01-16T15:01:19.048186+0000","last_peered":"2024-01-16T15:01:19.048186+0000","last_clean":"2024-01-16T15:01:19.048186+0000","last_became_active":"2024-01-16T15:01:19.044990+0000","last_became_peered":"2024-01-16T15:01:19.044990+0000","last_unstale":"2024-01-16T15:01:19.048186+0000","last_undegraded":"2024-01-16T15:01:19.048186+0000","last_fullsized":"2024-01-16T15:01:19.048186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:33.128266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.043413+0000","last_change":"2024-01-16T15:01:19.040779+0000","last_active":"2024-01-16T15:01:19.043413+0000","last_peered":"2024-01-16T15:01:19.043413+0000","last_clean":"2024-01-16T15:01:19.043413+0000","last_became_active":"2024-01-16T15:01:19.040486+0000","last_became_peered":"2024-01-16T15:01:19.040486+0000","last_unstale":"2024-01-16T15:01:19.043413+0000","last_undegraded":"2024-01-16T15:01:19.043413+0000","last_fullsized":"2024-01-16T15:01:19.043413+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:39.237725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018205+0000","last_change":"2024-01-16T15:01:19.048431+0000","last_active":"2024-01-16T15:01:20.018205+0000","last_peered":"2024-01-16T15:01:20.018205+0000","last_clean":"2024-01-16T15:01:20.018205+0000","last_became_active":"2024-01-16T15:01:19.048178+0000","last_became_peered":"2024-01-16T15:01:19.048178+0000","last_unstale":"2024-01-16T15:01:20.018205+0000","last_undegraded":"2024-01-16T15:01:20.018205+0000","last_fullsized":"2024-01-16T15:01:20.018205+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:10.016381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016051+0000","last_change":"2024-01-16T15:01:19.039285+0000","last_active":"2024-01-16T15:01:20.016051+0000","last_peered":"2024-01-16T15:01:20.016051+0000","last_clean":"2024-01-16T15:01:20.016051+0000","last_became_active":"2024-01-16T15:01:19.038635+0000","last_became_peered":"2024-01-16T15:01:19.038635+0000","last_unstale":"2024-01-16T15:01:20.016051+0000","last_undegraded":"2024-01-16T15:01:20.016051+0000","last_fullsized":"2024-01-16T15:01:20.016051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:16.439958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016827+0000","last_change":"2024-01-16T15:01:19.050193+0000","last_active":"2024-01-16T15:01:20.016827+0000","last_peered":"2024-01-16T15:01:20.016827+0000","last_clean":"2024-01-16T15:01:20.016827+0000","last_became_active":"2024-01-16T15:01:19.049907+0000","last_became_peered":"2024-01-16T15:01:19.049907+0000","last_unstale":"2024-01-16T15:01:20.016827+0000","last_undegraded":"2024-01-16T15:01:20.016827+0000","last_fullsized":"2024-01-16T15:01:20.016827+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:55.992504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.041813+0000","last_change":"2024-01-16T15:01:19.038959+0000","last_active":"2024-01-16T15:01:19.041813+0000","last_peered":"2024-01-16T15:01:19.041813+0000","last_clean":"2024-01-16T15:01:19.041813+0000","last_became_active":"2024-01-16T15:01:19.038700+0000","last_became_peered":"2024-01-16T15:01:19.038700+0000","last_unstale":"2024-01-16T15:01:19.041813+0000","last_undegraded":"2024-01-16T15:01:19.041813+0000","last_fullsized":"2024-01-16T15:01:19.041813+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:12.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.045793+0000","last_change":"2024-01-16T15:01:19.042419+0000","last_active":"2024-01-16T15:01:19.045793+0000","last_peered":"2024-01-16T15:01:19.045793+0000","last_clean":"2024-01-16T15:01:19.045793+0000","last_became_active":"2024-01-16T15:01:19.042053+0000","last_became_peered":"2024-01-16T15:01:19.042053+0000","last_unstale":"2024-01-16T15:01:19.045793+0000","last_undegraded":"2024-01-16T15:01:19.045793+0000","last_fullsized":"2024-01-16T15:01:19.045793+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:47.135061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048136+0000","last_change":"2024-01-16T15:01:19.044905+0000","last_active":"2024-01-16T15:01:19.048136+0000","last_peered":"2024-01-16T15:01:19.048136+0000","last_clean":"2024-01-16T15:01:19.048136+0000","last_became_active":"2024-01-16T15:01:19.044613+0000","last_became_peered":"2024-01-16T15:01:19.044613+0000","last_unstale":"2024-01-16T15:01:19.048136+0000","last_undegraded":"2024-01-16T15:01:19.048136+0000","last_fullsized":"2024-01-16T15:01:19.048136+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:03:59.796076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142845+0000","last_change":"2024-01-16T15:01:19.029251+0000","last_active":"2024-01-16T15:01:20.142845+0000","last_peered":"2024-01-16T15:01:20.142845+0000","last_clean":"2024-01-16T15:01:20.142845+0000","last_became_active":"2024-01-16T15:01:19.028375+0000","last_became_peered":"2024-01-16T15:01:19.028375+0000","last_unstale":"2024-01-16T15:01:20.142845+0000","last_undegraded":"2024-01-16T15:01:20.142845+0000","last_fullsized":"2024-01-16T15:01:20.142845+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:46.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142315+0000","last_change":"2024-01-16T15:01:19.036613+0000","last_active":"2024-01-16T15:01:20.142315+0000","last_peered":"2024-01-16T15:01:20.142315+0000","last_clean":"2024-01-16T15:01:20.142315+0000","last_became_active":"2024-01-16T15:01:19.029866+0000","last_became_peered":"2024-01-16T15:01:19.029866+0000","last_unstale":"2024-01-16T15:01:20.142315+0000","last_undegraded":"2024-01-16T15:01:20.142315+0000","last_fullsized":"2024-01-16T15:01:20.142315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:27:26.968382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.031933+0000","last_change":"2024-01-16T15:01:19.028495+0000","last_active":"2024-01-16T15:01:19.031933+0000","last_peered":"2024-01-16T15:01:19.031933+0000","last_clean":"2024-01-16T15:01:19.031933+0000","last_became_active":"2024-01-16T15:01:19.028058+0000","last_became_peered":"2024-01-16T15:01:19.028058+0000","last_unstale":"2024-01-16T15:01:19.031933+0000","last_undegraded":"2024-01-16T15:01:19.031933+0000","last_fullsized":"2024-01-16T15:01:19.031933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:39.562787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018025+0000","last_change":"2024-01-16T15:01:19.046995+0000","last_active":"2024-01-16T15:01:20.018025+0000","last_peered":"2024-01-16T15:01:20.018025+0000","last_clean":"2024-01-16T15:01:20.018025+0000","last_became_active":"2024-01-16T15:01:19.045662+0000","last_became_peered":"2024-01-16T15:01:19.045662+0000","last_unstale":"2024-01-16T15:01:20.018025+0000","last_undegraded":"2024-01-16T15:01:20.018025+0000","last_fullsized":"2024-01-16T15:01:20.018025+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:15.099576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048961+0000","last_change":"2024-01-16T15:01:19.043585+0000","last_active":"2024-01-16T15:01:19.048961+0000","last_peered":"2024-01-16T15:01:19.048961+0000","last_clean":"2024-01-16T15:01:19.048961+0000","last_became_active":"2024-01-16T15:01:19.043285+0000","last_became_peered":"2024-01-16T15:01:19.043285+0000","last_unstale":"2024-01-16T15:01:19.048961+0000","last_undegraded":"2024-01-16T15:01:19.048961+0000","last_fullsized":"2024-01-16T15:01:19.048961+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:16.354580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049341+0000","last_change":"2024-01-16T15:01:19.047682+0000","last_active":"2024-01-16T15:01:19.049341+0000","last_peered":"2024-01-16T15:01:19.049341+0000","last_clean":"2024-01-16T15:01:19.049341+0000","last_became_active":"2024-01-16T15:01:19.045795+0000","last_became_peered":"2024-01-16T15:01:19.045795+0000","last_unstale":"2024-01-16T15:01:19.049341+0000","last_undegraded":"2024-01-16T15:01:19.049341+0000","last_fullsized":"2024-01-16T15:01:19.049341+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:03.572984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048988+0000","last_change":"2024-01-16T15:01:19.042670+0000","last_active":"2024-01-16T15:01:19.048988+0000","last_peered":"2024-01-16T15:01:19.048988+0000","last_clean":"2024-01-16T15:01:19.048988+0000","last_became_active":"2024-01-16T15:01:19.042320+0000","last_became_peered":"2024-01-16T15:01:19.042320+0000","last_unstale":"2024-01-16T15:01:19.048988+0000","last_undegraded":"2024-01-16T15:01:19.048988+0000","last_fullsized":"2024-01-16T15:01:19.048988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:57.912755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018820+0000","last_change":"2024-01-16T15:01:19.042269+0000","last_active":"2024-01-16T15:01:20.018820+0000","last_peered":"2024-01-16T15:01:20.018820+0000","last_clean":"2024-01-16T15:01:20.018820+0000","last_became_active":"2024-01-16T15:01:19.041844+0000","last_became_peered":"2024-01-16T15:01:19.041844+0000","last_unstale":"2024-01-16T15:01:20.018820+0000","last_undegraded":"2024-01-16T15:01:20.018820+0000","last_fullsized":"2024-01-16T15:01:20.018820+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:39.180167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142406+0000","last_change":"2024-01-16T15:01:19.038023+0000","last_active":"2024-01-16T15:01:20.142406+0000","last_peered":"2024-01-16T15:01:20.142406+0000","last_clean":"2024-01-16T15:01:20.142406+0000","last_became_active":"2024-01-16T15:01:19.036102+0000","last_became_peered":"2024-01-16T15:01:19.036102+0000","last_unstale":"2024-01-16T15:01:20.142406+0000","last_undegraded":"2024-01-16T15:01:20.142406+0000","last_fullsized":"2024-01-16T15:01:20.142406+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:03.572984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015752+0000","last_change":"2024-01-16T15:01:19.042153+0000","last_active":"2024-01-16T15:01:20.015752+0000","last_peered":"2024-01-16T15:01:20.015752+0000","last_clean":"2024-01-16T15:01:20.015752+0000","last_became_active":"2024-01-16T15:01:19.041295+0000","last_became_peered":"2024-01-16T15:01:19.041295+0000","last_unstale":"2024-01-16T15:01:20.015752+0000","last_undegraded":"2024-01-16T15:01:20.015752+0000","last_fullsized":"2024-01-16T15:01:20.015752+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:48:24.148127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754183+0000","last_change":"2024-01-16T15:01:19.052938+0000","last_active":"2024-01-16T15:01:20.754183+0000","last_peered":"2024-01-16T15:01:20.754183+0000","last_clean":"2024-01-16T15:01:20.754183+0000","last_became_active":"2024-01-16T15:01:19.050222+0000","last_became_peered":"2024-01-16T15:01:19.050222+0000","last_unstale":"2024-01-16T15:01:20.754183+0000","last_undegraded":"2024-01-16T15:01:20.754183+0000","last_fullsized":"2024-01-16T15:01:20.754183+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:56.206938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.014995+0000","last_change":"2024-01-16T15:01:19.035534+0000","last_active":"2024-01-16T15:01:20.014995+0000","last_peered":"2024-01-16T15:01:20.014995+0000","last_clean":"2024-01-16T15:01:20.014995+0000","last_became_active":"2024-01-16T15:01:19.035154+0000","last_became_peered":"2024-01-16T15:01:19.035154+0000","last_unstale":"2024-01-16T15:01:20.014995+0000","last_undegraded":"2024-01-16T15:01:20.014995+0000","last_fullsized":"2024-01-16T15:01:20.014995+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:55.501685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.038593+0000","last_change":"2024-01-16T15:01:19.035154+0000","last_active":"2024-01-16T15:01:19.038593+0000","last_peered":"2024-01-16T15:01:19.038593+0000","last_clean":"2024-01-16T15:01:19.038593+0000","last_became_active":"2024-01-16T15:01:19.034725+0000","last_became_peered":"2024-01-16T15:01:19.034725+0000","last_unstale":"2024-01-16T15:01:19.038593+0000","last_undegraded":"2024-01-16T15:01:19.038593+0000","last_fullsized":"2024-01-16T15:01:19.038593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:57.912755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754811+0000","last_change":"2024-01-16T15:01:18.491800+0000","last_active":"2024-01-16T15:01:20.754811+0000","last_peered":"2024-01-16T15:01:20.754811+0000","last_clean":"2024-01-16T15:01:20.754811+0000","last_became_active":"2024-01-16T15:01:18.491505+0000","last_became_peered":"2024-01-16T15:01:18.491505+0000","last_unstale":"2024-01-16T15:01:20.754811+0000","last_undegraded":"2024-01-16T15:01:20.754811+0000","last_fullsized":"2024-01-16T15:01:20.754811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:33.163437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754689+0000","last_change":"2024-01-16T15:01:19.052360+0000","last_active":"2024-01-16T15:01:20.754689+0000","last_peered":"2024-01-16T15:01:20.754689+0000","last_clean":"2024-01-16T15:01:20.754689+0000","last_became_active":"2024-01-16T15:01:19.052048+0000","last_became_peered":"2024-01-16T15:01:19.052048+0000","last_unstale":"2024-01-16T15:01:20.754689+0000","last_undegraded":"2024-01-16T15:01:20.754689+0000","last_fullsized":"2024-01-16T15:01:20.754689+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:12.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.490224+0000","last_change":"2024-01-16T15:01:18.487981+0000","last_active":"2024-01-16T15:01:19.490224+0000","last_peered":"2024-01-16T15:01:19.490224+0000","last_clean":"2024-01-16T15:01:19.490224+0000","last_became_active":"2024-01-16T15:01:18.487348+0000","last_became_peered":"2024-01-16T15:01:18.487348+0000","last_unstale":"2024-01-16T15:01:19.490224+0000","last_undegraded":"2024-01-16T15:01:19.490224+0000","last_fullsized":"2024-01-16T15:01:19.490224+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:58.120947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.143369+0000","last_change":"2024-01-16T15:01:19.037180+0000","last_active":"2024-01-16T15:01:20.143369+0000","last_peered":"2024-01-16T15:01:20.143369+0000","last_clean":"2024-01-16T15:01:20.143369+0000","last_became_active":"2024-01-16T15:01:19.036289+0000","last_became_peered":"2024-01-16T15:01:19.036289+0000","last_unstale":"2024-01-16T15:01:20.143369+0000","last_undegraded":"2024-01-16T15:01:20.143369+0000","last_fullsized":"2024-01-16T15:01:20.143369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:05.861683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488139+0000","last_change":"2024-01-16T15:01:18.469397+0000","last_active":"2024-01-16T15:01:19.488139+0000","last_peered":"2024-01-16T15:01:19.488139+0000","last_clean":"2024-01-16T15:01:19.488139+0000","last_became_active":"2024-01-16T15:01:18.468952+0000","last_became_peered":"2024-01-16T15:01:18.468952+0000","last_unstale":"2024-01-16T15:01:19.488139+0000","last_undegraded":"2024-01-16T15:01:19.488139+0000","last_fullsized":"2024-01-16T15:01:19.488139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:04.847207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.046325+0000","last_change":"2024-01-16T15:01:19.039749+0000","last_active":"2024-01-16T15:01:19.046325+0000","last_peered":"2024-01-16T15:01:19.046325+0000","last_clean":"2024-01-16T15:01:19.046325+0000","last_became_active":"2024-01-16T15:01:19.039439+0000","last_became_peered":"2024-01-16T15:01:19.039439+0000","last_unstale":"2024-01-16T15:01:19.046325+0000","last_undegraded":"2024-01-16T15:01:19.046325+0000","last_fullsized":"2024-01-16T15:01:19.046325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:59.135423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.487032+0000","last_change":"2024-01-16T15:01:18.477058+0000","last_active":"2024-01-16T15:01:19.487032+0000","last_peered":"2024-01-16T15:01:19.487032+0000","last_clean":"2024-01-16T15:01:19.487032+0000","last_became_active":"2024-01-16T15:01:18.476800+0000","last_became_peered":"2024-01-16T15:01:18.476800+0000","last_unstale":"2024-01-16T15:01:19.487032+0000","last_undegraded":"2024-01-16T15:01:19.487032+0000","last_fullsized":"2024-01-16T15:01:19.487032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:27.299481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016168+0000","last_change":"2024-01-16T15:01:19.038648+0000","last_active":"2024-01-16T15:01:20.016168+0000","last_peered":"2024-01-16T15:01:20.016168+0000","last_clean":"2024-01-16T15:01:20.016168+0000","last_became_active":"2024-01-16T15:01:19.038226+0000","last_became_peered":"2024-01-16T15:01:19.038226+0000","last_unstale":"2024-01-16T15:01:20.016168+0000","last_undegraded":"2024-01-16T15:01:20.016168+0000","last_fullsized":"2024-01-16T15:01:20.016168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:06.302962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142664+0000","last_change":"2024-01-16T15:01:18.475512+0000","last_active":"2024-01-16T15:01:20.142664+0000","last_peered":"2024-01-16T15:01:20.142664+0000","last_clean":"2024-01-16T15:01:20.142664+0000","last_became_active":"2024-01-16T15:01:18.475241+0000","last_became_peered":"2024-01-16T15:01:18.475241+0000","last_unstale":"2024-01-16T15:01:20.142664+0000","last_undegraded":"2024-01-16T15:01:20.142664+0000","last_fullsized":"2024-01-16T15:01:20.142664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:33.163437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018509+0000","last_change":"2024-01-16T15:01:19.040981+0000","last_active":"2024-01-16T15:01:20.018509+0000","last_peered":"2024-01-16T15:01:20.018509+0000","last_clean":"2024-01-16T15:01:20.018509+0000","last_became_active":"2024-01-16T15:01:19.040698+0000","last_became_peered":"2024-01-16T15:01:19.040698+0000","last_unstale":"2024-01-16T15:01:20.018509+0000","last_undegraded":"2024-01-16T15:01:20.018509+0000","last_fullsized":"2024-01-16T15:01:20.018509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:55.501685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142262+0000","last_change":"2024-01-16T15:01:18.482628+0000","last_active":"2024-01-16T15:01:20.142262+0000","last_peered":"2024-01-16T15:01:20.142262+0000","last_clean":"2024-01-16T15:01:20.142262+0000","last_became_active":"2024-01-16T15:01:18.481456+0000","last_became_peered":"2024-01-16T15:01:18.481456+0000","last_unstale":"2024-01-16T15:01:20.142262+0000","last_undegraded":"2024-01-16T15:01:20.142262+0000","last_fullsized":"2024-01-16T15:01:20.142262+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:35.090050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142067+0000","last_change":"2024-01-16T15:01:19.044957+0000","last_active":"2024-01-16T15:01:20.142067+0000","last_peered":"2024-01-16T15:01:20.142067+0000","last_clean":"2024-01-16T15:01:20.142067+0000","last_became_active":"2024-01-16T15:01:19.044512+0000","last_became_peered":"2024-01-16T15:01:19.044512+0000","last_unstale":"2024-01-16T15:01:20.142067+0000","last_undegraded":"2024-01-16T15:01:20.142067+0000","last_fullsized":"2024-01-16T15:01:20.142067+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:12.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.029906+0000","last_change":"2024-01-16T15:01:18.480799+0000","last_active":"2024-01-16T15:01:19.029906+0000","last_peered":"2024-01-16T15:01:19.029906+0000","last_clean":"2024-01-16T15:01:19.029906+0000","last_became_active":"2024-01-16T15:01:18.478757+0000","last_became_peered":"2024-01-16T15:01:18.478757+0000","last_unstale":"2024-01-16T15:01:19.029906+0000","last_undegraded":"2024-01-16T15:01:19.029906+0000","last_fullsized":"2024-01-16T15:01:19.029906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:39.547663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049250+0000","last_change":"2024-01-16T15:01:19.044788+0000","last_active":"2024-01-16T15:01:19.049250+0000","last_peered":"2024-01-16T15:01:19.049250+0000","last_clean":"2024-01-16T15:01:19.049250+0000","last_became_active":"2024-01-16T15:01:19.044455+0000","last_became_peered":"2024-01-16T15:01:19.044455+0000","last_unstale":"2024-01-16T15:01:19.049250+0000","last_undegraded":"2024-01-16T15:01:19.049250+0000","last_fullsized":"2024-01-16T15:01:19.049250+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:08.071343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.141885+0000","last_change":"2024-01-16T15:01:18.485725+0000","last_active":"2024-01-16T15:01:20.141885+0000","last_peered":"2024-01-16T15:01:20.141885+0000","last_clean":"2024-01-16T15:01:20.141885+0000","last_became_active":"2024-01-16T15:01:18.485431+0000","last_became_peered":"2024-01-16T15:01:18.485431+0000","last_unstale":"2024-01-16T15:01:20.141885+0000","last_undegraded":"2024-01-16T15:01:20.141885+0000","last_fullsized":"2024-01-16T15:01:20.141885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:01.496327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019295+0000","last_change":"2024-01-16T15:01:19.040911+0000","last_active":"2024-01-16T15:01:20.019295+0000","last_peered":"2024-01-16T15:01:20.019295+0000","last_clean":"2024-01-16T15:01:20.019295+0000","last_became_active":"2024-01-16T15:01:19.039966+0000","last_became_peered":"2024-01-16T15:01:19.039966+0000","last_unstale":"2024-01-16T15:01:20.019295+0000","last_undegraded":"2024-01-16T15:01:20.019295+0000","last_fullsized":"2024-01-16T15:01:20.019295+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:07.512715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.032902+0000","last_change":"2024-01-16T15:01:18.493177+0000","last_active":"2024-01-16T15:01:19.032902+0000","last_peered":"2024-01-16T15:01:19.032902+0000","last_clean":"2024-01-16T15:01:19.032902+0000","last_became_active":"2024-01-16T15:01:18.492803+0000","last_became_peered":"2024-01-16T15:01:18.492803+0000","last_unstale":"2024-01-16T15:01:19.032902+0000","last_undegraded":"2024-01-16T15:01:19.032902+0000","last_fullsized":"2024-01-16T15:01:19.032902+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:32:40.055476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.042591+0000","last_change":"2024-01-16T15:01:19.039580+0000","last_active":"2024-01-16T15:01:19.042591+0000","last_peered":"2024-01-16T15:01:19.042591+0000","last_clean":"2024-01-16T15:01:19.042591+0000","last_became_active":"2024-01-16T15:01:19.039262+0000","last_became_peered":"2024-01-16T15:01:19.039262+0000","last_unstale":"2024-01-16T15:01:19.042591+0000","last_undegraded":"2024-01-16T15:01:19.042591+0000","last_fullsized":"2024-01-16T15:01:19.042591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.533005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.014376+0000","last_change":"2024-01-16T15:01:18.473665+0000","last_active":"2024-01-16T15:01:19.014376+0000","last_peered":"2024-01-16T15:01:19.014376+0000","last_clean":"2024-01-16T15:01:19.014376+0000","last_became_active":"2024-01-16T15:01:18.473233+0000","last_became_peered":"2024-01-16T15:01:18.473233+0000","last_unstale":"2024-01-16T15:01:19.014376+0000","last_undegraded":"2024-01-16T15:01:19.014376+0000","last_fullsized":"2024-01-16T15:01:19.014376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:52.680143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.043495+0000","last_change":"2024-01-16T15:01:19.041472+0000","last_active":"2024-01-16T15:01:19.043495+0000","last_peered":"2024-01-16T15:01:19.043495+0000","last_clean":"2024-01-16T15:01:19.043495+0000","last_became_active":"2024-01-16T15:01:19.041200+0000","last_became_peered":"2024-01-16T15:01:19.041200+0000","last_unstale":"2024-01-16T15:01:19.043495+0000","last_undegraded":"2024-01-16T15:01:19.043495+0000","last_fullsized":"2024-01-16T15:01:19.043495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:22.022593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019124+0000","last_change":"2024-01-16T15:01:18.492933+0000","last_active":"2024-01-16T15:01:20.019124+0000","last_peered":"2024-01-16T15:01:20.019124+0000","last_clean":"2024-01-16T15:01:20.019124+0000","last_became_active":"2024-01-16T15:01:18.492514+0000","last_became_peered":"2024-01-16T15:01:18.492514+0000","last_unstale":"2024-01-16T15:01:20.019124+0000","last_undegraded":"2024-01-16T15:01:20.019124+0000","last_fullsized":"2024-01-16T15:01:20.019124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:56.016453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018832+0000","last_change":"2024-01-16T15:01:19.048370+0000","last_active":"2024-01-16T15:01:20.018832+0000","last_peered":"2024-01-16T15:01:20.018832+0000","last_clean":"2024-01-16T15:01:20.018832+0000","last_became_active":"2024-01-16T15:01:19.047838+0000","last_became_peered":"2024-01-16T15:01:19.047838+0000","last_unstale":"2024-01-16T15:01:20.018832+0000","last_undegraded":"2024-01-16T15:01:20.018832+0000","last_fullsized":"2024-01-16T15:01:20.018832+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:48.466587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.496496+0000","last_change":"2024-01-16T15:01:18.473272+0000","last_active":"2024-01-16T15:01:19.496496+0000","last_peered":"2024-01-16T15:01:19.496496+0000","last_clean":"2024-01-16T15:01:19.496496+0000","last_became_active":"2024-01-16T15:01:18.472513+0000","last_became_peered":"2024-01-16T15:01:18.472513+0000","last_unstale":"2024-01-16T15:01:19.496496+0000","last_undegraded":"2024-01-16T15:01:19.496496+0000","last_fullsized":"2024-01-16T15:01:19.496496+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.784526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049464+0000","last_change":"2024-01-16T15:01:19.044492+0000","last_active":"2024-01-16T15:01:19.049464+0000","last_peered":"2024-01-16T15:01:19.049464+0000","last_clean":"2024-01-16T15:01:19.049464+0000","last_became_active":"2024-01-16T15:01:19.044233+0000","last_became_peered":"2024-01-16T15:01:19.044233+0000","last_unstale":"2024-01-16T15:01:19.049464+0000","last_undegraded":"2024-01-16T15:01:19.049464+0000","last_fullsized":"2024-01-16T15:01:19.049464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:58.091122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142917+0000","last_change":"2024-01-16T15:01:18.489034+0000","last_active":"2024-01-16T15:01:20.142917+0000","last_peered":"2024-01-16T15:01:20.142917+0000","last_clean":"2024-01-16T15:01:20.142917+0000","last_became_active":"2024-01-16T15:01:18.488759+0000","last_became_peered":"2024-01-16T15:01:18.488759+0000","last_unstale":"2024-01-16T15:01:20.142917+0000","last_undegraded":"2024-01-16T15:01:20.142917+0000","last_fullsized":"2024-01-16T15:01:20.142917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:57.567634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.037856+0000","last_change":"2024-01-16T15:01:19.033187+0000","last_active":"2024-01-16T15:01:19.037856+0000","last_peered":"2024-01-16T15:01:19.037856+0000","last_clean":"2024-01-16T15:01:19.037856+0000","last_became_active":"2024-01-16T15:01:19.032894+0000","last_became_peered":"2024-01-16T15:01:19.032894+0000","last_unstale":"2024-01-16T15:01:19.037856+0000","last_undegraded":"2024-01-16T15:01:19.037856+0000","last_fullsized":"2024-01-16T15:01:19.037856+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:56.206938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.030523+0000","last_change":"2024-01-16T15:01:18.489543+0000","last_active":"2024-01-16T15:01:19.030523+0000","last_peered":"2024-01-16T15:01:19.030523+0000","last_clean":"2024-01-16T15:01:19.030523+0000","last_became_active":"2024-01-16T15:01:18.489158+0000","last_became_peered":"2024-01-16T15:01:18.489158+0000","last_unstale":"2024-01-16T15:01:19.030523+0000","last_undegraded":"2024-01-16T15:01:19.030523+0000","last_fullsized":"2024-01-16T15:01:19.030523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:14.011515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015385+0000","last_change":"2024-01-16T15:01:19.042003+0000","last_active":"2024-01-16T15:01:20.015385+0000","last_peered":"2024-01-16T15:01:20.015385+0000","last_clean":"2024-01-16T15:01:20.015385+0000","last_became_active":"2024-01-16T15:01:19.041734+0000","last_became_peered":"2024-01-16T15:01:19.041734+0000","last_unstale":"2024-01-16T15:01:20.015385+0000","last_undegraded":"2024-01-16T15:01:20.015385+0000","last_fullsized":"2024-01-16T15:01:20.015385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:48.466587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.032333+0000","last_change":"2024-01-16T15:01:18.485309+0000","last_active":"2024-01-16T15:01:19.032333+0000","last_peered":"2024-01-16T15:01:19.032333+0000","last_clean":"2024-01-16T15:01:19.032333+0000","last_became_active":"2024-01-16T15:01:18.485020+0000","last_became_peered":"2024-01-16T15:01:18.485020+0000","last_unstale":"2024-01-16T15:01:19.032333+0000","last_undegraded":"2024-01-16T15:01:19.032333+0000","last_fullsized":"2024-01-16T15:01:19.032333+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.832883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754348+0000","last_change":"2024-01-16T15:01:19.052740+0000","last_active":"2024-01-16T15:01:20.754348+0000","last_peered":"2024-01-16T15:01:20.754348+0000","last_clean":"2024-01-16T15:01:20.754348+0000","last_became_active":"2024-01-16T15:01:19.049863+0000","last_became_peered":"2024-01-16T15:01:19.049863+0000","last_unstale":"2024-01-16T15:01:20.754348+0000","last_undegraded":"2024-01-16T15:01:20.754348+0000","last_fullsized":"2024-01-16T15:01:20.754348+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:47.135061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488077+0000","last_change":"2024-01-16T15:01:18.469296+0000","last_active":"2024-01-16T15:01:19.488077+0000","last_peered":"2024-01-16T15:01:19.488077+0000","last_clean":"2024-01-16T15:01:19.488077+0000","last_became_active":"2024-01-16T15:01:18.469041+0000","last_became_peered":"2024-01-16T15:01:18.469041+0000","last_unstale":"2024-01-16T15:01:19.488077+0000","last_undegraded":"2024-01-16T15:01:19.488077+0000","last_fullsized":"2024-01-16T15:01:19.488077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:45.704627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142676+0000","last_change":"2024-01-16T15:01:19.042733+0000","last_active":"2024-01-16T15:01:20.142676+0000","last_peered":"2024-01-16T15:01:20.142676+0000","last_clean":"2024-01-16T15:01:20.142676+0000","last_became_active":"2024-01-16T15:01:19.040078+0000","last_became_peered":"2024-01-16T15:01:19.040078+0000","last_unstale":"2024-01-16T15:01:20.142676+0000","last_undegraded":"2024-01-16T15:01:20.142676+0000","last_fullsized":"2024-01-16T15:01:20.142676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:40.562139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488582+0000","last_change":"2024-01-16T15:01:18.477106+0000","last_active":"2024-01-16T15:01:19.488582+0000","last_peered":"2024-01-16T15:01:19.488582+0000","last_clean":"2024-01-16T15:01:19.488582+0000","last_became_active":"2024-01-16T15:01:18.476072+0000","last_became_peered":"2024-01-16T15:01:18.476072+0000","last_unstale":"2024-01-16T15:01:19.488582+0000","last_undegraded":"2024-01-16T15:01:19.488582+0000","last_fullsized":"2024-01-16T15:01:19.488582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:37.460893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019257+0000","last_change":"2024-01-16T15:01:19.045922+0000","last_active":"2024-01-16T15:01:20.019257+0000","last_peered":"2024-01-16T15:01:20.019257+0000","last_clean":"2024-01-16T15:01:20.019257+0000","last_became_active":"2024-01-16T15:01:19.045649+0000","last_became_peered":"2024-01-16T15:01:19.045649+0000","last_unstale":"2024-01-16T15:01:20.019257+0000","last_undegraded":"2024-01-16T15:01:20.019257+0000","last_fullsized":"2024-01-16T15:01:20.019257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:48:24.148127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142554+0000","last_change":"2024-01-16T15:01:18.491363+0000","last_active":"2024-01-16T15:01:20.142554+0000","last_peered":"2024-01-16T15:01:20.142554+0000","last_clean":"2024-01-16T15:01:20.142554+0000","last_became_active":"2024-01-16T15:01:18.490548+0000","last_became_peered":"2024-01-16T15:01:18.490548+0000","last_unstale":"2024-01-16T15:01:20.142554+0000","last_undegraded":"2024-01-16T15:01:20.142554+0000","last_fullsized":"2024-01-16T15:01:20.142554+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:46.704589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049169+0000","last_change":"2024-01-16T15:01:19.044227+0000","last_active":"2024-01-16T15:01:19.049169+0000","last_peered":"2024-01-16T15:01:19.049169+0000","last_clean":"2024-01-16T15:01:19.049169+0000","last_became_active":"2024-01-16T15:01:19.041548+0000","last_became_peered":"2024-01-16T15:01:19.041548+0000","last_unstale":"2024-01-16T15:01:19.049169+0000","last_undegraded":"2024-01-16T15:01:19.049169+0000","last_fullsized":"2024-01-16T15:01:19.049169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:53.694619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142970+0000","last_change":"2024-01-16T15:01:18.484649+0000","last_active":"2024-01-16T15:01:20.142970+0000","last_peered":"2024-01-16T15:01:20.142970+0000","last_clean":"2024-01-16T15:01:20.142970+0000","last_became_active":"2024-01-16T15:01:18.483981+0000","last_became_peered":"2024-01-16T15:01:18.483981+0000","last_unstale":"2024-01-16T15:01:20.142970+0000","last_undegraded":"2024-01-16T15:01:20.142970+0000","last_fullsized":"2024-01-16T15:01:20.142970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:23.995689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048200+0000","last_change":"2024-01-16T15:01:19.045159+0000","last_active":"2024-01-16T15:01:19.048200+0000","last_peered":"2024-01-16T15:01:19.048200+0000","last_clean":"2024-01-16T15:01:19.048200+0000","last_became_active":"2024-01-16T15:01:19.043354+0000","last_became_peered":"2024-01-16T15:01:19.043354+0000","last_unstale":"2024-01-16T15:01:19.048200+0000","last_undegraded":"2024-01-16T15:01:19.048200+0000","last_fullsized":"2024-01-16T15:01:19.048200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:35.130556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.143177+0000","last_change":"2024-01-16T15:01:18.476597+0000","last_active":"2024-01-16T15:01:20.143177+0000","last_peered":"2024-01-16T15:01:20.143177+0000","last_clean":"2024-01-16T15:01:20.143177+0000","last_became_active":"2024-01-16T15:01:18.476310+0000","last_became_peered":"2024-01-16T15:01:18.476310+0000","last_unstale":"2024-01-16T15:01:20.143177+0000","last_undegraded":"2024-01-16T15:01:20.143177+0000","last_fullsized":"2024-01-16T15:01:20.143177+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:38.223249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049270+0000","last_change":"2024-01-16T15:01:19.047471+0000","last_active":"2024-01-16T15:01:19.049270+0000","last_peered":"2024-01-16T15:01:19.049270+0000","last_clean":"2024-01-16T15:01:19.049270+0000","last_became_active":"2024-01-16T15:01:19.047195+0000","last_became_peered":"2024-01-16T15:01:19.047195+0000","last_unstale":"2024-01-16T15:01:19.049270+0000","last_undegraded":"2024-01-16T15:01:19.049270+0000","last_fullsized":"2024-01-16T15:01:19.049270+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:32.591220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488632+0000","last_change":"2024-01-16T15:01:18.479262+0000","last_active":"2024-01-16T15:01:19.488632+0000","last_peered":"2024-01-16T15:01:19.488632+0000","last_clean":"2024-01-16T15:01:19.488632+0000","last_became_active":"2024-01-16T15:01:18.478965+0000","last_became_peered":"2024-01-16T15:01:18.478965+0000","last_unstale":"2024-01-16T15:01:19.488632+0000","last_undegraded":"2024-01-16T15:01:19.488632+0000","last_fullsized":"2024-01-16T15:01:19.488632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:56.981351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.046942+0000","last_change":"2024-01-16T15:01:19.039864+0000","last_active":"2024-01-16T15:01:19.046942+0000","last_peered":"2024-01-16T15:01:19.046942+0000","last_clean":"2024-01-16T15:01:19.046942+0000","last_became_active":"2024-01-16T15:01:19.036729+0000","last_became_peered":"2024-01-16T15:01:19.036729+0000","last_unstale":"2024-01-16T15:01:19.046942+0000","last_undegraded":"2024-01-16T15:01:19.046942+0000","last_fullsized":"2024-01-16T15:01:19.046942+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:35.299330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754379+0000","last_change":"2024-01-16T15:01:18.490027+0000","last_active":"2024-01-16T15:01:20.754379+0000","last_peered":"2024-01-16T15:01:20.754379+0000","last_clean":"2024-01-16T15:01:20.754379+0000","last_became_active":"2024-01-16T15:01:18.489615+0000","last_became_peered":"2024-01-16T15:01:18.489615+0000","last_unstale":"2024-01-16T15:01:20.754379+0000","last_undegraded":"2024-01-16T15:01:20.754379+0000","last_fullsized":"2024-01-16T15:01:20.754379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:35.090050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.047430+0000","last_change":"2024-01-16T15:01:19.038096+0000","last_active":"2024-01-16T15:01:19.047430+0000","last_peered":"2024-01-16T15:01:19.047430+0000","last_clean":"2024-01-16T15:01:19.047430+0000","last_became_active":"2024-01-16T15:01:19.036902+0000","last_became_peered":"2024-01-16T15:01:19.036902+0000","last_unstale":"2024-01-16T15:01:19.047430+0000","last_undegraded":"2024-01-16T15:01:19.047430+0000","last_fullsized":"2024-01-16T15:01:19.047430+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:16.914038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142037+0000","last_change":"2024-01-16T15:01:18.480620+0000","last_active":"2024-01-16T15:01:20.142037+0000","last_peered":"2024-01-16T15:01:20.142037+0000","last_clean":"2024-01-16T15:01:20.142037+0000","last_became_active":"2024-01-16T15:01:18.480281+0000","last_became_peered":"2024-01-16T15:01:18.480281+0000","last_unstale":"2024-01-16T15:01:20.142037+0000","last_undegraded":"2024-01-16T15:01:20.142037+0000","last_fullsized":"2024-01-16T15:01:20.142037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:46.120585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015125+0000","last_change":"2024-01-16T15:01:19.051501+0000","last_active":"2024-01-16T15:01:20.015125+0000","last_peered":"2024-01-16T15:01:20.015125+0000","last_clean":"2024-01-16T15:01:20.015125+0000","last_became_active":"2024-01-16T15:01:19.051087+0000","last_became_peered":"2024-01-16T15:01:19.051087+0000","last_unstale":"2024-01-16T15:01:20.015125+0000","last_undegraded":"2024-01-16T15:01:20.015125+0000","last_fullsized":"2024-01-16T15:01:20.015125+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:39.180167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.017910+0000","last_change":"2024-01-16T15:01:18.488108+0000","last_active":"2024-01-16T15:01:20.017910+0000","last_peered":"2024-01-16T15:01:20.017910+0000","last_clean":"2024-01-16T15:01:20.017910+0000","last_became_active":"2024-01-16T15:01:18.487832+0000","last_became_peered":"2024-01-16T15:01:18.487832+0000","last_unstale":"2024-01-16T15:01:20.017910+0000","last_undegraded":"2024-01-16T15:01:20.017910+0000","last_fullsized":"2024-01-16T15:01:20.017910+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:52.145878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048265+0000","last_change":"2024-01-16T15:01:19.046514+0000","last_active":"2024-01-16T15:01:19.048265+0000","last_peered":"2024-01-16T15:01:19.048265+0000","last_clean":"2024-01-16T15:01:19.048265+0000","last_became_active":"2024-01-16T15:01:19.046178+0000","last_became_peered":"2024-01-16T15:01:19.046178+0000","last_unstale":"2024-01-16T15:01:19.048265+0000","last_undegraded":"2024-01-16T15:01:19.048265+0000","last_fullsized":"2024-01-16T15:01:19.048265+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:47.607088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488902+0000","last_change":"2024-01-16T15:01:18.485980+0000","last_active":"2024-01-16T15:01:19.488902+0000","last_peered":"2024-01-16T15:01:19.488902+0000","last_clean":"2024-01-16T15:01:19.488902+0000","last_became_active":"2024-01-16T15:01:18.485628+0000","last_became_peered":"2024-01-16T15:01:18.485628+0000","last_unstale":"2024-01-16T15:01:19.488902+0000","last_undegraded":"2024-01-16T15:01:19.488902+0000","last_fullsized":"2024-01-16T15:01:19.488902+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:33.918678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019349+0000","last_change":"2024-01-16T15:01:19.043592+0000","last_active":"2024-01-16T15:01:20.019349+0000","last_peered":"2024-01-16T15:01:20.019349+0000","last_clean":"2024-01-16T15:01:20.019349+0000","last_became_active":"2024-01-16T15:01:19.043134+0000","last_became_peered":"2024-01-16T15:01:19.043134+0000","last_unstale":"2024-01-16T15:01:20.019349+0000","last_undegraded":"2024-01-16T15:01:20.019349+0000","last_fullsized":"2024-01-16T15:01:20.019349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:16.439958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.014771+0000","last_change":"2024-01-16T15:01:18.473486+0000","last_active":"2024-01-16T15:01:19.014771+0000","last_peered":"2024-01-16T15:01:19.014771+0000","last_clean":"2024-01-16T15:01:19.014771+0000","last_became_active":"2024-01-16T15:01:18.472875+0000","last_became_peered":"2024-01-16T15:01:18.472875+0000","last_unstale":"2024-01-16T15:01:19.014771+0000","last_undegraded":"2024-01-16T15:01:19.014771+0000","last_fullsized":"2024-01-16T15:01:19.014771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:57.076646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.042435+0000","last_change":"2024-01-16T15:01:19.038031+0000","last_active":"2024-01-16T15:01:19.042435+0000","last_peered":"2024-01-16T15:01:19.042435+0000","last_clean":"2024-01-16T15:01:19.042435+0000","last_became_active":"2024-01-16T15:01:19.037324+0000","last_became_peered":"2024-01-16T15:01:19.037324+0000","last_unstale":"2024-01-16T15:01:19.042435+0000","last_undegraded":"2024-01-16T15:01:19.042435+0000","last_fullsized":"2024-01-16T15:01:19.042435+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:08.957102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.031616+0000","last_change":"2024-01-16T15:01:18.480306+0000","last_active":"2024-01-16T15:01:19.031616+0000","last_peered":"2024-01-16T15:01:19.031616+0000","last_clean":"2024-01-16T15:01:19.031616+0000","last_became_active":"2024-01-16T15:01:18.479817+0000","last_became_peered":"2024-01-16T15:01:18.479817+0000","last_unstale":"2024-01-16T15:01:19.031616+0000","last_undegraded":"2024-01-16T15:01:19.031616+0000","last_fullsized":"2024-01-16T15:01:19.031616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:47.527171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754176+0000","last_change":"2024-01-16T15:01:19.046107+0000","last_active":"2024-01-16T15:01:20.754176+0000","last_peered":"2024-01-16T15:01:20.754176+0000","last_clean":"2024-01-16T15:01:20.754176+0000","last_became_active":"2024-01-16T15:01:19.045830+0000","last_became_peered":"2024-01-16T15:01:19.045830+0000","last_unstale":"2024-01-16T15:01:20.754176+0000","last_undegraded":"2024-01-16T15:01:20.754176+0000","last_fullsized":"2024-01-16T15:01:20.754176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:25.010165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.013592+0000","last_change":"2024-01-16T15:01:18.482133+0000","last_active":"2024-01-16T15:01:19.013592+0000","last_peered":"2024-01-16T15:01:19.013592+0000","last_clean":"2024-01-16T15:01:19.013592+0000","last_became_active":"2024-01-16T15:01:18.481693+0000","last_became_peered":"2024-01-16T15:01:18.481693+0000","last_unstale":"2024-01-16T15:01:19.013592+0000","last_undegraded":"2024-01-16T15:01:19.013592+0000","last_fullsized":"2024-01-16T15:01:19.013592+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:07.056868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048929+0000","last_change":"2024-01-16T15:01:19.047096+0000","last_active":"2024-01-16T15:01:19.048929+0000","last_peered":"2024-01-16T15:01:19.048929+0000","last_clean":"2024-01-16T15:01:19.048929+0000","last_became_active":"2024-01-16T15:01:19.046832+0000","last_became_peered":"2024-01-16T15:01:19.046832+0000","last_unstale":"2024-01-16T15:01:19.048929+0000","last_undegraded":"2024-01-16T15:01:19.048929+0000","last_fullsized":"2024-01-16T15:01:19.048929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:09:21.690178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.030673+0000","last_change":"2024-01-16T15:01:18.485941+0000","last_active":"2024-01-16T15:01:19.030673+0000","last_peered":"2024-01-16T15:01:19.030673+0000","last_clean":"2024-01-16T15:01:19.030673+0000","last_became_active":"2024-01-16T15:01:18.485618+0000","last_became_peered":"2024-01-16T15:01:18.485618+0000","last_unstale":"2024-01-16T15:01:19.030673+0000","last_undegraded":"2024-01-16T15:01:19.030673+0000","last_fullsized":"2024-01-16T15:01:19.030673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:24.383674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.042717+0000","last_change":"2024-01-16T15:01:19.038114+0000","last_active":"2024-01-16T15:01:19.042717+0000","last_peered":"2024-01-16T15:01:19.042717+0000","last_clean":"2024-01-16T15:01:19.042717+0000","last_became_active":"2024-01-16T15:01:19.036337+0000","last_became_peered":"2024-01-16T15:01:19.036337+0000","last_unstale":"2024-01-16T15:01:19.042717+0000","last_undegraded":"2024-01-16T15:01:19.042717+0000","last_fullsized":"2024-01-16T15:01:19.042717+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:34.177913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142300+0000","last_change":"2024-01-16T15:01:18.484537+0000","last_active":"2024-01-16T15:01:20.142300+0000","last_peered":"2024-01-16T15:01:20.142300+0000","last_clean":"2024-01-16T15:01:20.142300+0000","last_became_active":"2024-01-16T15:01:18.484243+0000","last_became_peered":"2024-01-16T15:01:18.484243+0000","last_unstale":"2024-01-16T15:01:20.142300+0000","last_undegraded":"2024-01-16T15:01:20.142300+0000","last_fullsized":"2024-01-16T15:01:20.142300+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:55.192462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142501+0000","last_change":"2024-01-16T15:01:19.035786+0000","last_active":"2024-01-16T15:01:20.142501+0000","last_peered":"2024-01-16T15:01:20.142501+0000","last_clean":"2024-01-16T15:01:20.142501+0000","last_became_active":"2024-01-16T15:01:19.035464+0000","last_became_peered":"2024-01-16T15:01:19.035464+0000","last_unstale":"2024-01-16T15:01:20.142501+0000","last_undegraded":"2024-01-16T15:01:20.142501+0000","last_fullsized":"2024-01-16T15:01:20.142501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:22.022593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.030523+0000","last_change":"2024-01-16T15:01:18.488173+0000","last_active":"2024-01-16T15:01:19.030523+0000","last_peered":"2024-01-16T15:01:19.030523+0000","last_clean":"2024-01-16T15:01:19.030523+0000","last_became_active":"2024-01-16T15:01:18.487709+0000","last_became_peered":"2024-01-16T15:01:18.487709+0000","last_unstale":"2024-01-16T15:01:19.030523+0000","last_undegraded":"2024-01-16T15:01:19.030523+0000","last_fullsized":"2024-01-16T15:01:19.030523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:16.978829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015934+0000","last_change":"2024-01-16T15:01:19.038894+0000","last_active":"2024-01-16T15:01:20.015934+0000","last_peered":"2024-01-16T15:01:20.015934+0000","last_clean":"2024-01-16T15:01:20.015934+0000","last_became_active":"2024-01-16T15:01:19.038447+0000","last_became_peered":"2024-01-16T15:01:19.038447+0000","last_unstale":"2024-01-16T15:01:20.015934+0000","last_undegraded":"2024-01-16T15:01:20.015934+0000","last_fullsized":"2024-01-16T15:01:20.015934+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:07.512715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754099+0000","last_change":"2024-01-16T15:01:18.488894+0000","last_active":"2024-01-16T15:01:20.754099+0000","last_peered":"2024-01-16T15:01:20.754099+0000","last_clean":"2024-01-16T15:01:20.754099+0000","last_became_active":"2024-01-16T15:01:18.488600+0000","last_became_peered":"2024-01-16T15:01:18.488600+0000","last_unstale":"2024-01-16T15:01:20.754099+0000","last_undegraded":"2024-01-16T15:01:20.754099+0000","last_fullsized":"2024-01-16T15:01:20.754099+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:44.184807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":176,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.024172+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:01:19.024172+0000","last_peered":"2024-01-16T15:01:19.024172+0000","last_clean":"2024-01-16T15:01:19.024172+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:01:19.024172+0000","last_undegraded":"2024-01-16T15:01:19.024172+0000","last_fullsized":"2024-01-16T15:01:19.024172+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":154466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.333}]},{"osd":1,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.712}]},{"osd":2,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.466},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.72799999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":1.052}]},{"osd":4,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":1.627}]},{"osd":5,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":1.9630000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.665,"15min":1.665},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.8899999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.673},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":3.4159999999999999,"5min":3.4159999999999999,"15min":3.4159999999999999},"last":1.7669999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":397312,"data_stored":153954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.77000000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.137,"15min":1.137},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.82999999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.397},{"interface":"front","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.714,"5min":2.714,"15min":2.714},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":1.294}]},{"osd":5,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.353},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.782,"5min":2.782,"15min":2.782},"last":1.45}]},{"osd":6,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.484},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.5179999999999998,"5min":3.5179999999999998,"15min":3.5179999999999998},"last":1.214}]},{"osd":7,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.583}]}]},{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":744308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":1.327}]},{"osd":1,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.056},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.1599999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.333,"15min":1.333},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.464},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":1.2589999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":1.288}]},{"osd":6,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.411},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.6680000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.5600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":385024,"data_stored":148222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":1.206},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.536}]},{"osd":2,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.276},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.379}]},{"osd":3,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":1.419},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.456}]},{"osd":4,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.597}]},{"osd":6,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.4870000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.625},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":3.903,"5min":3.903,"15min":3.903},"last":1.573}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":376832,"data_stored":147550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.026}]},{"osd":1,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.077}]},{"osd":2,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":1.522},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.2689999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.327},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.2190000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.413}]},{"osd":6,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.393},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.4650000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.569},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.347,"5min":2.347,"15min":2.347},"last":1.6240000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":380928,"data_stored":147620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.2130000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.79200000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.4059999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2,"5min":2,"15min":2},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.1639999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.278}]},{"osd":7,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.321},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"min":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"max":{"1min":2.54,"5min":2.54,"15min":2.54},"last":1.3440000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":393216,"data_stored":149184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.3779999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.0780000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.327},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.2669999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.4670000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.532},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.425}]},{"osd":6,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.665},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.5860000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":970752,"data_stored":737918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.432}]},{"osd":2,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.986,"5min":1.986,"15min":1.986},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.80000000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.89200000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4300000000000002,"5min":2.4300000000000002,"15min":2.4300000000000002},"last":1.0840000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.403,"15min":1.403},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.6459999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.874},{"interface":"front","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.0549999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":1.298}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:01:23.973 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:01:24.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.254+0000 7f7348400640 1 Processor -- start 2024-01-16T15:01:24.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.254+0000 7f7348400640 1 -- start start 2024-01-16T15:01:24.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.256+0000 7f7348400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f73400cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.256+0000 7f7348400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f734015f9c0 0x7f73400cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.256+0000 7f7348400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734015c5f0 con 0x7f734013df80 2024-01-16T15:01:24.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.256+0000 7f7348400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f734006c470 con 0x7f734015f9c0 2024-01-16T15:01:24.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.256+0000 7f7348400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734006c750 con 0x7f734013e360 2024-01-16T15:01:24.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.257+0000 7f7346400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f734015f9c0 0x7f73400cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.257+0000 7f7345000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f73400cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.257+0000 7f7346400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f734015f9c0 0x7f73400cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:34250/0 (socket says 172.21.15.53:34250) 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.257+0000 7f7346400640 1 -- 172.21.15.53:0/3357456067 learned_addr learned my addr 172.21.15.53:0/3357456067 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.258+0000 7f7345000640 1 -- 172.21.15.53:0/3357456067 >> v2:172.21.15.146:3300/0 conn(0x7f734015f9c0 msgr2=0x7f73400cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.258+0000 7f7345000640 1 --2- 172.21.15.53:0/3357456067 >> v2:172.21.15.146:3300/0 conn(0x7f734015f9c0 0x7f73400cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.258+0000 7f7345000640 1 -- 172.21.15.53:0/3357456067 >> v1:172.21.15.146:6789/0 conn(0x7f734013df80 legacy=0x7f734015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.258+0000 7f7345000640 1 -- 172.21.15.53:0/3357456067 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734006ca30 con 0x7f734013e360 2024-01-16T15:01:24.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.259+0000 7f7345000640 1 --2- 172.21.15.53:0/3357456067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f73400cf290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f73280097a0 tx=0x7f732802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcfe28373011662 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:24.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.259+0000 7f733fe00640 1 -- 172.21.15.53:0/3357456067 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732802d040 con 0x7f734013e360 2024-01-16T15:01:24.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.259+0000 7f733fe00640 1 -- 172.21.15.53:0/3357456067 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7328031810 con 0x7f734013e360 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 -- 172.21.15.53:0/3357456067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 msgr2=0x7f73400cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 --2- 172.21.15.53:0/3357456067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f73400cf290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f73280097a0 tx=0x7f732802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 -- 172.21.15.53:0/3357456067 shutdown_connections 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 --2- 172.21.15.53:0/3357456067 >> v2:172.21.15.146:3300/0 conn(0x7f734015f9c0 0x7f73400cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 --2- 172.21.15.53:0/3357456067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f73400cf290 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 -- 172.21.15.53:0/3357456067 >> 172.21.15.53:0/3357456067 conn(0x7f734006a800 msgr2=0x7f734006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 -- 172.21.15.53:0/3357456067 shutdown_connections 2024-01-16T15:01:24.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.260+0000 7f7348400640 1 -- 172.21.15.53:0/3357456067 wait complete. 2024-01-16T15:01:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.262+0000 7f7348400640 1 Processor -- start 2024-01-16T15:01:24.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.262+0000 7f7348400640 1 -- start start 2024-01-16T15:01:24.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.263+0000 7f7348400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 0x7f734013d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.263+0000 7f7348400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f734014eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.263+0000 7f7348400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f734015f9c0 0x7f734014f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.263+0000 7f7348400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f734014ffd0 con 0x7f734013df80 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.263+0000 7f7348400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73401f3210 con 0x7f734015f9c0 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.263+0000 7f7348400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f734014fa10 con 0x7f734013e360 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 0x7f734013d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 0x7f734013d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47770/0 (socket says 172.21.15.53:47770) 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 -- 172.21.15.53:0/3906157368 learned_addr learned my addr 172.21.15.53:0/3906157368 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7346400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f734015f9c0 0x7f734014f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345000640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f734014eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 -- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 msgr2=0x7f734014eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f734014eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 -- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f734015f9c0 msgr2=0x7f734014f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f734015f9c0 0x7f734014f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345a00640 1 -- 172.21.15.53:0/3906157368 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7328009400 con 0x7f734013df80 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.264+0000 7f7345000640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f734014eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.265+0000 7f7345a00640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 0x7f734013d520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f733000c430 tx=0x7f733000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.265+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73300052d0 con 0x7f734013df80 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.265+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7330005430 con 0x7f734013df80 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.265+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7330003830 con 0x7f734013df80 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.265+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f734014fcf0 con 0x7f734013df80 2024-01-16T15:01:24.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.265+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73400b0020 con 0x7f734013df80 2024-01-16T15:01:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.267+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73400afba0 con 0x7f734013df80 2024-01-16T15:01:24.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.267+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f7330003990 con 0x7f734013df80 2024-01-16T15:01:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.269+0000 7f733d600640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f731806b2c0 0x7f731806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.269+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f73300891d0 con 0x7f734013df80 2024-01-16T15:01:24.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.270+0000 7f7345000640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f731806b2c0 0x7f731806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.271+0000 7f7345000640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f731806b2c0 0x7f731806d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7328036000 tx=0x7f732803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:24.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.275+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f73400afba0 con 0x7f734013df80 2024-01-16T15:01:24.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.465+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f734006abe0 con 0x7f731806b2c0 2024-01-16T15:01:24.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.487+0000 7f733d600640 1 -- 172.21.15.53:0/3906157368 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286684 (secure 0 0 0) 0x7f734006abe0 con 0x7f731806b2c0 2024-01-16T15:01:24.473 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:24.474 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f731806b2c0 msgr2=0x7f731806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f731806b2c0 0x7f731806d780 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7328036000 tx=0x7f732803c040 comp rx=0 tx=0).stop 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 msgr2=0x7f734013d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 0x7f734013d520 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f733000c430 tx=0x7f733000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 shutdown_connections 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f731806b2c0 0x7f731806d780 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f734015f9c0 0x7f734014f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f734013e360 0x7f734014eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 --2- 172.21.15.53:0/3906157368 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f734013df80 0x7f734013d520 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 >> 172.21.15.53:0/3906157368 conn(0x7f734006a800 msgr2=0x7f73401536e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:24.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 shutdown_connections 2024-01-16T15:01:24.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.493+0000 7f7348400640 1 -- 172.21.15.53:0/3906157368 wait complete. 2024-01-16T15:01:24.494 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-01-16T15:01:24.201580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":95,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236984,"kb_used_data":4188,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737736,"statfs":{"total":773094113280,"available":772851441664,"internally_reserved":0,"allocated":4288512,"data_stored":2383222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":238276144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002102"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754626+0000","last_change":"2024-01-16T15:01:19.047111+0000","last_active":"2024-01-16T15:01:20.754626+0000","last_peered":"2024-01-16T15:01:20.754626+0000","last_clean":"2024-01-16T15:01:20.754626+0000","last_became_active":"2024-01-16T15:01:19.046844+0000","last_became_peered":"2024-01-16T15:01:19.046844+0000","last_unstale":"2024-01-16T15:01:20.754626+0000","last_undegraded":"2024-01-16T15:01:20.754626+0000","last_fullsized":"2024-01-16T15:01:20.754626+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.533005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.039308+0000","last_change":"2024-01-16T15:01:19.035158+0000","last_active":"2024-01-16T15:01:19.039308+0000","last_peered":"2024-01-16T15:01:19.039308+0000","last_clean":"2024-01-16T15:01:19.039308+0000","last_became_active":"2024-01-16T15:01:19.034893+0000","last_became_peered":"2024-01-16T15:01:19.034893+0000","last_unstale":"2024-01-16T15:01:19.039308+0000","last_undegraded":"2024-01-16T15:01:19.039308+0000","last_fullsized":"2024-01-16T15:01:19.039308+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:27:26.968382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019029+0000","last_change":"2024-01-16T15:01:19.048055+0000","last_active":"2024-01-16T15:01:20.019029+0000","last_peered":"2024-01-16T15:01:20.019029+0000","last_clean":"2024-01-16T15:01:20.019029+0000","last_became_active":"2024-01-16T15:01:19.046966+0000","last_became_peered":"2024-01-16T15:01:19.046966+0000","last_unstale":"2024-01-16T15:01:20.019029+0000","last_undegraded":"2024-01-16T15:01:20.019029+0000","last_fullsized":"2024-01-16T15:01:20.019029+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:18.453597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049185+0000","last_change":"2024-01-16T15:01:19.044420+0000","last_active":"2024-01-16T15:01:19.049185+0000","last_peered":"2024-01-16T15:01:19.049185+0000","last_clean":"2024-01-16T15:01:19.049185+0000","last_became_active":"2024-01-16T15:01:19.043953+0000","last_became_peered":"2024-01-16T15:01:19.043953+0000","last_unstale":"2024-01-16T15:01:19.049185+0000","last_undegraded":"2024-01-16T15:01:19.049185+0000","last_fullsized":"2024-01-16T15:01:19.049185+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:08.957102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.046920+0000","last_change":"2024-01-16T15:01:19.037628+0000","last_active":"2024-01-16T15:01:19.046920+0000","last_peered":"2024-01-16T15:01:19.046920+0000","last_clean":"2024-01-16T15:01:19.046920+0000","last_became_active":"2024-01-16T15:01:19.037334+0000","last_became_peered":"2024-01-16T15:01:19.037334+0000","last_unstale":"2024-01-16T15:01:19.046920+0000","last_undegraded":"2024-01-16T15:01:19.046920+0000","last_fullsized":"2024-01-16T15:01:19.046920+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:32:40.541057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.041145+0000","last_change":"2024-01-16T15:01:19.037650+0000","last_active":"2024-01-16T15:01:19.041145+0000","last_peered":"2024-01-16T15:01:19.041145+0000","last_clean":"2024-01-16T15:01:19.041145+0000","last_became_active":"2024-01-16T15:01:19.036793+0000","last_became_peered":"2024-01-16T15:01:19.036793+0000","last_unstale":"2024-01-16T15:01:19.041145+0000","last_undegraded":"2024-01-16T15:01:19.041145+0000","last_fullsized":"2024-01-16T15:01:19.041145+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:25.010165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015600+0000","last_change":"2024-01-16T15:01:19.041618+0000","last_active":"2024-01-16T15:01:20.015600+0000","last_peered":"2024-01-16T15:01:20.015600+0000","last_clean":"2024-01-16T15:01:20.015600+0000","last_became_active":"2024-01-16T15:01:19.041264+0000","last_became_peered":"2024-01-16T15:01:19.041264+0000","last_unstale":"2024-01-16T15:01:20.015600+0000","last_undegraded":"2024-01-16T15:01:20.015600+0000","last_fullsized":"2024-01-16T15:01:20.015600+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:18.453597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015002+0000","last_change":"2024-01-16T15:01:19.041987+0000","last_active":"2024-01-16T15:01:20.015002+0000","last_peered":"2024-01-16T15:01:20.015002+0000","last_clean":"2024-01-16T15:01:20.015002+0000","last_became_active":"2024-01-16T15:01:19.039018+0000","last_became_peered":"2024-01-16T15:01:19.039018+0000","last_unstale":"2024-01-16T15:01:20.015002+0000","last_undegraded":"2024-01-16T15:01:20.015002+0000","last_fullsized":"2024-01-16T15:01:20.015002+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:29.247514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.143062+0000","last_change":"2024-01-16T15:01:19.043165+0000","last_active":"2024-01-16T15:01:20.143062+0000","last_peered":"2024-01-16T15:01:20.143062+0000","last_clean":"2024-01-16T15:01:20.143062+0000","last_became_active":"2024-01-16T15:01:19.040497+0000","last_became_peered":"2024-01-16T15:01:19.040497+0000","last_unstale":"2024-01-16T15:01:20.143062+0000","last_undegraded":"2024-01-16T15:01:20.143062+0000","last_fullsized":"2024-01-16T15:01:20.143062+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:17.993305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048939+0000","last_change":"2024-01-16T15:01:19.043101+0000","last_active":"2024-01-16T15:01:19.048939+0000","last_peered":"2024-01-16T15:01:19.048939+0000","last_clean":"2024-01-16T15:01:19.048939+0000","last_became_active":"2024-01-16T15:01:19.042761+0000","last_became_peered":"2024-01-16T15:01:19.042761+0000","last_unstale":"2024-01-16T15:01:19.048939+0000","last_undegraded":"2024-01-16T15:01:19.048939+0000","last_fullsized":"2024-01-16T15:01:19.048939+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:53.580359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016576+0000","last_change":"2024-01-16T15:01:19.047751+0000","last_active":"2024-01-16T15:01:20.016576+0000","last_peered":"2024-01-16T15:01:20.016576+0000","last_clean":"2024-01-16T15:01:20.016576+0000","last_became_active":"2024-01-16T15:01:19.047275+0000","last_became_peered":"2024-01-16T15:01:19.047275+0000","last_unstale":"2024-01-16T15:01:20.016576+0000","last_undegraded":"2024-01-16T15:01:20.016576+0000","last_fullsized":"2024-01-16T15:01:20.016576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:19:58.174525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018831+0000","last_change":"2024-01-16T15:01:19.042987+0000","last_active":"2024-01-16T15:01:20.018831+0000","last_peered":"2024-01-16T15:01:20.018831+0000","last_clean":"2024-01-16T15:01:20.018831+0000","last_became_active":"2024-01-16T15:01:19.041451+0000","last_became_peered":"2024-01-16T15:01:19.041451+0000","last_unstale":"2024-01-16T15:01:20.018831+0000","last_undegraded":"2024-01-16T15:01:20.018831+0000","last_fullsized":"2024-01-16T15:01:20.018831+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:29.247514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048186+0000","last_change":"2024-01-16T15:01:19.045254+0000","last_active":"2024-01-16T15:01:19.048186+0000","last_peered":"2024-01-16T15:01:19.048186+0000","last_clean":"2024-01-16T15:01:19.048186+0000","last_became_active":"2024-01-16T15:01:19.044990+0000","last_became_peered":"2024-01-16T15:01:19.044990+0000","last_unstale":"2024-01-16T15:01:19.048186+0000","last_undegraded":"2024-01-16T15:01:19.048186+0000","last_fullsized":"2024-01-16T15:01:19.048186+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:33.128266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.043413+0000","last_change":"2024-01-16T15:01:19.040779+0000","last_active":"2024-01-16T15:01:19.043413+0000","last_peered":"2024-01-16T15:01:19.043413+0000","last_clean":"2024-01-16T15:01:19.043413+0000","last_became_active":"2024-01-16T15:01:19.040486+0000","last_became_peered":"2024-01-16T15:01:19.040486+0000","last_unstale":"2024-01-16T15:01:19.043413+0000","last_undegraded":"2024-01-16T15:01:19.043413+0000","last_fullsized":"2024-01-16T15:01:19.043413+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:39.237725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018205+0000","last_change":"2024-01-16T15:01:19.048431+0000","last_active":"2024-01-16T15:01:20.018205+0000","last_peered":"2024-01-16T15:01:20.018205+0000","last_clean":"2024-01-16T15:01:20.018205+0000","last_became_active":"2024-01-16T15:01:19.048178+0000","last_became_peered":"2024-01-16T15:01:19.048178+0000","last_unstale":"2024-01-16T15:01:20.018205+0000","last_undegraded":"2024-01-16T15:01:20.018205+0000","last_fullsized":"2024-01-16T15:01:20.018205+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:10.016381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016051+0000","last_change":"2024-01-16T15:01:19.039285+0000","last_active":"2024-01-16T15:01:20.016051+0000","last_peered":"2024-01-16T15:01:20.016051+0000","last_clean":"2024-01-16T15:01:20.016051+0000","last_became_active":"2024-01-16T15:01:19.038635+0000","last_became_peered":"2024-01-16T15:01:19.038635+0000","last_unstale":"2024-01-16T15:01:20.016051+0000","last_undegraded":"2024-01-16T15:01:20.016051+0000","last_fullsized":"2024-01-16T15:01:20.016051+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:16.439958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016827+0000","last_change":"2024-01-16T15:01:19.050193+0000","last_active":"2024-01-16T15:01:20.016827+0000","last_peered":"2024-01-16T15:01:20.016827+0000","last_clean":"2024-01-16T15:01:20.016827+0000","last_became_active":"2024-01-16T15:01:19.049907+0000","last_became_peered":"2024-01-16T15:01:19.049907+0000","last_unstale":"2024-01-16T15:01:20.016827+0000","last_undegraded":"2024-01-16T15:01:20.016827+0000","last_fullsized":"2024-01-16T15:01:20.016827+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:25:55.992504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.041813+0000","last_change":"2024-01-16T15:01:19.038959+0000","last_active":"2024-01-16T15:01:19.041813+0000","last_peered":"2024-01-16T15:01:19.041813+0000","last_clean":"2024-01-16T15:01:19.041813+0000","last_became_active":"2024-01-16T15:01:19.038700+0000","last_became_peered":"2024-01-16T15:01:19.038700+0000","last_unstale":"2024-01-16T15:01:19.041813+0000","last_undegraded":"2024-01-16T15:01:19.041813+0000","last_fullsized":"2024-01-16T15:01:19.041813+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:12.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.045793+0000","last_change":"2024-01-16T15:01:19.042419+0000","last_active":"2024-01-16T15:01:19.045793+0000","last_peered":"2024-01-16T15:01:19.045793+0000","last_clean":"2024-01-16T15:01:19.045793+0000","last_became_active":"2024-01-16T15:01:19.042053+0000","last_became_peered":"2024-01-16T15:01:19.042053+0000","last_unstale":"2024-01-16T15:01:19.045793+0000","last_undegraded":"2024-01-16T15:01:19.045793+0000","last_fullsized":"2024-01-16T15:01:19.045793+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:47.135061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048136+0000","last_change":"2024-01-16T15:01:19.044905+0000","last_active":"2024-01-16T15:01:19.048136+0000","last_peered":"2024-01-16T15:01:19.048136+0000","last_clean":"2024-01-16T15:01:19.048136+0000","last_became_active":"2024-01-16T15:01:19.044613+0000","last_became_peered":"2024-01-16T15:01:19.044613+0000","last_unstale":"2024-01-16T15:01:19.048136+0000","last_undegraded":"2024-01-16T15:01:19.048136+0000","last_fullsized":"2024-01-16T15:01:19.048136+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:03:59.796076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142845+0000","last_change":"2024-01-16T15:01:19.029251+0000","last_active":"2024-01-16T15:01:20.142845+0000","last_peered":"2024-01-16T15:01:20.142845+0000","last_clean":"2024-01-16T15:01:20.142845+0000","last_became_active":"2024-01-16T15:01:19.028375+0000","last_became_peered":"2024-01-16T15:01:19.028375+0000","last_unstale":"2024-01-16T15:01:20.142845+0000","last_undegraded":"2024-01-16T15:01:20.142845+0000","last_fullsized":"2024-01-16T15:01:20.142845+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:46.719103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142315+0000","last_change":"2024-01-16T15:01:19.036613+0000","last_active":"2024-01-16T15:01:20.142315+0000","last_peered":"2024-01-16T15:01:20.142315+0000","last_clean":"2024-01-16T15:01:20.142315+0000","last_became_active":"2024-01-16T15:01:19.029866+0000","last_became_peered":"2024-01-16T15:01:19.029866+0000","last_unstale":"2024-01-16T15:01:20.142315+0000","last_undegraded":"2024-01-16T15:01:20.142315+0000","last_fullsized":"2024-01-16T15:01:20.142315+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:27:26.968382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.031933+0000","last_change":"2024-01-16T15:01:19.028495+0000","last_active":"2024-01-16T15:01:19.031933+0000","last_peered":"2024-01-16T15:01:19.031933+0000","last_clean":"2024-01-16T15:01:19.031933+0000","last_became_active":"2024-01-16T15:01:19.028058+0000","last_became_peered":"2024-01-16T15:01:19.028058+0000","last_unstale":"2024-01-16T15:01:19.031933+0000","last_undegraded":"2024-01-16T15:01:19.031933+0000","last_fullsized":"2024-01-16T15:01:19.031933+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:39.562787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018025+0000","last_change":"2024-01-16T15:01:19.046995+0000","last_active":"2024-01-16T15:01:20.018025+0000","last_peered":"2024-01-16T15:01:20.018025+0000","last_clean":"2024-01-16T15:01:20.018025+0000","last_became_active":"2024-01-16T15:01:19.045662+0000","last_became_peered":"2024-01-16T15:01:19.045662+0000","last_unstale":"2024-01-16T15:01:20.018025+0000","last_undegraded":"2024-01-16T15:01:20.018025+0000","last_fullsized":"2024-01-16T15:01:20.018025+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:15.099576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048961+0000","last_change":"2024-01-16T15:01:19.043585+0000","last_active":"2024-01-16T15:01:19.048961+0000","last_peered":"2024-01-16T15:01:19.048961+0000","last_clean":"2024-01-16T15:01:19.048961+0000","last_became_active":"2024-01-16T15:01:19.043285+0000","last_became_peered":"2024-01-16T15:01:19.043285+0000","last_unstale":"2024-01-16T15:01:19.048961+0000","last_undegraded":"2024-01-16T15:01:19.048961+0000","last_fullsized":"2024-01-16T15:01:19.048961+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:16.354580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049341+0000","last_change":"2024-01-16T15:01:19.047682+0000","last_active":"2024-01-16T15:01:19.049341+0000","last_peered":"2024-01-16T15:01:19.049341+0000","last_clean":"2024-01-16T15:01:19.049341+0000","last_became_active":"2024-01-16T15:01:19.045795+0000","last_became_peered":"2024-01-16T15:01:19.045795+0000","last_unstale":"2024-01-16T15:01:19.049341+0000","last_undegraded":"2024-01-16T15:01:19.049341+0000","last_fullsized":"2024-01-16T15:01:19.049341+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:03.572984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048988+0000","last_change":"2024-01-16T15:01:19.042670+0000","last_active":"2024-01-16T15:01:19.048988+0000","last_peered":"2024-01-16T15:01:19.048988+0000","last_clean":"2024-01-16T15:01:19.048988+0000","last_became_active":"2024-01-16T15:01:19.042320+0000","last_became_peered":"2024-01-16T15:01:19.042320+0000","last_unstale":"2024-01-16T15:01:19.048988+0000","last_undegraded":"2024-01-16T15:01:19.048988+0000","last_fullsized":"2024-01-16T15:01:19.048988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:57.912755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018820+0000","last_change":"2024-01-16T15:01:19.042269+0000","last_active":"2024-01-16T15:01:20.018820+0000","last_peered":"2024-01-16T15:01:20.018820+0000","last_clean":"2024-01-16T15:01:20.018820+0000","last_became_active":"2024-01-16T15:01:19.041844+0000","last_became_peered":"2024-01-16T15:01:19.041844+0000","last_unstale":"2024-01-16T15:01:20.018820+0000","last_undegraded":"2024-01-16T15:01:20.018820+0000","last_fullsized":"2024-01-16T15:01:20.018820+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:39.180167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142406+0000","last_change":"2024-01-16T15:01:19.038023+0000","last_active":"2024-01-16T15:01:20.142406+0000","last_peered":"2024-01-16T15:01:20.142406+0000","last_clean":"2024-01-16T15:01:20.142406+0000","last_became_active":"2024-01-16T15:01:19.036102+0000","last_became_peered":"2024-01-16T15:01:19.036102+0000","last_unstale":"2024-01-16T15:01:20.142406+0000","last_undegraded":"2024-01-16T15:01:20.142406+0000","last_fullsized":"2024-01-16T15:01:20.142406+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:37:03.572984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015752+0000","last_change":"2024-01-16T15:01:19.042153+0000","last_active":"2024-01-16T15:01:20.015752+0000","last_peered":"2024-01-16T15:01:20.015752+0000","last_clean":"2024-01-16T15:01:20.015752+0000","last_became_active":"2024-01-16T15:01:19.041295+0000","last_became_peered":"2024-01-16T15:01:19.041295+0000","last_unstale":"2024-01-16T15:01:20.015752+0000","last_undegraded":"2024-01-16T15:01:20.015752+0000","last_fullsized":"2024-01-16T15:01:20.015752+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:48:24.148127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754183+0000","last_change":"2024-01-16T15:01:19.052938+0000","last_active":"2024-01-16T15:01:20.754183+0000","last_peered":"2024-01-16T15:01:20.754183+0000","last_clean":"2024-01-16T15:01:20.754183+0000","last_became_active":"2024-01-16T15:01:19.050222+0000","last_became_peered":"2024-01-16T15:01:19.050222+0000","last_unstale":"2024-01-16T15:01:20.754183+0000","last_undegraded":"2024-01-16T15:01:20.754183+0000","last_fullsized":"2024-01-16T15:01:20.754183+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:56.206938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.014995+0000","last_change":"2024-01-16T15:01:19.035534+0000","last_active":"2024-01-16T15:01:20.014995+0000","last_peered":"2024-01-16T15:01:20.014995+0000","last_clean":"2024-01-16T15:01:20.014995+0000","last_became_active":"2024-01-16T15:01:19.035154+0000","last_became_peered":"2024-01-16T15:01:19.035154+0000","last_unstale":"2024-01-16T15:01:20.014995+0000","last_undegraded":"2024-01-16T15:01:20.014995+0000","last_fullsized":"2024-01-16T15:01:20.014995+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:55.501685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.038593+0000","last_change":"2024-01-16T15:01:19.035154+0000","last_active":"2024-01-16T15:01:19.038593+0000","last_peered":"2024-01-16T15:01:19.038593+0000","last_clean":"2024-01-16T15:01:19.038593+0000","last_became_active":"2024-01-16T15:01:19.034725+0000","last_became_peered":"2024-01-16T15:01:19.034725+0000","last_unstale":"2024-01-16T15:01:19.038593+0000","last_undegraded":"2024-01-16T15:01:19.038593+0000","last_fullsized":"2024-01-16T15:01:19.038593+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:57.912755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"34'4","reported_seq":16,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754811+0000","last_change":"2024-01-16T15:01:18.491800+0000","last_active":"2024-01-16T15:01:20.754811+0000","last_peered":"2024-01-16T15:01:20.754811+0000","last_clean":"2024-01-16T15:01:20.754811+0000","last_became_active":"2024-01-16T15:01:18.491505+0000","last_became_peered":"2024-01-16T15:01:18.491505+0000","last_unstale":"2024-01-16T15:01:20.754811+0000","last_undegraded":"2024-01-16T15:01:20.754811+0000","last_fullsized":"2024-01-16T15:01:20.754811+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:33.163437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754689+0000","last_change":"2024-01-16T15:01:19.052360+0000","last_active":"2024-01-16T15:01:20.754689+0000","last_peered":"2024-01-16T15:01:20.754689+0000","last_clean":"2024-01-16T15:01:20.754689+0000","last_became_active":"2024-01-16T15:01:19.052048+0000","last_became_peered":"2024-01-16T15:01:19.052048+0000","last_unstale":"2024-01-16T15:01:20.754689+0000","last_undegraded":"2024-01-16T15:01:20.754689+0000","last_fullsized":"2024-01-16T15:01:20.754689+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:12.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.490224+0000","last_change":"2024-01-16T15:01:18.487981+0000","last_active":"2024-01-16T15:01:19.490224+0000","last_peered":"2024-01-16T15:01:19.490224+0000","last_clean":"2024-01-16T15:01:19.490224+0000","last_became_active":"2024-01-16T15:01:18.487348+0000","last_became_peered":"2024-01-16T15:01:18.487348+0000","last_unstale":"2024-01-16T15:01:19.490224+0000","last_undegraded":"2024-01-16T15:01:19.490224+0000","last_fullsized":"2024-01-16T15:01:19.490224+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:58.120947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.143369+0000","last_change":"2024-01-16T15:01:19.037180+0000","last_active":"2024-01-16T15:01:20.143369+0000","last_peered":"2024-01-16T15:01:20.143369+0000","last_clean":"2024-01-16T15:01:20.143369+0000","last_became_active":"2024-01-16T15:01:19.036289+0000","last_became_peered":"2024-01-16T15:01:19.036289+0000","last_unstale":"2024-01-16T15:01:20.143369+0000","last_undegraded":"2024-01-16T15:01:20.143369+0000","last_fullsized":"2024-01-16T15:01:20.143369+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:05.861683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488139+0000","last_change":"2024-01-16T15:01:18.469397+0000","last_active":"2024-01-16T15:01:19.488139+0000","last_peered":"2024-01-16T15:01:19.488139+0000","last_clean":"2024-01-16T15:01:19.488139+0000","last_became_active":"2024-01-16T15:01:18.468952+0000","last_became_peered":"2024-01-16T15:01:18.468952+0000","last_unstale":"2024-01-16T15:01:19.488139+0000","last_undegraded":"2024-01-16T15:01:19.488139+0000","last_fullsized":"2024-01-16T15:01:19.488139+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:04.847207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.046325+0000","last_change":"2024-01-16T15:01:19.039749+0000","last_active":"2024-01-16T15:01:19.046325+0000","last_peered":"2024-01-16T15:01:19.046325+0000","last_clean":"2024-01-16T15:01:19.046325+0000","last_became_active":"2024-01-16T15:01:19.039439+0000","last_became_peered":"2024-01-16T15:01:19.039439+0000","last_unstale":"2024-01-16T15:01:19.046325+0000","last_undegraded":"2024-01-16T15:01:19.046325+0000","last_fullsized":"2024-01-16T15:01:19.046325+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:59.135423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.487032+0000","last_change":"2024-01-16T15:01:18.477058+0000","last_active":"2024-01-16T15:01:19.487032+0000","last_peered":"2024-01-16T15:01:19.487032+0000","last_clean":"2024-01-16T15:01:19.487032+0000","last_became_active":"2024-01-16T15:01:18.476800+0000","last_became_peered":"2024-01-16T15:01:18.476800+0000","last_unstale":"2024-01-16T15:01:19.487032+0000","last_undegraded":"2024-01-16T15:01:19.487032+0000","last_fullsized":"2024-01-16T15:01:19.487032+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:01:27.299481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.016168+0000","last_change":"2024-01-16T15:01:19.038648+0000","last_active":"2024-01-16T15:01:20.016168+0000","last_peered":"2024-01-16T15:01:20.016168+0000","last_clean":"2024-01-16T15:01:20.016168+0000","last_became_active":"2024-01-16T15:01:19.038226+0000","last_became_peered":"2024-01-16T15:01:19.038226+0000","last_unstale":"2024-01-16T15:01:20.016168+0000","last_undegraded":"2024-01-16T15:01:20.016168+0000","last_fullsized":"2024-01-16T15:01:20.016168+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:59:06.302962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142664+0000","last_change":"2024-01-16T15:01:18.475512+0000","last_active":"2024-01-16T15:01:20.142664+0000","last_peered":"2024-01-16T15:01:20.142664+0000","last_clean":"2024-01-16T15:01:20.142664+0000","last_became_active":"2024-01-16T15:01:18.475241+0000","last_became_peered":"2024-01-16T15:01:18.475241+0000","last_unstale":"2024-01-16T15:01:20.142664+0000","last_undegraded":"2024-01-16T15:01:20.142664+0000","last_fullsized":"2024-01-16T15:01:20.142664+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:33.163437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018509+0000","last_change":"2024-01-16T15:01:19.040981+0000","last_active":"2024-01-16T15:01:20.018509+0000","last_peered":"2024-01-16T15:01:20.018509+0000","last_clean":"2024-01-16T15:01:20.018509+0000","last_became_active":"2024-01-16T15:01:19.040698+0000","last_became_peered":"2024-01-16T15:01:19.040698+0000","last_unstale":"2024-01-16T15:01:20.018509+0000","last_undegraded":"2024-01-16T15:01:20.018509+0000","last_fullsized":"2024-01-16T15:01:20.018509+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:55.501685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142262+0000","last_change":"2024-01-16T15:01:18.482628+0000","last_active":"2024-01-16T15:01:20.142262+0000","last_peered":"2024-01-16T15:01:20.142262+0000","last_clean":"2024-01-16T15:01:20.142262+0000","last_became_active":"2024-01-16T15:01:18.481456+0000","last_became_peered":"2024-01-16T15:01:18.481456+0000","last_unstale":"2024-01-16T15:01:20.142262+0000","last_undegraded":"2024-01-16T15:01:20.142262+0000","last_fullsized":"2024-01-16T15:01:20.142262+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:35.090050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142067+0000","last_change":"2024-01-16T15:01:19.044957+0000","last_active":"2024-01-16T15:01:20.142067+0000","last_peered":"2024-01-16T15:01:20.142067+0000","last_clean":"2024-01-16T15:01:20.142067+0000","last_became_active":"2024-01-16T15:01:19.044512+0000","last_became_peered":"2024-01-16T15:01:19.044512+0000","last_unstale":"2024-01-16T15:01:20.142067+0000","last_undegraded":"2024-01-16T15:01:20.142067+0000","last_fullsized":"2024-01-16T15:01:20.142067+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:17:12.455376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.029906+0000","last_change":"2024-01-16T15:01:18.480799+0000","last_active":"2024-01-16T15:01:19.029906+0000","last_peered":"2024-01-16T15:01:19.029906+0000","last_clean":"2024-01-16T15:01:19.029906+0000","last_became_active":"2024-01-16T15:01:18.478757+0000","last_became_peered":"2024-01-16T15:01:18.478757+0000","last_unstale":"2024-01-16T15:01:19.029906+0000","last_undegraded":"2024-01-16T15:01:19.029906+0000","last_fullsized":"2024-01-16T15:01:19.029906+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:39.547663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049250+0000","last_change":"2024-01-16T15:01:19.044788+0000","last_active":"2024-01-16T15:01:19.049250+0000","last_peered":"2024-01-16T15:01:19.049250+0000","last_clean":"2024-01-16T15:01:19.049250+0000","last_became_active":"2024-01-16T15:01:19.044455+0000","last_became_peered":"2024-01-16T15:01:19.044455+0000","last_unstale":"2024-01-16T15:01:19.049250+0000","last_undegraded":"2024-01-16T15:01:19.049250+0000","last_fullsized":"2024-01-16T15:01:19.049250+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:08.071343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.141885+0000","last_change":"2024-01-16T15:01:18.485725+0000","last_active":"2024-01-16T15:01:20.141885+0000","last_peered":"2024-01-16T15:01:20.141885+0000","last_clean":"2024-01-16T15:01:20.141885+0000","last_became_active":"2024-01-16T15:01:18.485431+0000","last_became_peered":"2024-01-16T15:01:18.485431+0000","last_unstale":"2024-01-16T15:01:20.141885+0000","last_undegraded":"2024-01-16T15:01:20.141885+0000","last_fullsized":"2024-01-16T15:01:20.141885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:01.496327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019295+0000","last_change":"2024-01-16T15:01:19.040911+0000","last_active":"2024-01-16T15:01:20.019295+0000","last_peered":"2024-01-16T15:01:20.019295+0000","last_clean":"2024-01-16T15:01:20.019295+0000","last_became_active":"2024-01-16T15:01:19.039966+0000","last_became_peered":"2024-01-16T15:01:19.039966+0000","last_unstale":"2024-01-16T15:01:20.019295+0000","last_undegraded":"2024-01-16T15:01:20.019295+0000","last_fullsized":"2024-01-16T15:01:20.019295+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:07.512715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.032902+0000","last_change":"2024-01-16T15:01:18.493177+0000","last_active":"2024-01-16T15:01:19.032902+0000","last_peered":"2024-01-16T15:01:19.032902+0000","last_clean":"2024-01-16T15:01:19.032902+0000","last_became_active":"2024-01-16T15:01:18.492803+0000","last_became_peered":"2024-01-16T15:01:18.492803+0000","last_unstale":"2024-01-16T15:01:19.032902+0000","last_undegraded":"2024-01-16T15:01:19.032902+0000","last_fullsized":"2024-01-16T15:01:19.032902+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:32:40.055476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.042591+0000","last_change":"2024-01-16T15:01:19.039580+0000","last_active":"2024-01-16T15:01:19.042591+0000","last_peered":"2024-01-16T15:01:19.042591+0000","last_clean":"2024-01-16T15:01:19.042591+0000","last_became_active":"2024-01-16T15:01:19.039262+0000","last_became_peered":"2024-01-16T15:01:19.039262+0000","last_unstale":"2024-01-16T15:01:19.042591+0000","last_undegraded":"2024-01-16T15:01:19.042591+0000","last_fullsized":"2024-01-16T15:01:19.042591+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.533005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.014376+0000","last_change":"2024-01-16T15:01:18.473665+0000","last_active":"2024-01-16T15:01:19.014376+0000","last_peered":"2024-01-16T15:01:19.014376+0000","last_clean":"2024-01-16T15:01:19.014376+0000","last_became_active":"2024-01-16T15:01:18.473233+0000","last_became_peered":"2024-01-16T15:01:18.473233+0000","last_unstale":"2024-01-16T15:01:19.014376+0000","last_undegraded":"2024-01-16T15:01:19.014376+0000","last_fullsized":"2024-01-16T15:01:19.014376+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:52.680143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.043495+0000","last_change":"2024-01-16T15:01:19.041472+0000","last_active":"2024-01-16T15:01:19.043495+0000","last_peered":"2024-01-16T15:01:19.043495+0000","last_clean":"2024-01-16T15:01:19.043495+0000","last_became_active":"2024-01-16T15:01:19.041200+0000","last_became_peered":"2024-01-16T15:01:19.041200+0000","last_unstale":"2024-01-16T15:01:19.043495+0000","last_undegraded":"2024-01-16T15:01:19.043495+0000","last_fullsized":"2024-01-16T15:01:19.043495+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:22.022593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"34'3","reported_seq":15,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019124+0000","last_change":"2024-01-16T15:01:18.492933+0000","last_active":"2024-01-16T15:01:20.019124+0000","last_peered":"2024-01-16T15:01:20.019124+0000","last_clean":"2024-01-16T15:01:20.019124+0000","last_became_active":"2024-01-16T15:01:18.492514+0000","last_became_peered":"2024-01-16T15:01:18.492514+0000","last_unstale":"2024-01-16T15:01:20.019124+0000","last_undegraded":"2024-01-16T15:01:20.019124+0000","last_fullsized":"2024-01-16T15:01:20.019124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:56.016453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.018832+0000","last_change":"2024-01-16T15:01:19.048370+0000","last_active":"2024-01-16T15:01:20.018832+0000","last_peered":"2024-01-16T15:01:20.018832+0000","last_clean":"2024-01-16T15:01:20.018832+0000","last_became_active":"2024-01-16T15:01:19.047838+0000","last_became_peered":"2024-01-16T15:01:19.047838+0000","last_unstale":"2024-01-16T15:01:20.018832+0000","last_undegraded":"2024-01-16T15:01:20.018832+0000","last_fullsized":"2024-01-16T15:01:20.018832+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:48.466587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"34'6","reported_seq":17,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.496496+0000","last_change":"2024-01-16T15:01:18.473272+0000","last_active":"2024-01-16T15:01:19.496496+0000","last_peered":"2024-01-16T15:01:19.496496+0000","last_clean":"2024-01-16T15:01:19.496496+0000","last_became_active":"2024-01-16T15:01:18.472513+0000","last_became_peered":"2024-01-16T15:01:18.472513+0000","last_unstale":"2024-01-16T15:01:19.496496+0000","last_undegraded":"2024-01-16T15:01:19.496496+0000","last_fullsized":"2024-01-16T15:01:19.496496+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:52.784526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049464+0000","last_change":"2024-01-16T15:01:19.044492+0000","last_active":"2024-01-16T15:01:19.049464+0000","last_peered":"2024-01-16T15:01:19.049464+0000","last_clean":"2024-01-16T15:01:19.049464+0000","last_became_active":"2024-01-16T15:01:19.044233+0000","last_became_peered":"2024-01-16T15:01:19.044233+0000","last_unstale":"2024-01-16T15:01:19.049464+0000","last_undegraded":"2024-01-16T15:01:19.049464+0000","last_fullsized":"2024-01-16T15:01:19.049464+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:58.091122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142917+0000","last_change":"2024-01-16T15:01:18.489034+0000","last_active":"2024-01-16T15:01:20.142917+0000","last_peered":"2024-01-16T15:01:20.142917+0000","last_clean":"2024-01-16T15:01:20.142917+0000","last_became_active":"2024-01-16T15:01:18.488759+0000","last_became_peered":"2024-01-16T15:01:18.488759+0000","last_unstale":"2024-01-16T15:01:20.142917+0000","last_undegraded":"2024-01-16T15:01:20.142917+0000","last_fullsized":"2024-01-16T15:01:20.142917+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:57.567634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.037856+0000","last_change":"2024-01-16T15:01:19.033187+0000","last_active":"2024-01-16T15:01:19.037856+0000","last_peered":"2024-01-16T15:01:19.037856+0000","last_clean":"2024-01-16T15:01:19.037856+0000","last_became_active":"2024-01-16T15:01:19.032894+0000","last_became_peered":"2024-01-16T15:01:19.032894+0000","last_unstale":"2024-01-16T15:01:19.037856+0000","last_undegraded":"2024-01-16T15:01:19.037856+0000","last_fullsized":"2024-01-16T15:01:19.037856+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:56.206938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.030523+0000","last_change":"2024-01-16T15:01:18.489543+0000","last_active":"2024-01-16T15:01:19.030523+0000","last_peered":"2024-01-16T15:01:19.030523+0000","last_clean":"2024-01-16T15:01:19.030523+0000","last_became_active":"2024-01-16T15:01:18.489158+0000","last_became_peered":"2024-01-16T15:01:18.489158+0000","last_unstale":"2024-01-16T15:01:19.030523+0000","last_undegraded":"2024-01-16T15:01:19.030523+0000","last_fullsized":"2024-01-16T15:01:19.030523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:22:14.011515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015385+0000","last_change":"2024-01-16T15:01:19.042003+0000","last_active":"2024-01-16T15:01:20.015385+0000","last_peered":"2024-01-16T15:01:20.015385+0000","last_clean":"2024-01-16T15:01:20.015385+0000","last_became_active":"2024-01-16T15:01:19.041734+0000","last_became_peered":"2024-01-16T15:01:19.041734+0000","last_unstale":"2024-01-16T15:01:20.015385+0000","last_undegraded":"2024-01-16T15:01:20.015385+0000","last_fullsized":"2024-01-16T15:01:20.015385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:48.466587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.032333+0000","last_change":"2024-01-16T15:01:18.485309+0000","last_active":"2024-01-16T15:01:19.032333+0000","last_peered":"2024-01-16T15:01:19.032333+0000","last_clean":"2024-01-16T15:01:19.032333+0000","last_became_active":"2024-01-16T15:01:18.485020+0000","last_became_peered":"2024-01-16T15:01:18.485020+0000","last_unstale":"2024-01-16T15:01:19.032333+0000","last_undegraded":"2024-01-16T15:01:19.032333+0000","last_fullsized":"2024-01-16T15:01:19.032333+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:29:41.832883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754348+0000","last_change":"2024-01-16T15:01:19.052740+0000","last_active":"2024-01-16T15:01:20.754348+0000","last_peered":"2024-01-16T15:01:20.754348+0000","last_clean":"2024-01-16T15:01:20.754348+0000","last_became_active":"2024-01-16T15:01:19.049863+0000","last_became_peered":"2024-01-16T15:01:19.049863+0000","last_unstale":"2024-01-16T15:01:20.754348+0000","last_undegraded":"2024-01-16T15:01:20.754348+0000","last_fullsized":"2024-01-16T15:01:20.754348+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:47.135061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"34'8","reported_seq":19,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488077+0000","last_change":"2024-01-16T15:01:18.469296+0000","last_active":"2024-01-16T15:01:19.488077+0000","last_peered":"2024-01-16T15:01:19.488077+0000","last_clean":"2024-01-16T15:01:19.488077+0000","last_became_active":"2024-01-16T15:01:18.469041+0000","last_became_peered":"2024-01-16T15:01:18.469041+0000","last_unstale":"2024-01-16T15:01:19.488077+0000","last_undegraded":"2024-01-16T15:01:19.488077+0000","last_fullsized":"2024-01-16T15:01:19.488077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:14:45.704627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142676+0000","last_change":"2024-01-16T15:01:19.042733+0000","last_active":"2024-01-16T15:01:20.142676+0000","last_peered":"2024-01-16T15:01:20.142676+0000","last_clean":"2024-01-16T15:01:20.142676+0000","last_became_active":"2024-01-16T15:01:19.040078+0000","last_became_peered":"2024-01-16T15:01:19.040078+0000","last_unstale":"2024-01-16T15:01:20.142676+0000","last_undegraded":"2024-01-16T15:01:20.142676+0000","last_fullsized":"2024-01-16T15:01:20.142676+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:40.562139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"34'1","reported_seq":12,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488582+0000","last_change":"2024-01-16T15:01:18.477106+0000","last_active":"2024-01-16T15:01:19.488582+0000","last_peered":"2024-01-16T15:01:19.488582+0000","last_clean":"2024-01-16T15:01:19.488582+0000","last_became_active":"2024-01-16T15:01:18.476072+0000","last_became_peered":"2024-01-16T15:01:18.476072+0000","last_unstale":"2024-01-16T15:01:19.488582+0000","last_undegraded":"2024-01-16T15:01:19.488582+0000","last_fullsized":"2024-01-16T15:01:19.488582+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:40:37.460893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019257+0000","last_change":"2024-01-16T15:01:19.045922+0000","last_active":"2024-01-16T15:01:20.019257+0000","last_peered":"2024-01-16T15:01:20.019257+0000","last_clean":"2024-01-16T15:01:20.019257+0000","last_became_active":"2024-01-16T15:01:19.045649+0000","last_became_peered":"2024-01-16T15:01:19.045649+0000","last_unstale":"2024-01-16T15:01:20.019257+0000","last_undegraded":"2024-01-16T15:01:20.019257+0000","last_fullsized":"2024-01-16T15:01:20.019257+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:48:24.148127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"34'2","reported_seq":14,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142554+0000","last_change":"2024-01-16T15:01:18.491363+0000","last_active":"2024-01-16T15:01:20.142554+0000","last_peered":"2024-01-16T15:01:20.142554+0000","last_clean":"2024-01-16T15:01:20.142554+0000","last_became_active":"2024-01-16T15:01:18.490548+0000","last_became_peered":"2024-01-16T15:01:18.490548+0000","last_unstale":"2024-01-16T15:01:20.142554+0000","last_undegraded":"2024-01-16T15:01:20.142554+0000","last_fullsized":"2024-01-16T15:01:20.142554+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:46.704589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049169+0000","last_change":"2024-01-16T15:01:19.044227+0000","last_active":"2024-01-16T15:01:19.049169+0000","last_peered":"2024-01-16T15:01:19.049169+0000","last_clean":"2024-01-16T15:01:19.049169+0000","last_became_active":"2024-01-16T15:01:19.041548+0000","last_became_peered":"2024-01-16T15:01:19.041548+0000","last_unstale":"2024-01-16T15:01:19.049169+0000","last_undegraded":"2024-01-16T15:01:19.049169+0000","last_fullsized":"2024-01-16T15:01:19.049169+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:53.694619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"34'1","reported_seq":13,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142970+0000","last_change":"2024-01-16T15:01:18.484649+0000","last_active":"2024-01-16T15:01:20.142970+0000","last_peered":"2024-01-16T15:01:20.142970+0000","last_clean":"2024-01-16T15:01:20.142970+0000","last_became_active":"2024-01-16T15:01:18.483981+0000","last_became_peered":"2024-01-16T15:01:18.483981+0000","last_unstale":"2024-01-16T15:01:20.142970+0000","last_undegraded":"2024-01-16T15:01:20.142970+0000","last_fullsized":"2024-01-16T15:01:20.142970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:23.995689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048200+0000","last_change":"2024-01-16T15:01:19.045159+0000","last_active":"2024-01-16T15:01:19.048200+0000","last_peered":"2024-01-16T15:01:19.048200+0000","last_clean":"2024-01-16T15:01:19.048200+0000","last_became_active":"2024-01-16T15:01:19.043354+0000","last_became_peered":"2024-01-16T15:01:19.043354+0000","last_unstale":"2024-01-16T15:01:19.048200+0000","last_undegraded":"2024-01-16T15:01:19.048200+0000","last_fullsized":"2024-01-16T15:01:19.048200+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:35.130556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.143177+0000","last_change":"2024-01-16T15:01:18.476597+0000","last_active":"2024-01-16T15:01:20.143177+0000","last_peered":"2024-01-16T15:01:20.143177+0000","last_clean":"2024-01-16T15:01:20.143177+0000","last_became_active":"2024-01-16T15:01:18.476310+0000","last_became_peered":"2024-01-16T15:01:18.476310+0000","last_unstale":"2024-01-16T15:01:20.143177+0000","last_undegraded":"2024-01-16T15:01:20.143177+0000","last_fullsized":"2024-01-16T15:01:20.143177+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:38.223249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.049270+0000","last_change":"2024-01-16T15:01:19.047471+0000","last_active":"2024-01-16T15:01:19.049270+0000","last_peered":"2024-01-16T15:01:19.049270+0000","last_clean":"2024-01-16T15:01:19.049270+0000","last_became_active":"2024-01-16T15:01:19.047195+0000","last_became_peered":"2024-01-16T15:01:19.047195+0000","last_unstale":"2024-01-16T15:01:19.049270+0000","last_undegraded":"2024-01-16T15:01:19.049270+0000","last_fullsized":"2024-01-16T15:01:19.049270+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:32.591220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488632+0000","last_change":"2024-01-16T15:01:18.479262+0000","last_active":"2024-01-16T15:01:19.488632+0000","last_peered":"2024-01-16T15:01:19.488632+0000","last_clean":"2024-01-16T15:01:19.488632+0000","last_became_active":"2024-01-16T15:01:18.478965+0000","last_became_peered":"2024-01-16T15:01:18.478965+0000","last_unstale":"2024-01-16T15:01:19.488632+0000","last_undegraded":"2024-01-16T15:01:19.488632+0000","last_fullsized":"2024-01-16T15:01:19.488632+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:56.981351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.046942+0000","last_change":"2024-01-16T15:01:19.039864+0000","last_active":"2024-01-16T15:01:19.046942+0000","last_peered":"2024-01-16T15:01:19.046942+0000","last_clean":"2024-01-16T15:01:19.046942+0000","last_became_active":"2024-01-16T15:01:19.036729+0000","last_became_peered":"2024-01-16T15:01:19.036729+0000","last_unstale":"2024-01-16T15:01:19.046942+0000","last_undegraded":"2024-01-16T15:01:19.046942+0000","last_fullsized":"2024-01-16T15:01:19.046942+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:35.299330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754379+0000","last_change":"2024-01-16T15:01:18.490027+0000","last_active":"2024-01-16T15:01:20.754379+0000","last_peered":"2024-01-16T15:01:20.754379+0000","last_clean":"2024-01-16T15:01:20.754379+0000","last_became_active":"2024-01-16T15:01:18.489615+0000","last_became_peered":"2024-01-16T15:01:18.489615+0000","last_unstale":"2024-01-16T15:01:20.754379+0000","last_undegraded":"2024-01-16T15:01:20.754379+0000","last_fullsized":"2024-01-16T15:01:20.754379+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:35.090050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.047430+0000","last_change":"2024-01-16T15:01:19.038096+0000","last_active":"2024-01-16T15:01:19.047430+0000","last_peered":"2024-01-16T15:01:19.047430+0000","last_clean":"2024-01-16T15:01:19.047430+0000","last_became_active":"2024-01-16T15:01:19.036902+0000","last_became_peered":"2024-01-16T15:01:19.036902+0000","last_unstale":"2024-01-16T15:01:19.047430+0000","last_undegraded":"2024-01-16T15:01:19.047430+0000","last_fullsized":"2024-01-16T15:01:19.047430+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:12:16.914038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142037+0000","last_change":"2024-01-16T15:01:18.480620+0000","last_active":"2024-01-16T15:01:20.142037+0000","last_peered":"2024-01-16T15:01:20.142037+0000","last_clean":"2024-01-16T15:01:20.142037+0000","last_became_active":"2024-01-16T15:01:18.480281+0000","last_became_peered":"2024-01-16T15:01:18.480281+0000","last_unstale":"2024-01-16T15:01:20.142037+0000","last_undegraded":"2024-01-16T15:01:20.142037+0000","last_fullsized":"2024-01-16T15:01:20.142037+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:46.120585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015125+0000","last_change":"2024-01-16T15:01:19.051501+0000","last_active":"2024-01-16T15:01:20.015125+0000","last_peered":"2024-01-16T15:01:20.015125+0000","last_clean":"2024-01-16T15:01:20.015125+0000","last_became_active":"2024-01-16T15:01:19.051087+0000","last_became_peered":"2024-01-16T15:01:19.051087+0000","last_unstale":"2024-01-16T15:01:20.015125+0000","last_undegraded":"2024-01-16T15:01:20.015125+0000","last_fullsized":"2024-01-16T15:01:20.015125+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:14:39.180167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.017910+0000","last_change":"2024-01-16T15:01:18.488108+0000","last_active":"2024-01-16T15:01:20.017910+0000","last_peered":"2024-01-16T15:01:20.017910+0000","last_clean":"2024-01-16T15:01:20.017910+0000","last_became_active":"2024-01-16T15:01:18.487832+0000","last_became_peered":"2024-01-16T15:01:18.487832+0000","last_unstale":"2024-01-16T15:01:20.017910+0000","last_undegraded":"2024-01-16T15:01:20.017910+0000","last_fullsized":"2024-01-16T15:01:20.017910+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:52.145878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048265+0000","last_change":"2024-01-16T15:01:19.046514+0000","last_active":"2024-01-16T15:01:19.048265+0000","last_peered":"2024-01-16T15:01:19.048265+0000","last_clean":"2024-01-16T15:01:19.048265+0000","last_became_active":"2024-01-16T15:01:19.046178+0000","last_became_peered":"2024-01-16T15:01:19.046178+0000","last_unstale":"2024-01-16T15:01:19.048265+0000","last_undegraded":"2024-01-16T15:01:19.048265+0000","last_fullsized":"2024-01-16T15:01:19.048265+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:47.607088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"34'2","reported_seq":13,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.488902+0000","last_change":"2024-01-16T15:01:18.485980+0000","last_active":"2024-01-16T15:01:19.488902+0000","last_peered":"2024-01-16T15:01:19.488902+0000","last_clean":"2024-01-16T15:01:19.488902+0000","last_became_active":"2024-01-16T15:01:18.485628+0000","last_became_peered":"2024-01-16T15:01:18.485628+0000","last_unstale":"2024-01-16T15:01:19.488902+0000","last_undegraded":"2024-01-16T15:01:19.488902+0000","last_fullsized":"2024-01-16T15:01:19.488902+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:05:33.918678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.019349+0000","last_change":"2024-01-16T15:01:19.043592+0000","last_active":"2024-01-16T15:01:20.019349+0000","last_peered":"2024-01-16T15:01:20.019349+0000","last_clean":"2024-01-16T15:01:20.019349+0000","last_became_active":"2024-01-16T15:01:19.043134+0000","last_became_peered":"2024-01-16T15:01:19.043134+0000","last_unstale":"2024-01-16T15:01:20.019349+0000","last_undegraded":"2024-01-16T15:01:20.019349+0000","last_fullsized":"2024-01-16T15:01:20.019349+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:16.439958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.014771+0000","last_change":"2024-01-16T15:01:18.473486+0000","last_active":"2024-01-16T15:01:19.014771+0000","last_peered":"2024-01-16T15:01:19.014771+0000","last_clean":"2024-01-16T15:01:19.014771+0000","last_became_active":"2024-01-16T15:01:18.472875+0000","last_became_peered":"2024-01-16T15:01:18.472875+0000","last_unstale":"2024-01-16T15:01:19.014771+0000","last_undegraded":"2024-01-16T15:01:19.014771+0000","last_fullsized":"2024-01-16T15:01:19.014771+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:57.076646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.042435+0000","last_change":"2024-01-16T15:01:19.038031+0000","last_active":"2024-01-16T15:01:19.042435+0000","last_peered":"2024-01-16T15:01:19.042435+0000","last_clean":"2024-01-16T15:01:19.042435+0000","last_became_active":"2024-01-16T15:01:19.037324+0000","last_became_peered":"2024-01-16T15:01:19.037324+0000","last_unstale":"2024-01-16T15:01:19.042435+0000","last_undegraded":"2024-01-16T15:01:19.042435+0000","last_fullsized":"2024-01-16T15:01:19.042435+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:08.957102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.031616+0000","last_change":"2024-01-16T15:01:18.480306+0000","last_active":"2024-01-16T15:01:19.031616+0000","last_peered":"2024-01-16T15:01:19.031616+0000","last_clean":"2024-01-16T15:01:19.031616+0000","last_became_active":"2024-01-16T15:01:18.479817+0000","last_became_peered":"2024-01-16T15:01:18.479817+0000","last_unstale":"2024-01-16T15:01:19.031616+0000","last_undegraded":"2024-01-16T15:01:19.031616+0000","last_fullsized":"2024-01-16T15:01:19.031616+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:37:47.527171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754176+0000","last_change":"2024-01-16T15:01:19.046107+0000","last_active":"2024-01-16T15:01:20.754176+0000","last_peered":"2024-01-16T15:01:20.754176+0000","last_clean":"2024-01-16T15:01:20.754176+0000","last_became_active":"2024-01-16T15:01:19.045830+0000","last_became_peered":"2024-01-16T15:01:19.045830+0000","last_unstale":"2024-01-16T15:01:20.754176+0000","last_undegraded":"2024-01-16T15:01:20.754176+0000","last_fullsized":"2024-01-16T15:01:20.754176+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:25.010165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.013592+0000","last_change":"2024-01-16T15:01:18.482133+0000","last_active":"2024-01-16T15:01:19.013592+0000","last_peered":"2024-01-16T15:01:19.013592+0000","last_clean":"2024-01-16T15:01:19.013592+0000","last_became_active":"2024-01-16T15:01:18.481693+0000","last_became_peered":"2024-01-16T15:01:18.481693+0000","last_unstale":"2024-01-16T15:01:19.013592+0000","last_undegraded":"2024-01-16T15:01:19.013592+0000","last_fullsized":"2024-01-16T15:01:19.013592+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:55:07.056868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.048929+0000","last_change":"2024-01-16T15:01:19.047096+0000","last_active":"2024-01-16T15:01:19.048929+0000","last_peered":"2024-01-16T15:01:19.048929+0000","last_clean":"2024-01-16T15:01:19.048929+0000","last_became_active":"2024-01-16T15:01:19.046832+0000","last_became_peered":"2024-01-16T15:01:19.046832+0000","last_unstale":"2024-01-16T15:01:19.048929+0000","last_undegraded":"2024-01-16T15:01:19.048929+0000","last_fullsized":"2024-01-16T15:01:19.048929+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:09:21.690178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.030673+0000","last_change":"2024-01-16T15:01:18.485941+0000","last_active":"2024-01-16T15:01:19.030673+0000","last_peered":"2024-01-16T15:01:19.030673+0000","last_clean":"2024-01-16T15:01:19.030673+0000","last_became_active":"2024-01-16T15:01:18.485618+0000","last_became_peered":"2024-01-16T15:01:18.485618+0000","last_unstale":"2024-01-16T15:01:19.030673+0000","last_undegraded":"2024-01-16T15:01:19.030673+0000","last_fullsized":"2024-01-16T15:01:19.030673+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:51:24.383674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":9,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.042717+0000","last_change":"2024-01-16T15:01:19.038114+0000","last_active":"2024-01-16T15:01:19.042717+0000","last_peered":"2024-01-16T15:01:19.042717+0000","last_clean":"2024-01-16T15:01:19.042717+0000","last_became_active":"2024-01-16T15:01:19.036337+0000","last_became_peered":"2024-01-16T15:01:19.036337+0000","last_unstale":"2024-01-16T15:01:19.042717+0000","last_undegraded":"2024-01-16T15:01:19.042717+0000","last_fullsized":"2024-01-16T15:01:19.042717+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:34.177913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142300+0000","last_change":"2024-01-16T15:01:18.484537+0000","last_active":"2024-01-16T15:01:20.142300+0000","last_peered":"2024-01-16T15:01:20.142300+0000","last_clean":"2024-01-16T15:01:20.142300+0000","last_became_active":"2024-01-16T15:01:18.484243+0000","last_became_peered":"2024-01-16T15:01:18.484243+0000","last_unstale":"2024-01-16T15:01:20.142300+0000","last_undegraded":"2024-01-16T15:01:20.142300+0000","last_fullsized":"2024-01-16T15:01:20.142300+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:55.192462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.142501+0000","last_change":"2024-01-16T15:01:19.035786+0000","last_active":"2024-01-16T15:01:20.142501+0000","last_peered":"2024-01-16T15:01:20.142501+0000","last_clean":"2024-01-16T15:01:20.142501+0000","last_became_active":"2024-01-16T15:01:19.035464+0000","last_became_peered":"2024-01-16T15:01:19.035464+0000","last_unstale":"2024-01-16T15:01:20.142501+0000","last_undegraded":"2024-01-16T15:01:20.142501+0000","last_fullsized":"2024-01-16T15:01:20.142501+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:22.022593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.030523+0000","last_change":"2024-01-16T15:01:18.488173+0000","last_active":"2024-01-16T15:01:19.030523+0000","last_peered":"2024-01-16T15:01:19.030523+0000","last_clean":"2024-01-16T15:01:19.030523+0000","last_became_active":"2024-01-16T15:01:18.487709+0000","last_became_peered":"2024-01-16T15:01:18.487709+0000","last_unstale":"2024-01-16T15:01:19.030523+0000","last_undegraded":"2024-01-16T15:01:19.030523+0000","last_fullsized":"2024-01-16T15:01:19.030523+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:42:16.978829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.015934+0000","last_change":"2024-01-16T15:01:19.038894+0000","last_active":"2024-01-16T15:01:20.015934+0000","last_peered":"2024-01-16T15:01:20.015934+0000","last_clean":"2024-01-16T15:01:20.015934+0000","last_became_active":"2024-01-16T15:01:19.038447+0000","last_became_peered":"2024-01-16T15:01:19.038447+0000","last_unstale":"2024-01-16T15:01:20.015934+0000","last_undegraded":"2024-01-16T15:01:20.015934+0000","last_fullsized":"2024-01-16T15:01:20.015934+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":33,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:18.443282+0000","last_clean_scrub_stamp":"2024-01-16T15:01:18.443282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:07.512715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-01-16T15:01:20.754099+0000","last_change":"2024-01-16T15:01:18.488894+0000","last_active":"2024-01-16T15:01:20.754099+0000","last_peered":"2024-01-16T15:01:20.754099+0000","last_clean":"2024-01-16T15:01:20.754099+0000","last_became_active":"2024-01-16T15:01:18.488600+0000","last_became_peered":"2024-01-16T15:01:18.488600+0000","last_unstale":"2024-01-16T15:01:20.754099+0000","last_undegraded":"2024-01-16T15:01:20.754099+0000","last_fullsized":"2024-01-16T15:01:20.754099+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:01:17.428806+0000","last_clean_scrub_stamp":"2024-01-16T15:01:17.428806+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:44.184807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":176,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-01-16T15:01:19.024172+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:01:19.024172+0000","last_peered":"2024-01-16T15:01:19.024172+0000","last_clean":"2024-01-16T15:01:19.024172+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:01:19.024172+0000","last_undegraded":"2024-01-16T15:01:19.024172+0000","last_fullsized":"2024-01-16T15:01:19.024172+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509461,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":397312,"data_stored":154466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.167,"5min":1.167,"15min":1.167},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.722,"5min":1.722,"15min":1.722},"last":1.333}]},{"osd":1,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.5740000000000001,"5min":1.5740000000000001,"15min":1.5740000000000001},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.246,"15min":1.246},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":1.8009999999999999,"5min":1.8009999999999999,"15min":1.8009999999999999},"last":1.712}]},{"osd":2,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.466},{"interface":"front","average":{"1min":1.399,"5min":1.399,"15min":1.399},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"max":{"1min":2.1840000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":0.72799999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"min":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":3.5710000000000002,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":1.052}]},{"osd":4,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.506,"15min":1.506},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":3.8170000000000002,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.5820000000000001,"5min":1.5820000000000001,"15min":1.5820000000000001},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.6629999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":1.627}]},{"osd":5,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.7030000000000001,"5min":1.7030000000000001,"15min":1.7030000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.8599999999999999,"5min":2.8599999999999999,"15min":2.8599999999999999},"last":1.9630000000000001},{"interface":"front","average":{"1min":1.665,"5min":1.665,"15min":1.665},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":2.5800000000000001,"5min":2.5800000000000001,"15min":2.5800000000000001},"last":1.8899999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.6279999999999999,"5min":1.6279999999999999,"15min":1.6279999999999999},"min":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"max":{"1min":2.383,"5min":2.383,"15min":2.383},"last":1.673},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.7450000000000001,"15min":1.7450000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":3.4159999999999999,"5min":3.4159999999999999,"15min":3.4159999999999999},"last":1.7669999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509463,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27416,"kb_used_data":388,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344424,"statfs":{"total":96636764160,"available":96608690176,"internally_reserved":0,"allocated":397312,"data_stored":153954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.77000000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.137,"15min":1.137},"min":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.7430000000000001,"5min":1.7430000000000001,"15min":1.7430000000000001},"last":1.171},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.0990000000000002},"last":0.82999999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.397},{"interface":"front","average":{"1min":1.276,"5min":1.276,"15min":1.276},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":2.714,"5min":2.714,"15min":2.714},"last":0.94199999999999995}]},{"osd":3,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2989999999999999,"15min":1.2989999999999999},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":2.637,"5min":2.637,"15min":2.637},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.393,"15min":1.393},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.9289999999999998,"5min":2.9289999999999998,"15min":2.9289999999999998},"last":1.294}]},{"osd":5,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.49,"15min":1.49},"min":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"max":{"1min":2.52,"5min":2.52,"15min":2.52},"last":1.353},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.782,"5min":2.782,"15min":2.782},"last":1.45}]},{"osd":6,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.831,"5min":2.831,"15min":2.831},"last":1.484},{"interface":"front","average":{"1min":1.548,"5min":1.548,"15min":1.548},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":3.5179999999999998,"5min":3.5179999999999998,"15min":3.5179999999999998},"last":1.214}]},{"osd":7,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.575,"15min":1.575},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.048,"5min":3.048,"15min":3.048},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.643,"5min":1.643,"15min":1.643},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":3.0190000000000001,"5min":3.0190000000000001,"15min":3.0190000000000001},"last":1.583}]}]},{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343856,"statfs":{"total":96636764160,"available":96608108544,"internally_reserved":0,"allocated":987136,"data_stored":744308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.014,"15min":1.014},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.611,"5min":1.611,"15min":1.611},"last":0.76900000000000002},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":1.327}]},{"osd":1,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.111,"5min":1.111,"15min":1.111},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"last":1.056},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.875,"5min":1.875,"15min":1.875},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.2470000000000001,"15min":1.2470000000000001},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.2509999999999999,"15min":1.2509999999999999},"min":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.901,"5min":1.901,"15min":1.901},"last":1.1599999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8089999999999999,"5min":1.8089999999999999,"15min":1.8089999999999999},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.333,"5min":1.333,"15min":1.333},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.7350000000000001,"5min":1.7350000000000001,"15min":1.7350000000000001},"last":1.5169999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.395,"15min":1.395},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.2879999999999998,"5min":2.2879999999999998,"15min":2.2879999999999998},"last":1.464},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4039999999999999,"15min":1.4039999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.2200000000000002,"5min":2.2200000000000002,"15min":2.2200000000000002},"last":1.2589999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:00:44 2024","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.978,"5min":1.978,"15min":1.978},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.387,"5min":2.387,"15min":2.387},"last":1.288}]},{"osd":6,"last update":"Tue Jan 16 15:00:51 2024","interfaces":[{"interface":"back","average":{"1min":1.5549999999999999,"5min":1.5549999999999999,"15min":1.5549999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.089,"5min":2.089,"15min":2.089},"last":1.411},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5660000000000001,"15min":1.5660000000000001},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.6680000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.5600000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":385024,"data_stored":148222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.645,"5min":1.645,"15min":1.645},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1060000000000001,"15min":1.1060000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":1.206},{"interface":"front","average":{"1min":1.079,"5min":1.079,"15min":1.079},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":1.8200000000000001},"last":1.536}]},{"osd":2,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.766,"5min":1.766,"15min":1.766},"last":1.276},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"min":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"last":1.379}]},{"osd":3,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.874},"max":{"1min":2.919,"5min":2.919,"15min":2.919},"last":1.419},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.3600000000000001,"15min":1.3600000000000001},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.93200000000000005},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.8779999999999999},"last":1.456}]},{"osd":4,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":1.296,"15min":1.296},"min":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.3859999999999999,"15min":1.3859999999999999},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.853,"5min":1.853,"15min":1.853},"last":1.597}]},{"osd":6,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9470000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.4870000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":3.2690000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.625},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":3.903,"5min":3.903,"15min":3.903},"last":1.573}]}]},{"osd":3,"up_from":14,"seq":60129542166,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31552,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340288,"statfs":{"total":96636764160,"available":96604454912,"internally_reserved":0,"allocated":376832,"data_stored":147550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.1899999999999999,"5min":2.1899999999999999,"15min":2.1899999999999999},"last":1.026}]},{"osd":1,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.6200000000000001,"5min":1.6200000000000001,"15min":1.6200000000000001},"last":1.077}]},{"osd":2,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":1.522},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":2.399,"5min":2.399,"15min":2.399},"last":1.2689999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.236,"15min":1.236},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.0449999999999999,"5min":2.0449999999999999,"15min":2.0449999999999999},"last":1.327},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3839999999999999,"15min":1.3839999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.2190000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.381,"15min":1.381},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.2360000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.2930000000000001,"5min":2.2930000000000001,"15min":2.2930000000000001},"last":1.413}]},{"osd":6,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.974,"5min":1.974,"15min":1.974},"last":1.393},{"interface":"front","average":{"1min":1.369,"5min":1.369,"15min":1.369},"min":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.4650000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.4139999999999999},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.569},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.347,"5min":2.347,"15min":2.347},"last":1.6240000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31548,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340292,"statfs":{"total":96636764160,"available":96604459008,"internally_reserved":0,"allocated":380928,"data_stored":147620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.038,"15min":1.038},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.022,"5min":1.022,"15min":1.022},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.6950000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.90700000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.5669999999999999,"5min":1.5669999999999999,"15min":1.5669999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":1.2130000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.88,"15min":0.88},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.7230000000000001,"5min":1.7230000000000001,"15min":1.7230000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.4289999999999998,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.79200000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3129999999999999,"15min":1.3129999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.931,"5min":1.931,"15min":1.931},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.429},"min":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.4059999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.753},"max":{"1min":2,"5min":2,"15min":2},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.349,"5min":1.349,"15min":1.349},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.1639999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3400000000000001,"15min":1.3400000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":1.982,"5min":1.982,"15min":1.982},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.466,"5min":1.466,"15min":1.466},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":1.278}]},{"osd":7,"last update":"Tue Jan 16 15:00:45 2024","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.53,"15min":1.53},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":2.1269999999999998,"5min":2.1269999999999998,"15min":2.1269999999999998},"last":1.321},{"interface":"front","average":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"min":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"max":{"1min":2.54,"5min":2.54,"15min":2.54},"last":1.3440000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542167,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31560,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340280,"statfs":{"total":96636764160,"available":96604446720,"internally_reserved":0,"allocated":393216,"data_stored":149184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.3939999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":1.3779999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.95799999999999996},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.135,"5min":1.135,"15min":1.135},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5609999999999999},"last":1.0780000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.081,"15min":1.081},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.327},{"interface":"front","average":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.2669999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.284,"15min":1.284},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.101,"5min":2.101,"15min":2.101},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.4690000000000001,"15min":1.4690000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.4670000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.337,"5min":1.337,"15min":1.337},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.998,"5min":1.998,"15min":1.998},"last":1.532},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":1.4450000000000001},"min":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.1389999999999998},"last":1.425}]},{"osd":6,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"max":{"1min":2.3809999999999998,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.665},{"interface":"front","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.0169999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":1.6990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:00:50 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.55,"15min":1.55},"min":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"max":{"1min":2.0590000000000002,"5min":2.0590000000000002,"15min":2.0590000000000002},"last":1.5860000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339716,"statfs":{"total":96636764160,"available":96603869184,"internally_reserved":0,"allocated":970752,"data_stored":737918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.8149999999999999,"5min":1.8149999999999999,"15min":1.8149999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.044,"5min":1.044,"15min":1.044},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.432}]},{"osd":2,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.107,"15min":1.107},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.986,"5min":1.986,"15min":1.986},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":2.1120000000000001,"5min":2.1120000000000001,"15min":2.1120000000000001},"last":0.80000000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.115,"15min":1.115},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":2.1440000000000001,"5min":2.1440000000000001,"15min":2.1440000000000001},"last":0.61899999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.367,"15min":1.367},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":2.3700000000000001,"5min":2.3700000000000001,"15min":2.3700000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.3170000000000002,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.89200000000000002}]},{"osd":5,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4140000000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.498,"5min":1.498,"15min":1.498},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":2.4300000000000002,"5min":2.4300000000000002,"15min":2.4300000000000002},"last":1.0840000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:00:48 2024","interfaces":[{"interface":"back","average":{"1min":1.403,"5min":1.403,"15min":1.403},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":2.6459999999999999,"5min":2.6459999999999999,"15min":2.6459999999999999},"last":0.874},{"interface":"front","average":{"1min":1.536,"5min":1.536,"15min":1.536},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.0549999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:00:47 2024","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.532,"15min":1.532},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.542,"5min":1.542,"15min":1.542},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":1.298}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:01:24.499 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:01:24.499 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:01:24.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.785+0000 7fb5e6800640 1 Processor -- start 2024-01-16T15:01:24.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.785+0000 7fb5e6800640 1 -- start start 2024-01-16T15:01:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.787+0000 7fb5e6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015ed20 0x7fb5e0141360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.787+0000 7fb5e6800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e013e440 con 0x7fb5e015f100 2024-01-16T15:01:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.787+0000 7fb5e6800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5e00ce1d0 con 0x7fb5e006c850 2024-01-16T15:01:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.787+0000 7fb5e6800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00ce4b0 con 0x7fb5e015ed20 2024-01-16T15:01:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.788+0000 7fb5d6800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb5e015f100 0x7fb5e00cd780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44582/0 (socket says 172.21.15.53:44582) 2024-01-16T15:01:24.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.788+0000 7fb5d6800640 1 -- 172.21.15.53:0/29248308 learned_addr learned my addr 172.21.15.53:0/29248308 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.788+0000 7fb5df400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015ed20 0x7fb5e0141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.788+0000 7fb5dea00640 1 -- 172.21.15.53:0/29248308 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 925221958 0 0) 0x7fb5e013e440 con 0x7fb5e015f100 2024-01-16T15:01:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.788+0000 7fb5dea00640 1 -- 172.21.15.53:0/29248308 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5c0003520 con 0x7fb5e015f100 2024-01-16T15:01:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.789+0000 7fb5df400640 1 -- 172.21.15.53:0/29248308 >> v1:172.21.15.146:6789/0 conn(0x7fb5e006c850 legacy=0x7fb5e006cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.789+0000 7fb5df400640 1 -- 172.21.15.53:0/29248308 >> v1:172.21.15.53:6789/0 conn(0x7fb5e015f100 legacy=0x7fb5e00cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.789+0000 7fb5df400640 1 -- 172.21.15.53:0/29248308 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5e00ce790 con 0x7fb5e015ed20 2024-01-16T15:01:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.790+0000 7fb5df400640 1 --2- 172.21.15.53:0/29248308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015ed20 0x7fb5e0141360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009dc0 tx=0x7fb5d002be80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da9e46462fb9d49e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5dea00640 1 -- 172.21.15.53:0/29248308 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d002cc30 con 0x7fb5e015ed20 2024-01-16T15:01:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5dea00640 1 -- 172.21.15.53:0/29248308 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5d002cd90 con 0x7fb5e015ed20 2024-01-16T15:01:24.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5dea00640 1 -- 172.21.15.53:0/29248308 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5d0034660 con 0x7fb5e015ed20 2024-01-16T15:01:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5e6800640 1 -- 172.21.15.53:0/29248308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015ed20 msgr2=0x7fb5e0141360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5e6800640 1 --2- 172.21.15.53:0/29248308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015ed20 0x7fb5e0141360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0009dc0 tx=0x7fb5d002be80 comp rx=0 tx=0).stop 2024-01-16T15:01:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5e6800640 1 -- 172.21.15.53:0/29248308 shutdown_connections 2024-01-16T15:01:24.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5e6800640 1 --2- 172.21.15.53:0/29248308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015ed20 0x7fb5e0141360 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.791+0000 7fb5e6800640 1 -- 172.21.15.53:0/29248308 >> 172.21.15.53:0/29248308 conn(0x7fb5e00b0be0 msgr2=0x7fb5e00b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.792+0000 7fb5e6800640 1 -- 172.21.15.53:0/29248308 shutdown_connections 2024-01-16T15:01:24.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.792+0000 7fb5e6800640 1 -- 172.21.15.53:0/29248308 wait complete. 2024-01-16T15:01:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.793+0000 7fb5e6800640 1 Processor -- start 2024-01-16T15:01:24.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.794+0000 7fb5e6800640 1 -- start start 2024-01-16T15:01:24.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.794+0000 7fb5e6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e006c850 0x7fb5e0153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.794+0000 7fb5e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 0x7fb5e0153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5e6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015f100 0x7fb5e01f29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5e6800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e0070ad0 con 0x7fb5e015ed20 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5e6800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e00704a0 con 0x7fb5e006c850 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5e6800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5e01f3690 con 0x7fb5e015f100 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 0x7fb5e0153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 0x7fb5e0153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47780/0 (socket says 172.21.15.53:47780) 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 -- 172.21.15.53:0/3605171776 learned_addr learned my addr 172.21.15.53:0/3605171776 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 -- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015f100 msgr2=0x7fb5e01f29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015f100 0x7fb5e01f29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 -- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e006c850 msgr2=0x7fb5e0153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e006c850 0x7fb5e0153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:24.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.795+0000 7fb5d6800640 1 -- 172.21.15.53:0/3605171776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d0009a70 con 0x7fb5e015ed20 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.796+0000 7fb5d6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 0x7fb5e0153d60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800c530 tx=0x7fb5c800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.796+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8005ce0 con 0x7fb5e015ed20 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.796+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c8005e40 con 0x7fb5e015ed20 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.796+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5e015f6c0 con 0x7fb5e015ed20 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.796+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c80072b0 con 0x7fb5e015ed20 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.796+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5e00706f0 con 0x7fb5e015ed20 2024-01-16T15:01:24.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.798+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5e01f3bb0 con 0x7fb5e015ed20 2024-01-16T15:01:24.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.798+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fb5c8007410 con 0x7fb5e015ed20 2024-01-16T15:01:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.799+0000 7fb5d7e00640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.799+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb5c8089130 con 0x7fb5e015ed20 2024-01-16T15:01:24.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.800+0000 7fb5df400640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:24.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.801+0000 7fb5df400640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0036000 tx=0x7fb5d002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:24.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:24.804+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5c80540f0 con 0x7fb5e015ed20 2024-01-16T15:01:25.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.069+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fb5e00b30c0 con 0x7fb5e015ed20 2024-01-16T15:01:25.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.073+0000 7fb5d7e00640 1 -- 172.21.15.53:0/3605171776 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7fb5c8057710 con 0x7fb5e015ed20 2024-01-16T15:01:25.054 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:01:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:01:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:01:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:01:25.054 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:01:25.054 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.075+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 msgr2=0x7fb5b806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.075+0000 7fb5e6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb5d0036000 tx=0x7fb5d002e000 comp rx=0 tx=0).stop 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.075+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 msgr2=0x7fb5e0153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.075+0000 7fb5e6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 0x7fb5e0153d60 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800c530 tx=0x7fb5c800c9f0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 shutdown_connections 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5b806b2c0 0x7fb5b806d780 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5e015f100 0x7fb5e01f29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5e015ed20 0x7fb5e0153d60 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 --2- 172.21.15.53:0/3605171776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5e006c850 0x7fb5e0153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 >> 172.21.15.53:0/3605171776 conn(0x7fb5e00b0be0 msgr2=0x7fb5e0152020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 shutdown_connections 2024-01-16T15:01:25.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.076+0000 7fb5e6800640 1 -- 172.21.15.53:0/3605171776 wait complete. 2024-01-16T15:01:25.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:01:25.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.318+0000 7f135a000640 1 Processor -- start 2024-01-16T15:01:25.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.319+0000 7f135a000640 1 -- start start 2024-01-16T15:01:25.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.320+0000 7f135a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354160920 0x7f13540b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.320+0000 7f135a000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13540cf9f0 0x7f13540cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.320+0000 7f135a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13540d0a10 con 0x7f1354161010 2024-01-16T15:01:25.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.320+0000 7f135a000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13540d03b0 con 0x7f13540cf9f0 2024-01-16T15:01:25.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.320+0000 7f135a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13540d1f50 con 0x7f1354160920 2024-01-16T15:01:25.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.321+0000 7f134bc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13540cf9f0 0x7f13540cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.321+0000 7f1359600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1354161010 0x7f13541614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:59290/0 (socket says 172.21.15.53:59290) 2024-01-16T15:01:25.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.321+0000 7f1358c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354160920 0x7f13540b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.321+0000 7f1359600640 1 -- 172.21.15.53:0/321825013 learned_addr learned my addr 172.21.15.53:0/321825013 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:25.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.322+0000 7f134bc00640 1 -- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354160920 msgr2=0x7f13540b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:25.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.322+0000 7f134bc00640 1 --2- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354160920 0x7f13540b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.322+0000 7f134bc00640 1 -- 172.21.15.53:0/321825013 >> v1:172.21.15.146:6789/0 conn(0x7f1354161010 legacy=0x7f13541614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.322+0000 7f134bc00640 1 -- 172.21.15.53:0/321825013 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13540cf080 con 0x7f13540cf9f0 2024-01-16T15:01:25.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.323+0000 7f134bc00640 1 --2- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540cfe70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f133c0097a0 tx=0x7f133c02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21b97249a5421ef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.324+0000 7f1353e00640 1 -- 172.21.15.53:0/321825013 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c02d040 con 0x7f13540cf9f0 2024-01-16T15:01:25.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.324+0000 7f1353e00640 1 -- 172.21.15.53:0/321825013 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c031810 con 0x7f13540cf9f0 2024-01-16T15:01:25.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.324+0000 7f1353e00640 1 -- 172.21.15.53:0/321825013 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c031ba0 con 0x7f13540cf9f0 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 -- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 msgr2=0x7f13540cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 --2- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540cfe70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f133c0097a0 tx=0x7f133c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 -- 172.21.15.53:0/321825013 shutdown_connections 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 --2- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540cfe70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 --2- 172.21.15.53:0/321825013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354160920 0x7f13540b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 -- 172.21.15.53:0/321825013 >> 172.21.15.53:0/321825013 conn(0x7f13540d3090 msgr2=0x7f13540d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 -- 172.21.15.53:0/321825013 shutdown_connections 2024-01-16T15:01:25.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.325+0000 7f135a000640 1 -- 172.21.15.53:0/321825013 wait complete. 2024-01-16T15:01:25.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.327+0000 7f135a000640 1 Processor -- start 2024-01-16T15:01:25.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.327+0000 7f135a000640 1 -- start start 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f135a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f135a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 0x7f135414d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f135a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354161010 0x7f13541f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f135a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135414daa0 con 0x7f1354160920 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f135a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13540ca160 con 0x7f13540cf9f0 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f135a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13541f3710 con 0x7f1354161010 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f134bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 0x7f135414d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f134bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 0x7f135414d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47796/0 (socket says 172.21.15.53:47796) 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.328+0000 7f134bc00640 1 -- 172.21.15.53:0/176593034 learned_addr learned my addr 172.21.15.53:0/176593034 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f1358c00640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f1359600640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354161010 0x7f13541f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f134bc00640 1 -- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354161010 msgr2=0x7f13541f2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f134bc00640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354161010 0x7f13541f2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f134bc00640 1 -- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 msgr2=0x7f13540b8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f134bc00640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540b8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f134bc00640 1 -- 172.21.15.53:0/176593034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c009400 con 0x7f1354160920 2024-01-16T15:01:25.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.329+0000 7f134bc00640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 0x7f135414d560 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f133c0098c0 tx=0x7f133c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.330+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c03ac60 con 0x7f1354160920 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.330+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c03ade0 con 0x7f1354160920 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.330+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c044630 con 0x7f1354160920 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.330+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13541f39f0 con 0x7f1354160920 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.330+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13540ca400 con 0x7f1354160920 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.331+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1354067450 con 0x7f1354160920 2024-01-16T15:01:25.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.332+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f133c04d450 con 0x7f1354160920 2024-01-16T15:01:25.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.333+0000 7f1351600640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f132c06b2c0 0x7f132c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:25.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.333+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f133c0abf20 con 0x7f1354160920 2024-01-16T15:01:25.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.333+0000 7f1358c00640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f132c06b2c0 0x7f132c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:25.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.335+0000 7f1358c00640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f132c06b2c0 0x7f132c06d780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13540b61c0 tx=0x7f1344008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:25.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.337+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f133c076cf0 con 0x7f1354160920 2024-01-16T15:01:25.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.570+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135408c7e0 con 0x7f1354160920 2024-01-16T15:01:25.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.572+0000 7f1351600640 1 -- 172.21.15.53:0/176593034 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f133c035090 con 0x7f1354160920 2024-01-16T15:01:25.553 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:01:25.553 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3324487785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3324487785},{"type":"v1","addr":"172.21.15.146:6836","nonce":3324487785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3124134221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3124134221},{"type":"v1","addr":"172.21.15.146:6837","nonce":3124134221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1994154705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1994154705},{"type":"v1","addr":"172.21.15.53:6834","nonce":1994154705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:01:18.926439+0000","modified":"2024-01-16T15:01:21.997432+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5190},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5190":{"gid":5190,"name":"c","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.53:6835/2729092299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2729092299},{"type":"v1","addr":"172.21.15.53:6835","nonce":2729092299}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-01-16T15:01:25.553 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 16 2024-01-16T15:01:25.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f132c06b2c0 msgr2=0x7f132c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f132c06b2c0 0x7f132c06d780 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f13540b61c0 tx=0x7f1344008040 comp rx=0 tx=0).stop 2024-01-16T15:01:25.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 msgr2=0x7f135414d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:25.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 0x7f135414d560 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f133c0098c0 tx=0x7f133c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 shutdown_connections 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f132c06b2c0 0x7f132c06d780 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1354161010 0x7f13541f2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1354160920 0x7f135414d560 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 --2- 172.21.15.53:0/176593034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13540cf9f0 0x7f13540b8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.574+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 >> 172.21.15.53:0/176593034 conn(0x7f13540d3090 msgr2=0x7f135406c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.575+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 shutdown_connections 2024-01-16T15:01:25.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:25.575+0000 7f135a000640 1 -- 172.21.15.53:0/176593034 wait complete. 2024-01-16T15:01:25.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:01:18.926439+0000', 'modified': '2024-01-16T15:01:21.997432+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5190}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5190': {'gid': 5190, 'name': 'c', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.53:6835/2729092299', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6833', 'nonce': 2729092299}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 2729092299}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:01:25.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:01:25.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:01:26.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.224+0000 7f105a600640 1 Processor -- start 2024-01-16T15:01:26.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.225+0000 7f105a600640 1 -- start start 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f105a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1054160920 0x7f10540b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f105a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 0x7f1054161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f105a600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10540d0a10 con 0x7f10540cf9f0 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f105a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10540d0330 con 0x7f1054160920 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f105a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540d1f50 con 0x7f10540cfea0 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f1053e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f10540cf9f0 0x7f1054161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44590/0 (socket says 172.21.15.53:44590) 2024-01-16T15:01:26.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.226+0000 7f1053e00640 1 -- 172.21.15.53:0/2305797678 learned_addr learned my addr 172.21.15.53:0/2305797678 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715095061 0 0) 0x7f10540d0a10 con 0x7f10540cf9f0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1059c00640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 0x7f1054161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1059200640 1 --2- 172.21.15.53:0/2305797678 >> v2:172.21.15.146:3300/0 conn(0x7f1054160920 0x7f10540b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1034003520 con 0x7f10540cf9f0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4271093017 0 0) 0x7f1034003520 con 0x7f10540cf9f0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 msgr2=0x7f1054161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 0x7f1054161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 >> v2:172.21.15.146:3300/0 conn(0x7f1054160920 msgr2=0x7f10540b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 --2- 172.21.15.53:0/2305797678 >> v2:172.21.15.146:3300/0 conn(0x7f1054160920 0x7f10540b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540cf080 con 0x7f10540cf9f0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f10440019f0 con 0x7f10540cf9f0 2024-01-16T15:01:26.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 >> v1:172.21.15.53:6789/0 conn(0x7f10540cf9f0 legacy=0x7f1054161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1034003970 0x7f1034003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10340047c0 0x7f1034004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 0x7f1034005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540d0330 con 0x7f10340047c0 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540d1f50 con 0x7f1034003970 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.227+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f103400b980 con 0x7f10540cfea0 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1053e00640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10340047c0 0x7f1034004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1059c00640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 0x7f1034005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1059200640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1034003970 0x7f1034003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1053e00640 1 -- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 msgr2=0x7f1034005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1053e00640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10540cfea0 0x7f1034005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1053e00640 1 -- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1034003970 msgr2=0x7f1034003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1053e00640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1034003970 0x7f1034003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.228+0000 7f1053e00640 1 -- 172.21.15.53:0/2305797678 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1034003520 con 0x7f10340047c0 2024-01-16T15:01:26.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.229+0000 7f1053e00640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10340047c0 0x7f1034004c40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f104400a830 tx=0x7f10340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e3fe9e2734181f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.229+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044034630 con 0x7f10340047c0 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.229+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104402f070 con 0x7f10340047c0 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.229+0000 7f1053400640 1 -- 172.21.15.53:0/2305797678 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440349d0 con 0x7f10340047c0 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f105a600640 1 -- 172.21.15.53:0/2305797678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10340047c0 msgr2=0x7f1034004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f105a600640 1 --2- 172.21.15.53:0/2305797678 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10340047c0 0x7f1034004c40 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f104400a830 tx=0x7f10340057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f1059c00640 1 -- 172.21.15.53:0/2305797678 reap_dead start 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f105a600640 1 -- 172.21.15.53:0/2305797678 shutdown_connections 2024-01-16T15:01:26.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f105a600640 1 -- 172.21.15.53:0/2305797678 >> 172.21.15.53:0/2305797678 conn(0x7f10540d3090 msgr2=0x7f10540d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f105a600640 1 -- 172.21.15.53:0/2305797678 shutdown_connections 2024-01-16T15:01:26.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.230+0000 7f105a600640 1 -- 172.21.15.53:0/2305797678 wait complete. 2024-01-16T15:01:26.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.232+0000 7f105a600640 1 Processor -- start 2024-01-16T15:01:26.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.232+0000 7f105a600640 1 -- start start 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f105a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10340047c0 0x7f10540cce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f105a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 0x7f105415d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f105a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10541508d0 0x7f105415dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f105a600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540ca690 con 0x7f1054150470 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f105a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10540b8bd0 con 0x7f10541508d0 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f105a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10540cd3c0 con 0x7f10340047c0 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f1053e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 0x7f105415d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f1053e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 0x7f105415d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47826/0 (socket says 172.21.15.53:47826) 2024-01-16T15:01:26.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.233+0000 7f1053e00640 1 -- 172.21.15.53:0/1816118888 learned_addr learned my addr 172.21.15.53:0/1816118888 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1059200640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10340047c0 0x7f10540cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1053e00640 1 -- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10340047c0 msgr2=0x7f10540cce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1053e00640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10340047c0 0x7f10540cce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1059c00640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10541508d0 0x7f105415dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1053e00640 1 -- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10541508d0 msgr2=0x7f105415dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1053e00640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10541508d0 0x7f105415dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1053e00640 1 -- 172.21.15.53:0/1816118888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10540b8610 con 0x7f1054150470 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1053e00640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 0x7f105415d980 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f104402cb70 tx=0x7f10540b4860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1050c00640 1 -- 172.21.15.53:0/1816118888 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044004b60 con 0x7f1054150470 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1050c00640 1 -- 172.21.15.53:0/1816118888 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1044004cc0 con 0x7f1054150470 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f1050c00640 1 -- 172.21.15.53:0/1816118888 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440358f0 con 0x7f1054150470 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.234+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10540b88f0 con 0x7f1054150470 2024-01-16T15:01:26.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.235+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10540b5fd0 con 0x7f1054150470 2024-01-16T15:01:26.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.237+0000 7f1050c00640 1 -- 172.21.15.53:0/1816118888 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f1044031030 con 0x7f1054150470 2024-01-16T15:01:26.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.238+0000 7f1050c00640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f102c06b2c0 0x7f102c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.238+0000 7f1050c00640 1 -- 172.21.15.53:0/1816118888 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f10440ac0a0 con 0x7f1054150470 2024-01-16T15:01:26.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.238+0000 7f105e877740 1 Processor -- start 2024-01-16T15:01:26.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.238+0000 7f1059200640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f102c06b2c0 0x7f102c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.238+0000 7f105e877740 1 -- start start 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6c070 0x5619f6f6c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f105e877740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f6b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6faf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f105e877740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6eca880 con 0x5619f6f66d70 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6ed9ae0 con 0x5619f6f6c070 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619f6ed71d0 con 0x5619f6f6b7b0 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.239+0000 7f1059200640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f102c06b2c0 0x7f102c06d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f103c009990 tx=0x7f103c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f6b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f6b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47834/0 (socket says 172.21.15.53:47834) 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 -- 172.21.15.53:0/1383431071 learned_addr learned my addr 172.21.15.53:0/1383431071 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1059200640 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6c070 0x5619f6f6c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 -- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 msgr2=0x5619f6faf730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6faf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 -- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6c070 msgr2=0x5619f6f6c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6c070 0x5619f6f6c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 -- 172.21.15.53:0/1383431071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619f6e9a860 con 0x5619f6f66d70 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1053e00640 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f6b270 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f10440aa470 tx=0x7f104407a5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f761956871b3a13 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1049600640 1 -- 172.21.15.53:0/1383431071 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1044078ed0 con 0x5619f6f66d70 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1049600640 1 -- 172.21.15.53:0/1383431071 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104400a480 con 0x5619f6f66d70 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.240+0000 7f1049600640 1 -- 172.21.15.53:0/1383431071 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10440829a0 con 0x5619f6f66d70 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 -- 172.21.15.53:0/1383431071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 msgr2=0x5619f6f6b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f6b270 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f10440aa470 tx=0x7f104407a5a0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 -- 172.21.15.53:0/1383431071 shutdown_connections 2024-01-16T15:01:26.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6faf730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f6b270 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 --2- 172.21.15.53:0/1383431071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6c070 0x5619f6f6c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 -- 172.21.15.53:0/1383431071 >> 172.21.15.53:0/1383431071 conn(0x5619f6f2dc70 msgr2=0x5619f6fb1b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 -- 172.21.15.53:0/1383431071 shutdown_connections 2024-01-16T15:01:26.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.241+0000 7f105e877740 1 -- 172.21.15.53:0/1383431071 wait complete. 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.242+0000 7f105e877740 1 -- start start 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.242+0000 7f105e877740 1 Processor -- start 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f8a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f8a370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6c070 0x5619f6f65dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6ed2ac0 con 0x5619f6f66d70 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6e6f2f0 con 0x5619f6f6b7b0 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619f6e8cc70 con 0x5619f6f6c070 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f105e877740 10 client.0 _ll_register_callbacks cb 0x5619f6e913b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f1059200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f8a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f1059200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f8a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47846/0 (socket says 172.21.15.53:47846) 2024-01-16T15:01:26.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.243+0000 7f1059200640 1 -- 172.21.15.53:0/3969274852 learned_addr learned my addr 172.21.15.53:0/3969274852 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1059200640 1 -- 172.21.15.53:0/3969274852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6c070 msgr2=0x5619f6f65dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1059200640 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6c070 0x5619f6f65dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1059200640 1 -- 172.21.15.53:0/3969274852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 msgr2=0x5619f6f8a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1059200640 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f8a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1059200640 1 -- 172.21.15.53:0/3969274852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10540b5fd0 con 0x5619f6f66d70 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1059200640 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f8a970 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f103c009760 tx=0x7f103c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2dc198934b43770 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1042000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.244+0000 7f1042000640 1 -- 172.21.15.53:0/3969274852 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c015e90 con 0x5619f6f66d70 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f1042000640 1 -- 172.21.15.53:0/3969274852 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f103c021620 con 0x5619f6f66d70 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5619f6ecbaa0 con 0x5619f6f66d70 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f1042000640 1 -- 172.21.15.53:0/3969274852 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f103c0217e0 con 0x5619f6f66d70 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 10 client.5349 fetch_fsmap learned FSMap version 16 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5619f6ed1140 con 0x5619f6f66d70 2024-01-16T15:01:26.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f1042000640 1 -- 172.21.15.53:0/3969274852 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5619f6ecbaa0 con 0x5619f6f66d70 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f1042000640 1 -- 172.21.15.53:0/3969274852 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f103c021940 con 0x5619f6f66d70 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 10 client.5349 fetch_fsmap finished waiting for FSMap version 16 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 10 client.5349 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 20 client.5349 populate_metadata read hostname 'smithi053' 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f6f66310 0x5619f6fb2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 4 client.5349 mds_command: new command op to 5190 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.245+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 --> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5619f6ecb7c0 con 0x5619f6f66310 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.246+0000 7f1053e00640 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f6f66310 0x5619f6fb2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.246+0000 7f1053e00640 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f6f66310 0x5619f6fb2c90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9747d8157e595a3 server_cookie=3ea54216ac97a335 in_seq=0 out_seq=0 2024-01-16T15:01:26.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.246+0000 7f1042000640 10 client.5349 ms_handle_connect on v2:172.21.15.53:6833/2729092299 2024-01-16T15:01:26.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f1042000640 1 -- 172.21.15.53:0/3969274852 <== mds.0 v2:172.21.15.53:6833/2729092299 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5619f6ecb7c0 con 0x5619f6f66310 2024-01-16T15:01:26.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f1042000640 10 client.5349 handle_command_reply: tid=0 2024-01-16T15:01:26.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 client.5349 shutdown 2024-01-16T15:01:26.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 msgr2=0x5619f6f8a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f8a970 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f103c009760 tx=0x7f103c004730 comp rx=0 tx=0).stop 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 shutdown_connections 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6c070 0x5619f6f65dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f6f66310 0x5619f6fb2c90 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f8a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 --2- 172.21.15.53:0/3969274852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f66d70 0x5619f6f8a970 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f1042000640 0 client.5349 ms_handle_reset on v2:172.21.15.53:6833/2729092299 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 >> 172.21.15.53:0/3969274852 conn(0x5619f6f2dc70 msgr2=0x5619f6f6c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 shutdown_connections 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.247+0000 7f105e877740 1 -- 172.21.15.53:0/3969274852 wait complete. 2024-01-16T15:01:26.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.248+0000 7f105e877740 20 client.5349 trim_cache size 0 max 16384 2024-01-16T15:01:26.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.264+0000 7f105e877740 1 Processor -- start 2024-01-16T15:01:26.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.264+0000 7f105e877740 1 -- start start 2024-01-16T15:01:26.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.265+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f66d70 0x5619f6f8a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.265+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f6adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.265+0000 7f105e877740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.265+0000 7f105e877740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6c35f20 con 0x5619f6f6c070 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.265+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6f6dc80 con 0x5619f6f6b7b0 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.265+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619f6fafb10 con 0x5619f6f66d70 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47850/0 (socket says 172.21.15.53:47850) 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 -- 172.21.15.53:0/2393250887 learned_addr learned my addr 172.21.15.53:0/2393250887 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059200640 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f66d70 0x5619f6f8a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 -- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f66d70 msgr2=0x5619f6f8a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f66d70 0x5619f6f8a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1053e00640 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f6adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 -- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 msgr2=0x5619f6f6adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f6adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 -- 172.21.15.53:0/2393250887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619f6e6eb80 con 0x5619f6f6c070 2024-01-16T15:01:26.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059200640 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f66d70 0x5619f6f8a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:26.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.266+0000 7f1059c00640 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92050 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f10540d3060 tx=0x7f104c009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f494621aa121dfc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f104a000640 1 -- 172.21.15.53:0/2393250887 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c004230 con 0x5619f6f6c070 2024-01-16T15:01:26.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f104a000640 1 -- 172.21.15.53:0/2393250887 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c004390 con 0x5619f6f6c070 2024-01-16T15:01:26.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f104a000640 1 -- 172.21.15.53:0/2393250887 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c0119d0 con 0x5619f6f6c070 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 -- 172.21.15.53:0/2393250887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 msgr2=0x5619f6f92050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92050 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f10540d3060 tx=0x7f104c009c00 comp rx=0 tx=0).stop 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 -- 172.21.15.53:0/2393250887 shutdown_connections 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92050 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f6b7b0 0x5619f6f6adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 --2- 172.21.15.53:0/2393250887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f66d70 0x5619f6f8a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 -- 172.21.15.53:0/2393250887 >> 172.21.15.53:0/2393250887 conn(0x5619f6f2dc70 msgr2=0x5619f6fb51d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 -- 172.21.15.53:0/2393250887 shutdown_connections 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.267+0000 7f105e877740 1 -- 172.21.15.53:0/2393250887 wait complete. 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.268+0000 7f105e877740 1 -- start start 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.268+0000 7f105e877740 1 Processor -- start 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f66d70 0x5619f6f8af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6fadbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f701c8a0 con 0x5619f6f6c070 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5619f6f6cd40 con 0x5619f6f66d70 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5619f6f6d000 con 0x5619f6f6b7b0 2024-01-16T15:01:26.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.269+0000 7f105e877740 10 client.0 _ll_register_callbacks cb 0x7f103c006dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47862/0 (socket says 172.21.15.53:47862) 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 -- 172.21.15.53:0/3216759132 learned_addr learned my addr 172.21.15.53:0/3216759132 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059200640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f66d70 0x5619f6f8af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1053e00640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6fadbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 -- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 msgr2=0x5619f6fadbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6fadbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 -- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f66d70 msgr2=0x5619f6f8af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f66d70 0x5619f6f8af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 -- 172.21.15.53:0/3216759132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c0094a0 con 0x5619f6f6c070 2024-01-16T15:01:26.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1059c00640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f104c018040 tx=0x7f104c009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c21bb8b44b9abde server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.270+0000 7f1042000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f1042000640 1 -- 172.21.15.53:0/3216759132 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c023070 con 0x5619f6f6c070 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f1042000640 1 -- 172.21.15.53:0/3216759132 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f104c028470 con 0x5619f6f6c070 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f104400a480 con 0x5619f6f6c070 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f1042000640 1 -- 172.21.15.53:0/3216759132 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f104c028630 con 0x5619f6f6c070 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 10 client.5355 fetch_fsmap learned FSMap version 16 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x5619f6e94170 con 0x5619f6f6c070 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f1042000640 1 -- 172.21.15.53:0/3216759132 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104400a480 con 0x5619f6f6c070 2024-01-16T15:01:26.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f1042000640 1 -- 172.21.15.53:0/3216759132 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 16) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f104c028790 con 0x5619f6f6c070 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 10 client.5355 fetch_fsmap finished waiting for FSMap version 16 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 10 client.5355 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 20 client.5355 populate_metadata read hostname 'smithi053' 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f701baf0 0x5619f701bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 4 client.5355 mds_command: new command op to 5190 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.271+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 --> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5619f6bd36c0 con 0x5619f701baf0 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.272+0000 7f1059200640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f701baf0 0x5619f701bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.272+0000 7f1059200640 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f701baf0 0x5619f701bed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e0583f8e021e79b server_cookie=d0fafcc057e6f7ff in_seq=0 out_seq=0 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.272+0000 7f1042000640 10 client.5355 ms_handle_connect on v2:172.21.15.53:6833/2729092299 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.272+0000 7f1042000640 1 -- 172.21.15.53:0/3216759132 <== mds.0 v2:172.21.15.53:6833/2729092299 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x5619f6bd36c0 con 0x5619f701baf0 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.272+0000 7f1042000640 10 client.5355 handle_command_reply: tid=0 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.272+0000 7f105e877740 1 client.5355 shutdown 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 msgr2=0x5619f6f92030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92030 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f104c018040 tx=0x7f104c009190 comp rx=0 tx=0).stop 2024-01-16T15:01:26.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 shutdown_connections 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x5619f6f6c070 0x5619f6f92030 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5619f6f6b7b0 0x5619f6fadbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.53:6833/2729092299,v1:172.21.15.53:6835/2729092299] conn(0x5619f701baf0 0x5619f701bed0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 --2- 172.21.15.53:0/3216759132 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5619f6f66d70 0x5619f6f8af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 >> 172.21.15.53:0/3216759132 conn(0x5619f6f2dc70 msgr2=0x5619f6f8a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f1042000640 0 client.5355 ms_handle_reset on v2:172.21.15.53:6833/2729092299 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 shutdown_connections 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 1 -- 172.21.15.53:0/3216759132 wait complete. 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.273+0000 7f105e877740 20 client.5355 trim_cache size 0 max 16384 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:01:26.258 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5190, 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 6.8065533909999996, 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 16, 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 35, 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 35, 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 15.628987517000001 2024-01-16T15:01:26.259 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f102c06b2c0 msgr2=0x7f102c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f102c06b2c0 0x7f102c06d780 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f103c009990 tx=0x7f103c008040 comp rx=0 tx=0).stop 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 msgr2=0x7f105415d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 0x7f105415d980 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f104402cb70 tx=0x7f10540b4860 comp rx=0 tx=0).stop 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 shutdown_connections 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f102c06b2c0 0x7f102c06d780 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10541508d0 0x7f105415dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1054150470 0x7f105415d980 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 --2- 172.21.15.53:0/1816118888 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10340047c0 0x7f10540cce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 >> 172.21.15.53:0/1816118888 conn(0x7f10540d3090 msgr2=0x7f105406c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 shutdown_connections 2024-01-16T15:01:26.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.274+0000 7f105a600640 1 -- 172.21.15.53:0/1816118888 wait complete. 2024-01-16T15:01:26.271 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:01:26.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.498+0000 7fd3fd800640 1 Processor -- start 2024-01-16T15:01:26.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.498+0000 7fd3fd800640 1 -- start start 2024-01-16T15:01:26.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.500+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 0x7fd3f80b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.500+0000 7fd3fd800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 0x7fd3f8071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.500+0000 7fd3fd800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f80702e0 con 0x7fd3f80b3650 2024-01-16T15:01:26.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.500+0000 7fd3fd800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f80705c0 con 0x7fd3f8162810 2024-01-16T15:01:26.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.500+0000 7fd3fd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f80708a0 con 0x7fd3f8162430 2024-01-16T15:01:26.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.501+0000 7fd3f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 0x7fd3f80b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.501+0000 7fd3efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 0x7fd3f8071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.501+0000 7fd3f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 0x7fd3f80b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:43288/0 (socket says 172.21.15.53:43288) 2024-01-16T15:01:26.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.501+0000 7fd3efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 0x7fd3f8071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:34378/0 (socket says 172.21.15.53:34378) 2024-01-16T15:01:26.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.501+0000 7fd3f7e00640 1 -- 172.21.15.53:0/93584366 learned_addr learned my addr 172.21.15.53:0/93584366 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.502+0000 7fd3f7e00640 1 -- 172.21.15.53:0/93584366 >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 msgr2=0x7fd3f8071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.502+0000 7fd3f7e00640 1 --2- 172.21.15.53:0/93584366 >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 0x7fd3f8071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.502+0000 7fd3f7e00640 1 -- 172.21.15.53:0/93584366 >> v1:172.21.15.146:6789/0 conn(0x7fd3f80b3650 legacy=0x7fd3f81650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.502+0000 7fd3f7e00640 1 -- 172.21.15.53:0/93584366 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f8070b80 con 0x7fd3f8162430 2024-01-16T15:01:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.502+0000 7fd3efe00640 1 --2- 172.21.15.53:0/93584366 >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 0x7fd3f8071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:01:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.503+0000 7fd3f7e00640 1 --2- 172.21.15.53:0/93584366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 0x7fd3f80b3110 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009870 tx=0x7fd3e802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3391c207fe6ce682 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3f7400640 1 -- 172.21.15.53:0/93584366 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8030840 con 0x7fd3f8162430 2024-01-16T15:01:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3f7400640 1 -- 172.21.15.53:0/93584366 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e80309a0 con 0x7fd3f8162430 2024-01-16T15:01:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3f7400640 1 -- 172.21.15.53:0/93584366 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e8030d30 con 0x7fd3f8162430 2024-01-16T15:01:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 -- 172.21.15.53:0/93584366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 msgr2=0x7fd3f80b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 --2- 172.21.15.53:0/93584366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 0x7fd3f80b3110 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8009870 tx=0x7fd3e802b980 comp rx=0 tx=0).stop 2024-01-16T15:01:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 -- 172.21.15.53:0/93584366 shutdown_connections 2024-01-16T15:01:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 --2- 172.21.15.53:0/93584366 >> v2:172.21.15.146:3300/0 conn(0x7fd3f8162810 0x7fd3f8071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 --2- 172.21.15.53:0/93584366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f8162430 0x7fd3f80b3110 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 -- 172.21.15.53:0/93584366 >> 172.21.15.53:0/93584366 conn(0x7fd3f806a800 msgr2=0x7fd3f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 -- 172.21.15.53:0/93584366 shutdown_connections 2024-01-16T15:01:26.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.504+0000 7fd3fd800640 1 -- 172.21.15.53:0/93584366 wait complete. 2024-01-16T15:01:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.506+0000 7fd3fd800640 1 Processor -- start 2024-01-16T15:01:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.506+0000 7fd3fd800640 1 -- start start 2024-01-16T15:01:26.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.507+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f80b3650 0x7fd3f814b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 0x7fd3f8156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3fd800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f81573b0 0x7fd3f81f8520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3fd800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f8157840 con 0x7fd3f8162430 2024-01-16T15:01:26.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3fd800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81f9ea0 con 0x7fd3f81573b0 2024-01-16T15:01:26.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3fd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f81f9290 con 0x7fd3f80b3650 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 0x7fd3f8156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 0x7fd3f8156e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47870/0 (socket says 172.21.15.53:47870) 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 -- 172.21.15.53:0/527897228 learned_addr learned my addr 172.21.15.53:0/527897228 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3f7e00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f80b3650 0x7fd3f814b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3fce00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f81573b0 0x7fd3f81f8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 -- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f80b3650 msgr2=0x7fd3f814b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f80b3650 0x7fd3f814b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 -- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f81573b0 msgr2=0x7fd3f81f8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f81573b0 0x7fd3f81f8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.508+0000 7fd3efe00640 1 -- 172.21.15.53:0/527897228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0009400 con 0x7fd3f8162430 2024-01-16T15:01:26.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3fce00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f81573b0 0x7fd3f81f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:26.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3efe00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 0x7fd3f8156e70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000e990 tx=0x7fd3e000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000c9e0 con 0x7fd3f8162430 2024-01-16T15:01:26.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e000cb40 con 0x7fd3f8162430 2024-01-16T15:01:26.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e000ced0 con 0x7fd3f8162430 2024-01-16T15:01:26.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e80094d0 con 0x7fd3f8162430 2024-01-16T15:01:26.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.509+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f81f9850 con 0x7fd3f8162430 2024-01-16T15:01:26.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.510+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f8067450 con 0x7fd3f8162430 2024-01-16T15:01:26.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.511+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd3e0013470 con 0x7fd3f8162430 2024-01-16T15:01:26.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.512+0000 7fd3f4c00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3d006b2c0 0x7fd3d006d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:26.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.512+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3e0089f10 con 0x7fd3f8162430 2024-01-16T15:01:26.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.513+0000 7fd3f7e00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3d006b2c0 0x7fd3d006d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:26.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.514+0000 7fd3f7e00640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3d006b2c0 0x7fd3d006d780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8036000 tx=0x7fd3e8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:26.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.517+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3e0054cf0 con 0x7fd3f8162430 2024-01-16T15:01:26.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.748+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fd3f81f9fe0 con 0x7fd3f8162430 2024-01-16T15:01:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.749+0000 7fd3f4c00640 1 -- 172.21.15.53:0/527897228 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fd3e0058310 con 0x7fd3f8162430 2024-01-16T15:01:26.730 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.751+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3d006b2c0 msgr2=0x7fd3d006d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.751+0000 7fd3fd800640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3d006b2c0 0x7fd3d006d780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fd3e8036000 tx=0x7fd3e8008030 comp rx=0 tx=0).stop 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.751+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 msgr2=0x7fd3f8156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.751+0000 7fd3fd800640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 0x7fd3f8156e70 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fd3e000e990 tx=0x7fd3e000ee50 comp rx=0 tx=0).stop 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 shutdown_connections 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3d006b2c0 0x7fd3d006d780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f81573b0 0x7fd3f81f8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f8162430 0x7fd3f8156e70 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 --2- 172.21.15.53:0/527897228 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f80b3650 0x7fd3f814b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:26.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 >> 172.21.15.53:0/527897228 conn(0x7fd3f806a800 msgr2=0x7fd3f80b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:26.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 shutdown_connections 2024-01-16T15:01:26.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:26.752+0000 7fd3fd800640 1 -- 172.21.15.53:0/527897228 wait complete. 2024-01-16T15:01:26.752 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:01:26.752 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:01:26.753 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:01:26.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:26.753 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:01:26.809 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42563sec preferred_lft 42563sec 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.810 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:01:26.811 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:26.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:26.812 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:01:26.908 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:01:26.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:26.914 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:01:26.967 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:01:26.967 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:01:26.967 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:01:26.967 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:01:26.967 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:01:26.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:26.967 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:01:26.968 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:01:26.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:01:26.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:01:26.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:01:26.968 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:01:27.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:27.160 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:01:27.212 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:01:27.212 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:01:27.213 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:01:27.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:27.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:01:27.306 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:01:27.306 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:01:27.306 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:27.362 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:01:27.362 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:01:27.362 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:27.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:01:27.517 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:01:27.575 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:01:27.576 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:01:27.577 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:01:27.578 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:01:27.578 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:01:27.578 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:01:27.578 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:01:27.578 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:01:27.579 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:27.579 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:01:27.636 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:01:27.636 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:01:27.642 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:01:27.717 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:01:27.759 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:01:27.760 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:01:27.761 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:01:27.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:27.762 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:01:27.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:01:27.844+0000 7fc4f124b480 -1 init, newargv = 0x5566d1f4b9f0 newargc=15 2024-01-16T15:01:27.825 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[48537]: starting ceph client 2024-01-16T15:01:27.833 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[48537]: starting fuse 2024-01-16T15:01:27.850 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:01:27.850 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:01:27.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:01:27.850 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:01:27.850 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:01:27.850 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:01:27.850 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:01:27.851 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:01:27.852 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:01:28.061 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.48537.asok 2024-01-16T15:01:28.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:01:28.079 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.48537.asok 2024-01-16T15:01:28.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:28.079 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.48537.asok status 2024-01-16T15:01:28.332 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:01:28.332 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:01:28.332 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:01:28.332 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:01:28.332 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:01:28.332 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "48537", 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5364, 2024-01-16T15:01:28.333 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 5364 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 4161577660 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:28.334 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 4161577660 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.5364 192.168.0.1:0/4161577660", 2024-01-16T15:01:28.335 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/4161577660", 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 16, 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 35, 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:01:28.336 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:01:28.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:28.348 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:01:28.348 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:28.405 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:01:28.405 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:28.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:28.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:01:28.504 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:01:28.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.744+0000 7fa9e7200640 1 Processor -- start 2024-01-16T15:01:28.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.745+0000 7fa9e7200640 1 -- start start 2024-01-16T15:01:28.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.746+0000 7fa9e7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9e0162810 0x7fa9e0071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.746+0000 7fa9e7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e0071c10 con 0x7fa9e0071780 2024-01-16T15:01:28.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.746+0000 7fa9e7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e00702e0 con 0x7fa9e0162430 2024-01-16T15:01:28.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.746+0000 7fa9e7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9e00705c0 con 0x7fa9e0162810 2024-01-16T15:01:28.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e6800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa9e0071780 0x7fa9e006c8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44596/0 (socket says 172.21.15.53:44596) 2024-01-16T15:01:28.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e6800640 1 -- 172.21.15.53:0/2875360217 learned_addr learned my addr 172.21.15.53:0/2875360217 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:28.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e5400640 1 --2- 172.21.15.53:0/2875360217 >> v2:172.21.15.146:3300/0 conn(0x7fa9e0162810 0x7fa9e0071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442671133 0 0) 0x7fa9e0071c10 con 0x7fa9e0071780 2024-01-16T15:01:28.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0003520 con 0x7fa9e0071780 2024-01-16T15:01:28.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2062101100 0 0) 0x7fa9c0003520 con 0x7fa9e0071780 2024-01-16T15:01:28.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 >> v2:172.21.15.146:3300/0 conn(0x7fa9e0162810 msgr2=0x7fa9e0071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:28.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e4a00640 1 --2- 172.21.15.53:0/2875360217 >> v2:172.21.15.146:3300/0 conn(0x7fa9e0162810 0x7fa9e0071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.747+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 >> v1:172.21.15.146:6789/0 conn(0x7fa9e0162430 legacy=0x7fa9e00b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e00708a0 con 0x7fa9e0071780 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fa9d80019f0 con 0x7fa9e0071780 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 >> v1:172.21.15.53:6789/0 conn(0x7fa9e0071780 legacy=0x7fa9e006c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c00039c0 0x7fa9c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9c00047c0 0x7fa9c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c0005180 0x7fa9c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e00705c0 con 0x7fa9c00039c0 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c000c080 con 0x7fa9c0005180 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c000c240 con 0x7fa9c00047c0 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e5e00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c00039c0 0x7fa9c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e5400640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9c00047c0 0x7fa9c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e5e00640 1 -- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9c00047c0 msgr2=0x7fa9c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e5e00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9c00047c0 0x7fa9c0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e6800640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c0005180 0x7fa9c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.748+0000 7fa9e5e00640 1 -- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c0005180 msgr2=0x7fa9c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.749+0000 7fa9e5e00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c0005180 0x7fa9c000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.749+0000 7fa9e5e00640 1 -- 172.21.15.53:0/2875360217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c0003520 con 0x7fa9c00039c0 2024-01-16T15:01:28.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.749+0000 7fa9e6800640 1 -- 172.21.15.53:0/2875360217 reap_dead start 2024-01-16T15:01:28.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.750+0000 7fa9e5e00640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c00039c0 0x7fa9c0003de0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0001750 tx=0x7fa9c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c53fa8e497ece68 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d00052d0 con 0x7fa9c00039c0 2024-01-16T15:01:28.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d002e050 con 0x7fa9c00039c0 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e4a00640 1 -- 172.21.15.53:0/2875360217 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0003830 con 0x7fa9c00039c0 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e7200640 1 -- 172.21.15.53:0/2875360217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c00039c0 msgr2=0x7fa9c0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e7200640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c00039c0 0x7fa9c0003de0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0001750 tx=0x7fa9c00057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e7200640 1 -- 172.21.15.53:0/2875360217 shutdown_connections 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e7200640 1 --2- 172.21.15.53:0/2875360217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9c00039c0 0x7fa9c0003de0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.751+0000 7fa9e7200640 1 -- 172.21.15.53:0/2875360217 >> 172.21.15.53:0/2875360217 conn(0x7fa9e006a800 msgr2=0x7fa9e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.752+0000 7fa9e7200640 1 -- 172.21.15.53:0/2875360217 shutdown_connections 2024-01-16T15:01:28.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.752+0000 7fa9e7200640 1 -- 172.21.15.53:0/2875360217 wait complete. 2024-01-16T15:01:28.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.753+0000 7fa9e7200640 1 Processor -- start 2024-01-16T15:01:28.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.754+0000 7fa9e7200640 1 -- start start 2024-01-16T15:01:28.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.754+0000 7fa9e7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c00039c0 0x7fa9e015a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9e0158df0 0x7fa9e01f7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 0x7fa9e0143fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e015a000 con 0x7fa9e0143b20 2024-01-16T15:01:28.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e01f7f60 con 0x7fa9c00039c0 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9e01593f0 con 0x7fa9e0158df0 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 0x7fa9e0143fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 0x7fa9e0143fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47894/0 (socket says 172.21.15.53:47894) 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 -- 172.21.15.53:0/318306652 learned_addr learned my addr 172.21.15.53:0/318306652 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e5400640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9e0158df0 0x7fa9e01f7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 -- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9e0158df0 msgr2=0x7fa9e01f7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9e0158df0 0x7fa9e01f7a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 -- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c00039c0 msgr2=0x7fa9e015a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c00039c0 0x7fa9e015a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.755+0000 7fa9e6800640 1 -- 172.21.15.53:0/318306652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e01596d0 con 0x7fa9e0143b20 2024-01-16T15:01:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.756+0000 7fa9e6800640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 0x7fa9e0143fa0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00b2bc0 tx=0x7fa9d800df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.756+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800b860 con 0x7fa9e0143b20 2024-01-16T15:01:28.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.756+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9d800b9e0 con 0x7fa9e0143b20 2024-01-16T15:01:28.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.756+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d800bd90 con 0x7fa9e0143b20 2024-01-16T15:01:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.756+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e01599b0 con 0x7fa9e0143b20 2024-01-16T15:01:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.756+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e01444e0 con 0x7fa9e0143b20 2024-01-16T15:01:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.757+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e0067450 con 0x7fa9e0143b20 2024-01-16T15:01:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.758+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fa9d8007070 con 0x7fa9e0143b20 2024-01-16T15:01:28.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.759+0000 7fa9de000640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa9b806b2f0 0x7fa9b806d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:28.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.759+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9d8089e70 con 0x7fa9e0143b20 2024-01-16T15:01:28.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.760+0000 7fa9e5e00640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa9b806b2f0 0x7fa9b806d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:28.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.761+0000 7fa9e5e00640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa9b806b2f0 0x7fa9b806d7b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0033000 tx=0x7fa9d002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.764+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa9d8054d70 con 0x7fa9e0143b20 2024-01-16T15:01:28.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.998+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa9e00b2eb0 con 0x7fa9e0143b20 2024-01-16T15:01:28.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:28.998+0000 7fa9de000640 1 -- 172.21.15.53:0/318306652 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa9d8058390 con 0x7fa9e0143b20 2024-01-16T15:01:28.979 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa9b806b2f0 msgr2=0x7fa9b806d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa9b806b2f0 0x7fa9b806d7b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0033000 tx=0x7fa9d002f000 comp rx=0 tx=0).stop 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 msgr2=0x7fa9e0143fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 0x7fa9e0143fa0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa9e00b2bc0 tx=0x7fa9d800df50 comp rx=0 tx=0).stop 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 shutdown_connections 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa9b806b2f0 0x7fa9b806d7b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9e0143b20 0x7fa9e0143fa0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa9e0158df0 0x7fa9e01f7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 --2- 172.21.15.53:0/318306652 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9c00039c0 0x7fa9e015a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 >> 172.21.15.53:0/318306652 conn(0x7fa9e006a800 msgr2=0x7fa9e00b1030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:28.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 shutdown_connections 2024-01-16T15:01:28.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:29.001+0000 7fa9e7200640 1 -- 172.21.15.53:0/318306652 wait complete. 2024-01-16T15:01:29.000 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:01:29.000 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:01:29.000 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:01:29.000 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:01:29.001 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:29.001 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:01:29.001 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:01:29.001 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:01:29.001 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:01:29.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.001 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:01:29.059 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42441sec preferred_lft 42441sec 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:01:29.060 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:01:29.061 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:01:29.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.062 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:01:29.147 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:01:29.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.152 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:01:29.204 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:01:29.204 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:01:29.204 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:01:29.204 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:01:29.204 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:01:29.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:01:29.205 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:01:29.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.376 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:01:29.423 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:01:29.423 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:01:29.423 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:01:29.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.423 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:01:29.515 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:01:29.515 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:01:29.515 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:29.567 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:01:29.568 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:01:29.568 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:29.618 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:01:29.707 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:01:29.759 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:01:29.760 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:01:29.761 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:01:29.762 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:01:29.762 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:01:29.762 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:01:29.762 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:01:29.762 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:01:29.762 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:01:29.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.763 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:01:29.809 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:01:29.809 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:01:29.813 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:01:29.884 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:01:29.925 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:01:29.926 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:01:29.927 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:01:29.927 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:01:29.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:29.927 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:01:29.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:01:30.007+0000 7f6775a41480 -1 init, newargv = 0x564df02c1c00 newargc=15 2024-01-16T15:01:29.987 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[42465]: starting ceph client 2024-01-16T15:01:30.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[42465]: starting fuse 2024-01-16T15:01:30.010 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:01:30.010 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:01:30.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:01:30.010 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:01:30.010 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:01:30.010 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:01:30.011 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:01:30.012 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:01:30.012 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:01:30.012 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:01:30.012 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:01:30.012 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:01:30.012 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:01:30.210 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.42465.asok 2024-01-16T15:01:30.215 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:01:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:01:30.228 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.42465.asok 2024-01-16T15:01:30.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:30.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.42465.asok status 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "42465", 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:01:30.464 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5134, 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 5134 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1734829176 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:01:30.465 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1734829176 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.5134 192.168.0.1:0/1734829176", 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/1734829176", 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 16, 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 35, 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:01:30.466 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:01:30.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:30.479 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:01:30.479 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:30.534 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:01:30.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:30.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:30.534 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:01:30.626 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:01:30.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.882+0000 7f2c89400640 1 Processor -- start 2024-01-16T15:01:30.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.882+0000 7f2c89400640 1 -- start start 2024-01-16T15:01:30.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.883+0000 7f2c89400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2c8415afb0 0x7f2c8413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.883+0000 7f2c89400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8413fae0 0x7f2c840b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.883+0000 7f2c89400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c8413ff30 con 0x7f2c8413cfa0 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.883+0000 7f2c89400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c840cdc40 con 0x7f2c8415afb0 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.883+0000 7f2c89400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840cdec0 con 0x7f2c8413fae0 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2c8415afb0 0x7f2c8413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2c8413cfa0 0x7f2c840cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44606/0 (socket says 172.21.15.53:44606) 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83e00640 1 -- 172.21.15.53:0/2259278322 learned_addr learned my addr 172.21.15.53:0/2259278322 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c88a00640 1 --2- 172.21.15.53:0/2259278322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8413fae0 0x7f2c840b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83400640 1 -- 172.21.15.53:0/2259278322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8413fae0 msgr2=0x7f2c840b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:30.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83400640 1 --2- 172.21.15.53:0/2259278322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8413fae0 0x7f2c840b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83400640 1 -- 172.21.15.53:0/2259278322 >> v1:172.21.15.53:6789/0 conn(0x7f2c8413cfa0 legacy=0x7f2c840cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.884+0000 7f2c83400640 1 -- 172.21.15.53:0/2259278322 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c840ce1a0 con 0x7f2c8415afb0 2024-01-16T15:01:30.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.885+0000 7f2c83400640 1 --2- 172.21.15.53:0/2259278322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8415afb0 0x7f2c8413f5a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0097a0 tx=0x7f2c6c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12aa4c778a5b95d7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.886+0000 7f2c82a00640 1 -- 172.21.15.53:0/2259278322 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c031780 con 0x7f2c8415afb0 2024-01-16T15:01:30.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.886+0000 7f2c82a00640 1 -- 172.21.15.53:0/2259278322 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c0318e0 con 0x7f2c8415afb0 2024-01-16T15:01:30.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.886+0000 7f2c82a00640 1 -- 172.21.15.53:0/2259278322 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c031c70 con 0x7f2c8415afb0 2024-01-16T15:01:30.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.886+0000 7f2c89400640 1 -- 172.21.15.53:0/2259278322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8415afb0 msgr2=0x7f2c8413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.886+0000 7f2c89400640 1 --2- 172.21.15.53:0/2259278322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8415afb0 0x7f2c8413f5a0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c0097a0 tx=0x7f2c6c02b820 comp rx=0 tx=0).stop 2024-01-16T15:01:30.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.887+0000 7f2c89400640 1 -- 172.21.15.53:0/2259278322 shutdown_connections 2024-01-16T15:01:30.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.887+0000 7f2c89400640 1 --2- 172.21.15.53:0/2259278322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8413fae0 0x7f2c840b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.887+0000 7f2c89400640 1 --2- 172.21.15.53:0/2259278322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8415afb0 0x7f2c8413f5a0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.887+0000 7f2c89400640 1 -- 172.21.15.53:0/2259278322 >> 172.21.15.53:0/2259278322 conn(0x7f2c8406a800 msgr2=0x7f2c8406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:30.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.887+0000 7f2c89400640 1 -- 172.21.15.53:0/2259278322 shutdown_connections 2024-01-16T15:01:30.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.887+0000 7f2c89400640 1 -- 172.21.15.53:0/2259278322 wait complete. 2024-01-16T15:01:30.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.888+0000 7f2c89400640 1 Processor -- start 2024-01-16T15:01:30.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.889+0000 7f2c89400640 1 -- start start 2024-01-16T15:01:30.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.889+0000 7f2c89400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c8413cfa0 0x7f2c841536a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.889+0000 7f2c89400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 0x7f2c841398b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c89400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8415afb0 0x7f2c8413b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c89400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840adb80 con 0x7f2c8413fae0 2024-01-16T15:01:30.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c89400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c840ade60 con 0x7f2c8413cfa0 2024-01-16T15:01:30.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c89400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c84137df0 con 0x7f2c8415afb0 2024-01-16T15:01:30.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 0x7f2c841398b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 0x7f2c841398b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47914/0 (socket says 172.21.15.53:47914) 2024-01-16T15:01:30.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 -- 172.21.15.53:0/4009375179 learned_addr learned my addr 172.21.15.53:0/4009375179 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:30.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83e00640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c8413cfa0 0x7f2c841536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c88a00640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8415afb0 0x7f2c8413b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 -- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8415afb0 msgr2=0x7f2c8413b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8415afb0 0x7f2c8413b2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 -- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c8413cfa0 msgr2=0x7f2c841536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c8413cfa0 0x7f2c841536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83400640 1 -- 172.21.15.53:0/4009375179 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c009400 con 0x7f2c8413fae0 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.890+0000 7f2c83e00640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c8413cfa0 0x7f2c841536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.891+0000 7f2c83400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 0x7f2c841398b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c034040 tx=0x7f2c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.891+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c032820 con 0x7f2c8413fae0 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.891+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6c0329a0 con 0x7f2c8413fae0 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.891+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6c032d50 con 0x7f2c8413fae0 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.891+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c841380d0 con 0x7f2c8413fae0 2024-01-16T15:01:30.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.892+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c8413c020 con 0x7f2c8413fae0 2024-01-16T15:01:30.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.893+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c84135270 con 0x7f2c8413fae0 2024-01-16T15:01:30.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.894+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f2c6c04e440 con 0x7f2c8413fae0 2024-01-16T15:01:30.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.895+0000 7f2c7be00640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c5c06b2c0 0x7f2c5c06d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:30.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.895+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2c6c0abe30 con 0x7f2c8413fae0 2024-01-16T15:01:30.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.896+0000 7f2c83e00640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c5c06b2c0 0x7f2c5c06d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:30.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.897+0000 7f2c83e00640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c5c06b2c0 0x7f2c5c06d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2c74005880 tx=0x7f2c74009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:30.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:30.900+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c84135270 con 0x7f2c8413fae0 2024-01-16T15:01:31.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:31.140+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2c84135270 con 0x7f2c8413fae0 2024-01-16T15:01:32.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.032+0000 7f2c7be00640 1 -- 172.21.15.53:0/4009375179 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 136+0+0 (secure 0 0 0) 0x7f2c84135270 con 0x7f2c8413fae0 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c5c06b2c0 msgr2=0x7f2c5c06d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c5c06b2c0 0x7f2c5c06d780 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2c74005880 tx=0x7f2c74009040 comp rx=0 tx=0).stop 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 msgr2=0x7f2c841398b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 0x7f2c841398b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c034040 tx=0x7f2c6c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 shutdown_connections 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c5c06b2c0 0x7f2c5c06d780 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c8415afb0 0x7f2c8413b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c8413fae0 0x7f2c841398b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.036+0000 7f2c89400640 1 --2- 172.21.15.53:0/4009375179 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c8413cfa0 0x7f2c841536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.037+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 >> 172.21.15.53:0/4009375179 conn(0x7f2c8406a800 msgr2=0x7f2c840cd530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.037+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 shutdown_connections 2024-01-16T15:01:32.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.037+0000 7f2c89400640 1 -- 172.21.15.53:0/4009375179 wait complete. 2024-01-16T15:01:32.042 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:01:32.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.286+0000 7fd09f200640 1 Processor -- start 2024-01-16T15:01:32.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.286+0000 7fd09f200640 1 -- start start 2024-01-16T15:01:32.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.287+0000 7fd09f200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0a013cfa0 0x7fd0a015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013fcb0 0x7fd0a00b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0a00cdec0 con 0x7fd0a00cd530 2024-01-16T15:01:32.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09f200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0a00ce1a0 con 0x7fd0a013cfa0 2024-01-16T15:01:32.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a00ce480 con 0x7fd0a013fcb0 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0a013cfa0 0x7fd0a015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0a013cfa0 0x7fd0a015afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55640/0 (socket says 172.21.15.53:55640) 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09de00640 1 -- 172.21.15.53:0/2000597073 learned_addr learned my addr 172.21.15.53:0/2000597073 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09e800640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013fcb0 0x7fd0a00b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09de00640 1 -- 172.21.15.53:0/2000597073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013fcb0 msgr2=0x7fd0a00b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.288+0000 7fd09de00640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013fcb0 0x7fd0a00b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.289+0000 7fd09de00640 1 -- 172.21.15.53:0/2000597073 >> v1:172.21.15.146:6789/0 conn(0x7fd0a00cd530 legacy=0x7fd0a013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.289+0000 7fd09de00640 1 -- 172.21.15.53:0/2000597073 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a00ce760 con 0x7fd0a013cfa0 2024-01-16T15:01:32.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.289+0000 7fd09e800640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013fcb0 0x7fd0a00b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:32.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.290+0000 7fd09de00640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a013cfa0 0x7fd0a015afb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd090009870 tx=0x7fd09002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57fe49f68f98290 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:32.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.290+0000 7fd09d400640 1 -- 172.21.15.53:0/2000597073 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0900317c0 con 0x7fd0a013cfa0 2024-01-16T15:01:32.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.290+0000 7fd09d400640 1 -- 172.21.15.53:0/2000597073 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090031920 con 0x7fd0a013cfa0 2024-01-16T15:01:32.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.290+0000 7fd09d400640 1 -- 172.21.15.53:0/2000597073 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090031cb0 con 0x7fd0a013cfa0 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 -- 172.21.15.53:0/2000597073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a013cfa0 msgr2=0x7fd0a015afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a013cfa0 0x7fd0a015afb0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fd090009870 tx=0x7fd09002b860 comp rx=0 tx=0).stop 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 -- 172.21.15.53:0/2000597073 shutdown_connections 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013fcb0 0x7fd0a00b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 --2- 172.21.15.53:0/2000597073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a013cfa0 0x7fd0a015afb0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 -- 172.21.15.53:0/2000597073 >> 172.21.15.53:0/2000597073 conn(0x7fd0a006a800 msgr2=0x7fd0a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 -- 172.21.15.53:0/2000597073 shutdown_connections 2024-01-16T15:01:32.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.291+0000 7fd09f200640 1 -- 172.21.15.53:0/2000597073 wait complete. 2024-01-16T15:01:32.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.293+0000 7fd09f200640 1 Processor -- start 2024-01-16T15:01:32.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.293+0000 7fd09f200640 1 -- start start 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 0x7fd0a0157620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013cfa0 0x7fd0a0153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0a013fcb0 0x7fd0a013a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a0153e80 con 0x7fd0a00cd530 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a01565a0 con 0x7fd0a013fcb0 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a0156880 con 0x7fd0a013cfa0 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 0x7fd0a0157620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 0x7fd0a0157620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55646/0 (socket says 172.21.15.53:55646) 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09de00640 1 -- 172.21.15.53:0/1644056096 learned_addr learned my addr 172.21.15.53:0/1644056096 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09e800640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0a013fcb0 0x7fd0a013a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09de00640 1 -- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013cfa0 msgr2=0x7fd0a0153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd097e00640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013cfa0 0x7fd0a0153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.294+0000 7fd09de00640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013cfa0 0x7fd0a0153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09de00640 1 -- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0a013fcb0 msgr2=0x7fd0a013a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09de00640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0a013fcb0 0x7fd0a013a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09de00640 1 -- 172.21.15.53:0/1644056096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0900094d0 con 0x7fd0a00cd530 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09e800640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0a013fcb0 0x7fd0a013a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09de00640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 0x7fd0a0157620 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd0900090b0 tx=0x7fd0900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090032920 con 0x7fd0a00cd530 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd090032a80 con 0x7fd0a00cd530 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd090032e30 con 0x7fd0a00cd530 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0a0156b60 con 0x7fd0a00cd530 2024-01-16T15:01:32.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.295+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0a0157120 con 0x7fd0a00cd530 2024-01-16T15:01:32.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.296+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0a0067450 con 0x7fd0a00cd530 2024-01-16T15:01:32.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.298+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fd09004d3f0 con 0x7fd0a00cd530 2024-01-16T15:01:32.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.298+0000 7fd096000640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd07806b2c0 0x7fd07806d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.299+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd0900abae0 con 0x7fd0a00cd530 2024-01-16T15:01:32.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.299+0000 7fd097e00640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd07806b2c0 0x7fd07806d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.300+0000 7fd097e00640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd07806b2c0 0x7fd07806d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd0a006d8e0 tx=0x7fd088009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:32.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.305+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd090076940 con 0x7fd0a00cd530 2024-01-16T15:01:32.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.523+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0a00b2580 con 0x7fd07806b2c0 2024-01-16T15:01:32.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.527+0000 7fd096000640 1 -- 172.21.15.53:0/1644056096 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fd088003a20 con 0x7fd07806b2c0 2024-01-16T15:01:32.507 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:01:32.507 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:01:32.507 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:01:32.507 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:01:32.508 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:01:32.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd07806b2c0 msgr2=0x7fd07806d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd07806b2c0 0x7fd07806d780 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd0a006d8e0 tx=0x7fd088009040 comp rx=0 tx=0).stop 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 msgr2=0x7fd0a0157620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 0x7fd0a0157620 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fd0900090b0 tx=0x7fd0900046c0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 shutdown_connections 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd07806b2c0 0x7fd07806d780 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0a013fcb0 0x7fd0a013a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.529+0000 7fd09f200640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0a013cfa0 0x7fd0a0153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.530+0000 7fd09f200640 1 --2- 172.21.15.53:0/1644056096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0a00cd530 0x7fd0a0157620 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.530+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 >> 172.21.15.53:0/1644056096 conn(0x7fd0a006a800 msgr2=0x7fd0a013c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.530+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 shutdown_connections 2024-01-16T15:01:32.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.530+0000 7fd09f200640 1 -- 172.21.15.53:0/1644056096 wait complete. 2024-01-16T15:01:32.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:01:32.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.776+0000 7fafada00640 1 Processor -- start 2024-01-16T15:01:32.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.776+0000 7fafada00640 1 -- start start 2024-01-16T15:01:32.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.777+0000 7fafada00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fafa80cfa90 0x7fafa80b85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.777+0000 7fafada00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80d09f0 con 0x7fafa80b7d30 2024-01-16T15:01:32.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.777+0000 7fafada00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80d3e70 con 0x7fafa80d1e30 2024-01-16T15:01:32.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.777+0000 7fafada00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80d3b90 con 0x7fafa80cfa90 2024-01-16T15:01:32.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.777+0000 7fafad000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fafa80b7d30 0x7fafa80ce470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:53046/0 (socket says 172.21.15.53:53046) 2024-01-16T15:01:32.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.777+0000 7fafad000640 1 -- 172.21.15.53:0/2896421 learned_addr learned my addr 172.21.15.53:0/2896421 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:32.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa7e00640 1 --2- 172.21.15.53:0/2896421 >> v2:172.21.15.146:3300/0 conn(0x7fafa80cfa90 0x7fafa80b85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342094550 0 0) 0x7fafa80d09f0 con 0x7fafa80b7d30 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf88003520 con 0x7fafa80b7d30 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789762950 0 0) 0x7faf88003520 con 0x7fafa80b7d30 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 >> v2:172.21.15.146:3300/0 conn(0x7fafa80cfa90 msgr2=0x7fafa80b85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 --2- 172.21.15.53:0/2896421 >> v2:172.21.15.146:3300/0 conn(0x7fafa80cfa90 0x7fafa80b85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 >> v1:172.21.15.146:6789/0 conn(0x7fafa80d1e30 legacy=0x7fafa80cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa80b5f60 con 0x7fafa80b7d30 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fafa00019f0 con 0x7fafa80b7d30 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.778+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 >> v1:172.21.15.53:6789/0 conn(0x7fafa80b7d30 legacy=0x7fafa80ce470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa6a00640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf880039c0 0x7faf88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa6a00640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf880047c0 0x7faf88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa6a00640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7faf8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa80d3b90 con 0x7faf88005180 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf8800c060 con 0x7faf880047c0 2024-01-16T15:01:32.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf8800c220 con 0x7faf880039c0 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafad000640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7faf8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa7400640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf880047c0 0x7faf88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafad000640 1 -- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf880039c0 msgr2=0x7faf88003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafad000640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf880039c0 0x7faf88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafad000640 1 -- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf880047c0 msgr2=0x7faf88004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafad000640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf880047c0 0x7faf88004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafad000640 1 -- 172.21.15.53:0/2896421 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf88003520 con 0x7faf88005180 2024-01-16T15:01:32.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.779+0000 7fafa7400640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faf880047c0 0x7faf88004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.780+0000 7fafad000640 1 -- 172.21.15.53:0/2896421 reap_dead start 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafad000640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7faf8800b980 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fafa000a760 tx=0x7faf880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a7bed2cf258605 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0034630 con 0x7faf88005180 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa002f070 con 0x7faf88005180 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafa6a00640 1 -- 172.21.15.53:0/2896421 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa0034a10 con 0x7faf88005180 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafada00640 1 -- 172.21.15.53:0/2896421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 msgr2=0x7faf8800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafada00640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7faf8800b980 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fafa000a760 tx=0x7faf880057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.781+0000 7fafada00640 1 -- 172.21.15.53:0/2896421 shutdown_connections 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.782+0000 7fafada00640 1 --2- 172.21.15.53:0/2896421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7faf8800b980 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.782+0000 7fafada00640 1 -- 172.21.15.53:0/2896421 >> 172.21.15.53:0/2896421 conn(0x7fafa80d3070 msgr2=0x7fafa80d3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.782+0000 7fafada00640 1 -- 172.21.15.53:0/2896421 shutdown_connections 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.782+0000 7fafada00640 1 -- 172.21.15.53:0/2896421 wait complete. 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.783+0000 7fafada00640 1 Processor -- start 2024-01-16T15:01:32.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 -- start start 2024-01-16T15:01:32.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7fafa809cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafa8155800 0x7fafa809ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafa8154540 0x7fafa81549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa8154f10 con 0x7faf88005180 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafa809d6e0 con 0x7fafa8155800 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.784+0000 7fafada00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa809d9c0 con 0x7fafa8154540 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7fafa809cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7fafa809cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55668/0 (socket says 172.21.15.53:55668) 2024-01-16T15:01:32.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 -- 172.21.15.53:0/35426773 learned_addr learned my addr 172.21.15.53:0/35426773 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7400640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafa8155800 0x7fafa809ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 -- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafa8154540 msgr2=0x7fafa81549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafa8154540 0x7fafa81549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 -- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafa8155800 msgr2=0x7fafa809ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafa8155800 0x7fafa809ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7e00640 1 -- 172.21.15.53:0/35426773 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa809dca0 con 0x7faf88005180 2024-01-16T15:01:32.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.785+0000 7fafa7400640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafa8155800 0x7fafa809ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.786+0000 7fafa7e00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7fafa809cde0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faf9800e7a0 tx=0x7faf9800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.786+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9800eeb0 con 0x7faf88005180 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.786+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9800c940 con 0x7faf88005180 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.786+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf9800cd10 con 0x7faf88005180 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.786+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa809df80 con 0x7faf88005180 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.786+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa809d320 con 0x7faf88005180 2024-01-16T15:01:32.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.787+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa8067450 con 0x7faf88005180 2024-01-16T15:01:32.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.788+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7faf98013470 con 0x7faf88005180 2024-01-16T15:01:32.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.789+0000 7faf9fe00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faf8006b310 0x7faf8006d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:32.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.789+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7faf98089c30 con 0x7faf88005180 2024-01-16T15:01:32.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.790+0000 7fafa7400640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faf8006b310 0x7faf8006d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:32.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.791+0000 7fafa7400640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faf8006b310 0x7faf8006d7d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faf90000c00 tx=0x7faf90009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:32.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.794+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faf98054b10 con 0x7faf88005180 2024-01-16T15:01:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.988+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fafa80d3700 con 0x7faf88005180 2024-01-16T15:01:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.989+0000 7faf9fe00640 1 -- 172.21.15.53:0/35426773 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7faf980170c0 con 0x7faf88005180 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faf8006b310 msgr2=0x7faf8006d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faf8006b310 0x7faf8006d7d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7faf90000c00 tx=0x7faf90009040 comp rx=0 tx=0).stop 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 msgr2=0x7fafa809cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7fafa809cde0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faf9800e7a0 tx=0x7faf9800ec60 comp rx=0 tx=0).stop 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 shutdown_connections 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faf8006b310 0x7faf8006d7d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fafa8154540 0x7fafa81549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fafa8155800 0x7fafa809ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 --2- 172.21.15.53:0/35426773 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf88005180 0x7fafa809cde0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 >> 172.21.15.53:0/35426773 conn(0x7fafa80d3070 msgr2=0x7fafa8160a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 shutdown_connections 2024-01-16T15:01:32.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:32.992+0000 7fafada00640 1 -- 172.21.15.53:0/35426773 wait complete. 2024-01-16T15:01:32.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:01:33.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.232+0000 7fba92e00640 1 Processor -- start 2024-01-16T15:01:33.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.232+0000 7fba92e00640 1 -- start start 2024-01-16T15:01:33.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.233+0000 7fba92e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba8c0cc500 0x7fba8c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba92e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 0x7fba8c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba92e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba8c070e50 con 0x7fba8c143c40 2024-01-16T15:01:33.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba92e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba8c0723c0 con 0x7fba8c0cc500 2024-01-16T15:01:33.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba92e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0cced0 con 0x7fba8c06d160 2024-01-16T15:01:33.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba91a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fba8c143c40 0x7fba8c071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:53050/0 (socket says 172.21.15.53:53050) 2024-01-16T15:01:33.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba91a00640 1 -- 172.21.15.53:0/67776378 learned_addr learned my addr 172.21.15.53:0/67776378 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:01:33.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889608522 0 0) 0x7fba8c070e50 con 0x7fba8c143c40 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba6c003520 con 0x7fba8c143c40 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba92400640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 0x7fba8c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.234+0000 7fba91000640 1 --2- 172.21.15.53:0/67776378 >> v2:172.21.15.146:3300/0 conn(0x7fba8c0cc500 0x7fba8c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202010651 0 0) 0x7fba6c003520 con 0x7fba8c143c40 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 msgr2=0x7fba8c06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 0x7fba8c06d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 >> v2:172.21.15.146:3300/0 conn(0x7fba8c0cc500 msgr2=0x7fba8c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 --2- 172.21.15.53:0/67776378 >> v2:172.21.15.146:3300/0 conn(0x7fba8c0cc500 0x7fba8c0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c144020 con 0x7fba8c143c40 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fba7c001690 con 0x7fba8c143c40 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 >> v1:172.21.15.53:6789/0 conn(0x7fba8c143c40 legacy=0x7fba8c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 0x7fba6c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6c003ff0 0x7fba6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c004d00 0x7fba6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0723c0 con 0x7fba6c003ff0 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0cced0 con 0x7fba6c004d00 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba6c00bb90 con 0x7fba8c06d160 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba91000640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6c003ff0 0x7fba6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.235+0000 7fba91a00640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 0x7fba6c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba92400640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c004d00 0x7fba6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba91000640 1 -- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 msgr2=0x7fba6c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba91000640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c06d160 0x7fba6c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba91000640 1 -- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c004d00 msgr2=0x7fba6c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba91000640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c004d00 0x7fba6c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba91000640 1 -- 172.21.15.53:0/67776378 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6c003520 con 0x7fba6c003ff0 2024-01-16T15:01:33.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.236+0000 7fba92400640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c004d00 0x7fba6c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.237+0000 7fba91000640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6c003ff0 0x7fba6c0047c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fba7400ba30 tx=0x7fba6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a289ea458ed6c21 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.237+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74032680 con 0x7fba6c003ff0 2024-01-16T15:01:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.237+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba740327e0 con 0x7fba6c003ff0 2024-01-16T15:01:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.237+0000 7fba8be00640 1 -- 172.21.15.53:0/67776378 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74032b70 con 0x7fba6c003ff0 2024-01-16T15:01:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92e00640 1 -- 172.21.15.53:0/67776378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6c003ff0 msgr2=0x7fba6c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92e00640 1 --2- 172.21.15.53:0/67776378 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6c003ff0 0x7fba6c0047c0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fba7400ba30 tx=0x7fba6c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92400640 1 -- 172.21.15.53:0/67776378 reap_dead start 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92e00640 1 -- 172.21.15.53:0/67776378 shutdown_connections 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92e00640 1 -- 172.21.15.53:0/67776378 >> 172.21.15.53:0/67776378 conn(0x7fba8c06a800 msgr2=0x7fba8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92e00640 1 -- 172.21.15.53:0/67776378 shutdown_connections 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.238+0000 7fba92e00640 1 -- 172.21.15.53:0/67776378 wait complete. 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.240+0000 7fba92e00640 1 Processor -- start 2024-01-16T15:01:33.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.240+0000 7fba92e00640 1 -- start start 2024-01-16T15:01:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba92e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c003ff0 0x7fba8c13c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba92e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 0x7fba8c1e7270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba92e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c0b2f50 0x7fba8c0b33d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba92e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c1e84d0 con 0x7fba8c1e6e10 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba92e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c0b3910 con 0x7fba6c003ff0 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba92e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba8c1f28d0 con 0x7fba8c0b2f50 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba91000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 0x7fba8c1e7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba91000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 0x7fba8c1e7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55696/0 (socket says 172.21.15.53:55696) 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba91000640 1 -- 172.21.15.53:0/863869169 learned_addr learned my addr 172.21.15.53:0/863869169 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.241+0000 7fba91a00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c003ff0 0x7fba8c13c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba92400640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c0b2f50 0x7fba8c0b33d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba91000640 1 -- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c0b2f50 msgr2=0x7fba8c0b33d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba91000640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c0b2f50 0x7fba8c0b33d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba91000640 1 -- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c003ff0 msgr2=0x7fba8c13c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba91000640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c003ff0 0x7fba8c13c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba91000640 1 -- 172.21.15.53:0/863869169 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c1e7950 con 0x7fba8c1e6e10 2024-01-16T15:01:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba91000640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 0x7fba8c1e7270 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fba7402bf00 tx=0x7fba74004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.242+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74004220 con 0x7fba8c1e6e10 2024-01-16T15:01:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.243+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba740043a0 con 0x7fba8c1e6e10 2024-01-16T15:01:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.243+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74030aa0 con 0x7fba8c1e6e10 2024-01-16T15:01:33.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.243+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba8c1e7c30 con 0x7fba8c1e6e10 2024-01-16T15:01:33.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.243+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba8c1e81f0 con 0x7fba8c1e6e10 2024-01-16T15:01:33.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.244+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba8c067450 con 0x7fba8c1e6e10 2024-01-16T15:01:33.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.245+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7fba74030c00 con 0x7fba8c1e6e10 2024-01-16T15:01:33.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.246+0000 7fba89600640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6406b2c0 0x7fba6406d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.246+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fba740ac760 con 0x7fba8c1e6e10 2024-01-16T15:01:33.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.247+0000 7fba91a00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6406b2c0 0x7fba6406d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.248+0000 7fba91a00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6406b2c0 0x7fba6406d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba7c000c00 tx=0x7fba7c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.251+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba74077530 con 0x7fba8c1e6e10 2024-01-16T15:01:33.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.472+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fba8c1f2a10 con 0x7fba6406b2c0 2024-01-16T15:01:33.465 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:01:33.484+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:33.465 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:01:33.484+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:33.465 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:01:33.484+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:33.465 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:01:33.484+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:33.465 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:01:33.484+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:01:33.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.569+0000 7fba89600640 1 -- 172.21.15.53:0/863869169 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fba8c1f2a10 con 0x7fba6406b2c0 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6406b2c0 msgr2=0x7fba6406d780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6406b2c0 0x7fba6406d780 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fba7c000c00 tx=0x7fba7c009040 comp rx=0 tx=0).stop 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 msgr2=0x7fba8c1e7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 0x7fba8c1e7270 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fba7402bf00 tx=0x7fba74004730 comp rx=0 tx=0).stop 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 shutdown_connections 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6406b2c0 0x7fba6406d780 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba8c0b2f50 0x7fba8c0b33d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba8c1e6e10 0x7fba8c1e7270 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 --2- 172.21.15.53:0/863869169 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6c003ff0 0x7fba8c13c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 >> 172.21.15.53:0/863869169 conn(0x7fba8c06a800 msgr2=0x7fba8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 shutdown_connections 2024-01-16T15:01:33.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.572+0000 7fba92e00640 1 -- 172.21.15.53:0/863869169 wait complete. 2024-01-16T15:01:33.574 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:01:33.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.818+0000 7f015b400640 1 Processor -- start 2024-01-16T15:01:33.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.818+0000 7f015b400640 1 -- start start 2024-01-16T15:01:33.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f015b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0154143ae0 0x7f01540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f015b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f015415e3b0 0x7f01540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f015b400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01540cfc30 con 0x7f01540cf7d0 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f015b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01540b1ee0 con 0x7f0154143ae0 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f015b400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01540b2780 con 0x7f015415e3b0 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f0158a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0154143ae0 0x7f01540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f0158a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0154143ae0 0x7f01540cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55708/0 (socket says 172.21.15.53:55708) 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.820+0000 7f014bc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f01540cf7d0 0x7f015415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:53064/0 (socket says 172.21.15.53:53064) 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0158a00640 1 -- 172.21.15.53:0/2262402486 learned_addr learned my addr 172.21.15.53:0/2262402486 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0159400640 1 --2- 172.21.15.53:0/2262402486 >> v2:172.21.15.146:3300/0 conn(0x7f015415e3b0 0x7f01540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0158a00640 1 -- 172.21.15.53:0/2262402486 >> v2:172.21.15.146:3300/0 conn(0x7f015415e3b0 msgr2=0x7f01540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0158a00640 1 --2- 172.21.15.53:0/2262402486 >> v2:172.21.15.146:3300/0 conn(0x7f015415e3b0 0x7f01540702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0158a00640 1 -- 172.21.15.53:0/2262402486 >> v1:172.21.15.53:6789/0 conn(0x7f01540cf7d0 legacy=0x7f015415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0158a00640 1 -- 172.21.15.53:0/2262402486 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f015415e990 con 0x7f0154143ae0 2024-01-16T15:01:33.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.821+0000 7f0159400640 1 --2- 172.21.15.53:0/2262402486 >> v2:172.21.15.146:3300/0 conn(0x7f015415e3b0 0x7f01540702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:01:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.823+0000 7f0158a00640 1 --2- 172.21.15.53:0/2262402486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01540cf290 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f015406e4f0 tx=0x7f014402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb5b24e04c96aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.823+0000 7f0153e00640 1 -- 172.21.15.53:0/2262402486 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f014402cb10 con 0x7f0154143ae0 2024-01-16T15:01:33.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.823+0000 7f0153e00640 1 -- 172.21.15.53:0/2262402486 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014402cc70 con 0x7f0154143ae0 2024-01-16T15:01:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.823+0000 7f0153e00640 1 -- 172.21.15.53:0/2262402486 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01440343f0 con 0x7f0154143ae0 2024-01-16T15:01:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 -- 172.21.15.53:0/2262402486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 msgr2=0x7f01540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 --2- 172.21.15.53:0/2262402486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01540cf290 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f015406e4f0 tx=0x7f014402bf70 comp rx=0 tx=0).stop 2024-01-16T15:01:33.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 -- 172.21.15.53:0/2262402486 shutdown_connections 2024-01-16T15:01:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 --2- 172.21.15.53:0/2262402486 >> v2:172.21.15.146:3300/0 conn(0x7f015415e3b0 0x7f01540702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 --2- 172.21.15.53:0/2262402486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01540cf290 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 -- 172.21.15.53:0/2262402486 >> 172.21.15.53:0/2262402486 conn(0x7f015406a800 msgr2=0x7f015406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 -- 172.21.15.53:0/2262402486 shutdown_connections 2024-01-16T15:01:33.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.824+0000 7f015b400640 1 -- 172.21.15.53:0/2262402486 wait complete. 2024-01-16T15:01:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.826+0000 7f015b400640 1 Processor -- start 2024-01-16T15:01:33.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.826+0000 7f015b400640 1 -- start start 2024-01-16T15:01:33.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f015b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01540cf7d0 0x7f0154132380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f015b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01541f3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f015b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f015415e3b0 0x7f01541414f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f015b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01541f4eb0 con 0x7f0154143ae0 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f015b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01541f48f0 con 0x7f01540cf7d0 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f015b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01541f4bd0 con 0x7f015415e3b0 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f014bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01541f3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f014bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01541f3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55714/0 (socket says 172.21.15.53:55714) 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.827+0000 7f014bc00640 1 -- 172.21.15.53:0/695852513 learned_addr learned my addr 172.21.15.53:0/695852513 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f0158a00640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01540cf7d0 0x7f0154132380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f0159400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f015415e3b0 0x7f01541414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f014bc00640 1 -- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f015415e3b0 msgr2=0x7f01541414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f014bc00640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f015415e3b0 0x7f01541414f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f014bc00640 1 -- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01540cf7d0 msgr2=0x7f0154132380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f014bc00640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01540cf7d0 0x7f0154132380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:33.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f014bc00640 1 -- 172.21.15.53:0/695852513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0144009b30 con 0x7f0154143ae0 2024-01-16T15:01:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f014bc00640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01541f3fe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f013c00c4d0 tx=0x7f013c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c007480 con 0x7f0154143ae0 2024-01-16T15:01:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f013c0075e0 con 0x7f0154143ae0 2024-01-16T15:01:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f013c0049e0 con 0x7f0154143ae0 2024-01-16T15:01:33.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.828+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0154131b80 con 0x7f0154143ae0 2024-01-16T15:01:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.829+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01541328c0 con 0x7f0154143ae0 2024-01-16T15:01:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.830+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01541320a0 con 0x7f0154143ae0 2024-01-16T15:01:33.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.831+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 5) v1 ==== 89539+0+0 (secure 0 0 0) 0x7f013c004b40 con 0x7f0154143ae0 2024-01-16T15:01:33.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.832+0000 7f0151600640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f012c06f6d0 0x7f012c071b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:01:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.832+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f013c088ca0 con 0x7f0154143ae0 2024-01-16T15:01:33.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.832+0000 7f0158a00640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f012c06f6d0 0x7f012c071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:01:33.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.834+0000 7f0158a00640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f012c06f6d0 0x7f012c071b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0144035000 tx=0x7f0144030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:01:33.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:33.837+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01541320a0 con 0x7f0154143ae0 2024-01-16T15:01:34.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.062+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f01541f4d40 con 0x7f012c06f6d0 2024-01-16T15:01:34.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.074+0000 7f0151600640 1 -- 172.21.15.53:0/695852513 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f01541f4d40 con 0x7f012c06f6d0 2024-01-16T15:01:34.055 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155 2024-01-16T15:01:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f012c06f6d0 msgr2=0x7f012c071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f012c06f6d0 0x7f012c071b90 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0144035000 tx=0x7f0144030000 comp rx=0 tx=0).stop 2024-01-16T15:01:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 msgr2=0x7f01541f3fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:01:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01541f3fe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f013c00c4d0 tx=0x7f013c00c990 comp rx=0 tx=0).stop 2024-01-16T15:01:34.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 shutdown_connections 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f012c06f6d0 0x7f012c071b90 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f015415e3b0 0x7f01541414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0154143ae0 0x7f01541f3fe0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 --2- 172.21.15.53:0/695852513 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01540cf7d0 0x7f0154132380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.077+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 >> 172.21.15.53:0/695852513 conn(0x7f015406a800 msgr2=0x7f0154142cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.078+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 shutdown_connections 2024-01-16T15:01:34.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:01:34.078+0000 7f015b400640 1 -- 172.21.15.53:0/695852513 wait complete. 2024-01-16T15:01:34.077 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155 2024-01-16T15:01:34.077 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:01:34.078 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:34.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:34.133 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:01:50.328 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:50.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:01:50.672 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:01:50.672 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:01:50.672 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.4556 s, 65.3 MB/s 2024-01-16T15:01:50.674 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:01:50.674 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:01:50.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:01:50.726 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:02:04.347 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:02:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:02:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.5639 s, 79.2 MB/s 2024-01-16T15:02:04.349 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:02:04.350 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:04.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:04.405 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:02:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:02:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:02:19.389 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.9225 s, 72.0 MB/s 2024-01-16T15:02:19.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:02:19.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.636+0000 7fcecd200640 1 Processor -- start 2024-01-16T15:02:19.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.636+0000 7fcecd200640 1 -- start start 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.638+0000 7fcecd200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcec8143810 0x7fcec8071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.638+0000 7fcecd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcec815d770 0x7fcec815dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.638+0000 7fcecd200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcec80ae620 con 0x7fcec80ae170 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.638+0000 7fcecd200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec8143bf0 con 0x7fcec8143810 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.638+0000 7fcecd200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcec806c730 con 0x7fcec815d770 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcec8143810 0x7fcec8071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec6a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcec80ae170 0x7fcec815e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49196/0 (socket says 172.21.15.53:49196) 2024-01-16T15:02:19.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec6a00640 1 -- 172.21.15.53:0/2875505511 learned_addr learned my addr 172.21.15.53:0/2875505511 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:19.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec7e00640 1 --2- 172.21.15.53:0/2875505511 >> v2:172.21.15.146:3300/0 conn(0x7fcec815d770 0x7fcec815dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec6000640 1 -- 172.21.15.53:0/2875505511 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242232278 0 0) 0x7fcec80ae620 con 0x7fcec80ae170 2024-01-16T15:02:19.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec6000640 1 -- 172.21.15.53:0/2875505511 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcea8003520 con 0x7fcec80ae170 2024-01-16T15:02:19.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec7400640 1 -- 172.21.15.53:0/2875505511 >> v2:172.21.15.146:3300/0 conn(0x7fcec815d770 msgr2=0x7fcec815dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:19.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec7400640 1 --2- 172.21.15.53:0/2875505511 >> v2:172.21.15.146:3300/0 conn(0x7fcec815d770 0x7fcec815dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec7400640 1 -- 172.21.15.53:0/2875505511 >> v1:172.21.15.53:6789/0 conn(0x7fcec80ae170 legacy=0x7fcec815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.639+0000 7fcec7400640 1 -- 172.21.15.53:0/2875505511 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec80724d0 con 0x7fcec8143810 2024-01-16T15:02:19.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.641+0000 7fcec7400640 1 --2- 172.21.15.53:0/2875505511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec8143810 0x7fcec8071e40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fcec80b4440 tx=0x7fceb802bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4933f6cac0834fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.642+0000 7fcec6000640 1 -- 172.21.15.53:0/2875505511 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb802cb60 con 0x7fcec8143810 2024-01-16T15:02:19.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.642+0000 7fcec6000640 1 -- 172.21.15.53:0/2875505511 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb802ccc0 con 0x7fcec8143810 2024-01-16T15:02:19.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.642+0000 7fcec6000640 1 -- 172.21.15.53:0/2875505511 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8034660 con 0x7fcec8143810 2024-01-16T15:02:19.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.642+0000 7fcecd200640 1 -- 172.21.15.53:0/2875505511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec8143810 msgr2=0x7fcec8071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.642+0000 7fcecd200640 1 --2- 172.21.15.53:0/2875505511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec8143810 0x7fcec8071e40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fcec80b4440 tx=0x7fceb802bdb0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.643+0000 7fcecd200640 1 -- 172.21.15.53:0/2875505511 shutdown_connections 2024-01-16T15:02:19.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.643+0000 7fcecd200640 1 --2- 172.21.15.53:0/2875505511 >> v2:172.21.15.146:3300/0 conn(0x7fcec815d770 0x7fcec815dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.643+0000 7fcecd200640 1 --2- 172.21.15.53:0/2875505511 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec8143810 0x7fcec8071e40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.643+0000 7fcecd200640 1 -- 172.21.15.53:0/2875505511 >> 172.21.15.53:0/2875505511 conn(0x7fcec806a7d0 msgr2=0x7fcec806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.643+0000 7fcecd200640 1 -- 172.21.15.53:0/2875505511 shutdown_connections 2024-01-16T15:02:19.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.643+0000 7fcecd200640 1 -- 172.21.15.53:0/2875505511 wait complete. 2024-01-16T15:02:19.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.644+0000 7fcecd200640 1 Processor -- start 2024-01-16T15:02:19.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.645+0000 7fcecd200640 1 -- start start 2024-01-16T15:02:19.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcecd200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 0x7fcec80cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcecd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec8143810 0x7fcec8160370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcecd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcec815d770 0x7fcec81608b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcecd200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec806d930 con 0x7fcec80ae170 2024-01-16T15:02:19.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcecd200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec815d150 con 0x7fcec8143810 2024-01-16T15:02:19.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcecd200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcec81f2fc0 con 0x7fcec815d770 2024-01-16T15:02:19.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcec7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 0x7fcec80cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcec7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 0x7fcec80cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36230/0 (socket says 172.21.15.53:36230) 2024-01-16T15:02:19.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcec7400640 1 -- 172.21.15.53:0/1147345008 learned_addr learned my addr 172.21.15.53:0/1147345008 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:19.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcec7e00640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcec815d770 0x7fcec81608b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.646+0000 7fcec6a00640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec8143810 0x7fcec8160370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7400640 1 -- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcec815d770 msgr2=0x7fcec81608b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7400640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcec815d770 0x7fcec81608b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7400640 1 -- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec8143810 msgr2=0x7fcec8160370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7400640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec8143810 0x7fcec8160370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7400640 1 -- 172.21.15.53:0/1147345008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceb8009a70 con 0x7fcec80ae170 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7e00640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcec815d770 0x7fcec81608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec6a00640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec8143810 0x7fcec8160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcec7400640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 0x7fcec80cb5e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fceb8037040 tx=0x7fceb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8040650 con 0x7fcec80ae170 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceb80407b0 con 0x7fcec80ae170 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceb8040b60 con 0x7fcec80ae170 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.647+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec81f32a0 con 0x7fcec80ae170 2024-01-16T15:02:19.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.648+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec81f3860 con 0x7fcec80ae170 2024-01-16T15:02:19.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.649+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec8067450 con 0x7fcec80ae170 2024-01-16T15:02:19.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.650+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fceb8040cc0 con 0x7fcec80ae170 2024-01-16T15:02:19.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.651+0000 7fcebf400640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcea006b350 0x7fcea006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:19.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.651+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fceb80acb70 con 0x7fcec80ae170 2024-01-16T15:02:19.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.651+0000 7fcec6a00640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcea006b350 0x7fcea006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:19.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.653+0000 7fcec6a00640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcea006b350 0x7fcea006d810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcec806e150 tx=0x7fceb0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:19.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.657+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fceb8077980 con 0x7fcec80ae170 2024-01-16T15:02:19.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.855+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fcec8146bb0 con 0x7fcec80ae170 2024-01-16T15:02:19.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.860+0000 7fcebf400640 1 -- 172.21.15.53:0/1147345008 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v11)=0 v11) v1 ==== 135+0+0 (secure 0 0 0) 0x7fceb807afa0 con 0x7fcec80ae170 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcea006b350 msgr2=0x7fcea006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcea006b350 0x7fcea006d810 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcec806e150 tx=0x7fceb0009040 comp rx=0 tx=0).stop 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 msgr2=0x7fcec80cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 0x7fcec80cb5e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fceb8037040 tx=0x7fceb8004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 shutdown_connections 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcea006b350 0x7fcea006d810 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcec815d770 0x7fcec81608b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcec8143810 0x7fcec8160370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 --2- 172.21.15.53:0/1147345008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcec80ae170 0x7fcec80cb5e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 >> 172.21.15.53:0/1147345008 conn(0x7fcec806a7d0 msgr2=0x7fcec81615a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 shutdown_connections 2024-01-16T15:02:19.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:19.863+0000 7fcecd200640 1 -- 172.21.15.53:0/1147345008 wait complete. 2024-01-16T15:02:19.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:02:20.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.111+0000 7f6da1200640 1 Processor -- start 2024-01-16T15:02:20.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.111+0000 7f6da1200640 1 -- start start 2024-01-16T15:02:20.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6da1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d9c162810 0x7f6d9c071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6da1200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6d9c071780 0x7f6d9c06c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6da1200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0702e0 con 0x7f6d9c162430 2024-01-16T15:02:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6da1200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d9c0705c0 con 0x7f6d9c071780 2024-01-16T15:02:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6da1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0708a0 con 0x7f6d9c162810 2024-01-16T15:02:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6d9b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6d9c162430 0x7f6d9c0b3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49204/0 (socket says 172.21.15.53:49204) 2024-01-16T15:02:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.113+0000 7f6d9b400640 1 -- 172.21.15.53:0/2996714922 learned_addr learned my addr 172.21.15.53:0/2996714922 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:20.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9be00640 1 --2- 172.21.15.53:0/2996714922 >> v2:172.21.15.146:3300/0 conn(0x7f6d9c071780 0x7f6d9c06c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615568366 0 0) 0x7f6d9c0702e0 con 0x7f6d9c162430 2024-01-16T15:02:20.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d9c162810 0x7f6d9c071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c003520 con 0x7f6d9c162430 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320235510 0 0) 0x7f6d7c003520 con 0x7f6d9c162430 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d9c162810 msgr2=0x7f6d9c071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d9c162810 0x7f6d9c071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 >> v2:172.21.15.146:3300/0 conn(0x7f6d9c071780 msgr2=0x7f6d9c06c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 --2- 172.21.15.53:0/2996714922 >> v2:172.21.15.146:3300/0 conn(0x7f6d9c071780 0x7f6d9c06c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c070b80 con 0x7f6d9c162430 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6d8c001670 con 0x7f6d9c162430 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 >> v1:172.21.15.53:6789/0 conn(0x7f6d9c162430 legacy=0x7f6d9c0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d7c003970 0x7f6d7c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c0047c0 0x7f6d7c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c005180 0x7f6d7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0705c0 con 0x7f6d7c0047c0 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c0708a0 con 0x7f6d7c003970 2024-01-16T15:02:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7c00bef0 con 0x7f6d7c005180 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.114+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d9c162810 0x7f6d9c071240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c0047c0 0x7f6d7c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9b400640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d7c003970 0x7f6d7c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9be00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c005180 0x7f6d7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9aa00640 1 -- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c005180 msgr2=0x7f6d7c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c005180 0x7f6d7c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9aa00640 1 -- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d7c003970 msgr2=0x7f6d7c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d7c003970 0x7f6d7c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.115+0000 7f6d9aa00640 1 -- 172.21.15.53:0/2996714922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c003520 con 0x7f6d7c0047c0 2024-01-16T15:02:20.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.116+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c0047c0 0x7f6d7c004c40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6d84006fd0 tx=0x7f6d7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eace4990ef4a39e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84034680 con 0x7f6d7c0047c0 2024-01-16T15:02:20.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d840347e0 con 0x7f6d7c0047c0 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6d9a000640 1 -- 172.21.15.53:0/2996714922 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d84034b70 con 0x7f6d7c0047c0 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6da1200640 1 -- 172.21.15.53:0/2996714922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c0047c0 msgr2=0x7f6d7c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6da1200640 1 --2- 172.21.15.53:0/2996714922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d7c0047c0 0x7f6d7c004c40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f6d84006fd0 tx=0x7f6d7c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6d9be00640 1 -- 172.21.15.53:0/2996714922 reap_dead start 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6da1200640 1 -- 172.21.15.53:0/2996714922 shutdown_connections 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6da1200640 1 -- 172.21.15.53:0/2996714922 >> 172.21.15.53:0/2996714922 conn(0x7f6d9c06a800 msgr2=0x7f6d9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6da1200640 1 -- 172.21.15.53:0/2996714922 shutdown_connections 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.117+0000 7f6da1200640 1 -- 172.21.15.53:0/2996714922 wait complete. 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.119+0000 7f6da1200640 1 Processor -- start 2024-01-16T15:02:20.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.119+0000 7f6da1200640 1 -- start start 2024-01-16T15:02:20.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.120+0000 7f6da1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c0047c0 0x7f6d9c143020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.120+0000 7f6da1200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 0x7f6d9c157090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.120+0000 7f6da1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d9c1fa880 0x7f6d9c1fad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.120+0000 7f6da1200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c1fb5b0 con 0x7f6d9c156c10 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.120+0000 7f6da1200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c1575d0 con 0x7f6d9c1fa880 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.120+0000 7f6da1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d9c146940 con 0x7f6d7c0047c0 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 0x7f6d9c157090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 0x7f6d9c157090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36250/0 (socket says 172.21.15.53:36250) 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 -- 172.21.15.53:0/3978890613 learned_addr learned my addr 172.21.15.53:0/3978890613 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9b400640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c0047c0 0x7f6d9c143020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 -- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c0047c0 msgr2=0x7f6d9c143020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c0047c0 0x7f6d9c143020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 -- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d9c1fa880 msgr2=0x7f6d9c1fad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d9c1fa880 0x7f6d9c1fad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 -- 172.21.15.53:0/3978890613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c1464b0 con 0x7f6d9c156c10 2024-01-16T15:02:20.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.121+0000 7f6d9aa00640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 0x7f6d9c157090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6d84037040 tx=0x7f6d84004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.122+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8403d810 con 0x7f6d9c156c10 2024-01-16T15:02:20.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.122+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d8403d970 con 0x7f6d9c156c10 2024-01-16T15:02:20.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.122+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d8403dce0 con 0x7f6d9c156c10 2024-01-16T15:02:20.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.122+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c145b80 con 0x7f6d9c156c10 2024-01-16T15:02:20.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.122+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c146140 con 0x7f6d9c156c10 2024-01-16T15:02:20.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.123+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d9c146640 con 0x7f6d9c156c10 2024-01-16T15:02:20.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.124+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f6d8403b040 con 0x7f6d9c156c10 2024-01-16T15:02:20.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.125+0000 7f6d93400640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6d7406b380 0x7f6d7406d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.125+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6d840ab180 con 0x7f6d9c156c10 2024-01-16T15:02:20.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.126+0000 7f6d9b400640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6d7406b380 0x7f6d7406d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.127+0000 7f6d9b400640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6d7406b380 0x7f6d7406d840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c001eb0 tx=0x7f6d8c000ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.130+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6d84075fa0 con 0x7f6d9c156c10 2024-01-16T15:02:20.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.357+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6d9c145970 con 0x7f6d7406b380 2024-01-16T15:02:20.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.393+0000 7f6d93400640 1 -- 172.21.15.53:0/3978890613 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6d9c145970 con 0x7f6d7406b380 2024-01-16T15:02:20.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.395+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6d7406b380 msgr2=0x7f6d7406d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.395+0000 7f6da1200640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6d7406b380 0x7f6d7406d840 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6d8c001eb0 tx=0x7f6d8c000ed0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.395+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 msgr2=0x7f6d9c157090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.395+0000 7f6da1200640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 0x7f6d9c157090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f6d84037040 tx=0x7f6d84004730 comp rx=0 tx=0).stop 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 shutdown_connections 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6d7406b380 0x7f6d7406d840 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6d9c1fa880 0x7f6d9c1fad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6d9c156c10 0x7f6d9c157090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 --2- 172.21.15.53:0/3978890613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6d7c0047c0 0x7f6d9c143020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 >> 172.21.15.53:0/3978890613 conn(0x7f6d9c06a800 msgr2=0x7f6d9c0cb3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 shutdown_connections 2024-01-16T15:02:20.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.396+0000 7f6da1200640 1 -- 172.21.15.53:0/3978890613 wait complete. 2024-01-16T15:02:20.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:02:20.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:20.501 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.643+0000 7f1c7be00640 1 Processor -- start 2024-01-16T15:02:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.643+0000 7f1c7be00640 1 -- start start 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c7be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1c7c0cf9a0 0x7f1c7c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c7be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c7c0b31d0 0x7f1c7c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c7be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c7c0d0320 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c7be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7c0d1c60 con 0x7f1c7c0cf9a0 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c7be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c7c0b2730 con 0x7f1c7c0b31d0 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c73e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1c7c0cf9a0 0x7f1c7c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c73e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1c7c0cf9a0 0x7f1c7c0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36264/0 (socket says 172.21.15.53:36264) 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c73e00640 1 -- 172.21.15.53:0/2514311318 learned_addr learned my addr 172.21.15.53:0/2514311318 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:20.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.645+0000 7f1c7b400640 1 --2- 172.21.15.53:0/2514311318 >> v2:172.21.15.146:3300/0 conn(0x7f1c7c0b31d0 0x7f1c7c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.646+0000 7f1c73e00640 1 -- 172.21.15.53:0/2514311318 >> v2:172.21.15.146:3300/0 conn(0x7f1c7c0b31d0 msgr2=0x7f1c7c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:20.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.646+0000 7f1c73e00640 1 --2- 172.21.15.53:0/2514311318 >> v2:172.21.15.146:3300/0 conn(0x7f1c7c0b31d0 0x7f1c7c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.646+0000 7f1c73e00640 1 -- 172.21.15.53:0/2514311318 >> v1:172.21.15.53:6789/0 conn(0x7f1c7c15d4b0 legacy=0x7f1c7c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.646+0000 7f1c73e00640 1 -- 172.21.15.53:0/2514311318 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c0d1ec0 con 0x7f1c7c0cf9a0 2024-01-16T15:02:20.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.647+0000 7f1c73e00640 1 --2- 172.21.15.53:0/2514311318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c0cfde0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c072a30 tx=0x7f1c6402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b98915a4c7d1aaf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.647+0000 7f1c7a000640 1 -- 172.21.15.53:0/2514311318 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64030860 con 0x7f1c7c0cf9a0 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.647+0000 7f1c7a000640 1 -- 172.21.15.53:0/2514311318 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c640309c0 con 0x7f1c7c0cf9a0 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.647+0000 7f1c7a000640 1 -- 172.21.15.53:0/2514311318 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c64030d50 con 0x7f1c7c0cf9a0 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 -- 172.21.15.53:0/2514311318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c0cf9a0 msgr2=0x7f1c7c0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 --2- 172.21.15.53:0/2514311318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c0cfde0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c072a30 tx=0x7f1c6402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 -- 172.21.15.53:0/2514311318 shutdown_connections 2024-01-16T15:02:20.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 --2- 172.21.15.53:0/2514311318 >> v2:172.21.15.146:3300/0 conn(0x7f1c7c0b31d0 0x7f1c7c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 --2- 172.21.15.53:0/2514311318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c0cfde0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 -- 172.21.15.53:0/2514311318 >> 172.21.15.53:0/2514311318 conn(0x7f1c7c0d3000 msgr2=0x7f1c7c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 -- 172.21.15.53:0/2514311318 shutdown_connections 2024-01-16T15:02:20.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.648+0000 7f1c7be00640 1 -- 172.21.15.53:0/2514311318 wait complete. 2024-01-16T15:02:20.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.650+0000 7f1c7be00640 1 Processor -- start 2024-01-16T15:02:20.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.650+0000 7f1c7be00640 1 -- start start 2024-01-16T15:02:20.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.651+0000 7f1c7be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c7c0b31d0 0x7f1c7c09ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.651+0000 7f1c7be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c09f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.651+0000 7f1c7be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 0x7f1c7c09d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.651+0000 7f1c7be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c09fdf0 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.651+0000 7f1c7be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c09e670 con 0x7f1c7c0cf9a0 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.651+0000 7f1c7be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c09e390 con 0x7f1c7c0b31d0 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 0x7f1c7c09d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 0x7f1c7c09d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36266/0 (socket says 172.21.15.53:36266) 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 -- 172.21.15.53:0/578086884 learned_addr learned my addr 172.21.15.53:0/578086884 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c73e00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c09f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7aa00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c7c0b31d0 0x7f1c7c09ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 -- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c7c0b31d0 msgr2=0x7f1c7c09ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c7c0b31d0 0x7f1c7c09ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 -- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c7c0cf9a0 msgr2=0x7f1c7c09f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.652+0000 7f1c7b400640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c09f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.653+0000 7f1c7b400640 1 -- 172.21.15.53:0/578086884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640094d0 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.653+0000 7f1c7b400640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 0x7f1c7c09d6c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400efc0 tx=0x7f1c7400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.654+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c74009ba0 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.654+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c7400f040 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.654+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c7c0ca410 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.654+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c740146c0 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.654+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7c0ca9d0 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.655+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7c06ca30 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.656+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f1c74014820 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.657+0000 7f1c72a00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c5406b350 0x7f1c5406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:20.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.657+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c7408afa0 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.658+0000 7f1c7aa00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c5406b350 0x7f1c5406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:20.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.659+0000 7f1c7aa00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c5406b350 0x7f1c5406d810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c000f60 tx=0x7f1c6c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:20.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.661+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c74055c90 con 0x7f1c7c15d4b0 2024-01-16T15:02:20.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.881+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c7c0d08e0 con 0x7f1c5406b350 2024-01-16T15:02:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.897+0000 7f1c72a00640 1 -- 172.21.15.53:0/578086884 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1c7c0d08e0 con 0x7f1c5406b350 2024-01-16T15:02:20.877 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c5406b350 msgr2=0x7f1c5406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c5406b350 0x7f1c5406d810 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1c6c000f60 tx=0x7f1c6c009040 comp rx=0 tx=0).stop 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 msgr2=0x7f1c7c09d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 0x7f1c7c09d6c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f1c7400efc0 tx=0x7f1c7400c490 comp rx=0 tx=0).stop 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 shutdown_connections 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c5406b350 0x7f1c5406d810 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c7c15d4b0 0x7f1c7c09d6c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c7c0cf9a0 0x7f1c7c09f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 --2- 172.21.15.53:0/578086884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c7c0b31d0 0x7f1c7c09ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 >> 172.21.15.53:0/578086884 conn(0x7f1c7c0d3000 msgr2=0x7f1c7c06c320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 shutdown_connections 2024-01-16T15:02:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:20.899+0000 7f1c7be00640 1 -- 172.21.15.53:0/578086884 wait complete. 2024-01-16T15:02:20.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:20.899 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155) 2024-01-16T15:02:21.005 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/ad00559f-725c-4030-85cf-8643acf19155 2024-01-16T15:02:21.005 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:02:21.005 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:21.006 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:02:21.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T15:02:21.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.246+0000 7fc37cc00640 1 Processor -- start 2024-01-16T15:02:21.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.247+0000 7fc37cc00640 1 -- start start 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.248+0000 7fc37cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc37813e360 0x7fc3780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.248+0000 7fc37cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc37815f9c0 0x7fc3780cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.248+0000 7fc37cc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc37815c5f0 con 0x7fc37813df80 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.248+0000 7fc37cc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37806c470 con 0x7fc37815f9c0 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.248+0000 7fc37cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc37806c750 con 0x7fc37813e360 2024-01-16T15:02:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc376800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc37815f9c0 0x7fc3780cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc375e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc37813df80 0x7fc37815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49224/0 (socket says 172.21.15.53:49224) 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc376800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc37815f9c0 0x7fc3780cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36280/0 (socket says 172.21.15.53:36280) 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc375e00640 1 -- 172.21.15.53:0/4204410726 learned_addr learned my addr 172.21.15.53:0/4204410726 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc36fe00640 1 --2- 172.21.15.53:0/4204410726 >> v2:172.21.15.146:3300/0 conn(0x7fc37813e360 0x7fc3780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc376800640 1 -- 172.21.15.53:0/4204410726 >> v2:172.21.15.146:3300/0 conn(0x7fc37813e360 msgr2=0x7fc3780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.249+0000 7fc376800640 1 --2- 172.21.15.53:0/4204410726 >> v2:172.21.15.146:3300/0 conn(0x7fc37813e360 0x7fc3780cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.250+0000 7fc376800640 1 -- 172.21.15.53:0/4204410726 >> v1:172.21.15.53:6789/0 conn(0x7fc37813df80 legacy=0x7fc37815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.250+0000 7fc376800640 1 -- 172.21.15.53:0/4204410726 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc37806ca30 con 0x7fc37815f9c0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.250+0000 7fc375400640 1 -- 172.21.15.53:0/4204410726 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566841945 0 0) 0x7fc37815c5f0 con 0x7fc37813df80 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.251+0000 7fc376800640 1 --2- 172.21.15.53:0/4204410726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37815f9c0 0x7fc3780cf7d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc37806ded0 tx=0x7fc37002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79858deef83e18fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.251+0000 7fc375400640 1 -- 172.21.15.53:0/4204410726 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc37002cc80 con 0x7fc37815f9c0 2024-01-16T15:02:21.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.251+0000 7fc375400640 1 -- 172.21.15.53:0/4204410726 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc37002cde0 con 0x7fc37815f9c0 2024-01-16T15:02:21.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.251+0000 7fc375400640 1 -- 172.21.15.53:0/4204410726 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc370034660 con 0x7fc37815f9c0 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 -- 172.21.15.53:0/4204410726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37815f9c0 msgr2=0x7fc3780cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 --2- 172.21.15.53:0/4204410726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37815f9c0 0x7fc3780cf7d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc37806ded0 tx=0x7fc37002bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 -- 172.21.15.53:0/4204410726 shutdown_connections 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 --2- 172.21.15.53:0/4204410726 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37815f9c0 0x7fc3780cf7d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 --2- 172.21.15.53:0/4204410726 >> v2:172.21.15.146:3300/0 conn(0x7fc37813e360 0x7fc3780cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 -- 172.21.15.53:0/4204410726 >> 172.21.15.53:0/4204410726 conn(0x7fc37806a800 msgr2=0x7fc37806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 -- 172.21.15.53:0/4204410726 shutdown_connections 2024-01-16T15:02:21.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.252+0000 7fc37cc00640 1 -- 172.21.15.53:0/4204410726 wait complete. 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.254+0000 7fc37cc00640 1 Processor -- start 2024-01-16T15:02:21.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.254+0000 7fc37cc00640 1 -- start start 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc37cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37813df80 0x7fc378153400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc37cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 0x7fc37815d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc37cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37815f9c0 0x7fc37815dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc37cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378153940 con 0x7fc37813e360 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc37cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37813c630 con 0x7fc37813df80 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc37cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc37813c910 con 0x7fc37815f9c0 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc36fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 0x7fc37815d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc36fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 0x7fc37815d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36284/0 (socket says 172.21.15.53:36284) 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc36fe00640 1 -- 172.21.15.53:0/2560272311 learned_addr learned my addr 172.21.15.53:0/2560272311 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc376800640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37815f9c0 0x7fc37815dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.255+0000 7fc375e00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37813df80 0x7fc378153400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36fe00640 1 -- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37815f9c0 msgr2=0x7fc37815dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36fe00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37815f9c0 0x7fc37815dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36fe00640 1 -- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37813df80 msgr2=0x7fc378153400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36fe00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37813df80 0x7fc378153400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36fe00640 1 -- 172.21.15.53:0/2560272311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc370009a70 con 0x7fc37813e360 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc375e00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37813df80 0x7fc378153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:21.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36fe00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 0x7fc37815d8a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc36000d7d0 tx=0x7fc36000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc360018070 con 0x7fc37813e360 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc360004020 con 0x7fc37813e360 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37813cbf0 con 0x7fc37813e360 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3600041b0 con 0x7fc37813e360 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.256+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37813d1b0 con 0x7fc37813e360 2024-01-16T15:02:21.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.257+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3781f2ef0 con 0x7fc37813e360 2024-01-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.258+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7fc360004310 con 0x7fc37813e360 2024-01-16T15:02:21.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.259+0000 7fc36e000640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc35006b350 0x7fc35006d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.259+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc360089b80 con 0x7fc37813e360 2024-01-16T15:02:21.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.260+0000 7fc375e00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc35006b350 0x7fc35006d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.261+0000 7fc375e00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc35006b350 0x7fc35006d810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc368000f60 tx=0x7fc368009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.264+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc3781f2ef0 con 0x7fc37813e360 2024-01-16T15:02:21.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.501+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3781559b0 con 0x7fc35006b350 2024-01-16T15:02:21.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.686+0000 7fc36e000640 1 -- 172.21.15.53:0/2560272311 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc3781559b0 con 0x7fc35006b350 2024-01-16T15:02:21.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc35006b350 msgr2=0x7fc35006d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc35006b350 0x7fc35006d810 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc368000f60 tx=0x7fc368009040 comp rx=0 tx=0).stop 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 msgr2=0x7fc37815d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 0x7fc37815d8a0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc36000d7d0 tx=0x7fc36000dc90 comp rx=0 tx=0).stop 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 shutdown_connections 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc35006b350 0x7fc35006d810 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc37815f9c0 0x7fc37815dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc37813e360 0x7fc37815d8a0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 --2- 172.21.15.53:0/2560272311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc37813df80 0x7fc378153400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 >> 172.21.15.53:0/2560272311 conn(0x7fc37806a800 msgr2=0x7fc3780af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 shutdown_connections 2024-01-16T15:02:21.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.690+0000 7fc37cc00640 1 -- 172.21.15.53:0/2560272311 wait complete. 2024-01-16T15:02:21.674 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.694+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.675 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.694+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.675 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.694+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.675 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.694+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.675 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.695+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.692 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:02:21.693 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:21.701 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.721+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.701 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.721+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.701 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.721+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.701 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.721+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.701 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:21.721+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:21.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.926+0000 7f4633400640 1 Processor -- start 2024-01-16T15:02:21.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.927+0000 7f4633400640 1 -- start start 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.928+0000 7f4633400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f462c09a170 0x7f462c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.928+0000 7f4633400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f462c06c6b0 0x7f462c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.928+0000 7f4633400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462c06cea0 con 0x7f462c0cd240 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.928+0000 7f4633400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462c06cbc0 con 0x7f462c09a170 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.928+0000 7f4633400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462c06f350 con 0x7f462c06c6b0 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f4630a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f462c0cd240 0x7f462c099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50076/0 (socket says 172.21.15.53:50076) 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f462be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f462c09a170 0x7f462c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f4630a00640 1 -- 172.21.15.53:0/3366494506 learned_addr learned my addr 172.21.15.53:0/3366494506 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f4631400640 1 --2- 172.21.15.53:0/3366494506 >> v2:172.21.15.146:3300/0 conn(0x7f462c06c6b0 0x7f462c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f462be00640 1 -- 172.21.15.53:0/3366494506 >> v2:172.21.15.146:3300/0 conn(0x7f462c06c6b0 msgr2=0x7f462c0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f462b400640 1 -- 172.21.15.53:0/3366494506 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343518881 0 0) 0x7f462c06cea0 con 0x7f462c0cd240 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f462be00640 1 --2- 172.21.15.53:0/3366494506 >> v2:172.21.15.146:3300/0 conn(0x7f462c06c6b0 0x7f462c0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f462be00640 1 -- 172.21.15.53:0/3366494506 >> v1:172.21.15.53:6789/0 conn(0x7f462c0cd240 legacy=0x7f462c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f462be00640 1 -- 172.21.15.53:0/3366494506 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462c06f630 con 0x7f462c09a170 2024-01-16T15:02:21.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.929+0000 7f4631400640 1 --2- 172.21.15.53:0/3366494506 >> v2:172.21.15.146:3300/0 conn(0x7f462c06c6b0 0x7f462c0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.931+0000 7f462be00640 1 --2- 172.21.15.53:0/3366494506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c09a170 0x7f462c0b3820 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f462c0cdc40 tx=0x7f461402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aaf7a3ba2477d60 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.931+0000 7f462b400640 1 -- 172.21.15.53:0/3366494506 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614031830 con 0x7f462c09a170 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.931+0000 7f462b400640 1 -- 172.21.15.53:0/3366494506 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4614031990 con 0x7f462c09a170 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.931+0000 7f462b400640 1 -- 172.21.15.53:0/3366494506 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4614031d20 con 0x7f462c09a170 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 -- 172.21.15.53:0/3366494506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c09a170 msgr2=0x7f462c0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 --2- 172.21.15.53:0/3366494506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c09a170 0x7f462c0b3820 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f462c0cdc40 tx=0x7f461402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 -- 172.21.15.53:0/3366494506 shutdown_connections 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 --2- 172.21.15.53:0/3366494506 >> v2:172.21.15.146:3300/0 conn(0x7f462c06c6b0 0x7f462c0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 --2- 172.21.15.53:0/3366494506 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c09a170 0x7f462c0b3820 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 -- 172.21.15.53:0/3366494506 >> 172.21.15.53:0/3366494506 conn(0x7f462c0b2d00 msgr2=0x7f462c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:21.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 -- 172.21.15.53:0/3366494506 shutdown_connections 2024-01-16T15:02:21.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.932+0000 7f4633400640 1 -- 172.21.15.53:0/3366494506 wait complete. 2024-01-16T15:02:21.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.934+0000 7f4633400640 1 Processor -- start 2024-01-16T15:02:21.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.934+0000 7f4633400640 1 -- start start 2024-01-16T15:02:21.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4633400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 0x7f462c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4633400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f462c09a170 0x7f462c0c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4633400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f462c0cd240 0x7f462c0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4633400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0b5930 con 0x7f462c06c6b0 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4633400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0b4d20 con 0x7f462c0cd240 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4633400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f462c0b5000 con 0x7f462c09a170 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4630a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 0x7f462c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4630a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 0x7f462c15b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58440/0 (socket says 172.21.15.53:58440) 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.935+0000 7f4630a00640 1 -- 172.21.15.53:0/2781989284 learned_addr learned my addr 172.21.15.53:0/2781989284 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4630a00640 1 -- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f462c09a170 msgr2=0x7f462c0c6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4630a00640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f462c09a170 0x7f462c0c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4630a00640 1 -- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f462c0cd240 msgr2=0x7f462c0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4631400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f462c0cd240 0x7f462c0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4630a00640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f462c0cd240 0x7f462c0c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4630a00640 1 -- 172.21.15.53:0/2781989284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4614009400 con 0x7f462c06c6b0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4631400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f462c0cd240 0x7f462c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4630a00640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 0x7f462c15b390 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f461c00c4d0 tx=0x7f461c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c0074e0 con 0x7f462c06c6b0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c007660 con 0x7f462c06c6b0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f461c004a20 con 0x7f462c06c6b0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462c0b52e0 con 0x7f462c06c6b0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.936+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462c0b4060 con 0x7f462c06c6b0 2024-01-16T15:02:21.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.938+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462c0675d0 con 0x7f462c06c6b0 2024-01-16T15:02:21.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.939+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 6) v1 ==== 89590+0+0 (secure 0 0 0) 0x7f461c004b80 con 0x7f462c06c6b0 2024-01-16T15:02:21.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.940+0000 7f4628c00640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f460406b350 0x7f460406d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:21.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.940+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f461c088e50 con 0x7f462c06c6b0 2024-01-16T15:02:21.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.941+0000 7f462be00640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f460406b350 0x7f460406d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:21.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.942+0000 7f462be00640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f460406b350 0x7f460406d810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4614036000 tx=0x7f4614030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:21.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:21.945+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f461c053c60 con 0x7f462c06c6b0 2024-01-16T15:02:22.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:22.180+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f462c0cd620 con 0x7f460406b350 2024-01-16T15:02:23.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:23.079+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f461c054030 con 0x7f462c06c6b0 2024-01-16T15:02:25.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.676+0000 7f4628c00640 1 -- 172.21.15.53:0/2781989284 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7f4614031b80 con 0x7f460406b350 2024-01-16T15:02:25.657 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:25.658 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f460406b350 msgr2=0x7f460406d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f460406b350 0x7f460406d810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4614036000 tx=0x7f4614030000 comp rx=0 tx=0).stop 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 msgr2=0x7f462c15b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 0x7f462c15b390 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f461c00c4d0 tx=0x7f461c00c990 comp rx=0 tx=0).stop 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 shutdown_connections 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f460406b350 0x7f460406d810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f462c0cd240 0x7f462c0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f462c09a170 0x7f462c0c6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 --2- 172.21.15.53:0/2781989284 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f462c06c6b0 0x7f462c15b390 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 >> 172.21.15.53:0/2781989284 conn(0x7f462c0b2d00 msgr2=0x7f462c0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 shutdown_connections 2024-01-16T15:02:25.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.682+0000 7f4633400640 1 -- 172.21.15.53:0/2781989284 wait complete. 2024-01-16T15:02:25.687 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:02:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.927+0000 7f2f6c200640 1 Processor -- start 2024-01-16T15:02:25.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.927+0000 7f2f6c200640 1 -- start start 2024-01-16T15:02:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.929+0000 7f2f6c200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f64143ae0 0x7f2f640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.929+0000 7f2f6c200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f640cfc30 con 0x7f2f6415e3b0 2024-01-16T15:02:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.929+0000 7f2f6c200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f640b1ee0 con 0x7f2f640cf7d0 2024-01-16T15:02:25.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.929+0000 7f2f6c200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f640b2780 con 0x7f2f64143ae0 2024-01-16T15:02:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.929+0000 7f2f6a200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2f6415e3b0 0x7f2f640702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50082/0 (socket says 172.21.15.53:50082) 2024-01-16T15:02:25.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.929+0000 7f2f6a200640 1 -- 172.21.15.53:0/2870644033 learned_addr learned my addr 172.21.15.53:0/2870644033 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f69800640 1 --2- 172.21.15.53:0/2870644033 >> v2:172.21.15.146:3300/0 conn(0x7f2f64143ae0 0x7f2f640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390385903 0 0) 0x7f2f640cfc30 con 0x7f2f6415e3b0 2024-01-16T15:02:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f44005730 con 0x7f2f6415e3b0 2024-01-16T15:02:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3849778812 0 0) 0x7f2f44005730 con 0x7f2f6415e3b0 2024-01-16T15:02:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 >> v2:172.21.15.146:3300/0 conn(0x7f2f64143ae0 msgr2=0x7f2f640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:25.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 --2- 172.21.15.53:0/2870644033 >> v2:172.21.15.146:3300/0 conn(0x7f2f64143ae0 0x7f2f640cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 >> v1:172.21.15.146:6789/0 conn(0x7f2f640cf7d0 legacy=0x7f2f6415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6415e990 con 0x7f2f6415e3b0 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2f5c0019f0 con 0x7f2f6415e3b0 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 >> v1:172.21.15.53:6789/0 conn(0x7f2f6415e3b0 legacy=0x7f2f640702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.930+0000 7f2f63e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f44005bd0 0x7f2f44005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f63e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f63e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f640b2780 con 0x7f2f440069d0 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4400e290 con 0x7f2f44007390 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f4400e450 con 0x7f2f44005bd0 2024-01-16T15:02:25.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f68e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f6a200640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f69800640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f44005bd0 0x7f2f44005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f68e00640 1 -- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f44005bd0 msgr2=0x7f2f44005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f68e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f44005bd0 0x7f2f44005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f68e00640 1 -- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f44007390 msgr2=0x7f2f4400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f68e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f68e00640 1 -- 172.21.15.53:0/2870644033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f44005730 con 0x7f2f440069d0 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f6a200640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f44007390 0x7f2f4400db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f6a200640 1 -- 172.21.15.53:0/2870644033 reap_dead start 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.931+0000 7f2f69800640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f44005bd0 0x7f2f44005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.933+0000 7f2f68e00640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c001750 tx=0x7f2f44007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a592731e4d4057b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.933+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c0052d0 con 0x7f2f440069d0 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.933+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4c02e050 con 0x7f2f440069d0 2024-01-16T15:02:25.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.933+0000 7f2f63e00640 1 -- 172.21.15.53:0/2870644033 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c003830 con 0x7f2f440069d0 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 -- 172.21.15.53:0/2870644033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 msgr2=0x7f2f44006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c001750 tx=0x7f2f44007a00 comp rx=0 tx=0).stop 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 -- 172.21.15.53:0/2870644033 shutdown_connections 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 --2- 172.21.15.53:0/2870644033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f44006e50 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 -- 172.21.15.53:0/2870644033 >> 172.21.15.53:0/2870644033 conn(0x7f2f6406a800 msgr2=0x7f2f6406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 -- 172.21.15.53:0/2870644033 shutdown_connections 2024-01-16T15:02:25.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.934+0000 7f2f6c200640 1 -- 172.21.15.53:0/2870644033 wait complete. 2024-01-16T15:02:25.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.936+0000 7f2f6c200640 1 Processor -- start 2024-01-16T15:02:25.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.936+0000 7f2f6c200640 1 -- start start 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f6c200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f641f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f6c200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f641375f0 0x7f2f64137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f6c200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f6406c7c0 0x7f2f6406cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f6c200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f64137fb0 con 0x7f2f440069d0 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f6c200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6406d180 con 0x7f2f6406c7c0 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f6c200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f640add80 con 0x7f2f641375f0 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f641f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f641f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58460/0 (socket says 172.21.15.53:58460) 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 -- 172.21.15.53:0/4093450394 learned_addr learned my addr 172.21.15.53:0/4093450394 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 -- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f641375f0 msgr2=0x7f2f64137a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f641375f0 0x7f2f64137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 -- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f6406c7c0 msgr2=0x7f2f6406cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f6406c7c0 0x7f2f6406cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.937+0000 7f2f69800640 1 -- 172.21.15.53:0/4093450394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f641381b0 con 0x7f2f440069d0 2024-01-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.938+0000 7f2f69800640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f641f27b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2f540107c0 tx=0x7f2f54010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.938+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5400eb60 con 0x7f2f440069d0 2024-01-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.938+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5400ecc0 con 0x7f2f440069d0 2024-01-16T15:02:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.938+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54010e70 con 0x7f2f440069d0 2024-01-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.938+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f64138490 con 0x7f2f440069d0 2024-01-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.938+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f64138a50 con 0x7f2f440069d0 2024-01-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.939+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f640675d0 con 0x7f2f440069d0 2024-01-16T15:02:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.940+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f2f5401c400 con 0x7f2f440069d0 2024-01-16T15:02:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.941+0000 7f2f61600640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3c06b470 0x7f2f3c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.941+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2f5408a290 con 0x7f2f440069d0 2024-01-16T15:02:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.942+0000 7f2f68e00640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3c06b470 0x7f2f3c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:25.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.943+0000 7f2f68e00640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3c06b470 0x7f2f3c06d930 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c033000 tx=0x7f2f4c02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:25.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:25.946+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f54054fd0 con 0x7f2f440069d0 2024-01-16T15:02:26.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.211+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2f641411a0 con 0x7f2f440069d0 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.212+0000 7f2f61600640 1 -- 172.21.15.53:0/4093450394 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1825 (secure 0 0 0) 0x7f2f540585f0 con 0x7f2f440069d0 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:02:26.193 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 173, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:02:26.194 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 35, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:02:26.195 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 793, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3222022809, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 6698369024, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 766395744256, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 56601152, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 0, 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 14 2024-01-16T15:02:26.196 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 17, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 2, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 2, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "c", 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5190 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:02:26.197 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 9, 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:02:24.226131+0000", 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.198 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": {} 2024-01-16T15:02:26.199 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3c06b470 msgr2=0x7f2f3c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3c06b470 0x7f2f3c06d930 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c033000 tx=0x7f2f4c02f000 comp rx=0 tx=0).stop 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 msgr2=0x7f2f641f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f641f27b0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f2f540107c0 tx=0x7f2f54010c80 comp rx=0 tx=0).stop 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 shutdown_connections 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3c06b470 0x7f2f3c06d930 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f6406c7c0 0x7f2f6406cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f641375f0 0x7f2f64137a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 --2- 172.21.15.53:0/4093450394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f440069d0 0x7f2f641f27b0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 >> 172.21.15.53:0/4093450394 conn(0x7f2f6406a800 msgr2=0x7f2f64141670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 shutdown_connections 2024-01-16T15:02:26.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.215+0000 7f2f6c200640 1 -- 172.21.15.53:0/4093450394 wait complete. 2024-01-16T15:02:26.215 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:02:26.215 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:26.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.456+0000 7fa853400640 1 Processor -- start 2024-01-16T15:02:26.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.456+0000 7fa853400640 1 -- start start 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa853400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa84c15d4b0 0x7fa84c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa853400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa84c0b41e0 0x7fa84c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa853400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa84c0d0320 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa853400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c0b2730 con 0x7fa84c15d4b0 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa853400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa84c0d1ec0 con 0x7fa84c0b41e0 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa852000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa84c15d4b0 0x7fa84c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa851600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa84c0cf9a0 0x7fa84c0cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50090/0 (socket says 172.21.15.53:50090) 2024-01-16T15:02:26.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.457+0000 7fa851600640 1 -- 172.21.15.53:0/3433414197 learned_addr learned my addr 172.21.15.53:0/3433414197 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa852a00640 1 --2- 172.21.15.53:0/3433414197 >> v2:172.21.15.146:3300/0 conn(0x7fa84c0b41e0 0x7fa84c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa852000640 1 -- 172.21.15.53:0/3433414197 >> v2:172.21.15.146:3300/0 conn(0x7fa84c0b41e0 msgr2=0x7fa84c0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa852000640 1 --2- 172.21.15.53:0/3433414197 >> v2:172.21.15.146:3300/0 conn(0x7fa84c0b41e0 0x7fa84c0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa852000640 1 -- 172.21.15.53:0/3433414197 >> v1:172.21.15.53:6789/0 conn(0x7fa84c0cf9a0 legacy=0x7fa84c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa852000640 1 -- 172.21.15.53:0/3433414197 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84c0b29d0 con 0x7fa84c15d4b0 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa852a00640 1 --2- 172.21.15.53:0/3433414197 >> v2:172.21.15.146:3300/0 conn(0x7fa84c0b41e0 0x7fa84c0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:26.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.458+0000 7fa850c00640 1 -- 172.21.15.53:0/3433414197 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 399311359 0 0) 0x7fa84c0d0320 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.459+0000 7fa852000640 1 --2- 172.21.15.53:0/3433414197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0d1550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0d0680 tx=0x7fa83c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e9d84dcd4b3335 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.459+0000 7fa850c00640 1 -- 172.21.15.53:0/3433414197 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c031830 con 0x7fa84c15d4b0 2024-01-16T15:02:26.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.459+0000 7fa850c00640 1 -- 172.21.15.53:0/3433414197 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa83c031990 con 0x7fa84c15d4b0 2024-01-16T15:02:26.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.459+0000 7fa850c00640 1 -- 172.21.15.53:0/3433414197 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa83c031d20 con 0x7fa84c15d4b0 2024-01-16T15:02:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 -- 172.21.15.53:0/3433414197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c15d4b0 msgr2=0x7fa84c0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 --2- 172.21.15.53:0/3433414197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0d1550 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fa84c0d0680 tx=0x7fa83c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 -- 172.21.15.53:0/3433414197 shutdown_connections 2024-01-16T15:02:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 --2- 172.21.15.53:0/3433414197 >> v2:172.21.15.146:3300/0 conn(0x7fa84c0b41e0 0x7fa84c0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 --2- 172.21.15.53:0/3433414197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0d1550 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 -- 172.21.15.53:0/3433414197 >> 172.21.15.53:0/3433414197 conn(0x7fa84c0d3000 msgr2=0x7fa84c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.460+0000 7fa853400640 1 -- 172.21.15.53:0/3433414197 shutdown_connections 2024-01-16T15:02:26.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.461+0000 7fa853400640 1 -- 172.21.15.53:0/3433414197 wait complete. 2024-01-16T15:02:26.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.462+0000 7fa853400640 1 Processor -- start 2024-01-16T15:02:26.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.462+0000 7fa853400640 1 -- start start 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.463+0000 7fa853400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84c0b41e0 0x7fa84c09d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.463+0000 7fa853400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 0x7fa84c09da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.463+0000 7fa853400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.463+0000 7fa853400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0b6a20 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.463+0000 7fa853400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0b1cc0 con 0x7fa84c15d4b0 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.463+0000 7fa853400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c0b6c70 con 0x7fa84c0b41e0 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 0x7fa84c09da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 0x7fa84c09da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58484/0 (socket says 172.21.15.53:58484) 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 -- 172.21.15.53:0/2714309887 learned_addr learned my addr 172.21.15.53:0/2714309887 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa852000640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84c0b41e0 0x7fa84c09d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa852a00640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 -- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84c0b41e0 msgr2=0x7fa84c09d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84c0b41e0 0x7fa84c09d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 -- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c15d4b0 msgr2=0x7fa84c0b0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0b0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa851600640 1 -- 172.21.15.53:0/2714309887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa83c009400 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.464+0000 7fa852a00640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.465+0000 7fa851600640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 0x7fa84c09da10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c4d0 tx=0x7fa83400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.465+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8340074e0 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.465+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834007660 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.465+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c15da10 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.465+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834004a20 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.465+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c0b7230 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.466+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c067450 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.468+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fa834004b80 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.468+0000 7fa84a000640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa82406b470 0x7fa82406d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.469+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8340894e0 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.469+0000 7fa852000640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa82406b470 0x7fa82406d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:26.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.470+0000 7fa852000640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa82406b470 0x7fa82406d930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa83c036000 tx=0x7fa83c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.473+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa834054030 con 0x7fa84c0cf9a0 2024-01-16T15:02:26.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.709+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa84c0d0cd0 con 0x7fa82406b470 2024-01-16T15:02:26.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.723+0000 7fa84a000640 1 -- 172.21.15.53:0/2714309887 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fa84c0d0cd0 con 0x7fa82406b470 2024-01-16T15:02:26.703 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:26.703 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:26.703 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:26.703 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:26.704 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:26.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa82406b470 msgr2=0x7fa82406d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa82406b470 0x7fa82406d930 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa83c036000 tx=0x7fa83c008030 comp rx=0 tx=0).stop 2024-01-16T15:02:26.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 msgr2=0x7fa84c09da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:26.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 0x7fa84c09da10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fa83400c4d0 tx=0x7fa83400c990 comp rx=0 tx=0).stop 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 shutdown_connections 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa82406b470 0x7fa82406d930 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa84c15d4b0 0x7fa84c0b0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa84c0cf9a0 0x7fa84c09da10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 --2- 172.21.15.53:0/2714309887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa84c0b41e0 0x7fa84c09d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.725+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 >> 172.21.15.53:0/2714309887 conn(0x7fa84c0d3000 msgr2=0x7fa84c0b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.726+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 shutdown_connections 2024-01-16T15:02:26.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:26.726+0000 7fa853400640 1 -- 172.21.15.53:0/2714309887 wait complete. 2024-01-16T15:02:26.725 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:27.726 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:27.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.000+0000 7f0c9ba00640 1 Processor -- start 2024-01-16T15:02:27.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.000+0000 7f0c9ba00640 1 -- start start 2024-01-16T15:02:27.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.001+0000 7f0c9ba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 0x7f0c940b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.001+0000 7f0c9ba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c94071190 con 0x7f0c9415da10 2024-01-16T15:02:27.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c9ba00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c94071470 con 0x7f0c94150760 2024-01-16T15:02:27.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c9ba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c94071750 con 0x7f0c94072230 2024-01-16T15:02:27.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c93e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0c9415da10 0x7f0c940b2bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50104/0 (socket says 172.21.15.53:50104) 2024-01-16T15:02:27.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c93e00640 1 -- 172.21.15.53:0/3746689049 learned_addr learned my addr 172.21.15.53:0/3746689049 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:27.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614124524 0 0) 0x7f0c94071190 con 0x7f0c9415da10 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c74003520 con 0x7f0c9415da10 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.002+0000 7f0c99a00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 0x7f0c940b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2345267286 0 0) 0x7f0c74003520 con 0x7f0c9415da10 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 msgr2=0x7f0c940b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 0x7f0c940b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 >> v1:172.21.15.146:6789/0 conn(0x7f0c94150760 legacy=0x7f0c9415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c94071a30 con 0x7f0c9415da10 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0c7c001670 con 0x7f0c9415da10 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 >> v1:172.21.15.53:6789/0 conn(0x7f0c9415da10 legacy=0x7f0c940b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 0x7f0c740039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c74003f70 0x7f0c740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c74004d00 0x7f0c74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c94071750 con 0x7f0c74003f70 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7400bac0 con 0x7f0c74004d00 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7400de90 con 0x7f0c94072230 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.003+0000 7f0c93e00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c74003f70 0x7f0c740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c99a00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c74004d00 0x7f0c74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c99000640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 0x7f0c740039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c93e00640 1 -- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 msgr2=0x7f0c740039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c93e00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c94072230 0x7f0c740039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c93e00640 1 -- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c74004d00 msgr2=0x7f0c74005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c93e00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c74004d00 0x7f0c74005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.004+0000 7f0c93e00640 1 -- 172.21.15.53:0/3746689049 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c74003520 con 0x7f0c74003f70 2024-01-16T15:02:27.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.005+0000 7f0c93e00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c74003f70 0x7f0c740047c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00c0e0 tx=0x7f0c740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40182ad3e4e7e3fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c005ad0 con 0x7f0c74003f70 2024-01-16T15:02:27.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c005c50 con 0x7f0c74003f70 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c93400640 1 -- 172.21.15.53:0/3746689049 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c007070 con 0x7f0c74003f70 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3746689049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c74003f70 msgr2=0x7f0c740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3746689049 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c74003f70 0x7f0c740047c0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00c0e0 tx=0x7f0c740057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c99a00640 1 -- 172.21.15.53:0/3746689049 reap_dead start 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3746689049 shutdown_connections 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.006+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3746689049 >> 172.21.15.53:0/3746689049 conn(0x7f0c9406a800 msgr2=0x7f0c9406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.007+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3746689049 shutdown_connections 2024-01-16T15:02:27.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.007+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3746689049 wait complete. 2024-01-16T15:02:27.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.009+0000 7f0c9ba00640 1 Processor -- start 2024-01-16T15:02:27.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.009+0000 7f0c9ba00640 1 -- start start 2024-01-16T15:02:27.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c9ba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c74003f70 0x7f0c940b15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c9ba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c941f2b90 0x7f0c940ce730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c9ba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 0x7f0c940b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c9ba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940b1320 con 0x7f0c940b0960 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c9ba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940cec70 con 0x7f0c941f2b90 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c9ba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940cc9b0 con 0x7f0c74003f70 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c99a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 0x7f0c940b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c99a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 0x7f0c940b0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58496/0 (socket says 172.21.15.53:58496) 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c99a00640 1 -- 172.21.15.53:0/3747381269 learned_addr learned my addr 172.21.15.53:0/3747381269 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c99000640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c74003f70 0x7f0c940b15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.010+0000 7f0c93e00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c941f2b90 0x7f0c940ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c99a00640 1 -- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c74003f70 msgr2=0x7f0c940b15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c99a00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c74003f70 0x7f0c940b15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c99a00640 1 -- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c941f2b90 msgr2=0x7f0c940ce730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c99a00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c941f2b90 0x7f0c940ce730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c99a00640 1 -- 172.21.15.53:0/3747381269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c940ccc90 con 0x7f0c940b0960 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c93e00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c941f2b90 0x7f0c940ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c99a00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 0x7f0c940b0de0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0c9406e190 tx=0x7f0c8c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c004500 con 0x7f0c940b0960 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c003610 con 0x7f0c940b0960 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c940ccf70 con 0x7f0c940b0960 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c8c0039e0 con 0x7f0c940b0960 2024-01-16T15:02:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.011+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c940cd530 con 0x7f0c940b0960 2024-01-16T15:02:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.013+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c940675d0 con 0x7f0c940b0960 2024-01-16T15:02:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.013+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f0c8c003b40 con 0x7f0c940b0960 2024-01-16T15:02:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.015+0000 7f0c90c00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c6c06b4a0 0x7f0c6c06d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.015+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0c8c08a240 con 0x7f0c940b0960 2024-01-16T15:02:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.020+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c8c054ec0 con 0x7f0c940b0960 2024-01-16T15:02:28.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.046+0000 7f0c99000640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c6c06b4a0 0x7f0c6c06d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:28.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.047+0000 7f0c99000640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c6c06b4a0 0x7f0c6c06d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c84007fd0 tx=0x7f0c84009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.346+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c941499f0 con 0x7f0c6c06b4a0 2024-01-16T15:02:28.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.374+0000 7f0c90c00640 1 -- 172.21.15.53:0/3747381269 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f0c941499f0 con 0x7f0c6c06b4a0 2024-01-16T15:02:28.356 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:02:28.357 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:28.358 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c6c06b4a0 msgr2=0x7f0c6c06d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c6c06b4a0 0x7f0c6c06d960 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f0c84007fd0 tx=0x7f0c84009040 comp rx=0 tx=0).stop 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 msgr2=0x7f0c940b0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 0x7f0c940b0de0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f0c9406e190 tx=0x7f0c8c011920 comp rx=0 tx=0).stop 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 shutdown_connections 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c6c06b4a0 0x7f0c6c06d960 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c940b0960 0x7f0c940b0de0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c941f2b90 0x7f0c940ce730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 --2- 172.21.15.53:0/3747381269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c74003f70 0x7f0c940b15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:28.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 >> 172.21.15.53:0/3747381269 conn(0x7f0c9406a800 msgr2=0x7f0c940b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:28.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 shutdown_connections 2024-01-16T15:02:28.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:28.377+0000 7f0c9ba00640 1 -- 172.21.15.53:0/3747381269 wait complete. 2024-01-16T15:02:28.393 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:29.394 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:29.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.701+0000 7f5538a00640 1 Processor -- start 2024-01-16T15:02:29.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.702+0000 7f5538a00640 1 -- start start 2024-01-16T15:02:29.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.703+0000 7f5538a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55300cd240 0x7f55300b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.703+0000 7f5538a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553006c6b0 0x7f5530099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.703+0000 7f5538a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5530099fa0 0x7f553006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.703+0000 7f5538a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553009a430 con 0x7f55300cd240 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.703+0000 7f5538a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553006cea0 con 0x7f5530099fa0 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.703+0000 7f5538a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553006cbc0 con 0x7f553006c6b0 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55300cd240 0x7f55300b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55300cd240 0x7f55300b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58498/0 (socket says 172.21.15.53:58498) 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 -- 172.21.15.53:0/946204489 learned_addr learned my addr 172.21.15.53:0/946204489 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5535600640 1 --2- 172.21.15.53:0/946204489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553006c6b0 0x7f5530099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 -- 172.21.15.53:0/946204489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553006c6b0 msgr2=0x7f5530099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 --2- 172.21.15.53:0/946204489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553006c6b0 0x7f5530099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 -- 172.21.15.53:0/946204489 >> v2:172.21.15.146:3300/0 conn(0x7f5530099fa0 msgr2=0x7f553006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 --2- 172.21.15.53:0/946204489 >> v2:172.21.15.146:3300/0 conn(0x7f5530099fa0 0x7f553006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.704+0000 7f5536000640 1 -- 172.21.15.53:0/946204489 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553006f350 con 0x7f55300cd240 2024-01-16T15:02:29.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.706+0000 7f5536000640 1 --2- 172.21.15.53:0/946204489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55300cd240 0x7f55300b3820 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f55300722b0 tx=0x7f552002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f6e3a6fddc58c73 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.706+0000 7f5534c00640 1 -- 172.21.15.53:0/946204489 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520031870 con 0x7f55300cd240 2024-01-16T15:02:29.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.706+0000 7f5534c00640 1 -- 172.21.15.53:0/946204489 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55200319d0 con 0x7f55300cd240 2024-01-16T15:02:29.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.706+0000 7f5534c00640 1 -- 172.21.15.53:0/946204489 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520031d60 con 0x7f55300cd240 2024-01-16T15:02:29.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.707+0000 7f5538a00640 1 -- 172.21.15.53:0/946204489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55300cd240 msgr2=0x7f55300b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.707+0000 7f5538a00640 1 --2- 172.21.15.53:0/946204489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55300cd240 0x7f55300b3820 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f55300722b0 tx=0x7f552002b910 comp rx=0 tx=0).stop 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 -- 172.21.15.53:0/946204489 shutdown_connections 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 --2- 172.21.15.53:0/946204489 >> v2:172.21.15.146:3300/0 conn(0x7f5530099fa0 0x7f553006b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 --2- 172.21.15.53:0/946204489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553006c6b0 0x7f5530099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 --2- 172.21.15.53:0/946204489 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55300cd240 0x7f55300b3820 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 -- 172.21.15.53:0/946204489 >> 172.21.15.53:0/946204489 conn(0x7f55300b2d00 msgr2=0x7f55300b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 -- 172.21.15.53:0/946204489 shutdown_connections 2024-01-16T15:02:29.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.708+0000 7f5538a00640 1 -- 172.21.15.53:0/946204489 wait complete. 2024-01-16T15:02:29.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.711+0000 7f5538a00640 1 Processor -- start 2024-01-16T15:02:29.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.711+0000 7f5538a00640 1 -- start start 2024-01-16T15:02:29.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5538a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 0x7f5530157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5538a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5530099fa0 0x7f55300c73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5538a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55300cd240 0x7f55300b4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5538a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55300b4f00 con 0x7f553006c6b0 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5538a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55300c5750 con 0x7f55300cd240 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5538a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55300c59a0 con 0x7f5530099fa0 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5536000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 0x7f5530157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5536000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 0x7f5530157070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58514/0 (socket says 172.21.15.53:58514) 2024-01-16T15:02:29.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.713+0000 7f5536000640 1 -- 172.21.15.53:0/562686091 learned_addr learned my addr 172.21.15.53:0/562686091 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:29.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55300cd240 0x7f55300b4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536000640 1 -- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5530099fa0 msgr2=0x7f55300c73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536000640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5530099fa0 0x7f55300c73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536000640 1 -- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55300cd240 msgr2=0x7f55300b4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:29.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536000640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55300cd240 0x7f55300b4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536000640 1 -- 172.21.15.53:0/562686091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55200094d0 con 0x7f553006c6b0 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55300cd240 0x7f55300b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.714+0000 7f5536000640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 0x7f5530157070 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5520000c00 tx=0x7f55200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.715+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552003abe0 con 0x7f553006c6b0 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.715+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552003ad40 con 0x7f553006c6b0 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.715+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520044610 con 0x7f553006c6b0 2024-01-16T15:02:29.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.715+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55300c5c80 con 0x7f553006c6b0 2024-01-16T15:02:29.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.715+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55300c6240 con 0x7f553006c6b0 2024-01-16T15:02:29.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.719+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55300675d0 con 0x7f553006c6b0 2024-01-16T15:02:29.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.719+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f552004d450 con 0x7f553006c6b0 2024-01-16T15:02:29.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.721+0000 7f552e000640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550806b470 0x7f550806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:29.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.721+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f552002d040 con 0x7f553006c6b0 2024-01-16T15:02:29.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.728+0000 7f5535600640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550806b470 0x7f550806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:29.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.728+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5520077110 con 0x7f553006c6b0 2024-01-16T15:02:29.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:29.740+0000 7f5535600640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550806b470 0x7f550806d930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5518006fd0 tx=0x7f5518008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:30.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.092+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f55300cd620 con 0x7f550806b470 2024-01-16T15:02:30.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.101+0000 7f552e000640 1 -- 172.21.15.53:0/562686091 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f551800be20 con 0x7f550806b470 2024-01-16T15:02:30.081 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:30.082 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:30.083 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:30.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550806b470 msgr2=0x7f550806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550806b470 0x7f550806d930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5518006fd0 tx=0x7f5518008040 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 msgr2=0x7f5530157070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 0x7f5530157070 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5520000c00 tx=0x7f55200046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 shutdown_connections 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550806b470 0x7f550806d930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55300cd240 0x7f55300b4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5530099fa0 0x7f55300c73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 --2- 172.21.15.53:0/562686091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553006c6b0 0x7f5530157070 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 >> 172.21.15.53:0/562686091 conn(0x7f55300b2d00 msgr2=0x7f5530160850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 shutdown_connections 2024-01-16T15:02:30.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:30.104+0000 7f5538a00640 1 -- 172.21.15.53:0/562686091 wait complete. 2024-01-16T15:02:30.102 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:31.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:31.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.359+0000 7f1fdfe00640 1 Processor -- start 2024-01-16T15:02:31.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.359+0000 7f1fdfe00640 1 -- start start 2024-01-16T15:02:31.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.360+0000 7f1fdfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fe01505b0 0x7f1fe00b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.360+0000 7f1fdfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 0x7f1fe006cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fdfe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe006d930 con 0x7f1fe006d500 2024-01-16T15:02:31.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fdfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe00b1e30 con 0x7f1fe01505b0 2024-01-16T15:02:31.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fdfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00cdc10 con 0x7f1fe015a690 2024-01-16T15:02:31.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fde000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 0x7f1fe006cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fdea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1fe01505b0 0x7f1fe00b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fde000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 0x7f1fe006cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:36256/0 (socket says 172.21.15.53:36256) 2024-01-16T15:02:31.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.361+0000 7f1fde000640 1 -- 172.21.15.53:0/4268720934 learned_addr learned my addr 172.21.15.53:0/4268720934 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:31.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.362+0000 7f1fde000640 1 -- 172.21.15.53:0/4268720934 >> v2:172.21.15.146:3300/0 conn(0x7f1fe01505b0 msgr2=0x7f1fe00b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:31.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.362+0000 7f1fde000640 1 --2- 172.21.15.53:0/4268720934 >> v2:172.21.15.146:3300/0 conn(0x7f1fe01505b0 0x7f1fe00b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.362+0000 7f1fde000640 1 -- 172.21.15.53:0/4268720934 >> v1:172.21.15.146:6789/0 conn(0x7f1fe006d500 legacy=0x7f1fe015d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.362+0000 7f1fde000640 1 -- 172.21.15.53:0/4268720934 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe00b20a0 con 0x7f1fe015a690 2024-01-16T15:02:31.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.364+0000 7f1fde000640 1 --2- 172.21.15.53:0/4268720934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 0x7f1fe006cfc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009870 tx=0x7f1fc802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2a2eb1ae16a3fc4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.364+0000 7f1fdd600640 1 -- 172.21.15.53:0/4268720934 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8030840 con 0x7f1fe015a690 2024-01-16T15:02:31.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.364+0000 7f1fdd600640 1 -- 172.21.15.53:0/4268720934 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc80309a0 con 0x7f1fe015a690 2024-01-16T15:02:31.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.364+0000 7f1fdd600640 1 -- 172.21.15.53:0/4268720934 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8030d30 con 0x7f1fe015a690 2024-01-16T15:02:31.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 -- 172.21.15.53:0/4268720934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 msgr2=0x7f1fe006cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/4268720934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 0x7f1fe006cfc0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009870 tx=0x7f1fc802b980 comp rx=0 tx=0).stop 2024-01-16T15:02:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 -- 172.21.15.53:0/4268720934 shutdown_connections 2024-01-16T15:02:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/4268720934 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe015a690 0x7f1fe006cfc0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/4268720934 >> v2:172.21.15.146:3300/0 conn(0x7f1fe01505b0 0x7f1fe00b3b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 -- 172.21.15.53:0/4268720934 >> 172.21.15.53:0/4268720934 conn(0x7f1fe006a7d0 msgr2=0x7f1fe006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.365+0000 7f1fdfe00640 1 -- 172.21.15.53:0/4268720934 shutdown_connections 2024-01-16T15:02:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.366+0000 7f1fdfe00640 1 -- 172.21.15.53:0/4268720934 wait complete. 2024-01-16T15:02:31.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.367+0000 7f1fdfe00640 1 Processor -- start 2024-01-16T15:02:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.367+0000 7f1fdfe00640 1 -- start start 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.368+0000 7f1fdfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe006d500 0x7f1fe014e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.368+0000 7f1fdfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe01505b0 0x7f1fe00b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.368+0000 7f1fdfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 0x7f1fe014a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.368+0000 7f1fdfe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe014b0a0 con 0x7f1fe015a690 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.368+0000 7f1fdfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe014adc0 con 0x7f1fe01505b0 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.368+0000 7f1fdfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe014db10 con 0x7f1fe006d500 2024-01-16T15:02:31.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 0x7f1fe014a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 0x7f1fe014a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58524/0 (socket says 172.21.15.53:58524) 2024-01-16T15:02:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 -- 172.21.15.53:0/872986287 learned_addr learned my addr 172.21.15.53:0/872986287 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdea00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe006d500 0x7f1fe014e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fde000640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe01505b0 0x7f1fe00b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 -- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe006d500 msgr2=0x7f1fe014e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe006d500 0x7f1fe014e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 -- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe01505b0 msgr2=0x7f1fe00b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe01505b0 0x7f1fe00b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdf400640 1 -- 172.21.15.53:0/872986287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc80094d0 con 0x7f1fe015a690 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.369+0000 7f1fdea00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe006d500 0x7f1fe014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.370+0000 7f1fdf400640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 0x7f1fe014a190 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c3c0 tx=0x7f1fd800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.370+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80052d0 con 0x7f1fe015a690 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.370+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8005430 con 0x7f1fe015a690 2024-01-16T15:02:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.370+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8003850 con 0x7f1fe015a690 2024-01-16T15:02:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.370+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe014ddf0 con 0x7f1fe015a690 2024-01-16T15:02:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.370+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe014eb30 con 0x7f1fe015a690 2024-01-16T15:02:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.371+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe014e310 con 0x7f1fe015a690 2024-01-16T15:02:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.372+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f1fd80039b0 con 0x7f1fe015a690 2024-01-16T15:02:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.373+0000 7f1fd7400640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fb806b470 0x7f1fb806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.373+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1fd8089490 con 0x7f1fe015a690 2024-01-16T15:02:31.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.374+0000 7f1fdea00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fb806b470 0x7f1fb806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:31.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.375+0000 7f1fdea00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fb806b470 0x7f1fb806d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00074c0 tx=0x7f1fd000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:31.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.378+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1fe014e310 con 0x7f1fe015a690 2024-01-16T15:02:31.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.610+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fe015aaf0 con 0x7f1fb806b470 2024-01-16T15:02:31.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.625+0000 7f1fd7400640 1 -- 172.21.15.53:0/872986287 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f1fd000f050 con 0x7f1fb806b470 2024-01-16T15:02:31.605 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:31.605 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:31.605 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:31.605 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:31.606 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:31.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fb806b470 msgr2=0x7f1fb806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fb806b470 0x7f1fb806d930 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00074c0 tx=0x7f1fd000b040 comp rx=0 tx=0).stop 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 msgr2=0x7f1fe014a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 0x7f1fe014a190 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800c3c0 tx=0x7f1fd800c880 comp rx=0 tx=0).stop 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 shutdown_connections 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fb806b470 0x7f1fb806d930 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe015a690 0x7f1fe014a190 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe01505b0 0x7f1fe00b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 --2- 172.21.15.53:0/872986287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe006d500 0x7f1fe014e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 >> 172.21.15.53:0/872986287 conn(0x7f1fe006a7d0 msgr2=0x7f1fe015d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 shutdown_connections 2024-01-16T15:02:31.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:31.628+0000 7f1fdfe00640 1 -- 172.21.15.53:0/872986287 wait complete. 2024-01-16T15:02:31.627 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:32.628 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:32.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.916+0000 7f4163200640 1 Processor -- start 2024-01-16T15:02:32.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.916+0000 7f4163200640 1 -- start start 2024-01-16T15:02:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.918+0000 7f4163200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f416406d6d0 0x7f41640b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.918+0000 7f4163200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f416415a690 0x7f41640ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.918+0000 7f4163200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41640b1e30 con 0x7f41641505b0 2024-01-16T15:02:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.918+0000 7f4163200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f416415ab20 con 0x7f416406d6d0 2024-01-16T15:02:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.918+0000 7f4163200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41640b20a0 con 0x7f416415a690 2024-01-16T15:02:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4161400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f416406d6d0 0x7f41640b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4161e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f41641505b0 0x7f416406cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41556/0 (socket says 172.21.15.53:41556) 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4161e00640 1 -- 172.21.15.53:0/455860430 learned_addr learned my addr 172.21.15.53:0/455860430 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4162800640 1 --2- 172.21.15.53:0/455860430 >> v2:172.21.15.146:3300/0 conn(0x7f416415a690 0x7f41640ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4161400640 1 -- 172.21.15.53:0/455860430 >> v2:172.21.15.146:3300/0 conn(0x7f416415a690 msgr2=0x7f41640ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4161400640 1 --2- 172.21.15.53:0/455860430 >> v2:172.21.15.146:3300/0 conn(0x7f416415a690 0x7f41640ccc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4160a00640 1 -- 172.21.15.53:0/455860430 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1376324133 0 0) 0x7f41640b1e30 con 0x7f41641505b0 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.919+0000 7f4161400640 1 -- 172.21.15.53:0/455860430 >> v1:172.21.15.53:6789/0 conn(0x7f41641505b0 legacy=0x7f416406cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.920+0000 7f4161400640 1 -- 172.21.15.53:0/455860430 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41640b2380 con 0x7f416406d6d0 2024-01-16T15:02:32.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.922+0000 7f4161400640 1 --2- 172.21.15.53:0/455860430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f416406d6d0 0x7f41640b3b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f416415d0f0 tx=0x7f414c02dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c414a95140a6e5a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.922+0000 7f4160a00640 1 -- 172.21.15.53:0/455860430 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c02f8a0 con 0x7f416406d6d0 2024-01-16T15:02:32.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.922+0000 7f4160a00640 1 -- 172.21.15.53:0/455860430 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c02fa00 con 0x7f416406d6d0 2024-01-16T15:02:32.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.922+0000 7f4160a00640 1 -- 172.21.15.53:0/455860430 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c02fd90 con 0x7f416406d6d0 2024-01-16T15:02:32.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 -- 172.21.15.53:0/455860430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f416406d6d0 msgr2=0x7f41640b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 --2- 172.21.15.53:0/455860430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f416406d6d0 0x7f41640b3b20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f416415d0f0 tx=0x7f414c02dae0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 -- 172.21.15.53:0/455860430 shutdown_connections 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 --2- 172.21.15.53:0/455860430 >> v2:172.21.15.146:3300/0 conn(0x7f416415a690 0x7f41640ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 --2- 172.21.15.53:0/455860430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f416406d6d0 0x7f41640b3b20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 -- 172.21.15.53:0/455860430 >> 172.21.15.53:0/455860430 conn(0x7f416406a7d0 msgr2=0x7f416406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 -- 172.21.15.53:0/455860430 shutdown_connections 2024-01-16T15:02:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.923+0000 7f4163200640 1 -- 172.21.15.53:0/455860430 wait complete. 2024-01-16T15:02:32.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.926+0000 7f4163200640 1 Processor -- start 2024-01-16T15:02:32.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.927+0000 7f4163200640 1 -- start start 2024-01-16T15:02:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.928+0000 7f4163200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f416406d6d0 0x7f41641f0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.928+0000 7f4163200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 0x7f41641f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.928+0000 7f4163200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f416415a690 0x7f4164150cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.928+0000 7f4163200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641f3dd0 con 0x7f41641505b0 2024-01-16T15:02:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.928+0000 7f4163200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641f3810 con 0x7f416406d6d0 2024-01-16T15:02:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.928+0000 7f4163200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41641f3af0 con 0x7f416415a690 2024-01-16T15:02:32.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 0x7f41641f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 0x7f41641f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50940/0 (socket says 172.21.15.53:50940) 2024-01-16T15:02:32.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 -- 172.21.15.53:0/3382084347 learned_addr learned my addr 172.21.15.53:0/3382084347 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:32.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161e00640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f416406d6d0 0x7f41641f0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4162800640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f416415a690 0x7f4164150cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 -- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f416415a690 msgr2=0x7f4164150cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f416415a690 0x7f4164150cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 -- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f416406d6d0 msgr2=0x7f41641f0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:32.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f416406d6d0 0x7f41641f0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4161400640 1 -- 172.21.15.53:0/3382084347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f414c009400 con 0x7f41641505b0 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.929+0000 7f4162800640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f416415a690 0x7f4164150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.930+0000 7f4161400640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 0x7f41641f2e20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f414c009890 tx=0x7f414c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.930+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c030ab0 con 0x7f41641505b0 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.930+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f414c030c30 con 0x7f41641505b0 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.930+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f414c042410 con 0x7f41641505b0 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.930+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416414b8e0 con 0x7f41641505b0 2024-01-16T15:02:32.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.931+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416414cee0 con 0x7f41641505b0 2024-01-16T15:02:32.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.932+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f416414be00 con 0x7f41641505b0 2024-01-16T15:02:32.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.933+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f414c042570 con 0x7f41641505b0 2024-01-16T15:02:32.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.934+0000 7f415a000640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f413c06b4c0 0x7f413c06d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:32.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.934+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f414c0abc50 con 0x7f41641505b0 2024-01-16T15:02:32.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.940+0000 7f4161e00640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f413c06b4c0 0x7f413c06d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:32.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.942+0000 7f4161e00640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f413c06b4c0 0x7f413c06d980 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4154000f60 tx=0x7f4154007d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:32.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:32.942+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f416414be00 con 0x7f41641505b0 2024-01-16T15:02:33.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.286+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f41640b41e0 con 0x7f413c06b4c0 2024-01-16T15:02:33.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.301+0000 7f415a000640 1 -- 172.21.15.53:0/3382084347 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f41540072e0 con 0x7f413c06b4c0 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:02:33.282 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:02:33.283 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:33.283 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:33.283 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:33.283 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:33.283 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f413c06b4c0 msgr2=0x7f413c06d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f413c06b4c0 0x7f413c06d980 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4154000f60 tx=0x7f4154007d50 comp rx=0 tx=0).stop 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 msgr2=0x7f41641f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 0x7f41641f2e20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f414c009890 tx=0x7f414c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 shutdown_connections 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f413c06b4c0 0x7f413c06d980 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f416415a690 0x7f4164150cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41641505b0 0x7f41641f2e20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 --2- 172.21.15.53:0/3382084347 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f416406d6d0 0x7f41641f0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:33.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.306+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 >> 172.21.15.53:0/3382084347 conn(0x7f416406a7d0 msgr2=0x7f41640ccbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:33.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.307+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 shutdown_connections 2024-01-16T15:02:33.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:33.307+0000 7f4163200640 1 -- 172.21.15.53:0/3382084347 wait complete. 2024-01-16T15:02:33.316 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:34.317 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:34.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.569+0000 7f961da00640 1 Processor -- start 2024-01-16T15:02:34.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.570+0000 7f961da00640 1 -- start start 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.571+0000 7f961da00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9618071e40 0x7f96180ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.571+0000 7f961da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f961815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.571+0000 7f961da00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f961815e140 con 0x7f9618143810 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.571+0000 7f961da00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9618143bf0 con 0x7f9618071e40 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.571+0000 7f961da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96180722f0 con 0x7f961815d770 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.572+0000 7f9617e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9618143810 0x7f961815e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41558/0 (socket says 172.21.15.53:41558) 2024-01-16T15:02:34.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.572+0000 7f9617e00640 1 -- 172.21.15.53:0/1665557739 learned_addr learned my addr 172.21.15.53:0/1665557739 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.572+0000 7f961d000640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f961815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.572+0000 7f960fe00640 1 --2- 172.21.15.53:0/1665557739 >> v2:172.21.15.146:3300/0 conn(0x7f9618071e40 0x7f96180ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.572+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169705277 0 0) 0x7f961815e140 con 0x7f9618143810 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.572+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8003520 con 0x7f9618143810 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400952220 0 0) 0x7f95f8003520 con 0x7f9618143810 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 msgr2=0x7f961815dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f961815dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 >> v2:172.21.15.146:3300/0 conn(0x7f9618071e40 msgr2=0x7f96180ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 --2- 172.21.15.53:0/1665557739 >> v2:172.21.15.146:3300/0 conn(0x7f9618071e40 0x7f96180ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961806c730 con 0x7f9618143810 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9608001670 con 0x7f9618143810 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 >> v1:172.21.15.53:6789/0 conn(0x7f9618143810 legacy=0x7f961815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95f8003970 0x7f95f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95f80047c0 0x7f95f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f95f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9618143bf0 con 0x7f95f80047c0 2024-01-16T15:02:34.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96180722f0 con 0x7f95f8003970 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95f800ba60 con 0x7f961815d770 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.573+0000 7f960fe00640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95f80047c0 0x7f95f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f9617e00640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95f8003970 0x7f95f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f961d000640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f95f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f960fe00640 1 -- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 msgr2=0x7f95f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f960fe00640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f95f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f960fe00640 1 -- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95f8003970 msgr2=0x7f95f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f960fe00640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f95f8003970 0x7f95f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f960fe00640 1 -- 172.21.15.53:0/1665557739 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f8003520 con 0x7f95f80047c0 2024-01-16T15:02:34.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.574+0000 7f961d000640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f961815d770 0x7f95f8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:34.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.575+0000 7f960fe00640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95f80047c0 0x7f95f8004c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9600009a90 tx=0x7f95f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb9572c632f29514 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.575+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960002c6b0 con 0x7f95f80047c0 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.575+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f960002c810 con 0x7f95f80047c0 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.575+0000 7f9617400640 1 -- 172.21.15.53:0/1665557739 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f960002cba0 con 0x7f95f80047c0 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961da00640 1 -- 172.21.15.53:0/1665557739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95f80047c0 msgr2=0x7f95f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961da00640 1 --2- 172.21.15.53:0/1665557739 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f95f80047c0 0x7f95f8004c40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9600009a90 tx=0x7f95f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961d000640 1 -- 172.21.15.53:0/1665557739 reap_dead start 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961da00640 1 -- 172.21.15.53:0/1665557739 shutdown_connections 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961da00640 1 -- 172.21.15.53:0/1665557739 >> 172.21.15.53:0/1665557739 conn(0x7f961806a7d0 msgr2=0x7f961806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961da00640 1 -- 172.21.15.53:0/1665557739 shutdown_connections 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.576+0000 7f961da00640 1 -- 172.21.15.53:0/1665557739 wait complete. 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.579+0000 7f961da00640 1 Processor -- start 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.579+0000 7f961da00640 1 -- start start 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.580+0000 7f961da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95f80047c0 0x7f961813bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.580+0000 7f961da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9618145410 0x7f96180ccf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.580+0000 7f961da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 0x7f9618143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.580+0000 7f961da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96180cd900 con 0x7f96180cd480 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.580+0000 7f961da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961814f4e0 con 0x7f9618145410 2024-01-16T15:02:34.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.580+0000 7f961da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961814e8d0 con 0x7f95f80047c0 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 0x7f9618143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 0x7f9618143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50948/0 (socket says 172.21.15.53:50948) 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 -- 172.21.15.53:0/3173753477 learned_addr learned my addr 172.21.15.53:0/3173753477 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f9617e00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95f80047c0 0x7f961813bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f960fe00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9618145410 0x7f96180ccf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 -- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95f80047c0 msgr2=0x7f961813bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95f80047c0 0x7f961813bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 -- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9618145410 msgr2=0x7f96180ccf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9618145410 0x7f96180ccf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f961d000640 1 -- 172.21.15.53:0/3173753477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f961814ebb0 con 0x7f96180cd480 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.581+0000 7f960fe00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9618145410 0x7f96180ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.582+0000 7f961d000640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 0x7f9618143d30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9618147090 tx=0x7f961000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.582+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610017400 con 0x7f96180cd480 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.582+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9610011040 con 0x7f96180cd480 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.582+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9610017780 con 0x7f96180cd480 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.582+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f961814ee90 con 0x7f96180cd480 2024-01-16T15:02:34.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.582+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9618145890 con 0x7f96180cd480 2024-01-16T15:02:34.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.583+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9618067450 con 0x7f96180cd480 2024-01-16T15:02:34.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.584+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f96100203b0 con 0x7f96180cd480 2024-01-16T15:02:34.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.585+0000 7f9614c00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f006b470 0x7f95f006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:34.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.585+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f961008a1d0 con 0x7f96180cd480 2024-01-16T15:02:34.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.586+0000 7f9617e00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f006b470 0x7f95f006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:34.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.587+0000 7f9617e00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f006b470 0x7f95f006d930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9608005990 tx=0x7f9608009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:34.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.590+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9610054cb0 con 0x7f96180cd480 2024-01-16T15:02:34.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.825+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f961806ccd0 con 0x7f95f006b470 2024-01-16T15:02:34.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.840+0000 7f9614c00640 1 -- 172.21.15.53:0/3173753477 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f9608001650 con 0x7f95f006b470 2024-01-16T15:02:34.820 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:34.820 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:34.820 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:34.820 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:34.820 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:34.821 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:34.822 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f006b470 msgr2=0x7f95f006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f006b470 0x7f95f006d930 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9608005990 tx=0x7f9608009040 comp rx=0 tx=0).stop 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 msgr2=0x7f9618143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 0x7f9618143d30 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f9618147090 tx=0x7f961000f4a0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 shutdown_connections 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f006b470 0x7f95f006d930 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96180cd480 0x7f9618143d30 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9618145410 0x7f96180ccf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 --2- 172.21.15.53:0/3173753477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f95f80047c0 0x7f961813bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:34.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 >> 172.21.15.53:0/3173753477 conn(0x7f961806a7d0 msgr2=0x7f9618142a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:34.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 shutdown_connections 2024-01-16T15:02:34.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:34.842+0000 7f961da00640 1 -- 172.21.15.53:0/3173753477 wait complete. 2024-01-16T15:02:34.840 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:35.841 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:36.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.084+0000 7f0fd0000640 1 Processor -- start 2024-01-16T15:02:36.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.084+0000 7f0fd0000640 1 -- start start 2024-01-16T15:02:36.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fd0000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fc8072230 0x7f0fc80b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fd0000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc8071190 con 0x7f0fc8150760 2024-01-16T15:02:36.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fd0000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc8071470 con 0x7f0fc815da10 2024-01-16T15:02:36.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fd0000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc8071750 con 0x7f0fc8072230 2024-01-16T15:02:36.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fcd600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0fc8150760 0x7f0fc815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41562/0 (socket says 172.21.15.53:41562) 2024-01-16T15:02:36.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fcd600640 1 -- 172.21.15.53:0/3300430903 learned_addr learned my addr 172.21.15.53:0/3300430903 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:36.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fce000640 1 --2- 172.21.15.53:0/3300430903 >> v2:172.21.15.53:3300/0 conn(0x7f0fc8072230 0x7f0fc80b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fc7e00640 1 -- 172.21.15.53:0/3300430903 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450850409 0 0) 0x7f0fc8071190 con 0x7f0fc8150760 2024-01-16T15:02:36.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.086+0000 7f0fc7e00640 1 -- 172.21.15.53:0/3300430903 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa8004630 con 0x7f0fc8150760 2024-01-16T15:02:36.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.087+0000 7f0fce000640 1 -- 172.21.15.53:0/3300430903 >> v1:172.21.15.146:6789/0 conn(0x7f0fc815da10 legacy=0x7f0fc80b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:36.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.087+0000 7f0fce000640 1 -- 172.21.15.53:0/3300430903 >> v1:172.21.15.53:6789/0 conn(0x7f0fc8150760 legacy=0x7f0fc815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.087+0000 7f0fce000640 1 -- 172.21.15.53:0/3300430903 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc8071a30 con 0x7f0fc8072230 2024-01-16T15:02:36.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.088+0000 7f0fce000640 1 --2- 172.21.15.53:0/3300430903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc80b32e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009d90 tx=0x7f0fc002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574e391e43d83ce server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.088+0000 7f0fc7e00640 1 -- 172.21.15.53:0/3300430903 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc002cc60 con 0x7f0fc8072230 2024-01-16T15:02:36.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.088+0000 7f0fc7e00640 1 -- 172.21.15.53:0/3300430903 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc002cdc0 con 0x7f0fc8072230 2024-01-16T15:02:36.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.088+0000 7f0fc7e00640 1 -- 172.21.15.53:0/3300430903 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0034660 con 0x7f0fc8072230 2024-01-16T15:02:36.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 -- 172.21.15.53:0/3300430903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 msgr2=0x7f0fc80b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 --2- 172.21.15.53:0/3300430903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc80b32e0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f0fc0009d90 tx=0x7f0fc002beb0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 -- 172.21.15.53:0/3300430903 shutdown_connections 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 --2- 172.21.15.53:0/3300430903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc80b32e0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 -- 172.21.15.53:0/3300430903 >> 172.21.15.53:0/3300430903 conn(0x7f0fc806a800 msgr2=0x7f0fc806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 -- 172.21.15.53:0/3300430903 shutdown_connections 2024-01-16T15:02:36.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.089+0000 7f0fd0000640 1 -- 172.21.15.53:0/3300430903 wait complete. 2024-01-16T15:02:36.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.091+0000 7f0fd0000640 1 Processor -- start 2024-01-16T15:02:36.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.091+0000 7f0fd0000640 1 -- start start 2024-01-16T15:02:36.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fd0000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc8156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fd0000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fc8150760 0x7f0fc814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fd0000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fc815da10 0x7f0fc8149990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fd0000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc814d470 con 0x7f0fc8072230 2024-01-16T15:02:36.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fd0000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc80ced10 con 0x7f0fc815da10 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fd0000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc8149ed0 con 0x7f0fc8150760 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fcd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc8156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fcd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc8156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50970/0 (socket says 172.21.15.53:50970) 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.092+0000 7f0fcd600640 1 -- 172.21.15.53:0/1238976714 learned_addr learned my addr 172.21.15.53:0/1238976714 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fce000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fc815da10 0x7f0fc8149990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fccc00640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fc8150760 0x7f0fc814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fcd600640 1 -- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fc8150760 msgr2=0x7f0fc814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fcd600640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fc8150760 0x7f0fc814cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fcd600640 1 -- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fc815da10 msgr2=0x7f0fc8149990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fcd600640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fc815da10 0x7f0fc8149990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fcd600640 1 -- 172.21.15.53:0/1238976714 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc0009a70 con 0x7f0fc8072230 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.093+0000 7f0fcd600640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc8156ab0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800c500 tx=0x7f0fb800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.094+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8005ce0 con 0x7f0fc8072230 2024-01-16T15:02:36.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.094+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8005e40 con 0x7f0fc8072230 2024-01-16T15:02:36.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.094+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc80cbfe0 con 0x7f0fc8072230 2024-01-16T15:02:36.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.094+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8007330 con 0x7f0fc8072230 2024-01-16T15:02:36.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.094+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc80cc5a0 con 0x7f0fc8072230 2024-01-16T15:02:36.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.095+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc80675d0 con 0x7f0fc8072230 2024-01-16T15:02:36.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.096+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f0fb8007550 con 0x7f0fc8072230 2024-01-16T15:02:36.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.097+0000 7f0fc5600640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa006b470 0x7f0fa006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:36.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.097+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0fb8089560 con 0x7f0fc8072230 2024-01-16T15:02:36.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.098+0000 7f0fccc00640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa006b470 0x7f0fa006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:36.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.099+0000 7f0fccc00640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa006b470 0x7f0fa006d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fc81542f0 tx=0x7f0fb0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:36.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.102+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fb8054200 con 0x7f0fc8072230 2024-01-16T15:02:36.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.380+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fc8150b40 con 0x7f0fa006b470 2024-01-16T15:02:36.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.395+0000 7f0fc5600640 1 -- 172.21.15.53:0/1238976714 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f0fb000a380 con 0x7f0fa006b470 2024-01-16T15:02:36.375 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:36.375 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:36.375 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:36.376 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa006b470 msgr2=0x7f0fa006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa006b470 0x7f0fa006d930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fc81542f0 tx=0x7f0fb0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 msgr2=0x7f0fc8156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc8156ab0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f0fb800c500 tx=0x7f0fb800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 shutdown_connections 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa006b470 0x7f0fa006d930 secure :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0fc81542f0 tx=0x7f0fb0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fc815da10 0x7f0fc8149990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fc8150760 0x7f0fc814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 --2- 172.21.15.53:0/1238976714 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fc8072230 0x7f0fc8156ab0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:36.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.401+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 >> 172.21.15.53:0/1238976714 conn(0x7f0fc806a800 msgr2=0x7f0fc806ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:36.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.405+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 shutdown_connections 2024-01-16T15:02:36.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:36.405+0000 7f0fd0000640 1 -- 172.21.15.53:0/1238976714 wait complete. 2024-01-16T15:02:36.412 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:37.413 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:37.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.712+0000 7f313e200640 1 Processor -- start 2024-01-16T15:02:37.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.715+0000 7f313e200640 1 -- start start 2024-01-16T15:02:37.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.717+0000 7f313e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.717+0000 7f313e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f31380cd840 0x7f3138140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.717+0000 7f313e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f31380cdc80 0x7f3138140da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.717+0000 7f313e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380ce220 con 0x7f31380cdc80 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.717+0000 7f313e200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31380ce500 con 0x7f31380cd840 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.717+0000 7f313e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31380ce7e0 con 0x7f313813df80 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.718+0000 7f3137e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f31380cdc80 0x7f3138140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.718+0000 7f3137e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f31380cdc80 0x7f3138140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50986/0 (socket says 172.21.15.53:50986) 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.718+0000 7f3137e00640 1 -- 172.21.15.53:0/2960342947 learned_addr learned my addr 172.21.15.53:0/2960342947 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.718+0000 7f3137400640 1 --2- 172.21.15.53:0/2960342947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.719+0000 7f3137e00640 1 -- 172.21.15.53:0/2960342947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 msgr2=0x7f313815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.719+0000 7f3137e00640 1 --2- 172.21.15.53:0/2960342947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.719+0000 7f3137e00640 1 -- 172.21.15.53:0/2960342947 >> v2:172.21.15.146:3300/0 conn(0x7f31380cd840 msgr2=0x7f3138140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.719+0000 7f3137e00640 1 --2- 172.21.15.53:0/2960342947 >> v2:172.21.15.146:3300/0 conn(0x7f31380cd840 0x7f3138140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.719+0000 7f3137e00640 1 -- 172.21.15.53:0/2960342947 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31380ceac0 con 0x7f31380cdc80 2024-01-16T15:02:37.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.720+0000 7f3137e00640 1 --2- 172.21.15.53:0/2960342947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cdc80 0x7f3138140da0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f31380b43a0 tx=0x7f312802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32a6cf360068bcc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.720+0000 7f3136000640 1 -- 172.21.15.53:0/2960342947 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128031880 con 0x7f31380cdc80 2024-01-16T15:02:37.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.720+0000 7f3136000640 1 -- 172.21.15.53:0/2960342947 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31280319e0 con 0x7f31380cdc80 2024-01-16T15:02:37.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.720+0000 7f3136000640 1 -- 172.21.15.53:0/2960342947 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3128031d70 con 0x7f31380cdc80 2024-01-16T15:02:37.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.721+0000 7f313e200640 1 -- 172.21.15.53:0/2960342947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cdc80 msgr2=0x7f3138140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.721+0000 7f313e200640 1 --2- 172.21.15.53:0/2960342947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cdc80 0x7f3138140da0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f31380b43a0 tx=0x7f312802b920 comp rx=0 tx=0).stop 2024-01-16T15:02:37.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 -- 172.21.15.53:0/2960342947 shutdown_connections 2024-01-16T15:02:37.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 --2- 172.21.15.53:0/2960342947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cdc80 0x7f3138140da0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 --2- 172.21.15.53:0/2960342947 >> v2:172.21.15.146:3300/0 conn(0x7f31380cd840 0x7f3138140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 --2- 172.21.15.53:0/2960342947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 -- 172.21.15.53:0/2960342947 >> 172.21.15.53:0/2960342947 conn(0x7f31380b0be0 msgr2=0x7f31380b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 -- 172.21.15.53:0/2960342947 shutdown_connections 2024-01-16T15:02:37.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.722+0000 7f313e200640 1 -- 172.21.15.53:0/2960342947 wait complete. 2024-01-16T15:02:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.725+0000 7f313e200640 1 Processor -- start 2024-01-16T15:02:37.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.725+0000 7f313e200640 1 -- start start 2024-01-16T15:02:37.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f313e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 0x7f31381320d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f313e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31380cdc80 0x7f3138144680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f313e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f313e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138132610 con 0x7f31380cd840 2024-01-16T15:02:37.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f313e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3138141e10 con 0x7f31380cdc80 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f313e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f313813fd70 con 0x7f313813df80 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.727+0000 7f3137400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 0x7f31381320d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 0x7f31381320d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50996/0 (socket says 172.21.15.53:50996) 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 -- 172.21.15.53:0/2166624618 learned_addr learned my addr 172.21.15.53:0/2166624618 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 -- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 msgr2=0x7f313815a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 -- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31380cdc80 msgr2=0x7f3138144680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31380cdc80 0x7f3138144680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:37.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 -- 172.21.15.53:0/2166624618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31280094d0 con 0x7f31380cd840 2024-01-16T15:02:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.728+0000 7f3137400640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 0x7f31381320d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3130007e40 tx=0x7f313000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.729+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3130014620 con 0x7f31380cd840 2024-01-16T15:02:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.729+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313000ee90 con 0x7f31380cd840 2024-01-16T15:02:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.729+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3138140050 con 0x7f31380cd840 2024-01-16T15:02:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.729+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f313813bf10 con 0x7f31380cd840 2024-01-16T15:02:37.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.729+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31300149c0 con 0x7f31380cd840 2024-01-16T15:02:37.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.731+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3108005250 con 0x7f31380cd840 2024-01-16T15:02:37.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.732+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f3130014b20 con 0x7f31380cd840 2024-01-16T15:02:37.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.733+0000 7f312f400640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f311006b470 0x7f311006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:37.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.734+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f313008a710 con 0x7f31380cd840 2024-01-16T15:02:37.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.736+0000 7f3136a00640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f311006b470 0x7f311006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:37.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.738+0000 7f3136a00640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f311006b470 0x7f311006d930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3138070630 tx=0x7f3120006c90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:37.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:37.741+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3130055390 con 0x7f31380cd840 2024-01-16T15:02:38.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.117+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3108000bc0 con 0x7f311006b470 2024-01-16T15:02:38.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.129+0000 7f312f400640 1 -- 172.21.15.53:0/2166624618 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f3108000bc0 con 0x7f311006b470 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:02:38.109 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:02:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:02:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:38.110 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:38.110 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:38.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f311006b470 msgr2=0x7f311006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f311006b470 0x7f311006d930 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3138070630 tx=0x7f3120006c90 comp rx=0 tx=0).stop 2024-01-16T15:02:38.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 msgr2=0x7f31381320d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 0x7f31381320d0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f3130007e40 tx=0x7f313000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 shutdown_connections 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f311006b470 0x7f311006d930 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f313813df80 0x7f313815a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f31380cdc80 0x7f3138144680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 --2- 172.21.15.53:0/2166624618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f31380cd840 0x7f31381320d0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 >> 172.21.15.53:0/2166624618 conn(0x7f31380b0be0 msgr2=0x7f3138071580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 shutdown_connections 2024-01-16T15:02:38.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:38.134+0000 7f313e200640 1 -- 172.21.15.53:0/2166624618 wait complete. 2024-01-16T15:02:38.135 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:02:39.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:02:39.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.395+0000 7fbdb5c00640 1 Processor -- start 2024-01-16T15:02:39.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.395+0000 7fbdb5c00640 1 -- start start 2024-01-16T15:02:39.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.396+0000 7fbdb5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 0x7fbdb0161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdb5c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdb00ce8f0 0x7fbdb00ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdb5c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdb01618a0 con 0x7fbdb00cfbe0 2024-01-16T15:02:39.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdb5c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdb00d0de0 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdb5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb00d10c0 con 0x7fbdb0160f80 2024-01-16T15:02:39.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbda5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbdb00ce8f0 0x7fbdb00ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdaea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 0x7fbdb0161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdaea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 0x7fbdb0161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49902/0 (socket says 172.21.15.53:49902) 2024-01-16T15:02:39.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.397+0000 7fbdaea00640 1 -- 172.21.15.53:0/284011142 learned_addr learned my addr 172.21.15.53:0/284011142 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:39.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.398+0000 7fbda5e00640 1 -- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 msgr2=0x7fbdb0161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:39.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.398+0000 7fbda5e00640 1 --2- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 0x7fbdb0161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.398+0000 7fbda5e00640 1 -- 172.21.15.53:0/284011142 >> v1:172.21.15.146:6789/0 conn(0x7fbdb00cfbe0 legacy=0x7fbdb00d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.398+0000 7fbda5e00640 1 -- 172.21.15.53:0/284011142 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb00d13a0 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.398+0000 7fbdaea00640 1 --2- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 0x7fbdb0161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:02:39.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.400+0000 7fbda5e00640 1 --2- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb00ced50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd980097a0 tx=0x7fbd9802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d903f652c2351c16 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.400+0000 7fbdae000640 1 -- 172.21.15.53:0/284011142 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9802d040 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.400+0000 7fbdae000640 1 -- 172.21.15.53:0/284011142 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd98031810 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.400+0000 7fbdae000640 1 -- 172.21.15.53:0/284011142 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd98031ba0 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 -- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb00ce8f0 msgr2=0x7fbdb00ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb00ced50 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbd980097a0 tx=0x7fbd9802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 -- 172.21.15.53:0/284011142 shutdown_connections 2024-01-16T15:02:39.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb00ced50 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/284011142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb0160f80 0x7fbdb0161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 -- 172.21.15.53:0/284011142 >> 172.21.15.53:0/284011142 conn(0x7fbdb009c550 msgr2=0x7fbdb00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:39.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 -- 172.21.15.53:0/284011142 shutdown_connections 2024-01-16T15:02:39.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.401+0000 7fbdb5c00640 1 -- 172.21.15.53:0/284011142 wait complete. 2024-01-16T15:02:39.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.403+0000 7fbdb5c00640 1 Processor -- start 2024-01-16T15:02:39.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.403+0000 7fbdb5c00640 1 -- start start 2024-01-16T15:02:39.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.404+0000 7fbdb5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb01557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.404+0000 7fbdb5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb00cfbe0 0x7fbdb00b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.404+0000 7fbdb5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb0160f80 0x7fbdb00b5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.404+0000 7fbdb5c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb01567e0 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.404+0000 7fbdb5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb015f280 con 0x7fbdb0160f80 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.404+0000 7fbdb5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdb015efa0 con 0x7fbdb00cfbe0 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb01557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb01557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51012/0 (socket says 172.21.15.53:51012) 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 -- 172.21.15.53:0/2032977011 learned_addr learned my addr 172.21.15.53:0/2032977011 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbda5e00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb00cfbe0 0x7fbdb00b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaf400640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb0160f80 0x7fbdb00b5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 -- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb00cfbe0 msgr2=0x7fbdb00b4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb00cfbe0 0x7fbdb00b4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 -- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb0160f80 msgr2=0x7fbdb00b5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:39.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb0160f80 0x7fbdb00b5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:39.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 -- 172.21.15.53:0/2032977011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda0009b30 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.405+0000 7fbdaea00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb01557f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009f20 tx=0x7fbda000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.406+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda0014480 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.406+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbda000f040 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.406+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbda00147e0 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.406+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd98009400 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.406+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdb0156470 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.407+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdb00675d0 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.408+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fbda0014940 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.409+0000 7fbda7400640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd8806b470 0x7fbd8806d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:39.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.409+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbda008a950 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.410+0000 7fbda5e00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd8806b470 0x7fbd8806d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:39.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.411+0000 7fbda5e00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd8806b470 0x7fbd8806d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd98036000 tx=0x7fbd9803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:39.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.414+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbda0055420 con 0x7fbdb00ce8f0 2024-01-16T15:02:39.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:39.646+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdb0161a30 con 0x7fbd8806b470 2024-01-16T15:02:40.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.675+0000 7fbda7400640 1 -- 172.21.15.53:0/2032977011 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fbdb0161a30 con 0x7fbd8806b470 2024-01-16T15:02:40.655 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:02:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:02:40.655 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:40.656 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd8806b470 msgr2=0x7fbd8806d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd8806b470 0x7fbd8806d930 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbd98036000 tx=0x7fbd9803c040 comp rx=0 tx=0).stop 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 msgr2=0x7fbdb01557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb01557f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbda0009f20 tx=0x7fbda000c550 comp rx=0 tx=0).stop 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 shutdown_connections 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd8806b470 0x7fbd8806d930 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdb0160f80 0x7fbdb00b5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdb00cfbe0 0x7fbdb00b4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 --2- 172.21.15.53:0/2032977011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdb00ce8f0 0x7fbdb01557f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 >> 172.21.15.53:0/2032977011 conn(0x7fbdb009c550 msgr2=0x7fbdb00d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:40.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 shutdown_connections 2024-01-16T15:02:40.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.679+0000 7fbdb5c00640 1 -- 172.21.15.53:0/2032977011 wait complete. 2024-01-16T15:02:40.679 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:02:40.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.923+0000 7f925b400640 1 Processor -- start 2024-01-16T15:02:40.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.924+0000 7f925b400640 1 -- start start 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9254160920 0x7f92540b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92540cf9f0 0x7f92540cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925b400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92540d0a10 con 0x7f9254161010 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92540d03e0 con 0x7f9254160920 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925b400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92540d1f50 con 0x7f92540cf9f0 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9254160920 0x7f92540b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9254160920 0x7f92540b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51020/0 (socket says 172.21.15.53:51020) 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.925+0000 7f925a000640 1 -- 172.21.15.53:0/3841649158 learned_addr learned my addr 172.21.15.53:0/3841649158 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.926+0000 7f9259600640 1 --2- 172.21.15.53:0/3841649158 >> v2:172.21.15.146:3300/0 conn(0x7f92540cf9f0 0x7f92540cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.926+0000 7f925a000640 1 -- 172.21.15.53:0/3841649158 >> v2:172.21.15.146:3300/0 conn(0x7f92540cf9f0 msgr2=0x7f92540cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.926+0000 7f925a000640 1 --2- 172.21.15.53:0/3841649158 >> v2:172.21.15.146:3300/0 conn(0x7f92540cf9f0 0x7f92540cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.926+0000 7f925a000640 1 -- 172.21.15.53:0/3841649158 >> v1:172.21.15.146:6789/0 conn(0x7f9254161010 legacy=0x7f9254161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:40.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.926+0000 7f925a000640 1 -- 172.21.15.53:0/3841649158 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92540cf080 con 0x7f9254160920 2024-01-16T15:02:40.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.927+0000 7f925a000640 1 --2- 172.21.15.53:0/3841649158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9254160920 0x7f92540b3f30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f925408c1f0 tx=0x7f923c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=748672b17b8fbb94 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:40.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.927+0000 7f9258c00640 1 -- 172.21.15.53:0/3841649158 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c031870 con 0x7f9254160920 2024-01-16T15:02:40.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.927+0000 7f9258c00640 1 -- 172.21.15.53:0/3841649158 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c0319d0 con 0x7f9254160920 2024-01-16T15:02:40.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.927+0000 7f9258c00640 1 -- 172.21.15.53:0/3841649158 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c031d60 con 0x7f9254160920 2024-01-16T15:02:40.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 -- 172.21.15.53:0/3841649158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9254160920 msgr2=0x7f92540b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 --2- 172.21.15.53:0/3841649158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9254160920 0x7f92540b3f30 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f925408c1f0 tx=0x7f923c02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:40.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 -- 172.21.15.53:0/3841649158 shutdown_connections 2024-01-16T15:02:40.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 --2- 172.21.15.53:0/3841649158 >> v2:172.21.15.146:3300/0 conn(0x7f92540cf9f0 0x7f92540cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 --2- 172.21.15.53:0/3841649158 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9254160920 0x7f92540b3f30 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 -- 172.21.15.53:0/3841649158 >> 172.21.15.53:0/3841649158 conn(0x7f92540d3090 msgr2=0x7f92540d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:40.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 -- 172.21.15.53:0/3841649158 shutdown_connections 2024-01-16T15:02:40.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.928+0000 7f925b400640 1 -- 172.21.15.53:0/3841649158 wait complete. 2024-01-16T15:02:40.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.930+0000 7f925b400640 1 Processor -- start 2024-01-16T15:02:40.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.930+0000 7f925b400640 1 -- start start 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 0x7f9254157970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9254160920 0x7f92540cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9254161010 0x7f92540cb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92540cbe00 con 0x7f92540cf9f0 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92540cc0e0 con 0x7f9254161010 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92540cc3c0 con 0x7f9254160920 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 0x7f9254157970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 0x7f9254157970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51032/0 (socket says 172.21.15.53:51032) 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.931+0000 7f925a000640 1 -- 172.21.15.53:0/504261133 learned_addr learned my addr 172.21.15.53:0/504261133 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925aa00640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9254161010 0x7f92540cb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f9259600640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9254160920 0x7f92540cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925a000640 1 -- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9254160920 msgr2=0x7f92540cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925a000640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9254160920 0x7f92540cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925a000640 1 -- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9254161010 msgr2=0x7f92540cb7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925a000640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9254161010 0x7f92540cb7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925a000640 1 -- 172.21.15.53:0/504261133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f923c0094d0 con 0x7f92540cf9f0 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925aa00640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9254161010 0x7f92540cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925a000640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 0x7f9254157970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f923c02b950 tx=0x7f923c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c03ad30 con 0x7f92540cf9f0 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923c0443f0 con 0x7f92540cf9f0 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923c0447a0 con 0x7f92540cf9f0 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.932+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92540cc6a0 con 0x7f92540cf9f0 2024-01-16T15:02:40.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.933+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92540b8a70 con 0x7f92540cf9f0 2024-01-16T15:02:40.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.934+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92540ccbc0 con 0x7f92540cf9f0 2024-01-16T15:02:40.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.935+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f923c04d450 con 0x7f92540cf9f0 2024-01-16T15:02:40.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.936+0000 7f9252000640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b470 0x7f922c06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:40.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.936+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f923c02d040 con 0x7f92540cf9f0 2024-01-16T15:02:40.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.936+0000 7f9259600640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b470 0x7f922c06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:40.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.937+0000 7f9259600640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b470 0x7f922c06d930 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92540b9010 tx=0x7f924400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:40.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:40.940+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92540ccbc0 con 0x7f92540cf9f0 2024-01-16T15:02:41.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.160+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f92540cc250 con 0x7f922c06b470 2024-01-16T15:02:41.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.164+0000 7f9252000640 1 -- 172.21.15.53:0/504261133 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f92540cc250 con 0x7f922c06b470 2024-01-16T15:02:41.143 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:02:41.143 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:41.143 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:02:41.144 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:41.144 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:02:41.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b470 msgr2=0x7f922c06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b470 0x7f922c06d930 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92540b9010 tx=0x7f924400b040 comp rx=0 tx=0).stop 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 msgr2=0x7f9254157970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 0x7f9254157970 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f923c02b950 tx=0x7f923c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 shutdown_connections 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f922c06b470 0x7f922c06d930 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9254161010 0x7f92540cb7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9254160920 0x7f92540cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 --2- 172.21.15.53:0/504261133 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92540cf9f0 0x7f9254157970 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 >> 172.21.15.53:0/504261133 conn(0x7f92540d3090 msgr2=0x7f925406c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 shutdown_connections 2024-01-16T15:02:41.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.167+0000 7f925b400640 1 -- 172.21.15.53:0/504261133 wait complete. 2024-01-16T15:02:41.165 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:02:41.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.410+0000 7f794d600640 1 Processor -- start 2024-01-16T15:02:41.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.411+0000 7f794d600640 1 -- start start 2024-01-16T15:02:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.412+0000 7f794d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813e360 0x7f79480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.412+0000 7f794d600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794815c5f0 con 0x7f794815f9c0 2024-01-16T15:02:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.412+0000 7f794d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f794806c470 con 0x7f794813df80 2024-01-16T15:02:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.412+0000 7f794d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f794806c750 con 0x7f794813e360 2024-01-16T15:02:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.413+0000 7f7947400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f794815f9c0 0x7f79480cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41572/0 (socket says 172.21.15.53:41572) 2024-01-16T15:02:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.413+0000 7f7947400640 1 -- 172.21.15.53:0/991054865 learned_addr learned my addr 172.21.15.53:0/991054865 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.413+0000 7f793fe00640 1 --2- 172.21.15.53:0/991054865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813e360 0x7f79480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.413+0000 7f7946000640 1 -- 172.21.15.53:0/991054865 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688161339 0 0) 0x7f794815c5f0 con 0x7f794815f9c0 2024-01-16T15:02:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.413+0000 7f7946000640 1 -- 172.21.15.53:0/991054865 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7928003520 con 0x7f794815f9c0 2024-01-16T15:02:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.414+0000 7f7946000640 1 -- 172.21.15.53:0/991054865 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857412795 0 0) 0x7f7928003520 con 0x7f794815f9c0 2024-01-16T15:02:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.414+0000 7f793fe00640 1 -- 172.21.15.53:0/991054865 >> v1:172.21.15.146:6789/0 conn(0x7f794813df80 legacy=0x7f794815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.414+0000 7f793fe00640 1 -- 172.21.15.53:0/991054865 >> v1:172.21.15.53:6789/0 conn(0x7f794815f9c0 legacy=0x7f79480cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.414+0000 7f793fe00640 1 -- 172.21.15.53:0/991054865 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f794806ca30 con 0x7f794813e360 2024-01-16T15:02:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.415+0000 7f793fe00640 1 --2- 172.21.15.53:0/991054865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813e360 0x7f79480cf290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79300097d0 tx=0x7f793002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=120c245aee1d6778 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.415+0000 7f7946000640 1 -- 172.21.15.53:0/991054865 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79400019f0 con 0x7f794813e360 2024-01-16T15:02:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f7946000640 1 -- 172.21.15.53:0/991054865 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f793002e050 con 0x7f794813e360 2024-01-16T15:02:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f7946000640 1 -- 172.21.15.53:0/991054865 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930030730 con 0x7f794813e360 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 -- 172.21.15.53:0/991054865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813e360 msgr2=0x7f79480cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 --2- 172.21.15.53:0/991054865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813e360 0x7f79480cf290 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f79300097d0 tx=0x7f793002b800 comp rx=0 tx=0).stop 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 -- 172.21.15.53:0/991054865 shutdown_connections 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 --2- 172.21.15.53:0/991054865 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813e360 0x7f79480cf290 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 -- 172.21.15.53:0/991054865 >> 172.21.15.53:0/991054865 conn(0x7f794806a800 msgr2=0x7f794806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 -- 172.21.15.53:0/991054865 shutdown_connections 2024-01-16T15:02:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.416+0000 7f794d600640 1 -- 172.21.15.53:0/991054865 wait complete. 2024-01-16T15:02:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.418+0000 7f794d600640 1 Processor -- start 2024-01-16T15:02:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.418+0000 7f794d600640 1 -- start start 2024-01-16T15:02:41.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f794d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813df80 0x7f79481f0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f794d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f794813e360 0x7f794814cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f794d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 0x7f794814d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f794d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79481f09c0 con 0x7f794815f9c0 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f794d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f794814da50 con 0x7f794813e360 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f794d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79481f16b0 con 0x7f794813df80 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.419+0000 7f7947400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 0x7f794814d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 0x7f794814d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51034/0 (socket says 172.21.15.53:51034) 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 -- 172.21.15.53:0/54174439 learned_addr learned my addr 172.21.15.53:0/54174439 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f793fe00640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f794813e360 0x7f794814cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7946a00640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813df80 0x7f79481f0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 -- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813df80 msgr2=0x7f79481f0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813df80 0x7f79481f0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 -- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f794813e360 msgr2=0x7f794814cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f794813e360 0x7f794814cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 -- 172.21.15.53:0/54174439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930009400 con 0x7f794815f9c0 2024-01-16T15:02:41.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.420+0000 7f7947400640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 0x7f794814d510 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7940001890 tx=0x7f794000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.421+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7940012400 con 0x7f794815f9c0 2024-01-16T15:02:41.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.421+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7940012560 con 0x7f794815f9c0 2024-01-16T15:02:41.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.421+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7940012910 con 0x7f794815f9c0 2024-01-16T15:02:41.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.421+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79481f1990 con 0x7f794815f9c0 2024-01-16T15:02:41.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.421+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79481f1f50 con 0x7f794815f9c0 2024-01-16T15:02:41.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.422+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79480675d0 con 0x7f794815f9c0 2024-01-16T15:02:41.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.423+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f7940012a70 con 0x7f794815f9c0 2024-01-16T15:02:41.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.424+0000 7f793ea00640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f792006b4a0 0x7f792006d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.424+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f794008a680 con 0x7f794815f9c0 2024-01-16T15:02:41.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.424+0000 7f7946a00640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f792006b4a0 0x7f792006d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.426+0000 7f7946a00640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f792006b4a0 0x7f792006d960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7938000f60 tx=0x7f7938009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.429+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7940055300 con 0x7f794815f9c0 2024-01-16T15:02:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.648+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f794806cff0 con 0x7f792006b4a0 2024-01-16T15:02:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.658+0000 7f793ea00640 1 -- 172.21.15.53:0/54174439 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f79380050d0 con 0x7f792006b4a0 2024-01-16T15:02:41.637 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:41.637 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:02:41.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:41.638 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:02:41.638 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:41.638 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:41.638 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1clone1" 2024-01-16T15:02:41.638 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:41.638 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:02:41.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f792006b4a0 msgr2=0x7f792006d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f792006b4a0 0x7f792006d960 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f7938000f60 tx=0x7f7938009040 comp rx=0 tx=0).stop 2024-01-16T15:02:41.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 msgr2=0x7f794814d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 0x7f794814d510 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f7940001890 tx=0x7f794000a430 comp rx=0 tx=0).stop 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 shutdown_connections 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f792006b4a0 0x7f792006d960 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f794815f9c0 0x7f794814d510 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f794813e360 0x7f794814cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 --2- 172.21.15.53:0/54174439 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f794813df80 0x7f79481f0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 >> 172.21.15.53:0/54174439 conn(0x7f794806a800 msgr2=0x7f794815e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 shutdown_connections 2024-01-16T15:02:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.661+0000 7f794d600640 1 -- 172.21.15.53:0/54174439 wait complete. 2024-01-16T15:02:41.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:02:41.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.900+0000 7f6303e00640 1 Processor -- start 2024-01-16T15:02:41.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.900+0000 7f6303e00640 1 -- start start 2024-01-16T15:02:41.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f6303e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f6304071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f6303e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63040717d0 0x7f63040cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f6303e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6304070e50 con 0x7f63040cca40 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f6303e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63040723c0 con 0x7f63040717d0 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f6303e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63040cce70 con 0x7f6304143c40 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f62fbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63040717d0 0x7f63040cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f62fbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63040717d0 0x7f63040cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49846/0 (socket says 172.21.15.53:49846) 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f62fbe00640 1 -- 172.21.15.53:0/3470668285 learned_addr learned my addr 172.21.15.53:0/3470668285 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:41.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.902+0000 7f6302a00640 1 --2- 172.21.15.53:0/3470668285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f6304071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.903+0000 7f62fbe00640 1 -- 172.21.15.53:0/3470668285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 msgr2=0x7f6304071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:41.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.903+0000 7f62fbe00640 1 --2- 172.21.15.53:0/3470668285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f6304071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.903+0000 7f62fbe00640 1 -- 172.21.15.53:0/3470668285 >> v1:172.21.15.146:6789/0 conn(0x7f63040cca40 legacy=0x7f630415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.903+0000 7f62fbe00640 1 -- 172.21.15.53:0/3470668285 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f630406e170 con 0x7f63040717d0 2024-01-16T15:02:41.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.904+0000 7f62fbe00640 1 --2- 172.21.15.53:0/3470668285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f63040cc500 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009870 tx=0x7f62ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edb99d36b326500 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.904+0000 7f6302000640 1 -- 172.21.15.53:0/3470668285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec031640 con 0x7f63040717d0 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.904+0000 7f6302000640 1 -- 172.21.15.53:0/3470668285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62ec0317a0 con 0x7f63040717d0 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.904+0000 7f6302000640 1 -- 172.21.15.53:0/3470668285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec031b30 con 0x7f63040717d0 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 -- 172.21.15.53:0/3470668285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 msgr2=0x7f63040cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 --2- 172.21.15.53:0/3470668285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f63040cc500 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009870 tx=0x7f62ec02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 -- 172.21.15.53:0/3470668285 shutdown_connections 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 --2- 172.21.15.53:0/3470668285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f63040cc500 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 --2- 172.21.15.53:0/3470668285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f6304071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 -- 172.21.15.53:0/3470668285 >> 172.21.15.53:0/3470668285 conn(0x7f630406a800 msgr2=0x7f630406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 -- 172.21.15.53:0/3470668285 shutdown_connections 2024-01-16T15:02:41.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.905+0000 7f6303e00640 1 -- 172.21.15.53:0/3470668285 wait complete. 2024-01-16T15:02:41.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.907+0000 7f6303e00640 1 Processor -- start 2024-01-16T15:02:41.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.907+0000 7f6303e00640 1 -- start start 2024-01-16T15:02:41.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6303e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f6304142a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6303e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63040cca40 0x7f63040ca180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6303e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f63040ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6303e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f630412fe20 con 0x7f63040717d0 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6303e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63040cad40 con 0x7f63040cca40 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6303e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f630412f7d0 con 0x7f6304143c40 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6302a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f6304142a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6302a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f6304142a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49858/0 (socket says 172.21.15.53:49858) 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.908+0000 7f6302a00640 1 -- 172.21.15.53:0/3023334558 learned_addr learned my addr 172.21.15.53:0/3023334558 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f62fbe00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63040cca40 0x7f63040ca180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6302a00640 1 -- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 msgr2=0x7f63040ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6303400640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f63040ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6302a00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f63040ca6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6302a00640 1 -- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63040cca40 msgr2=0x7f63040ca180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:41.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6302a00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63040cca40 0x7f63040ca180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6302a00640 1 -- 172.21.15.53:0/3023334558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec0094d0 con 0x7f63040717d0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f62fbe00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63040cca40 0x7f63040ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6302a00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f6304142a00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f62f400e870 tx=0x7f62f400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f4009510 con 0x7f63040717d0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f400eec0 con 0x7f63040717d0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62f400cb80 con 0x7f63040717d0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.909+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f630412fab0 con 0x7f63040717d0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.910+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63041302b0 con 0x7f63040717d0 2024-01-16T15:02:41.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.911+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6304067450 con 0x7f63040717d0 2024-01-16T15:02:41.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.912+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f62f400cd20 con 0x7f63040717d0 2024-01-16T15:02:41.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.912+0000 7f62faa00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62dc06b470 0x7f62dc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:41.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.913+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62f4089fa0 con 0x7f63040717d0 2024-01-16T15:02:41.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.913+0000 7f62fbe00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62dc06b470 0x7f62dc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:41.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.914+0000 7f62fbe00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62dc06b470 0x7f62dc06d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f62ec036000 tx=0x7f62ec03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:41.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:41.917+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62f40bce90 con 0x7f63040717d0 2024-01-16T15:02:42.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.142+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63040720e0 con 0x7f62dc06b470 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.177+0000 7f62faa00640 1 -- 172.21.15.53:0/3023334558 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f62ec031ad0 con 0x7f62dc06b470 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:42.157 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62dc06b470 msgr2=0x7f62dc06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62dc06b470 0x7f62dc06d930 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f62ec036000 tx=0x7f62ec03c040 comp rx=0 tx=0).stop 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 msgr2=0x7f6304142a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f6304142a00 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f62f400e870 tx=0x7f62f400ec70 comp rx=0 tx=0).stop 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 shutdown_connections 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62dc06b470 0x7f62dc06d930 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6304143c40 0x7f63040ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63040cca40 0x7f63040ca180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 --2- 172.21.15.53:0/3023334558 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63040717d0 0x7f6304142a00 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 >> 172.21.15.53:0/3023334558 conn(0x7f630406a800 msgr2=0x7f630406d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 shutdown_connections 2024-01-16T15:02:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.181+0000 7f6303e00640 1 -- 172.21.15.53:0/3023334558 wait complete. 2024-01-16T15:02:42.178 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:02:42.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.414+0000 7fef2b200640 1 Processor -- start 2024-01-16T15:02:42.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.415+0000 7fef2b200640 1 -- start start 2024-01-16T15:02:42.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef2c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef2b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c0d0a10 con 0x7fef2c0cf9f0 2024-01-16T15:02:42.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef2b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef2c161e30 con 0x7fef2c0cfe70 2024-01-16T15:02:42.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0d1f50 con 0x7fef2c160920 2024-01-16T15:02:42.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef23e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fef2c0cf9f0 0x7fef2c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54418/0 (socket says 172.21.15.53:54418) 2024-01-16T15:02:42.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef23e00640 1 -- 172.21.15.53:0/1565440478 learned_addr learned my addr 172.21.15.53:0/1565440478 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:42.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.416+0000 7fef29e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef2c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727090427 0 0) 0x7fef2c0d0a10 con 0x7fef2c0cf9f0 2024-01-16T15:02:42.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef0c003520 con 0x7fef2c0cf9f0 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2644105507 0 0) 0x7fef0c003520 con 0x7fef2c0cf9f0 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 msgr2=0x7fef2c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef2c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 >> v1:172.21.15.146:6789/0 conn(0x7fef2c0cfe70 legacy=0x7fef2c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0cf080 con 0x7fef2c0cf9f0 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fef14001670 con 0x7fef2c0cf9f0 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 >> v1:172.21.15.53:6789/0 conn(0x7fef2c0cf9f0 legacy=0x7fef2c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef0c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef0c004d00 0x7fef0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0d1f50 con 0x7fef0c0039a0 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c00ba30 con 0x7fef0c004d00 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.417+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c00bba0 con 0x7fef2c160920 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef29e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef0c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef23e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef2a800640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef0c004d00 0x7fef0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef29e00640 1 -- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 msgr2=0x7fef0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef29e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef0c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef29e00640 1 -- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef0c004d00 msgr2=0x7fef0c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef29e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef0c004d00 0x7fef0c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef29e00640 1 -- 172.21.15.53:0/1565440478 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c003520 con 0x7fef0c0039a0 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef23e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c160920 0x7fef0c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:42.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.418+0000 7fef2a800640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef0c004d00 0x7fef0c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:42.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.419+0000 7fef29e00640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef0c003d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009ac0 tx=0x7fef0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229620c052285e46 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.419+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c02c6b0 con 0x7fef0c0039a0 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.419+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c02c810 con 0x7fef0c0039a0 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.419+0000 7fef29400640 1 -- 172.21.15.53:0/1565440478 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c02cba0 con 0x7fef0c0039a0 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2b200640 1 -- 172.21.15.53:0/1565440478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 msgr2=0x7fef0c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2b200640 1 --2- 172.21.15.53:0/1565440478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef0c003d80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fef1c009ac0 tx=0x7fef0c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2a800640 1 -- 172.21.15.53:0/1565440478 reap_dead start 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2b200640 1 -- 172.21.15.53:0/1565440478 shutdown_connections 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2b200640 1 -- 172.21.15.53:0/1565440478 >> 172.21.15.53:0/1565440478 conn(0x7fef2c0d3090 msgr2=0x7fef2c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2b200640 1 -- 172.21.15.53:0/1565440478 shutdown_connections 2024-01-16T15:02:42.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.420+0000 7fef2b200640 1 -- 172.21.15.53:0/1565440478 wait complete. 2024-01-16T15:02:42.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.422+0000 7fef2b200640 1 Processor -- start 2024-01-16T15:02:42.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.422+0000 7fef2b200640 1 -- start start 2024-01-16T15:02:42.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef2b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef2c14d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c159210 0x7fef2c159690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef2b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef2c0ca1d0 0x7fef2c0ca680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef2b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0cbb80 con 0x7fef0c0039a0 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef2b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c159bd0 con 0x7fef2c0ca1d0 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef2c0caf00 con 0x7fef2c159210 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef29e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef2c14d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef29e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef2c14d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49866/0 (socket says 172.21.15.53:49866) 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.423+0000 7fef29e00640 1 -- 172.21.15.53:0/381956419 learned_addr learned my addr 172.21.15.53:0/381956419 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef23e00640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c159210 0x7fef2c159690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef29e00640 1 -- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c159210 msgr2=0x7fef2c159690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef29e00640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c159210 0x7fef2c159690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef29e00640 1 -- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef2c0ca1d0 msgr2=0x7fef2c0ca680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef2a800640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef2c0ca1d0 0x7fef2c0ca680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef29e00640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef2c0ca1d0 0x7fef2c0ca680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef29e00640 1 -- 172.21.15.53:0/381956419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2c0cb1e0 con 0x7fef0c0039a0 2024-01-16T15:02:42.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.424+0000 7fef29e00640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef2c14d600 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000c00 tx=0x7fef1c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.425+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c003810 con 0x7fef0c0039a0 2024-01-16T15:02:42.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.425+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef1c003970 con 0x7fef0c0039a0 2024-01-16T15:02:42.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.425+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2c0cb4c0 con 0x7fef0c0039a0 2024-01-16T15:02:42.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.425+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef1c003ce0 con 0x7fef0c0039a0 2024-01-16T15:02:42.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.425+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef2c0cabc0 con 0x7fef0c0039a0 2024-01-16T15:02:42.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.426+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef2c067450 con 0x7fef0c0039a0 2024-01-16T15:02:42.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.427+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7fef1c040450 con 0x7fef0c0039a0 2024-01-16T15:02:42.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.428+0000 7fef22000640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef0406b4a0 0x7fef0406d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.428+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fef1c0abfe0 con 0x7fef0c0039a0 2024-01-16T15:02:42.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.429+0000 7fef23e00640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef0406b4a0 0x7fef0406d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.430+0000 7fef23e00640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef0406b4a0 0x7fef0406d960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fef14000fd0 tx=0x7fef14009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.433+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fef1c076c60 con 0x7fef0c0039a0 2024-01-16T15:02:42.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.661+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fef2c0ce7e0 con 0x7fef0406b4a0 2024-01-16T15:02:42.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.736+0000 7fef22000640 1 -- 172.21.15.53:0/381956419 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fef1400d040 con 0x7fef0406b4a0 2024-01-16T15:02:42.716 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:42.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef0406b4a0 msgr2=0x7fef0406d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef0406b4a0 0x7fef0406d960 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fef14000fd0 tx=0x7fef14009040 comp rx=0 tx=0).stop 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 msgr2=0x7fef2c14d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef2c14d600 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fef1c000c00 tx=0x7fef1c004730 comp rx=0 tx=0).stop 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 shutdown_connections 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fef0406b4a0 0x7fef0406d960 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fef2c0ca1d0 0x7fef2c0ca680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fef2c159210 0x7fef2c159690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 --2- 172.21.15.53:0/381956419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fef0c0039a0 0x7fef2c14d600 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 >> 172.21.15.53:0/381956419 conn(0x7fef2c0d3090 msgr2=0x7fef2c06b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 shutdown_connections 2024-01-16T15:02:42.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.740+0000 7fef2b200640 1 -- 172.21.15.53:0/381956419 wait complete. 2024-01-16T15:02:42.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:02:42.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.969+0000 7f18f3e00640 1 Processor -- start 2024-01-16T15:02:42.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.969+0000 7f18f3e00640 1 -- start start 2024-01-16T15:02:42.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18f40ccb50 0x7f18f4143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f415f4e0 con 0x7f18f415bf10 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18f415f970 con 0x7f18f40ccb50 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4071130 con 0x7f18f40adf20 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18f40ccb50 0x7f18f4143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18f40ccb50 0x7f18f4143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49882/0 (socket says 172.21.15.53:49882) 2024-01-16T15:02:42.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.971+0000 7f18f3400640 1 -- 172.21.15.53:0/268171494 learned_addr learned my addr 172.21.15.53:0/268171494 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.972+0000 7f18f2a00640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.972+0000 7f18f3400640 1 -- 172.21.15.53:0/268171494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 msgr2=0x7f18f4070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.972+0000 7f18f3400640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f4070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.972+0000 7f18f3400640 1 -- 172.21.15.53:0/268171494 >> v1:172.21.15.146:6789/0 conn(0x7f18f415bf10 legacy=0x7f18f40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.972+0000 7f18f3400640 1 -- 172.21.15.53:0/268171494 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f415c390 con 0x7f18f40ccb50 2024-01-16T15:02:42.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.972+0000 7f18f2a00640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f4070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:42.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.973+0000 7f18f3400640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f40ccb50 0x7f18f4143d30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009870 tx=0x7f18ec02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4149468f5e4adbe6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.973+0000 7f18f2000640 1 -- 172.21.15.53:0/268171494 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0317c0 con 0x7f18f40ccb50 2024-01-16T15:02:42.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.973+0000 7f18f2000640 1 -- 172.21.15.53:0/268171494 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec031920 con 0x7f18f40ccb50 2024-01-16T15:02:42.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.973+0000 7f18f2000640 1 -- 172.21.15.53:0/268171494 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec031cb0 con 0x7f18f40ccb50 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.974+0000 7f18f3e00640 1 -- 172.21.15.53:0/268171494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f40ccb50 msgr2=0x7f18f4143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.974+0000 7f18f3e00640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f40ccb50 0x7f18f4143d30 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f18ec009870 tx=0x7f18ec02b860 comp rx=0 tx=0).stop 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.974+0000 7f18f3e00640 1 -- 172.21.15.53:0/268171494 shutdown_connections 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.974+0000 7f18f3e00640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f40ccb50 0x7f18f4143d30 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.974+0000 7f18f3e00640 1 --2- 172.21.15.53:0/268171494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f4070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.974+0000 7f18f3e00640 1 -- 172.21.15.53:0/268171494 >> 172.21.15.53:0/268171494 conn(0x7f18f406a7d0 msgr2=0x7f18f406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.975+0000 7f18f3e00640 1 -- 172.21.15.53:0/268171494 shutdown_connections 2024-01-16T15:02:42.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.975+0000 7f18f3e00640 1 -- 172.21.15.53:0/268171494 wait complete. 2024-01-16T15:02:42.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.977+0000 7f18f3e00640 1 Processor -- start 2024-01-16T15:02:42.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.977+0000 7f18f3e00640 1 -- start start 2024-01-16T15:02:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.978+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f41f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.978+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f40ccb50 0x7f18f40c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.978+0000 7f18f3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 0x7f18f40c54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.978+0000 7f18f3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f4139b60 con 0x7f18f415bf10 2024-01-16T15:02:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.978+0000 7f18f3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40c4400 con 0x7f18f40ccb50 2024-01-16T15:02:42.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.978+0000 7f18f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18f40c46e0 con 0x7f18f40adf20 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 0x7f18f40c54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 0x7f18f40c54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49888/0 (socket says 172.21.15.53:49888) 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 -- 172.21.15.53:0/4023872239 learned_addr learned my addr 172.21.15.53:0/4023872239 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18e9e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f40ccb50 0x7f18f40c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f2a00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f41f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 -- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 msgr2=0x7f18f41f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f41f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 -- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f40ccb50 msgr2=0x7f18f40c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f40ccb50 0x7f18f40c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18f3400640 1 -- 172.21.15.53:0/4023872239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18ec0094d0 con 0x7f18f415bf10 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.979+0000 7f18e9e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f40ccb50 0x7f18f40c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.980+0000 7f18f3400640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 0x7f18f40c54a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f18ec02b8a0 tx=0x7f18ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.980+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0329a0 con 0x7f18f415bf10 2024-01-16T15:02:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.980+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18ec032b20 con 0x7f18f415bf10 2024-01-16T15:02:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.980+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18ec0453f0 con 0x7f18f415bf10 2024-01-16T15:02:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.980+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f40c49c0 con 0x7f18f415bf10 2024-01-16T15:02:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.980+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f40c4f80 con 0x7f18f415bf10 2024-01-16T15:02:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.981+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f40ce1a0 con 0x7f18f415bf10 2024-01-16T15:02:42.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.982+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f18ec04d3f0 con 0x7f18f415bf10 2024-01-16T15:02:42.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.983+0000 7f18eb400640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18cc06b470 0x7f18cc06d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:42.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.983+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f18ec0abf40 con 0x7f18f415bf10 2024-01-16T15:02:42.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.984+0000 7f18f2a00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18cc06b470 0x7f18cc06d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:42.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.985+0000 7f18f2a00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18cc06b470 0x7f18cc06d930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f18e40075b0 tx=0x7f18e4007540 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:42.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:42.987+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f18ec076a90 con 0x7f18f415bf10 2024-01-16T15:02:43.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.216+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f18f415f170 con 0x7f18cc06b470 2024-01-16T15:02:43.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.259+0000 7f18eb400640 1 -- 172.21.15.53:0/4023872239 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f18e4007cf0 con 0x7f18cc06b470 2024-01-16T15:02:43.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18cc06b470 msgr2=0x7f18cc06d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18cc06b470 0x7f18cc06d930 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f18e40075b0 tx=0x7f18e4007540 comp rx=0 tx=0).stop 2024-01-16T15:02:43.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 msgr2=0x7f18f40c54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 0x7f18f40c54a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f18ec02b8a0 tx=0x7f18ec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 shutdown_connections 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f18cc06b470 0x7f18cc06d930 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18f415bf10 0x7f18f40c54a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18f40ccb50 0x7f18f40c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 --2- 172.21.15.53:0/4023872239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18f40adf20 0x7f18f41f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 >> 172.21.15.53:0/4023872239 conn(0x7f18f406a7d0 msgr2=0x7f18f415b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 shutdown_connections 2024-01-16T15:02:43.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.262+0000 7f18f3e00640 1 -- 172.21.15.53:0/4023872239 wait complete. 2024-01-16T15:02:43.248 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.268+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.248 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.268+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.248 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.268+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.249 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.268+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.249 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.268+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T15:02:43.278 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.298+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.278 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.298+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.278 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.298+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.278 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.298+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.278 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:02:43.298+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:02:43.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.488+0000 7f4426200640 1 Processor -- start 2024-01-16T15:02:43.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.488+0000 7f4426200640 1 -- start start 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f4426200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f44200cf9a0 0x7f4420070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f4426200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4420071690 con 0x7f442014b2c0 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f4426200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4420071970 con 0x7f4420070ea0 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f4426200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f442014b6a0 con 0x7f44200cf9a0 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f441f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f442014b2c0 0x7f44200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54420/0 (socket says 172.21.15.53:54420) 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f441f400640 1 -- 172.21.15.53:0/524363718 learned_addr learned my addr 172.21.15.53:0/524363718 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.490+0000 7f4417e00640 1 --2- 172.21.15.53:0/524363718 >> v2:172.21.15.53:3300/0 conn(0x7f44200cf9a0 0x7f4420070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.491+0000 7f441ea00640 1 -- 172.21.15.53:0/524363718 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986523644 0 0) 0x7f4420071690 con 0x7f442014b2c0 2024-01-16T15:02:43.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.491+0000 7f4417e00640 1 -- 172.21.15.53:0/524363718 >> v1:172.21.15.146:6789/0 conn(0x7f4420070ea0 legacy=0x7f44200cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.491+0000 7f4417e00640 1 -- 172.21.15.53:0/524363718 >> v1:172.21.15.53:6789/0 conn(0x7f442014b2c0 legacy=0x7f44200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.491+0000 7f4417e00640 1 -- 172.21.15.53:0/524363718 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4420071c50 con 0x7f44200cf9a0 2024-01-16T15:02:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.492+0000 7f4417e00640 1 --2- 172.21.15.53:0/524363718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44200cf9a0 0x7f4420070960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f44080097a0 tx=0x7f440802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf77d6f84ba8b2a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.492+0000 7f441ea00640 1 -- 172.21.15.53:0/524363718 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408031830 con 0x7f44200cf9a0 2024-01-16T15:02:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.492+0000 7f441ea00640 1 -- 172.21.15.53:0/524363718 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4408031990 con 0x7f44200cf9a0 2024-01-16T15:02:43.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.492+0000 7f441ea00640 1 -- 172.21.15.53:0/524363718 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4408031d20 con 0x7f44200cf9a0 2024-01-16T15:02:43.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 -- 172.21.15.53:0/524363718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44200cf9a0 msgr2=0x7f4420070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 --2- 172.21.15.53:0/524363718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44200cf9a0 0x7f4420070960 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f44080097a0 tx=0x7f440802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 -- 172.21.15.53:0/524363718 shutdown_connections 2024-01-16T15:02:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 --2- 172.21.15.53:0/524363718 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44200cf9a0 0x7f4420070960 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 -- 172.21.15.53:0/524363718 >> 172.21.15.53:0/524363718 conn(0x7f442006a800 msgr2=0x7f442006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 -- 172.21.15.53:0/524363718 shutdown_connections 2024-01-16T15:02:43.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.493+0000 7f4426200640 1 -- 172.21.15.53:0/524363718 wait complete. 2024-01-16T15:02:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.495+0000 7f4426200640 1 Processor -- start 2024-01-16T15:02:43.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.495+0000 7f4426200640 1 -- start start 2024-01-16T15:02:43.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f4426200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 0x7f44201437a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f4426200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44200cf9a0 0x7f4420161320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f4426200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442014b2c0 0x7f4420145680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f4426200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420161860 con 0x7f4420070ea0 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f4426200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4420140470 con 0x7f442014b2c0 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f4426200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4420140600 con 0x7f44200cf9a0 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 0x7f44201437a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 0x7f44201437a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49904/0 (socket says 172.21.15.53:49904) 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 -- 172.21.15.53:0/2810516944 learned_addr learned my addr 172.21.15.53:0/2810516944 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 -- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44200cf9a0 msgr2=0x7f4420161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44200cf9a0 0x7f4420161320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 -- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442014b2c0 msgr2=0x7f4420145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442014b2c0 0x7f4420145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.496+0000 7f441f400640 1 -- 172.21.15.53:0/2810516944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4408009400 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.497+0000 7f441f400640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 0x7f44201437a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f441000c430 tx=0x7f441000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.497+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44100074e0 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.497+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4410007660 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.497+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4410004a20 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.497+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4420145bc0 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.497+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4420144140 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.498+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44200675d0 con 0x7f4420070ea0 2024-01-16T15:02:43.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.499+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f4410004b80 con 0x7f4420070ea0 2024-01-16T15:02:43.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.500+0000 7f4417400640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f43f806b4c0 0x7f43f806d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:43.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.500+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4410089c70 con 0x7f4420070ea0 2024-01-16T15:02:43.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.501+0000 7f4417e00640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f43f806b4c0 0x7f43f806d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:43.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.502+0000 7f4417e00640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f43f806b4c0 0x7f43f806d980 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4408036000 tx=0x7f4408030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:43.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.505+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44100548d0 con 0x7f4420070ea0 2024-01-16T15:02:43.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.726+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f44200cbc40 con 0x7f43f806b4c0 2024-01-16T15:02:43.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.766+0000 7f4417400640 1 -- 172.21.15.53:0/2810516944 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4408031b80 con 0x7f43f806b4c0 2024-01-16T15:02:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f43f806b4c0 msgr2=0x7f43f806d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f43f806b4c0 0x7f43f806d980 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4408036000 tx=0x7f4408030000 comp rx=0 tx=0).stop 2024-01-16T15:02:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 msgr2=0x7f44201437a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 0x7f44201437a0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f441000c430 tx=0x7f441000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 shutdown_connections 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f43f806b4c0 0x7f43f806d980 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f442014b2c0 0x7f4420145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44200cf9a0 0x7f4420161320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 --2- 172.21.15.53:0/2810516944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4420070ea0 0x7f44201437a0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 >> 172.21.15.53:0/2810516944 conn(0x7f442006a800 msgr2=0x7f442006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 shutdown_connections 2024-01-16T15:02:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:43.769+0000 7f4426200640 1 -- 172.21.15.53:0/2810516944 wait complete. 2024-01-16T15:02:43.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:43.770 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:02:43.825 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-01-16T15:02:43.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:02:44.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.031+0000 7f0d6d800640 1 Processor -- start 2024-01-16T15:02:44.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.031+0000 7f0d6d800640 1 -- start start 2024-01-16T15:02:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.032+0000 7f0d6d800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0d6815aff0 0x7f0d680ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.032+0000 7f0d6d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 0x7f0d6806a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.032+0000 7f0d6d800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6815f270 con 0x7f0d6806b1b0 2024-01-16T15:02:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.032+0000 7f0d6d800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6806afd0 con 0x7f0d6815aff0 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.032+0000 7f0d6d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6815f550 con 0x7f0d6815ecd0 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0d6815aff0 0x7f0d680ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0d6815aff0 0x7f0d680ce160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49908/0 (socket says 172.21.15.53:49908) 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 -- 172.21.15.53:0/1557784402 learned_addr learned my addr 172.21.15.53:0/1557784402 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 -- 172.21.15.53:0/1557784402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 msgr2=0x7f0d6806a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 --2- 172.21.15.53:0/1557784402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 0x7f0d6806a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 -- 172.21.15.53:0/1557784402 >> v1:172.21.15.146:6789/0 conn(0x7f0d6806b1b0 legacy=0x7f0d680b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.033+0000 7f0d67e00640 1 -- 172.21.15.53:0/1557784402 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6815f830 con 0x7f0d6815aff0 2024-01-16T15:02:44.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.034+0000 7f0d67e00640 1 --2- 172.21.15.53:0/1557784402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6815aff0 0x7f0d680ce160 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0d680723e0 tx=0x7f0d5802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244f4dcd93bd4e72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:44.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d66a00640 1 -- 172.21.15.53:0/1557784402 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58031640 con 0x7f0d6815aff0 2024-01-16T15:02:44.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d66a00640 1 -- 172.21.15.53:0/1557784402 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d580317a0 con 0x7f0d6815aff0 2024-01-16T15:02:44.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d66a00640 1 -- 172.21.15.53:0/1557784402 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58031b30 con 0x7f0d6815aff0 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d6d800640 1 -- 172.21.15.53:0/1557784402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6815aff0 msgr2=0x7f0d680ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d6d800640 1 --2- 172.21.15.53:0/1557784402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6815aff0 0x7f0d680ce160 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f0d680723e0 tx=0x7f0d5802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d6d800640 1 -- 172.21.15.53:0/1557784402 shutdown_connections 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d6d800640 1 --2- 172.21.15.53:0/1557784402 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 0x7f0d6806a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d6d800640 1 --2- 172.21.15.53:0/1557784402 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6815aff0 0x7f0d680ce160 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.035+0000 7f0d6d800640 1 -- 172.21.15.53:0/1557784402 >> 172.21.15.53:0/1557784402 conn(0x7f0d680b1ba0 msgr2=0x7f0d680b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.036+0000 7f0d6d800640 1 -- 172.21.15.53:0/1557784402 shutdown_connections 2024-01-16T15:02:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.036+0000 7f0d6d800640 1 -- 172.21.15.53:0/1557784402 wait complete. 2024-01-16T15:02:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.037+0000 7f0d6d800640 1 Processor -- start 2024-01-16T15:02:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.038+0000 7f0d6d800640 1 -- start start 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.038+0000 7f0d6d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 0x7f0d681484c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.038+0000 7f0d6d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d6815aff0 0x7f0d6814b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.038+0000 7f0d6d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 0x7f0d68148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d6d800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814ef20 con 0x7f0d6806b1b0 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d6d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6814bb30 con 0x7f0d6815aff0 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d6d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d681f6ad0 con 0x7f0d6815ecd0 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 0x7f0d681484c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 0x7f0d681484c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49912/0 (socket says 172.21.15.53:49912) 2024-01-16T15:02:44.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 -- 172.21.15.53:0/3161947457 learned_addr learned my addr 172.21.15.53:0/3161947457 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 -- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 msgr2=0x7f0d68148b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 0x7f0d68148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 -- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d6815aff0 msgr2=0x7f0d6814b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d6815aff0 0x7f0d6814b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.039+0000 7f0d67e00640 1 -- 172.21.15.53:0/3161947457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d580094d0 con 0x7f0d6806b1b0 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.040+0000 7f0d67e00640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 0x7f0d681484c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0d58000c00 tx=0x7f0d580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.040+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5803ac60 con 0x7f0d6806b1b0 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.040+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5803ade0 con 0x7f0d6806b1b0 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.040+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d58044630 con 0x7f0d6806b1b0 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.040+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d680ae200 con 0x7f0d6806b1b0 2024-01-16T15:02:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.040+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d680b0a70 con 0x7f0d6806b1b0 2024-01-16T15:02:44.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.041+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d68067b60 con 0x7f0d6806b1b0 2024-01-16T15:02:44.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.042+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 7) v1 ==== 89692+0+0 (secure 0 0 0) 0x7f0d5802f050 con 0x7f0d6806b1b0 2024-01-16T15:02:44.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.043+0000 7f0d5fe00640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d4006b470 0x7f0d4006d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:44.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.043+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0d580ac200 con 0x7f0d6806b1b0 2024-01-16T15:02:44.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.044+0000 7f0d67400640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d4006b470 0x7f0d4006d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:44.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.045+0000 7f0d67400640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d4006b470 0x7f0d4006d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0d680b06c0 tx=0x7f0d50009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:44.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.048+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d58076d60 con 0x7f0d6806b1b0 2024-01-16T15:02:44.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:44.270+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d6815b3d0 con 0x7f0d4006b470 2024-01-16T15:02:44.542 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:02:44.562+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:02:44.635 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:02:45.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:45.206+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 8) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f0d58077130 con 0x7f0d6806b1b0 2024-01-16T15:02:47.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.238+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f0d580787d0 con 0x7f0d6806b1b0 2024-01-16T15:02:47.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.597+0000 7f0d5fe00640 1 -- 172.21.15.53:0/3161947457 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f0d500163f0 con 0x7f0d4006b470 2024-01-16T15:02:47.578 INFO:teuthology.orchestra.run.smithi053.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:02:47.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.602+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d4006b470 msgr2=0x7f0d4006d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.602+0000 7f0d6d800640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d4006b470 0x7f0d4006d930 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0d680b06c0 tx=0x7f0d50009250 comp rx=0 tx=0).stop 2024-01-16T15:02:47.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.602+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 msgr2=0x7f0d681484c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:47.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.602+0000 7f0d6d800640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 0x7f0d681484c0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0d58000c00 tx=0x7f0d580046c0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.603+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 shutdown_connections 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.603+0000 7f0d6d800640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d4006b470 0x7f0d4006d930 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.603+0000 7f0d6d800640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d6815ecd0 0x7f0d68148b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.603+0000 7f0d6d800640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d6815aff0 0x7f0d6814b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.603+0000 7f0d6d800640 1 --2- 172.21.15.53:0/3161947457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d6806b1b0 0x7f0d681484c0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.603+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 >> 172.21.15.53:0/3161947457 conn(0x7f0d680b1ba0 msgr2=0x7f0d6815d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.604+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 shutdown_connections 2024-01-16T15:02:47.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:47.604+0000 7f0d6d800640 1 -- 172.21.15.53:0/3161947457 wait complete. 2024-01-16T15:02:47.615 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:02:47.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:47.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:47.860 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:02:47.860 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:02:47.860 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:47.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:48.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.158+0000 7f7e88a00640 1 Processor -- start 2024-01-16T15:02:48.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.158+0000 7f7e88a00640 1 -- start start 2024-01-16T15:02:48.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.159+0000 7f7e88a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e840d0100 0x7f7e840b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.159+0000 7f7e88a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e840d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.159+0000 7f7e88a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e840d0a10 con 0x7f7e84160920 2024-01-16T15:02:48.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.159+0000 7f7e88a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e840cf080 con 0x7f7e840d0100 2024-01-16T15:02:48.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.159+0000 7f7e88a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840c6be0 con 0x7f7e840d0f40 2024-01-16T15:02:48.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.160+0000 7f7e81e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e840d0100 0x7f7e840b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.160+0000 7f7e81e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7e840d0100 0x7f7e840b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:59034/0 (socket says 172.21.15.53:59034) 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.160+0000 7f7e83200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e840d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.160+0000 7f7e81e00640 1 -- 172.21.15.53:0/2181728111 learned_addr learned my addr 172.21.15.53:0/2181728111 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.161+0000 7f7e81e00640 1 -- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 msgr2=0x7f7e840d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.161+0000 7f7e81e00640 1 --2- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e840d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.161+0000 7f7e81e00640 1 -- 172.21.15.53:0/2181728111 >> v1:172.21.15.146:6789/0 conn(0x7f7e84160920 legacy=0x7f7e840cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.161+0000 7f7e81e00640 1 -- 172.21.15.53:0/2181728111 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e840ce7e0 con 0x7f7e840d0100 2024-01-16T15:02:48.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.162+0000 7f7e81e00640 1 --2- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b3f30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009870 tx=0x7f7e6c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56fe97cd6d4769bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.162+0000 7f7e81400640 1 -- 172.21.15.53:0/2181728111 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c030840 con 0x7f7e840d0100 2024-01-16T15:02:48.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.162+0000 7f7e81400640 1 -- 172.21.15.53:0/2181728111 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e6c0309a0 con 0x7f7e840d0100 2024-01-16T15:02:48.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.162+0000 7f7e81400640 1 -- 172.21.15.53:0/2181728111 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e6c030d30 con 0x7f7e840d0100 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 -- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 msgr2=0x7f7e840b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b3f30 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7e6c009870 tx=0x7f7e6c02b980 comp rx=0 tx=0).stop 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 -- 172.21.15.53:0/2181728111 shutdown_connections 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e840d13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2181728111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b3f30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 -- 172.21.15.53:0/2181728111 >> 172.21.15.53:0/2181728111 conn(0x7f7e840d3090 msgr2=0x7f7e840d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 -- 172.21.15.53:0/2181728111 shutdown_connections 2024-01-16T15:02:48.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.163+0000 7f7e88a00640 1 -- 172.21.15.53:0/2181728111 wait complete. 2024-01-16T15:02:48.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.165+0000 7f7e88a00640 1 Processor -- start 2024-01-16T15:02:48.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.165+0000 7f7e88a00640 1 -- start start 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.166+0000 7f7e88a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.166+0000 7f7e88a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e841f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.166+0000 7f7e88a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 0x7f7e84148ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.166+0000 7f7e88a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8414f780 con 0x7f7e84160920 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.166+0000 7f7e88a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840cd4a0 con 0x7f7e840d0100 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.166+0000 7f7e88a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e840cd010 con 0x7f7e840d0f40 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 0x7f7e84148ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 0x7f7e84148ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49938/0 (socket says 172.21.15.53:49938) 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 -- 172.21.15.53:0/2120211144 learned_addr learned my addr 172.21.15.53:0/2120211144 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:48.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e81e00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e841f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e82800640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 -- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 msgr2=0x7f7e841f64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e841f64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 -- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 msgr2=0x7f7e840b6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 -- 172.21.15.53:0/2120211144 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e6c0094d0 con 0x7f7e84160920 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e82800640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.167+0000 7f7e83200640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 0x7f7e84148ba0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009dd0 tx=0x7f7e7c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.168+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c00ced0 con 0x7f7e84160920 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.168+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c00f040 con 0x7f7e84160920 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.168+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c0146a0 con 0x7f7e84160920 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.168+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e840b6a70 con 0x7f7e84160920 2024-01-16T15:02:48.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.168+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e840cc9c0 con 0x7f7e84160920 2024-01-16T15:02:48.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.169+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e84067450 con 0x7f7e84160920 2024-01-16T15:02:48.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.170+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7e7c014800 con 0x7f7e84160920 2024-01-16T15:02:48.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.171+0000 7f7e7aa00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.171+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f7e7c08aa20 con 0x7f7e84160920 2024-01-16T15:02:48.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.172+0000 7f7e82800640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.174+0000 7f7e82800640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e841f6190 tx=0x7f7e74009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.176+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7e7c055680 con 0x7f7e84160920 2024-01-16T15:02:48.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.366+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e8408c7e0 con 0x7f7e84160920 2024-01-16T15:02:48.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.367+0000 7f7e7aa00640 1 -- 172.21.15.53:0/2120211144 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12950 (secure 0 0 0) 0x7f7e7c018090 con 0x7f7e84160920 2024-01-16T15:02:48.346 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:48.346 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":38,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:02:47.579329+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/2729092299":"2024-01-17T15:02:44.562599+0000","172.21.15.53:6833/2729092299":"2024-01-17T15:02:44.562599+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:48.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e5c06b500 msgr2=0x7f7e5c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7e841f6190 tx=0x7f7e74009040 comp rx=0 tx=0).stop 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 msgr2=0x7f7e84148ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 0x7f7e84148ba0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c009dd0 tx=0x7f7e7c00c490 comp rx=0 tx=0).stop 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 shutdown_connections 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e5c06b500 0x7f7e5c06d9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7e84160920 0x7f7e84148ba0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7e840d0f40 0x7f7e841f64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 --2- 172.21.15.53:0/2120211144 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7e840d0100 0x7f7e840b6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 >> 172.21.15.53:0/2120211144 conn(0x7f7e840d3090 msgr2=0x7f7e8406b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 shutdown_connections 2024-01-16T15:02:48.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.370+0000 7f7e88a00640 1 -- 172.21.15.53:0/2120211144 wait complete. 2024-01-16T15:02:48.367 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:48.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:48.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.604+0000 7fa8aa200640 1 Processor -- start 2024-01-16T15:02:48.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.604+0000 7fa8aa200640 1 -- start start 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8aa200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8aa200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa8a415bf10 0x7fa8a40cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8aa200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a415f4e0 con 0x7fa8a40cc980 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8aa200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8a415f970 con 0x7fa8a415bf10 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8aa200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a40cce10 con 0x7fa8a40adf20 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8a9800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa8a40cc980 0x7fa8a4143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54438/0 (socket says 172.21.15.53:54438) 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8a9800640 1 -- 172.21.15.53:0/4200929876 learned_addr learned my addr 172.21.15.53:0/4200929876 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8a8e00640 1 --2- 172.21.15.53:0/4200929876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa89be00640 1 --2- 172.21.15.53:0/4200929876 >> v2:172.21.15.146:3300/0 conn(0x7fa8a415bf10 0x7fa8a40cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.606+0000 7fa8a3e00640 1 -- 172.21.15.53:0/4200929876 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227722977 0 0) 0x7fa8a415f4e0 con 0x7fa8a40cc980 2024-01-16T15:02:48.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.607+0000 7fa8a3e00640 1 -- 172.21.15.53:0/4200929876 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884003520 con 0x7fa8a40cc980 2024-01-16T15:02:48.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.607+0000 7fa8a8e00640 1 -- 172.21.15.53:0/4200929876 >> v2:172.21.15.146:3300/0 conn(0x7fa8a415bf10 msgr2=0x7fa8a40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:48.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.607+0000 7fa8a8e00640 1 --2- 172.21.15.53:0/4200929876 >> v2:172.21.15.146:3300/0 conn(0x7fa8a415bf10 0x7fa8a40cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.607+0000 7fa8a8e00640 1 -- 172.21.15.53:0/4200929876 >> v1:172.21.15.53:6789/0 conn(0x7fa8a40cc980 legacy=0x7fa8a4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.607+0000 7fa8a8e00640 1 -- 172.21.15.53:0/4200929876 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a4071130 con 0x7fa8a40adf20 2024-01-16T15:02:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.608+0000 7fa8a8e00640 1 --2- 172.21.15.53:0/4200929876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4070a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8940097a0 tx=0x7fa89402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c334209e19c5a064 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.608+0000 7fa8a3e00640 1 -- 172.21.15.53:0/4200929876 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89402e070 con 0x7fa8a40adf20 2024-01-16T15:02:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8a3e00640 1 -- 172.21.15.53:0/4200929876 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8940315a0 con 0x7fa8a40adf20 2024-01-16T15:02:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8a3e00640 1 -- 172.21.15.53:0/4200929876 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa894031930 con 0x7fa8a40adf20 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 -- 172.21.15.53:0/4200929876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 msgr2=0x7fa8a4070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 --2- 172.21.15.53:0/4200929876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4070a30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa8940097a0 tx=0x7fa89402b840 comp rx=0 tx=0).stop 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 -- 172.21.15.53:0/4200929876 shutdown_connections 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 --2- 172.21.15.53:0/4200929876 >> v2:172.21.15.146:3300/0 conn(0x7fa8a415bf10 0x7fa8a40cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 --2- 172.21.15.53:0/4200929876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4070a30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 -- 172.21.15.53:0/4200929876 >> 172.21.15.53:0/4200929876 conn(0x7fa8a406a7d0 msgr2=0x7fa8a406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 -- 172.21.15.53:0/4200929876 shutdown_connections 2024-01-16T15:02:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.609+0000 7fa8aa200640 1 -- 172.21.15.53:0/4200929876 wait complete. 2024-01-16T15:02:48.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.611+0000 7fa8aa200640 1 Processor -- start 2024-01-16T15:02:48.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.611+0000 7fa8aa200640 1 -- start start 2024-01-16T15:02:48.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.612+0000 7fa8aa200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4140790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.612+0000 7fa8aa200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8a40cc980 0x7fa8a40b2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.612+0000 7fa8aa200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 0x7fa8a40b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.612+0000 7fa8aa200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a40c4040 con 0x7fa8a415bf10 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.612+0000 7fa8aa200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a4141680 con 0x7fa8a40cc980 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.612+0000 7fa8aa200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8a4140cd0 con 0x7fa8a40adf20 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 0x7fa8a40b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 0x7fa8a40b30c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49954/0 (socket says 172.21.15.53:49954) 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 -- 172.21.15.53:0/2307451449 learned_addr learned my addr 172.21.15.53:0/2307451449 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a8e00640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4140790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa89be00640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8a40cc980 0x7fa8a40b2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 -- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 msgr2=0x7fa8a4140790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4140790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 -- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8a40cc980 msgr2=0x7fa8a40b2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8a40cc980 0x7fa8a40b2b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 -- 172.21.15.53:0/2307451449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa894009400 con 0x7fa8a415bf10 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa89be00640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8a40cc980 0x7fa8a40b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.613+0000 7fa8a9800640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 0x7fa8a40b30c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa89c00cb40 tx=0x7fa89c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.614+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c00aca0 con 0x7fa8a415bf10 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.614+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89c00ae00 con 0x7fa8a415bf10 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.614+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a40c4bb0 con 0x7fa8a415bf10 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.614+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89c012620 con 0x7fa8a415bf10 2024-01-16T15:02:48.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.614+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a4141310 con 0x7fa8a415bf10 2024-01-16T15:02:48.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.615+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a4067450 con 0x7fa8a415bf10 2024-01-16T15:02:48.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.616+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa89c007070 con 0x7fa8a415bf10 2024-01-16T15:02:48.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.617+0000 7fa8a1600640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa87c06b500 0x7fa87c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:48.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.617+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa89c089c70 con 0x7fa8a415bf10 2024-01-16T15:02:48.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.617+0000 7fa8a8e00640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa87c06b500 0x7fa87c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:48.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.618+0000 7fa8a8e00640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa87c06b500 0x7fa87c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa894036000 tx=0x7fa89403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:48.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.621+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa89c054b20 con 0x7fa8a415bf10 2024-01-16T15:02:48.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.814+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8a41417c0 con 0x7fa8a415bf10 2024-01-16T15:02:48.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.815+0000 7fa8a1600640 1 -- 172.21.15.53:0/2307451449 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa89c058140 con 0x7fa8a415bf10 2024-01-16T15:02:48.794 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:02:48.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa87c06b500 msgr2=0x7fa87c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa87c06b500 0x7fa87c06d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa894036000 tx=0x7fa89403d040 comp rx=0 tx=0).stop 2024-01-16T15:02:48.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 msgr2=0x7fa8a40b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:48.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 0x7fa8a40b30c0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fa89c00cb40 tx=0x7fa89c00a060 comp rx=0 tx=0).stop 2024-01-16T15:02:48.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 shutdown_connections 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa87c06b500 0x7fa87c06d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.817+0000 7fa8aa200640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8a415bf10 0x7fa8a40b30c0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.818+0000 7fa8aa200640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8a40cc980 0x7fa8a40b2b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.818+0000 7fa8aa200640 1 --2- 172.21.15.53:0/2307451449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8a40adf20 0x7fa8a4140790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.818+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 >> 172.21.15.53:0/2307451449 conn(0x7fa8a406a7d0 msgr2=0x7fa8a40b1e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.818+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 shutdown_connections 2024-01-16T15:02:48.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:48.818+0000 7fa8aa200640 1 -- 172.21.15.53:0/2307451449 wait complete. 2024-01-16T15:02:48.815 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:02:49.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.088+0000 7fe757200640 1 Processor -- start 2024-01-16T15:02:49.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.089+0000 7fe757200640 1 -- start start 2024-01-16T15:02:49.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.090+0000 7fe757200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75015e880 0x7fe75006ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.091+0000 7fe757200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7500cfa70 con 0x7fe750070490 2024-01-16T15:02:49.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.091+0000 7fe757200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7500cff00 con 0x7fe75009a170 2024-01-16T15:02:49.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.091+0000 7fe757200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75006fc70 con 0x7fe75015e880 2024-01-16T15:02:49.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.091+0000 7fe74f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe750070490 0x7fe750099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54444/0 (socket says 172.21.15.53:54444) 2024-01-16T15:02:49.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.091+0000 7fe74f400640 1 -- 172.21.15.53:0/1428907131 learned_addr learned my addr 172.21.15.53:0/1428907131 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:49.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.091+0000 7fe74fe00640 1 --2- 172.21.15.53:0/1428907131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75015e880 0x7fe75006ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3507761309 0 0) 0x7fe7500cfa70 con 0x7fe750070490 2024-01-16T15:02:49.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe730003520 con 0x7fe750070490 2024-01-16T15:02:49.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406022465 0 0) 0x7fe7500cff00 con 0x7fe75009a170 2024-01-16T15:02:49.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7500cfa70 con 0x7fe75009a170 2024-01-16T15:02:49.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74fe00640 1 -- 172.21.15.53:0/1428907131 >> v1:172.21.15.146:6789/0 conn(0x7fe75009a170 legacy=0x7fe75006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74fe00640 1 -- 172.21.15.53:0/1428907131 >> v1:172.21.15.53:6789/0 conn(0x7fe750070490 legacy=0x7fe750099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.092+0000 7fe74fe00640 1 -- 172.21.15.53:0/1428907131 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7500cf420 con 0x7fe75015e880 2024-01-16T15:02:49.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.094+0000 7fe74fe00640 1 --2- 172.21.15.53:0/1428907131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75015e880 0x7fe75006ff50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe7380097d0 tx=0x7fe73802b9f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a21b9c22fce3be97 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:49.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.095+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73802f810 con 0x7fe75015e880 2024-01-16T15:02:49.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.095+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe73802f970 con 0x7fe75015e880 2024-01-16T15:02:49.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.095+0000 7fe74ea00640 1 -- 172.21.15.53:0/1428907131 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73802fce0 con 0x7fe75015e880 2024-01-16T15:02:49.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.096+0000 7fe757200640 1 -- 172.21.15.53:0/1428907131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75015e880 msgr2=0x7fe75006ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.096+0000 7fe757200640 1 --2- 172.21.15.53:0/1428907131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75015e880 0x7fe75006ff50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe7380097d0 tx=0x7fe73802b9f0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.096+0000 7fe757200640 1 -- 172.21.15.53:0/1428907131 shutdown_connections 2024-01-16T15:02:49.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.096+0000 7fe757200640 1 --2- 172.21.15.53:0/1428907131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75015e880 0x7fe75006ff50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.096+0000 7fe757200640 1 -- 172.21.15.53:0/1428907131 >> 172.21.15.53:0/1428907131 conn(0x7fe7500b2d00 msgr2=0x7fe7500b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:49.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.096+0000 7fe757200640 1 -- 172.21.15.53:0/1428907131 shutdown_connections 2024-01-16T15:02:49.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.097+0000 7fe757200640 1 -- 172.21.15.53:0/1428907131 wait complete. 2024-01-16T15:02:49.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.098+0000 7fe757200640 1 Processor -- start 2024-01-16T15:02:49.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.099+0000 7fe757200640 1 -- start start 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.099+0000 7fe757200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe750070490 0x7fe750159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.099+0000 7fe757200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75009a170 0x7fe7501ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.099+0000 7fe757200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 0x7fe7500c4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe757200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe75015bbd0 con 0x7fe75015e880 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe757200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7502005a0 con 0x7fe750070490 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe757200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe750200880 con 0x7fe75009a170 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 0x7fe7500c4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 0x7fe7500c4ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49962/0 (socket says 172.21.15.53:49962) 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 -- 172.21.15.53:0/1132928722 learned_addr learned my addr 172.21.15.53:0/1132928722 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe74f400640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75009a170 0x7fe7501ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe74fe00640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe750070490 0x7fe750159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 -- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75009a170 msgr2=0x7fe7501ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75009a170 0x7fe7501ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 -- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe750070490 msgr2=0x7fe750159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe750070490 0x7fe750159da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.100+0000 7fe755200640 1 -- 172.21.15.53:0/1132928722 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe738009400 con 0x7fe75015e880 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe74fe00640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe750070490 0x7fe750159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:49.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe755200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 0x7fe7500c4ac0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c7a0 tx=0x7fe74800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:49.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe748003b70 con 0x7fe75015e880 2024-01-16T15:02:49.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe748003cd0 con 0x7fe75015e880 2024-01-16T15:02:49.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74800aa90 con 0x7fe75015e880 2024-01-16T15:02:49.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe750200b60 con 0x7fe75015e880 2024-01-16T15:02:49.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.101+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75015b7c0 con 0x7fe75015e880 2024-01-16T15:02:49.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.103+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe750200d50 con 0x7fe75015e880 2024-01-16T15:02:49.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.104+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fe74800abf0 con 0x7fe75015e880 2024-01-16T15:02:49.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.105+0000 7fe747e00640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe72806b550 0x7fe72806da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.105+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fe748089a50 con 0x7fe75015e880 2024-01-16T15:02:49.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.105+0000 7fe74fe00640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe72806b550 0x7fe72806da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.106+0000 7fe74fe00640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe72806b550 0x7fe72806da10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe738036000 tx=0x7fe738008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:49.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.112+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe750200d50 con 0x7fe75015e880 2024-01-16T15:02:49.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.342+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe750200d50 con 0x7fe75015e880 2024-01-16T15:02:49.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.343+0000 7fe747e00640 1 -- 172.21.15.53:0/1132928722 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v38) v1 ==== 80+0+122 (secure 0 0 0) 0x7fe750200d50 con 0x7fe75015e880 2024-01-16T15:02:49.322 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6835/2729092299 2024-01-17T15:02:44.562599+0000 2024-01-16T15:02:49.322 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6833/2729092299 2024-01-17T15:02:44.562599+0000 2024-01-16T15:02:49.323 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:02:49.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.345+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe72806b550 msgr2=0x7fe72806da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.345+0000 7fe757200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe72806b550 0x7fe72806da10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe738036000 tx=0x7fe738008030 comp rx=0 tx=0).stop 2024-01-16T15:02:49.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.345+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 msgr2=0x7fe7500c4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.345+0000 7fe757200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 0x7fe7500c4ac0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fe74800c7a0 tx=0x7fe74800cc60 comp rx=0 tx=0).stop 2024-01-16T15:02:49.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 shutdown_connections 2024-01-16T15:02:49.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe72806b550 0x7fe72806da10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe75015e880 0x7fe7500c4ac0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe75009a170 0x7fe7501ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 --2- 172.21.15.53:0/1132928722 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe750070490 0x7fe750159da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 >> 172.21.15.53:0/1132928722 conn(0x7fe7500b2d00 msgr2=0x7fe750099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 shutdown_connections 2024-01-16T15:02:49.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.346+0000 7fe757200640 1 -- 172.21.15.53:0/1132928722 wait complete. 2024-01-16T15:02:49.342 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:02:49.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:49.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:02:49.440 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:02:49.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:49.440 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:02:49.442 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[48537]: fuse finished with error 0 and tester_r 0 2024-01-16T15:02:49.655 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:02:49.656 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:49.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.964+0000 7ff99a200640 1 Processor -- start 2024-01-16T15:02:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.964+0000 7ff99a200640 1 -- start start 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.966+0000 7ff99a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.966+0000 7ff99a200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9940ce480 0x7ff9940ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.966+0000 7ff99a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff99415dd50 0x7ff99415e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.966+0000 7ff99a200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99415e700 con 0x7ff99415dd50 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.966+0000 7ff99a200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9940cf500 con 0x7ff9940ce480 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.966+0000 7ff99a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9940cef40 con 0x7ff994143c00 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.967+0000 7ff993e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff99415dd50 0x7ff99415e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.967+0000 7ff993e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff99415dd50 0x7ff99415e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49964/0 (socket says 172.21.15.53:49964) 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.967+0000 7ff993e00640 1 -- 172.21.15.53:0/1123990665 learned_addr learned my addr 172.21.15.53:0/1123990665 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.967+0000 7ff993400640 1 --2- 172.21.15.53:0/1123990665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.967+0000 7ff98be00640 1 --2- 172.21.15.53:0/1123990665 >> v2:172.21.15.146:3300/0 conn(0x7ff9940ce480 0x7ff9940ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.968+0000 7ff993e00640 1 -- 172.21.15.53:0/1123990665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 msgr2=0x7ff99415b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.968+0000 7ff993e00640 1 --2- 172.21.15.53:0/1123990665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.968+0000 7ff993e00640 1 -- 172.21.15.53:0/1123990665 >> v2:172.21.15.146:3300/0 conn(0x7ff9940ce480 msgr2=0x7ff9940ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.968+0000 7ff993e00640 1 --2- 172.21.15.53:0/1123990665 >> v2:172.21.15.146:3300/0 conn(0x7ff9940ce480 0x7ff9940ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.968+0000 7ff993e00640 1 -- 172.21.15.53:0/1123990665 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9940cf220 con 0x7ff99415dd50 2024-01-16T15:02:49.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.969+0000 7ff993e00640 1 --2- 172.21.15.53:0/1123990665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff99415e1c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff994070eb0 tx=0x7ff98c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aff5d2cda804989 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:49.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.969+0000 7ff992a00640 1 -- 172.21.15.53:0/1123990665 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c02ca70 con 0x7ff99415dd50 2024-01-16T15:02:49.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.969+0000 7ff992a00640 1 -- 172.21.15.53:0/1123990665 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c02cbd0 con 0x7ff99415dd50 2024-01-16T15:02:49.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.969+0000 7ff992a00640 1 -- 172.21.15.53:0/1123990665 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c0343f0 con 0x7ff99415dd50 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 -- 172.21.15.53:0/1123990665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 msgr2=0x7ff99415e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 --2- 172.21.15.53:0/1123990665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff99415e1c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7ff994070eb0 tx=0x7ff98c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 -- 172.21.15.53:0/1123990665 shutdown_connections 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 --2- 172.21.15.53:0/1123990665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff99415e1c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 --2- 172.21.15.53:0/1123990665 >> v2:172.21.15.146:3300/0 conn(0x7ff9940ce480 0x7ff9940ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 --2- 172.21.15.53:0/1123990665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 -- 172.21.15.53:0/1123990665 >> 172.21.15.53:0/1123990665 conn(0x7ff9940cff90 msgr2=0x7ff9940d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 -- 172.21.15.53:0/1123990665 shutdown_connections 2024-01-16T15:02:49.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.970+0000 7ff99a200640 1 -- 172.21.15.53:0/1123990665 wait complete. 2024-01-16T15:02:49.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.972+0000 7ff99a200640 1 Processor -- start 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.972+0000 7ff99a200640 1 -- start start 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff99a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9940ce480 0x7ff994138ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff99a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff99a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff994156620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff99a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff994139020 con 0x7ff99415dd50 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff99a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9941597b0 con 0x7ff9940ce480 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff99a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff994159a90 con 0x7ff994143c00 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.973+0000 7ff993e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff994156620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff994156620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49970/0 (socket says 172.21.15.53:49970) 2024-01-16T15:02:49.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 -- 172.21.15.53:0/558079707 learned_addr learned my addr 172.21.15.53:0/558079707 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993400640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9940ce480 0x7ff994138ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff98be00640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 -- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 msgr2=0x7ff99415b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 -- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9940ce480 msgr2=0x7ff994138ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9940ce480 0x7ff994138ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff993e00640 1 -- 172.21.15.53:0/558079707 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98c009b30 con 0x7ff99415dd50 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.974+0000 7ff98be00640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:49.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.975+0000 7ff993e00640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff994156620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036040 tx=0x7ff98c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.975+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c009510 con 0x7ff99415dd50 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.975+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff98c03e450 con 0x7ff99415dd50 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.975+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff98c03e800 con 0x7ff99415dd50 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.975+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff994159d70 con 0x7ff99415dd50 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.975+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99415baa0 con 0x7ff99415dd50 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.976+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9940675d0 con 0x7ff99415dd50 2024-01-16T15:02:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.977+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff98c03e960 con 0x7ff99415dd50 2024-01-16T15:02:49.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.978+0000 7ff98b400640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96c06b500 0x7ff96c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:49.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.978+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ff98c0ac810 con 0x7ff99415dd50 2024-01-16T15:02:49.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.979+0000 7ff993400640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96c06b500 0x7ff96c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:49.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.980+0000 7ff993400640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96c06b500 0x7ff96c06d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff984009aa0 tx=0x7ff984009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:49.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:49.982+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff98c0776b0 con 0x7ff99415dd50 2024-01-16T15:02:50.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.178+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff994154b80 con 0x7ff99415dd50 2024-01-16T15:02:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.179+0000 7ff98b400640 1 -- 172.21.15.53:0/558079707 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12950 (secure 0 0 0) 0x7ff98c07acd0 con 0x7ff99415dd50 2024-01-16T15:02:50.159 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:50.159 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":38,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:02:47.579329+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/2729092299":"2024-01-17T15:02:44.562599+0000","172.21.15.53:6833/2729092299":"2024-01-17T15:02:44.562599+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:50.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96c06b500 msgr2=0x7ff96c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96c06b500 0x7ff96c06d9c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff984009aa0 tx=0x7ff984009190 comp rx=0 tx=0).stop 2024-01-16T15:02:50.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 msgr2=0x7ff994156620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff994156620 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7ff98c036040 tx=0x7ff98c004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:50.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 shutdown_connections 2024-01-16T15:02:50.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96c06b500 0x7ff96c06d9c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff99415dd50 0x7ff994156620 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff994143c00 0x7ff99415b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 --2- 172.21.15.53:0/558079707 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9940ce480 0x7ff994138ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 >> 172.21.15.53:0/558079707 conn(0x7ff9940cff90 msgr2=0x7ff99415cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 shutdown_connections 2024-01-16T15:02:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.182+0000 7ff99a200640 1 -- 172.21.15.53:0/558079707 wait complete. 2024-01-16T15:02:50.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:50.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:50.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.427+0000 7f1953000640 1 Processor -- start 2024-01-16T15:02:50.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.427+0000 7f1953000640 1 -- start start 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.429+0000 7f1953000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 0x7f194c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.429+0000 7f1953000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f194c0cf7d0 0x7f194c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.429+0000 7f1953000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194c0cfbf0 con 0x7f194c070ea0 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.429+0000 7f1953000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194c14b6a0 con 0x7f194c0cf7d0 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.429+0000 7f1953000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c071690 con 0x7f194c14b2c0 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.430+0000 7f1951000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f194c070ea0 0x7f194c0b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54452/0 (socket says 172.21.15.53:54452) 2024-01-16T15:02:50.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.430+0000 7f1951000640 1 -- 172.21.15.53:0/2819969462 learned_addr learned my addr 172.21.15.53:0/2819969462 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.430+0000 7f194be00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 0x7f194c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.430+0000 7f194b400640 1 --2- 172.21.15.53:0/2819969462 >> v2:172.21.15.146:3300/0 conn(0x7f194c0cf7d0 0x7f194c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.430+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28146197 0 0) 0x7f194c0cfbf0 con 0x7f194c070ea0 2024-01-16T15:02:50.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.430+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f192c003520 con 0x7f194c070ea0 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2781108670 0 0) 0x7f192c003520 con 0x7f194c070ea0 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 msgr2=0x7f194c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 0x7f194c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 >> v2:172.21.15.146:3300/0 conn(0x7f194c0cf7d0 msgr2=0x7f194c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 --2- 172.21.15.53:0/2819969462 >> v2:172.21.15.146:3300/0 conn(0x7f194c0cf7d0 0x7f194c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c071970 con 0x7f194c070ea0 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f19440019f0 con 0x7f194c070ea0 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 >> v1:172.21.15.53:6789/0 conn(0x7f194c070ea0 legacy=0x7f194c0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f192c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 0x7f192c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f192c004d00 0x7f192c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c14b6a0 con 0x7f192c003970 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c071690 con 0x7f192c004d00 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f192c00b980 con 0x7f194c14b2c0 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194be00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f192c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f194b400640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 0x7f192c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.431+0000 7f1951000640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f192c004d00 0x7f192c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.432+0000 7f194be00640 1 -- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 msgr2=0x7f192c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.432+0000 7f194be00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c14b2c0 0x7f192c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.432+0000 7f194be00640 1 -- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f192c004d00 msgr2=0x7f192c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.432+0000 7f194be00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f192c004d00 0x7f192c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.432+0000 7f194be00640 1 -- 172.21.15.53:0/2819969462 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f192c003520 con 0x7f192c003970 2024-01-16T15:02:50.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.432+0000 7f1951000640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f192c004d00 0x7f192c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.433+0000 7f194be00640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f192c003d90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f193c009ac0 tx=0x7f192c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca11f3e018ff7e83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c02c6d0 con 0x7f192c003970 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c02c830 con 0x7f192c003970 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f194aa00640 1 -- 172.21.15.53:0/2819969462 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c02cbc0 con 0x7f192c003970 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1953000640 1 -- 172.21.15.53:0/2819969462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 msgr2=0x7f192c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1953000640 1 --2- 172.21.15.53:0/2819969462 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f192c003d90 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f193c009ac0 tx=0x7f192c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1951000640 1 -- 172.21.15.53:0/2819969462 reap_dead start 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1953000640 1 -- 172.21.15.53:0/2819969462 shutdown_connections 2024-01-16T15:02:50.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1953000640 1 -- 172.21.15.53:0/2819969462 >> 172.21.15.53:0/2819969462 conn(0x7f194c06a800 msgr2=0x7f194c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:50.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1953000640 1 -- 172.21.15.53:0/2819969462 shutdown_connections 2024-01-16T15:02:50.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.434+0000 7f1953000640 1 -- 172.21.15.53:0/2819969462 wait complete. 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.436+0000 7f1953000640 1 Processor -- start 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.436+0000 7f1953000640 1 -- start start 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f1953000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f194c161190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f1953000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c1417b0 0x7f194c1fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f1953000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f194c0cd0b0 0x7f194c0cd560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f1953000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c0cdaa0 con 0x7f192c003970 2024-01-16T15:02:50.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f1953000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194c1fb600 con 0x7f194c0cd0b0 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f1953000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194c1fb040 con 0x7f194c1417b0 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.437+0000 7f194be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f194c161190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f194c161190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49996/0 (socket says 172.21.15.53:49996) 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 -- 172.21.15.53:0/3442429111 learned_addr learned my addr 172.21.15.53:0/3442429111 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f1951000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f194c0cd0b0 0x7f194c0cd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194b400640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c1417b0 0x7f194c1fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 -- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c1417b0 msgr2=0x7f194c1fa6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c1417b0 0x7f194c1fa6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 -- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f194c0cd0b0 msgr2=0x7f194c0cd560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f194c0cd0b0 0x7f194c0cd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 -- 172.21.15.53:0/3442429111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194c1fb320 con 0x7f192c003970 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194b400640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c1417b0 0x7f194c1fa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:50.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.438+0000 7f194be00640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f194c161190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f193c02c270 tx=0x7f193c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:50.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.439+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c003810 con 0x7f192c003970 2024-01-16T15:02:50.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.439+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c003970 con 0x7f192c003970 2024-01-16T15:02:50.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.439+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c003d00 con 0x7f192c003970 2024-01-16T15:02:50.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.439+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194c163050 con 0x7f192c003970 2024-01-16T15:02:50.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.439+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f194c1616d0 con 0x7f192c003970 2024-01-16T15:02:50.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.440+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f194c0675d0 con 0x7f192c003970 2024-01-16T15:02:50.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.441+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f193c034440 con 0x7f192c003970 2024-01-16T15:02:50.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.442+0000 7f1943e00640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f192406b500 0x7f192406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.443+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f193c0abf90 con 0x7f192c003970 2024-01-16T15:02:50.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.443+0000 7f194b400640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f192406b500 0x7f192406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.444+0000 7f194b400640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f192406b500 0x7f192406d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f194c145510 tx=0x7f1934008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:50.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.447+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f193c076f60 con 0x7f192c003970 2024-01-16T15:02:50.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.641+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f194c0cb620 con 0x7f192c003970 2024-01-16T15:02:50.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.642+0000 7f1943e00640 1 -- 172.21.15.53:0/3442429111 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f193c04c0a0 con 0x7f192c003970 2024-01-16T15:02:50.622 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:02:50.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f192406b500 msgr2=0x7f192406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f192406b500 0x7f192406d9c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f194c145510 tx=0x7f1934008040 comp rx=0 tx=0).stop 2024-01-16T15:02:50.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 msgr2=0x7f194c161190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f194c161190 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f193c02c270 tx=0x7f193c004730 comp rx=0 tx=0).stop 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 shutdown_connections 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f192406b500 0x7f192406d9c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f194c0cd0b0 0x7f194c0cd560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f194c1417b0 0x7f194c1fa6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 --2- 172.21.15.53:0/3442429111 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f192c003970 0x7f194c161190 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 >> 172.21.15.53:0/3442429111 conn(0x7f194c06a800 msgr2=0x7f194c0af1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 shutdown_connections 2024-01-16T15:02:50.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.645+0000 7f1953000640 1 -- 172.21.15.53:0/3442429111 wait complete. 2024-01-16T15:02:50.632 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:50.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:02:50.647 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:02:50.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.919+0000 7fc6d0e00640 1 Processor -- start 2024-01-16T15:02:50.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.920+0000 7fc6d0e00640 1 -- start start 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.921+0000 7fc6d0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6cc13df80 0x7fc6cc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.921+0000 7fc6d0e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc6cc13e360 0x7fc6cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.921+0000 7fc6d0e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6cc06c470 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.921+0000 7fc6d0e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6cc06c750 con 0x7fc6cc13df80 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.921+0000 7fc6d0e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6cc06ca30 con 0x7fc6cc13e360 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6cc13df80 0x7fc6cc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc6cc13df80 0x7fc6cc15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50006/0 (socket says 172.21.15.53:50006) 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 -- 172.21.15.53:0/3127945946 learned_addr learned my addr 172.21.15.53:0/3127945946 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:50.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 -- 172.21.15.53:0/3127945946 >> v2:172.21.15.146:3300/0 conn(0x7fc6cc13e360 msgr2=0x7fc6cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9400640 1 --2- 172.21.15.53:0/3127945946 >> v2:172.21.15.146:3300/0 conn(0x7fc6cc13e360 0x7fc6cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 --2- 172.21.15.53:0/3127945946 >> v2:172.21.15.146:3300/0 conn(0x7fc6cc13e360 0x7fc6cc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 -- 172.21.15.53:0/3127945946 >> v1:172.21.15.146:6789/0 conn(0x7fc6cc15f9c0 legacy=0x7fc6cc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.922+0000 7fc6c9e00640 1 -- 172.21.15.53:0/3127945946 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc06cd10 con 0x7fc6cc13df80 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.924+0000 7fc6c9e00640 1 --2- 172.21.15.53:0/3127945946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc13df80 0x7fc6cc15bee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc06ded0 tx=0x7fc6bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7073e261d031f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.924+0000 7fc6c8a00640 1 -- 172.21.15.53:0/3127945946 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc031870 con 0x7fc6cc13df80 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.924+0000 7fc6c8a00640 1 -- 172.21.15.53:0/3127945946 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc0319d0 con 0x7fc6cc13df80 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.924+0000 7fc6c8a00640 1 -- 172.21.15.53:0/3127945946 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc031d60 con 0x7fc6cc13df80 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 -- 172.21.15.53:0/3127945946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc13df80 msgr2=0x7fc6cc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/3127945946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc13df80 0x7fc6cc15bee0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc06ded0 tx=0x7fc6bc02b910 comp rx=0 tx=0).stop 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 -- 172.21.15.53:0/3127945946 shutdown_connections 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/3127945946 >> v2:172.21.15.146:3300/0 conn(0x7fc6cc13e360 0x7fc6cc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/3127945946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc13df80 0x7fc6cc15bee0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 -- 172.21.15.53:0/3127945946 >> 172.21.15.53:0/3127945946 conn(0x7fc6cc06a800 msgr2=0x7fc6cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 -- 172.21.15.53:0/3127945946 shutdown_connections 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.925+0000 7fc6d0e00640 1 -- 172.21.15.53:0/3127945946 wait complete. 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.927+0000 7fc6d0e00640 1 Processor -- start 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.927+0000 7fc6d0e00640 1 -- start start 2024-01-16T15:02:50.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6d0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6cc13df80 0x7fc6cc14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6d0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6cc13e360 0x7fc6cc150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6d0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 0x7fc6cc1333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6d0e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc14c7e0 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6d0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc14ca00 con 0x7fc6cc13e360 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6d0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6cc14cce0 con 0x7fc6cc13df80 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6ca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 0x7fc6cc1333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6ca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 0x7fc6cc1333c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50020/0 (socket says 172.21.15.53:50020) 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.928+0000 7fc6ca800640 1 -- 172.21.15.53:0/2199709301 learned_addr learned my addr 172.21.15.53:0/2199709301 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6c9400640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6cc13e360 0x7fc6cc150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6c9e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6cc13df80 0x7fc6cc14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6ca800640 1 -- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6cc13df80 msgr2=0x7fc6cc14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6ca800640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6cc13df80 0x7fc6cc14d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6ca800640 1 -- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6cc13e360 msgr2=0x7fc6cc150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6ca800640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6cc13e360 0x7fc6cc150c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6ca800640 1 -- 172.21.15.53:0/2199709301 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc0094d0 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.929+0000 7fc6ca800640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 0x7fc6cc1333c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c360 tx=0x7fc6c400c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.930+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c40035f0 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.930+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c4003750 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.930+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6c4003b00 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.930+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6cc14cfc0 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.930+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6cc14c290 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.931+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6cc0675d0 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.932+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc6c4003c60 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.933+0000 7fc6c2000640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6a406b500 0x7fc6a406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:50.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.933+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc6c4088f10 con 0x7fc6cc15f9c0 2024-01-16T15:02:50.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.934+0000 7fc6c9e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6a406b500 0x7fc6a406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:50.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.935+0000 7fc6c9e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6a406b500 0x7fc6a406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc036000 tx=0x7fc6bc03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:50.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:50.937+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc6c4053d10 con 0x7fc6cc15f9c0 2024-01-16T15:02:51.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.144+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc6cc0cc100 con 0x7fc6cc15f9c0 2024-01-16T15:02:51.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.144+0000 7fc6c2000640 1 -- 172.21.15.53:0/2199709301 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v38) v1 ==== 80+0+122 (secure 0 0 0) 0x7fc6c4017080 con 0x7fc6cc15f9c0 2024-01-16T15:02:51.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6835/2729092299 2024-01-17T15:02:44.562599+0000 2024-01-16T15:02:51.124 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6833/2729092299 2024-01-17T15:02:44.562599+0000 2024-01-16T15:02:51.125 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.147+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6a406b500 msgr2=0x7fc6a406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.147+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6a406b500 0x7fc6a406d9c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc036000 tx=0x7fc6bc03c040 comp rx=0 tx=0).stop 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.147+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 msgr2=0x7fc6cc1333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.147+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 0x7fc6cc1333c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fc6c400c360 tx=0x7fc6c400c820 comp rx=0 tx=0).stop 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 shutdown_connections 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6a406b500 0x7fc6a406d9c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc6cc15f9c0 0x7fc6cc1333c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc6cc13e360 0x7fc6cc150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 --2- 172.21.15.53:0/2199709301 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc6cc13df80 0x7fc6cc14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 >> 172.21.15.53:0/2199709301 conn(0x7fc6cc06a800 msgr2=0x7fc6cc15ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:51.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 shutdown_connections 2024-01-16T15:02:51.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.148+0000 7fc6d0e00640 1 -- 172.21.15.53:0/2199709301 wait complete. 2024-01-16T15:02:51.147 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:02:51.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:51.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:02:51.242 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[42465]: fuse finished with error 0 and tester_r 0 2024-01-16T15:02:51.246 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:02:51.246 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:51.246 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:02:51.423 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:02:51.423 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:51.481 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:51.482 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:02:51.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:51.482 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:02:51.537 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:02:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:02:51.539 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:02:51.539 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:51.599 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:51.599 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:02:51.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:02:51.599 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:02:51.649 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:02:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:02:51.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:51.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.897+0000 7f4ba7c00640 1 Processor -- start 2024-01-16T15:02:51.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.897+0000 7f4ba7c00640 1 -- start start 2024-01-16T15:02:51.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.898+0000 7f4ba7c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ba013e360 0x7f4ba00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:51.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.898+0000 7f4ba7c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ba015f9c0 0x7f4ba00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:51.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.898+0000 7f4ba7c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba015c5f0 con 0x7f4ba013df80 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.898+0000 7f4ba7c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba006c470 con 0x7f4ba015f9c0 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.898+0000 7f4ba7c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba006c750 con 0x7f4ba013e360 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4ba013df80 0x7f4ba015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44574/0 (socket says 172.21.15.53:44574) 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5200640 1 -- 172.21.15.53:0/3547189987 learned_addr learned my addr 172.21.15.53:0/3547189987 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5c00640 1 --2- 172.21.15.53:0/3547189987 >> v2:172.21.15.53:3300/0 conn(0x7f4ba015f9c0 0x7f4ba00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5c00640 1 -- 172.21.15.53:0/3547189987 >> v2:172.21.15.146:3300/0 conn(0x7f4ba013e360 msgr2=0x7f4ba00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5c00640 1 --2- 172.21.15.53:0/3547189987 >> v2:172.21.15.146:3300/0 conn(0x7f4ba013e360 0x7f4ba00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5c00640 1 -- 172.21.15.53:0/3547189987 >> v1:172.21.15.53:6789/0 conn(0x7f4ba013df80 legacy=0x7f4ba015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:51.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.899+0000 7f4ba5c00640 1 -- 172.21.15.53:0/3547189987 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba006ca30 con 0x7f4ba015f9c0 2024-01-16T15:02:51.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.900+0000 7f4ba5c00640 1 --2- 172.21.15.53:0/3547189987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba015f9c0 0x7f4ba00cf7d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4ba006ded0 tx=0x7f4b9802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1b7718ede60cf6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4b9f400640 1 -- 172.21.15.53:0/3547189987 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98031830 con 0x7f4ba015f9c0 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4b9f400640 1 -- 172.21.15.53:0/3547189987 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98031990 con 0x7f4ba015f9c0 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4b9f400640 1 -- 172.21.15.53:0/3547189987 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b98031d20 con 0x7f4ba015f9c0 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3547189987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba015f9c0 msgr2=0x7f4ba00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3547189987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba015f9c0 0x7f4ba00cf7d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4ba006ded0 tx=0x7f4b9802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3547189987 shutdown_connections 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3547189987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba015f9c0 0x7f4ba00cf7d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3547189987 >> v2:172.21.15.146:3300/0 conn(0x7f4ba013e360 0x7f4ba00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3547189987 >> 172.21.15.53:0/3547189987 conn(0x7f4ba006a800 msgr2=0x7f4ba006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3547189987 shutdown_connections 2024-01-16T15:02:51.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.901+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3547189987 wait complete. 2024-01-16T15:02:51.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.903+0000 7f4ba7c00640 1 Processor -- start 2024-01-16T15:02:51.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.903+0000 7f4ba7c00640 1 -- start start 2024-01-16T15:02:51.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba7c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 0x7f4ba013cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:51.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba7c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba013e360 0x7f4ba013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:51.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba7c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba015f9c0 0x7f4ba00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:51.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba7c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0151490 con 0x7f4ba013df80 2024-01-16T15:02:51.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba7c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0151770 con 0x7f4ba015f9c0 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba7c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba0151a50 con 0x7f4ba013e360 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.904+0000 7f4ba5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 0x7f4ba013cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 0x7f4ba013cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45286/0 (socket says 172.21.15.53:45286) 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 -- 172.21.15.53:0/3131197597 learned_addr learned my addr 172.21.15.53:0/3131197597 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 -- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba013e360 msgr2=0x7f4ba013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba013e360 0x7f4ba013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 -- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba015f9c0 msgr2=0x7f4ba00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba015f9c0 0x7f4ba00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 -- 172.21.15.53:0/3131197597 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b98009400 con 0x7f4ba013df80 2024-01-16T15:02:51.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4ba5200640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 0x7f4ba013cda0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c400 tx=0x7f4b8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:51.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.905+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88005ce0 con 0x7f4ba013df80 2024-01-16T15:02:51.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.906+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b88005e40 con 0x7f4ba013df80 2024-01-16T15:02:51.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.906+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba0151d30 con 0x7f4ba013df80 2024-01-16T15:02:51.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.906+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b88007310 con 0x7f4ba013df80 2024-01-16T15:02:51.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.906+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba013d2e0 con 0x7f4ba013df80 2024-01-16T15:02:51.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.907+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba00675d0 con 0x7f4ba013df80 2024-01-16T15:02:51.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.908+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4b88007470 con 0x7f4ba013df80 2024-01-16T15:02:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.909+0000 7f4b9cc00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7806b500 0x7f4b7806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.909+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4b88089330 con 0x7f4ba013df80 2024-01-16T15:02:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.910+0000 7f4b9fe00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7806b500 0x7f4b7806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.911+0000 7f4b9fe00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7806b500 0x7f4b7806d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4ba01550b0 tx=0x7f4b90009250 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:51.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:51.913+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b880540a0 con 0x7f4ba013df80 2024-01-16T15:02:52.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.148+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ba006df00 con 0x7f4ba013df80 2024-01-16T15:02:52.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.149+0000 7f4b9cc00640 1 -- 172.21.15.53:0/3131197597 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f4b880576c0 con 0x7f4ba013df80 2024-01-16T15:02:52.129 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:52.129 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3324487785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3324487785},{"type":"v1","addr":"172.21.15.146:6836","nonce":3324487785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3124134221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3124134221},{"type":"v1","addr":"172.21.15.146:6837","nonce":3124134221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1994154705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1994154705},{"type":"v1","addr":"172.21.15.53:6834","nonce":1994154705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5595,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1853002600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1853002600},{"type":"v1","addr":"172.21.15.53:6835","nonce":1853002600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-01-16T15:02:52.129 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 19 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7806b500 msgr2=0x7f4b7806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7806b500 0x7f4b7806d9c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4ba01550b0 tx=0x7f4b90009250 comp rx=0 tx=0).stop 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 msgr2=0x7f4ba013cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 0x7f4ba013cda0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c400 tx=0x7f4b8800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 shutdown_connections 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7806b500 0x7f4b7806d9c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.152+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba015f9c0 0x7f4ba00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.153+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba013e360 0x7f4ba013d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.153+0000 7f4ba7c00640 1 --2- 172.21.15.53:0/3131197597 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba013df80 0x7f4ba013cda0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.153+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 >> 172.21.15.53:0/3131197597 conn(0x7f4ba006a800 msgr2=0x7f4ba00cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:52.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.153+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 shutdown_connections 2024-01-16T15:02:52.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.153+0000 7f4ba7c00640 1 -- 172.21.15.53:0/3131197597 wait complete. 2024-01-16T15:02:52.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:02:52.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.397+0000 7fc1c7800640 1 Processor -- start 2024-01-16T15:02:52.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.397+0000 7fc1c7800640 1 -- start start 2024-01-16T15:02:52.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.398+0000 7fc1c7800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc1c013df80 0x7fc1c015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.398+0000 7fc1c7800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc1c00cf9a0 0x7fc1c00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1c7800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c006c470 con 0x7fc1c013e360 2024-01-16T15:02:52.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1c7800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c006c750 con 0x7fc1c00cf9a0 2024-01-16T15:02:52.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1c7800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1c006ca30 con 0x7fc1c013df80 2024-01-16T15:02:52.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1b7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc1c013e360 0x7fc1c00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44580/0 (socket says 172.21.15.53:44580) 2024-01-16T15:02:52.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1b7e00640 1 -- 172.21.15.53:0/1584039649 learned_addr learned my addr 172.21.15.53:0/1584039649 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:52.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1c5800640 1 --2- 172.21.15.53:0/1584039649 >> v2:172.21.15.53:3300/0 conn(0x7fc1c00cf9a0 0x7fc1c00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.399+0000 7fc1c4e00640 1 --2- 172.21.15.53:0/1584039649 >> v2:172.21.15.146:3300/0 conn(0x7fc1c013df80 0x7fc1c015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.400+0000 7fc1c5800640 1 -- 172.21.15.53:0/1584039649 >> v2:172.21.15.146:3300/0 conn(0x7fc1c013df80 msgr2=0x7fc1c015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:52.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.400+0000 7fc1c5800640 1 --2- 172.21.15.53:0/1584039649 >> v2:172.21.15.146:3300/0 conn(0x7fc1c013df80 0x7fc1c015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.400+0000 7fc1c5800640 1 -- 172.21.15.53:0/1584039649 >> v1:172.21.15.53:6789/0 conn(0x7fc1c013e360 legacy=0x7fc1c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.400+0000 7fc1c5800640 1 -- 172.21.15.53:0/1584039649 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c006cd10 con 0x7fc1c00cf9a0 2024-01-16T15:02:52.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.401+0000 7fc1c5800640 1 --2- 172.21.15.53:0/1584039649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c00cd840 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0072770 tx=0x7fc1b802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db082c7e7f285eb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:52.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.401+0000 7fc1bfe00640 1 -- 172.21.15.53:0/1584039649 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b802cc80 con 0x7fc1c00cf9a0 2024-01-16T15:02:52.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1bfe00640 1 -- 172.21.15.53:0/1584039649 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1b802cde0 con 0x7fc1c00cf9a0 2024-01-16T15:02:52.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1bfe00640 1 -- 172.21.15.53:0/1584039649 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8034660 con 0x7fc1c00cf9a0 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 -- 172.21.15.53:0/1584039649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c00cf9a0 msgr2=0x7fc1c00cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 --2- 172.21.15.53:0/1584039649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c00cd840 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fc1c0072770 tx=0x7fc1b802bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 -- 172.21.15.53:0/1584039649 shutdown_connections 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 --2- 172.21.15.53:0/1584039649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c00cd840 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 --2- 172.21.15.53:0/1584039649 >> v2:172.21.15.146:3300/0 conn(0x7fc1c013df80 0x7fc1c015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 -- 172.21.15.53:0/1584039649 >> 172.21.15.53:0/1584039649 conn(0x7fc1c006a800 msgr2=0x7fc1c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 -- 172.21.15.53:0/1584039649 shutdown_connections 2024-01-16T15:02:52.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.402+0000 7fc1c7800640 1 -- 172.21.15.53:0/1584039649 wait complete. 2024-01-16T15:02:52.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.404+0000 7fc1c7800640 1 Processor -- start 2024-01-16T15:02:52.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.404+0000 7fc1c7800640 1 -- start start 2024-01-16T15:02:52.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.405+0000 7fc1c7800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c013d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.405+0000 7fc1c7800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 0x7fc1c0153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.405+0000 7fc1c7800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1c013e360 0x7fc1c0153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.405+0000 7fc1c7800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c014b130 con 0x7fc1c013df80 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.405+0000 7fc1c7800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c01515b0 con 0x7fc1c00cf9a0 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.405+0000 7fc1c7800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0151890 con 0x7fc1c013e360 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 0x7fc1c0153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 0x7fc1c0153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45302/0 (socket says 172.21.15.53:45302) 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 -- 172.21.15.53:0/135538619 learned_addr learned my addr 172.21.15.53:0/135538619 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1c4e00640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c013d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1c5800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1c013e360 0x7fc1c0153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 -- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1c013e360 msgr2=0x7fc1c0153680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1c013e360 0x7fc1c0153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 -- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1c00cf9a0 msgr2=0x7fc1c013d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c013d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 -- 172.21.15.53:0/135538619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b8009a70 con 0x7fc1c013df80 2024-01-16T15:02:52.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.406+0000 7fc1b7e00640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 0x7fc1c0153140 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c500 tx=0x7fc1a800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:52.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.407+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a8005ce0 con 0x7fc1c013df80 2024-01-16T15:02:52.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.407+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1a8005e40 con 0x7fc1c013df80 2024-01-16T15:02:52.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.407+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a80072f0 con 0x7fc1c013df80 2024-01-16T15:02:52.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.407+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c014b380 con 0x7fc1c013df80 2024-01-16T15:02:52.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.407+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c013dcf0 con 0x7fc1c013df80 2024-01-16T15:02:52.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.408+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c014b8a0 con 0x7fc1c013df80 2024-01-16T15:02:52.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.409+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc1a8007510 con 0x7fc1c013df80 2024-01-16T15:02:52.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.410+0000 7fc1bd600640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc19806b500 0x7fc19806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.410+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc1a8089160 con 0x7fc1c013df80 2024-01-16T15:02:52.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.411+0000 7fc1c4e00640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc19806b500 0x7fc19806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.412+0000 7fc1c4e00640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc19806b500 0x7fc19806d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0006fd0 tx=0x7fc1b0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:52.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.414+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1c014b8a0 con 0x7fc1c013df80 2024-01-16T15:02:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.610+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fc1c014b8a0 con 0x7fc1c013df80 2024-01-16T15:02:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.619+0000 7fc1bd600640 1 -- 172.21.15.53:0/135538619 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v12) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc1c014b8a0 con 0x7fc1c013df80 2024-01-16T15:02:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.622+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc19806b500 msgr2=0x7fc19806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.622+0000 7fc1c7800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc19806b500 0x7fc19806d9c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc1b0006fd0 tx=0x7fc1b0008040 comp rx=0 tx=0).stop 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.622+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 msgr2=0x7fc1c0153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.622+0000 7fc1c7800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 0x7fc1c0153140 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fc1a800c500 tx=0x7fc1a800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 shutdown_connections 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc19806b500 0x7fc19806d9c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc1c013e360 0x7fc1c0153680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc1c013df80 0x7fc1c0153140 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 --2- 172.21.15.53:0/135538619 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc1c00cf9a0 0x7fc1c013d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 >> 172.21.15.53:0/135538619 conn(0x7fc1c006a800 msgr2=0x7fc1c015d1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 shutdown_connections 2024-01-16T15:02:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.623+0000 7fc1c7800640 1 -- 172.21.15.53:0/135538619 wait complete. 2024-01-16T15:02:52.620 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:02:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.860+0000 7f1ee9400640 1 Processor -- start 2024-01-16T15:02:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.860+0000 7f1ee9400640 1 -- start start 2024-01-16T15:02:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ee40cd8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee9400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee406b670 con 0x7f1ee40cde00 2024-01-16T15:02:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee406b910 con 0x7f1ee415d2a0 2024-01-16T15:02:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40ce2b0 con 0x7f1ee406c6d0 2024-01-16T15:02:52.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee8a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1ee40cde00 0x7f1ee40b0840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44590/0 (socket says 172.21.15.53:44590) 2024-01-16T15:02:52.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee8a00640 1 -- 172.21.15.53:0/3864844703 learned_addr learned my addr 172.21.15.53:0/3864844703 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:52.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.862+0000 7f1ee3400640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ee40cd8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899214137 0 0) 0x7f1ee406b670 con 0x7f1ee40cde00 2024-01-16T15:02:52.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ec4003520 con 0x7f1ee40cde00 2024-01-16T15:02:52.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189548710 0 0) 0x7f1ec4003520 con 0x7f1ee40cde00 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 msgr2=0x7f1ee40cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ee40cd8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 >> v1:172.21.15.146:6789/0 conn(0x7f1ee415d2a0 legacy=0x7f1ee40cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee406bbf0 con 0x7f1ee40cde00 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1edc001670 con 0x7f1ee40cde00 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 >> v1:172.21.15.53:6789/0 conn(0x7f1ee40cde00 legacy=0x7f1ee40b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec40039a0 0x7f1ec4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ec40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ec4004d00 0x7f1ec4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40ce2b0 con 0x7f1ec4004d00 2024-01-16T15:02:52.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec400ba60 con 0x7f1ec40039a0 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.863+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec400bc20 con 0x7f1ee406c6d0 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ec4004d00 0x7f1ec4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee3e00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec40039a0 0x7f1ec4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee3400640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ec40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee8a00640 1 -- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 msgr2=0x7f1ec40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ec40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee8a00640 1 -- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec40039a0 msgr2=0x7f1ec4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ec40039a0 0x7f1ec4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee8a00640 1 -- 172.21.15.53:0/3864844703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec4003520 con 0x7f1ec4004d00 2024-01-16T15:02:52.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.864+0000 7f1ee3400640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ee406c6d0 0x7f1ec40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.865+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ec4004d00 0x7f1ec4005180 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1edc001eb0 tx=0x7f1ec40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6df3eda59b5a94 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.865+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc005ad0 con 0x7f1ec4004d00 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.865+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc005c30 con 0x7f1ec4004d00 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.865+0000 7f1ee2a00640 1 -- 172.21.15.53:0/3864844703 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc02ca00 con 0x7f1ec4004d00 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee9400640 1 -- 172.21.15.53:0/3864844703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ec4004d00 msgr2=0x7f1ec4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee9400640 1 --2- 172.21.15.53:0/3864844703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ec4004d00 0x7f1ec4005180 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f1edc001eb0 tx=0x7f1ec40057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee8a00640 1 -- 172.21.15.53:0/3864844703 reap_dead start 2024-01-16T15:02:52.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee9400640 1 -- 172.21.15.53:0/3864844703 shutdown_connections 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee9400640 1 -- 172.21.15.53:0/3864844703 >> 172.21.15.53:0/3864844703 conn(0x7f1ee40b2a30 msgr2=0x7f1ee40b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee9400640 1 -- 172.21.15.53:0/3864844703 shutdown_connections 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.866+0000 7f1ee9400640 1 -- 172.21.15.53:0/3864844703 wait complete. 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.869+0000 7f1ee9400640 1 Processor -- start 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.869+0000 7f1ee9400640 1 -- start start 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.870+0000 7f1ee9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4004d00 0x7f1ee41579c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.870+0000 7f1ee9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee415d800 0x7f1ee40c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.870+0000 7f1ee9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 0x7f1ee40aed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.870+0000 7f1ee9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee41582a0 con 0x7f1ee41611c0 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.870+0000 7f1ee9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40b47e0 con 0x7f1ee415d800 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.870+0000 7f1ee9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee40b4a30 con 0x7f1ec4004d00 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 0x7f1ee40aed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 0x7f1ee40aed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45310/0 (socket says 172.21.15.53:45310) 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 -- 172.21.15.53:0/1096184253 learned_addr learned my addr 172.21.15.53:0/1096184253 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee3e00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4004d00 0x7f1ee41579c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee3400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee415d800 0x7f1ee40c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 -- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4004d00 msgr2=0x7f1ee41579c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4004d00 0x7f1ee41579c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 -- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee415d800 msgr2=0x7f1ee40c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee415d800 0x7f1ee40c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.871+0000 7f1ee8a00640 1 -- 172.21.15.53:0/1096184253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee40b4d10 con 0x7f1ee41611c0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.872+0000 7f1ee8a00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 0x7f1ee40aed60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00c1d0 tx=0x7f1edc001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.872+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc001640 con 0x7f1ee41611c0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.872+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc03a070 con 0x7f1ee41611c0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.872+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee40b4ff0 con 0x7f1ee41611c0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.872+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1edc02c9c0 con 0x7f1ee41611c0 2024-01-16T15:02:52.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.872+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee40af2a0 con 0x7f1ee41611c0 2024-01-16T15:02:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.873+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee40ce600 con 0x7f1ee41611c0 2024-01-16T15:02:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.874+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f1edc032070 con 0x7f1ee41611c0 2024-01-16T15:02:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.875+0000 7f1edbe00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ebc06b500 0x7f1ebc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.875+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f1edc0aae80 con 0x7f1ee41611c0 2024-01-16T15:02:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.876+0000 7f1ee3e00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ebc06b500 0x7f1ebc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:52.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.877+0000 7f1ee3e00640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ebc06b500 0x7f1ebc06d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4007fd0 tx=0x7f1ed4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:52.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:52.879+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1edc075da0 con 0x7f1ee41611c0 2024-01-16T15:02:53.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.076+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1ee4067450 con 0x7f1ee41611c0 2024-01-16T15:02:53.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.083+0000 7f1edbe00640 1 -- 172.21.15.53:0/1096184253 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1edc0793c0 con 0x7f1ee41611c0 2024-01-16T15:02:53.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ebc06b500 msgr2=0x7f1ebc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ebc06b500 0x7f1ebc06d9c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ed4007fd0 tx=0x7f1ed4009040 comp rx=0 tx=0).stop 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 msgr2=0x7f1ee40aed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 0x7f1ee40aed60 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00c1d0 tx=0x7f1edc001930 comp rx=0 tx=0).stop 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 shutdown_connections 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ebc06b500 0x7f1ebc06d9c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ee41611c0 0x7f1ee40aed60 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ee415d800 0x7f1ee40c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 --2- 172.21.15.53:0/1096184253 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ec4004d00 0x7f1ee41579c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 >> 172.21.15.53:0/1096184253 conn(0x7f1ee40b2a30 msgr2=0x7f1ee40b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.086+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 shutdown_connections 2024-01-16T15:02:53.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.087+0000 7f1ee9400640 1 -- 172.21.15.53:0/1096184253 wait complete. 2024-01-16T15:02:53.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone' 2024-01-16T15:02:53.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.323+0000 7f9239a00640 1 Processor -- start 2024-01-16T15:02:53.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.323+0000 7f9239a00640 1 -- start start 2024-01-16T15:02:53.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9239a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f92340d0150 0x7f92340d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9239a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f92340ceea0 0x7f92340b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9239a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92340b2c70 con 0x7f923415d3c0 2024-01-16T15:02:53.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9239a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340cf6d0 con 0x7f92340d0150 2024-01-16T15:02:53.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9239a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92340cf3f0 con 0x7f92340ceea0 2024-01-16T15:02:53.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9233e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f923415d3c0 0x7f92340cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44596/0 (socket says 172.21.15.53:44596) 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9233e00640 1 -- 172.21.15.53:0/979101516 learned_addr learned my addr 172.21.15.53:0/979101516 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.325+0000 7f9233400640 1 --2- 172.21.15.53:0/979101516 >> v2:172.21.15.53:3300/0 conn(0x7f92340d0150 0x7f92340d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.326+0000 7f9239000640 1 --2- 172.21.15.53:0/979101516 >> v2:172.21.15.146:3300/0 conn(0x7f92340ceea0 0x7f92340b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.326+0000 7f9233400640 1 -- 172.21.15.53:0/979101516 >> v2:172.21.15.146:3300/0 conn(0x7f92340ceea0 msgr2=0x7f92340b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.326+0000 7f9233400640 1 --2- 172.21.15.53:0/979101516 >> v2:172.21.15.146:3300/0 conn(0x7f92340ceea0 0x7f92340b3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.326+0000 7f9233400640 1 -- 172.21.15.53:0/979101516 >> v1:172.21.15.53:6789/0 conn(0x7f923415d3c0 legacy=0x7f92340cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.326+0000 7f9233400640 1 -- 172.21.15.53:0/979101516 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92340b2620 con 0x7f92340d0150 2024-01-16T15:02:53.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.326+0000 7f9239000640 1 --2- 172.21.15.53:0/979101516 >> v2:172.21.15.146:3300/0 conn(0x7f92340ceea0 0x7f92340b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:53.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.327+0000 7f9233400640 1 --2- 172.21.15.53:0/979101516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340d0150 0x7f92340d1f20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9234072ad0 tx=0x7f921c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89aaef60c3e9ec26 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.327+0000 7f9232a00640 1 -- 172.21.15.53:0/979101516 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c031830 con 0x7f92340d0150 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.327+0000 7f9232a00640 1 -- 172.21.15.53:0/979101516 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f921c031990 con 0x7f92340d0150 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.327+0000 7f9232a00640 1 -- 172.21.15.53:0/979101516 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f921c031d20 con 0x7f92340d0150 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 -- 172.21.15.53:0/979101516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340d0150 msgr2=0x7f92340d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 --2- 172.21.15.53:0/979101516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340d0150 0x7f92340d1f20 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f9234072ad0 tx=0x7f921c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 -- 172.21.15.53:0/979101516 shutdown_connections 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 --2- 172.21.15.53:0/979101516 >> v2:172.21.15.146:3300/0 conn(0x7f92340ceea0 0x7f92340b3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 --2- 172.21.15.53:0/979101516 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340d0150 0x7f92340d1f20 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 -- 172.21.15.53:0/979101516 >> 172.21.15.53:0/979101516 conn(0x7f92340d3050 msgr2=0x7f92340d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 -- 172.21.15.53:0/979101516 shutdown_connections 2024-01-16T15:02:53.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.328+0000 7f9239a00640 1 -- 172.21.15.53:0/979101516 wait complete. 2024-01-16T15:02:53.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.330+0000 7f9239a00640 1 Processor -- start 2024-01-16T15:02:53.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.330+0000 7f9239a00640 1 -- start start 2024-01-16T15:02:53.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.331+0000 7f9239a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 0x7f92340c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.331+0000 7f9239a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92340d0150 0x7f92340c6b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.331+0000 7f9239a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f923415d3c0 0x7f92341f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.331+0000 7f9239a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92340c9b70 con 0x7f92340ceea0 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.331+0000 7f9239a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9234159a10 con 0x7f92340d0150 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.331+0000 7f9239a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92341f31f0 con 0x7f923415d3c0 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 0x7f92340c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 0x7f92340c8fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45324/0 (socket says 172.21.15.53:45324) 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 -- 172.21.15.53:0/3657637441 learned_addr learned my addr 172.21.15.53:0/3657637441 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9239000640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f923415d3c0 0x7f92341f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233400640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92340d0150 0x7f92340c6b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 -- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f923415d3c0 msgr2=0x7f92341f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f923415d3c0 0x7f92341f2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 -- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92340d0150 msgr2=0x7f92340c6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92340d0150 0x7f92340c6b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233e00640 1 -- 172.21.15.53:0/3657637441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c009400 con 0x7f92340ceea0 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.332+0000 7f9233400640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92340d0150 0x7f92340c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.333+0000 7f9233e00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 0x7f92340c8fa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f922400c480 tx=0x7f922400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.333+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9224005ce0 con 0x7f92340ceea0 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.333+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9224005e40 con 0x7f92340ceea0 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.333+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92240072f0 con 0x7f92340ceea0 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.333+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92340c95b0 con 0x7f92340ceea0 2024-01-16T15:02:53.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.333+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92340b6e60 con 0x7f92340ceea0 2024-01-16T15:02:53.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.334+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92340b6cc0 con 0x7f92340ceea0 2024-01-16T15:02:53.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.335+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9224007510 con 0x7f92340ceea0 2024-01-16T15:02:53.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.336+0000 7f922be00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f920c06b500 0x7f920c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.336+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f92240891c0 con 0x7f92340ceea0 2024-01-16T15:02:53.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.337+0000 7f9233400640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f920c06b500 0x7f920c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.338+0000 7f9233400640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f920c06b500 0x7f920c06d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f921c036000 tx=0x7f921c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:53.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.341+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f92340b6cc0 con 0x7f92340ceea0 2024-01-16T15:02:53.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.608+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]} v 0) v1 -- 0x7f92340b6cc0 con 0x7f92340ceea0 2024-01-16T15:02:53.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.621+0000 7f922be00640 1 -- 172.21.15.53:0/3657637441 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_1_clone"]}]=0 v165) v1 ==== 152+0+0 (secure 0 0 0) 0x7f92340b6cc0 con 0x7f92340ceea0 2024-01-16T15:02:53.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f920c06b500 msgr2=0x7f920c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f920c06b500 0x7f920c06d9c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f921c036000 tx=0x7f921c008030 comp rx=0 tx=0).stop 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 msgr2=0x7f92340c8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 0x7f92340c8fa0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f922400c480 tx=0x7f922400c940 comp rx=0 tx=0).stop 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 shutdown_connections 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f920c06b500 0x7f920c06d9c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f923415d3c0 0x7f92341f2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f92340d0150 0x7f92340c6b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 --2- 172.21.15.53:0/3657637441 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f92340ceea0 0x7f92340c8fa0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 >> 172.21.15.53:0/3657637441 conn(0x7f92340d3050 msgr2=0x7f92340b3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 shutdown_connections 2024-01-16T15:02:53.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.624+0000 7f9239a00640 1 -- 172.21.15.53:0/3657637441 wait complete. 2024-01-16T15:02:53.622 INFO:tasks.cephfs_test_runner:Test that when one clone is in progress, one progress bar is printed ... ok 2024-01-16T15:02:53.622 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:02:53.622 INFO:tasks.cephfs_test_runner:test_progress_bar_for_4_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:02:53.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:53.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.868+0000 7f50fca00640 1 Processor -- start 2024-01-16T15:02:53.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.868+0000 7f50fca00640 1 -- start start 2024-01-16T15:02:53.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.870+0000 7f50fca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f815afb0 0x7f50f813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.870+0000 7f50fca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f50f813fae0 0x7f50f80b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.870+0000 7f50fca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50f813ff00 con 0x7f50f813cfa0 2024-01-16T15:02:53.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.870+0000 7f50fca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50f80cdc40 con 0x7f50f813fae0 2024-01-16T15:02:53.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.870+0000 7f50fca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f80cdec0 con 0x7f50f815afb0 2024-01-16T15:02:53.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f50f813fae0 0x7f50f80b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f815afb0 0x7f50f813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f50f813fae0 0x7f50f80b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:59428/0 (socket says 172.21.15.53:59428) 2024-01-16T15:02:53.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 -- 172.21.15.53:0/2459852330 learned_addr learned my addr 172.21.15.53:0/2459852330 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:53.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 -- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f815afb0 msgr2=0x7f50f813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 --2- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f815afb0 0x7f50f813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 -- 172.21.15.53:0/2459852330 >> v1:172.21.15.146:6789/0 conn(0x7f50f813cfa0 legacy=0x7f50f80cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.871+0000 7f50f7e00640 1 -- 172.21.15.53:0/2459852330 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f80ce1a0 con 0x7f50f813fae0 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.873+0000 7f50f7e00640 1 --2- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813fae0 0x7f50f80b2040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f50f000d150 tx=0x7f50f002e0e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fb80a6698878207 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.873+0000 7f50f6a00640 1 -- 172.21.15.53:0/2459852330 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f002ea80 con 0x7f50f813fae0 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.873+0000 7f50f6a00640 1 -- 172.21.15.53:0/2459852330 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f002ebe0 con 0x7f50f813fae0 2024-01-16T15:02:53.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.873+0000 7f50f6a00640 1 -- 172.21.15.53:0/2459852330 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0009600 con 0x7f50f813fae0 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 -- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813fae0 msgr2=0x7f50f80b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 --2- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813fae0 0x7f50f80b2040 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f50f000d150 tx=0x7f50f002e0e0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 -- 172.21.15.53:0/2459852330 shutdown_connections 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 --2- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813fae0 0x7f50f80b2040 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 --2- 172.21.15.53:0/2459852330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f815afb0 0x7f50f813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 -- 172.21.15.53:0/2459852330 >> 172.21.15.53:0/2459852330 conn(0x7f50f806a800 msgr2=0x7f50f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:53.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 -- 172.21.15.53:0/2459852330 shutdown_connections 2024-01-16T15:02:53.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.874+0000 7f50fca00640 1 -- 172.21.15.53:0/2459852330 wait complete. 2024-01-16T15:02:53.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.876+0000 7f50fca00640 1 Processor -- start 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.876+0000 7f50fca00640 1 -- start start 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50fca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813cfa0 0x7f50f8153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50fca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f813fae0 0x7f50f8146a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50fca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 0x7f50f813c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50fca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f8136ef0 con 0x7f50f815afb0 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50fca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f813bd10 con 0x7f50f813cfa0 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50fca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50f813b100 con 0x7f50f813fae0 2024-01-16T15:02:53.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.877+0000 7f50f7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 0x7f50f813c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 0x7f50f813c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45340/0 (socket says 172.21.15.53:45340) 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 -- 172.21.15.53:0/2303425204 learned_addr learned my addr 172.21.15.53:0/2303425204 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7400640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813cfa0 0x7f50f8153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50efe00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f813fae0 0x7f50f8146a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 -- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f813fae0 msgr2=0x7f50f8146a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f813fae0 0x7f50f8146a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 -- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813cfa0 msgr2=0x7f50f8153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813cfa0 0x7f50f8153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 -- 172.21.15.53:0/2303425204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f000bd40 con 0x7f50f815afb0 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.878+0000 7f50f7e00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 0x7f50f813c1a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f50f000d150 tx=0x7f50f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.879+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f002ea80 con 0x7f50f815afb0 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.879+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f0009e30 con 0x7f50f815afb0 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.879+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50f0040880 con 0x7f50f815afb0 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.879+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f813b3e0 con 0x7f50f815afb0 2024-01-16T15:02:53.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.879+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f813b9a0 con 0x7f50f815afb0 2024-01-16T15:02:53.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.880+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f8067450 con 0x7f50f815afb0 2024-01-16T15:02:53.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.881+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f50f002ed50 con 0x7f50f815afb0 2024-01-16T15:02:53.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.882+0000 7f50ef400640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50d006b500 0x7f50d006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:53.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.882+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f50f00ae900 con 0x7f50f815afb0 2024-01-16T15:02:53.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.883+0000 7f50f7400640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50d006b500 0x7f50d006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:53.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.884+0000 7f50f7400640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50d006b500 0x7f50d006d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f50f80af4d0 tx=0x7f50e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:53.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:53.887+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50f0079700 con 0x7f50f815afb0 2024-01-16T15:02:54.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.082+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50f813be50 con 0x7f50f815afb0 2024-01-16T15:02:54.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.083+0000 7f50ef400640 1 -- 172.21.15.53:0/2303425204 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v38) v1 ==== 74+0+12950 (secure 0 0 0) 0x7f50f007cd20 con 0x7f50f815afb0 2024-01-16T15:02:54.063 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:54.063 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":38,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:02:47.579329+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/2729092299":"2024-01-17T15:02:44.562599+0000","172.21.15.53:6833/2729092299":"2024-01-17T15:02:44.562599+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:54.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50d006b500 msgr2=0x7f50d006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50d006b500 0x7f50d006d9c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f50f80af4d0 tx=0x7f50e8009040 comp rx=0 tx=0).stop 2024-01-16T15:02:54.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 msgr2=0x7f50f813c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 0x7f50f813c1a0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f50f000d150 tx=0x7f50f0004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:54.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 shutdown_connections 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50d006b500 0x7f50d006d9c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50f815afb0 0x7f50f813c1a0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50f813fae0 0x7f50f8146a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 --2- 172.21.15.53:0/2303425204 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50f813cfa0 0x7f50f8153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 >> 172.21.15.53:0/2303425204 conn(0x7f50f806a800 msgr2=0x7f50f80cd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 shutdown_connections 2024-01-16T15:02:54.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.086+0000 7f50fca00640 1 -- 172.21.15.53:0/2303425204 wait complete. 2024-01-16T15:02:54.085 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:54.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:54.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.333+0000 7f44b7e00640 1 Processor -- start 2024-01-16T15:02:54.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.333+0000 7f44b7e00640 1 -- start start 2024-01-16T15:02:54.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f44b0150760 0x7f44b0072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f44b015d300 0x7f44b015d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b00b2bd0 0x7f44b00b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b015dc80 con 0x7f44b015d300 2024-01-16T15:02:54.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b0071190 con 0x7f44b0150760 2024-01-16T15:02:54.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0071470 con 0x7f44b00b2bd0 2024-01-16T15:02:54.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f44b015d300 0x7f44b015d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f44b015d300 0x7f44b015d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45344/0 (socket says 172.21.15.53:45344) 2024-01-16T15:02:54.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b4a00640 1 -- 172.21.15.53:0/370792184 learned_addr learned my addr 172.21.15.53:0/370792184 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.335+0000 7f44b5400640 1 --2- 172.21.15.53:0/370792184 >> v2:172.21.15.146:3300/0 conn(0x7f44b0150760 0x7f44b0072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.336+0000 7f44b5e00640 1 --2- 172.21.15.53:0/370792184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b00b2bd0 0x7f44b00b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.336+0000 7f44b4a00640 1 -- 172.21.15.53:0/370792184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b00b2bd0 msgr2=0x7f44b00b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.336+0000 7f44b4a00640 1 --2- 172.21.15.53:0/370792184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b00b2bd0 0x7f44b00b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.336+0000 7f44b4a00640 1 -- 172.21.15.53:0/370792184 >> v2:172.21.15.146:3300/0 conn(0x7f44b0150760 msgr2=0x7f44b0072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.336+0000 7f44b4a00640 1 --2- 172.21.15.53:0/370792184 >> v2:172.21.15.146:3300/0 conn(0x7f44b0150760 0x7f44b0072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.336+0000 7f44b4a00640 1 -- 172.21.15.53:0/370792184 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b0071750 con 0x7f44b015d300 2024-01-16T15:02:54.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.337+0000 7f44b4a00640 1 --2- 172.21.15.53:0/370792184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b015d300 0x7f44b015d740 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f44b006e420 tx=0x7f44a002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b02d398b9187c6ba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:54.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.337+0000 7f44afe00640 1 -- 172.21.15.53:0/370792184 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0031870 con 0x7f44b015d300 2024-01-16T15:02:54.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.337+0000 7f44afe00640 1 -- 172.21.15.53:0/370792184 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a00319d0 con 0x7f44b015d300 2024-01-16T15:02:54.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.337+0000 7f44afe00640 1 -- 172.21.15.53:0/370792184 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0031d60 con 0x7f44b015d300 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 -- 172.21.15.53:0/370792184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b015d300 msgr2=0x7f44b015d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 --2- 172.21.15.53:0/370792184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b015d300 0x7f44b015d740 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f44b006e420 tx=0x7f44a002b910 comp rx=0 tx=0).stop 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 -- 172.21.15.53:0/370792184 shutdown_connections 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 --2- 172.21.15.53:0/370792184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b00b2bd0 0x7f44b00b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 --2- 172.21.15.53:0/370792184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b015d300 0x7f44b015d740 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 --2- 172.21.15.53:0/370792184 >> v2:172.21.15.146:3300/0 conn(0x7f44b0150760 0x7f44b0072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 -- 172.21.15.53:0/370792184 >> 172.21.15.53:0/370792184 conn(0x7f44b006a800 msgr2=0x7f44b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:54.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 -- 172.21.15.53:0/370792184 shutdown_connections 2024-01-16T15:02:54.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.338+0000 7f44b7e00640 1 -- 172.21.15.53:0/370792184 wait complete. 2024-01-16T15:02:54.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.340+0000 7f44b7e00640 1 Processor -- start 2024-01-16T15:02:54.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.340+0000 7f44b7e00640 1 -- start start 2024-01-16T15:02:54.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 0x7f44b0149360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b0150760 0x7f44b013f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b015d300 0x7f44b013fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b013ff60 con 0x7f44b00b2bd0 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0146dd0 con 0x7f44b015d300 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b0147070 con 0x7f44b0150760 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 0x7f44b0149360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 0x7f44b0149360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45356/0 (socket says 172.21.15.53:45356) 2024-01-16T15:02:54.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.341+0000 7f44b5400640 1 -- 172.21.15.53:0/420976300 learned_addr learned my addr 172.21.15.53:0/420976300 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b4a00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b0150760 0x7f44b013f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b015d300 0x7f44b013fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5400640 1 -- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b0150760 msgr2=0x7f44b013f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5400640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b0150760 0x7f44b013f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5400640 1 -- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b015d300 msgr2=0x7f44b013fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5400640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b015d300 0x7f44b013fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5400640 1 -- 172.21.15.53:0/420976300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a00094d0 con 0x7f44b00b2bd0 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b015d300 0x7f44b013fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44b5400640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 0x7f44b0149360 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f449800d800 tx=0x7f449800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:54.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.342+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449800b6c0 con 0x7f44b00b2bd0 2024-01-16T15:02:54.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.343+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449800b820 con 0x7f44b00b2bd0 2024-01-16T15:02:54.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.343+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449800bbb0 con 0x7f44b00b2bd0 2024-01-16T15:02:54.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.343+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b0147350 con 0x7f44b00b2bd0 2024-01-16T15:02:54.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.343+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b0147910 con 0x7f44b00b2bd0 2024-01-16T15:02:54.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.344+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b0147ad0 con 0x7f44b00b2bd0 2024-01-16T15:02:54.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.345+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f449800bd10 con 0x7f44b00b2bd0 2024-01-16T15:02:54.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.346+0000 7f44ad600640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f448806b530 0x7f448806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.346+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f449808a2c0 con 0x7f44b00b2bd0 2024-01-16T15:02:54.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.346+0000 7f44b4a00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f448806b530 0x7f448806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.348+0000 7f44b4a00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f448806b530 0x7f448806d9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f44a0036000 tx=0x7f44a003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:54.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.350+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4498055030 con 0x7f44b00b2bd0 2024-01-16T15:02:54.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.540+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44b00675d0 con 0x7f44b00b2bd0 2024-01-16T15:02:54.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.541+0000 7f44ad600640 1 -- 172.21.15.53:0/420976300 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v38) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4498058650 con 0x7f44b00b2bd0 2024-01-16T15:02:54.521 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:02:54.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f448806b530 msgr2=0x7f448806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f448806b530 0x7f448806d9f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f44a0036000 tx=0x7f44a003c040 comp rx=0 tx=0).stop 2024-01-16T15:02:54.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 msgr2=0x7f44b0149360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 0x7f44b0149360 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f449800d800 tx=0x7f449800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 shutdown_connections 2024-01-16T15:02:54.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f448806b530 0x7f448806d9f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f44b015d300 0x7f44b013fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f44b0150760 0x7f44b013f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 --2- 172.21.15.53:0/420976300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f44b00b2bd0 0x7f44b0149360 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 >> 172.21.15.53:0/420976300 conn(0x7f44b006a800 msgr2=0x7f44b006cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 shutdown_connections 2024-01-16T15:02:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.544+0000 7f44b7e00640 1 -- 172.21.15.53:0/420976300 wait complete. 2024-01-16T15:02:54.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:02:54.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.774+0000 7fa1b7200640 1 Processor -- start 2024-01-16T15:02:54.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.775+0000 7fa1b7200640 1 -- start start 2024-01-16T15:02:54.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.776+0000 7fa1b7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1b813cfa0 0x7fa1b815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.776+0000 7fa1b7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa1b80cd530 0x7fa1b813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b80cdec0 con 0x7fa1b80cd9e0 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b7200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b80ce1a0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b80ce480 con 0x7fa1b80cd530 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1b813cfa0 0x7fa1b815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa1b813cfa0 0x7fa1b815afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45366/0 (socket says 172.21.15.53:45366) 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 -- 172.21.15.53:0/2807167239 learned_addr learned my addr 172.21.15.53:0/2807167239 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1afe00640 1 --2- 172.21.15.53:0/2807167239 >> v2:172.21.15.146:3300/0 conn(0x7fa1b80cd530 0x7fa1b813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 -- 172.21.15.53:0/2807167239 >> v2:172.21.15.146:3300/0 conn(0x7fa1b80cd530 msgr2=0x7fa1b813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 --2- 172.21.15.53:0/2807167239 >> v2:172.21.15.146:3300/0 conn(0x7fa1b80cd530 0x7fa1b813f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 -- 172.21.15.53:0/2807167239 >> v1:172.21.15.146:6789/0 conn(0x7fa1b80cd9e0 legacy=0x7fa1b815d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:54.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.777+0000 7fa1b5e00640 1 -- 172.21.15.53:0/2807167239 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b80ce760 con 0x7fa1b813cfa0 2024-01-16T15:02:54.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.779+0000 7fa1b5e00640 1 --2- 172.21.15.53:0/2807167239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b815afb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa1b815b4f0 tx=0x7fa1a802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=678a5795b8d0842a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:54.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.779+0000 7fa1b5400640 1 -- 172.21.15.53:0/2807167239 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a80317e0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.779+0000 7fa1b5400640 1 -- 172.21.15.53:0/2807167239 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a8031940 con 0x7fa1b813cfa0 2024-01-16T15:02:54.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.779+0000 7fa1b5400640 1 -- 172.21.15.53:0/2807167239 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1a8031cd0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 -- 172.21.15.53:0/2807167239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 msgr2=0x7fa1b815afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 --2- 172.21.15.53:0/2807167239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b815afb0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fa1b815b4f0 tx=0x7fa1a802b880 comp rx=0 tx=0).stop 2024-01-16T15:02:54.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 -- 172.21.15.53:0/2807167239 shutdown_connections 2024-01-16T15:02:54.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 --2- 172.21.15.53:0/2807167239 >> v2:172.21.15.146:3300/0 conn(0x7fa1b80cd530 0x7fa1b813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 --2- 172.21.15.53:0/2807167239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b815afb0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 -- 172.21.15.53:0/2807167239 >> 172.21.15.53:0/2807167239 conn(0x7fa1b806a800 msgr2=0x7fa1b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:54.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 -- 172.21.15.53:0/2807167239 shutdown_connections 2024-01-16T15:02:54.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.780+0000 7fa1b7200640 1 -- 172.21.15.53:0/2807167239 wait complete. 2024-01-16T15:02:54.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.782+0000 7fa1b7200640 1 Processor -- start 2024-01-16T15:02:54.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.783+0000 7fa1b7200640 1 -- start start 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1b80cd530 0x7fa1b80adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1b80cd9e0 0x7fa1b81538e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b813c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b8159300 con 0x7fa1b813cfa0 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81595e0 con 0x7fa1b80cd9e0 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b81598c0 con 0x7fa1b80cd530 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b813c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b813c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45376/0 (socket says 172.21.15.53:45376) 2024-01-16T15:02:54.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.784+0000 7fa1b6800640 1 -- 172.21.15.53:0/3845498097 learned_addr learned my addr 172.21.15.53:0/3845498097 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b5e00640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1b80cd530 0x7fa1b80adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1afe00640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1b80cd9e0 0x7fa1b81538e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b6800640 1 -- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1b80cd530 msgr2=0x7fa1b80adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b6800640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1b80cd530 0x7fa1b80adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b6800640 1 -- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1b80cd9e0 msgr2=0x7fa1b81538e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b6800640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1b80cd9e0 0x7fa1b81538e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b6800640 1 -- 172.21.15.53:0/3845498097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a80094d0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1afe00640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1b80cd9e0 0x7fa1b81538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1b6800640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b813c1e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000c780 tx=0x7fa1b000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.785+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0003d80 con 0x7fa1b813cfa0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.786+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1b000ce30 con 0x7fa1b813cfa0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.786+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1b0005290 con 0x7fa1b813cfa0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.786+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b8159ba0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.786+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b814c500 con 0x7fa1b813cfa0 2024-01-16T15:02:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.787+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b815a0c0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.788+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa1b0006b10 con 0x7fa1b813cfa0 2024-01-16T15:02:54.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.789+0000 7fa1ae000640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa19006b500 0x7fa19006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:54.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.789+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa1b0088ff0 con 0x7fa1b813cfa0 2024-01-16T15:02:54.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.789+0000 7fa1b5e00640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa19006b500 0x7fa19006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:54.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.791+0000 7fa1b5e00640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa19006b500 0x7fa19006d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8036000 tx=0x7fa1a803d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:54.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:54.793+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa1b00bbe90 con 0x7fa1b813cfa0 2024-01-16T15:02:55.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.067+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7fa1b8067450 con 0x7fa1b813cfa0 2024-01-16T15:02:55.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.632+0000 7fa1ae000640 1 -- 172.21.15.53:0/3845498097 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v167) v1 ==== 156+0+0 (secure 0 0 0) 0x7fa1b0017070 con 0x7fa1b813cfa0 2024-01-16T15:02:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.636+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa19006b500 msgr2=0x7fa19006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.636+0000 7fa1b7200640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa19006b500 0x7fa19006d9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa1a8036000 tx=0x7fa1a803d040 comp rx=0 tx=0).stop 2024-01-16T15:02:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.636+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 msgr2=0x7fa1b813c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.636+0000 7fa1b7200640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b813c1e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fa1b000c780 tx=0x7fa1b000cc40 comp rx=0 tx=0).stop 2024-01-16T15:02:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 shutdown_connections 2024-01-16T15:02:55.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa19006b500 0x7fa19006d9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1b813cfa0 0x7fa1b813c1e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa1b80cd9e0 0x7fa1b81538e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 --2- 172.21.15.53:0/3845498097 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1b80cd530 0x7fa1b80adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 >> 172.21.15.53:0/3845498097 conn(0x7fa1b806a800 msgr2=0x7fa1b80b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:55.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 shutdown_connections 2024-01-16T15:02:55.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.637+0000 7fa1b7200640 1 -- 172.21.15.53:0/3845498097 wait complete. 2024-01-16T15:02:55.638 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:02:55.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.886+0000 7ffad4200640 1 Processor -- start 2024-01-16T15:02:55.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.886+0000 7ffad4200640 1 -- start start 2024-01-16T15:02:55.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.887+0000 7ffad4200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffacc143ae0 0x7ffacc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:55.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad4200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffacc0cf7d0 0x7ffacc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:55.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad4200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffacc0cfc30 con 0x7ffacc15e1e0 2024-01-16T15:02:55.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad4200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffacc0b1ee0 con 0x7ffacc0cf7d0 2024-01-16T15:02:55.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad4200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffacc0b2780 con 0x7ffacc143ae0 2024-01-16T15:02:55.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffacc0cf7d0 0x7ffacc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:55.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffacc0cf7d0 0x7ffacc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45390/0 (socket says 172.21.15.53:45390) 2024-01-16T15:02:55.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 -- 172.21.15.53:0/3950210231 learned_addr learned my addr 172.21.15.53:0/3950210231 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:55.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 -- 172.21.15.53:0/3950210231 >> v2:172.21.15.146:3300/0 conn(0x7ffacc143ae0 msgr2=0x7ffacc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:55.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 --2- 172.21.15.53:0/3950210231 >> v2:172.21.15.146:3300/0 conn(0x7ffacc143ae0 0x7ffacc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 -- 172.21.15.53:0/3950210231 >> v1:172.21.15.53:6789/0 conn(0x7ffacc15e1e0 legacy=0x7ffacc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:55.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.888+0000 7ffad0e00640 1 -- 172.21.15.53:0/3950210231 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc15e990 con 0x7ffacc0cf7d0 2024-01-16T15:02:55.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffad0e00640 1 --2- 172.21.15.53:0/3950210231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc0cf7d0 0x7ffacc15dca0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ffacc0cfea0 tx=0x7ffab402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31dd33409b56df3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffacbe00640 1 -- 172.21.15.53:0/3950210231 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4030860 con 0x7ffacc0cf7d0 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffacbe00640 1 -- 172.21.15.53:0/3950210231 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab40309c0 con 0x7ffacc0cf7d0 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffacbe00640 1 -- 172.21.15.53:0/3950210231 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4030d50 con 0x7ffacc0cf7d0 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffad4200640 1 -- 172.21.15.53:0/3950210231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc0cf7d0 msgr2=0x7ffacc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffad4200640 1 --2- 172.21.15.53:0/3950210231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc0cf7d0 0x7ffacc15dca0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7ffacc0cfea0 tx=0x7ffab402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffad4200640 1 -- 172.21.15.53:0/3950210231 shutdown_connections 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.890+0000 7ffad4200640 1 --2- 172.21.15.53:0/3950210231 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc0cf7d0 0x7ffacc15dca0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.891+0000 7ffad4200640 1 --2- 172.21.15.53:0/3950210231 >> v2:172.21.15.146:3300/0 conn(0x7ffacc143ae0 0x7ffacc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.891+0000 7ffad4200640 1 -- 172.21.15.53:0/3950210231 >> 172.21.15.53:0/3950210231 conn(0x7ffacc06a800 msgr2=0x7ffacc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.891+0000 7ffad4200640 1 -- 172.21.15.53:0/3950210231 shutdown_connections 2024-01-16T15:02:55.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.891+0000 7ffad4200640 1 -- 172.21.15.53:0/3950210231 wait complete. 2024-01-16T15:02:55.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.892+0000 7ffad4200640 1 Processor -- start 2024-01-16T15:02:55.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.893+0000 7ffad4200640 1 -- start start 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad4200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffacc0cf7d0 0x7ffacc159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad4200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 0x7ffacc0c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad4200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffacc15e1e0 0x7ffacc0ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad4200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc159f40 con 0x7ffacc143ae0 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad4200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc1410d0 con 0x7ffacc15e1e0 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad4200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffacc1404c0 con 0x7ffacc0cf7d0 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 0x7ffacc0c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 0x7ffacc0c3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45394/0 (socket says 172.21.15.53:45394) 2024-01-16T15:02:55.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 -- 172.21.15.53:0/2241187759 learned_addr learned my addr 172.21.15.53:0/2241187759 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 -- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffacc0cf7d0 msgr2=0x7ffacc159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffacc0cf7d0 0x7ffacc159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 -- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffacc15e1e0 msgr2=0x7ffacc0ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad2200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffacc15e1e0 0x7ffacc0ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.894+0000 7ffad0e00640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffacc15e1e0 0x7ffacc0ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffad0e00640 1 -- 172.21.15.53:0/2241187759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab40094d0 con 0x7ffacc143ae0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffad2200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffacc15e1e0 0x7ffacc0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffad0e00640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 0x7ffacc0c3ee0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffab4000c00 tx=0x7ffab4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4030850 con 0x7ffacc143ae0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab4039d90 con 0x7ffacc143ae0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4043650 con 0x7ffacc143ae0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffacc1407a0 con 0x7ffacc143ae0 2024-01-16T15:02:55.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.895+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffacc140d60 con 0x7ffacc143ae0 2024-01-16T15:02:55.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.897+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffacc15a4c0 con 0x7ffacc143ae0 2024-01-16T15:02:55.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.897+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ffab404c3b0 con 0x7ffacc143ae0 2024-01-16T15:02:55.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.899+0000 7ffac9600640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffaa406b500 0x7ffaa406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:55.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.899+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7ffab40ac660 con 0x7ffacc143ae0 2024-01-16T15:02:55.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.900+0000 7ffad1800640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffaa406b500 0x7ffaa406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:55.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.901+0000 7ffad1800640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffaa406b500 0x7ffaa406d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009710 tx=0x7ffabc006c90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:55.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:55.904+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffab4077350 con 0x7ffacc143ae0 2024-01-16T15:02:56.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.104+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ffacc0675d0 con 0x7ffacc143ae0 2024-01-16T15:02:56.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.109+0000 7ffac9600640 1 -- 172.21.15.53:0/2241187759 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7ffab407a970 con 0x7ffacc143ae0 2024-01-16T15:02:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffaa406b500 msgr2=0x7ffaa406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffaa406b500 0x7ffaa406d9c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffabc009710 tx=0x7ffabc006c90 comp rx=0 tx=0).stop 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 msgr2=0x7ffacc0c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 0x7ffacc0c3ee0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ffab4000c00 tx=0x7ffab4004730 comp rx=0 tx=0).stop 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 shutdown_connections 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffaa406b500 0x7ffaa406d9c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffacc15e1e0 0x7ffacc0ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffacc143ae0 0x7ffacc0c3ee0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 --2- 172.21.15.53:0/2241187759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffacc0cf7d0 0x7ffacc159a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 >> 172.21.15.53:0/2241187759 conn(0x7ffacc06a800 msgr2=0x7ffacc15b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 shutdown_connections 2024-01-16T15:02:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.112+0000 7ffad4200640 1 -- 172.21.15.53:0/2241187759 wait complete. 2024-01-16T15:02:56.111 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:02:56.111 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:56.230 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:02:56.348 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:02:56.349 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:02:56.349 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:56.406 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:56.407 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:02:56.407 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:56.463 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:56.463 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:02:56.463 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:56.521 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:02:56.521 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:02:56.521 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:02:56.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:02:56.575 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:02:56.575 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:02:56.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.825+0000 7f6909200640 1 Processor -- start 2024-01-16T15:02:56.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.826+0000 7f6909200640 1 -- start start 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.827+0000 7f6909200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69040cc500 0x7f69040cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.827+0000 7f6909200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f690406d160 0x7f690406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.827+0000 7f6909200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6904070e50 con 0x7f6904143c40 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.827+0000 7f6909200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040723c0 con 0x7f69040cc500 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.827+0000 7f6909200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69040cced0 con 0x7f690406d160 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f6903400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6904143c40 0x7f6904071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44614/0 (socket says 172.21.15.53:44614) 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f6903400640 1 -- 172.21.15.53:0/4165510100 learned_addr learned my addr 172.21.15.53:0/4165510100 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:56.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f68fa800640 1 --2- 172.21.15.53:0/4165510100 >> v2:172.21.15.53:3300/0 conn(0x7f69040cc500 0x7f69040cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f6903e00640 1 --2- 172.21.15.53:0/4165510100 >> v2:172.21.15.146:3300/0 conn(0x7f690406d160 0x7f690406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f68fa800640 1 -- 172.21.15.53:0/4165510100 >> v2:172.21.15.146:3300/0 conn(0x7f690406d160 msgr2=0x7f690406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:56.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f68fa800640 1 --2- 172.21.15.53:0/4165510100 >> v2:172.21.15.146:3300/0 conn(0x7f690406d160 0x7f690406d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f68fa800640 1 -- 172.21.15.53:0/4165510100 >> v1:172.21.15.53:6789/0 conn(0x7f6904143c40 legacy=0x7f6904071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.828+0000 7f68fa800640 1 -- 172.21.15.53:0/4165510100 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6904144020 con 0x7f69040cc500 2024-01-16T15:02:56.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.829+0000 7f68fa800640 1 --2- 172.21.15.53:0/4165510100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f69040cc990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6904070690 tx=0x7f68ec02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fb197acca26a87 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:56.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.830+0000 7f6902a00640 1 -- 172.21.15.53:0/4165510100 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec02cbe0 con 0x7f69040cc500 2024-01-16T15:02:56.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.830+0000 7f6902a00640 1 -- 172.21.15.53:0/4165510100 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec02cd40 con 0x7f69040cc500 2024-01-16T15:02:56.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.830+0000 7f6902a00640 1 -- 172.21.15.53:0/4165510100 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec034660 con 0x7f69040cc500 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.830+0000 7f6909200640 1 -- 172.21.15.53:0/4165510100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 msgr2=0x7f69040cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.830+0000 7f6909200640 1 --2- 172.21.15.53:0/4165510100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f69040cc990 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6904070690 tx=0x7f68ec02be30 comp rx=0 tx=0).stop 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.831+0000 7f6909200640 1 -- 172.21.15.53:0/4165510100 shutdown_connections 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.831+0000 7f6909200640 1 --2- 172.21.15.53:0/4165510100 >> v2:172.21.15.146:3300/0 conn(0x7f690406d160 0x7f690406d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.831+0000 7f6909200640 1 --2- 172.21.15.53:0/4165510100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f69040cc990 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.831+0000 7f6909200640 1 -- 172.21.15.53:0/4165510100 >> 172.21.15.53:0/4165510100 conn(0x7f690406a800 msgr2=0x7f690406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.831+0000 7f6909200640 1 -- 172.21.15.53:0/4165510100 shutdown_connections 2024-01-16T15:02:56.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.831+0000 7f6909200640 1 -- 172.21.15.53:0/4165510100 wait complete. 2024-01-16T15:02:56.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.832+0000 7f6909200640 1 Processor -- start 2024-01-16T15:02:56.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.833+0000 7f6909200640 1 -- start start 2024-01-16T15:02:56.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.833+0000 7f6909200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f690406d160 0x7f69041e6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.833+0000 7f6909200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f690415a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f6909200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6904143c40 0x7f690415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f6909200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041e2700 con 0x7f69040cc500 2024-01-16T15:02:56.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f6909200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69040a91f0 con 0x7f690406d160 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f6909200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69040a94d0 con 0x7f6904143c40 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f690415a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f690415a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45418/0 (socket says 172.21.15.53:45418) 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 -- 172.21.15.53:0/2040859178 learned_addr learned my addr 172.21.15.53:0/2040859178 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 -- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6904143c40 msgr2=0x7f690415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6904143c40 0x7f690415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 -- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f690406d160 msgr2=0x7f69041e6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f690406d160 0x7f69041e6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:56.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.834+0000 7f68fa800640 1 -- 172.21.15.53:0/2040859178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec009a70 con 0x7f69040cc500 2024-01-16T15:02:56.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.835+0000 7f68fa800640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f690415a4f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f68ec037040 tx=0x7f68ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:56.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.835+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec040650 con 0x7f69040cc500 2024-01-16T15:02:56.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.835+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ec0407b0 con 0x7f69040cc500 2024-01-16T15:02:56.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.835+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ec040b60 con 0x7f69040cc500 2024-01-16T15:02:56.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.835+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69040a97b0 con 0x7f69040cc500 2024-01-16T15:02:56.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.835+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041e73f0 con 0x7f69040cc500 2024-01-16T15:02:56.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.836+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6904067450 con 0x7f69040cc500 2024-01-16T15:02:56.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.837+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f68ec040cc0 con 0x7f69040cc500 2024-01-16T15:02:56.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.838+0000 7f68fbe00640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68dc06b500 0x7f68dc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:56.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.838+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f68ec0acc30 con 0x7f69040cc500 2024-01-16T15:02:56.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.839+0000 7f6903400640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68dc06b500 0x7f68dc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.840+0000 7f6903400640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68dc06b500 0x7f68dc06d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f68f4001550 tx=0x7f68f4007a40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:56.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:56.843+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f68ec0779a0 con 0x7f69040cc500 2024-01-16T15:02:57.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.077+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69041e2840 con 0x7f69040cc500 2024-01-16T15:02:57.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.078+0000 7f68fbe00640 1 -- 172.21.15.53:0/2040859178 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f68ec044e90 con 0x7f69040cc500 2024-01-16T15:02:57.057 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:57.057 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":19,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5008,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/3324487785","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3324487785},{"type":"v1","addr":"172.21.15.146:6836","nonce":3324487785}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5017,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3124134221","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3124134221},{"type":"v1","addr":"172.21.15.146:6837","nonce":3124134221}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5184,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1994154705","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1994154705},{"type":"v1","addr":"172.21.15.53:6834","nonce":1994154705}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":5595,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/1853002600","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1853002600},{"type":"v1","addr":"172.21.15.53:6835","nonce":1853002600}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2024-01-16T15:02:57.057 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 19 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.080+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68dc06b500 msgr2=0x7f68dc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.080+0000 7f6909200640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68dc06b500 0x7f68dc06d9c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f68f4001550 tx=0x7f68f4007a40 comp rx=0 tx=0).stop 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.080+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 msgr2=0x7f690415a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.080+0000 7f6909200640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f690415a4f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f68ec037040 tx=0x7f68ec004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 shutdown_connections 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68dc06b500 0x7f68dc06d9c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6904143c40 0x7f690415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69040cc500 0x7f690415a4f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 --2- 172.21.15.53:0/2040859178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f690406d160 0x7f69041e6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 >> 172.21.15.53:0/2040859178 conn(0x7f690406a800 msgr2=0x7f690406d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 shutdown_connections 2024-01-16T15:02:57.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.081+0000 7f6909200640 1 -- 172.21.15.53:0/2040859178 wait complete. 2024-01-16T15:02:57.080 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:02:57.080 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:02:57.081 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:02:57.081 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:57.081 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:02:57.081 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:02:57.081 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:02:57.081 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:57.081 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:02:57.082 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:02:57.082 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:02:57.082 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:57.082 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:02:57.082 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:02:57.082 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:02:57.082 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:02:57.101 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:02:57.102 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:02:57.103 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:02:57.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:02:57.107 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:02:57.107 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:02:57.108 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:02:57.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:02:57.150 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:02:57.150 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:57.155 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:02:57.155 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:57.156 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:02:57.156 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:57.161 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:02:57.161 DEBUG:teuthology.parallel:result is None 2024-01-16T15:02:57.161 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:02:57.306 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:02:57.314 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:02:57.317 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:02:57.322 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:02:57.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.377+0000 7f183c000640 1 Processor -- start 2024-01-16T15:02:57.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.377+0000 7f183c000640 1 -- start start 2024-01-16T15:02:57.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.378+0000 7f183c000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1834160f80 0x7f1834161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.378+0000 7f183c000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18340ced50 0x7f18340d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.378+0000 7f183c000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18341618a0 con 0x7f18340ce8f0 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.378+0000 7f183c000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18340cf1e0 con 0x7f18340ced50 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.378+0000 7f183c000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18340d0de0 con 0x7f1834160f80 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f183a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18340ced50 0x7f18340d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f1838c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f18340ce8f0 0x7f18340cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44652/0 (socket says 172.21.15.53:44652) 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f1838c00640 1 -- 172.21.15.53:0/635079954 learned_addr learned my addr 172.21.15.53:0/635079954 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f1839600640 1 --2- 172.21.15.53:0/635079954 >> v2:172.21.15.146:3300/0 conn(0x7f1834160f80 0x7f1834161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f183a000640 1 -- 172.21.15.53:0/635079954 >> v2:172.21.15.146:3300/0 conn(0x7f1834160f80 msgr2=0x7f1834161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f183a000640 1 --2- 172.21.15.53:0/635079954 >> v2:172.21.15.146:3300/0 conn(0x7f1834160f80 0x7f1834161360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f183a000640 1 -- 172.21.15.53:0/635079954 >> v1:172.21.15.53:6789/0 conn(0x7f18340ce8f0 legacy=0x7f18340cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f1833e00640 1 -- 172.21.15.53:0/635079954 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4094454242 0 0) 0x7f18341618a0 con 0x7f18340ce8f0 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f183a000640 1 -- 172.21.15.53:0/635079954 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18340d10c0 con 0x7f18340ced50 2024-01-16T15:02:57.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.379+0000 7f1839600640 1 --2- 172.21.15.53:0/635079954 >> v2:172.21.15.146:3300/0 conn(0x7f1834160f80 0x7f1834161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.380+0000 7f183a000640 1 --2- 172.21.15.53:0/635079954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18340ced50 0x7f18340d02f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f183408d490 tx=0x7f182c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7cabe4c9fff9573 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f1833e00640 1 -- 172.21.15.53:0/635079954 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c02cc80 con 0x7f18340ced50 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f1833e00640 1 -- 172.21.15.53:0/635079954 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c02cde0 con 0x7f18340ced50 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f1833e00640 1 -- 172.21.15.53:0/635079954 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c034660 con 0x7f18340ced50 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 -- 172.21.15.53:0/635079954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18340ced50 msgr2=0x7f18340d02f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 --2- 172.21.15.53:0/635079954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18340ced50 0x7f18340d02f0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f183408d490 tx=0x7f182c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 -- 172.21.15.53:0/635079954 shutdown_connections 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 --2- 172.21.15.53:0/635079954 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18340ced50 0x7f18340d02f0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 --2- 172.21.15.53:0/635079954 >> v2:172.21.15.146:3300/0 conn(0x7f1834160f80 0x7f1834161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 -- 172.21.15.53:0/635079954 >> 172.21.15.53:0/635079954 conn(0x7f183409c550 msgr2=0x7f18340d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 -- 172.21.15.53:0/635079954 shutdown_connections 2024-01-16T15:02:57.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.381+0000 7f183c000640 1 -- 172.21.15.53:0/635079954 wait complete. 2024-01-16T15:02:57.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.383+0000 7f183c000640 1 Processor -- start 2024-01-16T15:02:57.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.383+0000 7f183c000640 1 -- start start 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18340ce8f0 0x7f18340b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18340ced50 0x7f18340c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 0x7f18340b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18340caf20 con 0x7f1834160f80 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18340b20b0 con 0x7f18340ce8f0 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18340ca2e0 con 0x7f18340ced50 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.384+0000 7f183a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 0x7f18340b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 0x7f18340b0e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45476/0 (socket says 172.21.15.53:45476) 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 -- 172.21.15.53:0/942547095 learned_addr learned my addr 172.21.15.53:0/942547095 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f1838c00640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18340ced50 0x7f18340c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f1839600640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18340ce8f0 0x7f18340b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 -- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18340ced50 msgr2=0x7f18340c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18340ced50 0x7f18340c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 -- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18340ce8f0 msgr2=0x7f18340b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18340ce8f0 0x7f18340b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 -- 172.21.15.53:0/942547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182c009a70 con 0x7f1834160f80 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f1839600640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18340ce8f0 0x7f18340b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.385+0000 7f183a000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 0x7f18340b0e90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f182c009e80 tx=0x7f182c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.386+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c03e610 con 0x7f1834160f80 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.386+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182c03e770 con 0x7f1834160f80 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.386+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182c03eb60 con 0x7f1834160f80 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.386+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18340ca580 con 0x7f1834160f80 2024-01-16T15:02:57.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.386+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18340cab40 con 0x7f1834160f80 2024-01-16T15:02:57.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.387+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1834160830 con 0x7f1834160f80 2024-01-16T15:02:57.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.388+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f182c03b050 con 0x7f1834160f80 2024-01-16T15:02:57.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.389+0000 7f1831600640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f180c06b500 0x7f180c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:57.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.389+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f182c0ac580 con 0x7f1834160f80 2024-01-16T15:02:57.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.389+0000 7f1839600640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f180c06b500 0x7f180c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:57.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.390+0000 7f1839600640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f180c06b500 0x7f180c06d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1824006fd0 tx=0x7f1824008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:57.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.393+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f182c077430 con 0x7f1834160f80 2024-01-16T15:02:57.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:57.598+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f18340675d0 con 0x7f1834160f80 2024-01-16T15:02:58.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.674+0000 7f1831600640 1 -- 172.21.15.53:0/942547095 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v43) v1 ==== 97+0+0 (secure 0 0 0) 0x7f182c009450 con 0x7f1834160f80 2024-01-16T15:02:58.655 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f180c06b500 msgr2=0x7f180c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f180c06b500 0x7f180c06d9c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1824006fd0 tx=0x7f1824008040 comp rx=0 tx=0).stop 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 msgr2=0x7f18340b0e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 0x7f18340b0e90 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f182c009e80 tx=0x7f182c004a40 comp rx=0 tx=0).stop 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 shutdown_connections 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f180c06b500 0x7f180c06d9c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1834160f80 0x7f18340b0e90 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18340ced50 0x7f18340c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 --2- 172.21.15.53:0/942547095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18340ce8f0 0x7f18340b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.678+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 >> 172.21.15.53:0/942547095 conn(0x7f183409c550 msgr2=0x7f18340cdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:58.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.679+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 shutdown_connections 2024-01-16T15:02:58.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.679+0000 7f183c000640 1 -- 172.21.15.53:0/942547095 wait complete. 2024-01-16T15:02:58.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:02:58.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.918+0000 7f13dbe00640 1 Processor -- start 2024-01-16T15:02:58.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.919+0000 7f13dbe00640 1 -- start start 2024-01-16T15:02:58.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13dbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13d40cd240 0x7f13d40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 0x7f13d40cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13dbe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13d406cea0 con 0x7f13d406c6b0 2024-01-16T15:02:58.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13dbe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13d406cbc0 con 0x7f13d40cd240 2024-01-16T15:02:58.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13dbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d406f350 con 0x7f13d409a170 2024-01-16T15:02:58.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13d8a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f13d406c6b0 0x7f13d4099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44668/0 (socket says 172.21.15.53:44668) 2024-01-16T15:02:58.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.920+0000 7f13d8a00640 1 -- 172.21.15.53:0/3010441848 learned_addr learned my addr 172.21.15.53:0/3010441848 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:58.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d9400640 1 --2- 172.21.15.53:0/3010441848 >> v2:172.21.15.146:3300/0 conn(0x7f13d40cd240 0x7f13d40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d9e00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 0x7f13d40cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3643751033 0 0) 0x7f13d406cea0 con 0x7f13d406c6b0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b4003520 con 0x7f13d406c6b0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 674271684 0 0) 0x7f13b4003520 con 0x7f13d406c6b0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 msgr2=0x7f13d40cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 0x7f13d40cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 >> v2:172.21.15.146:3300/0 conn(0x7f13d40cd240 msgr2=0x7f13d40b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 --2- 172.21.15.53:0/3010441848 >> v2:172.21.15.146:3300/0 conn(0x7f13d40cd240 0x7f13d40b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d406f630 con 0x7f13d406c6b0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f13bc001670 con 0x7f13d406c6b0 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 >> v1:172.21.15.53:6789/0 conn(0x7f13d406c6b0 legacy=0x7f13d4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13b4003970 0x7f13b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 0x7f13b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4004d00 0x7f13b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d406cbc0 con 0x7f13b4003970 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d406f350 con 0x7f13b4004d00 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13b400ba30 con 0x7f13d409a170 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.921+0000 7f13d9400640 1 --2- 172.21.15.53:0/3010441848 >> v2:172.21.15.146:3300/0 conn(0x7f13d40cd240 0x7f13d40b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9400640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13b4003970 0x7f13b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d8a00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 0x7f13b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9e00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4004d00 0x7f13b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9400640 1 -- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 msgr2=0x7f13b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9400640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d409a170 0x7f13b40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9400640 1 -- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4004d00 msgr2=0x7f13b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9400640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4004d00 0x7f13b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.922+0000 7f13d9400640 1 -- 172.21.15.53:0/3010441848 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4003520 con 0x7f13b4003970 2024-01-16T15:02:58.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.923+0000 7f13d9400640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13b4003970 0x7f13b4003d90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f13c40098a0 tx=0x7f13b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9192dec13c73c94d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:58.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4037600 con 0x7f13b4003970 2024-01-16T15:02:58.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c4037760 con 0x7f13b4003970 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13d3e00640 1 -- 172.21.15.53:0/3010441848 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c4037ad0 con 0x7f13b4003970 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13dbe00640 1 -- 172.21.15.53:0/3010441848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13b4003970 msgr2=0x7f13b4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3010441848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13b4003970 0x7f13b4003d90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f13c40098a0 tx=0x7f13b40057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13d9e00640 1 -- 172.21.15.53:0/3010441848 reap_dead start 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13dbe00640 1 -- 172.21.15.53:0/3010441848 shutdown_connections 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13dbe00640 1 -- 172.21.15.53:0/3010441848 >> 172.21.15.53:0/3010441848 conn(0x7f13d40b2d00 msgr2=0x7f13d40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.924+0000 7f13dbe00640 1 -- 172.21.15.53:0/3010441848 shutdown_connections 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.925+0000 7f13dbe00640 1 -- 172.21.15.53:0/3010441848 wait complete. 2024-01-16T15:02:58.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.926+0000 7f13dbe00640 1 Processor -- start 2024-01-16T15:02:58.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.926+0000 7f13dbe00640 1 -- start start 2024-01-16T15:02:58.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.927+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4003970 0x7f13d40c8710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.927+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d414fbb0 0x7f13d414f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.927+0000 7f13dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 0x7f13d4151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.927+0000 7f13dbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40cb500 con 0x7f13d40ca9a0 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.927+0000 7f13dbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d414f910 con 0x7f13b4003970 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.927+0000 7f13dbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13d40b5250 con 0x7f13d414fbb0 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 0x7f13d4151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 0x7f13d4151b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45500/0 (socket says 172.21.15.53:45500) 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 -- 172.21.15.53:0/3374671564 learned_addr learned my addr 172.21.15.53:0/3374671564 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9400640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4003970 0x7f13d40c8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d8a00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d414fbb0 0x7f13d414f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 -- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d414fbb0 msgr2=0x7f13d414f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d414fbb0 0x7f13d414f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 -- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4003970 msgr2=0x7f13d40c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4003970 0x7f13d40c8710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 -- 172.21.15.53:0/3374671564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d40cb220 con 0x7f13d40ca9a0 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9400640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4003970 0x7f13d40c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:58.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d8a00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d414fbb0 0x7f13d414f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:58.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.928+0000 7f13d9e00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 0x7f13d4151b70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f13d40cb810 tx=0x7f13cc0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:58.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.929+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc00e070 con 0x7f13d40ca9a0 2024-01-16T15:02:58.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.929+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13cc011de0 con 0x7f13d40ca9a0 2024-01-16T15:02:58.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.929+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d40b54f0 con 0x7f13d40ca9a0 2024-01-16T15:02:58.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.929+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13cc003990 con 0x7f13d40ca9a0 2024-01-16T15:02:58.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.929+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d40cae50 con 0x7f13d40ca9a0 2024-01-16T15:02:58.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.930+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d40b50b0 con 0x7f13d40ca9a0 2024-01-16T15:02:58.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.931+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f13cc003af0 con 0x7f13d40ca9a0 2024-01-16T15:02:58.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.932+0000 7f13d1600640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13ac06b500 0x7f13ac06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:58.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.932+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f13cc08a400 con 0x7f13d40ca9a0 2024-01-16T15:02:58.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.932+0000 7f13d9400640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13ac06b500 0x7f13ac06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:58.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.934+0000 7f13d9400640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13ac06b500 0x7f13ac06d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c4031000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:58.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:58.937+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13d40b50b0 con 0x7f13d40ca9a0 2024-01-16T15:02:59.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.199+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f13d40b5a10 con 0x7f13d40ca9a0 2024-01-16T15:02:59.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.200+0000 7f13d1600640 1 -- 172.21.15.53:0/3374671564 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v9) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f13d40b5a10 con 0x7f13d40ca9a0 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.180 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:02:59.181 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.182 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.183 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:02:59.184 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.185 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow r", 2024-01-16T15:02:59.187 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:02:59.188 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:02:59.189 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:02:59.190 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:02:59.191 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:02:59.192 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:02:59.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.203+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13ac06b500 msgr2=0x7f13ac06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.203+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13ac06b500 0x7f13ac06d9c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f13c4036000 tx=0x7f13c4031000 comp rx=0 tx=0).stop 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.203+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 msgr2=0x7f13d4151b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.203+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 0x7f13d4151b70 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f13d40cb810 tx=0x7f13cc0115a0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 shutdown_connections 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13ac06b500 0x7f13ac06d9c0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13d40ca9a0 0x7f13d4151b70 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13d414fbb0 0x7f13d414f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 --2- 172.21.15.53:0/3374671564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13b4003970 0x7f13d40c8710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 >> 172.21.15.53:0/3374671564 conn(0x7f13d40b2d00 msgr2=0x7f13d40d0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 shutdown_connections 2024-01-16T15:02:59.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.204+0000 7f13dbe00640 1 -- 172.21.15.53:0/3374671564 wait complete. 2024-01-16T15:02:59.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:02:59.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.443+0000 7f9ff9200640 1 Processor -- start 2024-01-16T15:02:59.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.444+0000 7f9ff9200640 1 -- start start 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff9200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9ff40cc500 0x7f9ff40cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 0x7f9ff406d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff4070e50 con 0x7f9ff4143c40 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff9200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff40723c0 con 0x7f9ff40cc500 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40cced0 con 0x7f9ff406d160 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9ff4143c40 0x7f9ff4071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44670/0 (socket says 172.21.15.53:44670) 2024-01-16T15:02:59.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.445+0000 7f9ff3400640 1 -- 172.21.15.53:0/1651897983 learned_addr learned my addr 172.21.15.53:0/1651897983 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:02:59.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff3e00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 0x7f9ff406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582676851 0 0) 0x7f9ff4070e50 con 0x7f9ff4143c40 2024-01-16T15:02:59.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd4003520 con 0x7f9ff4143c40 2024-01-16T15:02:59.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9febe00640 1 --2- 172.21.15.53:0/1651897983 >> v2:172.21.15.146:3300/0 conn(0x7f9ff40cc500 0x7f9ff40cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 894385676 0 0) 0x7f9fd4003520 con 0x7f9ff4143c40 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 msgr2=0x7f9ff406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 0x7f9ff406d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 >> v2:172.21.15.146:3300/0 conn(0x7f9ff40cc500 msgr2=0x7f9ff40cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 --2- 172.21.15.53:0/1651897983 >> v2:172.21.15.146:3300/0 conn(0x7f9ff40cc500 0x7f9ff40cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4144020 con 0x7f9ff4143c40 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9fe4001690 con 0x7f9ff4143c40 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 >> v1:172.21.15.53:6789/0 conn(0x7f9ff4143c40 legacy=0x7f9ff4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fd4003970 0x7f9fd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fd40047c0 0x7f9fd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 0x7f9fd4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40723c0 con 0x7f9fd4003970 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40cced0 con 0x7f9fd40047c0 2024-01-16T15:02:59.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fd400ba00 con 0x7f9ff406d160 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.446+0000 7f9ff3400640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fd4003970 0x7f9fd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9febe00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fd40047c0 0x7f9fd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9ff3400640 1 -- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 msgr2=0x7f9fd4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9ff3400640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9ff406d160 0x7f9fd4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9ff3400640 1 -- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fd40047c0 msgr2=0x7f9fd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9ff3400640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fd40047c0 0x7f9fd4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9ff3400640 1 -- 172.21.15.53:0/1651897983 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fd4003520 con 0x7f9fd4003970 2024-01-16T15:02:59.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.447+0000 7f9febe00640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fd40047c0 0x7f9fd4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:59.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.448+0000 7f9ff3400640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fd4003970 0x7f9fd4003d90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400c160 tx=0x7f9fd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cf5cc2e268fdb3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:59.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.448+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4005ad0 con 0x7f9fd4003970 2024-01-16T15:02:59.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.448+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fe4005c30 con 0x7f9fd4003970 2024-01-16T15:02:59.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.448+0000 7f9ff2a00640 1 -- 172.21.15.53:0/1651897983 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe402c810 con 0x7f9fd4003970 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff9200640 1 -- 172.21.15.53:0/1651897983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fd4003970 msgr2=0x7f9fd4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff9200640 1 --2- 172.21.15.53:0/1651897983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fd4003970 0x7f9fd4003d90 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400c160 tx=0x7f9fd40057f0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff3e00640 1 -- 172.21.15.53:0/1651897983 reap_dead start 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff9200640 1 -- 172.21.15.53:0/1651897983 shutdown_connections 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff9200640 1 -- 172.21.15.53:0/1651897983 >> 172.21.15.53:0/1651897983 conn(0x7f9ff406a800 msgr2=0x7f9ff406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff9200640 1 -- 172.21.15.53:0/1651897983 shutdown_connections 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.449+0000 7f9ff9200640 1 -- 172.21.15.53:0/1651897983 wait complete. 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.451+0000 7f9ff9200640 1 Processor -- start 2024-01-16T15:02:59.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.451+0000 7f9ff9200640 1 -- start start 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fd4003970 0x7f9ff40b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff414d080 0x7f9ff414d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 0x7f9ff4141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff41421a0 con 0x7f9ff41413d0 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4142480 con 0x7f9ff414d080 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4142760 con 0x7f9fd4003970 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.452+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 0x7f9ff4141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 0x7f9ff4141850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45510/0 (socket says 172.21.15.53:45510) 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 -- 172.21.15.53:0/2401376239 learned_addr learned my addr 172.21.15.53:0/2401376239 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3400640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fd4003970 0x7f9ff40b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9febe00640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff414d080 0x7f9ff414d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 -- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fd4003970 msgr2=0x7f9ff40b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fd4003970 0x7f9ff40b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 -- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff414d080 msgr2=0x7f9ff414d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff414d080 0x7f9ff414d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 -- 172.21.15.53:0/2401376239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff4142a40 con 0x7f9ff41413d0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9febe00640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff414d080 0x7f9ff414d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9ff3e00640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 0x7f9ff4141850 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4144ca0 tx=0x7f9fec00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.453+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec010040 con 0x7f9ff41413d0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.454+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec00bae0 con 0x7f9ff41413d0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.454+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00be70 con 0x7f9ff41413d0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.454+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff4142d20 con 0x7f9ff41413d0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.454+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff4141d90 con 0x7f9ff41413d0 2024-01-16T15:02:59.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.455+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff4142f10 con 0x7f9ff41413d0 2024-01-16T15:02:59.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.456+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f9fec01f460 con 0x7f9ff41413d0 2024-01-16T15:02:59.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.457+0000 7f9feb400640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.457+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f9fec08a2e0 con 0x7f9ff41413d0 2024-01-16T15:02:59.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.457+0000 7f9ff3400640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.458+0000 7f9ff3400640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400c160 tx=0x7f9fe4037040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:59.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.461+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fec054fa0 con 0x7f9ff41413d0 2024-01-16T15:02:59.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.655+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ff4067450 con 0x7f9ff41413d0 2024-01-16T15:02:59.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.656+0000 7f9feb400640 1 -- 172.21.15.53:0/2401376239 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v43) v1 ==== 74+0+12821 (secure 0 0 0) 0x7f9fec0585c0 con 0x7f9ff41413d0 2024-01-16T15:02:59.636 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:02:59.636 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":43,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:02:58.670632+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:02:59.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fcc06b500 msgr2=0x7f9fcc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9fe400c160 tx=0x7f9fe4037040 comp rx=0 tx=0).stop 2024-01-16T15:02:59.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 msgr2=0x7f9ff4141850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 0x7f9ff4141850 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f9ff4144ca0 tx=0x7f9fec00b480 comp rx=0 tx=0).stop 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 shutdown_connections 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fcc06b500 0x7f9fcc06d9c0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9ff41413d0 0x7f9ff4141850 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9ff414d080 0x7f9ff414d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 --2- 172.21.15.53:0/2401376239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fd4003970 0x7f9ff40b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 >> 172.21.15.53:0/2401376239 conn(0x7f9ff406a800 msgr2=0x7f9ff4071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 shutdown_connections 2024-01-16T15:02:59.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.659+0000 7f9ff9200640 1 -- 172.21.15.53:0/2401376239 wait complete. 2024-01-16T15:02:59.659 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:02:59.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:02:59.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.904+0000 7fd239c00640 1 Processor -- start 2024-01-16T15:02:59.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.904+0000 7fd239c00640 1 -- start start 2024-01-16T15:02:59.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.906+0000 7fd239c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2340adf20 0x7fd234070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.906+0000 7fd239c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2340ccb50 0x7fd234143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.906+0000 7fd239c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd23415f4e0 con 0x7fd23415bf10 2024-01-16T15:02:59.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.906+0000 7fd239c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd23415f970 con 0x7fd2340adf20 2024-01-16T15:02:59.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.906+0000 7fd239c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd234071130 con 0x7fd2340ccb50 2024-01-16T15:02:59.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.906+0000 7fd233e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2340adf20 0x7fd234070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd239200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2340ccb50 0x7fd234143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd233e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd2340adf20 0x7fd234070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:59594/0 (socket says 172.21.15.53:59594) 2024-01-16T15:02:59.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd233e00640 1 -- 172.21.15.53:0/4165572872 learned_addr learned my addr 172.21.15.53:0/4165572872 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:59.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd233e00640 1 -- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2340ccb50 msgr2=0x7fd234143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:02:59.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd233e00640 1 --2- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2340ccb50 0x7fd234143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd233e00640 1 -- 172.21.15.53:0/4165572872 >> v1:172.21.15.146:6789/0 conn(0x7fd23415bf10 legacy=0x7fd2340cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.907+0000 7fd233e00640 1 -- 172.21.15.53:0/4165572872 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd23415c390 con 0x7fd2340adf20 2024-01-16T15:02:59.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.909+0000 7fd233e00640 1 --2- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd234070a30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd224009870 tx=0x7fd22402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=807b257b27c915e3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:59.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.909+0000 7fd232a00640 1 -- 172.21.15.53:0/4165572872 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224031640 con 0x7fd2340adf20 2024-01-16T15:02:59.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.909+0000 7fd232a00640 1 -- 172.21.15.53:0/4165572872 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2240317a0 con 0x7fd2340adf20 2024-01-16T15:02:59.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.909+0000 7fd232a00640 1 -- 172.21.15.53:0/4165572872 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd224031b30 con 0x7fd2340adf20 2024-01-16T15:02:59.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 -- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 msgr2=0x7fd234070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 --2- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd234070a30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fd224009870 tx=0x7fd22402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 -- 172.21.15.53:0/4165572872 shutdown_connections 2024-01-16T15:02:59.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 --2- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd2340ccb50 0x7fd234143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 --2- 172.21.15.53:0/4165572872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd234070a30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 -- 172.21.15.53:0/4165572872 >> 172.21.15.53:0/4165572872 conn(0x7fd23406a7d0 msgr2=0x7fd23406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:02:59.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 -- 172.21.15.53:0/4165572872 shutdown_connections 2024-01-16T15:02:59.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.910+0000 7fd239c00640 1 -- 172.21.15.53:0/4165572872 wait complete. 2024-01-16T15:02:59.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.912+0000 7fd239c00640 1 Processor -- start 2024-01-16T15:02:59.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.912+0000 7fd239c00640 1 -- start start 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd239c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd2341424f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd239c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 0x7fd23415cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd239c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd23415bf10 0x7fd23415d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd239c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23413c570 con 0x7fd2340ccb50 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd239c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340b1f10 con 0x7fd2340adf20 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd239c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340b21f0 con 0x7fd23415bf10 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd233400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 0x7fd23415cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd233400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 0x7fd23415cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45516/0 (socket says 172.21.15.53:45516) 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.913+0000 7fd233400640 1 -- 172.21.15.53:0/2162915561 learned_addr learned my addr 172.21.15.53:0/2162915561 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233e00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd2341424f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd239200640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd23415bf10 0x7fd23415d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233400640 1 -- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd23415bf10 msgr2=0x7fd23415d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233400640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd23415bf10 0x7fd23415d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233400640 1 -- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 msgr2=0x7fd2341424f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233400640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd2341424f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233400640 1 -- 172.21.15.53:0/2162915561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2240094d0 con 0x7fd2340ccb50 2024-01-16T15:02:59.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd233400640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 0x7fd23415cc10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00c3c0 tx=0x7fd21c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:59.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c0052d0 con 0x7fd2340ccb50 2024-01-16T15:02:59.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd21c005450 con 0x7fd2340ccb50 2024-01-16T15:02:59.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd21c007260 con 0x7fd2340ccb50 2024-01-16T15:02:59.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.914+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2340b24d0 con 0x7fd2340ccb50 2024-01-16T15:02:59.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.915+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23415d780 con 0x7fd2340ccb50 2024-01-16T15:02:59.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.916+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd23415aef0 con 0x7fd2340ccb50 2024-01-16T15:02:59.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.917+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd21c0073c0 con 0x7fd2340ccb50 2024-01-16T15:02:59.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.919+0000 7fd22be00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd20c06b500 0x7fd20c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:02:59.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.919+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fd21c088ee0 con 0x7fd2340ccb50 2024-01-16T15:02:59.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.919+0000 7fd233e00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd20c06b500 0x7fd20c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:02:59.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.921+0000 7fd233e00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd20c06b500 0x7fd20c06d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd224036000 tx=0x7fd22403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:02:59.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:02:59.922+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd21c053ea0 con 0x7fd2340ccb50 2024-01-16T15:03:00.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.116+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd234067450 con 0x7fd2340ccb50 2024-01-16T15:03:00.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.117+0000 7fd22be00640 1 -- 172.21.15.53:0/2162915561 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v43) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd21c0574c0 con 0x7fd2340ccb50 2024-01-16T15:03:00.096 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd20c06b500 msgr2=0x7fd20c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd20c06b500 0x7fd20c06d9c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fd224036000 tx=0x7fd22403c040 comp rx=0 tx=0).stop 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 msgr2=0x7fd23415cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 0x7fd23415cc10 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd21c00c3c0 tx=0x7fd21c00c880 comp rx=0 tx=0).stop 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 shutdown_connections 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd20c06b500 0x7fd20c06d9c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd23415bf10 0x7fd23415d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd2340ccb50 0x7fd23415cc10 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 --2- 172.21.15.53:0/2162915561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd2340adf20 0x7fd2341424f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.120+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 >> 172.21.15.53:0/2162915561 conn(0x7fd23406a7d0 msgr2=0x7fd2340cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:00.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.121+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 shutdown_connections 2024-01-16T15:03:00.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.121+0000 7fd239c00640 1 -- 172.21.15.53:0/2162915561 wait complete. 2024-01-16T15:03:00.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:03:00.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.349+0000 7fdd31a00640 1 Processor -- start 2024-01-16T15:03:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.350+0000 7fdd31a00640 1 -- start start 2024-01-16T15:03:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.351+0000 7fdd31a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c160920 0x7fdd2c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.351+0000 7fdd31a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdd2c0cfe70 0x7fdd2c161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.351+0000 7fdd31a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2c0d0a10 con 0x7fdd2c0cf9f0 2024-01-16T15:03:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.351+0000 7fdd31a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd2c0d1f50 con 0x7fdd2c0cfe70 2024-01-16T15:03:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.351+0000 7fdd31a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0cf080 con 0x7fdd2c160920 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.352+0000 7fdd23200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdd2c0cf9f0 0x7fdd2c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44686/0 (socket says 172.21.15.53:44686) 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.352+0000 7fdd23200640 1 -- 172.21.15.53:0/3067265566 learned_addr learned my addr 172.21.15.53:0/3067265566 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.352+0000 7fdd2be00640 1 --2- 172.21.15.53:0/3067265566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c160920 0x7fdd2c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.352+0000 7fdd31000640 1 --2- 172.21.15.53:0/3067265566 >> v2:172.21.15.146:3300/0 conn(0x7fdd2c0cfe70 0x7fdd2c161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.352+0000 7fdd2b400640 1 -- 172.21.15.53:0/3067265566 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572779466 0 0) 0x7fdd2c0d0a10 con 0x7fdd2c0cf9f0 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.352+0000 7fdd2b400640 1 -- 172.21.15.53:0/3067265566 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c003520 con 0x7fdd2c0cf9f0 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.353+0000 7fdd2b400640 1 -- 172.21.15.53:0/3067265566 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1398496450 0 0) 0x7fdd0c003520 con 0x7fdd2c0cf9f0 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.353+0000 7fdd2be00640 1 -- 172.21.15.53:0/3067265566 >> v2:172.21.15.146:3300/0 conn(0x7fdd2c0cfe70 msgr2=0x7fdd2c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.353+0000 7fdd2be00640 1 --2- 172.21.15.53:0/3067265566 >> v2:172.21.15.146:3300/0 conn(0x7fdd2c0cfe70 0x7fdd2c161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.353+0000 7fdd2be00640 1 -- 172.21.15.53:0/3067265566 >> v1:172.21.15.53:6789/0 conn(0x7fdd2c0cf9f0 legacy=0x7fdd2c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.353+0000 7fdd2be00640 1 -- 172.21.15.53:0/3067265566 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c0c6be0 con 0x7fdd2c160920 2024-01-16T15:03:00.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.353+0000 7fdd31000640 1 --2- 172.21.15.53:0/3067265566 >> v2:172.21.15.146:3300/0 conn(0x7fdd2c0cfe70 0x7fdd2c161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:03:00.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.354+0000 7fdd2be00640 1 --2- 172.21.15.53:0/3067265566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c160920 0x7fdd2c0b3f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0097a0 tx=0x7fdd1c02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74c3666c3de6e0e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.355+0000 7fdd2b400640 1 -- 172.21.15.53:0/3067265566 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd14001670 con 0x7fdd2c160920 2024-01-16T15:03:00.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.355+0000 7fdd2b400640 1 -- 172.21.15.53:0/3067265566 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1c02e050 con 0x7fdd2c160920 2024-01-16T15:03:00.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.355+0000 7fdd2b400640 1 -- 172.21.15.53:0/3067265566 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c030730 con 0x7fdd2c160920 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.355+0000 7fdd31a00640 1 -- 172.21.15.53:0/3067265566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c160920 msgr2=0x7fdd2c0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.355+0000 7fdd31a00640 1 --2- 172.21.15.53:0/3067265566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c160920 0x7fdd2c0b3f30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c0097a0 tx=0x7fdd1c02b790 comp rx=0 tx=0).stop 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.356+0000 7fdd31a00640 1 -- 172.21.15.53:0/3067265566 shutdown_connections 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.356+0000 7fdd31a00640 1 --2- 172.21.15.53:0/3067265566 >> v2:172.21.15.146:3300/0 conn(0x7fdd2c0cfe70 0x7fdd2c161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.356+0000 7fdd31a00640 1 --2- 172.21.15.53:0/3067265566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c160920 0x7fdd2c0b3f30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.356+0000 7fdd31a00640 1 -- 172.21.15.53:0/3067265566 >> 172.21.15.53:0/3067265566 conn(0x7fdd2c0d3090 msgr2=0x7fdd2c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.356+0000 7fdd31a00640 1 -- 172.21.15.53:0/3067265566 shutdown_connections 2024-01-16T15:03:00.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.356+0000 7fdd31a00640 1 -- 172.21.15.53:0/3067265566 wait complete. 2024-01-16T15:03:00.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.358+0000 7fdd31a00640 1 Processor -- start 2024-01-16T15:03:00.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.358+0000 7fdd31a00640 1 -- start start 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c0cf9f0 0x7fdd2c151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd2c0cfe70 0x7fdd2c161800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 0x7fdd2c0cc5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c151b50 con 0x7fdd2c160920 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c15f520 con 0x7fdd2c0cfe70 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c0cb550 con 0x7fdd2c0cf9f0 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 0x7fdd2c0cc5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 0x7fdd2c0cc5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45520/0 (socket says 172.21.15.53:45520) 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.360+0000 7fdd31000640 1 -- 172.21.15.53:0/1597621126 learned_addr learned my addr 172.21.15.53:0/1597621126 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd2be00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c0cf9f0 0x7fdd2c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd23200640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd2c0cfe70 0x7fdd2c161800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd31000640 1 -- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c0cf9f0 msgr2=0x7fdd2c151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd31000640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c0cf9f0 0x7fdd2c151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd31000640 1 -- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd2c0cfe70 msgr2=0x7fdd2c161800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd31000640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd2c0cfe70 0x7fdd2c161800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd31000640 1 -- 172.21.15.53:0/1597621126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd24009a70 con 0x7fdd2c160920 2024-01-16T15:03:00.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd31000640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 0x7fdd2c0cc5f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400efc0 tx=0x7fdd24009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.361+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24009d70 con 0x7fdd2c160920 2024-01-16T15:03:00.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.362+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd2400f040 con 0x7fdd2c160920 2024-01-16T15:03:00.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.362+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd24003990 con 0x7fdd2c160920 2024-01-16T15:03:00.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.362+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1c009400 con 0x7fdd2c160920 2024-01-16T15:03:00.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.362+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c0cbb10 con 0x7fdd2c160920 2024-01-16T15:03:00.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.363+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c067450 con 0x7fdd2c160920 2024-01-16T15:03:00.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.364+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fdd240144a0 con 0x7fdd2c160920 2024-01-16T15:03:00.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.365+0000 7fdd28c00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd0406b530 0x7fdd0406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.365+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdd2408aa00 con 0x7fdd2c160920 2024-01-16T15:03:00.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.365+0000 7fdd2be00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd0406b530 0x7fdd0406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.367+0000 7fdd2be00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd0406b530 0x7fdd0406d9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c034000 tx=0x7fdd1c02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.371+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd24055740 con 0x7fdd2c160920 2024-01-16T15:03:00.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.570+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdd2c08c7e0 con 0x7fdd2c160920 2024-01-16T15:03:00.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.571+0000 7fdd28c00640 1 -- 172.21.15.53:0/1597621126 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v43) v1 ==== 84+0+65 (secure 0 0 0) 0x7fdd24018090 con 0x7fdd2c160920 2024-01-16T15:03:00.550 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:00.550 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:03:00.550 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:03:00.550 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:03:00.550 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:03:00.551 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:03:00.551 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:03:00.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd0406b530 msgr2=0x7fdd0406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd0406b530 0x7fdd0406d9f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c034000 tx=0x7fdd1c02f000 comp rx=0 tx=0).stop 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 msgr2=0x7fdd2c0cc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 0x7fdd2c0cc5f0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fdd2400efc0 tx=0x7fdd24009f90 comp rx=0 tx=0).stop 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 shutdown_connections 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd0406b530 0x7fdd0406d9f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdd2c160920 0x7fdd2c0cc5f0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdd2c0cfe70 0x7fdd2c161800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 --2- 172.21.15.53:0/1597621126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdd2c0cf9f0 0x7fdd2c151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 >> 172.21.15.53:0/1597621126 conn(0x7fdd2c0d3090 msgr2=0x7fdd2c15b040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 shutdown_connections 2024-01-16T15:03:00.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.574+0000 7fdd31a00640 1 -- 172.21.15.53:0/1597621126 wait complete. 2024-01-16T15:03:00.573 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:03:00.573 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:03:00.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.810+0000 7f3d54c00640 1 Processor -- start 2024-01-16T15:03:00.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.810+0000 7f3d54c00640 1 -- start start 2024-01-16T15:03:00.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.812+0000 7f3d54c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 0x7f3d50071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.812+0000 7f3d54c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d50070e50 con 0x7f3d500717d0 2024-01-16T15:03:00.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.812+0000 7f3d54c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d50144020 con 0x7f3d500ccc10 2024-01-16T15:03:00.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.812+0000 7f3d54c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500723c0 con 0x7f3d50143c40 2024-01-16T15:03:00.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.812+0000 7f3d47e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3d500717d0 0x7f3d500cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44694/0 (socket says 172.21.15.53:44694) 2024-01-16T15:03:00.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.812+0000 7f3d47e00640 1 -- 172.21.15.53:0/3473415473 learned_addr learned my addr 172.21.15.53:0/3473415473 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:00.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.813+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 0x7f3d50071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.813+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180500439 0 0) 0x7f3d50070e50 con 0x7f3d500717d0 2024-01-16T15:03:00.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.813+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d30003520 con 0x7f3d500717d0 2024-01-16T15:03:00.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444508198 0 0) 0x7f3d50144020 con 0x7f3d500ccc10 2024-01-16T15:03:00.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d50070e50 con 0x7f3d500ccc10 2024-01-16T15:03:00.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3509781236 0 0) 0x7f3d30003520 con 0x7f3d500717d0 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 msgr2=0x7f3d50071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 0x7f3d50071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 >> v1:172.21.15.146:6789/0 conn(0x7f3d500ccc10 legacy=0x7f3d5006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d50071e00 con 0x7f3d500717d0 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3d38001670 con 0x7f3d500717d0 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 >> v1:172.21.15.53:6789/0 conn(0x7f3d500717d0 legacy=0x7f3d500cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d30003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300047c0 0x7f3d30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 0x7f3d300051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500723c0 con 0x7f3d30003b30 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3000ba90 con 0x7f3d300047c0 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3000bc50 con 0x7f3d50143c40 2024-01-16T15:03:00.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d30003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d4fe00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 0x7f3d300051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.814+0000 7f3d47e00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300047c0 0x7f3d30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.815+0000 7f3d4f400640 1 -- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 msgr2=0x7f3d300051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.815+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d50143c40 0x7f3d300051a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.815+0000 7f3d4f400640 1 -- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300047c0 msgr2=0x7f3d30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.815+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d300047c0 0x7f3d30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.815+0000 7f3d4f400640 1 -- 172.21.15.53:0/3473415473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d30003520 con 0x7f3d30003b30 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.816+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d30003f30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009ac0 tx=0x7f3d300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a46cf9a14b5962f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.816+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4002c6b0 con 0x7f3d30003b30 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.816+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d4002c810 con 0x7f3d30003b30 2024-01-16T15:03:00.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.816+0000 7f3d4ea00640 1 -- 172.21.15.53:0/3473415473 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d4002cba0 con 0x7f3d30003b30 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d54c00640 1 -- 172.21.15.53:0/3473415473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 msgr2=0x7f3d30003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3473415473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d30003f30 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009ac0 tx=0x7f3d300057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d4fe00640 1 -- 172.21.15.53:0/3473415473 reap_dead start 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d54c00640 1 -- 172.21.15.53:0/3473415473 shutdown_connections 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d54c00640 1 -- 172.21.15.53:0/3473415473 >> 172.21.15.53:0/3473415473 conn(0x7f3d5006a800 msgr2=0x7f3d5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d54c00640 1 -- 172.21.15.53:0/3473415473 shutdown_connections 2024-01-16T15:03:00.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.817+0000 7f3d54c00640 1 -- 172.21.15.53:0/3473415473 wait complete. 2024-01-16T15:03:00.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.819+0000 7f3d54c00640 1 Processor -- start 2024-01-16T15:03:00.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.819+0000 7f3d54c00640 1 -- start start 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d54c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d500b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d54c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d500c3d70 0x7f3d500c41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d54c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5013c200 0x7f3d5013c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d54c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d500c46f0 con 0x7f3d30003b30 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d54c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5015ab40 con 0x7f3d5013c200 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d54c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5013cbc0 con 0x7f3d500c3d70 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d4f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d500b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d4f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d500b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45542/0 (socket says 172.21.15.53:45542) 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.820+0000 7f3d4f400640 1 -- 172.21.15.53:0/3331671600 learned_addr learned my addr 172.21.15.53:0/3331671600 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d47e00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d500c3d70 0x7f3d500c41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4fe00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5013c200 0x7f3d5013c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4f400640 1 -- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d500c3d70 msgr2=0x7f3d500c41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d500c3d70 0x7f3d500c41b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4f400640 1 -- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5013c200 msgr2=0x7f3d5013c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5013c200 0x7f3d5013c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4f400640 1 -- 172.21.15.53:0/3331671600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d500c48f0 con 0x7f3d30003b30 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d4f400640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d500b30f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3d4002c250 tx=0x7f3d40004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.821+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40003810 con 0x7f3d30003b30 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.822+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d40003970 con 0x7f3d30003b30 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.822+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d500c4bd0 con 0x7f3d30003b30 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.822+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40003ce0 con 0x7f3d30003b30 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.822+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d500c5190 con 0x7f3d30003b30 2024-01-16T15:03:00.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.823+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d500cd970 con 0x7f3d30003b30 2024-01-16T15:03:00.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.824+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f3d40047450 con 0x7f3d30003b30 2024-01-16T15:03:00.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.825+0000 7f3d47400640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d2806b500 0x7f3d2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:00.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.825+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f3d400abfe0 con 0x7f3d30003b30 2024-01-16T15:03:00.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.825+0000 7f3d47e00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d2806b500 0x7f3d2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:00.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.827+0000 7f3d47e00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d2806b500 0x7f3d2806d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d38001200 tx=0x7f3d38000ef0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:00.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:00.830+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3d40076e70 con 0x7f3d30003b30 2024-01-16T15:03:01.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.032+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f3d50067450 con 0x7f3d30003b30 2024-01-16T15:03:01.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.701+0000 7f3d47400640 1 -- 172.21.15.53:0/3331671600 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v44) v1 ==== 138+0+0 (secure 0 0 0) 0x7f3d400044e0 con 0x7f3d30003b30 2024-01-16T15:03:01.681 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d2806b500 msgr2=0x7f3d2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d2806b500 0x7f3d2806d9c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3d38001200 tx=0x7f3d38000ef0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 msgr2=0x7f3d500b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d500b30f0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f3d4002c250 tx=0x7f3d40004730 comp rx=0 tx=0).stop 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 shutdown_connections 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3d2806b500 0x7f3d2806d9c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3d5013c200 0x7f3d5013c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3d500c3d70 0x7f3d500c41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 --2- 172.21.15.53:0/3331671600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3d30003b30 0x7f3d500b30f0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 >> 172.21.15.53:0/3331671600 conn(0x7f3d5006a800 msgr2=0x7f3d500cbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:01.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.705+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 shutdown_connections 2024-01-16T15:03:01.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.706+0000 7f3d54c00640 1 -- 172.21.15.53:0/3331671600 wait complete. 2024-01-16T15:03:01.714 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:03:01.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.949+0000 7f2c6be00640 1 Processor -- start 2024-01-16T15:03:01.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.949+0000 7f2c6be00640 1 -- start start 2024-01-16T15:03:01.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.951+0000 7f2c6be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2c6c0cc990 0x7f2c6c06d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.951+0000 7f2c6be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c070e50 con 0x7f2c6c143c40 2024-01-16T15:03:01.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.951+0000 7f2c6be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c6c144020 con 0x7f2c6c0cc500 2024-01-16T15:03:01.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.951+0000 7f2c6be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c6c0723c0 con 0x7f2c6c0cc990 2024-01-16T15:03:01.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.951+0000 7f2c6aa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2c6c143c40 0x7f2c6c071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46112/0 (socket says 172.21.15.53:46112) 2024-01-16T15:03:01.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.951+0000 7f2c6aa00640 1 -- 172.21.15.53:0/386396265 learned_addr learned my addr 172.21.15.53:0/386396265 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c6b400640 1 --2- 172.21.15.53:0/386396265 >> v2:172.21.15.146:3300/0 conn(0x7f2c6c0cc990 0x7f2c6c06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2972407981 0 0) 0x7f2c6c070e50 con 0x7f2c6c143c40 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c4c003520 con 0x7f2c6c143c40 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3947581161 0 0) 0x7f2c4c003520 con 0x7f2c6c143c40 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 >> v2:172.21.15.146:3300/0 conn(0x7f2c6c0cc990 msgr2=0x7f2c6c06d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 --2- 172.21.15.53:0/386396265 >> v2:172.21.15.146:3300/0 conn(0x7f2c6c0cc990 0x7f2c6c06d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 >> v1:172.21.15.146:6789/0 conn(0x7f2c6c0cc500 legacy=0x7f2c6c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c071e00 con 0x7f2c6c143c40 2024-01-16T15:03:01.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.952+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2c5c001670 con 0x7f2c6c143c40 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 >> v1:172.21.15.53:6789/0 conn(0x7f2c6c143c40 legacy=0x7f2c6c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c003970 0x7f2c4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4c0047c0 0x7f2c4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c4c005180 0x7f2c4c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0723c0 con 0x7f2c4c005180 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c00c120 con 0x7f2c4c003970 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c4c00c2e0 con 0x7f2c4c0047c0 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6b400640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c4c005180 0x7f2c4c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6aa00640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c003970 0x7f2c4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6a000640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4c0047c0 0x7f2c4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6b400640 1 -- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4c0047c0 msgr2=0x7f2c4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6b400640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4c0047c0 0x7f2c4c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6b400640 1 -- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c003970 msgr2=0x7f2c4c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:01.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6b400640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c003970 0x7f2c4c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6b400640 1 -- 172.21.15.53:0/386396265 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c003520 con 0x7f2c4c005180 2024-01-16T15:03:01.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.953+0000 7f2c6aa00640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c003970 0x7f2c4c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:01.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.954+0000 7f2c6a000640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c4c0047c0 0x7f2c4c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.954+0000 7f2c6b400640 1 -- 172.21.15.53:0/386396265 reap_dead start 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.955+0000 7f2c6b400640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c4c005180 0x7f2c4c00b980 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009fc0 tx=0x7f2c4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55b548168a4bb884 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.955+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c6402cc30 con 0x7f2c4c005180 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.955+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c6400f040 con 0x7f2c4c005180 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.955+0000 7f2c69600640 1 -- 172.21.15.53:0/386396265 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c64003990 con 0x7f2c4c005180 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 -- 172.21.15.53:0/386396265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c4c005180 msgr2=0x7f2c4c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c4c005180 0x7f2c4c00b980 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f2c64009fc0 tx=0x7f2c4c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 -- 172.21.15.53:0/386396265 shutdown_connections 2024-01-16T15:03:01.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 --2- 172.21.15.53:0/386396265 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c4c005180 0x7f2c4c00b980 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 -- 172.21.15.53:0/386396265 >> 172.21.15.53:0/386396265 conn(0x7f2c6c06a800 msgr2=0x7f2c6c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:01.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 -- 172.21.15.53:0/386396265 shutdown_connections 2024-01-16T15:03:01.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.956+0000 7f2c6be00640 1 -- 172.21.15.53:0/386396265 wait complete. 2024-01-16T15:03:01.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.958+0000 7f2c6be00640 1 Processor -- start 2024-01-16T15:03:01.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.958+0000 7f2c6be00640 1 -- start start 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c005180 0x7f2c6c15eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 0x7f2c6c13cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c6c0aa400 0x7f2c6c0aa8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c0aadf0 con 0x7f2c6c0a9770 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c15d9b0 con 0x7f2c4c005180 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c6c15de40 con 0x7f2c6c0aa400 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.959+0000 7f2c6a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 0x7f2c6c13cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 0x7f2c6c13cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41288/0 (socket says 172.21.15.53:41288) 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 -- 172.21.15.53:0/1715993629 learned_addr learned my addr 172.21.15.53:0/1715993629 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6aa00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c005180 0x7f2c6c15eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6b400640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c6c0aa400 0x7f2c6c0aa8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 -- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c6c0aa400 msgr2=0x7f2c6c0aa8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c6c0aa400 0x7f2c6c0aa8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 -- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c005180 msgr2=0x7f2c6c15eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c005180 0x7f2c6c15eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 -- 172.21.15.53:0/1715993629 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c15d0a0 con 0x7f2c6c0a9770 2024-01-16T15:03:01.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6b400640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c6c0aa400 0x7f2c6c0aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.960+0000 7f2c6a000640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 0x7f2c6c13cef0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400d770 tx=0x7f2c5400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.961+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c54005270 con 0x7f2c6c0a9770 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.961+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c540053d0 con 0x7f2c6c0a9770 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.961+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c5400b8f0 con 0x7f2c6c0a9770 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.961+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c6c15d360 con 0x7f2c6c0a9770 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.961+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c6c15f400 con 0x7f2c6c0a9770 2024-01-16T15:03:01.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.962+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c6c15db20 con 0x7f2c6c0a9770 2024-01-16T15:03:01.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.963+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f2c5400ba50 con 0x7f2c6c0a9770 2024-01-16T15:03:01.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.964+0000 7f2c62a00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c4406b530 0x7f2c4406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:01.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.965+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f2c54089e40 con 0x7f2c6c0a9770 2024-01-16T15:03:01.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.965+0000 7f2c6aa00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c4406b530 0x7f2c4406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:01.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.966+0000 7f2c6aa00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c4406b530 0x7f2c4406d9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0017a0 tx=0x7f2c5c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:01.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:01.969+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2c54054a20 con 0x7f2c6c0a9770 2024-01-16T15:03:02.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.167+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2c6c067450 con 0x7f2c6c0a9770 2024-01-16T15:03:02.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.716+0000 7f2c62a00640 1 -- 172.21.15.53:0/1715993629 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v45) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2c54019310 con 0x7f2c6c0a9770 2024-01-16T15:03:02.696 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:03:02.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.721+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c4406b530 msgr2=0x7f2c4406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:02.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.721+0000 7f2c6be00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c4406b530 0x7f2c4406d9f0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2c5c0017a0 tx=0x7f2c5c009040 comp rx=0 tx=0).stop 2024-01-16T15:03:02.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.721+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 msgr2=0x7f2c6c13cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:02.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.721+0000 7f2c6be00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 0x7f2c6c13cef0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f2c5400d770 tx=0x7f2c5400dc30 comp rx=0 tx=0).stop 2024-01-16T15:03:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.722+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 shutdown_connections 2024-01-16T15:03:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.722+0000 7f2c6be00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2c4406b530 0x7f2c4406d9f0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.722+0000 7f2c6be00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2c6c0aa400 0x7f2c6c0aa8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.722+0000 7f2c6be00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2c6c0a9770 0x7f2c6c13cef0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.722+0000 7f2c6be00640 1 --2- 172.21.15.53:0/1715993629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2c4c005180 0x7f2c6c15eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.722+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 >> 172.21.15.53:0/1715993629 conn(0x7f2c6c06a800 msgr2=0x7f2c6c0aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:02.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.723+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 shutdown_connections 2024-01-16T15:03:02.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.724+0000 7f2c6be00640 1 -- 172.21.15.53:0/1715993629 wait complete. 2024-01-16T15:03:02.740 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:03:02.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.947+0000 7fb06a200640 1 Processor -- start 2024-01-16T15:03:02.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.948+0000 7fb06a200640 1 -- start start 2024-01-16T15:03:02.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.949+0000 7fb06a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb06413e360 0x7fb0640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.949+0000 7fb06a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.949+0000 7fb06a200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb06415c5f0 con 0x7fb06413df80 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.949+0000 7fb06a200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb06406c470 con 0x7fb06413e360 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.949+0000 7fb06a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb06406c750 con 0x7fb06415f9c0 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb05a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb06413e360 0x7fb0640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb063400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb06413df80 0x7fb06415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46120/0 (socket says 172.21.15.53:46120) 2024-01-16T15:03:02.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb063400640 1 -- 172.21.15.53:0/4250539525 learned_addr learned my addr 172.21.15.53:0/4250539525 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb063e00640 1 --2- 172.21.15.53:0/4250539525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb05a800640 1 -- 172.21.15.53:0/4250539525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 msgr2=0x7fb0640cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb05a800640 1 --2- 172.21.15.53:0/4250539525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb05a800640 1 -- 172.21.15.53:0/4250539525 >> v1:172.21.15.53:6789/0 conn(0x7fb06413df80 legacy=0x7fb06415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb05a800640 1 -- 172.21.15.53:0/4250539525 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb06406ca30 con 0x7fb06413e360 2024-01-16T15:03:02.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.950+0000 7fb062a00640 1 -- 172.21.15.53:0/4250539525 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053616626 0 0) 0x7fb06415c5f0 con 0x7fb06413df80 2024-01-16T15:03:02.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.951+0000 7fb05a800640 1 --2- 172.21.15.53:0/4250539525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413e360 0x7fb0640cf290 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0097a0 tx=0x7fb04c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902eee30b1760af5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:02.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb062a00640 1 -- 172.21.15.53:0/4250539525 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c031830 con 0x7fb06413e360 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb062a00640 1 -- 172.21.15.53:0/4250539525 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb04c031990 con 0x7fb06413e360 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb062a00640 1 -- 172.21.15.53:0/4250539525 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb04c031d20 con 0x7fb06413e360 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 -- 172.21.15.53:0/4250539525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413e360 msgr2=0x7fb0640cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 --2- 172.21.15.53:0/4250539525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413e360 0x7fb0640cf290 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fb04c0097a0 tx=0x7fb04c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 -- 172.21.15.53:0/4250539525 shutdown_connections 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 --2- 172.21.15.53:0/4250539525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 --2- 172.21.15.53:0/4250539525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413e360 0x7fb0640cf290 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 -- 172.21.15.53:0/4250539525 >> 172.21.15.53:0/4250539525 conn(0x7fb06406a800 msgr2=0x7fb06406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 -- 172.21.15.53:0/4250539525 shutdown_connections 2024-01-16T15:03:02.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.952+0000 7fb06a200640 1 -- 172.21.15.53:0/4250539525 wait complete. 2024-01-16T15:03:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.954+0000 7fb06a200640 1 Processor -- start 2024-01-16T15:03:02.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.954+0000 7fb06a200640 1 -- start start 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb06a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 0x7fb06413ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb06a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb06413e360 0x7fb0640b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb06a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb06a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0641438b0 con 0x7fb06413df80 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb06a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb06413b070 con 0x7fb06413e360 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb06a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb064143210 con 0x7fb06415f9c0 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.955+0000 7fb063400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 0x7fb06413ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 0x7fb06413ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41296/0 (socket says 172.21.15.53:41296) 2024-01-16T15:03:02.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 -- 172.21.15.53:0/2149870021 learned_addr learned my addr 172.21.15.53:0/2149870021 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb05a800640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb06413e360 0x7fb0640b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063e00640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 -- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 msgr2=0x7fb0640b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 -- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb06413e360 msgr2=0x7fb0640b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb06413e360 0x7fb0640b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:02.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 -- 172.21.15.53:0/2149870021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb04c009400 con 0x7fb06413df80 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.956+0000 7fb063400640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 0x7fb06413ab30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb05400c400 tx=0x7fb05400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.957+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054005ce0 con 0x7fb06413df80 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.957+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb054005e40 con 0x7fb06413df80 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.957+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb064145e50 con 0x7fb06413df80 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.957+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb054007310 con 0x7fb06413df80 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.957+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb064145120 con 0x7fb06413df80 2024-01-16T15:03:02.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.958+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0640b4310 con 0x7fb06413df80 2024-01-16T15:03:02.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.959+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fb054007530 con 0x7fb06413df80 2024-01-16T15:03:02.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.960+0000 7fb05be00640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03c06b500 0x7fb03c06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:02.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.960+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5609+0+0 (secure 0 0 0) 0x7fb0540895c0 con 0x7fb06413df80 2024-01-16T15:03:02.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.961+0000 7fb05a800640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03c06b500 0x7fb03c06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:02.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.962+0000 7fb05a800640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03c06b500 0x7fb03c06d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb04c036000 tx=0x7fb04c03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:02.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:02.965+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb054054120 con 0x7fb06413df80 2024-01-16T15:03:03.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:03.201+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb0640675d0 con 0x7fb06413df80 2024-01-16T15:03:04.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.052+0000 7fb05be00640 1 -- 172.21.15.53:0/2149870021 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v21) v1 ==== 172+0+0 (secure 0 0 0) 0x7fb054057740 con 0x7fb06413df80 2024-01-16T15:03:04.032 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 6 and data pool 7 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.055+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03c06b500 msgr2=0x7fb03c06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.055+0000 7fb06a200640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03c06b500 0x7fb03c06d9c0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb04c036000 tx=0x7fb04c03c040 comp rx=0 tx=0).stop 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.055+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 msgr2=0x7fb06413ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.055+0000 7fb06a200640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 0x7fb06413ab30 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fb05400c400 tx=0x7fb05400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 shutdown_connections 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03c06b500 0x7fb03c06d9c0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb06415f9c0 0x7fb0640b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb06413e360 0x7fb0640b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 --2- 172.21.15.53:0/2149870021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb06413df80 0x7fb06413ab30 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 >> 172.21.15.53:0/2149870021 conn(0x7fb06406a800 msgr2=0x7fb06415dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:04.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 shutdown_connections 2024-01-16T15:03:04.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.056+0000 7fb06a200640 1 -- 172.21.15.53:0/2149870021 wait complete. 2024-01-16T15:03:04.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:03:04.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.275+0000 7f5130000640 1 Processor -- start 2024-01-16T15:03:04.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.275+0000 7f5130000640 1 -- start start 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.277+0000 7f5130000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f512813e360 0x7f51280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.277+0000 7f5130000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512815c5f0 con 0x7f512815f9c0 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.277+0000 7f5130000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f512806c470 con 0x7f512813df80 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.277+0000 7f5130000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f512806c750 con 0x7f512813e360 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.277+0000 7f512cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f512813e360 0x7f51280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f512cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f512813e360 0x7f51280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41300/0 (socket says 172.21.15.53:41300) 2024-01-16T15:03:04.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f512cc00640 1 -- 172.21.15.53:0/3138207043 learned_addr learned my addr 172.21.15.53:0/3138207043 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:04.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f512e000640 1 --1- 172.21.15.53:0/3138207043 >> v1:172.21.15.53:6789/0 conn(0x7f512815f9c0 0x7f51280cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46124/0 (socket says 172.21.15.53:46124) 2024-01-16T15:03:04.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f512cc00640 1 -- 172.21.15.53:0/3138207043 >> v1:172.21.15.146:6789/0 conn(0x7f512813df80 legacy=0x7f512815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:04.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f512cc00640 1 -- 172.21.15.53:0/3138207043 >> v1:172.21.15.53:6789/0 conn(0x7f512815f9c0 legacy=0x7f51280cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f5127e00640 1 -- 172.21.15.53:0/3138207043 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345613078 0 0) 0x7f512815c5f0 con 0x7f512815f9c0 2024-01-16T15:03:04.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.278+0000 7f512cc00640 1 -- 172.21.15.53:0/3138207043 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512806ca30 con 0x7f512813e360 2024-01-16T15:03:04.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.279+0000 7f512cc00640 1 --2- 172.21.15.53:0/3138207043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280cf290 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5110009870 tx=0x7f511002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19dbcaad723fbad4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:04.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.279+0000 7f5127e00640 1 -- 172.21.15.53:0/3138207043 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110030860 con 0x7f512813e360 2024-01-16T15:03:04.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5127e00640 1 -- 172.21.15.53:0/3138207043 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51100309c0 con 0x7f512813e360 2024-01-16T15:03:04.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5127e00640 1 -- 172.21.15.53:0/3138207043 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110030d50 con 0x7f512813e360 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5130000640 1 -- 172.21.15.53:0/3138207043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 msgr2=0x7f51280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5130000640 1 --2- 172.21.15.53:0/3138207043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280cf290 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5110009870 tx=0x7f511002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5130000640 1 -- 172.21.15.53:0/3138207043 shutdown_connections 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5130000640 1 --2- 172.21.15.53:0/3138207043 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280cf290 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5130000640 1 -- 172.21.15.53:0/3138207043 >> 172.21.15.53:0/3138207043 conn(0x7f512806a800 msgr2=0x7f512806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.280+0000 7f5130000640 1 -- 172.21.15.53:0/3138207043 shutdown_connections 2024-01-16T15:03:04.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.281+0000 7f5130000640 1 -- 172.21.15.53:0/3138207043 wait complete. 2024-01-16T15:03:04.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.283+0000 7f5130000640 1 Processor -- start 2024-01-16T15:03:04.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.283+0000 7f5130000640 1 -- start start 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f5130000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f512813df80 0x7f512813cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f5130000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f5130000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f512815f9c0 0x7f51280b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f5130000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5128151f00 con 0x7f512813e360 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f5130000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51281521e0 con 0x7f512815f9c0 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f5130000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51281524c0 con 0x7f512813df80 2024-01-16T15:03:04.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.284+0000 7f512cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41302/0 (socket says 172.21.15.53:41302) 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 -- 172.21.15.53:0/1612872623 learned_addr learned my addr 172.21.15.53:0/1612872623 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512d600640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f512813df80 0x7f512813cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512e000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f512815f9c0 0x7f51280b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 -- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f512813df80 msgr2=0x7f512813cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f512813df80 0x7f512813cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 -- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f512815f9c0 msgr2=0x7f51280b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f512815f9c0 0x7f51280b3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.285+0000 7f512cc00640 1 -- 172.21.15.53:0/1612872623 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51100094d0 con 0x7f512813e360 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.286+0000 7f512cc00640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280b2cf0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5110009870 tx=0x7f5110004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.286+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511003e040 con 0x7f512813e360 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.286+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5110039e20 con 0x7f512813e360 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.286+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5110004270 con 0x7f512813e360 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.286+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51281527a0 con 0x7f512813e360 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.286+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f512813d1c0 con 0x7f512813e360 2024-01-16T15:03:04.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.287+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5128152cc0 con 0x7f512813e360 2024-01-16T15:03:04.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.288+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f51100043d0 con 0x7f512813e360 2024-01-16T15:03:04.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.289+0000 7f5125600640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510006b500 0x7f510006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.289+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f51100ac810 con 0x7f512813e360 2024-01-16T15:03:04.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.290+0000 7f512d600640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510006b500 0x7f510006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.291+0000 7f512d600640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510006b500 0x7f510006d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5118009f20 tx=0x7f51180094d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:04.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.294+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5128152cc0 con 0x7f512813e360 2024-01-16T15:03:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.491+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5128152cc0 con 0x7f512813e360 2024-01-16T15:03:04.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.493+0000 7f5125600640 1 -- 172.21.15.53:0/1612872623 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+16490 (secure 0 0 0) 0x7f5128152cc0 con 0x7f512813e360 2024-01-16T15:03:04.473 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:04.473 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":46,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:03:03.202957+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:03:01.086339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:03:02.224069+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:03:04.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510006b500 msgr2=0x7f510006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510006b500 0x7f510006d9c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5118009f20 tx=0x7f51180094d0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 msgr2=0x7f51280b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280b2cf0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5110009870 tx=0x7f5110004730 comp rx=0 tx=0).stop 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 shutdown_connections 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510006b500 0x7f510006d9c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f512815f9c0 0x7f51280b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f512813e360 0x7f51280b2cf0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 --2- 172.21.15.53:0/1612872623 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f512813df80 0x7f512813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 >> 172.21.15.53:0/1612872623 conn(0x7f512806a800 msgr2=0x7f5128070660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 shutdown_connections 2024-01-16T15:03:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.496+0000 7f5130000640 1 -- 172.21.15.53:0/1612872623 wait complete. 2024-01-16T15:03:04.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:03:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.737+0000 7f4cc0e00640 1 Processor -- start 2024-01-16T15:03:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.737+0000 7f4cc0e00640 1 -- start start 2024-01-16T15:03:04.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cc0e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4cbc0cc990 0x7f4cbc06d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cc0e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc070e50 con 0x7f4cbc143c40 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cc0e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cbc144020 con 0x7f4cbc0cc500 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cc0e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cbc0723c0 con 0x7f4cbc0cc990 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cbb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4cbc143c40 0x7f4cbc071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46130/0 (socket says 172.21.15.53:46130) 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cbb400640 1 -- 172.21.15.53:0/2222801985 learned_addr learned my addr 172.21.15.53:0/2222801985 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.739+0000 7f4cbbe00640 1 --2- 172.21.15.53:0/2222801985 >> v2:172.21.15.146:3300/0 conn(0x7f4cbc0cc990 0x7f4cbc06d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245873189 0 0) 0x7f4cbc070e50 con 0x7f4cbc143c40 2024-01-16T15:03:04.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c9c003520 con 0x7f4cbc143c40 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088535977 0 0) 0x7f4c9c003520 con 0x7f4cbc143c40 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 >> v2:172.21.15.146:3300/0 conn(0x7f4cbc0cc990 msgr2=0x7f4cbc06d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 --2- 172.21.15.53:0/2222801985 >> v2:172.21.15.146:3300/0 conn(0x7f4cbc0cc990 0x7f4cbc06d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 >> v1:172.21.15.146:6789/0 conn(0x7f4cbc0cc500 legacy=0x7f4cbc06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc071e00 con 0x7f4cbc143c40 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4cac001670 con 0x7f4cbc143c40 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 >> v1:172.21.15.53:6789/0 conn(0x7f4cbc143c40 legacy=0x7f4cbc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9c003970 0x7f4c9c003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c0047c0 0x7f4c9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c9c005180 0x7f4c9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0723c0 con 0x7f4c9c003970 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00c0b0 con 0x7f4c9c005180 2024-01-16T15:03:04.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.740+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c00c270 con 0x7f4c9c0047c0 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbb400640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9c003970 0x7f4c9c003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbbe00640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c9c005180 0x7f4c9c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cb2800640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c0047c0 0x7f4c9c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbb400640 1 -- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c0047c0 msgr2=0x7f4c9c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbb400640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c0047c0 0x7f4c9c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbb400640 1 -- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c9c005180 msgr2=0x7f4c9c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbb400640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c9c005180 0x7f4c9c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbb400640 1 -- 172.21.15.53:0/2222801985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c003520 con 0x7f4c9c003970 2024-01-16T15:03:04.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.741+0000 7f4cbbe00640 1 -- 172.21.15.53:0/2222801985 reap_dead start 2024-01-16T15:03:04.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.742+0000 7f4cbb400640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9c003970 0x7f4c9c003d70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c210 tx=0x7f4c9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca9809d25d13c16 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:04.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.743+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac005ad0 con 0x7f4c9c003970 2024-01-16T15:03:04.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.743+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cac005c50 con 0x7f4c9c003970 2024-01-16T15:03:04.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.743+0000 7f4cbaa00640 1 -- 172.21.15.53:0/2222801985 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cac02cae0 con 0x7f4c9c003970 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.743+0000 7f4cc0e00640 1 -- 172.21.15.53:0/2222801985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9c003970 msgr2=0x7f4c9c003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.743+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9c003970 0x7f4c9c003d70 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4cac00c210 tx=0x7f4c9c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.744+0000 7f4cc0e00640 1 -- 172.21.15.53:0/2222801985 shutdown_connections 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.744+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/2222801985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c9c003970 0x7f4c9c003d70 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.744+0000 7f4cc0e00640 1 -- 172.21.15.53:0/2222801985 >> 172.21.15.53:0/2222801985 conn(0x7f4cbc06a800 msgr2=0x7f4cbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.744+0000 7f4cc0e00640 1 -- 172.21.15.53:0/2222801985 shutdown_connections 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.744+0000 7f4cc0e00640 1 -- 172.21.15.53:0/2222801985 wait complete. 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.745+0000 7f4cc0e00640 1 Processor -- start 2024-01-16T15:03:04.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.746+0000 7f4cc0e00640 1 -- start start 2024-01-16T15:03:04.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.746+0000 7f4cc0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c003970 0x7f4cbc1f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.746+0000 7f4cc0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cbc15d980 0x7f4cbc15de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cc0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 0x7f4cbc0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cc0e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc15e340 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cc0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0ae180 con 0x7f4cbc15d980 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cc0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cbc0ad570 con 0x7f4c9c003970 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 0x7f4cbc0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 0x7f4cbc0acd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41316/0 (socket says 172.21.15.53:41316) 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 -- 172.21.15.53:0/3688527755 learned_addr learned my addr 172.21.15.53:0/3688527755 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbb400640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c003970 0x7f4cbc1f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 -- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c003970 msgr2=0x7f4cbc1f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c003970 0x7f4cbc1f22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 -- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cbc15d980 msgr2=0x7f4cbc15de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cbc15d980 0x7f4cbc15de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.747+0000 7f4cbbe00640 1 -- 172.21.15.53:0/3688527755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cbc0ad850 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.748+0000 7f4cbbe00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 0x7f4cbc0acd50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0ce1a0 tx=0x7f4cb4011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:04.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.748+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb4004500 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.748+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cb4003610 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.748+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cb40039c0 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.748+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cbc0adb30 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.748+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cbc0ad290 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.749+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cbc067450 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.750+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f4cb4003b20 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.751+0000 7f4cb3e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c9406b500 0x7f4c9406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:04.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.751+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4cb4089ef0 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.751+0000 7f4cbb400640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c9406b500 0x7f4c9406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:04.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.753+0000 7f4cbb400640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c9406b500 0x7f4c9406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4cac034000 tx=0x7f4cac009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:04.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.755+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4cb4054960 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.946+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cbc0ae2f0 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.947+0000 7f4cb3e00640 1 -- 172.21.15.53:0/3688527755 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16477 (secure 0 0 0) 0x7f4cb4057f80 con 0x7f4cbc0ac8d0 2024-01-16T15:03:04.927 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:04.927 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":47,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:03:04.720588+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:03:01.086339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:03:02.224069+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c9406b500 msgr2=0x7f4c9406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c9406b500 0x7f4c9406d9c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4cac034000 tx=0x7f4cac009030 comp rx=0 tx=0).stop 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 msgr2=0x7f4cbc0acd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 0x7f4cbc0acd50 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4cbc0ce1a0 tx=0x7f4cb4011920 comp rx=0 tx=0).stop 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 shutdown_connections 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c9406b500 0x7f4c9406d9c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4cbc0ac8d0 0x7f4cbc0acd50 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4cbc15d980 0x7f4cbc15de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 --2- 172.21.15.53:0/3688527755 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c9c003970 0x7f4cbc1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 >> 172.21.15.53:0/3688527755 conn(0x7f4cbc06a800 msgr2=0x7f4cbc13e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:04.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 shutdown_connections 2024-01-16T15:03:04.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:04.950+0000 7f4cc0e00640 1 -- 172.21.15.53:0/3688527755 wait complete. 2024-01-16T15:03:04.952 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:03:05.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.191+0000 7f5136600640 1 Processor -- start 2024-01-16T15:03:05.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.192+0000 7f5136600640 1 -- start start 2024-01-16T15:03:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.193+0000 7f5136600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51300cf9a0 0x7f51300cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.193+0000 7f5136600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300d0320 con 0x7f513015d4b0 2024-01-16T15:03:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.193+0000 7f5136600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300d1c60 con 0x7f51300b31d0 2024-01-16T15:03:05.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.193+0000 7f5136600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51300b2730 con 0x7f51300cf9a0 2024-01-16T15:03:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.193+0000 7f5135200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f513015d4b0 0x7f51300d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46142/0 (socket says 172.21.15.53:46142) 2024-01-16T15:03:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.193+0000 7f5135200640 1 -- 172.21.15.53:0/3197133819 learned_addr learned my addr 172.21.15.53:0/3197133819 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f5127e00640 1 --2- 172.21.15.53:0/3197133819 >> v2:172.21.15.146:3300/0 conn(0x7f51300cf9a0 0x7f51300cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179118222 0 0) 0x7f51300d0320 con 0x7f513015d4b0 2024-01-16T15:03:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5110003520 con 0x7f513015d4b0 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1892244690 0 0) 0x7f5110003520 con 0x7f513015d4b0 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 >> v2:172.21.15.146:3300/0 conn(0x7f51300cf9a0 msgr2=0x7f51300cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 --2- 172.21.15.53:0/3197133819 >> v2:172.21.15.146:3300/0 conn(0x7f51300cf9a0 0x7f51300cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 >> v1:172.21.15.146:6789/0 conn(0x7f51300b31d0 legacy=0x7f51300b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51300d1ec0 con 0x7f513015d4b0 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.194+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5120001670 con 0x7f513015d4b0 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 >> v1:172.21.15.53:6789/0 conn(0x7f513015d4b0 legacy=0x7f51300d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51100039c0 0x7f5110003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51100047c0 0x7f5110004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f511000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300b2730 con 0x7f5110005180 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511000c080 con 0x7f51100047c0 2024-01-16T15:03:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511000c240 con 0x7f51100039c0 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f5135c00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f511000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f5135200640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51100039c0 0x7f5110003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.195+0000 7f5127e00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51100047c0 0x7f5110004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.196+0000 7f5135c00640 1 -- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51100039c0 msgr2=0x7f5110003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.196+0000 7f5135c00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51100039c0 0x7f5110003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.196+0000 7f5135c00640 1 -- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51100047c0 msgr2=0x7f5110004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.196+0000 7f5135c00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51100047c0 0x7f5110004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.196+0000 7f5135c00640 1 -- 172.21.15.53:0/3197133819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5110003520 con 0x7f5110005180 2024-01-16T15:03:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.196+0000 7f5127e00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51100047c0 0x7f5110004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:05.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f5135c00640 1 -- 172.21.15.53:0/3197133819 reap_dead start 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f5135c00640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f511000b980 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5128001e50 tx=0x7f51100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb56c2aa0cdc535 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128003b70 con 0x7f5110005180 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5128003d10 con 0x7f5110005180 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f512fe00640 1 -- 172.21.15.53:0/3197133819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5128005310 con 0x7f5110005180 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f5136600640 1 -- 172.21.15.53:0/3197133819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 msgr2=0x7f511000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.197+0000 7f5136600640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f511000b980 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f5128001e50 tx=0x7f51100057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.198+0000 7f5136600640 1 -- 172.21.15.53:0/3197133819 shutdown_connections 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.198+0000 7f5136600640 1 --2- 172.21.15.53:0/3197133819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f511000b980 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.198+0000 7f5136600640 1 -- 172.21.15.53:0/3197133819 >> 172.21.15.53:0/3197133819 conn(0x7f51300d3000 msgr2=0x7f51300d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.198+0000 7f5136600640 1 -- 172.21.15.53:0/3197133819 shutdown_connections 2024-01-16T15:03:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.198+0000 7f5136600640 1 -- 172.21.15.53:0/3197133819 wait complete. 2024-01-16T15:03:05.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.200+0000 7f5136600640 1 Processor -- start 2024-01-16T15:03:05.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.200+0000 7f5136600640 1 -- start start 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5136600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f5130154440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5136600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513015a4b0 0x7f513009d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5136600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513015bcf0 0x7f513009df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5136600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300a0870 con 0x7f5110005180 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5136600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51300a0b50 con 0x7f513015a4b0 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5136600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51301f4280 con 0x7f513015bcf0 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5135200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f5130154440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5135200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f5130154440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41328/0 (socket says 172.21.15.53:41328) 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.201+0000 7f5135200640 1 -- 172.21.15.53:0/202880307 learned_addr learned my addr 172.21.15.53:0/202880307 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5135200640 1 -- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513015bcf0 msgr2=0x7f513009df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5135200640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513015bcf0 0x7f513009df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5135200640 1 -- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513015a4b0 msgr2=0x7f513009d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5127e00640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513015a4b0 0x7f513009d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5135200640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513015a4b0 0x7f513009d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5135200640 1 -- 172.21.15.53:0/202880307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51301f4560 con 0x7f5110005180 2024-01-16T15:03:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5135200640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f5130154440 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f512000e730 tx=0x7f512000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512000c6c0 con 0x7f5110005180 2024-01-16T15:03:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512000c840 con 0x7f5110005180 2024-01-16T15:03:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130154980 con 0x7f5110005180 2024-01-16T15:03:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f512000cbf0 con 0x7f5110005180 2024-01-16T15:03:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.202+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51301f2f80 con 0x7f5110005180 2024-01-16T15:03:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.203+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51301f47a0 con 0x7f5110005180 2024-01-16T15:03:05.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.204+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f512000cd50 con 0x7f5110005180 2024-01-16T15:03:05.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.206+0000 7f512d600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510806b500 0x7f510806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:05.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.206+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5120089ea0 con 0x7f5110005180 2024-01-16T15:03:05.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.207+0000 7f5127e00640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510806b500 0x7f510806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:05.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.208+0000 7f5127e00640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510806b500 0x7f510806d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f51180099c0 tx=0x7f5118008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:05.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.210+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51301f47a0 con 0x7f5110005180 2024-01-16T15:03:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:05.447+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f51301f47a0 con 0x7f5110005180 2024-01-16T15:03:06.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.065+0000 7f512d600640 1 -- 172.21.15.53:0/202880307 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f51301f47a0 con 0x7f5110005180 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510806b500 msgr2=0x7f510806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510806b500 0x7f510806d9c0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f51180099c0 tx=0x7f5118008040 comp rx=0 tx=0).stop 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 msgr2=0x7f5130154440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f5130154440 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f512000e730 tx=0x7f512000ebf0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 shutdown_connections 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f510806b500 0x7f510806d9c0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f513015bcf0 0x7f513009df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.069+0000 7f5136600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f513015a4b0 0x7f513009d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.070+0000 7f5136600640 1 --2- 172.21.15.53:0/202880307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5110005180 0x7f5130154440 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.070+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 >> 172.21.15.53:0/202880307 conn(0x7f51300d3000 msgr2=0x7f51300b0060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.070+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 shutdown_connections 2024-01-16T15:03:06.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.070+0000 7f5136600640 1 -- 172.21.15.53:0/202880307 wait complete. 2024-01-16T15:03:06.071 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:03:06.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:03:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.317+0000 7efbfd200640 1 Processor -- start 2024-01-16T15:03:06.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.317+0000 7efbfd200640 1 -- start start 2024-01-16T15:03:06.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbfd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:06.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbfd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efbf80b41e0 0x7efbf80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:06.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbfd200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efbf80d0320 con 0x7efbf80cf9a0 2024-01-16T15:03:06.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbfd200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efbf80b2730 con 0x7efbf80b41e0 2024-01-16T15:03:06.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbfd200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80d1ec0 con 0x7efbf815d4b0 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbf7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efbf80b41e0 0x7efbf80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbf7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efbf80b41e0 0x7efbf80cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:37984/0 (socket says 172.21.15.53:37984) 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbf7e00640 1 -- 172.21.15.53:0/4153602449 learned_addr learned my addr 172.21.15.53:0/4153602449 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.319+0000 7efbf7400640 1 --2- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.320+0000 7efbf7e00640 1 -- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 msgr2=0x7efbf80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.320+0000 7efbf7e00640 1 --2- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.320+0000 7efbf7e00640 1 -- 172.21.15.53:0/4153602449 >> v1:172.21.15.146:6789/0 conn(0x7efbf80cf9a0 legacy=0x7efbf80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:06.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.320+0000 7efbf7e00640 1 -- 172.21.15.53:0/4153602449 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf80b29d0 con 0x7efbf80b41e0 2024-01-16T15:03:06.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.321+0000 7efbf7e00640 1 --2- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf80cfde0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009e80 tx=0x7efbf002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e64f715db9fd5be7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:06.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.322+0000 7efbf6000640 1 -- 172.21.15.53:0/4153602449 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf002caf0 con 0x7efbf80b41e0 2024-01-16T15:03:06.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.322+0000 7efbf6000640 1 -- 172.21.15.53:0/4153602449 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf002cc50 con 0x7efbf80b41e0 2024-01-16T15:03:06.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.322+0000 7efbf6000640 1 -- 172.21.15.53:0/4153602449 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf00343f0 con 0x7efbf80b41e0 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 -- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 msgr2=0x7efbf80cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 --2- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf80cfde0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7efbf0009e80 tx=0x7efbf002bf50 comp rx=0 tx=0).stop 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 -- 172.21.15.53:0/4153602449 shutdown_connections 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 --2- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf80cfde0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 --2- 172.21.15.53:0/4153602449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 -- 172.21.15.53:0/4153602449 >> 172.21.15.53:0/4153602449 conn(0x7efbf80d3000 msgr2=0x7efbf80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 -- 172.21.15.53:0/4153602449 shutdown_connections 2024-01-16T15:03:06.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.323+0000 7efbfd200640 1 -- 172.21.15.53:0/4153602449 wait complete. 2024-01-16T15:03:06.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.325+0000 7efbfd200640 1 Processor -- start 2024-01-16T15:03:06.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.325+0000 7efbfd200640 1 -- start start 2024-01-16T15:03:06.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbfd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf814e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbfd200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 0x7efbf80ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbfd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbfd200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80cbd40 con 0x7efbf80cf9a0 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbfd200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80ca760 con 0x7efbf80b41e0 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbfd200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efbf80caa40 con 0x7efbf815d4b0 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 0x7efbf80ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 0x7efbf80ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41342/0 (socket says 172.21.15.53:41342) 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbf6a00640 1 -- 172.21.15.53:0/875292356 learned_addr learned my addr 172.21.15.53:0/875292356 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.326+0000 7efbf7400640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf814e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf7e00640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf6a00640 1 -- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 msgr2=0x7efbf80cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf6a00640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf6a00640 1 -- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 msgr2=0x7efbf814e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf6a00640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf814e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf6a00640 1 -- 172.21.15.53:0/875292356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0009b30 con 0x7efbf80cf9a0 2024-01-16T15:03:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbf6a00640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 0x7efbf80ca0d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7efbe000c400 tx=0x7efbe000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe00052d0 con 0x7efbf80cf9a0 2024-01-16T15:03:06.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbe0005450 con 0x7efbf80cf9a0 2024-01-16T15:03:06.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efbf815da10 con 0x7efbf80cf9a0 2024-01-16T15:03:06.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbe0007260 con 0x7efbf80cf9a0 2024-01-16T15:03:06.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.327+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efbf80cb000 con 0x7efbf80cf9a0 2024-01-16T15:03:06.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.329+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efbf80d2cd0 con 0x7efbf80cf9a0 2024-01-16T15:03:06.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.330+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7efbe00073c0 con 0x7efbf80cf9a0 2024-01-16T15:03:06.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.331+0000 7efbef400640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbd006b500 0x7efbd006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:06.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.331+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efbe00893c0 con 0x7efbf80cf9a0 2024-01-16T15:03:06.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.331+0000 7efbf7400640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbd006b500 0x7efbd006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:06.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.333+0000 7efbf7400640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbd006b500 0x7efbd006d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009a30 tx=0x7efbe8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:06.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.335+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efbe0053f00 con 0x7efbf80cf9a0 2024-01-16T15:03:06.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:06.570+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7efbf8067450 con 0x7efbf80cf9a0 2024-01-16T15:03:07.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.076+0000 7efbef400640 1 -- 172.21.15.53:0/875292356 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v25) v1 ==== 122+0+0 (secure 0 0 0) 0x7efbe0057520 con 0x7efbf80cf9a0 2024-01-16T15:03:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.079+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbd006b500 msgr2=0x7efbd006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.079+0000 7efbfd200640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbd006b500 0x7efbd006d9c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7efbe8009a30 tx=0x7efbe8008040 comp rx=0 tx=0).stop 2024-01-16T15:03:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.079+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 msgr2=0x7efbf80ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.079+0000 7efbfd200640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 0x7efbf80ca0d0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7efbe000c400 tx=0x7efbe000c8c0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 shutdown_connections 2024-01-16T15:03:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efbd006b500 0x7efbd006d9c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efbf815d4b0 0x7efbf80cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efbf80cf9a0 0x7efbf80ca0d0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 --2- 172.21.15.53:0/875292356 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efbf80b41e0 0x7efbf814e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 >> 172.21.15.53:0/875292356 conn(0x7efbf80d3000 msgr2=0x7efbf806ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 shutdown_connections 2024-01-16T15:03:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.080+0000 7efbfd200640 1 -- 172.21.15.53:0/875292356 wait complete. 2024-01-16T15:03:07.080 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:03:07.081 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:03:07.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.310+0000 7f60ba800640 1 Processor -- start 2024-01-16T15:03:07.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.310+0000 7f60ba800640 1 -- start start 2024-01-16T15:03:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60ba800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 0x7f60b415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60ba800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40b1ee0 con 0x7f60b4143ae0 2024-01-16T15:03:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60ba800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b40b2780 con 0x7f60b415e1e0 2024-01-16T15:03:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60ba800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b415e990 con 0x7f60b40cf9a0 2024-01-16T15:03:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60b3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f60b4143ae0 0x7f60b40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46152/0 (socket says 172.21.15.53:46152) 2024-01-16T15:03:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60b3400640 1 -- 172.21.15.53:0/1032912160 learned_addr learned my addr 172.21.15.53:0/1032912160 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:07.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.312+0000 7f60abe00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 0x7f60b415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181852618 0 0) 0x7f60b40b1ee0 con 0x7f60b4143ae0 2024-01-16T15:03:07.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6094005730 con 0x7f60b4143ae0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173036366 0 0) 0x7f60b40b2780 con 0x7f60b415e1e0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60b40b1ee0 con 0x7f60b415e1e0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 139997059 0 0) 0x7f6094005730 con 0x7f60b4143ae0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 msgr2=0x7f60b415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 0x7f60b415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 >> v1:172.21.15.146:6789/0 conn(0x7f60b415e1e0 legacy=0x7f60b40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40b2130 con 0x7f60b4143ae0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.313+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f60a4001670 con 0x7f60b4143ae0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 >> v1:172.21.15.53:6789/0 conn(0x7f60b4143ae0 legacy=0x7f60b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 0x7f6094005d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60940062a0 0x7f60940069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6094006f10 0x7f6094007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b415e990 con 0x7f6094006f10 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609400dc70 con 0x7f60940062a0 2024-01-16T15:03:07.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f609400de30 con 0x7f60b40cf9a0 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b3e00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6094006f10 0x7f6094007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60abe00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60940062a0 0x7f60940069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.314+0000 7f60b3400640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 0x7f6094005d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.315+0000 7f60b3e00640 1 -- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 msgr2=0x7f6094005d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.315+0000 7f60b3e00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f60b40cf9a0 0x7f6094005d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.315+0000 7f60b3e00640 1 -- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60940062a0 msgr2=0x7f60940069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.315+0000 7f60b3e00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60940062a0 0x7f60940069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.315+0000 7f60b3e00640 1 -- 172.21.15.53:0/1032912160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6094005730 con 0x7f6094006f10 2024-01-16T15:03:07.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.316+0000 7f60b3e00640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6094006f10 0x7f6094007390 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f60ac001fd0 tx=0x7f6094007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9160190b329a3ed7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:07.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.316+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac003b70 con 0x7f6094006f10 2024-01-16T15:03:07.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.316+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60ac003cd0 con 0x7f6094006f10 2024-01-16T15:03:07.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.316+0000 7f60b2a00640 1 -- 172.21.15.53:0/1032912160 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0052f0 con 0x7f6094006f10 2024-01-16T15:03:07.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.316+0000 7f60ba800640 1 -- 172.21.15.53:0/1032912160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6094006f10 msgr2=0x7f6094007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.316+0000 7f60ba800640 1 --2- 172.21.15.53:0/1032912160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6094006f10 0x7f6094007390 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f60ac001fd0 tx=0x7f6094007a00 comp rx=0 tx=0).stop 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.317+0000 7f60b3e00640 1 -- 172.21.15.53:0/1032912160 reap_dead start 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.317+0000 7f60ba800640 1 -- 172.21.15.53:0/1032912160 shutdown_connections 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.317+0000 7f60ba800640 1 -- 172.21.15.53:0/1032912160 >> 172.21.15.53:0/1032912160 conn(0x7f60b406a800 msgr2=0x7f60b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.317+0000 7f60ba800640 1 -- 172.21.15.53:0/1032912160 shutdown_connections 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.317+0000 7f60ba800640 1 -- 172.21.15.53:0/1032912160 wait complete. 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.318+0000 7f60ba800640 1 Processor -- start 2024-01-16T15:03:07.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.319+0000 7f60ba800640 1 -- start start 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60ba800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6094006f10 0x7f60b40b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60ba800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 0x7f60b40aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60ba800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60b40ad120 0x7f60b4159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60ba800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b413fcd0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60ba800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40b0610 con 0x7f60b40ad120 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60ba800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b40b08f0 con 0x7f6094006f10 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 0x7f60b40aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 0x7f60b40aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41366/0 (socket says 172.21.15.53:41366) 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 -- 172.21.15.53:0/2028850354 learned_addr learned my addr 172.21.15.53:0/2028850354 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 -- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6094006f10 msgr2=0x7f60b40b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6094006f10 0x7f60b40b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 -- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60b40ad120 msgr2=0x7f60b4159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60b40ad120 0x7f60b4159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.320+0000 7f60abe00640 1 -- 172.21.15.53:0/2028850354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b40b0bd0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.321+0000 7f60abe00640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 0x7f60b40aff30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f609c011600 tx=0x7f609c011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.321+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c015870 con 0x7f60b40c3ec0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.321+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0159d0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.321+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c015d40 con 0x7f60b40c3ec0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.321+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b40b0eb0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.321+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b40ad5a0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.322+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b40675d0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.323+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f609c00f490 con 0x7f60b40c3ec0 2024-01-16T15:03:07.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.324+0000 7f60ab400640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f608c06b580 0x7f608c06da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.324+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f609c08a800 con 0x7f60b40c3ec0 2024-01-16T15:03:07.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.325+0000 7f60b3400640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f608c06b580 0x7f608c06da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.326+0000 7f60b3400640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f608c06b580 0x7f608c06da40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f60a4001050 tx=0x7f60a4001de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:07.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.329+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f609c0552f0 con 0x7f60b40c3ec0 2024-01-16T15:03:07.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.563+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60b406de40 con 0x7f60b40c3ec0 2024-01-16T15:03:07.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.564+0000 7f60ab400640 1 -- 172.21.15.53:0/2028850354 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f609c058910 con 0x7f60b40c3ec0 2024-01-16T15:03:07.545 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:07.545 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3977954404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3977954404},{"type":"v1","addr":"172.21.15.146:6835","nonce":3977954404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3669971149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3669971149},{"type":"v1","addr":"172.21.15.146:6837","nonce":3669971149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5709,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1838972240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1838972240},{"type":"v1","addr":"172.21.15.53:6833","nonce":1838972240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:03:03.203889+0000","modified":"2024-01-16T15:03:06.573035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5718":{"gid":5718,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/4293818684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":4293818684},{"type":"v1","addr":"172.21.15.53:6835","nonce":4293818684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:03:07.545 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 25 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f608c06b580 msgr2=0x7f608c06da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f608c06b580 0x7f608c06da40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f60a4001050 tx=0x7f60a4001de0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 msgr2=0x7f60b40aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 0x7f60b40aff30 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f609c011600 tx=0x7f609c011ac0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 shutdown_connections 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f608c06b580 0x7f608c06da40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f60b40ad120 0x7f60b4159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f60b40c3ec0 0x7f60b40aff30 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 --2- 172.21.15.53:0/2028850354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6094006f10 0x7f60b40b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 >> 172.21.15.53:0/2028850354 conn(0x7f60b406a800 msgr2=0x7f60b415cab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 shutdown_connections 2024-01-16T15:03:07.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.567+0000 7f60ba800640 1 -- 172.21.15.53:0/2028850354 wait complete. 2024-01-16T15:03:07.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:03:07.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.815+0000 7f3429200640 1 Processor -- start 2024-01-16T15:03:07.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.815+0000 7f3429200640 1 -- start start 2024-01-16T15:03:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.817+0000 7f3429200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f34240ae5a0 0x7f342415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.817+0000 7f3429200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34240cf290 0x7f34240cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.817+0000 7f3429200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3424070c40 con 0x7f34240b3ae0 2024-01-16T15:03:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.817+0000 7f3429200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3424071450 con 0x7f34240cf290 2024-01-16T15:03:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.817+0000 7f3429200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34240cfc60 con 0x7f34240ae5a0 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.818+0000 7f3422800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34240cf290 0x7f34240cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.818+0000 7f3422800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f34240cf290 0x7f34240cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41376/0 (socket says 172.21.15.53:41376) 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.818+0000 7f3422800640 1 -- 172.21.15.53:0/2844547794 learned_addr learned my addr 172.21.15.53:0/2844547794 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.818+0000 7f3421e00640 1 --2- 172.21.15.53:0/2844547794 >> v2:172.21.15.146:3300/0 conn(0x7f34240ae5a0 0x7f342415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.819+0000 7f3421e00640 1 -- 172.21.15.53:0/2844547794 >> v2:172.21.15.53:3300/0 conn(0x7f34240cf290 msgr2=0x7f34240cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.819+0000 7f3421e00640 1 --2- 172.21.15.53:0/2844547794 >> v2:172.21.15.53:3300/0 conn(0x7f34240cf290 0x7f34240cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.819+0000 7f3421e00640 1 -- 172.21.15.53:0/2844547794 >> v1:172.21.15.146:6789/0 conn(0x7f34240b3ae0 legacy=0x7f342406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.819+0000 7f3421e00640 1 -- 172.21.15.53:0/2844547794 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f342406c470 con 0x7f34240ae5a0 2024-01-16T15:03:07.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.820+0000 7f3421e00640 1 --2- 172.21.15.53:0/2844547794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f342415e430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f34240cfe20 tx=0x7f341402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=962327eae399307a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:07.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.820+0000 7f3421400640 1 -- 172.21.15.53:0/2844547794 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3414030610 con 0x7f34240ae5a0 2024-01-16T15:03:07.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3421400640 1 -- 172.21.15.53:0/2844547794 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3414030770 con 0x7f34240ae5a0 2024-01-16T15:03:07.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3421400640 1 -- 172.21.15.53:0/2844547794 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3414030b00 con 0x7f34240ae5a0 2024-01-16T15:03:07.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 -- 172.21.15.53:0/2844547794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 msgr2=0x7f342415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 --2- 172.21.15.53:0/2844547794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f342415e430 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f34240cfe20 tx=0x7f341402b960 comp rx=0 tx=0).stop 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 -- 172.21.15.53:0/2844547794 shutdown_connections 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 --2- 172.21.15.53:0/2844547794 >> v2:172.21.15.53:3300/0 conn(0x7f34240cf290 0x7f34240cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 --2- 172.21.15.53:0/2844547794 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f342415e430 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 -- 172.21.15.53:0/2844547794 >> 172.21.15.53:0/2844547794 conn(0x7f342406a800 msgr2=0x7f342406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 -- 172.21.15.53:0/2844547794 shutdown_connections 2024-01-16T15:03:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.821+0000 7f3429200640 1 -- 172.21.15.53:0/2844547794 wait complete. 2024-01-16T15:03:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.823+0000 7f3429200640 1 Processor -- start 2024-01-16T15:03:07.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.823+0000 7f3429200640 1 -- start start 2024-01-16T15:03:07.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.824+0000 7f3429200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f34240b05b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.824+0000 7f3429200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34240b3ae0 0x7f3424158230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.824+0000 7f3429200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 0x7f342414f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.824+0000 7f3429200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34241592a0 con 0x7f34240cf290 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.824+0000 7f3429200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3424157190 con 0x7f34240ae5a0 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.824+0000 7f3429200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3424157470 con 0x7f34240b3ae0 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 0x7f342414f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3421e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f34240b05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 0x7f342414f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41390/0 (socket says 172.21.15.53:41390) 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 -- 172.21.15.53:0/252812606 learned_addr learned my addr 172.21.15.53:0/252812606 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f341be00640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34240b3ae0 0x7f3424158230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 -- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34240b3ae0 msgr2=0x7f3424158230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34240b3ae0 0x7f3424158230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 -- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 msgr2=0x7f34240b05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f34240b05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.825+0000 7f3422800640 1 -- 172.21.15.53:0/252812606 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3414009400 con 0x7f34240cf290 2024-01-16T15:03:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.826+0000 7f3422800640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 0x7f342414f230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f341c00efc0 tx=0x7f341c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.826+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c009b60 con 0x7f34240cf290 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.826+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341c00f040 con 0x7f34240cf290 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.826+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3424157750 con 0x7f34240cf290 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.826+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341c003990 con 0x7f34240cf290 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.826+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3424157d10 con 0x7f34240cf290 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.828+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34240675d0 con 0x7f34240cf290 2024-01-16T15:03:07.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.829+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f341c003af0 con 0x7f34240cf290 2024-01-16T15:03:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.829+0000 7f341a000640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33fc06b500 0x7f33fc06d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.830+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f341c08a610 con 0x7f34240cf290 2024-01-16T15:03:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.830+0000 7f3421e00640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33fc06b500 0x7f33fc06d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:07.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.832+0000 7f3421e00640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33fc06b500 0x7f33fc06d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3414036000 tx=0x7f3414008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:07.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:07.834+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f341c0550a0 con 0x7f34240cf290 2024-01-16T15:03:08.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.026+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3424157f30 con 0x7f34240cf290 2024-01-16T15:03:08.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.028+0000 7f341a000640 1 -- 172.21.15.53:0/252812606 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+16477 (secure 0 0 0) 0x7f341c0586c0 con 0x7f34240cf290 2024-01-16T15:03:08.009 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:08.009 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":47,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:03:04.720588+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:03:01.086339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2024-01-16T15:03:02.224069+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:03:08.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33fc06b500 msgr2=0x7f33fc06d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33fc06b500 0x7f33fc06d9c0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3414036000 tx=0x7f3414008030 comp rx=0 tx=0).stop 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 msgr2=0x7f342414f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 0x7f342414f230 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f341c00efc0 tx=0x7f341c00c490 comp rx=0 tx=0).stop 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 shutdown_connections 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33fc06b500 0x7f33fc06d9c0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f34240cf290 0x7f342414f230 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f34240b3ae0 0x7f3424158230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 --2- 172.21.15.53:0/252812606 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f34240ae5a0 0x7f34240b05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 >> 172.21.15.53:0/252812606 conn(0x7f342406a800 msgr2=0x7f342415d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 shutdown_connections 2024-01-16T15:03:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.031+0000 7f3429200640 1 -- 172.21.15.53:0/252812606 wait complete. 2024-01-16T15:03:08.030 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:03:08.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:03:08.106 INFO:tasks.ceph.mds.c.smithi053.stderr:2024-01-16T15:03:08.126+0000 7fcfd7200640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:03:08.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.273+0000 7fa3b8400640 1 Processor -- start 2024-01-16T15:03:08.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.273+0000 7fa3b8400640 1 -- start start 2024-01-16T15:03:08.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b8400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013df80 0x7fa3b015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b8400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3b013e360 0x7fa3b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b8400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3b006c470 con 0x7fa3b015f9c0 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b8400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3b006c750 con 0x7fa3b013e360 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b8400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b006ca30 con 0x7fa3b013df80 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3b013e360 0x7fa3b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3b013e360 0x7fa3b00cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41394/0 (socket says 172.21.15.53:41394) 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.275+0000 7fa3b5000640 1 -- 172.21.15.53:0/234520362 learned_addr learned my addr 172.21.15.53:0/234520362 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:08.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.276+0000 7fa3b5a00640 1 --2- 172.21.15.53:0/234520362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013df80 0x7fa3b015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.276+0000 7fa3b5000640 1 -- 172.21.15.53:0/234520362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013df80 msgr2=0x7fa3b015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:08.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.276+0000 7fa3b5000640 1 --2- 172.21.15.53:0/234520362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013df80 0x7fa3b015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.276+0000 7fa3b5000640 1 -- 172.21.15.53:0/234520362 >> v1:172.21.15.146:6789/0 conn(0x7fa3b015f9c0 legacy=0x7fa3b01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.276+0000 7fa3b5000640 1 -- 172.21.15.53:0/234520362 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b006cd10 con 0x7fa3b013e360 2024-01-16T15:03:08.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.277+0000 7fa3b5000640 1 --2- 172.21.15.53:0/234520362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b013e360 0x7fa3b00cf290 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67cb4ca008fbb8f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:08.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.278+0000 7fa3afe00640 1 -- 172.21.15.53:0/234520362 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398030840 con 0x7fa3b013e360 2024-01-16T15:03:08.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.278+0000 7fa3afe00640 1 -- 172.21.15.53:0/234520362 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3980309a0 con 0x7fa3b013e360 2024-01-16T15:03:08.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.278+0000 7fa3afe00640 1 -- 172.21.15.53:0/234520362 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa398030d30 con 0x7fa3b013e360 2024-01-16T15:03:08.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.278+0000 7fa3b8400640 1 -- 172.21.15.53:0/234520362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b013e360 msgr2=0x7fa3b00cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.278+0000 7fa3b8400640 1 --2- 172.21.15.53:0/234520362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b013e360 0x7fa3b00cf290 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fa398009870 tx=0x7fa39802b980 comp rx=0 tx=0).stop 2024-01-16T15:03:08.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.279+0000 7fa3b8400640 1 -- 172.21.15.53:0/234520362 shutdown_connections 2024-01-16T15:03:08.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.279+0000 7fa3b8400640 1 --2- 172.21.15.53:0/234520362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b013e360 0x7fa3b00cf290 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.279+0000 7fa3b8400640 1 --2- 172.21.15.53:0/234520362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013df80 0x7fa3b015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.279+0000 7fa3b8400640 1 -- 172.21.15.53:0/234520362 >> 172.21.15.53:0/234520362 conn(0x7fa3b006a800 msgr2=0x7fa3b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:08.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.279+0000 7fa3b8400640 1 -- 172.21.15.53:0/234520362 shutdown_connections 2024-01-16T15:03:08.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.279+0000 7fa3b8400640 1 -- 172.21.15.53:0/234520362 wait complete. 2024-01-16T15:03:08.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.280+0000 7fa3b8400640 1 Processor -- start 2024-01-16T15:03:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.281+0000 7fa3b8400640 1 -- start start 2024-01-16T15:03:08.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b8400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b013df80 0x7fa3b013b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b8400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013e360 0x7fa3b0152580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b8400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 0x7fa3b013c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b8400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b013b300 con 0x7fa3b015f9c0 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b8400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b013bcd0 con 0x7fa3b013df80 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b8400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3b013a6f0 con 0x7fa3b013e360 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b6400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 0x7fa3b013c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b6400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 0x7fa3b013c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41398/0 (socket says 172.21.15.53:41398) 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b6400640 1 -- 172.21.15.53:0/1910955360 learned_addr learned my addr 172.21.15.53:0/1910955360 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b5a00640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b013df80 0x7fa3b013b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.282+0000 7fa3b5000640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013e360 0x7fa3b0152580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b6400640 1 -- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013e360 msgr2=0x7fa3b0152580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b6400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013e360 0x7fa3b0152580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b6400640 1 -- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b013df80 msgr2=0x7fa3b013b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b6400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b013df80 0x7fa3b013b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b6400640 1 -- 172.21.15.53:0/1910955360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3980094d0 con 0x7fa3b015f9c0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b6400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 0x7fa3b013c910 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c710 tx=0x7fa3a800cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8006d50 con 0x7fa3b015f9c0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3a800ce20 con 0x7fa3b015f9c0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a8003bb0 con 0x7fa3b015f9c0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3b013a9d0 con 0x7fa3b015f9c0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.283+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3b013af90 con 0x7fa3b015f9c0 2024-01-16T15:03:08.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.285+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3b013b470 con 0x7fa3b015f9c0 2024-01-16T15:03:08.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.286+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fa3a8003d10 con 0x7fa3b015f9c0 2024-01-16T15:03:08.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.286+0000 7fa3ad600640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa38806b530 0x7fa38806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.287+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa3a8089ba0 con 0x7fa3b015f9c0 2024-01-16T15:03:08.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.287+0000 7fa3b5a00640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa38806b530 0x7fa38806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.288+0000 7fa3b5a00640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa38806b530 0x7fa38806d9f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0006fd0 tx=0x7fa3a0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:08.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.291+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3a8058130 con 0x7fa3b015f9c0 2024-01-16T15:03:08.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.485+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa3b00ce850 con 0x7fa38806b530 2024-01-16T15:03:08.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.513+0000 7fa3ad600640 1 -- 172.21.15.53:0/1910955360 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287214 (secure 0 0 0) 0x7fa3a000f860 con 0x7fa38806b530 2024-01-16T15:03:08.498 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:08.500 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa38806b530 msgr2=0x7fa38806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa38806b530 0x7fa38806d9f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa3a0006fd0 tx=0x7fa3a0008040 comp rx=0 tx=0).stop 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 msgr2=0x7fa3b013c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 0x7fa3b013c910 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fa3a800c710 tx=0x7fa3a800cbd0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 shutdown_connections 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa38806b530 0x7fa38806d9f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3b015f9c0 0x7fa3b013c910 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3b013e360 0x7fa3b0152580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 --2- 172.21.15.53:0/1910955360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3b013df80 0x7fa3b013b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 >> 172.21.15.53:0/1910955360 conn(0x7fa3b006a800 msgr2=0x7fa3b006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 shutdown_connections 2024-01-16T15:03:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.520+0000 7fa3b8400640 1 -- 172.21.15.53:0/1910955360 wait complete. 2024-01-16T15:03:08.520 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-01-16T15:03:08.241432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263212,"kb_used_data":5420,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754711508,"statfs":{"total":773094113280,"available":772824584192,"internally_reserved":0,"allocated":5550080,"data_stored":2932912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002074"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730331+0000","last_change":"2024-01-16T15:03:03.756914+0000","last_active":"2024-01-16T15:03:04.730331+0000","last_peered":"2024-01-16T15:03:04.730331+0000","last_clean":"2024-01-16T15:03:04.730331+0000","last_became_active":"2024-01-16T15:03:03.756279+0000","last_became_peered":"2024-01-16T15:03:03.756279+0000","last_unstale":"2024-01-16T15:03:04.730331+0000","last_undegraded":"2024-01-16T15:03:04.730331+0000","last_fullsized":"2024-01-16T15:03:04.730331+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:13.143628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771361+0000","last_change":"2024-01-16T15:03:03.766530+0000","last_active":"2024-01-16T15:03:03.771361+0000","last_peered":"2024-01-16T15:03:03.771361+0000","last_clean":"2024-01-16T15:03:03.771361+0000","last_became_active":"2024-01-16T15:03:03.766168+0000","last_became_peered":"2024-01-16T15:03:03.766168+0000","last_unstale":"2024-01-16T15:03:03.771361+0000","last_undegraded":"2024-01-16T15:03:03.771361+0000","last_fullsized":"2024-01-16T15:03:03.771361+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:22.071687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732234+0000","last_change":"2024-01-16T15:03:03.751679+0000","last_active":"2024-01-16T15:03:04.732234+0000","last_peered":"2024-01-16T15:03:04.732234+0000","last_clean":"2024-01-16T15:03:04.732234+0000","last_became_active":"2024-01-16T15:03:03.751372+0000","last_became_peered":"2024-01-16T15:03:03.751372+0000","last_unstale":"2024-01-16T15:03:04.732234+0000","last_undegraded":"2024-01-16T15:03:04.732234+0000","last_fullsized":"2024-01-16T15:03:04.732234+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:59.920442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771050+0000","last_change":"2024-01-16T15:03:03.766324+0000","last_active":"2024-01-16T15:03:03.771050+0000","last_peered":"2024-01-16T15:03:03.771050+0000","last_clean":"2024-01-16T15:03:03.771050+0000","last_became_active":"2024-01-16T15:03:03.765790+0000","last_became_peered":"2024-01-16T15:03:03.765790+0000","last_unstale":"2024-01-16T15:03:03.771050+0000","last_undegraded":"2024-01-16T15:03:03.771050+0000","last_fullsized":"2024-01-16T15:03:03.771050+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:42.967775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.765923+0000","last_change":"2024-01-16T15:03:03.763765+0000","last_active":"2024-01-16T15:03:03.765923+0000","last_peered":"2024-01-16T15:03:03.765923+0000","last_clean":"2024-01-16T15:03:03.765923+0000","last_became_active":"2024-01-16T15:03:03.763447+0000","last_became_peered":"2024-01-16T15:03:03.763447+0000","last_unstale":"2024-01-16T15:03:03.765923+0000","last_undegraded":"2024-01-16T15:03:03.765923+0000","last_fullsized":"2024-01-16T15:03:03.765923+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:32.400381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767198+0000","last_change":"2024-01-16T15:03:03.763085+0000","last_active":"2024-01-16T15:03:03.767198+0000","last_peered":"2024-01-16T15:03:03.767198+0000","last_clean":"2024-01-16T15:03:03.767198+0000","last_became_active":"2024-01-16T15:03:03.762031+0000","last_became_peered":"2024-01-16T15:03:03.762031+0000","last_unstale":"2024-01-16T15:03:03.767198+0000","last_undegraded":"2024-01-16T15:03:03.767198+0000","last_fullsized":"2024-01-16T15:03:03.767198+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:32.242782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729402+0000","last_change":"2024-01-16T15:03:03.749130+0000","last_active":"2024-01-16T15:03:04.729402+0000","last_peered":"2024-01-16T15:03:04.729402+0000","last_clean":"2024-01-16T15:03:04.729402+0000","last_became_active":"2024-01-16T15:03:03.748855+0000","last_became_peered":"2024-01-16T15:03:03.748855+0000","last_unstale":"2024-01-16T15:03:04.729402+0000","last_undegraded":"2024-01-16T15:03:04.729402+0000","last_fullsized":"2024-01-16T15:03:04.729402+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:38.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.769179+0000","last_change":"2024-01-16T15:03:03.763236+0000","last_active":"2024-01-16T15:03:03.769179+0000","last_peered":"2024-01-16T15:03:03.769179+0000","last_clean":"2024-01-16T15:03:03.769179+0000","last_became_active":"2024-01-16T15:03:03.762904+0000","last_became_peered":"2024-01-16T15:03:03.762904+0000","last_unstale":"2024-01-16T15:03:03.769179+0000","last_undegraded":"2024-01-16T15:03:03.769179+0000","last_fullsized":"2024-01-16T15:03:03.769179+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:46.559713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.770313+0000","last_change":"2024-01-16T15:03:03.766115+0000","last_active":"2024-01-16T15:03:03.770313+0000","last_peered":"2024-01-16T15:03:03.770313+0000","last_clean":"2024-01-16T15:03:03.770313+0000","last_became_active":"2024-01-16T15:03:03.763415+0000","last_became_peered":"2024-01-16T15:03:03.763415+0000","last_unstale":"2024-01-16T15:03:03.770313+0000","last_undegraded":"2024-01-16T15:03:03.770313+0000","last_fullsized":"2024-01-16T15:03:03.770313+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:44.557239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768521+0000","last_change":"2024-01-16T15:03:03.763133+0000","last_active":"2024-01-16T15:03:03.768521+0000","last_peered":"2024-01-16T15:03:03.768521+0000","last_clean":"2024-01-16T15:03:03.768521+0000","last_became_active":"2024-01-16T15:03:03.762833+0000","last_became_peered":"2024-01-16T15:03:03.762833+0000","last_unstale":"2024-01-16T15:03:03.768521+0000","last_undegraded":"2024-01-16T15:03:03.768521+0000","last_fullsized":"2024-01-16T15:03:03.768521+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:51.286197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768487+0000","last_change":"2024-01-16T15:03:03.762409+0000","last_active":"2024-01-16T15:03:03.768487+0000","last_peered":"2024-01-16T15:03:03.768487+0000","last_clean":"2024-01-16T15:03:03.768487+0000","last_became_active":"2024-01-16T15:03:03.762025+0000","last_became_peered":"2024-01-16T15:03:03.762025+0000","last_unstale":"2024-01-16T15:03:03.768487+0000","last_undegraded":"2024-01-16T15:03:03.768487+0000","last_fullsized":"2024-01-16T15:03:03.768487+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:48.563036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.752461+0000","last_change":"2024-01-16T15:03:03.747009+0000","last_active":"2024-01-16T15:03:03.752461+0000","last_peered":"2024-01-16T15:03:03.752461+0000","last_clean":"2024-01-16T15:03:03.752461+0000","last_became_active":"2024-01-16T15:03:03.746345+0000","last_became_peered":"2024-01-16T15:03:03.746345+0000","last_unstale":"2024-01-16T15:03:03.752461+0000","last_undegraded":"2024-01-16T15:03:03.752461+0000","last_fullsized":"2024-01-16T15:03:03.752461+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:42.967775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027268+0000","last_change":"2024-01-16T15:03:03.761513+0000","last_active":"2024-01-16T15:03:05.027268+0000","last_peered":"2024-01-16T15:03:05.027268+0000","last_clean":"2024-01-16T15:03:05.027268+0000","last_became_active":"2024-01-16T15:03:03.761220+0000","last_became_peered":"2024-01-16T15:03:03.761220+0000","last_unstale":"2024-01-16T15:03:05.027268+0000","last_undegraded":"2024-01-16T15:03:05.027268+0000","last_fullsized":"2024-01-16T15:03:05.027268+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:42.829656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.765131+0000","last_change":"2024-01-16T15:03:03.750530+0000","last_active":"2024-01-16T15:03:03.765131+0000","last_peered":"2024-01-16T15:03:03.765131+0000","last_clean":"2024-01-16T15:03:03.765131+0000","last_became_active":"2024-01-16T15:03:03.747908+0000","last_became_peered":"2024-01-16T15:03:03.747908+0000","last_unstale":"2024-01-16T15:03:03.765131+0000","last_undegraded":"2024-01-16T15:03:03.765131+0000","last_fullsized":"2024-01-16T15:03:03.765131+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:32:14.533934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.762656+0000","last_change":"2024-01-16T15:03:03.757148+0000","last_active":"2024-01-16T15:03:03.762656+0000","last_peered":"2024-01-16T15:03:03.762656+0000","last_clean":"2024-01-16T15:03:03.762656+0000","last_became_active":"2024-01-16T15:03:03.755613+0000","last_became_peered":"2024-01-16T15:03:03.755613+0000","last_unstale":"2024-01-16T15:03:03.762656+0000","last_undegraded":"2024-01-16T15:03:03.762656+0000","last_fullsized":"2024-01-16T15:03:03.762656+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:05.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524458+0000","last_change":"2024-01-16T15:03:03.767628+0000","last_active":"2024-01-16T15:03:05.524458+0000","last_peered":"2024-01-16T15:03:05.524458+0000","last_clean":"2024-01-16T15:03:05.524458+0000","last_became_active":"2024-01-16T15:03:03.767293+0000","last_became_peered":"2024-01-16T15:03:03.767293+0000","last_unstale":"2024-01-16T15:03:05.524458+0000","last_undegraded":"2024-01-16T15:03:05.524458+0000","last_fullsized":"2024-01-16T15:03:05.524458+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:53.126320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733406+0000","last_change":"2024-01-16T15:03:03.753799+0000","last_active":"2024-01-16T15:03:04.733406+0000","last_peered":"2024-01-16T15:03:04.733406+0000","last_clean":"2024-01-16T15:03:04.733406+0000","last_became_active":"2024-01-16T15:03:03.753494+0000","last_became_peered":"2024-01-16T15:03:03.753494+0000","last_unstale":"2024-01-16T15:03:04.733406+0000","last_undegraded":"2024-01-16T15:03:04.733406+0000","last_fullsized":"2024-01-16T15:03:04.733406+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:14.400108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732937+0000","last_change":"2024-01-16T15:03:03.769228+0000","last_active":"2024-01-16T15:03:04.732937+0000","last_peered":"2024-01-16T15:03:04.732937+0000","last_clean":"2024-01-16T15:03:04.732937+0000","last_became_active":"2024-01-16T15:03:03.768967+0000","last_became_peered":"2024-01-16T15:03:03.768967+0000","last_unstale":"2024-01-16T15:03:04.732937+0000","last_undegraded":"2024-01-16T15:03:04.732937+0000","last_fullsized":"2024-01-16T15:03:04.732937+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:42.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525793+0000","last_change":"2024-01-16T15:03:03.770152+0000","last_active":"2024-01-16T15:03:05.525793+0000","last_peered":"2024-01-16T15:03:05.525793+0000","last_clean":"2024-01-16T15:03:05.525793+0000","last_became_active":"2024-01-16T15:03:03.769919+0000","last_became_peered":"2024-01-16T15:03:03.769919+0000","last_unstale":"2024-01-16T15:03:05.525793+0000","last_undegraded":"2024-01-16T15:03:05.525793+0000","last_fullsized":"2024-01-16T15:03:05.525793+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:15.453517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027392+0000","last_change":"2024-01-16T15:03:03.769582+0000","last_active":"2024-01-16T15:03:05.027392+0000","last_peered":"2024-01-16T15:03:05.027392+0000","last_clean":"2024-01-16T15:03:05.027392+0000","last_became_active":"2024-01-16T15:03:03.768731+0000","last_became_peered":"2024-01-16T15:03:03.768731+0000","last_unstale":"2024-01-16T15:03:05.027392+0000","last_undegraded":"2024-01-16T15:03:05.027392+0000","last_fullsized":"2024-01-16T15:03:05.027392+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:17.546707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.762982+0000","last_change":"2024-01-16T15:03:03.760051+0000","last_active":"2024-01-16T15:03:03.762982+0000","last_peered":"2024-01-16T15:03:03.762982+0000","last_clean":"2024-01-16T15:03:03.762982+0000","last_became_active":"2024-01-16T15:03:03.759757+0000","last_became_peered":"2024-01-16T15:03:03.759757+0000","last_unstale":"2024-01-16T15:03:03.762982+0000","last_undegraded":"2024-01-16T15:03:03.762982+0000","last_fullsized":"2024-01-16T15:03:03.762982+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:09.541105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768960+0000","last_change":"2024-01-16T15:03:03.757236+0000","last_active":"2024-01-16T15:03:03.768960+0000","last_peered":"2024-01-16T15:03:03.768960+0000","last_clean":"2024-01-16T15:03:03.768960+0000","last_became_active":"2024-01-16T15:03:03.756941+0000","last_became_peered":"2024-01-16T15:03:03.756941+0000","last_unstale":"2024-01-16T15:03:03.768960+0000","last_undegraded":"2024-01-16T15:03:03.768960+0000","last_fullsized":"2024-01-16T15:03:03.768960+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:35.561556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524811+0000","last_change":"2024-01-16T15:03:03.758872+0000","last_active":"2024-01-16T15:03:05.524811+0000","last_peered":"2024-01-16T15:03:05.524811+0000","last_clean":"2024-01-16T15:03:05.524811+0000","last_became_active":"2024-01-16T15:03:03.758569+0000","last_became_peered":"2024-01-16T15:03:03.758569+0000","last_unstale":"2024-01-16T15:03:05.524811+0000","last_undegraded":"2024-01-16T15:03:05.524811+0000","last_fullsized":"2024-01-16T15:03:05.524811+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:56.907068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524429+0000","last_change":"2024-01-16T15:03:03.758659+0000","last_active":"2024-01-16T15:03:05.524429+0000","last_peered":"2024-01-16T15:03:05.524429+0000","last_clean":"2024-01-16T15:03:05.524429+0000","last_became_active":"2024-01-16T15:03:03.758390+0000","last_became_peered":"2024-01-16T15:03:03.758390+0000","last_unstale":"2024-01-16T15:03:05.524429+0000","last_undegraded":"2024-01-16T15:03:05.524429+0000","last_fullsized":"2024-01-16T15:03:05.524429+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:23.546151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525471+0000","last_change":"2024-01-16T15:03:03.763284+0000","last_active":"2024-01-16T15:03:05.525471+0000","last_peered":"2024-01-16T15:03:05.525471+0000","last_clean":"2024-01-16T15:03:05.525471+0000","last_became_active":"2024-01-16T15:03:03.762944+0000","last_became_peered":"2024-01-16T15:03:03.762944+0000","last_unstale":"2024-01-16T15:03:05.525471+0000","last_undegraded":"2024-01-16T15:03:05.525471+0000","last_fullsized":"2024-01-16T15:03:05.525471+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:13.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027187+0000","last_change":"2024-01-16T15:03:03.757757+0000","last_active":"2024-01-16T15:03:05.027187+0000","last_peered":"2024-01-16T15:03:05.027187+0000","last_clean":"2024-01-16T15:03:05.027187+0000","last_became_active":"2024-01-16T15:03:03.757467+0000","last_became_peered":"2024-01-16T15:03:03.757467+0000","last_unstale":"2024-01-16T15:03:05.027187+0000","last_undegraded":"2024-01-16T15:03:05.027187+0000","last_fullsized":"2024-01-16T15:03:05.027187+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:54.306598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732313+0000","last_change":"2024-01-16T15:03:03.755329+0000","last_active":"2024-01-16T15:03:04.732313+0000","last_peered":"2024-01-16T15:03:04.732313+0000","last_clean":"2024-01-16T15:03:04.732313+0000","last_became_active":"2024-01-16T15:03:03.753828+0000","last_became_peered":"2024-01-16T15:03:03.753828+0000","last_unstale":"2024-01-16T15:03:04.732313+0000","last_undegraded":"2024-01-16T15:03:04.732313+0000","last_fullsized":"2024-01-16T15:03:04.732313+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:33.207831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.752479+0000","last_change":"2024-01-16T15:03:03.747091+0000","last_active":"2024-01-16T15:03:03.752479+0000","last_peered":"2024-01-16T15:03:03.752479+0000","last_clean":"2024-01-16T15:03:03.752479+0000","last_became_active":"2024-01-16T15:03:03.746793+0000","last_became_peered":"2024-01-16T15:03:03.746793+0000","last_unstale":"2024-01-16T15:03:03.752479+0000","last_undegraded":"2024-01-16T15:03:03.752479+0000","last_fullsized":"2024-01-16T15:03:03.752479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:51.286197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524816+0000","last_change":"2024-01-16T15:03:03.756962+0000","last_active":"2024-01-16T15:03:05.524816+0000","last_peered":"2024-01-16T15:03:05.524816+0000","last_clean":"2024-01-16T15:03:05.524816+0000","last_became_active":"2024-01-16T15:03:03.756513+0000","last_became_peered":"2024-01-16T15:03:03.756513+0000","last_unstale":"2024-01-16T15:03:05.524816+0000","last_undegraded":"2024-01-16T15:03:05.524816+0000","last_fullsized":"2024-01-16T15:03:05.524816+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:34.124804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732216+0000","last_change":"2024-01-16T15:03:03.750861+0000","last_active":"2024-01-16T15:03:04.732216+0000","last_peered":"2024-01-16T15:03:04.732216+0000","last_clean":"2024-01-16T15:03:04.732216+0000","last_became_active":"2024-01-16T15:03:03.750493+0000","last_became_peered":"2024-01-16T15:03:03.750493+0000","last_unstale":"2024-01-16T15:03:04.732216+0000","last_undegraded":"2024-01-16T15:03:04.732216+0000","last_fullsized":"2024-01-16T15:03:04.732216+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:05.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.758833+0000","last_change":"2024-01-16T15:03:03.754509+0000","last_active":"2024-01-16T15:03:03.758833+0000","last_peered":"2024-01-16T15:03:03.758833+0000","last_clean":"2024-01-16T15:03:03.758833+0000","last_became_active":"2024-01-16T15:03:03.754199+0000","last_became_peered":"2024-01-16T15:03:03.754199+0000","last_unstale":"2024-01-16T15:03:03.758833+0000","last_undegraded":"2024-01-16T15:03:03.758833+0000","last_fullsized":"2024-01-16T15:03:03.758833+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:56.424560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027343+0000","last_change":"2024-01-16T15:03:03.748787+0000","last_active":"2024-01-16T15:03:05.027343+0000","last_peered":"2024-01-16T15:03:05.027343+0000","last_clean":"2024-01-16T15:03:05.027343+0000","last_became_active":"2024-01-16T15:03:03.747052+0000","last_became_peered":"2024-01-16T15:03:03.747052+0000","last_unstale":"2024-01-16T15:03:05.027343+0000","last_undegraded":"2024-01-16T15:03:05.027343+0000","last_fullsized":"2024-01-16T15:03:05.027343+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:58.887884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.762491+0000","last_change":"2024-01-16T15:03:03.755138+0000","last_active":"2024-01-16T15:03:03.762491+0000","last_peered":"2024-01-16T15:03:03.762491+0000","last_clean":"2024-01-16T15:03:03.762491+0000","last_became_active":"2024-01-16T15:03:03.754647+0000","last_became_peered":"2024-01-16T15:03:03.754647+0000","last_unstale":"2024-01-16T15:03:03.762491+0000","last_undegraded":"2024-01-16T15:03:03.762491+0000","last_fullsized":"2024-01-16T15:03:03.762491+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:23.421216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"46'4","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.074382+0000","last_change":"2024-01-16T15:03:02.747015+0000","last_active":"2024-01-16T15:03:04.074382+0000","last_peered":"2024-01-16T15:03:04.074382+0000","last_clean":"2024-01-16T15:03:04.074382+0000","last_became_active":"2024-01-16T15:03:02.746705+0000","last_became_peered":"2024-01-16T15:03:02.746705+0000","last_unstale":"2024-01-16T15:03:04.074382+0000","last_undegraded":"2024-01-16T15:03:04.074382+0000","last_fullsized":"2024-01-16T15:03:04.074382+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:42:44.105957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525709+0000","last_change":"2024-01-16T15:03:03.761945+0000","last_active":"2024-01-16T15:03:05.525709+0000","last_peered":"2024-01-16T15:03:05.525709+0000","last_clean":"2024-01-16T15:03:05.525709+0000","last_became_active":"2024-01-16T15:03:03.761691+0000","last_became_peered":"2024-01-16T15:03:03.761691+0000","last_unstale":"2024-01-16T15:03:05.525709+0000","last_undegraded":"2024-01-16T15:03:05.525709+0000","last_fullsized":"2024-01-16T15:03:05.525709+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:36.354147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733081+0000","last_change":"2024-01-16T15:03:02.744417+0000","last_active":"2024-01-16T15:03:04.733081+0000","last_peered":"2024-01-16T15:03:04.733081+0000","last_clean":"2024-01-16T15:03:04.733081+0000","last_became_active":"2024-01-16T15:03:02.744117+0000","last_became_peered":"2024-01-16T15:03:02.744117+0000","last_unstale":"2024-01-16T15:03:04.733081+0000","last_undegraded":"2024-01-16T15:03:04.733081+0000","last_fullsized":"2024-01-16T15:03:04.733081+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:54:23.826072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768169+0000","last_change":"2024-01-16T15:03:03.764508+0000","last_active":"2024-01-16T15:03:03.768169+0000","last_peered":"2024-01-16T15:03:03.768169+0000","last_clean":"2024-01-16T15:03:03.768169+0000","last_became_active":"2024-01-16T15:03:03.764247+0000","last_became_peered":"2024-01-16T15:03:03.764247+0000","last_unstale":"2024-01-16T15:03:03.768169+0000","last_undegraded":"2024-01-16T15:03:03.768169+0000","last_fullsized":"2024-01-16T15:03:03.768169+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:56.336015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.077003+0000","last_change":"2024-01-16T15:03:02.738219+0000","last_active":"2024-01-16T15:03:04.077003+0000","last_peered":"2024-01-16T15:03:04.077003+0000","last_clean":"2024-01-16T15:03:04.077003+0000","last_became_active":"2024-01-16T15:03:02.737153+0000","last_became_peered":"2024-01-16T15:03:02.737153+0000","last_unstale":"2024-01-16T15:03:04.077003+0000","last_undegraded":"2024-01-16T15:03:04.077003+0000","last_fullsized":"2024-01-16T15:03:04.077003+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:55.319844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.764636+0000","last_change":"2024-01-16T15:03:03.756833+0000","last_active":"2024-01-16T15:03:03.764636+0000","last_peered":"2024-01-16T15:03:03.764636+0000","last_clean":"2024-01-16T15:03:03.764636+0000","last_became_active":"2024-01-16T15:03:03.756549+0000","last_became_peered":"2024-01-16T15:03:03.756549+0000","last_unstale":"2024-01-16T15:03:03.764636+0000","last_undegraded":"2024-01-16T15:03:03.764636+0000","last_fullsized":"2024-01-16T15:03:03.764636+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:13.357896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729953+0000","last_change":"2024-01-16T15:03:02.739810+0000","last_active":"2024-01-16T15:03:04.729953+0000","last_peered":"2024-01-16T15:03:04.729953+0000","last_clean":"2024-01-16T15:03:04.729953+0000","last_became_active":"2024-01-16T15:03:02.739459+0000","last_became_peered":"2024-01-16T15:03:02.739459+0000","last_unstale":"2024-01-16T15:03:04.729953+0000","last_undegraded":"2024-01-16T15:03:04.729953+0000","last_fullsized":"2024-01-16T15:03:04.729953+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.046251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732573+0000","last_change":"2024-01-16T15:03:03.756387+0000","last_active":"2024-01-16T15:03:04.732573+0000","last_peered":"2024-01-16T15:03:04.732573+0000","last_clean":"2024-01-16T15:03:04.732573+0000","last_became_active":"2024-01-16T15:03:03.756069+0000","last_became_peered":"2024-01-16T15:03:03.756069+0000","last_unstale":"2024-01-16T15:03:04.732573+0000","last_undegraded":"2024-01-16T15:03:04.732573+0000","last_fullsized":"2024-01-16T15:03:04.732573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:16.134109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"46'1","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.072859+0000","last_change":"2024-01-16T15:03:02.745265+0000","last_active":"2024-01-16T15:03:04.072859+0000","last_peered":"2024-01-16T15:03:04.072859+0000","last_clean":"2024-01-16T15:03:04.072859+0000","last_became_active":"2024-01-16T15:03:02.743903+0000","last_became_peered":"2024-01-16T15:03:02.743903+0000","last_unstale":"2024-01-16T15:03:04.072859+0000","last_undegraded":"2024-01-16T15:03:04.072859+0000","last_fullsized":"2024-01-16T15:03:04.072859+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:13.383937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771059+0000","last_change":"2024-01-16T15:03:03.766518+0000","last_active":"2024-01-16T15:03:03.771059+0000","last_peered":"2024-01-16T15:03:03.771059+0000","last_clean":"2024-01-16T15:03:03.771059+0000","last_became_active":"2024-01-16T15:03:03.765246+0000","last_became_peered":"2024-01-16T15:03:03.765246+0000","last_unstale":"2024-01-16T15:03:03.771059+0000","last_undegraded":"2024-01-16T15:03:03.771059+0000","last_fullsized":"2024-01-16T15:03:03.771059+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:22.071687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732666+0000","last_change":"2024-01-16T15:03:02.738149+0000","last_active":"2024-01-16T15:03:04.732666+0000","last_peered":"2024-01-16T15:03:04.732666+0000","last_clean":"2024-01-16T15:03:04.732666+0000","last_became_active":"2024-01-16T15:03:02.736301+0000","last_became_peered":"2024-01-16T15:03:02.736301+0000","last_unstale":"2024-01-16T15:03:04.732666+0000","last_undegraded":"2024-01-16T15:03:04.732666+0000","last_fullsized":"2024-01-16T15:03:04.732666+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:22.204469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.769161+0000","last_change":"2024-01-16T15:03:03.758057+0000","last_active":"2024-01-16T15:03:03.769161+0000","last_peered":"2024-01-16T15:03:03.769161+0000","last_clean":"2024-01-16T15:03:03.769161+0000","last_became_active":"2024-01-16T15:03:03.757782+0000","last_became_peered":"2024-01-16T15:03:03.757782+0000","last_unstale":"2024-01-16T15:03:03.769161+0000","last_undegraded":"2024-01-16T15:03:03.769161+0000","last_fullsized":"2024-01-16T15:03:03.769161+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:33.626652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.729004+0000","last_change":"2024-01-16T15:03:02.744706+0000","last_active":"2024-01-16T15:03:03.729004+0000","last_peered":"2024-01-16T15:03:03.729004+0000","last_clean":"2024-01-16T15:03:03.729004+0000","last_became_active":"2024-01-16T15:03:02.743633+0000","last_became_peered":"2024-01-16T15:03:02.743633+0000","last_unstale":"2024-01-16T15:03:03.729004+0000","last_undegraded":"2024-01-16T15:03:03.729004+0000","last_fullsized":"2024-01-16T15:03:03.729004+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:05:28.811804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525178+0000","last_change":"2024-01-16T15:03:03.758610+0000","last_active":"2024-01-16T15:03:05.525178+0000","last_peered":"2024-01-16T15:03:05.525178+0000","last_clean":"2024-01-16T15:03:05.525178+0000","last_became_active":"2024-01-16T15:03:03.758351+0000","last_became_peered":"2024-01-16T15:03:03.758351+0000","last_unstale":"2024-01-16T15:03:05.525178+0000","last_undegraded":"2024-01-16T15:03:05.525178+0000","last_fullsized":"2024-01-16T15:03:05.525178+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:05:42.600626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.743994+0000","last_change":"2024-01-16T15:03:02.745507+0000","last_active":"2024-01-16T15:03:03.743994+0000","last_peered":"2024-01-16T15:03:03.743994+0000","last_clean":"2024-01-16T15:03:03.743994+0000","last_became_active":"2024-01-16T15:03:02.745183+0000","last_became_peered":"2024-01-16T15:03:02.745183+0000","last_unstale":"2024-01-16T15:03:03.743994+0000","last_undegraded":"2024-01-16T15:03:03.743994+0000","last_fullsized":"2024-01-16T15:03:03.743994+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:12.341724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733064+0000","last_change":"2024-01-16T15:03:03.767005+0000","last_active":"2024-01-16T15:03:04.733064+0000","last_peered":"2024-01-16T15:03:04.733064+0000","last_clean":"2024-01-16T15:03:04.733064+0000","last_became_active":"2024-01-16T15:03:03.766717+0000","last_became_peered":"2024-01-16T15:03:03.766717+0000","last_unstale":"2024-01-16T15:03:04.733064+0000","last_undegraded":"2024-01-16T15:03:04.733064+0000","last_fullsized":"2024-01-16T15:03:04.733064+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:09.541105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733498+0000","last_change":"2024-01-16T15:03:02.743197+0000","last_active":"2024-01-16T15:03:04.733498+0000","last_peered":"2024-01-16T15:03:04.733498+0000","last_clean":"2024-01-16T15:03:04.733498+0000","last_became_active":"2024-01-16T15:03:02.740663+0000","last_became_peered":"2024-01-16T15:03:02.740663+0000","last_unstale":"2024-01-16T15:03:04.733498+0000","last_undegraded":"2024-01-16T15:03:04.733498+0000","last_fullsized":"2024-01-16T15:03:04.733498+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:14.437345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730675+0000","last_change":"2024-01-16T15:03:03.750469+0000","last_active":"2024-01-16T15:03:04.730675+0000","last_peered":"2024-01-16T15:03:04.730675+0000","last_clean":"2024-01-16T15:03:04.730675+0000","last_became_active":"2024-01-16T15:03:03.750080+0000","last_became_peered":"2024-01-16T15:03:03.750080+0000","last_unstale":"2024-01-16T15:03:04.730675+0000","last_undegraded":"2024-01-16T15:03:04.730675+0000","last_fullsized":"2024-01-16T15:03:04.730675+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:24.974640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.751476+0000","last_change":"2024-01-16T15:03:02.742151+0000","last_active":"2024-01-16T15:03:03.751476+0000","last_peered":"2024-01-16T15:03:03.751476+0000","last_clean":"2024-01-16T15:03:03.751476+0000","last_became_active":"2024-01-16T15:03:02.741865+0000","last_became_peered":"2024-01-16T15:03:02.741865+0000","last_unstale":"2024-01-16T15:03:03.751476+0000","last_undegraded":"2024-01-16T15:03:03.751476+0000","last_fullsized":"2024-01-16T15:03:03.751476+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:45.978118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.770007+0000","last_change":"2024-01-16T15:03:03.764762+0000","last_active":"2024-01-16T15:03:03.770007+0000","last_peered":"2024-01-16T15:03:03.770007+0000","last_clean":"2024-01-16T15:03:03.770007+0000","last_became_active":"2024-01-16T15:03:03.764485+0000","last_became_peered":"2024-01-16T15:03:03.764485+0000","last_unstale":"2024-01-16T15:03:03.770007+0000","last_undegraded":"2024-01-16T15:03:03.770007+0000","last_fullsized":"2024-01-16T15:03:03.770007+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:35.292757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"46'3","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730591+0000","last_change":"2024-01-16T15:03:02.746772+0000","last_active":"2024-01-16T15:03:04.730591+0000","last_peered":"2024-01-16T15:03:04.730591+0000","last_clean":"2024-01-16T15:03:04.730591+0000","last_became_active":"2024-01-16T15:03:02.746453+0000","last_became_peered":"2024-01-16T15:03:02.746453+0000","last_unstale":"2024-01-16T15:03:04.730591+0000","last_undegraded":"2024-01-16T15:03:04.730591+0000","last_fullsized":"2024-01-16T15:03:04.730591+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:41.169885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730362+0000","last_change":"2024-01-16T15:03:03.759965+0000","last_active":"2024-01-16T15:03:04.730362+0000","last_peered":"2024-01-16T15:03:04.730362+0000","last_clean":"2024-01-16T15:03:04.730362+0000","last_became_active":"2024-01-16T15:03:03.759090+0000","last_became_peered":"2024-01-16T15:03:03.759090+0000","last_unstale":"2024-01-16T15:03:04.730362+0000","last_undegraded":"2024-01-16T15:03:04.730362+0000","last_fullsized":"2024-01-16T15:03:04.730362+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:13.538220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"46'6","reported_seq":18,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027436+0000","last_change":"2024-01-16T15:03:02.738505+0000","last_active":"2024-01-16T15:03:05.027436+0000","last_peered":"2024-01-16T15:03:05.027436+0000","last_clean":"2024-01-16T15:03:05.027436+0000","last_became_active":"2024-01-16T15:03:02.738235+0000","last_became_peered":"2024-01-16T15:03:02.738235+0000","last_unstale":"2024-01-16T15:03:05.027436+0000","last_undegraded":"2024-01-16T15:03:05.027436+0000","last_fullsized":"2024-01-16T15:03:05.027436+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:03:51.358036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525814+0000","last_change":"2024-01-16T15:03:03.768539+0000","last_active":"2024-01-16T15:03:05.525814+0000","last_peered":"2024-01-16T15:03:05.525814+0000","last_clean":"2024-01-16T15:03:05.525814+0000","last_became_active":"2024-01-16T15:03:03.768201+0000","last_became_peered":"2024-01-16T15:03:03.768201+0000","last_unstale":"2024-01-16T15:03:05.525814+0000","last_undegraded":"2024-01-16T15:03:05.525814+0000","last_fullsized":"2024-01-16T15:03:05.525814+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:23.220641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.069124+0000","last_change":"2024-01-16T15:03:02.750156+0000","last_active":"2024-01-16T15:03:04.069124+0000","last_peered":"2024-01-16T15:03:04.069124+0000","last_clean":"2024-01-16T15:03:04.069124+0000","last_became_active":"2024-01-16T15:03:02.749815+0000","last_became_peered":"2024-01-16T15:03:02.749815+0000","last_unstale":"2024-01-16T15:03:04.069124+0000","last_undegraded":"2024-01-16T15:03:04.069124+0000","last_fullsized":"2024-01-16T15:03:04.069124+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:09.262264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768603+0000","last_change":"2024-01-16T15:03:03.764953+0000","last_active":"2024-01-16T15:03:03.768603+0000","last_peered":"2024-01-16T15:03:03.768603+0000","last_clean":"2024-01-16T15:03:03.768603+0000","last_became_active":"2024-01-16T15:03:03.756011+0000","last_became_peered":"2024-01-16T15:03:03.756011+0000","last_unstale":"2024-01-16T15:03:03.768603+0000","last_undegraded":"2024-01-16T15:03:03.768603+0000","last_fullsized":"2024-01-16T15:03:03.768603+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:57.932933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729552+0000","last_change":"2024-01-16T15:03:02.737611+0000","last_active":"2024-01-16T15:03:04.729552+0000","last_peered":"2024-01-16T15:03:04.729552+0000","last_clean":"2024-01-16T15:03:04.729552+0000","last_became_active":"2024-01-16T15:03:02.737290+0000","last_became_peered":"2024-01-16T15:03:02.737290+0000","last_unstale":"2024-01-16T15:03:04.729552+0000","last_undegraded":"2024-01-16T15:03:04.729552+0000","last_fullsized":"2024-01-16T15:03:04.729552+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:56.916762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767418+0000","last_change":"2024-01-16T15:03:03.762312+0000","last_active":"2024-01-16T15:03:03.767418+0000","last_peered":"2024-01-16T15:03:03.767418+0000","last_clean":"2024-01-16T15:03:03.767418+0000","last_became_active":"2024-01-16T15:03:03.762036+0000","last_became_peered":"2024-01-16T15:03:03.762036+0000","last_unstale":"2024-01-16T15:03:03.767418+0000","last_undegraded":"2024-01-16T15:03:03.767418+0000","last_fullsized":"2024-01-16T15:03:03.767418+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:23.220641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.742331+0000","last_change":"2024-01-16T15:03:02.749895+0000","last_active":"2024-01-16T15:03:03.742331+0000","last_peered":"2024-01-16T15:03:03.742331+0000","last_clean":"2024-01-16T15:03:03.742331+0000","last_became_active":"2024-01-16T15:03:02.749499+0000","last_became_peered":"2024-01-16T15:03:02.749499+0000","last_unstale":"2024-01-16T15:03:03.742331+0000","last_undegraded":"2024-01-16T15:03:03.742331+0000","last_fullsized":"2024-01-16T15:03:03.742331+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:58.904271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729443+0000","last_change":"2024-01-16T15:03:03.761477+0000","last_active":"2024-01-16T15:03:04.729443+0000","last_peered":"2024-01-16T15:03:04.729443+0000","last_clean":"2024-01-16T15:03:04.729443+0000","last_became_active":"2024-01-16T15:03:03.761137+0000","last_became_peered":"2024-01-16T15:03:03.761137+0000","last_unstale":"2024-01-16T15:03:04.729443+0000","last_undegraded":"2024-01-16T15:03:04.729443+0000","last_fullsized":"2024-01-16T15:03:04.729443+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:56:58.032962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"46'8","reported_seq":19,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.076906+0000","last_change":"2024-01-16T15:03:02.751726+0000","last_active":"2024-01-16T15:03:04.076906+0000","last_peered":"2024-01-16T15:03:04.076906+0000","last_clean":"2024-01-16T15:03:04.076906+0000","last_became_active":"2024-01-16T15:03:02.751314+0000","last_became_peered":"2024-01-16T15:03:02.751314+0000","last_unstale":"2024-01-16T15:03:04.076906+0000","last_undegraded":"2024-01-16T15:03:04.076906+0000","last_fullsized":"2024-01-16T15:03:04.076906+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:23.092743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027379+0000","last_change":"2024-01-16T15:03:03.737335+0000","last_active":"2024-01-16T15:03:05.027379+0000","last_peered":"2024-01-16T15:03:05.027379+0000","last_clean":"2024-01-16T15:03:05.027379+0000","last_became_active":"2024-01-16T15:03:03.736322+0000","last_became_peered":"2024-01-16T15:03:03.736322+0000","last_unstale":"2024-01-16T15:03:05.027379+0000","last_undegraded":"2024-01-16T15:03:05.027379+0000","last_fullsized":"2024-01-16T15:03:05.027379+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:41.092236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027375+0000","last_change":"2024-01-16T15:03:02.741842+0000","last_active":"2024-01-16T15:03:05.027375+0000","last_peered":"2024-01-16T15:03:05.027375+0000","last_clean":"2024-01-16T15:03:05.027375+0000","last_became_active":"2024-01-16T15:03:02.740116+0000","last_became_peered":"2024-01-16T15:03:02.740116+0000","last_unstale":"2024-01-16T15:03:05.027375+0000","last_undegraded":"2024-01-16T15:03:05.027375+0000","last_fullsized":"2024-01-16T15:03:05.027375+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:08.333486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768188+0000","last_change":"2024-01-16T15:03:03.756583+0000","last_active":"2024-01-16T15:03:03.768188+0000","last_peered":"2024-01-16T15:03:03.768188+0000","last_clean":"2024-01-16T15:03:03.768188+0000","last_became_active":"2024-01-16T15:03:03.756248+0000","last_became_peered":"2024-01-16T15:03:03.756248+0000","last_unstale":"2024-01-16T15:03:03.768188+0000","last_undegraded":"2024-01-16T15:03:03.768188+0000","last_fullsized":"2024-01-16T15:03:03.768188+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:48.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733457+0000","last_change":"2024-01-16T15:03:02.743957+0000","last_active":"2024-01-16T15:03:04.733457+0000","last_peered":"2024-01-16T15:03:04.733457+0000","last_clean":"2024-01-16T15:03:04.733457+0000","last_became_active":"2024-01-16T15:03:02.743681+0000","last_became_peered":"2024-01-16T15:03:02.743681+0000","last_unstale":"2024-01-16T15:03:04.733457+0000","last_undegraded":"2024-01-16T15:03:04.733457+0000","last_fullsized":"2024-01-16T15:03:04.733457+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:12.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525102+0000","last_change":"2024-01-16T15:03:03.770866+0000","last_active":"2024-01-16T15:03:05.525102+0000","last_peered":"2024-01-16T15:03:05.525102+0000","last_clean":"2024-01-16T15:03:05.525102+0000","last_became_active":"2024-01-16T15:03:03.770561+0000","last_became_peered":"2024-01-16T15:03:03.770561+0000","last_unstale":"2024-01-16T15:03:05.525102+0000","last_undegraded":"2024-01-16T15:03:05.525102+0000","last_fullsized":"2024-01-16T15:03:05.525102+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:18.862583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"46'1","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.076481+0000","last_change":"2024-01-16T15:03:02.742090+0000","last_active":"2024-01-16T15:03:04.076481+0000","last_peered":"2024-01-16T15:03:04.076481+0000","last_clean":"2024-01-16T15:03:04.076481+0000","last_became_active":"2024-01-16T15:03:02.741828+0000","last_became_peered":"2024-01-16T15:03:02.741828+0000","last_unstale":"2024-01-16T15:03:04.076481+0000","last_undegraded":"2024-01-16T15:03:04.076481+0000","last_fullsized":"2024-01-16T15:03:04.076481+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:56.916762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525257+0000","last_change":"2024-01-16T15:03:03.763483+0000","last_active":"2024-01-16T15:03:05.525257+0000","last_peered":"2024-01-16T15:03:05.525257+0000","last_clean":"2024-01-16T15:03:05.525257+0000","last_became_active":"2024-01-16T15:03:03.762707+0000","last_became_peered":"2024-01-16T15:03:03.762707+0000","last_unstale":"2024-01-16T15:03:05.525257+0000","last_undegraded":"2024-01-16T15:03:05.525257+0000","last_fullsized":"2024-01-16T15:03:05.525257+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:54:24.842244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.755594+0000","last_change":"2024-01-16T15:03:02.744774+0000","last_active":"2024-01-16T15:03:03.755594+0000","last_peered":"2024-01-16T15:03:03.755594+0000","last_clean":"2024-01-16T15:03:03.755594+0000","last_became_active":"2024-01-16T15:03:02.744496+0000","last_became_peered":"2024-01-16T15:03:02.744496+0000","last_unstale":"2024-01-16T15:03:03.755594+0000","last_undegraded":"2024-01-16T15:03:03.755594+0000","last_fullsized":"2024-01-16T15:03:03.755594+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:44.812709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767226+0000","last_change":"2024-01-16T15:03:03.763309+0000","last_active":"2024-01-16T15:03:03.767226+0000","last_peered":"2024-01-16T15:03:03.767226+0000","last_clean":"2024-01-16T15:03:03.767226+0000","last_became_active":"2024-01-16T15:03:03.762966+0000","last_became_peered":"2024-01-16T15:03:03.762966+0000","last_unstale":"2024-01-16T15:03:03.767226+0000","last_undegraded":"2024-01-16T15:03:03.767226+0000","last_fullsized":"2024-01-16T15:03:03.767226+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:16.134109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.073247+0000","last_change":"2024-01-16T15:03:02.746052+0000","last_active":"2024-01-16T15:03:04.073247+0000","last_peered":"2024-01-16T15:03:04.073247+0000","last_clean":"2024-01-16T15:03:04.073247+0000","last_became_active":"2024-01-16T15:03:02.744858+0000","last_became_peered":"2024-01-16T15:03:02.744858+0000","last_unstale":"2024-01-16T15:03:04.073247+0000","last_undegraded":"2024-01-16T15:03:04.073247+0000","last_fullsized":"2024-01-16T15:03:04.073247+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:00.408282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.758276+0000","last_change":"2024-01-16T15:03:03.753202+0000","last_active":"2024-01-16T15:03:03.758276+0000","last_peered":"2024-01-16T15:03:03.758276+0000","last_clean":"2024-01-16T15:03:03.758276+0000","last_became_active":"2024-01-16T15:03:03.752916+0000","last_became_peered":"2024-01-16T15:03:03.752916+0000","last_unstale":"2024-01-16T15:03:03.758276+0000","last_undegraded":"2024-01-16T15:03:03.758276+0000","last_fullsized":"2024-01-16T15:03:03.758276+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:46.559713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.731711+0000","last_change":"2024-01-16T15:03:02.746271+0000","last_active":"2024-01-16T15:03:03.731711+0000","last_peered":"2024-01-16T15:03:03.731711+0000","last_clean":"2024-01-16T15:03:03.731711+0000","last_became_active":"2024-01-16T15:03:02.745996+0000","last_became_peered":"2024-01-16T15:03:02.745996+0000","last_unstale":"2024-01-16T15:03:03.731711+0000","last_undegraded":"2024-01-16T15:03:03.731711+0000","last_fullsized":"2024-01-16T15:03:03.731711+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:07.483449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027548+0000","last_change":"2024-01-16T15:03:03.750756+0000","last_active":"2024-01-16T15:03:05.027548+0000","last_peered":"2024-01-16T15:03:05.027548+0000","last_clean":"2024-01-16T15:03:05.027548+0000","last_became_active":"2024-01-16T15:03:03.750170+0000","last_became_peered":"2024-01-16T15:03:03.750170+0000","last_unstale":"2024-01-16T15:03:05.027548+0000","last_undegraded":"2024-01-16T15:03:05.027548+0000","last_fullsized":"2024-01-16T15:03:05.027548+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:45:37.475530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.743830+0000","last_change":"2024-01-16T15:03:02.743223+0000","last_active":"2024-01-16T15:03:03.743830+0000","last_peered":"2024-01-16T15:03:03.743830+0000","last_clean":"2024-01-16T15:03:03.743830+0000","last_became_active":"2024-01-16T15:03:02.742953+0000","last_became_peered":"2024-01-16T15:03:02.742953+0000","last_unstale":"2024-01-16T15:03:03.743830+0000","last_undegraded":"2024-01-16T15:03:03.743830+0000","last_fullsized":"2024-01-16T15:03:03.743830+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:22.405045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.765260+0000","last_change":"2024-01-16T15:03:03.751298+0000","last_active":"2024-01-16T15:03:03.765260+0000","last_peered":"2024-01-16T15:03:03.765260+0000","last_clean":"2024-01-16T15:03:03.765260+0000","last_became_active":"2024-01-16T15:03:03.750669+0000","last_became_peered":"2024-01-16T15:03:03.750669+0000","last_unstale":"2024-01-16T15:03:03.765260+0000","last_undegraded":"2024-01-16T15:03:03.765260+0000","last_fullsized":"2024-01-16T15:03:03.765260+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:29.856711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729296+0000","last_change":"2024-01-16T15:03:02.737666+0000","last_active":"2024-01-16T15:03:04.729296+0000","last_peered":"2024-01-16T15:03:04.729296+0000","last_clean":"2024-01-16T15:03:04.729296+0000","last_became_active":"2024-01-16T15:03:02.737386+0000","last_became_peered":"2024-01-16T15:03:02.737386+0000","last_unstale":"2024-01-16T15:03:04.729296+0000","last_undegraded":"2024-01-16T15:03:04.729296+0000","last_fullsized":"2024-01-16T15:03:04.729296+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:21.055515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525680+0000","last_change":"2024-01-16T15:03:03.770134+0000","last_active":"2024-01-16T15:03:05.525680+0000","last_peered":"2024-01-16T15:03:05.525680+0000","last_clean":"2024-01-16T15:03:05.525680+0000","last_became_active":"2024-01-16T15:03:03.769815+0000","last_became_peered":"2024-01-16T15:03:03.769815+0000","last_unstale":"2024-01-16T15:03:05.525680+0000","last_undegraded":"2024-01-16T15:03:05.525680+0000","last_fullsized":"2024-01-16T15:03:05.525680+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:44.028816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.740920+0000","last_change":"2024-01-16T15:03:02.743715+0000","last_active":"2024-01-16T15:03:03.740920+0000","last_peered":"2024-01-16T15:03:03.740920+0000","last_clean":"2024-01-16T15:03:03.740920+0000","last_became_active":"2024-01-16T15:03:02.740742+0000","last_became_peered":"2024-01-16T15:03:02.740742+0000","last_unstale":"2024-01-16T15:03:03.740920+0000","last_undegraded":"2024-01-16T15:03:03.740920+0000","last_fullsized":"2024-01-16T15:03:03.740920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:15.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767951+0000","last_change":"2024-01-16T15:03:03.756955+0000","last_active":"2024-01-16T15:03:03.767951+0000","last_peered":"2024-01-16T15:03:03.767951+0000","last_clean":"2024-01-16T15:03:03.767951+0000","last_became_active":"2024-01-16T15:03:03.756648+0000","last_became_peered":"2024-01-16T15:03:03.756648+0000","last_unstale":"2024-01-16T15:03:03.767951+0000","last_undegraded":"2024-01-16T15:03:03.767951+0000","last_fullsized":"2024-01-16T15:03:03.767951+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:05:29.827975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730790+0000","last_change":"2024-01-16T15:03:02.747244+0000","last_active":"2024-01-16T15:03:04.730790+0000","last_peered":"2024-01-16T15:03:04.730790+0000","last_clean":"2024-01-16T15:03:04.730790+0000","last_became_active":"2024-01-16T15:03:02.746069+0000","last_became_peered":"2024-01-16T15:03:02.746069+0000","last_unstale":"2024-01-16T15:03:04.730790+0000","last_undegraded":"2024-01-16T15:03:04.730790+0000","last_fullsized":"2024-01-16T15:03:04.730790+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:42.172985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730616+0000","last_change":"2024-01-16T15:03:03.755079+0000","last_active":"2024-01-16T15:03:04.730616+0000","last_peered":"2024-01-16T15:03:04.730616+0000","last_clean":"2024-01-16T15:03:04.730616+0000","last_became_active":"2024-01-16T15:03:03.754251+0000","last_became_peered":"2024-01-16T15:03:03.754251+0000","last_unstale":"2024-01-16T15:03:04.730616+0000","last_undegraded":"2024-01-16T15:03:04.730616+0000","last_fullsized":"2024-01-16T15:03:04.730616+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:04.449847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.739677+0000","last_change":"2024-01-16T15:03:02.741707+0000","last_active":"2024-01-16T15:03:03.739677+0000","last_peered":"2024-01-16T15:03:03.739677+0000","last_clean":"2024-01-16T15:03:03.739677+0000","last_became_active":"2024-01-16T15:03:02.741350+0000","last_became_peered":"2024-01-16T15:03:02.741350+0000","last_unstale":"2024-01-16T15:03:03.739677+0000","last_undegraded":"2024-01-16T15:03:03.739677+0000","last_fullsized":"2024-01-16T15:03:03.739677+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:31.384210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.763188+0000","last_change":"2024-01-16T15:03:03.759755+0000","last_active":"2024-01-16T15:03:03.763188+0000","last_peered":"2024-01-16T15:03:03.763188+0000","last_clean":"2024-01-16T15:03:03.763188+0000","last_became_active":"2024-01-16T15:03:03.758872+0000","last_became_peered":"2024-01-16T15:03:03.758872+0000","last_unstale":"2024-01-16T15:03:03.763188+0000","last_undegraded":"2024-01-16T15:03:03.763188+0000","last_fullsized":"2024-01-16T15:03:03.763188+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:33.207831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730072+0000","last_change":"2024-01-16T15:03:02.747745+0000","last_active":"2024-01-16T15:03:04.730072+0000","last_peered":"2024-01-16T15:03:04.730072+0000","last_clean":"2024-01-16T15:03:04.730072+0000","last_became_active":"2024-01-16T15:03:02.747445+0000","last_became_peered":"2024-01-16T15:03:02.747445+0000","last_unstale":"2024-01-16T15:03:04.730072+0000","last_undegraded":"2024-01-16T15:03:04.730072+0000","last_fullsized":"2024-01-16T15:03:04.730072+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:59.087146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.769920+0000","last_change":"2024-01-16T15:03:03.764495+0000","last_active":"2024-01-16T15:03:03.769920+0000","last_peered":"2024-01-16T15:03:03.769920+0000","last_clean":"2024-01-16T15:03:03.769920+0000","last_became_active":"2024-01-16T15:03:03.764246+0000","last_became_peered":"2024-01-16T15:03:03.764246+0000","last_unstale":"2024-01-16T15:03:03.769920+0000","last_undegraded":"2024-01-16T15:03:03.769920+0000","last_fullsized":"2024-01-16T15:03:03.769920+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:42.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732809+0000","last_change":"2024-01-16T15:03:02.743333+0000","last_active":"2024-01-16T15:03:04.732809+0000","last_peered":"2024-01-16T15:03:04.732809+0000","last_clean":"2024-01-16T15:03:04.732809+0000","last_became_active":"2024-01-16T15:03:02.742989+0000","last_became_peered":"2024-01-16T15:03:02.742989+0000","last_unstale":"2024-01-16T15:03:04.732809+0000","last_undegraded":"2024-01-16T15:03:04.732809+0000","last_fullsized":"2024-01-16T15:03:04.732809+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:15.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.753216+0000","last_change":"2024-01-16T15:03:03.747977+0000","last_active":"2024-01-16T15:03:03.753216+0000","last_peered":"2024-01-16T15:03:03.753216+0000","last_clean":"2024-01-16T15:03:03.753216+0000","last_became_active":"2024-01-16T15:03:03.747699+0000","last_became_peered":"2024-01-16T15:03:03.747699+0000","last_unstale":"2024-01-16T15:03:03.753216+0000","last_undegraded":"2024-01-16T15:03:03.753216+0000","last_fullsized":"2024-01-16T15:03:03.753216+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:46.994289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.738068+0000","last_change":"2024-01-16T15:03:02.744665+0000","last_active":"2024-01-16T15:03:03.738068+0000","last_peered":"2024-01-16T15:03:03.738068+0000","last_clean":"2024-01-16T15:03:03.738068+0000","last_became_active":"2024-01-16T15:03:02.744384+0000","last_became_peered":"2024-01-16T15:03:02.744384+0000","last_unstale":"2024-01-16T15:03:03.738068+0000","last_undegraded":"2024-01-16T15:03:03.738068+0000","last_fullsized":"2024-01-16T15:03:03.738068+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:23.092743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.731961+0000","last_change":"2024-01-16T15:03:03.759969+0000","last_active":"2024-01-16T15:03:04.731961+0000","last_peered":"2024-01-16T15:03:04.731961+0000","last_clean":"2024-01-16T15:03:04.731961+0000","last_became_active":"2024-01-16T15:03:03.759720+0000","last_became_peered":"2024-01-16T15:03:03.759720+0000","last_unstale":"2024-01-16T15:03:04.731961+0000","last_undegraded":"2024-01-16T15:03:04.731961+0000","last_fullsized":"2024-01-16T15:03:04.731961+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:56.424560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.744814+0000","last_change":"2024-01-16T15:03:02.751074+0000","last_active":"2024-01-16T15:03:03.744814+0000","last_peered":"2024-01-16T15:03:03.744814+0000","last_clean":"2024-01-16T15:03:03.744814+0000","last_became_active":"2024-01-16T15:03:02.750131+0000","last_became_peered":"2024-01-16T15:03:02.750131+0000","last_unstale":"2024-01-16T15:03:03.744814+0000","last_undegraded":"2024-01-16T15:03:03.744814+0000","last_fullsized":"2024-01-16T15:03:03.744814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:55.319844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771092+0000","last_change":"2024-01-16T15:03:03.767680+0000","last_active":"2024-01-16T15:03:03.771092+0000","last_peered":"2024-01-16T15:03:03.771092+0000","last_clean":"2024-01-16T15:03:03.771092+0000","last_became_active":"2024-01-16T15:03:03.767262+0000","last_became_peered":"2024-01-16T15:03:03.767262+0000","last_unstale":"2024-01-16T15:03:03.771092+0000","last_undegraded":"2024-01-16T15:03:03.771092+0000","last_fullsized":"2024-01-16T15:03:03.771092+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:32.400381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.076052+0000","last_change":"2024-01-16T15:03:02.748067+0000","last_active":"2024-01-16T15:03:04.076052+0000","last_peered":"2024-01-16T15:03:04.076052+0000","last_clean":"2024-01-16T15:03:04.076052+0000","last_became_active":"2024-01-16T15:03:02.747807+0000","last_became_peered":"2024-01-16T15:03:02.747807+0000","last_unstale":"2024-01-16T15:03:04.076052+0000","last_undegraded":"2024-01-16T15:03:04.076052+0000","last_fullsized":"2024-01-16T15:03:04.076052+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:01.474775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":187,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.737188+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:03:03.737188+0000","last_peered":"2024-01-16T15:03:03.737188+0000","last_clean":"2024-01-16T15:03:03.737188+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:03:03.737188+0000","last_undegraded":"2024-01-16T15:03:03.737188+0000","last_fullsized":"2024-01-16T15:03:03.737188+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":223029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.2110000000000001,"15min":1.159},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":9.7899999999999991,"5min":9.7899999999999991,"15min":9.7899999999999991},"last":1.23},{"interface":"front","average":{"1min":1.222,"5min":1.171,"15min":1.1679999999999999},"min":{"1min":0.77700000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.232,"15min":1.153},"min":{"1min":0.84099999999999997,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":8.3379999999999992,"5min":8.3379999999999992,"15min":8.3379999999999992},"last":0.871},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.248,"15min":1.246},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.0449999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.381,"15min":1.391},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.3460000000000001,"5min":3.3460000000000001,"15min":3.3460000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.278,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0510000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4159999999999999,"15min":1.4339999999999999},"min":{"1min":0.77600000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.665,"5min":3.665,"15min":3.665},"last":1.395},{"interface":"front","average":{"1min":1.417,"5min":1.512,"15min":1.573},"min":{"1min":0.88400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":1.98,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":1.3420000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.431,"15min":1.4810000000000001},"min":{"1min":0.873,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.1520000000000001,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.524,"15min":1.5620000000000001},"min":{"1min":0.89700000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.1789999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":1.597}]},{"osd":5,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.7010000000000001,"15min":1.702},"min":{"1min":1.0640000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":3.6299999999999999,"5min":3.6299999999999999,"15min":3.6299999999999999},"last":1.671},{"interface":"front","average":{"1min":1.718,"5min":1.6519999999999999,"15min":1.6599999999999999},"min":{"1min":0.93400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.798,"5min":3.798,"15min":3.798},"last":1.536}]},{"osd":7,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.621,"15min":1.625},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":4.0060000000000002,"5min":4.0060000000000002,"15min":4.0060000000000002},"last":1.298},{"interface":"front","average":{"1min":1.712,"5min":1.708,"15min":1.732},"min":{"1min":1.093,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":3.2709999999999999,"5min":3.4159999999999999,"15min":3.4159999999999999},"last":1.722}]}]},{"osd":4,"up_from":15,"seq":64424509484,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":557056,"data_stored":223969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.109,"15min":1.0449999999999999},"min":{"1min":0.67700000000000005,"5min":0.622,"15min":0.622},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1279999999999999,"15min":1.121},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0640000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2370000000000001,"15min":1.1699999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9370000000000001,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.2849999999999999,"15min":1.2410000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":5.04,"5min":5.04,"15min":5.04},"last":1.1060000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3340000000000001,"15min":1.2689999999999999},"min":{"1min":0.80500000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.871,"5min":10.129,"15min":10.129},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.3520000000000001,"15min":1.3009999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":1.393}]},{"osd":3,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.302,"15min":1.3},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2599999999999998,"5min":2.637,"15min":2.637},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.74,"5min":1.532,"15min":1.4390000000000001},"min":{"1min":0.78500000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":7.8940000000000001,"5min":9.8109999999999999,"15min":9.8109999999999999},"last":1.345}]},{"osd":5,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.5980000000000001,"15min":1.526},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":11.823,"5min":11.823,"15min":11.823},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.68200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":1.4730000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.546,"15min":1.514},"min":{"1min":0.59099999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.855,"5min":9.7219999999999995,"15min":9.7219999999999995},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.9690000000000001,"5min":1.6220000000000001,"15min":1.5720000000000001},"min":{"1min":0.91500000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"last":1.5069999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5349999999999999,"15min":1.5609999999999999},"min":{"1min":0.63100000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.3929999999999998,"5min":3.048,"15min":3.048},"last":1.302},{"interface":"front","average":{"1min":2.2869999999999999,"5min":1.8999999999999999,"15min":1.728},"min":{"1min":0.74299999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":13.561999999999999,"5min":13.561999999999999,"15min":13.561999999999999},"last":1.4330000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":812815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.042,"15min":1.0229999999999999},"min":{"1min":0.81000000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.64000000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4990000000000001,"5min":1.746,"15min":1.746},"last":0.61799999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.1180000000000001,"15min":1.113},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2130000000000001,"15min":1.173},"min":{"1min":0.97799999999999998,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":3.5609999999999999,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.748}]},{"osd":2,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.25,"15min":1.248},"min":{"1min":0.89700000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.2649999999999999,"15min":1.2549999999999999},"min":{"1min":1.0009999999999999,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.0269999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.3660000000000001,"15min":1.3440000000000001},"min":{"1min":0.98799999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.6000000000000001,"5min":4.2249999999999996,"15min":4.2249999999999996},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.3400000000000001,"15min":1.335},"min":{"1min":1.1040000000000001,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.115}]},{"osd":4,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.383,"15min":1.391},"min":{"1min":0.88800000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.71,"5min":2.71,"15min":2.71},"last":1.395},{"interface":"front","average":{"1min":1.494,"5min":1.403,"15min":1.403},"min":{"1min":1.0449999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.2599999999999998,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.26}]},{"osd":5,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.4179999999999999,"15min":1.407},"min":{"1min":1.0549999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.548,"5min":2.548,"15min":2.548},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.5549999999999999,"15min":1.5580000000000001},"min":{"1min":1.095,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":1.508}]},{"osd":6,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.5449999999999999,"15min":1.5509999999999999},"min":{"1min":1.141,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.3690000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.349},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.571,"15min":1.5669999999999999},"min":{"1min":1.095,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.6600000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.0569999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":216127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.0820000000000001,"15min":1.042},"min":{"1min":0.65000000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.1459999999999999,"5min":3.1459999999999999,"15min":3.1459999999999999},"last":1.349},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.2070000000000001,"15min":1.095},"min":{"1min":0.68799999999999994,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":10.074999999999999,"5min":10.074999999999999,"15min":10.074999999999999},"last":0.97199999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.129,"15min":1.113},"min":{"1min":0.72699999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.921,"5min":2.625,"15min":2.625},"last":1.319},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.1599999999999999,"15min":1.1060000000000001},"min":{"1min":0.86599999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.69899999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.2310000000000001,"15min":1.1930000000000001},"min":{"1min":0.94499999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8140000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.367,"5min":1.2470000000000001,"15min":1.21},"min":{"1min":0.97699999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.796,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.329,"15min":1.3360000000000001},"min":{"1min":0.93100000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":3.4249999999999998,"5min":3.4249999999999998,"15min":3.4249999999999998},"last":1.224},{"interface":"front","average":{"1min":2.1920000000000002,"5min":1.5149999999999999,"15min":1.411},"min":{"1min":1.0429999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":14.647,"5min":14.647,"15min":14.647},"last":1.03}]},{"osd":4,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3220000000000001,"15min":1.304},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.409,"15min":1.393},"min":{"1min":0.90400000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.2629999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.755,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7270000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.4219999999999999,"15min":1.403},"min":{"1min":1.1180000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.9219999999999999,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.494}]},{"osd":7,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.544,"15min":1.5529999999999999},"min":{"1min":1.1599999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.8720000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.637,"15min":1.649},"min":{"1min":1.333,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8540000000000001,"5min":3.903,"15min":3.903},"last":1.6020000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35872,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335968,"statfs":{"total":96636764160,"available":96600031232,"internally_reserved":0,"allocated":544768,"data_stored":216695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.86299999999999999,"15min":0.81999999999999995},"min":{"1min":0.73099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":2.6099999999999999},"last":1.22},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.0069999999999999,"15min":0.97999999999999998},"min":{"1min":0.68600000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.5110000000000001,"5min":4.5110000000000001,"15min":4.5110000000000001},"last":0.76900000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.94099999999999995,"15min":0.879},"min":{"1min":0.58099999999999996,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":5.4269999999999996,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.1000000000000001,"15min":1.0349999999999999},"min":{"1min":0.79100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":1.361}]},{"osd":2,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.175,"15min":1.105},"min":{"1min":0.73899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.4299999999999999},{"interface":"front","average":{"1min":2.012,"5min":1.286,"15min":1.147},"min":{"1min":0.84999999999999998,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":14.577999999999999,"5min":14.577999999999999,"15min":14.577999999999999},"last":1.099}]},{"osd":4,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.351,"15min":1.274},"min":{"1min":0.94999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":5.4900000000000002,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.421,"15min":1.3959999999999999},"min":{"1min":0.96199999999999997,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.4970000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.46,"15min":1.407},"min":{"1min":1.0209999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":5.3520000000000003,"5min":5.3520000000000003,"15min":5.3520000000000003},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.484,"15min":1.4430000000000001},"min":{"1min":1.0469999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.556}]},{"osd":6,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.5009999999999999,"15min":1.429},"min":{"1min":1.0960000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":8.4730000000000008,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":1.149},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.498,"15min":1.4119999999999999},"min":{"1min":1.2689999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":1.387}]},{"osd":7,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":1.52,"15min":1.4490000000000001},"min":{"1min":1.2170000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":4.0830000000000002,"5min":4.0830000000000002,"15min":4.0830000000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.9430000000000001,"5min":1.5800000000000001,"15min":1.4950000000000001},"min":{"1min":1.385,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":5.5510000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":1.5289999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":548864,"data_stored":217691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.436,"15min":0.436},"max":{"1min":1.3129999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.018,"5min":1.0169999999999999,"15min":1.02},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5700000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":3.839,"5min":3.839,"15min":3.839},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.0940000000000001,"15min":1.0720000000000001},"min":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.0189999999999999,"15min":0.92600000000000005},"min":{"1min":0.60899999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":9.1530000000000005,"5min":9.1530000000000005,"15min":9.1530000000000005},"last":0.88},{"interface":"front","average":{"1min":1.379,"5min":1.3149999999999999,"15min":1.3120000000000001},"min":{"1min":0.82999999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.857,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.82699999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.292,"15min":1.306},"min":{"1min":0.93000000000000005,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.407,"15min":1.421},"min":{"1min":0.93899999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.093,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.24}]},{"osd":5,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.3580000000000001,"15min":1.3220000000000001},"min":{"1min":0.91100000000000003,"5min":0.753,"15min":0.753},"max":{"1min":9.6189999999999998,"5min":9.6189999999999998,"15min":9.6189999999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.4319999999999999,"15min":1.3759999999999999},"min":{"1min":0.91600000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":9.423,"5min":9.423,"15min":9.423},"last":1.04}]},{"osd":6,"last update":"Tue Jan 16 15:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.46,"15min":1.3799999999999999},"min":{"1min":0.93400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":9.5359999999999996,"5min":9.5359999999999996,"15min":9.5359999999999996},"last":1.01},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.5549999999999999,"15min":1.4950000000000001},"min":{"1min":1.0049999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":1.502}]},{"osd":7,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4970000000000001,"15min":1.5189999999999999},"min":{"1min":1.121,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.024,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.403},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.542,"15min":1.583},"min":{"1min":0.98699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.992,"5min":2.54,"15min":2.54},"last":1.204}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35868,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335972,"statfs":{"total":96636764160,"available":96600035328,"internally_reserved":0,"allocated":536576,"data_stored":216127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":0.96399999999999997,"15min":0.84799999999999998},"min":{"1min":0.57999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":10.667999999999999,"5min":10.667999999999999,"15min":10.667999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.97199999999999998,"15min":0.91600000000000004},"min":{"1min":0.46500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.02,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.98999999999999999,"15min":0.96799999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.3200000000000001,"15min":1.196},"min":{"1min":0.82299999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.805,"5min":10.805,"15min":10.805},"last":1.0900000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.238,"15min":1.133},"min":{"1min":0.75800000000000001,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":10.991,"5min":10.991,"15min":10.991},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.1639999999999999,"15min":1.0840000000000001},"min":{"1min":0.50600000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":0.94999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2589999999999999,"15min":1.2749999999999999},"min":{"1min":0.93200000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.157},{"interface":"front","average":{"1min":2.0310000000000001,"5min":1.5700000000000001,"15min":1.502},"min":{"1min":1.111,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":11.157999999999999,"5min":11.157999999999999,"15min":11.157999999999999},"last":1.847}]},{"osd":5,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.381,"15min":1.351},"min":{"1min":0.90000000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.913},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.4470000000000001,"15min":1.4450000000000001},"min":{"1min":0.83399999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":2.157}]},{"osd":6,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.4810000000000001,"15min":1.4690000000000001},"min":{"1min":1.0549999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.1920000000000002,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.786},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.4359999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":2.0459999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5229999999999999,"15min":1.5409999999999999},"min":{"1min":0.83099999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":2.2690000000000001},{"interface":"front","average":{"1min":2.0939999999999999,"5min":1.627,"15min":1.5289999999999999},"min":{"1min":1.266,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":11.063000000000001,"5min":11.063000000000001,"15min":11.063000000000001},"last":1.9670000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":806459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.98399999999999999,"15min":0.95799999999999996},"min":{"1min":0.59699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":0.499},{"interface":"front","average":{"1min":1.236,"5min":1.048,"15min":1.0449999999999999},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.432},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.0469999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.923,"5min":1.262,"15min":1.1579999999999999},"min":{"1min":0.71899999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":17.032,"5min":17.032,"15min":17.032},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.163,"15min":1.1140000000000001},"min":{"1min":0.626,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":6.1470000000000002,"5min":6.1470000000000002,"15min":6.1470000000000002},"last":1.2689999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.228,"15min":1.1519999999999999},"min":{"1min":0.67000000000000004,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":10.795,"5min":10.795,"15min":10.795},"last":0.71799999999999997},{"interface":"front","average":{"1min":2.0150000000000001,"5min":1.4119999999999999,"15min":1.319},"min":{"1min":0.93500000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":14.992000000000001,"5min":14.992000000000001,"15min":14.992000000000001},"last":0.69699999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.4430000000000001,"15min":1.3919999999999999},"min":{"1min":1.107,"5min":0.625,"15min":0.625},"max":{"1min":4.8529999999999998,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.425,"15min":1.3260000000000001},"min":{"1min":1.0129999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":3.2749999999999999,"5min":9.8719999999999999,"15min":9.8719999999999999},"last":1.147}]},{"osd":5,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":2.2309999999999999,"5min":1.5029999999999999,"15min":1.393},"min":{"1min":1.038,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":10.606999999999999,"5min":10.606999999999999,"15min":10.606999999999999},"last":0.88},{"interface":"front","average":{"1min":1.623,"5min":1.5149999999999999,"15min":1.5029999999999999},"min":{"1min":1.089,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.2010000000000001,"5min":3.2010000000000001,"15min":3.2010000000000001},"last":0.92500000000000004}]},{"osd":6,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.5069999999999999,"15min":1.4370000000000001},"min":{"1min":1.0980000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":10.69,"5min":10.69,"15min":10.69},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.714,"5min":1.5600000000000001,"15min":1.544},"min":{"1min":1.085,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":0.89400000000000002}]},{"osd":7,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.496,"15min":1.52},"min":{"1min":0.84999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"last":1.6240000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.6719999999999999,"15min":1.585},"min":{"1min":1.264,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":1.137}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:03:08.527 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:03:08.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.806+0000 7f8e87400640 1 Processor -- start 2024-01-16T15:03:08.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.806+0000 7f8e87400640 1 -- start start 2024-01-16T15:03:08.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e87400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e80071410 0x7f8e80146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e87400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8e8006c7c0 0x7f8e800717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e87400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071d30 0x7f8e800b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e87400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8006cc50 con 0x7f8e80071410 2024-01-16T15:03:08.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e87400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e8006cec0 con 0x7f8e8006c7c0 2024-01-16T15:03:08.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e87400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e8006d1a0 con 0x7f8e80071d30 2024-01-16T15:03:08.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.808+0000 7f8e86000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e80071410 0x7f8e80146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e80071410 0x7f8e80146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41406/0 (socket says 172.21.15.53:41406) 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 -- 172.21.15.53:0/962506584 learned_addr learned my addr 172.21.15.53:0/962506584 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86a00640 1 --2- 172.21.15.53:0/962506584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071d30 0x7f8e800b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e85600640 1 --2- 172.21.15.53:0/962506584 >> v2:172.21.15.146:3300/0 conn(0x7f8e8006c7c0 0x7f8e800717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 -- 172.21.15.53:0/962506584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071d30 msgr2=0x7f8e800b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 --2- 172.21.15.53:0/962506584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071d30 0x7f8e800b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 -- 172.21.15.53:0/962506584 >> v2:172.21.15.146:3300/0 conn(0x7f8e8006c7c0 msgr2=0x7f8e800717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 --2- 172.21.15.53:0/962506584 >> v2:172.21.15.146:3300/0 conn(0x7f8e8006c7c0 0x7f8e800717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e86000640 1 -- 172.21.15.53:0/962506584 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e8006d480 con 0x7f8e80071410 2024-01-16T15:03:08.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.809+0000 7f8e85600640 1 --2- 172.21.15.53:0/962506584 >> v2:172.21.15.146:3300/0 conn(0x7f8e8006c7c0 0x7f8e800717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:03:08.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.811+0000 7f8e86000640 1 --2- 172.21.15.53:0/962506584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e80071410 0x7f8e80146770 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8e80146ef0 tx=0x7f8e7802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31639f734164e7d9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:08.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.811+0000 7f8e84c00640 1 -- 172.21.15.53:0/962506584 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7802ca70 con 0x7f8e80071410 2024-01-16T15:03:08.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e84c00640 1 -- 172.21.15.53:0/962506584 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7802cbd0 con 0x7f8e80071410 2024-01-16T15:03:08.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e84c00640 1 -- 172.21.15.53:0/962506584 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780343f0 con 0x7f8e80071410 2024-01-16T15:03:08.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 -- 172.21.15.53:0/962506584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e80071410 msgr2=0x7f8e80146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 --2- 172.21.15.53:0/962506584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e80071410 0x7f8e80146770 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8e80146ef0 tx=0x7f8e7802bed0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 -- 172.21.15.53:0/962506584 shutdown_connections 2024-01-16T15:03:08.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 --2- 172.21.15.53:0/962506584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071d30 0x7f8e800b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 --2- 172.21.15.53:0/962506584 >> v2:172.21.15.146:3300/0 conn(0x7f8e8006c7c0 0x7f8e800717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 --2- 172.21.15.53:0/962506584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e80071410 0x7f8e80146770 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 -- 172.21.15.53:0/962506584 >> 172.21.15.53:0/962506584 conn(0x7f8e8006a800 msgr2=0x7f8e8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:08.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 -- 172.21.15.53:0/962506584 shutdown_connections 2024-01-16T15:03:08.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.812+0000 7f8e87400640 1 -- 172.21.15.53:0/962506584 wait complete. 2024-01-16T15:03:08.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.814+0000 7f8e87400640 1 Processor -- start 2024-01-16T15:03:08.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.815+0000 7f8e87400640 1 -- start start 2024-01-16T15:03:08.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e87400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 0x7f8e800c4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e87400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071410 0x7f8e8015c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e87400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e80071d30 0x7f8e800b2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e87400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800c3f10 con 0x7f8e8006c7c0 2024-01-16T15:03:08.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e87400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800c41f0 con 0x7f8e80071d30 2024-01-16T15:03:08.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e87400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e800c44d0 con 0x7f8e80071410 2024-01-16T15:03:08.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 0x7f8e800c4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e86000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 0x7f8e800c4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41420/0 (socket says 172.21.15.53:41420) 2024-01-16T15:03:08.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.816+0000 7f8e86000640 1 -- 172.21.15.53:0/395310897 learned_addr learned my addr 172.21.15.53:0/395310897 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86000640 1 -- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071410 msgr2=0x7f8e8015c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86000640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071410 0x7f8e8015c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86000640 1 -- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e80071d30 msgr2=0x7f8e800b2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86a00640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e80071d30 0x7f8e800b2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86000640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e80071d30 0x7f8e800b2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86000640 1 -- 172.21.15.53:0/395310897 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e78009b30 con 0x7f8e8006c7c0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e86000640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 0x7f8e800c4fb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8e78000f80 tx=0x7f8e78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e78009510 con 0x7f8e8006c7c0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7803e450 con 0x7f8e8006c7c0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7803e800 con 0x7f8e8006c7c0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.817+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e800c47b0 con 0x7f8e8006c7c0 2024-01-16T15:03:08.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.818+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e800c54f0 con 0x7f8e8006c7c0 2024-01-16T15:03:08.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.819+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e800c4c70 con 0x7f8e8006c7c0 2024-01-16T15:03:08.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.820+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f8e7803e960 con 0x7f8e8006c7c0 2024-01-16T15:03:08.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.821+0000 7f8e7e000640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5806b500 0x7f8e5806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:08.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.821+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8e780ace20 con 0x7f8e8006c7c0 2024-01-16T15:03:08.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.821+0000 7f8e85600640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5806b500 0x7f8e5806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:08.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.823+0000 7f8e85600640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5806b500 0x7f8e5806d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006e80 tx=0x7f8e68008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:08.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:08.825+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e780779c0 con 0x7f8e8006c7c0 2024-01-16T15:03:08.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.017+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8e80072180 con 0x7f8e5806b500 2024-01-16T15:03:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.040+0000 7f8e7e000640 1 -- 172.21.15.53:0/395310897 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287214 (secure 0 0 0) 0x7f8e68009d80 con 0x7f8e5806b500 2024-01-16T15:03:09.024 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:09.026 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5806b500 msgr2=0x7f8e5806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5806b500 0x7f8e5806d9c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8e68006e80 tx=0x7f8e68008040 comp rx=0 tx=0).stop 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 msgr2=0x7f8e800c4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 0x7f8e800c4fb0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f8e78000f80 tx=0x7f8e78004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 shutdown_connections 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5806b500 0x7f8e5806d9c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e80071d30 0x7f8e800b2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e80071410 0x7f8e8015c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 --2- 172.21.15.53:0/395310897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8006c7c0 0x7f8e800c4fb0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 >> 172.21.15.53:0/395310897 conn(0x7f8e8006a800 msgr2=0x7f8e801465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.046+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 shutdown_connections 2024-01-16T15:03:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.047+0000 7f8e87400640 1 -- 172.21.15.53:0/395310897 wait complete. 2024-01-16T15:03:09.049 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-01-16T15:03:08.241432+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":101,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":263212,"kb_used_data":5420,"kb_used_omap":12,"kb_used_meta":257651,"kb_avail":754711508,"statfs":{"total":773094113280,"available":772824584192,"internally_reserved":0,"allocated":5550080,"data_stored":2932912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":263835184},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002074"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730331+0000","last_change":"2024-01-16T15:03:03.756914+0000","last_active":"2024-01-16T15:03:04.730331+0000","last_peered":"2024-01-16T15:03:04.730331+0000","last_clean":"2024-01-16T15:03:04.730331+0000","last_became_active":"2024-01-16T15:03:03.756279+0000","last_became_peered":"2024-01-16T15:03:03.756279+0000","last_unstale":"2024-01-16T15:03:04.730331+0000","last_undegraded":"2024-01-16T15:03:04.730331+0000","last_fullsized":"2024-01-16T15:03:04.730331+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:13.143628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771361+0000","last_change":"2024-01-16T15:03:03.766530+0000","last_active":"2024-01-16T15:03:03.771361+0000","last_peered":"2024-01-16T15:03:03.771361+0000","last_clean":"2024-01-16T15:03:03.771361+0000","last_became_active":"2024-01-16T15:03:03.766168+0000","last_became_peered":"2024-01-16T15:03:03.766168+0000","last_unstale":"2024-01-16T15:03:03.771361+0000","last_undegraded":"2024-01-16T15:03:03.771361+0000","last_fullsized":"2024-01-16T15:03:03.771361+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:22.071687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732234+0000","last_change":"2024-01-16T15:03:03.751679+0000","last_active":"2024-01-16T15:03:04.732234+0000","last_peered":"2024-01-16T15:03:04.732234+0000","last_clean":"2024-01-16T15:03:04.732234+0000","last_became_active":"2024-01-16T15:03:03.751372+0000","last_became_peered":"2024-01-16T15:03:03.751372+0000","last_unstale":"2024-01-16T15:03:04.732234+0000","last_undegraded":"2024-01-16T15:03:04.732234+0000","last_fullsized":"2024-01-16T15:03:04.732234+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:59.920442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771050+0000","last_change":"2024-01-16T15:03:03.766324+0000","last_active":"2024-01-16T15:03:03.771050+0000","last_peered":"2024-01-16T15:03:03.771050+0000","last_clean":"2024-01-16T15:03:03.771050+0000","last_became_active":"2024-01-16T15:03:03.765790+0000","last_became_peered":"2024-01-16T15:03:03.765790+0000","last_unstale":"2024-01-16T15:03:03.771050+0000","last_undegraded":"2024-01-16T15:03:03.771050+0000","last_fullsized":"2024-01-16T15:03:03.771050+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:42.967775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.765923+0000","last_change":"2024-01-16T15:03:03.763765+0000","last_active":"2024-01-16T15:03:03.765923+0000","last_peered":"2024-01-16T15:03:03.765923+0000","last_clean":"2024-01-16T15:03:03.765923+0000","last_became_active":"2024-01-16T15:03:03.763447+0000","last_became_peered":"2024-01-16T15:03:03.763447+0000","last_unstale":"2024-01-16T15:03:03.765923+0000","last_undegraded":"2024-01-16T15:03:03.765923+0000","last_fullsized":"2024-01-16T15:03:03.765923+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:32.400381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767198+0000","last_change":"2024-01-16T15:03:03.763085+0000","last_active":"2024-01-16T15:03:03.767198+0000","last_peered":"2024-01-16T15:03:03.767198+0000","last_clean":"2024-01-16T15:03:03.767198+0000","last_became_active":"2024-01-16T15:03:03.762031+0000","last_became_peered":"2024-01-16T15:03:03.762031+0000","last_unstale":"2024-01-16T15:03:03.767198+0000","last_undegraded":"2024-01-16T15:03:03.767198+0000","last_fullsized":"2024-01-16T15:03:03.767198+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:32.242782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729402+0000","last_change":"2024-01-16T15:03:03.749130+0000","last_active":"2024-01-16T15:03:04.729402+0000","last_peered":"2024-01-16T15:03:04.729402+0000","last_clean":"2024-01-16T15:03:04.729402+0000","last_became_active":"2024-01-16T15:03:03.748855+0000","last_became_peered":"2024-01-16T15:03:03.748855+0000","last_unstale":"2024-01-16T15:03:04.729402+0000","last_undegraded":"2024-01-16T15:03:04.729402+0000","last_fullsized":"2024-01-16T15:03:04.729402+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:38.492632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.769179+0000","last_change":"2024-01-16T15:03:03.763236+0000","last_active":"2024-01-16T15:03:03.769179+0000","last_peered":"2024-01-16T15:03:03.769179+0000","last_clean":"2024-01-16T15:03:03.769179+0000","last_became_active":"2024-01-16T15:03:03.762904+0000","last_became_peered":"2024-01-16T15:03:03.762904+0000","last_unstale":"2024-01-16T15:03:03.769179+0000","last_undegraded":"2024-01-16T15:03:03.769179+0000","last_fullsized":"2024-01-16T15:03:03.769179+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:46.559713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.770313+0000","last_change":"2024-01-16T15:03:03.766115+0000","last_active":"2024-01-16T15:03:03.770313+0000","last_peered":"2024-01-16T15:03:03.770313+0000","last_clean":"2024-01-16T15:03:03.770313+0000","last_became_active":"2024-01-16T15:03:03.763415+0000","last_became_peered":"2024-01-16T15:03:03.763415+0000","last_unstale":"2024-01-16T15:03:03.770313+0000","last_undegraded":"2024-01-16T15:03:03.770313+0000","last_fullsized":"2024-01-16T15:03:03.770313+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:13:44.557239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768521+0000","last_change":"2024-01-16T15:03:03.763133+0000","last_active":"2024-01-16T15:03:03.768521+0000","last_peered":"2024-01-16T15:03:03.768521+0000","last_clean":"2024-01-16T15:03:03.768521+0000","last_became_active":"2024-01-16T15:03:03.762833+0000","last_became_peered":"2024-01-16T15:03:03.762833+0000","last_unstale":"2024-01-16T15:03:03.768521+0000","last_undegraded":"2024-01-16T15:03:03.768521+0000","last_fullsized":"2024-01-16T15:03:03.768521+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:51.286197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768487+0000","last_change":"2024-01-16T15:03:03.762409+0000","last_active":"2024-01-16T15:03:03.768487+0000","last_peered":"2024-01-16T15:03:03.768487+0000","last_clean":"2024-01-16T15:03:03.768487+0000","last_became_active":"2024-01-16T15:03:03.762025+0000","last_became_peered":"2024-01-16T15:03:03.762025+0000","last_unstale":"2024-01-16T15:03:03.768487+0000","last_undegraded":"2024-01-16T15:03:03.768487+0000","last_fullsized":"2024-01-16T15:03:03.768487+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:29:48.563036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.752461+0000","last_change":"2024-01-16T15:03:03.747009+0000","last_active":"2024-01-16T15:03:03.752461+0000","last_peered":"2024-01-16T15:03:03.752461+0000","last_clean":"2024-01-16T15:03:03.752461+0000","last_became_active":"2024-01-16T15:03:03.746345+0000","last_became_peered":"2024-01-16T15:03:03.746345+0000","last_unstale":"2024-01-16T15:03:03.752461+0000","last_undegraded":"2024-01-16T15:03:03.752461+0000","last_fullsized":"2024-01-16T15:03:03.752461+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:42.967775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027268+0000","last_change":"2024-01-16T15:03:03.761513+0000","last_active":"2024-01-16T15:03:05.027268+0000","last_peered":"2024-01-16T15:03:05.027268+0000","last_clean":"2024-01-16T15:03:05.027268+0000","last_became_active":"2024-01-16T15:03:03.761220+0000","last_became_peered":"2024-01-16T15:03:03.761220+0000","last_unstale":"2024-01-16T15:03:05.027268+0000","last_undegraded":"2024-01-16T15:03:05.027268+0000","last_fullsized":"2024-01-16T15:03:05.027268+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:34:42.829656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.765131+0000","last_change":"2024-01-16T15:03:03.750530+0000","last_active":"2024-01-16T15:03:03.765131+0000","last_peered":"2024-01-16T15:03:03.765131+0000","last_clean":"2024-01-16T15:03:03.765131+0000","last_became_active":"2024-01-16T15:03:03.747908+0000","last_became_peered":"2024-01-16T15:03:03.747908+0000","last_unstale":"2024-01-16T15:03:03.765131+0000","last_undegraded":"2024-01-16T15:03:03.765131+0000","last_fullsized":"2024-01-16T15:03:03.765131+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:32:14.533934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.762656+0000","last_change":"2024-01-16T15:03:03.757148+0000","last_active":"2024-01-16T15:03:03.762656+0000","last_peered":"2024-01-16T15:03:03.762656+0000","last_clean":"2024-01-16T15:03:03.762656+0000","last_became_active":"2024-01-16T15:03:03.755613+0000","last_became_peered":"2024-01-16T15:03:03.755613+0000","last_unstale":"2024-01-16T15:03:03.762656+0000","last_undegraded":"2024-01-16T15:03:03.762656+0000","last_fullsized":"2024-01-16T15:03:03.762656+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:05.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524458+0000","last_change":"2024-01-16T15:03:03.767628+0000","last_active":"2024-01-16T15:03:05.524458+0000","last_peered":"2024-01-16T15:03:05.524458+0000","last_clean":"2024-01-16T15:03:05.524458+0000","last_became_active":"2024-01-16T15:03:03.767293+0000","last_became_peered":"2024-01-16T15:03:03.767293+0000","last_unstale":"2024-01-16T15:03:05.524458+0000","last_undegraded":"2024-01-16T15:03:05.524458+0000","last_fullsized":"2024-01-16T15:03:05.524458+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:53.126320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733406+0000","last_change":"2024-01-16T15:03:03.753799+0000","last_active":"2024-01-16T15:03:04.733406+0000","last_peered":"2024-01-16T15:03:04.733406+0000","last_clean":"2024-01-16T15:03:04.733406+0000","last_became_active":"2024-01-16T15:03:03.753494+0000","last_became_peered":"2024-01-16T15:03:03.753494+0000","last_unstale":"2024-01-16T15:03:04.733406+0000","last_undegraded":"2024-01-16T15:03:04.733406+0000","last_fullsized":"2024-01-16T15:03:04.733406+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:14.400108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732937+0000","last_change":"2024-01-16T15:03:03.769228+0000","last_active":"2024-01-16T15:03:04.732937+0000","last_peered":"2024-01-16T15:03:04.732937+0000","last_clean":"2024-01-16T15:03:04.732937+0000","last_became_active":"2024-01-16T15:03:03.768967+0000","last_became_peered":"2024-01-16T15:03:03.768967+0000","last_unstale":"2024-01-16T15:03:04.732937+0000","last_undegraded":"2024-01-16T15:03:04.732937+0000","last_fullsized":"2024-01-16T15:03:04.732937+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:42.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525793+0000","last_change":"2024-01-16T15:03:03.770152+0000","last_active":"2024-01-16T15:03:05.525793+0000","last_peered":"2024-01-16T15:03:05.525793+0000","last_clean":"2024-01-16T15:03:05.525793+0000","last_became_active":"2024-01-16T15:03:03.769919+0000","last_became_peered":"2024-01-16T15:03:03.769919+0000","last_unstale":"2024-01-16T15:03:05.525793+0000","last_undegraded":"2024-01-16T15:03:05.525793+0000","last_fullsized":"2024-01-16T15:03:05.525793+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:15.453517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027392+0000","last_change":"2024-01-16T15:03:03.769582+0000","last_active":"2024-01-16T15:03:05.027392+0000","last_peered":"2024-01-16T15:03:05.027392+0000","last_clean":"2024-01-16T15:03:05.027392+0000","last_became_active":"2024-01-16T15:03:03.768731+0000","last_became_peered":"2024-01-16T15:03:03.768731+0000","last_unstale":"2024-01-16T15:03:05.027392+0000","last_undegraded":"2024-01-16T15:03:05.027392+0000","last_fullsized":"2024-01-16T15:03:05.027392+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:28:17.546707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.762982+0000","last_change":"2024-01-16T15:03:03.760051+0000","last_active":"2024-01-16T15:03:03.762982+0000","last_peered":"2024-01-16T15:03:03.762982+0000","last_clean":"2024-01-16T15:03:03.762982+0000","last_became_active":"2024-01-16T15:03:03.759757+0000","last_became_peered":"2024-01-16T15:03:03.759757+0000","last_unstale":"2024-01-16T15:03:03.762982+0000","last_undegraded":"2024-01-16T15:03:03.762982+0000","last_fullsized":"2024-01-16T15:03:03.762982+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:09.541105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768960+0000","last_change":"2024-01-16T15:03:03.757236+0000","last_active":"2024-01-16T15:03:03.768960+0000","last_peered":"2024-01-16T15:03:03.768960+0000","last_clean":"2024-01-16T15:03:03.768960+0000","last_became_active":"2024-01-16T15:03:03.756941+0000","last_became_peered":"2024-01-16T15:03:03.756941+0000","last_unstale":"2024-01-16T15:03:03.768960+0000","last_undegraded":"2024-01-16T15:03:03.768960+0000","last_fullsized":"2024-01-16T15:03:03.768960+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:13:35.561556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524811+0000","last_change":"2024-01-16T15:03:03.758872+0000","last_active":"2024-01-16T15:03:05.524811+0000","last_peered":"2024-01-16T15:03:05.524811+0000","last_clean":"2024-01-16T15:03:05.524811+0000","last_became_active":"2024-01-16T15:03:03.758569+0000","last_became_peered":"2024-01-16T15:03:03.758569+0000","last_unstale":"2024-01-16T15:03:05.524811+0000","last_undegraded":"2024-01-16T15:03:05.524811+0000","last_fullsized":"2024-01-16T15:03:05.524811+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:07:56.907068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524429+0000","last_change":"2024-01-16T15:03:03.758659+0000","last_active":"2024-01-16T15:03:05.524429+0000","last_peered":"2024-01-16T15:03:05.524429+0000","last_clean":"2024-01-16T15:03:05.524429+0000","last_became_active":"2024-01-16T15:03:03.758390+0000","last_became_peered":"2024-01-16T15:03:03.758390+0000","last_unstale":"2024-01-16T15:03:05.524429+0000","last_undegraded":"2024-01-16T15:03:05.524429+0000","last_fullsized":"2024-01-16T15:03:05.524429+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:23.546151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525471+0000","last_change":"2024-01-16T15:03:03.763284+0000","last_active":"2024-01-16T15:03:05.525471+0000","last_peered":"2024-01-16T15:03:05.525471+0000","last_clean":"2024-01-16T15:03:05.525471+0000","last_became_active":"2024-01-16T15:03:03.762944+0000","last_became_peered":"2024-01-16T15:03:03.762944+0000","last_unstale":"2024-01-16T15:03:05.525471+0000","last_undegraded":"2024-01-16T15:03:05.525471+0000","last_fullsized":"2024-01-16T15:03:05.525471+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:13.874608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027187+0000","last_change":"2024-01-16T15:03:03.757757+0000","last_active":"2024-01-16T15:03:05.027187+0000","last_peered":"2024-01-16T15:03:05.027187+0000","last_clean":"2024-01-16T15:03:05.027187+0000","last_became_active":"2024-01-16T15:03:03.757467+0000","last_became_peered":"2024-01-16T15:03:03.757467+0000","last_unstale":"2024-01-16T15:03:05.027187+0000","last_undegraded":"2024-01-16T15:03:05.027187+0000","last_fullsized":"2024-01-16T15:03:05.027187+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:54.306598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732313+0000","last_change":"2024-01-16T15:03:03.755329+0000","last_active":"2024-01-16T15:03:04.732313+0000","last_peered":"2024-01-16T15:03:04.732313+0000","last_clean":"2024-01-16T15:03:04.732313+0000","last_became_active":"2024-01-16T15:03:03.753828+0000","last_became_peered":"2024-01-16T15:03:03.753828+0000","last_unstale":"2024-01-16T15:03:04.732313+0000","last_undegraded":"2024-01-16T15:03:04.732313+0000","last_fullsized":"2024-01-16T15:03:04.732313+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:33.207831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.752479+0000","last_change":"2024-01-16T15:03:03.747091+0000","last_active":"2024-01-16T15:03:03.752479+0000","last_peered":"2024-01-16T15:03:03.752479+0000","last_clean":"2024-01-16T15:03:03.752479+0000","last_became_active":"2024-01-16T15:03:03.746793+0000","last_became_peered":"2024-01-16T15:03:03.746793+0000","last_unstale":"2024-01-16T15:03:03.752479+0000","last_undegraded":"2024-01-16T15:03:03.752479+0000","last_fullsized":"2024-01-16T15:03:03.752479+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:51.286197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.524816+0000","last_change":"2024-01-16T15:03:03.756962+0000","last_active":"2024-01-16T15:03:05.524816+0000","last_peered":"2024-01-16T15:03:05.524816+0000","last_clean":"2024-01-16T15:03:05.524816+0000","last_became_active":"2024-01-16T15:03:03.756513+0000","last_became_peered":"2024-01-16T15:03:03.756513+0000","last_unstale":"2024-01-16T15:03:05.524816+0000","last_undegraded":"2024-01-16T15:03:05.524816+0000","last_fullsized":"2024-01-16T15:03:05.524816+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:57:34.124804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732216+0000","last_change":"2024-01-16T15:03:03.750861+0000","last_active":"2024-01-16T15:03:04.732216+0000","last_peered":"2024-01-16T15:03:04.732216+0000","last_clean":"2024-01-16T15:03:04.732216+0000","last_became_active":"2024-01-16T15:03:03.750493+0000","last_became_peered":"2024-01-16T15:03:03.750493+0000","last_unstale":"2024-01-16T15:03:04.732216+0000","last_undegraded":"2024-01-16T15:03:04.732216+0000","last_fullsized":"2024-01-16T15:03:04.732216+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:05.286531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.758833+0000","last_change":"2024-01-16T15:03:03.754509+0000","last_active":"2024-01-16T15:03:03.758833+0000","last_peered":"2024-01-16T15:03:03.758833+0000","last_clean":"2024-01-16T15:03:03.758833+0000","last_became_active":"2024-01-16T15:03:03.754199+0000","last_became_peered":"2024-01-16T15:03:03.754199+0000","last_unstale":"2024-01-16T15:03:03.758833+0000","last_undegraded":"2024-01-16T15:03:03.758833+0000","last_fullsized":"2024-01-16T15:03:03.758833+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:56.424560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027343+0000","last_change":"2024-01-16T15:03:03.748787+0000","last_active":"2024-01-16T15:03:05.027343+0000","last_peered":"2024-01-16T15:03:05.027343+0000","last_clean":"2024-01-16T15:03:05.027343+0000","last_became_active":"2024-01-16T15:03:03.747052+0000","last_became_peered":"2024-01-16T15:03:03.747052+0000","last_unstale":"2024-01-16T15:03:05.027343+0000","last_undegraded":"2024-01-16T15:03:05.027343+0000","last_fullsized":"2024-01-16T15:03:05.027343+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:58.887884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.762491+0000","last_change":"2024-01-16T15:03:03.755138+0000","last_active":"2024-01-16T15:03:03.762491+0000","last_peered":"2024-01-16T15:03:03.762491+0000","last_clean":"2024-01-16T15:03:03.762491+0000","last_became_active":"2024-01-16T15:03:03.754647+0000","last_became_peered":"2024-01-16T15:03:03.754647+0000","last_unstale":"2024-01-16T15:03:03.762491+0000","last_undegraded":"2024-01-16T15:03:03.762491+0000","last_fullsized":"2024-01-16T15:03:03.762491+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:23.421216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"46'4","reported_seq":15,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.074382+0000","last_change":"2024-01-16T15:03:02.747015+0000","last_active":"2024-01-16T15:03:04.074382+0000","last_peered":"2024-01-16T15:03:04.074382+0000","last_clean":"2024-01-16T15:03:04.074382+0000","last_became_active":"2024-01-16T15:03:02.746705+0000","last_became_peered":"2024-01-16T15:03:02.746705+0000","last_unstale":"2024-01-16T15:03:04.074382+0000","last_undegraded":"2024-01-16T15:03:04.074382+0000","last_fullsized":"2024-01-16T15:03:04.074382+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:42:44.105957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525709+0000","last_change":"2024-01-16T15:03:03.761945+0000","last_active":"2024-01-16T15:03:05.525709+0000","last_peered":"2024-01-16T15:03:05.525709+0000","last_clean":"2024-01-16T15:03:05.525709+0000","last_became_active":"2024-01-16T15:03:03.761691+0000","last_became_peered":"2024-01-16T15:03:03.761691+0000","last_unstale":"2024-01-16T15:03:05.525709+0000","last_undegraded":"2024-01-16T15:03:05.525709+0000","last_fullsized":"2024-01-16T15:03:05.525709+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:36.354147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733081+0000","last_change":"2024-01-16T15:03:02.744417+0000","last_active":"2024-01-16T15:03:04.733081+0000","last_peered":"2024-01-16T15:03:04.733081+0000","last_clean":"2024-01-16T15:03:04.733081+0000","last_became_active":"2024-01-16T15:03:02.744117+0000","last_became_peered":"2024-01-16T15:03:02.744117+0000","last_unstale":"2024-01-16T15:03:04.733081+0000","last_undegraded":"2024-01-16T15:03:04.733081+0000","last_fullsized":"2024-01-16T15:03:04.733081+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:54:23.826072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768169+0000","last_change":"2024-01-16T15:03:03.764508+0000","last_active":"2024-01-16T15:03:03.768169+0000","last_peered":"2024-01-16T15:03:03.768169+0000","last_clean":"2024-01-16T15:03:03.768169+0000","last_became_active":"2024-01-16T15:03:03.764247+0000","last_became_peered":"2024-01-16T15:03:03.764247+0000","last_unstale":"2024-01-16T15:03:03.768169+0000","last_undegraded":"2024-01-16T15:03:03.768169+0000","last_fullsized":"2024-01-16T15:03:03.768169+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:56.336015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.077003+0000","last_change":"2024-01-16T15:03:02.738219+0000","last_active":"2024-01-16T15:03:04.077003+0000","last_peered":"2024-01-16T15:03:04.077003+0000","last_clean":"2024-01-16T15:03:04.077003+0000","last_became_active":"2024-01-16T15:03:02.737153+0000","last_became_peered":"2024-01-16T15:03:02.737153+0000","last_unstale":"2024-01-16T15:03:04.077003+0000","last_undegraded":"2024-01-16T15:03:04.077003+0000","last_fullsized":"2024-01-16T15:03:04.077003+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:55.319844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.764636+0000","last_change":"2024-01-16T15:03:03.756833+0000","last_active":"2024-01-16T15:03:03.764636+0000","last_peered":"2024-01-16T15:03:03.764636+0000","last_clean":"2024-01-16T15:03:03.764636+0000","last_became_active":"2024-01-16T15:03:03.756549+0000","last_became_peered":"2024-01-16T15:03:03.756549+0000","last_unstale":"2024-01-16T15:03:03.764636+0000","last_undegraded":"2024-01-16T15:03:03.764636+0000","last_fullsized":"2024-01-16T15:03:03.764636+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:13.357896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729953+0000","last_change":"2024-01-16T15:03:02.739810+0000","last_active":"2024-01-16T15:03:04.729953+0000","last_peered":"2024-01-16T15:03:04.729953+0000","last_clean":"2024-01-16T15:03:04.729953+0000","last_became_active":"2024-01-16T15:03:02.739459+0000","last_became_peered":"2024-01-16T15:03:02.739459+0000","last_unstale":"2024-01-16T15:03:04.729953+0000","last_undegraded":"2024-01-16T15:03:04.729953+0000","last_fullsized":"2024-01-16T15:03:04.729953+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:00:02.046251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732573+0000","last_change":"2024-01-16T15:03:03.756387+0000","last_active":"2024-01-16T15:03:04.732573+0000","last_peered":"2024-01-16T15:03:04.732573+0000","last_clean":"2024-01-16T15:03:04.732573+0000","last_became_active":"2024-01-16T15:03:03.756069+0000","last_became_peered":"2024-01-16T15:03:03.756069+0000","last_unstale":"2024-01-16T15:03:04.732573+0000","last_undegraded":"2024-01-16T15:03:04.732573+0000","last_fullsized":"2024-01-16T15:03:04.732573+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:16.134109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"46'1","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.072859+0000","last_change":"2024-01-16T15:03:02.745265+0000","last_active":"2024-01-16T15:03:04.072859+0000","last_peered":"2024-01-16T15:03:04.072859+0000","last_clean":"2024-01-16T15:03:04.072859+0000","last_became_active":"2024-01-16T15:03:02.743903+0000","last_became_peered":"2024-01-16T15:03:02.743903+0000","last_unstale":"2024-01-16T15:03:04.072859+0000","last_undegraded":"2024-01-16T15:03:04.072859+0000","last_fullsized":"2024-01-16T15:03:04.072859+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:13.383937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771059+0000","last_change":"2024-01-16T15:03:03.766518+0000","last_active":"2024-01-16T15:03:03.771059+0000","last_peered":"2024-01-16T15:03:03.771059+0000","last_clean":"2024-01-16T15:03:03.771059+0000","last_became_active":"2024-01-16T15:03:03.765246+0000","last_became_peered":"2024-01-16T15:03:03.765246+0000","last_unstale":"2024-01-16T15:03:03.771059+0000","last_undegraded":"2024-01-16T15:03:03.771059+0000","last_fullsized":"2024-01-16T15:03:03.771059+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:22.071687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732666+0000","last_change":"2024-01-16T15:03:02.738149+0000","last_active":"2024-01-16T15:03:04.732666+0000","last_peered":"2024-01-16T15:03:04.732666+0000","last_clean":"2024-01-16T15:03:04.732666+0000","last_became_active":"2024-01-16T15:03:02.736301+0000","last_became_peered":"2024-01-16T15:03:02.736301+0000","last_unstale":"2024-01-16T15:03:04.732666+0000","last_undegraded":"2024-01-16T15:03:04.732666+0000","last_fullsized":"2024-01-16T15:03:04.732666+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:22.204469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.769161+0000","last_change":"2024-01-16T15:03:03.758057+0000","last_active":"2024-01-16T15:03:03.769161+0000","last_peered":"2024-01-16T15:03:03.769161+0000","last_clean":"2024-01-16T15:03:03.769161+0000","last_became_active":"2024-01-16T15:03:03.757782+0000","last_became_peered":"2024-01-16T15:03:03.757782+0000","last_unstale":"2024-01-16T15:03:03.769161+0000","last_undegraded":"2024-01-16T15:03:03.769161+0000","last_fullsized":"2024-01-16T15:03:03.769161+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:56:33.626652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.729004+0000","last_change":"2024-01-16T15:03:02.744706+0000","last_active":"2024-01-16T15:03:03.729004+0000","last_peered":"2024-01-16T15:03:03.729004+0000","last_clean":"2024-01-16T15:03:03.729004+0000","last_became_active":"2024-01-16T15:03:02.743633+0000","last_became_peered":"2024-01-16T15:03:02.743633+0000","last_unstale":"2024-01-16T15:03:03.729004+0000","last_undegraded":"2024-01-16T15:03:03.729004+0000","last_fullsized":"2024-01-16T15:03:03.729004+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:05:28.811804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525178+0000","last_change":"2024-01-16T15:03:03.758610+0000","last_active":"2024-01-16T15:03:05.525178+0000","last_peered":"2024-01-16T15:03:05.525178+0000","last_clean":"2024-01-16T15:03:05.525178+0000","last_became_active":"2024-01-16T15:03:03.758351+0000","last_became_peered":"2024-01-16T15:03:03.758351+0000","last_unstale":"2024-01-16T15:03:05.525178+0000","last_undegraded":"2024-01-16T15:03:05.525178+0000","last_fullsized":"2024-01-16T15:03:05.525178+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:05:42.600626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.743994+0000","last_change":"2024-01-16T15:03:02.745507+0000","last_active":"2024-01-16T15:03:03.743994+0000","last_peered":"2024-01-16T15:03:03.743994+0000","last_clean":"2024-01-16T15:03:03.743994+0000","last_became_active":"2024-01-16T15:03:02.745183+0000","last_became_peered":"2024-01-16T15:03:02.745183+0000","last_unstale":"2024-01-16T15:03:03.743994+0000","last_undegraded":"2024-01-16T15:03:03.743994+0000","last_fullsized":"2024-01-16T15:03:03.743994+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:15:12.341724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733064+0000","last_change":"2024-01-16T15:03:03.767005+0000","last_active":"2024-01-16T15:03:04.733064+0000","last_peered":"2024-01-16T15:03:04.733064+0000","last_clean":"2024-01-16T15:03:04.733064+0000","last_became_active":"2024-01-16T15:03:03.766717+0000","last_became_peered":"2024-01-16T15:03:03.766717+0000","last_unstale":"2024-01-16T15:03:04.733064+0000","last_undegraded":"2024-01-16T15:03:04.733064+0000","last_fullsized":"2024-01-16T15:03:04.733064+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:27:09.541105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733498+0000","last_change":"2024-01-16T15:03:02.743197+0000","last_active":"2024-01-16T15:03:04.733498+0000","last_peered":"2024-01-16T15:03:04.733498+0000","last_clean":"2024-01-16T15:03:04.733498+0000","last_became_active":"2024-01-16T15:03:02.740663+0000","last_became_peered":"2024-01-16T15:03:02.740663+0000","last_unstale":"2024-01-16T15:03:04.733498+0000","last_undegraded":"2024-01-16T15:03:04.733498+0000","last_fullsized":"2024-01-16T15:03:04.733498+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:01:14.437345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730675+0000","last_change":"2024-01-16T15:03:03.750469+0000","last_active":"2024-01-16T15:03:04.730675+0000","last_peered":"2024-01-16T15:03:04.730675+0000","last_clean":"2024-01-16T15:03:04.730675+0000","last_became_active":"2024-01-16T15:03:03.750080+0000","last_became_peered":"2024-01-16T15:03:03.750080+0000","last_unstale":"2024-01-16T15:03:04.730675+0000","last_undegraded":"2024-01-16T15:03:04.730675+0000","last_fullsized":"2024-01-16T15:03:04.730675+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:51:24.974640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.751476+0000","last_change":"2024-01-16T15:03:02.742151+0000","last_active":"2024-01-16T15:03:03.751476+0000","last_peered":"2024-01-16T15:03:03.751476+0000","last_clean":"2024-01-16T15:03:03.751476+0000","last_became_active":"2024-01-16T15:03:02.741865+0000","last_became_peered":"2024-01-16T15:03:02.741865+0000","last_unstale":"2024-01-16T15:03:03.751476+0000","last_undegraded":"2024-01-16T15:03:03.751476+0000","last_fullsized":"2024-01-16T15:03:03.751476+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:45.978118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.770007+0000","last_change":"2024-01-16T15:03:03.764762+0000","last_active":"2024-01-16T15:03:03.770007+0000","last_peered":"2024-01-16T15:03:03.770007+0000","last_clean":"2024-01-16T15:03:03.770007+0000","last_became_active":"2024-01-16T15:03:03.764485+0000","last_became_peered":"2024-01-16T15:03:03.764485+0000","last_unstale":"2024-01-16T15:03:03.770007+0000","last_undegraded":"2024-01-16T15:03:03.770007+0000","last_fullsized":"2024-01-16T15:03:03.770007+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:54:35.292757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"46'3","reported_seq":15,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730591+0000","last_change":"2024-01-16T15:03:02.746772+0000","last_active":"2024-01-16T15:03:04.730591+0000","last_peered":"2024-01-16T15:03:04.730591+0000","last_clean":"2024-01-16T15:03:04.730591+0000","last_became_active":"2024-01-16T15:03:02.746453+0000","last_became_peered":"2024-01-16T15:03:02.746453+0000","last_unstale":"2024-01-16T15:03:04.730591+0000","last_undegraded":"2024-01-16T15:03:04.730591+0000","last_fullsized":"2024-01-16T15:03:04.730591+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:41.169885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730362+0000","last_change":"2024-01-16T15:03:03.759965+0000","last_active":"2024-01-16T15:03:04.730362+0000","last_peered":"2024-01-16T15:03:04.730362+0000","last_clean":"2024-01-16T15:03:04.730362+0000","last_became_active":"2024-01-16T15:03:03.759090+0000","last_became_peered":"2024-01-16T15:03:03.759090+0000","last_unstale":"2024-01-16T15:03:04.730362+0000","last_undegraded":"2024-01-16T15:03:04.730362+0000","last_fullsized":"2024-01-16T15:03:04.730362+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:13.538220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"46'6","reported_seq":18,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027436+0000","last_change":"2024-01-16T15:03:02.738505+0000","last_active":"2024-01-16T15:03:05.027436+0000","last_peered":"2024-01-16T15:03:05.027436+0000","last_clean":"2024-01-16T15:03:05.027436+0000","last_became_active":"2024-01-16T15:03:02.738235+0000","last_became_peered":"2024-01-16T15:03:02.738235+0000","last_unstale":"2024-01-16T15:03:05.027436+0000","last_undegraded":"2024-01-16T15:03:05.027436+0000","last_fullsized":"2024-01-16T15:03:05.027436+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:03:51.358036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525814+0000","last_change":"2024-01-16T15:03:03.768539+0000","last_active":"2024-01-16T15:03:05.525814+0000","last_peered":"2024-01-16T15:03:05.525814+0000","last_clean":"2024-01-16T15:03:05.525814+0000","last_became_active":"2024-01-16T15:03:03.768201+0000","last_became_peered":"2024-01-16T15:03:03.768201+0000","last_unstale":"2024-01-16T15:03:05.525814+0000","last_undegraded":"2024-01-16T15:03:05.525814+0000","last_fullsized":"2024-01-16T15:03:05.525814+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:23.220641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.069124+0000","last_change":"2024-01-16T15:03:02.750156+0000","last_active":"2024-01-16T15:03:04.069124+0000","last_peered":"2024-01-16T15:03:04.069124+0000","last_clean":"2024-01-16T15:03:04.069124+0000","last_became_active":"2024-01-16T15:03:02.749815+0000","last_became_peered":"2024-01-16T15:03:02.749815+0000","last_unstale":"2024-01-16T15:03:04.069124+0000","last_undegraded":"2024-01-16T15:03:04.069124+0000","last_fullsized":"2024-01-16T15:03:04.069124+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:13:09.262264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768603+0000","last_change":"2024-01-16T15:03:03.764953+0000","last_active":"2024-01-16T15:03:03.768603+0000","last_peered":"2024-01-16T15:03:03.768603+0000","last_clean":"2024-01-16T15:03:03.768603+0000","last_became_active":"2024-01-16T15:03:03.756011+0000","last_became_peered":"2024-01-16T15:03:03.756011+0000","last_unstale":"2024-01-16T15:03:03.768603+0000","last_undegraded":"2024-01-16T15:03:03.768603+0000","last_fullsized":"2024-01-16T15:03:03.768603+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:57.932933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729552+0000","last_change":"2024-01-16T15:03:02.737611+0000","last_active":"2024-01-16T15:03:04.729552+0000","last_peered":"2024-01-16T15:03:04.729552+0000","last_clean":"2024-01-16T15:03:04.729552+0000","last_became_active":"2024-01-16T15:03:02.737290+0000","last_became_peered":"2024-01-16T15:03:02.737290+0000","last_unstale":"2024-01-16T15:03:04.729552+0000","last_undegraded":"2024-01-16T15:03:04.729552+0000","last_fullsized":"2024-01-16T15:03:04.729552+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:56.916762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767418+0000","last_change":"2024-01-16T15:03:03.762312+0000","last_active":"2024-01-16T15:03:03.767418+0000","last_peered":"2024-01-16T15:03:03.767418+0000","last_clean":"2024-01-16T15:03:03.767418+0000","last_became_active":"2024-01-16T15:03:03.762036+0000","last_became_peered":"2024-01-16T15:03:03.762036+0000","last_unstale":"2024-01-16T15:03:03.767418+0000","last_undegraded":"2024-01-16T15:03:03.767418+0000","last_fullsized":"2024-01-16T15:03:03.767418+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:10:23.220641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.742331+0000","last_change":"2024-01-16T15:03:02.749895+0000","last_active":"2024-01-16T15:03:03.742331+0000","last_peered":"2024-01-16T15:03:03.742331+0000","last_clean":"2024-01-16T15:03:03.742331+0000","last_became_active":"2024-01-16T15:03:02.749499+0000","last_became_peered":"2024-01-16T15:03:02.749499+0000","last_unstale":"2024-01-16T15:03:03.742331+0000","last_undegraded":"2024-01-16T15:03:03.742331+0000","last_fullsized":"2024-01-16T15:03:03.742331+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:58.904271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729443+0000","last_change":"2024-01-16T15:03:03.761477+0000","last_active":"2024-01-16T15:03:04.729443+0000","last_peered":"2024-01-16T15:03:04.729443+0000","last_clean":"2024-01-16T15:03:04.729443+0000","last_became_active":"2024-01-16T15:03:03.761137+0000","last_became_peered":"2024-01-16T15:03:03.761137+0000","last_unstale":"2024-01-16T15:03:04.729443+0000","last_undegraded":"2024-01-16T15:03:04.729443+0000","last_fullsized":"2024-01-16T15:03:04.729443+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:56:58.032962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"46'8","reported_seq":19,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.076906+0000","last_change":"2024-01-16T15:03:02.751726+0000","last_active":"2024-01-16T15:03:04.076906+0000","last_peered":"2024-01-16T15:03:04.076906+0000","last_clean":"2024-01-16T15:03:04.076906+0000","last_became_active":"2024-01-16T15:03:02.751314+0000","last_became_peered":"2024-01-16T15:03:02.751314+0000","last_unstale":"2024-01-16T15:03:04.076906+0000","last_undegraded":"2024-01-16T15:03:04.076906+0000","last_fullsized":"2024-01-16T15:03:04.076906+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:23.092743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027379+0000","last_change":"2024-01-16T15:03:03.737335+0000","last_active":"2024-01-16T15:03:05.027379+0000","last_peered":"2024-01-16T15:03:05.027379+0000","last_clean":"2024-01-16T15:03:05.027379+0000","last_became_active":"2024-01-16T15:03:03.736322+0000","last_became_peered":"2024-01-16T15:03:03.736322+0000","last_unstale":"2024-01-16T15:03:05.027379+0000","last_undegraded":"2024-01-16T15:03:05.027379+0000","last_fullsized":"2024-01-16T15:03:05.027379+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:41.092236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"46'1","reported_seq":13,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027375+0000","last_change":"2024-01-16T15:03:02.741842+0000","last_active":"2024-01-16T15:03:05.027375+0000","last_peered":"2024-01-16T15:03:05.027375+0000","last_clean":"2024-01-16T15:03:05.027375+0000","last_became_active":"2024-01-16T15:03:02.740116+0000","last_became_peered":"2024-01-16T15:03:02.740116+0000","last_unstale":"2024-01-16T15:03:05.027375+0000","last_undegraded":"2024-01-16T15:03:05.027375+0000","last_fullsized":"2024-01-16T15:03:05.027375+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:08.333486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.768188+0000","last_change":"2024-01-16T15:03:03.756583+0000","last_active":"2024-01-16T15:03:03.768188+0000","last_peered":"2024-01-16T15:03:03.768188+0000","last_clean":"2024-01-16T15:03:03.768188+0000","last_became_active":"2024-01-16T15:03:03.756248+0000","last_became_peered":"2024-01-16T15:03:03.756248+0000","last_unstale":"2024-01-16T15:03:03.768188+0000","last_undegraded":"2024-01-16T15:03:03.768188+0000","last_fullsized":"2024-01-16T15:03:03.768188+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:48.552742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"46'2","reported_seq":14,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.733457+0000","last_change":"2024-01-16T15:03:02.743957+0000","last_active":"2024-01-16T15:03:04.733457+0000","last_peered":"2024-01-16T15:03:04.733457+0000","last_clean":"2024-01-16T15:03:04.733457+0000","last_became_active":"2024-01-16T15:03:02.743681+0000","last_became_peered":"2024-01-16T15:03:02.743681+0000","last_unstale":"2024-01-16T15:03:04.733457+0000","last_undegraded":"2024-01-16T15:03:04.733457+0000","last_fullsized":"2024-01-16T15:03:04.733457+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:12.858437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525102+0000","last_change":"2024-01-16T15:03:03.770866+0000","last_active":"2024-01-16T15:03:05.525102+0000","last_peered":"2024-01-16T15:03:05.525102+0000","last_clean":"2024-01-16T15:03:05.525102+0000","last_became_active":"2024-01-16T15:03:03.770561+0000","last_became_peered":"2024-01-16T15:03:03.770561+0000","last_unstale":"2024-01-16T15:03:05.525102+0000","last_undegraded":"2024-01-16T15:03:05.525102+0000","last_fullsized":"2024-01-16T15:03:05.525102+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:18.862583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"46'1","reported_seq":12,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.076481+0000","last_change":"2024-01-16T15:03:02.742090+0000","last_active":"2024-01-16T15:03:04.076481+0000","last_peered":"2024-01-16T15:03:04.076481+0000","last_clean":"2024-01-16T15:03:04.076481+0000","last_became_active":"2024-01-16T15:03:02.741828+0000","last_became_peered":"2024-01-16T15:03:02.741828+0000","last_unstale":"2024-01-16T15:03:04.076481+0000","last_undegraded":"2024-01-16T15:03:04.076481+0000","last_fullsized":"2024-01-16T15:03:04.076481+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:28:56.916762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525257+0000","last_change":"2024-01-16T15:03:03.763483+0000","last_active":"2024-01-16T15:03:05.525257+0000","last_peered":"2024-01-16T15:03:05.525257+0000","last_clean":"2024-01-16T15:03:05.525257+0000","last_became_active":"2024-01-16T15:03:03.762707+0000","last_became_peered":"2024-01-16T15:03:03.762707+0000","last_unstale":"2024-01-16T15:03:05.525257+0000","last_undegraded":"2024-01-16T15:03:05.525257+0000","last_fullsized":"2024-01-16T15:03:05.525257+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:54:24.842244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.755594+0000","last_change":"2024-01-16T15:03:02.744774+0000","last_active":"2024-01-16T15:03:03.755594+0000","last_peered":"2024-01-16T15:03:03.755594+0000","last_clean":"2024-01-16T15:03:03.755594+0000","last_became_active":"2024-01-16T15:03:02.744496+0000","last_became_peered":"2024-01-16T15:03:02.744496+0000","last_unstale":"2024-01-16T15:03:03.755594+0000","last_undegraded":"2024-01-16T15:03:03.755594+0000","last_fullsized":"2024-01-16T15:03:03.755594+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:44.812709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767226+0000","last_change":"2024-01-16T15:03:03.763309+0000","last_active":"2024-01-16T15:03:03.767226+0000","last_peered":"2024-01-16T15:03:03.767226+0000","last_clean":"2024-01-16T15:03:03.767226+0000","last_became_active":"2024-01-16T15:03:03.762966+0000","last_became_peered":"2024-01-16T15:03:03.762966+0000","last_unstale":"2024-01-16T15:03:03.767226+0000","last_undegraded":"2024-01-16T15:03:03.767226+0000","last_fullsized":"2024-01-16T15:03:03.767226+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:09:16.134109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.073247+0000","last_change":"2024-01-16T15:03:02.746052+0000","last_active":"2024-01-16T15:03:04.073247+0000","last_peered":"2024-01-16T15:03:04.073247+0000","last_clean":"2024-01-16T15:03:04.073247+0000","last_became_active":"2024-01-16T15:03:02.744858+0000","last_became_peered":"2024-01-16T15:03:02.744858+0000","last_unstale":"2024-01-16T15:03:04.073247+0000","last_undegraded":"2024-01-16T15:03:04.073247+0000","last_fullsized":"2024-01-16T15:03:04.073247+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:05:00.408282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.758276+0000","last_change":"2024-01-16T15:03:03.753202+0000","last_active":"2024-01-16T15:03:03.758276+0000","last_peered":"2024-01-16T15:03:03.758276+0000","last_clean":"2024-01-16T15:03:03.758276+0000","last_became_active":"2024-01-16T15:03:03.752916+0000","last_became_peered":"2024-01-16T15:03:03.752916+0000","last_unstale":"2024-01-16T15:03:03.758276+0000","last_undegraded":"2024-01-16T15:03:03.758276+0000","last_fullsized":"2024-01-16T15:03:03.758276+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:46.559713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.731711+0000","last_change":"2024-01-16T15:03:02.746271+0000","last_active":"2024-01-16T15:03:03.731711+0000","last_peered":"2024-01-16T15:03:03.731711+0000","last_clean":"2024-01-16T15:03:03.731711+0000","last_became_active":"2024-01-16T15:03:02.745996+0000","last_became_peered":"2024-01-16T15:03:02.745996+0000","last_unstale":"2024-01-16T15:03:03.731711+0000","last_undegraded":"2024-01-16T15:03:03.731711+0000","last_fullsized":"2024-01-16T15:03:03.731711+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:22:07.483449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.027548+0000","last_change":"2024-01-16T15:03:03.750756+0000","last_active":"2024-01-16T15:03:05.027548+0000","last_peered":"2024-01-16T15:03:05.027548+0000","last_clean":"2024-01-16T15:03:05.027548+0000","last_became_active":"2024-01-16T15:03:03.750170+0000","last_became_peered":"2024-01-16T15:03:03.750170+0000","last_unstale":"2024-01-16T15:03:05.027548+0000","last_undegraded":"2024-01-16T15:03:05.027548+0000","last_fullsized":"2024-01-16T15:03:05.027548+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:45:37.475530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.743830+0000","last_change":"2024-01-16T15:03:02.743223+0000","last_active":"2024-01-16T15:03:03.743830+0000","last_peered":"2024-01-16T15:03:03.743830+0000","last_clean":"2024-01-16T15:03:03.743830+0000","last_became_active":"2024-01-16T15:03:02.742953+0000","last_became_peered":"2024-01-16T15:03:02.742953+0000","last_unstale":"2024-01-16T15:03:03.743830+0000","last_undegraded":"2024-01-16T15:03:03.743830+0000","last_fullsized":"2024-01-16T15:03:03.743830+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:11:22.405045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.765260+0000","last_change":"2024-01-16T15:03:03.751298+0000","last_active":"2024-01-16T15:03:03.765260+0000","last_peered":"2024-01-16T15:03:03.765260+0000","last_clean":"2024-01-16T15:03:03.765260+0000","last_became_active":"2024-01-16T15:03:03.750669+0000","last_became_peered":"2024-01-16T15:03:03.750669+0000","last_unstale":"2024-01-16T15:03:03.765260+0000","last_undegraded":"2024-01-16T15:03:03.765260+0000","last_fullsized":"2024-01-16T15:03:03.765260+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:26:29.856711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.729296+0000","last_change":"2024-01-16T15:03:02.737666+0000","last_active":"2024-01-16T15:03:04.729296+0000","last_peered":"2024-01-16T15:03:04.729296+0000","last_clean":"2024-01-16T15:03:04.729296+0000","last_became_active":"2024-01-16T15:03:02.737386+0000","last_became_peered":"2024-01-16T15:03:02.737386+0000","last_unstale":"2024-01-16T15:03:04.729296+0000","last_undegraded":"2024-01-16T15:03:04.729296+0000","last_fullsized":"2024-01-16T15:03:04.729296+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:21.055515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:05.525680+0000","last_change":"2024-01-16T15:03:03.770134+0000","last_active":"2024-01-16T15:03:05.525680+0000","last_peered":"2024-01-16T15:03:05.525680+0000","last_clean":"2024-01-16T15:03:05.525680+0000","last_became_active":"2024-01-16T15:03:03.769815+0000","last_became_peered":"2024-01-16T15:03:03.769815+0000","last_unstale":"2024-01-16T15:03:05.525680+0000","last_undegraded":"2024-01-16T15:03:05.525680+0000","last_fullsized":"2024-01-16T15:03:05.525680+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:44.028816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.740920+0000","last_change":"2024-01-16T15:03:02.743715+0000","last_active":"2024-01-16T15:03:03.740920+0000","last_peered":"2024-01-16T15:03:03.740920+0000","last_clean":"2024-01-16T15:03:03.740920+0000","last_became_active":"2024-01-16T15:03:02.740742+0000","last_became_peered":"2024-01-16T15:03:02.740742+0000","last_unstale":"2024-01-16T15:03:03.740920+0000","last_undegraded":"2024-01-16T15:03:03.740920+0000","last_fullsized":"2024-01-16T15:03:03.740920+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:15.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.767951+0000","last_change":"2024-01-16T15:03:03.756955+0000","last_active":"2024-01-16T15:03:03.767951+0000","last_peered":"2024-01-16T15:03:03.767951+0000","last_clean":"2024-01-16T15:03:03.767951+0000","last_became_active":"2024-01-16T15:03:03.756648+0000","last_became_peered":"2024-01-16T15:03:03.756648+0000","last_unstale":"2024-01-16T15:03:03.767951+0000","last_undegraded":"2024-01-16T15:03:03.767951+0000","last_fullsized":"2024-01-16T15:03:03.767951+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:05:29.827975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730790+0000","last_change":"2024-01-16T15:03:02.747244+0000","last_active":"2024-01-16T15:03:04.730790+0000","last_peered":"2024-01-16T15:03:04.730790+0000","last_clean":"2024-01-16T15:03:04.730790+0000","last_became_active":"2024-01-16T15:03:02.746069+0000","last_became_peered":"2024-01-16T15:03:02.746069+0000","last_unstale":"2024-01-16T15:03:04.730790+0000","last_undegraded":"2024-01-16T15:03:04.730790+0000","last_fullsized":"2024-01-16T15:03:04.730790+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:42.172985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730616+0000","last_change":"2024-01-16T15:03:03.755079+0000","last_active":"2024-01-16T15:03:04.730616+0000","last_peered":"2024-01-16T15:03:04.730616+0000","last_clean":"2024-01-16T15:03:04.730616+0000","last_became_active":"2024-01-16T15:03:03.754251+0000","last_became_peered":"2024-01-16T15:03:03.754251+0000","last_unstale":"2024-01-16T15:03:04.730616+0000","last_undegraded":"2024-01-16T15:03:04.730616+0000","last_fullsized":"2024-01-16T15:03:04.730616+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:07:04.449847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.739677+0000","last_change":"2024-01-16T15:03:02.741707+0000","last_active":"2024-01-16T15:03:03.739677+0000","last_peered":"2024-01-16T15:03:03.739677+0000","last_clean":"2024-01-16T15:03:03.739677+0000","last_became_active":"2024-01-16T15:03:02.741350+0000","last_became_peered":"2024-01-16T15:03:02.741350+0000","last_unstale":"2024-01-16T15:03:03.739677+0000","last_undegraded":"2024-01-16T15:03:03.739677+0000","last_fullsized":"2024-01-16T15:03:03.739677+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:31.384210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.763188+0000","last_change":"2024-01-16T15:03:03.759755+0000","last_active":"2024-01-16T15:03:03.763188+0000","last_peered":"2024-01-16T15:03:03.763188+0000","last_clean":"2024-01-16T15:03:03.763188+0000","last_became_active":"2024-01-16T15:03:03.758872+0000","last_became_peered":"2024-01-16T15:03:03.758872+0000","last_unstale":"2024-01-16T15:03:03.763188+0000","last_undegraded":"2024-01-16T15:03:03.763188+0000","last_fullsized":"2024-01-16T15:03:03.763188+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:09:33.207831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.730072+0000","last_change":"2024-01-16T15:03:02.747745+0000","last_active":"2024-01-16T15:03:04.730072+0000","last_peered":"2024-01-16T15:03:04.730072+0000","last_clean":"2024-01-16T15:03:04.730072+0000","last_became_active":"2024-01-16T15:03:02.747445+0000","last_became_peered":"2024-01-16T15:03:02.747445+0000","last_unstale":"2024-01-16T15:03:04.730072+0000","last_undegraded":"2024-01-16T15:03:04.730072+0000","last_fullsized":"2024-01-16T15:03:04.730072+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:59.087146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.769920+0000","last_change":"2024-01-16T15:03:03.764495+0000","last_active":"2024-01-16T15:03:03.769920+0000","last_peered":"2024-01-16T15:03:03.769920+0000","last_clean":"2024-01-16T15:03:03.769920+0000","last_became_active":"2024-01-16T15:03:03.764246+0000","last_became_peered":"2024-01-16T15:03:03.764246+0000","last_unstale":"2024-01-16T15:03:03.769920+0000","last_undegraded":"2024-01-16T15:03:03.769920+0000","last_fullsized":"2024-01-16T15:03:03.769920+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:42.994598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.732809+0000","last_change":"2024-01-16T15:03:02.743333+0000","last_active":"2024-01-16T15:03:04.732809+0000","last_peered":"2024-01-16T15:03:04.732809+0000","last_clean":"2024-01-16T15:03:04.732809+0000","last_became_active":"2024-01-16T15:03:02.742989+0000","last_became_peered":"2024-01-16T15:03:02.742989+0000","last_unstale":"2024-01-16T15:03:04.732809+0000","last_undegraded":"2024-01-16T15:03:04.732809+0000","last_fullsized":"2024-01-16T15:03:04.732809+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:15.541618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.753216+0000","last_change":"2024-01-16T15:03:03.747977+0000","last_active":"2024-01-16T15:03:03.753216+0000","last_peered":"2024-01-16T15:03:03.753216+0000","last_clean":"2024-01-16T15:03:03.753216+0000","last_became_active":"2024-01-16T15:03:03.747699+0000","last_became_peered":"2024-01-16T15:03:03.747699+0000","last_unstale":"2024-01-16T15:03:03.753216+0000","last_undegraded":"2024-01-16T15:03:03.753216+0000","last_fullsized":"2024-01-16T15:03:03.753216+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:46.994289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.738068+0000","last_change":"2024-01-16T15:03:02.744665+0000","last_active":"2024-01-16T15:03:03.738068+0000","last_peered":"2024-01-16T15:03:03.738068+0000","last_clean":"2024-01-16T15:03:03.738068+0000","last_became_active":"2024-01-16T15:03:02.744384+0000","last_became_peered":"2024-01-16T15:03:02.744384+0000","last_unstale":"2024-01-16T15:03:03.738068+0000","last_undegraded":"2024-01-16T15:03:03.738068+0000","last_fullsized":"2024-01-16T15:03:03.738068+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:23.092743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.731961+0000","last_change":"2024-01-16T15:03:03.759969+0000","last_active":"2024-01-16T15:03:04.731961+0000","last_peered":"2024-01-16T15:03:04.731961+0000","last_clean":"2024-01-16T15:03:04.731961+0000","last_became_active":"2024-01-16T15:03:03.759720+0000","last_became_peered":"2024-01-16T15:03:03.759720+0000","last_unstale":"2024-01-16T15:03:04.731961+0000","last_undegraded":"2024-01-16T15:03:04.731961+0000","last_fullsized":"2024-01-16T15:03:04.731961+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:56.424560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":11,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.744814+0000","last_change":"2024-01-16T15:03:02.751074+0000","last_active":"2024-01-16T15:03:03.744814+0000","last_peered":"2024-01-16T15:03:03.744814+0000","last_clean":"2024-01-16T15:03:03.744814+0000","last_became_active":"2024-01-16T15:03:02.750131+0000","last_became_peered":"2024-01-16T15:03:02.750131+0000","last_unstale":"2024-01-16T15:03:03.744814+0000","last_undegraded":"2024-01-16T15:03:03.744814+0000","last_fullsized":"2024-01-16T15:03:03.744814+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:41:55.319844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":9,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.771092+0000","last_change":"2024-01-16T15:03:03.767680+0000","last_active":"2024-01-16T15:03:03.771092+0000","last_peered":"2024-01-16T15:03:03.771092+0000","last_clean":"2024-01-16T15:03:03.771092+0000","last_became_active":"2024-01-16T15:03:03.767262+0000","last_became_peered":"2024-01-16T15:03:03.767262+0000","last_unstale":"2024-01-16T15:03:03.771092+0000","last_undegraded":"2024-01-16T15:03:03.771092+0000","last_fullsized":"2024-01-16T15:03:03.771092+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:02.702922+0000","last_clean_scrub_stamp":"2024-01-16T15:03:02.702922+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:41:32.400381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"46'2","reported_seq":13,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:04.076052+0000","last_change":"2024-01-16T15:03:02.748067+0000","last_active":"2024-01-16T15:03:04.076052+0000","last_peered":"2024-01-16T15:03:04.076052+0000","last_clean":"2024-01-16T15:03:04.076052+0000","last_became_active":"2024-01-16T15:03:02.747807+0000","last_became_peered":"2024-01-16T15:03:02.747807+0000","last_unstale":"2024-01-16T15:03:04.076052+0000","last_undegraded":"2024-01-16T15:03:04.076052+0000","last_fullsized":"2024-01-16T15:03:04.076052+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":44,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:03:01.686750+0000","last_clean_scrub_stamp":"2024-01-16T15:03:01.686750+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:01.474775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":187,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-01-16T15:03:03.737188+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:03:03.737188+0000","last_peered":"2024-01-16T15:03:03.737188+0000","last_clean":"2024-01-16T15:03:03.737188+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:03:03.737188+0000","last_undegraded":"2024-01-16T15:03:03.737188+0000","last_fullsized":"2024-01-16T15:03:03.737188+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509482,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31724,"kb_used_data":540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340116,"statfs":{"total":96636764160,"available":96604278784,"internally_reserved":0,"allocated":552960,"data_stored":223029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.2110000000000001,"15min":1.159},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":9.7899999999999991,"5min":9.7899999999999991,"15min":9.7899999999999991},"last":1.23},{"interface":"front","average":{"1min":1.222,"5min":1.171,"15min":1.1679999999999999},"min":{"1min":0.77700000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.8240000000000001,"5min":1.8240000000000001,"15min":1.8240000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.232,"15min":1.153},"min":{"1min":0.84099999999999997,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":8.3379999999999992,"5min":8.3379999999999992,"15min":8.3379999999999992},"last":0.871},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.248,"15min":1.246},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":2.4329999999999998,"5min":2.4329999999999998,"15min":2.4329999999999998},"last":1.0449999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.381,"15min":1.391},"min":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":3.3460000000000001,"5min":3.3460000000000001,"15min":3.3460000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.278,"5min":1.3320000000000001,"15min":1.3759999999999999},"min":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":2.0510000000000002,"5min":2.1840000000000002,"15min":2.1840000000000002},"last":1.1200000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4159999999999999,"15min":1.4339999999999999},"min":{"1min":0.77600000000000002,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":3.665,"5min":3.665,"15min":3.665},"last":1.395},{"interface":"front","average":{"1min":1.417,"5min":1.512,"15min":1.573},"min":{"1min":0.88400000000000001,"5min":0.745,"15min":0.745},"max":{"1min":1.98,"5min":3.5710000000000002,"15min":3.5710000000000002},"last":1.3420000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.431,"15min":1.4810000000000001},"min":{"1min":0.873,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.1520000000000001,"5min":3.8170000000000002,"15min":3.8170000000000002},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.524,"15min":1.5620000000000001},"min":{"1min":0.89700000000000002,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":3.1789999999999998,"5min":3.6629999999999998,"15min":3.6629999999999998},"last":1.597}]},{"osd":5,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.762,"5min":1.7010000000000001,"15min":1.702},"min":{"1min":1.0640000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":3.6299999999999999,"5min":3.6299999999999999,"15min":3.6299999999999999},"last":1.671},{"interface":"front","average":{"1min":1.718,"5min":1.6519999999999999,"15min":1.6599999999999999},"min":{"1min":0.93400000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.798,"5min":3.798,"15min":3.798},"last":1.536}]},{"osd":7,"last update":"Tue Jan 16 15:02:50 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.621,"15min":1.625},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":4.0060000000000002,"5min":4.0060000000000002,"15min":4.0060000000000002},"last":1.298},{"interface":"front","average":{"1min":1.712,"5min":1.708,"15min":1.732},"min":{"1min":1.093,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":3.2709999999999999,"5min":3.4159999999999999,"15min":3.4159999999999999},"last":1.722}]}]},{"osd":4,"up_from":15,"seq":64424509484,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31728,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340112,"statfs":{"total":96636764160,"available":96604274688,"internally_reserved":0,"allocated":557056,"data_stored":223969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.109,"15min":1.0449999999999999},"min":{"1min":0.67700000000000005,"5min":0.622,"15min":0.622},"max":{"1min":5.2969999999999997,"5min":5.2969999999999997,"15min":5.2969999999999997},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.1279999999999999,"15min":1.121},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":2.3450000000000002,"5min":2.3450000000000002,"15min":2.3450000000000002},"last":1.0640000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.175,"5min":1.2370000000000001,"15min":1.1699999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.9370000000000001,"5min":9.9450000000000003,"15min":9.9450000000000003},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.2849999999999999,"15min":1.2410000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":5.04,"5min":5.04,"15min":5.04},"last":1.1060000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3340000000000001,"15min":1.2689999999999999},"min":{"1min":0.80500000000000005,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.871,"5min":10.129,"15min":10.129},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.3520000000000001,"15min":1.3009999999999999},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":5.1920000000000002},"last":1.393}]},{"osd":3,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.3440000000000001,"5min":1.302,"15min":1.3},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.2599999999999998,"5min":2.637,"15min":2.637},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.74,"5min":1.532,"15min":1.4390000000000001},"min":{"1min":0.78500000000000003,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":7.8940000000000001,"5min":9.8109999999999999,"15min":9.8109999999999999},"last":1.345}]},{"osd":5,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":2.1059999999999999,"5min":1.5980000000000001,"15min":1.526},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":11.823,"5min":11.823,"15min":11.823},"last":1.6839999999999999},{"interface":"front","average":{"1min":1.9139999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.68200000000000005,"5min":0.502,"15min":0.502},"max":{"1min":11.936999999999999,"5min":11.936999999999999,"15min":11.936999999999999},"last":1.4730000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.546,"15min":1.514},"min":{"1min":0.59099999999999997,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":2.855,"5min":9.7219999999999995,"15min":9.7219999999999995},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.9690000000000001,"5min":1.6220000000000001,"15min":1.5720000000000001},"min":{"1min":0.91500000000000004,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":6.7039999999999997,"5min":6.7039999999999997,"15min":6.7039999999999997},"last":1.5069999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:02:56 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5349999999999999,"15min":1.5609999999999999},"min":{"1min":0.63100000000000001,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":2.3929999999999998,"5min":3.048,"15min":3.048},"last":1.302},{"interface":"front","average":{"1min":2.2869999999999999,"5min":1.8999999999999999,"15min":1.728},"min":{"1min":0.74299999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":13.561999999999999,"5min":13.561999999999999,"15min":13.561999999999999},"last":1.4330000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542186,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32304,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339536,"statfs":{"total":96636764160,"available":96603684864,"internally_reserved":0,"allocated":1142784,"data_stored":812815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.042,"15min":1.0229999999999999},"min":{"1min":0.81000000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.8660000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0249999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.64000000000000001,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.4990000000000001,"5min":1.746,"15min":1.746},"last":0.61799999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.1180000000000001,"15min":1.113},"min":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.88600000000000001},{"interface":"front","average":{"1min":1.4350000000000001,"5min":1.2130000000000001,"15min":1.173},"min":{"1min":0.97799999999999998,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":3.5609999999999999,"5min":3.5609999999999999,"15min":3.5609999999999999},"last":0.748}]},{"osd":2,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.25,"15min":1.248},"min":{"1min":0.89700000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.1990000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.2649999999999999,"15min":1.2549999999999999},"min":{"1min":1.0009999999999999,"5min":0.81100000000000005,"15min":0.81100000000000005},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.0269999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.3660000000000001,"15min":1.3440000000000001},"min":{"1min":0.98799999999999999,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.6000000000000001,"5min":4.2249999999999996,"15min":4.2249999999999996},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.456,"5min":1.3400000000000001,"15min":1.335},"min":{"1min":1.1040000000000001,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":2.0670000000000002,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":1.115}]},{"osd":4,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.383,"15min":1.391},"min":{"1min":0.88800000000000001,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":2.71,"5min":2.71,"15min":2.71},"last":1.395},{"interface":"front","average":{"1min":1.494,"5min":1.403,"15min":1.403},"min":{"1min":1.0449999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":2.2599999999999998,"5min":2.2599999999999998,"15min":2.2599999999999998},"last":1.26}]},{"osd":5,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.595,"5min":1.4179999999999999,"15min":1.407},"min":{"1min":1.0549999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.548,"5min":2.548,"15min":2.548},"last":0.95699999999999996},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.5549999999999999,"15min":1.5580000000000001},"min":{"1min":1.095,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":2.9489999999999998,"5min":2.9489999999999998,"15min":2.9489999999999998},"last":1.508}]},{"osd":6,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.575,"5min":1.5449999999999999,"15min":1.5509999999999999},"min":{"1min":1.141,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.3690000000000002,"5min":2.3690000000000002,"15min":2.3690000000000002},"last":1.349},{"interface":"front","average":{"1min":1.5960000000000001,"5min":1.571,"15min":1.5669999999999999},"min":{"1min":1.095,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":2.6600000000000001,"5min":2.6680000000000001,"15min":2.6680000000000001},"last":1.0569999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31708,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340132,"statfs":{"total":96636764160,"available":96604295168,"internally_reserved":0,"allocated":536576,"data_stored":216127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.0820000000000001,"15min":1.042},"min":{"1min":0.65000000000000002,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.1459999999999999,"5min":3.1459999999999999,"15min":3.1459999999999999},"last":1.349},{"interface":"front","average":{"1min":1.9279999999999999,"5min":1.2070000000000001,"15min":1.095},"min":{"1min":0.68799999999999994,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":10.074999999999999,"5min":10.074999999999999,"15min":10.074999999999999},"last":0.97199999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.129,"15min":1.113},"min":{"1min":0.72699999999999998,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.921,"5min":2.625,"15min":2.625},"last":1.319},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.1599999999999999,"15min":1.1060000000000001},"min":{"1min":0.86599999999999999,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":4.6200000000000001,"5min":4.6200000000000001,"15min":4.6200000000000001},"last":0.69899999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.2310000000000001,"15min":1.1930000000000001},"min":{"1min":0.94499999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.8140000000000001,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.367,"5min":1.2470000000000001,"15min":1.21},"min":{"1min":0.97699999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.796,"5min":2.4220000000000002,"15min":2.4220000000000002},"last":1.3879999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.329,"15min":1.3360000000000001},"min":{"1min":0.93100000000000005,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":3.4249999999999998,"5min":3.4249999999999998,"15min":3.4249999999999998},"last":1.224},{"interface":"front","average":{"1min":2.1920000000000002,"5min":1.5149999999999999,"15min":1.411},"min":{"1min":1.0429999999999999,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":14.647,"5min":14.647,"15min":14.647},"last":1.03}]},{"osd":4,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.3180000000000001,"5min":1.3220000000000001,"15min":1.304},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.764,"5min":1.764,"15min":1.764},"last":1.1220000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.409,"15min":1.393},"min":{"1min":0.90400000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":1.8720000000000001},"last":1.2629999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:02:54 2024","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.3380000000000001,"15min":1.3380000000000001},"min":{"1min":0.755,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.7270000000000001,"5min":1.9470000000000001,"15min":1.9470000000000001},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.4219999999999999,"15min":1.403},"min":{"1min":1.1180000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.9219999999999999,"5min":2.1560000000000001,"15min":2.1560000000000001},"last":1.494}]},{"osd":7,"last update":"Tue Jan 16 15:02:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.544,"15min":1.5529999999999999},"min":{"1min":1.1599999999999999,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.8720000000000001,"5min":3.2690000000000001,"15min":3.2690000000000001},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.637,"15min":1.649},"min":{"1min":1.333,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":1.8540000000000001,"5min":3.903,"15min":3.903},"last":1.6020000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35872,"kb_used_data":532,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335968,"statfs":{"total":96636764160,"available":96600031232,"internally_reserved":0,"allocated":544768,"data_stored":216695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":0.86299999999999999,"15min":0.81999999999999995},"min":{"1min":0.73099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":2.6099999999999999,"5min":2.6099999999999999,"15min":2.6099999999999999},"last":1.22},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.0069999999999999,"15min":0.97999999999999998},"min":{"1min":0.68600000000000005,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":4.5110000000000001,"5min":4.5110000000000001,"15min":4.5110000000000001},"last":0.76900000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":0.94099999999999995,"15min":0.879},"min":{"1min":0.58099999999999996,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":5.4269999999999996,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.1000000000000001,"15min":1.0349999999999999},"min":{"1min":0.79100000000000004,"5min":0.497,"15min":0.497},"max":{"1min":3.7559999999999998,"5min":3.7559999999999998,"15min":3.7559999999999998},"last":1.361}]},{"osd":2,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.175,"15min":1.105},"min":{"1min":0.73899999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":1.4299999999999999},{"interface":"front","average":{"1min":2.012,"5min":1.286,"15min":1.147},"min":{"1min":0.84999999999999998,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":14.577999999999999,"5min":14.577999999999999,"15min":14.577999999999999},"last":1.099}]},{"osd":4,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.7490000000000001,"5min":1.351,"15min":1.274},"min":{"1min":0.94999999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":5.4900000000000002,"5min":5.4900000000000002,"15min":5.4900000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5720000000000001,"5min":1.421,"15min":1.3959999999999999},"min":{"1min":0.96199999999999997,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":2.468,"5min":2.468,"15min":2.468},"last":1.4970000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.46,"15min":1.407},"min":{"1min":1.0209999999999999,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":5.3520000000000003,"5min":5.3520000000000003,"15min":5.3520000000000003},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.6850000000000001,"5min":1.484,"15min":1.4430000000000001},"min":{"1min":1.0469999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.556}]},{"osd":6,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.5009999999999999,"15min":1.429},"min":{"1min":1.0960000000000001,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":8.4730000000000008,"5min":8.4730000000000008,"15min":8.4730000000000008},"last":1.149},{"interface":"front","average":{"1min":1.8959999999999999,"5min":1.498,"15min":1.4119999999999999},"min":{"1min":1.2689999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"max":{"1min":5.1799999999999997,"5min":5.1799999999999997,"15min":5.1799999999999997},"last":1.387}]},{"osd":7,"last update":"Tue Jan 16 15:02:51 2024","interfaces":[{"interface":"back","average":{"1min":1.806,"5min":1.52,"15min":1.4490000000000001},"min":{"1min":1.2170000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":4.0830000000000002,"5min":4.0830000000000002,"15min":4.0830000000000002},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.9430000000000001,"5min":1.5800000000000001,"15min":1.4950000000000001},"min":{"1min":1.385,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":5.5510000000000002,"5min":5.5510000000000002,"15min":5.5510000000000002},"last":1.5289999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":536,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":548864,"data_stored":217691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.98199999999999998,"15min":1.0189999999999999},"min":{"1min":0.497,"5min":0.436,"15min":0.436},"max":{"1min":1.3129999999999999,"5min":2.1139999999999999,"15min":2.1139999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.018,"5min":1.0169999999999999,"15min":1.02},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.5700000000000001,"5min":1.6950000000000001,"15min":1.6950000000000001},"last":0.57099999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":3.839,"5min":3.839,"15min":3.839},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.0940000000000001,"15min":1.0720000000000001},"min":{"1min":0.57599999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":1.9330000000000001},"last":0.68300000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.0189999999999999,"15min":0.92600000000000005},"min":{"1min":0.60899999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":9.1530000000000005,"5min":9.1530000000000005,"15min":9.1530000000000005},"last":0.88},{"interface":"front","average":{"1min":1.379,"5min":1.3149999999999999,"15min":1.3120000000000001},"min":{"1min":0.82999999999999996,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.857,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":0.82699999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.292,"15min":1.306},"min":{"1min":0.93000000000000005,"5min":0.91500000000000004,"15min":0.91500000000000004},"max":{"1min":1.9730000000000001,"5min":1.9730000000000001,"15min":1.9730000000000001},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.419,"5min":1.407,"15min":1.421},"min":{"1min":0.93899999999999995,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":2.093,"5min":2.1040000000000001,"15min":2.1040000000000001},"last":1.24}]},{"osd":5,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.704,"5min":1.3580000000000001,"15min":1.3220000000000001},"min":{"1min":0.91100000000000003,"5min":0.753,"15min":0.753},"max":{"1min":9.6189999999999998,"5min":9.6189999999999998,"15min":9.6189999999999998},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.7310000000000001,"5min":1.4319999999999999,"15min":1.3759999999999999},"min":{"1min":0.91600000000000004,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":9.423,"5min":9.423,"15min":9.423},"last":1.04}]},{"osd":6,"last update":"Tue Jan 16 15:02:55 2024","interfaces":[{"interface":"back","average":{"1min":1.853,"5min":1.46,"15min":1.3799999999999999},"min":{"1min":0.93400000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":9.5359999999999996,"5min":9.5359999999999996,"15min":9.5359999999999996},"last":1.01},{"interface":"front","average":{"1min":1.9119999999999999,"5min":1.5549999999999999,"15min":1.4950000000000001},"min":{"1min":1.0049999999999999,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":9.7119999999999997,"5min":9.7119999999999997,"15min":9.7119999999999997},"last":1.502}]},{"osd":7,"last update":"Tue Jan 16 15:02:49 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4970000000000001,"15min":1.5189999999999999},"min":{"1min":1.121,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.024,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.403},{"interface":"front","average":{"1min":1.5089999999999999,"5min":1.542,"15min":1.583},"min":{"1min":0.98699999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.992,"5min":2.54,"15min":2.54},"last":1.204}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":35868,"kb_used_data":524,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335972,"statfs":{"total":96636764160,"available":96600035328,"internally_reserved":0,"allocated":536576,"data_stored":216127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":0.96399999999999997,"15min":0.84799999999999998},"min":{"1min":0.57999999999999996,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":10.667999999999999,"5min":10.667999999999999,"15min":10.667999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.1000000000000001,"5min":0.97199999999999998,"15min":0.91600000000000004},"min":{"1min":0.46500000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":2.02,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":0.83399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":0.98999999999999999,"15min":0.96799999999999997},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.3560000000000001},{"interface":"front","average":{"1min":1.8520000000000001,"5min":1.3200000000000001,"15min":1.196},"min":{"1min":0.82299999999999995,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":10.805,"5min":10.805,"15min":10.805},"last":1.0900000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.7370000000000001,"5min":1.238,"15min":1.133},"min":{"1min":0.75800000000000001,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":10.991,"5min":10.991,"15min":10.991},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.1639999999999999,"15min":1.0840000000000001},"min":{"1min":0.50600000000000001,"5min":0.443,"15min":0.443},"max":{"1min":7.3449999999999998,"5min":7.3449999999999998,"15min":7.3449999999999998},"last":0.94999999999999996}]},{"osd":4,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.2589999999999999,"15min":1.2749999999999999},"min":{"1min":0.93200000000000005,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.157},{"interface":"front","average":{"1min":2.0310000000000001,"5min":1.5700000000000001,"15min":1.502},"min":{"1min":1.111,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":11.157999999999999,"5min":11.157999999999999,"15min":11.157999999999999},"last":1.847}]},{"osd":5,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.381,"15min":1.351},"min":{"1min":0.90000000000000002,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.913},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.4470000000000001,"15min":1.4450000000000001},"min":{"1min":0.83399999999999996,"5min":0.78100000000000003,"15min":0.78100000000000003},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":2.157}]},{"osd":6,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.4810000000000001,"15min":1.4690000000000001},"min":{"1min":1.0549999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":2.1920000000000002,"5min":2.3809999999999998,"15min":2.3809999999999998},"last":1.786},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.76800000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":2.4359999999999999,"5min":3.0169999999999999,"15min":3.0169999999999999},"last":2.0459999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:02:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5229999999999999,"15min":1.5409999999999999},"min":{"1min":0.83099999999999996,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.3889999999999998,"5min":2.3889999999999998,"15min":2.3889999999999998},"last":2.2690000000000001},{"interface":"front","average":{"1min":2.0939999999999999,"5min":1.627,"15min":1.5289999999999999},"min":{"1min":1.266,"5min":0.89400000000000002,"15min":0.89400000000000002},"max":{"1min":11.063000000000001,"5min":11.063000000000001,"15min":11.063000000000001},"last":1.9670000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32288,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339552,"statfs":{"total":96636764160,"available":96603701248,"internally_reserved":0,"allocated":1130496,"data_stored":806459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":0.98399999999999999,"15min":0.95799999999999996},"min":{"1min":0.59699999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":2.4500000000000002},"last":0.499},{"interface":"front","average":{"1min":1.236,"5min":1.048,"15min":1.0449999999999999},"min":{"1min":0.47499999999999998,"5min":0.432,"15min":0.432},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.0469999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.923,"5min":1.262,"15min":1.1579999999999999},"min":{"1min":0.71899999999999997,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":17.032,"5min":17.032,"15min":17.032},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.163,"15min":1.1140000000000001},"min":{"1min":0.626,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":6.1470000000000002,"5min":6.1470000000000002,"15min":6.1470000000000002},"last":1.2689999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8460000000000001,"5min":1.228,"15min":1.1519999999999999},"min":{"1min":0.67000000000000004,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":10.795,"5min":10.795,"15min":10.795},"last":0.71799999999999997},{"interface":"front","average":{"1min":2.0150000000000001,"5min":1.4119999999999999,"15min":1.319},"min":{"1min":0.93500000000000005,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":14.992000000000001,"5min":14.992000000000001,"15min":14.992000000000001},"last":0.69699999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.4430000000000001,"15min":1.3919999999999999},"min":{"1min":1.107,"5min":0.625,"15min":0.625},"max":{"1min":4.8529999999999998,"5min":4.8529999999999998,"15min":4.8529999999999998},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.425,"15min":1.3260000000000001},"min":{"1min":1.0129999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":3.2749999999999999,"5min":9.8719999999999999,"15min":9.8719999999999999},"last":1.147}]},{"osd":5,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":2.2309999999999999,"5min":1.5029999999999999,"15min":1.393},"min":{"1min":1.038,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":10.606999999999999,"5min":10.606999999999999,"15min":10.606999999999999},"last":0.88},{"interface":"front","average":{"1min":1.623,"5min":1.5149999999999999,"15min":1.5029999999999999},"min":{"1min":1.089,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":3.2010000000000001,"5min":3.2010000000000001,"15min":3.2010000000000001},"last":0.92500000000000004}]},{"osd":6,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":2.004,"5min":1.5069999999999999,"15min":1.4370000000000001},"min":{"1min":1.0980000000000001,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":10.69,"5min":10.69,"15min":10.69},"last":0.82499999999999996},{"interface":"front","average":{"1min":1.714,"5min":1.5600000000000001,"15min":1.544},"min":{"1min":1.085,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":0.89400000000000002}]},{"osd":7,"last update":"Tue Jan 16 15:02:48 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.496,"15min":1.52},"min":{"1min":0.84999999999999998,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.8100000000000001,"5min":2.8100000000000001,"15min":2.8100000000000001},"last":1.6240000000000001},{"interface":"front","average":{"1min":2.2530000000000001,"5min":1.6719999999999999,"15min":1.585},"min":{"1min":1.264,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":10.736000000000001,"5min":10.736000000000001,"15min":10.736000000000001},"last":1.137}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:03:09.053 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:03:09.053 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:03:09.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.345+0000 7fd64f000640 1 Processor -- start 2024-01-16T15:03:09.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.346+0000 7fd64f000640 1 -- start start 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.347+0000 7fd64f000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd648155950 0x7fd6480ce340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.347+0000 7fd64f000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6480b2ce0 0x7fd6480b3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.347+0000 7fd64f000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd64815edb0 con 0x7fd64815c440 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.347+0000 7fd64f000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480b36d0 con 0x7fd6480b2ce0 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.347+0000 7fd64f000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6480cd300 con 0x7fd648155950 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6480b2ce0 0x7fd6480b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6480b2ce0 0x7fd6480b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41422/0 (socket says 172.21.15.53:41422) 2024-01-16T15:03:09.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 -- 172.21.15.53:0/2842081134 learned_addr learned my addr 172.21.15.53:0/2842081134 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd647e00640 1 --2- 172.21.15.53:0/2842081134 >> v2:172.21.15.146:3300/0 conn(0x7fd648155950 0x7fd6480ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 -- 172.21.15.53:0/2842081134 >> v2:172.21.15.146:3300/0 conn(0x7fd648155950 msgr2=0x7fd6480ce340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 --2- 172.21.15.53:0/2842081134 >> v2:172.21.15.146:3300/0 conn(0x7fd648155950 0x7fd6480ce340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 -- 172.21.15.53:0/2842081134 >> v1:172.21.15.146:6789/0 conn(0x7fd64815c440 legacy=0x7fd64815f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd64d000640 1 -- 172.21.15.53:0/2842081134 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6480cd5e0 con 0x7fd6480b2ce0 2024-01-16T15:03:09.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.348+0000 7fd647e00640 1 --2- 172.21.15.53:0/2842081134 >> v2:172.21.15.146:3300/0 conn(0x7fd648155950 0x7fd6480ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:09.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.349+0000 7fd64d000640 1 --2- 172.21.15.53:0/2842081134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6480b2ce0 0x7fd6480b3190 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd64806b2f0 tx=0x7fd64002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18dafb3f3e0fb69 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:09.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.350+0000 7fd646a00640 1 -- 172.21.15.53:0/2842081134 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640031870 con 0x7fd6480b2ce0 2024-01-16T15:03:09.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.350+0000 7fd646a00640 1 -- 172.21.15.53:0/2842081134 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6400319d0 con 0x7fd6480b2ce0 2024-01-16T15:03:09.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.350+0000 7fd646a00640 1 -- 172.21.15.53:0/2842081134 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd640031d60 con 0x7fd6480b2ce0 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 -- 172.21.15.53:0/2842081134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6480b2ce0 msgr2=0x7fd6480b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 --2- 172.21.15.53:0/2842081134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6480b2ce0 0x7fd6480b3190 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd64806b2f0 tx=0x7fd64002b910 comp rx=0 tx=0).stop 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 -- 172.21.15.53:0/2842081134 shutdown_connections 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 --2- 172.21.15.53:0/2842081134 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6480b2ce0 0x7fd6480b3190 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 --2- 172.21.15.53:0/2842081134 >> v2:172.21.15.146:3300/0 conn(0x7fd648155950 0x7fd6480ce340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 -- 172.21.15.53:0/2842081134 >> 172.21.15.53:0/2842081134 conn(0x7fd648066450 msgr2=0x7fd64806a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:09.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 -- 172.21.15.53:0/2842081134 shutdown_connections 2024-01-16T15:03:09.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.351+0000 7fd64f000640 1 -- 172.21.15.53:0/2842081134 wait complete. 2024-01-16T15:03:09.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.353+0000 7fd64f000640 1 Processor -- start 2024-01-16T15:03:09.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.354+0000 7fd64f000640 1 -- start start 2024-01-16T15:03:09.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd64f000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480b2ce0 0x7fd6481e5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd64f000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 0x7fd648152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd64f000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd64815c440 0x7fd6481e7ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd64f000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd648152fe0 con 0x7fd648155950 2024-01-16T15:03:09.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd64f000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6481e5690 con 0x7fd6480b2ce0 2024-01-16T15:03:09.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd64f000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd648152540 con 0x7fd64815c440 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd647400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 0x7fd648152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd647400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 0x7fd648152000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41434/0 (socket says 172.21.15.53:41434) 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.355+0000 7fd647400640 1 -- 172.21.15.53:0/3540850208 learned_addr learned my addr 172.21.15.53:0/3540850208 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647e00640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480b2ce0 0x7fd6481e5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647400640 1 -- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd64815c440 msgr2=0x7fd6481e7ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd64d000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd64815c440 0x7fd6481e7ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647400640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd64815c440 0x7fd6481e7ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647400640 1 -- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480b2ce0 msgr2=0x7fd6481e5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647400640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480b2ce0 0x7fd6481e5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647400640 1 -- 172.21.15.53:0/3540850208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6400094d0 con 0x7fd648155950 2024-01-16T15:03:09.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.356+0000 7fd647400640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 0x7fd648152000 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd63000b3c0 tx=0x7fd63000b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:09.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.357+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630011830 con 0x7fd648155950 2024-01-16T15:03:09.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.357+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd630011990 con 0x7fd648155950 2024-01-16T15:03:09.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.357+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd630011d20 con 0x7fd648155950 2024-01-16T15:03:09.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.357+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6481e58c0 con 0x7fd648155950 2024-01-16T15:03:09.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.357+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6481e5e80 con 0x7fd648155950 2024-01-16T15:03:09.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.358+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd648067670 con 0x7fd648155950 2024-01-16T15:03:09.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.359+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd63001b460 con 0x7fd648155950 2024-01-16T15:03:09.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.360+0000 7fd63fe00640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd62006b500 0x7fd62006d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.361+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd630089d60 con 0x7fd648155950 2024-01-16T15:03:09.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.361+0000 7fd647e00640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd62006b500 0x7fd62006d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.363+0000 7fd647e00640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd62006b500 0x7fd62006d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd6380075b0 tx=0x7fd638007540 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:09.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.365+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd630054860 con 0x7fd648155950 2024-01-16T15:03:09.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.631+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fd6480cdef0 con 0x7fd648155950 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.636+0000 7fd63fe00640 1 -- 172.21.15.53:0/3540850208 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v10) v1 ==== 190+0+147 (secure 0 0 0) 0x7fd630057e80 con 0x7fd648155950 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:03:09.616 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:03:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.638+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd62006b500 msgr2=0x7fd62006d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.638+0000 7fd64f000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd62006b500 0x7fd62006d9c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd6380075b0 tx=0x7fd638007540 comp rx=0 tx=0).stop 2024-01-16T15:03:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.638+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 msgr2=0x7fd648152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.638+0000 7fd64f000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 0x7fd648152000 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd63000b3c0 tx=0x7fd63000b880 comp rx=0 tx=0).stop 2024-01-16T15:03:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 shutdown_connections 2024-01-16T15:03:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd62006b500 0x7fd62006d9c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd64815c440 0x7fd6481e7ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd648155950 0x7fd648152000 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 --2- 172.21.15.53:0/3540850208 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6480b2ce0 0x7fd6481e5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 >> 172.21.15.53:0/3540850208 conn(0x7fd648066450 msgr2=0x7fd64815b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 shutdown_connections 2024-01-16T15:03:09.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.639+0000 7fd64f000640 1 -- 172.21.15.53:0/3540850208 wait complete. 2024-01-16T15:03:09.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:03:09.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.869+0000 7f1a61e00640 1 Processor -- start 2024-01-16T15:03:09.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.869+0000 7f1a61e00640 1 -- start start 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a61e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c143c40 0x7f1a5c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a61e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a5c0717d0 0x7f1a5c0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a61e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5c070e50 con 0x7f1a5c0cca40 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a61e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a5c0723c0 con 0x7f1a5c0717d0 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a61e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0cce70 con 0x7f1a5c143c40 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a53e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a5c0717d0 0x7f1a5c0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a53e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a5c0717d0 0x7f1a5c0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41442/0 (socket says 172.21.15.53:41442) 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a53e00640 1 -- 172.21.15.53:0/3449447828 learned_addr learned my addr 172.21.15.53:0/3449447828 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a61400640 1 --1- 172.21.15.53:0/3449447828 >> v1:172.21.15.53:6789/0 conn(0x7f1a5c0cca40 0x7f1a5c15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46156/0 (socket says 172.21.15.53:46156) 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.871+0000 7f1a60a00640 1 --2- 172.21.15.53:0/3449447828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c143c40 0x7f1a5c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.872+0000 7f1a53e00640 1 -- 172.21.15.53:0/3449447828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c143c40 msgr2=0x7f1a5c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:09.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.872+0000 7f1a53e00640 1 --2- 172.21.15.53:0/3449447828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c143c40 0x7f1a5c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.872+0000 7f1a53e00640 1 -- 172.21.15.53:0/3449447828 >> v1:172.21.15.53:6789/0 conn(0x7f1a5c0cca40 legacy=0x7f1a5c15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.872+0000 7f1a53e00640 1 -- 172.21.15.53:0/3449447828 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a5c06e170 con 0x7f1a5c0717d0 2024-01-16T15:03:09.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.872+0000 7f1a5be00640 1 -- 172.21.15.53:0/3449447828 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470937284 0 0) 0x7f1a5c070e50 con 0x7f1a5c0cca40 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.873+0000 7f1a53e00640 1 --2- 172.21.15.53:0/3449447828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0cc500 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009870 tx=0x7f1a4402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64b8d752339ee343 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.873+0000 7f1a5be00640 1 -- 172.21.15.53:0/3449447828 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44031640 con 0x7f1a5c0717d0 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.873+0000 7f1a5be00640 1 -- 172.21.15.53:0/3449447828 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a440317a0 con 0x7f1a5c0717d0 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.873+0000 7f1a5be00640 1 -- 172.21.15.53:0/3449447828 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a44031b30 con 0x7f1a5c0717d0 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 -- 172.21.15.53:0/3449447828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c0717d0 msgr2=0x7f1a5c0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 --2- 172.21.15.53:0/3449447828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0cc500 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1a44009870 tx=0x7f1a4402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 -- 172.21.15.53:0/3449447828 shutdown_connections 2024-01-16T15:03:09.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 --2- 172.21.15.53:0/3449447828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0cc500 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 --2- 172.21.15.53:0/3449447828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c143c40 0x7f1a5c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 -- 172.21.15.53:0/3449447828 >> 172.21.15.53:0/3449447828 conn(0x7f1a5c06a800 msgr2=0x7f1a5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:09.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 -- 172.21.15.53:0/3449447828 shutdown_connections 2024-01-16T15:03:09.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.874+0000 7f1a61e00640 1 -- 172.21.15.53:0/3449447828 wait complete. 2024-01-16T15:03:09.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.876+0000 7f1a61e00640 1 Processor -- start 2024-01-16T15:03:09.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.876+0000 7f1a61e00640 1 -- start start 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c0cca40 0x7f1a5c0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 0x7f1a5c1f37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c1f33d0 con 0x7f1a5c143c40 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0c8aa0 con 0x7f1a5c0717d0 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5c0ca3a0 con 0x7f1a5c0cca40 2024-01-16T15:03:09.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 0x7f1a5c1f37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 0x7f1a5c1f37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41444/0 (socket says 172.21.15.53:41444) 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 -- 172.21.15.53:0/2537596182 learned_addr learned my addr 172.21.15.53:0/2537596182 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a60a00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a53e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c0cca40 0x7f1a5c0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 -- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c0cca40 msgr2=0x7f1a5c0c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c0cca40 0x7f1a5c0c54c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 -- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a5c0717d0 msgr2=0x7f1a5c0c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.877+0000 7f1a61400640 1 -- 172.21.15.53:0/2537596182 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a440094d0 con 0x7f1a5c143c40 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.878+0000 7f1a61400640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 0x7f1a5c1f37f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c7a0 tx=0x7f1a5400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.878+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400a640 con 0x7f1a5c143c40 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.878+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5400a7a0 con 0x7f1a5c143c40 2024-01-16T15:03:09.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.878+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5400abd0 con 0x7f1a5c143c40 2024-01-16T15:03:09.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.878+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a5c0c5a00 con 0x7f1a5c143c40 2024-01-16T15:03:09.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.878+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a5c0ca890 con 0x7f1a5c143c40 2024-01-16T15:03:09.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.879+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a5c1f3510 con 0x7f1a5c143c40 2024-01-16T15:03:09.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.880+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f1a5400ad30 con 0x7f1a5c143c40 2024-01-16T15:03:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.881+0000 7f1a59600640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a3406b530 0x7f1a3406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.881+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a540893b0 con 0x7f1a5c143c40 2024-01-16T15:03:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.882+0000 7f1a60a00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a3406b530 0x7f1a3406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:09.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.883+0000 7f1a60a00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a3406b530 0x7f1a3406d9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006fd0 tx=0x7f1a4c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:09.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:09.886+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a5c1f3510 con 0x7f1a5c143c40 2024-01-16T15:03:10.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.110+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a5c1f3510 con 0x7f1a5c143c40 2024-01-16T15:03:10.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.111+0000 7f1a59600640 1 -- 172.21.15.53:0/2537596182 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f1a5c1f3510 con 0x7f1a5c143c40 2024-01-16T15:03:10.092 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:03:10.092 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":25,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3977954404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3977954404},{"type":"v1","addr":"172.21.15.146:6835","nonce":3977954404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3669971149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3669971149},{"type":"v1","addr":"172.21.15.146:6837","nonce":3669971149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5709,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1838972240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1838972240},{"type":"v1","addr":"172.21.15.53:6833","nonce":1838972240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20}],"filesystems":[{"mdsmap":{"epoch":25,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:03:03.203889+0000","modified":"2024-01-16T15:03:06.573035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5718},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5718":{"gid":5718,"name":"c","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6835/4293818684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":4293818684},{"type":"v1","addr":"172.21.15.53:6835","nonce":4293818684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-01-16T15:03:10.092 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 25 2024-01-16T15:03:10.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a3406b530 msgr2=0x7f1a3406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a3406b530 0x7f1a3406d9f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c006fd0 tx=0x7f1a4c008040 comp rx=0 tx=0).stop 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 msgr2=0x7f1a5c1f37f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 0x7f1a5c1f37f0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1a5400c7a0 tx=0x7f1a5400cc60 comp rx=0 tx=0).stop 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 shutdown_connections 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a3406b530 0x7f1a3406d9f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a5c143c40 0x7f1a5c1f37f0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a5c0cca40 0x7f1a5c0c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 --2- 172.21.15.53:0/2537596182 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a5c0717d0 0x7f1a5c0c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 >> 172.21.15.53:0/2537596182 conn(0x7f1a5c06a800 msgr2=0x7f1a5c0b02e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 shutdown_connections 2024-01-16T15:03:10.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.114+0000 7f1a61e00640 1 -- 172.21.15.53:0/2537596182 wait complete. 2024-01-16T15:03:10.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 25, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:03:03.203889+0000', 'modified': '2024-01-16T15:03:06.573035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5718}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5718': {'gid': 5718, 'name': 'c', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6835/4293818684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6834', 'nonce': 4293818684}, {'type': 'v1', 'addr': '172.21.15.53:6835', 'nonce': 4293818684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:03:10.515 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:03:10.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:03:10.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.769+0000 7fd3efe00640 1 Processor -- start 2024-01-16T15:03:10.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.770+0000 7fd3efe00640 1 -- start start 2024-01-16T15:03:10.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.771+0000 7fd3efe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3f0143c40 0x7fd3f0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.771+0000 7fd3efe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3f00717d0 0x7fd3f00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.771+0000 7fd3efe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f0070e50 con 0x7fd3f00cca40 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.771+0000 7fd3efe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f00723c0 con 0x7fd3f0143c40 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.771+0000 7fd3efe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3f00cce40 con 0x7fd3f00717d0 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3eea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3f0143c40 0x7fd3f0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3ef400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd3f00cca40 0x7fd3f015eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:46170/0 (socket says 172.21.15.53:46170) 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3ef400640 1 -- 172.21.15.53:0/1125282608 learned_addr learned my addr 172.21.15.53:0/1125282608 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1125282608 >> v2:172.21.15.146:3300/0 conn(0x7fd3f00717d0 0x7fd3f00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3eea00640 1 -- 172.21.15.53:0/1125282608 >> v2:172.21.15.146:3300/0 conn(0x7fd3f00717d0 msgr2=0x7fd3f00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1125282608 >> v2:172.21.15.146:3300/0 conn(0x7fd3f00717d0 0x7fd3f00cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3eea00640 1 -- 172.21.15.53:0/1125282608 >> v1:172.21.15.53:6789/0 conn(0x7fd3f00cca40 legacy=0x7fd3f015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.772+0000 7fd3eea00640 1 -- 172.21.15.53:0/1125282608 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f006e170 con 0x7fd3f0143c40 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.773+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1125282608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f0143c40 0x7fd3f0071290 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00b44a0 tx=0x7fd3e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07b5281fee11162 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3ee000640 1 -- 172.21.15.53:0/1125282608 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e002e070 con 0x7fd3f0143c40 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3ee000640 1 -- 172.21.15.53:0/1125282608 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e00315a0 con 0x7fd3f0143c40 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3ee000640 1 -- 172.21.15.53:0/1125282608 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0031930 con 0x7fd3f0143c40 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 -- 172.21.15.53:0/1125282608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f0143c40 msgr2=0x7fd3f0071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1125282608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f0143c40 0x7fd3f0071290 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fd3f00b44a0 tx=0x7fd3e002b840 comp rx=0 tx=0).stop 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 -- 172.21.15.53:0/1125282608 shutdown_connections 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1125282608 >> v2:172.21.15.146:3300/0 conn(0x7fd3f00717d0 0x7fd3f00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1125282608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f0143c40 0x7fd3f0071290 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 -- 172.21.15.53:0/1125282608 >> 172.21.15.53:0/1125282608 conn(0x7fd3f006a800 msgr2=0x7fd3f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.774+0000 7fd3efe00640 1 -- 172.21.15.53:0/1125282608 shutdown_connections 2024-01-16T15:03:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.775+0000 7fd3efe00640 1 -- 172.21.15.53:0/1125282608 wait complete. 2024-01-16T15:03:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.776+0000 7fd3efe00640 1 Processor -- start 2024-01-16T15:03:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.776+0000 7fd3efe00640 1 -- start start 2024-01-16T15:03:10.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.777+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 0x7fd3f01f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.777+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f00cca40 0x7fd3f00c8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.777+0000 7fd3efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f0143c40 0x7fd3f00c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.777+0000 7fd3efe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01f71d0 con 0x7fd3f00717d0 2024-01-16T15:03:10.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.777+0000 7fd3efe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01f6930 con 0x7fd3f00cca40 2024-01-16T15:03:10.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.777+0000 7fd3efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f01f6c10 con 0x7fd3f0143c40 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 0x7fd3f01f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 0x7fd3f01f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41466/0 (socket says 172.21.15.53:41466) 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 -- 172.21.15.53:0/1802498377 learned_addr learned my addr 172.21.15.53:0/1802498377 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f00cca40 0x7fd3f00c8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3ef400640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f0143c40 0x7fd3f00c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 -- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f0143c40 msgr2=0x7fd3f00c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f0143c40 0x7fd3f00c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 -- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f00cca40 msgr2=0x7fd3f00c8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f00cca40 0x7fd3f00c8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.778+0000 7fd3eea00640 1 -- 172.21.15.53:0/1802498377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e0009400 con 0x7fd3f00717d0 2024-01-16T15:03:10.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.779+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 0x7fd3f01f62a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009890 tx=0x7fd3e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.779+0000 7fd3e6a00640 1 -- 172.21.15.53:0/1802498377 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032840 con 0x7fd3f00717d0 2024-01-16T15:03:10.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.779+0000 7fd3e6a00640 1 -- 172.21.15.53:0/1802498377 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e00329a0 con 0x7fd3f00717d0 2024-01-16T15:03:10.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.779+0000 7fd3e6a00640 1 -- 172.21.15.53:0/1802498377 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0032d50 con 0x7fd3f00717d0 2024-01-16T15:03:10.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.779+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f01f6ef0 con 0x7fd3f00717d0 2024-01-16T15:03:10.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.779+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f00ca340 con 0x7fd3f00717d0 2024-01-16T15:03:10.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.781+0000 7fd3e6a00640 1 -- 172.21.15.53:0/1802498377 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fd3e003b3f0 con 0x7fd3f00717d0 2024-01-16T15:03:10.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.782+0000 7fd3e6a00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c806b500 0x7fd3c806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.782+0000 7fd3e6a00640 1 -- 172.21.15.53:0/1802498377 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3e00ac0c0 con 0x7fd3f00717d0 2024-01-16T15:03:10.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.783+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c806b500 0x7fd3c806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.783+0000 7fd3f8987740 1 Processor -- start 2024-01-16T15:03:10.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.783+0000 7fd3f8987740 1 -- start start 2024-01-16T15:03:10.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c806b500 0x7fd3c806d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd3f01349f0 tx=0x7fd3d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x558894585450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x558894584270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945c8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3f8987740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588944e3880 con 0x55889457fd70 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588944f2ae0 con 0x5588945847b0 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588944f01d0 con 0x558894585070 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x558894584270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x558894584270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41472/0 (socket says 172.21.15.53:41472) 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.784+0000 7fd3e7e00640 1 -- 172.21.15.53:0/735479180 learned_addr learned my addr 172.21.15.53:0/735479180 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3eea00640 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x558894585450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3ef400640 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945c8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3e7e00640 1 -- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 msgr2=0x558894585450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x558894585450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3e7e00640 1 -- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 msgr2=0x5588945c8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945c8730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3e7e00640 1 -- 172.21.15.53:0/735479180 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588944b3860 con 0x55889457fd70 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3eea00640 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x558894585450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x558894584270 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800fa60 tx=0x7fd3d8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=331e2919b5c0e740 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3df400640 1 -- 172.21.15.53:0/735479180 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8014be0 con 0x55889457fd70 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3df400640 1 -- 172.21.15.53:0/735479180 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d8019070 con 0x55889457fd70 2024-01-16T15:03:10.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.785+0000 7fd3df400640 1 -- 172.21.15.53:0/735479180 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d8020440 con 0x55889457fd70 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 -- 172.21.15.53:0/735479180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 msgr2=0x558894584270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x558894584270 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fd3d800fa60 tx=0x7fd3d8012000 comp rx=0 tx=0).stop 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 -- 172.21.15.53:0/735479180 shutdown_connections 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945c8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x558894584270 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 --2- 172.21.15.53:0/735479180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x558894585450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 -- 172.21.15.53:0/735479180 >> 172.21.15.53:0/735479180 conn(0x558894546c70 msgr2=0x5588945cab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 -- 172.21.15.53:0/735479180 shutdown_connections 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.786+0000 7fd3f8987740 1 -- 172.21.15.53:0/735479180 wait complete. 2024-01-16T15:03:10.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.787+0000 7fd3f8987740 1 -- start start 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.787+0000 7fd3f8987740 1 Processor -- start 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945aabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945ab120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588944ebac0 con 0x55889457fd70 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588944882f0 con 0x5588945847b0 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588944a5c70 con 0x558894585070 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3f8987740 10 client.0 _ll_register_callbacks cb 0x5588944aa3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945cdbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41488/0 (socket says 172.21.15.53:41488) 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.788+0000 7fd3eea00640 1 -- 172.21.15.53:0/1524284646 learned_addr learned my addr 172.21.15.53:0/1524284646 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:10.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945aabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3ef400640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945ab120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3eea00640 1 -- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 msgr2=0x5588945ab120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945ab120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3eea00640 1 -- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 msgr2=0x5588945aabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945aabe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3eea00640 1 -- 172.21.15.53:0/1524284646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d800e040 con 0x55889457fd70 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945cdbc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd3e002be50 tx=0x7fd3e007af30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd4819a31bf9e9b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1524284646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e007abd0 con 0x55889457fd70 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1524284646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3e0082a70 con 0x55889457fd70 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5588944e4aa0 con 0x55889457fd70 2024-01-16T15:03:10.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.789+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1524284646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3e0079c60 con 0x55889457fd70 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 10 client.5853 fetch_fsmap learned FSMap version 25 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5588944ea140 con 0x55889457fd70 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1524284646 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5588944e4aa0 con 0x55889457fd70 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1524284646 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd3e00825e0 con 0x55889457fd70 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 10 client.5853 fetch_fsmap finished waiting for FSMap version 25 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 10 client.5853 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 20 client.5853 populate_metadata read hostname 'smithi053' 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945a2d20 0x5588945a3800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 4 client.5853 mds_command: new command op to 5718 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 --> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588944e47c0 con 0x5588945a2d20 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.790+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945a2d20 0x5588945a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.791+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945a2d20 0x5588945a3800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11bf4dd206221ac server_cookie=543f89265d8db5a5 in_seq=0 out_seq=0 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.791+0000 7fd3d7e00640 10 client.5853 ms_handle_connect on v2:172.21.15.53:6834/4293818684 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1524284646 <== mds.0 v2:172.21.15.53:6834/4293818684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5588944e47c0 con 0x5588945a2d20 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3d7e00640 10 client.5853 handle_command_reply: tid=0 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 client.5853 shutdown 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 msgr2=0x5588945cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945cdbc0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fd3e002be50 tx=0x7fd3e007af30 comp rx=0 tx=0).stop 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 shutdown_connections 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945ab120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945a2d20 0x5588945a3800 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945aabe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1524284646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945cdbc0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3d7e00640 0 client.5853 ms_handle_reset on v2:172.21.15.53:6834/4293818684 2024-01-16T15:03:10.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 >> 172.21.15.53:0/1524284646 conn(0x558894546c70 msgr2=0x5588945c7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 shutdown_connections 2024-01-16T15:03:10.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 1 -- 172.21.15.53:0/1524284646 wait complete. 2024-01-16T15:03:10.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.792+0000 7fd3f8987740 20 client.5853 trim_cache size 0 max 16384 2024-01-16T15:03:10.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.809+0000 7fd3f8987740 1 Processor -- start 2024-01-16T15:03:10.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.809+0000 7fd3f8987740 1 -- start start 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945d2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945dd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945a1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3f8987740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889424ef20 con 0x55889457fd70 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588945a2320 con 0x5588945847b0 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x5588945cd310 con 0x558894585070 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945d2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945d2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41492/0 (socket says 172.21.15.53:41492) 2024-01-16T15:03:10.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 -- 172.21.15.53:0/3701601112 learned_addr learned my addr 172.21.15.53:0/3701601112 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:10.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 -- 172.21.15.53:0/3701601112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 msgr2=0x5588945a1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945a1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 -- 172.21.15.53:0/3701601112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 msgr2=0x5588945dd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945dd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.810+0000 7fd3eea00640 1 -- 172.21.15.53:0/3701601112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558894487b80 con 0x55889457fd70 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.811+0000 7fd3eea00640 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945d2620 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0032040 tx=0x7fd3e0090000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b17f091d50806b6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.811+0000 7fd3dfe00640 1 -- 172.21.15.53:0/3701601112 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e0082a70 con 0x55889457fd70 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.811+0000 7fd3dfe00640 1 -- 172.21.15.53:0/3701601112 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e0079c60 con 0x55889457fd70 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.811+0000 7fd3dfe00640 1 -- 172.21.15.53:0/3701601112 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e007a9d0 con 0x55889457fd70 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.811+0000 7fd3f8987740 1 -- 172.21.15.53:0/3701601112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 msgr2=0x5588945d2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.811+0000 7fd3f8987740 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945d2620 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0032040 tx=0x7fd3e0090000 comp rx=0 tx=0).stop 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 -- 172.21.15.53:0/3701601112 shutdown_connections 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x558894585070 0x5588945a1680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x5588945847b0 0x5588945dd2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 --2- 172.21.15.53:0/3701601112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55889457fd70 0x5588945d2620 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 -- 172.21.15.53:0/3701601112 >> 172.21.15.53:0/3701601112 conn(0x558894546c70 msgr2=0x55889457b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 -- 172.21.15.53:0/3701601112 shutdown_connections 2024-01-16T15:03:10.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 -- 172.21.15.53:0/3701601112 wait complete. 2024-01-16T15:03:10.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.812+0000 7fd3f8987740 1 -- start start 2024-01-16T15:03:10.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.813+0000 7fd3f8987740 1 Processor -- start 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.813+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55889457fd70 0x5588945ac4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.813+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5588945847b0 0x5588945d9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3f8987740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 0x5588945d9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3f8987740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588945ccc70 con 0x558894585070 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x558894634cc0 con 0x55889457fd70 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3f8987740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x558894634ad0 con 0x5588945847b0 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3f8987740 10 client.0 _ll_register_callbacks cb 0x5588945ced10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:03:10.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 0x5588945d9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 0x5588945d9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41494/0 (socket says 172.21.15.53:41494) 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 -- 172.21.15.53:0/1151423701 learned_addr learned my addr 172.21.15.53:0/1151423701 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3e7e00640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5588945847b0 0x5588945d9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55889457fd70 0x5588945ac4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 -- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5588945847b0 msgr2=0x5588945d9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5588945847b0 0x5588945d9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 -- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55889457fd70 msgr2=0x5588945ac4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55889457fd70 0x5588945ac4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.814+0000 7fd3ef400640 1 -- 172.21.15.53:0/1151423701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588944ea140 con 0x558894585070 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3ef400640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 0x5588945d9950 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80012e0 tx=0x7fd3e8010c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb9ebcd6f6b1358 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3d7e00640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:03:10.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1151423701 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e800e9a0 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1151423701 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3e800eb00 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd3d8019070 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1151423701 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3e800ecc0 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3f8987740 10 client.5859 fetch_fsmap learned FSMap version 25 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.815+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=2+}) v3 -- 0x5588944ad170 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1151423701 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d8019070 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1151423701 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 25) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fd3e8012040 con 0x558894585070 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3f8987740 10 client.5859 fetch_fsmap finished waiting for FSMap version 25 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3f8987740 10 client.5859 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3f8987740 20 client.5859 populate_metadata read hostname 'smithi053' 2024-01-16T15:03:10.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945cc3a0 0x5588945d1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3f8987740 4 client.5859 mds_command: new command op to 5718 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 --> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5588941ec6c0 con 0x5588945cc3a0 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945cc3a0 0x5588945d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3eea00640 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945cc3a0 0x5588945d1240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3418cac1beec9ad1 server_cookie=d59e84c6ddc7a222 in_seq=0 out_seq=0 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.816+0000 7fd3d7e00640 10 client.5859 ms_handle_connect on v2:172.21.15.53:6834/4293818684 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3d7e00640 1 -- 172.21.15.53:0/1151423701 <== mds.0 v2:172.21.15.53:6834/4293818684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x5588941ec6c0 con 0x5588945cc3a0 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3d7e00640 10 client.5859 handle_command_reply: tid=0 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 client.5859 shutdown 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 msgr2=0x5588945d9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 0x5588945d9950 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fd3e80012e0 tx=0x7fd3e8010c70 comp rx=0 tx=0).stop 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 shutdown_connections 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x558894585070 0x5588945d9950 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x5588945847b0 0x5588945d9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.53:6834/4293818684,v1:172.21.15.53:6835/4293818684] conn(0x5588945cc3a0 0x5588945d1240 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 --2- 172.21.15.53:0/1151423701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55889457fd70 0x5588945ac4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 >> 172.21.15.53:0/1151423701 conn(0x558894546c70 msgr2=0x5588945dc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3d7e00640 0 client.5859 ms_handle_reset on v2:172.21.15.53:6834/4293818684 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 shutdown_connections 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 1 -- 172.21.15.53:0/1151423701 wait complete. 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.817+0000 7fd3f8987740 20 client.5859 trim_cache size 0 max 16384 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:03:10.802 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5718, 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 6.7643492619999996, 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 25, 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 47, 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 47, 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 13.47720052 2024-01-16T15:03:10.803 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:03:10.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c806b500 msgr2=0x7fd3c806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c806b500 0x7fd3c806d9c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fd3f01349f0 tx=0x7fd3d8008040 comp rx=0 tx=0).stop 2024-01-16T15:03:10.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 msgr2=0x7fd3f01f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:10.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 0x7fd3f01f62a0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fd3e0009890 tx=0x7fd3e00046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 shutdown_connections 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c806b500 0x7fd3c806d9c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3f0143c40 0x7fd3f00c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3f00cca40 0x7fd3f00c8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 --2- 172.21.15.53:0/1802498377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3f00717d0 0x7fd3f01f62a0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 >> 172.21.15.53:0/1802498377 conn(0x7fd3f006a800 msgr2=0x7fd3f013f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 shutdown_connections 2024-01-16T15:03:10.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:10.818+0000 7fd3efe00640 1 -- 172.21.15.53:0/1802498377 wait complete. 2024-01-16T15:03:10.814 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:03:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.039+0000 7ff375000640 1 Processor -- start 2024-01-16T15:03:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.039+0000 7ff375000640 1 -- start start 2024-01-16T15:03:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff375000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff370071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff375000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3700717d0 0x7ff3700cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff375000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3700cca40 0x7ff37015eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff375000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3700723c0 con 0x7ff3700cca40 2024-01-16T15:03:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff375000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff370070e50 con 0x7ff3700717d0 2024-01-16T15:03:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff375000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3700cce70 con 0x7ff370143c40 2024-01-16T15:03:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff36fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3700cca40 0x7ff37015eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff36fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3700cca40 0x7ff37015eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41496/0 (socket says 172.21.15.53:41496) 2024-01-16T15:03:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.041+0000 7ff36fe00640 1 -- 172.21.15.53:0/1108662669 learned_addr learned my addr 172.21.15.53:0/1108662669 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:11.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff367e00640 1 --2- 172.21.15.53:0/1108662669 >> v2:172.21.15.146:3300/0 conn(0x7ff3700717d0 0x7ff3700cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff36f400640 1 --2- 172.21.15.53:0/1108662669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff370071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff36fe00640 1 -- 172.21.15.53:0/1108662669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 msgr2=0x7ff370071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff36fe00640 1 --2- 172.21.15.53:0/1108662669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff370071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff36fe00640 1 -- 172.21.15.53:0/1108662669 >> v2:172.21.15.146:3300/0 conn(0x7ff3700717d0 msgr2=0x7ff3700cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff36fe00640 1 --2- 172.21.15.53:0/1108662669 >> v2:172.21.15.146:3300/0 conn(0x7ff3700717d0 0x7ff3700cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.042+0000 7ff36fe00640 1 -- 172.21.15.53:0/1108662669 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff370144020 con 0x7ff3700cca40 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.043+0000 7ff36fe00640 1 --2- 172.21.15.53:0/1108662669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37015eab0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff37006e440 tx=0x7ff36802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d059e9643c9479d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.043+0000 7ff36ea00640 1 -- 172.21.15.53:0/1108662669 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff36802c9f0 con 0x7ff3700cca40 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.043+0000 7ff36ea00640 1 -- 172.21.15.53:0/1108662669 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff36802cb50 con 0x7ff3700cca40 2024-01-16T15:03:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.043+0000 7ff36ea00640 1 -- 172.21.15.53:0/1108662669 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3680343f0 con 0x7ff3700cca40 2024-01-16T15:03:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 -- 172.21.15.53:0/1108662669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 msgr2=0x7ff37015eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 --2- 172.21.15.53:0/1108662669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37015eab0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7ff37006e440 tx=0x7ff36802be50 comp rx=0 tx=0).stop 2024-01-16T15:03:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 -- 172.21.15.53:0/1108662669 shutdown_connections 2024-01-16T15:03:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 --2- 172.21.15.53:0/1108662669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37015eab0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 --2- 172.21.15.53:0/1108662669 >> v2:172.21.15.146:3300/0 conn(0x7ff3700717d0 0x7ff3700cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 --2- 172.21.15.53:0/1108662669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff370071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 -- 172.21.15.53:0/1108662669 >> 172.21.15.53:0/1108662669 conn(0x7ff37006a800 msgr2=0x7ff37006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:11.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 -- 172.21.15.53:0/1108662669 shutdown_connections 2024-01-16T15:03:11.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.044+0000 7ff375000640 1 -- 172.21.15.53:0/1108662669 wait complete. 2024-01-16T15:03:11.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.046+0000 7ff375000640 1 Processor -- start 2024-01-16T15:03:11.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.046+0000 7ff375000640 1 -- start start 2024-01-16T15:03:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff375000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3700717d0 0x7ff3700c5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff375000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37013ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff375000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff37013cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff375000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37013d5c0 con 0x7ff3700cca40 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff375000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37013d8a0 con 0x7ff3700717d0 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff375000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37013db80 con 0x7ff370143c40 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff367e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37013ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff367e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37013ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41504/0 (socket says 172.21.15.53:41504) 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.047+0000 7ff367e00640 1 -- 172.21.15.53:0/4088147308 learned_addr learned my addr 172.21.15.53:0/4088147308 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff36f400640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3700717d0 0x7ff3700c5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff36fe00640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff37013cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367e00640 1 -- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 msgr2=0x7ff37013cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367e00640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff37013cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367e00640 1 -- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3700717d0 msgr2=0x7ff3700c5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367e00640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3700717d0 0x7ff3700c5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367e00640 1 -- 172.21.15.53:0/4088147308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff368009b30 con 0x7ff3700cca40 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff36f400640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3700717d0 0x7ff3700c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367e00640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37013ecf0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff35800d5f0 tx=0x7ff35800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3580125f0 con 0x7ff3700cca40 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff358012750 con 0x7ff3700cca40 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37013de60 con 0x7ff3700cca40 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.048+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff358012b00 con 0x7ff3700cca40 2024-01-16T15:03:11.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.049+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37013e660 con 0x7ff3700cca40 2024-01-16T15:03:11.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.050+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff370067450 con 0x7ff3700cca40 2024-01-16T15:03:11.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.051+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7ff358012c60 con 0x7ff3700cca40 2024-01-16T15:03:11.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.052+0000 7ff367400640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff34806b500 0x7ff34806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:11.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.052+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff35808a4e0 con 0x7ff3700cca40 2024-01-16T15:03:11.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.052+0000 7ff36f400640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff34806b500 0x7ff34806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:11.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.053+0000 7ff36f400640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff34806b500 0x7ff34806d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff360006fd0 tx=0x7ff360008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.056+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff358054fe0 con 0x7ff3700cca40 2024-01-16T15:03:11.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.293+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff3700cb9a0 con 0x7ff3700cca40 2024-01-16T15:03:11.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.293+0000 7ff367400640 1 -- 172.21.15.53:0/4088147308 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff358058600 con 0x7ff3700cca40 2024-01-16T15:03:11.274 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff34806b500 msgr2=0x7ff34806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff34806b500 0x7ff34806d9c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff360006fd0 tx=0x7ff360008040 comp rx=0 tx=0).stop 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 msgr2=0x7ff37013ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37013ecf0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff35800d5f0 tx=0x7ff35800dab0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 shutdown_connections 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff34806b500 0x7ff34806d9c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff370143c40 0x7ff37013cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3700cca40 0x7ff37013ecf0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 --2- 172.21.15.53:0/4088147308 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3700717d0 0x7ff3700c5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:11.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 >> 172.21.15.53:0/4088147308 conn(0x7ff37006a800 msgr2=0x7ff37006d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:11.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 shutdown_connections 2024-01-16T15:03:11.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:11.296+0000 7ff375000640 1 -- 172.21.15.53:0/4088147308 wait complete. 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:03:11.296 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:03:11.297 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:03:11.297 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:03:11.297 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:11.297 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:03:11.348 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:03:11.349 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42459sec preferred_lft 42459sec 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:03:11.350 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:03:11.351 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.352 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:03:11.352 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:03:11.352 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:03:11.352 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:11.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:11.353 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:03:11.441 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:03:11.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:11.480 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:03:11.542 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:03:11.543 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:03:11.543 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:03:11.543 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:03:11.543 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:03:11.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:11.544 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:03:11.544 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:03:11.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:03:11.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:03:11.544 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:03:11.544 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:03:11.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:11.754 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:03:11.807 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:03:11.807 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:03:11.807 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:03:11.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:11.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:03:11.903 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:03:11.903 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:03:11.904 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:03:11.958 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:03:11.958 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:03:11.958 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:03:12.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:03:12.114 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:03:12.171 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:03:12.171 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:03:12.171 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:03:12.172 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:03:12.173 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:03:12.174 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:03:12.174 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:03:12.174 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:03:12.174 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:03:12.174 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:03:12.174 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:03:12.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:12.175 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:03:12.232 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:03:12.232 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:03:12.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:03:12.330 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:03:12.376 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:03:12.376 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:03:12.376 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:03:12.377 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:03:12.378 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:03:12.379 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:03:12.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:12.381 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:03:12.446 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:03:12.466+0000 7f19ec252480 -1 init, newargv = 0x55857fadcab0 newargc=15 2024-01-16T15:03:12.447 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[52085]: starting ceph client 2024-01-16T15:03:12.456 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[52085]: starting fuse 2024-01-16T15:03:12.469 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:03:12.470 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:03:12.470 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:03:12.471 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:03:12.472 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:03:12.688 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.52085.asok 2024-01-16T15:03:12.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:12 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:03:12.706 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52085.asok 2024-01-16T15:03:12.706 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:12.706 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52085.asok status 2024-01-16T15:03:12.953 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "52085", 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:03:12.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5871, 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 5871 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:03:12.955 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 4259469329 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:03:12.956 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 4259469329 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.5871 192.168.0.1:0/4259469329", 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/4259469329", 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 25, 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 47, 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:03:12.957 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:03:12.958 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:03:12.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:12.970 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:03:12.970 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:03:13.029 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:03:13.029 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:03:13.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:13.030 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:03:13.135 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:03:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.383+0000 7fc593400640 1 Processor -- start 2024-01-16T15:03:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.384+0000 7fc593400640 1 -- start start 2024-01-16T15:03:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.385+0000 7fc593400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc58c143c40 0x7fc58c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.385+0000 7fc593400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c070e50 con 0x7fc58c0ccc10 2024-01-16T15:03:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.385+0000 7fc593400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c144020 con 0x7fc58c0717d0 2024-01-16T15:03:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.385+0000 7fc593400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c0723c0 con 0x7fc58c143c40 2024-01-16T15:03:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.386+0000 7fc592a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc58c0ccc10 0x7fc58c06d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36832/0 (socket says 172.21.15.53:36832) 2024-01-16T15:03:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.386+0000 7fc592a00640 1 -- 172.21.15.53:0/2672758293 learned_addr learned my addr 172.21.15.53:0/2672758293 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.386+0000 7fc592000640 1 --2- 172.21.15.53:0/2672758293 >> v2:172.21.15.146:3300/0 conn(0x7fc58c143c40 0x7fc58c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.386+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2884705188 0 0) 0x7fc58c070e50 con 0x7fc58c0ccc10 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.386+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc56c003520 con 0x7fc58c0ccc10 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3679612344 0 0) 0x7fc56c003520 con 0x7fc58c0ccc10 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 >> v2:172.21.15.146:3300/0 conn(0x7fc58c143c40 msgr2=0x7fc58c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 --2- 172.21.15.53:0/2672758293 >> v2:172.21.15.146:3300/0 conn(0x7fc58c143c40 0x7fc58c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 >> v1:172.21.15.146:6789/0 conn(0x7fc58c0717d0 legacy=0x7fc58c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c071e00 con 0x7fc58c0ccc10 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fc57c001670 con 0x7fc58c0ccc10 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 >> v1:172.21.15.53:6789/0 conn(0x7fc58c0ccc10 legacy=0x7fc58c06d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc56c003970 0x7fc56c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c0047c0 0x7fc56c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc56c005180 0x7fc56c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c0723c0 con 0x7fc56c003970 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c00c120 con 0x7fc56c0047c0 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c00c2e0 con 0x7fc56c005180 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc592000640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc56c003970 0x7fc56c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.387+0000 7fc591600640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c0047c0 0x7fc56c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc592000640 1 -- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc56c005180 msgr2=0x7fc56c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc592000640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc56c005180 0x7fc56c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc592000640 1 -- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c0047c0 msgr2=0x7fc56c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc592000640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c0047c0 0x7fc56c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc592000640 1 -- 172.21.15.53:0/2672758293 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56c003520 con 0x7fc56c003970 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc592a00640 1 -- 172.21.15.53:0/2672758293 reap_dead start 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.388+0000 7fc591600640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c0047c0 0x7fc56c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.389+0000 7fc592000640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc56c003970 0x7fc56c003d90 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc584009ac0 tx=0x7fc56c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecd30ed875fc32 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.389+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58402c6d0 con 0x7fc56c003970 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.389+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58402c830 con 0x7fc56c003970 2024-01-16T15:03:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.389+0000 7fc590c00640 1 -- 172.21.15.53:0/2672758293 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58402cbc0 con 0x7fc56c003970 2024-01-16T15:03:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.389+0000 7fc593400640 1 -- 172.21.15.53:0/2672758293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc56c003970 msgr2=0x7fc56c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.389+0000 7fc593400640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc56c003970 0x7fc56c003d90 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fc584009ac0 tx=0x7fc56c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.390+0000 7fc593400640 1 -- 172.21.15.53:0/2672758293 shutdown_connections 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.390+0000 7fc593400640 1 --2- 172.21.15.53:0/2672758293 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc56c003970 0x7fc56c003d90 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.390+0000 7fc593400640 1 -- 172.21.15.53:0/2672758293 >> 172.21.15.53:0/2672758293 conn(0x7fc58c06a800 msgr2=0x7fc58c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.390+0000 7fc593400640 1 -- 172.21.15.53:0/2672758293 shutdown_connections 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.390+0000 7fc593400640 1 -- 172.21.15.53:0/2672758293 wait complete. 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.391+0000 7fc593400640 1 Processor -- start 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.392+0000 7fc593400640 1 -- start start 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.392+0000 7fc593400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c003970 0x7fc58c0b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.392+0000 7fc593400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc58c15a2e0 0x7fc58c0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.392+0000 7fc593400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 0x7fc58c1eea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc593400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c0b2600 con 0x7fc58c15de10 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc593400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c15ab50 con 0x7fc56c003970 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc593400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c15c910 con 0x7fc58c15a2e0 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 0x7fc58c1eea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 0x7fc58c1eea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54882/0 (socket says 172.21.15.53:54882) 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 -- 172.21.15.53:0/3908535086 learned_addr learned my addr 172.21.15.53:0/3908535086 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592000640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c003970 0x7fc58c0b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 -- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc58c15a2e0 msgr2=0x7fc58c0b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc58c15a2e0 0x7fc58c0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 -- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c003970 msgr2=0x7fc58c0b3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c003970 0x7fc58c0b3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592a00640 1 -- 172.21.15.53:0/3908535086 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c15cbf0 con 0x7fc58c15de10 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.393+0000 7fc592000640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c003970 0x7fc58c0b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.394+0000 7fc592a00640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 0x7fc58c1eea20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc57c000f60 tx=0x7fc57c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.394+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c0072d0 con 0x7fc58c15de10 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.394+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c00ced0 con 0x7fc58c15de10 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.394+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c007670 con 0x7fc58c15de10 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.394+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c0b2850 con 0x7fc58c15de10 2024-01-16T15:03:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.394+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c0b3570 con 0x7fc58c15de10 2024-01-16T15:03:13.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.395+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c067450 con 0x7fc58c15de10 2024-01-16T15:03:13.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.396+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fc57c0047e0 con 0x7fc58c15de10 2024-01-16T15:03:13.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.398+0000 7fc58a000640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc56406b500 0x7fc56406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:13.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.398+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc57c089620 con 0x7fc58c15de10 2024-01-16T15:03:13.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.398+0000 7fc592000640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc56406b500 0x7fc56406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:13.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.399+0000 7fc592000640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc56406b500 0x7fc56406d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc584036000 tx=0x7fc584030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:13.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.403+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc57c054120 con 0x7fc58c15de10 2024-01-16T15:03:13.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.634+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc58c0ceb30 con 0x7fc58c15de10 2024-01-16T15:03:13.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.634+0000 7fc58a000640 1 -- 172.21.15.53:0/3908535086 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v25) v1 ==== 53+0+73 (secure 0 0 0) 0x7fc57c01a060 con 0x7fc58c15de10 2024-01-16T15:03:13.614 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.636+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc56406b500 msgr2=0x7fc56406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.636+0000 7fc593400640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc56406b500 0x7fc56406d9c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fc584036000 tx=0x7fc584030000 comp rx=0 tx=0).stop 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.636+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 msgr2=0x7fc58c1eea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.636+0000 7fc593400640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 0x7fc58c1eea20 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fc57c000f60 tx=0x7fc57c00cb90 comp rx=0 tx=0).stop 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 shutdown_connections 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc56406b500 0x7fc56406d9c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58c15de10 0x7fc58c1eea20 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc58c15a2e0 0x7fc58c0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 --2- 172.21.15.53:0/3908535086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc56c003970 0x7fc58c0b3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 >> 172.21.15.53:0/3908535086 conn(0x7fc58c06a800 msgr2=0x7fc58c13e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 shutdown_connections 2024-01-16T15:03:13.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:13.637+0000 7fc593400640 1 -- 172.21.15.53:0/3908535086 wait complete. 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:03:13.637 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:03:13.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:13.638 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:03:13.697 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42336sec preferred_lft 42336sec 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:03:13.698 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:03:13.699 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:03:13.700 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:03:13.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:13.701 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:03:13.790 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:03:13.794 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:13.794 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:03:13.844 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:03:13.844 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:03:13.844 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:03:13.844 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:03:13.844 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:03:13.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:13.844 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:03:13.844 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:03:13.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:03:13.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:03:13.845 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:03:13.845 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:03:14.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:14.021 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:03:14.071 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:03:14.071 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:03:14.072 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:03:14.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:14.072 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:03:14.163 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:03:14.163 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:03:14.163 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:03:14.217 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:03:14.217 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:03:14.217 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:03:14.268 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:03:14.359 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:03:14.414 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:03:14.414 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:03:14.415 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:03:14.416 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:03:14.417 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:03:14.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:14.418 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:03:14.471 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:03:14.471 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:03:14.476 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:03:14.549 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:03:14.591 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:03:14.591 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:03:14.592 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:03:14.593 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:03:14.594 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:03:14.594 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:03:14.594 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:03:14.594 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:03:14.594 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:03:14.594 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:03:14.595 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:14.595 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:03:14.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:03:14.683+0000 7f8c5ac35480 -1 init, newargv = 0x555c581e3660 newargc=15 2024-01-16T15:03:14.663 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[43192]: starting ceph client 2024-01-16T15:03:14.674 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[43192]: starting fuse 2024-01-16T15:03:14.682 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:03:14.682 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:03:14.682 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:03:14.682 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:03:14.683 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:03:14.684 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:03:14.874 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.43192.asok 2024-01-16T15:03:14.879 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:03:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:03:14.892 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.43192.asok 2024-01-16T15:03:14.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:14.892 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.43192.asok status 2024-01-16T15:03:15.122 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:03:15.122 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "43192", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5443, 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:03:15.123 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 5443 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3710677017 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 3710677017 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.5443 192.168.0.1:0/3710677017", 2024-01-16T15:03:15.124 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/3710677017", 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 25, 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 47, 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:03:15.125 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:03:15.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:15.138 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:03:15.138 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:03:15.189 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:03:15.189 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:03:15.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:15.189 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:03:15.282 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:03:15.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.534+0000 7fce32200640 1 Processor -- start 2024-01-16T15:03:15.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.535+0000 7fce32200640 1 -- start start 2024-01-16T15:03:15.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.536+0000 7fce32200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fce2c0d0100 0x7fce2c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.536+0000 7fce32200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0d0a10 con 0x7fce2c160920 2024-01-16T15:03:15.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.536+0000 7fce32200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce2c0cf080 con 0x7fce2c0d0f40 2024-01-16T15:03:15.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.536+0000 7fce32200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce2c0c6be0 con 0x7fce2c0d0100 2024-01-16T15:03:15.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.537+0000 7fce30e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fce2c160920 0x7fce2c0cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36838/0 (socket says 172.21.15.53:36838) 2024-01-16T15:03:15.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.537+0000 7fce30e00640 1 -- 172.21.15.53:0/425683592 learned_addr learned my addr 172.21.15.53:0/425683592 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:15.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.537+0000 7fce2be00640 1 --2- 172.21.15.53:0/425683592 >> v2:172.21.15.146:3300/0 conn(0x7fce2c0d0100 0x7fce2c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.537+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016169995 0 0) 0x7fce2c0d0a10 con 0x7fce2c160920 2024-01-16T15:03:15.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.537+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce0c003520 con 0x7fce2c160920 2024-01-16T15:03:15.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909113492 0 0) 0x7fce0c003520 con 0x7fce2c160920 2024-01-16T15:03:15.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 >> v2:172.21.15.146:3300/0 conn(0x7fce2c0d0100 msgr2=0x7fce2c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 --2- 172.21.15.53:0/425683592 >> v2:172.21.15.146:3300/0 conn(0x7fce2c0d0100 0x7fce2c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 >> v1:172.21.15.146:6789/0 conn(0x7fce2c0d0f40 legacy=0x7fce2c0d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c0ce7e0 con 0x7fce2c160920 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fce1c001670 con 0x7fce2c160920 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 >> v1:172.21.15.53:6789/0 conn(0x7fce2c160920 legacy=0x7fce2c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0c003970 0x7fce0c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce0c0047c0 0x7fce0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c005180 0x7fce0c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0c6be0 con 0x7fce0c003970 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c00c020 con 0x7fce0c005180 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce0c00c1e0 con 0x7fce0c0047c0 2024-01-16T15:03:15.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.538+0000 7fce30e00640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0c003970 0x7fce0c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce2be00640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce0c0047c0 0x7fce0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce31800640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c005180 0x7fce0c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce30e00640 1 -- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce0c0047c0 msgr2=0x7fce0c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce30e00640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce0c0047c0 0x7fce0c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce30e00640 1 -- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c005180 msgr2=0x7fce0c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce30e00640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c005180 0x7fce0c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce31800640 1 -- 172.21.15.53:0/425683592 reap_dead start 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce30e00640 1 -- 172.21.15.53:0/425683592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c003520 con 0x7fce0c003970 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.539+0000 7fce2be00640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce0c0047c0 0x7fce0c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.540+0000 7fce30e00640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0c003970 0x7fce0c003d90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c190 tx=0x7fce0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a8a767d66a9cbd6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.540+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c005ad0 con 0x7fce0c003970 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.540+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce1c005c30 con 0x7fce0c003970 2024-01-16T15:03:15.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.540+0000 7fce2b400640 1 -- 172.21.15.53:0/425683592 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce1c02caf0 con 0x7fce0c003970 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 -- 172.21.15.53:0/425683592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0c003970 msgr2=0x7fce0c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0c003970 0x7fce0c003d90 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fce1c00c190 tx=0x7fce0c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 -- 172.21.15.53:0/425683592 shutdown_connections 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 --2- 172.21.15.53:0/425683592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce0c003970 0x7fce0c003d90 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 -- 172.21.15.53:0/425683592 >> 172.21.15.53:0/425683592 conn(0x7fce2c0d3090 msgr2=0x7fce2c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 -- 172.21.15.53:0/425683592 shutdown_connections 2024-01-16T15:03:15.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.541+0000 7fce32200640 1 -- 172.21.15.53:0/425683592 wait complete. 2024-01-16T15:03:15.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.543+0000 7fce32200640 1 Processor -- start 2024-01-16T15:03:15.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.543+0000 7fce32200640 1 -- start start 2024-01-16T15:03:15.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce32200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c003970 0x7fce2c0ccc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce32200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce2c153f60 0x7fce2c0b7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce32200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 0x7fce2c155cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce32200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0b8890 con 0x7fce2c0b8410 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce32200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0cd1e0 con 0x7fce0c003970 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce32200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2c0cab70 con 0x7fce2c153f60 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce31800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 0x7fce2c155cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.544+0000 7fce31800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 0x7fce2c155cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54892/0 (socket says 172.21.15.53:54892) 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 -- 172.21.15.53:0/2475333834 learned_addr learned my addr 172.21.15.53:0/2475333834 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce2be00640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce2c153f60 0x7fce2c0b7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce30e00640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c003970 0x7fce2c0ccc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 -- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce2c153f60 msgr2=0x7fce2c0b7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce2c153f60 0x7fce2c0b7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 -- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c003970 msgr2=0x7fce2c0ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c003970 0x7fce2c0ccc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 -- 172.21.15.53:0/2475333834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce2c0cae50 con 0x7fce2c0b8410 2024-01-16T15:03:15.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce31800640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 0x7fce2c155cc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0d3060 tx=0x7fce2400d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24003d80 con 0x7fce2c0b8410 2024-01-16T15:03:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce240050f0 con 0x7fce2c0b8410 2024-01-16T15:03:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.545+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce24005460 con 0x7fce2c0b8410 2024-01-16T15:03:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.546+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce2c0cb130 con 0x7fce2c0b8410 2024-01-16T15:03:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.546+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce2c1543e0 con 0x7fce2c0b8410 2024-01-16T15:03:15.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.547+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce2c0cb650 con 0x7fce2c0b8410 2024-01-16T15:03:15.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.548+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fce240055c0 con 0x7fce2c0b8410 2024-01-16T15:03:15.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.549+0000 7fce28c00640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce0406b530 0x7fce0406d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:15.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.549+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fce24089970 con 0x7fce2c0b8410 2024-01-16T15:03:15.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.549+0000 7fce30e00640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce0406b530 0x7fce0406d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:15.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.551+0000 7fce30e00640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce0406b530 0x7fce0406d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce1c034000 tx=0x7fce1c009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:15.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.554+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fce2c0cb650 con 0x7fce2c0b8410 2024-01-16T15:03:15.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:15.789+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fce2c0cb650 con 0x7fce2c0b8410 2024-01-16T15:03:16.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.110+0000 7fce28c00640 1 -- 172.21.15.53:0/2475333834 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 136+0+0 (secure 0 0 0) 0x7fce2c0cb650 con 0x7fce2c0b8410 2024-01-16T15:03:16.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce0406b530 msgr2=0x7fce0406d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce0406b530 0x7fce0406d9f0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fce1c034000 tx=0x7fce1c009030 comp rx=0 tx=0).stop 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 msgr2=0x7fce2c155cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 0x7fce2c155cc0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0d3060 tx=0x7fce2400d9f0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 shutdown_connections 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fce0406b530 0x7fce0406d9f0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fce2c0b8410 0x7fce2c155cc0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fce2c153f60 0x7fce2c0b7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 --2- 172.21.15.53:0/2475333834 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fce0c003970 0x7fce2c0ccc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 >> 172.21.15.53:0/2475333834 conn(0x7fce2c0d3090 msgr2=0x7fce2c0c7330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 shutdown_connections 2024-01-16T15:03:16.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.114+0000 7fce32200640 1 -- 172.21.15.53:0/2475333834 wait complete. 2024-01-16T15:03:16.116 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:03:16.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.359+0000 7fcc4b400640 1 Processor -- start 2024-01-16T15:03:16.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.359+0000 7fcc4b400640 1 -- start start 2024-01-16T15:03:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc4b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc440cd240 0x7fcc440b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc4b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4409a170 0x7fcc440cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc4b400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4406cea0 con 0x7fcc4406c6b0 2024-01-16T15:03:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc4b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4406cbc0 con 0x7fcc440cd240 2024-01-16T15:03:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc4b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4406f350 con 0x7fcc4409a170 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc48a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc440cd240 0x7fcc440b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc43e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcc4406c6b0 0x7fcc44099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36850/0 (socket says 172.21.15.53:36850) 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.361+0000 7fcc43e00640 1 -- 172.21.15.53:0/825556380 learned_addr learned my addr 172.21.15.53:0/825556380 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc49400640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4409a170 0x7fcc440cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc48a00640 1 -- 172.21.15.53:0/825556380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4409a170 msgr2=0x7fcc440cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc48a00640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4409a170 0x7fcc440cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc48a00640 1 -- 172.21.15.53:0/825556380 >> v1:172.21.15.53:6789/0 conn(0x7fcc4406c6b0 legacy=0x7fcc44099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc43400640 1 -- 172.21.15.53:0/825556380 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894860650 0 0) 0x7fcc4406cea0 con 0x7fcc4406c6b0 2024-01-16T15:03:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc48a00640 1 -- 172.21.15.53:0/825556380 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4406f630 con 0x7fcc440cd240 2024-01-16T15:03:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.362+0000 7fcc49400640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4409a170 0x7fcc440cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.363+0000 7fcc48a00640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc440b3820 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009d90 tx=0x7fcc3402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5d24f3f2d7e507 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.363+0000 7fcc43400640 1 -- 172.21.15.53:0/825556380 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3402cc60 con 0x7fcc440cd240 2024-01-16T15:03:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.363+0000 7fcc43400640 1 -- 172.21.15.53:0/825556380 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3402cdc0 con 0x7fcc440cd240 2024-01-16T15:03:16.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.363+0000 7fcc43400640 1 -- 172.21.15.53:0/825556380 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34034660 con 0x7fcc440cd240 2024-01-16T15:03:16.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 -- 172.21.15.53:0/825556380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 msgr2=0x7fcc440b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc440b3820 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009d90 tx=0x7fcc3402beb0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 -- 172.21.15.53:0/825556380 shutdown_connections 2024-01-16T15:03:16.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4409a170 0x7fcc440cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 --2- 172.21.15.53:0/825556380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc440b3820 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 -- 172.21.15.53:0/825556380 >> 172.21.15.53:0/825556380 conn(0x7fcc440b2d00 msgr2=0x7fcc440b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 -- 172.21.15.53:0/825556380 shutdown_connections 2024-01-16T15:03:16.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.364+0000 7fcc4b400640 1 -- 172.21.15.53:0/825556380 wait complete. 2024-01-16T15:03:16.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.366+0000 7fcc4b400640 1 Processor -- start 2024-01-16T15:03:16.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.366+0000 7fcc4b400640 1 -- start start 2024-01-16T15:03:16.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc4b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4406c6b0 0x7fcc4415da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc4b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc4409a170 0x7fcc4415e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc4b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc4415ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc4b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440b4e80 con 0x7fcc440cd240 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc4b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4415df60 con 0x7fcc4409a170 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc4b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc440b4200 con 0x7fcc4406c6b0 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc49400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc4415ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc49400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc4415ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54918/0 (socket says 172.21.15.53:54918) 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.367+0000 7fcc49400640 1 -- 172.21.15.53:0/4174734118 learned_addr learned my addr 172.21.15.53:0/4174734118 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc48a00640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4406c6b0 0x7fcc4415da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc43e00640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc4409a170 0x7fcc4415e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc49400640 1 -- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4406c6b0 msgr2=0x7fcc4415da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc49400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4406c6b0 0x7fcc4415da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc49400640 1 -- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc4409a170 msgr2=0x7fcc4415e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc49400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc4409a170 0x7fcc4415e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc49400640 1 -- 172.21.15.53:0/4174734118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc34009a70 con 0x7fcc440cd240 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc49400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc4415ec10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c820 tx=0x7fcc3c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c007a30 con 0x7fcc440cd240 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c007b90 con 0x7fcc440cd240 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3c00ce10 con 0x7fcc440cd240 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.368+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc440b44e0 con 0x7fcc440cd240 2024-01-16T15:03:16.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.369+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc440b4aa0 con 0x7fcc440cd240 2024-01-16T15:03:16.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.370+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc440b4fc0 con 0x7fcc440cd240 2024-01-16T15:03:16.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.371+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7fcc3c0113f0 con 0x7fcc440cd240 2024-01-16T15:03:16.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.372+0000 7fcc40c00640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc1c06b530 0x7fcc1c06d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.372+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc3c08a250 con 0x7fcc440cd240 2024-01-16T15:03:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.373+0000 7fcc48a00640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc1c06b530 0x7fcc1c06d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.374+0000 7fcc48a00640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc1c06b530 0x7fcc1c06d9f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcc34036000 tx=0x7fcc34030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.377+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc440b4fc0 con 0x7fcc440cd240 2024-01-16T15:03:16.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.600+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc440b4fc0 con 0x7fcc1c06b530 2024-01-16T15:03:16.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.604+0000 7fcc40c00640 1 -- 172.21.15.53:0/4174734118 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcc440b4fc0 con 0x7fcc1c06b530 2024-01-16T15:03:16.584 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:03:16.584 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:03:16.584 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:03:16.584 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:03:16.584 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:03:16.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.606+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc1c06b530 msgr2=0x7fcc1c06d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.606+0000 7fcc4b400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc1c06b530 0x7fcc1c06d9f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcc34036000 tx=0x7fcc34030000 comp rx=0 tx=0).stop 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.606+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 msgr2=0x7fcc4415ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.606+0000 7fcc4b400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc4415ec10 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcc3c00c820 tx=0x7fcc3c00cce0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 shutdown_connections 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc1c06b530 0x7fcc1c06d9f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc440cd240 0x7fcc4415ec10 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc4409a170 0x7fcc4415e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 --2- 172.21.15.53:0/4174734118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc4406c6b0 0x7fcc4415da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 >> 172.21.15.53:0/4174734118 conn(0x7fcc440b2d00 msgr2=0x7fcc440b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 shutdown_connections 2024-01-16T15:03:16.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.607+0000 7fcc4b400640 1 -- 172.21.15.53:0/4174734118 wait complete. 2024-01-16T15:03:16.606 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:03:16.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.848+0000 7f8936c00640 1 Processor -- start 2024-01-16T15:03:16.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.848+0000 7f8936c00640 1 -- start start 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.849+0000 7f8936c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89301447f0 0x7f893015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.849+0000 7f8936c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f893015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.849+0000 7f8936c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893015c5c0 con 0x7f893006c470 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.849+0000 7f8936c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893006d690 con 0x7f89301447f0 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.849+0000 7f8936c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89300cebb0 con 0x7f89300cccf0 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89301447f0 0x7f893015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89301447f0 0x7f893015c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54932/0 (socket says 172.21.15.53:54932) 2024-01-16T15:03:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 -- 172.21.15.53:0/383241065 learned_addr learned my addr 172.21.15.53:0/383241065 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8936200640 1 --2- 172.21.15.53:0/383241065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f893015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 -- 172.21.15.53:0/383241065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 msgr2=0x7f893015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 --2- 172.21.15.53:0/383241065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f893015c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 -- 172.21.15.53:0/383241065 >> v1:172.21.15.146:6789/0 conn(0x7f893006c470 legacy=0x7f89300cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.850+0000 7f8935800640 1 -- 172.21.15.53:0/383241065 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893006ca10 con 0x7f89301447f0 2024-01-16T15:03:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.851+0000 7f8935800640 1 --2- 172.21.15.53:0/383241065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89301447f0 0x7f893015c080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f89300cecf0 tx=0x7f892002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92cdde57c84baf91 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f892fe00640 1 -- 172.21.15.53:0/383241065 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89200317c0 con 0x7f89301447f0 2024-01-16T15:03:16.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f892fe00640 1 -- 172.21.15.53:0/383241065 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8920031920 con 0x7f89301447f0 2024-01-16T15:03:16.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f892fe00640 1 -- 172.21.15.53:0/383241065 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920031cb0 con 0x7f89301447f0 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 -- 172.21.15.53:0/383241065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89301447f0 msgr2=0x7f893015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 --2- 172.21.15.53:0/383241065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89301447f0 0x7f893015c080 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f89300cecf0 tx=0x7f892002b860 comp rx=0 tx=0).stop 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 -- 172.21.15.53:0/383241065 shutdown_connections 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 --2- 172.21.15.53:0/383241065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f893015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 --2- 172.21.15.53:0/383241065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89301447f0 0x7f893015c080 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 -- 172.21.15.53:0/383241065 >> 172.21.15.53:0/383241065 conn(0x7f893006a3e0 msgr2=0x7f893006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 -- 172.21.15.53:0/383241065 shutdown_connections 2024-01-16T15:03:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.852+0000 7f8936c00640 1 -- 172.21.15.53:0/383241065 wait complete. 2024-01-16T15:03:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.854+0000 7f8936c00640 1 Processor -- start 2024-01-16T15:03:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.854+0000 7f8936c00640 1 -- start start 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8936c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 0x7f8930140480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8936c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f89300cb510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8936c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89301447f0 0x7f89300cba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8936c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893012ee60 con 0x7f893006c470 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8936c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89300cbf90 con 0x7f89301447f0 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8936c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893012e550 con 0x7f89300cccf0 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.855+0000 7f8935800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 0x7f8930140480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 0x7f8930140480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54944/0 (socket says 172.21.15.53:54944) 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 -- 172.21.15.53:0/3954890618 learned_addr learned my addr 172.21.15.53:0/3954890618 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8934e00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f89300cb510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8936200640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89301447f0 0x7f89300cba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 -- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 msgr2=0x7f89300cb510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f89300cb510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 -- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89301447f0 msgr2=0x7f89300cba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89301447f0 0x7f89300cba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 -- 172.21.15.53:0/3954890618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89200094d0 con 0x7f893006c470 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8936200640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89301447f0 0x7f89300cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.856+0000 7f8935800640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 0x7f8930140480 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8920000c00 tx=0x7f89200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.857+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920032920 con 0x7f893006c470 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.857+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8920032aa0 con 0x7f893006c470 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.857+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8920032e50 con 0x7f893006c470 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.857+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8930131a40 con 0x7f893006c470 2024-01-16T15:03:16.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.857+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893012e7a0 con 0x7f893006c470 2024-01-16T15:03:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.858+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893012efa0 con 0x7f893006c470 2024-01-16T15:03:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.859+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f892004d3f0 con 0x7f893006c470 2024-01-16T15:03:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.860+0000 7f892d600640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f890806b530 0x7f890806d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.860+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f89200abec0 con 0x7f893006c470 2024-01-16T15:03:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.861+0000 7f8934e00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f890806b530 0x7f890806d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.862+0000 7f8934e00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f890806b530 0x7f890806d9f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8930143ef0 tx=0x7f8918009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:16.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:16.865+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89200769c0 con 0x7f893006c470 2024-01-16T15:03:17.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.057+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8930067450 con 0x7f893006c470 2024-01-16T15:03:17.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.058+0000 7f892d600640 1 -- 172.21.15.53:0/3954890618 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8920079fe0 con 0x7f893006c470 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f890806b530 msgr2=0x7f890806d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f890806b530 0x7f890806d9f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f8930143ef0 tx=0x7f8918009040 comp rx=0 tx=0).stop 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 msgr2=0x7f8930140480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 0x7f8930140480 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8920000c00 tx=0x7f89200046c0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 shutdown_connections 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f890806b530 0x7f890806d9f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89301447f0 0x7f89300cba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89300cccf0 0x7f89300cb510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 --2- 172.21.15.53:0/3954890618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f893006c470 0x7f8930140480 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.060+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 >> 172.21.15.53:0/3954890618 conn(0x7f893006a3e0 msgr2=0x7f893015c7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.061+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 shutdown_connections 2024-01-16T15:03:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.061+0000 7f8936c00640 1 -- 172.21.15.53:0/3954890618 wait complete. 2024-01-16T15:03:17.061 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:03:17.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.305+0000 7f7d54e00640 1 Processor -- start 2024-01-16T15:03:17.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.305+0000 7f7d54e00640 1 -- start start 2024-01-16T15:03:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d50143c40 0x7f7d50071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d54e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7d500ccc10 0x7f7d5006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d54e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d50070e50 con 0x7f7d500717d0 2024-01-16T15:03:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d54e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d500723c0 con 0x7f7d500ccc10 2024-01-16T15:03:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d54e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d50144020 con 0x7f7d50143c40 2024-01-16T15:03:17.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d4fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7d500ccc10 0x7f7d5006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d4fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7d500ccc10 0x7f7d5006d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54954/0 (socket says 172.21.15.53:54954) 2024-01-16T15:03:17.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d4fe00640 1 -- 172.21.15.53:0/1656038281 learned_addr learned my addr 172.21.15.53:0/1656038281 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:17.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.307+0000 7f7d4f400640 1 --2- 172.21.15.53:0/1656038281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d50143c40 0x7f7d50071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.308+0000 7f7d4fe00640 1 -- 172.21.15.53:0/1656038281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d50143c40 msgr2=0x7f7d50071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.308+0000 7f7d4fe00640 1 --2- 172.21.15.53:0/1656038281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d50143c40 0x7f7d50071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.308+0000 7f7d4fe00640 1 -- 172.21.15.53:0/1656038281 >> v1:172.21.15.146:6789/0 conn(0x7f7d500717d0 legacy=0x7f7d500cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.308+0000 7f7d4fe00640 1 -- 172.21.15.53:0/1656038281 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d50071e00 con 0x7f7d500ccc10 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.309+0000 7f7d4fe00640 1 --2- 172.21.15.53:0/1656038281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d500ccc10 0x7f7d5006d160 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009e50 tx=0x7f7d4802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=792d7dd1ce8a993f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.309+0000 7f7d4ea00640 1 -- 172.21.15.53:0/1656038281 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4802ca80 con 0x7f7d500ccc10 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.309+0000 7f7d4ea00640 1 -- 172.21.15.53:0/1656038281 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4802cbe0 con 0x7f7d500ccc10 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.309+0000 7f7d4ea00640 1 -- 172.21.15.53:0/1656038281 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d480343f0 con 0x7f7d500ccc10 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 -- 172.21.15.53:0/1656038281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d500ccc10 msgr2=0x7f7d5006d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 --2- 172.21.15.53:0/1656038281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d500ccc10 0x7f7d5006d160 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009e50 tx=0x7f7d4802bee0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 -- 172.21.15.53:0/1656038281 shutdown_connections 2024-01-16T15:03:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 --2- 172.21.15.53:0/1656038281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d500ccc10 0x7f7d5006d160 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 --2- 172.21.15.53:0/1656038281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d50143c40 0x7f7d50071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 -- 172.21.15.53:0/1656038281 >> 172.21.15.53:0/1656038281 conn(0x7f7d5006a800 msgr2=0x7f7d5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 -- 172.21.15.53:0/1656038281 shutdown_connections 2024-01-16T15:03:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.310+0000 7f7d54e00640 1 -- 172.21.15.53:0/1656038281 wait complete. 2024-01-16T15:03:17.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.312+0000 7f7d54e00640 1 Processor -- start 2024-01-16T15:03:17.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.312+0000 7f7d54e00640 1 -- start start 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d500717d0 0x7f7d500b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d500ccc10 0x7f7d500add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d54e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 0x7f7d50142280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d54e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d500b2fc0 con 0x7f7d50143c40 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d54e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d501f3880 con 0x7f7d500ccc10 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d54e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d501f35a0 con 0x7f7d500717d0 2024-01-16T15:03:17.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d4fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 0x7f7d50142280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d4fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 0x7f7d50142280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54968/0 (socket says 172.21.15.53:54968) 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.313+0000 7f7d4fe00640 1 -- 172.21.15.53:0/55102377 learned_addr learned my addr 172.21.15.53:0/55102377 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4f400640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d500717d0 0x7f7d500b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d47e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d500ccc10 0x7f7d500add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4fe00640 1 -- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d500717d0 msgr2=0x7f7d500b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4fe00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d500717d0 0x7f7d500b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4fe00640 1 -- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d500ccc10 msgr2=0x7f7d500add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4fe00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d500ccc10 0x7f7d500add80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4fe00640 1 -- 172.21.15.53:0/55102377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48009b30 con 0x7f7d50143c40 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d47e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d500ccc10 0x7f7d500add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d4fe00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 0x7f7d50142280 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009e50 tx=0x7f7d48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.314+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d48009510 con 0x7f7d50143c40 2024-01-16T15:03:17.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.315+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d4803e450 con 0x7f7d50143c40 2024-01-16T15:03:17.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.315+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d500b1fa0 con 0x7f7d50143c40 2024-01-16T15:03:17.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.315+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d4803e800 con 0x7f7d50143c40 2024-01-16T15:03:17.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.315+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d500b2560 con 0x7f7d50143c40 2024-01-16T15:03:17.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.316+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d500b2780 con 0x7f7d50143c40 2024-01-16T15:03:17.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.317+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f7d4802cd50 con 0x7f7d50143c40 2024-01-16T15:03:17.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.318+0000 7f7d47400640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d2806b500 0x7f7d2806d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.318+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7d480acb50 con 0x7f7d50143c40 2024-01-16T15:03:17.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.318+0000 7f7d4f400640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d2806b500 0x7f7d2806d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.320+0000 7f7d4f400640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d2806b500 0x7f7d2806d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d40006fd0 tx=0x7f7d40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.322+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7d480775c0 con 0x7f7d50143c40 2024-01-16T15:03:17.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.544+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f7d5006cc40 con 0x7f7d2806b500 2024-01-16T15:03:17.535 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:03:17.556+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:17.536 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:03:17.556+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:17.536 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:03:17.556+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:17.536 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:03:17.556+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:17.536 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:03:17.556+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:03:17.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.636+0000 7f7d47400640 1 -- 172.21.15.53:0/55102377 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7d4000f860 con 0x7f7d2806b500 2024-01-16T15:03:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d2806b500 msgr2=0x7f7d2806d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d2806b500 0x7f7d2806d9c0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f7d40006fd0 tx=0x7f7d40008040 comp rx=0 tx=0).stop 2024-01-16T15:03:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 msgr2=0x7f7d50142280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 0x7f7d50142280 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f7d48009e50 tx=0x7f7d48004a40 comp rx=0 tx=0).stop 2024-01-16T15:03:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 shutdown_connections 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7d2806b500 0x7f7d2806d9c0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7d50143c40 0x7f7d50142280 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7d500ccc10 0x7f7d500add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 --2- 172.21.15.53:0/55102377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7d500717d0 0x7f7d500b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 >> 172.21.15.53:0/55102377 conn(0x7f7d5006a800 msgr2=0x7f7d500af170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 shutdown_connections 2024-01-16T15:03:17.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.639+0000 7f7d54e00640 1 -- 172.21.15.53:0/55102377 wait complete. 2024-01-16T15:03:17.639 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:03:17.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.882+0000 7f9402e00640 1 Processor -- start 2024-01-16T15:03:17.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.882+0000 7f9402e00640 1 -- start start 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.883+0000 7f9402e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93fc09a170 0x7f93fc0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.883+0000 7f9402e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f93fc06c6b0 0x7f93fc0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.883+0000 7f9402e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93fc06cea0 con 0x7f93fc0cd240 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.883+0000 7f9402e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93fc06cbc0 con 0x7f93fc09a170 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.883+0000 7f9402e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93fc06f350 con 0x7f93fc06c6b0 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93fc09a170 0x7f93fc0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93fc09a170 0x7f93fc0b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54974/0 (socket says 172.21.15.53:54974) 2024-01-16T15:03:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 -- 172.21.15.53:0/1366704804 learned_addr learned my addr 172.21.15.53:0/1366704804 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:17.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f9400e00640 1 --2- 172.21.15.53:0/1366704804 >> v2:172.21.15.146:3300/0 conn(0x7f93fc06c6b0 0x7f93fc0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 -- 172.21.15.53:0/1366704804 >> v2:172.21.15.146:3300/0 conn(0x7f93fc06c6b0 msgr2=0x7f93fc0cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:03:17.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 --2- 172.21.15.53:0/1366704804 >> v2:172.21.15.146:3300/0 conn(0x7f93fc06c6b0 0x7f93fc0cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 -- 172.21.15.53:0/1366704804 >> v1:172.21.15.53:6789/0 conn(0x7f93fc0cd240 legacy=0x7f93fc099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93fb400640 1 -- 172.21.15.53:0/1366704804 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc06f630 con 0x7f93fc09a170 2024-01-16T15:03:17.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f93faa00640 1 -- 172.21.15.53:0/1366704804 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925238302 0 0) 0x7f93fc06cea0 con 0x7f93fc0cd240 2024-01-16T15:03:17.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.884+0000 7f9400e00640 1 --2- 172.21.15.53:0/1366704804 >> v2:172.21.15.146:3300/0 conn(0x7f93fc06c6b0 0x7f93fc0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.886+0000 7f93fb400640 1 --2- 172.21.15.53:0/1366704804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0b3820 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0cdc40 tx=0x7f93e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=576c02148a4a77b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.886+0000 7f93faa00640 1 -- 172.21.15.53:0/1366704804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4030860 con 0x7f93fc09a170 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.886+0000 7f93faa00640 1 -- 172.21.15.53:0/1366704804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e40309c0 con 0x7f93fc09a170 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.886+0000 7f93faa00640 1 -- 172.21.15.53:0/1366704804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4030d50 con 0x7f93fc09a170 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 -- 172.21.15.53:0/1366704804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 msgr2=0x7f93fc0b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 --2- 172.21.15.53:0/1366704804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0b3820 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f93fc0cdc40 tx=0x7f93e402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 -- 172.21.15.53:0/1366704804 shutdown_connections 2024-01-16T15:03:17.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 --2- 172.21.15.53:0/1366704804 >> v2:172.21.15.146:3300/0 conn(0x7f93fc06c6b0 0x7f93fc0cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 --2- 172.21.15.53:0/1366704804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0b3820 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 -- 172.21.15.53:0/1366704804 >> 172.21.15.53:0/1366704804 conn(0x7f93fc0b2d00 msgr2=0x7f93fc0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:17.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 -- 172.21.15.53:0/1366704804 shutdown_connections 2024-01-16T15:03:17.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.887+0000 7f9402e00640 1 -- 172.21.15.53:0/1366704804 wait complete. 2024-01-16T15:03:17.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.888+0000 7f9402e00640 1 Processor -- start 2024-01-16T15:03:17.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.889+0000 7f9402e00640 1 -- start start 2024-01-16T15:03:17.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.889+0000 7f9402e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93fc06c6b0 0x7f93fc15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9402e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0c6830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9402e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93fc0cd240 0x7f93fc0c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9402e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0b5930 con 0x7f93fc09a170 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9402e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0b4d20 con 0x7f93fc0cd240 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9402e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93fc0b5000 con 0x7f93fc06c6b0 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0c6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0c6830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54978/0 (socket says 172.21.15.53:54978) 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 -- 172.21.15.53:0/2611053463 learned_addr learned my addr 172.21.15.53:0/2611053463 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fbe00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93fc06c6b0 0x7f93fc15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9400e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93fc0cd240 0x7f93fc0c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 -- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93fc06c6b0 msgr2=0x7f93fc15b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93fc06c6b0 0x7f93fc15b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 -- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93fc0cd240 msgr2=0x7f93fc0c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:17.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93fc0cd240 0x7f93fc0c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:17.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f93fb400640 1 -- 172.21.15.53:0/2611053463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e40094d0 con 0x7f93fc09a170 2024-01-16T15:03:17.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.890+0000 7f9400e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93fc0cd240 0x7f93fc0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:03:17.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.891+0000 7f93fb400640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0c6830 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f93e402b9e0 tx=0x7f93e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.891+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e4030850 con 0x7f93fc09a170 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.891+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93e4043400 con 0x7f93fc09a170 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.891+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93e40437b0 con 0x7f93fc09a170 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.891+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93fc0b52e0 con 0x7f93fc09a170 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.891+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93fc0b4060 con 0x7f93fc09a170 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.893+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93fc0675d0 con 0x7f93fc09a170 2024-01-16T15:03:17.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.893+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 9) v1 ==== 89743+0+0 (secure 0 0 0) 0x7f93e4043910 con 0x7f93fc09a170 2024-01-16T15:03:17.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.894+0000 7f93f3e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93d406b500 0x7f93d406d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:03:17.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.895+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f93e403e070 con 0x7f93fc09a170 2024-01-16T15:03:17.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.895+0000 7f93fbe00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93d406b500 0x7f93d406d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:03:17.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.897+0000 7f93fbe00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93d406b500 0x7f93d406d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:03:17.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:17.899+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93e4077f30 con 0x7f93fc09a170 2024-01-16T15:03:18.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.123+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f93fc0cd620 con 0x7f93d406b500 2024-01-16T15:03:18.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.136+0000 7f93f3e00640 1 -- 172.21.15.53:0/2611053463 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f93ec001c50 con 0x7f93d406b500 2024-01-16T15:03:18.116 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e 2024-01-16T15:03:18.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93d406b500 msgr2=0x7f93d406d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93d406b500 0x7f93d406d9c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93ec000f60 tx=0x7f93ec009040 comp rx=0 tx=0).stop 2024-01-16T15:03:18.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 msgr2=0x7f93fc0c6830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:03:18.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0c6830 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f93e402b9e0 tx=0x7f93e4004730 comp rx=0 tx=0).stop 2024-01-16T15:03:18.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 shutdown_connections 2024-01-16T15:03:18.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93d406b500 0x7f93d406d9c0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93fc0cd240 0x7f93fc0c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93fc09a170 0x7f93fc0c6830 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 --2- 172.21.15.53:0/2611053463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93fc06c6b0 0x7f93fc15b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:03:18.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 >> 172.21.15.53:0/2611053463 conn(0x7f93fc0b2d00 msgr2=0x7f93fc0b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:03:18.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 shutdown_connections 2024-01-16T15:03:18.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:03:18.139+0000 7f9402e00640 1 -- 172.21.15.53:0/2611053463 wait complete. 2024-01-16T15:03:18.138 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e 2024-01-16T15:03:18.138 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:03:18.138 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:18.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:18.193 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:20.811 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:20.820 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:35.238 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:03:35.238 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:03:35.239 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.96 s, 63.3 MB/s 2024-01-16T15:03:35.241 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:03:35.241 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:35.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:35.299 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:49.909 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:03:49.909 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:03:49.909 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.5462 s, 73.8 MB/s 2024-01-16T15:03:49.910 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:03:49.911 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:03:49.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:03:49.966 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:03:50.963 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:03:50.972 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:04.388 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:04:04.388 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:04:04.388 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.3632 s, 74.8 MB/s 2024-01-16T15:04:04.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:04:04.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.635+0000 7f3682c00640 1 Processor -- start 2024-01-16T15:04:04.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.636+0000 7f3682c00640 1 -- start start 2024-01-16T15:04:04.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.637+0000 7f3682c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f367c0c9870 0x7f367c0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.637+0000 7f3682c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c161740 0x7f367c0c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.637+0000 7f3682c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c0cf4c0 con 0x7f367c0d0560 2024-01-16T15:04:04.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.637+0000 7f3682c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c0d2ef0 con 0x7f367c0c9870 2024-01-16T15:04:04.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.637+0000 7f3682c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0c9c50 con 0x7f367c161740 2024-01-16T15:04:04.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.638+0000 7f3680c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c161740 0x7f367c0c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.638+0000 7f367be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f367c0c9870 0x7f367c0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.638+0000 7f367be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f367c0c9870 0x7f367c0ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:56744/0 (socket says 172.21.15.53:56744) 2024-01-16T15:04:04.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.638+0000 7f3680c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c161740 0x7f367c0c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:33000/0 (socket says 172.21.15.53:33000) 2024-01-16T15:04:04.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.638+0000 7f367be00640 1 -- 172.21.15.53:0/885679030 learned_addr learned my addr 172.21.15.53:0/885679030 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:04.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.639+0000 7f367be00640 1 -- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c161740 msgr2=0x7f367c0c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:04.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.639+0000 7f367be00640 1 --2- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c161740 0x7f367c0c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.639+0000 7f367be00640 1 -- 172.21.15.53:0/885679030 >> v1:172.21.15.146:6789/0 conn(0x7f367c0d0560 legacy=0x7f367c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.639+0000 7f367be00640 1 -- 172.21.15.53:0/885679030 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c0cf7a0 con 0x7f367c0c9870 2024-01-16T15:04:04.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.641+0000 7f367be00640 1 --2- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c0c9870 0x7f367c0ceb50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f366c0098a0 tx=0x7f366c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b43749ece2d26b8e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:04.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.641+0000 7f367aa00640 1 -- 172.21.15.53:0/885679030 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c030840 con 0x7f367c0c9870 2024-01-16T15:04:04.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.641+0000 7f367aa00640 1 -- 172.21.15.53:0/885679030 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c0309a0 con 0x7f367c0c9870 2024-01-16T15:04:04.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.641+0000 7f367aa00640 1 -- 172.21.15.53:0/885679030 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c030d30 con 0x7f367c0c9870 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 -- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c0c9870 msgr2=0x7f367c0ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 --2- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c0c9870 0x7f367c0ceb50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f366c0098a0 tx=0x7f366c02b980 comp rx=0 tx=0).stop 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 -- 172.21.15.53:0/885679030 shutdown_connections 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 --2- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c161740 0x7f367c0c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 --2- 172.21.15.53:0/885679030 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c0c9870 0x7f367c0ceb50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 -- 172.21.15.53:0/885679030 >> 172.21.15.53:0/885679030 conn(0x7f367c09c560 msgr2=0x7f367c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 -- 172.21.15.53:0/885679030 shutdown_connections 2024-01-16T15:04:04.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.642+0000 7f3682c00640 1 -- 172.21.15.53:0/885679030 wait complete. 2024-01-16T15:04:04.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.644+0000 7f3682c00640 1 Processor -- start 2024-01-16T15:04:04.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.645+0000 7f3682c00640 1 -- start start 2024-01-16T15:04:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f3682c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 0x7f367c0b0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f3682c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c0d0560 0x7f367c06b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f3682c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c161740 0x7f367c1502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f3682c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0ca4c0 con 0x7f367c0c9870 2024-01-16T15:04:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f3682c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0ca7a0 con 0x7f367c161740 2024-01-16T15:04:04.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f3682c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c0caa80 con 0x7f367c0d0560 2024-01-16T15:04:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f367be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 0x7f367c0b0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f367be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 0x7f367c0b0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51652/0 (socket says 172.21.15.53:51652) 2024-01-16T15:04:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.646+0000 7f367be00640 1 -- 172.21.15.53:0/2675893456 learned_addr learned my addr 172.21.15.53:0/2675893456 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367b400640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c0d0560 0x7f367c06b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f3680c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c161740 0x7f367c1502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367be00640 1 -- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c0d0560 msgr2=0x7f367c06b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367be00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c0d0560 0x7f367c06b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367be00640 1 -- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c161740 msgr2=0x7f367c1502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367be00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c161740 0x7f367c1502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367be00640 1 -- 172.21.15.53:0/2675893456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c0094d0 con 0x7f367c0c9870 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f3680c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c161740 0x7f367c1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.647+0000 7f367be00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 0x7f367c0b0980 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f366c000c00 tx=0x7f366c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.648+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c039c90 con 0x7f367c0c9870 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.648+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c039e10 con 0x7f367c0c9870 2024-01-16T15:04:04.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.648+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c043640 con 0x7f367c0c9870 2024-01-16T15:04:04.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.648+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c0cad60 con 0x7f367c0c9870 2024-01-16T15:04:04.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.648+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c0b0ec0 con 0x7f367c0c9870 2024-01-16T15:04:04.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.649+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c0cb280 con 0x7f367c0c9870 2024-01-16T15:04:04.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.650+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f366c0437a0 con 0x7f367c0c9870 2024-01-16T15:04:04.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.651+0000 7f3673e00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365406b590 0x7f365406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:04.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.651+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f366c03e070 con 0x7f367c0c9870 2024-01-16T15:04:04.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.652+0000 7f367b400640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365406b590 0x7f365406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:04.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.653+0000 7f367b400640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365406b590 0x7f365406da50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f367c06cfa0 tx=0x7f3664009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:04.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.656+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f366c077bc0 con 0x7f367c0c9870 2024-01-16T15:04:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.850+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f367c0675d0 con 0x7f367c0c9870 2024-01-16T15:04:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.858+0000 7f3673e00640 1 -- 172.21.15.53:0/2675893456 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v15)=0 v15) v1 ==== 131+0+0 (secure 0 0 0) 0x7f366c0340c0 con 0x7f367c0c9870 2024-01-16T15:04:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365406b590 msgr2=0x7f365406da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365406b590 0x7f365406da50 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f367c06cfa0 tx=0x7f3664009040 comp rx=0 tx=0).stop 2024-01-16T15:04:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 msgr2=0x7f367c0b0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 0x7f367c0b0980 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f366c000c00 tx=0x7f366c004730 comp rx=0 tx=0).stop 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 shutdown_connections 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365406b590 0x7f365406da50 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f367c161740 0x7f367c1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f367c0d0560 0x7f367c06b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 --2- 172.21.15.53:0/2675893456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f367c0c9870 0x7f367c0b0980 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 >> 172.21.15.53:0/2675893456 conn(0x7f367c09c560 msgr2=0x7f367c0d3030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 shutdown_connections 2024-01-16T15:04:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:04.861+0000 7f3682c00640 1 -- 172.21.15.53:0/2675893456 wait complete. 2024-01-16T15:04:04.860 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:04:05.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.100+0000 7fd88c200640 1 Processor -- start 2024-01-16T15:04:05.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.100+0000 7fd88c200640 1 -- start start 2024-01-16T15:04:05.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.102+0000 7fd88c200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd88415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.102+0000 7fd88c200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd8840cf290 0x7fd8840cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.102+0000 7fd88c200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd884070c40 con 0x7fd8840b3ae0 2024-01-16T15:04:05.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.102+0000 7fd88c200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd884071450 con 0x7fd8840cf290 2024-01-16T15:04:05.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.102+0000 7fd88c200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8840cfc60 con 0x7fd8840ae5a0 2024-01-16T15:04:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd8840cf290 0x7fd8840cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd8840cf290 0x7fd8840cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51666/0 (socket says 172.21.15.53:51666) 2024-01-16T15:04:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 -- 172.21.15.53:0/1875609337 learned_addr learned my addr 172.21.15.53:0/1875609337 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd889800640 1 --2- 172.21.15.53:0/1875609337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd88415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 -- 172.21.15.53:0/1875609337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 msgr2=0x7fd88415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:05.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 --2- 172.21.15.53:0/1875609337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd88415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 -- 172.21.15.53:0/1875609337 >> v1:172.21.15.146:6789/0 conn(0x7fd8840b3ae0 legacy=0x7fd88406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:05.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.103+0000 7fd88a200640 1 -- 172.21.15.53:0/1875609337 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd88406c470 con 0x7fd8840cf290 2024-01-16T15:04:05.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.105+0000 7fd88a200640 1 --2- 172.21.15.53:0/1875609337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840cf290 0x7fd8840cf720 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009e50 tx=0x7fd87c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55be523dad1ade79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.105+0000 7fd883e00640 1 -- 172.21.15.53:0/1875609337 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c02cb00 con 0x7fd8840cf290 2024-01-16T15:04:05.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.105+0000 7fd883e00640 1 -- 172.21.15.53:0/1875609337 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c02cc60 con 0x7fd8840cf290 2024-01-16T15:04:05.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.105+0000 7fd883e00640 1 -- 172.21.15.53:0/1875609337 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c0343f0 con 0x7fd8840cf290 2024-01-16T15:04:05.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 -- 172.21.15.53:0/1875609337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840cf290 msgr2=0x7fd8840cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 --2- 172.21.15.53:0/1875609337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840cf290 0x7fd8840cf720 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009e50 tx=0x7fd87c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 -- 172.21.15.53:0/1875609337 shutdown_connections 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 --2- 172.21.15.53:0/1875609337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840cf290 0x7fd8840cf720 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 --2- 172.21.15.53:0/1875609337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd88415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 -- 172.21.15.53:0/1875609337 >> 172.21.15.53:0/1875609337 conn(0x7fd88406a800 msgr2=0x7fd88406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 -- 172.21.15.53:0/1875609337 shutdown_connections 2024-01-16T15:04:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.106+0000 7fd88c200640 1 -- 172.21.15.53:0/1875609337 wait complete. 2024-01-16T15:04:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.108+0000 7fd88c200640 1 Processor -- start 2024-01-16T15:04:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.108+0000 7fd88c200640 1 -- start start 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88c200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd8841f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88c200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 0x7fd8841f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88c200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8840cf290 0x7fd8841512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88c200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd884158cd0 con 0x7fd8840b3ae0 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88c200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd884150230 con 0x7fd8840cf290 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88c200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8841504f0 con 0x7fd8840ae5a0 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd888e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 0x7fd8841f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd888e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 0x7fd8841f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51680/0 (socket says 172.21.15.53:51680) 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd888e00640 1 -- 172.21.15.53:0/3951035838 learned_addr learned my addr 172.21.15.53:0/3951035838 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd88a200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8840cf290 0x7fd8841512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.109+0000 7fd889800640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd8841f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd888e00640 1 -- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 msgr2=0x7fd8841f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd888e00640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd8841f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd888e00640 1 -- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8840cf290 msgr2=0x7fd8841512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd888e00640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8840cf290 0x7fd8841512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd888e00640 1 -- 172.21.15.53:0/3951035838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87c009b30 con 0x7fd8840b3ae0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd888e00640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 0x7fd8841f2660 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c430 tx=0x7fd86c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c011790 con 0x7fd8840b3ae0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd86c0118f0 con 0x7fd8840b3ae0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd86c011ce0 con 0x7fd8840b3ae0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8841507d0 con 0x7fd8840b3ae0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.110+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd884150d90 con 0x7fd8840b3ae0 2024-01-16T15:04:05.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.112+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8840675d0 con 0x7fd8840b3ae0 2024-01-16T15:04:05.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.113+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fd86c01c460 con 0x7fd8840b3ae0 2024-01-16T15:04:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.114+0000 7fd881600640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85c06b590 0x7fd85c06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.114+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd86c08a060 con 0x7fd8840b3ae0 2024-01-16T15:04:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.115+0000 7fd889800640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85c06b590 0x7fd85c06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.116+0000 7fd889800640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85c06b590 0x7fd85c06da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd874006fd0 tx=0x7fd874008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.119+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd86c0bce90 con 0x7fd8840b3ae0 2024-01-16T15:04:05.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.321+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fd884150f50 con 0x7fd8840b3ae0 2024-01-16T15:04:05.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.327+0000 7fd881600640 1 -- 172.21.15.53:0/3951035838 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v16)=0 v16) v1 ==== 135+0+0 (secure 0 0 0) 0x7fd86c054b00 con 0x7fd8840b3ae0 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85c06b590 msgr2=0x7fd85c06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85c06b590 0x7fd85c06da50 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd874006fd0 tx=0x7fd874008040 comp rx=0 tx=0).stop 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 msgr2=0x7fd8841f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 0x7fd8841f2660 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00c430 tx=0x7fd86c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 shutdown_connections 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85c06b590 0x7fd85c06da50 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd8840cf290 0x7fd8841512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd8840b3ae0 0x7fd8841f2660 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 --2- 172.21.15.53:0/3951035838 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd8840ae5a0 0x7fd8841f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 >> 172.21.15.53:0/3951035838 conn(0x7fd88406a800 msgr2=0x7fd8840b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 shutdown_connections 2024-01-16T15:04:05.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.330+0000 7fd88c200640 1 -- 172.21.15.53:0/3951035838 wait complete. 2024-01-16T15:04:05.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:04:05.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.569+0000 7ff79ca00640 1 Processor -- start 2024-01-16T15:04:05.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.570+0000 7ff79ca00640 1 -- start start 2024-01-16T15:04:05.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.571+0000 7ff79ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071f00 0x7ff798146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.571+0000 7ff79ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 0x7ff798166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.571+0000 7ff79ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980b2e10 con 0x7ff798071410 2024-01-16T15:04:05.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.571+0000 7ff79ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff79806cec0 con 0x7ff79806c7c0 2024-01-16T15:04:05.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.571+0000 7ff79ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff79806d1a0 con 0x7ff798071f00 2024-01-16T15:04:05.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.572+0000 7ff797200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 0x7ff798166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.572+0000 7ff78fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071f00 0x7ff798146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.572+0000 7ff797200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 0x7ff798166cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:56764/0 (socket says 172.21.15.53:56764) 2024-01-16T15:04:05.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.572+0000 7ff797200640 1 -- 172.21.15.53:0/245131482 learned_addr learned my addr 172.21.15.53:0/245131482 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:05.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.573+0000 7ff78fe00640 1 -- 172.21.15.53:0/245131482 >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 msgr2=0x7ff798166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:05.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.573+0000 7ff78fe00640 1 --2- 172.21.15.53:0/245131482 >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 0x7ff798166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.573+0000 7ff78fe00640 1 -- 172.21.15.53:0/245131482 >> v1:172.21.15.146:6789/0 conn(0x7ff798071410 legacy=0x7ff7980717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.573+0000 7ff78fe00640 1 -- 172.21.15.53:0/245131482 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79806d480 con 0x7ff798071f00 2024-01-16T15:04:05.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.573+0000 7ff797200640 1 --2- 172.21.15.53:0/245131482 >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 0x7ff798166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:05.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.574+0000 7ff78fe00640 1 --2- 172.21.15.53:0/245131482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071f00 0x7ff798146770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7800097a0 tx=0x7ff78002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d8fa20a8b72e558 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.575+0000 7ff795e00640 1 -- 172.21.15.53:0/245131482 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78002d040 con 0x7ff798071f00 2024-01-16T15:04:05.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.575+0000 7ff795e00640 1 -- 172.21.15.53:0/245131482 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff780031810 con 0x7ff798071f00 2024-01-16T15:04:05.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.575+0000 7ff795e00640 1 -- 172.21.15.53:0/245131482 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff780031ba0 con 0x7ff798071f00 2024-01-16T15:04:05.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.575+0000 7ff79ca00640 1 -- 172.21.15.53:0/245131482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071f00 msgr2=0x7ff798146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 --2- 172.21.15.53:0/245131482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071f00 0x7ff798146770 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff7800097a0 tx=0x7ff78002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 -- 172.21.15.53:0/245131482 shutdown_connections 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 --2- 172.21.15.53:0/245131482 >> v2:172.21.15.146:3300/0 conn(0x7ff79806c7c0 0x7ff798166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 --2- 172.21.15.53:0/245131482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071f00 0x7ff798146770 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 -- 172.21.15.53:0/245131482 >> 172.21.15.53:0/245131482 conn(0x7ff79806a800 msgr2=0x7ff79806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 -- 172.21.15.53:0/245131482 shutdown_connections 2024-01-16T15:04:05.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.576+0000 7ff79ca00640 1 -- 172.21.15.53:0/245131482 wait complete. 2024-01-16T15:04:05.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.577+0000 7ff79ca00640 1 Processor -- start 2024-01-16T15:04:05.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.578+0000 7ff79ca00640 1 -- start start 2024-01-16T15:04:05.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff79ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff79806c7c0 0x7ff798144a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff79ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071410 0x7ff79815bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff79ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 0x7ff79813e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff79ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7981459f0 con 0x7ff798071f00 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff79ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff798145150 con 0x7ff79806c7c0 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff79ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff798145430 con 0x7ff798071410 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff797200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 0x7ff79813e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff797200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 0x7ff79813e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51692/0 (socket says 172.21.15.53:51692) 2024-01-16T15:04:05.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff797200640 1 -- 172.21.15.53:0/4038778450 learned_addr learned my addr 172.21.15.53:0/4038778450 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:05.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff78fe00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071410 0x7ff79815bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff796800640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff79806c7c0 0x7ff798144a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.579+0000 7ff797200640 1 -- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071410 msgr2=0x7ff79815bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff797200640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071410 0x7ff79815bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff797200640 1 -- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff79806c7c0 msgr2=0x7ff798144a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff797200640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff79806c7c0 0x7ff798144a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff797200640 1 -- 172.21.15.53:0/4038778450 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff790009b30 con 0x7ff798071f00 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff78fe00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071410 0x7ff79815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff797200640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 0x7ff79813e9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff79000efc0 tx=0x7ff79000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff790018070 con 0x7ff798071f00 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff79000f040 con 0x7ff798071f00 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff780009400 con 0x7ff798071f00 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79000ce00 con 0x7ff798071f00 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.580+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79813cb50 con 0x7ff798071f00 2024-01-16T15:04:05.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.582+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff79813c9b0 con 0x7ff798071f00 2024-01-16T15:04:05.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.583+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7ff790014480 con 0x7ff798071f00 2024-01-16T15:04:05.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.583+0000 7ff78ea00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff77006b590 0x7ff77006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:05.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.583+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff79008a460 con 0x7ff798071f00 2024-01-16T15:04:05.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.584+0000 7ff796800640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff77006b590 0x7ff77006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:05.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.585+0000 7ff796800640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff77006b590 0x7ff77006da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f60 tx=0x7ff788009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.589+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff79813c9b0 con 0x7ff798071f00 2024-01-16T15:04:05.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.814+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7ff79806cbe0 con 0x7ff77006b590 2024-01-16T15:04:05.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.855+0000 7ff78ea00640 1 -- 172.21.15.53:0/4038778450 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff788001c70 con 0x7ff77006b590 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff77006b590 msgr2=0x7ff77006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff77006b590 0x7ff77006da50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ff788000f60 tx=0x7ff788009040 comp rx=0 tx=0).stop 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 msgr2=0x7ff79813e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 0x7ff79813e9c0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7ff79000efc0 tx=0x7ff79000c490 comp rx=0 tx=0).stop 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 shutdown_connections 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff77006b590 0x7ff77006da50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff798071f00 0x7ff79813e9c0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff798071410 0x7ff79815bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 --2- 172.21.15.53:0/4038778450 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff79806c7c0 0x7ff798144a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:05.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 >> 172.21.15.53:0/4038778450 conn(0x7ff79806a800 msgr2=0x7ff7980b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:05.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 shutdown_connections 2024-01-16T15:04:05.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:05.858+0000 7ff79ca00640 1 -- 172.21.15.53:0/4038778450 wait complete. 2024-01-16T15:04:05.857 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:04:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.099+0000 7fe7e9200640 1 Processor -- start 2024-01-16T15:04:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.099+0000 7fe7e9200640 1 -- start start 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7e9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7e40b3ae0 0x7fe7e415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7e9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e4070c40 con 0x7fe7e40cf290 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7e9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e406c470 con 0x7fe7e40ae5a0 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7e9200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e4071450 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7d9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7e40b3ae0 0x7fe7e415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7e2800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe7e40cf290 0x7fe7e40b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47460/0 (socket says 172.21.15.53:47460) 2024-01-16T15:04:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.101+0000 7fe7e2800640 1 -- 172.21.15.53:0/3070779336 learned_addr learned my addr 172.21.15.53:0/3070779336 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.102+0000 7fe7d9200640 1 -- 172.21.15.53:0/3070779336 >> v1:172.21.15.146:6789/0 conn(0x7fe7e40ae5a0 legacy=0x7fe7e406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.102+0000 7fe7d9200640 1 -- 172.21.15.53:0/3070779336 >> v1:172.21.15.53:6789/0 conn(0x7fe7e40cf290 legacy=0x7fe7e40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.102+0000 7fe7d9200640 1 -- 172.21.15.53:0/3070779336 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e406c750 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.103+0000 7fe7d9200640 1 --2- 172.21.15.53:0/3070779336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e415e430 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0097a0 tx=0x7fe7cc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6759e54a6052d7a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.103+0000 7fe7e1400640 1 -- 172.21.15.53:0/3070779336 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc030610 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.103+0000 7fe7e1400640 1 -- 172.21.15.53:0/3070779336 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc030770 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.103+0000 7fe7e1400640 1 -- 172.21.15.53:0/3070779336 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc030b00 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 -- 172.21.15.53:0/3070779336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 msgr2=0x7fe7e415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 --2- 172.21.15.53:0/3070779336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e415e430 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0097a0 tx=0x7fe7cc02b960 comp rx=0 tx=0).stop 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 -- 172.21.15.53:0/3070779336 shutdown_connections 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 --2- 172.21.15.53:0/3070779336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e415e430 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 -- 172.21.15.53:0/3070779336 >> 172.21.15.53:0/3070779336 conn(0x7fe7e406a800 msgr2=0x7fe7e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 -- 172.21.15.53:0/3070779336 shutdown_connections 2024-01-16T15:04:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.104+0000 7fe7e9200640 1 -- 172.21.15.53:0/3070779336 wait complete. 2024-01-16T15:04:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.106+0000 7fe7e9200640 1 Processor -- start 2024-01-16T15:04:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.106+0000 7fe7e9200640 1 -- start start 2024-01-16T15:04:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40ae5a0 0x7fe7e40acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e40cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40cf290 0x7fe7e40cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7e9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40ccfe0 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7e9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40cc360 con 0x7fe7e40cf290 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7e9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40cc640 con 0x7fe7e40ae5a0 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7d9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e40cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7d9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e40cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51714/0 (socket says 172.21.15.53:51714) 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.107+0000 7fe7d9200640 1 -- 172.21.15.53:0/1027452904 learned_addr learned my addr 172.21.15.53:0/1027452904 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7e1e00640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40ae5a0 0x7fe7e40acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7e2800640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40cf290 0x7fe7e40cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7d9200640 1 -- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40ae5a0 msgr2=0x7fe7e40acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7d9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40ae5a0 0x7fe7e40acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7d9200640 1 -- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40cf290 msgr2=0x7fe7e40cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7d9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40cf290 0x7fe7e40cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7d9200640 1 -- 172.21.15.53:0/1027452904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7cc009400 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7e2800640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40cf290 0x7fe7e40cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7d9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e40cd400 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098c0 tx=0x7fe7cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.108+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc03e040 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.109+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc039cb0 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.109+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc044630 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.109+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e40cc920 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.109+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e40c40e0 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.110+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e40cd120 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.111+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fe7cc043390 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.112+0000 7fe7daa00640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b590 0x7fe7bc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:06.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.112+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7cc0ac4a0 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.113+0000 7fe7e1e00640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b590 0x7fe7bc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:06.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.114+0000 7fe7e1e00640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b590 0x7fe7bc06da50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4004ea0 tx=0x7fe7d4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:06.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.116+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7e40cd120 con 0x7fe7e40b3ae0 2024-01-16T15:04:06.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.344+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7e4070550 con 0x7fe7bc06b590 2024-01-16T15:04:06.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.357+0000 7fe7daa00640 1 -- 172.21.15.53:0/1027452904 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7fe7d4005420 con 0x7fe7bc06b590 2024-01-16T15:04:06.336 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b590 msgr2=0x7fe7bc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b590 0x7fe7bc06da50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4004ea0 tx=0x7fe7d4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 msgr2=0x7fe7e40cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e40cd400 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc0098c0 tx=0x7fe7cc004730 comp rx=0 tx=0).stop 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 shutdown_connections 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b590 0x7fe7bc06da50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7e40cf290 0x7fe7e40cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e40b3ae0 0x7fe7e40cd400 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 --2- 172.21.15.53:0/1027452904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40ae5a0 0x7fe7e40acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:06.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 >> 172.21.15.53:0/1027452904 conn(0x7fe7e406a800 msgr2=0x7fe7e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:06.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 shutdown_connections 2024-01-16T15:04:06.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:06.359+0000 7fe7e9200640 1 -- 172.21.15.53:0/1027452904 wait complete. 2024-01-16T15:04:06.359 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:06.359 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e) 2024-01-16T15:04:06.420 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e 2024-01-16T15:04:06.420 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:04:06.420 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:04:06.421 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:04:08.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:04:08.422 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e) 2024-01-16T15:04:08.492 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/e6066642-f447-4767-b221-31824a61958e 2024-01-16T15:04:08.492 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:04:08.492 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:04:08.493 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:04:08.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone48371 2024-01-16T15:04:08.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.747+0000 7f86cce00640 1 Processor -- start 2024-01-16T15:04:08.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.747+0000 7f86cce00640 1 -- start start 2024-01-16T15:04:08.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86cce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f86c813cfa0 0x7f86c815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86cce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c80cd530 0x7f86c813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86cce00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86c80cd9b0 0x7f86c813fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86cce00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c80cdec0 con 0x7f86c80cd9b0 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86cce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c80ce1a0 con 0x7f86c813cfa0 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86cce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c80ce480 con 0x7f86c80cd530 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.749+0000 7f86c7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86c80cd9b0 0x7f86c813fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f86c80cd9b0 0x7f86c813fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51722/0 (socket says 172.21.15.53:51722) 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 -- 172.21.15.53:0/3895556232 learned_addr learned my addr 172.21.15.53:0/3895556232 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86be800640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c80cd530 0x7f86c813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7400640 1 --2- 172.21.15.53:0/3895556232 >> v2:172.21.15.146:3300/0 conn(0x7f86c813cfa0 0x7f86c815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 -- 172.21.15.53:0/3895556232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c80cd530 msgr2=0x7f86c813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c80cd530 0x7f86c813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 -- 172.21.15.53:0/3895556232 >> v2:172.21.15.146:3300/0 conn(0x7f86c813cfa0 msgr2=0x7f86c815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 --2- 172.21.15.53:0/3895556232 >> v2:172.21.15.146:3300/0 conn(0x7f86c813cfa0 0x7f86c815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7e00640 1 -- 172.21.15.53:0/3895556232 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c80ce760 con 0x7f86c80cd9b0 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86be800640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c80cd530 0x7f86c813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.750+0000 7f86c7400640 1 --2- 172.21.15.53:0/3895556232 >> v2:172.21.15.146:3300/0 conn(0x7f86c813cfa0 0x7f86c815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:08.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.751+0000 7f86c7e00640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd9b0 0x7f86c813fae0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f86c806cf00 tx=0x7f86c002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=534b3d9714456eaf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.752+0000 7f86c6a00640 1 -- 172.21.15.53:0/3895556232 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c002c9e0 con 0x7f86c80cd9b0 2024-01-16T15:04:08.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.752+0000 7f86c6a00640 1 -- 172.21.15.53:0/3895556232 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86c002cb40 con 0x7f86c80cd9b0 2024-01-16T15:04:08.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.752+0000 7f86c6a00640 1 -- 172.21.15.53:0/3895556232 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86c00343f0 con 0x7f86c80cd9b0 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.752+0000 7f86cce00640 1 -- 172.21.15.53:0/3895556232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd9b0 msgr2=0x7f86c813fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.752+0000 7f86cce00640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd9b0 0x7f86c813fae0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f86c806cf00 tx=0x7f86c002be40 comp rx=0 tx=0).stop 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 -- 172.21.15.53:0/3895556232 shutdown_connections 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd9b0 0x7f86c813fae0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 --2- 172.21.15.53:0/3895556232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c80cd530 0x7f86c813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 --2- 172.21.15.53:0/3895556232 >> v2:172.21.15.146:3300/0 conn(0x7f86c813cfa0 0x7f86c815afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 -- 172.21.15.53:0/3895556232 >> 172.21.15.53:0/3895556232 conn(0x7f86c806a800 msgr2=0x7f86c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 -- 172.21.15.53:0/3895556232 shutdown_connections 2024-01-16T15:04:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.753+0000 7f86cce00640 1 -- 172.21.15.53:0/3895556232 wait complete. 2024-01-16T15:04:08.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.755+0000 7f86cce00640 1 Processor -- start 2024-01-16T15:04:08.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.755+0000 7f86cce00640 1 -- start start 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86cce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 0x7f86c815b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86cce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86c80cd9b0 0x7f86c815bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86cce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c813cfa0 0x7f86c81e7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86cce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c814f4f0 con 0x7f86c80cd530 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86cce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c814f740 con 0x7f86c80cd9b0 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86cce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c814fa20 con 0x7f86c813cfa0 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 0x7f86c815b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 0x7f86c815b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51736/0 (socket says 172.21.15.53:51736) 2024-01-16T15:04:08.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86c7400640 1 -- 172.21.15.53:0/1593359310 learned_addr learned my addr 172.21.15.53:0/1593359310 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86be800640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86c80cd9b0 0x7f86c815bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86c7e00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c813cfa0 0x7f86c81e7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86c7400640 1 -- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c813cfa0 msgr2=0x7f86c81e7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.756+0000 7f86c7400640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c813cfa0 0x7f86c81e7540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86c7400640 1 -- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86c80cd9b0 msgr2=0x7f86c815bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86c7400640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86c80cd9b0 0x7f86c815bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86c7400640 1 -- 172.21.15.53:0/1593359310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c0009b30 con 0x7f86c80cd530 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86be800640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86c80cd9b0 0x7f86c815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86c7400640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 0x7f86c815b5e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f86b800d7d0 tx=0x7f86b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8004030 con 0x7f86c80cd530 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b8004190 con 0x7f86c80cd530 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b8004520 con 0x7f86c80cd530 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c814fd00 con 0x7f86c80cd530 2024-01-16T15:04:08.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.757+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c81502c0 con 0x7f86c80cd530 2024-01-16T15:04:08.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.758+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c8067450 con 0x7f86c80cd530 2024-01-16T15:04:08.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.759+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f86b800b6c0 con 0x7f86c80cd530 2024-01-16T15:04:08.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.760+0000 7f86bfe00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f86a006b590 0x7f86a006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:08.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.760+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f86b808a160 con 0x7f86c80cd530 2024-01-16T15:04:08.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.761+0000 7f86be800640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f86a006b590 0x7f86a006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.762+0000 7f86be800640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f86a006b590 0x7f86a006da50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f86b0006fd0 tx=0x7f86b0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:08.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.765+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f86b8054c90 con 0x7f86c80cd530 2024-01-16T15:04:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:08.992+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f86c8155640 con 0x7f86a006b590 2024-01-16T15:04:09.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.150+0000 7f86bfe00640 1 -- 172.21.15.53:0/1593359310 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f86c8155640 con 0x7f86a006b590 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f86a006b590 msgr2=0x7f86a006da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f86a006b590 0x7f86a006da50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f86b0006fd0 tx=0x7f86b0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 msgr2=0x7f86c815b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 0x7f86c815b5e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f86b800d7d0 tx=0x7f86b800dc90 comp rx=0 tx=0).stop 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 shutdown_connections 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f86a006b590 0x7f86a006da50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f86c813cfa0 0x7f86c81e7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f86c80cd9b0 0x7f86c815bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 --2- 172.21.15.53:0/1593359310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f86c80cd530 0x7f86c815b5e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 >> 172.21.15.53:0/1593359310 conn(0x7f86c806a800 msgr2=0x7f86c806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:09.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 shutdown_connections 2024-01-16T15:04:09.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.154+0000 7f86cce00640 1 -- 172.21.15.53:0/1593359310 wait complete. 2024-01-16T15:04:09.137 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.159+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.137 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.159+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.137 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.159+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.137 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.159+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.137 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.159+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone48372 2024-01-16T15:04:09.167 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.188+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.167 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.188+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.167 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.188+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.167 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.188+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.167 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:09.188+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:09.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.396+0000 7fb1dae00640 1 Processor -- start 2024-01-16T15:04:09.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.396+0000 7fb1dae00640 1 -- start start 2024-01-16T15:04:09.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.397+0000 7fb1dae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb1d4071410 0x7fb1d4146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:09.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.397+0000 7fb1dae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1dae00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1d40b2e10 con 0x7fb1d406c7c0 2024-01-16T15:04:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1dae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1d406cc50 con 0x7fb1d4071410 2024-01-16T15:04:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1dae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d406cec0 con 0x7fb1d4071f00 2024-01-16T15:04:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1d9a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb1d4071410 0x7fb1d4146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1d9000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb1d406c7c0 0x7fb1d40717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47470/0 (socket says 172.21.15.53:47470) 2024-01-16T15:04:09.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1d9000640 1 -- 172.21.15.53:0/2046005962 learned_addr learned my addr 172.21.15.53:0/2046005962 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1da400640 1 --2- 172.21.15.53:0/2046005962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1d9a00640 1 -- 172.21.15.53:0/2046005962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 msgr2=0x7fb1d4166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1d9a00640 1 --2- 172.21.15.53:0/2046005962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4166cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.398+0000 7fb1d9a00640 1 -- 172.21.15.53:0/2046005962 >> v1:172.21.15.53:6789/0 conn(0x7fb1d406c7c0 legacy=0x7fb1d40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.399+0000 7fb1d9a00640 1 -- 172.21.15.53:0/2046005962 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d406d1a0 con 0x7fb1d4071410 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.399+0000 7fb1d3e00640 1 -- 172.21.15.53:0/2046005962 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661741014 0 0) 0x7fb1d40b2e10 con 0x7fb1d406c7c0 2024-01-16T15:04:09.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.400+0000 7fb1d9a00640 1 --2- 172.21.15.53:0/2046005962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d4071410 0x7fb1d4146770 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009d90 tx=0x7fb1c402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a22e3c4f8a3cd107 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:09.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.400+0000 7fb1d3e00640 1 -- 172.21.15.53:0/2046005962 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c402cc50 con 0x7fb1d4071410 2024-01-16T15:04:09.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.400+0000 7fb1d3e00640 1 -- 172.21.15.53:0/2046005962 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c402cdb0 con 0x7fb1d4071410 2024-01-16T15:04:09.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.400+0000 7fb1d3e00640 1 -- 172.21.15.53:0/2046005962 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4034660 con 0x7fb1d4071410 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 -- 172.21.15.53:0/2046005962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d4071410 msgr2=0x7fb1d4146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 --2- 172.21.15.53:0/2046005962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d4071410 0x7fb1d4146770 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009d90 tx=0x7fb1c402bea0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 -- 172.21.15.53:0/2046005962 shutdown_connections 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 --2- 172.21.15.53:0/2046005962 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4166cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 --2- 172.21.15.53:0/2046005962 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d4071410 0x7fb1d4146770 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 -- 172.21.15.53:0/2046005962 >> 172.21.15.53:0/2046005962 conn(0x7fb1d406a800 msgr2=0x7fb1d406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 -- 172.21.15.53:0/2046005962 shutdown_connections 2024-01-16T15:04:09.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.401+0000 7fb1dae00640 1 -- 172.21.15.53:0/2046005962 wait complete. 2024-01-16T15:04:09.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.403+0000 7fb1dae00640 1 Processor -- start 2024-01-16T15:04:09.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.403+0000 7fb1dae00640 1 -- start start 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1dae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 0x7fb1d41302d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1dae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1d4071410 0x7fb1d41f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1dae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4141ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1dae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4132300 con 0x7fb1d406c7c0 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1dae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d41325b0 con 0x7fb1d4071410 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1dae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d4132890 con 0x7fb1d4071f00 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1d9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 0x7fb1d41302d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1d9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 0x7fb1d41302d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51744/0 (socket says 172.21.15.53:51744) 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1d9a00640 1 -- 172.21.15.53:0/3904791779 learned_addr learned my addr 172.21.15.53:0/3904791779 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1d9000640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1d4071410 0x7fb1d41f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1d9a00640 1 -- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 msgr2=0x7fb1d4141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1da400640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4141ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.404+0000 7fb1d9a00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4141ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d9a00640 1 -- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1d4071410 msgr2=0x7fb1d41f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d9a00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1d4071410 0x7fb1d41f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d9a00640 1 -- 172.21.15.53:0/3904791779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1c4009a70 con 0x7fb1d406c7c0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d9000640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1d4071410 0x7fb1d41f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d9a00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 0x7fb1d41302d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009d90 tx=0x7fb1c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4040650 con 0x7fb1d406c7c0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1c40407b0 con 0x7fb1d406c7c0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d4132b70 con 0x7fb1d406c7c0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4040b60 con 0x7fb1d406c7c0 2024-01-16T15:04:09.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.405+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d41f2830 con 0x7fb1d406c7c0 2024-01-16T15:04:09.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.406+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d4133090 con 0x7fb1d406c7c0 2024-01-16T15:04:09.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.408+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7fb1c4040cc0 con 0x7fb1d406c7c0 2024-01-16T15:04:09.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.409+0000 7fb1d1600640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1ac06b590 0x7fb1ac06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:09.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.409+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb1c40acf00 con 0x7fb1d406c7c0 2024-01-16T15:04:09.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.410+0000 7fb1d9000640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1ac06b590 0x7fb1ac06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:09.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.411+0000 7fb1d9000640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1ac06b590 0x7fb1ac06da50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4072470 tx=0x7fb1bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:09.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.413+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1d4133090 con 0x7fb1d406c7c0 2024-01-16T15:04:09.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.643+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone48372", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1d406da40 con 0x7fb1ac06b590 2024-01-16T15:04:09.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.804+0000 7fb1d1600640 1 -- 172.21.15.53:0/3904791779 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb1bc001d50 con 0x7fb1ac06b590 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1ac06b590 msgr2=0x7fb1ac06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1ac06b590 0x7fb1ac06da50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb1d4072470 tx=0x7fb1bc009040 comp rx=0 tx=0).stop 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 msgr2=0x7fb1d41302d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 0x7fb1d41302d0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4009d90 tx=0x7fb1c4004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 shutdown_connections 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1ac06b590 0x7fb1ac06da50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1d4071f00 0x7fb1d4141ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1d4071410 0x7fb1d41f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 --2- 172.21.15.53:0/3904791779 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1d406c7c0 0x7fb1d41302d0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:09.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.808+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 >> 172.21.15.53:0/3904791779 conn(0x7fb1d406a800 msgr2=0x7fb1d40b02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:09.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.809+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 shutdown_connections 2024-01-16T15:04:09.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:09.809+0000 7fb1dae00640 1 -- 172.21.15.53:0/3904791779 wait complete. 2024-01-16T15:04:09.812 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone48373 2024-01-16T15:04:10.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.060+0000 7f51ebc00640 1 Processor -- start 2024-01-16T15:04:10.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.060+0000 7f51ebc00640 1 -- start start 2024-01-16T15:04:10.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.062+0000 7f51ebc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e415e880 0x7f51e406ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.062+0000 7f51ebc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f51e4070490 0x7f51e4099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.062+0000 7f51ebc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51e4099fa0 0x7f51e40cbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.062+0000 7f51ebc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e409a410 con 0x7f51e4099fa0 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.062+0000 7f51ebc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e40cfa70 con 0x7f51e4070490 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.062+0000 7f51ebc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40cff00 con 0x7f51e415e880 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51e4099fa0 0x7f51e40cbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51e4099fa0 0x7f51e40cbb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51746/0 (socket says 172.21.15.53:51746) 2024-01-16T15:04:10.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 -- 172.21.15.53:0/1776214884 learned_addr learned my addr 172.21.15.53:0/1776214884 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:10.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51dbe00640 1 --2- 172.21.15.53:0/1776214884 >> v2:172.21.15.146:3300/0 conn(0x7f51e4070490 0x7f51e4099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9200640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e415e880 0x7f51e406ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 -- 172.21.15.53:0/1776214884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e415e880 msgr2=0x7f51e406ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e415e880 0x7f51e406ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 -- 172.21.15.53:0/1776214884 >> v2:172.21.15.146:3300/0 conn(0x7f51e4070490 msgr2=0x7f51e4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:10.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 --2- 172.21.15.53:0/1776214884 >> v2:172.21.15.146:3300/0 conn(0x7f51e4070490 0x7f51e4099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9c00640 1 -- 172.21.15.53:0/1776214884 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e406fc70 con 0x7f51e4099fa0 2024-01-16T15:04:10.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.063+0000 7f51e9200640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e415e880 0x7f51e406ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:10.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.065+0000 7f51e9c00640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40cbb20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f51e40749d0 tx=0x7f51dc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b670e4a481bb5cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.065+0000 7f51e3e00640 1 -- 172.21.15.53:0/1776214884 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc02cb00 con 0x7f51e4099fa0 2024-01-16T15:04:10.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.065+0000 7f51e3e00640 1 -- 172.21.15.53:0/1776214884 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51dc02cc60 con 0x7f51e4099fa0 2024-01-16T15:04:10.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.065+0000 7f51e3e00640 1 -- 172.21.15.53:0/1776214884 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51dc0343f0 con 0x7f51e4099fa0 2024-01-16T15:04:10.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 -- 172.21.15.53:0/1776214884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 msgr2=0x7f51e40cbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40cbb20 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f51e40749d0 tx=0x7f51dc02bf60 comp rx=0 tx=0).stop 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 -- 172.21.15.53:0/1776214884 shutdown_connections 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40cbb20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 --2- 172.21.15.53:0/1776214884 >> v2:172.21.15.146:3300/0 conn(0x7f51e4070490 0x7f51e4099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 --2- 172.21.15.53:0/1776214884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e415e880 0x7f51e406ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 -- 172.21.15.53:0/1776214884 >> 172.21.15.53:0/1776214884 conn(0x7f51e40b2d00 msgr2=0x7f51e40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 -- 172.21.15.53:0/1776214884 shutdown_connections 2024-01-16T15:04:10.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.066+0000 7f51ebc00640 1 -- 172.21.15.53:0/1776214884 wait complete. 2024-01-16T15:04:10.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.068+0000 7f51ebc00640 1 Processor -- start 2024-01-16T15:04:10.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.068+0000 7f51ebc00640 1 -- start start 2024-01-16T15:04:10.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.069+0000 7f51ebc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e4070490 0x7f51e40b69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.069+0000 7f51ebc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.069+0000 7f51ebc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51e415e880 0x7f51e40b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.069+0000 7f51ebc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40c92d0 con 0x7f51e4099fa0 2024-01-16T15:04:10.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.069+0000 7f51ebc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51e406ba00 con 0x7f51e415e880 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.069+0000 7f51ebc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51e40c8d10 con 0x7f51e4070490 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51760/0 (socket says 172.21.15.53:51760) 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 -- 172.21.15.53:0/730821519 learned_addr learned my addr 172.21.15.53:0/730821519 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51e9200640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e4070490 0x7f51e40b69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51e9c00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51e415e880 0x7f51e40b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 -- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e4070490 msgr2=0x7f51e40b69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e4070490 0x7f51e40b69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 -- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51e415e880 msgr2=0x7f51e40b0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51e415e880 0x7f51e40b0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.070+0000 7f51dbe00640 1 -- 172.21.15.53:0/730821519 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51dc009b30 con 0x7f51e4099fa0 2024-01-16T15:04:10.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.071+0000 7f51dbe00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40c82e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00d800 tx=0x7f51cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.071+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc0096a0 con 0x7f51e4099fa0 2024-01-16T15:04:10.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.071+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51cc009800 con 0x7f51e4099fa0 2024-01-16T15:04:10.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.071+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51cc009990 con 0x7f51e4099fa0 2024-01-16T15:04:10.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.071+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e40c8ff0 con 0x7f51e4099fa0 2024-01-16T15:04:10.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.071+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e40b6f20 con 0x7f51e4099fa0 2024-01-16T15:04:10.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.072+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e406b4f0 con 0x7f51e4099fa0 2024-01-16T15:04:10.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.073+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 10) v1 ==== 89794+0+0 (secure 0 0 0) 0x7f51cc009af0 con 0x7f51e4099fa0 2024-01-16T15:04:10.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.074+0000 7f51e1600640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51bc06b590 0x7f51bc06da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:10.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.074+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f51cc08a5a0 con 0x7f51e4099fa0 2024-01-16T15:04:10.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.075+0000 7f51e9200640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51bc06b590 0x7f51bc06da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:10.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.076+0000 7f51e9200640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51bc06b590 0x7f51bc06da50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f51d400a990 tx=0x7f51d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:10.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.079+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51e406b4f0 con 0x7f51e4099fa0 2024-01-16T15:04:10.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:10.310+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone48373", "target": ["mon-mgr", ""]}) v1 -- 0x7f51e40cba70 con 0x7f51bc06b590 2024-01-16T15:04:11.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:11.106+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f51cc055380 con 0x7f51e4099fa0 2024-01-16T15:04:12.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.532+0000 7f51e1600640 1 -- 172.21.15.53:0/730821519 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f51e40cba70 con 0x7f51bc06b590 2024-01-16T15:04:12.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.535+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51bc06b590 msgr2=0x7f51bc06da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:12.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.535+0000 7f51ebc00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51bc06b590 0x7f51bc06da50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f51d400a990 tx=0x7f51d4008040 comp rx=0 tx=0).stop 2024-01-16T15:04:12.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.535+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 msgr2=0x7f51e40c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.535+0000 7f51ebc00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40c82e0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f51cc00d800 tx=0x7f51cc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 shutdown_connections 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51bc06b590 0x7f51bc06da50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51e415e880 0x7f51e40b0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51e4099fa0 0x7f51e40c82e0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 --2- 172.21.15.53:0/730821519 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51e4070490 0x7f51e40b69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 >> 172.21.15.53:0/730821519 conn(0x7f51e40b2d00 msgr2=0x7f51e415c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 shutdown_connections 2024-01-16T15:04:12.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.536+0000 7f51ebc00640 1 -- 172.21.15.53:0/730821519 wait complete. 2024-01-16T15:04:12.539 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone48374 2024-01-16T15:04:12.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.776+0000 7f1183200640 1 Processor -- start 2024-01-16T15:04:12.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.776+0000 7f1183200640 1 -- start start 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1183200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1184143c40 0x7f1184071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1183200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f11840717d0 0x7f11840cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1183200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1184070e50 con 0x7f11840cca40 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1183200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11840723c0 con 0x7f1184143c40 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1183200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11840cce40 con 0x7f11840717d0 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1181e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1184143c40 0x7f1184071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1181e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1184143c40 0x7f1184071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39126/0 (socket says 172.21.15.53:39126) 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.778+0000 7f1181e00640 1 -- 172.21.15.53:0/902236241 learned_addr learned my addr 172.21.15.53:0/902236241 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.779+0000 7f1181400640 1 --2- 172.21.15.53:0/902236241 >> v2:172.21.15.146:3300/0 conn(0x7f11840717d0 0x7f11840cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:12.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.779+0000 7f1181e00640 1 -- 172.21.15.53:0/902236241 >> v2:172.21.15.146:3300/0 conn(0x7f11840717d0 msgr2=0x7f11840cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:12.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.779+0000 7f1181e00640 1 --2- 172.21.15.53:0/902236241 >> v2:172.21.15.146:3300/0 conn(0x7f11840717d0 0x7f11840cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.779+0000 7f1181e00640 1 -- 172.21.15.53:0/902236241 >> v1:172.21.15.146:6789/0 conn(0x7f11840cca40 legacy=0x7f118415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:12.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.779+0000 7f1181e00640 1 -- 172.21.15.53:0/902236241 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118406e170 con 0x7f1184143c40 2024-01-16T15:04:12.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.780+0000 7f1181e00640 1 --2- 172.21.15.53:0/902236241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1184143c40 0x7f1184071290 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f11840b44a0 tx=0x7f117402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=100396f7774a9c1f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.780+0000 7f1180a00640 1 -- 172.21.15.53:0/902236241 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11740317e0 con 0x7f1184143c40 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.780+0000 7f1180a00640 1 -- 172.21.15.53:0/902236241 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174031940 con 0x7f1184143c40 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.780+0000 7f1180a00640 1 -- 172.21.15.53:0/902236241 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1174031cd0 con 0x7f1184143c40 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 -- 172.21.15.53:0/902236241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1184143c40 msgr2=0x7f1184071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 --2- 172.21.15.53:0/902236241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1184143c40 0x7f1184071290 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f11840b44a0 tx=0x7f117402b880 comp rx=0 tx=0).stop 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 -- 172.21.15.53:0/902236241 shutdown_connections 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 --2- 172.21.15.53:0/902236241 >> v2:172.21.15.146:3300/0 conn(0x7f11840717d0 0x7f11840cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 --2- 172.21.15.53:0/902236241 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1184143c40 0x7f1184071290 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 -- 172.21.15.53:0/902236241 >> 172.21.15.53:0/902236241 conn(0x7f118406a800 msgr2=0x7f118406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 -- 172.21.15.53:0/902236241 shutdown_connections 2024-01-16T15:04:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.781+0000 7f1183200640 1 -- 172.21.15.53:0/902236241 wait complete. 2024-01-16T15:04:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.783+0000 7f1183200640 1 Processor -- start 2024-01-16T15:04:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.783+0000 7f1183200640 1 -- start start 2024-01-16T15:04:12.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1183200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 0x7f11841f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:12.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1183200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11840cca40 0x7f11840aad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:12.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1183200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1184143c40 0x7f11840afc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:12.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1183200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11840ac4a0 con 0x7f11840717d0 2024-01-16T15:04:12.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1183200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11840a93e0 con 0x7f11840cca40 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1183200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11840a9690 con 0x7f1184143c40 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.784+0000 7f1181e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 0x7f11841f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 0x7f11841f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39132/0 (socket says 172.21.15.53:39132) 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 -- 172.21.15.53:0/3911073085 learned_addr learned my addr 172.21.15.53:0/3911073085 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181400640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11840cca40 0x7f11840aad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1182800640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1184143c40 0x7f11840afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 -- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1184143c40 msgr2=0x7f11840afc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1184143c40 0x7f11840afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 -- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11840cca40 msgr2=0x7f11840aad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11840cca40 0x7f11840aad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 -- 172.21.15.53:0/3911073085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11740094d0 con 0x7f11840717d0 2024-01-16T15:04:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.785+0000 7f1181e00640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 0x7f11841f3e10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f117402b8c0 tx=0x7f11740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.786+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11740329c0 con 0x7f11840717d0 2024-01-16T15:04:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.786+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1174032b40 con 0x7f11840717d0 2024-01-16T15:04:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.786+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11740453f0 con 0x7f11840717d0 2024-01-16T15:04:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.786+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11840a9970 con 0x7f11840717d0 2024-01-16T15:04:12.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.786+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11841f4350 con 0x7f11840717d0 2024-01-16T15:04:12.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.787+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1184067450 con 0x7f11840717d0 2024-01-16T15:04:12.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.788+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f117404d3f0 con 0x7f11840717d0 2024-01-16T15:04:12.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.789+0000 7f117a000640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f115c06b6b0 0x7f115c06db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:12.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.789+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f11740ac2f0 con 0x7f11840717d0 2024-01-16T15:04:12.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.790+0000 7f1181400640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f115c06b6b0 0x7f115c06db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:12.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.791+0000 7f1181400640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f115c06b6b0 0x7f115c06db70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f118413cd00 tx=0x7f116c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:12.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:12.794+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1174076b70 con 0x7f11840717d0 2024-01-16T15:04:13.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:13.024+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone48374", "target": ["mon-mgr", ""]}) v1 -- 0x7f11840720e0 con 0x7f115c06b6b0 2024-01-16T15:04:17.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.530+0000 7f117a000640 1 -- 172.21.15.53:0/3911073085 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f116c00c050 con 0x7f115c06b6b0 2024-01-16T15:04:17.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.533+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f115c06b6b0 msgr2=0x7f115c06db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.533+0000 7f1183200640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f115c06b6b0 0x7f115c06db70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f118413cd00 tx=0x7f116c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:17.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.533+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 msgr2=0x7f11841f3e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.533+0000 7f1183200640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 0x7f11841f3e10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f117402b8c0 tx=0x7f11740046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 shutdown_connections 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f115c06b6b0 0x7f115c06db70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1184143c40 0x7f11840afc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f11840cca40 0x7f11840aad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 --2- 172.21.15.53:0/3911073085 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f11840717d0 0x7f11841f3e10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 >> 172.21.15.53:0/3911073085 conn(0x7f118406a800 msgr2=0x7f11841402e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 shutdown_connections 2024-01-16T15:04:17.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.534+0000 7f1183200640 1 -- 172.21.15.53:0/3911073085 wait complete. 2024-01-16T15:04:17.536 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:04:17.537 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:17.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.780+0000 7f9102a00640 1 Processor -- start 2024-01-16T15:04:17.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.780+0000 7f9102a00640 1 -- start start 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f9102a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f90fc0cf9a0 0x7f90fc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f9102a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f90fc15e1e0 0x7f90fc0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f9102a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90fc0b1ee0 con 0x7f90fc143ae0 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f9102a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc0b2780 con 0x7f90fc0cf9a0 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f9102a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90fc15e990 con 0x7f90fc15e1e0 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f90fb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f90fc0cf9a0 0x7f90fc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f90fbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f90fc143ae0 0x7f90fc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33194/0 (socket says 172.21.15.53:33194) 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f90fbe00640 1 -- 172.21.15.53:0/481690823 learned_addr learned my addr 172.21.15.53:0/481690823 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.782+0000 7f90fb400640 1 --2- 172.21.15.53:0/481690823 >> v2:172.21.15.53:3300/0 conn(0x7f90fc0cf9a0 0x7f90fc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39140/0 (socket says 172.21.15.53:39140) 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.783+0000 7f9100a00640 1 --2- 172.21.15.53:0/481690823 >> v2:172.21.15.146:3300/0 conn(0x7f90fc15e1e0 0x7f90fc0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.783+0000 7f90fb400640 1 -- 172.21.15.53:0/481690823 >> v2:172.21.15.146:3300/0 conn(0x7f90fc15e1e0 msgr2=0x7f90fc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.783+0000 7f90fb400640 1 --2- 172.21.15.53:0/481690823 >> v2:172.21.15.146:3300/0 conn(0x7f90fc15e1e0 0x7f90fc0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.783+0000 7f90fb400640 1 -- 172.21.15.53:0/481690823 >> v1:172.21.15.53:6789/0 conn(0x7f90fc143ae0 legacy=0x7f90fc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.783+0000 7f90fb400640 1 -- 172.21.15.53:0/481690823 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90fc0b2130 con 0x7f90fc0cf9a0 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.783+0000 7f9100a00640 1 --2- 172.21.15.53:0/481690823 >> v2:172.21.15.146:3300/0 conn(0x7f90fc15e1e0 0x7f90fc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:17.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.784+0000 7f90fb400640 1 --2- 172.21.15.53:0/481690823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc15dca0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0ae9d0 tx=0x7f90e402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bf422a0625ffc5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f90faa00640 1 -- 172.21.15.53:0/481690823 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e402ee70 con 0x7f90fc0cf9a0 2024-01-16T15:04:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f90faa00640 1 -- 172.21.15.53:0/481690823 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e4036440 con 0x7f90fc0cf9a0 2024-01-16T15:04:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f90faa00640 1 -- 172.21.15.53:0/481690823 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90e40367d0 con 0x7f90fc0cf9a0 2024-01-16T15:04:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 -- 172.21.15.53:0/481690823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc0cf9a0 msgr2=0x7f90fc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 --2- 172.21.15.53:0/481690823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc15dca0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f90fc0ae9d0 tx=0x7f90e402e0c0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 -- 172.21.15.53:0/481690823 shutdown_connections 2024-01-16T15:04:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 --2- 172.21.15.53:0/481690823 >> v2:172.21.15.146:3300/0 conn(0x7f90fc15e1e0 0x7f90fc0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 --2- 172.21.15.53:0/481690823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc15dca0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 -- 172.21.15.53:0/481690823 >> 172.21.15.53:0/481690823 conn(0x7f90fc06a800 msgr2=0x7f90fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.785+0000 7f9102a00640 1 -- 172.21.15.53:0/481690823 shutdown_connections 2024-01-16T15:04:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.786+0000 7f9102a00640 1 -- 172.21.15.53:0/481690823 wait complete. 2024-01-16T15:04:17.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.787+0000 7f9102a00640 1 Processor -- start 2024-01-16T15:04:17.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.787+0000 7f9102a00640 1 -- start start 2024-01-16T15:04:17.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.788+0000 7f9102a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc06c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.788+0000 7f9102a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90fc143ae0 0x7f90fc06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.788+0000 7f9102a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 0x7f90fc141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.788+0000 7f9102a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc130b20 con 0x7f90fc15e1e0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.788+0000 7f9102a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc130e00 con 0x7f90fc0cf9a0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.788+0000 7f9102a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90fc1310e0 con 0x7f90fc143ae0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 0x7f90fc141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 0x7f90fc141400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39142/0 (socket says 172.21.15.53:39142) 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 -- 172.21.15.53:0/926915804 learned_addr learned my addr 172.21.15.53:0/926915804 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f90fbe00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f90fb400640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90fc143ae0 0x7f90fc06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 -- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90fc143ae0 msgr2=0x7f90fc06c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90fc143ae0 0x7f90fc06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 -- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90fc0cf9a0 msgr2=0x7f90fc06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc06c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 -- 172.21.15.53:0/926915804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e400b6a0 con 0x7f90fc15e1e0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.789+0000 7f9100a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 0x7f90fc141400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f90f400cef0 tx=0x7f90f400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.790+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400d840 con 0x7f90fc15e1e0 2024-01-16T15:04:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.790+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f400d9c0 con 0x7f90fc15e1e0 2024-01-16T15:04:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.790+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90f400dd70 con 0x7f90fc15e1e0 2024-01-16T15:04:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.790+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90fc0ace80 con 0x7f90fc15e1e0 2024-01-16T15:04:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.790+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90fc12e2d0 con 0x7f90fc15e1e0 2024-01-16T15:04:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.791+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90fc1312a0 con 0x7f90fc15e1e0 2024-01-16T15:04:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.792+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f90f401e3b0 con 0x7f90fc15e1e0 2024-01-16T15:04:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.793+0000 7f90f3e00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90d406b730 0x7f90d406dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.793+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f90f408adf0 con 0x7f90fc15e1e0 2024-01-16T15:04:17.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.794+0000 7f90fbe00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90d406b730 0x7f90d406dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:17.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.795+0000 7f90fbe00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90d406b730 0x7f90d406dbf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f90ec005880 tx=0x7f90ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:17.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:17.798+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90f4055740 con 0x7f90fc15e1e0 2024-01-16T15:04:18.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:18.028+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f90fc15e670 con 0x7f90d406b730 2024-01-16T15:04:21.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:21.125 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.386+0000 7f90f3e00640 1 -- 172.21.15.53:0/926915804 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f90ec001340 con 0x7f90d406b730 2024-01-16T15:04:22.365 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:22.366 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90d406b730 msgr2=0x7f90d406dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90d406b730 0x7f90d406dbf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f90ec005880 tx=0x7f90ec009040 comp rx=0 tx=0).stop 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 msgr2=0x7f90fc141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 0x7f90fc141400 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f90f400cef0 tx=0x7f90f400a710 comp rx=0 tx=0).stop 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 shutdown_connections 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90d406b730 0x7f90d406dbf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f90fc15e1e0 0x7f90fc141400 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f90fc143ae0 0x7f90fc06c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 --2- 172.21.15.53:0/926915804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f90fc0cf9a0 0x7f90fc06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:22.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 >> 172.21.15.53:0/926915804 conn(0x7f90fc06a800 msgr2=0x7f90fc0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:22.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.390+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 shutdown_connections 2024-01-16T15:04:22.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:22.391+0000 7f9102a00640 1 -- 172.21.15.53:0/926915804 wait complete. 2024-01-16T15:04:22.392 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:04:23.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.653+0000 7ff201600640 1 Processor -- start 2024-01-16T15:04:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.653+0000 7ff201600640 1 -- start start 2024-01-16T15:04:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff201600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc0ccb50 0x7ff1fc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff201600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1fc15bf10 0x7ff1fc143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff201600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1fc15f4e0 con 0x7ff1fc0adf20 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff201600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1fc15f970 con 0x7ff1fc15bf10 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff201600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc15c3c0 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff200c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff1fc15bf10 0x7ff1fc143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff1fbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff1fc0adf20 0x7ff1fc0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37230/0 (socket says 172.21.15.53:37230) 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.655+0000 7ff1fbe00640 1 -- 172.21.15.53:0/4169546546 learned_addr learned my addr 172.21.15.53:0/4169546546 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.656+0000 7ff200c00640 1 -- 172.21.15.53:0/4169546546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc0ccb50 msgr2=0x7ff1fc070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.656+0000 7ff200c00640 1 --2- 172.21.15.53:0/4169546546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc0ccb50 0x7ff1fc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.656+0000 7ff200c00640 1 -- 172.21.15.53:0/4169546546 >> v1:172.21.15.53:6789/0 conn(0x7ff1fc0adf20 legacy=0x7ff1fc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:23.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.656+0000 7ff200c00640 1 -- 172.21.15.53:0/4169546546 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc071130 con 0x7ff1fc15bf10 2024-01-16T15:04:23.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.656+0000 7ff1fb400640 1 -- 172.21.15.53:0/4169546546 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860138396 0 0) 0x7ff1fc15f4e0 con 0x7ff1fc0adf20 2024-01-16T15:04:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.657+0000 7ff200c00640 1 --2- 172.21.15.53:0/4169546546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc15bf10 0x7ff1fc143d30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009d90 tx=0x7ff1f402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495f659b8693434f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.657+0000 7ff1fb400640 1 -- 172.21.15.53:0/4169546546 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f402cbc0 con 0x7ff1fc15bf10 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff1fb400640 1 -- 172.21.15.53:0/4169546546 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f402cd20 con 0x7ff1fc15bf10 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff1fb400640 1 -- 172.21.15.53:0/4169546546 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4034660 con 0x7ff1fc15bf10 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff201600640 1 -- 172.21.15.53:0/4169546546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc15bf10 msgr2=0x7ff1fc143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff201600640 1 --2- 172.21.15.53:0/4169546546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc15bf10 0x7ff1fc143d30 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4009d90 tx=0x7ff1f402be10 comp rx=0 tx=0).stop 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff201600640 1 -- 172.21.15.53:0/4169546546 shutdown_connections 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff201600640 1 --2- 172.21.15.53:0/4169546546 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc15bf10 0x7ff1fc143d30 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff201600640 1 --2- 172.21.15.53:0/4169546546 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc0ccb50 0x7ff1fc070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.658+0000 7ff201600640 1 -- 172.21.15.53:0/4169546546 >> 172.21.15.53:0/4169546546 conn(0x7ff1fc06a7d0 msgr2=0x7ff1fc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.659+0000 7ff201600640 1 -- 172.21.15.53:0/4169546546 shutdown_connections 2024-01-16T15:04:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.659+0000 7ff201600640 1 -- 172.21.15.53:0/4169546546 wait complete. 2024-01-16T15:04:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.661+0000 7ff201600640 1 Processor -- start 2024-01-16T15:04:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.661+0000 7ff201600640 1 -- start start 2024-01-16T15:04:23.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff201600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0adf20 0x7ff1fc1386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff201600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 0x7ff1fc0b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff201600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc15bf10 0x7ff1fc15a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff201600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc15ae50 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff201600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc135c10 con 0x7ff1fc0adf20 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff201600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc139010 con 0x7ff1fc15bf10 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff1f3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 0x7ff1fc0b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff1f3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 0x7ff1fc0b38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42394/0 (socket says 172.21.15.53:42394) 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.662+0000 7ff1f3e00640 1 -- 172.21.15.53:0/494466840 learned_addr learned my addr 172.21.15.53:0/494466840 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff200c00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc15bf10 0x7ff1fc15a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1fbe00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0adf20 0x7ff1fc1386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f3e00640 1 -- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc15bf10 msgr2=0x7ff1fc15a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f3e00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc15bf10 0x7ff1fc15a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f3e00640 1 -- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0adf20 msgr2=0x7ff1fc1386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f3e00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0adf20 0x7ff1fc1386f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f3e00640 1 -- 172.21.15.53:0/494466840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f4009a70 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1fbe00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0adf20 0x7ff1fc1386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f3e00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 0x7ff1fc0b38e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b450 tx=0x7ff1e400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4011890 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e40119f0 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e4011da0 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.663+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1fc134e50 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.664+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1fc135410 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.665+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1fc067450 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.666+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7ff1e401a400 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.667+0000 7ff1f8c00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1d406b6b0 0x7ff1d406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:23.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.667+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff1e408a700 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.667+0000 7ff1fbe00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1d406b6b0 0x7ff1d406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.669+0000 7ff1fbe00640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1d406b6b0 0x7ff1d406db70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec000f60 tx=0x7ff1ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.672+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1e4054f70 con 0x7ff1fc0ccb50 2024-01-16T15:04:23.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.910+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7ff1fc15f170 con 0x7ff1d406b6b0 2024-01-16T15:04:23.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.937+0000 7ff1f8c00640 1 -- 172.21.15.53:0/494466840 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7ff1ec001d50 con 0x7ff1d406b6b0 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:23.918 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:23.919 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:23.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1d406b6b0 msgr2=0x7ff1d406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:23.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1d406b6b0 0x7ff1d406db70 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec000f60 tx=0x7ff1ec009040 comp rx=0 tx=0).stop 2024-01-16T15:04:23.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 msgr2=0x7ff1fc0b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:23.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 0x7ff1fc0b38e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b450 tx=0x7ff1e400b910 comp rx=0 tx=0).stop 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 shutdown_connections 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1d406b6b0 0x7ff1d406db70 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff1fc15bf10 0x7ff1fc15a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff1fc0ccb50 0x7ff1fc0b38e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 --2- 172.21.15.53:0/494466840 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff1fc0adf20 0x7ff1fc1386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 >> 172.21.15.53:0/494466840 conn(0x7ff1fc06a7d0 msgr2=0x7ff1fc15b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 shutdown_connections 2024-01-16T15:04:23.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:23.940+0000 7ff201600640 1 -- 172.21.15.53:0/494466840 wait complete. 2024-01-16T15:04:23.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:04:24.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.175+0000 7f4aa2800640 1 Processor -- start 2024-01-16T15:04:24.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.175+0000 7f4aa2800640 1 -- start start 2024-01-16T15:04:24.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.176+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.176+0000 7f4aa2800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.176+0000 7f4aa2800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9c071690 con 0x7f4a9c14b2c0 2024-01-16T15:04:24.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.176+0000 7f4aa2800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a9c14b6a0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.176+0000 7f4aa2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c071970 con 0x7f4a9c0cf9a0 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.177+0000 7f4a9b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4a9c14b2c0 0x7f4a9c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37232/0 (socket says 172.21.15.53:37232) 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.177+0000 7f4a9b400640 1 -- 172.21.15.53:0/3328714337 learned_addr learned my addr 172.21.15.53:0/3328714337 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.177+0000 7f4a9be00640 1 --2- 172.21.15.53:0/3328714337 >> v2:172.21.15.53:3300/0 conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.177+0000 7f4a93e00640 1 --2- 172.21.15.53:0/3328714337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.177+0000 7f4a9be00640 1 -- 172.21.15.53:0/3328714337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c0cf9a0 msgr2=0x7f4a9c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.178+0000 7f4a9be00640 1 --2- 172.21.15.53:0/3328714337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.178+0000 7f4a9be00640 1 -- 172.21.15.53:0/3328714337 >> v1:172.21.15.53:6789/0 conn(0x7f4a9c14b2c0 legacy=0x7f4a9c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.178+0000 7f4a9be00640 1 -- 172.21.15.53:0/3328714337 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c071c50 con 0x7f4a9c070ea0 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.179+0000 7f4a9be00640 1 --2- 172.21.15.53:0/3328714337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009d90 tx=0x7f4a9402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce0945324d3b50a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.179+0000 7f4a9aa00640 1 -- 172.21.15.53:0/3328714337 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9402cc60 con 0x7f4a9c070ea0 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.179+0000 7f4a9aa00640 1 -- 172.21.15.53:0/3328714337 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9402cdc0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.179+0000 7f4a9aa00640 1 -- 172.21.15.53:0/3328714337 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94034660 con 0x7f4a9c070ea0 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 -- 172.21.15.53:0/3328714337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 msgr2=0x7f4a9c0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 --2- 172.21.15.53:0/3328714337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009d90 tx=0x7f4a9402beb0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 -- 172.21.15.53:0/3328714337 shutdown_connections 2024-01-16T15:04:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 --2- 172.21.15.53:0/3328714337 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c0cc160 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 --2- 172.21.15.53:0/3328714337 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c0cf9a0 0x7f4a9c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 -- 172.21.15.53:0/3328714337 >> 172.21.15.53:0/3328714337 conn(0x7f4a9c06a800 msgr2=0x7f4a9c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 -- 172.21.15.53:0/3328714337 shutdown_connections 2024-01-16T15:04:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.180+0000 7f4aa2800640 1 -- 172.21.15.53:0/3328714337 wait complete. 2024-01-16T15:04:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.182+0000 7f4aa2800640 1 Processor -- start 2024-01-16T15:04:24.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.182+0000 7f4aa2800640 1 -- start start 2024-01-16T15:04:24.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c143ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1455b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4aa2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c14b2c0 0x7f4a9c1610f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4aa2800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c144b60 con 0x7f4a9c070ea0 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4aa2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c145120 con 0x7f4a9c0cf9a0 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4aa2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c164980 con 0x7f4a9c14b2c0 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4a9b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c143ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4a9b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c143ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42406/0 (socket says 172.21.15.53:42406) 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.183+0000 7f4a9b400640 1 -- 172.21.15.53:0/1798078294 learned_addr learned my addr 172.21.15.53:0/1798078294 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9be00640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c14b2c0 0x7f4a9c1610f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a93e00640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1455b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9b400640 1 -- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c14b2c0 msgr2=0x7f4a9c1610f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9b400640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c14b2c0 0x7f4a9c1610f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9b400640 1 -- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9c0cf9a0 msgr2=0x7f4a9c1455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9b400640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1455b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9b400640 1 -- 172.21.15.53:0/1798078294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94009a70 con 0x7f4a9c070ea0 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a9b400640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c143ca0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c4b0 tx=0x7f4a8c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.184+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c005ce0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.185+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a8c005e40 con 0x7f4a9c070ea0 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.185+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c007270 con 0x7f4a9c070ea0 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.185+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a9c1441e0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.185+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a9c144db0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.186+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a9c0675d0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.187+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f4a8c0073d0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.188+0000 7f4a93400640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7406b6b0 0x7f4a7406db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.188+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f4a8c0896e0 con 0x7f4a9c070ea0 2024-01-16T15:04:24.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.188+0000 7f4a93e00640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7406b6b0 0x7f4a7406db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.190+0000 7f4a93e00640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7406b6b0 0x7f4a7406db70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c164380 tx=0x7f4a84008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.192+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a8c054090 con 0x7f4a9c070ea0 2024-01-16T15:04:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.457+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f4a9c06ae30 con 0x7f4a9c070ea0 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.458+0000 7f4a93400640 1 -- 172.21.15.53:0/1798078294 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1853 (secure 0 0 0) 0x7f4a8c017750 con 0x7f4a9c070ea0 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:04:24.437 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:04:24.438 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 291, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 49, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:04:24.439 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:04:24.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 800, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3222214628, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 6721536000, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 766372577280, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 597, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 10747, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 0, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 3 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 26, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 3, 2024-01-16T15:04:24.441 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 3, 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "c", 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5718 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:04:24.442 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 11, 2024-01-16T15:04:24.443 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:04:08.266313+0000", 2024-01-16T15:04:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:04:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:24.444 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": {} 2024-01-16T15:04:24.444 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7406b6b0 msgr2=0x7f4a7406db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7406b6b0 0x7f4a7406db70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f4a9c164380 tx=0x7f4a84008040 comp rx=0 tx=0).stop 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 msgr2=0x7f4a9c143ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c143ca0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c4b0 tx=0x7f4a8c00c970 comp rx=0 tx=0).stop 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 shutdown_connections 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7406b6b0 0x7f4a7406db70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9c14b2c0 0x7f4a9c1610f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9c0cf9a0 0x7f4a9c1455b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 --2- 172.21.15.53:0/1798078294 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a9c070ea0 0x7f4a9c143ca0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 >> 172.21.15.53:0/1798078294 conn(0x7f4a9c06a800 msgr2=0x7f4a9c06cea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:24.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 shutdown_connections 2024-01-16T15:04:24.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.461+0000 7f4aa2800640 1 -- 172.21.15.53:0/1798078294 wait complete. 2024-01-16T15:04:24.460 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:24.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:24.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.727+0000 7f216ce00640 1 Processor -- start 2024-01-16T15:04:24.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.727+0000 7f216ce00640 1 -- start start 2024-01-16T15:04:24.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f216ce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2168150760 0x7f2168072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f216ce00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f216815d300 0x7f216815d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f216ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21680b2bd0 0x7f21680b3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f216ce00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f216815dc80 con 0x7f216815d300 2024-01-16T15:04:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f216ce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2168071190 con 0x7f2168150760 2024-01-16T15:04:24.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f216ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2168071470 con 0x7f21680b2bd0 2024-01-16T15:04:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.730+0000 7f2165400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f216815d300 0x7f216815d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f216815d300 0x7f216815d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42412/0 (socket says 172.21.15.53:42412) 2024-01-16T15:04:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 -- 172.21.15.53:0/1032623236 learned_addr learned my addr 172.21.15.53:0/1032623236 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165e00640 1 --2- 172.21.15.53:0/1032623236 >> v2:172.21.15.146:3300/0 conn(0x7f2168150760 0x7f2168072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2166800640 1 --2- 172.21.15.53:0/1032623236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21680b2bd0 0x7f21680b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 -- 172.21.15.53:0/1032623236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21680b2bd0 msgr2=0x7f21680b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 --2- 172.21.15.53:0/1032623236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21680b2bd0 0x7f21680b3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 -- 172.21.15.53:0/1032623236 >> v2:172.21.15.146:3300/0 conn(0x7f2168150760 msgr2=0x7f2168072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 --2- 172.21.15.53:0/1032623236 >> v2:172.21.15.146:3300/0 conn(0x7f2168150760 0x7f2168072230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.731+0000 7f2165400640 1 -- 172.21.15.53:0/1032623236 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2168071750 con 0x7f216815d300 2024-01-16T15:04:24.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.734+0000 7f2165400640 1 --2- 172.21.15.53:0/1032623236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216815d740 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f216806e420 tx=0x7f215802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a57302d3b0d53 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.735+0000 7f2164a00640 1 -- 172.21.15.53:0/1032623236 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f215802ca80 con 0x7f216815d300 2024-01-16T15:04:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.735+0000 7f2164a00640 1 -- 172.21.15.53:0/1032623236 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f215802cbe0 con 0x7f216815d300 2024-01-16T15:04:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.735+0000 7f2164a00640 1 -- 172.21.15.53:0/1032623236 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21580343f0 con 0x7f216815d300 2024-01-16T15:04:24.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.736+0000 7f216ce00640 1 -- 172.21.15.53:0/1032623236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 msgr2=0x7f216815d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.736+0000 7f216ce00640 1 --2- 172.21.15.53:0/1032623236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216815d740 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f216806e420 tx=0x7f215802bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.737+0000 7f216ce00640 1 -- 172.21.15.53:0/1032623236 shutdown_connections 2024-01-16T15:04:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.737+0000 7f216ce00640 1 --2- 172.21.15.53:0/1032623236 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21680b2bd0 0x7f21680b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.737+0000 7f216ce00640 1 --2- 172.21.15.53:0/1032623236 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216815d740 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.737+0000 7f216ce00640 1 --2- 172.21.15.53:0/1032623236 >> v2:172.21.15.146:3300/0 conn(0x7f2168150760 0x7f2168072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.737+0000 7f216ce00640 1 -- 172.21.15.53:0/1032623236 >> 172.21.15.53:0/1032623236 conn(0x7f216806a800 msgr2=0x7f216806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:24.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.738+0000 7f216ce00640 1 -- 172.21.15.53:0/1032623236 shutdown_connections 2024-01-16T15:04:24.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.738+0000 7f216ce00640 1 -- 172.21.15.53:0/1032623236 wait complete. 2024-01-16T15:04:24.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.740+0000 7f216ce00640 1 Processor -- start 2024-01-16T15:04:24.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.741+0000 7f216ce00640 1 -- start start 2024-01-16T15:04:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.743+0000 7f216ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21680b2bd0 0x7f2168158b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.743+0000 7f216ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2168150760 0x7f21681477a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.743+0000 7f216ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216814f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.743+0000 7f216ce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21681483a0 con 0x7f216815d300 2024-01-16T15:04:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.743+0000 7f216ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2168158800 con 0x7f21680b2bd0 2024-01-16T15:04:24.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.743+0000 7f216ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f216814ceb0 con 0x7f2168150760 2024-01-16T15:04:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.744+0000 7f2166800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216814f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.744+0000 7f2166800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216814f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42418/0 (socket says 172.21.15.53:42418) 2024-01-16T15:04:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.744+0000 7f2166800640 1 -- 172.21.15.53:0/404269198 learned_addr learned my addr 172.21.15.53:0/404269198 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:24.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.744+0000 7f2165400640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2168150760 0x7f21681477a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.745+0000 7f2166800640 1 -- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2168150760 msgr2=0x7f21681477a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.745+0000 7f2166800640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2168150760 0x7f21681477a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.745+0000 7f2166800640 1 -- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21680b2bd0 msgr2=0x7f2168158b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.745+0000 7f2166800640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21680b2bd0 0x7f2168158b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.745+0000 7f2166800640 1 -- 172.21.15.53:0/404269198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2158009b30 con 0x7f216815d300 2024-01-16T15:04:24.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.746+0000 7f2166800640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216814f5c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f216000d830 tx=0x7f216000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.746+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21600096c0 con 0x7f216815d300 2024-01-16T15:04:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.746+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2160009840 con 0x7f216815d300 2024-01-16T15:04:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.746+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2160009bf0 con 0x7f216815d300 2024-01-16T15:04:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.746+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f216814d190 con 0x7f216815d300 2024-01-16T15:04:24.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.747+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f216814d750 con 0x7f216815d300 2024-01-16T15:04:24.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.749+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f216814d940 con 0x7f216815d300 2024-01-16T15:04:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.750+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f2160014410 con 0x7f216815d300 2024-01-16T15:04:24.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.751+0000 7f215e000640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f214006b6b0 0x7f214006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:24.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.751+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f216008a8f0 con 0x7f216815d300 2024-01-16T15:04:24.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.755+0000 7f2165e00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f214006b6b0 0x7f214006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:24.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.759+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21600550b0 con 0x7f216815d300 2024-01-16T15:04:24.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:24.781+0000 7f2165e00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f214006b6b0 0x7f214006db70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2150003860 tx=0x7f2150006ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:25.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.180+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f2168150b40 con 0x7f214006b6b0 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:25.205 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:25.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.226+0000 7f215e000640 1 -- 172.21.15.53:0/404269198 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f2150003e40 con 0x7f214006b6b0 2024-01-16T15:04:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.231+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f214006b6b0 msgr2=0x7f214006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.231+0000 7f216ce00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f214006b6b0 0x7f214006db70 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2150003860 tx=0x7f2150006ed0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.231+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 msgr2=0x7f216814f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:25.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.231+0000 7f216ce00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216814f5c0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f216000d830 tx=0x7f216000dcf0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.232+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 shutdown_connections 2024-01-16T15:04:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.232+0000 7f216ce00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f214006b6b0 0x7f214006db70 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.232+0000 7f216ce00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f216815d300 0x7f216814f5c0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.232+0000 7f216ce00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2168150760 0x7f21681477a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.232+0000 7f216ce00640 1 --2- 172.21.15.53:0/404269198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21680b2bd0 0x7f2168158b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:25.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.232+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 >> 172.21.15.53:0/404269198 conn(0x7f216806a800 msgr2=0x7f2168070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.233+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 shutdown_connections 2024-01-16T15:04:25.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:25.234+0000 7f216ce00640 1 -- 172.21.15.53:0/404269198 wait complete. 2024-01-16T15:04:25.240 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:26.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:26.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.535+0000 7fcf9be00640 1 Processor -- start 2024-01-16T15:04:26.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.536+0000 7fcf9be00640 1 -- start start 2024-01-16T15:04:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf9be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcf940cfcf0 0x7fcf94160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf9be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcf940d3910 0x7fcf94160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf9be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940d4160 con 0x7fcf940d00d0 2024-01-16T15:04:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf9be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf940ceff0 con 0x7fcf940cfcf0 2024-01-16T15:04:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf9be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf940cf410 con 0x7fcf940d3910 2024-01-16T15:04:26.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf99400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcf940cfcf0 0x7fcf94160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf99400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcf940cfcf0 0x7fcf94160700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42428/0 (socket says 172.21.15.53:42428) 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.538+0000 7fcf99400640 1 -- 172.21.15.53:0/2527779253 learned_addr learned my addr 172.21.15.53:0/2527779253 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.539+0000 7fcf93e00640 1 -- 172.21.15.53:0/2527779253 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2303445023 0 0) 0x7fcf940d4160 con 0x7fcf940d00d0 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.539+0000 7fcf93e00640 1 -- 172.21.15.53:0/2527779253 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf74003520 con 0x7fcf940d00d0 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.539+0000 7fcf99400640 1 -- 172.21.15.53:0/2527779253 >> v2:172.21.15.146:3300/0 conn(0x7fcf940d3910 msgr2=0x7fcf94160c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.539+0000 7fcf99400640 1 --2- 172.21.15.53:0/2527779253 >> v2:172.21.15.146:3300/0 conn(0x7fcf940d3910 0x7fcf94160c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.539+0000 7fcf99400640 1 -- 172.21.15.53:0/2527779253 >> v1:172.21.15.53:6789/0 conn(0x7fcf940d00d0 legacy=0x7fcf940b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.539+0000 7fcf99400640 1 -- 172.21.15.53:0/2527779253 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf940b2420 con 0x7fcf940cfcf0 2024-01-16T15:04:26.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.541+0000 7fcf99400640 1 --2- 172.21.15.53:0/2527779253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940cfcf0 0x7fcf94160700 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcf9408b640 tx=0x7fcf8402b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1d138af567fa95a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.541+0000 7fcf93e00640 1 -- 172.21.15.53:0/2527779253 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84030800 con 0x7fcf940cfcf0 2024-01-16T15:04:26.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.542+0000 7fcf93e00640 1 -- 172.21.15.53:0/2527779253 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf84030960 con 0x7fcf940cfcf0 2024-01-16T15:04:26.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.542+0000 7fcf93e00640 1 -- 172.21.15.53:0/2527779253 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf84030cf0 con 0x7fcf940cfcf0 2024-01-16T15:04:26.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.543+0000 7fcf9be00640 1 -- 172.21.15.53:0/2527779253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940cfcf0 msgr2=0x7fcf94160700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.543+0000 7fcf9be00640 1 --2- 172.21.15.53:0/2527779253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940cfcf0 0x7fcf94160700 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fcf9408b640 tx=0x7fcf8402b940 comp rx=0 tx=0).stop 2024-01-16T15:04:26.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.543+0000 7fcf9be00640 1 -- 172.21.15.53:0/2527779253 shutdown_connections 2024-01-16T15:04:26.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.543+0000 7fcf9be00640 1 --2- 172.21.15.53:0/2527779253 >> v2:172.21.15.146:3300/0 conn(0x7fcf940d3910 0x7fcf94160c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.543+0000 7fcf9be00640 1 --2- 172.21.15.53:0/2527779253 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940cfcf0 0x7fcf94160700 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.543+0000 7fcf9be00640 1 -- 172.21.15.53:0/2527779253 >> 172.21.15.53:0/2527779253 conn(0x7fcf9409c530 msgr2=0x7fcf940d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:26.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.544+0000 7fcf9be00640 1 -- 172.21.15.53:0/2527779253 shutdown_connections 2024-01-16T15:04:26.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.544+0000 7fcf9be00640 1 -- 172.21.15.53:0/2527779253 wait complete. 2024-01-16T15:04:26.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.547+0000 7fcf9be00640 1 Processor -- start 2024-01-16T15:04:26.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.548+0000 7fcf9be00640 1 -- start start 2024-01-16T15:04:26.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.549+0000 7fcf9be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 0x7fcf9415b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf99400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 0x7fcf9415b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf9be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940d00d0 0x7fcf940b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf9be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf940d3910 0x7fcf940b8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf9be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415a730 con 0x7fcf940d00d0 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf9be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415a980 con 0x7fcf940d3910 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf9be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9415ac60 con 0x7fcf940cfcf0 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf99400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 0x7fcf9415b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:57556/0 (socket says 172.21.15.53:57556) 2024-01-16T15:04:26.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.551+0000 7fcf99400640 1 -- 172.21.15.53:0/692586899 learned_addr learned my addr 172.21.15.53:0/692586899 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:26.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.553+0000 7fcf98a00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940d00d0 0x7fcf940b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.554+0000 7fcf99e00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf940d3910 0x7fcf940b8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.556+0000 7fcf98a00640 1 -- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 msgr2=0x7fcf9415b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.556+0000 7fcf98a00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 0x7fcf9415b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.556+0000 7fcf98a00640 1 -- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf940d3910 msgr2=0x7fcf940b8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.556+0000 7fcf98a00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf940d3910 0x7fcf940b8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.556+0000 7fcf98a00640 1 -- 172.21.15.53:0/692586899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf840094d0 con 0x7fcf940d00d0 2024-01-16T15:04:26.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.556+0000 7fcf99400640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 0x7fcf9415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:26.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.557+0000 7fcf98a00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940d00d0 0x7fcf940b2fe0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00b3e0 tx=0x7fcf7c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.559+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c011830 con 0x7fcf940d00d0 2024-01-16T15:04:26.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.559+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf7c011990 con 0x7fcf940d00d0 2024-01-16T15:04:26.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.559+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9415af40 con 0x7fcf940d00d0 2024-01-16T15:04:26.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.559+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7c011d40 con 0x7fcf940d00d0 2024-01-16T15:04:26.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.559+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9415d710 con 0x7fcf940d00d0 2024-01-16T15:04:26.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.564+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9415b460 con 0x7fcf940d00d0 2024-01-16T15:04:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.565+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7fcf7c01a400 con 0x7fcf940d00d0 2024-01-16T15:04:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.567+0000 7fcf91600640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf6c06b780 0x7fcf6c06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.567+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcf7c08a2e0 con 0x7fcf940d00d0 2024-01-16T15:04:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.572+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf9415b460 con 0x7fcf940d00d0 2024-01-16T15:04:26.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.580+0000 7fcf99400640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf6c06b780 0x7fcf6c06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:26.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.582+0000 7fcf99400640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf6c06b780 0x7fcf6c06dc40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcf8402b980 tx=0x7fcf84008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:26.990+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf940cec10 con 0x7fcf6c06b780 2024-01-16T15:04:26.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.011+0000 7fcf91600640 1 -- 172.21.15.53:0/692586899 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fcf84030bd0 con 0x7fcf6c06b780 2024-01-16T15:04:26.991 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:26.992 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:26.993 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:26.993 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:26.993 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:26.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.018+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf6c06b780 msgr2=0x7fcf6c06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.018+0000 7fcf9be00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf6c06b780 0x7fcf6c06dc40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcf8402b980 tx=0x7fcf84008030 comp rx=0 tx=0).stop 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.018+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940d00d0 msgr2=0x7fcf940b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.018+0000 7fcf9be00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940d00d0 0x7fcf940b2fe0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcf7c00b3e0 tx=0x7fcf7c00b8a0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.019+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 shutdown_connections 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.019+0000 7fcf9be00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf6c06b780 0x7fcf6c06dc40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.019+0000 7fcf9be00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf940d3910 0x7fcf940b8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.019+0000 7fcf9be00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf940d00d0 0x7fcf940b2fe0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.019+0000 7fcf9be00640 1 --2- 172.21.15.53:0/692586899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf940cfcf0 0x7fcf9415b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:26.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.019+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 >> 172.21.15.53:0/692586899 conn(0x7fcf9409c530 msgr2=0x7fcf940d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:27.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.022+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 shutdown_connections 2024-01-16T15:04:27.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:27.022+0000 7fcf9be00640 1 -- 172.21.15.53:0/692586899 wait complete. 2024-01-16T15:04:27.042 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:27.417 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:27.436+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:27.417 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:27.436+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:27.417 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:27.436+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:27.417 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:27.436+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:27.417 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:27.436+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:28.043 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:28.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.296+0000 7f6b20800640 1 Processor -- start 2024-01-16T15:04:28.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.296+0000 7f6b20800640 1 -- start start 2024-01-16T15:04:28.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b20800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6b1813df80 0x7f6b1815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b20800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 0x7f6b180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b20800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1806c470 con 0x7f6b1815f9c0 2024-01-16T15:04:28.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b20800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b1806c750 con 0x7f6b1813df80 2024-01-16T15:04:28.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b20800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1806ca30 con 0x7f6b1813e360 2024-01-16T15:04:28.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b1e800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6b1815f9c0 0x7f6b181408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37244/0 (socket says 172.21.15.53:37244) 2024-01-16T15:04:28.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b1e800640 1 -- 172.21.15.53:0/3440156408 learned_addr learned my addr 172.21.15.53:0/3440156408 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b1de00640 1 --2- 172.21.15.53:0/3440156408 >> v2:172.21.15.146:3300/0 conn(0x7f6b1813df80 0x7f6b1815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.298+0000 7f6b1d400640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 0x7f6b180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745480293 0 0) 0x7f6b1806c470 con 0x7f6b1815f9c0 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6af8003520 con 0x7f6b1815f9c0 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2806908076 0 0) 0x7f6af8003520 con 0x7f6b1815f9c0 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 msgr2=0x7f6b180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 0x7f6b180cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 >> v2:172.21.15.146:3300/0 conn(0x7f6b1813df80 msgr2=0x7f6b1815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:28.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 --2- 172.21.15.53:0/3440156408 >> v2:172.21.15.146:3300/0 conn(0x7f6b1813df80 0x7f6b1815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b1806cd10 con 0x7f6b1815f9c0 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6b100019f0 con 0x7f6b1815f9c0 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 >> v1:172.21.15.53:6789/0 conn(0x7f6b1815f9c0 legacy=0x7f6b181408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8003970 0x7f6af8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 0x7f6af80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6af8004d00 0x7f6af8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1806c750 con 0x7f6af8004d00 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1806ca30 con 0x7f6af8003970 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.299+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6af800b980 con 0x7f6b1813e360 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1e800640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6af8004d00 0x7f6af8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1de00640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8003970 0x7f6af8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1e800640 1 -- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 msgr2=0x7f6af80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1e800640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b1813e360 0x7f6af80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1e800640 1 -- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8003970 msgr2=0x7f6af8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:28.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1e800640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8003970 0x7f6af8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.300+0000 7f6b1e800640 1 -- 172.21.15.53:0/3440156408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af8003520 con 0x7f6af8004d00 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.301+0000 7f6b1e800640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6af8004d00 0x7f6af8005180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000a900 tx=0x7f6af80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0bc0997cf46506a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10034630 con 0x7f6af8004d00 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1002f070 con 0x7f6af8004d00 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b1ca00640 1 -- 172.21.15.53:0/3440156408 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b100349d0 con 0x7f6af8004d00 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b20800640 1 -- 172.21.15.53:0/3440156408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6af8004d00 msgr2=0x7f6af8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b20800640 1 --2- 172.21.15.53:0/3440156408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6af8004d00 0x7f6af8005180 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f6b1000a900 tx=0x7f6af80057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b1e800640 1 -- 172.21.15.53:0/3440156408 reap_dead start 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b20800640 1 -- 172.21.15.53:0/3440156408 shutdown_connections 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.302+0000 7f6b20800640 1 -- 172.21.15.53:0/3440156408 >> 172.21.15.53:0/3440156408 conn(0x7f6b1806a800 msgr2=0x7f6b1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.303+0000 7f6b20800640 1 -- 172.21.15.53:0/3440156408 shutdown_connections 2024-01-16T15:04:28.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.303+0000 7f6b20800640 1 -- 172.21.15.53:0/3440156408 wait complete. 2024-01-16T15:04:28.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.305+0000 7f6b20800640 1 Processor -- start 2024-01-16T15:04:28.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.305+0000 7f6b20800640 1 -- start start 2024-01-16T15:04:28.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.306+0000 7f6b20800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8004d00 0x7f6b181f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.306+0000 7f6b20800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b181f2f70 0x7f6b1814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.306+0000 7f6b20800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 0x7f6b1814b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.306+0000 7f6b20800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1814ded0 con 0x7f6b18131510 2024-01-16T15:04:28.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.306+0000 7f6b20800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b1814dbf0 con 0x7f6af8004d00 2024-01-16T15:04:28.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.306+0000 7f6b20800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b18130a30 con 0x7f6b181f2f70 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.307+0000 7f6b1e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 0x7f6b1814b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.307+0000 7f6b1e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 0x7f6b1814b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42466/0 (socket says 172.21.15.53:42466) 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.307+0000 7f6b1e800640 1 -- 172.21.15.53:0/1672698479 learned_addr learned my addr 172.21.15.53:0/1672698479 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.307+0000 7f6b1de00640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8004d00 0x7f6b181f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.307+0000 7f6b1d400640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b181f2f70 0x7f6b1814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b1e800640 1 -- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b181f2f70 msgr2=0x7f6b1814b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b1e800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b181f2f70 0x7f6b1814b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b1e800640 1 -- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8004d00 msgr2=0x7f6b181f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b1e800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8004d00 0x7f6b181f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b1e800640 1 -- 172.21.15.53:0/1672698479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b18130d10 con 0x7f6b18131510 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b1e800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 0x7f6b1814b540 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6b1002cb70 tx=0x7f6b10001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b10002000 con 0x7f6b18131510 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b10034640 con 0x7f6b18131510 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b100349b0 con 0x7f6b18131510 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.308+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b18130ff0 con 0x7f6b18131510 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.309+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b1814d050 con 0x7f6b18131510 2024-01-16T15:04:28.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.310+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b181311e0 con 0x7f6b18131510 2024-01-16T15:04:28.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.311+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 11) v1 ==== 89896+0+0 (secure 0 0 0) 0x7f6b10031030 con 0x7f6b18131510 2024-01-16T15:04:28.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.312+0000 7f6b16000640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006b6b0 0x7f6af006db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:28.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.312+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6b100ab8b0 con 0x7f6b18131510 2024-01-16T15:04:28.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.313+0000 7f6b1de00640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006b6b0 0x7f6af006db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:28.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.314+0000 7f6b1de00640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006b6b0 0x7f6af006db70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6b1814a920 tx=0x7f6b08008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:28.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.317+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b100760a0 con 0x7f6b18131510 2024-01-16T15:04:28.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.551+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b181308b0 con 0x7f6af006b6b0 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.566+0000 7f6b16000640 1 -- 172.21.15.53:0/1672698479 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f6b08014ac0 con 0x7f6af006b6b0 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:28.545 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:28.546 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:28.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006b6b0 msgr2=0x7f6af006db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006b6b0 0x7f6af006db70 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f6b1814a920 tx=0x7f6b08008040 comp rx=0 tx=0).stop 2024-01-16T15:04:28.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 msgr2=0x7f6b1814b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:28.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 0x7f6b1814b540 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f6b1002cb70 tx=0x7f6b10001360 comp rx=0 tx=0).stop 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 shutdown_connections 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006b6b0 0x7f6af006db70 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b18131510 0x7f6b1814b540 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b181f2f70 0x7f6b1814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 --2- 172.21.15.53:0/1672698479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6af8004d00 0x7f6b181f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 >> 172.21.15.53:0/1672698479 conn(0x7f6b1806a800 msgr2=0x7f6b1815c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 shutdown_connections 2024-01-16T15:04:28.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:28.569+0000 7f6b20800640 1 -- 172.21.15.53:0/1672698479 wait complete. 2024-01-16T15:04:28.568 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:29.569 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:29.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.832+0000 7fa3cfe00640 1 Processor -- start 2024-01-16T15:04:29.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.833+0000 7fa3cfe00640 1 -- start start 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.834+0000 7fa3cfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c8161a80 0x7fa3c80c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.834+0000 7fa3cfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3c80b2620 0x7fa3c80c4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.834+0000 7fa3cfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3c80cf9c0 0x7fa3c80cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.834+0000 7fa3cfe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c8161e60 con 0x7fa3c80cf9c0 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.834+0000 7fa3cfe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c806bd90 con 0x7fa3c80b2620 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.834+0000 7fa3cfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80adcd0 con 0x7fa3c8161a80 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3c80cf9c0 0x7fa3c80cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa3c80cf9c0 0x7fa3c80cfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42482/0 (socket says 172.21.15.53:42482) 2024-01-16T15:04:29.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 -- 172.21.15.53:0/578597566 learned_addr learned my addr 172.21.15.53:0/578597566 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:29.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cca00640 1 --2- 172.21.15.53:0/578597566 >> v2:172.21.15.146:3300/0 conn(0x7fa3c80b2620 0x7fa3c80c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cd400640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c8161a80 0x7fa3c80c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 -- 172.21.15.53:0/578597566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c8161a80 msgr2=0x7fa3c80c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:29.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c8161a80 0x7fa3c80c3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 -- 172.21.15.53:0/578597566 >> v2:172.21.15.146:3300/0 conn(0x7fa3c80b2620 msgr2=0x7fa3c80c4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:29.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 --2- 172.21.15.53:0/578597566 >> v2:172.21.15.146:3300/0 conn(0x7fa3c80b2620 0x7fa3c80c4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cde00640 1 -- 172.21.15.53:0/578597566 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c80d1e70 con 0x7fa3c80cf9c0 2024-01-16T15:04:29.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.835+0000 7fa3cd400640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c8161a80 0x7fa3c80c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:29.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.837+0000 7fa3cde00640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c80cf9c0 0x7fa3c80cfe10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8070e20 tx=0x7fa3c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a2328028ba43b47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:29.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3c7e00640 1 -- 172.21.15.53:0/578597566 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0031870 con 0x7fa3c80cf9c0 2024-01-16T15:04:29.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3c7e00640 1 -- 172.21.15.53:0/578597566 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c00319d0 con 0x7fa3c80cf9c0 2024-01-16T15:04:29.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3c7e00640 1 -- 172.21.15.53:0/578597566 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0031d60 con 0x7fa3c80cf9c0 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 -- 172.21.15.53:0/578597566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c80cf9c0 msgr2=0x7fa3c80cfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c80cf9c0 0x7fa3c80cfe10 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fa3c8070e20 tx=0x7fa3c002b910 comp rx=0 tx=0).stop 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 -- 172.21.15.53:0/578597566 shutdown_connections 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c80cf9c0 0x7fa3c80cfe10 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/578597566 >> v2:172.21.15.146:3300/0 conn(0x7fa3c80b2620 0x7fa3c80c4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/578597566 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c8161a80 0x7fa3c80c3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 -- 172.21.15.53:0/578597566 >> 172.21.15.53:0/578597566 conn(0x7fa3c80d1020 msgr2=0x7fa3c80d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.838+0000 7fa3cfe00640 1 -- 172.21.15.53:0/578597566 shutdown_connections 2024-01-16T15:04:29.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.839+0000 7fa3cfe00640 1 -- 172.21.15.53:0/578597566 wait complete. 2024-01-16T15:04:29.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.841+0000 7fa3cfe00640 1 Processor -- start 2024-01-16T15:04:29.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.841+0000 7fa3cfe00640 1 -- start start 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80b2620 0x7fa3c81f9de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cf9c0 0x7fa3c80cc6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 0x7fa3c80ccbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cfe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80c7f30 con 0x7fa3c8161a80 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80c7970 con 0x7fa3c80b2620 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c80c7c50 con 0x7fa3c80cf9c0 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 0x7fa3c80ccbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 0x7fa3c80ccbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42494/0 (socket says 172.21.15.53:42494) 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cde00640 1 -- 172.21.15.53:0/1503705628 learned_addr learned my addr 172.21.15.53:0/1503705628 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cd400640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80b2620 0x7fa3c81f9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.842+0000 7fa3cca00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cf9c0 0x7fa3c80cc6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cde00640 1 -- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cf9c0 msgr2=0x7fa3c80cc6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cde00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cf9c0 0x7fa3c80cc6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cde00640 1 -- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80b2620 msgr2=0x7fa3c81f9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cde00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80b2620 0x7fa3c81f9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cde00640 1 -- 172.21.15.53:0/1503705628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00094d0 con 0x7fa3c8161a80 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cd400640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80b2620 0x7fa3c81f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cde00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 0x7fa3c80ccbe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00090b0 tx=0x7fa3c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c003ac60 con 0x7fa3c8161a80 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3c003adc0 con 0x7fa3c8161a80 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c0044630 con 0x7fa3c8161a80 2024-01-16T15:04:29.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.843+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c81480a0 con 0x7fa3c8161a80 2024-01-16T15:04:29.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.844+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c80ae160 con 0x7fa3c8161a80 2024-01-16T15:04:29.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.845+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c8147f00 con 0x7fa3c8161a80 2024-01-16T15:04:29.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.846+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7fa3c004d450 con 0x7fa3c8161a80 2024-01-16T15:04:29.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.847+0000 7fa3c5600640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa3a006b740 0x7fa3a006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:29.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.847+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa3c002d040 con 0x7fa3c8161a80 2024-01-16T15:04:29.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.848+0000 7fa3cd400640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa3a006b740 0x7fa3a006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:29.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.850+0000 7fa3cd400640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa3a006b740 0x7fa3a006dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80c78a0 tx=0x7fa3b0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:29.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:29.857+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3c8147f00 con 0x7fa3c8161a80 2024-01-16T15:04:30.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.247+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3c806bab0 con 0x7fa3a006b740 2024-01-16T15:04:30.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.305+0000 7fa3c5600640 1 -- 172.21.15.53:0/1503705628 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fa3b000a210 con 0x7fa3a006b740 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:30.284 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:30.285 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:30.285 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:30.285 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:30.285 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:30.285 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:30.285 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:30.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa3a006b740 msgr2=0x7fa3a006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa3a006b740 0x7fa3a006dc00 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa3c80c78a0 tx=0x7fa3b0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 msgr2=0x7fa3c80ccbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 0x7fa3c80ccbe0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fa3c00090b0 tx=0x7fa3c00046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 shutdown_connections 2024-01-16T15:04:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa3a006b740 0x7fa3a006dc00 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa3c8161a80 0x7fa3c80ccbe0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3c80cf9c0 0x7fa3c80cc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 --2- 172.21.15.53:0/1503705628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa3c80b2620 0x7fa3c81f9de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.308+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 >> 172.21.15.53:0/1503705628 conn(0x7fa3c80d1020 msgr2=0x7fa3c8160080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.309+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 shutdown_connections 2024-01-16T15:04:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:30.309+0000 7fa3cfe00640 1 -- 172.21.15.53:0/1503705628 wait complete. 2024-01-16T15:04:30.321 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:31.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:31.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.653+0000 7f7fae200640 1 Processor -- start 2024-01-16T15:04:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.654+0000 7f7fae200640 1 -- start start 2024-01-16T15:04:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.656+0000 7f7fae200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7fa814b2c0 0x7f7fa80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.656+0000 7f7fae200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa80cfbf0 con 0x7f7fa80cf7d0 2024-01-16T15:04:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.656+0000 7f7fae200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fa8071690 con 0x7f7fa8071070 2024-01-16T15:04:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.656+0000 7f7fae200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa814b6a0 con 0x7f7fa814b2c0 2024-01-16T15:04:31.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.657+0000 7f7f9fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7fa80cf7d0 0x7f7fa8070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37254/0 (socket says 172.21.15.53:37254) 2024-01-16T15:04:31.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.657+0000 7f7f9fe00640 1 -- 172.21.15.53:0/3663298817 learned_addr learned my addr 172.21.15.53:0/3663298817 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.658+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383197026 0 0) 0x7f7fa80cfbf0 con 0x7f7fa80cf7d0 2024-01-16T15:04:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.658+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f88003520 con 0x7f7fa80cf7d0 2024-01-16T15:04:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2518791269 0 0) 0x7f7f88003520 con 0x7f7fa80cf7d0 2024-01-16T15:04:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 >> v2:172.21.15.146:3300/0 conn(0x7f7fa814b2c0 msgr2=0x7f7fa80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 --2- 172.21.15.53:0/3663298817 >> v2:172.21.15.146:3300/0 conn(0x7f7fa814b2c0 0x7f7fa80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 >> v1:172.21.15.146:6789/0 conn(0x7f7fa8071070 legacy=0x7f7fa80cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa8071970 con 0x7f7fa80cf7d0 2024-01-16T15:04:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7f90001670 con 0x7f7fa80cf7d0 2024-01-16T15:04:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 >> v1:172.21.15.53:6789/0 conn(0x7f7fa80cf7d0 legacy=0x7f7fa8070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880039c0 0x7f7f88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f880047c0 0x7f7f88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f880052a0 0x7f7f8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa814b6a0 con 0x7f7f880047c0 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8800c1f0 con 0x7f7f880052a0 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f8800c3b0 con 0x7f7f880039c0 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.659+0000 7f7f9fe00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f880047c0 0x7f7f88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7fa7e00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f880052a0 0x7f7f8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7f9fe00640 1 -- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880039c0 msgr2=0x7f7f88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7f9fe00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880039c0 0x7f7f88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7f9fe00640 1 -- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f880052a0 msgr2=0x7f7f8800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7f9fe00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f880052a0 0x7f7f8800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7f9fe00640 1 -- 172.21.15.53:0/3663298817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f88003520 con 0x7f7f880047c0 2024-01-16T15:04:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7fa7e00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f880052a0 0x7f7f8800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.660+0000 7f7fa7e00640 1 -- 172.21.15.53:0/3663298817 reap_dead start 2024-01-16T15:04:31.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.662+0000 7f7f9fe00640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f880047c0 0x7f7f88004c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c160 tx=0x7f7f880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf66f1b181728354 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.663+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f90005ad0 con 0x7f7f880047c0 2024-01-16T15:04:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.663+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f90005c30 con 0x7f7f880047c0 2024-01-16T15:04:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.663+0000 7f7fa6a00640 1 -- 172.21.15.53:0/3663298817 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9002ca20 con 0x7f7f880047c0 2024-01-16T15:04:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.665+0000 7f7fae200640 1 -- 172.21.15.53:0/3663298817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f880047c0 msgr2=0x7f7f88004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.665+0000 7f7fae200640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f880047c0 0x7f7f88004c40 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f7f9000c160 tx=0x7f7f880057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.666+0000 7f7fae200640 1 -- 172.21.15.53:0/3663298817 shutdown_connections 2024-01-16T15:04:31.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.666+0000 7f7fae200640 1 --2- 172.21.15.53:0/3663298817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f880047c0 0x7f7f88004c40 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.667+0000 7f7fae200640 1 -- 172.21.15.53:0/3663298817 >> 172.21.15.53:0/3663298817 conn(0x7f7fa806a800 msgr2=0x7f7fa806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:31.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.670+0000 7f7fae200640 1 -- 172.21.15.53:0/3663298817 shutdown_connections 2024-01-16T15:04:31.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.670+0000 7f7fae200640 1 -- 172.21.15.53:0/3663298817 wait complete. 2024-01-16T15:04:31.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.677+0000 7f7fae200640 1 Processor -- start 2024-01-16T15:04:31.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.677+0000 7f7fae200640 1 -- start start 2024-01-16T15:04:31.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.679+0000 7f7fae200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880047c0 0x7f7fa8145a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.679+0000 7f7fae200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa8162710 0x7f7fa80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.680+0000 7f7fae200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 0x7f7fa80cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.680+0000 7f7fae200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa8160ac0 con 0x7f7fa8162b90 2024-01-16T15:04:31.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.680+0000 7f7fae200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa81f7810 con 0x7f7fa8162710 2024-01-16T15:04:31.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.680+0000 7f7fae200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa81f7af0 con 0x7f7f880047c0 2024-01-16T15:04:31.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.681+0000 7f7fa7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880047c0 0x7f7fa8145a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.681+0000 7f7fa7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 0x7f7fa80cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.681+0000 7f7f9fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa8162710 0x7f7fa80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.681+0000 7f7fa7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 0x7f7fa80cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42512/0 (socket says 172.21.15.53:42512) 2024-01-16T15:04:31.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.681+0000 7f7fa7e00640 1 -- 172.21.15.53:0/869212090 learned_addr learned my addr 172.21.15.53:0/869212090 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:31.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.683+0000 7f7fa7e00640 1 -- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880047c0 msgr2=0x7f7fa8145a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.683+0000 7f7fa7e00640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880047c0 0x7f7fa8145a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.683+0000 7f7fa7e00640 1 -- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa8162710 msgr2=0x7f7fa80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:31.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.683+0000 7f7fa7e00640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa8162710 0x7f7fa80cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:31.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.683+0000 7f7fa7e00640 1 -- 172.21.15.53:0/869212090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa81f7dd0 con 0x7f7fa8162b90 2024-01-16T15:04:31.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.685+0000 7f7fa7e00640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 0x7f7fa80cd720 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80cfe50 tx=0x7f7fa000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:31.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.686+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0017070 con 0x7f7fa8162b90 2024-01-16T15:04:31.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.686+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fa000de50 con 0x7f7fa8162b90 2024-01-16T15:04:31.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.686+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fa0006d70 con 0x7f7fa8162b90 2024-01-16T15:04:31.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.687+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa8161f10 con 0x7f7fa8162b90 2024-01-16T15:04:31.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.687+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa8163410 con 0x7f7fa8162b90 2024-01-16T15:04:31.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.689+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa8162430 con 0x7f7fa8162b90 2024-01-16T15:04:31.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.693+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7f7fa0005070 con 0x7f7fa8162b90 2024-01-16T15:04:31.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.695+0000 7f7f9f400640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f8006b790 0x7f7f8006dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:31.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.700+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f7fa0058fe0 con 0x7f7fa8162b90 2024-01-16T15:04:31.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.704+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7fa0054c20 con 0x7f7fa8162b90 2024-01-16T15:04:31.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.714+0000 7f7fa7400640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f8006b790 0x7f7f8006dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:31.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:31.716+0000 7f7fa7400640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f8006b790 0x7f7f8006dc50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80695b0 tx=0x7f7f98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:32.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.102+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f7fa81f7c60 con 0x7f7f8006b790 2024-01-16T15:04:32.124 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:32.124 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:32.124 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:32.124 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:32.124 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:32.124 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:32.125 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.142+0000 7f7f9f400640 1 -- 172.21.15.53:0/869212090 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f7fa81f7c60 con 0x7f7f8006b790 2024-01-16T15:04:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.146+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f8006b790 msgr2=0x7f7f8006dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.146+0000 7f7fae200640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f8006b790 0x7f7f8006dc50 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80695b0 tx=0x7f7f98008040 comp rx=0 tx=0).stop 2024-01-16T15:04:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.147+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 msgr2=0x7f7fa80cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:32.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.147+0000 7f7fae200640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 0x7f7fa80cd720 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f7fa80cfe50 tx=0x7f7fa000dc00 comp rx=0 tx=0).stop 2024-01-16T15:04:32.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.148+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 shutdown_connections 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.149+0000 7f7fae200640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f8006b790 0x7f7f8006dc50 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.149+0000 7f7fae200640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7fa8162b90 0x7f7fa80cd720 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.149+0000 7f7fae200640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7fa8162710 0x7f7fa80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.149+0000 7f7fae200640 1 --2- 172.21.15.53:0/869212090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f880047c0 0x7f7fa8145a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.149+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 >> 172.21.15.53:0/869212090 conn(0x7f7fa806a800 msgr2=0x7f7fa80b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.149+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 shutdown_connections 2024-01-16T15:04:32.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:32.150+0000 7f7fae200640 1 -- 172.21.15.53:0/869212090 wait complete. 2024-01-16T15:04:32.169 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:32.534 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:32.545+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:32.534 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:32.545+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:32.534 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:32.545+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:32.534 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:32.545+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:32.534 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:32.545+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:33.171 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.457+0000 7f5f9ae00640 1 Processor -- start 2024-01-16T15:04:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.458+0000 7f5f9ae00640 1 -- start start 2024-01-16T15:04:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.460+0000 7f5f9ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f94143ae0 0x7f5f940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.460+0000 7f5f9ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f940cf7d0 0x7f5f9415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.460+0000 7f5f9ae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f940cfc30 con 0x7f5f9415e1e0 2024-01-16T15:04:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.460+0000 7f5f9ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f940b1ee0 con 0x7f5f94143ae0 2024-01-16T15:04:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.460+0000 7f5f9ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f940b2780 con 0x7f5f940cf7d0 2024-01-16T15:04:33.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.461+0000 7f5f93e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f94143ae0 0x7f5f940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.461+0000 7f5f93e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f94143ae0 0x7f5f940cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:55676/0 (socket says 172.21.15.53:55676) 2024-01-16T15:04:33.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.461+0000 7f5f93e00640 1 -- 172.21.15.53:0/917346759 learned_addr learned my addr 172.21.15.53:0/917346759 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:33.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.462+0000 7f5f93400640 1 --2- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f940cf7d0 0x7f5f9415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.463+0000 7f5f93e00640 1 -- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f940cf7d0 msgr2=0x7f5f9415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:33.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.463+0000 7f5f93e00640 1 --2- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f940cf7d0 0x7f5f9415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.463+0000 7f5f93e00640 1 -- 172.21.15.53:0/917346759 >> v1:172.21.15.146:6789/0 conn(0x7f5f9415e1e0 legacy=0x7f5f9415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:33.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.463+0000 7f5f93e00640 1 -- 172.21.15.53:0/917346759 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9415e990 con 0x7f5f94143ae0 2024-01-16T15:04:33.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.466+0000 7f5f93e00640 1 --2- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f94143ae0 0x7f5f940cf290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009870 tx=0x7f5f8402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=463e76ebc4ea3139 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:33.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.467+0000 7f5f92a00640 1 -- 172.21.15.53:0/917346759 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84031640 con 0x7f5f94143ae0 2024-01-16T15:04:33.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.467+0000 7f5f92a00640 1 -- 172.21.15.53:0/917346759 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f840317a0 con 0x7f5f94143ae0 2024-01-16T15:04:33.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.467+0000 7f5f92a00640 1 -- 172.21.15.53:0/917346759 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84031b30 con 0x7f5f94143ae0 2024-01-16T15:04:33.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.468+0000 7f5f9ae00640 1 -- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f94143ae0 msgr2=0x7f5f940cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.468+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f94143ae0 0x7f5f940cf290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009870 tx=0x7f5f8402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.470+0000 7f5f9ae00640 1 -- 172.21.15.53:0/917346759 shutdown_connections 2024-01-16T15:04:33.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.470+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f940cf7d0 0x7f5f9415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.470+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/917346759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f94143ae0 0x7f5f940cf290 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.470+0000 7f5f9ae00640 1 -- 172.21.15.53:0/917346759 >> 172.21.15.53:0/917346759 conn(0x7f5f9406a800 msgr2=0x7f5f9406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:33.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.470+0000 7f5f9ae00640 1 -- 172.21.15.53:0/917346759 shutdown_connections 2024-01-16T15:04:33.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.471+0000 7f5f9ae00640 1 -- 172.21.15.53:0/917346759 wait complete. 2024-01-16T15:04:33.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.475+0000 7f5f9ae00640 1 Processor -- start 2024-01-16T15:04:33.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.475+0000 7f5f9ae00640 1 -- start start 2024-01-16T15:04:33.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f9ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 0x7f5f94159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f9ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f94143ae0 0x7f5f94201690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f9ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f9415e1e0 0x7f5f94201bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f9ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f942021b0 con 0x7f5f940cf7d0 2024-01-16T15:04:33.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f9ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f94202490 con 0x7f5f9415e1e0 2024-01-16T15:04:33.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f9ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f94202770 con 0x7f5f94143ae0 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f93e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 0x7f5f94159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f93e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 0x7f5f94159220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44076/0 (socket says 172.21.15.53:44076) 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f93e00640 1 -- 172.21.15.53:0/138909268 learned_addr learned my addr 172.21.15.53:0/138909268 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.477+0000 7f5f93400640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f94143ae0 0x7f5f94201690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.478+0000 7f5f93e00640 1 -- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f94143ae0 msgr2=0x7f5f94201690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.478+0000 7f5f93e00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f94143ae0 0x7f5f94201690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.478+0000 7f5f93e00640 1 -- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f9415e1e0 msgr2=0x7f5f94201bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.478+0000 7f5f93e00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f9415e1e0 0x7f5f94201bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.478+0000 7f5f93e00640 1 -- 172.21.15.53:0/138909268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f840094d0 con 0x7f5f940cf7d0 2024-01-16T15:04:33.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.479+0000 7f5f93e00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 0x7f5f94159220 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009870 tx=0x7f5f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:33.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.479+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8403a930 con 0x7f5f940cf7d0 2024-01-16T15:04:33.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.479+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8403aa90 con 0x7f5f940cf7d0 2024-01-16T15:04:33.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.479+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8403ae40 con 0x7f5f940cf7d0 2024-01-16T15:04:33.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.480+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f94202a50 con 0x7f5f940cf7d0 2024-01-16T15:04:33.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.480+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f94072420 con 0x7f5f940cf7d0 2024-01-16T15:04:33.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.481+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f94202f70 con 0x7f5f940cf7d0 2024-01-16T15:04:33.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.483+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7f5f8402f050 con 0x7f5f940cf7d0 2024-01-16T15:04:33.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.484+0000 7f5f8be00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f6c06b740 0x7f5f6c06dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:33.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.484+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f5f840ac500 con 0x7f5f940cf7d0 2024-01-16T15:04:33.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.492+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f94202f70 con 0x7f5f940cf7d0 2024-01-16T15:04:33.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.492+0000 7f5f93400640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f6c06b740 0x7f5f6c06dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:33.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.495+0000 7f5f93400640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f6c06b740 0x7f5f6c06dc00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5f9406b2f0 tx=0x7f5f7c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:33.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.905+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f9415e640 con 0x7f5f6c06b740 2024-01-16T15:04:33.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.973+0000 7f5f8be00640 1 -- 172.21.15.53:0/138909268 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f5f7c009820 con 0x7f5f6c06b740 2024-01-16T15:04:33.952 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:33.952 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:33.952 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:33.952 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:33.952 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:33.952 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:33.953 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:33.954 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f6c06b740 msgr2=0x7f5f6c06dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f6c06b740 0x7f5f6c06dc00 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f5f9406b2f0 tx=0x7f5f7c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 msgr2=0x7f5f94159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 0x7f5f94159220 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f5f84009870 tx=0x7f5f840046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 shutdown_connections 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f6c06b740 0x7f5f6c06dc00 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f9415e1e0 0x7f5f94201bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f94143ae0 0x7f5f94201690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 --2- 172.21.15.53:0/138909268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f940cf7d0 0x7f5f94159220 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:33.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.975+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 >> 172.21.15.53:0/138909268 conn(0x7f5f9406a800 msgr2=0x7f5f940702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:33.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.976+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 shutdown_connections 2024-01-16T15:04:33.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:33.976+0000 7f5f9ae00640 1 -- 172.21.15.53:0/138909268 wait complete. 2024-01-16T15:04:33.979 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:34.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:35.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.236+0000 7fa98ca00640 1 Processor -- start 2024-01-16T15:04:35.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.236+0000 7fa98ca00640 1 -- start start 2024-01-16T15:04:35.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa98ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9880adf20 0x7fa988070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa98ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa98ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa98815f4e0 con 0x7fa9880cc980 2024-01-16T15:04:35.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa98ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa98815f970 con 0x7fa9880adf20 2024-01-16T15:04:35.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa98ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880cce10 con 0x7fa98815bf10 2024-01-16T15:04:35.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa986800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9880adf20 0x7fa988070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.238+0000 7fa97dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.239+0000 7fa986800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa9880adf20 0x7fa988070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:55694/0 (socket says 172.21.15.53:55694) 2024-01-16T15:04:35.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.239+0000 7fa986800640 1 -- 172.21.15.53:0/2976941691 learned_addr learned my addr 172.21.15.53:0/2976941691 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:35.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.239+0000 7fa986800640 1 -- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 msgr2=0x7fa9880cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:35.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.239+0000 7fa986800640 1 --2- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.239+0000 7fa986800640 1 -- 172.21.15.53:0/2976941691 >> v1:172.21.15.146:6789/0 conn(0x7fa9880cc980 legacy=0x7fa988143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.239+0000 7fa986800640 1 -- 172.21.15.53:0/2976941691 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa988071130 con 0x7fa9880adf20 2024-01-16T15:04:35.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.241+0000 7fa986800640 1 --2- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880adf20 0x7fa988070a30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa978009e50 tx=0x7fa97802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e88bc4dcbe3c01d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.241+0000 7fa985e00640 1 -- 172.21.15.53:0/2976941691 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97802ca70 con 0x7fa9880adf20 2024-01-16T15:04:35.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.241+0000 7fa985e00640 1 -- 172.21.15.53:0/2976941691 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97802cbd0 con 0x7fa9880adf20 2024-01-16T15:04:35.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.241+0000 7fa985e00640 1 -- 172.21.15.53:0/2976941691 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9780343f0 con 0x7fa9880adf20 2024-01-16T15:04:35.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 -- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880adf20 msgr2=0x7fa988070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880adf20 0x7fa988070a30 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa978009e50 tx=0x7fa97802bed0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 -- 172.21.15.53:0/2976941691 shutdown_connections 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2976941691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880adf20 0x7fa988070a30 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 -- 172.21.15.53:0/2976941691 >> 172.21.15.53:0/2976941691 conn(0x7fa98806a7d0 msgr2=0x7fa98806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 -- 172.21.15.53:0/2976941691 shutdown_connections 2024-01-16T15:04:35.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.242+0000 7fa98ca00640 1 -- 172.21.15.53:0/2976941691 wait complete. 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.244+0000 7fa98ca00640 1 Processor -- start 2024-01-16T15:04:35.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.244+0000 7fa98ca00640 1 -- start start 2024-01-16T15:04:35.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa98ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 0x7fa9880b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa98ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880cc980 0x7fa988157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa98ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa98ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880ac0d0 con 0x7fa9880adf20 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa98ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880b36b0 con 0x7fa9880cc980 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa98ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9880b2450 con 0x7fa98815bf10 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa986800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 0x7fa9880b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa986800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 0x7fa9880b1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44078/0 (socket says 172.21.15.53:44078) 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.245+0000 7fa986800640 1 -- 172.21.15.53:0/2828377880 learned_addr learned my addr 172.21.15.53:0/2828377880 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa97dc00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880cc980 0x7fa988157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa987200640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa986800640 1 -- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 msgr2=0x7fa9880b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa986800640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa986800640 1 -- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880cc980 msgr2=0x7fa988157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa986800640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880cc980 0x7fa988157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa986800640 1 -- 172.21.15.53:0/2828377880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa978009b30 con 0x7fa9880adf20 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa987200640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa986800640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 0x7fa9880b1e70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa978009e50 tx=0x7fa978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa978009510 con 0x7fa9880adf20 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa97803e450 con 0x7fa9880adf20 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa97803e800 con 0x7fa9880adf20 2024-01-16T15:04:35.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.246+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9880b2730 con 0x7fa9880adf20 2024-01-16T15:04:35.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.247+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9880b2cf0 con 0x7fa9880adf20 2024-01-16T15:04:35.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.248+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa988067450 con 0x7fa9880adf20 2024-01-16T15:04:35.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.249+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 12) v1 ==== 89947+0+0 (secure 0 0 0) 0x7fa97803e960 con 0x7fa9880adf20 2024-01-16T15:04:35.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.249+0000 7fa97f400640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa96006b740 0x7fa96006dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:35.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.249+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fa9780ad200 con 0x7fa9880adf20 2024-01-16T15:04:35.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.250+0000 7fa97dc00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa96006b740 0x7fa96006dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:35.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.251+0000 7fa97dc00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa96006b740 0x7fa96006dc00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9880a9a40 tx=0x7fa970008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:35.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.254+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa978077930 con 0x7fa9880adf20 2024-01-16T15:04:35.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.274+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fa978077d00 con 0x7fa9880adf20 2024-01-16T15:04:35.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.484+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7fa98815f170 con 0x7fa96006b740 2024-01-16T15:04:35.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.498+0000 7fa97f400640 1 -- 172.21.15.53:0/2828377880 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fa970009800 con 0x7fa96006b740 2024-01-16T15:04:35.476 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:35.477 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:35.478 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:35.478 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:35.478 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:35.478 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:35.478 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa96006b740 msgr2=0x7fa96006dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa96006b740 0x7fa96006dc00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fa9880a9a40 tx=0x7fa970008040 comp rx=0 tx=0).stop 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 msgr2=0x7fa9880b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 0x7fa9880b1e70 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7fa978009e50 tx=0x7fa978004a40 comp rx=0 tx=0).stop 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 shutdown_connections 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa96006b740 0x7fa96006dc00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa98815bf10 0x7fa9880b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa9880cc980 0x7fa988157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 --2- 172.21.15.53:0/2828377880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa9880adf20 0x7fa9880b1e70 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.500+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 >> 172.21.15.53:0/2828377880 conn(0x7fa98806a7d0 msgr2=0x7fa9880cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.501+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 shutdown_connections 2024-01-16T15:04:35.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:35.501+0000 7fa98ca00640 1 -- 172.21.15.53:0/2828377880 wait complete. 2024-01-16T15:04:35.497 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:36.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:36.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.838+0000 7fb672200640 1 Processor -- start 2024-01-16T15:04:36.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.838+0000 7fb672200640 1 -- start start 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.840+0000 7fb672200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb66c15d4b0 0x7fb66c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.840+0000 7fb672200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c0b2730 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.840+0000 7fb672200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb66c0d1ec0 con 0x7fb66c0b41e0 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.840+0000 7fb672200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb66c0b29d0 con 0x7fb66c15d4b0 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.841+0000 7fb66be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb66c0cf9a0 0x7fb66c0cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:45162/0 (socket says 172.21.15.53:45162) 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.841+0000 7fb66be00640 1 -- 172.21.15.53:0/1532704908 learned_addr learned my addr 172.21.15.53:0/1532704908 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:36.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.841+0000 7fb670e00640 1 --2- 172.21.15.53:0/1532704908 >> v2:172.21.15.53:3300/0 conn(0x7fb66c15d4b0 0x7fb66c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.841+0000 7fb66b400640 1 -- 172.21.15.53:0/1532704908 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641629600 0 0) 0x7fb66c0b2730 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.842+0000 7fb66b400640 1 -- 172.21.15.53:0/1532704908 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb64c003520 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.842+0000 7fb670e00640 1 -- 172.21.15.53:0/1532704908 >> v1:172.21.15.146:6789/0 conn(0x7fb66c0b41e0 legacy=0x7fb66c0ce500 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:04:36.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.842+0000 7fb670e00640 1 -- 172.21.15.53:0/1532704908 >> v1:172.21.15.53:6789/0 conn(0x7fb66c0cf9a0 legacy=0x7fb66c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.842+0000 7fb670e00640 1 -- 172.21.15.53:0/1532704908 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c0b2cb0 con 0x7fb66c15d4b0 2024-01-16T15:04:36.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.844+0000 7fb670e00640 1 --2- 172.21.15.53:0/1532704908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c15d4b0 0x7fb66c0d1550 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb6540097a0 tx=0x7fb65402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fbedc33d7f301e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.844+0000 7fb66b400640 1 -- 172.21.15.53:0/1532704908 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654031830 con 0x7fb66c15d4b0 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.844+0000 7fb66b400640 1 -- 172.21.15.53:0/1532704908 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb654031990 con 0x7fb66c15d4b0 2024-01-16T15:04:36.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.844+0000 7fb66b400640 1 -- 172.21.15.53:0/1532704908 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654031d20 con 0x7fb66c15d4b0 2024-01-16T15:04:36.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.846+0000 7fb672200640 1 -- 172.21.15.53:0/1532704908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c15d4b0 msgr2=0x7fb66c0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.846+0000 7fb672200640 1 --2- 172.21.15.53:0/1532704908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c15d4b0 0x7fb66c0d1550 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fb6540097a0 tx=0x7fb65402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.846+0000 7fb672200640 1 -- 172.21.15.53:0/1532704908 shutdown_connections 2024-01-16T15:04:36.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.846+0000 7fb672200640 1 --2- 172.21.15.53:0/1532704908 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c15d4b0 0x7fb66c0d1550 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.846+0000 7fb672200640 1 -- 172.21.15.53:0/1532704908 >> 172.21.15.53:0/1532704908 conn(0x7fb66c0d3000 msgr2=0x7fb66c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:36.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.847+0000 7fb672200640 1 -- 172.21.15.53:0/1532704908 shutdown_connections 2024-01-16T15:04:36.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.847+0000 7fb672200640 1 -- 172.21.15.53:0/1532704908 wait complete. 2024-01-16T15:04:36.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.853+0000 7fb672200640 1 Processor -- start 2024-01-16T15:04:36.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.856+0000 7fb672200640 1 -- start start 2024-01-16T15:04:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.857+0000 7fb672200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb66c0b41e0 0x7fb66c0b8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.857+0000 7fb672200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 0x7fb66c0cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.857+0000 7fb672200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb66c15d4b0 0x7fb66c14ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.857+0000 7fb672200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c1f2ee0 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.857+0000 7fb672200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c0cd440 con 0x7fb66c15d4b0 2024-01-16T15:04:36.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.857+0000 7fb672200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66c09f6a0 con 0x7fb66c0b41e0 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.858+0000 7fb66be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 0x7fb66c0cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.858+0000 7fb66be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 0x7fb66c0cbf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44102/0 (socket says 172.21.15.53:44102) 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.858+0000 7fb66be00640 1 -- 172.21.15.53:0/2707492090 learned_addr learned my addr 172.21.15.53:0/2707492090 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.858+0000 7fb671800640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb66c15d4b0 0x7fb66c14ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.858+0000 7fb670e00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb66c0b41e0 0x7fb66c0b8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.859+0000 7fb66be00640 1 -- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb66c0b41e0 msgr2=0x7fb66c0b8440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.859+0000 7fb66be00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb66c0b41e0 0x7fb66c0b8440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.859+0000 7fb66be00640 1 -- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb66c15d4b0 msgr2=0x7fb66c14ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:36.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.859+0000 7fb66be00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb66c15d4b0 0x7fb66c14ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:36.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.859+0000 7fb66be00640 1 -- 172.21.15.53:0/2707492090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb654009400 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.859+0000 7fb66be00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 0x7fb66c0cbf00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00c430 tx=0x7fb65c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:36.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.860+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c005ce0 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.860+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb65c005e40 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.860+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65c0072f0 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.860+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb66c15da10 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.860+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c0cce80 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.862+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb66c09f860 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.864+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fb65c007450 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.867+0000 7fb668c00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb64406b7d0 0x7fb64406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:36.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.867+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fb65c0899e0 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.872+0000 7fb670e00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb64406b7d0 0x7fb64406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:36.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.874+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb65c054230 con 0x7fb66c0cf9a0 2024-01-16T15:04:36.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:36.877+0000 7fb670e00640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb64406b7d0 0x7fb64406dc90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb654036000 tx=0x7fb65403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:37.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.248+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7fb66c15d890 con 0x7fb64406b7d0 2024-01-16T15:04:37.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.356+0000 7fb668c00640 1 -- 172.21.15.53:0/2707492090 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fb654031b00 con 0x7fb64406b7d0 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:37.336 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:37.337 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:37.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb64406b7d0 msgr2=0x7fb64406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb64406b7d0 0x7fb64406dc90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb654036000 tx=0x7fb65403c040 comp rx=0 tx=0).stop 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 msgr2=0x7fb66c0cbf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 0x7fb66c0cbf00 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fb65c00c430 tx=0x7fb65c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 shutdown_connections 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb64406b7d0 0x7fb64406dc90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb66c15d4b0 0x7fb66c14ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb66c0cf9a0 0x7fb66c0cbf00 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 --2- 172.21.15.53:0/2707492090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb66c0b41e0 0x7fb66c0b8440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 >> 172.21.15.53:0/2707492090 conn(0x7fb66c0d3000 msgr2=0x7fb66c0cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 shutdown_connections 2024-01-16T15:04:37.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:37.369+0000 7fb672200640 1 -- 172.21.15.53:0/2707492090 wait complete. 2024-01-16T15:04:37.385 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:38.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:38.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.706+0000 7f6e57200640 1 Processor -- start 2024-01-16T15:04:38.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.706+0000 7f6e57200640 1 -- start start 2024-01-16T15:04:38.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.708+0000 7f6e57200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6e500cd170 0x7f6e50071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.708+0000 7f6e57200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e50071690 con 0x7f6e500cf9a0 2024-01-16T15:04:38.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.708+0000 7f6e57200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e50071970 con 0x7f6e5014b2c0 2024-01-16T15:04:38.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.708+0000 7f6e57200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e5014b6a0 con 0x7f6e500cd170 2024-01-16T15:04:38.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.709+0000 7f6e4f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6e500cf9a0 0x7f6e50070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:45174/0 (socket says 172.21.15.53:45174) 2024-01-16T15:04:38.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.709+0000 7f6e4f400640 1 -- 172.21.15.53:0/13281299 learned_addr learned my addr 172.21.15.53:0/13281299 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:38.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.710+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840659375 0 0) 0x7f6e50071690 con 0x7f6e500cf9a0 2024-01-16T15:04:38.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.710+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e30003520 con 0x7f6e500cf9a0 2024-01-16T15:04:38.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e55200640 1 --2- 172.21.15.53:0/13281299 >> v2:172.21.15.146:3300/0 conn(0x7f6e500cd170 0x7f6e50071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520108199 0 0) 0x7f6e30003520 con 0x7f6e500cf9a0 2024-01-16T15:04:38.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 >> v2:172.21.15.146:3300/0 conn(0x7f6e500cd170 msgr2=0x7f6e50071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:38.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 --2- 172.21.15.53:0/13281299 >> v2:172.21.15.146:3300/0 conn(0x7f6e500cd170 0x7f6e50071070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 >> v1:172.21.15.146:6789/0 conn(0x7f6e5014b2c0 legacy=0x7f6e500cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:38.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e50071c50 con 0x7f6e500cf9a0 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6e38001670 con 0x7f6e500cf9a0 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 >> v1:172.21.15.53:6789/0 conn(0x7f6e500cf9a0 legacy=0x7f6e50070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e30003970 0x7f6e30003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30005180 0x7f6e3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5014b6a0 con 0x7f6e300047c0 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3000c0b0 con 0x7f6e30005180 2024-01-16T15:04:38.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.711+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3000c270 con 0x7f6e30003970 2024-01-16T15:04:38.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.713+0000 7f6e4f400640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.714+0000 7f6e55200640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30005180 0x7f6e3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.714+0000 7f6e4f400640 1 -- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e30003970 msgr2=0x7f6e30003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:38.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.714+0000 7f6e4f400640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e30003970 0x7f6e30003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.714+0000 7f6e4f400640 1 -- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30005180 msgr2=0x7f6e3000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:38.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.715+0000 7f6e4f400640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e30005180 0x7f6e3000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.715+0000 7f6e4f400640 1 -- 172.21.15.53:0/13281299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e50071c50 con 0x7f6e300047c0 2024-01-16T15:04:38.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.716+0000 7f6e55200640 1 -- 172.21.15.53:0/13281299 reap_dead start 2024-01-16T15:04:38.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.719+0000 7f6e4f400640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c260 tx=0x7f6e300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d9ced01fca64142 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:38.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.720+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e38005ad0 con 0x7f6e300047c0 2024-01-16T15:04:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.720+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e38005c30 con 0x7f6e300047c0 2024-01-16T15:04:38.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.720+0000 7f6e4ea00640 1 -- 172.21.15.53:0/13281299 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e3802c8d0 con 0x7f6e300047c0 2024-01-16T15:04:38.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.723+0000 7f6e57200640 1 -- 172.21.15.53:0/13281299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e300047c0 msgr2=0x7f6e30004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.723+0000 7f6e57200640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c260 tx=0x7f6e300057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.723+0000 7f6e57200640 1 -- 172.21.15.53:0/13281299 shutdown_connections 2024-01-16T15:04:38.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.723+0000 7f6e57200640 1 --2- 172.21.15.53:0/13281299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e300047c0 0x7f6e30004c40 secure :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f6e3800c260 tx=0x7f6e300057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.723+0000 7f6e57200640 1 -- 172.21.15.53:0/13281299 >> 172.21.15.53:0/13281299 conn(0x7f6e5006a800 msgr2=0x7f6e5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:38.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.725+0000 7f6e57200640 1 -- 172.21.15.53:0/13281299 shutdown_connections 2024-01-16T15:04:38.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.725+0000 7f6e57200640 1 -- 172.21.15.53:0/13281299 wait complete. 2024-01-16T15:04:38.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.729+0000 7f6e57200640 1 Processor -- start 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.735+0000 7f6e57200640 1 -- start start 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.737+0000 7f6e57200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 0x7f6e50162790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.737+0000 7f6e57200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e501fa2f0 0x7f6e501447b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.737+0000 7f6e57200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e50144cf0 0x7f6e50142e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.738+0000 7f6e57200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50145180 con 0x7f6e501f9f10 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.738+0000 7f6e57200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e50143e80 con 0x7f6e50144cf0 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.738+0000 7f6e57200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5013fff0 con 0x7f6e501fa2f0 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.738+0000 7f6e4fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 0x7f6e50162790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.738+0000 7f6e4fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 0x7f6e50162790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44120/0 (socket says 172.21.15.53:44120) 2024-01-16T15:04:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.738+0000 7f6e4fe00640 1 -- 172.21.15.53:0/1113972100 learned_addr learned my addr 172.21.15.53:0/1113972100 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.739+0000 7f6e4f400640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e501fa2f0 0x7f6e501447b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.739+0000 7f6e55200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e50144cf0 0x7f6e50142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.740+0000 7f6e4fe00640 1 -- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e501fa2f0 msgr2=0x7f6e501447b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.740+0000 7f6e4fe00640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e501fa2f0 0x7f6e501447b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.740+0000 7f6e4fe00640 1 -- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e50144cf0 msgr2=0x7f6e50142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.740+0000 7f6e4fe00640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e50144cf0 0x7f6e50142e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.740+0000 7f6e4fe00640 1 -- 172.21.15.53:0/1113972100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5013f9e0 con 0x7f6e501f9f10 2024-01-16T15:04:38.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.741+0000 7f6e4fe00640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 0x7f6e50162790 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c3f0 tx=0x7f6e4000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:38.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.748+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40011790 con 0x7f6e501f9f10 2024-01-16T15:04:38.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.748+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e400118f0 con 0x7f6e501f9f10 2024-01-16T15:04:38.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.748+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e40011ca0 con 0x7f6e501f9f10 2024-01-16T15:04:38.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.748+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e50143ba0 con 0x7f6e501f9f10 2024-01-16T15:04:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.754+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e50143370 con 0x7f6e501f9f10 2024-01-16T15:04:38.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.756+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5013f840 con 0x7f6e501f9f10 2024-01-16T15:04:38.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.759+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f6e4001c460 con 0x7f6e501f9f10 2024-01-16T15:04:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.760+0000 7f6e47e00640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e2806b800 0x7f6e2806dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.762+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f6e4008d070 con 0x7f6e501f9f10 2024-01-16T15:04:38.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.766+0000 7f6e4f400640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e2806b800 0x7f6e2806dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:38.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.768+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e5013f840 con 0x7f6e501f9f10 2024-01-16T15:04:38.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:38.798+0000 7f6e4f400640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e2806b800 0x7f6e2806dcc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6e38034000 tx=0x7f6e38009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:39.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.146+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e500cbc40 con 0x7f6e2806b800 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.270+0000 7f6e47e00640 1 -- 172.21.15.53:0/1113972100 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f6e3803a040 con 0x7f6e2806b800 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:39.249 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:39.250 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:39.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e2806b800 msgr2=0x7f6e2806dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e2806b800 0x7f6e2806dcc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6e38034000 tx=0x7f6e38009030 comp rx=0 tx=0).stop 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 msgr2=0x7f6e50162790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 0x7f6e50162790 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c3f0 tx=0x7f6e4000c7f0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 shutdown_connections 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e2806b800 0x7f6e2806dcc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e50144cf0 0x7f6e50142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e501fa2f0 0x7f6e501447b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 --2- 172.21.15.53:0/1113972100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e501f9f10 0x7f6e50162790 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:39.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 >> 172.21.15.53:0/1113972100 conn(0x7f6e5006a800 msgr2=0x7f6e5006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:39.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 shutdown_connections 2024-01-16T15:04:39.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:39.273+0000 7f6e57200640 1 -- 172.21.15.53:0/1113972100 wait complete. 2024-01-16T15:04:39.276 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:40.277 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.606+0000 7fcd06200640 1 Processor -- start 2024-01-16T15:04:40.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.606+0000 7fcd06200640 1 -- start start 2024-01-16T15:04:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.608+0000 7fcd06200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd00161d30 0x7fcd000b3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.609+0000 7fcd06200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd00161b90 con 0x7fcd000b3a50 2024-01-16T15:04:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.609+0000 7fcd06200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd00070840 con 0x7fcd0006d3b0 2024-01-16T15:04:40.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.609+0000 7fcd06200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd000702c0 con 0x7fcd00161d30 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.610+0000 7fccff400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd00161d30 0x7fcd000b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.610+0000 7fccff400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd00161d30 0x7fcd000b3510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:55758/0 (socket says 172.21.15.53:55758) 2024-01-16T15:04:40.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.610+0000 7fccff400640 1 -- 172.21.15.53:0/3083267207 learned_addr learned my addr 172.21.15.53:0/3083267207 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.612+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169562 0 0) 0x7fccf8001490 con 0x7fcd000b3a50 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.612+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce0003520 con 0x7fcd000b3a50 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.612+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 389879120 0 0) 0x7fcce0003520 con 0x7fcd000b3a50 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.612+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 >> v2:172.21.15.146:3300/0 conn(0x7fcd00161d30 msgr2=0x7fcd000b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.612+0000 7fccfe000640 1 --2- 172.21.15.53:0/3083267207 >> v2:172.21.15.146:3300/0 conn(0x7fcd00161d30 0x7fcd000b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 >> v1:172.21.15.146:6789/0 conn(0x7fcd0006d3b0 legacy=0x7fcd0016b2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd00162e80 con 0x7fcd000b3a50 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fccf8001b80 con 0x7fcd000b3a50 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 >> v1:172.21.15.53:6789/0 conn(0x7fcd000b3a50 legacy=0x7fcd00162110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce0003970 0x7fcce0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce00047c0 0x7fcce0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcce000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd000702c0 con 0x7fcce0005180 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcce000c000 con 0x7fcce00047c0 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.613+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce000c1c0 con 0x7fcce0003970 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.614+0000 7fccffe00640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcce000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.615+0000 7fccffe00640 1 -- 172.21.15.53:0/3083267207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce0003970 msgr2=0x7fcce0003d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.615+0000 7fccffe00640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcce0003970 0x7fcce0003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.615+0000 7fccffe00640 1 -- 172.21.15.53:0/3083267207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce00047c0 msgr2=0x7fcce0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.615+0000 7fccffe00640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcce00047c0 0x7fcce0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.615+0000 7fccffe00640 1 -- 172.21.15.53:0/3083267207 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd00162e80 con 0x7fcce0005180 2024-01-16T15:04:40.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.621+0000 7fccffe00640 1 -- 172.21.15.53:0/3083267207 reap_dead start 2024-01-16T15:04:40.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.622+0000 7fccffe00640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcce000b980 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcce8001650 tx=0x7fcce00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f59df4e4d407c79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.623+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce8005270 con 0x7fcce0005180 2024-01-16T15:04:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.623+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcce802e070 con 0x7fcce0005180 2024-01-16T15:04:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.624+0000 7fcd06200640 1 -- 172.21.15.53:0/3083267207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 msgr2=0x7fcce000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.624+0000 7fccfe000640 1 -- 172.21.15.53:0/3083267207 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcce8005270 con 0x7fcce0005180 2024-01-16T15:04:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.624+0000 7fcd06200640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcce000b980 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcce8001650 tx=0x7fcce00057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.625+0000 7fcd06200640 1 -- 172.21.15.53:0/3083267207 shutdown_connections 2024-01-16T15:04:40.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.625+0000 7fcd06200640 1 --2- 172.21.15.53:0/3083267207 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcce000b980 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.625+0000 7fcd06200640 1 -- 172.21.15.53:0/3083267207 >> 172.21.15.53:0/3083267207 conn(0x7fcd000b0be0 msgr2=0x7fcd000b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:40.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.627+0000 7fcd06200640 1 -- 172.21.15.53:0/3083267207 shutdown_connections 2024-01-16T15:04:40.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.627+0000 7fcd06200640 1 -- 172.21.15.53:0/3083267207 wait complete. 2024-01-16T15:04:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.632+0000 7fcd06200640 1 Processor -- start 2024-01-16T15:04:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.633+0000 7fcd06200640 1 -- start start 2024-01-16T15:04:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.639+0000 7fcd06200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcd001fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.639+0000 7fcd06200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd001591b0 0x7fcd0015a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.639+0000 7fcd06200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0015c250 0x7fcd00202ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.639+0000 7fcd06200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015b1b0 con 0x7fcce0005180 2024-01-16T15:04:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.639+0000 7fcd06200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015b490 con 0x7fcd001591b0 2024-01-16T15:04:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.639+0000 7fcd06200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd0015b770 con 0x7fcd0015c250 2024-01-16T15:04:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.640+0000 7fccff400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcd001fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.640+0000 7fccff400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcd001fcc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44140/0 (socket says 172.21.15.53:44140) 2024-01-16T15:04:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.640+0000 7fccff400640 1 -- 172.21.15.53:0/3870972398 learned_addr learned my addr 172.21.15.53:0/3870972398 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.641+0000 7fccffe00640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0015c250 0x7fcd00202ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.642+0000 7fccff400640 1 -- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0015c250 msgr2=0x7fcd00202ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.642+0000 7fccfea00640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd001591b0 0x7fcd0015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.643+0000 7fccff400640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0015c250 0x7fcd00202ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.643+0000 7fccff400640 1 -- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd001591b0 msgr2=0x7fcd0015a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:40.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.643+0000 7fccff400640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd001591b0 0x7fcd0015a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:40.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.643+0000 7fccff400640 1 -- 172.21.15.53:0/3870972398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0015ba50 con 0x7fcce0005180 2024-01-16T15:04:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.643+0000 7fccfea00640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd001591b0 0x7fcd0015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:40.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.649+0000 7fccff400640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcd001fcc30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fccf000e7f0 tx=0x7fccf000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:40.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.653+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000c940 con 0x7fcce0005180 2024-01-16T15:04:40.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.653+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccf000caa0 con 0x7fcce0005180 2024-01-16T15:04:40.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.653+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccf000ce50 con 0x7fcce0005180 2024-01-16T15:04:40.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.653+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd0015bd30 con 0x7fcce0005180 2024-01-16T15:04:40.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.653+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd00203220 con 0x7fcce0005180 2024-01-16T15:04:40.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.656+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd0015bec0 con 0x7fcce0005180 2024-01-16T15:04:40.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.657+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fccf0013470 con 0x7fcce0005180 2024-01-16T15:04:40.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.660+0000 7fccf7400640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fccd806b850 0x7fccd806dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:40.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.661+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fccf008a360 con 0x7fcce0005180 2024-01-16T15:04:40.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.665+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccf0054cd0 con 0x7fcce0005180 2024-01-16T15:04:40.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.693+0000 7fccfea00640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fccd806b850 0x7fccd806dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:40.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:40.777+0000 7fccfea00640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fccd806b850 0x7fccd806dd10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcd001595d0 tx=0x7fccf8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:41.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.043+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd0015b600 con 0x7fccd806b850 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.126+0000 7fccf7400640 1 -- 172.21.15.53:0/3870972398 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fcd0015b600 con 0x7fccd806b850 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:41.105 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:41.106 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:41.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fccd806b850 msgr2=0x7fccd806dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fccd806b850 0x7fccd806dd10 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fcd001595d0 tx=0x7fccf8009040 comp rx=0 tx=0).stop 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 msgr2=0x7fcd001fcc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcd001fcc30 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fccf000e7f0 tx=0x7fccf000ecb0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 shutdown_connections 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fccd806b850 0x7fccd806dd10 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd0015c250 0x7fcd00202ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd001591b0 0x7fcd0015a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 --2- 172.21.15.53:0/3870972398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcce0005180 0x7fcd001fcc30 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.130+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 >> 172.21.15.53:0/3870972398 conn(0x7fcd000b0be0 msgr2=0x7fcd0015f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.131+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 shutdown_connections 2024-01-16T15:04:41.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:41.131+0000 7fcd06200640 1 -- 172.21.15.53:0/3870972398 wait complete. 2024-01-16T15:04:41.141 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:42.142 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48371 2024-01-16T15:04:42.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.431+0000 7fcbbae00640 1 Processor -- start 2024-01-16T15:04:42.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.431+0000 7fcbbae00640 1 -- start start 2024-01-16T15:04:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.432+0000 7fcbbae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcbb40cd240 0x7fcbb40b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.432+0000 7fcbbae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcbb409a170 0x7fcbb40cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.432+0000 7fcbbae00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb406cea0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.432+0000 7fcbbae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb406cbc0 con 0x7fcbb409a170 2024-01-16T15:04:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.432+0000 7fcbbae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb406f350 con 0x7fcbb40cd240 2024-01-16T15:04:42.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcbb409a170 0x7fcbb40cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcbb40cd240 0x7fcbb40b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcbb409a170 0x7fcbb40cf640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42124/0 (socket says 172.21.15.53:42124) 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 -- 172.21.15.53:0/961528318 learned_addr learned my addr 172.21.15.53:0/961528318 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 -- 172.21.15.53:0/961528318 >> v2:172.21.15.146:3300/0 conn(0x7fcbb40cd240 msgr2=0x7fcbb40b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 --2- 172.21.15.53:0/961528318 >> v2:172.21.15.146:3300/0 conn(0x7fcbb40cd240 0x7fcbb40b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 -- 172.21.15.53:0/961528318 >> v1:172.21.15.53:6789/0 conn(0x7fcbb406c6b0 legacy=0x7fcbb4099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb8e00640 1 -- 172.21.15.53:0/961528318 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb406f630 con 0x7fcbb409a170 2024-01-16T15:04:42.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.433+0000 7fcbb3e00640 1 --2- 172.21.15.53:0/961528318 >> v2:172.21.15.146:3300/0 conn(0x7fcbb40cd240 0x7fcbb40b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:04:42.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.435+0000 7fcbb8e00640 1 --2- 172.21.15.53:0/961528318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb40cf640 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4070a10 tx=0x7fcbac02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59860364f47b0c9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.435+0000 7fcbb2a00640 1 -- 172.21.15.53:0/961528318 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac02cb20 con 0x7fcbb409a170 2024-01-16T15:04:42.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.435+0000 7fcbb2a00640 1 -- 172.21.15.53:0/961528318 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbac02cc80 con 0x7fcbb409a170 2024-01-16T15:04:42.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.435+0000 7fcbb2a00640 1 -- 172.21.15.53:0/961528318 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbac0343f0 con 0x7fcbb409a170 2024-01-16T15:04:42.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.435+0000 7fcbbae00640 1 -- 172.21.15.53:0/961528318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 msgr2=0x7fcbb40cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.435+0000 7fcbbae00640 1 --2- 172.21.15.53:0/961528318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb40cf640 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4070a10 tx=0x7fcbac02bf80 comp rx=0 tx=0).stop 2024-01-16T15:04:42.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.436+0000 7fcbbae00640 1 -- 172.21.15.53:0/961528318 shutdown_connections 2024-01-16T15:04:42.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.436+0000 7fcbbae00640 1 --2- 172.21.15.53:0/961528318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb40cf640 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.436+0000 7fcbbae00640 1 --2- 172.21.15.53:0/961528318 >> v2:172.21.15.146:3300/0 conn(0x7fcbb40cd240 0x7fcbb40b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.436+0000 7fcbbae00640 1 -- 172.21.15.53:0/961528318 >> 172.21.15.53:0/961528318 conn(0x7fcbb40b2d00 msgr2=0x7fcbb40b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:42.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.436+0000 7fcbbae00640 1 -- 172.21.15.53:0/961528318 shutdown_connections 2024-01-16T15:04:42.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.436+0000 7fcbbae00640 1 -- 172.21.15.53:0/961528318 wait complete. 2024-01-16T15:04:42.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.438+0000 7fcbbae00640 1 Processor -- start 2024-01-16T15:04:42.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.438+0000 7fcbbae00640 1 -- start start 2024-01-16T15:04:42.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 0x7fcbb414f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb4151ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbbae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbb40cd240 0x7fcbb4154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbbae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40ca2f0 con 0x7fcbb409a170 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbbae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40c65b0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbbae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb40c9ca0 con 0x7fcbb40cd240 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbb3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb4151ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbb3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 0x7fcbb414f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbb3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 0x7fcbb414f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:51120/0 (socket says 172.21.15.53:51120) 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.439+0000 7fcbb3e00640 1 -- 172.21.15.53:0/2074437316 learned_addr learned my addr 172.21.15.53:0/2074437316 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb8e00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbb40cd240 0x7fcbb4154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb3e00640 1 -- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbb40cd240 msgr2=0x7fcbb4154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb3e00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbb40cd240 0x7fcbb4154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb3e00640 1 -- 172.21.15.53:0/2074437316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 msgr2=0x7fcbb4151ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb3e00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb4151ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb3e00640 1 -- 172.21.15.53:0/2074437316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba4009400 con 0x7fcbb406c6b0 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbb3e00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 0x7fcbb414f7d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcba400ebe0 tx=0x7fcba400c520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.440+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4009760 con 0x7fcbb406c6b0 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.441+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba400cc30 con 0x7fcbb406c6b0 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.441+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba400eee0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.441+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbac009b30 con 0x7fcbb406c6b0 2024-01-16T15:04:42.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.441+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb40c98c0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.442+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb40c9ae0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.443+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fcba4013470 con 0x7fcbb406c6b0 2024-01-16T15:04:42.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.444+0000 7fcbabe00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb8c06b7d0 0x7fcb8c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:42.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.444+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fcba408a5d0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.444+0000 7fcbb3400640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb8c06b7d0 0x7fcb8c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:42.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.445+0000 7fcbb3400640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb8c06b7d0 0x7fcb8c06dc90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4099ad0 tx=0x7fcb9c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:42.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.449+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcba4054eb0 con 0x7fcbb406c6b0 2024-01-16T15:04:42.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:42.681+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7fcbb40cd620 con 0x7fcb8c06b7d0 2024-01-16T15:04:43.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.510+0000 7fcbabe00640 1 -- 172.21.15.53:0/2074437316 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fcb9c001400 con 0x7fcb8c06b7d0 2024-01-16T15:04:43.490 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:04:43.490 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:43.490 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:43.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb8c06b7d0 msgr2=0x7fcb8c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb8c06b7d0 0x7fcb8c06dc90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4099ad0 tx=0x7fcb9c009040 comp rx=0 tx=0).stop 2024-01-16T15:04:43.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 msgr2=0x7fcbb414f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 0x7fcbb414f7d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fcba400ebe0 tx=0x7fcba400c520 comp rx=0 tx=0).stop 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 shutdown_connections 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb8c06b7d0 0x7fcb8c06dc90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbb40cd240 0x7fcbb4154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbb409a170 0x7fcbb4151ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 --2- 172.21.15.53:0/2074437316 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbb406c6b0 0x7fcbb414f7d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 >> 172.21.15.53:0/2074437316 conn(0x7fcbb40b2d00 msgr2=0x7fcbb40b30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 shutdown_connections 2024-01-16T15:04:43.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.514+0000 7fcbbae00640 1 -- 172.21.15.53:0/2074437316 wait complete. 2024-01-16T15:04:43.513 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:43.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48372 2024-01-16T15:04:43.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.763+0000 7f253ce00640 1 Processor -- start 2024-01-16T15:04:43.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.763+0000 7f253ce00640 1 -- start start 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f253ce00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25380ccb50 0x7f2538070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f253ce00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253815f4e0 con 0x7f25380adf20 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f253ce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2538071130 con 0x7f253815bf10 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f253ce00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f253815f970 con 0x7f25380ccb50 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f2537400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f25380adf20 0x7f25380cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41080/0 (socket says 172.21.15.53:41080) 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f252fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25380ccb50 0x7f2538070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.765+0000 7f2537400640 1 -- 172.21.15.53:0/1663273611 learned_addr learned my addr 172.21.15.53:0/1663273611 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.766+0000 7f252fe00640 1 -- 172.21.15.53:0/1663273611 >> v1:172.21.15.146:6789/0 conn(0x7f253815bf10 legacy=0x7f2538143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.766+0000 7f252fe00640 1 -- 172.21.15.53:0/1663273611 >> v1:172.21.15.53:6789/0 conn(0x7f25380adf20 legacy=0x7f25380cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.766+0000 7f252fe00640 1 -- 172.21.15.53:0/1663273611 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253815ebb0 con 0x7f25380ccb50 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.766+0000 7f2536a00640 1 -- 172.21.15.53:0/1663273611 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723748689 0 0) 0x7f253815f4e0 con 0x7f25380adf20 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.767+0000 7f252fe00640 1 --2- 172.21.15.53:0/1663273611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25380ccb50 0x7f2538070a30 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f25200097a0 tx=0x7f252002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7c9aab1d2b3c7f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.767+0000 7f2536a00640 1 -- 172.21.15.53:0/1663273611 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252002e070 con 0x7f25380ccb50 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.767+0000 7f2536a00640 1 -- 172.21.15.53:0/1663273611 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25200315a0 con 0x7f25380ccb50 2024-01-16T15:04:43.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.767+0000 7f2536a00640 1 -- 172.21.15.53:0/1663273611 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2520031930 con 0x7f25380ccb50 2024-01-16T15:04:43.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 -- 172.21.15.53:0/1663273611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25380ccb50 msgr2=0x7f2538070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 --2- 172.21.15.53:0/1663273611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25380ccb50 0x7f2538070a30 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f25200097a0 tx=0x7f252002b840 comp rx=0 tx=0).stop 2024-01-16T15:04:43.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 -- 172.21.15.53:0/1663273611 shutdown_connections 2024-01-16T15:04:43.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 --2- 172.21.15.53:0/1663273611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25380ccb50 0x7f2538070a30 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 -- 172.21.15.53:0/1663273611 >> 172.21.15.53:0/1663273611 conn(0x7f253806a7d0 msgr2=0x7f253806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 -- 172.21.15.53:0/1663273611 shutdown_connections 2024-01-16T15:04:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.768+0000 7f253ce00640 1 -- 172.21.15.53:0/1663273611 wait complete. 2024-01-16T15:04:43.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.770+0000 7f253ce00640 1 Processor -- start 2024-01-16T15:04:43.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.770+0000 7f253ce00640 1 -- start start 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f253ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25380adf20 0x7f253815a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f253ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25380ccb50 0x7f25380abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f253ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 0x7f25380ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f253ce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380aadf0 con 0x7f253815bf10 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f253ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380aa1e0 con 0x7f25380adf20 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f253ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380aa4c0 con 0x7f25380ccb50 2024-01-16T15:04:43.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f2537e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 0x7f25380ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f2537e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 0x7f25380ac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42142/0 (socket says 172.21.15.53:42142) 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.771+0000 7f2537e00640 1 -- 172.21.15.53:0/915886354 learned_addr learned my addr 172.21.15.53:0/915886354 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537400640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25380adf20 0x7f253815a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537e00640 1 -- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25380ccb50 msgr2=0x7f25380abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537e00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25380ccb50 0x7f25380abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537e00640 1 -- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25380adf20 msgr2=0x7f253815a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537e00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25380adf20 0x7f253815a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537e00640 1 -- 172.21.15.53:0/915886354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2520009400 con 0x7f253815bf10 2024-01-16T15:04:43.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f2537e00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 0x7f25380ac2b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f253000c780 tx=0x7f253000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2530004b60 con 0x7f253815bf10 2024-01-16T15:04:43.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253000ce90 con 0x7f253815bf10 2024-01-16T15:04:43.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253000a850 con 0x7f253815bf10 2024-01-16T15:04:43.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25380aa7a0 con 0x7f253815bf10 2024-01-16T15:04:43.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.772+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253815ad90 con 0x7f253815bf10 2024-01-16T15:04:43.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.773+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25380aaf60 con 0x7f253815bf10 2024-01-16T15:04:43.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.774+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f253000a9b0 con 0x7f253815bf10 2024-01-16T15:04:43.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.775+0000 7f252f400640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f251006b7d0 0x7f251006dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:43.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.775+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f253008a5f0 con 0x7f253815bf10 2024-01-16T15:04:43.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.776+0000 7f2537400640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f251006b7d0 0x7f251006dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:43.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.777+0000 7f2537400640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f251006b7d0 0x7f251006dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2528000f60 tx=0x7f2528009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:43.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:43.780+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2530054e40 con 0x7f253815bf10 2024-01-16T15:04:44.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.058+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48372", "target": ["mon-mgr", ""]}) v1 -- 0x7f253815f170 con 0x7f251006b7d0 2024-01-16T15:04:44.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.141+0000 7f252f400640 1 -- 172.21.15.53:0/915886354 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f2528001270 con 0x7f251006b7d0 2024-01-16T15:04:44.121 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:44.121 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:44.121 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:04:44.121 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:44.122 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.147+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f251006b7d0 msgr2=0x7f251006dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.147+0000 7f253ce00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f251006b7d0 0x7f251006dc90 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2528000f60 tx=0x7f2528009040 comp rx=0 tx=0).stop 2024-01-16T15:04:44.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.147+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 msgr2=0x7f25380ac2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.147+0000 7f253ce00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 0x7f25380ac2b0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f253000c780 tx=0x7f253000cc40 comp rx=0 tx=0).stop 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.148+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 shutdown_connections 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.148+0000 7f253ce00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f251006b7d0 0x7f251006dc90 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.148+0000 7f253ce00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f253815bf10 0x7f25380ac2b0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.148+0000 7f253ce00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25380ccb50 0x7f25380abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.148+0000 7f253ce00640 1 --2- 172.21.15.53:0/915886354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25380adf20 0x7f253815a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.148+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 >> 172.21.15.53:0/915886354 conn(0x7f253806a7d0 msgr2=0x7f253815b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.150+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 shutdown_connections 2024-01-16T15:04:44.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.150+0000 7f253ce00640 1 -- 172.21.15.53:0/915886354 wait complete. 2024-01-16T15:04:44.153 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:44.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48373 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.454+0000 7f8ef6800640 1 Processor -- start 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.454+0000 7f8ef6800640 1 -- start start 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8ef6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef0143ae0 0x7f8ef00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8ef6800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8ef00cf7d0 0x7f8ef015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8ef6800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ef00cfc30 con 0x7f8ef015e1e0 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8ef6800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ef00b1ee0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8ef6800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef00b2780 con 0x7f8ef0143ae0 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8eeea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8ef00cf7d0 0x7f8ef015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8eeea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8ef00cf7d0 0x7f8ef015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42144/0 (socket says 172.21.15.53:42144) 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8eefe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8ef015e1e0 0x7f8ef015a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41094/0 (socket says 172.21.15.53:41094) 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8eeea00640 1 -- 172.21.15.53:0/3398203149 learned_addr learned my addr 172.21.15.53:0/3398203149 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.456+0000 7f8eef400640 1 --2- 172.21.15.53:0/3398203149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef0143ae0 0x7f8ef00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.457+0000 7f8eeea00640 1 -- 172.21.15.53:0/3398203149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef0143ae0 msgr2=0x7f8ef00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.457+0000 7f8eeea00640 1 --2- 172.21.15.53:0/3398203149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef0143ae0 0x7f8ef00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.457+0000 7f8eeea00640 1 -- 172.21.15.53:0/3398203149 >> v1:172.21.15.53:6789/0 conn(0x7f8ef015e1e0 legacy=0x7f8ef015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.457+0000 7f8eeea00640 1 -- 172.21.15.53:0/3398203149 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef015e990 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.457+0000 7f8eee000640 1 -- 172.21.15.53:0/3398203149 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898396320 0 0) 0x7f8ef00cfc30 con 0x7f8ef015e1e0 2024-01-16T15:04:44.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.458+0000 7f8eeea00640 1 --2- 172.21.15.53:0/3398203149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef015dca0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800bb10 tx=0x7f8ed802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315d49ac81ab474c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.458+0000 7f8eee000640 1 -- 172.21.15.53:0/3398203149 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8036440 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.458+0000 7f8eee000640 1 -- 172.21.15.53:0/3398203149 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ed80365a0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.458+0000 7f8eee000640 1 -- 172.21.15.53:0/3398203149 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ed8036930 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 -- 172.21.15.53:0/3398203149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 msgr2=0x7f8ef015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 --2- 172.21.15.53:0/3398203149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef015dca0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800bb10 tx=0x7f8ed802e150 comp rx=0 tx=0).stop 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 -- 172.21.15.53:0/3398203149 shutdown_connections 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 --2- 172.21.15.53:0/3398203149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef015dca0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 --2- 172.21.15.53:0/3398203149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef0143ae0 0x7f8ef00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 -- 172.21.15.53:0/3398203149 >> 172.21.15.53:0/3398203149 conn(0x7f8ef006a800 msgr2=0x7f8ef006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 -- 172.21.15.53:0/3398203149 shutdown_connections 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.459+0000 7f8ef6800640 1 -- 172.21.15.53:0/3398203149 wait complete. 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.461+0000 7f8ef6800640 1 Processor -- start 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.461+0000 7f8ef6800640 1 -- start start 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8ef6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef01f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8ef6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef0143ae0 0x7f8ef0141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8ef6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef015e1e0 0x7f8ef0141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8ef6800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef01f65f0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8ef6800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef01f3510 con 0x7f8ef0143ae0 2024-01-16T15:04:44.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8ef6800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ef01f5fa0 con 0x7f8ef015e1e0 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8eef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef01f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8eef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef01f6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42160/0 (socket says 172.21.15.53:42160) 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.462+0000 7f8eef400640 1 -- 172.21.15.53:0/557835400 learned_addr learned my addr 172.21.15.53:0/557835400 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eeea00640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef0143ae0 0x7f8ef0141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eefe00640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef015e1e0 0x7f8ef0141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eef400640 1 -- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef015e1e0 msgr2=0x7f8ef0141bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eef400640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef015e1e0 0x7f8ef0141bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eef400640 1 -- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef0143ae0 msgr2=0x7f8ef0141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eef400640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef0143ae0 0x7f8ef0141690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eef400640 1 -- 172.21.15.53:0/557835400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed800b770 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.463+0000 7f8eef400640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef01f6a80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000d800 tx=0x7f8ee000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.464+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee000b6c0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.464+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ee000b840 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.464+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ee000bbf0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.464+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ef01f6280 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.464+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ef01f37f0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.466+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ef00675d0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.467+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f8ee000bd50 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.468+0000 7f8ee7400640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8ec806b7d0 0x7f8ec806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:44.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.468+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f8ee008aa30 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.469+0000 7f8eeea00640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8ec806b7d0 0x7f8ec806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:44.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.470+0000 7f8eeea00640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8ec806b7d0 0x7f8ec806dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8038000 tx=0x7f8ed8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:44.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.474+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ee00551d0 con 0x7f8ef00cf7d0 2024-01-16T15:04:44.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.889+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48373", "target": ["mon-mgr", ""]}) v1 -- 0x7f8ef0142110 con 0x7f8ec806b7d0 2024-01-16T15:04:44.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.943+0000 7f8ee7400640 1 -- 172.21.15.53:0/557835400 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f8ef0142110 con 0x7f8ec806b7d0 2024-01-16T15:04:44.922 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:44.922 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:44.922 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:44.922 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:44.923 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:44.924 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:44.924 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:44.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.946+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8ec806b7d0 msgr2=0x7f8ec806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.946+0000 7f8ef6800640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8ec806b7d0 0x7f8ec806dc90 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8038000 tx=0x7f8ed8008040 comp rx=0 tx=0).stop 2024-01-16T15:04:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.946+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 msgr2=0x7f8ef01f6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.946+0000 7f8ef6800640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef01f6a80 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f8ee000d800 tx=0x7f8ee000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 shutdown_connections 2024-01-16T15:04:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8ec806b7d0 0x7f8ec806dc90 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8ef015e1e0 0x7f8ef0141bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8ef0143ae0 0x7f8ef0141690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 --2- 172.21.15.53:0/557835400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8ef00cf7d0 0x7f8ef01f6a80 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 >> 172.21.15.53:0/557835400 conn(0x7f8ef006a800 msgr2=0x7f8ef015b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 shutdown_connections 2024-01-16T15:04:44.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:44.947+0000 7f8ef6800640 1 -- 172.21.15.53:0/557835400 wait complete. 2024-01-16T15:04:44.949 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:45.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48373 2024-01-16T15:04:46.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.264+0000 7fd53d000640 1 Processor -- start 2024-01-16T15:04:46.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.264+0000 7fd53d000640 1 -- start start 2024-01-16T15:04:46.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.267+0000 7fd53d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53806a3e0 0x7fd5380ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.267+0000 7fd53d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5380d0b60 0x7fd5380aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.267+0000 7fd53d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd538071de0 con 0x7fd53806a7c0 2024-01-16T15:04:46.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.267+0000 7fd53d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5380af320 con 0x7fd5380d0b60 2024-01-16T15:04:46.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.267+0000 7fd53d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd538072030 con 0x7fd53806a3e0 2024-01-16T15:04:46.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.268+0000 7fd52fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5380d0b60 0x7fd5380aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.268+0000 7fd52fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5380d0b60 0x7fd5380aede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:51146/0 (socket says 172.21.15.53:51146) 2024-01-16T15:04:46.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.268+0000 7fd52fe00640 1 -- 172.21.15.53:0/2430061021 learned_addr learned my addr 172.21.15.53:0/2430061021 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:46.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.269+0000 7fd535e00640 1 --2- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53806a3e0 0x7fd5380ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.269+0000 7fd52fe00640 1 -- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53806a3e0 msgr2=0x7fd5380ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:46.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.269+0000 7fd52fe00640 1 --2- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53806a3e0 0x7fd5380ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.269+0000 7fd52fe00640 1 -- 172.21.15.53:0/2430061021 >> v1:172.21.15.146:6789/0 conn(0x7fd53806a7c0 legacy=0x7fd53806f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.269+0000 7fd52fe00640 1 -- 172.21.15.53:0/2430061021 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd538072310 con 0x7fd5380d0b60 2024-01-16T15:04:46.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.275+0000 7fd52fe00640 1 --2- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5380d0b60 0x7fd5380aede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd528009870 tx=0x7fd52802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f8b60b410c6c419 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.278+0000 7fd535400640 1 -- 172.21.15.53:0/2430061021 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528030840 con 0x7fd5380d0b60 2024-01-16T15:04:46.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.278+0000 7fd535400640 1 -- 172.21.15.53:0/2430061021 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5280309a0 con 0x7fd5380d0b60 2024-01-16T15:04:46.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.278+0000 7fd535400640 1 -- 172.21.15.53:0/2430061021 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd528030d30 con 0x7fd5380d0b60 2024-01-16T15:04:46.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 -- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5380d0b60 msgr2=0x7fd5380aede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 --2- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5380d0b60 0x7fd5380aede0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd528009870 tx=0x7fd52802b980 comp rx=0 tx=0).stop 2024-01-16T15:04:46.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 -- 172.21.15.53:0/2430061021 shutdown_connections 2024-01-16T15:04:46.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 --2- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5380d0b60 0x7fd5380aede0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 --2- 172.21.15.53:0/2430061021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd53806a3e0 0x7fd5380ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 -- 172.21.15.53:0/2430061021 >> 172.21.15.53:0/2430061021 conn(0x7fd5380cfef0 msgr2=0x7fd5380d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:46.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 -- 172.21.15.53:0/2430061021 shutdown_connections 2024-01-16T15:04:46.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.279+0000 7fd53d000640 1 -- 172.21.15.53:0/2430061021 wait complete. 2024-01-16T15:04:46.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.281+0000 7fd53d000640 1 Processor -- start 2024-01-16T15:04:46.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.281+0000 7fd53d000640 1 -- start start 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd53d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 0x7fd538147d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd53d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53806a7c0 0x7fd538159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd53d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5380d0b60 0x7fd538159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd53d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd538148260 con 0x7fd53806a3e0 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd53d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53815b0d0 con 0x7fd53806a7c0 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd53d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53815a4c0 con 0x7fd5380d0b60 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 0x7fd538147d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 0x7fd538147d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42170/0 (socket says 172.21.15.53:42170) 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 -- 172.21.15.53:0/1415261614 learned_addr learned my addr 172.21.15.53:0/1415261614 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:46.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 -- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5380d0b60 msgr2=0x7fd538159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5380d0b60 0x7fd538159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 -- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53806a7c0 msgr2=0x7fd538159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53806a7c0 0x7fd538159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.282+0000 7fd535e00640 1 -- 172.21.15.53:0/1415261614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5280094d0 con 0x7fd53806a3e0 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.283+0000 7fd535e00640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 0x7fd538147d20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd52000d800 tx=0x7fd52000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.283+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52000b6c0 con 0x7fd53806a3e0 2024-01-16T15:04:46.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.283+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52000b820 con 0x7fd53806a3e0 2024-01-16T15:04:46.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.283+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52000bbb0 con 0x7fd53806a3e0 2024-01-16T15:04:46.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.283+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53815a7a0 con 0x7fd53806a3e0 2024-01-16T15:04:46.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.283+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53815ad60 con 0x7fd53806a3e0 2024-01-16T15:04:46.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.284+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5380675d0 con 0x7fd53806a3e0 2024-01-16T15:04:46.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.285+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd52000bd10 con 0x7fd53806a3e0 2024-01-16T15:04:46.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.286+0000 7fd52e000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd51006b820 0x7fd51006dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:46.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.286+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd52008a960 con 0x7fd53806a3e0 2024-01-16T15:04:46.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.292+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd520055180 con 0x7fd53806a3e0 2024-01-16T15:04:46.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.296+0000 7fd52fe00640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd51006b820 0x7fd51006dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:46.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.298+0000 7fd52fe00640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd51006b820 0x7fd51006dce0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd528036000 tx=0x7fd52803b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:46.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.693+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48373", "target": ["mon-mgr", ""]}) v1 -- 0x7fd538160bb0 con 0x7fd51006b820 2024-01-16T15:04:46.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.775+0000 7fd52e000640 1 -- 172.21.15.53:0/1415261614 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fd538160bb0 con 0x7fd51006b820 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:46.754 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:46.755 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:46.755 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:46.755 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:46.755 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:46.755 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:46.755 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd51006b820 msgr2=0x7fd51006dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd51006b820 0x7fd51006dce0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fd528036000 tx=0x7fd52803b040 comp rx=0 tx=0).stop 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 msgr2=0x7fd538147d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 0x7fd538147d20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd52000d800 tx=0x7fd52000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 shutdown_connections 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd51006b820 0x7fd51006dce0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5380d0b60 0x7fd538159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd53806a7c0 0x7fd538159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 --2- 172.21.15.53:0/1415261614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd53806a3e0 0x7fd538147d20 secure :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fd52000d800 tx=0x7fd52000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 >> 172.21.15.53:0/1415261614 conn(0x7fd5380cfef0 msgr2=0x7fd5380addd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 shutdown_connections 2024-01-16T15:04:46.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:46.779+0000 7fd53d000640 1 -- 172.21.15.53:0/1415261614 wait complete. 2024-01-16T15:04:46.788 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:47.789 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48373 2024-01-16T15:04:48.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.039+0000 7f0e3fe00640 1 Processor -- start 2024-01-16T15:04:48.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.039+0000 7f0e3fe00640 1 -- start start 2024-01-16T15:04:48.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.040+0000 7f0e3fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e4015d4b0 0x7f0e400d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.040+0000 7f0e3fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0e400cf9a0 0x7f0e400cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.040+0000 7f0e3fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e400d0320 con 0x7f0e400cddf0 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.040+0000 7f0e3fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400b2730 con 0x7f0e400cf9a0 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.040+0000 7f0e3fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e400d1ec0 con 0x7f0e4015d4b0 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e37e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0e400cf9a0 0x7f0e400cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e3f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0e400cddf0 0x7f0e400ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41106/0 (socket says 172.21.15.53:41106) 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e3f400640 1 -- 172.21.15.53:0/2653530075 learned_addr learned my addr 172.21.15.53:0/2653530075 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e3ea00640 1 --2- 172.21.15.53:0/2653530075 >> v2:172.21.15.146:3300/0 conn(0x7f0e4015d4b0 0x7f0e400d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e37e00640 1 -- 172.21.15.53:0/2653530075 >> v2:172.21.15.146:3300/0 conn(0x7f0e4015d4b0 msgr2=0x7f0e400d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e37e00640 1 --2- 172.21.15.53:0/2653530075 >> v2:172.21.15.146:3300/0 conn(0x7f0e4015d4b0 0x7f0e400d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e37e00640 1 -- 172.21.15.53:0/2653530075 >> v1:172.21.15.53:6789/0 conn(0x7f0e400cddf0 legacy=0x7f0e400ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.041+0000 7f0e37e00640 1 -- 172.21.15.53:0/2653530075 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e400b29d0 con 0x7f0e400cf9a0 2024-01-16T15:04:48.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.042+0000 7f0e37e00640 1 --2- 172.21.15.53:0/2653530075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cfde0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0e4008b5b0 tx=0x7f0e2802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d64f609c916ad1b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.043+0000 7f0e3e000640 1 -- 172.21.15.53:0/2653530075 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28031830 con 0x7f0e400cf9a0 2024-01-16T15:04:48.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.043+0000 7f0e3e000640 1 -- 172.21.15.53:0/2653530075 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e28031990 con 0x7f0e400cf9a0 2024-01-16T15:04:48.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.043+0000 7f0e3e000640 1 -- 172.21.15.53:0/2653530075 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28031d20 con 0x7f0e400cf9a0 2024-01-16T15:04:48.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.043+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2653530075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cf9a0 msgr2=0x7f0e400cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.043+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2653530075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cfde0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f0e4008b5b0 tx=0x7f0e2802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.044+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2653530075 shutdown_connections 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.044+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2653530075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cfde0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.044+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2653530075 >> v2:172.21.15.146:3300/0 conn(0x7f0e4015d4b0 0x7f0e400d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.044+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2653530075 >> 172.21.15.53:0/2653530075 conn(0x7f0e400d3000 msgr2=0x7f0e400d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.044+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2653530075 shutdown_connections 2024-01-16T15:04:48.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.044+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2653530075 wait complete. 2024-01-16T15:04:48.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.046+0000 7f0e3fe00640 1 Processor -- start 2024-01-16T15:04:48.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.046+0000 7f0e3fe00640 1 -- start start 2024-01-16T15:04:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 0x7f0e40152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cb530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e4015d4b0 0x7f0e400cce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e400cb110 con 0x7f0e400cddf0 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e40152d60 con 0x7f0e400cf9a0 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e400caae0 con 0x7f0e4015d4b0 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 0x7f0e40152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 0x7f0e40152820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42192/0 (socket says 172.21.15.53:42192) 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.047+0000 7f0e3ea00640 1 -- 172.21.15.53:0/2019720949 learned_addr learned my addr 172.21.15.53:0/2019720949 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:48.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e37e00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3ea00640 1 -- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e4015d4b0 msgr2=0x7f0e400cce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3f400640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e4015d4b0 0x7f0e400cce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3ea00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e4015d4b0 0x7f0e400cce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3ea00640 1 -- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e400cf9a0 msgr2=0x7f0e400cb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3ea00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3ea00640 1 -- 172.21.15.53:0/2019720949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28009400 con 0x7f0e400cddf0 2024-01-16T15:04:48.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e37e00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:48.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e3ea00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 0x7f0e40152820 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000d620 tx=0x7f0e3000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.048+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e300125f0 con 0x7f0e400cddf0 2024-01-16T15:04:48.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.049+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e30012750 con 0x7f0e400cddf0 2024-01-16T15:04:48.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.049+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e30012b00 con 0x7f0e400cddf0 2024-01-16T15:04:48.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.049+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e4015da10 con 0x7f0e400cddf0 2024-01-16T15:04:48.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.049+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e4009e510 con 0x7f0e400cddf0 2024-01-16T15:04:48.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.050+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e400cb380 con 0x7f0e400cddf0 2024-01-16T15:04:48.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.051+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f0e30012c60 con 0x7f0e400cddf0 2024-01-16T15:04:48.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.051+0000 7f0e36a00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e1806b7d0 0x7f0e1806dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.052+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f0e3008a860 con 0x7f0e400cddf0 2024-01-16T15:04:48.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.052+0000 7f0e37e00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e1806b7d0 0x7f0e1806dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.053+0000 7f0e37e00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e1806b7d0 0x7f0e1806dc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0e28036000 tx=0x7f0e28008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.056+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e30055090 con 0x7f0e400cddf0 2024-01-16T15:04:48.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.289+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48373", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e4015d890 con 0x7f0e1806b7d0 2024-01-16T15:04:48.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.439+0000 7f0e36a00640 1 -- 172.21.15.53:0/2019720949 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f0e28031b80 con 0x7f0e1806b7d0 2024-01-16T15:04:48.419 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:48.419 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:48.419 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:04:48.419 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:48.419 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:48.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e1806b7d0 msgr2=0x7f0e1806dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e1806b7d0 0x7f0e1806dc90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f0e28036000 tx=0x7f0e28008030 comp rx=0 tx=0).stop 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 msgr2=0x7f0e40152820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 0x7f0e40152820 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f0e3000d620 tx=0x7f0e3000dae0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 shutdown_connections 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e1806b7d0 0x7f0e1806dc90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e4015d4b0 0x7f0e400cce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e400cf9a0 0x7f0e400cb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 --2- 172.21.15.53:0/2019720949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e400cddf0 0x7f0e40152820 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 >> 172.21.15.53:0/2019720949 conn(0x7f0e400d3000 msgr2=0x7f0e400b31d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 shutdown_connections 2024-01-16T15:04:48.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.442+0000 7f0e3fe00640 1 -- 172.21.15.53:0/2019720949 wait complete. 2024-01-16T15:04:48.444 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:04:48.444 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48374 2024-01-16T15:04:48.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.689+0000 7fe7e9200640 1 Processor -- start 2024-01-16T15:04:48.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.690+0000 7fe7e9200640 1 -- start start 2024-01-16T15:04:48.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e9200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7e40ccb50 0x7fe7e4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 0x7fe7e4143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e415f4e0 con 0x7fe7e40adf20 2024-01-16T15:04:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e9200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e415f970 con 0x7fe7e40ccb50 2024-01-16T15:04:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e415c3c0 con 0x7fe7e415bf10 2024-01-16T15:04:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe7e40adf20 0x7fe7e40cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41110/0 (socket says 172.21.15.53:41110) 2024-01-16T15:04:48.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e3400640 1 -- 172.21.15.53:0/3503019580 learned_addr learned my addr 172.21.15.53:0/3503019580 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:48.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948171724 0 0) 0x7fe7e415f4e0 con 0x7fe7e40adf20 2024-01-16T15:04:48.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe7c4003520 con 0x7fe7e40adf20 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.692+0000 7fe7da800640 1 --2- 172.21.15.53:0/3503019580 >> v2:172.21.15.146:3300/0 conn(0x7fe7e40ccb50 0x7fe7e4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e3e00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 0x7fe7e4143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4167799277 0 0) 0x7fe7c4003520 con 0x7fe7e40adf20 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 msgr2=0x7fe7e4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 0x7fe7e4143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 >> v2:172.21.15.146:3300/0 conn(0x7fe7e40ccb50 msgr2=0x7fe7e4070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 --2- 172.21.15.53:0/3503019580 >> v2:172.21.15.146:3300/0 conn(0x7fe7e40ccb50 0x7fe7e4070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4071130 con 0x7fe7e40adf20 2024-01-16T15:04:48.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe7d4001670 con 0x7fe7e40adf20 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 >> v1:172.21.15.53:6789/0 conn(0x7fe7e40adf20 legacy=0x7fe7e40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4003970 0x7fe7c4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 0x7fe7c40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7c4004d00 0x7fe7c4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e415f970 con 0x7fe7c4004d00 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e415c3c0 con 0x7fe7c4003970 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7c400bb10 con 0x7fe7e415bf10 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.693+0000 7fe7e3e00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7c4004d00 0x7fe7c4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7da800640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 0x7fe7c40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7e3400640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4003970 0x7fe7c4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7e3e00640 1 -- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 msgr2=0x7fe7c40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7e3e00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e415bf10 0x7fe7c40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7e3e00640 1 -- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4003970 msgr2=0x7fe7c4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7e3e00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4003970 0x7fe7c4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.694+0000 7fe7e3e00640 1 -- 172.21.15.53:0/3503019580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7c4003520 con 0x7fe7c4004d00 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.695+0000 7fe7e3e00640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7c4004d00 0x7fe7c4005180 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ba30 tx=0x7fe7c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86b05d3b2eb33ec server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.695+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc032640 con 0x7fe7c4004d00 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.695+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7dc0327a0 con 0x7fe7c4004d00 2024-01-16T15:04:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.695+0000 7fe7e2a00640 1 -- 172.21.15.53:0/3503019580 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7dc032b30 con 0x7fe7c4004d00 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e9200640 1 -- 172.21.15.53:0/3503019580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7c4004d00 msgr2=0x7fe7c4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e9200640 1 --2- 172.21.15.53:0/3503019580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7c4004d00 0x7fe7c4005180 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fe7dc00ba30 tx=0x7fe7c40057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e3e00640 1 -- 172.21.15.53:0/3503019580 reap_dead start 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e9200640 1 -- 172.21.15.53:0/3503019580 shutdown_connections 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e9200640 1 -- 172.21.15.53:0/3503019580 >> 172.21.15.53:0/3503019580 conn(0x7fe7e406a7d0 msgr2=0x7fe7e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e9200640 1 -- 172.21.15.53:0/3503019580 shutdown_connections 2024-01-16T15:04:48.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.696+0000 7fe7e9200640 1 -- 172.21.15.53:0/3503019580 wait complete. 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.698+0000 7fe7e9200640 1 Processor -- start 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.698+0000 7fe7e9200640 1 -- start start 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4004d00 0x7fe7e40b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 0x7fe7e41f2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7e9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40c3d00 0x7fe7e40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7e9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e4141790 con 0x7fe7e4141310 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7e9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e40ae900 con 0x7fe7c4004d00 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7e9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e41f3570 con 0x7fe7e40c3d00 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.699+0000 7fe7da800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 0x7fe7e41f2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 0x7fe7e41f2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42212/0 (socket says 172.21.15.53:42212) 2024-01-16T15:04:48.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 -- 172.21.15.53:0/192739738 learned_addr learned my addr 172.21.15.53:0/192739738 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:48.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7e3400640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4004d00 0x7fe7e40b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7e3e00640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40c3d00 0x7fe7e40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 -- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40c3d00 msgr2=0x7fe7e40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40c3d00 0x7fe7e40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 -- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4004d00 msgr2=0x7fe7e40b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4004d00 0x7fe7e40b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 -- 172.21.15.53:0/192739738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e41f3850 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.700+0000 7fe7da800640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 0x7fe7e41f2d20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009fd0 tx=0x7fe7cc00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.701+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc0145f0 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.701+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7cc014750 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.701+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7cc014ae0 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.701+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e41f3b30 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.701+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e41f3260 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.702+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e4067450 con 0x7fe7e4141310 2024-01-16T15:04:48.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.703+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fe7cc014c40 con 0x7fe7e4141310 2024-01-16T15:04:48.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.704+0000 7fe7dbe00640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b800 0x7fe7bc06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:48.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.704+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fe7cc08b310 con 0x7fe7e4141310 2024-01-16T15:04:48.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.705+0000 7fe7e3400640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b800 0x7fe7bc06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:48.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.706+0000 7fe7e3400640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b800 0x7fe7bc06dcc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4000c00 tx=0x7fe7d4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:48.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.709+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7cc0559a0 con 0x7fe7e4141310 2024-01-16T15:04:48.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.947+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48374", "target": ["mon-mgr", ""]}) v1 -- 0x7fe7e415f170 con 0x7fe7bc06b800 2024-01-16T15:04:48.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.971+0000 7fe7dbe00640 1 -- 172.21.15.53:0/192739738 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fe7d4001650 con 0x7fe7bc06b800 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:48.951 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:48.952 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:48.952 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:48.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b800 msgr2=0x7fe7bc06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b800 0x7fe7bc06dcc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4000c00 tx=0x7fe7d4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:48.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 msgr2=0x7fe7e41f2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:48.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 0x7fe7e41f2d20 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009fd0 tx=0x7fe7cc00f520 comp rx=0 tx=0).stop 2024-01-16T15:04:48.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 shutdown_connections 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7bc06b800 0x7fe7bc06dcc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7e40c3d00 0x7fe7e40ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7e4141310 0x7fe7e41f2d20 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 --2- 172.21.15.53:0/192739738 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7c4004d00 0x7fe7e40b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 >> 172.21.15.53:0/192739738 conn(0x7fe7e406a7d0 msgr2=0x7fe7e4143590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 shutdown_connections 2024-01-16T15:04:48.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:48.974+0000 7fe7e9200640 1 -- 172.21.15.53:0/192739738 wait complete. 2024-01-16T15:04:48.977 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:49.978 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48374 2024-01-16T15:04:50.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.290+0000 7faf0ac00640 1 Processor -- start 2024-01-16T15:04:50.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.291+0000 7faf0ac00640 1 -- start start 2024-01-16T15:04:50.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.292+0000 7faf0ac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faf0415e3b0 0x7faf04164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.292+0000 7faf0ac00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf040b1ee0 con 0x7faf040cf9a0 2024-01-16T15:04:50.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.292+0000 7faf0ac00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf040b2780 con 0x7faf04143ae0 2024-01-16T15:04:50.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.292+0000 7faf0ac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0415e990 con 0x7faf0415e3b0 2024-01-16T15:04:50.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.292+0000 7faefbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faf040cf9a0 0x7faf0415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41122/0 (socket says 172.21.15.53:41122) 2024-01-16T15:04:50.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.292+0000 7faefbe00640 1 -- 172.21.15.53:0/2947712297 learned_addr learned my addr 172.21.15.53:0/2947712297 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442976224 0 0) 0x7faf040b1ee0 con 0x7faf040cf9a0 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee4003520 con 0x7faf040cf9a0 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3491600412 0 0) 0x7faee4003520 con 0x7faf040cf9a0 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 >> v2:172.21.15.146:3300/0 conn(0x7faf0415e3b0 msgr2=0x7faf04164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 --2- 172.21.15.53:0/2947712297 >> v2:172.21.15.146:3300/0 conn(0x7faf0415e3b0 0x7faf04164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 >> v1:172.21.15.146:6789/0 conn(0x7faf04143ae0 legacy=0x7faf040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf040b2130 con 0x7faf040cf9a0 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7faeec001670 con 0x7faf040cf9a0 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 >> v1:172.21.15.53:6789/0 conn(0x7faf040cf9a0 legacy=0x7faf0415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee4003970 0x7faee4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee40047c0 0x7faee4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40051f0 0x7faee400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0415e990 con 0x7faee40047c0 2024-01-16T15:04:50.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faee400c150 con 0x7faee40051f0 2024-01-16T15:04:50.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.294+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faee400c310 con 0x7faee4003970 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.295+0000 7faefbe00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee40047c0 0x7faee4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.295+0000 7faf03e00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee4003970 0x7faee4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.296+0000 7faefbe00640 1 -- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee4003970 msgr2=0x7faee4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.296+0000 7faefbe00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faee4003970 0x7faee4003d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.296+0000 7faefbe00640 1 -- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40051f0 msgr2=0x7faee400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.296+0000 7faefbe00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40051f0 0x7faee400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.296+0000 7faefbe00640 1 -- 172.21.15.53:0/2947712297 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf040b2130 con 0x7faee40047c0 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.296+0000 7faf08c00640 1 -- 172.21.15.53:0/2947712297 reap_dead start 2024-01-16T15:04:50.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.297+0000 7faefbe00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee40047c0 0x7faee4004c40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7faeec00c370 tx=0x7faee40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dff0a650e0d0d64 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.297+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec02cc00 con 0x7faee40047c0 2024-01-16T15:04:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.298+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec02cd60 con 0x7faee40047c0 2024-01-16T15:04:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.298+0000 7faf03400640 1 -- 172.21.15.53:0/2947712297 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec005af0 con 0x7faee40047c0 2024-01-16T15:04:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 -- 172.21.15.53:0/2947712297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee40047c0 msgr2=0x7faee4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee40047c0 0x7faee4004c40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7faeec00c370 tx=0x7faee40057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 -- 172.21.15.53:0/2947712297 shutdown_connections 2024-01-16T15:04:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 --2- 172.21.15.53:0/2947712297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faee40047c0 0x7faee4004c40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 -- 172.21.15.53:0/2947712297 >> 172.21.15.53:0/2947712297 conn(0x7faf0406a800 msgr2=0x7faf0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 -- 172.21.15.53:0/2947712297 shutdown_connections 2024-01-16T15:04:50.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.299+0000 7faf0ac00640 1 -- 172.21.15.53:0/2947712297 wait complete. 2024-01-16T15:04:50.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.303+0000 7faf0ac00640 1 Processor -- start 2024-01-16T15:04:50.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.303+0000 7faf0ac00640 1 -- start start 2024-01-16T15:04:50.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf0ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40047c0 0x7faf0415bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf0ac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 0x7faf0415a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf0ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf04159d10 0x7faf0406c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf0ac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0415bbb0 con 0x7faf04159890 2024-01-16T15:04:50.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf0ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0415af30 con 0x7faee40047c0 2024-01-16T15:04:50.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf0ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0415b210 con 0x7faf04159d10 2024-01-16T15:04:50.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faefbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 0x7faf0415a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faefbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 0x7faf0415a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42228/0 (socket says 172.21.15.53:42228) 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faefbe00640 1 -- 172.21.15.53:0/814571781 learned_addr learned my addr 172.21.15.53:0/814571781 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.305+0000 7faf08c00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf04159d10 0x7faf0406c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.306+0000 7faefbe00640 1 -- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf04159d10 msgr2=0x7faf0406c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.306+0000 7faefbe00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf04159d10 0x7faf0406c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.306+0000 7faefbe00640 1 -- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40047c0 msgr2=0x7faf0415bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.306+0000 7faefbe00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40047c0 0x7faf0415bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.306+0000 7faefbe00640 1 -- 172.21.15.53:0/814571781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0415b4f0 con 0x7faf04159890 2024-01-16T15:04:50.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.306+0000 7faefbe00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 0x7faf0415a4e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7faeec02c5a0 tx=0x7faeec001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.307+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec03c070 con 0x7faf04159890 2024-01-16T15:04:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.307+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeec004da0 con 0x7faf04159890 2024-01-16T15:04:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.307+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeec007270 con 0x7faf04159890 2024-01-16T15:04:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.307+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0415b7d0 con 0x7faf04159890 2024-01-16T15:04:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.307+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0415aa20 con 0x7faf04159890 2024-01-16T15:04:50.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.308+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0415be20 con 0x7faf04159890 2024-01-16T15:04:50.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.310+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7faeec032070 con 0x7faf04159890 2024-01-16T15:04:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.310+0000 7faf00c00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faedc06b7d0 0x7faedc06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.311+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7faeec0ab500 con 0x7faf04159890 2024-01-16T15:04:50.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.315+0000 7faf03e00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faedc06b7d0 0x7faedc06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:50.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.315+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faeec075d30 con 0x7faf04159890 2024-01-16T15:04:50.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.317+0000 7faf03e00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faedc06b7d0 0x7faedc06dc90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf040695b0 tx=0x7faef4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:50.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.672+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48374", "target": ["mon-mgr", ""]}) v1 -- 0x7faf0415dca0 con 0x7faedc06b7d0 2024-01-16T15:04:50.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.689+0000 7faf00c00640 1 -- 172.21.15.53:0/814571781 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7faef40011c0 con 0x7faedc06b7d0 2024-01-16T15:04:50.669 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:50.669 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:50.669 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:04:50.669 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:04:50.669 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:50.670 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.694+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faedc06b7d0 msgr2=0x7faedc06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.694+0000 7faf0ac00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faedc06b7d0 0x7faedc06dc90 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7faf040695b0 tx=0x7faef4009040 comp rx=0 tx=0).stop 2024-01-16T15:04:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.694+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 msgr2=0x7faf0415a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.694+0000 7faf0ac00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 0x7faf0415a4e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7faeec02c5a0 tx=0x7faeec001a00 comp rx=0 tx=0).stop 2024-01-16T15:04:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 shutdown_connections 2024-01-16T15:04:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faedc06b7d0 0x7faedc06dc90 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faf04159d10 0x7faf0406c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faf04159890 0x7faf0415a4e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 --2- 172.21.15.53:0/814571781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faee40047c0 0x7faf0415bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 >> 172.21.15.53:0/814571781 conn(0x7faf0406a800 msgr2=0x7faf040ce660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 shutdown_connections 2024-01-16T15:04:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:50.696+0000 7faf0ac00640 1 -- 172.21.15.53:0/814571781 wait complete. 2024-01-16T15:04:50.700 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:04:51.270 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:51.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:04:51.701 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone48374 2024-01-16T15:04:51.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.956+0000 7fd76a600640 1 Processor -- start 2024-01-16T15:04:51.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.956+0000 7fd76a600640 1 -- start start 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd76a600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd76415b320 0x7fd7640ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd76a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd7640ce5a0 0x7fd7640ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd76a600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640cc730 con 0x7fd7640ccbc0 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd76a600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7640cbe00 con 0x7fd76415b320 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd76a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7640cc0e0 con 0x7fd7640ce5a0 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd769200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd76415b320 0x7fd7640ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd769200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd76415b320 0x7fd7640ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34822/0 (socket says 172.21.15.53:34822) 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd763e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd7640ccbc0 0x7fd76415b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38568/0 (socket says 172.21.15.53:38568) 2024-01-16T15:04:51.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.957+0000 7fd769200640 1 -- 172.21.15.53:0/107824852 learned_addr learned my addr 172.21.15.53:0/107824852 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:51.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.958+0000 7fd769c00640 1 --2- 172.21.15.53:0/107824852 >> v2:172.21.15.146:3300/0 conn(0x7fd7640ce5a0 0x7fd7640ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.958+0000 7fd769200640 1 -- 172.21.15.53:0/107824852 >> v2:172.21.15.146:3300/0 conn(0x7fd7640ce5a0 msgr2=0x7fd7640ceae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:51.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.958+0000 7fd769200640 1 --2- 172.21.15.53:0/107824852 >> v2:172.21.15.146:3300/0 conn(0x7fd7640ce5a0 0x7fd7640ceae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.958+0000 7fd769200640 1 -- 172.21.15.53:0/107824852 >> v1:172.21.15.53:6789/0 conn(0x7fd7640ccbc0 legacy=0x7fd76415b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.958+0000 7fd769200640 1 -- 172.21.15.53:0/107824852 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7640cc3c0 con 0x7fd76415b320 2024-01-16T15:04:51.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.959+0000 7fd769200640 1 --2- 172.21.15.53:0/107824852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76415b320 0x7fd7640ce060 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd7640d0630 tx=0x7fd75402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4efa6d71fa665090 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd763400640 1 -- 172.21.15.53:0/107824852 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754030860 con 0x7fd76415b320 2024-01-16T15:04:51.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd763400640 1 -- 172.21.15.53:0/107824852 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7540309c0 con 0x7fd76415b320 2024-01-16T15:04:51.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd763400640 1 -- 172.21.15.53:0/107824852 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754030d50 con 0x7fd76415b320 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 -- 172.21.15.53:0/107824852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76415b320 msgr2=0x7fd7640ce060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 --2- 172.21.15.53:0/107824852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76415b320 0x7fd7640ce060 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fd7640d0630 tx=0x7fd75402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 -- 172.21.15.53:0/107824852 shutdown_connections 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 --2- 172.21.15.53:0/107824852 >> v2:172.21.15.146:3300/0 conn(0x7fd7640ce5a0 0x7fd7640ceae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 --2- 172.21.15.53:0/107824852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd76415b320 0x7fd7640ce060 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 -- 172.21.15.53:0/107824852 >> 172.21.15.53:0/107824852 conn(0x7fd7640b2a60 msgr2=0x7fd7640b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.960+0000 7fd76a600640 1 -- 172.21.15.53:0/107824852 shutdown_connections 2024-01-16T15:04:51.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.961+0000 7fd76a600640 1 -- 172.21.15.53:0/107824852 wait complete. 2024-01-16T15:04:51.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.962+0000 7fd76a600640 1 Processor -- start 2024-01-16T15:04:51.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.962+0000 7fd76a600640 1 -- start start 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.963+0000 7fd76a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 0x7fd7641f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.963+0000 7fd76a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7640ce5a0 0x7fd7640c7fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.963+0000 7fd76a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd76415b320 0x7fd76409c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.963+0000 7fd76a600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641f36d0 con 0x7fd7640ccbc0 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.963+0000 7fd76a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76409d190 con 0x7fd76415b320 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.963+0000 7fd76a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd76409d470 con 0x7fd7640ce5a0 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 0x7fd7641f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 0x7fd7641f3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34832/0 (socket says 172.21.15.53:34832) 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 -- 172.21.15.53:0/1425878466 learned_addr learned my addr 172.21.15.53:0/1425878466 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd763e00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7640ce5a0 0x7fd7640c7fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769c00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd76415b320 0x7fd76409c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 -- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7640ce5a0 msgr2=0x7fd7640c7fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7640ce5a0 0x7fd7640c7fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 -- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd76415b320 msgr2=0x7fd76409c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd76415b320 0x7fd76409c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 -- 172.21.15.53:0/1425878466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7540094d0 con 0x7fd7640ccbc0 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769c00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd76415b320 0x7fd76409c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd763e00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7640ce5a0 0x7fd7640c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:51.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.964+0000 7fd769200640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 0x7fd7641f3190 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd75402b9e0 tx=0x7fd754004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.965+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754039b60 con 0x7fd7640ccbc0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.965+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd754039ce0 con 0x7fd7640ccbc0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.965+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd754043650 con 0x7fd7640ccbc0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.965+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76409d750 con 0x7fd7640ccbc0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.965+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76409dd10 con 0x7fd7640ccbc0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.966+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd764067b60 con 0x7fd7640ccbc0 2024-01-16T15:04:51.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.967+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7fd75402f3b0 con 0x7fd7640ccbc0 2024-01-16T15:04:51.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.968+0000 7fd760c00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd73c06b7d0 0x7fd73c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:51.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.968+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fd7540ac750 con 0x7fd7640ccbc0 2024-01-16T15:04:51.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.969+0000 7fd763e00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd73c06b7d0 0x7fd73c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:51.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.970+0000 7fd763e00640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd73c06b7d0 0x7fd73c06dc90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd7640c77c0 tx=0x7fd74c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:51.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:51.972+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd754076ea0 con 0x7fd7640ccbc0 2024-01-16T15:04:52.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.206+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone48374", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7640b2840 con 0x7fd73c06b7d0 2024-01-16T15:04:52.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.375+0000 7fd760c00640 1 -- 172.21.15.53:0/1425878466 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fd7640b2840 con 0x7fd73c06b7d0 2024-01-16T15:04:52.355 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:04:52.356 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:04:52.356 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:04:52.356 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:52.356 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:04:52.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd73c06b7d0 msgr2=0x7fd73c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd73c06b7d0 0x7fd73c06dc90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd7640c77c0 tx=0x7fd74c009040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 msgr2=0x7fd7641f3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 0x7fd7641f3190 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fd75402b9e0 tx=0x7fd754004730 comp rx=0 tx=0).stop 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 shutdown_connections 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd73c06b7d0 0x7fd73c06dc90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd76415b320 0x7fd76409c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd7640ce5a0 0x7fd7640c7fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 --2- 172.21.15.53:0/1425878466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd7640ccbc0 0x7fd7641f3190 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.379+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 >> 172.21.15.53:0/1425878466 conn(0x7fd7640b2a60 msgr2=0x7fd76406cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.380+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 shutdown_connections 2024-01-16T15:04:52.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.380+0000 7fd76a600640 1 -- 172.21.15.53:0/1425878466 wait complete. 2024-01-16T15:04:52.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:04:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.620+0000 7ff679400640 1 Processor -- start 2024-01-16T15:04:52.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.620+0000 7ff679400640 1 -- start start 2024-01-16T15:04:52.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff679400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff679400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff679400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff67406c470 con 0x7ff67415f9c0 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff679400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff67406c750 con 0x7ff67413df80 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff679400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67406ca30 con 0x7ff67413e360 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff672800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff673200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff67415f9c0 0x7ff6741408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38584/0 (socket says 172.21.15.53:38584) 2024-01-16T15:04:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.622+0000 7ff673200640 1 -- 172.21.15.53:0/2616272432 learned_addr learned my addr 172.21.15.53:0/2616272432 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff671e00640 1 -- 172.21.15.53:0/2616272432 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054590506 0 0) 0x7ff67406c470 con 0x7ff67415f9c0 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff66be00640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff671e00640 1 -- 172.21.15.53:0/2616272432 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff654003520 con 0x7ff67415f9c0 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff672800640 1 -- 172.21.15.53:0/2616272432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 msgr2=0x7ff6740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff672800640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff672800640 1 -- 172.21.15.53:0/2616272432 >> v1:172.21.15.53:6789/0 conn(0x7ff67415f9c0 legacy=0x7ff6741408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff672800640 1 -- 172.21.15.53:0/2616272432 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67406cd10 con 0x7ff67413df80 2024-01-16T15:04:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.623+0000 7ff66be00640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.624+0000 7ff672800640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff67415bee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff6640097a0 tx=0x7ff66402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d83286205ce581 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.624+0000 7ff671e00640 1 -- 172.21.15.53:0/2616272432 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664031830 con 0x7ff67413df80 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.624+0000 7ff671e00640 1 -- 172.21.15.53:0/2616272432 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff664031990 con 0x7ff67413df80 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.624+0000 7ff671e00640 1 -- 172.21.15.53:0/2616272432 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664031d00 con 0x7ff67413df80 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 -- 172.21.15.53:0/2616272432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 msgr2=0x7ff67415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff67415bee0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7ff6640097a0 tx=0x7ff66402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 -- 172.21.15.53:0/2616272432 shutdown_connections 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 --2- 172.21.15.53:0/2616272432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff67415bee0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 -- 172.21.15.53:0/2616272432 >> 172.21.15.53:0/2616272432 conn(0x7ff67406a800 msgr2=0x7ff67406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 -- 172.21.15.53:0/2616272432 shutdown_connections 2024-01-16T15:04:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.625+0000 7ff679400640 1 -- 172.21.15.53:0/2616272432 wait complete. 2024-01-16T15:04:52.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.627+0000 7ff679400640 1 Processor -- start 2024-01-16T15:04:52.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.627+0000 7ff679400640 1 -- start start 2024-01-16T15:04:52.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff679400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff6741f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff679400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff679400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff67415f9c0 0x7ff6740b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff679400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6741f2a00 con 0x7ff67413df80 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff679400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6741341c0 con 0x7ff67415f9c0 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff679400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6741344a0 con 0x7ff67413e360 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff672800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff6741f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff672800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff6741f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34856/0 (socket says 172.21.15.53:34856) 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.628+0000 7ff672800640 1 -- 172.21.15.53:0/1378911936 learned_addr learned my addr 172.21.15.53:0/1378911936 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff66be00640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff673200640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff67415f9c0 0x7ff6740b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff672800640 1 -- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 msgr2=0x7ff6740b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff672800640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff672800640 1 -- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff67415f9c0 msgr2=0x7ff6740b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff672800640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff67415f9c0 0x7ff6740b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff672800640 1 -- 172.21.15.53:0/1378911936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664009400 con 0x7ff67413df80 2024-01-16T15:04:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff672800640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff6741f24c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff66402b910 tx=0x7ff6640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66403a8f0 con 0x7ff67413df80 2024-01-16T15:04:52.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66403aa70 con 0x7ff67413df80 2024-01-16T15:04:52.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66403ae20 con 0x7ff67413df80 2024-01-16T15:04:52.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.629+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff674134780 con 0x7ff67413df80 2024-01-16T15:04:52.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.630+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff674134d40 con 0x7ff67413df80 2024-01-16T15:04:52.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.631+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6740675d0 con 0x7ff67413df80 2024-01-16T15:04:52.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.632+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7ff66404d450 con 0x7ff67413df80 2024-01-16T15:04:52.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.632+0000 7ff66aa00640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff64c06b7d0 0x7ff64c06dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:52.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.633+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7ff6640ac590 con 0x7ff67413df80 2024-01-16T15:04:52.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.633+0000 7ff66be00640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff64c06b7d0 0x7ff64c06dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:52.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.634+0000 7ff66be00640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff64c06b7d0 0x7ff64c06dc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff6740b3b80 tx=0x7ff65c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:52.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.637+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff664076c20 con 0x7ff67413df80 2024-01-16T15:04:52.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.853+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff6740ce850 con 0x7ff64c06b7d0 2024-01-16T15:04:52.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.857+0000 7ff66aa00640 1 -- 172.21.15.53:0/1378911936 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff65c00a660 con 0x7ff64c06b7d0 2024-01-16T15:04:52.836 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:04:52.836 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:52.836 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:04:52.836 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:52.836 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.858+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff64c06b7d0 msgr2=0x7ff64c06dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.858+0000 7ff679400640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff64c06b7d0 0x7ff64c06dc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff6740b3b80 tx=0x7ff65c008040 comp rx=0 tx=0).stop 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 msgr2=0x7ff6741f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff6741f24c0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7ff66402b910 tx=0x7ff6640046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 shutdown_connections 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff64c06b7d0 0x7ff64c06dc90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff67415f9c0 0x7ff6740b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff67413e360 0x7ff6740b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 --2- 172.21.15.53:0/1378911936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff67413df80 0x7ff6741f24c0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:52.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 >> 172.21.15.53:0/1378911936 conn(0x7ff67406a800 msgr2=0x7ff6740b1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:52.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 shutdown_connections 2024-01-16T15:04:52.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:52.859+0000 7ff679400640 1 -- 172.21.15.53:0/1378911936 wait complete. 2024-01-16T15:04:52.858 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:04:53.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.104+0000 7f1d5be00640 1 Processor -- start 2024-01-16T15:04:53.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.104+0000 7f1d5be00640 1 -- start start 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.106+0000 7f1d5be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1d5c0cf9a0 0x7f1d5c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.106+0000 7f1d5be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0d0320 con 0x7f1d5c0b31d0 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.106+0000 7f1d5be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d5c0d1c60 con 0x7f1d5c15d4b0 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.106+0000 7f1d5be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b2730 con 0x7f1d5c0cf9a0 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.106+0000 7f1d5b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1d5c0b31d0 0x7f1d5c0b3660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38586/0 (socket says 172.21.15.53:38586) 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.106+0000 7f1d5b400640 1 -- 172.21.15.53:0/3642186781 learned_addr learned my addr 172.21.15.53:0/3642186781 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:53.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d5a000640 1 --2- 172.21.15.53:0/3642186781 >> v2:172.21.15.146:3300/0 conn(0x7f1d5c0cf9a0 0x7f1d5c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541039675 0 0) 0x7f1d5c0d0320 con 0x7f1d5c0b31d0 2024-01-16T15:04:53.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d3c003520 con 0x7f1d5c0b31d0 2024-01-16T15:04:53.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1951401748 0 0) 0x7f1d3c003520 con 0x7f1d5c0b31d0 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 >> v2:172.21.15.146:3300/0 conn(0x7f1d5c0cf9a0 msgr2=0x7f1d5c0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 --2- 172.21.15.53:0/3642186781 >> v2:172.21.15.146:3300/0 conn(0x7f1d5c0cf9a0 0x7f1d5c0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 >> v1:172.21.15.146:6789/0 conn(0x7f1d5c15d4b0 legacy=0x7f1d5c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0d1ec0 con 0x7f1d5c0b31d0 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1d540019f0 con 0x7f1d5c0b31d0 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 >> v1:172.21.15.53:6789/0 conn(0x7f1d5c0b31d0 legacy=0x7f1d5c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d3c003970 0x7f1d3c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.107+0000 7f1d59600640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c0047c0 0x7f1d3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d59600640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c005180 0x7f1d3c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b2730 con 0x7f1d3c003970 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c00c020 con 0x7f1d3c0047c0 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c00c1e0 con 0x7f1d3c005180 2024-01-16T15:04:53.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d3c003970 0x7f1d3c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5a000640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c0047c0 0x7f1d3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5aa00640 1 -- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c005180 msgr2=0x7f1d3c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c005180 0x7f1d3c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5aa00640 1 -- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c0047c0 msgr2=0x7f1d3c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c0047c0 0x7f1d3c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5aa00640 1 -- 172.21.15.53:0/3642186781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c003520 con 0x7f1d3c003970 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5a000640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c0047c0 0x7f1d3c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:53.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.108+0000 7f1d5b400640 1 -- 172.21.15.53:0/3642186781 reap_dead start 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.109+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d3c003970 0x7f1d3c003d90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c001750 tx=0x7f1d3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f4c8f24999a4d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c0052d0 con 0x7f1d3c003970 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d4c02e050 con 0x7f1d3c003970 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d59600640 1 -- 172.21.15.53:0/3642186781 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4c003830 con 0x7f1d3c003970 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d5be00640 1 -- 172.21.15.53:0/3642186781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d3c003970 msgr2=0x7f1d3c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d5be00640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d3c003970 0x7f1d3c003d90 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c001750 tx=0x7f1d3c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d5be00640 1 -- 172.21.15.53:0/3642186781 shutdown_connections 2024-01-16T15:04:53.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d5be00640 1 --2- 172.21.15.53:0/3642186781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d3c003970 0x7f1d3c003d90 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.110+0000 7f1d5be00640 1 -- 172.21.15.53:0/3642186781 >> 172.21.15.53:0/3642186781 conn(0x7f1d5c0d3000 msgr2=0x7f1d5c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.111+0000 7f1d5be00640 1 -- 172.21.15.53:0/3642186781 shutdown_connections 2024-01-16T15:04:53.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.111+0000 7f1d5be00640 1 -- 172.21.15.53:0/3642186781 wait complete. 2024-01-16T15:04:53.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.112+0000 7f1d5be00640 1 Processor -- start 2024-01-16T15:04:53.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.113+0000 7f1d5be00640 1 -- start start 2024-01-16T15:04:53.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.113+0000 7f1d5be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c003970 0x7f1d5c0cae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.113+0000 7f1d5be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5c0b11f0 0x7f1d5c1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 0x7f1d5c0ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b1a20 con 0x7f1d5c0cc670 2024-01-16T15:04:53.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b1d00 con 0x7f1d5c0b11f0 2024-01-16T15:04:53.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d5c0b1fe0 con 0x7f1d3c003970 2024-01-16T15:04:53.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 0x7f1d5c0ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 0x7f1d5c0ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34878/0 (socket says 172.21.15.53:34878) 2024-01-16T15:04:53.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5b400640 1 -- 172.21.15.53:0/226433176 learned_addr learned my addr 172.21.15.53:0/226433176 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5a000640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5c0b11f0 0x7f1d5c1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c003970 0x7f1d5c0cae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5b400640 1 -- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c003970 msgr2=0x7f1d5c0cae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5b400640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c003970 0x7f1d5c0cae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.114+0000 7f1d5b400640 1 -- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5c0b11f0 msgr2=0x7f1d5c1f2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d5b400640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5c0b11f0 0x7f1d5c1f2550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d5b400640 1 -- 172.21.15.53:0/226433176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0b6a90 con 0x7f1d5c0cc670 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d5b400640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 0x7f1d5c0ccaf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c06aaa0 tx=0x7f1d5400cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400ce50 con 0x7f1d5c0cc670 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d5400a610 con 0x7f1d5c0cc670 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d5c0b6d70 con 0x7f1d5c0cc670 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5400a9c0 con 0x7f1d5c0cc670 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.115+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d5c0cd030 con 0x7f1d5c0cc670 2024-01-16T15:04:53.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.117+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d5c0b6f60 con 0x7f1d5c0cc670 2024-01-16T15:04:53.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.117+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f1d5400ab20 con 0x7f1d5c0cc670 2024-01-16T15:04:53.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.118+0000 7f1d52a00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d3406b7d0 0x7f1d3406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.118+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1d5408a0f0 con 0x7f1d5c0cc670 2024-01-16T15:04:53.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.119+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d3406b7d0 0x7f1d3406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.120+0000 7f1d5aa00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d3406b7d0 0x7f1d3406dc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c033000 tx=0x7f1d4c009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.123+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d54054920 con 0x7f1d5c0cc670 2024-01-16T15:04:53.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.351+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d5c0b1e70 con 0x7f1d3406b7d0 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.367+0000 7f1d52a00640 1 -- 172.21.15.53:0/226433176 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+234 (secure 0 0 0) 0x7f1d5c0b1e70 con 0x7f1d3406b7d0 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone48374" 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone48372" 2024-01-16T15:04:53.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone48371" 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone48373" 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:53.348 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d3406b7d0 msgr2=0x7f1d3406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d3406b7d0 0x7f1d3406dc90 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c033000 tx=0x7f1d4c009030 comp rx=0 tx=0).stop 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 msgr2=0x7f1d5c0ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 0x7f1d5c0ccaf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c06aaa0 tx=0x7f1d5400cc00 comp rx=0 tx=0).stop 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 shutdown_connections 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d3406b7d0 0x7f1d3406dc90 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d5c0cc670 0x7f1d5c0ccaf0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d5c0b11f0 0x7f1d5c1f2550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 --2- 172.21.15.53:0/226433176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c003970 0x7f1d5c0cae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 >> 172.21.15.53:0/226433176 conn(0x7f1d5c0d3000 msgr2=0x7f1d5c06c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 shutdown_connections 2024-01-16T15:04:53.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.370+0000 7f1d5be00640 1 -- 172.21.15.53:0/226433176 wait complete. 2024-01-16T15:04:53.370 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone48374 2024-01-16T15:04:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.611+0000 7f75a9e00640 1 Processor -- start 2024-01-16T15:04:53.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.611+0000 7f75a9e00640 1 -- start start 2024-01-16T15:04:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.613+0000 7f75a9e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f75a40ce8f0 0x7f75a40ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.613+0000 7f75a9e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a40d0de0 con 0x7f75a40b2a00 2024-01-16T15:04:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.613+0000 7f75a9e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75a40d10c0 con 0x7f75a4160f80 2024-01-16T15:04:53.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.613+0000 7f75a9e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75a40d13a0 con 0x7f75a40ce8f0 2024-01-16T15:04:53.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.613+0000 7f75a3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f75a40b2a00 0x7f75a40b2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38588/0 (socket says 172.21.15.53:38588) 2024-01-16T15:04:53.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.613+0000 7f75a3400640 1 -- 172.21.15.53:0/3380058680 learned_addr learned my addr 172.21.15.53:0/3380058680 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:53.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f759be00640 1 --2- 172.21.15.53:0/3380058680 >> v2:172.21.15.146:3300/0 conn(0x7f75a40ce8f0 0x7f75a40ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027586944 0 0) 0x7f75a40d0de0 con 0x7f75a40b2a00 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584003520 con 0x7f75a40b2a00 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77369613 0 0) 0x7f7584003520 con 0x7f75a40b2a00 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 >> v2:172.21.15.146:3300/0 conn(0x7f75a40ce8f0 msgr2=0x7f75a40ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 --2- 172.21.15.53:0/3380058680 >> v2:172.21.15.146:3300/0 conn(0x7f75a40ce8f0 0x7f75a40ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 >> v1:172.21.15.146:6789/0 conn(0x7f75a4160f80 legacy=0x7f75a4161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40d1680 con 0x7f75a40b2a00 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f759c0019f0 con 0x7f75a40b2a00 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 >> v1:172.21.15.53:6789/0 conn(0x7f75a40b2a00 legacy=0x7f75a40b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7584003970 0x7f7584003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7584005180 0x7f758400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40d13a0 con 0x7f7584003970 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758400c020 con 0x7f75840047c0 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.614+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758400c1e0 con 0x7f7584005180 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a2a00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7584003970 0x7f7584003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f759be00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a2a00640 1 -- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7584005180 msgr2=0x7f758400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a2a00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7584005180 0x7f758400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a2a00640 1 -- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75840047c0 msgr2=0x7f7584004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a2a00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a2a00640 1 -- 172.21.15.53:0/3380058680 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584003520 con 0x7f7584003970 2024-01-16T15:04:53.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.615+0000 7f75a3400640 1 -- 172.21.15.53:0/3380058680 reap_dead start 2024-01-16T15:04:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.616+0000 7f75a2a00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7584003970 0x7f7584003d90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7594001750 tx=0x7f75840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e0b17bfbc72bd0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940052d0 con 0x7f7584003970 2024-01-16T15:04:53.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759402e050 con 0x7f7584003970 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a2000640 1 -- 172.21.15.53:0/3380058680 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7594003830 con 0x7f7584003970 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a9e00640 1 -- 172.21.15.53:0/3380058680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7584003970 msgr2=0x7f7584003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a9e00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7584003970 0x7f7584003d90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7594001750 tx=0x7f75840057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a9e00640 1 -- 172.21.15.53:0/3380058680 shutdown_connections 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a9e00640 1 --2- 172.21.15.53:0/3380058680 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7584003970 0x7f7584003d90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.617+0000 7f75a9e00640 1 -- 172.21.15.53:0/3380058680 >> 172.21.15.53:0/3380058680 conn(0x7f75a409c550 msgr2=0x7f75a40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.618+0000 7f75a9e00640 1 -- 172.21.15.53:0/3380058680 shutdown_connections 2024-01-16T15:04:53.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.618+0000 7f75a9e00640 1 -- 172.21.15.53:0/3380058680 wait complete. 2024-01-16T15:04:53.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.619+0000 7f75a9e00640 1 Processor -- start 2024-01-16T15:04:53.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.620+0000 7f75a9e00640 1 -- start start 2024-01-16T15:04:53.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.620+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7584003970 0x7f75a41516d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.620+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 0x7f75a4151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f75a9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75a40caca0 0x7f75a40b0e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f75a9e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4151550 con 0x7f75a4152490 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f75a9e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a4152910 con 0x7f7584003970 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f75a9e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75a40cb520 con 0x7f75a40caca0 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 0x7f75a4151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 0x7f75a4151010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34900/0 (socket says 172.21.15.53:34900) 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 -- 172.21.15.53:0/1921302372 learned_addr learned my addr 172.21.15.53:0/1921302372 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 -- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75a40caca0 msgr2=0x7f75a40b0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75a40caca0 0x7f75a40b0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 -- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7584003970 msgr2=0x7f75a41516d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7584003970 0x7f75a41516d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.621+0000 7f759be00640 1 -- 172.21.15.53:0/1921302372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75a40c7e30 con 0x7f75a4152490 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.622+0000 7f759be00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 0x7f75a4151010 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f758c011700 tx=0x7f758c011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.622+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c00f9c0 con 0x7f75a4152490 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.622+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f758c00fb20 con 0x7f75a4152490 2024-01-16T15:04:53.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.622+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f758c00fe90 con 0x7f75a4152490 2024-01-16T15:04:53.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.622+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a40c8110 con 0x7f75a4152490 2024-01-16T15:04:53.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.622+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a40cb800 con 0x7f75a4152490 2024-01-16T15:04:53.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.623+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a40675d0 con 0x7f75a4152490 2024-01-16T15:04:53.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.625+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f758c015460 con 0x7f75a4152490 2024-01-16T15:04:53.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.626+0000 7f759b400640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f757c06b800 0x7f757c06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:53.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.626+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f758c08a8e0 con 0x7f75a4152490 2024-01-16T15:04:53.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.626+0000 7f75a2a00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f757c06b800 0x7f757c06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:53.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.627+0000 7f75a2a00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f757c06b800 0x7f757c06dcc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7594033000 tx=0x7f759402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:53.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.630+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f758c055080 con 0x7f75a4152490 2024-01-16T15:04:53.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.849+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone48374", "target": ["mon-mgr", ""]}) v1 -- 0x7f75a40d1c40 con 0x7f757c06b800 2024-01-16T15:04:53.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.900+0000 7f759b400640 1 -- 172.21.15.53:0/1921302372 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7594007a30 con 0x7f757c06b800 2024-01-16T15:04:53.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.902+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f757c06b800 msgr2=0x7f757c06dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.902+0000 7f75a9e00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f757c06b800 0x7f757c06dcc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f7594033000 tx=0x7f759402f000 comp rx=0 tx=0).stop 2024-01-16T15:04:53.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.902+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 msgr2=0x7f75a4151010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:53.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.902+0000 7f75a9e00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 0x7f75a4151010 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f758c011700 tx=0x7f758c011bc0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 shutdown_connections 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f757c06b800 0x7f757c06dcc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f75a40caca0 0x7f75a40b0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f75a4152490 0x7f75a4151010 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 --2- 172.21.15.53:0/1921302372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7584003970 0x7f75a41516d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 >> 172.21.15.53:0/1921302372 conn(0x7f75a409c550 msgr2=0x7f75a406c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 shutdown_connections 2024-01-16T15:04:53.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:53.903+0000 7f75a9e00640 1 -- 172.21.15.53:0/1921302372 wait complete. 2024-01-16T15:04:53.889 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.910+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.889 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.910+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.889 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.910+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.889 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.910+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.889 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.910+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.907 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:04:53.923 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.945+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.923 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.945+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.924 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.945+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.924 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.945+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:53.924 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:04:53.945+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:04:54.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.149+0000 7f47e3a00640 1 Processor -- start 2024-01-16T15:04:54.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.149+0000 7f47e3a00640 1 -- start start 2024-01-16T15:04:54.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e3a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f47dc0cf9a0 0x7f47dc15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e3a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc070c40 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e3a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47dc06c470 con 0x7f47dc0ae5a0 2024-01-16T15:04:54.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e3a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47dc071450 con 0x7f47dc0cf9a0 2024-01-16T15:04:54.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47d3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f47dc0b3ae0 0x7f47dc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38596/0 (socket says 172.21.15.53:38596) 2024-01-16T15:04:54.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47d3e00640 1 -- 172.21.15.53:0/2865676331 learned_addr learned my addr 172.21.15.53:0/2865676331 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:54.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e1a00640 1 --2- 172.21.15.53:0/2865676331 >> v2:172.21.15.53:3300/0 conn(0x7f47dc0cf9a0 0x7f47dc15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e1a00640 1 -- 172.21.15.53:0/2865676331 >> v1:172.21.15.146:6789/0 conn(0x7f47dc0ae5a0 legacy=0x7f47dc06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.151+0000 7f47e1a00640 1 -- 172.21.15.53:0/2865676331 >> v1:172.21.15.53:6789/0 conn(0x7f47dc0b3ae0 legacy=0x7f47dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.152+0000 7f47e1a00640 1 -- 172.21.15.53:0/2865676331 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc06c750 con 0x7f47dc0cf9a0 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.152+0000 7f47dbe00640 1 -- 172.21.15.53:0/2865676331 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559874903 0 0) 0x7f47dc070c40 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.153+0000 7f47e1a00640 1 --2- 172.21.15.53:0/2865676331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc15e430 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009d90 tx=0x7f47d402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88d05f9e2d56259 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.153+0000 7f47dbe00640 1 -- 172.21.15.53:0/2865676331 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d402cd00 con 0x7f47dc0cf9a0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.153+0000 7f47dbe00640 1 -- 172.21.15.53:0/2865676331 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d402ce60 con 0x7f47dc0cf9a0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.153+0000 7f47dbe00640 1 -- 172.21.15.53:0/2865676331 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4034660 con 0x7f47dc0cf9a0 2024-01-16T15:04:54.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 -- 172.21.15.53:0/2865676331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0cf9a0 msgr2=0x7f47dc15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 --2- 172.21.15.53:0/2865676331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc15e430 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f47d4009d90 tx=0x7f47d402bf50 comp rx=0 tx=0).stop 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 -- 172.21.15.53:0/2865676331 shutdown_connections 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 --2- 172.21.15.53:0/2865676331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc15e430 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 -- 172.21.15.53:0/2865676331 >> 172.21.15.53:0/2865676331 conn(0x7f47dc06a800 msgr2=0x7f47dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 -- 172.21.15.53:0/2865676331 shutdown_connections 2024-01-16T15:04:54.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.154+0000 7f47e3a00640 1 -- 172.21.15.53:0/2865676331 wait complete. 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.156+0000 7f47e3a00640 1 Processor -- start 2024-01-16T15:04:54.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.156+0000 7f47e3a00640 1 -- start start 2024-01-16T15:04:54.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.157+0000 7f47e3a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47dc0ae5a0 0x7f47dc149e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.157+0000 7f47e3a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 0x7f47dc0cdae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.157+0000 7f47e3a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc0cb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47e3a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc1484a0 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47e3a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc0ce610 con 0x7f47dc0cf9a0 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47e3a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47dc14a3b0 con 0x7f47dc0ae5a0 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 0x7f47dc0cdae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 0x7f47dc0cdae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34912/0 (socket says 172.21.15.53:34912) 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47d3e00640 1 -- 172.21.15.53:0/124244505 learned_addr learned my addr 172.21.15.53:0/124244505 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47e1000640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47dc0ae5a0 0x7f47dc149e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.158+0000 7f47e1a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc0cb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d3e00640 1 -- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47dc0ae5a0 msgr2=0x7f47dc149e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d3e00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47dc0ae5a0 0x7f47dc149e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d3e00640 1 -- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47dc0cf9a0 msgr2=0x7f47dc0cb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d3e00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc0cb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d3e00640 1 -- 172.21.15.53:0/124244505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d4009a70 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d3e00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 0x7f47dc0cdae0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f47c400c530 tx=0x7f47c400c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.159+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c40074e0 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.160+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c4007660 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.160+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c4004a20 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.160+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47dc147890 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.160+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47dc147e50 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.161+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47dc148650 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.162+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 13) v1 ==== 89998+0+0 (secure 0 0 0) 0x7f47c4004b80 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.163+0000 7f47d9600640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b406b7d0 0x7f47b406dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.164+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f47c40894b0 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.164+0000 7f47e1000640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b406b7d0 0x7f47b406dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:54.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.165+0000 7f47e1000640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b406b7d0 0x7f47b406dc90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f47cc004f60 tx=0x7f47cc007e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:54.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.168+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47dc148650 con 0x7f47dc0b3ae0 2024-01-16T15:04:54.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:54.396+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f47dc070550 con 0x7f47b406b7d0 2024-01-16T15:04:55.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:55.125+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f47c40541a0 con 0x7f47dc0b3ae0 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.374+0000 7f47d9600640 1 -- 172.21.15.53:0/124244505 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f47cc003940 con 0x7f47b406b7d0 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:04:57.354 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b406b7d0 msgr2=0x7f47b406dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b406b7d0 0x7f47b406dc90 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f47cc004f60 tx=0x7f47cc007e30 comp rx=0 tx=0).stop 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 msgr2=0x7f47dc0cdae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 0x7f47dc0cdae0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f47c400c530 tx=0x7f47c400c9f0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 shutdown_connections 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b406b7d0 0x7f47b406dc90 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47dc0cf9a0 0x7f47dc0cb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47dc0b3ae0 0x7f47dc0cdae0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 --2- 172.21.15.53:0/124244505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47dc0ae5a0 0x7f47dc149e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 >> 172.21.15.53:0/124244505 conn(0x7f47dc06a800 msgr2=0x7f47dc06cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.377+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 shutdown_connections 2024-01-16T15:04:57.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.378+0000 7f47e3a00640 1 -- 172.21.15.53:0/124244505 wait complete. 2024-01-16T15:04:57.378 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:04:57.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.628+0000 7f41dd200640 1 Processor -- start 2024-01-16T15:04:57.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.628+0000 7f41dd200640 1 -- start start 2024-01-16T15:04:57.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41dd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41dd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f41d80cf7d0 0x7f41d8070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41dd200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f41d8070ea0 0x7f41d80b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41dd200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d814b6a0 con 0x7f41d8070ea0 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41dd200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d80cfbf0 con 0x7f41d80cf7d0 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41dd200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8071690 con 0x7f41d814b2c0 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41d6800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f41d8070ea0 0x7f41d80b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41d6800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f41d8070ea0 0x7f41d80b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34928/0 (socket says 172.21.15.53:34928) 2024-01-16T15:04:57.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.630+0000 7f41d6800640 1 -- 172.21.15.53:0/3948679081 learned_addr learned my addr 172.21.15.53:0/3948679081 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d6800640 1 -- 172.21.15.53:0/3948679081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 msgr2=0x7f41d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d5e00640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d6800640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d6800640 1 -- 172.21.15.53:0/3948679081 >> v2:172.21.15.146:3300/0 conn(0x7f41d80cf7d0 msgr2=0x7f41d8070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d5400640 1 --2- 172.21.15.53:0/3948679081 >> v2:172.21.15.146:3300/0 conn(0x7f41d80cf7d0 0x7f41d8070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d6800640 1 --2- 172.21.15.53:0/3948679081 >> v2:172.21.15.146:3300/0 conn(0x7f41d80cf7d0 0x7f41d8070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d6800640 1 -- 172.21.15.53:0/3948679081 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d8071970 con 0x7f41d8070ea0 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d5e00640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:57.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.631+0000 7f41d5400640 1 --2- 172.21.15.53:0/3948679081 >> v2:172.21.15.146:3300/0 conn(0x7f41d80cf7d0 0x7f41d8070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:57.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.632+0000 7f41d6800640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d80b1ee0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f41d80cfed0 tx=0x7f41d002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83a59b744c6d9e1c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.632+0000 7f41d4a00640 1 -- 172.21.15.53:0/3948679081 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d002ca80 con 0x7f41d8070ea0 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41d4a00640 1 -- 172.21.15.53:0/3948679081 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d002cbe0 con 0x7f41d8070ea0 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41d4a00640 1 -- 172.21.15.53:0/3948679081 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41d00343f0 con 0x7f41d8070ea0 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 -- 172.21.15.53:0/3948679081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 msgr2=0x7f41d80b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d80b1ee0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f41d80cfed0 tx=0x7f41d002bee0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 -- 172.21.15.53:0/3948679081 shutdown_connections 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d80b1ee0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 --2- 172.21.15.53:0/3948679081 >> v2:172.21.15.146:3300/0 conn(0x7f41d80cf7d0 0x7f41d8070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 --2- 172.21.15.53:0/3948679081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 -- 172.21.15.53:0/3948679081 >> 172.21.15.53:0/3948679081 conn(0x7f41d806a800 msgr2=0x7f41d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 -- 172.21.15.53:0/3948679081 shutdown_connections 2024-01-16T15:04:57.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.633+0000 7f41dd200640 1 -- 172.21.15.53:0/3948679081 wait complete. 2024-01-16T15:04:57.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.635+0000 7f41dd200640 1 Processor -- start 2024-01-16T15:04:57.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.635+0000 7f41dd200640 1 -- start start 2024-01-16T15:04:57.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41dd200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d8161420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41dd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41d80cf7d0 0x7f41d8163c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41dd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80abb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41dd200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8161960 con 0x7f41d8070ea0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41dd200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8161e10 con 0x7f41d80cf7d0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41dd200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d81620f0 con 0x7f41d814b2c0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.636+0000 7f41d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d8161420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d8161420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34940/0 (socket says 172.21.15.53:34940) 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 -- 172.21.15.53:0/2574725810 learned_addr learned my addr 172.21.15.53:0/2574725810 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5400640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41d80cf7d0 0x7f41d8163c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d6800640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80abb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 -- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 msgr2=0x7f41d80abb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80abb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 -- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41d80cf7d0 msgr2=0x7f41d8163c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41d80cf7d0 0x7f41d8163c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 -- 172.21.15.53:0/2574725810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d0009b30 con 0x7f41d8070ea0 2024-01-16T15:04:57.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.637+0000 7f41d5e00640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d8161420 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c7f0 tx=0x7f41c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.638+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c8007a30 con 0x7f41d8070ea0 2024-01-16T15:04:57.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.638+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c8007b90 con 0x7f41d8070ea0 2024-01-16T15:04:57.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.638+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c800ce40 con 0x7f41d8070ea0 2024-01-16T15:04:57.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.638+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d81623d0 con 0x7f41d8070ea0 2024-01-16T15:04:57.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.638+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d8162990 con 0x7f41d8070ea0 2024-01-16T15:04:57.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.639+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d80675d0 con 0x7f41d8070ea0 2024-01-16T15:04:57.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.640+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f41c80113f0 con 0x7f41d8070ea0 2024-01-16T15:04:57.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.641+0000 7f41ce000640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41b006b970 0x7f41b006de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:57.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.641+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f41c808a9f0 con 0x7f41d8070ea0 2024-01-16T15:04:57.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.641+0000 7f41d5400640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41b006b970 0x7f41b006de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:57.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.642+0000 7f41d5400640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41b006b970 0x7f41b006de30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f41d8164fd0 tx=0x7f41c0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:57.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.645+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41c8054e10 con 0x7f41d8070ea0 2024-01-16T15:04:57.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.873+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f41d80cbc40 con 0x7f41b006b970 2024-01-16T15:04:57.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.947+0000 7f41ce000640 1 -- 172.21.15.53:0/2574725810 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f41c000c050 con 0x7f41b006b970 2024-01-16T15:04:57.927 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41b006b970 msgr2=0x7f41b006de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41b006b970 0x7f41b006de30 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f41d8164fd0 tx=0x7f41c0008040 comp rx=0 tx=0).stop 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 msgr2=0x7f41d8161420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d8161420 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f41c800c7f0 tx=0x7f41c800cbf0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 shutdown_connections 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41b006b970 0x7f41b006de30 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41d814b2c0 0x7f41d80abb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41d80cf7d0 0x7f41d8163c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 --2- 172.21.15.53:0/2574725810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41d8070ea0 0x7f41d8161420 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 >> 172.21.15.53:0/2574725810 conn(0x7f41d806a800 msgr2=0x7f41d80aece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 shutdown_connections 2024-01-16T15:04:57.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:57.951+0000 7f41dd200640 1 -- 172.21.15.53:0/2574725810 wait complete. 2024-01-16T15:04:57.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:04:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.191+0000 7f974bc00640 1 Processor -- start 2024-01-16T15:04:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.191+0000 7f974bc00640 1 -- start start 2024-01-16T15:04:58.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.193+0000 7f974bc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f974413df80 0x7f974415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.193+0000 7f974bc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413e360 0x7f97440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.193+0000 7f974bc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974406c470 con 0x7f974415f9c0 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.193+0000 7f974bc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f974406c750 con 0x7f974413df80 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.193+0000 7f974bc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f974406ca30 con 0x7f974413e360 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f974413df80 0x7f974415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f974413df80 0x7f974415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34948/0 (socket says 172.21.15.53:34948) 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 -- 172.21.15.53:0/1552576289 learned_addr learned my addr 172.21.15.53:0/1552576289 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f973be00640 1 --2- 172.21.15.53:0/1552576289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413e360 0x7f97440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 -- 172.21.15.53:0/1552576289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413e360 msgr2=0x7f97440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 --2- 172.21.15.53:0/1552576289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413e360 0x7f97440cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 -- 172.21.15.53:0/1552576289 >> v1:172.21.15.53:6789/0 conn(0x7f974415f9c0 legacy=0x7f97441408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.194+0000 7f9749200640 1 -- 172.21.15.53:0/1552576289 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f974406cd10 con 0x7f974413df80 2024-01-16T15:04:58.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.195+0000 7f9749200640 1 --2- 172.21.15.53:0/1552576289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413df80 0x7f974415bee0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9734009870 tx=0x7f973402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ea74dd479698c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.196+0000 7f9743e00640 1 -- 172.21.15.53:0/1552576289 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734030840 con 0x7f974413df80 2024-01-16T15:04:58.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.196+0000 7f9743e00640 1 -- 172.21.15.53:0/1552576289 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97340309a0 con 0x7f974413df80 2024-01-16T15:04:58.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.196+0000 7f9743e00640 1 -- 172.21.15.53:0/1552576289 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9734030d30 con 0x7f974413df80 2024-01-16T15:04:58.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 -- 172.21.15.53:0/1552576289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413df80 msgr2=0x7f974415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 --2- 172.21.15.53:0/1552576289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413df80 0x7f974415bee0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f9734009870 tx=0x7f973402b980 comp rx=0 tx=0).stop 2024-01-16T15:04:58.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 -- 172.21.15.53:0/1552576289 shutdown_connections 2024-01-16T15:04:58.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 --2- 172.21.15.53:0/1552576289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413e360 0x7f97440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 --2- 172.21.15.53:0/1552576289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413df80 0x7f974415bee0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 -- 172.21.15.53:0/1552576289 >> 172.21.15.53:0/1552576289 conn(0x7f974406a800 msgr2=0x7f974406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 -- 172.21.15.53:0/1552576289 shutdown_connections 2024-01-16T15:04:58.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.197+0000 7f974bc00640 1 -- 172.21.15.53:0/1552576289 wait complete. 2024-01-16T15:04:58.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.199+0000 7f974bc00640 1 Processor -- start 2024-01-16T15:04:58.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.199+0000 7f974bc00640 1 -- start start 2024-01-16T15:04:58.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.200+0000 7f974bc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413df80 0x7f974413d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f974bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 0x7f9744151550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f974bc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f974415f9c0 0x7f9744151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f974bc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f974413de10 con 0x7f974413e360 2024-01-16T15:04:58.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f974bc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9744153360 con 0x7f974415f9c0 2024-01-16T15:04:58.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f974bc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97441f3010 con 0x7f974413df80 2024-01-16T15:04:58.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 0x7f9744151550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 0x7f9744151550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34954/0 (socket says 172.21.15.53:34954) 2024-01-16T15:04:58.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 -- 172.21.15.53:0/3787873272 learned_addr learned my addr 172.21.15.53:0/3787873272 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f9749200640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413df80 0x7f974413d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f9749c00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f974415f9c0 0x7f9744151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 -- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413df80 msgr2=0x7f974413d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413df80 0x7f974413d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 -- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f974415f9c0 msgr2=0x7f9744151a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f974415f9c0 0x7f9744151a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f973be00640 1 -- 172.21.15.53:0/3787873272 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97340094d0 con 0x7f974413e360 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f9749200640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413df80 0x7f974413d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.201+0000 7f9749c00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f974415f9c0 0x7f9744151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.202+0000 7f973be00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 0x7f9744151550 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c7f0 tx=0x7f972c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.202+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c004220 con 0x7f974413e360 2024-01-16T15:04:58.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.202+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c0043a0 con 0x7f974413e360 2024-01-16T15:04:58.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.202+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c007860 con 0x7f974413e360 2024-01-16T15:04:58.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.202+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97440b2f50 con 0x7f974413e360 2024-01-16T15:04:58.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.202+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97440b3510 con 0x7f974413e360 2024-01-16T15:04:58.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.203+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97441f3250 con 0x7f974413e360 2024-01-16T15:04:58.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.205+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f972c007bd0 con 0x7f974413e360 2024-01-16T15:04:58.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.206+0000 7f9741600640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f971c06b920 0x7f971c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.206+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f972c08a6a0 con 0x7f974413e360 2024-01-16T15:04:58.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.206+0000 7f9749200640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f971c06b920 0x7f971c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.208+0000 7f9749200640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f971c06b920 0x7f971c06dde0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9734036000 tx=0x7f973403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.211+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f97441f3250 con 0x7f974413e360 2024-01-16T15:04:58.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.437+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f97440ce850 con 0x7f971c06b920 2024-01-16T15:04:58.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.487+0000 7f9741600640 1 -- 172.21.15.53:0/3787873272 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9734030b90 con 0x7f971c06b920 2024-01-16T15:04:58.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f971c06b920 msgr2=0x7f971c06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f971c06b920 0x7f971c06dde0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f9734036000 tx=0x7f973403b040 comp rx=0 tx=0).stop 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 msgr2=0x7f9744151550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 0x7f9744151550 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f972c00c7f0 tx=0x7f972c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 shutdown_connections 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f971c06b920 0x7f971c06dde0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f974415f9c0 0x7f9744151a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f974413e360 0x7f9744151550 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 --2- 172.21.15.53:0/3787873272 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f974413df80 0x7f974413d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 >> 172.21.15.53:0/3787873272 conn(0x7f974406a800 msgr2=0x7f974415e860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.490+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 shutdown_connections 2024-01-16T15:04:58.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.491+0000 7f974bc00640 1 -- 172.21.15.53:0/3787873272 wait complete. 2024-01-16T15:04:58.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone48372 2024-01-16T15:04:58.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.726+0000 7fada7e00640 1 Processor -- start 2024-01-16T15:04:58.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.726+0000 7fada7e00640 1 -- start start 2024-01-16T15:04:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fada7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8160920 0x7fada80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fada7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fada80cf9f0 0x7fada80cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fada7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fada8161010 0x7fada81614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fada7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80d03b0 con 0x7fada80cf9f0 2024-01-16T15:04:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fada7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fada80d0a10 con 0x7fada8161010 2024-01-16T15:04:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fada7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada80cf080 con 0x7fada8160920 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fad9fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fada80cf9f0 0x7fada80cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fad9fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fada80cf9f0 0x7fada80cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34968/0 (socket says 172.21.15.53:34968) 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.728+0000 7fad9fe00640 1 -- 172.21.15.53:0/3918408205 learned_addr learned my addr 172.21.15.53:0/3918408205 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fad9fe00640 1 -- 172.21.15.53:0/3918408205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8160920 msgr2=0x7fada80b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fada7400640 1 --2- 172.21.15.53:0/3918408205 >> v2:172.21.15.146:3300/0 conn(0x7fada8161010 0x7fada81614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fada6a00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8160920 0x7fada80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fad9fe00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8160920 0x7fada80b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fad9fe00640 1 -- 172.21.15.53:0/3918408205 >> v2:172.21.15.146:3300/0 conn(0x7fada8161010 msgr2=0x7fada81614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fad9fe00640 1 --2- 172.21.15.53:0/3918408205 >> v2:172.21.15.146:3300/0 conn(0x7fada8161010 0x7fada81614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fad9fe00640 1 -- 172.21.15.53:0/3918408205 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada80c6be0 con 0x7fada80cf9f0 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fada6a00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8160920 0x7fada80b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.729+0000 7fada7400640 1 --2- 172.21.15.53:0/3918408205 >> v2:172.21.15.146:3300/0 conn(0x7fada8161010 0x7fada81614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:04:58.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.730+0000 7fad9fe00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada80cf9f0 0x7fada80cfe70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fada80c6930 tx=0x7fad9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49aa08f0cdbaf92d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.730+0000 7fada6000640 1 -- 172.21.15.53:0/3918408205 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90031870 con 0x7fada80cf9f0 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.730+0000 7fada6000640 1 -- 172.21.15.53:0/3918408205 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad900319d0 con 0x7fada80cf9f0 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.730+0000 7fada6000640 1 -- 172.21.15.53:0/3918408205 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90031d60 con 0x7fada80cf9f0 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 -- 172.21.15.53:0/3918408205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada80cf9f0 msgr2=0x7fada80cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada80cf9f0 0x7fada80cfe70 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fada80c6930 tx=0x7fad9002b910 comp rx=0 tx=0).stop 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 -- 172.21.15.53:0/3918408205 shutdown_connections 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 --2- 172.21.15.53:0/3918408205 >> v2:172.21.15.146:3300/0 conn(0x7fada8161010 0x7fada81614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada80cf9f0 0x7fada80cfe70 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 --2- 172.21.15.53:0/3918408205 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8160920 0x7fada80b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 -- 172.21.15.53:0/3918408205 >> 172.21.15.53:0/3918408205 conn(0x7fada80d3090 msgr2=0x7fada80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 -- 172.21.15.53:0/3918408205 shutdown_connections 2024-01-16T15:04:58.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.731+0000 7fada7e00640 1 -- 172.21.15.53:0/3918408205 wait complete. 2024-01-16T15:04:58.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.734+0000 7fada7e00640 1 Processor -- start 2024-01-16T15:04:58.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.734+0000 7fada7e00640 1 -- start start 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fada7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada80cf9f0 0x7fada814d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fada7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 0x7fada80cc840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fada7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8161010 0x7fada81f32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fada7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada814da80 con 0x7fada8160920 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fada7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada80cb7a0 con 0x7fada80cf9f0 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fada7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fada80cba80 con 0x7fada8161010 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 0x7fada80cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 0x7fada80cc840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34970/0 (socket says 172.21.15.53:34970) 2024-01-16T15:04:58.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 -- 172.21.15.53:0/1973598178 learned_addr learned my addr 172.21.15.53:0/1973598178 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:58.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 -- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8161010 msgr2=0x7fada81f32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8161010 0x7fada81f32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 -- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada80cf9f0 msgr2=0x7fada814d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada80cf9f0 0x7fada814d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.736+0000 7fad9fe00640 1 -- 172.21.15.53:0/1973598178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad900094d0 con 0x7fada8160920 2024-01-16T15:04:58.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.737+0000 7fad9fe00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 0x7fada80cc840 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fad90000c00 tx=0x7fad900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.737+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90031840 con 0x7fada8160920 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.737+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad900443f0 con 0x7fada8160920 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.737+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada80cbd60 con 0x7fada8160920 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.737+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad900447a0 con 0x7fada8160920 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.737+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada80cc320 con 0x7fada8160920 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.738+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada80cc510 con 0x7fada8160920 2024-01-16T15:04:58.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.739+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fad9004d450 con 0x7fada8160920 2024-01-16T15:04:58.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.740+0000 7fad9ea00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad8006b8f0 0x7fad8006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:58.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.740+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7fad9002d040 con 0x7fada8160920 2024-01-16T15:04:58.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.741+0000 7fada6a00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad8006b8f0 0x7fad8006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:58.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.742+0000 7fada6a00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad8006b8f0 0x7fad8006ddb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fad9800a8b0 tx=0x7fad98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:58.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.745+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad90077470 con 0x7fada8160920 2024-01-16T15:04:58.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:58.966+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone48372", "target": ["mon-mgr", ""]}) v1 -- 0x7fada80ceda0 con 0x7fad8006b8f0 2024-01-16T15:04:58.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.009+0000 7fad9ea00640 1 -- 172.21.15.53:0/1973598178 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fad9800a4a0 con 0x7fad8006b8f0 2024-01-16T15:04:58.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.011+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad8006b8f0 msgr2=0x7fad8006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.011+0000 7fada7e00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad8006b8f0 0x7fad8006ddb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fad9800a8b0 tx=0x7fad98008040 comp rx=0 tx=0).stop 2024-01-16T15:04:58.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.011+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 msgr2=0x7fada80cc840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:58.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.011+0000 7fada7e00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 0x7fada80cc840 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fad90000c00 tx=0x7fad900046c0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 shutdown_connections 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad8006b8f0 0x7fad8006ddb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fada8161010 0x7fada81f32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fada8160920 0x7fada80cc840 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 --2- 172.21.15.53:0/1973598178 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fada80cf9f0 0x7fada814d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 >> 172.21.15.53:0/1973598178 conn(0x7fada80d3090 msgr2=0x7fada815aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 shutdown_connections 2024-01-16T15:04:58.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.012+0000 7fada7e00640 1 -- 172.21.15.53:0/1973598178 wait complete. 2024-01-16T15:04:59.010 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone48371 2024-01-16T15:04:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.237+0000 7f99ed000640 1 Processor -- start 2024-01-16T15:04:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.237+0000 7f99ed000640 1 -- start start 2024-01-16T15:04:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99ed000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99e815d3c0 0x7f99e80d0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99ed000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 0x7f99e80afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99ed000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99e80d12f0 con 0x7f99e80b01b0 2024-01-16T15:04:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99ed000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99e80ce5d0 con 0x7f99e815d3c0 2024-01-16T15:04:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99ed000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80b23c0 con 0x7f99e80d0e90 2024-01-16T15:04:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99e7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f99e80b01b0 0x7f99e80ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38612/0 (socket says 172.21.15.53:38612) 2024-01-16T15:04:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99e7e00640 1 -- 172.21.15.53:0/3019483944 learned_addr learned my addr 172.21.15.53:0/3019483944 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:04:59.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99e7400640 1 --2- 172.21.15.53:0/3019483944 >> v2:172.21.15.146:3300/0 conn(0x7f99e815d3c0 0x7f99e80d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.239+0000 7f99e6a00640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 0x7f99e80afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453628557 0 0) 0x7f99e0001110 con 0x7f99e80b01b0 2024-01-16T15:04:59.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c8003520 con 0x7f99e80b01b0 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864115212 0 0) 0x7f99c8003520 con 0x7f99e80b01b0 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 msgr2=0x7f99e80afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 0x7f99e80afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 >> v2:172.21.15.146:3300/0 conn(0x7f99e815d3c0 msgr2=0x7f99e80d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 --2- 172.21.15.53:0/3019483944 >> v2:172.21.15.146:3300/0 conn(0x7f99e815d3c0 0x7f99e80d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80d0800 con 0x7f99e80b01b0 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f99e0001800 con 0x7f99e80b01b0 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 >> v1:172.21.15.53:6789/0 conn(0x7f99e80b01b0 legacy=0x7f99e80ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80047c0 0x7f99c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 0x7f99c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80ce5d0 con 0x7f99c8003970 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80b23c0 con 0x7f99c80047c0 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c800b9d0 con 0x7f99e80d0e90 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.240+0000 7f99e7400640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.241+0000 7f99e6a00640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80047c0 0x7f99c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.241+0000 7f99e7400640 1 -- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 msgr2=0x7f99c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.241+0000 7f99e7400640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80d0e90 0x7f99c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.241+0000 7f99e7400640 1 -- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80047c0 msgr2=0x7f99c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.241+0000 7f99e7400640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80047c0 0x7f99c8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.241+0000 7f99e7400640 1 -- 172.21.15.53:0/3019483944 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c8003520 con 0x7f99c8003970 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.242+0000 7f99e7400640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99c8003d90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009af0 tx=0x7f99c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f13c58099d3411 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.242+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d802c710 con 0x7f99c8003970 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.242+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d802c870 con 0x7f99c8003970 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.242+0000 7f99e6000640 1 -- 172.21.15.53:0/3019483944 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d802cc00 con 0x7f99c8003970 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99ed000640 1 -- 172.21.15.53:0/3019483944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 msgr2=0x7f99c8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99ed000640 1 --2- 172.21.15.53:0/3019483944 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99c8003d90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f99d8009af0 tx=0x7f99c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99e7e00640 1 -- 172.21.15.53:0/3019483944 reap_dead start 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99ed000640 1 -- 172.21.15.53:0/3019483944 shutdown_connections 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99ed000640 1 -- 172.21.15.53:0/3019483944 >> 172.21.15.53:0/3019483944 conn(0x7f99e80d30b0 msgr2=0x7f99e80d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99ed000640 1 -- 172.21.15.53:0/3019483944 shutdown_connections 2024-01-16T15:04:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.243+0000 7f99ed000640 1 -- 172.21.15.53:0/3019483944 wait complete. 2024-01-16T15:04:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.245+0000 7f99ed000640 1 Processor -- start 2024-01-16T15:04:59.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.245+0000 7f99ed000640 1 -- start start 2024-01-16T15:04:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99ed000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99e8150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99ed000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80ccd00 0x7f99e80b78b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99ed000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99e80cd180 0x7f99e80b7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99ed000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80cac40 con 0x7f99c8003970 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99ed000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80b64c0 con 0x7f99e80cd180 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99ed000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99e80ca650 con 0x7f99e80ccd00 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99e7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99e8150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99e7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99e8150520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34994/0 (socket says 172.21.15.53:34994) 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.246+0000 7f99e7400640 1 -- 172.21.15.53:0/3941100078 learned_addr learned my addr 172.21.15.53:0/3941100078 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99e7400640 1 -- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80ccd00 msgr2=0x7f99e80b78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99e7400640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80ccd00 0x7f99e80b78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99e7400640 1 -- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99e80cd180 msgr2=0x7f99e80b7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99e7400640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99e80cd180 0x7f99e80b7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99e7400640 1 -- 172.21.15.53:0/3941100078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e80b5cb0 con 0x7f99c8003970 2024-01-16T15:04:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99e7400640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99e8150520 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f99d8000c00 tx=0x7f99d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8003810 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d8003990 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d8003d40 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e80b8330 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.247+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99e80b5f00 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.248+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99e8067b60 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.249+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f99d803b070 con 0x7f99c8003970 2024-01-16T15:04:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.250+0000 7f99df400640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99c006b920 0x7f99c006dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.250+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f99d80ac3d0 con 0x7f99c8003970 2024-01-16T15:04:59.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.251+0000 7f99e6a00640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99c006b920 0x7f99c006dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.252+0000 7f99e6a00640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99c006b920 0x7f99c006dde0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f99e8159da0 tx=0x7f99d0009750 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.255+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99d80769c0 con 0x7f99c8003970 2024-01-16T15:04:59.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.468+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone48371", "target": ["mon-mgr", ""]}) v1 -- 0x7f99e80cdbc0 con 0x7f99c006b920 2024-01-16T15:04:59.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.511+0000 7f99df400640 1 -- 172.21.15.53:0/3941100078 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f99d0011040 con 0x7f99c006b920 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99c006b920 msgr2=0x7f99c006dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99c006b920 0x7f99c006dde0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f99e8159da0 tx=0x7f99d0009750 comp rx=0 tx=0).stop 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 msgr2=0x7f99e8150520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99e8150520 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f99d8000c00 tx=0x7f99d8004730 comp rx=0 tx=0).stop 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 shutdown_connections 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99c006b920 0x7f99c006dde0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99e80cd180 0x7f99e80b7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99e80ccd00 0x7f99e80b78b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 --2- 172.21.15.53:0/3941100078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c8003970 0x7f99e8150520 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 >> 172.21.15.53:0/3941100078 conn(0x7f99e80d30b0 msgr2=0x7f99e815df90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 shutdown_connections 2024-01-16T15:04:59.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.515+0000 7f99ed000640 1 -- 172.21.15.53:0/3941100078 wait complete. 2024-01-16T15:04:59.513 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone48373 2024-01-16T15:04:59.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.746+0000 7f1aedc00640 1 Processor -- start 2024-01-16T15:04:59.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.746+0000 7f1aedc00640 1 -- start start 2024-01-16T15:04:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1aedc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ae813df80 0x7f1ae815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1aedc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1aedc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1ae815f9c0 0x7f1ae80cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1aedc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae806c470 con 0x7f1ae813df80 2024-01-16T15:04:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1aedc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ae806c750 con 0x7f1ae815f9c0 2024-01-16T15:04:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1aedc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae806ca30 con 0x7f1ae813e360 2024-01-16T15:04:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1ae6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ae813df80 0x7f1ae815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1ae6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ae813df80 0x7f1ae815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35002/0 (socket says 172.21.15.53:35002) 2024-01-16T15:04:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.748+0000 7f1ae6a00640 1 -- 172.21.15.53:0/4057943138 learned_addr learned my addr 172.21.15.53:0/4057943138 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae7400640 1 --2- 172.21.15.53:0/4057943138 >> v2:172.21.15.146:3300/0 conn(0x7f1ae815f9c0 0x7f1ae80cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae6000640 1 --2- 172.21.15.53:0/4057943138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae6a00640 1 -- 172.21.15.53:0/4057943138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 msgr2=0x7f1ae80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae6a00640 1 --2- 172.21.15.53:0/4057943138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae6a00640 1 -- 172.21.15.53:0/4057943138 >> v2:172.21.15.146:3300/0 conn(0x7f1ae815f9c0 msgr2=0x7f1ae80cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae6a00640 1 --2- 172.21.15.53:0/4057943138 >> v2:172.21.15.146:3300/0 conn(0x7f1ae815f9c0 0x7f1ae80cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.749+0000 7f1ae6a00640 1 -- 172.21.15.53:0/4057943138 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae806cd10 con 0x7f1ae813df80 2024-01-16T15:04:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.750+0000 7f1ae6a00640 1 --2- 172.21.15.53:0/4057943138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae815bee0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1ae806e6d0 tx=0x7f1ad802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194b1c213f1ba1c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.750+0000 7f1ae5600640 1 -- 172.21.15.53:0/4057943138 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad802ca70 con 0x7f1ae813df80 2024-01-16T15:04:59.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.750+0000 7f1ae5600640 1 -- 172.21.15.53:0/4057943138 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad802cbd0 con 0x7f1ae813df80 2024-01-16T15:04:59.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.750+0000 7f1ae5600640 1 -- 172.21.15.53:0/4057943138 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad80343f0 con 0x7f1ae813df80 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 -- 172.21.15.53:0/4057943138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 msgr2=0x7f1ae815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 --2- 172.21.15.53:0/4057943138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae815bee0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f1ae806e6d0 tx=0x7f1ad802bed0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 -- 172.21.15.53:0/4057943138 shutdown_connections 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 --2- 172.21.15.53:0/4057943138 >> v2:172.21.15.146:3300/0 conn(0x7f1ae815f9c0 0x7f1ae80cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 --2- 172.21.15.53:0/4057943138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 --2- 172.21.15.53:0/4057943138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae815bee0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 -- 172.21.15.53:0/4057943138 >> 172.21.15.53:0/4057943138 conn(0x7f1ae806a800 msgr2=0x7f1ae806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 -- 172.21.15.53:0/4057943138 shutdown_connections 2024-01-16T15:04:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.751+0000 7f1aedc00640 1 -- 172.21.15.53:0/4057943138 wait complete. 2024-01-16T15:04:59.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.753+0000 7f1aedc00640 1 Processor -- start 2024-01-16T15:04:59.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.753+0000 7f1aedc00640 1 -- start start 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.754+0000 7f1aedc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae813b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.754+0000 7f1aedc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae813f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.754+0000 7f1aedc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae815f9c0 0x7f1ae813c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.754+0000 7f1aedc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae813a160 con 0x7f1ae813df80 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.754+0000 7f1aedc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae813a3b0 con 0x7f1ae815f9c0 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.754+0000 7f1aedc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae813a690 con 0x7f1ae813e360 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae813b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae813b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35016/0 (socket says 172.21.15.53:35016) 2024-01-16T15:04:59.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 -- 172.21.15.53:0/2465158820 learned_addr learned my addr 172.21.15.53:0/2465158820 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:04:59.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae7400640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae815f9c0 0x7f1ae813c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 -- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 msgr2=0x7f1ae813f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae813f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 -- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae815f9c0 msgr2=0x7f1ae813c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae815f9c0 0x7f1ae813c020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 -- 172.21.15.53:0/2465158820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8009b30 con 0x7f1ae813df80 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.755+0000 7f1ae6a00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae813b450 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000f80 tx=0x7f1ad8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.756+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803e450 con 0x7f1ae813df80 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.756+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad803e5b0 con 0x7f1ae813df80 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.756+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad803e960 con 0x7f1ae813df80 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.756+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae813a970 con 0x7f1ae813df80 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.756+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae813af30 con 0x7f1ae813df80 2024-01-16T15:04:59.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.757+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae80675d0 con 0x7f1ae813df80 2024-01-16T15:04:59.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.758+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f1ad803eac0 con 0x7f1ae813df80 2024-01-16T15:04:59.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.759+0000 7f1adea00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac006b8f0 0x7f1ac006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:04:59.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.759+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f1ad80ad550 con 0x7f1ae813df80 2024-01-16T15:04:59.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.760+0000 7f1ae6000640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac006b8f0 0x7f1ac006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:04:59.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.761+0000 7f1ae6000640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac006b8f0 0x7f1ac006ddb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8148070 tx=0x7f1ad0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:04:59.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.764+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad8077a40 con 0x7f1ae813df80 2024-01-16T15:04:59.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:04:59.989+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone48373", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ae81408d0 con 0x7f1ac006b8f0 2024-01-16T15:05:00.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.031+0000 7f1adea00640 1 -- 172.21.15.53:0/2465158820 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1ad0009e40 con 0x7f1ac006b8f0 2024-01-16T15:05:00.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac006b8f0 msgr2=0x7f1ac006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac006b8f0 0x7f1ac006ddb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8148070 tx=0x7f1ad0008040 comp rx=0 tx=0).stop 2024-01-16T15:05:00.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 msgr2=0x7f1ae813b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae813b450 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8000f80 tx=0x7f1ad8004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 shutdown_connections 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac006b8f0 0x7f1ac006ddb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ae815f9c0 0x7f1ae813c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ae813e360 0x7f1ae813f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 --2- 172.21.15.53:0/2465158820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ae813df80 0x7f1ae813b450 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 >> 172.21.15.53:0/2465158820 conn(0x7f1ae806a800 msgr2=0x7f1ae806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 shutdown_connections 2024-01-16T15:05:00.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.035+0000 7f1aedc00640 1 -- 172.21.15.53:0/2465158820 wait complete. 2024-01-16T15:05:00.035 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:00.035 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:05:00.100 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-01-16T15:05:00.101 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:05:00.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.339+0000 7f7453600640 1 Processor -- start 2024-01-16T15:05:00.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.340+0000 7f7453600640 1 -- start start 2024-01-16T15:05:00.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.341+0000 7f7453600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f744c0b3ae0 0x7f744c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.341+0000 7f7453600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0cf290 0x7f744c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.341+0000 7f7453600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f744c070c40 con 0x7f744c0ae5a0 2024-01-16T15:05:00.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.341+0000 7f7453600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f744c071450 con 0x7f744c0b3ae0 2024-01-16T15:05:00.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.341+0000 7f7453600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744c0cfc60 con 0x7f744c0cf290 2024-01-16T15:05:00.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.342+0000 7f744be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f744c0b3ae0 0x7f744c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.342+0000 7f744be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f744c0b3ae0 0x7f744c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35018/0 (socket says 172.21.15.53:35018) 2024-01-16T15:05:00.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.342+0000 7f744be00640 1 -- 172.21.15.53:0/4080795770 learned_addr learned my addr 172.21.15.53:0/4080795770 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:00.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.342+0000 7f7451600640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0cf290 0x7f744c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.343+0000 7f744be00640 1 -- 172.21.15.53:0/4080795770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0cf290 msgr2=0x7f744c0cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:00.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.343+0000 7f744be00640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0cf290 0x7f744c0cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.343+0000 7f744be00640 1 -- 172.21.15.53:0/4080795770 >> v1:172.21.15.53:6789/0 conn(0x7f744c0ae5a0 legacy=0x7f744c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.343+0000 7f744be00640 1 -- 172.21.15.53:0/4080795770 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f744c06c470 con 0x7f744c0b3ae0 2024-01-16T15:05:00.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.343+0000 7f744b400640 1 -- 172.21.15.53:0/4080795770 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157273171 0 0) 0x7f744c070c40 con 0x7f744c0ae5a0 2024-01-16T15:05:00.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.343+0000 7f7451600640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0cf290 0x7f744c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:00.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.344+0000 7f744be00640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c15e430 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7434009870 tx=0x7f743402ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f96c286675bc44 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.344+0000 7f744b400640 1 -- 172.21.15.53:0/4080795770 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743402f830 con 0x7f744c0b3ae0 2024-01-16T15:05:00.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.344+0000 7f744b400640 1 -- 172.21.15.53:0/4080795770 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f743402f990 con 0x7f744c0b3ae0 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.344+0000 7f744b400640 1 -- 172.21.15.53:0/4080795770 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743402fd20 con 0x7f744c0b3ae0 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 -- 172.21.15.53:0/4080795770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 msgr2=0x7f744c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c15e430 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f7434009870 tx=0x7f743402ba10 comp rx=0 tx=0).stop 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 -- 172.21.15.53:0/4080795770 shutdown_connections 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0cf290 0x7f744c0cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 --2- 172.21.15.53:0/4080795770 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c15e430 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 -- 172.21.15.53:0/4080795770 >> 172.21.15.53:0/4080795770 conn(0x7f744c06a800 msgr2=0x7f744c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 -- 172.21.15.53:0/4080795770 shutdown_connections 2024-01-16T15:05:00.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.345+0000 7f7453600640 1 -- 172.21.15.53:0/4080795770 wait complete. 2024-01-16T15:05:00.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.347+0000 7f7453600640 1 Processor -- start 2024-01-16T15:05:00.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.347+0000 7f7453600640 1 -- start start 2024-01-16T15:05:00.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f7453600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0ae5a0 0x7f744c14f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f7453600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c150860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f7453600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f744c0cf290 0x7f744c153840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f7453600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c15ef20 con 0x7f744c0b3ae0 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f7453600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f744c1f2a30 con 0x7f744c0cf290 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f7453600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f744c15f170 con 0x7f744c0ae5a0 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.348+0000 7f744be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c150860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c150860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35022/0 (socket says 172.21.15.53:35022) 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 -- 172.21.15.53:0/1792413000 learned_addr learned my addr 172.21.15.53:0/1792413000 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f7450c00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0ae5a0 0x7f744c14f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f7451600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f744c0cf290 0x7f744c153840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 -- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0ae5a0 msgr2=0x7f744c14f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0ae5a0 0x7f744c14f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 -- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f744c0cf290 msgr2=0x7f744c153840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f744c0cf290 0x7f744c153840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 -- 172.21.15.53:0/1792413000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74340094d0 con 0x7f744c0b3ae0 2024-01-16T15:05:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f7451600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f744c0cf290 0x7f744c153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:00.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.349+0000 7f744be00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c150860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7434000c00 tx=0x7f74340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.350+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f743403b040 con 0x7f744c0b3ae0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.350+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7434037dd0 con 0x7f744c0b3ae0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.350+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74340426c0 con 0x7f744c0b3ae0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.350+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f744c15f450 con 0x7f744c0b3ae0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.350+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f744c15fa10 con 0x7f744c0b3ae0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.351+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f744c0675d0 con 0x7f744c0b3ae0 2024-01-16T15:05:00.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.352+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 14) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f7434042820 con 0x7f744c0b3ae0 2024-01-16T15:05:00.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.353+0000 7f7448c00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f742406b8f0 0x7f742406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:00.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.353+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5767+0+0 (secure 0 0 0) 0x7f74340adaf0 con 0x7f744c0b3ae0 2024-01-16T15:05:00.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.354+0000 7f7450c00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f742406b8f0 0x7f742406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:00.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.355+0000 7f7450c00640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f742406b8f0 0x7f742406ddb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f743c000f60 tx=0x7f743c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:00.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.358+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7434077e90 con 0x7f744c0b3ae0 2024-01-16T15:05:00.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:00.578+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f744c15f2e0 con 0x7f742406b8f0 2024-01-16T15:05:00.618 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:05:00.639+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:05:00.718 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:05:03.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.172+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7434078260 con 0x7f744c0b3ae0 2024-01-16T15:05:03.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.674+0000 7f7448c00640 1 -- 172.21.15.53:0/1792413000 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f744c15f2e0 con 0x7f742406b8f0 2024-01-16T15:05:03.654 INFO:teuthology.orchestra.run.smithi053.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:05:03.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f742406b8f0 msgr2=0x7f742406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f742406b8f0 0x7f742406ddb0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f743c000f60 tx=0x7f743c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:03.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 msgr2=0x7f744c150860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c150860 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f7434000c00 tx=0x7f74340046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 shutdown_connections 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f742406b8f0 0x7f742406ddb0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f744c0cf290 0x7f744c153840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f744c0b3ae0 0x7f744c150860 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 --2- 172.21.15.53:0/1792413000 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f744c0ae5a0 0x7f744c14f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:03.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.680+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 >> 172.21.15.53:0/1792413000 conn(0x7f744c06a800 msgr2=0x7f744c0b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:03.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.681+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 shutdown_connections 2024-01-16T15:05:03.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:03.681+0000 7f7453600640 1 -- 172.21.15.53:0/1792413000 wait complete. 2024-01-16T15:05:03.692 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:05:03.692 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:05:03.843 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:05:03.965 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:05:03.965 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:05:03.965 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:04.044 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:04.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.362+0000 7f9111400640 1 Processor -- start 2024-01-16T15:05:04.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.362+0000 7f9111400640 1 -- start start 2024-01-16T15:05:04.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.364+0000 7f9111400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c0b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.364+0000 7f9111400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 0x7f910c071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.364+0000 7f9111400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f910c06cb10 0x7f910c06cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.364+0000 7f9111400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910c0708d0 con 0x7f910c06cb10 2024-01-16T15:05:04.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.364+0000 7f9111400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f910c070bb0 con 0x7f910c070160 2024-01-16T15:05:04.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.364+0000 7f9111400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c070e90 con 0x7f910c06c730 2024-01-16T15:05:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.365+0000 7f910b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 0x7f910c071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.365+0000 7f910be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c0b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.366+0000 7f9110a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f910c06cb10 0x7f910c06cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.366+0000 7f910b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 0x7f910c071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:46012/0 (socket says 172.21.15.53:46012) 2024-01-16T15:05:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.366+0000 7f910b400640 1 -- 172.21.15.53:0/2819158412 learned_addr learned my addr 172.21.15.53:0/2819158412 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.367+0000 7f9110a00640 1 -- 172.21.15.53:0/2819158412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 msgr2=0x7f910c0b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.367+0000 7f9110a00640 1 --2- 172.21.15.53:0/2819158412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c0b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.367+0000 7f9110a00640 1 -- 172.21.15.53:0/2819158412 >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 msgr2=0x7f910c071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.367+0000 7f9110a00640 1 --2- 172.21.15.53:0/2819158412 >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 0x7f910c071970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.367+0000 7f9110a00640 1 -- 172.21.15.53:0/2819158412 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f910c071170 con 0x7f910c06cb10 2024-01-16T15:05:04.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.369+0000 7f910b400640 1 --2- 172.21.15.53:0/2819158412 >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 0x7f910c071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:04.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.369+0000 7f9110a00640 1 --2- 172.21.15.53:0/2819158412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c06cf80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f910c06e350 tx=0x7f910402c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211c32607fee96a8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.370+0000 7f910aa00640 1 -- 172.21.15.53:0/2819158412 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910402cc60 con 0x7f910c06cb10 2024-01-16T15:05:04.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.370+0000 7f910aa00640 1 -- 172.21.15.53:0/2819158412 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910402cdc0 con 0x7f910c06cb10 2024-01-16T15:05:04.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.370+0000 7f910aa00640 1 -- 172.21.15.53:0/2819158412 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104034660 con 0x7f910c06cb10 2024-01-16T15:05:04.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 -- 172.21.15.53:0/2819158412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 msgr2=0x7f910c06cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 --2- 172.21.15.53:0/2819158412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c06cf80 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f910c06e350 tx=0x7f910402c0c0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 -- 172.21.15.53:0/2819158412 shutdown_connections 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 --2- 172.21.15.53:0/2819158412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c06cf80 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 --2- 172.21.15.53:0/2819158412 >> v2:172.21.15.146:3300/0 conn(0x7f910c070160 0x7f910c071970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 --2- 172.21.15.53:0/2819158412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c0b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.371+0000 7f9111400640 1 -- 172.21.15.53:0/2819158412 >> 172.21.15.53:0/2819158412 conn(0x7f910c06a7d0 msgr2=0x7f910c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.372+0000 7f9111400640 1 -- 172.21.15.53:0/2819158412 shutdown_connections 2024-01-16T15:05:04.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.372+0000 7f9111400640 1 -- 172.21.15.53:0/2819158412 wait complete. 2024-01-16T15:05:04.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.375+0000 7f9111400640 1 Processor -- start 2024-01-16T15:05:04.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.375+0000 7f9111400640 1 -- start start 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f9111400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c15d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f9111400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c1473c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f910be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f910be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c15d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:46494/0 (socket says 172.21.15.53:46494) 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f910be00640 1 -- 172.21.15.53:0/1373581477 learned_addr learned my addr 172.21.15.53:0/1373581477 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c070160 0x7f910c15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f910b400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c1473c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c15cc00 con 0x7f910c06cb10 2024-01-16T15:05:04.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f910c15d560 con 0x7f910c070160 2024-01-16T15:05:04.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.377+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f910c1450a0 con 0x7f910c06c730 2024-01-16T15:05:04.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f9110a00640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c070160 0x7f910c15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f910be00640 1 -- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c070160 msgr2=0x7f910c15b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f910be00640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c070160 0x7f910c15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f910be00640 1 -- 172.21.15.53:0/1373581477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 msgr2=0x7f910c1473c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f910be00640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c1473c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f910be00640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9104009a70 con 0x7f910c06c730 2024-01-16T15:05:04.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.378+0000 7f910b400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c1473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:05:04.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.379+0000 7f910be00640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c15d020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00e8a0 tx=0x7f90fc00ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.380+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00ee90 con 0x7f910c06c730 2024-01-16T15:05:04.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.381+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90fc00c940 con 0x7f910c06c730 2024-01-16T15:05:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.381+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90fc00ee90 con 0x7f910c06c730 2024-01-16T15:05:04.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.381+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f910c15bf80 con 0x7f910c06c730 2024-01-16T15:05:04.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.383+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f910c15c540 con 0x7f910c06c730 2024-01-16T15:05:04.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.385+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f910c0723e0 con 0x7f910c06c730 2024-01-16T15:05:04.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.387+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f90fc00cab0 con 0x7f910c06c730 2024-01-16T15:05:04.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.388+0000 7f9103e00640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90e406b8c0 0x7f90e406dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.388+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f90fc089f50 con 0x7f910c06c730 2024-01-16T15:05:04.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.389+0000 7f910b400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90e406b8c0 0x7f90e406dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.392+0000 7f910b400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90e406b8c0 0x7f90e406dd80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f910c145510 tx=0x7f90f4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.400+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f90fc054900 con 0x7f910c06c730 2024-01-16T15:05:04.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.623+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f910c067450 con 0x7f910c06c730 2024-01-16T15:05:04.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.626+0000 7f9103e00640 1 -- 172.21.15.53:0/1373581477 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+12950 (secure 0 0 0) 0x7f90fc021020 con 0x7f910c06c730 2024-01-16T15:05:04.606 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:04.606 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":52,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:03.654310+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/4293818684":"2024-01-17T15:05:00.640610+0000","172.21.15.53:6834/4293818684":"2024-01-17T15:05:00.640610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:04.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90e406b8c0 msgr2=0x7f90e406dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90e406b8c0 0x7f90e406dd80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f910c145510 tx=0x7f90f4008040 comp rx=0 tx=0).stop 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 msgr2=0x7f910c15d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c15d020 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f90fc00e8a0 tx=0x7f90fc00ed60 comp rx=0 tx=0).stop 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 shutdown_connections 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f90e406b8c0 0x7f90e406dd80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f910c070160 0x7f910c15b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f910c06cb10 0x7f910c1473c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 --2- 172.21.15.53:0/1373581477 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f910c06c730 0x7f910c15d020 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 >> 172.21.15.53:0/1373581477 conn(0x7f910c06a7d0 msgr2=0x7f910c0b0280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 shutdown_connections 2024-01-16T15:05:04.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.628+0000 7f9111400640 1 -- 172.21.15.53:0/1373581477 wait complete. 2024-01-16T15:05:04.626 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:05:04.627 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:05:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.850+0000 7f4553000640 1 Processor -- start 2024-01-16T15:05:04.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.850+0000 7f4553000640 1 -- start start 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4553000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f454c0adf20 0x7f454c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4553000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c15bf10 0x7f454c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4553000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f454c0cc980 0x7f454c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4553000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454c15f970 con 0x7f454c0adf20 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4553000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f454c15f4e0 con 0x7f454c0cc980 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4553000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454c0cce10 con 0x7f454c15bf10 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4551c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f454c0adf20 0x7f454c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4551c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f454c0adf20 0x7f454c070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46852/0 (socket says 172.21.15.53:46852) 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.852+0000 7f4551c00640 1 -- 172.21.15.53:0/1777134657 learned_addr learned my addr 172.21.15.53:0/1777134657 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4551200640 1 --2- 172.21.15.53:0/1777134657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c15bf10 0x7f454c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4552600640 1 --2- 172.21.15.53:0/1777134657 >> v2:172.21.15.146:3300/0 conn(0x7f454c0cc980 0x7f454c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4551c00640 1 -- 172.21.15.53:0/1777134657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c15bf10 msgr2=0x7f454c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4551c00640 1 --2- 172.21.15.53:0/1777134657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c15bf10 0x7f454c0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4551c00640 1 -- 172.21.15.53:0/1777134657 >> v2:172.21.15.146:3300/0 conn(0x7f454c0cc980 msgr2=0x7f454c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4551c00640 1 --2- 172.21.15.53:0/1777134657 >> v2:172.21.15.146:3300/0 conn(0x7f454c0cc980 0x7f454c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4551c00640 1 -- 172.21.15.53:0/1777134657 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c071130 con 0x7f454c0adf20 2024-01-16T15:05:04.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.853+0000 7f4552600640 1 --2- 172.21.15.53:0/1777134657 >> v2:172.21.15.146:3300/0 conn(0x7f454c0cc980 0x7f454c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.854+0000 7f4551c00640 1 --2- 172.21.15.53:0/1777134657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c0adf20 0x7f454c070a30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f454c06e490 tx=0x7f453c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30eb35e280913073 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.854+0000 7f454be00640 1 -- 172.21.15.53:0/1777134657 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0317e0 con 0x7f454c0adf20 2024-01-16T15:05:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.854+0000 7f454be00640 1 -- 172.21.15.53:0/1777134657 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c031940 con 0x7f454c0adf20 2024-01-16T15:05:04.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.854+0000 7f454be00640 1 -- 172.21.15.53:0/1777134657 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c031cd0 con 0x7f454c0adf20 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 -- 172.21.15.53:0/1777134657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c0adf20 msgr2=0x7f454c070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 --2- 172.21.15.53:0/1777134657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c0adf20 0x7f454c070a30 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f454c06e490 tx=0x7f453c02b880 comp rx=0 tx=0).stop 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 -- 172.21.15.53:0/1777134657 shutdown_connections 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 --2- 172.21.15.53:0/1777134657 >> v2:172.21.15.146:3300/0 conn(0x7f454c0cc980 0x7f454c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 --2- 172.21.15.53:0/1777134657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c15bf10 0x7f454c0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 --2- 172.21.15.53:0/1777134657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c0adf20 0x7f454c070a30 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 -- 172.21.15.53:0/1777134657 >> 172.21.15.53:0/1777134657 conn(0x7f454c06a7d0 msgr2=0x7f454c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 -- 172.21.15.53:0/1777134657 shutdown_connections 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.855+0000 7f4553000640 1 -- 172.21.15.53:0/1777134657 wait complete. 2024-01-16T15:05:04.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.857+0000 7f4553000640 1 Processor -- start 2024-01-16T15:05:04.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.857+0000 7f4553000640 1 -- start start 2024-01-16T15:05:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4553000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0adf20 0x7f454c13f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4553000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c0cc980 0x7f454c13e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4553000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 0x7f454c14e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4553000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c1426c0 con 0x7f454c15bf10 2024-01-16T15:05:04.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4553000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454c0b2a40 con 0x7f454c0cc980 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4553000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f454c0b2d20 con 0x7f454c0adf20 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4552600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 0x7f454c14e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4552600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 0x7f454c14e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46858/0 (socket says 172.21.15.53:46858) 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4552600640 1 -- 172.21.15.53:0/1235306075 learned_addr learned my addr 172.21.15.53:0/1235306075 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4551c00640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0adf20 0x7f454c13f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.858+0000 7f4551200640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c0cc980 0x7f454c13e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4552600640 1 -- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0adf20 msgr2=0x7f454c13f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4552600640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0adf20 0x7f454c13f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4552600640 1 -- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c0cc980 msgr2=0x7f454c13e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:04.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4552600640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c0cc980 0x7f454c13e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4552600640 1 -- 172.21.15.53:0/1235306075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c0094d0 con 0x7f454c15bf10 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4551200640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c0cc980 0x7f454c13e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4552600640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 0x7f454c14e680 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4544007e10 tx=0x7f454400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544003b90 con 0x7f454c15bf10 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454400f040 con 0x7f454c15bf10 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454400de80 con 0x7f454c15bf10 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f454c0b3000 con 0x7f454c15bf10 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.859+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f454c13ed30 con 0x7f454c15bf10 2024-01-16T15:05:04.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.861+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f454c0b28a0 con 0x7f454c15bf10 2024-01-16T15:05:04.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.862+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4544004030 con 0x7f454c15bf10 2024-01-16T15:05:04.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.862+0000 7f4549600640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452406b860 0x7f452406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.862+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f454408a740 con 0x7f454c15bf10 2024-01-16T15:05:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.863+0000 7f4551c00640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452406b860 0x7f452406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:04.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.864+0000 7f4551c00640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452406b860 0x7f452406dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f453c036000 tx=0x7f453c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:04.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:04.867+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f454c0b28a0 con 0x7f454c15bf10 2024-01-16T15:05:05.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.055+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f454c0b28a0 con 0x7f454c15bf10 2024-01-16T15:05:05.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.055+0000 7f4549600640 1 -- 172.21.15.53:0/1235306075 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f454401a020 con 0x7f454c15bf10 2024-01-16T15:05:05.035 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:05:05.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452406b860 msgr2=0x7f452406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452406b860 0x7f452406dd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f453c036000 tx=0x7f453c03d040 comp rx=0 tx=0).stop 2024-01-16T15:05:05.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 msgr2=0x7f454c14e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 0x7f454c14e680 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f4544007e10 tx=0x7f454400d4a0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 shutdown_connections 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452406b860 0x7f452406dd20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f454c15bf10 0x7f454c14e680 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f454c0cc980 0x7f454c13e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 --2- 172.21.15.53:0/1235306075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f454c0adf20 0x7f454c13f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 >> 172.21.15.53:0/1235306075 conn(0x7f454c06a7d0 msgr2=0x7f454c06ce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 shutdown_connections 2024-01-16T15:05:05.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.058+0000 7f4553000640 1 -- 172.21.15.53:0/1235306075 wait complete. 2024-01-16T15:05:05.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:05:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.294+0000 7f5c3e200640 1 Processor -- start 2024-01-16T15:05:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.295+0000 7f5c3e200640 1 -- start start 2024-01-16T15:05:05.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c3e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c380d0150 0x7f5c380d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c3e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380b2c70 con 0x7f5c3815d3c0 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c3e200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380cf6d0 con 0x7f5c380ceea0 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c3e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c380cf3f0 con 0x7f5c380d0150 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c37e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c380d0150 0x7f5c380d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c37e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c380d0150 0x7f5c380d1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46860/0 (socket says 172.21.15.53:46860) 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.296+0000 7f5c37e00640 1 -- 172.21.15.53:0/3863209446 learned_addr learned my addr 172.21.15.53:0/3863209446 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.297+0000 7f5c37e00640 1 -- 172.21.15.53:0/3863209446 >> v1:172.21.15.146:6789/0 conn(0x7f5c380ceea0 legacy=0x7f5c380b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.297+0000 7f5c37e00640 1 -- 172.21.15.53:0/3863209446 >> v1:172.21.15.53:6789/0 conn(0x7f5c3815d3c0 legacy=0x7f5c380cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.297+0000 7f5c37e00640 1 -- 172.21.15.53:0/3863209446 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c380b2620 con 0x7f5c380d0150 2024-01-16T15:05:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.298+0000 7f5c37e00640 1 --2- 172.21.15.53:0/3863209446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c380d0150 0x7f5c380d1f20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009870 tx=0x7f5c2002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea9c4db5cd300e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.298+0000 7f5c37400640 1 -- 172.21.15.53:0/3863209446 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20030850 con 0x7f5c380d0150 2024-01-16T15:05:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.298+0000 7f5c37400640 1 -- 172.21.15.53:0/3863209446 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c200309b0 con 0x7f5c380d0150 2024-01-16T15:05:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.298+0000 7f5c37400640 1 -- 172.21.15.53:0/3863209446 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20030d40 con 0x7f5c380d0150 2024-01-16T15:05:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 -- 172.21.15.53:0/3863209446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c380d0150 msgr2=0x7f5c380d1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 --2- 172.21.15.53:0/3863209446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c380d0150 0x7f5c380d1f20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f5c20009870 tx=0x7f5c2002b990 comp rx=0 tx=0).stop 2024-01-16T15:05:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 -- 172.21.15.53:0/3863209446 shutdown_connections 2024-01-16T15:05:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 --2- 172.21.15.53:0/3863209446 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c380d0150 0x7f5c380d1f20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 -- 172.21.15.53:0/3863209446 >> 172.21.15.53:0/3863209446 conn(0x7f5c380d3050 msgr2=0x7f5c380d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 -- 172.21.15.53:0/3863209446 shutdown_connections 2024-01-16T15:05:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.299+0000 7f5c3e200640 1 -- 172.21.15.53:0/3863209446 wait complete. 2024-01-16T15:05:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.301+0000 7f5c3e200640 1 Processor -- start 2024-01-16T15:05:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.301+0000 7f5c3e200640 1 -- start start 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c380ceea0 0x7f5c38155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c380d0150 0x7f5c3815f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 0x7f5c3815daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3815ef50 con 0x7f5c3815d3c0 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3815e2d0 con 0x7f5c380d0150 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3815e5b0 con 0x7f5c380ceea0 2024-01-16T15:05:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 0x7f5c3815daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 0x7f5c3815daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46870/0 (socket says 172.21.15.53:46870) 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.302+0000 7f5c3d800640 1 -- 172.21.15.53:0/1900329436 learned_addr learned my addr 172.21.15.53:0/1900329436 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c37e00640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c380d0150 0x7f5c3815f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3d800640 1 -- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c380ceea0 msgr2=0x7f5c38155c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3d800640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c380ceea0 0x7f5c38155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3d800640 1 -- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c380d0150 msgr2=0x7f5c3815f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3d800640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c380d0150 0x7f5c3815f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3d800640 1 -- 172.21.15.53:0/1900329436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c200094d0 con 0x7f5c3815d3c0 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3d800640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 0x7f5c3815daa0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c410 tx=0x7f5c3000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30011790 con 0x7f5c3815d3c0 2024-01-16T15:05:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c300118f0 con 0x7f5c3815d3c0 2024-01-16T15:05:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c3815e890 con 0x7f5c3815d3c0 2024-01-16T15:05:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c30011ce0 con 0x7f5c3815d3c0 2024-01-16T15:05:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.303+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3815dfe0 con 0x7f5c3815d3c0 2024-01-16T15:05:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.305+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3815f090 con 0x7f5c3815d3c0 2024-01-16T15:05:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.305+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f5c3001c460 con 0x7f5c3815d3c0 2024-01-16T15:05:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.306+0000 7f5c34c00640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c1006b890 0x7f5c1006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.306+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f5c30089fc0 con 0x7f5c3815d3c0 2024-01-16T15:05:05.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.307+0000 7f5c3ce00640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c1006b890 0x7f5c1006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.308+0000 7f5c3ce00640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c1006b890 0x7f5c1006dd50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5c280012e0 tx=0x7f5c28007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:05.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.311+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c3815f090 con 0x7f5c3815d3c0 2024-01-16T15:05:05.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.510+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5c3815f090 con 0x7f5c3815d3c0 2024-01-16T15:05:05.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.510+0000 7f5c34c00640 1 -- 172.21.15.53:0/1900329436 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v52) v1 ==== 80+0+122 (secure 0 0 0) 0x7f5c3815f090 con 0x7f5c3815d3c0 2024-01-16T15:05:05.490 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6835/4293818684 2024-01-17T15:05:00.640610+0000 2024-01-16T15:05:05.490 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6834/4293818684 2024-01-17T15:05:00.640610+0000 2024-01-16T15:05:05.490 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c1006b890 msgr2=0x7f5c1006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c1006b890 0x7f5c1006dd50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5c280012e0 tx=0x7f5c28007d30 comp rx=0 tx=0).stop 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 msgr2=0x7f5c3815daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 0x7f5c3815daa0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f5c3000c410 tx=0x7f5c3000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 shutdown_connections 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c1006b890 0x7f5c1006dd50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c3815d3c0 0x7f5c3815daa0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c380d0150 0x7f5c3815f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 --2- 172.21.15.53:0/1900329436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c380ceea0 0x7f5c38155c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:05.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 >> 172.21.15.53:0/1900329436 conn(0x7f5c380d3050 msgr2=0x7f5c3806aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:05.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 shutdown_connections 2024-01-16T15:05:05.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:05.513+0000 7f5c3e200640 1 -- 172.21.15.53:0/1900329436 wait complete. 2024-01-16T15:05:05.509 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:05:05.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:05.509 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:05.602 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:05:05.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:05.603 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:05:05.603 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[52085]: fuse finished with error 0 and tester_r 0 2024-01-16T15:05:05.854 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:05:05.854 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:05.913 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:06.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.166+0000 7f00e7200640 1 Processor -- start 2024-01-16T15:05:06.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.166+0000 7f00e7200640 1 -- start start 2024-01-16T15:05:06.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00e8146770 0x7f00e806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e80b2e10 con 0x7f00e8071410 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00e8146bc0 con 0x7f00e8071f00 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e7200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00e806cec0 con 0x7f00e8146770 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e5e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f00e8071410 0x7f00e80717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:60880/0 (socket says 172.21.15.53:60880) 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e5e00640 1 -- 172.21.15.53:0/2863375723 learned_addr learned my addr 172.21.15.53:0/2863375723 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.168+0000 7f00e6800640 1 --2- 172.21.15.53:0/2863375723 >> v2:172.21.15.53:3300/0 conn(0x7f00e8146770 0x7f00e806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.169+0000 7f00e5400640 1 -- 172.21.15.53:0/2863375723 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3671768929 0 0) 0x7f00e80b2e10 con 0x7f00e8071410 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.169+0000 7f00e5400640 1 -- 172.21.15.53:0/2863375723 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00c8003520 con 0x7f00e8071410 2024-01-16T15:05:06.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.169+0000 7f00e6800640 1 -- 172.21.15.53:0/2863375723 >> v1:172.21.15.146:6789/0 conn(0x7f00e8071f00 legacy=0x7f00e8164ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.169+0000 7f00e6800640 1 -- 172.21.15.53:0/2863375723 >> v1:172.21.15.53:6789/0 conn(0x7f00e8071410 legacy=0x7f00e80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.169+0000 7f00e6800640 1 -- 172.21.15.53:0/2863375723 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e806d1a0 con 0x7f00e8146770 2024-01-16T15:05:06.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.170+0000 7f00e6800640 1 --2- 172.21.15.53:0/2863375723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8146770 0x7f00e806c7c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009d90 tx=0x7f00e002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562d711b2a052372 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.170+0000 7f00e5400640 1 -- 172.21.15.53:0/2863375723 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e002cbd0 con 0x7f00e8146770 2024-01-16T15:05:06.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.170+0000 7f00e5400640 1 -- 172.21.15.53:0/2863375723 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e002cd30 con 0x7f00e8146770 2024-01-16T15:05:06.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.170+0000 7f00e5400640 1 -- 172.21.15.53:0/2863375723 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e002cea0 con 0x7f00e8146770 2024-01-16T15:05:06.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 -- 172.21.15.53:0/2863375723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8146770 msgr2=0x7f00e806c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 --2- 172.21.15.53:0/2863375723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8146770 0x7f00e806c7c0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f00e0009d90 tx=0x7f00e002be20 comp rx=0 tx=0).stop 2024-01-16T15:05:06.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 -- 172.21.15.53:0/2863375723 shutdown_connections 2024-01-16T15:05:06.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 --2- 172.21.15.53:0/2863375723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8146770 0x7f00e806c7c0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 -- 172.21.15.53:0/2863375723 >> 172.21.15.53:0/2863375723 conn(0x7f00e806a800 msgr2=0x7f00e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 -- 172.21.15.53:0/2863375723 shutdown_connections 2024-01-16T15:05:06.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.171+0000 7f00e7200640 1 -- 172.21.15.53:0/2863375723 wait complete. 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.173+0000 7f00e7200640 1 Processor -- start 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.173+0000 7f00e7200640 1 -- start start 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 0x7f00e81f21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00e8071f00 0x7f00e8141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00e8146770 0x7f00e80b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e81f2730 con 0x7f00e8071410 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e80c3d70 con 0x7f00e8146770 2024-01-16T15:05:06.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00e8142690 con 0x7f00e8071f00 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 0x7f00e81f21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 0x7f00e81f21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46888/0 (socket says 172.21.15.53:46888) 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 -- 172.21.15.53:0/925979217 learned_addr learned my addr 172.21.15.53:0/925979217 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 -- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00e8071f00 msgr2=0x7f00e8141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00e8071f00 0x7f00e8141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 -- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00e8146770 msgr2=0x7f00e80b0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e6800640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00e8146770 0x7f00e80b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.174+0000 7f00e5e00640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00e8146770 0x7f00e80b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00e5e00640 1 -- 172.21.15.53:0/925979217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e0009a70 con 0x7f00e8071410 2024-01-16T15:05:06.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00e5e00640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 0x7f00e81f21f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f00d800c500 tx=0x7f00d800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d8005ce0 con 0x7f00e8071410 2024-01-16T15:05:06.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d8005e40 con 0x7f00e8071410 2024-01-16T15:05:06.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d8007330 con 0x7f00e8071410 2024-01-16T15:05:06.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e8142970 con 0x7f00e8071410 2024-01-16T15:05:06.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.175+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e80c4310 con 0x7f00e8071410 2024-01-16T15:05:06.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.177+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e8067450 con 0x7f00e8071410 2024-01-16T15:05:06.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.177+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f00d8007550 con 0x7f00e8071410 2024-01-16T15:05:06.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.178+0000 7f00de000640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00c006b860 0x7f00c006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.178+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f00d8089950 con 0x7f00e8071410 2024-01-16T15:05:06.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.179+0000 7f00dfe00640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00c006b860 0x7f00c006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.180+0000 7f00dfe00640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00c006b860 0x7f00c006dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f00e8142e90 tx=0x7f00d00079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.183+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00d8054360 con 0x7f00e8071410 2024-01-16T15:05:06.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.378+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00e8070b20 con 0x7f00e8071410 2024-01-16T15:05:06.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.380+0000 7f00de000640 1 -- 172.21.15.53:0/925979217 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+12950 (secure 0 0 0) 0x7f00d8057980 con 0x7f00e8071410 2024-01-16T15:05:06.359 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:06.360 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":52,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:03.654310+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/4293818684":"2024-01-17T15:05:00.640610+0000","172.21.15.53:6834/4293818684":"2024-01-17T15:05:00.640610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00c006b860 msgr2=0x7f00c006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00c006b860 0x7f00c006dd20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f00e8142e90 tx=0x7f00d00079b0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 msgr2=0x7f00e81f21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 0x7f00e81f21f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f00d800c500 tx=0x7f00d800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 shutdown_connections 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00c006b860 0x7f00c006dd20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00e8146770 0x7f00e80b0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00e8071f00 0x7f00e8141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 --2- 172.21.15.53:0/925979217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00e8071410 0x7f00e81f21f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.382+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 >> 172.21.15.53:0/925979217 conn(0x7f00e806a800 msgr2=0x7f00e806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.383+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 shutdown_connections 2024-01-16T15:05:06.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.383+0000 7f00e7200640 1 -- 172.21.15.53:0/925979217 wait complete. 2024-01-16T15:05:06.378 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:05:06.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:05:06.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.617+0000 7fc5b5200640 1 Processor -- start 2024-01-16T15:05:06.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.617+0000 7fc5b5200640 1 -- start start 2024-01-16T15:05:06.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.619+0000 7fc5b5200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5b00adf20 0x7fc5b0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.619+0000 7fc5b5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 0x7fc5b00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.619+0000 7fc5b5200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5b015f4e0 con 0x7fc5b00cc980 2024-01-16T15:05:06.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.619+0000 7fc5b5200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5b015f970 con 0x7fc5b00adf20 2024-01-16T15:05:06.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.619+0000 7fc5b5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00cce10 con 0x7fc5b015bf10 2024-01-16T15:05:06.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.620+0000 7fc5a6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 0x7fc5b00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.620+0000 7fc5af400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5b00adf20 0x7fc5b0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.620+0000 7fc5a6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 0x7fc5b00cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:46538/0 (socket says 172.21.15.53:46538) 2024-01-16T15:05:06.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.620+0000 7fc5a6800640 1 -- 172.21.15.53:0/2924566191 learned_addr learned my addr 172.21.15.53:0/2924566191 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:06.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.621+0000 7fc5a6800640 1 -- 172.21.15.53:0/2924566191 >> v2:172.21.15.146:3300/0 conn(0x7fc5b00adf20 msgr2=0x7fc5b0070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:06.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.621+0000 7fc5a6800640 1 --2- 172.21.15.53:0/2924566191 >> v2:172.21.15.146:3300/0 conn(0x7fc5b00adf20 0x7fc5b0070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.621+0000 7fc5a6800640 1 -- 172.21.15.53:0/2924566191 >> v1:172.21.15.146:6789/0 conn(0x7fc5b00cc980 legacy=0x7fc5b0143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.621+0000 7fc5a6800640 1 -- 172.21.15.53:0/2924566191 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b0071130 con 0x7fc5b015bf10 2024-01-16T15:05:06.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.621+0000 7fc5af400640 1 --2- 172.21.15.53:0/2924566191 >> v2:172.21.15.146:3300/0 conn(0x7fc5b00adf20 0x7fc5b0070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:05:06.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.622+0000 7fc5a6800640 1 --2- 172.21.15.53:0/2924566191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 0x7fc5b00cc440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc598009870 tx=0x7fc59802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=606dc6768c4bace9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.623+0000 7fc5aea00640 1 -- 172.21.15.53:0/2924566191 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598031640 con 0x7fc5b015bf10 2024-01-16T15:05:06.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.623+0000 7fc5aea00640 1 -- 172.21.15.53:0/2924566191 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5980317a0 con 0x7fc5b015bf10 2024-01-16T15:05:06.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.623+0000 7fc5aea00640 1 -- 172.21.15.53:0/2924566191 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc598031b30 con 0x7fc5b015bf10 2024-01-16T15:05:06.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 -- 172.21.15.53:0/2924566191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 msgr2=0x7fc5b00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 --2- 172.21.15.53:0/2924566191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 0x7fc5b00cc440 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc598009870 tx=0x7fc59802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 -- 172.21.15.53:0/2924566191 shutdown_connections 2024-01-16T15:05:06.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 --2- 172.21.15.53:0/2924566191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b015bf10 0x7fc5b00cc440 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 --2- 172.21.15.53:0/2924566191 >> v2:172.21.15.146:3300/0 conn(0x7fc5b00adf20 0x7fc5b0070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 -- 172.21.15.53:0/2924566191 >> 172.21.15.53:0/2924566191 conn(0x7fc5b006a7d0 msgr2=0x7fc5b006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 -- 172.21.15.53:0/2924566191 shutdown_connections 2024-01-16T15:05:06.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.624+0000 7fc5b5200640 1 -- 172.21.15.53:0/2924566191 wait complete. 2024-01-16T15:05:06.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.626+0000 7fc5b5200640 1 Processor -- start 2024-01-16T15:05:06.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.626+0000 7fc5b5200640 1 -- start start 2024-01-16T15:05:06.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5b5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b00adf20 0x7fc5b00b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5b5200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b00cc980 0x7fc5b00b1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5b5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 0x7fc5b013a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5b5200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00b2df0 con 0x7fc5b015bf10 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5b5200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00b2830 con 0x7fc5b00cc980 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5b5200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5b00b2b10 con 0x7fc5b00adf20 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 0x7fc5b013a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 0x7fc5b013a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46890/0 (socket says 172.21.15.53:46890) 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.627+0000 7fc5afe00640 1 -- 172.21.15.53:0/1309532793 learned_addr learned my addr 172.21.15.53:0/1309532793 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5a6800640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b00cc980 0x7fc5b00b1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5af400640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b00adf20 0x7fc5b00b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5afe00640 1 -- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b00adf20 msgr2=0x7fc5b00b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5afe00640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b00adf20 0x7fc5b00b37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5afe00640 1 -- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b00cc980 msgr2=0x7fc5b00b1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5afe00640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b00cc980 0x7fc5b00b1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5afe00640 1 -- 172.21.15.53:0/1309532793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5a0009a70 con 0x7fc5b015bf10 2024-01-16T15:05:06.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5afe00640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 0x7fc5b013a8e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c390 tx=0x7fc5a800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.628+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a80052d0 con 0x7fc5b015bf10 2024-01-16T15:05:06.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.629+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5a8005430 con 0x7fc5b015bf10 2024-01-16T15:05:06.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.629+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8003810 con 0x7fc5b015bf10 2024-01-16T15:05:06.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.629+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5980094d0 con 0x7fc5b015bf10 2024-01-16T15:05:06.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.629+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5b013a3c0 con 0x7fc5b015bf10 2024-01-16T15:05:06.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.630+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b0066eb0 con 0x7fc5b015bf10 2024-01-16T15:05:06.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.631+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fc5a8003970 con 0x7fc5b015bf10 2024-01-16T15:05:06.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.632+0000 7fc5a7e00640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc58806b860 0x7fc58806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:06.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.632+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fc5a80897d0 con 0x7fc5b015bf10 2024-01-16T15:05:06.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.632+0000 7fc5af400640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc58806b860 0x7fc58806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:06.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.633+0000 7fc5af400640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc58806b860 0x7fc58806dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00b09f0 tx=0x7fc5a000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:06.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.636+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5a8054160 con 0x7fc5b015bf10 2024-01-16T15:05:06.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.824+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc5b0067450 con 0x7fc5b015bf10 2024-01-16T15:05:06.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.825+0000 7fc5a7e00640 1 -- 172.21.15.53:0/1309532793 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc5a8057780 con 0x7fc5b015bf10 2024-01-16T15:05:06.804 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:05:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.827+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc58806b860 msgr2=0x7fc58806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.827+0000 7fc5b5200640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc58806b860 0x7fc58806dd20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc5b00b09f0 tx=0x7fc5a000b040 comp rx=0 tx=0).stop 2024-01-16T15:05:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 msgr2=0x7fc5b013a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 0x7fc5b013a8e0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800c390 tx=0x7fc5a800c850 comp rx=0 tx=0).stop 2024-01-16T15:05:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 shutdown_connections 2024-01-16T15:05:06.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc58806b860 0x7fc58806dd20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5b015bf10 0x7fc5b013a8e0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5b00cc980 0x7fc5b00b1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 --2- 172.21.15.53:0/1309532793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5b00adf20 0x7fc5b00b37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:06.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 >> 172.21.15.53:0/1309532793 conn(0x7fc5b006a7d0 msgr2=0x7fc5b015dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:06.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 shutdown_connections 2024-01-16T15:05:06.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:06.828+0000 7fc5b5200640 1 -- 172.21.15.53:0/1309532793 wait complete. 2024-01-16T15:05:06.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:05:07.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.062+0000 7f499b400640 1 Processor -- start 2024-01-16T15:05:07.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.062+0000 7f499b400640 1 -- start start 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f499b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f499b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f499413e360 0x7f49940cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f499b400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f499406c470 con 0x7f499415f9c0 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f499b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f499406c750 con 0x7f499413e360 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f499b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499406ca30 con 0x7f499413df80 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f498be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f499413e360 0x7f49940cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f4999400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f499415f9c0 0x7f49941408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:60884/0 (socket says 172.21.15.53:60884) 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f4999400640 1 -- 172.21.15.53:0/348142057 learned_addr learned my addr 172.21.15.53:0/348142057 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:07.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.064+0000 7f4998a00640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.065+0000 7f498be00640 1 -- 172.21.15.53:0/348142057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 msgr2=0x7f499415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.065+0000 7f498be00640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.065+0000 7f498be00640 1 -- 172.21.15.53:0/348142057 >> v1:172.21.15.53:6789/0 conn(0x7f499415f9c0 legacy=0x7f49941408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.065+0000 7f498be00640 1 -- 172.21.15.53:0/348142057 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f499406cd10 con 0x7f499413e360 2024-01-16T15:05:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.065+0000 7f4993e00640 1 -- 172.21.15.53:0/348142057 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828638480 0 0) 0x7f499406c470 con 0x7f499415f9c0 2024-01-16T15:05:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.065+0000 7f4998a00640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.066+0000 7f498be00640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499413e360 0x7f49940cf290 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f497c0097a0 tx=0x7f497c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a45c66774b27392e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.066+0000 7f4993e00640 1 -- 172.21.15.53:0/348142057 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c02d040 con 0x7f499413e360 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.066+0000 7f4993e00640 1 -- 172.21.15.53:0/348142057 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f497c031810 con 0x7f499413e360 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.066+0000 7f4993e00640 1 -- 172.21.15.53:0/348142057 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f497c031ba0 con 0x7f499413e360 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.066+0000 7f499b400640 1 -- 172.21.15.53:0/348142057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499413e360 msgr2=0x7f49940cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.066+0000 7f499b400640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499413e360 0x7f49940cf290 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f497c0097a0 tx=0x7f497c02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.067+0000 7f499b400640 1 -- 172.21.15.53:0/348142057 shutdown_connections 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.067+0000 7f499b400640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499413e360 0x7f49940cf290 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.067+0000 7f499b400640 1 --2- 172.21.15.53:0/348142057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.067+0000 7f499b400640 1 -- 172.21.15.53:0/348142057 >> 172.21.15.53:0/348142057 conn(0x7f499406a800 msgr2=0x7f499406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.067+0000 7f499b400640 1 -- 172.21.15.53:0/348142057 shutdown_connections 2024-01-16T15:05:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.067+0000 7f499b400640 1 -- 172.21.15.53:0/348142057 wait complete. 2024-01-16T15:05:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.068+0000 7f499b400640 1 Processor -- start 2024-01-16T15:05:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.068+0000 7f499b400640 1 -- start start 2024-01-16T15:05:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.069+0000 7f499b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499414f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.069+0000 7f499b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f499413e360 0x7f499414d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.069+0000 7f499b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 0x7f499413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.069+0000 7f499b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499414f5f0 con 0x7f499415f9c0 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.069+0000 7f499b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499414dd10 con 0x7f499413e360 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.069+0000 7f499b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49941f42c0 con 0x7f499413df80 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 0x7f499413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 0x7f499413d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46908/0 (socket says 172.21.15.53:46908) 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 -- 172.21.15.53:0/1273615436 learned_addr learned my addr 172.21.15.53:0/1273615436 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4998a00640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499414f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 -- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 msgr2=0x7f499414f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f498be00640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f499413e360 0x7f499414d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499414f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 -- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f499413e360 msgr2=0x7f499414d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f499413e360 0x7f499414d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 -- 172.21.15.53:0/1273615436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497c009400 con 0x7f499415f9c0 2024-01-16T15:05:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.070+0000 7f4999400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 0x7f499413d7b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f498c00c7a0 tx=0x7f498c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.071+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00a640 con 0x7f499415f9c0 2024-01-16T15:05:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.071+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c00a7a0 con 0x7f499415f9c0 2024-01-16T15:05:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.071+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49941f45a0 con 0x7f499415f9c0 2024-01-16T15:05:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.071+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c00abd0 con 0x7f499415f9c0 2024-01-16T15:05:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.071+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49941f4b60 con 0x7f499415f9c0 2024-01-16T15:05:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.072+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49940675d0 con 0x7f499415f9c0 2024-01-16T15:05:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.073+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f498c00ad30 con 0x7f499415f9c0 2024-01-16T15:05:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.074+0000 7f4991600640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f496c06b860 0x7f496c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.074+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f498c089b90 con 0x7f499415f9c0 2024-01-16T15:05:07.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.075+0000 7f4998a00640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f496c06b860 0x7f496c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.076+0000 7f4998a00640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f496c06b860 0x7f496c06dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f498400b840 tx=0x7f4984008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:07.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.078+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f498c054450 con 0x7f499415f9c0 2024-01-16T15:05:07.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.278+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f499406ac80 con 0x7f499415f9c0 2024-01-16T15:05:07.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.279+0000 7f4991600640 1 -- 172.21.15.53:0/1273615436 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v52) v1 ==== 80+0+122 (secure 0 0 0) 0x7f498c018080 con 0x7f499415f9c0 2024-01-16T15:05:07.259 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6835/4293818684 2024-01-17T15:05:00.640610+0000 2024-01-16T15:05:07.259 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6834/4293818684 2024-01-17T15:05:00.640610+0000 2024-01-16T15:05:07.259 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:05:07.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.281+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f496c06b860 msgr2=0x7f496c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.281+0000 7f499b400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f496c06b860 0x7f496c06dd20 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f498400b840 tx=0x7f4984008040 comp rx=0 tx=0).stop 2024-01-16T15:05:07.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 msgr2=0x7f499413d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:07.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 0x7f499413d7b0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f498c00c7a0 tx=0x7f498c00cc60 comp rx=0 tx=0).stop 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 shutdown_connections 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f496c06b860 0x7f496c06dd20 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f499415f9c0 0x7f499413d7b0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f499413e360 0x7f499414d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 --2- 172.21.15.53:0/1273615436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f499413df80 0x7f499414f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 >> 172.21.15.53:0/1273615436 conn(0x7f499406a800 msgr2=0x7f499415cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 shutdown_connections 2024-01-16T15:05:07.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:07.282+0000 7f499b400640 1 -- 172.21.15.53:0/1273615436 wait complete. 2024-01-16T15:05:07.279 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:05:07.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:07.279 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:07.374 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[43192]: fuse finished with error 0 and tester_r 0 2024-01-16T15:05:07.379 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:05:07.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:07.379 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:05:07.564 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:05:07.564 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:07.621 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:05:07.621 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:05:07.621 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:07.622 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:05:07.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:05:07.677 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:05:07.677 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:05:07.677 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:07.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:05:07.733 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:05:07.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:07.733 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:05:07.786 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:05:07.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:05:07.787 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:08.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.037+0000 7f657e400640 1 Processor -- start 2024-01-16T15:05:08.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.038+0000 7f657e400640 1 -- start start 2024-01-16T15:05:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.039+0000 7f657e400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65781607a0 0x7f65780b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.039+0000 7f657e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65780b3eb0 0x7f65781610c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.039+0000 7f657e400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65780b2b90 con 0x7f6578160b80 2024-01-16T15:05:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.039+0000 7f657e400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65780d21d0 con 0x7f65781607a0 2024-01-16T15:05:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.039+0000 7f657e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65780b2e30 con 0x7f65780b3eb0 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6578160b80 0x7f6578161600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:60894/0 (socket says 172.21.15.53:60894) 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65781607a0 0x7f65780b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657da00640 1 -- 172.21.15.53:0/2709503606 learned_addr learned my addr 172.21.15.53:0/2709503606 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f656fc00640 1 --2- 172.21.15.53:0/2709503606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65780b3eb0 0x7f65781610c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657d000640 1 -- 172.21.15.53:0/2709503606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65780b3eb0 msgr2=0x7f65781610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657d000640 1 --2- 172.21.15.53:0/2709503606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65780b3eb0 0x7f65781610c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657d000640 1 -- 172.21.15.53:0/2709503606 >> v1:172.21.15.53:6789/0 conn(0x7f6578160b80 legacy=0x7f6578161600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.040+0000 7f657d000640 1 -- 172.21.15.53:0/2709503606 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65780b3110 con 0x7f65781607a0 2024-01-16T15:05:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.041+0000 7f657d000640 1 --2- 172.21.15.53:0/2709503606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65781607a0 0x7f65780b4330 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f65680097a0 tx=0x7f656802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121a84524d60cb92 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.042+0000 7f6577e00640 1 -- 172.21.15.53:0/2709503606 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f656802d040 con 0x7f65781607a0 2024-01-16T15:05:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.042+0000 7f6577e00640 1 -- 172.21.15.53:0/2709503606 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6568031810 con 0x7f65781607a0 2024-01-16T15:05:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.042+0000 7f6577e00640 1 -- 172.21.15.53:0/2709503606 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6568031ba0 con 0x7f65781607a0 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.042+0000 7f657e400640 1 -- 172.21.15.53:0/2709503606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65781607a0 msgr2=0x7f65780b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.042+0000 7f657e400640 1 --2- 172.21.15.53:0/2709503606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65781607a0 0x7f65780b4330 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f65680097a0 tx=0x7f656802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.043+0000 7f657e400640 1 -- 172.21.15.53:0/2709503606 shutdown_connections 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.043+0000 7f657e400640 1 --2- 172.21.15.53:0/2709503606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65780b3eb0 0x7f65781610c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.043+0000 7f657e400640 1 --2- 172.21.15.53:0/2709503606 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65781607a0 0x7f65780b4330 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.043+0000 7f657e400640 1 -- 172.21.15.53:0/2709503606 >> 172.21.15.53:0/2709503606 conn(0x7f657809c590 msgr2=0x7f65780d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.043+0000 7f657e400640 1 -- 172.21.15.53:0/2709503606 shutdown_connections 2024-01-16T15:05:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.043+0000 7f657e400640 1 -- 172.21.15.53:0/2709503606 wait complete. 2024-01-16T15:05:08.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.044+0000 7f657e400640 1 Processor -- start 2024-01-16T15:05:08.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.045+0000 7f657e400640 1 -- start start 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657e400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65780b3eb0 0x7f65780ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657e400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65781607a0 0x7f657815b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657e400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 0x7f65780d0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657e400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65780c6f20 con 0x7f6578160b80 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657e400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65780c71c0 con 0x7f65780b3eb0 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657e400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65780c74a0 con 0x7f65781607a0 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 0x7f65780d0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 0x7f65780d0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46926/0 (socket says 172.21.15.53:46926) 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 -- 172.21.15.53:0/424256762 learned_addr learned my addr 172.21.15.53:0/424256762 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657d000640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65780b3eb0 0x7f65780ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f656fc00640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65781607a0 0x7f657815b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 -- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65781607a0 msgr2=0x7f657815b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65781607a0 0x7f657815b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 -- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65780b3eb0 msgr2=0x7f65780ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65780b3eb0 0x7f65780ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657da00640 1 -- 172.21.15.53:0/424256762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6568009400 con 0x7f6578160b80 2024-01-16T15:05:08.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f656fc00640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65781607a0 0x7f657815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:08.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.046+0000 7f657d000640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65780b3eb0 0x7f65780ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:08.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.047+0000 7f657da00640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 0x7f65780d0e00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f657000cbc0 tx=0x7f657000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.047+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000ad40 con 0x7f6578160b80 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.047+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6570012400 con 0x7f6578160b80 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.047+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657000aea0 con 0x7f6578160b80 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.047+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65780c7780 con 0x7f6578160b80 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.047+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657815b840 con 0x7f6578160b80 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.048+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65780c7ca0 con 0x7f6578160b80 2024-01-16T15:05:08.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.049+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f6570007070 con 0x7f6578160b80 2024-01-16T15:05:08.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.050+0000 7f6575600640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f655006b860 0x7f655006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.050+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f657008a2f0 con 0x7f6578160b80 2024-01-16T15:05:08.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.051+0000 7f657d000640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f655006b860 0x7f655006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.052+0000 7f657d000640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f655006b860 0x7f655006dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6568036000 tx=0x7f656803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.055+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6570054bd0 con 0x7f6578160b80 2024-01-16T15:05:08.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.278+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65780d0550 con 0x7f6578160b80 2024-01-16T15:05:08.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.278+0000 7f6575600640 1 -- 172.21.15.53:0/424256762 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6570017090 con 0x7f6578160b80 2024-01-16T15:05:08.258 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:08.258 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3977954404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3977954404},{"type":"v1","addr":"172.21.15.146:6835","nonce":3977954404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3669971149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3669971149},{"type":"v1","addr":"172.21.15.146:6837","nonce":3669971149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5709,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1838972240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1838972240},{"type":"v1","addr":"172.21.15.53:6833","nonce":1838972240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3852579477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3852579477},{"type":"v1","addr":"172.21.15.53:6835","nonce":3852579477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-01-16T15:05:08.258 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 28 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f655006b860 msgr2=0x7f655006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f655006b860 0x7f655006dd20 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6568036000 tx=0x7f656803c040 comp rx=0 tx=0).stop 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 msgr2=0x7f65780d0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 0x7f65780d0e00 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f657000cbc0 tx=0x7f657000a430 comp rx=0 tx=0).stop 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 shutdown_connections 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f655006b860 0x7f655006dd20 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6578160b80 0x7f65780d0e00 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65781607a0 0x7f657815b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 --2- 172.21.15.53:0/424256762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65780b3eb0 0x7f65780ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 >> 172.21.15.53:0/424256762 conn(0x7f657809c590 msgr2=0x7f6578161030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.281+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 shutdown_connections 2024-01-16T15:05:08.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.282+0000 7f657e400640 1 -- 172.21.15.53:0/424256762 wait complete. 2024-01-16T15:05:08.279 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:05:08.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.520+0000 7f53e7c00640 1 Processor -- start 2024-01-16T15:05:08.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.520+0000 7f53e7c00640 1 -- start start 2024-01-16T15:05:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.521+0000 7f53e7c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53e013e360 0x7f53e00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e7c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53e015f9c0 0x7f53e00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e7c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e015c5f0 con 0x7f53e013df80 2024-01-16T15:05:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e7c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e006c470 con 0x7f53e015f9c0 2024-01-16T15:05:08.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e7c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53e006c750 con 0x7f53e013e360 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53e015f9c0 0x7f53e00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53e015f9c0 0x7f53e00cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46936/0 (socket says 172.21.15.53:46936) 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 -- 172.21.15.53:0/2519016919 learned_addr learned my addr 172.21.15.53:0/2519016919 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53dfe00640 1 --2- 172.21.15.53:0/2519016919 >> v2:172.21.15.146:3300/0 conn(0x7f53e013e360 0x7f53e00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 -- 172.21.15.53:0/2519016919 >> v2:172.21.15.146:3300/0 conn(0x7f53e013e360 msgr2=0x7f53e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 --2- 172.21.15.53:0/2519016919 >> v2:172.21.15.146:3300/0 conn(0x7f53e013e360 0x7f53e00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 -- 172.21.15.53:0/2519016919 >> v1:172.21.15.146:6789/0 conn(0x7f53e013df80 legacy=0x7f53e015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.522+0000 7f53e5c00640 1 -- 172.21.15.53:0/2519016919 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e006ca30 con 0x7f53e015f9c0 2024-01-16T15:05:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.524+0000 7f53e5c00640 1 --2- 172.21.15.53:0/2519016919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e015f9c0 0x7f53e00cf7d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f53e00ccae0 tx=0x7f53d802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c7afc7baac2dbe6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.524+0000 7f53df400640 1 -- 172.21.15.53:0/2519016919 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d802ca50 con 0x7f53e015f9c0 2024-01-16T15:05:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.524+0000 7f53df400640 1 -- 172.21.15.53:0/2519016919 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d802cbb0 con 0x7f53e015f9c0 2024-01-16T15:05:08.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.524+0000 7f53df400640 1 -- 172.21.15.53:0/2519016919 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d80343f0 con 0x7f53e015f9c0 2024-01-16T15:05:08.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 -- 172.21.15.53:0/2519016919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e015f9c0 msgr2=0x7f53e00cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 --2- 172.21.15.53:0/2519016919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e015f9c0 0x7f53e00cf7d0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f53e00ccae0 tx=0x7f53d802beb0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 -- 172.21.15.53:0/2519016919 shutdown_connections 2024-01-16T15:05:08.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 --2- 172.21.15.53:0/2519016919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e015f9c0 0x7f53e00cf7d0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 --2- 172.21.15.53:0/2519016919 >> v2:172.21.15.146:3300/0 conn(0x7f53e013e360 0x7f53e00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 -- 172.21.15.53:0/2519016919 >> 172.21.15.53:0/2519016919 conn(0x7f53e006a800 msgr2=0x7f53e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 -- 172.21.15.53:0/2519016919 shutdown_connections 2024-01-16T15:05:08.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.525+0000 7f53e7c00640 1 -- 172.21.15.53:0/2519016919 wait complete. 2024-01-16T15:05:08.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.527+0000 7f53e7c00640 1 Processor -- start 2024-01-16T15:05:08.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.527+0000 7f53e7c00640 1 -- start start 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e7c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 0x7f53e013d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e7c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53e013e360 0x7f53e00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e7c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53e015f9c0 0x7f53e00b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e7c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00aaaa0 con 0x7f53e013df80 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e7c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00a9a40 con 0x7f53e015f9c0 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e7c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53e00a9ce0 con 0x7f53e013e360 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 0x7f53e013d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 0x7f53e013d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46948/0 (socket says 172.21.15.53:46948) 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 -- 172.21.15.53:0/299964166 learned_addr learned my addr 172.21.15.53:0/299964166 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53e015f9c0 0x7f53e00b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 -- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53e013e360 msgr2=0x7f53e00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53dfe00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53e013e360 0x7f53e00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53e013e360 0x7f53e00b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 -- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53e015f9c0 msgr2=0x7f53e00b3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.528+0000 7f53e5200640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53e015f9c0 0x7f53e00b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53e5200640 1 -- 172.21.15.53:0/299964166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d8009b30 con 0x7f53e013df80 2024-01-16T15:05:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53e5200640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 0x7f53e013d940 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f53d000b3f0 tx=0x7f53d000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0011830 con 0x7f53e013df80 2024-01-16T15:05:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53d0011990 con 0x7f53e013df80 2024-01-16T15:05:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53d0011d40 con 0x7f53e013df80 2024-01-16T15:05:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e00a9fc0 con 0x7f53e013df80 2024-01-16T15:05:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.529+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e00aa580 con 0x7f53e013df80 2024-01-16T15:05:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.531+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e0073c30 con 0x7f53e013df80 2024-01-16T15:05:08.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.532+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f53d001a400 con 0x7f53e013df80 2024-01-16T15:05:08.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.532+0000 7f53dcc00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f53b806b860 0x7f53b806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.532+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f53d008a170 con 0x7f53e013df80 2024-01-16T15:05:08.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.533+0000 7f53dfe00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f53b806b860 0x7f53b806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.534+0000 7f53dfe00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f53b806b860 0x7f53b806dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f53c8006fd0 tx=0x7f53c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.537+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53e0073c30 con 0x7f53e013df80 2024-01-16T15:05:08.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.733+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f53e0073c30 con 0x7f53e013df80 2024-01-16T15:05:08.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.740+0000 7f53dcc00640 1 -- 172.21.15.53:0/299964166 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v17) v1 ==== 120+0+0 (secure 0 0 0) 0x7f53e0073c30 con 0x7f53e013df80 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f53b806b860 msgr2=0x7f53b806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f53b806b860 0x7f53b806dd20 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f53c8006fd0 tx=0x7f53c8008040 comp rx=0 tx=0).stop 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 msgr2=0x7f53e013d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 0x7f53e013d940 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f53d000b3f0 tx=0x7f53d000b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 shutdown_connections 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f53b806b860 0x7f53b806dd20 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53e015f9c0 0x7f53e00b3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53e013e360 0x7f53e00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 --2- 172.21.15.53:0/299964166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53e013df80 0x7f53e013d940 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.743+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 >> 172.21.15.53:0/299964166 conn(0x7f53e006a800 msgr2=0x7f53e015a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.744+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 shutdown_connections 2024-01-16T15:05:08.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.744+0000 7f53e7c00640 1 -- 172.21.15.53:0/299964166 wait complete. 2024-01-16T15:05:08.743 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:05:08.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.985+0000 7f223f200640 1 Processor -- start 2024-01-16T15:05:08.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.986+0000 7f223f200640 1 -- start start 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.987+0000 7f223f200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22400cf9a0 0x7f22400cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.987+0000 7f223f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.987+0000 7f223f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22400d0320 con 0x7f224015d4b0 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.987+0000 7f223f200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22400d1c60 con 0x7f22400cf9a0 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.987+0000 7f223f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b2730 con 0x7f22400b31d0 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22400cf9a0 0x7f22400cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22400cf9a0 0x7f22400cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46956/0 (socket says 172.21.15.53:46956) 2024-01-16T15:05:08.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 -- 172.21.15.53:0/59279948 learned_addr learned my addr 172.21.15.53:0/59279948 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:08.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223e800640 1 --2- 172.21.15.53:0/59279948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 -- 172.21.15.53:0/59279948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 msgr2=0x7f22400b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 --2- 172.21.15.53:0/59279948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 -- 172.21.15.53:0/59279948 >> v1:172.21.15.146:6789/0 conn(0x7f224015d4b0 legacy=0x7f22400d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:08.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.988+0000 7f223d400640 1 -- 172.21.15.53:0/59279948 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22400d1ec0 con 0x7f22400cf9a0 2024-01-16T15:05:08.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.989+0000 7f223d400640 1 --2- 172.21.15.53:0/59279948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22400cf9a0 0x7f22400cfde0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2230009e50 tx=0x7f223002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d9ca68398ac268 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223ca00640 1 -- 172.21.15.53:0/59279948 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223002ca70 con 0x7f22400cf9a0 2024-01-16T15:05:08.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223ca00640 1 -- 172.21.15.53:0/59279948 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223002cbd0 con 0x7f22400cf9a0 2024-01-16T15:05:08.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223ca00640 1 -- 172.21.15.53:0/59279948 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22300343f0 con 0x7f22400cf9a0 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223f200640 1 -- 172.21.15.53:0/59279948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22400cf9a0 msgr2=0x7f22400cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223f200640 1 --2- 172.21.15.53:0/59279948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22400cf9a0 0x7f22400cfde0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2230009e50 tx=0x7f223002bed0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223f200640 1 -- 172.21.15.53:0/59279948 shutdown_connections 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223f200640 1 --2- 172.21.15.53:0/59279948 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223f200640 1 --2- 172.21.15.53:0/59279948 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22400cf9a0 0x7f22400cfde0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.990+0000 7f223f200640 1 -- 172.21.15.53:0/59279948 >> 172.21.15.53:0/59279948 conn(0x7f22400d3000 msgr2=0x7f22400d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.991+0000 7f223f200640 1 -- 172.21.15.53:0/59279948 shutdown_connections 2024-01-16T15:05:08.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.991+0000 7f223f200640 1 -- 172.21.15.53:0/59279948 wait complete. 2024-01-16T15:05:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.992+0000 7f223f200640 1 Processor -- start 2024-01-16T15:05:08.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.992+0000 7f223f200640 1 -- start start 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.993+0000 7f223f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.993+0000 7f223f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22400cf9a0 0x7f22400affb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.993+0000 7f223f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 0x7f224009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.993+0000 7f223f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b16f0 con 0x7f224015d4b0 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.993+0000 7f223f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22400a0d90 con 0x7f22400cf9a0 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.993+0000 7f223f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22400b0a70 con 0x7f22400b31d0 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 0x7f224009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 0x7f224009cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46958/0 (socket says 172.21.15.53:46958) 2024-01-16T15:05:08.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 -- 172.21.15.53:0/2975071515 learned_addr learned my addr 172.21.15.53:0/2975071515 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:08.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223de00640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223d400640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22400cf9a0 0x7f22400affb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 -- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 msgr2=0x7f22400a0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400a0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 -- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22400cf9a0 msgr2=0x7f22400affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22400cf9a0 0x7f22400affb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 -- 172.21.15.53:0/2975071515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2230009b30 con 0x7f224015d4b0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.994+0000 7f223e800640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 0x7f224009cce0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f223800c550 tx=0x7f223800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.995+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238004220 con 0x7f224015d4b0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.995+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2238004380 con 0x7f224015d4b0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.995+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22400b0d50 con 0x7f224015d4b0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.995+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2238003830 con 0x7f224015d4b0 2024-01-16T15:05:08.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.995+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22400b1310 con 0x7f224015d4b0 2024-01-16T15:05:08.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.996+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2240067450 con 0x7f224015d4b0 2024-01-16T15:05:08.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.997+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f2238003990 con 0x7f224015d4b0 2024-01-16T15:05:08.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.999+0000 7f2236000640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f221806b860 0x7f221806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:08.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.999+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f223808a0e0 con 0x7f224015d4b0 2024-01-16T15:05:08.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:08.999+0000 7f223de00640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f221806b860 0x7f221806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:08.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.000+0000 7f223de00640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f221806b860 0x7f221806dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f224015c620 tx=0x7f2228009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:08.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.003+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22380bce90 con 0x7f224015d4b0 2024-01-16T15:05:09.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.200+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f2240073e10 con 0x7f224015d4b0 2024-01-16T15:05:09.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.206+0000 7f2236000640 1 -- 172.21.15.53:0/2975071515 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v18) v1 ==== 115+0+0 (secure 0 0 0) 0x7f223801e0b0 con 0x7f224015d4b0 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f221806b860 msgr2=0x7f221806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f221806b860 0x7f221806dd20 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f224015c620 tx=0x7f2228009040 comp rx=0 tx=0).stop 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 msgr2=0x7f224009cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 0x7f224009cce0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f223800c550 tx=0x7f223800ca10 comp rx=0 tx=0).stop 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 shutdown_connections 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f221806b860 0x7f221806dd20 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f224015d4b0 0x7f224009cce0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22400cf9a0 0x7f22400affb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 --2- 172.21.15.53:0/2975071515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22400b31d0 0x7f22400a0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 >> 172.21.15.53:0/2975071515 conn(0x7f22400d3000 msgr2=0x7f224006c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 shutdown_connections 2024-01-16T15:05:09.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.209+0000 7f223f200640 1 -- 172.21.15.53:0/2975071515 wait complete. 2024-01-16T15:05:09.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:05:09.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.460+0000 7f0c59c00640 1 Processor -- start 2024-01-16T15:05:09.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.460+0000 7f0c59c00640 1 -- start start 2024-01-16T15:05:09.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.462+0000 7f0c59c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 0x7f0c54070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.462+0000 7f0c59c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540706a0 con 0x7f0c5406cb10 2024-01-16T15:05:09.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.462+0000 7f0c59c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540708d0 con 0x7f0c5406c730 2024-01-16T15:05:09.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.462+0000 7f0c59c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540b3290 con 0x7f0c540b2e00 2024-01-16T15:05:09.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c59200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0c5406cb10 0x7f0c5406cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:60904/0 (socket says 172.21.15.53:60904) 2024-01-16T15:05:09.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c59200640 1 -- 172.21.15.53:0/3314709467 learned_addr learned my addr 172.21.15.53:0/3314709467 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:09.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c4be00640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 0x7f0c54070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821656089 0 0) 0x7f0c540706a0 con 0x7f0c5406cb10 2024-01-16T15:05:09.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c34003520 con 0x7f0c5406cb10 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3942876860 0 0) 0x7f0c34003520 con 0x7f0c5406cb10 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 msgr2=0x7f0c54070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.463+0000 7f0c53400640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 0x7f0c54070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 >> v1:172.21.15.146:6789/0 conn(0x7f0c5406c730 legacy=0x7f0c54071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c54070bb0 con 0x7f0c5406cb10 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0c4c0019f0 con 0x7f0c5406cb10 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 >> v1:172.21.15.53:6789/0 conn(0x7f0c5406cb10 legacy=0x7f0c5406cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c340039a0 0x7f0c34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 0x7f0c340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34004d00 0x7f0c34005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3400bab0 con 0x7f0c340039a0 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3400bc20 con 0x7f0c34004d00 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3400bde0 con 0x7f0c540b2e00 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53e00640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c340039a0 0x7f0c34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c59200640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34004d00 0x7f0c34005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53e00640 1 -- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 msgr2=0x7f0c340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53e00640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540b2e00 0x7f0c340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53e00640 1 -- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34004d00 msgr2=0x7f0c34005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53e00640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c34004d00 0x7f0c34005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.464+0000 7f0c53e00640 1 -- 172.21.15.53:0/3314709467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c34003520 con 0x7f0c340039a0 2024-01-16T15:05:09.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c53e00640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c340039a0 0x7f0c34003dc0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0c44001750 tx=0x7f0c340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96365375386ba617 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c440052d0 con 0x7f0c340039a0 2024-01-16T15:05:09.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4402e050 con 0x7f0c340039a0 2024-01-16T15:05:09.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c53400640 1 -- 172.21.15.53:0/3314709467 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c440035f0 con 0x7f0c340039a0 2024-01-16T15:05:09.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c59c00640 1 -- 172.21.15.53:0/3314709467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c340039a0 msgr2=0x7f0c34003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3314709467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c340039a0 0x7f0c34003dc0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f0c44001750 tx=0x7f0c340057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.466+0000 7f0c59200640 1 -- 172.21.15.53:0/3314709467 reap_dead start 2024-01-16T15:05:09.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.467+0000 7f0c59c00640 1 -- 172.21.15.53:0/3314709467 shutdown_connections 2024-01-16T15:05:09.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.467+0000 7f0c59c00640 1 -- 172.21.15.53:0/3314709467 >> 172.21.15.53:0/3314709467 conn(0x7f0c5406a7d0 msgr2=0x7f0c5406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.467+0000 7f0c59c00640 1 -- 172.21.15.53:0/3314709467 shutdown_connections 2024-01-16T15:05:09.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.467+0000 7f0c59c00640 1 -- 172.21.15.53:0/3314709467 wait complete. 2024-01-16T15:05:09.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.468+0000 7f0c59c00640 1 Processor -- start 2024-01-16T15:05:09.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.469+0000 7f0c59c00640 1 -- start start 2024-01-16T15:05:09.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c59c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c340039a0 0x7f0c54149e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c59c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 0x7f0c541f7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c59c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540ad630 0x7f0c540adab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c59c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415fa10 con 0x7f0c54143b60 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c59c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54143fe0 con 0x7f0c340039a0 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c59c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541f8540 con 0x7f0c540ad630 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 0x7f0c541f7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 0x7f0c541f7cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46968/0 (socket says 172.21.15.53:46968) 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 -- 172.21.15.53:0/3186443364 learned_addr learned my addr 172.21.15.53:0/3186443364 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 -- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540ad630 msgr2=0x7f0c540adab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540ad630 0x7f0c540adab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 -- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c340039a0 msgr2=0x7f0c54149e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c340039a0 0x7f0c54149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.470+0000 7f0c4be00640 1 -- 172.21.15.53:0/3186443364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c541f8820 con 0x7f0c54143b60 2024-01-16T15:05:09.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.471+0000 7f0c4be00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 0x7f0c541f7cf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00e7c0 tx=0x7f0c3c00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.471+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00eed0 con 0x7f0c54143b60 2024-01-16T15:05:09.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.471+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c3c00c940 con 0x7f0c54143b60 2024-01-16T15:05:09.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.471+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00ccd0 con 0x7f0c54143b60 2024-01-16T15:05:09.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.471+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c541f8b00 con 0x7f0c54143b60 2024-01-16T15:05:09.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.471+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c541f90c0 con 0x7f0c54143b60 2024-01-16T15:05:09.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.472+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c54067d40 con 0x7f0c54143b60 2024-01-16T15:05:09.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.473+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f0c3c013470 con 0x7f0c54143b60 2024-01-16T15:05:09.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.475+0000 7f0c50c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c2c06b890 0x7f0c2c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.475+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0c3c08a330 con 0x7f0c54143b60 2024-01-16T15:05:09.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.475+0000 7f0c53e00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c2c06b890 0x7f0c2c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.477+0000 7f0c53e00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c2c06b890 0x7f0c2c06dd50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0c44033000 tx=0x7f0c4402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.480+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c3c054c30 con 0x7f0c54143b60 2024-01-16T15:05:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.679+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f0c5406e0f0 con 0x7f0c54143b60 2024-01-16T15:05:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.684+0000 7f0c50c00640 1 -- 172.21.15.53:0/3186443364 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0c3c058250 con 0x7f0c54143b60 2024-01-16T15:05:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.687+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c2c06b890 msgr2=0x7f0c2c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.687+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c2c06b890 0x7f0c2c06dd50 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f0c44033000 tx=0x7f0c4402f000 comp rx=0 tx=0).stop 2024-01-16T15:05:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.687+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 msgr2=0x7f0c541f7cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.687+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 0x7f0c541f7cf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00e7c0 tx=0x7f0c3c00ec80 comp rx=0 tx=0).stop 2024-01-16T15:05:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 shutdown_connections 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c2c06b890 0x7f0c2c06dd50 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c540ad630 0x7f0c540adab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c54143b60 0x7f0c541f7cf0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 --2- 172.21.15.53:0/3186443364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c340039a0 0x7f0c54149e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 >> 172.21.15.53:0/3186443364 conn(0x7f0c5406a7d0 msgr2=0x7f0c540cd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 shutdown_connections 2024-01-16T15:05:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.688+0000 7f0c59c00640 1 -- 172.21.15.53:0/3186443364 wait complete. 2024-01-16T15:05:09.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones' 2024-01-16T15:05:09.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.924+0000 7f47d7200640 1 Processor -- start 2024-01-16T15:05:09.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.924+0000 7f47d7200640 1 -- start start 2024-01-16T15:05:09.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47d806c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d80b2e10 con 0x7f47d8071f00 2024-01-16T15:05:09.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d8146bc0 con 0x7f47d8071410 2024-01-16T15:05:09.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d806cec0 con 0x7f47d8146770 2024-01-16T15:05:09.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47cfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f47d8071f00 0x7f47d8164ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:60916/0 (socket says 172.21.15.53:60916) 2024-01-16T15:05:09.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47cfe00640 1 -- 172.21.15.53:0/1226284371 learned_addr learned my addr 172.21.15.53:0/1226284371 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:09.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d6800640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47d806c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881652949 0 0) 0x7f47d80b2e10 con 0x7f47d8071f00 2024-01-16T15:05:09.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.926+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b8003520 con 0x7f47d8071f00 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4047689979 0 0) 0x7f47b8003520 con 0x7f47d8071f00 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 msgr2=0x7f47d806c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47d806c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 >> v1:172.21.15.146:6789/0 conn(0x7f47d8071410 legacy=0x7f47d80717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d806d1a0 con 0x7f47d8071f00 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f47c0001670 con 0x7f47d8071f00 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 >> v1:172.21.15.53:6789/0 conn(0x7f47d8071f00 legacy=0x7f47d8164ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47b80039a0 0x7f47b8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47b8004d00 0x7f47b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d806cec0 con 0x7f47b8004d00 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47b800bae0 con 0x7f47b80039a0 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b800bca0 con 0x7f47d8146770 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.927+0000 7f47d6800640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47b8004d00 0x7f47b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47d5e00640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47b80039a0 0x7f47b8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47cfe00640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47d6800640 1 -- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 msgr2=0x7f47b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47d6800640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47d6800640 1 -- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47b80039a0 msgr2=0x7f47b8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47d6800640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47b80039a0 0x7f47b8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47d6800640 1 -- 172.21.15.53:0/1226284371 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b8003520 con 0x7f47b8004d00 2024-01-16T15:05:09.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.928+0000 7f47cfe00640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d8146770 0x7f47b80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.929+0000 7f47d6800640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47b8004d00 0x7f47b8005180 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009fc0 tx=0x7f47b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee81375410adc110 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.929+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d002cc20 con 0x7f47b8004d00 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.929+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d000f040 con 0x7f47b8004d00 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.929+0000 7f47d5400640 1 -- 172.21.15.53:0/1226284371 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d0034440 con 0x7f47b8004d00 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d7200640 1 -- 172.21.15.53:0/1226284371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47b8004d00 msgr2=0x7f47b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d7200640 1 --2- 172.21.15.53:0/1226284371 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47b8004d00 0x7f47b8005180 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f47d0009fc0 tx=0x7f47b80057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d6800640 1 -- 172.21.15.53:0/1226284371 reap_dead start 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d7200640 1 -- 172.21.15.53:0/1226284371 shutdown_connections 2024-01-16T15:05:09.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d7200640 1 -- 172.21.15.53:0/1226284371 >> 172.21.15.53:0/1226284371 conn(0x7f47d806a800 msgr2=0x7f47d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:09.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d7200640 1 -- 172.21.15.53:0/1226284371 shutdown_connections 2024-01-16T15:05:09.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.930+0000 7f47d7200640 1 -- 172.21.15.53:0/1226284371 wait complete. 2024-01-16T15:05:09.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.932+0000 7f47d7200640 1 Processor -- start 2024-01-16T15:05:09.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.932+0000 7f47d7200640 1 -- start start 2024-01-16T15:05:09.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 0x7f47d815e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d815fb70 0x7f47d80cbc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d8144bc0 0x7f47d813c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d815f830 con 0x7f47d81447e0 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d815ef90 con 0x7f47d8144bc0 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d815f270 con 0x7f47d815fb70 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 0x7f47d815e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 0x7f47d815e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46986/0 (socket says 172.21.15.53:46986) 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.933+0000 7f47d5e00640 1 -- 172.21.15.53:0/3069440194 learned_addr learned my addr 172.21.15.53:0/3069440194 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:09.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47cfe00640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d815fb70 0x7f47d80cbc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d6800640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d8144bc0 0x7f47d813c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d5e00640 1 -- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d815fb70 msgr2=0x7f47d80cbc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d5e00640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d815fb70 0x7f47d80cbc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d5e00640 1 -- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d8144bc0 msgr2=0x7f47d813c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:09.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d5e00640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d8144bc0 0x7f47d813c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:09.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d5e00640 1 -- 172.21.15.53:0/3069440194 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d815f550 con 0x7f47d81447e0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d6800640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d8144bc0 0x7f47d813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47d5e00640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 0x7f47d815e740 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f47c800e550 tx=0x7f47c800ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c80052e0 con 0x7f47d81447e0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.934+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47c8005440 con 0x7f47d81447e0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.935+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47c8003810 con 0x7f47d81447e0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.935+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d813c310 con 0x7f47d81447e0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.935+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d80cc1a0 con 0x7f47d81447e0 2024-01-16T15:05:09.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.936+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d813c170 con 0x7f47d81447e0 2024-01-16T15:05:09.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.937+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f47c8003970 con 0x7f47d81447e0 2024-01-16T15:05:09.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.938+0000 7f47ce000640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b006b860 0x7f47b006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:09.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.938+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f47c808a070 con 0x7f47d81447e0 2024-01-16T15:05:09.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.938+0000 7f47cfe00640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b006b860 0x7f47b006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:09.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.939+0000 7f47cfe00640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b006b860 0x7f47b006dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f47c0001eb0 tx=0x7f47c0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:09.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:09.943+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47d813c170 con 0x7f47d81447e0 2024-01-16T15:05:10.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:10.213+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]} v 0) v1 -- 0x7f47d813c170 con 0x7f47d81447e0 2024-01-16T15:05:11.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.124+0000 7f47ce000640 1 -- 172.21.15.53:0/3069440194 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bar_for_4_clones"]}]=0 v261) v1 ==== 153+0+0 (secure 0 0 0) 0x7f47d813c170 con 0x7f47d81447e0 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b006b860 msgr2=0x7f47b006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b006b860 0x7f47b006dd20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f47c0001eb0 tx=0x7f47c0009040 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 msgr2=0x7f47d815e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 0x7f47d815e740 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f47c800e550 tx=0x7f47c800ea10 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 shutdown_connections 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47b006b860 0x7f47b006dd20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47d8144bc0 0x7f47d813c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47d815fb70 0x7f47d80cbc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 --2- 172.21.15.53:0/3069440194 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47d81447e0 0x7f47d815e740 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 >> 172.21.15.53:0/3069440194 conn(0x7f47d806a800 msgr2=0x7f47d80b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.128+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 shutdown_connections 2024-01-16T15:05:11.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.129+0000 7f47d7200640 1 -- 172.21.15.53:0/3069440194 wait complete. 2024-01-16T15:05:11.128 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:05:11.128 INFO:tasks.cephfs_test_runner:Starting test: test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:05:11.128 INFO:tasks.cephfs_test_runner:test_progress_bars_for_7_clones (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:05:11.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.374+0000 7f4823200640 1 Processor -- start 2024-01-16T15:05:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.375+0000 7f4823200640 1 -- start start 2024-01-16T15:05:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4823200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f481c160920 0x7f481c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4823200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c0d0a10 con 0x7f481c0cfea0 2024-01-16T15:05:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4823200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f481c161e30 con 0x7f481c0cf9f0 2024-01-16T15:05:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4823200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f481c0d1f50 con 0x7f481c160920 2024-01-16T15:05:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4821e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f481c160920 0x7f481c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4821e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f481c160920 0x7f481c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46996/0 (socket says 172.21.15.53:46996) 2024-01-16T15:05:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.376+0000 7f4821e00640 1 -- 172.21.15.53:0/3723191292 learned_addr learned my addr 172.21.15.53:0/3723191292 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.377+0000 7f4821e00640 1 -- 172.21.15.53:0/3723191292 >> v1:172.21.15.146:6789/0 conn(0x7f481c0cf9f0 legacy=0x7f481c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.377+0000 7f4821e00640 1 -- 172.21.15.53:0/3723191292 >> v1:172.21.15.53:6789/0 conn(0x7f481c0cfea0 legacy=0x7f481c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.377+0000 7f4821e00640 1 -- 172.21.15.53:0/3723191292 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f481c0cf080 con 0x7f481c160920 2024-01-16T15:05:11.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.378+0000 7f4821e00640 1 --2- 172.21.15.53:0/3723191292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c160920 0x7f481c0b3f30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f480c009870 tx=0x7f480c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83cf00b907373339 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.378+0000 7f4820a00640 1 -- 172.21.15.53:0/3723191292 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c030860 con 0x7f481c160920 2024-01-16T15:05:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.378+0000 7f4820a00640 1 -- 172.21.15.53:0/3723191292 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f480c0309c0 con 0x7f481c160920 2024-01-16T15:05:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.378+0000 7f4820a00640 1 -- 172.21.15.53:0/3723191292 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f480c030d50 con 0x7f481c160920 2024-01-16T15:05:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 -- 172.21.15.53:0/3723191292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c160920 msgr2=0x7f481c0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 --2- 172.21.15.53:0/3723191292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c160920 0x7f481c0b3f30 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f480c009870 tx=0x7f480c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 -- 172.21.15.53:0/3723191292 shutdown_connections 2024-01-16T15:05:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 --2- 172.21.15.53:0/3723191292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c160920 0x7f481c0b3f30 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 -- 172.21.15.53:0/3723191292 >> 172.21.15.53:0/3723191292 conn(0x7f481c0d3090 msgr2=0x7f481c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 -- 172.21.15.53:0/3723191292 shutdown_connections 2024-01-16T15:05:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.379+0000 7f4823200640 1 -- 172.21.15.53:0/3723191292 wait complete. 2024-01-16T15:05:11.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.381+0000 7f4823200640 1 Processor -- start 2024-01-16T15:05:11.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.381+0000 7f4823200640 1 -- start start 2024-01-16T15:05:11.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4823200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c0cf9f0 0x7f481c15cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4823200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 0x7f481c150c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4823200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c160920 0x7f481c0b8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4823200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c15d1e0 con 0x7f481c0cfea0 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4823200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0ca5f0 con 0x7f481c0cf9f0 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4823200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f481c0ca8d0 con 0x7f481c160920 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4821400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 0x7f481c150c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4821400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 0x7f481c150c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47008/0 (socket says 172.21.15.53:47008) 2024-01-16T15:05:11.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.382+0000 7f4821400640 1 -- 172.21.15.53:0/540672183 learned_addr learned my addr 172.21.15.53:0/540672183 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:11.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821e00640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c0cf9f0 0x7f481c15cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4822800640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c160920 0x7f481c0b8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821400640 1 -- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c160920 msgr2=0x7f481c0b8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821400640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c160920 0x7f481c0b8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821400640 1 -- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c0cf9f0 msgr2=0x7f481c15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821400640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c0cf9f0 0x7f481c15cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821400640 1 -- 172.21.15.53:0/540672183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f480c0094d0 con 0x7f481c0cfea0 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821e00640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c0cf9f0 0x7f481c15cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4822800640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c160920 0x7f481c0b8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.383+0000 7f4821400640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 0x7f481c150c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f480400b3e0 tx=0x7f480400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.384+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804011830 con 0x7f481c0cfea0 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.384+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48040119b0 con 0x7f481c0cfea0 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.384+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4804011d60 con 0x7f481c0cfea0 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.384+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f481c0cabb0 con 0x7f481c0cfea0 2024-01-16T15:05:11.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.384+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f481c0cb170 con 0x7f481c0cfea0 2024-01-16T15:05:11.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.385+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f481c0cb390 con 0x7f481c0cfea0 2024-01-16T15:05:11.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.386+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f480401a400 con 0x7f481c0cfea0 2024-01-16T15:05:11.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.387+0000 7f481a000640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47f406b860 0x7f47f406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.387+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f480408a400 con 0x7f481c0cfea0 2024-01-16T15:05:11.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.388+0000 7f4821e00640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47f406b860 0x7f47f406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.389+0000 7f4821e00640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47f406b860 0x7f47f406dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f480c036000 tx=0x7f480c03b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.392+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4804054da0 con 0x7f481c0cfea0 2024-01-16T15:05:11.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.587+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f481c067450 con 0x7f481c0cfea0 2024-01-16T15:05:11.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.588+0000 7f481a000640 1 -- 172.21.15.53:0/540672183 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+12950 (secure 0 0 0) 0x7f48040583c0 con 0x7f481c0cfea0 2024-01-16T15:05:11.567 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:11.567 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":52,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:03.654310+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6835/4293818684":"2024-01-17T15:05:00.640610+0000","172.21.15.53:6834/4293818684":"2024-01-17T15:05:00.640610+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:11.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47f406b860 msgr2=0x7f47f406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47f406b860 0x7f47f406dd20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f480c036000 tx=0x7f480c03b040 comp rx=0 tx=0).stop 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 msgr2=0x7f481c150c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 0x7f481c150c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f480400b3e0 tx=0x7f480400b8a0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 shutdown_connections 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47f406b860 0x7f47f406dd20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f481c160920 0x7f481c0b8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f481c0cfea0 0x7f481c150c20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 --2- 172.21.15.53:0/540672183 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f481c0cf9f0 0x7f481c15cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 >> 172.21.15.53:0/540672183 conn(0x7f481c0d3090 msgr2=0x7f481c15b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 shutdown_connections 2024-01-16T15:05:11.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.591+0000 7f4823200640 1 -- 172.21.15.53:0/540672183 wait complete. 2024-01-16T15:05:11.588 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:05:11.588 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:05:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.835+0000 7f0a37800640 1 Processor -- start 2024-01-16T15:05:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.836+0000 7f0a37800640 1 -- start start 2024-01-16T15:05:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.837+0000 7f0a37800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 0x7f0a3015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.837+0000 7f0a37800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a3013e360 0x7f0a300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.837+0000 7f0a37800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a3006c470 con 0x7f0a3015f9c0 2024-01-16T15:05:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.837+0000 7f0a37800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a3006c750 con 0x7f0a3013e360 2024-01-16T15:05:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.837+0000 7f0a37800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3006ca30 con 0x7f0a3013df80 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.838+0000 7f0a34e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 0x7f0a3015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.838+0000 7f0a27c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0a3013e360 0x7f0a300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.838+0000 7f0a34e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 0x7f0a3015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:46652/0 (socket says 172.21.15.53:46652) 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.838+0000 7f0a34e00640 1 -- 172.21.15.53:0/486765194 learned_addr learned my addr 172.21.15.53:0/486765194 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.839+0000 7f0a27c00640 1 -- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 msgr2=0x7f0a3015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.839+0000 7f0a27c00640 1 --2- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 0x7f0a3015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.839+0000 7f0a27c00640 1 -- 172.21.15.53:0/486765194 >> v1:172.21.15.146:6789/0 conn(0x7f0a3015f9c0 legacy=0x7f0a301408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.839+0000 7f0a27c00640 1 -- 172.21.15.53:0/486765194 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a3006cd10 con 0x7f0a3013e360 2024-01-16T15:05:11.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.839+0000 7f0a34e00640 1 --2- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 0x7f0a3015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:05:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.840+0000 7f0a27c00640 1 --2- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3013e360 0x7f0a300cf290 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0a180097a0 tx=0x7f0a1802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d227947a006ec152 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.841+0000 7f0a2fe00640 1 -- 172.21.15.53:0/486765194 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1802d040 con 0x7f0a3013e360 2024-01-16T15:05:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.841+0000 7f0a2fe00640 1 -- 172.21.15.53:0/486765194 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a18031810 con 0x7f0a3013e360 2024-01-16T15:05:11.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.841+0000 7f0a2fe00640 1 -- 172.21.15.53:0/486765194 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a18031ba0 con 0x7f0a3013e360 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 -- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3013e360 msgr2=0x7f0a300cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 --2- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3013e360 0x7f0a300cf290 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0a180097a0 tx=0x7f0a1802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 -- 172.21.15.53:0/486765194 shutdown_connections 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 --2- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3013e360 0x7f0a300cf290 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 --2- 172.21.15.53:0/486765194 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013df80 0x7f0a3015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 -- 172.21.15.53:0/486765194 >> 172.21.15.53:0/486765194 conn(0x7f0a3006a800 msgr2=0x7f0a3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 -- 172.21.15.53:0/486765194 shutdown_connections 2024-01-16T15:05:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.842+0000 7f0a37800640 1 -- 172.21.15.53:0/486765194 wait complete. 2024-01-16T15:05:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.844+0000 7f0a37800640 1 Processor -- start 2024-01-16T15:05:11.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.844+0000 7f0a37800640 1 -- start start 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a37800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 0x7f0a3014dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a37800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013e360 0x7f0a300b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a37800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3015f9c0 0x7f0a3013bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a37800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3014bfb0 con 0x7f0a3013df80 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a37800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3014c290 con 0x7f0a3015f9c0 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a37800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a3014c570 con 0x7f0a3013e360 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a34e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 0x7f0a3014dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a34e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 0x7f0a3014dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47016/0 (socket says 172.21.15.53:47016) 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a34e00640 1 -- 172.21.15.53:0/3724782931 learned_addr learned my addr 172.21.15.53:0/3724782931 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.845+0000 7f0a35800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3015f9c0 0x7f0a3013bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a27c00640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013e360 0x7f0a300b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a34e00640 1 -- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013e360 msgr2=0x7f0a300b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a34e00640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013e360 0x7f0a300b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a34e00640 1 -- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3015f9c0 msgr2=0x7f0a3013bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:11.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a34e00640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3015f9c0 0x7f0a3013bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:11.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a34e00640 1 -- 172.21.15.53:0/3724782931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a20009b30 con 0x7f0a3013df80 2024-01-16T15:05:11.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a34e00640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 0x7f0a3014dda0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000efc0 tx=0x7f0a2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a2000ce60 con 0x7f0a3013df80 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a2000f040 con 0x7f0a3013df80 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a20014680 con 0x7f0a3013df80 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.846+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a18009400 con 0x7f0a3013df80 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.847+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a301e5ea0 con 0x7f0a3013df80 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.848+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a301e6060 con 0x7f0a3013df80 2024-01-16T15:05:11.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.849+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f0a200147e0 con 0x7f0a3013df80 2024-01-16T15:05:11.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.849+0000 7f0a2d600640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a0806b860 0x7f0a0806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:11.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.850+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f0a2008ad40 con 0x7f0a3013df80 2024-01-16T15:05:11.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.850+0000 7f0a27c00640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a0806b860 0x7f0a0806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:11.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.851+0000 7f0a27c00640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a0806b860 0x7f0a0806dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0a18036000 tx=0x7f0a1803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:11.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:11.854+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a200554f0 con 0x7f0a3013df80 2024-01-16T15:05:12.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.047+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0a300675d0 con 0x7f0a3013df80 2024-01-16T15:05:12.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.048+0000 7f0a2d600640 1 -- 172.21.15.53:0/3724782931 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v52) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0a20058b10 con 0x7f0a3013df80 2024-01-16T15:05:12.027 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a0806b860 msgr2=0x7f0a0806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a0806b860 0x7f0a0806dd20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f0a18036000 tx=0x7f0a1803c040 comp rx=0 tx=0).stop 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 msgr2=0x7f0a3014dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 0x7f0a3014dda0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f0a2000efc0 tx=0x7f0a2000c490 comp rx=0 tx=0).stop 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 shutdown_connections 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0a0806b860 0x7f0a0806dd20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a3015f9c0 0x7f0a3013bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a3013e360 0x7f0a300b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 --2- 172.21.15.53:0/3724782931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a3013df80 0x7f0a3014dda0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.051+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 >> 172.21.15.53:0/3724782931 conn(0x7f0a3006a800 msgr2=0x7f0a301549e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.052+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 shutdown_connections 2024-01-16T15:05:12.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.052+0000 7f0a37800640 1 -- 172.21.15.53:0/3724782931 wait complete. 2024-01-16T15:05:12.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:05:12.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.301+0000 7f4b37400640 1 Processor -- start 2024-01-16T15:05:12.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.302+0000 7f4b37400640 1 -- start start 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.303+0000 7f4b37400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4b3013e360 0x7f4b300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.303+0000 7f4b37400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3015c5f0 con 0x7f4b3015f9c0 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.303+0000 7f4b37400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b3006c470 con 0x7f4b3013df80 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.303+0000 7f4b37400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b3006c750 con 0x7f4b3013e360 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.304+0000 7f4b2fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4b3013e360 0x7f4b300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.304+0000 7f4b35400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4b3015f9c0 0x7f4b300cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36636/0 (socket says 172.21.15.53:36636) 2024-01-16T15:05:12.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.304+0000 7f4b35400640 1 -- 172.21.15.53:0/2872379848 learned_addr learned my addr 172.21.15.53:0/2872379848 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:12.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.304+0000 7f4b2fe00640 1 -- 172.21.15.53:0/2872379848 >> v1:172.21.15.146:6789/0 conn(0x7f4b3013df80 legacy=0x7f4b3015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:12.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.304+0000 7f4b2fe00640 1 -- 172.21.15.53:0/2872379848 >> v1:172.21.15.53:6789/0 conn(0x7f4b3015f9c0 legacy=0x7f4b300cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.304+0000 7f4b2fe00640 1 -- 172.21.15.53:0/2872379848 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3006ca30 con 0x7f4b3013e360 2024-01-16T15:05:12.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.305+0000 7f4b2fe00640 1 --2- 172.21.15.53:0/2872379848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3013e360 0x7f4b300cf290 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4b180097a0 tx=0x7f4b1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48936b046a99b6e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b2f400640 1 -- 172.21.15.53:0/2872379848 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18031830 con 0x7f4b3013e360 2024-01-16T15:05:12.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b2f400640 1 -- 172.21.15.53:0/2872379848 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b18031990 con 0x7f4b3013e360 2024-01-16T15:05:12.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b2f400640 1 -- 172.21.15.53:0/2872379848 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b18031d20 con 0x7f4b3013e360 2024-01-16T15:05:12.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b37400640 1 -- 172.21.15.53:0/2872379848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3013e360 msgr2=0x7f4b300cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b37400640 1 --2- 172.21.15.53:0/2872379848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3013e360 0x7f4b300cf290 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f4b180097a0 tx=0x7f4b1802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b37400640 1 -- 172.21.15.53:0/2872379848 shutdown_connections 2024-01-16T15:05:12.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b37400640 1 --2- 172.21.15.53:0/2872379848 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3013e360 0x7f4b300cf290 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.306+0000 7f4b37400640 1 -- 172.21.15.53:0/2872379848 >> 172.21.15.53:0/2872379848 conn(0x7f4b3006a800 msgr2=0x7f4b3006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:12.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.307+0000 7f4b37400640 1 -- 172.21.15.53:0/2872379848 shutdown_connections 2024-01-16T15:05:12.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.307+0000 7f4b37400640 1 -- 172.21.15.53:0/2872379848 wait complete. 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.308+0000 7f4b37400640 1 Processor -- start 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.309+0000 7f4b37400640 1 -- start start 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b37400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b3013df80 0x7f4b3013d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b37400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b3013e360 0x7f4b3014ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b37400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 0x7f4b301505d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b37400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3014f1d0 con 0x7f4b3015f9c0 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b37400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b3014e800 con 0x7f4b3013e360 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b37400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b301f16e0 con 0x7f4b3013df80 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 0x7f4b301505d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 0x7f4b301505d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43680/0 (socket says 172.21.15.53:43680) 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 -- 172.21.15.53:0/1477416957 learned_addr learned my addr 172.21.15.53:0/1477416957 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b2fe00640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b3013e360 0x7f4b3014ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 -- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b3013df80 msgr2=0x7f4b3013d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b34a00640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b3013df80 0x7f4b3013d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b3013df80 0x7f4b3013d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 -- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b3013e360 msgr2=0x7f4b3014ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b3013e360 0x7f4b3014ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b35400640 1 -- 172.21.15.53:0/1477416957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18009400 con 0x7f4b3015f9c0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.310+0000 7f4b2fe00640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b3013e360 0x7f4b3014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.311+0000 7f4b35400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 0x7f4b301505d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4b28002230 tx=0x7f4b2800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.311+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b2800ac30 con 0x7f4b3015f9c0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.311+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b2800ad90 con 0x7f4b3015f9c0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.311+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b28012600 con 0x7f4b3015f9c0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.311+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b3014e1b0 con 0x7f4b3015f9c0 2024-01-16T15:05:12.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.311+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b30150b10 con 0x7f4b3015f9c0 2024-01-16T15:05:12.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.312+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b3014e970 con 0x7f4b3015f9c0 2024-01-16T15:05:12.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.313+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4b28007070 con 0x7f4b3015f9c0 2024-01-16T15:05:12.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.314+0000 7f4b2cc00640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b0806b860 0x7f4b0806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:12.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.314+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f4b2808a0b0 con 0x7f4b3015f9c0 2024-01-16T15:05:12.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.315+0000 7f4b34a00640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b0806b860 0x7f4b0806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:12.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.316+0000 7f4b34a00640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b0806b860 0x7f4b0806dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000a750 tx=0x7f4b20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:12.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.319+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b28054ab0 con 0x7f4b3015f9c0 2024-01-16T15:05:12.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:12.589+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7f4b300675d0 con 0x7f4b3015f9c0 2024-01-16T15:05:13.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.135+0000 7f4b2cc00640 1 -- 172.21.15.53:0/1477416957 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v263) v1 ==== 157+0+0 (secure 0 0 0) 0x7f4b280170e0 con 0x7f4b3015f9c0 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b0806b860 msgr2=0x7f4b0806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b0806b860 0x7f4b0806dd20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4b2000a750 tx=0x7f4b20008040 comp rx=0 tx=0).stop 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 msgr2=0x7f4b301505d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 0x7f4b301505d0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f4b28002230 tx=0x7f4b2800cf70 comp rx=0 tx=0).stop 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 shutdown_connections 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b0806b860 0x7f4b0806dd20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b3015f9c0 0x7f4b301505d0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b3013e360 0x7f4b3014ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 --2- 172.21.15.53:0/1477416957 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b3013df80 0x7f4b3013d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 >> 172.21.15.53:0/1477416957 conn(0x7f4b3006a800 msgr2=0x7f4b3015d410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 shutdown_connections 2024-01-16T15:05:13.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.139+0000 7f4b37400640 1 -- 172.21.15.53:0/1477416957 wait complete. 2024-01-16T15:05:13.138 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:05:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.382+0000 7f3ec9200640 1 Processor -- start 2024-01-16T15:05:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.383+0000 7f3ec9200640 1 -- start start 2024-01-16T15:05:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.384+0000 7f3ec9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ec4160920 0x7f3ec40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.384+0000 7f3ec9200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ec40cf9f0 0x7f3ec40cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.384+0000 7f3ec9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec40d0a10 con 0x7f3ec4161010 2024-01-16T15:05:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.384+0000 7f3ec9200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40d03e0 con 0x7f3ec4160920 2024-01-16T15:05:13.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.384+0000 7f3ec9200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40d1f50 con 0x7f3ec40cf9f0 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.384+0000 7f3ec3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ec4160920 0x7f3ec40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ec4160920 0x7f3ec40b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43682/0 (socket says 172.21.15.53:43682) 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec3400640 1 -- 172.21.15.53:0/2823813094 learned_addr learned my addr 172.21.15.53:0/2823813094 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec2a00640 1 --2- 172.21.15.53:0/2823813094 >> v2:172.21.15.146:3300/0 conn(0x7f3ec40cf9f0 0x7f3ec40cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec3400640 1 -- 172.21.15.53:0/2823813094 >> v2:172.21.15.146:3300/0 conn(0x7f3ec40cf9f0 msgr2=0x7f3ec40cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec3400640 1 --2- 172.21.15.53:0/2823813094 >> v2:172.21.15.146:3300/0 conn(0x7f3ec40cf9f0 0x7f3ec40cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec3400640 1 -- 172.21.15.53:0/2823813094 >> v1:172.21.15.146:6789/0 conn(0x7f3ec4161010 legacy=0x7f3ec4161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec3400640 1 -- 172.21.15.53:0/2823813094 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec40cf080 con 0x7f3ec4160920 2024-01-16T15:05:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.385+0000 7f3ec2a00640 1 --2- 172.21.15.53:0/2823813094 >> v2:172.21.15.146:3300/0 conn(0x7f3ec40cf9f0 0x7f3ec40cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.387+0000 7f3ec3400640 1 --2- 172.21.15.53:0/2823813094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec40b3f30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3ec408c1f0 tx=0x7f3eb402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e0867e37bd07f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.387+0000 7f3ec2000640 1 -- 172.21.15.53:0/2823813094 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4031870 con 0x7f3ec4160920 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.387+0000 7f3ec2000640 1 -- 172.21.15.53:0/2823813094 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb40319d0 con 0x7f3ec4160920 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.387+0000 7f3ec2000640 1 -- 172.21.15.53:0/2823813094 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4031d60 con 0x7f3ec4160920 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 -- 172.21.15.53:0/2823813094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 msgr2=0x7f3ec40b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 --2- 172.21.15.53:0/2823813094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec40b3f30 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f3ec408c1f0 tx=0x7f3eb402b910 comp rx=0 tx=0).stop 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 -- 172.21.15.53:0/2823813094 shutdown_connections 2024-01-16T15:05:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 --2- 172.21.15.53:0/2823813094 >> v2:172.21.15.146:3300/0 conn(0x7f3ec40cf9f0 0x7f3ec40cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 --2- 172.21.15.53:0/2823813094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec40b3f30 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 -- 172.21.15.53:0/2823813094 >> 172.21.15.53:0/2823813094 conn(0x7f3ec40d3090 msgr2=0x7f3ec40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 -- 172.21.15.53:0/2823813094 shutdown_connections 2024-01-16T15:05:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.388+0000 7f3ec9200640 1 -- 172.21.15.53:0/2823813094 wait complete. 2024-01-16T15:05:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.390+0000 7f3ec9200640 1 Processor -- start 2024-01-16T15:05:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.391+0000 7f3ec9200640 1 -- start start 2024-01-16T15:05:13.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ec40cf9f0 0x7f3ec40b5e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec41484f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ec4161010 0x7f3ec4158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec4148a30 con 0x7f3ec4160920 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40b8d80 con 0x7f3ec40cf9f0 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec4157c60 con 0x7f3ec4161010 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec41484f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec41484f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43690/0 (socket says 172.21.15.53:43690) 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.392+0000 7f3ec2a00640 1 -- 172.21.15.53:0/3100123900 learned_addr learned my addr 172.21.15.53:0/3100123900 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec3400640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ec40cf9f0 0x7f3ec40b5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec3e00640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ec4161010 0x7f3ec4158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec2a00640 1 -- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ec4161010 msgr2=0x7f3ec4158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec2a00640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ec4161010 0x7f3ec4158740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec2a00640 1 -- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ec40cf9f0 msgr2=0x7f3ec40b5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec2a00640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ec40cf9f0 0x7f3ec40b5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec2a00640 1 -- 172.21.15.53:0/3100123900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40094d0 con 0x7f3ec4160920 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.393+0000 7f3ec2a00640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec41484f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00ce00 tx=0x7f3eac00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.394+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00f890 con 0x7f3ec4160920 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.394+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac00f9f0 con 0x7f3ec4160920 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.394+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00fdc0 con 0x7f3ec4160920 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.394+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec4157f40 con 0x7f3ec4160920 2024-01-16T15:05:13.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.394+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec4155a60 con 0x7f3ec4160920 2024-01-16T15:05:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.395+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec4067450 con 0x7f3ec4160920 2024-01-16T15:05:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.396+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f3eac00b040 con 0x7f3ec4160920 2024-01-16T15:05:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.397+0000 7f3ebb400640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e9c06b890 0x7f3e9c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:13.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.397+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f3eac08ade0 con 0x7f3ec4160920 2024-01-16T15:05:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.398+0000 7f3ec3400640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e9c06b890 0x7f3e9c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:13.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.399+0000 7f3ec3400640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e9c06b890 0x7f3e9c06dd50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4036000 tx=0x7f3eb403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:13.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.402+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3eac090020 con 0x7f3ec4160920 2024-01-16T15:05:13.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.597+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3ec408c7e0 con 0x7f3ec4160920 2024-01-16T15:05:13.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.602+0000 7f3ebb400640 1 -- 172.21.15.53:0/3100123900 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3eac01d070 con 0x7f3ec4160920 2024-01-16T15:05:13.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e9c06b890 msgr2=0x7f3e9c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e9c06b890 0x7f3e9c06dd50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4036000 tx=0x7f3eb403c040 comp rx=0 tx=0).stop 2024-01-16T15:05:13.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 msgr2=0x7f3ec41484f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec41484f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00ce00 tx=0x7f3eac00a650 comp rx=0 tx=0).stop 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 shutdown_connections 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3e9c06b890 0x7f3e9c06dd50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ec4161010 0x7f3ec4158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ec4160920 0x7f3ec41484f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 --2- 172.21.15.53:0/3100123900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ec40cf9f0 0x7f3ec40b5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 >> 172.21.15.53:0/3100123900 conn(0x7f3ec40d3090 msgr2=0x7f3ec40d3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 shutdown_connections 2024-01-16T15:05:13.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:13.605+0000 7f3ec9200640 1 -- 172.21.15.53:0/3100123900 wait complete. 2024-01-16T15:05:13.603 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:05:13.603 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:05:13.724 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:05:13.842 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:05:13.842 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:05:13.842 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:13.901 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:05:13.901 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:05:13.902 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:13.957 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:05:13.958 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:05:13.958 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:14.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:05:14.019 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:05:14.019 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:14.078 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:05:14.078 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:05:14.078 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:14.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.334+0000 7faa5b400640 1 Processor -- start 2024-01-16T15:05:14.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.335+0000 7faa5b400640 1 -- start start 2024-01-16T15:05:14.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faa54160920 0x7faa540b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa540cf9f0 0x7faa540cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa54161010 0x7faa541614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5b400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa540d03b0 con 0x7faa54160920 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5b400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa540d0a10 con 0x7faa54161010 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa540cf080 con 0x7faa540cf9f0 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.336+0000 7faa5a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faa54160920 0x7faa540b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faa54160920 0x7faa540b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43696/0 (socket says 172.21.15.53:43696) 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 -- 172.21.15.53:0/2411249083 learned_addr learned my addr 172.21.15.53:0/2411249083 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5aa00640 1 --2- 172.21.15.53:0/2411249083 >> v2:172.21.15.146:3300/0 conn(0x7faa54161010 0x7faa541614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa59600640 1 --2- 172.21.15.53:0/2411249083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa540cf9f0 0x7faa540cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 -- 172.21.15.53:0/2411249083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa540cf9f0 msgr2=0x7faa540cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 --2- 172.21.15.53:0/2411249083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa540cf9f0 0x7faa540cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 -- 172.21.15.53:0/2411249083 >> v2:172.21.15.146:3300/0 conn(0x7faa54161010 msgr2=0x7faa541614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 --2- 172.21.15.53:0/2411249083 >> v2:172.21.15.146:3300/0 conn(0x7faa54161010 0x7faa541614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5a000640 1 -- 172.21.15.53:0/2411249083 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa540c6be0 con 0x7faa54160920 2024-01-16T15:05:14.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.337+0000 7faa5aa00640 1 --2- 172.21.15.53:0/2411249083 >> v2:172.21.15.146:3300/0 conn(0x7faa54161010 0x7faa541614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:14.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.338+0000 7faa5a000640 1 --2- 172.21.15.53:0/2411249083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa54160920 0x7faa540b3f30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7faa540c6930 tx=0x7faa4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb1407e6d3d84adc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.339+0000 7faa58c00640 1 -- 172.21.15.53:0/2411249083 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4402ca70 con 0x7faa54160920 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.339+0000 7faa58c00640 1 -- 172.21.15.53:0/2411249083 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4402cbd0 con 0x7faa54160920 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.339+0000 7faa58c00640 1 -- 172.21.15.53:0/2411249083 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa440343f0 con 0x7faa54160920 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.339+0000 7faa5b400640 1 -- 172.21.15.53:0/2411249083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa54160920 msgr2=0x7faa540b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.339+0000 7faa5b400640 1 --2- 172.21.15.53:0/2411249083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa54160920 0x7faa540b3f30 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7faa540c6930 tx=0x7faa4402bed0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 -- 172.21.15.53:0/2411249083 shutdown_connections 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 --2- 172.21.15.53:0/2411249083 >> v2:172.21.15.146:3300/0 conn(0x7faa54161010 0x7faa541614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 --2- 172.21.15.53:0/2411249083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa540cf9f0 0x7faa540cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 --2- 172.21.15.53:0/2411249083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa54160920 0x7faa540b3f30 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 -- 172.21.15.53:0/2411249083 >> 172.21.15.53:0/2411249083 conn(0x7faa540d3090 msgr2=0x7faa540d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 -- 172.21.15.53:0/2411249083 shutdown_connections 2024-01-16T15:05:14.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.340+0000 7faa5b400640 1 -- 172.21.15.53:0/2411249083 wait complete. 2024-01-16T15:05:14.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.342+0000 7faa5b400640 1 Processor -- start 2024-01-16T15:05:14.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.342+0000 7faa5b400640 1 -- start start 2024-01-16T15:05:14.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 0x7faa540b86c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa54160920 0x7faa5414d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa54161010 0x7faa541f3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa54157b50 con 0x7faa540cf9f0 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa540cb6c0 con 0x7faa54160920 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa540cb9a0 con 0x7faa54161010 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 0x7faa540b86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 0x7faa540b86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43710/0 (socket says 172.21.15.53:43710) 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 -- 172.21.15.53:0/2545690584 learned_addr learned my addr 172.21.15.53:0/2545690584 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 -- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa54161010 msgr2=0x7faa541f3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa54161010 0x7faa541f3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 -- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa54160920 msgr2=0x7faa5414d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa54160920 0x7faa5414d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.343+0000 7faa5a000640 1 -- 172.21.15.53:0/2545690584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa44009b30 con 0x7faa540cf9f0 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.344+0000 7faa5a000640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 0x7faa540b86c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7faa44000f80 tx=0x7faa44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.344+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4403e450 con 0x7faa540cf9f0 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.344+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4403e5f0 con 0x7faa540cf9f0 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.344+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4403e9a0 con 0x7faa540cf9f0 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.344+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa540cbc80 con 0x7faa540cf9f0 2024-01-16T15:05:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.344+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa540cc240 con 0x7faa540cf9f0 2024-01-16T15:05:14.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.345+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa540ced80 con 0x7faa540cf9f0 2024-01-16T15:05:14.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.346+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7faa4403eb00 con 0x7faa540cf9f0 2024-01-16T15:05:14.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.347+0000 7faa52000640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa2c06b860 0x7faa2c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.347+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7faa440ad330 con 0x7faa540cf9f0 2024-01-16T15:05:14.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.348+0000 7faa59600640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa2c06b860 0x7faa2c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.349+0000 7faa59600640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa2c06b860 0x7faa2c06dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7faa5415fca0 tx=0x7faa3c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.351+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa540ced80 con 0x7faa540cf9f0 2024-01-16T15:05:14.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.583+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa540ced80 con 0x7faa540cf9f0 2024-01-16T15:05:14.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.584+0000 7faa52000640 1 -- 172.21.15.53:0/2545690584 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+3705 (secure 0 0 0) 0x7faa540ced80 con 0x7faa540cf9f0 2024-01-16T15:05:14.563 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:14.563 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":28,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3977954404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3977954404},{"type":"v1","addr":"172.21.15.146:6835","nonce":3977954404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3669971149","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":3669971149},{"type":"v1","addr":"172.21.15.146:6837","nonce":3669971149}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":5709,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/1838972240","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1838972240},{"type":"v1","addr":"172.21.15.53:6833","nonce":1838972240}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":20},{"gid":6213,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3852579477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3852579477},{"type":"v1","addr":"172.21.15.53:6835","nonce":3852579477}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":28}],"filesystems":[]} 2024-01-16T15:05:14.563 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 28 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa2c06b860 msgr2=0x7faa2c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa2c06b860 0x7faa2c06dd20 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7faa5415fca0 tx=0x7faa3c008040 comp rx=0 tx=0).stop 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 msgr2=0x7faa540b86c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 0x7faa540b86c0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7faa44000f80 tx=0x7faa44004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 shutdown_connections 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa2c06b860 0x7faa2c06dd20 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa54161010 0x7faa541f3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa54160920 0x7faa5414d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 --2- 172.21.15.53:0/2545690584 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa540cf9f0 0x7faa540b86c0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 >> 172.21.15.53:0/2545690584 conn(0x7faa540d3090 msgr2=0x7faa5406aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 shutdown_connections 2024-01-16T15:05:14.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.588+0000 7faa5b400640 1 -- 172.21.15.53:0/2545690584 wait complete. 2024-01-16T15:05:14.587 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:05:14.588 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:05:14.588 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:05:14.588 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:14.588 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:05:14.588 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:05:14.588 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:05:14.589 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:14.589 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:05:14.589 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:05:14.589 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:05:14.589 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:14.589 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:05:14.589 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:05:14.590 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:05:14.590 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:05:14.605 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:05:14.605 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:05:14.608 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:05:14.609 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:05:14.610 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:05:14.610 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:05:14.612 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:05:14.612 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:05:14.656 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:05:14.656 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:14.659 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:05:14.660 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:14.663 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:05:14.663 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:14.666 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:05:14.666 DEBUG:teuthology.parallel:result is None 2024-01-16T15:05:14.666 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:05:14.818 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:05:14.823 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:05:14.826 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:05:14.826 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:05:14.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.879+0000 7f2843800640 1 Processor -- start 2024-01-16T15:05:14.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.879+0000 7f2843800640 1 -- start start 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.880+0000 7f2843800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c13e360 0x7f283c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.880+0000 7f2843800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f283c15f9c0 0x7f283c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.880+0000 7f2843800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f283c15c5f0 con 0x7f283c13df80 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.880+0000 7f2843800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f283c06c470 con 0x7f283c15f9c0 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.880+0000 7f2843800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c06c750 con 0x7f283c13e360 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f283c15f9c0 0x7f283c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f283c15f9c0 0x7f283c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43738/0 (socket says 172.21.15.53:43738) 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 -- 172.21.15.53:0/2853587608 learned_addr learned my addr 172.21.15.53:0/2853587608 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2840e00640 1 --1- 172.21.15.53:0/2853587608 >> v1:172.21.15.53:6789/0 conn(0x7f283c13df80 0x7f283c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36660/0 (socket says 172.21.15.53:36660) 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f283be00640 1 --2- 172.21.15.53:0/2853587608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c13e360 0x7f283c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 -- 172.21.15.53:0/2853587608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c13e360 msgr2=0x7f283c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:14.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 --2- 172.21.15.53:0/2853587608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c13e360 0x7f283c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 -- 172.21.15.53:0/2853587608 >> v1:172.21.15.53:6789/0 conn(0x7f283c13df80 legacy=0x7f283c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.881+0000 7f2841800640 1 -- 172.21.15.53:0/2853587608 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f283c06ca30 con 0x7f283c15f9c0 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.882+0000 7f2841800640 1 --2- 172.21.15.53:0/2853587608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c15f9c0 0x7f283c0cf7d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2834009870 tx=0x7f283402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9654720eac693f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.882+0000 7f283b400640 1 -- 172.21.15.53:0/2853587608 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834030840 con 0x7f283c15f9c0 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f283b400640 1 -- 172.21.15.53:0/2853587608 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28340309a0 con 0x7f283c15f9c0 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f283b400640 1 -- 172.21.15.53:0/2853587608 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2834030d30 con 0x7f283c15f9c0 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 -- 172.21.15.53:0/2853587608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c15f9c0 msgr2=0x7f283c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 --2- 172.21.15.53:0/2853587608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c15f9c0 0x7f283c0cf7d0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f2834009870 tx=0x7f283402b980 comp rx=0 tx=0).stop 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 -- 172.21.15.53:0/2853587608 shutdown_connections 2024-01-16T15:05:14.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 --2- 172.21.15.53:0/2853587608 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c15f9c0 0x7f283c0cf7d0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 --2- 172.21.15.53:0/2853587608 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c13e360 0x7f283c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 -- 172.21.15.53:0/2853587608 >> 172.21.15.53:0/2853587608 conn(0x7f283c06a800 msgr2=0x7f283c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:14.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 -- 172.21.15.53:0/2853587608 shutdown_connections 2024-01-16T15:05:14.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.883+0000 7f2843800640 1 -- 172.21.15.53:0/2853587608 wait complete. 2024-01-16T15:05:14.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.885+0000 7f2843800640 1 Processor -- start 2024-01-16T15:05:14.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.885+0000 7f2843800640 1 -- start start 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2843800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c13df80 0x7f283c157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2843800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 0x7f283c158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2843800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c15f9c0 0x7f283c158650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2843800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c1577b0 con 0x7f283c13e360 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2843800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f283c158b90 con 0x7f283c13df80 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2843800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f283c156bc0 con 0x7f283c15f9c0 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f283be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 0x7f283c158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f283be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 0x7f283c158110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43746/0 (socket says 172.21.15.53:43746) 2024-01-16T15:05:14.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f283be00640 1 -- 172.21.15.53:0/3576326762 learned_addr learned my addr 172.21.15.53:0/3576326762 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.886+0000 7f2841800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c15f9c0 0x7f283c158650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f2840e00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c13df80 0x7f283c157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f283be00640 1 -- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c15f9c0 msgr2=0x7f283c158650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f283be00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c15f9c0 0x7f283c158650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f283be00640 1 -- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c13df80 msgr2=0x7f283c157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f283be00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c13df80 0x7f283c157bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f283be00640 1 -- 172.21.15.53:0/3576326762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28340094d0 con 0x7f283c13e360 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f2840e00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c13df80 0x7f283c157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f283be00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 0x7f283c158110 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f282400d800 tx=0x7f282400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282400b6c0 con 0x7f283c13e360 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f282400b820 con 0x7f283c13e360 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f282400bbb0 con 0x7f283c13e360 2024-01-16T15:05:14.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.887+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f283c156e10 con 0x7f283c13e360 2024-01-16T15:05:14.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.888+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f283c1573d0 con 0x7f283c13e360 2024-01-16T15:05:14.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.889+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f283c157a20 con 0x7f283c13e360 2024-01-16T15:05:14.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.890+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f282400bd10 con 0x7f283c13e360 2024-01-16T15:05:14.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.890+0000 7f2838c00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f281406b860 0x7f281406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:14.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.891+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 4943+0+0 (secure 0 0 0) 0x7f282408a770 con 0x7f283c13e360 2024-01-16T15:05:14.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.891+0000 7f2840e00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f281406b860 0x7f281406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:14.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.892+0000 7f2840e00640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f281406b860 0x7f281406dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f282c007fd0 tx=0x7f282c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:14.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:14.895+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28240550c0 con 0x7f283c13e360 2024-01-16T15:05:15.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.102+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f283c0675d0 con 0x7f283c13e360 2024-01-16T15:05:15.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.164+0000 7f2838c00640 1 -- 172.21.15.53:0/3576326762 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v54) v1 ==== 97+0+0 (secure 0 0 0) 0x7f28240586e0 con 0x7f283c13e360 2024-01-16T15:05:15.142 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f281406b860 msgr2=0x7f281406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f281406b860 0x7f281406dd20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f282c007fd0 tx=0x7f282c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 msgr2=0x7f283c158110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 0x7f283c158110 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f282400d800 tx=0x7f282400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 shutdown_connections 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f281406b860 0x7f281406dd20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f283c15f9c0 0x7f283c158650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f283c13e360 0x7f283c158110 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 --2- 172.21.15.53:0/3576326762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f283c13df80 0x7f283c157bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 >> 172.21.15.53:0/3576326762 conn(0x7f283c06a800 msgr2=0x7f283c15beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:15.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.167+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 shutdown_connections 2024-01-16T15:05:15.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.168+0000 7f2843800640 1 -- 172.21.15.53:0/3576326762 wait complete. 2024-01-16T15:05:15.167 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:05:15.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.398+0000 7fe83a000640 1 Processor -- start 2024-01-16T15:05:15.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.398+0000 7fe83a000640 1 -- start start 2024-01-16T15:05:15.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.400+0000 7fe83a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 0x7fe834167dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.400+0000 7fe83a000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe83406d480 con 0x7fe834071ce0 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.400+0000 7fe83a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8340b3490 con 0x7fe834146850 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.400+0000 7fe83a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340b3770 con 0x7fe8340adcd0 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.401+0000 7fe82be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe834071ce0 0x7fe8340ad5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36670/0 (socket says 172.21.15.53:36670) 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.401+0000 7fe82be00640 1 -- 172.21.15.53:0/186571254 learned_addr learned my addr 172.21.15.53:0/186571254 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.401+0000 7fe839600640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 0x7fe834167dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.401+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 661315154 0 0) 0x7fe83406d480 con 0x7fe834071ce0 2024-01-16T15:05:15.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.401+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe814003520 con 0x7fe834071ce0 2024-01-16T15:05:15.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384969860 0 0) 0x7fe814003520 con 0x7fe834071ce0 2024-01-16T15:05:15.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 msgr2=0x7fe834167dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 0x7fe834167dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 >> v1:172.21.15.146:6789/0 conn(0x7fe834146850 legacy=0x7fe8340715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8340b3a50 con 0x7fe834071ce0 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe81c001670 con 0x7fe834071ce0 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 >> v1:172.21.15.53:6789/0 conn(0x7fe834071ce0 legacy=0x7fe8340ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8140039a0 0x7fe814003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8140047c0 0x7fe814004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 0x7fe814005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8340b3770 con 0x7fe8140039a0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe81400ba60 con 0x7fe8140047c0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe81400bc20 con 0x7fe8340adcd0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe838c00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8140039a0 0x7fe814003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.402+0000 7fe82be00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8140047c0 0x7fe814004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.403+0000 7fe838c00640 1 -- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 msgr2=0x7fe814005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.403+0000 7fe838c00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8340adcd0 0x7fe814005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.403+0000 7fe838c00640 1 -- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8140047c0 msgr2=0x7fe814004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.403+0000 7fe838c00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8140047c0 0x7fe814004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.403+0000 7fe838c00640 1 -- 172.21.15.53:0/186571254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe814003520 con 0x7fe8140039a0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe838c00640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8140039a0 0x7fe814003dc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe824001720 tx=0x7fe8140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90abffefd5ece4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:15.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8240052d0 con 0x7fe8140039a0 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe82402e050 con 0x7fe8140039a0 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe833e00640 1 -- 172.21.15.53:0/186571254 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8240035f0 con 0x7fe8140039a0 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe83a000640 1 -- 172.21.15.53:0/186571254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8140039a0 msgr2=0x7fe814003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe83a000640 1 --2- 172.21.15.53:0/186571254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8140039a0 0x7fe814003dc0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe824001720 tx=0x7fe8140057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.404+0000 7fe839600640 1 -- 172.21.15.53:0/186571254 reap_dead start 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.405+0000 7fe83a000640 1 -- 172.21.15.53:0/186571254 shutdown_connections 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.405+0000 7fe83a000640 1 -- 172.21.15.53:0/186571254 >> 172.21.15.53:0/186571254 conn(0x7fe83406a3e0 msgr2=0x7fe83406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.405+0000 7fe83a000640 1 -- 172.21.15.53:0/186571254 shutdown_connections 2024-01-16T15:05:15.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.405+0000 7fe83a000640 1 -- 172.21.15.53:0/186571254 wait complete. 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.406+0000 7fe83a000640 1 Processor -- start 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.407+0000 7fe83a000640 1 -- start start 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.407+0000 7fe83a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8140039a0 0x7fe834142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.407+0000 7fe83a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 0x7fe8341332e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.407+0000 7fe83a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834133820 0x7fe834133ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe83a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8341341e0 con 0x7fe834142590 2024-01-16T15:05:15.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe83a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe834142a10 con 0x7fe834133820 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe83a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe834143840 con 0x7fe8140039a0 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 0x7fe8341332e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 0x7fe8341332e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43778/0 (socket says 172.21.15.53:43778) 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 -- 172.21.15.53:0/1959636548 learned_addr learned my addr 172.21.15.53:0/1959636548 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe838c00640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8140039a0 0x7fe834142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 -- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8140039a0 msgr2=0x7fe834142050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8140039a0 0x7fe834142050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 -- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834133820 msgr2=0x7fe834133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834133820 0x7fe834133ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.408+0000 7fe82be00640 1 -- 172.21.15.53:0/1959636548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe83415caf0 con 0x7fe834142590 2024-01-16T15:05:15.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.409+0000 7fe82be00640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 0x7fe8341332e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00e760 tx=0x7fe81c00eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:15.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.409+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c00c6c0 con 0x7fe834142590 2024-01-16T15:05:15.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.409+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe81c00c840 con 0x7fe834142590 2024-01-16T15:05:15.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.409+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81c00cbf0 con 0x7fe834142590 2024-01-16T15:05:15.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.409+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe834143cd0 con 0x7fe834142590 2024-01-16T15:05:15.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.409+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe834142f10 con 0x7fe834142590 2024-01-16T15:05:15.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.410+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8341439b0 con 0x7fe834142590 2024-01-16T15:05:15.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.411+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe81c00cd50 con 0x7fe834142590 2024-01-16T15:05:15.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.412+0000 7fe831600640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe80c06b890 0x7fe80c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.412+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe81c089ed0 con 0x7fe834142590 2024-01-16T15:05:15.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.413+0000 7fe838c00640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe80c06b890 0x7fe80c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.414+0000 7fe838c00640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe80c06b890 0x7fe80c06dd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe824033000 tx=0x7fe824009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:15.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.417+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe81c0547e0 con 0x7fe834142590 2024-01-16T15:05:15.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.676+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe8341439b0 con 0x7fe834142590 2024-01-16T15:05:15.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.677+0000 7fe831600640 1 -- 172.21.15.53:0/1959636548 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v10) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe81c057e00 con 0x7fe834142590 2024-01-16T15:05:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:15.655 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:05:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:05:15.655 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.656 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.657 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.658 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:05:15.659 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:05:15.660 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.661 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.662 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:05:15.663 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.664 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:05:15.665 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow r", 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.666 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:05:15.667 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.668 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:05:15.669 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.670 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:05:15.671 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:05:15.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe80c06b890 msgr2=0x7fe80c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe80c06b890 0x7fe80c06dd50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe824033000 tx=0x7fe824009030 comp rx=0 tx=0).stop 2024-01-16T15:05:15.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 msgr2=0x7fe8341332e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 0x7fe8341332e0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe81c00e760 tx=0x7fe81c00eb60 comp rx=0 tx=0).stop 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 shutdown_connections 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe80c06b890 0x7fe80c06dd50 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe834133820 0x7fe834133ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe834142590 0x7fe8341332e0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 --2- 172.21.15.53:0/1959636548 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8140039a0 0x7fe834142050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 >> 172.21.15.53:0/1959636548 conn(0x7fe83406a3e0 msgr2=0x7fe8340ad5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 shutdown_connections 2024-01-16T15:05:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.679+0000 7fe83a000640 1 -- 172.21.15.53:0/1959636548 wait complete. 2024-01-16T15:05:15.676 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:15.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.913+0000 7f2810600640 1 Processor -- start 2024-01-16T15:05:15.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.913+0000 7f2810600640 1 -- start start 2024-01-16T15:05:15.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f2810600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f280813df80 0x7f280815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f2810600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f2810600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f280815f9c0 0x7f28080cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f2810600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280806c470 con 0x7f280813df80 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f2810600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f280806c750 con 0x7f280815f9c0 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f2810600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f280806ca30 con 0x7f280813e360 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f280dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f280813df80 0x7f280815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f280dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f280813df80 0x7f280815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43790/0 (socket says 172.21.15.53:43790) 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.915+0000 7f280dc00640 1 -- 172.21.15.53:0/3547398332 learned_addr learned my addr 172.21.15.53:0/3547398332 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.916+0000 7f280dc00640 1 -- 172.21.15.53:0/3547398332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 msgr2=0x7f28080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.916+0000 7f280dc00640 1 --2- 172.21.15.53:0/3547398332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.916+0000 7f280dc00640 1 -- 172.21.15.53:0/3547398332 >> v2:172.21.15.146:3300/0 conn(0x7f280815f9c0 msgr2=0x7f28080cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.916+0000 7f280dc00640 1 --2- 172.21.15.53:0/3547398332 >> v2:172.21.15.146:3300/0 conn(0x7f280815f9c0 0x7f28080cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.916+0000 7f280dc00640 1 -- 172.21.15.53:0/3547398332 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280806cd10 con 0x7f280813df80 2024-01-16T15:05:15.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.917+0000 7f280dc00640 1 --2- 172.21.15.53:0/3547398332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f280815bee0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f280806e6d0 tx=0x7f27f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b36f9525f7ba3b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:15.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.917+0000 7f2807e00640 1 -- 172.21.15.53:0/3547398332 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8031870 con 0x7f280813df80 2024-01-16T15:05:15.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.917+0000 7f2807e00640 1 -- 172.21.15.53:0/3547398332 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f80319d0 con 0x7f280813df80 2024-01-16T15:05:15.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.917+0000 7f2807e00640 1 -- 172.21.15.53:0/3547398332 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8031d60 con 0x7f280813df80 2024-01-16T15:05:15.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 -- 172.21.15.53:0/3547398332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 msgr2=0x7f280815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 --2- 172.21.15.53:0/3547398332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f280815bee0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f280806e6d0 tx=0x7f27f802b910 comp rx=0 tx=0).stop 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 -- 172.21.15.53:0/3547398332 shutdown_connections 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 --2- 172.21.15.53:0/3547398332 >> v2:172.21.15.146:3300/0 conn(0x7f280815f9c0 0x7f28080cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 --2- 172.21.15.53:0/3547398332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28080cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 --2- 172.21.15.53:0/3547398332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f280815bee0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 -- 172.21.15.53:0/3547398332 >> 172.21.15.53:0/3547398332 conn(0x7f280806a800 msgr2=0x7f280806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 -- 172.21.15.53:0/3547398332 shutdown_connections 2024-01-16T15:05:15.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.918+0000 7f2810600640 1 -- 172.21.15.53:0/3547398332 wait complete. 2024-01-16T15:05:15.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.920+0000 7f2810600640 1 Processor -- start 2024-01-16T15:05:15.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.920+0000 7f2810600640 1 -- start start 2024-01-16T15:05:15.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f2810600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f2808154760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f2810600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28081f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f2810600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280815f9c0 0x7f280815a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f2810600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2808154ca0 con 0x7f280813df80 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f2810600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080706a0 con 0x7f280815f9c0 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f2810600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080b38d0 con 0x7f280813e360 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f280dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f2808154760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f280dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f2808154760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43792/0 (socket says 172.21.15.53:43792) 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.921+0000 7f280dc00640 1 -- 172.21.15.53:0/3254078318 learned_addr learned my addr 172.21.15.53:0/3254078318 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280e600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280815f9c0 0x7f280815a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280d200640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28081f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280dc00640 1 -- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 msgr2=0x7f28081f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280dc00640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28081f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280dc00640 1 -- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280815f9c0 msgr2=0x7f280815a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:15.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280dc00640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280815f9c0 0x7f280815a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280dc00640 1 -- 172.21.15.53:0/3254078318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f80094d0 con 0x7f280813df80 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280d200640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28081f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280e600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280815f9c0 0x7f280815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f280dc00640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f2808154760 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f27f802b950 tx=0x7f27f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f803ad10 con 0x7f280813df80 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f803ae70 con 0x7f280813df80 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27f8044610 con 0x7f280813df80 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.922+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28080b3bb0 con 0x7f280813df80 2024-01-16T15:05:15.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.923+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2808070f20 con 0x7f280813df80 2024-01-16T15:05:15.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.924+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28080675d0 con 0x7f280813df80 2024-01-16T15:05:15.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.925+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f27f8044770 con 0x7f280813df80 2024-01-16T15:05:15.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.926+0000 7f2805600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b860 0x7f27e006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:15.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.926+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f27f802d040 con 0x7f280813df80 2024-01-16T15:05:15.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.927+0000 7f280d200640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b860 0x7f27e006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:15.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.928+0000 7f280d200640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b860 0x7f27e006dd20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f27f0006fd0 tx=0x7f27f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:15.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:15.931+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f27f8077470 con 0x7f280813df80 2024-01-16T15:05:16.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.121+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f280815eb50 con 0x7f280813df80 2024-01-16T15:05:16.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.123+0000 7f2805600640 1 -- 172.21.15.53:0/3254078318 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+12821 (secure 0 0 0) 0x7f27f807aa90 con 0x7f280813df80 2024-01-16T15:05:16.102 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:16.102 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":56,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:15.177646+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":45,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b860 msgr2=0x7f27e006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b860 0x7f27e006dd20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f27f0006fd0 tx=0x7f27f0008040 comp rx=0 tx=0).stop 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 msgr2=0x7f2808154760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f2808154760 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f27f802b950 tx=0x7f27f80046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 shutdown_connections 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27e006b860 0x7f27e006dd20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f280815f9c0 0x7f280815a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f280813e360 0x7f28081f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 --2- 172.21.15.53:0/3254078318 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f280813df80 0x7f2808154760 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 >> 172.21.15.53:0/3254078318 conn(0x7f280806a800 msgr2=0x7f280813eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:16.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 shutdown_connections 2024-01-16T15:05:16.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.126+0000 7f2810600640 1 -- 172.21.15.53:0/3254078318 wait complete. 2024-01-16T15:05:16.124 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:05:16.124 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:05:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.371+0000 7f596da00640 1 Processor -- start 2024-01-16T15:05:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.371+0000 7f596da00640 1 -- start start 2024-01-16T15:05:16.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f596da00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f596815b030 0x7f59680b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f596da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59680b31b0 0x7f59680b4e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f596da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b5380 0x7f596815ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f596da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59680b3610 con 0x7f59680b31b0 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f596da00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59680d0780 con 0x7f596815b030 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f596da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59680cf8f0 con 0x7f59680b5380 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f5966000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59680b31b0 0x7f59680b4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f5966000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f59680b31b0 0x7f59680b4e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43796/0 (socket says 172.21.15.53:43796) 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f5966000640 1 -- 172.21.15.53:0/2117250412 learned_addr learned my addr 172.21.15.53:0/2117250412 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f5966a00640 1 --2- 172.21.15.53:0/2117250412 >> v2:172.21.15.146:3300/0 conn(0x7f596815b030 0x7f59680b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.373+0000 7f5967400640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b5380 0x7f596815ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5966000640 1 -- 172.21.15.53:0/2117250412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b5380 msgr2=0x7f596815ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5966000640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b5380 0x7f596815ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5966000640 1 -- 172.21.15.53:0/2117250412 >> v2:172.21.15.146:3300/0 conn(0x7f596815b030 msgr2=0x7f59680b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5966000640 1 --2- 172.21.15.53:0/2117250412 >> v2:172.21.15.146:3300/0 conn(0x7f596815b030 0x7f59680b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5966000640 1 -- 172.21.15.53:0/2117250412 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59680b5810 con 0x7f59680b31b0 2024-01-16T15:05:16.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5967400640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b5380 0x7f596815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:16.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.374+0000 7f5966a00640 1 --2- 172.21.15.53:0/2117250412 >> v2:172.21.15.146:3300/0 conn(0x7f596815b030 0x7f59680b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.375+0000 7f5966000640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b31b0 0x7f59680b4e40 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f59680b61d0 tx=0x7f595002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a4b4a7620de8661 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.375+0000 7f5965600640 1 -- 172.21.15.53:0/2117250412 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950031870 con 0x7f59680b31b0 2024-01-16T15:05:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.375+0000 7f5965600640 1 -- 172.21.15.53:0/2117250412 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59500319d0 con 0x7f59680b31b0 2024-01-16T15:05:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.375+0000 7f5965600640 1 -- 172.21.15.53:0/2117250412 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950031d60 con 0x7f59680b31b0 2024-01-16T15:05:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 -- 172.21.15.53:0/2117250412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b31b0 msgr2=0x7f59680b4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b31b0 0x7f59680b4e40 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f59680b61d0 tx=0x7f595002b910 comp rx=0 tx=0).stop 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 -- 172.21.15.53:0/2117250412 shutdown_connections 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b5380 0x7f596815ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 --2- 172.21.15.53:0/2117250412 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b31b0 0x7f59680b4e40 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 --2- 172.21.15.53:0/2117250412 >> v2:172.21.15.146:3300/0 conn(0x7f596815b030 0x7f59680b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 -- 172.21.15.53:0/2117250412 >> 172.21.15.53:0/2117250412 conn(0x7f596806a3e0 msgr2=0x7f596806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 -- 172.21.15.53:0/2117250412 shutdown_connections 2024-01-16T15:05:16.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.376+0000 7f596da00640 1 -- 172.21.15.53:0/2117250412 wait complete. 2024-01-16T15:05:16.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.378+0000 7f596da00640 1 Processor -- start 2024-01-16T15:05:16.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.378+0000 7f596da00640 1 -- start start 2024-01-16T15:05:16.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.379+0000 7f596da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b31b0 0x7f596806bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.379+0000 7f596da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 0x7f5968151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.379+0000 7f596da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815b030 0x7f596806c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.379+0000 7f596da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596806bb40 con 0x7f59680b5380 2024-01-16T15:05:16.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.379+0000 7f596da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f596806d080 con 0x7f596815b030 2024-01-16T15:05:16.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.379+0000 7f596da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f596806d360 con 0x7f59680b31b0 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 0x7f5968151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 0x7f5968151d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43808/0 (socket says 172.21.15.53:43808) 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 -- 172.21.15.53:0/2610515917 learned_addr learned my addr 172.21.15.53:0/2610515917 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966a00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b31b0 0x7f596806bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5967400640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815b030 0x7f596806c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 -- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b31b0 msgr2=0x7f596806bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b31b0 0x7f596806bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 -- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815b030 msgr2=0x7f596806c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815b030 0x7f596806c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5966000640 1 -- 172.21.15.53:0/2610515917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59500094d0 con 0x7f59680b5380 2024-01-16T15:05:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.380+0000 7f5967400640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815b030 0x7f596806c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:16.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.381+0000 7f5966000640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 0x7f5968151d00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f595002b950 tx=0x7f59500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.381+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950031860 con 0x7f59680b5380 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.381+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59500443f0 con 0x7f59680b5380 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.381+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59500447a0 con 0x7f59680b5380 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.381+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f596806b4f0 con 0x7f59680b5380 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.381+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5968152240 con 0x7f59680b5380 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.382+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5968070090 con 0x7f59680b5380 2024-01-16T15:05:16.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.383+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f595004d450 con 0x7f59680b5380 2024-01-16T15:05:16.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.384+0000 7f595ea00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f594006b860 0x7f594006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.384+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f595002d040 con 0x7f59680b5380 2024-01-16T15:05:16.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.385+0000 7f5966a00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f594006b860 0x7f594006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.386+0000 7f5966a00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f594006b860 0x7f594006dd20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f595800a8d0 tx=0x7f5958008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.388+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5950077410 con 0x7f59680b5380 2024-01-16T15:05:16.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.585+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59680675d0 con 0x7f59680b5380 2024-01-16T15:05:16.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.586+0000 7f595ea00640 1 -- 172.21.15.53:0/2610515917 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v56) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5950035020 con 0x7f59680b5380 2024-01-16T15:05:16.564 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:05:16.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f594006b860 msgr2=0x7f594006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f594006b860 0x7f594006dd20 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f595800a8d0 tx=0x7f5958008040 comp rx=0 tx=0).stop 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 msgr2=0x7f5968151d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 0x7f5968151d00 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f595002b950 tx=0x7f59500046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 shutdown_connections 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f594006b860 0x7f594006dd20 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f596815b030 0x7f596806c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f59680b5380 0x7f5968151d00 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 --2- 172.21.15.53:0/2610515917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f59680b31b0 0x7f596806bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 >> 172.21.15.53:0/2610515917 conn(0x7f596806a3e0 msgr2=0x7f59680b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 shutdown_connections 2024-01-16T15:05:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.589+0000 7f596da00640 1 -- 172.21.15.53:0/2610515917 wait complete. 2024-01-16T15:05:16.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:05:16.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.831+0000 7fa014e00640 1 Processor -- start 2024-01-16T15:05:16.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.831+0000 7fa014e00640 1 -- start start 2024-01-16T15:05:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.833+0000 7fa014e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0100cca50 0x7fa01015ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.833+0000 7fa014e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa01015f200 0x7fa0100b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.833+0000 7fa014e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0100cc5c0 con 0x7fa01015b260 2024-01-16T15:05:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.833+0000 7fa014e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa01015f690 con 0x7fa0100cca50 2024-01-16T15:05:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.833+0000 7fa014e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa01015b640 con 0x7fa01015f200 2024-01-16T15:05:16.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.833+0000 7fa006800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0100cca50 0x7fa01015ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa006800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0100cca50 0x7fa01015ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43810/0 (socket says 172.21.15.53:43810) 2024-01-16T15:05:16.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa006800640 1 -- 172.21.15.53:0/3930383518 learned_addr learned my addr 172.21.15.53:0/3930383518 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:16.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa00fe00640 1 --2- 172.21.15.53:0/3930383518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa01015f200 0x7fa0100b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa006800640 1 -- 172.21.15.53:0/3930383518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa01015f200 msgr2=0x7fa0100b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa006800640 1 --2- 172.21.15.53:0/3930383518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa01015f200 0x7fa0100b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa006800640 1 -- 172.21.15.53:0/3930383518 >> v1:172.21.15.146:6789/0 conn(0x7fa01015b260 legacy=0x7fa0100cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:16.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.834+0000 7fa006800640 1 -- 172.21.15.53:0/3930383518 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0100cfef0 con 0x7fa0100cca50 2024-01-16T15:05:16.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.835+0000 7fa006800640 1 --2- 172.21.15.53:0/3930383518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cca50 0x7fa01015ecc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80098a0 tx=0x7f9ff802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e111d49732af2b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.835+0000 7fa00ea00640 1 -- 172.21.15.53:0/3930383518 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8031640 con 0x7fa0100cca50 2024-01-16T15:05:16.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa00ea00640 1 -- 172.21.15.53:0/3930383518 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ff80317a0 con 0x7fa0100cca50 2024-01-16T15:05:16.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa00ea00640 1 -- 172.21.15.53:0/3930383518 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff8031b30 con 0x7fa0100cca50 2024-01-16T15:05:16.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 -- 172.21.15.53:0/3930383518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cca50 msgr2=0x7fa01015ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 --2- 172.21.15.53:0/3930383518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cca50 0x7fa01015ecc0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f9ff80098a0 tx=0x7f9ff802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 -- 172.21.15.53:0/3930383518 shutdown_connections 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 --2- 172.21.15.53:0/3930383518 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa01015f200 0x7fa0100b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 --2- 172.21.15.53:0/3930383518 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0100cca50 0x7fa01015ecc0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 -- 172.21.15.53:0/3930383518 >> 172.21.15.53:0/3930383518 conn(0x7fa0100d0e50 msgr2=0x7fa0100d1260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 -- 172.21.15.53:0/3930383518 shutdown_connections 2024-01-16T15:05:16.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.836+0000 7fa014e00640 1 -- 172.21.15.53:0/3930383518 wait complete. 2024-01-16T15:05:16.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.838+0000 7fa014e00640 1 Processor -- start 2024-01-16T15:05:16.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.838+0000 7fa014e00640 1 -- start start 2024-01-16T15:05:16.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa014e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0100cca50 0x7fa0100aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa014e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa01015b260 0x7fa01015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa014e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 0x7fa01015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa014e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b0170 con 0x7fa01015f200 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa014e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100b0590 con 0x7fa01015b260 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa014e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0100af4f0 con 0x7fa0100cca50 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa00fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 0x7fa01015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa00fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 0x7fa01015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43822/0 (socket says 172.21.15.53:43822) 2024-01-16T15:05:16.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.839+0000 7fa00fe00640 1 -- 172.21.15.53:0/3105297637 learned_addr learned my addr 172.21.15.53:0/3105297637 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa00fe00640 1 -- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0100cca50 msgr2=0x7fa0100aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa00fe00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0100cca50 0x7fa0100aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa00fe00640 1 -- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa01015b260 msgr2=0x7fa01015b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa00fe00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa01015b260 0x7fa01015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa00fe00640 1 -- 172.21.15.53:0/3105297637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff80094d0 con 0x7fa01015f200 2024-01-16T15:05:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa00fe00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 0x7fa01015bd00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa008007e90 tx=0x7fa00800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008014420 con 0x7fa01015f200 2024-01-16T15:05:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.840+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0080145a0 con 0x7fa01015f200 2024-01-16T15:05:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.841+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0100af7d0 con 0x7fa01015f200 2024-01-16T15:05:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.841+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008014950 con 0x7fa01015f200 2024-01-16T15:05:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.841+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0100afd90 con 0x7fa01015f200 2024-01-16T15:05:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.842+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa010067450 con 0x7fa01015f200 2024-01-16T15:05:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.843+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fa008014ab0 con 0x7fa01015f200 2024-01-16T15:05:16.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.844+0000 7fa007e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fe806b860 0x7f9fe806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:16.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.844+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fa00808ab90 con 0x7fa01015f200 2024-01-16T15:05:16.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.844+0000 7fa00f400640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fe806b860 0x7f9fe806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.846+0000 7fa00f400640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fe806b860 0x7f9fe806dd20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa000003cc0 tx=0x7fa0000079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:16.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:16.848+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa00808ce60 con 0x7fa01015f200 2024-01-16T15:05:17.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.046+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa01006ca80 con 0x7fa01015f200 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.047+0000 7fa007e00640 1 -- 172.21.15.53:0/3105297637 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v56) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa008055570 con 0x7fa01015f200 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:17.025 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fe806b860 msgr2=0x7f9fe806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fe806b860 0x7f9fe806dd20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa000003cc0 tx=0x7fa0000079b0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 msgr2=0x7fa01015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 0x7fa01015bd00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fa008007e90 tx=0x7fa00800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 shutdown_connections 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fe806b860 0x7f9fe806dd20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa01015f200 0x7fa01015bd00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa01015b260 0x7fa01015b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 --2- 172.21.15.53:0/3105297637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0100cca50 0x7fa0100aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 >> 172.21.15.53:0/3105297637 conn(0x7fa0100d0e50 msgr2=0x7fa01006e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 shutdown_connections 2024-01-16T15:05:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.049+0000 7fa014e00640 1 -- 172.21.15.53:0/3105297637 wait complete. 2024-01-16T15:05:17.046 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:05:17.047 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:05:17.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.297+0000 7fdf54e00640 1 Processor -- start 2024-01-16T15:05:17.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.297+0000 7fdf54e00640 1 -- start start 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.298+0000 7fdf54e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdf50143c40 0x7fdf50071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.298+0000 7fdf54e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 0x7fdf5006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.298+0000 7fdf54e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf50070e50 con 0x7fdf500717d0 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf54e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf500723c0 con 0x7fdf50143c40 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf54e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf50144020 con 0x7fdf500ccc10 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf47e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdf500717d0 0x7fdf500cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36686/0 (socket says 172.21.15.53:36686) 2024-01-16T15:05:17.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf47e00640 1 -- 172.21.15.53:0/4183827750 learned_addr learned my addr 172.21.15.53:0/4183827750 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:17.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf4f400640 1 --2- 172.21.15.53:0/4183827750 >> v2:172.21.15.146:3300/0 conn(0x7fdf50143c40 0x7fdf50071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf4fe00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 0x7fdf5006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.299+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320132326 0 0) 0x7fdf50070e50 con 0x7fdf500717d0 2024-01-16T15:05:17.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf30003520 con 0x7fdf500717d0 2024-01-16T15:05:17.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1894243063 0 0) 0x7fdf30003520 con 0x7fdf500717d0 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 msgr2=0x7fdf5006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 0x7fdf5006d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 >> v2:172.21.15.146:3300/0 conn(0x7fdf50143c40 msgr2=0x7fdf50071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 --2- 172.21.15.53:0/4183827750 >> v2:172.21.15.146:3300/0 conn(0x7fdf50143c40 0x7fdf50071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf50071e00 con 0x7fdf500717d0 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fdf38001670 con 0x7fdf500717d0 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 >> v1:172.21.15.53:6789/0 conn(0x7fdf500717d0 legacy=0x7fdf500cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30003970 0x7fdf30003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 0x7fdf300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf30004d00 0x7fdf30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf500723c0 con 0x7fdf30004d00 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf50144020 con 0x7fdf30003970 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3000b980 con 0x7fdf500ccc10 2024-01-16T15:05:17.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.300+0000 7fdf4fe00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf30004d00 0x7fdf30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.301+0000 7fdf4fe00640 1 -- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 msgr2=0x7fdf300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.301+0000 7fdf4fe00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf500ccc10 0x7fdf300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.301+0000 7fdf4fe00640 1 -- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30003970 msgr2=0x7fdf30003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:17.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.301+0000 7fdf4fe00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30003970 0x7fdf30003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.301+0000 7fdf4fe00640 1 -- 172.21.15.53:0/4183827750 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf30003520 con 0x7fdf30004d00 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.302+0000 7fdf4fe00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf30004d00 0x7fdf30005180 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009fc0 tx=0x7fdf300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1a1945da7d8215 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.302+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf4802cc50 con 0x7fdf30004d00 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.302+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf4800f040 con 0x7fdf30004d00 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.302+0000 7fdf4ea00640 1 -- 172.21.15.53:0/4183827750 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf48034440 con 0x7fdf30004d00 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf54e00640 1 -- 172.21.15.53:0/4183827750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf30004d00 msgr2=0x7fdf30005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf54e00640 1 --2- 172.21.15.53:0/4183827750 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf30004d00 0x7fdf30005180 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009fc0 tx=0x7fdf300057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf4fe00640 1 -- 172.21.15.53:0/4183827750 reap_dead start 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf54e00640 1 -- 172.21.15.53:0/4183827750 shutdown_connections 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf54e00640 1 -- 172.21.15.53:0/4183827750 >> 172.21.15.53:0/4183827750 conn(0x7fdf5006a800 msgr2=0x7fdf5006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf54e00640 1 -- 172.21.15.53:0/4183827750 shutdown_connections 2024-01-16T15:05:17.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.303+0000 7fdf54e00640 1 -- 172.21.15.53:0/4183827750 wait complete. 2024-01-16T15:05:17.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.305+0000 7fdf54e00640 1 Processor -- start 2024-01-16T15:05:17.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.305+0000 7fdf54e00640 1 -- start start 2024-01-16T15:05:17.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf54e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004d00 0x7fdf5015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf54e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 0x7fdf500b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf54e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf501f4370 0x7fdf500b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf54e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf501f72e0 con 0x7fdf501f3f10 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf54e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5013c710 con 0x7fdf30004d00 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf54e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5013c9f0 con 0x7fdf501f4370 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 0x7fdf500b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.306+0000 7fdf47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 0x7fdf500b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43840/0 (socket says 172.21.15.53:43840) 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 -- 172.21.15.53:0/3823497921 learned_addr learned my addr 172.21.15.53:0/3823497921 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf4f400640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004d00 0x7fdf5015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf4fe00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf501f4370 0x7fdf500b35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 -- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf501f4370 msgr2=0x7fdf500b35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf501f4370 0x7fdf500b35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 -- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004d00 msgr2=0x7fdf5015a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004d00 0x7fdf5015a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 -- 172.21.15.53:0/3823497921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5013ccd0 con 0x7fdf501f3f10 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf4f400640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004d00 0x7fdf5015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.307+0000 7fdf47e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 0x7fdf500b3080 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800c7c0 tx=0x7fdf3800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.308+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf380074e0 con 0x7fdf501f3f10 2024-01-16T15:05:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.308+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf38007640 con 0x7fdf501f3f10 2024-01-16T15:05:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.308+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf380049e0 con 0x7fdf501f3f10 2024-01-16T15:05:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.308+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5013cfb0 con 0x7fdf501f3f10 2024-01-16T15:05:17.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.308+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5013d4b0 con 0x7fdf501f3f10 2024-01-16T15:05:17.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.309+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf50067450 con 0x7fdf501f3f10 2024-01-16T15:05:17.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.310+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fdf38004b40 con 0x7fdf501f3f10 2024-01-16T15:05:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.311+0000 7fdf47400640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf2806b860 0x7fdf2806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.311+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fdf38089820 con 0x7fdf501f3f10 2024-01-16T15:05:17.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.312+0000 7fdf4f400640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf2806b860 0x7fdf2806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.313+0000 7fdf4f400640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf2806b860 0x7fdf2806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009a40 tx=0x7fdf40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:17.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.315+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf380541c0 con 0x7fdf501f3f10 2024-01-16T15:05:17.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:17.510+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fdf500ceb30 con 0x7fdf501f3f10 2024-01-16T15:05:18.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.184+0000 7fdf47400640 1 -- 172.21.15.53:0/3823497921 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v57) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdf380180b0 con 0x7fdf501f3f10 2024-01-16T15:05:18.163 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:05:18.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf2806b860 msgr2=0x7fdf2806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf2806b860 0x7fdf2806dd20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fdf40009a40 tx=0x7fdf40008040 comp rx=0 tx=0).stop 2024-01-16T15:05:18.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 msgr2=0x7fdf500b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 0x7fdf500b3080 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fdf3800c7c0 tx=0x7fdf3800cc80 comp rx=0 tx=0).stop 2024-01-16T15:05:18.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 shutdown_connections 2024-01-16T15:05:18.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf2806b860 0x7fdf2806dd20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf501f4370 0x7fdf500b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf501f3f10 0x7fdf500b3080 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 --2- 172.21.15.53:0/3823497921 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf30004d00 0x7fdf5015a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 >> 172.21.15.53:0/3823497921 conn(0x7fdf5006a800 msgr2=0x7fdf500adc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 shutdown_connections 2024-01-16T15:05:18.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.189+0000 7fdf54e00640 1 -- 172.21.15.53:0/3823497921 wait complete. 2024-01-16T15:05:18.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:05:18.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.424+0000 7fb3d9400640 1 Processor -- start 2024-01-16T15:05:18.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.425+0000 7fb3d9400640 1 -- start start 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3d4161890 0x7fb3d4070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 0x7fb3d41539b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d40714e0 con 0x7fb3d40cef70 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d9400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d40cd390 con 0x7fb3d4161890 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d406aea0 con 0x7fb3d40710c0 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 0x7fb3d41539b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 0x7fb3d41539b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49092/0 (socket says 172.21.15.53:49092) 2024-01-16T15:05:18.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d8a00640 1 -- 172.21.15.53:0/758914930 learned_addr learned my addr 172.21.15.53:0/758914930 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:18.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.427+0000 7fb3d3400640 1 --2- 172.21.15.53:0/758914930 >> v2:172.21.15.146:3300/0 conn(0x7fb3d4161890 0x7fb3d4070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.428+0000 7fb3d8a00640 1 -- 172.21.15.53:0/758914930 >> v2:172.21.15.146:3300/0 conn(0x7fb3d4161890 msgr2=0x7fb3d4070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:18.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.428+0000 7fb3d8a00640 1 --2- 172.21.15.53:0/758914930 >> v2:172.21.15.146:3300/0 conn(0x7fb3d4161890 0x7fb3d4070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.428+0000 7fb3d8a00640 1 -- 172.21.15.53:0/758914930 >> v1:172.21.15.146:6789/0 conn(0x7fb3d40cef70 legacy=0x7fb3d406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.428+0000 7fb3d8a00640 1 -- 172.21.15.53:0/758914930 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d406b180 con 0x7fb3d40710c0 2024-01-16T15:05:18.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.429+0000 7fb3d8a00640 1 --2- 172.21.15.53:0/758914930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 0x7fb3d41539b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009870 tx=0x7fb3cc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ccf62ea5391ecac server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.430+0000 7fb3d2a00640 1 -- 172.21.15.53:0/758914930 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc031640 con 0x7fb3d40710c0 2024-01-16T15:05:18.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.430+0000 7fb3d2a00640 1 -- 172.21.15.53:0/758914930 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc0317a0 con 0x7fb3d40710c0 2024-01-16T15:05:18.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.430+0000 7fb3d2a00640 1 -- 172.21.15.53:0/758914930 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc031b30 con 0x7fb3d40710c0 2024-01-16T15:05:18.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.430+0000 7fb3d9400640 1 -- 172.21.15.53:0/758914930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 msgr2=0x7fb3d41539b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.430+0000 7fb3d9400640 1 --2- 172.21.15.53:0/758914930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 0x7fb3d41539b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc009870 tx=0x7fb3cc02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.431+0000 7fb3d9400640 1 -- 172.21.15.53:0/758914930 shutdown_connections 2024-01-16T15:05:18.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.431+0000 7fb3d9400640 1 --2- 172.21.15.53:0/758914930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d40710c0 0x7fb3d41539b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.431+0000 7fb3d9400640 1 --2- 172.21.15.53:0/758914930 >> v2:172.21.15.146:3300/0 conn(0x7fb3d4161890 0x7fb3d4070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.431+0000 7fb3d9400640 1 -- 172.21.15.53:0/758914930 >> 172.21.15.53:0/758914930 conn(0x7fb3d40cc1d0 msgr2=0x7fb3d40cc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:18.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.431+0000 7fb3d9400640 1 -- 172.21.15.53:0/758914930 shutdown_connections 2024-01-16T15:05:18.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.431+0000 7fb3d9400640 1 -- 172.21.15.53:0/758914930 wait complete. 2024-01-16T15:05:18.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.432+0000 7fb3d9400640 1 Processor -- start 2024-01-16T15:05:18.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.433+0000 7fb3d9400640 1 -- start start 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.433+0000 7fb3d9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3d40710c0 0x7fb3d41565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 0x7fb3d41e6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d4161890 0x7fb3d4146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d41e7000 con 0x7fb3d40cef70 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d41e8120 con 0x7fb3d40710c0 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d41e8400 con 0x7fb3d4161890 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 0x7fb3d41e6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 0x7fb3d41e6ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43856/0 (socket says 172.21.15.53:43856) 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 -- 172.21.15.53:0/1958577744 learned_addr learned my addr 172.21.15.53:0/1958577744 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3e00640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3d40710c0 0x7fb3d41565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d8a00640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d4161890 0x7fb3d4146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 -- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d4161890 msgr2=0x7fb3d4146090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d4161890 0x7fb3d4146090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 -- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3d40710c0 msgr2=0x7fb3d41565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.434+0000 7fb3d3400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3d40710c0 0x7fb3d41565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3d3400640 1 -- 172.21.15.53:0/1958577744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3cc0094d0 con 0x7fb3d40cef70 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3d3400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 0x7fb3d41e6ac0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00efc0 tx=0x7fb3bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc009bd0 con 0x7fb3d40cef70 2024-01-16T15:05:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3bc00f040 con 0x7fb3d40cef70 2024-01-16T15:05:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d41e86e0 con 0x7fb3d40cef70 2024-01-16T15:05:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3bc0146a0 con 0x7fb3d40cef70 2024-01-16T15:05:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.435+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d41465d0 con 0x7fb3d40cef70 2024-01-16T15:05:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.437+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d4067b60 con 0x7fb3d40cef70 2024-01-16T15:05:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.438+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb3bc014800 con 0x7fb3d40cef70 2024-01-16T15:05:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.439+0000 7fb3cbe00640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3ac06b860 0x7fb3ac06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.439+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5227+0+0 (secure 0 0 0) 0x7fb3bc08b160 con 0x7fb3d40cef70 2024-01-16T15:05:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.439+0000 7fb3d3e00640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3ac06b860 0x7fb3ac06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:18.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.440+0000 7fb3d3e00640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3ac06b860 0x7fb3ac06dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4007fd0 tx=0x7fb3c4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:18.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.444+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3bc055930 con 0x7fb3d40cef70 2024-01-16T15:05:18.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:18.637+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fb3d40668e0 con 0x7fb3d40cef70 2024-01-16T15:05:19.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.199+0000 7fb3cbe00640 1 -- 172.21.15.53:0/1958577744 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v58) v1 ==== 170+0+0 (secure 0 0 0) 0x7fb3bc058f50 con 0x7fb3d40cef70 2024-01-16T15:05:19.178 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:05:19.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.203+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3ac06b860 msgr2=0x7fb3ac06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.203+0000 7fb3d9400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3ac06b860 0x7fb3ac06dd20 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb3c4007fd0 tx=0x7fb3c4009040 comp rx=0 tx=0).stop 2024-01-16T15:05:19.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.203+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 msgr2=0x7fb3d41e6ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.203+0000 7fb3d9400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 0x7fb3d41e6ac0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fb3bc00efc0 tx=0x7fb3bc00c490 comp rx=0 tx=0).stop 2024-01-16T15:05:19.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.204+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 shutdown_connections 2024-01-16T15:05:19.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.204+0000 7fb3d9400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3ac06b860 0x7fb3ac06dd20 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.204+0000 7fb3d9400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3d4161890 0x7fb3d4146090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.204+0000 7fb3d9400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3d40cef70 0x7fb3d41e6ac0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.204+0000 7fb3d9400640 1 --2- 172.21.15.53:0/1958577744 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3d40710c0 0x7fb3d41565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.204+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 >> 172.21.15.53:0/1958577744 conn(0x7fb3d40cc1d0 msgr2=0x7fb3d406a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:19.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.205+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 shutdown_connections 2024-01-16T15:05:19.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.205+0000 7fb3d9400640 1 -- 172.21.15.53:0/1958577744 wait complete. 2024-01-16T15:05:19.221 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:05:19.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.429+0000 7f455a400640 1 Processor -- start 2024-01-16T15:05:19.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.429+0000 7f455a400640 1 -- start start 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.430+0000 7f455a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 0x7f4554166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f455a400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455415c5f0 con 0x7f455413df80 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f455a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f455406c470 con 0x7f455413e360 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f455a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455406c750 con 0x7f45540cf9a0 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f4553400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f455413df80 0x7f455415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36698/0 (socket says 172.21.15.53:36698) 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f4553400640 1 -- 172.21.15.53:0/3075242406 learned_addr learned my addr 172.21.15.53:0/3075242406 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f4553e00640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 0x7f4554166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166759974 0 0) 0x7f455415c5f0 con 0x7f455413df80 2024-01-16T15:05:19.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.431+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534003520 con 0x7f455413df80 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1640527537 0 0) 0x7f4534003520 con 0x7f455413df80 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 msgr2=0x7f4554166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 0x7f4554166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 >> v1:172.21.15.146:6789/0 conn(0x7f455413e360 legacy=0x7f45540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455406ca30 con 0x7f455413df80 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4544001670 con 0x7f455413df80 2024-01-16T15:05:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 >> v1:172.21.15.53:6789/0 conn(0x7f455413df80 legacy=0x7f455415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 0x7f45340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4534003f20 0x7f45340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4534004d00 0x7f4534005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455406c750 con 0x7f4534004d00 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f453400ba60 con 0x7f4534003f20 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f453400bc20 con 0x7f45540cf9a0 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.432+0000 7f4553e00640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4534004d00 0x7f4534005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4553400640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 0x7f45340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4552a00640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4534003f20 0x7f45340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4553e00640 1 -- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 msgr2=0x7f45340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4553e00640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45540cf9a0 0x7f45340039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4553e00640 1 -- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4534003f20 msgr2=0x7f45340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4553e00640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4534003f20 0x7f45340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.433+0000 7f4553e00640 1 -- 172.21.15.53:0/3075242406 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534003520 con 0x7f4534004d00 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.434+0000 7f4553e00640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4534004d00 0x7f4534005180 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f454c009f90 tx=0x7f45340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f33f08a791cc5f8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.434+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c02cbd0 con 0x7f4534004d00 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454c00f040 con 0x7f4534004d00 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f4552000640 1 -- 172.21.15.53:0/3075242406 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c034440 con 0x7f4534004d00 2024-01-16T15:05:19.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f455a400640 1 -- 172.21.15.53:0/3075242406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4534004d00 msgr2=0x7f4534005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f455a400640 1 --2- 172.21.15.53:0/3075242406 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4534004d00 0x7f4534005180 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f454c009f90 tx=0x7f45340057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f4553e00640 1 -- 172.21.15.53:0/3075242406 reap_dead start 2024-01-16T15:05:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f455a400640 1 -- 172.21.15.53:0/3075242406 shutdown_connections 2024-01-16T15:05:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f455a400640 1 -- 172.21.15.53:0/3075242406 >> 172.21.15.53:0/3075242406 conn(0x7f455406a800 msgr2=0x7f455406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f455a400640 1 -- 172.21.15.53:0/3075242406 shutdown_connections 2024-01-16T15:05:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.435+0000 7f455a400640 1 -- 172.21.15.53:0/3075242406 wait complete. 2024-01-16T15:05:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.437+0000 7f455a400640 1 Processor -- start 2024-01-16T15:05:19.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.437+0000 7f455a400640 1 -- start start 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.438+0000 7f455a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f455413d730 0x7f4554142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.438+0000 7f455a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f455413db10 0x7f45540b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.438+0000 7f455a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 0x7f45540b3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.438+0000 7f455a400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554140bb0 con 0x7f4554143180 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.438+0000 7f455a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4554143700 con 0x7f455413db10 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.438+0000 7f455a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45541439e0 con 0x7f455413d730 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 0x7f45540b3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 0x7f45540b3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43864/0 (socket says 172.21.15.53:43864) 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 -- 172.21.15.53:0/3702738408 learned_addr learned my addr 172.21.15.53:0/3702738408 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f455413d730 0x7f4554142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4552a00640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f455413db10 0x7f45540b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 -- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f455413d730 msgr2=0x7f4554142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f455413d730 0x7f4554142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 -- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f455413db10 msgr2=0x7f45540b3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f455413db10 0x7f45540b3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.439+0000 7f4553e00640 1 -- 172.21.15.53:0/3702738408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4554140dc0 con 0x7f4554143180 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.440+0000 7f4553e00640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 0x7f45540b3d70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f454c037040 tx=0x7f454c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.440+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c034d50 con 0x7f4554143180 2024-01-16T15:05:19.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.440+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454c02de60 con 0x7f4554143180 2024-01-16T15:05:19.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.440+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c003bb0 con 0x7f4554143180 2024-01-16T15:05:19.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.440+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45541410a0 con 0x7f4554143180 2024-01-16T15:05:19.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.440+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45541e4b00 con 0x7f4554143180 2024-01-16T15:05:19.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.441+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4554143ba0 con 0x7f4554143180 2024-01-16T15:05:19.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.444+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f454c003d10 con 0x7f4554143180 2024-01-16T15:05:19.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.444+0000 7f454b400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452c06b860 0x7f452c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:19.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.445+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f454c0abf00 con 0x7f4554143180 2024-01-16T15:05:19.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.445+0000 7f4553400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452c06b860 0x7f452c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:19.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.446+0000 7f4553400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452c06b860 0x7f452c06dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4554154a80 tx=0x7f4544007040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:19.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.448+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f454c0767a0 con 0x7f4554143180 2024-01-16T15:05:19.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:19.686+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f4554141290 con 0x7f4554143180 2024-01-16T15:05:20.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.203+0000 7f454b400640 1 -- 172.21.15.53:0/3702738408 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v31) v1 ==== 172+0+0 (secure 0 0 0) 0x7f454c0450c0 con 0x7f4554143180 2024-01-16T15:05:20.182 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 8 and data pool 9 2024-01-16T15:05:20.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452c06b860 msgr2=0x7f452c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452c06b860 0x7f452c06dd20 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f4554154a80 tx=0x7f4544007040 comp rx=0 tx=0).stop 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 msgr2=0x7f45540b3d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 0x7f45540b3d70 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f454c037040 tx=0x7f454c009750 comp rx=0 tx=0).stop 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 shutdown_connections 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f452c06b860 0x7f452c06dd20 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4554143180 0x7f45540b3d70 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f455413db10 0x7f45540b3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 --2- 172.21.15.53:0/3702738408 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f455413d730 0x7f4554142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 >> 172.21.15.53:0/3702738408 conn(0x7f455406a800 msgr2=0x7f455415aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 shutdown_connections 2024-01-16T15:05:20.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.208+0000 7f455a400640 1 -- 172.21.15.53:0/3702738408 wait complete. 2024-01-16T15:05:20.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:20.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.426+0000 7f4aa0a00640 1 Processor -- start 2024-01-16T15:05:20.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.426+0000 7f4aa0a00640 1 -- start start 2024-01-16T15:05:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.427+0000 7f4aa0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 0x7f4a9815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.427+0000 7f4aa0a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9806c470 con 0x7f4a980cf9a0 2024-01-16T15:05:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.427+0000 7f4aa0a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a9806c750 con 0x7f4a9813e360 2024-01-16T15:05:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.427+0000 7f4aa0a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9806ca30 con 0x7f4a9813df80 2024-01-16T15:05:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.428+0000 7f4a9ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4a980cf9a0 0x7f4a980cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36706/0 (socket says 172.21.15.53:36706) 2024-01-16T15:05:20.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.428+0000 7f4a9ea00640 1 -- 172.21.15.53:0/3232662394 learned_addr learned my addr 172.21.15.53:0/3232662394 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.428+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539991847 0 0) 0x7f4a9806c470 con 0x7f4a980cf9a0 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.428+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a78003520 con 0x7f4a980cf9a0 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.428+0000 7f4a9e000640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 0x7f4a9815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038609302 0 0) 0x7f4a78003520 con 0x7f4a980cf9a0 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 msgr2=0x7f4a9815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 0x7f4a9815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 >> v1:172.21.15.146:6789/0 conn(0x7f4a9813e360 legacy=0x7f4a980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9806cd10 con 0x7f4a980cf9a0 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4a900019f0 con 0x7f4a980cf9a0 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 >> v1:172.21.15.53:6789/0 conn(0x7f4a980cf9a0 legacy=0x7f4a980cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a78003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a780047c0 0x7f4a78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 0x7f4a780051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9806ca30 con 0x7f4a780039a0 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7800bab0 con 0x7f4a780047c0 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a7800bc70 con 0x7f4a9813df80 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9e000640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a78003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9d600640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a780047c0 0x7f4a78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.429+0000 7f4a9ea00640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 0x7f4a780051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.430+0000 7f4a9e000640 1 -- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 msgr2=0x7f4a780051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.430+0000 7f4a9e000640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a9813df80 0x7f4a780051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.430+0000 7f4a9e000640 1 -- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a780047c0 msgr2=0x7f4a78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.430+0000 7f4a9e000640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a780047c0 0x7f4a78004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.430+0000 7f4a9e000640 1 -- 172.21.15.53:0/3232662394 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a78003520 con 0x7f4a780039a0 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.430+0000 7f4a9d600640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a780047c0 0x7f4a78004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.431+0000 7f4a9e000640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a78003dc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800ba30 tx=0x7f4a780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211f4d6c17ab4a7e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.431+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88032640 con 0x7f4a780039a0 2024-01-16T15:05:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a880327a0 con 0x7f4a780039a0 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4a9cc00640 1 -- 172.21.15.53:0/3232662394 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88032b30 con 0x7f4a780039a0 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4aa0a00640 1 -- 172.21.15.53:0/3232662394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 msgr2=0x7f4a78003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/3232662394 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a78003dc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f4a8800ba30 tx=0x7f4a780057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4a9ea00640 1 -- 172.21.15.53:0/3232662394 reap_dead start 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4aa0a00640 1 -- 172.21.15.53:0/3232662394 shutdown_connections 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4aa0a00640 1 -- 172.21.15.53:0/3232662394 >> 172.21.15.53:0/3232662394 conn(0x7f4a9806a800 msgr2=0x7f4a9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4aa0a00640 1 -- 172.21.15.53:0/3232662394 shutdown_connections 2024-01-16T15:05:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.432+0000 7f4aa0a00640 1 -- 172.21.15.53:0/3232662394 wait complete. 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.434+0000 7f4aa0a00640 1 Processor -- start 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.434+0000 7f4aa0a00640 1 -- start start 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4aa0a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a9813d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4aa0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9813d8a0 0x7f4a98156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4aa0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a980b2cf0 0x7f4a980b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4aa0a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98151960 con 0x7f4a780039a0 2024-01-16T15:05:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4aa0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a98153540 con 0x7f4a9813d8a0 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4aa0a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9813d720 con 0x7f4a980b2cf0 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4a9e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a9813d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4a9e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a9813d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43878/0 (socket says 172.21.15.53:43878) 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.435+0000 7f4a9e000640 1 -- 172.21.15.53:0/2302220254 learned_addr learned my addr 172.21.15.53:0/2302220254 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9d600640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9813d8a0 0x7f4a98156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9ea00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a980b2cf0 0x7f4a980b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9e000640 1 -- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a980b2cf0 msgr2=0x7f4a980b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9e000640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a980b2cf0 0x7f4a980b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9e000640 1 -- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9813d8a0 msgr2=0x7f4a98156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9e000640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9813d8a0 0x7f4a98156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9e000640 1 -- 172.21.15.53:0/2302220254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a98072310 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9ea00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a980b2cf0 0x7f4a980b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a9e000640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a9813d1e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4a8802bf90 tx=0x7f4a88004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a88004220 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a88004380 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a8803c870 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a98152ec0 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.436+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a98156d30 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.437+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a981537b0 con 0x7f4a780039a0 2024-01-16T15:05:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.438+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f4a88044070 con 0x7f4a780039a0 2024-01-16T15:05:20.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.439+0000 7f4a96000640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7006b860 0x7f4a7006dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.439+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4a880acb20 con 0x7f4a780039a0 2024-01-16T15:05:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.440+0000 7f4a9d600640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7006b860 0x7f4a7006dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.441+0000 7f4a9d600640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7006b860 0x7f4a7006dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4a9815e720 tx=0x7f4a80009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.444+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a880770f0 con 0x7f4a780039a0 2024-01-16T15:05:20.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.628+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a98152d20 con 0x7f4a780039a0 2024-01-16T15:05:20.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.630+0000 7f4a96000640 1 -- 172.21.15.53:0/2302220254 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16542 (secure 0 0 0) 0x7f4a98152d20 con 0x7f4a780039a0 2024-01-16T15:05:20.609 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:20.610 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":59,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:19.687671+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:17.563281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:18.694511+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:20.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7006b860 msgr2=0x7f4a7006dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7006b860 0x7f4a7006dd20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f4a9815e720 tx=0x7f4a80009040 comp rx=0 tx=0).stop 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 msgr2=0x7f4a9813d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a9813d1e0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f4a8802bf90 tx=0x7f4a88004730 comp rx=0 tx=0).stop 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 shutdown_connections 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a7006b860 0x7f4a7006dd20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a980b2cf0 0x7f4a980b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a9813d8a0 0x7f4a98156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 --2- 172.21.15.53:0/2302220254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a780039a0 0x7f4a9813d1e0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 >> 172.21.15.53:0/2302220254 conn(0x7f4a9806a800 msgr2=0x7f4a980cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 shutdown_connections 2024-01-16T15:05:20.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.633+0000 7f4aa0a00640 1 -- 172.21.15.53:0/2302220254 wait complete. 2024-01-16T15:05:20.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:20.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.874+0000 7f0104c00640 1 Processor -- start 2024-01-16T15:05:20.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.874+0000 7f0104c00640 1 -- start start 2024-01-16T15:05:20.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.876+0000 7f0104c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f010013df80 0x7f010015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.876+0000 7f0104c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f01000cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.876+0000 7f0104c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010006c470 con 0x7f010013e360 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.876+0000 7f0104c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f010006c750 con 0x7f010013df80 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.876+0000 7f0104c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010006ca30 con 0x7f01000cf9a0 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f010013df80 0x7f010015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f010013df80 0x7f010015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43890/0 (socket says 172.21.15.53:43890) 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 -- 172.21.15.53:0/3802860810 learned_addr learned my addr 172.21.15.53:0/3802860810 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:20.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fe800640 1 --2- 172.21.15.53:0/3802860810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f01000cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 -- 172.21.15.53:0/3802860810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 msgr2=0x7f01000cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:20.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 --2- 172.21.15.53:0/3802860810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f01000cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 -- 172.21.15.53:0/3802860810 >> v1:172.21.15.146:6789/0 conn(0x7f010013e360 legacy=0x7f01000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.877+0000 7f00fde00640 1 -- 172.21.15.53:0/3802860810 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f010006cd10 con 0x7f010013df80 2024-01-16T15:05:20.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.878+0000 7f00fde00640 1 --2- 172.21.15.53:0/3802860810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013df80 0x7f010015bee0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e80ccccda09d3da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.879+0000 7f00fca00640 1 -- 172.21.15.53:0/3802860810 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8030840 con 0x7f010013df80 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.879+0000 7f00fca00640 1 -- 172.21.15.53:0/3802860810 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e80309a0 con 0x7f010013df80 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.879+0000 7f00fca00640 1 -- 172.21.15.53:0/3802860810 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8030d30 con 0x7f010013df80 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.879+0000 7f0104c00640 1 -- 172.21.15.53:0/3802860810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013df80 msgr2=0x7f010015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.879+0000 7f0104c00640 1 --2- 172.21.15.53:0/3802860810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013df80 0x7f010015bee0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e802b980 comp rx=0 tx=0).stop 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.880+0000 7f0104c00640 1 -- 172.21.15.53:0/3802860810 shutdown_connections 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.880+0000 7f0104c00640 1 --2- 172.21.15.53:0/3802860810 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f01000cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.880+0000 7f0104c00640 1 --2- 172.21.15.53:0/3802860810 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013df80 0x7f010015bee0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.880+0000 7f0104c00640 1 -- 172.21.15.53:0/3802860810 >> 172.21.15.53:0/3802860810 conn(0x7f010006a800 msgr2=0x7f010006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:20.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.880+0000 7f0104c00640 1 -- 172.21.15.53:0/3802860810 shutdown_connections 2024-01-16T15:05:20.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.880+0000 7f0104c00640 1 -- 172.21.15.53:0/3802860810 wait complete. 2024-01-16T15:05:20.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.881+0000 7f0104c00640 1 Processor -- start 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.882+0000 7f0104c00640 1 -- start start 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.882+0000 7f0104c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f010013cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.882+0000 7f0104c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010013df80 0x7f01001e6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.882+0000 7f0104c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 0x7f01001f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.882+0000 7f0104c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01001f4a10 con 0x7f010013e360 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f0104c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01001f4cf0 con 0x7f010013df80 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f0104c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01001f4fd0 con 0x7f01000cf9a0 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 0x7f01001f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 0x7f01001f5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43904/0 (socket says 172.21.15.53:43904) 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 -- 172.21.15.53:0/1246600844 learned_addr learned my addr 172.21.15.53:0/1246600844 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fd400640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010013df80 0x7f01001e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fde00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f010013cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 -- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 msgr2=0x7f010013cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f010013cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 -- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010013df80 msgr2=0x7f01001e6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010013df80 0x7f01001e6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fe800640 1 -- 172.21.15.53:0/1246600844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e80094d0 con 0x7f010013e360 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.883+0000 7f00fde00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f010013cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:20.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.884+0000 7f00fe800640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 0x7f01001f5ab0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f00f800d5a0 tx=0x7f00f800da60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.884+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f80125f0 con 0x7f010013e360 2024-01-16T15:05:20.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.884+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f8012750 con 0x7f010013e360 2024-01-16T15:05:20.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.884+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8012ae0 con 0x7f010013e360 2024-01-16T15:05:20.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.884+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01001f52b0 con 0x7f010013e360 2024-01-16T15:05:20.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.884+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f010013d4b0 con 0x7f010013e360 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.885+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01001f57d0 con 0x7f010013e360 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.886+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f00f8012c40 con 0x7f010013e360 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.888+0000 7f00f6000640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d806b860 0x7f00d806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:20.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.888+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f00f808aac0 con 0x7f010013e360 2024-01-16T15:05:20.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.889+0000 7f00fde00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d806b860 0x7f00d806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:20.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.890+0000 7f00fde00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d806b860 0x7f00d806dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f00e8036000 tx=0x7f00e803b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:20.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:20.893+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01001f57d0 con 0x7f010013e360 2024-01-16T15:05:21.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.089+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01001f57d0 con 0x7f010013e360 2024-01-16T15:05:21.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.092+0000 7f00f6000640 1 -- 172.21.15.53:0/1246600844 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+16542 (secure 0 0 0) 0x7f01001f57d0 con 0x7f010013e360 2024-01-16T15:05:21.071 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:21.071 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":59,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:19.687671+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:17.563281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:18.694511+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d806b860 msgr2=0x7f00d806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d806b860 0x7f00d806dd20 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f00e8036000 tx=0x7f00e803b040 comp rx=0 tx=0).stop 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 msgr2=0x7f01001f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 0x7f01001f5ab0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f00f800d5a0 tx=0x7f00f800da60 comp rx=0 tx=0).stop 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 shutdown_connections 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d806b860 0x7f00d806dd20 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f010013e360 0x7f01001f5ab0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f010013df80 0x7f01001e6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 --2- 172.21.15.53:0/1246600844 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01000cf9a0 0x7f010013cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 >> 172.21.15.53:0/1246600844 conn(0x7f010006a800 msgr2=0x7f01000b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:21.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 shutdown_connections 2024-01-16T15:05:21.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.095+0000 7f0104c00640 1 -- 172.21.15.53:0/1246600844 wait complete. 2024-01-16T15:05:21.093 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:05:21.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.340+0000 7f6793000640 1 Processor -- start 2024-01-16T15:05:21.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.340+0000 7f6793000640 1 -- start start 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.341+0000 7f6793000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f678c15d4b0 0x7f678c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.341+0000 7f6793000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.341+0000 7f6793000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678c0d0320 con 0x7f678c0cf9a0 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.341+0000 7f6793000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678c0b2730 con 0x7f678c15d4b0 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.341+0000 7f6793000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0d1ec0 con 0x7f678c0b41e0 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f678c15d4b0 0x7f678c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f678c15d4b0 0x7f678c0d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43906/0 (socket says 172.21.15.53:43906) 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 -- 172.21.15.53:0/4102866883 learned_addr learned my addr 172.21.15.53:0/4102866883 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6792600640 1 --2- 172.21.15.53:0/4102866883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 -- 172.21.15.53:0/4102866883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 msgr2=0x7f678c0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:21.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 --2- 172.21.15.53:0/4102866883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 -- 172.21.15.53:0/4102866883 >> v1:172.21.15.146:6789/0 conn(0x7f678c0cf9a0 legacy=0x7f678c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:21.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.342+0000 7f6791c00640 1 -- 172.21.15.53:0/4102866883 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c0b29d0 con 0x7f678c15d4b0 2024-01-16T15:05:21.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.344+0000 7f6791c00640 1 --2- 172.21.15.53:0/4102866883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c15d4b0 0x7f678c0d1550 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f677c009e50 tx=0x7f677c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f6e005ac024c98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:21.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.344+0000 7f678be00640 1 -- 172.21.15.53:0/4102866883 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c02ca70 con 0x7f678c15d4b0 2024-01-16T15:05:21.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.344+0000 7f678be00640 1 -- 172.21.15.53:0/4102866883 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f677c02cbd0 con 0x7f678c15d4b0 2024-01-16T15:05:21.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.344+0000 7f678be00640 1 -- 172.21.15.53:0/4102866883 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f677c0343f0 con 0x7f678c15d4b0 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 -- 172.21.15.53:0/4102866883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c15d4b0 msgr2=0x7f678c0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 --2- 172.21.15.53:0/4102866883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c15d4b0 0x7f678c0d1550 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f677c009e50 tx=0x7f677c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 -- 172.21.15.53:0/4102866883 shutdown_connections 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 --2- 172.21.15.53:0/4102866883 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 --2- 172.21.15.53:0/4102866883 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c15d4b0 0x7f678c0d1550 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 -- 172.21.15.53:0/4102866883 >> 172.21.15.53:0/4102866883 conn(0x7f678c0d3000 msgr2=0x7f678c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:21.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 -- 172.21.15.53:0/4102866883 shutdown_connections 2024-01-16T15:05:21.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.345+0000 7f6793000640 1 -- 172.21.15.53:0/4102866883 wait complete. 2024-01-16T15:05:21.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.347+0000 7f6793000640 1 Processor -- start 2024-01-16T15:05:21.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.347+0000 7f6793000640 1 -- start start 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6793000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c09d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6793000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 0x7f678c15a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6793000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f678c15d4b0 0x7f678c09fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6793000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0ca550 con 0x7f678c0cf9a0 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6793000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f678c15ab30 con 0x7f678c15d4b0 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6793000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f678c0a0420 con 0x7f678c0b41e0 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6791200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 0x7f678c15a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6791200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 0x7f678c15a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43920/0 (socket says 172.21.15.53:43920) 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6791200640 1 -- 172.21.15.53:0/3738528851 learned_addr learned my addr 172.21.15.53:0/3738528851 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6792600640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f678c15d4b0 0x7f678c09fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6791c00640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c09d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6791200640 1 -- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 msgr2=0x7f678c09d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.349+0000 7f6791200640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c09d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6791200640 1 -- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f678c15d4b0 msgr2=0x7f678c09fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6791200640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f678c15d4b0 0x7f678c09fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6791200640 1 -- 172.21.15.53:0/3738528851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c009b30 con 0x7f678c0cf9a0 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6791c00640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6791200640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 0x7f678c15a5f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f677400c430 tx=0x7f677400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:21.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011790 con 0x7f678c0cf9a0 2024-01-16T15:05:21.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67740118f0 con 0x7f678c0cf9a0 2024-01-16T15:05:21.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6774011ca0 con 0x7f678c0cf9a0 2024-01-16T15:05:21.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f678c15da10 con 0x7f678c0cf9a0 2024-01-16T15:05:21.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.350+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678c0a09e0 con 0x7f678c0cf9a0 2024-01-16T15:05:21.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.352+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f678c0ca690 con 0x7f678c0cf9a0 2024-01-16T15:05:21.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.352+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f677401c460 con 0x7f678c0cf9a0 2024-01-16T15:05:21.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.353+0000 7f6789600640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f676406b860 0x7f676406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:21.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.353+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f677408a500 con 0x7f678c0cf9a0 2024-01-16T15:05:21.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.354+0000 7f6791c00640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f676406b860 0x7f676406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:21.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.355+0000 7f6791c00640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f676406b860 0x7f676406dd20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f677c035000 tx=0x7f677c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:21.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.358+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f678c0ca690 con 0x7f678c0cf9a0 2024-01-16T15:05:21.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:21.449 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:21.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:21.592+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f678c0ca690 con 0x7f678c0cf9a0 2024-01-16T15:05:22.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.247+0000 7f6789600640 1 -- 172.21.15.53:0/3738528851 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v34) v1 ==== 118+0+0 (secure 0 0 0) 0x7f678c0ca690 con 0x7f678c0cf9a0 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f676406b860 msgr2=0x7f676406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f676406b860 0x7f676406dd20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f677c035000 tx=0x7f677c030000 comp rx=0 tx=0).stop 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 msgr2=0x7f678c15a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 0x7f678c15a5f0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f677400c430 tx=0x7f677400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 shutdown_connections 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f676406b860 0x7f676406dd20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f678c15d4b0 0x7f678c09fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f678c0cf9a0 0x7f678c15a5f0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 --2- 172.21.15.53:0/3738528851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f678c0b41e0 0x7f678c09d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.250+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 >> 172.21.15.53:0/3738528851 conn(0x7f678c0d3000 msgr2=0x7f678c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:22.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.251+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 shutdown_connections 2024-01-16T15:05:22.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.251+0000 7f6793000640 1 -- 172.21.15.53:0/3738528851 wait complete. 2024-01-16T15:05:22.251 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:05:22.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:05:22.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.544+0000 7fd4f3000640 1 Processor -- start 2024-01-16T15:05:22.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.544+0000 7fd4f3000640 1 -- start start 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4f3000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd4ec13df80 0x7fd4ec15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4f3000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd4ec0cf9a0 0x7fd4ec0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4f3000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4ec06c470 con 0x7fd4ec13e360 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4f3000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec06c750 con 0x7fd4ec13df80 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4f3000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4ec06ca30 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4ebe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd4ec13df80 0x7fd4ec15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4eb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd4ec13e360 0x7fd4ec0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40314/0 (socket says 172.21.15.53:40314) 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4eb400640 1 -- 172.21.15.53:0/4057677823 learned_addr learned my addr 172.21.15.53:0/4057677823 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:22.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4ebe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd4ec13df80 0x7fd4ec15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52164/0 (socket says 172.21.15.53:52164) 2024-01-16T15:05:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4f1000640 1 --2- 172.21.15.53:0/4057677823 >> v2:172.21.15.146:3300/0 conn(0x7fd4ec0cf9a0 0x7fd4ec0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4ebe00640 1 -- 172.21.15.53:0/4057677823 >> v2:172.21.15.146:3300/0 conn(0x7fd4ec0cf9a0 msgr2=0x7fd4ec0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4ebe00640 1 --2- 172.21.15.53:0/4057677823 >> v2:172.21.15.146:3300/0 conn(0x7fd4ec0cf9a0 0x7fd4ec0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4ebe00640 1 -- 172.21.15.53:0/4057677823 >> v1:172.21.15.53:6789/0 conn(0x7fd4ec13e360 legacy=0x7fd4ec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.546+0000 7fd4ebe00640 1 -- 172.21.15.53:0/4057677823 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ec06cd10 con 0x7fd4ec13df80 2024-01-16T15:05:22.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.547+0000 7fd4f1000640 1 --2- 172.21.15.53:0/4057677823 >> v2:172.21.15.146:3300/0 conn(0x7fd4ec0cf9a0 0x7fd4ec0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.548+0000 7fd4ebe00640 1 --2- 172.21.15.53:0/4057677823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec13df80 0x7fd4ec15bee0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0ccb10 tx=0x7fd4dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8426fae8277e8c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.548+0000 7fd4eaa00640 1 -- 172.21.15.53:0/4057677823 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc031830 con 0x7fd4ec13df80 2024-01-16T15:05:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.548+0000 7fd4eaa00640 1 -- 172.21.15.53:0/4057677823 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc031990 con 0x7fd4ec13df80 2024-01-16T15:05:22.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.548+0000 7fd4eaa00640 1 -- 172.21.15.53:0/4057677823 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc031d20 con 0x7fd4ec13df80 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 -- 172.21.15.53:0/4057677823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec13df80 msgr2=0x7fd4ec15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 --2- 172.21.15.53:0/4057677823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec13df80 0x7fd4ec15bee0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec0ccb10 tx=0x7fd4dc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 -- 172.21.15.53:0/4057677823 shutdown_connections 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 --2- 172.21.15.53:0/4057677823 >> v2:172.21.15.146:3300/0 conn(0x7fd4ec0cf9a0 0x7fd4ec0cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 --2- 172.21.15.53:0/4057677823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec13df80 0x7fd4ec15bee0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 -- 172.21.15.53:0/4057677823 >> 172.21.15.53:0/4057677823 conn(0x7fd4ec06a800 msgr2=0x7fd4ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 -- 172.21.15.53:0/4057677823 shutdown_connections 2024-01-16T15:05:22.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.549+0000 7fd4f3000640 1 -- 172.21.15.53:0/4057677823 wait complete. 2024-01-16T15:05:22.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.551+0000 7fd4f3000640 1 Processor -- start 2024-01-16T15:05:22.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.551+0000 7fd4f3000640 1 -- start start 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4f3000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 0x7fd4ec071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4f3000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec13df80 0x7fd4ec132590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4f3000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec13e360 0x7fd4ec15a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4f3000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec1f3640 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4f3000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0a8e40 con 0x7fd4ec13e360 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4f3000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4ec0a90e0 con 0x7fd4ec13df80 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 0x7fd4ec071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 0x7fd4ec071ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52168/0 (socket says 172.21.15.53:52168) 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4ebe00640 1 -- 172.21.15.53:0/446451867 learned_addr learned my addr 172.21.15.53:0/446451867 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:22.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4ebe00640 1 -- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec13df80 msgr2=0x7fd4ec132590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.552+0000 7fd4ebe00640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec13df80 0x7fd4ec132590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4f1000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec13e360 0x7fd4ec15a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4ebe00640 1 -- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec13e360 msgr2=0x7fd4ec15a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4ebe00640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec13e360 0x7fd4ec15a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4ebe00640 1 -- 172.21.15.53:0/446451867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4dc009400 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4ebe00640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 0x7fd4ec071ee0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009890 tx=0x7fd4dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc03ba70 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4dc03bbf0 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc045400 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4ec0a93c0 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.553+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4ec132ad0 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.555+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4ec0675d0 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.556+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fd4dc04e440 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.557+0000 7fd4e3e00640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4c406b890 0x7fd4c406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:22.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.557+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd4dc0acac0 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.558+0000 7fd4eb400640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4c406b890 0x7fd4c406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:22.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.559+0000 7fd4eb400640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4c406b890 0x7fd4c406dd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec1545a0 tx=0x7fd4d4007a40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:22.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.562+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd4dc076f70 con 0x7fd4ec0cf9a0 2024-01-16T15:05:22.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:22.794+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd4ec06df00 con 0x7fd4ec0cf9a0 2024-01-16T15:05:23.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.257+0000 7fd4e3e00640 1 -- 172.21.15.53:0/446451867 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v35) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd4dc0350c0 con 0x7fd4ec0cf9a0 2024-01-16T15:05:23.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4c406b890 msgr2=0x7fd4c406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4c406b890 0x7fd4c406dd50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec1545a0 tx=0x7fd4d4007a40 comp rx=0 tx=0).stop 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 msgr2=0x7fd4ec071ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 0x7fd4ec071ee0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009890 tx=0x7fd4dc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 shutdown_connections 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4c406b890 0x7fd4c406dd50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd4ec13e360 0x7fd4ec15a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd4ec13df80 0x7fd4ec132590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 --2- 172.21.15.53:0/446451867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd4ec0cf9a0 0x7fd4ec071ee0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.261+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 >> 172.21.15.53:0/446451867 conn(0x7fd4ec06a800 msgr2=0x7fd4ec06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.262+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 shutdown_connections 2024-01-16T15:05:23.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.262+0000 7fd4f3000640 1 -- 172.21.15.53:0/446451867 wait complete. 2024-01-16T15:05:23.261 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:05:23.261 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:23.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.498+0000 7f029e200640 1 Processor -- start 2024-01-16T15:05:23.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.499+0000 7f029e200640 1 -- start start 2024-01-16T15:05:23.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.500+0000 7f029e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0298159590 0x7f0298070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.500+0000 7f029e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02980b2450 0x7f02980b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.500+0000 7f029e200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0298159a20 con 0x7f02980b3c20 2024-01-16T15:05:23.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.500+0000 7f029e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0298071790 con 0x7f0298159590 2024-01-16T15:05:23.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.500+0000 7f029e200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0298071100 con 0x7f02980b2450 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f029ce00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f02980b3c20 0x7f02980ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40324/0 (socket says 172.21.15.53:40324) 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f029ce00640 1 -- 172.21.15.53:0/2747082899 learned_addr learned my addr 172.21.15.53:0/2747082899 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297e00640 1 --2- 172.21.15.53:0/2747082899 >> v2:172.21.15.53:3300/0 conn(0x7f0298159590 0x7f0298070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f029d800640 1 --2- 172.21.15.53:0/2747082899 >> v2:172.21.15.146:3300/0 conn(0x7f02980b2450 0x7f02980b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297400640 1 -- 172.21.15.53:0/2747082899 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083496285 0 0) 0x7f0298159a20 con 0x7f02980b3c20 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297400640 1 -- 172.21.15.53:0/2747082899 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0278005730 con 0x7f02980b3c20 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297e00640 1 -- 172.21.15.53:0/2747082899 >> v2:172.21.15.146:3300/0 conn(0x7f02980b2450 msgr2=0x7f02980b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297e00640 1 --2- 172.21.15.53:0/2747082899 >> v2:172.21.15.146:3300/0 conn(0x7f02980b2450 0x7f02980b28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297e00640 1 -- 172.21.15.53:0/2747082899 >> v1:172.21.15.53:6789/0 conn(0x7f02980b3c20 legacy=0x7f02980ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.501+0000 7f0297e00640 1 -- 172.21.15.53:0/2747082899 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298070860 con 0x7f0298159590 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.503+0000 7f0297e00640 1 --2- 172.21.15.53:0/2747082899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0298159590 0x7f0298070160 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f02980b37a0 tx=0x7f028002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751e2444c7478b53 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.503+0000 7f0297400640 1 -- 172.21.15.53:0/2747082899 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028002e070 con 0x7f0298159590 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.503+0000 7f0297400640 1 -- 172.21.15.53:0/2747082899 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02800315a0 con 0x7f0298159590 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.503+0000 7f0297400640 1 -- 172.21.15.53:0/2747082899 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280031930 con 0x7f0298159590 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.503+0000 7f029e200640 1 -- 172.21.15.53:0/2747082899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0298159590 msgr2=0x7f0298070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.503+0000 7f029e200640 1 --2- 172.21.15.53:0/2747082899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0298159590 0x7f0298070160 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f02980b37a0 tx=0x7f028002b840 comp rx=0 tx=0).stop 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.504+0000 7f029e200640 1 -- 172.21.15.53:0/2747082899 shutdown_connections 2024-01-16T15:05:23.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.504+0000 7f029e200640 1 --2- 172.21.15.53:0/2747082899 >> v2:172.21.15.146:3300/0 conn(0x7f02980b2450 0x7f02980b28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.504+0000 7f029e200640 1 --2- 172.21.15.53:0/2747082899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0298159590 0x7f0298070160 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.504+0000 7f029e200640 1 -- 172.21.15.53:0/2747082899 >> 172.21.15.53:0/2747082899 conn(0x7f029806a800 msgr2=0x7f029806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:23.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.504+0000 7f029e200640 1 -- 172.21.15.53:0/2747082899 shutdown_connections 2024-01-16T15:05:23.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.504+0000 7f029e200640 1 -- 172.21.15.53:0/2747082899 wait complete. 2024-01-16T15:05:23.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.505+0000 7f029e200640 1 Processor -- start 2024-01-16T15:05:23.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.506+0000 7f029e200640 1 -- start start 2024-01-16T15:05:23.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02980b2450 0x7f0298153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 0x7f0298153d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0298159590 0x7f02981542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029815b9a0 con 0x7f02980b3c20 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f029815adb0 con 0x7f0298159590 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f029815b070 con 0x7f02980b2450 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 0x7f0298153d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 0x7f0298153d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52178/0 (socket says 172.21.15.53:52178) 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 -- 172.21.15.53:0/895256752 learned_addr learned my addr 172.21.15.53:0/895256752 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029d800640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0298159590 0x7f02981542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f029ce00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02980b2450 0x7f0298153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 -- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02980b2450 msgr2=0x7f0298153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02980b2450 0x7f0298153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 -- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0298159590 msgr2=0x7f02981542a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.507+0000 7f0297e00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0298159590 0x7f02981542a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f0297e00640 1 -- 172.21.15.53:0/895256752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0280009400 con 0x7f02980b3c20 2024-01-16T15:05:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f029ce00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02980b2450 0x7f0298153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:23.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f0297e00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 0x7f0298153d60 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0280009890 tx=0x7f02800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:23.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280032840 con 0x7f02980b3c20 2024-01-16T15:05:23.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0280031f00 con 0x7f02980b3c20 2024-01-16T15:05:23.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280032be0 con 0x7f02980b3c20 2024-01-16T15:05:23.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.508+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f029815b350 con 0x7f02980b3c20 2024-01-16T15:05:23.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.509+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029813fc00 con 0x7f02980b3c20 2024-01-16T15:05:23.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.510+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0298153600 con 0x7f02980b3c20 2024-01-16T15:05:23.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.511+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f0280032d40 con 0x7f02980b3c20 2024-01-16T15:05:23.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.512+0000 7f0294c00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027006b890 0x7f027006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.512+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02800ac5e0 con 0x7f02980b3c20 2024-01-16T15:05:23.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.512+0000 7f029ce00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027006b890 0x7f027006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.513+0000 7f029ce00640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027006b890 0x7f027006dd50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0288005c00 tx=0x7f0288009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:23.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.517+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0280076b80 con 0x7f02980b3c20 2024-01-16T15:05:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.747+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0298071300 con 0x7f02980b3c20 2024-01-16T15:05:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.748+0000 7f0294c00640 1 -- 172.21.15.53:0/895256752 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4919 (secure 0 0 0) 0x7f028007a1a0 con 0x7f02980b3c20 2024-01-16T15:05:23.726 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:23.727 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/2632791628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2632791628},{"type":"v1","addr":"172.21.15.53:6833","nonce":2632791628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5680,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3474334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3474334502},{"type":"v1","addr":"172.21.15.53:6835","nonce":3474334502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5683,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2549681777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2549681777},{"type":"v1","addr":"172.21.15.146:6836","nonce":2549681777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:05:19.688137+0000","modified":"2024-01-16T15:05:22.796011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5686":{"gid":5686,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3568122877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3568122877},{"type":"v1","addr":"172.21.15.146:6837","nonce":3568122877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:05:23.727 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 35 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027006b890 msgr2=0x7f027006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027006b890 0x7f027006dd50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f0288005c00 tx=0x7f0288009040 comp rx=0 tx=0).stop 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 msgr2=0x7f0298153d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 0x7f0298153d60 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0280009890 tx=0x7f02800046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 shutdown_connections 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027006b890 0x7f027006dd50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0298159590 0x7f02981542a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02980b3c20 0x7f0298153d60 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 --2- 172.21.15.53:0/895256752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02980b2450 0x7f0298153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 >> 172.21.15.53:0/895256752 conn(0x7f029806a800 msgr2=0x7f029806d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 shutdown_connections 2024-01-16T15:05:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.751+0000 7f029e200640 1 -- 172.21.15.53:0/895256752 wait complete. 2024-01-16T15:05:23.747 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:05:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.984+0000 7fbdf7400640 1 Processor -- start 2024-01-16T15:05:23.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.984+0000 7fbdf7400640 1 -- start start 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbdf00cc990 0x7fbdf006d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf7400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf0070e50 con 0x7fbdf0143c40 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf7400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdf0144020 con 0x7fbdf00cc500 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf7400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdf00723c0 con 0x7fbdf00cc990 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf6000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbdf0143c40 0x7fbdf0071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40336/0 (socket says 172.21.15.53:40336) 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf6000640 1 -- 172.21.15.53:0/662951503 learned_addr learned my addr 172.21.15.53:0/662951503 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:23.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.986+0000 7fbdf6a00640 1 --2- 172.21.15.53:0/662951503 >> v2:172.21.15.53:3300/0 conn(0x7fbdf00cc990 0x7fbdf006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.987+0000 7fbdf6a00640 1 -- 172.21.15.53:0/662951503 >> v1:172.21.15.146:6789/0 conn(0x7fbdf00cc500 legacy=0x7fbdf006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.987+0000 7fbdf6a00640 1 -- 172.21.15.53:0/662951503 >> v1:172.21.15.53:6789/0 conn(0x7fbdf0143c40 legacy=0x7fbdf0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.987+0000 7fbdf6a00640 1 -- 172.21.15.53:0/662951503 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf0071e00 con 0x7fbdf00cc990 2024-01-16T15:05:23.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.988+0000 7fbdf6a00640 1 --2- 172.21.15.53:0/662951503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf00cc990 0x7fbdf006d870 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbde80097a0 tx=0x7fbde802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f6d2e39544a97f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.988+0000 7fbdf4c00640 1 -- 172.21.15.53:0/662951503 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde802e070 con 0x7fbdf00cc990 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.988+0000 7fbdf4c00640 1 -- 172.21.15.53:0/662951503 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80315a0 con 0x7fbdf00cc990 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.988+0000 7fbdf4c00640 1 -- 172.21.15.53:0/662951503 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8031930 con 0x7fbdf00cc990 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 -- 172.21.15.53:0/662951503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf00cc990 msgr2=0x7fbdf006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 --2- 172.21.15.53:0/662951503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf00cc990 0x7fbdf006d870 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fbde80097a0 tx=0x7fbde802b840 comp rx=0 tx=0).stop 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 -- 172.21.15.53:0/662951503 shutdown_connections 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 --2- 172.21.15.53:0/662951503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf00cc990 0x7fbdf006d870 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 -- 172.21.15.53:0/662951503 >> 172.21.15.53:0/662951503 conn(0x7fbdf006a800 msgr2=0x7fbdf006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:23.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 -- 172.21.15.53:0/662951503 shutdown_connections 2024-01-16T15:05:23.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.989+0000 7fbdf7400640 1 -- 172.21.15.53:0/662951503 wait complete. 2024-01-16T15:05:23.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.991+0000 7fbdf7400640 1 Processor -- start 2024-01-16T15:05:23.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.991+0000 7fbdf7400640 1 -- start start 2024-01-16T15:05:23.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf7400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf00cc500 0x7fbdf0154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf7400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf00cc990 0x7fbdf0133e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 0x7fbdf0136d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf7400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf013d830 con 0x7fbdf0143c40 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf7400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf01358d0 con 0x7fbdf00cc990 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf7400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf0135030 con 0x7fbdf00cc500 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 0x7fbdf0136d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 0x7fbdf0136d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52192/0 (socket says 172.21.15.53:52192) 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 -- 172.21.15.53:0/2627550700 learned_addr learned my addr 172.21.15.53:0/2627550700 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf5600640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf00cc990 0x7fbdf0133e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6000640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf00cc500 0x7fbdf0154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 -- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf00cc500 msgr2=0x7fbdf0154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf00cc500 0x7fbdf0154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 -- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf00cc990 msgr2=0x7fbdf0133e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.992+0000 7fbdf6a00640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf00cc990 0x7fbdf0133e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdf6a00640 1 -- 172.21.15.53:0/2627550700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8009400 con 0x7fbdf0143c40 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdf6a00640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 0x7fbdf0136d30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbde80097a0 tx=0x7fbde80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8032840 con 0x7fbdf0143c40 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde80329c0 con 0x7fbdf0143c40 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbde8032d70 con 0x7fbdf0143c40 2024-01-16T15:05:23.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf0135310 con 0x7fbdf0143c40 2024-01-16T15:05:23.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.993+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf013d4c0 con 0x7fbdf0143c40 2024-01-16T15:05:23.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.995+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf013d320 con 0x7fbdf0143c40 2024-01-16T15:05:23.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.996+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fbde803b3f0 con 0x7fbdf0143c40 2024-01-16T15:05:23.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.997+0000 7fbdee000640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbdc806b890 0x7fbdc806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:23.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.997+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbde80ac690 con 0x7fbdf0143c40 2024-01-16T15:05:23.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.998+0000 7fbdf6000640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbdc806b890 0x7fbdc806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:23.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:23.999+0000 7fbdf6000640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbdc806b890 0x7fbdc806dd50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8005880 tx=0x7fbdd8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:23.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.002+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbdf013d320 con 0x7fbdf0143c40 2024-01-16T15:05:24.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.198+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbdf013d320 con 0x7fbdf0143c40 2024-01-16T15:05:24.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.200+0000 7fbdee000640 1 -- 172.21.15.53:0/2627550700 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16529 (secure 0 0 0) 0x7fbdf013d320 con 0x7fbdf0143c40 2024-01-16T15:05:24.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:24.180 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":60,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:05:21.204096+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:05:17.563281+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"59","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2024-01-16T15:05:18.694511+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:05:24.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbdc806b890 msgr2=0x7fbdc806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbdc806b890 0x7fbdc806dd50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8005880 tx=0x7fbdd8009040 comp rx=0 tx=0).stop 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 msgr2=0x7fbdf0136d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 0x7fbdf0136d30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fbde80097a0 tx=0x7fbde80046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 shutdown_connections 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbdc806b890 0x7fbdc806dd50 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbdf0143c40 0x7fbdf0136d30 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbdf00cc990 0x7fbdf0133e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 --2- 172.21.15.53:0/2627550700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbdf00cc500 0x7fbdf0154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 >> 172.21.15.53:0/2627550700 conn(0x7fbdf006a800 msgr2=0x7fbdf00b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 shutdown_connections 2024-01-16T15:05:24.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.204+0000 7fbdf7400640 1 -- 172.21.15.53:0/2627550700 wait complete. 2024-01-16T15:05:24.201 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:05:24.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:05:24.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.443+0000 7f9fa2200640 1 Processor -- start 2024-01-16T15:05:24.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.444+0000 7f9fa2200640 1 -- start start 2024-01-16T15:05:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.445+0000 7f9fa2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 0x7f9f9c0ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.445+0000 7f9fa2200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9f9c0aede0 0x7f9f9c06f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.445+0000 7f9fa2200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c071de0 con 0x7f9f9c06a3e0 2024-01-16T15:05:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.445+0000 7f9fa2200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c0af270 con 0x7f9f9c0aede0 2024-01-16T15:05:24.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.446+0000 7f9fa2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c072030 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.446+0000 7f9f9aa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 0x7f9f9c0ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.446+0000 7f9f9aa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 0x7f9f9c0ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:47380/0 (socket says 172.21.15.53:47380) 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.446+0000 7f9f9aa00640 1 -- 172.21.15.53:0/1800190466 learned_addr learned my addr 172.21.15.53:0/1800190466 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.447+0000 7f9f9be00640 1 --2- 172.21.15.53:0/1800190466 >> v2:172.21.15.146:3300/0 conn(0x7f9f9c0aede0 0x7f9f9c06f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.447+0000 7f9f9aa00640 1 -- 172.21.15.53:0/1800190466 >> v2:172.21.15.146:3300/0 conn(0x7f9f9c0aede0 msgr2=0x7f9f9c06f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.447+0000 7f9f9aa00640 1 --2- 172.21.15.53:0/1800190466 >> v2:172.21.15.146:3300/0 conn(0x7f9f9c0aede0 0x7f9f9c06f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.447+0000 7f9f9aa00640 1 -- 172.21.15.53:0/1800190466 >> v1:172.21.15.146:6789/0 conn(0x7f9f9c06a3e0 legacy=0x7f9f9c06a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.447+0000 7f9f9aa00640 1 -- 172.21.15.53:0/1800190466 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c072310 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.448+0000 7f9f9aa00640 1 --2- 172.21.15.53:0/1800190466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 0x7f9f9c0ac8d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009e50 tx=0x7f9f8c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=893e93273165d698 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:24.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.449+0000 7f9f9a000640 1 -- 172.21.15.53:0/1800190466 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c02cb00 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.449+0000 7f9f9a000640 1 -- 172.21.15.53:0/1800190466 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c02cc60 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.449+0000 7f9f9a000640 1 -- 172.21.15.53:0/1800190466 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0343f0 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 -- 172.21.15.53:0/1800190466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 msgr2=0x7f9f9c0ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1800190466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 0x7f9f9c0ac8d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009e50 tx=0x7f9f8c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 -- 172.21.15.53:0/1800190466 shutdown_connections 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1800190466 >> v2:172.21.15.146:3300/0 conn(0x7f9f9c0aede0 0x7f9f9c06f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1800190466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0d0b60 0x7f9f9c0ac8d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 -- 172.21.15.53:0/1800190466 >> 172.21.15.53:0/1800190466 conn(0x7f9f9c0cfef0 msgr2=0x7f9f9c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 -- 172.21.15.53:0/1800190466 shutdown_connections 2024-01-16T15:05:24.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.450+0000 7f9fa2200640 1 -- 172.21.15.53:0/1800190466 wait complete. 2024-01-16T15:05:24.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.452+0000 7f9fa2200640 1 Processor -- start 2024-01-16T15:05:24.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.453+0000 7f9fa2200640 1 -- start start 2024-01-16T15:05:24.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9fa2200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f9c06a3e0 0x7f9f9c147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9fa2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0aede0 0x7f9f9c0cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9fa2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 0x7f9f9c1f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9fa2200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c147f40 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9fa2200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c15aed0 con 0x7f9f9c06a3e0 2024-01-16T15:05:24.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9fa2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c0ce200 con 0x7f9f9c0aede0 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9f9be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 0x7f9f9c1f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9f9be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 0x7f9f9c1f3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52200/0 (socket says 172.21.15.53:52200) 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.454+0000 7f9f9be00640 1 -- 172.21.15.53:0/1992444404 learned_addr learned my addr 172.21.15.53:0/1992444404 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9aa00640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0aede0 0x7f9f9c0cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9b400640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f9c06a3e0 0x7f9f9c147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9be00640 1 -- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0aede0 msgr2=0x7f9f9c0cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9be00640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0aede0 0x7f9f9c0cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9be00640 1 -- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f9c06a3e0 msgr2=0x7f9f9c147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9be00640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f9c06a3e0 0x7f9f9c147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9be00640 1 -- 172.21.15.53:0/1992444404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c009b30 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9b400640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f9c06a3e0 0x7f9f9c147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.455+0000 7f9f9be00640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 0x7f9f9c1f3490 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400d800 tx=0x7f9f9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.456+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9400b6c0 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.456+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f9400b840 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.456+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f9400bbf0 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.456+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c0cc940 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.456+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c0ccf00 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.457+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c073c00 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.458+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f9f9400bd50 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.459+0000 7f9f93400640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f7406b860 0x7f9f7406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.459+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9f9408aad0 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.460+0000 7f9f9b400640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f7406b860 0x7f9f7406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.461+0000 7f9f9b400640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f7406b860 0x7f9f7406dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9f84007fd0 tx=0x7f9f84009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:24.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.463+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f940550d0 con 0x7f9f9c0d0b60 2024-01-16T15:05:24.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.654+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9f9c06e880 con 0x7f9f7406b860 2024-01-16T15:05:24.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.682+0000 7f9f93400640 1 -- 172.21.15.53:0/1992444404 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286894 (secure 0 0 0) 0x7f9f840013e0 con 0x7f9f7406b860 2024-01-16T15:05:24.666 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:24.668 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:05:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.688+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f7406b860 msgr2=0x7f9f7406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.688+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f7406b860 0x7f9f7406dd20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9f84007fd0 tx=0x7f9f84009040 comp rx=0 tx=0).stop 2024-01-16T15:05:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.688+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 msgr2=0x7f9f9c1f3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.688+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 0x7f9f9c1f3490 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f9f9400d800 tx=0x7f9f9400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 shutdown_connections 2024-01-16T15:05:24.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f7406b860 0x7f9f7406dd20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f9c0d0b60 0x7f9f9c1f3490 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f9c0aede0 0x7f9f9c0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 --2- 172.21.15.53:0/1992444404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f9c06a3e0 0x7f9f9c147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 >> 172.21.15.53:0/1992444404 conn(0x7f9f9c0cfef0 msgr2=0x7f9f9c0ae370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 shutdown_connections 2024-01-16T15:05:24.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.689+0000 7f9fa2200640 1 -- 172.21.15.53:0/1992444404 wait complete. 2024-01-16T15:05:24.691 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-01-16T15:05:24.307040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":305972,"kb_used_data":6584,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754668748,"statfs":{"total":773094113280,"available":772780797952,"internally_reserved":0,"allocated":6742016,"data_stored":3479254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002204"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.239895+0000","last_change":"2024-01-16T15:05:20.238820+0000","last_active":"2024-01-16T15:05:20.239895+0000","last_peered":"2024-01-16T15:05:20.239895+0000","last_clean":"2024-01-16T15:05:20.239895+0000","last_became_active":"2024-01-16T15:05:20.238515+0000","last_became_peered":"2024-01-16T15:05:20.238515+0000","last_unstale":"2024-01-16T15:05:20.239895+0000","last_undegraded":"2024-01-16T15:05:20.239895+0000","last_fullsized":"2024-01-16T15:05:20.239895+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:11.361119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255504+0000","last_change":"2024-01-16T15:05:20.252971+0000","last_active":"2024-01-16T15:05:20.255504+0000","last_peered":"2024-01-16T15:05:20.255504+0000","last_clean":"2024-01-16T15:05:20.255504+0000","last_became_active":"2024-01-16T15:05:20.252446+0000","last_became_peered":"2024-01-16T15:05:20.252446+0000","last_unstale":"2024-01-16T15:05:20.255504+0000","last_undegraded":"2024-01-16T15:05:20.255504+0000","last_fullsized":"2024-01-16T15:05:20.255504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:59.541512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.236175+0000","last_change":"2024-01-16T15:05:20.229623+0000","last_active":"2024-01-16T15:05:20.236175+0000","last_peered":"2024-01-16T15:05:20.236175+0000","last_clean":"2024-01-16T15:05:20.236175+0000","last_became_active":"2024-01-16T15:05:20.229286+0000","last_became_peered":"2024-01-16T15:05:20.229286+0000","last_unstale":"2024-01-16T15:05:20.236175+0000","last_undegraded":"2024-01-16T15:05:20.236175+0000","last_fullsized":"2024-01-16T15:05:20.236175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:22.954805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242762+0000","last_change":"2024-01-16T15:05:20.237777+0000","last_active":"2024-01-16T15:05:20.242762+0000","last_peered":"2024-01-16T15:05:20.242762+0000","last_clean":"2024-01-16T15:05:20.242762+0000","last_became_active":"2024-01-16T15:05:20.237515+0000","last_became_peered":"2024-01-16T15:05:20.237515+0000","last_unstale":"2024-01-16T15:05:20.242762+0000","last_undegraded":"2024-01-16T15:05:20.242762+0000","last_fullsized":"2024-01-16T15:05:20.242762+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:37.705713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.235458+0000","last_change":"2024-01-16T15:05:20.233436+0000","last_active":"2024-01-16T15:05:20.235458+0000","last_peered":"2024-01-16T15:05:20.235458+0000","last_clean":"2024-01-16T15:05:20.235458+0000","last_became_active":"2024-01-16T15:05:20.230693+0000","last_became_peered":"2024-01-16T15:05:20.230693+0000","last_unstale":"2024-01-16T15:05:20.235458+0000","last_undegraded":"2024-01-16T15:05:20.235458+0000","last_fullsized":"2024-01-16T15:05:20.235458+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:41.663640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.241921+0000","last_change":"2024-01-16T15:05:20.235127+0000","last_active":"2024-01-16T15:05:20.241921+0000","last_peered":"2024-01-16T15:05:20.241921+0000","last_clean":"2024-01-16T15:05:20.241921+0000","last_became_active":"2024-01-16T15:05:20.234852+0000","last_became_peered":"2024-01-16T15:05:20.234852+0000","last_unstale":"2024-01-16T15:05:20.241921+0000","last_undegraded":"2024-01-16T15:05:20.241921+0000","last_fullsized":"2024-01-16T15:05:20.241921+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:13:12.014658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216642+0000","last_change":"2024-01-16T15:05:20.240118+0000","last_active":"2024-01-16T15:05:21.216642+0000","last_peered":"2024-01-16T15:05:21.216642+0000","last_clean":"2024-01-16T15:05:21.216642+0000","last_became_active":"2024-01-16T15:05:20.239664+0000","last_became_peered":"2024-01-16T15:05:20.239664+0000","last_unstale":"2024-01-16T15:05:21.216642+0000","last_undegraded":"2024-01-16T15:05:21.216642+0000","last_fullsized":"2024-01-16T15:05:21.216642+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:28.378083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.238625+0000","last_change":"2024-01-16T15:05:20.232961+0000","last_active":"2024-01-16T15:05:20.238625+0000","last_peered":"2024-01-16T15:05:20.238625+0000","last_clean":"2024-01-16T15:05:20.238625+0000","last_became_active":"2024-01-16T15:05:20.232681+0000","last_became_peered":"2024-01-16T15:05:20.232681+0000","last_unstale":"2024-01-16T15:05:20.238625+0000","last_undegraded":"2024-01-16T15:05:20.238625+0000","last_fullsized":"2024-01-16T15:05:20.238625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:22.874597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.218944+0000","last_change":"2024-01-16T15:05:20.240572+0000","last_active":"2024-01-16T15:05:21.218944+0000","last_peered":"2024-01-16T15:05:21.218944+0000","last_clean":"2024-01-16T15:05:21.218944+0000","last_became_active":"2024-01-16T15:05:20.240296+0000","last_became_peered":"2024-01-16T15:05:20.240296+0000","last_unstale":"2024-01-16T15:05:21.218944+0000","last_undegraded":"2024-01-16T15:05:21.218944+0000","last_fullsized":"2024-01-16T15:05:21.218944+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:55.299913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.248030+0000","last_change":"2024-01-16T15:05:20.243791+0000","last_active":"2024-01-16T15:05:20.248030+0000","last_peered":"2024-01-16T15:05:20.248030+0000","last_clean":"2024-01-16T15:05:20.248030+0000","last_became_active":"2024-01-16T15:05:20.243513+0000","last_became_peered":"2024-01-16T15:05:20.243513+0000","last_unstale":"2024-01-16T15:05:20.248030+0000","last_undegraded":"2024-01-16T15:05:20.248030+0000","last_fullsized":"2024-01-16T15:05:20.248030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:12.737053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255293+0000","last_change":"2024-01-16T15:05:20.253744+0000","last_active":"2024-01-16T15:05:20.255293+0000","last_peered":"2024-01-16T15:05:20.255293+0000","last_clean":"2024-01-16T15:05:20.255293+0000","last_became_active":"2024-01-16T15:05:20.253426+0000","last_became_peered":"2024-01-16T15:05:20.253426+0000","last_unstale":"2024-01-16T15:05:20.255293+0000","last_undegraded":"2024-01-16T15:05:20.255293+0000","last_fullsized":"2024-01-16T15:05:20.255293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:17.634683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.240102+0000","last_change":"2024-01-16T15:05:20.232550+0000","last_active":"2024-01-16T15:05:20.240102+0000","last_peered":"2024-01-16T15:05:20.240102+0000","last_clean":"2024-01-16T15:05:20.240102+0000","last_became_active":"2024-01-16T15:05:20.232293+0000","last_became_peered":"2024-01-16T15:05:20.232293+0000","last_unstale":"2024-01-16T15:05:20.240102+0000","last_undegraded":"2024-01-16T15:05:20.240102+0000","last_fullsized":"2024-01-16T15:05:20.240102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:35.867624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423576+0000","last_change":"2024-01-16T15:05:20.235880+0000","last_active":"2024-01-16T15:05:21.423576+0000","last_peered":"2024-01-16T15:05:21.423576+0000","last_clean":"2024-01-16T15:05:21.423576+0000","last_became_active":"2024-01-16T15:05:20.235577+0000","last_became_peered":"2024-01-16T15:05:20.235577+0000","last_unstale":"2024-01-16T15:05:21.423576+0000","last_undegraded":"2024-01-16T15:05:21.423576+0000","last_fullsized":"2024-01-16T15:05:21.423576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:05.263225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216944+0000","last_change":"2024-01-16T15:05:20.241773+0000","last_active":"2024-01-16T15:05:21.216944+0000","last_peered":"2024-01-16T15:05:21.216944+0000","last_clean":"2024-01-16T15:05:21.216944+0000","last_became_active":"2024-01-16T15:05:20.241269+0000","last_became_peered":"2024-01-16T15:05:20.241269+0000","last_unstale":"2024-01-16T15:05:21.216944+0000","last_undegraded":"2024-01-16T15:05:21.216944+0000","last_fullsized":"2024-01-16T15:05:21.216944+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:10:03.363520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216805+0000","last_change":"2024-01-16T15:05:20.247822+0000","last_active":"2024-01-16T15:05:21.216805+0000","last_peered":"2024-01-16T15:05:21.216805+0000","last_clean":"2024-01-16T15:05:21.216805+0000","last_became_active":"2024-01-16T15:05:20.247567+0000","last_became_peered":"2024-01-16T15:05:20.247567+0000","last_unstale":"2024-01-16T15:05:21.216805+0000","last_undegraded":"2024-01-16T15:05:21.216805+0000","last_fullsized":"2024-01-16T15:05:21.216805+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:39.169632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.233523+0000","last_change":"2024-01-16T15:05:20.229308+0000","last_active":"2024-01-16T15:05:20.233523+0000","last_peered":"2024-01-16T15:05:20.233523+0000","last_clean":"2024-01-16T15:05:20.233523+0000","last_became_active":"2024-01-16T15:05:20.228992+0000","last_became_peered":"2024-01-16T15:05:20.228992+0000","last_unstale":"2024-01-16T15:05:20.233523+0000","last_undegraded":"2024-01-16T15:05:20.233523+0000","last_fullsized":"2024-01-16T15:05:20.233523+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:35.158758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252068+0000","last_change":"2024-01-16T15:05:20.247343+0000","last_active":"2024-01-16T15:05:20.252068+0000","last_peered":"2024-01-16T15:05:20.252068+0000","last_clean":"2024-01-16T15:05:20.252068+0000","last_became_active":"2024-01-16T15:05:20.238700+0000","last_became_peered":"2024-01-16T15:05:20.238700+0000","last_unstale":"2024-01-16T15:05:20.252068+0000","last_undegraded":"2024-01-16T15:05:20.252068+0000","last_fullsized":"2024-01-16T15:05:20.252068+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:18.897950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243266+0000","last_change":"2024-01-16T15:05:20.241942+0000","last_active":"2024-01-16T15:05:20.243266+0000","last_peered":"2024-01-16T15:05:20.243266+0000","last_clean":"2024-01-16T15:05:20.243266+0000","last_became_active":"2024-01-16T15:05:20.241666+0000","last_became_peered":"2024-01-16T15:05:20.241666+0000","last_unstale":"2024-01-16T15:05:20.243266+0000","last_undegraded":"2024-01-16T15:05:20.243266+0000","last_fullsized":"2024-01-16T15:05:20.243266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:50.528201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.239640+0000","last_change":"2024-01-16T15:05:20.238255+0000","last_active":"2024-01-16T15:05:20.239640+0000","last_peered":"2024-01-16T15:05:20.239640+0000","last_clean":"2024-01-16T15:05:20.239640+0000","last_became_active":"2024-01-16T15:05:20.237815+0000","last_became_peered":"2024-01-16T15:05:20.237815+0000","last_unstale":"2024-01-16T15:05:20.239640+0000","last_undegraded":"2024-01-16T15:05:20.239640+0000","last_fullsized":"2024-01-16T15:05:20.239640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:10:04.890676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253079+0000","last_change":"2024-01-16T15:05:20.248596+0000","last_active":"2024-01-16T15:05:20.253079+0000","last_peered":"2024-01-16T15:05:20.253079+0000","last_clean":"2024-01-16T15:05:20.253079+0000","last_became_active":"2024-01-16T15:05:20.248097+0000","last_became_peered":"2024-01-16T15:05:20.248097+0000","last_unstale":"2024-01-16T15:05:20.253079+0000","last_undegraded":"2024-01-16T15:05:20.253079+0000","last_fullsized":"2024-01-16T15:05:20.253079+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:54.977369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242445+0000","last_change":"2024-01-16T15:05:20.238168+0000","last_active":"2024-01-16T15:05:20.242445+0000","last_peered":"2024-01-16T15:05:20.242445+0000","last_clean":"2024-01-16T15:05:20.242445+0000","last_became_active":"2024-01-16T15:05:20.237874+0000","last_became_peered":"2024-01-16T15:05:20.237874+0000","last_unstale":"2024-01-16T15:05:20.242445+0000","last_undegraded":"2024-01-16T15:05:20.242445+0000","last_fullsized":"2024-01-16T15:05:20.242445+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:37.192618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243650+0000","last_change":"2024-01-16T15:05:20.238080+0000","last_active":"2024-01-16T15:05:20.243650+0000","last_peered":"2024-01-16T15:05:20.243650+0000","last_clean":"2024-01-16T15:05:20.243650+0000","last_became_active":"2024-01-16T15:05:20.237391+0000","last_became_peered":"2024-01-16T15:05:20.237391+0000","last_unstale":"2024-01-16T15:05:20.243650+0000","last_undegraded":"2024-01-16T15:05:20.243650+0000","last_fullsized":"2024-01-16T15:05:20.243650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:08.900121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250757+0000","last_change":"2024-01-16T15:05:20.245358+0000","last_active":"2024-01-16T15:05:20.250757+0000","last_peered":"2024-01-16T15:05:20.250757+0000","last_clean":"2024-01-16T15:05:20.250757+0000","last_became_active":"2024-01-16T15:05:20.244029+0000","last_became_peered":"2024-01-16T15:05:20.244029+0000","last_unstale":"2024-01-16T15:05:20.250757+0000","last_undegraded":"2024-01-16T15:05:20.250757+0000","last_fullsized":"2024-01-16T15:05:20.250757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:13:12.014658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423422+0000","last_change":"2024-01-16T15:05:20.233598+0000","last_active":"2024-01-16T15:05:21.423422+0000","last_peered":"2024-01-16T15:05:21.423422+0000","last_clean":"2024-01-16T15:05:21.423422+0000","last_became_active":"2024-01-16T15:05:20.232505+0000","last_became_peered":"2024-01-16T15:05:20.232505+0000","last_unstale":"2024-01-16T15:05:21.423422+0000","last_undegraded":"2024-01-16T15:05:21.423422+0000","last_fullsized":"2024-01-16T15:05:21.423422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:16:27.966705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252691+0000","last_change":"2024-01-16T15:05:20.249771+0000","last_active":"2024-01-16T15:05:20.252691+0000","last_peered":"2024-01-16T15:05:20.252691+0000","last_clean":"2024-01-16T15:05:20.252691+0000","last_became_active":"2024-01-16T15:05:20.249479+0000","last_became_peered":"2024-01-16T15:05:20.249479+0000","last_unstale":"2024-01-16T15:05:20.252691+0000","last_undegraded":"2024-01-16T15:05:20.252691+0000","last_fullsized":"2024-01-16T15:05:20.252691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:06.994033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242781+0000","last_change":"2024-01-16T15:05:20.237332+0000","last_active":"2024-01-16T15:05:20.242781+0000","last_peered":"2024-01-16T15:05:20.242781+0000","last_clean":"2024-01-16T15:05:20.242781+0000","last_became_active":"2024-01-16T15:05:20.236885+0000","last_became_peered":"2024-01-16T15:05:20.236885+0000","last_unstale":"2024-01-16T15:05:20.242781+0000","last_undegraded":"2024-01-16T15:05:20.242781+0000","last_fullsized":"2024-01-16T15:05:20.242781+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:22.954805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242172+0000","last_change":"2024-01-16T15:05:20.236199+0000","last_active":"2024-01-16T15:05:20.242172+0000","last_peered":"2024-01-16T15:05:20.242172+0000","last_clean":"2024-01-16T15:05:20.242172+0000","last_became_active":"2024-01-16T15:05:20.235919+0000","last_became_peered":"2024-01-16T15:05:20.235919+0000","last_unstale":"2024-01-16T15:05:20.242172+0000","last_undegraded":"2024-01-16T15:05:20.242172+0000","last_fullsized":"2024-01-16T15:05:20.242172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:26.943440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247151+0000","last_change":"2024-01-16T15:05:20.242282+0000","last_active":"2024-01-16T15:05:20.247151+0000","last_peered":"2024-01-16T15:05:20.247151+0000","last_clean":"2024-01-16T15:05:20.247151+0000","last_became_active":"2024-01-16T15:05:20.241794+0000","last_became_peered":"2024-01-16T15:05:20.241794+0000","last_unstale":"2024-01-16T15:05:20.247151+0000","last_undegraded":"2024-01-16T15:05:20.247151+0000","last_fullsized":"2024-01-16T15:05:20.247151+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:45.588041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252049+0000","last_change":"2024-01-16T15:05:20.246393+0000","last_active":"2024-01-16T15:05:20.252049+0000","last_peered":"2024-01-16T15:05:20.252049+0000","last_clean":"2024-01-16T15:05:20.252049+0000","last_became_active":"2024-01-16T15:05:20.245756+0000","last_became_peered":"2024-01-16T15:05:20.245756+0000","last_unstale":"2024-01-16T15:05:20.252049+0000","last_undegraded":"2024-01-16T15:05:20.252049+0000","last_fullsized":"2024-01-16T15:05:20.252049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:01.117429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.235517+0000","last_change":"2024-01-16T15:05:20.229665+0000","last_active":"2024-01-16T15:05:20.235517+0000","last_peered":"2024-01-16T15:05:20.235517+0000","last_clean":"2024-01-16T15:05:20.235517+0000","last_became_active":"2024-01-16T15:05:20.229359+0000","last_became_peered":"2024-01-16T15:05:20.229359+0000","last_unstale":"2024-01-16T15:05:20.235517+0000","last_undegraded":"2024-01-16T15:05:20.235517+0000","last_fullsized":"2024-01-16T15:05:20.235517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:45:38.160267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253377+0000","last_change":"2024-01-16T15:05:20.251101+0000","last_active":"2024-01-16T15:05:20.253377+0000","last_peered":"2024-01-16T15:05:20.253377+0000","last_clean":"2024-01-16T15:05:20.253377+0000","last_became_active":"2024-01-16T15:05:20.250798+0000","last_became_peered":"2024-01-16T15:05:20.250798+0000","last_unstale":"2024-01-16T15:05:20.253377+0000","last_undegraded":"2024-01-16T15:05:20.253377+0000","last_fullsized":"2024-01-16T15:05:20.253377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:54.235722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253095+0000","last_change":"2024-01-16T15:05:20.245790+0000","last_active":"2024-01-16T15:05:20.253095+0000","last_peered":"2024-01-16T15:05:20.253095+0000","last_clean":"2024-01-16T15:05:20.253095+0000","last_became_active":"2024-01-16T15:05:20.244190+0000","last_became_peered":"2024-01-16T15:05:20.244190+0000","last_unstale":"2024-01-16T15:05:20.253095+0000","last_undegraded":"2024-01-16T15:05:20.253095+0000","last_fullsized":"2024-01-16T15:05:20.253095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:37.192618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252064+0000","last_change":"2024-01-16T15:05:20.240511+0000","last_active":"2024-01-16T15:05:20.252064+0000","last_peered":"2024-01-16T15:05:20.252064+0000","last_clean":"2024-01-16T15:05:20.252064+0000","last_became_active":"2024-01-16T15:05:20.240166+0000","last_became_peered":"2024-01-16T15:05:20.240166+0000","last_unstale":"2024-01-16T15:05:20.252064+0000","last_undegraded":"2024-01-16T15:05:20.252064+0000","last_fullsized":"2024-01-16T15:05:20.252064+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:57.672017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"59'4","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.245332+0000","last_change":"2024-01-16T15:05:19.227915+0000","last_active":"2024-01-16T15:05:20.245332+0000","last_peered":"2024-01-16T15:05:20.245332+0000","last_clean":"2024-01-16T15:05:20.245332+0000","last_became_active":"2024-01-16T15:05:19.227530+0000","last_became_peered":"2024-01-16T15:05:19.227530+0000","last_unstale":"2024-01-16T15:05:20.245332+0000","last_undegraded":"2024-01-16T15:05:20.245332+0000","last_fullsized":"2024-01-16T15:05:20.245332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:49.516558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247952+0000","last_change":"2024-01-16T15:05:20.243233+0000","last_active":"2024-01-16T15:05:20.247952+0000","last_peered":"2024-01-16T15:05:20.247952+0000","last_clean":"2024-01-16T15:05:20.247952+0000","last_became_active":"2024-01-16T15:05:20.242946+0000","last_became_peered":"2024-01-16T15:05:20.242946+0000","last_unstale":"2024-01-16T15:05:20.247952+0000","last_undegraded":"2024-01-16T15:05:20.247952+0000","last_fullsized":"2024-01-16T15:05:20.247952+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:10.475751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.251485+0000","last_change":"2024-01-16T15:05:19.235007+0000","last_active":"2024-01-16T15:05:20.251485+0000","last_peered":"2024-01-16T15:05:20.251485+0000","last_clean":"2024-01-16T15:05:20.251485+0000","last_became_active":"2024-01-16T15:05:19.234525+0000","last_became_peered":"2024-01-16T15:05:19.234525+0000","last_unstale":"2024-01-16T15:05:20.251485+0000","last_undegraded":"2024-01-16T15:05:20.251485+0000","last_fullsized":"2024-01-16T15:05:20.251485+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:58.597011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.245915+0000","last_change":"2024-01-16T15:05:20.240863+0000","last_active":"2024-01-16T15:05:20.245915+0000","last_peered":"2024-01-16T15:05:20.245915+0000","last_clean":"2024-01-16T15:05:20.245915+0000","last_became_active":"2024-01-16T15:05:20.240048+0000","last_became_peered":"2024-01-16T15:05:20.240048+0000","last_unstale":"2024-01-16T15:05:20.245915+0000","last_undegraded":"2024-01-16T15:05:20.245915+0000","last_fullsized":"2024-01-16T15:05:20.245915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:51.432970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.246947+0000","last_change":"2024-01-16T15:05:19.227210+0000","last_active":"2024-01-16T15:05:20.246947+0000","last_peered":"2024-01-16T15:05:20.246947+0000","last_clean":"2024-01-16T15:05:20.246947+0000","last_became_active":"2024-01-16T15:05:19.224759+0000","last_became_peered":"2024-01-16T15:05:19.224759+0000","last_unstale":"2024-01-16T15:05:20.246947+0000","last_undegraded":"2024-01-16T15:05:20.246947+0000","last_fullsized":"2024-01-16T15:05:20.246947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:12.268974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242247+0000","last_change":"2024-01-16T15:05:20.238501+0000","last_active":"2024-01-16T15:05:20.242247+0000","last_peered":"2024-01-16T15:05:20.242247+0000","last_clean":"2024-01-16T15:05:20.242247+0000","last_became_active":"2024-01-16T15:05:20.238193+0000","last_became_peered":"2024-01-16T15:05:20.238193+0000","last_unstale":"2024-01-16T15:05:20.242247+0000","last_undegraded":"2024-01-16T15:05:20.242247+0000","last_fullsized":"2024-01-16T15:05:20.242247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:32.160025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.244067+0000","last_change":"2024-01-16T15:05:19.231086+0000","last_active":"2024-01-16T15:05:20.244067+0000","last_peered":"2024-01-16T15:05:20.244067+0000","last_clean":"2024-01-16T15:05:20.244067+0000","last_became_active":"2024-01-16T15:05:19.230818+0000","last_became_peered":"2024-01-16T15:05:19.230818+0000","last_unstale":"2024-01-16T15:05:20.244067+0000","last_undegraded":"2024-01-16T15:05:20.244067+0000","last_fullsized":"2024-01-16T15:05:20.244067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:31.148382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243671+0000","last_change":"2024-01-16T15:05:20.238283+0000","last_active":"2024-01-16T15:05:20.243671+0000","last_peered":"2024-01-16T15:05:20.243671+0000","last_clean":"2024-01-16T15:05:20.243671+0000","last_became_active":"2024-01-16T15:05:20.237795+0000","last_became_peered":"2024-01-16T15:05:20.237795+0000","last_unstale":"2024-01-16T15:05:20.243671+0000","last_undegraded":"2024-01-16T15:05:20.243671+0000","last_fullsized":"2024-01-16T15:05:20.243671+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:43.007200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"59'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.217353+0000","last_change":"2024-01-16T15:05:19.227138+0000","last_active":"2024-01-16T15:05:21.217353+0000","last_peered":"2024-01-16T15:05:21.217353+0000","last_clean":"2024-01-16T15:05:21.217353+0000","last_became_active":"2024-01-16T15:05:19.226552+0000","last_became_peered":"2024-01-16T15:05:19.226552+0000","last_unstale":"2024-01-16T15:05:21.217353+0000","last_undegraded":"2024-01-16T15:05:21.217353+0000","last_fullsized":"2024-01-16T15:05:21.217353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:35.343390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252107+0000","last_change":"2024-01-16T15:05:20.247345+0000","last_active":"2024-01-16T15:05:20.252107+0000","last_peered":"2024-01-16T15:05:20.252107+0000","last_clean":"2024-01-16T15:05:20.252107+0000","last_became_active":"2024-01-16T15:05:20.246041+0000","last_became_peered":"2024-01-16T15:05:20.246041+0000","last_unstale":"2024-01-16T15:05:20.252107+0000","last_undegraded":"2024-01-16T15:05:20.252107+0000","last_fullsized":"2024-01-16T15:05:20.252107+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:10.117242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.240583+0000","last_change":"2024-01-16T15:05:19.226406+0000","last_active":"2024-01-16T15:05:20.240583+0000","last_peered":"2024-01-16T15:05:20.240583+0000","last_clean":"2024-01-16T15:05:20.240583+0000","last_became_active":"2024-01-16T15:05:19.226188+0000","last_became_peered":"2024-01-16T15:05:19.226188+0000","last_unstale":"2024-01-16T15:05:20.240583+0000","last_undegraded":"2024-01-16T15:05:20.240583+0000","last_fullsized":"2024-01-16T15:05:20.240583+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:39.002852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423689+0000","last_change":"2024-01-16T15:05:20.218703+0000","last_active":"2024-01-16T15:05:21.423689+0000","last_peered":"2024-01-16T15:05:21.423689+0000","last_clean":"2024-01-16T15:05:21.423689+0000","last_became_active":"2024-01-16T15:05:20.217556+0000","last_became_peered":"2024-01-16T15:05:20.217556+0000","last_unstale":"2024-01-16T15:05:21.423689+0000","last_undegraded":"2024-01-16T15:05:21.423689+0000","last_fullsized":"2024-01-16T15:05:21.423689+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:10:40.745856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.224878+0000","last_change":"2024-01-16T15:05:19.230317+0000","last_active":"2024-01-16T15:05:20.224878+0000","last_peered":"2024-01-16T15:05:20.224878+0000","last_clean":"2024-01-16T15:05:20.224878+0000","last_became_active":"2024-01-16T15:05:19.228514+0000","last_became_peered":"2024-01-16T15:05:19.228514+0000","last_unstale":"2024-01-16T15:05:20.224878+0000","last_undegraded":"2024-01-16T15:05:20.224878+0000","last_fullsized":"2024-01-16T15:05:20.224878+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:34.147115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.238855+0000","last_change":"2024-01-16T15:05:20.234236+0000","last_active":"2024-01-16T15:05:20.238855+0000","last_peered":"2024-01-16T15:05:20.238855+0000","last_clean":"2024-01-16T15:05:20.238855+0000","last_became_active":"2024-01-16T15:05:20.232878+0000","last_became_peered":"2024-01-16T15:05:20.232878+0000","last_unstale":"2024-01-16T15:05:20.238855+0000","last_undegraded":"2024-01-16T15:05:20.238855+0000","last_fullsized":"2024-01-16T15:05:20.238855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:23.127401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216459+0000","last_change":"2024-01-16T15:05:19.229933+0000","last_active":"2024-01-16T15:05:21.216459+0000","last_peered":"2024-01-16T15:05:21.216459+0000","last_clean":"2024-01-16T15:05:21.216459+0000","last_became_active":"2024-01-16T15:05:19.229571+0000","last_became_peered":"2024-01-16T15:05:19.229571+0000","last_unstale":"2024-01-16T15:05:21.216459+0000","last_undegraded":"2024-01-16T15:05:21.216459+0000","last_fullsized":"2024-01-16T15:05:21.216459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:29:22.696249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.218944+0000","last_change":"2024-01-16T15:05:20.246391+0000","last_active":"2024-01-16T15:05:21.218944+0000","last_peered":"2024-01-16T15:05:21.218944+0000","last_clean":"2024-01-16T15:05:21.218944+0000","last_became_active":"2024-01-16T15:05:20.245765+0000","last_became_peered":"2024-01-16T15:05:20.245765+0000","last_unstale":"2024-01-16T15:05:21.218944+0000","last_undegraded":"2024-01-16T15:05:21.218944+0000","last_fullsized":"2024-01-16T15:05:21.218944+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:52.207259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.214830+0000","last_change":"2024-01-16T15:05:19.224716+0000","last_active":"2024-01-16T15:05:20.214830+0000","last_peered":"2024-01-16T15:05:20.214830+0000","last_clean":"2024-01-16T15:05:20.214830+0000","last_became_active":"2024-01-16T15:05:19.223449+0000","last_became_peered":"2024-01-16T15:05:19.223449+0000","last_unstale":"2024-01-16T15:05:20.214830+0000","last_undegraded":"2024-01-16T15:05:20.214830+0000","last_fullsized":"2024-01-16T15:05:20.214830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:37.070315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242650+0000","last_change":"2024-01-16T15:05:20.241259+0000","last_active":"2024-01-16T15:05:20.242650+0000","last_peered":"2024-01-16T15:05:20.242650+0000","last_clean":"2024-01-16T15:05:20.242650+0000","last_became_active":"2024-01-16T15:05:20.240922+0000","last_became_peered":"2024-01-16T15:05:20.240922+0000","last_unstale":"2024-01-16T15:05:20.242650+0000","last_undegraded":"2024-01-16T15:05:20.242650+0000","last_fullsized":"2024-01-16T15:05:20.242650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:32.160025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.232920+0000","last_change":"2024-01-16T15:05:19.229019+0000","last_active":"2024-01-16T15:05:20.232920+0000","last_peered":"2024-01-16T15:05:20.232920+0000","last_clean":"2024-01-16T15:05:20.232920+0000","last_became_active":"2024-01-16T15:05:19.228725+0000","last_became_peered":"2024-01-16T15:05:19.228725+0000","last_unstale":"2024-01-16T15:05:20.232920+0000","last_undegraded":"2024-01-16T15:05:20.232920+0000","last_fullsized":"2024-01-16T15:05:20.232920+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:00:08.760888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.254884+0000","last_change":"2024-01-16T15:05:20.251459+0000","last_active":"2024-01-16T15:05:20.254884+0000","last_peered":"2024-01-16T15:05:20.254884+0000","last_clean":"2024-01-16T15:05:20.254884+0000","last_became_active":"2024-01-16T15:05:20.251194+0000","last_became_peered":"2024-01-16T15:05:20.251194+0000","last_unstale":"2024-01-16T15:05:20.254884+0000","last_undegraded":"2024-01-16T15:05:20.254884+0000","last_fullsized":"2024-01-16T15:05:20.254884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:03.339363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"59'3","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247432+0000","last_change":"2024-01-16T15:05:19.225885+0000","last_active":"2024-01-16T15:05:20.247432+0000","last_peered":"2024-01-16T15:05:20.247432+0000","last_clean":"2024-01-16T15:05:20.247432+0000","last_became_active":"2024-01-16T15:05:19.225376+0000","last_became_peered":"2024-01-16T15:05:19.225376+0000","last_unstale":"2024-01-16T15:05:20.247432+0000","last_undegraded":"2024-01-16T15:05:20.247432+0000","last_fullsized":"2024-01-16T15:05:20.247432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:50.421327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242822+0000","last_change":"2024-01-16T15:05:20.240925+0000","last_active":"2024-01-16T15:05:20.242822+0000","last_peered":"2024-01-16T15:05:20.242822+0000","last_clean":"2024-01-16T15:05:20.242822+0000","last_became_active":"2024-01-16T15:05:20.240024+0000","last_became_peered":"2024-01-16T15:05:20.240024+0000","last_unstale":"2024-01-16T15:05:20.242822+0000","last_undegraded":"2024-01-16T15:05:20.242822+0000","last_fullsized":"2024-01-16T15:05:20.242822+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:48.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"59'6","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.218036+0000","last_change":"2024-01-16T15:05:19.216746+0000","last_active":"2024-01-16T15:05:21.218036+0000","last_peered":"2024-01-16T15:05:21.218036+0000","last_clean":"2024-01-16T15:05:21.218036+0000","last_became_active":"2024-01-16T15:05:19.216359+0000","last_became_peered":"2024-01-16T15:05:19.216359+0000","last_unstale":"2024-01-16T15:05:21.218036+0000","last_undegraded":"2024-01-16T15:05:21.218036+0000","last_fullsized":"2024-01-16T15:05:21.218036+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:35.458895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253314+0000","last_change":"2024-01-16T15:05:20.248867+0000","last_active":"2024-01-16T15:05:20.253314+0000","last_peered":"2024-01-16T15:05:20.253314+0000","last_clean":"2024-01-16T15:05:20.253314+0000","last_became_active":"2024-01-16T15:05:20.248277+0000","last_became_peered":"2024-01-16T15:05:20.248277+0000","last_unstale":"2024-01-16T15:05:20.253314+0000","last_undegraded":"2024-01-16T15:05:20.253314+0000","last_fullsized":"2024-01-16T15:05:20.253314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:50.528201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255602+0000","last_change":"2024-01-16T15:05:19.227725+0000","last_active":"2024-01-16T15:05:20.255602+0000","last_peered":"2024-01-16T15:05:20.255602+0000","last_clean":"2024-01-16T15:05:20.255602+0000","last_became_active":"2024-01-16T15:05:19.227387+0000","last_became_peered":"2024-01-16T15:05:19.227387+0000","last_unstale":"2024-01-16T15:05:20.255602+0000","last_undegraded":"2024-01-16T15:05:20.255602+0000","last_fullsized":"2024-01-16T15:05:20.255602+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:04.014231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.222750+0000","last_change":"2024-01-16T15:05:20.217677+0000","last_active":"2024-01-16T15:05:20.222750+0000","last_peered":"2024-01-16T15:05:20.222750+0000","last_clean":"2024-01-16T15:05:20.222750+0000","last_became_active":"2024-01-16T15:05:20.217092+0000","last_became_peered":"2024-01-16T15:05:20.217092+0000","last_unstale":"2024-01-16T15:05:20.222750+0000","last_undegraded":"2024-01-16T15:05:20.222750+0000","last_fullsized":"2024-01-16T15:05:20.222750+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:37.705713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423137+0000","last_change":"2024-01-16T15:05:19.223095+0000","last_active":"2024-01-16T15:05:21.423137+0000","last_peered":"2024-01-16T15:05:21.423137+0000","last_clean":"2024-01-16T15:05:21.423137+0000","last_became_active":"2024-01-16T15:05:19.222815+0000","last_became_peered":"2024-01-16T15:05:19.222815+0000","last_unstale":"2024-01-16T15:05:21.423137+0000","last_undegraded":"2024-01-16T15:05:21.423137+0000","last_fullsized":"2024-01-16T15:05:21.423137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:56.417205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216372+0000","last_change":"2024-01-16T15:05:20.236853+0000","last_active":"2024-01-16T15:05:21.216372+0000","last_peered":"2024-01-16T15:05:21.216372+0000","last_clean":"2024-01-16T15:05:21.216372+0000","last_became_active":"2024-01-16T15:05:20.236400+0000","last_became_peered":"2024-01-16T15:05:20.236400+0000","last_unstale":"2024-01-16T15:05:21.216372+0000","last_undegraded":"2024-01-16T15:05:21.216372+0000","last_fullsized":"2024-01-16T15:05:21.216372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:14.926936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.225273+0000","last_change":"2024-01-16T15:05:19.231103+0000","last_active":"2024-01-16T15:05:20.225273+0000","last_peered":"2024-01-16T15:05:20.225273+0000","last_clean":"2024-01-16T15:05:20.225273+0000","last_became_active":"2024-01-16T15:05:19.230564+0000","last_became_peered":"2024-01-16T15:05:19.230564+0000","last_unstale":"2024-01-16T15:05:20.225273+0000","last_undegraded":"2024-01-16T15:05:20.225273+0000","last_fullsized":"2024-01-16T15:05:20.225273+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:13:11.003015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250778+0000","last_change":"2024-01-16T15:05:20.245528+0000","last_active":"2024-01-16T15:05:20.250778+0000","last_peered":"2024-01-16T15:05:20.250778+0000","last_clean":"2024-01-16T15:05:20.250778+0000","last_became_active":"2024-01-16T15:05:20.244419+0000","last_became_peered":"2024-01-16T15:05:20.244419+0000","last_unstale":"2024-01-16T15:05:20.250778+0000","last_undegraded":"2024-01-16T15:05:20.250778+0000","last_fullsized":"2024-01-16T15:05:20.250778+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.014495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"59'8","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.246667+0000","last_change":"2024-01-16T15:05:19.226677+0000","last_active":"2024-01-16T15:05:20.246667+0000","last_peered":"2024-01-16T15:05:20.246667+0000","last_clean":"2024-01-16T15:05:20.246667+0000","last_became_active":"2024-01-16T15:05:19.226289+0000","last_became_peered":"2024-01-16T15:05:19.226289+0000","last_unstale":"2024-01-16T15:05:20.246667+0000","last_undegraded":"2024-01-16T15:05:20.246667+0000","last_fullsized":"2024-01-16T15:05:20.246667+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:34.855981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.236111+0000","last_change":"2024-01-16T15:05:20.231030+0000","last_active":"2024-01-16T15:05:20.236111+0000","last_peered":"2024-01-16T15:05:20.236111+0000","last_clean":"2024-01-16T15:05:20.236111+0000","last_became_active":"2024-01-16T15:05:20.230741+0000","last_became_peered":"2024-01-16T15:05:20.230741+0000","last_unstale":"2024-01-16T15:05:20.236111+0000","last_undegraded":"2024-01-16T15:05:20.236111+0000","last_fullsized":"2024-01-16T15:05:20.236111+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:57.672017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255522+0000","last_change":"2024-01-16T15:05:19.234403+0000","last_active":"2024-01-16T15:05:20.255522+0000","last_peered":"2024-01-16T15:05:20.255522+0000","last_clean":"2024-01-16T15:05:20.255522+0000","last_became_active":"2024-01-16T15:05:19.230980+0000","last_became_peered":"2024-01-16T15:05:19.230980+0000","last_unstale":"2024-01-16T15:05:20.255522+0000","last_undegraded":"2024-01-16T15:05:20.255522+0000","last_fullsized":"2024-01-16T15:05:20.255522+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:21.416746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250675+0000","last_change":"2024-01-16T15:05:20.247941+0000","last_active":"2024-01-16T15:05:20.250675+0000","last_peered":"2024-01-16T15:05:20.250675+0000","last_clean":"2024-01-16T15:05:20.250675+0000","last_became_active":"2024-01-16T15:05:20.247645+0000","last_became_peered":"2024-01-16T15:05:20.247645+0000","last_unstale":"2024-01-16T15:05:20.250675+0000","last_undegraded":"2024-01-16T15:05:20.250675+0000","last_fullsized":"2024-01-16T15:05:20.250675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:37.192618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252000+0000","last_change":"2024-01-16T15:05:19.224765+0000","last_active":"2024-01-16T15:05:20.252000+0000","last_peered":"2024-01-16T15:05:20.252000+0000","last_clean":"2024-01-16T15:05:20.252000+0000","last_became_active":"2024-01-16T15:05:19.224394+0000","last_became_peered":"2024-01-16T15:05:19.224394+0000","last_unstale":"2024-01-16T15:05:20.252000+0000","last_undegraded":"2024-01-16T15:05:20.252000+0000","last_fullsized":"2024-01-16T15:05:20.252000+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:13.820308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253059+0000","last_change":"2024-01-16T15:05:20.248357+0000","last_active":"2024-01-16T15:05:20.253059+0000","last_peered":"2024-01-16T15:05:20.253059+0000","last_clean":"2024-01-16T15:05:20.253059+0000","last_became_active":"2024-01-16T15:05:20.247636+0000","last_became_peered":"2024-01-16T15:05:20.247636+0000","last_unstale":"2024-01-16T15:05:20.253059+0000","last_undegraded":"2024-01-16T15:05:20.253059+0000","last_fullsized":"2024-01-16T15:05:20.253059+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:21.447360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243269+0000","last_change":"2024-01-16T15:05:19.228334+0000","last_active":"2024-01-16T15:05:20.243269+0000","last_peered":"2024-01-16T15:05:20.243269+0000","last_clean":"2024-01-16T15:05:20.243269+0000","last_became_active":"2024-01-16T15:05:19.227858+0000","last_became_peered":"2024-01-16T15:05:19.227858+0000","last_unstale":"2024-01-16T15:05:20.243269+0000","last_undegraded":"2024-01-16T15:05:20.243269+0000","last_fullsized":"2024-01-16T15:05:20.243269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:37.070315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243688+0000","last_change":"2024-01-16T15:05:20.242055+0000","last_active":"2024-01-16T15:05:20.243688+0000","last_peered":"2024-01-16T15:05:20.243688+0000","last_clean":"2024-01-16T15:05:20.243688+0000","last_became_active":"2024-01-16T15:05:20.241731+0000","last_became_peered":"2024-01-16T15:05:20.241731+0000","last_unstale":"2024-01-16T15:05:20.243688+0000","last_undegraded":"2024-01-16T15:05:20.243688+0000","last_fullsized":"2024-01-16T15:05:20.243688+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:35.158758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.221282+0000","last_change":"2024-01-16T15:05:19.236671+0000","last_active":"2024-01-16T15:05:20.221282+0000","last_peered":"2024-01-16T15:05:20.221282+0000","last_clean":"2024-01-16T15:05:20.221282+0000","last_became_active":"2024-01-16T15:05:19.236332+0000","last_became_peered":"2024-01-16T15:05:19.236332+0000","last_unstale":"2024-01-16T15:05:20.221282+0000","last_undegraded":"2024-01-16T15:05:20.221282+0000","last_fullsized":"2024-01-16T15:05:20.221282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:30.621371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253332+0000","last_change":"2024-01-16T15:05:20.249108+0000","last_active":"2024-01-16T15:05:20.253332+0000","last_peered":"2024-01-16T15:05:20.253332+0000","last_clean":"2024-01-16T15:05:20.253332+0000","last_became_active":"2024-01-16T15:05:20.248623+0000","last_became_peered":"2024-01-16T15:05:20.248623+0000","last_unstale":"2024-01-16T15:05:20.253332+0000","last_undegraded":"2024-01-16T15:05:20.253332+0000","last_fullsized":"2024-01-16T15:05:20.253332+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:36.528292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.256400+0000","last_change":"2024-01-16T15:05:19.228416+0000","last_active":"2024-01-16T15:05:20.256400+0000","last_peered":"2024-01-16T15:05:20.256400+0000","last_clean":"2024-01-16T15:05:20.256400+0000","last_became_active":"2024-01-16T15:05:19.227029+0000","last_became_peered":"2024-01-16T15:05:19.227029+0000","last_unstale":"2024-01-16T15:05:20.256400+0000","last_undegraded":"2024-01-16T15:05:20.256400+0000","last_fullsized":"2024-01-16T15:05:20.256400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:08.433686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242355+0000","last_change":"2024-01-16T15:05:20.236188+0000","last_active":"2024-01-16T15:05:20.242355+0000","last_peered":"2024-01-16T15:05:20.242355+0000","last_clean":"2024-01-16T15:05:20.242355+0000","last_became_active":"2024-01-16T15:05:20.235807+0000","last_became_peered":"2024-01-16T15:05:20.235807+0000","last_unstale":"2024-01-16T15:05:20.242355+0000","last_undegraded":"2024-01-16T15:05:20.242355+0000","last_fullsized":"2024-01-16T15:05:20.242355+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:38.081958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.222052+0000","last_change":"2024-01-16T15:05:19.231277+0000","last_active":"2024-01-16T15:05:20.222052+0000","last_peered":"2024-01-16T15:05:20.222052+0000","last_clean":"2024-01-16T15:05:20.222052+0000","last_became_active":"2024-01-16T15:05:19.225403+0000","last_became_peered":"2024-01-16T15:05:19.225403+0000","last_unstale":"2024-01-16T15:05:20.222052+0000","last_undegraded":"2024-01-16T15:05:20.222052+0000","last_fullsized":"2024-01-16T15:05:20.222052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:13.484758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.232287+0000","last_change":"2024-01-16T15:05:20.226031+0000","last_active":"2024-01-16T15:05:20.232287+0000","last_peered":"2024-01-16T15:05:20.232287+0000","last_clean":"2024-01-16T15:05:20.232287+0000","last_became_active":"2024-01-16T15:05:20.225735+0000","last_became_peered":"2024-01-16T15:05:20.225735+0000","last_unstale":"2024-01-16T15:05:20.232287+0000","last_undegraded":"2024-01-16T15:05:20.232287+0000","last_fullsized":"2024-01-16T15:05:20.232287+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:48.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.225648+0000","last_change":"2024-01-16T15:05:19.237578+0000","last_active":"2024-01-16T15:05:20.225648+0000","last_peered":"2024-01-16T15:05:20.225648+0000","last_clean":"2024-01-16T15:05:20.225648+0000","last_became_active":"2024-01-16T15:05:19.237373+0000","last_became_peered":"2024-01-16T15:05:19.237373+0000","last_unstale":"2024-01-16T15:05:20.225648+0000","last_undegraded":"2024-01-16T15:05:20.225648+0000","last_fullsized":"2024-01-16T15:05:20.225648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:12.419974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242621+0000","last_change":"2024-01-16T15:05:20.237130+0000","last_active":"2024-01-16T15:05:20.242621+0000","last_peered":"2024-01-16T15:05:20.242621+0000","last_clean":"2024-01-16T15:05:20.242621+0000","last_became_active":"2024-01-16T15:05:20.236569+0000","last_became_peered":"2024-01-16T15:05:20.236569+0000","last_unstale":"2024-01-16T15:05:20.242621+0000","last_undegraded":"2024-01-16T15:05:20.242621+0000","last_fullsized":"2024-01-16T15:05:20.242621+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:45:38.160267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.254344+0000","last_change":"2024-01-16T15:05:19.235333+0000","last_active":"2024-01-16T15:05:20.254344+0000","last_peered":"2024-01-16T15:05:20.254344+0000","last_clean":"2024-01-16T15:05:20.254344+0000","last_became_active":"2024-01-16T15:05:19.235056+0000","last_became_peered":"2024-01-16T15:05:19.235056+0000","last_unstale":"2024-01-16T15:05:20.254344+0000","last_undegraded":"2024-01-16T15:05:20.254344+0000","last_fullsized":"2024-01-16T15:05:20.254344+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:35.962422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242721+0000","last_change":"2024-01-16T15:05:20.237813+0000","last_active":"2024-01-16T15:05:20.242721+0000","last_peered":"2024-01-16T15:05:20.242721+0000","last_clean":"2024-01-16T15:05:20.242721+0000","last_became_active":"2024-01-16T15:05:20.237005+0000","last_became_peered":"2024-01-16T15:05:20.237005+0000","last_unstale":"2024-01-16T15:05:20.242721+0000","last_undegraded":"2024-01-16T15:05:20.242721+0000","last_fullsized":"2024-01-16T15:05:20.242721+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:48.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.221674+0000","last_change":"2024-01-16T15:05:19.226502+0000","last_active":"2024-01-16T15:05:20.221674+0000","last_peered":"2024-01-16T15:05:20.221674+0000","last_clean":"2024-01-16T15:05:20.221674+0000","last_became_active":"2024-01-16T15:05:19.226217+0000","last_became_peered":"2024-01-16T15:05:19.226217+0000","last_unstale":"2024-01-16T15:05:20.221674+0000","last_undegraded":"2024-01-16T15:05:20.221674+0000","last_fullsized":"2024-01-16T15:05:20.221674+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:06:03.890562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.254873+0000","last_change":"2024-01-16T15:05:20.249136+0000","last_active":"2024-01-16T15:05:20.254873+0000","last_peered":"2024-01-16T15:05:20.254873+0000","last_clean":"2024-01-16T15:05:20.254873+0000","last_became_active":"2024-01-16T15:05:20.248868+0000","last_became_peered":"2024-01-16T15:05:20.248868+0000","last_unstale":"2024-01-16T15:05:20.254873+0000","last_undegraded":"2024-01-16T15:05:20.254873+0000","last_fullsized":"2024-01-16T15:05:20.254873+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:42.584977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423622+0000","last_change":"2024-01-16T15:05:19.225718+0000","last_active":"2024-01-16T15:05:21.423622+0000","last_peered":"2024-01-16T15:05:21.423622+0000","last_clean":"2024-01-16T15:05:21.423622+0000","last_became_active":"2024-01-16T15:05:19.225157+0000","last_became_peered":"2024-01-16T15:05:19.225157+0000","last_unstale":"2024-01-16T15:05:21.423622+0000","last_undegraded":"2024-01-16T15:05:21.423622+0000","last_fullsized":"2024-01-16T15:05:21.423622+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:57:18.591656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.245074+0000","last_change":"2024-01-16T15:05:20.240238+0000","last_active":"2024-01-16T15:05:20.245074+0000","last_peered":"2024-01-16T15:05:20.245074+0000","last_clean":"2024-01-16T15:05:20.245074+0000","last_became_active":"2024-01-16T15:05:20.239928+0000","last_became_peered":"2024-01-16T15:05:20.239928+0000","last_unstale":"2024-01-16T15:05:20.245074+0000","last_undegraded":"2024-01-16T15:05:20.245074+0000","last_fullsized":"2024-01-16T15:05:20.245074+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:13.431617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.226282+0000","last_change":"2024-01-16T15:05:19.227933+0000","last_active":"2024-01-16T15:05:20.226282+0000","last_peered":"2024-01-16T15:05:20.226282+0000","last_clean":"2024-01-16T15:05:20.226282+0000","last_became_active":"2024-01-16T15:05:19.227287+0000","last_became_peered":"2024-01-16T15:05:19.227287+0000","last_unstale":"2024-01-16T15:05:20.226282+0000","last_undegraded":"2024-01-16T15:05:20.226282+0000","last_fullsized":"2024-01-16T15:05:20.226282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:56.660374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252159+0000","last_change":"2024-01-16T15:05:20.244766+0000","last_active":"2024-01-16T15:05:20.252159+0000","last_peered":"2024-01-16T15:05:20.252159+0000","last_clean":"2024-01-16T15:05:20.252159+0000","last_became_active":"2024-01-16T15:05:20.244495+0000","last_became_peered":"2024-01-16T15:05:20.244495+0000","last_unstale":"2024-01-16T15:05:20.252159+0000","last_undegraded":"2024-01-16T15:05:20.252159+0000","last_fullsized":"2024-01-16T15:05:20.252159+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:23.544704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.229827+0000","last_change":"2024-01-16T15:05:19.232700+0000","last_active":"2024-01-16T15:05:20.229827+0000","last_peered":"2024-01-16T15:05:20.229827+0000","last_clean":"2024-01-16T15:05:20.229827+0000","last_became_active":"2024-01-16T15:05:19.231172+0000","last_became_peered":"2024-01-16T15:05:19.231172+0000","last_unstale":"2024-01-16T15:05:20.229827+0000","last_undegraded":"2024-01-16T15:05:20.229827+0000","last_fullsized":"2024-01-16T15:05:20.229827+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:57.272125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250788+0000","last_change":"2024-01-16T15:05:20.247212+0000","last_active":"2024-01-16T15:05:20.250788+0000","last_peered":"2024-01-16T15:05:20.250788+0000","last_clean":"2024-01-16T15:05:20.250788+0000","last_became_active":"2024-01-16T15:05:20.246855+0000","last_became_peered":"2024-01-16T15:05:20.246855+0000","last_unstale":"2024-01-16T15:05:20.250788+0000","last_undegraded":"2024-01-16T15:05:20.250788+0000","last_fullsized":"2024-01-16T15:05:20.250788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:13.431617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.217051+0000","last_change":"2024-01-16T15:05:19.224066+0000","last_active":"2024-01-16T15:05:21.217051+0000","last_peered":"2024-01-16T15:05:21.217051+0000","last_clean":"2024-01-16T15:05:21.217051+0000","last_became_active":"2024-01-16T15:05:19.223742+0000","last_became_peered":"2024-01-16T15:05:19.223742+0000","last_unstale":"2024-01-16T15:05:21.217051+0000","last_undegraded":"2024-01-16T15:05:21.217051+0000","last_fullsized":"2024-01-16T15:05:21.217051+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:34.042174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247869+0000","last_change":"2024-01-16T15:05:20.243131+0000","last_active":"2024-01-16T15:05:20.247869+0000","last_peered":"2024-01-16T15:05:20.247869+0000","last_clean":"2024-01-16T15:05:20.247869+0000","last_became_active":"2024-01-16T15:05:20.242844+0000","last_became_peered":"2024-01-16T15:05:20.242844+0000","last_unstale":"2024-01-16T15:05:20.247869+0000","last_undegraded":"2024-01-16T15:05:20.247869+0000","last_fullsized":"2024-01-16T15:05:20.247869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:01.528474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.221924+0000","last_change":"2024-01-16T15:05:19.223723+0000","last_active":"2024-01-16T15:05:20.221924+0000","last_peered":"2024-01-16T15:05:20.221924+0000","last_clean":"2024-01-16T15:05:20.221924+0000","last_became_active":"2024-01-16T15:05:19.223421+0000","last_became_peered":"2024-01-16T15:05:19.223421+0000","last_unstale":"2024-01-16T15:05:20.221924+0000","last_undegraded":"2024-01-16T15:05:20.221924+0000","last_fullsized":"2024-01-16T15:05:20.221924+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:43.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.237185+0000","last_change":"2024-01-16T15:05:20.228580+0000","last_active":"2024-01-16T15:05:20.237185+0000","last_peered":"2024-01-16T15:05:20.237185+0000","last_clean":"2024-01-16T15:05:20.237185+0000","last_became_active":"2024-01-16T15:05:20.228285+0000","last_became_peered":"2024-01-16T15:05:20.228285+0000","last_unstale":"2024-01-16T15:05:20.237185+0000","last_undegraded":"2024-01-16T15:05:20.237185+0000","last_fullsized":"2024-01-16T15:05:20.237185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:32.091740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.228510+0000","last_change":"2024-01-16T15:05:19.235230+0000","last_active":"2024-01-16T15:05:20.228510+0000","last_peered":"2024-01-16T15:05:20.228510+0000","last_clean":"2024-01-16T15:05:20.228510+0000","last_became_active":"2024-01-16T15:05:19.234945+0000","last_became_peered":"2024-01-16T15:05:19.234945+0000","last_unstale":"2024-01-16T15:05:20.228510+0000","last_undegraded":"2024-01-16T15:05:20.228510+0000","last_fullsized":"2024-01-16T15:05:20.228510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:45.715224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253110+0000","last_change":"2024-01-16T15:05:20.248535+0000","last_active":"2024-01-16T15:05:20.253110+0000","last_peered":"2024-01-16T15:05:20.253110+0000","last_clean":"2024-01-16T15:05:20.253110+0000","last_became_active":"2024-01-16T15:05:20.247506+0000","last_became_peered":"2024-01-16T15:05:20.247506+0000","last_unstale":"2024-01-16T15:05:20.253110+0000","last_undegraded":"2024-01-16T15:05:20.253110+0000","last_fullsized":"2024-01-16T15:05:20.253110+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:55.299913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423329+0000","last_change":"2024-01-16T15:05:19.231368+0000","last_active":"2024-01-16T15:05:21.423329+0000","last_peered":"2024-01-16T15:05:21.423329+0000","last_clean":"2024-01-16T15:05:21.423329+0000","last_became_active":"2024-01-16T15:05:19.231051+0000","last_became_peered":"2024-01-16T15:05:19.231051+0000","last_unstale":"2024-01-16T15:05:21.423329+0000","last_undegraded":"2024-01-16T15:05:21.423329+0000","last_fullsized":"2024-01-16T15:05:21.423329+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:28.750083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":200,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:05:19.210941+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:05:19.210941+0000","last_peered":"2024-01-16T15:05:19.210941+0000","last_clean":"2024-01-16T15:05:19.210941+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:05:19.210941+0000","last_undegraded":"2024-01-16T15:05:19.210941+0000","last_fullsized":"2024-01-16T15:05:19.210941+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":700416,"data_stored":291407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.3560000000000001,"15min":1.208},"min":{"1min":0.746,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.4809999999999999,"5min":9.7899999999999991,"15min":9.7899999999999991},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.2929999999999999,"15min":1.2090000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":9.5920000000000005,"5min":9.5920000000000005,"15min":9.5920000000000005},"last":1.569}]},{"osd":1,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.5649999999999999,"15min":1.264},"min":{"1min":0.745,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":12.577999999999999,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.014},{"interface":"front","average":{"1min":1.831,"5min":1.353,"15min":1.2809999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":1.121}]},{"osd":2,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.3919999999999999,"15min":1.3939999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":5.931,"5min":5.931,"15min":5.931},"last":1.242},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.335,"15min":1.377},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":1.6779999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6440000000000001,"5min":1.661,"15min":1.516},"min":{"1min":0.91900000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":12.654999999999999,"5min":12.654999999999999,"15min":12.654999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":2.153,"5min":1.5660000000000001,"15min":1.591},"min":{"1min":0.80000000000000004,"5min":0.745,"15min":0.745},"max":{"1min":8.657,"5min":8.657,"15min":8.657},"last":0.93100000000000005}]},{"osd":4,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.4350000000000001,"15min":1.482},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":9.6980000000000004},"last":1.3959999999999999},{"interface":"front","average":{"1min":3.3359999999999999,"5min":1.851,"15min":1.671},"min":{"1min":0.92400000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":12.506,"5min":12.506,"15min":12.506},"last":1.73}]},{"osd":5,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":3.4729999999999999,"5min":2.0470000000000002,"15min":1.8169999999999999},"min":{"1min":1.339,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":12.606,"5min":12.606,"15min":12.606},"last":1.956},{"interface":"front","average":{"1min":2.1579999999999999,"5min":1.698,"15min":1.6759999999999999},"min":{"1min":0.75900000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":9.5449999999999999,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":1.5149999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.655,"15min":1.637},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":9.7309999999999999,"5min":9.7309999999999999,"15min":9.7309999999999999},"last":1.792},{"interface":"front","average":{"1min":2.9359999999999999,"5min":1.9319999999999999,"15min":1.8069999999999999},"min":{"1min":1.2889999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":12.630000000000001,"5min":12.630000000000001,"15min":12.630000000000001},"last":1.859}]}]},{"osd":4,"up_from":15,"seq":64424509511,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36056,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335784,"statfs":{"total":96636764160,"available":96599842816,"internally_reserved":0,"allocated":724992,"data_stored":299319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":1.49,"15min":1.1719999999999999},"min":{"1min":0.75900000000000001,"5min":0.622,"15min":0.622},"max":{"1min":6.4619999999999997,"5min":14.993,"15min":14.993},"last":1.012},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.266,"15min":1.167},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":1.351}]},{"osd":1,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":2.383,"5min":1.4990000000000001,"15min":1.2569999999999999},"min":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":10.821999999999999,"5min":10.821999999999999,"15min":10.821999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.7470000000000001,"5min":1.391,"15min":1.276},"min":{"1min":0.88300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":6.6660000000000004,"5min":6.6660000000000004,"15min":6.6660000000000004},"last":1.3169999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4129999999999998,"5min":1.571,"15min":1.3480000000000001},"min":{"1min":0.88400000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":16.457999999999998,"5min":16.457999999999998,"15min":16.457999999999998},"last":1.294},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.4319999999999999,"15min":1.3280000000000001},"min":{"1min":0.68799999999999994,"5min":0.627,"15min":0.627},"max":{"1min":6.7389999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"last":1.5669999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":1.7450000000000001,"15min":1.4470000000000001},"min":{"1min":1.1419999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":26.741,"5min":26.741,"15min":26.741},"last":1.506},{"interface":"front","average":{"1min":3.4910000000000001,"5min":1.948,"15min":1.5780000000000001},"min":{"1min":0.95499999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":26.701000000000001,"5min":26.701000000000001,"15min":26.701000000000001},"last":1.4770000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":1.6279999999999999,"15min":1.536},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.5949999999999998,"5min":11.823,"15min":11.823},"last":1.228},{"interface":"front","average":{"1min":3.327,"5min":1.9279999999999999,"15min":1.647},"min":{"1min":0.72599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":26.597000000000001,"5min":26.597000000000001,"15min":26.597000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":2.3220000000000001,"5min":1.6739999999999999,"15min":1.556},"min":{"1min":0.54800000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":10.986000000000001,"5min":10.986000000000001,"15min":10.986000000000001},"last":1.163},{"interface":"front","average":{"1min":1.871,"5min":1.6579999999999999,"15min":1.5840000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.7839999999999998},"last":1.4470000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":3.351,"5min":1.865,"15min":1.671},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":26.683,"5min":26.683,"15min":26.683},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.907,"15min":1.7310000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":6.5410000000000004,"5min":13.561999999999999,"15min":13.561999999999999},"last":1.5329999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1282048,"data_stored":875000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.2070000000000001,"15min":1.0780000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":8.5180000000000007,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.95699999999999996},{"interface":"front","average":{"1min":2.8889999999999998,"5min":1.4550000000000001,"15min":1.175},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":30.126000000000001,"5min":30.126000000000001,"15min":30.126000000000001},"last":1.282}]},{"osd":1,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1990000000000001,"15min":1.1399999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":1.0720000000000001},{"interface":"front","average":{"1min":2.8250000000000002,"5min":1.593,"15min":1.3},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":21.032,"5min":21.032,"15min":21.032},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3300000000000001,"15min":1.274},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":1.458},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.369,"15min":1.29},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":1.3759999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3899999999999999,"15min":1.3520000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.752,"5min":4.2249999999999996,"15min":4.2249999999999996},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3979999999999999,"15min":1.3540000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":3.4300000000000002,"5min":3.653,"15min":3.653},"last":1.4330000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.4950000000000001,"15min":1.4279999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":14.659000000000001,"5min":14.659000000000001,"15min":14.659000000000001},"last":1.143},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.544,"15min":1.45},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.8650000000000002,"5min":11.433,"15min":11.433},"last":1.246}]},{"osd":5,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2869999999999999,"5min":1.599,"15min":1.4670000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":18.132000000000001,"5min":18.132000000000001,"15min":18.132000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.869,"5min":1.641,"15min":1.587},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":9.2370000000000001,"5min":9.2370000000000001,"15min":9.2370000000000001},"last":1.3169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.542,"15min":1.55},"min":{"1min":0.72099999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.4649999999999999,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.486},{"interface":"front","average":{"1min":3.214,"5min":1.915,"15min":1.6819999999999999},"min":{"1min":0.68400000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":29.991,"5min":29.991,"15min":29.991},"last":1.204}]}]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.2529999999999999,"15min":1.099},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.0760000000000001,"5min":1.417,"15min":1.165},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":1.073}]},{"osd":1,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.244,"15min":1.1519999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":5.0750000000000002,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.819,"5min":1.3280000000000001,"15min":1.1619999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":9.5210000000000008,"5min":9.5210000000000008,"15min":9.5210000000000008},"last":1.1220000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.359,"15min":1.236},"min":{"1min":0.66900000000000004,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4970000000000001,"15min":1.2929999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":11.321,"5min":11.321,"15min":11.321},"last":0.84699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.4650000000000001,"15min":1.381},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":6.7450000000000001,"5min":6.7450000000000001,"15min":6.7450000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.5860000000000001,"15min":1.4350000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":6.1669999999999998,"5min":14.647,"15min":14.647},"last":1.3959999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":2.9780000000000002,"5min":1.675,"15min":1.4219999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":24.436,"5min":24.436,"15min":24.436},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.4419999999999999,"15min":1.4039999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.4900000000000002,"5min":4.4900000000000002,"15min":4.4900000000000002},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":1.774,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":21.192,"5min":21.192,"15min":21.192},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.536,"15min":1.4410000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.208}]},{"osd":7,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":3.7050000000000001,"5min":1.95,"15min":1.6879999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":32.405999999999999,"5min":32.405999999999999,"15min":32.405999999999999},"last":1.286},{"interface":"front","average":{"1min":3.0920000000000001,"5min":1.9099999999999999,"15min":1.74},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":24.367000000000001,"5min":24.367000000000001,"15min":24.367000000000001},"last":1.456}]}]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40172,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331668,"statfs":{"total":96636764160,"available":96595628032,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":2.7869999999999999,"5min":1.302,"15min":0.96599999999999997},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":31.562999999999999,"5min":31.562999999999999,"15min":31.562999999999999},"last":0.996},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.153,"15min":1.0289999999999999},"min":{"1min":0.49299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":8.4879999999999995,"5min":8.4879999999999995,"15min":8.4879999999999995},"last":0.75900000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.0529999999999999,"15min":0.91600000000000004},"min":{"1min":0.60999999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.4780000000000002,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":1.083},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.2989999999999999,"15min":1.101},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.992,"5min":6.992,"15min":6.992},"last":1.129}]},{"osd":2,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.371,"15min":1.171},"min":{"1min":0.68000000000000005,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.0529999999999999,"5min":7.0529999999999999,"15min":7.0529999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4119999999999999,"15min":1.1890000000000001},"min":{"1min":0.64700000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.9729999999999999,"5min":14.577999999999999,"15min":14.577999999999999},"last":0.84499999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.4490000000000001,"15min":1.3069999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":1.52},{"interface":"front","average":{"1min":2.109,"5min":1.6180000000000001,"15min":1.462},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":7.1379999999999999,"5min":7.1379999999999999,"15min":7.1379999999999999},"last":2.0110000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.5469999999999999,"15min":1.4359999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":2.625,"5min":1.788,"15min":1.544},"min":{"1min":0.90600000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":17.613,"5min":17.613,"15min":17.613},"last":1.931}]},{"osd":6,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":3.0059999999999998,"5min":1.887,"15min":1.5580000000000001},"min":{"1min":1.0449999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":13.182,"5min":13.182,"15min":13.182},"last":1.4059999999999999},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.7569999999999999,"15min":1.498},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":18.481000000000002,"5min":18.481000000000002,"15min":18.481000000000002},"last":1.748}]},{"osd":7,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.6399999999999999,"15min":1.4890000000000001},"min":{"1min":0.89600000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":3.2829999999999999,"5min":4.0830000000000002,"15min":4.0830000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":1.8540000000000001,"15min":1.5860000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":19.183,"5min":19.183,"15min":19.183},"last":1.595}]}]},{"osd":2,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40176,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94499999999999995,"15min":1.0069999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.026,"15min":1.0229999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.6850000000000001,"5min":3.6850000000000001,"15min":3.6850000000000001},"last":0.999}]},{"osd":1,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0600000000000001,"15min":1.0429999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7909999999999999,"5min":3.839,"15min":3.839},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.177,"5min":1.135,"15min":1.0860000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.091}]},{"osd":3,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1399999999999999,"15min":0.96599999999999997},"min":{"1min":0.503,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6179999999999999,"5min":9.1530000000000005,"15min":9.1530000000000005},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.325,"15min":1.3149999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.1459999999999999,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.6910000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.3580000000000001,"15min":1.3280000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":1.597},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.4330000000000001,"15min":1.4299999999999999},"min":{"1min":0.84799999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":4.859,"5min":4.859,"15min":4.859},"last":1.1539999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.448,"15min":1.3520000000000001},"min":{"1min":0.95699999999999996,"5min":0.753,"15min":0.753},"max":{"1min":4.3609999999999998,"5min":9.6189999999999998,"15min":9.6189999999999998},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.48,"15min":1.3919999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.1469999999999998,"5min":9.423,"15min":9.423},"last":1.323}]},{"osd":6,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.512,"15min":1.397},"min":{"1min":0.93100000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.0310000000000001,"5min":9.5359999999999996,"15min":9.5359999999999996},"last":1.647},{"interface":"front","average":{"1min":3.3250000000000002,"5min":1.966,"15min":1.6319999999999999},"min":{"1min":0.89300000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":21.564,"5min":21.564,"15min":21.564},"last":1.4259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4710000000000001,"15min":1.51},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0550000000000002,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.444},{"interface":"front","average":{"1min":1.613,"5min":1.5229999999999999,"15min":1.577},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.3069999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542215,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40176,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.04,"15min":0.873},"min":{"1min":0.41599999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.954,"5min":10.667999999999999,"15min":10.667999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.1259999999999999,"15min":0.96799999999999997},"min":{"1min":0.59099999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.1020000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.03,"15min":0.98199999999999998},"min":{"1min":0.53600000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3799999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.05},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.3480000000000001,"15min":1.206},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.772,"5min":10.805,"15min":10.805},"last":1.1339999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3899999999999999,"15min":1.1839999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9159999999999999,"5min":10.991,"15min":10.991},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.952,"5min":1.393,"15min":1.161},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":10.430999999999999,"5min":10.430999999999999,"15min":10.430999999999999},"last":1.0089999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.284,"15min":1.284},"min":{"1min":0.747,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.5489999999999999,"15min":1.4950000000000001},"min":{"1min":0.72999999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.875,"5min":11.157999999999999,"15min":11.157999999999999},"last":1.29}]},{"osd":5,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.4890000000000001,"15min":1.387},"min":{"1min":0.71199999999999997,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":4.9359999999999999,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":1.341},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.5429999999999999,"15min":1.4770000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.7530000000000001,"5min":3.7530000000000001,"15min":3.7530000000000001},"last":1.1799999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.6799999999999999,"15min":1.536},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.23},{"interface":"front","average":{"1min":2.4039999999999999,"5min":1.734,"15min":1.554},"min":{"1min":0.81200000000000006,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"last":1.3680000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.0859999999999999,"15min":1.728},"min":{"1min":1.1890000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":30.834,"5min":30.834,"15min":30.834},"last":1.3149999999999999},{"interface":"front","average":{"1min":3.069,"5min":2.0099999999999998,"15min":1.657},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":27.263999999999999,"5min":27.263999999999999,"15min":27.263999999999999},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40756,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331084,"statfs":{"total":96636764160,"available":96595030016,"internally_reserved":0,"allocated":1282048,"data_stored":875000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.944,"5min":1.1759999999999999,"15min":1.022},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":22.132999999999999,"5min":22.132999999999999,"15min":22.132999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":4.1799999999999997,"5min":1.681,"15min":1.256},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":50.429000000000002,"5min":50.429000000000002,"15min":50.429000000000002},"last":0.51200000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.202,"5min":1.478,"15min":1.23},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":22.716999999999999,"5min":22.716999999999999,"15min":22.716999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":4.4909999999999997,"5min":1.867,"15min":1.349},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":50.581000000000003,"5min":50.581000000000003,"15min":50.581000000000003},"last":0.59899999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.2809999999999999,"15min":1.1699999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.6070000000000002,"5min":10.795,"15min":10.795},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.1560000000000001,"5min":1.5940000000000001,"15min":1.3799999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":22.254999999999999,"5min":22.254999999999999,"15min":22.254999999999999},"last":0.624}]},{"osd":4,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.6299999999999999,"5min":1.7030000000000001,"15min":1.4790000000000001},"min":{"1min":0.72599999999999998,"5min":0.625,"15min":0.625},"max":{"1min":22.407,"5min":22.407,"15min":22.407},"last":1.1910000000000001},{"interface":"front","average":{"1min":4.726,"5min":2.153,"15min":1.569},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":50.497,"5min":50.497,"15min":50.497},"last":0.88800000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.9449999999999998,"5min":1.841,"15min":1.5049999999999999},"min":{"1min":0.70999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":14.112,"5min":14.112,"15min":14.112},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.5840000000000001,"5min":1.7270000000000001,"15min":1.5740000000000001},"min":{"1min":0.78000000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":22.882999999999999,"5min":22.882999999999999,"15min":22.882999999999999},"last":1.1020000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":3.048,"5min":1.857,"15min":1.554},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":23.405000000000001,"5min":23.405000000000001,"15min":23.405000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":3.3090000000000002,"5min":1.911,"15min":1.661},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":23.547000000000001,"5min":23.547000000000001,"15min":23.547000000000001},"last":1.1519999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":5.2839999999999998,"5min":2.2959999999999998,"15min":1.786},"min":{"1min":0.86399999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":50.542000000000002,"5min":50.542000000000002,"15min":50.542000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.7529999999999999,"15min":1.6120000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.9850000000000003,"5min":10.736000000000001,"15min":10.736000000000001},"last":1.002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:05:24.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:05:24.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.977+0000 7ff6c7e00640 1 Processor -- start 2024-01-16T15:05:24.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.977+0000 7ff6c7e00640 1 -- start start 2024-01-16T15:05:24.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.979+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 0x7ff6c80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.979+0000 7ff6c7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff6c806d160 0x7ff6c806d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.979+0000 7ff6c7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c8070e50 con 0x7ff6c8143c40 2024-01-16T15:05:24.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.979+0000 7ff6c7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c80723c0 con 0x7ff6c806d160 2024-01-16T15:05:24.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.979+0000 7ff6c7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8144020 con 0x7ff6c80cc500 2024-01-16T15:05:24.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.980+0000 7ff6bde00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 0x7ff6c80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.980+0000 7ff6bde00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 0x7ff6c80cc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:47404/0 (socket says 172.21.15.53:47404) 2024-01-16T15:05:24.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.980+0000 7ff6bde00640 1 -- 172.21.15.53:0/715253297 learned_addr learned my addr 172.21.15.53:0/715253297 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:24.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.980+0000 7ff6c7400640 1 --2- 172.21.15.53:0/715253297 >> v2:172.21.15.146:3300/0 conn(0x7ff6c806d160 0x7ff6c806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.981+0000 7ff6bde00640 1 -- 172.21.15.53:0/715253297 >> v2:172.21.15.146:3300/0 conn(0x7ff6c806d160 msgr2=0x7ff6c806d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:24.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.981+0000 7ff6bde00640 1 --2- 172.21.15.53:0/715253297 >> v2:172.21.15.146:3300/0 conn(0x7ff6c806d160 0x7ff6c806d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.981+0000 7ff6bde00640 1 -- 172.21.15.53:0/715253297 >> v1:172.21.15.146:6789/0 conn(0x7ff6c8143c40 legacy=0x7ff6c8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.981+0000 7ff6bde00640 1 -- 172.21.15.53:0/715253297 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c80ccea0 con 0x7ff6c80cc500 2024-01-16T15:05:24.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.981+0000 7ff6c7400640 1 --2- 172.21.15.53:0/715253297 >> v2:172.21.15.146:3300/0 conn(0x7ff6c806d160 0x7ff6c806d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:24.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.982+0000 7ff6bde00640 1 --2- 172.21.15.53:0/715253297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 0x7ff6c80cc960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009870 tx=0x7ff6b002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82085753db1b498f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:24.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.983+0000 7ff6c6000640 1 -- 172.21.15.53:0/715253297 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0031640 con 0x7ff6c80cc500 2024-01-16T15:05:24.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.983+0000 7ff6c6000640 1 -- 172.21.15.53:0/715253297 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b00317a0 con 0x7ff6c80cc500 2024-01-16T15:05:24.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.983+0000 7ff6c6000640 1 -- 172.21.15.53:0/715253297 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0031b30 con 0x7ff6c80cc500 2024-01-16T15:05:24.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 -- 172.21.15.53:0/715253297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 msgr2=0x7ff6c80cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/715253297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 0x7ff6c80cc960 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0009870 tx=0x7ff6b002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 -- 172.21.15.53:0/715253297 shutdown_connections 2024-01-16T15:05:24.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/715253297 >> v2:172.21.15.146:3300/0 conn(0x7ff6c806d160 0x7ff6c806d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/715253297 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c80cc500 0x7ff6c80cc960 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 -- 172.21.15.53:0/715253297 >> 172.21.15.53:0/715253297 conn(0x7ff6c806a800 msgr2=0x7ff6c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:24.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 -- 172.21.15.53:0/715253297 shutdown_connections 2024-01-16T15:05:24.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.984+0000 7ff6c7e00640 1 -- 172.21.15.53:0/715253297 wait complete. 2024-01-16T15:05:24.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.986+0000 7ff6c7e00640 1 Processor -- start 2024-01-16T15:05:24.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.987+0000 7ff6c7e00640 1 -- start start 2024-01-16T15:05:24.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c806d160 0x7ff6c81f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 0x7ff6c80add10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6c7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c8143c40 0x7ff6c80b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6c7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c813aea0 con 0x7ff6c80cc500 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6c7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c80ac8d0 con 0x7ff6c8143c40 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6c7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c81f3280 con 0x7ff6c806d160 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6bde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 0x7ff6c80add10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6bde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 0x7ff6c80add10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52204/0 (socket says 172.21.15.53:52204) 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.988+0000 7ff6bde00640 1 -- 172.21.15.53:0/900942737 learned_addr learned my addr 172.21.15.53:0/900942737 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6c6a00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c806d160 0x7ff6c81f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6c7400640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c8143c40 0x7ff6c80b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6bde00640 1 -- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c806d160 msgr2=0x7ff6c81f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6bde00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c806d160 0x7ff6c81f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6bde00640 1 -- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c8143c40 msgr2=0x7ff6c80b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6bde00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c8143c40 0x7ff6c80b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6bde00640 1 -- 172.21.15.53:0/900942737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b00094d0 con 0x7ff6c80cc500 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6c6a00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c806d160 0x7ff6c81f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.989+0000 7ff6bde00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 0x7ff6c80add10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0000c00 tx=0x7ff6b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.990+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b003ac60 con 0x7ff6c80cc500 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.990+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b003adc0 con 0x7ff6c80cc500 2024-01-16T15:05:24.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.990+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c814c580 con 0x7ff6c80cc500 2024-01-16T15:05:24.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.990+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0044630 con 0x7ff6c80cc500 2024-01-16T15:05:24.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.990+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c814cb40 con 0x7ff6c80cc500 2024-01-16T15:05:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.991+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c80cdf30 con 0x7ff6c80cc500 2024-01-16T15:05:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.992+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ff6b002f050 con 0x7ff6c80cc500 2024-01-16T15:05:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.993+0000 7ff6bf400640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff6a006b890 0x7ff6a006dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.993+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff6b00ac650 con 0x7ff6c80cc500 2024-01-16T15:05:24.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.994+0000 7ff6c6a00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff6a006b890 0x7ff6a006dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:24.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.995+0000 7ff6c6a00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff6a006b890 0x7ff6a006dd50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80012e0 tx=0x7ff6b8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:24.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:24.997+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6b0076b00 con 0x7ff6c80cc500 2024-01-16T15:05:25.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.189+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6c806c960 con 0x7ff6a006b890 2024-01-16T15:05:25.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.214+0000 7ff6bf400640 1 -- 172.21.15.53:0/900942737 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286894 (secure 0 0 0) 0x7ff6b8001ad0 con 0x7ff6a006b890 2024-01-16T15:05:25.197 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:25.199 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.220+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff6a006b890 msgr2=0x7ff6a006dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.220+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff6a006b890 0x7ff6a006dd50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff6b80012e0 tx=0x7ff6b8009040 comp rx=0 tx=0).stop 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.220+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 msgr2=0x7ff6c80add10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.220+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 0x7ff6c80add10 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff6b0000c00 tx=0x7ff6b00046c0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 shutdown_connections 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff6a006b890 0x7ff6a006dd50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c8143c40 0x7ff6c80b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c80cc500 0x7ff6c80add10 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 --2- 172.21.15.53:0/900942737 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c806d160 0x7ff6c81f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 >> 172.21.15.53:0/900942737 conn(0x7ff6c806a800 msgr2=0x7ff6c8071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 shutdown_connections 2024-01-16T15:05:25.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.221+0000 7ff6c7e00640 1 -- 172.21.15.53:0/900942737 wait complete. 2024-01-16T15:05:25.220 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-01-16T15:05:24.307040+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":60,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":305972,"kb_used_data":6584,"kb_used_omap":12,"kb_used_meta":299251,"kb_avail":754668748,"statfs":{"total":773094113280,"available":772780797952,"internally_reserved":0,"allocated":6742016,"data_stored":3479254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":306433584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":33,"apply_latency_ms":33,"commit_latency_ns":33000000,"apply_latency_ns":33000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002204"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.239895+0000","last_change":"2024-01-16T15:05:20.238820+0000","last_active":"2024-01-16T15:05:20.239895+0000","last_peered":"2024-01-16T15:05:20.239895+0000","last_clean":"2024-01-16T15:05:20.239895+0000","last_became_active":"2024-01-16T15:05:20.238515+0000","last_became_peered":"2024-01-16T15:05:20.238515+0000","last_unstale":"2024-01-16T15:05:20.239895+0000","last_undegraded":"2024-01-16T15:05:20.239895+0000","last_fullsized":"2024-01-16T15:05:20.239895+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:30:11.361119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255504+0000","last_change":"2024-01-16T15:05:20.252971+0000","last_active":"2024-01-16T15:05:20.255504+0000","last_peered":"2024-01-16T15:05:20.255504+0000","last_clean":"2024-01-16T15:05:20.255504+0000","last_became_active":"2024-01-16T15:05:20.252446+0000","last_became_peered":"2024-01-16T15:05:20.252446+0000","last_unstale":"2024-01-16T15:05:20.255504+0000","last_undegraded":"2024-01-16T15:05:20.255504+0000","last_fullsized":"2024-01-16T15:05:20.255504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:44:59.541512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.236175+0000","last_change":"2024-01-16T15:05:20.229623+0000","last_active":"2024-01-16T15:05:20.236175+0000","last_peered":"2024-01-16T15:05:20.236175+0000","last_clean":"2024-01-16T15:05:20.236175+0000","last_became_active":"2024-01-16T15:05:20.229286+0000","last_became_peered":"2024-01-16T15:05:20.229286+0000","last_unstale":"2024-01-16T15:05:20.236175+0000","last_undegraded":"2024-01-16T15:05:20.236175+0000","last_fullsized":"2024-01-16T15:05:20.236175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:22.954805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242762+0000","last_change":"2024-01-16T15:05:20.237777+0000","last_active":"2024-01-16T15:05:20.242762+0000","last_peered":"2024-01-16T15:05:20.242762+0000","last_clean":"2024-01-16T15:05:20.242762+0000","last_became_active":"2024-01-16T15:05:20.237515+0000","last_became_peered":"2024-01-16T15:05:20.237515+0000","last_unstale":"2024-01-16T15:05:20.242762+0000","last_undegraded":"2024-01-16T15:05:20.242762+0000","last_fullsized":"2024-01-16T15:05:20.242762+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:37.705713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.235458+0000","last_change":"2024-01-16T15:05:20.233436+0000","last_active":"2024-01-16T15:05:20.235458+0000","last_peered":"2024-01-16T15:05:20.235458+0000","last_clean":"2024-01-16T15:05:20.235458+0000","last_became_active":"2024-01-16T15:05:20.230693+0000","last_became_peered":"2024-01-16T15:05:20.230693+0000","last_unstale":"2024-01-16T15:05:20.235458+0000","last_undegraded":"2024-01-16T15:05:20.235458+0000","last_fullsized":"2024-01-16T15:05:20.235458+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:41.663640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.241921+0000","last_change":"2024-01-16T15:05:20.235127+0000","last_active":"2024-01-16T15:05:20.241921+0000","last_peered":"2024-01-16T15:05:20.241921+0000","last_clean":"2024-01-16T15:05:20.241921+0000","last_became_active":"2024-01-16T15:05:20.234852+0000","last_became_peered":"2024-01-16T15:05:20.234852+0000","last_unstale":"2024-01-16T15:05:20.241921+0000","last_undegraded":"2024-01-16T15:05:20.241921+0000","last_fullsized":"2024-01-16T15:05:20.241921+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:13:12.014658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216642+0000","last_change":"2024-01-16T15:05:20.240118+0000","last_active":"2024-01-16T15:05:21.216642+0000","last_peered":"2024-01-16T15:05:21.216642+0000","last_clean":"2024-01-16T15:05:21.216642+0000","last_became_active":"2024-01-16T15:05:20.239664+0000","last_became_peered":"2024-01-16T15:05:20.239664+0000","last_unstale":"2024-01-16T15:05:21.216642+0000","last_undegraded":"2024-01-16T15:05:21.216642+0000","last_fullsized":"2024-01-16T15:05:21.216642+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:28.378083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.238625+0000","last_change":"2024-01-16T15:05:20.232961+0000","last_active":"2024-01-16T15:05:20.238625+0000","last_peered":"2024-01-16T15:05:20.238625+0000","last_clean":"2024-01-16T15:05:20.238625+0000","last_became_active":"2024-01-16T15:05:20.232681+0000","last_became_peered":"2024-01-16T15:05:20.232681+0000","last_unstale":"2024-01-16T15:05:20.238625+0000","last_undegraded":"2024-01-16T15:05:20.238625+0000","last_fullsized":"2024-01-16T15:05:20.238625+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:22.874597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.218944+0000","last_change":"2024-01-16T15:05:20.240572+0000","last_active":"2024-01-16T15:05:21.218944+0000","last_peered":"2024-01-16T15:05:21.218944+0000","last_clean":"2024-01-16T15:05:21.218944+0000","last_became_active":"2024-01-16T15:05:20.240296+0000","last_became_peered":"2024-01-16T15:05:20.240296+0000","last_unstale":"2024-01-16T15:05:21.218944+0000","last_undegraded":"2024-01-16T15:05:21.218944+0000","last_fullsized":"2024-01-16T15:05:21.218944+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:55.299913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.248030+0000","last_change":"2024-01-16T15:05:20.243791+0000","last_active":"2024-01-16T15:05:20.248030+0000","last_peered":"2024-01-16T15:05:20.248030+0000","last_clean":"2024-01-16T15:05:20.248030+0000","last_became_active":"2024-01-16T15:05:20.243513+0000","last_became_peered":"2024-01-16T15:05:20.243513+0000","last_unstale":"2024-01-16T15:05:20.248030+0000","last_undegraded":"2024-01-16T15:05:20.248030+0000","last_fullsized":"2024-01-16T15:05:20.248030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:47:12.737053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255293+0000","last_change":"2024-01-16T15:05:20.253744+0000","last_active":"2024-01-16T15:05:20.255293+0000","last_peered":"2024-01-16T15:05:20.255293+0000","last_clean":"2024-01-16T15:05:20.255293+0000","last_became_active":"2024-01-16T15:05:20.253426+0000","last_became_peered":"2024-01-16T15:05:20.253426+0000","last_unstale":"2024-01-16T15:05:20.255293+0000","last_undegraded":"2024-01-16T15:05:20.255293+0000","last_fullsized":"2024-01-16T15:05:20.255293+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:43:17.634683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.240102+0000","last_change":"2024-01-16T15:05:20.232550+0000","last_active":"2024-01-16T15:05:20.240102+0000","last_peered":"2024-01-16T15:05:20.240102+0000","last_clean":"2024-01-16T15:05:20.240102+0000","last_became_active":"2024-01-16T15:05:20.232293+0000","last_became_peered":"2024-01-16T15:05:20.232293+0000","last_unstale":"2024-01-16T15:05:20.240102+0000","last_undegraded":"2024-01-16T15:05:20.240102+0000","last_fullsized":"2024-01-16T15:05:20.240102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:35.867624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423576+0000","last_change":"2024-01-16T15:05:20.235880+0000","last_active":"2024-01-16T15:05:21.423576+0000","last_peered":"2024-01-16T15:05:21.423576+0000","last_clean":"2024-01-16T15:05:21.423576+0000","last_became_active":"2024-01-16T15:05:20.235577+0000","last_became_peered":"2024-01-16T15:05:20.235577+0000","last_unstale":"2024-01-16T15:05:21.423576+0000","last_undegraded":"2024-01-16T15:05:21.423576+0000","last_fullsized":"2024-01-16T15:05:21.423576+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:13:05.263225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216944+0000","last_change":"2024-01-16T15:05:20.241773+0000","last_active":"2024-01-16T15:05:21.216944+0000","last_peered":"2024-01-16T15:05:21.216944+0000","last_clean":"2024-01-16T15:05:21.216944+0000","last_became_active":"2024-01-16T15:05:20.241269+0000","last_became_peered":"2024-01-16T15:05:20.241269+0000","last_unstale":"2024-01-16T15:05:21.216944+0000","last_undegraded":"2024-01-16T15:05:21.216944+0000","last_fullsized":"2024-01-16T15:05:21.216944+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:10:03.363520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216805+0000","last_change":"2024-01-16T15:05:20.247822+0000","last_active":"2024-01-16T15:05:21.216805+0000","last_peered":"2024-01-16T15:05:21.216805+0000","last_clean":"2024-01-16T15:05:21.216805+0000","last_became_active":"2024-01-16T15:05:20.247567+0000","last_became_peered":"2024-01-16T15:05:20.247567+0000","last_unstale":"2024-01-16T15:05:21.216805+0000","last_undegraded":"2024-01-16T15:05:21.216805+0000","last_fullsized":"2024-01-16T15:05:21.216805+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:48:39.169632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.233523+0000","last_change":"2024-01-16T15:05:20.229308+0000","last_active":"2024-01-16T15:05:20.233523+0000","last_peered":"2024-01-16T15:05:20.233523+0000","last_clean":"2024-01-16T15:05:20.233523+0000","last_became_active":"2024-01-16T15:05:20.228992+0000","last_became_peered":"2024-01-16T15:05:20.228992+0000","last_unstale":"2024-01-16T15:05:20.233523+0000","last_undegraded":"2024-01-16T15:05:20.233523+0000","last_fullsized":"2024-01-16T15:05:20.233523+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:35.158758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252068+0000","last_change":"2024-01-16T15:05:20.247343+0000","last_active":"2024-01-16T15:05:20.252068+0000","last_peered":"2024-01-16T15:05:20.252068+0000","last_clean":"2024-01-16T15:05:20.252068+0000","last_became_active":"2024-01-16T15:05:20.238700+0000","last_became_peered":"2024-01-16T15:05:20.238700+0000","last_unstale":"2024-01-16T15:05:20.252068+0000","last_undegraded":"2024-01-16T15:05:20.252068+0000","last_fullsized":"2024-01-16T15:05:20.252068+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:41:18.897950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243266+0000","last_change":"2024-01-16T15:05:20.241942+0000","last_active":"2024-01-16T15:05:20.243266+0000","last_peered":"2024-01-16T15:05:20.243266+0000","last_clean":"2024-01-16T15:05:20.243266+0000","last_became_active":"2024-01-16T15:05:20.241666+0000","last_became_peered":"2024-01-16T15:05:20.241666+0000","last_unstale":"2024-01-16T15:05:20.243266+0000","last_undegraded":"2024-01-16T15:05:20.243266+0000","last_fullsized":"2024-01-16T15:05:20.243266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:50.528201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.239640+0000","last_change":"2024-01-16T15:05:20.238255+0000","last_active":"2024-01-16T15:05:20.239640+0000","last_peered":"2024-01-16T15:05:20.239640+0000","last_clean":"2024-01-16T15:05:20.239640+0000","last_became_active":"2024-01-16T15:05:20.237815+0000","last_became_peered":"2024-01-16T15:05:20.237815+0000","last_unstale":"2024-01-16T15:05:20.239640+0000","last_undegraded":"2024-01-16T15:05:20.239640+0000","last_fullsized":"2024-01-16T15:05:20.239640+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:10:04.890676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253079+0000","last_change":"2024-01-16T15:05:20.248596+0000","last_active":"2024-01-16T15:05:20.253079+0000","last_peered":"2024-01-16T15:05:20.253079+0000","last_clean":"2024-01-16T15:05:20.253079+0000","last_became_active":"2024-01-16T15:05:20.248097+0000","last_became_peered":"2024-01-16T15:05:20.248097+0000","last_unstale":"2024-01-16T15:05:20.253079+0000","last_undegraded":"2024-01-16T15:05:20.253079+0000","last_fullsized":"2024-01-16T15:05:20.253079+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:54.977369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242445+0000","last_change":"2024-01-16T15:05:20.238168+0000","last_active":"2024-01-16T15:05:20.242445+0000","last_peered":"2024-01-16T15:05:20.242445+0000","last_clean":"2024-01-16T15:05:20.242445+0000","last_became_active":"2024-01-16T15:05:20.237874+0000","last_became_peered":"2024-01-16T15:05:20.237874+0000","last_unstale":"2024-01-16T15:05:20.242445+0000","last_undegraded":"2024-01-16T15:05:20.242445+0000","last_fullsized":"2024-01-16T15:05:20.242445+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:37.192618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243650+0000","last_change":"2024-01-16T15:05:20.238080+0000","last_active":"2024-01-16T15:05:20.243650+0000","last_peered":"2024-01-16T15:05:20.243650+0000","last_clean":"2024-01-16T15:05:20.243650+0000","last_became_active":"2024-01-16T15:05:20.237391+0000","last_became_peered":"2024-01-16T15:05:20.237391+0000","last_unstale":"2024-01-16T15:05:20.243650+0000","last_undegraded":"2024-01-16T15:05:20.243650+0000","last_fullsized":"2024-01-16T15:05:20.243650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:27:08.900121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250757+0000","last_change":"2024-01-16T15:05:20.245358+0000","last_active":"2024-01-16T15:05:20.250757+0000","last_peered":"2024-01-16T15:05:20.250757+0000","last_clean":"2024-01-16T15:05:20.250757+0000","last_became_active":"2024-01-16T15:05:20.244029+0000","last_became_peered":"2024-01-16T15:05:20.244029+0000","last_unstale":"2024-01-16T15:05:20.250757+0000","last_undegraded":"2024-01-16T15:05:20.250757+0000","last_fullsized":"2024-01-16T15:05:20.250757+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:13:12.014658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423422+0000","last_change":"2024-01-16T15:05:20.233598+0000","last_active":"2024-01-16T15:05:21.423422+0000","last_peered":"2024-01-16T15:05:21.423422+0000","last_clean":"2024-01-16T15:05:21.423422+0000","last_became_active":"2024-01-16T15:05:20.232505+0000","last_became_peered":"2024-01-16T15:05:20.232505+0000","last_unstale":"2024-01-16T15:05:21.423422+0000","last_undegraded":"2024-01-16T15:05:21.423422+0000","last_fullsized":"2024-01-16T15:05:21.423422+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:16:27.966705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252691+0000","last_change":"2024-01-16T15:05:20.249771+0000","last_active":"2024-01-16T15:05:20.252691+0000","last_peered":"2024-01-16T15:05:20.252691+0000","last_clean":"2024-01-16T15:05:20.252691+0000","last_became_active":"2024-01-16T15:05:20.249479+0000","last_became_peered":"2024-01-16T15:05:20.249479+0000","last_unstale":"2024-01-16T15:05:20.252691+0000","last_undegraded":"2024-01-16T15:05:20.252691+0000","last_fullsized":"2024-01-16T15:05:20.252691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:06.994033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242781+0000","last_change":"2024-01-16T15:05:20.237332+0000","last_active":"2024-01-16T15:05:20.242781+0000","last_peered":"2024-01-16T15:05:20.242781+0000","last_clean":"2024-01-16T15:05:20.242781+0000","last_became_active":"2024-01-16T15:05:20.236885+0000","last_became_peered":"2024-01-16T15:05:20.236885+0000","last_unstale":"2024-01-16T15:05:20.242781+0000","last_undegraded":"2024-01-16T15:05:20.242781+0000","last_fullsized":"2024-01-16T15:05:20.242781+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:56:22.954805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242172+0000","last_change":"2024-01-16T15:05:20.236199+0000","last_active":"2024-01-16T15:05:20.242172+0000","last_peered":"2024-01-16T15:05:20.242172+0000","last_clean":"2024-01-16T15:05:20.242172+0000","last_became_active":"2024-01-16T15:05:20.235919+0000","last_became_peered":"2024-01-16T15:05:20.235919+0000","last_unstale":"2024-01-16T15:05:20.242172+0000","last_undegraded":"2024-01-16T15:05:20.242172+0000","last_fullsized":"2024-01-16T15:05:20.242172+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:39:26.943440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247151+0000","last_change":"2024-01-16T15:05:20.242282+0000","last_active":"2024-01-16T15:05:20.247151+0000","last_peered":"2024-01-16T15:05:20.247151+0000","last_clean":"2024-01-16T15:05:20.247151+0000","last_became_active":"2024-01-16T15:05:20.241794+0000","last_became_peered":"2024-01-16T15:05:20.241794+0000","last_unstale":"2024-01-16T15:05:20.247151+0000","last_undegraded":"2024-01-16T15:05:20.247151+0000","last_fullsized":"2024-01-16T15:05:20.247151+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:45.588041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252049+0000","last_change":"2024-01-16T15:05:20.246393+0000","last_active":"2024-01-16T15:05:20.252049+0000","last_peered":"2024-01-16T15:05:20.252049+0000","last_clean":"2024-01-16T15:05:20.252049+0000","last_became_active":"2024-01-16T15:05:20.245756+0000","last_became_peered":"2024-01-16T15:05:20.245756+0000","last_unstale":"2024-01-16T15:05:20.252049+0000","last_undegraded":"2024-01-16T15:05:20.252049+0000","last_fullsized":"2024-01-16T15:05:20.252049+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:01.117429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.235517+0000","last_change":"2024-01-16T15:05:20.229665+0000","last_active":"2024-01-16T15:05:20.235517+0000","last_peered":"2024-01-16T15:05:20.235517+0000","last_clean":"2024-01-16T15:05:20.235517+0000","last_became_active":"2024-01-16T15:05:20.229359+0000","last_became_peered":"2024-01-16T15:05:20.229359+0000","last_unstale":"2024-01-16T15:05:20.235517+0000","last_undegraded":"2024-01-16T15:05:20.235517+0000","last_fullsized":"2024-01-16T15:05:20.235517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:45:38.160267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253377+0000","last_change":"2024-01-16T15:05:20.251101+0000","last_active":"2024-01-16T15:05:20.253377+0000","last_peered":"2024-01-16T15:05:20.253377+0000","last_clean":"2024-01-16T15:05:20.253377+0000","last_became_active":"2024-01-16T15:05:20.250798+0000","last_became_peered":"2024-01-16T15:05:20.250798+0000","last_unstale":"2024-01-16T15:05:20.253377+0000","last_undegraded":"2024-01-16T15:05:20.253377+0000","last_fullsized":"2024-01-16T15:05:20.253377+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:57:54.235722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253095+0000","last_change":"2024-01-16T15:05:20.245790+0000","last_active":"2024-01-16T15:05:20.253095+0000","last_peered":"2024-01-16T15:05:20.253095+0000","last_clean":"2024-01-16T15:05:20.253095+0000","last_became_active":"2024-01-16T15:05:20.244190+0000","last_became_peered":"2024-01-16T15:05:20.244190+0000","last_unstale":"2024-01-16T15:05:20.253095+0000","last_undegraded":"2024-01-16T15:05:20.253095+0000","last_fullsized":"2024-01-16T15:05:20.253095+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:37.192618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252064+0000","last_change":"2024-01-16T15:05:20.240511+0000","last_active":"2024-01-16T15:05:20.252064+0000","last_peered":"2024-01-16T15:05:20.252064+0000","last_clean":"2024-01-16T15:05:20.252064+0000","last_became_active":"2024-01-16T15:05:20.240166+0000","last_became_peered":"2024-01-16T15:05:20.240166+0000","last_unstale":"2024-01-16T15:05:20.252064+0000","last_undegraded":"2024-01-16T15:05:20.252064+0000","last_fullsized":"2024-01-16T15:05:20.252064+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:57.672017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"59'4","reported_seq":15,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.245332+0000","last_change":"2024-01-16T15:05:19.227915+0000","last_active":"2024-01-16T15:05:20.245332+0000","last_peered":"2024-01-16T15:05:20.245332+0000","last_clean":"2024-01-16T15:05:20.245332+0000","last_became_active":"2024-01-16T15:05:19.227530+0000","last_became_peered":"2024-01-16T15:05:19.227530+0000","last_unstale":"2024-01-16T15:05:20.245332+0000","last_undegraded":"2024-01-16T15:05:20.245332+0000","last_fullsized":"2024-01-16T15:05:20.245332+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:49.516558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247952+0000","last_change":"2024-01-16T15:05:20.243233+0000","last_active":"2024-01-16T15:05:20.247952+0000","last_peered":"2024-01-16T15:05:20.247952+0000","last_clean":"2024-01-16T15:05:20.247952+0000","last_became_active":"2024-01-16T15:05:20.242946+0000","last_became_peered":"2024-01-16T15:05:20.242946+0000","last_unstale":"2024-01-16T15:05:20.247952+0000","last_undegraded":"2024-01-16T15:05:20.247952+0000","last_fullsized":"2024-01-16T15:05:20.247952+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:10.475751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.251485+0000","last_change":"2024-01-16T15:05:19.235007+0000","last_active":"2024-01-16T15:05:20.251485+0000","last_peered":"2024-01-16T15:05:20.251485+0000","last_clean":"2024-01-16T15:05:20.251485+0000","last_became_active":"2024-01-16T15:05:19.234525+0000","last_became_peered":"2024-01-16T15:05:19.234525+0000","last_unstale":"2024-01-16T15:05:20.251485+0000","last_undegraded":"2024-01-16T15:05:20.251485+0000","last_fullsized":"2024-01-16T15:05:20.251485+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:51:58.597011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.245915+0000","last_change":"2024-01-16T15:05:20.240863+0000","last_active":"2024-01-16T15:05:20.245915+0000","last_peered":"2024-01-16T15:05:20.245915+0000","last_clean":"2024-01-16T15:05:20.245915+0000","last_became_active":"2024-01-16T15:05:20.240048+0000","last_became_peered":"2024-01-16T15:05:20.240048+0000","last_unstale":"2024-01-16T15:05:20.245915+0000","last_undegraded":"2024-01-16T15:05:20.245915+0000","last_fullsized":"2024-01-16T15:05:20.245915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:51.432970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.246947+0000","last_change":"2024-01-16T15:05:19.227210+0000","last_active":"2024-01-16T15:05:20.246947+0000","last_peered":"2024-01-16T15:05:20.246947+0000","last_clean":"2024-01-16T15:05:20.246947+0000","last_became_active":"2024-01-16T15:05:19.224759+0000","last_became_peered":"2024-01-16T15:05:19.224759+0000","last_unstale":"2024-01-16T15:05:20.246947+0000","last_undegraded":"2024-01-16T15:05:20.246947+0000","last_fullsized":"2024-01-16T15:05:20.246947+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:12.268974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242247+0000","last_change":"2024-01-16T15:05:20.238501+0000","last_active":"2024-01-16T15:05:20.242247+0000","last_peered":"2024-01-16T15:05:20.242247+0000","last_clean":"2024-01-16T15:05:20.242247+0000","last_became_active":"2024-01-16T15:05:20.238193+0000","last_became_peered":"2024-01-16T15:05:20.238193+0000","last_unstale":"2024-01-16T15:05:20.242247+0000","last_undegraded":"2024-01-16T15:05:20.242247+0000","last_fullsized":"2024-01-16T15:05:20.242247+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:32.160025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.244067+0000","last_change":"2024-01-16T15:05:19.231086+0000","last_active":"2024-01-16T15:05:20.244067+0000","last_peered":"2024-01-16T15:05:20.244067+0000","last_clean":"2024-01-16T15:05:20.244067+0000","last_became_active":"2024-01-16T15:05:19.230818+0000","last_became_peered":"2024-01-16T15:05:19.230818+0000","last_unstale":"2024-01-16T15:05:20.244067+0000","last_undegraded":"2024-01-16T15:05:20.244067+0000","last_fullsized":"2024-01-16T15:05:20.244067+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:31.148382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243671+0000","last_change":"2024-01-16T15:05:20.238283+0000","last_active":"2024-01-16T15:05:20.243671+0000","last_peered":"2024-01-16T15:05:20.243671+0000","last_clean":"2024-01-16T15:05:20.243671+0000","last_became_active":"2024-01-16T15:05:20.237795+0000","last_became_peered":"2024-01-16T15:05:20.237795+0000","last_unstale":"2024-01-16T15:05:20.243671+0000","last_undegraded":"2024-01-16T15:05:20.243671+0000","last_fullsized":"2024-01-16T15:05:20.243671+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:46:43.007200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"59'1","reported_seq":13,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.217353+0000","last_change":"2024-01-16T15:05:19.227138+0000","last_active":"2024-01-16T15:05:21.217353+0000","last_peered":"2024-01-16T15:05:21.217353+0000","last_clean":"2024-01-16T15:05:21.217353+0000","last_became_active":"2024-01-16T15:05:19.226552+0000","last_became_peered":"2024-01-16T15:05:19.226552+0000","last_unstale":"2024-01-16T15:05:21.217353+0000","last_undegraded":"2024-01-16T15:05:21.217353+0000","last_fullsized":"2024-01-16T15:05:21.217353+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:35.343390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252107+0000","last_change":"2024-01-16T15:05:20.247345+0000","last_active":"2024-01-16T15:05:20.252107+0000","last_peered":"2024-01-16T15:05:20.252107+0000","last_clean":"2024-01-16T15:05:20.252107+0000","last_became_active":"2024-01-16T15:05:20.246041+0000","last_became_peered":"2024-01-16T15:05:20.246041+0000","last_unstale":"2024-01-16T15:05:20.252107+0000","last_undegraded":"2024-01-16T15:05:20.252107+0000","last_fullsized":"2024-01-16T15:05:20.252107+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:30:10.117242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.240583+0000","last_change":"2024-01-16T15:05:19.226406+0000","last_active":"2024-01-16T15:05:20.240583+0000","last_peered":"2024-01-16T15:05:20.240583+0000","last_clean":"2024-01-16T15:05:20.240583+0000","last_became_active":"2024-01-16T15:05:19.226188+0000","last_became_peered":"2024-01-16T15:05:19.226188+0000","last_unstale":"2024-01-16T15:05:20.240583+0000","last_undegraded":"2024-01-16T15:05:20.240583+0000","last_fullsized":"2024-01-16T15:05:20.240583+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:39.002852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423689+0000","last_change":"2024-01-16T15:05:20.218703+0000","last_active":"2024-01-16T15:05:21.423689+0000","last_peered":"2024-01-16T15:05:21.423689+0000","last_clean":"2024-01-16T15:05:21.423689+0000","last_became_active":"2024-01-16T15:05:20.217556+0000","last_became_peered":"2024-01-16T15:05:20.217556+0000","last_unstale":"2024-01-16T15:05:21.423689+0000","last_undegraded":"2024-01-16T15:05:21.423689+0000","last_fullsized":"2024-01-16T15:05:21.423689+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:10:40.745856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.224878+0000","last_change":"2024-01-16T15:05:19.230317+0000","last_active":"2024-01-16T15:05:20.224878+0000","last_peered":"2024-01-16T15:05:20.224878+0000","last_clean":"2024-01-16T15:05:20.224878+0000","last_became_active":"2024-01-16T15:05:19.228514+0000","last_became_peered":"2024-01-16T15:05:19.228514+0000","last_unstale":"2024-01-16T15:05:20.224878+0000","last_undegraded":"2024-01-16T15:05:20.224878+0000","last_fullsized":"2024-01-16T15:05:20.224878+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:34.147115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.238855+0000","last_change":"2024-01-16T15:05:20.234236+0000","last_active":"2024-01-16T15:05:20.238855+0000","last_peered":"2024-01-16T15:05:20.238855+0000","last_clean":"2024-01-16T15:05:20.238855+0000","last_became_active":"2024-01-16T15:05:20.232878+0000","last_became_peered":"2024-01-16T15:05:20.232878+0000","last_unstale":"2024-01-16T15:05:20.238855+0000","last_undegraded":"2024-01-16T15:05:20.238855+0000","last_fullsized":"2024-01-16T15:05:20.238855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:23.127401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216459+0000","last_change":"2024-01-16T15:05:19.229933+0000","last_active":"2024-01-16T15:05:21.216459+0000","last_peered":"2024-01-16T15:05:21.216459+0000","last_clean":"2024-01-16T15:05:21.216459+0000","last_became_active":"2024-01-16T15:05:19.229571+0000","last_became_peered":"2024-01-16T15:05:19.229571+0000","last_unstale":"2024-01-16T15:05:21.216459+0000","last_undegraded":"2024-01-16T15:05:21.216459+0000","last_fullsized":"2024-01-16T15:05:21.216459+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:29:22.696249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.218944+0000","last_change":"2024-01-16T15:05:20.246391+0000","last_active":"2024-01-16T15:05:21.218944+0000","last_peered":"2024-01-16T15:05:21.218944+0000","last_clean":"2024-01-16T15:05:21.218944+0000","last_became_active":"2024-01-16T15:05:20.245765+0000","last_became_peered":"2024-01-16T15:05:20.245765+0000","last_unstale":"2024-01-16T15:05:21.218944+0000","last_undegraded":"2024-01-16T15:05:21.218944+0000","last_fullsized":"2024-01-16T15:05:21.218944+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:52.207259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.214830+0000","last_change":"2024-01-16T15:05:19.224716+0000","last_active":"2024-01-16T15:05:20.214830+0000","last_peered":"2024-01-16T15:05:20.214830+0000","last_clean":"2024-01-16T15:05:20.214830+0000","last_became_active":"2024-01-16T15:05:19.223449+0000","last_became_peered":"2024-01-16T15:05:19.223449+0000","last_unstale":"2024-01-16T15:05:20.214830+0000","last_undegraded":"2024-01-16T15:05:20.214830+0000","last_fullsized":"2024-01-16T15:05:20.214830+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:37.070315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242650+0000","last_change":"2024-01-16T15:05:20.241259+0000","last_active":"2024-01-16T15:05:20.242650+0000","last_peered":"2024-01-16T15:05:20.242650+0000","last_clean":"2024-01-16T15:05:20.242650+0000","last_became_active":"2024-01-16T15:05:20.240922+0000","last_became_peered":"2024-01-16T15:05:20.240922+0000","last_unstale":"2024-01-16T15:05:20.242650+0000","last_undegraded":"2024-01-16T15:05:20.242650+0000","last_fullsized":"2024-01-16T15:05:20.242650+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:26:32.160025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.232920+0000","last_change":"2024-01-16T15:05:19.229019+0000","last_active":"2024-01-16T15:05:20.232920+0000","last_peered":"2024-01-16T15:05:20.232920+0000","last_clean":"2024-01-16T15:05:20.232920+0000","last_became_active":"2024-01-16T15:05:19.228725+0000","last_became_peered":"2024-01-16T15:05:19.228725+0000","last_unstale":"2024-01-16T15:05:20.232920+0000","last_undegraded":"2024-01-16T15:05:20.232920+0000","last_fullsized":"2024-01-16T15:05:20.232920+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:00:08.760888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.254884+0000","last_change":"2024-01-16T15:05:20.251459+0000","last_active":"2024-01-16T15:05:20.254884+0000","last_peered":"2024-01-16T15:05:20.254884+0000","last_clean":"2024-01-16T15:05:20.254884+0000","last_became_active":"2024-01-16T15:05:20.251194+0000","last_became_peered":"2024-01-16T15:05:20.251194+0000","last_unstale":"2024-01-16T15:05:20.254884+0000","last_undegraded":"2024-01-16T15:05:20.254884+0000","last_fullsized":"2024-01-16T15:05:20.254884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:03.339363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"59'3","reported_seq":14,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247432+0000","last_change":"2024-01-16T15:05:19.225885+0000","last_active":"2024-01-16T15:05:20.247432+0000","last_peered":"2024-01-16T15:05:20.247432+0000","last_clean":"2024-01-16T15:05:20.247432+0000","last_became_active":"2024-01-16T15:05:19.225376+0000","last_became_peered":"2024-01-16T15:05:19.225376+0000","last_unstale":"2024-01-16T15:05:20.247432+0000","last_undegraded":"2024-01-16T15:05:20.247432+0000","last_fullsized":"2024-01-16T15:05:20.247432+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:12:50.421327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242822+0000","last_change":"2024-01-16T15:05:20.240925+0000","last_active":"2024-01-16T15:05:20.242822+0000","last_peered":"2024-01-16T15:05:20.242822+0000","last_clean":"2024-01-16T15:05:20.242822+0000","last_became_active":"2024-01-16T15:05:20.240024+0000","last_became_peered":"2024-01-16T15:05:20.240024+0000","last_unstale":"2024-01-16T15:05:20.242822+0000","last_undegraded":"2024-01-16T15:05:20.242822+0000","last_fullsized":"2024-01-16T15:05:20.242822+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:48.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"59'6","reported_seq":18,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.218036+0000","last_change":"2024-01-16T15:05:19.216746+0000","last_active":"2024-01-16T15:05:21.218036+0000","last_peered":"2024-01-16T15:05:21.218036+0000","last_clean":"2024-01-16T15:05:21.218036+0000","last_became_active":"2024-01-16T15:05:19.216359+0000","last_became_peered":"2024-01-16T15:05:19.216359+0000","last_unstale":"2024-01-16T15:05:21.218036+0000","last_undegraded":"2024-01-16T15:05:21.218036+0000","last_fullsized":"2024-01-16T15:05:21.218036+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:07:35.458895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253314+0000","last_change":"2024-01-16T15:05:20.248867+0000","last_active":"2024-01-16T15:05:20.253314+0000","last_peered":"2024-01-16T15:05:20.253314+0000","last_clean":"2024-01-16T15:05:20.253314+0000","last_became_active":"2024-01-16T15:05:20.248277+0000","last_became_peered":"2024-01-16T15:05:20.248277+0000","last_unstale":"2024-01-16T15:05:20.253314+0000","last_undegraded":"2024-01-16T15:05:20.253314+0000","last_fullsized":"2024-01-16T15:05:20.253314+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:50.528201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255602+0000","last_change":"2024-01-16T15:05:19.227725+0000","last_active":"2024-01-16T15:05:20.255602+0000","last_peered":"2024-01-16T15:05:20.255602+0000","last_clean":"2024-01-16T15:05:20.255602+0000","last_became_active":"2024-01-16T15:05:19.227387+0000","last_became_peered":"2024-01-16T15:05:19.227387+0000","last_unstale":"2024-01-16T15:05:20.255602+0000","last_undegraded":"2024-01-16T15:05:20.255602+0000","last_fullsized":"2024-01-16T15:05:20.255602+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:04.014231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.222750+0000","last_change":"2024-01-16T15:05:20.217677+0000","last_active":"2024-01-16T15:05:20.222750+0000","last_peered":"2024-01-16T15:05:20.222750+0000","last_clean":"2024-01-16T15:05:20.222750+0000","last_became_active":"2024-01-16T15:05:20.217092+0000","last_became_peered":"2024-01-16T15:05:20.217092+0000","last_unstale":"2024-01-16T15:05:20.222750+0000","last_undegraded":"2024-01-16T15:05:20.222750+0000","last_fullsized":"2024-01-16T15:05:20.222750+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:37.705713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423137+0000","last_change":"2024-01-16T15:05:19.223095+0000","last_active":"2024-01-16T15:05:21.423137+0000","last_peered":"2024-01-16T15:05:21.423137+0000","last_clean":"2024-01-16T15:05:21.423137+0000","last_became_active":"2024-01-16T15:05:19.222815+0000","last_became_peered":"2024-01-16T15:05:19.222815+0000","last_unstale":"2024-01-16T15:05:21.423137+0000","last_undegraded":"2024-01-16T15:05:21.423137+0000","last_fullsized":"2024-01-16T15:05:21.423137+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:56.417205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.216372+0000","last_change":"2024-01-16T15:05:20.236853+0000","last_active":"2024-01-16T15:05:21.216372+0000","last_peered":"2024-01-16T15:05:21.216372+0000","last_clean":"2024-01-16T15:05:21.216372+0000","last_became_active":"2024-01-16T15:05:20.236400+0000","last_became_peered":"2024-01-16T15:05:20.236400+0000","last_unstale":"2024-01-16T15:05:21.216372+0000","last_undegraded":"2024-01-16T15:05:21.216372+0000","last_fullsized":"2024-01-16T15:05:21.216372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:14.926936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.225273+0000","last_change":"2024-01-16T15:05:19.231103+0000","last_active":"2024-01-16T15:05:20.225273+0000","last_peered":"2024-01-16T15:05:20.225273+0000","last_clean":"2024-01-16T15:05:20.225273+0000","last_became_active":"2024-01-16T15:05:19.230564+0000","last_became_peered":"2024-01-16T15:05:19.230564+0000","last_unstale":"2024-01-16T15:05:20.225273+0000","last_undegraded":"2024-01-16T15:05:20.225273+0000","last_fullsized":"2024-01-16T15:05:20.225273+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:13:11.003015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250778+0000","last_change":"2024-01-16T15:05:20.245528+0000","last_active":"2024-01-16T15:05:20.250778+0000","last_peered":"2024-01-16T15:05:20.250778+0000","last_clean":"2024-01-16T15:05:20.250778+0000","last_became_active":"2024-01-16T15:05:20.244419+0000","last_became_peered":"2024-01-16T15:05:20.244419+0000","last_unstale":"2024-01-16T15:05:20.250778+0000","last_undegraded":"2024-01-16T15:05:20.250778+0000","last_fullsized":"2024-01-16T15:05:20.250778+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.014495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"59'8","reported_seq":19,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.246667+0000","last_change":"2024-01-16T15:05:19.226677+0000","last_active":"2024-01-16T15:05:20.246667+0000","last_peered":"2024-01-16T15:05:20.246667+0000","last_clean":"2024-01-16T15:05:20.246667+0000","last_became_active":"2024-01-16T15:05:19.226289+0000","last_became_peered":"2024-01-16T15:05:19.226289+0000","last_unstale":"2024-01-16T15:05:20.246667+0000","last_undegraded":"2024-01-16T15:05:20.246667+0000","last_fullsized":"2024-01-16T15:05:20.246667+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:08:34.855981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.236111+0000","last_change":"2024-01-16T15:05:20.231030+0000","last_active":"2024-01-16T15:05:20.236111+0000","last_peered":"2024-01-16T15:05:20.236111+0000","last_clean":"2024-01-16T15:05:20.236111+0000","last_became_active":"2024-01-16T15:05:20.230741+0000","last_became_peered":"2024-01-16T15:05:20.230741+0000","last_unstale":"2024-01-16T15:05:20.236111+0000","last_undegraded":"2024-01-16T15:05:20.236111+0000","last_fullsized":"2024-01-16T15:05:20.236111+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:57.672017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.255522+0000","last_change":"2024-01-16T15:05:19.234403+0000","last_active":"2024-01-16T15:05:20.255522+0000","last_peered":"2024-01-16T15:05:20.255522+0000","last_clean":"2024-01-16T15:05:20.255522+0000","last_became_active":"2024-01-16T15:05:19.230980+0000","last_became_peered":"2024-01-16T15:05:19.230980+0000","last_unstale":"2024-01-16T15:05:20.255522+0000","last_undegraded":"2024-01-16T15:05:20.255522+0000","last_fullsized":"2024-01-16T15:05:20.255522+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:49:21.416746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250675+0000","last_change":"2024-01-16T15:05:20.247941+0000","last_active":"2024-01-16T15:05:20.250675+0000","last_peered":"2024-01-16T15:05:20.250675+0000","last_clean":"2024-01-16T15:05:20.250675+0000","last_became_active":"2024-01-16T15:05:20.247645+0000","last_became_peered":"2024-01-16T15:05:20.247645+0000","last_unstale":"2024-01-16T15:05:20.250675+0000","last_undegraded":"2024-01-16T15:05:20.250675+0000","last_fullsized":"2024-01-16T15:05:20.250675+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:37.192618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252000+0000","last_change":"2024-01-16T15:05:19.224765+0000","last_active":"2024-01-16T15:05:20.252000+0000","last_peered":"2024-01-16T15:05:20.252000+0000","last_clean":"2024-01-16T15:05:20.252000+0000","last_became_active":"2024-01-16T15:05:19.224394+0000","last_became_peered":"2024-01-16T15:05:19.224394+0000","last_unstale":"2024-01-16T15:05:20.252000+0000","last_undegraded":"2024-01-16T15:05:20.252000+0000","last_fullsized":"2024-01-16T15:05:20.252000+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:13.820308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253059+0000","last_change":"2024-01-16T15:05:20.248357+0000","last_active":"2024-01-16T15:05:20.253059+0000","last_peered":"2024-01-16T15:05:20.253059+0000","last_clean":"2024-01-16T15:05:20.253059+0000","last_became_active":"2024-01-16T15:05:20.247636+0000","last_became_peered":"2024-01-16T15:05:20.247636+0000","last_unstale":"2024-01-16T15:05:20.253059+0000","last_undegraded":"2024-01-16T15:05:20.253059+0000","last_fullsized":"2024-01-16T15:05:20.253059+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:21.447360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"59'1","reported_seq":12,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243269+0000","last_change":"2024-01-16T15:05:19.228334+0000","last_active":"2024-01-16T15:05:20.243269+0000","last_peered":"2024-01-16T15:05:20.243269+0000","last_clean":"2024-01-16T15:05:20.243269+0000","last_became_active":"2024-01-16T15:05:19.227858+0000","last_became_peered":"2024-01-16T15:05:19.227858+0000","last_unstale":"2024-01-16T15:05:20.243269+0000","last_undegraded":"2024-01-16T15:05:20.243269+0000","last_fullsized":"2024-01-16T15:05:20.243269+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:37.070315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.243688+0000","last_change":"2024-01-16T15:05:20.242055+0000","last_active":"2024-01-16T15:05:20.243688+0000","last_peered":"2024-01-16T15:05:20.243688+0000","last_clean":"2024-01-16T15:05:20.243688+0000","last_became_active":"2024-01-16T15:05:20.241731+0000","last_became_peered":"2024-01-16T15:05:20.241731+0000","last_unstale":"2024-01-16T15:05:20.243688+0000","last_undegraded":"2024-01-16T15:05:20.243688+0000","last_fullsized":"2024-01-16T15:05:20.243688+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:28:35.158758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.221282+0000","last_change":"2024-01-16T15:05:19.236671+0000","last_active":"2024-01-16T15:05:20.221282+0000","last_peered":"2024-01-16T15:05:20.221282+0000","last_clean":"2024-01-16T15:05:20.221282+0000","last_became_active":"2024-01-16T15:05:19.236332+0000","last_became_peered":"2024-01-16T15:05:19.236332+0000","last_unstale":"2024-01-16T15:05:20.221282+0000","last_undegraded":"2024-01-16T15:05:20.221282+0000","last_fullsized":"2024-01-16T15:05:20.221282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:30.621371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253332+0000","last_change":"2024-01-16T15:05:20.249108+0000","last_active":"2024-01-16T15:05:20.253332+0000","last_peered":"2024-01-16T15:05:20.253332+0000","last_clean":"2024-01-16T15:05:20.253332+0000","last_became_active":"2024-01-16T15:05:20.248623+0000","last_became_peered":"2024-01-16T15:05:20.248623+0000","last_unstale":"2024-01-16T15:05:20.253332+0000","last_undegraded":"2024-01-16T15:05:20.253332+0000","last_fullsized":"2024-01-16T15:05:20.253332+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:36.528292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.256400+0000","last_change":"2024-01-16T15:05:19.228416+0000","last_active":"2024-01-16T15:05:20.256400+0000","last_peered":"2024-01-16T15:05:20.256400+0000","last_clean":"2024-01-16T15:05:20.256400+0000","last_became_active":"2024-01-16T15:05:19.227029+0000","last_became_peered":"2024-01-16T15:05:19.227029+0000","last_unstale":"2024-01-16T15:05:20.256400+0000","last_undegraded":"2024-01-16T15:05:20.256400+0000","last_fullsized":"2024-01-16T15:05:20.256400+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:08.433686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242355+0000","last_change":"2024-01-16T15:05:20.236188+0000","last_active":"2024-01-16T15:05:20.242355+0000","last_peered":"2024-01-16T15:05:20.242355+0000","last_clean":"2024-01-16T15:05:20.242355+0000","last_became_active":"2024-01-16T15:05:20.235807+0000","last_became_peered":"2024-01-16T15:05:20.235807+0000","last_unstale":"2024-01-16T15:05:20.242355+0000","last_undegraded":"2024-01-16T15:05:20.242355+0000","last_fullsized":"2024-01-16T15:05:20.242355+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:38.081958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.222052+0000","last_change":"2024-01-16T15:05:19.231277+0000","last_active":"2024-01-16T15:05:20.222052+0000","last_peered":"2024-01-16T15:05:20.222052+0000","last_clean":"2024-01-16T15:05:20.222052+0000","last_became_active":"2024-01-16T15:05:19.225403+0000","last_became_peered":"2024-01-16T15:05:19.225403+0000","last_unstale":"2024-01-16T15:05:20.222052+0000","last_undegraded":"2024-01-16T15:05:20.222052+0000","last_fullsized":"2024-01-16T15:05:20.222052+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:13.484758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.232287+0000","last_change":"2024-01-16T15:05:20.226031+0000","last_active":"2024-01-16T15:05:20.232287+0000","last_peered":"2024-01-16T15:05:20.232287+0000","last_clean":"2024-01-16T15:05:20.232287+0000","last_became_active":"2024-01-16T15:05:20.225735+0000","last_became_peered":"2024-01-16T15:05:20.225735+0000","last_unstale":"2024-01-16T15:05:20.232287+0000","last_undegraded":"2024-01-16T15:05:20.232287+0000","last_fullsized":"2024-01-16T15:05:20.232287+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:48.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.225648+0000","last_change":"2024-01-16T15:05:19.237578+0000","last_active":"2024-01-16T15:05:20.225648+0000","last_peered":"2024-01-16T15:05:20.225648+0000","last_clean":"2024-01-16T15:05:20.225648+0000","last_became_active":"2024-01-16T15:05:19.237373+0000","last_became_peered":"2024-01-16T15:05:19.237373+0000","last_unstale":"2024-01-16T15:05:20.225648+0000","last_undegraded":"2024-01-16T15:05:20.225648+0000","last_fullsized":"2024-01-16T15:05:20.225648+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:12.419974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242621+0000","last_change":"2024-01-16T15:05:20.237130+0000","last_active":"2024-01-16T15:05:20.242621+0000","last_peered":"2024-01-16T15:05:20.242621+0000","last_clean":"2024-01-16T15:05:20.242621+0000","last_became_active":"2024-01-16T15:05:20.236569+0000","last_became_peered":"2024-01-16T15:05:20.236569+0000","last_unstale":"2024-01-16T15:05:20.242621+0000","last_undegraded":"2024-01-16T15:05:20.242621+0000","last_fullsized":"2024-01-16T15:05:20.242621+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:45:38.160267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"59'2","reported_seq":13,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.254344+0000","last_change":"2024-01-16T15:05:19.235333+0000","last_active":"2024-01-16T15:05:20.254344+0000","last_peered":"2024-01-16T15:05:20.254344+0000","last_clean":"2024-01-16T15:05:20.254344+0000","last_became_active":"2024-01-16T15:05:19.235056+0000","last_became_peered":"2024-01-16T15:05:19.235056+0000","last_unstale":"2024-01-16T15:05:20.254344+0000","last_undegraded":"2024-01-16T15:05:20.254344+0000","last_fullsized":"2024-01-16T15:05:20.254344+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:11:35.962422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.242721+0000","last_change":"2024-01-16T15:05:20.237813+0000","last_active":"2024-01-16T15:05:20.242721+0000","last_peered":"2024-01-16T15:05:20.242721+0000","last_clean":"2024-01-16T15:05:20.242721+0000","last_became_active":"2024-01-16T15:05:20.237005+0000","last_became_peered":"2024-01-16T15:05:20.237005+0000","last_unstale":"2024-01-16T15:05:20.242721+0000","last_undegraded":"2024-01-16T15:05:20.242721+0000","last_fullsized":"2024-01-16T15:05:20.242721+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:49:48.598197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.221674+0000","last_change":"2024-01-16T15:05:19.226502+0000","last_active":"2024-01-16T15:05:20.221674+0000","last_peered":"2024-01-16T15:05:20.221674+0000","last_clean":"2024-01-16T15:05:20.221674+0000","last_became_active":"2024-01-16T15:05:19.226217+0000","last_became_peered":"2024-01-16T15:05:19.226217+0000","last_unstale":"2024-01-16T15:05:20.221674+0000","last_undegraded":"2024-01-16T15:05:20.221674+0000","last_fullsized":"2024-01-16T15:05:20.221674+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:06:03.890562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.254873+0000","last_change":"2024-01-16T15:05:20.249136+0000","last_active":"2024-01-16T15:05:20.254873+0000","last_peered":"2024-01-16T15:05:20.254873+0000","last_clean":"2024-01-16T15:05:20.254873+0000","last_became_active":"2024-01-16T15:05:20.248868+0000","last_became_peered":"2024-01-16T15:05:20.248868+0000","last_unstale":"2024-01-16T15:05:20.254873+0000","last_undegraded":"2024-01-16T15:05:20.254873+0000","last_fullsized":"2024-01-16T15:05:20.254873+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:50:42.584977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423622+0000","last_change":"2024-01-16T15:05:19.225718+0000","last_active":"2024-01-16T15:05:21.423622+0000","last_peered":"2024-01-16T15:05:21.423622+0000","last_clean":"2024-01-16T15:05:21.423622+0000","last_became_active":"2024-01-16T15:05:19.225157+0000","last_became_peered":"2024-01-16T15:05:19.225157+0000","last_unstale":"2024-01-16T15:05:21.423622+0000","last_undegraded":"2024-01-16T15:05:21.423622+0000","last_fullsized":"2024-01-16T15:05:21.423622+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:57:18.591656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.245074+0000","last_change":"2024-01-16T15:05:20.240238+0000","last_active":"2024-01-16T15:05:20.245074+0000","last_peered":"2024-01-16T15:05:20.245074+0000","last_clean":"2024-01-16T15:05:20.245074+0000","last_became_active":"2024-01-16T15:05:20.239928+0000","last_became_peered":"2024-01-16T15:05:20.239928+0000","last_unstale":"2024-01-16T15:05:20.245074+0000","last_undegraded":"2024-01-16T15:05:20.245074+0000","last_fullsized":"2024-01-16T15:05:20.245074+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:13.431617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.226282+0000","last_change":"2024-01-16T15:05:19.227933+0000","last_active":"2024-01-16T15:05:20.226282+0000","last_peered":"2024-01-16T15:05:20.226282+0000","last_clean":"2024-01-16T15:05:20.226282+0000","last_became_active":"2024-01-16T15:05:19.227287+0000","last_became_peered":"2024-01-16T15:05:19.227287+0000","last_unstale":"2024-01-16T15:05:20.226282+0000","last_undegraded":"2024-01-16T15:05:20.226282+0000","last_fullsized":"2024-01-16T15:05:20.226282+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:08:56.660374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.252159+0000","last_change":"2024-01-16T15:05:20.244766+0000","last_active":"2024-01-16T15:05:20.252159+0000","last_peered":"2024-01-16T15:05:20.252159+0000","last_clean":"2024-01-16T15:05:20.252159+0000","last_became_active":"2024-01-16T15:05:20.244495+0000","last_became_peered":"2024-01-16T15:05:20.244495+0000","last_unstale":"2024-01-16T15:05:20.252159+0000","last_undegraded":"2024-01-16T15:05:20.252159+0000","last_fullsized":"2024-01-16T15:05:20.252159+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:28:23.544704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.229827+0000","last_change":"2024-01-16T15:05:19.232700+0000","last_active":"2024-01-16T15:05:20.229827+0000","last_peered":"2024-01-16T15:05:20.229827+0000","last_clean":"2024-01-16T15:05:20.229827+0000","last_became_active":"2024-01-16T15:05:19.231172+0000","last_became_peered":"2024-01-16T15:05:19.231172+0000","last_unstale":"2024-01-16T15:05:20.229827+0000","last_undegraded":"2024-01-16T15:05:20.229827+0000","last_fullsized":"2024-01-16T15:05:20.229827+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:29:57.272125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.250788+0000","last_change":"2024-01-16T15:05:20.247212+0000","last_active":"2024-01-16T15:05:20.250788+0000","last_peered":"2024-01-16T15:05:20.250788+0000","last_clean":"2024-01-16T15:05:20.250788+0000","last_became_active":"2024-01-16T15:05:20.246855+0000","last_became_peered":"2024-01-16T15:05:20.246855+0000","last_unstale":"2024-01-16T15:05:20.250788+0000","last_undegraded":"2024-01-16T15:05:20.250788+0000","last_fullsized":"2024-01-16T15:05:20.250788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:13.431617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.217051+0000","last_change":"2024-01-16T15:05:19.224066+0000","last_active":"2024-01-16T15:05:21.217051+0000","last_peered":"2024-01-16T15:05:21.217051+0000","last_clean":"2024-01-16T15:05:21.217051+0000","last_became_active":"2024-01-16T15:05:19.223742+0000","last_became_peered":"2024-01-16T15:05:19.223742+0000","last_unstale":"2024-01-16T15:05:21.217051+0000","last_undegraded":"2024-01-16T15:05:21.217051+0000","last_fullsized":"2024-01-16T15:05:21.217051+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:16:34.042174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.247869+0000","last_change":"2024-01-16T15:05:20.243131+0000","last_active":"2024-01-16T15:05:20.247869+0000","last_peered":"2024-01-16T15:05:20.247869+0000","last_clean":"2024-01-16T15:05:20.247869+0000","last_became_active":"2024-01-16T15:05:20.242844+0000","last_became_peered":"2024-01-16T15:05:20.242844+0000","last_unstale":"2024-01-16T15:05:20.247869+0000","last_undegraded":"2024-01-16T15:05:20.247869+0000","last_fullsized":"2024-01-16T15:05:20.247869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:34:01.528474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.221924+0000","last_change":"2024-01-16T15:05:19.223723+0000","last_active":"2024-01-16T15:05:20.221924+0000","last_peered":"2024-01-16T15:05:20.221924+0000","last_clean":"2024-01-16T15:05:20.221924+0000","last_became_active":"2024-01-16T15:05:19.223421+0000","last_became_peered":"2024-01-16T15:05:19.223421+0000","last_unstale":"2024-01-16T15:05:20.221924+0000","last_undegraded":"2024-01-16T15:05:20.221924+0000","last_fullsized":"2024-01-16T15:05:20.221924+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:43.062084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.237185+0000","last_change":"2024-01-16T15:05:20.228580+0000","last_active":"2024-01-16T15:05:20.237185+0000","last_peered":"2024-01-16T15:05:20.237185+0000","last_clean":"2024-01-16T15:05:20.237185+0000","last_became_active":"2024-01-16T15:05:20.228285+0000","last_became_peered":"2024-01-16T15:05:20.228285+0000","last_unstale":"2024-01-16T15:05:20.237185+0000","last_undegraded":"2024-01-16T15:05:20.237185+0000","last_fullsized":"2024-01-16T15:05:20.237185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:56:32.091740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":11,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.228510+0000","last_change":"2024-01-16T15:05:19.235230+0000","last_active":"2024-01-16T15:05:20.228510+0000","last_peered":"2024-01-16T15:05:20.228510+0000","last_clean":"2024-01-16T15:05:20.228510+0000","last_became_active":"2024-01-16T15:05:19.234945+0000","last_became_peered":"2024-01-16T15:05:19.234945+0000","last_unstale":"2024-01-16T15:05:20.228510+0000","last_undegraded":"2024-01-16T15:05:20.228510+0000","last_fullsized":"2024-01-16T15:05:20.228510+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:45.715224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":9,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-01-16T15:05:20.253110+0000","last_change":"2024-01-16T15:05:20.248535+0000","last_active":"2024-01-16T15:05:20.253110+0000","last_peered":"2024-01-16T15:05:20.253110+0000","last_clean":"2024-01-16T15:05:20.253110+0000","last_became_active":"2024-01-16T15:05:20.247506+0000","last_became_peered":"2024-01-16T15:05:20.247506+0000","last_unstale":"2024-01-16T15:05:20.253110+0000","last_undegraded":"2024-01-16T15:05:20.253110+0000","last_fullsized":"2024-01-16T15:05:20.253110+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:19.185996+0000","last_clean_scrub_stamp":"2024-01-16T15:05:19.185996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:35:55.299913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-01-16T15:05:21.423329+0000","last_change":"2024-01-16T15:05:19.231368+0000","last_active":"2024-01-16T15:05:21.423329+0000","last_peered":"2024-01-16T15:05:21.423329+0000","last_clean":"2024-01-16T15:05:21.423329+0000","last_became_active":"2024-01-16T15:05:19.231051+0000","last_became_peered":"2024-01-16T15:05:19.231051+0000","last_unstale":"2024-01-16T15:05:21.423329+0000","last_undegraded":"2024-01-16T15:05:21.423329+0000","last_fullsized":"2024-01-16T15:05:21.423329+0000","mapping_epoch":57,"log_start":"0'0","ondisk_log_start":"0'0","created":57,"last_epoch_clean":58,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:05:18.174353+0000","last_clean_scrub_stamp":"2024-01-16T15:05:18.174353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:50:28.750083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":200,"reported_epoch":58,"state":"active+clean","last_fresh":"2024-01-16T15:05:19.210941+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:05:19.210941+0000","last_peered":"2024-01-16T15:05:19.210941+0000","last_clean":"2024-01-16T15:05:19.210941+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:05:19.210941+0000","last_undegraded":"2024-01-16T15:05:19.210941+0000","last_fullsized":"2024-01-16T15:05:19.210941+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509509,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36024,"kb_used_data":684,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335816,"statfs":{"total":96636764160,"available":96599875584,"internally_reserved":0,"allocated":700416,"data_stored":291407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.9630000000000001,"5min":1.3560000000000001,"15min":1.208},"min":{"1min":0.746,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":6.4809999999999999,"5min":9.7899999999999991,"15min":9.7899999999999991},"last":1.3440000000000001},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.2929999999999999,"15min":1.2090000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":9.5920000000000005,"5min":9.5920000000000005,"15min":9.5920000000000005},"last":1.569}]},{"osd":1,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.5649999999999999,"15min":1.264},"min":{"1min":0.745,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":12.577999999999999,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.014},{"interface":"front","average":{"1min":1.831,"5min":1.353,"15min":1.2809999999999999},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":6.5300000000000002,"5min":6.5300000000000002,"15min":6.5300000000000002},"last":1.121}]},{"osd":2,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.6779999999999999,"5min":1.3919999999999999,"15min":1.3939999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":5.931,"5min":5.931,"15min":5.931},"last":1.242},{"interface":"front","average":{"1min":1.6160000000000001,"5min":1.335,"15min":1.377},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":9.6669999999999998,"5min":9.6669999999999998,"15min":9.6669999999999998},"last":1.6779999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6440000000000001,"5min":1.661,"15min":1.516},"min":{"1min":0.91900000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":12.654999999999999,"5min":12.654999999999999,"15min":12.654999999999999},"last":1.2769999999999999},{"interface":"front","average":{"1min":2.153,"5min":1.5660000000000001,"15min":1.591},"min":{"1min":0.80000000000000004,"5min":0.745,"15min":0.745},"max":{"1min":8.657,"5min":8.657,"15min":8.657},"last":0.93100000000000005}]},{"osd":4,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.804,"5min":1.4350000000000001,"15min":1.482},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":9.6980000000000004,"5min":9.6980000000000004,"15min":9.6980000000000004},"last":1.3959999999999999},{"interface":"front","average":{"1min":3.3359999999999999,"5min":1.851,"15min":1.671},"min":{"1min":0.92400000000000004,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":12.506,"5min":12.506,"15min":12.506},"last":1.73}]},{"osd":5,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":3.4729999999999999,"5min":2.0470000000000002,"15min":1.8169999999999999},"min":{"1min":1.339,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":12.606,"5min":12.606,"15min":12.606},"last":1.956},{"interface":"front","average":{"1min":2.1579999999999999,"5min":1.698,"15min":1.6759999999999999},"min":{"1min":0.75900000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":9.5449999999999999,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":1.5149999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:58 2024","interfaces":[{"interface":"back","average":{"1min":1.992,"5min":1.655,"15min":1.637},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":9.7309999999999999,"5min":9.7309999999999999,"15min":9.7309999999999999},"last":1.792},{"interface":"front","average":{"1min":2.9359999999999999,"5min":1.9319999999999999,"15min":1.8069999999999999},"min":{"1min":1.2889999999999999,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":12.630000000000001,"5min":12.630000000000001,"15min":12.630000000000001},"last":1.859}]}]},{"osd":4,"up_from":15,"seq":64424509511,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36056,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335784,"statfs":{"total":96636764160,"available":96599842816,"internally_reserved":0,"allocated":724992,"data_stored":299319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.931,"5min":1.49,"15min":1.1719999999999999},"min":{"1min":0.75900000000000001,"5min":0.622,"15min":0.622},"max":{"1min":6.4619999999999997,"5min":14.993,"15min":14.993},"last":1.012},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.266,"15min":1.167},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":1.351}]},{"osd":1,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":2.383,"5min":1.4990000000000001,"15min":1.2569999999999999},"min":{"1min":0.57299999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":10.821999999999999,"5min":10.821999999999999,"15min":10.821999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.7470000000000001,"5min":1.391,"15min":1.276},"min":{"1min":0.88300000000000001,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":6.6660000000000004,"5min":6.6660000000000004,"15min":6.6660000000000004},"last":1.3169999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":2.4129999999999998,"5min":1.571,"15min":1.3480000000000001},"min":{"1min":0.88400000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":16.457999999999998,"5min":16.457999999999998,"15min":16.457999999999998},"last":1.294},{"interface":"front","average":{"1min":1.6990000000000001,"5min":1.4319999999999999,"15min":1.3280000000000001},"min":{"1min":0.68799999999999994,"5min":0.627,"15min":0.627},"max":{"1min":6.7389999999999999,"5min":6.7389999999999999,"15min":6.7389999999999999},"last":1.5669999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":3.4660000000000002,"5min":1.7450000000000001,"15min":1.4470000000000001},"min":{"1min":1.1419999999999999,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":26.741,"5min":26.741,"15min":26.741},"last":1.506},{"interface":"front","average":{"1min":3.4910000000000001,"5min":1.948,"15min":1.5780000000000001},"min":{"1min":0.95499999999999996,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":26.701000000000001,"5min":26.701000000000001,"15min":26.701000000000001},"last":1.4770000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":1.6279999999999999,"15min":1.536},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":6.5949999999999998,"5min":11.823,"15min":11.823},"last":1.228},{"interface":"front","average":{"1min":3.327,"5min":1.9279999999999999,"15min":1.647},"min":{"1min":0.72599999999999998,"5min":0.502,"15min":0.502},"max":{"1min":26.597000000000001,"5min":26.597000000000001,"15min":26.597000000000001},"last":0.85399999999999998}]},{"osd":6,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":2.3220000000000001,"5min":1.6739999999999999,"15min":1.556},"min":{"1min":0.54800000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":10.986000000000001,"5min":10.986000000000001,"15min":10.986000000000001},"last":1.163},{"interface":"front","average":{"1min":1.871,"5min":1.6579999999999999,"15min":1.5840000000000001},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":6.7839999999999998,"5min":6.7839999999999998,"15min":6.7839999999999998},"last":1.4470000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:05:01 2024","interfaces":[{"interface":"back","average":{"1min":3.351,"5min":1.865,"15min":1.671},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":26.683,"5min":26.683,"15min":26.683},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.907,"15min":1.7310000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":6.5410000000000004,"5min":13.561999999999999,"15min":13.561999999999999},"last":1.5329999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542213,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36596,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335244,"statfs":{"total":96636764160,"available":96599289856,"internally_reserved":0,"allocated":1282048,"data_stored":875000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.2070000000000001,"15min":1.0780000000000001},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":8.5180000000000007,"5min":8.5180000000000007,"15min":8.5180000000000007},"last":0.95699999999999996},{"interface":"front","average":{"1min":2.8889999999999998,"5min":1.4550000000000001,"15min":1.175},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":30.126000000000001,"5min":30.126000000000001,"15min":30.126000000000001},"last":1.282}]},{"osd":1,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.1990000000000001,"15min":1.1399999999999999},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":3.339,"5min":3.339,"15min":3.339},"last":1.0720000000000001},{"interface":"front","average":{"1min":2.8250000000000002,"5min":1.593,"15min":1.3},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":21.032,"5min":21.032,"15min":21.032},"last":1.3480000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.4379999999999999,"5min":1.3300000000000001,"15min":1.274},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":1.458},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.369,"15min":1.29},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":4.4989999999999997,"5min":4.4989999999999997,"15min":4.4989999999999997},"last":1.3759999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3899999999999999,"15min":1.3520000000000001},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.752,"5min":4.2249999999999996,"15min":4.2249999999999996},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.3979999999999999,"15min":1.3540000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":3.4300000000000002,"5min":3.653,"15min":3.653},"last":1.4330000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8080000000000001,"5min":1.4950000000000001,"15min":1.4279999999999999},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":14.659000000000001,"5min":14.659000000000001,"15min":14.659000000000001},"last":1.143},{"interface":"front","average":{"1min":1.5880000000000001,"5min":1.544,"15min":1.45},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":5.8650000000000002,"5min":11.433,"15min":11.433},"last":1.246}]},{"osd":5,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":2.2869999999999999,"5min":1.599,"15min":1.4670000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":18.132000000000001,"5min":18.132000000000001,"15min":18.132000000000001},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.869,"5min":1.641,"15min":1.587},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":9.2370000000000001,"5min":9.2370000000000001,"15min":9.2370000000000001},"last":1.3169999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.542,"15min":1.55},"min":{"1min":0.72099999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":3.4649999999999999,"5min":3.7730000000000001,"15min":3.7730000000000001},"last":1.486},{"interface":"front","average":{"1min":3.214,"5min":1.915,"15min":1.6819999999999999},"min":{"1min":0.68400000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"max":{"1min":29.991,"5min":29.991,"15min":29.991},"last":1.204}]}]},{"osd":5,"up_from":14,"seq":60129542214,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36016,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335824,"statfs":{"total":96636764160,"available":96599883776,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.2529999999999999,"15min":1.099},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.2509999999999999},{"interface":"front","average":{"1min":2.0760000000000001,"5min":1.417,"15min":1.165},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":11.109,"5min":11.109,"15min":11.109},"last":1.073}]},{"osd":1,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.244,"15min":1.1519999999999999},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":5.0750000000000002,"5min":5.0750000000000002,"15min":5.0750000000000002},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.819,"5min":1.3280000000000001,"15min":1.1619999999999999},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":9.5210000000000008,"5min":9.5210000000000008,"15min":9.5210000000000008},"last":1.1220000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":1.359,"15min":1.236},"min":{"1min":0.66900000000000004,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":4.7949999999999999,"5min":4.7949999999999999,"15min":4.7949999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":2.3929999999999998,"5min":1.4970000000000001,"15min":1.2929999999999999},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":11.321,"5min":11.321,"15min":11.321},"last":0.84699999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":2.0219999999999998,"5min":1.4650000000000001,"15min":1.381},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":6.7450000000000001,"5min":6.7450000000000001,"15min":6.7450000000000001},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.754,"5min":1.5860000000000001,"15min":1.4350000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":6.1669999999999998,"5min":14.647,"15min":14.647},"last":1.3959999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":2.9780000000000002,"5min":1.675,"15min":1.4219999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":24.436,"5min":24.436,"15min":24.436},"last":1.3520000000000001},{"interface":"front","average":{"1min":1.54,"5min":1.4419999999999999,"15min":1.4039999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.4900000000000002,"5min":4.4900000000000002,"15min":4.4900000000000002},"last":1.504}]},{"osd":6,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":2.9319999999999999,"5min":1.774,"15min":1.4830000000000001},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":21.192,"5min":21.192,"15min":21.192},"last":0.98099999999999998},{"interface":"front","average":{"1min":1.8300000000000001,"5min":1.536,"15min":1.4410000000000001},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":6.0970000000000004,"5min":6.0970000000000004,"15min":6.0970000000000004},"last":1.208}]},{"osd":7,"last update":"Tue Jan 16 15:04:50 2024","interfaces":[{"interface":"back","average":{"1min":3.7050000000000001,"5min":1.95,"15min":1.6879999999999999},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":32.405999999999999,"5min":32.405999999999999,"15min":32.405999999999999},"last":1.286},{"interface":"front","average":{"1min":3.0920000000000001,"5min":1.9099999999999999,"15min":1.74},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":24.367000000000001,"5min":24.367000000000001,"15min":24.367000000000001},"last":1.456}]}]},{"osd":3,"up_from":14,"seq":60129542214,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40172,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331668,"statfs":{"total":96636764160,"available":96595628032,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":2.7869999999999999,"5min":1.302,"15min":0.96599999999999997},"min":{"1min":0.36199999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":31.562999999999999,"5min":31.562999999999999,"15min":31.562999999999999},"last":0.996},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.153,"15min":1.0289999999999999},"min":{"1min":0.49299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":8.4879999999999995,"5min":8.4879999999999995,"15min":8.4879999999999995},"last":0.75900000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.0529999999999999,"15min":0.91600000000000004},"min":{"1min":0.60999999999999999,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":3.4780000000000002,"5min":5.4269999999999996,"15min":5.4269999999999996},"last":1.083},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.2989999999999999,"15min":1.101},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":6.992,"5min":6.992,"15min":6.992},"last":1.129}]},{"osd":2,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.371,"15min":1.171},"min":{"1min":0.68000000000000005,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":7.0529999999999999,"5min":7.0529999999999999,"15min":7.0529999999999999},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4119999999999999,"15min":1.1890000000000001},"min":{"1min":0.64700000000000002,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":7.9729999999999999,"5min":14.577999999999999,"15min":14.577999999999999},"last":0.84499999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.4490000000000001,"15min":1.3069999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":7.7990000000000004,"5min":7.7990000000000004,"15min":7.7990000000000004},"last":1.52},{"interface":"front","average":{"1min":2.109,"5min":1.6180000000000001,"15min":1.462},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":7.1379999999999999,"5min":7.1379999999999999,"15min":7.1379999999999999},"last":2.0110000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.79,"5min":1.5469999999999999,"15min":1.4359999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":5.9489999999999998,"5min":5.9489999999999998,"15min":5.9489999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":2.625,"5min":1.788,"15min":1.544},"min":{"1min":0.90600000000000003,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":17.613,"5min":17.613,"15min":17.613},"last":1.931}]},{"osd":6,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":3.0059999999999998,"5min":1.887,"15min":1.5580000000000001},"min":{"1min":1.0449999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":13.182,"5min":13.182,"15min":13.182},"last":1.4059999999999999},{"interface":"front","average":{"1min":2.4790000000000001,"5min":1.7569999999999999,"15min":1.498},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":18.481000000000002,"5min":18.481000000000002,"15min":18.481000000000002},"last":1.748}]},{"osd":7,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.6399999999999999,"15min":1.4890000000000001},"min":{"1min":0.89600000000000002,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":3.2829999999999999,"5min":4.0830000000000002,"15min":4.0830000000000002},"last":1.1930000000000001},{"interface":"front","average":{"1min":2.6469999999999998,"5min":1.8540000000000001,"15min":1.5860000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":19.183,"5min":19.183,"15min":19.183},"last":1.595}]}]},{"osd":2,"up_from":14,"seq":60129542213,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40176,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.94499999999999995,"15min":1.0069999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.4990000000000001,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.026,"15min":1.0229999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":3.6850000000000001,"5min":3.6850000000000001,"15min":3.6850000000000001},"last":0.999}]},{"osd":1,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.0600000000000001,"15min":1.0429999999999999},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7909999999999999,"5min":3.839,"15min":3.839},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.177,"5min":1.135,"15min":1.0860000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.091}]},{"osd":3,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1399999999999999,"15min":0.96599999999999997},"min":{"1min":0.503,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":2.6179999999999999,"5min":9.1530000000000005,"15min":9.1530000000000005},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.325,"15min":1.3149999999999999},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":2.1459999999999999,"5min":2.4289999999999998,"15min":2.4289999999999998},"last":1.6910000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.5449999999999999,"5min":1.3580000000000001,"15min":1.3280000000000001},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":6.4459999999999997,"5min":6.4459999999999997,"15min":6.4459999999999997},"last":1.597},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.4330000000000001,"15min":1.4299999999999999},"min":{"1min":0.84799999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"max":{"1min":4.859,"5min":4.859,"15min":4.859},"last":1.1539999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.448,"15min":1.3520000000000001},"min":{"1min":0.95699999999999996,"5min":0.753,"15min":0.753},"max":{"1min":4.3609999999999998,"5min":9.6189999999999998,"15min":9.6189999999999998},"last":1.4950000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.48,"15min":1.3919999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":2.1469999999999998,"5min":9.423,"15min":9.423},"last":1.323}]},{"osd":6,"last update":"Tue Jan 16 15:04:56 2024","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.512,"15min":1.397},"min":{"1min":0.93100000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.0310000000000001,"5min":9.5359999999999996,"15min":9.5359999999999996},"last":1.647},{"interface":"front","average":{"1min":3.3250000000000002,"5min":1.966,"15min":1.6319999999999999},"min":{"1min":0.89300000000000002,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":21.564,"5min":21.564,"15min":21.564},"last":1.4259999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:53 2024","interfaces":[{"interface":"back","average":{"1min":1.4670000000000001,"5min":1.4710000000000001,"15min":1.51},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.0550000000000002,"5min":2.4169999999999998,"15min":2.4169999999999998},"last":1.444},{"interface":"front","average":{"1min":1.613,"5min":1.5229999999999999,"15min":1.577},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":4.2510000000000003,"5min":4.2510000000000003,"15min":4.2510000000000003},"last":1.3069999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542215,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40176,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331664,"statfs":{"total":96636764160,"available":96595623936,"internally_reserved":0,"allocated":688128,"data_stored":284632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":1.04,"15min":0.873},"min":{"1min":0.41599999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.954,"5min":10.667999999999999,"15min":10.667999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.1259999999999999,"15min":0.96799999999999997},"min":{"1min":0.59099999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":4.8410000000000002},"last":1.1020000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":1.03,"15min":0.98199999999999998},"min":{"1min":0.53600000000000003,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.3799999999999999,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.05},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.3480000000000001,"15min":1.206},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.772,"5min":10.805,"15min":10.805},"last":1.1339999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.3899999999999999,"15min":1.1839999999999999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":3.9159999999999999,"5min":10.991,"15min":10.991},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.952,"5min":1.393,"15min":1.161},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":10.430999999999999,"5min":10.430999999999999,"15min":10.430999999999999},"last":1.0089999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.284,"15min":1.284},"min":{"1min":0.747,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.236,"5min":1.5489999999999999,"15min":1.4950000000000001},"min":{"1min":0.72999999999999998,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.875,"5min":11.157999999999999,"15min":11.157999999999999},"last":1.29}]},{"osd":5,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.4890000000000001,"15min":1.387},"min":{"1min":0.71199999999999997,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":4.9359999999999999,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":1.341},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.5429999999999999,"15min":1.4770000000000001},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":3.7530000000000001,"5min":3.7530000000000001,"15min":3.7530000000000001},"last":1.1799999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":2.2160000000000002,"5min":1.6799999999999999,"15min":1.536},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":10.057,"5min":10.057,"15min":10.057},"last":1.23},{"interface":"front","average":{"1min":2.4039999999999999,"5min":1.734,"15min":1.554},"min":{"1min":0.81200000000000006,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":18.850999999999999,"5min":18.850999999999999,"15min":18.850999999999999},"last":1.3680000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:04:59 2024","interfaces":[{"interface":"back","average":{"1min":4.0730000000000004,"5min":2.0859999999999999,"15min":1.728},"min":{"1min":1.1890000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":30.834,"5min":30.834,"15min":30.834},"last":1.3149999999999999},{"interface":"front","average":{"1min":3.069,"5min":2.0099999999999998,"15min":1.657},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":27.263999999999999,"5min":27.263999999999999,"15min":27.263999999999999},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542213,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40756,"kb_used_data":1252,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331084,"statfs":{"total":96636764160,"available":96595030016,"internally_reserved":0,"allocated":1282048,"data_stored":875000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.944,"5min":1.1759999999999999,"15min":1.022},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":22.132999999999999,"5min":22.132999999999999,"15min":22.132999999999999},"last":0.69899999999999995},{"interface":"front","average":{"1min":4.1799999999999997,"5min":1.681,"15min":1.256},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":50.429000000000002,"5min":50.429000000000002,"15min":50.429000000000002},"last":0.51200000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.202,"5min":1.478,"15min":1.23},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":22.716999999999999,"5min":22.716999999999999,"15min":22.716999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":4.4909999999999997,"5min":1.867,"15min":1.349},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":50.581000000000003,"5min":50.581000000000003,"15min":50.581000000000003},"last":0.59899999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.2809999999999999,"15min":1.1699999999999999},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":4.6070000000000002,"5min":10.795,"15min":10.795},"last":0.65300000000000002},{"interface":"front","average":{"1min":2.1560000000000001,"5min":1.5940000000000001,"15min":1.3799999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":22.254999999999999,"5min":22.254999999999999,"15min":22.254999999999999},"last":0.624}]},{"osd":4,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.6299999999999999,"5min":1.7030000000000001,"15min":1.4790000000000001},"min":{"1min":0.72599999999999998,"5min":0.625,"15min":0.625},"max":{"1min":22.407,"5min":22.407,"15min":22.407},"last":1.1910000000000001},{"interface":"front","average":{"1min":4.726,"5min":2.153,"15min":1.569},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":50.497,"5min":50.497,"15min":50.497},"last":0.88800000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":2.9449999999999998,"5min":1.841,"15min":1.5049999999999999},"min":{"1min":0.70999999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":14.112,"5min":14.112,"15min":14.112},"last":0.73799999999999999},{"interface":"front","average":{"1min":2.5840000000000001,"5min":1.7270000000000001,"15min":1.5740000000000001},"min":{"1min":0.78000000000000003,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":22.882999999999999,"5min":22.882999999999999,"15min":22.882999999999999},"last":1.1020000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":3.048,"5min":1.857,"15min":1.554},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":23.405000000000001,"5min":23.405000000000001,"15min":23.405000000000001},"last":0.86099999999999999},{"interface":"front","average":{"1min":3.3090000000000002,"5min":1.911,"15min":1.661},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":23.547000000000001,"5min":23.547000000000001,"15min":23.547000000000001},"last":1.1519999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:04:52 2024","interfaces":[{"interface":"back","average":{"1min":5.2839999999999998,"5min":2.2959999999999998,"15min":1.786},"min":{"1min":0.86399999999999999,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":50.542000000000002,"5min":50.542000000000002,"15min":50.542000000000002},"last":0.96999999999999997},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.7529999999999999,"15min":1.6120000000000001},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.9850000000000003,"5min":10.736000000000001,"15min":10.736000000000001},"last":1.002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:05:25.225 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:05:25.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:05:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.521+0000 7f01d3400640 1 Processor -- start 2024-01-16T15:05:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.521+0000 7f01d3400640 1 -- start start 2024-01-16T15:05:25.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01cc15e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d3400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f01cc15e5f0 0x7f01cc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d3400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01cc13f6b0 con 0x7f01cc06d900 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d3400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01cc1402f0 con 0x7f01cc15e5f0 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d3400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc06dd90 con 0x7f01cc13ce60 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d2a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f01cc06d900 0x7f01cc071200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40340/0 (socket says 172.21.15.53:40340) 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.523+0000 7f01d2a00640 1 -- 172.21.15.53:0/3445773104 learned_addr learned my addr 172.21.15.53:0/3445773104 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d2000640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01cc15e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d1600640 1 --2- 172.21.15.53:0/3445773104 >> v2:172.21.15.146:3300/0 conn(0x7f01cc15e5f0 0x7f01cc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093996232 0 0) 0x7f01cc13f6b0 con 0x7f01cc06d900 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01ac003520 con 0x7f01cc06d900 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76834232 0 0) 0x7f01ac003520 con 0x7f01cc06d900 2024-01-16T15:05:25.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 msgr2=0x7f01cc15e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01cc15e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 >> v2:172.21.15.146:3300/0 conn(0x7f01cc15e5f0 msgr2=0x7f01cc06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 --2- 172.21.15.53:0/3445773104 >> v2:172.21.15.146:3300/0 conn(0x7f01cc15e5f0 0x7f01cc06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc13d240 con 0x7f01cc06d900 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.524+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f01c40019f0 con 0x7f01cc06d900 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 >> v1:172.21.15.53:6789/0 conn(0x7f01cc06d900 legacy=0x7f01cc071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01ac003970 0x7f01ac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac0047c0 0x7f01ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1402f0 con 0x7f01ac003970 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc06dd90 con 0x7f01ac0047c0 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01ac00b980 con 0x7f01cc13ce60 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2000640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01ac003970 0x7f01ac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d1600640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac0047c0 0x7f01ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2a00640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2000640 1 -- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 msgr2=0x7f01ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2000640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2000640 1 -- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac0047c0 msgr2=0x7f01ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2000640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac0047c0 0x7f01ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2000640 1 -- 172.21.15.53:0/3445773104 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01ac003520 con 0x7f01ac003970 2024-01-16T15:05:25.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d1600640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac0047c0 0x7f01ac004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:25.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.525+0000 7f01d2a00640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13ce60 0x7f01ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:25.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.527+0000 7f01d2000640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01ac003970 0x7f01ac003d90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f01bc009ac0 tx=0x7f01ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79bb62869397137d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:25.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.527+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc02c6b0 con 0x7f01ac003970 2024-01-16T15:05:25.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.527+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01bc02c810 con 0x7f01ac003970 2024-01-16T15:05:25.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.527+0000 7f01d0c00640 1 -- 172.21.15.53:0/3445773104 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01bc02cba0 con 0x7f01ac003970 2024-01-16T15:05:25.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d3400640 1 -- 172.21.15.53:0/3445773104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01ac003970 msgr2=0x7f01ac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d3400640 1 --2- 172.21.15.53:0/3445773104 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01ac003970 0x7f01ac003d90 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f01bc009ac0 tx=0x7f01ac0057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d2a00640 1 -- 172.21.15.53:0/3445773104 reap_dead start 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d3400640 1 -- 172.21.15.53:0/3445773104 shutdown_connections 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d3400640 1 -- 172.21.15.53:0/3445773104 >> 172.21.15.53:0/3445773104 conn(0x7f01cc0b0b70 msgr2=0x7f01cc0b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d3400640 1 -- 172.21.15.53:0/3445773104 shutdown_connections 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.528+0000 7f01d3400640 1 -- 172.21.15.53:0/3445773104 wait complete. 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.530+0000 7f01d3400640 1 Processor -- start 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.530+0000 7f01d3400640 1 -- start start 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac003970 0x7f01cc0af040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13e7c0 0x7f01cc135680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 0x7f01cc138c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d3400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc1391c0 con 0x7f01cc1387d0 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d3400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc13b680 con 0x7f01ac003970 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d3400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01cc13a5e0 con 0x7f01cc13e7c0 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 0x7f01cc138c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 0x7f01cc138c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52226/0 (socket says 172.21.15.53:52226) 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 -- 172.21.15.53:0/2463681702 learned_addr learned my addr 172.21.15.53:0/2463681702 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d1600640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13e7c0 0x7f01cc135680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2000640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac003970 0x7f01cc0af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 -- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13e7c0 msgr2=0x7f01cc135680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13e7c0 0x7f01cc135680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 -- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac003970 msgr2=0x7f01cc0af040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac003970 0x7f01cc0af040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.531+0000 7f01d2a00640 1 -- 172.21.15.53:0/2463681702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01cc13a8c0 con 0x7f01cc1387d0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.532+0000 7f01d2a00640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 0x7f01cc138c80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0b25e0 tx=0x7f01c400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.532+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4001990 con 0x7f01cc1387d0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.532+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01c400bb80 con 0x7f01cc1387d0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.532+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01cc13aba0 con 0x7f01cc1387d0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.532+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01c4007070 con 0x7f01cc1387d0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.532+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01cc13b160 con 0x7f01cc1387d0 2024-01-16T15:05:25.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.533+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01cc13fbf0 con 0x7f01cc1387d0 2024-01-16T15:05:25.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.534+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f01c40071d0 con 0x7f01cc1387d0 2024-01-16T15:05:25.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.535+0000 7f01ca000640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01a406b8e0 0x7f01a406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:25.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.535+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f01c408aa80 con 0x7f01cc1387d0 2024-01-16T15:05:25.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.536+0000 7f01d2000640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01a406b8e0 0x7f01a406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:25.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.537+0000 7f01d2000640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01a406b8e0 0x7f01a406dda0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f01bc036000 tx=0x7f01bc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:25.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.540+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01c4054fb0 con 0x7f01cc1387d0 2024-01-16T15:05:25.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.807+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f01cc067450 con 0x7f01cc1387d0 2024-01-16T15:05:25.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.810+0000 7f01ca000640 1 -- 172.21.15.53:0/2463681702 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+147 (secure 0 0 0) 0x7f01c40585d0 con 0x7f01cc1387d0 2024-01-16T15:05:25.789 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:05:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:05:25.789 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:05:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:05:25.790 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:05:25.790 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:05:25.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.813+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01a406b8e0 msgr2=0x7f01a406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.813+0000 7f01d3400640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01a406b8e0 0x7f01a406dda0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f01bc036000 tx=0x7f01bc030000 comp rx=0 tx=0).stop 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.813+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 msgr2=0x7f01cc138c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.813+0000 7f01d3400640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 0x7f01cc138c80 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f01cc0b25e0 tx=0x7f01c400b4a0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.813+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 shutdown_connections 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01a406b8e0 0x7f01a406dda0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01cc1387d0 0x7f01cc138c80 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01cc13e7c0 0x7f01cc135680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 --2- 172.21.15.53:0/2463681702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01ac003970 0x7f01cc0af040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 >> 172.21.15.53:0/2463681702 conn(0x7f01cc0b0b70 msgr2=0x7f01cc0ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:25.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 shutdown_connections 2024-01-16T15:05:25.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:25.814+0000 7f01d3400640 1 -- 172.21.15.53:0/2463681702 wait complete. 2024-01-16T15:05:25.813 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:05:26.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.047+0000 7fbf32200640 1 Processor -- start 2024-01-16T15:05:26.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.047+0000 7fbf32200640 1 -- start start 2024-01-16T15:05:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf32200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbf2c15b7a0 0x7fbf2c0b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf32200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbf2c0cfec0 0x7fbf2c0d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf32200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf2c0b2130 con 0x7fbf2c144b70 2024-01-16T15:05:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf32200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c0d0890 con 0x7fbf2c0cfec0 2024-01-16T15:05:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf32200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf2c144f50 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf2be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbf2c0cfec0 0x7fbf2c0d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf2be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbf2c0cfec0 0x7fbf2c0d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52234/0 (socket says 172.21.15.53:52234) 2024-01-16T15:05:26.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf2be00640 1 -- 172.21.15.53:0/597283645 learned_addr learned my addr 172.21.15.53:0/597283645 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.049+0000 7fbf2be00640 1 -- 172.21.15.53:0/597283645 >> v2:172.21.15.146:3300/0 conn(0x7fbf2c15b7a0 msgr2=0x7fbf2c0b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.050+0000 7fbf2be00640 1 --2- 172.21.15.53:0/597283645 >> v2:172.21.15.146:3300/0 conn(0x7fbf2c15b7a0 0x7fbf2c0b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.050+0000 7fbf2be00640 1 -- 172.21.15.53:0/597283645 >> v1:172.21.15.146:6789/0 conn(0x7fbf2c144b70 legacy=0x7fbf2c15f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.050+0000 7fbf2be00640 1 -- 172.21.15.53:0/597283645 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf2c15edc0 con 0x7fbf2c0cfec0 2024-01-16T15:05:26.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.051+0000 7fbf2be00640 1 --2- 172.21.15.53:0/597283645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c0d0350 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0afba0 tx=0x7fbf2402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c24049510c2323c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.051+0000 7fbf2aa00640 1 -- 172.21.15.53:0/597283645 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2402ca80 con 0x7fbf2c0cfec0 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.051+0000 7fbf2aa00640 1 -- 172.21.15.53:0/597283645 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2402cbe0 con 0x7fbf2c0cfec0 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.051+0000 7fbf2aa00640 1 -- 172.21.15.53:0/597283645 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf240343f0 con 0x7fbf2c0cfec0 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.051+0000 7fbf32200640 1 -- 172.21.15.53:0/597283645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0cfec0 msgr2=0x7fbf2c0d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.051+0000 7fbf32200640 1 --2- 172.21.15.53:0/597283645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c0d0350 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c0afba0 tx=0x7fbf2402bee0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.052+0000 7fbf32200640 1 -- 172.21.15.53:0/597283645 shutdown_connections 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.052+0000 7fbf32200640 1 --2- 172.21.15.53:0/597283645 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c0d0350 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.052+0000 7fbf32200640 1 --2- 172.21.15.53:0/597283645 >> v2:172.21.15.146:3300/0 conn(0x7fbf2c15b7a0 0x7fbf2c0b1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.052+0000 7fbf32200640 1 -- 172.21.15.53:0/597283645 >> 172.21.15.53:0/597283645 conn(0x7fbf2c06b4f0 msgr2=0x7fbf2c06b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.052+0000 7fbf32200640 1 -- 172.21.15.53:0/597283645 shutdown_connections 2024-01-16T15:05:26.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.052+0000 7fbf32200640 1 -- 172.21.15.53:0/597283645 wait complete. 2024-01-16T15:05:26.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.053+0000 7fbf32200640 1 Processor -- start 2024-01-16T15:05:26.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.054+0000 7fbf32200640 1 -- start start 2024-01-16T15:05:26.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf32200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c13d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf32200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf2c144b70 0x7fbf2c15e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf32200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 0x7fbf2c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf32200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c15d540 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf32200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c15c900 con 0x7fbf2c0cfec0 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf32200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf2c15cba0 con 0x7fbf2c144b70 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf2be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 0x7fbf2c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf2be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 0x7fbf2c159a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52242/0 (socket says 172.21.15.53:52242) 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.055+0000 7fbf2be00640 1 -- 172.21.15.53:0/1048867341 learned_addr learned my addr 172.21.15.53:0/1048867341 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2b400640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c13d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf23e00640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf2c144b70 0x7fbf2c15e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2be00640 1 -- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf2c144b70 msgr2=0x7fbf2c15e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2be00640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf2c144b70 0x7fbf2c15e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2be00640 1 -- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf2c0cfec0 msgr2=0x7fbf2c13d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2be00640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c13d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2be00640 1 -- 172.21.15.53:0/1048867341 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24009b30 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.056+0000 7fbf2be00640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 0x7fbf2c159a80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbf24036040 tx=0x7fbf24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.057+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2403e450 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.057+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2403e5d0 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.057+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2403e980 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.057+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf2c15ce80 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.057+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2c15e950 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.058+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf2c15d680 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.059+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fbf2402cd50 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.060+0000 7fbf23400640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf0406b860 0x7fbf0406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.060+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbf240ad260 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.060+0000 7fbf2b400640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf0406b860 0x7fbf0406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.062+0000 7fbf2b400640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf0406b860 0x7fbf0406dd20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c007fd0 tx=0x7fbf1c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.064+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbf2c15d680 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.156 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-01-16T15:05:26.177+0000 7f59ace00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:05:26.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.292+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf2c15d680 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.294+0000 7fbf23400640 1 -- 172.21.15.53:0/1048867341 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+4919 (secure 0 0 0) 0x7fbf2c15d680 con 0x7fbf2c15b7a0 2024-01-16T15:05:26.272 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:05:26.273 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":35,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/2632791628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2632791628},{"type":"v1","addr":"172.21.15.53:6833","nonce":2632791628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5680,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3474334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3474334502},{"type":"v1","addr":"172.21.15.53:6835","nonce":3474334502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5683,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2549681777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2549681777},{"type":"v1","addr":"172.21.15.146:6836","nonce":2549681777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:05:19.688137+0000","modified":"2024-01-16T15:05:22.796011+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5686},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5686":{"gid":5686,"name":"d","rank":0,"incarnation":31,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6837/3568122877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3568122877},{"type":"v1","addr":"172.21.15.146:6837","nonce":3568122877}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-01-16T15:05:26.273 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 35 2024-01-16T15:05:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf0406b860 msgr2=0x7fbf0406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf0406b860 0x7fbf0406dd20 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbf1c007fd0 tx=0x7fbf1c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 msgr2=0x7fbf2c159a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 0x7fbf2c159a80 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fbf24036040 tx=0x7fbf24004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 shutdown_connections 2024-01-16T15:05:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbf0406b860 0x7fbf0406dd20 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbf2c15b7a0 0x7fbf2c159a80 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbf2c144b70 0x7fbf2c15e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 --2- 172.21.15.53:0/1048867341 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbf2c0cfec0 0x7fbf2c13d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 >> 172.21.15.53:0/1048867341 conn(0x7fbf2c06b4f0 msgr2=0x7fbf2c06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.296+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 shutdown_connections 2024-01-16T15:05:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.297+0000 7fbf32200640 1 -- 172.21.15.53:0/1048867341 wait complete. 2024-01-16T15:05:26.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:05:19.688137+0000', 'modified': '2024-01-16T15:05:22.796011+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5686}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5686': {'gid': 5686, 'name': 'd', 'rank': 0, 'incarnation': 31, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6837/3568122877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6835', 'nonce': 3568122877}, {'type': 'v1', 'addr': '172.21.15.146:6837', 'nonce': 3568122877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:05:26.697 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:05:26.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T15:05:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.945+0000 7fa6e2000640 1 Processor -- start 2024-01-16T15:05:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.946+0000 7fa6e2000640 1 -- start start 2024-01-16T15:05:26.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e2000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa6dc071410 0x7fa6dc146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa6dc06c7c0 0x7fa6dc0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc071d30 0x7fa6dc0b1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e2000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6dc06cc50 con 0x7fa6dc071410 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e2000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6dc06cec0 con 0x7fa6dc06c7c0 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc06d1a0 con 0x7fa6dc071d30 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa6dc071410 0x7fa6dc146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa6dc071410 0x7fa6dc146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52248/0 (socket says 172.21.15.53:52248) 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e0c00640 1 -- 172.21.15.53:0/2840483426 learned_addr learned my addr 172.21.15.53:0/2840483426 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6e1600640 1 --2- 172.21.15.53:0/2840483426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc071d30 0x7fa6dc0b1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.948+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2840483426 >> v2:172.21.15.146:3300/0 conn(0x7fa6dc06c7c0 0x7fa6dc0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.949+0000 7fa6e0c00640 1 -- 172.21.15.53:0/2840483426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc071d30 msgr2=0x7fa6dc0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.949+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2840483426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc071d30 0x7fa6dc0b1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.949+0000 7fa6e0c00640 1 -- 172.21.15.53:0/2840483426 >> v2:172.21.15.146:3300/0 conn(0x7fa6dc06c7c0 msgr2=0x7fa6dc0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.949+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2840483426 >> v2:172.21.15.146:3300/0 conn(0x7fa6dc06c7c0 0x7fa6dc0717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.949+0000 7fa6e0c00640 1 -- 172.21.15.53:0/2840483426 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6dc06d480 con 0x7fa6dc071410 2024-01-16T15:05:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.950+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2840483426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc146770 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc146ef0 tx=0x7fa6cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca9e6fcfa520440 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6db400640 1 -- 172.21.15.53:0/2840483426 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc031870 con 0x7fa6dc071410 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6db400640 1 -- 172.21.15.53:0/2840483426 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc0319d0 con 0x7fa6dc071410 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6db400640 1 -- 172.21.15.53:0/2840483426 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6cc031d60 con 0x7fa6dc071410 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 -- 172.21.15.53:0/2840483426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 msgr2=0x7fa6dc146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 --2- 172.21.15.53:0/2840483426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc146770 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa6dc146ef0 tx=0x7fa6cc02b910 comp rx=0 tx=0).stop 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 -- 172.21.15.53:0/2840483426 shutdown_connections 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 --2- 172.21.15.53:0/2840483426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc071d30 0x7fa6dc0b1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 --2- 172.21.15.53:0/2840483426 >> v2:172.21.15.146:3300/0 conn(0x7fa6dc06c7c0 0x7fa6dc0717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 --2- 172.21.15.53:0/2840483426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc146770 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 -- 172.21.15.53:0/2840483426 >> 172.21.15.53:0/2840483426 conn(0x7fa6dc06a800 msgr2=0x7fa6dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.951+0000 7fa6e2000640 1 -- 172.21.15.53:0/2840483426 shutdown_connections 2024-01-16T15:05:26.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.952+0000 7fa6e2000640 1 -- 172.21.15.53:0/2840483426 wait complete. 2024-01-16T15:05:26.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.953+0000 7fa6e2000640 1 Processor -- start 2024-01-16T15:05:26.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.953+0000 7fa6e2000640 1 -- start start 2024-01-16T15:05:26.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.954+0000 7fa6e2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc06c7c0 0x7fa6dc1f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.954+0000 7fa6e2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc1f5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.954+0000 7fa6e2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6dc071d30 0x7fa6dc15f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.954+0000 7fa6e2000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc15d4a0 con 0x7fa6dc071410 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.954+0000 7fa6e2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc1f3280 con 0x7fa6dc071d30 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.954+0000 7fa6e2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6dc13d6c0 con 0x7fa6dc06c7c0 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc1f5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc1f5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52262/0 (socket says 172.21.15.53:52262) 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 -- 172.21.15.53:0/54902261 learned_addr learned my addr 172.21.15.53:0/54902261 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6e1600640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6dc071d30 0x7fa6dc15f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc06c7c0 0x7fa6dc1f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 -- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc06c7c0 msgr2=0x7fa6dc1f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc06c7c0 0x7fa6dc1f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 -- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6dc071d30 msgr2=0x7fa6dc15f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6dc071d30 0x7fa6dc15f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 -- 172.21.15.53:0/54902261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6cc0094d0 con 0x7fa6dc071410 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc06c7c0 0x7fa6dc1f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:26.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.955+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc1f5a90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4007ef0 tx=0x7fa6c400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.956+0000 7fa6d8c00640 1 -- 172.21.15.53:0/54902261 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4014620 con 0x7fa6dc071410 2024-01-16T15:05:26.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.956+0000 7fa6d8c00640 1 -- 172.21.15.53:0/54902261 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4014780 con 0x7fa6dc071410 2024-01-16T15:05:26.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.956+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6dc15d6f0 con 0x7fa6dc071410 2024-01-16T15:05:26.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.956+0000 7fa6d8c00640 1 -- 172.21.15.53:0/54902261 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4014b30 con 0x7fa6dc071410 2024-01-16T15:05:26.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.956+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6dc15dcb0 con 0x7fa6dc071410 2024-01-16T15:05:26.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.958+0000 7fa6d8c00640 1 -- 172.21.15.53:0/54902261 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fa6c4014c90 con 0x7fa6dc071410 2024-01-16T15:05:26.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.959+0000 7fa6d8c00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa6b406b860 0x7fa6b406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.959+0000 7fa6d8c00640 1 -- 172.21.15.53:0/54902261 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6c408b050 con 0x7fa6dc071410 2024-01-16T15:05:26.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.960+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa6b406b860 0x7fa6b406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.960+0000 7fa6e6378740 1 Processor -- start 2024-01-16T15:05:26.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.960+0000 7fa6e6378740 1 -- start start 2024-01-16T15:05:26.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec279ed450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec279ec270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e6378740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2794b880 con 0x55ec279ec7b0 2024-01-16T15:05:26.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec2795aae0 con 0x55ec279e7d70 2024-01-16T15:05:26.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec279581d0 con 0x55ec279ed070 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa6b406b860 0x7fa6b406dd20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc036000 tx=0x7fa6cc03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52274/0 (socket says 172.21.15.53:52274) 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 -- 172.21.15.53:0/156838330 learned_addr learned my addr 172.21.15.53:0/156838330 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec279ed450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec279ec270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 -- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 msgr2=0x55ec279ed450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec279ed450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 -- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 msgr2=0x55ec279ec270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec279ec270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e1600640 1 -- 172.21.15.53:0/156838330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec2791b860 con 0x55ec279ec7b0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec279ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.961+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec279ec270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.962+0000 7fa6e1600640 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30730 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400d7f0 tx=0x7fa6d400dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=539c18710ba24006 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.962+0000 7fa6d1600640 1 -- 172.21.15.53:0/156838330 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d400f9c0 con 0x55ec279ec7b0 2024-01-16T15:05:26.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.962+0000 7fa6d1600640 1 -- 172.21.15.53:0/156838330 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d400fb20 con 0x55ec279ec7b0 2024-01-16T15:05:26.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.962+0000 7fa6d1600640 1 -- 172.21.15.53:0/156838330 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d400feb0 con 0x55ec279ec7b0 2024-01-16T15:05:26.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 -- 172.21.15.53:0/156838330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 msgr2=0x55ec27a30730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30730 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400d7f0 tx=0x7fa6d400dcb0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 -- 172.21.15.53:0/156838330 shutdown_connections 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30730 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec279ec270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 --2- 172.21.15.53:0/156838330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec279ed450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 -- 172.21.15.53:0/156838330 >> 172.21.15.53:0/156838330 conn(0x55ec279aec70 msgr2=0x55ec27a32b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 -- 172.21.15.53:0/156838330 shutdown_connections 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 -- 172.21.15.53:0/156838330 wait complete. 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.963+0000 7fa6e6378740 1 -- start start 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.964+0000 7fa6e6378740 1 Processor -- start 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a03f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a15470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a161d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec27953ac0 con 0x55ec279ec7b0 2024-01-16T15:05:26.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec278f02f0 con 0x55ec279e7d70 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec2790dc70 con 0x55ec279ed070 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6e6378740 10 client.0 _ll_register_callbacks cb 0x55ec279123b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a15470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a15470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52286/0 (socket says 172.21.15.53:52286) 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2778213420 learned_addr learned my addr 172.21.15.53:0/2778213420 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 msgr2=0x55ec27a161d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a161d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 msgr2=0x55ec27a03f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a03f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.965+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2778213420 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d400e040 con 0x55ec279ec7b0 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a15470 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4058fd0 tx=0x7fa6c40599d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4104d188c29eb43f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6ca000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6ca000640 1 -- 172.21.15.53:0/2778213420 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c4088ba0 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6ca000640 1 -- 172.21.15.53:0/2778213420 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6c40615d0 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ec2794caa0 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6ca000640 1 -- 172.21.15.53:0/2778213420 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6c40587c0 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6e6378740 10 client.6471 fetch_fsmap learned FSMap version 35 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55ec27952140 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6ca000640 1 -- 172.21.15.53:0/2778213420 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4074030 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6ca000640 1 -- 172.21.15.53:0/2778213420 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 35) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa6c4061140 con 0x55ec279ec7b0 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6e6378740 10 client.6471 fetch_fsmap finished waiting for FSMap version 35 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6e6378740 10 client.6471 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.966+0000 7fa6e6378740 20 client.6471 populate_metadata read hostname 'smithi053' 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.967+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27a12b90 0x55ec27a375a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.967+0000 7fa6e6378740 4 client.6471 mds_command: new command op to 5686 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.967+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 --> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec2794c7c0 con 0x55ec27a12b90 2024-01-16T15:05:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.967+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27a12b90 0x55ec27a375a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.968+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27a12b90 0x55ec27a375a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59ba20b82e8d4178 server_cookie=4833d362e4281447 in_seq=0 out_seq=0 2024-01-16T15:05:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.968+0000 7fa6ca000640 10 client.6471 ms_handle_connect on v2:172.21.15.146:6835/3568122877 2024-01-16T15:05:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.970+0000 7fa6ca000640 1 -- 172.21.15.53:0/2778213420 <== mds.0 v2:172.21.15.146:6835/3568122877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ec2794c7c0 con 0x55ec27a12b90 2024-01-16T15:05:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.970+0000 7fa6ca000640 10 client.6471 handle_command_reply: tid=0 2024-01-16T15:05:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.970+0000 7fa6e6378740 1 client.6471 shutdown 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 msgr2=0x55ec27a15470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a15470 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4058fd0 tx=0x7fa6c40599d0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 shutdown_connections 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27a12b90 0x55ec27a375a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a161d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a15470 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2778213420 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a03f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6ca000640 0 client.6471 ms_handle_reset on v2:172.21.15.146:6835/3568122877 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 >> 172.21.15.53:0/2778213420 conn(0x55ec279aec70 msgr2=0x55ec279ed450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 shutdown_connections 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 1 -- 172.21.15.53:0/2778213420 wait complete. 2024-01-16T15:05:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.971+0000 7fa6e6378740 20 client.6471 trim_cache size 0 max 16384 2024-01-16T15:05:26.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.988+0000 7fa6e6378740 1 Processor -- start 2024-01-16T15:05:26.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.988+0000 7fa6e6378740 1 -- start start 2024-01-16T15:05:26.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a11a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec279e6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a35ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6e6378740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec276b6f20 con 0x55ec279ec7b0 2024-01-16T15:05:26.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec27a165b0 con 0x55ec279e7d70 2024-01-16T15:05:26.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec279e8c40 con 0x55ec279ed070 2024-01-16T15:05:26.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec279e6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec279e6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52296/0 (socket says 172.21.15.53:52296) 2024-01-16T15:05:26.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.989+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2268661684 learned_addr learned my addr 172.21.15.53:0/2268661684 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a11a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6e1600640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a35ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 msgr2=0x55ec27a35ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a35ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 msgr2=0x55ec27a11a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a11a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2268661684 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec278efb80 con 0x55ec279ec7b0 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a11a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6e1600640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a35ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec279e6fc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4088f50 tx=0x7fa6c401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922dda61b803a7d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6d2000640 1 -- 172.21.15.53:0/2268661684 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40615d0 con 0x55ec279ec7b0 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6d2000640 1 -- 172.21.15.53:0/2268661684 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c4074070 con 0x55ec279ec7b0 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.990+0000 7fa6d2000640 1 -- 172.21.15.53:0/2268661684 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c40587c0 con 0x55ec279ec7b0 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 -- 172.21.15.53:0/2268661684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 msgr2=0x55ec279e6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec279e6fc0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4088f50 tx=0x7fa6c401f000 comp rx=0 tx=0).stop 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 -- 172.21.15.53:0/2268661684 shutdown_connections 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27a35ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec279e6fc0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2268661684 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a11a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 -- 172.21.15.53:0/2268661684 >> 172.21.15.53:0/2268661684 conn(0x55ec279aec70 msgr2=0x55ec27a388c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 -- 172.21.15.53:0/2268661684 shutdown_connections 2024-01-16T15:05:26.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.991+0000 7fa6e6378740 1 -- 172.21.15.53:0/2268661684 wait complete. 2024-01-16T15:05:26.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.992+0000 7fa6e6378740 1 -- start start 2024-01-16T15:05:26.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.992+0000 7fa6e6378740 1 Processor -- start 2024-01-16T15:05:26.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a9c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27aa0a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec27a9cc70 con 0x55ec279ec7b0 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec27aa0330 con 0x55ec279e7d70 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55ec27a30f40 con 0x55ec279ed070 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6e6378740 10 client.0 _ll_register_callbacks cb 0x7fa6c4055610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52306/0 (socket says 172.21.15.53:52306) 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.993+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2912560841 learned_addr learned my addr 172.21.15.53:0/2912560841 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a9c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6e1600640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27aa0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 msgr2=0x55ec27aa0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27aa0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 msgr2=0x55ec27a9c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a9c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6dbe00640 1 -- 172.21.15.53:0/2912560841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec27952140 con 0x55ec279ec7b0 2024-01-16T15:05:26.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6dbe00640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30530 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4059260 tx=0x7fa6c40094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff34225c72ab1c26 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6ca000640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6ca000640 1 -- 172.21.15.53:0/2912560841 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6c405f1d0 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6ca000640 1 -- 172.21.15.53:0/2912560841 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa6c405f330 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.994+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6d400fb20 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6ca000640 1 -- 172.21.15.53:0/2912560841 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6c405f4f0 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 10 client.6477 fetch_fsmap learned FSMap version 35 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0x55ec27915170 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6ca000640 1 -- 172.21.15.53:0/2912560841 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d400fb20 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6ca000640 1 -- 172.21.15.53:0/2912560841 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 35) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa6c40550c0 con 0x55ec279ec7b0 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 10 client.6477 fetch_fsmap finished waiting for FSMap version 35 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 10 client.6477 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 20 client.6477 populate_metadata read hostname 'smithi053' 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27aa0fa0 0x55ec27a13ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 4 client.6477 mds_command: new command op to 5686 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:05:26.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.995+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 --> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ec276546c0 con 0x55ec27aa0fa0 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.996+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27aa0fa0 0x55ec27a13ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6e0c00640 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27aa0fa0 0x55ec27a13ad0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b6a2b7b40f121f9 server_cookie=39b786c01901f200 in_seq=0 out_seq=0 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6ca000640 10 client.6477 ms_handle_connect on v2:172.21.15.146:6835/3568122877 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6ca000640 1 -- 172.21.15.53:0/2912560841 <== mds.0 v2:172.21.15.146:6835/3568122877 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55ec276546c0 con 0x55ec27aa0fa0 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6ca000640 10 client.6477 handle_command_reply: tid=0 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6e6378740 1 client.6477 shutdown 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 msgr2=0x55ec27a30530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.997+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30530 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4059260 tx=0x7fa6c40094d0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 shutdown_connections 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:6835/3568122877,v1:172.21.15.146:6837/3568122877] conn(0x55ec27aa0fa0 0x55ec27a13ad0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55ec279ed070 0x55ec27aa0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55ec279ec7b0 0x55ec27a30530 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 --2- 172.21.15.53:0/2912560841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55ec279e7d70 0x55ec27a9c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6ca000640 0 client.6477 ms_handle_reset on v2:172.21.15.146:6835/3568122877 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 >> 172.21.15.53:0/2912560841 conn(0x55ec279aec70 msgr2=0x55ec279eb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 shutdown_connections 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 1 -- 172.21.15.53:0/2912560841 wait complete. 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e6378740 20 client.6477 trim_cache size 0 max 16384 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5686, 2024-01-16T15:05:26.978 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 6.7897080900000004, 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 35, 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 60, 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 60, 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 12.145414151000001 2024-01-16T15:05:26.979 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa6b406b860 msgr2=0x7fa6b406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e2000640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa6b406b860 0x7fa6b406dd20 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fa6cc036000 tx=0x7fa6cc03c040 comp rx=0 tx=0).stop 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 msgr2=0x7fa6dc1f5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.998+0000 7fa6e2000640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc1f5a90 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa6c4007ef0 tx=0x7fa6c400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 shutdown_connections 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa6b406b860 0x7fa6b406dd20 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6dc071d30 0x7fa6dc15f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6dc071410 0x7fa6dc1f5a90 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 --2- 172.21.15.53:0/54902261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa6dc06c7c0 0x7fa6dc1f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 >> 172.21.15.53:0/54902261 conn(0x7fa6dc06a800 msgr2=0x7fa6dc1465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:26.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 shutdown_connections 2024-01-16T15:05:26.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:26.999+0000 7fa6e2000640 1 -- 172.21.15.53:0/54902261 wait complete. 2024-01-16T15:05:26.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:05:27.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.213+0000 7f2478c00640 1 Processor -- start 2024-01-16T15:05:27.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.213+0000 7f2478c00640 1 -- start start 2024-01-16T15:05:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2478c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f247415afb0 0x7f247413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2478c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247413ff30 con 0x7f247413fae0 2024-01-16T15:05:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2478c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24740cdc40 con 0x7f247413cfa0 2024-01-16T15:05:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2478c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24740cdec0 con 0x7f247415afb0 2024-01-16T15:05:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2473e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f247413fae0 0x7f24740b2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40350/0 (socket says 172.21.15.53:40350) 2024-01-16T15:05:27.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2473e00640 1 -- 172.21.15.53:0/2003184429 learned_addr learned my addr 172.21.15.53:0/2003184429 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:27.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.215+0000 7f2472a00640 1 --2- 172.21.15.53:0/2003184429 >> v2:172.21.15.146:3300/0 conn(0x7f247415afb0 0x7f247413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554515899 0 0) 0x7f246c001490 con 0x7f247413fae0 2024-01-16T15:05:27.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454005730 con 0x7f247413fae0 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 440327364 0 0) 0x7f2454005730 con 0x7f247413fae0 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 >> v2:172.21.15.146:3300/0 conn(0x7f247415afb0 msgr2=0x7f247413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 --2- 172.21.15.53:0/2003184429 >> v2:172.21.15.146:3300/0 conn(0x7f247415afb0 0x7f247413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 >> v1:172.21.15.146:6789/0 conn(0x7f247413cfa0 legacy=0x7f24740cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24740ce1a0 con 0x7f247413fae0 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f246c001b80 con 0x7f247413fae0 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 >> v1:172.21.15.53:6789/0 conn(0x7f247413fae0 legacy=0x7f24740b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2454005bd0 0x7f2454005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f2454006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24540073e0 0x7f245400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24740cdec0 con 0x7f24540069d0 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245400e330 con 0x7f24540073e0 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.216+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245400e4f0 con 0x7f2454005bd0 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2472a00640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f2454006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2472a00640 1 -- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2454005bd0 msgr2=0x7f2454005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2473400640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2454005bd0 0x7f2454005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2472a00640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2454005bd0 0x7f2454005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2472a00640 1 -- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24540073e0 msgr2=0x7f245400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2472a00640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24540073e0 0x7f245400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2472a00640 1 -- 172.21.15.53:0/2003184429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454005730 con 0x7f24540069d0 2024-01-16T15:05:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2473e00640 1 -- 172.21.15.53:0/2003184429 reap_dead start 2024-01-16T15:05:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.217+0000 7f2473400640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2454005bd0 0x7f2454005ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.218+0000 7f2472a00640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f2454006e50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f245c009a90 tx=0x7f2454007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d108a6b1d9dc783 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.218+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c02c6b0 con 0x7f24540069d0 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245c02c810 con 0x7f24540069d0 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2472000640 1 -- 172.21.15.53:0/2003184429 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245c02cba0 con 0x7f24540069d0 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 -- 172.21.15.53:0/2003184429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 msgr2=0x7f2454006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f2454006e50 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f245c009a90 tx=0x7f2454007a00 comp rx=0 tx=0).stop 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 -- 172.21.15.53:0/2003184429 shutdown_connections 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 --2- 172.21.15.53:0/2003184429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f2454006e50 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 -- 172.21.15.53:0/2003184429 >> 172.21.15.53:0/2003184429 conn(0x7f247406a800 msgr2=0x7f247406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 -- 172.21.15.53:0/2003184429 shutdown_connections 2024-01-16T15:05:27.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.219+0000 7f2478c00640 1 -- 172.21.15.53:0/2003184429 wait complete. 2024-01-16T15:05:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.221+0000 7f2478c00640 1 Processor -- start 2024-01-16T15:05:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.221+0000 7f2478c00640 1 -- start start 2024-01-16T15:05:27.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2478c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f24740adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2478c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24740ac8d0 0x7f24741549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2478c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24741f7640 0x7f24741f7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2478c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24740acd50 con 0x7f24540069d0 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2478c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24740ad000 con 0x7f24740ac8d0 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2478c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24740ad2e0 con 0x7f24741f7640 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2473400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f24740adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2473400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f24740adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52318/0 (socket says 172.21.15.53:52318) 2024-01-16T15:05:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.222+0000 7f2473400640 1 -- 172.21.15.53:0/3438940395 learned_addr learned my addr 172.21.15.53:0/3438940395 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2472a00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24740ac8d0 0x7f24741549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2473e00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24741f7640 0x7f24741f7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2473400640 1 -- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24741f7640 msgr2=0x7f24741f7ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2473400640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24741f7640 0x7f24741f7ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2473400640 1 -- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24740ac8d0 msgr2=0x7f24741549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2473400640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24740ac8d0 0x7f24741549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.223+0000 7f2473400640 1 -- 172.21.15.53:0/3438940395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24740ad5c0 con 0x7f24540069d0 2024-01-16T15:05:27.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.224+0000 7f2473400640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f24740adff0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f246400d480 tx=0x7f246400d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:27.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.224+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2464005480 con 0x7f24540069d0 2024-01-16T15:05:27.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.224+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24741f7120 con 0x7f24540069d0 2024-01-16T15:05:27.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.224+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2474155550 con 0x7f24540069d0 2024-01-16T15:05:27.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.225+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2464003610 con 0x7f24540069d0 2024-01-16T15:05:27.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.225+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24640039c0 con 0x7f24540069d0 2024-01-16T15:05:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.226+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2444005250 con 0x7f24540069d0 2024-01-16T15:05:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.228+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f246401c460 con 0x7f24540069d0 2024-01-16T15:05:27.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.229+0000 7f246b400640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f244c06b860 0x7f244c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:27.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.229+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f246408af60 con 0x7f24540069d0 2024-01-16T15:05:27.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.230+0000 7f2472a00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f244c06b860 0x7f244c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:27.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.231+0000 7f2472a00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f244c06b860 0x7f244c06dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f245c036000 tx=0x7f245c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:27.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.232+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24640556a0 con 0x7f24540069d0 2024-01-16T15:05:27.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.459+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f24440057d0 con 0x7f24540069d0 2024-01-16T15:05:27.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.460+0000 7f246b400640 1 -- 172.21.15.53:0/3438940395 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) v1 ==== 53+0+73 (secure 0 0 0) 0x7f2464058cc0 con 0x7f24540069d0 2024-01-16T15:05:27.438 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f244c06b860 msgr2=0x7f244c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f244c06b860 0x7f244c06dd20 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f245c036000 tx=0x7f245c030000 comp rx=0 tx=0).stop 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 msgr2=0x7f24740adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f24740adff0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f246400d480 tx=0x7f246400d940 comp rx=0 tx=0).stop 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 shutdown_connections 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f244c06b860 0x7f244c06dd20 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24741f7640 0x7f24741f7ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24740ac8d0 0x7f24741549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 --2- 172.21.15.53:0/3438940395 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24540069d0 0x7f24740adff0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.462+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 >> 172.21.15.53:0/3438940395 conn(0x7f247406a800 msgr2=0x7f2474071310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:27.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.463+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 shutdown_connections 2024-01-16T15:05:27.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:27.463+0000 7f2478c00640 1 -- 172.21.15.53:0/3438940395 wait complete. 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:05:27.458 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:27.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:27.459 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:05:27.509 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:05:27.509 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:05:27.509 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:05:27.509 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.509 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:05:27.509 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42323sec preferred_lft 42323sec 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.510 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:05:27.511 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.512 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:27.512 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:27.512 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:05:27.512 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:27.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:27.513 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:05:27.595 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:05:27.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:27.600 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:05:27.653 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:05:27.653 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:05:27.653 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:05:27.653 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:05:27.654 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:05:27.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:27.655 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:05:27.655 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:05:27.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:05:27.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:05:27.655 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:05:27.655 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:05:27.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:27.839 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:05:27.894 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:05:27.894 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:05:27.895 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:27.896 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:27.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:05:27.994 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:05:27.994 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:27.994 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:28.051 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:05:28.051 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:28.051 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:28.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:05:28.205 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:28.263 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:05:28.264 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:28.265 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:28.266 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:28.267 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:28.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:28.268 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:05:28.316 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:05:28.316 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:05:28.322 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:05:28.397 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:28.443 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:05:28.444 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:28.445 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:05:28.446 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:28.447 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:05:28.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:28.448 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:05:28.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:05:28.528+0000 7fe3a0042480 -1 init, newargv = 0x561c3bae1ac0 newargc=15 2024-01-16T15:05:28.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[56436]: starting ceph client 2024-01-16T15:05:28.518 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[56436]: starting fuse 2024-01-16T15:05:28.532 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:05:28.532 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:05:28.533 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:05:28.534 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:05:28.535 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:05:28.535 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:05:28.535 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:05:28.535 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:05:28.736 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.56436.asok 2024-01-16T15:05:28.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:05:28.754 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56436.asok 2024-01-16T15:05:28.754 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:28.754 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56436.asok status 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "56436", 2024-01-16T15:05:29.000 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 6489, 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 6489 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 1192326847 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:29.001 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 1192326847 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.6489 192.168.0.1:0/1192326847", 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/1192326847", 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 35, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 60, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:05:29.002 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:05:29.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:29.015 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:05:29.015 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:29.072 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:05:29.072 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:29.072 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:29.072 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:05:29.175 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:05:29.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.404+0000 7fe88a200640 1 Processor -- start 2024-01-16T15:05:29.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.405+0000 7fe88a200640 1 -- start start 2024-01-16T15:05:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.406+0000 7fe88a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8840cd240 0x7fe8840b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.406+0000 7fe88a200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe88406c6b0 0x7fe884099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.406+0000 7fe88a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe884099fa0 0x7fe88406b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.406+0000 7fe88a200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88409a430 con 0x7fe884099fa0 2024-01-16T15:05:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.406+0000 7fe88a200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe88406cea0 con 0x7fe88406c6b0 2024-01-16T15:05:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.406+0000 7fe88a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88406cbc0 con 0x7fe8840cd240 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe884099fa0 0x7fe88406b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe884099fa0 0x7fe88406b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52334/0 (socket says 172.21.15.53:52334) 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 -- 172.21.15.53:0/4011273468 learned_addr learned my addr 172.21.15.53:0/4011273468 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883400640 1 --2- 172.21.15.53:0/4011273468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8840cd240 0x7fe8840b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe87be00640 1 --2- 172.21.15.53:0/4011273468 >> v2:172.21.15.146:3300/0 conn(0x7fe88406c6b0 0x7fe884099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 -- 172.21.15.53:0/4011273468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8840cd240 msgr2=0x7fe8840b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 --2- 172.21.15.53:0/4011273468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8840cd240 0x7fe8840b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 -- 172.21.15.53:0/4011273468 >> v2:172.21.15.146:3300/0 conn(0x7fe88406c6b0 msgr2=0x7fe884099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 --2- 172.21.15.53:0/4011273468 >> v2:172.21.15.146:3300/0 conn(0x7fe88406c6b0 0x7fe884099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.407+0000 7fe883e00640 1 -- 172.21.15.53:0/4011273468 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe88406f350 con 0x7fe884099fa0 2024-01-16T15:05:29.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.408+0000 7fe883e00640 1 --2- 172.21.15.53:0/4011273468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe88406b4f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe8840749d0 tx=0x7fe87c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d2a9082765a2f3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:29.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe882a00640 1 -- 172.21.15.53:0/4011273468 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c02ca70 con 0x7fe884099fa0 2024-01-16T15:05:29.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe882a00640 1 -- 172.21.15.53:0/4011273468 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87c02cbd0 con 0x7fe884099fa0 2024-01-16T15:05:29.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe882a00640 1 -- 172.21.15.53:0/4011273468 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87c0343f0 con 0x7fe884099fa0 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 -- 172.21.15.53:0/4011273468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 msgr2=0x7fe88406b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 --2- 172.21.15.53:0/4011273468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe88406b4f0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fe8840749d0 tx=0x7fe87c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 -- 172.21.15.53:0/4011273468 shutdown_connections 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 --2- 172.21.15.53:0/4011273468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe88406b4f0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 --2- 172.21.15.53:0/4011273468 >> v2:172.21.15.146:3300/0 conn(0x7fe88406c6b0 0x7fe884099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 --2- 172.21.15.53:0/4011273468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8840cd240 0x7fe8840b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.409+0000 7fe88a200640 1 -- 172.21.15.53:0/4011273468 >> 172.21.15.53:0/4011273468 conn(0x7fe8840b2d00 msgr2=0x7fe8840b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.410+0000 7fe88a200640 1 -- 172.21.15.53:0/4011273468 shutdown_connections 2024-01-16T15:05:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.410+0000 7fe88a200640 1 -- 172.21.15.53:0/4011273468 wait complete. 2024-01-16T15:05:29.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.411+0000 7fe88a200640 1 Processor -- start 2024-01-16T15:05:29.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.411+0000 7fe88a200640 1 -- start start 2024-01-16T15:05:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.412+0000 7fe88a200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe88406c6b0 0x7fe8840c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.412+0000 7fe88a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe8840c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.412+0000 7fe88a200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8840cd240 0x7fe8840b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.412+0000 7fe88a200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840c5ee0 con 0x7fe884099fa0 2024-01-16T15:05:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.412+0000 7fe88a200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840b5430 con 0x7fe8840cd240 2024-01-16T15:05:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.412+0000 7fe88a200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8840b5660 con 0x7fe88406c6b0 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe8840c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe8840c59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52350/0 (socket says 172.21.15.53:52350) 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 -- 172.21.15.53:0/4152615637 learned_addr learned my addr 172.21.15.53:0/4152615637 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe883400640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe88406c6b0 0x7fe8840c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe883e00640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8840cd240 0x7fe8840b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 -- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe88406c6b0 msgr2=0x7fe8840c82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe88406c6b0 0x7fe8840c82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 -- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8840cd240 msgr2=0x7fe8840b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8840cd240 0x7fe8840b4ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe87be00640 1 -- 172.21.15.53:0/4152615637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe87c009b30 con 0x7fe884099fa0 2024-01-16T15:05:29.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.413+0000 7fe883400640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe88406c6b0 0x7fe8840c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.414+0000 7fe87be00640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe8840c59a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d5c0 tx=0x7fe86c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.414+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0125f0 con 0x7fe884099fa0 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.414+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe86c012750 con 0x7fe884099fa0 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.414+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe86c012b00 con 0x7fe884099fa0 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.414+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8840b5940 con 0x7fe884099fa0 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.414+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8840c66d0 con 0x7fe884099fa0 2024-01-16T15:05:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.416+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8840675d0 con 0x7fe884099fa0 2024-01-16T15:05:29.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.416+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fe86c012c60 con 0x7fe884099fa0 2024-01-16T15:05:29.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.417+0000 7fe87b400640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe85c06b860 0x7fe85c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:29.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.417+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe86c08aa60 con 0x7fe884099fa0 2024-01-16T15:05:29.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.418+0000 7fe883400640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe85c06b860 0x7fe85c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:29.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.419+0000 7fe883400640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe85c06b860 0x7fe85c06dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe874006fd0 tx=0x7fe874008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:29.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.425+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe86c0550f0 con 0x7fe884099fa0 2024-01-16T15:05:29.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.658+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe8840cfb70 con 0x7fe884099fa0 2024-01-16T15:05:29.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.659+0000 7fe87b400640 1 -- 172.21.15.53:0/4152615637 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v35) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe86c058710 con 0x7fe884099fa0 2024-01-16T15:05:29.637 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:05:29.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe85c06b860 msgr2=0x7fe85c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe85c06b860 0x7fe85c06dd20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fe874006fd0 tx=0x7fe874008040 comp rx=0 tx=0).stop 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 msgr2=0x7fe8840c59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe8840c59a0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00d5c0 tx=0x7fe86c00da80 comp rx=0 tx=0).stop 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 shutdown_connections 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe85c06b860 0x7fe85c06dd20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8840cd240 0x7fe8840b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe884099fa0 0x7fe8840c59a0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 --2- 172.21.15.53:0/4152615637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe88406c6b0 0x7fe8840c82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 >> 172.21.15.53:0/4152615637 conn(0x7fe8840b2d00 msgr2=0x7fe88406b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 shutdown_connections 2024-01-16T15:05:29.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:29.661+0000 7fe88a200640 1 -- 172.21.15.53:0/4152615637 wait complete. 2024-01-16T15:05:29.657 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:05:29.657 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:05:29.658 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:29.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:29.658 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:29.717 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42200sec preferred_lft 42200sec 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:05:29.718 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:05:29.719 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:05:29.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:29.720 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:05:29.810 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:05:29.814 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:29.814 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:05:29.862 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:05:29.862 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:05:29.862 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:05:29.862 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:05:29.862 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:05:29.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:29.863 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:05:29.863 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:05:29.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:05:29.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:05:29.863 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:05:29.863 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:05:30.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:30.033 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:05:30.084 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:05:30.084 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:05:30.085 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:30.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:30.085 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:05:30.176 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:05:30.176 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:30.176 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:30.231 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:05:30.231 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:05:30.232 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:30.285 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:05:30.377 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:05:30.431 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:30.432 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:30.433 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:30.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:30.434 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:05:30.479 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:05:30.479 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:05:30.527 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:05:30.606 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:05:30.648 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:05:30.649 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:30.650 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:05:30.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:30.651 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:05:30.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:05:30.738+0000 7f12c9585480 -1 init, newargv = 0x5592d98d26f0 newargc=15 2024-01-16T15:05:30.717 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44041]: starting ceph client 2024-01-16T15:05:30.726 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44041]: starting fuse 2024-01-16T15:05:30.739 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:05:30.739 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:05:30.739 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:05:30.740 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:05:30.741 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:05:30.741 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:05:30.741 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:05:30.741 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:05:30.942 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.44041.asok 2024-01-16T15:05:30.946 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:05:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:05:30.959 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44041.asok 2024-01-16T15:05:30.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:30.959 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44041.asok status 2024-01-16T15:05:31.190 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:05:31.190 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "44041", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 5788, 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:05:31.191 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 5788 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1396882169 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 1396882169 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.5788 192.168.0.1:0/1396882169", 2024-01-16T15:05:31.192 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/1396882169", 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 35, 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 60, 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:05:31.193 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:05:31.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:31.205 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:05:31.205 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:31.253 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:05:31.253 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:31.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:31.253 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:05:31.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:05:31.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.648+0000 7ffbb5000640 1 Processor -- start 2024-01-16T15:05:31.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.649+0000 7ffbb5000640 1 -- start start 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.650+0000 7ffbb5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb00cc500 0x7ffbb00cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.650+0000 7ffbb5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffbb006d160 0x7ffbb006d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffbb5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbb0070e50 con 0x7ffbb0143c40 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffbb5000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbb00723c0 con 0x7ffbb006d160 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffbb5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0144020 con 0x7ffbb00cc500 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffbafe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffbb006d160 0x7ffbb006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:31.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffbaf400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ffbb0143c40 0x7ffbb0071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40352/0 (socket says 172.21.15.53:40352) 2024-01-16T15:05:31.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffbaf400640 1 -- 172.21.15.53:0/59524005 learned_addr learned my addr 172.21.15.53:0/59524005 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:31.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.651+0000 7ffba7e00640 1 --2- 172.21.15.53:0/59524005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb00cc500 0x7ffbb00cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:31.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.652+0000 7ffbafe00640 1 -- 172.21.15.53:0/59524005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb00cc500 msgr2=0x7ffbb00cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:31.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.652+0000 7ffbafe00640 1 --2- 172.21.15.53:0/59524005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb00cc500 0x7ffbb00cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:31.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.652+0000 7ffbafe00640 1 -- 172.21.15.53:0/59524005 >> v1:172.21.15.53:6789/0 conn(0x7ffbb0143c40 legacy=0x7ffbb0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.652+0000 7ffbafe00640 1 -- 172.21.15.53:0/59524005 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb00ccea0 con 0x7ffbb006d160 2024-01-16T15:05:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.652+0000 7ffbaea00640 1 -- 172.21.15.53:0/59524005 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114262230 0 0) 0x7ffbb0070e50 con 0x7ffbb0143c40 2024-01-16T15:05:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.653+0000 7ffbafe00640 1 --2- 172.21.15.53:0/59524005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb006d160 0x7ffbb006d590 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009d90 tx=0x7ffba802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f83e9f89d02215 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.653+0000 7ffbaea00640 1 -- 172.21.15.53:0/59524005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba802cbd0 con 0x7ffbb006d160 2024-01-16T15:05:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.653+0000 7ffbaea00640 1 -- 172.21.15.53:0/59524005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba802cd30 con 0x7ffbb006d160 2024-01-16T15:05:31.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.653+0000 7ffbaea00640 1 -- 172.21.15.53:0/59524005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba802cea0 con 0x7ffbb006d160 2024-01-16T15:05:31.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 -- 172.21.15.53:0/59524005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb006d160 msgr2=0x7ffbb006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:31.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 --2- 172.21.15.53:0/59524005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb006d160 0x7ffbb006d590 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009d90 tx=0x7ffba802be20 comp rx=0 tx=0).stop 2024-01-16T15:05:31.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 -- 172.21.15.53:0/59524005 shutdown_connections 2024-01-16T15:05:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 --2- 172.21.15.53:0/59524005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb006d160 0x7ffbb006d590 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 --2- 172.21.15.53:0/59524005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb00cc500 0x7ffbb00cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 -- 172.21.15.53:0/59524005 >> 172.21.15.53:0/59524005 conn(0x7ffbb006a800 msgr2=0x7ffbb006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 -- 172.21.15.53:0/59524005 shutdown_connections 2024-01-16T15:05:31.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.654+0000 7ffbb5000640 1 -- 172.21.15.53:0/59524005 wait complete. 2024-01-16T15:05:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.656+0000 7ffbb5000640 1 Processor -- start 2024-01-16T15:05:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.656+0000 7ffbb5000640 1 -- start start 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbb5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb006d160 0x7ffbb013e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbb5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbb00cc500 0x7ffbb012f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbb5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 0x7ffbb00b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbb5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb00b39e0 con 0x7ffbb0143c40 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbb5000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb00b2dd0 con 0x7ffbb00cc500 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbb5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb00b30b0 con 0x7ffbb006d160 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 0x7ffbb00b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 0x7ffbb00b2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52368/0 (socket says 172.21.15.53:52368) 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.657+0000 7ffbafe00640 1 -- 172.21.15.53:0/3236160637 learned_addr learned my addr 172.21.15.53:0/3236160637 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbaf400640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb006d160 0x7ffbb013e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffba7e00640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbb00cc500 0x7ffbb012f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbafe00640 1 -- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb006d160 msgr2=0x7ffbb013e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbafe00640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb006d160 0x7ffbb013e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbafe00640 1 -- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbb00cc500 msgr2=0x7ffbb012f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbafe00640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbb00cc500 0x7ffbb012f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbafe00640 1 -- 172.21.15.53:0/3236160637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba8009a70 con 0x7ffbb0143c40 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffba7e00640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbb00cc500 0x7ffbb012f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.658+0000 7ffbafe00640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 0x7ffbb00b2570 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009d90 tx=0x7ffba8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.659+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba802cea0 con 0x7ffbb0143c40 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.659+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffba8040670 con 0x7ffbb0143c40 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.659+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8040a20 con 0x7ffbb0143c40 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.659+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb00b3390 con 0x7ffbb0143c40 2024-01-16T15:05:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.659+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb00b2ab0 con 0x7ffbb0143c40 2024-01-16T15:05:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.660+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0067450 con 0x7ffbb0143c40 2024-01-16T15:05:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.661+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7ffba8040b80 con 0x7ffbb0143c40 2024-01-16T15:05:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.662+0000 7ffba7400640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb8806b860 0x7ffb8806dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.662+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ffba80ad3a0 con 0x7ffbb0143c40 2024-01-16T15:05:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.663+0000 7ffbaf400640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb8806b860 0x7ffb8806dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.664+0000 7ffbaf400640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb8806b860 0x7ffb8806dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffba0000f60 tx=0x7ffba0007c00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:31.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.667+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffba8077960 con 0x7ffbb0143c40 2024-01-16T15:05:31.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:31.898+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ffbb00b3b20 con 0x7ffbb0143c40 2024-01-16T15:05:32.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.311+0000 7ffba7400640 1 -- 172.21.15.53:0/3236160637 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 136+0+0 (secure 0 0 0) 0x7ffba807af80 con 0x7ffbb0143c40 2024-01-16T15:05:32.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.314+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb8806b860 msgr2=0x7ffb8806dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.314+0000 7ffbb5000640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb8806b860 0x7ffb8806dd20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ffba0000f60 tx=0x7ffba0007c00 comp rx=0 tx=0).stop 2024-01-16T15:05:32.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 msgr2=0x7ffbb00b2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 0x7ffbb00b2570 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7ffba8009d90 tx=0x7ffba8004a40 comp rx=0 tx=0).stop 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 shutdown_connections 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb8806b860 0x7ffb8806dd20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffbb0143c40 0x7ffbb00b2570 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffbb00cc500 0x7ffbb012f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 --2- 172.21.15.53:0/3236160637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffbb006d160 0x7ffbb013e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 >> 172.21.15.53:0/3236160637 conn(0x7ffbb006a800 msgr2=0x7ffbb006d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 shutdown_connections 2024-01-16T15:05:32.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.315+0000 7ffbb5000640 1 -- 172.21.15.53:0/3236160637 wait complete. 2024-01-16T15:05:32.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:05:32.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.559+0000 7fb027200640 1 Processor -- start 2024-01-16T15:05:32.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.560+0000 7fb027200640 1 -- start start 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.561+0000 7fb027200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb020143c40 0x7fb020071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.561+0000 7fb027200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0200ccc10 0x7fb02006d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.561+0000 7fb027200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb020070e50 con 0x7fb0200717d0 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.561+0000 7fb027200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0200723c0 con 0x7fb020143c40 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.561+0000 7fb027200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb020144020 con 0x7fb0200ccc10 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb020143c40 0x7fb020071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb020143c40 0x7fb020071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47870/0 (socket says 172.21.15.53:47870) 2024-01-16T15:05:32.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 -- 172.21.15.53:0/2497582299 learned_addr learned my addr 172.21.15.53:0/2497582299 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:32.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb026800640 1 --2- 172.21.15.53:0/2497582299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0200ccc10 0x7fb02006d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:32.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 -- 172.21.15.53:0/2497582299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0200ccc10 msgr2=0x7fb02006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:32.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 --2- 172.21.15.53:0/2497582299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0200ccc10 0x7fb02006d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 -- 172.21.15.53:0/2497582299 >> v1:172.21.15.146:6789/0 conn(0x7fb0200717d0 legacy=0x7fb0200cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:32.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.562+0000 7fb025e00640 1 -- 172.21.15.53:0/2497582299 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb020071e00 con 0x7fb020143c40 2024-01-16T15:05:32.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.563+0000 7fb025e00640 1 --2- 172.21.15.53:0/2497582299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb020143c40 0x7fb020071290 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb010009e50 tx=0x7fb01002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73921713aa0b64ed server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:32.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.564+0000 7fb024a00640 1 -- 172.21.15.53:0/2497582299 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01002c9e0 con 0x7fb020143c40 2024-01-16T15:05:32.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.564+0000 7fb024a00640 1 -- 172.21.15.53:0/2497582299 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01002cb40 con 0x7fb020143c40 2024-01-16T15:05:32.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.564+0000 7fb024a00640 1 -- 172.21.15.53:0/2497582299 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0100343f0 con 0x7fb020143c40 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.564+0000 7fb027200640 1 -- 172.21.15.53:0/2497582299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb020143c40 msgr2=0x7fb020071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.564+0000 7fb027200640 1 --2- 172.21.15.53:0/2497582299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb020143c40 0x7fb020071290 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb010009e50 tx=0x7fb01002be40 comp rx=0 tx=0).stop 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.565+0000 7fb027200640 1 -- 172.21.15.53:0/2497582299 shutdown_connections 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.565+0000 7fb027200640 1 --2- 172.21.15.53:0/2497582299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0200ccc10 0x7fb02006d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.565+0000 7fb027200640 1 --2- 172.21.15.53:0/2497582299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb020143c40 0x7fb020071290 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.565+0000 7fb027200640 1 -- 172.21.15.53:0/2497582299 >> 172.21.15.53:0/2497582299 conn(0x7fb02006a800 msgr2=0x7fb02006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:32.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.565+0000 7fb027200640 1 -- 172.21.15.53:0/2497582299 shutdown_connections 2024-01-16T15:05:32.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.565+0000 7fb027200640 1 -- 172.21.15.53:0/2497582299 wait complete. 2024-01-16T15:05:32.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.567+0000 7fb027200640 1 Processor -- start 2024-01-16T15:05:32.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.567+0000 7fb027200640 1 -- start start 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb027200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0200717d0 0x7fb0201f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb027200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 0x7fb020138810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb027200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb020143c40 0x7fb020138d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb027200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020141980 con 0x7fb0200ccc10 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb027200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb020141050 con 0x7fb0200717d0 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb027200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb020141330 con 0x7fb020143c40 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb025400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 0x7fb020138810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb025400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 0x7fb020138810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47874/0 (socket says 172.21.15.53:47874) 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb025400640 1 -- 172.21.15.53:0/4119719220 learned_addr learned my addr 172.21.15.53:0/4119719220 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb025e00640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0200717d0 0x7fb0201f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:32.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.568+0000 7fb026800640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb020143c40 0x7fb020138d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb025400640 1 -- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb020143c40 msgr2=0x7fb020138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb025400640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb020143c40 0x7fb020138d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb025400640 1 -- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0200717d0 msgr2=0x7fb0201f2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb025400640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0200717d0 0x7fb0201f2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb025400640 1 -- 172.21.15.53:0/4119719220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb010009b30 con 0x7fb0200ccc10 2024-01-16T15:05:32.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb025400640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 0x7fb020138810 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb00800b3f0 tx=0x7fb00800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008011830 con 0x7fb0200ccc10 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb008011990 con 0x7fb0200ccc10 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb008011d40 con 0x7fb0200ccc10 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb020141610 con 0x7fb0200ccc10 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.569+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02014b140 con 0x7fb0200ccc10 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.570+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb020141b30 con 0x7fb0200ccc10 2024-01-16T15:05:32.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.571+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7fb00801a400 con 0x7fb0200ccc10 2024-01-16T15:05:32.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.572+0000 7fb01e000640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faff806b890 0x7faff806dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:32.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.572+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb00808a440 con 0x7fb0200ccc10 2024-01-16T15:05:32.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.573+0000 7fb025e00640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faff806b890 0x7faff806dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:32.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.574+0000 7fb025e00640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faff806b890 0x7faff806dd50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb010035000 tx=0x7fb010030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:32.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.577+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb020141b30 con 0x7fb0200ccc10 2024-01-16T15:05:32.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.799+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb02006cc40 con 0x7faff806b890 2024-01-16T15:05:32.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.803+0000 7fb01e000640 1 -- 172.21.15.53:0/4119719220 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb010044080 con 0x7faff806b890 2024-01-16T15:05:32.781 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:05:32.781 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:05:32.781 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:05:32.781 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:05:32.781 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:05:32.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faff806b890 msgr2=0x7faff806dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faff806b890 0x7faff806dd50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb010035000 tx=0x7fb010030000 comp rx=0 tx=0).stop 2024-01-16T15:05:32.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 msgr2=0x7fb020138810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 0x7fb020138810 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb00800b3f0 tx=0x7fb00800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 shutdown_connections 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faff806b890 0x7faff806dd50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb020143c40 0x7fb020138d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb0200ccc10 0x7fb020138810 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 --2- 172.21.15.53:0/4119719220 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb0200717d0 0x7fb0201f2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 >> 172.21.15.53:0/4119719220 conn(0x7fb02006a800 msgr2=0x7fb0200aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 shutdown_connections 2024-01-16T15:05:32.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:32.806+0000 7fb027200640 1 -- 172.21.15.53:0/4119719220 wait complete. 2024-01-16T15:05:32.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:05:33.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.049+0000 7f02a9c00640 1 Processor -- start 2024-01-16T15:05:33.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.049+0000 7f02a9c00640 1 -- start start 2024-01-16T15:05:33.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a9c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02a413df80 0x7f02a415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40cf9a0 0x7f02a40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a9c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a406c470 con 0x7f02a413e360 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a9c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a406c750 con 0x7f02a413df80 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a406ca30 con 0x7f02a40cf9a0 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02a413df80 0x7f02a415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02a413df80 0x7f02a415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47888/0 (socket says 172.21.15.53:47888) 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.051+0000 7f02a2a00640 1 -- 172.21.15.53:0/2222248783 learned_addr learned my addr 172.21.15.53:0/2222248783 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:33.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.052+0000 7f02a3400640 1 --2- 172.21.15.53:0/2222248783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40cf9a0 0x7f02a40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.052+0000 7f02a2a00640 1 -- 172.21.15.53:0/2222248783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40cf9a0 msgr2=0x7f02a40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:33.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.052+0000 7f02a2a00640 1 --2- 172.21.15.53:0/2222248783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40cf9a0 0x7f02a40cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.052+0000 7f02a2a00640 1 -- 172.21.15.53:0/2222248783 >> v1:172.21.15.146:6789/0 conn(0x7f02a413e360 legacy=0x7f02a40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.052+0000 7f02a2a00640 1 -- 172.21.15.53:0/2222248783 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a406cd10 con 0x7f02a413df80 2024-01-16T15:05:33.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.054+0000 7f02a2a00640 1 --2- 172.21.15.53:0/2222248783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a413df80 0x7f02a415bee0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f029402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=891e4e67a4e3f89c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:33.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a2000640 1 -- 172.21.15.53:0/2222248783 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294030840 con 0x7f02a413df80 2024-01-16T15:05:33.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a2000640 1 -- 172.21.15.53:0/2222248783 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02940309a0 con 0x7f02a413df80 2024-01-16T15:05:33.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a2000640 1 -- 172.21.15.53:0/2222248783 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294030d30 con 0x7f02a413df80 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a9c00640 1 -- 172.21.15.53:0/2222248783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a413df80 msgr2=0x7f02a415bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2222248783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a413df80 0x7f02a415bee0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f029402b980 comp rx=0 tx=0).stop 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a9c00640 1 -- 172.21.15.53:0/2222248783 shutdown_connections 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2222248783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40cf9a0 0x7f02a40cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2222248783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a413df80 0x7f02a415bee0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.055+0000 7f02a9c00640 1 -- 172.21.15.53:0/2222248783 >> 172.21.15.53:0/2222248783 conn(0x7f02a406a800 msgr2=0x7f02a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.056+0000 7f02a9c00640 1 -- 172.21.15.53:0/2222248783 shutdown_connections 2024-01-16T15:05:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.056+0000 7f02a9c00640 1 -- 172.21.15.53:0/2222248783 wait complete. 2024-01-16T15:05:33.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.057+0000 7f02a9c00640 1 Processor -- start 2024-01-16T15:05:33.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.057+0000 7f02a9c00640 1 -- start start 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.058+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 0x7f02a415a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.058+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a413df80 0x7f02a41f2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.058+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a413e360 0x7f02a41f2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.058+0000 7f02a9c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a40b33d0 con 0x7f02a40cf9a0 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.058+0000 7f02a9c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a40b2750 con 0x7f02a413df80 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.058+0000 7f02a9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a40b2a30 con 0x7f02a413e360 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 0x7f02a415a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 0x7f02a415a650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47890/0 (socket says 172.21.15.53:47890) 2024-01-16T15:05:33.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 -- 172.21.15.53:0/831072355 learned_addr learned my addr 172.21.15.53:0/831072355 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f029be00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a413df80 0x7f02a41f2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 -- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a413e360 msgr2=0x7f02a41f2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a413e360 0x7f02a41f2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 -- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a413df80 msgr2=0x7f02a41f2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a413df80 0x7f02a41f2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.059+0000 7f02a2a00640 1 -- 172.21.15.53:0/831072355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02940094d0 con 0x7f02a40cf9a0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.060+0000 7f02a2a00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 0x7f02a415a650 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f0294004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.060+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294039c90 con 0x7f02a40cf9a0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.060+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0294039df0 con 0x7f02a40cf9a0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.060+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a40b2d10 con 0x7f02a40cf9a0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.060+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294043620 con 0x7f02a40cf9a0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.060+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a40b4190 con 0x7f02a40cf9a0 2024-01-16T15:05:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.061+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a40b3510 con 0x7f02a40cf9a0 2024-01-16T15:05:33.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.062+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f0294043780 con 0x7f02a40cf9a0 2024-01-16T15:05:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.063+0000 7f029aa00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06b860 0x7f027c06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.063+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f029403e070 con 0x7f02a40cf9a0 2024-01-16T15:05:33.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.064+0000 7f029be00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06b860 0x7f027c06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.065+0000 7f029be00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06b860 0x7f027c06dd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02a413df50 tx=0x7f028c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.067+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02a40b3510 con 0x7f02a40cf9a0 2024-01-16T15:05:33.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.267+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f02a40b3510 con 0x7f02a40cf9a0 2024-01-16T15:05:33.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.267+0000 7f029aa00640 1 -- 172.21.15.53:0/831072355 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f02a40b3510 con 0x7f02a40cf9a0 2024-01-16T15:05:33.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06b860 msgr2=0x7f027c06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06b860 0x7f027c06dd20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02a413df50 tx=0x7f028c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 msgr2=0x7f02a415a650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 0x7f02a415a650 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f0294004730 comp rx=0 tx=0).stop 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 shutdown_connections 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06b860 0x7f027c06dd20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a413e360 0x7f02a41f2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a413df80 0x7f02a41f2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 --2- 172.21.15.53:0/831072355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40cf9a0 0x7f02a415a650 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 >> 172.21.15.53:0/831072355 conn(0x7f02a406a800 msgr2=0x7f02a40b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 shutdown_connections 2024-01-16T15:05:33.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.270+0000 7f02a9c00640 1 -- 172.21.15.53:0/831072355 wait complete. 2024-01-16T15:05:33.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:05:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.526+0000 7f7dfc800640 1 Processor -- start 2024-01-16T15:05:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.526+0000 7f7dfc800640 1 -- start start 2024-01-16T15:05:33.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.528+0000 7f7dfc800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df40cf9a0 0x7f7df415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.528+0000 7f7dfc800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7df415e1e0 0x7f7df40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.528+0000 7f7dfc800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7df40b1ee0 con 0x7f7df4143ae0 2024-01-16T15:05:33.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.528+0000 7f7dfc800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7df40b2780 con 0x7f7df415e1e0 2024-01-16T15:05:33.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.528+0000 7f7dfc800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df415e990 con 0x7f7df40cf9a0 2024-01-16T15:05:33.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7dfa800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7df415e1e0 0x7f7df40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7df9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df40cf9a0 0x7f7df415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7dfa800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7df415e1e0 0x7f7df40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:52622/0 (socket says 172.21.15.53:52622) 2024-01-16T15:05:33.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7dfa800640 1 -- 172.21.15.53:0/2223487583 learned_addr learned my addr 172.21.15.53:0/2223487583 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:33.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7dfa800640 1 -- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df40cf9a0 msgr2=0x7f7df415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:33.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7dfa800640 1 --2- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df40cf9a0 0x7f7df415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.529+0000 7f7dfa800640 1 -- 172.21.15.53:0/2223487583 >> v1:172.21.15.146:6789/0 conn(0x7f7df4143ae0 legacy=0x7f7df40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.530+0000 7f7dfa800640 1 -- 172.21.15.53:0/2223487583 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7df40b2130 con 0x7f7df415e1e0 2024-01-16T15:05:33.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.531+0000 7f7dfa800640 1 --2- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df415e1e0 0x7f7df40702c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009e50 tx=0x7f7dec02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb301968d8eb7443 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:33.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.532+0000 7f7df8a00640 1 -- 172.21.15.53:0/2223487583 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec02cb00 con 0x7f7df415e1e0 2024-01-16T15:05:33.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.532+0000 7f7df8a00640 1 -- 172.21.15.53:0/2223487583 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dec02cc60 con 0x7f7df415e1e0 2024-01-16T15:05:33.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.532+0000 7f7df8a00640 1 -- 172.21.15.53:0/2223487583 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dec0343f0 con 0x7f7df415e1e0 2024-01-16T15:05:33.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 -- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df415e1e0 msgr2=0x7f7df40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 --2- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df415e1e0 0x7f7df40702c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7dec009e50 tx=0x7f7dec02bf60 comp rx=0 tx=0).stop 2024-01-16T15:05:33.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 -- 172.21.15.53:0/2223487583 shutdown_connections 2024-01-16T15:05:33.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 --2- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df415e1e0 0x7f7df40702c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 --2- 172.21.15.53:0/2223487583 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df40cf9a0 0x7f7df415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 -- 172.21.15.53:0/2223487583 >> 172.21.15.53:0/2223487583 conn(0x7f7df406a800 msgr2=0x7f7df406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:33.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 -- 172.21.15.53:0/2223487583 shutdown_connections 2024-01-16T15:05:33.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.533+0000 7f7dfc800640 1 -- 172.21.15.53:0/2223487583 wait complete. 2024-01-16T15:05:33.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.535+0000 7f7dfc800640 1 Processor -- start 2024-01-16T15:05:33.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.536+0000 7f7dfc800640 1 -- start start 2024-01-16T15:05:33.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7dfc800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df40cf9a0 0x7f7df406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7dfc800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 0x7f7df40c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7dfc800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df415e1e0 0x7f7df40afa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7dfc800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df413fd40 con 0x7f7df4143ae0 2024-01-16T15:05:33.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7dfc800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7df40ae3e0 con 0x7f7df40cf9a0 2024-01-16T15:05:33.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7dfc800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7df413ffb0 con 0x7f7df415e1e0 2024-01-16T15:05:33.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7df9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 0x7f7df40c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7df9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 0x7f7df40c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47896/0 (socket says 172.21.15.53:47896) 2024-01-16T15:05:33.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.537+0000 7f7df9400640 1 -- 172.21.15.53:0/558632299 learned_addr learned my addr 172.21.15.53:0/558632299 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:33.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9e00640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df40cf9a0 0x7f7df406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7dfa800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df415e1e0 0x7f7df40afa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9400640 1 -- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df415e1e0 msgr2=0x7f7df40afa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9400640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df415e1e0 0x7f7df40afa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9400640 1 -- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df40cf9a0 msgr2=0x7f7df406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9400640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df40cf9a0 0x7f7df406c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9400640 1 -- 172.21.15.53:0/558632299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dec009b30 con 0x7f7df4143ae0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df9400640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 0x7f7df40c3e50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00e9b0 tx=0x7f7ddc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.538+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc009670 con 0x7f7df4143ae0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.539+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ddc00c9a0 con 0x7f7df4143ae0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.539+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7df4140290 con 0x7f7df4143ae0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.539+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc00cd50 con 0x7f7df4143ae0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.539+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7df40ae070 con 0x7f7df4143ae0 2024-01-16T15:05:33.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.540+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7df40675d0 con 0x7f7df4143ae0 2024-01-16T15:05:33.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.541+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f7ddc013470 con 0x7f7df4143ae0 2024-01-16T15:05:33.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.542+0000 7f7df2000640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7dcc06b860 0x7f7dcc06dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:33.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.543+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7ddc08a8d0 con 0x7f7df4143ae0 2024-01-16T15:05:33.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.543+0000 7f7df9e00640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7dcc06b860 0x7f7dcc06dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:33.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.544+0000 7f7df9e00640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7dcc06b860 0x7f7dcc06dd20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7df415dad0 tx=0x7f7de4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:33.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.547+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ddc054e90 con 0x7f7df4143ae0 2024-01-16T15:05:33.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.769+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f7df415e640 con 0x7f7dcc06b860 2024-01-16T15:05:33.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:05:33.779+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:33.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:05:33.779+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:33.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:05:33.779+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:33.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:05:33.779+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:33.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:05:33.779+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:05:33.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.843+0000 7f7df2000640 1 -- 172.21.15.53:0/558632299 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7de4001360 con 0x7f7dcc06b860 2024-01-16T15:05:33.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7dcc06b860 msgr2=0x7f7dcc06dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7dcc06b860 0x7f7dcc06dd20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7df415dad0 tx=0x7f7de4009040 comp rx=0 tx=0).stop 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 msgr2=0x7f7df40c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 0x7f7df40c3e50 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f7ddc00e9b0 tx=0x7f7ddc00ee70 comp rx=0 tx=0).stop 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 shutdown_connections 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7dcc06b860 0x7f7dcc06dd20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7df415e1e0 0x7f7df40afa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7df4143ae0 0x7f7df40c3e50 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 --2- 172.21.15.53:0/558632299 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7df40cf9a0 0x7f7df406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 >> 172.21.15.53:0/558632299 conn(0x7f7df406a800 msgr2=0x7f7df406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 shutdown_connections 2024-01-16T15:05:33.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:33.846+0000 7f7dfc800640 1 -- 172.21.15.53:0/558632299 wait complete. 2024-01-16T15:05:33.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:05:34.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.092+0000 7f158b200640 1 Processor -- start 2024-01-16T15:05:34.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.092+0000 7f158b200640 1 -- start start 2024-01-16T15:05:34.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.094+0000 7f158b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f158415f010 0x7f15840b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.094+0000 7f158b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158415f620 con 0x7f158415eb60 2024-01-16T15:05:34.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.094+0000 7f158b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158415f900 con 0x7f1584071290 2024-01-16T15:05:34.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.094+0000 7f158b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840ccb10 con 0x7f158415f010 2024-01-16T15:05:34.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.095+0000 7f1589400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f158415eb60 0x7f15840cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56390/0 (socket says 172.21.15.53:56390) 2024-01-16T15:05:34.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.095+0000 7f1589400640 1 -- 172.21.15.53:0/2557855938 learned_addr learned my addr 172.21.15.53:0/2557855938 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.095+0000 7f158a800640 1 --2- 172.21.15.53:0/2557855938 >> v2:172.21.15.146:3300/0 conn(0x7f158415f010 0x7f15840b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.095+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255050186 0 0) 0x7f158415f620 con 0x7f158415eb60 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.095+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1564003520 con 0x7f158415eb60 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186865047 0 0) 0x7f1564003520 con 0x7f158415eb60 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 >> v2:172.21.15.146:3300/0 conn(0x7f158415f010 msgr2=0x7f15840b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 --2- 172.21.15.53:0/2557855938 >> v2:172.21.15.146:3300/0 conn(0x7f158415f010 0x7f15840b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 >> v1:172.21.15.146:6789/0 conn(0x7f1584071290 legacy=0x7f1584071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15840ccd20 con 0x7f158415eb60 2024-01-16T15:05:34.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f156c001670 con 0x7f158415eb60 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 >> v1:172.21.15.53:6789/0 conn(0x7f158415eb60 legacy=0x7f15840cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f1564003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15640047c0 0x7f1564004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1564005180 0x7f156400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15840ccb10 con 0x7f1564003970 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f156400c020 con 0x7f15640047c0 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f156400c1e0 con 0x7f1564005180 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.096+0000 7f1589e00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f1564003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589400640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15640047c0 0x7f1564004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589e00640 1 -- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1564005180 msgr2=0x7f156400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589e00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1564005180 0x7f156400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589e00640 1 -- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15640047c0 msgr2=0x7f1564004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:05:34.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589e00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15640047c0 0x7f1564004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589e00640 1 -- 172.21.15.53:0/2557855938 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1564003520 con 0x7f1564003970 2024-01-16T15:05:34.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f1589400640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15640047c0 0x7f1564004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:34.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.097+0000 7f158a800640 1 -- 172.21.15.53:0/2557855938 reap_dead start 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.098+0000 7f1589e00640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f1564003d90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1574001750 tx=0x7f15640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a243c5b395c4b405 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.098+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15740052d0 con 0x7f1564003970 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.098+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157402e050 con 0x7f1564003970 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.098+0000 7f1588a00640 1 -- 172.21.15.53:0/2557855938 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574003830 con 0x7f1564003970 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 -- 172.21.15.53:0/2557855938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 msgr2=0x7f1564003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f1564003d90 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f1574001750 tx=0x7f15640057f0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 -- 172.21.15.53:0/2557855938 shutdown_connections 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 --2- 172.21.15.53:0/2557855938 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f1564003d90 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 -- 172.21.15.53:0/2557855938 >> 172.21.15.53:0/2557855938 conn(0x7f15840b0ed0 msgr2=0x7f15840cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:34.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 -- 172.21.15.53:0/2557855938 shutdown_connections 2024-01-16T15:05:34.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.099+0000 7f158b200640 1 -- 172.21.15.53:0/2557855938 wait complete. 2024-01-16T15:05:34.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.101+0000 7f158b200640 1 Processor -- start 2024-01-16T15:05:34.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.101+0000 7f158b200640 1 -- start start 2024-01-16T15:05:34.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f158b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f158415a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f158b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15840abb20 0x7f15840c3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f158b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15840c4050 0x7f15840a9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f158b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15840c44b0 con 0x7f1564003970 2024-01-16T15:05:34.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f158b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584131440 con 0x7f15840c4050 2024-01-16T15:05:34.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f158b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1584130830 con 0x7f15840abb20 2024-01-16T15:05:34.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f1589e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f158415a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f1589e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f158415a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47920/0 (socket says 172.21.15.53:47920) 2024-01-16T15:05:34.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.102+0000 7f1589e00640 1 -- 172.21.15.53:0/122840650 learned_addr learned my addr 172.21.15.53:0/122840650 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:05:34.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f158a800640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15840c4050 0x7f15840a9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589400640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15840abb20 0x7f15840c3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589e00640 1 -- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15840abb20 msgr2=0x7f15840c3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589e00640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15840abb20 0x7f15840c3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589e00640 1 -- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15840c4050 msgr2=0x7f15840a9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589e00640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15840c4050 0x7f15840a9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589e00640 1 -- 172.21.15.53:0/122840650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1584130b10 con 0x7f1564003970 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f158a800640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15840c4050 0x7f15840a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589400640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15840abb20 0x7f15840c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1589e00640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f158415a7d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1574005040 tx=0x7f1574001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.103+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574007a30 con 0x7f1564003970 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.104+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15740069e0 con 0x7f1564003970 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.104+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1584130df0 con 0x7f1564003970 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.104+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1574006d70 con 0x7f1564003970 2024-01-16T15:05:34.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.104+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15840abfa0 con 0x7f1564003970 2024-01-16T15:05:34.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.105+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15840c4430 con 0x7f1564003970 2024-01-16T15:05:34.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.106+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 15) v1 ==== 90049+0+0 (secure 0 0 0) 0x7f15740035f0 con 0x7f1564003970 2024-01-16T15:05:34.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.107+0000 7f1582000640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f155c06b890 0x7f155c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:05:34.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.107+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15740ac6c0 con 0x7f1564003970 2024-01-16T15:05:34.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.107+0000 7f1589400640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f155c06b890 0x7f155c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:05:34.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.108+0000 7f1589400640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f155c06b890 0x7f155c06dd50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f15840ab770 tx=0x7f156c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:05:34.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.113+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1574076ce0 con 0x7f1564003970 2024-01-16T15:05:34.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.341+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1584144990 con 0x7f155c06b890 2024-01-16T15:05:34.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.353+0000 7f1582000640 1 -- 172.21.15.53:0/122840650 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f156c00d040 con 0x7f155c06b890 2024-01-16T15:05:34.332 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c 2024-01-16T15:05:34.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.355+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f155c06b890 msgr2=0x7f155c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.355+0000 7f158b200640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f155c06b890 0x7f155c06dd50 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f15840ab770 tx=0x7f156c009040 comp rx=0 tx=0).stop 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 msgr2=0x7f158415a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f158415a7d0 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f1574005040 tx=0x7f1574001120 comp rx=0 tx=0).stop 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 shutdown_connections 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f155c06b890 0x7f155c06dd50 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15840c4050 0x7f15840a9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15840abb20 0x7f15840c3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 --2- 172.21.15.53:0/122840650 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1564003970 0x7f158415a7d0 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 >> 172.21.15.53:0/122840650 conn(0x7f15840b0ed0 msgr2=0x7f15840b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 shutdown_connections 2024-01-16T15:05:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:05:34.356+0000 7f158b200640 1 -- 172.21.15.53:0/122840650 wait complete. 2024-01-16T15:05:34.353 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c 2024-01-16T15:05:34.353 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:05:34.353 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:34.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:34.402 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:50.678 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:05:50.678 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:05:50.678 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 16.1466 s, 66.5 MB/s 2024-01-16T15:05:50.680 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:05:50.681 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:05:50.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:05:50.741 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:05:51.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:05:51.588 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:05.486 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:06:05.486 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:06:05.486 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.6781 s, 73.2 MB/s 2024-01-16T15:06:05.487 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:06:05.488 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:06:05.544 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:06:05.544 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:06:20.012 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:06:20.012 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:06:20.012 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.4015 s, 74.6 MB/s 2024-01-16T15:06:20.014 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:06:20.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.258+0000 7f7813800640 1 Processor -- start 2024-01-16T15:06:20.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.258+0000 7f7813800640 1 -- start start 2024-01-16T15:06:20.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.261+0000 7f7813800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f780c15e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.261+0000 7f7813800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780c06b690 con 0x7f780c0c6860 2024-01-16T15:06:20.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.261+0000 7f7813800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f780c06b970 con 0x7f780c0c7920 2024-01-16T15:06:20.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.261+0000 7f7813800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780c0c6060 con 0x7f780c15e040 2024-01-16T15:06:20.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.262+0000 7f7811800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f780c0c6860 0x7f780c0d1020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36414/0 (socket says 172.21.15.53:36414) 2024-01-16T15:06:20.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.262+0000 7f7811800640 1 -- 172.21.15.53:0/3153293290 learned_addr learned my addr 172.21.15.53:0/3153293290 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:20.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.262+0000 7f7810e00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f780c15e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.262+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069993522 0 0) 0x7f780c06b690 con 0x7f780c0c6860 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.262+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77ec003520 con 0x7f780c0c6860 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1356530366 0 0) 0x7f77ec003520 con 0x7f780c0c6860 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 msgr2=0x7f780c15e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f780c15e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 >> v1:172.21.15.146:6789/0 conn(0x7f780c0c7920 legacy=0x7f780c0cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c0c6340 con 0x7f780c0c6860 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f78040019f0 con 0x7f780c0c6860 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 >> v1:172.21.15.53:6789/0 conn(0x7f780c0c6860 legacy=0x7f780c0d1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f77ec0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ec003f70 0x7f77ec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77ec004d00 0x7f77ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c0c6060 con 0x7f77ec003f70 2024-01-16T15:06:20.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec00bac0 con 0x7f77ec004d00 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77ec00bc80 con 0x7f780c15e040 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f780be00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ec003f70 0x7f77ec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.263+0000 7f7810e00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f77ec0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.264+0000 7f780be00640 1 -- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 msgr2=0x7f77ec0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.264+0000 7f780be00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f77ec0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.264+0000 7f780be00640 1 -- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77ec004d00 msgr2=0x7f77ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.264+0000 7f780be00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f77ec004d00 0x7f77ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.264+0000 7f780be00640 1 -- 172.21.15.53:0/3153293290 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77ec003520 con 0x7f77ec003f70 2024-01-16T15:06:20.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.264+0000 7f7810e00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f780c15e040 0x7f77ec0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:20.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.265+0000 7f780be00640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ec003f70 0x7f77ec0047c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f77f4001750 tx=0x7f77ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d641e485a7000d00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.265+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f40052d0 con 0x7f77ec003f70 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.265+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f402e050 con 0x7f77ec003f70 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.265+0000 7f780b400640 1 -- 172.21.15.53:0/3153293290 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4003830 con 0x7f77ec003f70 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7813800640 1 -- 172.21.15.53:0/3153293290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ec003f70 msgr2=0x7f77ec0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7813800640 1 --2- 172.21.15.53:0/3153293290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77ec003f70 0x7f77ec0047c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f77f4001750 tx=0x7f77ec0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7811800640 1 -- 172.21.15.53:0/3153293290 reap_dead start 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7813800640 1 -- 172.21.15.53:0/3153293290 shutdown_connections 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7813800640 1 -- 172.21.15.53:0/3153293290 >> 172.21.15.53:0/3153293290 conn(0x7f780c0b2ce0 msgr2=0x7f780c0b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7813800640 1 -- 172.21.15.53:0/3153293290 shutdown_connections 2024-01-16T15:06:20.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.266+0000 7f7813800640 1 -- 172.21.15.53:0/3153293290 wait complete. 2024-01-16T15:06:20.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.268+0000 7f7813800640 1 Processor -- start 2024-01-16T15:06:20.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.268+0000 7f7813800640 1 -- start start 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f7813800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77ec003f70 0x7f780c0b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f7813800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 0x7f780c1466e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f7813800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f780c0ae9c0 0x7f780c146c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f7813800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c1e8450 con 0x7f780c0ae560 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f7813800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780c0b08f0 con 0x7f780c0ae9c0 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f7813800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f780c1e7810 con 0x7f77ec003f70 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.269+0000 7f780be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 0x7f780c1466e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 0x7f780c1466e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59578/0 (socket says 172.21.15.53:59578) 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 -- 172.21.15.53:0/3579912674 learned_addr learned my addr 172.21.15.53:0/3579912674 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:20.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f7810e00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77ec003f70 0x7f780c0b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f7811800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f780c0ae9c0 0x7f780c146c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 -- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77ec003f70 msgr2=0x7f780c0b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77ec003f70 0x7f780c0b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 -- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f780c0ae9c0 msgr2=0x7f780c146c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f780c0ae9c0 0x7f780c146c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 -- 172.21.15.53:0/3579912674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780c0cf4f0 con 0x7f780c0ae560 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.270+0000 7f780be00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 0x7f780c1466e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f77f4005040 tx=0x7f77f4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.271+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4007a30 con 0x7f780c0ae560 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.271+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77f40069e0 con 0x7f780c0ae560 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.271+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780c1e7ab0 con 0x7f780c0ae560 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.271+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4006d90 con 0x7f780c0ae560 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.271+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780c1e8070 con 0x7f780c0ae560 2024-01-16T15:06:20.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.272+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f780c06b690 con 0x7f780c0ae560 2024-01-16T15:06:20.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.273+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f77f40035f0 con 0x7f780c0ae560 2024-01-16T15:06:20.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.274+0000 7f7808c00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f77e406b8f0 0x7f77e406ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.275+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f77f40ac7f0 con 0x7f780c0ae560 2024-01-16T15:06:20.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.275+0000 7f7810e00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f77e406b8f0 0x7f77e406ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.276+0000 7f7810e00640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f77e406b8f0 0x7f77e406ddb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009aa0 tx=0x7f77fc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.279+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77f4076bd0 con 0x7f780c0ae560 2024-01-16T15:06:20.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.474+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f780c0ce4f0 con 0x7f780c0ae560 2024-01-16T15:06:20.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.479+0000 7f7808c00640 1 -- 172.21.15.53:0/3579912674 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v21)=0 v21) v1 ==== 131+0+0 (secure 0 0 0) 0x7f77f407a1f0 con 0x7f780c0ae560 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.482+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f77e406b8f0 msgr2=0x7f77e406ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.482+0000 7f7813800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f77e406b8f0 0x7f77e406ddb0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f77fc009aa0 tx=0x7f77fc008040 comp rx=0 tx=0).stop 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.482+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 msgr2=0x7f780c1466e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.482+0000 7f7813800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 0x7f780c1466e0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f77f4005040 tx=0x7f77f4001120 comp rx=0 tx=0).stop 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 shutdown_connections 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f77e406b8f0 0x7f77e406ddb0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f780c0ae9c0 0x7f780c146c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f780c0ae560 0x7f780c1466e0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 --2- 172.21.15.53:0/3579912674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f77ec003f70 0x7f780c0b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 >> 172.21.15.53:0/3579912674 conn(0x7f780c0b2ce0 msgr2=0x7f780c0cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 shutdown_connections 2024-01-16T15:06:20.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.483+0000 7f7813800640 1 -- 172.21.15.53:0/3579912674 wait complete. 2024-01-16T15:06:20.479 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 2 2024-01-16T15:06:20.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.727+0000 7f8049200640 1 Processor -- start 2024-01-16T15:06:20.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.727+0000 7f8049200640 1 -- start start 2024-01-16T15:06:20.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.729+0000 7f8049200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f804413e360 0x7f80440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.729+0000 7f8049200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804415c5f0 con 0x7f804413df80 2024-01-16T15:06:20.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.729+0000 7f8049200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f804406c470 con 0x7f804415f9c0 2024-01-16T15:06:20.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.729+0000 7f8049200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f804406c750 con 0x7f804413e360 2024-01-16T15:06:20.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8041e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f804413df80 0x7f804415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36420/0 (socket says 172.21.15.53:36420) 2024-01-16T15:06:20.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8041e00640 1 -- 172.21.15.53:0/2998080596 learned_addr learned my addr 172.21.15.53:0/2998080596 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:20.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8041400640 1 --2- 172.21.15.53:0/2998080596 >> v2:172.21.15.53:3300/0 conn(0x7f804413e360 0x7f80440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8040a00640 1 -- 172.21.15.53:0/2998080596 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054576947 0 0) 0x7f804415c5f0 con 0x7f804413df80 2024-01-16T15:06:20.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8040a00640 1 -- 172.21.15.53:0/2998080596 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8024003520 con 0x7f804413df80 2024-01-16T15:06:20.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8041400640 1 -- 172.21.15.53:0/2998080596 >> v1:172.21.15.146:6789/0 conn(0x7f804415f9c0 legacy=0x7f80440cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:20.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8041400640 1 -- 172.21.15.53:0/2998080596 >> v1:172.21.15.53:6789/0 conn(0x7f804413df80 legacy=0x7f804415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.730+0000 7f8041400640 1 -- 172.21.15.53:0/2998080596 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f804406ca30 con 0x7f804413e360 2024-01-16T15:06:20.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.731+0000 7f8041400640 1 --2- 172.21.15.53:0/2998080596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804413e360 0x7f80440cf290 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f802c0097a0 tx=0x7f802c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a126f0ed028d9e00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8040a00640 1 -- 172.21.15.53:0/2998080596 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c031830 con 0x7f804413e360 2024-01-16T15:06:20.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8040a00640 1 -- 172.21.15.53:0/2998080596 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c031990 con 0x7f804413e360 2024-01-16T15:06:20.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8040a00640 1 -- 172.21.15.53:0/2998080596 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c031d20 con 0x7f804413e360 2024-01-16T15:06:20.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8049200640 1 -- 172.21.15.53:0/2998080596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804413e360 msgr2=0x7f80440cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8049200640 1 --2- 172.21.15.53:0/2998080596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804413e360 0x7f80440cf290 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f802c0097a0 tx=0x7f802c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8049200640 1 -- 172.21.15.53:0/2998080596 shutdown_connections 2024-01-16T15:06:20.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.732+0000 7f8049200640 1 --2- 172.21.15.53:0/2998080596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804413e360 0x7f80440cf290 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.733+0000 7f8049200640 1 -- 172.21.15.53:0/2998080596 >> 172.21.15.53:0/2998080596 conn(0x7f804406a800 msgr2=0x7f804406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.733+0000 7f8049200640 1 -- 172.21.15.53:0/2998080596 shutdown_connections 2024-01-16T15:06:20.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.733+0000 7f8049200640 1 -- 172.21.15.53:0/2998080596 wait complete. 2024-01-16T15:06:20.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.734+0000 7f8049200640 1 Processor -- start 2024-01-16T15:06:20.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.735+0000 7f8049200640 1 -- start start 2024-01-16T15:06:20.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.735+0000 7f8049200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804413df80 0x7f804413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.735+0000 7f8049200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f804413e360 0x7f80441e6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8049200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 0x7f804415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8049200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80441e67d0 con 0x7f804415f9c0 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8049200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8044151830 con 0x7f804413e360 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8049200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80440b2130 con 0x7f804413df80 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 0x7f804415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 0x7f804415d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59596/0 (socket says 172.21.15.53:59596) 2024-01-16T15:06:20.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 -- 172.21.15.53:0/3473709624 learned_addr learned my addr 172.21.15.53:0/3473709624 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:20.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 -- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804413df80 msgr2=0x7f804413d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804413df80 0x7f804413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 -- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f804413e360 msgr2=0x7f80441e6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8041400640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f804413e360 0x7f80441e6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f804413e360 0x7f80441e6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.736+0000 7f8042800640 1 -- 172.21.15.53:0/3473709624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c009400 con 0x7f804415f9c0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.737+0000 7f8042800640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 0x7f804415d8a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f803c00c700 tx=0x7f803c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.737+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c00aa50 con 0x7f804415f9c0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.737+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f803c00abb0 con 0x7f804415f9c0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.737+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80440b2410 con 0x7f804415f9c0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.737+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f803c00ce10 con 0x7f804415f9c0 2024-01-16T15:06:20.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.737+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80440b29d0 con 0x7f804415f9c0 2024-01-16T15:06:20.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.738+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80440675d0 con 0x7f804415f9c0 2024-01-16T15:06:20.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.739+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f803c012420 con 0x7f804415f9c0 2024-01-16T15:06:20.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.740+0000 7f803a000640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f801c06b920 0x7f801c06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:20.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.740+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f803c08ac10 con 0x7f804415f9c0 2024-01-16T15:06:20.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.741+0000 7f8041e00640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f801c06b920 0x7f801c06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:20.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.742+0000 7f8041e00640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f801c06b920 0x7f801c06dde0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f804415c780 tx=0x7f8034007930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:20.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.753+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f803c055100 con 0x7f804415f9c0 2024-01-16T15:06:20.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.955+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f804406df00 con 0x7f804415f9c0 2024-01-16T15:06:20.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.961+0000 7f803a000640 1 -- 172.21.15.53:0/3473709624 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v22)=0 v22) v1 ==== 135+0+0 (secure 0 0 0) 0x7f803c0160c0 con 0x7f804415f9c0 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f801c06b920 msgr2=0x7f801c06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f801c06b920 0x7f801c06dde0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f804415c780 tx=0x7f8034007930 comp rx=0 tx=0).stop 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 msgr2=0x7f804415d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 0x7f804415d8a0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f803c00c700 tx=0x7f803c00cbc0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 shutdown_connections 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f801c06b920 0x7f801c06dde0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f804415f9c0 0x7f804415d8a0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f804413e360 0x7f80441e6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 --2- 172.21.15.53:0/3473709624 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f804413df80 0x7f804413d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 >> 172.21.15.53:0/3473709624 conn(0x7f804406a800 msgr2=0x7f80440706a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:20.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 shutdown_connections 2024-01-16T15:06:20.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:20.964+0000 7f8049200640 1 -- 172.21.15.53:0/3473709624 wait complete. 2024-01-16T15:06:20.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:06:21.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.207+0000 7f46a2800640 1 Processor -- start 2024-01-16T15:06:21.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.208+0000 7f46a2800640 1 -- start start 2024-01-16T15:06:21.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.209+0000 7f46a2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c0b3f30 0x7f469c0d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.209+0000 7f46a2800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0d0a10 con 0x7f469c0d0100 2024-01-16T15:06:21.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.209+0000 7f46a2800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f469c0cf080 con 0x7f469c160920 2024-01-16T15:06:21.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.209+0000 7f46a2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0c6be0 con 0x7f469c0b3f30 2024-01-16T15:06:21.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.209+0000 7f46a0a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f469c0d0100 0x7f469c0d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36434/0 (socket says 172.21.15.53:36434) 2024-01-16T15:06:21.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.209+0000 7f46a0a00640 1 -- 172.21.15.53:0/2484723606 learned_addr learned my addr 172.21.15.53:0/2484723606 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:21.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f46a1e00640 1 --2- 172.21.15.53:0/2484723606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c0b3f30 0x7f469c0d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f469be00640 1 -- 172.21.15.53:0/2484723606 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374744064 0 0) 0x7f469c0d0a10 con 0x7f469c0d0100 2024-01-16T15:06:21.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f469be00640 1 -- 172.21.15.53:0/2484723606 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f467c003520 con 0x7f469c0d0100 2024-01-16T15:06:21.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f469be00640 1 -- 172.21.15.53:0/2484723606 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2252890309 0 0) 0x7f467c003520 con 0x7f469c0d0100 2024-01-16T15:06:21.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f46a1e00640 1 -- 172.21.15.53:0/2484723606 >> v1:172.21.15.146:6789/0 conn(0x7f469c160920 legacy=0x7f469c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f46a1e00640 1 -- 172.21.15.53:0/2484723606 >> v1:172.21.15.53:6789/0 conn(0x7f469c0d0100 legacy=0x7f469c0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.210+0000 7f46a1e00640 1 -- 172.21.15.53:0/2484723606 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f469c0ce7e0 con 0x7f469c0b3f30 2024-01-16T15:06:21.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.212+0000 7f46a1e00640 1 --2- 172.21.15.53:0/2484723606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c0b3f30 0x7f469c0d1650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4694009dc0 tx=0x7f469402bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24c4b325488860b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.212+0000 7f469be00640 1 -- 172.21.15.53:0/2484723606 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684001670 con 0x7f469c0b3f30 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.212+0000 7f469be00640 1 -- 172.21.15.53:0/2484723606 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469402c910 con 0x7f469c0b3f30 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.212+0000 7f469be00640 1 -- 172.21.15.53:0/2484723606 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469402cca0 con 0x7f469c0b3f30 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 -- 172.21.15.53:0/2484723606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c0b3f30 msgr2=0x7f469c0d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 --2- 172.21.15.53:0/2484723606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c0b3f30 0x7f469c0d1650 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4694009dc0 tx=0x7f469402bde0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 -- 172.21.15.53:0/2484723606 shutdown_connections 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 --2- 172.21.15.53:0/2484723606 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c0b3f30 0x7f469c0d1650 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 -- 172.21.15.53:0/2484723606 >> 172.21.15.53:0/2484723606 conn(0x7f469c0d3090 msgr2=0x7f469c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:21.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 -- 172.21.15.53:0/2484723606 shutdown_connections 2024-01-16T15:06:21.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.213+0000 7f46a2800640 1 -- 172.21.15.53:0/2484723606 wait complete. 2024-01-16T15:06:21.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.215+0000 7f46a2800640 1 Processor -- start 2024-01-16T15:06:21.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.215+0000 7f46a2800640 1 -- start start 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 0x7f469c0ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f469c0d0100 0x7f469c0b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c160920 0x7f469c15eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a2800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c153910 con 0x7f469c0b3f30 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0b83f0 con 0x7f469c0d0100 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f469c0b77e0 con 0x7f469c160920 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 0x7f469c0ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 0x7f469c0ca2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59606/0 (socket says 172.21.15.53:59606) 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.216+0000 7f46a1400640 1 -- 172.21.15.53:0/2837215826 learned_addr learned my addr 172.21.15.53:0/2837215826 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1e00640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c160920 0x7f469c15eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a0a00640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f469c0d0100 0x7f469c0b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1400640 1 -- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c160920 msgr2=0x7f469c15eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1400640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c160920 0x7f469c15eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1400640 1 -- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f469c0d0100 msgr2=0x7f469c0b7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1400640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f469c0d0100 0x7f469c0b7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1400640 1 -- 172.21.15.53:0/2837215826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4694009a70 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.217+0000 7f46a1400640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 0x7f469c0ca2f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f468c00c3c0 tx=0x7f468c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.218+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c0052d0 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.218+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468c005450 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.218+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468c007260 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.218+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469c0b7ac0 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.218+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469c0b8080 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.219+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469c0cd070 con 0x7f469c0b3f30 2024-01-16T15:06:21.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.220+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f468c0073c0 con 0x7f469c0b3f30 2024-01-16T15:06:21.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.221+0000 7f4699600640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f467406b920 0x7f467406dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.221+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f468c089aa0 con 0x7f469c0b3f30 2024-01-16T15:06:21.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.221+0000 7f46a0a00640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f467406b920 0x7f467406dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.223+0000 7f46a0a00640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f467406b920 0x7f467406dde0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f469c15df80 tx=0x7f4684009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.225+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f468c054020 con 0x7f469c0b3f30 2024-01-16T15:06:21.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.450+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f469c160d00 con 0x7f467406b920 2024-01-16T15:06:21.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.489+0000 7f4699600640 1 -- 172.21.15.53:0/2837215826 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46840058d0 con 0x7f467406b920 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f467406b920 msgr2=0x7f467406dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f467406b920 0x7f467406dde0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f469c15df80 tx=0x7f4684009040 comp rx=0 tx=0).stop 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 msgr2=0x7f469c0ca2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 0x7f469c0ca2f0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f468c00c3c0 tx=0x7f468c00c880 comp rx=0 tx=0).stop 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 shutdown_connections 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f467406b920 0x7f467406dde0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f469c160920 0x7f469c15eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f469c0d0100 0x7f469c0b7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 --2- 172.21.15.53:0/2837215826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f469c0b3f30 0x7f469c0ca2f0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 >> 172.21.15.53:0/2837215826 conn(0x7f469c0d3090 msgr2=0x7f469c0c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:21.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 shutdown_connections 2024-01-16T15:06:21.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.492+0000 7f46a2800640 1 -- 172.21.15.53:0/2837215826 wait complete. 2024-01-16T15:06:21.490 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:06:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.736+0000 7f394ee00640 1 Processor -- start 2024-01-16T15:06:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.737+0000 7f394ee00640 1 -- start start 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.738+0000 7f394ee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f39480cf9a0 0x7f39480cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.738+0000 7f394ee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f39480b31d0 0x7f39480b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394ee00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39480d0320 con 0x7f394815d4b0 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394ee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39480d1c60 con 0x7f39480cf9a0 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394ee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39480b2730 con 0x7f39480b31d0 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f394815d4b0 0x7f39480d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36450/0 (socket says 172.21.15.53:36450) 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394da00640 1 -- 172.21.15.53:0/502253486 learned_addr learned my addr 172.21.15.53:0/502253486 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:21.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394d000640 1 --2- 172.21.15.53:0/502253486 >> v2:172.21.15.53:3300/0 conn(0x7f39480cf9a0 0x7f39480cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394d000640 1 -- 172.21.15.53:0/502253486 >> v2:172.21.15.146:3300/0 conn(0x7f39480b31d0 msgr2=0x7f39480b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394d000640 1 --2- 172.21.15.53:0/502253486 >> v2:172.21.15.146:3300/0 conn(0x7f39480b31d0 0x7f39480b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394d000640 1 -- 172.21.15.53:0/502253486 >> v1:172.21.15.53:6789/0 conn(0x7f394815d4b0 legacy=0x7f39480d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f3947e00640 1 -- 172.21.15.53:0/502253486 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825132388 0 0) 0x7f39480d0320 con 0x7f394815d4b0 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.739+0000 7f394d000640 1 -- 172.21.15.53:0/502253486 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39480d1ec0 con 0x7f39480cf9a0 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.741+0000 7f394d000640 1 --2- 172.21.15.53:0/502253486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480cf9a0 0x7f39480cfde0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3948072a30 tx=0x7f393002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f933f967a1aa21d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.741+0000 7f3947e00640 1 -- 172.21.15.53:0/502253486 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930031830 con 0x7f39480cf9a0 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.741+0000 7f3947e00640 1 -- 172.21.15.53:0/502253486 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3930031990 con 0x7f39480cf9a0 2024-01-16T15:06:21.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.741+0000 7f3947e00640 1 -- 172.21.15.53:0/502253486 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930031d20 con 0x7f39480cf9a0 2024-01-16T15:06:21.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 -- 172.21.15.53:0/502253486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480cf9a0 msgr2=0x7f39480cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 --2- 172.21.15.53:0/502253486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480cf9a0 0x7f39480cfde0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f3948072a30 tx=0x7f393002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 -- 172.21.15.53:0/502253486 shutdown_connections 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 --2- 172.21.15.53:0/502253486 >> v2:172.21.15.146:3300/0 conn(0x7f39480b31d0 0x7f39480b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 --2- 172.21.15.53:0/502253486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480cf9a0 0x7f39480cfde0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 -- 172.21.15.53:0/502253486 >> 172.21.15.53:0/502253486 conn(0x7f39480d3000 msgr2=0x7f39480d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 -- 172.21.15.53:0/502253486 shutdown_connections 2024-01-16T15:06:21.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.742+0000 7f394ee00640 1 -- 172.21.15.53:0/502253486 wait complete. 2024-01-16T15:06:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.744+0000 7f394ee00640 1 Processor -- start 2024-01-16T15:06:21.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.744+0000 7f394ee00640 1 -- start start 2024-01-16T15:06:21.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 0x7f394809d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39480cf9a0 0x7f39480ccf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f394815d4b0 0x7f3948155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39480cd440 con 0x7f39480b31d0 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f394809e5e0 con 0x7f394815d4b0 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39480ca530 con 0x7f39480cf9a0 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 0x7f394809d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.745+0000 7f394da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 0x7f394809d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59610/0 (socket says 172.21.15.53:59610) 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 -- 172.21.15.53:0/3438975955 learned_addr learned my addr 172.21.15.53:0/3438975955 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394e400640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f394815d4b0 0x7f3948155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394d000640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39480cf9a0 0x7f39480ccf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 -- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39480cf9a0 msgr2=0x7f39480ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39480cf9a0 0x7f39480ccf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 -- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f394815d4b0 msgr2=0x7f3948155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f394815d4b0 0x7f3948155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 -- 172.21.15.53:0/3438975955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3930009400 con 0x7f39480b31d0 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394e400640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f394815d4b0 0x7f3948155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.746+0000 7f394da00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 0x7f394809d6f0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f393800c430 tx=0x7f393800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.747+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39380074e0 con 0x7f39480b31d0 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.747+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3938007660 con 0x7f39480b31d0 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.747+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938004a20 con 0x7f39480b31d0 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.747+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3948156390 con 0x7f39480b31d0 2024-01-16T15:06:21.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.747+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f394809e020 con 0x7f39480b31d0 2024-01-16T15:06:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.748+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3948067450 con 0x7f39480b31d0 2024-01-16T15:06:21.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.749+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f3938004b80 con 0x7f39480b31d0 2024-01-16T15:06:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.750+0000 7f3945600640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f392006b8f0 0x7f392006ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.750+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f393808a580 con 0x7f39480b31d0 2024-01-16T15:06:21.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.751+0000 7f394d000640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f392006b8f0 0x7f392006ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:21.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.752+0000 7f394d000640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f392006b8f0 0x7f392006ddb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3930036000 tx=0x7f393003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:21.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.755+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3938054b00 con 0x7f39480b31d0 2024-01-16T15:06:21.744 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:21.749 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:21.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.984+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f39480d21a0 con 0x7f392006b8f0 2024-01-16T15:06:21.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:21.998+0000 7f3945600640 1 -- 172.21.15.53:0/3438975955 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f393003a420 con 0x7f392006b8f0 2024-01-16T15:06:21.976 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c 2024-01-16T15:06:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f392006b8f0 msgr2=0x7f392006ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f392006b8f0 0x7f392006ddb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f3930036000 tx=0x7f393003c040 comp rx=0 tx=0).stop 2024-01-16T15:06:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 msgr2=0x7f394809d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 0x7f394809d6f0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f393800c430 tx=0x7f393800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 shutdown_connections 2024-01-16T15:06:21.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f392006b8f0 0x7f392006ddb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f394815d4b0 0x7f3948155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39480cf9a0 0x7f39480ccf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 --2- 172.21.15.53:0/3438975955 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39480b31d0 0x7f394809d6f0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:21.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 >> 172.21.15.53:0/3438975955 conn(0x7f39480d3000 msgr2=0x7f394806ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:21.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 shutdown_connections 2024-01-16T15:06:21.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:22.001+0000 7f394ee00640 1 -- 172.21.15.53:0/3438975955 wait complete. 2024-01-16T15:06:21.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:06:21.999 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c) 2024-01-16T15:06:22.067 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c 2024-01-16T15:06:22.067 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:06:22.067 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:22.069 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:06:24.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:06:24.070 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c) 2024-01-16T15:06:24.141 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c 2024-01-16T15:06:24.141 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:06:24.141 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:24.142 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:06:26.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:06:26.143 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c) 2024-01-16T15:06:26.210 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/36b9e7ea-df7d-40a4-bcf0-9c37854b939c 2024-01-16T15:06:26.210 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:06:26.210 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:26.211 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:06:26.211 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30231 2024-01-16T15:06:26.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.466+0000 7fe3df200640 1 Processor -- start 2024-01-16T15:06:26.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.466+0000 7fe3df200640 1 -- start start 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3df200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00d0dc0 0x7fe3e00d11a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3df200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe3e00cf1d0 0x7fe3e00cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3df200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e00b1e00 con 0x7fe3e015d410 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3df200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e00cfbc0 con 0x7fe3e00cf1d0 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3df200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e00b1b20 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3de800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe3e00cf1d0 0x7fe3e00cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3de800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe3e00cf1d0 0x7fe3e00cf680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49752/0 (socket says 172.21.15.53:49752) 2024-01-16T15:06:26.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3de800640 1 -- 172.21.15.53:0/3198610449 learned_addr learned my addr 172.21.15.53:0/3198610449 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3dde00640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00d0dc0 0x7fe3e00d11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.468+0000 7fe3de800640 1 -- 172.21.15.53:0/3198610449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00d0dc0 msgr2=0x7fe3e00d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.469+0000 7fe3de800640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00d0dc0 0x7fe3e00d11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.469+0000 7fe3de800640 1 -- 172.21.15.53:0/3198610449 >> v1:172.21.15.146:6789/0 conn(0x7fe3e015d410 legacy=0x7fe3e006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.469+0000 7fe3de800640 1 -- 172.21.15.53:0/3198610449 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e006b620 con 0x7fe3e00cf1d0 2024-01-16T15:06:26.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.469+0000 7fe3dde00640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00d0dc0 0x7fe3e00d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.470+0000 7fe3de800640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00cf1d0 0x7fe3e00cf680 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009e50 tx=0x7fe3d802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eab6e8f57218ba5c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.470+0000 7fe3dca00640 1 -- 172.21.15.53:0/3198610449 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d802c9e0 con 0x7fe3e00cf1d0 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.470+0000 7fe3dca00640 1 -- 172.21.15.53:0/3198610449 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d802cb40 con 0x7fe3e00cf1d0 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.470+0000 7fe3dca00640 1 -- 172.21.15.53:0/3198610449 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3d80343f0 con 0x7fe3e00cf1d0 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 -- 172.21.15.53:0/3198610449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00cf1d0 msgr2=0x7fe3e00cf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00cf1d0 0x7fe3e00cf680 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe3d8009e50 tx=0x7fe3d802be40 comp rx=0 tx=0).stop 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 -- 172.21.15.53:0/3198610449 shutdown_connections 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00cf1d0 0x7fe3e00cf680 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 --2- 172.21.15.53:0/3198610449 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00d0dc0 0x7fe3e00d11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 -- 172.21.15.53:0/3198610449 >> 172.21.15.53:0/3198610449 conn(0x7fe3e0099a60 msgr2=0x7fe3e0099e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:26.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 -- 172.21.15.53:0/3198610449 shutdown_connections 2024-01-16T15:06:26.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.471+0000 7fe3df200640 1 -- 172.21.15.53:0/3198610449 wait complete. 2024-01-16T15:06:26.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.473+0000 7fe3df200640 1 Processor -- start 2024-01-16T15:06:26.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.473+0000 7fe3df200640 1 -- start start 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3df200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00cf1d0 0x7fe3e0153290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3df200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 0x7fe3e015bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3df200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3e015d410 0x7fe3e0159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3df200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e015ac60 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3df200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e015af40 con 0x7fe3e015d410 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3df200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e015b220 con 0x7fe3e00cf1d0 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3dd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 0x7fe3e015bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3dd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 0x7fe3e015bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49756/0 (socket says 172.21.15.53:49756) 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3dd400640 1 -- 172.21.15.53:0/2823018417 learned_addr learned my addr 172.21.15.53:0/2823018417 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3dde00640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00cf1d0 0x7fe3e0153290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.474+0000 7fe3de800640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3e015d410 0x7fe3e0159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dd400640 1 -- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00cf1d0 msgr2=0x7fe3e0153290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dd400640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00cf1d0 0x7fe3e0153290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dd400640 1 -- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3e015d410 msgr2=0x7fe3e0159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dd400640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3e015d410 0x7fe3e0159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dd400640 1 -- 172.21.15.53:0/2823018417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d8009b30 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dde00640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00cf1d0 0x7fe3e0153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:26.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3dd400640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 0x7fe3e015bfe0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800c430 tx=0x7fe3c800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8011790 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c80118f0 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e015b500 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.475+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c8011ce0 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.476+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e015bac0 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.477+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e015cc50 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.478+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7fe3c801c460 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.479+0000 7fe3d6000640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe3b806b8f0 0x7fe3b806ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:26.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.479+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe3c808a6a0 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.479+0000 7fe3dde00640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe3b806b8f0 0x7fe3b806ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:26.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.481+0000 7fe3dde00640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe3b806b8f0 0x7fe3b806ddb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0006fd0 tx=0x7fe3d0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:26.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.483+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe3c8054bd0 con 0x7fe3e00d0dc0 2024-01-16T15:06:26.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.715+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30231", "target": ["mon-mgr", ""]}) v1 -- 0x7fe3e00d16e0 con 0x7fe3b806b8f0 2024-01-16T15:06:26.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.877+0000 7fe3d6000640 1 -- 172.21.15.53:0/2823018417 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe3d000c040 con 0x7fe3b806b8f0 2024-01-16T15:06:26.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe3b806b8f0 msgr2=0x7fe3b806ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe3b806b8f0 0x7fe3b806ddb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe3d0006fd0 tx=0x7fe3d0008040 comp rx=0 tx=0).stop 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 msgr2=0x7fe3e015bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 0x7fe3e015bfe0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe3c800c430 tx=0x7fe3c800c8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 shutdown_connections 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe3b806b8f0 0x7fe3b806ddb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe3e015d410 0x7fe3e0159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe3e00d0dc0 0x7fe3e015bfe0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 --2- 172.21.15.53:0/2823018417 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe3e00cf1d0 0x7fe3e0153290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.880+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 >> 172.21.15.53:0/2823018417 conn(0x7fe3e0099a60 msgr2=0x7fe3e00cb710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.881+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 shutdown_connections 2024-01-16T15:06:26.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:26.881+0000 7fe3df200640 1 -- 172.21.15.53:0/2823018417 wait complete. 2024-01-16T15:06:26.861 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.884+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.861 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.884+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.861 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.884+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.861 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.884+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.861 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.884+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.878 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30232 2024-01-16T15:06:26.885 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.907+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.885 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.907+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.885 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.907+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.885 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.907+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:26.885 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:26.907+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:27.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.131+0000 7f7003e00640 1 Processor -- start 2024-01-16T15:06:27.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.132+0000 7f7003e00640 1 -- start start 2024-01-16T15:06:27.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.133+0000 7f7003e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70040d0100 0x7f70040b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.133+0000 7f7003e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f70040d0f40 0x7f70040d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.133+0000 7f7003e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70040d0a10 con 0x7f7004160920 2024-01-16T15:06:27.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.133+0000 7f7003e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70040cf080 con 0x7f70040d0f40 2024-01-16T15:06:27.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.133+0000 7f7003e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c6be0 con 0x7f70040d0100 2024-01-16T15:06:27.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7002a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7004160920 0x7f70040cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42658/0 (socket says 172.21.15.53:42658) 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7002a00640 1 -- 172.21.15.53:0/3764577958 learned_addr learned my addr 172.21.15.53:0/3764577958 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7003400640 1 --2- 172.21.15.53:0/3764577958 >> v2:172.21.15.53:3300/0 conn(0x7f70040d0f40 0x7f70040d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7002000640 1 --2- 172.21.15.53:0/3764577958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70040d0100 0x7f70040b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7001600640 1 -- 172.21.15.53:0/3764577958 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1873343999 0 0) 0x7f70040d0a10 con 0x7f7004160920 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7001600640 1 -- 172.21.15.53:0/3764577958 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4003520 con 0x7f7004160920 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7003400640 1 -- 172.21.15.53:0/3764577958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70040d0100 msgr2=0x7f70040b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7003400640 1 --2- 172.21.15.53:0/3764577958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70040d0100 0x7f70040b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7003400640 1 -- 172.21.15.53:0/3764577958 >> v1:172.21.15.53:6789/0 conn(0x7f7004160920 legacy=0x7f70040cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.134+0000 7f7003400640 1 -- 172.21.15.53:0/3764577958 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70040ce7e0 con 0x7f70040d0f40 2024-01-16T15:06:27.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.136+0000 7f7003400640 1 --2- 172.21.15.53:0/3764577958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0f40 0x7f70040d13f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0097a0 tx=0x7f6ffc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8eebb259ee4015 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.136+0000 7f7001600640 1 -- 172.21.15.53:0/3764577958 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc02d040 con 0x7f70040d0f40 2024-01-16T15:06:27.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.136+0000 7f7001600640 1 -- 172.21.15.53:0/3764577958 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc031810 con 0x7f70040d0f40 2024-01-16T15:06:27.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.136+0000 7f7001600640 1 -- 172.21.15.53:0/3764577958 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc031ba0 con 0x7f70040d0f40 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.136+0000 7f7003e00640 1 -- 172.21.15.53:0/3764577958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0f40 msgr2=0x7f70040d13f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.136+0000 7f7003e00640 1 --2- 172.21.15.53:0/3764577958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0f40 0x7f70040d13f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f6ffc0097a0 tx=0x7f6ffc02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.137+0000 7f7003e00640 1 -- 172.21.15.53:0/3764577958 shutdown_connections 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.137+0000 7f7003e00640 1 --2- 172.21.15.53:0/3764577958 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0f40 0x7f70040d13f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.137+0000 7f7003e00640 1 --2- 172.21.15.53:0/3764577958 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70040d0100 0x7f70040b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.137+0000 7f7003e00640 1 -- 172.21.15.53:0/3764577958 >> 172.21.15.53:0/3764577958 conn(0x7f70040d3090 msgr2=0x7f70040d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.137+0000 7f7003e00640 1 -- 172.21.15.53:0/3764577958 shutdown_connections 2024-01-16T15:06:27.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.137+0000 7f7003e00640 1 -- 172.21.15.53:0/3764577958 wait complete. 2024-01-16T15:06:27.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.138+0000 7f7003e00640 1 Processor -- start 2024-01-16T15:06:27.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.139+0000 7f7003e00640 1 -- start start 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.139+0000 7f7003e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 0x7f700415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7003e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70040d0f40 0x7f700406c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7003e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7004160920 0x7f700406c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7003e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70041574c0 con 0x7f70040d0100 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7003e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70041568b0 con 0x7f70040d0f40 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7003e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7004156b90 con 0x7f7004160920 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 0x7f700415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 0x7f700415faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49786/0 (socket says 172.21.15.53:49786) 2024-01-16T15:06:27.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 -- 172.21.15.53:0/3422386733 learned_addr learned my addr 172.21.15.53:0/3422386733 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:27.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 -- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7004160920 msgr2=0x7f700406c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7003400640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7004160920 0x7f700406c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7004160920 0x7f700406c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 -- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70040d0f40 msgr2=0x7f700406c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002000640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70040d0f40 0x7f700406c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70040d0f40 0x7f700406c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.140+0000 7f7002a00640 1 -- 172.21.15.53:0/3422386733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc009400 con 0x7f70040d0100 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.141+0000 7f7002a00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 0x7f700415faf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400cbc0 tx=0x7f6ff400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.141+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400ad40 con 0x7f70040d0100 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.141+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4012400 con 0x7f70040d0100 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.141+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff400aea0 con 0x7f70040d0100 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.141+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7004156e70 con 0x7f70040d0100 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.141+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7004157950 con 0x7f70040d0100 2024-01-16T15:06:27.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.142+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7004067450 con 0x7f70040d0100 2024-01-16T15:06:27.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.144+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f6ff4007070 con 0x7f70040d0100 2024-01-16T15:06:27.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.144+0000 7f6ffaa00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6fdc06b920 0x7f6fdc06dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.145+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6ff408a6a0 con 0x7f70040d0100 2024-01-16T15:06:27.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.145+0000 7f7002000640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6fdc06b920 0x7f6fdc06dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.146+0000 7f7002000640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6fdc06b920 0x7f6fdc06dde0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6fec006fd0 tx=0x7f6fec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.149+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ff4054cb0 con 0x7f70040d0100 2024-01-16T15:06:27.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.379+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30232", "target": ["mon-mgr", ""]}) v1 -- 0x7f7004160d00 con 0x7f6fdc06b920 2024-01-16T15:06:27.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.546+0000 7f6ffaa00640 1 -- 172.21.15.53:0/3422386733 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fec009480 con 0x7f6fdc06b920 2024-01-16T15:06:27.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6fdc06b920 msgr2=0x7f6fdc06dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6fdc06b920 0x7f6fdc06dde0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6fec006fd0 tx=0x7f6fec008040 comp rx=0 tx=0).stop 2024-01-16T15:06:27.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 msgr2=0x7f700415faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 0x7f700415faf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400cbc0 tx=0x7f6ff400a430 comp rx=0 tx=0).stop 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 shutdown_connections 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6fdc06b920 0x7f6fdc06dde0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7004160920 0x7f700406c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70040d0f40 0x7f700406c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 --2- 172.21.15.53:0/3422386733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70040d0100 0x7f700415faf0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 >> 172.21.15.53:0/3422386733 conn(0x7f70040d3090 msgr2=0x7f70040c94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 shutdown_connections 2024-01-16T15:06:27.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.549+0000 7f7003e00640 1 -- 172.21.15.53:0/3422386733 wait complete. 2024-01-16T15:06:27.550 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30233 2024-01-16T15:06:27.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.806+0000 7f33ea000640 1 Processor -- start 2024-01-16T15:06:27.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.806+0000 7f33ea000640 1 -- start start 2024-01-16T15:06:27.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33ea000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33e4160920 0x7f33e40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33ea000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cfea0 0x7f33e4161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33ea000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33e40d0a10 con 0x7f33e40cf9f0 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33ea000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33e40d0330 con 0x7f33e4160920 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33ea000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40d1f50 con 0x7f33e40cfea0 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33e8c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33e4160920 0x7f33e40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33e8c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33e4160920 0x7f33e40b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49788/0 (socket says 172.21.15.53:49788) 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.808+0000 7f33e8c00640 1 -- 172.21.15.53:0/1703965306 learned_addr learned my addr 172.21.15.53:0/1703965306 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.809+0000 7f33e9600640 1 --2- 172.21.15.53:0/1703965306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cfea0 0x7f33e4161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.809+0000 7f33e8c00640 1 -- 172.21.15.53:0/1703965306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cfea0 msgr2=0x7f33e4161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.809+0000 7f33e8c00640 1 --2- 172.21.15.53:0/1703965306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cfea0 0x7f33e4161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.809+0000 7f33e8c00640 1 -- 172.21.15.53:0/1703965306 >> v1:172.21.15.146:6789/0 conn(0x7f33e40cf9f0 legacy=0x7f33e4161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:27.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.809+0000 7f33e8c00640 1 -- 172.21.15.53:0/1703965306 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e40cf080 con 0x7f33e4160920 2024-01-16T15:06:27.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.810+0000 7f33e8c00640 1 --2- 172.21.15.53:0/1703965306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40b3f30 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009870 tx=0x7f33d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17de0da5872bfc79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.810+0000 7f33e3400640 1 -- 172.21.15.53:0/1703965306 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4031640 con 0x7f33e4160920 2024-01-16T15:06:27.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.810+0000 7f33e3400640 1 -- 172.21.15.53:0/1703965306 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33d40317a0 con 0x7f33e4160920 2024-01-16T15:06:27.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.810+0000 7f33e3400640 1 -- 172.21.15.53:0/1703965306 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33d4031b30 con 0x7f33e4160920 2024-01-16T15:06:27.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 -- 172.21.15.53:0/1703965306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 msgr2=0x7f33e40b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 --2- 172.21.15.53:0/1703965306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40b3f30 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7f33d4009870 tx=0x7f33d402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 -- 172.21.15.53:0/1703965306 shutdown_connections 2024-01-16T15:06:27.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 --2- 172.21.15.53:0/1703965306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cfea0 0x7f33e4161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 --2- 172.21.15.53:0/1703965306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40b3f30 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 -- 172.21.15.53:0/1703965306 >> 172.21.15.53:0/1703965306 conn(0x7f33e40d3090 msgr2=0x7f33e40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 -- 172.21.15.53:0/1703965306 shutdown_connections 2024-01-16T15:06:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.811+0000 7f33ea000640 1 -- 172.21.15.53:0/1703965306 wait complete. 2024-01-16T15:06:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.813+0000 7f33ea000640 1 Processor -- start 2024-01-16T15:06:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.813+0000 7f33ea000640 1 -- start start 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33ea000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cf9f0 0x7f33e40b7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33ea000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33e40cfea0 0x7f33e40b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33ea000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40c8d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33ea000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40cbd90 con 0x7f33e4160920 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33ea000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40cc910 con 0x7f33e40cfea0 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33ea000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33e40cbfe0 con 0x7f33e40cf9f0 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.814+0000 7f33e9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40c8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40c8d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49792/0 (socket says 172.21.15.53:49792) 2024-01-16T15:06:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 -- 172.21.15.53:0/3254847455 learned_addr learned my addr 172.21.15.53:0/3254847455 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e3e00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33e40cfea0 0x7f33e40b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e8c00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cf9f0 0x7f33e40b7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 -- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cf9f0 msgr2=0x7f33e40b7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cf9f0 0x7f33e40b7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 -- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33e40cfea0 msgr2=0x7f33e40b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33e40cfea0 0x7f33e40b6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 -- 172.21.15.53:0/3254847455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d40094d0 con 0x7f33e4160920 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e3e00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33e40cfea0 0x7f33e40b6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e8c00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cf9f0 0x7f33e40b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.815+0000 7f33e9600640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40c8d00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f33dc007e70 tx=0x7f33dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.816+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc014620 con 0x7f33e4160920 2024-01-16T15:06:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.816+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc0147a0 con 0x7f33e4160920 2024-01-16T15:06:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.816+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc014b50 con 0x7f33e4160920 2024-01-16T15:06:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.816+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33e40cc2c0 con 0x7f33e4160920 2024-01-16T15:06:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.816+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33e40b6ba0 con 0x7f33e4160920 2024-01-16T15:06:27.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.817+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33e40ccaa0 con 0x7f33e4160920 2024-01-16T15:06:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.818+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 16) v1 ==== 90100+0+0 (secure 0 0 0) 0x7f33dc014cb0 con 0x7f33e4160920 2024-01-16T15:06:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.819+0000 7f33e0c00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33bc06b8f0 0x7f33bc06ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.819+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f33dc08b0b0 con 0x7f33e4160920 2024-01-16T15:06:27.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.820+0000 7f33e8c00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33bc06b8f0 0x7f33bc06ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.821+0000 7f33e8c00640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33bc06b8f0 0x7f33bc06ddb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f33d4036000 tx=0x7f33d403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:27.824+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33dc0555e0 con 0x7f33e4160920 2024-01-16T15:06:28.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:28.054+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30233", "target": ["mon-mgr", ""]}) v1 -- 0x7f33e4066640 con 0x7f33bc06b8f0 2024-01-16T15:06:29.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:29.177+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f33dc0559b0 con 0x7f33e4160920 2024-01-16T15:06:30.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.050+0000 7f33e0c00640 1 -- 172.21.15.53:0/3254847455 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f33e4066640 con 0x7f33bc06b8f0 2024-01-16T15:06:30.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.053+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33bc06b8f0 msgr2=0x7f33bc06ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.053+0000 7f33ea000640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33bc06b8f0 0x7f33bc06ddb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f33d4036000 tx=0x7f33d403c040 comp rx=0 tx=0).stop 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.053+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 msgr2=0x7f33e40c8d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.053+0000 7f33ea000640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40c8d00 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f33dc007e70 tx=0x7f33dc00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.053+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 shutdown_connections 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.053+0000 7f33ea000640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f33bc06b8f0 0x7f33bc06ddb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.054+0000 7f33ea000640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33e4160920 0x7f33e40c8d00 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.054+0000 7f33ea000640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33e40cfea0 0x7f33e40b6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.054+0000 7f33ea000640 1 --2- 172.21.15.53:0/3254847455 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33e40cf9f0 0x7f33e40b7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.054+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 >> 172.21.15.53:0/3254847455 conn(0x7f33e40d3090 msgr2=0x7f33e406c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.054+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 shutdown_connections 2024-01-16T15:06:30.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.054+0000 7f33ea000640 1 -- 172.21.15.53:0/3254847455 wait complete. 2024-01-16T15:06:30.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30234 2024-01-16T15:06:30.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.307+0000 7f4ede000640 1 Processor -- start 2024-01-16T15:06:30.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.307+0000 7f4ede000640 1 -- start start 2024-01-16T15:06:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.309+0000 7f4ede000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 0x7f4ed815a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.309+0000 7f4ede000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 0x7f4ed815db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.309+0000 7f4ede000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ed80ac630 con 0x7f4ed80cf030 2024-01-16T15:06:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.309+0000 7f4ede000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed80adab0 con 0x7f4ed80ad630 2024-01-16T15:06:30.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.309+0000 7f4ede000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80ac890 con 0x7f4ed815ad70 2024-01-16T15:06:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.310+0000 7f4edd600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 0x7f4ed815db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.310+0000 7f4ecfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 0x7f4ed815a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.310+0000 7f4ecfe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 0x7f4ed815a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:52434/0 (socket says 172.21.15.53:52434) 2024-01-16T15:06:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.310+0000 7f4edd600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 0x7f4ed815db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:60130/0 (socket says 172.21.15.53:60130) 2024-01-16T15:06:30.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.310+0000 7f4ecfe00640 1 -- 172.21.15.53:0/1476592704 learned_addr learned my addr 172.21.15.53:0/1476592704 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:30.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.311+0000 7f4edd600640 1 -- 172.21.15.53:0/1476592704 >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 msgr2=0x7f4ed815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:30.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.311+0000 7f4edd600640 1 --2- 172.21.15.53:0/1476592704 >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 0x7f4ed815a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.311+0000 7f4edd600640 1 -- 172.21.15.53:0/1476592704 >> v1:172.21.15.146:6789/0 conn(0x7f4ed80cf030 legacy=0x7f4ed80cf410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.311+0000 7f4edd600640 1 -- 172.21.15.53:0/1476592704 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed80acb50 con 0x7f4ed815ad70 2024-01-16T15:06:30.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.311+0000 7f4ecfe00640 1 --2- 172.21.15.53:0/1476592704 >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 0x7f4ed815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:30.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.312+0000 7f4edd600640 1 --2- 172.21.15.53:0/1476592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 0x7f4ed815db30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009d90 tx=0x7f4ed002e0c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65fcb86d4d6ec75d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.313+0000 7f4ed7e00640 1 -- 172.21.15.53:0/1476592704 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed002ee70 con 0x7f4ed815ad70 2024-01-16T15:06:30.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.313+0000 7f4ed7e00640 1 -- 172.21.15.53:0/1476592704 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0036440 con 0x7f4ed815ad70 2024-01-16T15:06:30.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.313+0000 7f4ed7e00640 1 -- 172.21.15.53:0/1476592704 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00367d0 con 0x7f4ed815ad70 2024-01-16T15:06:30.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 -- 172.21.15.53:0/1476592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 msgr2=0x7f4ed815db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 --2- 172.21.15.53:0/1476592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 0x7f4ed815db30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009d90 tx=0x7f4ed002e0c0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 -- 172.21.15.53:0/1476592704 shutdown_connections 2024-01-16T15:06:30.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 --2- 172.21.15.53:0/1476592704 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed815ad70 0x7f4ed815db30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 --2- 172.21.15.53:0/1476592704 >> v2:172.21.15.146:3300/0 conn(0x7f4ed80ad630 0x7f4ed815a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 -- 172.21.15.53:0/1476592704 >> 172.21.15.53:0/1476592704 conn(0x7f4ed806a800 msgr2=0x7f4ed806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:30.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 -- 172.21.15.53:0/1476592704 shutdown_connections 2024-01-16T15:06:30.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.314+0000 7f4ede000640 1 -- 172.21.15.53:0/1476592704 wait complete. 2024-01-16T15:06:30.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.316+0000 7f4ede000640 1 Processor -- start 2024-01-16T15:06:30.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.316+0000 7f4ede000640 1 -- start start 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4ede000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed80ad630 0x7f4ed8152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4ede000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed80cf030 0x7f4ed8153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4ede000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 0x7f4ed81423a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4ede000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8152d40 con 0x7f4ed815ad70 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4ede000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed8151db0 con 0x7f4ed80cf030 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4ede000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed809ad50 con 0x7f4ed80ad630 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.317+0000 7f4edd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 0x7f4ed81423a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 0x7f4ed81423a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49796/0 (socket says 172.21.15.53:49796) 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 -- 172.21.15.53:0/693902221 learned_addr learned my addr 172.21.15.53:0/693902221 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4ecfe00640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed80cf030 0x7f4ed8153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edcc00640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed80ad630 0x7f4ed8152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 -- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed80ad630 msgr2=0x7f4ed8152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed80ad630 0x7f4ed8152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 -- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed80cf030 msgr2=0x7f4ed8153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:30.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed80cf030 0x7f4ed8153ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:30.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.318+0000 7f4edd600640 1 -- 172.21.15.53:0/693902221 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ec00094d0 con 0x7f4ed815ad70 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.319+0000 7f4edd600640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 0x7f4ed81423a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009eb0 tx=0x7f4ed0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.319+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed00408e0 con 0x7f4ed815ad70 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.319+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed0040a40 con 0x7f4ed815ad70 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.319+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed0040df0 con 0x7f4ed815ad70 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.319+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed0009a70 con 0x7f4ed815ad70 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.319+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed8152000 con 0x7f4ed815ad70 2024-01-16T15:06:30.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.320+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed8067450 con 0x7f4ed815ad70 2024-01-16T15:06:30.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.321+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f4ed003d050 con 0x7f4ed815ad70 2024-01-16T15:06:30.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.322+0000 7f4ed5600640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4eb006ba10 0x7f4eb006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:30.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.322+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ed00ad2f0 con 0x7f4ed815ad70 2024-01-16T15:06:30.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.323+0000 7f4edcc00640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4eb006ba10 0x7f4eb006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:30.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.324+0000 7f4edcc00640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4eb006ba10 0x7f4eb006ded0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8001640 tx=0x7f4ec8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:30.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.327+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ed00ad000 con 0x7f4ed815ad70 2024-01-16T15:06:30.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:30.558+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30234", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ed815b200 con 0x7f4eb006ba10 2024-01-16T15:06:35.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.050+0000 7f4ed5600640 1 -- 172.21.15.53:0/693902221 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4ec80012c0 con 0x7f4eb006ba10 2024-01-16T15:06:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.053+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4eb006ba10 msgr2=0x7f4eb006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.053+0000 7f4ede000640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4eb006ba10 0x7f4eb006ded0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4ec8001640 tx=0x7f4ec8009040 comp rx=0 tx=0).stop 2024-01-16T15:06:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.053+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 msgr2=0x7f4ed81423a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.053+0000 7f4ede000640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 0x7f4ed81423a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f4ed0009eb0 tx=0x7f4ed0004a40 comp rx=0 tx=0).stop 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 shutdown_connections 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4eb006ba10 0x7f4eb006ded0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ed815ad70 0x7f4ed81423a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ed80cf030 0x7f4ed8153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 --2- 172.21.15.53:0/693902221 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ed80ad630 0x7f4ed8152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 >> 172.21.15.53:0/693902221 conn(0x7f4ed806a800 msgr2=0x7f4ed806cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 shutdown_connections 2024-01-16T15:06:35.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.054+0000 7f4ede000640 1 -- 172.21.15.53:0/693902221 wait complete. 2024-01-16T15:06:35.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30235 2024-01-16T15:06:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.310+0000 7f2006400640 1 Processor -- start 2024-01-16T15:06:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.311+0000 7f2006400640 1 -- start start 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.312+0000 7f2006400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20000d0100 0x7f20000b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.312+0000 7f2006400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000d0a10 con 0x7f2000160920 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.312+0000 7f2006400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000cf080 con 0x7f20000d0f40 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.312+0000 7f2006400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20000c6be0 con 0x7f20000d0100 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.312+0000 7f2005000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2000160920 0x7f20000cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56866/0 (socket says 172.21.15.53:56866) 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.312+0000 7f2005000640 1 -- 172.21.15.53:0/541906476 learned_addr learned my addr 172.21.15.53:0/541906476 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fffe00640 1 --2- 172.21.15.53:0/541906476 >> v2:172.21.15.146:3300/0 conn(0x7f20000d0100 0x7f20000b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548496134 0 0) 0x7f20000d0a10 con 0x7f2000160920 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe0003520 con 0x7f2000160920 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3244261972 0 0) 0x7f1fe0003520 con 0x7f2000160920 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 >> v2:172.21.15.146:3300/0 conn(0x7f20000d0100 msgr2=0x7f20000b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 --2- 172.21.15.53:0/541906476 >> v2:172.21.15.146:3300/0 conn(0x7f20000d0100 0x7f20000b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 >> v1:172.21.15.146:6789/0 conn(0x7f20000d0f40 legacy=0x7f20000d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000ce7e0 con 0x7f2000160920 2024-01-16T15:06:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1ff0001670 con 0x7f2000160920 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 >> v1:172.21.15.53:6789/0 conn(0x7f2000160920 legacy=0x7f20000cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe0003970 0x7f1fe0003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00047c0 0x7f1fe0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.313+0000 7f1fff400640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0005180 0x7f1fe000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20000c6be0 con 0x7f1fe0003970 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe000c0b0 con 0x7f1fe0005180 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe000c270 con 0x7f1fe00047c0 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005000640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe0003970 0x7f1fe0003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005a00640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0005180 0x7f1fe000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005000640 1 -- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00047c0 msgr2=0x7f1fe0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005000640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe00047c0 0x7f1fe0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005000640 1 -- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0005180 msgr2=0x7f1fe000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:35.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005000640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0005180 0x7f1fe000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005000640 1 -- 172.21.15.53:0/541906476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0003520 con 0x7f1fe0003970 2024-01-16T15:06:35.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005a00640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1fe0005180 0x7f1fe000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:35.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.314+0000 7f2005a00640 1 -- 172.21.15.53:0/541906476 reap_dead start 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.315+0000 7f2005000640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe0003970 0x7f1fe0003d70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000c210 tx=0x7f1fe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761ed9676e2c8fd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.316+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff0005ad0 con 0x7f1fe0003970 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.316+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff0005c30 con 0x7f1fe0003970 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.316+0000 7f1fff400640 1 -- 172.21.15.53:0/541906476 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff002c890 con 0x7f1fe0003970 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.316+0000 7f2006400640 1 -- 172.21.15.53:0/541906476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe0003970 msgr2=0x7f1fe0003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.316+0000 7f2006400640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe0003970 0x7f1fe0003d70 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f1ff000c210 tx=0x7f1fe00057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.317+0000 7f2006400640 1 -- 172.21.15.53:0/541906476 shutdown_connections 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.317+0000 7f2006400640 1 --2- 172.21.15.53:0/541906476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1fe0003970 0x7f1fe0003d70 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.317+0000 7f2006400640 1 -- 172.21.15.53:0/541906476 >> 172.21.15.53:0/541906476 conn(0x7f20000d3090 msgr2=0x7f20000d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.317+0000 7f2006400640 1 -- 172.21.15.53:0/541906476 shutdown_connections 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.317+0000 7f2006400640 1 -- 172.21.15.53:0/541906476 wait complete. 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.318+0000 7f2006400640 1 Processor -- start 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.319+0000 7f2006400640 1 -- start start 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2006400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe0003970 0x7f20000b7450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2006400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2000157860 0x7f2000157ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2006400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 0x7f2000156400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2006400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2000158440 con 0x7f2000155fc0 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2006400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20001f7200 con 0x7f2000157860 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2006400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2000158220 con 0x7f1fe0003970 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 0x7f2000156400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 0x7f2000156400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46654/0 (socket says 172.21.15.53:46654) 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 -- 172.21.15.53:0/1378576532 learned_addr learned my addr 172.21.15.53:0/1378576532 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005000640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe0003970 0x7f20000b7450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f1fffe00640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2000157860 0x7f2000157ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 -- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe0003970 msgr2=0x7f20000b7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe0003970 0x7f20000b7450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 -- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2000157860 msgr2=0x7f2000157ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.320+0000 7f2005a00640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2000157860 0x7f2000157ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f2005a00640 1 -- 172.21.15.53:0/1378576532 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2000159d00 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f1fffe00640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2000157860 0x7f2000157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f2005a00640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 0x7f2000156400 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f200008c2c0 tx=0x7f1ff800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8006d50 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff8004960 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff8004d10 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20001f6440 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.321+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20001f6a00 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.322+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2000067450 con 0x7f2000155fc0 2024-01-16T15:06:35.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.323+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f1ff8005030 con 0x7f2000155fc0 2024-01-16T15:06:35.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.324+0000 7f1ffcc00640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd806ba40 0x7f1fd806df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:35.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.324+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ff808a020 con 0x7f2000155fc0 2024-01-16T15:06:35.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.325+0000 7f2005000640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd806ba40 0x7f1fd806df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:35.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.326+0000 7f2005000640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd806ba40 0x7f1fd806df00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0034000 tx=0x7f1ff0009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:35.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.329+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ff8087460 con 0x7f2000155fc0 2024-01-16T15:06:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:35.560+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7f2000160d00 con 0x7f1fd806ba40 2024-01-16T15:06:40.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.040+0000 7f1ffcc00640 1 -- 172.21.15.53:0/1378576532 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1ff003d040 con 0x7f1fd806ba40 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.043+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd806ba40 msgr2=0x7f1fd806df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.043+0000 7f2006400640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd806ba40 0x7f1fd806df00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0034000 tx=0x7f1ff0009030 comp rx=0 tx=0).stop 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.043+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 msgr2=0x7f2000156400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.043+0000 7f2006400640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 0x7f2000156400 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f200008c2c0 tx=0x7f1ff800d9d0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 shutdown_connections 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fd806ba40 0x7f1fd806df00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2000155fc0 0x7f2000156400 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2000157860 0x7f2000157ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 --2- 172.21.15.53:0/1378576532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1fe0003970 0x7f20000b7450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 >> 172.21.15.53:0/1378576532 conn(0x7f20000d3090 msgr2=0x7f20000c6d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 shutdown_connections 2024-01-16T15:06:40.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.044+0000 7f2006400640 1 -- 172.21.15.53:0/1378576532 wait complete. 2024-01-16T15:06:40.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30236 2024-01-16T15:06:40.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.304+0000 7f4ad9400640 1 Processor -- start 2024-01-16T15:06:40.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.305+0000 7f4ad9400640 1 -- start start 2024-01-16T15:06:40.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.306+0000 7f4ad9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad40b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.306+0000 7f4ad9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 0x7f4ad406c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.306+0000 7f4ad9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40702e0 con 0x7f4ad4162810 2024-01-16T15:06:40.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.306+0000 7f4ad9400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad40705c0 con 0x7f4ad40b3650 2024-01-16T15:06:40.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.306+0000 7f4ad9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40708a0 con 0x7f4ad4162430 2024-01-16T15:06:40.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.307+0000 7f4ad8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 0x7f4ad406c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.307+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad40b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.307+0000 7f4ad8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 0x7f4ad406c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:49698/0 (socket says 172.21.15.53:49698) 2024-01-16T15:06:40.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.307+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad40b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:42836/0 (socket says 172.21.15.53:42836) 2024-01-16T15:06:40.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.307+0000 7f4ad8a00640 1 -- 172.21.15.53:0/1148653702 learned_addr learned my addr 172.21.15.53:0/1148653702 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:40.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.307+0000 7f4acbe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f4ad4162810 0x7f4ad4071240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:50390/0 (socket says 172.21.15.53:50390) 2024-01-16T15:06:40.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.308+0000 7f4ad3e00640 1 -- 172.21.15.53:0/1148653702 >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 msgr2=0x7f4ad406c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.308+0000 7f4ad3e00640 1 --2- 172.21.15.53:0/1148653702 >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 0x7f4ad406c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.308+0000 7f4ad3e00640 1 -- 172.21.15.53:0/1148653702 >> v1:172.21.15.146:6789/0 conn(0x7f4ad4162810 legacy=0x7f4ad4071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.308+0000 7f4ad3e00640 1 -- 172.21.15.53:0/1148653702 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4070b80 con 0x7f4ad4162430 2024-01-16T15:06:40.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.308+0000 7f4ad8a00640 1 --2- 172.21.15.53:0/1148653702 >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 0x7f4ad406c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:40.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.308+0000 7f4ad3400640 1 -- 172.21.15.53:0/1148653702 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406922737 0 0) 0x7f4ad40702e0 con 0x7f4ad4162810 2024-01-16T15:06:40.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.310+0000 7f4ad3e00640 1 --2- 172.21.15.53:0/1148653702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad40b3110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40097a0 tx=0x7f4ac402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3006e8c76b49eea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.310+0000 7f4ad3400640 1 -- 172.21.15.53:0/1148653702 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac402d040 con 0x7f4ad4162430 2024-01-16T15:06:40.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.310+0000 7f4ad3400640 1 -- 172.21.15.53:0/1148653702 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ac4031810 con 0x7f4ad4162430 2024-01-16T15:06:40.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.310+0000 7f4ad3400640 1 -- 172.21.15.53:0/1148653702 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4031ba0 con 0x7f4ad4162430 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 -- 172.21.15.53:0/1148653702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 msgr2=0x7f4ad40b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1148653702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad40b3110 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4ac40097a0 tx=0x7f4ac402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 -- 172.21.15.53:0/1148653702 shutdown_connections 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1148653702 >> v2:172.21.15.146:3300/0 conn(0x7f4ad40b3650 0x7f4ad406c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1148653702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad40b3110 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 -- 172.21.15.53:0/1148653702 >> 172.21.15.53:0/1148653702 conn(0x7f4ad406a800 msgr2=0x7f4ad406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 -- 172.21.15.53:0/1148653702 shutdown_connections 2024-01-16T15:06:40.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.311+0000 7f4ad9400640 1 -- 172.21.15.53:0/1148653702 wait complete. 2024-01-16T15:06:40.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.313+0000 7f4ad9400640 1 Processor -- start 2024-01-16T15:06:40.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.313+0000 7f4ad9400640 1 -- start start 2024-01-16T15:06:40.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.314+0000 7f4ad9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40b3650 0x7f4ad415b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.314+0000 7f4ad9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad41463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.314+0000 7f4ad9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 0x7f4ad4144e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.314+0000 7f4ad9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad415ba40 con 0x7f4ad4159370 2024-01-16T15:06:40.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.314+0000 7f4ad9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad41449b0 con 0x7f4ad40b3650 2024-01-16T15:06:40.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.314+0000 7f4ad9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4146930 con 0x7f4ad4162430 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 0x7f4ad4144e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40b3650 0x7f4ad415b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4acbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad41463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 0x7f4ad4144e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46660/0 (socket says 172.21.15.53:46660) 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 -- 172.21.15.53:0/1744188525 learned_addr learned my addr 172.21.15.53:0/1744188525 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 -- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 msgr2=0x7f4ad41463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad41463f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 -- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40b3650 msgr2=0x7f4ad415b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40b3650 0x7f4ad415b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4ad8a00640 1 -- 172.21.15.53:0/1744188525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4acc009b30 con 0x7f4ad4159370 2024-01-16T15:06:40.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.315+0000 7f4acbe00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad41463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.316+0000 7f4ad8a00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 0x7f4ad4144e40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4acc00efc0 tx=0x7f4acc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.316+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc018070 con 0x7f4ad4159370 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.316+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4acc00f040 con 0x7f4ad4159370 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.316+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4acc00ce00 con 0x7f4ad4159370 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.316+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac4009400 con 0x7f4ad4159370 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.316+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad4144080 con 0x7f4ad4159370 2024-01-16T15:06:40.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.317+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad4144b40 con 0x7f4ad4159370 2024-01-16T15:06:40.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.318+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f4acc014480 con 0x7f4ad4159370 2024-01-16T15:06:40.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.319+0000 7f4ad0c00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aac06ba10 0x7f4aac06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.319+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4acc08acb0 con 0x7f4ad4159370 2024-01-16T15:06:40.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.320+0000 7f4ad3e00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aac06ba10 0x7f4aac06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.321+0000 7f4ad3e00640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aac06ba10 0x7f4aac06ded0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4036000 tx=0x7f4ac403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.324+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4acc088500 con 0x7f4ad4159370 2024-01-16T15:06:40.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.556+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30236", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ad4067450 con 0x7f4aac06ba10 2024-01-16T15:06:40.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.719+0000 7f4ad0c00640 1 -- 172.21.15.53:0/1744188525 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4ad4067450 con 0x7f4aac06ba10 2024-01-16T15:06:40.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.721+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aac06ba10 msgr2=0x7f4aac06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.721+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aac06ba10 0x7f4aac06ded0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f4ac4036000 tx=0x7f4ac403c040 comp rx=0 tx=0).stop 2024-01-16T15:06:40.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 msgr2=0x7f4ad4144e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 0x7f4ad4144e40 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f4acc00efc0 tx=0x7f4acc00c490 comp rx=0 tx=0).stop 2024-01-16T15:06:40.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 shutdown_connections 2024-01-16T15:06:40.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aac06ba10 0x7f4aac06ded0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4159370 0x7f4ad4144e40 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad4162430 0x7f4ad41463f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 --2- 172.21.15.53:0/1744188525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40b3650 0x7f4ad415b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 >> 172.21.15.53:0/1744188525 conn(0x7f4ad406a800 msgr2=0x7f4ad406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 shutdown_connections 2024-01-16T15:06:40.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.722+0000 7f4ad9400640 1 -- 172.21.15.53:0/1744188525 wait complete. 2024-01-16T15:06:40.720 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone30237 2024-01-16T15:06:40.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.968+0000 7fe08d000640 1 Processor -- start 2024-01-16T15:06:40.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.968+0000 7fe08d000640 1 -- start start 2024-01-16T15:06:40.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.970+0000 7fe08d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0880cc940 0x7fe0880cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.970+0000 7fe08d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe08806a3e0 0x7fe08806a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.970+0000 7fe08d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0880cfcc0 con 0x7fe088144c60 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.970+0000 7fe08d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08806adb0 con 0x7fe0880cc940 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.970+0000 7fe08d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe088071a20 con 0x7fe08806a3e0 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0880cc940 0x7fe0880cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0880cc940 0x7fe0880cf780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46666/0 (socket says 172.21.15.53:46666) 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 -- 172.21.15.53:0/1735073292 learned_addr learned my addr 172.21.15.53:0/1735073292 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe086800640 1 --2- 172.21.15.53:0/1735073292 >> v2:172.21.15.146:3300/0 conn(0x7fe08806a3e0 0x7fe08806a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 -- 172.21.15.53:0/1735073292 >> v2:172.21.15.146:3300/0 conn(0x7fe08806a3e0 msgr2=0x7fe08806a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:40.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 --2- 172.21.15.53:0/1735073292 >> v2:172.21.15.146:3300/0 conn(0x7fe08806a3e0 0x7fe08806a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 -- 172.21.15.53:0/1735073292 >> v1:172.21.15.146:6789/0 conn(0x7fe088144c60 legacy=0x7fe08806e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.971+0000 7fe07fe00640 1 -- 172.21.15.53:0/1735073292 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0880ccd80 con 0x7fe0880cc940 2024-01-16T15:06:40.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.973+0000 7fe07fe00640 1 --2- 172.21.15.53:0/1735073292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0880cc940 0x7fe0880cf780 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe08806d040 tx=0x7fe07002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c71290dd8b829537 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.973+0000 7fe085400640 1 -- 172.21.15.53:0/1735073292 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070030860 con 0x7fe0880cc940 2024-01-16T15:06:40.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.973+0000 7fe085400640 1 -- 172.21.15.53:0/1735073292 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0700309c0 con 0x7fe0880cc940 2024-01-16T15:06:40.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.973+0000 7fe085400640 1 -- 172.21.15.53:0/1735073292 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070030d50 con 0x7fe0880cc940 2024-01-16T15:06:40.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 -- 172.21.15.53:0/1735073292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0880cc940 msgr2=0x7fe0880cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 --2- 172.21.15.53:0/1735073292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0880cc940 0x7fe0880cf780 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe08806d040 tx=0x7fe07002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 -- 172.21.15.53:0/1735073292 shutdown_connections 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 --2- 172.21.15.53:0/1735073292 >> v2:172.21.15.146:3300/0 conn(0x7fe08806a3e0 0x7fe08806a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 --2- 172.21.15.53:0/1735073292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0880cc940 0x7fe0880cf780 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 -- 172.21.15.53:0/1735073292 >> 172.21.15.53:0/1735073292 conn(0x7fe0880cfef0 msgr2=0x7fe0880d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 -- 172.21.15.53:0/1735073292 shutdown_connections 2024-01-16T15:06:40.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.974+0000 7fe08d000640 1 -- 172.21.15.53:0/1735073292 wait complete. 2024-01-16T15:06:40.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.976+0000 7fe08d000640 1 Processor -- start 2024-01-16T15:06:40.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.976+0000 7fe08d000640 1 -- start start 2024-01-16T15:06:40.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe08d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe08806a3e0 0x7fe08813fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe08d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0880cc940 0x7fe0881309b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe08d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 0x7fe088130ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe08d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08813dd90 con 0x7fe088144c60 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe08d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0881301f0 con 0x7fe08806a3e0 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe08d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe08812f5e0 con 0x7fe0880cc940 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe086800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 0x7fe088130ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe086800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 0x7fe088130ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46682/0 (socket says 172.21.15.53:46682) 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe086800640 1 -- 172.21.15.53:0/71315931 learned_addr learned my addr 172.21.15.53:0/71315931 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.977+0000 7fe085e00640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe08806a3e0 0x7fe08813fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe07fe00640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0880cc940 0x7fe0881309b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe086800640 1 -- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0880cc940 msgr2=0x7fe0881309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe086800640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0880cc940 0x7fe0881309b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe086800640 1 -- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe08806a3e0 msgr2=0x7fe08813fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:40.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe086800640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe08806a3e0 0x7fe08813fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:40.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe086800640 1 -- 172.21.15.53:0/71315931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0700094d0 con 0x7fe088144c60 2024-01-16T15:06:40.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe086800640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 0x7fe088130ef0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe08000efc0 tx=0x7fe08000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080014480 con 0x7fe088144c60 2024-01-16T15:06:40.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe08000f040 con 0x7fe088144c60 2024-01-16T15:06:40.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080014820 con 0x7fe088144c60 2024-01-16T15:06:40.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.978+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe08812f8c0 con 0x7fe088144c60 2024-01-16T15:06:40.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.979+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe08812fe80 con 0x7fe088144c60 2024-01-16T15:06:40.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.980+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0880675d0 con 0x7fe088144c60 2024-01-16T15:06:40.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.981+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fe080014980 con 0x7fe088144c60 2024-01-16T15:06:40.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.982+0000 7fe07e000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe06006ba40 0x7fe06006df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:40.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.982+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe08008b9e0 con 0x7fe088144c60 2024-01-16T15:06:40.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.982+0000 7fe085e00640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe06006ba40 0x7fe06006df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:40.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.983+0000 7fe085e00640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe06006ba40 0x7fe06006df00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe078007fd0 tx=0x7fe078009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:40.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:40.986+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0800891a0 con 0x7fe088144c60 2024-01-16T15:06:41.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.216+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone30237", "target": ["mon-mgr", ""]}) v1 -- 0x7fe088072570 con 0x7fe06006ba40 2024-01-16T15:06:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.393+0000 7fe07e000640 1 -- 172.21.15.53:0/71315931 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe088072570 con 0x7fe06006ba40 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe06006ba40 msgr2=0x7fe06006df00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe06006ba40 0x7fe06006df00 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe078007fd0 tx=0x7fe078009040 comp rx=0 tx=0).stop 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 msgr2=0x7fe088130ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 0x7fe088130ef0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fe08000efc0 tx=0x7fe08000c490 comp rx=0 tx=0).stop 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 shutdown_connections 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe06006ba40 0x7fe06006df00 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe088144c60 0x7fe088130ef0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0880cc940 0x7fe0881309b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 --2- 172.21.15.53:0/71315931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe08806a3e0 0x7fe08813fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 >> 172.21.15.53:0/71315931 conn(0x7fe0880cfef0 msgr2=0x7fe0880d02d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 shutdown_connections 2024-01-16T15:06:41.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.397+0000 7fe08d000640 1 -- 172.21.15.53:0/71315931 wait complete. 2024-01-16T15:06:41.397 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:06:41.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30231 2024-01-16T15:06:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.647+0000 7fbb86e00640 1 Processor -- start 2024-01-16T15:06:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.647+0000 7fbb86e00640 1 -- start start 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb86e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbb8006b8c0 0x7fbb800ce160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb86e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8015f270 con 0x7fbb8015aff0 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb86e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb8015f550 con 0x7fbb8015ecd0 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb86e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb8006afd0 con 0x7fbb8006b8c0 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb85a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbb8015aff0 0x7fbb8006b1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56870/0 (socket says 172.21.15.53:56870) 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb85000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbb8006b8c0 0x7fbb800ce160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.649+0000 7fbb85a00640 1 -- 172.21.15.53:0/751541897 learned_addr learned my addr 172.21.15.53:0/751541897 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.650+0000 7fbb7fe00640 1 -- 172.21.15.53:0/751541897 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3599879056 0 0) 0x7fbb8015f270 con 0x7fbb8015aff0 2024-01-16T15:06:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.650+0000 7fbb7fe00640 1 -- 172.21.15.53:0/751541897 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb60003520 con 0x7fbb8015aff0 2024-01-16T15:06:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.650+0000 7fbb85000640 1 -- 172.21.15.53:0/751541897 >> v1:172.21.15.146:6789/0 conn(0x7fbb8015ecd0 legacy=0x7fbb800b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.650+0000 7fbb85000640 1 -- 172.21.15.53:0/751541897 >> v1:172.21.15.53:6789/0 conn(0x7fbb8015aff0 legacy=0x7fbb8006b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.650+0000 7fbb85000640 1 -- 172.21.15.53:0/751541897 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8015f830 con 0x7fbb8006b8c0 2024-01-16T15:06:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.651+0000 7fbb85000640 1 --2- 172.21.15.53:0/751541897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb800ce160 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbb680097a0 tx=0x7fbb6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0cfe4e11e3aa24e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb7fe00640 1 -- 172.21.15.53:0/751541897 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68031830 con 0x7fbb8006b8c0 2024-01-16T15:06:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb7fe00640 1 -- 172.21.15.53:0/751541897 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb68031990 con 0x7fbb8006b8c0 2024-01-16T15:06:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb7fe00640 1 -- 172.21.15.53:0/751541897 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb68031d20 con 0x7fbb8006b8c0 2024-01-16T15:06:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb86e00640 1 -- 172.21.15.53:0/751541897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 msgr2=0x7fbb800ce160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb86e00640 1 --2- 172.21.15.53:0/751541897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb800ce160 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbb680097a0 tx=0x7fbb6802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb86e00640 1 -- 172.21.15.53:0/751541897 shutdown_connections 2024-01-16T15:06:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb86e00640 1 --2- 172.21.15.53:0/751541897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb800ce160 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.652+0000 7fbb86e00640 1 -- 172.21.15.53:0/751541897 >> 172.21.15.53:0/751541897 conn(0x7fbb800b1ba0 msgr2=0x7fbb800b1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:41.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.653+0000 7fbb86e00640 1 -- 172.21.15.53:0/751541897 shutdown_connections 2024-01-16T15:06:41.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.653+0000 7fbb86e00640 1 -- 172.21.15.53:0/751541897 wait complete. 2024-01-16T15:06:41.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.654+0000 7fbb86e00640 1 Processor -- start 2024-01-16T15:06:41.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.655+0000 7fbb86e00640 1 -- start start 2024-01-16T15:06:41.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb86e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb80159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb86e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb8015aff0 0x7fbb801f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb86e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb8015ecd0 0x7fbb801469f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb86e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8015d4d0 con 0x7fbb8006b8c0 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb86e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb801f6ea0 con 0x7fbb8015aff0 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb86e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb801f2d30 con 0x7fbb8015ecd0 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb80159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb80159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46698/0 (socket says 172.21.15.53:46698) 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 -- 172.21.15.53:0/2011148975 learned_addr learned my addr 172.21.15.53:0/2011148975 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85000640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb8015aff0 0x7fbb801f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 -- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb8015ecd0 msgr2=0x7fbb801469f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb8015ecd0 0x7fbb801469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 -- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb8015aff0 msgr2=0x7fbb801f6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb8015aff0 0x7fbb801f6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.656+0000 7fbb85a00640 1 -- 172.21.15.53:0/2011148975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb68009400 con 0x7fbb8006b8c0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.657+0000 7fbb85a00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb80159270 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000c500 tx=0x7fbb7000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.657+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70005ce0 con 0x7fbb8006b8c0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.657+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb70005e40 con 0x7fbb8006b8c0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.657+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb70007330 con 0x7fbb8006b8c0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.657+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb801f3010 con 0x7fbb8006b8c0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.657+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb801f35d0 con 0x7fbb8006b8c0 2024-01-16T15:06:41.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.658+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb80067b60 con 0x7fbb8006b8c0 2024-01-16T15:06:41.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.659+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fbb70007550 con 0x7fbb8006b8c0 2024-01-16T15:06:41.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.660+0000 7fbb7d600640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb5806ba60 0x7fbb5806df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:41.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.660+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbb7008a380 con 0x7fbb8006b8c0 2024-01-16T15:06:41.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.661+0000 7fbb85000640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb5806ba60 0x7fbb5806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:41.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.662+0000 7fbb85000640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb5806ba60 0x7fbb5806df20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbb68036000 tx=0x7fbb68008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:41.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.665+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb70087950 con 0x7fbb8006b8c0 2024-01-16T15:06:41.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.898+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30231", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb8015b3d0 con 0x7fbb5806ba60 2024-01-16T15:06:41.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.922+0000 7fbb7d600640 1 -- 172.21.15.53:0/2011148975 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7fbb68031b00 con 0x7fbb5806ba60 2024-01-16T15:06:41.900 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:41.902 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb5806ba60 msgr2=0x7fbb5806df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb5806ba60 0x7fbb5806df20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fbb68036000 tx=0x7fbb68008030 comp rx=0 tx=0).stop 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 msgr2=0x7fbb80159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb80159270 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7fbb7000c500 tx=0x7fbb7000c9c0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 shutdown_connections 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb5806ba60 0x7fbb5806df20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb8015ecd0 0x7fbb801469f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb8015aff0 0x7fbb801f6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 --2- 172.21.15.53:0/2011148975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb8006b8c0 0x7fbb80159270 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:41.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 >> 172.21.15.53:0/2011148975 conn(0x7fbb800b1ba0 msgr2=0x7fbb80153d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:41.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 shutdown_connections 2024-01-16T15:06:41.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:41.924+0000 7fbb86e00640 1 -- 172.21.15.53:0/2011148975 wait complete. 2024-01-16T15:06:41.921 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:06:41.921 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30237 2024-01-16T15:06:42.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.167+0000 7f83fda00640 1 Processor -- start 2024-01-16T15:06:42.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.167+0000 7f83fda00640 1 -- start start 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83fda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f83f813df80 0x7f83f815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83fda00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83fda00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83f806c470 con 0x7f83f815f9c0 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83fda00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f806c750 con 0x7f83f813df80 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83fda00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f806ca30 con 0x7f83f813e360 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83f6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f83f813df80 0x7f83f815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83f7400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f83f815f9c0 0x7f83f81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57424/0 (socket says 172.21.15.53:57424) 2024-01-16T15:06:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.169+0000 7f83f7400640 1 -- 172.21.15.53:0/3102207050 learned_addr learned my addr 172.21.15.53:0/3102207050 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.170+0000 7f83f6000640 1 --2- 172.21.15.53:0/3102207050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.170+0000 7f83f6a00640 1 -- 172.21.15.53:0/3102207050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 msgr2=0x7f83f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.170+0000 7f83f6a00640 1 --2- 172.21.15.53:0/3102207050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.170+0000 7f83f6a00640 1 -- 172.21.15.53:0/3102207050 >> v1:172.21.15.53:6789/0 conn(0x7f83f815f9c0 legacy=0x7f83f81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.170+0000 7f83f6a00640 1 -- 172.21.15.53:0/3102207050 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f806cd10 con 0x7f83f813df80 2024-01-16T15:06:42.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.171+0000 7f83f6a00640 1 --2- 172.21.15.53:0/3102207050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f813df80 0x7f83f815bee0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f83e00097a0 tx=0x7f83e002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd009eef73865562 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.171+0000 7f83f5600640 1 -- 172.21.15.53:0/3102207050 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e002d040 con 0x7f83f813df80 2024-01-16T15:06:42.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.171+0000 7f83f5600640 1 -- 172.21.15.53:0/3102207050 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e0031810 con 0x7f83f813df80 2024-01-16T15:06:42.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.171+0000 7f83f5600640 1 -- 172.21.15.53:0/3102207050 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e0031ba0 con 0x7f83f813df80 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 -- 172.21.15.53:0/3102207050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f813df80 msgr2=0x7f83f815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 --2- 172.21.15.53:0/3102207050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f813df80 0x7f83f815bee0 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f83e00097a0 tx=0x7f83e002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 -- 172.21.15.53:0/3102207050 shutdown_connections 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 --2- 172.21.15.53:0/3102207050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 --2- 172.21.15.53:0/3102207050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f813df80 0x7f83f815bee0 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 -- 172.21.15.53:0/3102207050 >> 172.21.15.53:0/3102207050 conn(0x7f83f806a800 msgr2=0x7f83f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 -- 172.21.15.53:0/3102207050 shutdown_connections 2024-01-16T15:06:42.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.172+0000 7f83fda00640 1 -- 172.21.15.53:0/3102207050 wait complete. 2024-01-16T15:06:42.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.174+0000 7f83fda00640 1 Processor -- start 2024-01-16T15:06:42.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.174+0000 7f83fda00640 1 -- start start 2024-01-16T15:06:42.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83fda00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83f813df80 0x7f83f813d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83fda00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f814ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83fda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 0x7f83f80afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83fda00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f814c810 con 0x7f83f815f9c0 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83fda00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f814bac0 con 0x7f83f813df80 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83fda00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f814b500 con 0x7f83f813e360 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.175+0000 7f83f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 0x7f83f80afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 0x7f83f80afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54608/0 (socket says 172.21.15.53:54608) 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 -- 172.21.15.53:0/3095100330 learned_addr learned my addr 172.21.15.53:0/3095100330 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f6a00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83f813df80 0x7f83f813d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f6000640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f814ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 -- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 msgr2=0x7f83f814ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f814ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 -- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83f813df80 msgr2=0x7f83f813d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83f813df80 0x7f83f813d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 -- 172.21.15.53:0/3095100330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e0009400 con 0x7f83f815f9c0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f6000640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f814ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.176+0000 7f83f7400640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 0x7f83f80afab0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f83f000cbc0 tx=0x7f83f000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.177+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f000ad40 con 0x7f83f815f9c0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.177+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f0012400 con 0x7f83f815f9c0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.177+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f000aea0 con 0x7f83f815f9c0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.177+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f814b7e0 con 0x7f83f815f9c0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.177+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f813dc50 con 0x7f83f815f9c0 2024-01-16T15:06:42.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.178+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f814c980 con 0x7f83f815f9c0 2024-01-16T15:06:42.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.179+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f83f0007070 con 0x7f83f815f9c0 2024-01-16T15:06:42.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.181+0000 7f83eea00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83d006ba10 0x7f83d006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.181+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f83f0088430 con 0x7f83f815f9c0 2024-01-16T15:06:42.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.181+0000 7f83f6a00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83d006ba10 0x7f83d006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.182+0000 7f83f6a00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83d006ba10 0x7f83d006ded0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f83e0036000 tx=0x7f83e003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.185+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f83f0088140 con 0x7f83f815f9c0 2024-01-16T15:06:42.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.479+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30237", "target": ["mon-mgr", ""]}) v1 -- 0x7f83f80ce850 con 0x7f83d006ba10 2024-01-16T15:06:42.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.506+0000 7f83eea00640 1 -- 172.21.15.53:0/3095100330 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f83e0031b00 con 0x7f83d006ba10 2024-01-16T15:06:42.485 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:42.485 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:42.486 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:42.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.509+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83d006ba10 msgr2=0x7f83d006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83d006ba10 0x7f83d006ded0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f83e0036000 tx=0x7f83e003c040 comp rx=0 tx=0).stop 2024-01-16T15:06:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 msgr2=0x7f83f80afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 0x7f83f80afab0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f83f000cbc0 tx=0x7f83f000a430 comp rx=0 tx=0).stop 2024-01-16T15:06:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 shutdown_connections 2024-01-16T15:06:42.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83d006ba10 0x7f83d006ded0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83f815f9c0 0x7f83f80afab0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83f813e360 0x7f83f814ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 --2- 172.21.15.53:0/3095100330 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83f813df80 0x7f83f813d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 >> 172.21.15.53:0/3095100330 conn(0x7f83f806a800 msgr2=0x7f83f815cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 shutdown_connections 2024-01-16T15:06:42.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.510+0000 7f83fda00640 1 -- 172.21.15.53:0/3095100330 wait complete. 2024-01-16T15:06:42.528 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.847+0000 7f1049a00640 1 Processor -- start 2024-01-16T15:06:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.847+0000 7f1049a00640 1 -- start start 2024-01-16T15:06:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.849+0000 7f1049a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f10440cf9a0 0x7f104415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.849+0000 7f1049a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 0x7f10440702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.849+0000 7f1049a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10440b1ee0 con 0x7f1044143ae0 2024-01-16T15:06:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.849+0000 7f1049a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10440b2780 con 0x7f10440cf9a0 2024-01-16T15:06:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.849+0000 7f1049a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f104415e990 con 0x7f104415e1e0 2024-01-16T15:06:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.850+0000 7f103be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f10440cf9a0 0x7f104415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.850+0000 7f1043400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 0x7f10440702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.850+0000 7f103be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f10440cf9a0 0x7f104415dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:39708/0 (socket says 172.21.15.53:39708) 2024-01-16T15:06:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.850+0000 7f103be00640 1 -- 172.21.15.53:0/60188443 learned_addr learned my addr 172.21.15.53:0/60188443 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.850+0000 7f1043400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 0x7f10440702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:34116/0 (socket says 172.21.15.53:34116) 2024-01-16T15:06:42.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.851+0000 7f103be00640 1 -- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 msgr2=0x7f10440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:42.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.851+0000 7f103be00640 1 --2- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 0x7f10440702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.851+0000 7f103be00640 1 -- 172.21.15.53:0/60188443 >> v1:172.21.15.146:6789/0 conn(0x7f1044143ae0 legacy=0x7f10440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.851+0000 7f103be00640 1 -- 172.21.15.53:0/60188443 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10440b2130 con 0x7f10440cf9a0 2024-01-16T15:06:42.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.851+0000 7f1043400640 1 --2- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 0x7f10440702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:42.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.854+0000 7f103be00640 1 --2- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 0x7f104415dca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f102c009870 tx=0x7f102c02db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48c7bf14c0860aad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.854+0000 7f1042000640 1 -- 172.21.15.53:0/60188443 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c02e940 con 0x7f10440cf9a0 2024-01-16T15:06:42.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.854+0000 7f1042000640 1 -- 172.21.15.53:0/60188443 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102c02eaa0 con 0x7f10440cf9a0 2024-01-16T15:06:42.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.854+0000 7f1042000640 1 -- 172.21.15.53:0/60188443 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c02ee30 con 0x7f10440cf9a0 2024-01-16T15:06:42.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.855+0000 7f1049a00640 1 -- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 msgr2=0x7f104415dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.855+0000 7f1049a00640 1 --2- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 0x7f104415dca0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f102c009870 tx=0x7f102c02db90 comp rx=0 tx=0).stop 2024-01-16T15:06:42.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.856+0000 7f1049a00640 1 -- 172.21.15.53:0/60188443 shutdown_connections 2024-01-16T15:06:42.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.856+0000 7f1049a00640 1 --2- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f104415e1e0 0x7f10440702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.856+0000 7f1049a00640 1 --2- 172.21.15.53:0/60188443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 0x7f104415dca0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.856+0000 7f1049a00640 1 -- 172.21.15.53:0/60188443 >> 172.21.15.53:0/60188443 conn(0x7f104406a800 msgr2=0x7f104406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:42.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.856+0000 7f1049a00640 1 -- 172.21.15.53:0/60188443 shutdown_connections 2024-01-16T15:06:42.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.856+0000 7f1049a00640 1 -- 172.21.15.53:0/60188443 wait complete. 2024-01-16T15:06:42.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.859+0000 7f1049a00640 1 Processor -- start 2024-01-16T15:06:42.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.859+0000 7f1049a00640 1 -- start start 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.861+0000 7f1049a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 0x7f104413f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.861+0000 7f1049a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1044143ae0 0x7f10440cc610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1049a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 0x7f104413d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1049a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10440cd920 con 0x7f104415e1e0 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1049a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104413dea0 con 0x7f10440cf9a0 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1049a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10440ccd10 con 0x7f1044143ae0 2024-01-16T15:06:42.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1043400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 0x7f104413d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1043400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 0x7f104413d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54616/0 (socket says 172.21.15.53:54616) 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.862+0000 7f1043400640 1 -- 172.21.15.53:0/760685418 learned_addr learned my addr 172.21.15.53:0/760685418 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.863+0000 7f1043400640 1 -- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1044143ae0 msgr2=0x7f10440cc610 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.863+0000 7f1043400640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1044143ae0 0x7f10440cc610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.863+0000 7f1043400640 1 -- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 msgr2=0x7f104413f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.863+0000 7f1043400640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 0x7f104413f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.863+0000 7f1043400640 1 -- 172.21.15.53:0/760685418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c009a70 con 0x7f104415e1e0 2024-01-16T15:06:42.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.863+0000 7f1043400640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 0x7f104413d960 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f103c009e60 tx=0x7f103c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:42.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.864+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c00cdf0 con 0x7f104415e1e0 2024-01-16T15:06:42.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.864+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f103c00f040 con 0x7f104415e1e0 2024-01-16T15:06:42.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.864+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c0094d0 con 0x7f104415e1e0 2024-01-16T15:06:42.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.864+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f103c003bb0 con 0x7f104415e1e0 2024-01-16T15:06:42.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.864+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10440cd2d0 con 0x7f104415e1e0 2024-01-16T15:06:42.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.866+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10440675d0 con 0x7f104415e1e0 2024-01-16T15:06:42.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.868+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7f103c014480 con 0x7f104415e1e0 2024-01-16T15:06:42.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.869+0000 7f103aa00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f101c06ba90 0x7f101c06df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:42.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.869+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f103c08b600 con 0x7f104415e1e0 2024-01-16T15:06:42.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.874+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f103c088e40 con 0x7f104415e1e0 2024-01-16T15:06:42.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.899+0000 7f1042a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f101c06ba90 0x7f101c06df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:42.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:42.904+0000 7f1042a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f101c06ba90 0x7f101c06df50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1034000f60 tx=0x7f10340079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:43.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.395+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f10440affa0 con 0x7f104415e1e0 2024-01-16T15:06:43.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.396+0000 7f103aa00640 1 -- 172.21.15.53:0/760685418 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7f103c018090 con 0x7f104415e1e0 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:43.376 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 430, 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:43.377 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:43.378 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 804, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3222367969, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 6761144320, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 766332968960, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 767, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 31304, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 0, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 7 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:43.379 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:43.380 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:43.381 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:43.382 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.400+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f101c06ba90 msgr2=0x7f101c06df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.400+0000 7f1049a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f101c06ba90 0x7f101c06df50 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f1034000f60 tx=0x7f10340079b0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.400+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 msgr2=0x7f104413d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.400+0000 7f1049a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 0x7f104413d960 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f103c009e60 tx=0x7f103c00c490 comp rx=0 tx=0).stop 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 shutdown_connections 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f101c06ba90 0x7f101c06df50 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f104415e1e0 0x7f104413d960 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1044143ae0 0x7f10440cc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 --2- 172.21.15.53:0/760685418 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10440cf9a0 0x7f104413f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:43.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 >> 172.21.15.53:0/760685418 conn(0x7f104406a800 msgr2=0x7f104406c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:43.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.401+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 shutdown_connections 2024-01-16T15:06:43.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:43.402+0000 7f1049a00640 1 -- 172.21.15.53:0/760685418 wait complete. 2024-01-16T15:06:44.418 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:44.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.738+0000 7fb2a3000640 1 Processor -- start 2024-01-16T15:06:44.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.739+0000 7fb2a3000640 1 -- start start 2024-01-16T15:06:44.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.740+0000 7fb2a3000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb29c160f80 0x7fb29c161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:44.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.741+0000 7fb2a3000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb29c0ce8f0 0x7fb29c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:44.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.741+0000 7fb2a3000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb29c1618a0 con 0x7fb29c0cfbe0 2024-01-16T15:06:44.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.741+0000 7fb2a3000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c0d0de0 con 0x7fb29c160f80 2024-01-16T15:06:44.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.741+0000 7fb2a3000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb29c0d10c0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb29c160f80 0x7fb29c161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:44.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb29c160f80 0x7fb29c161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54628/0 (socket says 172.21.15.53:54628) 2024-01-16T15:06:44.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 -- 172.21.15.53:0/826826408 learned_addr learned my addr 172.21.15.53:0/826826408 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:44.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 -- 172.21.15.53:0/826826408 >> v2:172.21.15.146:3300/0 conn(0x7fb29c0ce8f0 msgr2=0x7fb29c0ced50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:06:44.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 --2- 172.21.15.53:0/826826408 >> v2:172.21.15.146:3300/0 conn(0x7fb29c0ce8f0 0x7fb29c0ced50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 -- 172.21.15.53:0/826826408 >> v1:172.21.15.146:6789/0 conn(0x7fb29c0cfbe0 legacy=0x7fb29c0d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:44.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.742+0000 7fb29be00640 1 -- 172.21.15.53:0/826826408 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb29c0d13a0 con 0x7fb29c160f80 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.744+0000 7fb29be00640 1 --2- 172.21.15.53:0/826826408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c160f80 0x7fb29c161360 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0b5d10 tx=0x7fb28c02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c88d6e5f585fe2aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.745+0000 7fb29aa00640 1 -- 172.21.15.53:0/826826408 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c031890 con 0x7fb29c160f80 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.745+0000 7fb29aa00640 1 -- 172.21.15.53:0/826826408 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c0319f0 con 0x7fb29c160f80 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.745+0000 7fb29aa00640 1 -- 172.21.15.53:0/826826408 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c031890 con 0x7fb29c160f80 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.747+0000 7fb2a3000640 1 -- 172.21.15.53:0/826826408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c160f80 msgr2=0x7fb29c161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.747+0000 7fb2a3000640 1 --2- 172.21.15.53:0/826826408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c160f80 0x7fb29c161360 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0b5d10 tx=0x7fb28c02b930 comp rx=0 tx=0).stop 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.748+0000 7fb2a3000640 1 -- 172.21.15.53:0/826826408 shutdown_connections 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.748+0000 7fb2a3000640 1 --2- 172.21.15.53:0/826826408 >> v2:172.21.15.146:3300/0 conn(0x7fb29c0ce8f0 0x7fb29c0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.748+0000 7fb2a3000640 1 --2- 172.21.15.53:0/826826408 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c160f80 0x7fb29c161360 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.748+0000 7fb2a3000640 1 -- 172.21.15.53:0/826826408 >> 172.21.15.53:0/826826408 conn(0x7fb29c09c550 msgr2=0x7fb29c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:44.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.748+0000 7fb2a3000640 1 -- 172.21.15.53:0/826826408 shutdown_connections 2024-01-16T15:06:44.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.748+0000 7fb2a3000640 1 -- 172.21.15.53:0/826826408 wait complete. 2024-01-16T15:06:44.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.751+0000 7fb2a3000640 1 Processor -- start 2024-01-16T15:06:44.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.751+0000 7fb2a3000640 1 -- start start 2024-01-16T15:06:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.752+0000 7fb2a3000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 0x7fb29c15a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.752+0000 7fb2a3000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb29c0cfbe0 0x7fb29c0cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.752+0000 7fb2a3000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb29c0ce290 0x7fb29c0c7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.752+0000 7fb2a3000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0ce720 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.752+0000 7fb2a3000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c0c79f0 con 0x7fb29c0cfbe0 2024-01-16T15:06:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.752+0000 7fb2a3000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb29c15a8a0 con 0x7fb29c0ce290 2024-01-16T15:06:44.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.753+0000 7fb29be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 0x7fb29c15a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:44.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.753+0000 7fb29be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 0x7fb29c15a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54642/0 (socket says 172.21.15.53:54642) 2024-01-16T15:06:44.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.753+0000 7fb29be00640 1 -- 172.21.15.53:0/1322321863 learned_addr learned my addr 172.21.15.53:0/1322321863 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:44.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.753+0000 7fb29be00640 1 -- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb29c0ce290 msgr2=0x7fb29c0c7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:44.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.753+0000 7fb29be00640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb29c0ce290 0x7fb29c0c7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.754+0000 7fb29be00640 1 -- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb29c0cfbe0 msgr2=0x7fb29c0cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:44.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.754+0000 7fb29be00640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb29c0cfbe0 0x7fb29c0cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:44.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.754+0000 7fb29be00640 1 -- 172.21.15.53:0/1322321863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb28c0094d0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.754+0000 7fb29be00640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 0x7fb29c15a360 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb28c02b970 tx=0x7fb28c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.755+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c003aa0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.755+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb28c03f070 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.755+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb28c003e00 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.756+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29c0c6de0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.756+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb29c0c73a0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.758+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb29c0675d0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.759+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 17) v1 ==== 90202+0+0 (secure 0 0 0) 0x7fb28c046480 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.760+0000 7fb293e00640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb27406ba10 0x7fb27406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:44.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.761+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb28c041070 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.762+0000 7fb29b400640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb27406ba10 0x7fb27406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:44.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.767+0000 7fb29b400640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb27406ba10 0x7fb27406ded0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0666f0 tx=0x7fb284006c40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:44.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:44.770+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb28c0aafb0 con 0x7fb29c0ce8f0 2024-01-16T15:06:44.934 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:44.956+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:44.934 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:44.956+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:44.934 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:44.956+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:44.934 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:44.956+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:44.934 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:44.956+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:45.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.206+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb29c0d3920 con 0x7fb29c0ce8f0 2024-01-16T15:06:45.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.207+0000 7fb293e00640 1 -- 172.21.15.53:0/1322321863 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2337 (secure 0 0 0) 0x7fb28c04b940 con 0x7fb29c0ce8f0 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:45.186 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 432, 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:45.187 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:45.188 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 804, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3222367969, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 6761144320, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 766332968960, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 767, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 31304, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 0, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 7 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:45.189 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:45.190 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:45.191 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "2 ongoing cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "5 ongoing+pending cloning jobs; Avg progress = 0.0% (0s)\n [............................] ", 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:45.192 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb27406ba10 msgr2=0x7fb27406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb27406ba10 0x7fb27406ded0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb29c0666f0 tx=0x7fb284006c40 comp rx=0 tx=0).stop 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 msgr2=0x7fb29c15a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 0x7fb29c15a360 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fb28c02b970 tx=0x7fb28c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 shutdown_connections 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb27406ba10 0x7fb27406ded0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb29c0ce290 0x7fb29c0c7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb29c0cfbe0 0x7fb29c0cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 --2- 172.21.15.53:0/1322321863 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb29c0ce8f0 0x7fb29c15a360 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:45.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 >> 172.21.15.53:0/1322321863 conn(0x7fb29c09c550 msgr2=0x7fb29c0b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:45.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 shutdown_connections 2024-01-16T15:06:45.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:45.209+0000 7fb2a3000640 1 -- 172.21.15.53:0/1322321863 wait complete. 2024-01-16T15:06:47.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:47.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.475+0000 7f9dc0e00640 1 Processor -- start 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.475+0000 7f9dc0e00640 1 -- start start 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.477+0000 7f9dc0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc0cd170 0x7f9dbc071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.477+0000 7f9dc0e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc071690 con 0x7f9dbc0cf9a0 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.477+0000 7f9dc0e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dbc071970 con 0x7f9dbc14b2c0 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.477+0000 7f9dc0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc14b6a0 con 0x7f9dbc0cd170 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.477+0000 7f9db9400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9dbc0cf9a0 0x7f9dbc070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57436/0 (socket says 172.21.15.53:57436) 2024-01-16T15:06:47.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.477+0000 7f9db9400640 1 -- 172.21.15.53:0/2095555655 learned_addr learned my addr 172.21.15.53:0/2095555655 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:47.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566356248 0 0) 0x7f9dbc071690 con 0x7f9dbc0cf9a0 2024-01-16T15:06:47.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d9c003520 con 0x7f9dbc0cf9a0 2024-01-16T15:06:47.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3466396602 0 0) 0x7f9d9c003520 con 0x7f9dbc0cf9a0 2024-01-16T15:06:47.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc0cd170 msgr2=0x7f9dbc071070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:47.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc0cd170 0x7f9dbc071070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 >> v1:172.21.15.146:6789/0 conn(0x7f9dbc14b2c0 legacy=0x7f9dbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:47.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc071c50 con 0x7f9dbc0cf9a0 2024-01-16T15:06:47.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9dac001670 con 0x7f9dbc0cf9a0 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 >> v1:172.21.15.53:6789/0 conn(0x7f9dbc0cf9a0 legacy=0x7f9dbc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d9c0039a0 0x7f9d9c003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0047c0 0x7f9d9c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.478+0000 7f9db8a00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9c0052a0 0x7f9d9c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc14b6a0 con 0x7f9d9c0039a0 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c00c100 con 0x7f9d9c0047c0 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9c00c2c0 con 0x7f9d9c0052a0 2024-01-16T15:06:47.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db9e00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d9c0039a0 0x7f9d9c003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db9e00640 1 -- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9c0052a0 msgr2=0x7f9d9c00b980 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db9e00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d9c0052a0 0x7f9d9c00b980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db9e00640 1 -- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0047c0 msgr2=0x7f9d9c004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db9e00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0047c0 0x7f9d9c004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.479+0000 7f9db9e00640 1 -- 172.21.15.53:0/2095555655 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc071c50 con 0x7f9d9c0039a0 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.481+0000 7f9db9e00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d9c0039a0 0x7f9d9c003da0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9da4003d80 tx=0x7f9d9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64425e72f185d554 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.481+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da4032600 con 0x7f9d9c0039a0 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.481+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9da402e050 con 0x7f9d9c0039a0 2024-01-16T15:06:47.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.481+0000 7f9db8a00640 1 -- 172.21.15.53:0/2095555655 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9da40329a0 con 0x7f9d9c0039a0 2024-01-16T15:06:47.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.482+0000 7f9dc0e00640 1 -- 172.21.15.53:0/2095555655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d9c0039a0 msgr2=0x7f9d9c003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.482+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/2095555655 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d9c0039a0 0x7f9d9c003da0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f9da4003d80 tx=0x7f9d9c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.482+0000 7f9dba800640 1 -- 172.21.15.53:0/2095555655 reap_dead start 2024-01-16T15:06:47.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.482+0000 7f9dc0e00640 1 -- 172.21.15.53:0/2095555655 shutdown_connections 2024-01-16T15:06:47.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.482+0000 7f9dc0e00640 1 -- 172.21.15.53:0/2095555655 >> 172.21.15.53:0/2095555655 conn(0x7f9dbc06a800 msgr2=0x7f9dbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:47.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.482+0000 7f9dc0e00640 1 -- 172.21.15.53:0/2095555655 shutdown_connections 2024-01-16T15:06:47.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.483+0000 7f9dc0e00640 1 -- 172.21.15.53:0/2095555655 wait complete. 2024-01-16T15:06:47.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.486+0000 7f9dc0e00640 1 Processor -- start 2024-01-16T15:06:47.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.487+0000 7f9dc0e00640 1 -- start start 2024-01-16T15:06:47.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dc0e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0039a0 0x7f9dbc15e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dc0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc1634f0 0x7f9dbc163a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dc0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 0x7f9dbc0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dc0e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc15e8c0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dc0e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc1fa500 con 0x7f9d9c0039a0 2024-01-16T15:06:47.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dc0e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9dbc1fa7e0 con 0x7f9dbc1634f0 2024-01-16T15:06:47.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dba800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 0x7f9dbc0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dba800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 0x7f9dbc0cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54654/0 (socket says 172.21.15.53:54654) 2024-01-16T15:06:47.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.488+0000 7f9dba800640 1 -- 172.21.15.53:0/1782682874 learned_addr learned my addr 172.21.15.53:0/1782682874 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:47.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.489+0000 7f9dba800640 1 -- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc1634f0 msgr2=0x7f9dbc163a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.489+0000 7f9dba800640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc1634f0 0x7f9dbc163a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.489+0000 7f9dba800640 1 -- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0039a0 msgr2=0x7f9dbc15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:47.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.489+0000 7f9dba800640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0039a0 0x7f9dbc15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:47.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.489+0000 7f9dba800640 1 -- 172.21.15.53:0/1782682874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dbc1faac0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.491+0000 7f9dba800640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 0x7f9dbc0cc060 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0cd550 tx=0x7f9db400c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.492+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db400c840 con 0x7f9dbc163fd0 2024-01-16T15:06:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.492+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9db400c9a0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.492+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9db400cd10 con 0x7f9dbc163fd0 2024-01-16T15:06:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.492+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dbc1fada0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.492+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dbc0cc5a0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.494+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dbc1fb2c0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.495+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) v1 ==== 90253+0+0 (secure 0 0 0) 0x7f9db4012020 con 0x7f9dbc163fd0 2024-01-16T15:06:47.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.497+0000 7f9db2000640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d9406bba0 0x7f9d9406e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:47.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.498+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9db4087aa0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.499+0000 7f9db9e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d9406bba0 0x7f9d9406e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:47.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.503+0000 7f9db9e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d9406bba0 0x7f9d9406e060 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9da4034000 tx=0x7f9da402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:47.504+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9dbc1fb2c0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.010+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9dbc1fb2c0 con 0x7f9dbc163fd0 2024-01-16T15:06:47.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.010+0000 7f9db2000640 1 -- 172.21.15.53:0/1782682874 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2423 (secure 0 0 0) 0x7f9db401a090 con 0x7f9dbc163fd0 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:47.990 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:47.991 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 435, 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:47.992 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:47.993 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1242, 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 5059473595, 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 9481039872, 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 763613073408, 2024-01-16T15:06:47.994 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 155828099, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 153052359, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 38, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 43 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:47.995 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:47.996 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:47.997 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "2 ongoing cloning jobs; Avg progress = 33.33% (5s)\n [=========...................] (remaining: 10s)", 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.33329999446868896, 2024-01-16T15:06:47.998 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (5s)\n [==..........................] (remaining: 48s)", 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:47.999 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d9406bba0 msgr2=0x7f9d9406e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d9406bba0 0x7f9d9406e060 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f9da4034000 tx=0x7f9da402f000 comp rx=0 tx=0).stop 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 msgr2=0x7f9dbc0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 0x7f9dbc0cc060 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc0cd550 tx=0x7f9db400c040 comp rx=0 tx=0).stop 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 shutdown_connections 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d9406bba0 0x7f9d9406e060 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9dbc163fd0 0x7f9dbc0cc060 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9dbc1634f0 0x7f9dbc163a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 --2- 172.21.15.53:0/1782682874 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d9c0039a0 0x7f9dbc15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:48.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 >> 172.21.15.53:0/1782682874 conn(0x7f9dbc06a800 msgr2=0x7f9dbc06cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:48.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 shutdown_connections 2024-01-16T15:06:48.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:48.013+0000 7f9dc0e00640 1 -- 172.21.15.53:0/1782682874 wait complete. 2024-01-16T15:06:49.908 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:49.930+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:49.908 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:49.930+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:49.908 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:49.930+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:49.908 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:49.930+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:49.909 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:06:49.930+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:06:50.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:50.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.296+0000 7fbd2c400640 1 Processor -- start 2024-01-16T15:06:50.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.296+0000 7fbd2c400640 1 -- start start 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd2c400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd240cf9a0 0x7fbd24166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd2c400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2415c5f0 con 0x7fbd2413df80 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd2c400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd2406c470 con 0x7fbd2413e360 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd2c400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd2406c750 con 0x7fbd240cf9a0 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd29a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbd2413df80 0x7fbd2415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57440/0 (socket says 172.21.15.53:57440) 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd29a00640 1 -- 172.21.15.53:0/4024340504 learned_addr learned my addr 172.21.15.53:0/4024340504 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:50.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.298+0000 7fbd2a400640 1 --2- 172.21.15.53:0/4024340504 >> v2:172.21.15.53:3300/0 conn(0x7fbd240cf9a0 0x7fbd24166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.299+0000 7fbd23e00640 1 -- 172.21.15.53:0/4024340504 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591948580 0 0) 0x7fbd2415c5f0 con 0x7fbd2413df80 2024-01-16T15:06:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.299+0000 7fbd23e00640 1 -- 172.21.15.53:0/4024340504 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04003520 con 0x7fbd2413df80 2024-01-16T15:06:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.299+0000 7fbd2a400640 1 -- 172.21.15.53:0/4024340504 >> v1:172.21.15.146:6789/0 conn(0x7fbd2413e360 legacy=0x7fbd240cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.299+0000 7fbd2a400640 1 -- 172.21.15.53:0/4024340504 >> v1:172.21.15.53:6789/0 conn(0x7fbd2413df80 legacy=0x7fbd2415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.299+0000 7fbd2a400640 1 -- 172.21.15.53:0/4024340504 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2406ca30 con 0x7fbd240cf9a0 2024-01-16T15:06:50.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.300+0000 7fbd2a400640 1 --2- 172.21.15.53:0/4024340504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cf9a0 0x7fbd24166030 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009d90 tx=0x7fbd1c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9eeaa0c20250b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.301+0000 7fbd23e00640 1 -- 172.21.15.53:0/4024340504 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c02cc60 con 0x7fbd240cf9a0 2024-01-16T15:06:50.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.301+0000 7fbd23e00640 1 -- 172.21.15.53:0/4024340504 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c02cdc0 con 0x7fbd240cf9a0 2024-01-16T15:06:50.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.301+0000 7fbd23e00640 1 -- 172.21.15.53:0/4024340504 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c034660 con 0x7fbd240cf9a0 2024-01-16T15:06:50.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.302+0000 7fbd2c400640 1 -- 172.21.15.53:0/4024340504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cf9a0 msgr2=0x7fbd24166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.302+0000 7fbd2c400640 1 --2- 172.21.15.53:0/4024340504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cf9a0 0x7fbd24166030 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009d90 tx=0x7fbd1c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.303+0000 7fbd2c400640 1 -- 172.21.15.53:0/4024340504 shutdown_connections 2024-01-16T15:06:50.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.303+0000 7fbd2c400640 1 --2- 172.21.15.53:0/4024340504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cf9a0 0x7fbd24166030 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.303+0000 7fbd2c400640 1 -- 172.21.15.53:0/4024340504 >> 172.21.15.53:0/4024340504 conn(0x7fbd2406a800 msgr2=0x7fbd2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.303+0000 7fbd2c400640 1 -- 172.21.15.53:0/4024340504 shutdown_connections 2024-01-16T15:06:50.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.303+0000 7fbd2c400640 1 -- 172.21.15.53:0/4024340504 wait complete. 2024-01-16T15:06:50.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.306+0000 7fbd2c400640 1 Processor -- start 2024-01-16T15:06:50.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.307+0000 7fbd2c400640 1 -- start start 2024-01-16T15:06:50.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.309+0000 7fbd2c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd240cf9a0 0x7fbd2413d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.309+0000 7fbd2c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2413df80 0x7fbd240702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.309+0000 7fbd2c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2413e360 0x7fbd241fab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.309+0000 7fbd2c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24158100 con 0x7fbd2413df80 2024-01-16T15:06:50.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.309+0000 7fbd2c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2415aaf0 con 0x7fbd240cf9a0 2024-01-16T15:06:50.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.309+0000 7fbd2c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24157760 con 0x7fbd2413e360 2024-01-16T15:06:50.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.310+0000 7fbd29000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2413df80 0x7fbd240702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.310+0000 7fbd29a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd240cf9a0 0x7fbd2413d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.310+0000 7fbd29a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd240cf9a0 0x7fbd2413d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:39744/0 (socket says 172.21.15.53:39744) 2024-01-16T15:06:50.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.310+0000 7fbd29a00640 1 -- 172.21.15.53:0/2271516409 learned_addr learned my addr 172.21.15.53:0/2271516409 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:50.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.310+0000 7fbd2a400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2413e360 0x7fbd241fab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.311+0000 7fbd29000640 1 -- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2413e360 msgr2=0x7fbd241fab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.311+0000 7fbd29000640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2413e360 0x7fbd241fab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.311+0000 7fbd29000640 1 -- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd240cf9a0 msgr2=0x7fbd2413d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.311+0000 7fbd29000640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd240cf9a0 0x7fbd2413d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.311+0000 7fbd29000640 1 -- 172.21.15.53:0/2271516409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009a70 con 0x7fbd2413df80 2024-01-16T15:06:50.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.311+0000 7fbd29000640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2413df80 0x7fbd240702c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00c3e0 tx=0x7fbd0c00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.314+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c0035f0 con 0x7fbd2413df80 2024-01-16T15:06:50.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.315+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0c003750 con 0x7fbd2413df80 2024-01-16T15:06:50.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.315+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c003ae0 con 0x7fbd2413df80 2024-01-16T15:06:50.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.315+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd24157a40 con 0x7fbd2413df80 2024-01-16T15:06:50.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.315+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd2413d720 con 0x7fbd2413df80 2024-01-16T15:06:50.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.319+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24158240 con 0x7fbd2413df80 2024-01-16T15:06:50.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.321+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 18) v1 ==== 90253+0+0 (secure 0 0 0) 0x7fbd0c003c40 con 0x7fbd2413df80 2024-01-16T15:06:50.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.322+0000 7fbd21600640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06baf0 0x7fbcfc06dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:50.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.323+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbd0c087770 con 0x7fbd2413df80 2024-01-16T15:06:50.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.328+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd24158240 con 0x7fbd2413df80 2024-01-16T15:06:50.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.341+0000 7fbd29a00640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06baf0 0x7fbcfc06dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:50.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.352+0000 7fbd29a00640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06baf0 0x7fbcfc06dfb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbd14007fd0 tx=0x7fbd14009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:50.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.468+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fbd0c054660 con 0x7fbd2413df80 2024-01-16T15:06:50.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.821+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fbd24158240 con 0x7fbd2413df80 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.821+0000 7fbd21600640 1 -- 172.21.15.53:0/2271516409 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2424 (secure 0 0 0) 0x7fbd24158240 con 0x7fbd2413df80 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:50.800 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 437, 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:50.801 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:50.802 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1586, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 6506605689, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 12511092736, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 760583020544, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 276713511, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 273592081, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 66, 2024-01-16T15:06:50.803 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 73 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:50.804 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:50.805 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "3 ongoing cloning jobs; Avg progress = 22.22% (7s)\n [======......................] (remaining: 25s)", 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.22220000624656677, 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (7s)\n [==..........................] (remaining: 69s)", 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:50.806 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.823+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06baf0 msgr2=0x7fbcfc06dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.823+0000 7fbd2c400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06baf0 0x7fbcfc06dfb0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fbd14007fd0 tx=0x7fbd14009040 comp rx=0 tx=0).stop 2024-01-16T15:06:50.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.823+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2413df80 msgr2=0x7fbd240702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.823+0000 7fbd2c400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2413df80 0x7fbd240702c0 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00c3e0 tx=0x7fbd0c00c8a0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 shutdown_connections 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06baf0 0x7fbcfc06dfb0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2413e360 0x7fbd241fab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2413df80 0x7fbd240702c0 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 --2- 172.21.15.53:0/2271516409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd240cf9a0 0x7fbd2413d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 >> 172.21.15.53:0/2271516409 conn(0x7fbd2406a800 msgr2=0x7fbd240b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 shutdown_connections 2024-01-16T15:06:50.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:50.824+0000 7fbd2c400640 1 -- 172.21.15.53:0/2271516409 wait complete. 2024-01-16T15:06:51.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:51.927 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:06:52.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:53.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.169+0000 7f8b92e00640 1 Processor -- start 2024-01-16T15:06:53.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.170+0000 7f8b92e00640 1 -- start start 2024-01-16T15:06:53.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.173+0000 7f8b92e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8b8c0d0560 0x7f8b8c161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.173+0000 7f8b92e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d2260 0x7f8b8c161570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.173+0000 7f8b92e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c0cf260 con 0x7f8b8c0c9870 2024-01-16T15:06:53.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.173+0000 7f8b92e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c0d2ef0 con 0x7f8b8c0d0560 2024-01-16T15:06:53.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.173+0000 7f8b92e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c9c50 con 0x7f8b8c0d2260 2024-01-16T15:06:53.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.174+0000 7f8b8be00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8b8c0c9870 0x7f8b8c0ceb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:40892/0 (socket says 172.21.15.53:40892) 2024-01-16T15:06:53.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.174+0000 7f8b8be00640 1 -- 172.21.15.53:0/2192120183 learned_addr learned my addr 172.21.15.53:0/2192120183 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:53.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.174+0000 7f8b90e00640 1 --2- 172.21.15.53:0/2192120183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d2260 0x7f8b8c161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.176+0000 7f8b90e00640 1 -- 172.21.15.53:0/2192120183 >> v2:172.21.15.146:3300/0 conn(0x7f8b8c0d0560 msgr2=0x7f8b8c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:53.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.176+0000 7f8b90e00640 1 --2- 172.21.15.53:0/2192120183 >> v2:172.21.15.146:3300/0 conn(0x7f8b8c0d0560 0x7f8b8c161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.176+0000 7f8b90e00640 1 -- 172.21.15.53:0/2192120183 >> v1:172.21.15.146:6789/0 conn(0x7f8b8c0c9870 legacy=0x7f8b8c0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.176+0000 7f8b90e00640 1 -- 172.21.15.53:0/2192120183 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c0cf4c0 con 0x7f8b8c0d2260 2024-01-16T15:06:53.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.178+0000 7f8b90e00640 1 --2- 172.21.15.53:0/2192120183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d2260 0x7f8b8c161570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b740097a0 tx=0x7f8b7402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=927b1946d5870552 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.185+0000 7f8b8aa00640 1 -- 172.21.15.53:0/2192120183 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7402d040 con 0x7f8b8c0d2260 2024-01-16T15:06:53.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.185+0000 7f8b8aa00640 1 -- 172.21.15.53:0/2192120183 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74031810 con 0x7f8b8c0d2260 2024-01-16T15:06:53.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.185+0000 7f8b8aa00640 1 -- 172.21.15.53:0/2192120183 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74031ba0 con 0x7f8b8c0d2260 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 -- 172.21.15.53:0/2192120183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d2260 msgr2=0x7f8b8c161570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 --2- 172.21.15.53:0/2192120183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d2260 0x7f8b8c161570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8b740097a0 tx=0x7f8b7402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 -- 172.21.15.53:0/2192120183 shutdown_connections 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 --2- 172.21.15.53:0/2192120183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d2260 0x7f8b8c161570 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 --2- 172.21.15.53:0/2192120183 >> v2:172.21.15.146:3300/0 conn(0x7f8b8c0d0560 0x7f8b8c161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 -- 172.21.15.53:0/2192120183 >> 172.21.15.53:0/2192120183 conn(0x7f8b8c09c560 msgr2=0x7f8b8c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 -- 172.21.15.53:0/2192120183 shutdown_connections 2024-01-16T15:06:53.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.186+0000 7f8b92e00640 1 -- 172.21.15.53:0/2192120183 wait complete. 2024-01-16T15:06:53.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.189+0000 7f8b92e00640 1 Processor -- start 2024-01-16T15:06:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.189+0000 7f8b92e00640 1 -- start start 2024-01-16T15:06:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.190+0000 7f8b92e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 0x7f8b8c0700c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.190+0000 7f8b92e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d0560 0x7f8b8c15dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.190+0000 7f8b92e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b8c0d2260 0x7f8b8c06b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.190+0000 7f8b92e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0714d0 con 0x7f8b8c0d2260 2024-01-16T15:06:53.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.190+0000 7f8b92e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c15ed20 con 0x7f8b8c0c9870 2024-01-16T15:06:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.190+0000 7f8b92e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c070600 con 0x7f8b8c0d0560 2024-01-16T15:06:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.191+0000 7f8b8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 0x7f8b8c0700c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.191+0000 7f8b90e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b8c0d2260 0x7f8b8c06b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.191+0000 7f8b8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 0x7f8b8c0700c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:57894/0 (socket says 172.21.15.53:57894) 2024-01-16T15:06:53.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.191+0000 7f8b8be00640 1 -- 172.21.15.53:0/908377859 learned_addr learned my addr 172.21.15.53:0/908377859 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:53.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.192+0000 7f8b90e00640 1 -- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d0560 msgr2=0x7f8b8c15dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.192+0000 7f8b90e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d0560 0x7f8b8c15dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.192+0000 7f8b90e00640 1 -- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 msgr2=0x7f8b8c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.192+0000 7f8b90e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 0x7f8b8c0700c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.192+0000 7f8b90e00640 1 -- 172.21.15.53:0/908377859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b74009400 con 0x7f8b8c0d2260 2024-01-16T15:06:53.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.192+0000 7f8b8be00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 0x7f8b8c0700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.193+0000 7f8b90e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b8c0d2260 0x7f8b8c06b4b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c06ac70 tx=0x7f8b740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.193+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7403ae60 con 0x7f8b8c0d2260 2024-01-16T15:06:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.193+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b74045450 con 0x7f8b8c0d2260 2024-01-16T15:06:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.193+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c0708c0 con 0x7f8b8c0d2260 2024-01-16T15:06:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.193+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b74045800 con 0x7f8b8c0d2260 2024-01-16T15:06:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.193+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c070e80 con 0x7f8b8c0d2260 2024-01-16T15:06:53.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.195+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c0675d0 con 0x7f8b8c0d2260 2024-01-16T15:06:53.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.196+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f8b74045960 con 0x7f8b8c0d2260 2024-01-16T15:06:53.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.199+0000 7f8b83e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8b6406bb30 0x7f8b6406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:53.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.199+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b740aa940 con 0x7f8b8c0d2260 2024-01-16T15:06:53.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.203+0000 7f8b8be00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8b6406bb30 0x7f8b6406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:53.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.206+0000 7f8b8be00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8b6406bb30 0x7f8b6406dff0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8b840012e0 tx=0x7f8b84001a20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:53.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.207+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8b74077200 con 0x7f8b8c0d2260 2024-01-16T15:06:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.711+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8b8c0d0a20 con 0x7f8b8c0d2260 2024-01-16T15:06:53.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.712+0000 7f8b83e00640 1 -- 172.21.15.53:0/908377859 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7f8b7407a820 con 0x7f8b8c0d2260 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:53.690 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 440, 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:53.691 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:53.692 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:53.693 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1816, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 7471437559, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 14549721088, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 758544392192, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 365793030, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 353949801, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 88, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 95 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:53.694 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:53.695 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:53.696 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.335% (10s)\n [=========...................] (remaining: 21s)", 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.33335000276565552, 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:53.697 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.049% (10s)\n [=====.......................] (remaining: 45s)", 2024-01-16T15:06:53.698 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.19048571586608887, 2024-01-16T15:06:53.698 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:53.698 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:53.698 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:53.698 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8b6406bb30 msgr2=0x7f8b6406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8b6406bb30 0x7f8b6406dff0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8b840012e0 tx=0x7f8b84001a20 comp rx=0 tx=0).stop 2024-01-16T15:06:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b8c0d2260 msgr2=0x7f8b8c06b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b8c0d2260 0x7f8b8c06b4b0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f8b8c06ac70 tx=0x7f8b740046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 shutdown_connections 2024-01-16T15:06:53.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8b6406bb30 0x7f8b6406dff0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8b8c0d2260 0x7f8b8c06b4b0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8b8c0d0560 0x7f8b8c15dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 --2- 172.21.15.53:0/908377859 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8b8c0c9870 0x7f8b8c0700c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.714+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 >> 172.21.15.53:0/908377859 conn(0x7f8b8c09c560 msgr2=0x7f8b8c160f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.715+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 shutdown_connections 2024-01-16T15:06:53.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:53.715+0000 7f8b92e00640 1 -- 172.21.15.53:0/908377859 wait complete. 2024-01-16T15:06:55.717 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:56.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.047+0000 7fdf3e800640 1 Processor -- start 2024-01-16T15:06:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.049+0000 7fdf3e800640 1 -- start start 2024-01-16T15:06:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf3e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 0x7fdf38161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf3e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdf380ced50 0x7fdf380d02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf3e800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf380cf1b0 con 0x7fdf380ce8f0 2024-01-16T15:06:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf36a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdf380ce8f0 0x7fdf380cfbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36948/0 (socket says 172.21.15.53:36948) 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf3e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf381618a0 con 0x7fdf380ced50 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf3e800640 1 -- 172.21.15.53:0/2035655525 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380d0de0 con 0x7fdf38160f80 2024-01-16T15:06:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.052+0000 7fdf36a00640 1 -- 172.21.15.53:0/2035655525 learned_addr learned my addr 172.21.15.53:0/2035655525 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:56.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.053+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162885166 0 0) 0x7fdf380cf1b0 con 0x7fdf380ce8f0 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.053+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf18003520 con 0x7fdf380ce8f0 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839527535 0 0) 0x7fdf18003520 con 0x7fdf380ce8f0 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 msgr2=0x7fdf38161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 0x7fdf38161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 >> v2:172.21.15.146:3300/0 conn(0x7fdf380ced50 msgr2=0x7fdf380d02f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:06:56.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 --2- 172.21.15.53:0/2035655525 >> v2:172.21.15.146:3300/0 conn(0x7fdf380ced50 0x7fdf380d02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380d10c0 con 0x7fdf380ce8f0 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fdf20001670 con 0x7fdf380ce8f0 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.054+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 >> v1:172.21.15.53:6789/0 conn(0x7fdf380ce8f0 legacy=0x7fdf380cfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.055+0000 7fdf36000640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf18003970 0x7fdf18003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.055+0000 7fdf36000640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf180047c0 0x7fdf18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.055+0000 7fdf36000640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 0x7fdf180051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.055+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf381618a0 con 0x7fdf180047c0 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.055+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380d0de0 con 0x7fdf18003970 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.055+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1800bb60 con 0x7fdf38160f80 2024-01-16T15:06:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.056+0000 7fdf36a00640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf180047c0 0x7fdf18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.057+0000 7fdf37400640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf18003970 0x7fdf18003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.057+0000 7fdf37e00640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 0x7fdf180051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.059+0000 7fdf37400640 1 -- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 msgr2=0x7fdf180051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.059+0000 7fdf37400640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf38160f80 0x7fdf180051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.059+0000 7fdf37400640 1 -- 172.21.15.53:0/2035655525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf180047c0 msgr2=0x7fdf18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.059+0000 7fdf37400640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf180047c0 0x7fdf18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.059+0000 7fdf37400640 1 -- 172.21.15.53:0/2035655525 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf18003520 con 0x7fdf18003970 2024-01-16T15:06:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.059+0000 7fdf36a00640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf180047c0 0x7fdf18004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:06:56.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.063+0000 7fdf37400640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf18003970 0x7fdf18003d50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009860 tx=0x7fdf180057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d96eb3d9800bb8e4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.067+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28032600 con 0x7fdf18003970 2024-01-16T15:06:56.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.067+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2802e050 con 0x7fdf18003970 2024-01-16T15:06:56.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.067+0000 7fdf36000640 1 -- 172.21.15.53:0/2035655525 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf280329c0 con 0x7fdf18003970 2024-01-16T15:06:56.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.069+0000 7fdf3e800640 1 -- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf18003970 msgr2=0x7fdf18003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.069+0000 7fdf3e800640 1 --2- 172.21.15.53:0/2035655525 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf18003970 0x7fdf18003d50 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fdf28009860 tx=0x7fdf180057f0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.069+0000 7fdf37e00640 1 -- 172.21.15.53:0/2035655525 reap_dead start 2024-01-16T15:06:56.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.072+0000 7fdf3e800640 1 -- 172.21.15.53:0/2035655525 shutdown_connections 2024-01-16T15:06:56.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.072+0000 7fdf3e800640 1 -- 172.21.15.53:0/2035655525 >> 172.21.15.53:0/2035655525 conn(0x7fdf3809c550 msgr2=0x7fdf380d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.072+0000 7fdf3e800640 1 -- 172.21.15.53:0/2035655525 shutdown_connections 2024-01-16T15:06:56.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.073+0000 7fdf3e800640 1 -- 172.21.15.53:0/2035655525 wait complete. 2024-01-16T15:06:56.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.076+0000 7fdf3e800640 1 Processor -- start 2024-01-16T15:06:56.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.076+0000 7fdf3e800640 1 -- start start 2024-01-16T15:06:56.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.077+0000 7fdf3e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 0x7fdf3815c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.077+0000 7fdf3e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf3815b740 0x7fdf380ccaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.077+0000 7fdf3e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf380b18a0 0x7fdf380b1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.077+0000 7fdf3e800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf380b2260 con 0x7fdf18003970 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.077+0000 7fdf3e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3815ce20 con 0x7fdf380b18a0 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.077+0000 7fdf3e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf3815bf90 con 0x7fdf3815b740 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.078+0000 7fdf37400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 0x7fdf3815c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.078+0000 7fdf37400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 0x7fdf3815c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54152/0 (socket says 172.21.15.53:54152) 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.078+0000 7fdf37400640 1 -- 172.21.15.53:0/1390921541 learned_addr learned my addr 172.21.15.53:0/1390921541 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:56.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.078+0000 7fdf37e00640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf380b18a0 0x7fdf380b1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.079+0000 7fdf37400640 1 -- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf3815b740 msgr2=0x7fdf380ccaa0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.079+0000 7fdf37400640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf3815b740 0x7fdf380ccaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.079+0000 7fdf37400640 1 -- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf380b18a0 msgr2=0x7fdf380b1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.079+0000 7fdf37400640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf380b18a0 0x7fdf380b1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.079+0000 7fdf37400640 1 -- 172.21.15.53:0/1390921541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf380ccfe0 con 0x7fdf18003970 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.079+0000 7fdf37e00640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf380b18a0 0x7fdf380b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:06:56.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.080+0000 7fdf37400640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 0x7fdf3815c790 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000c00 tx=0x7fdf280047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.080+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2803e630 con 0x7fdf18003970 2024-01-16T15:06:56.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.080+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf2803e790 con 0x7fdf18003970 2024-01-16T15:06:56.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.080+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf2803eb40 con 0x7fdf18003970 2024-01-16T15:06:56.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.080+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf380c6e70 con 0x7fdf18003970 2024-01-16T15:06:56.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.080+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf380c7430 con 0x7fdf18003970 2024-01-16T15:06:56.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.082+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38071930 con 0x7fdf18003970 2024-01-16T15:06:56.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.085+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fdf2803eca0 con 0x7fdf18003970 2024-01-16T15:06:56.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.086+0000 7fdf2f400640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf1006bb30 0x7fdf1006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:56.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.086+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf280aa7d0 con 0x7fdf18003970 2024-01-16T15:06:56.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.091+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdf2807aa40 con 0x7fdf18003970 2024-01-16T15:06:56.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.128+0000 7fdf36a00640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf1006bb30 0x7fdf1006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:56.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.149+0000 7fdf36a00640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf1006bb30 0x7fdf1006dff0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdf20000c00 tx=0x7fdf20008ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:56.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.584+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fdf380675d0 con 0x7fdf18003970 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.586+0000 7fdf2f400640 1 -- 172.21.15.53:0/1390921541 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fdf2804b990 con 0x7fdf18003970 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:56.564 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 443, 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:56.565 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:56.566 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1816, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 7471437559, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 14549721088, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 758544392192, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 365791221, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 353934730, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 87, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 92 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:56.567 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:56.568 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:56.569 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 17s)", 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.41667500138282776, 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 39s)", 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.23810000717639923, 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:56.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.591+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf1006bb30 msgr2=0x7fdf1006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.591+0000 7fdf3e800640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf1006bb30 0x7fdf1006dff0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdf20000c00 tx=0x7fdf20008ca0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.591+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 msgr2=0x7fdf3815c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.591+0000 7fdf3e800640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 0x7fdf3815c790 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fdf28000c00 tx=0x7fdf280047d0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.592+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 shutdown_connections 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.592+0000 7fdf3e800640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdf1006bb30 0x7fdf1006dff0 secure :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fdf20000c00 tx=0x7fdf20008ca0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.592+0000 7fdf3e800640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdf380b18a0 0x7fdf380b1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.592+0000 7fdf3e800640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdf3815b740 0x7fdf380ccaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.592+0000 7fdf3e800640 1 --2- 172.21.15.53:0/1390921541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdf18003970 0x7fdf3815c790 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.592+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 >> 172.21.15.53:0/1390921541 conn(0x7fdf3809c550 msgr2=0x7fdf380b3940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:56.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.593+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 shutdown_connections 2024-01-16T15:06:56.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:56.594+0000 7fdf3e800640 1 -- 172.21.15.53:0/1390921541 wait complete. 2024-01-16T15:06:58.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:06:58.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.869+0000 7fb2b6400640 1 Processor -- start 2024-01-16T15:06:58.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.870+0000 7fb2b6400640 1 -- start start 2024-01-16T15:06:58.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2b6400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2b6400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb2b015e1e0 0x7fb2b00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2b6400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b00b1ee0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2b6400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b00b2780 con 0x7fb2b015e1e0 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2b6400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b015e990 con 0x7fb2b00cf9a0 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2af400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb2b0143ae0 0x7fb2b00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:40908/0 (socket says 172.21.15.53:40908) 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2af400640 1 -- 172.21.15.53:0/3724412493 learned_addr learned my addr 172.21.15.53:0/3724412493 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2aea00640 1 --2- 172.21.15.53:0/3724412493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.872+0000 7fb2afe00640 1 --2- 172.21.15.53:0/3724412493 >> v2:172.21.15.146:3300/0 conn(0x7fb2b015e1e0 0x7fb2b00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.873+0000 7fb2aea00640 1 -- 172.21.15.53:0/3724412493 >> v2:172.21.15.146:3300/0 conn(0x7fb2b015e1e0 msgr2=0x7fb2b00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:06:58.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.873+0000 7fb2ae000640 1 -- 172.21.15.53:0/3724412493 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429244762 0 0) 0x7fb2b00b1ee0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.873+0000 7fb2aea00640 1 --2- 172.21.15.53:0/3724412493 >> v2:172.21.15.146:3300/0 conn(0x7fb2b015e1e0 0x7fb2b00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.873+0000 7fb2aea00640 1 -- 172.21.15.53:0/3724412493 >> v1:172.21.15.146:6789/0 conn(0x7fb2b0143ae0 legacy=0x7fb2b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.873+0000 7fb2aea00640 1 -- 172.21.15.53:0/3724412493 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b00b2130 con 0x7fb2b00cf9a0 2024-01-16T15:06:58.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.874+0000 7fb2aea00640 1 --2- 172.21.15.53:0/3724412493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b015dca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2980097a0 tx=0x7fb29802dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb22f02378fee54 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.875+0000 7fb2ae000640 1 -- 172.21.15.53:0/3724412493 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29802f8a0 con 0x7fb2b00cf9a0 2024-01-16T15:06:58.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.875+0000 7fb2ae000640 1 -- 172.21.15.53:0/3724412493 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb29802fa00 con 0x7fb2b00cf9a0 2024-01-16T15:06:58.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.875+0000 7fb2ae000640 1 -- 172.21.15.53:0/3724412493 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29802fd90 con 0x7fb2b00cf9a0 2024-01-16T15:06:58.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.875+0000 7fb2b6400640 1 -- 172.21.15.53:0/3724412493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 msgr2=0x7fb2b015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.875+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3724412493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b015dca0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb2980097a0 tx=0x7fb29802dae0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.876+0000 7fb2b6400640 1 -- 172.21.15.53:0/3724412493 shutdown_connections 2024-01-16T15:06:58.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.876+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3724412493 >> v2:172.21.15.146:3300/0 conn(0x7fb2b015e1e0 0x7fb2b00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.876+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3724412493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b015dca0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.876+0000 7fb2b6400640 1 -- 172.21.15.53:0/3724412493 >> 172.21.15.53:0/3724412493 conn(0x7fb2b006a800 msgr2=0x7fb2b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:58.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.876+0000 7fb2b6400640 1 -- 172.21.15.53:0/3724412493 shutdown_connections 2024-01-16T15:06:58.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.876+0000 7fb2b6400640 1 -- 172.21.15.53:0/3724412493 wait complete. 2024-01-16T15:06:58.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.878+0000 7fb2b6400640 1 Processor -- start 2024-01-16T15:06:58.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.878+0000 7fb2b6400640 1 -- start start 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.879+0000 7fb2b6400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.879+0000 7fb2b6400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 0x7fb2b015a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.879+0000 7fb2b6400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b015e1e0 0x7fb2b015aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.879+0000 7fb2b6400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b00cde20 con 0x7fb2b0143ae0 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.879+0000 7fb2b6400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b00a9320 con 0x7fb2b015e1e0 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.879+0000 7fb2b6400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b00cd4f0 con 0x7fb2b00cf9a0 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 0x7fb2b015a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 0x7fb2b015a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54158/0 (socket says 172.21.15.53:54158) 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 -- 172.21.15.53:0/3684285993 learned_addr learned my addr 172.21.15.53:0/3684285993 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2afe00640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b015e1e0 0x7fb2b015aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2af400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 -- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 msgr2=0x7fb2b0159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b0159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 -- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b015e1e0 msgr2=0x7fb2b015aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b015e1e0 0x7fb2b015aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.880+0000 7fb2aea00640 1 -- 172.21.15.53:0/3684285993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb298009400 con 0x7fb2b0143ae0 2024-01-16T15:06:58.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.881+0000 7fb2aea00640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 0x7fb2b015a4e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb29802e0f0 tx=0x7fb2980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.881+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298030ab0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.881+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb298030c30 con 0x7fb2b0143ae0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.881+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb298042410 con 0x7fb2b0143ae0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.881+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b00cd7d0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.881+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b00ce2b0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.882+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b00cdfd0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.883+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb298039400 con 0x7fb2b0143ae0 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.884+0000 7fb2a7400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bb60 0x7fb28806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.884+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2980aa1d0 con 0x7fb2b0143ae0 2024-01-16T15:06:58.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.885+0000 7fb2af400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bb60 0x7fb28806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:06:58.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.885+0000 7fb2af400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bb60 0x7fb28806e020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0000f60 tx=0x7fb2a0007d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:06:58.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:58.889+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2b00cdfd0 con 0x7fb2b0143ae0 2024-01-16T15:06:59.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.155+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb2b00675d0 con 0x7fb2b0143ae0 2024-01-16T15:06:59.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.157+0000 7fb2a7400640 1 -- 172.21.15.53:0/3684285993 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2429 (secure 0 0 0) 0x7fb298047090 con 0x7fb2b0143ae0 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:06:59.135 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 446, 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:06:59.136 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:06:59.137 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 2432, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 10055128928, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 19337007104, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 753757106176, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 579756572, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 568952065, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 139, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 144 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:06:59.138 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:06:59.139 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:06:59.140 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 41.668% (12s)\n [===========.................] (remaining: 17s)", 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.41667500138282776, 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 23.81% (12s)\n [======......................] (remaining: 39s)", 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.23810000717639923, 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:06:59.141 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.159+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bb60 msgr2=0x7fb28806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bb60 0x7fb28806e020 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0000f60 tx=0x7fb2a0007d50 comp rx=0 tx=0).stop 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 msgr2=0x7fb2b015a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 0x7fb2b015a4e0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb29802e0f0 tx=0x7fb2980046c0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 shutdown_connections 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bb60 0x7fb28806e020 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b015e1e0 0x7fb2b015aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b0143ae0 0x7fb2b015a4e0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 --2- 172.21.15.53:0/3684285993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b00cf9a0 0x7fb2b0159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:06:59.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 >> 172.21.15.53:0/3684285993 conn(0x7fb2b006a800 msgr2=0x7fb2b00cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:06:59.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 shutdown_connections 2024-01-16T15:06:59.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:06:59.160+0000 7fb2b6400640 1 -- 172.21.15.53:0/3684285993 wait complete. 2024-01-16T15:07:01.159 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:07:01.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.480+0000 7eff4ce00640 1 Processor -- start 2024-01-16T15:07:01.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.481+0000 7eff4ce00640 1 -- start start 2024-01-16T15:07:01.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.484+0000 7eff4ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4815bfa0 0x7eff480cec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.484+0000 7eff4ce00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff4806d2b0 0x7eff481657e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.484+0000 7eff4ce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff48165350 con 0x7eff480d0ec0 2024-01-16T15:07:01.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.484+0000 7eff4ce00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff4806b590 con 0x7eff4806d2b0 2024-01-16T15:07:01.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.484+0000 7eff4ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff4806b800 con 0x7eff4815bfa0 2024-01-16T15:07:01.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.485+0000 7eff46800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff4806d2b0 0x7eff481657e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.486+0000 7eff46800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7eff4806d2b0 0x7eff481657e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54168/0 (socket says 172.21.15.53:54168) 2024-01-16T15:07:01.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.486+0000 7eff46800640 1 -- 172.21.15.53:0/500612326 learned_addr learned my addr 172.21.15.53:0/500612326 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:01.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.489+0000 7eff46800640 1 -- 172.21.15.53:0/500612326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4815bfa0 msgr2=0x7eff480cec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:01.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.489+0000 7eff46800640 1 --2- 172.21.15.53:0/500612326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4815bfa0 0x7eff480cec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.489+0000 7eff46800640 1 -- 172.21.15.53:0/500612326 >> v1:172.21.15.146:6789/0 conn(0x7eff480d0ec0 legacy=0x7eff480d12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:01.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.489+0000 7eff46800640 1 -- 172.21.15.53:0/500612326 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4806bae0 con 0x7eff4806d2b0 2024-01-16T15:07:01.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.491+0000 7eff46800640 1 --2- 172.21.15.53:0/500612326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481657e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7eff30009870 tx=0x7eff3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ccd0a103f6dc6a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.492+0000 7eff44a00640 1 -- 172.21.15.53:0/500612326 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30031640 con 0x7eff4806d2b0 2024-01-16T15:07:01.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.492+0000 7eff44a00640 1 -- 172.21.15.53:0/500612326 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff300317a0 con 0x7eff4806d2b0 2024-01-16T15:07:01.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.492+0000 7eff44a00640 1 -- 172.21.15.53:0/500612326 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30031b30 con 0x7eff4806d2b0 2024-01-16T15:07:01.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.493+0000 7eff4ce00640 1 -- 172.21.15.53:0/500612326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 msgr2=0x7eff481657e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.493+0000 7eff4ce00640 1 --2- 172.21.15.53:0/500612326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481657e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7eff30009870 tx=0x7eff3002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.493+0000 7eff4ce00640 1 -- 172.21.15.53:0/500612326 shutdown_connections 2024-01-16T15:07:01.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.493+0000 7eff4ce00640 1 --2- 172.21.15.53:0/500612326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481657e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.494+0000 7eff4ce00640 1 --2- 172.21.15.53:0/500612326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff4815bfa0 0x7eff480cec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.494+0000 7eff4ce00640 1 -- 172.21.15.53:0/500612326 >> 172.21.15.53:0/500612326 conn(0x7eff480b2ce0 msgr2=0x7eff480b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:01.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.494+0000 7eff4ce00640 1 -- 172.21.15.53:0/500612326 shutdown_connections 2024-01-16T15:07:01.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.494+0000 7eff4ce00640 1 -- 172.21.15.53:0/500612326 wait complete. 2024-01-16T15:07:01.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.497+0000 7eff4ce00640 1 Processor -- start 2024-01-16T15:07:01.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.497+0000 7eff4ce00640 1 -- start start 2024-01-16T15:07:01.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.498+0000 7eff4ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481f8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.498+0000 7eff4ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff480d0ec0 0x7eff481469b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.498+0000 7eff4ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff480c5f50 0x7eff480c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.499+0000 7eff4ce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff480c6940 con 0x7eff4806d2b0 2024-01-16T15:07:01.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.499+0000 7eff4ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff480b4c60 con 0x7eff480d0ec0 2024-01-16T15:07:01.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.499+0000 7eff4ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff480b3db0 con 0x7eff480c5f50 2024-01-16T15:07:01.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.500+0000 7eff45e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481f8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.500+0000 7eff45e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481f8920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54182/0 (socket says 172.21.15.53:54182) 2024-01-16T15:07:01.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.500+0000 7eff45e00640 1 -- 172.21.15.53:0/2322690637 learned_addr learned my addr 172.21.15.53:0/2322690637 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:01.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.500+0000 7eff45400640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff480d0ec0 0x7eff481469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.501+0000 7eff45e00640 1 -- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff480c5f50 msgr2=0x7eff480c6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.501+0000 7eff45e00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff480c5f50 0x7eff480c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.501+0000 7eff45e00640 1 -- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff480d0ec0 msgr2=0x7eff481469b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.501+0000 7eff45e00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff480d0ec0 0x7eff481469b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.501+0000 7eff45e00640 1 -- 172.21.15.53:0/2322690637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff300094d0 con 0x7eff4806d2b0 2024-01-16T15:07:01.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.501+0000 7eff45e00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481f8920 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c440 tx=0x7eff3800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.502+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff38005480 con 0x7eff4806d2b0 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.502+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff380035f0 con 0x7eff4806d2b0 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.502+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff480b4050 con 0x7eff4806d2b0 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.502+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff380039c0 con 0x7eff4806d2b0 2024-01-16T15:07:01.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.502+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff480b4610 con 0x7eff4806d2b0 2024-01-16T15:07:01.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.504+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff480675d0 con 0x7eff4806d2b0 2024-01-16T15:07:01.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.509+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7eff38003b20 con 0x7eff4806d2b0 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.510+0000 7eff3e000640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2006bc00 0x7eff2006e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:01.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.510+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7eff38087fd0 con 0x7eff4806d2b0 2024-01-16T15:07:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.514+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7eff38054770 con 0x7eff4806d2b0 2024-01-16T15:07:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.521+0000 7eff45400640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2006bc00 0x7eff2006e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.523+0000 7eff45400640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2006bc00 0x7eff2006e0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7eff480cb780 tx=0x7eff4000d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:01.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:01.999+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7eff48158ab0 con 0x7eff4806d2b0 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:07:01.978 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 449, 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.979 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:07:01.980 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 2955, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 12232092213, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 23795085312, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 749299027968, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 574154040, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 573478919, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 137, 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 145 2024-01-16T15:07:01.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:01.982 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:07:01.983 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "2 ongoing cloning jobs; Avg progress = 50.0% (18s)\n [==============..............] (remaining: 18s)", 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.5, 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 42.857% (18s)\n [============................] (remaining: 24s)", 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.4285714328289032, 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:01.984 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.000+0000 7eff3e000640 1 -- 172.21.15.53:0/2322690637 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2411 (secure 0 0 0) 0x7eff38017030 con 0x7eff4806d2b0 2024-01-16T15:07:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2006bc00 msgr2=0x7eff2006e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2006bc00 0x7eff2006e0c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7eff480cb780 tx=0x7eff4000d040 comp rx=0 tx=0).stop 2024-01-16T15:07:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 msgr2=0x7eff481f8920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481f8920 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7eff3800c440 tx=0x7eff3800c900 comp rx=0 tx=0).stop 2024-01-16T15:07:01.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 shutdown_connections 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7eff2006bc00 0x7eff2006e0c0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7eff480c5f50 0x7eff480c6400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7eff480d0ec0 0x7eff481469b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 --2- 172.21.15.53:0/2322690637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7eff4806d2b0 0x7eff481f8920 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.005+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 >> 172.21.15.53:0/2322690637 conn(0x7eff480b2ce0 msgr2=0x7eff480b30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.006+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 shutdown_connections 2024-01-16T15:07:01.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:02.006+0000 7eff4ce00640 1 -- 172.21.15.53:0/2322690637 wait complete. 2024-01-16T15:07:04.016 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:07:04.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.339+0000 7f095c400640 1 Processor -- start 2024-01-16T15:07:04.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.340+0000 7f095c400640 1 -- start start 2024-01-16T15:07:04.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.341+0000 7f095c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f09540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.341+0000 7f095c400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f095415e3b0 0x7f09540702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.341+0000 7f095c400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09540cfc30 con 0x7f09540cf7d0 2024-01-16T15:07:04.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.341+0000 7f095c400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09540b1ee0 con 0x7f095415e3b0 2024-01-16T15:07:04.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.341+0000 7f095c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09540b2780 con 0x7f0954143ae0 2024-01-16T15:07:04.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f09540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f09540cf7d0 0x7f095415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:51576/0 (socket says 172.21.15.53:51576) 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959000640 1 -- 172.21.15.53:0/3315829265 learned_addr learned my addr 172.21.15.53:0/3315829265 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f095a400640 1 --2- 172.21.15.53:0/3315829265 >> v2:172.21.15.146:3300/0 conn(0x7f095415e3b0 0x7f09540702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959a00640 1 -- 172.21.15.53:0/3315829265 >> v2:172.21.15.146:3300/0 conn(0x7f095415e3b0 msgr2=0x7f09540702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959a00640 1 --2- 172.21.15.53:0/3315829265 >> v2:172.21.15.146:3300/0 conn(0x7f095415e3b0 0x7f09540702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959a00640 1 -- 172.21.15.53:0/3315829265 >> v1:172.21.15.146:6789/0 conn(0x7f09540cf7d0 legacy=0x7f095415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0959a00640 1 -- 172.21.15.53:0/3315829265 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095415e990 con 0x7f0954143ae0 2024-01-16T15:07:04.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.342+0000 7f0953e00640 1 -- 172.21.15.53:0/3315829265 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645870527 0 0) 0x7f09540cfc30 con 0x7f09540cf7d0 2024-01-16T15:07:04.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.344+0000 7f0959a00640 1 --2- 172.21.15.53:0/3315829265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f09540cf290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f09440097a0 tx=0x7f094402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edc348ba28adf9c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.344+0000 7f0953e00640 1 -- 172.21.15.53:0/3315829265 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094402d040 con 0x7f0954143ae0 2024-01-16T15:07:04.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.344+0000 7f0953e00640 1 -- 172.21.15.53:0/3315829265 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0944031810 con 0x7f0954143ae0 2024-01-16T15:07:04.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.344+0000 7f0953e00640 1 -- 172.21.15.53:0/3315829265 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0944031ba0 con 0x7f0954143ae0 2024-01-16T15:07:04.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 -- 172.21.15.53:0/3315829265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 msgr2=0x7f09540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 --2- 172.21.15.53:0/3315829265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f09540cf290 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f09440097a0 tx=0x7f094402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 -- 172.21.15.53:0/3315829265 shutdown_connections 2024-01-16T15:07:04.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 --2- 172.21.15.53:0/3315829265 >> v2:172.21.15.146:3300/0 conn(0x7f095415e3b0 0x7f09540702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 --2- 172.21.15.53:0/3315829265 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f09540cf290 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 -- 172.21.15.53:0/3315829265 >> 172.21.15.53:0/3315829265 conn(0x7f095406a800 msgr2=0x7f095406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:04.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 -- 172.21.15.53:0/3315829265 shutdown_connections 2024-01-16T15:07:04.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.345+0000 7f095c400640 1 -- 172.21.15.53:0/3315829265 wait complete. 2024-01-16T15:07:04.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.347+0000 7f095c400640 1 Processor -- start 2024-01-16T15:07:04.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.347+0000 7f095c400640 1 -- start start 2024-01-16T15:07:04.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.348+0000 7f095c400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540cf7d0 0x7f095413d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.348+0000 7f095c400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f095415aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.348+0000 7f095c400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 0x7f095415af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.348+0000 7f095c400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095413cc90 con 0x7f095415e3b0 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.348+0000 7f095c400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f095413c080 con 0x7f09540cf7d0 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.348+0000 7f095c400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f095413c360 con 0x7f0954143ae0 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 0x7f095415af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 0x7f095415af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47666/0 (socket says 172.21.15.53:47666) 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 -- 172.21.15.53:0/73787670 learned_addr learned my addr 172.21.15.53:0/73787670 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:04.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f0959a00640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540cf7d0 0x7f095413d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f0959000640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f095415aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 -- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 msgr2=0x7f095415aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f095415aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 -- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540cf7d0 msgr2=0x7f095413d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540cf7d0 0x7f095413d120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f095a400640 1 -- 172.21.15.53:0/73787670 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0944009400 con 0x7f095415e3b0 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f0959000640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f095415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.349+0000 7f0959a00640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540cf7d0 0x7f095413d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.350+0000 7f095a400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 0x7f095415af70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f094c00efc0 tx=0x7f094c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.350+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c009ba0 con 0x7f095415e3b0 2024-01-16T15:07:04.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.350+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f094c00f040 con 0x7f095415e3b0 2024-01-16T15:07:04.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.350+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095413c640 con 0x7f095415e3b0 2024-01-16T15:07:04.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.350+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f094c0146c0 con 0x7f095415e3b0 2024-01-16T15:07:04.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.350+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f095413d660 con 0x7f095415e3b0 2024-01-16T15:07:04.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.352+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095413ce00 con 0x7f095415e3b0 2024-01-16T15:07:04.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.354+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f094c014820 con 0x7f095415e3b0 2024-01-16T15:07:04.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.355+0000 7f0951600640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f092c06bb30 0x7f092c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:04.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.356+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f094c089220 con 0x7f095415e3b0 2024-01-16T15:07:04.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.359+0000 7f0959a00640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f092c06bb30 0x7f092c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:04.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.361+0000 7f0959a00640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f092c06bb30 0x7f092c06dff0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0944036000 tx=0x7f094403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:04.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.364+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f094c055890 con 0x7f095415e3b0 2024-01-16T15:07:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.822+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f09540b1c20 con 0x7f095415e3b0 2024-01-16T15:07:04.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.823+0000 7f0951600640 1 -- 172.21.15.53:0/73787670 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2428 (secure 0 0 0) 0x7f094c018990 con 0x7f095415e3b0 2024-01-16T15:07:04.802 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:04.802 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:07:04.803 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 451, 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:07:04.804 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:07:04.805 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 3242, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 13431863122, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 26678620160, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 746415493120, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:07:04.806 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 575209429, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 576647511, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 138, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 149 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:07:04.807 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:07:04.808 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:07:04.809 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "2 ongoing cloning jobs; Avg progress = 66.67% (21s)\n [==================..........] (remaining: 10s)", 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.66670000553131104, 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 47.62% (21s)\n [=============...............] (remaining: 23s)", 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.47620001435279846, 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:04.810 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:04.811 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.828+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f092c06bb30 msgr2=0x7f092c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.828+0000 7f095c400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f092c06bb30 0x7f092c06dff0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f0944036000 tx=0x7f094403c040 comp rx=0 tx=0).stop 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.828+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 msgr2=0x7f095415af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.828+0000 7f095c400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 0x7f095415af70 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f094c00efc0 tx=0x7f094c00c490 comp rx=0 tx=0).stop 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 shutdown_connections 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f092c06bb30 0x7f092c06dff0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f095415e3b0 0x7f095415af70 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0954143ae0 0x7f095415aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 --2- 172.21.15.53:0/73787670 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09540cf7d0 0x7f095413d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:04.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 >> 172.21.15.53:0/73787670 conn(0x7f095406a800 msgr2=0x7f09540b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 shutdown_connections 2024-01-16T15:07:04.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:04.829+0000 7f095c400640 1 -- 172.21.15.53:0/73787670 wait complete. 2024-01-16T15:07:06.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:07:07.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.080+0000 7f02a9c00640 1 Processor -- start 2024-01-16T15:07:07.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.080+0000 7f02a9c00640 1 -- start start 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.082+0000 7f02a9c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02a4143c40 0x7f02a4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.082+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40ccc10 0x7f02a406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.082+0000 7f02a9c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a4070e50 con 0x7f02a40717d0 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.082+0000 7f02a9c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a40723c0 con 0x7f02a4143c40 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.082+0000 7f02a9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4144020 con 0x7f02a40ccc10 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.082+0000 7f02a3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02a4143c40 0x7f02a4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02a4143c40 0x7f02a4071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47676/0 (socket says 172.21.15.53:47676) 2024-01-16T15:07:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a3e00640 1 -- 172.21.15.53:0/3727383864 learned_addr learned my addr 172.21.15.53:0/3727383864 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a9200640 1 --2- 172.21.15.53:0/3727383864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40ccc10 0x7f02a406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a3e00640 1 -- 172.21.15.53:0/3727383864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40ccc10 msgr2=0x7f02a406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a3e00640 1 --2- 172.21.15.53:0/3727383864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40ccc10 0x7f02a406d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a3e00640 1 -- 172.21.15.53:0/3727383864 >> v1:172.21.15.146:6789/0 conn(0x7f02a40717d0 legacy=0x7f02a40cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.083+0000 7f02a3e00640 1 -- 172.21.15.53:0/3727383864 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a4071e00 con 0x7f02a4143c40 2024-01-16T15:07:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.084+0000 7f02a3e00640 1 --2- 172.21.15.53:0/3727383864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a4143c40 0x7f02a4071290 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f029402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab60fedcac773e38 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a2a00640 1 -- 172.21.15.53:0/3727383864 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02940317c0 con 0x7f02a4143c40 2024-01-16T15:07:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a2a00640 1 -- 172.21.15.53:0/3727383864 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0294031920 con 0x7f02a4143c40 2024-01-16T15:07:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a2a00640 1 -- 172.21.15.53:0/3727383864 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0294031cb0 con 0x7f02a4143c40 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 -- 172.21.15.53:0/3727383864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a4143c40 msgr2=0x7f02a4071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 --2- 172.21.15.53:0/3727383864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a4143c40 0x7f02a4071290 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f0294009870 tx=0x7f029402b860 comp rx=0 tx=0).stop 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 -- 172.21.15.53:0/3727383864 shutdown_connections 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 --2- 172.21.15.53:0/3727383864 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40ccc10 0x7f02a406d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 --2- 172.21.15.53:0/3727383864 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a4143c40 0x7f02a4071290 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 -- 172.21.15.53:0/3727383864 >> 172.21.15.53:0/3727383864 conn(0x7f02a406a800 msgr2=0x7f02a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 -- 172.21.15.53:0/3727383864 shutdown_connections 2024-01-16T15:07:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.085+0000 7f02a9c00640 1 -- 172.21.15.53:0/3727383864 wait complete. 2024-01-16T15:07:07.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.087+0000 7f02a9c00640 1 Processor -- start 2024-01-16T15:07:07.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.087+0000 7f02a9c00640 1 -- start start 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.088+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40717d0 0x7f02a415a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 0x7f02a41eeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a9c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a4143c40 0x7f02a40af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a9c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4140a60 con 0x7f02a40ccc10 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a9c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4140d20 con 0x7f02a4143c40 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a4141000 con 0x7f02a40717d0 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 0x7f02a41eeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 0x7f02a41eeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47680/0 (socket says 172.21.15.53:47680) 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 -- 172.21.15.53:0/2080172889 learned_addr learned my addr 172.21.15.53:0/2080172889 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3e00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40717d0 0x7f02a415a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a9200640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a4143c40 0x7f02a40af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 -- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40717d0 msgr2=0x7f02a415a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40717d0 0x7f02a415a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 -- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a4143c40 msgr2=0x7f02a40af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a4143c40 0x7f02a40af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.089+0000 7f02a3400640 1 -- 172.21.15.53:0/2080172889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02940094d0 con 0x7f02a40ccc10 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.090+0000 7f02a3400640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 0x7f02a41eeee0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f028c00b3f0 tx=0x7f028c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.090+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c011830 con 0x7f02a40ccc10 2024-01-16T15:07:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.090+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f028c011990 con 0x7f02a40ccc10 2024-01-16T15:07:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.090+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a41412e0 con 0x7f02a40ccc10 2024-01-16T15:07:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.090+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028c011d40 con 0x7f02a40ccc10 2024-01-16T15:07:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.090+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a41ef420 con 0x7f02a40ccc10 2024-01-16T15:07:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.092+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a4067450 con 0x7f02a40ccc10 2024-01-16T15:07:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.092+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f028c01a400 con 0x7f02a40ccc10 2024-01-16T15:07:07.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.093+0000 7f029be00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06bb30 0x7f027c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:07.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.094+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f028c0883c0 con 0x7f02a40ccc10 2024-01-16T15:07:07.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.094+0000 7f02a3e00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06bb30 0x7f027c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:07.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.095+0000 7f02a3e00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06bb30 0x7f027c06dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0294036000 tx=0x7f029403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:07.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.098+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f028c054ab0 con 0x7f02a40ccc10 2024-01-16T15:07:07.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.364+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f02a40cbab0 con 0x7f02a40ccc10 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.365+0000 7f029be00640 1 -- 172.21.15.53:0/2080172889 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2404 (secure 0 0 0) 0x7f028c015090 con 0x7f02a40ccc10 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:07:07.343 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 454, 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:07:07.344 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 60, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:07:07.345 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 3739, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 15533209163, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 30631231488, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 742462881792, 2024-01-16T15:07:07.346 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 669903869, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 671329776, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 160, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 170 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 36, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 4, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:07.347 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 4, 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 5686 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:07:07.348 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 13, 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:06:26.332926+0000", 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "9598e1c6-adb6-436d-939a-5583937c4736": { 2024-01-16T15:07:07.349 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "Subvolume \"subvol_clone30237\" has been 100.0% cloned (25s)\n [============================] ", 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 1, 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: "c308c820-4f47-4fc6-a421-8eeb0c2f6732": { 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 57.143% (25s)\n [================............] (remaining: 19s)", 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.57142859697341919, 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:07.350 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06bb30 msgr2=0x7f027c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06bb30 0x7f027c06dff0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f0294036000 tx=0x7f029403d040 comp rx=0 tx=0).stop 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 msgr2=0x7f02a41eeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 0x7f02a41eeee0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f028c00b3f0 tx=0x7f028c00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 shutdown_connections 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f027c06bb30 0x7f027c06dff0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02a4143c40 0x7f02a40af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02a40ccc10 0x7f02a41eeee0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 --2- 172.21.15.53:0/2080172889 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02a40717d0 0x7f02a415a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:07.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 >> 172.21.15.53:0/2080172889 conn(0x7f02a406a800 msgr2=0x7f02a40b00b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:07.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 shutdown_connections 2024-01-16T15:07:07.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:07.368+0000 7f02a9c00640 1 -- 172.21.15.53:0/2080172889 wait complete. 2024-01-16T15:07:09.367 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:09.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30231 2024-01-16T15:07:09.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.625+0000 7f9bce400640 1 Processor -- start 2024-01-16T15:07:09.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.625+0000 7f9bce400640 1 -- start start 2024-01-16T15:07:09.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.626+0000 7f9bce400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc80ccb50 0x7f9bc8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.626+0000 7f9bce400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9bc815bf10 0x7f9bc8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.626+0000 7f9bce400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc815f4e0 con 0x7f9bc80adf20 2024-01-16T15:07:09.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.626+0000 7f9bce400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc815f970 con 0x7f9bc815bf10 2024-01-16T15:07:09.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.626+0000 7f9bce400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc815c3c0 con 0x7f9bc80ccb50 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9bc815bf10 0x7f9bc8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9bc815bf10 0x7f9bc8143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47692/0 (socket says 172.21.15.53:47692) 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 -- 172.21.15.53:0/4178969343 learned_addr learned my addr 172.21.15.53:0/4178969343 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 -- 172.21.15.53:0/4178969343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc80ccb50 msgr2=0x7f9bc8070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bc7e00640 1 --2- 172.21.15.53:0/4178969343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc80ccb50 0x7f9bc8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 --2- 172.21.15.53:0/4178969343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc80ccb50 0x7f9bc8070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 -- 172.21.15.53:0/4178969343 >> v1:172.21.15.146:6789/0 conn(0x7f9bc80adf20 legacy=0x7f9bc80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:09.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.627+0000 7f9bcda00640 1 -- 172.21.15.53:0/4178969343 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8071130 con 0x7f9bc815bf10 2024-01-16T15:07:09.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.628+0000 7f9bcda00640 1 --2- 172.21.15.53:0/4178969343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc815bf10 0x7f9bc8143d30 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009870 tx=0x7f9bc002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67f9edb372e34349 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.629+0000 7f9bc7400640 1 -- 172.21.15.53:0/4178969343 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc00317c0 con 0x7f9bc815bf10 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.629+0000 7f9bc7400640 1 -- 172.21.15.53:0/4178969343 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc0031920 con 0x7f9bc815bf10 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.629+0000 7f9bc7400640 1 -- 172.21.15.53:0/4178969343 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc0031cb0 con 0x7f9bc815bf10 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.629+0000 7f9bce400640 1 -- 172.21.15.53:0/4178969343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc815bf10 msgr2=0x7f9bc8143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.629+0000 7f9bce400640 1 --2- 172.21.15.53:0/4178969343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc815bf10 0x7f9bc8143d30 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0009870 tx=0x7f9bc002b860 comp rx=0 tx=0).stop 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.630+0000 7f9bce400640 1 -- 172.21.15.53:0/4178969343 shutdown_connections 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.630+0000 7f9bce400640 1 --2- 172.21.15.53:0/4178969343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc815bf10 0x7f9bc8143d30 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.630+0000 7f9bce400640 1 --2- 172.21.15.53:0/4178969343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc80ccb50 0x7f9bc8070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.630+0000 7f9bce400640 1 -- 172.21.15.53:0/4178969343 >> 172.21.15.53:0/4178969343 conn(0x7f9bc806a7d0 msgr2=0x7f9bc806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.630+0000 7f9bce400640 1 -- 172.21.15.53:0/4178969343 shutdown_connections 2024-01-16T15:07:09.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.630+0000 7f9bce400640 1 -- 172.21.15.53:0/4178969343 wait complete. 2024-01-16T15:07:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.631+0000 7f9bce400640 1 Processor -- start 2024-01-16T15:07:09.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.632+0000 7f9bce400640 1 -- start start 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bce400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bc80adf20 0x7f9bc80c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bce400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 0x7f9bc81f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bce400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc815bf10 0x7f9bc8132fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bce400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc8132bc0 con 0x7f9bc80ccb50 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bce400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc81f7380 con 0x7f9bc80adf20 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bce400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80c4770 con 0x7f9bc815bf10 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 0x7f9bc81f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 0x7f9bc81f64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47708/0 (socket says 172.21.15.53:47708) 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 -- 172.21.15.53:0/3278351751 learned_addr learned my addr 172.21.15.53:0/3278351751 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 -- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc815bf10 msgr2=0x7f9bc8132fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc815bf10 0x7f9bc8132fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 -- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bc80adf20 msgr2=0x7f9bc80c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bc80adf20 0x7f9bc80c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.633+0000 7f9bc7e00640 1 -- 172.21.15.53:0/3278351751 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc00094d0 con 0x7f9bc80ccb50 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.634+0000 7f9bc7e00640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 0x7f9bc81f64c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000c7f0 tx=0x7f9bb000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.634+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0004220 con 0x7f9bc80ccb50 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.634+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb0004380 con 0x7f9bc80ccb50 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.634+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0007840 con 0x7f9bc80ccb50 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.634+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc81f6dc0 con 0x7f9bc80ccb50 2024-01-16T15:07:09.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.634+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc81327e0 con 0x7f9bc80ccb50 2024-01-16T15:07:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.635+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc8132d00 con 0x7f9bc80ccb50 2024-01-16T15:07:09.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.636+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f9bb0007bb0 con 0x7f9bc80ccb50 2024-01-16T15:07:09.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.637+0000 7f9bc4c00640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ba006bb80 0x7f9ba006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:09.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.637+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9bb0088580 con 0x7f9bc80ccb50 2024-01-16T15:07:09.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.638+0000 7f9bcd000640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ba006bb80 0x7f9ba006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:09.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.639+0000 7f9bcd000640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ba006bb80 0x7f9ba006e040 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80043c0 tx=0x7f9bb8007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:09.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.642+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9bc8132d00 con 0x7f9bc80ccb50 2024-01-16T15:07:09.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:09.872+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30231", "target": ["mon-mgr", ""]}) v1 -- 0x7f9bc815f170 con 0x7f9ba006bb80 2024-01-16T15:07:11.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.748+0000 7f9bc4c00640 1 -- 172.21.15.53:0/3278351751 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f9bb8003fb0 con 0x7f9ba006bb80 2024-01-16T15:07:11.727 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:11.727 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:11.727 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:11.727 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:11.727 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ba006bb80 msgr2=0x7f9ba006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ba006bb80 0x7f9ba006e040 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f9bb80043c0 tx=0x7f9bb8007d30 comp rx=0 tx=0).stop 2024-01-16T15:07:11.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 msgr2=0x7f9bc81f64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 0x7f9bc81f64c0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9bb000c7f0 tx=0x7f9bb000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 shutdown_connections 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ba006bb80 0x7f9ba006e040 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9bc815bf10 0x7f9bc8132fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9bc80ccb50 0x7f9bc81f64c0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 --2- 172.21.15.53:0/3278351751 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9bc80adf20 0x7f9bc80c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 >> 172.21.15.53:0/3278351751 conn(0x7f9bc806a7d0 msgr2=0x7f9bc80cc160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 shutdown_connections 2024-01-16T15:07:11.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:11.752+0000 7f9bce400640 1 -- 172.21.15.53:0/3278351751 wait complete. 2024-01-16T15:07:11.755 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:11.755 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30232 2024-01-16T15:07:12.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.061+0000 7f7049c00640 1 Processor -- start 2024-01-16T15:07:12.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.061+0000 7f7049c00640 1 -- start start 2024-01-16T15:07:12.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.062+0000 7f7049c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70440ad880 0x7f70440717b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.062+0000 7f7049c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f70440ac8d0 0x7f704406b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7049c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f704415e680 con 0x7f704406b930 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7049c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f704406bdc0 con 0x7f70440ac8d0 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7049c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7044071cf0 con 0x7f70440ad880 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7043400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f704406b930 0x7f7044160ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48154/0 (socket says 172.21.15.53:48154) 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7043400640 1 -- 172.21.15.53:0/2945121767 learned_addr learned my addr 172.21.15.53:0/2945121767 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:12.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7042000640 1 --2- 172.21.15.53:0/2945121767 >> v2:172.21.15.146:3300/0 conn(0x7f70440ac8d0 0x7f704406b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.063+0000 7f7042a00640 1 --2- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70440ad880 0x7f70440717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7041600640 1 -- 172.21.15.53:0/2945121767 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551754905 0 0) 0x7f704415e680 con 0x7f704406b930 2024-01-16T15:07:12.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7041600640 1 -- 172.21.15.53:0/2945121767 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7024003520 con 0x7f704406b930 2024-01-16T15:07:12.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7042000640 1 -- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70440ad880 msgr2=0x7f70440717b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7041600640 1 -- 172.21.15.53:0/2945121767 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2979822579 0 0) 0x7f7024003520 con 0x7f704406b930 2024-01-16T15:07:12.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7042000640 1 --2- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70440ad880 0x7f70440717b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7042000640 1 -- 172.21.15.53:0/2945121767 >> v1:172.21.15.53:6789/0 conn(0x7f704406b930 legacy=0x7f7044160ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.064+0000 7f7042000640 1 -- 172.21.15.53:0/2945121767 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704415dd50 con 0x7f70440ac8d0 2024-01-16T15:07:12.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.066+0000 7f7042000640 1 --2- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 0x7f704406b3f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f702c0097a0 tx=0x7f702c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7bf99a209030816 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.066+0000 7f7041600640 1 -- 172.21.15.53:0/2945121767 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702c02f050 con 0x7f70440ac8d0 2024-01-16T15:07:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.066+0000 7f7041600640 1 -- 172.21.15.53:0/2945121767 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c031500 con 0x7f70440ac8d0 2024-01-16T15:07:12.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.066+0000 7f7041600640 1 -- 172.21.15.53:0/2945121767 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034001670 con 0x7f70440ac8d0 2024-01-16T15:07:12.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.067+0000 7f7049c00640 1 -- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 msgr2=0x7f704406b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.067+0000 7f7049c00640 1 --2- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 0x7f704406b3f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f702c0097a0 tx=0x7f702c02b820 comp rx=0 tx=0).stop 2024-01-16T15:07:12.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.068+0000 7f7049c00640 1 -- 172.21.15.53:0/2945121767 shutdown_connections 2024-01-16T15:07:12.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.068+0000 7f7049c00640 1 --2- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 0x7f704406b3f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.068+0000 7f7049c00640 1 --2- 172.21.15.53:0/2945121767 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f70440ad880 0x7f70440717b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.068+0000 7f7049c00640 1 -- 172.21.15.53:0/2945121767 >> 172.21.15.53:0/2945121767 conn(0x7f7044070a80 msgr2=0x7f7044070e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.068+0000 7f7049c00640 1 -- 172.21.15.53:0/2945121767 shutdown_connections 2024-01-16T15:07:12.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.068+0000 7f7049c00640 1 -- 172.21.15.53:0/2945121767 wait complete. 2024-01-16T15:07:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.070+0000 7f7049c00640 1 Processor -- start 2024-01-16T15:07:12.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.070+0000 7f7049c00640 1 -- start start 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7049c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704406b930 0x7f704414c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7049c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 0x7f704414e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7049c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 0x7f70440b34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7049c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70440cc110 con 0x7f70440ad880 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7049c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70440cc9b0 con 0x7f70440ac8d0 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7049c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f704414f320 con 0x7f704406b930 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7043400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 0x7f70440b34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7043400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 0x7f70440b34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39164/0 (socket says 172.21.15.53:39164) 2024-01-16T15:07:12.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.072+0000 7f7043400640 1 -- 172.21.15.53:0/826563079 learned_addr learned my addr 172.21.15.53:0/826563079 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7043400640 1 -- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704406b930 msgr2=0x7f704414c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7043400640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704406b930 0x7f704414c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7043400640 1 -- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 msgr2=0x7f704414e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7043400640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 0x7f704414e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7043400640 1 -- 172.21.15.53:0/826563079 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f702c009400 con 0x7f70440ad880 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7043400640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 0x7f70440b34e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f703400c750 tx=0x7f703400cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034005ad0 con 0x7f70440ad880 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7034005c30 con 0x7f70440ad880 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7034007070 con 0x7f70440ad880 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.073+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70440cc360 con 0x7f70440ad880 2024-01-16T15:07:12.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.074+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f704406e940 con 0x7f70440ad880 2024-01-16T15:07:12.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.075+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70440675d0 con 0x7f70440ad880 2024-01-16T15:07:12.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.077+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f70340071d0 con 0x7f70440ad880 2024-01-16T15:07:12.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.078+0000 7f703aa00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f701c06bb30 0x7f701c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.079+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f703408a670 con 0x7f70440ad880 2024-01-16T15:07:12.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.082+0000 7f7042a00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f701c06bb30 0x7f701c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.087+0000 7f7042a00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f701c06bb30 0x7f701c06dff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f703c00e190 tx=0x7f703c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.087+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7034054430 con 0x7f70440ad880 2024-01-16T15:07:12.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.457+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30232", "target": ["mon-mgr", ""]}) v1 -- 0x7f704415da00 con 0x7f701c06bb30 2024-01-16T15:07:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.494+0000 7f703aa00640 1 -- 172.21.15.53:0/826563079 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f703c00f040 con 0x7f701c06bb30 2024-01-16T15:07:12.472 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:12.472 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:12.472 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:12.472 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:12.472 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:12.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.501+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f701c06bb30 msgr2=0x7f701c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.501+0000 7f7049c00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f701c06bb30 0x7f701c06dff0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f703c00e190 tx=0x7f703c00b040 comp rx=0 tx=0).stop 2024-01-16T15:07:12.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.501+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 msgr2=0x7f70440b34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.501+0000 7f7049c00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 0x7f70440b34e0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f703400c750 tx=0x7f703400cc10 comp rx=0 tx=0).stop 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 shutdown_connections 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f701c06bb30 0x7f701c06dff0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f70440ad880 0x7f70440b34e0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f70440ac8d0 0x7f704414e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 --2- 172.21.15.53:0/826563079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f704406b930 0x7f704414c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 >> 172.21.15.53:0/826563079 conn(0x7f7044070a80 msgr2=0x7f704406ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:12.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 shutdown_connections 2024-01-16T15:07:12.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.502+0000 7f7049c00640 1 -- 172.21.15.53:0/826563079 wait complete. 2024-01-16T15:07:12.517 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:12.517 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30233 2024-01-16T15:07:12.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.802+0000 7fe0e9200640 1 Processor -- start 2024-01-16T15:07:12.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.803+0000 7fe0e9200640 1 -- start start 2024-01-16T15:07:12.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.804+0000 7fe0e9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0e4150760 0x7fe0e4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.804+0000 7fe0e9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e415d300 0x7fe0e415d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.804+0000 7fe0e9200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe0e40b2bd0 0x7fe0e40b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.804+0000 7fe0e9200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e415dc80 con 0x7fe0e4150760 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.804+0000 7fe0e9200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0e4071190 con 0x7fe0e40b2bd0 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.804+0000 7fe0e9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4071470 con 0x7fe0e415d300 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0e4150760 0x7fe0e4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe0e4150760 0x7fe0e4072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39176/0 (socket says 172.21.15.53:39176) 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 -- 172.21.15.53:0/26624668 learned_addr learned my addr 172.21.15.53:0/26624668 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e2800640 1 --2- 172.21.15.53:0/26624668 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40b2bd0 0x7fe0e40b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0d9200640 1 --2- 172.21.15.53:0/26624668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e415d300 0x7fe0e415d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 -- 172.21.15.53:0/26624668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e415d300 msgr2=0x7fe0e415d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 --2- 172.21.15.53:0/26624668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e415d300 0x7fe0e415d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 -- 172.21.15.53:0/26624668 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40b2bd0 msgr2=0x7fe0e40b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:12.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 --2- 172.21.15.53:0/26624668 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40b2bd0 0x7fe0e40b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.805+0000 7fe0e1e00640 1 -- 172.21.15.53:0/26624668 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e4071750 con 0x7fe0e4150760 2024-01-16T15:07:12.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.807+0000 7fe0e1e00640 1 --2- 172.21.15.53:0/26624668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e4072230 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40cbc10 tx=0x7fe0d402e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c762503cf620a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.808+0000 7fe0e1400640 1 -- 172.21.15.53:0/26624668 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d402ec90 con 0x7fe0e4150760 2024-01-16T15:07:12.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.808+0000 7fe0e1400640 1 -- 172.21.15.53:0/26624668 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0d402edf0 con 0x7fe0e4150760 2024-01-16T15:07:12.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.808+0000 7fe0e1400640 1 -- 172.21.15.53:0/26624668 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0d4036610 con 0x7fe0e4150760 2024-01-16T15:07:12.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 -- 172.21.15.53:0/26624668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 msgr2=0x7fe0e4072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 --2- 172.21.15.53:0/26624668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e4072230 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fe0e40cbc10 tx=0x7fe0d402e0f0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 -- 172.21.15.53:0/26624668 shutdown_connections 2024-01-16T15:07:12.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 --2- 172.21.15.53:0/26624668 >> v2:172.21.15.146:3300/0 conn(0x7fe0e40b2bd0 0x7fe0e40b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 --2- 172.21.15.53:0/26624668 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e415d300 0x7fe0e415d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 --2- 172.21.15.53:0/26624668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e4072230 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 -- 172.21.15.53:0/26624668 >> 172.21.15.53:0/26624668 conn(0x7fe0e406a800 msgr2=0x7fe0e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:12.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 -- 172.21.15.53:0/26624668 shutdown_connections 2024-01-16T15:07:12.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.809+0000 7fe0e9200640 1 -- 172.21.15.53:0/26624668 wait complete. 2024-01-16T15:07:12.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.812+0000 7fe0e9200640 1 Processor -- start 2024-01-16T15:07:12.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.812+0000 7fe0e9200640 1 -- start start 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.813+0000 7fe0e9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e40b2bd0 0x7fe0e415c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.813+0000 7fe0e9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e41f65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.813+0000 7fe0e9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e415d300 0x7fe0e41f6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.813+0000 7fe0e9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e4145400 con 0x7fe0e4150760 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.813+0000 7fe0e9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e414c320 con 0x7fe0e415d300 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.813+0000 7fe0e9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e414c600 con 0x7fe0e40b2bd0 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.814+0000 7fe0d9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e41f65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.814+0000 7fe0d9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e41f65f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39188/0 (socket says 172.21.15.53:39188) 2024-01-16T15:07:12.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.814+0000 7fe0d9200640 1 -- 172.21.15.53:0/2285887209 learned_addr learned my addr 172.21.15.53:0/2285887209 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:12.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.814+0000 7fe0e2800640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e415d300 0x7fe0e41f6b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0e1e00640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e40b2bd0 0x7fe0e415c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0d9200640 1 -- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e40b2bd0 msgr2=0x7fe0e415c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0d9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e40b2bd0 0x7fe0e415c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0d9200640 1 -- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e415d300 msgr2=0x7fe0e41f6b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0d9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e415d300 0x7fe0e41f6b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0d9200640 1 -- 172.21.15.53:0/2285887209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d400d040 con 0x7fe0e4150760 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.815+0000 7fe0d9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e41f65f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00d800 tx=0x7fe0cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.816+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00b6c0 con 0x7fe0e4150760 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.816+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0cc00b820 con 0x7fe0e4150760 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.816+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0cc00bbb0 con 0x7fe0e4150760 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.816+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0e414c8e0 con 0x7fe0e4150760 2024-01-16T15:07:12.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.816+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0e414d0e0 con 0x7fe0e4150760 2024-01-16T15:07:12.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.817+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0e40675d0 con 0x7fe0e4150760 2024-01-16T15:07:12.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.819+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fe0cc00bd10 con 0x7fe0e4150760 2024-01-16T15:07:12.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.819+0000 7fe0daa00640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0bc06bb30 0x7fe0bc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:12.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.820+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe0cc088aa0 con 0x7fe0e4150760 2024-01-16T15:07:12.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.824+0000 7fe0e1e00640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0bc06bb30 0x7fe0bc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:12.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.830+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe0cc0551f0 con 0x7fe0e4150760 2024-01-16T15:07:12.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:12.849+0000 7fe0e1e00640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0bc06bb30 0x7fe0bc06dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4037000 tx=0x7fe0d4032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.156+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30233", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0e414c770 con 0x7fe0bc06bb30 2024-01-16T15:07:13.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.716+0000 7fe0daa00640 1 -- 172.21.15.53:0/2285887209 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fe0e414c770 con 0x7fe0bc06bb30 2024-01-16T15:07:13.695 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:13.695 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:13.695 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:13.695 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:13.695 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0bc06bb30 msgr2=0x7fe0bc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0bc06bb30 0x7fe0bc06dff0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe0d4037000 tx=0x7fe0d4032000 comp rx=0 tx=0).stop 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 msgr2=0x7fe0e41f65f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e41f65f0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fe0cc00d800 tx=0x7fe0cc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 shutdown_connections 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0bc06bb30 0x7fe0bc06dff0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe0e415d300 0x7fe0e41f6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe0e4150760 0x7fe0e41f65f0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 --2- 172.21.15.53:0/2285887209 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe0e40b2bd0 0x7fe0e415c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 >> 172.21.15.53:0/2285887209 conn(0x7fe0e406a800 msgr2=0x7fe0e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 shutdown_connections 2024-01-16T15:07:13.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.719+0000 7fe0e9200640 1 -- 172.21.15.53:0/2285887209 wait complete. 2024-01-16T15:07:13.713 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:13.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30234 2024-01-16T15:07:13.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.920+0000 7f4ebd000640 1 Processor -- start 2024-01-16T15:07:13.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.920+0000 7f4ebd000640 1 -- start start 2024-01-16T15:07:13.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4ebd000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4eb80adf20 0x7f4eb8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4ebd000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4eb8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4ebd000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb815f4e0 con 0x7f4eb815bf10 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4ebd000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb815f970 con 0x7f4eb80adf20 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4ebd000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8071130 con 0x7f4eb80ccb50 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4eafe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4eb815bf10 0x7f4eb80cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48162/0 (socket says 172.21.15.53:48162) 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.922+0000 7f4eafe00640 1 -- 172.21.15.53:0/1884501505 learned_addr learned my addr 172.21.15.53:0/1884501505 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb7e00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4eb8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1884501505 >> v2:172.21.15.146:3300/0 conn(0x7f4eb80adf20 0x7f4eb8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084783495 0 0) 0x7f4eb815f4e0 con 0x7f4eb815bf10 2024-01-16T15:07:13.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e98003520 con 0x7f4eb815bf10 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2820276462 0 0) 0x7f4e98003520 con 0x7f4eb815bf10 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 msgr2=0x7f4eb8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4eb8143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 >> v2:172.21.15.146:3300/0 conn(0x7f4eb80adf20 msgr2=0x7f4eb8070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 --2- 172.21.15.53:0/1884501505 >> v2:172.21.15.146:3300/0 conn(0x7f4eb80adf20 0x7f4eb8070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb815c390 con 0x7f4eb815bf10 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4ea0001670 con 0x7f4eb815bf10 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1884501505 >> v2:172.21.15.146:3300/0 conn(0x7f4eb80adf20 0x7f4eb8070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 >> v1:172.21.15.53:6789/0 conn(0x7f4eb815bf10 legacy=0x7f4eb80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4e98003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98003f40 0x7f4e980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e98004d00 0x7f4e98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb815f970 con 0x7f4e98004d00 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb8071130 con 0x7f4e98003f40 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.923+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9800bae0 con 0x7f4eb80ccb50 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7e00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e98004d00 0x7f4e98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4e98003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7e00640 1 -- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 msgr2=0x7f4e98003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eafe00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98003f40 0x7f4e980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7e00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4e98003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7e00640 1 -- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98003f40 msgr2=0x7f4e980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:13.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7e00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98003f40 0x7f4e980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7e00640 1 -- 172.21.15.53:0/1884501505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e98003520 con 0x7f4e98004d00 2024-01-16T15:07:13.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb80ccb50 0x7f4e98003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:13.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.924+0000 7f4eafe00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98003f40 0x7f4e980047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.925+0000 7f4eb7e00640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e98004d00 0x7f4e98005180 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009f90 tx=0x7f4e980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c3def305b52338 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb002cbd0 con 0x7f4e98004d00 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eb000f040 con 0x7f4e98004d00 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4eb6a00640 1 -- 172.21.15.53:0/1884501505 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb0034440 con 0x7f4e98004d00 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4ebd000640 1 -- 172.21.15.53:0/1884501505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e98004d00 msgr2=0x7f4e98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1884501505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e98004d00 0x7f4e98005180 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4eb0009f90 tx=0x7f4e980057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4eb7e00640 1 -- 172.21.15.53:0/1884501505 reap_dead start 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4ebd000640 1 -- 172.21.15.53:0/1884501505 shutdown_connections 2024-01-16T15:07:13.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4ebd000640 1 -- 172.21.15.53:0/1884501505 >> 172.21.15.53:0/1884501505 conn(0x7f4eb806a7d0 msgr2=0x7f4eb806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.926+0000 7f4ebd000640 1 -- 172.21.15.53:0/1884501505 shutdown_connections 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.927+0000 7f4ebd000640 1 -- 172.21.15.53:0/1884501505 wait complete. 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.928+0000 7f4ebd000640 1 Processor -- start 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.929+0000 7f4ebd000640 1 -- start start 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4ebd000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98004d00 0x7f4eb80aba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4ebd000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 0x7f4eb81422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4ebd000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb81427e0 0x7f4eb806d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4ebd000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb812e7d0 con 0x7f4eb80abfb0 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4ebd000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb81433d0 con 0x7f4e98004d00 2024-01-16T15:07:13.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4ebd000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb812e4a0 con 0x7f4eb81427e0 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4eafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 0x7f4eb81422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4eafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 0x7f4eb81422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39206/0 (socket says 172.21.15.53:39206) 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4eafe00640 1 -- 172.21.15.53:0/1559967870 learned_addr learned my addr 172.21.15.53:0/1559967870 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4eb7e00640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb81427e0 0x7f4eb806d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.930+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98004d00 0x7f4eb80aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eafe00640 1 -- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb81427e0 msgr2=0x7f4eb806d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eafe00640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb81427e0 0x7f4eb806d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eafe00640 1 -- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98004d00 msgr2=0x7f4eb80aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eafe00640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98004d00 0x7f4eb80aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:13.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eafe00640 1 -- 172.21.15.53:0/1559967870 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb8140c50 con 0x7f4eb80abfb0 2024-01-16T15:07:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eafe00640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 0x7f4eb81422a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000d8e0 tx=0x7f4ea000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00074e0 con 0x7f4eb80abfb0 2024-01-16T15:07:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea0007640 con 0x7f4eb80abfb0 2024-01-16T15:07:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0004a00 con 0x7f4eb80abfb0 2024-01-16T15:07:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb8142e10 con 0x7f4eb80abfb0 2024-01-16T15:07:13.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.931+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb806db10 con 0x7f4eb80abfb0 2024-01-16T15:07:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.932+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb812e980 con 0x7f4eb80abfb0 2024-01-16T15:07:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.933+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4ea0004b60 con 0x7f4eb80abfb0 2024-01-16T15:07:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.934+0000 7f4eaf400640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e9006bb60 0x7f4e9006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:13.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.934+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4ea0088030 con 0x7f4eb80abfb0 2024-01-16T15:07:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.935+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e9006bb60 0x7f4e9006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:13.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.936+0000 7f4eb7400640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e9006bb60 0x7f4e9006e020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8006fd0 tx=0x7f4ea8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:13.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:13.939+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4eb812e980 con 0x7f4eb80abfb0 2024-01-16T15:07:14.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.225+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30234", "target": ["mon-mgr", ""]}) v1 -- 0x7f4eb815f170 con 0x7f4e9006bb60 2024-01-16T15:07:14.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.336+0000 7f4eaf400640 1 -- 172.21.15.53:0/1559967870 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f4ea800e510 con 0x7f4e9006bb60 2024-01-16T15:07:14.314 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:14.315 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:14.315 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:14.315 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:14.315 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:14.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e9006bb60 msgr2=0x7f4e9006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e9006bb60 0x7f4e9006e020 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8006fd0 tx=0x7f4ea8008040 comp rx=0 tx=0).stop 2024-01-16T15:07:14.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 msgr2=0x7f4eb81422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 0x7f4eb81422a0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000d8e0 tx=0x7f4ea000dda0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 shutdown_connections 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e9006bb60 0x7f4e9006e020 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eb81427e0 0x7f4eb806d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4eb80abfb0 0x7f4eb81422a0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 --2- 172.21.15.53:0/1559967870 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e98004d00 0x7f4eb80aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 >> 172.21.15.53:0/1559967870 conn(0x7f4eb806a7d0 msgr2=0x7f4eb806cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 shutdown_connections 2024-01-16T15:07:14.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.340+0000 7f4ebd000640 1 -- 172.21.15.53:0/1559967870 wait complete. 2024-01-16T15:07:14.348 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:14.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30235 2024-01-16T15:07:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.663+0000 7fde95600640 1 Processor -- start 2024-01-16T15:07:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.664+0000 7fde95600640 1 -- start start 2024-01-16T15:07:14.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.665+0000 7fde95600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde900cf9a0 0x7fde900cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.665+0000 7fde95600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde900b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.665+0000 7fde95600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde900d0320 con 0x7fde9015d4b0 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.665+0000 7fde95600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde900d1c60 con 0x7fde900cf9a0 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.665+0000 7fde95600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde900b2730 con 0x7fde900b31d0 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde900cf9a0 0x7fde900cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde900cf9a0 0x7fde900cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39222/0 (socket says 172.21.15.53:39222) 2024-01-16T15:07:14.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 -- 172.21.15.53:0/980694823 learned_addr learned my addr 172.21.15.53:0/980694823 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:14.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde94c00640 1 --2- 172.21.15.53:0/980694823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde900b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 -- 172.21.15.53:0/980694823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 msgr2=0x7fde900b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:14.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 --2- 172.21.15.53:0/980694823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde900b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 -- 172.21.15.53:0/980694823 >> v1:172.21.15.146:6789/0 conn(0x7fde9015d4b0 legacy=0x7fde900d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.666+0000 7fde8f400640 1 -- 172.21.15.53:0/980694823 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde900d1ec0 con 0x7fde900cf9a0 2024-01-16T15:07:14.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.668+0000 7fde8f400640 1 --2- 172.21.15.53:0/980694823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde900cfde0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fde78009870 tx=0x7fde7802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fc85c77373e507c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.668+0000 7fde8ea00640 1 -- 172.21.15.53:0/980694823 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78030840 con 0x7fde900cf9a0 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.668+0000 7fde8ea00640 1 -- 172.21.15.53:0/980694823 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde780309a0 con 0x7fde900cf9a0 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.668+0000 7fde8ea00640 1 -- 172.21.15.53:0/980694823 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78030d30 con 0x7fde900cf9a0 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.669+0000 7fde95600640 1 -- 172.21.15.53:0/980694823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 msgr2=0x7fde900cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.669+0000 7fde95600640 1 --2- 172.21.15.53:0/980694823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde900cfde0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7fde78009870 tx=0x7fde7802b980 comp rx=0 tx=0).stop 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.670+0000 7fde95600640 1 -- 172.21.15.53:0/980694823 shutdown_connections 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.670+0000 7fde95600640 1 --2- 172.21.15.53:0/980694823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde900b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.670+0000 7fde95600640 1 --2- 172.21.15.53:0/980694823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde900cfde0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.670+0000 7fde95600640 1 -- 172.21.15.53:0/980694823 >> 172.21.15.53:0/980694823 conn(0x7fde900d3000 msgr2=0x7fde900d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:14.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.670+0000 7fde95600640 1 -- 172.21.15.53:0/980694823 shutdown_connections 2024-01-16T15:07:14.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.671+0000 7fde95600640 1 -- 172.21.15.53:0/980694823 wait complete. 2024-01-16T15:07:14.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.675+0000 7fde95600640 1 Processor -- start 2024-01-16T15:07:14.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.675+0000 7fde95600640 1 -- start start 2024-01-16T15:07:14.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.676+0000 7fde95600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde90154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.677+0000 7fde95600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde9009f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.677+0000 7fde95600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde9015d4b0 0x7fde900ce300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.677+0000 7fde95600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9009e2e0 con 0x7fde900cf9a0 2024-01-16T15:07:14.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.677+0000 7fde95600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde900a0180 con 0x7fde9015d4b0 2024-01-16T15:07:14.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.677+0000 7fde95600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde900a0460 con 0x7fde900b31d0 2024-01-16T15:07:14.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.679+0000 7fde8fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde90154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.679+0000 7fde8f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde9009f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.679+0000 7fde94c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde9015d4b0 0x7fde900ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.679+0000 7fde8f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde9009f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39230/0 (socket says 172.21.15.53:39230) 2024-01-16T15:07:14.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.679+0000 7fde8f400640 1 -- 172.21.15.53:0/2569905372 learned_addr learned my addr 172.21.15.53:0/2569905372 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:14.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.680+0000 7fde8f400640 1 -- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 msgr2=0x7fde90154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.680+0000 7fde8f400640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde90154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.680+0000 7fde8f400640 1 -- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde9015d4b0 msgr2=0x7fde900ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:14.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.680+0000 7fde8f400640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde9015d4b0 0x7fde900ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:14.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.680+0000 7fde8f400640 1 -- 172.21.15.53:0/2569905372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde780094d0 con 0x7fde900cf9a0 2024-01-16T15:07:14.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.680+0000 7fde8f400640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde9009f6a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fde78000c00 tx=0x7fde78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:14.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.681+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78039a90 con 0x7fde900cf9a0 2024-01-16T15:07:14.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.681+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde78039bf0 con 0x7fde900cf9a0 2024-01-16T15:07:14.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.681+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde900a0740 con 0x7fde900cf9a0 2024-01-16T15:07:14.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.681+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde78043410 con 0x7fde900cf9a0 2024-01-16T15:07:14.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.682+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde901548c0 con 0x7fde900cf9a0 2024-01-16T15:07:14.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.683+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde900b7000 con 0x7fde900cf9a0 2024-01-16T15:07:14.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.686+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fde7802f390 con 0x7fde900cf9a0 2024-01-16T15:07:14.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.687+0000 7fde87e00640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde6806bb30 0x7fde6806dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:14.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.688+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde780aa8c0 con 0x7fde900cf9a0 2024-01-16T15:07:14.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.688+0000 7fde8fe00640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde6806bb30 0x7fde6806dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:14.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.692+0000 7fde8fe00640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde6806bb30 0x7fde6806dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fde80000f60 tx=0x7fde80009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:14.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:14.696+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde780771e0 con 0x7fde900cf9a0 2024-01-16T15:07:15.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.094+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7fde900d08e0 con 0x7fde6806bb30 2024-01-16T15:07:15.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.131+0000 7fde87e00640 1 -- 172.21.15.53:0/2569905372 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fde900d08e0 con 0x7fde6806bb30 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:15.110 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:15.111 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:15.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde6806bb30 msgr2=0x7fde6806dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde6806bb30 0x7fde6806dff0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fde80000f60 tx=0x7fde80009040 comp rx=0 tx=0).stop 2024-01-16T15:07:15.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 msgr2=0x7fde9009f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:15.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde9009f6a0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7fde78000c00 tx=0x7fde78004730 comp rx=0 tx=0).stop 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 shutdown_connections 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde6806bb30 0x7fde6806dff0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde9015d4b0 0x7fde900ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde900cf9a0 0x7fde9009f6a0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 --2- 172.21.15.53:0/2569905372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde900b31d0 0x7fde90154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 >> 172.21.15.53:0/2569905372 conn(0x7fde900d3000 msgr2=0x7fde900b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 shutdown_connections 2024-01-16T15:07:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:15.134+0000 7fde95600640 1 -- 172.21.15.53:0/2569905372 wait complete. 2024-01-16T15:07:15.131 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:16.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30235 2024-01-16T15:07:16.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.418+0000 7fbc48600640 1 Processor -- start 2024-01-16T15:07:16.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.418+0000 7fbc48600640 1 -- start start 2024-01-16T15:07:16.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.420+0000 7fbc48600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbc4013df80 0x7fbc4015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.420+0000 7fbc48600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc400cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.420+0000 7fbc48600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4006c470 con 0x7fbc4013e360 2024-01-16T15:07:16.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.420+0000 7fbc48600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4006c750 con 0x7fbc4013df80 2024-01-16T15:07:16.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.420+0000 7fbc48600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4006ca30 con 0x7fbc400cf9a0 2024-01-16T15:07:16.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.421+0000 7fbc46600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc400cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.421+0000 7fbc45200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fbc4013e360 0x7fbc400cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:54724/0 (socket says 172.21.15.53:54724) 2024-01-16T15:07:16.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.421+0000 7fbc45200640 1 -- 172.21.15.53:0/731376033 learned_addr learned my addr 172.21.15.53:0/731376033 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:16.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.421+0000 7fbc45c00640 1 --2- 172.21.15.53:0/731376033 >> v2:172.21.15.146:3300/0 conn(0x7fbc4013df80 0x7fbc4015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.421+0000 7fbc45c00640 1 -- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 msgr2=0x7fbc400cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:16.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.422+0000 7fbc45c00640 1 --2- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc400cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.422+0000 7fbc45c00640 1 -- 172.21.15.53:0/731376033 >> v1:172.21.15.146:6789/0 conn(0x7fbc4013e360 legacy=0x7fbc400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.422+0000 7fbc45c00640 1 -- 172.21.15.53:0/731376033 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4006cd10 con 0x7fbc4013df80 2024-01-16T15:07:16.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.423+0000 7fbc45c00640 1 --2- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013df80 0x7fbc4015bee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbc300097a0 tx=0x7fbc3002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a10f8da1841d62e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:16.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.424+0000 7fbc3fe00640 1 -- 172.21.15.53:0/731376033 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031830 con 0x7fbc4013df80 2024-01-16T15:07:16.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.424+0000 7fbc3fe00640 1 -- 172.21.15.53:0/731376033 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc30031990 con 0x7fbc4013df80 2024-01-16T15:07:16.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.424+0000 7fbc3fe00640 1 -- 172.21.15.53:0/731376033 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc30031d20 con 0x7fbc4013df80 2024-01-16T15:07:16.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 -- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013df80 msgr2=0x7fbc4015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 --2- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013df80 0x7fbc4015bee0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fbc300097a0 tx=0x7fbc3002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 -- 172.21.15.53:0/731376033 shutdown_connections 2024-01-16T15:07:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 --2- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc400cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 --2- 172.21.15.53:0/731376033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013df80 0x7fbc4015bee0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 -- 172.21.15.53:0/731376033 >> 172.21.15.53:0/731376033 conn(0x7fbc4006a800 msgr2=0x7fbc4006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 -- 172.21.15.53:0/731376033 shutdown_connections 2024-01-16T15:07:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.425+0000 7fbc48600640 1 -- 172.21.15.53:0/731376033 wait complete. 2024-01-16T15:07:16.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.427+0000 7fbc48600640 1 Processor -- start 2024-01-16T15:07:16.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.428+0000 7fbc48600640 1 -- start start 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc48600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc40156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc48600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 0x7fbc401f1d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc48600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013e360 0x7fbc4013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc48600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc4013a150 con 0x7fbc4013df80 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc48600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc40157000 con 0x7fbc4013e360 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc48600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc401391f0 con 0x7fbc400cf9a0 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc45200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 0x7fbc401f1d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc45200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 0x7fbc401f1d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39232/0 (socket says 172.21.15.53:39232) 2024-01-16T15:07:16.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.429+0000 7fbc45200640 1 -- 172.21.15.53:0/2915331025 learned_addr learned my addr 172.21.15.53:0/2915331025 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc45200640 1 -- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 msgr2=0x7fbc40156ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc45200640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc40156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc45200640 1 -- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013e360 msgr2=0x7fbc4013c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc45200640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013e360 0x7fbc4013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc45200640 1 -- 172.21.15.53:0/2915331025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc30009400 con 0x7fbc4013df80 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc45200640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 0x7fbc401f1d90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c440 tx=0x7fbc28001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc28005ce0 con 0x7fbc4013df80 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc28005e40 con 0x7fbc4013df80 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.430+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc280072d0 con 0x7fbc4013df80 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.431+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc401394d0 con 0x7fbc4013df80 2024-01-16T15:07:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.431+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc40139a90 con 0x7fbc4013df80 2024-01-16T15:07:16.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.432+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4013a290 con 0x7fbc4013df80 2024-01-16T15:07:16.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.433+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fbc28007430 con 0x7fbc4013df80 2024-01-16T15:07:16.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.434+0000 7fbc3d600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbc1806bb60 0x7fbc1806e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:16.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.434+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbc28087920 con 0x7fbc4013df80 2024-01-16T15:07:16.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.439+0000 7fbc45c00640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbc1806bb60 0x7fbc1806e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:16.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.440+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbc4013a290 con 0x7fbc4013df80 2024-01-16T15:07:16.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.441+0000 7fbc45c00640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbc1806bb60 0x7fbc1806e020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbc30036000 tx=0x7fbc30030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:16.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.836+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc4015f9c0 con 0x7fbc1806bb60 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.873+0000 7fbc3d600640 1 -- 172.21.15.53:0/2915331025 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fbc30031b00 con 0x7fbc1806bb60 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:16.851 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:16.852 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbc1806bb60 msgr2=0x7fbc1806e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbc1806bb60 0x7fbc1806e020 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbc30036000 tx=0x7fbc30030000 comp rx=0 tx=0).stop 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 msgr2=0x7fbc401f1d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 0x7fbc401f1d90 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbc2800c440 tx=0x7fbc28001e70 comp rx=0 tx=0).stop 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 shutdown_connections 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbc1806bb60 0x7fbc1806e020 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbc4013e360 0x7fbc4013c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbc4013df80 0x7fbc401f1d90 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.878+0000 7fbc48600640 1 --2- 172.21.15.53:0/2915331025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbc400cf9a0 0x7fbc40156ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:16.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.879+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 >> 172.21.15.53:0/2915331025 conn(0x7fbc4006a800 msgr2=0x7fbc4006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:16.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.879+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 shutdown_connections 2024-01-16T15:07:16.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:16.879+0000 7fbc48600640 1 -- 172.21.15.53:0/2915331025 wait complete. 2024-01-16T15:07:16.890 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:17.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30235 2024-01-16T15:07:18.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.198+0000 7f5d5d200640 1 Processor -- start 2024-01-16T15:07:18.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.198+0000 7f5d5d200640 1 -- start start 2024-01-16T15:07:18.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.200+0000 7f5d5d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5d58143ae0 0x7f5d580cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.200+0000 7f5d5d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d580cfc30 con 0x7f5d580cf7d0 2024-01-16T15:07:18.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.200+0000 7f5d5d200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d580b1ee0 con 0x7f5d5815e3b0 2024-01-16T15:07:18.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.200+0000 7f5d5d200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d580b2780 con 0x7f5d58143ae0 2024-01-16T15:07:18.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.201+0000 7f5d55400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5d580cf7d0 0x7f5d5815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48172/0 (socket says 172.21.15.53:48172) 2024-01-16T15:07:18.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.201+0000 7f5d55400640 1 -- 172.21.15.53:0/3293546612 learned_addr learned my addr 172.21.15.53:0/3293546612 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:18.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.201+0000 7f5d55e00640 1 --2- 172.21.15.53:0/3293546612 >> v2:172.21.15.53:3300/0 conn(0x7f5d58143ae0 0x7f5d580cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.201+0000 7f5d54a00640 1 -- 172.21.15.53:0/3293546612 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3419558147 0 0) 0x7f5d580cfc30 con 0x7f5d580cf7d0 2024-01-16T15:07:18.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.201+0000 7f5d54a00640 1 -- 172.21.15.53:0/3293546612 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d38005730 con 0x7f5d580cf7d0 2024-01-16T15:07:18.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.202+0000 7f5d55e00640 1 -- 172.21.15.53:0/3293546612 >> v1:172.21.15.146:6789/0 conn(0x7f5d5815e3b0 legacy=0x7f5d580702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:18.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.202+0000 7f5d54a00640 1 -- 172.21.15.53:0/3293546612 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2425334966 0 0) 0x7f5d38005730 con 0x7f5d580cf7d0 2024-01-16T15:07:18.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.202+0000 7f5d55e00640 1 -- 172.21.15.53:0/3293546612 >> v1:172.21.15.53:6789/0 conn(0x7f5d580cf7d0 legacy=0x7f5d5815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.202+0000 7f5d55e00640 1 -- 172.21.15.53:0/3293546612 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5815e990 con 0x7f5d58143ae0 2024-01-16T15:07:18.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.203+0000 7f5d55e00640 1 --2- 172.21.15.53:0/3293546612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d580cf290 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009d90 tx=0x7f5d4802bd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb08de239a4fd90 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.204+0000 7f5d54a00640 1 -- 172.21.15.53:0/3293546612 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40001670 con 0x7f5d58143ae0 2024-01-16T15:07:18.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.204+0000 7f5d54a00640 1 -- 172.21.15.53:0/3293546612 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4802c8b0 con 0x7f5d58143ae0 2024-01-16T15:07:18.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.204+0000 7f5d54a00640 1 -- 172.21.15.53:0/3293546612 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4802cc40 con 0x7f5d58143ae0 2024-01-16T15:07:18.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 -- 172.21.15.53:0/3293546612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 msgr2=0x7f5d580cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 --2- 172.21.15.53:0/3293546612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d580cf290 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f5d48009d90 tx=0x7f5d4802bd80 comp rx=0 tx=0).stop 2024-01-16T15:07:18.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 -- 172.21.15.53:0/3293546612 shutdown_connections 2024-01-16T15:07:18.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 --2- 172.21.15.53:0/3293546612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d580cf290 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 -- 172.21.15.53:0/3293546612 >> 172.21.15.53:0/3293546612 conn(0x7f5d5806a800 msgr2=0x7f5d5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 -- 172.21.15.53:0/3293546612 shutdown_connections 2024-01-16T15:07:18.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.205+0000 7f5d5d200640 1 -- 172.21.15.53:0/3293546612 wait complete. 2024-01-16T15:07:18.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.207+0000 7f5d5d200640 1 Processor -- start 2024-01-16T15:07:18.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.207+0000 7f5d5d200640 1 -- start start 2024-01-16T15:07:18.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d5d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d580cf7d0 0x7f5d5813eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d5d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d58140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d5d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5815e3b0 0x7f5d580702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d5d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d580a9280 con 0x7f5d58143ae0 2024-01-16T15:07:18.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d5d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5813dd80 con 0x7f5d5815e3b0 2024-01-16T15:07:18.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d5d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5813eff0 con 0x7f5d580cf7d0 2024-01-16T15:07:18.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d55400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d58140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d55400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d58140dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39254/0 (socket says 172.21.15.53:39254) 2024-01-16T15:07:18.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d55400640 1 -- 172.21.15.53:0/1793009131 learned_addr learned my addr 172.21.15.53:0/1793009131 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d55e00640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d580cf7d0 0x7f5d5813eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.209+0000 7f5d56800640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5815e3b0 0x7f5d580702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d55400640 1 -- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d580cf7d0 msgr2=0x7f5d5813eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d55400640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d580cf7d0 0x7f5d5813eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d55400640 1 -- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5815e3b0 msgr2=0x7f5d580702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d55400640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5815e3b0 0x7f5d580702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d55400640 1 -- 172.21.15.53:0/1793009131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d48009a70 con 0x7f5d58143ae0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d56800640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5815e3b0 0x7f5d580702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d55400640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d58140dd0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000c710 tx=0x7f5d4000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40005ad0 con 0x7f5d58143ae0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d40005c30 con 0x7f5d58143ae0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.210+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d40007070 con 0x7f5d58143ae0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.211+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5813d170 con 0x7f5d58143ae0 2024-01-16T15:07:18.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.211+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5813d730 con 0x7f5d58143ae0 2024-01-16T15:07:18.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.212+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d580cbf50 con 0x7f5d58143ae0 2024-01-16T15:07:18.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.214+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5d400071f0 con 0x7f5d58143ae0 2024-01-16T15:07:18.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.217+0000 7f5d4e000640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d3006bbb0 0x7f5d3006e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:18.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.218+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5d400883b0 con 0x7f5d58143ae0 2024-01-16T15:07:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.228+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5d40054ae0 con 0x7f5d58143ae0 2024-01-16T15:07:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.229+0000 7f5d55e00640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d3006bbb0 0x7f5d3006e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:18.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.230+0000 7f5d55e00640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d3006bbb0 0x7f5d3006e070 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d48036000 tx=0x7f5d48030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:18.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.710+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7f5d580b2410 con 0x7f5d3006bbb0 2024-01-16T15:07:18.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.737+0000 7f5d4e000640 1 -- 172.21.15.53:0/1793009131 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f5d48009450 con 0x7f5d3006bbb0 2024-01-16T15:07:18.715 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:18.715 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:18.715 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:18.715 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:18.715 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 33.33333333333333, 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "1 GB/3 GB", 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "1/3", 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:18.716 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.744+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d3006bbb0 msgr2=0x7f5d3006e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.744+0000 7f5d5d200640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d3006bbb0 0x7f5d3006e070 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d48036000 tx=0x7f5d48030000 comp rx=0 tx=0).stop 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.744+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 msgr2=0x7f5d58140dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.744+0000 7f5d5d200640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d58140dd0 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f5d4000c710 tx=0x7f5d4000cbd0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.744+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 shutdown_connections 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.745+0000 7f5d5d200640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5d3006bbb0 0x7f5d3006e070 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5d48036000 tx=0x7f5d48030000 comp rx=0 tx=0).stop 2024-01-16T15:07:18.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.745+0000 7f5d5d200640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5d5815e3b0 0x7f5d580702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.745+0000 7f5d5d200640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5d58143ae0 0x7f5d58140dd0 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.745+0000 7f5d5d200640 1 --2- 172.21.15.53:0/1793009131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5d580cf7d0 0x7f5d5813eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:18.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.745+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 >> 172.21.15.53:0/1793009131 conn(0x7f5d5806a800 msgr2=0x7f5d5815a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:18.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.747+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 shutdown_connections 2024-01-16T15:07:18.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:18.747+0000 7f5d5d200640 1 -- 172.21.15.53:0/1793009131 wait complete. 2024-01-16T15:07:18.758 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:19.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30235 2024-01-16T15:07:20.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.100+0000 7fa8f9200640 1 Processor -- start 2024-01-16T15:07:20.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.100+0000 7fa8f9200640 1 -- start start 2024-01-16T15:07:20.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.102+0000 7fa8f9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8f4153a50 0x7fa8f40b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.102+0000 7fa8f9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f4070e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.102+0000 7fa8f9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8f4159e20 con 0x7fa8f41598d0 2024-01-16T15:07:20.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.102+0000 7fa8f9200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8f4153e30 con 0x7fa8f4153a50 2024-01-16T15:07:20.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.102+0000 7fa8f9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f40709b0 con 0x7fa8f40cfad0 2024-01-16T15:07:20.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.103+0000 7fa8f1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8f4153a50 0x7fa8f40b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.103+0000 7fa8f1400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa8f41598d0 0x7fa8f40cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48180/0 (socket says 172.21.15.53:48180) 2024-01-16T15:07:20.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.103+0000 7fa8f1400640 1 -- 172.21.15.53:0/3657674928 learned_addr learned my addr 172.21.15.53:0/3657674928 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:20.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.103+0000 7fa8f2800640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f4070e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f0a00640 1 -- 172.21.15.53:0/3657674928 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137664857 0 0) 0x7fa8f4159e20 con 0x7fa8f41598d0 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f0a00640 1 -- 172.21.15.53:0/3657674928 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4003520 con 0x7fa8f41598d0 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f1e00640 1 -- 172.21.15.53:0/3657674928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 msgr2=0x7fa8f4070e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f1e00640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f4070e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f1e00640 1 -- 172.21.15.53:0/3657674928 >> v1:172.21.15.53:6789/0 conn(0x7fa8f41598d0 legacy=0x7fa8f40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f1e00640 1 -- 172.21.15.53:0/3657674928 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8f415a100 con 0x7fa8f4153a50 2024-01-16T15:07:20.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.104+0000 7fa8f2800640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f4070e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:20.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.105+0000 7fa8f1e00640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f40b2ab0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009d90 tx=0x7fa8ec02bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fd18cc74dd2b2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.106+0000 7fa8f0a00640 1 -- 172.21.15.53:0/3657674928 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec02ccf0 con 0x7fa8f4153a50 2024-01-16T15:07:20.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.106+0000 7fa8f0a00640 1 -- 172.21.15.53:0/3657674928 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8ec02ce50 con 0x7fa8f4153a50 2024-01-16T15:07:20.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.107+0000 7fa8f9200640 1 -- 172.21.15.53:0/3657674928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 msgr2=0x7fa8f40b2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.107+0000 7fa8f9200640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f40b2ab0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009d90 tx=0x7fa8ec02bf40 comp rx=0 tx=0).stop 2024-01-16T15:07:20.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.107+0000 7fa8f9200640 1 -- 172.21.15.53:0/3657674928 shutdown_connections 2024-01-16T15:07:20.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.107+0000 7fa8f9200640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f4070e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.107+0000 7fa8f9200640 1 --2- 172.21.15.53:0/3657674928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f40b2ab0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.107+0000 7fa8f9200640 1 -- 172.21.15.53:0/3657674928 >> 172.21.15.53:0/3657674928 conn(0x7fa8f406a800 msgr2=0x7fa8f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:20.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.108+0000 7fa8f9200640 1 -- 172.21.15.53:0/3657674928 shutdown_connections 2024-01-16T15:07:20.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.108+0000 7fa8f9200640 1 -- 172.21.15.53:0/3657674928 wait complete. 2024-01-16T15:07:20.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.110+0000 7fa8f9200640 1 Processor -- start 2024-01-16T15:07:20.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.110+0000 7fa8f9200640 1 -- start start 2024-01-16T15:07:20.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.111+0000 7fa8f9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f414ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.111+0000 7fa8f9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f414a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.111+0000 7fa8f9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8f41598d0 0x7fa8f4151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.111+0000 7fa8f9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4151110 con 0x7fa8f4153a50 2024-01-16T15:07:20.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.111+0000 7fa8f9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f4150500 con 0x7fa8f41598d0 2024-01-16T15:07:20.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.111+0000 7fa8f9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f41507e0 con 0x7fa8f40cfad0 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f414a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f414a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39272/0 (socket says 172.21.15.53:39272) 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 -- 172.21.15.53:0/1884761467 learned_addr learned my addr 172.21.15.53:0/1884761467 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 -- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 msgr2=0x7fa8f414ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f414ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 -- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8f41598d0 msgr2=0x7fa8f4151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8f41598d0 0x7fa8f4151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.112+0000 7fa8f1400640 1 -- 172.21.15.53:0/1884761467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8ec009a70 con 0x7fa8f4153a50 2024-01-16T15:07:20.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.113+0000 7fa8f1400640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f414a4e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c410 tx=0x7fa8e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.114+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e4005ce0 con 0x7fa8f4153a50 2024-01-16T15:07:20.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.114+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4150ac0 con 0x7fa8f4153a50 2024-01-16T15:07:20.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.114+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f414f340 con 0x7fa8f4153a50 2024-01-16T15:07:20.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.114+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8e4005e40 con 0x7fa8f4153a50 2024-01-16T15:07:20.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.114+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8e40072d0 con 0x7fa8f4153a50 2024-01-16T15:07:20.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.116+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f415a680 con 0x7fa8f4153a50 2024-01-16T15:07:20.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.117+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fa8e4007430 con 0x7fa8f4153a50 2024-01-16T15:07:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.120+0000 7fa8ea000640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8cc06bb30 0x7fa8cc06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:20.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.120+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa8e4087b90 con 0x7fa8f4153a50 2024-01-16T15:07:20.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.126+0000 7fa8f1e00640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8cc06bb30 0x7fa8cc06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:20.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.126+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8e4054370 con 0x7fa8f4153a50 2024-01-16T15:07:20.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.127+0000 7fa8f1e00640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8cc06bb30 0x7fa8cc06dff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec036000 tx=0x7fa8ec02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:20.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.519+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7fa8f414f550 con 0x7fa8cc06bb30 2024-01-16T15:07:20.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.532+0000 7fa8ea000640 1 -- 172.21.15.53:0/1884761467 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fa8f414f550 con 0x7fa8cc06bb30 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:07:20.510 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:20.511 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:20.511 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:20.511 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:20.511 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:20.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8cc06bb30 msgr2=0x7fa8cc06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8cc06bb30 0x7fa8cc06dff0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec036000 tx=0x7fa8ec02e000 comp rx=0 tx=0).stop 2024-01-16T15:07:20.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 msgr2=0x7fa8f414a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f414a4e0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fa8e400c410 tx=0x7fa8e400c8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 shutdown_connections 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8cc06bb30 0x7fa8cc06dff0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8f41598d0 0x7fa8f4151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8f4153a50 0x7fa8f414a4e0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 --2- 172.21.15.53:0/1884761467 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8f40cfad0 0x7fa8f414ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.534+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 >> 172.21.15.53:0/1884761467 conn(0x7fa8f406a800 msgr2=0x7fa8f40b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.535+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 shutdown_connections 2024-01-16T15:07:20.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:20.535+0000 7fa8f9200640 1 -- 172.21.15.53:0/1884761467 wait complete. 2024-01-16T15:07:20.538 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:21.540 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30235 2024-01-16T15:07:21.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.882+0000 7f68b2000640 1 Processor -- start 2024-01-16T15:07:21.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.883+0000 7f68b2000640 1 -- start start 2024-01-16T15:07:21.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68b2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68b2000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f68ac06c6b0 0x7f68ac099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68b2000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68ac099fa0 0x7f68ac06b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68b2000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac09a430 con 0x7f68ac06c6b0 2024-01-16T15:07:21.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68b2000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68ac06cea0 con 0x7f68ac099fa0 2024-01-16T15:07:21.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68b2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac06cbc0 con 0x7f68ac0cd240 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68aaa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f68ac06c6b0 0x7f68ac099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68aaa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f68ac06c6b0 0x7f68ac099a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43958/0 (socket says 172.21.15.53:43958) 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.885+0000 7f68aaa00640 1 -- 172.21.15.53:0/2086844543 learned_addr learned my addr 172.21.15.53:0/2086844543 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68ab400640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68abe00640 1 --2- 172.21.15.53:0/2086844543 >> v2:172.21.15.146:3300/0 conn(0x7f68ac099fa0 0x7f68ac06b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68aaa00640 1 -- 172.21.15.53:0/2086844543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 msgr2=0x7f68ac0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68aaa00640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68aaa00640 1 -- 172.21.15.53:0/2086844543 >> v2:172.21.15.146:3300/0 conn(0x7f68ac099fa0 msgr2=0x7f68ac06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68aaa00640 1 --2- 172.21.15.53:0/2086844543 >> v2:172.21.15.146:3300/0 conn(0x7f68ac099fa0 0x7f68ac06b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68aaa00640 1 -- 172.21.15.53:0/2086844543 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac06f350 con 0x7f68ac06c6b0 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68ab400640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:21.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.886+0000 7f68abe00640 1 --2- 172.21.15.53:0/2086844543 >> v2:172.21.15.146:3300/0 conn(0x7f68ac099fa0 0x7f68ac06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:21.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.888+0000 7f68aaa00640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac099a60 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0722b0 tx=0x7f689402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c2fcb2edaca899f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.888+0000 7f68aa000640 1 -- 172.21.15.53:0/2086844543 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894031860 con 0x7f68ac06c6b0 2024-01-16T15:07:21.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.888+0000 7f68aa000640 1 -- 172.21.15.53:0/2086844543 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68940319c0 con 0x7f68ac06c6b0 2024-01-16T15:07:21.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.888+0000 7f68aa000640 1 -- 172.21.15.53:0/2086844543 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6894031d50 con 0x7f68ac06c6b0 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 -- 172.21.15.53:0/2086844543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 msgr2=0x7f68ac099a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac099a60 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f68ac0722b0 tx=0x7f689402b900 comp rx=0 tx=0).stop 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 -- 172.21.15.53:0/2086844543 shutdown_connections 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 --2- 172.21.15.53:0/2086844543 >> v2:172.21.15.146:3300/0 conn(0x7f68ac099fa0 0x7f68ac06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac099a60 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 --2- 172.21.15.53:0/2086844543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 -- 172.21.15.53:0/2086844543 >> 172.21.15.53:0/2086844543 conn(0x7f68ac0b2d00 msgr2=0x7f68ac0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 -- 172.21.15.53:0/2086844543 shutdown_connections 2024-01-16T15:07:21.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.889+0000 7f68b2000640 1 -- 172.21.15.53:0/2086844543 wait complete. 2024-01-16T15:07:21.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.891+0000 7f68b2000640 1 Processor -- start 2024-01-16T15:07:21.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.891+0000 7f68b2000640 1 -- start start 2024-01-16T15:07:21.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68b2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac15f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68b2000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68ac099fa0 0x7f68ac1f0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68b2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac1f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68ab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac15f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68b2000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1f6bb0 con 0x7f68ac06c6b0 2024-01-16T15:07:21.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68b2000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1f3030 con 0x7f68ac099fa0 2024-01-16T15:07:21.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68b2000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68ac1f6e50 con 0x7f68ac0cd240 2024-01-16T15:07:21.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68ab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac15f570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43974/0 (socket says 172.21.15.53:43974) 2024-01-16T15:07:21.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.893+0000 7f68ab400640 1 -- 172.21.15.53:0/2137228600 learned_addr learned my addr 172.21.15.53:0/2137228600 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.894+0000 7f68ab400640 1 -- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 msgr2=0x7f68ac1f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.894+0000 7f68ab400640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac1f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.894+0000 7f68ab400640 1 -- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68ac099fa0 msgr2=0x7f68ac1f0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.894+0000 7f68ab400640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68ac099fa0 0x7f68ac1f0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.894+0000 7f68ab400640 1 -- 172.21.15.53:0/2137228600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68940094d0 con 0x7f68ac06c6b0 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.894+0000 7f68ab400640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac15f570 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f689c00c7f0 tx=0x7f689c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.895+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c007a30 con 0x7f68ac06c6b0 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.895+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689c007b90 con 0x7f68ac06c6b0 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.895+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f689c00ce40 con 0x7f68ac06c6b0 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.895+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68ac1f7130 con 0x7f68ac06c6b0 2024-01-16T15:07:21.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.895+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68ac1f3300 con 0x7f68ac06c6b0 2024-01-16T15:07:21.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.897+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68ac06fc30 con 0x7f68ac06c6b0 2024-01-16T15:07:21.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.897+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f689c0113f0 con 0x7f68ac06c6b0 2024-01-16T15:07:21.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.899+0000 7f68a3400640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f688406bb30 0x7f688406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:21.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.899+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f689c088730 con 0x7f68ac06c6b0 2024-01-16T15:07:21.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.901+0000 7f68aaa00640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f688406bb30 0x7f688406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.902+0000 7f68aaa00640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f688406bb30 0x7f688406dff0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6894036000 tx=0x7f6894030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:21.905+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f689c054e80 con 0x7f68ac06c6b0 2024-01-16T15:07:22.068 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:22.073 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:22.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.328+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7f68ac0cd620 con 0x7f688406bb30 2024-01-16T15:07:22.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.376+0000 7f68a3400640 1 -- 172.21.15.53:0/2137228600 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f6894031c30 con 0x7f688406bb30 2024-01-16T15:07:22.356 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:22.356 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:22.356 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:22.357 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:22.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.382+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f688406bb30 msgr2=0x7f688406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.382+0000 7f68b2000640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f688406bb30 0x7f688406dff0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6894036000 tx=0x7f6894030000 comp rx=0 tx=0).stop 2024-01-16T15:07:22.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.382+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 msgr2=0x7f68ac15f570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:22.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.382+0000 7f68b2000640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac15f570 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f689c00c7f0 tx=0x7f689c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.383+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 shutdown_connections 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.383+0000 7f68b2000640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f688406bb30 0x7f688406dff0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.383+0000 7f68b2000640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68ac0cd240 0x7f68ac1f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.383+0000 7f68b2000640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68ac099fa0 0x7f68ac1f0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.383+0000 7f68b2000640 1 --2- 172.21.15.53:0/2137228600 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68ac06c6b0 0x7f68ac15f570 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.383+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 >> 172.21.15.53:0/2137228600 conn(0x7f68ac0b2d00 msgr2=0x7f68ac0cc590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:22.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.384+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 shutdown_connections 2024-01-16T15:07:22.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:22.384+0000 7f68b2000640 1 -- 172.21.15.53:0/2137228600 wait complete. 2024-01-16T15:07:22.390 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:23.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30235 2024-01-16T15:07:23.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.741+0000 7f4e3cc00640 1 Processor -- start 2024-01-16T15:07:23.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.741+0000 7f4e3cc00640 1 -- start start 2024-01-16T15:07:23.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.743+0000 7f4e3cc00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4e380cf9a0 0x7f4e3815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.743+0000 7f4e3cc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e380b1ee0 con 0x7f4e3815e1e0 2024-01-16T15:07:23.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.743+0000 7f4e3cc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e380b2780 con 0x7f4e38143ae0 2024-01-16T15:07:23.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.743+0000 7f4e3cc00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e3815e990 con 0x7f4e380cf9a0 2024-01-16T15:07:23.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.744+0000 7f4e36800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4e3815e1e0 0x7f4e380702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55036/0 (socket says 172.21.15.53:55036) 2024-01-16T15:07:23.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.744+0000 7f4e36800640 1 -- 172.21.15.53:0/2371114379 learned_addr learned my addr 172.21.15.53:0/2371114379 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:23.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.744+0000 7f4e35400640 1 --2- 172.21.15.53:0/2371114379 >> v2:172.21.15.146:3300/0 conn(0x7f4e380cf9a0 0x7f4e3815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445055747 0 0) 0x7f4e380b1ee0 con 0x7f4e3815e1e0 2024-01-16T15:07:23.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e18003520 con 0x7f4e3815e1e0 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 276681762 0 0) 0x7f4e18003520 con 0x7f4e3815e1e0 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 >> v2:172.21.15.146:3300/0 conn(0x7f4e380cf9a0 msgr2=0x7f4e3815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 --2- 172.21.15.53:0/2371114379 >> v2:172.21.15.146:3300/0 conn(0x7f4e380cf9a0 0x7f4e3815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 >> v1:172.21.15.146:6789/0 conn(0x7f4e38143ae0 legacy=0x7f4e380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e380b2130 con 0x7f4e3815e1e0 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4e300019f0 con 0x7f4e3815e1e0 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.745+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 >> v1:172.21.15.53:6789/0 conn(0x7f4e3815e1e0 legacy=0x7f4e380702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e34a00640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e180039c0 0x7f4e18003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e34a00640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e180047c0 0x7f4e18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e34a00640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18005180 0x7f4e1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3815e990 con 0x7f4e18005180 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18010490 con 0x7f4e180047c0 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18010650 con 0x7f4e180039c0 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e36800640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18005180 0x7f4e1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.746+0000 7f4e35e00640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e180039c0 0x7f4e18003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.747+0000 7f4e35400640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e180047c0 0x7f4e18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.747+0000 7f4e36800640 1 -- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e180039c0 msgr2=0x7f4e18003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.747+0000 7f4e36800640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e180039c0 0x7f4e18003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.747+0000 7f4e36800640 1 -- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e180047c0 msgr2=0x7f4e18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:23.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.747+0000 7f4e36800640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e180047c0 0x7f4e18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.747+0000 7f4e36800640 1 -- 172.21.15.53:0/2371114379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e18003520 con 0x7f4e18005180 2024-01-16T15:07:23.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.749+0000 7f4e36800640 1 -- 172.21.15.53:0/2371114379 reap_dead start 2024-01-16T15:07:23.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.749+0000 7f4e36800640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18005180 0x7f4e1800b980 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000a760 tx=0x7f4e180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13056ecc474501c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.749+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30034630 con 0x7f4e18005180 2024-01-16T15:07:23.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.749+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3002f070 con 0x7f4e18005180 2024-01-16T15:07:23.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.749+0000 7f4e34a00640 1 -- 172.21.15.53:0/2371114379 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e300349b0 con 0x7f4e18005180 2024-01-16T15:07:23.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.750+0000 7f4e3cc00640 1 -- 172.21.15.53:0/2371114379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18005180 msgr2=0x7f4e1800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.750+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18005180 0x7f4e1800b980 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4e3000a760 tx=0x7f4e180057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.750+0000 7f4e3cc00640 1 -- 172.21.15.53:0/2371114379 shutdown_connections 2024-01-16T15:07:23.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.750+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/2371114379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e18005180 0x7f4e1800b980 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.750+0000 7f4e3cc00640 1 -- 172.21.15.53:0/2371114379 >> 172.21.15.53:0/2371114379 conn(0x7f4e3806a800 msgr2=0x7f4e3806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:23.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.751+0000 7f4e3cc00640 1 -- 172.21.15.53:0/2371114379 shutdown_connections 2024-01-16T15:07:23.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.751+0000 7f4e3cc00640 1 -- 172.21.15.53:0/2371114379 wait complete. 2024-01-16T15:07:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.753+0000 7f4e3cc00640 1 Processor -- start 2024-01-16T15:07:23.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.753+0000 7f4e3cc00640 1 -- start start 2024-01-16T15:07:23.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.754+0000 7f4e3cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e18005180 0x7f4e3814d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.754+0000 7f4e3cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3813c280 0x7f4e3813c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.754+0000 7f4e3cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 0x7f4e3815b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.754+0000 7f4e3cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3815b8b0 con 0x7f4e3815aef0 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.755+0000 7f4e3cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3813cc20 con 0x7f4e18005180 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.755+0000 7f4e3cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e3814dff0 con 0x7f4e3813c280 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.755+0000 7f4e36800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 0x7f4e3815b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.755+0000 7f4e36800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 0x7f4e3815b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43984/0 (socket says 172.21.15.53:43984) 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.755+0000 7f4e36800640 1 -- 172.21.15.53:0/511760952 learned_addr learned my addr 172.21.15.53:0/511760952 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.756+0000 7f4e36800640 1 -- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3813c280 msgr2=0x7f4e3813c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.756+0000 7f4e36800640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3813c280 0x7f4e3813c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.756+0000 7f4e36800640 1 -- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e18005180 msgr2=0x7f4e3814d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:23.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.756+0000 7f4e36800640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e18005180 0x7f4e3814d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:23.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.756+0000 7f4e36800640 1 -- 172.21.15.53:0/511760952 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3815bad0 con 0x7f4e3815aef0 2024-01-16T15:07:23.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.756+0000 7f4e36800640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 0x7f4e3815b370 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4e30002230 tx=0x7f4e30001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:23.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.757+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30001930 con 0x7f4e3815aef0 2024-01-16T15:07:23.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.757+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e30004e20 con 0x7f4e3815aef0 2024-01-16T15:07:23.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.757+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e30001ab0 con 0x7f4e3815aef0 2024-01-16T15:07:23.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.757+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e3815bdb0 con 0x7f4e3815aef0 2024-01-16T15:07:23.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.757+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e380cc6e0 con 0x7f4e3815aef0 2024-01-16T15:07:23.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.759+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e380675d0 con 0x7f4e3815aef0 2024-01-16T15:07:23.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.762+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f4e30031030 con 0x7f4e3815aef0 2024-01-16T15:07:23.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.765+0000 7f4e2e000640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e1006bb80 0x7f4e1006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:23.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.765+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4e300aa140 con 0x7f4e3815aef0 2024-01-16T15:07:23.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.767+0000 7f4e35e00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e1006bb80 0x7f4e1006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:23.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.773+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e30076870 con 0x7f4e3815aef0 2024-01-16T15:07:23.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:23.790+0000 7f4e35e00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e1006bb80 0x7f4e1006e040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4e280016d0 tx=0x7f4e28009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.181+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e3815e670 con 0x7f4e1006bb80 2024-01-16T15:07:24.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.469+0000 7f4e2e000640 1 -- 172.21.15.53:0/511760952 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f4e280013b0 con 0x7f4e1006bb80 2024-01-16T15:07:24.448 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:24.448 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:24.449 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.473+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e1006bb80 msgr2=0x7f4e1006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.473+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e1006bb80 0x7f4e1006e040 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f4e280016d0 tx=0x7f4e28009040 comp rx=0 tx=0).stop 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.473+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 msgr2=0x7f4e3815b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 0x7f4e3815b370 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4e30002230 tx=0x7f4e30001cb0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 shutdown_connections 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e1006bb80 0x7f4e1006e040 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e3815aef0 0x7f4e3815b370 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e3813c280 0x7f4e3813c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 --2- 172.21.15.53:0/511760952 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e18005180 0x7f4e3814d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 >> 172.21.15.53:0/511760952 conn(0x7f4e3806a800 msgr2=0x7f4e38142b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 shutdown_connections 2024-01-16T15:07:24.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.474+0000 7f4e3cc00640 1 -- 172.21.15.53:0/511760952 wait complete. 2024-01-16T15:07:24.488 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:24.488 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30236 2024-01-16T15:07:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.814+0000 7fb473a00640 1 Processor -- start 2024-01-16T15:07:24.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.814+0000 7fb473a00640 1 -- start start 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.816+0000 7fb473a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c160f80 0x7fb46c161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.816+0000 7fb473a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb46c0ce8f0 0x7fb46c0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.816+0000 7fb473a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb46c0cfbe0 0x7fb46c0d0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.816+0000 7fb473a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46c1618a0 con 0x7fb46c0ce8f0 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.816+0000 7fb473a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb46c0d0de0 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.816+0000 7fb473a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0d10c0 con 0x7fb46c160f80 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb46c0ce8f0 0x7fb46c0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb46c0ce8f0 0x7fb46c0ced50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43986/0 (socket says 172.21.15.53:43986) 2024-01-16T15:07:24.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 -- 172.21.15.53:0/3724909673 learned_addr learned my addr 172.21.15.53:0/3724909673 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb471a00640 1 --2- 172.21.15.53:0/3724909673 >> v2:172.21.15.146:3300/0 conn(0x7fb46c0cfbe0 0x7fb46c0d0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 -- 172.21.15.53:0/3724909673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c160f80 msgr2=0x7fb46c161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 --2- 172.21.15.53:0/3724909673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c160f80 0x7fb46c161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 -- 172.21.15.53:0/3724909673 >> v2:172.21.15.146:3300/0 conn(0x7fb46c0cfbe0 msgr2=0x7fb46c0d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 --2- 172.21.15.53:0/3724909673 >> v2:172.21.15.146:3300/0 conn(0x7fb46c0cfbe0 0x7fb46c0d0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.817+0000 7fb46be00640 1 -- 172.21.15.53:0/3724909673 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb46c0d13a0 con 0x7fb46c0ce8f0 2024-01-16T15:07:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.818+0000 7fb46be00640 1 --2- 172.21.15.53:0/3724909673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0ce8f0 0x7fb46c0ced50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0c6a40 tx=0x7fb45402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf160517a0fbe433 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.819+0000 7fb46b400640 1 -- 172.21.15.53:0/3724909673 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454031870 con 0x7fb46c0ce8f0 2024-01-16T15:07:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.819+0000 7fb46b400640 1 -- 172.21.15.53:0/3724909673 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4540319d0 con 0x7fb46c0ce8f0 2024-01-16T15:07:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.819+0000 7fb46b400640 1 -- 172.21.15.53:0/3724909673 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454031d60 con 0x7fb46c0ce8f0 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.819+0000 7fb473a00640 1 -- 172.21.15.53:0/3724909673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0ce8f0 msgr2=0x7fb46c0ced50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.819+0000 7fb473a00640 1 --2- 172.21.15.53:0/3724909673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0ce8f0 0x7fb46c0ced50 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0c6a40 tx=0x7fb45402b910 comp rx=0 tx=0).stop 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 -- 172.21.15.53:0/3724909673 shutdown_connections 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 --2- 172.21.15.53:0/3724909673 >> v2:172.21.15.146:3300/0 conn(0x7fb46c0cfbe0 0x7fb46c0d0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 --2- 172.21.15.53:0/3724909673 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0ce8f0 0x7fb46c0ced50 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 --2- 172.21.15.53:0/3724909673 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c160f80 0x7fb46c161360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 -- 172.21.15.53:0/3724909673 >> 172.21.15.53:0/3724909673 conn(0x7fb46c09c550 msgr2=0x7fb46c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 -- 172.21.15.53:0/3724909673 shutdown_connections 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.820+0000 7fb473a00640 1 -- 172.21.15.53:0/3724909673 wait complete. 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.821+0000 7fb473a00640 1 Processor -- start 2024-01-16T15:07:24.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.822+0000 7fb473a00640 1 -- start start 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.822+0000 7fb473a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c0ce8f0 0x7fb46c1585d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb473a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 0x7fb46c15df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb473a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb46c160f80 0x7fb46c0b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb473a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c15dad0 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb473a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0ccfb0 con 0x7fb46c160f80 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb473a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c0cd290 con 0x7fb46c0ce8f0 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 0x7fb46c15df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 0x7fb46c15df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44000/0 (socket says 172.21.15.53:44000) 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 -- 172.21.15.53:0/2846110783 learned_addr learned my addr 172.21.15.53:0/2846110783 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb471a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb46c160f80 0x7fb46c0b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 -- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c0ce8f0 msgr2=0x7fb46c1585d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c0ce8f0 0x7fb46c1585d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 -- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb46c160f80 msgr2=0x7fb46c0b0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb46c160f80 0x7fb46c0b0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb46be00640 1 -- 172.21.15.53:0/2846110783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4540094d0 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.823+0000 7fb471a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb46c160f80 0x7fb46c0b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.824+0000 7fb46be00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 0x7fb46c15df60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb454000c00 tx=0x7fb4540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.824+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45403abe0 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.824+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45403ad40 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.824+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454044610 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.824+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c15d760 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.824+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c15e4a0 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.825+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb46c0675d0 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.826+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb45404d450 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.827+0000 7fb468c00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb44406bb30 0x7fb44406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.827+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb45402d040 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.832+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb454077380 con 0x7fb46c0cfbe0 2024-01-16T15:07:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.834+0000 7fb471000640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb44406bb30 0x7fb44406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:24.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:24.840+0000 7fb471000640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb44406bb30 0x7fb44406dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009670 tx=0x7fb45c006bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:25.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.259+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30236", "target": ["mon-mgr", ""]}) v1 -- 0x7fb46c0ce5e0 con 0x7fb44406bb30 2024-01-16T15:07:25.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.294+0000 7fb468c00640 1 -- 172.21.15.53:0/2846110783 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7fb46c0ce5e0 con 0x7fb44406bb30 2024-01-16T15:07:25.274 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:25.275 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:25.276 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:25.276 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb44406bb30 msgr2=0x7fb44406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb44406bb30 0x7fb44406dff0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb45c009670 tx=0x7fb45c006bf0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 msgr2=0x7fb46c15df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 0x7fb46c15df60 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fb454000c00 tx=0x7fb4540046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 shutdown_connections 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb44406bb30 0x7fb44406dff0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb46c160f80 0x7fb46c0b0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb46c0cfbe0 0x7fb46c15df60 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 --2- 172.21.15.53:0/2846110783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb46c0ce8f0 0x7fb46c1585d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:25.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.303+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 >> 172.21.15.53:0/2846110783 conn(0x7fb46c09c550 msgr2=0x7fb46c0cecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.304+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 shutdown_connections 2024-01-16T15:07:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:25.304+0000 7fb473a00640 1 -- 172.21.15.53:0/2846110783 wait complete. 2024-01-16T15:07:25.312 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:26.312 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30236 2024-01-16T15:07:26.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.566+0000 7f56aaa00640 1 Processor -- start 2024-01-16T15:07:26.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.566+0000 7f56aaa00640 1 -- start start 2024-01-16T15:07:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56aaa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f56a40adf20 0x7f56a4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56aaa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a415f4e0 con 0x7f56a40ccb50 2024-01-16T15:07:26.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56aaa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a4071130 con 0x7f56a415bf10 2024-01-16T15:07:26.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56aaa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a415f970 con 0x7f56a40adf20 2024-01-16T15:07:26.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56a9600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f56a40adf20 0x7f56a4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56aa000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f56a40ccb50 0x7f56a4143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55040/0 (socket says 172.21.15.53:55040) 2024-01-16T15:07:26.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.568+0000 7f56aa000640 1 -- 172.21.15.53:0/1813771483 learned_addr learned my addr 172.21.15.53:0/1813771483 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.569+0000 7f56a9600640 1 -- 172.21.15.53:0/1813771483 >> v1:172.21.15.146:6789/0 conn(0x7f56a415bf10 legacy=0x7f56a40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.569+0000 7f56a9600640 1 -- 172.21.15.53:0/1813771483 >> v1:172.21.15.53:6789/0 conn(0x7f56a40ccb50 legacy=0x7f56a4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.569+0000 7f56a9600640 1 -- 172.21.15.53:0/1813771483 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a415c390 con 0x7f56a40adf20 2024-01-16T15:07:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.569+0000 7f56a3e00640 1 -- 172.21.15.53:0/1813771483 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761629943 0 0) 0x7f56a415f4e0 con 0x7f56a40ccb50 2024-01-16T15:07:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.570+0000 7f56a9600640 1 --2- 172.21.15.53:0/1813771483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40adf20 0x7f56a4070a30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f56940097a0 tx=0x7f569402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90b359b59ac22eef server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.570+0000 7f56a3e00640 1 -- 172.21.15.53:0/1813771483 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569402e070 con 0x7f56a40adf20 2024-01-16T15:07:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.570+0000 7f56a3e00640 1 -- 172.21.15.53:0/1813771483 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56940315a0 con 0x7f56a40adf20 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.570+0000 7f56a3e00640 1 -- 172.21.15.53:0/1813771483 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694031930 con 0x7f56a40adf20 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 -- 172.21.15.53:0/1813771483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40adf20 msgr2=0x7f56a4070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 --2- 172.21.15.53:0/1813771483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40adf20 0x7f56a4070a30 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f56940097a0 tx=0x7f569402b840 comp rx=0 tx=0).stop 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 -- 172.21.15.53:0/1813771483 shutdown_connections 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 --2- 172.21.15.53:0/1813771483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40adf20 0x7f56a4070a30 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 -- 172.21.15.53:0/1813771483 >> 172.21.15.53:0/1813771483 conn(0x7f56a406a7d0 msgr2=0x7f56a406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 -- 172.21.15.53:0/1813771483 shutdown_connections 2024-01-16T15:07:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.571+0000 7f56aaa00640 1 -- 172.21.15.53:0/1813771483 wait complete. 2024-01-16T15:07:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.572+0000 7f56aaa00640 1 Processor -- start 2024-01-16T15:07:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.573+0000 7f56aaa00640 1 -- start start 2024-01-16T15:07:26.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.573+0000 7f56aaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56a40adf20 0x7f56a40b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.573+0000 7f56aaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 0x7f56a40c96c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56aaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56a415bf10 0x7f56a415c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56aaa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40abb80 con 0x7f56a40ccb50 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56aaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a413ab70 con 0x7f56a40adf20 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56aaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a40b3e90 con 0x7f56a415bf10 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 0x7f56a40c96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 0x7f56a40c96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44022/0 (socket says 172.21.15.53:44022) 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 -- 172.21.15.53:0/524683978 learned_addr learned my addr 172.21.15.53:0/524683978 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 -- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56a415bf10 msgr2=0x7f56a415c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56a415bf10 0x7f56a415c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 -- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56a40adf20 msgr2=0x7f56a40b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56a40adf20 0x7f56a40b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 -- 172.21.15.53:0/524683978 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694009400 con 0x7f56a40ccb50 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.574+0000 7f56a8c00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 0x7f56a40c96c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f568c00b3f0 tx=0x7f568c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.575+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c011830 con 0x7f56a40ccb50 2024-01-16T15:07:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.575+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f568c011990 con 0x7f56a40ccb50 2024-01-16T15:07:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.575+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f568c011d40 con 0x7f56a40ccb50 2024-01-16T15:07:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.575+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a40b2390 con 0x7f56a40ccb50 2024-01-16T15:07:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.575+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a40b2950 con 0x7f56a40ccb50 2024-01-16T15:07:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.576+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a4067450 con 0x7f56a40ccb50 2024-01-16T15:07:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.577+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f568c01a400 con 0x7f56a40ccb50 2024-01-16T15:07:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.578+0000 7f56a1600640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f567c06bb30 0x7f567c06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.578+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f568c088420 con 0x7f56a40ccb50 2024-01-16T15:07:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.579+0000 7f56a9600640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f567c06bb30 0x7f567c06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:26.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.581+0000 7f56a9600640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f567c06bb30 0x7f567c06dff0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5694036000 tx=0x7f5694030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:26.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.582+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f568c054b10 con 0x7f56a40ccb50 2024-01-16T15:07:26.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.807+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30236", "target": ["mon-mgr", ""]}) v1 -- 0x7f56a415f170 con 0x7f567c06bb30 2024-01-16T15:07:26.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.927+0000 7f56a1600640 1 -- 172.21.15.53:0/524683978 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f5694004040 con 0x7f567c06bb30 2024-01-16T15:07:26.905 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:26.905 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:26.905 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:26.905 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:26.905 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:26.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f567c06bb30 msgr2=0x7f567c06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f567c06bb30 0x7f567c06dff0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5694036000 tx=0x7f5694030000 comp rx=0 tx=0).stop 2024-01-16T15:07:26.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 msgr2=0x7f56a40c96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:26.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 0x7f56a40c96c0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f568c00b3f0 tx=0x7f568c00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 shutdown_connections 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f567c06bb30 0x7f567c06dff0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f56a415bf10 0x7f56a415c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f56a40ccb50 0x7f56a40c96c0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 --2- 172.21.15.53:0/524683978 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f56a40adf20 0x7f56a40b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.930+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 >> 172.21.15.53:0/524683978 conn(0x7f56a406a7d0 msgr2=0x7f56a40ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.931+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 shutdown_connections 2024-01-16T15:07:26.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:26.931+0000 7f56aaa00640 1 -- 172.21.15.53:0/524683978 wait complete. 2024-01-16T15:07:26.930 INFO:tasks.cephfs.test_volumes:expected clone state = ('complete',) 2024-01-16T15:07:26.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30237 2024-01-16T15:07:27.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.179+0000 7f3151400640 1 Processor -- start 2024-01-16T15:07:27.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.179+0000 7f3151400640 1 -- start start 2024-01-16T15:07:27.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f3151400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f314c13e360 0x7f314c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f3151400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c15c5f0 con 0x7f314c15f9c0 2024-01-16T15:07:27.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f3151400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c06c470 con 0x7f314c13df80 2024-01-16T15:07:27.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f3151400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314c06c750 con 0x7f314c13e360 2024-01-16T15:07:27.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f314b200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f314c15f9c0 0x7f314c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55044/0 (socket says 172.21.15.53:55044) 2024-01-16T15:07:27.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f314b200640 1 -- 172.21.15.53:0/2786986460 learned_addr learned my addr 172.21.15.53:0/2786986460 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:27.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.181+0000 7f3141c00640 1 --2- 172.21.15.53:0/2786986460 >> v2:172.21.15.53:3300/0 conn(0x7f314c13e360 0x7f314c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.182+0000 7f3149e00640 1 -- 172.21.15.53:0/2786986460 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 354527000 0 0) 0x7f314c15c5f0 con 0x7f314c15f9c0 2024-01-16T15:07:27.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.182+0000 7f3149e00640 1 -- 172.21.15.53:0/2786986460 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312c003520 con 0x7f314c15f9c0 2024-01-16T15:07:27.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.182+0000 7f3141c00640 1 -- 172.21.15.53:0/2786986460 >> v1:172.21.15.146:6789/0 conn(0x7f314c13df80 legacy=0x7f314c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.182+0000 7f3141c00640 1 -- 172.21.15.53:0/2786986460 >> v1:172.21.15.53:6789/0 conn(0x7f314c15f9c0 legacy=0x7f314c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.182+0000 7f3141c00640 1 -- 172.21.15.53:0/2786986460 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c06ca30 con 0x7f314c13e360 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.183+0000 7f3141c00640 1 --2- 172.21.15.53:0/2786986460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c13e360 0x7f314c0cf290 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f31340097a0 tx=0x7f313402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90fa55aa023669b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3149e00640 1 -- 172.21.15.53:0/2786986460 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134030610 con 0x7f314c13e360 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3149e00640 1 -- 172.21.15.53:0/2786986460 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3134030770 con 0x7f314c13e360 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3149e00640 1 -- 172.21.15.53:0/2786986460 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3134030b00 con 0x7f314c13e360 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 -- 172.21.15.53:0/2786986460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c13e360 msgr2=0x7f314c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 --2- 172.21.15.53:0/2786986460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c13e360 0x7f314c0cf290 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f31340097a0 tx=0x7f313402b960 comp rx=0 tx=0).stop 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 -- 172.21.15.53:0/2786986460 shutdown_connections 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 --2- 172.21.15.53:0/2786986460 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c13e360 0x7f314c0cf290 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 -- 172.21.15.53:0/2786986460 >> 172.21.15.53:0/2786986460 conn(0x7f314c06a800 msgr2=0x7f314c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 -- 172.21.15.53:0/2786986460 shutdown_connections 2024-01-16T15:07:27.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.184+0000 7f3151400640 1 -- 172.21.15.53:0/2786986460 wait complete. 2024-01-16T15:07:27.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.186+0000 7f3151400640 1 Processor -- start 2024-01-16T15:07:27.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.186+0000 7f3151400640 1 -- start start 2024-01-16T15:07:27.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f3151400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f314c13df80 0x7f314c071ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f3151400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f314c13e360 0x7f314c0b3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f3151400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 0x7f314c15a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f3151400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c071a10 con 0x7f314c15f9c0 2024-01-16T15:07:27.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f3151400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c13d890 con 0x7f314c13df80 2024-01-16T15:07:27.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f3151400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c070e00 con 0x7f314c13e360 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.187+0000 7f314b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 0x7f314c15a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 0x7f314c15a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44028/0 (socket says 172.21.15.53:44028) 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 -- 172.21.15.53:0/83910344 learned_addr learned my addr 172.21.15.53:0/83910344 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f3141c00640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f314c13e360 0x7f314c0b3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314a800640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f314c13df80 0x7f314c071ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 -- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f314c13e360 msgr2=0x7f314c0b3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f314c13e360 0x7f314c0b3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 -- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f314c13df80 msgr2=0x7f314c071ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f314c13df80 0x7f314c071ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 -- 172.21.15.53:0/83910344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3134009400 con 0x7f314c15f9c0 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314a800640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f314c13df80 0x7f314c071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f3141c00640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f314c13e360 0x7f314c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.188+0000 7f314b200640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 0x7f314c15a1f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f314400cbc0 tx=0x7f314400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.189+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f314400aca0 con 0x7f314c15f9c0 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.189+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f314400ae00 con 0x7f314c15f9c0 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.189+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3144012620 con 0x7f314c15f9c0 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.189+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c0710e0 con 0x7f314c15f9c0 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.189+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c0716a0 con 0x7f314c15f9c0 2024-01-16T15:07:27.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.190+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314c071b80 con 0x7f314c15f9c0 2024-01-16T15:07:27.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.191+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f3144007070 con 0x7f314c15f9c0 2024-01-16T15:07:27.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.192+0000 7f3143400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f312406bb30 0x7f312406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:27.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.192+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3144088710 con 0x7f314c15f9c0 2024-01-16T15:07:27.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.193+0000 7f314a800640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f312406bb30 0x7f312406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:27.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.194+0000 7f314a800640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f312406bb30 0x7f312406dff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f313c000f60 tx=0x7f313c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:27.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.197+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3144054d70 con 0x7f314c15f9c0 2024-01-16T15:07:27.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.443+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30237", "target": ["mon-mgr", ""]}) v1 -- 0x7f314c06cff0 con 0x7f312406bb30 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.478+0000 7f3143400640 1 -- 172.21.15.53:0/83910344 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f313c001340 con 0x7f312406bb30 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:27.457 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:27.458 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:27.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f312406bb30 msgr2=0x7f312406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f312406bb30 0x7f312406dff0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f313c000f60 tx=0x7f313c009040 comp rx=0 tx=0).stop 2024-01-16T15:07:27.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 msgr2=0x7f314c15a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:27.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 0x7f314c15a1f0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f314400cbc0 tx=0x7f314400a060 comp rx=0 tx=0).stop 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 shutdown_connections 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f312406bb30 0x7f312406dff0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f314c15f9c0 0x7f314c15a1f0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f314c13e360 0x7f314c0b3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 --2- 172.21.15.53:0/83910344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f314c13df80 0x7f314c071ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.481+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 >> 172.21.15.53:0/83910344 conn(0x7f314c06a800 msgr2=0x7f314c15cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.482+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 shutdown_connections 2024-01-16T15:07:27.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:27.482+0000 7f3151400640 1 -- 172.21.15.53:0/83910344 wait complete. 2024-01-16T15:07:27.484 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:28.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30237 2024-01-16T15:07:28.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.793+0000 7f5f5dc00640 1 Processor -- start 2024-01-16T15:07:28.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.793+0000 7f5f5dc00640 1 -- start start 2024-01-16T15:07:28.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.796+0000 7f5f5dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f580709d0 0x7f5f58164d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.796+0000 7f5f5dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f58071a00 con 0x7f5f58143a50 2024-01-16T15:07:28.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.796+0000 7f5f5dc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f58143e30 con 0x7f5f580cfad0 2024-01-16T15:07:28.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.796+0000 7f5f5dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580b24f0 con 0x7f5f580709d0 2024-01-16T15:07:28.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.796+0000 7f5f56a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5f58143a50 0x7f5f580cf3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55048/0 (socket says 172.21.15.53:55048) 2024-01-16T15:07:28.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.797+0000 7f5f56a00640 1 -- 172.21.15.53:0/3925024201 learned_addr learned my addr 172.21.15.53:0/3925024201 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.797+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484679257 0 0) 0x7f5f58071a00 con 0x7f5f58143a50 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.797+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f38003520 con 0x7f5f58143a50 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661685078 0 0) 0x7f5f38003520 con 0x7f5f58143a50 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f580709d0 msgr2=0x7f5f58164d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f580709d0 0x7f5f58164d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 >> v1:172.21.15.146:6789/0 conn(0x7f5f580cfad0 legacy=0x7f5f580702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f580b1ee0 con 0x7f5f58143a50 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5f40001670 con 0x7f5f58143a50 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 >> v1:172.21.15.53:6789/0 conn(0x7f5f58143a50 legacy=0x7f5f580cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f580709d0 0x7f5f380039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f380040e0 0x7f5f380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f38004e20 0x7f5f3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580b24f0 con 0x7f5f38004e20 2024-01-16T15:07:28.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f38005530 con 0x7f5f380040e0 2024-01-16T15:07:28.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3800e0d0 con 0x7f5f580709d0 2024-01-16T15:07:28.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.798+0000 7f5f57400640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f38004e20 0x7f5f3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f4fe00640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f380040e0 0x7f5f380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f57400640 1 -- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f580709d0 msgr2=0x7f5f380039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f57400640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f580709d0 0x7f5f380039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f57400640 1 -- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f380040e0 msgr2=0x7f5f380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f57400640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f380040e0 0x7f5f380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f57400640 1 -- 172.21.15.53:0/3925024201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f38003520 con 0x7f5f38004e20 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.799+0000 7f5f4fe00640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f380040e0 0x7f5f380047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.800+0000 7f5f57400640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f38004e20 0x7f5f3800b980 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009e10 tx=0x7f5f380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2003b291259d8a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.800+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5002cd50 con 0x7f5f38004e20 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.800+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f50034440 con 0x7f5f38004e20 2024-01-16T15:07:28.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.800+0000 7f5f56000640 1 -- 172.21.15.53:0/3925024201 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f500347f0 con 0x7f5f38004e20 2024-01-16T15:07:28.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f5dc00640 1 -- 172.21.15.53:0/3925024201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f38004e20 msgr2=0x7f5f3800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/3925024201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f38004e20 0x7f5f3800b980 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f5f50009e10 tx=0x7f5f380057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f57400640 1 -- 172.21.15.53:0/3925024201 reap_dead start 2024-01-16T15:07:28.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f5dc00640 1 -- 172.21.15.53:0/3925024201 shutdown_connections 2024-01-16T15:07:28.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f5dc00640 1 -- 172.21.15.53:0/3925024201 >> 172.21.15.53:0/3925024201 conn(0x7f5f5806a800 msgr2=0x7f5f5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:28.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f5dc00640 1 -- 172.21.15.53:0/3925024201 shutdown_connections 2024-01-16T15:07:28.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.802+0000 7f5f5dc00640 1 -- 172.21.15.53:0/3925024201 wait complete. 2024-01-16T15:07:28.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.805+0000 7f5f5dc00640 1 Processor -- start 2024-01-16T15:07:28.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.805+0000 7f5f5dc00640 1 -- start start 2024-01-16T15:07:28.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.806+0000 7f5f5dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f38004e20 0x7f5f581389a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.806+0000 7f5f5dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f581578f0 0x7f5f5815dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.806+0000 7f5f5dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 0x7f5f58154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.806+0000 7f5f5dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f580ad310 con 0x7f5f5815beb0 2024-01-16T15:07:28.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.806+0000 7f5f5dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f581559d0 con 0x7f5f581578f0 2024-01-16T15:07:28.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.806+0000 7f5f5dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5815e180 con 0x7f5f38004e20 2024-01-16T15:07:28.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.807+0000 7f5f57400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 0x7f5f58154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.807+0000 7f5f57400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 0x7f5f58154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44048/0 (socket says 172.21.15.53:44048) 2024-01-16T15:07:28.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.807+0000 7f5f57400640 1 -- 172.21.15.53:0/1617252533 learned_addr learned my addr 172.21.15.53:0/1617252533 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:28.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.807+0000 7f5f56a00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f38004e20 0x7f5f581389a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.808+0000 7f5f4fe00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f581578f0 0x7f5f5815dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.808+0000 7f5f57400640 1 -- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f38004e20 msgr2=0x7f5f581389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.808+0000 7f5f57400640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f38004e20 0x7f5f581389a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.808+0000 7f5f57400640 1 -- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f581578f0 msgr2=0x7f5f5815dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:28.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.808+0000 7f5f57400640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f581578f0 0x7f5f5815dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:28.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.808+0000 7f5f57400640 1 -- 172.21.15.53:0/1617252533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f580ac970 con 0x7f5f5815beb0 2024-01-16T15:07:28.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.809+0000 7f5f57400640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 0x7f5f58154c30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5f50037040 tx=0x7f5f50004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:28.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.810+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5003e410 con 0x7f5f5815beb0 2024-01-16T15:07:28.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.810+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f580acc50 con 0x7f5f5815beb0 2024-01-16T15:07:28.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.810+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5815e3b0 con 0x7f5f5815beb0 2024-01-16T15:07:28.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.811+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5003e590 con 0x7f5f5815beb0 2024-01-16T15:07:28.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.811+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5003e940 con 0x7f5f5815beb0 2024-01-16T15:07:28.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.815+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f28005250 con 0x7f5f5815beb0 2024-01-16T15:07:28.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.815+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f5f5003b050 con 0x7f5f5815beb0 2024-01-16T15:07:28.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.819+0000 7f5f4ea00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f3006bb30 0x7f5f3006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:28.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.819+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5f500aaa60 con 0x7f5f5815beb0 2024-01-16T15:07:28.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.825+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f50077220 con 0x7f5f5815beb0 2024-01-16T15:07:28.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.827+0000 7f5f56a00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f3006bb30 0x7f5f3006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:28.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:28.829+0000 7f5f56a00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f3006bb30 0x7f5f3006dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5f40004ae0 tx=0x7f5f400019b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:29.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.221+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30237", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f28000bc0 con 0x7f5f3006bb30 2024-01-16T15:07:29.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.243+0000 7f5f4ea00640 1 -- 172.21.15.53:0/1617252533 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+359 (secure 0 0 0) 0x7f5f28000bc0 con 0x7f5f3006bb30 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 66.66666666666666, 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "2 GB/3 GB", 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "2/3", 2024-01-16T15:07:29.222 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:07:29.223 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:07:29.223 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:29.223 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:29.223 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:29.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.248+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f3006bb30 msgr2=0x7f5f3006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.248+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f3006bb30 0x7f5f3006dff0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5f40004ae0 tx=0x7f5f400019b0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.248+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 msgr2=0x7f5f58154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:29.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.248+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 0x7f5f58154c30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5f50037040 tx=0x7f5f50004b50 comp rx=0 tx=0).stop 2024-01-16T15:07:29.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 shutdown_connections 2024-01-16T15:07:29.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f3006bb30 0x7f5f3006dff0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f5815beb0 0x7f5f58154c30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f581578f0 0x7f5f5815dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 --2- 172.21.15.53:0/1617252533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f38004e20 0x7f5f581389a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:29.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 >> 172.21.15.53:0/1617252533 conn(0x7f5f5806a800 msgr2=0x7f5f5815a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:29.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.249+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 shutdown_connections 2024-01-16T15:07:29.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:29.250+0000 7f5f5dc00640 1 -- 172.21.15.53:0/1617252533 wait complete. 2024-01-16T15:07:29.259 INFO:tasks.cephfs.test_volumes:current clone state = in-progress 2024-01-16T15:07:30.259 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone30237 2024-01-16T15:07:30.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.516+0000 7f62d9200640 1 Processor -- start 2024-01-16T15:07:30.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.516+0000 7f62d9200640 1 -- start start 2024-01-16T15:07:30.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.518+0000 7f62d9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 0x7f62d4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.518+0000 7f62d9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d415f4e0 con 0x7f62d415bf10 2024-01-16T15:07:30.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.518+0000 7f62d9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62d4071130 con 0x7f62d40ccb50 2024-01-16T15:07:30.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.518+0000 7f62d9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d415f970 con 0x7f62d40adf20 2024-01-16T15:07:30.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.518+0000 7f62cbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f62d415bf10 0x7f62d40cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55050/0 (socket says 172.21.15.53:55050) 2024-01-16T15:07:30.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.518+0000 7f62cbe00640 1 -- 172.21.15.53:0/2143812035 learned_addr learned my addr 172.21.15.53:0/2143812035 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:30.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756754352 0 0) 0x7f62d415f4e0 con 0x7f62d415bf10 2024-01-16T15:07:30.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b4003520 con 0x7f62d415bf10 2024-01-16T15:07:30.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d3400640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 0x7f62d4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 698322629 0 0) 0x7f62b4003520 con 0x7f62d415bf10 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 msgr2=0x7f62d4070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 0x7f62d4070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 >> v1:172.21.15.146:6789/0 conn(0x7f62d40ccb50 legacy=0x7f62d4143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d415c390 con 0x7f62d415bf10 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f62bc001670 con 0x7f62d415bf10 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 >> v1:172.21.15.53:6789/0 conn(0x7f62d415bf10 legacy=0x7f62d40cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62b4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 0x7f62b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62b4004d00 0x7f62b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d415f970 con 0x7f62b40039a0 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b400bc00 con 0x7f62b4004d00 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.519+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62b400bdc0 con 0x7f62d40adf20 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.520+0000 7f62d3400640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62b4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.520+0000 7f62d3400640 1 -- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 msgr2=0x7f62b40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.520+0000 7f62d3400640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40adf20 0x7f62b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.520+0000 7f62d3400640 1 -- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62b4004d00 msgr2=0x7f62b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:30.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.520+0000 7f62d3400640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62b4004d00 0x7f62b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.520+0000 7f62d3400640 1 -- 172.21.15.53:0/2143812035 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b4003520 con 0x7f62b40039a0 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.521+0000 7f62d3400640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62b4003dc0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f62c400ba30 tx=0x7f62b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74dccd66c4f42a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.521+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4032680 con 0x7f62b40039a0 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.521+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c40327e0 con 0x7f62b40039a0 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.521+0000 7f62d2a00640 1 -- 172.21.15.53:0/2143812035 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4032b70 con 0x7f62b40039a0 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d9200640 1 -- 172.21.15.53:0/2143812035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 msgr2=0x7f62b4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d9200640 1 --2- 172.21.15.53:0/2143812035 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62b4003dc0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f62c400ba30 tx=0x7f62b40057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d3e00640 1 -- 172.21.15.53:0/2143812035 reap_dead start 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d9200640 1 -- 172.21.15.53:0/2143812035 shutdown_connections 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d9200640 1 -- 172.21.15.53:0/2143812035 >> 172.21.15.53:0/2143812035 conn(0x7f62d406a7d0 msgr2=0x7f62d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:30.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d9200640 1 -- 172.21.15.53:0/2143812035 shutdown_connections 2024-01-16T15:07:30.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.522+0000 7f62d9200640 1 -- 172.21.15.53:0/2143812035 wait complete. 2024-01-16T15:07:30.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.524+0000 7f62d9200640 1 Processor -- start 2024-01-16T15:07:30.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.524+0000 7f62d9200640 1 -- start start 2024-01-16T15:07:30.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.525+0000 7f62d9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62d40ac050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.525+0000 7f62d9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d41421b0 0x7f62d40c3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.525+0000 7f62d9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40c4240 0x7f62d41f0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.525+0000 7f62d9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d41f1b20 con 0x7f62b40039a0 2024-01-16T15:07:30.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.525+0000 7f62d9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d40c46c0 con 0x7f62d41421b0 2024-01-16T15:07:30.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.525+0000 7f62d9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62d40a9ce0 con 0x7f62d40c4240 2024-01-16T15:07:30.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62d40ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62d40ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44068/0 (socket says 172.21.15.53:44068) 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 -- 172.21.15.53:0/1197159910 learned_addr learned my addr 172.21.15.53:0/1197159910 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62cbe00640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d41421b0 0x7f62d40c3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3e00640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40c4240 0x7f62d41f0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 -- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40c4240 msgr2=0x7f62d41f0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40c4240 0x7f62d41f0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 -- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d41421b0 msgr2=0x7f62d40c3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d41421b0 0x7f62d40c3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62d3400640 1 -- 172.21.15.53:0/1197159910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d41f1840 con 0x7f62b40039a0 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.526+0000 7f62cbe00640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d41421b0 0x7f62d40c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.527+0000 7f62d3400640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62d40ac050 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f62c402bf00 tx=0x7f62c4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.527+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4030a20 con 0x7f62b40039a0 2024-01-16T15:07:30.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.527+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c4030ba0 con 0x7f62b40039a0 2024-01-16T15:07:30.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.527+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c403b420 con 0x7f62b40039a0 2024-01-16T15:07:30.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.527+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d40a9690 con 0x7f62b40039a0 2024-01-16T15:07:30.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.527+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d4142630 con 0x7f62b40039a0 2024-01-16T15:07:30.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.528+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62d4066eb0 con 0x7f62b40039a0 2024-01-16T15:07:30.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.529+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f62c403b580 con 0x7f62b40039a0 2024-01-16T15:07:30.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.530+0000 7f62cb400640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62ac06bb30 0x7f62ac06dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:30.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.531+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f62c40aae40 con 0x7f62b40039a0 2024-01-16T15:07:30.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.531+0000 7f62cbe00640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62ac06bb30 0x7f62ac06dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:30.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.532+0000 7f62cbe00640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62ac06bb30 0x7f62ac06dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f62bc001820 tx=0x7f62bc001930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:30.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.535+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62c4077650 con 0x7f62b40039a0 2024-01-16T15:07:30.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:30.766+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone30237", "target": ["mon-mgr", ""]}) v1 -- 0x7f62d40a9510 con 0x7f62ac06bb30 2024-01-16T15:07:34.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.883+0000 7f62cb400640 1 -- 172.21.15.53:0/1197159910 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f62bc005870 con 0x7f62ac06bb30 2024-01-16T15:07:34.862 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:34.862 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:07:34.862 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:07:34.862 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:34.862 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:34.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.886+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62ac06bb30 msgr2=0x7f62ac06dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:34.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.886+0000 7f62d9200640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62ac06bb30 0x7f62ac06dff0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f62bc001820 tx=0x7f62bc001930 comp rx=0 tx=0).stop 2024-01-16T15:07:34.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.886+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 msgr2=0x7f62d40ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.886+0000 7f62d9200640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62d40ac050 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f62c402bf00 tx=0x7f62c4004730 comp rx=0 tx=0).stop 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 shutdown_connections 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62ac06bb30 0x7f62ac06dff0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62d40c4240 0x7f62d41f0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62d41421b0 0x7f62d40c3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 --2- 172.21.15.53:0/1197159910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62b40039a0 0x7f62d40ac050 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 >> 172.21.15.53:0/1197159910 conn(0x7f62d406a7d0 msgr2=0x7f62d40713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 shutdown_connections 2024-01-16T15:07:34.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:34.887+0000 7f62d9200640 1 -- 172.21.15.53:0/1197159910 wait complete. 2024-01-16T15:07:34.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:07:35.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.132+0000 7f6f52400640 1 Processor -- start 2024-01-16T15:07:35.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.133+0000 7f6f52400640 1 -- start start 2024-01-16T15:07:35.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.134+0000 7f6f52400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f52400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6f4c071f00 0x7f6f4c166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f52400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f4c0b2e10 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f52400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f4c06cc20 con 0x7f6f4c071f00 2024-01-16T15:07:35.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f52400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c06cec0 con 0x7f6f4c071410 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6f4c071f00 0x7f6f4c166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6f4c071f00 0x7f6f4c166cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48128/0 (socket says 172.21.15.53:48128) 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51a00640 1 -- 172.21.15.53:0/437700239 learned_addr learned my addr 172.21.15.53:0/437700239 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51000640 1 --2- 172.21.15.53:0/437700239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51a00640 1 -- 172.21.15.53:0/437700239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 msgr2=0x7f6f4c146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51a00640 1 --2- 172.21.15.53:0/437700239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c146770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.135+0000 7f6f51a00640 1 -- 172.21.15.53:0/437700239 >> v1:172.21.15.146:6789/0 conn(0x7f6f4c06c7c0 legacy=0x7f6f4c0717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:35.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.136+0000 7f6f51a00640 1 -- 172.21.15.53:0/437700239 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f4c06d1a0 con 0x7f6f4c071f00 2024-01-16T15:07:35.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.137+0000 7f6f51a00640 1 --2- 172.21.15.53:0/437700239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c071f00 0x7f6f4c166cf0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009e50 tx=0x7f6f4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9422ebd0b3694 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.137+0000 7f6f4b400640 1 -- 172.21.15.53:0/437700239 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4402ca70 con 0x7f6f4c071f00 2024-01-16T15:07:35.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.137+0000 7f6f4b400640 1 -- 172.21.15.53:0/437700239 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4402cbd0 con 0x7f6f4c071f00 2024-01-16T15:07:35.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.137+0000 7f6f4b400640 1 -- 172.21.15.53:0/437700239 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f440343f0 con 0x7f6f4c071f00 2024-01-16T15:07:35.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 -- 172.21.15.53:0/437700239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c071f00 msgr2=0x7f6f4c166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 --2- 172.21.15.53:0/437700239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c071f00 0x7f6f4c166cf0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f6f44009e50 tx=0x7f6f4402bed0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 -- 172.21.15.53:0/437700239 shutdown_connections 2024-01-16T15:07:35.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 --2- 172.21.15.53:0/437700239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c071f00 0x7f6f4c166cf0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 --2- 172.21.15.53:0/437700239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 -- 172.21.15.53:0/437700239 >> 172.21.15.53:0/437700239 conn(0x7f6f4c06a800 msgr2=0x7f6f4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:35.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 -- 172.21.15.53:0/437700239 shutdown_connections 2024-01-16T15:07:35.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.138+0000 7f6f52400640 1 -- 172.21.15.53:0/437700239 wait complete. 2024-01-16T15:07:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.140+0000 7f6f52400640 1 Processor -- start 2024-01-16T15:07:35.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.140+0000 7f6f52400640 1 -- start start 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f52400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 0x7f6f4c1f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f52400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c13aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f52400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f4c071f00 0x7f6f4c15c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f52400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c1f6fc0 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f52400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c141510 con 0x7f6f4c071f00 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f52400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4c1417f0 con 0x7f6f4c071410 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.141+0000 7f6f51000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 0x7f6f4c1f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 0x7f6f4c1f6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48142/0 (socket says 172.21.15.53:48142) 2024-01-16T15:07:35.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 -- 172.21.15.53:0/674288715 learned_addr learned my addr 172.21.15.53:0/674288715 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 -- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 msgr2=0x7f6f4c13aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c13aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 -- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f4c071f00 msgr2=0x7f6f4c15c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f4c071f00 0x7f6f4c15c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 -- 172.21.15.53:0/674288715 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f44009b30 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.142+0000 7f6f51000640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 0x7f6f4c1f6a80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c00c7f0 tx=0x7f6f3c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.143+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c004220 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.143+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f3c0043a0 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.143+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3c007860 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.143+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4c141ad0 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.143+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4c142090 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.144+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f4c0b3fb0 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.145+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6f3c007bd0 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.147+0000 7f6f48c00640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f2406bb30 0x7f6f2406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.147+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f3c088630 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.147+0000 7f6f4be00640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f2406bb30 0x7f6f2406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.149+0000 7f6f4be00640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f2406bb30 0x7f6f2406dff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6f34006fd0 tx=0x7f6f34006bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.152+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f3c054d20 con 0x7f6f4c06c7c0 2024-01-16T15:07:35.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.373+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f4c06da40 con 0x7f6f2406bb30 2024-01-16T15:07:35.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.377+0000 7f6f48c00640 1 -- 172.21.15.53:0/674288715 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f6f34013430 con 0x7f6f2406bb30 2024-01-16T15:07:35.355 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:07:35.355 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.356 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:07:35.356 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:35.356 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:07:35.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f2406bb30 msgr2=0x7f6f2406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f2406bb30 0x7f6f2406dff0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f6f34006fd0 tx=0x7f6f34006bc0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 msgr2=0x7f6f4c1f6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 0x7f6f4c1f6a80 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f6f3c00c7f0 tx=0x7f6f3c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 shutdown_connections 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f2406bb30 0x7f6f2406dff0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f4c071f00 0x7f6f4c15c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f4c071410 0x7f6f4c13aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 --2- 172.21.15.53:0/674288715 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f4c06c7c0 0x7f6f4c1f6a80 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 >> 172.21.15.53:0/674288715 conn(0x7f6f4c06a800 msgr2=0x7f6f4c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 shutdown_connections 2024-01-16T15:07:35.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.380+0000 7f6f52400640 1 -- 172.21.15.53:0/674288715 wait complete. 2024-01-16T15:07:35.379 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:07:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.627+0000 7fb7ae600640 1 Processor -- start 2024-01-16T15:07:35.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.627+0000 7fb7ae600640 1 -- start start 2024-01-16T15:07:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7ae600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a815b2a0 0x7fb7a80cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7ae600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb7a80cfaf0 0x7fb7a80cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7ae600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7a80d00a0 con 0x7fb7a80cdea0 2024-01-16T15:07:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7ae600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7a80ce330 con 0x7fb7a80cfaf0 2024-01-16T15:07:35.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7ae600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a80b4010 con 0x7fb7a815b2a0 2024-01-16T15:07:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7a7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb7a80cdea0 0x7fb7a80cbd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58894/0 (socket says 172.21.15.53:58894) 2024-01-16T15:07:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7a7e00640 1 -- 172.21.15.53:0/4090101176 learned_addr learned my addr 172.21.15.53:0/4090101176 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:35.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7a6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb7a80cfaf0 0x7fb7a80cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.629+0000 7fb7a7400640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a815b2a0 0x7fb7a80cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a6000640 1 -- 172.21.15.53:0/4090101176 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4287126538 0 0) 0x7fb7a80d00a0 con 0x7fb7a80cdea0 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a6000640 1 -- 172.21.15.53:0/4090101176 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb788003520 con 0x7fb7a80cdea0 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a6a00640 1 -- 172.21.15.53:0/4090101176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a815b2a0 msgr2=0x7fb7a80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a6a00640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a815b2a0 0x7fb7a80cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a6a00640 1 -- 172.21.15.53:0/4090101176 >> v1:172.21.15.53:6789/0 conn(0x7fb7a80cdea0 legacy=0x7fb7a80cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a6a00640 1 -- 172.21.15.53:0/4090101176 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7a80b42f0 con 0x7fb7a80cfaf0 2024-01-16T15:07:35.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.630+0000 7fb7a7400640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a815b2a0 0x7fb7a80cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.631+0000 7fb7a6a00640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cfaf0 0x7fb7a80cd960 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb7900097a0 tx=0x7fb79002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8437f626db597e2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.631+0000 7fb7a6000640 1 -- 172.21.15.53:0/4090101176 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790031830 con 0x7fb7a80cfaf0 2024-01-16T15:07:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.631+0000 7fb7a6000640 1 -- 172.21.15.53:0/4090101176 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb790031990 con 0x7fb7a80cfaf0 2024-01-16T15:07:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.631+0000 7fb7a6000640 1 -- 172.21.15.53:0/4090101176 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb790031d20 con 0x7fb7a80cfaf0 2024-01-16T15:07:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 -- 172.21.15.53:0/4090101176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cfaf0 msgr2=0x7fb7a80cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cfaf0 0x7fb7a80cd960 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb7900097a0 tx=0x7fb79002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 -- 172.21.15.53:0/4090101176 shutdown_connections 2024-01-16T15:07:35.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cfaf0 0x7fb7a80cd960 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 --2- 172.21.15.53:0/4090101176 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a815b2a0 0x7fb7a80cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 -- 172.21.15.53:0/4090101176 >> 172.21.15.53:0/4090101176 conn(0x7fb7a80d1020 msgr2=0x7fb7a80d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:35.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 -- 172.21.15.53:0/4090101176 shutdown_connections 2024-01-16T15:07:35.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.632+0000 7fb7ae600640 1 -- 172.21.15.53:0/4090101176 wait complete. 2024-01-16T15:07:35.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.634+0000 7fb7ae600640 1 Processor -- start 2024-01-16T15:07:35.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.634+0000 7fb7ae600640 1 -- start start 2024-01-16T15:07:35.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7ae600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 0x7fb7a81566a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7ae600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a80cfaf0 0x7fb7a81f3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7ae600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a815b2a0 0x7fb7a815c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7ae600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a81f3400 con 0x7fb7a80cdea0 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7ae600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a815ba40 con 0x7fb7a815b2a0 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7ae600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7a815bd20 con 0x7fb7a80cfaf0 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 0x7fb7a81566a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 0x7fb7a81566a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48150/0 (socket says 172.21.15.53:48150) 2024-01-16T15:07:35.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.635+0000 7fb7a7400640 1 -- 172.21.15.53:0/3462875482 learned_addr learned my addr 172.21.15.53:0/3462875482 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:35.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a6a00640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a80cfaf0 0x7fb7a81f3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7e00640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a815b2a0 0x7fb7a815c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7400640 1 -- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a80cfaf0 msgr2=0x7fb7a81f3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7400640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a80cfaf0 0x7fb7a81f3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7400640 1 -- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a815b2a0 msgr2=0x7fb7a815c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7400640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a815b2a0 0x7fb7a815c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7400640 1 -- 172.21.15.53:0/3462875482 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb790009400 con 0x7fb7a80cdea0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7a7400640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 0x7fb7a81566a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb798009fa0 tx=0x7fb79800d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798003b90 con 0x7fb7a80cdea0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb79800f040 con 0x7fb7a80cdea0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.636+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7a81f3020 con 0x7fb7a80cdea0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.637+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb798004030 con 0x7fb7a80cdea0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.637+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7a8158de0 con 0x7fb7a80cdea0 2024-01-16T15:07:35.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.638+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a81f3670 con 0x7fb7a80cdea0 2024-01-16T15:07:35.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.639+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb798004190 con 0x7fb7a80cdea0 2024-01-16T15:07:35.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.640+0000 7fb79f400640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb78006bb30 0x7fb78006dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:35.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.640+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb798088870 con 0x7fb7a80cdea0 2024-01-16T15:07:35.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.640+0000 7fb7a6a00640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb78006bb30 0x7fb78006dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:35.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.642+0000 7fb7a6a00640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb78006bb30 0x7fb78006dff0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb790036000 tx=0x7fb79003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:35.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.648+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb798055010 con 0x7fb7a80cdea0 2024-01-16T15:07:35.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.870+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb7a80b48b0 con 0x7fb78006bb30 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.892+0000 7fb79f400640 1 -- 172.21.15.53:0/3462875482 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7fb790031b80 con 0x7fb78006bb30 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30233" 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30232" 2024-01-16T15:07:35.870 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30235" 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30231" 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30237" 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30236" 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone30234" 2024-01-16T15:07:35.871 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:35.872 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:07:35.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb78006bb30 msgr2=0x7fb78006dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb78006bb30 0x7fb78006dff0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fb790036000 tx=0x7fb79003c040 comp rx=0 tx=0).stop 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 msgr2=0x7fb7a81566a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 0x7fb7a81566a0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fb798009fa0 tx=0x7fb79800d560 comp rx=0 tx=0).stop 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 shutdown_connections 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb78006bb30 0x7fb78006dff0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7a815b2a0 0x7fb7a815c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7a80cfaf0 0x7fb7a81f3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 --2- 172.21.15.53:0/3462875482 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb7a80cdea0 0x7fb7a81566a0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 >> 172.21.15.53:0/3462875482 conn(0x7fb7a80d1020 msgr2=0x7fb7a806b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 shutdown_connections 2024-01-16T15:07:35.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:35.894+0000 7fb7ae600640 1 -- 172.21.15.53:0/3462875482 wait complete. 2024-01-16T15:07:35.893 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30233 2024-01-16T15:07:36.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.148+0000 7f6f23e00640 1 Processor -- start 2024-01-16T15:07:36.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.148+0000 7f6f23e00640 1 -- start start 2024-01-16T15:07:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.149+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 0x7f6f1c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.149+0000 7f6f23e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f1c06c6b0 0x7f6f1c099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f23e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f1c06cea0 con 0x7f6f1c099fa0 2024-01-16T15:07:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f23e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f1c09a430 con 0x7f6f1c06c6b0 2024-01-16T15:07:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f23e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c06cbc0 con 0x7f6f1c0cd240 2024-01-16T15:07:36.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f21e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6f1c099fa0 0x7f6f1c06b4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58908/0 (socket says 172.21.15.53:58908) 2024-01-16T15:07:36.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f21e00640 1 -- 172.21.15.53:0/3352230349 learned_addr learned my addr 172.21.15.53:0/3352230349 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:36.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f21400640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 0x7f6f1c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f20a00640 1 --2- 172.21.15.53:0/3352230349 >> v2:172.21.15.146:3300/0 conn(0x7f6f1c06c6b0 0x7f6f1c099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.150+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698018535 0 0) 0x7f6f1c06cea0 con 0x7f6f1c099fa0 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6efc003520 con 0x7f6f1c099fa0 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1143653251 0 0) 0x7f6efc003520 con 0x7f6f1c099fa0 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 msgr2=0x7f6f1c0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 0x7f6f1c0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 >> v2:172.21.15.146:3300/0 conn(0x7f6f1c06c6b0 msgr2=0x7f6f1c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 --2- 172.21.15.53:0/3352230349 >> v2:172.21.15.146:3300/0 conn(0x7f6f1c06c6b0 0x7f6f1c099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c06f350 con 0x7f6f1c099fa0 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6f14001670 con 0x7f6f1c099fa0 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 >> v1:172.21.15.53:6789/0 conn(0x7f6f1c099fa0 legacy=0x7f6f1c06b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6efc003970 0x7f6efc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc0047c0 0x7f6efc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 0x7f6efc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c09a430 con 0x7f6efc003970 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c06cbc0 con 0x7f6efc0047c0 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6efc00b980 con 0x7f6f1c0cd240 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.151+0000 7f6f21400640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6efc003970 0x7f6efc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.152+0000 7f6f20a00640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc0047c0 0x7f6efc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.152+0000 7f6f21400640 1 -- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 msgr2=0x7f6efc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.152+0000 7f6f21400640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f1c0cd240 0x7f6efc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.152+0000 7f6f21400640 1 -- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc0047c0 msgr2=0x7f6efc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.152+0000 7f6f21400640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6efc0047c0 0x7f6efc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.152+0000 7f6f21400640 1 -- 172.21.15.53:0/3352230349 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6efc003520 con 0x7f6efc003970 2024-01-16T15:07:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.153+0000 7f6f21400640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6efc003970 0x7f6efc003d90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ac0 tx=0x7f6efc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf73b96c2f58bf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.153+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c02c6d0 con 0x7f6efc003970 2024-01-16T15:07:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.153+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f0c02c830 con 0x7f6efc003970 2024-01-16T15:07:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.153+0000 7f6f1be00640 1 -- 172.21.15.53:0/3352230349 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c02cbc0 con 0x7f6efc003970 2024-01-16T15:07:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f23e00640 1 -- 172.21.15.53:0/3352230349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6efc003970 msgr2=0x7f6efc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3352230349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6efc003970 0x7f6efc003d90 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ac0 tx=0x7f6efc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f21e00640 1 -- 172.21.15.53:0/3352230349 reap_dead start 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f23e00640 1 -- 172.21.15.53:0/3352230349 shutdown_connections 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f23e00640 1 -- 172.21.15.53:0/3352230349 >> 172.21.15.53:0/3352230349 conn(0x7f6f1c0b2d00 msgr2=0x7f6f1c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f23e00640 1 -- 172.21.15.53:0/3352230349 shutdown_connections 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.154+0000 7f6f23e00640 1 -- 172.21.15.53:0/3352230349 wait complete. 2024-01-16T15:07:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.156+0000 7f6f23e00640 1 Processor -- start 2024-01-16T15:07:36.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.156+0000 7f6f23e00640 1 -- start start 2024-01-16T15:07:36.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc003970 0x7f6f1c1f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 0x7f6f1c0c6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f23e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f1c0cb2c0 0x7f6f1c0c6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f23e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c1f2b80 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f23e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c0c5240 con 0x7f6f1c0cb2c0 2024-01-16T15:07:36.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f23e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1c0c8930 con 0x7f6efc003970 2024-01-16T15:07:36.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.157+0000 7f6f20a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 0x7f6f1c0c6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 0x7f6f1c0c6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48156/0 (socket says 172.21.15.53:48156) 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 -- 172.21.15.53:0/3996144551 learned_addr learned my addr 172.21.15.53:0/3996144551 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f21400640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc003970 0x7f6f1c1f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f21e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f1c0cb2c0 0x7f6f1c0c6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 -- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc003970 msgr2=0x7f6f1c1f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc003970 0x7f6f1c1f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 -- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f1c0cb2c0 msgr2=0x7f6f1c0c6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f1c0cb2c0 0x7f6f1c0c6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 -- 172.21.15.53:0/3996144551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c0c8650 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.158+0000 7f6f20a00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 0x7f6f1c0c6440 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009fd0 tx=0x7f6f040115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.159+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0400e030 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.159+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f04003990 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.159+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f04003b20 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.159+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1c0c54e0 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.159+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f1c0c6ec0 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.160+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f1c0c50a0 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.161+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7f6f04003c80 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.162+0000 7f6f19600640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ef406bb30 0x7f6ef406dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.162+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6f0400f070 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.163+0000 7f6f21400640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ef406bb30 0x7f6ef406dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.164+0000 7f6f21400640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ef406bb30 0x7f6ef406dff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c036000 tx=0x7f6f0c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.166+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f1c0c50a0 con 0x7f6f1c0c7ba0 2024-01-16T15:07:36.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.395+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30233", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f1c0cd620 con 0x7f6ef406bb30 2024-01-16T15:07:36.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.437+0000 7f6f19600640 1 -- 172.21.15.53:0/3996144551 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f0c043070 con 0x7f6ef406bb30 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ef406bb30 msgr2=0x7f6ef406dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ef406bb30 0x7f6ef406dff0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c036000 tx=0x7f6f0c030000 comp rx=0 tx=0).stop 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 msgr2=0x7f6f1c0c6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 0x7f6f1c0c6440 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009fd0 tx=0x7f6f040115a0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 shutdown_connections 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ef406bb30 0x7f6ef406dff0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f1c0cb2c0 0x7f6f1c0c6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f1c0c7ba0 0x7f6f1c0c6440 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 --2- 172.21.15.53:0/3996144551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6efc003970 0x7f6f1c1f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 >> 172.21.15.53:0/3996144551 conn(0x7f6f1c0b2d00 msgr2=0x7f6f1c0cf4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 shutdown_connections 2024-01-16T15:07:36.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.439+0000 7f6f23e00640 1 -- 172.21.15.53:0/3996144551 wait complete. 2024-01-16T15:07:36.424 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.446+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.424 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.446+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.424 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.446+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.424 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.446+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.425 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.446+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:07:36.448 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.470+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.448 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.470+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.448 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.470+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.448 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.470+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.448 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:07:36.470+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:07:36.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.668+0000 7fb2be800640 1 Processor -- start 2024-01-16T15:07:36.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.668+0000 7fb2be800640 1 -- start start 2024-01-16T15:07:36.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2be800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b80cc500 0x7fb2b80cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2be800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb2b806d160 0x7fb2b806d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2be800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b8070e50 con 0x7fb2b8143c40 2024-01-16T15:07:36.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2be800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b80723c0 con 0x7fb2b806d160 2024-01-16T15:07:36.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2be800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b8144020 con 0x7fb2b80cc500 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2bd400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb2b8143c40 0x7fb2b8071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58920/0 (socket says 172.21.15.53:58920) 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2bd400640 1 -- 172.21.15.53:0/1125576928 learned_addr learned my addr 172.21.15.53:0/1125576928 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2bde00640 1 --2- 172.21.15.53:0/1125576928 >> v2:172.21.15.53:3300/0 conn(0x7fb2b806d160 0x7fb2b806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2bca00640 1 --2- 172.21.15.53:0/1125576928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b80cc500 0x7fb2b80cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.670+0000 7fb2b7e00640 1 -- 172.21.15.53:0/1125576928 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319187156 0 0) 0x7fb2b8070e50 con 0x7fb2b8143c40 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.671+0000 7fb2b7e00640 1 -- 172.21.15.53:0/1125576928 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb298003520 con 0x7fb2b8143c40 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.671+0000 7fb2bde00640 1 -- 172.21.15.53:0/1125576928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b80cc500 msgr2=0x7fb2b80cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.671+0000 7fb2bde00640 1 --2- 172.21.15.53:0/1125576928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b80cc500 0x7fb2b80cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.671+0000 7fb2bde00640 1 -- 172.21.15.53:0/1125576928 >> v1:172.21.15.53:6789/0 conn(0x7fb2b8143c40 legacy=0x7fb2b8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.671+0000 7fb2bde00640 1 -- 172.21.15.53:0/1125576928 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b80ccea0 con 0x7fb2b806d160 2024-01-16T15:07:36.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.672+0000 7fb2bde00640 1 --2- 172.21.15.53:0/1125576928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b806d160 0x7fb2b806d590 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009d90 tx=0x7fb2b002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f554df0b58b3c75 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.672+0000 7fb2b7e00640 1 -- 172.21.15.53:0/1125576928 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b002cbc0 con 0x7fb2b806d160 2024-01-16T15:07:36.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.672+0000 7fb2b7e00640 1 -- 172.21.15.53:0/1125576928 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b002cd20 con 0x7fb2b806d160 2024-01-16T15:07:36.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.672+0000 7fb2b7e00640 1 -- 172.21.15.53:0/1125576928 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0034640 con 0x7fb2b806d160 2024-01-16T15:07:36.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 -- 172.21.15.53:0/1125576928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b806d160 msgr2=0x7fb2b806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 --2- 172.21.15.53:0/1125576928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b806d160 0x7fb2b806d590 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009d90 tx=0x7fb2b002be10 comp rx=0 tx=0).stop 2024-01-16T15:07:36.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 -- 172.21.15.53:0/1125576928 shutdown_connections 2024-01-16T15:07:36.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 --2- 172.21.15.53:0/1125576928 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b806d160 0x7fb2b806d590 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 --2- 172.21.15.53:0/1125576928 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b80cc500 0x7fb2b80cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 -- 172.21.15.53:0/1125576928 >> 172.21.15.53:0/1125576928 conn(0x7fb2b806a800 msgr2=0x7fb2b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:36.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 -- 172.21.15.53:0/1125576928 shutdown_connections 2024-01-16T15:07:36.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.673+0000 7fb2be800640 1 -- 172.21.15.53:0/1125576928 wait complete. 2024-01-16T15:07:36.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.675+0000 7fb2be800640 1 Processor -- start 2024-01-16T15:07:36.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.675+0000 7fb2be800640 1 -- start start 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2be800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b806d160 0x7fb2b815a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2be800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b80cc500 0x7fb2b80c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2be800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 0x7fb2b80c7340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2be800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b80cb3f0 con 0x7fb2b8143c40 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2be800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b815ae30 con 0x7fb2b80cc500 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2be800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b80ca770 con 0x7fb2b806d160 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2bde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 0x7fb2b80c7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2bde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 0x7fb2b80c7340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48172/0 (socket says 172.21.15.53:48172) 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.676+0000 7fb2bde00640 1 -- 172.21.15.53:0/4223444881 learned_addr learned my addr 172.21.15.53:0/4223444881 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bd400640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b806d160 0x7fb2b815a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bca00640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b80cc500 0x7fb2b80c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bde00640 1 -- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b806d160 msgr2=0x7fb2b815a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bde00640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b806d160 0x7fb2b815a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bde00640 1 -- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b80cc500 msgr2=0x7fb2b80c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bde00640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b80cc500 0x7fb2b80c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bde00640 1 -- 172.21.15.53:0/4223444881 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b0009a70 con 0x7fb2b8143c40 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bd400640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b806d160 0x7fb2b815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bca00640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b80cc500 0x7fb2b80c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:36.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2bde00640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 0x7fb2b80c7340 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009eb0 tx=0x7fb2b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b003e610 con 0x7fb2b8143c40 2024-01-16T15:07:36.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2b002ce90 con 0x7fb2b8143c40 2024-01-16T15:07:36.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2b003e9f0 con 0x7fb2b8143c40 2024-01-16T15:07:36.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b80caa50 con 0x7fb2b8143c40 2024-01-16T15:07:36.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.677+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b80cb010 con 0x7fb2b8143c40 2024-01-16T15:07:36.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.679+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b80ce210 con 0x7fb2b8143c40 2024-01-16T15:07:36.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.680+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 19) v1 ==== 90304+0+0 (secure 0 0 0) 0x7fb2b003b050 con 0x7fb2b8143c40 2024-01-16T15:07:36.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.680+0000 7fb2b5600640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb29006bb60 0x7fb29006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:36.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.681+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb2b00aa640 con 0x7fb2b8143c40 2024-01-16T15:07:36.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.681+0000 7fb2bd400640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb29006bb60 0x7fb29006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:36.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.682+0000 7fb2bd400640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb29006bb60 0x7fb29006e020 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005880 tx=0x7fb2a8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:36.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.686+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2b0076d90 con 0x7fb2b8143c40 2024-01-16T15:07:36.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:36.912+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2b806c960 con 0x7fb29006bb60 2024-01-16T15:07:39.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.307+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fb2b0077160 con 0x7fb2b8143c40 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.892+0000 7fb2b5600640 1 -- 172.21.15.53:0/4223444881 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fb2a8001350 con 0x7fb29006bb60 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:39.871 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:07:39.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb29006bb60 msgr2=0x7fb29006e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb29006bb60 0x7fb29006e020 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8005880 tx=0x7fb2a8009040 comp rx=0 tx=0).stop 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 msgr2=0x7fb2b80c7340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 0x7fb2b80c7340 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb2b0009eb0 tx=0x7fb2b0004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 shutdown_connections 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb29006bb60 0x7fb29006e020 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b8143c40 0x7fb2b80c7340 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b80cc500 0x7fb2b80c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 --2- 172.21.15.53:0/4223444881 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b806d160 0x7fb2b815a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 >> 172.21.15.53:0/4223444881 conn(0x7fb2b806a800 msgr2=0x7fb2b80acbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 shutdown_connections 2024-01-16T15:07:39.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:39.896+0000 7fb2be800640 1 -- 172.21.15.53:0/4223444881 wait complete. 2024-01-16T15:07:39.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:07:40.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.150+0000 7f995be00640 1 Processor -- start 2024-01-16T15:07:40.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.150+0000 7f995be00640 1 -- start start 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.151+0000 7f995be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f995c1447f0 0x7f995c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.151+0000 7f995be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c0cccf0 0x7f995c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.151+0000 7f995be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995c15c5c0 con 0x7f995c06c470 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.151+0000 7f995be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f995c06d690 con 0x7f995c1447f0 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.151+0000 7f995be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0cebb0 con 0x7f995c0cccf0 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f995c1447f0 0x7f995c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995a000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f995c06c470 0x7f995c0cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58922/0 (socket says 172.21.15.53:58922) 2024-01-16T15:07:40.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995a000640 1 -- 172.21.15.53:0/3947956123 learned_addr learned my addr 172.21.15.53:0/3947956123 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:40.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995b400640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c0cccf0 0x7f995c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995aa00640 1 -- 172.21.15.53:0/3947956123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c0cccf0 msgr2=0x7f995c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:40.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995aa00640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c0cccf0 0x7f995c15c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995aa00640 1 -- 172.21.15.53:0/3947956123 >> v1:172.21.15.53:6789/0 conn(0x7f995c06c470 legacy=0x7f995c0cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995aa00640 1 -- 172.21.15.53:0/3947956123 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c06ca10 con 0x7f995c1447f0 2024-01-16T15:07:40.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.152+0000 7f995b400640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c0cccf0 0x7f995c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:40.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.154+0000 7f995aa00640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c1447f0 0x7f995c15c080 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f995c0cecf0 tx=0x7f994c02b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f76f9e17a42dcae2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:40.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.154+0000 7f9959600640 1 -- 172.21.15.53:0/3947956123 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c031770 con 0x7f995c1447f0 2024-01-16T15:07:40.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.154+0000 7f9959600640 1 -- 172.21.15.53:0/3947956123 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c0318d0 con 0x7f995c1447f0 2024-01-16T15:07:40.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.154+0000 7f9959600640 1 -- 172.21.15.53:0/3947956123 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c031c60 con 0x7f995c1447f0 2024-01-16T15:07:40.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.154+0000 7f995be00640 1 -- 172.21.15.53:0/3947956123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c1447f0 msgr2=0x7f995c15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.154+0000 7f995be00640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c1447f0 0x7f995c15c080 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f995c0cecf0 tx=0x7f994c02b810 comp rx=0 tx=0).stop 2024-01-16T15:07:40.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.155+0000 7f995be00640 1 -- 172.21.15.53:0/3947956123 shutdown_connections 2024-01-16T15:07:40.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.155+0000 7f995be00640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c0cccf0 0x7f995c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.155+0000 7f995be00640 1 --2- 172.21.15.53:0/3947956123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c1447f0 0x7f995c15c080 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.155+0000 7f995be00640 1 -- 172.21.15.53:0/3947956123 >> 172.21.15.53:0/3947956123 conn(0x7f995c06a3e0 msgr2=0x7f995c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:40.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.155+0000 7f995be00640 1 -- 172.21.15.53:0/3947956123 shutdown_connections 2024-01-16T15:07:40.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.155+0000 7f995be00640 1 -- 172.21.15.53:0/3947956123 wait complete. 2024-01-16T15:07:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.157+0000 7f995be00640 1 Processor -- start 2024-01-16T15:07:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.157+0000 7f995be00640 1 -- start start 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 0x7f995c0c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f995c0cccf0 0x7f995c0c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c1447f0 0x7f995c1350e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0aa830 con 0x7f995c06c470 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0aab10 con 0x7f995c0cccf0 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f995c0aadf0 con 0x7f995c1447f0 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 0x7f995c0c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 0x7f995c0c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48194/0 (socket says 172.21.15.53:48194) 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995aa00640 1 -- 172.21.15.53:0/2141447937 learned_addr learned my addr 172.21.15.53:0/2141447937 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995aa00640 1 -- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c1447f0 msgr2=0x7f995c1350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995a000640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f995c0cccf0 0x7f995c0c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995b400640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c1447f0 0x7f995c1350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995aa00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c1447f0 0x7f995c1350e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.158+0000 7f995aa00640 1 -- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f995c0cccf0 msgr2=0x7f995c0c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f995aa00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f995c0cccf0 0x7f995c0c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f995aa00640 1 -- 172.21.15.53:0/2141447937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c009400 con 0x7f995c06c470 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f995a000640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f995c0cccf0 0x7f995c0c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f995aa00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 0x7f995c0c3de0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f994c000c00 tx=0x7f994c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c0328d0 con 0x7f995c06c470 2024-01-16T15:07:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994c032a50 con 0x7f995c06c470 2024-01-16T15:07:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f994c032e00 con 0x7f995c06c470 2024-01-16T15:07:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.159+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995c0ab0d0 con 0x7f995c06c470 2024-01-16T15:07:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.160+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f995c0c4320 con 0x7f995c06c470 2024-01-16T15:07:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.160+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f995c067450 con 0x7f995c06c470 2024-01-16T15:07:40.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.162+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f994c04e450 con 0x7f995c06c470 2024-01-16T15:07:40.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.162+0000 7f9953400640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f993406bc50 0x7f993406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.163+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f994c0aa8c0 con 0x7f995c06c470 2024-01-16T15:07:40.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.163+0000 7f995a000640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f993406bc50 0x7f993406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.164+0000 7f995a000640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f993406bc50 0x7f993406e110 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9944007fd0 tx=0x7f9944009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:40.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.167+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f994c0770a0 con 0x7f995c06c470 2024-01-16T15:07:40.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.401+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f995c06d2b0 con 0x7f993406bc50 2024-01-16T15:07:40.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.470+0000 7f9953400640 1 -- 172.21.15.53:0/2141447937 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f99440013d0 con 0x7f993406bc50 2024-01-16T15:07:40.448 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f993406bc50 msgr2=0x7f993406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f993406bc50 0x7f993406e110 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f9944007fd0 tx=0x7f9944009040 comp rx=0 tx=0).stop 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 msgr2=0x7f995c0c3de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 0x7f995c0c3de0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f994c000c00 tx=0x7f994c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 shutdown_connections 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f993406bc50 0x7f993406e110 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f995c1447f0 0x7f995c1350e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f995c0cccf0 0x7f995c0c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 --2- 172.21.15.53:0/2141447937 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f995c06c470 0x7f995c0c3de0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 >> 172.21.15.53:0/2141447937 conn(0x7f995c06a3e0 msgr2=0x7f995c0af440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:40.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 shutdown_connections 2024-01-16T15:07:40.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.473+0000 7f995be00640 1 -- 172.21.15.53:0/2141447937 wait complete. 2024-01-16T15:07:40.474 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:07:40.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.726+0000 7fd1f7200640 1 Processor -- start 2024-01-16T15:07:40.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.726+0000 7fd1f7200640 1 -- start start 2024-01-16T15:07:40.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.727+0000 7fd1f7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd1f80cfde0 0x7fd1f80b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f80d1c60 con 0x7fd1f815d4b0 2024-01-16T15:07:40.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1f80b2730 con 0x7fd1f80cf9a0 2024-01-16T15:07:40.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1f80d1ec0 con 0x7fd1f80cfde0 2024-01-16T15:07:40.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f5e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd1f815d4b0 0x7fd1f80d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58936/0 (socket says 172.21.15.53:58936) 2024-01-16T15:07:40.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f5e00640 1 -- 172.21.15.53:0/1377216466 learned_addr learned my addr 172.21.15.53:0/1377216466 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f6800640 1 --2- 172.21.15.53:0/1377216466 >> v2:172.21.15.146:3300/0 conn(0x7fd1f80cfde0 0x7fd1f80b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225119860 0 0) 0x7fd1f80d1c60 con 0x7fd1f815d4b0 2024-01-16T15:07:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.728+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d8003520 con 0x7fd1f815d4b0 2024-01-16T15:07:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58378176 0 0) 0x7fd1d8003520 con 0x7fd1f815d4b0 2024-01-16T15:07:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 >> v2:172.21.15.146:3300/0 conn(0x7fd1f80cfde0 msgr2=0x7fd1f80b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:40.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 --2- 172.21.15.53:0/1377216466 >> v2:172.21.15.146:3300/0 conn(0x7fd1f80cfde0 0x7fd1f80b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 >> v1:172.21.15.146:6789/0 conn(0x7fd1f80cf9a0 legacy=0x7fd1f80b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80b29d0 con 0x7fd1f815d4b0 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd1e8001670 con 0x7fd1f815d4b0 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 >> v1:172.21.15.53:6789/0 conn(0x7fd1f815d4b0 legacy=0x7fd1f80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d8003970 0x7fd1d8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d80047c0 0x7fd1d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d8005180 0x7fd1d800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80d1ec0 con 0x7fd1d80047c0 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d800c020 con 0x7fd1d8003970 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.729+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1d800c1e0 con 0x7fd1d8005180 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5400640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d80047c0 0x7fd1d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5e00640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d8003970 0x7fd1d8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f6800640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d8005180 0x7fd1d800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5400640 1 -- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d8005180 msgr2=0x7fd1d800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5400640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d8005180 0x7fd1d800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5400640 1 -- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d8003970 msgr2=0x7fd1d8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5400640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1d8003970 0x7fd1d8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f5400640 1 -- 172.21.15.53:0/1377216466 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d8003520 con 0x7fd1d80047c0 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.730+0000 7fd1f6800640 1 -- 172.21.15.53:0/1377216466 reap_dead start 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.731+0000 7fd1f5400640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d80047c0 0x7fd1d8004c40 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0001750 tx=0x7fd1d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecbf4bce533b2e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.731+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00052d0 con 0x7fd1d80047c0 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e002e050 con 0x7fd1d80047c0 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f4a00640 1 -- 172.21.15.53:0/1377216466 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0003830 con 0x7fd1d80047c0 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 -- 172.21.15.53:0/1377216466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d80047c0 msgr2=0x7fd1d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d80047c0 0x7fd1d8004c40 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fd1e0001750 tx=0x7fd1d80057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 -- 172.21.15.53:0/1377216466 shutdown_connections 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 --2- 172.21.15.53:0/1377216466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1d80047c0 0x7fd1d8004c40 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 -- 172.21.15.53:0/1377216466 >> 172.21.15.53:0/1377216466 conn(0x7fd1f80d3000 msgr2=0x7fd1f80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:40.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 -- 172.21.15.53:0/1377216466 shutdown_connections 2024-01-16T15:07:40.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.732+0000 7fd1f7200640 1 -- 172.21.15.53:0/1377216466 wait complete. 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.734+0000 7fd1f7200640 1 Processor -- start 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.734+0000 7fd1f7200640 1 -- start start 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.735+0000 7fd1f7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d80047c0 0x7fd1f81f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.735+0000 7fd1f7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 0x7fd1f80b7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.735+0000 7fd1f7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1f80c7cd0 0x7fd1f80b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.735+0000 7fd1f7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80b6840 con 0x7fd1f80c7850 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.735+0000 7fd1f7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80b5bc0 con 0x7fd1f80c7cd0 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.735+0000 7fd1f7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1f80b5ea0 con 0x7fd1d80047c0 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 0x7fd1f80b7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 0x7fd1f80b7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48210/0 (socket says 172.21.15.53:48210) 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 -- 172.21.15.53:0/2127303949 learned_addr learned my addr 172.21.15.53:0/2127303949 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f6800640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1f80c7cd0 0x7fd1f80b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5e00640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d80047c0 0x7fd1f81f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 -- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d80047c0 msgr2=0x7fd1f81f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d80047c0 0x7fd1f81f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 -- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1f80c7cd0 msgr2=0x7fd1f80b8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1f80c7cd0 0x7fd1f80b8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f5400640 1 -- 172.21.15.53:0/2127303949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1f80b6180 con 0x7fd1f80c7850 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.736+0000 7fd1f6800640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1f80c7cd0 0x7fd1f80b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.737+0000 7fd1f5400640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 0x7fd1f80b7e20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd1e002bc00 tx=0x7fd1e0001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.737+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e00076c0 con 0x7fd1f80c7850 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.737+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1e0007a30 con 0x7fd1f80c7850 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.737+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1e0006be0 con 0x7fd1f80c7850 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.737+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1f80b6460 con 0x7fd1f80c7850 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.737+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1f80caea0 con 0x7fd1f80c7850 2024-01-16T15:07:40.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.738+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1f8067450 con 0x7fd1f80c7850 2024-01-16T15:07:40.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.739+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fd1e0006d40 con 0x7fd1f80c7850 2024-01-16T15:07:40.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.740+0000 7fd1ee000640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd1d006bc50 0x7fd1d006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:40.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.741+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd1e00aa340 con 0x7fd1f80c7850 2024-01-16T15:07:40.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.741+0000 7fd1f5e00640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd1d006bc50 0x7fd1d006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:40.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.742+0000 7fd1f5e00640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd1d006bc50 0x7fd1d006e110 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8001c90 tx=0x7fd1e8000ff0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:40.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.744+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd1e0076a00 con 0x7fd1f80c7850 2024-01-16T15:07:40.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:40.967+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1f80b6010 con 0x7fd1d006bc50 2024-01-16T15:07:40.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.012+0000 7fd1ee000640 1 -- 172.21.15.53:0/2127303949 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd1f80b6010 con 0x7fd1d006bc50 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd1d006bc50 msgr2=0x7fd1d006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd1d006bc50 0x7fd1d006e110 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fd1e8001c90 tx=0x7fd1e8000ff0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 msgr2=0x7fd1f80b7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 0x7fd1f80b7e20 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fd1e002bc00 tx=0x7fd1e0001120 comp rx=0 tx=0).stop 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 shutdown_connections 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd1d006bc50 0x7fd1d006e110 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd1f80c7cd0 0x7fd1f80b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd1f80c7850 0x7fd1f80b7e20 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 --2- 172.21.15.53:0/2127303949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd1d80047c0 0x7fd1f81f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 >> 172.21.15.53:0/2127303949 conn(0x7fd1f80d3000 msgr2=0x7fd1f80b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 shutdown_connections 2024-01-16T15:07:40.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.015+0000 7fd1f7200640 1 -- 172.21.15.53:0/2127303949 wait complete. 2024-01-16T15:07:41.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30232 2024-01-16T15:07:41.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.260+0000 7febebe00640 1 Processor -- start 2024-01-16T15:07:41.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.260+0000 7febebe00640 1 -- start start 2024-01-16T15:07:41.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febebe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febec0d0100 0x7febec0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febebe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7febec0d0f40 0x7febec0d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febebe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febec0d0a10 con 0x7febec160920 2024-01-16T15:07:41.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febebe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec0cf080 con 0x7febec0d0100 2024-01-16T15:07:41.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febebe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febec0c6be0 con 0x7febec0d0f40 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febe3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febec0d0100 0x7febec0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febeaa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7febec160920 0x7febec0cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58952/0 (socket says 172.21.15.53:58952) 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febeaa00640 1 -- 172.21.15.53:0/4191125467 learned_addr learned my addr 172.21.15.53:0/4191125467 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febe3e00640 1 -- 172.21.15.53:0/4191125467 >> v2:172.21.15.146:3300/0 conn(0x7febec0d0f40 msgr2=0x7febec0d1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febe3e00640 1 --2- 172.21.15.53:0/4191125467 >> v2:172.21.15.146:3300/0 conn(0x7febec0d0f40 0x7febec0d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febe3e00640 1 -- 172.21.15.53:0/4191125467 >> v1:172.21.15.53:6789/0 conn(0x7febec160920 legacy=0x7febec0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.262+0000 7febe3e00640 1 -- 172.21.15.53:0/4191125467 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febec0ce7e0 con 0x7febec0d0100 2024-01-16T15:07:41.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.264+0000 7febe3e00640 1 --2- 172.21.15.53:0/4191125467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec0b3f30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7febec1620b0 tx=0x7febd402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8950ce890b0e26ff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:41.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.264+0000 7febea000640 1 -- 172.21.15.53:0/4191125467 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4031830 con 0x7febec0d0100 2024-01-16T15:07:41.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.264+0000 7febea000640 1 -- 172.21.15.53:0/4191125467 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd4031990 con 0x7febec0d0100 2024-01-16T15:07:41.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.264+0000 7febea000640 1 -- 172.21.15.53:0/4191125467 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd4031d20 con 0x7febec0d0100 2024-01-16T15:07:41.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.264+0000 7febebe00640 1 -- 172.21.15.53:0/4191125467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 msgr2=0x7febec0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.264+0000 7febebe00640 1 --2- 172.21.15.53:0/4191125467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec0b3f30 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7febec1620b0 tx=0x7febd402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.265+0000 7febebe00640 1 -- 172.21.15.53:0/4191125467 shutdown_connections 2024-01-16T15:07:41.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.265+0000 7febebe00640 1 --2- 172.21.15.53:0/4191125467 >> v2:172.21.15.146:3300/0 conn(0x7febec0d0f40 0x7febec0d1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.265+0000 7febebe00640 1 --2- 172.21.15.53:0/4191125467 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec0b3f30 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.265+0000 7febebe00640 1 -- 172.21.15.53:0/4191125467 >> 172.21.15.53:0/4191125467 conn(0x7febec0d3090 msgr2=0x7febec0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:41.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.265+0000 7febebe00640 1 -- 172.21.15.53:0/4191125467 shutdown_connections 2024-01-16T15:07:41.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.265+0000 7febebe00640 1 -- 172.21.15.53:0/4191125467 wait complete. 2024-01-16T15:07:41.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.266+0000 7febebe00640 1 Processor -- start 2024-01-16T15:07:41.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.267+0000 7febebe00640 1 -- start start 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.267+0000 7febebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec15a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.267+0000 7febebe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febec0d0f40 0x7febec0b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.267+0000 7febebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febec160920 0x7febec0b4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.267+0000 7febebe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec15bde0 con 0x7febec0d0100 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.267+0000 7febebe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febec0b63c0 con 0x7febec0d0f40 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febebe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febec15ab70 con 0x7febec160920 2024-01-16T15:07:41.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec15a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec15a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48230/0 (socket says 172.21.15.53:48230) 2024-01-16T15:07:41.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 -- 172.21.15.53:0/1112964047 learned_addr learned my addr 172.21.15.53:0/1112964047 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febe3e00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febec0d0f40 0x7febec0b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 -- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febec160920 msgr2=0x7febec0b4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febec160920 0x7febec0b4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 -- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febec0d0f40 msgr2=0x7febec0b6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febec0d0f40 0x7febec0b6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.268+0000 7febeaa00640 1 -- 172.21.15.53:0/1112964047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4009400 con 0x7febec0d0100 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.269+0000 7febeaa00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec15a630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c480 tx=0x7febdc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.269+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc005ce0 con 0x7febec0d0100 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.269+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc005e40 con 0x7febec0d0100 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.269+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc0072f0 con 0x7febec0d0100 2024-01-16T15:07:41.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.269+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febec0b60e0 con 0x7febec0d0100 2024-01-16T15:07:41.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.269+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febec0ca570 con 0x7febec0d0100 2024-01-16T15:07:41.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.270+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febec067450 con 0x7febec0d0100 2024-01-16T15:07:41.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.273+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7febdc007510 con 0x7febec0d0100 2024-01-16T15:07:41.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.273+0000 7febe2a00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febc406bc50 0x7febc406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.273+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7febdc087ba0 con 0x7febec0d0100 2024-01-16T15:07:41.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.274+0000 7febe3e00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febc406bc50 0x7febc406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.275+0000 7febe3e00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febc406bc50 0x7febc406e110 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7febd4036000 tx=0x7febd4008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:41.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.278+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febdc054310 con 0x7febec0d0100 2024-01-16T15:07:41.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.509+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30232", "target": ["mon-mgr", ""]}) v1 -- 0x7febec160d00 con 0x7febc406bc50 2024-01-16T15:07:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.554+0000 7febe2a00640 1 -- 172.21.15.53:0/1112964047 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7febd4045400 con 0x7febc406bc50 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febc406bc50 msgr2=0x7febc406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febc406bc50 0x7febc406e110 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7febd4036000 tx=0x7febd4008030 comp rx=0 tx=0).stop 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 msgr2=0x7febec15a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec15a630 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c480 tx=0x7febdc00c940 comp rx=0 tx=0).stop 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 shutdown_connections 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febc406bc50 0x7febc406e110 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febec160920 0x7febec0b4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febec0d0f40 0x7febec0b6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 --2- 172.21.15.53:0/1112964047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febec0d0100 0x7febec15a630 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 >> 172.21.15.53:0/1112964047 conn(0x7febec0d3090 msgr2=0x7febec0d3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 shutdown_connections 2024-01-16T15:07:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.557+0000 7febebe00640 1 -- 172.21.15.53:0/1112964047 wait complete. 2024-01-16T15:07:41.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30235 2024-01-16T15:07:41.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.792+0000 7fa026c00640 1 Processor -- start 2024-01-16T15:07:41.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.793+0000 7fa026c00640 1 -- start start 2024-01-16T15:07:41.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.794+0000 7fa026c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa02013f8d0 0x7fa0200ce7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.794+0000 7fa026c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0200cfa50 con 0x7fa02006aa30 2024-01-16T15:07:41.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.794+0000 7fa026c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0200cee40 con 0x7fa02006a650 2024-01-16T15:07:41.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.794+0000 7fa026c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0201417e0 con 0x7fa02013f8d0 2024-01-16T15:07:41.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.795+0000 7fa024c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa02006aa30 0x7fa0201400c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:58954/0 (socket says 172.21.15.53:58954) 2024-01-16T15:07:41.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.795+0000 7fa024c00640 1 -- 172.21.15.53:0/742574481 learned_addr learned my addr 172.21.15.53:0/742574481 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:41.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.795+0000 7fa01f400640 1 --2- 172.21.15.53:0/742574481 >> v2:172.21.15.146:3300/0 conn(0x7fa02013f8d0 0x7fa0200ce7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.795+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360659817 0 0) 0x7fa0200cfa50 con 0x7fa02006aa30 2024-01-16T15:07:41.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.795+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa000003520 con 0x7fa02006aa30 2024-01-16T15:07:41.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 17782811 0 0) 0x7fa000003520 con 0x7fa02006aa30 2024-01-16T15:07:41.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 >> v2:172.21.15.146:3300/0 conn(0x7fa02013f8d0 msgr2=0x7fa0200ce7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:41.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 --2- 172.21.15.53:0/742574481 >> v2:172.21.15.146:3300/0 conn(0x7fa02013f8d0 0x7fa0200ce7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 >> v1:172.21.15.146:6789/0 conn(0x7fa02006a650 legacy=0x7fa020071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0200cf120 con 0x7fa02006aa30 2024-01-16T15:07:41.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fa018001670 con 0x7fa02006aa30 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 >> v1:172.21.15.53:6789/0 conn(0x7fa02006aa30 legacy=0x7fa0201400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000039c0 0x7fa000003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0000047c0 0x7fa000004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0000051d0 0x7fa00000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0201417e0 con 0x7fa0000051d0 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c120 con 0x7fa0000039c0 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00000c2e0 con 0x7fa0000047c0 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa024c00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0000051d0 0x7fa00000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.796+0000 7fa01fe00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000039c0 0x7fa000003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.797+0000 7fa01f400640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0000047c0 0x7fa000004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.797+0000 7fa024c00640 1 -- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0000047c0 msgr2=0x7fa000004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.797+0000 7fa024c00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0000047c0 0x7fa000004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.797+0000 7fa024c00640 1 -- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000039c0 msgr2=0x7fa000003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:41.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.797+0000 7fa024c00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000039c0 0x7fa000003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.797+0000 7fa024c00640 1 -- 172.21.15.53:0/742574481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa000003520 con 0x7fa0000051d0 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.798+0000 7fa024c00640 1 -- 172.21.15.53:0/742574481 reap_dead start 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.798+0000 7fa024c00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0000051d0 0x7fa00000b980 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa018001eb0 tx=0x7fa0000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9da22df78e6bedd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.798+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa018005ad0 con 0x7fa0000051d0 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.798+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa018005c30 con 0x7fa0000051d0 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.798+0000 7fa01ea00640 1 -- 172.21.15.53:0/742574481 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01802c7c0 con 0x7fa0000051d0 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 -- 172.21.15.53:0/742574481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0000051d0 msgr2=0x7fa00000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0000051d0 0x7fa00000b980 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7fa018001eb0 tx=0x7fa0000057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 -- 172.21.15.53:0/742574481 shutdown_connections 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 --2- 172.21.15.53:0/742574481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0000051d0 0x7fa00000b980 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 -- 172.21.15.53:0/742574481 >> 172.21.15.53:0/742574481 conn(0x7fa0200b1bf0 msgr2=0x7fa0200b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:41.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 -- 172.21.15.53:0/742574481 shutdown_connections 2024-01-16T15:07:41.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.799+0000 7fa026c00640 1 -- 172.21.15.53:0/742574481 wait complete. 2024-01-16T15:07:41.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.801+0000 7fa026c00640 1 Processor -- start 2024-01-16T15:07:41.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.801+0000 7fa026c00640 1 -- start start 2024-01-16T15:07:41.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa026c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000051d0 0x7fa0200ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa026c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 0x7fa02015b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa026c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa02015c8d0 0x7fa02015cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa026c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02015c4e0 con 0x7fa02015a700 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa026c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0200b3200 con 0x7fa0000051d0 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa026c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02015d290 con 0x7fa02015c8d0 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa01f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 0x7fa02015b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.802+0000 7fa01f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 0x7fa02015b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48246/0 (socket says 172.21.15.53:48246) 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 -- 172.21.15.53:0/1365229986 learned_addr learned my addr 172.21.15.53:0/1365229986 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa024c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa02015c8d0 0x7fa02015cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01fe00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000051d0 0x7fa0200ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 -- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa02015c8d0 msgr2=0x7fa02015cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa02015c8d0 0x7fa02015cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 -- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000051d0 msgr2=0x7fa0200ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000051d0 0x7fa0200ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:41.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 -- 172.21.15.53:0/1365229986 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02015b9a0 con 0x7fa02015a700 2024-01-16T15:07:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.803+0000 7fa01f400640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 0x7fa02015b460 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa008010890 tx=0x7fa008010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.804+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa008009610 con 0x7fa02015a700 2024-01-16T15:07:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.804+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00800eb60 con 0x7fa02015a700 2024-01-16T15:07:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.804+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00801c400 con 0x7fa02015a700 2024-01-16T15:07:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.804+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa02015bc40 con 0x7fa02015a700 2024-01-16T15:07:41.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.804+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa02015c200 con 0x7fa02015a700 2024-01-16T15:07:41.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.805+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0200675d0 con 0x7fa02015a700 2024-01-16T15:07:41.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.806+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fa00801c560 con 0x7fa02015a700 2024-01-16T15:07:41.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.807+0000 7fa017e00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ff806bc80 0x7f9ff806e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:41.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.808+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa00808b640 con 0x7fa02015a700 2024-01-16T15:07:41.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.808+0000 7fa01fe00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ff806bc80 0x7f9ff806e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:41.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.810+0000 7fa01fe00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ff806bc80 0x7f9ff806e140 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa010007fd0 tx=0x7fa010009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:41.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:41.812+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa008055380 con 0x7fa02015a700 2024-01-16T15:07:42.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.026+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30235", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0200cf6e0 con 0x7f9ff806bc80 2024-01-16T15:07:42.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.077+0000 7fa017e00640 1 -- 172.21.15.53:0/1365229986 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa010005740 con 0x7f9ff806bc80 2024-01-16T15:07:42.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.079+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ff806bc80 msgr2=0x7f9ff806e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.079+0000 7fa026c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ff806bc80 0x7f9ff806e140 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa010007fd0 tx=0x7fa010009040 comp rx=0 tx=0).stop 2024-01-16T15:07:42.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.079+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 msgr2=0x7fa02015b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.079+0000 7fa026c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 0x7fa02015b460 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fa008010890 tx=0x7fa008010d50 comp rx=0 tx=0).stop 2024-01-16T15:07:42.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 shutdown_connections 2024-01-16T15:07:42.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9ff806bc80 0x7f9ff806e140 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa02015c8d0 0x7fa02015cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa02015a700 0x7fa02015b460 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 --2- 172.21.15.53:0/1365229986 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0000051d0 0x7fa0200ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 >> 172.21.15.53:0/1365229986 conn(0x7fa0200b1bf0 msgr2=0x7fa0200b1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:42.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 shutdown_connections 2024-01-16T15:07:42.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.080+0000 7fa026c00640 1 -- 172.21.15.53:0/1365229986 wait complete. 2024-01-16T15:07:42.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30231 2024-01-16T15:07:42.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.319+0000 7f8d4d400640 1 Processor -- start 2024-01-16T15:07:42.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.319+0000 7f8d4d400640 1 -- start start 2024-01-16T15:07:42.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d4d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8d48160f80 0x7f8d48161360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d4d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d480ce8f0 0x7f8d480ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d4d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d481618a0 con 0x7f8d480cfbe0 2024-01-16T15:07:42.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d4d400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d480d0de0 con 0x7f8d48160f80 2024-01-16T15:07:42.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d4d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d480d10c0 con 0x7f8d480ce8f0 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d46800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8d48160f80 0x7f8d48161360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d46800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8d48160f80 0x7f8d48161360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46888/0 (socket says 172.21.15.53:46888) 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.320+0000 7f8d46800640 1 -- 172.21.15.53:0/2535006562 learned_addr learned my addr 172.21.15.53:0/2535006562 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.321+0000 7f8d45e00640 1 --2- 172.21.15.53:0/2535006562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d480ce8f0 0x7f8d480ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.321+0000 7f8d46800640 1 -- 172.21.15.53:0/2535006562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d480ce8f0 msgr2=0x7f8d480ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.321+0000 7f8d46800640 1 --2- 172.21.15.53:0/2535006562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d480ce8f0 0x7f8d480ced50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.321+0000 7f8d46800640 1 -- 172.21.15.53:0/2535006562 >> v1:172.21.15.146:6789/0 conn(0x7f8d480cfbe0 legacy=0x7f8d480d0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:42.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.321+0000 7f8d46800640 1 -- 172.21.15.53:0/2535006562 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d480d13a0 con 0x7f8d48160f80 2024-01-16T15:07:42.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.322+0000 7f8d46800640 1 --2- 172.21.15.53:0/2535006562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d48160f80 0x7f8d48161360 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8d38009870 tx=0x7f8d3802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e1e835a520aaecd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.323+0000 7f8d45400640 1 -- 172.21.15.53:0/2535006562 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d38031640 con 0x7f8d48160f80 2024-01-16T15:07:42.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.323+0000 7f8d45400640 1 -- 172.21.15.53:0/2535006562 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d380317a0 con 0x7f8d48160f80 2024-01-16T15:07:42.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.323+0000 7f8d45400640 1 -- 172.21.15.53:0/2535006562 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d38031b30 con 0x7f8d48160f80 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 -- 172.21.15.53:0/2535006562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d48160f80 msgr2=0x7f8d48161360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 --2- 172.21.15.53:0/2535006562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d48160f80 0x7f8d48161360 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f8d38009870 tx=0x7f8d3802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 -- 172.21.15.53:0/2535006562 shutdown_connections 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 --2- 172.21.15.53:0/2535006562 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d480ce8f0 0x7f8d480ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 --2- 172.21.15.53:0/2535006562 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d48160f80 0x7f8d48161360 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 -- 172.21.15.53:0/2535006562 >> 172.21.15.53:0/2535006562 conn(0x7f8d4809c550 msgr2=0x7f8d480d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:42.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 -- 172.21.15.53:0/2535006562 shutdown_connections 2024-01-16T15:07:42.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.324+0000 7f8d4d400640 1 -- 172.21.15.53:0/2535006562 wait complete. 2024-01-16T15:07:42.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.325+0000 7f8d4d400640 1 Processor -- start 2024-01-16T15:07:42.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.326+0000 7f8d4d400640 1 -- start start 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.326+0000 7f8d4d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 0x7f8d481f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d4d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d480cfbe0 0x7f8d480c7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d4d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d48160f80 0x7f8d48154de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d4d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d480c8100 con 0x7f8d480ce8f0 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d4d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d48155320 con 0x7f8d480cfbe0 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d4d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d481f73a0 con 0x7f8d48160f80 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 0x7f8d481f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 0x7f8d481f6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46892/0 (socket says 172.21.15.53:46892) 2024-01-16T15:07:42.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 -- 172.21.15.53:0/3927054567 learned_addr learned my addr 172.21.15.53:0/3927054567 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d45e00640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d480cfbe0 0x7f8d480c7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 -- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d48160f80 msgr2=0x7f8d48154de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d48160f80 0x7f8d48154de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 -- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d480cfbe0 msgr2=0x7f8d480c7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d480cfbe0 0x7f8d480c7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d46800640 1 -- 172.21.15.53:0/3927054567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d380094d0 con 0x7f8d480ce8f0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.327+0000 7f8d45e00640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d480cfbe0 0x7f8d480c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.328+0000 7f8d46800640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 0x7f8d481f6490 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8d38009870 tx=0x7f8d380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.328+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d3803ac60 con 0x7f8d480ce8f0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.328+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d3803adc0 con 0x7f8d480ce8f0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.328+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d38044630 con 0x7f8d480ce8f0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.328+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d481f6b00 con 0x7f8d480ce8f0 2024-01-16T15:07:42.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.328+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d481f70c0 con 0x7f8d480ce8f0 2024-01-16T15:07:42.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.329+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d480675d0 con 0x7f8d480ce8f0 2024-01-16T15:07:42.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.330+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f8d3802f050 con 0x7f8d480ce8f0 2024-01-16T15:07:42.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.331+0000 7f8d3ea00640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d2006bc50 0x7f8d2006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.331+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d380aa650 con 0x7f8d480ce8f0 2024-01-16T15:07:42.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.332+0000 7f8d45e00640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d2006bc50 0x7f8d2006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.333+0000 7f8d45e00640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d2006bc50 0x7f8d2006e110 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8d30006fd0 tx=0x7f8d30008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.336+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d38076d10 con 0x7f8d480ce8f0 2024-01-16T15:07:42.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.552+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30231", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d480d1c40 con 0x7f8d2006bc50 2024-01-16T15:07:42.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.596+0000 7f8d3ea00640 1 -- 172.21.15.53:0/3927054567 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8d30009530 con 0x7f8d2006bc50 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d2006bc50 msgr2=0x7f8d2006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d2006bc50 0x7f8d2006e110 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f8d30006fd0 tx=0x7f8d30008040 comp rx=0 tx=0).stop 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 msgr2=0x7f8d481f6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 0x7f8d481f6490 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f8d38009870 tx=0x7f8d380046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 shutdown_connections 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d2006bc50 0x7f8d2006e110 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d48160f80 0x7f8d48154de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d480cfbe0 0x7f8d480c7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 --2- 172.21.15.53:0/3927054567 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d480ce8f0 0x7f8d481f6490 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 >> 172.21.15.53:0/3927054567 conn(0x7f8d4809c550 msgr2=0x7f8d4806c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:42.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.598+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 shutdown_connections 2024-01-16T15:07:42.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.599+0000 7f8d4d400640 1 -- 172.21.15.53:0/3927054567 wait complete. 2024-01-16T15:07:42.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30237 2024-01-16T15:07:42.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.838+0000 7f1dfb200640 1 Processor -- start 2024-01-16T15:07:42.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.838+0000 7f1dfb200640 1 -- start start 2024-01-16T15:07:42.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1dfb200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1df415d450 0x7f1df40cf760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1dfb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1df40b2bc0 0x7f1df40b3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1dfb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1df40ceff0 con 0x7f1df40b2020 2024-01-16T15:07:42.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1dfb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40d0800 con 0x7f1df40b2bc0 2024-01-16T15:07:42.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1dfb200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1df40cea30 con 0x7f1df415d450 2024-01-16T15:07:42.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1dfa800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1df40b2bc0 0x7f1df40b3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1df9400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1df40b2020 0x7f1df40cfca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51716/0 (socket says 172.21.15.53:51716) 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.840+0000 7f1df9400640 1 -- 172.21.15.53:0/986957300 learned_addr learned my addr 172.21.15.53:0/986957300 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.841+0000 7f1df9e00640 1 --2- 172.21.15.53:0/986957300 >> v2:172.21.15.146:3300/0 conn(0x7f1df415d450 0x7f1df40cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.841+0000 7f1dfa800640 1 -- 172.21.15.53:0/986957300 >> v2:172.21.15.146:3300/0 conn(0x7f1df415d450 msgr2=0x7f1df40cf760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.841+0000 7f1dfa800640 1 --2- 172.21.15.53:0/986957300 >> v2:172.21.15.146:3300/0 conn(0x7f1df415d450 0x7f1df40cf760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.841+0000 7f1dfa800640 1 -- 172.21.15.53:0/986957300 >> v1:172.21.15.53:6789/0 conn(0x7f1df40b2020 legacy=0x7f1df40cfca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.841+0000 7f1dfa800640 1 -- 172.21.15.53:0/986957300 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df40ced10 con 0x7f1df40b2bc0 2024-01-16T15:07:42.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.841+0000 7f1df8a00640 1 -- 172.21.15.53:0/986957300 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720179268 0 0) 0x7f1df40ceff0 con 0x7f1df40b2020 2024-01-16T15:07:42.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.842+0000 7f1dfa800640 1 --2- 172.21.15.53:0/986957300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2bc0 0x7f1df40b3050 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1df406abd0 tx=0x7f1dec02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fec1d374d33fd646 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.842+0000 7f1df8a00640 1 -- 172.21.15.53:0/986957300 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec02cd00 con 0x7f1df40b2bc0 2024-01-16T15:07:42.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.842+0000 7f1df8a00640 1 -- 172.21.15.53:0/986957300 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec02ce60 con 0x7f1df40b2bc0 2024-01-16T15:07:42.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.842+0000 7f1df8a00640 1 -- 172.21.15.53:0/986957300 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec034660 con 0x7f1df40b2bc0 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 -- 172.21.15.53:0/986957300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2bc0 msgr2=0x7f1df40b3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 --2- 172.21.15.53:0/986957300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2bc0 0x7f1df40b3050 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7f1df406abd0 tx=0x7f1dec02bf50 comp rx=0 tx=0).stop 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 -- 172.21.15.53:0/986957300 shutdown_connections 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 --2- 172.21.15.53:0/986957300 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2bc0 0x7f1df40b3050 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 --2- 172.21.15.53:0/986957300 >> v2:172.21.15.146:3300/0 conn(0x7f1df415d450 0x7f1df40cf760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 -- 172.21.15.53:0/986957300 >> 172.21.15.53:0/986957300 conn(0x7f1df40d2f80 msgr2=0x7f1df40d3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 -- 172.21.15.53:0/986957300 shutdown_connections 2024-01-16T15:07:42.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.843+0000 7f1dfb200640 1 -- 172.21.15.53:0/986957300 wait complete. 2024-01-16T15:07:42.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.845+0000 7f1dfb200640 1 Processor -- start 2024-01-16T15:07:42.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.845+0000 7f1dfb200640 1 -- start start 2024-01-16T15:07:42.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1dfb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 0x7f1df4159580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1dfb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df40b2bc0 0x7f1df40ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1dfb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df415d450 0x7f1df40ca590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1dfb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df40c83b0 con 0x7f1df40b2020 2024-01-16T15:07:42.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1dfb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df40cabc0 con 0x7f1df415d450 2024-01-16T15:07:42.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1dfb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1df40caea0 con 0x7f1df40b2bc0 2024-01-16T15:07:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.846+0000 7f1df9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 0x7f1df4159580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 0x7f1df4159580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46906/0 (socket says 172.21.15.53:46906) 2024-01-16T15:07:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 -- 172.21.15.53:0/2417225067 learned_addr learned my addr 172.21.15.53:0/2417225067 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1dfa800640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df415d450 0x7f1df40ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9400640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df40b2bc0 0x7f1df40ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 -- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df40b2bc0 msgr2=0x7f1df40ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df40b2bc0 0x7f1df40ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 -- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df415d450 msgr2=0x7f1df40ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df415d450 0x7f1df40ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 -- 172.21.15.53:0/2417225067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec009a70 con 0x7f1df40b2020 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.847+0000 7f1df9e00640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 0x7f1df4159580 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1de400d800 tx=0x7f1de400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.848+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400b6c0 con 0x7f1df40b2020 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.848+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1de400b820 con 0x7f1df40b2020 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.848+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de400bbb0 con 0x7f1df40b2020 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.848+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df40cb180 con 0x7f1df40b2020 2024-01-16T15:07:42.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.848+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df40cb740 con 0x7f1df40b2020 2024-01-16T15:07:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.849+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df40cb900 con 0x7f1df40b2020 2024-01-16T15:07:42.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.850+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f1de400bd10 con 0x7f1df40b2020 2024-01-16T15:07:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.851+0000 7f1df2000640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dcc06bc50 0x7f1dcc06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:42.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.851+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1de4088cc0 con 0x7f1df40b2020 2024-01-16T15:07:42.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.852+0000 7f1df9400640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dcc06bc50 0x7f1dcc06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:42.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.853+0000 7f1df9400640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dcc06bc50 0x7f1dcc06e110 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1df415bb30 tx=0x7f1ddc007c00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:42.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:42.856+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1de4055340 con 0x7f1df40b2020 2024-01-16T15:07:43.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.078+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30237", "target": ["mon-mgr", ""]}) v1 -- 0x7f1df415e5e0 con 0x7f1dcc06bc50 2024-01-16T15:07:43.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.122+0000 7f1df2000640 1 -- 172.21.15.53:0/2417225067 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1df415e5e0 con 0x7f1dcc06bc50 2024-01-16T15:07:43.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dcc06bc50 msgr2=0x7f1dcc06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dcc06bc50 0x7f1dcc06e110 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1df415bb30 tx=0x7f1ddc007c00 comp rx=0 tx=0).stop 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 msgr2=0x7f1df4159580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 0x7f1df4159580 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f1de400d800 tx=0x7f1de400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 shutdown_connections 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dcc06bc50 0x7f1dcc06e110 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1df415d450 0x7f1df40ca590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1df40b2bc0 0x7f1df40ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 --2- 172.21.15.53:0/2417225067 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1df40b2020 0x7f1df4159580 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 >> 172.21.15.53:0/2417225067 conn(0x7f1df40d2f80 msgr2=0x7f1df406c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 shutdown_connections 2024-01-16T15:07:43.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.124+0000 7f1dfb200640 1 -- 172.21.15.53:0/2417225067 wait complete. 2024-01-16T15:07:43.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30236 2024-01-16T15:07:43.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.364+0000 7fca57200640 1 Processor -- start 2024-01-16T15:07:43.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.364+0000 7fca57200640 1 -- start start 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca57200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca5815afb0 0x7fca5813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca57200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fca5813fae0 0x7fca580b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca57200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5813ff30 con 0x7fca5813cfa0 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca57200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca580cdc40 con 0x7fca5815afb0 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca57200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca580cdec0 con 0x7fca5813fae0 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca55e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fca5813cfa0 0x7fca580cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51722/0 (socket says 172.21.15.53:51722) 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.366+0000 7fca55e00640 1 -- 172.21.15.53:0/2855085693 learned_addr learned my addr 172.21.15.53:0/2855085693 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:43.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca55400640 1 --2- 172.21.15.53:0/2855085693 >> v2:172.21.15.53:3300/0 conn(0x7fca5815afb0 0x7fca5813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:43.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca56800640 1 --2- 172.21.15.53:0/2855085693 >> v2:172.21.15.146:3300/0 conn(0x7fca5813fae0 0x7fca580b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:43.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca54a00640 1 -- 172.21.15.53:0/2855085693 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442630738 0 0) 0x7fca40001110 con 0x7fca5813cfa0 2024-01-16T15:07:43.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca54a00640 1 -- 172.21.15.53:0/2855085693 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca38003520 con 0x7fca5813cfa0 2024-01-16T15:07:43.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca55400640 1 -- 172.21.15.53:0/2855085693 >> v2:172.21.15.146:3300/0 conn(0x7fca5813fae0 msgr2=0x7fca580b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:43.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca55400640 1 --2- 172.21.15.53:0/2855085693 >> v2:172.21.15.146:3300/0 conn(0x7fca5813fae0 0x7fca580b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca55400640 1 -- 172.21.15.53:0/2855085693 >> v1:172.21.15.53:6789/0 conn(0x7fca5813cfa0 legacy=0x7fca580cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.367+0000 7fca55400640 1 -- 172.21.15.53:0/2855085693 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca580ce1a0 con 0x7fca5815afb0 2024-01-16T15:07:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca55400640 1 --2- 172.21.15.53:0/2855085693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5815afb0 0x7fca5813f5a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fca58154a90 tx=0x7fca4802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d6d74eec47adc65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca54a00640 1 -- 172.21.15.53:0/2855085693 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca4802cbf0 con 0x7fca5815afb0 2024-01-16T15:07:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca54a00640 1 -- 172.21.15.53:0/2855085693 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca4802cd50 con 0x7fca5815afb0 2024-01-16T15:07:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca54a00640 1 -- 172.21.15.53:0/2855085693 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca48034660 con 0x7fca5815afb0 2024-01-16T15:07:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca57200640 1 -- 172.21.15.53:0/2855085693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5815afb0 msgr2=0x7fca5813f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca57200640 1 --2- 172.21.15.53:0/2855085693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5815afb0 0x7fca5813f5a0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fca58154a90 tx=0x7fca4802be40 comp rx=0 tx=0).stop 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca57200640 1 -- 172.21.15.53:0/2855085693 shutdown_connections 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca57200640 1 --2- 172.21.15.53:0/2855085693 >> v2:172.21.15.146:3300/0 conn(0x7fca5813fae0 0x7fca580b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca57200640 1 --2- 172.21.15.53:0/2855085693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5815afb0 0x7fca5813f5a0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.369+0000 7fca57200640 1 -- 172.21.15.53:0/2855085693 >> 172.21.15.53:0/2855085693 conn(0x7fca5806a800 msgr2=0x7fca5806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.370+0000 7fca57200640 1 -- 172.21.15.53:0/2855085693 shutdown_connections 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.370+0000 7fca57200640 1 -- 172.21.15.53:0/2855085693 wait complete. 2024-01-16T15:07:43.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.371+0000 7fca57200640 1 Processor -- start 2024-01-16T15:07:43.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.371+0000 7fca57200640 1 -- start start 2024-01-16T15:07:43.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.372+0000 7fca57200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 0x7fca581564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.372+0000 7fca57200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca5813fae0 0x7fca58138830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.372+0000 7fca57200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca5815afb0 0x7fca58156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.372+0000 7fca57200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca581597c0 con 0x7fca5813cfa0 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.372+0000 7fca57200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca58158bb0 con 0x7fca5813fae0 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.372+0000 7fca57200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca58158e90 con 0x7fca5815afb0 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 0x7fca581564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 0x7fca581564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46918/0 (socket says 172.21.15.53:46918) 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 -- 172.21.15.53:0/3622544547 learned_addr learned my addr 172.21.15.53:0/3622544547 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 -- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca5815afb0 msgr2=0x7fca58156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca5815afb0 0x7fca58156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 -- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca5813fae0 msgr2=0x7fca58138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca5813fae0 0x7fca58138830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 -- 172.21.15.53:0/3622544547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48009a70 con 0x7fca5813cfa0 2024-01-16T15:07:43.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.373+0000 7fca55e00640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 0x7fca581564f0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c700 tx=0x7fca4000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.374+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40005ce0 con 0x7fca5813cfa0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.374+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca40005e40 con 0x7fca5813cfa0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.374+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca58159170 con 0x7fca5813cfa0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.374+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca40007290 con 0x7fca5813cfa0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.374+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca58138100 con 0x7fca5813cfa0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.375+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca580b2500 con 0x7fca5813cfa0 2024-01-16T15:07:43.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.376+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fca400074b0 con 0x7fca5813cfa0 2024-01-16T15:07:43.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.377+0000 7fca4e000640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca3006bc50 0x7fca3006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:43.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.377+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fca40087b70 con 0x7fca5813cfa0 2024-01-16T15:07:43.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.377+0000 7fca55400640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca3006bc50 0x7fca3006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:43.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.378+0000 7fca55400640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca3006bc50 0x7fca3006e110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fca48036000 tx=0x7fca4802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:43.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.381+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca40057cc0 con 0x7fca5813cfa0 2024-01-16T15:07:43.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.692+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30236", "target": ["mon-mgr", ""]}) v1 -- 0x7fca5813c7a0 con 0x7fca3006bc50 2024-01-16T15:07:43.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.725+0000 7fca4e000640 1 -- 172.21.15.53:0/3622544547 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fca5813c7a0 con 0x7fca3006bc50 2024-01-16T15:07:43.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca3006bc50 msgr2=0x7fca3006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca3006bc50 0x7fca3006e110 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fca48036000 tx=0x7fca4802e000 comp rx=0 tx=0).stop 2024-01-16T15:07:43.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 msgr2=0x7fca581564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:43.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 0x7fca581564f0 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c700 tx=0x7fca4000cb00 comp rx=0 tx=0).stop 2024-01-16T15:07:43.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 shutdown_connections 2024-01-16T15:07:43.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca3006bc50 0x7fca3006e110 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca5815afb0 0x7fca58156cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca5813fae0 0x7fca58138830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 --2- 172.21.15.53:0/3622544547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca5813cfa0 0x7fca581564f0 secure :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fca4000c700 tx=0x7fca4000cb00 comp rx=0 tx=0).stop 2024-01-16T15:07:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.730+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 >> 172.21.15.53:0/3622544547 conn(0x7fca5806a800 msgr2=0x7fca5815b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.731+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 shutdown_connections 2024-01-16T15:07:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:43.731+0000 7fca57200640 1 -- 172.21.15.53:0/3622544547 wait complete. 2024-01-16T15:07:43.734 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone30234 2024-01-16T15:07:44.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.033+0000 7fb9cb200640 1 Processor -- start 2024-01-16T15:07:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.033+0000 7fb9cb200640 1 -- start start 2024-01-16T15:07:44.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.036+0000 7fb9cb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb9c406e380 0x7fb9c40ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.036+0000 7fb9cb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c415d840 con 0x7fb9c40b30b0 2024-01-16T15:07:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.036+0000 7fb9cb200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c4072190 con 0x7fb9c415d460 2024-01-16T15:07:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.036+0000 7fb9cb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c4071490 con 0x7fb9c406e380 2024-01-16T15:07:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.036+0000 7fb9c3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb9c40b30b0 0x7fb9c406d280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51738/0 (socket says 172.21.15.53:51738) 2024-01-16T15:07:44.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.036+0000 7fb9c3400640 1 -- 172.21.15.53:0/3830442717 learned_addr learned my addr 172.21.15.53:0/3830442717 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:44.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.037+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 620796872 0 0) 0x7fb9c415d840 con 0x7fb9c40b30b0 2024-01-16T15:07:44.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.037+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4003520 con 0x7fb9c40b30b0 2024-01-16T15:07:44.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.037+0000 7fb9c9200640 1 --2- 172.21.15.53:0/3830442717 >> v2:172.21.15.53:3300/0 conn(0x7fb9c406e380 0x7fb9c40ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4086527971 0 0) 0x7fb9a4003520 con 0x7fb9c40b30b0 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 >> v2:172.21.15.53:3300/0 conn(0x7fb9c406e380 msgr2=0x7fb9c40ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 --2- 172.21.15.53:0/3830442717 >> v2:172.21.15.53:3300/0 conn(0x7fb9c406e380 0x7fb9c40ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 >> v1:172.21.15.146:6789/0 conn(0x7fb9c415d460 legacy=0x7fb9c40b29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c4072470 con 0x7fb9c40b30b0 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb9ac001670 con 0x7fb9c40b30b0 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 >> v1:172.21.15.53:6789/0 conn(0x7fb9c40b30b0 legacy=0x7fb9c406d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4003970 0x7fb9a4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4071490 con 0x7fb9a4005180 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a400c0b0 con 0x7fb9a4003970 2024-01-16T15:07:44.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.038+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a400c270 con 0x7fb9a40047c0 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.039+0000 7fb9c3e00640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4003970 0x7fb9a4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.039+0000 7fb9c9200640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.040+0000 7fb9c3400640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.040+0000 7fb9c9200640 1 -- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9a40047c0 msgr2=0x7fb9a4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.040+0000 7fb9c9200640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.040+0000 7fb9c9200640 1 -- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4003970 msgr2=0x7fb9a4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.040+0000 7fb9c9200640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4003970 0x7fb9a4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.040+0000 7fb9c9200640 1 -- 172.21.15.53:0/3830442717 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c4072470 con 0x7fb9a4005180 2024-01-16T15:07:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.041+0000 7fb9c3400640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.041+0000 7fb9c3e00640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4003970 0x7fb9a4003d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.041+0000 7fb9c9200640 1 -- 172.21.15.53:0/3830442717 reap_dead start 2024-01-16T15:07:44.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.042+0000 7fb9c9200640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00e3a0 tx=0x7fb9a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a23ede1f11dc5cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:44.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.042+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc034440 con 0x7fb9a4005180 2024-01-16T15:07:44.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.042+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc00f040 con 0x7fb9a4005180 2024-01-16T15:07:44.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.042+0000 7fb9c2a00640 1 -- 172.21.15.53:0/3830442717 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9bc0347c0 con 0x7fb9a4005180 2024-01-16T15:07:44.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.043+0000 7fb9cb200640 1 -- 172.21.15.53:0/3830442717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9a4005180 msgr2=0x7fb9a400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:44.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.043+0000 7fb9cb200640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fb9bc00e3a0 tx=0x7fb9a40057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.045+0000 7fb9cb200640 1 -- 172.21.15.53:0/3830442717 shutdown_connections 2024-01-16T15:07:44.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.045+0000 7fb9cb200640 1 --2- 172.21.15.53:0/3830442717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.045+0000 7fb9cb200640 1 -- 172.21.15.53:0/3830442717 >> 172.21.15.53:0/3830442717 conn(0x7fb9c40b0be0 msgr2=0x7fb9c40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:44.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.045+0000 7fb9cb200640 1 -- 172.21.15.53:0/3830442717 shutdown_connections 2024-01-16T15:07:44.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.045+0000 7fb9cb200640 1 -- 172.21.15.53:0/3830442717 wait complete. 2024-01-16T15:07:44.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.048+0000 7fb9cb200640 1 Processor -- start 2024-01-16T15:07:44.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.048+0000 7fb9cb200640 1 -- start start 2024-01-16T15:07:44.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.049+0000 7fb9cb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 0x7fb9c414f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.049+0000 7fb9cb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c414ff00 0x7fb9c415fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.051+0000 7fb9cb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c41603c0 0x7fb9c4160840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.051+0000 7fb9cb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c415f3b0 con 0x7fb9c41603c0 2024-01-16T15:07:44.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.051+0000 7fb9cb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c4150380 con 0x7fb9a4005180 2024-01-16T15:07:44.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.051+0000 7fb9cb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c415edf0 con 0x7fb9c414ff00 2024-01-16T15:07:44.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 0x7fb9c414f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 0x7fb9c414f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:58738/0 (socket says 172.21.15.53:58738) 2024-01-16T15:07:44.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 -- 172.21.15.53:0/2183443127 learned_addr learned my addr 172.21.15.53:0/2183443127 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c9200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c41603c0 0x7fb9c4160840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3400640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c414ff00 0x7fb9c415fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 -- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c414ff00 msgr2=0x7fb9c415fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c414ff00 0x7fb9c415fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 -- 172.21.15.53:0/2183443127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c41603c0 msgr2=0x7fb9c4160840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c41603c0 0x7fb9c4160840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3e00640 1 -- 172.21.15.53:0/2183443127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c415f0d0 con 0x7fb9a4005180 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.052+0000 7fb9c3400640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c414ff00 0x7fb9c415fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:44.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.053+0000 7fb9c9200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c41603c0 0x7fb9c4160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:44.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.053+0000 7fb9c3e00640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 0x7fb9c414f9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4014de0 tx=0x7fb9b4017490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:44.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.053+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4019040 con 0x7fb9a4005180 2024-01-16T15:07:44.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.054+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b4007220 con 0x7fb9a4005180 2024-01-16T15:07:44.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.054+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c41513c0 con 0x7fb9a4005180 2024-01-16T15:07:44.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.054+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c4150cc0 con 0x7fb9a4005180 2024-01-16T15:07:44.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.054+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9b4007610 con 0x7fb9a4005180 2024-01-16T15:07:44.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.057+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb994005250 con 0x7fb9a4005180 2024-01-16T15:07:44.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.059+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7fb9b40067d0 con 0x7fb9a4005180 2024-01-16T15:07:44.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.061+0000 7fb9bbe00640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb99c06bd20 0x7fb99c06e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:44.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.061+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9b4090f30 con 0x7fb9a4005180 2024-01-16T15:07:44.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.062+0000 7fb9c3400640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb99c06bd20 0x7fb99c06e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:44.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.064+0000 7fb9c3400640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb99c06bd20 0x7fb99c06e1e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0017a0 tx=0x7fb9ac009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:44.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.069+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9b405d5f0 con 0x7fb9a4005180 2024-01-16T15:07:44.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.379+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone30234", "target": ["mon-mgr", ""]}) v1 -- 0x7fb994000bc0 con 0x7fb99c06bd20 2024-01-16T15:07:44.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.418+0000 7fb9bbe00640 1 -- 172.21.15.53:0/2183443127 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb994000bc0 con 0x7fb99c06bd20 2024-01-16T15:07:44.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb99c06bd20 msgr2=0x7fb99c06e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:44.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb99c06bd20 0x7fb99c06e1e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fb9ac0017a0 tx=0x7fb9ac009040 comp rx=0 tx=0).stop 2024-01-16T15:07:44.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 msgr2=0x7fb9c414f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 0x7fb9c414f9c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4014de0 tx=0x7fb9b4017490 comp rx=0 tx=0).stop 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 shutdown_connections 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb99c06bd20 0x7fb99c06e1e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb9c41603c0 0x7fb9c4160840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb9c414ff00 0x7fb9c415fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 --2- 172.21.15.53:0/2183443127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb9a4005180 0x7fb9c414f9c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 >> 172.21.15.53:0/2183443127 conn(0x7fb9c40b0be0 msgr2=0x7fb9c41598b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.420+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 shutdown_connections 2024-01-16T15:07:44.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:44.421+0000 7fb9cb200640 1 -- 172.21.15.53:0/2183443127 wait complete. 2024-01-16T15:07:44.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:44.417 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:07:44.786 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-01-16T15:07:44.788 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:07:45.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.090+0000 7f774a800640 1 Processor -- start 2024-01-16T15:07:45.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.090+0000 7f774a800640 1 -- start start 2024-01-16T15:07:45.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.092+0000 7f774a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f774414b2c0 0x7f77440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.092+0000 7f774a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7744071070 0x7f77440cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.092+0000 7f774a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77440cfbf0 con 0x7f77440cf7d0 2024-01-16T15:07:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.092+0000 7f774a800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774414b6a0 con 0x7f774414b2c0 2024-01-16T15:07:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.092+0000 7f774a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7744071690 con 0x7f7744071070 2024-01-16T15:07:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.093+0000 7f7743e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7744071070 0x7f77440cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:45.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.093+0000 7f7743e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7744071070 0x7f77440cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:58746/0 (socket says 172.21.15.53:58746) 2024-01-16T15:07:45.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.093+0000 7f7743400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f774414b2c0 0x7f77440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:45.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.093+0000 7f7743e00640 1 -- 172.21.15.53:0/45447120 learned_addr learned my addr 172.21.15.53:0/45447120 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:45.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.093+0000 7f7742a00640 1 --1- 172.21.15.53:0/45447120 >> v1:172.21.15.146:6789/0 conn(0x7f77440cf7d0 0x7f7744070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:42244/0 (socket says 172.21.15.53:42244) 2024-01-16T15:07:45.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.094+0000 7f7743e00640 1 -- 172.21.15.53:0/45447120 >> v2:172.21.15.53:3300/0 conn(0x7f774414b2c0 msgr2=0x7f77440cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:45.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.094+0000 7f7743e00640 1 --2- 172.21.15.53:0/45447120 >> v2:172.21.15.53:3300/0 conn(0x7f774414b2c0 0x7f77440cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.094+0000 7f7743e00640 1 -- 172.21.15.53:0/45447120 >> v1:172.21.15.146:6789/0 conn(0x7f77440cf7d0 legacy=0x7f7744070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:45.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.094+0000 7f7743e00640 1 -- 172.21.15.53:0/45447120 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744071970 con 0x7f7744071070 2024-01-16T15:07:45.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.094+0000 7f7743400640 1 --2- 172.21.15.53:0/45447120 >> v2:172.21.15.53:3300/0 conn(0x7f774414b2c0 0x7f77440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:45.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.096+0000 7f7743e00640 1 --2- 172.21.15.53:0/45447120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77440cc160 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7744165330 tx=0x7f773402b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1556b87a8b785a73 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:45.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.097+0000 7f7742000640 1 -- 172.21.15.53:0/45447120 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734030860 con 0x7f7744071070 2024-01-16T15:07:45.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.097+0000 7f7742000640 1 -- 172.21.15.53:0/45447120 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77340309c0 con 0x7f7744071070 2024-01-16T15:07:45.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.097+0000 7f7742000640 1 -- 172.21.15.53:0/45447120 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7734030d50 con 0x7f7744071070 2024-01-16T15:07:45.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.098+0000 7f774a800640 1 -- 172.21.15.53:0/45447120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 msgr2=0x7f77440cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:45.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.098+0000 7f774a800640 1 --2- 172.21.15.53:0/45447120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77440cc160 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f7744165330 tx=0x7f773402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.098+0000 7f774a800640 1 -- 172.21.15.53:0/45447120 shutdown_connections 2024-01-16T15:07:45.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.098+0000 7f774a800640 1 --2- 172.21.15.53:0/45447120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77440cc160 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.098+0000 7f774a800640 1 --2- 172.21.15.53:0/45447120 >> v2:172.21.15.53:3300/0 conn(0x7f774414b2c0 0x7f77440cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.098+0000 7f774a800640 1 -- 172.21.15.53:0/45447120 >> 172.21.15.53:0/45447120 conn(0x7f774406a800 msgr2=0x7f774406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:45.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.100+0000 7f774a800640 1 -- 172.21.15.53:0/45447120 shutdown_connections 2024-01-16T15:07:45.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.100+0000 7f774a800640 1 -- 172.21.15.53:0/45447120 wait complete. 2024-01-16T15:07:45.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.102+0000 7f774a800640 1 Processor -- start 2024-01-16T15:07:45.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.102+0000 7f774a800640 1 -- start start 2024-01-16T15:07:45.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.103+0000 7f774a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77441420a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:45.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f774a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77440cf7d0 0x7f7744162b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:45.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f774a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774414b2c0 0x7f774415fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f774a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77441425e0 con 0x7f77440cf7d0 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f774a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7744142e60 con 0x7f7744071070 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f774a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7744143140 con 0x7f774414b2c0 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7742a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77440cf7d0 0x7f7744162b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7743400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77441420a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7743e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774414b2c0 0x7f774415fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7743400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77441420a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:58754/0 (socket says 172.21.15.53:58754) 2024-01-16T15:07:45.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7743400640 1 -- 172.21.15.53:0/942268152 learned_addr learned my addr 172.21.15.53:0/942268152 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7742a00640 1 -- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774414b2c0 msgr2=0x7f774415fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7742a00640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774414b2c0 0x7f774415fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7742a00640 1 -- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 msgr2=0x7f77441420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7742a00640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77441420a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7742a00640 1 -- 172.21.15.53:0/942268152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f773c009a70 con 0x7f77440cf7d0 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7743e00640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774414b2c0 0x7f774415fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:45.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.105+0000 7f7743400640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77441420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:45.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.106+0000 7f7742a00640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77440cf7d0 0x7f7744162b70 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c430 tx=0x7f772c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:45.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.107+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c0074e0 con 0x7f77440cf7d0 2024-01-16T15:07:45.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.107+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f772c007640 con 0x7f77440cf7d0 2024-01-16T15:07:45.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.107+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f772c004a00 con 0x7f77440cf7d0 2024-01-16T15:07:45.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.107+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77340094d0 con 0x7f77440cf7d0 2024-01-16T15:07:45.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.107+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7744143700 con 0x7f77440cf7d0 2024-01-16T15:07:45.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.110+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7714005250 con 0x7f77440cf7d0 2024-01-16T15:07:45.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.113+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 20) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f772c004b60 con 0x7f77440cf7d0 2024-01-16T15:07:45.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.114+0000 7f773b400640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f771c06bc50 0x7f771c06e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:45.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.114+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f772c087ed0 con 0x7f77440cf7d0 2024-01-16T15:07:45.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.116+0000 7f7743400640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f771c06bc50 0x7f771c06e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.117+0000 7f7743400640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f771c06bc50 0x7f771c06e110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f773c003de0 tx=0x7f773c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.122+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f772c0546a0 con 0x7f77440cf7d0 2024-01-16T15:07:45.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:45.529+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f7714000bc0 con 0x7f771c06bc50 2024-01-16T15:07:45.541 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:07:45.564+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:07:45.637 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:07:47.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:47.220+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f772c054a70 con 0x7f77440cf7d0 2024-01-16T15:07:48.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.594+0000 7f773b400640 1 -- 172.21.15.53:0/942268152 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f7714000bc0 con 0x7f771c06bc50 2024-01-16T15:07:48.572 INFO:teuthology.orchestra.run.smithi053.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.598+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f771c06bc50 msgr2=0x7f771c06e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.598+0000 7f774a800640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f771c06bc50 0x7f771c06e110 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f773c003de0 tx=0x7f773c00b040 comp rx=0 tx=0).stop 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.598+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77440cf7d0 msgr2=0x7f7744162b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.598+0000 7f774a800640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77440cf7d0 0x7f7744162b70 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f772c00c430 tx=0x7f772c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 shutdown_connections 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f771c06bc50 0x7f771c06e110 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774414b2c0 0x7f774415fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f77440cf7d0 0x7f7744162b70 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 --2- 172.21.15.53:0/942268152 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7744071070 0x7f77441420a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 >> 172.21.15.53:0/942268152 conn(0x7f774406a800 msgr2=0x7f7744149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 shutdown_connections 2024-01-16T15:07:48.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:48.599+0000 7f774a800640 1 -- 172.21.15.53:0/942268152 wait complete. 2024-01-16T15:07:48.598 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:07:48.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:48.744 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:48.904 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:07:48.905 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:07:48.905 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:48.971 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:49.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.197+0000 7f836dc00640 1 Processor -- start 2024-01-16T15:07:49.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.197+0000 7f836dc00640 1 -- start start 2024-01-16T15:07:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.198+0000 7f836dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 0x7f836815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f836dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836806c470 con 0x7f836813e360 2024-01-16T15:07:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f836dc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f836806c750 con 0x7f83680cf9a0 2024-01-16T15:07:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f836dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f836806ca30 con 0x7f836813df80 2024-01-16T15:07:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f835fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f836813e360 0x7f83680cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51744/0 (socket says 172.21.15.53:51744) 2024-01-16T15:07:49.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f835fe00640 1 -- 172.21.15.53:0/536882140 learned_addr learned my addr 172.21.15.53:0/536882140 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677218104 0 0) 0x7f836806c470 con 0x7f836813e360 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8348003520 con 0x7f836813e360 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.199+0000 7f8366a00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 0x7f836815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3144874974 0 0) 0x7f8348003520 con 0x7f836813e360 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 msgr2=0x7f836815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 0x7f836815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 >> v1:172.21.15.146:6789/0 conn(0x7f83680cf9a0 legacy=0x7f83680cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f836806cd10 con 0x7f836813e360 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8350001670 con 0x7f836813e360 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 >> v1:172.21.15.53:6789/0 conn(0x7f836813e360 legacy=0x7f83680cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83480039a0 0x7f8348003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83480047c0 0x7f8348004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 0x7f8348005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836806ca30 con 0x7f83480039a0 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f834800bb10 con 0x7f83480047c0 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f834800bca0 con 0x7f836813df80 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366a00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83480039a0 0x7f8348003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8367400640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 0x7f8348005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f835fe00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83480047c0 0x7f8348004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366a00640 1 -- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 msgr2=0x7f8348005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366a00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f836813df80 0x7f8348005230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366a00640 1 -- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83480047c0 msgr2=0x7f8348004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:49.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366a00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83480047c0 0x7f8348004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.200+0000 7f8366a00640 1 -- 172.21.15.53:0/536882140 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8348003520 con 0x7f83480039a0 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f8366a00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83480039a0 0x7f8348003da0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f835800ba30 tx=0x7f83480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9976e93635ee0d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358032640 con 0x7f83480039a0 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83580327a0 con 0x7f83480039a0 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f8366000640 1 -- 172.21.15.53:0/536882140 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8358032b30 con 0x7f83480039a0 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f836dc00640 1 -- 172.21.15.53:0/536882140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83480039a0 msgr2=0x7f8348003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f836dc00640 1 --2- 172.21.15.53:0/536882140 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83480039a0 0x7f8348003da0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f835800ba30 tx=0x7f83480057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.202+0000 7f8367400640 1 -- 172.21.15.53:0/536882140 reap_dead start 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.203+0000 7f836dc00640 1 -- 172.21.15.53:0/536882140 shutdown_connections 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.203+0000 7f836dc00640 1 -- 172.21.15.53:0/536882140 >> 172.21.15.53:0/536882140 conn(0x7f836806a800 msgr2=0x7f836806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.203+0000 7f836dc00640 1 -- 172.21.15.53:0/536882140 shutdown_connections 2024-01-16T15:07:49.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.203+0000 7f836dc00640 1 -- 172.21.15.53:0/536882140 wait complete. 2024-01-16T15:07:49.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.204+0000 7f836dc00640 1 Processor -- start 2024-01-16T15:07:49.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.204+0000 7f836dc00640 1 -- start start 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f836dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83480039a0 0x7f83681474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f836dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 0x7f836813d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f836dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8368146930 0x7f8368146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f836dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836813de10 con 0x7f83680afb60 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f836dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83681472f0 con 0x7f8368146930 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f836dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83680b40d0 con 0x7f83480039a0 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.205+0000 7f835fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 0x7f836813d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 0x7f836813d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46954/0 (socket says 172.21.15.53:46954) 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 -- 172.21.15.53:0/3955293381 learned_addr learned my addr 172.21.15.53:0/3955293381 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f8366a00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83480039a0 0x7f83681474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 -- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83480039a0 msgr2=0x7f83681474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83480039a0 0x7f83681474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 -- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8368146930 msgr2=0x7f8368146db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8368146930 0x7f8368146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 -- 172.21.15.53:0/3955293381 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83680affc0 con 0x7f83680afb60 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835fe00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 0x7f836813d8d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f835000d7f0 tx=0x7f835000dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350007560 con 0x7f83680afb60 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83500047e0 con 0x7f83680afb60 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8350004b70 con 0x7f83680afb60 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.206+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83680b3830 con 0x7f83680afb60 2024-01-16T15:07:49.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.207+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83680b3df0 con 0x7f83680afb60 2024-01-16T15:07:49.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.208+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83680675d0 con 0x7f83680afb60 2024-01-16T15:07:49.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.208+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8350004cd0 con 0x7f83680afb60 2024-01-16T15:07:49.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.209+0000 7f835ea00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f834006bbf0 0x7f834006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.209+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f8350087e40 con 0x7f83680afb60 2024-01-16T15:07:49.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.210+0000 7f8366a00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f834006bbf0 0x7f834006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.211+0000 7f8366a00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f834006bbf0 0x7f834006e0b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f835800ba00 tx=0x7f8358030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:49.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.214+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8350054700 con 0x7f83680afb60 2024-01-16T15:07:49.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.399+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f836815beb0 con 0x7f83680afb60 2024-01-16T15:07:49.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.401+0000 7f835ea00640 1 -- 172.21.15.53:0/3955293381 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12995 (secure 0 0 0) 0x7f8350019090 con 0x7f83680afb60 2024-01-16T15:07:49.379 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:49.379 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":64,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:07:48.577053+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6837/3568122877":"2024-01-17T15:07:45.564678+0000","172.21.15.146:6835/3568122877":"2024-01-17T15:07:45.564678+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:49.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f834006bbf0 msgr2=0x7f834006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f834006bbf0 0x7f834006e0b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f835800ba00 tx=0x7f8358030000 comp rx=0 tx=0).stop 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 msgr2=0x7f836813d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 0x7f836813d8d0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f835000d7f0 tx=0x7f835000dcb0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 shutdown_connections 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f834006bbf0 0x7f834006e0b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8368146930 0x7f8368146db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83680afb60 0x7f836813d8d0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 --2- 172.21.15.53:0/3955293381 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83480039a0 0x7f83681474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 >> 172.21.15.53:0/3955293381 conn(0x7f836806a800 msgr2=0x7f836815d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 shutdown_connections 2024-01-16T15:07:49.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.404+0000 7f836dc00640 1 -- 172.21.15.53:0/3955293381 wait complete. 2024-01-16T15:07:49.398 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:49.398 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:49.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.639+0000 7f0c67400640 1 Processor -- start 2024-01-16T15:07:49.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.639+0000 7f0c67400640 1 -- start start 2024-01-16T15:07:49.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.641+0000 7f0c67400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c6006ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.641+0000 7f0c67400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0c6009a170 0x7f0c6006e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.641+0000 7f0c67400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600cfa70 con 0x7f0c60070490 2024-01-16T15:07:49.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.641+0000 7f0c67400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c600cff00 con 0x7f0c6009a170 2024-01-16T15:07:49.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.641+0000 7f0c67400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c6006fc70 con 0x7f0c6015e880 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.641+0000 7f0c65400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0c6009a170 0x7f0c6006e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.642+0000 7f0c65400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0c6009a170 0x7f0c6006e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46960/0 (socket says 172.21.15.53:46960) 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.642+0000 7f0c65400640 1 -- 172.21.15.53:0/1684092320 learned_addr learned my addr 172.21.15.53:0/1684092320 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.642+0000 7f0c64a00640 1 --2- 172.21.15.53:0/1684092320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c6006ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.643+0000 7f0c64a00640 1 -- 172.21.15.53:0/1684092320 >> v2:172.21.15.53:3300/0 conn(0x7f0c6009a170 msgr2=0x7f0c6006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.643+0000 7f0c64a00640 1 --2- 172.21.15.53:0/1684092320 >> v2:172.21.15.53:3300/0 conn(0x7f0c6009a170 0x7f0c6006e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.643+0000 7f0c64a00640 1 -- 172.21.15.53:0/1684092320 >> v1:172.21.15.146:6789/0 conn(0x7f0c60070490 legacy=0x7f0c60099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:49.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.643+0000 7f0c64a00640 1 -- 172.21.15.53:0/1684092320 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c600cf420 con 0x7f0c6015e880 2024-01-16T15:07:49.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.647+0000 7f0c64a00640 1 --2- 172.21.15.53:0/1684092320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c6006ff50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c480097a0 tx=0x7f0c4802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7cdf01ee369ff47 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:49.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.648+0000 7f0c5f400640 1 -- 172.21.15.53:0/1684092320 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48031830 con 0x7f0c6015e880 2024-01-16T15:07:49.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.648+0000 7f0c5f400640 1 -- 172.21.15.53:0/1684092320 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c48031990 con 0x7f0c6015e880 2024-01-16T15:07:49.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.648+0000 7f0c5f400640 1 -- 172.21.15.53:0/1684092320 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48031d20 con 0x7f0c6015e880 2024-01-16T15:07:49.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.649+0000 7f0c67400640 1 -- 172.21.15.53:0/1684092320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 msgr2=0x7f0c6006ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.649+0000 7f0c67400640 1 --2- 172.21.15.53:0/1684092320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c6006ff50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0c480097a0 tx=0x7f0c4802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.650+0000 7f0c67400640 1 -- 172.21.15.53:0/1684092320 shutdown_connections 2024-01-16T15:07:49.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.650+0000 7f0c67400640 1 --2- 172.21.15.53:0/1684092320 >> v2:172.21.15.53:3300/0 conn(0x7f0c6009a170 0x7f0c6006e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.650+0000 7f0c67400640 1 --2- 172.21.15.53:0/1684092320 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c6006ff50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.650+0000 7f0c67400640 1 -- 172.21.15.53:0/1684092320 >> 172.21.15.53:0/1684092320 conn(0x7f0c600b2d00 msgr2=0x7f0c600b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:49.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.650+0000 7f0c67400640 1 -- 172.21.15.53:0/1684092320 shutdown_connections 2024-01-16T15:07:49.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.650+0000 7f0c67400640 1 -- 172.21.15.53:0/1684092320 wait complete. 2024-01-16T15:07:49.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.654+0000 7f0c67400640 1 Processor -- start 2024-01-16T15:07:49.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.657+0000 7f0c67400640 1 -- start start 2024-01-16T15:07:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.658+0000 7f0c67400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60070490 0x7f0c601577a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.659+0000 7f0c67400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c6009a170 0x7f0c600b10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.659+0000 7f0c67400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c600aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.659+0000 7f0c67400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c60157ce0 con 0x7f0c60070490 2024-01-16T15:07:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.659+0000 7f0c67400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600b1620 con 0x7f0c6009a170 2024-01-16T15:07:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.659+0000 7f0c67400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c600af1a0 con 0x7f0c6015e880 2024-01-16T15:07:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.659+0000 7f0c65400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c600aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.660+0000 7f0c65400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c600aec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:39074/0 (socket says 172.21.15.53:39074) 2024-01-16T15:07:49.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.660+0000 7f0c65400640 1 -- 172.21.15.53:0/3181081469 learned_addr learned my addr 172.21.15.53:0/3181081469 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:49.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.660+0000 7f0c64a00640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60070490 0x7f0c601577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.661+0000 7f0c5fe00640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c6009a170 0x7f0c600b10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.661+0000 7f0c65400640 1 -- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c6009a170 msgr2=0x7f0c600b10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.661+0000 7f0c65400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c6009a170 0x7f0c600b10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.661+0000 7f0c65400640 1 -- 172.21.15.53:0/3181081469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60070490 msgr2=0x7f0c601577a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.662+0000 7f0c65400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60070490 0x7f0c601577a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.662+0000 7f0c65400640 1 -- 172.21.15.53:0/3181081469 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c48009400 con 0x7f0c6015e880 2024-01-16T15:07:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.662+0000 7f0c64a00640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60070490 0x7f0c601577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:07:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.663+0000 7f0c65400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c600aec60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800e840 tx=0x7f0c5800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.664+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c58009660 con 0x7f0c6015e880 2024-01-16T15:07:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.664+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5800c940 con 0x7f0c6015e880 2024-01-16T15:07:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.664+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c5800cd10 con 0x7f0c6015e880 2024-01-16T15:07:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.664+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c600b1880 con 0x7f0c6015e880 2024-01-16T15:07:49.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.664+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c600b1e40 con 0x7f0c6015e880 2024-01-16T15:07:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.666+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c600675d0 con 0x7f0c6015e880 2024-01-16T15:07:49.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.668+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f0c58013470 con 0x7f0c6015e880 2024-01-16T15:07:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.669+0000 7f0c5cc00640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c3806bc10 0x7f0c3806e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.669+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f0c58088820 con 0x7f0c6015e880 2024-01-16T15:07:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.671+0000 7f0c64a00640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c3806bc10 0x7f0c3806e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:49.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.673+0000 7f0c64a00640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c3806bc10 0x7f0c3806e0d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0c48036000 tx=0x7f0c4803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:49.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.679+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c58054fa0 con 0x7f0c6015e880 2024-01-16T15:07:49.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.980+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0c60067d70 con 0x7f0c6015e880 2024-01-16T15:07:49.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.981+0000 7f0c5cc00640 1 -- 172.21.15.53:0/3181081469 <== mon.2 v2:172.21.15.146:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0c580210a0 con 0x7f0c6015e880 2024-01-16T15:07:49.958 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.983+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c3806bc10 msgr2=0x7f0c3806e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.983+0000 7f0c67400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c3806bc10 0x7f0c3806e0d0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f0c48036000 tx=0x7f0c4803c040 comp rx=0 tx=0).stop 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.983+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 msgr2=0x7f0c600aec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.983+0000 7f0c67400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c600aec60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0c5800e840 tx=0x7f0c5800ed00 comp rx=0 tx=0).stop 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.983+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 shutdown_connections 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.983+0000 7f0c67400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c3806bc10 0x7f0c3806e0d0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.984+0000 7f0c67400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c6015e880 0x7f0c600aec60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.984+0000 7f0c67400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c6009a170 0x7f0c600b10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.984+0000 7f0c67400640 1 --2- 172.21.15.53:0/3181081469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c60070490 0x7f0c601577a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:49.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.984+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 >> 172.21.15.53:0/3181081469 conn(0x7f0c600b2d00 msgr2=0x7f0c600b5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:49.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.984+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 shutdown_connections 2024-01-16T15:07:49.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:49.984+0000 7f0c67400640 1 -- 172.21.15.53:0/3181081469 wait complete. 2024-01-16T15:07:49.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:07:50.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.178+0000 7fbd2cc00640 1 Processor -- start 2024-01-16T15:07:50.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.178+0000 7fbd2cc00640 1 -- start start 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd2cc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd28143c40 0x7fbd28071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd280cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd2cc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd28070e50 con 0x7fbd280cca40 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd2cc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd280723c0 con 0x7fbd28143c40 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280cce70 con 0x7fbd280717d0 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd27e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbd280cca40 0x7fbd2815eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51746/0 (socket says 172.21.15.53:51746) 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd27e00640 1 -- 172.21.15.53:0/548685050 learned_addr learned my addr 172.21.15.53:0/548685050 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.180+0000 7fbd27400640 1 --2- 172.21.15.53:0/548685050 >> v2:172.21.15.53:3300/0 conn(0x7fbd28143c40 0x7fbd28071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.181+0000 7fbd1fe00640 1 --2- 172.21.15.53:0/548685050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd280cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.181+0000 7fbd27400640 1 -- 172.21.15.53:0/548685050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 msgr2=0x7fbd280cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:50.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.181+0000 7fbd27400640 1 --2- 172.21.15.53:0/548685050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd280cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.181+0000 7fbd27400640 1 -- 172.21.15.53:0/548685050 >> v1:172.21.15.53:6789/0 conn(0x7fbd280cca40 legacy=0x7fbd2815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:50.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.181+0000 7fbd26a00640 1 -- 172.21.15.53:0/548685050 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90499333 0 0) 0x7fbd28070e50 con 0x7fbd280cca40 2024-01-16T15:07:50.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.181+0000 7fbd27400640 1 -- 172.21.15.53:0/548685050 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd2806e170 con 0x7fbd28143c40 2024-01-16T15:07:50.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.182+0000 7fbd27400640 1 --2- 172.21.15.53:0/548685050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28071290 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbd180097a0 tx=0x7fbd1802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e9c33210752b64d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:50.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd26a00640 1 -- 172.21.15.53:0/548685050 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1802e070 con 0x7fbd28143c40 2024-01-16T15:07:50.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd26a00640 1 -- 172.21.15.53:0/548685050 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd180315a0 con 0x7fbd28143c40 2024-01-16T15:07:50.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd26a00640 1 -- 172.21.15.53:0/548685050 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd18031930 con 0x7fbd28143c40 2024-01-16T15:07:50.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd2cc00640 1 -- 172.21.15.53:0/548685050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 msgr2=0x7fbd28071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:50.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/548685050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28071290 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7fbd180097a0 tx=0x7fbd1802b840 comp rx=0 tx=0).stop 2024-01-16T15:07:50.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd2cc00640 1 -- 172.21.15.53:0/548685050 shutdown_connections 2024-01-16T15:07:50.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/548685050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd280cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.183+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/548685050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28071290 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.184+0000 7fbd2cc00640 1 -- 172.21.15.53:0/548685050 >> 172.21.15.53:0/548685050 conn(0x7fbd2806a800 msgr2=0x7fbd2806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:50.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.184+0000 7fbd2cc00640 1 -- 172.21.15.53:0/548685050 shutdown_connections 2024-01-16T15:07:50.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.184+0000 7fbd2cc00640 1 -- 172.21.15.53:0/548685050 wait complete. 2024-01-16T15:07:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.186+0000 7fbd2cc00640 1 Processor -- start 2024-01-16T15:07:50.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.186+0000 7fbd2cc00640 1 -- start start 2024-01-16T15:07:50.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.187+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd28141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:50.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.187+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd280cca40 0x7fbd280ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:50.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.187+0000 7fbd2cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28157a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:50.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.187+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280ad820 con 0x7fbd28143c40 2024-01-16T15:07:50.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.187+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd28141e70 con 0x7fbd280cca40 2024-01-16T15:07:50.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.187+0000 7fbd2cc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd280b1f90 con 0x7fbd280717d0 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28157a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28157a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46986/0 (socket says 172.21.15.53:46986) 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 -- 172.21.15.53:0/1025718233 learned_addr learned my addr 172.21.15.53:0/1025718233 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27400640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd28141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd1fe00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd280cca40 0x7fbd280ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 -- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 msgr2=0x7fbd28141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd28141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 -- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd280cca40 msgr2=0x7fbd280ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd280cca40 0x7fbd280ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd27e00640 1 -- 172.21.15.53:0/1025718233 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd18009400 con 0x7fbd28143c40 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.188+0000 7fbd1fe00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd280cca40 0x7fbd280ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:07:50.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.189+0000 7fbd27e00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28157a20 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000c7e0 tx=0x7fbd2000cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:50.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.189+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2000a640 con 0x7fbd28143c40 2024-01-16T15:07:50.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.189+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd2000a7c0 con 0x7fbd28143c40 2024-01-16T15:07:50.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.189+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd2000ab70 con 0x7fbd28143c40 2024-01-16T15:07:50.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.189+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd280b2270 con 0x7fbd28143c40 2024-01-16T15:07:50.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.189+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd280b2830 con 0x7fbd28143c40 2024-01-16T15:07:50.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.190+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd281434d0 con 0x7fbd28143c40 2024-01-16T15:07:50.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.191+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fbd2000acd0 con 0x7fbd28143c40 2024-01-16T15:07:50.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.192+0000 7fbd1f400640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd0006bbc0 0x7fbd0006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:50.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.192+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fbd20087980 con 0x7fbd28143c40 2024-01-16T15:07:50.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.193+0000 7fbd27400640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd0006bbc0 0x7fbd0006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:50.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.194+0000 7fbd27400640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd0006bbc0 0x7fbd0006e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbd18036000 tx=0x7fbd1803d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.196+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd20054350 con 0x7fbd28143c40 2024-01-16T15:07:50.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.395+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbd280b2a50 con 0x7fbd28143c40 2024-01-16T15:07:50.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.396+0000 7fbd1f400640 1 -- 172.21.15.53:0/1025718233 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v64) v1 ==== 80+0+124 (secure 0 0 0) 0x7fbd20018080 con 0x7fbd28143c40 2024-01-16T15:07:50.374 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6837/3568122877 2024-01-17T15:07:45.564678+0000 2024-01-16T15:07:50.374 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6835/3568122877 2024-01-17T15:07:45.564678+0000 2024-01-16T15:07:50.374 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd0006bbc0 msgr2=0x7fbd0006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd0006bbc0 0x7fbd0006e080 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbd18036000 tx=0x7fbd1803d040 comp rx=0 tx=0).stop 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 msgr2=0x7fbd28157a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28157a20 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7fbd2000c7e0 tx=0x7fbd2000cca0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 shutdown_connections 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbd0006bbc0 0x7fbd0006e080 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd28143c40 0x7fbd28157a20 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd280cca40 0x7fbd280ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 --2- 172.21.15.53:0/1025718233 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd280717d0 0x7fbd28141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:50.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.398+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 >> 172.21.15.53:0/1025718233 conn(0x7fbd2806a800 msgr2=0x7fbd2806d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:50.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.399+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 shutdown_connections 2024-01-16T15:07:50.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:50.399+0000 7fbd2cc00640 1 -- 172.21.15.53:0/1025718233 wait complete. 2024-01-16T15:07:50.392 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:07:50.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:50.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:07:50.483 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:07:50.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:50.483 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:07:50.489 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[56436]: fuse finished with error 0 and tester_r 0 2024-01-16T15:07:50.710 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:07:50.710 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:50.765 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:50.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.021+0000 7f1de3200640 1 Processor -- start 2024-01-16T15:07:50.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.021+0000 7f1de3200640 1 -- start start 2024-01-16T15:07:51.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1de3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1de41542b0 0x7f1de4160e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1de3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1de3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de40cbe50 con 0x7f1de40ccde0 2024-01-16T15:07:51.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1de3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de40cc410 con 0x7f1de41542b0 2024-01-16T15:07:51.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1de3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4154690 con 0x7f1de4161340 2024-01-16T15:07:51.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1de1e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1de41542b0 0x7f1de4160e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.023+0000 7f1dd9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1dd9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de40cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:39090/0 (socket says 172.21.15.53:39090) 2024-01-16T15:07:51.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1dd9200640 1 -- 172.21.15.53:0/1229682247 learned_addr learned my addr 172.21.15.53:0/1229682247 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:51.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1de1e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1de41542b0 0x7f1de4160e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:58790/0 (socket says 172.21.15.53:58790) 2024-01-16T15:07:51.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1de1e00640 1 -- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 msgr2=0x7f1de40cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1de1e00640 1 --2- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de40cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1de1e00640 1 -- 172.21.15.53:0/1229682247 >> v1:172.21.15.146:6789/0 conn(0x7f1de40ccde0 legacy=0x7f1de406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.024+0000 7f1de1e00640 1 -- 172.21.15.53:0/1229682247 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de4161810 con 0x7f1de41542b0 2024-01-16T15:07:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.026+0000 7f1de1e00640 1 --2- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de41542b0 0x7f1de4160e00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009d90 tx=0x7f1dd402bf30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f59a43a3ef384778 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.026+0000 7f1de1400640 1 -- 172.21.15.53:0/1229682247 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd402cce0 con 0x7f1de41542b0 2024-01-16T15:07:51.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.026+0000 7f1de1400640 1 -- 172.21.15.53:0/1229682247 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd402ce40 con 0x7f1de41542b0 2024-01-16T15:07:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.026+0000 7f1de1400640 1 -- 172.21.15.53:0/1229682247 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd4034660 con 0x7f1de41542b0 2024-01-16T15:07:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 -- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de41542b0 msgr2=0x7f1de4160e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 --2- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de41542b0 0x7f1de4160e00 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4009d90 tx=0x7f1dd402bf30 comp rx=0 tx=0).stop 2024-01-16T15:07:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 -- 172.21.15.53:0/1229682247 shutdown_connections 2024-01-16T15:07:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 --2- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de40cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 --2- 172.21.15.53:0/1229682247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de41542b0 0x7f1de4160e00 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 -- 172.21.15.53:0/1229682247 >> 172.21.15.53:0/1229682247 conn(0x7f1de40b4150 msgr2=0x7f1de40b4810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 -- 172.21.15.53:0/1229682247 shutdown_connections 2024-01-16T15:07:51.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.027+0000 7f1de3200640 1 -- 172.21.15.53:0/1229682247 wait complete. 2024-01-16T15:07:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.029+0000 7f1de3200640 1 Processor -- start 2024-01-16T15:07:51.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.029+0000 7f1de3200640 1 -- start start 2024-01-16T15:07:51.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1de3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de40ccde0 0x7f1de4145ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1de3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 0x7f1de40cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1de3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de406da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1de3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4146010 con 0x7f1de41542b0 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1de3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de41f64d0 con 0x7f1de40ccde0 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1de3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de41f67b0 con 0x7f1de4161340 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1dd9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 0x7f1de40cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1dd9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 0x7f1de40cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46998/0 (socket says 172.21.15.53:46998) 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.030+0000 7f1dd9200640 1 -- 172.21.15.53:0/365582013 learned_addr learned my addr 172.21.15.53:0/365582013 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1dd9200640 1 -- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 msgr2=0x7f1de406da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1dd9200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de406da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1dd9200640 1 -- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de40ccde0 msgr2=0x7f1de4145ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1dd9200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de40ccde0 0x7f1de4145ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1dd9200640 1 -- 172.21.15.53:0/365582013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd4009a70 con 0x7f1de41542b0 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1dd9200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 0x7f1de40cae10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00e870 tx=0x7f1dcc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00c940 con 0x7f1de41542b0 2024-01-16T15:07:51.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dcc00caa0 con 0x7f1de41542b0 2024-01-16T15:07:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc00ce30 con 0x7f1de41542b0 2024-01-16T15:07:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de41f6a90 con 0x7f1de41542b0 2024-01-16T15:07:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.031+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de41f7050 con 0x7f1de41542b0 2024-01-16T15:07:51.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.033+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de4067b60 con 0x7f1de41542b0 2024-01-16T15:07:51.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.034+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1dcc013470 con 0x7f1de41542b0 2024-01-16T15:07:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.035+0000 7f1ddaa00640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dbc06bbc0 0x7f1dbc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.035+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1dcc0887b0 con 0x7f1de41542b0 2024-01-16T15:07:51.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.035+0000 7f1de1e00640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dbc06bbc0 0x7f1dbc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.036+0000 7f1de1e00640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dbc06bbc0 0x7f1dbc06e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4036000 tx=0x7f1dd402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.039+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1dcc054e80 con 0x7f1de41542b0 2024-01-16T15:07:51.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.232+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1de4070b10 con 0x7f1de41542b0 2024-01-16T15:07:51.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.233+0000 7f1ddaa00640 1 -- 172.21.15.53:0/365582013 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12995 (secure 0 0 0) 0x7f1dcc017090 con 0x7f1de41542b0 2024-01-16T15:07:51.211 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:51.211 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":64,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:07:48.577053+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6837/3568122877":"2024-01-17T15:07:45.564678+0000","172.21.15.146:6835/3568122877":"2024-01-17T15:07:45.564678+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dbc06bbc0 msgr2=0x7f1dbc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dbc06bbc0 0x7f1dbc06e080 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f1dd4036000 tx=0x7f1dd402e000 comp rx=0 tx=0).stop 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 msgr2=0x7f1de40cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 0x7f1de40cae10 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f1dcc00e870 tx=0x7f1dcc00ed30 comp rx=0 tx=0).stop 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 shutdown_connections 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1dbc06bbc0 0x7f1dbc06e080 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1de4161340 0x7f1de406da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1de41542b0 0x7f1de40cae10 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 --2- 172.21.15.53:0/365582013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1de40ccde0 0x7f1de4145ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 >> 172.21.15.53:0/365582013 conn(0x7f1de40b4150 msgr2=0x7f1de40acab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 shutdown_connections 2024-01-16T15:07:51.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.236+0000 7f1de3200640 1 -- 172.21.15.53:0/365582013 wait complete. 2024-01-16T15:07:51.232 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:51.232 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:51.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.482+0000 7fb2b4c00640 1 Processor -- start 2024-01-16T15:07:51.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.482+0000 7fb2b4c00640 1 -- start start 2024-01-16T15:07:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.483+0000 7fb2b4c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb2b013df80 0x7fb2b015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2b4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2b4c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2b006c470 con 0x7fb2b015f9c0 2024-01-16T15:07:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2b4c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2b006c750 con 0x7fb2b013df80 2024-01-16T15:07:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2b4c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b006ca30 con 0x7fb2b013e360 2024-01-16T15:07:51.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2ade00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb2b013df80 0x7fb2b015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2ade00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb2b013df80 0x7fb2b015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47010/0 (socket says 172.21.15.53:47010) 2024-01-16T15:07:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2ade00640 1 -- 172.21.15.53:0/3315829707 learned_addr learned my addr 172.21.15.53:0/3315829707 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.484+0000 7fb2ad400640 1 --2- 172.21.15.53:0/3315829707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.485+0000 7fb2ade00640 1 -- 172.21.15.53:0/3315829707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 msgr2=0x7fb2b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.485+0000 7fb2ade00640 1 --2- 172.21.15.53:0/3315829707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.485+0000 7fb2ade00640 1 -- 172.21.15.53:0/3315829707 >> v1:172.21.15.146:6789/0 conn(0x7fb2b015f9c0 legacy=0x7fb2b01408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.485+0000 7fb2ade00640 1 -- 172.21.15.53:0/3315829707 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2b006cd10 con 0x7fb2b013df80 2024-01-16T15:07:51.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.486+0000 7fb2ade00640 1 --2- 172.21.15.53:0/3315829707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b013df80 0x7fb2b015bee0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009870 tx=0x7fb2a002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2489efb125c1fcd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.486+0000 7fb2aca00640 1 -- 172.21.15.53:0/3315829707 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0030840 con 0x7fb2b013df80 2024-01-16T15:07:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.486+0000 7fb2aca00640 1 -- 172.21.15.53:0/3315829707 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a00309a0 con 0x7fb2b013df80 2024-01-16T15:07:51.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.486+0000 7fb2aca00640 1 -- 172.21.15.53:0/3315829707 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a0030d30 con 0x7fb2b013df80 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3315829707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b013df80 msgr2=0x7fb2b015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3315829707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b013df80 0x7fb2b015bee0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0009870 tx=0x7fb2a002b980 comp rx=0 tx=0).stop 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3315829707 shutdown_connections 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3315829707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3315829707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b013df80 0x7fb2b015bee0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3315829707 >> 172.21.15.53:0/3315829707 conn(0x7fb2b006a800 msgr2=0x7fb2b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3315829707 shutdown_connections 2024-01-16T15:07:51.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.487+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3315829707 wait complete. 2024-01-16T15:07:51.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.489+0000 7fb2b4c00640 1 Processor -- start 2024-01-16T15:07:51.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.489+0000 7fb2b4c00640 1 -- start start 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2b4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b013df80 0x7fb2b013d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2b4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b014ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2b4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 0x7fb2b00afab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2b4c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b014c810 con 0x7fb2b015f9c0 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2b4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b014bac0 con 0x7fb2b013df80 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2b4c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b014b500 con 0x7fb2b013e360 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2ae800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 0x7fb2b00afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2ae800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 0x7fb2b00afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47014/0 (socket says 172.21.15.53:47014) 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.490+0000 7fb2ae800640 1 -- 172.21.15.53:0/3843864911 learned_addr learned my addr 172.21.15.53:0/3843864911 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ad400640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b014ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ade00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b013df80 0x7fb2b013d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ae800640 1 -- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 msgr2=0x7fb2b014ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ae800640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b014ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ae800640 1 -- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b013df80 msgr2=0x7fb2b013d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ae800640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b013df80 0x7fb2b013d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ae800640 1 -- 172.21.15.53:0/3843864911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a00094d0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2ae800640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 0x7fb2b00afab0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800c7a0 tx=0x7fb2a800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.491+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8003b70 con 0x7fb2b015f9c0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.492+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2a8003cf0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.492+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b014b7e0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.492+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8003ea0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.492+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b013dc50 con 0x7fb2b015f9c0 2024-01-16T15:07:51.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.493+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b014c980 con 0x7fb2b015f9c0 2024-01-16T15:07:51.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.494+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb2a800aa90 con 0x7fb2b015f9c0 2024-01-16T15:07:51.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.495+0000 7fb2a6000640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bbc0 0x7fb28806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.495+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb2a8088310 con 0x7fb2b015f9c0 2024-01-16T15:07:51.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.495+0000 7fb2ade00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bbc0 0x7fb28806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.496+0000 7fb2ade00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bbc0 0x7fb28806e080 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0036000 tx=0x7fb2a003b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.499+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2a8054bd0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.694+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2b00675d0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.695+0000 7fb2a6000640 1 -- 172.21.15.53:0/3843864911 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2a80581f0 con 0x7fb2b015f9c0 2024-01-16T15:07:51.673 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bbc0 msgr2=0x7fb28806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bbc0 0x7fb28806e080 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0036000 tx=0x7fb2a003b040 comp rx=0 tx=0).stop 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 msgr2=0x7fb2b00afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 0x7fb2b00afab0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fb2a800c7a0 tx=0x7fb2a800cc60 comp rx=0 tx=0).stop 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 shutdown_connections 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb28806bbc0 0x7fb28806e080 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2b015f9c0 0x7fb2b00afab0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2b013e360 0x7fb2b014ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 --2- 172.21.15.53:0/3843864911 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2b013df80 0x7fb2b013d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.698+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 >> 172.21.15.53:0/3843864911 conn(0x7fb2b006a800 msgr2=0x7fb2b015cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.699+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 shutdown_connections 2024-01-16T15:07:51.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.699+0000 7fb2b4c00640 1 -- 172.21.15.53:0/3843864911 wait complete. 2024-01-16T15:07:51.696 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:07:51.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.937+0000 7fed23e00640 1 Processor -- start 2024-01-16T15:07:51.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.937+0000 7fed23e00640 1 -- start start 2024-01-16T15:07:51.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.939+0000 7fed23e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24160920 0x7fed240b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.939+0000 7fed23e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed240cf9f0 0x7fed240cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.939+0000 7fed23e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240d0a10 con 0x7fed24161010 2024-01-16T15:07:51.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.939+0000 7fed23e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed240d03b0 con 0x7fed240cf9f0 2024-01-16T15:07:51.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.939+0000 7fed23e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed240d1f50 con 0x7fed24160920 2024-01-16T15:07:51.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.940+0000 7fed1be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed240cf9f0 0x7fed240cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.940+0000 7fed22a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24160920 0x7fed240b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.940+0000 7fed1be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed240cf9f0 0x7fed240cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:60410/0 (socket says 172.21.15.53:60410) 2024-01-16T15:07:51.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.940+0000 7fed1be00640 1 -- 172.21.15.53:0/4274681521 learned_addr learned my addr 172.21.15.53:0/4274681521 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:51.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.941+0000 7fed22a00640 1 -- 172.21.15.53:0/4274681521 >> v2:172.21.15.146:3300/0 conn(0x7fed240cf9f0 msgr2=0x7fed240cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:51.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.941+0000 7fed22a00640 1 --2- 172.21.15.53:0/4274681521 >> v2:172.21.15.146:3300/0 conn(0x7fed240cf9f0 0x7fed240cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.941+0000 7fed22a00640 1 -- 172.21.15.53:0/4274681521 >> v1:172.21.15.146:6789/0 conn(0x7fed24161010 legacy=0x7fed241614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.941+0000 7fed22a00640 1 -- 172.21.15.53:0/4274681521 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240cf080 con 0x7fed24160920 2024-01-16T15:07:51.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.942+0000 7fed22a00640 1 --2- 172.21.15.53:0/4274681521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24160920 0x7fed240b3f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed140097a0 tx=0x7fed1402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f23192b3606a0cf4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.943+0000 7fed22000640 1 -- 172.21.15.53:0/4274681521 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14031830 con 0x7fed24160920 2024-01-16T15:07:51.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.943+0000 7fed22000640 1 -- 172.21.15.53:0/4274681521 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed14031990 con 0x7fed24160920 2024-01-16T15:07:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.943+0000 7fed22000640 1 -- 172.21.15.53:0/4274681521 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed14031d20 con 0x7fed24160920 2024-01-16T15:07:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 -- 172.21.15.53:0/4274681521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24160920 msgr2=0x7fed240b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 --2- 172.21.15.53:0/4274681521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24160920 0x7fed240b3f30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fed140097a0 tx=0x7fed1402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 -- 172.21.15.53:0/4274681521 shutdown_connections 2024-01-16T15:07:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 --2- 172.21.15.53:0/4274681521 >> v2:172.21.15.146:3300/0 conn(0x7fed240cf9f0 0x7fed240cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 --2- 172.21.15.53:0/4274681521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24160920 0x7fed240b3f30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 -- 172.21.15.53:0/4274681521 >> 172.21.15.53:0/4274681521 conn(0x7fed240d3090 msgr2=0x7fed240d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 -- 172.21.15.53:0/4274681521 shutdown_connections 2024-01-16T15:07:51.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.944+0000 7fed23e00640 1 -- 172.21.15.53:0/4274681521 wait complete. 2024-01-16T15:07:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.946+0000 7fed23e00640 1 Processor -- start 2024-01-16T15:07:51.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.946+0000 7fed23e00640 1 -- start start 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed23e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 0x7fed2415d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed23e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed24160920 0x7fed2415c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed23e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24161010 0x7fed241502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed23e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2415cc90 con 0x7fed240cf9f0 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed23e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2415dad0 con 0x7fed24160920 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed23e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2415ddb0 con 0x7fed24161010 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed22a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 0x7fed2415d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed22a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 0x7fed2415d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44730/0 (socket says 172.21.15.53:44730) 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed22a00640 1 -- 172.21.15.53:0/2653739667 learned_addr learned my addr 172.21.15.53:0/2653739667 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.947+0000 7fed1be00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed24160920 0x7fed2415c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed23400640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24161010 0x7fed241502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed22a00640 1 -- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24161010 msgr2=0x7fed241502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed22a00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24161010 0x7fed241502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed22a00640 1 -- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed24160920 msgr2=0x7fed2415c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed22a00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed24160920 0x7fed2415c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed22a00640 1 -- 172.21.15.53:0/2653739667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed14009400 con 0x7fed240cf9f0 2024-01-16T15:07:51.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed23400640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24161010 0x7fed241502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed22a00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 0x7fed2415d440 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fed140098c0 tx=0x7fed140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1403b970 con 0x7fed240cf9f0 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1403baf0 con 0x7fed240cf9f0 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1403bea0 con 0x7fed240cf9f0 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.948+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2415e090 con 0x7fed240cf9f0 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.949+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed240b79a0 con 0x7fed240cf9f0 2024-01-16T15:07:51.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.950+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed2415dc10 con 0x7fed240cf9f0 2024-01-16T15:07:51.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.951+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fed1404e450 con 0x7fed240cf9f0 2024-01-16T15:07:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.952+0000 7fed1aa00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecfc06bbc0 0x7fecfc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:51.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.952+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fed140aa590 con 0x7fed240cf9f0 2024-01-16T15:07:51.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.953+0000 7fed1be00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecfc06bbc0 0x7fecfc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:51.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.954+0000 7fed1be00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecfc06bbc0 0x7fecfc06e080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fed240ccfc0 tx=0x7fed0c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:51.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:51.957+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed2415dc10 con 0x7fed240cf9f0 2024-01-16T15:07:52.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.165+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fed2415dc10 con 0x7fed240cf9f0 2024-01-16T15:07:52.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.166+0000 7fed1aa00640 1 -- 172.21.15.53:0/2653739667 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v64) v1 ==== 80+0+124 (secure 0 0 0) 0x7fed2415dc10 con 0x7fed240cf9f0 2024-01-16T15:07:52.143 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6837/3568122877 2024-01-17T15:07:45.564678+0000 2024-01-16T15:07:52.143 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6835/3568122877 2024-01-17T15:07:45.564678+0000 2024-01-16T15:07:52.143 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:07:52.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.168+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecfc06bbc0 msgr2=0x7fecfc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.168+0000 7fed23e00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecfc06bbc0 0x7fecfc06e080 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fed240ccfc0 tx=0x7fed0c008040 comp rx=0 tx=0).stop 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.168+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 msgr2=0x7fed2415d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.168+0000 7fed23e00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 0x7fed2415d440 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fed140098c0 tx=0x7fed140046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 shutdown_connections 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecfc06bbc0 0x7fecfc06e080 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed24161010 0x7fed241502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed24160920 0x7fed2415c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 --2- 172.21.15.53:0/2653739667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed240cf9f0 0x7fed2415d440 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 >> 172.21.15.53:0/2653739667 conn(0x7fed240d3090 msgr2=0x7fed2406c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 shutdown_connections 2024-01-16T15:07:52.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.169+0000 7fed23e00640 1 -- 172.21.15.53:0/2653739667 wait complete. 2024-01-16T15:07:52.163 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:07:52.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:52.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:07:52.252 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:52.259 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44041]: fuse finished with error 0 and tester_r 0 2024-01-16T15:07:52.259 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:07:52.270 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:07:52.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:52.271 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:07:52.482 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:07:52.482 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:52.542 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:07:52.542 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:07:52.542 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:52.542 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:07:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:07:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:07:52.601 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:07:52.601 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:52.656 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:07:52.656 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:07:52.656 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:07:52.656 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:07:52.712 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:07:52.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:07:52.713 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:07:52.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.967+0000 7f9332800640 1 Processor -- start 2024-01-16T15:07:52.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.967+0000 7f9332800640 1 -- start start 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9332800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f932c13cfa0 0x7f932c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9332800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f932c13fcb0 0x7f932c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9332800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f932c0cdec0 con 0x7f932c0cd530 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9332800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c0ce1a0 con 0x7f932c13cfa0 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9332800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f932c0ce480 con 0x7f932c13fcb0 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9331400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f932c13cfa0 0x7f932c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9330a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f932c0cd530 0x7f932c13f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33922/0 (socket says 172.21.15.53:33922) 2024-01-16T15:07:52.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9331400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f932c13cfa0 0x7f932c15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44732/0 (socket says 172.21.15.53:44732) 2024-01-16T15:07:52.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.969+0000 7f9330a00640 1 -- 172.21.15.53:0/1453928316 learned_addr learned my addr 172.21.15.53:0/1453928316 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:52.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.970+0000 7f9331400640 1 -- 172.21.15.53:0/1453928316 >> v2:172.21.15.146:3300/0 conn(0x7f932c13fcb0 msgr2=0x7f932c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:52.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.970+0000 7f9331400640 1 --2- 172.21.15.53:0/1453928316 >> v2:172.21.15.146:3300/0 conn(0x7f932c13fcb0 0x7f932c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.970+0000 7f9331400640 1 -- 172.21.15.53:0/1453928316 >> v1:172.21.15.53:6789/0 conn(0x7f932c0cd530 legacy=0x7f932c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.970+0000 7f9331400640 1 -- 172.21.15.53:0/1453928316 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c0ce760 con 0x7f932c13cfa0 2024-01-16T15:07:52.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.971+0000 7f9331400640 1 --2- 172.21.15.53:0/1453928316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c15afb0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f932c0b3fd0 tx=0x7f931c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9322176579d6164e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:52.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.971+0000 7f932be00640 1 -- 172.21.15.53:0/1453928316 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c02e070 con 0x7f932c13cfa0 2024-01-16T15:07:52.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.971+0000 7f932be00640 1 -- 172.21.15.53:0/1453928316 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c0315a0 con 0x7f932c13cfa0 2024-01-16T15:07:52.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.971+0000 7f932be00640 1 -- 172.21.15.53:0/1453928316 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c031930 con 0x7f932c13cfa0 2024-01-16T15:07:52.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 -- 172.21.15.53:0/1453928316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 msgr2=0x7f932c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 --2- 172.21.15.53:0/1453928316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c15afb0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f932c0b3fd0 tx=0x7f931c02b840 comp rx=0 tx=0).stop 2024-01-16T15:07:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 -- 172.21.15.53:0/1453928316 shutdown_connections 2024-01-16T15:07:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 --2- 172.21.15.53:0/1453928316 >> v2:172.21.15.146:3300/0 conn(0x7f932c13fcb0 0x7f932c0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 --2- 172.21.15.53:0/1453928316 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c15afb0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 -- 172.21.15.53:0/1453928316 >> 172.21.15.53:0/1453928316 conn(0x7f932c06a800 msgr2=0x7f932c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 -- 172.21.15.53:0/1453928316 shutdown_connections 2024-01-16T15:07:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.972+0000 7f9332800640 1 -- 172.21.15.53:0/1453928316 wait complete. 2024-01-16T15:07:52.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.974+0000 7f9332800640 1 Processor -- start 2024-01-16T15:07:52.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.974+0000 7f9332800640 1 -- start start 2024-01-16T15:07:52.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.975+0000 7f9332800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c0cd530 0x7f932c0ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.975+0000 7f9332800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c13b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.975+0000 7f9332800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f932c13fcb0 0x7f932c156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.975+0000 7f9332800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0aae20 con 0x7f932c13cfa0 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.975+0000 7f9332800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0abcc0 con 0x7f932c0cd530 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.975+0000 7f9332800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f932c0acb10 con 0x7f932c13fcb0 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c13b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c13b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44740/0 (socket says 172.21.15.53:44740) 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 -- 172.21.15.53:0/2321832643 learned_addr learned my addr 172.21.15.53:0/2321832643 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9331400640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c0cd530 0x7f932c0ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9331e00640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f932c13fcb0 0x7f932c156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 -- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f932c13fcb0 msgr2=0x7f932c156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f932c13fcb0 0x7f932c156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 -- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c0cd530 msgr2=0x7f932c0ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:52.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c0cd530 0x7f932c0ac1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:52.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9330a00640 1 -- 172.21.15.53:0/2321832643 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c009400 con 0x7f932c13cfa0 2024-01-16T15:07:52.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.976+0000 7f9331e00640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f932c13fcb0 0x7f932c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:52.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.977+0000 7f9330a00640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c13b570 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f931400c4a0 tx=0x7f931400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:52.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.977+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314005ce0 con 0x7f932c13cfa0 2024-01-16T15:07:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.977+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9314005e40 con 0x7f932c13cfa0 2024-01-16T15:07:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.977+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314007270 con 0x7f932c13cfa0 2024-01-16T15:07:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.977+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c0acdf0 con 0x7f932c13cfa0 2024-01-16T15:07:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.977+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c0ac720 con 0x7f932c13cfa0 2024-01-16T15:07:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.978+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c0ad210 con 0x7f932c13cfa0 2024-01-16T15:07:52.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.979+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f9314007490 con 0x7f932c13cfa0 2024-01-16T15:07:52.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.980+0000 7f9329600640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f930406bbc0 0x7f930406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:52.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.980+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f93140878f0 con 0x7f932c13cfa0 2024-01-16T15:07:52.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.981+0000 7f9331400640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f930406bbc0 0x7f930406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:52.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.982+0000 7f9331400640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f930406bbc0 0x7f930406e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f931c036000 tx=0x7f931c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:52.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:52.984+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f932c0ad210 con 0x7f932c13cfa0 2024-01-16T15:07:53.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.221+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f932c0ad210 con 0x7f932c13cfa0 2024-01-16T15:07:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.222+0000 7f9329600640 1 -- 172.21.15.53:0/2321832643 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f932c0ad210 con 0x7f932c13cfa0 2024-01-16T15:07:53.200 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:53.200 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/2632791628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2632791628},{"type":"v1","addr":"172.21.15.53:6833","nonce":2632791628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3341240307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3341240307},{"type":"v1","addr":"172.21.15.146:6837","nonce":3341240307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5680,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3474334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3474334502},{"type":"v1","addr":"172.21.15.53:6835","nonce":3474334502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5683,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2549681777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2549681777},{"type":"v1","addr":"172.21.15.146:6836","nonce":2549681777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-01-16T15:07:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 38 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f930406bbc0 msgr2=0x7f930406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f930406bbc0 0x7f930406e080 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f931c036000 tx=0x7f931c03d040 comp rx=0 tx=0).stop 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 msgr2=0x7f932c13b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c13b570 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f931400c4a0 tx=0x7f931400c960 comp rx=0 tx=0).stop 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 shutdown_connections 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f930406bbc0 0x7f930406e080 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f932c13fcb0 0x7f932c156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f932c13cfa0 0x7f932c13b570 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 --2- 172.21.15.53:0/2321832643 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f932c0cd530 0x7f932c0ac1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 >> 172.21.15.53:0/2321832643 conn(0x7f932c06a800 msgr2=0x7f932c0b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 shutdown_connections 2024-01-16T15:07:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.225+0000 7f9332800640 1 -- 172.21.15.53:0/2321832643 wait complete. 2024-01-16T15:07:53.222 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:07:53.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.473+0000 7f7ec2e00640 1 Processor -- start 2024-01-16T15:07:53.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.473+0000 7f7ec2e00640 1 -- start start 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec2e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7ebc144c60 0x7f7ebc0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec2e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7ebc0cf780 0x7f7ebc06e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec2e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ebc071a20 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec2e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ebc0ccfd0 con 0x7f7ebc0cf780 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec2e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ebc0cd2b0 con 0x7f7ebc144c60 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec0e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7ebc06a3e0 0x7f7ebc071eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33930/0 (socket says 172.21.15.53:33930) 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7ec0e00640 1 -- 172.21.15.53:0/4015333288 learned_addr learned my addr 172.21.15.53:0/4015333288 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.475+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/4015333288 >> v2:172.21.15.53:3300/0 conn(0x7f7ebc0cf780 0x7f7ebc06e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.476+0000 7f7ebbe00640 1 --2- 172.21.15.53:0/4015333288 >> v2:172.21.15.146:3300/0 conn(0x7f7ebc144c60 0x7f7ebc0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.476+0000 7f7eb3e00640 1 -- 172.21.15.53:0/4015333288 >> v2:172.21.15.146:3300/0 conn(0x7f7ebc144c60 msgr2=0x7f7ebc0cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:53.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.476+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/4015333288 >> v2:172.21.15.146:3300/0 conn(0x7f7ebc144c60 0x7f7ebc0cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.476+0000 7f7eb3e00640 1 -- 172.21.15.53:0/4015333288 >> v1:172.21.15.53:6789/0 conn(0x7f7ebc06a3e0 legacy=0x7f7ebc071eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.476+0000 7f7eb3e00640 1 -- 172.21.15.53:0/4015333288 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ebc0713d0 con 0x7f7ebc0cf780 2024-01-16T15:07:53.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.476+0000 7f7ebb400640 1 -- 172.21.15.53:0/4015333288 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3108255745 0 0) 0x7f7ebc071a20 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.477+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/4015333288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc06e7e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc06b4c0 tx=0x7f7ea402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=420a379e516b381 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ebb400640 1 -- 172.21.15.53:0/4015333288 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4031830 con 0x7f7ebc0cf780 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ebb400640 1 -- 172.21.15.53:0/4015333288 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea4031990 con 0x7f7ebc0cf780 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ebb400640 1 -- 172.21.15.53:0/4015333288 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ea4031d20 con 0x7f7ebc0cf780 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 -- 172.21.15.53:0/4015333288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0cf780 msgr2=0x7f7ebc06e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/4015333288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc06e7e0 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc06b4c0 tx=0x7f7ea402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 -- 172.21.15.53:0/4015333288 shutdown_connections 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/4015333288 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc06e7e0 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/4015333288 >> v2:172.21.15.146:3300/0 conn(0x7f7ebc144c60 0x7f7ebc0cc940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 -- 172.21.15.53:0/4015333288 >> 172.21.15.53:0/4015333288 conn(0x7f7ebc0cfef0 msgr2=0x7f7ebc0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 -- 172.21.15.53:0/4015333288 shutdown_connections 2024-01-16T15:07:53.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.478+0000 7f7ec2e00640 1 -- 172.21.15.53:0/4015333288 wait complete. 2024-01-16T15:07:53.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.480+0000 7f7ec2e00640 1 Processor -- start 2024-01-16T15:07:53.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.480+0000 7f7ec2e00640 1 -- start start 2024-01-16T15:07:53.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ec2e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 0x7f7ebc13da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ec2e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc13cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ec2e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ebc144c60 0x7f7ebc1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ec2e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc15a0c0 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ec2e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc159180 con 0x7f7ebc0cf780 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ec2e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ebc159440 con 0x7f7ebc144c60 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.481+0000 7f7ebbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 0x7f7ebc13da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 0x7f7ebc13da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44754/0 (socket says 172.21.15.53:44754) 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 -- 172.21.15.53:0/1524706936 learned_addr learned my addr 172.21.15.53:0/1524706936 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc13cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ec0e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ebc144c60 0x7f7ebc1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 -- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ebc144c60 msgr2=0x7f7ebc1551f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ebc144c60 0x7f7ebc1551f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 -- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ebc0cf780 msgr2=0x7f7ebc13cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc13cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 -- 172.21.15.53:0/1524706936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ea4009400 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc13cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:53.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ec0e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ebc144c60 0x7f7ebc1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:53.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.482+0000 7f7ebbe00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 0x7f7ebc13da20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00d5f0 tx=0x7f7eac00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:53.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.483+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac0125f0 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.483+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac012750 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.483+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7eac012b00 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.483+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ebc159720 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.483+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ebc159ce0 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.484+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ebc073c30 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.485+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f7eac012c60 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.486+0000 7f7eb8c00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e9406bbc0 0x7f7e9406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.486+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7eac088b70 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.486+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e9406bbc0 0x7f7e9406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.488+0000 7f7eb3e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e9406bbc0 0x7f7e9406e080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4036000 tx=0x7f7ea4008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:53.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.491+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ebc073c30 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.687+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f7ebc0c9c70 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.693+0000 7f7eb8c00640 1 -- 172.21.15.53:0/1524706936 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v23) v1 ==== 120+0+0 (secure 0 0 0) 0x7f7eac055240 con 0x7f7ebc06a3e0 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e9406bbc0 msgr2=0x7f7e9406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e9406bbc0 0x7f7e9406e080 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f7ea4036000 tx=0x7f7ea4008030 comp rx=0 tx=0).stop 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 msgr2=0x7f7ebc13da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 0x7f7ebc13da20 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7eac00d5f0 tx=0x7f7eac00dab0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 shutdown_connections 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7e9406bbc0 0x7f7e9406e080 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ebc144c60 0x7f7ebc1551f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ebc0cf780 0x7f7ebc13cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 --2- 172.21.15.53:0/1524706936 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ebc06a3e0 0x7f7ebc13da20 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 >> 172.21.15.53:0/1524706936 conn(0x7f7ebc0cfef0 msgr2=0x7f7ebc15d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:53.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 shutdown_connections 2024-01-16T15:07:53.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.696+0000 7f7ec2e00640 1 -- 172.21.15.53:0/1524706936 wait complete. 2024-01-16T15:07:53.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:07:53.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.943+0000 7f2d9d000640 1 Processor -- start 2024-01-16T15:07:53.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.943+0000 7f2d9d000640 1 -- start start 2024-01-16T15:07:53.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.945+0000 7f2d9d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2d981505b0 0x7f2d980b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.945+0000 7f2d9d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b1e30 con 0x7f2d9815a690 2024-01-16T15:07:53.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.945+0000 7f2d9d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9815ab20 con 0x7f2d9806d6d0 2024-01-16T15:07:53.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.945+0000 7f2d9d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d980b20a0 con 0x7f2d981505b0 2024-01-16T15:07:53.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.945+0000 7f2d96a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2d9815a690 0x7f2d9806cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33934/0 (socket says 172.21.15.53:33934) 2024-01-16T15:07:53.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.945+0000 7f2d96a00640 1 -- 172.21.15.53:0/398668895 learned_addr learned my addr 172.21.15.53:0/398668895 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:53.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d97400640 1 --2- 172.21.15.53:0/398668895 >> v2:172.21.15.146:3300/0 conn(0x7f2d981505b0 0x7f2d980b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255395978 0 0) 0x7f2d980b1e30 con 0x7f2d9815a690 2024-01-16T15:07:53.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d78003520 con 0x7f2d9815a690 2024-01-16T15:07:53.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2519319293 0 0) 0x7f2d78003520 con 0x7f2d9815a690 2024-01-16T15:07:53.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 >> v2:172.21.15.146:3300/0 conn(0x7f2d981505b0 msgr2=0x7f2d980b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:53.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 --2- 172.21.15.53:0/398668895 >> v2:172.21.15.146:3300/0 conn(0x7f2d981505b0 0x7f2d980b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 >> v1:172.21.15.146:6789/0 conn(0x7f2d9806d6d0 legacy=0x7f2d980ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980b2380 con 0x7f2d9815a690 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2d80001670 con 0x7f2d9815a690 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 >> v1:172.21.15.53:6789/0 conn(0x7f2d9815a690 legacy=0x7f2d9806cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d78003970 0x7f2d78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d780047c0 0x7f2d78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d78005180 0x7f2d78005950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d980b20a0 con 0x7f2d78003970 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78005ff0 con 0x7f2d78005180 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.946+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d780061b0 con 0x7f2d780047c0 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97400640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d78003970 0x7f2d78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97e00640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d78005180 0x7f2d78005950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97400640 1 -- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d780047c0 msgr2=0x7f2d78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97400640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d780047c0 0x7f2d78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97400640 1 -- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d78005180 msgr2=0x7f2d78005950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97400640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d78005180 0x7f2d78005950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97400640 1 -- 172.21.15.53:0/398668895 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d78003520 con 0x7f2d78003970 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97e00640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d78005180 0x7f2d78005950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:53.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.947+0000 7f2d97e00640 1 -- 172.21.15.53:0/398668895 reap_dead start 2024-01-16T15:07:53.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.948+0000 7f2d97400640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d78003970 0x7f2d78003d90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009ac0 tx=0x7f2d780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3f8da84409b0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8802c6b0 con 0x7f2d78003970 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8802c810 con 0x7f2d78003970 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d96000640 1 -- 172.21.15.53:0/398668895 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8802cba0 con 0x7f2d78003970 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 -- 172.21.15.53:0/398668895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d78003970 msgr2=0x7f2d78003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d78003970 0x7f2d78003d90 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f2d88009ac0 tx=0x7f2d780057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 -- 172.21.15.53:0/398668895 shutdown_connections 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 --2- 172.21.15.53:0/398668895 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d78003970 0x7f2d78003d90 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 -- 172.21.15.53:0/398668895 >> 172.21.15.53:0/398668895 conn(0x7f2d9806a7d0 msgr2=0x7f2d9806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 -- 172.21.15.53:0/398668895 shutdown_connections 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.949+0000 7f2d9d000640 1 -- 172.21.15.53:0/398668895 wait complete. 2024-01-16T15:07:53.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.951+0000 7f2d9d000640 1 Processor -- start 2024-01-16T15:07:53.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.951+0000 7f2d9d000640 1 -- start start 2024-01-16T15:07:53.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.952+0000 7f2d9d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d78003970 0x7f2d981588e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.952+0000 7f2d9d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 0x7f2d980a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.952+0000 7f2d9d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d98142420 0x7f2d981428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.952+0000 7f2d9d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9815ccd0 con 0x7f2d9815c510 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.952+0000 7f2d9d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d98141f90 con 0x7f2d98142420 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.952+0000 7f2d9d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d9815c9f0 con 0x7f2d78003970 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 0x7f2d980a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 0x7f2d980a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44780/0 (socket says 172.21.15.53:44780) 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 -- 172.21.15.53:0/2887081800 learned_addr learned my addr 172.21.15.53:0/2887081800 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d97400640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d78003970 0x7f2d981588e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d97e00640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d98142420 0x7f2d981428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 -- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d78003970 msgr2=0x7f2d981588e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d78003970 0x7f2d981588e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 -- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d98142420 msgr2=0x7f2d981428a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d98142420 0x7f2d981428a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 -- 172.21.15.53:0/2887081800 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d98141940 con 0x7f2d9815c510 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.953+0000 7f2d96a00640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 0x7f2d980a84c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c7f0 tx=0x7f2d8000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.954+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d800074e0 con 0x7f2d9815c510 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.954+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d80007640 con 0x7f2d9815c510 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.954+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d80004a00 con 0x7f2d9815c510 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.954+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d98141c20 con 0x7f2d9815c510 2024-01-16T15:07:53.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.954+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d98159000 con 0x7f2d9815c510 2024-01-16T15:07:53.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.955+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d98067450 con 0x7f2d9815c510 2024-01-16T15:07:53.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.956+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f2d80004b60 con 0x7f2d9815c510 2024-01-16T15:07:53.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.957+0000 7f2d8f400640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2d7006bc40 0x7f2d7006e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:53.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.957+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2d80087c10 con 0x7f2d9815c510 2024-01-16T15:07:53.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.958+0000 7f2d97400640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2d7006bc40 0x7f2d7006e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:53.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.959+0000 7f2d97400640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2d7006bc40 0x7f2d7006e100 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2d88036000 tx=0x7f2d88030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:53.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:53.962+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2d80054440 con 0x7f2d9815c510 2024-01-16T15:07:54.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.154+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f2d98141de0 con 0x7f2d9815c510 2024-01-16T15:07:54.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.159+0000 7f2d8f400640 1 -- 172.21.15.53:0/2887081800 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v24) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2d80018030 con 0x7f2d9815c510 2024-01-16T15:07:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2d7006bc40 msgr2=0x7f2d7006e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2d7006bc40 0x7f2d7006e100 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f2d88036000 tx=0x7f2d88030000 comp rx=0 tx=0).stop 2024-01-16T15:07:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 msgr2=0x7f2d980a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 0x7f2d980a84c0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f2d8000c7f0 tx=0x7f2d8000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 shutdown_connections 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2d7006bc40 0x7f2d7006e100 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2d98142420 0x7f2d981428a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2d9815c510 0x7f2d980a84c0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 --2- 172.21.15.53:0/2887081800 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2d78003970 0x7f2d981588e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.162+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 >> 172.21.15.53:0/2887081800 conn(0x7f2d9806a7d0 msgr2=0x7f2d9815b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:54.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.163+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 shutdown_connections 2024-01-16T15:07:54.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.163+0000 7f2d9d000640 1 -- 172.21.15.53:0/2887081800 wait complete. 2024-01-16T15:07:54.160 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:07:54.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.408+0000 7fe7f7e00640 1 Processor -- start 2024-01-16T15:07:54.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.408+0000 7fe7f7e00640 1 -- start start 2024-01-16T15:07:54.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.410+0000 7fe7f7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7f80adf20 0x7fe7f8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.410+0000 7fe7f7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe7f80ccb50 0x7fe7f8143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.410+0000 7fe7f7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7f815f4e0 con 0x7fe7f815bf10 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.410+0000 7fe7f7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f815f970 con 0x7fe7f80adf20 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.410+0000 7fe7f7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7f8071130 con 0x7fe7f80ccb50 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7f80adf20 0x7fe7f8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7f80adf20 0x7fe7f8070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44782/0 (socket says 172.21.15.53:44782) 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 -- 172.21.15.53:0/3432866348 learned_addr learned my addr 172.21.15.53:0/3432866348 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f7400640 1 --2- 172.21.15.53:0/3432866348 >> v2:172.21.15.146:3300/0 conn(0x7fe7f80ccb50 0x7fe7f8143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 -- 172.21.15.53:0/3432866348 >> v2:172.21.15.146:3300/0 conn(0x7fe7f80ccb50 msgr2=0x7fe7f8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 --2- 172.21.15.53:0/3432866348 >> v2:172.21.15.146:3300/0 conn(0x7fe7f80ccb50 0x7fe7f8143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 -- 172.21.15.53:0/3432866348 >> v1:172.21.15.146:6789/0 conn(0x7fe7f815bf10 legacy=0x7fe7f80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.411+0000 7fe7f6a00640 1 -- 172.21.15.53:0/3432866348 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f815c390 con 0x7fe7f80adf20 2024-01-16T15:07:54.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.412+0000 7fe7f6a00640 1 --2- 172.21.15.53:0/3432866348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f80adf20 0x7fe7f8070a30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe7f806ff20 tx=0x7fe7e802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83c0f546d34c4c91 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.413+0000 7fe7f5600640 1 -- 172.21.15.53:0/3432866348 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e80317e0 con 0x7fe7f80adf20 2024-01-16T15:07:54.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.413+0000 7fe7f5600640 1 -- 172.21.15.53:0/3432866348 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e8031940 con 0x7fe7f80adf20 2024-01-16T15:07:54.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.413+0000 7fe7f5600640 1 -- 172.21.15.53:0/3432866348 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e8031cd0 con 0x7fe7f80adf20 2024-01-16T15:07:54.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.413+0000 7fe7f7e00640 1 -- 172.21.15.53:0/3432866348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f80adf20 msgr2=0x7fe7f8070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.413+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/3432866348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f80adf20 0x7fe7f8070a30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe7f806ff20 tx=0x7fe7e802b880 comp rx=0 tx=0).stop 2024-01-16T15:07:54.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.414+0000 7fe7f7e00640 1 -- 172.21.15.53:0/3432866348 shutdown_connections 2024-01-16T15:07:54.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.414+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/3432866348 >> v2:172.21.15.146:3300/0 conn(0x7fe7f80ccb50 0x7fe7f8143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.414+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/3432866348 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f80adf20 0x7fe7f8070a30 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.414+0000 7fe7f7e00640 1 -- 172.21.15.53:0/3432866348 >> 172.21.15.53:0/3432866348 conn(0x7fe7f806a7d0 msgr2=0x7fe7f806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:54.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.414+0000 7fe7f7e00640 1 -- 172.21.15.53:0/3432866348 shutdown_connections 2024-01-16T15:07:54.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.414+0000 7fe7f7e00640 1 -- 172.21.15.53:0/3432866348 wait complete. 2024-01-16T15:07:54.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.416+0000 7fe7f7e00640 1 Processor -- start 2024-01-16T15:07:54.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.416+0000 7fe7f7e00640 1 -- start start 2024-01-16T15:07:54.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.417+0000 7fe7f7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7f80adf20 0x7fe7f8140fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.417+0000 7fe7f7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7f80ccb50 0x7fe7f813d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.417+0000 7fe7f7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 0x7fe7f813bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.417+0000 7fe7f7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f813ecf0 con 0x7fe7f815bf10 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.417+0000 7fe7f7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f813de40 con 0x7fe7f80ccb50 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.417+0000 7fe7f7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7f813e0e0 con 0x7fe7f80adf20 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 0x7fe7f813bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 0x7fe7f813bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44784/0 (socket says 172.21.15.53:44784) 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 -- 172.21.15.53:0/225242106 learned_addr learned my addr 172.21.15.53:0/225242106 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f6a00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7f80adf20 0x7fe7f8140fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f6000640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7f80ccb50 0x7fe7f813d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 -- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7f80adf20 msgr2=0x7fe7f8140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7f80adf20 0x7fe7f8140fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 -- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7f80ccb50 msgr2=0x7fe7f813d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7f80ccb50 0x7fe7f813d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.418+0000 7fe7f7400640 1 -- 172.21.15.53:0/225242106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e80094d0 con 0x7fe7f815bf10 2024-01-16T15:07:54.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.419+0000 7fe7f7400640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 0x7fe7f813bd00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0007e40 tx=0x7fe7f000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.419+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f0014620 con 0x7fe7f815bf10 2024-01-16T15:07:54.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.419+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f000ee90 con 0x7fe7f815bf10 2024-01-16T15:07:54.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.419+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f00149e0 con 0x7fe7f815bf10 2024-01-16T15:07:54.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.419+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7f813e3c0 con 0x7fe7f815bf10 2024-01-16T15:07:54.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.419+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7f813e980 con 0x7fe7f815bf10 2024-01-16T15:07:54.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.420+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7f813eea0 con 0x7fe7f815bf10 2024-01-16T15:07:54.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.422+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe7f0014b40 con 0x7fe7f815bf10 2024-01-16T15:07:54.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.423+0000 7fe7eea00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7d006bbc0 0x7fe7d006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.423+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe7f0088e60 con 0x7fe7f815bf10 2024-01-16T15:07:54.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.423+0000 7fe7f6a00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7d006bbc0 0x7fe7d006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.424+0000 7fe7f6a00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7d006bbc0 0x7fe7d006e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8036000 tx=0x7fe7e803d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.428+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7f813eea0 con 0x7fe7f815bf10 2024-01-16T15:07:54.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.624+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe7f80ce1a0 con 0x7fe7f815bf10 2024-01-16T15:07:54.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.630+0000 7fe7eea00640 1 -- 172.21.15.53:0/225242106 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v25) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe7f0055720 con 0x7fe7f815bf10 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7d006bbc0 msgr2=0x7fe7d006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7d006bbc0 0x7fe7d006e080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe7e8036000 tx=0x7fe7e803d040 comp rx=0 tx=0).stop 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 msgr2=0x7fe7f813bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 0x7fe7f813bd00 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe7f0007e40 tx=0x7fe7f000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 shutdown_connections 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe7d006bbc0 0x7fe7d006e080 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7f815bf10 0x7fe7f813bd00 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7f80ccb50 0x7fe7f813d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 --2- 172.21.15.53:0/225242106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7f80adf20 0x7fe7f8140fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 >> 172.21.15.53:0/225242106 conn(0x7fe7f806a7d0 msgr2=0x7fe7f806abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:54.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.632+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 shutdown_connections 2024-01-16T15:07:54.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.633+0000 7fe7f7e00640 1 -- 172.21.15.53:0/225242106 wait complete. 2024-01-16T15:07:54.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones' 2024-01-16T15:07:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.871+0000 7fd51e000640 1 Processor -- start 2024-01-16T15:07:54.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.872+0000 7fd51e000640 1 -- start start 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.873+0000 7fd51e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5180b3ae0 0x7fd51815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.873+0000 7fd51e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5180cf290 0x7fd5180cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.873+0000 7fd51e000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd518070c40 con 0x7fd5180ae5a0 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.873+0000 7fd51e000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd518071450 con 0x7fd5180cf290 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.873+0000 7fd51e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5180cfc60 con 0x7fd5180b3ae0 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5180cf290 0x7fd5180cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5180cf290 0x7fd5180cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44794/0 (socket says 172.21.15.53:44794) 2024-01-16T15:07:54.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 -- 172.21.15.53:0/4224775751 learned_addr learned my addr 172.21.15.53:0/4224775751 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd516a00640 1 --2- 172.21.15.53:0/4224775751 >> v2:172.21.15.146:3300/0 conn(0x7fd5180b3ae0 0x7fd51815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 -- 172.21.15.53:0/4224775751 >> v2:172.21.15.146:3300/0 conn(0x7fd5180b3ae0 msgr2=0x7fd51815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 --2- 172.21.15.53:0/4224775751 >> v2:172.21.15.146:3300/0 conn(0x7fd5180b3ae0 0x7fd51815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 -- 172.21.15.53:0/4224775751 >> v1:172.21.15.146:6789/0 conn(0x7fd5180ae5a0 legacy=0x7fd51806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:54.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd517e00640 1 -- 172.21.15.53:0/4224775751 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd51806c470 con 0x7fd5180cf290 2024-01-16T15:07:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.874+0000 7fd516a00640 1 --2- 172.21.15.53:0/4224775751 >> v2:172.21.15.146:3300/0 conn(0x7fd5180b3ae0 0x7fd51815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.876+0000 7fd517e00640 1 --2- 172.21.15.53:0/4224775751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd5180cf720 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd5180cfea0 tx=0x7fd51002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71eb48c8546ceba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.876+0000 7fd516000640 1 -- 172.21.15.53:0/4224775751 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51002cb10 con 0x7fd5180cf290 2024-01-16T15:07:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.876+0000 7fd516000640 1 -- 172.21.15.53:0/4224775751 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51002cc70 con 0x7fd5180cf290 2024-01-16T15:07:54.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.876+0000 7fd516000640 1 -- 172.21.15.53:0/4224775751 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5100343f0 con 0x7fd5180cf290 2024-01-16T15:07:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.876+0000 7fd51e000640 1 -- 172.21.15.53:0/4224775751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 msgr2=0x7fd5180cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.876+0000 7fd51e000640 1 --2- 172.21.15.53:0/4224775751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd5180cf720 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fd5180cfea0 tx=0x7fd51002bf70 comp rx=0 tx=0).stop 2024-01-16T15:07:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.877+0000 7fd51e000640 1 -- 172.21.15.53:0/4224775751 shutdown_connections 2024-01-16T15:07:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.877+0000 7fd51e000640 1 --2- 172.21.15.53:0/4224775751 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd5180cf720 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.877+0000 7fd51e000640 1 --2- 172.21.15.53:0/4224775751 >> v2:172.21.15.146:3300/0 conn(0x7fd5180b3ae0 0x7fd51815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.877+0000 7fd51e000640 1 -- 172.21.15.53:0/4224775751 >> 172.21.15.53:0/4224775751 conn(0x7fd51806a800 msgr2=0x7fd51806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.877+0000 7fd51e000640 1 -- 172.21.15.53:0/4224775751 shutdown_connections 2024-01-16T15:07:54.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.877+0000 7fd51e000640 1 -- 172.21.15.53:0/4224775751 wait complete. 2024-01-16T15:07:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.879+0000 7fd51e000640 1 Processor -- start 2024-01-16T15:07:54.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.879+0000 7fd51e000640 1 -- start start 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd51e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5180ae5a0 0x7fd5181539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd51e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5180b3ae0 0x7fd51814b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd51e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd51815d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd51e000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd518155de0 con 0x7fd5180cf290 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd51e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814b860 con 0x7fd5180ae5a0 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd51e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd51814f300 con 0x7fd5180b3ae0 2024-01-16T15:07:54.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd51815d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd51815d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44800/0 (socket says 172.21.15.53:44800) 2024-01-16T15:07:54.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 -- 172.21.15.53:0/2629507677 learned_addr learned my addr 172.21.15.53:0/2629507677 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517400640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5180ae5a0 0x7fd5181539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 -- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5180b3ae0 msgr2=0x7fd51814b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5180b3ae0 0x7fd51814b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 -- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5180ae5a0 msgr2=0x7fd5181539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5180ae5a0 0x7fd5181539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.880+0000 7fd517e00640 1 -- 172.21.15.53:0/2629507677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd510009b30 con 0x7fd5180cf290 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.881+0000 7fd517e00640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd51815d820 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd510009f40 tx=0x7fd510004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.881+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51003e450 con 0x7fd5180cf290 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.881+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd51003e5d0 con 0x7fd5180cf290 2024-01-16T15:07:54.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.881+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd51003e980 con 0x7fd5180cf290 2024-01-16T15:07:54.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.881+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd518160880 con 0x7fd5180cf290 2024-01-16T15:07:54.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.881+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5181562a0 con 0x7fd5180cf290 2024-01-16T15:07:54.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.882+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5180675d0 con 0x7fd5180cf290 2024-01-16T15:07:54.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.883+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd51003eae0 con 0x7fd5180cf290 2024-01-16T15:07:54.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.884+0000 7fd50f400640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4f006bbc0 0x7fd4f006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:54.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.884+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd5100ab660 con 0x7fd5180cf290 2024-01-16T15:07:54.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.885+0000 7fd517400640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4f006bbc0 0x7fd4f006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:54.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.886+0000 7fd517400640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4f006bbc0 0x7fd4f006e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd508003d90 tx=0x7fd508009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:54.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:54.889+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd510077f20 con 0x7fd5180cf290 2024-01-16T15:07:55.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.151+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]} v 0) v1 -- 0x7fd51806b3f0 con 0x7fd5180cf290 2024-01-16T15:07:55.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.199+0000 7fd50f400640 1 -- 172.21.15.53:0/2629507677 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_bars_for_7_clones"]}]=0 v374) v1 ==== 154+0+0 (secure 0 0 0) 0x7fd510033e80 con 0x7fd5180cf290 2024-01-16T15:07:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4f006bbc0 msgr2=0x7fd4f006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4f006bbc0 0x7fd4f006e080 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fd508003d90 tx=0x7fd508009040 comp rx=0 tx=0).stop 2024-01-16T15:07:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 msgr2=0x7fd51815d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd51815d820 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fd510009f40 tx=0x7fd510004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 shutdown_connections 2024-01-16T15:07:55.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd4f006bbc0 0x7fd4f006e080 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5180cf290 0x7fd51815d820 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5180b3ae0 0x7fd51814b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 --2- 172.21.15.53:0/2629507677 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5180ae5a0 0x7fd5181539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 >> 172.21.15.53:0/2629507677 conn(0x7fd51806a800 msgr2=0x7fd51815aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:55.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 shutdown_connections 2024-01-16T15:07:55.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.203+0000 7fd51e000640 1 -- 172.21.15.53:0/2629507677 wait complete. 2024-01-16T15:07:55.199 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:07:55.199 INFO:tasks.cephfs_test_runner:Starting test: test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) X = number of clones issued. 2024-01-16T15:07:55.199 INFO:tasks.cephfs_test_runner:test_progress_drops_when_new_jobs_are_added (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:07:55.200 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:55.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.442+0000 7fb70a400640 1 Processor -- start 2024-01-16T15:07:55.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.443+0000 7fb70a400640 1 -- start start 2024-01-16T15:07:55.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.444+0000 7fb70a400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb704143c40 0x7fb704071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.444+0000 7fb70a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7040cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.444+0000 7fb70a400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb704070e50 con 0x7fb7040cca40 2024-01-16T15:07:55.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.444+0000 7fb70a400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7040723c0 con 0x7fb704143c40 2024-01-16T15:07:55.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.444+0000 7fb70a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040cce70 con 0x7fb7040717d0 2024-01-16T15:07:55.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7040cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb709000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb704143c40 0x7fb704071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7040cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:38142/0 (socket says 172.21.15.53:38142) 2024-01-16T15:07:55.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 -- 172.21.15.53:0/3085229218 learned_addr learned my addr 172.21.15.53:0/3085229218 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:55.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 -- 172.21.15.53:0/3085229218 >> v2:172.21.15.146:3300/0 conn(0x7fb704143c40 msgr2=0x7fb704071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:55.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 --2- 172.21.15.53:0/3085229218 >> v2:172.21.15.146:3300/0 conn(0x7fb704143c40 0x7fb704071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 -- 172.21.15.53:0/3085229218 >> v1:172.21.15.146:6789/0 conn(0x7fb7040cca40 legacy=0x7fb70415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.445+0000 7fb6fbe00640 1 -- 172.21.15.53:0/3085229218 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70406e170 con 0x7fb7040717d0 2024-01-16T15:07:55.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.447+0000 7fb6fbe00640 1 --2- 172.21.15.53:0/3085229218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7040cc500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009870 tx=0x7fb6ec02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8431276b5d957a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:55.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.447+0000 7fb703e00640 1 -- 172.21.15.53:0/3085229218 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec031640 con 0x7fb7040717d0 2024-01-16T15:07:55.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.447+0000 7fb703e00640 1 -- 172.21.15.53:0/3085229218 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6ec0317a0 con 0x7fb7040717d0 2024-01-16T15:07:55.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.447+0000 7fb703e00640 1 -- 172.21.15.53:0/3085229218 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec031b30 con 0x7fb7040717d0 2024-01-16T15:07:55.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 -- 172.21.15.53:0/3085229218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 msgr2=0x7fb7040cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 --2- 172.21.15.53:0/3085229218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7040cc500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec009870 tx=0x7fb6ec02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 -- 172.21.15.53:0/3085229218 shutdown_connections 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 --2- 172.21.15.53:0/3085229218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7040cc500 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 --2- 172.21.15.53:0/3085229218 >> v2:172.21.15.146:3300/0 conn(0x7fb704143c40 0x7fb704071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 -- 172.21.15.53:0/3085229218 >> 172.21.15.53:0/3085229218 conn(0x7fb70406a800 msgr2=0x7fb70406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 -- 172.21.15.53:0/3085229218 shutdown_connections 2024-01-16T15:07:55.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.448+0000 7fb70a400640 1 -- 172.21.15.53:0/3085229218 wait complete. 2024-01-16T15:07:55.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.450+0000 7fb70a400640 1 Processor -- start 2024-01-16T15:07:55.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.450+0000 7fb70a400640 1 -- start start 2024-01-16T15:07:55.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.451+0000 7fb70a400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7041f6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.451+0000 7fb70a400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7040cca40 0x7fb7041f3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.451+0000 7fb70a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 0x7fb70413e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.451+0000 7fb70a400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041f6bc0 con 0x7fb704143c40 2024-01-16T15:07:55.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.451+0000 7fb70a400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041f6ea0 con 0x7fb7040cca40 2024-01-16T15:07:55.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.451+0000 7fb70a400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041f7180 con 0x7fb7040717d0 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 0x7fb70413e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 0x7fb70413e1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44806/0 (socket says 172.21.15.53:44806) 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 -- 172.21.15.53:0/97934803 learned_addr learned my addr 172.21.15.53:0/97934803 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709000640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7041f6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb6fbe00640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7040cca40 0x7fb7041f3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 -- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 msgr2=0x7fb7041f6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7041f6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 -- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7040cca40 msgr2=0x7fb7041f3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.452+0000 7fb709a00640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7040cca40 0x7fb7041f3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb709a00640 1 -- 172.21.15.53:0/97934803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6ec0094d0 con 0x7fb704143c40 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb709a00640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 0x7fb70413e1f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00c7a0 tx=0x7fb6fc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc003b70 con 0x7fb704143c40 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6fc003cf0 con 0x7fb704143c40 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc003ea0 con 0x7fb704143c40 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7041f2e90 con 0x7fb704143c40 2024-01-16T15:07:55.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.453+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7041f68b0 con 0x7fb704143c40 2024-01-16T15:07:55.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.454+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb704067450 con 0x7fb704143c40 2024-01-16T15:07:55.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.456+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb6fc00aa90 con 0x7fb704143c40 2024-01-16T15:07:55.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.457+0000 7fb701600640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6dc06bbc0 0x7fb6dc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.458+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb6fc088d40 con 0x7fb704143c40 2024-01-16T15:07:55.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.458+0000 7fb709000640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6dc06bbc0 0x7fb6dc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.459+0000 7fb709000640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6dc06bbc0 0x7fb6dc06e080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb70413b5d0 tx=0x7fb6f4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:55.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.461+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6fc055600 con 0x7fb704143c40 2024-01-16T15:07:55.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.658+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb704140f00 con 0x7fb704143c40 2024-01-16T15:07:55.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.661+0000 7fb701600640 1 -- 172.21.15.53:0/97934803 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+12995 (secure 0 0 0) 0x7fb6fc058c20 con 0x7fb704143c40 2024-01-16T15:07:55.638 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:55.639 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":64,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:07:48.577053+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6837/3568122877":"2024-01-17T15:07:45.564678+0000","172.21.15.146:6835/3568122877":"2024-01-17T15:07:45.564678+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:07:55.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6dc06bbc0 msgr2=0x7fb6dc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6dc06bbc0 0x7fb6dc06e080 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fb70413b5d0 tx=0x7fb6f4009190 comp rx=0 tx=0).stop 2024-01-16T15:07:55.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 msgr2=0x7fb70413e1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 0x7fb70413e1f0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00c7a0 tx=0x7fb6fc00cc60 comp rx=0 tx=0).stop 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 shutdown_connections 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6dc06bbc0 0x7fb6dc06e080 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb704143c40 0x7fb70413e1f0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb7040cca40 0x7fb7041f3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 --2- 172.21.15.53:0/97934803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb7040717d0 0x7fb7041f6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 >> 172.21.15.53:0/97934803 conn(0x7fb70406a800 msgr2=0x7fb7040aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 shutdown_connections 2024-01-16T15:07:55.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.664+0000 7fb70a400640 1 -- 172.21.15.53:0/97934803 wait complete. 2024-01-16T15:07:55.661 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:07:55.661 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:07:55.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.903+0000 7fccb5e00640 1 Processor -- start 2024-01-16T15:07:55.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.904+0000 7fccb5e00640 1 -- start start 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.905+0000 7fccb5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015ec60 0x7fccb00cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.905+0000 7fccb5e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fccb015f040 0x7fccb00cb950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccb5e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb00cc1a0 con 0x7fccb015ab00 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccb5e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb00cf2f0 con 0x7fccb015f040 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccb5e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00cc480 con 0x7fccb015ec60 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccafe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fccb015f040 0x7fccb00cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccb5400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fccb015ab00 0x7fccb015af90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33940/0 (socket says 172.21.15.53:33940) 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccb5400640 1 -- 172.21.15.53:0/3724690350 learned_addr learned my addr 172.21.15.53:0/3724690350 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.906+0000 7fccb4a00640 1 --2- 172.21.15.53:0/3724690350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015ec60 0x7fccb00cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.907+0000 7fccafe00640 1 -- 172.21.15.53:0/3724690350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015ec60 msgr2=0x7fccb00cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:55.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.907+0000 7fccafe00640 1 --2- 172.21.15.53:0/3724690350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015ec60 0x7fccb00cedb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.907+0000 7fccafe00640 1 -- 172.21.15.53:0/3724690350 >> v1:172.21.15.53:6789/0 conn(0x7fccb015ab00 legacy=0x7fccb015af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.907+0000 7fccafe00640 1 -- 172.21.15.53:0/3724690350 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00cc760 con 0x7fccb015f040 2024-01-16T15:07:55.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.908+0000 7fccafe00640 1 --2- 172.21.15.53:0/3724690350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015f040 0x7fccb00cb950 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fcc980097a0 tx=0x7fcc9802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a82f65cf5ecf148a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:55.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.908+0000 7fccaf400640 1 -- 172.21.15.53:0/3724690350 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9802d040 con 0x7fccb015f040 2024-01-16T15:07:55.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.908+0000 7fccaf400640 1 -- 172.21.15.53:0/3724690350 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc98031810 con 0x7fccb015f040 2024-01-16T15:07:55.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.908+0000 7fccaf400640 1 -- 172.21.15.53:0/3724690350 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc98031ba0 con 0x7fccb015f040 2024-01-16T15:07:55.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 -- 172.21.15.53:0/3724690350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015f040 msgr2=0x7fccb00cb950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 --2- 172.21.15.53:0/3724690350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015f040 0x7fccb00cb950 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7fcc980097a0 tx=0x7fcc9802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 -- 172.21.15.53:0/3724690350 shutdown_connections 2024-01-16T15:07:55.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 --2- 172.21.15.53:0/3724690350 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015f040 0x7fccb00cb950 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 --2- 172.21.15.53:0/3724690350 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015ec60 0x7fccb00cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 -- 172.21.15.53:0/3724690350 >> 172.21.15.53:0/3724690350 conn(0x7fccb00d0eb0 msgr2=0x7fccb00d12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:55.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 -- 172.21.15.53:0/3724690350 shutdown_connections 2024-01-16T15:07:55.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.909+0000 7fccb5e00640 1 -- 172.21.15.53:0/3724690350 wait complete. 2024-01-16T15:07:55.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.912+0000 7fccb5e00640 1 Processor -- start 2024-01-16T15:07:55.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.912+0000 7fccb5e00640 1 -- start start 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.913+0000 7fccb5e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb015ab00 0x7fccb00c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.913+0000 7fccb5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 0x7fccb00c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.913+0000 7fccb5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015f040 0x7fccb0140fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.913+0000 7fccb5e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb015b880 con 0x7fccb015ec60 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.913+0000 7fccb5e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb015bb60 con 0x7fccb015ab00 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.913+0000 7fccb5e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00c6700 con 0x7fccb015f040 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 0x7fccb00c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 0x7fccb00c6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44824/0 (socket says 172.21.15.53:44824) 2024-01-16T15:07:55.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 -- 172.21.15.53:0/1549764042 learned_addr learned my addr 172.21.15.53:0/1549764042 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:55.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccb4a00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb015ab00 0x7fccb00c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccb5400640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015f040 0x7fccb0140fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 -- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015f040 msgr2=0x7fccb0140fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015f040 0x7fccb0140fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 -- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb015ab00 msgr2=0x7fccb00c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb015ab00 0x7fccb00c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.914+0000 7fccafe00640 1 -- 172.21.15.53:0/1549764042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc98009400 con 0x7fccb015ec60 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.915+0000 7fccafe00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 0x7fccb00c6c20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fcc98000c00 tx=0x7fcc980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.915+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9803ad60 con 0x7fccb015ec60 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.915+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc980443f0 con 0x7fccb015ec60 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.915+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc980447a0 con 0x7fccb015ec60 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.915+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb01542e0 con 0x7fccb015ec60 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.915+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb00c63c0 con 0x7fccb015ec60 2024-01-16T15:07:55.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.916+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb0067b60 con 0x7fccb015ec60 2024-01-16T15:07:55.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.918+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fcc9804d450 con 0x7fccb015ec60 2024-01-16T15:07:55.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.919+0000 7fccacc00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8806bbc0 0x7fcc8806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:55.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.919+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fcc980aab80 con 0x7fccb015ec60 2024-01-16T15:07:55.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.920+0000 7fccb4a00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8806bbc0 0x7fcc8806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:55.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.921+0000 7fccb4a00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8806bbc0 0x7fcc8806e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcca0006fd0 tx=0x7fcca0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:55.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:55.924+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc98077560 con 0x7fccb015ec60 2024-01-16T15:07:56.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.118+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fccb015b9c0 con 0x7fccb015ec60 2024-01-16T15:07:56.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.119+0000 7fccacc00640 1 -- 172.21.15.53:0/1549764042 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7fccb015b9c0 con 0x7fccb015ec60 2024-01-16T15:07:56.097 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8806bbc0 msgr2=0x7fcc8806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8806bbc0 0x7fcc8806e080 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fcca0006fd0 tx=0x7fcca0008040 comp rx=0 tx=0).stop 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 msgr2=0x7fccb00c6c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 0x7fccb00c6c20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fcc98000c00 tx=0x7fcc980046c0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 shutdown_connections 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8806bbc0 0x7fcc8806e080 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb015f040 0x7fccb0140fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb015ec60 0x7fccb00c6c20 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 --2- 172.21.15.53:0/1549764042 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb015ab00 0x7fccb00c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.122+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 >> 172.21.15.53:0/1549764042 conn(0x7fccb00d0eb0 msgr2=0x7fccb006ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.123+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 shutdown_connections 2024-01-16T15:07:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.123+0000 7fccb5e00640 1 -- 172.21.15.53:0/1549764042 wait complete. 2024-01-16T15:07:56.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:07:56.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.354+0000 7facdce00640 1 Processor -- start 2024-01-16T15:07:56.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.354+0000 7facdce00640 1 -- start start 2024-01-16T15:07:56.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.356+0000 7facdce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facd813df80 0x7facd815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:56.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.356+0000 7facdce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd813e360 0x7facd80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:56.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.356+0000 7facdce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd806c470 con 0x7facd815f9c0 2024-01-16T15:07:56.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.356+0000 7facdce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facd806c750 con 0x7facd813df80 2024-01-16T15:07:56.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.356+0000 7facdce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd806ca30 con 0x7facd813e360 2024-01-16T15:07:56.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facd813df80 0x7facd815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:56.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7faccd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd813e360 0x7facd80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:56.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd5e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7facd813df80 0x7facd815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:60560/0 (socket says 172.21.15.53:60560) 2024-01-16T15:07:56.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd6800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7facd815f9c0 0x7facd81408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:43446/0 (socket says 172.21.15.53:43446) 2024-01-16T15:07:56.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd5e00640 1 -- 172.21.15.53:0/3797324060 learned_addr learned my addr 172.21.15.53:0/3797324060 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:56.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd5e00640 1 -- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd813e360 msgr2=0x7facd80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:56.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd5e00640 1 --2- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd813e360 0x7facd80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.357+0000 7facd5e00640 1 -- 172.21.15.53:0/3797324060 >> v1:172.21.15.146:6789/0 conn(0x7facd815f9c0 legacy=0x7facd81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:56.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.358+0000 7facd5e00640 1 -- 172.21.15.53:0/3797324060 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd806cd10 con 0x7facd813df80 2024-01-16T15:07:56.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.359+0000 7facd5e00640 1 --2- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813df80 0x7facd815bee0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7facc8009870 tx=0x7facc802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2ffa51a4f65b215 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:56.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.360+0000 7facd5400640 1 -- 172.21.15.53:0/3797324060 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8030840 con 0x7facd813df80 2024-01-16T15:07:56.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.360+0000 7facd5400640 1 -- 172.21.15.53:0/3797324060 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc80309a0 con 0x7facd813df80 2024-01-16T15:07:56.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.360+0000 7facd5400640 1 -- 172.21.15.53:0/3797324060 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8030d30 con 0x7facd813df80 2024-01-16T15:07:56.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 -- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813df80 msgr2=0x7facd815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 --2- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813df80 0x7facd815bee0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7facc8009870 tx=0x7facc802b980 comp rx=0 tx=0).stop 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 -- 172.21.15.53:0/3797324060 shutdown_connections 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 --2- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd813e360 0x7facd80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 --2- 172.21.15.53:0/3797324060 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813df80 0x7facd815bee0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 -- 172.21.15.53:0/3797324060 >> 172.21.15.53:0/3797324060 conn(0x7facd806a800 msgr2=0x7facd806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 -- 172.21.15.53:0/3797324060 shutdown_connections 2024-01-16T15:07:56.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.361+0000 7facdce00640 1 -- 172.21.15.53:0/3797324060 wait complete. 2024-01-16T15:07:56.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.363+0000 7facdce00640 1 Processor -- start 2024-01-16T15:07:56.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.364+0000 7facdce00640 1 -- start start 2024-01-16T15:07:56.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facdce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 0x7facd80b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:56.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facdce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813e360 0x7facd80a8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:56.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facdce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd815f9c0 0x7facd80a92b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:56.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facdce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80a97f0 con 0x7facd813df80 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facdce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facd80b42f0 con 0x7facd813e360 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facdce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facd8150b60 con 0x7facd815f9c0 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facd5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 0x7facd80b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facd5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 0x7facd80b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44830/0 (socket says 172.21.15.53:44830) 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.365+0000 7facd5e00640 1 -- 172.21.15.53:0/2739312224 learned_addr learned my addr 172.21.15.53:0/2739312224 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd6800640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd815f9c0 0x7facd80a92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7faccd200640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813e360 0x7facd80a8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd5e00640 1 -- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd815f9c0 msgr2=0x7facd80a92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd5e00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd815f9c0 0x7facd80a92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd5e00640 1 -- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813e360 msgr2=0x7facd80a8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd5e00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813e360 0x7facd80a8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd5e00640 1 -- 172.21.15.53:0/2739312224 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facc80094d0 con 0x7facd813df80 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7faccd200640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813e360 0x7facd80a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:56.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.366+0000 7facd5e00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 0x7facd80b2f60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7facc8000c00 tx=0x7facc8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:56.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.367+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8039a90 con 0x7facd813df80 2024-01-16T15:07:56.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.367+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facc8039c10 con 0x7facd813df80 2024-01-16T15:07:56.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.367+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facc8043410 con 0x7facd813df80 2024-01-16T15:07:56.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.367+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd8151b70 con 0x7facd813df80 2024-01-16T15:07:56.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.367+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facd8151090 con 0x7facd813df80 2024-01-16T15:07:56.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.368+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facd815ee80 con 0x7facd813df80 2024-01-16T15:07:56.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.369+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7facc802f390 con 0x7facd813df80 2024-01-16T15:07:56.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.370+0000 7faccea00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facb006bbc0 0x7facb006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:56.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.370+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7facc80aa5d0 con 0x7facd813df80 2024-01-16T15:07:56.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.371+0000 7faccd200640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facb006bbc0 0x7facb006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:56.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.372+0000 7faccd200640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facb006bbc0 0x7facb006e080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7facd8150b10 tx=0x7facc0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:56.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.377+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7facc8076e90 con 0x7facd813df80 2024-01-16T15:07:56.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:56.641+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7facd80675d0 con 0x7facd813df80 2024-01-16T15:07:57.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.212+0000 7faccea00640 1 -- 172.21.15.53:0/2739312224 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v376) v1 ==== 169+0+0 (secure 0 0 0) 0x7facc80480f0 con 0x7facd813df80 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facb006bbc0 msgr2=0x7facb006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facb006bbc0 0x7facb006e080 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7facd8150b10 tx=0x7facc0008040 comp rx=0 tx=0).stop 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 msgr2=0x7facd80b2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 0x7facd80b2f60 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7facc8000c00 tx=0x7facc8004730 comp rx=0 tx=0).stop 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 shutdown_connections 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facb006bbc0 0x7facb006e080 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7facd815f9c0 0x7facd80a92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7facd813e360 0x7facd80a8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 --2- 172.21.15.53:0/2739312224 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7facd813df80 0x7facd80b2f60 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 >> 172.21.15.53:0/2739312224 conn(0x7facd806a800 msgr2=0x7facd80af400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 shutdown_connections 2024-01-16T15:07:57.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.216+0000 7facdce00640 1 -- 172.21.15.53:0/2739312224 wait complete. 2024-01-16T15:07:57.213 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:07:57.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.464+0000 7f888be00640 1 Processor -- start 2024-01-16T15:07:57.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.464+0000 7f888be00640 1 -- start start 2024-01-16T15:07:57.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.465+0000 7f888be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f888c1447f0 0x7f888c15c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:57.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.465+0000 7f888be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c0cccf0 0x7f888c15c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:57.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.465+0000 7f888be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c15c5c0 con 0x7f888c06c470 2024-01-16T15:07:57.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c06d690 con 0x7f888c1447f0 2024-01-16T15:07:57.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0cebb0 con 0x7f888c0cccf0 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f888c1447f0 0x7f888c15c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f888c1447f0 0x7f888c15c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44836/0 (socket says 172.21.15.53:44836) 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 -- 172.21.15.53:0/1052109217 learned_addr learned my addr 172.21.15.53:0/1052109217 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 -- 172.21.15.53:0/1052109217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c0cccf0 msgr2=0x7f888c15c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 --2- 172.21.15.53:0/1052109217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c0cccf0 0x7f888c15c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 -- 172.21.15.53:0/1052109217 >> v1:172.21.15.146:6789/0 conn(0x7f888c06c470 legacy=0x7f888c0cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.466+0000 7f888aa00640 1 -- 172.21.15.53:0/1052109217 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c06ca10 con 0x7f888c1447f0 2024-01-16T15:07:57.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.467+0000 7f888aa00640 1 --2- 172.21.15.53:0/1052109217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c15c080 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f888c0cecf0 tx=0x7f887c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d4f1f99b60cb94c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:57.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888a000640 1 -- 172.21.15.53:0/1052109217 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c0317c0 con 0x7f888c1447f0 2024-01-16T15:07:57.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888a000640 1 -- 172.21.15.53:0/1052109217 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c031920 con 0x7f888c1447f0 2024-01-16T15:07:57.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888a000640 1 -- 172.21.15.53:0/1052109217 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c031cb0 con 0x7f888c1447f0 2024-01-16T15:07:57.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 -- 172.21.15.53:0/1052109217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 msgr2=0x7f888c15c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 --2- 172.21.15.53:0/1052109217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c15c080 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f888c0cecf0 tx=0x7f887c02b860 comp rx=0 tx=0).stop 2024-01-16T15:07:57.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 -- 172.21.15.53:0/1052109217 shutdown_connections 2024-01-16T15:07:57.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 --2- 172.21.15.53:0/1052109217 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c0cccf0 0x7f888c15c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 --2- 172.21.15.53:0/1052109217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c15c080 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 -- 172.21.15.53:0/1052109217 >> 172.21.15.53:0/1052109217 conn(0x7f888c06a3e0 msgr2=0x7f888c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:57.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 -- 172.21.15.53:0/1052109217 shutdown_connections 2024-01-16T15:07:57.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.468+0000 7f888be00640 1 -- 172.21.15.53:0/1052109217 wait complete. 2024-01-16T15:07:57.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.470+0000 7f888be00640 1 Processor -- start 2024-01-16T15:07:57.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.470+0000 7f888be00640 1 -- start start 2024-01-16T15:07:57.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.471+0000 7f888be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c06c470 0x7f888c140150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:57.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.471+0000 7f888be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c0cccf0 0x7f888c0aabd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:57.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.471+0000 7f888be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c1f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:57.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.471+0000 7f888be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c070e30 con 0x7f888c1447f0 2024-01-16T15:07:57.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.471+0000 7f888be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c070220 con 0x7f888c0cccf0 2024-01-16T15:07:57.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.471+0000 7f888be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c070500 con 0x7f888c06c470 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c1f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c1f6140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44850/0 (socket says 172.21.15.53:44850) 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 -- 172.21.15.53:0/1350408949 learned_addr learned my addr 172.21.15.53:0/1350408949 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888aa00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c06c470 0x7f888c140150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f8883e00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c0cccf0 0x7f888c0aabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 -- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c06c470 msgr2=0x7f888c140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c06c470 0x7f888c140150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 -- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c0cccf0 msgr2=0x7f888c0aabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c0cccf0 0x7f888c0aabd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 -- 172.21.15.53:0/1350408949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f887c0094d0 con 0x7f888c1447f0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.472+0000 7f888b400640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c1f6140 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f888400b970 tx=0x7f888400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.473+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888400cb40 con 0x7f888c1447f0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.473+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f888400ccc0 con 0x7f888c1447f0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.473+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884014680 con 0x7f888c1447f0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.473+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c0707e0 con 0x7f888c1447f0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.473+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c140690 con 0x7f888c1447f0 2024-01-16T15:07:57.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.474+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c070fa0 con 0x7f888c1447f0 2024-01-16T15:07:57.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.475+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f888401d3b0 con 0x7f888c1447f0 2024-01-16T15:07:57.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.476+0000 7f8882a00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bc10 0x7f886406e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:57.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.476+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f8884088660 con 0x7f888c1447f0 2024-01-16T15:07:57.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.476+0000 7f888aa00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bc10 0x7f886406e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:57.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.478+0000 7f888aa00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bc10 0x7f886406e0d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f887c036000 tx=0x7f887c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:57.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.480+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8884054d30 con 0x7f888c1447f0 2024-01-16T15:07:57.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.671+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f888c067450 con 0x7f888c1447f0 2024-01-16T15:07:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.676+0000 7f8882a00640 1 -- 172.21.15.53:0/1350408949 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8884019070 con 0x7f888c1447f0 2024-01-16T15:07:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bc10 msgr2=0x7f886406e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bc10 0x7f886406e0d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f887c036000 tx=0x7f887c03d040 comp rx=0 tx=0).stop 2024-01-16T15:07:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 msgr2=0x7f888c1f6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c1f6140 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f888400b970 tx=0x7f888400be30 comp rx=0 tx=0).stop 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 shutdown_connections 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bc10 0x7f886406e0d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c1447f0 0x7f888c1f6140 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c0cccf0 0x7f888c0aabd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 --2- 172.21.15.53:0/1350408949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c06c470 0x7f888c140150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.679+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 >> 172.21.15.53:0/1350408949 conn(0x7f888c06a3e0 msgr2=0x7f888c0afcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.680+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 shutdown_connections 2024-01-16T15:07:57.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:57.680+0000 7f888be00640 1 -- 172.21.15.53:0/1350408949 wait complete. 2024-01-16T15:07:57.676 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:07:57.677 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:57.789 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:07:57.914 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:07:57.914 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:07:57.914 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:57.973 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:07:57.973 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:07:57.973 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:58.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:07:58.033 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:07:58.033 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:58.092 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:07:58.092 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:07:58.092 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:07:58.148 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:07:58.148 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:07:58.149 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:07:58.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.395+0000 7f6ea6c00640 1 Processor -- start 2024-01-16T15:07:58.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.395+0000 7f6ea6c00640 1 -- start start 2024-01-16T15:07:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 0x7f6ea0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea6c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0070e50 con 0x7f6ea00ccc10 2024-01-16T15:07:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea6c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0144020 con 0x7f6ea00717d0 2024-01-16T15:07:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00723c0 con 0x7f6ea0143c40 2024-01-16T15:07:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea6200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6ea00ccc10 0x7f6ea006d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33956/0 (socket says 172.21.15.53:33956) 2024-01-16T15:07:58.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea6200640 1 -- 172.21.15.53:0/1568567687 learned_addr learned my addr 172.21.15.53:0/1568567687 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:07:58.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.397+0000 7f6ea5800640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 0x7f6ea0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86462387 0 0) 0x7f6ea0070e50 con 0x7f6ea00ccc10 2024-01-16T15:07:58.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e80003520 con 0x7f6ea00ccc10 2024-01-16T15:07:58.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459180339 0 0) 0x7f6e80003520 con 0x7f6ea00ccc10 2024-01-16T15:07:58.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 msgr2=0x7f6ea0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:58.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 0x7f6ea0071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 >> v1:172.21.15.146:6789/0 conn(0x7f6ea00717d0 legacy=0x7f6ea00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea0071e00 con 0x7f6ea00ccc10 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6e98001670 con 0x7f6ea00ccc10 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 >> v1:172.21.15.53:6789/0 conn(0x7f6ea00ccc10 legacy=0x7f6ea006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e800039a0 0x7f6e80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6e80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 0x7f6e80005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00723c0 con 0x7f6e800047c0 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8000bb10 con 0x7f6e800039a0 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8000bcd0 con 0x7f6ea0143c40 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.398+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6e80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea6200640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 0x7f6e80005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea5800640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e800039a0 0x7f6e80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea4e00640 1 -- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 msgr2=0x7f6e80005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea0143c40 0x7f6e80005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea4e00640 1 -- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e800039a0 msgr2=0x7f6e80003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:58.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e800039a0 0x7f6e80003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.399+0000 7f6ea4e00640 1 -- 172.21.15.53:0/1568567687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e80003520 con 0x7f6e800047c0 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.400+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6e80004c40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001750 tx=0x7f6e800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d8c70fdc89856c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.400+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880052d0 con 0x7f6e800047c0 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.400+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8802e050 con 0x7f6e800047c0 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.400+0000 7f6e9fe00640 1 -- 172.21.15.53:0/1568567687 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880035f0 con 0x7f6e800047c0 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6c00640 1 -- 172.21.15.53:0/1568567687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 msgr2=0x7f6e80004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/1568567687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6e80004c40 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001750 tx=0x7f6e800057f0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6200640 1 -- 172.21.15.53:0/1568567687 reap_dead start 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6c00640 1 -- 172.21.15.53:0/1568567687 shutdown_connections 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6c00640 1 -- 172.21.15.53:0/1568567687 >> 172.21.15.53:0/1568567687 conn(0x7f6ea006a800 msgr2=0x7f6ea006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6c00640 1 -- 172.21.15.53:0/1568567687 shutdown_connections 2024-01-16T15:07:58.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.401+0000 7f6ea6c00640 1 -- 172.21.15.53:0/1568567687 wait complete. 2024-01-16T15:07:58.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.403+0000 7f6ea6c00640 1 Processor -- start 2024-01-16T15:07:58.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.403+0000 7f6ea6c00640 1 -- start start 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea6c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6ea00c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea00c8380 0x7f6ea00b2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea00b3120 0x7f6ea012d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea6c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea012e550 con 0x7f6e800047c0 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00b35a0 con 0x7f6ea00c8380 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea00c3f10 con 0x7f6ea00b3120 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6ea00c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6ea00c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44876/0 (socket says 172.21.15.53:44876) 2024-01-16T15:07:58.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.404+0000 7f6ea5800640 1 -- 172.21.15.53:0/2595651654 learned_addr learned my addr 172.21.15.53:0/2595651654 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea6200640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea00b3120 0x7f6ea012d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea5800640 1 -- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea00b3120 msgr2=0x7f6ea012d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea5800640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea00b3120 0x7f6ea012d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea00c8380 0x7f6ea00b2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea5800640 1 -- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea00c8380 msgr2=0x7f6ea00b2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea5800640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea00c8380 0x7f6ea00b2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea5800640 1 -- 172.21.15.53:0/2595651654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00c41f0 con 0x7f6e800047c0 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea6200640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea00b3120 0x7f6ea012d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:58.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea5800640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6ea00c4cd0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6e90007f30 tx=0x7f6e90011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9000e030 con 0x7f6e800047c0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90016470 con 0x7f6e800047c0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90016820 con 0x7f6e800047c0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea00c44d0 con 0x7f6e800047c0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.405+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea00c8800 con 0x7f6e800047c0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.406+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea00c4050 con 0x7f6e800047c0 2024-01-16T15:07:58.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.407+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f6e90016980 con 0x7f6e800047c0 2024-01-16T15:07:58.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.409+0000 7f6e9d600640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e7806bbc0 0x7f6e7806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.409+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6e9000f070 con 0x7f6e800047c0 2024-01-16T15:07:58.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.409+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e7806bbc0 0x7f6e7806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.411+0000 7f6ea4e00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e7806bbc0 0x7f6e7806e080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001750 tx=0x7f6e88009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:58.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.414+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ea00c4050 con 0x7f6e800047c0 2024-01-16T15:07:58.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.647+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ea00c4050 con 0x7f6e800047c0 2024-01-16T15:07:58.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.648+0000 7f6e9d600640 1 -- 172.21.15.53:0/2595651654 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f6ea00c4050 con 0x7f6e800047c0 2024-01-16T15:07:58.625 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:58.625 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":38,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5240,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6833/2632791628","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2632791628},{"type":"v1","addr":"172.21.15.53:6833","nonce":2632791628}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":29},{"gid":5525,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/3341240307","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":3341240307},{"type":"v1","addr":"172.21.15.146:6837","nonce":3341240307}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":38},{"gid":5680,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3474334502","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6834","nonce":3474334502},{"type":"v1","addr":"172.21.15.53:6835","nonce":3474334502}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30},{"gid":5683,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2549681777","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2549681777},{"type":"v1","addr":"172.21.15.146:6836","nonce":2549681777}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[]} 2024-01-16T15:07:58.626 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 38 2024-01-16T15:07:58.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.650+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e7806bbc0 msgr2=0x7f6e7806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e7806bbc0 0x7f6e7806e080 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001750 tx=0x7f6e88009030 comp rx=0 tx=0).stop 2024-01-16T15:07:58.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 msgr2=0x7f6ea00c4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6ea00c4cd0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f6e90007f30 tx=0x7f6e90011660 comp rx=0 tx=0).stop 2024-01-16T15:07:58.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 shutdown_connections 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e7806bbc0 0x7f6e7806e080 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea00b3120 0x7f6ea012d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea00c8380 0x7f6ea00b2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 --2- 172.21.15.53:0/2595651654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e800047c0 0x7f6ea00c4cd0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 >> 172.21.15.53:0/2595651654 conn(0x7f6ea006a800 msgr2=0x7f6ea00ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 shutdown_connections 2024-01-16T15:07:58.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.651+0000 7f6ea6c00640 1 -- 172.21.15.53:0/2595651654 wait complete. 2024-01-16T15:07:58.647 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:07:58.647 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:07:58.647 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:07:58.647 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:58.647 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:07:58.647 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:07:58.647 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:07:58.648 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:58.648 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:07:58.648 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:07:58.648 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:07:58.648 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:58.648 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:07:58.648 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:07:58.649 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:07:58.649 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:07:58.663 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:07:58.664 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:07:58.668 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:07:58.668 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:07:58.674 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:07:58.674 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:07:58.675 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:07:58.675 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:07:58.715 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:07:58.716 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:58.722 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:07:58.722 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:58.723 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:07:58.723 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:58.729 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:07:58.729 DEBUG:teuthology.parallel:result is None 2024-01-16T15:07:58.730 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:07:58.877 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:07:58.877 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:07:58.879 INFO:tasks.ceph.mds.a.smithi053.stderr:2024-01-16T15:07:58.902+0000 7f3f75600640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2024-01-16T15:07:58.886 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:07:58.886 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:07:58.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.944+0000 7fe7c4600640 1 Processor -- start 2024-01-16T15:07:58.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.945+0000 7fe7c4600640 1 -- start start 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c4600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13df80 0x7fe7bc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c4600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7bc13e360 0x7fe7bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c4600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc06c470 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c4600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7bc06c750 con 0x7fe7bc13e360 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c4600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc06ca30 con 0x7fe7bc13df80 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c1200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7bc13e360 0x7fe7bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c1200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7bc13e360 0x7fe7bc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44932/0 (socket says 172.21.15.53:44932) 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c1200640 1 -- 172.21.15.53:0/3661515615 learned_addr learned my addr 172.21.15.53:0/3661515615 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:58.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.946+0000 7fe7c1c00640 1 --2- 172.21.15.53:0/3661515615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13df80 0x7fe7bc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.947+0000 7fe7c1200640 1 -- 172.21.15.53:0/3661515615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13df80 msgr2=0x7fe7bc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:58.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.947+0000 7fe7c1200640 1 --2- 172.21.15.53:0/3661515615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13df80 0x7fe7bc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.947+0000 7fe7c1200640 1 -- 172.21.15.53:0/3661515615 >> v1:172.21.15.146:6789/0 conn(0x7fe7bc15f9c0 legacy=0x7fe7bc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.947+0000 7fe7c1200640 1 -- 172.21.15.53:0/3661515615 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc06cd10 con 0x7fe7bc13e360 2024-01-16T15:07:58.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.948+0000 7fe7c1200640 1 --2- 172.21.15.53:0/3661515615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc13e360 0x7fe7bc0cf290 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009870 tx=0x7fe7a402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af038faf699117f1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:58.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.948+0000 7fe7bbe00640 1 -- 172.21.15.53:0/3661515615 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4004230 con 0x7fe7bc13e360 2024-01-16T15:07:58.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.948+0000 7fe7bbe00640 1 -- 172.21.15.53:0/3661515615 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a402e050 con 0x7fe7bc13e360 2024-01-16T15:07:58.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.948+0000 7fe7bbe00640 1 -- 172.21.15.53:0/3661515615 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4003610 con 0x7fe7bc13e360 2024-01-16T15:07:58.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661515615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc13e360 msgr2=0x7fe7bc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661515615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc13e360 0x7fe7bc0cf290 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009870 tx=0x7fe7a402b7e0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661515615 shutdown_connections 2024-01-16T15:07:58.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661515615 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc13e360 0x7fe7bc0cf290 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661515615 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13df80 0x7fe7bc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661515615 >> 172.21.15.53:0/3661515615 conn(0x7fe7bc06a800 msgr2=0x7fe7bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:58.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661515615 shutdown_connections 2024-01-16T15:07:58.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.949+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661515615 wait complete. 2024-01-16T15:07:58.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.951+0000 7fe7c4600640 1 Processor -- start 2024-01-16T15:07:58.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.951+0000 7fe7c4600640 1 -- start start 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c4600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7bc13df80 0x7fe7bc1540f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c4600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13e360 0x7fe7bc13d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c4600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 0x7fe7bc13d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c4600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc14efd0 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c4600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc14abc0 con 0x7fe7bc13df80 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c4600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc14aea0 con 0x7fe7bc13e360 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c2600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 0x7fe7bc13d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c1c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7bc13df80 0x7fe7bc1540f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c2600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 0x7fe7bc13d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44948/0 (socket says 172.21.15.53:44948) 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c2600640 1 -- 172.21.15.53:0/3661423082 learned_addr learned my addr 172.21.15.53:0/3661423082 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.952+0000 7fe7c1200640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13e360 0x7fe7bc13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c2600640 1 -- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13e360 msgr2=0x7fe7bc13d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c2600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13e360 0x7fe7bc13d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c2600640 1 -- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7bc13df80 msgr2=0x7fe7bc1540f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c2600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7bc13df80 0x7fe7bc1540f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c2600640 1 -- 172.21.15.53:0/3661423082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a40094d0 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c2600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 0x7fe7bc13d950 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400caf0 tx=0x7fe7b40020d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:58.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4003da0 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b400a700 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400aad0 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc14b180 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.953+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc154630 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.955+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc154880 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.956+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe7b400ac30 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.957+0000 7fe7b9600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79406bc10 0x7fe79406e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:58.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.957+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe7b4087ee0 con 0x7fe7bc15f9c0 2024-01-16T15:07:58.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.958+0000 7fe7c1c00640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79406bc10 0x7fe79406e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:58.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.958+0000 7fe7c1c00640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79406bc10 0x7fe79406e0d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009aa0 tx=0x7fe7ac009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:58.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:58.962+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7b4054850 con 0x7fe7bc15f9c0 2024-01-16T15:07:59.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.167+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe7bc14b6a0 con 0x7fe7bc15f9c0 2024-01-16T15:07:59.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.235+0000 7fe7b9600640 1 -- 172.21.15.53:0/3661423082 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v66) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe7bc14b6a0 con 0x7fe7bc15f9c0 2024-01-16T15:07:59.212 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79406bc10 msgr2=0x7fe79406e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79406bc10 0x7fe79406e0d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fe7ac009aa0 tx=0x7fe7ac009190 comp rx=0 tx=0).stop 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 msgr2=0x7fe7bc13d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 0x7fe7bc13d950 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400caf0 tx=0x7fe7b40020d0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 shutdown_connections 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79406bc10 0x7fe79406e0d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7bc15f9c0 0x7fe7bc13d950 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7bc13e360 0x7fe7bc13d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 --2- 172.21.15.53:0/3661423082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7bc13df80 0x7fe7bc1540f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 >> 172.21.15.53:0/3661423082 conn(0x7fe7bc06a800 msgr2=0x7fe7bc0afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:59.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 shutdown_connections 2024-01-16T15:07:59.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.238+0000 7fe7c4600640 1 -- 172.21.15.53:0/3661423082 wait complete. 2024-01-16T15:07:59.241 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:07:59.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.481+0000 7fcd61800640 1 Processor -- start 2024-01-16T15:07:59.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.481+0000 7fcd61800640 1 -- start start 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.482+0000 7fcd61800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15e670 0x7fcd5c0b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.482+0000 7fcd61800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0ce6a0 0x7fcd5c0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.482+0000 7fcd61800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd5c15ea50 0x7fcd5c0cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.482+0000 7fcd61800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5c0cf170 con 0x7fcd5c15ea50 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.482+0000 7fcd61800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd5c0cca90 con 0x7fcd5c0ce6a0 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.482+0000 7fcd61800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c0ccd70 con 0x7fcd5c15e670 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd5c15ea50 0x7fcd5c0cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd5c15ea50 0x7fcd5c0cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44960/0 (socket says 172.21.15.53:44960) 2024-01-16T15:07:59.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 -- 172.21.15.53:0/2390238109 learned_addr learned my addr 172.21.15.53:0/2390238109 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:59.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd5be00640 1 --2- 172.21.15.53:0/2390238109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15e670 0x7fcd5c0b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd5b400640 1 --2- 172.21.15.53:0/2390238109 >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0ce6a0 0x7fcd5c0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 -- 172.21.15.53:0/2390238109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15e670 msgr2=0x7fcd5c0b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:59.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 --2- 172.21.15.53:0/2390238109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15e670 0x7fcd5c0b3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 -- 172.21.15.53:0/2390238109 >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0ce6a0 msgr2=0x7fcd5c0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:59.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 --2- 172.21.15.53:0/2390238109 >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0ce6a0 0x7fcd5c0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.483+0000 7fcd60e00640 1 -- 172.21.15.53:0/2390238109 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c0ceab0 con 0x7fcd5c15ea50 2024-01-16T15:07:59.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.485+0000 7fcd60e00640 1 --2- 172.21.15.53:0/2390238109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c15ea50 0x7fcd5c0cfad0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c071d50 tx=0x7fcd5402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ce2d1ed6affa39f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:59.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.485+0000 7fcd5aa00640 1 -- 172.21.15.53:0/2390238109 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5402ca80 con 0x7fcd5c15ea50 2024-01-16T15:07:59.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.485+0000 7fcd5aa00640 1 -- 172.21.15.53:0/2390238109 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5402cbe0 con 0x7fcd5c15ea50 2024-01-16T15:07:59.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.485+0000 7fcd5aa00640 1 -- 172.21.15.53:0/2390238109 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd540343f0 con 0x7fcd5c15ea50 2024-01-16T15:07:59.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 -- 172.21.15.53:0/2390238109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c15ea50 msgr2=0x7fcd5c0cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 --2- 172.21.15.53:0/2390238109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c15ea50 0x7fcd5c0cfad0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c071d50 tx=0x7fcd5402bee0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 -- 172.21.15.53:0/2390238109 shutdown_connections 2024-01-16T15:07:59.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 --2- 172.21.15.53:0/2390238109 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c15ea50 0x7fcd5c0cfad0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 --2- 172.21.15.53:0/2390238109 >> v2:172.21.15.146:3300/0 conn(0x7fcd5c0ce6a0 0x7fcd5c0cf590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 --2- 172.21.15.53:0/2390238109 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15e670 0x7fcd5c0b3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 -- 172.21.15.53:0/2390238109 >> 172.21.15.53:0/2390238109 conn(0x7fcd5c0b2a30 msgr2=0x7fcd5c0b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:59.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 -- 172.21.15.53:0/2390238109 shutdown_connections 2024-01-16T15:07:59.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.486+0000 7fcd61800640 1 -- 172.21.15.53:0/2390238109 wait complete. 2024-01-16T15:07:59.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.488+0000 7fcd61800640 1 Processor -- start 2024-01-16T15:07:59.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.488+0000 7fcd61800640 1 -- start start 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd61800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e670 0x7fcd5c1f2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd61800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15ea50 0x7fcd5c1533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd61800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 0x7fcd5c06be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd61800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c154260 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd61800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c154500 con 0x7fcd5c15e670 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd61800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c1547e0 con 0x7fcd5c15ea50 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd60e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 0x7fcd5c06be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd60e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 0x7fcd5c06be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44964/0 (socket says 172.21.15.53:44964) 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.489+0000 7fcd60e00640 1 -- 172.21.15.53:0/348178843 learned_addr learned my addr 172.21.15.53:0/348178843 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd5be00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e670 0x7fcd5c1f2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd5b400640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15ea50 0x7fcd5c1533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd60e00640 1 -- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15ea50 msgr2=0x7fcd5c1533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd60e00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15ea50 0x7fcd5c1533f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd60e00640 1 -- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e670 msgr2=0x7fcd5c1f2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd60e00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e670 0x7fcd5c1f2c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd60e00640 1 -- 172.21.15.53:0/348178843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd54009b30 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.490+0000 7fcd60e00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 0x7fcd5c06be70 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcd54036040 tx=0x7fcd54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.491+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd54009510 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.491+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5403e450 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.491+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5403e800 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.491+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5c154ac0 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.491+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c153930 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.492+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c154ce0 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.493+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fcd5402cd50 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.494+0000 7fcd53e00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd3406bbc0 0x7fcd3406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.494+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fcd540aaff0 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.494+0000 7fcd5be00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd3406bbc0 0x7fcd3406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.496+0000 7fcd5be00640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd3406bbc0 0x7fcd3406e080 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c006fd0 tx=0x7fcd4c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:59.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.498+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd540778b0 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.762+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fcd5c067b60 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.763+0000 7fcd53e00640 1 -- 172.21.15.53:0/348178843 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fcd5403e960 con 0x7fcd5c0c9f20 2024-01-16T15:07:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:07:59.740 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:07:59.740 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.741 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.742 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.743 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.744 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.745 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.746 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.747 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow r", 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.748 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.749 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:07:59.750 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:07:59.751 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:07:59.752 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd3406bbc0 msgr2=0x7fcd3406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd3406bbc0 0x7fcd3406e080 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c006fd0 tx=0x7fcd4c008040 comp rx=0 tx=0).stop 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 msgr2=0x7fcd5c06be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 0x7fcd5c06be70 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fcd54036040 tx=0x7fcd54004a40 comp rx=0 tx=0).stop 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 shutdown_connections 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd3406bbc0 0x7fcd3406e080 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd5c0c9f20 0x7fcd5c06be70 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd5c15ea50 0x7fcd5c1533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 --2- 172.21.15.53:0/348178843 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd5c15e670 0x7fcd5c1f2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 >> 172.21.15.53:0/348178843 conn(0x7fcd5c0b2a30 msgr2=0x7fcd5c0b62d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.766+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 shutdown_connections 2024-01-16T15:07:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:07:59.767+0000 7fcd61800640 1 -- 172.21.15.53:0/348178843 wait complete. 2024-01-16T15:07:59.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:07:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.010+0000 7fccb9000640 1 Processor -- start 2024-01-16T15:07:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.010+0000 7fccb9000640 1 -- start start 2024-01-16T15:07:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.012+0000 7fccb9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.012+0000 7fccb9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fccb40717d0 0x7fccb40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.012+0000 7fccb9000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb4070e50 con 0x7fccb40cca40 2024-01-16T15:07:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.012+0000 7fccb9000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb40723c0 con 0x7fccb40717d0 2024-01-16T15:07:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.012+0000 7fccb9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb40cce70 con 0x7fccb4143c40 2024-01-16T15:07:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.012+0000 7fccabe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fccb40717d0 0x7fccb40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccabe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fccb40717d0 0x7fccb40cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44966/0 (socket says 172.21.15.53:44966) 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccabe00640 1 -- 172.21.15.53:0/994733079 learned_addr learned my addr 172.21.15.53:0/994733079 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccb3400640 1 --2- 172.21.15.53:0/994733079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccabe00640 1 -- 172.21.15.53:0/994733079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 msgr2=0x7fccb4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccabe00640 1 --2- 172.21.15.53:0/994733079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccabe00640 1 -- 172.21.15.53:0/994733079 >> v1:172.21.15.146:6789/0 conn(0x7fccb40cca40 legacy=0x7fccb415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:07:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.013+0000 7fccabe00640 1 -- 172.21.15.53:0/994733079 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb406e170 con 0x7fccb40717d0 2024-01-16T15:07:59.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.015+0000 7fccabe00640 1 --2- 172.21.15.53:0/994733079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb40cc500 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009870 tx=0x7fcc9c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5717f24d6deb12d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:59.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.015+0000 7fccb2a00640 1 -- 172.21.15.53:0/994733079 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c0317c0 con 0x7fccb40717d0 2024-01-16T15:07:59.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.015+0000 7fccb2a00640 1 -- 172.21.15.53:0/994733079 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9c031920 con 0x7fccb40717d0 2024-01-16T15:07:59.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.015+0000 7fccb2a00640 1 -- 172.21.15.53:0/994733079 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9c031cb0 con 0x7fccb40717d0 2024-01-16T15:07:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 -- 172.21.15.53:0/994733079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 msgr2=0x7fccb40cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 --2- 172.21.15.53:0/994733079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb40cc500 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c009870 tx=0x7fcc9c02b860 comp rx=0 tx=0).stop 2024-01-16T15:07:59.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 -- 172.21.15.53:0/994733079 shutdown_connections 2024-01-16T15:07:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 --2- 172.21.15.53:0/994733079 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb40cc500 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 --2- 172.21.15.53:0/994733079 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 -- 172.21.15.53:0/994733079 >> 172.21.15.53:0/994733079 conn(0x7fccb406a800 msgr2=0x7fccb406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:07:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 -- 172.21.15.53:0/994733079 shutdown_connections 2024-01-16T15:07:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.016+0000 7fccb9000640 1 -- 172.21.15.53:0/994733079 wait complete. 2024-01-16T15:07:59.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.018+0000 7fccb9000640 1 Processor -- start 2024-01-16T15:07:59.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.018+0000 7fccb9000640 1 -- start start 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb414eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb40cca40 0x7fccb4151540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb9000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb413a780 con 0x7fccb40717d0 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb414f020 con 0x7fccb40cca40 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4139f20 con 0x7fccb4143c40 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb414eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb414eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44970/0 (socket says 172.21.15.53:44970) 2024-01-16T15:07:59.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.019+0000 7fccb3400640 1 -- 172.21.15.53:0/787300121 learned_addr learned my addr 172.21.15.53:0/787300121 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccabe00640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb40cca40 0x7fccb4151540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3e00640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3400640 1 -- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 msgr2=0x7fccb4154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3400640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3400640 1 -- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb40cca40 msgr2=0x7fccb4151540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3400640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb40cca40 0x7fccb4151540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3400640 1 -- 172.21.15.53:0/787300121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc9c0094d0 con 0x7fccb40717d0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccabe00640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb40cca40 0x7fccb4151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccb3400640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb414eae0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcca400d620 tx=0x7fcca400dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.020+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400b620 con 0x7fccb40717d0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.021+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca400b780 con 0x7fccb40717d0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.021+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400bb30 con 0x7fccb40717d0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.021+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb4151a80 con 0x7fccb40717d0 2024-01-16T15:07:59.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.021+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb413bec0 con 0x7fccb40717d0 2024-01-16T15:08:00.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.022+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4067450 con 0x7fccb40717d0 2024-01-16T15:08:00.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.023+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fcca400bc90 con 0x7fccb40717d0 2024-01-16T15:08:00.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.024+0000 7fccab400640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8c06bbc0 0x7fcc8c06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.024+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fcca4088b70 con 0x7fccb40717d0 2024-01-16T15:08:00.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.025+0000 7fccabe00640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8c06bbc0 0x7fcc8c06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.026+0000 7fccabe00640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8c06bbc0 0x7fcc8c06e080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c036000 tx=0x7fcc9c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.028+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcca40bde90 con 0x7fccb40717d0 2024-01-16T15:08:00.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.220+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb40ceb30 con 0x7fccb40717d0 2024-01-16T15:08:00.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.222+0000 7fccab400640 1 -- 172.21.15.53:0/787300121 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+12864 (secure 0 0 0) 0x7fcca4055240 con 0x7fccb40717d0 2024-01-16T15:08:00.200 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:00.200 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":68,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:07:59.248377+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8c06bbc0 msgr2=0x7fcc8c06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8c06bbc0 0x7fcc8c06e080 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fcc9c036000 tx=0x7fcc9c03d040 comp rx=0 tx=0).stop 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 msgr2=0x7fccb414eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb414eae0 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fcca400d620 tx=0x7fcca400dae0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 shutdown_connections 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc8c06bbc0 0x7fcc8c06e080 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccb4143c40 0x7fccb4154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccb40cca40 0x7fccb4151540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 --2- 172.21.15.53:0/787300121 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccb40717d0 0x7fccb414eae0 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 >> 172.21.15.53:0/787300121 conn(0x7fccb406a800 msgr2=0x7fccb406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.225+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 shutdown_connections 2024-01-16T15:08:00.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.226+0000 7fccb9000640 1 -- 172.21.15.53:0/787300121 wait complete. 2024-01-16T15:08:00.226 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:08:00.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:08:00.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.472+0000 7ff209400640 1 Processor -- start 2024-01-16T15:08:00.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.472+0000 7ff209400640 1 -- start start 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.473+0000 7ff209400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff2040ae5a0 0x7ff20415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.473+0000 7ff209400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff2040b3ae0 0x7ff20406cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.473+0000 7ff209400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff204070c40 con 0x7ff20406d120 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.473+0000 7ff209400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff204071450 con 0x7ff2040ae5a0 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.473+0000 7ff209400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20406c470 con 0x7ff2040b3ae0 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.474+0000 7ff202800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff2040ae5a0 0x7ff20415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.474+0000 7ff202800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff2040ae5a0 0x7ff20415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44972/0 (socket says 172.21.15.53:44972) 2024-01-16T15:08:00.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.474+0000 7ff202800640 1 -- 172.21.15.53:0/1041861384 learned_addr learned my addr 172.21.15.53:0/1041861384 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:00.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.474+0000 7ff202800640 1 -- 172.21.15.53:0/1041861384 >> v2:172.21.15.146:3300/0 conn(0x7ff2040b3ae0 msgr2=0x7ff20406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:00.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.475+0000 7ff202800640 1 --2- 172.21.15.53:0/1041861384 >> v2:172.21.15.146:3300/0 conn(0x7ff2040b3ae0 0x7ff20406cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.475+0000 7ff202800640 1 -- 172.21.15.53:0/1041861384 >> v1:172.21.15.146:6789/0 conn(0x7ff20406d120 legacy=0x7ff2040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:00.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.475+0000 7ff202800640 1 -- 172.21.15.53:0/1041861384 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20406c750 con 0x7ff2040ae5a0 2024-01-16T15:08:00.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.476+0000 7ff202800640 1 --2- 172.21.15.53:0/1041861384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2040ae5a0 0x7ff20415e430 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff2040cfea0 tx=0x7ff1f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f5810aa4020283 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.476+0000 7ff201400640 1 -- 172.21.15.53:0/1041861384 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4031870 con 0x7ff2040ae5a0 2024-01-16T15:08:00.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff201400640 1 -- 172.21.15.53:0/1041861384 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f40319d0 con 0x7ff2040ae5a0 2024-01-16T15:08:00.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff201400640 1 -- 172.21.15.53:0/1041861384 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4031d60 con 0x7ff2040ae5a0 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 -- 172.21.15.53:0/1041861384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2040ae5a0 msgr2=0x7ff20415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 --2- 172.21.15.53:0/1041861384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2040ae5a0 0x7ff20415e430 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7ff2040cfea0 tx=0x7ff1f402b910 comp rx=0 tx=0).stop 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 -- 172.21.15.53:0/1041861384 shutdown_connections 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 --2- 172.21.15.53:0/1041861384 >> v2:172.21.15.146:3300/0 conn(0x7ff2040b3ae0 0x7ff20406cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 --2- 172.21.15.53:0/1041861384 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff2040ae5a0 0x7ff20415e430 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 -- 172.21.15.53:0/1041861384 >> 172.21.15.53:0/1041861384 conn(0x7ff20406a800 msgr2=0x7ff20406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.477+0000 7ff209400640 1 -- 172.21.15.53:0/1041861384 shutdown_connections 2024-01-16T15:08:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.478+0000 7ff209400640 1 -- 172.21.15.53:0/1041861384 wait complete. 2024-01-16T15:08:00.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.479+0000 7ff209400640 1 Processor -- start 2024-01-16T15:08:00.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.479+0000 7ff209400640 1 -- start start 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.480+0000 7ff209400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 0x7ff2040cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.480+0000 7ff209400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2040ae5a0 0x7ff20414ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.480+0000 7ff209400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2040b3ae0 0x7ff2040ce120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.480+0000 7ff209400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2041f70b0 con 0x7ff20406d120 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.480+0000 7ff209400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cb940 con 0x7ff2040ae5a0 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.480+0000 7ff209400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2040cbc20 con 0x7ff2040b3ae0 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 0x7ff2040cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 0x7ff2040cc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44982/0 (socket says 172.21.15.53:44982) 2024-01-16T15:08:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 -- 172.21.15.53:0/871456173 learned_addr learned my addr 172.21.15.53:0/871456173 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 -- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2040b3ae0 msgr2=0x7ff2040ce120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2040b3ae0 0x7ff2040ce120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff201e00640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2040ae5a0 0x7ff20414ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 -- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2040ae5a0 msgr2=0x7ff20414ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2040ae5a0 0x7ff20414ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff202800640 1 -- 172.21.15.53:0/871456173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f40094d0 con 0x7ff20406d120 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.481+0000 7ff201e00640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2040ae5a0 0x7ff20414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.482+0000 7ff202800640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 0x7ff2040cc700 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000c00 tx=0x7ff1f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.482+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f403ad10 con 0x7ff20406d120 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.482+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f403ae70 con 0x7ff20406d120 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.482+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4044610 con 0x7ff20406d120 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.482+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2040cbf00 con 0x7ff20406d120 2024-01-16T15:08:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.482+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2040ccc40 con 0x7ff20406d120 2024-01-16T15:08:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.483+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2040cc420 con 0x7ff20406d120 2024-01-16T15:08:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.484+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7ff1f404d450 con 0x7ff20406d120 2024-01-16T15:08:00.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.485+0000 7ff1faa00640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1dc06bbc0 0x7ff1dc06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.485+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7ff1f402d040 con 0x7ff20406d120 2024-01-16T15:08:00.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.486+0000 7ff201e00640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1dc06bbc0 0x7ff1dc06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.487+0000 7ff201e00640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1dc06bbc0 0x7ff1dc06e080 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff20415d3d0 tx=0x7ff1ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.490+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff1f40dfe90 con 0x7ff20406d120 2024-01-16T15:08:00.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.683+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2040675d0 con 0x7ff20406d120 2024-01-16T15:08:00.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.684+0000 7ff1faa00640 1 -- 172.21.15.53:0/871456173 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff1f4077480 con 0x7ff20406d120 2024-01-16T15:08:00.661 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1dc06bbc0 msgr2=0x7ff1dc06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1dc06bbc0 0x7ff1dc06e080 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff20415d3d0 tx=0x7ff1ec008040 comp rx=0 tx=0).stop 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 msgr2=0x7ff2040cc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 0x7ff2040cc700 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4000c00 tx=0x7ff1f40046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 shutdown_connections 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1dc06bbc0 0x7ff1dc06e080 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2040b3ae0 0x7ff2040ce120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2040ae5a0 0x7ff20414ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 --2- 172.21.15.53:0/871456173 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff20406d120 0x7ff2040cc700 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 >> 172.21.15.53:0/871456173 conn(0x7ff20406a800 msgr2=0x7ff2040cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:00.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.687+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 shutdown_connections 2024-01-16T15:08:00.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.688+0000 7ff209400640 1 -- 172.21.15.53:0/871456173 wait complete. 2024-01-16T15:08:00.685 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:08:00.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.928+0000 7fd985600640 1 Processor -- start 2024-01-16T15:08:00.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.928+0000 7fd985600640 1 -- start start 2024-01-16T15:08:00.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.929+0000 7fd985600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd98015e6e0 0x7fd9800cb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd985600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd9800c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd985600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9800b21f0 con 0x7fd9800cbdf0 2024-01-16T15:08:00.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd985600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9800ce540 con 0x7fd98015e6e0 2024-01-16T15:08:00.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd985600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800d1cb0 con 0x7fd9800c49c0 2024-01-16T15:08:00.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd97fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd98015e6e0 0x7fd9800cb8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd984c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd9800c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd984c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd9800c4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:38378/0 (socket says 172.21.15.53:38378) 2024-01-16T15:08:00.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.930+0000 7fd984c00640 1 -- 172.21.15.53:0/3391189216 learned_addr learned my addr 172.21.15.53:0/3391189216 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:00.908 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.931+0000 7fd984c00640 1 -- 172.21.15.53:0/3391189216 >> v2:172.21.15.146:3300/0 conn(0x7fd98015e6e0 msgr2=0x7fd9800cb8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:00.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.931+0000 7fd984c00640 1 --2- 172.21.15.53:0/3391189216 >> v2:172.21.15.146:3300/0 conn(0x7fd98015e6e0 0x7fd9800cb8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.931+0000 7fd984c00640 1 -- 172.21.15.53:0/3391189216 >> v1:172.21.15.146:6789/0 conn(0x7fd9800cbdf0 legacy=0x7fd9800cc230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.931+0000 7fd984c00640 1 -- 172.21.15.53:0/3391189216 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9800b24d0 con 0x7fd9800c49c0 2024-01-16T15:08:00.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.933+0000 7fd984c00640 1 --2- 172.21.15.53:0/3391189216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd9800c4df0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd978009e50 tx=0x7fd97802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71a318e24ee2d5c8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.933+0000 7fd97ea00640 1 -- 172.21.15.53:0/3391189216 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97802cb00 con 0x7fd9800c49c0 2024-01-16T15:08:00.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd97ea00640 1 -- 172.21.15.53:0/3391189216 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97802cc60 con 0x7fd9800c49c0 2024-01-16T15:08:00.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd97ea00640 1 -- 172.21.15.53:0/3391189216 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9780343f0 con 0x7fd9800c49c0 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 -- 172.21.15.53:0/3391189216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 msgr2=0x7fd9800c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 --2- 172.21.15.53:0/3391189216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd9800c4df0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd978009e50 tx=0x7fd97802bf60 comp rx=0 tx=0).stop 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 -- 172.21.15.53:0/3391189216 shutdown_connections 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 --2- 172.21.15.53:0/3391189216 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd9800c4df0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 --2- 172.21.15.53:0/3391189216 >> v2:172.21.15.146:3300/0 conn(0x7fd98015e6e0 0x7fd9800cb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 -- 172.21.15.53:0/3391189216 >> 172.21.15.53:0/3391189216 conn(0x7fd9800d0de0 msgr2=0x7fd9800d11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 -- 172.21.15.53:0/3391189216 shutdown_connections 2024-01-16T15:08:00.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.934+0000 7fd985600640 1 -- 172.21.15.53:0/3391189216 wait complete. 2024-01-16T15:08:00.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.936+0000 7fd985600640 1 Processor -- start 2024-01-16T15:08:00.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.936+0000 7fd985600640 1 -- start start 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.937+0000 7fd985600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd980150270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.937+0000 7fd985600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9800cbdf0 0x7fd98006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.937+0000 7fd985600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 0x7fd98006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.937+0000 7fd985600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98009e360 con 0x7fd98015e6e0 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.937+0000 7fd985600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98009e080 con 0x7fd9800cbdf0 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.937+0000 7fd985600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9800c8240 con 0x7fd9800c49c0 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 0x7fd98006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 0x7fd98006e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44988/0 (socket says 172.21.15.53:44988) 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 -- 172.21.15.53:0/1436218791 learned_addr learned my addr 172.21.15.53:0/1436218791 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd97fe00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd980150270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd97f400640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9800cbdf0 0x7fd98006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 -- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 msgr2=0x7fd980150270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd980150270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 -- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9800cbdf0 msgr2=0x7fd98006e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9800cbdf0 0x7fd98006e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd984c00640 1 -- 172.21.15.53:0/1436218791 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd978009b30 con 0x7fd98015e6e0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.938+0000 7fd97f400640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9800cbdf0 0x7fd98006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.939+0000 7fd984c00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 0x7fd98006e890 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd978009e50 tx=0x7fd978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.939+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97803e450 con 0x7fd98015e6e0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.939+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97803e5b0 con 0x7fd98015e6e0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.939+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97803e960 con 0x7fd98015e6e0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.939+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9800c8520 con 0x7fd98015e6e0 2024-01-16T15:08:00.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.939+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98009d3c0 con 0x7fd98015e6e0 2024-01-16T15:08:00.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.940+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9800c8a40 con 0x7fd98015e6e0 2024-01-16T15:08:00.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.941+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fd97803eac0 con 0x7fd98015e6e0 2024-01-16T15:08:00.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.942+0000 7fd977e00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95806bbc0 0x7fd95806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:00.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.942+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fd9780ab100 con 0x7fd98015e6e0 2024-01-16T15:08:00.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.943+0000 7fd97fe00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95806bbc0 0x7fd95806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:00.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.944+0000 7fd97fe00640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95806bbc0 0x7fd95806e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd970009ad0 tx=0x7fd970008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:00.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:00.947+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd9800c8a40 con 0x7fd98015e6e0 2024-01-16T15:08:01.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.146+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd9800c8a40 con 0x7fd98015e6e0 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.147+0000 7fd977e00640 1 -- 172.21.15.53:0/1436218791 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v68) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd9800c8a40 con 0x7fd98015e6e0 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:08:01.124 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:08:01.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.149+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95806bbc0 msgr2=0x7fd95806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.149+0000 7fd985600640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95806bbc0 0x7fd95806e080 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd970009ad0 tx=0x7fd970008040 comp rx=0 tx=0).stop 2024-01-16T15:08:01.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.149+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 msgr2=0x7fd98006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.149+0000 7fd985600640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 0x7fd98006e890 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fd978009e50 tx=0x7fd978004a40 comp rx=0 tx=0).stop 2024-01-16T15:08:01.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 shutdown_connections 2024-01-16T15:08:01.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95806bbc0 0x7fd95806e080 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98015e6e0 0x7fd98006e890 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd9800cbdf0 0x7fd98006e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 --2- 172.21.15.53:0/1436218791 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd9800c49c0 0x7fd980150270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 >> 172.21.15.53:0/1436218791 conn(0x7fd9800d0de0 msgr2=0x7fd98015cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:01.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 shutdown_connections 2024-01-16T15:08:01.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.150+0000 7fd985600640 1 -- 172.21.15.53:0/1436218791 wait complete. 2024-01-16T15:08:01.148 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:08:01.148 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:08:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.390+0000 7fb4d9600640 1 Processor -- start 2024-01-16T15:08:01.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.390+0000 7fb4d9600640 1 -- start start 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.391+0000 7fb4d9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4d40d0220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.391+0000 7fb4d9600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb4d40d0e90 0x7fb4d40afc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.391+0000 7fb4d9600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4d40d12f0 con 0x7fb4d40b01b0 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.391+0000 7fb4d9600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4d40ce5d0 con 0x7fb4d40d0e90 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.391+0000 7fb4d9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40b23c0 con 0x7fb4d415d3c0 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.392+0000 7fb4d8c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb4d40b01b0 0x7fb4d40ce9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33996/0 (socket says 172.21.15.53:33996) 2024-01-16T15:08:01.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.392+0000 7fb4d8c00640 1 -- 172.21.15.53:0/3556400805 learned_addr learned my addr 172.21.15.53:0/3556400805 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:01.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.392+0000 7fb4d3400640 1 --2- 172.21.15.53:0/3556400805 >> v2:172.21.15.146:3300/0 conn(0x7fb4d40d0e90 0x7fb4d40afc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.392+0000 7fb4d3e00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4d40d0220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.392+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489304703 0 0) 0x7fb4cc001110 con 0x7fb4d40b01b0 2024-01-16T15:08:01.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.392+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4b4003520 con 0x7fb4d40b01b0 2024-01-16T15:08:01.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3011160706 0 0) 0x7fb4b4003520 con 0x7fb4d40b01b0 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 msgr2=0x7fb4d40d0220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4d40d0220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 >> v2:172.21.15.146:3300/0 conn(0x7fb4d40d0e90 msgr2=0x7fb4d40afc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 --2- 172.21.15.53:0/3556400805 >> v2:172.21.15.146:3300/0 conn(0x7fb4d40d0e90 0x7fb4d40afc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d40d0800 con 0x7fb4d40b01b0 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb4cc001800 con 0x7fb4d40b01b0 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 >> v1:172.21.15.53:6789/0 conn(0x7fb4d40b01b0 legacy=0x7fb4d40ce9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4003970 0x7fb4b4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4b40047c0 0x7fb4b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40ce5d0 con 0x7fb4b40047c0 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40b23c0 con 0x7fb4b4003970 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b400ba30 con 0x7fb4d415d3c0 2024-01-16T15:08:01.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.393+0000 7fb4d3400640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4b40047c0 0x7fb4b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d3e00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4003970 0x7fb4b4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d8c00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d3400640 1 -- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 msgr2=0x7fb4b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d3400640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d3400640 1 -- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4003970 msgr2=0x7fb4b4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d3400640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b4003970 0x7fb4b4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d3400640 1 -- 172.21.15.53:0/3556400805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b4003520 con 0x7fb4b40047c0 2024-01-16T15:08:01.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.394+0000 7fb4d8c00640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d415d3c0 0x7fb4b4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.395+0000 7fb4d3400640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4b40047c0 0x7fb4b4004c40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009f90 tx=0x7fb4b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98df3237e739948a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.395+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc02cb30 con 0x7fb4b40047c0 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.395+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4bc00f040 con 0x7fb4b40047c0 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.395+0000 7fb4d2a00640 1 -- 172.21.15.53:0/3556400805 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc034440 con 0x7fb4b40047c0 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d9600640 1 -- 172.21.15.53:0/3556400805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4b40047c0 msgr2=0x7fb4b4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d9600640 1 --2- 172.21.15.53:0/3556400805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4b40047c0 0x7fb4b4004c40 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009f90 tx=0x7fb4b40057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d8c00640 1 -- 172.21.15.53:0/3556400805 reap_dead start 2024-01-16T15:08:01.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d9600640 1 -- 172.21.15.53:0/3556400805 shutdown_connections 2024-01-16T15:08:01.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d9600640 1 -- 172.21.15.53:0/3556400805 >> 172.21.15.53:0/3556400805 conn(0x7fb4d40d30b0 msgr2=0x7fb4d40d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:01.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d9600640 1 -- 172.21.15.53:0/3556400805 shutdown_connections 2024-01-16T15:08:01.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.396+0000 7fb4d9600640 1 -- 172.21.15.53:0/3556400805 wait complete. 2024-01-16T15:08:01.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.398+0000 7fb4d9600640 1 Processor -- start 2024-01-16T15:08:01.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.398+0000 7fb4d9600640 1 -- start start 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b40047c0 0x7fb4d415bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d41f6e30 0x7fb4d40b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 0x7fb4d40b4510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d9600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40cd030 con 0x7fb4d40b7d50 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40b8220 con 0x7fb4b40047c0 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d40b8500 con 0x7fb4d41f6e30 2024-01-16T15:08:01.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 0x7fb4d40b4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 0x7fb4d40b4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45004/0 (socket says 172.21.15.53:45004) 2024-01-16T15:08:01.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d8c00640 1 -- 172.21.15.53:0/1888058578 learned_addr learned my addr 172.21.15.53:0/1888058578 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d8c00640 1 -- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d41f6e30 msgr2=0x7fb4d40b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.399+0000 7fb4d3400640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d41f6e30 0x7fb4d40b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d3e00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b40047c0 0x7fb4d415bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d8c00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d41f6e30 0x7fb4d40b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d8c00640 1 -- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b40047c0 msgr2=0x7fb4d415bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d8c00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b40047c0 0x7fb4d415bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d8c00640 1 -- 172.21.15.53:0/1888058578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d41f6350 con 0x7fb4d40b7d50 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d3400640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d41f6e30 0x7fb4d40b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d8c00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 0x7fb4d40b4510 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fb4d408b3c0 tx=0x7fb4cc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc007230 con 0x7fb4d40b7d50 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4cc007390 con 0x7fb4d40b7d50 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc0047e0 con 0x7fb4d40b7d50 2024-01-16T15:08:01.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d41f6630 con 0x7fb4d40b7d50 2024-01-16T15:08:01.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.400+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d40c9040 con 0x7fb4d40b7d50 2024-01-16T15:08:01.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.401+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4d40c9200 con 0x7fb4d40b7d50 2024-01-16T15:08:01.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.403+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fb4cc004940 con 0x7fb4d40b7d50 2024-01-16T15:08:01.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.404+0000 7fb4cbe00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb4ac06bbc0 0x7fb4ac06e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:01.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.404+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fb4cc08a370 con 0x7fb4d40b7d50 2024-01-16T15:08:01.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.404+0000 7fb4d3e00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb4ac06bbc0 0x7fb4ac06e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:01.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.405+0000 7fb4d3e00640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb4ac06bbc0 0x7fb4ac06e080 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40b7470 tx=0x7fb4c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:01.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.408+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4cc054930 con 0x7fb4d40b7d50 2024-01-16T15:08:01.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:01.607+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb4d41f61b0 con 0x7fb4d40b7d50 2024-01-16T15:08:02.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.269+0000 7fb4cbe00640 1 -- 172.21.15.53:0/1888058578 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v69) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb4d41f61b0 con 0x7fb4d40b7d50 2024-01-16T15:08:02.247 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:08:02.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb4ac06bbc0 msgr2=0x7fb4ac06e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:02.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb4ac06bbc0 0x7fb4ac06e080 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fb4d40b7470 tx=0x7fb4c4008040 comp rx=0 tx=0).stop 2024-01-16T15:08:02.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 msgr2=0x7fb4d40b4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:02.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 0x7fb4d40b4510 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fb4d408b3c0 tx=0x7fb4cc00dc00 comp rx=0 tx=0).stop 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 shutdown_connections 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb4ac06bbc0 0x7fb4ac06e080 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb4d40b7d50 0x7fb4d40b4510 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb4d41f6e30 0x7fb4d40b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 --2- 172.21.15.53:0/1888058578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb4b40047c0 0x7fb4d415bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.273+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 >> 172.21.15.53:0/1888058578 conn(0x7fb4d40d30b0 msgr2=0x7fb4d406c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.274+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 shutdown_connections 2024-01-16T15:08:02.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.274+0000 7fb4d9600640 1 -- 172.21.15.53:0/1888058578 wait complete. 2024-01-16T15:08:02.276 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:08:02.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.518+0000 7f9fd0c00640 1 Processor -- start 2024-01-16T15:08:02.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.518+0000 7f9fd0c00640 1 -- start start 2024-01-16T15:08:02.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.520+0000 7f9fd0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9fcc0702e0 0x7f9fcc0706c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.520+0000 7f9fd0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9fcc15c0c0 0x7f9fcc0b2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.520+0000 7f9fd0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fcc0b3790 con 0x7f9fcc161ec0 2024-01-16T15:08:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.520+0000 7f9fd0c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fcc070c00 con 0x7f9fcc0702e0 2024-01-16T15:08:02.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.520+0000 7f9fd0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fcc0b2b80 con 0x7f9fcc15c0c0 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9fcc0702e0 0x7f9fcc0706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9fcc0702e0 0x7f9fcc0706c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45348/0 (socket says 172.21.15.53:45348) 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fca800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9fcc161ec0 0x7f9fcc1653d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50750/0 (socket says 172.21.15.53:50750) 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 -- 172.21.15.53:0/1135937646 learned_addr learned my addr 172.21.15.53:0/1135937646 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9400640 1 --2- 172.21.15.53:0/1135937646 >> v2:172.21.15.146:3300/0 conn(0x7f9fcc15c0c0 0x7f9fcc0b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 -- 172.21.15.53:0/1135937646 >> v2:172.21.15.146:3300/0 conn(0x7f9fcc15c0c0 msgr2=0x7f9fcc0b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 --2- 172.21.15.53:0/1135937646 >> v2:172.21.15.146:3300/0 conn(0x7f9fcc15c0c0 0x7f9fcc0b2280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc8a00640 1 -- 172.21.15.53:0/1135937646 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 625512166 0 0) 0x7f9fcc0b3790 con 0x7f9fcc161ec0 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 -- 172.21.15.53:0/1135937646 >> v1:172.21.15.53:6789/0 conn(0x7f9fcc161ec0 legacy=0x7f9fcc1653d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:02.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.521+0000 7f9fc9e00640 1 -- 172.21.15.53:0/1135937646 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fcc0b2e60 con 0x7f9fcc0702e0 2024-01-16T15:08:02.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.523+0000 7f9fc9e00640 1 --2- 172.21.15.53:0/1135937646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc0706c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc06e730 tx=0x7f9fbc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcba0b5441aad6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:02.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.523+0000 7f9fc8a00640 1 -- 172.21.15.53:0/1135937646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc030860 con 0x7f9fcc0702e0 2024-01-16T15:08:02.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.523+0000 7f9fc8a00640 1 -- 172.21.15.53:0/1135937646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fbc0309c0 con 0x7f9fcc0702e0 2024-01-16T15:08:02.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.523+0000 7f9fc8a00640 1 -- 172.21.15.53:0/1135937646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fbc030d50 con 0x7f9fcc0702e0 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 -- 172.21.15.53:0/1135937646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc0702e0 msgr2=0x7f9fcc0706c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/1135937646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc0706c0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f9fcc06e730 tx=0x7f9fbc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 -- 172.21.15.53:0/1135937646 shutdown_connections 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/1135937646 >> v2:172.21.15.146:3300/0 conn(0x7f9fcc15c0c0 0x7f9fcc0b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/1135937646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc0706c0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 -- 172.21.15.53:0/1135937646 >> 172.21.15.53:0/1135937646 conn(0x7f9fcc0b0be0 msgr2=0x7f9fcc0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:02.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 -- 172.21.15.53:0/1135937646 shutdown_connections 2024-01-16T15:08:02.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.524+0000 7f9fd0c00640 1 -- 172.21.15.53:0/1135937646 wait complete. 2024-01-16T15:08:02.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.526+0000 7f9fd0c00640 1 Processor -- start 2024-01-16T15:08:02.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.526+0000 7f9fd0c00640 1 -- start start 2024-01-16T15:08:02.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fd0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:02.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fd0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fcc15c0c0 0x7f9fcc164d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:02.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fd0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 0x7f9fcc15e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:02.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fd0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc1648e0 con 0x7f9fcc161ec0 2024-01-16T15:08:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fd0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc15de00 con 0x7f9fcc0702e0 2024-01-16T15:08:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fd0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fcc163fd0 con 0x7f9fcc15c0c0 2024-01-16T15:08:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 0x7f9fcc15e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 0x7f9fcc15e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45356/0 (socket says 172.21.15.53:45356) 2024-01-16T15:08:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fca800640 1 -- 172.21.15.53:0/2526360776 learned_addr learned my addr 172.21.15.53:0/2526360776 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:02.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fc9e00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.527+0000 7f9fca800640 1 -- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fcc15c0c0 msgr2=0x7f9fcc164d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fca800640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fcc15c0c0 0x7f9fcc164d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fca800640 1 -- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fcc0702e0 msgr2=0x7f9fcc156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fca800640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fca800640 1 -- 172.21.15.53:0/2526360776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0094d0 con 0x7f9fcc161ec0 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fca800640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 0x7f9fcc15e560 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400c7b0 tx=0x7f9fc400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc400a5e0 con 0x7f9fcc161ec0 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fc400a760 con 0x7f9fcc161ec0 2024-01-16T15:08:02.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fc400ab10 con 0x7f9fcc161ec0 2024-01-16T15:08:02.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fcc157290 con 0x7f9fcc161ec0 2024-01-16T15:08:02.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.528+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fcc164220 con 0x7f9fcc161ec0 2024-01-16T15:08:02.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.530+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fcc0cb730 con 0x7f9fcc161ec0 2024-01-16T15:08:02.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.530+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f9fc400ac70 con 0x7f9fcc161ec0 2024-01-16T15:08:02.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.531+0000 7f9fc2000640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fa406bbc0 0x7f9fa406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:02.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.531+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5255+0+0 (secure 0 0 0) 0x7f9fc4087820 con 0x7f9fcc161ec0 2024-01-16T15:08:02.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.532+0000 7f9fc9e00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fa406bbc0 0x7f9fa406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:02.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.533+0000 7f9fc9e00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fa406bbc0 0x7f9fa406e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc036000 tx=0x7f9fbc008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:02.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.537+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9fc4054140 con 0x7f9fcc161ec0 2024-01-16T15:08:02.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:02.731+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f9fcc06e120 con 0x7f9fcc161ec0 2024-01-16T15:08:03.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.282+0000 7f9fc2000640 1 -- 172.21.15.53:0/2526360776 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v70) v1 ==== 170+0+0 (secure 0 0 0) 0x7f9fc40180d0 con 0x7f9fcc161ec0 2024-01-16T15:08:03.260 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:08:03.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fa406bbc0 msgr2=0x7f9fa406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fa406bbc0 0x7f9fa406e080 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9fbc036000 tx=0x7f9fbc008030 comp rx=0 tx=0).stop 2024-01-16T15:08:03.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 msgr2=0x7f9fcc15e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 0x7f9fcc15e560 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f9fc400c7b0 tx=0x7f9fc400cc70 comp rx=0 tx=0).stop 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 shutdown_connections 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9fa406bbc0 0x7f9fa406e080 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9fcc161ec0 0x7f9fcc15e560 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9fcc15c0c0 0x7f9fcc164d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 --2- 172.21.15.53:0/2526360776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9fcc0702e0 0x7f9fcc156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.286+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 >> 172.21.15.53:0/2526360776 conn(0x7f9fcc0b0be0 msgr2=0x7f9fcc06c770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:03.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.287+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 shutdown_connections 2024-01-16T15:08:03.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.287+0000 7f9fd0c00640 1 -- 172.21.15.53:0/2526360776 wait complete. 2024-01-16T15:08:03.298 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:08:03.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.515+0000 7ff8d8600640 1 Processor -- start 2024-01-16T15:08:03.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.515+0000 7ff8d8600640 1 -- start start 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d8600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d00cf9a0 0x7ff8d015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d8600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff8d015e1e0 0x7ff8d00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d8600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8d00b1ee0 con 0x7ff8d0143ae0 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d8600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8d00b2780 con 0x7ff8d015e1e0 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d8600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d015e990 con 0x7ff8d00cf9a0 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d5c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff8d0143ae0 0x7ff8d00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50754/0 (socket says 172.21.15.53:50754) 2024-01-16T15:08:03.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d5c00640 1 -- 172.21.15.53:0/1060300657 learned_addr learned my addr 172.21.15.53:0/1060300657 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:03.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d6600640 1 --2- 172.21.15.53:0/1060300657 >> v2:172.21.15.53:3300/0 conn(0x7ff8d015e1e0 0x7ff8d00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.517+0000 7ff8d5200640 1 --2- 172.21.15.53:0/1060300657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d00cf9a0 0x7ff8d015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.518+0000 7ff8d6600640 1 -- 172.21.15.53:0/1060300657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d00cf9a0 msgr2=0x7ff8d015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:03.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.518+0000 7ff8cfe00640 1 -- 172.21.15.53:0/1060300657 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912940539 0 0) 0x7ff8d00b1ee0 con 0x7ff8d0143ae0 2024-01-16T15:08:03.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.518+0000 7ff8d6600640 1 --2- 172.21.15.53:0/1060300657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d00cf9a0 0x7ff8d015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.518+0000 7ff8d6600640 1 -- 172.21.15.53:0/1060300657 >> v1:172.21.15.53:6789/0 conn(0x7ff8d0143ae0 legacy=0x7ff8d00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.518+0000 7ff8d6600640 1 -- 172.21.15.53:0/1060300657 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8d00b2130 con 0x7ff8d015e1e0 2024-01-16T15:08:03.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.519+0000 7ff8d6600640 1 --2- 172.21.15.53:0/1060300657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d00702c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80097a0 tx=0x7ff8c802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319050732ce2d061 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:03.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.519+0000 7ff8cfe00640 1 -- 172.21.15.53:0/1060300657 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8031830 con 0x7ff8d015e1e0 2024-01-16T15:08:03.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.519+0000 7ff8cfe00640 1 -- 172.21.15.53:0/1060300657 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c8031990 con 0x7ff8d015e1e0 2024-01-16T15:08:03.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.519+0000 7ff8cfe00640 1 -- 172.21.15.53:0/1060300657 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8031d20 con 0x7ff8d015e1e0 2024-01-16T15:08:03.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 -- 172.21.15.53:0/1060300657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 msgr2=0x7ff8d00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1060300657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d00702c0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff8c80097a0 tx=0x7ff8c802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 -- 172.21.15.53:0/1060300657 shutdown_connections 2024-01-16T15:08:03.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1060300657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d00702c0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1060300657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d00cf9a0 0x7ff8d015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 -- 172.21.15.53:0/1060300657 >> 172.21.15.53:0/1060300657 conn(0x7ff8d006a800 msgr2=0x7ff8d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:03.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 -- 172.21.15.53:0/1060300657 shutdown_connections 2024-01-16T15:08:03.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.520+0000 7ff8d8600640 1 -- 172.21.15.53:0/1060300657 wait complete. 2024-01-16T15:08:03.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.522+0000 7ff8d8600640 1 Processor -- start 2024-01-16T15:08:03.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.522+0000 7ff8d8600640 1 -- start start 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d8600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d00cf9a0 0x7ff8d013c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d8600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d0143ae0 0x7ff8d0139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d8600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d01590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d8600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d013caf0 con 0x7ff8d015e1e0 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d8600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d013ae40 con 0x7ff8d00cf9a0 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d8600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8d013b120 con 0x7ff8d0143ae0 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d01590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d01590d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45370/0 (socket says 172.21.15.53:45370) 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 -- 172.21.15.53:0/1574580592 learned_addr learned my addr 172.21.15.53:0/1574580592 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 -- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d0143ae0 msgr2=0x7ff8d0139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d0143ae0 0x7ff8d0139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 -- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d00cf9a0 msgr2=0x7ff8d013c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:03.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d00cf9a0 0x7ff8d013c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:03.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.523+0000 7ff8d6600640 1 -- 172.21.15.53:0/1574580592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8c8009400 con 0x7ff8d015e1e0 2024-01-16T15:08:03.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.524+0000 7ff8d6600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d01590d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8034040 tx=0x7ff8c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:03.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.524+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c8004230 con 0x7ff8d015e1e0 2024-01-16T15:08:03.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.524+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8c8004390 con 0x7ff8d015e1e0 2024-01-16T15:08:03.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.524+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8c803a870 con 0x7ff8d015e1e0 2024-01-16T15:08:03.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.524+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8d0139030 con 0x7ff8d015e1e0 2024-01-16T15:08:03.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.524+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8d01395f0 con 0x7ff8d015e1e0 2024-01-16T15:08:03.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.525+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8d00675d0 con 0x7ff8d015e1e0 2024-01-16T15:08:03.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.527+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7ff8c803a9d0 con 0x7ff8d015e1e0 2024-01-16T15:08:03.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.527+0000 7ff8cd600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff8a806bbc0 0x7ff8a806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:03.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.528+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 5637+0+0 (secure 0 0 0) 0x7ff8c80aa710 con 0x7ff8d015e1e0 2024-01-16T15:08:03.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.528+0000 7ff8d5c00640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff8a806bbc0 0x7ff8a806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:03.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.529+0000 7ff8d5c00640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff8a806bbc0 0x7ff8a806e080 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0001560 tx=0x7ff8c0007ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:03.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.533+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff8c8076fd0 con 0x7ff8d015e1e0 2024-01-16T15:08:03.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:03.762+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff8d01397b0 con 0x7ff8d015e1e0 2024-01-16T15:08:04.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.305+0000 7ff8cd600640 1 -- 172.21.15.53:0/1574580592 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v41) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff8c807a5f0 con 0x7ff8d015e1e0 2024-01-16T15:08:04.282 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 10 and data pool 11 2024-01-16T15:08:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.313+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff8a806bbc0 msgr2=0x7ff8a806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.313+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff8a806bbc0 0x7ff8a806e080 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0001560 tx=0x7ff8c0007ca0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 msgr2=0x7ff8d01590d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d01590d0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff8c8034040 tx=0x7ff8c80046c0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 shutdown_connections 2024-01-16T15:08:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff8a806bbc0 0x7ff8a806e080 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff8c0001560 tx=0x7ff8c0007ca0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8d015e1e0 0x7ff8d01590d0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8d0143ae0 0x7ff8d0139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 --2- 172.21.15.53:0/1574580592 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8d00cf9a0 0x7ff8d013c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.314+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 >> 172.21.15.53:0/1574580592 conn(0x7ff8d006a800 msgr2=0x7ff8d006cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.315+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 shutdown_connections 2024-01-16T15:08:04.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.315+0000 7ff8d8600640 1 -- 172.21.15.53:0/1574580592 wait complete. 2024-01-16T15:08:04.320 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:08:04.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.519+0000 7f1f32800640 1 Processor -- start 2024-01-16T15:08:04.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.520+0000 7f1f32800640 1 -- start start 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.521+0000 7f1f32800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1f2c0cc500 0x7f1f2c0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.521+0000 7f1f32800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1f2c06d160 0x7f1f2c06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.521+0000 7f1f32800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f2c070e50 con 0x7f1f2c143c40 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.521+0000 7f1f32800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c0723c0 con 0x7f1f2c06d160 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.521+0000 7f1f32800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2c0cced0 con 0x7f1f2c0cc500 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.521+0000 7f1f31e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1f2c06d160 0x7f1f2c06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f31e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1f2c06d160 0x7f1f2c06d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45382/0 (socket says 172.21.15.53:45382) 2024-01-16T15:08:04.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f31e00640 1 -- 172.21.15.53:0/2051975285 learned_addr learned my addr 172.21.15.53:0/2051975285 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:04.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f30a00640 1 --2- 172.21.15.53:0/2051975285 >> v2:172.21.15.146:3300/0 conn(0x7f1f2c0cc500 0x7f1f2c0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f31e00640 1 -- 172.21.15.53:0/2051975285 >> v2:172.21.15.146:3300/0 conn(0x7f1f2c0cc500 msgr2=0x7f1f2c0cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:04.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f31e00640 1 --2- 172.21.15.53:0/2051975285 >> v2:172.21.15.146:3300/0 conn(0x7f1f2c0cc500 0x7f1f2c0cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f31e00640 1 -- 172.21.15.53:0/2051975285 >> v1:172.21.15.146:6789/0 conn(0x7f1f2c143c40 legacy=0x7f1f2c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:04.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.522+0000 7f1f31e00640 1 -- 172.21.15.53:0/2051975285 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c144020 con 0x7f1f2c06d160 2024-01-16T15:08:04.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.523+0000 7f1f31e00640 1 --2- 172.21.15.53:0/2051975285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c06d160 0x7f1f2c06d560 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0b44a0 tx=0x7f1f2402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=242315486608a72d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:04.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f2be00640 1 -- 172.21.15.53:0/2051975285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2402c9f0 con 0x7f1f2c06d160 2024-01-16T15:08:04.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f2be00640 1 -- 172.21.15.53:0/2051975285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2402cb50 con 0x7f1f2c06d160 2024-01-16T15:08:04.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f2be00640 1 -- 172.21.15.53:0/2051975285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f240343f0 con 0x7f1f2c06d160 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f32800640 1 -- 172.21.15.53:0/2051975285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c06d160 msgr2=0x7f1f2c06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f32800640 1 --2- 172.21.15.53:0/2051975285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c06d160 0x7f1f2c06d560 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0b44a0 tx=0x7f1f2402be50 comp rx=0 tx=0).stop 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f32800640 1 -- 172.21.15.53:0/2051975285 shutdown_connections 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f32800640 1 --2- 172.21.15.53:0/2051975285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c06d160 0x7f1f2c06d560 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f32800640 1 --2- 172.21.15.53:0/2051975285 >> v2:172.21.15.146:3300/0 conn(0x7f1f2c0cc500 0x7f1f2c0cc990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.524+0000 7f1f32800640 1 -- 172.21.15.53:0/2051975285 >> 172.21.15.53:0/2051975285 conn(0x7f1f2c06a800 msgr2=0x7f1f2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.525+0000 7f1f32800640 1 -- 172.21.15.53:0/2051975285 shutdown_connections 2024-01-16T15:08:04.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.525+0000 7f1f32800640 1 -- 172.21.15.53:0/2051975285 wait complete. 2024-01-16T15:08:04.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.526+0000 7f1f32800640 1 Processor -- start 2024-01-16T15:08:04.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.526+0000 7f1f32800640 1 -- start start 2024-01-16T15:08:04.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.527+0000 7f1f32800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2c06d160 0x7f1f2c13d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.527+0000 7f1f32800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f2c0cc500 0x7f1f2c0acad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.527+0000 7f1f32800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 0x7f1f2c0ad010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.527+0000 7f1f32800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c13dce0 con 0x7f1f2c143c40 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.527+0000 7f1f32800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0ad620 con 0x7f1f2c0cc500 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.527+0000 7f1f32800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2c0ad900 con 0x7f1f2c06d160 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 0x7f1f2c0ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 0x7f1f2c0ad010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45396/0 (socket says 172.21.15.53:45396) 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 -- 172.21.15.53:0/1819905398 learned_addr learned my addr 172.21.15.53:0/1819905398 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31400640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2c06d160 0x7f1f2c13d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 -- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2c06d160 msgr2=0x7f1f2c13d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2c06d160 0x7f1f2c13d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 -- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f2c0cc500 msgr2=0x7f1f2c0acad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f30a00640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f2c0cc500 0x7f1f2c0acad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f2c0cc500 0x7f1f2c0acad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 -- 172.21.15.53:0/1819905398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f24009b30 con 0x7f1f2c143c40 2024-01-16T15:08:04.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.528+0000 7f1f31e00640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 0x7f1f2c0ad010 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1f24036040 tx=0x7f1f24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:04.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.529+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24009510 con 0x7f1f2c143c40 2024-01-16T15:08:04.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.529+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2403e450 con 0x7f1f2c143c40 2024-01-16T15:08:04.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.529+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2403e7e0 con 0x7f1f2c143c40 2024-01-16T15:08:04.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.529+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2c0adbe0 con 0x7f1f2c143c40 2024-01-16T15:08:04.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.529+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2c0ae1a0 con 0x7f1f2c143c40 2024-01-16T15:08:04.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.530+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2c067450 con 0x7f1f2c143c40 2024-01-16T15:08:04.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.531+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f1f2403e940 con 0x7f1f2c143c40 2024-01-16T15:08:04.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.532+0000 7f1f29600640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1f0406bbc0 0x7f1f0406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.532+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1f240ab280 con 0x7f1f2c143c40 2024-01-16T15:08:04.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.533+0000 7f1f31400640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1f0406bbc0 0x7f1f0406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.534+0000 7f1f31400640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1f0406bbc0 0x7f1f0406e080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00a750 tx=0x7f1f1c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:04.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.537+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1f24077b40 con 0x7f1f2c143c40 2024-01-16T15:08:04.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.732+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f2c15aec0 con 0x7f1f2c143c40 2024-01-16T15:08:04.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.735+0000 7f1f29600640 1 -- 172.21.15.53:0/1819905398 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f1f24043090 con 0x7f1f2c143c40 2024-01-16T15:08:04.713 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:04.713 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":71,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:08:03.764091+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:08:01.660404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:08:02.783705+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:08:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.738+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1f0406bbc0 msgr2=0x7f1f0406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.738+0000 7f1f32800640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1f0406bbc0 0x7f1f0406e080 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c00a750 tx=0x7f1f1c008040 comp rx=0 tx=0).stop 2024-01-16T15:08:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.738+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 msgr2=0x7f1f2c0ad010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.738+0000 7f1f32800640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 0x7f1f2c0ad010 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1f24036040 tx=0x7f1f24004a40 comp rx=0 tx=0).stop 2024-01-16T15:08:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 shutdown_connections 2024-01-16T15:08:04.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1f0406bbc0 0x7f1f0406e080 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1f2c143c40 0x7f1f2c0ad010 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1f2c0cc500 0x7f1f2c0acad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 --2- 172.21.15.53:0/1819905398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1f2c06d160 0x7f1f2c13d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 >> 172.21.15.53:0/1819905398 conn(0x7f1f2c06a800 msgr2=0x7f1f2c071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 shutdown_connections 2024-01-16T15:08:04.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.739+0000 7f1f32800640 1 -- 172.21.15.53:0/1819905398 wait complete. 2024-01-16T15:08:04.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:08:04.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.976+0000 7f8893800640 1 Processor -- start 2024-01-16T15:08:04.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.976+0000 7f8893800640 1 -- start start 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.978+0000 7f8893800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f888c144c60 0x7f888c0cc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.978+0000 7f8893800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f888c06a3e0 0x7f888c06a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.978+0000 7f8893800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888c071a20 con 0x7f888c0cf780 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.978+0000 7f8893800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c06adb0 con 0x7f888c144c60 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.978+0000 7f8893800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888c06eef0 con 0x7f888c06a3e0 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f888be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f888c0cf780 0x7f888c06e7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50756/0 (socket says 172.21.15.53:50756) 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f888be00640 1 -- 172.21.15.53:0/3541420059 learned_addr learned my addr 172.21.15.53:0/3541420059 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:04.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f8890e00640 1 --2- 172.21.15.53:0/3541420059 >> v2:172.21.15.53:3300/0 conn(0x7f888c144c60 0x7f888c0cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f8891800640 1 --2- 172.21.15.53:0/3541420059 >> v2:172.21.15.146:3300/0 conn(0x7f888c06a3e0 0x7f888c06a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f888b400640 1 -- 172.21.15.53:0/3541420059 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1350171471 0 0) 0x7f888c071a20 con 0x7f888c0cf780 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f8890e00640 1 -- 172.21.15.53:0/3541420059 >> v2:172.21.15.146:3300/0 conn(0x7f888c06a3e0 msgr2=0x7f888c06a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f8890e00640 1 --2- 172.21.15.53:0/3541420059 >> v2:172.21.15.146:3300/0 conn(0x7f888c06a3e0 0x7f888c06a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f8890e00640 1 -- 172.21.15.53:0/3541420059 >> v1:172.21.15.53:6789/0 conn(0x7f888c0cf780 legacy=0x7f888c06e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.979+0000 7f8890e00640 1 -- 172.21.15.53:0/3541420059 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888c0ccfd0 con 0x7f888c144c60 2024-01-16T15:08:04.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.980+0000 7f8890e00640 1 --2- 172.21.15.53:0/3541420059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c144c60 0x7f888c0cc940 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f888c06d040 tx=0x7f887402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47dd253eeffd5186 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:04.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f888b400640 1 -- 172.21.15.53:0/3541420059 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874031830 con 0x7f888c144c60 2024-01-16T15:08:04.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f888b400640 1 -- 172.21.15.53:0/3541420059 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8874031990 con 0x7f888c144c60 2024-01-16T15:08:04.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f888b400640 1 -- 172.21.15.53:0/3541420059 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8874031d20 con 0x7f888c144c60 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f8893800640 1 -- 172.21.15.53:0/3541420059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c144c60 msgr2=0x7f888c0cc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f8893800640 1 --2- 172.21.15.53:0/3541420059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c144c60 0x7f888c0cc940 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f888c06d040 tx=0x7f887402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f8893800640 1 -- 172.21.15.53:0/3541420059 shutdown_connections 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f8893800640 1 --2- 172.21.15.53:0/3541420059 >> v2:172.21.15.146:3300/0 conn(0x7f888c06a3e0 0x7f888c06a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f8893800640 1 --2- 172.21.15.53:0/3541420059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c144c60 0x7f888c0cc940 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.981+0000 7f8893800640 1 -- 172.21.15.53:0/3541420059 >> 172.21.15.53:0/3541420059 conn(0x7f888c0cfef0 msgr2=0x7f888c0d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.982+0000 7f8893800640 1 -- 172.21.15.53:0/3541420059 shutdown_connections 2024-01-16T15:08:04.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.982+0000 7f8893800640 1 -- 172.21.15.53:0/3541420059 wait complete. 2024-01-16T15:08:04.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.983+0000 7f8893800640 1 Processor -- start 2024-01-16T15:08:04.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.983+0000 7f8893800640 1 -- start start 2024-01-16T15:08:04.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.984+0000 7f8893800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c06a3e0 0x7f888c0b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.984+0000 7f8893800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 0x7f888c15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.984+0000 7f8893800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c144c60 0x7f888c0cb350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.984+0000 7f8893800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0cabe0 con 0x7f888c0cf780 2024-01-16T15:08:04.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.984+0000 7f8893800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0b35a0 con 0x7f888c06a3e0 2024-01-16T15:08:04.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.984+0000 7f8893800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888c0b3840 con 0x7f888c144c60 2024-01-16T15:08:04.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f888be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 0x7f888c15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f888be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 0x7f888c15c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45406/0 (socket says 172.21.15.53:45406) 2024-01-16T15:08:04.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f888be00640 1 -- 172.21.15.53:0/2475179692 learned_addr learned my addr 172.21.15.53:0/2475179692 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:04.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f8890e00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c06a3e0 0x7f888c0b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f8891800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c144c60 0x7f888c0cb350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f888be00640 1 -- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c144c60 msgr2=0x7f888c0cb350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.985+0000 7f888be00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c144c60 0x7f888c0cb350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f888be00640 1 -- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c06a3e0 msgr2=0x7f888c0b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f888be00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c06a3e0 0x7f888c0b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f888be00640 1 -- 172.21.15.53:0/2475179692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8874009400 con 0x7f888c0cf780 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f888be00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 0x7f888c15c260 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c890 tx=0x7f887c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00a640 con 0x7f888c0cf780 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f887c00a7c0 con 0x7f888c0cf780 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f887c00ab70 con 0x7f888c0cf780 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888c0b3b20 con 0x7f888c0cf780 2024-01-16T15:08:04.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.986+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c0cae30 con 0x7f888c0cf780 2024-01-16T15:08:04.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.988+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888c0cb050 con 0x7f888c0cf780 2024-01-16T15:08:04.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.989+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f887c00acd0 con 0x7f888c0cf780 2024-01-16T15:08:04.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.989+0000 7f8888c00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bbc0 0x7f886406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:04.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.990+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f887c0879c0 con 0x7f888c0cf780 2024-01-16T15:08:04.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.990+0000 7f8890e00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bbc0 0x7f886406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:04.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.991+0000 7f8890e00640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bbc0 0x7f886406e080 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8874036000 tx=0x7f8874008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:04.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:04.994+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f887c0542e0 con 0x7f888c0cf780 2024-01-16T15:08:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.190+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f888c0675d0 con 0x7f888c0cf780 2024-01-16T15:08:05.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.193+0000 7f8888c00640 1 -- 172.21.15.53:0/2475179692 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16588 (secure 0 0 0) 0x7f887c057900 con 0x7f888c0cf780 2024-01-16T15:08:05.170 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:05.171 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":71,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:08:03.764091+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:08:01.660404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:08:02.783705+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bbc0 msgr2=0x7f886406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bbc0 0x7f886406e080 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f8874036000 tx=0x7f8874008030 comp rx=0 tx=0).stop 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 msgr2=0x7f888c15c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 0x7f888c15c260 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f887c00c890 tx=0x7f887c00cd50 comp rx=0 tx=0).stop 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 shutdown_connections 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f886406bbc0 0x7f886406e080 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f888c144c60 0x7f888c0cb350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f888c0cf780 0x7f888c15c260 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 --2- 172.21.15.53:0/2475179692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f888c06a3e0 0x7f888c0b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.195+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 >> 172.21.15.53:0/2475179692 conn(0x7f888c0cfef0 msgr2=0x7f888c06e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.196+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 shutdown_connections 2024-01-16T15:08:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.196+0000 7f8893800640 1 -- 172.21.15.53:0/2475179692 wait complete. 2024-01-16T15:08:05.191 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:08:05.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.443+0000 7f72f2e00640 1 Processor -- start 2024-01-16T15:08:05.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.443+0000 7f72f2e00640 1 -- start start 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72f2e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 0x7f72ec15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72f2e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f72ec0cf9a0 0x7f72ec0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72f2e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ec06c470 con 0x7f72ec13e360 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72f2e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72ec06c750 con 0x7f72ec0cf9a0 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72f2e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec06ca30 con 0x7f72ec13df80 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72eb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f72ec13e360 0x7f72ec0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50766/0 (socket says 172.21.15.53:50766) 2024-01-16T15:08:05.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.445+0000 7f72eb400640 1 -- 172.21.15.53:0/2161157554 learned_addr learned my addr 172.21.15.53:0/2161157554 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:05.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72f0e00640 1 --2- 172.21.15.53:0/2161157554 >> v2:172.21.15.146:3300/0 conn(0x7f72ec0cf9a0 0x7f72ec0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72ebe00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 0x7f72ec15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054477163 0 0) 0x7f72ec06c470 con 0x7f72ec13e360 2024-01-16T15:08:05.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72cc003520 con 0x7f72ec13e360 2024-01-16T15:08:05.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3103864204 0 0) 0x7f72cc003520 con 0x7f72ec13e360 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 msgr2=0x7f72ec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 0x7f72ec15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 >> v2:172.21.15.146:3300/0 conn(0x7f72ec0cf9a0 msgr2=0x7f72ec0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 --2- 172.21.15.53:0/2161157554 >> v2:172.21.15.146:3300/0 conn(0x7f72ec0cf9a0 0x7f72ec0cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec06cd10 con 0x7f72ec13e360 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f72d4001670 con 0x7f72ec13e360 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 >> v1:172.21.15.53:6789/0 conn(0x7f72ec13e360 legacy=0x7f72ec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc003970 0x7f72cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72cc0047c0 0x7f72cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 0x7f72cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec06c750 con 0x7f72cc0047c0 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec06ca30 con 0x7f72cc003970 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.446+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc00b980 con 0x7f72ec13df80 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72eb400640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72cc0047c0 0x7f72cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72ebe00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc003970 0x7f72cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72eb400640 1 -- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 msgr2=0x7f72cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72eb400640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec13df80 0x7f72cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72eb400640 1 -- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc003970 msgr2=0x7f72cc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72eb400640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc003970 0x7f72cc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.447+0000 7f72eb400640 1 -- 172.21.15.53:0/2161157554 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc003520 con 0x7f72cc0047c0 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.448+0000 7f72eb400640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72cc0047c0 0x7f72cc004c40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c110 tx=0x7f72cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0e405bb90dc0175 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.448+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d4005ad0 con 0x7f72cc0047c0 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.448+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72d4005c50 con 0x7f72cc0047c0 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.448+0000 7f72eaa00640 1 -- 172.21.15.53:0/2161157554 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72d402c810 con 0x7f72cc0047c0 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f2e00640 1 -- 172.21.15.53:0/2161157554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72cc0047c0 msgr2=0x7f72cc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2161157554 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72cc0047c0 0x7f72cc004c40 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f72d400c110 tx=0x7f72cc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f0e00640 1 -- 172.21.15.53:0/2161157554 reap_dead start 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f2e00640 1 -- 172.21.15.53:0/2161157554 shutdown_connections 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f2e00640 1 -- 172.21.15.53:0/2161157554 >> 172.21.15.53:0/2161157554 conn(0x7f72ec06a800 msgr2=0x7f72ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f2e00640 1 -- 172.21.15.53:0/2161157554 shutdown_connections 2024-01-16T15:08:05.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.449+0000 7f72f2e00640 1 -- 172.21.15.53:0/2161157554 wait complete. 2024-01-16T15:08:05.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.452+0000 7f72f2e00640 1 Processor -- start 2024-01-16T15:08:05.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.452+0000 7f72f2e00640 1 -- start start 2024-01-16T15:08:05.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.453+0000 7f72f2e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc0047c0 0x7f72ec1583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.453+0000 7f72f2e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec0c3ec0 0x7f72ec159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.453+0000 7f72f2e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 0x7f72ec15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.453+0000 7f72f2e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec159ff0 con 0x7f72ec13d7f0 2024-01-16T15:08:05.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.453+0000 7f72f2e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec0c4340 con 0x7f72cc0047c0 2024-01-16T15:08:05.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.453+0000 7f72f2e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72ec158a10 con 0x7f72ec0c3ec0 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 0x7f72ec15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 0x7f72ec15d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45432/0 (socket says 172.21.15.53:45432) 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 -- 172.21.15.53:0/2991851202 learned_addr learned my addr 172.21.15.53:0/2991851202 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72ebe00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc0047c0 0x7f72ec1583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72eb400640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec0c3ec0 0x7f72ec159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 -- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec0c3ec0 msgr2=0x7f72ec159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec0c3ec0 0x7f72ec159ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 -- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc0047c0 msgr2=0x7f72ec1583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc0047c0 0x7f72ec1583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 -- 172.21.15.53:0/2991851202 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec158cf0 con 0x7f72ec13d7f0 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.454+0000 7f72f0e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 0x7f72ec15d8a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0cc770 tx=0x7f72e400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:05.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.455+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4010040 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.455+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e400bce0 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.455+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ec158fd0 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.455+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4014640 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.455+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ec159590 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.456+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ec0675d0 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.458+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f72e40147a0 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.458+0000 7f72e3e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72c406bc10 0x7f72c406e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.459+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f72e4088b00 con 0x7f72ec13d7f0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.459+0000 7f72ebe00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72c406bc10 0x7f72c406e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.460+0000 7f72ebe00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72c406bc10 0x7f72c406e0d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009930 tx=0x7f72dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:05.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.463+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f72e4055210 con 0x7f72ec13d7f0 2024-01-16T15:08:05.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:05.700+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f72ec159750 con 0x7f72ec13d7f0 2024-01-16T15:08:06.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.326+0000 7f72e3e00640 1 -- 172.21.15.53:0/2991851202 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v44) v1 ==== 118+0+0 (secure 0 0 0) 0x7f72e4058830 con 0x7f72ec13d7f0 2024-01-16T15:08:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.329+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72c406bc10 msgr2=0x7f72c406e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.329+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72c406bc10 0x7f72c406e0d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009930 tx=0x7f72dc008040 comp rx=0 tx=0).stop 2024-01-16T15:08:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.329+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 msgr2=0x7f72ec15d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:06.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.329+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 0x7f72ec15d8a0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0cc770 tx=0x7f72e400b480 comp rx=0 tx=0).stop 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 shutdown_connections 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f72c406bc10 0x7f72c406e0d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f72ec13d7f0 0x7f72ec15d8a0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f72ec0c3ec0 0x7f72ec159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 --2- 172.21.15.53:0/2991851202 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f72cc0047c0 0x7f72ec1583c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 >> 172.21.15.53:0/2991851202 conn(0x7f72ec06a800 msgr2=0x7f72ec0afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 shutdown_connections 2024-01-16T15:08:06.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.330+0000 7f72f2e00640 1 -- 172.21.15.53:0/2991851202 wait complete. 2024-01-16T15:08:06.330 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:08:06.330 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:08:06.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.577+0000 7f8f5d600640 1 Processor -- start 2024-01-16T15:08:06.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.578+0000 7f8f5d600640 1 -- start start 2024-01-16T15:08:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.579+0000 7f8f5d600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f5813fcb0 0x7f8f580cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.579+0000 7f8f5d600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f580ce440 con 0x7f8f5813cf10 2024-01-16T15:08:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f5d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f580ce720 con 0x7f8f580cdc20 2024-01-16T15:08:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f5d600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f580cea00 con 0x7f8f5813fcb0 2024-01-16T15:08:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f57e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8f5813cf10 0x7f8f5813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50774/0 (socket says 172.21.15.53:50774) 2024-01-16T15:08:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f57e00640 1 -- 172.21.15.53:0/3382549289 learned_addr learned my addr 172.21.15.53:0/3382549289 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f57400640 1 --2- 172.21.15.53:0/3382549289 >> v2:172.21.15.146:3300/0 conn(0x7f8f5813fcb0 0x7f8f580cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225358606 0 0) 0x7f8f580ce440 con 0x7f8f5813cf10 2024-01-16T15:08:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.580+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f38003520 con 0x7f8f5813cf10 2024-01-16T15:08:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1223187560 0 0) 0x7f8f38003520 con 0x7f8f5813cf10 2024-01-16T15:08:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 >> v2:172.21.15.146:3300/0 conn(0x7f8f5813fcb0 msgr2=0x7f8f580cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 --2- 172.21.15.53:0/3382549289 >> v2:172.21.15.146:3300/0 conn(0x7f8f5813fcb0 0x7f8f580cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 >> v1:172.21.15.146:6789/0 conn(0x7f8f580cdc20 legacy=0x7f8f580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f580b1d90 con 0x7f8f5813cf10 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8f48001670 con 0x7f8f5813cf10 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 >> v1:172.21.15.53:6789/0 conn(0x7f8f5813cf10 legacy=0x7f8f5813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f380039c0 0x7f8f38003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f380047c0 0x7f8f38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f38005180 0x7f8f3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f580cea00 con 0x7f8f380039c0 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3800c050 con 0x7f8f38005180 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f3800c210 con 0x7f8f380047c0 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.581+0000 7f8f57e00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f380039c0 0x7f8f38003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f57e00640 1 -- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f380047c0 msgr2=0x7f8f38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f57e00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f380047c0 0x7f8f38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f57e00640 1 -- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f38005180 msgr2=0x7f8f3800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f5cc00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f38005180 0x7f8f3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f57e00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f38005180 0x7f8f3800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f57e00640 1 -- 172.21.15.53:0/3382549289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f38003520 con 0x7f8f380039c0 2024-01-16T15:08:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f5cc00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f38005180 0x7f8f3800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.582+0000 7f8f5cc00640 1 -- 172.21.15.53:0/3382549289 reap_dead start 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.583+0000 7f8f57e00640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f380039c0 0x7f8f38003dc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c190 tx=0x7f8f380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab259da9b8977c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.583+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f48005ad0 con 0x7f8f380039c0 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.583+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f48005c30 con 0x7f8f380039c0 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.583+0000 7f8f56a00640 1 -- 172.21.15.53:0/3382549289 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4802c810 con 0x7f8f380039c0 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 -- 172.21.15.53:0/3382549289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f380039c0 msgr2=0x7f8f38003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f380039c0 0x7f8f38003dc0 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f8f4800c190 tx=0x7f8f380057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 -- 172.21.15.53:0/3382549289 shutdown_connections 2024-01-16T15:08:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 --2- 172.21.15.53:0/3382549289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f380039c0 0x7f8f38003dc0 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 -- 172.21.15.53:0/3382549289 >> 172.21.15.53:0/3382549289 conn(0x7f8f5806a7d0 msgr2=0x7f8f5806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 -- 172.21.15.53:0/3382549289 shutdown_connections 2024-01-16T15:08:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.584+0000 7f8f5d600640 1 -- 172.21.15.53:0/3382549289 wait complete. 2024-01-16T15:08:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.586+0000 7f8f5d600640 1 Processor -- start 2024-01-16T15:08:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.586+0000 7f8f5d600640 1 -- start start 2024-01-16T15:08:06.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380039c0 0x7f8f58138110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f580711b0 0x7f8f58152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 0x7f8f581527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f581581f0 con 0x7f8f58071630 2024-01-16T15:08:06.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f581f21c0 con 0x7f8f380039c0 2024-01-16T15:08:06.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f581f24a0 con 0x7f8f580711b0 2024-01-16T15:08:06.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 0x7f8f581527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 0x7f8f581527d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45446/0 (socket says 172.21.15.53:45446) 2024-01-16T15:08:06.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.587+0000 7f8f5cc00640 1 -- 172.21.15.53:0/1101069596 learned_addr learned my addr 172.21.15.53:0/1101069596 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:06.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f57400640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f580711b0 0x7f8f58152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f57e00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380039c0 0x7f8f58138110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5cc00640 1 -- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f580711b0 msgr2=0x7f8f58152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:06.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5cc00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f580711b0 0x7f8f58152290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5cc00640 1 -- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380039c0 msgr2=0x7f8f58138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5cc00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380039c0 0x7f8f58138110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5cc00640 1 -- 172.21.15.53:0/1101069596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f581f2780 con 0x7f8f58071630 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5cc00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 0x7f8f581527d0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f8f580ae9b0 tx=0x7f8f5000d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50006d50 con 0x7f8f58071630 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f50004980 con 0x7f8f58071630 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f581f2a60 con 0x7f8f58071630 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50004d70 con 0x7f8f58071630 2024-01-16T15:08:06.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.588+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f58138eb0 con 0x7f8f58071630 2024-01-16T15:08:06.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.589+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f58139040 con 0x7f8f58071630 2024-01-16T15:08:06.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.590+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f8f50012600 con 0x7f8f58071630 2024-01-16T15:08:06.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.591+0000 7f8f4fe00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f3006bbc0 0x7f8f3006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:06.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.591+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8f50087820 con 0x7f8f58071630 2024-01-16T15:08:06.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.592+0000 7f8f57e00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f3006bbc0 0x7f8f3006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.593+0000 7f8f57e00640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f3006bbc0 0x7f8f3006e080 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8f48034000 tx=0x7f8f48009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:06.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.596+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f50054050 con 0x7f8f58071630 2024-01-16T15:08:06.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:06.829+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8f580b2490 con 0x7f8f58071630 2024-01-16T15:08:07.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.339+0000 7f8f4fe00640 1 -- 172.21.15.53:0/1101069596 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v45) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8f50057670 con 0x7f8f58071630 2024-01-16T15:08:07.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f3006bbc0 msgr2=0x7f8f3006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f3006bbc0 0x7f8f3006e080 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8f48034000 tx=0x7f8f48009030 comp rx=0 tx=0).stop 2024-01-16T15:08:07.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 msgr2=0x7f8f581527d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 0x7f8f581527d0 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f8f580ae9b0 tx=0x7f8f5000d9d0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 shutdown_connections 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f3006bbc0 0x7f8f3006e080 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f58071630 0x7f8f581527d0 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f580711b0 0x7f8f58152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 --2- 172.21.15.53:0/1101069596 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f380039c0 0x7f8f58138110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 >> 172.21.15.53:0/1101069596 conn(0x7f8f5806a7d0 msgr2=0x7f8f5806cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 shutdown_connections 2024-01-16T15:08:07.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.343+0000 7f8f5d600640 1 -- 172.21.15.53:0/1101069596 wait complete. 2024-01-16T15:08:07.345 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:08:07.345 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:08:07.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.585+0000 7efd02800640 1 Processor -- start 2024-01-16T15:08:07.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.585+0000 7efd02800640 1 -- start start 2024-01-16T15:08:07.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.586+0000 7efd02800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.586+0000 7efd02800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcfc0717d0 0x7efcfc0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.586+0000 7efd02800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcfc070e50 con 0x7efcfc0cca40 2024-01-16T15:08:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.586+0000 7efd02800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcfc0723c0 con 0x7efcfc0717d0 2024-01-16T15:08:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.586+0000 7efd02800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc0cce70 con 0x7efcfc143c40 2024-01-16T15:08:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.587+0000 7efd00a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcfc0717d0 0x7efcfc0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.587+0000 7efd00a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7efcfc0717d0 0x7efcfc0cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:39246/0 (socket says 172.21.15.53:39246) 2024-01-16T15:08:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.587+0000 7efd01400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.587+0000 7efd00a00640 1 -- 172.21.15.53:0/1501664391 learned_addr learned my addr 172.21.15.53:0/1501664391 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.588+0000 7efd00a00640 1 -- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 msgr2=0x7efcfc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.588+0000 7efd00a00640 1 --2- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.588+0000 7efd00a00640 1 -- 172.21.15.53:0/1501664391 >> v1:172.21.15.146:6789/0 conn(0x7efcfc0cca40 legacy=0x7efcfc15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.588+0000 7efd00a00640 1 -- 172.21.15.53:0/1501664391 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcfc06e170 con 0x7efcfc0717d0 2024-01-16T15:08:07.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.590+0000 7efd00a00640 1 --2- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0717d0 0x7efcfc0cc500 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efce4009870 tx=0x7efce402b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8056c03d7dac3a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efcfbe00640 1 -- 172.21.15.53:0/1501664391 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4031640 con 0x7efcfc0717d0 2024-01-16T15:08:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efcfbe00640 1 -- 172.21.15.53:0/1501664391 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce40317a0 con 0x7efcfc0717d0 2024-01-16T15:08:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efcfbe00640 1 -- 172.21.15.53:0/1501664391 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4031b30 con 0x7efcfc0717d0 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efd02800640 1 -- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0717d0 msgr2=0x7efcfc0cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efd02800640 1 --2- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0717d0 0x7efcfc0cc500 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7efce4009870 tx=0x7efce402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efd02800640 1 -- 172.21.15.53:0/1501664391 shutdown_connections 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efd02800640 1 --2- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0717d0 0x7efcfc0cc500 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efd02800640 1 --2- 172.21.15.53:0/1501664391 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.591+0000 7efd02800640 1 -- 172.21.15.53:0/1501664391 >> 172.21.15.53:0/1501664391 conn(0x7efcfc06a800 msgr2=0x7efcfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.592+0000 7efd02800640 1 -- 172.21.15.53:0/1501664391 shutdown_connections 2024-01-16T15:08:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.592+0000 7efd02800640 1 -- 172.21.15.53:0/1501664391 wait complete. 2024-01-16T15:08:07.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.593+0000 7efd02800640 1 Processor -- start 2024-01-16T15:08:07.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.593+0000 7efd02800640 1 -- start start 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.594+0000 7efd02800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 0x7efcfc12fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.594+0000 7efd02800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0cca40 0x7efcfc0aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.594+0000 7efd02800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc0af500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.594+0000 7efd02800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc133500 con 0x7efcfc0717d0 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.594+0000 7efd02800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc12f890 con 0x7efcfc0cca40 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.594+0000 7efd02800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcfc1f70e0 con 0x7efcfc143c40 2024-01-16T15:08:07.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 0x7efcfc12fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:07.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 0x7efcfc12fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45456/0 (socket says 172.21.15.53:45456) 2024-01-16T15:08:07.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 -- 172.21.15.53:0/824476631 learned_addr learned my addr 172.21.15.53:0/824476631 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:07.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd00a00640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0cca40 0x7efcfc0aefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:07.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01e00640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc0af500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 -- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 msgr2=0x7efcfc0af500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc0af500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 -- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0cca40 msgr2=0x7efcfc0aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0cca40 0x7efcfc0aefc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.595+0000 7efd01400640 1 -- 172.21.15.53:0/824476631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce40094d0 con 0x7efcfc0717d0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.596+0000 7efd01400640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 0x7efcfc12fd20 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7efcec00e870 tx=0x7efcec00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.596+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec0094f0 con 0x7efcfc0717d0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.596+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcec00c940 con 0x7efcfc0717d0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.596+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcfc1f73c0 con 0x7efcfc0717d0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.596+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcec00ccf0 con 0x7efcfc0717d0 2024-01-16T15:08:07.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.596+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcfc12f520 con 0x7efcfc0717d0 2024-01-16T15:08:07.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.598+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcfc067450 con 0x7efcfc0717d0 2024-01-16T15:08:07.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.599+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7efcec013470 con 0x7efcfc0717d0 2024-01-16T15:08:07.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.600+0000 7efcf9600640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efcd406bbc0 0x7efcd406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:07.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.600+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efcec0887c0 con 0x7efcfc0717d0 2024-01-16T15:08:07.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.600+0000 7efd00a00640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efcd406bbc0 0x7efcd406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:07.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.602+0000 7efd00a00640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efcd406bbc0 0x7efcd406e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efce4036000 tx=0x7efce403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:07.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.604+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efcec054e90 con 0x7efcfc0717d0 2024-01-16T15:08:07.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.837+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcfc0ceb30 con 0x7efcfc0717d0 2024-01-16T15:08:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.838+0000 7efcf9600640 1 -- 172.21.15.53:0/824476631 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4918 (secure 0 0 0) 0x7efcec0584b0 con 0x7efcfc0717d0 2024-01-16T15:08:07.816 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:07.816 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1106083775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1106083775},{"type":"v1","addr":"172.21.15.53:6834","nonce":1106083775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2940561403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2940561403},{"type":"v1","addr":"172.21.15.146:6837","nonce":2940561403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7029,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3081217400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3081217400},{"type":"v1","addr":"172.21.15.53:6835","nonce":3081217400}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:08:03.764651+0000","modified":"2024-01-16T15:08:06.830496+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/867019165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":867019165},{"type":"v1","addr":"172.21.15.146:6836","nonce":867019165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:08:07.816 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 45 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efcd406bbc0 msgr2=0x7efcd406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efcd406bbc0 0x7efcd406e080 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7efce4036000 tx=0x7efce403c040 comp rx=0 tx=0).stop 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 msgr2=0x7efcfc12fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 0x7efcfc12fd20 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7efcec00e870 tx=0x7efcec00ed30 comp rx=0 tx=0).stop 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 shutdown_connections 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efcd406bbc0 0x7efcd406e080 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efcfc143c40 0x7efcfc0af500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efcfc0cca40 0x7efcfc0aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 --2- 172.21.15.53:0/824476631 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efcfc0717d0 0x7efcfc12fd20 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 >> 172.21.15.53:0/824476631 conn(0x7efcfc06a800 msgr2=0x7efcfc06d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 shutdown_connections 2024-01-16T15:08:07.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:07.841+0000 7efd02800640 1 -- 172.21.15.53:0/824476631 wait complete. 2024-01-16T15:08:07.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:08:08.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.083+0000 7f5bc6800640 1 Processor -- start 2024-01-16T15:08:08.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.084+0000 7f5bc6800640 1 -- start start 2024-01-16T15:08:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.085+0000 7f5bc6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 0x7f5bc00b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.085+0000 7f5bc6800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc006cea0 con 0x7f5bc006c6b0 2024-01-16T15:08:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.085+0000 7f5bc6800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc006cbc0 con 0x7f5bc00cd240 2024-01-16T15:08:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.085+0000 7f5bc6800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc006f350 con 0x7f5bc009a170 2024-01-16T15:08:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.086+0000 7f5bbfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5bc006c6b0 0x7f5bc00cf640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50788/0 (socket says 172.21.15.53:50788) 2024-01-16T15:08:08.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.086+0000 7f5bbfe00640 1 -- 172.21.15.53:0/32631499 learned_addr learned my addr 172.21.15.53:0/32631499 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.086+0000 7f5bb6800640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 0x7f5bc00b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.086+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464543118 0 0) 0x7f5bc006cea0 con 0x7f5bc006c6b0 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.086+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ba0003520 con 0x7f5bc006c6b0 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2367076642 0 0) 0x7f5ba0003520 con 0x7f5bc006c6b0 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 msgr2=0x7f5bc00b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 0x7f5bc00b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 >> v1:172.21.15.146:6789/0 conn(0x7f5bc00cd240 legacy=0x7f5bc0099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc006f630 con 0x7f5bc006c6b0 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5bb80019f0 con 0x7f5bc006c6b0 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 >> v1:172.21.15.53:6789/0 conn(0x7f5bc006c6b0 legacy=0x7f5bc00cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba00039a0 0x7f5ba0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ba00047c0 0x7f5ba0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 0x7f5ba0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc006f350 con 0x7f5ba00047c0 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba000ba30 con 0x7f5ba00039a0 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba000bbf0 con 0x7f5bc009a170 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bb6800640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ba00047c0 0x7f5ba0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bbf400640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba00039a0 0x7f5ba0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bb6800640 1 -- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 msgr2=0x7f5ba0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.087+0000 7f5bb6800640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bc009a170 0x7f5ba0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.088+0000 7f5bb6800640 1 -- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba00039a0 msgr2=0x7f5ba0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.088+0000 7f5bb6800640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ba00039a0 0x7f5ba0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.088+0000 7f5bb6800640 1 -- 172.21.15.53:0/32631499 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba0003520 con 0x7f5ba00047c0 2024-01-16T15:08:08.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.089+0000 7f5bb6800640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ba00047c0 0x7f5ba0004c40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8009a90 tx=0x7f5ba00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42188c67aaf2d6d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:08.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.089+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba802c6b0 con 0x7f5ba00047c0 2024-01-16T15:08:08.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.089+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba802c810 con 0x7f5ba00047c0 2024-01-16T15:08:08.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.089+0000 7f5bbea00640 1 -- 172.21.15.53:0/32631499 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba802cba0 con 0x7f5ba00047c0 2024-01-16T15:08:08.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bc6800640 1 -- 172.21.15.53:0/32631499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ba00047c0 msgr2=0x7f5ba0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bc6800640 1 --2- 172.21.15.53:0/32631499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ba00047c0 0x7f5ba0004c40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7f5ba8009a90 tx=0x7f5ba00057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bbfe00640 1 -- 172.21.15.53:0/32631499 reap_dead start 2024-01-16T15:08:08.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bc6800640 1 -- 172.21.15.53:0/32631499 shutdown_connections 2024-01-16T15:08:08.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bc6800640 1 -- 172.21.15.53:0/32631499 >> 172.21.15.53:0/32631499 conn(0x7f5bc00b2d00 msgr2=0x7f5bc00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:08.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bc6800640 1 -- 172.21.15.53:0/32631499 shutdown_connections 2024-01-16T15:08:08.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.090+0000 7f5bc6800640 1 -- 172.21.15.53:0/32631499 wait complete. 2024-01-16T15:08:08.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.092+0000 7f5bc6800640 1 Processor -- start 2024-01-16T15:08:08.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.092+0000 7f5bc6800640 1 -- start start 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bc6800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ba00047c0 0x7f5bc00c6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bc6800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc015b130 0x7f5bc00b1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bc6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 0x7f5bc00c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bc6800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00b4f30 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bc6800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00b1f20 con 0x7f5bc015b130 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bc6800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc00b4060 con 0x7f5ba00047c0 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 0x7f5bc00c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 0x7f5bc00c4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45482/0 (socket says 172.21.15.53:45482) 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 -- 172.21.15.53:0/4103866540 learned_addr learned my addr 172.21.15.53:0/4103866540 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bb6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc015b130 0x7f5bc00b1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbf400640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ba00047c0 0x7f5bc00c6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 -- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ba00047c0 msgr2=0x7f5bc00c6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ba00047c0 0x7f5bc00c6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 -- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc015b130 msgr2=0x7f5bc00b1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.093+0000 7f5bbfe00640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc015b130 0x7f5bc00b1560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bbfe00640 1 -- 172.21.15.53:0/4103866540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc00c70a0 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bbf400640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ba00047c0 0x7f5bc00c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bb6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc015b130 0x7f5bc00b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bbfe00640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 0x7f5bc00c4d20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00709e0 tx=0x7f5bb800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800ce50 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb800a610 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb800a9c0 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc00b42b0 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.094+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc00b4870 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.095+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc00675d0 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.097+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f5bb800ab20 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.098+0000 7f5bb7e00640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9806bbc0 0x7f5b9806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.098+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5bb80879e0 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.099+0000 7f5bbf400640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9806bbc0 0x7f5b9806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.100+0000 7f5bbf400640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9806bbc0 0x7f5b9806e080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5bb00016b0 tx=0x7f5bb0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:08.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.103+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bb8057d30 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.296+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5bc00cf670 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.299+0000 7f5bb7e00640 1 -- 172.21.15.53:0/4103866540 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16575 (secure 0 0 0) 0x7f5bb80578c0 con 0x7f5bc00b1aa0 2024-01-16T15:08:08.277 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:08.277 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":72,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:08:05.303274+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:08:01.660404+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2024-01-16T15:08:02.783705+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:08:08.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9806bbc0 msgr2=0x7f5b9806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9806bbc0 0x7f5b9806e080 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f5bb00016b0 tx=0x7f5bb0009040 comp rx=0 tx=0).stop 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 msgr2=0x7f5bc00c4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 0x7f5bc00c4d20 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f5bc00709e0 tx=0x7f5bb800cc00 comp rx=0 tx=0).stop 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 shutdown_connections 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9806bbc0 0x7f5b9806e080 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bc00b1aa0 0x7f5bc00c4d20 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bc015b130 0x7f5bc00b1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 --2- 172.21.15.53:0/4103866540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ba00047c0 0x7f5bc00c6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 >> 172.21.15.53:0/4103866540 conn(0x7f5bc00b2d00 msgr2=0x7f5bc01607c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 shutdown_connections 2024-01-16T15:08:08.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.302+0000 7f5bc6800640 1 -- 172.21.15.53:0/4103866540 wait complete. 2024-01-16T15:08:08.297 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:08:08.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:08:08.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.529+0000 7fccd8600640 1 Processor -- start 2024-01-16T15:08:08.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.530+0000 7fccd8600640 1 -- start start 2024-01-16T15:08:08.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.531+0000 7fccd8600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fccd00ae5a0 0x7fccd015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.531+0000 7fccd8600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 0x7fccd00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.531+0000 7fccd8600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd0070c40 con 0x7fccd00b3ae0 2024-01-16T15:08:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.531+0000 7fccd8600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd0071450 con 0x7fccd00ae5a0 2024-01-16T15:08:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.531+0000 7fccd8600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd00cfc60 con 0x7fccd00cf290 2024-01-16T15:08:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.532+0000 7fccd5c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fccd00ae5a0 0x7fccd015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.532+0000 7fccd6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 0x7fccd00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.532+0000 7fccd6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 0x7fccd00cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:37788/0 (socket says 172.21.15.53:37788) 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.532+0000 7fccd6600640 1 -- 172.21.15.53:0/2286827415 learned_addr learned my addr 172.21.15.53:0/2286827415 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.532+0000 7fccd5200640 1 --1- 172.21.15.53:0/2286827415 >> v1:172.21.15.146:6789/0 conn(0x7fccd00b3ae0 0x7fccd006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:54352/0 (socket says 172.21.15.53:54352) 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.533+0000 7fccd5c00640 1 -- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 msgr2=0x7fccd00cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.533+0000 7fccd5c00640 1 --2- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 0x7fccd00cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.533+0000 7fccd5c00640 1 -- 172.21.15.53:0/2286827415 >> v1:172.21.15.146:6789/0 conn(0x7fccd00b3ae0 legacy=0x7fccd006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.533+0000 7fccd5c00640 1 -- 172.21.15.53:0/2286827415 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd006c470 con 0x7fccd00ae5a0 2024-01-16T15:08:08.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.533+0000 7fccd6600640 1 --2- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 0x7fccd00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:08:08.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.534+0000 7fccd5c00640 1 --2- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00ae5a0 0x7fccd015e430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fccc00097a0 tx=0x7fccc002b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f9239da33e04d2b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:08.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.535+0000 7fcccfe00640 1 -- 172.21.15.53:0/2286827415 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0030800 con 0x7fccd00ae5a0 2024-01-16T15:08:08.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.535+0000 7fcccfe00640 1 -- 172.21.15.53:0/2286827415 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc0030960 con 0x7fccd00ae5a0 2024-01-16T15:08:08.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.535+0000 7fcccfe00640 1 -- 172.21.15.53:0/2286827415 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0030cf0 con 0x7fccd00ae5a0 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.535+0000 7fccd8600640 1 -- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00ae5a0 msgr2=0x7fccd015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.535+0000 7fccd8600640 1 --2- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00ae5a0 0x7fccd015e430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fccc00097a0 tx=0x7fccc002b940 comp rx=0 tx=0).stop 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.536+0000 7fccd8600640 1 -- 172.21.15.53:0/2286827415 shutdown_connections 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.536+0000 7fccd8600640 1 --2- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00cf290 0x7fccd00cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.536+0000 7fccd8600640 1 --2- 172.21.15.53:0/2286827415 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00ae5a0 0x7fccd015e430 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.536+0000 7fccd8600640 1 -- 172.21.15.53:0/2286827415 >> 172.21.15.53:0/2286827415 conn(0x7fccd006a800 msgr2=0x7fccd006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.536+0000 7fccd8600640 1 -- 172.21.15.53:0/2286827415 shutdown_connections 2024-01-16T15:08:08.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.536+0000 7fccd8600640 1 -- 172.21.15.53:0/2286827415 wait complete. 2024-01-16T15:08:08.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.537+0000 7fccd8600640 1 Processor -- start 2024-01-16T15:08:08.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.538+0000 7fccd8600640 1 -- start start 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.538+0000 7fccd8600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00b3ae0 0x7fccd014f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd8600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00cf290 0x7fccd01f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd8600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 0x7fccd015ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd8600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd01f6cb0 con 0x7fccd01f6800 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd8600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd01574c0 con 0x7fccd00cf290 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd8600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd015faf0 con 0x7fccd00b3ae0 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 0x7fccd015ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 0x7fccd015ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45490/0 (socket says 172.21.15.53:45490) 2024-01-16T15:08:08.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 -- 172.21.15.53:0/2586213099 learned_addr learned my addr 172.21.15.53:0/2586213099 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd5200640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00cf290 0x7fccd01f62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd5c00640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00b3ae0 0x7fccd014f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 -- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00b3ae0 msgr2=0x7fccd014f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00b3ae0 0x7fccd014f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 -- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00cf290 msgr2=0x7fccd01f62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00cf290 0x7fccd01f62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.539+0000 7fccd6600640 1 -- 172.21.15.53:0/2586213099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc8009b30 con 0x7fccd01f6800 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.540+0000 7fccd6600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 0x7fccd015ed80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fccc800efc0 tx=0x7fccc800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.540+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8018070 con 0x7fccd01f6800 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.540+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc800f040 con 0x7fccd01f6800 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.540+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc800ce00 con 0x7fccd01f6800 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.540+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc0009400 con 0x7fccd01f6800 2024-01-16T15:08:08.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.540+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd01600b0 con 0x7fccd01f6800 2024-01-16T15:08:08.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.541+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd0157680 con 0x7fccd01f6800 2024-01-16T15:08:08.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.542+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fccc8014480 con 0x7fccd01f6800 2024-01-16T15:08:08.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.543+0000 7fcccd600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcca806bbf0 0x7fcca806e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:08.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.543+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fccc8088950 con 0x7fccd01f6800 2024-01-16T15:08:08.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.544+0000 7fccd5c00640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcca806bbf0 0x7fcca806e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:08.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.545+0000 7fccd5c00640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcca806bbf0 0x7fcca806e0b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036000 tx=0x7fccc003b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:08.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.548+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fccc8055020 con 0x7fccd01f6800 2024-01-16T15:08:08.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.728+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fccd0071170 con 0x7fcca806bbf0 2024-01-16T15:08:08.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.756+0000 7fcccd600640 1 -- 172.21.15.53:0/2586213099 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286515 (secure 0 0 0) 0x7fccc0030600 con 0x7fcca806bbf0 2024-01-16T15:08:08.738 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:08.740 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:08:08.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcca806bbf0 msgr2=0x7fcca806e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcca806bbf0 0x7fcca806e0b0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fccc0036000 tx=0x7fccc003b040 comp rx=0 tx=0).stop 2024-01-16T15:08:08.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 msgr2=0x7fccd015ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:08.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 0x7fccd015ed80 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fccc800efc0 tx=0x7fccc800c490 comp rx=0 tx=0).stop 2024-01-16T15:08:08.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 shutdown_connections 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcca806bbf0 0x7fcca806e0b0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fccd01f6800 0x7fccd015ed80 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fccd00cf290 0x7fccd01f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 --2- 172.21.15.53:0/2586213099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fccd00b3ae0 0x7fccd014f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 >> 172.21.15.53:0/2586213099 conn(0x7fccd006a800 msgr2=0x7fccd015aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 shutdown_connections 2024-01-16T15:08:08.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:08.763+0000 7fccd8600640 1 -- 172.21.15.53:0/2586213099 wait complete. 2024-01-16T15:08:08.758 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-01-16T15:08:08.375888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":405944,"kb_used_data":7936,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754568776,"statfs":{"total":773094113280,"available":772678426624,"internally_reserved":0,"allocated":8126464,"data_stored":4088380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":407424560},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001938"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326974+0000","last_change":"2024-01-16T15:08:04.318085+0000","last_active":"2024-01-16T15:08:04.326974+0000","last_peered":"2024-01-16T15:08:04.326974+0000","last_clean":"2024-01-16T15:08:04.326974+0000","last_became_active":"2024-01-16T15:08:04.314969+0000","last_became_peered":"2024-01-16T15:08:04.314969+0000","last_unstale":"2024-01-16T15:08:04.326974+0000","last_undegraded":"2024-01-16T15:08:04.326974+0000","last_fullsized":"2024-01-16T15:08:04.326974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.041958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877459+0000","last_change":"2024-01-16T15:08:04.323490+0000","last_active":"2024-01-16T15:08:05.877459+0000","last_peered":"2024-01-16T15:08:05.877459+0000","last_clean":"2024-01-16T15:08:05.877459+0000","last_became_active":"2024-01-16T15:08:04.322913+0000","last_became_peered":"2024-01-16T15:08:04.322913+0000","last_unstale":"2024-01-16T15:08:05.877459+0000","last_undegraded":"2024-01-16T15:08:05.877459+0000","last_fullsized":"2024-01-16T15:08:05.877459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:45.702276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.312523+0000","last_change":"2024-01-16T15:08:04.307919+0000","last_active":"2024-01-16T15:08:04.312523+0000","last_peered":"2024-01-16T15:08:04.312523+0000","last_clean":"2024-01-16T15:08:04.312523+0000","last_became_active":"2024-01-16T15:08:04.307650+0000","last_became_peered":"2024-01-16T15:08:04.307650+0000","last_unstale":"2024-01-16T15:08:04.312523+0000","last_undegraded":"2024-01-16T15:08:04.312523+0000","last_fullsized":"2024-01-16T15:08:04.312523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:00.013331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317866+0000","last_change":"2024-01-16T15:08:04.319430+0000","last_active":"2024-01-16T15:08:05.317866+0000","last_peered":"2024-01-16T15:08:05.317866+0000","last_clean":"2024-01-16T15:08:05.317866+0000","last_became_active":"2024-01-16T15:08:04.318684+0000","last_became_peered":"2024-01-16T15:08:04.318684+0000","last_unstale":"2024-01-16T15:08:05.317866+0000","last_undegraded":"2024-01-16T15:08:05.317866+0000","last_fullsized":"2024-01-16T15:08:05.317866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:22.339868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.321986+0000","last_change":"2024-01-16T15:08:04.319295+0000","last_active":"2024-01-16T15:08:04.321986+0000","last_peered":"2024-01-16T15:08:04.321986+0000","last_clean":"2024-01-16T15:08:04.321986+0000","last_became_active":"2024-01-16T15:08:04.319005+0000","last_became_peered":"2024-01-16T15:08:04.319005+0000","last_unstale":"2024-01-16T15:08:04.321986+0000","last_undegraded":"2024-01-16T15:08:04.321986+0000","last_fullsized":"2024-01-16T15:08:04.321986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:40.492931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876359+0000","last_change":"2024-01-16T15:08:04.328640+0000","last_active":"2024-01-16T15:08:05.876359+0000","last_peered":"2024-01-16T15:08:05.876359+0000","last_clean":"2024-01-16T15:08:05.876359+0000","last_became_active":"2024-01-16T15:08:04.328356+0000","last_became_peered":"2024-01-16T15:08:04.328356+0000","last_unstale":"2024-01-16T15:08:05.876359+0000","last_undegraded":"2024-01-16T15:08:05.876359+0000","last_fullsized":"2024-01-16T15:08:05.876359+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:56.030381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.324330+0000","last_change":"2024-01-16T15:08:04.322407+0000","last_active":"2024-01-16T15:08:04.324330+0000","last_peered":"2024-01-16T15:08:04.324330+0000","last_clean":"2024-01-16T15:08:04.324330+0000","last_became_active":"2024-01-16T15:08:04.321973+0000","last_became_peered":"2024-01-16T15:08:04.321973+0000","last_unstale":"2024-01-16T15:08:04.324330+0000","last_undegraded":"2024-01-16T15:08:04.324330+0000","last_fullsized":"2024-01-16T15:08:04.324330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:02.906263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876327+0000","last_change":"2024-01-16T15:08:04.313503+0000","last_active":"2024-01-16T15:08:05.876327+0000","last_peered":"2024-01-16T15:08:05.876327+0000","last_clean":"2024-01-16T15:08:05.876327+0000","last_became_active":"2024-01-16T15:08:04.312882+0000","last_became_peered":"2024-01-16T15:08:04.312882+0000","last_unstale":"2024-01-16T15:08:05.876327+0000","last_undegraded":"2024-01-16T15:08:05.876327+0000","last_fullsized":"2024-01-16T15:08:05.876327+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:48:04.785493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314392+0000","last_change":"2024-01-16T15:08:04.311731+0000","last_active":"2024-01-16T15:08:04.314392+0000","last_peered":"2024-01-16T15:08:04.314392+0000","last_clean":"2024-01-16T15:08:04.314392+0000","last_became_active":"2024-01-16T15:08:04.311413+0000","last_became_peered":"2024-01-16T15:08:04.311413+0000","last_unstale":"2024-01-16T15:08:04.314392+0000","last_undegraded":"2024-01-16T15:08:04.314392+0000","last_fullsized":"2024-01-16T15:08:04.314392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:38.851861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319644+0000","last_change":"2024-01-16T15:08:04.315893+0000","last_active":"2024-01-16T15:08:05.319644+0000","last_peered":"2024-01-16T15:08:05.319644+0000","last_clean":"2024-01-16T15:08:05.319644+0000","last_became_active":"2024-01-16T15:08:04.315435+0000","last_became_peered":"2024-01-16T15:08:04.315435+0000","last_unstale":"2024-01-16T15:08:05.319644+0000","last_undegraded":"2024-01-16T15:08:05.319644+0000","last_fullsized":"2024-01-16T15:08:05.319644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:27:40.947545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.311725+0000","last_change":"2024-01-16T15:08:04.308397+0000","last_active":"2024-01-16T15:08:04.311725+0000","last_peered":"2024-01-16T15:08:04.311725+0000","last_clean":"2024-01-16T15:08:04.311725+0000","last_became_active":"2024-01-16T15:08:04.308060+0000","last_became_peered":"2024-01-16T15:08:04.308060+0000","last_unstale":"2024-01-16T15:08:04.311725+0000","last_undegraded":"2024-01-16T15:08:04.311725+0000","last_fullsized":"2024-01-16T15:08:04.311725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:29.417861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317688+0000","last_change":"2024-01-16T15:08:04.316673+0000","last_active":"2024-01-16T15:08:05.317688+0000","last_peered":"2024-01-16T15:08:05.317688+0000","last_clean":"2024-01-16T15:08:05.317688+0000","last_became_active":"2024-01-16T15:08:04.312492+0000","last_became_peered":"2024-01-16T15:08:04.312492+0000","last_unstale":"2024-01-16T15:08:05.317688+0000","last_undegraded":"2024-01-16T15:08:05.317688+0000","last_fullsized":"2024-01-16T15:08:05.317688+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:37.200622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.320022+0000","last_change":"2024-01-16T15:08:04.317681+0000","last_active":"2024-01-16T15:08:04.320022+0000","last_peered":"2024-01-16T15:08:04.320022+0000","last_clean":"2024-01-16T15:08:04.320022+0000","last_became_active":"2024-01-16T15:08:04.317080+0000","last_became_peered":"2024-01-16T15:08:04.317080+0000","last_unstale":"2024-01-16T15:08:04.320022+0000","last_undegraded":"2024-01-16T15:08:04.320022+0000","last_fullsized":"2024-01-16T15:08:04.320022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:19.115023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.331358+0000","last_change":"2024-01-16T15:08:04.329809+0000","last_active":"2024-01-16T15:08:04.331358+0000","last_peered":"2024-01-16T15:08:04.331358+0000","last_clean":"2024-01-16T15:08:04.331358+0000","last_became_active":"2024-01-16T15:08:04.329526+0000","last_became_peered":"2024-01-16T15:08:04.329526+0000","last_unstale":"2024-01-16T15:08:04.331358+0000","last_undegraded":"2024-01-16T15:08:04.331358+0000","last_fullsized":"2024-01-16T15:08:04.331358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:25.877853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314705+0000","last_change":"2024-01-16T15:08:04.312436+0000","last_active":"2024-01-16T15:08:04.314705+0000","last_peered":"2024-01-16T15:08:04.314705+0000","last_clean":"2024-01-16T15:08:04.314705+0000","last_became_active":"2024-01-16T15:08:04.312131+0000","last_became_peered":"2024-01-16T15:08:04.312131+0000","last_unstale":"2024-01-16T15:08:04.314705+0000","last_undegraded":"2024-01-16T15:08:04.314705+0000","last_fullsized":"2024-01-16T15:08:04.314705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:31.718991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.320384+0000","last_change":"2024-01-16T15:08:04.317948+0000","last_active":"2024-01-16T15:08:05.320384+0000","last_peered":"2024-01-16T15:08:05.320384+0000","last_clean":"2024-01-16T15:08:05.320384+0000","last_became_active":"2024-01-16T15:08:04.316680+0000","last_became_peered":"2024-01-16T15:08:04.316680+0000","last_unstale":"2024-01-16T15:08:05.320384+0000","last_undegraded":"2024-01-16T15:08:05.320384+0000","last_fullsized":"2024-01-16T15:08:05.320384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:12.054682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314412+0000","last_change":"2024-01-16T15:08:04.310501+0000","last_active":"2024-01-16T15:08:04.314412+0000","last_peered":"2024-01-16T15:08:04.314412+0000","last_clean":"2024-01-16T15:08:04.314412+0000","last_became_active":"2024-01-16T15:08:04.310158+0000","last_became_peered":"2024-01-16T15:08:04.310158+0000","last_unstale":"2024-01-16T15:08:04.314412+0000","last_undegraded":"2024-01-16T15:08:04.314412+0000","last_fullsized":"2024-01-16T15:08:04.314412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:26.755643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317705+0000","last_change":"2024-01-16T15:08:04.335659+0000","last_active":"2024-01-16T15:08:05.317705+0000","last_peered":"2024-01-16T15:08:05.317705+0000","last_clean":"2024-01-16T15:08:05.317705+0000","last_became_active":"2024-01-16T15:08:04.335181+0000","last_became_peered":"2024-01-16T15:08:04.335181+0000","last_unstale":"2024-01-16T15:08:05.317705+0000","last_undegraded":"2024-01-16T15:08:05.317705+0000","last_fullsized":"2024-01-16T15:08:05.317705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:16.523173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877090+0000","last_change":"2024-01-16T15:08:04.325332+0000","last_active":"2024-01-16T15:08:05.877090+0000","last_peered":"2024-01-16T15:08:05.877090+0000","last_clean":"2024-01-16T15:08:05.877090+0000","last_became_active":"2024-01-16T15:08:04.324175+0000","last_became_peered":"2024-01-16T15:08:04.324175+0000","last_unstale":"2024-01-16T15:08:05.877090+0000","last_undegraded":"2024-01-16T15:08:05.877090+0000","last_fullsized":"2024-01-16T15:08:05.877090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:00.978684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.324254+0000","last_change":"2024-01-16T15:08:04.322958+0000","last_active":"2024-01-16T15:08:04.324254+0000","last_peered":"2024-01-16T15:08:04.324254+0000","last_clean":"2024-01-16T15:08:04.324254+0000","last_became_active":"2024-01-16T15:08:04.321906+0000","last_became_peered":"2024-01-16T15:08:04.321906+0000","last_unstale":"2024-01-16T15:08:04.324254+0000","last_undegraded":"2024-01-16T15:08:04.324254+0000","last_fullsized":"2024-01-16T15:08:04.324254+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:11.951966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319286+0000","last_change":"2024-01-16T15:08:04.304645+0000","last_active":"2024-01-16T15:08:05.319286+0000","last_peered":"2024-01-16T15:08:05.319286+0000","last_clean":"2024-01-16T15:08:05.319286+0000","last_became_active":"2024-01-16T15:08:04.304054+0000","last_became_peered":"2024-01-16T15:08:04.304054+0000","last_unstale":"2024-01-16T15:08:05.319286+0000","last_undegraded":"2024-01-16T15:08:05.319286+0000","last_fullsized":"2024-01-16T15:08:05.319286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:41.087608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319262+0000","last_change":"2024-01-16T15:08:04.314852+0000","last_active":"2024-01-16T15:08:05.319262+0000","last_peered":"2024-01-16T15:08:05.319262+0000","last_clean":"2024-01-16T15:08:05.319262+0000","last_became_active":"2024-01-16T15:08:04.314559+0000","last_became_peered":"2024-01-16T15:08:04.314559+0000","last_unstale":"2024-01-16T15:08:05.319262+0000","last_undegraded":"2024-01-16T15:08:05.319262+0000","last_fullsized":"2024-01-16T15:08:05.319262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:33:28.389208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.337285+0000","last_change":"2024-01-16T15:08:04.332155+0000","last_active":"2024-01-16T15:08:04.337285+0000","last_peered":"2024-01-16T15:08:04.337285+0000","last_clean":"2024-01-16T15:08:04.337285+0000","last_became_active":"2024-01-16T15:08:04.329534+0000","last_became_peered":"2024-01-16T15:08:04.329534+0000","last_unstale":"2024-01-16T15:08:04.337285+0000","last_undegraded":"2024-01-16T15:08:04.337285+0000","last_fullsized":"2024-01-16T15:08:04.337285+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:19.115023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.338624+0000","last_change":"2024-01-16T15:08:04.332782+0000","last_active":"2024-01-16T15:08:04.338624+0000","last_peered":"2024-01-16T15:08:04.338624+0000","last_clean":"2024-01-16T15:08:04.338624+0000","last_became_active":"2024-01-16T15:08:04.332517+0000","last_became_peered":"2024-01-16T15:08:04.332517+0000","last_unstale":"2024-01-16T15:08:04.338624+0000","last_undegraded":"2024-01-16T15:08:04.338624+0000","last_fullsized":"2024-01-16T15:08:04.338624+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:20.168426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319646+0000","last_change":"2024-01-16T15:08:04.304272+0000","last_active":"2024-01-16T15:08:05.319646+0000","last_peered":"2024-01-16T15:08:05.319646+0000","last_clean":"2024-01-16T15:08:05.319646+0000","last_became_active":"2024-01-16T15:08:04.302567+0000","last_became_peered":"2024-01-16T15:08:04.302567+0000","last_unstale":"2024-01-16T15:08:05.319646+0000","last_undegraded":"2024-01-16T15:08:05.319646+0000","last_fullsized":"2024-01-16T15:08:05.319646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:01.847441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336572+0000","last_change":"2024-01-16T15:08:04.335315+0000","last_active":"2024-01-16T15:08:04.336572+0000","last_peered":"2024-01-16T15:08:04.336572+0000","last_clean":"2024-01-16T15:08:04.336572+0000","last_became_active":"2024-01-16T15:08:04.335040+0000","last_became_peered":"2024-01-16T15:08:04.335040+0000","last_unstale":"2024-01-16T15:08:04.336572+0000","last_undegraded":"2024-01-16T15:08:04.336572+0000","last_fullsized":"2024-01-16T15:08:04.336572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:25.214223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.321116+0000","last_change":"2024-01-16T15:08:04.318374+0000","last_active":"2024-01-16T15:08:04.321116+0000","last_peered":"2024-01-16T15:08:04.321116+0000","last_clean":"2024-01-16T15:08:04.321116+0000","last_became_active":"2024-01-16T15:08:04.314762+0000","last_became_peered":"2024-01-16T15:08:04.314762+0000","last_unstale":"2024-01-16T15:08:04.321116+0000","last_undegraded":"2024-01-16T15:08:04.321116+0000","last_fullsized":"2024-01-16T15:08:04.321116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:42.934564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319404+0000","last_change":"2024-01-16T15:08:04.316813+0000","last_active":"2024-01-16T15:08:05.319404+0000","last_peered":"2024-01-16T15:08:05.319404+0000","last_clean":"2024-01-16T15:08:05.319404+0000","last_became_active":"2024-01-16T15:08:04.316456+0000","last_became_peered":"2024-01-16T15:08:04.316456+0000","last_unstale":"2024-01-16T15:08:05.319404+0000","last_undegraded":"2024-01-16T15:08:05.319404+0000","last_fullsized":"2024-01-16T15:08:05.319404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:37.858357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.323205+0000","last_change":"2024-01-16T15:08:04.321861+0000","last_active":"2024-01-16T15:08:04.323205+0000","last_peered":"2024-01-16T15:08:04.323205+0000","last_clean":"2024-01-16T15:08:04.323205+0000","last_became_active":"2024-01-16T15:08:04.321437+0000","last_became_peered":"2024-01-16T15:08:04.321437+0000","last_unstale":"2024-01-16T15:08:04.323205+0000","last_undegraded":"2024-01-16T15:08:04.323205+0000","last_fullsized":"2024-01-16T15:08:04.323205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:29.185899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.338017+0000","last_change":"2024-01-16T15:08:04.336138+0000","last_active":"2024-01-16T15:08:04.338017+0000","last_peered":"2024-01-16T15:08:04.338017+0000","last_clean":"2024-01-16T15:08:04.338017+0000","last_became_active":"2024-01-16T15:08:04.335717+0000","last_became_peered":"2024-01-16T15:08:04.335717+0000","last_unstale":"2024-01-16T15:08:04.338017+0000","last_undegraded":"2024-01-16T15:08:04.338017+0000","last_fullsized":"2024-01-16T15:08:04.338017+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:19.115023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336473+0000","last_change":"2024-01-16T15:08:04.333829+0000","last_active":"2024-01-16T15:08:04.336473+0000","last_peered":"2024-01-16T15:08:04.336473+0000","last_clean":"2024-01-16T15:08:04.336473+0000","last_became_active":"2024-01-16T15:08:04.333555+0000","last_became_peered":"2024-01-16T15:08:04.333555+0000","last_unstale":"2024-01-16T15:08:04.336473+0000","last_undegraded":"2024-01-16T15:08:04.336473+0000","last_fullsized":"2024-01-16T15:08:04.336473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:06.794696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876488+0000","last_change":"2024-01-16T15:08:04.320098+0000","last_active":"2024-01-16T15:08:05.876488+0000","last_peered":"2024-01-16T15:08:05.876488+0000","last_clean":"2024-01-16T15:08:05.876488+0000","last_became_active":"2024-01-16T15:08:04.319356+0000","last_became_peered":"2024-01-16T15:08:04.319356+0000","last_unstale":"2024-01-16T15:08:05.876488+0000","last_undegraded":"2024-01-16T15:08:05.876488+0000","last_fullsized":"2024-01-16T15:08:05.876488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:19.111663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336747+0000","last_change":"2024-01-16T15:08:04.331908+0000","last_active":"2024-01-16T15:08:04.336747+0000","last_peered":"2024-01-16T15:08:04.336747+0000","last_clean":"2024-01-16T15:08:04.336747+0000","last_became_active":"2024-01-16T15:08:04.331565+0000","last_became_peered":"2024-01-16T15:08:04.331565+0000","last_unstale":"2024-01-16T15:08:04.336747+0000","last_undegraded":"2024-01-16T15:08:04.336747+0000","last_fullsized":"2024-01-16T15:08:04.336747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:11.951966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.337089+0000","last_change":"2024-01-16T15:08:03.311514+0000","last_active":"2024-01-16T15:08:04.337089+0000","last_peered":"2024-01-16T15:08:04.337089+0000","last_clean":"2024-01-16T15:08:04.337089+0000","last_became_active":"2024-01-16T15:08:03.311238+0000","last_became_peered":"2024-01-16T15:08:03.311238+0000","last_unstale":"2024-01-16T15:08:04.337089+0000","last_undegraded":"2024-01-16T15:08:04.337089+0000","last_fullsized":"2024-01-16T15:08:04.337089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:36.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.312524+0000","last_change":"2024-01-16T15:08:04.310480+0000","last_active":"2024-01-16T15:08:04.312524+0000","last_peered":"2024-01-16T15:08:04.312524+0000","last_clean":"2024-01-16T15:08:04.312524+0000","last_became_active":"2024-01-16T15:08:04.310121+0000","last_became_peered":"2024-01-16T15:08:04.310121+0000","last_unstale":"2024-01-16T15:08:04.312524+0000","last_undegraded":"2024-01-16T15:08:04.312524+0000","last_fullsized":"2024-01-16T15:08:04.312524+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:54.608877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.331753+0000","last_change":"2024-01-16T15:08:03.312504+0000","last_active":"2024-01-16T15:08:04.331753+0000","last_peered":"2024-01-16T15:08:04.331753+0000","last_clean":"2024-01-16T15:08:04.331753+0000","last_became_active":"2024-01-16T15:08:03.312176+0000","last_became_peered":"2024-01-16T15:08:03.312176+0000","last_unstale":"2024-01-16T15:08:04.331753+0000","last_undegraded":"2024-01-16T15:08:04.331753+0000","last_fullsized":"2024-01-16T15:08:04.331753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:02.787365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876502+0000","last_change":"2024-01-16T15:08:04.315592+0000","last_active":"2024-01-16T15:08:05.876502+0000","last_peered":"2024-01-16T15:08:05.876502+0000","last_clean":"2024-01-16T15:08:05.876502+0000","last_became_active":"2024-01-16T15:08:04.315286+0000","last_became_peered":"2024-01-16T15:08:04.315286+0000","last_unstale":"2024-01-16T15:08:05.876502+0000","last_undegraded":"2024-01-16T15:08:05.876502+0000","last_fullsized":"2024-01-16T15:08:05.876502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:49.221249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319927+0000","last_change":"2024-01-16T15:08:03.312380+0000","last_active":"2024-01-16T15:08:05.319927+0000","last_peered":"2024-01-16T15:08:05.319927+0000","last_clean":"2024-01-16T15:08:05.319927+0000","last_became_active":"2024-01-16T15:08:03.311822+0000","last_became_peered":"2024-01-16T15:08:03.311822+0000","last_unstale":"2024-01-16T15:08:05.319927+0000","last_undegraded":"2024-01-16T15:08:05.319927+0000","last_fullsized":"2024-01-16T15:08:05.319927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:46.968161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.311703+0000","last_change":"2024-01-16T15:08:04.300254+0000","last_active":"2024-01-16T15:08:04.311703+0000","last_peered":"2024-01-16T15:08:04.311703+0000","last_clean":"2024-01-16T15:08:04.311703+0000","last_became_active":"2024-01-16T15:08:04.299401+0000","last_became_peered":"2024-01-16T15:08:04.299401+0000","last_unstale":"2024-01-16T15:08:04.311703+0000","last_undegraded":"2024-01-16T15:08:04.311703+0000","last_fullsized":"2024-01-16T15:08:04.311703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:31.718991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876773+0000","last_change":"2024-01-16T15:08:03.315357+0000","last_active":"2024-01-16T15:08:05.876773+0000","last_peered":"2024-01-16T15:08:05.876773+0000","last_clean":"2024-01-16T15:08:05.876773+0000","last_became_active":"2024-01-16T15:08:03.315064+0000","last_became_peered":"2024-01-16T15:08:03.315064+0000","last_unstale":"2024-01-16T15:08:05.876773+0000","last_undegraded":"2024-01-16T15:08:05.876773+0000","last_fullsized":"2024-01-16T15:08:05.876773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:25.472504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.329729+0000","last_change":"2024-01-16T15:08:04.326228+0000","last_active":"2024-01-16T15:08:04.329729+0000","last_peered":"2024-01-16T15:08:04.329729+0000","last_clean":"2024-01-16T15:08:04.329729+0000","last_became_active":"2024-01-16T15:08:04.325893+0000","last_became_peered":"2024-01-16T15:08:04.325893+0000","last_unstale":"2024-01-16T15:08:04.329729+0000","last_undegraded":"2024-01-16T15:08:04.329729+0000","last_fullsized":"2024-01-16T15:08:04.329729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:20.168426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.329818+0000","last_change":"2024-01-16T15:08:03.310423+0000","last_active":"2024-01-16T15:08:04.329818+0000","last_peered":"2024-01-16T15:08:04.329818+0000","last_clean":"2024-01-16T15:08:04.329818+0000","last_became_active":"2024-01-16T15:08:03.308253+0000","last_became_peered":"2024-01-16T15:08:03.308253+0000","last_unstale":"2024-01-16T15:08:04.329818+0000","last_undegraded":"2024-01-16T15:08:04.329818+0000","last_fullsized":"2024-01-16T15:08:04.329818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:34.374011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319288+0000","last_change":"2024-01-16T15:08:04.301339+0000","last_active":"2024-01-16T15:08:05.319288+0000","last_peered":"2024-01-16T15:08:05.319288+0000","last_clean":"2024-01-16T15:08:05.319288+0000","last_became_active":"2024-01-16T15:08:04.300419+0000","last_became_peered":"2024-01-16T15:08:04.300419+0000","last_unstale":"2024-01-16T15:08:05.319288+0000","last_undegraded":"2024-01-16T15:08:05.319288+0000","last_fullsized":"2024-01-16T15:08:05.319288+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:04.609348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876629+0000","last_change":"2024-01-16T15:08:03.312036+0000","last_active":"2024-01-16T15:08:05.876629+0000","last_peered":"2024-01-16T15:08:05.876629+0000","last_clean":"2024-01-16T15:08:05.876629+0000","last_became_active":"2024-01-16T15:08:03.311300+0000","last_became_peered":"2024-01-16T15:08:03.311300+0000","last_unstale":"2024-01-16T15:08:05.876629+0000","last_undegraded":"2024-01-16T15:08:05.876629+0000","last_fullsized":"2024-01-16T15:08:05.876629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:31.818904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.315158+0000","last_change":"2024-01-16T15:08:04.311575+0000","last_active":"2024-01-16T15:08:04.315158+0000","last_peered":"2024-01-16T15:08:04.315158+0000","last_clean":"2024-01-16T15:08:04.315158+0000","last_became_active":"2024-01-16T15:08:04.310670+0000","last_became_peered":"2024-01-16T15:08:04.310670+0000","last_unstale":"2024-01-16T15:08:04.315158+0000","last_undegraded":"2024-01-16T15:08:04.315158+0000","last_fullsized":"2024-01-16T15:08:04.315158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:03:51.644650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317116+0000","last_change":"2024-01-16T15:08:03.310875+0000","last_active":"2024-01-16T15:08:05.317116+0000","last_peered":"2024-01-16T15:08:05.317116+0000","last_clean":"2024-01-16T15:08:05.317116+0000","last_became_active":"2024-01-16T15:08:03.310276+0000","last_became_peered":"2024-01-16T15:08:03.310276+0000","last_unstale":"2024-01-16T15:08:05.317116+0000","last_undegraded":"2024-01-16T15:08:05.317116+0000","last_fullsized":"2024-01-16T15:08:05.317116+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:14.858194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.309613+0000","last_change":"2024-01-16T15:08:04.307417+0000","last_active":"2024-01-16T15:08:04.309613+0000","last_peered":"2024-01-16T15:08:04.309613+0000","last_clean":"2024-01-16T15:08:04.309613+0000","last_became_active":"2024-01-16T15:08:04.306078+0000","last_became_peered":"2024-01-16T15:08:04.306078+0000","last_unstale":"2024-01-16T15:08:04.309613+0000","last_undegraded":"2024-01-16T15:08:04.309613+0000","last_fullsized":"2024-01-16T15:08:04.309613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:21.211461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.296394+0000","last_change":"2024-01-16T15:08:03.304692+0000","last_active":"2024-01-16T15:08:04.296394+0000","last_peered":"2024-01-16T15:08:04.296394+0000","last_clean":"2024-01-16T15:08:04.296394+0000","last_became_active":"2024-01-16T15:08:03.303212+0000","last_became_peered":"2024-01-16T15:08:03.303212+0000","last_unstale":"2024-01-16T15:08:04.296394+0000","last_undegraded":"2024-01-16T15:08:04.296394+0000","last_fullsized":"2024-01-16T15:08:04.296394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:46.421758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336488+0000","last_change":"2024-01-16T15:08:04.334738+0000","last_active":"2024-01-16T15:08:04.336488+0000","last_peered":"2024-01-16T15:08:04.336488+0000","last_clean":"2024-01-16T15:08:04.336488+0000","last_became_active":"2024-01-16T15:08:04.334408+0000","last_became_peered":"2024-01-16T15:08:04.334408+0000","last_unstale":"2024-01-16T15:08:04.336488+0000","last_undegraded":"2024-01-16T15:08:04.336488+0000","last_fullsized":"2024-01-16T15:08:04.336488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:56.787564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319743+0000","last_change":"2024-01-16T15:08:03.308379+0000","last_active":"2024-01-16T15:08:05.319743+0000","last_peered":"2024-01-16T15:08:05.319743+0000","last_clean":"2024-01-16T15:08:05.319743+0000","last_became_active":"2024-01-16T15:08:03.308095+0000","last_became_peered":"2024-01-16T15:08:03.308095+0000","last_unstale":"2024-01-16T15:08:05.319743+0000","last_undegraded":"2024-01-16T15:08:05.319743+0000","last_fullsized":"2024-01-16T15:08:05.319743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:49.003048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326951+0000","last_change":"2024-01-16T15:08:04.319354+0000","last_active":"2024-01-16T15:08:04.326951+0000","last_peered":"2024-01-16T15:08:04.326951+0000","last_clean":"2024-01-16T15:08:04.326951+0000","last_became_active":"2024-01-16T15:08:04.319032+0000","last_became_peered":"2024-01-16T15:08:04.319032+0000","last_unstale":"2024-01-16T15:08:04.326951+0000","last_undegraded":"2024-01-16T15:08:04.326951+0000","last_fullsized":"2024-01-16T15:08:04.326951+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:16.809488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877506+0000","last_change":"2024-01-16T15:08:03.316929+0000","last_active":"2024-01-16T15:08:05.877506+0000","last_peered":"2024-01-16T15:08:05.877506+0000","last_clean":"2024-01-16T15:08:05.877506+0000","last_became_active":"2024-01-16T15:08:03.316201+0000","last_became_peered":"2024-01-16T15:08:03.316201+0000","last_unstale":"2024-01-16T15:08:05.877506+0000","last_undegraded":"2024-01-16T15:08:05.877506+0000","last_fullsized":"2024-01-16T15:08:05.877506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:11:38.592403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876606+0000","last_change":"2024-01-16T15:08:04.326679+0000","last_active":"2024-01-16T15:08:05.876606+0000","last_peered":"2024-01-16T15:08:05.876606+0000","last_clean":"2024-01-16T15:08:05.876606+0000","last_became_active":"2024-01-16T15:08:04.323081+0000","last_became_peered":"2024-01-16T15:08:04.323081+0000","last_unstale":"2024-01-16T15:08:05.876606+0000","last_undegraded":"2024-01-16T15:08:05.876606+0000","last_fullsized":"2024-01-16T15:08:05.876606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:17.712427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.330987+0000","last_change":"2024-01-16T15:08:03.313202+0000","last_active":"2024-01-16T15:08:04.330987+0000","last_peered":"2024-01-16T15:08:04.330987+0000","last_clean":"2024-01-16T15:08:04.330987+0000","last_became_active":"2024-01-16T15:08:03.312944+0000","last_became_peered":"2024-01-16T15:08:03.312944+0000","last_unstale":"2024-01-16T15:08:04.330987+0000","last_undegraded":"2024-01-16T15:08:04.330987+0000","last_fullsized":"2024-01-16T15:08:04.330987+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:07.334338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.315082+0000","last_change":"2024-01-16T15:08:04.311198+0000","last_active":"2024-01-16T15:08:04.315082+0000","last_peered":"2024-01-16T15:08:04.315082+0000","last_clean":"2024-01-16T15:08:04.315082+0000","last_became_active":"2024-01-16T15:08:04.310912+0000","last_became_peered":"2024-01-16T15:08:04.310912+0000","last_unstale":"2024-01-16T15:08:04.315082+0000","last_undegraded":"2024-01-16T15:08:04.315082+0000","last_fullsized":"2024-01-16T15:08:04.315082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:21.001264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.318016+0000","last_change":"2024-01-16T15:08:03.311820+0000","last_active":"2024-01-16T15:08:05.318016+0000","last_peered":"2024-01-16T15:08:05.318016+0000","last_clean":"2024-01-16T15:08:05.318016+0000","last_became_active":"2024-01-16T15:08:03.311542+0000","last_became_peered":"2024-01-16T15:08:03.311542+0000","last_unstale":"2024-01-16T15:08:05.318016+0000","last_undegraded":"2024-01-16T15:08:05.318016+0000","last_fullsized":"2024-01-16T15:08:05.318016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:58.716201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.311323+0000","last_change":"2024-01-16T15:08:04.309096+0000","last_active":"2024-01-16T15:08:04.311323+0000","last_peered":"2024-01-16T15:08:04.311323+0000","last_clean":"2024-01-16T15:08:04.311323+0000","last_became_active":"2024-01-16T15:08:04.308828+0000","last_became_peered":"2024-01-16T15:08:04.308828+0000","last_unstale":"2024-01-16T15:08:04.311323+0000","last_undegraded":"2024-01-16T15:08:04.311323+0000","last_fullsized":"2024-01-16T15:08:04.311323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:30.062633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317282+0000","last_change":"2024-01-16T15:08:03.307012+0000","last_active":"2024-01-16T15:08:05.317282+0000","last_peered":"2024-01-16T15:08:05.317282+0000","last_clean":"2024-01-16T15:08:05.317282+0000","last_became_active":"2024-01-16T15:08:03.306725+0000","last_became_peered":"2024-01-16T15:08:03.306725+0000","last_unstale":"2024-01-16T15:08:05.317282+0000","last_undegraded":"2024-01-16T15:08:05.317282+0000","last_fullsized":"2024-01-16T15:08:05.317282+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:59.878283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.332766+0000","last_change":"2024-01-16T15:08:04.330823+0000","last_active":"2024-01-16T15:08:04.332766+0000","last_peered":"2024-01-16T15:08:04.332766+0000","last_clean":"2024-01-16T15:08:04.332766+0000","last_became_active":"2024-01-16T15:08:04.330553+0000","last_became_peered":"2024-01-16T15:08:04.330553+0000","last_unstale":"2024-01-16T15:08:04.332766+0000","last_undegraded":"2024-01-16T15:08:04.332766+0000","last_fullsized":"2024-01-16T15:08:04.332766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:57:56.702039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876868+0000","last_change":"2024-01-16T15:08:03.315290+0000","last_active":"2024-01-16T15:08:05.876868+0000","last_peered":"2024-01-16T15:08:05.876868+0000","last_clean":"2024-01-16T15:08:05.876868+0000","last_became_active":"2024-01-16T15:08:03.314984+0000","last_became_peered":"2024-01-16T15:08:03.314984+0000","last_unstale":"2024-01-16T15:08:05.876868+0000","last_undegraded":"2024-01-16T15:08:05.876868+0000","last_fullsized":"2024-01-16T15:08:05.876868+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:02:49.155475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.329448+0000","last_change":"2024-01-16T15:08:04.327612+0000","last_active":"2024-01-16T15:08:04.329448+0000","last_peered":"2024-01-16T15:08:04.329448+0000","last_clean":"2024-01-16T15:08:04.329448+0000","last_became_active":"2024-01-16T15:08:04.327246+0000","last_became_peered":"2024-01-16T15:08:04.327246+0000","last_unstale":"2024-01-16T15:08:04.329448+0000","last_undegraded":"2024-01-16T15:08:04.329448+0000","last_fullsized":"2024-01-16T15:08:04.329448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:29.417861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314907+0000","last_change":"2024-01-16T15:08:03.316462+0000","last_active":"2024-01-16T15:08:04.314907+0000","last_peered":"2024-01-16T15:08:04.314907+0000","last_clean":"2024-01-16T15:08:04.314907+0000","last_became_active":"2024-01-16T15:08:03.316004+0000","last_became_peered":"2024-01-16T15:08:03.316004+0000","last_unstale":"2024-01-16T15:08:04.314907+0000","last_undegraded":"2024-01-16T15:08:04.314907+0000","last_fullsized":"2024-01-16T15:08:04.314907+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:18.148893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.309907+0000","last_change":"2024-01-16T15:08:04.308122+0000","last_active":"2024-01-16T15:08:04.309907+0000","last_peered":"2024-01-16T15:08:04.309907+0000","last_clean":"2024-01-16T15:08:04.309907+0000","last_became_active":"2024-01-16T15:08:04.307828+0000","last_became_peered":"2024-01-16T15:08:04.307828+0000","last_unstale":"2024-01-16T15:08:04.309907+0000","last_undegraded":"2024-01-16T15:08:04.309907+0000","last_fullsized":"2024-01-16T15:08:04.309907+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:53:17.628006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.318817+0000","last_change":"2024-01-16T15:08:03.297584+0000","last_active":"2024-01-16T15:08:05.318817+0000","last_peered":"2024-01-16T15:08:05.318817+0000","last_clean":"2024-01-16T15:08:05.318817+0000","last_became_active":"2024-01-16T15:08:03.297283+0000","last_became_peered":"2024-01-16T15:08:03.297283+0000","last_unstale":"2024-01-16T15:08:05.318817+0000","last_undegraded":"2024-01-16T15:08:05.318817+0000","last_fullsized":"2024-01-16T15:08:05.318817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:12.124956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319763+0000","last_change":"2024-01-16T15:08:04.315716+0000","last_active":"2024-01-16T15:08:05.319763+0000","last_peered":"2024-01-16T15:08:05.319763+0000","last_clean":"2024-01-16T15:08:05.319763+0000","last_became_active":"2024-01-16T15:08:04.314955+0000","last_became_peered":"2024-01-16T15:08:04.314955+0000","last_unstale":"2024-01-16T15:08:05.319763+0000","last_undegraded":"2024-01-16T15:08:05.319763+0000","last_fullsized":"2024-01-16T15:08:05.319763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:17.749199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877104+0000","last_change":"2024-01-16T15:08:03.314796+0000","last_active":"2024-01-16T15:08:05.877104+0000","last_peered":"2024-01-16T15:08:05.877104+0000","last_clean":"2024-01-16T15:08:05.877104+0000","last_became_active":"2024-01-16T15:08:03.314401+0000","last_became_peered":"2024-01-16T15:08:03.314401+0000","last_unstale":"2024-01-16T15:08:05.877104+0000","last_undegraded":"2024-01-16T15:08:05.877104+0000","last_fullsized":"2024-01-16T15:08:05.877104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:23.014039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.319485+0000","last_change":"2024-01-16T15:08:04.315796+0000","last_active":"2024-01-16T15:08:04.319485+0000","last_peered":"2024-01-16T15:08:04.319485+0000","last_clean":"2024-01-16T15:08:04.319485+0000","last_became_active":"2024-01-16T15:08:04.311547+0000","last_became_peered":"2024-01-16T15:08:04.311547+0000","last_unstale":"2024-01-16T15:08:04.319485+0000","last_undegraded":"2024-01-16T15:08:04.319485+0000","last_fullsized":"2024-01-16T15:08:04.319485+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:18.007918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319792+0000","last_change":"2024-01-16T15:08:03.317728+0000","last_active":"2024-01-16T15:08:05.319792+0000","last_peered":"2024-01-16T15:08:05.319792+0000","last_clean":"2024-01-16T15:08:05.319792+0000","last_became_active":"2024-01-16T15:08:03.317115+0000","last_became_peered":"2024-01-16T15:08:03.317115+0000","last_unstale":"2024-01-16T15:08:05.319792+0000","last_undegraded":"2024-01-16T15:08:05.319792+0000","last_fullsized":"2024-01-16T15:08:05.319792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:31.426794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317903+0000","last_change":"2024-01-16T15:08:04.312648+0000","last_active":"2024-01-16T15:08:05.317903+0000","last_peered":"2024-01-16T15:08:05.317903+0000","last_clean":"2024-01-16T15:08:05.317903+0000","last_became_active":"2024-01-16T15:08:04.312334+0000","last_became_peered":"2024-01-16T15:08:04.312334+0000","last_unstale":"2024-01-16T15:08:05.317903+0000","last_undegraded":"2024-01-16T15:08:05.317903+0000","last_fullsized":"2024-01-16T15:08:05.317903+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:06:21.368154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877275+0000","last_change":"2024-01-16T15:08:03.314555+0000","last_active":"2024-01-16T15:08:05.877275+0000","last_peered":"2024-01-16T15:08:05.877275+0000","last_clean":"2024-01-16T15:08:05.877275+0000","last_became_active":"2024-01-16T15:08:03.314250+0000","last_became_peered":"2024-01-16T15:08:03.314250+0000","last_unstale":"2024-01-16T15:08:05.877275+0000","last_undegraded":"2024-01-16T15:08:05.877275+0000","last_fullsized":"2024-01-16T15:08:05.877275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:10.313446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.325699+0000","last_change":"2024-01-16T15:08:04.323321+0000","last_active":"2024-01-16T15:08:04.325699+0000","last_peered":"2024-01-16T15:08:04.325699+0000","last_clean":"2024-01-16T15:08:04.325699+0000","last_became_active":"2024-01-16T15:08:04.323030+0000","last_became_peered":"2024-01-16T15:08:04.323030+0000","last_unstale":"2024-01-16T15:08:04.325699+0000","last_undegraded":"2024-01-16T15:08:04.325699+0000","last_fullsized":"2024-01-16T15:08:04.325699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:33:36.602276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.285974+0000","last_change":"2024-01-16T15:08:03.306768+0000","last_active":"2024-01-16T15:08:04.285974+0000","last_peered":"2024-01-16T15:08:04.285974+0000","last_clean":"2024-01-16T15:08:04.285974+0000","last_became_active":"2024-01-16T15:08:03.306422+0000","last_became_peered":"2024-01-16T15:08:03.306422+0000","last_unstale":"2024-01-16T15:08:04.285974+0000","last_undegraded":"2024-01-16T15:08:04.285974+0000","last_fullsized":"2024-01-16T15:08:04.285974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:20.195660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.317349+0000","last_change":"2024-01-16T15:08:04.313524+0000","last_active":"2024-01-16T15:08:04.317349+0000","last_peered":"2024-01-16T15:08:04.317349+0000","last_clean":"2024-01-16T15:08:04.317349+0000","last_became_active":"2024-01-16T15:08:04.311859+0000","last_became_peered":"2024-01-16T15:08:04.311859+0000","last_unstale":"2024-01-16T15:08:04.317349+0000","last_undegraded":"2024-01-16T15:08:04.317349+0000","last_fullsized":"2024-01-16T15:08:04.317349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:27:50.675877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876914+0000","last_change":"2024-01-16T15:08:03.318388+0000","last_active":"2024-01-16T15:08:05.876914+0000","last_peered":"2024-01-16T15:08:05.876914+0000","last_clean":"2024-01-16T15:08:05.876914+0000","last_became_active":"2024-01-16T15:08:03.316993+0000","last_became_peered":"2024-01-16T15:08:03.316993+0000","last_unstale":"2024-01-16T15:08:05.876914+0000","last_undegraded":"2024-01-16T15:08:05.876914+0000","last_fullsized":"2024-01-16T15:08:05.876914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:01.554258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.325417+0000","last_change":"2024-01-16T15:08:04.324472+0000","last_active":"2024-01-16T15:08:04.325417+0000","last_peered":"2024-01-16T15:08:04.325417+0000","last_clean":"2024-01-16T15:08:04.325417+0000","last_became_active":"2024-01-16T15:08:04.323999+0000","last_became_peered":"2024-01-16T15:08:04.323999+0000","last_unstale":"2024-01-16T15:08:04.325417+0000","last_undegraded":"2024-01-16T15:08:04.325417+0000","last_fullsized":"2024-01-16T15:08:04.325417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:45.225887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.315061+0000","last_change":"2024-01-16T15:08:03.314819+0000","last_active":"2024-01-16T15:08:04.315061+0000","last_peered":"2024-01-16T15:08:04.315061+0000","last_clean":"2024-01-16T15:08:04.315061+0000","last_became_active":"2024-01-16T15:08:03.314562+0000","last_became_peered":"2024-01-16T15:08:03.314562+0000","last_unstale":"2024-01-16T15:08:04.315061+0000","last_undegraded":"2024-01-16T15:08:04.315061+0000","last_fullsized":"2024-01-16T15:08:04.315061+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:41.596206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.321397+0000","last_change":"2024-01-16T15:08:04.316178+0000","last_active":"2024-01-16T15:08:04.321397+0000","last_peered":"2024-01-16T15:08:04.321397+0000","last_clean":"2024-01-16T15:08:04.321397+0000","last_became_active":"2024-01-16T15:08:04.315608+0000","last_became_peered":"2024-01-16T15:08:04.315608+0000","last_unstale":"2024-01-16T15:08:04.321397+0000","last_undegraded":"2024-01-16T15:08:04.321397+0000","last_fullsized":"2024-01-16T15:08:04.321397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.300239+0000","last_change":"2024-01-16T15:08:03.314684+0000","last_active":"2024-01-16T15:08:04.300239+0000","last_peered":"2024-01-16T15:08:04.300239+0000","last_clean":"2024-01-16T15:08:04.300239+0000","last_became_active":"2024-01-16T15:08:03.314325+0000","last_became_peered":"2024-01-16T15:08:03.314325+0000","last_unstale":"2024-01-16T15:08:04.300239+0000","last_undegraded":"2024-01-16T15:08:04.300239+0000","last_fullsized":"2024-01-16T15:08:04.300239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:15.793686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314546+0000","last_change":"2024-01-16T15:08:04.312391+0000","last_active":"2024-01-16T15:08:04.314546+0000","last_peered":"2024-01-16T15:08:04.314546+0000","last_clean":"2024-01-16T15:08:04.314546+0000","last_became_active":"2024-01-16T15:08:04.312026+0000","last_became_peered":"2024-01-16T15:08:04.312026+0000","last_unstale":"2024-01-16T15:08:04.314546+0000","last_undegraded":"2024-01-16T15:08:04.314546+0000","last_fullsized":"2024-01-16T15:08:04.314546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.041958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877384+0000","last_change":"2024-01-16T15:08:03.315816+0000","last_active":"2024-01-16T15:08:05.877384+0000","last_peered":"2024-01-16T15:08:05.877384+0000","last_clean":"2024-01-16T15:08:05.877384+0000","last_became_active":"2024-01-16T15:08:03.315417+0000","last_became_peered":"2024-01-16T15:08:03.315417+0000","last_unstale":"2024-01-16T15:08:05.877384+0000","last_undegraded":"2024-01-16T15:08:05.877384+0000","last_fullsized":"2024-01-16T15:08:05.877384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:42.145047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326195+0000","last_change":"2024-01-16T15:08:04.325323+0000","last_active":"2024-01-16T15:08:04.326195+0000","last_peered":"2024-01-16T15:08:04.326195+0000","last_clean":"2024-01-16T15:08:04.326195+0000","last_became_active":"2024-01-16T15:08:04.324401+0000","last_became_peered":"2024-01-16T15:08:04.324401+0000","last_unstale":"2024-01-16T15:08:04.326195+0000","last_undegraded":"2024-01-16T15:08:04.326195+0000","last_fullsized":"2024-01-16T15:08:04.326195+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:39:17.050489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.320439+0000","last_change":"2024-01-16T15:08:03.312423+0000","last_active":"2024-01-16T15:08:04.320439+0000","last_peered":"2024-01-16T15:08:04.320439+0000","last_clean":"2024-01-16T15:08:04.320439+0000","last_became_active":"2024-01-16T15:08:03.312112+0000","last_became_peered":"2024-01-16T15:08:03.312112+0000","last_unstale":"2024-01-16T15:08:04.320439+0000","last_undegraded":"2024-01-16T15:08:04.320439+0000","last_fullsized":"2024-01-16T15:08:04.320439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:10.936164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326202+0000","last_change":"2024-01-16T15:08:04.318122+0000","last_active":"2024-01-16T15:08:04.326202+0000","last_peered":"2024-01-16T15:08:04.326202+0000","last_clean":"2024-01-16T15:08:04.326202+0000","last_became_active":"2024-01-16T15:08:04.317834+0000","last_became_peered":"2024-01-16T15:08:04.317834+0000","last_unstale":"2024-01-16T15:08:04.326202+0000","last_undegraded":"2024-01-16T15:08:04.326202+0000","last_fullsized":"2024-01-16T15:08:04.326202+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:21.545902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.302925+0000","last_change":"2024-01-16T15:08:03.306305+0000","last_active":"2024-01-16T15:08:04.302925+0000","last_peered":"2024-01-16T15:08:04.302925+0000","last_clean":"2024-01-16T15:08:04.302925+0000","last_became_active":"2024-01-16T15:08:03.306027+0000","last_became_peered":"2024-01-16T15:08:03.306027+0000","last_unstale":"2024-01-16T15:08:04.302925+0000","last_undegraded":"2024-01-16T15:08:04.302925+0000","last_fullsized":"2024-01-16T15:08:04.302925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:24.907779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.338139+0000","last_change":"2024-01-16T15:08:04.332372+0000","last_active":"2024-01-16T15:08:04.338139+0000","last_peered":"2024-01-16T15:08:04.338139+0000","last_clean":"2024-01-16T15:08:04.338139+0000","last_became_active":"2024-01-16T15:08:04.331881+0000","last_became_peered":"2024-01-16T15:08:04.331881+0000","last_unstale":"2024-01-16T15:08:04.338139+0000","last_undegraded":"2024-01-16T15:08:04.338139+0000","last_fullsized":"2024-01-16T15:08:04.338139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:03.803166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.298386+0000","last_change":"2024-01-16T15:08:03.307706+0000","last_active":"2024-01-16T15:08:04.298386+0000","last_peered":"2024-01-16T15:08:04.298386+0000","last_clean":"2024-01-16T15:08:04.298386+0000","last_became_active":"2024-01-16T15:08:03.306510+0000","last_became_peered":"2024-01-16T15:08:03.306510+0000","last_unstale":"2024-01-16T15:08:04.298386+0000","last_undegraded":"2024-01-16T15:08:04.298386+0000","last_fullsized":"2024-01-16T15:08:04.298386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.530101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.312566+0000","last_change":"2024-01-16T15:08:04.309368+0000","last_active":"2024-01-16T15:08:04.312566+0000","last_peered":"2024-01-16T15:08:04.312566+0000","last_clean":"2024-01-16T15:08:04.312566+0000","last_became_active":"2024-01-16T15:08:04.309087+0000","last_became_peered":"2024-01-16T15:08:04.309087+0000","last_unstale":"2024-01-16T15:08:04.312566+0000","last_undegraded":"2024-01-16T15:08:04.312566+0000","last_fullsized":"2024-01-16T15:08:04.312566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:55.413459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.295260+0000","last_change":"2024-01-16T15:08:03.317378+0000","last_active":"2024-01-16T15:08:04.295260+0000","last_peered":"2024-01-16T15:08:04.295260+0000","last_clean":"2024-01-16T15:08:04.295260+0000","last_became_active":"2024-01-16T15:08:03.317100+0000","last_became_peered":"2024-01-16T15:08:03.317100+0000","last_unstale":"2024-01-16T15:08:04.295260+0000","last_undegraded":"2024-01-16T15:08:04.295260+0000","last_fullsized":"2024-01-16T15:08:04.295260+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:08.889310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876336+0000","last_change":"2024-01-16T15:08:04.319275+0000","last_active":"2024-01-16T15:08:05.876336+0000","last_peered":"2024-01-16T15:08:05.876336+0000","last_clean":"2024-01-16T15:08:05.876336+0000","last_became_active":"2024-01-16T15:08:04.318315+0000","last_became_peered":"2024-01-16T15:08:04.318315+0000","last_unstale":"2024-01-16T15:08:05.876336+0000","last_undegraded":"2024-01-16T15:08:05.876336+0000","last_fullsized":"2024-01-16T15:08:05.876336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:43.326777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.316977+0000","last_change":"2024-01-16T15:08:03.306371+0000","last_active":"2024-01-16T15:08:05.316977+0000","last_peered":"2024-01-16T15:08:05.316977+0000","last_clean":"2024-01-16T15:08:05.316977+0000","last_became_active":"2024-01-16T15:08:03.306064+0000","last_became_peered":"2024-01-16T15:08:03.306064+0000","last_unstale":"2024-01-16T15:08:05.316977+0000","last_undegraded":"2024-01-16T15:08:05.316977+0000","last_fullsized":"2024-01-16T15:08:05.316977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:11.513682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317974+0000","last_change":"2024-01-16T15:08:04.308853+0000","last_active":"2024-01-16T15:08:05.317974+0000","last_peered":"2024-01-16T15:08:05.317974+0000","last_clean":"2024-01-16T15:08:05.317974+0000","last_became_active":"2024-01-16T15:08:04.308415+0000","last_became_peered":"2024-01-16T15:08:04.308415+0000","last_unstale":"2024-01-16T15:08:05.317974+0000","last_undegraded":"2024-01-16T15:08:05.317974+0000","last_fullsized":"2024-01-16T15:08:05.317974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:34.254450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.318750+0000","last_change":"2024-01-16T15:08:03.299481+0000","last_active":"2024-01-16T15:08:05.318750+0000","last_peered":"2024-01-16T15:08:05.318750+0000","last_clean":"2024-01-16T15:08:05.318750+0000","last_became_active":"2024-01-16T15:08:03.299064+0000","last_became_peered":"2024-01-16T15:08:03.299064+0000","last_unstale":"2024-01-16T15:08:05.318750+0000","last_undegraded":"2024-01-16T15:08:05.318750+0000","last_fullsized":"2024-01-16T15:08:05.318750+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:27.669781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876062+0000","last_change":"2024-01-16T15:08:04.316799+0000","last_active":"2024-01-16T15:08:05.876062+0000","last_peered":"2024-01-16T15:08:05.876062+0000","last_clean":"2024-01-16T15:08:05.876062+0000","last_became_active":"2024-01-16T15:08:04.316487+0000","last_became_peered":"2024-01-16T15:08:04.316487+0000","last_unstale":"2024-01-16T15:08:05.876062+0000","last_undegraded":"2024-01-16T15:08:05.876062+0000","last_fullsized":"2024-01-16T15:08:05.876062+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:58.169539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876566+0000","last_change":"2024-01-16T15:08:03.314311+0000","last_active":"2024-01-16T15:08:05.876566+0000","last_peered":"2024-01-16T15:08:05.876566+0000","last_clean":"2024-01-16T15:08:05.876566+0000","last_became_active":"2024-01-16T15:08:03.313852+0000","last_became_peered":"2024-01-16T15:08:03.313852+0000","last_unstale":"2024-01-16T15:08:05.876566+0000","last_undegraded":"2024-01-16T15:08:05.876566+0000","last_fullsized":"2024-01-16T15:08:05.876566+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:11.723335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.322459+0000","last_change":"2024-01-16T15:08:04.319584+0000","last_active":"2024-01-16T15:08:04.322459+0000","last_peered":"2024-01-16T15:08:04.322459+0000","last_clean":"2024-01-16T15:08:04.322459+0000","last_became_active":"2024-01-16T15:08:04.319243+0000","last_became_peered":"2024-01-16T15:08:04.319243+0000","last_unstale":"2024-01-16T15:08:04.322459+0000","last_undegraded":"2024-01-16T15:08:04.322459+0000","last_fullsized":"2024-01-16T15:08:04.322459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:11.309019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876611+0000","last_change":"2024-01-16T15:08:03.313819+0000","last_active":"2024-01-16T15:08:05.876611+0000","last_peered":"2024-01-16T15:08:05.876611+0000","last_clean":"2024-01-16T15:08:05.876611+0000","last_became_active":"2024-01-16T15:08:03.313445+0000","last_became_peered":"2024-01-16T15:08:03.313445+0000","last_unstale":"2024-01-16T15:08:05.876611+0000","last_undegraded":"2024-01-16T15:08:05.876611+0000","last_fullsized":"2024-01-16T15:08:05.876611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:33.361363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":213,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:08:03.297083+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:08:03.297083+0000","last_peered":"2024-01-16T15:08:03.297083+0000","last_clean":"2024-01-16T15:08:03.297083+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:08:03.297083+0000","last_undegraded":"2024-01-16T15:08:03.297083+0000","last_fullsized":"2024-01-16T15:08:03.297083+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509542,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45924,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325916,"statfs":{"total":96636764160,"available":96589737984,"internally_reserved":0,"allocated":876544,"data_stored":367691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.9419999999999999,"15min":1.429},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.55100000000000005},"max":{"1min":14.1,"5min":27.853999999999999,"15min":27.853999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.615,"15min":1.3180000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":4.149,"5min":19.827999999999999,"15min":19.827999999999999},"last":0.93899999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.607,"15min":1.3180000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":10.326000000000001,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.216},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.5900000000000001,"15min":1.361},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":3.6059999999999999,"5min":8.9469999999999992,"15min":8.9469999999999992},"last":1.6060000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.5920000000000001,"15min":1.456},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":10.696999999999999,"5min":10.696999999999999,"15min":10.696999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.625,"5min":1.7070000000000001,"15min":1.4790000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":5.9329999999999998,"5min":19.866,"15min":19.866},"last":1.3320000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":1.8540000000000001,"15min":1.571},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":16.353000000000002,"5min":16.353000000000002,"15min":16.353000000000002},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.7729999999999999,"15min":1.629},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":3.536,"5min":8.9009999999999998,"15min":8.9009999999999998},"last":1.423}]},{"osd":4,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.7669999999999999,"15min":1.5680000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":4.6310000000000002,"5min":19.707000000000001,"15min":19.707000000000001},"last":1.139},{"interface":"front","average":{"1min":2.1539999999999999,"5min":2.0409999999999999,"15min":1.7150000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":10.609999999999999,"5min":12.506,"15min":12.506},"last":1.696}]},{"osd":5,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.2200000000000002,"15min":1.875},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":10.388,"5min":12.606,"15min":12.606},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.823,"15min":1.7130000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.6850000000000001,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":1.5029999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":2.069,"15min":1.772},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":3.8300000000000001,"5min":33.155999999999999,"15min":33.155999999999999},"last":1.853},{"interface":"front","average":{"1min":2.1499999999999999,"5min":2.7909999999999999,"15min":2.081},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":11.553000000000001,"5min":46.305,"15min":46.305},"last":1.478}]}]},{"osd":4,"up_from":15,"seq":64424509544,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50084,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321756,"statfs":{"total":96636764160,"available":96585478144,"internally_reserved":0,"allocated":872448,"data_stored":367669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.9159999999999999,"15min":1.3460000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.0990000000000002,"5min":18.818000000000001,"15min":18.818000000000001},"last":2.0640000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.3959999999999999,"15min":1.214},"min":{"1min":0.46300000000000002,"5min":0.432,"15min":0.432},"max":{"1min":3.722,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.91800000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.9339999999999999,"15min":1.4359999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":5.742,"5min":18.347000000000001,"15min":18.347000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.708,"15min":1.4039999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.1429999999999998,"5min":18.858000000000001,"15min":18.858000000000001},"last":1.579}]},{"osd":2,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.996,"15min":1.522},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":4.3380000000000001,"5min":16.794,"15min":16.794},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.714,"15min":1.4470000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.226,"5min":18.905000000000001,"15min":18.905000000000001},"last":1.181}]},{"osd":3,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.944,"15min":1.5149999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.169,"5min":26.741,"15min":26.741},"last":1.361},{"interface":"front","average":{"1min":1.5720000000000001,"5min":2.23,"15min":1.718},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":5.1719999999999997,"5min":26.701000000000001,"15min":26.701000000000001},"last":1.5389999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.9019999999999999,"15min":1.663},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":2.0449999999999999,"5min":18.919,"15min":18.919},"last":1.444},{"interface":"front","average":{"1min":1.355,"5min":1.907,"15min":1.661},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.768,"5min":26.597000000000001,"15min":26.597000000000001},"last":1.768}]},{"osd":6,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.9299999999999999,"15min":1.6579999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.47699999999999998},"max":{"1min":4.2439999999999998,"5min":16.902000000000001,"15min":16.902000000000001},"last":1.74},{"interface":"front","average":{"1min":1.4590000000000001,"5min":2.0019999999999998,"15min":1.724},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":2.2240000000000002,"5min":18.952999999999999,"15min":18.952999999999999},"last":1.712}]},{"osd":7,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.946,"15min":1.6850000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.6259999999999999,"5min":26.683,"15min":26.683},"last":1.621},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.954,"15min":1.8320000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":4.6260000000000003,"5min":16.927,"15min":16.927},"last":1.819}]}]},{"osd":7,"up_from":14,"seq":60129542246,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1458176,"data_stored":952255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.875,"15min":1.3100000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":10.51,"5min":15.661,"15min":15.661},"last":2.9420000000000002},{"interface":"front","average":{"1min":3.431,"5min":2.0630000000000002,"15min":1.377},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":21.262,"5min":30.126000000000001,"15min":30.126000000000001},"last":5.399}]},{"osd":1,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.1859999999999999,"5min":1.4870000000000001,"15min":1.238},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":8.3149999999999995,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":1.125},{"interface":"front","average":{"1min":3.4380000000000002,"5min":2.0670000000000002,"15min":1.478},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":21.298999999999999,"5min":21.298999999999999,"15min":21.298999999999999},"last":6.8739999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.6799999999999999,"15min":1.3919999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":10.571999999999999,"5min":12.446999999999999,"15min":12.446999999999999},"last":4.9489999999999998},{"interface":"front","average":{"1min":1.9890000000000001,"5min":1.575,"15min":1.363},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":8.2759999999999998,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":1.5660000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.8149999999999999,"15min":1.5049999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":10.653,"5min":11.779,"15min":11.779},"last":3.7589999999999999},{"interface":"front","average":{"1min":2.6349999999999998,"5min":1.7290000000000001,"15min":1.4670000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":17.347999999999999,"5min":17.347999999999999,"15min":17.347999999999999},"last":3.9390000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:07:04 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.7689999999999999,"15min":1.5089999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.989000000000001,"5min":14.659000000000001,"15min":14.659000000000001},"last":2.996},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6160000000000001,"15min":1.468},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.8319999999999999,"5min":11.433,"15min":11.433},"last":2.351}]},{"osd":5,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":5.7130000000000001,"5min":2.4470000000000001,"15min":1.756},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":26.23,"5min":26.23,"15min":26.23},"last":7.899},{"interface":"front","average":{"1min":3.2610000000000001,"5min":2.1389999999999998,"15min":1.7509999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":21.312000000000001,"5min":21.312000000000001,"15min":21.312000000000001},"last":7.7539999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:07:04 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.6200000000000001,"15min":1.5720000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":5.8949999999999996,"5min":5.8949999999999996,"15min":5.8949999999999996},"last":1.7430000000000001},{"interface":"front","average":{"1min":3.5179999999999998,"5min":2.2930000000000001,"15min":1.8080000000000001},"min":{"1min":1.3140000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":20.881,"5min":29.991,"15min":29.991},"last":5.2830000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45916,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325924,"statfs":{"total":96636764160,"available":96589746176,"internally_reserved":0,"allocated":860160,"data_stored":361361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.3370000000000002,"5min":1.6739999999999999,"15min":1.2589999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":10.606,"5min":10.606,"15min":10.606},"last":0.872},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.486,"15min":1.244},"min":{"1min":0.68799999999999994,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.95,"5min":11.109,"15min":11.109},"last":1.153}]},{"osd":1,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.3660000000000001,"5min":1.821,"15min":1.3520000000000001},"min":{"1min":0.67400000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":13.837999999999999,"5min":17.693999999999999,"15min":17.693999999999999},"last":1.373},{"interface":"front","average":{"1min":2.5179999999999998,"5min":1.762,"15min":1.333},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":10.775,"5min":10.775,"15min":10.775},"last":1.002}]},{"osd":2,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.621,"5min":1.7450000000000001,"15min":1.383},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":10.702999999999999,"5min":10.702999999999999,"15min":10.702999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.649,"15min":1.3620000000000001},"min":{"1min":0.77400000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":3.0609999999999999,"5min":11.321,"15min":11.321},"last":1.3080000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.367,"5min":2.1890000000000001,"15min":1.619},"min":{"1min":0.93000000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":13.875,"5min":23.109000000000002,"15min":23.109000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":2.569,"5min":1.788,"15min":1.554},"min":{"1min":0.91000000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.645,"5min":10.645,"15min":14.647},"last":1.1100000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:07:02 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":2.194,"15min":1.603},"min":{"1min":0.90500000000000003,"5min":0.442,"15min":0.442},"max":{"1min":17.718,"5min":24.436,"15min":24.436},"last":1.657},{"interface":"front","average":{"1min":2.9630000000000001,"5min":1.7709999999999999,"15min":1.5189999999999999},"min":{"1min":0.68700000000000006,"5min":0.434,"15min":0.434},"max":{"1min":10.541,"5min":10.541,"15min":10.541},"last":1.5649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:07:02 2024","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":1.95,"15min":1.54},"min":{"1min":0.79000000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.401,"5min":21.192,"15min":21.192},"last":1.4199999999999999},{"interface":"front","average":{"1min":4.6399999999999997,"5min":2.2160000000000002,"15min":1.673},"min":{"1min":1.26,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":23.276,"5min":23.276,"15min":23.276},"last":1.849}]},{"osd":7,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":2.2970000000000002,"15min":1.7989999999999999},"min":{"1min":0.67000000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":10.605,"5min":32.405999999999999,"15min":32.405999999999999},"last":1.512},{"interface":"front","average":{"1min":2.9649999999999999,"5min":2.5299999999999998,"15min":1.9410000000000001},"min":{"1min":0.68200000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":13.715999999999999,"5min":24.367000000000001,"15min":24.367000000000001},"last":2.0640000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542247,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54236,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317604,"statfs":{"total":96636764160,"available":96581226496,"internally_reserved":0,"allocated":864256,"data_stored":361383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.375,"15min":1.012},"min":{"1min":0.55000000000000004,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.1120000000000001,"5min":31.562999999999999,"15min":31.562999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2250000000000001,"15min":1.0660000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.1050000000000004,"5min":8.4879999999999995,"15min":8.4879999999999995},"last":0.67800000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.1950000000000001,"15min":0.99399999999999999},"min":{"1min":0.53100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.506,"5min":15.569000000000001,"15min":15.569000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.357,"15min":1.153},"min":{"1min":0.69199999999999995,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.728,"5min":6.992,"15min":6.992},"last":1.216}]},{"osd":2,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.3779999999999999,"15min":1.208},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.36399999999999999},"max":{"1min":1.623,"5min":7.0529999999999999,"15min":7.0529999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2709999999999999,"15min":1.2110000000000001},"min":{"1min":0.67300000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.089,"5min":7.9729999999999999,"15min":14.577999999999999},"last":1.339}]},{"osd":4,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.7470000000000001,"15min":1.4450000000000001},"min":{"1min":1.042,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.641,"5min":20.978999999999999,"15min":20.978999999999999},"last":1.073},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.635,"15min":1.48},"min":{"1min":0.85099999999999998,"5min":0.748,"15min":0.748},"max":{"1min":2.2490000000000001,"5min":7.1379999999999999,"15min":7.1379999999999999},"last":1.115}]},{"osd":5,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":1.8640000000000001,"15min":1.5680000000000001},"min":{"1min":0.94199999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":20.663,"5min":20.663,"15min":20.663},"last":1.258},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.861,"15min":1.589},"min":{"1min":0.998,"5min":0.81399999999999995,"15min":0.76000000000000001},"max":{"1min":7.0460000000000003,"5min":17.613,"15min":17.613},"last":1.3029999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.875,"15min":1.5900000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.2850000000000001,"5min":13.182,"15min":13.182},"last":1.153},{"interface":"front","average":{"1min":2.5899999999999999,"5min":2.0649999999999999,"15min":1.6439999999999999},"min":{"1min":1.0269999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":20.593,"5min":20.593,"15min":20.593},"last":1.514}]},{"osd":7,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.6910000000000001,"15min":1.5409999999999999},"min":{"1min":1.0409999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":4.0679999999999996,"5min":5.4459999999999997,"15min":5.4459999999999997},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.671,"5min":2.7360000000000002,"15min":1.923},"min":{"1min":0.99099999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.8149999999999999,"5min":49.811,"15min":49.811},"last":1.3740000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50076,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321764,"statfs":{"total":96636764160,"available":96585486336,"internally_reserved":0,"allocated":868352,"data_stored":362883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.94499999999999995,"15min":0.98799999999999999},"min":{"1min":0.39600000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.391,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.874},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.081,"15min":1.04},"min":{"1min":0.35799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.7909999999999999,"5min":4.7809999999999997,"15min":4.7809999999999997},"last":0.81200000000000006}]},{"osd":1,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0580000000000001,"15min":1.044},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.3199999999999998,"5min":2.5779999999999998,"15min":3.839},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.1899999999999999,"15min":1.115},"min":{"1min":0.53100000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.8260000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.143}]},{"osd":3,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.077,"15min":0.99199999999999999},"min":{"1min":0.64300000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.611,"5min":2.6179999999999999,"15min":9.1530000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.3720000000000001,"15min":1.3320000000000001},"min":{"1min":0.623,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.0179999999999998,"5min":4.9610000000000003,"15min":4.9610000000000003},"last":1.268}]},{"osd":4,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.589,"15min":1.3979999999999999},"min":{"1min":0.69999999999999996,"5min":0.621,"15min":0.621},"max":{"1min":2.3820000000000001,"5min":18.016999999999999,"15min":18.016999999999999},"last":1.323},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.504,"15min":1.4470000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":3.2090000000000001,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":1.5569999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.623,"15min":1.429},"min":{"1min":0.84599999999999997,"5min":0.746,"15min":0.746},"max":{"1min":7.944,"5min":7.944,"15min":9.6189999999999998},"last":1.387},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.48,"15min":1.4199999999999999},"min":{"1min":0.90300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":6.6040000000000001,"5min":6.6040000000000001,"15min":9.423},"last":1.1950000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.913,"5min":1.5960000000000001,"15min":1.4650000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":9.5359999999999996},"last":1.504},{"interface":"front","average":{"1min":1.704,"5min":1.96,"15min":1.6599999999999999},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.72999999999999998},"max":{"1min":6.6870000000000003,"5min":21.564,"15min":21.564},"last":1.5940000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.5529999999999999,"15min":1.526},"min":{"1min":0.95099999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.5899999999999999,"5min":9.8610000000000007,"15min":9.8610000000000007},"last":1.6870000000000001},{"interface":"front","average":{"1min":2.3490000000000002,"5min":2.1560000000000001,"15min":1.7669999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":13.074,"5min":39.685000000000002,"15min":39.685000000000002},"last":1.637}]}]},{"osd":1,"up_from":14,"seq":60129542248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54244,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317596,"statfs":{"total":96636764160,"available":96581218304,"internally_reserved":0,"allocated":876544,"data_stored":363499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.92400000000000004,"15min":0.89200000000000002},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.278,"5min":2.278,"15min":10.667999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.3009999999999999,"15min":1.054},"min":{"1min":0.502,"5min":0.41699999999999998,"15min":0.39600000000000002},"max":{"1min":1.4019999999999999,"5min":17.5,"15min":17.5},"last":0.80900000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.2210000000000001,"15min":1.056},"min":{"1min":0.60799999999999998,"5min":0.52000000000000002,"15min":0.51000000000000001},"max":{"1min":1.538,"5min":6.6120000000000001,"15min":6.6120000000000001},"last":1.04},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.3420000000000001,"15min":1.2649999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6839999999999999,"5min":6.2759999999999998,"15min":10.805},"last":1.4390000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.6830000000000001,"15min":1.3340000000000001},"min":{"1min":0.55400000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.635,"5min":22.736000000000001,"15min":22.736000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.5269999999999999,"15min":1.2450000000000001},"min":{"1min":0.49399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.7110000000000001,"5min":17.361000000000001,"15min":17.361000000000001},"last":1.127}]},{"osd":4,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.609,"15min":1.3839999999999999},"min":{"1min":0.79400000000000004,"5min":0.628,"15min":0.628},"max":{"1min":9.0899999999999999,"5min":10.577999999999999,"15min":10.577999999999999},"last":1.181},{"interface":"front","average":{"1min":1.349,"5min":1.492,"15min":1.51},"min":{"1min":0.81299999999999994,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.696,"5min":4.5609999999999999,"15min":11.157999999999999},"last":1.5629999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.7529999999999999,"15min":1.49},"min":{"1min":0.84699999999999998,"5min":0.71199999999999997,"15min":0.68400000000000005},"max":{"1min":1.8620000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"last":1.296},{"interface":"front","average":{"1min":1.835,"5min":1.732,"15min":1.5409999999999999},"min":{"1min":0.88200000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":8.2989999999999995,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.6240000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":2,"15min":1.6479999999999999},"min":{"1min":0.80300000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.3540000000000001,"5min":17.428999999999998,"15min":17.428999999999998},"last":1.363},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1779999999999999,"15min":1.7030000000000001},"min":{"1min":0.92500000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":4.891,"5min":18.850999999999999,"15min":18.850999999999999},"last":1.5069999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":2.2469999999999999,"15min":1.7729999999999999},"min":{"1min":1.0389999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.6440000000000001,"5min":30.834,"15min":30.834},"last":1.413},{"interface":"front","average":{"1min":1.5189999999999999,"5min":2.0049999999999999,"15min":1.704},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.8460000000000001,"5min":27.263999999999999,"15min":27.263999999999999},"last":1.694}]}]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54804,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317036,"statfs":{"total":96636764160,"available":96580644864,"internally_reserved":0,"allocated":1449984,"data_stored":951639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.375,"15min":1.101},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.4900000000000002,"5min":22.132999999999999,"15min":22.132999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.651,"5min":1.919,"15min":1.337},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":12.41,"5min":50.429000000000002,"15min":50.429000000000002},"last":0.73499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.6040000000000001,"15min":1.3240000000000001},"min":{"1min":0.45300000000000001,"5min":0.38,"15min":0.38},"max":{"1min":4.4189999999999996,"5min":22.716999999999999,"15min":22.716999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.9379999999999999,"15min":1.397},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":7.2549999999999999,"5min":50.581000000000003,"15min":50.581000000000003},"last":1.0649999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.161,"15min":1.1679999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.716,"5min":4.6070000000000002,"15min":10.795},"last":1.583},{"interface":"front","average":{"1min":1.554,"5min":1.788,"15min":1.4910000000000001},"min":{"1min":0.56200000000000006,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":4.5750000000000002,"5min":22.254999999999999,"15min":22.254999999999999},"last":1.137}]},{"osd":4,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":2.266,"5min":2.1869999999999998,"15min":1.6659999999999999},"min":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":0.625},"max":{"1min":12.093,"5min":22.407,"15min":22.407},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.7809999999999999,"5min":2.423,"15min":1.708},"min":{"1min":0.76400000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":7.3200000000000003,"5min":50.497,"15min":50.497},"last":1.3560000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.8,"15min":1.5469999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":3.1880000000000002,"5min":14.112,"15min":14.112},"last":1.3149999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.238,"15min":1.75},"min":{"1min":0.79900000000000004,"5min":0.78000000000000003,"15min":0.71399999999999997},"max":{"1min":9.827,"5min":22.882999999999999,"15min":22.882999999999999},"last":1.6220000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":2.9950000000000001,"5min":2.1789999999999998,"15min":1.696},"min":{"1min":0.67500000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":31.387,"5min":31.387,"15min":31.387},"last":1.458},{"interface":"front","average":{"1min":2.2160000000000002,"5min":2.3959999999999999,"15min":1.8300000000000001},"min":{"1min":0.876,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":6.4889999999999999,"5min":23.547000000000001,"15min":23.547000000000001},"last":1.4990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0489999999999999,"5min":2.54,"15min":1.8560000000000001},"min":{"1min":0.84999999999999998,"5min":0.79300000000000004,"15min":0.75600000000000001},"max":{"1min":7.3769999999999998,"5min":50.542000000000002,"15min":50.542000000000002},"last":1.546},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.893,"15min":1.702},"min":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.6619999999999999,"5min":15.747,"15min":15.747},"last":1.6519999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:08:08.764 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:08:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.049+0000 7f6830400640 1 Processor -- start 2024-01-16T15:08:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.050+0000 7f6830400640 1 -- start start 2024-01-16T15:08:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.051+0000 7f6830400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6828143ae0 0x7f68280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.051+0000 7f6830400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f68280cf7d0 0x7f682815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.051+0000 7f6830400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682815e1e0 0x7f682815a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.051+0000 7f6830400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68280cfc30 con 0x7f6828143ae0 2024-01-16T15:08:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.051+0000 7f6830400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68280b1ee0 con 0x7f68280cf7d0 2024-01-16T15:08:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.051+0000 7f6830400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68280b2780 con 0x7f682815e1e0 2024-01-16T15:08:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.052+0000 7f682da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6828143ae0 0x7f68280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.052+0000 7f682da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6828143ae0 0x7f68280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45502/0 (socket says 172.21.15.53:45502) 2024-01-16T15:08:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.052+0000 7f682da00640 1 -- 172.21.15.53:0/2966713068 learned_addr learned my addr 172.21.15.53:0/2966713068 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.052+0000 7f682d000640 1 --2- 172.21.15.53:0/2966713068 >> v2:172.21.15.146:3300/0 conn(0x7f68280cf7d0 0x7f682815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.052+0000 7f682e400640 1 --2- 172.21.15.53:0/2966713068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682815e1e0 0x7f682815a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.053+0000 7f682da00640 1 -- 172.21.15.53:0/2966713068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682815e1e0 msgr2=0x7f682815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.053+0000 7f682da00640 1 --2- 172.21.15.53:0/2966713068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682815e1e0 0x7f682815a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.053+0000 7f682da00640 1 -- 172.21.15.53:0/2966713068 >> v2:172.21.15.146:3300/0 conn(0x7f68280cf7d0 msgr2=0x7f682815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.053+0000 7f682da00640 1 --2- 172.21.15.53:0/2966713068 >> v2:172.21.15.146:3300/0 conn(0x7f68280cf7d0 0x7f682815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.053+0000 7f682da00640 1 -- 172.21.15.53:0/2966713068 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682815e990 con 0x7f6828143ae0 2024-01-16T15:08:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.054+0000 7f682da00640 1 --2- 172.21.15.53:0/2966713068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6828143ae0 0x7f68280cf290 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f68280cb830 tx=0x7f681802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fe2e81cb45f0647 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.054+0000 7f6827e00640 1 -- 172.21.15.53:0/2966713068 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681802ca70 con 0x7f6828143ae0 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.054+0000 7f6827e00640 1 -- 172.21.15.53:0/2966713068 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681802cbd0 con 0x7f6828143ae0 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.054+0000 7f6827e00640 1 -- 172.21.15.53:0/2966713068 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68180343f0 con 0x7f6828143ae0 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 -- 172.21.15.53:0/2966713068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6828143ae0 msgr2=0x7f68280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 --2- 172.21.15.53:0/2966713068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6828143ae0 0x7f68280cf290 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f68280cb830 tx=0x7f681802bed0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 -- 172.21.15.53:0/2966713068 shutdown_connections 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 --2- 172.21.15.53:0/2966713068 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682815e1e0 0x7f682815a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 --2- 172.21.15.53:0/2966713068 >> v2:172.21.15.146:3300/0 conn(0x7f68280cf7d0 0x7f682815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 --2- 172.21.15.53:0/2966713068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6828143ae0 0x7f68280cf290 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 -- 172.21.15.53:0/2966713068 >> 172.21.15.53:0/2966713068 conn(0x7f682806a800 msgr2=0x7f682806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 -- 172.21.15.53:0/2966713068 shutdown_connections 2024-01-16T15:08:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.055+0000 7f6830400640 1 -- 172.21.15.53:0/2966713068 wait complete. 2024-01-16T15:08:09.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.057+0000 7f6830400640 1 Processor -- start 2024-01-16T15:08:09.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.057+0000 7f6830400640 1 -- start start 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f6830400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 0x7f6828157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f6830400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6828143ae0 0x7f68280a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f6830400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682815e1e0 0x7f682815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f6830400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68280a9520 con 0x7f68280cf7d0 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f6830400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682813b350 con 0x7f682815e1e0 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f6830400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68280a8f60 con 0x7f6828143ae0 2024-01-16T15:08:09.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 0x7f6828157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 0x7f6828157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45512/0 (socket says 172.21.15.53:45512) 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 -- 172.21.15.53:0/59874025 learned_addr learned my addr 172.21.15.53:0/59874025 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 -- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6828143ae0 msgr2=0x7f68280a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6828143ae0 0x7f68280a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682e400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682815e1e0 0x7f682815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 -- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682815e1e0 msgr2=0x7f682815ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682815e1e0 0x7f682815ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.058+0000 7f682da00640 1 -- 172.21.15.53:0/59874025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6818009b30 con 0x7f68280cf7d0 2024-01-16T15:08:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.059+0000 7f682da00640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 0x7f6828157a70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6818009e20 tx=0x7f6818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.059+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6818009510 con 0x7f68280cf7d0 2024-01-16T15:08:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.059+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68280a9240 con 0x7f68280cf7d0 2024-01-16T15:08:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.059+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f681803e450 con 0x7f68280cf7d0 2024-01-16T15:08:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.059+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f681803e7e0 con 0x7f68280cf7d0 2024-01-16T15:08:09.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.059+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6828157fb0 con 0x7f68280cf7d0 2024-01-16T15:08:09.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.061+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682813b1b0 con 0x7f68280cf7d0 2024-01-16T15:08:09.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.062+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f681803e940 con 0x7f68280cf7d0 2024-01-16T15:08:09.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.063+0000 7f6825600640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f680006bbc0 0x7f680006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.063+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f68180ab250 con 0x7f68280cf7d0 2024-01-16T15:08:09.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.063+0000 7f682d000640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f680006bbc0 0x7f680006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.065+0000 7f682d000640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f680006bbc0 0x7f680006e080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f68281f6610 tx=0x7f6810008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:09.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.067+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f682813b1b0 con 0x7f68280cf7d0 2024-01-16T15:08:09.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.256+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f682815e670 con 0x7f680006bbc0 2024-01-16T15:08:09.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.284+0000 7f6825600640 1 -- 172.21.15.53:0/59874025 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286515 (secure 0 0 0) 0x7f6810009480 con 0x7f680006bbc0 2024-01-16T15:08:09.270 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.289+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f680006bbc0 msgr2=0x7f680006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.289+0000 7f6830400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f680006bbc0 0x7f680006e080 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f68281f6610 tx=0x7f6810008040 comp rx=0 tx=0).stop 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.289+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 msgr2=0x7f6828157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.289+0000 7f6830400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 0x7f6828157a70 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f6818009e20 tx=0x7f6818004a40 comp rx=0 tx=0).stop 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 shutdown_connections 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f680006bbc0 0x7f680006e080 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682815e1e0 0x7f682815ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6828143ae0 0x7f68280a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 --2- 172.21.15.53:0/59874025 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68280cf7d0 0x7f6828157a70 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 >> 172.21.15.53:0/59874025 conn(0x7f682806a800 msgr2=0x7f68280af530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:09.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 shutdown_connections 2024-01-16T15:08:09.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.290+0000 7f6830400640 1 -- 172.21.15.53:0/59874025 wait complete. 2024-01-16T15:08:09.289 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-01-16T15:08:08.375888+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":125,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":405944,"kb_used_data":7936,"kb_used_omap":12,"kb_used_meta":397875,"kb_avail":754568776,"statfs":{"total":773094113280,"available":772678426624,"internally_reserved":0,"allocated":8126464,"data_stored":4088380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":407424560},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001938"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326974+0000","last_change":"2024-01-16T15:08:04.318085+0000","last_active":"2024-01-16T15:08:04.326974+0000","last_peered":"2024-01-16T15:08:04.326974+0000","last_clean":"2024-01-16T15:08:04.326974+0000","last_became_active":"2024-01-16T15:08:04.314969+0000","last_became_peered":"2024-01-16T15:08:04.314969+0000","last_unstale":"2024-01-16T15:08:04.326974+0000","last_undegraded":"2024-01-16T15:08:04.326974+0000","last_fullsized":"2024-01-16T15:08:04.326974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.041958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877459+0000","last_change":"2024-01-16T15:08:04.323490+0000","last_active":"2024-01-16T15:08:05.877459+0000","last_peered":"2024-01-16T15:08:05.877459+0000","last_clean":"2024-01-16T15:08:05.877459+0000","last_became_active":"2024-01-16T15:08:04.322913+0000","last_became_peered":"2024-01-16T15:08:04.322913+0000","last_unstale":"2024-01-16T15:08:05.877459+0000","last_undegraded":"2024-01-16T15:08:05.877459+0000","last_fullsized":"2024-01-16T15:08:05.877459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:07:45.702276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.312523+0000","last_change":"2024-01-16T15:08:04.307919+0000","last_active":"2024-01-16T15:08:04.312523+0000","last_peered":"2024-01-16T15:08:04.312523+0000","last_clean":"2024-01-16T15:08:04.312523+0000","last_became_active":"2024-01-16T15:08:04.307650+0000","last_became_peered":"2024-01-16T15:08:04.307650+0000","last_unstale":"2024-01-16T15:08:04.312523+0000","last_undegraded":"2024-01-16T15:08:04.312523+0000","last_fullsized":"2024-01-16T15:08:04.312523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:24:00.013331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317866+0000","last_change":"2024-01-16T15:08:04.319430+0000","last_active":"2024-01-16T15:08:05.317866+0000","last_peered":"2024-01-16T15:08:05.317866+0000","last_clean":"2024-01-16T15:08:05.317866+0000","last_became_active":"2024-01-16T15:08:04.318684+0000","last_became_peered":"2024-01-16T15:08:04.318684+0000","last_unstale":"2024-01-16T15:08:05.317866+0000","last_undegraded":"2024-01-16T15:08:05.317866+0000","last_fullsized":"2024-01-16T15:08:05.317866+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:41:22.339868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.321986+0000","last_change":"2024-01-16T15:08:04.319295+0000","last_active":"2024-01-16T15:08:04.321986+0000","last_peered":"2024-01-16T15:08:04.321986+0000","last_clean":"2024-01-16T15:08:04.321986+0000","last_became_active":"2024-01-16T15:08:04.319005+0000","last_became_peered":"2024-01-16T15:08:04.319005+0000","last_unstale":"2024-01-16T15:08:04.321986+0000","last_undegraded":"2024-01-16T15:08:04.321986+0000","last_fullsized":"2024-01-16T15:08:04.321986+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:40.492931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876359+0000","last_change":"2024-01-16T15:08:04.328640+0000","last_active":"2024-01-16T15:08:05.876359+0000","last_peered":"2024-01-16T15:08:05.876359+0000","last_clean":"2024-01-16T15:08:05.876359+0000","last_became_active":"2024-01-16T15:08:04.328356+0000","last_became_peered":"2024-01-16T15:08:04.328356+0000","last_unstale":"2024-01-16T15:08:05.876359+0000","last_undegraded":"2024-01-16T15:08:05.876359+0000","last_fullsized":"2024-01-16T15:08:05.876359+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:09:56.030381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.324330+0000","last_change":"2024-01-16T15:08:04.322407+0000","last_active":"2024-01-16T15:08:04.324330+0000","last_peered":"2024-01-16T15:08:04.324330+0000","last_clean":"2024-01-16T15:08:04.324330+0000","last_became_active":"2024-01-16T15:08:04.321973+0000","last_became_peered":"2024-01-16T15:08:04.321973+0000","last_unstale":"2024-01-16T15:08:04.324330+0000","last_undegraded":"2024-01-16T15:08:04.324330+0000","last_fullsized":"2024-01-16T15:08:04.324330+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:06:02.906263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876327+0000","last_change":"2024-01-16T15:08:04.313503+0000","last_active":"2024-01-16T15:08:05.876327+0000","last_peered":"2024-01-16T15:08:05.876327+0000","last_clean":"2024-01-16T15:08:05.876327+0000","last_became_active":"2024-01-16T15:08:04.312882+0000","last_became_peered":"2024-01-16T15:08:04.312882+0000","last_unstale":"2024-01-16T15:08:05.876327+0000","last_undegraded":"2024-01-16T15:08:05.876327+0000","last_fullsized":"2024-01-16T15:08:05.876327+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:48:04.785493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314392+0000","last_change":"2024-01-16T15:08:04.311731+0000","last_active":"2024-01-16T15:08:04.314392+0000","last_peered":"2024-01-16T15:08:04.314392+0000","last_clean":"2024-01-16T15:08:04.314392+0000","last_became_active":"2024-01-16T15:08:04.311413+0000","last_became_peered":"2024-01-16T15:08:04.311413+0000","last_unstale":"2024-01-16T15:08:04.314392+0000","last_undegraded":"2024-01-16T15:08:04.314392+0000","last_fullsized":"2024-01-16T15:08:04.314392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:38.851861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319644+0000","last_change":"2024-01-16T15:08:04.315893+0000","last_active":"2024-01-16T15:08:05.319644+0000","last_peered":"2024-01-16T15:08:05.319644+0000","last_clean":"2024-01-16T15:08:05.319644+0000","last_became_active":"2024-01-16T15:08:04.315435+0000","last_became_peered":"2024-01-16T15:08:04.315435+0000","last_unstale":"2024-01-16T15:08:05.319644+0000","last_undegraded":"2024-01-16T15:08:05.319644+0000","last_fullsized":"2024-01-16T15:08:05.319644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:27:40.947545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.311725+0000","last_change":"2024-01-16T15:08:04.308397+0000","last_active":"2024-01-16T15:08:04.311725+0000","last_peered":"2024-01-16T15:08:04.311725+0000","last_clean":"2024-01-16T15:08:04.311725+0000","last_became_active":"2024-01-16T15:08:04.308060+0000","last_became_peered":"2024-01-16T15:08:04.308060+0000","last_unstale":"2024-01-16T15:08:04.311725+0000","last_undegraded":"2024-01-16T15:08:04.311725+0000","last_fullsized":"2024-01-16T15:08:04.311725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:29.417861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317688+0000","last_change":"2024-01-16T15:08:04.316673+0000","last_active":"2024-01-16T15:08:05.317688+0000","last_peered":"2024-01-16T15:08:05.317688+0000","last_clean":"2024-01-16T15:08:05.317688+0000","last_became_active":"2024-01-16T15:08:04.312492+0000","last_became_peered":"2024-01-16T15:08:04.312492+0000","last_unstale":"2024-01-16T15:08:05.317688+0000","last_undegraded":"2024-01-16T15:08:05.317688+0000","last_fullsized":"2024-01-16T15:08:05.317688+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:37.200622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.320022+0000","last_change":"2024-01-16T15:08:04.317681+0000","last_active":"2024-01-16T15:08:04.320022+0000","last_peered":"2024-01-16T15:08:04.320022+0000","last_clean":"2024-01-16T15:08:04.320022+0000","last_became_active":"2024-01-16T15:08:04.317080+0000","last_became_peered":"2024-01-16T15:08:04.317080+0000","last_unstale":"2024-01-16T15:08:04.320022+0000","last_undegraded":"2024-01-16T15:08:04.320022+0000","last_fullsized":"2024-01-16T15:08:04.320022+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:19.115023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.331358+0000","last_change":"2024-01-16T15:08:04.329809+0000","last_active":"2024-01-16T15:08:04.331358+0000","last_peered":"2024-01-16T15:08:04.331358+0000","last_clean":"2024-01-16T15:08:04.331358+0000","last_became_active":"2024-01-16T15:08:04.329526+0000","last_became_peered":"2024-01-16T15:08:04.329526+0000","last_unstale":"2024-01-16T15:08:04.331358+0000","last_undegraded":"2024-01-16T15:08:04.331358+0000","last_fullsized":"2024-01-16T15:08:04.331358+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:25.877853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314705+0000","last_change":"2024-01-16T15:08:04.312436+0000","last_active":"2024-01-16T15:08:04.314705+0000","last_peered":"2024-01-16T15:08:04.314705+0000","last_clean":"2024-01-16T15:08:04.314705+0000","last_became_active":"2024-01-16T15:08:04.312131+0000","last_became_peered":"2024-01-16T15:08:04.312131+0000","last_unstale":"2024-01-16T15:08:04.314705+0000","last_undegraded":"2024-01-16T15:08:04.314705+0000","last_fullsized":"2024-01-16T15:08:04.314705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:31.718991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.320384+0000","last_change":"2024-01-16T15:08:04.317948+0000","last_active":"2024-01-16T15:08:05.320384+0000","last_peered":"2024-01-16T15:08:05.320384+0000","last_clean":"2024-01-16T15:08:05.320384+0000","last_became_active":"2024-01-16T15:08:04.316680+0000","last_became_peered":"2024-01-16T15:08:04.316680+0000","last_unstale":"2024-01-16T15:08:05.320384+0000","last_undegraded":"2024-01-16T15:08:05.320384+0000","last_fullsized":"2024-01-16T15:08:05.320384+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:55:12.054682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314412+0000","last_change":"2024-01-16T15:08:04.310501+0000","last_active":"2024-01-16T15:08:04.314412+0000","last_peered":"2024-01-16T15:08:04.314412+0000","last_clean":"2024-01-16T15:08:04.314412+0000","last_became_active":"2024-01-16T15:08:04.310158+0000","last_became_peered":"2024-01-16T15:08:04.310158+0000","last_unstale":"2024-01-16T15:08:04.314412+0000","last_undegraded":"2024-01-16T15:08:04.314412+0000","last_fullsized":"2024-01-16T15:08:04.314412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:02:26.755643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317705+0000","last_change":"2024-01-16T15:08:04.335659+0000","last_active":"2024-01-16T15:08:05.317705+0000","last_peered":"2024-01-16T15:08:05.317705+0000","last_clean":"2024-01-16T15:08:05.317705+0000","last_became_active":"2024-01-16T15:08:04.335181+0000","last_became_peered":"2024-01-16T15:08:04.335181+0000","last_unstale":"2024-01-16T15:08:05.317705+0000","last_undegraded":"2024-01-16T15:08:05.317705+0000","last_fullsized":"2024-01-16T15:08:05.317705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:32:16.523173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877090+0000","last_change":"2024-01-16T15:08:04.325332+0000","last_active":"2024-01-16T15:08:05.877090+0000","last_peered":"2024-01-16T15:08:05.877090+0000","last_clean":"2024-01-16T15:08:05.877090+0000","last_became_active":"2024-01-16T15:08:04.324175+0000","last_became_peered":"2024-01-16T15:08:04.324175+0000","last_unstale":"2024-01-16T15:08:05.877090+0000","last_undegraded":"2024-01-16T15:08:05.877090+0000","last_fullsized":"2024-01-16T15:08:05.877090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:46:00.978684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.324254+0000","last_change":"2024-01-16T15:08:04.322958+0000","last_active":"2024-01-16T15:08:04.324254+0000","last_peered":"2024-01-16T15:08:04.324254+0000","last_clean":"2024-01-16T15:08:04.324254+0000","last_became_active":"2024-01-16T15:08:04.321906+0000","last_became_peered":"2024-01-16T15:08:04.321906+0000","last_unstale":"2024-01-16T15:08:04.324254+0000","last_undegraded":"2024-01-16T15:08:04.324254+0000","last_fullsized":"2024-01-16T15:08:04.324254+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:11.951966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319286+0000","last_change":"2024-01-16T15:08:04.304645+0000","last_active":"2024-01-16T15:08:05.319286+0000","last_peered":"2024-01-16T15:08:05.319286+0000","last_clean":"2024-01-16T15:08:05.319286+0000","last_became_active":"2024-01-16T15:08:04.304054+0000","last_became_peered":"2024-01-16T15:08:04.304054+0000","last_unstale":"2024-01-16T15:08:05.319286+0000","last_undegraded":"2024-01-16T15:08:05.319286+0000","last_fullsized":"2024-01-16T15:08:05.319286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:19:41.087608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319262+0000","last_change":"2024-01-16T15:08:04.314852+0000","last_active":"2024-01-16T15:08:05.319262+0000","last_peered":"2024-01-16T15:08:05.319262+0000","last_clean":"2024-01-16T15:08:05.319262+0000","last_became_active":"2024-01-16T15:08:04.314559+0000","last_became_peered":"2024-01-16T15:08:04.314559+0000","last_unstale":"2024-01-16T15:08:05.319262+0000","last_undegraded":"2024-01-16T15:08:05.319262+0000","last_fullsized":"2024-01-16T15:08:05.319262+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:33:28.389208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.337285+0000","last_change":"2024-01-16T15:08:04.332155+0000","last_active":"2024-01-16T15:08:04.337285+0000","last_peered":"2024-01-16T15:08:04.337285+0000","last_clean":"2024-01-16T15:08:04.337285+0000","last_became_active":"2024-01-16T15:08:04.329534+0000","last_became_peered":"2024-01-16T15:08:04.329534+0000","last_unstale":"2024-01-16T15:08:04.337285+0000","last_undegraded":"2024-01-16T15:08:04.337285+0000","last_fullsized":"2024-01-16T15:08:04.337285+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:19.115023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.338624+0000","last_change":"2024-01-16T15:08:04.332782+0000","last_active":"2024-01-16T15:08:04.338624+0000","last_peered":"2024-01-16T15:08:04.338624+0000","last_clean":"2024-01-16T15:08:04.338624+0000","last_became_active":"2024-01-16T15:08:04.332517+0000","last_became_peered":"2024-01-16T15:08:04.332517+0000","last_unstale":"2024-01-16T15:08:04.338624+0000","last_undegraded":"2024-01-16T15:08:04.338624+0000","last_fullsized":"2024-01-16T15:08:04.338624+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:20.168426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319646+0000","last_change":"2024-01-16T15:08:04.304272+0000","last_active":"2024-01-16T15:08:05.319646+0000","last_peered":"2024-01-16T15:08:05.319646+0000","last_clean":"2024-01-16T15:08:05.319646+0000","last_became_active":"2024-01-16T15:08:04.302567+0000","last_became_peered":"2024-01-16T15:08:04.302567+0000","last_unstale":"2024-01-16T15:08:05.319646+0000","last_undegraded":"2024-01-16T15:08:05.319646+0000","last_fullsized":"2024-01-16T15:08:05.319646+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:01.847441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336572+0000","last_change":"2024-01-16T15:08:04.335315+0000","last_active":"2024-01-16T15:08:04.336572+0000","last_peered":"2024-01-16T15:08:04.336572+0000","last_clean":"2024-01-16T15:08:04.336572+0000","last_became_active":"2024-01-16T15:08:04.335040+0000","last_became_peered":"2024-01-16T15:08:04.335040+0000","last_unstale":"2024-01-16T15:08:04.336572+0000","last_undegraded":"2024-01-16T15:08:04.336572+0000","last_fullsized":"2024-01-16T15:08:04.336572+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:25.214223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.321116+0000","last_change":"2024-01-16T15:08:04.318374+0000","last_active":"2024-01-16T15:08:04.321116+0000","last_peered":"2024-01-16T15:08:04.321116+0000","last_clean":"2024-01-16T15:08:04.321116+0000","last_became_active":"2024-01-16T15:08:04.314762+0000","last_became_peered":"2024-01-16T15:08:04.314762+0000","last_unstale":"2024-01-16T15:08:04.321116+0000","last_undegraded":"2024-01-16T15:08:04.321116+0000","last_fullsized":"2024-01-16T15:08:04.321116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:19:42.934564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319404+0000","last_change":"2024-01-16T15:08:04.316813+0000","last_active":"2024-01-16T15:08:05.319404+0000","last_peered":"2024-01-16T15:08:05.319404+0000","last_clean":"2024-01-16T15:08:05.319404+0000","last_became_active":"2024-01-16T15:08:04.316456+0000","last_became_peered":"2024-01-16T15:08:04.316456+0000","last_unstale":"2024-01-16T15:08:05.319404+0000","last_undegraded":"2024-01-16T15:08:05.319404+0000","last_fullsized":"2024-01-16T15:08:05.319404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:37.858357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.323205+0000","last_change":"2024-01-16T15:08:04.321861+0000","last_active":"2024-01-16T15:08:04.323205+0000","last_peered":"2024-01-16T15:08:04.323205+0000","last_clean":"2024-01-16T15:08:04.323205+0000","last_became_active":"2024-01-16T15:08:04.321437+0000","last_became_peered":"2024-01-16T15:08:04.321437+0000","last_unstale":"2024-01-16T15:08:04.323205+0000","last_undegraded":"2024-01-16T15:08:04.323205+0000","last_fullsized":"2024-01-16T15:08:04.323205+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:29.185899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.338017+0000","last_change":"2024-01-16T15:08:04.336138+0000","last_active":"2024-01-16T15:08:04.338017+0000","last_peered":"2024-01-16T15:08:04.338017+0000","last_clean":"2024-01-16T15:08:04.338017+0000","last_became_active":"2024-01-16T15:08:04.335717+0000","last_became_peered":"2024-01-16T15:08:04.335717+0000","last_unstale":"2024-01-16T15:08:04.338017+0000","last_undegraded":"2024-01-16T15:08:04.338017+0000","last_fullsized":"2024-01-16T15:08:04.338017+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:39:19.115023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336473+0000","last_change":"2024-01-16T15:08:04.333829+0000","last_active":"2024-01-16T15:08:04.336473+0000","last_peered":"2024-01-16T15:08:04.336473+0000","last_clean":"2024-01-16T15:08:04.336473+0000","last_became_active":"2024-01-16T15:08:04.333555+0000","last_became_peered":"2024-01-16T15:08:04.333555+0000","last_unstale":"2024-01-16T15:08:04.336473+0000","last_undegraded":"2024-01-16T15:08:04.336473+0000","last_fullsized":"2024-01-16T15:08:04.336473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:19:06.794696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876488+0000","last_change":"2024-01-16T15:08:04.320098+0000","last_active":"2024-01-16T15:08:05.876488+0000","last_peered":"2024-01-16T15:08:05.876488+0000","last_clean":"2024-01-16T15:08:05.876488+0000","last_became_active":"2024-01-16T15:08:04.319356+0000","last_became_peered":"2024-01-16T15:08:04.319356+0000","last_unstale":"2024-01-16T15:08:05.876488+0000","last_undegraded":"2024-01-16T15:08:05.876488+0000","last_fullsized":"2024-01-16T15:08:05.876488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:09:19.111663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336747+0000","last_change":"2024-01-16T15:08:04.331908+0000","last_active":"2024-01-16T15:08:04.336747+0000","last_peered":"2024-01-16T15:08:04.336747+0000","last_clean":"2024-01-16T15:08:04.336747+0000","last_became_active":"2024-01-16T15:08:04.331565+0000","last_became_peered":"2024-01-16T15:08:04.331565+0000","last_unstale":"2024-01-16T15:08:04.336747+0000","last_undegraded":"2024-01-16T15:08:04.336747+0000","last_fullsized":"2024-01-16T15:08:04.336747+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:11.951966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"71'4","reported_seq":15,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.337089+0000","last_change":"2024-01-16T15:08:03.311514+0000","last_active":"2024-01-16T15:08:04.337089+0000","last_peered":"2024-01-16T15:08:04.337089+0000","last_clean":"2024-01-16T15:08:04.337089+0000","last_became_active":"2024-01-16T15:08:03.311238+0000","last_became_peered":"2024-01-16T15:08:03.311238+0000","last_unstale":"2024-01-16T15:08:04.337089+0000","last_undegraded":"2024-01-16T15:08:04.337089+0000","last_fullsized":"2024-01-16T15:08:04.337089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:36.717980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.312524+0000","last_change":"2024-01-16T15:08:04.310480+0000","last_active":"2024-01-16T15:08:04.312524+0000","last_peered":"2024-01-16T15:08:04.312524+0000","last_clean":"2024-01-16T15:08:04.312524+0000","last_became_active":"2024-01-16T15:08:04.310121+0000","last_became_peered":"2024-01-16T15:08:04.310121+0000","last_unstale":"2024-01-16T15:08:04.312524+0000","last_undegraded":"2024-01-16T15:08:04.312524+0000","last_fullsized":"2024-01-16T15:08:04.312524+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:51:54.608877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.331753+0000","last_change":"2024-01-16T15:08:03.312504+0000","last_active":"2024-01-16T15:08:04.331753+0000","last_peered":"2024-01-16T15:08:04.331753+0000","last_clean":"2024-01-16T15:08:04.331753+0000","last_became_active":"2024-01-16T15:08:03.312176+0000","last_became_peered":"2024-01-16T15:08:03.312176+0000","last_unstale":"2024-01-16T15:08:04.331753+0000","last_undegraded":"2024-01-16T15:08:04.331753+0000","last_fullsized":"2024-01-16T15:08:04.331753+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:02.787365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876502+0000","last_change":"2024-01-16T15:08:04.315592+0000","last_active":"2024-01-16T15:08:05.876502+0000","last_peered":"2024-01-16T15:08:05.876502+0000","last_clean":"2024-01-16T15:08:05.876502+0000","last_became_active":"2024-01-16T15:08:04.315286+0000","last_became_peered":"2024-01-16T15:08:04.315286+0000","last_unstale":"2024-01-16T15:08:05.876502+0000","last_undegraded":"2024-01-16T15:08:05.876502+0000","last_fullsized":"2024-01-16T15:08:05.876502+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:49.221249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319927+0000","last_change":"2024-01-16T15:08:03.312380+0000","last_active":"2024-01-16T15:08:05.319927+0000","last_peered":"2024-01-16T15:08:05.319927+0000","last_clean":"2024-01-16T15:08:05.319927+0000","last_became_active":"2024-01-16T15:08:03.311822+0000","last_became_peered":"2024-01-16T15:08:03.311822+0000","last_unstale":"2024-01-16T15:08:05.319927+0000","last_undegraded":"2024-01-16T15:08:05.319927+0000","last_fullsized":"2024-01-16T15:08:05.319927+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:58:46.968161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.311703+0000","last_change":"2024-01-16T15:08:04.300254+0000","last_active":"2024-01-16T15:08:04.311703+0000","last_peered":"2024-01-16T15:08:04.311703+0000","last_clean":"2024-01-16T15:08:04.311703+0000","last_became_active":"2024-01-16T15:08:04.299401+0000","last_became_peered":"2024-01-16T15:08:04.299401+0000","last_unstale":"2024-01-16T15:08:04.311703+0000","last_undegraded":"2024-01-16T15:08:04.311703+0000","last_fullsized":"2024-01-16T15:08:04.311703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:18:31.718991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876773+0000","last_change":"2024-01-16T15:08:03.315357+0000","last_active":"2024-01-16T15:08:05.876773+0000","last_peered":"2024-01-16T15:08:05.876773+0000","last_clean":"2024-01-16T15:08:05.876773+0000","last_became_active":"2024-01-16T15:08:03.315064+0000","last_became_peered":"2024-01-16T15:08:03.315064+0000","last_unstale":"2024-01-16T15:08:05.876773+0000","last_undegraded":"2024-01-16T15:08:05.876773+0000","last_fullsized":"2024-01-16T15:08:05.876773+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:25.472504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.329729+0000","last_change":"2024-01-16T15:08:04.326228+0000","last_active":"2024-01-16T15:08:04.329729+0000","last_peered":"2024-01-16T15:08:04.329729+0000","last_clean":"2024-01-16T15:08:04.329729+0000","last_became_active":"2024-01-16T15:08:04.325893+0000","last_became_peered":"2024-01-16T15:08:04.325893+0000","last_unstale":"2024-01-16T15:08:04.329729+0000","last_undegraded":"2024-01-16T15:08:04.329729+0000","last_fullsized":"2024-01-16T15:08:04.329729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:20.168426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"71'1","reported_seq":12,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.329818+0000","last_change":"2024-01-16T15:08:03.310423+0000","last_active":"2024-01-16T15:08:04.329818+0000","last_peered":"2024-01-16T15:08:04.329818+0000","last_clean":"2024-01-16T15:08:04.329818+0000","last_became_active":"2024-01-16T15:08:03.308253+0000","last_became_peered":"2024-01-16T15:08:03.308253+0000","last_unstale":"2024-01-16T15:08:04.329818+0000","last_undegraded":"2024-01-16T15:08:04.329818+0000","last_fullsized":"2024-01-16T15:08:04.329818+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:34.374011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319288+0000","last_change":"2024-01-16T15:08:04.301339+0000","last_active":"2024-01-16T15:08:05.319288+0000","last_peered":"2024-01-16T15:08:05.319288+0000","last_clean":"2024-01-16T15:08:05.319288+0000","last_became_active":"2024-01-16T15:08:04.300419+0000","last_became_peered":"2024-01-16T15:08:04.300419+0000","last_unstale":"2024-01-16T15:08:05.319288+0000","last_undegraded":"2024-01-16T15:08:05.319288+0000","last_fullsized":"2024-01-16T15:08:05.319288+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:24:04.609348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"71'1","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876629+0000","last_change":"2024-01-16T15:08:03.312036+0000","last_active":"2024-01-16T15:08:05.876629+0000","last_peered":"2024-01-16T15:08:05.876629+0000","last_clean":"2024-01-16T15:08:05.876629+0000","last_became_active":"2024-01-16T15:08:03.311300+0000","last_became_peered":"2024-01-16T15:08:03.311300+0000","last_unstale":"2024-01-16T15:08:05.876629+0000","last_undegraded":"2024-01-16T15:08:05.876629+0000","last_fullsized":"2024-01-16T15:08:05.876629+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:31.818904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.315158+0000","last_change":"2024-01-16T15:08:04.311575+0000","last_active":"2024-01-16T15:08:04.315158+0000","last_peered":"2024-01-16T15:08:04.315158+0000","last_clean":"2024-01-16T15:08:04.315158+0000","last_became_active":"2024-01-16T15:08:04.310670+0000","last_became_peered":"2024-01-16T15:08:04.310670+0000","last_unstale":"2024-01-16T15:08:04.315158+0000","last_undegraded":"2024-01-16T15:08:04.315158+0000","last_fullsized":"2024-01-16T15:08:04.315158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:03:51.644650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317116+0000","last_change":"2024-01-16T15:08:03.310875+0000","last_active":"2024-01-16T15:08:05.317116+0000","last_peered":"2024-01-16T15:08:05.317116+0000","last_clean":"2024-01-16T15:08:05.317116+0000","last_became_active":"2024-01-16T15:08:03.310276+0000","last_became_peered":"2024-01-16T15:08:03.310276+0000","last_unstale":"2024-01-16T15:08:05.317116+0000","last_undegraded":"2024-01-16T15:08:05.317116+0000","last_fullsized":"2024-01-16T15:08:05.317116+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:47:14.858194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.309613+0000","last_change":"2024-01-16T15:08:04.307417+0000","last_active":"2024-01-16T15:08:04.309613+0000","last_peered":"2024-01-16T15:08:04.309613+0000","last_clean":"2024-01-16T15:08:04.309613+0000","last_became_active":"2024-01-16T15:08:04.306078+0000","last_became_peered":"2024-01-16T15:08:04.306078+0000","last_unstale":"2024-01-16T15:08:04.309613+0000","last_undegraded":"2024-01-16T15:08:04.309613+0000","last_fullsized":"2024-01-16T15:08:04.309613+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:21.211461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.296394+0000","last_change":"2024-01-16T15:08:03.304692+0000","last_active":"2024-01-16T15:08:04.296394+0000","last_peered":"2024-01-16T15:08:04.296394+0000","last_clean":"2024-01-16T15:08:04.296394+0000","last_became_active":"2024-01-16T15:08:03.303212+0000","last_became_peered":"2024-01-16T15:08:03.303212+0000","last_unstale":"2024-01-16T15:08:04.296394+0000","last_undegraded":"2024-01-16T15:08:04.296394+0000","last_fullsized":"2024-01-16T15:08:04.296394+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:59:46.421758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.336488+0000","last_change":"2024-01-16T15:08:04.334738+0000","last_active":"2024-01-16T15:08:04.336488+0000","last_peered":"2024-01-16T15:08:04.336488+0000","last_clean":"2024-01-16T15:08:04.336488+0000","last_became_active":"2024-01-16T15:08:04.334408+0000","last_became_peered":"2024-01-16T15:08:04.334408+0000","last_unstale":"2024-01-16T15:08:04.336488+0000","last_undegraded":"2024-01-16T15:08:04.336488+0000","last_fullsized":"2024-01-16T15:08:04.336488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:00:56.787564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319743+0000","last_change":"2024-01-16T15:08:03.308379+0000","last_active":"2024-01-16T15:08:05.319743+0000","last_peered":"2024-01-16T15:08:05.319743+0000","last_clean":"2024-01-16T15:08:05.319743+0000","last_became_active":"2024-01-16T15:08:03.308095+0000","last_became_peered":"2024-01-16T15:08:03.308095+0000","last_unstale":"2024-01-16T15:08:05.319743+0000","last_undegraded":"2024-01-16T15:08:05.319743+0000","last_fullsized":"2024-01-16T15:08:05.319743+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:49.003048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326951+0000","last_change":"2024-01-16T15:08:04.319354+0000","last_active":"2024-01-16T15:08:04.326951+0000","last_peered":"2024-01-16T15:08:04.326951+0000","last_clean":"2024-01-16T15:08:04.326951+0000","last_became_active":"2024-01-16T15:08:04.319032+0000","last_became_peered":"2024-01-16T15:08:04.319032+0000","last_unstale":"2024-01-16T15:08:04.326951+0000","last_undegraded":"2024-01-16T15:08:04.326951+0000","last_fullsized":"2024-01-16T15:08:04.326951+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:16.809488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877506+0000","last_change":"2024-01-16T15:08:03.316929+0000","last_active":"2024-01-16T15:08:05.877506+0000","last_peered":"2024-01-16T15:08:05.877506+0000","last_clean":"2024-01-16T15:08:05.877506+0000","last_became_active":"2024-01-16T15:08:03.316201+0000","last_became_peered":"2024-01-16T15:08:03.316201+0000","last_unstale":"2024-01-16T15:08:05.877506+0000","last_undegraded":"2024-01-16T15:08:05.877506+0000","last_fullsized":"2024-01-16T15:08:05.877506+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:11:38.592403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876606+0000","last_change":"2024-01-16T15:08:04.326679+0000","last_active":"2024-01-16T15:08:05.876606+0000","last_peered":"2024-01-16T15:08:05.876606+0000","last_clean":"2024-01-16T15:08:05.876606+0000","last_became_active":"2024-01-16T15:08:04.323081+0000","last_became_peered":"2024-01-16T15:08:04.323081+0000","last_unstale":"2024-01-16T15:08:05.876606+0000","last_undegraded":"2024-01-16T15:08:05.876606+0000","last_fullsized":"2024-01-16T15:08:05.876606+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:01:17.712427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"71'3","reported_seq":14,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.330987+0000","last_change":"2024-01-16T15:08:03.313202+0000","last_active":"2024-01-16T15:08:04.330987+0000","last_peered":"2024-01-16T15:08:04.330987+0000","last_clean":"2024-01-16T15:08:04.330987+0000","last_became_active":"2024-01-16T15:08:03.312944+0000","last_became_peered":"2024-01-16T15:08:03.312944+0000","last_unstale":"2024-01-16T15:08:04.330987+0000","last_undegraded":"2024-01-16T15:08:04.330987+0000","last_fullsized":"2024-01-16T15:08:04.330987+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:52:07.334338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.315082+0000","last_change":"2024-01-16T15:08:04.311198+0000","last_active":"2024-01-16T15:08:04.315082+0000","last_peered":"2024-01-16T15:08:04.315082+0000","last_clean":"2024-01-16T15:08:04.315082+0000","last_became_active":"2024-01-16T15:08:04.310912+0000","last_became_peered":"2024-01-16T15:08:04.310912+0000","last_unstale":"2024-01-16T15:08:04.315082+0000","last_undegraded":"2024-01-16T15:08:04.315082+0000","last_fullsized":"2024-01-16T15:08:04.315082+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:20:21.001264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"71'6","reported_seq":18,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.318016+0000","last_change":"2024-01-16T15:08:03.311820+0000","last_active":"2024-01-16T15:08:05.318016+0000","last_peered":"2024-01-16T15:08:05.318016+0000","last_clean":"2024-01-16T15:08:05.318016+0000","last_became_active":"2024-01-16T15:08:03.311542+0000","last_became_peered":"2024-01-16T15:08:03.311542+0000","last_unstale":"2024-01-16T15:08:05.318016+0000","last_undegraded":"2024-01-16T15:08:05.318016+0000","last_fullsized":"2024-01-16T15:08:05.318016+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:32:58.716201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.311323+0000","last_change":"2024-01-16T15:08:04.309096+0000","last_active":"2024-01-16T15:08:04.311323+0000","last_peered":"2024-01-16T15:08:04.311323+0000","last_clean":"2024-01-16T15:08:04.311323+0000","last_became_active":"2024-01-16T15:08:04.308828+0000","last_became_peered":"2024-01-16T15:08:04.308828+0000","last_unstale":"2024-01-16T15:08:04.311323+0000","last_undegraded":"2024-01-16T15:08:04.311323+0000","last_fullsized":"2024-01-16T15:08:04.311323+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:30.062633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317282+0000","last_change":"2024-01-16T15:08:03.307012+0000","last_active":"2024-01-16T15:08:05.317282+0000","last_peered":"2024-01-16T15:08:05.317282+0000","last_clean":"2024-01-16T15:08:05.317282+0000","last_became_active":"2024-01-16T15:08:03.306725+0000","last_became_peered":"2024-01-16T15:08:03.306725+0000","last_unstale":"2024-01-16T15:08:05.317282+0000","last_undegraded":"2024-01-16T15:08:05.317282+0000","last_fullsized":"2024-01-16T15:08:05.317282+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:50:59.878283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.332766+0000","last_change":"2024-01-16T15:08:04.330823+0000","last_active":"2024-01-16T15:08:04.332766+0000","last_peered":"2024-01-16T15:08:04.332766+0000","last_clean":"2024-01-16T15:08:04.332766+0000","last_became_active":"2024-01-16T15:08:04.330553+0000","last_became_peered":"2024-01-16T15:08:04.330553+0000","last_unstale":"2024-01-16T15:08:04.332766+0000","last_undegraded":"2024-01-16T15:08:04.332766+0000","last_fullsized":"2024-01-16T15:08:04.332766+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:57:56.702039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876868+0000","last_change":"2024-01-16T15:08:03.315290+0000","last_active":"2024-01-16T15:08:05.876868+0000","last_peered":"2024-01-16T15:08:05.876868+0000","last_clean":"2024-01-16T15:08:05.876868+0000","last_became_active":"2024-01-16T15:08:03.314984+0000","last_became_peered":"2024-01-16T15:08:03.314984+0000","last_unstale":"2024-01-16T15:08:05.876868+0000","last_undegraded":"2024-01-16T15:08:05.876868+0000","last_fullsized":"2024-01-16T15:08:05.876868+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:02:49.155475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.329448+0000","last_change":"2024-01-16T15:08:04.327612+0000","last_active":"2024-01-16T15:08:04.329448+0000","last_peered":"2024-01-16T15:08:04.329448+0000","last_clean":"2024-01-16T15:08:04.329448+0000","last_became_active":"2024-01-16T15:08:04.327246+0000","last_became_peered":"2024-01-16T15:08:04.327246+0000","last_unstale":"2024-01-16T15:08:04.329448+0000","last_undegraded":"2024-01-16T15:08:04.329448+0000","last_fullsized":"2024-01-16T15:08:04.329448+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:15:29.417861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314907+0000","last_change":"2024-01-16T15:08:03.316462+0000","last_active":"2024-01-16T15:08:04.314907+0000","last_peered":"2024-01-16T15:08:04.314907+0000","last_clean":"2024-01-16T15:08:04.314907+0000","last_became_active":"2024-01-16T15:08:03.316004+0000","last_became_peered":"2024-01-16T15:08:03.316004+0000","last_unstale":"2024-01-16T15:08:04.314907+0000","last_undegraded":"2024-01-16T15:08:04.314907+0000","last_fullsized":"2024-01-16T15:08:04.314907+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:14:18.148893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.309907+0000","last_change":"2024-01-16T15:08:04.308122+0000","last_active":"2024-01-16T15:08:04.309907+0000","last_peered":"2024-01-16T15:08:04.309907+0000","last_clean":"2024-01-16T15:08:04.309907+0000","last_became_active":"2024-01-16T15:08:04.307828+0000","last_became_peered":"2024-01-16T15:08:04.307828+0000","last_unstale":"2024-01-16T15:08:04.309907+0000","last_undegraded":"2024-01-16T15:08:04.309907+0000","last_fullsized":"2024-01-16T15:08:04.309907+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:53:17.628006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"71'8","reported_seq":20,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.318817+0000","last_change":"2024-01-16T15:08:03.297584+0000","last_active":"2024-01-16T15:08:05.318817+0000","last_peered":"2024-01-16T15:08:05.318817+0000","last_clean":"2024-01-16T15:08:05.318817+0000","last_became_active":"2024-01-16T15:08:03.297283+0000","last_became_peered":"2024-01-16T15:08:03.297283+0000","last_unstale":"2024-01-16T15:08:05.318817+0000","last_undegraded":"2024-01-16T15:08:05.318817+0000","last_fullsized":"2024-01-16T15:08:05.318817+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:09:12.124956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319763+0000","last_change":"2024-01-16T15:08:04.315716+0000","last_active":"2024-01-16T15:08:05.319763+0000","last_peered":"2024-01-16T15:08:05.319763+0000","last_clean":"2024-01-16T15:08:05.319763+0000","last_became_active":"2024-01-16T15:08:04.314955+0000","last_became_peered":"2024-01-16T15:08:04.314955+0000","last_unstale":"2024-01-16T15:08:05.319763+0000","last_undegraded":"2024-01-16T15:08:05.319763+0000","last_fullsized":"2024-01-16T15:08:05.319763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:31:17.749199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"71'1","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877104+0000","last_change":"2024-01-16T15:08:03.314796+0000","last_active":"2024-01-16T15:08:05.877104+0000","last_peered":"2024-01-16T15:08:05.877104+0000","last_clean":"2024-01-16T15:08:05.877104+0000","last_became_active":"2024-01-16T15:08:03.314401+0000","last_became_peered":"2024-01-16T15:08:03.314401+0000","last_unstale":"2024-01-16T15:08:05.877104+0000","last_undegraded":"2024-01-16T15:08:05.877104+0000","last_fullsized":"2024-01-16T15:08:05.877104+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:35:23.014039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.319485+0000","last_change":"2024-01-16T15:08:04.315796+0000","last_active":"2024-01-16T15:08:04.319485+0000","last_peered":"2024-01-16T15:08:04.319485+0000","last_clean":"2024-01-16T15:08:04.319485+0000","last_became_active":"2024-01-16T15:08:04.311547+0000","last_became_peered":"2024-01-16T15:08:04.311547+0000","last_unstale":"2024-01-16T15:08:04.319485+0000","last_undegraded":"2024-01-16T15:08:04.319485+0000","last_fullsized":"2024-01-16T15:08:04.319485+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:23:18.007918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.319792+0000","last_change":"2024-01-16T15:08:03.317728+0000","last_active":"2024-01-16T15:08:05.319792+0000","last_peered":"2024-01-16T15:08:05.319792+0000","last_clean":"2024-01-16T15:08:05.319792+0000","last_became_active":"2024-01-16T15:08:03.317115+0000","last_became_peered":"2024-01-16T15:08:03.317115+0000","last_unstale":"2024-01-16T15:08:05.319792+0000","last_undegraded":"2024-01-16T15:08:05.319792+0000","last_fullsized":"2024-01-16T15:08:05.319792+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:31.426794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317903+0000","last_change":"2024-01-16T15:08:04.312648+0000","last_active":"2024-01-16T15:08:05.317903+0000","last_peered":"2024-01-16T15:08:05.317903+0000","last_clean":"2024-01-16T15:08:05.317903+0000","last_became_active":"2024-01-16T15:08:04.312334+0000","last_became_peered":"2024-01-16T15:08:04.312334+0000","last_unstale":"2024-01-16T15:08:05.317903+0000","last_undegraded":"2024-01-16T15:08:05.317903+0000","last_fullsized":"2024-01-16T15:08:05.317903+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:06:21.368154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"71'1","reported_seq":13,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877275+0000","last_change":"2024-01-16T15:08:03.314555+0000","last_active":"2024-01-16T15:08:05.877275+0000","last_peered":"2024-01-16T15:08:05.877275+0000","last_clean":"2024-01-16T15:08:05.877275+0000","last_became_active":"2024-01-16T15:08:03.314250+0000","last_became_peered":"2024-01-16T15:08:03.314250+0000","last_unstale":"2024-01-16T15:08:05.877275+0000","last_undegraded":"2024-01-16T15:08:05.877275+0000","last_fullsized":"2024-01-16T15:08:05.877275+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:15:10.313446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.325699+0000","last_change":"2024-01-16T15:08:04.323321+0000","last_active":"2024-01-16T15:08:04.325699+0000","last_peered":"2024-01-16T15:08:04.325699+0000","last_clean":"2024-01-16T15:08:04.325699+0000","last_became_active":"2024-01-16T15:08:04.323030+0000","last_became_peered":"2024-01-16T15:08:04.323030+0000","last_unstale":"2024-01-16T15:08:04.325699+0000","last_undegraded":"2024-01-16T15:08:04.325699+0000","last_fullsized":"2024-01-16T15:08:04.325699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:33:36.602276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.285974+0000","last_change":"2024-01-16T15:08:03.306768+0000","last_active":"2024-01-16T15:08:04.285974+0000","last_peered":"2024-01-16T15:08:04.285974+0000","last_clean":"2024-01-16T15:08:04.285974+0000","last_became_active":"2024-01-16T15:08:03.306422+0000","last_became_peered":"2024-01-16T15:08:03.306422+0000","last_unstale":"2024-01-16T15:08:04.285974+0000","last_undegraded":"2024-01-16T15:08:04.285974+0000","last_fullsized":"2024-01-16T15:08:04.285974+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:20.195660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.317349+0000","last_change":"2024-01-16T15:08:04.313524+0000","last_active":"2024-01-16T15:08:04.317349+0000","last_peered":"2024-01-16T15:08:04.317349+0000","last_clean":"2024-01-16T15:08:04.317349+0000","last_became_active":"2024-01-16T15:08:04.311859+0000","last_became_peered":"2024-01-16T15:08:04.311859+0000","last_unstale":"2024-01-16T15:08:04.317349+0000","last_undegraded":"2024-01-16T15:08:04.317349+0000","last_fullsized":"2024-01-16T15:08:04.317349+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:27:50.675877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876914+0000","last_change":"2024-01-16T15:08:03.318388+0000","last_active":"2024-01-16T15:08:05.876914+0000","last_peered":"2024-01-16T15:08:05.876914+0000","last_clean":"2024-01-16T15:08:05.876914+0000","last_became_active":"2024-01-16T15:08:03.316993+0000","last_became_peered":"2024-01-16T15:08:03.316993+0000","last_unstale":"2024-01-16T15:08:05.876914+0000","last_undegraded":"2024-01-16T15:08:05.876914+0000","last_fullsized":"2024-01-16T15:08:05.876914+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:26:01.554258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.325417+0000","last_change":"2024-01-16T15:08:04.324472+0000","last_active":"2024-01-16T15:08:04.325417+0000","last_peered":"2024-01-16T15:08:04.325417+0000","last_clean":"2024-01-16T15:08:04.325417+0000","last_became_active":"2024-01-16T15:08:04.323999+0000","last_became_peered":"2024-01-16T15:08:04.323999+0000","last_unstale":"2024-01-16T15:08:04.325417+0000","last_undegraded":"2024-01-16T15:08:04.325417+0000","last_fullsized":"2024-01-16T15:08:04.325417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:45.225887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.315061+0000","last_change":"2024-01-16T15:08:03.314819+0000","last_active":"2024-01-16T15:08:04.315061+0000","last_peered":"2024-01-16T15:08:04.315061+0000","last_clean":"2024-01-16T15:08:04.315061+0000","last_became_active":"2024-01-16T15:08:03.314562+0000","last_became_peered":"2024-01-16T15:08:03.314562+0000","last_unstale":"2024-01-16T15:08:04.315061+0000","last_undegraded":"2024-01-16T15:08:04.315061+0000","last_fullsized":"2024-01-16T15:08:04.315061+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:33:41.596206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.321397+0000","last_change":"2024-01-16T15:08:04.316178+0000","last_active":"2024-01-16T15:08:04.321397+0000","last_peered":"2024-01-16T15:08:04.321397+0000","last_clean":"2024-01-16T15:08:04.321397+0000","last_became_active":"2024-01-16T15:08:04.315608+0000","last_became_peered":"2024-01-16T15:08:04.315608+0000","last_unstale":"2024-01-16T15:08:04.321397+0000","last_undegraded":"2024-01-16T15:08:04.321397+0000","last_fullsized":"2024-01-16T15:08:04.321397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.581063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.300239+0000","last_change":"2024-01-16T15:08:03.314684+0000","last_active":"2024-01-16T15:08:04.300239+0000","last_peered":"2024-01-16T15:08:04.300239+0000","last_clean":"2024-01-16T15:08:04.300239+0000","last_became_active":"2024-01-16T15:08:03.314325+0000","last_became_peered":"2024-01-16T15:08:03.314325+0000","last_unstale":"2024-01-16T15:08:04.300239+0000","last_undegraded":"2024-01-16T15:08:04.300239+0000","last_fullsized":"2024-01-16T15:08:04.300239+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:28:15.793686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.314546+0000","last_change":"2024-01-16T15:08:04.312391+0000","last_active":"2024-01-16T15:08:04.314546+0000","last_peered":"2024-01-16T15:08:04.314546+0000","last_clean":"2024-01-16T15:08:04.314546+0000","last_became_active":"2024-01-16T15:08:04.312026+0000","last_became_peered":"2024-01-16T15:08:04.312026+0000","last_unstale":"2024-01-16T15:08:04.314546+0000","last_undegraded":"2024-01-16T15:08:04.314546+0000","last_fullsized":"2024-01-16T15:08:04.314546+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:58:27.041958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.877384+0000","last_change":"2024-01-16T15:08:03.315816+0000","last_active":"2024-01-16T15:08:05.877384+0000","last_peered":"2024-01-16T15:08:05.877384+0000","last_clean":"2024-01-16T15:08:05.877384+0000","last_became_active":"2024-01-16T15:08:03.315417+0000","last_became_peered":"2024-01-16T15:08:03.315417+0000","last_unstale":"2024-01-16T15:08:05.877384+0000","last_undegraded":"2024-01-16T15:08:05.877384+0000","last_fullsized":"2024-01-16T15:08:05.877384+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:42.145047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326195+0000","last_change":"2024-01-16T15:08:04.325323+0000","last_active":"2024-01-16T15:08:04.326195+0000","last_peered":"2024-01-16T15:08:04.326195+0000","last_clean":"2024-01-16T15:08:04.326195+0000","last_became_active":"2024-01-16T15:08:04.324401+0000","last_became_peered":"2024-01-16T15:08:04.324401+0000","last_unstale":"2024-01-16T15:08:04.326195+0000","last_undegraded":"2024-01-16T15:08:04.326195+0000","last_fullsized":"2024-01-16T15:08:04.326195+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:39:17.050489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.320439+0000","last_change":"2024-01-16T15:08:03.312423+0000","last_active":"2024-01-16T15:08:04.320439+0000","last_peered":"2024-01-16T15:08:04.320439+0000","last_clean":"2024-01-16T15:08:04.320439+0000","last_became_active":"2024-01-16T15:08:03.312112+0000","last_became_peered":"2024-01-16T15:08:03.312112+0000","last_unstale":"2024-01-16T15:08:04.320439+0000","last_undegraded":"2024-01-16T15:08:04.320439+0000","last_fullsized":"2024-01-16T15:08:04.320439+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:51:10.936164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.326202+0000","last_change":"2024-01-16T15:08:04.318122+0000","last_active":"2024-01-16T15:08:04.326202+0000","last_peered":"2024-01-16T15:08:04.326202+0000","last_clean":"2024-01-16T15:08:04.326202+0000","last_became_active":"2024-01-16T15:08:04.317834+0000","last_became_peered":"2024-01-16T15:08:04.317834+0000","last_unstale":"2024-01-16T15:08:04.326202+0000","last_undegraded":"2024-01-16T15:08:04.326202+0000","last_fullsized":"2024-01-16T15:08:04.326202+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:21.545902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.302925+0000","last_change":"2024-01-16T15:08:03.306305+0000","last_active":"2024-01-16T15:08:04.302925+0000","last_peered":"2024-01-16T15:08:04.302925+0000","last_clean":"2024-01-16T15:08:04.302925+0000","last_became_active":"2024-01-16T15:08:03.306027+0000","last_became_peered":"2024-01-16T15:08:03.306027+0000","last_unstale":"2024-01-16T15:08:04.302925+0000","last_undegraded":"2024-01-16T15:08:04.302925+0000","last_fullsized":"2024-01-16T15:08:04.302925+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:24.907779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.338139+0000","last_change":"2024-01-16T15:08:04.332372+0000","last_active":"2024-01-16T15:08:04.338139+0000","last_peered":"2024-01-16T15:08:04.338139+0000","last_clean":"2024-01-16T15:08:04.338139+0000","last_became_active":"2024-01-16T15:08:04.331881+0000","last_became_peered":"2024-01-16T15:08:04.331881+0000","last_unstale":"2024-01-16T15:08:04.338139+0000","last_undegraded":"2024-01-16T15:08:04.338139+0000","last_fullsized":"2024-01-16T15:08:04.338139+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:11:03.803166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.298386+0000","last_change":"2024-01-16T15:08:03.307706+0000","last_active":"2024-01-16T15:08:04.298386+0000","last_peered":"2024-01-16T15:08:04.298386+0000","last_clean":"2024-01-16T15:08:04.298386+0000","last_became_active":"2024-01-16T15:08:03.306510+0000","last_became_peered":"2024-01-16T15:08:03.306510+0000","last_unstale":"2024-01-16T15:08:04.298386+0000","last_undegraded":"2024-01-16T15:08:04.298386+0000","last_fullsized":"2024-01-16T15:08:04.298386+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:58:20.530101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.312566+0000","last_change":"2024-01-16T15:08:04.309368+0000","last_active":"2024-01-16T15:08:04.312566+0000","last_peered":"2024-01-16T15:08:04.312566+0000","last_clean":"2024-01-16T15:08:04.312566+0000","last_became_active":"2024-01-16T15:08:04.309087+0000","last_became_peered":"2024-01-16T15:08:04.309087+0000","last_unstale":"2024-01-16T15:08:04.312566+0000","last_undegraded":"2024-01-16T15:08:04.312566+0000","last_fullsized":"2024-01-16T15:08:04.312566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:55.413459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":11,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.295260+0000","last_change":"2024-01-16T15:08:03.317378+0000","last_active":"2024-01-16T15:08:04.295260+0000","last_peered":"2024-01-16T15:08:04.295260+0000","last_clean":"2024-01-16T15:08:04.295260+0000","last_became_active":"2024-01-16T15:08:03.317100+0000","last_became_peered":"2024-01-16T15:08:03.317100+0000","last_unstale":"2024-01-16T15:08:04.295260+0000","last_undegraded":"2024-01-16T15:08:04.295260+0000","last_fullsized":"2024-01-16T15:08:04.295260+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:53:08.889310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876336+0000","last_change":"2024-01-16T15:08:04.319275+0000","last_active":"2024-01-16T15:08:05.876336+0000","last_peered":"2024-01-16T15:08:05.876336+0000","last_clean":"2024-01-16T15:08:05.876336+0000","last_became_active":"2024-01-16T15:08:04.318315+0000","last_became_peered":"2024-01-16T15:08:04.318315+0000","last_unstale":"2024-01-16T15:08:05.876336+0000","last_undegraded":"2024-01-16T15:08:05.876336+0000","last_fullsized":"2024-01-16T15:08:05.876336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:43.326777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.316977+0000","last_change":"2024-01-16T15:08:03.306371+0000","last_active":"2024-01-16T15:08:05.316977+0000","last_peered":"2024-01-16T15:08:05.316977+0000","last_clean":"2024-01-16T15:08:05.316977+0000","last_became_active":"2024-01-16T15:08:03.306064+0000","last_became_peered":"2024-01-16T15:08:03.306064+0000","last_unstale":"2024-01-16T15:08:05.316977+0000","last_undegraded":"2024-01-16T15:08:05.316977+0000","last_fullsized":"2024-01-16T15:08:05.316977+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:05:11.513682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.317974+0000","last_change":"2024-01-16T15:08:04.308853+0000","last_active":"2024-01-16T15:08:05.317974+0000","last_peered":"2024-01-16T15:08:05.317974+0000","last_clean":"2024-01-16T15:08:05.317974+0000","last_became_active":"2024-01-16T15:08:04.308415+0000","last_became_peered":"2024-01-16T15:08:04.308415+0000","last_unstale":"2024-01-16T15:08:05.317974+0000","last_undegraded":"2024-01-16T15:08:05.317974+0000","last_fullsized":"2024-01-16T15:08:05.317974+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:44:34.254450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.318750+0000","last_change":"2024-01-16T15:08:03.299481+0000","last_active":"2024-01-16T15:08:05.318750+0000","last_peered":"2024-01-16T15:08:05.318750+0000","last_clean":"2024-01-16T15:08:05.318750+0000","last_became_active":"2024-01-16T15:08:03.299064+0000","last_became_peered":"2024-01-16T15:08:03.299064+0000","last_unstale":"2024-01-16T15:08:05.318750+0000","last_undegraded":"2024-01-16T15:08:05.318750+0000","last_fullsized":"2024-01-16T15:08:05.318750+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:40:27.669781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876062+0000","last_change":"2024-01-16T15:08:04.316799+0000","last_active":"2024-01-16T15:08:05.876062+0000","last_peered":"2024-01-16T15:08:05.876062+0000","last_clean":"2024-01-16T15:08:05.876062+0000","last_became_active":"2024-01-16T15:08:04.316487+0000","last_became_peered":"2024-01-16T15:08:04.316487+0000","last_unstale":"2024-01-16T15:08:05.876062+0000","last_undegraded":"2024-01-16T15:08:05.876062+0000","last_fullsized":"2024-01-16T15:08:05.876062+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:59:58.169539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876566+0000","last_change":"2024-01-16T15:08:03.314311+0000","last_active":"2024-01-16T15:08:05.876566+0000","last_peered":"2024-01-16T15:08:05.876566+0000","last_clean":"2024-01-16T15:08:05.876566+0000","last_became_active":"2024-01-16T15:08:03.313852+0000","last_became_peered":"2024-01-16T15:08:03.313852+0000","last_unstale":"2024-01-16T15:08:05.876566+0000","last_undegraded":"2024-01-16T15:08:05.876566+0000","last_fullsized":"2024-01-16T15:08:05.876566+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:12:11.723335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":9,"reported_epoch":71,"state":"active+clean","last_fresh":"2024-01-16T15:08:04.322459+0000","last_change":"2024-01-16T15:08:04.319584+0000","last_active":"2024-01-16T15:08:04.322459+0000","last_peered":"2024-01-16T15:08:04.322459+0000","last_clean":"2024-01-16T15:08:04.322459+0000","last_became_active":"2024-01-16T15:08:04.319243+0000","last_became_peered":"2024-01-16T15:08:04.319243+0000","last_unstale":"2024-01-16T15:08:04.322459+0000","last_undegraded":"2024-01-16T15:08:04.322459+0000","last_fullsized":"2024-01-16T15:08:04.322459+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:03.270007+0000","last_clean_scrub_stamp":"2024-01-16T15:08:03.270007+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:11.309019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"71'2","reported_seq":14,"reported_epoch":72,"state":"active+clean","last_fresh":"2024-01-16T15:08:05.876611+0000","last_change":"2024-01-16T15:08:03.313819+0000","last_active":"2024-01-16T15:08:05.876611+0000","last_peered":"2024-01-16T15:08:05.876611+0000","last_clean":"2024-01-16T15:08:05.876611+0000","last_became_active":"2024-01-16T15:08:03.313445+0000","last_became_peered":"2024-01-16T15:08:03.313445+0000","last_unstale":"2024-01-16T15:08:05.876611+0000","last_undegraded":"2024-01-16T15:08:05.876611+0000","last_fullsized":"2024-01-16T15:08:05.876611+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:08:02.254206+0000","last_clean_scrub_stamp":"2024-01-16T15:08:02.254206+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:35:33.361363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":213,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-01-16T15:08:03.297083+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:08:03.297083+0000","last_peered":"2024-01-16T15:08:03.297083+0000","last_clean":"2024-01-16T15:08:03.297083+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:08:03.297083+0000","last_undegraded":"2024-01-16T15:08:03.297083+0000","last_fullsized":"2024-01-16T15:08:03.297083+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509542,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45924,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325916,"statfs":{"total":96636764160,"available":96589737984,"internally_reserved":0,"allocated":876544,"data_stored":367691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.915,"5min":1.9419999999999999,"15min":1.429},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.55100000000000005},"max":{"1min":14.1,"5min":27.853999999999999,"15min":27.853999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.615,"15min":1.3180000000000001},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":4.149,"5min":19.827999999999999,"15min":19.827999999999999},"last":0.93899999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.607,"15min":1.3180000000000001},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":10.326000000000001,"5min":12.577999999999999,"15min":12.577999999999999},"last":1.216},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.5900000000000001,"15min":1.361},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":3.6059999999999999,"5min":8.9469999999999992,"15min":8.9469999999999992},"last":1.6060000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6459999999999999,"5min":1.5920000000000001,"15min":1.456},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":10.696999999999999,"5min":10.696999999999999,"15min":10.696999999999999},"last":1.5529999999999999},{"interface":"front","average":{"1min":1.625,"5min":1.7070000000000001,"15min":1.4790000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":5.9329999999999998,"5min":19.866,"15min":19.866},"last":1.3320000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8600000000000001,"5min":1.8540000000000001,"15min":1.571},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":16.353000000000002,"5min":16.353000000000002,"15min":16.353000000000002},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.7729999999999999,"15min":1.629},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":3.536,"5min":8.9009999999999998,"15min":8.9009999999999998},"last":1.423}]},{"osd":4,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.7669999999999999,"15min":1.5680000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":4.6310000000000002,"5min":19.707000000000001,"15min":19.707000000000001},"last":1.139},{"interface":"front","average":{"1min":2.1539999999999999,"5min":2.0409999999999999,"15min":1.7150000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":10.609999999999999,"5min":12.506,"15min":12.506},"last":1.696}]},{"osd":5,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8720000000000001,"5min":2.2200000000000002,"15min":1.875},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":10.388,"5min":12.606,"15min":12.606},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.823,"15min":1.7130000000000001},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":2.6850000000000001,"5min":9.5449999999999999,"15min":9.5449999999999999},"last":1.5029999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":2.069,"15min":1.772},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":3.8300000000000001,"5min":33.155999999999999,"15min":33.155999999999999},"last":1.853},{"interface":"front","average":{"1min":2.1499999999999999,"5min":2.7909999999999999,"15min":2.081},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":11.553000000000001,"5min":46.305,"15min":46.305},"last":1.478}]}]},{"osd":4,"up_from":15,"seq":64424509544,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50084,"kb_used_data":852,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321756,"statfs":{"total":96636764160,"available":96585478144,"internally_reserved":0,"allocated":872448,"data_stored":367669,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.9159999999999999,"15min":1.3460000000000001},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":2.0990000000000002,"5min":18.818000000000001,"15min":18.818000000000001},"last":2.0640000000000001},{"interface":"front","average":{"1min":1.232,"5min":1.3959999999999999,"15min":1.214},"min":{"1min":0.46300000000000002,"5min":0.432,"15min":0.432},"max":{"1min":3.722,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.91800000000000004}]},{"osd":1,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4730000000000001,"5min":1.9339999999999999,"15min":1.4359999999999999},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":5.742,"5min":18.347000000000001,"15min":18.347000000000001},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.175,"5min":1.708,"15min":1.4039999999999999},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":2.1429999999999998,"5min":18.858000000000001,"15min":18.858000000000001},"last":1.579}]},{"osd":2,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.996,"15min":1.522},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":4.3380000000000001,"5min":16.794,"15min":16.794},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.714,"15min":1.4470000000000001},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":2.226,"5min":18.905000000000001,"15min":18.905000000000001},"last":1.181}]},{"osd":3,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8500000000000001,"5min":1.944,"15min":1.5149999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":13.169,"5min":26.741,"15min":26.741},"last":1.361},{"interface":"front","average":{"1min":1.5720000000000001,"5min":2.23,"15min":1.718},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":5.1719999999999997,"5min":26.701000000000001,"15min":26.701000000000001},"last":1.5389999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.9019999999999999,"15min":1.663},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":2.0449999999999999,"5min":18.919,"15min":18.919},"last":1.444},{"interface":"front","average":{"1min":1.355,"5min":1.907,"15min":1.661},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.768,"5min":26.597000000000001,"15min":26.597000000000001},"last":1.768}]},{"osd":6,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.9299999999999999,"15min":1.6579999999999999},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.47699999999999998},"max":{"1min":4.2439999999999998,"5min":16.902000000000001,"15min":16.902000000000001},"last":1.74},{"interface":"front","average":{"1min":1.4590000000000001,"5min":2.0019999999999998,"15min":1.724},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":2.2240000000000002,"5min":18.952999999999999,"15min":18.952999999999999},"last":1.712}]},{"osd":7,"last update":"Tue Jan 16 15:08:06 2024","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.946,"15min":1.6850000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":2.6259999999999999,"5min":26.683,"15min":26.683},"last":1.621},{"interface":"front","average":{"1min":1.6819999999999999,"5min":1.954,"15min":1.8320000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":4.6260000000000003,"5min":16.927,"15min":16.927},"last":1.819}]}]},{"osd":7,"up_from":14,"seq":60129542246,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50660,"kb_used_data":1424,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321180,"statfs":{"total":96636764160,"available":96584888320,"internally_reserved":0,"allocated":1458176,"data_stored":952255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":50395594},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.8779999999999999,"5min":1.875,"15min":1.3100000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":10.51,"5min":15.661,"15min":15.661},"last":2.9420000000000002},{"interface":"front","average":{"1min":3.431,"5min":2.0630000000000002,"15min":1.377},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":21.262,"5min":30.126000000000001,"15min":30.126000000000001},"last":5.399}]},{"osd":1,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.1859999999999999,"5min":1.4870000000000001,"15min":1.238},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":8.3149999999999995,"5min":8.3149999999999995,"15min":8.3149999999999995},"last":1.125},{"interface":"front","average":{"1min":3.4380000000000002,"5min":2.0670000000000002,"15min":1.478},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":21.298999999999999,"5min":21.298999999999999,"15min":21.298999999999999},"last":6.8739999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":1.6799999999999999,"15min":1.3919999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":10.571999999999999,"5min":12.446999999999999,"15min":12.446999999999999},"last":4.9489999999999998},{"interface":"front","average":{"1min":1.9890000000000001,"5min":1.575,"15min":1.363},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":8.2759999999999998,"5min":8.2759999999999998,"15min":8.2759999999999998},"last":1.5660000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.8149999999999999,"15min":1.5049999999999999},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":10.653,"5min":11.779,"15min":11.779},"last":3.7589999999999999},{"interface":"front","average":{"1min":2.6349999999999998,"5min":1.7290000000000001,"15min":1.4670000000000001},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":17.347999999999999,"5min":17.347999999999999,"15min":17.347999999999999},"last":3.9390000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:07:04 2024","interfaces":[{"interface":"back","average":{"1min":2.6629999999999998,"5min":1.7689999999999999,"15min":1.5089999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":10.989000000000001,"5min":14.659000000000001,"15min":14.659000000000001},"last":2.996},{"interface":"front","average":{"1min":1.7150000000000001,"5min":1.6160000000000001,"15min":1.468},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":5.8319999999999999,"5min":11.433,"15min":11.433},"last":2.351}]},{"osd":5,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":5.7130000000000001,"5min":2.4470000000000001,"15min":1.756},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":26.23,"5min":26.23,"15min":26.23},"last":7.899},{"interface":"front","average":{"1min":3.2610000000000001,"5min":2.1389999999999998,"15min":1.7509999999999999},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":21.312000000000001,"5min":21.312000000000001,"15min":21.312000000000001},"last":7.7539999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:07:04 2024","interfaces":[{"interface":"back","average":{"1min":1.927,"5min":1.6200000000000001,"15min":1.5720000000000001},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":5.8949999999999996,"5min":5.8949999999999996,"15min":5.8949999999999996},"last":1.7430000000000001},{"interface":"front","average":{"1min":3.5179999999999998,"5min":2.2930000000000001,"15min":1.8080000000000001},"min":{"1min":1.3140000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":20.881,"5min":29.991,"15min":29.991},"last":5.2830000000000004}]}]},{"osd":5,"up_from":14,"seq":60129542247,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45916,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325924,"statfs":{"total":96636764160,"available":96589746176,"internally_reserved":0,"allocated":860160,"data_stored":361361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.3370000000000002,"5min":1.6739999999999999,"15min":1.2589999999999999},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":10.606,"5min":10.606,"15min":10.606},"last":0.872},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.486,"15min":1.244},"min":{"1min":0.68799999999999994,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.95,"5min":11.109,"15min":11.109},"last":1.153}]},{"osd":1,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.3660000000000001,"5min":1.821,"15min":1.3520000000000001},"min":{"1min":0.67400000000000004,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":13.837999999999999,"5min":17.693999999999999,"15min":17.693999999999999},"last":1.373},{"interface":"front","average":{"1min":2.5179999999999998,"5min":1.762,"15min":1.333},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":10.775,"5min":10.775,"15min":10.775},"last":1.002}]},{"osd":2,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.621,"5min":1.7450000000000001,"15min":1.383},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":10.702999999999999,"5min":10.702999999999999,"15min":10.702999999999999},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.409,"5min":1.649,"15min":1.3620000000000001},"min":{"1min":0.77400000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":3.0609999999999999,"5min":11.321,"15min":11.321},"last":1.3080000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.367,"5min":2.1890000000000001,"15min":1.619},"min":{"1min":0.93000000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":13.875,"5min":23.109000000000002,"15min":23.109000000000002},"last":1.2649999999999999},{"interface":"front","average":{"1min":2.569,"5min":1.788,"15min":1.554},"min":{"1min":0.91000000000000003,"5min":0.624,"15min":0.624},"max":{"1min":10.645,"5min":10.645,"15min":14.647},"last":1.1100000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:07:02 2024","interfaces":[{"interface":"back","average":{"1min":3.8119999999999998,"5min":2.194,"15min":1.603},"min":{"1min":0.90500000000000003,"5min":0.442,"15min":0.442},"max":{"1min":17.718,"5min":24.436,"15min":24.436},"last":1.657},{"interface":"front","average":{"1min":2.9630000000000001,"5min":1.7709999999999999,"15min":1.5189999999999999},"min":{"1min":0.68700000000000006,"5min":0.434,"15min":0.434},"max":{"1min":10.541,"5min":10.541,"15min":10.541},"last":1.5649999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:07:02 2024","interfaces":[{"interface":"back","average":{"1min":2.1269999999999998,"5min":1.95,"15min":1.54},"min":{"1min":0.79000000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":12.401,"5min":21.192,"15min":21.192},"last":1.4199999999999999},{"interface":"front","average":{"1min":4.6399999999999997,"5min":2.2160000000000002,"15min":1.673},"min":{"1min":1.26,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":23.276,"5min":23.276,"15min":23.276},"last":1.849}]},{"osd":7,"last update":"Tue Jan 16 15:07:55 2024","interfaces":[{"interface":"back","average":{"1min":2.399,"5min":2.2970000000000002,"15min":1.7989999999999999},"min":{"1min":0.67000000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":10.605,"5min":32.405999999999999,"15min":32.405999999999999},"last":1.512},{"interface":"front","average":{"1min":2.9649999999999999,"5min":2.5299999999999998,"15min":1.9410000000000001},"min":{"1min":0.68200000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":13.715999999999999,"5min":24.367000000000001,"15min":24.367000000000001},"last":2.0640000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542247,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54236,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317604,"statfs":{"total":96636764160,"available":96581226496,"internally_reserved":0,"allocated":864256,"data_stored":361383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":54655431},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.016,"5min":1.375,"15min":1.012},"min":{"1min":0.55000000000000004,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":2.1120000000000001,"5min":31.562999999999999,"15min":31.562999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2250000000000001,"15min":1.0660000000000001},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":5.1050000000000004,"5min":8.4879999999999995,"15min":8.4879999999999995},"last":0.67800000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":0.91300000000000003,"5min":1.1950000000000001,"15min":0.99399999999999999},"min":{"1min":0.53100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.506,"5min":15.569000000000001,"15min":15.569000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.357,"15min":1.153},"min":{"1min":0.69199999999999995,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.728,"5min":6.992,"15min":6.992},"last":1.216}]},{"osd":2,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.3779999999999999,"15min":1.208},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.36399999999999999},"max":{"1min":1.623,"5min":7.0529999999999999,"15min":7.0529999999999999},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.2709999999999999,"15min":1.2110000000000001},"min":{"1min":0.67300000000000004,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":2.089,"5min":7.9729999999999999,"15min":14.577999999999999},"last":1.339}]},{"osd":4,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.7470000000000001,"15min":1.4450000000000001},"min":{"1min":1.042,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.641,"5min":20.978999999999999,"15min":20.978999999999999},"last":1.073},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.635,"15min":1.48},"min":{"1min":0.85099999999999998,"5min":0.748,"15min":0.748},"max":{"1min":2.2490000000000001,"5min":7.1379999999999999,"15min":7.1379999999999999},"last":1.115}]},{"osd":5,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":2.4950000000000001,"5min":1.8640000000000001,"15min":1.5680000000000001},"min":{"1min":0.94199999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":20.663,"5min":20.663,"15min":20.663},"last":1.258},{"interface":"front","average":{"1min":1.8260000000000001,"5min":1.861,"15min":1.589},"min":{"1min":0.998,"5min":0.81399999999999995,"15min":0.76000000000000001},"max":{"1min":7.0460000000000003,"5min":17.613,"15min":17.613},"last":1.3029999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.875,"15min":1.5900000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.2850000000000001,"5min":13.182,"15min":13.182},"last":1.153},{"interface":"front","average":{"1min":2.5899999999999999,"5min":2.0649999999999999,"15min":1.6439999999999999},"min":{"1min":1.0269999999999999,"5min":0.76400000000000001,"15min":0.76400000000000001},"max":{"1min":20.593,"5min":20.593,"15min":20.593},"last":1.514}]},{"osd":7,"last update":"Tue Jan 16 15:08:04 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":1.6910000000000001,"15min":1.5409999999999999},"min":{"1min":1.0409999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":4.0679999999999996,"5min":5.4459999999999997,"15min":5.4459999999999997},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.671,"5min":2.7360000000000002,"15min":1.923},"min":{"1min":0.99099999999999999,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":3.8149999999999999,"5min":49.811,"15min":49.811},"last":1.3740000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542246,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50076,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321764,"statfs":{"total":96636764160,"available":96585486336,"internally_reserved":0,"allocated":868352,"data_stored":362883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86099999999999999,"5min":0.94499999999999995,"15min":0.98799999999999999},"min":{"1min":0.39600000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":2.391,"5min":2.4990000000000001,"15min":2.4990000000000001},"last":0.874},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.081,"15min":1.04},"min":{"1min":0.35799999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.7909999999999999,"5min":4.7809999999999997,"15min":4.7809999999999997},"last":0.81200000000000006}]},{"osd":1,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0580000000000001,"15min":1.044},"min":{"1min":0.442,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":2.3199999999999998,"5min":2.5779999999999998,"15min":3.839},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.1899999999999999,"15min":1.115},"min":{"1min":0.53100000000000003,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.8260000000000001,"5min":4.8849999999999998,"15min":4.8849999999999998},"last":1.143}]},{"osd":3,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":1.077,"15min":0.99199999999999999},"min":{"1min":0.64300000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.611,"5min":2.6179999999999999,"15min":9.1530000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.224,"5min":1.3720000000000001,"15min":1.3320000000000001},"min":{"1min":0.623,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":2.0179999999999998,"5min":4.9610000000000003,"15min":4.9610000000000003},"last":1.268}]},{"osd":4,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.589,"15min":1.3979999999999999},"min":{"1min":0.69999999999999996,"5min":0.621,"15min":0.621},"max":{"1min":2.3820000000000001,"5min":18.016999999999999,"15min":18.016999999999999},"last":1.323},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.504,"15min":1.4470000000000001},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":3.2090000000000001,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":1.5569999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0259999999999998,"5min":1.623,"15min":1.429},"min":{"1min":0.84599999999999997,"5min":0.746,"15min":0.746},"max":{"1min":7.944,"5min":7.944,"15min":9.6189999999999998},"last":1.387},{"interface":"front","average":{"1min":1.6579999999999999,"5min":1.48,"15min":1.4199999999999999},"min":{"1min":0.90300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":6.6040000000000001,"5min":6.6040000000000001,"15min":9.423},"last":1.1950000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.913,"5min":1.5960000000000001,"15min":1.4650000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":6.6509999999999998,"5min":6.6509999999999998,"15min":9.5359999999999996},"last":1.504},{"interface":"front","average":{"1min":1.704,"5min":1.96,"15min":1.6599999999999999},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.72999999999999998},"max":{"1min":6.6870000000000003,"5min":21.564,"15min":21.564},"last":1.5940000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:07:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.5529999999999999,"15min":1.526},"min":{"1min":0.95099999999999996,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":2.5899999999999999,"5min":9.8610000000000007,"15min":9.8610000000000007},"last":1.6870000000000001},{"interface":"front","average":{"1min":2.3490000000000002,"5min":2.1560000000000001,"15min":1.7669999999999999},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":13.074,"5min":39.685000000000002,"15min":39.685000000000002},"last":1.637}]}]},{"osd":1,"up_from":14,"seq":60129542248,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54244,"kb_used_data":856,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317596,"statfs":{"total":96636764160,"available":96581218304,"internally_reserved":0,"allocated":876544,"data_stored":363499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.92400000000000004,"15min":0.89200000000000002},"min":{"1min":0.34899999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":2.278,"5min":2.278,"15min":10.667999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.3009999999999999,"15min":1.054},"min":{"1min":0.502,"5min":0.41699999999999998,"15min":0.39600000000000002},"max":{"1min":1.4019999999999999,"5min":17.5,"15min":17.5},"last":0.80900000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.2210000000000001,"15min":1.056},"min":{"1min":0.60799999999999998,"5min":0.52000000000000002,"15min":0.51000000000000001},"max":{"1min":1.538,"5min":6.6120000000000001,"15min":6.6120000000000001},"last":1.04},{"interface":"front","average":{"1min":1.1739999999999999,"5min":1.3420000000000001,"15min":1.2649999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6839999999999999,"5min":6.2759999999999998,"15min":10.805},"last":1.4390000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.6830000000000001,"15min":1.3340000000000001},"min":{"1min":0.55400000000000005,"5min":0.443,"15min":0.443},"max":{"1min":1.635,"5min":22.736000000000001,"15min":22.736000000000001},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.1000000000000001,"5min":1.5269999999999999,"15min":1.2450000000000001},"min":{"1min":0.49399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.7110000000000001,"5min":17.361000000000001,"15min":17.361000000000001},"last":1.127}]},{"osd":4,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.6930000000000001,"5min":1.609,"15min":1.3839999999999999},"min":{"1min":0.79400000000000004,"5min":0.628,"15min":0.628},"max":{"1min":9.0899999999999999,"5min":10.577999999999999,"15min":10.577999999999999},"last":1.181},{"interface":"front","average":{"1min":1.349,"5min":1.492,"15min":1.51},"min":{"1min":0.81299999999999994,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.696,"5min":4.5609999999999999,"15min":11.157999999999999},"last":1.5629999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.7529999999999999,"15min":1.49},"min":{"1min":0.84699999999999998,"5min":0.71199999999999997,"15min":0.68400000000000005},"max":{"1min":1.8620000000000001,"5min":17.469000000000001,"15min":17.469000000000001},"last":1.296},{"interface":"front","average":{"1min":1.835,"5min":1.732,"15min":1.5409999999999999},"min":{"1min":0.88200000000000001,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":8.2989999999999995,"5min":8.2989999999999995,"15min":8.2989999999999995},"last":1.6240000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":2,"15min":1.6479999999999999},"min":{"1min":0.80300000000000005,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":3.3540000000000001,"5min":17.428999999999998,"15min":17.428999999999998},"last":1.363},{"interface":"front","average":{"1min":1.6639999999999999,"5min":2.1779999999999999,"15min":1.7030000000000001},"min":{"1min":0.92500000000000004,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":4.891,"5min":18.850999999999999,"15min":18.850999999999999},"last":1.5069999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:08:02 2024","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":2.2469999999999999,"15min":1.7729999999999999},"min":{"1min":1.0389999999999999,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":3.6440000000000001,"5min":30.834,"15min":30.834},"last":1.413},{"interface":"front","average":{"1min":1.5189999999999999,"5min":2.0049999999999999,"15min":1.704},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":2.8460000000000001,"5min":27.263999999999999,"15min":27.263999999999999},"last":1.694}]}]},{"osd":0,"up_from":14,"seq":60129542246,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54804,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94317036,"statfs":{"total":96636764160,"available":96580644864,"internally_reserved":0,"allocated":1449984,"data_stored":951639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.375,"15min":1.101},"min":{"1min":0.39800000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.4900000000000002,"5min":22.132999999999999,"15min":22.132999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":1.651,"5min":1.919,"15min":1.337},"min":{"1min":0.39300000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":12.41,"5min":50.429000000000002,"15min":50.429000000000002},"last":0.73499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3779999999999999,"5min":1.6040000000000001,"15min":1.3240000000000001},"min":{"1min":0.45300000000000001,"5min":0.38,"15min":0.38},"max":{"1min":4.4189999999999996,"5min":22.716999999999999,"15min":22.716999999999999},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.9379999999999999,"15min":1.397},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":7.2549999999999999,"5min":50.581000000000003,"15min":50.581000000000003},"last":1.0649999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.161,"15min":1.1679999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.716,"5min":4.6070000000000002,"15min":10.795},"last":1.583},{"interface":"front","average":{"1min":1.554,"5min":1.788,"15min":1.4910000000000001},"min":{"1min":0.56200000000000006,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":4.5750000000000002,"5min":22.254999999999999,"15min":22.254999999999999},"last":1.137}]},{"osd":4,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":2.266,"5min":2.1869999999999998,"15min":1.6659999999999999},"min":{"1min":0.79600000000000004,"5min":0.72599999999999998,"15min":0.625},"max":{"1min":12.093,"5min":22.407,"15min":22.407},"last":1.2769999999999999},{"interface":"front","average":{"1min":1.7809999999999999,"5min":2.423,"15min":1.708},"min":{"1min":0.76400000000000001,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":7.3200000000000003,"5min":50.497,"15min":50.497},"last":1.3560000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":1.3660000000000001,"5min":1.8,"15min":1.5469999999999999},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":3.1880000000000002,"5min":14.112,"15min":14.112},"last":1.3149999999999999},{"interface":"front","average":{"1min":2.1600000000000001,"5min":2.238,"15min":1.75},"min":{"1min":0.79900000000000004,"5min":0.78000000000000003,"15min":0.71399999999999997},"max":{"1min":9.827,"5min":22.882999999999999,"15min":22.882999999999999},"last":1.6220000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":2.9950000000000001,"5min":2.1789999999999998,"15min":1.696},"min":{"1min":0.67500000000000004,"5min":0.63300000000000001,"15min":0.63300000000000001},"max":{"1min":31.387,"5min":31.387,"15min":31.387},"last":1.458},{"interface":"front","average":{"1min":2.2160000000000002,"5min":2.3959999999999999,"15min":1.8300000000000001},"min":{"1min":0.876,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":6.4889999999999999,"5min":23.547000000000001,"15min":23.547000000000001},"last":1.4990000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:08:00 2024","interfaces":[{"interface":"back","average":{"1min":2.0489999999999999,"5min":2.54,"15min":1.8560000000000001},"min":{"1min":0.84999999999999998,"5min":0.79300000000000004,"15min":0.75600000000000001},"max":{"1min":7.3769999999999998,"5min":50.542000000000002,"15min":50.542000000000002},"last":1.546},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.893,"15min":1.702},"min":{"1min":0.60499999999999998,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":4.6619999999999999,"5min":15.747,"15min":15.747},"last":1.6519999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:08:09.295 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:08:09.295 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:08:09.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.591+0000 7f4a37e00640 1 Processor -- start 2024-01-16T15:08:09.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.592+0000 7f4a37e00640 1 -- start start 2024-01-16T15:08:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.593+0000 7f4a37e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a380b0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.593+0000 7f4a37e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4a380b1f30 0x7f4a380b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.593+0000 7f4a37e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a380d0050 con 0x7f4a3815d190 2024-01-16T15:08:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.593+0000 7f4a37e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a380d02a0 con 0x7f4a380b1f30 2024-01-16T15:08:09.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.593+0000 7f4a37e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380d0580 con 0x7f4a380d1d00 2024-01-16T15:08:09.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4a380b1f30 0x7f4a380b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4a380b1f30 0x7f4a380b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45518/0 (socket says 172.21.15.53:45518) 2024-01-16T15:08:09.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 -- 172.21.15.53:0/703408859 learned_addr learned my addr 172.21.15.53:0/703408859 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:09.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a2fe00640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a380b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 -- 172.21.15.53:0/703408859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 msgr2=0x7f4a380b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a380b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 -- 172.21.15.53:0/703408859 >> v1:172.21.15.146:6789/0 conn(0x7f4a3815d190 legacy=0x7f4a380d15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a37400640 1 -- 172.21.15.53:0/703408859 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a380d0860 con 0x7f4a380b1f30 2024-01-16T15:08:09.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.594+0000 7f4a2fe00640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a380b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.596+0000 7f4a37400640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380b06c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009e50 tx=0x7f4a3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c133823634889c46 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.596+0000 7f4a36000640 1 -- 172.21.15.53:0/703408859 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3002ca80 con 0x7f4a380b1f30 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.596+0000 7f4a36000640 1 -- 172.21.15.53:0/703408859 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3002cbe0 con 0x7f4a380b1f30 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.596+0000 7f4a36000640 1 -- 172.21.15.53:0/703408859 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a300343f0 con 0x7f4a380b1f30 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.596+0000 7f4a37e00640 1 -- 172.21.15.53:0/703408859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 msgr2=0x7f4a380b06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.596+0000 7f4a37e00640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380b06c0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009e50 tx=0x7f4a3002bee0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.597+0000 7f4a37e00640 1 -- 172.21.15.53:0/703408859 shutdown_connections 2024-01-16T15:08:09.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.597+0000 7f4a37e00640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380b06c0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.597+0000 7f4a37e00640 1 --2- 172.21.15.53:0/703408859 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a380b0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.597+0000 7f4a37e00640 1 -- 172.21.15.53:0/703408859 >> 172.21.15.53:0/703408859 conn(0x7f4a380d3010 msgr2=0x7f4a380d3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:09.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.597+0000 7f4a37e00640 1 -- 172.21.15.53:0/703408859 shutdown_connections 2024-01-16T15:08:09.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.597+0000 7f4a37e00640 1 -- 172.21.15.53:0/703408859 wait complete. 2024-01-16T15:08:09.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.598+0000 7f4a37e00640 1 Processor -- start 2024-01-16T15:08:09.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.599+0000 7f4a37e00640 1 -- start start 2024-01-16T15:08:09.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a38151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3815d190 0x7f4a380b7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38151a40 con 0x7f4a380b1f30 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380b7250 con 0x7f4a3815d190 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380b6640 con 0x7f4a380d1d00 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45524/0 (socket says 172.21.15.53:45524) 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 -- 172.21.15.53:0/2063863197 learned_addr learned my addr 172.21.15.53:0/2063863197 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a2fe00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a38151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a37400640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3815d190 0x7f4a380b7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 -- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 msgr2=0x7f4a38151500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a38151500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 -- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3815d190 msgr2=0x7f4a380b7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3815d190 0x7f4a380b7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a36a00640 1 -- 172.21.15.53:0/2063863197 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a30009b30 con 0x7f4a380b1f30 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.600+0000 7f4a2fe00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a38151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.601+0000 7f4a36a00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380cd1f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c450 tx=0x7f4a2800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.601+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280052d0 con 0x7f4a380b1f30 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.601+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a28005430 con 0x7f4a380b1f30 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.601+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28012870 con 0x7f4a380b1f30 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.601+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a380b6920 con 0x7f4a380b1f30 2024-01-16T15:08:09.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.601+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a380b6ee0 con 0x7f4a380b1f30 2024-01-16T15:08:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.602+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a380b70a0 con 0x7f4a380b1f30 2024-01-16T15:08:09.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.604+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f4a280129d0 con 0x7f4a380b1f30 2024-01-16T15:08:09.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.605+0000 7f4a2ea00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1006bbc0 0x7f4a1006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:09.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.605+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4a28087f80 con 0x7f4a380b1f30 2024-01-16T15:08:09.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.606+0000 7f4a2fe00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1006bbc0 0x7f4a1006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:09.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.607+0000 7f4a2fe00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1006bbc0 0x7f4a1006e080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4a20006fd0 tx=0x7f4a20008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:09.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.610+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a280548a0 con 0x7f4a380b1f30 2024-01-16T15:08:09.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.879+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f4a38067450 con 0x7f4a380b1f30 2024-01-16T15:08:09.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.884+0000 7f4a2ea00640 1 -- 172.21.15.53:0/2063863197 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+147 (secure 0 0 0) 0x7f4a28057ec0 con 0x7f4a380b1f30 2024-01-16T15:08:09.862 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:08:09.862 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:08:09.862 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:08:09.862 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:08:09.862 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:08:09.862 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1006bbc0 msgr2=0x7f4a1006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1006bbc0 0x7f4a1006e080 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f4a20006fd0 tx=0x7f4a20008040 comp rx=0 tx=0).stop 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 msgr2=0x7f4a380cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380cd1f0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c450 tx=0x7f4a2800c910 comp rx=0 tx=0).stop 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 shutdown_connections 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1006bbc0 0x7f4a1006e080 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3815d190 0x7f4a380b7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a380d1d00 0x7f4a38151500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 --2- 172.21.15.53:0/2063863197 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a380b1f30 0x7f4a380cd1f0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.887+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 >> 172.21.15.53:0/2063863197 conn(0x7f4a380d3010 msgr2=0x7f4a3815df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:09.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.888+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 shutdown_connections 2024-01-16T15:08:09.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:09.888+0000 7f4a37e00640 1 -- 172.21.15.53:0/2063863197 wait complete. 2024-01-16T15:08:09.885 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:08:10.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.125+0000 7fac4b200640 1 Processor -- start 2024-01-16T15:08:10.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.125+0000 7fac4b200640 1 -- start start 2024-01-16T15:08:10.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.126+0000 7fac4b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 0x7fac4c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:10.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac4b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fac4c13fcb0 0x7fac4c0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:10.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac4b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac4c0cdec0 con 0x7fac4c0cd530 2024-01-16T15:08:10.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac4b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4c0ce1a0 con 0x7fac4c13fcb0 2024-01-16T15:08:10.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac4b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c0ce480 con 0x7fac4c13cfa0 2024-01-16T15:08:10.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac49e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 0x7fac4c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:10.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac49e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 0x7fac4c15afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:37854/0 (socket says 172.21.15.53:37854) 2024-01-16T15:08:10.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac4a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fac4c13fcb0 0x7fac4c0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:10.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.127+0000 7fac49e00640 1 -- 172.21.15.53:0/2290535732 learned_addr learned my addr 172.21.15.53:0/2290535732 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:10.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.128+0000 7fac49e00640 1 -- 172.21.15.53:0/2290535732 >> v2:172.21.15.146:3300/0 conn(0x7fac4c13fcb0 msgr2=0x7fac4c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:10.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.128+0000 7fac49e00640 1 --2- 172.21.15.53:0/2290535732 >> v2:172.21.15.146:3300/0 conn(0x7fac4c13fcb0 0x7fac4c0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.128+0000 7fac49e00640 1 -- 172.21.15.53:0/2290535732 >> v1:172.21.15.146:6789/0 conn(0x7fac4c0cd530 legacy=0x7fac4c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.128+0000 7fac49e00640 1 -- 172.21.15.53:0/2290535732 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c0ce760 con 0x7fac4c13cfa0 2024-01-16T15:08:10.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.130+0000 7fac49e00640 1 --2- 172.21.15.53:0/2290535732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 0x7fac4c15afb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009870 tx=0x7fac3c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48234d2b6ab2e1ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:10.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.130+0000 7fac48a00640 1 -- 172.21.15.53:0/2290535732 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031640 con 0x7fac4c13cfa0 2024-01-16T15:08:10.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.130+0000 7fac48a00640 1 -- 172.21.15.53:0/2290535732 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0317a0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac48a00640 1 -- 172.21.15.53:0/2290535732 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031b30 con 0x7fac4c13cfa0 2024-01-16T15:08:10.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac4b200640 1 -- 172.21.15.53:0/2290535732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 msgr2=0x7fac4c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac4b200640 1 --2- 172.21.15.53:0/2290535732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 0x7fac4c15afb0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009870 tx=0x7fac3c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac4b200640 1 -- 172.21.15.53:0/2290535732 shutdown_connections 2024-01-16T15:08:10.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac4b200640 1 --2- 172.21.15.53:0/2290535732 >> v2:172.21.15.146:3300/0 conn(0x7fac4c13fcb0 0x7fac4c0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac4b200640 1 --2- 172.21.15.53:0/2290535732 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13cfa0 0x7fac4c15afb0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.131+0000 7fac4b200640 1 -- 172.21.15.53:0/2290535732 >> 172.21.15.53:0/2290535732 conn(0x7fac4c06a800 msgr2=0x7fac4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:10.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.132+0000 7fac4b200640 1 -- 172.21.15.53:0/2290535732 shutdown_connections 2024-01-16T15:08:10.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.132+0000 7fac4b200640 1 -- 172.21.15.53:0/2290535732 wait complete. 2024-01-16T15:08:10.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.134+0000 7fac4b200640 1 Processor -- start 2024-01-16T15:08:10.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.134+0000 7fac4b200640 1 -- start start 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.135+0000 7fac4b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac4c0cd530 0x7fac4c13b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.135+0000 7fac4b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 0x7fac4c13bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.135+0000 7fac4b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13fcb0 0x7fac4c13f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.135+0000 7fac4b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c154ac0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.135+0000 7fac4b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c154d10 con 0x7fac4c0cd530 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.135+0000 7fac4b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4c154ff0 con 0x7fac4c13fcb0 2024-01-16T15:08:10.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 0x7fac4c13bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 0x7fac4c13bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45526/0 (socket says 172.21.15.53:45526) 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 -- 172.21.15.53:0/1726208438 learned_addr learned my addr 172.21.15.53:0/1726208438 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac4a800640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13fcb0 0x7fac4c13f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49e00640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac4c0cd530 0x7fac4c13b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 -- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13fcb0 msgr2=0x7fac4c13f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13fcb0 0x7fac4c13f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 -- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac4c0cd530 msgr2=0x7fac4c13b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac4c0cd530 0x7fac4c13b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.136+0000 7fac49400640 1 -- 172.21.15.53:0/1726208438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0094d0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.137+0000 7fac49400640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 0x7fac4c13bdf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fac3400c450 tx=0x7fac3400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.137+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac340052d0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.137+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac34005430 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.137+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac34012870 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.137+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac4c1552d0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.137+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac4c155890 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.139+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4c067450 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.139+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fac340129d0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.140+0000 7fac42000640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac2406bbc0 0x7fac2406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:10.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.140+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fac34087fe0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.141+0000 7fac49e00640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac2406bbc0 0x7fac2406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:10.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.142+0000 7fac49e00640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac2406bbc0 0x7fac2406e080 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036000 tx=0x7fac3c03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:10.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.145+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fac340548a0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.232 INFO:tasks.ceph.mds.d.smithi146.stderr:2024-01-16T15:08:10.254+0000 7f91e4600640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:08:10.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.377+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac4c0b4000 con 0x7fac4c13cfa0 2024-01-16T15:08:10.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.379+0000 7fac42000640 1 -- 172.21.15.53:0/1726208438 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fac34057ec0 con 0x7fac4c13cfa0 2024-01-16T15:08:10.357 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:08:10.357 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":45,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1106083775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1106083775},{"type":"v1","addr":"172.21.15.53:6834","nonce":1106083775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2940561403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2940561403},{"type":"v1","addr":"172.21.15.146:6837","nonce":2940561403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7029,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3081217400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3081217400},{"type":"v1","addr":"172.21.15.53:6835","nonce":3081217400}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:08:03.764651+0000","modified":"2024-01-16T15:08:06.830496+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7032},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7032":{"gid":7032,"name":"d","rank":0,"incarnation":41,"state":"up:active","state_seq":3,"addr":"172.21.15.146:6836/867019165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":867019165},{"type":"v1","addr":"172.21.15.146:6836","nonce":867019165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-01-16T15:08:10.357 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 45 2024-01-16T15:08:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac2406bbc0 msgr2=0x7fac2406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac2406bbc0 0x7fac2406e080 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fac3c036000 tx=0x7fac3c03c040 comp rx=0 tx=0).stop 2024-01-16T15:08:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 msgr2=0x7fac4c13bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:10.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 0x7fac4c13bdf0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7fac3400c450 tx=0x7fac3400c910 comp rx=0 tx=0).stop 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 shutdown_connections 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fac2406bbc0 0x7fac2406e080 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fac4c13fcb0 0x7fac4c13f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fac4c13cfa0 0x7fac4c13bdf0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 --2- 172.21.15.53:0/1726208438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fac4c0cd530 0x7fac4c13b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 >> 172.21.15.53:0/1726208438 conn(0x7fac4c06a800 msgr2=0x7fac4c0af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.382+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 shutdown_connections 2024-01-16T15:08:10.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:10.383+0000 7fac4b200640 1 -- 172.21.15.53:0/1726208438 wait complete. 2024-01-16T15:08:10.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:08:03.764651+0000', 'modified': '2024-01-16T15:08:06.830496+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7032}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7032': {'gid': 7032, 'name': 'd', 'rank': 0, 'incarnation': 41, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.146:6836/867019165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.146:6834', 'nonce': 867019165}, {'type': 'v1', 'addr': '172.21.15.146:6836', 'nonce': 867019165}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:08:10.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:08:10.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-01-16T15:08:11.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.034+0000 7fe7bf000640 1 Processor -- start 2024-01-16T15:08:11.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.034+0000 7fe7bf000640 1 -- start start 2024-01-16T15:08:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bf000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7b80cf9a0 0x7fe7b815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bf000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b8070c40 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bf000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b806c470 con 0x7fe7b80ae5a0 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bf000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b8071450 con 0x7fe7b80cf9a0 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bd000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7b80cf9a0 0x7fe7b815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bd000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7b80cf9a0 0x7fe7b815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45536/0 (socket says 172.21.15.53:45536) 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7afe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe7b80b3ae0 0x7fe7b80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50792/0 (socket says 172.21.15.53:50792) 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.036+0000 7fe7bd000640 1 -- 172.21.15.53:0/1171260465 learned_addr learned my addr 172.21.15.53:0/1171260465 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.037+0000 7fe7bd000640 1 -- 172.21.15.53:0/1171260465 >> v1:172.21.15.146:6789/0 conn(0x7fe7b80ae5a0 legacy=0x7fe7b806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.037+0000 7fe7bd000640 1 -- 172.21.15.53:0/1171260465 >> v1:172.21.15.53:6789/0 conn(0x7fe7b80b3ae0 legacy=0x7fe7b80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.037+0000 7fe7bd000640 1 -- 172.21.15.53:0/1171260465 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b806c750 con 0x7fe7b80cf9a0 2024-01-16T15:08:11.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.038+0000 7fe7bd000640 1 --2- 172.21.15.53:0/1171260465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815e430 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009e50 tx=0x7fe7b002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea7e1e56d1fecb2b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.038+0000 7fe7b7400640 1 -- 172.21.15.53:0/1171260465 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b002cbb0 con 0x7fe7b80cf9a0 2024-01-16T15:08:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.038+0000 7fe7b7400640 1 -- 172.21.15.53:0/1171260465 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b002cd10 con 0x7fe7b80cf9a0 2024-01-16T15:08:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.038+0000 7fe7b7400640 1 -- 172.21.15.53:0/1171260465 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0034610 con 0x7fe7b80cf9a0 2024-01-16T15:08:11.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 -- 172.21.15.53:0/1171260465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80cf9a0 msgr2=0x7fe7b815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1171260465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815e430 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009e50 tx=0x7fe7b002c010 comp rx=0 tx=0).stop 2024-01-16T15:08:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 -- 172.21.15.53:0/1171260465 shutdown_connections 2024-01-16T15:08:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1171260465 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815e430 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 -- 172.21.15.53:0/1171260465 >> 172.21.15.53:0/1171260465 conn(0x7fe7b806a800 msgr2=0x7fe7b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 -- 172.21.15.53:0/1171260465 shutdown_connections 2024-01-16T15:08:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.039+0000 7fe7bf000640 1 -- 172.21.15.53:0/1171260465 wait complete. 2024-01-16T15:08:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.041+0000 7fe7bf000640 1 Processor -- start 2024-01-16T15:08:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.041+0000 7fe7bf000640 1 -- start start 2024-01-16T15:08:11.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7bf000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7b80ae5a0 0x7fe7b81f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7bf000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 0x7fe7b8153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7bf000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7bf000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b80cb980 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7bf000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b81f4380 con 0x7fe7b80cf9a0 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7bf000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b8153840 con 0x7fe7b80ae5a0 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.042+0000 7fe7afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 0x7fe7b8153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 0x7fe7b8153300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45548/0 (socket says 172.21.15.53:45548) 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 -- 172.21.15.53:0/1543978248 learned_addr learned my addr 172.21.15.53:0/1543978248 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7b80ae5a0 0x7fe7b81f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7bd000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 -- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7b80ae5a0 msgr2=0x7fe7b81f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7b80ae5a0 0x7fe7b81f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 -- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7b80cf9a0 msgr2=0x7fe7b815fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 -- 172.21.15.53:0/1543978248 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b0009b30 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.043+0000 7fe7afe00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 0x7fe7b8153300 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c4d0 tx=0x7fe7a000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.044+0000 7fe7b4c00640 1 -- 172.21.15.53:0/1543978248 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0007480 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.044+0000 7fe7b4c00640 1 -- 172.21.15.53:0/1543978248 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a00075e0 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.044+0000 7fe7b4c00640 1 -- 172.21.15.53:0/1543978248 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a00049e0 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.044+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b81f6e90 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.044+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b81f7450 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.046+0000 7fe7b4c00640 1 -- 172.21.15.53:0/1543978248 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe7a0004b40 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.047+0000 7fe7b4c00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79006bbc0 0x7fe79006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.047+0000 7fe7b4c00640 1 -- 172.21.15.53:0/1543978248 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe7a00874e0 con 0x7fe7b80b3ae0 2024-01-16T15:08:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.047+0000 7fe7c1c5c740 1 Processor -- start 2024-01-16T15:08:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.048+0000 7fe7c1c5c740 1 -- start start 2024-01-16T15:08:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.048+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79006bbc0 0x7fe79006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.048+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cf88280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.048+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cf870a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfcb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98cee6880 con 0x55a98cf87ea0 2024-01-16T15:08:11.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98cef5ae0 con 0x55a98cf82ba0 2024-01-16T15:08:11.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a98cef31d0 con 0x55a98cf875e0 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cf88280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cf88280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45554/0 (socket says 172.21.15.53:45554) 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 -- 172.21.15.53:0/661775139 learned_addr learned my addr 172.21.15.53:0/661775139 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 -- 172.21.15.53:0/661775139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 msgr2=0x55a98cfcb560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfcb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 -- 172.21.15.53:0/661775139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 msgr2=0x55a98cf870a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cf870a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 -- 172.21.15.53:0/661775139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a98ceb6860 con 0x55a98cf87ea0 2024-01-16T15:08:11.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.049+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79006bbc0 0x7fe79006e080 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8004ea0 tx=0x7fe7a8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.050+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cf88280 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80055f0 tx=0x7fe7a8019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab0cf35892475dba server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.050+0000 7fe7acc00640 1 -- 172.21.15.53:0/661775139 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a80188f0 con 0x55a98cf87ea0 2024-01-16T15:08:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.050+0000 7fe7acc00640 1 -- 172.21.15.53:0/661775139 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a8018a50 con 0x55a98cf87ea0 2024-01-16T15:08:11.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.050+0000 7fe7acc00640 1 -- 172.21.15.53:0/661775139 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8018dc0 con 0x55a98cf87ea0 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.050+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/661775139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 msgr2=0x55a98cf88280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.050+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cf88280 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fe7a80055f0 tx=0x7fe7a8019000 comp rx=0 tx=0).stop 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/661775139 shutdown_connections 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfcb560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cf870a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/661775139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cf88280 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/661775139 >> 172.21.15.53:0/661775139 conn(0x55a98cf49c70 msgr2=0x55a98cfcd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/661775139 shutdown_connections 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.051+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/661775139 wait complete. 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.052+0000 7fe7c1c5c740 1 -- start start 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.052+0000 7fe7c1c5c740 1 Processor -- start 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 0x55a98cfb3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf875e0 0x55a98cfb5820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf87ea0 0x55a98cfca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98ceeeac0 con 0x55a98cf82ba0 2024-01-16T15:08:11.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98ce8b2f0 con 0x55a98cf875e0 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a98cea8c70 con 0x55a98cf87ea0 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7c1c5c740 10 client.0 _ll_register_callbacks cb 0x55a98cead3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 0x55a98cfb3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 0x55a98cfb3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45564/0 (socket says 172.21.15.53:45564) 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.053+0000 7fe7b7e00640 1 -- 172.21.15.53:0/3845451876 learned_addr learned my addr 172.21.15.53:0/3845451876 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7bd000640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf87ea0 0x55a98cfca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7afe00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf875e0 0x55a98cfb5820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7b7e00640 1 -- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf87ea0 msgr2=0x55a98cfca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf87ea0 0x55a98cfca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7b7e00640 1 -- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf875e0 msgr2=0x55a98cfb5820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf875e0 0x55a98cfb5820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7b7e00640 1 -- 172.21.15.53:0/3845451876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a800f040 con 0x55a98cf82ba0 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7bd000640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf87ea0 0x55a98cfca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7afe00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf875e0 0x55a98cfb5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 0x55a98cfb3080 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009010 tx=0x7fe7a80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f9e37467a2cdd7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7a5600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7a5600640 1 -- 172.21.15.53:0/3845451876 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a8020940 con 0x55a98cf82ba0 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7a5600640 1 -- 172.21.15.53:0/3845451876 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe7a8020aa0 con 0x55a98cf82ba0 2024-01-16T15:08:11.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.054+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a98cee7aa0 con 0x55a98cf82ba0 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7a5600640 1 -- 172.21.15.53:0/3845451876 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7a8020c60 con 0x55a98cf82ba0 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 10 client.7155 fetch_fsmap learned FSMap version 45 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x55a98ceed140 con 0x55a98cf82ba0 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7a5600640 1 -- 172.21.15.53:0/3845451876 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a98cee7aa0 con 0x55a98cf82ba0 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7a5600640 1 -- 172.21.15.53:0/3845451876 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 45) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe7a8035430 con 0x55a98cf82ba0 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 10 client.7155 fetch_fsmap finished waiting for FSMap version 45 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 10 client.7155 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 20 client.7155 populate_metadata read hostname 'smithi053' 2024-01-16T15:08:11.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfaed30 0x55a98cfd4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 4 client.7155 mds_command: new command op to 7032 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:08:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.055+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 --> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a98cee77c0 con 0x55a98cfaed30 2024-01-16T15:08:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.056+0000 7fe7afe00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfaed30 0x55a98cfd4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.057+0000 7fe7afe00640 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfaed30 0x55a98cfd4090 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4746356807a0f7c5 server_cookie=cdaa08ef75bc872e in_seq=0 out_seq=0 2024-01-16T15:08:11.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.057+0000 7fe7a5600640 10 client.7155 ms_handle_connect on v2:172.21.15.146:6834/867019165 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7a5600640 1 -- 172.21.15.53:0/3845451876 <== mds.0 v2:172.21.15.146:6834/867019165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fe7a0086a90 con 0x55a98cfaed30 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7a5600640 10 client.7155 handle_command_reply: tid=0 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 client.7155 shutdown 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 msgr2=0x55a98cfb3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 0x55a98cfb3080 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8009010 tx=0x7fe7a80036f0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 shutdown_connections 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfaed30 0x55a98cfd4090 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf87ea0 0x55a98cfca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf875e0 0x55a98cfb5820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/3845451876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf82ba0 0x55a98cfb3080 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7a5600640 0 client.7155 ms_handle_reset on v2:172.21.15.146:6834/867019165 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 >> 172.21.15.53:0/3845451876 conn(0x55a98cf49c70 msgr2=0x55a98cfd0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 shutdown_connections 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.059+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/3845451876 wait complete. 2024-01-16T15:08:11.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.060+0000 7fe7c1c5c740 20 client.7155 trim_cache size 0 max 16384 2024-01-16T15:08:11.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.077+0000 7fe7c1c5c740 1 Processor -- start 2024-01-16T15:08:11.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.077+0000 7fe7c1c5c740 1 -- start start 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cfb5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfc8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98cc51f20 con 0x55a98cf87ea0 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98cfd1da0 con 0x55a98cf82ba0 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a98cfca5a0 con 0x55a98cf875e0 2024-01-16T15:08:11.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7bd000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7bd000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45572/0 (socket says 172.21.15.53:45572) 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.078+0000 7fe7bd000640 1 -- 172.21.15.53:0/2544245579 learned_addr learned my addr 172.21.15.53:0/2544245579 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7afe00640 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfc8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cfb5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7bd000640 1 -- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 msgr2=0x55a98cfc8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7bd000640 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfc8a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7bd000640 1 -- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 msgr2=0x55a98cfb5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7bd000640 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cfb5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7bd000640 1 -- 172.21.15.53:0/2544245579 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a98ce8ab80 con 0x55a98cf87ea0 2024-01-16T15:08:11.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.079+0000 7fe7bd000640 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8f80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe7b00093e0 tx=0x7fe7b002e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87afaf9b1fa76a87 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7ad600640 1 -- 172.21.15.53:0/2544245579 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b003d5f0 con 0x55a98cf87ea0 2024-01-16T15:08:11.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7ad600640 1 -- 172.21.15.53:0/2544245579 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b003d750 con 0x55a98cf87ea0 2024-01-16T15:08:11.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7ad600640 1 -- 172.21.15.53:0/2544245579 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b003db00 con 0x55a98cf87ea0 2024-01-16T15:08:11.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/2544245579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 msgr2=0x55a98cfc8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8f80 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fe7b00093e0 tx=0x7fe7b002e070 comp rx=0 tx=0).stop 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/2544245579 shutdown_connections 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8f80 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf875e0 0x55a98cfc8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/2544245579 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf82ba0 0x55a98cfb5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/2544245579 >> 172.21.15.53:0/2544245579 conn(0x55a98cf49c70 msgr2=0x55a98cfb2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/2544245579 shutdown_connections 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.080+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/2544245579 wait complete. 2024-01-16T15:08:11.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.081+0000 7fe7c1c5c740 1 -- start start 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.081+0000 7fe7c1c5c740 1 Processor -- start 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf82ba0 0x55a98cfcaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 0x55a98cfcafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98cfcbff0 con 0x55a98cf875e0 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a98d0389c0 con 0x55a98cf87ea0 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a98cfb5120 con 0x55a98cf82ba0 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.082+0000 7fe7c1c5c740 10 client.0 _ll_register_callbacks cb 0x55a98cfb68b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 0x55a98cfcafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 0x55a98cfcafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45582/0 (socket says 172.21.15.53:45582) 2024-01-16T15:08:11.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 -- 172.21.15.53:0/4016311196 learned_addr learned my addr 172.21.15.53:0/4016311196 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 -- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf82ba0 msgr2=0x55a98cfcaaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf82ba0 0x55a98cfcaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7bd000640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf82ba0 0x55a98cfcaaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 -- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf87ea0 msgr2=0x55a98cfc8a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 -- 172.21.15.53:0/4016311196 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b806c750 con 0x55a98cf875e0 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7afe00640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 0x55a98cfcafe0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0087450 tx=0x7fe7a005fe20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06d2b95bcdfd20e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7a5600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.083+0000 7fe7a5600640 1 -- 172.21.15.53:0/4016311196 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a0054220 con 0x55a98cf875e0 2024-01-16T15:08:11.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7a5600640 1 -- 172.21.15.53:0/4016311196 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe7a0054380 con 0x55a98cf875e0 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7a8018a50 con 0x55a98cf875e0 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7a5600640 1 -- 172.21.15.53:0/4016311196 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7a0057000 con 0x55a98cf875e0 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 10 client.7161 fetch_fsmap learned FSMap version 45 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=2+}) v3 -- 0x55a98ceb0170 con 0x55a98cf875e0 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7a5600640 1 -- 172.21.15.53:0/4016311196 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a8018a50 con 0x55a98cf875e0 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7a5600640 1 -- 172.21.15.53:0/4016311196 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 45) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fe7a0076170 con 0x55a98cf875e0 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 10 client.7161 fetch_fsmap finished waiting for FSMap version 45 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 10 client.7161 resolve_mds: resolved name 'd' to daemon mds.d 2024-01-16T15:08:11.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 20 client.7161 populate_metadata read hostname 'smithi053' 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfc9480 0x55a98d0371d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 4 client.7161 mds_command: new command op to 7032 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.084+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 --> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a98cbef6c0 con 0x55a98cfc9480 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.085+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfc9480 0x55a98d0371d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7b7e00640 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfc9480 0x55a98d0371d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad10d9f1a6685305 server_cookie=55402ca87b6b527 in_seq=0 out_seq=0 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7a5600640 10 client.7161 ms_handle_connect on v2:172.21.15.146:6834/867019165 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7a5600640 1 -- 172.21.15.53:0/4016311196 <== mds.0 v2:172.21.15.146:6834/867019165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55a98cbef6c0 con 0x55a98cfc9480 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7a5600640 10 client.7161 handle_command_reply: tid=0 2024-01-16T15:08:11.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 client.7161 shutdown 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 msgr2=0x55a98cfcafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 0x55a98cfcafe0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7fe7a0087450 tx=0x7fe7a005fe20 comp rx=0 tx=0).stop 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 shutdown_connections 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:6834/867019165,v1:172.21.15.146:6836/867019165] conn(0x55a98cfc9480 0x55a98d0371d0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a98cf87ea0 0x55a98cfc8a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a98cf875e0 0x55a98cfcafe0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 --2- 172.21.15.53:0/4016311196 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a98cf82ba0 0x55a98cfcaaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7a5600640 0 client.7161 ms_handle_reset on v2:172.21.15.146:6834/867019165 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.086+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 >> 172.21.15.53:0/4016311196 conn(0x55a98cf49c70 msgr2=0x55a98cfaf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 shutdown_connections 2024-01-16T15:08:11.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7c1c5c740 1 -- 172.21.15.53:0/4016311196 wait complete. 2024-01-16T15:08:11.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7c1c5c740 20 client.7161 trim_cache size 0 max 16384 2024-01-16T15:08:11.071 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:08:11.071 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:08:11.071 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:08:11.071 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 7032, 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 6.7809030379999999, 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 45, 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 72, 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 72, 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 12.173503886000001 2024-01-16T15:08:11.072 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79006bbc0 msgr2=0x7fe79006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79006bbc0 0x7fe79006e080 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fe7a8004ea0 tx=0x7fe7a8009040 comp rx=0 tx=0).stop 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 msgr2=0x7fe7b8153300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 0x7fe7b8153300 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fe7a000c4d0 tx=0x7fe7a000c990 comp rx=0 tx=0).stop 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 shutdown_connections 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe79006bbc0 0x7fe79006e080 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7b80cf9a0 0x7fe7b815fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7b80b3ae0 0x7fe7b8153300 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 --2- 172.21.15.53:0/1543978248 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7b80ae5a0 0x7fe7b81f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.087+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 >> 172.21.15.53:0/1543978248 conn(0x7fe7b806a800 msgr2=0x7fe7b80cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.088+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 shutdown_connections 2024-01-16T15:08:11.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.088+0000 7fe7bf000640 1 -- 172.21.15.53:0/1543978248 wait complete. 2024-01-16T15:08:11.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:08:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.326+0000 7f07cec00640 1 Processor -- start 2024-01-16T15:08:11.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.327+0000 7f07cec00640 1 -- start start 2024-01-16T15:08:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07cec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07cec00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07c80cf7d0 0x7f07c815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07cec00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c80cfc30 con 0x7f07c815e1e0 2024-01-16T15:08:11.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07cec00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c80b1ee0 con 0x7f07c80cf7d0 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07cec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80b2780 con 0x7f07c8143ae0 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07c7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f07c80cf7d0 0x7f07c815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07ccc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f07c815e1e0 0x7f07c815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50794/0 (socket says 172.21.15.53:50794) 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.328+0000 7f07ccc00640 1 -- 172.21.15.53:0/272674254 learned_addr learned my addr 172.21.15.53:0/272674254 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c7e00640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c7400640 1 -- 172.21.15.53:0/272674254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 msgr2=0x7f07c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c7400640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c7400640 1 -- 172.21.15.53:0/272674254 >> v1:172.21.15.53:6789/0 conn(0x7f07c815e1e0 legacy=0x7f07c815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c7400640 1 -- 172.21.15.53:0/272674254 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c815e990 con 0x7f07c80cf7d0 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c7e00640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:11.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.329+0000 7f07c6a00640 1 -- 172.21.15.53:0/272674254 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718154985 0 0) 0x7f07c80cfc30 con 0x7f07c815e1e0 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.330+0000 7f07c7400640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c80cf7d0 0x7f07c815dca0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9059622d7741a18 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.330+0000 7f07c6a00640 1 -- 172.21.15.53:0/272674254 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b002d040 con 0x7f07c80cf7d0 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.330+0000 7f07c6a00640 1 -- 172.21.15.53:0/272674254 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b0031810 con 0x7f07c80cf7d0 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.330+0000 7f07c6a00640 1 -- 172.21.15.53:0/272674254 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b0031ba0 con 0x7f07c80cf7d0 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 -- 172.21.15.53:0/272674254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c80cf7d0 msgr2=0x7f07c815dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c80cf7d0 0x7f07c815dca0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 -- 172.21.15.53:0/272674254 shutdown_connections 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c80cf7d0 0x7f07c815dca0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 --2- 172.21.15.53:0/272674254 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 -- 172.21.15.53:0/272674254 >> 172.21.15.53:0/272674254 conn(0x7f07c806a800 msgr2=0x7f07c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 -- 172.21.15.53:0/272674254 shutdown_connections 2024-01-16T15:08:11.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.331+0000 7f07cec00640 1 -- 172.21.15.53:0/272674254 wait complete. 2024-01-16T15:08:11.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.333+0000 7f07cec00640 1 Processor -- start 2024-01-16T15:08:11.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.333+0000 7f07cec00640 1 -- start start 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07cec00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c80cf7d0 0x7f07c80c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07cec00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07cec00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 0x7f07c80cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07cec00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c80cb7f0 con 0x7f07c815e1e0 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07cec00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8134a30 con 0x7f07c80cf7d0 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07cec00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c81352d0 con 0x7f07c8143ae0 2024-01-16T15:08:11.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07ccc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 0x7f07c80cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07ccc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 0x7f07c80cb2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45608/0 (socket says 172.21.15.53:45608) 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07ccc00640 1 -- 172.21.15.53:0/78919354 learned_addr learned my addr 172.21.15.53:0/78919354 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07c7400640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.334+0000 7f07c7e00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c80cf7d0 0x7f07c80c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07ccc00640 1 -- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 msgr2=0x7f07c80cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07ccc00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07ccc00640 1 -- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c80cf7d0 msgr2=0x7f07c80c6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07ccc00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c80cf7d0 0x7f07c80c6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07ccc00640 1 -- 172.21.15.53:0/78919354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b0009400 con 0x7f07c815e1e0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07c7e00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c80cf7d0 0x7f07c80c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07ccc00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 0x7f07c80cb2b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f07c000c7b0 tx=0x7f07c000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000a640 con 0x7f07c815e1e0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c000a7c0 con 0x7f07c815e1e0 2024-01-16T15:08:11.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c000ab70 con 0x7f07c815e1e0 2024-01-16T15:08:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.335+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c80c4240 con 0x7f07c815e1e0 2024-01-16T15:08:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.336+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c8134c80 con 0x7f07c815e1e0 2024-01-16T15:08:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.337+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c80675d0 con 0x7f07c815e1e0 2024-01-16T15:08:11.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.338+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f07c000acd0 con 0x7f07c815e1e0 2024-01-16T15:08:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.339+0000 7f07bfe00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f07a006bbc0 0x7f07a006e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.339+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f07c0087860 con 0x7f07c815e1e0 2024-01-16T15:08:11.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.339+0000 7f07c7e00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f07a006bbc0 0x7f07a006e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:11.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.340+0000 7f07c7e00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f07a006bbc0 0x7f07a006e080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07b8006fd0 tx=0x7f07b8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:11.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.343+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f07c0054180 con 0x7f07c815e1e0 2024-01-16T15:08:11.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.581+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07c8135410 con 0x7f07c815e1e0 2024-01-16T15:08:11.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.582+0000 7f07bfe00640 1 -- 172.21.15.53:0/78919354 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7f07c00577a0 con 0x7f07c815e1e0 2024-01-16T15:08:11.559 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:08:11.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f07a006bbc0 msgr2=0x7f07a006e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f07a006bbc0 0x7f07a006e080 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f07b8006fd0 tx=0x7f07b8008040 comp rx=0 tx=0).stop 2024-01-16T15:08:11.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 msgr2=0x7f07c80cb2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 0x7f07c80cb2b0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f07c000c7b0 tx=0x7f07c000cc70 comp rx=0 tx=0).stop 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 shutdown_connections 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f07a006bbc0 0x7f07a006e080 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f07c815e1e0 0x7f07c80cb2b0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f07c8143ae0 0x7f07c80cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 --2- 172.21.15.53:0/78919354 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f07c80cf7d0 0x7f07c80c6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 >> 172.21.15.53:0/78919354 conn(0x7f07c806a800 msgr2=0x7f07c80704d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 shutdown_connections 2024-01-16T15:08:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:11.585+0000 7f07cec00640 1 -- 172.21.15.53:0/78919354 wait complete. 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:08:11.583 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:08:11.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:11.583 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:08:11.637 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:08:11.637 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:11.638 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 42159sec preferred_lft 42159sec 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.639 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:08:11.640 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:11.641 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:11.641 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:08:11.734 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:08:11.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:11.738 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:08:11.794 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:08:11.794 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:08:11.794 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:08:11.795 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:08:11.795 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:08:11.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:11.796 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:08:11.796 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:08:11.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:08:11.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:08:11.796 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:08:11.796 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:08:11.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:11.996 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:08:12.049 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:08:12.049 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:08:12.050 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:08:12.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:12.050 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:08:12.149 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:08:12.149 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:08:12.149 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:12.206 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:08:12.207 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:08:12.207 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:12.262 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:08:12.354 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:08:12.404 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:08:12.404 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:08:12.404 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:08:12.404 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:08:12.404 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:08:12.405 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:12.406 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:12.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:12.407 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:08:12.462 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:08:12.462 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:08:12.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:08:12.547 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:08:12.596 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:08:12.597 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:08:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:08:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:08:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:12.598 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:12.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:12.599 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:08:12.661 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:08:12.684+0000 7fc674e3b480 -1 init, newargv = 0x560e77a2fb30 newargc=15 2024-01-16T15:08:12.662 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[61266]: starting ceph client 2024-01-16T15:08:12.672 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[61266]: starting fuse 2024-01-16T15:08:13.742 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:08:13.848 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:08:13.907 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:08:13.907 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:08:13.907 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:08:13.907 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:08:13.908 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:13.909 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:08:13.910 INFO:teuthology.orchestra.run.smithi053.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:13.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:13.911 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:08:13.967 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:08:13.967 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:08:13.968 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:08:13.969 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:08:14.158 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.61266.asok 2024-01-16T15:08:14.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:08:14.177 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.61266.asok 2024-01-16T15:08:14.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:14.177 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.61266.asok status 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "61266", 2024-01-16T15:08:14.422 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 7176, 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 7176 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 2099710275 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:08:14.423 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 2099710275 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.7176 192.168.0.1:0/2099710275", 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/2099710275", 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 45, 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 72, 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:08:14.424 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:08:14.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:14.438 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:08:14.438 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:14.490 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:08:14.490 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:14.491 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:14.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:08:14.589 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:08:14.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.841+0000 7fad05800640 1 Processor -- start 2024-01-16T15:08:14.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.841+0000 7fad05800640 1 -- start start 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.842+0000 7fad05800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad0015d500 0x7fad000cd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7fad05800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7fad05800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad000b2490 con 0x7fad0015d8e0 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7fad05800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0006b240 con 0x7fad0015d500 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7fad05800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad000cffc0 con 0x7fad000cc720 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad0015d500 0x7fad000cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fad0015d500 0x7fad000cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46428/0 (socket says 172.21.15.53:46428) 2024-01-16T15:08:14.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 -- 172.21.15.53:0/672761537 learned_addr learned my addr 172.21.15.53:0/672761537 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:14.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 -- 172.21.15.53:0/672761537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 msgr2=0x7fad000cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:14.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 --2- 172.21.15.53:0/672761537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 -- 172.21.15.53:0/672761537 >> v1:172.21.15.146:6789/0 conn(0x7fad0015d8e0 legacy=0x7fad0006b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:14.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.843+0000 7facffe00640 1 -- 172.21.15.53:0/672761537 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad000d02a0 con 0x7fad0015d500 2024-01-16T15:08:14.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.845+0000 7facffe00640 1 --2- 172.21.15.53:0/672761537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad000cd790 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7facf0009870 tx=0x7facf002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01d91a58a1792c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.845+0000 7facfea00640 1 -- 172.21.15.53:0/672761537 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0031640 con 0x7fad0015d500 2024-01-16T15:08:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.845+0000 7facfea00640 1 -- 172.21.15.53:0/672761537 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf00317a0 con 0x7fad0015d500 2024-01-16T15:08:14.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.845+0000 7facfea00640 1 -- 172.21.15.53:0/672761537 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7facf0031b30 con 0x7fad0015d500 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 -- 172.21.15.53:0/672761537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 msgr2=0x7fad000cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 --2- 172.21.15.53:0/672761537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad000cd790 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7facf0009870 tx=0x7facf002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 -- 172.21.15.53:0/672761537 shutdown_connections 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 --2- 172.21.15.53:0/672761537 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 --2- 172.21.15.53:0/672761537 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad000cd790 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 -- 172.21.15.53:0/672761537 >> 172.21.15.53:0/672761537 conn(0x7fad000d0f10 msgr2=0x7fad000d1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:14.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 -- 172.21.15.53:0/672761537 shutdown_connections 2024-01-16T15:08:14.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.846+0000 7fad05800640 1 -- 172.21.15.53:0/672761537 wait complete. 2024-01-16T15:08:14.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.848+0000 7fad05800640 1 Processor -- start 2024-01-16T15:08:14.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.848+0000 7fad05800640 1 -- start start 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7fad05800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000c8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7fad05800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad001e87c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7fad05800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad0015d8e0 0x7fad001e7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7fad05800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000c8ad0 con 0x7fad0015d500 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7fad05800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad000c8db0 con 0x7fad0015d8e0 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7fad05800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad000c9090 con 0x7fad000cc720 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7facff400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad001e87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:14.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.849+0000 7facff400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad001e87c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46432/0 (socket says 172.21.15.53:46432) 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 -- 172.21.15.53:0/841123907 learned_addr learned my addr 172.21.15.53:0/841123907 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facffe00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000c8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7fad04e00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad0015d8e0 0x7fad001e7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 -- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 msgr2=0x7fad000c8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000c8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 -- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad0015d8e0 msgr2=0x7fad001e7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad0015d8e0 0x7fad001e7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 -- 172.21.15.53:0/841123907 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facf00094d0 con 0x7fad0015d500 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7fad04e00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad0015d8e0 0x7fad001e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facffe00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:14.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.850+0000 7facff400640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad001e87c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7face800b3e0 tx=0x7face800b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.851+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8011830 con 0x7fad0015d500 2024-01-16T15:08:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.851+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7face80119b0 con 0x7fad0015d500 2024-01-16T15:08:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.851+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7face8011d60 con 0x7fad0015d500 2024-01-16T15:08:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.851+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad000c9370 con 0x7fad0015d500 2024-01-16T15:08:14.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.851+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad000c8780 con 0x7fad0015d500 2024-01-16T15:08:14.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.852+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad000c9890 con 0x7fad0015d500 2024-01-16T15:08:14.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.853+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7face801a400 con 0x7fad0015d500 2024-01-16T15:08:14.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.854+0000 7facf7e00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facd806bbc0 0x7facd806e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:14.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.854+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7face8088630 con 0x7fad0015d500 2024-01-16T15:08:14.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.854+0000 7facffe00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facd806bbc0 0x7facd806e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:14.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.856+0000 7facffe00640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facd806bbc0 0x7facd806e080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7facf0036000 tx=0x7facf003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:14.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:14.859+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad000c9890 con 0x7fad0015d500 2024-01-16T15:08:15.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.095+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fad000c9890 con 0x7fad0015d500 2024-01-16T15:08:15.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.096+0000 7facf7e00640 1 -- 172.21.15.53:0/841123907 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v45) v1 ==== 53+0+73 (secure 0 0 0) 0x7fad000c9890 con 0x7fad0015d500 2024-01-16T15:08:15.074 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:08:15.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facd806bbc0 msgr2=0x7facd806e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:15.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facd806bbc0 0x7facd806e080 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7facf0036000 tx=0x7facf003c040 comp rx=0 tx=0).stop 2024-01-16T15:08:15.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 msgr2=0x7fad001e87c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:15.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad001e87c0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7face800b3e0 tx=0x7face800b8a0 comp rx=0 tx=0).stop 2024-01-16T15:08:15.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 shutdown_connections 2024-01-16T15:08:15.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7facd806bbc0 0x7facd806e080 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:15.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad0015d8e0 0x7fad001e7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:15.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad0015d500 0x7fad001e87c0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:15.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.098+0000 7fad05800640 1 --2- 172.21.15.53:0/841123907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad000cc720 0x7fad000c8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:15.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.099+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 >> 172.21.15.53:0/841123907 conn(0x7fad000d0f10 msgr2=0x7fad000c4a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:15.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.099+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 shutdown_connections 2024-01-16T15:08:15.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:15.099+0000 7fad05800640 1 -- 172.21.15.53:0/841123907 wait complete. 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:08:15.097 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:08:15.098 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:08:15.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.098 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:08:15.157 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:08:15.157 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:08:15.157 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:08:15.157 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.157 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:08:15.157 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 42035sec preferred_lft 42035sec 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.158 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:08:15.159 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:08:15.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.160 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:08:15.244 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:08:15.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.248 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:08:15.297 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:08:15.297 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:08:15.297 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:08:15.298 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:08:15.298 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:08:15.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.298 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:08:15.298 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:08:15.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:08:15.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:08:15.298 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:08:15.299 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:08:15.478 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.478 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:08:15.530 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:08:15.530 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:08:15.531 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:08:15.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.531 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:08:15.628 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:08:15.628 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:08:15.628 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:08:15.681 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:08:15.681 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:08:15.681 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:08:15.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:08:15.824 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:08:15.880 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:08:15.881 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:08:15.882 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:08:15.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:15.883 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:08:15.934 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:08:15.935 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:08:15.939 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:08:16.016 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:08:16.060 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:08:16.061 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:08:16.062 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:08:16.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:16.063 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:08:16.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:08:16.148+0000 7f731ee3c480 -1 init, newargv = 0x55d9248366e0 newargc=15 2024-01-16T15:08:16.126 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44911]: starting ceph client 2024-01-16T15:08:16.136 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44911]: starting fuse 2024-01-16T15:08:16.148 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:08:16.148 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:08:16.149 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:08:16.150 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:08:16.342 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.44911.asok 2024-01-16T15:08:16.346 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:08:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:08:16.360 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44911.asok 2024-01-16T15:08:16.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:16.360 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44911.asok status 2024-01-16T15:08:16.599 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:08:16.599 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:08:16.599 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:08:16.599 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "44911", 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6187, 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 6187 2024-01-16T15:08:16.600 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2447443791 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2447443791 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.6187 192.168.0.1:0/2447443791", 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/2447443791", 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 45, 2024-01-16T15:08:16.601 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 72, 2024-01-16T15:08:16.602 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:08:16.602 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:08:16.602 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:08:16.602 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:08:16.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:16.614 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:08:16.614 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:08:16.665 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:08:16.666 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:08:16.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:16.666 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:08:16.757 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:08:16.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.015+0000 7fba42000640 1 Processor -- start 2024-01-16T15:08:16.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.015+0000 7fba42000640 1 -- start start 2024-01-16T15:08:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.017+0000 7fba42000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba3c070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.017+0000 7fba42000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c071460 con 0x7fba3c157c30 2024-01-16T15:08:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba42000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba3c15deb0 con 0x7fba3c071030 2024-01-16T15:08:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba42000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15ea30 con 0x7fba3c0cd600 2024-01-16T15:08:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba3b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fba3c157c30 0x7fba3c1618f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:32880/0 (socket says 172.21.15.53:32880) 2024-01-16T15:08:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba3b400640 1 -- 172.21.15.53:0/913595833 learned_addr learned my addr 172.21.15.53:0/913595833 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba33e00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba3c070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149961329 0 0) 0x7fba3c071460 con 0x7fba3c157c30 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.018+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba1c003520 con 0x7fba3c157c30 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1076488504 0 0) 0x7fba1c003520 con 0x7fba3c157c30 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 msgr2=0x7fba3c070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba3c070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 >> v1:172.21.15.146:6789/0 conn(0x7fba3c071030 legacy=0x7fba3c06dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c15e190 con 0x7fba3c157c30 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fba2c001690 con 0x7fba3c157c30 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 >> v1:172.21.15.53:6789/0 conn(0x7fba3c157c30 legacy=0x7fba3c1618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba1c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1c004d00 0x7fba1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c15ea30 con 0x7fba1c003f70 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c00bac0 con 0x7fba1c004d00 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba1c00bc80 con 0x7fba3c0cd600 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.019+0000 7fba33e00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba3b400640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba1c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba3be00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1c004d00 0x7fba1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba33e00640 1 -- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 msgr2=0x7fba1c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba33e00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba1c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba33e00640 1 -- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1c004d00 msgr2=0x7fba1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:16.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba33e00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba1c004d00 0x7fba1c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:17.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba33e00640 1 -- 172.21.15.53:0/913595833 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba1c003520 con 0x7fba1c003f70 2024-01-16T15:08:17.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.020+0000 7fba3b400640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c0cd600 0x7fba1c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.021+0000 7fba33e00640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba1c0047c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fba24009ac0 tx=0x7fba1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80850591384f1e49 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.021+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2402c600 con 0x7fba1c003f70 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.021+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2402c760 con 0x7fba1c003f70 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.021+0000 7fba3aa00640 1 -- 172.21.15.53:0/913595833 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2402caf0 con 0x7fba1c003f70 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba42000640 1 -- 172.21.15.53:0/913595833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 msgr2=0x7fba1c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba42000640 1 --2- 172.21.15.53:0/913595833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba1c0047c0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fba24009ac0 tx=0x7fba1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba3be00640 1 -- 172.21.15.53:0/913595833 reap_dead start 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba42000640 1 -- 172.21.15.53:0/913595833 shutdown_connections 2024-01-16T15:08:17.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba42000640 1 -- 172.21.15.53:0/913595833 >> 172.21.15.53:0/913595833 conn(0x7fba3c0cf700 msgr2=0x7fba3c0cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:17.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba42000640 1 -- 172.21.15.53:0/913595833 shutdown_connections 2024-01-16T15:08:17.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.022+0000 7fba42000640 1 -- 172.21.15.53:0/913595833 wait complete. 2024-01-16T15:08:17.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.024+0000 7fba42000640 1 Processor -- start 2024-01-16T15:08:17.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.024+0000 7fba42000640 1 -- start start 2024-01-16T15:08:17.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.025+0000 7fba42000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba3c15ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.025+0000 7fba42000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c145ed0 0x7fba3c15b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.025+0000 7fba42000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba3c0acc00 0x7fba3c0ad090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.025+0000 7fba42000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c1464f0 con 0x7fba1c003f70 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.025+0000 7fba42000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0ad9c0 con 0x7fba3c0acc00 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.025+0000 7fba42000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c0adca0 con 0x7fba3c145ed0 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba3c15ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba3c15ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46452/0 (socket says 172.21.15.53:46452) 2024-01-16T15:08:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 -- 172.21.15.53:0/464657142 learned_addr learned my addr 172.21.15.53:0/464657142 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba33e00640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c145ed0 0x7fba3c15b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3be00640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba3c0acc00 0x7fba3c0ad090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 -- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c145ed0 msgr2=0x7fba3c15b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c145ed0 0x7fba3c15b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 -- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba3c0acc00 msgr2=0x7fba3c0ad090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba3c0acc00 0x7fba3c0ad090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 -- 172.21.15.53:0/464657142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba3c0adf80 con 0x7fba1c003f70 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.026+0000 7fba3b400640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba3c15ae50 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00e8c0 tx=0x7fba2c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.027+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00c870 con 0x7fba1c003f70 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.027+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba2c00c9d0 con 0x7fba1c003f70 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.027+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c00cd80 con 0x7fba1c003f70 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.027+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c0ae260 con 0x7fba1c003f70 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.027+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c0ca640 con 0x7fba1c003f70 2024-01-16T15:08:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.028+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c15da50 con 0x7fba1c003f70 2024-01-16T15:08:17.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.029+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fba2c01d460 con 0x7fba1c003f70 2024-01-16T15:08:17.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.030+0000 7fba33400640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba1406bbc0 0x7fba1406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:17.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.030+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fba2c088740 con 0x7fba1c003f70 2024-01-16T15:08:17.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.031+0000 7fba33e00640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba1406bbc0 0x7fba1406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.032+0000 7fba33e00640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba1406bbc0 0x7fba1406e080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fba24036000 tx=0x7fba24030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.035+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba2c054e10 con 0x7fba1c003f70 2024-01-16T15:08:17.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:17.271+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fba3c0675d0 con 0x7fba1c003f70 2024-01-16T15:08:18.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.232+0000 7fba33400640 1 -- 172.21.15.53:0/464657142 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v46) v1 ==== 136+0+0 (secure 0 0 0) 0x7fba2c058430 con 0x7fba1c003f70 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba1406bbc0 msgr2=0x7fba1406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba1406bbc0 0x7fba1406e080 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fba24036000 tx=0x7fba24030000 comp rx=0 tx=0).stop 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 msgr2=0x7fba3c15ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba3c15ae50 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00e8c0 tx=0x7fba2c00ed80 comp rx=0 tx=0).stop 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 shutdown_connections 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba1406bbc0 0x7fba1406e080 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba3c0acc00 0x7fba3c0ad090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba3c145ed0 0x7fba3c15b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 --2- 172.21.15.53:0/464657142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba1c003f70 0x7fba3c15ae50 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.237+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 >> 172.21.15.53:0/464657142 conn(0x7fba3c0cf700 msgr2=0x7fba3c0cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:18.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.238+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 shutdown_connections 2024-01-16T15:08:18.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.238+0000 7fba42000640 1 -- 172.21.15.53:0/464657142 wait complete. 2024-01-16T15:08:18.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:08:18.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.487+0000 7fe67d000640 1 Processor -- start 2024-01-16T15:08:18.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.487+0000 7fe67d000640 1 -- start start 2024-01-16T15:08:18.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.488+0000 7fe67d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 0x7fe67815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.488+0000 7fe67d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780b1ee0 con 0x7fe678143ae0 2024-01-16T15:08:18.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.488+0000 7fe67d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6780b2780 con 0x7fe67815e1e0 2024-01-16T15:08:18.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.488+0000 7fe67d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe67815e990 con 0x7fe6780cf9a0 2024-01-16T15:08:18.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.489+0000 7fe675e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe678143ae0 0x7fe6780cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:32890/0 (socket says 172.21.15.53:32890) 2024-01-16T15:08:18.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.489+0000 7fe675e00640 1 -- 172.21.15.53:0/2805780793 learned_addr learned my addr 172.21.15.53:0/2805780793 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.489+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736461958 0 0) 0x7fe6780b1ee0 con 0x7fe678143ae0 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.489+0000 7fe66fe00640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 0x7fe67815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.489+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe658003520 con 0x7fe678143ae0 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3385094068 0 0) 0x7fe658003520 con 0x7fe678143ae0 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 msgr2=0x7fe67815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 0x7fe67815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 >> v1:172.21.15.146:6789/0 conn(0x7fe67815e1e0 legacy=0x7fe6780702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6780b2130 con 0x7fe678143ae0 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe668001670 con 0x7fe678143ae0 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 >> v1:172.21.15.53:6789/0 conn(0x7fe678143ae0 legacy=0x7fe6780cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 0x7fe6580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe658004d00 0x7fe658005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe67815e990 con 0x7fe658004d00 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65800dc70 con 0x7fe6580039a0 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe65800de30 con 0x7fe6780cf9a0 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.490+0000 7fe676800640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe658004d00 0x7fe658005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe66fe00640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 0x7fe6580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe675e00640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe676800640 1 -- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 msgr2=0x7fe6580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe676800640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6780cf9a0 0x7fe6580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe676800640 1 -- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 msgr2=0x7fe658003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe676800640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6580039a0 0x7fe658003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.491+0000 7fe676800640 1 -- 172.21.15.53:0/2805780793 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658003520 con 0x7fe658004d00 2024-01-16T15:08:18.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.492+0000 7fe676800640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe658004d00 0x7fe658005180 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe670001e20 tx=0x7fe6580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=383bc48f56dcfcf6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:18.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.492+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670003b70 con 0x7fe658004d00 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.492+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670003cd0 con 0x7fe658004d00 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.492+0000 7fe675400640 1 -- 172.21.15.53:0/2805780793 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670005310 con 0x7fe658004d00 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe67d000640 1 -- 172.21.15.53:0/2805780793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe658004d00 msgr2=0x7fe658005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe67d000640 1 --2- 172.21.15.53:0/2805780793 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe658004d00 0x7fe658005180 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7fe670001e20 tx=0x7fe6580057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe676800640 1 -- 172.21.15.53:0/2805780793 reap_dead start 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe67d000640 1 -- 172.21.15.53:0/2805780793 shutdown_connections 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe67d000640 1 -- 172.21.15.53:0/2805780793 >> 172.21.15.53:0/2805780793 conn(0x7fe67806a800 msgr2=0x7fe67806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe67d000640 1 -- 172.21.15.53:0/2805780793 shutdown_connections 2024-01-16T15:08:18.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.493+0000 7fe67d000640 1 -- 172.21.15.53:0/2805780793 wait complete. 2024-01-16T15:08:18.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.495+0000 7fe67d000640 1 Processor -- start 2024-01-16T15:08:18.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.495+0000 7fe67d000640 1 -- start start 2024-01-16T15:08:18.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe67d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe678159d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe67d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f72c0 0x7fe678136000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe67d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 0x7fe6780c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe67d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6780c5150 con 0x7fe6780c4760 2024-01-16T15:08:18.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe67d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe678136970 con 0x7fe658004d00 2024-01-16T15:08:18.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe67d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe678136c50 con 0x7fe6781f72c0 2024-01-16T15:08:18.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe676800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 0x7fe6780c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe676800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 0x7fe6780c4c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46480/0 (socket says 172.21.15.53:46480) 2024-01-16T15:08:18.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.496+0000 7fe676800640 1 -- 172.21.15.53:0/1748402975 learned_addr learned my addr 172.21.15.53:0/1748402975 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:18.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe66fe00640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f72c0 0x7fe678136000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe675e00640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe678159d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe676800640 1 -- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f72c0 msgr2=0x7fe678136000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe676800640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f72c0 0x7fe678136000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe676800640 1 -- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 msgr2=0x7fe678159d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe676800640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe678159d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe676800640 1 -- 172.21.15.53:0/1748402975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe678136f30 con 0x7fe6780c4760 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe676800640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 0x7fe6780c4c10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe670005040 tx=0x7fe6700014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67003e070 con 0x7fe6780c4760 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe670003e40 con 0x7fe6780c4760 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe670006d50 con 0x7fe6780c4760 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.497+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe678137210 con 0x7fe6780c4760 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.498+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe678136540 con 0x7fe6780c4760 2024-01-16T15:08:18.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.499+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6780675d0 con 0x7fe6780c4760 2024-01-16T15:08:18.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.500+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7fe670007030 con 0x7fe6780c4760 2024-01-16T15:08:18.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.501+0000 7fe66e000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006bbf0 0x7fe65006e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.501+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe6700a9920 con 0x7fe6780c4760 2024-01-16T15:08:18.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.501+0000 7fe675e00640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006bbf0 0x7fe65006e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.502+0000 7fe675e00640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006bbf0 0x7fe65006e0b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe668001050 tx=0x7fe668009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:18.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.505+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe670075ff0 con 0x7fe6780c4760 2024-01-16T15:08:18.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.731+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe67806b160 con 0x7fe65006bbf0 2024-01-16T15:08:18.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.735+0000 7fe66e000640 1 -- 172.21.15.53:0/1748402975 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe67806b160 con 0x7fe65006bbf0 2024-01-16T15:08:18.713 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:08:18.713 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:08:18.713 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:08:18.713 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:08:18.713 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:08:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.737+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006bbf0 msgr2=0x7fe65006e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.737+0000 7fe67d000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006bbf0 0x7fe65006e0b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fe668001050 tx=0x7fe668009040 comp rx=0 tx=0).stop 2024-01-16T15:08:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 msgr2=0x7fe6780c4c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 0x7fe6780c4c10 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fe670005040 tx=0x7fe6700014a0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 shutdown_connections 2024-01-16T15:08:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe65006bbf0 0x7fe65006e0b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6780c4760 0x7fe6780c4c10 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6781f72c0 0x7fe678136000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 --2- 172.21.15.53:0/1748402975 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe658004d00 0x7fe678159d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 >> 172.21.15.53:0/1748402975 conn(0x7fe67806a800 msgr2=0x7fe6780ac940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 shutdown_connections 2024-01-16T15:08:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.738+0000 7fe67d000640 1 -- 172.21.15.53:0/1748402975 wait complete. 2024-01-16T15:08:18.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:08:18.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.984+0000 7f3c72e00640 1 Processor -- start 2024-01-16T15:08:18.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.985+0000 7f3c72e00640 1 -- start start 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.986+0000 7f3c72e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 0x7f3c6c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.986+0000 7f3c72e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3c6c0cfe70 0x7f3c6c161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.986+0000 7f3c72e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0d0a10 con 0x7f3c6c0cf9f0 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.986+0000 7f3c72e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c0d1f50 con 0x7f3c6c0cfe70 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.986+0000 7f3c72e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0cf080 con 0x7f3c6c160920 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.986+0000 7f3c71000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3c6c0cf9f0 0x7f3c6c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:32906/0 (socket says 172.21.15.53:32906) 2024-01-16T15:08:18.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c71000640 1 -- 172.21.15.53:0/1504317118 learned_addr learned my addr 172.21.15.53:0/1504317118 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c71a00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 0x7f3c6c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c72400640 1 --2- 172.21.15.53:0/1504317118 >> v2:172.21.15.146:3300/0 conn(0x7f3c6c0cfe70 0x7f3c6c161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403706314 0 0) 0x7f3c6c0d0a10 con 0x7f3c6c0cf9f0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c4c003520 con 0x7f3c6c0cf9f0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1710600462 0 0) 0x7f3c4c003520 con 0x7f3c6c0cf9f0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 msgr2=0x7f3c6c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 0x7f3c6c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 >> v2:172.21.15.146:3300/0 conn(0x7f3c6c0cfe70 msgr2=0x7f3c6c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 --2- 172.21.15.53:0/1504317118 >> v2:172.21.15.146:3300/0 conn(0x7f3c6c0cfe70 0x7f3c6c161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0c6be0 con 0x7f3c6c0cf9f0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.987+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3c54001670 con 0x7f3c6c0cf9f0 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 >> v1:172.21.15.53:6789/0 conn(0x7f3c6c0cf9f0 legacy=0x7f3c6c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c4c003970 0x7f3c4c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c0047c0 0x7f3c4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 0x7f3c4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0d1f50 con 0x7f3c4c003970 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0cf080 con 0x7f3c4c0047c0 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c4c00b980 con 0x7f3c6c160920 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c71a00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c4c003970 0x7f3c4c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c71a00640 1 -- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 msgr2=0x7f3c4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c71a00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c160920 0x7f3c4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c71a00640 1 -- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c0047c0 msgr2=0x7f3c4c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c71a00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c0047c0 0x7f3c4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.988+0000 7f3c71a00640 1 -- 172.21.15.53:0/1504317118 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c003520 con 0x7f3c4c003970 2024-01-16T15:08:18.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.989+0000 7f3c71a00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c4c003970 0x7f3c4c003d90 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009fc0 tx=0x7f3c4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1ccce6492d4d04d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:18.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c02cc40 con 0x7f3c4c003970 2024-01-16T15:08:18.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5c00f040 con 0x7f3c4c003970 2024-01-16T15:08:18.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c6be00640 1 -- 172.21.15.53:0/1504317118 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5c034440 con 0x7f3c4c003970 2024-01-16T15:08:18.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72e00640 1 -- 172.21.15.53:0/1504317118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c4c003970 msgr2=0x7f3c4c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72e00640 1 --2- 172.21.15.53:0/1504317118 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c4c003970 0x7f3c4c003d90 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c009fc0 tx=0x7f3c4c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72400640 1 -- 172.21.15.53:0/1504317118 reap_dead start 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72e00640 1 -- 172.21.15.53:0/1504317118 shutdown_connections 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72e00640 1 -- 172.21.15.53:0/1504317118 >> 172.21.15.53:0/1504317118 conn(0x7f3c6c0d3090 msgr2=0x7f3c6c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72e00640 1 -- 172.21.15.53:0/1504317118 shutdown_connections 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.990+0000 7f3c72e00640 1 -- 172.21.15.53:0/1504317118 wait complete. 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.992+0000 7f3c72e00640 1 Processor -- start 2024-01-16T15:08:18.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.993+0000 7f3c72e00640 1 -- start start 2024-01-16T15:08:18.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.993+0000 7f3c72e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c003970 0x7f3c6c0b87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.993+0000 7f3c72e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c0b71a0 0x7f3c6c1511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 0x7f3c6c0c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b8390 con 0x7f3c6c1540d0 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b7710 con 0x7f3c4c003970 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b79f0 con 0x7f3c6c0b71a0 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 0x7f3c6c0c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 0x7f3c6c0c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46486/0 (socket says 172.21.15.53:46486) 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 -- 172.21.15.53:0/683405050 learned_addr learned my addr 172.21.15.53:0/683405050 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c71000640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c0b71a0 0x7f3c6c1511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c71a00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c003970 0x7f3c6c0b87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 -- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c0b71a0 msgr2=0x7f3c6c1511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c0b71a0 0x7f3c6c1511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 -- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c003970 msgr2=0x7f3c6c0b87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c003970 0x7f3c6c0b87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c72400640 1 -- 172.21.15.53:0/683405050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0b7cd0 con 0x7f3c6c1540d0 2024-01-16T15:08:18.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.994+0000 7f3c71000640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c0b71a0 0x7f3c6c1511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:18.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.995+0000 7f3c71a00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c003970 0x7f3c6c0b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:08:18.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.995+0000 7f3c72400640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 0x7f3c6c0c8f20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c08c060 tx=0x7f3c6400ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:18.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.995+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400c680 con 0x7f3c6c1540d0 2024-01-16T15:08:18.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.996+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6400c7e0 con 0x7f3c6c1540d0 2024-01-16T15:08:18.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.996+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6400cb90 con 0x7f3c6c1540d0 2024-01-16T15:08:18.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.996+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c0b7fb0 con 0x7f3c6c1540d0 2024-01-16T15:08:18.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.996+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c154730 con 0x7f3c6c1540d0 2024-01-16T15:08:18.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.997+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c067450 con 0x7f3c6c1540d0 2024-01-16T15:08:18.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.998+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f3c6400ccf0 con 0x7f3c6c1540d0 2024-01-16T15:08:18.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.999+0000 7f3c69600640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3c4406bbc0 0x7f3c4406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:18.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.999+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3c6408a680 con 0x7f3c6c1540d0 2024-01-16T15:08:18.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:18.999+0000 7f3c71a00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3c4406bbc0 0x7f3c4406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:18.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.001+0000 7f3c71a00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3c4406bbc0 0x7f3c4406e080 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c036000 tx=0x7f3c5c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:18.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.003+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3c640543c0 con 0x7f3c6c1540d0 2024-01-16T15:08:19.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.194+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f3c6c08c7e0 con 0x7f3c6c1540d0 2024-01-16T15:08:19.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.195+0000 7f3c69600640 1 -- 172.21.15.53:0/683405050 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v26)=0 v26) v1 ==== 122+0+0 (secure 0 0 0) 0x7f3c640579e0 con 0x7f3c6c1540d0 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.197+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3c4406bbc0 msgr2=0x7f3c4406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.197+0000 7f3c72e00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3c4406bbc0 0x7f3c4406e080 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f3c5c036000 tx=0x7f3c5c02e000 comp rx=0 tx=0).stop 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.197+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 msgr2=0x7f3c6c0c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.197+0000 7f3c72e00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 0x7f3c6c0c8f20 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c08c060 tx=0x7f3c6400ec00 comp rx=0 tx=0).stop 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 shutdown_connections 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3c4406bbc0 0x7f3c4406e080 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3c6c1540d0 0x7f3c6c0c8f20 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3c6c0b71a0 0x7f3c6c1511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 --2- 172.21.15.53:0/683405050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3c4c003970 0x7f3c6c0b87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 >> 172.21.15.53:0/683405050 conn(0x7f3c6c0d3090 msgr2=0x7f3c6c160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:19.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 shutdown_connections 2024-01-16T15:08:19.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.198+0000 7f3c72e00640 1 -- 172.21.15.53:0/683405050 wait complete. 2024-01-16T15:08:19.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:08:19.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.433+0000 7f389c000640 1 Processor -- start 2024-01-16T15:08:19.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.434+0000 7f389c000640 1 -- start start 2024-01-16T15:08:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.435+0000 7f389c000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f389413df80 0x7f389415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.435+0000 7f389c000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389406c470 con 0x7f38940cf9a0 2024-01-16T15:08:19.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f389c000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389406c750 con 0x7f389413e360 2024-01-16T15:08:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f389c000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f389406ca30 con 0x7f389413df80 2024-01-16T15:08:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f389a000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f38940cf9a0 0x7f38940cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:32918/0 (socket says 172.21.15.53:32918) 2024-01-16T15:08:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f389a000640 1 -- 172.21.15.53:0/830491313 learned_addr learned my addr 172.21.15.53:0/830491313 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f3899600640 1 --2- 172.21.15.53:0/830491313 >> v2:172.21.15.146:3300/0 conn(0x7f389413df80 0x7f389415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2189463332 0 0) 0x7f389406c470 con 0x7f38940cf9a0 2024-01-16T15:08:19.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.436+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3874003520 con 0x7f38940cf9a0 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 425296831 0 0) 0x7f3874003520 con 0x7f38940cf9a0 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 >> v2:172.21.15.146:3300/0 conn(0x7f389413df80 msgr2=0x7f389415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 --2- 172.21.15.53:0/830491313 >> v2:172.21.15.146:3300/0 conn(0x7f389413df80 0x7f389415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 >> v1:172.21.15.146:6789/0 conn(0x7f389413e360 legacy=0x7f38940cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f389406cd10 con 0x7f38940cf9a0 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f388c0019f0 con 0x7f38940cf9a0 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 >> v1:172.21.15.53:6789/0 conn(0x7f38940cf9a0 legacy=0x7f38940cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3874003970 0x7f3874003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38740047c0 0x7f3874004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3874005180 0x7f387400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389406ca30 con 0x7f3874003970 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387400c020 con 0x7f38740047c0 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387400c1e0 con 0x7f3874005180 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.437+0000 7f3899600640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3874003970 0x7f3874003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f3898c00640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38740047c0 0x7f3874004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f3899600640 1 -- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3874005180 msgr2=0x7f387400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f3899600640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3874005180 0x7f387400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f3899600640 1 -- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38740047c0 msgr2=0x7f3874004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f3899600640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38740047c0 0x7f3874004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f389a000640 1 -- 172.21.15.53:0/830491313 reap_dead start 2024-01-16T15:08:19.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.438+0000 7f3899600640 1 -- 172.21.15.53:0/830491313 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3874003520 con 0x7f3874003970 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.439+0000 7f3899600640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3874003970 0x7f3874003d90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3884009ac0 tx=0x7f38740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f1cda9e8263b03 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.439+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388402c6b0 con 0x7f3874003970 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.439+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388402c810 con 0x7f3874003970 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.439+0000 7f3893e00640 1 -- 172.21.15.53:0/830491313 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388402cba0 con 0x7f3874003970 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 -- 172.21.15.53:0/830491313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3874003970 msgr2=0x7f3874003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3874003970 0x7f3874003d90 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f3884009ac0 tx=0x7f38740057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 -- 172.21.15.53:0/830491313 shutdown_connections 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 --2- 172.21.15.53:0/830491313 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3874003970 0x7f3874003d90 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 -- 172.21.15.53:0/830491313 >> 172.21.15.53:0/830491313 conn(0x7f389406a800 msgr2=0x7f389406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 -- 172.21.15.53:0/830491313 shutdown_connections 2024-01-16T15:08:19.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.440+0000 7f389c000640 1 -- 172.21.15.53:0/830491313 wait complete. 2024-01-16T15:08:19.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.442+0000 7f389c000640 1 Processor -- start 2024-01-16T15:08:19.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.442+0000 7f389c000640 1 -- start start 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.443+0000 7f389c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874003970 0x7f38940b1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.443+0000 7f389c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3894071fd0 0x7f389414f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.443+0000 7f389c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 0x7f38940af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.443+0000 7f389c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38941561a0 con 0x7f38940702c0 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3894155e20 con 0x7f3874003970 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3894070bc0 con 0x7f3894071fd0 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 0x7f38940af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 0x7f38940af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46504/0 (socket says 172.21.15.53:46504) 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 -- 172.21.15.53:0/3989979429 learned_addr learned my addr 172.21.15.53:0/3989979429 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:19.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f3898c00640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3894071fd0 0x7f389414f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f3899600640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874003970 0x7f38940b1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 -- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3894071fd0 msgr2=0x7f389414f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3894071fd0 0x7f389414f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 -- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874003970 msgr2=0x7f38940b1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.444+0000 7f389a000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874003970 0x7f38940b1e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.445+0000 7f389a000640 1 -- 172.21.15.53:0/3989979429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3894070ea0 con 0x7f38940702c0 2024-01-16T15:08:19.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.445+0000 7f389a000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 0x7f38940af070 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f38940cfea0 tx=0x7f388c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.445+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00cea0 con 0x7f38940702c0 2024-01-16T15:08:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.445+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f388c00a610 con 0x7f38940702c0 2024-01-16T15:08:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.445+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00a9a0 con 0x7f38940702c0 2024-01-16T15:08:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.445+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3894071180 con 0x7f38940702c0 2024-01-16T15:08:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.446+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389414f670 con 0x7f38940702c0 2024-01-16T15:08:19.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.447+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3894071370 con 0x7f38940702c0 2024-01-16T15:08:19.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.448+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f388c00ab00 con 0x7f38940702c0 2024-01-16T15:08:19.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.449+0000 7f3891600640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f386c06bbf0 0x7f386c06e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:19.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.449+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f388c087890 con 0x7f38940702c0 2024-01-16T15:08:19.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.450+0000 7f3899600640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f386c06bbf0 0x7f386c06e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:19.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.451+0000 7f3899600640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f386c06bbf0 0x7f386c06e0b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3884036000 tx=0x7f3884030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:19.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.454+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f388c0541b0 con 0x7f38940702c0 2024-01-16T15:08:19.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.679+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f38940cf290 con 0x7f386c06bbf0 2024-01-16T15:08:19.670 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:08:19.692+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:19.670 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:08:19.692+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:19.670 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:08:19.692+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:19.670 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:08:19.692+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:19.670 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:08:19.692+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:08:19.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.768+0000 7f3891600640 1 -- 172.21.15.53:0/3989979429 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38840040c0 con 0x7f386c06bbf0 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f386c06bbf0 msgr2=0x7f386c06e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f386c06bbf0 0x7f386c06e0b0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3884036000 tx=0x7f3884030000 comp rx=0 tx=0).stop 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 msgr2=0x7f38940af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 0x7f38940af070 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f38940cfea0 tx=0x7f388c00cc50 comp rx=0 tx=0).stop 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 shutdown_connections 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f386c06bbf0 0x7f386c06e0b0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f38940702c0 0x7f38940af070 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3894071fd0 0x7f389414f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 --2- 172.21.15.53:0/3989979429 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3874003970 0x7f38940b1e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:19.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 >> 172.21.15.53:0/3989979429 conn(0x7f389406a800 msgr2=0x7f38940cd840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:19.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 shutdown_connections 2024-01-16T15:08:19.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:19.771+0000 7f389c000640 1 -- 172.21.15.53:0/3989979429 wait complete. 2024-01-16T15:08:19.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:08:20.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.025+0000 7f798b200640 1 Processor -- start 2024-01-16T15:08:20.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.025+0000 7f798b200640 1 -- start start 2024-01-16T15:08:20.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.027+0000 7f798b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 0x7f798c0b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.027+0000 7f798b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0b1e30 con 0x7f798c1505b0 2024-01-16T15:08:20.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.027+0000 7f798b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f798c0b20a0 con 0x7f798c15a690 2024-01-16T15:08:20.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.027+0000 7f798b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0b2380 con 0x7f798c06d6d0 2024-01-16T15:08:20.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.027+0000 7f7989e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f798c1505b0 0x7f798c06cfc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:32928/0 (socket says 172.21.15.53:32928) 2024-01-16T15:08:20.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.027+0000 7f7989e00640 1 -- 172.21.15.53:0/4150944646 learned_addr learned my addr 172.21.15.53:0/4150944646 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7989400640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 0x7f798c0b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678634410 0 0) 0x7f798c0b1e30 con 0x7f798c1505b0 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f796c003520 con 0x7f798c1505b0 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1080405012 0 0) 0x7f796c003520 con 0x7f798c1505b0 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 msgr2=0x7f798c0b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 0x7f798c0b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 >> v1:172.21.15.146:6789/0 conn(0x7f798c15a690 legacy=0x7f798c0ccc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c0b2660 con 0x7f798c1505b0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7974001670 con 0x7f798c1505b0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 >> v1:172.21.15.53:6789/0 conn(0x7f798c1505b0 legacy=0x7f798c06cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f796c0039a0 0x7f796c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.028+0000 7f7988a00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f796c0047c0 0x7f796c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7988a00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 0x7f796c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0b2380 con 0x7f796c0039a0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f796c00ba30 con 0x7f796c0047c0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f796c00bbf0 con 0x7f798c06d6d0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989e00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f796c0039a0 0x7f796c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989400640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f796c0047c0 0x7f796c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989e00640 1 -- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 msgr2=0x7f796c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:20.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989e00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f798c06d6d0 0x7f796c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989e00640 1 -- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f796c0047c0 msgr2=0x7f796c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:08:20.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989e00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f796c0047c0 0x7f796c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.029+0000 7f7989e00640 1 -- 172.21.15.53:0/4150944646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f796c003520 con 0x7f796c0039a0 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.030+0000 7f7989e00640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f796c0039a0 0x7f796c003dc0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7974007ca0 tx=0x7f796c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8e4174e09f94f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797402ec80 con 0x7f796c0039a0 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797402ede0 con 0x7f796c0039a0 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f7988a00640 1 -- 172.21.15.53:0/4150944646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7974005af0 con 0x7f796c0039a0 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f798b200640 1 -- 172.21.15.53:0/4150944646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f796c0039a0 msgr2=0x7f796c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f798b200640 1 --2- 172.21.15.53:0/4150944646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f796c0039a0 0x7f796c003dc0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f7974007ca0 tx=0x7f796c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f798a800640 1 -- 172.21.15.53:0/4150944646 reap_dead start 2024-01-16T15:08:20.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f798b200640 1 -- 172.21.15.53:0/4150944646 shutdown_connections 2024-01-16T15:08:20.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.031+0000 7f798b200640 1 -- 172.21.15.53:0/4150944646 >> 172.21.15.53:0/4150944646 conn(0x7f798c06a7d0 msgr2=0x7f798c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:20.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.032+0000 7f798b200640 1 -- 172.21.15.53:0/4150944646 shutdown_connections 2024-01-16T15:08:20.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.032+0000 7f798b200640 1 -- 172.21.15.53:0/4150944646 wait complete. 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.033+0000 7f798b200640 1 Processor -- start 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.034+0000 7f798b200640 1 -- start start 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.034+0000 7f798b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f796c0039a0 0x7f798c15b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.034+0000 7f798b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 0x7f798c0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.034+0000 7f798b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f798c0b13b0 0x7f798c14a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f798b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c13b420 con 0x7f798c15b7b0 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f798b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798c15bc30 con 0x7f798c0b13b0 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f798b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f798c0b1810 con 0x7f796c0039a0 2024-01-16T15:08:20.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 0x7f798c0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 0x7f798c0b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46530/0 (socket says 172.21.15.53:46530) 2024-01-16T15:08:20.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 -- 172.21.15.53:0/1803489230 learned_addr learned my addr 172.21.15.53:0/1803489230 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f798a800640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f798c0b13b0 0x7f798c14a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 -- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f796c0039a0 msgr2=0x7f798c15b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989e00640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f796c0039a0 0x7f798c15b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f796c0039a0 0x7f798c15b270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 -- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f798c0b13b0 msgr2=0x7f798c14a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f798c0b13b0 0x7f798c14a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.035+0000 7f7989400640 1 -- 172.21.15.53:0/1803489230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f798c14b0d0 con 0x7f798c15b7b0 2024-01-16T15:08:20.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.036+0000 7f7989400640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 0x7f798c0b0e70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f797c007e60 tx=0x7f797c010580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:20.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.036+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c00f040 con 0x7f798c15b7b0 2024-01-16T15:08:20.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.036+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797c00a2d0 con 0x7f798c15b7b0 2024-01-16T15:08:20.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.036+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798c14b3b0 con 0x7f798c15b7b0 2024-01-16T15:08:20.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.036+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c015670 con 0x7f798c15b7b0 2024-01-16T15:08:20.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.036+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798c13b630 con 0x7f798c15b7b0 2024-01-16T15:08:20.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.038+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798c13b7f0 con 0x7f798c15b7b0 2024-01-16T15:08:20.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.038+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 21) v1 ==== 90355+0+0 (secure 0 0 0) 0x7f797c0157d0 con 0x7f798c15b7b0 2024-01-16T15:08:20.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.039+0000 7f7982000640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f796406bbc0 0x7f796406e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:08:20.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.040+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f797c089140 con 0x7f798c15b7b0 2024-01-16T15:08:20.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.040+0000 7f7989e00640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f796406bbc0 0x7f796406e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:08:20.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.041+0000 7f7989e00640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f796406bbc0 0x7f796406e080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7974036000 tx=0x7f7974009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:08:20.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.044+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f797c055a00 con 0x7f798c15b7b0 2024-01-16T15:08:20.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.269+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f798c14b8d0 con 0x7f796406bbc0 2024-01-16T15:08:20.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.281+0000 7f7982000640 1 -- 172.21.15.53:0/1803489230 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f798c14b8d0 con 0x7f796406bbc0 2024-01-16T15:08:20.259 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f796406bbc0 msgr2=0x7f796406e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f796406bbc0 0x7f796406e080 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f7974036000 tx=0x7f7974009030 comp rx=0 tx=0).stop 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 msgr2=0x7f798c0b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 0x7f798c0b0e70 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f797c007e60 tx=0x7f797c010580 comp rx=0 tx=0).stop 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 shutdown_connections 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f796406bbc0 0x7f796406e080 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f798c0b13b0 0x7f798c14a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f798c15b7b0 0x7f798c0b0e70 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 --2- 172.21.15.53:0/1803489230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f796c0039a0 0x7f798c15b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:08:20.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.283+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 >> 172.21.15.53:0/1803489230 conn(0x7f798c06a7d0 msgr2=0x7f798c159bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:08:20.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.284+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 shutdown_connections 2024-01-16T15:08:20.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:08:20.284+0000 7f798b200640 1 -- 172.21.15.53:0/1803489230 wait complete. 2024-01-16T15:08:20.279 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea 2024-01-16T15:08:20.279 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:08:20.279 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:08:20.330 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:20.330 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:08:22.393 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:22.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:36.353 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:08:36.353 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:08:36.353 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.9399 s, 67.4 MB/s 2024-01-16T15:08:36.355 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:08:36.355 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:08:36.418 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:36.418 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:08:51.255 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:08:51.255 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:08:51.255 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.7672 s, 72.7 MB/s 2024-01-16T15:08:51.257 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:08:51.257 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:08:51.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:08:51.313 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:08:52.545 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:08:52.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:05.365 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:09:05.365 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:09:05.365 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.9945 s, 76.7 MB/s 2024-01-16T15:09:05.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 10 2024-01-16T15:09:05.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.614+0000 7f0601000640 1 Processor -- start 2024-01-16T15:09:05.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.614+0000 7f0601000640 1 -- start start 2024-01-16T15:09:05.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.616+0000 7f0601000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f05fc13df80 0x7f05fc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.616+0000 7f0601000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05fc13e360 0x7f05fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.616+0000 7f0601000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc06c470 con 0x7f05fc15f9c0 2024-01-16T15:09:05.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.616+0000 7f0601000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc06c750 con 0x7f05fc13e360 2024-01-16T15:09:05.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.616+0000 7f0601000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc06ca30 con 0x7f05fc13df80 2024-01-16T15:09:05.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05fc13e360 0x7f05fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05fc13e360 0x7f05fc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36066/0 (socket says 172.21.15.53:36066) 2024-01-16T15:09:05.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05fa800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f05fc15f9c0 0x7f05fc1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56480/0 (socket says 172.21.15.53:56480) 2024-01-16T15:09:05.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 -- 172.21.15.53:0/1119442951 learned_addr learned my addr 172.21.15.53:0/1119442951 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:05.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f9e00640 1 --2- 172.21.15.53:0/1119442951 >> v2:172.21.15.146:3300/0 conn(0x7f05fc13df80 0x7f05fc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 -- 172.21.15.53:0/1119442951 >> v2:172.21.15.146:3300/0 conn(0x7f05fc13df80 msgr2=0x7f05fc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:05.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 --2- 172.21.15.53:0/1119442951 >> v2:172.21.15.146:3300/0 conn(0x7f05fc13df80 0x7f05fc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 -- 172.21.15.53:0/1119442951 >> v1:172.21.15.53:6789/0 conn(0x7f05fc15f9c0 legacy=0x7f05fc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.617+0000 7f05f3e00640 1 -- 172.21.15.53:0/1119442951 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc06cd10 con 0x7f05fc13e360 2024-01-16T15:09:05.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.618+0000 7f05f3e00640 1 --2- 172.21.15.53:0/1119442951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13e360 0x7f05fc0cf290 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f05fc072770 tx=0x7f05e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9508e0909f7d3faf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.619+0000 7f05f9400640 1 -- 172.21.15.53:0/1119442951 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4030860 con 0x7f05fc13e360 2024-01-16T15:09:05.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.619+0000 7f05f9400640 1 -- 172.21.15.53:0/1119442951 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e40309c0 con 0x7f05fc13e360 2024-01-16T15:09:05.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.619+0000 7f05f9400640 1 -- 172.21.15.53:0/1119442951 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4030d50 con 0x7f05fc13e360 2024-01-16T15:09:05.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.619+0000 7f0601000640 1 -- 172.21.15.53:0/1119442951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13e360 msgr2=0x7f05fc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.619+0000 7f0601000640 1 --2- 172.21.15.53:0/1119442951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13e360 0x7f05fc0cf290 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7f05fc072770 tx=0x7f05e402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.620+0000 7f0601000640 1 -- 172.21.15.53:0/1119442951 shutdown_connections 2024-01-16T15:09:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.620+0000 7f0601000640 1 --2- 172.21.15.53:0/1119442951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13e360 0x7f05fc0cf290 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.620+0000 7f0601000640 1 --2- 172.21.15.53:0/1119442951 >> v2:172.21.15.146:3300/0 conn(0x7f05fc13df80 0x7f05fc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.620+0000 7f0601000640 1 -- 172.21.15.53:0/1119442951 >> 172.21.15.53:0/1119442951 conn(0x7f05fc06a800 msgr2=0x7f05fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.620+0000 7f0601000640 1 -- 172.21.15.53:0/1119442951 shutdown_connections 2024-01-16T15:09:05.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.620+0000 7f0601000640 1 -- 172.21.15.53:0/1119442951 wait complete. 2024-01-16T15:09:05.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.621+0000 7f0601000640 1 Processor -- start 2024-01-16T15:09:05.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.622+0000 7f0601000640 1 -- start start 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f0601000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 0x7f05fc14dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f0601000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc13e360 0x7f05fc1f2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f0601000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15f9c0 0x7f05fc1f6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f0601000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1f29d0 con 0x7f05fc13df80 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f0601000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc151570 con 0x7f05fc15f9c0 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f0601000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1f7020 con 0x7f05fc13e360 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f05f9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 0x7f05fc14dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f05f9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 0x7f05fc14dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36070/0 (socket says 172.21.15.53:36070) 2024-01-16T15:09:05.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f05f9e00640 1 -- 172.21.15.53:0/243225239 learned_addr learned my addr 172.21.15.53:0/243225239 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:05.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f05f9e00640 1 -- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc13e360 msgr2=0x7f05fc1f2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f05f3e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc13e360 0x7f05fc1f2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.623+0000 7f05fa800640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15f9c0 0x7f05fc1f6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f9e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc13e360 0x7f05fc1f2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f9e00640 1 -- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15f9c0 msgr2=0x7f05fc1f6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f9e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15f9c0 0x7f05fc1f6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f9e00640 1 -- 172.21.15.53:0/243225239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e40094d0 con 0x7f05fc13df80 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f3e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc13e360 0x7f05fc1f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05fa800640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15f9c0 0x7f05fc1f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f9e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 0x7f05fc14dae0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00d620 tx=0x7f05ec00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec00b620 con 0x7f05fc13df80 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05ec00b780 con 0x7f05fc13df80 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05ec00bb30 con 0x7f05fc13df80 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.624+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc14eab0 con 0x7f05fc13df80 2024-01-16T15:09:05.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.625+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc14e7d0 con 0x7f05fc13df80 2024-01-16T15:09:05.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.626+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc0675d0 con 0x7f05fc13df80 2024-01-16T15:09:05.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.627+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f05ec00bc90 con 0x7f05fc13df80 2024-01-16T15:09:05.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.628+0000 7f05f2000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406bc50 0x7f05d406e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:05.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.628+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05ec088bb0 con 0x7f05fc13df80 2024-01-16T15:09:05.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.629+0000 7f05f3e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406bc50 0x7f05d406e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:05.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.630+0000 7f05f3e00640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406bc50 0x7f05d406e110 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f05e4036000 tx=0x7f05e403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:05.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.634+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05ec055270 con 0x7f05fc13df80 2024-01-16T15:09:05.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.835+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f05fc14d8a0 con 0x7f05fc13df80 2024-01-16T15:09:05.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.843+0000 7f05f2000640 1 -- 172.21.15.53:0/243225239 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v27)=0 v27) v1 ==== 136+0+0 (secure 0 0 0) 0x7f05ec058890 con 0x7f05fc13df80 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406bc50 msgr2=0x7f05d406e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406bc50 0x7f05d406e110 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f05e4036000 tx=0x7f05e403b040 comp rx=0 tx=0).stop 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 msgr2=0x7f05fc14dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 0x7f05fc14dae0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f05ec00d620 tx=0x7f05ec00dae0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 shutdown_connections 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406bc50 0x7f05d406e110 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15f9c0 0x7f05fc1f6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc13e360 0x7f05fc1f2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 --2- 172.21.15.53:0/243225239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc13df80 0x7f05fc14dae0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:05.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 >> 172.21.15.53:0/243225239 conn(0x7f05fc06a800 msgr2=0x7f05fc0affe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:05.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 shutdown_connections 2024-01-16T15:09:05.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:05.846+0000 7f0601000640 1 -- 172.21.15.53:0/243225239 wait complete. 2024-01-16T15:09:05.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:09:06.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.090+0000 7f7f30a00640 1 Processor -- start 2024-01-16T15:09:06.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.091+0000 7f7f30a00640 1 -- start start 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.092+0000 7f7f30a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7f2813df80 0x7f7f2815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.092+0000 7f7f30a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2813e360 0x7f7f280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.092+0000 7f7f30a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7f2815f9c0 0x7f7f280cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.092+0000 7f7f30a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2806c470 con 0x7f7f2813df80 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.092+0000 7f7f30a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f2806c750 con 0x7f7f2815f9c0 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.092+0000 7f7f30a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2806ca30 con 0x7f7f2813e360 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7f2813df80 0x7f7f2815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7f2813df80 0x7f7f2815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36080/0 (socket says 172.21.15.53:36080) 2024-01-16T15:09:06.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 -- 172.21.15.53:0/567303069 learned_addr learned my addr 172.21.15.53:0/567303069 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2ea00640 1 --2- 172.21.15.53:0/567303069 >> v2:172.21.15.146:3300/0 conn(0x7f7f2815f9c0 0x7f7f280cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2d600640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2813e360 0x7f7f280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 -- 172.21.15.53:0/567303069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2813e360 msgr2=0x7f7f280cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2813e360 0x7f7f280cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 -- 172.21.15.53:0/567303069 >> v2:172.21.15.146:3300/0 conn(0x7f7f2815f9c0 msgr2=0x7f7f280cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 --2- 172.21.15.53:0/567303069 >> v2:172.21.15.146:3300/0 conn(0x7f7f2815f9c0 0x7f7f280cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2e000640 1 -- 172.21.15.53:0/567303069 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f2806cd10 con 0x7f7f2813df80 2024-01-16T15:09:06.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.093+0000 7f7f2d600640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2813e360 0x7f7f280cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:06.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.095+0000 7f7f2e000640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f2815bee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7f2806e6d0 tx=0x7f7f1802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3cd3f8a20db5539 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.095+0000 7f7f2cc00640 1 -- 172.21.15.53:0/567303069 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1802ca80 con 0x7f7f2813df80 2024-01-16T15:09:06.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.095+0000 7f7f2cc00640 1 -- 172.21.15.53:0/567303069 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1802cbe0 con 0x7f7f2813df80 2024-01-16T15:09:06.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.095+0000 7f7f2cc00640 1 -- 172.21.15.53:0/567303069 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f180343f0 con 0x7f7f2813df80 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 -- 172.21.15.53:0/567303069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 msgr2=0x7f7f2815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f2815bee0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f7f2806e6d0 tx=0x7f7f1802bee0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 -- 172.21.15.53:0/567303069 shutdown_connections 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 --2- 172.21.15.53:0/567303069 >> v2:172.21.15.146:3300/0 conn(0x7f7f2815f9c0 0x7f7f280cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2813e360 0x7f7f280cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 --2- 172.21.15.53:0/567303069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f2815bee0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 -- 172.21.15.53:0/567303069 >> 172.21.15.53:0/567303069 conn(0x7f7f2806a800 msgr2=0x7f7f2806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 -- 172.21.15.53:0/567303069 shutdown_connections 2024-01-16T15:09:06.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.096+0000 7f7f30a00640 1 -- 172.21.15.53:0/567303069 wait complete. 2024-01-16T15:09:06.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.098+0000 7f7f30a00640 1 Processor -- start 2024-01-16T15:09:06.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.098+0000 7f7f30a00640 1 -- start start 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f30a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f280a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f30a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f2813e360 0x7f7f28071580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f30a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2815f9c0 0x7f7f280b3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f30a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f280a8a00 con 0x7f7f2813df80 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f30a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f281f31e0 con 0x7f7f2813e360 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f30a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f281f34c0 con 0x7f7f2815f9c0 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f2e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f280a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f2e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f280a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36086/0 (socket says 172.21.15.53:36086) 2024-01-16T15:09:06.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.099+0000 7f7f2e000640 1 -- 172.21.15.53:0/3603312095 learned_addr learned my addr 172.21.15.53:0/3603312095 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:06.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2d600640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f2813e360 0x7f7f28071580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2ea00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2815f9c0 0x7f7f280b3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2e000640 1 -- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2815f9c0 msgr2=0x7f7f280b3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2e000640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2815f9c0 0x7f7f280b3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2e000640 1 -- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f2813e360 msgr2=0x7f7f28071580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2e000640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f2813e360 0x7f7f28071580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2e000640 1 -- 172.21.15.53:0/3603312095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f18009b30 con 0x7f7f2813df80 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2d600640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f2813e360 0x7f7f28071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2ea00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2815f9c0 0x7f7f280b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f2e000640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f280a84c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7f18000f80 tx=0x7f7f18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18009510 con 0x7f7f2813df80 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1803e450 con 0x7f7f2813df80 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f1803e800 con 0x7f7f2813df80 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.100+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f281f37a0 con 0x7f7f2813df80 2024-01-16T15:09:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.101+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f28071ac0 con 0x7f7f2813df80 2024-01-16T15:09:06.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.102+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f28071d10 con 0x7f7f2813df80 2024-01-16T15:09:06.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.103+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f7f1802cd50 con 0x7f7f2813df80 2024-01-16T15:09:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.104+0000 7f7f26000640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f0006bca0 0x7f7f0006e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.104+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7f180ab230 con 0x7f7f2813df80 2024-01-16T15:09:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.104+0000 7f7f2d600640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f0006bca0 0x7f7f0006e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.106+0000 7f7f2d600640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f0006bca0 0x7f7f0006e160 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7f10006fd0 tx=0x7f7f10008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:06.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.108+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f180778f0 con 0x7f7f2813df80 2024-01-16T15:09:06.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.335+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f281408d0 con 0x7f7f0006bca0 2024-01-16T15:09:06.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.369+0000 7f7f26000640 1 -- 172.21.15.53:0/3603312095 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7f1000c040 con 0x7f7f0006bca0 2024-01-16T15:09:06.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.371+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f0006bca0 msgr2=0x7f7f0006e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.371+0000 7f7f30a00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f0006bca0 0x7f7f0006e160 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f7f10006fd0 tx=0x7f7f10008040 comp rx=0 tx=0).stop 2024-01-16T15:09:06.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.371+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 msgr2=0x7f7f280a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:06.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.371+0000 7f7f30a00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f280a84c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f7f18000f80 tx=0x7f7f18004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 shutdown_connections 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f0006bca0 0x7f7f0006e160 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f2815f9c0 0x7f7f280b3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f2813e360 0x7f7f28071580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 --2- 172.21.15.53:0/3603312095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f2813df80 0x7f7f280a84c0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 >> 172.21.15.53:0/3603312095 conn(0x7f7f2806a800 msgr2=0x7f7f2806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 shutdown_connections 2024-01-16T15:09:06.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:06.372+0000 7f7f30a00640 1 -- 172.21.15.53:0/3603312095 wait complete. 2024-01-16T15:09:06.372 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:09:07.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.023+0000 7f05cde00640 1 Processor -- start 2024-01-16T15:09:07.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.023+0000 7f05cde00640 1 -- start start 2024-01-16T15:09:07.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.025+0000 7f05cde00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f05c813fcb0 0x7f05c80cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.025+0000 7f05cde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05c80cdc20 0x7f05c80b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.025+0000 7f05cde00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05c80ce440 con 0x7f05c813cf10 2024-01-16T15:09:07.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.025+0000 7f05cde00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c80ce720 con 0x7f05c80cdc20 2024-01-16T15:09:07.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.025+0000 7f05cde00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05c80cea00 con 0x7f05c813fcb0 2024-01-16T15:09:07.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.025+0000 7f05cca00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f05c813cf10 0x7f05c813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56490/0 (socket says 172.21.15.53:56490) 2024-01-16T15:09:07.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05cca00640 1 -- 172.21.15.53:0/3757082772 learned_addr learned my addr 172.21.15.53:0/3757082772 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:07.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05cd400640 1 --2- 172.21.15.53:0/3757082772 >> v2:172.21.15.53:3300/0 conn(0x7f05c80cdc20 0x7f05c80b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05c7e00640 1 --2- 172.21.15.53:0/3757082772 >> v2:172.21.15.146:3300/0 conn(0x7f05c813fcb0 0x7f05c80cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05c7400640 1 -- 172.21.15.53:0/3757082772 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971557841 0 0) 0x7f05c80ce440 con 0x7f05c813cf10 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05c7400640 1 -- 172.21.15.53:0/3757082772 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05a8003520 con 0x7f05c813cf10 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05cd400640 1 -- 172.21.15.53:0/3757082772 >> v2:172.21.15.146:3300/0 conn(0x7f05c813fcb0 msgr2=0x7f05c80cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05cd400640 1 --2- 172.21.15.53:0/3757082772 >> v2:172.21.15.146:3300/0 conn(0x7f05c813fcb0 0x7f05c80cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05cd400640 1 -- 172.21.15.53:0/3757082772 >> v1:172.21.15.53:6789/0 conn(0x7f05c813cf10 legacy=0x7f05c813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.026+0000 7f05cd400640 1 -- 172.21.15.53:0/3757082772 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c80b1d90 con 0x7f05c80cdc20 2024-01-16T15:09:07.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.027+0000 7f05cd400640 1 --2- 172.21.15.53:0/3757082772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c80b2590 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f05c8157f90 tx=0x7f05c002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f895a0156c2f17d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.028+0000 7f05c7400640 1 -- 172.21.15.53:0/3757082772 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c002cc80 con 0x7f05c80cdc20 2024-01-16T15:09:07.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.028+0000 7f05c7400640 1 -- 172.21.15.53:0/3757082772 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05c002cde0 con 0x7f05c80cdc20 2024-01-16T15:09:07.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.028+0000 7f05c7400640 1 -- 172.21.15.53:0/3757082772 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05c0034660 con 0x7f05c80cdc20 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.028+0000 7f05cde00640 1 -- 172.21.15.53:0/3757082772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 msgr2=0x7f05c80b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.028+0000 7f05cde00640 1 --2- 172.21.15.53:0/3757082772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c80b2590 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f05c8157f90 tx=0x7f05c002bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.029+0000 7f05cde00640 1 -- 172.21.15.53:0/3757082772 shutdown_connections 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.029+0000 7f05cde00640 1 --2- 172.21.15.53:0/3757082772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c80b2590 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.029+0000 7f05cde00640 1 --2- 172.21.15.53:0/3757082772 >> v2:172.21.15.146:3300/0 conn(0x7f05c813fcb0 0x7f05c80cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.029+0000 7f05cde00640 1 -- 172.21.15.53:0/3757082772 >> 172.21.15.53:0/3757082772 conn(0x7f05c806a7d0 msgr2=0x7f05c806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.029+0000 7f05cde00640 1 -- 172.21.15.53:0/3757082772 shutdown_connections 2024-01-16T15:09:07.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.029+0000 7f05cde00640 1 -- 172.21.15.53:0/3757082772 wait complete. 2024-01-16T15:09:07.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.030+0000 7f05cde00640 1 Processor -- start 2024-01-16T15:09:07.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.031+0000 7f05cde00640 1 -- start start 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.031+0000 7f05cde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cde00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05c813cf10 0x7f05c81f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cde00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05c813fcb0 0x7f05c814aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cde00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c813abf0 con 0x7f05c80cdc20 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cde00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05c813aed0 con 0x7f05c813cf10 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cde00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05c813b1b0 con 0x7f05c813fcb0 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c813dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36102/0 (socket says 172.21.15.53:36102) 2024-01-16T15:09:07.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 -- 172.21.15.53:0/298092257 learned_addr learned my addr 172.21.15.53:0/298092257 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:07.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05c7e00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05c813cf10 0x7f05c81f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cd400640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05c813fcb0 0x7f05c814aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 -- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05c813fcb0 msgr2=0x7f05c814aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05c813fcb0 0x7f05c814aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 -- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05c813cf10 msgr2=0x7f05c81f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05c813cf10 0x7f05c81f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05cca00640 1 -- 172.21.15.53:0/298092257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05c0009a70 con 0x7f05c80cdc20 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.032+0000 7f05c7e00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05c813cf10 0x7f05c81f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.033+0000 7f05cca00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c813dd00 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f05b800c500 tx=0x7f05b800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.033+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8005ce0 con 0x7f05c80cdc20 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.033+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b8005e40 con 0x7f05c80cdc20 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.033+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05c813b490 con 0x7f05c80cdc20 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.033+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b8007330 con 0x7f05c80cdc20 2024-01-16T15:09:07.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.033+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05c814d810 con 0x7f05c80cdc20 2024-01-16T15:09:07.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.034+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05c813b9b0 con 0x7f05c80cdc20 2024-01-16T15:09:07.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.035+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f05b8007550 con 0x7f05c80cdc20 2024-01-16T15:09:07.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.036+0000 7f05c4c00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05a006bc50 0x7f05a006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:07.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.036+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05b808a700 con 0x7f05c80cdc20 2024-01-16T15:09:07.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.037+0000 7f05c7e00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05a006bc50 0x7f05a006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:07.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.038+0000 7f05c7e00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05a006bc50 0x7f05a006e110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f05c814dcf0 tx=0x7f05b0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:07.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.041+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05c813b9b0 con 0x7f05c80cdc20 2024-01-16T15:09:07.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.264+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f05c8159e40 con 0x7f05a006bc50 2024-01-16T15:09:07.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.276+0000 7f05c4c00640 1 -- 172.21.15.53:0/298092257 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f05c8159e40 con 0x7f05a006bc50 2024-01-16T15:09:07.254 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05a006bc50 msgr2=0x7f05a006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05a006bc50 0x7f05a006e110 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f05c814dcf0 tx=0x7f05b0008040 comp rx=0 tx=0).stop 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 msgr2=0x7f05c813dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c813dd00 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f05b800c500 tx=0x7f05b800c9c0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 shutdown_connections 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05a006bc50 0x7f05a006e110 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05c813fcb0 0x7f05c814aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05c813cf10 0x7f05c81f5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 --2- 172.21.15.53:0/298092257 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05c80cdc20 0x7f05c813dd00 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:07.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 >> 172.21.15.53:0/298092257 conn(0x7f05c806a7d0 msgr2=0x7f05c8159730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:07.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 shutdown_connections 2024-01-16T15:09:07.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:07.279+0000 7f05cde00640 1 -- 172.21.15.53:0/298092257 wait complete. 2024-01-16T15:09:07.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:07.276 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea) 2024-01-16T15:09:07.340 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea 2024-01-16T15:09:07.340 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:09:07.340 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:07.341 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:09:09.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:09:09.343 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea) 2024-01-16T15:09:09.414 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/72819b89-74fc-461a-bcaf-2e44009d56ea 2024-01-16T15:09:09.414 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:09:09.414 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:09.415 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:09:09.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54141 2024-01-16T15:09:09.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.674+0000 7f5494e00640 1 Processor -- start 2024-01-16T15:09:09.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.674+0000 7f5494e00640 1 -- start start 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f5494e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f549013df80 0x7f549015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f5494e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f54900cf9a0 0x7f54900cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f5494e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549006c470 con 0x7f549013e360 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f5494e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549006c750 con 0x7f549013df80 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f5494e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549006ca30 con 0x7f54900cf9a0 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f548de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f549013df80 0x7f549015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f548de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f549013df80 0x7f549015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36106/0 (socket says 172.21.15.53:36106) 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.676+0000 7f548de00640 1 -- 172.21.15.53:0/103249290 learned_addr learned my addr 172.21.15.53:0/103249290 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.677+0000 7f548e800640 1 --2- 172.21.15.53:0/103249290 >> v2:172.21.15.146:3300/0 conn(0x7f54900cf9a0 0x7f54900cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.677+0000 7f548de00640 1 -- 172.21.15.53:0/103249290 >> v2:172.21.15.146:3300/0 conn(0x7f54900cf9a0 msgr2=0x7f54900cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.677+0000 7f548de00640 1 --2- 172.21.15.53:0/103249290 >> v2:172.21.15.146:3300/0 conn(0x7f54900cf9a0 0x7f54900cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.677+0000 7f548de00640 1 -- 172.21.15.53:0/103249290 >> v1:172.21.15.146:6789/0 conn(0x7f549013e360 legacy=0x7f54900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:09.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.677+0000 7f548de00640 1 -- 172.21.15.53:0/103249290 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f549006cd10 con 0x7f549013df80 2024-01-16T15:09:09.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.678+0000 7f548de00640 1 --2- 172.21.15.53:0/103249290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f549015bee0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f54900ccb10 tx=0x7f548002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8574d1955b2ce17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.679+0000 7f548d400640 1 -- 172.21.15.53:0/103249290 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480031870 con 0x7f549013df80 2024-01-16T15:09:09.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.679+0000 7f548d400640 1 -- 172.21.15.53:0/103249290 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54800319d0 con 0x7f549013df80 2024-01-16T15:09:09.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.679+0000 7f548d400640 1 -- 172.21.15.53:0/103249290 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480031d60 con 0x7f549013df80 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 -- 172.21.15.53:0/103249290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 msgr2=0x7f549015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 --2- 172.21.15.53:0/103249290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f549015bee0 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f54900ccb10 tx=0x7f548002b910 comp rx=0 tx=0).stop 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 -- 172.21.15.53:0/103249290 shutdown_connections 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 --2- 172.21.15.53:0/103249290 >> v2:172.21.15.146:3300/0 conn(0x7f54900cf9a0 0x7f54900cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 --2- 172.21.15.53:0/103249290 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f549015bee0 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 -- 172.21.15.53:0/103249290 >> 172.21.15.53:0/103249290 conn(0x7f549006a800 msgr2=0x7f549006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 -- 172.21.15.53:0/103249290 shutdown_connections 2024-01-16T15:09:09.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.680+0000 7f5494e00640 1 -- 172.21.15.53:0/103249290 wait complete. 2024-01-16T15:09:09.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.682+0000 7f5494e00640 1 Processor -- start 2024-01-16T15:09:09.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.682+0000 7f5494e00640 1 -- start start 2024-01-16T15:09:09.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5494e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54900cf9a0 0x7f54901e49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5494e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f54901393b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5494e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f549013e360 0x7f54901f33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5494e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54901e4f10 con 0x7f549013df80 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5494e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549013b6e0 con 0x7f549013e360 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5494e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549013b9c0 con 0x7f54900cf9a0 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f54901393b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f54901393b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36108/0 (socket says 172.21.15.53:36108) 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 -- 172.21.15.53:0/459538385 learned_addr learned my addr 172.21.15.53:0/459538385 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f548de00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54900cf9a0 0x7f54901e49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f548e800640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f549013e360 0x7f54901f33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 -- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54900cf9a0 msgr2=0x7f54901e49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54900cf9a0 0x7f54901e49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 -- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f549013e360 msgr2=0x7f54901f33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:09.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f549013e360 0x7f54901f33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:09.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f5487e00640 1 -- 172.21.15.53:0/459538385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800094d0 con 0x7f549013df80 2024-01-16T15:09:09.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f548e800640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f549013e360 0x7f54901f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:09.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.684+0000 7f548de00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54900cf9a0 0x7f54901e49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:09.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.685+0000 7f5487e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f54901393b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f547800c400 tx=0x7f547800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.685+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478011790 con 0x7f549013df80 2024-01-16T15:09:09.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.685+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5478011930 con 0x7f549013df80 2024-01-16T15:09:09.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.685+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478011ce0 con 0x7f549013df80 2024-01-16T15:09:09.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.685+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549013bca0 con 0x7f549013df80 2024-01-16T15:09:09.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.685+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549013c260 con 0x7f549013df80 2024-01-16T15:09:09.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.686+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54900cd120 con 0x7f549013df80 2024-01-16T15:09:09.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.688+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f547801c460 con 0x7f549013df80 2024-01-16T15:09:09.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.688+0000 7f5486000640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f546806bc50 0x7f546806e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:09.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.689+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5478088750 con 0x7f549013df80 2024-01-16T15:09:09.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.689+0000 7f548de00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f546806bc50 0x7f546806e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:09.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.690+0000 7f548de00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f546806bc50 0x7f546806e110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5480036000 tx=0x7f548003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:09.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.694+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f54780bde90 con 0x7f549013df80 2024-01-16T15:09:09.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:09.928+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54141", "target": ["mon-mgr", ""]}) v1 -- 0x7f549015f9c0 con 0x7f546806bc50 2024-01-16T15:09:10.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.087+0000 7f5486000640 1 -- 172.21.15.53:0/459538385 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5480031c40 con 0x7f546806bc50 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f546806bc50 msgr2=0x7f546806e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f546806bc50 0x7f546806e110 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5480036000 tx=0x7f548003c040 comp rx=0 tx=0).stop 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 msgr2=0x7f54901393b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f54901393b0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f547800c400 tx=0x7f547800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 shutdown_connections 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f546806bc50 0x7f546806e110 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f549013e360 0x7f54901f33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f549013df80 0x7f54901393b0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 --2- 172.21.15.53:0/459538385 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f54900cf9a0 0x7f54901e49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 >> 172.21.15.53:0/459538385 conn(0x7f549006a800 msgr2=0x7f54900af9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 shutdown_connections 2024-01-16T15:09:10.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.091+0000 7f5494e00640 1 -- 172.21.15.53:0/459538385 wait complete. 2024-01-16T15:09:10.075 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.097+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.075 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.097+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.075 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.097+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.075 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.097+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.075 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.097+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.091 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54142 2024-01-16T15:09:10.096 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.119+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.096 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.119+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.096 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.119+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.096 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.119+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.096 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:10.119+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.345+0000 7f5dddc00640 1 Processor -- start 2024-01-16T15:09:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.346+0000 7f5dddc00640 1 -- start start 2024-01-16T15:09:10.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.347+0000 7f5dddc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80ce8f0 0x7f5dd80ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.347+0000 7f5dddc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5dd80b2a00 0x7f5dd80b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.347+0000 7f5dddc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dd80d0de0 con 0x7f5dd8160f80 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.347+0000 7f5dddc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd80d10c0 con 0x7f5dd80b2a00 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.347+0000 7f5dddc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80d13a0 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.347+0000 7f5dd7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5dd80b2a00 0x7f5dd80b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5dd80b2a00 0x7f5dd80b2e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36110/0 (socket says 172.21.15.53:36110) 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd7400640 1 -- 172.21.15.53:0/503778059 learned_addr learned my addr 172.21.15.53:0/503778059 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd6000640 1 --2- 172.21.15.53:0/503778059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80ce8f0 0x7f5dd80ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd7400640 1 -- 172.21.15.53:0/503778059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80ce8f0 msgr2=0x7f5dd80ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd7400640 1 --2- 172.21.15.53:0/503778059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80ce8f0 0x7f5dd80ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd7400640 1 -- 172.21.15.53:0/503778059 >> v1:172.21.15.146:6789/0 conn(0x7f5dd8160f80 legacy=0x7f5dd8161360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.348+0000 7f5dd7400640 1 -- 172.21.15.53:0/503778059 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd80d1680 con 0x7f5dd80b2a00 2024-01-16T15:09:10.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.350+0000 7f5dd7400640 1 --2- 172.21.15.53:0/503778059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80b2a00 0x7f5dd80b2e60 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009870 tx=0x7f5dd002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c06e36040831be server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.350+0000 7f5dd5600640 1 -- 172.21.15.53:0/503778059 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0030840 con 0x7f5dd80b2a00 2024-01-16T15:09:10.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.350+0000 7f5dd5600640 1 -- 172.21.15.53:0/503778059 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dd00309a0 con 0x7f5dd80b2a00 2024-01-16T15:09:10.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.350+0000 7f5dd5600640 1 -- 172.21.15.53:0/503778059 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dd0030d30 con 0x7f5dd80b2a00 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 -- 172.21.15.53:0/503778059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80b2a00 msgr2=0x7f5dd80b2e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 --2- 172.21.15.53:0/503778059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80b2a00 0x7f5dd80b2e60 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f5dd0009870 tx=0x7f5dd002b980 comp rx=0 tx=0).stop 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 -- 172.21.15.53:0/503778059 shutdown_connections 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 --2- 172.21.15.53:0/503778059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80b2a00 0x7f5dd80b2e60 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 --2- 172.21.15.53:0/503778059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80ce8f0 0x7f5dd80ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 -- 172.21.15.53:0/503778059 >> 172.21.15.53:0/503778059 conn(0x7f5dd809c550 msgr2=0x7f5dd80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 -- 172.21.15.53:0/503778059 shutdown_connections 2024-01-16T15:09:10.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.351+0000 7f5dddc00640 1 -- 172.21.15.53:0/503778059 wait complete. 2024-01-16T15:09:10.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.353+0000 7f5dddc00640 1 Processor -- start 2024-01-16T15:09:10.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.353+0000 7f5dddc00640 1 -- start start 2024-01-16T15:09:10.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dddc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80b2a00 0x7f5dd8152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dddc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 0x7f5dd8156f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dddc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd8160f80 0x7f5dd8157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dddc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd8158080 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dddc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd8160270 con 0x7f5dd8160f80 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dddc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd8157ac0 con 0x7f5dd80b2a00 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dd6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 0x7f5dd8156f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dd6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 0x7f5dd8156f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36118/0 (socket says 172.21.15.53:36118) 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dd6000640 1 -- 172.21.15.53:0/2162535262 learned_addr learned my addr 172.21.15.53:0/2162535262 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dd7400640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd8160f80 0x7f5dd8157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dd6a00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80b2a00 0x7f5dd8152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.354+0000 7f5dd6000640 1 -- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80b2a00 msgr2=0x7f5dd8152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dd6000640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80b2a00 0x7f5dd8152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dd6000640 1 -- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd8160f80 msgr2=0x7f5dd8157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dd6000640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd8160f80 0x7f5dd8157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dd6000640 1 -- 172.21.15.53:0/2162535262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd00094d0 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dd7400640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd8160f80 0x7f5dd8157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dd6000640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 0x7f5dd8156f50 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0007ef0 tx=0x7f5dc000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0014620 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc0014780 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5dc0014b30 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd8157da0 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.355+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd80b04b0 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.357+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd80675d0 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.357+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f5dc0014c90 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.358+0000 7f5dcea00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5db006bc50 0x7f5db006e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:10.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.358+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5dc0089080 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.359+0000 7f5dd6a00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5db006bc50 0x7f5db006e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:10.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.360+0000 7f5dd6a00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5db006bc50 0x7f5db006e110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8000f60 tx=0x7f5dc8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:10.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.363+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5dc00557a0 con 0x7f5dd80ce8f0 2024-01-16T15:09:10.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.593+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54142", "target": ["mon-mgr", ""]}) v1 -- 0x7f5dd80d1c40 con 0x7f5db006bc50 2024-01-16T15:09:10.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.773+0000 7f5dcea00640 1 -- 172.21.15.53:0/2162535262 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5dc8001240 con 0x7f5db006bc50 2024-01-16T15:09:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5db006bc50 msgr2=0x7f5db006e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5db006bc50 0x7f5db006e110 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5dc8000f60 tx=0x7f5dc8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:10.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 msgr2=0x7f5dd8156f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 0x7f5dd8156f50 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f5dc0007ef0 tx=0x7f5dc000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 shutdown_connections 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5db006bc50 0x7f5db006e110 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5dd8160f80 0x7f5dd8157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5dd80ce8f0 0x7f5dd8156f50 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 --2- 172.21.15.53:0/2162535262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5dd80b2a00 0x7f5dd8152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 >> 172.21.15.53:0/2162535262 conn(0x7f5dd809c550 msgr2=0x7f5dd80d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 shutdown_connections 2024-01-16T15:09:10.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:10.777+0000 7f5dddc00640 1 -- 172.21.15.53:0/2162535262 wait complete. 2024-01-16T15:09:10.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54143 2024-01-16T15:09:11.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.027+0000 7f5572e00640 1 Processor -- start 2024-01-16T15:09:11.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.027+0000 7f5572e00640 1 -- start start 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.029+0000 7f5572e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f556c162430 0x7f556c0b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.029+0000 7f5572e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 0x7f556c071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.029+0000 7f5572e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c0702e0 con 0x7f556c0b3650 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.029+0000 7f5572e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c0705c0 con 0x7f556c162430 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.029+0000 7f5572e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c0708a0 con 0x7f556c162810 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.030+0000 7f5571000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 0x7f556c071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.030+0000 7f5571000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 0x7f556c071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:33522/0 (socket says 172.21.15.53:33522) 2024-01-16T15:09:11.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.030+0000 7f5571000640 1 -- 172.21.15.53:0/1254839532 learned_addr learned my addr 172.21.15.53:0/1254839532 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:11.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.030+0000 7f5571a00640 1 --2- 172.21.15.53:0/1254839532 >> v2:172.21.15.146:3300/0 conn(0x7f556c162430 0x7f556c0b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.031+0000 7f5571000640 1 -- 172.21.15.53:0/1254839532 >> v2:172.21.15.146:3300/0 conn(0x7f556c162430 msgr2=0x7f556c0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:11.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.031+0000 7f5571000640 1 --2- 172.21.15.53:0/1254839532 >> v2:172.21.15.146:3300/0 conn(0x7f556c162430 0x7f556c0b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.031+0000 7f5571000640 1 -- 172.21.15.53:0/1254839532 >> v1:172.21.15.146:6789/0 conn(0x7f556c0b3650 legacy=0x7f556c1650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.031+0000 7f5571000640 1 -- 172.21.15.53:0/1254839532 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c070b80 con 0x7f556c162810 2024-01-16T15:09:11.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.032+0000 7f5571000640 1 --2- 172.21.15.53:0/1254839532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 0x7f556c071240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5554009870 tx=0x7f555402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd63642ebb02241b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.032+0000 7f556be00640 1 -- 172.21.15.53:0/1254839532 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554030840 con 0x7f556c162810 2024-01-16T15:09:11.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.033+0000 7f556be00640 1 -- 172.21.15.53:0/1254839532 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55540309a0 con 0x7f556c162810 2024-01-16T15:09:11.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.033+0000 7f556be00640 1 -- 172.21.15.53:0/1254839532 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554030d30 con 0x7f556c162810 2024-01-16T15:09:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.033+0000 7f5572e00640 1 -- 172.21.15.53:0/1254839532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 msgr2=0x7f556c071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.033+0000 7f5572e00640 1 --2- 172.21.15.53:0/1254839532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 0x7f556c071240 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5554009870 tx=0x7f555402b980 comp rx=0 tx=0).stop 2024-01-16T15:09:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.034+0000 7f5572e00640 1 -- 172.21.15.53:0/1254839532 shutdown_connections 2024-01-16T15:09:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.034+0000 7f5572e00640 1 --2- 172.21.15.53:0/1254839532 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c162810 0x7f556c071240 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.034+0000 7f5572e00640 1 --2- 172.21.15.53:0/1254839532 >> v2:172.21.15.146:3300/0 conn(0x7f556c162430 0x7f556c0b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.034+0000 7f5572e00640 1 -- 172.21.15.53:0/1254839532 >> 172.21.15.53:0/1254839532 conn(0x7f556c06a800 msgr2=0x7f556c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.034+0000 7f5572e00640 1 -- 172.21.15.53:0/1254839532 shutdown_connections 2024-01-16T15:09:11.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.034+0000 7f5572e00640 1 -- 172.21.15.53:0/1254839532 wait complete. 2024-01-16T15:09:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.036+0000 7f5572e00640 1 Processor -- start 2024-01-16T15:09:11.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.036+0000 7f5572e00640 1 -- start start 2024-01-16T15:09:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f556c0b3650 0x7f556c146bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 0x7f556c162eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c0b0910 0x7f556c143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c1450f0 con 0x7f556c162430 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c147100 con 0x7f556c0b3650 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c1444e0 con 0x7f556c0b0910 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5571000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 0x7f556c162eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5571000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 0x7f556c162eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36126/0 (socket says 172.21.15.53:36126) 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5571000640 1 -- 172.21.15.53:0/743658080 learned_addr learned my addr 172.21.15.53:0/743658080 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5571a00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f556c0b3650 0x7f556c146bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.038+0000 7f5572400640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c0b0910 0x7f556c143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5571000640 1 -- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c0b0910 msgr2=0x7f556c143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5571000640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c0b0910 0x7f556c143520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5571000640 1 -- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f556c0b3650 msgr2=0x7f556c146bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5571000640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f556c0b3650 0x7f556c146bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5571000640 1 -- 172.21.15.53:0/743658080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55540094d0 con 0x7f556c162430 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5572400640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c0b0910 0x7f556c143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:11.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.039+0000 7f5571000640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 0x7f556c162eb0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f555402b9c0 tx=0x7f55540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.040+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555403e070 con 0x7f556c162430 2024-01-16T15:09:11.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.040+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5554039a10 con 0x7f556c162430 2024-01-16T15:09:11.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.040+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5554039de0 con 0x7f556c162430 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.040+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1447c0 con 0x7f556c162430 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.040+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c144d80 con 0x7f556c162430 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.041+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c144f40 con 0x7f556c162430 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.042+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 22) v1 ==== 90406+0+0 (secure 0 0 0) 0x7f555404b400 con 0x7f556c162430 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.043+0000 7f5569600640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f554406bc80 0x7f554406e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.043+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f55540ad6c0 con 0x7f556c162430 2024-01-16T15:09:11.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.043+0000 7f5571a00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f554406bc80 0x7f554406e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:11.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.045+0000 7f5571a00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f554406bc80 0x7f554406e140 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f555c009b40 tx=0x7f555c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:11.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.048+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55540771e0 con 0x7f556c162430 2024-01-16T15:09:11.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.271+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f55540775b0 con 0x7f556c162430 2024-01-16T15:09:11.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:11.282+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54143", "target": ["mon-mgr", ""]}) v1 -- 0x7f556c0ce780 con 0x7f554406bc80 2024-01-16T15:09:14.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.090+0000 7f5569600640 1 -- 172.21.15.53:0/743658080 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f556c0ce780 con 0x7f554406bc80 2024-01-16T15:09:14.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f554406bc80 msgr2=0x7f554406e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f554406bc80 0x7f554406e140 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f555c009b40 tx=0x7f555c009190 comp rx=0 tx=0).stop 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 msgr2=0x7f556c162eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 0x7f556c162eb0 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7f555402b9c0 tx=0x7f55540046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 shutdown_connections 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f554406bc80 0x7f554406e140 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f556c0b0910 0x7f556c143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f556c162430 0x7f556c162eb0 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 --2- 172.21.15.53:0/743658080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f556c0b3650 0x7f556c146bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.094+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 >> 172.21.15.53:0/743658080 conn(0x7f556c06a800 msgr2=0x7f556c071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.095+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 shutdown_connections 2024-01-16T15:09:14.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.095+0000 7f5572e00640 1 -- 172.21.15.53:0/743658080 wait complete. 2024-01-16T15:09:14.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54144 2024-01-16T15:09:14.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.348+0000 7f7aa3e00640 1 Processor -- start 2024-01-16T15:09:14.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.348+0000 7f7aa3e00640 1 -- start start 2024-01-16T15:09:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.350+0000 7f7aa3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7aa4071f00 0x7f7aa4146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.350+0000 7f7aa3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa40b2e10 con 0x7f7aa406c7c0 2024-01-16T15:09:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.350+0000 7f7aa3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aa406cc10 con 0x7f7aa4071410 2024-01-16T15:09:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.350+0000 7f7aa3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aa406cec0 con 0x7f7aa4071f00 2024-01-16T15:09:14.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.350+0000 7f7aa3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7aa406c7c0 0x7f7aa4166cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47068/0 (socket says 172.21.15.53:47068) 2024-01-16T15:09:14.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.350+0000 7f7aa3400640 1 -- 172.21.15.53:0/1456926858 learned_addr learned my addr 172.21.15.53:0/1456926858 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.351+0000 7f7aa2000640 1 --2- 172.21.15.53:0/1456926858 >> v2:172.21.15.146:3300/0 conn(0x7f7aa4071f00 0x7f7aa4146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.351+0000 7f7aa1600640 1 -- 172.21.15.53:0/1456926858 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119306954 0 0) 0x7f7aa40b2e10 con 0x7f7aa406c7c0 2024-01-16T15:09:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.351+0000 7f7aa1600640 1 -- 172.21.15.53:0/1456926858 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a84003520 con 0x7f7aa406c7c0 2024-01-16T15:09:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.352+0000 7f7aa2000640 1 -- 172.21.15.53:0/1456926858 >> v1:172.21.15.146:6789/0 conn(0x7f7aa4071410 legacy=0x7f7aa40717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.352+0000 7f7aa2000640 1 -- 172.21.15.53:0/1456926858 >> v1:172.21.15.53:6789/0 conn(0x7f7aa406c7c0 legacy=0x7f7aa4166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.352+0000 7f7aa2000640 1 -- 172.21.15.53:0/1456926858 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa406d1a0 con 0x7f7aa4071f00 2024-01-16T15:09:14.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.353+0000 7f7aa2000640 1 --2- 172.21.15.53:0/1456926858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa4146770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0097a0 tx=0x7f7a8c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e6123f59d7bb856 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.353+0000 7f7aa1600640 1 -- 172.21.15.53:0/1456926858 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c031830 con 0x7f7aa4071f00 2024-01-16T15:09:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.353+0000 7f7aa1600640 1 -- 172.21.15.53:0/1456926858 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c031990 con 0x7f7aa4071f00 2024-01-16T15:09:14.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa1600640 1 -- 172.21.15.53:0/1456926858 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c031d20 con 0x7f7aa4071f00 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa3e00640 1 -- 172.21.15.53:0/1456926858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 msgr2=0x7f7aa4146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/1456926858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa4146770 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0097a0 tx=0x7f7a8c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa3e00640 1 -- 172.21.15.53:0/1456926858 shutdown_connections 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/1456926858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa4146770 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa3e00640 1 -- 172.21.15.53:0/1456926858 >> 172.21.15.53:0/1456926858 conn(0x7f7aa406a800 msgr2=0x7f7aa406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.354+0000 7f7aa3e00640 1 -- 172.21.15.53:0/1456926858 shutdown_connections 2024-01-16T15:09:14.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.355+0000 7f7aa3e00640 1 -- 172.21.15.53:0/1456926858 wait complete. 2024-01-16T15:09:14.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.356+0000 7f7aa3e00640 1 Processor -- start 2024-01-16T15:09:14.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 -- start start 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7aa406c7c0 0x7f7aa4132d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 0x7f7aa41452f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa40c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa40c3d70 con 0x7f7aa4071410 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa415ed00 con 0x7f7aa4071f00 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.357+0000 7f7aa3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa415efe0 con 0x7f7aa406c7c0 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 0x7f7aa41452f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 0x7f7aa41452f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58434/0 (socket says 172.21.15.53:58434) 2024-01-16T15:09:14.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 -- 172.21.15.53:0/3478080817 learned_addr learned my addr 172.21.15.53:0/3478080817 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:14.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2a00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7aa406c7c0 0x7f7aa4132d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa3400640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa40c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 -- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7aa406c7c0 msgr2=0x7f7aa4132d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7aa406c7c0 0x7f7aa4132d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 -- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 msgr2=0x7f7aa40c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa40c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2000640 1 -- 172.21.15.53:0/3478080817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8c009400 con 0x7f7aa4071410 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa2a00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7aa406c7c0 0x7f7aa4132d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.358+0000 7f7aa3400640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa40c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.359+0000 7f7aa2000640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 0x7f7aa41452f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0097a0 tx=0x7f7a8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.359+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c004230 con 0x7f7aa4071410 2024-01-16T15:09:14.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.359+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8c004390 con 0x7f7aa4071410 2024-01-16T15:09:14.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.359+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a8c03a870 con 0x7f7aa4071410 2024-01-16T15:09:14.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.359+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa415f2c0 con 0x7f7aa4071410 2024-01-16T15:09:14.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.359+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa4145830 con 0x7f7aa4071410 2024-01-16T15:09:14.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.361+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa40c3ee0 con 0x7f7aa4071410 2024-01-16T15:09:14.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.361+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f7a8c03a9d0 con 0x7f7aa4071410 2024-01-16T15:09:14.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.362+0000 7f7a9b400640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a7c06bdc0 0x7f7a7c06e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:14.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.362+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7a8c0aa6f0 con 0x7f7aa4071410 2024-01-16T15:09:14.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.363+0000 7f7aa2a00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a7c06bdc0 0x7f7a7c06e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:14.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.364+0000 7f7aa2a00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a7c06bdc0 0x7f7a7c06e280 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7a94000f60 tx=0x7f7a94009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:14.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.367+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7aa40c3ee0 con 0x7f7aa4071410 2024-01-16T15:09:14.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:14.596+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7f7aa406da40 con 0x7f7a7c06bdc0 2024-01-16T15:09:19.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.099+0000 7f7a9b400640 1 -- 172.21.15.53:0/3478080817 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7a94001d30 con 0x7f7a7c06bdc0 2024-01-16T15:09:19.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.102+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a7c06bdc0 msgr2=0x7f7a7c06e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.102+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a7c06bdc0 0x7f7a7c06e280 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7a94000f60 tx=0x7f7a94009040 comp rx=0 tx=0).stop 2024-01-16T15:09:19.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.102+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 msgr2=0x7f7aa41452f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.102+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 0x7f7aa41452f0 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0097a0 tx=0x7f7a8c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 shutdown_connections 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a7c06bdc0 0x7f7a7c06e280 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7aa4071f00 0x7f7aa40c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7aa4071410 0x7f7aa41452f0 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 --2- 172.21.15.53:0/3478080817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7aa406c7c0 0x7f7aa4132d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 >> 172.21.15.53:0/3478080817 conn(0x7f7aa406a800 msgr2=0x7f7aa40717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 shutdown_connections 2024-01-16T15:09:19.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.103+0000 7f7aa3e00640 1 -- 172.21.15.53:0/3478080817 wait complete. 2024-01-16T15:09:19.105 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54145 2024-01-16T15:09:19.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.356+0000 7fe8e2c00640 1 Processor -- start 2024-01-16T15:09:19.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.356+0000 7fe8e2c00640 1 -- start start 2024-01-16T15:09:19.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e2c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe8dc0cc500 0x7fe8dc0cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 0x7fe8dc06d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e2c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8dc070e50 con 0x7fe8dc143c40 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e2c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8dc0723c0 con 0x7fe8dc0cc500 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc0cced0 con 0x7fe8dc06d160 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe8dc0cc500 0x7fe8dc0cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e1800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe8dc143c40 0x7fe8dc071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47076/0 (socket says 172.21.15.53:47076) 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.358+0000 7fe8e1800640 1 -- 172.21.15.53:0/1496097429 learned_addr learned my addr 172.21.15.53:0/1496097429 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.359+0000 7fe8e0e00640 1 -- 172.21.15.53:0/1496097429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 msgr2=0x7fe8dc06d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.359+0000 7fe8e0e00640 1 --2- 172.21.15.53:0/1496097429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 0x7fe8dc06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.359+0000 7fe8e0e00640 1 -- 172.21.15.53:0/1496097429 >> v1:172.21.15.53:6789/0 conn(0x7fe8dc143c40 legacy=0x7fe8dc071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.359+0000 7fe8e0e00640 1 -- 172.21.15.53:0/1496097429 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc144020 con 0x7fe8dc0cc500 2024-01-16T15:09:19.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.360+0000 7fe8e0e00640 1 --2- 172.21.15.53:0/1496097429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc0cc990 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40097a0 tx=0x7fe8c402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8af5405fab11c8c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.360+0000 7fe8dbe00640 1 -- 172.21.15.53:0/1496097429 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4031780 con 0x7fe8dc0cc500 2024-01-16T15:09:19.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.360+0000 7fe8dbe00640 1 -- 172.21.15.53:0/1496097429 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c40318e0 con 0x7fe8dc0cc500 2024-01-16T15:09:19.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.360+0000 7fe8dbe00640 1 -- 172.21.15.53:0/1496097429 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4031c70 con 0x7fe8dc0cc500 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 -- 172.21.15.53:0/1496097429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 msgr2=0x7fe8dc0cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/1496097429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc0cc990 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fe8c40097a0 tx=0x7fe8c402b820 comp rx=0 tx=0).stop 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 -- 172.21.15.53:0/1496097429 shutdown_connections 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/1496097429 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 0x7fe8dc06d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/1496097429 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc0cc990 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 -- 172.21.15.53:0/1496097429 >> 172.21.15.53:0/1496097429 conn(0x7fe8dc06a800 msgr2=0x7fe8dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:19.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 -- 172.21.15.53:0/1496097429 shutdown_connections 2024-01-16T15:09:19.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.361+0000 7fe8e2c00640 1 -- 172.21.15.53:0/1496097429 wait complete. 2024-01-16T15:09:19.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.363+0000 7fe8e2c00640 1 Processor -- start 2024-01-16T15:09:19.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.363+0000 7fe8e2c00640 1 -- start start 2024-01-16T15:09:19.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 0x7fe8dc137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc143c40 0x7fe8dc1f04a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc135720 con 0x7fe8dc0cc500 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc1354a0 con 0x7fe8dc143c40 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8dc13a170 con 0x7fe8dc06d160 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc134f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58452/0 (socket says 172.21.15.53:58452) 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 -- 172.21.15.53:0/2258802039 learned_addr learned my addr 172.21.15.53:0/2258802039 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 -- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 msgr2=0x7fe8dc137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2200640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc143c40 0x7fe8dc1f04a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 0x7fe8dc137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 -- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc143c40 msgr2=0x7fe8dc1f04a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc143c40 0x7fe8dc1f04a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:19.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e0e00640 1 -- 172.21.15.53:0/2258802039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c4009400 con 0x7fe8dc0cc500 2024-01-16T15:09:19.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.365+0000 7fe8e2200640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc143c40 0x7fe8dc1f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:19.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.366+0000 7fe8e0e00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc134f60 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4034040 tx=0x7fe8c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.366+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c40328e0 con 0x7fe8dc0cc500 2024-01-16T15:09:19.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.366+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8c4032a60 con 0x7fe8dc0cc500 2024-01-16T15:09:19.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.366+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8c4032e10 con 0x7fe8dc0cc500 2024-01-16T15:09:19.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.366+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8dc13a450 con 0x7fe8dc0cc500 2024-01-16T15:09:19.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.366+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8dc13aa10 con 0x7fe8dc0cc500 2024-01-16T15:09:19.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.368+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8dc13ac00 con 0x7fe8dc0cc500 2024-01-16T15:09:19.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.369+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fe8c404d3f0 con 0x7fe8dc0cc500 2024-01-16T15:09:19.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.369+0000 7fe8d9600640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8b406bd70 0x7fe8b406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:19.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.370+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe8c40aa280 con 0x7fe8dc0cc500 2024-01-16T15:09:19.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.370+0000 7fe8e1800640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8b406bd70 0x7fe8b406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:19.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.371+0000 7fe8e1800640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8b406bd70 0x7fe8b406e230 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc005880 tx=0x7fe8cc007a00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.374+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8c40b0030 con 0x7fe8dc0cc500 2024-01-16T15:09:19.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:19.606+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54145", "target": ["mon-mgr", ""]}) v1 -- 0x7fe8dc06c960 con 0x7fe8b406bd70 2024-01-16T15:09:22.691 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:22.700 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:24.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.093+0000 7fe8d9600640 1 -- 172.21.15.53:0/2258802039 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe8cc0058b0 con 0x7fe8b406bd70 2024-01-16T15:09:24.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.095+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8b406bd70 msgr2=0x7fe8b406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.095+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8b406bd70 0x7fe8b406e230 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fe8cc005880 tx=0x7fe8cc007a00 comp rx=0 tx=0).stop 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.095+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 msgr2=0x7fe8dc134f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.095+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc134f60 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fe8c4034040 tx=0x7fe8c40046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.095+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 shutdown_connections 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8b406bd70 0x7fe8b406e230 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8dc143c40 0x7fe8dc1f04a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8dc0cc500 0x7fe8dc134f60 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 --2- 172.21.15.53:0/2258802039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8dc06d160 0x7fe8dc137f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 >> 172.21.15.53:0/2258802039 conn(0x7fe8dc06a800 msgr2=0x7fe8dc071290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 shutdown_connections 2024-01-16T15:09:24.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.096+0000 7fe8e2c00640 1 -- 172.21.15.53:0/2258802039 wait complete. 2024-01-16T15:09:24.096 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54146 2024-01-16T15:09:24.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.347+0000 7f0129000640 1 Processor -- start 2024-01-16T15:09:24.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.347+0000 7f0129000640 1 -- start start 2024-01-16T15:09:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.349+0000 7f0129000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0124143c40 0x7f0124071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.349+0000 7f0129000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124070e50 con 0x7f01240717d0 2024-01-16T15:09:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.349+0000 7f0129000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124144020 con 0x7f01240ccc10 2024-01-16T15:09:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.349+0000 7f0129000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01240723c0 con 0x7f0124143c40 2024-01-16T15:09:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.349+0000 7f0122a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f01240717d0 0x7f01240cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44196/0 (socket says 172.21.15.53:44196) 2024-01-16T15:09:24.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.349+0000 7f0122a00640 1 -- 172.21.15.53:0/1184472860 learned_addr learned my addr 172.21.15.53:0/1184472860 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1441898253 0 0) 0x7f0124070e50 con 0x7f01240717d0 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0104003520 con 0x7f01240717d0 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0123400640 1 --2- 172.21.15.53:0/1184472860 >> v2:172.21.15.146:3300/0 conn(0x7f0124143c40 0x7f0124071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924665734 0 0) 0x7f0104003520 con 0x7f01240717d0 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 >> v2:172.21.15.146:3300/0 conn(0x7f0124143c40 msgr2=0x7f0124071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 --2- 172.21.15.53:0/1184472860 >> v2:172.21.15.146:3300/0 conn(0x7f0124143c40 0x7f0124071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 >> v1:172.21.15.146:6789/0 conn(0x7f01240ccc10 legacy=0x7f012406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124071e00 con 0x7f01240717d0 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.350+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f010c001670 con 0x7f01240717d0 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 >> v1:172.21.15.53:6789/0 conn(0x7f01240717d0 legacy=0x7f01240cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01040039c0 0x7f0104003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01040047c0 0x7f0104004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f010400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01240723c0 con 0x7f01040052a0 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f010400c1c0 con 0x7f01040039c0 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f010400c380 con 0x7f01040047c0 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0123e00640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f010400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0123400640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01040039c0 0x7f0104003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.351+0000 7f0122a00640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01040047c0 0x7f0104004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.352+0000 7f0123e00640 1 -- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01040047c0 msgr2=0x7f0104004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.352+0000 7f0123e00640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f01040047c0 0x7f0104004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.352+0000 7f0123e00640 1 -- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01040039c0 msgr2=0x7f0104003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.352+0000 7f0123e00640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f01040039c0 0x7f0104003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.352+0000 7f0123e00640 1 -- 172.21.15.53:0/1184472860 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0104003520 con 0x7f01040052a0 2024-01-16T15:09:24.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.353+0000 7f0123e00640 1 -- 172.21.15.53:0/1184472860 reap_dead start 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.353+0000 7f0123e00640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f010400b980 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f011c001e50 tx=0x7f01040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa4bd71609023e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.353+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c003b70 con 0x7f01040052a0 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.353+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c003d10 con 0x7f01040052a0 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.353+0000 7f0122000640 1 -- 172.21.15.53:0/1184472860 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f011c005310 con 0x7f01040052a0 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 -- 172.21.15.53:0/1184472860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 msgr2=0x7f010400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f010400b980 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7f011c001e50 tx=0x7f01040057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 -- 172.21.15.53:0/1184472860 shutdown_connections 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 --2- 172.21.15.53:0/1184472860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f010400b980 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 -- 172.21.15.53:0/1184472860 >> 172.21.15.53:0/1184472860 conn(0x7f012406a800 msgr2=0x7f012406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 -- 172.21.15.53:0/1184472860 shutdown_connections 2024-01-16T15:09:24.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.354+0000 7f0129000640 1 -- 172.21.15.53:0/1184472860 wait complete. 2024-01-16T15:09:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.356+0000 7f0129000640 1 Processor -- start 2024-01-16T15:09:24.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.356+0000 7f0129000640 1 -- start start 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.357+0000 7f0129000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f012415a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.357+0000 7f0129000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f012413a250 0x7f01241386f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.357+0000 7f0129000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0124138c30 0x7f01240b2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.357+0000 7f0129000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0124139dc0 con 0x7f01040052a0 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.357+0000 7f0129000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f012415a8a0 con 0x7f0124138c30 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.357+0000 7f0129000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01241391b0 con 0x7f012413a250 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f012415a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f012415a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48806/0 (socket says 172.21.15.53:48806) 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 -- 172.21.15.53:0/462326725 learned_addr learned my addr 172.21.15.53:0/462326725 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0122a00640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f012413a250 0x7f01241386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 -- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f012413a250 msgr2=0x7f01241386f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f012413a250 0x7f01241386f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 -- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0124138c30 msgr2=0x7f01240b2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0124138c30 0x7f01240b2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.358+0000 7f0123400640 1 -- 172.21.15.53:0/462326725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124139490 con 0x7f01040052a0 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.359+0000 7f0123400640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f012415a360 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f011400b7c0 tx=0x7f011400bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.359+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114010830 con 0x7f01040052a0 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.359+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0114010990 con 0x7f01040052a0 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.359+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0114010d20 con 0x7f01040052a0 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.359+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0124139770 con 0x7f01040052a0 2024-01-16T15:09:24.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.359+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012415aca0 con 0x7f01040052a0 2024-01-16T15:09:24.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.360+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0124067450 con 0x7f01040052a0 2024-01-16T15:09:24.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.361+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f0114014430 con 0x7f01040052a0 2024-01-16T15:09:24.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.362+0000 7f011b400640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00fc06bda0 0x7f00fc06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:24.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.362+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f011408b6f0 con 0x7f01040052a0 2024-01-16T15:09:24.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.363+0000 7f0122a00640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00fc06bda0 0x7f00fc06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:24.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.364+0000 7f0122a00640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00fc06bda0 0x7f00fc06e260 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f01241f6580 tx=0x7f010c001b50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:24.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.367+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f01140553e0 con 0x7f01040052a0 2024-01-16T15:09:24.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.596+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54146", "target": ["mon-mgr", ""]}) v1 -- 0x7f012406cc40 con 0x7f00fc06bda0 2024-01-16T15:09:24.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.768+0000 7f011b400640 1 -- 172.21.15.53:0/462326725 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f010c001490 con 0x7f00fc06bda0 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00fc06bda0 msgr2=0x7f00fc06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00fc06bda0 0x7f00fc06e260 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f01241f6580 tx=0x7f010c001b50 comp rx=0 tx=0).stop 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 msgr2=0x7f012415a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f012415a360 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7f011400b7c0 tx=0x7f011400bc80 comp rx=0 tx=0).stop 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 shutdown_connections 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00fc06bda0 0x7f00fc06e260 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0124138c30 0x7f01240b2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f012413a250 0x7f01241386f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 --2- 172.21.15.53:0/462326725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f01040052a0 0x7f012415a360 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:24.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 >> 172.21.15.53:0/462326725 conn(0x7f012406a800 msgr2=0x7f01240adda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 shutdown_connections 2024-01-16T15:09:24.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:24.771+0000 7f0129000640 1 -- 172.21.15.53:0/462326725 wait complete. 2024-01-16T15:09:24.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54147 2024-01-16T15:09:24.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.021+0000 7f16fca00640 1 Processor -- start 2024-01-16T15:09:24.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.021+0000 7f16fca00640 1 -- start start 2024-01-16T15:09:25.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.023+0000 7f16fca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 0x7f16f80b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.023+0000 7f16fca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16f8160b80 0x7f16f81617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.023+0000 7f16fca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16f80b2b90 con 0x7f16f81607a0 2024-01-16T15:09:25.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.023+0000 7f16fca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16f80d21d0 con 0x7f16f8160b80 2024-01-16T15:09:25.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.023+0000 7f16fca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80b2e30 con 0x7f16f80b3eb0 2024-01-16T15:09:25.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 0x7f16f80b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 0x7f16f80b4330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:48884/0 (socket says 172.21.15.53:48884) 2024-01-16T15:09:25.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 -- 172.21.15.53:0/1029924387 learned_addr learned my addr 172.21.15.53:0/1029924387 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:25.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16f7e00640 1 --2- 172.21.15.53:0/1029924387 >> v2:172.21.15.146:3300/0 conn(0x7f16f8160b80 0x7f16f81617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 -- 172.21.15.53:0/1029924387 >> v2:172.21.15.146:3300/0 conn(0x7f16f8160b80 msgr2=0x7f16f81617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 --2- 172.21.15.53:0/1029924387 >> v2:172.21.15.146:3300/0 conn(0x7f16f8160b80 0x7f16f81617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 -- 172.21.15.53:0/1029924387 >> v1:172.21.15.146:6789/0 conn(0x7f16f81607a0 legacy=0x7f16f81610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.024+0000 7f16efe00640 1 -- 172.21.15.53:0/1029924387 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f80b3110 con 0x7f16f80b3eb0 2024-01-16T15:09:25.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.026+0000 7f16efe00640 1 --2- 172.21.15.53:0/1029924387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 0x7f16f80b4330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009870 tx=0x7f16e002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4298502531486cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.026+0000 7f16f6a00640 1 -- 172.21.15.53:0/1029924387 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0030840 con 0x7f16f80b3eb0 2024-01-16T15:09:25.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.026+0000 7f16f6a00640 1 -- 172.21.15.53:0/1029924387 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e00309a0 con 0x7f16f80b3eb0 2024-01-16T15:09:25.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.026+0000 7f16f6a00640 1 -- 172.21.15.53:0/1029924387 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e0030d30 con 0x7f16f80b3eb0 2024-01-16T15:09:25.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 -- 172.21.15.53:0/1029924387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 msgr2=0x7f16f80b4330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 --2- 172.21.15.53:0/1029924387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 0x7f16f80b4330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f16e0009870 tx=0x7f16e002b980 comp rx=0 tx=0).stop 2024-01-16T15:09:25.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 -- 172.21.15.53:0/1029924387 shutdown_connections 2024-01-16T15:09:25.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 --2- 172.21.15.53:0/1029924387 >> v2:172.21.15.146:3300/0 conn(0x7f16f8160b80 0x7f16f81617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 --2- 172.21.15.53:0/1029924387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f80b3eb0 0x7f16f80b4330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 -- 172.21.15.53:0/1029924387 >> 172.21.15.53:0/1029924387 conn(0x7f16f809c590 msgr2=0x7f16f80d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 -- 172.21.15.53:0/1029924387 shutdown_connections 2024-01-16T15:09:25.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.027+0000 7f16fca00640 1 -- 172.21.15.53:0/1029924387 wait complete. 2024-01-16T15:09:25.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.029+0000 7f16fca00640 1 Processor -- start 2024-01-16T15:09:25.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.029+0000 7f16fca00640 1 -- start start 2024-01-16T15:09:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.030+0000 7f16fca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 0x7f16f80b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.030+0000 7f16fca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f81607a0 0x7f16f8151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.030+0000 7f16fca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f8160b80 0x7f16f80ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.030+0000 7f16fca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80cc080 con 0x7f16f80b3eb0 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.030+0000 7f16fca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80cbac0 con 0x7f16f8160b80 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.030+0000 7f16fca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16f80cbda0 con 0x7f16f81607a0 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 0x7f16f80b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 0x7f16f80b5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48812/0 (socket says 172.21.15.53:48812) 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 -- 172.21.15.53:0/3644671847 learned_addr learned my addr 172.21.15.53:0/3644671847 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7e00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f8160b80 0x7f16f80ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16efe00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f81607a0 0x7f16f8151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 -- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f81607a0 msgr2=0x7f16f8151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f81607a0 0x7f16f8151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 -- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f8160b80 msgr2=0x7f16f80ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f8160b80 0x7f16f80ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.031+0000 7f16f7400640 1 -- 172.21.15.53:0/3644671847 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16e00094d0 con 0x7f16f80b3eb0 2024-01-16T15:09:25.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.032+0000 7f16f7400640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 0x7f16f80b5e70 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c450 tx=0x7f16e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.032+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e80052d0 con 0x7f16f80b3eb0 2024-01-16T15:09:25.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.032+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e8005430 con 0x7f16f80b3eb0 2024-01-16T15:09:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.032+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16e8012870 con 0x7f16f80b3eb0 2024-01-16T15:09:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.032+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16f80cc450 con 0x7f16f80b3eb0 2024-01-16T15:09:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.032+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16f80b7660 con 0x7f16f80b3eb0 2024-01-16T15:09:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.034+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16f80b77f0 con 0x7f16f80b3eb0 2024-01-16T15:09:25.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.035+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7f16e80129d0 con 0x7f16f80b3eb0 2024-01-16T15:09:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.035+0000 7f16ef400640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f16d006bd70 0x7f16d006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.036+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f16e8088020 con 0x7f16f80b3eb0 2024-01-16T15:09:25.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.036+0000 7f16efe00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f16d006bd70 0x7f16d006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.037+0000 7f16efe00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f16d006bd70 0x7f16d006e230 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f16e0036000 tx=0x7f16e003b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.040+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16e808ce60 con 0x7f16f80b3eb0 2024-01-16T15:09:25.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.270+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54147", "target": ["mon-mgr", ""]}) v1 -- 0x7f16f80b39b0 con 0x7f16d006bd70 2024-01-16T15:09:25.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.438+0000 7f16ef400640 1 -- 172.21.15.53:0/3644671847 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f16e0030b90 con 0x7f16d006bd70 2024-01-16T15:09:25.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f16d006bd70 msgr2=0x7f16d006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f16d006bd70 0x7f16d006e230 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f16e0036000 tx=0x7f16e003b040 comp rx=0 tx=0).stop 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 msgr2=0x7f16f80b5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 0x7f16f80b5e70 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7f16e800c450 tx=0x7f16e800c910 comp rx=0 tx=0).stop 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 shutdown_connections 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f16d006bd70 0x7f16d006e230 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16f8160b80 0x7f16f80ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16f81607a0 0x7f16f8151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 --2- 172.21.15.53:0/3644671847 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16f80b3eb0 0x7f16f80b5e70 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 >> 172.21.15.53:0/3644671847 conn(0x7f16f809c590 msgr2=0x7f16f80d1ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 shutdown_connections 2024-01-16T15:09:25.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.441+0000 7f16fca00640 1 -- 172.21.15.53:0/3644671847 wait complete. 2024-01-16T15:09:25.440 INFO:tasks.cephfs.test_volumes:expected clone state = ('in-progress',) 2024-01-16T15:09:25.440 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54144 2024-01-16T15:09:25.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.686+0000 7fcc8a600640 1 Processor -- start 2024-01-16T15:09:25.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.686+0000 7fcc8a600640 1 -- start start 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.687+0000 7fcc8a600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc8415d4b0 0x7fcc840d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.687+0000 7fcc8a600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcc840cf9a0 0x7fcc840cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc8a600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc840d0320 con 0x7fcc840cddf0 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc8a600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840b2730 con 0x7fcc8415d4b0 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc8a600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840d1ec0 con 0x7fcc840cf9a0 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc89200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc8415d4b0 0x7fcc840d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc89200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcc8415d4b0 0x7fcc840d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48822/0 (socket says 172.21.15.53:48822) 2024-01-16T15:09:25.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc89200640 1 -- 172.21.15.53:0/475503473 learned_addr learned my addr 172.21.15.53:0/475503473 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:25.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc83e00640 1 --2- 172.21.15.53:0/475503473 >> v2:172.21.15.146:3300/0 conn(0x7fcc840cf9a0 0x7fcc840cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc89200640 1 -- 172.21.15.53:0/475503473 >> v2:172.21.15.146:3300/0 conn(0x7fcc840cf9a0 msgr2=0x7fcc840cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.688+0000 7fcc89200640 1 --2- 172.21.15.53:0/475503473 >> v2:172.21.15.146:3300/0 conn(0x7fcc840cf9a0 0x7fcc840cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.689+0000 7fcc89200640 1 -- 172.21.15.53:0/475503473 >> v1:172.21.15.146:6789/0 conn(0x7fcc840cddf0 legacy=0x7fcc840ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:25.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.689+0000 7fcc89200640 1 -- 172.21.15.53:0/475503473 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc840b29d0 con 0x7fcc8415d4b0 2024-01-16T15:09:25.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.689+0000 7fcc83e00640 1 --2- 172.21.15.53:0/475503473 >> v2:172.21.15.146:3300/0 conn(0x7fcc840cf9a0 0x7fcc840cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:25.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.690+0000 7fcc89200640 1 --2- 172.21.15.53:0/475503473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc840d1550 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fcc8408b580 tx=0x7fcc7402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63b550e99ad28e84 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.690+0000 7fcc83400640 1 -- 172.21.15.53:0/475503473 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74031870 con 0x7fcc8415d4b0 2024-01-16T15:09:25.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.690+0000 7fcc83400640 1 -- 172.21.15.53:0/475503473 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc740319d0 con 0x7fcc8415d4b0 2024-01-16T15:09:25.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.690+0000 7fcc83400640 1 -- 172.21.15.53:0/475503473 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc74031d60 con 0x7fcc8415d4b0 2024-01-16T15:09:25.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 -- 172.21.15.53:0/475503473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 msgr2=0x7fcc840d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 --2- 172.21.15.53:0/475503473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc840d1550 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fcc8408b580 tx=0x7fcc7402b910 comp rx=0 tx=0).stop 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 -- 172.21.15.53:0/475503473 shutdown_connections 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 --2- 172.21.15.53:0/475503473 >> v2:172.21.15.146:3300/0 conn(0x7fcc840cf9a0 0x7fcc840cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 --2- 172.21.15.53:0/475503473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc840d1550 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 -- 172.21.15.53:0/475503473 >> 172.21.15.53:0/475503473 conn(0x7fcc840d3000 msgr2=0x7fcc840d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 -- 172.21.15.53:0/475503473 shutdown_connections 2024-01-16T15:09:25.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.691+0000 7fcc8a600640 1 -- 172.21.15.53:0/475503473 wait complete. 2024-01-16T15:09:25.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.693+0000 7fcc8a600640 1 Processor -- start 2024-01-16T15:09:25.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.693+0000 7fcc8a600640 1 -- start start 2024-01-16T15:09:25.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc8a600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc840cddf0 0x7fcc8415b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc8a600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc840cf9a0 0x7fcc840b71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc8a600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc841f2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc8a600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840b76f0 con 0x7fcc8415d4b0 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc8a600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840b8460 con 0x7fcc840cddf0 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc8a600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840b66d0 con 0x7fcc840cf9a0 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc89c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc841f2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc89c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc841f2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48824/0 (socket says 172.21.15.53:48824) 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.694+0000 7fcc89c00640 1 -- 172.21.15.53:0/2344623264 learned_addr learned my addr 172.21.15.53:0/2344623264 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc83e00640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc840cf9a0 0x7fcc840b71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89200640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc840cddf0 0x7fcc8415b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89c00640 1 -- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc840cf9a0 msgr2=0x7fcc840b71b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89c00640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc840cf9a0 0x7fcc840b71b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89c00640 1 -- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc840cddf0 msgr2=0x7fcc8415b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89c00640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc840cddf0 0x7fcc8415b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89c00640 1 -- 172.21.15.53:0/2344623264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc740094d0 con 0x7fcc8415d4b0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc89c00640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc841f2b50 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00c780 tx=0x7fcc7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c004b60 con 0x7fcc8415d4b0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7c00ce90 con 0x7fcc8415d4b0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.695+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc7c00a850 con 0x7fcc8415d4b0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.696+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8415da10 con 0x7fcc8415d4b0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.696+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc840b6c90 con 0x7fcc8415d4b0 2024-01-16T15:09:25.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.697+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc84067450 con 0x7fcc8415d4b0 2024-01-16T15:09:25.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.698+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fcc7c00a9b0 con 0x7fcc8415d4b0 2024-01-16T15:09:25.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.699+0000 7fcc80c00640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc5c06bda0 0x7fcc5c06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:25.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.699+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcc7c0888e0 con 0x7fcc8415d4b0 2024-01-16T15:09:25.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.700+0000 7fcc89200640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc5c06bda0 0x7fcc5c06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:25.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.701+0000 7fcc89200640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc5c06bda0 0x7fcc5c06e260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcc74036000 tx=0x7fcc7403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:25.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.704+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc7c054f70 con 0x7fcc8415d4b0 2024-01-16T15:09:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.941+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7fcc840b1a60 con 0x7fcc5c06bda0 2024-01-16T15:09:25.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.959+0000 7fcc80c00640 1 -- 172.21.15.53:0/2344623264 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fcc840b1a60 con 0x7fcc5c06bda0 2024-01-16T15:09:25.936 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:09:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:09:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:09:25.936 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:09:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:09:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:09:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:25.937 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:25.937 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.960+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc5c06bda0 msgr2=0x7fcc5c06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.960+0000 7fcc8a600640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc5c06bda0 0x7fcc5c06e260 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fcc74036000 tx=0x7fcc7403c040 comp rx=0 tx=0).stop 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.960+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 msgr2=0x7fcc841f2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.960+0000 7fcc8a600640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc841f2b50 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fcc7c00c780 tx=0x7fcc7c00cc40 comp rx=0 tx=0).stop 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 shutdown_connections 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc5c06bda0 0x7fcc5c06e260 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc8415d4b0 0x7fcc841f2b50 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc840cf9a0 0x7fcc840b71b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 --2- 172.21.15.53:0/2344623264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc840cddf0 0x7fcc8415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:25.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 >> 172.21.15.53:0/2344623264 conn(0x7fcc840d3000 msgr2=0x7fcc8406c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:25.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 shutdown_connections 2024-01-16T15:09:25.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:25.961+0000 7fcc8a600640 1 -- 172.21.15.53:0/2344623264 wait complete. 2024-01-16T15:09:25.959 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:09:26.960 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54144 2024-01-16T15:09:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.215+0000 7fd011600640 1 Processor -- start 2024-01-16T15:09:27.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.216+0000 7fd011600640 1 -- start start 2024-01-16T15:09:27.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd011600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd00c143ae0 0x7fd00c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd011600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd00c0cf7d0 0x7fd00c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd011600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd00c0cfc30 con 0x7fd00c15e1e0 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd011600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c0b1ee0 con 0x7fd00c143ae0 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd011600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd00c0b2780 con 0x7fd00c0cf7d0 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd00c143ae0 0x7fd00c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd00c143ae0 0x7fd00c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48826/0 (socket says 172.21.15.53:48826) 2024-01-16T15:09:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 -- 172.21.15.53:0/2196768239 learned_addr learned my addr 172.21.15.53:0/2196768239 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 -- 172.21.15.53:0/2196768239 >> v2:172.21.15.146:3300/0 conn(0x7fd00c0cf7d0 msgr2=0x7fd00c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 --2- 172.21.15.53:0/2196768239 >> v2:172.21.15.146:3300/0 conn(0x7fd00c0cf7d0 0x7fd00c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 -- 172.21.15.53:0/2196768239 >> v1:172.21.15.146:6789/0 conn(0x7fd00c15e1e0 legacy=0x7fd00c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.218+0000 7fd00aa00640 1 -- 172.21.15.53:0/2196768239 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd00c15e990 con 0x7fd00c143ae0 2024-01-16T15:09:27.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.219+0000 7fd00aa00640 1 --2- 172.21.15.53:0/2196768239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c0cf290 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0cb830 tx=0x7fcffc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79f6bcbc03e887b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.220+0000 7fd009600640 1 -- 172.21.15.53:0/2196768239 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc02ca80 con 0x7fd00c143ae0 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.220+0000 7fd009600640 1 -- 172.21.15.53:0/2196768239 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcffc02cbe0 con 0x7fd00c143ae0 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.220+0000 7fd009600640 1 -- 172.21.15.53:0/2196768239 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcffc0343f0 con 0x7fd00c143ae0 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 -- 172.21.15.53:0/2196768239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 msgr2=0x7fd00c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 --2- 172.21.15.53:0/2196768239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c0cf290 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0cb830 tx=0x7fcffc02bee0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 -- 172.21.15.53:0/2196768239 shutdown_connections 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 --2- 172.21.15.53:0/2196768239 >> v2:172.21.15.146:3300/0 conn(0x7fd00c0cf7d0 0x7fd00c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 --2- 172.21.15.53:0/2196768239 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c0cf290 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 -- 172.21.15.53:0/2196768239 >> 172.21.15.53:0/2196768239 conn(0x7fd00c06a800 msgr2=0x7fd00c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 -- 172.21.15.53:0/2196768239 shutdown_connections 2024-01-16T15:09:27.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.222+0000 7fd011600640 1 -- 172.21.15.53:0/2196768239 wait complete. 2024-01-16T15:09:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.223+0000 7fd011600640 1 Processor -- start 2024-01-16T15:09:27.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.223+0000 7fd011600640 1 -- start start 2024-01-16T15:09:27.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd00c0cf7d0 0x7fd00c0afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c13f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd00c15e1e0 0x7fd00c1593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c1f3fb0 con 0x7fd00c143ae0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c1f4290 con 0x7fd00c15e1e0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd00c1f4570 con 0x7fd00c0cf7d0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c13f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c13f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48838/0 (socket says 172.21.15.53:48838) 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 -- 172.21.15.53:0/3138241419 learned_addr learned my addr 172.21.15.53:0/3138241419 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00aa00640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd00c0cf7d0 0x7fd00c0afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00b400640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd00c15e1e0 0x7fd00c1593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 -- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd00c0cf7d0 msgr2=0x7fd00c0afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd00c0cf7d0 0x7fd00c0afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 -- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd00c15e1e0 msgr2=0x7fd00c1593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd00c15e1e0 0x7fd00c1593d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 -- 172.21.15.53:0/3138241419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc009b30 con 0x7fd00c143ae0 2024-01-16T15:09:27.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd00a000640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c13f9a0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b410 tx=0x7fcff400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4011830 con 0x7fd00c143ae0 2024-01-16T15:09:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcff40119b0 con 0x7fd00c143ae0 2024-01-16T15:09:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcff4011d60 con 0x7fd00c143ae0 2024-01-16T15:09:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.226+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd00c1f4850 con 0x7fd00c143ae0 2024-01-16T15:09:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.227+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd00c0b0220 con 0x7fd00c143ae0 2024-01-16T15:09:27.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.228+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd00c0b2bf0 con 0x7fd00c143ae0 2024-01-16T15:09:27.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.230+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fcff401a400 con 0x7fd00c143ae0 2024-01-16T15:09:27.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.230+0000 7fd002a00640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcfe406bd70 0x7fcfe406e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:27.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.230+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcff40886c0 con 0x7fd00c143ae0 2024-01-16T15:09:27.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.230+0000 7fd00aa00640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcfe406bd70 0x7fcfe406e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:27.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.232+0000 7fd00aa00640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcfe406bd70 0x7fcfe406e230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcffc035000 tx=0x7fcffc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:27.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.235+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd00c0b2bf0 con 0x7fd00c143ae0 2024-01-16T15:09:27.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.468+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7fd00c15e670 con 0x7fcfe406bd70 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.490+0000 7fd002a00640 1 -- 172.21.15.53:0/3138241419 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fcffc034470 con 0x7fcfe406bd70 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:09:27.468 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:09:27.469 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:09:27.469 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:09:27.469 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:09:27.469 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:27.469 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:27.469 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:27.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcfe406bd70 msgr2=0x7fcfe406e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcfe406bd70 0x7fcfe406e230 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fcffc035000 tx=0x7fcffc030000 comp rx=0 tx=0).stop 2024-01-16T15:09:27.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 msgr2=0x7fd00c13f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:27.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c13f9a0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fcff400b410 tx=0x7fcff400b8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 shutdown_connections 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcfe406bd70 0x7fcfe406e230 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd00c15e1e0 0x7fd00c1593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd00c143ae0 0x7fd00c13f9a0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 --2- 172.21.15.53:0/3138241419 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd00c0cf7d0 0x7fd00c0afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 >> 172.21.15.53:0/3138241419 conn(0x7fd00c06a800 msgr2=0x7fd00c15b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.493+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 shutdown_connections 2024-01-16T15:09:27.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:27.494+0000 7fd011600640 1 -- 172.21.15.53:0/3138241419 wait complete. 2024-01-16T15:09:27.494 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:09:28.495 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54144 2024-01-16T15:09:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.760+0000 7fb317200640 1 Processor -- start 2024-01-16T15:09:28.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.760+0000 7fb317200640 1 -- start start 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb317200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb318070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb317200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb31815bf10 0x7fb318143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb317200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31815f4e0 con 0x7fb3180adf20 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb317200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb31815f970 con 0x7fb31815bf10 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb317200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb31815c3c0 con 0x7fb3180ccb50 2024-01-16T15:09:28.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb315e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb3180adf20 0x7fb3180cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44210/0 (socket says 172.21.15.53:44210) 2024-01-16T15:09:28.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.762+0000 7fb315e00640 1 -- 172.21.15.53:0/1458827642 learned_addr learned my addr 172.21.15.53:0/1458827642 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb30fe00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb318070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb316800640 1 --2- 172.21.15.53:0/1458827642 >> v2:172.21.15.146:3300/0 conn(0x7fb31815bf10 0x7fb318143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474092039 0 0) 0x7fb31815f4e0 con 0x7fb3180adf20 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2f8003520 con 0x7fb3180adf20 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3594712797 0 0) 0x7fb2f8003520 con 0x7fb3180adf20 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 msgr2=0x7fb318070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb318070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 >> v2:172.21.15.146:3300/0 conn(0x7fb31815bf10 msgr2=0x7fb318143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 --2- 172.21.15.53:0/1458827642 >> v2:172.21.15.146:3300/0 conn(0x7fb31815bf10 0x7fb318143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb318071130 con 0x7fb3180adf20 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb308001670 con 0x7fb3180adf20 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 >> v1:172.21.15.53:6789/0 conn(0x7fb3180adf20 legacy=0x7fb3180cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb2f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f80047c0 0x7fb2f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.763+0000 7fb315400640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb2f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31815f970 con 0x7fb2f8003970 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb31815c3c0 con 0x7fb2f80047c0 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f800b9d0 con 0x7fb3180ccb50 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315e00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb2f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb30fe00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f80047c0 0x7fb2f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb316800640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb2f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315e00640 1 -- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 msgr2=0x7fb2f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315e00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb2f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315e00640 1 -- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f80047c0 msgr2=0x7fb2f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315e00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f80047c0 0x7fb2f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb315e00640 1 -- 172.21.15.53:0/1458827642 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f8003520 con 0x7fb2f8003970 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb316800640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3180ccb50 0x7fb2f8005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.764+0000 7fb30fe00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2f80047c0 0x7fb2f8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.765+0000 7fb315e00640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb2f8003d90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb30800c110 tx=0x7fb2f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4dc84837f915567 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308005ad0 con 0x7fb2f8003970 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb308005c50 con 0x7fb2f8003970 2024-01-16T15:09:28.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb315400640 1 -- 172.21.15.53:0/1458827642 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30802c7e0 con 0x7fb2f8003970 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb317200640 1 -- 172.21.15.53:0/1458827642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 msgr2=0x7fb2f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb317200640 1 --2- 172.21.15.53:0/1458827642 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb2f8003d90 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fb30800c110 tx=0x7fb2f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb316800640 1 -- 172.21.15.53:0/1458827642 reap_dead start 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb317200640 1 -- 172.21.15.53:0/1458827642 shutdown_connections 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.766+0000 7fb317200640 1 -- 172.21.15.53:0/1458827642 >> 172.21.15.53:0/1458827642 conn(0x7fb31806a7d0 msgr2=0x7fb31806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:28.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.767+0000 7fb317200640 1 -- 172.21.15.53:0/1458827642 shutdown_connections 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.767+0000 7fb317200640 1 -- 172.21.15.53:0/1458827642 wait complete. 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.768+0000 7fb317200640 1 Processor -- start 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.769+0000 7fb317200640 1 -- start start 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.769+0000 7fb317200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb318142240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.769+0000 7fb317200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3181f6070 0x7fb3180b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb317200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3180ab510 0x7fb31806caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb317200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181f6550 con 0x7fb2f8003970 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb317200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181f6830 con 0x7fb3180ab510 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb317200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3181f6b10 con 0x7fb3181f6070 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb315e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb318142240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb315e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb318142240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48854/0 (socket says 172.21.15.53:48854) 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb315e00640 1 -- 172.21.15.53:0/4137768614 learned_addr learned my addr 172.21.15.53:0/4137768614 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:28.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb316800640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3180ab510 0x7fb31806caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.770+0000 7fb30fe00640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3181f6070 0x7fb3180b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb315e00640 1 -- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3181f6070 msgr2=0x7fb3180b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb315e00640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3181f6070 0x7fb3180b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb315e00640 1 -- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3180ab510 msgr2=0x7fb31806caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb315e00640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3180ab510 0x7fb31806caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb315e00640 1 -- 172.21.15.53:0/4137768614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3181f6df0 con 0x7fb2f8003970 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb315e00640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb318142240 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb30802c380 tx=0x7fb31815b490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb30803c040 con 0x7fb2f8003970 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb30802cbf0 con 0x7fb2f8003970 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb308001ba0 con 0x7fb2f8003970 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb318142780 con 0x7fb2f8003970 2024-01-16T15:09:28.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.771+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3180ab9a0 con 0x7fb2f8003970 2024-01-16T15:09:28.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.772+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb318067450 con 0x7fb2f8003970 2024-01-16T15:09:28.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.773+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 23) v1 ==== 90508+0+0 (secure 0 0 0) 0x7fb308032070 con 0x7fb2f8003970 2024-01-16T15:09:28.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.774+0000 7fb30e000640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb2f006bd70 0x7fb2f006e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:28.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.775+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3080a9360 con 0x7fb2f8003970 2024-01-16T15:09:28.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.775+0000 7fb30fe00640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb2f006bd70 0x7fb2f006e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:28.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.776+0000 7fb30fe00640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb2f006bd70 0x7fb2f006e230 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb31806dfa0 tx=0x7fb300008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:28.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:28.779+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb308075a50 con 0x7fb2f8003970 2024-01-16T15:09:28.939 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:28.962+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:28.939 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:28.962+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:28.939 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:28.962+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:28.939 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:28.962+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:28.939 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:28.962+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:28.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:29.020+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7fb31815f170 con 0x7fb2f006bd70 2024-01-16T15:09:31.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:31.496+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 24) v1 ==== 90559+0+0 (secure 0 0 0) 0x7fb308075e20 con 0x7fb2f8003970 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.939+0000 7fb30e000640 1 -- 172.21.15.53:0/4137768614 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fb3000097d0 con 0x7fb2f006bd70 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:09:33.917 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:09:33.918 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:33.918 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:33.918 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.943+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb2f006bd70 msgr2=0x7fb2f006e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.943+0000 7fb317200640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb2f006bd70 0x7fb2f006e230 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb31806dfa0 tx=0x7fb300008040 comp rx=0 tx=0).stop 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.943+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 msgr2=0x7fb318142240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.943+0000 7fb317200640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb318142240 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7fb30802c380 tx=0x7fb31815b490 comp rx=0 tx=0).stop 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.943+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 shutdown_connections 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.943+0000 7fb317200640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb2f006bd70 0x7fb2f006e230 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.944+0000 7fb317200640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3180ab510 0x7fb31806caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.944+0000 7fb317200640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3181f6070 0x7fb3180b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.944+0000 7fb317200640 1 --2- 172.21.15.53:0/4137768614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2f8003970 0x7fb318142240 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.944+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 >> 172.21.15.53:0/4137768614 conn(0x7fb31806a7d0 msgr2=0x7fb318071690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:33.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.944+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 shutdown_connections 2024-01-16T15:09:33.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:33.944+0000 7fb317200640 1 -- 172.21.15.53:0/4137768614 wait complete. 2024-01-16T15:09:33.935 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:33.958+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:33.935 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:33.958+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:33.935 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:33.958+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:33.935 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:33.958+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:33.935 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:09:33.958+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:09:33.943 INFO:tasks.cephfs.test_volumes:current clone state = pending 2024-01-16T15:09:34.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54144 2024-01-16T15:09:35.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.307+0000 7f0c8a000640 1 Processor -- start 2024-01-16T15:09:35.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.307+0000 7f0c8a000640 1 -- start start 2024-01-16T15:09:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.311+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c840cfde0 0x7f0c840b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.311+0000 7f0c8a000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840d1c60 con 0x7f0c840cf9a0 2024-01-16T15:09:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.311+0000 7f0c8a000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c840b2730 con 0x7f0c8415d4b0 2024-01-16T15:09:35.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.311+0000 7f0c8a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c840d1ec0 con 0x7f0c840cfde0 2024-01-16T15:09:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.312+0000 7f0c83e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0c840cf9a0 0x7f0c840b31d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41138/0 (socket says 172.21.15.53:41138) 2024-01-16T15:09:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.312+0000 7f0c83e00640 1 -- 172.21.15.53:0/3850444773 learned_addr learned my addr 172.21.15.53:0/3850444773 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.312+0000 7f0c89600640 1 --2- 172.21.15.53:0/3850444773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c840cfde0 0x7f0c840b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.312+0000 7f0c83400640 1 -- 172.21.15.53:0/3850444773 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832059721 0 0) 0x7f0c840d1c60 con 0x7f0c840cf9a0 2024-01-16T15:09:35.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.312+0000 7f0c83400640 1 -- 172.21.15.53:0/3850444773 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c64003520 con 0x7f0c840cf9a0 2024-01-16T15:09:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.313+0000 7f0c83400640 1 -- 172.21.15.53:0/3850444773 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3310043172 0 0) 0x7f0c64003520 con 0x7f0c840cf9a0 2024-01-16T15:09:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.313+0000 7f0c89600640 1 -- 172.21.15.53:0/3850444773 >> v1:172.21.15.146:6789/0 conn(0x7f0c8415d4b0 legacy=0x7f0c840d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.313+0000 7f0c89600640 1 -- 172.21.15.53:0/3850444773 >> v1:172.21.15.53:6789/0 conn(0x7f0c840cf9a0 legacy=0x7f0c840b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.313+0000 7f0c89600640 1 -- 172.21.15.53:0/3850444773 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c840b29d0 con 0x7f0c840cfde0 2024-01-16T15:09:35.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.315+0000 7f0c89600640 1 --2- 172.21.15.53:0/3850444773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c840cfde0 0x7f0c840b38e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009dc0 tx=0x7f0c7c02bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb8144bcd98d6328 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.317+0000 7f0c83400640 1 -- 172.21.15.53:0/3850444773 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c001670 con 0x7f0c840cfde0 2024-01-16T15:09:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.317+0000 7f0c83400640 1 -- 172.21.15.53:0/3850444773 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c02c910 con 0x7f0c840cfde0 2024-01-16T15:09:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.317+0000 7f0c83400640 1 -- 172.21.15.53:0/3850444773 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c02cca0 con 0x7f0c840cfde0 2024-01-16T15:09:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 -- 172.21.15.53:0/3850444773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c840cfde0 msgr2=0x7f0c840b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 --2- 172.21.15.53:0/3850444773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c840cfde0 0x7f0c840b38e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c009dc0 tx=0x7f0c7c02bde0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 -- 172.21.15.53:0/3850444773 shutdown_connections 2024-01-16T15:09:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 --2- 172.21.15.53:0/3850444773 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c840cfde0 0x7f0c840b38e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 -- 172.21.15.53:0/3850444773 >> 172.21.15.53:0/3850444773 conn(0x7f0c840d3000 msgr2=0x7f0c840d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 -- 172.21.15.53:0/3850444773 shutdown_connections 2024-01-16T15:09:35.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.318+0000 7f0c8a000640 1 -- 172.21.15.53:0/3850444773 wait complete. 2024-01-16T15:09:35.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.321+0000 7f0c8a000640 1 Processor -- start 2024-01-16T15:09:35.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.322+0000 7f0c8a000640 1 -- start start 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.324+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c840cf9a0 0x7f0c8414f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.324+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 0x7f0c841ecf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.324+0000 7f0c8a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c8415d4b0 0x7f0c841ed490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.324+0000 7f0c8a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8414fa10 con 0x7f0c840cfde0 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.324+0000 7f0c8a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8409e950 con 0x7f0c840cf9a0 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.324+0000 7f0c8a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c8409ebf0 con 0x7f0c8415d4b0 2024-01-16T15:09:35.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.325+0000 7f0c83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 0x7f0c841ecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.325+0000 7f0c83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 0x7f0c841ecf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33524/0 (socket says 172.21.15.53:33524) 2024-01-16T15:09:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.325+0000 7f0c83e00640 1 -- 172.21.15.53:0/2981282990 learned_addr learned my addr 172.21.15.53:0/2981282990 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.325+0000 7f0c83e00640 1 -- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c8415d4b0 msgr2=0x7f0c841ed490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.326+0000 7f0c83e00640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c8415d4b0 0x7f0c841ed490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.326+0000 7f0c83e00640 1 -- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c840cf9a0 msgr2=0x7f0c8414f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.326+0000 7f0c83e00640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c840cf9a0 0x7f0c8414f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.326+0000 7f0c83e00640 1 -- 172.21.15.53:0/2981282990 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c009a70 con 0x7f0c840cfde0 2024-01-16T15:09:35.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.326+0000 7f0c83e00640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 0x7f0c841ecf50 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c760 tx=0x7f0c6c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.327+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c005ad0 con 0x7f0c840cfde0 2024-01-16T15:09:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.327+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c005c30 con 0x7f0c840cfde0 2024-01-16T15:09:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.327+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c007070 con 0x7f0c840cfde0 2024-01-16T15:09:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.327+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c8409eed0 con 0x7f0c840cfde0 2024-01-16T15:09:35.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.327+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c8409f490 con 0x7f0c840cfde0 2024-01-16T15:09:35.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.329+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c840b1380 con 0x7f0c840cfde0 2024-01-16T15:09:35.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.330+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f0c6c0071d0 con 0x7f0c840cfde0 2024-01-16T15:09:35.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.331+0000 7f0c80c00640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c5c06be90 0x7f0c5c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:35.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.332+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0c6c088480 con 0x7f0c840cfde0 2024-01-16T15:09:35.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.332+0000 7f0c88c00640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c5c06be90 0x7f0c5c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:35.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.337+0000 7f0c88c00640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c5c06be90 0x7f0c5c06e350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0c74000f60 tx=0x7f0c740079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:35.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.339+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0c6c054c00 con 0x7f0c840cfde0 2024-01-16T15:09:35.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.703+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c840d21a0 con 0x7f0c5c06be90 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.827+0000 7f0c80c00640 1 -- 172.21.15.53:0/2981282990 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+344 (secure 0 0 0) 0x7f0c74001210 con 0x7f0c5c06be90 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "in-progress", 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:35.804 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_report": { 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: "percentage completed": 0.0, 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: "amount copied": "0 B/3 GB", 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: "regfiles copied": "0/3", 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: "subdirs copied": "0/1", 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: "symlinks copied": "0/0" 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:35.805 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c5c06be90 msgr2=0x7f0c5c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c5c06be90 0x7f0c5c06e350 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f0c74000f60 tx=0x7f0c740079b0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 msgr2=0x7f0c841ecf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 0x7f0c841ecf50 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c00c760 tx=0x7f0c6c00cc20 comp rx=0 tx=0).stop 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 shutdown_connections 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0c5c06be90 0x7f0c5c06e350 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c8415d4b0 0x7f0c841ed490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.831+0000 7f0c8a000640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c840cfde0 0x7f0c841ecf50 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.832+0000 7f0c8a000640 1 --2- 172.21.15.53:0/2981282990 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c840cf9a0 0x7f0c8414f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.832+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 >> 172.21.15.53:0/2981282990 conn(0x7f0c840d3000 msgr2=0x7f0c840cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.832+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 shutdown_connections 2024-01-16T15:09:35.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:35.832+0000 7f0c8a000640 1 -- 172.21.15.53:0/2981282990 wait complete. 2024-01-16T15:09:35.833 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:09:35.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54147 2024-01-16T15:09:36.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.145+0000 7fbbc4e00640 1 Processor -- start 2024-01-16T15:09:36.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.145+0000 7fbbc4e00640 1 -- start start 2024-01-16T15:09:36.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.147+0000 7fbbc4e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbbc01608b0 0x7fbbc00d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.147+0000 7fbbc4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00c6c00 0x7fbbc00cfb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.147+0000 7fbbc4e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc00cf6c0 con 0x7fbbc0160c90 2024-01-16T15:09:36.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.147+0000 7fbbc4e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc00d1450 con 0x7fbbc01608b0 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.147+0000 7fbbc4e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00b4390 con 0x7fbbc00c6c00 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.147+0000 7fbbbf400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbbc01608b0 0x7fbbc00d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.148+0000 7fbbbf400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbbc01608b0 0x7fbbc00d3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33532/0 (socket says 172.21.15.53:33532) 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.148+0000 7fbbbf400640 1 -- 172.21.15.53:0/1178014243 learned_addr learned my addr 172.21.15.53:0/1178014243 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.148+0000 7fbbbf400640 1 -- 172.21.15.53:0/1178014243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00c6c00 msgr2=0x7fbbc00cfb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.148+0000 7fbbbf400640 1 --2- 172.21.15.53:0/1178014243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00c6c00 0x7fbbc00cfb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.148+0000 7fbbbf400640 1 -- 172.21.15.53:0/1178014243 >> v1:172.21.15.146:6789/0 conn(0x7fbbc0160c90 legacy=0x7fbbc00d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:36.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.148+0000 7fbbbf400640 1 -- 172.21.15.53:0/1178014243 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc00ce7e0 con 0x7fbbc01608b0 2024-01-16T15:09:36.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.150+0000 7fbbbf400640 1 --2- 172.21.15.53:0/1178014243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc01608b0 0x7fbbc00d3bd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb002bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f54cf42dc935ed0d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.150+0000 7fbbbe000640 1 -- 172.21.15.53:0/1178014243 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb002c950 con 0x7fbbc01608b0 2024-01-16T15:09:36.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.150+0000 7fbbbe000640 1 -- 172.21.15.53:0/1178014243 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb002cab0 con 0x7fbbc01608b0 2024-01-16T15:09:36.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.150+0000 7fbbbe000640 1 -- 172.21.15.53:0/1178014243 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb002ce40 con 0x7fbbc01608b0 2024-01-16T15:09:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.151+0000 7fbbc4e00640 1 -- 172.21.15.53:0/1178014243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc01608b0 msgr2=0x7fbbc00d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.151+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/1178014243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc01608b0 0x7fbbc00d3bd0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb002bdb0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.151+0000 7fbbc4e00640 1 -- 172.21.15.53:0/1178014243 shutdown_connections 2024-01-16T15:09:36.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.151+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/1178014243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc00c6c00 0x7fbbc00cfb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.151+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/1178014243 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc01608b0 0x7fbbc00d3bd0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.151+0000 7fbbc4e00640 1 -- 172.21.15.53:0/1178014243 >> 172.21.15.53:0/1178014243 conn(0x7fbbc00d30b0 msgr2=0x7fbbc00d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.152+0000 7fbbc4e00640 1 -- 172.21.15.53:0/1178014243 shutdown_connections 2024-01-16T15:09:36.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.152+0000 7fbbc4e00640 1 -- 172.21.15.53:0/1178014243 wait complete. 2024-01-16T15:09:36.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.153+0000 7fbbc4e00640 1 Processor -- start 2024-01-16T15:09:36.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 -- start start 2024-01-16T15:09:36.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 0x7fbbc01528f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc01608b0 0x7fbbc00cb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0160c90 0x7fbbc01536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00ccb20 con 0x7fbbc00c6c00 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00cbf10 con 0x7fbbc01608b0 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.154+0000 7fbbc4e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc00cc1f0 con 0x7fbbc0160c90 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 0x7fbbc01528f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 0x7fbbc01528f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33546/0 (socket says 172.21.15.53:33546) 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 -- 172.21.15.53:0/640537117 learned_addr learned my addr 172.21.15.53:0/640537117 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbea00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc01608b0 0x7fbbc00cb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 -- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0160c90 msgr2=0x7fbbc01536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0160c90 0x7fbbc01536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 -- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc01608b0 msgr2=0x7fbbc00cb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc01608b0 0x7fbbc00cb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.155+0000 7fbbbf400640 1 -- 172.21.15.53:0/640537117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb0009b30 con 0x7fbbc00c6c00 2024-01-16T15:09:36.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.156+0000 7fbbbf400640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 0x7fbbc01528f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.156+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb003e450 con 0x7fbbc00c6c00 2024-01-16T15:09:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.156+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb003e5b0 con 0x7fbbc00c6c00 2024-01-16T15:09:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.156+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb003e960 con 0x7fbbc00c6c00 2024-01-16T15:09:36.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.156+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc00cc4d0 con 0x7fbbc00c6c00 2024-01-16T15:09:36.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.157+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc00cbb00 con 0x7fbbc00c6c00 2024-01-16T15:09:36.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.158+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc00cbcc0 con 0x7fbbc00c6c00 2024-01-16T15:09:36.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.160+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fbbb003eac0 con 0x7fbbc00c6c00 2024-01-16T15:09:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.162+0000 7fbbb7400640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806be90 0x7fbb9806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.162+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbbb00ab4c0 con 0x7fbbc00c6c00 2024-01-16T15:09:36.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.171+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbbb00e0e90 con 0x7fbbc00c6c00 2024-01-16T15:09:36.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.186+0000 7fbbbea00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806be90 0x7fbb9806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.198+0000 7fbbbea00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806be90 0x7fbb9806e350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00cf400 tx=0x7fbba8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.540+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54147", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbc00b4000 con 0x7fbb9806be90 2024-01-16T15:09:36.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.583+0000 7fbbb7400640 1 -- 172.21.15.53:0/640537117 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7fbbc00b4000 con 0x7fbb9806be90 2024-01-16T15:09:36.560 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:36.561 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:36.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806be90 msgr2=0x7fbb9806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806be90 0x7fbb9806e350 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00cf400 tx=0x7fbba8009040 comp rx=0 tx=0).stop 2024-01-16T15:09:36.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 msgr2=0x7fbbc01528f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 0x7fbbc01528f0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb0004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:36.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 shutdown_connections 2024-01-16T15:09:36.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb9806be90 0x7fbb9806e350 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbbc0160c90 0x7fbbc01536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbbc01608b0 0x7fbbc00cb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 --2- 172.21.15.53:0/640537117 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbbc00c6c00 0x7fbbc01528f0 secure :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7fbbb0009e50 tx=0x7fbbb0004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:36.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.587+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 >> 172.21.15.53:0/640537117 conn(0x7fbbc00d30b0 msgr2=0x7fbbc00b4af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.588+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 shutdown_connections 2024-01-16T15:09:36.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.589+0000 7fbbc4e00640 1 -- 172.21.15.53:0/640537117 wait complete. 2024-01-16T15:09:36.595 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:09:36.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.883+0000 7f4694800640 1 Processor -- start 2024-01-16T15:09:36.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.883+0000 7f4694800640 1 -- start start 2024-01-16T15:09:36.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.885+0000 7f4694800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15d320 0x7f468c06cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.885+0000 7f4694800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c153d30 con 0x7f468c15ce90 2024-01-16T15:09:36.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.885+0000 7f4694800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c15d790 con 0x7f468c153950 2024-01-16T15:09:36.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.885+0000 7f4694800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0b2b70 con 0x7f468c15d320 2024-01-16T15:09:36.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.886+0000 7f4691400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f468c15ce90 0x7f468c06c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41146/0 (socket says 172.21.15.53:41146) 2024-01-16T15:09:36.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.886+0000 7f4691400640 1 -- 172.21.15.53:0/4052448215 learned_addr learned my addr 172.21.15.53:0/4052448215 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:36.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.886+0000 7f4692800640 1 --2- 172.21.15.53:0/4052448215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15d320 0x7f468c06cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.887+0000 7f4690a00640 1 -- 172.21.15.53:0/4052448215 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676166961 0 0) 0x7f468c153d30 con 0x7f468c15ce90 2024-01-16T15:09:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.887+0000 7f4690a00640 1 -- 172.21.15.53:0/4052448215 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f466c003520 con 0x7f468c15ce90 2024-01-16T15:09:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.887+0000 7f4692800640 1 -- 172.21.15.53:0/4052448215 >> v1:172.21.15.146:6789/0 conn(0x7f468c153950 legacy=0x7f468c0cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.887+0000 7f4692800640 1 -- 172.21.15.53:0/4052448215 >> v1:172.21.15.53:6789/0 conn(0x7f468c15ce90 legacy=0x7f468c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.888+0000 7f4692800640 1 -- 172.21.15.53:0/4052448215 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c0b2e50 con 0x7f468c15d320 2024-01-16T15:09:36.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.889+0000 7f4692800640 1 --2- 172.21.15.53:0/4052448215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15d320 0x7f468c06cb80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4684009dc0 tx=0x7f468402bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=276366f14069c05d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.889+0000 7f4690a00640 1 -- 172.21.15.53:0/4052448215 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f468402ccc0 con 0x7f468c15d320 2024-01-16T15:09:36.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.889+0000 7f4690a00640 1 -- 172.21.15.53:0/4052448215 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468402ce20 con 0x7f468c15d320 2024-01-16T15:09:36.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.889+0000 7f4690a00640 1 -- 172.21.15.53:0/4052448215 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684034660 con 0x7f468c15d320 2024-01-16T15:09:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.890+0000 7f4694800640 1 -- 172.21.15.53:0/4052448215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15d320 msgr2=0x7f468c06cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.890+0000 7f4694800640 1 --2- 172.21.15.53:0/4052448215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15d320 0x7f468c06cb80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4684009dc0 tx=0x7f468402bf10 comp rx=0 tx=0).stop 2024-01-16T15:09:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.891+0000 7f4694800640 1 -- 172.21.15.53:0/4052448215 shutdown_connections 2024-01-16T15:09:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.891+0000 7f4694800640 1 --2- 172.21.15.53:0/4052448215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15d320 0x7f468c06cb80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.891+0000 7f4694800640 1 -- 172.21.15.53:0/4052448215 >> 172.21.15.53:0/4052448215 conn(0x7f468c0b0be0 msgr2=0x7f468c0b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.891+0000 7f4694800640 1 -- 172.21.15.53:0/4052448215 shutdown_connections 2024-01-16T15:09:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.891+0000 7f4694800640 1 -- 172.21.15.53:0/4052448215 wait complete. 2024-01-16T15:09:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.893+0000 7f4694800640 1 Processor -- start 2024-01-16T15:09:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.893+0000 7f4694800640 1 -- start start 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4694800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f468c153950 0x7f468c14c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4694800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15ce90 0x7f468c1490d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4694800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 0x7f468c136c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4694800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c1568d0 con 0x7f468c15d320 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4694800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c156440 con 0x7f468c153950 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4694800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c149610 con 0x7f468c15ce90 2024-01-16T15:09:36.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4692800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 0x7f468c136c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4692800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 0x7f468c136c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33548/0 (socket says 172.21.15.53:33548) 2024-01-16T15:09:36.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.895+0000 7f4692800640 1 -- 172.21.15.53:0/2005221454 learned_addr learned my addr 172.21.15.53:0/2005221454 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.896+0000 7f4692800640 1 -- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15ce90 msgr2=0x7f468c1490d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.896+0000 7f4692800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15ce90 0x7f468c1490d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.896+0000 7f4692800640 1 -- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f468c153950 msgr2=0x7f468c14c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.896+0000 7f4692800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f468c153950 0x7f468c14c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.896+0000 7f4692800640 1 -- 172.21.15.53:0/2005221454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684009a70 con 0x7f468c15d320 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.896+0000 7f4692800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 0x7f468c136c10 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f4684037040 tx=0x7f4684004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.897+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684040650 con 0x7f468c15d320 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.897+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46840407b0 con 0x7f468c15d320 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.897+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4684040b60 con 0x7f468c15d320 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.897+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c149870 con 0x7f468c15d320 2024-01-16T15:09:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.897+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c155df0 con 0x7f468c15d320 2024-01-16T15:09:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.898+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c0b1fa0 con 0x7f468c15d320 2024-01-16T15:09:36.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.900+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f4684040cc0 con 0x7f468c15d320 2024-01-16T15:09:36.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.901+0000 7f468a000640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f466406be90 0x7f466406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:36.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.901+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f46840ab2a0 con 0x7f468c15d320 2024-01-16T15:09:36.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.902+0000 7f4691e00640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f466406be90 0x7f466406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:36.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.907+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4684077d00 con 0x7f468c15d320 2024-01-16T15:09:36.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:36.911+0000 7f4691e00640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f466406be90 0x7f466406e350 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f467c0014b0 tx=0x7f467c007990 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:37.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.291+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f468c0675d0 con 0x7f468c15d320 2024-01-16T15:09:37.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.291+0000 7f468a000640 1 -- 172.21.15.53:0/2005221454 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2347 (secure 0 0 0) 0x7f4684044940 con 0x7f468c15d320 2024-01-16T15:09:37.268 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:37.268 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:37.268 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:37.268 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:37.268 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:37.269 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 604, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:37.270 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 921, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3713213020, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 7505874944, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 765588238336, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:37.271 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 47154821, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 40892048, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 12, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 17 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:37.272 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:37.273 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:37.274 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:37.275 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:37.275 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:37.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.293+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f466406be90 msgr2=0x7f466406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.293+0000 7f4694800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f466406be90 0x7f466406e350 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f467c0014b0 tx=0x7f467c007990 comp rx=0 tx=0).stop 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.293+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 msgr2=0x7f468c136c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.293+0000 7f4694800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 0x7f468c136c10 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f4684037040 tx=0x7f4684004ab0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 shutdown_connections 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f466406be90 0x7f466406e350 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f468c15d320 0x7f468c136c10 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f468c15ce90 0x7f468c1490d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 --2- 172.21.15.53:0/2005221454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f468c153950 0x7f468c14c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 >> 172.21.15.53:0/2005221454 conn(0x7f468c0b0be0 msgr2=0x7f468c157910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 shutdown_connections 2024-01-16T15:09:37.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:37.294+0000 7f4694800640 1 -- 172.21.15.53:0/2005221454 wait complete. 2024-01-16T15:09:38.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:38.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.545+0000 7f8175600640 1 Processor -- start 2024-01-16T15:09:38.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.545+0000 7f8175600640 1 -- start start 2024-01-16T15:09:38.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.546+0000 7f8175600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f817013e360 0x7f81700cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.546+0000 7f8175600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 0x7f81700cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.546+0000 7f8175600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f817015c5f0 con 0x7f817013df80 2024-01-16T15:09:38.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.546+0000 7f8175600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f817006c470 con 0x7f817013e360 2024-01-16T15:09:38.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.546+0000 7f8175600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f817006c750 con 0x7f817015f9c0 2024-01-16T15:09:38.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.547+0000 7f816ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f817013df80 0x7f817015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41148/0 (socket says 172.21.15.53:41148) 2024-01-16T15:09:38.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.547+0000 7f816ea00640 1 -- 172.21.15.53:0/1914239762 learned_addr learned my addr 172.21.15.53:0/1914239762 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.547+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994221342 0 0) 0x7f817015c5f0 con 0x7f817013df80 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.547+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8150003520 con 0x7f817013df80 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.547+0000 7f816f400640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 0x7f81700cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.547+0000 7f8167e00640 1 --2- 172.21.15.53:0/1914239762 >> v2:172.21.15.146:3300/0 conn(0x7f817013e360 0x7f81700cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754306983 0 0) 0x7f8150003520 con 0x7f817013df80 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 msgr2=0x7f81700cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 0x7f81700cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 >> v2:172.21.15.146:3300/0 conn(0x7f817013e360 msgr2=0x7f81700cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 --2- 172.21.15.53:0/1914239762 >> v2:172.21.15.146:3300/0 conn(0x7f817013e360 0x7f81700cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817006ca30 con 0x7f817013df80 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8160001690 con 0x7f817013df80 2024-01-16T15:09:38.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 >> v1:172.21.15.53:6789/0 conn(0x7f817013df80 legacy=0x7f817015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f8150003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81500047c0 0x7f8150004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 0x7f8150005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817006c470 con 0x7f8150003970 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817006c750 con 0x7f81500047c0 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815000ba80 con 0x7f817015f9c0 2024-01-16T15:09:38.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.548+0000 7f816ea00640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f8150003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.549+0000 7f816ea00640 1 -- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 msgr2=0x7f8150005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.549+0000 7f816ea00640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f817015f9c0 0x7f8150005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.549+0000 7f816ea00640 1 -- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81500047c0 msgr2=0x7f8150004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.549+0000 7f816ea00640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f81500047c0 0x7f8150004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.549+0000 7f816ea00640 1 -- 172.21.15.53:0/1914239762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8150003520 con 0x7f8150003970 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.550+0000 7f816ea00640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f8150003d90 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f817015c780 tx=0x7f81500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7254f81e87c40d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.550+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160005ad0 con 0x7f8150003970 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.550+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8160005c50 con 0x7f8150003970 2024-01-16T15:09:38.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.550+0000 7f816e000640 1 -- 172.21.15.53:0/1914239762 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f816002ca00 con 0x7f8150003970 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f8175600640 1 -- 172.21.15.53:0/1914239762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 msgr2=0x7f8150003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f8175600640 1 --2- 172.21.15.53:0/1914239762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f8150003d90 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f817015c780 tx=0x7f81500057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f816f400640 1 -- 172.21.15.53:0/1914239762 reap_dead start 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f8175600640 1 -- 172.21.15.53:0/1914239762 shutdown_connections 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f8175600640 1 -- 172.21.15.53:0/1914239762 >> 172.21.15.53:0/1914239762 conn(0x7f817006a800 msgr2=0x7f817006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f8175600640 1 -- 172.21.15.53:0/1914239762 shutdown_connections 2024-01-16T15:09:38.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.551+0000 7f8175600640 1 -- 172.21.15.53:0/1914239762 wait complete. 2024-01-16T15:09:38.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.553+0000 7f8175600640 1 Processor -- start 2024-01-16T15:09:38.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.553+0000 7f8175600640 1 -- start start 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f8175600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f817015ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f8175600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817013d910 0x7f817013d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f8175600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81700b3290 0x7f81700b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f8175600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81701f4910 con 0x7f8150003970 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f8175600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817014bcf0 con 0x7f817013d910 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f8175600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81700b3c50 con 0x7f81700b3290 2024-01-16T15:09:38.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.554+0000 7f816ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f817015ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f817015ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33566/0 (socket says 172.21.15.53:33566) 2024-01-16T15:09:38.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 -- 172.21.15.53:0/34139930 learned_addr learned my addr 172.21.15.53:0/34139930 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816f400640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81700b3290 0x7f81700b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f8167e00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817013d910 0x7f817013d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 -- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81700b3290 msgr2=0x7f81700b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81700b3290 0x7f81700b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 -- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817013d910 msgr2=0x7f817013d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817013d910 0x7f817013d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 -- 172.21.15.53:0/34139930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817015b040 con 0x7f8150003970 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f8167e00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817013d910 0x7f817013d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:38.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.555+0000 7f816ea00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f817015ab00 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f816000c250 tx=0x7f8160001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.556+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81600016a0 con 0x7f8150003970 2024-01-16T15:09:38.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.556+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8160005e20 con 0x7f8150003970 2024-01-16T15:09:38.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.556+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81701f3d20 con 0x7f8150003970 2024-01-16T15:09:38.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.556+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8160007070 con 0x7f8150003970 2024-01-16T15:09:38.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.556+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f817013d720 con 0x7f8150003970 2024-01-16T15:09:38.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.557+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81700675d0 con 0x7f8150003970 2024-01-16T15:09:38.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.558+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f8160032070 con 0x7f8150003970 2024-01-16T15:09:38.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.559+0000 7f8166a00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f814806be90 0x7f814806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:38.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.559+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f81600a9cb0 con 0x7f8150003970 2024-01-16T15:09:38.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.560+0000 7f8167e00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f814806be90 0x7f814806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:38.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.561+0000 7f8167e00640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f814806be90 0x7f814806e350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8158006a90 tx=0x7f8158008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:38.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.564+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8160076540 con 0x7f8150003970 2024-01-16T15:09:38.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.831+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f817006df00 con 0x7f8150003970 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.832+0000 7f8166a00640 1 -- 172.21.15.53:0/34139930 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2347 (secure 0 0 0) 0x7f8160046940 con 0x7f8150003970 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:38.809 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 605, 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:38.810 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:38.811 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 921, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3713213020, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 7505874944, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 765588238336, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 47154821, 2024-01-16T15:09:38.812 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 40892048, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 12, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 17 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:38.813 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.814 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 0.0% (9s)\n [............................] ", 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0, 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:38.815 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:38.816 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:38.816 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f814806be90 msgr2=0x7f814806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f814806be90 0x7f814806e350 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8158006a90 tx=0x7f8158008040 comp rx=0 tx=0).stop 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 msgr2=0x7f817015ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f817015ab00 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f816000c250 tx=0x7f8160001950 comp rx=0 tx=0).stop 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 shutdown_connections 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f814806be90 0x7f814806e350 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f81700b3290 0x7f81700b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f817013d910 0x7f817013d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 --2- 172.21.15.53:0/34139930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8150003970 0x7f817015ab00 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 >> 172.21.15.53:0/34139930 conn(0x7f817006a800 msgr2=0x7f817015e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.835+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 shutdown_connections 2024-01-16T15:09:38.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:38.836+0000 7f8175600640 1 -- 172.21.15.53:0/34139930 wait complete. 2024-01-16T15:09:39.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:40.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.091+0000 7f9409a00640 1 Processor -- start 2024-01-16T15:09:40.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.092+0000 7f9409a00640 1 -- start start 2024-01-16T15:09:40.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.093+0000 7f9409a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9404150760 0x7f9404072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.093+0000 7f9409a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f940415d300 0x7f940415d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.093+0000 7f9409a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f94040b2bd0 0x7f94040b3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.093+0000 7f9409a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f940415dc80 con 0x7f94040b2bd0 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.093+0000 7f9409a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9404071190 con 0x7f940415d300 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.093+0000 7f9409a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9404071470 con 0x7f9404150760 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f94040b2bd0 0x7f94040b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f94040b2bd0 0x7f94040b3060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33578/0 (socket says 172.21.15.53:33578) 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 -- 172.21.15.53:0/2793624366 learned_addr learned my addr 172.21.15.53:0/2793624366 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f93fbe00640 1 --2- 172.21.15.53:0/2793624366 >> v2:172.21.15.146:3300/0 conn(0x7f940415d300 0x7f940415d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9402a00640 1 --2- 172.21.15.53:0/2793624366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9404150760 0x7f9404072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 -- 172.21.15.53:0/2793624366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9404150760 msgr2=0x7f9404072230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 --2- 172.21.15.53:0/2793624366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9404150760 0x7f9404072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 -- 172.21.15.53:0/2793624366 >> v2:172.21.15.146:3300/0 conn(0x7f940415d300 msgr2=0x7f940415d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:40.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 --2- 172.21.15.53:0/2793624366 >> v2:172.21.15.146:3300/0 conn(0x7f940415d300 0x7f940415d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.094+0000 7f9403400640 1 -- 172.21.15.53:0/2793624366 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9404071750 con 0x7f94040b2bd0 2024-01-16T15:09:40.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.095+0000 7f9403400640 1 --2- 172.21.15.53:0/2793624366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94040b2bd0 0x7f94040b3060 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f940406e0c0 tx=0x7f93fc02e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6fcea99712bb979 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9402000640 1 -- 172.21.15.53:0/2793624366 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc02ea80 con 0x7f94040b2bd0 2024-01-16T15:09:40.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9402000640 1 -- 172.21.15.53:0/2793624366 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc02ebe0 con 0x7f94040b2bd0 2024-01-16T15:09:40.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9402000640 1 -- 172.21.15.53:0/2793624366 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc009600 con 0x7f94040b2bd0 2024-01-16T15:09:40.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 -- 172.21.15.53:0/2793624366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94040b2bd0 msgr2=0x7f94040b3060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 --2- 172.21.15.53:0/2793624366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94040b2bd0 0x7f94040b3060 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f940406e0c0 tx=0x7f93fc02e0e0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 -- 172.21.15.53:0/2793624366 shutdown_connections 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 --2- 172.21.15.53:0/2793624366 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f94040b2bd0 0x7f94040b3060 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 --2- 172.21.15.53:0/2793624366 >> v2:172.21.15.146:3300/0 conn(0x7f940415d300 0x7f940415d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 --2- 172.21.15.53:0/2793624366 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9404150760 0x7f9404072230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.096+0000 7f9409a00640 1 -- 172.21.15.53:0/2793624366 >> 172.21.15.53:0/2793624366 conn(0x7f940406a800 msgr2=0x7f940406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.097+0000 7f9409a00640 1 -- 172.21.15.53:0/2793624366 shutdown_connections 2024-01-16T15:09:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.097+0000 7f9409a00640 1 -- 172.21.15.53:0/2793624366 wait complete. 2024-01-16T15:09:40.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.098+0000 7f9409a00640 1 Processor -- start 2024-01-16T15:09:40.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.098+0000 7f9409a00640 1 -- start start 2024-01-16T15:09:40.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.099+0000 7f9409a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94040b2bd0 0x7f940414ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.099+0000 7f9409a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9404150760 0x7f94041672e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.099+0000 7f9409a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 0x7f940414f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.099+0000 7f9409a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f940414f300 con 0x7f940415d300 2024-01-16T15:09:40.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.099+0000 7f9409a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94041654f0 con 0x7f9404150760 2024-01-16T15:09:40.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.099+0000 7f9409a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9404165790 con 0x7f94040b2bd0 2024-01-16T15:09:40.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 0x7f940414f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 0x7f940414f630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33586/0 (socket says 172.21.15.53:33586) 2024-01-16T15:09:40.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 -- 172.21.15.53:0/4253527550 learned_addr learned my addr 172.21.15.53:0/4253527550 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 -- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94040b2bd0 msgr2=0x7f940414ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94040b2bd0 0x7f940414ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 -- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9404150760 msgr2=0x7f94041672e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9404150760 0x7f94041672e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.100+0000 7f9403400640 1 -- 172.21.15.53:0/4253527550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93fc00bd40 con 0x7f940415d300 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.101+0000 7f9403400640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 0x7f940414f630 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f93fc038040 tx=0x7f93fc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.101+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc02ea80 con 0x7f940415d300 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.101+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93fc036800 con 0x7f940415d300 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.101+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93fc036bb0 con 0x7f940415d300 2024-01-16T15:09:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.101+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9404165a70 con 0x7f940415d300 2024-01-16T15:09:40.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.101+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9404167d30 con 0x7f940415d300 2024-01-16T15:09:40.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.102+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9404165f90 con 0x7f940415d300 2024-01-16T15:09:40.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.103+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f93fc02ed50 con 0x7f940415d300 2024-01-16T15:09:40.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.104+0000 7f93faa00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93dc06be90 0x7f93dc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:40.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.104+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f93fc0ab720 con 0x7f940415d300 2024-01-16T15:09:40.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.105+0000 7f9402a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93dc06be90 0x7f93dc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:40.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.106+0000 7f9402a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93dc06be90 0x7f93dc06e350 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006b10 tx=0x7f93f4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:40.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.108+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9404165f90 con 0x7f940415d300 2024-01-16T15:09:40.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.383+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9404165f90 con 0x7f940415d300 2024-01-16T15:09:40.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.384+0000 7f93faa00640 1 -- 172.21.15.53:0/4253527550 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2423 (secure 0 0 0) 0x7f9404165f90 con 0x7f940415d300 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:40.361 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 607, 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:40.362 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1007, 2024-01-16T15:09:40.363 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 4073923265, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 8084684800, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 765009428480, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 83477954, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 70915607, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 20, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 21 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:40.364 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:40.365 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:40.366 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:40.367 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:40.367 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:40.367 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:40.367 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:40.367 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.386+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93dc06be90 msgr2=0x7f93dc06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.386+0000 7f9409a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93dc06be90 0x7f93dc06e350 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f93f4006b10 tx=0x7f93f4008040 comp rx=0 tx=0).stop 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.386+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 msgr2=0x7f940414f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.386+0000 7f9409a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 0x7f940414f630 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f93fc038040 tx=0x7f93fc004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 shutdown_connections 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93dc06be90 0x7f93dc06e350 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f940415d300 0x7f940414f630 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9404150760 0x7f94041672e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 --2- 172.21.15.53:0/4253527550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f94040b2bd0 0x7f940414ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:40.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 >> 172.21.15.53:0/4253527550 conn(0x7f940406a800 msgr2=0x7f94040b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:40.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 shutdown_connections 2024-01-16T15:09:40.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:40.387+0000 7f9409a00640 1 -- 172.21.15.53:0/4253527550 wait complete. 2024-01-16T15:09:41.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:41.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.638+0000 7fc5e5a00640 1 Processor -- start 2024-01-16T15:09:41.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.639+0000 7fc5e5a00640 1 -- start start 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.640+0000 7fc5e5a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc5e00cc990 0x7fc5e006d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.640+0000 7fc5e5a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e0070e50 con 0x7fc5e0143c40 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.640+0000 7fc5e5a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e0144020 con 0x7fc5e00cc500 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.640+0000 7fc5e5a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e00723c0 con 0x7fc5e00cc990 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5dfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc5e0143c40 0x7fc5e0071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41158/0 (socket says 172.21.15.53:41158) 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5dfe00640 1 -- 172.21.15.53:0/1201367759 learned_addr learned my addr 172.21.15.53:0/1201367759 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:41.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5e5000640 1 --2- 172.21.15.53:0/1201367759 >> v2:172.21.15.53:3300/0 conn(0x7fc5e00cc990 0x7fc5e006d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5dea00640 1 -- 172.21.15.53:0/1201367759 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2668217120 0 0) 0x7fc5e0070e50 con 0x7fc5e0143c40 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5e5000640 1 -- 172.21.15.53:0/1201367759 >> v1:172.21.15.146:6789/0 conn(0x7fc5e00cc500 legacy=0x7fc5e006d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5e5000640 1 -- 172.21.15.53:0/1201367759 >> v1:172.21.15.53:6789/0 conn(0x7fc5e0143c40 legacy=0x7fc5e0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.641+0000 7fc5e5000640 1 -- 172.21.15.53:0/1201367759 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0071e00 con 0x7fc5e00cc990 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.643+0000 7fc5e5000640 1 --2- 172.21.15.53:0/1201367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc990 0x7fc5e006d870 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009d90 tx=0x7fc5d802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73ab63bca39608e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.643+0000 7fc5dea00640 1 -- 172.21.15.53:0/1201367759 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d802cc60 con 0x7fc5e00cc990 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.643+0000 7fc5dea00640 1 -- 172.21.15.53:0/1201367759 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d802cdc0 con 0x7fc5e00cc990 2024-01-16T15:09:41.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.643+0000 7fc5dea00640 1 -- 172.21.15.53:0/1201367759 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d8034660 con 0x7fc5e00cc990 2024-01-16T15:09:41.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.643+0000 7fc5e5a00640 1 -- 172.21.15.53:0/1201367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc990 msgr2=0x7fc5e006d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.643+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/1201367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc990 0x7fc5e006d870 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fc5d8009d90 tx=0x7fc5d802beb0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.644+0000 7fc5e5a00640 1 -- 172.21.15.53:0/1201367759 shutdown_connections 2024-01-16T15:09:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.644+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/1201367759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc990 0x7fc5e006d870 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.644+0000 7fc5e5a00640 1 -- 172.21.15.53:0/1201367759 >> 172.21.15.53:0/1201367759 conn(0x7fc5e006a800 msgr2=0x7fc5e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.644+0000 7fc5e5a00640 1 -- 172.21.15.53:0/1201367759 shutdown_connections 2024-01-16T15:09:41.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.644+0000 7fc5e5a00640 1 -- 172.21.15.53:0/1201367759 wait complete. 2024-01-16T15:09:41.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.645+0000 7fc5e5a00640 1 Processor -- start 2024-01-16T15:09:41.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.646+0000 7fc5e5a00640 1 -- start start 2024-01-16T15:09:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.646+0000 7fc5e5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 0x7fc5e013ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5e5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5e00cc990 0x7fc5e00ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5e5a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5e0143c40 0x7fc5e00aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5e5a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00abbe0 con 0x7fc5e00cc500 2024-01-16T15:09:41.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5e5a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e01f52e0 con 0x7fc5e0143c40 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5e5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e00ab520 con 0x7fc5e00cc990 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 0x7fc5e013ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 0x7fc5e013ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33614/0 (socket says 172.21.15.53:33614) 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 -- 172.21.15.53:0/2223602328 learned_addr learned my addr 172.21.15.53:0/2223602328 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 -- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5e00cc990 msgr2=0x7fc5e00ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5e00cc990 0x7fc5e00ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 -- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5e0143c40 msgr2=0x7fc5e00aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5e0143c40 0x7fc5e00aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.647+0000 7fc5dfe00640 1 -- 172.21.15.53:0/2223602328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d8009a70 con 0x7fc5e00cc500 2024-01-16T15:09:41.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.648+0000 7fc5dfe00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 0x7fc5e013ca70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000c3f0 tx=0x7fc5d000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.648+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d00074e0 con 0x7fc5e00cc500 2024-01-16T15:09:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.648+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d0007640 con 0x7fc5e00cc500 2024-01-16T15:09:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.648+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d0004a00 con 0x7fc5e00cc500 2024-01-16T15:09:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.648+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e00ab800 con 0x7fc5e00cc500 2024-01-16T15:09:41.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.648+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e01f55c0 con 0x7fc5e00cc500 2024-01-16T15:09:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.649+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e00abd20 con 0x7fc5e00cc500 2024-01-16T15:09:41.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.650+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc5d0004b60 con 0x7fc5e00cc500 2024-01-16T15:09:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.652+0000 7fc5d7e00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5b806be90 0x7fc5b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.652+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc5d00885d0 con 0x7fc5e00cc500 2024-01-16T15:09:41.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.652+0000 7fc5df400640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5b806be90 0x7fc5b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:41.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.654+0000 7fc5df400640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5b806be90 0x7fc5b806e350 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8000f60 tx=0x7fc5c80079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:41.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.656+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc5e00abd20 con 0x7fc5e00cc500 2024-01-16T15:09:41.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.919+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc5e00abd20 con 0x7fc5e00cc500 2024-01-16T15:09:41.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.920+0000 7fc5d7e00640 1 -- 172.21.15.53:0/2223602328 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fc5e00abd20 con 0x7fc5e00cc500 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.897 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:41.898 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 608, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:41.899 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:41.900 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1222, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 4979927995, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 10148347904, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 762945765376, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 149494731, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 146366260, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 36, 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 40 2024-01-16T15:09:41.901 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:41.902 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:41.903 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:41.904 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:41.905 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5b806be90 msgr2=0x7fc5b806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5b806be90 0x7fc5b806e350 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fc5c8000f60 tx=0x7fc5c80079b0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 msgr2=0x7fc5e013ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 0x7fc5e013ca70 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7fc5d000c3f0 tx=0x7fc5d000c8b0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 shutdown_connections 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc5b806be90 0x7fc5b806e350 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5e0143c40 0x7fc5e00aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5e00cc990 0x7fc5e00ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 --2- 172.21.15.53:0/2223602328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5e00cc500 0x7fc5e013ca70 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.922+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 >> 172.21.15.53:0/2223602328 conn(0x7fc5e006a800 msgr2=0x7fc5e00afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.923+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 shutdown_connections 2024-01-16T15:09:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:41.923+0000 7fc5e5a00640 1 -- 172.21.15.53:0/2223602328 wait complete. 2024-01-16T15:09:42.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:43.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.173+0000 7ff397e00640 1 Processor -- start 2024-01-16T15:09:43.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.173+0000 7ff397e00640 1 -- start start 2024-01-16T15:09:43.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.175+0000 7ff397e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3980b3f30 0x7ff3980d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.175+0000 7ff397e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980d0a10 con 0x7ff398160920 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.175+0000 7ff397e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3980cf080 con 0x7ff3980d0100 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.175+0000 7ff397e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3980c6be0 con 0x7ff3980b3f30 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.175+0000 7ff396a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff398160920 0x7ff3980cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33864/0 (socket says 172.21.15.53:33864) 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.175+0000 7ff396a00640 1 -- 172.21.15.53:0/1492623309 learned_addr learned my addr 172.21.15.53:0/1492623309 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff397400640 1 --2- 172.21.15.53:0/1492623309 >> v2:172.21.15.146:3300/0 conn(0x7ff3980b3f30 0x7ff3980d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572267961 0 0) 0x7ff3980d0a10 con 0x7ff398160920 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff378003520 con 0x7ff398160920 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 695794167 0 0) 0x7ff378003520 con 0x7ff398160920 2024-01-16T15:09:43.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 >> v2:172.21.15.146:3300/0 conn(0x7ff3980b3f30 msgr2=0x7ff3980d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 --2- 172.21.15.53:0/1492623309 >> v2:172.21.15.146:3300/0 conn(0x7ff3980b3f30 0x7ff3980d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 >> v1:172.21.15.146:6789/0 conn(0x7ff3980d0100 legacy=0x7ff3980d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980ce7e0 con 0x7ff398160920 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7ff388001670 con 0x7ff398160920 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 >> v1:172.21.15.53:6789/0 conn(0x7ff398160920 legacy=0x7ff3980cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3780047c0 0x7ff378004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff37800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980c6be0 con 0x7ff378005180 2024-01-16T15:09:43.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37800c080 con 0x7ff3780047c0 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.176+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37800c240 con 0x7ff3780039c0 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff397400640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff37800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff396a00640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff397400640 1 -- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3780039c0 msgr2=0x7ff378003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff397400640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff397400640 1 -- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3780047c0 msgr2=0x7ff378004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff397400640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3780047c0 0x7ff378004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff397400640 1 -- 172.21.15.53:0/1492623309 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff378003520 con 0x7ff378005180 2024-01-16T15:09:43.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.177+0000 7ff396a00640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3780039c0 0x7ff378003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:43.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.178+0000 7ff397400640 1 -- 172.21.15.53:0/1492623309 reap_dead start 2024-01-16T15:09:43.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.178+0000 7ff397400640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff37800b980 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff390009fc0 tx=0x7ff3780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae491400465e87f6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:43.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff39002cc30 con 0x7ff378005180 2024-01-16T15:09:43.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39000f040 con 0x7ff378005180 2024-01-16T15:09:43.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff396000640 1 -- 172.21.15.53:0/1492623309 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff390003990 con 0x7ff378005180 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 -- 172.21.15.53:0/1492623309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 msgr2=0x7ff37800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff37800b980 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7ff390009fc0 tx=0x7ff3780057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 -- 172.21.15.53:0/1492623309 shutdown_connections 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 --2- 172.21.15.53:0/1492623309 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff37800b980 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 -- 172.21.15.53:0/1492623309 >> 172.21.15.53:0/1492623309 conn(0x7ff3980d3090 msgr2=0x7ff3980d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 -- 172.21.15.53:0/1492623309 shutdown_connections 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.179+0000 7ff397e00640 1 -- 172.21.15.53:0/1492623309 wait complete. 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.181+0000 7ff397e00640 1 Processor -- start 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.181+0000 7ff397e00640 1 -- start start 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff397e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff39814e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff397e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3980cbd50 0x7ff3980cc1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff397e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3980ca160 0x7ff3980ca5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff397e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980cc710 con 0x7ff378005180 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff397e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3980b5d80 con 0x7ff3980cbd50 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff397e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff39814ea20 con 0x7ff3980ca160 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.182+0000 7ff396a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff39814e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff39814e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37456/0 (socket says 172.21.15.53:37456) 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 -- 172.21.15.53:0/2368879 learned_addr learned my addr 172.21.15.53:0/2368879 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff397400640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3980ca160 0x7ff3980ca5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 -- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3980ca160 msgr2=0x7ff3980ca5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3980ca160 0x7ff3980ca5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 -- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3980cbd50 msgr2=0x7ff3980cc1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff38fe00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3980cbd50 0x7ff3980cc1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3980cbd50 0x7ff3980cc1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 -- 172.21.15.53:0/2368879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3980cae50 con 0x7ff378005180 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.183+0000 7ff396a00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff39814e4e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7ff38800d7d0 tx=0x7ff38800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.184+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3880074e0 con 0x7ff378005180 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.184+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff388007640 con 0x7ff378005180 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.184+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3880049e0 con 0x7ff378005180 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.184+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3980cb130 con 0x7ff378005180 2024-01-16T15:09:43.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.184+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3980cb6f0 con 0x7ff378005180 2024-01-16T15:09:43.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.185+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff39815dd10 con 0x7ff378005180 2024-01-16T15:09:43.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.186+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff388004b40 con 0x7ff378005180 2024-01-16T15:09:43.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.187+0000 7ff38ea00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff37006bec0 0x7ff37006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:43.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.187+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff388087e20 con 0x7ff378005180 2024-01-16T15:09:43.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.187+0000 7ff38fe00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff37006bec0 0x7ff37006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:43.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.189+0000 7ff38fe00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff37006bec0 0x7ff37006e380 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff3981f6770 tx=0x7ff380009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:43.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.191+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3880546e0 con 0x7ff378005180 2024-01-16T15:09:43.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.458+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff398067450 con 0x7ff378005180 2024-01-16T15:09:43.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.459+0000 7ff38ea00640 1 -- 172.21.15.53:0/2368879 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7ff388019070 con 0x7ff378005180 2024-01-16T15:09:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:43.437 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:43.437 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:43.438 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 610, 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:43.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:43.440 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1315, 2024-01-16T15:09:43.441 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 5369998267, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 11167563776, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 761926549504, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 178836674, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 178846566, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 43, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 46 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:43.442 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:43.443 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:43.444 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:43.445 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:43.446 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:43.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff37006bec0 msgr2=0x7ff37006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff37006bec0 0x7ff37006e380 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7ff3981f6770 tx=0x7ff380009040 comp rx=0 tx=0).stop 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 msgr2=0x7ff39814e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff39814e4e0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7ff38800d7d0 tx=0x7ff38800dc90 comp rx=0 tx=0).stop 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 shutdown_connections 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff37006bec0 0x7ff37006e380 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3980ca160 0x7ff3980ca5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3980cbd50 0x7ff3980cc1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 --2- 172.21.15.53:0/2368879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff378005180 0x7ff39814e4e0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.462+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 >> 172.21.15.53:0/2368879 conn(0x7ff3980d3090 msgr2=0x7ff39806c650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:43.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.463+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 shutdown_connections 2024-01-16T15:09:43.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:43.463+0000 7ff397e00640 1 -- 172.21.15.53:0/2368879 wait complete. 2024-01-16T15:09:44.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.754+0000 7f47ed600640 1 Processor -- start 2024-01-16T15:09:44.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.754+0000 7f47ed600640 1 -- start start 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.756+0000 7f47ed600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f47e80cd170 0x7f47e8071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.756+0000 7f47ed600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e8071690 con 0x7f47e80cf9a0 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.756+0000 7f47ed600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e8071970 con 0x7f47e814b2c0 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.756+0000 7f47ed600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e814b6a0 con 0x7f47e80cd170 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.757+0000 7f47e6000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f47e80cf9a0 0x7f47e8070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33868/0 (socket says 172.21.15.53:33868) 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.757+0000 7f47e6000640 1 -- 172.21.15.53:0/3807220452 learned_addr learned my addr 172.21.15.53:0/3807220452 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:44.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.757+0000 7f47e7400640 1 --2- 172.21.15.53:0/3807220452 >> v2:172.21.15.146:3300/0 conn(0x7f47e80cd170 0x7f47e8071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.757+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020220639 0 0) 0x7f47e8071690 con 0x7f47e80cf9a0 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.757+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c8003520 con 0x7f47e80cf9a0 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2459224766 0 0) 0x7f47c8003520 con 0x7f47e80cf9a0 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 >> v2:172.21.15.146:3300/0 conn(0x7f47e80cd170 msgr2=0x7f47e8071070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 --2- 172.21.15.53:0/3807220452 >> v2:172.21.15.146:3300/0 conn(0x7f47e80cd170 0x7f47e8071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 >> v1:172.21.15.146:6789/0 conn(0x7f47e814b2c0 legacy=0x7f47e80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8071c50 con 0x7f47e80cf9a0 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f47d0001670 con 0x7f47e80cf9a0 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 >> v1:172.21.15.53:6789/0 conn(0x7f47e80cf9a0 legacy=0x7f47e8070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47c8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47c80047c0 0x7f47c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47c80052a0 0x7f47c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e814b6a0 con 0x7f47c80039c0 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c800c1f0 con 0x7f47c80052a0 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c800c3b0 con 0x7f47c80047c0 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.758+0000 7f47e6a00640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47c8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.759+0000 7f47e6a00640 1 -- 172.21.15.53:0/3807220452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47c80047c0 msgr2=0x7f47c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.759+0000 7f47e6a00640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47c80047c0 0x7f47c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.759+0000 7f47e6a00640 1 -- 172.21.15.53:0/3807220452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47c80052a0 msgr2=0x7f47c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.759+0000 7f47e6a00640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47c80052a0 0x7f47c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.759+0000 7f47e6a00640 1 -- 172.21.15.53:0/3807220452 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c8003520 con 0x7f47c80039c0 2024-01-16T15:09:44.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.759+0000 7f47e7400640 1 -- 172.21.15.53:0/3807220452 reap_dead start 2024-01-16T15:09:44.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.760+0000 7f47e6a00640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47c8003dc0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f47e0001d20 tx=0x7f47c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0807a3e03b47648 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.760+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0003b90 con 0x7f47c80039c0 2024-01-16T15:09:44.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.760+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e0003d10 con 0x7f47c80039c0 2024-01-16T15:09:44.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.760+0000 7f47e5600640 1 -- 172.21.15.53:0/3807220452 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00052b0 con 0x7f47c80039c0 2024-01-16T15:09:44.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.761+0000 7f47ed600640 1 -- 172.21.15.53:0/3807220452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 msgr2=0x7f47c8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.761+0000 7f47ed600640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47c8003dc0 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f47e0001d20 tx=0x7f47c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.761+0000 7f47ed600640 1 -- 172.21.15.53:0/3807220452 shutdown_connections 2024-01-16T15:09:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.761+0000 7f47ed600640 1 --2- 172.21.15.53:0/3807220452 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47c8003dc0 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.761+0000 7f47ed600640 1 -- 172.21.15.53:0/3807220452 >> 172.21.15.53:0/3807220452 conn(0x7f47e806a800 msgr2=0x7f47e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.762+0000 7f47ed600640 1 -- 172.21.15.53:0/3807220452 shutdown_connections 2024-01-16T15:09:44.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.762+0000 7f47ed600640 1 -- 172.21.15.53:0/3807220452 wait complete. 2024-01-16T15:09:44.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.764+0000 7f47ed600640 1 Processor -- start 2024-01-16T15:09:44.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.764+0000 7f47ed600640 1 -- start start 2024-01-16T15:09:44.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47ed600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47e8162d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47ed600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47e806cb80 0x7f47e81632b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47ed600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47e80cde80 0x7f47e81637f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47ed600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e8161d60 con 0x7f47c80039c0 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47ed600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e81628e0 con 0x7f47e806cb80 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47ed600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e8161fb0 con 0x7f47e80cde80 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47e6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47e8162d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47e6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47e8162d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37476/0 (socket says 172.21.15.53:37476) 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.765+0000 7f47e6a00640 1 -- 172.21.15.53:0/1507060114 learned_addr learned my addr 172.21.15.53:0/1507060114 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6000640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47e806cb80 0x7f47e81632b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e7400640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47e80cde80 0x7f47e81637f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6a00640 1 -- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47e80cde80 msgr2=0x7f47e81637f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6a00640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47e80cde80 0x7f47e81637f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6a00640 1 -- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47e806cb80 msgr2=0x7f47e81632b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:44.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6a00640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47e806cb80 0x7f47e81632b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:44.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6a00640 1 -- 172.21.15.53:0/1507060114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8162290 con 0x7f47c80039c0 2024-01-16T15:09:44.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6000640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47e806cb80 0x7f47e81632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:44.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.766+0000 7f47e6a00640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47e8162d70 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f47e002be80 tx=0x7f47e0001330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:44.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.767+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0007dd0 con 0x7f47c80039c0 2024-01-16T15:09:44.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.767+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e0007040 con 0x7f47c80039c0 2024-01-16T15:09:44.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.767+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e00073d0 con 0x7f47c80039c0 2024-01-16T15:09:44.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.767+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e8162570 con 0x7f47c80039c0 2024-01-16T15:09:44.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.767+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e80afb80 con 0x7f47c80039c0 2024-01-16T15:09:44.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.769+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e8072610 con 0x7f47c80039c0 2024-01-16T15:09:44.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.769+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f47e0042020 con 0x7f47c80039c0 2024-01-16T15:09:44.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.770+0000 7f47dea00640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47c006bec0 0x7f47c006e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:44.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.771+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f47e00aa9a0 con 0x7f47c80039c0 2024-01-16T15:09:44.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.776+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f47e00772c0 con 0x7f47c80039c0 2024-01-16T15:09:44.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.798+0000 7f47e6000640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47c006bec0 0x7f47c006e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:44.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:44.806+0000 7f47e6000640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47c006bec0 0x7f47c006e380 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f47e814b000 tx=0x7f47d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:45.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.128+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f47e80afd70 con 0x7f47c80039c0 2024-01-16T15:09:45.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.130+0000 7f47dea00640 1 -- 172.21.15.53:0/1507060114 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f47e004a990 con 0x7f47c80039c0 2024-01-16T15:09:45.111 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:45.111 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:45.112 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 612, 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:45.113 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:45.114 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1315, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 5369998267, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 11167563776, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 761926549504, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 178836674, 2024-01-16T15:09:45.115 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 178846566, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 43, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 46 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:45.116 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:45.117 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:45.118 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:45.119 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:45.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47c006bec0 msgr2=0x7f47c006e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47c006bec0 0x7f47c006e380 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f47e814b000 tx=0x7f47d0009040 comp rx=0 tx=0).stop 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 msgr2=0x7f47e8162d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47e8162d70 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f47e002be80 tx=0x7f47e0001330 comp rx=0 tx=0).stop 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 shutdown_connections 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f47c006bec0 0x7f47c006e380 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f47e80cde80 0x7f47e81637f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f47e806cb80 0x7f47e81632b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 --2- 172.21.15.53:0/1507060114 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f47c80039c0 0x7f47e8162d70 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.136+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 >> 172.21.15.53:0/1507060114 conn(0x7f47e806a800 msgr2=0x7f47e806c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.137+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 shutdown_connections 2024-01-16T15:09:45.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:45.137+0000 7f47ed600640 1 -- 172.21.15.53:0/1507060114 wait complete. 2024-01-16T15:09:46.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:46.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.417+0000 7fb3ec000640 1 Processor -- start 2024-01-16T15:09:46.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.417+0000 7fb3ec000640 1 -- start start 2024-01-16T15:09:46.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3ec000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb3e40cf9a0 0x7fb3e415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3ec000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb3e415e1e0 0x7fb3e40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3ec000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3e40b1ee0 con 0x7fb3e4143ae0 2024-01-16T15:09:46.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3ec000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e40b2780 con 0x7fb3e415e1e0 2024-01-16T15:09:46.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3ec000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e415e990 con 0x7fb3e40cf9a0 2024-01-16T15:09:46.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3e9600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb3e4143ae0 0x7fb3e40cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33872/0 (socket says 172.21.15.53:33872) 2024-01-16T15:09:46.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3e9600640 1 -- 172.21.15.53:0/1055677208 learned_addr learned my addr 172.21.15.53:0/1055677208 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:46.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.420+0000 7fb3ea000640 1 --2- 172.21.15.53:0/1055677208 >> v2:172.21.15.53:3300/0 conn(0x7fb3e415e1e0 0x7fb3e40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.421+0000 7fb3ea000640 1 -- 172.21.15.53:0/1055677208 >> v2:172.21.15.146:3300/0 conn(0x7fb3e40cf9a0 msgr2=0x7fb3e415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:46.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.421+0000 7fb3ea000640 1 --2- 172.21.15.53:0/1055677208 >> v2:172.21.15.146:3300/0 conn(0x7fb3e40cf9a0 0x7fb3e415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.421+0000 7fb3ea000640 1 -- 172.21.15.53:0/1055677208 >> v1:172.21.15.53:6789/0 conn(0x7fb3e4143ae0 legacy=0x7fb3e40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.421+0000 7fb3ea000640 1 -- 172.21.15.53:0/1055677208 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40b2130 con 0x7fb3e415e1e0 2024-01-16T15:09:46.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.423+0000 7fb3ea000640 1 --2- 172.21.15.53:0/1055677208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e40702c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40cb830 tx=0x7fb3dc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f8e27c01da455a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.424+0000 7fb3e3e00640 1 -- 172.21.15.53:0/1055677208 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc02cc60 con 0x7fb3e415e1e0 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.424+0000 7fb3e3e00640 1 -- 172.21.15.53:0/1055677208 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc02cdc0 con 0x7fb3e415e1e0 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.424+0000 7fb3e3e00640 1 -- 172.21.15.53:0/1055677208 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc034660 con 0x7fb3e415e1e0 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 -- 172.21.15.53:0/1055677208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 msgr2=0x7fb3e40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 --2- 172.21.15.53:0/1055677208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e40702c0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7fb3e40cb830 tx=0x7fb3dc02beb0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 -- 172.21.15.53:0/1055677208 shutdown_connections 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 --2- 172.21.15.53:0/1055677208 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e40702c0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 --2- 172.21.15.53:0/1055677208 >> v2:172.21.15.146:3300/0 conn(0x7fb3e40cf9a0 0x7fb3e415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 -- 172.21.15.53:0/1055677208 >> 172.21.15.53:0/1055677208 conn(0x7fb3e406a800 msgr2=0x7fb3e406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 -- 172.21.15.53:0/1055677208 shutdown_connections 2024-01-16T15:09:46.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.425+0000 7fb3ec000640 1 -- 172.21.15.53:0/1055677208 wait complete. 2024-01-16T15:09:46.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.429+0000 7fb3ec000640 1 Processor -- start 2024-01-16T15:09:46.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.429+0000 7fb3ec000640 1 -- start start 2024-01-16T15:09:46.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ec000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3e40cf9a0 0x7fb3e41376d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ec000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3e4143ae0 0x7fb3e41f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ec000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e406caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ec000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4139300 con 0x7fb3e415e1e0 2024-01-16T15:09:46.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ec000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e41395e0 con 0x7fb3e40cf9a0 2024-01-16T15:09:46.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ec000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e41398c0 con 0x7fb3e4143ae0 2024-01-16T15:09:46.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ea000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e406caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ea000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e406caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37496/0 (socket says 172.21.15.53:37496) 2024-01-16T15:09:46.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.431+0000 7fb3ea000640 1 -- 172.21.15.53:0/2958114861 learned_addr learned my addr 172.21.15.53:0/2958114861 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:46.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.432+0000 7fb3ea000640 1 -- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3e4143ae0 msgr2=0x7fb3e41f75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.432+0000 7fb3ea000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3e4143ae0 0x7fb3e41f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.432+0000 7fb3ea000640 1 -- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3e40cf9a0 msgr2=0x7fb3e41376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.432+0000 7fb3ea000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3e40cf9a0 0x7fb3e41376d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.432+0000 7fb3ea000640 1 -- 172.21.15.53:0/2958114861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc009a70 con 0x7fb3e415e1e0 2024-01-16T15:09:46.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.433+0000 7fb3ea000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e406caf0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009d90 tx=0x7fb3dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.434+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc03e610 con 0x7fb3e415e1e0 2024-01-16T15:09:46.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.434+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc03e770 con 0x7fb3e415e1e0 2024-01-16T15:09:46.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.434+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc03eb20 con 0x7fb3e415e1e0 2024-01-16T15:09:46.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.435+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e4139ba0 con 0x7fb3e415e1e0 2024-01-16T15:09:46.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.435+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e4137df0 con 0x7fb3e415e1e0 2024-01-16T15:09:46.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.438+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b4005250 con 0x7fb3e415e1e0 2024-01-16T15:09:46.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.439+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb3dc03b050 con 0x7fb3e415e1e0 2024-01-16T15:09:46.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.441+0000 7fb3e1600640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3bc06bee0 0x7fb3bc06e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:46.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.441+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb3dc0aadf0 con 0x7fb3e415e1e0 2024-01-16T15:09:46.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.444+0000 7fb3e9600640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3bc06bee0 0x7fb3bc06e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:46.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.447+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb3dc0776f0 con 0x7fb3e415e1e0 2024-01-16T15:09:46.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.463+0000 7fb3e9600640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3bc06bee0 0x7fb3bc06e3a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40015f0 tx=0x7fb3d4007a40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:46.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.902+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb3b40057d0 con 0x7fb3e415e1e0 2024-01-16T15:09:46.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.903+0000 7fb3e1600640 1 -- 172.21.15.53:0/2958114861 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fb3dc04e990 con 0x7fb3e415e1e0 2024-01-16T15:09:46.880 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:46.881 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 613, 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:46.882 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:46.883 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1469, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 6015990633, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 12111413248, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 760982700032, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 241612041, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 232550088, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 58, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 61 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:46.884 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:46.885 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:46.886 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:46.887 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3bc06bee0 msgr2=0x7fb3bc06e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3bc06bee0 0x7fb3bc06e3a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40015f0 tx=0x7fb3d4007a40 comp rx=0 tx=0).stop 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 msgr2=0x7fb3e406caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e406caf0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009d90 tx=0x7fb3dc004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 shutdown_connections 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb3bc06bee0 0x7fb3bc06e3a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb3e415e1e0 0x7fb3e406caf0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb3e4143ae0 0x7fb3e41f75b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 --2- 172.21.15.53:0/2958114861 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb3e40cf9a0 0x7fb3e41376d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 >> 172.21.15.53:0/2958114861 conn(0x7fb3e406a800 msgr2=0x7fb3e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 shutdown_connections 2024-01-16T15:09:46.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:46.907+0000 7fb3ec000640 1 -- 172.21.15.53:0/2958114861 wait complete. 2024-01-16T15:09:47.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:48.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.219+0000 7f3392600640 1 Processor -- start 2024-01-16T15:09:48.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.219+0000 7f3392600640 1 -- start start 2024-01-16T15:09:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.221+0000 7f3392600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c143ae0 0x7f338c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.221+0000 7f3392600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f338c0cf7d0 0x7f338c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.221+0000 7f3392600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f338c0cfc30 con 0x7f338c15e1e0 2024-01-16T15:09:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.221+0000 7f3392600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f338c0b1ee0 con 0x7f338c0cf7d0 2024-01-16T15:09:48.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.221+0000 7f3392600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c0b2780 con 0x7f338c143ae0 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.221+0000 7f3382800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f338c0cf7d0 0x7f338c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f3382800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f338c0cf7d0 0x7f338c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37498/0 (socket says 172.21.15.53:37498) 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f3382800640 1 -- 172.21.15.53:0/2982167766 learned_addr learned my addr 172.21.15.53:0/2982167766 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f338b400640 1 --2- 172.21.15.53:0/2982167766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c143ae0 0x7f338c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f338aa00640 1 -- 172.21.15.53:0/2982167766 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542097217 0 0) 0x7f338c0cfc30 con 0x7f338c15e1e0 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f338aa00640 1 -- 172.21.15.53:0/2982167766 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f336c003520 con 0x7f338c15e1e0 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f3382800640 1 -- 172.21.15.53:0/2982167766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c143ae0 msgr2=0x7f338c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f3382800640 1 --2- 172.21.15.53:0/2982167766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c143ae0 0x7f338c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f3382800640 1 -- 172.21.15.53:0/2982167766 >> v1:172.21.15.53:6789/0 conn(0x7f338c15e1e0 legacy=0x7f338c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.222+0000 7f3382800640 1 -- 172.21.15.53:0/2982167766 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c15e990 con 0x7f338c0cf7d0 2024-01-16T15:09:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.224+0000 7f3382800640 1 --2- 172.21.15.53:0/2982167766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c0cf7d0 0x7f338c15dca0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f3374009870 tx=0x7f337402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b13f5a04ab19b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.224+0000 7f338aa00640 1 -- 172.21.15.53:0/2982167766 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374031640 con 0x7f338c0cf7d0 2024-01-16T15:09:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.224+0000 7f338aa00640 1 -- 172.21.15.53:0/2982167766 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33740317a0 con 0x7f338c0cf7d0 2024-01-16T15:09:48.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.224+0000 7f338aa00640 1 -- 172.21.15.53:0/2982167766 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3374031b30 con 0x7f338c0cf7d0 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.225+0000 7f3392600640 1 -- 172.21.15.53:0/2982167766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c0cf7d0 msgr2=0x7f338c15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.225+0000 7f3392600640 1 --2- 172.21.15.53:0/2982167766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c0cf7d0 0x7f338c15dca0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f3374009870 tx=0x7f337402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.225+0000 7f3392600640 1 -- 172.21.15.53:0/2982167766 shutdown_connections 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.225+0000 7f3392600640 1 --2- 172.21.15.53:0/2982167766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c0cf7d0 0x7f338c15dca0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.225+0000 7f3392600640 1 --2- 172.21.15.53:0/2982167766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c143ae0 0x7f338c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.225+0000 7f3392600640 1 -- 172.21.15.53:0/2982167766 >> 172.21.15.53:0/2982167766 conn(0x7f338c06a800 msgr2=0x7f338c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.226+0000 7f3392600640 1 -- 172.21.15.53:0/2982167766 shutdown_connections 2024-01-16T15:09:48.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.226+0000 7f3392600640 1 -- 172.21.15.53:0/2982167766 wait complete. 2024-01-16T15:09:48.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.227+0000 7f3392600640 1 Processor -- start 2024-01-16T15:09:48.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.228+0000 7f3392600640 1 -- start start 2024-01-16T15:09:48.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.228+0000 7f3392600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c0cf7d0 0x7f338c13b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f3392600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f338c143ae0 0x7f338c13c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f3392600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 0x7f338c0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f3392600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c13c7e0 con 0x7f338c15e1e0 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f3392600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c13a6a0 con 0x7f338c143ae0 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f3392600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f338c13a980 con 0x7f338c0cf7d0 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 0x7f338c0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 0x7f338c0a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37500/0 (socket says 172.21.15.53:37500) 2024-01-16T15:09:48.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 -- 172.21.15.53:0/2484341827 learned_addr learned my addr 172.21.15.53:0/2484341827 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f3382800640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f338c143ae0 0x7f338c13c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 -- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c0cf7d0 msgr2=0x7f338c13b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c0cf7d0 0x7f338c13b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 -- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f338c143ae0 msgr2=0x7f338c13c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f338c143ae0 0x7f338c13c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.229+0000 7f338be00640 1 -- 172.21.15.53:0/2484341827 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740094d0 con 0x7f338c15e1e0 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.230+0000 7f338be00640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 0x7f338c0a84c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f338400c3e0 tx=0x7f338400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.230+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33840035f0 con 0x7f338c15e1e0 2024-01-16T15:09:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.230+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3384003750 con 0x7f338c15e1e0 2024-01-16T15:09:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.230+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384003b40 con 0x7f338c15e1e0 2024-01-16T15:09:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.230+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c13ac60 con 0x7f338c15e1e0 2024-01-16T15:09:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.230+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c156cf0 con 0x7f338c15e1e0 2024-01-16T15:09:48.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.231+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c0675d0 con 0x7f338c15e1e0 2024-01-16T15:09:48.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.233+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3384003ca0 con 0x7f338c15e1e0 2024-01-16T15:09:48.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.235+0000 7f3383e00640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f336406bec0 0x7f336406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:48.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.235+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3384087e30 con 0x7f338c15e1e0 2024-01-16T15:09:48.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.236+0000 7f338b400640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f336406bec0 0x7f336406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:48.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.237+0000 7f338b400640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f336406bec0 0x7f336406e380 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f337c0070b0 tx=0x7f337c007040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:48.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.239+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33840544d0 con 0x7f338c15e1e0 2024-01-16T15:09:48.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.650+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f338c13cf40 con 0x7f338c15e1e0 2024-01-16T15:09:48.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.651+0000 7f3383e00640 1 -- 172.21.15.53:0/2484341827 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7f3384017020 con 0x7f338c15e1e0 2024-01-16T15:09:48.627 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:48.628 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 615, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:48.629 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1469, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 6015990633, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 12111413248, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 760982700032, 2024-01-16T15:09:48.630 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 241612041, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 232550088, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 58, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 61 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:48.631 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:48.632 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 9.523% (13s)\n [==..........................] (remaining: 2m)", 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.095228567719459534, 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 16.665% (13s)\n [====........................] (remaining: 66s)", 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.16664999723434448, 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:48.633 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:48.634 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:48.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f336406bec0 msgr2=0x7f336406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f336406bec0 0x7f336406e380 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f337c0070b0 tx=0x7f337c007040 comp rx=0 tx=0).stop 2024-01-16T15:09:48.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 msgr2=0x7f338c0a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 0x7f338c0a84c0 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f338400c3e0 tx=0x7f338400c8a0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 shutdown_connections 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f336406bec0 0x7f336406e380 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f338c15e1e0 0x7f338c0a84c0 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f338c143ae0 0x7f338c13c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 --2- 172.21.15.53:0/2484341827 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f338c0cf7d0 0x7f338c13b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.655+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 >> 172.21.15.53:0/2484341827 conn(0x7f338c06a800 msgr2=0x7f338c0708a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.656+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 shutdown_connections 2024-01-16T15:09:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:48.656+0000 7f3392600640 1 -- 172.21.15.53:0/2484341827 wait complete. 2024-01-16T15:09:49.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:49.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.950+0000 7ff82e600640 1 Processor -- start 2024-01-16T15:09:49.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.951+0000 7ff82e600640 1 -- start start 2024-01-16T15:09:49.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.954+0000 7ff82e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8280ce8f0 0x7ff8280ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.954+0000 7ff82e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff8280b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.954+0000 7ff82e600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8280d0de0 con 0x7ff828160f80 2024-01-16T15:09:49.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.954+0000 7ff82e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8280d10c0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.954+0000 7ff82e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8280d13a0 con 0x7ff8280b2a00 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.955+0000 7ff81e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8280ce8f0 0x7ff8280ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.956+0000 7ff81e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff8280ce8f0 0x7ff8280ced30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:55402/0 (socket says 172.21.15.53:55402) 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.956+0000 7ff81e800640 1 -- 172.21.15.53:0/2996237015 learned_addr learned my addr 172.21.15.53:0/2996237015 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.957+0000 7ff81e800640 1 -- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 msgr2=0x7ff8280b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.957+0000 7ff81e800640 1 --2- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff8280b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.957+0000 7ff81e800640 1 -- 172.21.15.53:0/2996237015 >> v1:172.21.15.146:6789/0 conn(0x7ff828160f80 legacy=0x7ff828161360 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.957+0000 7ff81e800640 1 -- 172.21.15.53:0/2996237015 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8280d1680 con 0x7ff8280ce8f0 2024-01-16T15:09:49.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.959+0000 7ff81e800640 1 --2- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8280ce8f0 0x7ff8280ced30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff810009870 tx=0x7ff81002b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=914cf74de2cec722 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.959+0000 7ff826a00640 1 -- 172.21.15.53:0/2996237015 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810031640 con 0x7ff8280ce8f0 2024-01-16T15:09:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.959+0000 7ff826a00640 1 -- 172.21.15.53:0/2996237015 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8100317a0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.959+0000 7ff826a00640 1 -- 172.21.15.53:0/2996237015 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810031b30 con 0x7ff8280ce8f0 2024-01-16T15:09:49.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.960+0000 7ff82e600640 1 -- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8280ce8f0 msgr2=0x7ff8280ced30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.960+0000 7ff82e600640 1 --2- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8280ce8f0 0x7ff8280ced30 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7ff810009870 tx=0x7ff81002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.961+0000 7ff82e600640 1 -- 172.21.15.53:0/2996237015 shutdown_connections 2024-01-16T15:09:49.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.961+0000 7ff82e600640 1 --2- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff8280b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.961+0000 7ff82e600640 1 --2- 172.21.15.53:0/2996237015 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff8280ce8f0 0x7ff8280ced30 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.961+0000 7ff82e600640 1 -- 172.21.15.53:0/2996237015 >> 172.21.15.53:0/2996237015 conn(0x7ff82809c550 msgr2=0x7ff8280d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:49.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.961+0000 7ff82e600640 1 -- 172.21.15.53:0/2996237015 shutdown_connections 2024-01-16T15:09:49.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.961+0000 7ff82e600640 1 -- 172.21.15.53:0/2996237015 wait complete. 2024-01-16T15:09:49.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.963+0000 7ff82e600640 1 Processor -- start 2024-01-16T15:09:49.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.963+0000 7ff82e600640 1 -- start start 2024-01-16T15:09:49.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.964+0000 7ff82e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff828155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.964+0000 7ff82e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 0x7ff828157c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.964+0000 7ff82e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff828160f80 0x7ff8280b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.964+0000 7ff82e600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8280cad40 con 0x7ff8280ce8f0 2024-01-16T15:09:49.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.964+0000 7ff82e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8281564a0 con 0x7ff828160f80 2024-01-16T15:09:49.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.964+0000 7ff82e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff828160670 con 0x7ff8280b2a00 2024-01-16T15:09:49.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 0x7ff828157c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 0x7ff828157c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37502/0 (socket says 172.21.15.53:37502) 2024-01-16T15:09:49.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 -- 172.21.15.53:0/3254151809 learned_addr learned my addr 172.21.15.53:0/3254151809 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff827e00640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff828160f80 0x7ff8280b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff827400640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff828155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 -- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 msgr2=0x7ff828155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff828155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 -- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff828160f80 msgr2=0x7ff8280b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff828160f80 0x7ff8280b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.965+0000 7ff81e800640 1 -- 172.21.15.53:0/3254151809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8100094d0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.966+0000 7ff81e800640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 0x7ff828157c20 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff810000c00 tx=0x7ff8100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.967+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff81003ac60 con 0x7ff8280ce8f0 2024-01-16T15:09:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.967+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff81003ade0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.967+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff810044630 con 0x7ff8280ce8f0 2024-01-16T15:09:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.967+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8280ca410 con 0x7ff8280ce8f0 2024-01-16T15:09:49.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.967+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8280ca9d0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.969+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8280675d0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.970+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff81002f050 con 0x7ff8280ce8f0 2024-01-16T15:09:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.971+0000 7ff81fe00640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff80006be90 0x7ff80006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:49.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.972+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff8100aa7f0 con 0x7ff8280ce8f0 2024-01-16T15:09:49.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.976+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff810076e50 con 0x7ff8280ce8f0 2024-01-16T15:09:49.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:49.992+0000 7ff827400640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff80006be90 0x7ff80006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:50.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.033+0000 7ff827400640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff80006be90 0x7ff80006e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff818009f60 tx=0x7ff8180094f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:50.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.467+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff828067e80 con 0x7ff8280ce8f0 2024-01-16T15:09:50.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.467+0000 7ff81fe00640 1 -- 172.21.15.53:0/3254151809 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7ff81007a470 con 0x7ff8280ce8f0 2024-01-16T15:09:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:50.444 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:50.444 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:50.445 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 617, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.446 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 1579, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 6477364073, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 13008994304, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 760085118976, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 234932703, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 230055008, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 56, 2024-01-16T15:09:50.447 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 58 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:50.448 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.449 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 96s)", 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.19045713543891907, 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.33329999446868896, 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:50.450 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:50.451 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:50.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.470+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff80006be90 msgr2=0x7ff80006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.470+0000 7ff82e600640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff80006be90 0x7ff80006e350 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7ff818009f60 tx=0x7ff8180094f0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.470+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 msgr2=0x7ff828157c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.470+0000 7ff82e600640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 0x7ff828157c20 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7ff810000c00 tx=0x7ff8100046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 shutdown_connections 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff80006be90 0x7ff80006e350 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff828160f80 0x7ff8280b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff8280ce8f0 0x7ff828157c20 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 --2- 172.21.15.53:0/3254151809 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff8280b2a00 0x7ff828155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 >> 172.21.15.53:0/3254151809 conn(0x7ff82809c550 msgr2=0x7ff8280b5590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 shutdown_connections 2024-01-16T15:09:50.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:50.471+0000 7ff82e600640 1 -- 172.21.15.53:0/3254151809 wait complete. 2024-01-16T15:09:51.471 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:51.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.721+0000 7fa60ea00640 1 Processor -- start 2024-01-16T15:09:51.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.721+0000 7fa60ea00640 1 -- start start 2024-01-16T15:09:51.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.722+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa608071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.722+0000 7fa60ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa6080717d0 0x7fa6080cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.723+0000 7fa60ea00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa608070e50 con 0x7fa6080cca40 2024-01-16T15:09:51.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.723+0000 7fa60ea00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6080723c0 con 0x7fa6080717d0 2024-01-16T15:09:51.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.723+0000 7fa60ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6080cce70 con 0x7fa608143c40 2024-01-16T15:09:51.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.723+0000 7fa60d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa608071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.723+0000 7fa60d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa608071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:59918/0 (socket says 172.21.15.53:59918) 2024-01-16T15:09:51.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.723+0000 7fa60d600640 1 -- 172.21.15.53:0/499324489 learned_addr learned my addr 172.21.15.53:0/499324489 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:51.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.724+0000 7fa60cc00640 1 --2- 172.21.15.53:0/499324489 >> v2:172.21.15.146:3300/0 conn(0x7fa6080717d0 0x7fa6080cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.724+0000 7fa60d600640 1 -- 172.21.15.53:0/499324489 >> v2:172.21.15.146:3300/0 conn(0x7fa6080717d0 msgr2=0x7fa6080cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:51.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.724+0000 7fa60d600640 1 --2- 172.21.15.53:0/499324489 >> v2:172.21.15.146:3300/0 conn(0x7fa6080717d0 0x7fa6080cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.724+0000 7fa60d600640 1 -- 172.21.15.53:0/499324489 >> v1:172.21.15.146:6789/0 conn(0x7fa6080cca40 legacy=0x7fa60815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.724+0000 7fa60d600640 1 -- 172.21.15.53:0/499324489 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa60806e170 con 0x7fa608143c40 2024-01-16T15:09:51.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.726+0000 7fa60d600640 1 --2- 172.21.15.53:0/499324489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa608071290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009870 tx=0x7fa5f802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb2c87576424b992 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:51.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.726+0000 7fa607e00640 1 -- 172.21.15.53:0/499324489 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8031640 con 0x7fa608143c40 2024-01-16T15:09:51.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.726+0000 7fa607e00640 1 -- 172.21.15.53:0/499324489 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f80317a0 con 0x7fa608143c40 2024-01-16T15:09:51.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.726+0000 7fa607e00640 1 -- 172.21.15.53:0/499324489 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f8031b30 con 0x7fa608143c40 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 -- 172.21.15.53:0/499324489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 msgr2=0x7fa608071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 --2- 172.21.15.53:0/499324489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa608071290 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8009870 tx=0x7fa5f802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 -- 172.21.15.53:0/499324489 shutdown_connections 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 --2- 172.21.15.53:0/499324489 >> v2:172.21.15.146:3300/0 conn(0x7fa6080717d0 0x7fa6080cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 --2- 172.21.15.53:0/499324489 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa608071290 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 -- 172.21.15.53:0/499324489 >> 172.21.15.53:0/499324489 conn(0x7fa60806a800 msgr2=0x7fa60806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 -- 172.21.15.53:0/499324489 shutdown_connections 2024-01-16T15:09:51.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.727+0000 7fa60ea00640 1 -- 172.21.15.53:0/499324489 wait complete. 2024-01-16T15:09:51.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.729+0000 7fa60ea00640 1 Processor -- start 2024-01-16T15:09:51.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.729+0000 7fa60ea00640 1 -- start start 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 0x7fa608142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6080cca40 0x7fa6081f2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa6081f2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60ea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa608137be0 con 0x7fa6080717d0 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6081f72e0 con 0x7fa6080cca40 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa608137240 con 0x7fa608143c40 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 0x7fa608142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 0x7fa608142970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37516/0 (socket says 172.21.15.53:37516) 2024-01-16T15:09:51.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.730+0000 7fa60d600640 1 -- 172.21.15.53:0/3939906867 learned_addr learned my addr 172.21.15.53:0/3939906867 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:51.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60cc00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6080cca40 0x7fa6081f2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60d600640 1 -- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 msgr2=0x7fa6081f2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60e000640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa6081f2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60d600640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa6081f2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60d600640 1 -- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6080cca40 msgr2=0x7fa6081f2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60d600640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6080cca40 0x7fa6081f2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60d600640 1 -- 172.21.15.53:0/3939906867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f80094d0 con 0x7fa6080717d0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60e000640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa6081f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60cc00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6080cca40 0x7fa6081f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.731+0000 7fa60d600640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 0x7fa608142970 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80090b0 tx=0x7fa5f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.732+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f803a9f0 con 0x7fa6080717d0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.732+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5f803ab90 con 0x7fa6080717d0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.732+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f80443f0 con 0x7fa6080717d0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.732+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa608137520 con 0x7fa6080717d0 2024-01-16T15:09:51.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.732+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa608138000 con 0x7fa6080717d0 2024-01-16T15:09:51.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.733+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6080cb6d0 con 0x7fa6080717d0 2024-01-16T15:09:51.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.734+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fa5f802f050 con 0x7fa6080717d0 2024-01-16T15:09:51.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.735+0000 7fa605600640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa5e006be90 0x7fa5e006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:51.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.735+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa5f80aa720 con 0x7fa6080717d0 2024-01-16T15:09:51.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.736+0000 7fa60cc00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa5e006be90 0x7fa5e006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:51.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.737+0000 7fa60cc00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa5e006be90 0x7fa5e006e350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa6080af1d0 tx=0x7fa5f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:51.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:51.740+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa5f8076ea0 con 0x7fa6080717d0 2024-01-16T15:09:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.010+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa608137e50 con 0x7fa6080717d0 2024-01-16T15:09:51.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.011+0000 7fa605600640 1 -- 172.21.15.53:0/3939906867 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2426 (secure 0 0 0) 0x7fa5f8035090 con 0x7fa6080717d0 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:51.991 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 619, 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:51.992 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:51.993 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 2040, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 8411019590, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 16712802304, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 756381310976, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 350478545, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 360967713, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 83, 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 91 2024-01-16T15:09:51.994 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:51.995 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:51.996 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 19.046% (22s)\n [=====.......................] (remaining: 96s)", 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.19045713543891907, 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 33.33% (22s)\n [=========...................] (remaining: 45s)", 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.33329999446868896, 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:51.997 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa5e006be90 msgr2=0x7fa5e006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa5e006be90 0x7fa5e006e350 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fa6080af1d0 tx=0x7fa5f0008040 comp rx=0 tx=0).stop 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 msgr2=0x7fa608142970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 0x7fa608142970 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80090b0 tx=0x7fa5f80046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 shutdown_connections 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa5e006be90 0x7fa5e006e350 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa608143c40 0x7fa6081f2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa6080cca40 0x7fa6081f2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.014+0000 7fa60ea00640 1 --2- 172.21.15.53:0/3939906867 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa6080717d0 0x7fa608142970 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:51.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.015+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 >> 172.21.15.53:0/3939906867 conn(0x7fa60806a800 msgr2=0x7fa6081401e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:51.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.015+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 shutdown_connections 2024-01-16T15:09:51.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:52.015+0000 7fa60ea00640 1 -- 172.21.15.53:0/3939906867 wait complete. 2024-01-16T15:09:52.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:52.906 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:09:53.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:09:53.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.259+0000 7f13a4000640 1 Processor -- start 2024-01-16T15:09:53.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.259+0000 7f13a4000640 1 -- start start 2024-01-16T15:09:53.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.261+0000 7f13a4000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f139c13df80 0x7f139c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.261+0000 7f13a4000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c06c470 con 0x7f139c13e360 2024-01-16T15:09:53.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.261+0000 7f13a4000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139c06c750 con 0x7f139c0cf9a0 2024-01-16T15:09:53.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.261+0000 7f13a4000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139c06ca30 con 0x7f139c13df80 2024-01-16T15:09:53.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.262+0000 7f13a0c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f139c13e360 0x7f139c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48576/0 (socket says 172.21.15.53:48576) 2024-01-16T15:09:53.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.262+0000 7f13a0c00640 1 -- 172.21.15.53:0/405564336 learned_addr learned my addr 172.21.15.53:0/405564336 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.262+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045822533 0 0) 0x7f139c06c470 con 0x7f139c13e360 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.262+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f137c003520 con 0x7f139c13e360 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.262+0000 7f13a1600640 1 --2- 172.21.15.53:0/405564336 >> v2:172.21.15.146:3300/0 conn(0x7f139c13df80 0x7f139c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 478984061 0 0) 0x7f137c003520 con 0x7f139c13e360 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 >> v2:172.21.15.146:3300/0 conn(0x7f139c13df80 msgr2=0x7f139c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 --2- 172.21.15.53:0/405564336 >> v2:172.21.15.146:3300/0 conn(0x7f139c13df80 0x7f139c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 >> v1:172.21.15.146:6789/0 conn(0x7f139c0cf9a0 legacy=0x7f139c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c06cd10 con 0x7f139c13e360 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1384001670 con 0x7f139c13e360 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 >> v1:172.21.15.53:6789/0 conn(0x7f139c13e360 legacy=0x7f139c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137c003970 0x7f137c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c0047c0 0x7f137c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137c005180 0x7f137c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c06ca30 con 0x7f137c003970 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137c00c0d0 con 0x7f137c005180 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137c00c290 con 0x7f137c0047c0 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f13a1600640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137c003970 0x7f137c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f13a0c00640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c0047c0 0x7f137c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.263+0000 7f13a2000640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137c005180 0x7f137c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.264+0000 7f13a1600640 1 -- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c0047c0 msgr2=0x7f137c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.264+0000 7f13a1600640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c0047c0 0x7f137c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.264+0000 7f13a1600640 1 -- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137c005180 msgr2=0x7f137c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.264+0000 7f13a1600640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137c005180 0x7f137c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.264+0000 7f13a1600640 1 -- 172.21.15.53:0/405564336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f137c003520 con 0x7f137c003970 2024-01-16T15:09:53.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.264+0000 7f13a2000640 1 -- 172.21.15.53:0/405564336 reap_dead start 2024-01-16T15:09:53.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.265+0000 7f13a1600640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137c003970 0x7f137c003d90 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f138c00ba30 tx=0x7f137c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84a3faf140d23043 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.265+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c032660 con 0x7f137c003970 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.265+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c0327c0 con 0x7f137c003970 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.265+0000 7f139be00640 1 -- 172.21.15.53:0/405564336 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c032b50 con 0x7f137c003970 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 -- 172.21.15.53:0/405564336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137c003970 msgr2=0x7f137c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137c003970 0x7f137c003d90 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f138c00ba30 tx=0x7f137c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 -- 172.21.15.53:0/405564336 shutdown_connections 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 --2- 172.21.15.53:0/405564336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137c003970 0x7f137c003d90 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 -- 172.21.15.53:0/405564336 >> 172.21.15.53:0/405564336 conn(0x7f139c06a800 msgr2=0x7f139c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 -- 172.21.15.53:0/405564336 shutdown_connections 2024-01-16T15:09:53.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.266+0000 7f13a4000640 1 -- 172.21.15.53:0/405564336 wait complete. 2024-01-16T15:09:53.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.268+0000 7f13a4000640 1 Processor -- start 2024-01-16T15:09:53.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.268+0000 7f13a4000640 1 -- start start 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a4000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c003970 0x7f139c152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 0x7f139c1f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a4000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f139c0cd840 0x7f139c0cdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c0ce200 con 0x7f139c1e5890 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a4000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139c152ba0 con 0x7f139c0cd840 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a4000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f139c152e80 con 0x7f137c003970 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 0x7f139c1f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 0x7f139c1f2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39376/0 (socket says 172.21.15.53:39376) 2024-01-16T15:09:53.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a0c00640 1 -- 172.21.15.53:0/39705667 learned_addr learned my addr 172.21.15.53:0/39705667 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a1600640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c003970 0x7f139c152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a2000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f139c0cd840 0x7f139c0cdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.269+0000 7f13a0c00640 1 -- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c003970 msgr2=0x7f139c152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a0c00640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c003970 0x7f139c152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a0c00640 1 -- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f139c0cd840 msgr2=0x7f139c0cdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a0c00640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f139c0cd840 0x7f139c0cdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a0c00640 1 -- 172.21.15.53:0/39705667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c153160 con 0x7f139c1e5890 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a1600640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c003970 0x7f139c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:53.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a2000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f139c0cd840 0x7f139c0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a0c00640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 0x7f139c1f2b30 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f138400e700 tx=0x7f138400ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400ee10 con 0x7f139c1e5890 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138400c870 con 0x7f139c1e5890 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138400cc20 con 0x7f139c1e5890 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f139c153440 con 0x7f139c1e5890 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.270+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139c1f3070 con 0x7f139c1e5890 2024-01-16T15:09:53.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.272+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139c0675d0 con 0x7f139c1e5890 2024-01-16T15:09:53.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.272+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f138401d460 con 0x7f139c1e5890 2024-01-16T15:09:53.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.273+0000 7f1399600640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f137406bec0 0x7f137406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:53.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.273+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1384088730 con 0x7f139c1e5890 2024-01-16T15:09:53.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.274+0000 7f13a1600640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f137406bec0 0x7f137406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:53.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.275+0000 7f13a1600640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f137406bec0 0x7f137406e380 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f138c00ba00 tx=0x7f138c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:53.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.279+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1384054d90 con 0x7f139c1e5890 2024-01-16T15:09:53.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.543+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f139c0b04d0 con 0x7f139c1e5890 2024-01-16T15:09:53.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.544+0000 7f1399600640 1 -- 172.21.15.53:0/39705667 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2409 (secure 0 0 0) 0x7f13840583b0 con 0x7f139c1e5890 2024-01-16T15:09:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:09:53.521 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:09:53.521 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:09:53.522 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 620, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:09:53.523 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 2300, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 9501538630, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 19049029632, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 754045083648, 2024-01-16T15:09:53.524 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 378056301, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 376324184, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 90, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 94 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:09:53.525 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.526 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "7 ongoing+pending cloning jobs; Avg progress = 28.571% (26s)\n [========....................] (remaining: 66s)", 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.28571429848670959, 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 50.0% (26s)\n [==============..............] (remaining: 26s)", 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.5, 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:09:53.527 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:09:53.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f137406bec0 msgr2=0x7f137406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f137406bec0 0x7f137406e380 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f138c00ba00 tx=0x7f138c030000 comp rx=0 tx=0).stop 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 msgr2=0x7f139c1f2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 0x7f139c1f2b30 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f138400e700 tx=0x7f138400ebc0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 shutdown_connections 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f137406bec0 0x7f137406e380 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f139c0cd840 0x7f139c0cdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f139c1e5890 0x7f139c1f2b30 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 --2- 172.21.15.53:0/39705667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137c003970 0x7f139c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 >> 172.21.15.53:0/39705667 conn(0x7f139c06a800 msgr2=0x7f139c0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 shutdown_connections 2024-01-16T15:09:53.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:53.547+0000 7f13a4000640 1 -- 172.21.15.53:0/39705667 wait complete. 2024-01-16T15:09:54.542 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54148 2024-01-16T15:09:54.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.801+0000 7f7eebe00640 1 Processor -- start 2024-01-16T15:09:54.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.801+0000 7f7eebe00640 1 -- start start 2024-01-16T15:09:54.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.802+0000 7f7eebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 0x7f7eec0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.802+0000 7f7eebe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec070e50 con 0x7f7eec143c40 2024-01-16T15:09:54.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7eebe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eec0ccea0 con 0x7f7eec06d160 2024-01-16T15:09:54.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7eebe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0723c0 con 0x7f7eec0cc500 2024-01-16T15:09:54.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7eeaa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7eec143c40 0x7f7eec071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48588/0 (socket says 172.21.15.53:48588) 2024-01-16T15:09:54.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7eeaa00640 1 -- 172.21.15.53:0/880497235 learned_addr learned my addr 172.21.15.53:0/880497235 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7eea000640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 0x7f7eec0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615843803 0 0) 0x7f7eec070e50 con 0x7f7eec143c40 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.803+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ecc003520 con 0x7f7eec143c40 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2323680217 0 0) 0x7f7ecc003520 con 0x7f7eec143c40 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 msgr2=0x7f7eec0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 0x7f7eec0cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 >> v1:172.21.15.146:6789/0 conn(0x7f7eec06d160 legacy=0x7f7eec06d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec144020 con 0x7f7eec143c40 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7edc001670 con 0x7f7eec143c40 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 >> v1:172.21.15.53:6789/0 conn(0x7f7eec143c40 legacy=0x7f7eec071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ecc0039a0 0x7f7ecc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ecc0047c0 0x7f7ecc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 0x7f7ecc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec0723c0 con 0x7f7ecc0047c0 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc00bb40 con 0x7f7ecc0039a0 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ecc00bd00 con 0x7f7eec0cc500 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.804+0000 7f7eea000640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ecc0047c0 0x7f7ecc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eeaa00640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ecc0039a0 0x7f7ecc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eeb400640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 0x7f7ecc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eea000640 1 -- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 msgr2=0x7f7ecc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eea000640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7eec0cc500 0x7f7ecc005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eea000640 1 -- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ecc0039a0 msgr2=0x7f7ecc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eea000640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ecc0039a0 0x7f7ecc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eea000640 1 -- 172.21.15.53:0/880497235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ecc003520 con 0x7f7ecc0047c0 2024-01-16T15:09:54.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.805+0000 7f7eeaa00640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7ecc0039a0 0x7f7ecc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.806+0000 7f7eea000640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ecc0047c0 0x7f7ecc004c40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009ac0 tx=0x7f7ecc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=765c7d760af167c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.806+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed402c6b0 con 0x7f7ecc0047c0 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.806+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed402c810 con 0x7f7ecc0047c0 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.806+0000 7f7ee9600640 1 -- 172.21.15.53:0/880497235 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ed402cba0 con 0x7f7ecc0047c0 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eebe00640 1 -- 172.21.15.53:0/880497235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ecc0047c0 msgr2=0x7f7ecc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eebe00640 1 --2- 172.21.15.53:0/880497235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7ecc0047c0 0x7f7ecc004c40 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f7ed4009ac0 tx=0x7f7ecc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eeb400640 1 -- 172.21.15.53:0/880497235 reap_dead start 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eebe00640 1 -- 172.21.15.53:0/880497235 shutdown_connections 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eebe00640 1 -- 172.21.15.53:0/880497235 >> 172.21.15.53:0/880497235 conn(0x7f7eec06a800 msgr2=0x7f7eec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eebe00640 1 -- 172.21.15.53:0/880497235 shutdown_connections 2024-01-16T15:09:54.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.807+0000 7f7eebe00640 1 -- 172.21.15.53:0/880497235 wait complete. 2024-01-16T15:09:54.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.809+0000 7f7eebe00640 1 Processor -- start 2024-01-16T15:09:54.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.809+0000 7f7eebe00640 1 -- start start 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ecc0047c0 0x7f7eec1f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eebe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eec1423e0 0x7f7eec142860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 0x7f7eec1f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eebe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec1f7240 con 0x7f7eec1f6850 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eebe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec13ae80 con 0x7f7eec1423e0 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eebe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eec13b720 con 0x7f7ecc0047c0 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.810+0000 7f7eeb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 0x7f7eec1f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 0x7f7eec1f6d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39396/0 (socket says 172.21.15.53:39396) 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 -- 172.21.15.53:0/804166230 learned_addr learned my addr 172.21.15.53:0/804166230 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeaa00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ecc0047c0 0x7f7eec1f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eea000640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eec1423e0 0x7f7eec142860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 -- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ecc0047c0 msgr2=0x7f7eec1f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ecc0047c0 0x7f7eec1f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 -- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eec1423e0 msgr2=0x7f7eec142860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eec1423e0 0x7f7eec142860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 -- 172.21.15.53:0/804166230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eec142ee0 con 0x7f7eec1f6850 2024-01-16T15:09:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eea000640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eec1423e0 0x7f7eec142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:54.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.811+0000 7f7eeb400640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 0x7f7eec1f6d00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4000f80 tx=0x7f7ee400e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.812+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4003b80 con 0x7f7eec1f6850 2024-01-16T15:09:54.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.812+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ee4003d00 con 0x7f7eec1f6850 2024-01-16T15:09:54.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.812+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ee4005310 con 0x7f7eec1f6850 2024-01-16T15:09:54.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.812+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eec13b0d0 con 0x7f7eec1f6850 2024-01-16T15:09:54.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.812+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eec13ff60 con 0x7f7eec1f6850 2024-01-16T15:09:54.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.813+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eec0ce210 con 0x7f7eec1f6850 2024-01-16T15:09:54.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.814+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f7ee4005470 con 0x7f7eec1f6850 2024-01-16T15:09:54.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.815+0000 7f7ee2a00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7ec406bec0 0x7f7ec406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:54.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.815+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7ee4088520 con 0x7f7eec1f6850 2024-01-16T15:09:54.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.816+0000 7f7eeaa00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7ec406bec0 0x7f7ec406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:54.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.817+0000 7f7eeaa00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7ec406bec0 0x7f7ec406e380 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7edc000ff0 tx=0x7f7edc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:54.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:54.819+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7ee4054b80 con 0x7f7eec1f6850 2024-01-16T15:09:55.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.049+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54148", "target": ["mon-mgr", ""]}) v1 -- 0x7f7eec06c960 con 0x7f7ec406bec0 2024-01-16T15:09:55.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.223+0000 7f7ee2a00640 1 -- 172.21.15.53:0/804166230 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7edc00d050 con 0x7f7ec406bec0 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7ec406bec0 msgr2=0x7f7ec406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7ec406bec0 0x7f7ec406e380 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f7edc000ff0 tx=0x7f7edc009040 comp rx=0 tx=0).stop 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 msgr2=0x7f7eec1f6d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 0x7f7eec1f6d00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f7ee4000f80 tx=0x7f7ee400e9c0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 shutdown_connections 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7ec406bec0 0x7f7ec406e380 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7eec1f6850 0x7f7eec1f6d00 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7eec1423e0 0x7f7eec142860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 --2- 172.21.15.53:0/804166230 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7ecc0047c0 0x7f7eec1f4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 >> 172.21.15.53:0/804166230 conn(0x7f7eec06a800 msgr2=0x7f7eec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.226+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 shutdown_connections 2024-01-16T15:09:55.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.227+0000 7f7eebe00640 1 -- 172.21.15.53:0/804166230 wait complete. 2024-01-16T15:09:55.226 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone54149 2024-01-16T15:09:55.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.485+0000 7f2930000640 1 Processor -- start 2024-01-16T15:09:55.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.485+0000 7f2930000640 1 -- start start 2024-01-16T15:09:55.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f2930000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f292815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f2930000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2928070c40 con 0x7f29280b3ae0 2024-01-16T15:09:55.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f2930000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292806c470 con 0x7f29280cf290 2024-01-16T15:09:55.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f2930000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2928071450 con 0x7f29280ae5a0 2024-01-16T15:09:55.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f292cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f29280b3ae0 0x7f292806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48594/0 (socket says 172.21.15.53:48594) 2024-01-16T15:09:55.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f292cc00640 1 -- 172.21.15.53:0/1407411181 learned_addr learned my addr 172.21.15.53:0/1407411181 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.487+0000 7f292d600640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f292815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506161334 0 0) 0x7f2928070c40 con 0x7f29280b3ae0 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2908003520 con 0x7f29280b3ae0 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2314911063 0 0) 0x7f2908003520 con 0x7f29280b3ae0 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 msgr2=0x7f292815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f292815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 >> v1:172.21.15.146:6789/0 conn(0x7f29280cf290 legacy=0x7f29280b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292806c750 con 0x7f29280b3ae0 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2910001670 con 0x7f29280b3ae0 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 >> v1:172.21.15.53:6789/0 conn(0x7f29280b3ae0 legacy=0x7f292806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29080039a0 0x7f2908003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29080047c0 0x7f2908004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f2908005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2928071450 con 0x7f29080047c0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f290800baf0 con 0x7f29080039a0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.488+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f290800bcb0 con 0x7f29280ae5a0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292cc00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29080047c0 0x7f2908004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292d600640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29080039a0 0x7f2908003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292e000640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f2908005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292cc00640 1 -- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 msgr2=0x7f2908005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292cc00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f2908005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292cc00640 1 -- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29080039a0 msgr2=0x7f2908003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292cc00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29080039a0 0x7f2908003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292cc00640 1 -- 172.21.15.53:0/1407411181 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2908003520 con 0x7f29080047c0 2024-01-16T15:09:55.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.489+0000 7f292e000640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29280ae5a0 0x7f2908005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:55.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.490+0000 7f292cc00640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29080047c0 0x7f2908004c40 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f2910007f90 tx=0x7f29080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c53cf5a2355441a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.490+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910005ad0 con 0x7f29080047c0 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2910005c30 con 0x7f29080047c0 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2927e00640 1 -- 172.21.15.53:0/1407411181 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2910007070 con 0x7f29080047c0 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2930000640 1 -- 172.21.15.53:0/1407411181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29080047c0 msgr2=0x7f2908004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2930000640 1 --2- 172.21.15.53:0/1407411181 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29080047c0 0x7f2908004c40 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f2910007f90 tx=0x7f29080057f0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f292e000640 1 -- 172.21.15.53:0/1407411181 reap_dead start 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2930000640 1 -- 172.21.15.53:0/1407411181 shutdown_connections 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2930000640 1 -- 172.21.15.53:0/1407411181 >> 172.21.15.53:0/1407411181 conn(0x7f292806a800 msgr2=0x7f292806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2930000640 1 -- 172.21.15.53:0/1407411181 shutdown_connections 2024-01-16T15:09:55.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.491+0000 7f2930000640 1 -- 172.21.15.53:0/1407411181 wait complete. 2024-01-16T15:09:55.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.493+0000 7f2930000640 1 Processor -- start 2024-01-16T15:09:55.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.494+0000 7f2930000640 1 -- start start 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f2930000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29080047c0 0x7f29280cc280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f2930000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29281f6270 0x7f2928155750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f2930000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 0x7f2928155c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f2930000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2928160640 con 0x7f29281f66d0 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f2930000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292815fba0 con 0x7f29281f6270 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f2930000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29281f6de0 con 0x7f29080047c0 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f292e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 0x7f2928155c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f292e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 0x7f2928155c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39408/0 (socket says 172.21.15.53:39408) 2024-01-16T15:09:55.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f292e000640 1 -- 172.21.15.53:0/648374096 learned_addr learned my addr 172.21.15.53:0/648374096 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.495+0000 7f292d600640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29080047c0 0x7f29280cc280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292cc00640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29281f6270 0x7f2928155750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292e000640 1 -- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29080047c0 msgr2=0x7f29280cc280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292e000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29080047c0 0x7f29280cc280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292e000640 1 -- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29281f6270 msgr2=0x7f2928155750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292e000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29281f6270 0x7f2928155750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292e000640 1 -- 172.21.15.53:0/648374096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280cbbd0 con 0x7f29281f66d0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292cc00640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29281f6270 0x7f2928155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f292e000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 0x7f2928155c90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f29280aeb00 tx=0x7f292000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920003b70 con 0x7f29281f66d0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920003cf0 con 0x7f29281f66d0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292815fe40 con 0x7f29281f66d0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29200052b0 con 0x7f29281f66d0 2024-01-16T15:09:55.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.496+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29280cbeb0 con 0x7f29281f66d0 2024-01-16T15:09:55.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.498+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29280675d0 con 0x7f29281f66d0 2024-01-16T15:09:55.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.499+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2920005410 con 0x7f29281f66d0 2024-01-16T15:09:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.500+0000 7f2925600640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f290006be90 0x7f290006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:55.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.500+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2920087f80 con 0x7f29281f66d0 2024-01-16T15:09:55.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.500+0000 7f292d600640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f290006be90 0x7f290006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:55.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.501+0000 7f292d600640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f290006be90 0x7f290006e350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2918009a10 tx=0x7f2918008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:55.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.504+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2920054840 con 0x7f29281f66d0 2024-01-16T15:09:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.731+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone54149", "target": ["mon-mgr", ""]}) v1 -- 0x7f2928070550 con 0x7f290006be90 2024-01-16T15:09:55.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.901+0000 7f2925600640 1 -- 172.21.15.53:0/648374096 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2918004020 con 0x7f290006be90 2024-01-16T15:09:55.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f290006be90 msgr2=0x7f290006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f290006be90 0x7f290006e350 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f2918009a10 tx=0x7f2918008040 comp rx=0 tx=0).stop 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 msgr2=0x7f2928155c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 0x7f2928155c90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f29280aeb00 tx=0x7f292000dc00 comp rx=0 tx=0).stop 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 shutdown_connections 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f290006be90 0x7f290006e350 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29281f66d0 0x7f2928155c90 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29281f6270 0x7f2928155750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 --2- 172.21.15.53:0/648374096 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29080047c0 0x7f29280cc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.904+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 >> 172.21.15.53:0/648374096 conn(0x7f292806a800 msgr2=0x7f292815da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.905+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 shutdown_connections 2024-01-16T15:09:55.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:55.905+0000 7f2930000640 1 -- 172.21.15.53:0/648374096 wait complete. 2024-01-16T15:09:55.903 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541410 2024-01-16T15:09:56.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.153+0000 7f3683200640 1 Processor -- start 2024-01-16T15:09:56.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.153+0000 7f3683200640 1 -- start start 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.155+0000 7f3683200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f368415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.155+0000 7f3683200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f36840cd530 0x7f368413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.155+0000 7f3683200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36840cdec0 con 0x7f36840cd9b0 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.155+0000 7f3683200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36840ce1a0 con 0x7f36840cd530 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.155+0000 7f3683200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36840ce480 con 0x7f368413cfa0 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.155+0000 7f3679200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f36840cd530 0x7f368413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3679200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f36840cd530 0x7f368413f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39416/0 (socket says 172.21.15.53:39416) 2024-01-16T15:09:56.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3679200640 1 -- 172.21.15.53:0/1704458892 learned_addr learned my addr 172.21.15.53:0/1704458892 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:56.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3681e00640 1 --2- 172.21.15.53:0/1704458892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f368415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3679200640 1 -- 172.21.15.53:0/1704458892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 msgr2=0x7f368415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3679200640 1 --2- 172.21.15.53:0/1704458892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f368415afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3679200640 1 -- 172.21.15.53:0/1704458892 >> v1:172.21.15.53:6789/0 conn(0x7f36840cd9b0 legacy=0x7f368415d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3679200640 1 -- 172.21.15.53:0/1704458892 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36840ce760 con 0x7f36840cd530 2024-01-16T15:09:56.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.156+0000 7f3681400640 1 -- 172.21.15.53:0/1704458892 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 169320646 0 0) 0x7f36840cdec0 con 0x7f36840cd9b0 2024-01-16T15:09:56.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.157+0000 7f3679200640 1 --2- 172.21.15.53:0/1704458892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd530 0x7f368413f5a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f366c009870 tx=0x7f366c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b67e42675c7f3fb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.158+0000 7f3681400640 1 -- 172.21.15.53:0/1704458892 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c031640 con 0x7f36840cd530 2024-01-16T15:09:56.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.158+0000 7f3681400640 1 -- 172.21.15.53:0/1704458892 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c0317a0 con 0x7f36840cd530 2024-01-16T15:09:56.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.158+0000 7f3681400640 1 -- 172.21.15.53:0/1704458892 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c031b30 con 0x7f36840cd530 2024-01-16T15:09:56.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.158+0000 7f3683200640 1 -- 172.21.15.53:0/1704458892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd530 msgr2=0x7f368413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.158+0000 7f3683200640 1 --2- 172.21.15.53:0/1704458892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd530 0x7f368413f5a0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f366c009870 tx=0x7f366c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.159+0000 7f3683200640 1 -- 172.21.15.53:0/1704458892 shutdown_connections 2024-01-16T15:09:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.159+0000 7f3683200640 1 --2- 172.21.15.53:0/1704458892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd530 0x7f368413f5a0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.159+0000 7f3683200640 1 --2- 172.21.15.53:0/1704458892 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f368415afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.159+0000 7f3683200640 1 -- 172.21.15.53:0/1704458892 >> 172.21.15.53:0/1704458892 conn(0x7f368406a800 msgr2=0x7f368406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.159+0000 7f3683200640 1 -- 172.21.15.53:0/1704458892 shutdown_connections 2024-01-16T15:09:56.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.159+0000 7f3683200640 1 -- 172.21.15.53:0/1704458892 wait complete. 2024-01-16T15:09:56.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.161+0000 7f3683200640 1 Processor -- start 2024-01-16T15:09:56.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.161+0000 7f3683200640 1 -- start start 2024-01-16T15:09:56.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3683200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36840cd530 0x7f3684150240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3683200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 0x7f3684159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3683200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f3684159c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3683200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368412d770 con 0x7f36840cd9b0 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3683200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36841f3fd0 con 0x7f36840cd530 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3683200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368412cdd0 con 0x7f368413cfa0 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3679200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 0x7f3684159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3679200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 0x7f3684159740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39430/0 (socket says 172.21.15.53:39430) 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.162+0000 7f3679200640 1 -- 172.21.15.53:0/1507095699 learned_addr learned my addr 172.21.15.53:0/1507095699 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3682800640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f3684159c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3681e00640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36840cd530 0x7f3684150240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3679200640 1 -- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 msgr2=0x7f3684159c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3679200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f3684159c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3679200640 1 -- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36840cd530 msgr2=0x7f3684150240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3679200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36840cd530 0x7f3684150240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3679200640 1 -- 172.21.15.53:0/1507095699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366c0094d0 con 0x7f36840cd9b0 2024-01-16T15:09:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3682800640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f3684159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3681e00640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36840cd530 0x7f3684150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:56.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.163+0000 7f3679200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 0x7f3684159740 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f366c000c00 tx=0x7f366c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.164+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c03ac60 con 0x7f36840cd9b0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.164+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c03ade0 con 0x7f36840cd9b0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.164+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c044630 con 0x7f36840cd9b0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.164+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368412d0b0 con 0x7f36840cd9b0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.164+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3684150780 con 0x7f36840cd9b0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.165+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368412d8b0 con 0x7f36840cd9b0 2024-01-16T15:09:56.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.166+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f366c02f050 con 0x7f36840cd9b0 2024-01-16T15:09:56.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.167+0000 7f367aa00640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365c06be90 0x7f365c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.167+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f366c0aa6c0 con 0x7f36840cd9b0 2024-01-16T15:09:56.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.168+0000 7f3681e00640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365c06be90 0x7f365c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.169+0000 7f3681e00640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365c06be90 0x7f365c06e350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3674006fd0 tx=0x7f3674008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.172+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f368412d8b0 con 0x7f36840cd9b0 2024-01-16T15:09:56.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.410+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541410", "target": ["mon-mgr", ""]}) v1 -- 0x7f368412d8b0 con 0x7f365c06be90 2024-01-16T15:09:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.582+0000 7f367aa00640 1 -- 172.21.15.53:0/1507095699 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f368412d8b0 con 0x7f365c06be90 2024-01-16T15:09:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.585+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365c06be90 msgr2=0x7f365c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365c06be90 0x7f365c06e350 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3674006fd0 tx=0x7f3674008040 comp rx=0 tx=0).stop 2024-01-16T15:09:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 msgr2=0x7f3684159740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 0x7f3684159740 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f366c000c00 tx=0x7f366c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 shutdown_connections 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f365c06be90 0x7f365c06e350 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f368413cfa0 0x7f3684159c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36840cd9b0 0x7f3684159740 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 --2- 172.21.15.53:0/1507095699 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36840cd530 0x7f3684150240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 >> 172.21.15.53:0/1507095699 conn(0x7f368406a800 msgr2=0x7f36840aefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 shutdown_connections 2024-01-16T15:09:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.586+0000 7f3683200640 1 -- 172.21.15.53:0/1507095699 wait complete. 2024-01-16T15:09:56.586 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541411 2024-01-16T15:09:56.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.839+0000 7f89d1200640 1 Processor -- start 2024-01-16T15:09:56.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.839+0000 7f89d1200640 1 -- start start 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.841+0000 7f89d1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.841+0000 7f89d1200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89cc15f9c0 0x7f89cc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.841+0000 7f89d1200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89cc15c5f0 con 0x7f89cc13df80 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.841+0000 7f89d1200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89cc06c470 con 0x7f89cc15f9c0 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.841+0000 7f89d1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc06c750 con 0x7f89cc13e360 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.841+0000 7f89ca800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89cc15f9c0 0x7f89cc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89ca800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f89cc15f9c0 0x7f89cc0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39432/0 (socket says 172.21.15.53:39432) 2024-01-16T15:09:56.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89ca800640 1 -- 172.21.15.53:0/4131825611 learned_addr learned my addr 172.21.15.53:0/4131825611 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:56.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89c9400640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89ca800640 1 -- 172.21.15.53:0/4131825611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 msgr2=0x7f89cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89ca800640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89ca800640 1 -- 172.21.15.53:0/4131825611 >> v1:172.21.15.146:6789/0 conn(0x7f89cc13df80 legacy=0x7f89cc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:56.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89ca800640 1 -- 172.21.15.53:0/4131825611 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89cc06ca30 con 0x7f89cc15f9c0 2024-01-16T15:09:56.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.842+0000 7f89c9400640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:09:56.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.843+0000 7f89ca800640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc0cf7d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009e50 tx=0x7f89c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd0be4563b6efa7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.844+0000 7f89c8a00640 1 -- 172.21.15.53:0/4131825611 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c402ca70 con 0x7f89cc15f9c0 2024-01-16T15:09:56.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.844+0000 7f89c8a00640 1 -- 172.21.15.53:0/4131825611 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c402cbd0 con 0x7f89cc15f9c0 2024-01-16T15:09:56.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.844+0000 7f89c8a00640 1 -- 172.21.15.53:0/4131825611 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c40343f0 con 0x7f89cc15f9c0 2024-01-16T15:09:56.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.844+0000 7f89d1200640 1 -- 172.21.15.53:0/4131825611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 msgr2=0x7f89cc0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.844+0000 7f89d1200640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc0cf7d0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f89c4009e50 tx=0x7f89c402bed0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.845+0000 7f89d1200640 1 -- 172.21.15.53:0/4131825611 shutdown_connections 2024-01-16T15:09:56.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.845+0000 7f89d1200640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc0cf7d0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.845+0000 7f89d1200640 1 --2- 172.21.15.53:0/4131825611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.845+0000 7f89d1200640 1 -- 172.21.15.53:0/4131825611 >> 172.21.15.53:0/4131825611 conn(0x7f89cc06a800 msgr2=0x7f89cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:56.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.845+0000 7f89d1200640 1 -- 172.21.15.53:0/4131825611 shutdown_connections 2024-01-16T15:09:56.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.845+0000 7f89d1200640 1 -- 172.21.15.53:0/4131825611 wait complete. 2024-01-16T15:09:56.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.847+0000 7f89d1200640 1 Processor -- start 2024-01-16T15:09:56.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.847+0000 7f89d1200640 1 -- start start 2024-01-16T15:09:56.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89d1200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89cc13df80 0x7f89cc14e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89d1200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89d1200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc13d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89d1200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1f6950 con 0x7f89cc15f9c0 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89d1200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc1f6340 con 0x7f89cc13df80 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89d1200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89cc14f280 con 0x7f89cc13e360 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc13d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc13d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39436/0 (socket says 172.21.15.53:39436) 2024-01-16T15:09:56.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 -- 172.21.15.53:0/1099398858 learned_addr learned my addr 172.21.15.53:0/1099398858 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89c9400640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 -- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 msgr2=0x7f89cc0b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0b2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 -- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89cc13df80 msgr2=0x7f89cc14e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89cc13df80 0x7f89cc14e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.848+0000 7f89ca800640 1 -- 172.21.15.53:0/1099398858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89c4009b30 con 0x7f89cc15f9c0 2024-01-16T15:09:56.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.849+0000 7f89ca800640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc13d3a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f89c4036040 tx=0x7f89c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.849+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c403e450 con 0x7f89cc15f9c0 2024-01-16T15:09:56.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.849+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89c403e5d0 con 0x7f89cc15f9c0 2024-01-16T15:09:56.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.849+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89c403e980 con 0x7f89cc15f9c0 2024-01-16T15:09:56.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.849+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89cc14efa0 con 0x7f89cc15f9c0 2024-01-16T15:09:56.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.849+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89cc14e650 con 0x7f89cc15f9c0 2024-01-16T15:09:56.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.850+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89cc1f61a0 con 0x7f89cc15f9c0 2024-01-16T15:09:56.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.851+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f89c403eae0 con 0x7f89cc15f9c0 2024-01-16T15:09:56.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.852+0000 7f89c2000640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f89a406be90 0x7f89a406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:56.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.852+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f89c40ab420 con 0x7f89cc15f9c0 2024-01-16T15:09:56.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.853+0000 7f89c9e00640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f89a406be90 0x7f89a406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:56.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.854+0000 7f89c9e00640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f89a406be90 0x7f89a406e350 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f89bc006fd0 tx=0x7f89bc006bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:56.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:56.857+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f89cc1f61a0 con 0x7f89cc15f9c0 2024-01-16T15:09:57.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.085+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541411", "target": ["mon-mgr", ""]}) v1 -- 0x7f89cc06cff0 con 0x7f89a406be90 2024-01-16T15:09:57.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.433+0000 7f89c2000640 1 -- 172.21.15.53:0/1099398858 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f89bc009280 con 0x7f89a406be90 2024-01-16T15:09:57.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f89a406be90 msgr2=0x7f89a406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f89a406be90 0x7f89a406e350 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f89bc006fd0 tx=0x7f89bc006bc0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 msgr2=0x7f89cc13d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc13d3a0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f89c4036040 tx=0x7f89c4004a40 comp rx=0 tx=0).stop 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 shutdown_connections 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f89a406be90 0x7f89a406e350 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f89cc15f9c0 0x7f89cc13d3a0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f89cc13e360 0x7f89cc0b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 --2- 172.21.15.53:0/1099398858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f89cc13df80 0x7f89cc14e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.437+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 >> 172.21.15.53:0/1099398858 conn(0x7f89cc06a800 msgr2=0x7f89cc15d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.438+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 shutdown_connections 2024-01-16T15:09:57.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.438+0000 7f89d1200640 1 -- 172.21.15.53:0/1099398858 wait complete. 2024-01-16T15:09:57.444 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541412 2024-01-16T15:09:57.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.721+0000 7fde0be00640 1 Processor -- start 2024-01-16T15:09:57.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.721+0000 7fde0be00640 1 -- start start 2024-01-16T15:09:57.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.723+0000 7fde0be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde0c160920 0x7fde0c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.723+0000 7fde0be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c0d0a10 con 0x7fde0c0cfea0 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.723+0000 7fde0be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0c161e30 con 0x7fde0c0cf9f0 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.723+0000 7fde0be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde0c0d1f50 con 0x7fde0c160920 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.723+0000 7fde0aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde0c160920 0x7fde0c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.724+0000 7fde0aa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fde0c160920 0x7fde0c0b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39440/0 (socket says 172.21.15.53:39440) 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.724+0000 7fde0b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fde0c0cfea0 0x7fde0c161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48614/0 (socket says 172.21.15.53:48614) 2024-01-16T15:09:57.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.724+0000 7fde0aa00640 1 -- 172.21.15.53:0/2376278660 learned_addr learned my addr 172.21.15.53:0/2376278660 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:57.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.724+0000 7fde0aa00640 1 -- 172.21.15.53:0/2376278660 >> v1:172.21.15.146:6789/0 conn(0x7fde0c0cf9f0 legacy=0x7fde0c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:57.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.724+0000 7fde0aa00640 1 -- 172.21.15.53:0/2376278660 >> v1:172.21.15.53:6789/0 conn(0x7fde0c0cfea0 legacy=0x7fde0c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.724+0000 7fde0aa00640 1 -- 172.21.15.53:0/2376278660 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0c0cf080 con 0x7fde0c160920 2024-01-16T15:09:57.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.725+0000 7fde0aa00640 1 --2- 172.21.15.53:0/2376278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c0b3f30 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009870 tx=0x7fddfc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44333f1d701da395 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.726+0000 7fde09600640 1 -- 172.21.15.53:0/2376278660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc030860 con 0x7fde0c160920 2024-01-16T15:09:57.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.726+0000 7fde09600640 1 -- 172.21.15.53:0/2376278660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddfc0309c0 con 0x7fde0c160920 2024-01-16T15:09:57.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.726+0000 7fde09600640 1 -- 172.21.15.53:0/2376278660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddfc030d50 con 0x7fde0c160920 2024-01-16T15:09:57.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 -- 172.21.15.53:0/2376278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 msgr2=0x7fde0c0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 --2- 172.21.15.53:0/2376278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c0b3f30 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7fddfc009870 tx=0x7fddfc02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 -- 172.21.15.53:0/2376278660 shutdown_connections 2024-01-16T15:09:57.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 --2- 172.21.15.53:0/2376278660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c0b3f30 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 -- 172.21.15.53:0/2376278660 >> 172.21.15.53:0/2376278660 conn(0x7fde0c0d3090 msgr2=0x7fde0c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:57.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 -- 172.21.15.53:0/2376278660 shutdown_connections 2024-01-16T15:09:57.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.727+0000 7fde0be00640 1 -- 172.21.15.53:0/2376278660 wait complete. 2024-01-16T15:09:57.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.729+0000 7fde0be00640 1 Processor -- start 2024-01-16T15:09:57.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.729+0000 7fde0be00640 1 -- start start 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cf9f0 0x7fde0c151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c0cfea0 0x7fde0c15e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0b6160 con 0x7fde0c160920 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c0b8410 con 0x7fde0c0cf9f0 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0c15d8f0 con 0x7fde0c0cfea0 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39448/0 (socket says 172.21.15.53:39448) 2024-01-16T15:09:57.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.730+0000 7fde0b400640 1 -- 172.21.15.53:0/483709024 learned_addr learned my addr 172.21.15.53:0/483709024 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:57.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.731+0000 7fde0b400640 1 -- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c0cfea0 msgr2=0x7fde0c15e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.731+0000 7fde0b400640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c0cfea0 0x7fde0c15e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.731+0000 7fde0b400640 1 -- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cf9f0 msgr2=0x7fde0c151510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:57.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.731+0000 7fde0b400640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cf9f0 0x7fde0c151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.731+0000 7fde0b400640 1 -- 172.21.15.53:0/483709024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddfc0094d0 con 0x7fde0c160920 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.731+0000 7fde0b400640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c15e680 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c780 tx=0x7fde0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.732+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400ce90 con 0x7fde0c160920 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.732+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0400a640 con 0x7fde0c160920 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.732+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0c151a50 con 0x7fde0c160920 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.732+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0400a9f0 con 0x7fde0c160920 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.732+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde0c15d610 con 0x7fde0c160920 2024-01-16T15:09:57.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.733+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde0c067450 con 0x7fde0c160920 2024-01-16T15:09:57.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.734+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fde0400ab50 con 0x7fde0c160920 2024-01-16T15:09:57.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.735+0000 7fde02a00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406bee0 0x7fdde406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:57.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.735+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fde04087f40 con 0x7fde0c160920 2024-01-16T15:09:57.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.740+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde04058120 con 0x7fde0c160920 2024-01-16T15:09:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.749+0000 7fde0aa00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406bee0 0x7fdde406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:57.754+0000 7fde0aa00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406bee0 0x7fdde406e3a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fddfc036000 tx=0x7fddfc03b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.063+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541412", "target": ["mon-mgr", ""]}) v1 -- 0x7fde0c0ce7e0 con 0x7fdde406bee0 2024-01-16T15:09:58.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.381+0000 7fde02a00640 1 -- 172.21.15.53:0/483709024 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fddfc030bb0 con 0x7fdde406bee0 2024-01-16T15:09:58.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.387+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406bee0 msgr2=0x7fdde406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.387+0000 7fde0be00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406bee0 0x7fdde406e3a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fddfc036000 tx=0x7fddfc03b040 comp rx=0 tx=0).stop 2024-01-16T15:09:58.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.387+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 msgr2=0x7fde0c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.387+0000 7fde0be00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c15e680 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7fde0400c780 tx=0x7fde0400cc40 comp rx=0 tx=0).stop 2024-01-16T15:09:58.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.389+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 shutdown_connections 2024-01-16T15:09:58.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.389+0000 7fde0be00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdde406bee0 0x7fdde406e3a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.389+0000 7fde0be00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fde0c160920 0x7fde0c15e680 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.389+0000 7fde0be00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde0c0cfea0 0x7fde0c15e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.389+0000 7fde0be00640 1 --2- 172.21.15.53:0/483709024 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0c0cf9f0 0x7fde0c151510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.389+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 >> 172.21.15.53:0/483709024 conn(0x7fde0c0d3090 msgr2=0x7fde0c0b47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.390+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 shutdown_connections 2024-01-16T15:09:58.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.391+0000 7fde0be00640 1 -- 172.21.15.53:0/483709024 wait complete. 2024-01-16T15:09:58.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541413 2024-01-16T15:09:58.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.690+0000 7f5b16400640 1 Processor -- start 2024-01-16T15:09:58.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.691+0000 7f5b16400640 1 -- start start 2024-01-16T15:09:58.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.692+0000 7f5b16400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b100cd170 0x7f5b10071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.692+0000 7f5b16400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b10071690 con 0x7f5b1014b2c0 2024-01-16T15:09:58.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.692+0000 7f5b16400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b10071970 con 0x7f5b100cf9a0 2024-01-16T15:09:58.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.692+0000 7f5b16400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b1014b6a0 con 0x7f5b100cd170 2024-01-16T15:09:58.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b100cd170 0x7f5b10071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b100cd170 0x7f5b10071070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39450/0 (socket says 172.21.15.53:39450) 2024-01-16T15:09:58.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0fe00640 1 -- 172.21.15.53:0/3773887761 learned_addr learned my addr 172.21.15.53:0/3773887761 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:58.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0fe00640 1 -- 172.21.15.53:0/3773887761 >> v1:172.21.15.146:6789/0 conn(0x7f5b100cf9a0 legacy=0x7f5b10070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:58.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0fe00640 1 -- 172.21.15.53:0/3773887761 >> v1:172.21.15.53:6789/0 conn(0x7f5b1014b2c0 legacy=0x7f5b100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0fe00640 1 -- 172.21.15.53:0/3773887761 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b10071c50 con 0x7f5b100cd170 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.693+0000 7f5b0e000640 1 -- 172.21.15.53:0/3773887761 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658227038 0 0) 0x7f5b10071690 con 0x7f5b1014b2c0 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.694+0000 7f5b0fe00640 1 --2- 172.21.15.53:0/3773887761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cd170 0x7f5b10071070 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009e50 tx=0x7f5b0802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95642e0a72871351 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.695+0000 7f5b0e000640 1 -- 172.21.15.53:0/3773887761 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0802cb20 con 0x7f5b100cd170 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.695+0000 7f5b0e000640 1 -- 172.21.15.53:0/3773887761 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b0802cc80 con 0x7f5b100cd170 2024-01-16T15:09:58.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.695+0000 7f5b0e000640 1 -- 172.21.15.53:0/3773887761 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b080343f0 con 0x7f5b100cd170 2024-01-16T15:09:58.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.695+0000 7f5b16400640 1 -- 172.21.15.53:0/3773887761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cd170 msgr2=0x7f5b10071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.695+0000 7f5b16400640 1 --2- 172.21.15.53:0/3773887761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cd170 0x7f5b10071070 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7f5b08009e50 tx=0x7f5b0802bf80 comp rx=0 tx=0).stop 2024-01-16T15:09:58.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.696+0000 7f5b16400640 1 -- 172.21.15.53:0/3773887761 shutdown_connections 2024-01-16T15:09:58.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.696+0000 7f5b16400640 1 --2- 172.21.15.53:0/3773887761 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cd170 0x7f5b10071070 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.696+0000 7f5b16400640 1 -- 172.21.15.53:0/3773887761 >> 172.21.15.53:0/3773887761 conn(0x7f5b1006a800 msgr2=0x7f5b1006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:58.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.696+0000 7f5b16400640 1 -- 172.21.15.53:0/3773887761 shutdown_connections 2024-01-16T15:09:58.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.696+0000 7f5b16400640 1 -- 172.21.15.53:0/3773887761 wait complete. 2024-01-16T15:09:58.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.698+0000 7f5b16400640 1 Processor -- start 2024-01-16T15:09:58.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.698+0000 7f5b16400640 1 -- start start 2024-01-16T15:09:58.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.700+0000 7f5b16400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b100cd170 0x7f5b1013f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.700+0000 7f5b16400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 0x7f5b1015bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.700+0000 7f5b16400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1014b2c0 0x7f5b1015c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.700+0000 7f5b16400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b10163720 con 0x7f5b100cf9a0 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.700+0000 7f5b16400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1014a950 con 0x7f5b1014b2c0 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.700+0000 7f5b16400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b101630d0 con 0x7f5b100cd170 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.701+0000 7f5b0ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 0x7f5b1015bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.701+0000 7f5b0ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 0x7f5b1015bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39458/0 (socket says 172.21.15.53:39458) 2024-01-16T15:09:58.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.701+0000 7f5b0ea00640 1 -- 172.21.15.53:0/572602917 learned_addr learned my addr 172.21.15.53:0/572602917 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.701+0000 7f5b0ea00640 1 -- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b100cd170 msgr2=0x7f5b1013f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.701+0000 7f5b0ea00640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b100cd170 0x7f5b1013f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.702+0000 7f5b0ea00640 1 -- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1014b2c0 msgr2=0x7f5b1015c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.702+0000 7f5b0ea00640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1014b2c0 0x7f5b1015c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.702+0000 7f5b0ea00640 1 -- 172.21.15.53:0/572602917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08009b30 con 0x7f5b100cf9a0 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.702+0000 7f5b0ea00640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 0x7f5b1015bd00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c410 tx=0x7f5b0000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:58.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.703+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b000035f0 con 0x7f5b100cf9a0 2024-01-16T15:09:58.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.703+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00003750 con 0x7f5b100cf9a0 2024-01-16T15:09:58.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.703+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b101633b0 con 0x7f5b100cf9a0 2024-01-16T15:09:58.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.703+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b10163bb0 con 0x7f5b100cf9a0 2024-01-16T15:09:58.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.703+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00003b40 con 0x7f5b100cf9a0 2024-01-16T15:09:58.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.705+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae0005250 con 0x7f5b100cf9a0 2024-01-16T15:09:58.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.707+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f5b00003ca0 con 0x7f5b100cf9a0 2024-01-16T15:09:58.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.711+0000 7f5b07400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ae806bf60 0x7f5ae806e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:58.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.711+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5b00087ec0 con 0x7f5b100cf9a0 2024-01-16T15:09:58.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.716+0000 7f5b0f400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ae806bf60 0x7f5ae806e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:58.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.719+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b00054520 con 0x7f5b100cf9a0 2024-01-16T15:09:58.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:58.721+0000 7f5b0f400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ae806bf60 0x7f5ae806e420 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001560 tx=0x7f5af8007930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.048+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541413", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ae0000c80 con 0x7f5ae806bf60 2024-01-16T15:09:59.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.433+0000 7f5b07400640 1 -- 172.21.15.53:0/572602917 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5ae0000c80 con 0x7f5ae806bf60 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ae806bf60 msgr2=0x7f5ae806e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ae806bf60 0x7f5ae806e420 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5af8001560 tx=0x7f5af8007930 comp rx=0 tx=0).stop 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 msgr2=0x7f5b1015bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 0x7f5b1015bd00 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7f5b0000c410 tx=0x7f5b0000c8d0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 shutdown_connections 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ae806bf60 0x7f5ae806e420 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b1014b2c0 0x7f5b1015c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b100cf9a0 0x7f5b1015bd00 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 --2- 172.21.15.53:0/572602917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b100cd170 0x7f5b1013f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 >> 172.21.15.53:0/572602917 conn(0x7f5b1006a800 msgr2=0x7f5b100708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:59.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 shutdown_connections 2024-01-16T15:09:59.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.436+0000 7f5b16400640 1 -- 172.21.15.53:0/572602917 wait complete. 2024-01-16T15:09:59.444 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541414 2024-01-16T15:09:59.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.739+0000 7fc2aa200640 1 Processor -- start 2024-01-16T15:09:59.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.739+0000 7fc2aa200640 1 -- start start 2024-01-16T15:09:59.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2aa200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a415b7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2aa200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2a40cfec0 0x7fc2a40d0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2aa200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2a4144f50 con 0x7fc2a40b1bf0 2024-01-16T15:09:59.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2aa200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2a40d0890 con 0x7fc2a40cfec0 2024-01-16T15:09:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2aa200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a415edc0 con 0x7fc2a4144b70 2024-01-16T15:09:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2a3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2a40cfec0 0x7fc2a40d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2a3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2a40cfec0 0x7fc2a40d0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39462/0 (socket says 172.21.15.53:39462) 2024-01-16T15:09:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc2a3e00640 1 -- 172.21.15.53:0/1454012920 learned_addr learned my addr 172.21.15.53:0/1454012920 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.741+0000 7fc29be00640 1 --1- 172.21.15.53:0/1454012920 >> v1:172.21.15.53:6789/0 conn(0x7fc2a40b1bf0 0x7fc2a415f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48630/0 (socket says 172.21.15.53:48630) 2024-01-16T15:09:59.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.742+0000 7fc2a3400640 1 --2- 172.21.15.53:0/1454012920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a415b7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.742+0000 7fc2a3e00640 1 -- 172.21.15.53:0/1454012920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 msgr2=0x7fc2a415b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:09:59.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.742+0000 7fc2a3e00640 1 --2- 172.21.15.53:0/1454012920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a415b7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.742+0000 7fc2a3e00640 1 -- 172.21.15.53:0/1454012920 >> v1:172.21.15.53:6789/0 conn(0x7fc2a40b1bf0 legacy=0x7fc2a415f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.742+0000 7fc2a3e00640 1 -- 172.21.15.53:0/1454012920 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a40b2360 con 0x7fc2a40cfec0 2024-01-16T15:09:59.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.743+0000 7fc2a3e00640 1 --2- 172.21.15.53:0/1454012920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40cfec0 0x7fc2a40d0350 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009e50 tx=0x7fc29c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c164a8ce69801cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.744+0000 7fc2a2a00640 1 -- 172.21.15.53:0/1454012920 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c02cb00 con 0x7fc2a40cfec0 2024-01-16T15:09:59.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.744+0000 7fc2a2a00640 1 -- 172.21.15.53:0/1454012920 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc29c02cc60 con 0x7fc2a40cfec0 2024-01-16T15:09:59.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.744+0000 7fc2a2a00640 1 -- 172.21.15.53:0/1454012920 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c0343f0 con 0x7fc2a40cfec0 2024-01-16T15:09:59.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 -- 172.21.15.53:0/1454012920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40cfec0 msgr2=0x7fc2a40d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 --2- 172.21.15.53:0/1454012920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40cfec0 0x7fc2a40d0350 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009e50 tx=0x7fc29c02bf60 comp rx=0 tx=0).stop 2024-01-16T15:09:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 -- 172.21.15.53:0/1454012920 shutdown_connections 2024-01-16T15:09:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 --2- 172.21.15.53:0/1454012920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40cfec0 0x7fc2a40d0350 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 --2- 172.21.15.53:0/1454012920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a415b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 -- 172.21.15.53:0/1454012920 >> 172.21.15.53:0/1454012920 conn(0x7fc2a406b4f0 msgr2=0x7fc2a406b900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:09:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 -- 172.21.15.53:0/1454012920 shutdown_connections 2024-01-16T15:09:59.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.745+0000 7fc2aa200640 1 -- 172.21.15.53:0/1454012920 wait complete. 2024-01-16T15:09:59.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.747+0000 7fc2aa200640 1 Processor -- start 2024-01-16T15:09:59.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.748+0000 7fc2aa200640 1 -- start start 2024-01-16T15:09:59.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2aa200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 0x7fc2a415aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2aa200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 0x7fc2a41431b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2aa200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a41436f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2aa200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a412c6c0 con 0x7fc2a40b1bf0 2024-01-16T15:09:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2aa200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a40c9f30 con 0x7fc2a40cfec0 2024-01-16T15:09:59.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2aa200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a40ca210 con 0x7fc2a4144b70 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.749+0000 7fc2a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 0x7fc2a415aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.750+0000 7fc29be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 0x7fc2a41431b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.750+0000 7fc29be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 0x7fc2a41431b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:35626/0 (socket says 172.21.15.53:35626) 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.750+0000 7fc29be00640 1 -- 172.21.15.53:0/3909774774 learned_addr learned my addr 172.21.15.53:0/3909774774 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.750+0000 7fc2a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 0x7fc2a415aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39474/0 (socket says 172.21.15.53:39474) 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.750+0000 7fc2a3e00640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a41436f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.751+0000 7fc29be00640 1 -- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 msgr2=0x7fc2a41436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.751+0000 7fc29be00640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a41436f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.751+0000 7fc29be00640 1 -- 172.21.15.53:0/3909774774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 msgr2=0x7fc2a415aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:09:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.751+0000 7fc29be00640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 0x7fc2a415aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:09:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.751+0000 7fc29be00640 1 -- 172.21.15.53:0/3909774774 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc29c009b30 con 0x7fc2a40cfec0 2024-01-16T15:09:59.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.751+0000 7fc2a3400640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 0x7fc2a415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:09:59.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.752+0000 7fc29be00640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 0x7fc2a41431b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c4d0 tx=0x7fc28c00c990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.753+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c007480 con 0x7fc2a40cfec0 2024-01-16T15:09:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.753+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc28c0075e0 con 0x7fc2a40cfec0 2024-01-16T15:09:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.753+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a412c070 con 0x7fc2a40cfec0 2024-01-16T15:09:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.753+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc28c0049e0 con 0x7fc2a40cfec0 2024-01-16T15:09:59.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.753+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a415aff0 con 0x7fc2a40cfec0 2024-01-16T15:09:59.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.754+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2a413f9e0 con 0x7fc2a40cfec0 2024-01-16T15:09:59.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.758+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fc28c004b40 con 0x7fc2a40cfec0 2024-01-16T15:09:59.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.760+0000 7fc29b400640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc27c06be90 0x7fc27c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:09:59.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.760+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc28c088140 con 0x7fc2a40cfec0 2024-01-16T15:09:59.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.772+0000 7fc2a3400640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc27c06be90 0x7fc27c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:09:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.776+0000 7fc2a3400640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc27c06be90 0x7fc27c06e350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc294007d90 tx=0x7fc294008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:09:59.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:09:59.781+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc28c054850 con 0x7fc2a40cfec0 2024-01-16T15:10:00.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.157+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541414", "target": ["mon-mgr", ""]}) v1 -- 0x7fc2a40b2920 con 0x7fc27c06be90 2024-01-16T15:10:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.296+0000 7fc29b400640 1 -- 172.21.15.53:0/3909774774 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc2940079e0 con 0x7fc27c06be90 2024-01-16T15:10:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc27c06be90 msgr2=0x7fc27c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc27c06be90 0x7fc27c06e350 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fc294007d90 tx=0x7fc294008040 comp rx=0 tx=0).stop 2024-01-16T15:10:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 msgr2=0x7fc2a41431b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 0x7fc2a41431b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc28c00c4d0 tx=0x7fc28c00c990 comp rx=0 tx=0).stop 2024-01-16T15:10:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 shutdown_connections 2024-01-16T15:10:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc27c06be90 0x7fc27c06e350 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2a4144b70 0x7fc2a41436f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2a40cfec0 0x7fc2a41431b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 --2- 172.21.15.53:0/3909774774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2a40b1bf0 0x7fc2a415aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.298+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 >> 172.21.15.53:0/3909774774 conn(0x7fc2a406b4f0 msgr2=0x7fc2a406b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.299+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 shutdown_connections 2024-01-16T15:10:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.299+0000 7fc2aa200640 1 -- 172.21.15.53:0/3909774774 wait complete. 2024-01-16T15:10:00.292 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541415 2024-01-16T15:10:00.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.515+0000 7f5542800640 1 Processor -- start 2024-01-16T15:10:00.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.515+0000 7f5542800640 1 -- start start 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5542800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f553c13cfa0 0x7f553c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5542800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c0cd530 0x7f553c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5542800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f553c0cd9b0 0x7f553c13fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5542800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553c0cdec0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5542800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553c0ce1a0 con 0x7f553c13cfa0 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5542800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c0ce480 con 0x7f553c0cd530 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.517+0000 7f5541e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f553c0cd9b0 0x7f553c13fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f553c0cd9b0 0x7f553c13fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39484/0 (socket says 172.21.15.53:39484) 2024-01-16T15:10:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 -- 172.21.15.53:0/1296925541 learned_addr learned my addr 172.21.15.53:0/1296925541 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 -- 172.21.15.53:0/1296925541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c0cd530 msgr2=0x7f553c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 --2- 172.21.15.53:0/1296925541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c0cd530 0x7f553c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 -- 172.21.15.53:0/1296925541 >> v2:172.21.15.146:3300/0 conn(0x7f553c13cfa0 msgr2=0x7f553c15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 --2- 172.21.15.53:0/1296925541 >> v2:172.21.15.146:3300/0 conn(0x7f553c13cfa0 0x7f553c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.518+0000 7f5541e00640 1 -- 172.21.15.53:0/1296925541 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553c0ce760 con 0x7f553c0cd9b0 2024-01-16T15:10:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.519+0000 7f5541e00640 1 --2- 172.21.15.53:0/1296925541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c13fae0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f553c06cf00 tx=0x7f553402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15611a34dea86db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.519+0000 7f553be00640 1 -- 172.21.15.53:0/1296925541 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f553402c9e0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.519+0000 7f553be00640 1 -- 172.21.15.53:0/1296925541 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f553402cb40 con 0x7f553c0cd9b0 2024-01-16T15:10:00.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.519+0000 7f553be00640 1 -- 172.21.15.53:0/1296925541 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55340343f0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 -- 172.21.15.53:0/1296925541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 msgr2=0x7f553c13fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 --2- 172.21.15.53:0/1296925541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c13fae0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f553c06cf00 tx=0x7f553402be40 comp rx=0 tx=0).stop 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 -- 172.21.15.53:0/1296925541 shutdown_connections 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 --2- 172.21.15.53:0/1296925541 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c13fae0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 --2- 172.21.15.53:0/1296925541 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c0cd530 0x7f553c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 --2- 172.21.15.53:0/1296925541 >> v2:172.21.15.146:3300/0 conn(0x7f553c13cfa0 0x7f553c15afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 -- 172.21.15.53:0/1296925541 >> 172.21.15.53:0/1296925541 conn(0x7f553c06a800 msgr2=0x7f553c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:00.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 -- 172.21.15.53:0/1296925541 shutdown_connections 2024-01-16T15:10:00.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.520+0000 7f5542800640 1 -- 172.21.15.53:0/1296925541 wait complete. 2024-01-16T15:10:00.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.522+0000 7f5542800640 1 Processor -- start 2024-01-16T15:10:00.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.522+0000 7f5542800640 1 -- start start 2024-01-16T15:10:00.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.523+0000 7f5542800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f553c0cd530 0x7f553c135840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5542800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c1572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5542800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c13cfa0 0x7f553c13c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5542800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c155eb0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5542800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553c1552a0 con 0x7f553c0cd530 2024-01-16T15:10:00.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5542800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553c155580 con 0x7f553c13cfa0 2024-01-16T15:10:00.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c1572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c1572d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39496/0 (socket says 172.21.15.53:39496) 2024-01-16T15:10:00.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 -- 172.21.15.53:0/4254241392 learned_addr learned my addr 172.21.15.53:0/4254241392 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 -- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c13cfa0 msgr2=0x7f553c13c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c13cfa0 0x7f553c13c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 -- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f553c0cd530 msgr2=0x7f553c135840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f553c0cd530 0x7f553c135840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.524+0000 7f5540a00640 1 -- 172.21.15.53:0/4254241392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534009b30 con 0x7f553c0cd9b0 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.525+0000 7f5540a00640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c1572d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f552400b450 tx=0x7f552400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.525+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524011890 con 0x7f553c0cd9b0 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.525+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55240119f0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.525+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524011da0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.525+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553c155860 con 0x7f553c0cd9b0 2024-01-16T15:10:00.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.525+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553c13a5b0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.526+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f553c13a7d0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.527+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f552401a400 con 0x7f553c0cd9b0 2024-01-16T15:10:00.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.528+0000 7f5539600640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f551406be90 0x7f551406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:00.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.529+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5524088be0 con 0x7f553c0cd9b0 2024-01-16T15:10:00.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.529+0000 7f5541400640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f551406be90 0x7f551406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:00.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.530+0000 7f5541400640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f551406be90 0x7f551406e350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f552c009670 tx=0x7f552c006bf0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:00.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.533+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5524055240 con 0x7f553c0cd9b0 2024-01-16T15:10:00.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:00.765+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541415", "target": ["mon-mgr", ""]}) v1 -- 0x7f553c067450 con 0x7f551406be90 2024-01-16T15:10:01.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.188+0000 7f5539600640 1 -- 172.21.15.53:0/4254241392 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f553c067450 con 0x7f551406be90 2024-01-16T15:10:01.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f551406be90 msgr2=0x7f551406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f551406be90 0x7f551406e350 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f552c009670 tx=0x7f552c006bf0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 msgr2=0x7f553c1572d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c1572d0 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f552400b450 tx=0x7f552400b910 comp rx=0 tx=0).stop 2024-01-16T15:10:01.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 shutdown_connections 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f551406be90 0x7f551406e350 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f553c13cfa0 0x7f553c13c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f553c0cd9b0 0x7f553c1572d0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 --2- 172.21.15.53:0/4254241392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f553c0cd530 0x7f553c135840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 >> 172.21.15.53:0/4254241392 conn(0x7f553c06a800 msgr2=0x7f553c15b9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 shutdown_connections 2024-01-16T15:10:01.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.192+0000 7f5542800640 1 -- 172.21.15.53:0/4254241392 wait complete. 2024-01-16T15:10:01.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541416 2024-01-16T15:10:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.517+0000 7f2825000640 1 Processor -- start 2024-01-16T15:10:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.517+0000 7f2825000640 1 -- start start 2024-01-16T15:10:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.519+0000 7f2825000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200d17e0 0x7f28200d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.519+0000 7f2825000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f28200b3890 0x7f28200b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.519+0000 7f2825000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28200d2180 con 0x7f282015d610 2024-01-16T15:10:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.519+0000 7f2825000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28200d1390 con 0x7f28200b3890 2024-01-16T15:10:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.519+0000 7f2825000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28200b3400 con 0x7f28200d17e0 2024-01-16T15:10:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.520+0000 7f281de00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f282015d610 0x7f28200ce820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:46122/0 (socket says 172.21.15.53:46122) 2024-01-16T15:10:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.520+0000 7f281de00640 1 -- 172.21.15.53:0/4367364 learned_addr learned my addr 172.21.15.53:0/4367364 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.520+0000 7f281e800640 1 --2- 172.21.15.53:0/4367364 >> v2:172.21.15.53:3300/0 conn(0x7f28200b3890 0x7f28200b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.520+0000 7f281d400640 1 --2- 172.21.15.53:0/4367364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200d17e0 0x7f28200d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.521+0000 7f281ca00640 1 -- 172.21.15.53:0/4367364 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3573891834 0 0) 0x7f28200d2180 con 0x7f282015d610 2024-01-16T15:10:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.521+0000 7f281ca00640 1 -- 172.21.15.53:0/4367364 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2800003520 con 0x7f282015d610 2024-01-16T15:10:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.521+0000 7f281e800640 1 -- 172.21.15.53:0/4367364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200d17e0 msgr2=0x7f28200d1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.521+0000 7f281e800640 1 --2- 172.21.15.53:0/4367364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200d17e0 0x7f28200d1c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.521+0000 7f281e800640 1 -- 172.21.15.53:0/4367364 >> v1:172.21.15.146:6789/0 conn(0x7f282015d610 legacy=0x7f28200ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.521+0000 7f281e800640 1 -- 172.21.15.53:0/4367364 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28200b27f0 con 0x7f28200b3890 2024-01-16T15:10:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.523+0000 7f281e800640 1 --2- 172.21.15.53:0/4367364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28200b3890 0x7f28200b3d20 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f2818009d90 tx=0x7f281802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3a88c766e9ca75a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.524+0000 7f281ca00640 1 -- 172.21.15.53:0/4367364 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281802cc70 con 0x7f28200b3890 2024-01-16T15:10:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.524+0000 7f281ca00640 1 -- 172.21.15.53:0/4367364 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281802cdd0 con 0x7f28200b3890 2024-01-16T15:10:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.524+0000 7f281ca00640 1 -- 172.21.15.53:0/4367364 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2818034660 con 0x7f28200b3890 2024-01-16T15:10:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.525+0000 7f2825000640 1 -- 172.21.15.53:0/4367364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28200b3890 msgr2=0x7f28200b3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.525+0000 7f2825000640 1 --2- 172.21.15.53:0/4367364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28200b3890 0x7f28200b3d20 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f2818009d90 tx=0x7f281802bee0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.526+0000 7f2825000640 1 -- 172.21.15.53:0/4367364 shutdown_connections 2024-01-16T15:10:01.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.526+0000 7f2825000640 1 --2- 172.21.15.53:0/4367364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f28200b3890 0x7f28200b3d20 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.526+0000 7f2825000640 1 --2- 172.21.15.53:0/4367364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200d17e0 0x7f28200d1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.526+0000 7f2825000640 1 -- 172.21.15.53:0/4367364 >> 172.21.15.53:0/4367364 conn(0x7f282009c540 msgr2=0x7f28200d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:01.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.526+0000 7f2825000640 1 -- 172.21.15.53:0/4367364 shutdown_connections 2024-01-16T15:10:01.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.526+0000 7f2825000640 1 -- 172.21.15.53:0/4367364 wait complete. 2024-01-16T15:10:01.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.530+0000 7f2825000640 1 Processor -- start 2024-01-16T15:10:01.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.530+0000 7f2825000640 1 -- start start 2024-01-16T15:10:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.532+0000 7f2825000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200b3890 0x7f28200c9b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.532+0000 7f2825000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28200d17e0 0x7f28201f2970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.533+0000 7f2825000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 0x7f28201f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.533+0000 7f2825000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28201f6270 con 0x7f282015d610 2024-01-16T15:10:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.533+0000 7f2825000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28201f5f60 con 0x7f28200d17e0 2024-01-16T15:10:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.533+0000 7f2825000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28201f2eb0 con 0x7f28200b3890 2024-01-16T15:10:01.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.534+0000 7f281e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 0x7f28201f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.534+0000 7f281e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 0x7f28201f6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39512/0 (socket says 172.21.15.53:39512) 2024-01-16T15:10:01.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.534+0000 7f281e800640 1 -- 172.21.15.53:0/2037384443 learned_addr learned my addr 172.21.15.53:0/2037384443 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:01.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.534+0000 7f281d400640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28200d17e0 0x7f28201f2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.534+0000 7f281de00640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200b3890 0x7f28200c9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.535+0000 7f281e800640 1 -- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200b3890 msgr2=0x7f28200c9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.535+0000 7f281e800640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200b3890 0x7f28200c9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.535+0000 7f281e800640 1 -- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28200d17e0 msgr2=0x7f28201f2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.535+0000 7f281e800640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28200d17e0 0x7f28201f2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.535+0000 7f281e800640 1 -- 172.21.15.53:0/2037384443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2818009a70 con 0x7f282015d610 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.535+0000 7f281de00640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200b3890 0x7f28200c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:01.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.536+0000 7f281e800640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 0x7f28201f6670 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2818009eb0 tx=0x7f2818004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:01.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.537+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281803e610 con 0x7f282015d610 2024-01-16T15:10:01.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.537+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281803e770 con 0x7f282015d610 2024-01-16T15:10:01.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.537+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281803eb20 con 0x7f282015d610 2024-01-16T15:10:01.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.538+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28201f3130 con 0x7f282015d610 2024-01-16T15:10:01.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.538+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28201f36f0 con 0x7f282015d610 2024-01-16T15:10:01.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.540+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28201f38e0 con 0x7f282015d610 2024-01-16T15:10:01.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.542+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f281803b050 con 0x7f282015d610 2024-01-16T15:10:01.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.544+0000 7f2816000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27f806bee0 0x7f27f806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:01.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.544+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f28180aae40 con 0x7f282015d610 2024-01-16T15:10:01.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.553+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2818077860 con 0x7f282015d610 2024-01-16T15:10:01.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.563+0000 7f281de00640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27f806bee0 0x7f27f806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:01.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.574+0000 7f281de00640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27f806bee0 0x7f27f806e3a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2810005880 tx=0x7f2810009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:01.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:01.940+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541416", "target": ["mon-mgr", ""]}) v1 -- 0x7f28200ca0c0 con 0x7f27f806bee0 2024-01-16T15:10:02.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.429+0000 7f2816000640 1 -- 172.21.15.53:0/2037384443 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2810001350 con 0x7f27f806bee0 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27f806bee0 msgr2=0x7f27f806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27f806bee0 0x7f27f806e3a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f2810005880 tx=0x7f2810009040 comp rx=0 tx=0).stop 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 msgr2=0x7f28201f6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 0x7f28201f6670 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2818009eb0 tx=0x7f2818004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 shutdown_connections 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f27f806bee0 0x7f27f806e3a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282015d610 0x7f28201f6670 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f28200d17e0 0x7f28201f2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 --2- 172.21.15.53:0/2037384443 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f28200b3890 0x7f28200c9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 >> 172.21.15.53:0/2037384443 conn(0x7f282009c540 msgr2=0x7f28200b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 shutdown_connections 2024-01-16T15:10:02.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.434+0000 7f2825000640 1 -- 172.21.15.53:0/2037384443 wait complete. 2024-01-16T15:10:02.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541417 2024-01-16T15:10:02.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.779+0000 7fcf58e00640 1 Processor -- start 2024-01-16T15:10:02.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.779+0000 7fcf58e00640 1 -- start start 2024-01-16T15:10:02.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.781+0000 7fcf58e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf54160700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.781+0000 7fcf58e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcf540d3910 0x7fcf54160c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.781+0000 7fcf58e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf540d4160 con 0x7fcf540d00d0 2024-01-16T15:10:02.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.781+0000 7fcf58e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf540ceff0 con 0x7fcf540d3910 2024-01-16T15:10:02.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.781+0000 7fcf58e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540cf410 con 0x7fcf540cfcf0 2024-01-16T15:10:02.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.783+0000 7fcf51400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcf540d3910 0x7fcf54160c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.783+0000 7fcf51400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcf540d3910 0x7fcf54160c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:35388/0 (socket says 172.21.15.53:35388) 2024-01-16T15:10:02.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.783+0000 7fcf51400640 1 -- 172.21.15.53:0/1570408770 learned_addr learned my addr 172.21.15.53:0/1570408770 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:02.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.785+0000 7fcf51400640 1 -- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 msgr2=0x7fcf54160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.785+0000 7fcf51400640 1 --2- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf54160700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.785+0000 7fcf51400640 1 -- 172.21.15.53:0/1570408770 >> v1:172.21.15.146:6789/0 conn(0x7fcf540d00d0 legacy=0x7fcf540b2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.785+0000 7fcf51400640 1 -- 172.21.15.53:0/1570408770 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf540b2420 con 0x7fcf540d3910 2024-01-16T15:10:02.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.787+0000 7fcf51400640 1 --2- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d3910 0x7fcf54160c40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0098a0 tx=0x7fcf3c02b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c9691ad3013fec2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.789+0000 7fcf50a00640 1 -- 172.21.15.53:0/1570408770 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c031640 con 0x7fcf540d3910 2024-01-16T15:10:02.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.789+0000 7fcf50a00640 1 -- 172.21.15.53:0/1570408770 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf3c0317a0 con 0x7fcf540d3910 2024-01-16T15:10:02.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.789+0000 7fcf50a00640 1 -- 172.21.15.53:0/1570408770 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c031b30 con 0x7fcf540d3910 2024-01-16T15:10:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 -- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d3910 msgr2=0x7fcf54160c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 --2- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d3910 0x7fcf54160c40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fcf3c0098a0 tx=0x7fcf3c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 -- 172.21.15.53:0/1570408770 shutdown_connections 2024-01-16T15:10:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 --2- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d3910 0x7fcf54160c40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 --2- 172.21.15.53:0/1570408770 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf54160700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 -- 172.21.15.53:0/1570408770 >> 172.21.15.53:0/1570408770 conn(0x7fcf5409c530 msgr2=0x7fcf540d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 -- 172.21.15.53:0/1570408770 shutdown_connections 2024-01-16T15:10:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.790+0000 7fcf58e00640 1 -- 172.21.15.53:0/1570408770 wait complete. 2024-01-16T15:10:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.792+0000 7fcf58e00640 1 Processor -- start 2024-01-16T15:10:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.793+0000 7fcf58e00640 1 -- start start 2024-01-16T15:10:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.794+0000 7fcf58e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf540b6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.794+0000 7fcf58e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d00d0 0x7fcf5415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.794+0000 7fcf58e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 0x7fcf541f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.794+0000 7fcf58e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540b7120 con 0x7fcf540d3910 2024-01-16T15:10:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.794+0000 7fcf58e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540c88e0 con 0x7fcf540d00d0 2024-01-16T15:10:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.794+0000 7fcf58e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf540b79b0 con 0x7fcf540cfcf0 2024-01-16T15:10:02.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.795+0000 7fcf52800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 0x7fcf541f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.795+0000 7fcf52800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 0x7fcf541f6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33156/0 (socket says 172.21.15.53:33156) 2024-01-16T15:10:02.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.795+0000 7fcf52800640 1 -- 172.21.15.53:0/864235336 learned_addr learned my addr 172.21.15.53:0/864235336 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:02.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.795+0000 7fcf51400640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d00d0 0x7fcf5415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.795+0000 7fcf51e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf540b6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.796+0000 7fcf52800640 1 -- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 msgr2=0x7fcf540b6050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.796+0000 7fcf52800640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf540b6050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.796+0000 7fcf52800640 1 -- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d00d0 msgr2=0x7fcf5415d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.796+0000 7fcf52800640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d00d0 0x7fcf5415d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.796+0000 7fcf52800640 1 -- 172.21.15.53:0/864235336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf3c0094d0 con 0x7fcf540d3910 2024-01-16T15:10:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.796+0000 7fcf52800640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 0x7fcf541f6cb0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00c740 tx=0x7fcf4c00cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.797+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c006d50 con 0x7fcf540d3910 2024-01-16T15:10:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.797+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf540ca2d0 con 0x7fcf540d3910 2024-01-16T15:10:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.798+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf540b6b60 con 0x7fcf540d3910 2024-01-16T15:10:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.798+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf4c00ce50 con 0x7fcf540d3910 2024-01-16T15:10:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.798+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c003bb0 con 0x7fcf540d3910 2024-01-16T15:10:02.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.800+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf24005250 con 0x7fcf540d3910 2024-01-16T15:10:02.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.802+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fcf4c003d10 con 0x7fcf540d3910 2024-01-16T15:10:02.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.803+0000 7fcf4a000640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf2c06be90 0x7fcf2c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:02.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.803+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcf4c088280 con 0x7fcf540d3910 2024-01-16T15:10:02.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.807+0000 7fcf51e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf2c06be90 0x7fcf2c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:02.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.808+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf4c054940 con 0x7fcf540d3910 2024-01-16T15:10:02.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:02.815+0000 7fcf51e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf2c06be90 0x7fcf2c06e350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcf440098e0 tx=0x7fcf44009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:03.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.194+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541417", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf24000c80 con 0x7fcf2c06be90 2024-01-16T15:10:03.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.690+0000 7fcf4a000640 1 -- 172.21.15.53:0/864235336 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcf24000c80 con 0x7fcf2c06be90 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf2c06be90 msgr2=0x7fcf2c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf2c06be90 0x7fcf2c06e350 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fcf440098e0 tx=0x7fcf44009060 comp rx=0 tx=0).stop 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 msgr2=0x7fcf541f6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 0x7fcf541f6cb0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7fcf4c00c740 tx=0x7fcf4c00cc00 comp rx=0 tx=0).stop 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 shutdown_connections 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf2c06be90 0x7fcf2c06e350 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf540d3910 0x7fcf541f6cb0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf540d00d0 0x7fcf5415d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 --2- 172.21.15.53:0/864235336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcf540cfcf0 0x7fcf540b6050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 >> 172.21.15.53:0/864235336 conn(0x7fcf5409c530 msgr2=0x7fcf540b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:03.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 shutdown_connections 2024-01-16T15:10:03.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.694+0000 7fcf58e00640 1 -- 172.21.15.53:0/864235336 wait complete. 2024-01-16T15:10:03.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541418 2024-01-16T15:10:03.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.971+0000 7f3032a00640 1 Processor -- start 2024-01-16T15:10:03.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.971+0000 7f3032a00640 1 -- start start 2024-01-16T15:10:03.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.972+0000 7f3032a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:03.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.972+0000 7f3032a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c06c470 con 0x7f302c0cf9a0 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f3032a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c06c750 con 0x7f302c13e360 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f3032a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c06ca30 con 0x7f302c13df80 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f302b400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f302c13e360 0x7f302c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:40228/0 (socket says 172.21.15.53:40228) 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f302b400640 1 -- 172.21.15.53:0/4198532691 learned_addr learned my addr 172.21.15.53:0/4198532691 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f302be00640 1 --2- 172.21.15.53:0/4198532691 >> v2:172.21.15.146:3300/0 conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f302aa00640 1 -- 172.21.15.53:0/4198532691 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840333296 0 0) 0x7f302c06c470 con 0x7f302c0cf9a0 2024-01-16T15:10:03.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.973+0000 7f302aa00640 1 -- 172.21.15.53:0/4198532691 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c003520 con 0x7f302c0cf9a0 2024-01-16T15:10:03.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.974+0000 7f302aa00640 1 -- 172.21.15.53:0/4198532691 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270476961 0 0) 0x7f302c06c750 con 0x7f302c13e360 2024-01-16T15:10:03.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.974+0000 7f302be00640 1 -- 172.21.15.53:0/4198532691 >> v1:172.21.15.146:6789/0 conn(0x7f302c13e360 legacy=0x7f302c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.974+0000 7f302be00640 1 -- 172.21.15.53:0/4198532691 >> v1:172.21.15.53:6789/0 conn(0x7f302c0cf9a0 legacy=0x7f302c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.974+0000 7f302be00640 1 -- 172.21.15.53:0/4198532691 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c06cd10 con 0x7f302c13df80 2024-01-16T15:10:03.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.975+0000 7f302be00640 1 --2- 172.21.15.53:0/4198532691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15bee0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f301c0097a0 tx=0x7f301c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cf97deeab912a77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:03.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.975+0000 7f302aa00640 1 -- 172.21.15.53:0/4198532691 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c030610 con 0x7f302c13df80 2024-01-16T15:10:03.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.975+0000 7f302aa00640 1 -- 172.21.15.53:0/4198532691 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c030770 con 0x7f302c13df80 2024-01-16T15:10:03.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.975+0000 7f302aa00640 1 -- 172.21.15.53:0/4198532691 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c030b00 con 0x7f302c13df80 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 -- 172.21.15.53:0/4198532691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 msgr2=0x7f302c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 --2- 172.21.15.53:0/4198532691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15bee0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f301c0097a0 tx=0x7f301c02b960 comp rx=0 tx=0).stop 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 -- 172.21.15.53:0/4198532691 shutdown_connections 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 --2- 172.21.15.53:0/4198532691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 -- 172.21.15.53:0/4198532691 >> 172.21.15.53:0/4198532691 conn(0x7f302c06a800 msgr2=0x7f302c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 -- 172.21.15.53:0/4198532691 shutdown_connections 2024-01-16T15:10:03.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.976+0000 7f3032a00640 1 -- 172.21.15.53:0/4198532691 wait complete. 2024-01-16T15:10:03.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.978+0000 7f3032a00640 1 Processor -- start 2024-01-16T15:10:03.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.978+0000 7f3032a00640 1 -- start start 2024-01-16T15:10:03.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f3032a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c14d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:03.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f3032a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c153c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f3032a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c13d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f3032a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c14da50 con 0x7f302c0cf9a0 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f3032a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c070480 con 0x7f302c13df80 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f3032a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c15a750 con 0x7f302c13e360 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f302be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c14d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f302b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c153c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f302be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c14d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33168/0 (socket says 172.21.15.53:33168) 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.979+0000 7f302be00640 1 -- 172.21.15.53:0/1470150782 learned_addr learned my addr 172.21.15.53:0/1470150782 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f3030a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c13d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302be00640 1 -- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 msgr2=0x7f302c13d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302be00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c13d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302be00640 1 -- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 msgr2=0x7f302c153c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:03.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302be00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c153c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:03.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302be00640 1 -- 172.21.15.53:0/1470150782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c009400 con 0x7f302c0cf9a0 2024-01-16T15:10:03.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302b400640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.980+0000 7f302be00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c14d510 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f301c0098c0 tx=0x7f301c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.981+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c03e040 con 0x7f302c0cf9a0 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.981+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c039c10 con 0x7f302c0cf9a0 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.981+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c030e30 con 0x7f302c0cf9a0 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.981+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c159eb0 con 0x7f302c0cf9a0 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.981+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c15a470 con 0x7f302c0cf9a0 2024-01-16T15:10:03.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.982+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c0675d0 con 0x7f302c0cf9a0 2024-01-16T15:10:03.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.983+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f301c043390 con 0x7f302c0cf9a0 2024-01-16T15:10:03.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.984+0000 7f3023e00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406be90 0x7f300406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:03.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.984+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f301c0aa980 con 0x7f302c0cf9a0 2024-01-16T15:10:03.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.984+0000 7f302b400640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406be90 0x7f300406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:03.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.985+0000 7f302b400640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406be90 0x7f300406e350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3014007b90 tx=0x7f3014007ae0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:03.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:03.989+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f301c077150 con 0x7f302c0cf9a0 2024-01-16T15:10:04.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:04.219+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541418", "target": ["mon-mgr", ""]}) v1 -- 0x7f302c15f9c0 con 0x7f300406be90 2024-01-16T15:10:11.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.145+0000 7f3023e00640 1 -- 172.21.15.53:0/1470150782 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3014004b40 con 0x7f300406be90 2024-01-16T15:10:11.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.150+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406be90 msgr2=0x7f300406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.150+0000 7f3032a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406be90 0x7f300406e350 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3014007b90 tx=0x7f3014007ae0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.150+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 msgr2=0x7f302c14d510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.150+0000 7f3032a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c14d510 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f301c0098c0 tx=0x7f301c004730 comp rx=0 tx=0).stop 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 shutdown_connections 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406be90 0x7f300406e350 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c13d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c153c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 --2- 172.21.15.53:0/1470150782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c14d510 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 >> 172.21.15.53:0/1470150782 conn(0x7f302c06a800 msgr2=0x7f302c15e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:11.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.151+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 shutdown_connections 2024-01-16T15:10:11.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.152+0000 7f3032a00640 1 -- 172.21.15.53:0/1470150782 wait complete. 2024-01-16T15:10:11.166 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541419 2024-01-16T15:10:11.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.468+0000 7f057ae00640 1 Processor -- start 2024-01-16T15:10:11.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.468+0000 7f057ae00640 1 -- start start 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.470+0000 7f057ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f057413e360 0x7f05740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.470+0000 7f057ae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f057415f9c0 0x7f05740cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.470+0000 7f057ae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f057415c5f0 con 0x7f057413df80 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.471+0000 7f057ae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057406c470 con 0x7f057415f9c0 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.471+0000 7f057ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f057406c750 con 0x7f057413e360 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.471+0000 7f0578e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f057415f9c0 0x7f05740cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.471+0000 7f0578e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f057415f9c0 0x7f05740cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33182/0 (socket says 172.21.15.53:33182) 2024-01-16T15:10:11.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.471+0000 7f0578e00640 1 -- 172.21.15.53:0/205409507 learned_addr learned my addr 172.21.15.53:0/205409507 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.472+0000 7f0578e00640 1 -- 172.21.15.53:0/205409507 >> v2:172.21.15.146:3300/0 conn(0x7f057413e360 msgr2=0x7f05740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.472+0000 7f0578e00640 1 --2- 172.21.15.53:0/205409507 >> v2:172.21.15.146:3300/0 conn(0x7f057413e360 0x7f05740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.472+0000 7f0578e00640 1 -- 172.21.15.53:0/205409507 >> v1:172.21.15.146:6789/0 conn(0x7f057413df80 legacy=0x7f057415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:11.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.472+0000 7f0578e00640 1 -- 172.21.15.53:0/205409507 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f057406ca30 con 0x7f057415f9c0 2024-01-16T15:10:11.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.474+0000 7f0578e00640 1 --2- 172.21.15.53:0/205409507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f05740cf7d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f05740ccae0 tx=0x7f056c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5107e60e773c76 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.474+0000 7f0572a00640 1 -- 172.21.15.53:0/205409507 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c02ca60 con 0x7f057415f9c0 2024-01-16T15:10:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f0572a00640 1 -- 172.21.15.53:0/205409507 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c02cbc0 con 0x7f057415f9c0 2024-01-16T15:10:11.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f0572a00640 1 -- 172.21.15.53:0/205409507 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c0343f0 con 0x7f057415f9c0 2024-01-16T15:10:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f057ae00640 1 -- 172.21.15.53:0/205409507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 msgr2=0x7f05740cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f057ae00640 1 --2- 172.21.15.53:0/205409507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f05740cf7d0 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f05740ccae0 tx=0x7f056c02bec0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f057ae00640 1 -- 172.21.15.53:0/205409507 shutdown_connections 2024-01-16T15:10:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f057ae00640 1 --2- 172.21.15.53:0/205409507 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f05740cf7d0 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f057ae00640 1 --2- 172.21.15.53:0/205409507 >> v2:172.21.15.146:3300/0 conn(0x7f057413e360 0x7f05740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.475+0000 7f057ae00640 1 -- 172.21.15.53:0/205409507 >> 172.21.15.53:0/205409507 conn(0x7f057406a800 msgr2=0x7f057406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:11.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.476+0000 7f057ae00640 1 -- 172.21.15.53:0/205409507 shutdown_connections 2024-01-16T15:10:11.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.476+0000 7f057ae00640 1 -- 172.21.15.53:0/205409507 wait complete. 2024-01-16T15:10:11.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.478+0000 7f057ae00640 1 Processor -- start 2024-01-16T15:10:11.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.479+0000 7f057ae00640 1 -- start start 2024-01-16T15:10:11.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.480+0000 7f057ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057413df80 0x7f0574156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:11.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f057ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057413e360 0x7f057413d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:11.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f057ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f057415a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:11.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f057ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057414aa80 con 0x7f057415f9c0 2024-01-16T15:10:11.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f057ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0574149e70 con 0x7f057413df80 2024-01-16T15:10:11.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f057ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057414a150 con 0x7f057413e360 2024-01-16T15:10:11.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f0578e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f057415a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:11.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f0578e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f057415a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33186/0 (socket says 172.21.15.53:33186) 2024-01-16T15:10:11.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.481+0000 7f0578e00640 1 -- 172.21.15.53:0/1971841038 learned_addr learned my addr 172.21.15.53:0/1971841038 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.482+0000 7f0578e00640 1 -- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057413e360 msgr2=0x7f057413d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.482+0000 7f0578e00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057413e360 0x7f057413d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.482+0000 7f0578e00640 1 -- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057413df80 msgr2=0x7f0574156350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.482+0000 7f0578e00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057413df80 0x7f0574156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.482+0000 7f0578e00640 1 -- 172.21.15.53:0/1971841038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f056c009b30 con 0x7f057415f9c0 2024-01-16T15:10:11.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.483+0000 7f0578e00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f057415a790 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f056c036040 tx=0x7f056c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.483+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c03e450 con 0x7f057415f9c0 2024-01-16T15:10:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.483+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c03e5d0 con 0x7f057415f9c0 2024-01-16T15:10:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.483+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c03e980 con 0x7f057415f9c0 2024-01-16T15:10:11.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.483+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057414a430 con 0x7f057415f9c0 2024-01-16T15:10:11.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.484+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057413dc70 con 0x7f057415f9c0 2024-01-16T15:10:11.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.488+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f056c03eae0 con 0x7f057415f9c0 2024-01-16T15:10:11.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.489+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05740675d0 con 0x7f057415f9c0 2024-01-16T15:10:11.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.490+0000 7f056be00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f054c06be90 0x7f054c06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:11.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.491+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f056c0ab470 con 0x7f057415f9c0 2024-01-16T15:10:11.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.497+0000 7f0573e00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f054c06be90 0x7f054c06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:11.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.502+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f056c0e0e90 con 0x7f057415f9c0 2024-01-16T15:10:11.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.510+0000 7f0573e00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f054c06be90 0x7f054c06e350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0564006fd0 tx=0x7f0564006bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:11.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:11.865+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541419", "target": ["mon-mgr", ""]}) v1 -- 0x7f057406cff0 con 0x7f054c06be90 2024-01-16T15:10:12.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.417+0000 7f056be00640 1 -- 172.21.15.53:0/1971841038 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0564009280 con 0x7f054c06be90 2024-01-16T15:10:12.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.421+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f054c06be90 msgr2=0x7f054c06e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:12.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.421+0000 7f057ae00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f054c06be90 0x7f054c06e350 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f0564006fd0 tx=0x7f0564006bc0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.421+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 msgr2=0x7f057415a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:12.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.421+0000 7f057ae00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f057415a790 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f056c036040 tx=0x7f056c004a40 comp rx=0 tx=0).stop 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 shutdown_connections 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f054c06be90 0x7f054c06e350 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f057415f9c0 0x7f057415a790 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f057413e360 0x7f057413d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 --2- 172.21.15.53:0/1971841038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f057413df80 0x7f0574156350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 >> 172.21.15.53:0/1971841038 conn(0x7f057406a800 msgr2=0x7f05740af380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 shutdown_connections 2024-01-16T15:10:12.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.422+0000 7f057ae00640 1 -- 172.21.15.53:0/1971841038 wait complete. 2024-01-16T15:10:12.426 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone541420 2024-01-16T15:10:12.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.756+0000 7fba13000640 1 Processor -- start 2024-01-16T15:10:12.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.757+0000 7fba13000640 1 -- start start 2024-01-16T15:10:12.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.760+0000 7fba13000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c0cf9a0 0x7fba0c070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:12.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.760+0000 7fba13000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba0c070ea0 0x7fba0c0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:12.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.760+0000 7fba13000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0c071690 con 0x7fba0c14b2c0 2024-01-16T15:10:12.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.760+0000 7fba13000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0c14b6a0 con 0x7fba0c070ea0 2024-01-16T15:10:12.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.760+0000 7fba13000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c071970 con 0x7fba0c0cf9a0 2024-01-16T15:10:12.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.762+0000 7fba11000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba0c070ea0 0x7fba0c0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:12.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.762+0000 7fba0be00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fba0c14b2c0 0x7fba0c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:41246/0 (socket says 172.21.15.53:41246) 2024-01-16T15:10:12.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.762+0000 7fba0be00640 1 -- 172.21.15.53:0/3194097306 learned_addr learned my addr 172.21.15.53:0/3194097306 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:12.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.762+0000 7fba03e00640 1 --2- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c0cf9a0 0x7fba0c070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:12.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.764+0000 7fba0b400640 1 -- 172.21.15.53:0/3194097306 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1388265728 0 0) 0x7fba0c071690 con 0x7fba0c14b2c0 2024-01-16T15:10:12.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.764+0000 7fba0b400640 1 -- 172.21.15.53:0/3194097306 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec003520 con 0x7fba0c14b2c0 2024-01-16T15:10:12.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.764+0000 7fba11000640 1 -- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c0cf9a0 msgr2=0x7fba0c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:12.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.764+0000 7fba11000640 1 --2- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c0cf9a0 0x7fba0c070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.765+0000 7fba11000640 1 -- 172.21.15.53:0/3194097306 >> v1:172.21.15.146:6789/0 conn(0x7fba0c14b2c0 legacy=0x7fba0c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:12.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.765+0000 7fba11000640 1 -- 172.21.15.53:0/3194097306 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0c071c50 con 0x7fba0c070ea0 2024-01-16T15:10:12.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.767+0000 7fba11000640 1 --2- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c070ea0 0x7fba0c0cc160 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fba04009d90 tx=0x7fba0402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30a8d12c73d839e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:12.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.768+0000 7fba0b400640 1 -- 172.21.15.53:0/3194097306 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0402cc60 con 0x7fba0c070ea0 2024-01-16T15:10:12.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.768+0000 7fba0b400640 1 -- 172.21.15.53:0/3194097306 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0402cdc0 con 0x7fba0c070ea0 2024-01-16T15:10:12.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.768+0000 7fba0b400640 1 -- 172.21.15.53:0/3194097306 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04034660 con 0x7fba0c070ea0 2024-01-16T15:10:12.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.769+0000 7fba13000640 1 -- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c070ea0 msgr2=0x7fba0c0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:12.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.769+0000 7fba13000640 1 --2- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c070ea0 0x7fba0c0cc160 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fba04009d90 tx=0x7fba0402beb0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.769+0000 7fba13000640 1 -- 172.21.15.53:0/3194097306 shutdown_connections 2024-01-16T15:10:12.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.769+0000 7fba13000640 1 --2- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c070ea0 0x7fba0c0cc160 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.769+0000 7fba13000640 1 --2- 172.21.15.53:0/3194097306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c0cf9a0 0x7fba0c070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.769+0000 7fba13000640 1 -- 172.21.15.53:0/3194097306 >> 172.21.15.53:0/3194097306 conn(0x7fba0c06a800 msgr2=0x7fba0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:12.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.770+0000 7fba13000640 1 -- 172.21.15.53:0/3194097306 shutdown_connections 2024-01-16T15:10:12.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.770+0000 7fba13000640 1 -- 172.21.15.53:0/3194097306 wait complete. 2024-01-16T15:10:12.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.773+0000 7fba13000640 1 Processor -- start 2024-01-16T15:10:12.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.774+0000 7fba13000640 1 -- start start 2024-01-16T15:10:12.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.777+0000 7fba13000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 0x7fba0c161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:12.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.777+0000 7fba13000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c0cf9a0 0x7fba0c147390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:12.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.777+0000 7fba13000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 0x7fba0c1478d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:12.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.777+0000 7fba13000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c145200 con 0x7fba0c14b2c0 2024-01-16T15:10:12.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.777+0000 7fba13000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c143e80 con 0x7fba0c0cf9a0 2024-01-16T15:10:12.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.777+0000 7fba13000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c144160 con 0x7fba0c070ea0 2024-01-16T15:10:12.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.778+0000 7fba11000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 0x7fba0c1478d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.779+0000 7fba0be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 0x7fba0c161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.780+0000 7fba0be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 0x7fba0c161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:47234/0 (socket says 172.21.15.53:47234) 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.780+0000 7fba0be00640 1 -- 172.21.15.53:0/2099326468 learned_addr learned my addr 172.21.15.53:0/2099326468 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.782+0000 7fba0be00640 1 -- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c0cf9a0 msgr2=0x7fba0c147390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.779+0000 7fba03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c0cf9a0 0x7fba0c147390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.779+0000 7fba11000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 0x7fba0c1478d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33630/0 (socket says 172.21.15.53:33630) 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.784+0000 7fba0be00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c0cf9a0 0x7fba0c147390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.784+0000 7fba0be00640 1 -- 172.21.15.53:0/2099326468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 msgr2=0x7fba0c1478d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:12.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.784+0000 7fba0be00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 0x7fba0c1478d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.784+0000 7fba0be00640 1 -- 172.21.15.53:0/2099326468 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba04009a70 con 0x7fba0c070ea0 2024-01-16T15:10:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.784+0000 7fba11000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 0x7fba0c1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.784+0000 7fba03e00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c0cf9a0 0x7fba0c147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:12.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.785+0000 7fba0be00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 0x7fba0c161e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00c390 tx=0x7fb9fc00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:12.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.786+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc005ce0 con 0x7fba0c070ea0 2024-01-16T15:10:12.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.786+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc005e40 con 0x7fba0c070ea0 2024-01-16T15:10:12.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.786+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc0072d0 con 0x7fba0c070ea0 2024-01-16T15:10:12.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.787+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c144440 con 0x7fba0c070ea0 2024-01-16T15:10:12.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.787+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c144a00 con 0x7fba0c070ea0 2024-01-16T15:10:12.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.789+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c0675d0 con 0x7fba0c070ea0 2024-01-16T15:10:12.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.798+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb9fc007430 con 0x7fba0c070ea0 2024-01-16T15:10:12.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.799+0000 7fba08c00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9e406bee0 0x7fb9e406e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:12.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.799+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb9fc088170 con 0x7fba0c070ea0 2024-01-16T15:10:12.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.804+0000 7fba03e00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9e406bee0 0x7fb9e406e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:12.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.811+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9fc054980 con 0x7fba0c070ea0 2024-01-16T15:10:12.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:12.812+0000 7fba03e00640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9e406bee0 0x7fb9e406e3a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0ae970 tx=0x7fb9f4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.193+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone541420", "target": ["mon-mgr", ""]}) v1 -- 0x7fba0c071330 con 0x7fb9e406bee0 2024-01-16T15:10:13.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.568+0000 7fba08c00640 1 -- 172.21.15.53:0/2099326468 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb9f4009730 con 0x7fb9e406bee0 2024-01-16T15:10:13.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.570+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9e406bee0 msgr2=0x7fb9e406e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.570+0000 7fba13000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9e406bee0 0x7fb9e406e3a0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fba0c0ae970 tx=0x7fb9f4008040 comp rx=0 tx=0).stop 2024-01-16T15:10:13.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.570+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 msgr2=0x7fba0c161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.570+0000 7fba13000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 0x7fba0c161e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc00c390 tx=0x7fb9fc00c850 comp rx=0 tx=0).stop 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.570+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 shutdown_connections 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9e406bee0 0x7fb9e406e3a0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0c14b2c0 0x7fba0c1478d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0c0cf9a0 0x7fba0c147390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 --2- 172.21.15.53:0/2099326468 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0c070ea0 0x7fba0c161e70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 >> 172.21.15.53:0/2099326468 conn(0x7fba0c06a800 msgr2=0x7fba0c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 shutdown_connections 2024-01-16T15:10:13.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.571+0000 7fba13000640 1 -- 172.21.15.53:0/2099326468 wait complete. 2024-01-16T15:10:13.566 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json-pretty 2024-01-16T15:10:13.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.777+0000 7f2014a00640 1 Processor -- start 2024-01-16T15:10:13.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.777+0000 7f2014a00640 1 -- start start 2024-01-16T15:10:13.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f2014a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 0x7f20100b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f2014a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20100d0a10 con 0x7f20100cf9f0 2024-01-16T15:10:13.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f2014a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2010161e30 con 0x7f20100cfe70 2024-01-16T15:10:13.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f2014a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100d1f50 con 0x7f2010160920 2024-01-16T15:10:13.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f2007e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f20100cf9f0 0x7f2010161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56000/0 (socket says 172.21.15.53:56000) 2024-01-16T15:10:13.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f2007e00640 1 -- 172.21.15.53:0/2758759256 learned_addr learned my addr 172.21.15.53:0/2758759256 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:13.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.779+0000 7f200f400640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 0x7f20100b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855411874 0 0) 0x7f20100d0a10 con 0x7f20100cf9f0 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ff0003520 con 0x7f20100cf9f0 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 653046984 0 0) 0x7f1ff0003520 con 0x7f20100cf9f0 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 msgr2=0x7f20100b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 0x7f20100b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 >> v1:172.21.15.146:6789/0 conn(0x7f20100cfe70 legacy=0x7f2010161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100cf080 con 0x7f20100cf9f0 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1ff8001670 con 0x7f20100cf9f0 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 >> v1:172.21.15.53:6789/0 conn(0x7f20100cf9f0 legacy=0x7f2010161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ff00039a0 0x7f1ff0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 0x7f1ff00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ff0004d00 0x7f1ff0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100d1f50 con 0x7f1ff00039a0 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000ba30 con 0x7f1ff0004d00 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.780+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff000bbf0 con 0x7f2010160920 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200f400640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ff00039a0 0x7f1ff0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f2007e00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 0x7f1ff00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200f400640 1 -- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 msgr2=0x7f1ff00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200f400640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2010160920 0x7f1ff00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200f400640 1 -- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ff0004d00 msgr2=0x7f1ff0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200fe00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ff0004d00 0x7f1ff0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200f400640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ff0004d00 0x7f1ff0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.781+0000 7f200f400640 1 -- 172.21.15.53:0/2758759256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff0003520 con 0x7f1ff00039a0 2024-01-16T15:10:13.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.782+0000 7f200f400640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ff00039a0 0x7f1ff0003dc0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2000009a90 tx=0x7f1ff00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c08b6f97826c1e49 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200002c680 con 0x7f1ff00039a0 2024-01-16T15:10:13.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200002c7e0 con 0x7f1ff00039a0 2024-01-16T15:10:13.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f200ea00640 1 -- 172.21.15.53:0/2758759256 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200002cb70 con 0x7f1ff00039a0 2024-01-16T15:10:13.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f2014a00640 1 -- 172.21.15.53:0/2758759256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ff00039a0 msgr2=0x7f1ff0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f2014a00640 1 --2- 172.21.15.53:0/2758759256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ff00039a0 0x7f1ff0003dc0 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f2000009a90 tx=0x7f1ff00057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f200fe00640 1 -- 172.21.15.53:0/2758759256 reap_dead start 2024-01-16T15:10:13.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f2014a00640 1 -- 172.21.15.53:0/2758759256 shutdown_connections 2024-01-16T15:10:13.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f2014a00640 1 -- 172.21.15.53:0/2758759256 >> 172.21.15.53:0/2758759256 conn(0x7f20100d3090 msgr2=0x7f20100d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:13.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f2014a00640 1 -- 172.21.15.53:0/2758759256 shutdown_connections 2024-01-16T15:10:13.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.783+0000 7f2014a00640 1 -- 172.21.15.53:0/2758759256 wait complete. 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.785+0000 7f2014a00640 1 Processor -- start 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.785+0000 7f2014a00640 1 -- start start 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.786+0000 7f2014a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff00039a0 0x7f20101514a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.786+0000 7f2014a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100c8a90 0x7f20100c8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.786+0000 7f2014a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 0x7f20101525c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.786+0000 7f2014a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20100c9450 con 0x7f2010152110 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.786+0000 7f2014a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010153f70 con 0x7f20100c8a90 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.786+0000 7f2014a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2010152ed0 con 0x7f1ff00039a0 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 0x7f20101525c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 0x7f20101525c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33646/0 (socket says 172.21.15.53:33646) 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 -- 172.21.15.53:0/3530316476 learned_addr learned my addr 172.21.15.53:0/3530316476 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:13.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200f400640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff00039a0 0x7f20101514a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f2007e00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100c8a90 0x7f20100c8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 -- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff00039a0 msgr2=0x7f20101514a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff00039a0 0x7f20101514a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 -- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100c8a90 msgr2=0x7f20100c8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100c8a90 0x7f20100c8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 -- 172.21.15.53:0/3530316476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20101531b0 con 0x7f2010152110 2024-01-16T15:10:13.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.787+0000 7f200fe00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 0x7f20101525c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f201008bfe0 tx=0x7f200800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.788+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008003b70 con 0x7f2010152110 2024-01-16T15:10:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.788+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2008003cd0 con 0x7f2010152110 2024-01-16T15:10:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.788+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080052b0 con 0x7f2010152110 2024-01-16T15:10:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.788+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2010153490 con 0x7f2010152110 2024-01-16T15:10:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.788+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2010153a50 con 0x7f2010152110 2024-01-16T15:10:13.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.789+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2010067450 con 0x7f2010152110 2024-01-16T15:10:13.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.790+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f2008005410 con 0x7f2010152110 2024-01-16T15:10:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.791+0000 7f2007400640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fe806bec0 0x7f1fe806e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.791+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2008088060 con 0x7f2010152110 2024-01-16T15:10:13.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.792+0000 7f200f400640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fe806bec0 0x7f1fe806e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:13.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.793+0000 7f200f400640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fe806bec0 0x7f1fe806e380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2000036000 tx=0x7f2000030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:13.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:13.796+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2008054920 con 0x7f2010152110 2024-01-16T15:10:14.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.068+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f201008c7e0 con 0x7f2010152110 2024-01-16T15:10:14.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.070+0000 7f2007400640 1 -- 172.21.15.53:0/3530316476 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2427 (secure 0 0 0) 0x7f2008018030 con 0x7f2010152110 2024-01-16T15:10:14.047 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:10:14.047 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:10:14.048 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 641, 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 74, 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:10:14.049 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:10:14.050 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 3417, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 14136891263, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 28707553280, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 744386560000, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 270726291, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 267609678, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 65, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 72 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 46, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 5, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:10:14.051 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 5, 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "d", 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 7032 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:10:14.052 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 15, 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:09:10.402536+0000", 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": { 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "15695185-627b-4b7f-b3e8-28484fd7a91b": { 2024-01-16T15:10:14.053 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "17 ongoing+pending cloning jobs; Avg progress = 19.608% (39s)\n [=====.......................] (remaining: 2m)", 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.19608235359191895, 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: "43ee1060-9288-471d-9abd-61ae6747addf": { 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: "message": "4 ongoing cloning jobs; Avg progress = 83.335% (39s)\n [=======================.....] (remaining: 7s)", 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: "progress": 0.83335000276565552, 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: "add_to_ceph_s": true 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:10:14.054 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:10:14.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.072+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fe806bec0 msgr2=0x7f1fe806e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.072+0000 7f2014a00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fe806bec0 0x7f1fe806e380 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2000036000 tx=0x7f2000030000 comp rx=0 tx=0).stop 2024-01-16T15:10:14.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.072+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 msgr2=0x7f20101525c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.072+0000 7f2014a00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 0x7f20101525c0 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f201008bfe0 tx=0x7f200800dc00 comp rx=0 tx=0).stop 2024-01-16T15:10:14.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 shutdown_connections 2024-01-16T15:10:14.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1fe806bec0 0x7f1fe806e380 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2010152110 0x7f20101525c0 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20100c8a90 0x7f20100c8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 --2- 172.21.15.53:0/3530316476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ff00039a0 0x7f20101514a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 >> 172.21.15.53:0/3530316476 conn(0x7f20100d3090 msgr2=0x7f201015afd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:14.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 shutdown_connections 2024-01-16T15:10:14.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.073+0000 7f2014a00640 1 -- 172.21.15.53:0/3530316476 wait complete. 2024-01-16T15:10:14.069 INFO:tasks.cephfs.test_volumes:new_onpen_p is less than onpen_p. 2024-01-16T15:10:14.069 INFO:tasks.cephfs.test_volumes:new_onpen_p = 0.19608235359191895; onpen_p = 0.2857142984867096 2024-01-16T15:10:14.069 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54141 2024-01-16T15:10:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.321+0000 7f6e03e00640 1 Processor -- start 2024-01-16T15:10:14.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.322+0000 7f6e03e00640 1 -- start start 2024-01-16T15:10:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.323+0000 7f6e03e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040adf20 0x7f6e04070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.323+0000 7f6e03e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6e0415bf10 0x7f6e040cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.323+0000 7f6e03e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6e040cc980 0x7f6e04143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.323+0000 7f6e03e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0415f970 con 0x7f6e0415bf10 2024-01-16T15:10:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.323+0000 7f6e03e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e0415f4e0 con 0x7f6e040cc980 2024-01-16T15:10:14.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.323+0000 7f6e03e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040cce10 con 0x7f6e040adf20 2024-01-16T15:10:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6e0415bf10 0x7f6e040cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6e0415bf10 0x7f6e040cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33660/0 (socket says 172.21.15.53:33660) 2024-01-16T15:10:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02000640 1 -- 172.21.15.53:0/844123165 learned_addr learned my addr 172.21.15.53:0/844123165 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02a00640 1 --2- 172.21.15.53:0/844123165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040adf20 0x7f6e04070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e03400640 1 --2- 172.21.15.53:0/844123165 >> v2:172.21.15.146:3300/0 conn(0x7f6e040cc980 0x7f6e04143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02000640 1 -- 172.21.15.53:0/844123165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040adf20 msgr2=0x7f6e04070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02000640 1 --2- 172.21.15.53:0/844123165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040adf20 0x7f6e04070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.324+0000 7f6e02000640 1 -- 172.21.15.53:0/844123165 >> v2:172.21.15.146:3300/0 conn(0x7f6e040cc980 msgr2=0x7f6e04143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.325+0000 7f6e02000640 1 --2- 172.21.15.53:0/844123165 >> v2:172.21.15.146:3300/0 conn(0x7f6e040cc980 0x7f6e04143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.325+0000 7f6e02000640 1 -- 172.21.15.53:0/844123165 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e04071130 con 0x7f6e0415bf10 2024-01-16T15:10:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.326+0000 7f6e02000640 1 --2- 172.21.15.53:0/844123165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040cc440 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6e0406e490 tx=0x7f6dec02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71804a6375554a1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.326+0000 7f6e01600640 1 -- 172.21.15.53:0/844123165 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec0317e0 con 0x7f6e0415bf10 2024-01-16T15:10:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.326+0000 7f6e01600640 1 -- 172.21.15.53:0/844123165 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec031940 con 0x7f6e0415bf10 2024-01-16T15:10:14.304 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.326+0000 7f6e01600640 1 -- 172.21.15.53:0/844123165 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dec031cd0 con 0x7f6e0415bf10 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 -- 172.21.15.53:0/844123165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 msgr2=0x7f6e040cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 --2- 172.21.15.53:0/844123165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040cc440 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f6e0406e490 tx=0x7f6dec02b880 comp rx=0 tx=0).stop 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 -- 172.21.15.53:0/844123165 shutdown_connections 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 --2- 172.21.15.53:0/844123165 >> v2:172.21.15.146:3300/0 conn(0x7f6e040cc980 0x7f6e04143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 --2- 172.21.15.53:0/844123165 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040cc440 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 --2- 172.21.15.53:0/844123165 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040adf20 0x7f6e04070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 -- 172.21.15.53:0/844123165 >> 172.21.15.53:0/844123165 conn(0x7f6e0406a7d0 msgr2=0x7f6e0406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 -- 172.21.15.53:0/844123165 shutdown_connections 2024-01-16T15:10:14.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.327+0000 7f6e03e00640 1 -- 172.21.15.53:0/844123165 wait complete. 2024-01-16T15:10:14.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.329+0000 7f6e03e00640 1 Processor -- start 2024-01-16T15:10:14.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.329+0000 7f6e03e00640 1 -- start start 2024-01-16T15:10:14.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e040adf20 0x7f6e0415a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040cc980 0x7f6e040ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0415ae30 con 0x7f6e0415bf10 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e040b1e90 con 0x7f6e040adf20 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0413bba0 con 0x7f6e040cc980 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33668/0 (socket says 172.21.15.53:33668) 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.330+0000 7f6e03400640 1 -- 172.21.15.53:0/3282394965 learned_addr learned my addr 172.21.15.53:0/3282394965 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e02000640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040cc980 0x7f6e040ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e02a00640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e040adf20 0x7f6e0415a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e03400640 1 -- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040cc980 msgr2=0x7f6e040ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e03400640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e040cc980 0x7f6e040ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e03400640 1 -- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e040adf20 msgr2=0x7f6e0415a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:14.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e03400640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e040adf20 0x7f6e0415a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:14.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e03400640 1 -- 172.21.15.53:0/3282394965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec0094d0 con 0x7f6e0415bf10 2024-01-16T15:10:14.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e02a00640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e040adf20 0x7f6e0415a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:14.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6e03400640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e0415bf10 0x7f6e040ace10 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc007e70 tx=0x7f6dfc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc014620 con 0x7f6e0415bf10 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.331+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc0147a0 con 0x7f6e0415bf10 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.332+0000 7f6e03e00640 1 -- 172.21.15.53:0/3282394965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0413be80 con 0x7f6e0415bf10 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.332+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc014b50 con 0x7f6e0415bf10 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.332+0000 7f6e03e00640 1 -- 172.21.15.53:0/3282394965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e040b23c0 con 0x7f6e0415bf10 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.333+0000 7f6e03e00640 1 -- 172.21.15.53:0/3282394965 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e0413c0c0 con 0x7f6e0415bf10 2024-01-16T15:10:14.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.334+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f6dfc014cb0 con 0x7f6e0415bf10 2024-01-16T15:10:14.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.334+0000 7f6dfb400640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ddc06be90 0x7f6ddc06e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:14.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.335+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6dfc088fa0 con 0x7f6e0415bf10 2024-01-16T15:10:14.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.335+0000 7f6e02a00640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ddc06be90 0x7f6ddc06e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:14.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.336+0000 7f6e02a00640 1 --2- 172.21.15.53:0/3282394965 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ddc06be90 0x7f6ddc06e350 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f6df40099a0 tx=0x7f6df4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:14.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.339+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6dfc055860 con 0x7f6e0415bf10 2024-01-16T15:10:14.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.573+0000 7f6e03e00640 1 -- 172.21.15.53:0/3282394965 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54141", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e0415ee90 con 0x7f6ddc06be90 2024-01-16T15:10:14.571 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:14.594+0000 7f6135000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:14.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:14.594+0000 7f6dfb400640 1 -- 172.21.15.53:0/3282394965 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f6df4009610 con 0x7f6ddc06be90 2024-01-16T15:10:14.572 INFO:teuthology.orchestra.run.smithi053.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:14.586 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:10:16.587 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54142 2024-01-16T15:10:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.842+0000 7f956fe00640 1 Processor -- start 2024-01-16T15:10:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.842+0000 7f956fe00640 1 -- start start 2024-01-16T15:10:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570071e40 0x7f95700ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f957015d770 0x7f957015dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f957015e140 con 0x7f9570143810 2024-01-16T15:10:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9570143bf0 con 0x7f957015d770 2024-01-16T15:10:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95700722c0 con 0x7f9570071e40 2024-01-16T15:10:16.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9570143810 0x7f957015e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56006/0 (socket says 172.21.15.53:56006) 2024-01-16T15:10:16.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956ea00640 1 -- 172.21.15.53:0/2744201270 learned_addr learned my addr 172.21.15.53:0/2744201270 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f956f400640 1 --2- 172.21.15.53:0/2744201270 >> v2:172.21.15.53:3300/0 conn(0x7f957015d770 0x7f957015dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.844+0000 7f9567e00640 1 --2- 172.21.15.53:0/2744201270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570071e40 0x7f95700ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.845+0000 7f956e000640 1 -- 172.21.15.53:0/2744201270 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574210916 0 0) 0x7f957015e140 con 0x7f9570143810 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.845+0000 7f956e000640 1 -- 172.21.15.53:0/2744201270 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9550003520 con 0x7f9570143810 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.845+0000 7f956f400640 1 -- 172.21.15.53:0/2744201270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570071e40 msgr2=0x7f95700ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.845+0000 7f956f400640 1 --2- 172.21.15.53:0/2744201270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570071e40 0x7f95700ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.845+0000 7f956f400640 1 -- 172.21.15.53:0/2744201270 >> v1:172.21.15.53:6789/0 conn(0x7f9570143810 legacy=0x7f957015e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.845+0000 7f956f400640 1 -- 172.21.15.53:0/2744201270 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957006c730 con 0x7f957015d770 2024-01-16T15:10:16.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.846+0000 7f956f400640 1 --2- 172.21.15.53:0/2744201270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f957015d770 0x7f957015dc00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f9568009d90 tx=0x7f956802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb580f38176b1c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.846+0000 7f956e000640 1 -- 172.21.15.53:0/2744201270 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956802cb40 con 0x7f957015d770 2024-01-16T15:10:16.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.846+0000 7f956e000640 1 -- 172.21.15.53:0/2744201270 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956802cca0 con 0x7f957015d770 2024-01-16T15:10:16.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.846+0000 7f956e000640 1 -- 172.21.15.53:0/2744201270 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568034660 con 0x7f957015d770 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 -- 172.21.15.53:0/2744201270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f957015d770 msgr2=0x7f957015dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 --2- 172.21.15.53:0/2744201270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f957015d770 0x7f957015dc00 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f9568009d90 tx=0x7f956802bd90 comp rx=0 tx=0).stop 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 -- 172.21.15.53:0/2744201270 shutdown_connections 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 --2- 172.21.15.53:0/2744201270 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f957015d770 0x7f957015dc00 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 --2- 172.21.15.53:0/2744201270 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570071e40 0x7f95700ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 -- 172.21.15.53:0/2744201270 >> 172.21.15.53:0/2744201270 conn(0x7f957006a7d0 msgr2=0x7f957006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 -- 172.21.15.53:0/2744201270 shutdown_connections 2024-01-16T15:10:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.847+0000 7f956fe00640 1 -- 172.21.15.53:0/2744201270 wait complete. 2024-01-16T15:10:16.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.849+0000 7f956fe00640 1 Processor -- start 2024-01-16T15:10:16.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.849+0000 7f956fe00640 1 -- start start 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9570071e40 0x7f95701f22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570143810 0x7f95700b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f957015d770 0x7f9570143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95700b00d0 con 0x7f9570071e40 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95700b02e0 con 0x7f957015d770 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95700b05c0 con 0x7f9570143810 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9570071e40 0x7f95701f22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.850+0000 7f956ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9570071e40 0x7f95701f22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33682/0 (socket says 172.21.15.53:33682) 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 -- 172.21.15.53:0/1136787073 learned_addr learned my addr 172.21.15.53:0/1136787073 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f9567e00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570143810 0x7f95700b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956f400640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f957015d770 0x7f9570143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 -- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570143810 msgr2=0x7f95700b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9570143810 0x7f95700b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 -- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f957015d770 msgr2=0x7f9570143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f957015d770 0x7f9570143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 -- 172.21.15.53:0/1136787073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9568009a70 con 0x7f9570071e40 2024-01-16T15:10:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956f400640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f957015d770 0x7f9570143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.851+0000 7f956ea00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9570071e40 0x7f95701f22a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f956000c500 tx=0x7f956000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.852+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560005ce0 con 0x7f9570071e40 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.852+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9560005e40 con 0x7f9570071e40 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.852+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9560007330 con 0x7f9570071e40 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.852+0000 7f956fe00640 1 -- 172.21.15.53:0/1136787073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95700b08a0 con 0x7f9570071e40 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.852+0000 7f956fe00640 1 -- 172.21.15.53:0/1136787073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95701f27e0 con 0x7f9570071e40 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.853+0000 7f956fe00640 1 -- 172.21.15.53:0/1136787073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95700b0dc0 con 0x7f9570071e40 2024-01-16T15:10:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.854+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f9560007550 con 0x7f9570071e40 2024-01-16T15:10:16.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.855+0000 7f9566a00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f954806be90 0x7f954806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:16.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.855+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9560087eb0 con 0x7f9570071e40 2024-01-16T15:10:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.856+0000 7f9567e00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f954806be90 0x7f954806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.857+0000 7f9567e00640 1 --2- 172.21.15.53:0/1136787073 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f954806be90 0x7f954806e350 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f95701f5f90 tx=0x7f9558008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:16.860+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95700b0dc0 con 0x7f9570071e40 2024-01-16T15:10:17.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:17.087+0000 7f956fe00640 1 -- 172.21.15.53:0/1136787073 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54142", "target": ["mon-mgr", ""]}) v1 -- 0x7f95700b0450 con 0x7f954806be90 2024-01-16T15:10:17.091 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:17.114+0000 7f6135000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:17.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:17.114+0000 7f9566a00640 1 -- 172.21.15.53:0/1136787073 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7f95700b0450 con 0x7f954806be90 2024-01-16T15:10:17.091 INFO:teuthology.orchestra.run.smithi053.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:17.109 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:10:19.110 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54143 2024-01-16T15:10:19.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.371+0000 7fed86e00640 1 Processor -- start 2024-01-16T15:10:19.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.371+0000 7fed86e00640 1 -- start start 2024-01-16T15:10:19.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.373+0000 7fed86e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed8013cf10 0x7fed8013f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.373+0000 7fed86e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800ce440 con 0x7fed8013fae0 2024-01-16T15:10:19.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.373+0000 7fed86e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800ce720 con 0x7fed800cddf0 2024-01-16T15:10:19.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.373+0000 7fed86e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed8013ff90 con 0x7fed8013cf10 2024-01-16T15:10:19.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.373+0000 7fed85000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fed8013fae0 0x7fed800cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56018/0 (socket says 172.21.15.53:56018) 2024-01-16T15:10:19.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.373+0000 7fed85000640 1 -- 172.21.15.53:0/1348701188 learned_addr learned my addr 172.21.15.53:0/1348701188 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266917162 0 0) 0x7fed800ce440 con 0x7fed8013fae0 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed60003520 con 0x7fed8013fae0 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1590274111 0 0) 0x7fed60003520 con 0x7fed8013fae0 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 >> v2:172.21.15.146:3300/0 conn(0x7fed8013cf10 msgr2=0x7fed8013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 --2- 172.21.15.53:0/1348701188 >> v2:172.21.15.146:3300/0 conn(0x7fed8013cf10 0x7fed8013f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 >> v1:172.21.15.146:6789/0 conn(0x7fed800cddf0 legacy=0x7fed800b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed800cea00 con 0x7fed8013fae0 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fed68001670 con 0x7fed8013fae0 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 >> v1:172.21.15.53:6789/0 conn(0x7fed8013fae0 legacy=0x7fed800cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:19.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.374+0000 7fed7fe00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed60003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed7fe00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600047c0 0x7fed60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed7fe00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed60005380 0x7fed6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8013ff90 con 0x7fed600039c0 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed6000c1f0 con 0x7fed600047c0 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6000c3b0 con 0x7fed60005380 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed85a00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed60003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed85a00640 1 -- 172.21.15.53:0/1348701188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed60005380 msgr2=0x7fed6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed85a00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed60005380 0x7fed6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed85a00640 1 -- 172.21.15.53:0/1348701188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600047c0 msgr2=0x7fed60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed85a00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed600047c0 0x7fed60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed85a00640 1 -- 172.21.15.53:0/1348701188 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60003520 con 0x7fed600039c0 2024-01-16T15:10:19.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.375+0000 7fed86400640 1 -- 172.21.15.53:0/1348701188 reap_dead start 2024-01-16T15:10:19.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.376+0000 7fed85a00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed60003de0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fed70009e40 tx=0x7fed600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56271bc4eab1d3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:19.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7002cd70 con 0x7fed600039c0 2024-01-16T15:10:19.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed70034440 con 0x7fed600039c0 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed7fe00640 1 -- 172.21.15.53:0/1348701188 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed700347f0 con 0x7fed600039c0 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed86e00640 1 -- 172.21.15.53:0/1348701188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 msgr2=0x7fed60003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed86e00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed60003de0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7fed70009e40 tx=0x7fed600057f0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed86e00640 1 -- 172.21.15.53:0/1348701188 shutdown_connections 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed86e00640 1 --2- 172.21.15.53:0/1348701188 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed60003de0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed86e00640 1 -- 172.21.15.53:0/1348701188 >> 172.21.15.53:0/1348701188 conn(0x7fed8006a7d0 msgr2=0x7fed8006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.377+0000 7fed86e00640 1 -- 172.21.15.53:0/1348701188 shutdown_connections 2024-01-16T15:10:19.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.378+0000 7fed86e00640 1 -- 172.21.15.53:0/1348701188 wait complete. 2024-01-16T15:10:19.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.379+0000 7fed86e00640 1 Processor -- start 2024-01-16T15:10:19.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.380+0000 7fed86e00640 1 -- start start 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.380+0000 7fed86e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed80070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed86e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed80150ec0 0x7fed8015da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed86e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8015df60 0x7fed80152a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed86e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8015e3e0 con 0x7fed600039c0 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed86e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80151d60 con 0x7fed8015df60 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed86e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed80152600 con 0x7fed80150ec0 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed80070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed80070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33702/0 (socket says 172.21.15.53:33702) 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 -- 172.21.15.53:0/2512410171 learned_addr learned my addr 172.21.15.53:0/2512410171 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 -- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed80150ec0 msgr2=0x7fed8015da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:19.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 --2- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed80150ec0 0x7fed8015da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 -- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8015df60 msgr2=0x7fed80152a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:19.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 --2- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed8015df60 0x7fed80152a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:19.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.381+0000 7fed85a00640 1 -- 172.21.15.53:0/2512410171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8013eae0 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.382+0000 7fed85a00640 1 --2- 172.21.15.53:0/2512410171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed600039c0 0x7fed80070b70 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7fed70000f80 tx=0x7fed70004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.382+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7003e610 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.382+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7003e770 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.382+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7003eb20 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.382+0000 7fed86e00640 1 -- 172.21.15.53:0/2512410171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80151fb0 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.382+0000 7fed86e00640 1 -- 172.21.15.53:0/2512410171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8015b560 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.383+0000 7fed86e00640 1 -- 172.21.15.53:0/2512410171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8015e380 con 0x7fed600039c0 2024-01-16T15:10:19.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.384+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fed7003b050 con 0x7fed600039c0 2024-01-16T15:10:19.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.385+0000 7fed7d600640 1 --2- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed5806be90 0x7fed5806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:19.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.385+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fed700aad70 con 0x7fed600039c0 2024-01-16T15:10:19.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.386+0000 7fed85000640 1 --2- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed5806be90 0x7fed5806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:19.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.387+0000 7fed85000640 1 --2- 172.21.15.53:0/2512410171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fed5806be90 0x7fed5806e350 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fed8013a2d0 tx=0x7fed68001540 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:19.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.390+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed8015e380 con 0x7fed600039c0 2024-01-16T15:10:19.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.621+0000 7fed86e00640 1 -- 172.21.15.53:0/2512410171 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54143", "target": ["mon-mgr", ""]}) v1 -- 0x7fed800b2070 con 0x7fed5806be90 2024-01-16T15:10:19.623 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:19.646+0000 7f6135000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:19.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:19.647+0000 7fed7d600640 1 -- 172.21.15.53:0/2512410171 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fed68001bf0 con 0x7fed5806be90 2024-01-16T15:10:19.624 INFO:teuthology.orchestra.run.smithi053.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:10:21.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54144 2024-01-16T15:10:21.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.900+0000 7fbffe400640 1 Processor -- start 2024-01-16T15:10:21.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.900+0000 7fbffe400640 1 -- start start 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffe400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffe400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbff80cddf0 0x7fbff80b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffe400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbff80ce440 con 0x7fbff813fae0 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffe400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbff813ff60 con 0x7fbff80cddf0 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffe400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff80ce720 con 0x7fbff813cf10 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbff80cddf0 0x7fbff80b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbff80cddf0 0x7fbff80b2590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42824/0 (socket says 172.21.15.53:42824) 2024-01-16T15:10:21.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.902+0000 7fbffda00640 1 -- 172.21.15.53:0/1665837861 learned_addr learned my addr 172.21.15.53:0/1665837861 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.903+0000 7fbffd000640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.903+0000 7fbffda00640 1 -- 172.21.15.53:0/1665837861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 msgr2=0x7fbff813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.903+0000 7fbffda00640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.903+0000 7fbffda00640 1 -- 172.21.15.53:0/1665837861 >> v1:172.21.15.146:6789/0 conn(0x7fbff813fae0 legacy=0x7fbff80cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.903+0000 7fbffda00640 1 -- 172.21.15.53:0/1665837861 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff80cea00 con 0x7fbff80cddf0 2024-01-16T15:10:21.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.903+0000 7fbffd000640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:21.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.904+0000 7fbffda00640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff80cddf0 0x7fbff80b2590 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009e50 tx=0x7fbff002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55aa75437d48ce7b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:21.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.904+0000 7fbff7e00640 1 -- 172.21.15.53:0/1665837861 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff002ca70 con 0x7fbff80cddf0 2024-01-16T15:10:21.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.904+0000 7fbff7e00640 1 -- 172.21.15.53:0/1665837861 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff002cbd0 con 0x7fbff80cddf0 2024-01-16T15:10:21.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.904+0000 7fbff7e00640 1 -- 172.21.15.53:0/1665837861 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff00343f0 con 0x7fbff80cddf0 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 -- 172.21.15.53:0/1665837861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff80cddf0 msgr2=0x7fbff80b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff80cddf0 0x7fbff80b2590 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009e50 tx=0x7fbff002bed0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 -- 172.21.15.53:0/1665837861 shutdown_connections 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff80cddf0 0x7fbff80b2590 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 --2- 172.21.15.53:0/1665837861 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 -- 172.21.15.53:0/1665837861 >> 172.21.15.53:0/1665837861 conn(0x7fbff806a7d0 msgr2=0x7fbff806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 -- 172.21.15.53:0/1665837861 shutdown_connections 2024-01-16T15:10:21.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.905+0000 7fbffe400640 1 -- 172.21.15.53:0/1665837861 wait complete. 2024-01-16T15:10:21.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.907+0000 7fbffe400640 1 Processor -- start 2024-01-16T15:10:21.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.907+0000 7fbffe400640 1 -- start start 2024-01-16T15:10:21.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffe400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbff80cddf0 0x7fbff815b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:21.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffe400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff81538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:21.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffe400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff813fae0 0x7fbff806cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffe400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff8154360 con 0x7fbff813fae0 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffe400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbff815baa0 con 0x7fbff80cddf0 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffe400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbff813df70 con 0x7fbff813cf10 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff813fae0 0x7fbff806cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff813fae0 0x7fbff806cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42834/0 (socket says 172.21.15.53:42834) 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.908+0000 7fbffda00640 1 -- 172.21.15.53:0/1948977492 learned_addr learned my addr 172.21.15.53:0/1948977492 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffd000640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbff80cddf0 0x7fbff815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbfefe00640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff81538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffda00640 1 -- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 msgr2=0x7fbff81538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffda00640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff81538d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffda00640 1 -- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbff80cddf0 msgr2=0x7fbff815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffda00640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbff80cddf0 0x7fbff815b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffda00640 1 -- 172.21.15.53:0/1948977492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbff0009b30 con 0x7fbff813fae0 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbfefe00640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbff813cf10 0x7fbff81538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:21.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbffda00640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbff813fae0 0x7fbff806cfc0 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fbff0009e50 tx=0x7fbff0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:21.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.909+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff0009510 con 0x7fbff813fae0 2024-01-16T15:10:21.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.910+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff003e450 con 0x7fbff813fae0 2024-01-16T15:10:21.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.910+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff003e800 con 0x7fbff813fae0 2024-01-16T15:10:21.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.910+0000 7fbffe400640 1 -- 172.21.15.53:0/1948977492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff813e250 con 0x7fbff813fae0 2024-01-16T15:10:21.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.910+0000 7fbffe400640 1 -- 172.21.15.53:0/1948977492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbff813e810 con 0x7fbff813fae0 2024-01-16T15:10:21.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.911+0000 7fbffe400640 1 -- 172.21.15.53:0/1948977492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbff8067450 con 0x7fbff813fae0 2024-01-16T15:10:21.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.912+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fbff003e960 con 0x7fbff813fae0 2024-01-16T15:10:21.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.913+0000 7fbff5600640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbfd006be90 0x7fbfd006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:21.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.913+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbff00ab470 con 0x7fbff813fae0 2024-01-16T15:10:21.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.913+0000 7fbffd000640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbfd006be90 0x7fbfd006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:21.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.915+0000 7fbffd000640 1 --2- 172.21.15.53:0/1948977492 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbfd006be90 0x7fbfd006e350 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbfe800a990 tx=0x7fbfe8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:21.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:21.918+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbff0077de0 con 0x7fbff813fae0 2024-01-16T15:10:22.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:22.153+0000 7fbffe400640 1 -- 172.21.15.53:0/1948977492 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7fbff80b2070 con 0x7fbfd006be90 2024-01-16T15:10:23.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:23.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:28.922 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:28.945+0000 7f6135000640 -1 mgr.server reply reply (22) Invalid argument cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:28.946+0000 7fbff5600640 1 -- 172.21.15.53:0/1948977492 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -22 cannot cancel -- clone finished (check clone status)) v1 ==== 60+0+0 (secure 0 0 0) 0x7fbfe800a660 con 0x7fbfd006be90 2024-01-16T15:10:28.923 INFO:teuthology.orchestra.run.smithi053.stderr:Error EINVAL: cannot cancel -- clone finished (check clone status) 2024-01-16T15:10:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-01-16T15:10:30.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54145 2024-01-16T15:10:31.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.205+0000 7fb873200640 1 Processor -- start 2024-01-16T15:10:31.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.205+0000 7fb873200640 1 -- start start 2024-01-16T15:10:31.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.207+0000 7fb873200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0ae5a0 0x7fb86c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:31.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.207+0000 7fb873200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb86c0cf290 0x7fb86c0cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:31.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.207+0000 7fb873200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c070c40 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.207+0000 7fb873200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c071450 con 0x7fb86c0cf290 2024-01-16T15:10:31.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.207+0000 7fb873200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0cfc60 con 0x7fb86c0ae5a0 2024-01-16T15:10:31.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb871200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb86c0cf290 0x7fb86c0cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:31.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb871200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb86c0cf290 0x7fb86c0cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42838/0 (socket says 172.21.15.53:42838) 2024-01-16T15:10:31.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb871200640 1 -- 172.21.15.53:0/789941466 learned_addr learned my addr 172.21.15.53:0/789941466 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:31.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb871200640 1 -- 172.21.15.53:0/789941466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0ae5a0 msgr2=0x7fb86c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:31.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb86be00640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0ae5a0 0x7fb86c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:31.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb871200640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0ae5a0 0x7fb86c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:31.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.208+0000 7fb871200640 1 -- 172.21.15.53:0/789941466 >> v1:172.21.15.146:6789/0 conn(0x7fb86c0b3ae0 legacy=0x7fb86c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:31.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.209+0000 7fb871200640 1 -- 172.21.15.53:0/789941466 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c06c470 con 0x7fb86c0cf290 2024-01-16T15:10:31.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.209+0000 7fb86be00640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0ae5a0 0x7fb86c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:31.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.210+0000 7fb871200640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0cf290 0x7fb86c0cf720 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fb864009870 tx=0x7fb86402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e275b205b02597a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:31.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.210+0000 7fb86aa00640 1 -- 172.21.15.53:0/789941466 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864030840 con 0x7fb86c0cf290 2024-01-16T15:10:31.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.210+0000 7fb86aa00640 1 -- 172.21.15.53:0/789941466 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8640309a0 con 0x7fb86c0cf290 2024-01-16T15:10:31.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.210+0000 7fb86aa00640 1 -- 172.21.15.53:0/789941466 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb864030d30 con 0x7fb86c0cf290 2024-01-16T15:10:31.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 -- 172.21.15.53:0/789941466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0cf290 msgr2=0x7fb86c0cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0cf290 0x7fb86c0cf720 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fb864009870 tx=0x7fb86402b980 comp rx=0 tx=0).stop 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 -- 172.21.15.53:0/789941466 shutdown_connections 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0cf290 0x7fb86c0cf720 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 --2- 172.21.15.53:0/789941466 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0ae5a0 0x7fb86c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 -- 172.21.15.53:0/789941466 >> 172.21.15.53:0/789941466 conn(0x7fb86c06a800 msgr2=0x7fb86c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 -- 172.21.15.53:0/789941466 shutdown_connections 2024-01-16T15:10:31.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.211+0000 7fb873200640 1 -- 172.21.15.53:0/789941466 wait complete. 2024-01-16T15:10:31.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.213+0000 7fb873200640 1 Processor -- start 2024-01-16T15:10:31.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.213+0000 7fb873200640 1 -- start start 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb873200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb86c0ae5a0 0x7fb86c1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb873200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 0x7fb86c1f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb873200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0cf290 0x7fb86c15fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb873200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c14e200 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb873200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c14e4e0 con 0x7fb86c0ae5a0 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb873200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c14e7c0 con 0x7fb86c0cf290 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb86b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 0x7fb86c1f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb86b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 0x7fb86c1f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42844/0 (socket says 172.21.15.53:42844) 2024-01-16T15:10:31.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.214+0000 7fb86b400640 1 -- 172.21.15.53:0/379120126 learned_addr learned my addr 172.21.15.53:0/379120126 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb871200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0cf290 0x7fb86c15fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86be00640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb86c0ae5a0 0x7fb86c1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86b400640 1 -- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0cf290 msgr2=0x7fb86c15fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86b400640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0cf290 0x7fb86c15fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86b400640 1 -- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb86c0ae5a0 msgr2=0x7fb86c1565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86b400640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb86c0ae5a0 0x7fb86c1565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86b400640 1 -- 172.21.15.53:0/379120126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8640094d0 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86be00640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb86c0ae5a0 0x7fb86c1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb86b400640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 0x7fb86c1f2530 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c7c0 tx=0x7fb85400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:31.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85400aa50 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85400abb0 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.215+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb854012420 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.216+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c156dc0 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.216+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c156b20 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.217+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c1572e0 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.218+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 25) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fb854012580 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.219+0000 7fb863e00640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb84406bec0 0x7fb84406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:31.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.219+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb854088a90 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.220+0000 7fb86be00640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb84406bec0 0x7fb84406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:31.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.221+0000 7fb86be00640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb84406bec0 0x7fb84406e380 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb85c006fd0 tx=0x7fb85c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:31.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.224+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb86c1572e0 con 0x7fb86c0b3ae0 2024-01-16T15:10:31.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:31.456+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54145", "target": ["mon-mgr", ""]}) v1 -- 0x7fb86c14e650 con 0x7fb84406bec0 2024-01-16T15:10:31.447 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:31.470+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:31.447 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:31.470+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:31.447 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:31.470+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:31.447 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:31.470+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:31.447 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:10:31.470+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:10:33.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:33.356+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb854055490 con 0x7fb86c0b3ae0 2024-01-16T15:10:39.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.226+0000 7fb863e00640 1 -- 172.21.15.53:0/379120126 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb86c14e650 con 0x7fb84406bec0 2024-01-16T15:10:39.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.230+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb84406bec0 msgr2=0x7fb84406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.230+0000 7fb873200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb84406bec0 0x7fb84406e380 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fb85c006fd0 tx=0x7fb85c008040 comp rx=0 tx=0).stop 2024-01-16T15:10:39.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.230+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 msgr2=0x7fb86c1f2530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:39.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.230+0000 7fb873200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 0x7fb86c1f2530 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fb85400c7c0 tx=0x7fb85400cc80 comp rx=0 tx=0).stop 2024-01-16T15:10:39.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.230+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 shutdown_connections 2024-01-16T15:10:39.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb84406bec0 0x7fb84406e380 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb86c0cf290 0x7fb86c15fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb86c0b3ae0 0x7fb86c1f2530 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 --2- 172.21.15.53:0/379120126 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb86c0ae5a0 0x7fb86c1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:39.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 >> 172.21.15.53:0/379120126 conn(0x7fb86c06a800 msgr2=0x7fb86c15d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:39.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 shutdown_connections 2024-01-16T15:10:39.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:39.231+0000 7fb873200640 1 -- 172.21.15.53:0/379120126 wait complete. 2024-01-16T15:10:41.233 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54146 2024-01-16T15:10:41.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.535+0000 7f8f91400640 1 Processor -- start 2024-01-16T15:10:41.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.535+0000 7f8f91400640 1 -- start start 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.536+0000 7f8f91400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8f8c0b7bf0 0x7f8f8c0ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.536+0000 7f8f91400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c0b6750 con 0x7f8f8c15d590 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.536+0000 7f8f91400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8c0cf1f0 con 0x7f8f8c0d0280 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.536+0000 7f8f91400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8c0cef10 con 0x7f8f8c0b7bf0 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.537+0000 7f8f8a800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8f8c15d590 0x7f8f8c0cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:36330/0 (socket says 172.21.15.53:36330) 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.537+0000 7f8f8a800640 1 -- 172.21.15.53:0/4152451487 learned_addr learned my addr 172.21.15.53:0/4152451487 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.537+0000 7f8f8b200640 1 --2- 172.21.15.53:0/4152451487 >> v2:172.21.15.146:3300/0 conn(0x7f8f8c0b7bf0 0x7f8f8c0ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.538+0000 7f8f89400640 1 -- 172.21.15.53:0/4152451487 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552538963 0 0) 0x7f8f8c0b6750 con 0x7f8f8c15d590 2024-01-16T15:10:41.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.538+0000 7f8f89400640 1 -- 172.21.15.53:0/4152451487 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f6c003520 con 0x7f8f8c15d590 2024-01-16T15:10:41.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.538+0000 7f8f8b200640 1 -- 172.21.15.53:0/4152451487 >> v1:172.21.15.146:6789/0 conn(0x7f8f8c0d0280 legacy=0x7f8f8c0ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:41.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.538+0000 7f8f8b200640 1 -- 172.21.15.53:0/4152451487 >> v1:172.21.15.53:6789/0 conn(0x7f8f8c15d590 legacy=0x7f8f8c0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:41.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.538+0000 7f8f8b200640 1 -- 172.21.15.53:0/4152451487 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8c0b6100 con 0x7f8f8c0b7bf0 2024-01-16T15:10:41.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.540+0000 7f8f8b200640 1 --2- 172.21.15.53:0/4152451487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0ce8c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8f740097a0 tx=0x7f8f7402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d497b4989547a823 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:41.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.542+0000 7f8f89400640 1 -- 172.21.15.53:0/4152451487 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74030610 con 0x7f8f8c0b7bf0 2024-01-16T15:10:41.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.542+0000 7f8f89400640 1 -- 172.21.15.53:0/4152451487 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f74030770 con 0x7f8f8c0b7bf0 2024-01-16T15:10:41.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.542+0000 7f8f89400640 1 -- 172.21.15.53:0/4152451487 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74030b00 con 0x7f8f8c0b7bf0 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.543+0000 7f8f91400640 1 -- 172.21.15.53:0/4152451487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c0b7bf0 msgr2=0x7f8f8c0ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.543+0000 7f8f91400640 1 --2- 172.21.15.53:0/4152451487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0ce8c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8f740097a0 tx=0x7f8f7402b960 comp rx=0 tx=0).stop 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.543+0000 7f8f91400640 1 -- 172.21.15.53:0/4152451487 shutdown_connections 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.543+0000 7f8f91400640 1 --2- 172.21.15.53:0/4152451487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0ce8c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.543+0000 7f8f91400640 1 -- 172.21.15.53:0/4152451487 >> 172.21.15.53:0/4152451487 conn(0x7f8f8c09c540 msgr2=0x7f8f8c0d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.543+0000 7f8f91400640 1 -- 172.21.15.53:0/4152451487 shutdown_connections 2024-01-16T15:10:41.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.544+0000 7f8f91400640 1 -- 172.21.15.53:0/4152451487 wait complete. 2024-01-16T15:10:41.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.546+0000 7f8f91400640 1 Processor -- start 2024-01-16T15:10:41.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.547+0000 7f8f91400640 1 -- start start 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.548+0000 7f8f91400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.548+0000 7f8f91400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 0x7f8f8c15eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.548+0000 7f8f91400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c15d590 0x7f8f8c15b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.548+0000 7f8f91400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c0c9be0 con 0x7f8f8c0d0280 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.548+0000 7f8f91400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c15f960 con 0x7f8f8c15d590 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.548+0000 7f8f91400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8c0c5fc0 con 0x7f8f8c0b7bf0 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.549+0000 7f8f89e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 0x7f8f8c15eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.549+0000 7f8f89e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 0x7f8f8c15eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:38182/0 (socket says 172.21.15.53:38182) 2024-01-16T15:10:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.549+0000 7f8f89e00640 1 -- 172.21.15.53:0/3675329237 learned_addr learned my addr 172.21.15.53:0/3675329237 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:41.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.549+0000 7f8f8a800640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:41.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.549+0000 7f8f8b200640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c15d590 0x7f8f8c15b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:41.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.550+0000 7f8f89e00640 1 -- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f8c0b7bf0 msgr2=0x7f8f8c0c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:41.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.550+0000 7f8f89e00640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0c96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:41.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.550+0000 7f8f89e00640 1 -- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c15d590 msgr2=0x7f8f8c15b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:41.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.550+0000 7f8f89e00640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c15d590 0x7f8f8c15b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:41.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.550+0000 7f8f89e00640 1 -- 172.21.15.53:0/3675329237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f74009400 con 0x7f8f8c0d0280 2024-01-16T15:10:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.551+0000 7f8f89e00640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 0x7f8f8c15eb00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00c480 tx=0x7f8f7c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.551+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c0052d0 con 0x7f8f8c0d0280 2024-01-16T15:10:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.551+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f7c005430 con 0x7f8f8c0d0280 2024-01-16T15:10:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.551+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c15f040 con 0x7f8f8c0d0280 2024-01-16T15:10:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.551+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c003830 con 0x7f8f8c0d0280 2024-01-16T15:10:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.551+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8c15f310 con 0x7f8f8c0d0280 2024-01-16T15:10:41.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.553+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f8c0675d0 con 0x7f8f8c0d0280 2024-01-16T15:10:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.555+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8f7c003990 con 0x7f8f8c0d0280 2024-01-16T15:10:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.556+0000 7f8f82a00640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6406bf50 0x7f8f6406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:41.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.556+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8f7c087cd0 con 0x7f8f8c0d0280 2024-01-16T15:10:41.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.566+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f7c054420 con 0x7f8f8c0d0280 2024-01-16T15:10:41.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.571+0000 7f8f8a800640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6406bf50 0x7f8f6406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:41.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.581+0000 7f8f8a800640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6406bf50 0x7f8f6406e410 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8f84001560 tx=0x7f8f84001a90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:41.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:41.961+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54146", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f8c15d970 con 0x7f8f6406bf50 2024-01-16T15:10:44.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.031+0000 7f8f82a00640 1 -- 172.21.15.53:0/3675329237 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8f84001f60 con 0x7f8f6406bf50 2024-01-16T15:10:44.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6406bf50 msgr2=0x7f8f6406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:44.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6406bf50 0x7f8f6406e410 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8f84001560 tx=0x7f8f84001a90 comp rx=0 tx=0).stop 2024-01-16T15:10:44.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 msgr2=0x7f8f8c15eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:44.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 0x7f8f8c15eb00 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00c480 tx=0x7f8f7c00c940 comp rx=0 tx=0).stop 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 shutdown_connections 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6406bf50 0x7f8f6406e410 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8c15d590 0x7f8f8c15b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8c0d0280 0x7f8f8c15eb00 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 --2- 172.21.15.53:0/3675329237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f8c0b7bf0 0x7f8f8c0c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 >> 172.21.15.53:0/3675329237 conn(0x7f8f8c09c540 msgr2=0x7f8f8c0ce040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 shutdown_connections 2024-01-16T15:10:44.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:44.034+0000 7f8f91400640 1 -- 172.21.15.53:0/3675329237 wait complete. 2024-01-16T15:10:46.036 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54147 2024-01-16T15:10:46.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.295+0000 7f6fbe000640 1 Processor -- start 2024-01-16T15:10:46.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.296+0000 7f6fbe000640 1 -- start start 2024-01-16T15:10:46.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.297+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb815a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.297+0000 7f6fbe000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6fb815d8a0 0x7f6fb8164520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.297+0000 7f6fbe000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb80b1e30 con 0x7f6fb81505b0 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.297+0000 7f6fbe000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb815abd0 con 0x7f6fb815d8a0 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.297+0000 7f6fbe000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb80b20a0 con 0x7f6fb80b3b20 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.298+0000 7f6fafe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6fb81505b0 0x7f6fb815d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55778/0 (socket says 172.21.15.53:55778) 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.298+0000 7f6fafe00640 1 -- 172.21.15.53:0/621744138 learned_addr learned my addr 172.21.15.53:0/621744138 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.298+0000 7f6fbcc00640 1 --2- 172.21.15.53:0/621744138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb815a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.298+0000 7f6fbd600640 1 --2- 172.21.15.53:0/621744138 >> v2:172.21.15.146:3300/0 conn(0x7f6fb815d8a0 0x7f6fb8164520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.299+0000 7f6fb7e00640 1 -- 172.21.15.53:0/621744138 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593462521 0 0) 0x7f6fb80b1e30 con 0x7f6fb81505b0 2024-01-16T15:10:46.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.299+0000 7f6fb7e00640 1 -- 172.21.15.53:0/621744138 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f98003520 con 0x7f6fb81505b0 2024-01-16T15:10:46.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.299+0000 7f6fbcc00640 1 -- 172.21.15.53:0/621744138 >> v2:172.21.15.146:3300/0 conn(0x7f6fb815d8a0 msgr2=0x7f6fb8164520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:46.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.299+0000 7f6fbcc00640 1 --2- 172.21.15.53:0/621744138 >> v2:172.21.15.146:3300/0 conn(0x7f6fb815d8a0 0x7f6fb8164520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.299+0000 7f6fbcc00640 1 -- 172.21.15.53:0/621744138 >> v1:172.21.15.53:6789/0 conn(0x7f6fb81505b0 legacy=0x7f6fb815d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.299+0000 7f6fbcc00640 1 -- 172.21.15.53:0/621744138 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb80b2380 con 0x7f6fb80b3b20 2024-01-16T15:10:46.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.300+0000 7f6fbcc00640 1 --2- 172.21.15.53:0/621744138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb815a690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80097a0 tx=0x7f6fa802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ede4fb10f0bf3a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:46.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.301+0000 7f6fb7e00640 1 -- 172.21.15.53:0/621744138 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802d040 con 0x7f6fb80b3b20 2024-01-16T15:10:46.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.301+0000 7f6fb7e00640 1 -- 172.21.15.53:0/621744138 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa8031810 con 0x7f6fb80b3b20 2024-01-16T15:10:46.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.301+0000 7f6fb7e00640 1 -- 172.21.15.53:0/621744138 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8031ba0 con 0x7f6fb80b3b20 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 -- 172.21.15.53:0/621744138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 msgr2=0x7f6fb815a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 --2- 172.21.15.53:0/621744138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb815a690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6fa80097a0 tx=0x7f6fa802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 -- 172.21.15.53:0/621744138 shutdown_connections 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 --2- 172.21.15.53:0/621744138 >> v2:172.21.15.146:3300/0 conn(0x7f6fb815d8a0 0x7f6fb8164520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 --2- 172.21.15.53:0/621744138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb815a690 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 -- 172.21.15.53:0/621744138 >> 172.21.15.53:0/621744138 conn(0x7f6fb806a7d0 msgr2=0x7f6fb806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 -- 172.21.15.53:0/621744138 shutdown_connections 2024-01-16T15:10:46.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.302+0000 7f6fbe000640 1 -- 172.21.15.53:0/621744138 wait complete. 2024-01-16T15:10:46.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.305+0000 7f6fbe000640 1 Processor -- start 2024-01-16T15:10:46.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.305+0000 7f6fbe000640 1 -- start start 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb814b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 0x7f6fb81527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fbe000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb815d8a0 0x7f6fb8152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fbe000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb814eb20 con 0x7f6fb81505b0 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fbe000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb814bae0 con 0x7f6fb815d8a0 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fbe000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb814bdc0 con 0x7f6fb80b3b20 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 0x7f6fb81527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.306+0000 7f6fafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 0x7f6fb81527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53582/0 (socket says 172.21.15.53:53582) 2024-01-16T15:10:46.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 -- 172.21.15.53:0/392412424 learned_addr learned my addr 172.21.15.53:0/392412424 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fbcc00640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb814b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fbd600640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb815d8a0 0x7f6fb8152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 -- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 msgr2=0x7f6fb814b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb814b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 -- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb815d8a0 msgr2=0x7f6fb8152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb815d8a0 0x7f6fb8152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 -- 172.21.15.53:0/392412424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa8009400 con 0x7f6fb81505b0 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.307+0000 7f6fafe00640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 0x7f6fb81527c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c430 tx=0x7f6fa000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.308+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0005ce0 con 0x7f6fb81505b0 2024-01-16T15:10:46.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.308+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa0005e40 con 0x7f6fb81505b0 2024-01-16T15:10:46.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.308+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa00072f0 con 0x7f6fb81505b0 2024-01-16T15:10:46.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.308+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb814e7b0 con 0x7f6fb81505b0 2024-01-16T15:10:46.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.308+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb8133130 con 0x7f6fb81505b0 2024-01-16T15:10:46.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.309+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb814ec90 con 0x7f6fb81505b0 2024-01-16T15:10:46.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.310+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6fa0007450 con 0x7f6fb81505b0 2024-01-16T15:10:46.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.311+0000 7f6fb5600640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f9006bf20 0x7f6f9006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:46.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.312+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6fa0087ef0 con 0x7f6fb81505b0 2024-01-16T15:10:46.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.312+0000 7f6fbcc00640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f9006bf20 0x7f6f9006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:46.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.313+0000 7f6fbcc00640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f9006bf20 0x7f6f9006e3e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8036000 tx=0x7f6fa803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:46.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.316+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6fa0057fc0 con 0x7f6fb81505b0 2024-01-16T15:10:46.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:46.549+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54147", "target": ["mon-mgr", ""]}) v1 -- 0x7f6fb80b41e0 con 0x7f6f9006bf20 2024-01-16T15:10:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.697+0000 7f6fb5600640 1 -- 172.21.15.53:0/392412424 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6fa8031ad0 con 0x7f6f9006bf20 2024-01-16T15:10:52.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f9006bf20 msgr2=0x7f6f9006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:52.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f9006bf20 0x7f6f9006e3e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8036000 tx=0x7f6fa803c040 comp rx=0 tx=0).stop 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 msgr2=0x7f6fb81527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 0x7f6fb81527c0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000c430 tx=0x7f6fa000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 shutdown_connections 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f9006bf20 0x7f6f9006e3e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6fb815d8a0 0x7f6fb8152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fb81505b0 0x7f6fb81527c0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 --2- 172.21.15.53:0/392412424 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fb80b3b20 0x7f6fb814b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 >> 172.21.15.53:0/392412424 conn(0x7f6fb806a7d0 msgr2=0x7f6fb80733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 shutdown_connections 2024-01-16T15:10:52.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:52.700+0000 7f6fbe000640 1 -- 172.21.15.53:0/392412424 wait complete. 2024-01-16T15:10:53.208 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:53.213 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:10:54.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54148 2024-01-16T15:10:54.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.955+0000 7f41f7c00640 1 Processor -- start 2024-01-16T15:10:54.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.955+0000 7f41f7c00640 1 -- start start 2024-01-16T15:10:54.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41f7c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f41f00b2620 0x7f41f00b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41f7c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f006bd90 con 0x7f41f00c4700 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41f7c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41f00adcd0 con 0x7f41f0161a80 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41f7c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41f0161e60 con 0x7f41f00b2620 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41f5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f41f00b2620 0x7f41f00b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41efe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f41f00c4700 0x7f41f00b0400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51916/0 (socket says 172.21.15.53:51916) 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.957+0000 7f41efe00640 1 -- 172.21.15.53:0/2912116630 learned_addr learned my addr 172.21.15.53:0/2912116630 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:10:54.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.958+0000 7f41f5c00640 1 -- 172.21.15.53:0/2912116630 >> v1:172.21.15.146:6789/0 conn(0x7f41f0161a80 legacy=0x7f41f00c3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:10:54.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.958+0000 7f41f5c00640 1 -- 172.21.15.53:0/2912116630 >> v1:172.21.15.53:6789/0 conn(0x7f41f00c4700 legacy=0x7f41f00b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.958+0000 7f41f5c00640 1 -- 172.21.15.53:0/2912116630 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f00d1e70 con 0x7f41f00b2620 2024-01-16T15:10:54.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.958+0000 7f41ef400640 1 -- 172.21.15.53:0/2912116630 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 570410849 0 0) 0x7f41f006bd90 con 0x7f41f00c4700 2024-01-16T15:10:54.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.959+0000 7f41f5c00640 1 --2- 172.21.15.53:0/2912116630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f00b0b10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009d90 tx=0x7f41e802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c183fd167abfbb6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:54.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.959+0000 7f41ef400640 1 -- 172.21.15.53:0/2912116630 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e802cbd0 con 0x7f41f00b2620 2024-01-16T15:10:54.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.959+0000 7f41ef400640 1 -- 172.21.15.53:0/2912116630 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e802cd30 con 0x7f41f00b2620 2024-01-16T15:10:54.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.959+0000 7f41ef400640 1 -- 172.21.15.53:0/2912116630 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e802cea0 con 0x7f41f00b2620 2024-01-16T15:10:54.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 -- 172.21.15.53:0/2912116630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 msgr2=0x7f41f00b0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 --2- 172.21.15.53:0/2912116630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f00b0b10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f41e8009d90 tx=0x7f41e802be20 comp rx=0 tx=0).stop 2024-01-16T15:10:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 -- 172.21.15.53:0/2912116630 shutdown_connections 2024-01-16T15:10:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 --2- 172.21.15.53:0/2912116630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f00b0b10 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 -- 172.21.15.53:0/2912116630 >> 172.21.15.53:0/2912116630 conn(0x7f41f00d1020 msgr2=0x7f41f00d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:10:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 -- 172.21.15.53:0/2912116630 shutdown_connections 2024-01-16T15:10:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.960+0000 7f41f7c00640 1 -- 172.21.15.53:0/2912116630 wait complete. 2024-01-16T15:10:54.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.962+0000 7f41f7c00640 1 Processor -- start 2024-01-16T15:10:54.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.962+0000 7f41f7c00640 1 -- start start 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f7c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f015dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f7c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41f00c4700 0x7f41f00cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f7c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41f0161a80 0x7f41f00ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f7c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00cc590 con 0x7f41f00b2620 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f7c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00cb910 con 0x7f41f00c4700 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f7c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41f00cbbf0 con 0x7f41f0161a80 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.963+0000 7f41f5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f015dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f015dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47834/0 (socket says 172.21.15.53:47834) 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 -- 172.21.15.53:0/1985587322 learned_addr learned my addr 172.21.15.53:0/1985587322 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:10:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41efe00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41f00c4700 0x7f41f00cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 -- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41f0161a80 msgr2=0x7f41f00ce190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41f0161a80 0x7f41f00ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 -- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41f00c4700 msgr2=0x7f41f00cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41f00c4700 0x7f41f00cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 -- 172.21.15.53:0/1985587322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e8009a70 con 0x7f41f00b2620 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.964+0000 7f41f5200640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f015dad0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c430 tx=0x7f41e000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.965+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0011790 con 0x7f41f00b2620 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.965+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e00118f0 con 0x7f41f00b2620 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.965+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e0011ce0 con 0x7f41f00b2620 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.965+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41f00cbed0 con 0x7f41f00b2620 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.965+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41f015eae0 con 0x7f41f00b2620 2024-01-16T15:10:54.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.966+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f00675d0 con 0x7f41f00b2620 2024-01-16T15:10:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.967+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f41e001c460 con 0x7f41f00b2620 2024-01-16T15:10:54.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.969+0000 7f41ecc00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41c806bf20 0x7f41c806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:10:54.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.970+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f41e0088550 con 0x7f41f00b2620 2024-01-16T15:10:54.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.970+0000 7f41efe00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41c806bf20 0x7f41c806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:10:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.971+0000 7f41efe00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41c806bf20 0x7f41c806e3e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f41f006eb00 tx=0x7f41d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:10:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:54.973+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f41e0054e10 con 0x7f41f00b2620 2024-01-16T15:10:55.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:10:55.209+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54148", "target": ["mon-mgr", ""]}) v1 -- 0x7f41f006bab0 con 0x7f41c806bf20 2024-01-16T15:11:04.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.730+0000 7f41ecc00640 1 -- 172.21.15.53:0/1985587322 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f41d8001300 con 0x7f41c806bf20 2024-01-16T15:11:04.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41c806bf20 msgr2=0x7f41c806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41c806bf20 0x7f41c806e3e0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f41f006eb00 tx=0x7f41d8009040 comp rx=0 tx=0).stop 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 msgr2=0x7f41f015dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f015dad0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f41e000c430 tx=0x7f41e000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 shutdown_connections 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f41c806bf20 0x7f41c806e3e0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f41f0161a80 0x7f41f00ce190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f41f00c4700 0x7f41f00cdc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 --2- 172.21.15.53:0/1985587322 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f41f00b2620 0x7f41f015dad0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 >> 172.21.15.53:0/1985587322 conn(0x7f41f00d1020 msgr2=0x7f41f00b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 shutdown_connections 2024-01-16T15:11:04.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:04.733+0000 7f41f7c00640 1 -- 172.21.15.53:0/1985587322 wait complete. 2024-01-16T15:11:06.733 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone54149 2024-01-16T15:11:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.054+0000 7feb62600640 1 Processor -- start 2024-01-16T15:11:07.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.055+0000 7feb62600640 1 -- start start 2024-01-16T15:11:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.057+0000 7feb62600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feb5c14b2c0 0x7feb5c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.057+0000 7feb62600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c0cfbf0 con 0x7feb5c071070 2024-01-16T15:11:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.057+0000 7feb62600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb5c071690 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.057+0000 7feb62600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb5c14b6a0 con 0x7feb5c14b2c0 2024-01-16T15:11:07.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.058+0000 7feb5aa00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7feb5c0cf7d0 0x7feb5c070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:39804/0 (socket says 172.21.15.53:39804) 2024-01-16T15:11:07.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.058+0000 7feb5aa00640 1 -- 172.21.15.53:0/815141397 learned_addr learned my addr 172.21.15.53:0/815141397 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:11:07.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.058+0000 7feb5b400640 1 --2- 172.21.15.53:0/815141397 >> v2:172.21.15.146:3300/0 conn(0x7feb5c14b2c0 0x7feb5c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.059+0000 7feb5b400640 1 -- 172.21.15.53:0/815141397 >> v1:172.21.15.146:6789/0 conn(0x7feb5c0cf7d0 legacy=0x7feb5c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.059+0000 7feb5b400640 1 -- 172.21.15.53:0/815141397 >> v1:172.21.15.53:6789/0 conn(0x7feb5c071070 legacy=0x7feb5c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.059+0000 7feb5b400640 1 -- 172.21.15.53:0/815141397 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c071970 con 0x7feb5c14b2c0 2024-01-16T15:11:07.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.063+0000 7feb5b400640 1 --2- 172.21.15.53:0/815141397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c0cf290 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0097a0 tx=0x7feb4c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=986273266d64647b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.066+0000 7feb5a000640 1 -- 172.21.15.53:0/815141397 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c030610 con 0x7feb5c14b2c0 2024-01-16T15:11:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.066+0000 7feb5a000640 1 -- 172.21.15.53:0/815141397 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb4c030770 con 0x7feb5c14b2c0 2024-01-16T15:11:07.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.066+0000 7feb5a000640 1 -- 172.21.15.53:0/815141397 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb4c030b00 con 0x7feb5c14b2c0 2024-01-16T15:11:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.067+0000 7feb62600640 1 -- 172.21.15.53:0/815141397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 msgr2=0x7feb5c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.067+0000 7feb62600640 1 --2- 172.21.15.53:0/815141397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c0cf290 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7feb4c0097a0 tx=0x7feb4c02b960 comp rx=0 tx=0).stop 2024-01-16T15:11:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.068+0000 7feb62600640 1 -- 172.21.15.53:0/815141397 shutdown_connections 2024-01-16T15:11:07.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.068+0000 7feb62600640 1 --2- 172.21.15.53:0/815141397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c0cf290 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.068+0000 7feb62600640 1 -- 172.21.15.53:0/815141397 >> 172.21.15.53:0/815141397 conn(0x7feb5c06a800 msgr2=0x7feb5c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.068+0000 7feb62600640 1 -- 172.21.15.53:0/815141397 shutdown_connections 2024-01-16T15:11:07.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.068+0000 7feb62600640 1 -- 172.21.15.53:0/815141397 wait complete. 2024-01-16T15:11:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.070+0000 7feb62600640 1 Processor -- start 2024-01-16T15:11:07.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.070+0000 7feb62600640 1 -- start start 2024-01-16T15:11:07.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.071+0000 7feb62600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb5c071070 0x7feb5c1ed970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.071+0000 7feb62600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 0x7feb5c1479a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.071+0000 7feb62600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c147ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.071+0000 7feb62600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c15caa0 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.071+0000 7feb62600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c15d900 con 0x7feb5c14b2c0 2024-01-16T15:11:07.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.071+0000 7feb62600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb5c1edeb0 con 0x7feb5c071070 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 0x7feb5c1479a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 0x7feb5c1479a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35352/0 (socket says 172.21.15.53:35352) 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5aa00640 1 -- 172.21.15.53:0/2489125051 learned_addr learned my addr 172.21.15.53:0/2489125051 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5b400640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb5c071070 0x7feb5c1ed970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5be00640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c147ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5aa00640 1 -- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb5c071070 msgr2=0x7feb5c1ed970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.072+0000 7feb5aa00640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb5c071070 0x7feb5c1ed970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb5aa00640 1 -- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 msgr2=0x7feb5c147ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb5aa00640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c147ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb5aa00640 1 -- 172.21.15.53:0/2489125051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb4c009400 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb5be00640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb5aa00640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 0x7feb5c1479a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c430 tx=0x7feb4400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb44005ce0 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb44005e40 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb440072f0 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.073+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb5c15ccf0 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.074+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb5c15d2b0 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.075+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb5c1ea090 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.077+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7feb44007510 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.078+0000 7feb53400640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feb3406bf70 0x7feb3406e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:07.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.078+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feb44087ea0 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.083+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feb5c1ea090 con 0x7feb5c0cf7d0 2024-01-16T15:11:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.084+0000 7feb5b400640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feb3406bf70 0x7feb3406e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.087+0000 7feb5b400640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feb3406bf70 0x7feb3406e430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7feb4c036000 tx=0x7feb4c03b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:07.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:07.375+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone54149", "target": ["mon-mgr", ""]}) v1 -- 0x7feb5c0cbc40 con 0x7feb3406bf70 2024-01-16T15:11:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.829+0000 7feb53400640 1 -- 172.21.15.53:0/2489125051 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7feb4c030960 con 0x7feb3406bf70 2024-01-16T15:11:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feb3406bf70 msgr2=0x7feb3406e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feb3406bf70 0x7feb3406e430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7feb4c036000 tx=0x7feb4c03b040 comp rx=0 tx=0).stop 2024-01-16T15:11:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 msgr2=0x7feb5c1479a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 0x7feb5c1479a0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7feb4400c430 tx=0x7feb4400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 shutdown_connections 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feb3406bf70 0x7feb3406e430 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feb5c14b2c0 0x7feb5c147ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feb5c0cf7d0 0x7feb5c1479a0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 --2- 172.21.15.53:0/2489125051 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feb5c071070 0x7feb5c1ed970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 >> 172.21.15.53:0/2489125051 conn(0x7feb5c06a800 msgr2=0x7feb5c149bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 shutdown_connections 2024-01-16T15:11:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:12.833+0000 7feb62600640 1 -- 172.21.15.53:0/2489125051 wait complete. 2024-01-16T15:11:14.832 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541410 2024-01-16T15:11:15.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.134+0000 7f3114e00640 1 Processor -- start 2024-01-16T15:11:15.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.134+0000 7f3114e00640 1 -- start start 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.135+0000 7f3114e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3110072230 0x7f31100b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f3114e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3110071190 con 0x7f3110150760 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f3114e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3110071470 con 0x7f311015da10 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f3114e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3110071750 con 0x7f3110072230 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f310de00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3110150760 0x7f311015d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59586/0 (socket says 172.21.15.53:59586) 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f310de00640 1 -- 172.21.15.53:0/579910071 learned_addr learned my addr 172.21.15.53:0/579910071 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:11:15.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f310e800640 1 --2- 172.21.15.53:0/579910071 >> v2:172.21.15.146:3300/0 conn(0x7f3110072230 0x7f31100b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.136+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923471111 0 0) 0x7f3110071190 con 0x7f3110150760 2024-01-16T15:11:15.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f0003520 con 0x7f3110150760 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1767958464 0 0) 0x7f30f0003520 con 0x7f3110150760 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 >> v2:172.21.15.146:3300/0 conn(0x7f3110072230 msgr2=0x7f31100b32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 --2- 172.21.15.53:0/579910071 >> v2:172.21.15.146:3300/0 conn(0x7f3110072230 0x7f31100b32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 >> v1:172.21.15.146:6789/0 conn(0x7f311015da10 legacy=0x7f31100b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110071a30 con 0x7f3110150760 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3100001670 con 0x7f3110150760 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310e800640 1 --2- 172.21.15.53:0/579910071 >> v2:172.21.15.146:3300/0 conn(0x7f3110072230 0x7f31100b32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:15.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 >> v1:172.21.15.53:6789/0 conn(0x7f3110150760 legacy=0x7f311015d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30f0003970 0x7f30f0003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f00047c0 0x7f30f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f0005180 0x7f30f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110071750 con 0x7f30f0005180 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f000c0b0 con 0x7f30f00047c0 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.137+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f000c270 con 0x7f30f0003970 2024-01-16T15:11:15.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.138+0000 7f310e800640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f0005180 0x7f30f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.138+0000 7f3107e00640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f00047c0 0x7f30f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.138+0000 7f310e800640 1 -- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30f0003970 msgr2=0x7f30f0003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.138+0000 7f310e800640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30f0003970 0x7f30f0003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.138+0000 7f310e800640 1 -- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f00047c0 msgr2=0x7f30f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.138+0000 7f310e800640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f00047c0 0x7f30f0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.139+0000 7f310e800640 1 -- 172.21.15.53:0/579910071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f0003520 con 0x7f30f0005180 2024-01-16T15:11:15.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.140+0000 7f310e800640 1 -- 172.21.15.53:0/579910071 reap_dead start 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.140+0000 7f310e800640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f0005180 0x7f30f000b980 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f31080073e0 tx=0x7f30f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=632ac8dc0f28518f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.140+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310802cdc0 con 0x7f30f0005180 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.140+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3108034440 con 0x7f30f0005180 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.140+0000 7f310d400640 1 -- 172.21.15.53:0/579910071 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31080347f0 con 0x7f30f0005180 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 -- 172.21.15.53:0/579910071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f0005180 msgr2=0x7f30f000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f0005180 0x7f30f000b980 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f31080073e0 tx=0x7f30f00057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 -- 172.21.15.53:0/579910071 shutdown_connections 2024-01-16T15:11:15.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 --2- 172.21.15.53:0/579910071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30f0005180 0x7f30f000b980 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 -- 172.21.15.53:0/579910071 >> 172.21.15.53:0/579910071 conn(0x7f311006a800 msgr2=0x7f311006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:15.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 -- 172.21.15.53:0/579910071 shutdown_connections 2024-01-16T15:11:15.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.141+0000 7f3114e00640 1 -- 172.21.15.53:0/579910071 wait complete. 2024-01-16T15:11:15.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.143+0000 7f3114e00640 1 Processor -- start 2024-01-16T15:11:15.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 -- start start 2024-01-16T15:11:15.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f0005180 0x7f31100cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3110137000 0x7f31101333d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 0x7f3110140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110159780 con 0x7f3110133910 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3110134910 con 0x7f30f0005180 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.144+0000 7f3114e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3110133ec0 con 0x7f3110137000 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 0x7f3110140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 0x7f3110140e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41538/0 (socket says 172.21.15.53:41538) 2024-01-16T15:11:15.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 -- 172.21.15.53:0/2197823151 learned_addr learned my addr 172.21.15.53:0/2197823151 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310de00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f0005180 0x7f31100cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f3107e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3110137000 0x7f31101333d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 -- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3110137000 msgr2=0x7f31101333d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3110137000 0x7f31101333d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 -- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f0005180 msgr2=0x7f31100cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f0005180 0x7f31100cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:15.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310e800640 1 -- 172.21.15.53:0/2197823151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31101594a0 con 0x7f3110133910 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f3107e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3110137000 0x7f31101333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.145+0000 7f310de00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f0005180 0x7f31100cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.146+0000 7f310e800640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 0x7f3110140e00 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f310802c830 tx=0x7f3108004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.146+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108041070 con 0x7f3110133910 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.146+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310802cdd0 con 0x7f3110133910 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.146+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3110134160 con 0x7f3110133910 2024-01-16T15:11:15.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.146+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108003bb0 con 0x7f3110133910 2024-01-16T15:11:15.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.146+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100ce340 con 0x7f3110133910 2024-01-16T15:11:15.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.147+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31100675d0 con 0x7f3110133910 2024-01-16T15:11:15.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.148+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f310803b070 con 0x7f3110133910 2024-01-16T15:11:15.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.149+0000 7f3106000640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30e806bf50 0x7f30e806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:15.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.149+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f31080a9ac0 con 0x7f3110133910 2024-01-16T15:11:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.154+0000 7f310de00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30e806bf50 0x7f30e806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:15.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.156+0000 7f310de00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30e806bf50 0x7f30e806e410 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3100001eb0 tx=0x7f31000017e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:15.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.156+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f31080763e0 con 0x7f3110133910 2024-01-16T15:11:15.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:15.547+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541410", "target": ["mon-mgr", ""]}) v1 -- 0x7f3110150b40 con 0x7f30e806bf50 2024-01-16T15:11:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.453+0000 7f3106000640 1 -- 172.21.15.53:0/2197823151 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3100001a70 con 0x7f30e806bf50 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30e806bf50 msgr2=0x7f30e806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30e806bf50 0x7f30e806e410 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3100001eb0 tx=0x7f31000017e0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 msgr2=0x7f3110140e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 0x7f3110140e00 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f310802c830 tx=0x7f3108004a40 comp rx=0 tx=0).stop 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 shutdown_connections 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30e806bf50 0x7f30e806e410 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3110133910 0x7f3110140e00 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3110137000 0x7f31101333d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 --2- 172.21.15.53:0/2197823151 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30f0005180 0x7f31100cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 >> 172.21.15.53:0/2197823151 conn(0x7f311006a800 msgr2=0x7f31100b2d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.457+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 shutdown_connections 2024-01-16T15:11:17.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:17.458+0000 7f3114e00640 1 -- 172.21.15.53:0/2197823151 wait complete. 2024-01-16T15:11:19.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541411 2024-01-16T15:11:19.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.712+0000 7fa31d000640 1 Processor -- start 2024-01-16T15:11:19.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.713+0000 7fa31d000640 1 -- start start 2024-01-16T15:11:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.714+0000 7fa31d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 0x7fa3180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.714+0000 7fa31d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa3180cf7d0 0x7fa31815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.714+0000 7fa31d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3180cfc30 con 0x7fa31815e1e0 2024-01-16T15:11:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.714+0000 7fa31d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3180b1ee0 con 0x7fa3180cf7d0 2024-01-16T15:11:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.714+0000 7fa31d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3180b2780 con 0x7fa318143ae0 2024-01-16T15:11:19.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.715+0000 7fa315e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 0x7fa3180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.715+0000 7fa315e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 0x7fa3180cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:49138/0 (socket says 172.21.15.53:49138) 2024-01-16T15:11:19.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.715+0000 7fa315e00640 1 -- 172.21.15.53:0/1349379635 learned_addr learned my addr 172.21.15.53:0/1349379635 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:19.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.715+0000 7fa30d200640 1 --2- 172.21.15.53:0/1349379635 >> v2:172.21.15.146:3300/0 conn(0x7fa3180cf7d0 0x7fa31815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.716+0000 7fa315400640 1 -- 172.21.15.53:0/1349379635 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959732760 0 0) 0x7fa3180cfc30 con 0x7fa31815e1e0 2024-01-16T15:11:19.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.716+0000 7fa315e00640 1 -- 172.21.15.53:0/1349379635 >> v2:172.21.15.146:3300/0 conn(0x7fa3180cf7d0 msgr2=0x7fa31815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:19.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.716+0000 7fa315e00640 1 --2- 172.21.15.53:0/1349379635 >> v2:172.21.15.146:3300/0 conn(0x7fa3180cf7d0 0x7fa31815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.716+0000 7fa315e00640 1 -- 172.21.15.53:0/1349379635 >> v1:172.21.15.146:6789/0 conn(0x7fa31815e1e0 legacy=0x7fa31815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.716+0000 7fa315e00640 1 -- 172.21.15.53:0/1349379635 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa31815e990 con 0x7fa318143ae0 2024-01-16T15:11:19.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.718+0000 7fa315e00640 1 --2- 172.21.15.53:0/1349379635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 0x7fa3180cf290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa308009e50 tx=0x7fa30802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=347d7bfe62b9cc35 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:19.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.718+0000 7fa315400640 1 -- 172.21.15.53:0/1349379635 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa30802cb00 con 0x7fa318143ae0 2024-01-16T15:11:19.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.718+0000 7fa315400640 1 -- 172.21.15.53:0/1349379635 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa30802cc60 con 0x7fa318143ae0 2024-01-16T15:11:19.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.718+0000 7fa315400640 1 -- 172.21.15.53:0/1349379635 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3080343f0 con 0x7fa318143ae0 2024-01-16T15:11:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 -- 172.21.15.53:0/1349379635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 msgr2=0x7fa3180cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 --2- 172.21.15.53:0/1349379635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 0x7fa3180cf290 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa308009e50 tx=0x7fa30802bf60 comp rx=0 tx=0).stop 2024-01-16T15:11:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 -- 172.21.15.53:0/1349379635 shutdown_connections 2024-01-16T15:11:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 --2- 172.21.15.53:0/1349379635 >> v2:172.21.15.146:3300/0 conn(0x7fa3180cf7d0 0x7fa31815dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 --2- 172.21.15.53:0/1349379635 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa318143ae0 0x7fa3180cf290 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 -- 172.21.15.53:0/1349379635 >> 172.21.15.53:0/1349379635 conn(0x7fa31806a800 msgr2=0x7fa31806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 -- 172.21.15.53:0/1349379635 shutdown_connections 2024-01-16T15:11:19.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.719+0000 7fa31d000640 1 -- 172.21.15.53:0/1349379635 wait complete. 2024-01-16T15:11:19.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.721+0000 7fa31d000640 1 Processor -- start 2024-01-16T15:11:19.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.722+0000 7fa31d000640 1 -- start start 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa31d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3180cf7d0 0x7fa31814d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa31d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa318143ae0 0x7fa318137cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa31d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 0x7fa31813d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa31d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa318138200 con 0x7fa31815e1e0 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa31d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa31813e040 con 0x7fa318143ae0 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa31d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa31813e320 con 0x7fa3180cf7d0 2024-01-16T15:11:19.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa316800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 0x7fa31813d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa316800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 0x7fa31813d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41540/0 (socket says 172.21.15.53:41540) 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.723+0000 7fa316800640 1 -- 172.21.15.53:0/2864656008 learned_addr learned my addr 172.21.15.53:0/2864656008 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa30d200640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa318143ae0 0x7fa318137cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa315e00640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3180cf7d0 0x7fa31814d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa316800640 1 -- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3180cf7d0 msgr2=0x7fa31814d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa316800640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3180cf7d0 0x7fa31814d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa316800640 1 -- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa318143ae0 msgr2=0x7fa318137cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa316800640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa318143ae0 0x7fa318137cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa316800640 1 -- 172.21.15.53:0/2864656008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa308009b30 con 0x7fa31815e1e0 2024-01-16T15:11:19.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.724+0000 7fa316800640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 0x7fa31813d7d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c3e0 tx=0x7fa31000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:19.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.725+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3100035f0 con 0x7fa31815e1e0 2024-01-16T15:11:19.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.725+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa310003750 con 0x7fa31815e1e0 2024-01-16T15:11:19.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.725+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310003b00 con 0x7fa31815e1e0 2024-01-16T15:11:19.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.725+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa31813e600 con 0x7fa31815e1e0 2024-01-16T15:11:19.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.725+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa31813ebc0 con 0x7fa31815e1e0 2024-01-16T15:11:19.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.726+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3180b2bf0 con 0x7fa31815e1e0 2024-01-16T15:11:19.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.727+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fa310003c60 con 0x7fa31815e1e0 2024-01-16T15:11:19.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.728+0000 7fa30ea00640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2f006bf20 0x7fa2f006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:19.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.728+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa310087e10 con 0x7fa31815e1e0 2024-01-16T15:11:19.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.729+0000 7fa315e00640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2f006bf20 0x7fa2f006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:19.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.730+0000 7fa315e00640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2f006bf20 0x7fa2f006e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa308035000 tx=0x7fa308030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:19.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.733+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa3180b2bf0 con 0x7fa31815e1e0 2024-01-16T15:11:19.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:19.964+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541411", "target": ["mon-mgr", ""]}) v1 -- 0x7fa31813e490 con 0x7fa2f006bf20 2024-01-16T15:11:23.368 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:23.377 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:26.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.339+0000 7fa30ea00640 1 -- 172.21.15.53:0/2864656008 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa31813e490 con 0x7fa2f006bf20 2024-01-16T15:11:26.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2f006bf20 msgr2=0x7fa2f006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2f006bf20 0x7fa2f006e3e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fa308035000 tx=0x7fa308030000 comp rx=0 tx=0).stop 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 msgr2=0x7fa31813d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 0x7fa31813d7d0 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fa31000c3e0 tx=0x7fa31000c8a0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 shutdown_connections 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa2f006bf20 0x7fa2f006e3e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa31815e1e0 0x7fa31813d7d0 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa318143ae0 0x7fa318137cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 --2- 172.21.15.53:0/2864656008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa3180cf7d0 0x7fa31814d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 >> 172.21.15.53:0/2864656008 conn(0x7fa31806a800 msgr2=0x7fa3180b0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 shutdown_connections 2024-01-16T15:11:26.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:26.342+0000 7fa31d000640 1 -- 172.21.15.53:0/2864656008 wait complete. 2024-01-16T15:11:28.343 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541412 2024-01-16T15:11:28.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.609+0000 7f5f58c00640 1 Processor -- start 2024-01-16T15:11:28.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.609+0000 7f5f58c00640 1 -- start start 2024-01-16T15:11:28.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.611+0000 7f5f58c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5f540ae5a0 0x7f5f5415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.611+0000 7f5f58c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f54070c40 con 0x7f5f540cf290 2024-01-16T15:11:28.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.611+0000 7f5f58c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f5406c470 con 0x7f5f540b3ae0 2024-01-16T15:11:28.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.611+0000 7f5f58c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f54071450 con 0x7f5f540ae5a0 2024-01-16T15:11:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.611+0000 7f5f52800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5f540cf290 0x7f5f540b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49738/0 (socket says 172.21.15.53:49738) 2024-01-16T15:11:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.611+0000 7f5f52800640 1 -- 172.21.15.53:0/870558001 learned_addr learned my addr 172.21.15.53:0/870558001 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:11:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44961243 0 0) 0x7f5f54070c40 con 0x7f5f540cf290 2024-01-16T15:11:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f34003520 con 0x7f5f540cf290 2024-01-16T15:11:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51e00640 1 --2- 172.21.15.53:0/870558001 >> v2:172.21.15.146:3300/0 conn(0x7f5f540ae5a0 0x7f5f5415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1412662559 0 0) 0x7f5f34003520 con 0x7f5f540cf290 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 >> v2:172.21.15.146:3300/0 conn(0x7f5f540ae5a0 msgr2=0x7f5f5415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 --2- 172.21.15.53:0/870558001 >> v2:172.21.15.146:3300/0 conn(0x7f5f540ae5a0 0x7f5f5415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 >> v1:172.21.15.146:6789/0 conn(0x7f5f540b3ae0 legacy=0x7f5f5406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5406c750 con 0x7f5f540cf290 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5f4c0019f0 con 0x7f5f540cf290 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 >> v1:172.21.15.53:6789/0 conn(0x7f5f540cf290 legacy=0x7f5f540b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f34003970 0x7f5f34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f340047c0 0x7f5f34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f3400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f54071450 con 0x7f5f34005180 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3400c0b0 con 0x7f5f340047c0 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.612+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3400c270 con 0x7f5f34003970 2024-01-16T15:11:28.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f52800640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f3400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f4be00640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f340047c0 0x7f5f34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f51e00640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f34003970 0x7f5f34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f52800640 1 -- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f34003970 msgr2=0x7f5f34003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f52800640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f34003970 0x7f5f34003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f52800640 1 -- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f340047c0 msgr2=0x7f5f34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f52800640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f340047c0 0x7f5f34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f52800640 1 -- 172.21.15.53:0/870558001 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f34003520 con 0x7f5f34005180 2024-01-16T15:11:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.613+0000 7f5f4be00640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f340047c0 0x7f5f34004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:28.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.614+0000 7f5f52800640 1 -- 172.21.15.53:0/870558001 reap_dead start 2024-01-16T15:11:28.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.614+0000 7f5f52800640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f3400b980 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00a800 tx=0x7f5f340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=197df74053efd5f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.614+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c034630 con 0x7f5f34005180 2024-01-16T15:11:28.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.614+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4c02f070 con 0x7f5f34005180 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.614+0000 7f5f51400640 1 -- 172.21.15.53:0/870558001 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4c034a10 con 0x7f5f34005180 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 -- 172.21.15.53:0/870558001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 msgr2=0x7f5f3400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f3400b980 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7f5f4c00a800 tx=0x7f5f340057f0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 -- 172.21.15.53:0/870558001 shutdown_connections 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 --2- 172.21.15.53:0/870558001 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f3400b980 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 -- 172.21.15.53:0/870558001 >> 172.21.15.53:0/870558001 conn(0x7f5f5406a800 msgr2=0x7f5f5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 -- 172.21.15.53:0/870558001 shutdown_connections 2024-01-16T15:11:28.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.615+0000 7f5f58c00640 1 -- 172.21.15.53:0/870558001 wait complete. 2024-01-16T15:11:28.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.617+0000 7f5f58c00640 1 Processor -- start 2024-01-16T15:11:28.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.617+0000 7f5f58c00640 1 -- start start 2024-01-16T15:11:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.618+0000 7f5f58c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f54151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.618+0000 7f5f58c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f5415b940 0x7f5f5415cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.618+0000 7f5f58c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f5414fa60 0x7f5f540b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.618+0000 7f5f58c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f540cec70 con 0x7f5f34005180 2024-01-16T15:11:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.618+0000 7f5f58c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5414fee0 con 0x7f5f5414fa60 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.618+0000 7f5f58c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f5415bf00 con 0x7f5f5415b940 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f54151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f54151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34596/0 (socket says 172.21.15.53:34596) 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 -- 172.21.15.53:0/2842964302 learned_addr learned my addr 172.21.15.53:0/2842964302 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f52800640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f5414fa60 0x7f5f540b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f4be00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f5415b940 0x7f5f5415cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 -- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f5415b940 msgr2=0x7f5f5415cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f5415b940 0x7f5f5415cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 -- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f5414fa60 msgr2=0x7f5f540b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f5414fa60 0x7f5f540b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 -- 172.21.15.53:0/2842964302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5415c1e0 con 0x7f5f34005180 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f52800640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f5414fa60 0x7f5f540b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.619+0000 7f5f51e00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f54151530 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e7e0 tx=0x7f5f4400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.620+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4400c6d0 con 0x7f5f34005180 2024-01-16T15:11:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.620+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f4400c830 con 0x7f5f34005180 2024-01-16T15:11:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.620+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f4400cba0 con 0x7f5f34005180 2024-01-16T15:11:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.620+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f5415c4c0 con 0x7f5f34005180 2024-01-16T15:11:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.620+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f5415ca80 con 0x7f5f34005180 2024-01-16T15:11:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.621+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f5415cc40 con 0x7f5f34005180 2024-01-16T15:11:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.622+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f5f4400cd00 con 0x7f5f34005180 2024-01-16T15:11:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.623+0000 7f5f4a000640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f2c06bf20 0x7f5f2c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.623+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5f44088430 con 0x7f5f34005180 2024-01-16T15:11:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.624+0000 7f5f4be00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f2c06bf20 0x7f5f2c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.625+0000 7f5f4be00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f2c06bf20 0x7f5f2c06e3e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c000f60 tx=0x7f5f3c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:28.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.627+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5f44054cf0 con 0x7f5f34005180 2024-01-16T15:11:28.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:28.866+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541412", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f54070550 con 0x7f5f2c06bf20 2024-01-16T15:11:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.129+0000 7f5f4a000640 1 -- 172.21.15.53:0/2842964302 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5f3c00d040 con 0x7f5f2c06bf20 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.132+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f2c06bf20 msgr2=0x7f5f2c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.132+0000 7f5f58c00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f2c06bf20 0x7f5f2c06e3e0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f5f3c000f60 tx=0x7f5f3c009040 comp rx=0 tx=0).stop 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.132+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 msgr2=0x7f5f54151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.132+0000 7f5f58c00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f54151530 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f5f4400e7e0 tx=0x7f5f4400eca0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 shutdown_connections 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5f2c06bf20 0x7f5f2c06e3e0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5f5414fa60 0x7f5f540b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5f5415b940 0x7f5f5415cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 --2- 172.21.15.53:0/2842964302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5f34005180 0x7f5f54151530 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 >> 172.21.15.53:0/2842964302 conn(0x7f5f5406a800 msgr2=0x7f5f541618b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:38.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 shutdown_connections 2024-01-16T15:11:38.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:38.133+0000 7f5f58c00640 1 -- 172.21.15.53:0/2842964302 wait complete. 2024-01-16T15:11:40.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541413 2024-01-16T15:11:40.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.432+0000 7ff0e3200640 1 Processor -- start 2024-01-16T15:11:40.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.432+0000 7ff0e3200640 1 -- start start 2024-01-16T15:11:40.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0e3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc15afc0 0x7ff0dc15d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0e3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff0dc06cc60 0x7ff0dc06d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0e3200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0dc15b3a0 con 0x7ff0dc0b2c00 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0e3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0dc0d11c0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0e3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0d14a0 con 0x7ff0dc15afc0 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0db400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff0dc06cc60 0x7ff0dc06d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0db400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff0dc06cc60 0x7ff0dc06d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36958/0 (socket says 172.21.15.53:36958) 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0db400640 1 -- 172.21.15.53:0/3949028048 learned_addr learned my addr 172.21.15.53:0/3949028048 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:40.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.434+0000 7ff0dbe00640 1 --2- 172.21.15.53:0/3949028048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc15afc0 0x7ff0dc15d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.435+0000 7ff0db400640 1 -- 172.21.15.53:0/3949028048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc15afc0 msgr2=0x7ff0dc15d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:40.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.435+0000 7ff0db400640 1 --2- 172.21.15.53:0/3949028048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc15afc0 0x7ff0dc15d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.435+0000 7ff0db400640 1 -- 172.21.15.53:0/3949028048 >> v1:172.21.15.146:6789/0 conn(0x7ff0dc0b2c00 legacy=0x7ff0dc161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.435+0000 7ff0db400640 1 -- 172.21.15.53:0/3949028048 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0dc0d1780 con 0x7ff0dc06cc60 2024-01-16T15:11:40.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.437+0000 7ff0db400640 1 --2- 172.21.15.53:0/3949028048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc06d0e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009870 tx=0x7ff0c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad40c47e397213b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.437+0000 7ff0daa00640 1 -- 172.21.15.53:0/3949028048 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4031640 con 0x7ff0dc06cc60 2024-01-16T15:11:40.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.437+0000 7ff0daa00640 1 -- 172.21.15.53:0/3949028048 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0c40317a0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.437+0000 7ff0daa00640 1 -- 172.21.15.53:0/3949028048 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0c4031b30 con 0x7ff0dc06cc60 2024-01-16T15:11:40.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.438+0000 7ff0e3200640 1 -- 172.21.15.53:0/3949028048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 msgr2=0x7ff0dc06d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.438+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3949028048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc06d0e0 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009870 tx=0x7ff0c402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.439+0000 7ff0e3200640 1 -- 172.21.15.53:0/3949028048 shutdown_connections 2024-01-16T15:11:40.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.439+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3949028048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc06d0e0 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.439+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3949028048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc15afc0 0x7ff0dc15d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.439+0000 7ff0e3200640 1 -- 172.21.15.53:0/3949028048 >> 172.21.15.53:0/3949028048 conn(0x7ff0dc06e7a0 msgr2=0x7ff0dc06eb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:40.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.439+0000 7ff0e3200640 1 -- 172.21.15.53:0/3949028048 shutdown_connections 2024-01-16T15:11:40.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.439+0000 7ff0e3200640 1 -- 172.21.15.53:0/3949028048 wait complete. 2024-01-16T15:11:40.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.441+0000 7ff0e3200640 1 Processor -- start 2024-01-16T15:11:40.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.441+0000 7ff0e3200640 1 -- start start 2024-01-16T15:11:40.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.442+0000 7ff0e3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc0c42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.442+0000 7ff0e3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc0b2c00 0x7ff0dc0ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.442+0000 7ff0e3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc15afc0 0x7ff0dc159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.442+0000 7ff0e3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0cc1e0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.442+0000 7ff0e3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0cb5d0 con 0x7ff0dc15afc0 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.442+0000 7ff0e3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0dc0cb8b0 con 0x7ff0dc0b2c00 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc0c42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc0c42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36960/0 (socket says 172.21.15.53:36960) 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 -- 172.21.15.53:0/3267225781 learned_addr learned my addr 172.21.15.53:0/3267225781 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0db400640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc0b2c00 0x7ff0dc0ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0e1200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc15afc0 0x7ff0dc159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 -- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc0b2c00 msgr2=0x7ff0dc0ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc0b2c00 0x7ff0dc0ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 -- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc15afc0 msgr2=0x7ff0dc159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc15afc0 0x7ff0dc159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0dbe00640 1 -- 172.21.15.53:0/3267225781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40094d0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.443+0000 7ff0e1200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc15afc0 0x7ff0dc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:11:40.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.444+0000 7ff0dbe00640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc0c42a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00e870 tx=0x7ff0cc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.444+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc0094f0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.444+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc00c940 con 0x7ff0dc06cc60 2024-01-16T15:11:40.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.444+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00ccf0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.444+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0dc0cbb90 con 0x7ff0dc06cc60 2024-01-16T15:11:40.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.444+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0dc159b40 con 0x7ff0dc06cc60 2024-01-16T15:11:40.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.446+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0dc0675d0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.447+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 26) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff0cc013470 con 0x7ff0dc06cc60 2024-01-16T15:11:40.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.449+0000 7ff0d3e00640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0b406bf20 0x7ff0b406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:40.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.449+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff0cc088930 con 0x7ff0dc06cc60 2024-01-16T15:11:40.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.450+0000 7ff0db400640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0b406bf20 0x7ff0b406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:40.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.452+0000 7ff0db400640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0b406bf20 0x7ff0b406e3e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4036000 tx=0x7ff0c403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:40.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.454+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff0cc054fc0 con 0x7ff0dc06cc60 2024-01-16T15:11:40.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:40.846+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541413", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0dc160400 con 0x7ff0b406bf20 2024-01-16T15:11:41.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:41.399+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff0cc055390 con 0x7ff0dc06cc60 2024-01-16T15:11:46.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.338+0000 7ff0d3e00640 1 -- 172.21.15.53:0/3267225781 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff0dc160400 con 0x7ff0b406bf20 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.341+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0b406bf20 msgr2=0x7ff0b406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.341+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0b406bf20 0x7ff0b406e3e0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4036000 tx=0x7ff0c403c040 comp rx=0 tx=0).stop 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 msgr2=0x7ff0dc0c42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc0c42a0 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00e870 tx=0x7ff0cc00ed30 comp rx=0 tx=0).stop 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 shutdown_connections 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff0b406bf20 0x7ff0b406e3e0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0dc15afc0 0x7ff0dc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0dc0b2c00 0x7ff0dc0ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 --2- 172.21.15.53:0/3267225781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0dc06cc60 0x7ff0dc0c42a0 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:46.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 >> 172.21.15.53:0/3267225781 conn(0x7ff0dc06e7a0 msgr2=0x7ff0dc06ffd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:46.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 shutdown_connections 2024-01-16T15:11:46.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:46.342+0000 7ff0e3200640 1 -- 172.21.15.53:0/3267225781 wait complete. 2024-01-16T15:11:48.341 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541414 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.655+0000 7f1506200640 1 Processor -- start 2024-01-16T15:11:48.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.656+0000 7f1506200640 1 -- start start 2024-01-16T15:11:48.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.657+0000 7f1506200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1500143ae0 0x7f15000cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.657+0000 7f1506200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f15000cf7d0 0x7f150015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.657+0000 7f1506200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15000cfc30 con 0x7f150015e1e0 2024-01-16T15:11:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.657+0000 7f1506200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15000b1ee0 con 0x7f1500143ae0 2024-01-16T15:11:48.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.657+0000 7f1506200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15000b2780 con 0x7f15000cf7d0 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14ff400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1500143ae0 0x7f15000cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14ff400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1500143ae0 0x7f15000cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50302/0 (socket says 172.21.15.53:50302) 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14ff400640 1 -- 172.21.15.53:0/1030482393 learned_addr learned my addr 172.21.15.53:0/1030482393 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14fea00640 1 --2- 172.21.15.53:0/1030482393 >> v2:172.21.15.146:3300/0 conn(0x7f15000cf7d0 0x7f150015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14ff400640 1 -- 172.21.15.53:0/1030482393 >> v2:172.21.15.146:3300/0 conn(0x7f15000cf7d0 msgr2=0x7f150015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14ff400640 1 --2- 172.21.15.53:0/1030482393 >> v2:172.21.15.146:3300/0 conn(0x7f15000cf7d0 0x7f150015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.658+0000 7f14ff400640 1 -- 172.21.15.53:0/1030482393 >> v1:172.21.15.146:6789/0 conn(0x7f150015e1e0 legacy=0x7f150015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.659+0000 7f14ff400640 1 -- 172.21.15.53:0/1030482393 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f150015e990 con 0x7f1500143ae0 2024-01-16T15:11:48.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.660+0000 7f14ff400640 1 --2- 172.21.15.53:0/1030482393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1500143ae0 0x7f15000cf290 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f15000cb830 tx=0x7f14f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3810a25e37e607a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.661+0000 7f14fe000640 1 -- 172.21.15.53:0/1030482393 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0030860 con 0x7f1500143ae0 2024-01-16T15:11:48.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.661+0000 7f14fe000640 1 -- 172.21.15.53:0/1030482393 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f00309c0 con 0x7f1500143ae0 2024-01-16T15:11:48.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.661+0000 7f14fe000640 1 -- 172.21.15.53:0/1030482393 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f0030d50 con 0x7f1500143ae0 2024-01-16T15:11:48.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.662+0000 7f1506200640 1 -- 172.21.15.53:0/1030482393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1500143ae0 msgr2=0x7f15000cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.662+0000 7f1506200640 1 --2- 172.21.15.53:0/1030482393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1500143ae0 0x7f15000cf290 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f15000cb830 tx=0x7f14f002b9a0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.662+0000 7f1506200640 1 -- 172.21.15.53:0/1030482393 shutdown_connections 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.662+0000 7f1506200640 1 --2- 172.21.15.53:0/1030482393 >> v2:172.21.15.146:3300/0 conn(0x7f15000cf7d0 0x7f150015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.662+0000 7f1506200640 1 --2- 172.21.15.53:0/1030482393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1500143ae0 0x7f15000cf290 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.662+0000 7f1506200640 1 -- 172.21.15.53:0/1030482393 >> 172.21.15.53:0/1030482393 conn(0x7f150006a800 msgr2=0x7f150006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.663+0000 7f1506200640 1 -- 172.21.15.53:0/1030482393 shutdown_connections 2024-01-16T15:11:48.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.663+0000 7f1506200640 1 -- 172.21.15.53:0/1030482393 wait complete. 2024-01-16T15:11:48.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.665+0000 7f1506200640 1 Processor -- start 2024-01-16T15:11:48.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.666+0000 7f1506200640 1 -- start start 2024-01-16T15:11:48.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.667+0000 7f1506200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 0x7f150015d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.667+0000 7f1506200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1500143ae0 0x7f1500141610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.667+0000 7f1506200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f150015e1e0 0x7f15001f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.667+0000 7f1506200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150015d5d0 con 0x7f150015e1e0 2024-01-16T15:11:48.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.667+0000 7f1506200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f150015b020 con 0x7f1500143ae0 2024-01-16T15:11:48.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.667+0000 7f1506200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f150015b300 con 0x7f15000cf7d0 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.668+0000 7f14ff400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 0x7f150015d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.668+0000 7f14fea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1500143ae0 0x7f1500141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.668+0000 7f14fea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1500143ae0 0x7f1500141610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:54024/0 (socket says 172.21.15.53:54024) 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.668+0000 7f14fea00640 1 -- 172.21.15.53:0/1720235854 learned_addr learned my addr 172.21.15.53:0/1720235854 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.668+0000 7f14ff400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 0x7f150015d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:43092/0 (socket says 172.21.15.53:43092) 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.668+0000 7f14ffe00640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f150015e1e0 0x7f15001f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.669+0000 7f14ffe00640 1 -- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 msgr2=0x7f150015d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.669+0000 7f14ffe00640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 0x7f150015d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.669+0000 7f14ffe00640 1 -- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1500143ae0 msgr2=0x7f1500141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:48.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.669+0000 7f14ffe00640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1500143ae0 0x7f1500141610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:48.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.669+0000 7f14ffe00640 1 -- 172.21.15.53:0/1720235854 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00094d0 con 0x7f150015e1e0 2024-01-16T15:11:48.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.669+0000 7f14ff400640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 0x7f150015d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:48.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.670+0000 7f14ffe00640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f150015e1e0 0x7f15001f1dd0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f14f800c690 tx=0x7f14f800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:48.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.671+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8003b70 con 0x7f150015e1e0 2024-01-16T15:11:48.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.671+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14f8003cd0 con 0x7f150015e1e0 2024-01-16T15:11:48.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.671+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14f8005310 con 0x7f150015e1e0 2024-01-16T15:11:48.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.671+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f150015b5e0 con 0x7f150015e1e0 2024-01-16T15:11:48.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.671+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f150015bba0 con 0x7f150015e1e0 2024-01-16T15:11:48.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.673+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15000b2bf0 con 0x7f150015e1e0 2024-01-16T15:11:48.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.675+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f14f8005470 con 0x7f150015e1e0 2024-01-16T15:11:48.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.677+0000 7f14f7400640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f14d806bee0 0x7f14d806e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:48.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.677+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f14f80884c0 con 0x7f150015e1e0 2024-01-16T15:11:48.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.682+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15000b2bf0 con 0x7f150015e1e0 2024-01-16T15:11:48.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.692+0000 7f14ff400640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f14d806bee0 0x7f14d806e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:48.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:48.701+0000 7f14ff400640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f14d806bee0 0x7f14d806e3a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f14f0036000 tx=0x7f14f003b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:49.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:49.069+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541414", "target": ["mon-mgr", ""]}) v1 -- 0x7f150015e670 con 0x7f14d806bee0 2024-01-16T15:11:51.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.266+0000 7f14f7400640 1 -- 172.21.15.53:0/1720235854 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f14f0030c30 con 0x7f14d806bee0 2024-01-16T15:11:51.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f14d806bee0 msgr2=0x7f14d806e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:51.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f14d806bee0 0x7f14d806e3a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f14f0036000 tx=0x7f14f003b040 comp rx=0 tx=0).stop 2024-01-16T15:11:51.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f150015e1e0 msgr2=0x7f15001f1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:51.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f150015e1e0 0x7f15001f1dd0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f14f800c690 tx=0x7f14f800cb50 comp rx=0 tx=0).stop 2024-01-16T15:11:51.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 shutdown_connections 2024-01-16T15:11:51.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f14d806bee0 0x7f14d806e3a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f150015e1e0 0x7f15001f1dd0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1500143ae0 0x7f1500141610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 --2- 172.21.15.53:0/1720235854 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15000cf7d0 0x7f150015d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:51.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 >> 172.21.15.53:0/1720235854 conn(0x7f150006a800 msgr2=0x7f15000ad3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:51.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 shutdown_connections 2024-01-16T15:11:51.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:51.269+0000 7f1506200640 1 -- 172.21.15.53:0/1720235854 wait complete. 2024-01-16T15:11:53.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541415 2024-01-16T15:11:53.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.523+0000 7fe4e0c00640 1 Processor -- start 2024-01-16T15:11:53.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.523+0000 7fe4e0c00640 1 -- start start 2024-01-16T15:11:53.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.525+0000 7fe4e0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13df80 0x7fe4dc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:53.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.525+0000 7fe4e0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4dc13e360 0x7fe4dc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:53.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.525+0000 7fe4e0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc06c470 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.525+0000 7fe4e0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc06c750 con 0x7fe4dc13e360 2024-01-16T15:11:53.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.525+0000 7fe4e0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc06ca30 con 0x7fe4dc13df80 2024-01-16T15:11:53.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4d9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13df80 0x7fe4dc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:53.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4d1200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe4dc13e360 0x7fe4dc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:53.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4da800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fe4dc15f9c0 0x7fe4dc1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:33712/0 (socket says 172.21.15.53:33712) 2024-01-16T15:11:53.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4da800640 1 -- 172.21.15.53:0/626569208 learned_addr learned my addr 172.21.15.53:0/626569208 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:11:53.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4d9e00640 1 -- 172.21.15.53:0/626569208 >> v2:172.21.15.146:3300/0 conn(0x7fe4dc13e360 msgr2=0x7fe4dc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:11:53.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4d9e00640 1 --2- 172.21.15.53:0/626569208 >> v2:172.21.15.146:3300/0 conn(0x7fe4dc13e360 0x7fe4dc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.526+0000 7fe4d9e00640 1 -- 172.21.15.53:0/626569208 >> v1:172.21.15.146:6789/0 conn(0x7fe4dc15f9c0 legacy=0x7fe4dc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:53.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.527+0000 7fe4d9e00640 1 -- 172.21.15.53:0/626569208 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc06cd10 con 0x7fe4dc13df80 2024-01-16T15:11:53.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.528+0000 7fe4d9e00640 1 --2- 172.21.15.53:0/626569208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13df80 0x7fe4dc15bee0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0097a0 tx=0x7fe4cc02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4f176d5bd9e5098 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:53.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4d9400640 1 -- 172.21.15.53:0/626569208 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc031830 con 0x7fe4dc13df80 2024-01-16T15:11:53.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4d9400640 1 -- 172.21.15.53:0/626569208 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4cc031990 con 0x7fe4dc13df80 2024-01-16T15:11:53.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4d9400640 1 -- 172.21.15.53:0/626569208 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc031d20 con 0x7fe4dc13df80 2024-01-16T15:11:53.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 -- 172.21.15.53:0/626569208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13df80 msgr2=0x7fe4dc15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:53.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/626569208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13df80 0x7fe4dc15bee0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0097a0 tx=0x7fe4cc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 -- 172.21.15.53:0/626569208 shutdown_connections 2024-01-16T15:11:53.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/626569208 >> v2:172.21.15.146:3300/0 conn(0x7fe4dc13e360 0x7fe4dc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/626569208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13df80 0x7fe4dc15bee0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 -- 172.21.15.53:0/626569208 >> 172.21.15.53:0/626569208 conn(0x7fe4dc06a800 msgr2=0x7fe4dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:53.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 -- 172.21.15.53:0/626569208 shutdown_connections 2024-01-16T15:11:53.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.529+0000 7fe4e0c00640 1 -- 172.21.15.53:0/626569208 wait complete. 2024-01-16T15:11:53.507 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:53.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.531+0000 7fe4e0c00640 1 Processor -- start 2024-01-16T15:11:53.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.531+0000 7fe4e0c00640 1 -- start start 2024-01-16T15:11:53.512 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:11:53.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.532+0000 7fe4e0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4dc13df80 0x7fe4dc14f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:53.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4e0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13e360 0x7fe4dc1f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:53.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4e0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 0x7fe4dc0b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:53.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4e0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc1f41e0 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4e0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc14fd30 con 0x7fe4dc13df80 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4e0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc14e5a0 con 0x7fe4dc13e360 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 0x7fe4dc0b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 0x7fe4dc0b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58986/0 (socket says 172.21.15.53:58986) 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 -- 172.21.15.53:0/3671879825 learned_addr learned my addr 172.21.15.53:0/3671879825 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4d1200640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13e360 0x7fe4dc1f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4d9e00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4dc13df80 0x7fe4dc14f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 -- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13e360 msgr2=0x7fe4dc1f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13e360 0x7fe4dc1f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 -- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4dc13df80 msgr2=0x7fe4dc14f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4dc13df80 0x7fe4dc14f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4da800640 1 -- 172.21.15.53:0/3671879825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4cc009400 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.533+0000 7fe4d9e00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4dc13df80 0x7fe4dc14f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:11:53.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.534+0000 7fe4da800640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 0x7fe4dc0b2cf0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400cbc0 tx=0x7fe4d400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.534+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400ad40 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.534+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d4012400 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.534+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4d400aea0 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.534+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc14e880 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.534+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc14ee40 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.535+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc1f4390 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.536+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fe4d4007070 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.537+0000 7fe4d2a00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe4b406bec0 0x7fe4b406e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:11:53.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.537+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe4d4088630 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.538+0000 7fe4d9e00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe4b406bec0 0x7fe4b406e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:11:53.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.539+0000 7fe4d9e00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe4b406bec0 0x7fe4b406e380 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc036000 tx=0x7fe4cc03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:11:53.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.542+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe4dc1f4390 con 0x7fe4dc15f9c0 2024-01-16T15:11:53.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:53.776+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541415", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4dc15eac0 con 0x7fe4b406bec0 2024-01-16T15:11:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.775+0000 7fe4d2a00640 1 -- 172.21.15.53:0/3671879825 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe4dc15eac0 con 0x7fe4b406bec0 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe4b406bec0 msgr2=0x7fe4b406e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe4b406bec0 0x7fe4b406e380 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc036000 tx=0x7fe4cc03c040 comp rx=0 tx=0).stop 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 msgr2=0x7fe4dc0b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 0x7fe4dc0b2cf0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400cbc0 tx=0x7fe4d400a430 comp rx=0 tx=0).stop 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 shutdown_connections 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe4b406bec0 0x7fe4b406e380 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4dc15f9c0 0x7fe4dc0b2cf0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe4dc13e360 0x7fe4dc1f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 --2- 172.21.15.53:0/3671879825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe4dc13df80 0x7fe4dc14f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:11:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.779+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 >> 172.21.15.53:0/3671879825 conn(0x7fe4dc06a800 msgr2=0x7fe4dc15d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:11:59.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.780+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 shutdown_connections 2024-01-16T15:11:59.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:11:59.780+0000 7fe4e0c00640 1 -- 172.21.15.53:0/3671879825 wait complete. 2024-01-16T15:12:01.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541416 2024-01-16T15:12:02.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.040+0000 7ff21b200640 1 Processor -- start 2024-01-16T15:12:02.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.040+0000 7ff21b200640 1 -- start start 2024-01-16T15:12:02.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.041+0000 7ff21b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff21c0cf9a0 0x7ff21c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:02.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.041+0000 7ff21b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff21c0b31d0 0x7ff21c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.041+0000 7ff21b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21c0d0320 con 0x7ff21c15d4b0 2024-01-16T15:12:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.041+0000 7ff21b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21c0d1c60 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.041+0000 7ff21b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff21c0b2730 con 0x7ff21c0b31d0 2024-01-16T15:12:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff21c15d4b0 0x7ff21c0d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56348/0 (socket says 172.21.15.53:56348) 2024-01-16T15:12:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219e00640 1 -- 172.21.15.53:0/4073386713 learned_addr learned my addr 172.21.15.53:0/4073386713 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219400640 1 --2- 172.21.15.53:0/4073386713 >> v2:172.21.15.53:3300/0 conn(0x7ff21c0cf9a0 0x7ff21c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff21a800640 1 --2- 172.21.15.53:0/4073386713 >> v2:172.21.15.146:3300/0 conn(0x7ff21c0b31d0 0x7ff21c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff218a00640 1 -- 172.21.15.53:0/4073386713 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813543195 0 0) 0x7ff21c0d0320 con 0x7ff21c15d4b0 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff218a00640 1 -- 172.21.15.53:0/4073386713 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc003520 con 0x7ff21c15d4b0 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219400640 1 -- 172.21.15.53:0/4073386713 >> v2:172.21.15.146:3300/0 conn(0x7ff21c0b31d0 msgr2=0x7ff21c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219400640 1 --2- 172.21.15.53:0/4073386713 >> v2:172.21.15.146:3300/0 conn(0x7ff21c0b31d0 0x7ff21c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219400640 1 -- 172.21.15.53:0/4073386713 >> v1:172.21.15.53:6789/0 conn(0x7ff21c15d4b0 legacy=0x7ff21c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff219400640 1 -- 172.21.15.53:0/4073386713 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff21c0d1ec0 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.042+0000 7ff21a800640 1 --2- 172.21.15.53:0/4073386713 >> v2:172.21.15.146:3300/0 conn(0x7ff21c0b31d0 0x7ff21c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:02.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.043+0000 7ff219400640 1 --2- 172.21.15.53:0/4073386713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0cfde0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff21c072a30 tx=0x7ff20402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60c6932196b257b2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:02.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff218a00640 1 -- 172.21.15.53:0/4073386713 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204031830 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff218a00640 1 -- 172.21.15.53:0/4073386713 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204031990 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff218a00640 1 -- 172.21.15.53:0/4073386713 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204031d20 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff21b200640 1 -- 172.21.15.53:0/4073386713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 msgr2=0x7ff21c0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff21b200640 1 --2- 172.21.15.53:0/4073386713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0cfde0 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7ff21c072a30 tx=0x7ff20402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff21b200640 1 -- 172.21.15.53:0/4073386713 shutdown_connections 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.044+0000 7ff21b200640 1 --2- 172.21.15.53:0/4073386713 >> v2:172.21.15.146:3300/0 conn(0x7ff21c0b31d0 0x7ff21c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.045+0000 7ff21b200640 1 --2- 172.21.15.53:0/4073386713 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0cfde0 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.045+0000 7ff21b200640 1 -- 172.21.15.53:0/4073386713 >> 172.21.15.53:0/4073386713 conn(0x7ff21c0d3000 msgr2=0x7ff21c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.045+0000 7ff21b200640 1 -- 172.21.15.53:0/4073386713 shutdown_connections 2024-01-16T15:12:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.045+0000 7ff21b200640 1 -- 172.21.15.53:0/4073386713 wait complete. 2024-01-16T15:12:02.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 Processor -- start 2024-01-16T15:12:02.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 -- start start 2024-01-16T15:12:02.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff21c0b31d0 0x7ff21c152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21c15d4b0 0x7ff21c09dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c09f550 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c152a40 con 0x7ff21c15d4b0 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.047+0000 7ff21b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21c1f3ba0 con 0x7ff21c0b31d0 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0ca0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47058/0 (socket says 172.21.15.53:47058) 2024-01-16T15:12:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 -- 172.21.15.53:0/3557284692 learned_addr learned my addr 172.21.15.53:0/3557284692 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219e00640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff21c0b31d0 0x7ff21c152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 -- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff21c0b31d0 msgr2=0x7ff21c152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff21c0b31d0 0x7ff21c152500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 -- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21c15d4b0 msgr2=0x7ff21c09dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21c15d4b0 0x7ff21c09dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.048+0000 7ff219400640 1 -- 172.21.15.53:0/3557284692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204009400 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.049+0000 7ff219400640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0ca0d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ff204000c00 tx=0x7ff2040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.049+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20403a830 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.049+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20403a9b0 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.049+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20403ad60 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.049+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff21c09ce80 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.049+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff21c09d440 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.050+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff21c067450 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.051+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7ff20404d450 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.052+0000 7ff212000640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1f406be90 0x7ff1f406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.052+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff2040aa960 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.053+0000 7ff219e00640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1f406be90 0x7ff1f406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:02.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.054+0000 7ff219e00640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1f406be90 0x7ff1f406e350 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0011e0 tx=0x7ff20c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:02.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.058+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff204076f00 con 0x7ff21c0cf9a0 2024-01-16T15:12:02.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:02.293+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541416", "target": ["mon-mgr", ""]}) v1 -- 0x7ff21c0667d0 con 0x7ff1f406be90 2024-01-16T15:12:11.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.622+0000 7ff212000640 1 -- 172.21.15.53:0/3557284692 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff21c0667d0 con 0x7ff1f406be90 2024-01-16T15:12:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1f406be90 msgr2=0x7ff1f406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1f406be90 0x7ff1f406e350 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0011e0 tx=0x7ff20c009040 comp rx=0 tx=0).stop 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 msgr2=0x7ff21c0ca0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0ca0d0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7ff204000c00 tx=0x7ff2040046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 shutdown_connections 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff1f406be90 0x7ff1f406e350 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff21c15d4b0 0x7ff21c09dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff21c0cf9a0 0x7ff21c0ca0d0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 --2- 172.21.15.53:0/3557284692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff21c0b31d0 0x7ff21c152500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 >> 172.21.15.53:0/3557284692 conn(0x7ff21c0d3000 msgr2=0x7ff21c0cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 shutdown_connections 2024-01-16T15:12:11.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:11.626+0000 7ff21b200640 1 -- 172.21.15.53:0/3557284692 wait complete. 2024-01-16T15:12:13.624 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541417 2024-01-16T15:12:13.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.903+0000 7fd05fe00640 1 Processor -- start 2024-01-16T15:12:13.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.903+0000 7fd05fe00640 1 -- start start 2024-01-16T15:12:13.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.904+0000 7fd05fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0580cf9a0 0x7fd058166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.904+0000 7fd05fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05815c5f0 con 0x7fd05813df80 2024-01-16T15:12:13.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.904+0000 7fd05fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd05806c470 con 0x7fd05813e360 2024-01-16T15:12:13.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.904+0000 7fd05fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05806c750 con 0x7fd0580cf9a0 2024-01-16T15:12:13.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.904+0000 7fd05d400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd05813df80 0x7fd05815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:53588/0 (socket says 172.21.15.53:53588) 2024-01-16T15:12:13.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.904+0000 7fd05d400640 1 -- 172.21.15.53:0/2793760208 learned_addr learned my addr 172.21.15.53:0/2793760208 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:13.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd05de00640 1 --2- 172.21.15.53:0/2793760208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0580cf9a0 0x7fd058166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962047934 0 0) 0x7fd05815c5f0 con 0x7fd05813df80 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd038003520 con 0x7fd05813df80 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167818358 0 0) 0x7fd05806c470 con 0x7fd05813e360 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd05815c5f0 con 0x7fd05813e360 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1462221135 0 0) 0x7fd038003520 con 0x7fd05813df80 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd05de00640 1 -- 172.21.15.53:0/2793760208 >> v1:172.21.15.146:6789/0 conn(0x7fd05813e360 legacy=0x7fd0580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd05de00640 1 -- 172.21.15.53:0/2793760208 >> v1:172.21.15.53:6789/0 conn(0x7fd05813df80 legacy=0x7fd05815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.905+0000 7fd05de00640 1 -- 172.21.15.53:0/2793760208 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05806ca30 con 0x7fd0580cf9a0 2024-01-16T15:12:13.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.906+0000 7fd05de00640 1 --2- 172.21.15.53:0/2793760208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0580cf9a0 0x7fd058166030 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd050009dc0 tx=0x7fd05002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e6e0f92d5c8dccf7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048001670 con 0x7fd0580cf9a0 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05002c910 con 0x7fd0580cf9a0 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd057e00640 1 -- 172.21.15.53:0/2793760208 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05002cca0 con 0x7fd0580cf9a0 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd05fe00640 1 -- 172.21.15.53:0/2793760208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0580cf9a0 msgr2=0x7fd058166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd05fe00640 1 --2- 172.21.15.53:0/2793760208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0580cf9a0 0x7fd058166030 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd050009dc0 tx=0x7fd05002bde0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd05fe00640 1 -- 172.21.15.53:0/2793760208 shutdown_connections 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd05fe00640 1 --2- 172.21.15.53:0/2793760208 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0580cf9a0 0x7fd058166030 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd05fe00640 1 -- 172.21.15.53:0/2793760208 >> 172.21.15.53:0/2793760208 conn(0x7fd05806a800 msgr2=0x7fd05806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.907+0000 7fd05fe00640 1 -- 172.21.15.53:0/2793760208 shutdown_connections 2024-01-16T15:12:13.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.908+0000 7fd05fe00640 1 -- 172.21.15.53:0/2793760208 wait complete. 2024-01-16T15:12:13.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.909+0000 7fd05fe00640 1 Processor -- start 2024-01-16T15:12:13.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.910+0000 7fd05fe00640 1 -- start start 2024-01-16T15:12:13.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.911+0000 7fd05fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 0x7fd05813bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd05813df80 0x7fd05815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd05813e360 0x7fd0580b2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0581e4b30 con 0x7fd0580cf9a0 2024-01-16T15:12:13.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd05813b790 con 0x7fd05813df80 2024-01-16T15:12:13.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd05813ab80 con 0x7fd05813e360 2024-01-16T15:12:13.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 0x7fd05813bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 0x7fd05813bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41578/0 (socket says 172.21.15.53:41578) 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.912+0000 7fd05d400640 1 -- 172.21.15.53:0/3270678876 learned_addr learned my addr 172.21.15.53:0/3270678876 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05de00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd05813e360 0x7fd0580b2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05ca00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd05813df80 0x7fd05815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05d400640 1 -- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd05813e360 msgr2=0x7fd0580b2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05d400640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd05813e360 0x7fd0580b2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05d400640 1 -- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd05813df80 msgr2=0x7fd05815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05d400640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd05813df80 0x7fd05815d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05d400640 1 -- 172.21.15.53:0/3270678876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd050009a70 con 0x7fd0580cf9a0 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05de00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd05813e360 0x7fd0580b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05ca00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd05813df80 0x7fd05815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.913+0000 7fd05d400640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 0x7fd05813bc20 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c690 tx=0x7fd04800cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.914+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048005ad0 con 0x7fd0580cf9a0 2024-01-16T15:12:13.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.914+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd048005c50 con 0x7fd0580cf9a0 2024-01-16T15:12:13.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.914+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd05813ae60 con 0x7fd0580cf9a0 2024-01-16T15:12:13.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.914+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd048007070 con 0x7fd0580cf9a0 2024-01-16T15:12:13.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.914+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd05813c160 con 0x7fd0580cf9a0 2024-01-16T15:12:13.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.915+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0580675d0 con 0x7fd0580cf9a0 2024-01-16T15:12:13.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.918+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fd0480071d0 con 0x7fd0580cf9a0 2024-01-16T15:12:13.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.919+0000 7fd055600640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd03006be90 0x7fd03006e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:13.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.920+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd048087fa0 con 0x7fd0580cf9a0 2024-01-16T15:12:13.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.921+0000 7fd05ca00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd03006be90 0x7fd03006e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:13.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.923+0000 7fd05ca00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd03006be90 0x7fd03006e350 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd0580b36e0 tx=0x7fd040009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:13.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:13.926+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd048054660 con 0x7fd0580cf9a0 2024-01-16T15:12:14.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:14.283+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541417", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0581559b0 con 0x7fd03006be90 2024-01-16T15:12:19.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.809+0000 7fd055600640 1 -- 172.21.15.53:0/3270678876 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd0581559b0 con 0x7fd03006be90 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd03006be90 msgr2=0x7fd03006e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd03006be90 0x7fd03006e350 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd0580b36e0 tx=0x7fd040009040 comp rx=0 tx=0).stop 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 msgr2=0x7fd05813bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 0x7fd05813bc20 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7fd04800c690 tx=0x7fd04800cb50 comp rx=0 tx=0).stop 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 shutdown_connections 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd03006be90 0x7fd03006e350 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd05813e360 0x7fd0580b2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd05813df80 0x7fd05815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 --2- 172.21.15.53:0/3270678876 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0580cf9a0 0x7fd05813bc20 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 >> 172.21.15.53:0/3270678876 conn(0x7fd05806a800 msgr2=0x7fd05806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 shutdown_connections 2024-01-16T15:12:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:19.813+0000 7fd05fe00640 1 -- 172.21.15.53:0/3270678876 wait complete. 2024-01-16T15:12:21.810 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541418 2024-01-16T15:12:22.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.115+0000 7f3869200640 1 Processor -- start 2024-01-16T15:12:22.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.115+0000 7f3869200640 1 -- start start 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3869200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f386406a3e0 0x7f38640ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3869200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f38640d0b60 0x7f38640aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3869200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3864071de0 con 0x7f386406a7c0 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3869200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38640af320 con 0x7f386406a3e0 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3869200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3864072030 con 0x7f38640d0b60 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3861e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f386406a3e0 0x7f38640ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3861e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f386406a3e0 0x7f38640ac8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32906/0 (socket says 172.21.15.53:32906) 2024-01-16T15:12:22.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.117+0000 7f3861e00640 1 -- 172.21.15.53:0/3207694397 learned_addr learned my addr 172.21.15.53:0/3207694397 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.118+0000 7f3859200640 1 --2- 172.21.15.53:0/3207694397 >> v2:172.21.15.146:3300/0 conn(0x7f38640d0b60 0x7f38640aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.118+0000 7f3861e00640 1 -- 172.21.15.53:0/3207694397 >> v2:172.21.15.146:3300/0 conn(0x7f38640d0b60 msgr2=0x7f38640aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.118+0000 7f3861e00640 1 --2- 172.21.15.53:0/3207694397 >> v2:172.21.15.146:3300/0 conn(0x7f38640d0b60 0x7f38640aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.118+0000 7f3861e00640 1 -- 172.21.15.53:0/3207694397 >> v1:172.21.15.146:6789/0 conn(0x7f386406a7c0 legacy=0x7f386406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.118+0000 7f3861e00640 1 -- 172.21.15.53:0/3207694397 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3864072310 con 0x7f386406a3e0 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.118+0000 7f3859200640 1 --2- 172.21.15.53:0/3207694397 >> v2:172.21.15.146:3300/0 conn(0x7f38640d0b60 0x7f38640aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:22.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.119+0000 7f3861e00640 1 --2- 172.21.15.53:0/3207694397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a3e0 0x7f38640ac8d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f38640b6120 tx=0x7f385402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330f85a233df8026 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.119+0000 7f3861400640 1 -- 172.21.15.53:0/3207694397 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854031870 con 0x7f386406a3e0 2024-01-16T15:12:22.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.119+0000 7f3861400640 1 -- 172.21.15.53:0/3207694397 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38540319d0 con 0x7f386406a3e0 2024-01-16T15:12:22.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.119+0000 7f3861400640 1 -- 172.21.15.53:0/3207694397 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3854031d60 con 0x7f386406a3e0 2024-01-16T15:12:22.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.120+0000 7f3869200640 1 -- 172.21.15.53:0/3207694397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a3e0 msgr2=0x7f38640ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.120+0000 7f3869200640 1 --2- 172.21.15.53:0/3207694397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a3e0 0x7f38640ac8d0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f38640b6120 tx=0x7f385402b910 comp rx=0 tx=0).stop 2024-01-16T15:12:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.120+0000 7f3869200640 1 -- 172.21.15.53:0/3207694397 shutdown_connections 2024-01-16T15:12:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.120+0000 7f3869200640 1 --2- 172.21.15.53:0/3207694397 >> v2:172.21.15.146:3300/0 conn(0x7f38640d0b60 0x7f38640aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.120+0000 7f3869200640 1 --2- 172.21.15.53:0/3207694397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a3e0 0x7f38640ac8d0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.120+0000 7f3869200640 1 -- 172.21.15.53:0/3207694397 >> 172.21.15.53:0/3207694397 conn(0x7f38640cfef0 msgr2=0x7f38640d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.121+0000 7f3869200640 1 -- 172.21.15.53:0/3207694397 shutdown_connections 2024-01-16T15:12:22.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.121+0000 7f3869200640 1 -- 172.21.15.53:0/3207694397 wait complete. 2024-01-16T15:12:22.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.123+0000 7f3869200640 1 Processor -- start 2024-01-16T15:12:22.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.123+0000 7f3869200640 1 -- start start 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3869200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f386406a3e0 0x7f3864153070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3869200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 0x7f38640ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3869200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38640d0b60 0x7f3864155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3869200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640cdf60 con 0x7f386406a7c0 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3869200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38640cd350 con 0x7f38640d0b60 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3869200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38640cd630 con 0x7f386406a3e0 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3859200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 0x7f38640ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3859200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 0x7f38640ce3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32916/0 (socket says 172.21.15.53:32916) 2024-01-16T15:12:22.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.124+0000 7f3859200640 1 -- 172.21.15.53:0/2703043405 learned_addr learned my addr 172.21.15.53:0/2703043405 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:22.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3862800640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38640d0b60 0x7f3864155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3859200640 1 -- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f386406a3e0 msgr2=0x7f3864153070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3859200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f386406a3e0 0x7f3864153070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3859200640 1 -- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38640d0b60 msgr2=0x7f3864155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:22.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3859200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38640d0b60 0x7f3864155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:22.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3859200640 1 -- 172.21.15.53:0/2703043405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38540094d0 con 0x7f386406a7c0 2024-01-16T15:12:22.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.125+0000 7f3859200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 0x7f38640ce3f0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f384c00c7f0 tx=0x7f384c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.126+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c00cea0 con 0x7f386406a7c0 2024-01-16T15:12:22.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.126+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c007a30 con 0x7f386406a7c0 2024-01-16T15:12:22.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.126+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c007dc0 con 0x7f386406a7c0 2024-01-16T15:12:22.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.126+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38640cd910 con 0x7f386406a7c0 2024-01-16T15:12:22.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.126+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38640ce930 con 0x7f386406a7c0 2024-01-16T15:12:22.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.128+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38640675d0 con 0x7f386406a7c0 2024-01-16T15:12:22.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.129+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f384c0113f0 con 0x7f386406a7c0 2024-01-16T15:12:22.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.130+0000 7f385aa00640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f383c06bec0 0x7f383c06e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:22.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.130+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f384c088730 con 0x7f386406a7c0 2024-01-16T15:12:22.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.132+0000 7f3861e00640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f383c06bec0 0x7f383c06e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:22.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.135+0000 7f3861e00640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f383c06bec0 0x7f383c06e380 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3854036000 tx=0x7f3854030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:22.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.142+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f384c054ff0 con 0x7f386406a7c0 2024-01-16T15:12:22.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:22.516+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541418", "target": ["mon-mgr", ""]}) v1 -- 0x7f38640cd7a0 con 0x7f383c06bec0 2024-01-16T15:12:23.640 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:23.648 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:24.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.711+0000 7f385aa00640 1 -- 172.21.15.53:0/2703043405 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f38640cd7a0 con 0x7f383c06bec0 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f383c06bec0 msgr2=0x7f383c06e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f383c06bec0 0x7f383c06e380 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f3854036000 tx=0x7f3854030000 comp rx=0 tx=0).stop 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 msgr2=0x7f38640ce3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 0x7f38640ce3f0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f384c00c7f0 tx=0x7f384c00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 shutdown_connections 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f383c06bec0 0x7f383c06e380 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f38640d0b60 0x7f3864155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f386406a7c0 0x7f38640ce3f0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 --2- 172.21.15.53:0/2703043405 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f386406a3e0 0x7f3864153070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 >> 172.21.15.53:0/2703043405 conn(0x7f38640cfef0 msgr2=0x7f38640b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 shutdown_connections 2024-01-16T15:12:24.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:24.714+0000 7f3869200640 1 -- 172.21.15.53:0/2703043405 wait complete. 2024-01-16T15:12:26.711 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541419 2024-01-16T15:12:26.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.969+0000 7f25c1400640 1 Processor -- start 2024-01-16T15:12:26.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.970+0000 7f25c1400640 1 -- start start 2024-01-16T15:12:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.971+0000 7f25c1400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25bc13cf10 0x7f25bc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.971+0000 7f25c1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25bc13fae0 0x7f25bc0cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.971+0000 7f25c1400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc0ce440 con 0x7f25bc0cdc20 2024-01-16T15:12:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.971+0000 7f25c1400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc13ff90 con 0x7f25bc13fae0 2024-01-16T15:12:26.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.971+0000 7f25c1400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc0ce720 con 0x7f25bc13cf10 2024-01-16T15:12:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25b3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25bc13fae0 0x7f25bc0cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25c0a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f25bc0cdc20 0x7f25bc15d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51306/0 (socket says 172.21.15.53:51306) 2024-01-16T15:12:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25c0a00640 1 -- 172.21.15.53:0/4194431297 learned_addr learned my addr 172.21.15.53:0/4194431297 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25bbe00640 1 --2- 172.21.15.53:0/4194431297 >> v2:172.21.15.146:3300/0 conn(0x7f25bc13cf10 0x7f25bc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25b3e00640 1 -- 172.21.15.53:0/4194431297 >> v2:172.21.15.146:3300/0 conn(0x7f25bc13cf10 msgr2=0x7f25bc13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:26.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25b3e00640 1 --2- 172.21.15.53:0/4194431297 >> v2:172.21.15.146:3300/0 conn(0x7f25bc13cf10 0x7f25bc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25b3e00640 1 -- 172.21.15.53:0/4194431297 >> v1:172.21.15.53:6789/0 conn(0x7f25bc0cdc20 legacy=0x7f25bc15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25b3e00640 1 -- 172.21.15.53:0/4194431297 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc0cea00 con 0x7f25bc13fae0 2024-01-16T15:12:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.972+0000 7f25bb400640 1 -- 172.21.15.53:0/4194431297 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775346592 0 0) 0x7f25bc0ce440 con 0x7f25bc0cdc20 2024-01-16T15:12:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.973+0000 7f25b3e00640 1 --2- 172.21.15.53:0/4194431297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13fae0 0x7f25bc0cd6e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f25bc06e390 tx=0x7f25a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4685d627df476cda server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.974+0000 7f25bb400640 1 -- 172.21.15.53:0/4194431297 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4031830 con 0x7f25bc13fae0 2024-01-16T15:12:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.974+0000 7f25bb400640 1 -- 172.21.15.53:0/4194431297 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a4031990 con 0x7f25bc13fae0 2024-01-16T15:12:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.974+0000 7f25bb400640 1 -- 172.21.15.53:0/4194431297 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a4031d20 con 0x7f25bc13fae0 2024-01-16T15:12:26.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.974+0000 7f25c1400640 1 -- 172.21.15.53:0/4194431297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13fae0 msgr2=0x7f25bc0cd6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.974+0000 7f25c1400640 1 --2- 172.21.15.53:0/4194431297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13fae0 0x7f25bc0cd6e0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f25bc06e390 tx=0x7f25a402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.975+0000 7f25c1400640 1 -- 172.21.15.53:0/4194431297 shutdown_connections 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.975+0000 7f25c1400640 1 --2- 172.21.15.53:0/4194431297 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13fae0 0x7f25bc0cd6e0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.975+0000 7f25c1400640 1 --2- 172.21.15.53:0/4194431297 >> v2:172.21.15.146:3300/0 conn(0x7f25bc13cf10 0x7f25bc13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.975+0000 7f25c1400640 1 -- 172.21.15.53:0/4194431297 >> 172.21.15.53:0/4194431297 conn(0x7f25bc06a7d0 msgr2=0x7f25bc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.975+0000 7f25c1400640 1 -- 172.21.15.53:0/4194431297 shutdown_connections 2024-01-16T15:12:26.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.975+0000 7f25c1400640 1 -- 172.21.15.53:0/4194431297 wait complete. 2024-01-16T15:12:26.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.976+0000 7f25c1400640 1 Processor -- start 2024-01-16T15:12:26.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 -- start start 2024-01-16T15:12:26.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25bc0cdc20 0x7f25bc15d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 0x7f25bc159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25bc13fae0 0x7f25bc136540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc15a170 con 0x7f25bc13cf10 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc15bf70 con 0x7f25bc13fae0 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.977+0000 7f25c1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc15cdd0 con 0x7f25bc0cdc20 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 0x7f25bc159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 0x7f25bc159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32936/0 (socket says 172.21.15.53:32936) 2024-01-16T15:12:26.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 -- 172.21.15.53:0/655796103 learned_addr learned my addr 172.21.15.53:0/655796103 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:26.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25bbe00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25bc0cdc20 0x7f25bc15d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25c0a00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25bc13fae0 0x7f25bc136540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 -- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25bc0cdc20 msgr2=0x7f25bc15d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25bc0cdc20 0x7f25bc15d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 -- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25bc13fae0 msgr2=0x7f25bc136540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25bc13fae0 0x7f25bc136540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25b3e00640 1 -- 172.21.15.53:0/655796103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25a4009400 con 0x7f25bc13cf10 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25bbe00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25bc0cdc20 0x7f25bc15d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.978+0000 7f25c0a00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25bc13fae0 0x7f25bc136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.979+0000 7f25b3e00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 0x7f25bc159c30 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009890 tx=0x7f25a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.979+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a403aa60 con 0x7f25bc13cf10 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.979+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25a403abc0 con 0x7f25bc13cf10 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.979+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc15c1c0 con 0x7f25bc13cf10 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.979+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25a40423f0 con 0x7f25bc13cf10 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.979+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc15c780 con 0x7f25bc13cf10 2024-01-16T15:12:26.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.980+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25bc067450 con 0x7f25bc13cf10 2024-01-16T15:12:26.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.981+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f25a404b3b0 con 0x7f25bc13cf10 2024-01-16T15:12:26.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.982+0000 7f25b8c00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f259406be90 0x7f259406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:26.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.982+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f25a40aa7e0 con 0x7f25bc13cf10 2024-01-16T15:12:26.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.983+0000 7f25bbe00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f259406be90 0x7f259406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:26.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.984+0000 7f25bbe00640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f259406be90 0x7f259406e350 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f25ac008890 tx=0x7f25ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:26.987+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25a4076f60 con 0x7f25bc13cf10 2024-01-16T15:12:27.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:27.221+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541419", "target": ["mon-mgr", ""]}) v1 -- 0x7f25bc0b2070 con 0x7f259406be90 2024-01-16T15:12:33.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.292+0000 7f25b8c00640 1 -- 172.21.15.53:0/655796103 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f25ac00c040 con 0x7f259406be90 2024-01-16T15:12:33.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f259406be90 msgr2=0x7f259406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:33.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f259406be90 0x7f259406e350 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f25ac008890 tx=0x7f25ac008040 comp rx=0 tx=0).stop 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 msgr2=0x7f25bc159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 0x7f25bc159c30 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f25a4009890 tx=0x7f25a40046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 shutdown_connections 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f259406be90 0x7f259406e350 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25bc13fae0 0x7f25bc136540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25bc13cf10 0x7f25bc159c30 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 --2- 172.21.15.53:0/655796103 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25bc0cdc20 0x7f25bc15d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.297+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 >> 172.21.15.53:0/655796103 conn(0x7f25bc06a7d0 msgr2=0x7f25bc13d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.298+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 shutdown_connections 2024-01-16T15:12:33.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:33.298+0000 7f25c1400640 1 -- 172.21.15.53:0/655796103 wait complete. 2024-01-16T15:12:35.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone541420 2024-01-16T15:12:35.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.555+0000 7f3cf4000640 1 Processor -- start 2024-01-16T15:12:35.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.555+0000 7f3cf4000640 1 -- start start 2024-01-16T15:12:35.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.557+0000 7f3cf4000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cec13df80 0x7f3cec15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.557+0000 7f3cf4000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cec06c470 con 0x7f3cec0cf9a0 2024-01-16T15:12:35.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.557+0000 7f3cf4000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cec06c750 con 0x7f3cec13e360 2024-01-16T15:12:35.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.557+0000 7f3cf4000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cec06ca30 con 0x7f3cec13df80 2024-01-16T15:12:35.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.557+0000 7f3cf2000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3cec0cf9a0 0x7f3cec0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:60302/0 (socket says 172.21.15.53:60302) 2024-01-16T15:12:35.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.557+0000 7f3cf2000640 1 -- 172.21.15.53:0/1742557561 learned_addr learned my addr 172.21.15.53:0/1742557561 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:35.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cf1600640 1 --2- 172.21.15.53:0/1742557561 >> v2:172.21.15.146:3300/0 conn(0x7f3cec13df80 0x7f3cec15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956150832 0 0) 0x7f3cec06c470 con 0x7f3cec0cf9a0 2024-01-16T15:12:35.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc003520 con 0x7f3cec0cf9a0 2024-01-16T15:12:35.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 922068265 0 0) 0x7f3ccc003520 con 0x7f3cec0cf9a0 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 >> v2:172.21.15.146:3300/0 conn(0x7f3cec13df80 msgr2=0x7f3cec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 --2- 172.21.15.53:0/1742557561 >> v2:172.21.15.146:3300/0 conn(0x7f3cec13df80 0x7f3cec15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 >> v1:172.21.15.146:6789/0 conn(0x7f3cec13e360 legacy=0x7f3cec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec06cd10 con 0x7f3cec0cf9a0 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3ce4001670 con 0x7f3cec0cf9a0 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 >> v1:172.21.15.53:6789/0 conn(0x7f3cec0cf9a0 legacy=0x7f3cec0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc0039c0 0x7f3ccc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc0047c0 0x7f3ccc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005180 0x7f3ccc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec06ca30 con 0x7f3ccc0039c0 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00c080 con 0x7f3ccc0047c0 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.558+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00c240 con 0x7f3ccc005180 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf1600640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc0039c0 0x7f3ccc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc0047c0 0x7f3ccc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf2000640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005180 0x7f3ccc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf1600640 1 -- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005180 msgr2=0x7f3ccc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf1600640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005180 0x7f3ccc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf1600640 1 -- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc0047c0 msgr2=0x7f3ccc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf1600640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc0047c0 0x7f3ccc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf1600640 1 -- 172.21.15.53:0/1742557561 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc003520 con 0x7f3ccc0039c0 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc0047c0 0x7f3ccc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf2000640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005180 0x7f3ccc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.559+0000 7f3cf2000640 1 -- 172.21.15.53:0/1742557561 reap_dead start 2024-01-16T15:12:35.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.560+0000 7f3cf1600640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc0039c0 0x7f3ccc003de0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009ac0 tx=0x7f3ccc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6313d41dc974327 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc02c6b0 con 0x7f3ccc0039c0 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cdc02c810 con 0x7f3ccc0039c0 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cebe00640 1 -- 172.21.15.53:0/1742557561 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc02cba0 con 0x7f3ccc0039c0 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 -- 172.21.15.53:0/1742557561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc0039c0 msgr2=0x7f3ccc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc0039c0 0x7f3ccc003de0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009ac0 tx=0x7f3ccc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 -- 172.21.15.53:0/1742557561 shutdown_connections 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 --2- 172.21.15.53:0/1742557561 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc0039c0 0x7f3ccc003de0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 -- 172.21.15.53:0/1742557561 >> 172.21.15.53:0/1742557561 conn(0x7f3cec06a800 msgr2=0x7f3cec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:35.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 -- 172.21.15.53:0/1742557561 shutdown_connections 2024-01-16T15:12:35.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.561+0000 7f3cf4000640 1 -- 172.21.15.53:0/1742557561 wait complete. 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.563+0000 7f3cf4000640 1 Processor -- start 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.563+0000 7f3cf4000640 1 -- start start 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf4000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc0039c0 0x7f3cec15db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf4000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 0x7f3cec0cb980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf4000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0c3f30 0x7f3cec0c43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf4000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0cbec0 con 0x7f3cec0cb500 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf4000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0c4ca0 con 0x7f3cec0c3f30 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf4000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec14c000 con 0x7f3ccc0039c0 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.564+0000 7f3cf0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 0x7f3cec0cb980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 0x7f3cec0cb980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47846/0 (socket says 172.21.15.53:47846) 2024-01-16T15:12:35.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 -- 172.21.15.53:0/120216780 learned_addr learned my addr 172.21.15.53:0/120216780 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:35.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf2000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0c3f30 0x7f3cec0c43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf1600640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc0039c0 0x7f3cec15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 -- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc0039c0 msgr2=0x7f3cec15db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc0039c0 0x7f3cec15db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 -- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0c3f30 msgr2=0x7f3cec0c43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0c3f30 0x7f3cec0c43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 -- 172.21.15.53:0/120216780 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec14c2e0 con 0x7f3cec0cb500 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf1600640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc0039c0 0x7f3cec15db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf2000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0c3f30 0x7f3cec0c43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.565+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 0x7f3cec0cb980 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400db10 tx=0x7f3cd40020a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.566+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4017070 con 0x7f3cec0cb500 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.566+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd4003ba0 con 0x7f3cec0cb500 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.566+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4006b50 con 0x7f3cec0cb500 2024-01-16T15:12:35.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.566+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec14c5c0 con 0x7f3cec0cb500 2024-01-16T15:12:35.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.566+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec0c4920 con 0x7f3cec0cb500 2024-01-16T15:12:35.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.567+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cec14be60 con 0x7f3cec0cb500 2024-01-16T15:12:35.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.568+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f3cd4006cb0 con 0x7f3cec0cb500 2024-01-16T15:12:35.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.569+0000 7f3ce9600640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406be90 0x7f3cc406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:35.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.569+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3cd4088270 con 0x7f3cec0cb500 2024-01-16T15:12:35.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.570+0000 7f3cf1600640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406be90 0x7f3cc406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:35.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.571+0000 7f3cf1600640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406be90 0x7f3cc406e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc036000 tx=0x7f3cdc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:35.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.574+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cec14be60 con 0x7f3cec0cb500 2024-01-16T15:12:35.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:35.810+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone541420", "target": ["mon-mgr", ""]}) v1 -- 0x7f3cec0cf290 con 0x7f3cc406be90 2024-01-16T15:12:49.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.107+0000 7f3ce9600640 1 -- 172.21.15.53:0/120216780 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f3cdc0041c0 con 0x7f3cc406be90 2024-01-16T15:12:49.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406be90 msgr2=0x7f3cc406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406be90 0x7f3cc406e350 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc036000 tx=0x7f3cdc030000 comp rx=0 tx=0).stop 2024-01-16T15:12:49.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 msgr2=0x7f3cec0cb980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:49.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 0x7f3cec0cb980 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400db10 tx=0x7f3cd40020a0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 shutdown_connections 2024-01-16T15:12:49.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406be90 0x7f3cc406e350 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0c3f30 0x7f3cec0c43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0cb500 0x7f3cec0cb980 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 --2- 172.21.15.53:0/120216780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc0039c0 0x7f3cec15db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 >> 172.21.15.53:0/120216780 conn(0x7f3cec06a800 msgr2=0x7f3cec0af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 shutdown_connections 2024-01-16T15:12:49.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:49.111+0000 7f3cf4000640 1 -- 172.21.15.53:0/120216780 wait complete. 2024-01-16T15:12:51.111 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:51.112 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54141 2024-01-16T15:12:51.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.370+0000 7f49e5c00640 1 Processor -- start 2024-01-16T15:12:51.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.370+0000 7f49e5c00640 1 -- start start 2024-01-16T15:12:51.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49e5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00ae5a0 0x7f49e015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49e5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49e00cf290 0x7f49e00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49e5c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e0070c40 con 0x7f49e00b3ae0 2024-01-16T15:12:51.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49e5c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e0071450 con 0x7f49e00cf290 2024-01-16T15:12:51.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49e5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e00cfc60 con 0x7f49e00ae5a0 2024-01-16T15:12:51.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49df400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49e00cf290 0x7f49e00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49df400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49e00cf290 0x7f49e00cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49042/0 (socket says 172.21.15.53:49042) 2024-01-16T15:12:51.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49d7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f49e00b3ae0 0x7f49e006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54190/0 (socket says 172.21.15.53:54190) 2024-01-16T15:12:51.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.372+0000 7f49df400640 1 -- 172.21.15.53:0/2792397921 learned_addr learned my addr 172.21.15.53:0/2792397921 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:51.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.373+0000 7f49dea00640 1 --2- 172.21.15.53:0/2792397921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00ae5a0 0x7f49e015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.373+0000 7f49df400640 1 -- 172.21.15.53:0/2792397921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00ae5a0 msgr2=0x7f49e015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.373+0000 7f49df400640 1 --2- 172.21.15.53:0/2792397921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00ae5a0 0x7f49e015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.373+0000 7f49df400640 1 -- 172.21.15.53:0/2792397921 >> v1:172.21.15.53:6789/0 conn(0x7f49e00b3ae0 legacy=0x7f49e006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.373+0000 7f49df400640 1 -- 172.21.15.53:0/2792397921 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e006c470 con 0x7f49e00cf290 2024-01-16T15:12:51.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.373+0000 7f49de000640 1 -- 172.21.15.53:0/2792397921 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763415503 0 0) 0x7f49e0070c40 con 0x7f49e00b3ae0 2024-01-16T15:12:51.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.374+0000 7f49df400640 1 --2- 172.21.15.53:0/2792397921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00cf290 0x7f49e00cf720 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e50 tx=0x7f49d802bff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f7cc3685214684 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.374+0000 7f49de000640 1 -- 172.21.15.53:0/2792397921 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d802cb90 con 0x7f49e00cf290 2024-01-16T15:12:51.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.374+0000 7f49de000640 1 -- 172.21.15.53:0/2792397921 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d802ccf0 con 0x7f49e00cf290 2024-01-16T15:12:51.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.374+0000 7f49de000640 1 -- 172.21.15.53:0/2792397921 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8034610 con 0x7f49e00cf290 2024-01-16T15:12:51.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 -- 172.21.15.53:0/2792397921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00cf290 msgr2=0x7f49e00cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 --2- 172.21.15.53:0/2792397921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00cf290 0x7f49e00cf720 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009e50 tx=0x7f49d802bff0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 -- 172.21.15.53:0/2792397921 shutdown_connections 2024-01-16T15:12:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 --2- 172.21.15.53:0/2792397921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00cf290 0x7f49e00cf720 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 --2- 172.21.15.53:0/2792397921 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00ae5a0 0x7f49e015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 -- 172.21.15.53:0/2792397921 >> 172.21.15.53:0/2792397921 conn(0x7f49e006a800 msgr2=0x7f49e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 -- 172.21.15.53:0/2792397921 shutdown_connections 2024-01-16T15:12:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.375+0000 7f49e5c00640 1 -- 172.21.15.53:0/2792397921 wait complete. 2024-01-16T15:12:51.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.377+0000 7f49e5c00640 1 Processor -- start 2024-01-16T15:12:51.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.377+0000 7f49e5c00640 1 -- start start 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.378+0000 7f49e5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 0x7f49e0159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.378+0000 7f49e5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00b3ae0 0x7f49e014f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.378+0000 7f49e5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49e00cf290 0x7f49e014f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49e5c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e01f3fe0 con 0x7f49e00ae5a0 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49e5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e014fed0 con 0x7f49e00cf290 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49e5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e01f6760 con 0x7f49e00b3ae0 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 0x7f49e0159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 0x7f49e0159890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49056/0 (socket says 172.21.15.53:49056) 2024-01-16T15:12:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 -- 172.21.15.53:0/3135038604 learned_addr learned my addr 172.21.15.53:0/3135038604 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49d7e00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00b3ae0 0x7f49e014f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49df400640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49e00cf290 0x7f49e014f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 -- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00b3ae0 msgr2=0x7f49e014f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00b3ae0 0x7f49e014f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 -- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49e00cf290 msgr2=0x7f49e014f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49e00cf290 0x7f49e014f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.379+0000 7f49dea00640 1 -- 172.21.15.53:0/3135038604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d8009b30 con 0x7f49e00ae5a0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.380+0000 7f49dea00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 0x7f49e0159890 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f49d000d800 tx=0x7f49d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.380+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000b6c0 con 0x7f49e00ae5a0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.380+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d000b820 con 0x7f49e00ae5a0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.380+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d000bbb0 con 0x7f49e00ae5a0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.380+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e01f3d00 con 0x7f49e00ae5a0 2024-01-16T15:12:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.380+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e01f6d00 con 0x7f49e00ae5a0 2024-01-16T15:12:51.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.381+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e01f6ec0 con 0x7f49e00ae5a0 2024-01-16T15:12:51.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.382+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f49d000bd10 con 0x7f49e00ae5a0 2024-01-16T15:12:51.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.383+0000 7f49d6a00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b806be90 0x7f49b806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.383+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f49d0088db0 con 0x7f49e00ae5a0 2024-01-16T15:12:51.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.384+0000 7f49d7e00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b806be90 0x7f49b806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.385+0000 7f49d7e00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b806be90 0x7f49b806e350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f49e01506e0 tx=0x7f49c8007b50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.389+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49d0055410 con 0x7f49e00ae5a0 2024-01-16T15:12:51.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.615+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54141", "target": ["mon-mgr", ""]}) v1 -- 0x7f49e0071170 con 0x7f49b806be90 2024-01-16T15:12:51.605 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:12:51.629+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:51.605 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:12:51.629+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:51.605 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:12:51.629+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:51.605 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:12:51.629+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:51.606 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:12:51.629+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:12:51.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.662+0000 7f49d6a00640 1 -- 172.21.15.53:0/3135038604 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f49c80076a0 con 0x7f49b806be90 2024-01-16T15:12:51.641 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:51.641 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:51.641 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:12:51.641 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:51.641 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.665+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b806be90 msgr2=0x7f49b806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.665+0000 7f49e5c00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b806be90 0x7f49b806e350 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f49e01506e0 tx=0x7f49c8007b50 comp rx=0 tx=0).stop 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.665+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 msgr2=0x7f49e0159890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.665+0000 7f49e5c00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 0x7f49e0159890 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f49d000d800 tx=0x7f49d000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 shutdown_connections 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b806be90 0x7f49b806e350 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49e00cf290 0x7f49e014f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49e00b3ae0 0x7f49e014f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 --2- 172.21.15.53:0/3135038604 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49e00ae5a0 0x7f49e0159890 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 >> 172.21.15.53:0/3135038604 conn(0x7f49e006a800 msgr2=0x7f49e0160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 shutdown_connections 2024-01-16T15:12:51.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.666+0000 7f49e5c00640 1 -- 172.21.15.53:0/3135038604 wait complete. 2024-01-16T15:12:51.663 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:51.663 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54142 2024-01-16T15:12:51.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.908+0000 7fbb9be00640 1 Processor -- start 2024-01-16T15:12:51.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.908+0000 7fbb9be00640 1 -- start start 2024-01-16T15:12:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb9be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 0x7fbb9c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb9be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c06c730 con 0x7fbb9c143810 2024-01-16T15:12:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb9be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb9c0724d0 con 0x7fbb9c15d770 2024-01-16T15:12:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb9be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c143bf0 con 0x7fbb9c071e40 2024-01-16T15:12:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb9aa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbb9c143810 0x7fbb9c15e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49476/0 (socket says 172.21.15.53:49476) 2024-01-16T15:12:51.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb9aa00640 1 -- 172.21.15.53:0/1198618074 learned_addr learned my addr 172.21.15.53:0/1198618074 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.910+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388871308 0 0) 0x7fbb9c06c730 con 0x7fbb9c143810 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb7c003520 con 0x7fbb9c143810 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb9a000640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 0x7fbb9c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3896876118 0 0) 0x7fbb7c003520 con 0x7fbb9c143810 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 msgr2=0x7fbb9c0ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 0x7fbb9c0ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 >> v1:172.21.15.146:6789/0 conn(0x7fbb9c15d770 legacy=0x7fbb9c15dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c06c9f0 con 0x7fbb9c143810 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fbb8c001670 con 0x7fbb9c143810 2024-01-16T15:12:51.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 >> v1:172.21.15.53:6789/0 conn(0x7fbb9c143810 legacy=0x7fbb9c15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 0x7fbb7c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb7c004020 0x7fbb7c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb7c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c143bf0 con 0x7fbb7c004e20 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c005640 con 0x7fbb7c004020 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb7c00bec0 con 0x7fbb9c071e40 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.911+0000 7fbb9b400640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb7c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9aa00640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 0x7fbb7c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9a000640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb7c004020 0x7fbb7c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9b400640 1 -- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 msgr2=0x7fbb7c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9b400640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071e40 0x7fbb7c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9b400640 1 -- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb7c004020 msgr2=0x7fbb7c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9b400640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb7c004020 0x7fbb7c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.912+0000 7fbb9b400640 1 -- 172.21.15.53:0/1198618074 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb7c003520 con 0x7fbb7c004e20 2024-01-16T15:12:51.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.913+0000 7fbb9b400640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb7c00b980 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fbb94001e50 tx=0x7fbb7c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f5d86354cbdab9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.913+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94003b70 con 0x7fbb7c004e20 2024-01-16T15:12:51.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.913+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb94003cd0 con 0x7fbb7c004e20 2024-01-16T15:12:51.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.913+0000 7fbb99600640 1 -- 172.21.15.53:0/1198618074 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb94005310 con 0x7fbb7c004e20 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9be00640 1 -- 172.21.15.53:0/1198618074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 msgr2=0x7fbb7c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1198618074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb7c00b980 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fbb94001e50 tx=0x7fbb7c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9b400640 1 -- 172.21.15.53:0/1198618074 reap_dead start 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9be00640 1 -- 172.21.15.53:0/1198618074 shutdown_connections 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9be00640 1 -- 172.21.15.53:0/1198618074 >> 172.21.15.53:0/1198618074 conn(0x7fbb9c06a7d0 msgr2=0x7fbb9c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9be00640 1 -- 172.21.15.53:0/1198618074 shutdown_connections 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.914+0000 7fbb9be00640 1 -- 172.21.15.53:0/1198618074 wait complete. 2024-01-16T15:12:51.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.916+0000 7fbb9be00640 1 Processor -- start 2024-01-16T15:12:51.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.916+0000 7fbb9be00640 1 -- start start 2024-01-16T15:12:51.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb9c132790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb9c071580 0x7fbb9c153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071a00 0x7fbb9c154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c132cd0 con 0x7fbb7c004e20 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c1f6010 con 0x7fbb9c071580 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9c1f62f0 con 0x7fbb9c071a00 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb9c132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb9c132790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57456/0 (socket says 172.21.15.53:57456) 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 -- 172.21.15.53:0/1719429162 learned_addr learned my addr 172.21.15.53:0/1719429162 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9a000640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb9c071580 0x7fbb9c153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 -- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071a00 msgr2=0x7fbb9c154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9b400640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071a00 0x7fbb9c154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071a00 0x7fbb9c154370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 -- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb9c071580 msgr2=0x7fbb9c153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.917+0000 7fbb9aa00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb9c071580 0x7fbb9c153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:51.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb9aa00640 1 -- 172.21.15.53:0/1719429162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb9c1f65d0 con 0x7fbb7c004e20 2024-01-16T15:12:51.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb9a000640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb9c071580 0x7fbb9c153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:51.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb9aa00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb9c132790 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e760 tx=0x7fbb8c00ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c00ee70 con 0x7fbb7c004e20 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb8c00c870 con 0x7fbb7c004e20 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb8c00cc40 con 0x7fbb7c004e20 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb9c1f68b0 con 0x7fbb7c004e20 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.918+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb9c1f5ca0 con 0x7fbb7c004e20 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.919+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb9c067450 con 0x7fbb7c004e20 2024-01-16T15:12:51.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.920+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7fbb8c01d460 con 0x7fbb7c004e20 2024-01-16T15:12:51.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.921+0000 7fbb92a00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb7406be90 0x7fbb7406e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:51.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.921+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fbb8c0887d0 con 0x7fbb7c004e20 2024-01-16T15:12:51.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.922+0000 7fbb9a000640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb7406be90 0x7fbb7406e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:51.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.923+0000 7fbb9a000640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb7406be90 0x7fbb7406e350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0b4080 tx=0x7fbb84008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:51.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:51.925+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb8c054e30 con 0x7fbb7c004e20 2024-01-16T15:12:52.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.157+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54142", "target": ["mon-mgr", ""]}) v1 -- 0x7fbb9c06cfb0 con 0x7fbb7406be90 2024-01-16T15:12:52.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.180+0000 7fbb92a00640 1 -- 172.21.15.53:0/1719429162 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7fbb8400b670 con 0x7fbb7406be90 2024-01-16T15:12:52.158 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:52.158 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:52.158 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:12:52.158 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:52.158 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb7406be90 msgr2=0x7fbb7406e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb7406be90 0x7fbb7406e350 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c0b4080 tx=0x7fbb84008040 comp rx=0 tx=0).stop 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 msgr2=0x7fbb9c132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb9c132790 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c00e760 tx=0x7fbb8c00ec20 comp rx=0 tx=0).stop 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 shutdown_connections 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb7406be90 0x7fbb7406e350 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb9c071a00 0x7fbb9c154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb9c071580 0x7fbb9c153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 --2- 172.21.15.53:0/1719429162 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb7c004e20 0x7fbb9c132790 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 >> 172.21.15.53:0/1719429162 conn(0x7fbb9c06a7d0 msgr2=0x7fbb9c15fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 shutdown_connections 2024-01-16T15:12:52.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.183+0000 7fbb9be00640 1 -- 172.21.15.53:0/1719429162 wait complete. 2024-01-16T15:12:52.180 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:52.180 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54143 2024-01-16T15:12:52.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.428+0000 7f1a15a00640 1 Processor -- start 2024-01-16T15:12:52.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.428+0000 7f1a15a00640 1 -- start start 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.429+0000 7f1a15a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a100cf9a0 0x7f1a1015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a1015e1e0 0x7f1a100702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a15a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a100b1ee0 con 0x7f1a10143ae0 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a15a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a100b2780 con 0x7f1a100cf9a0 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a15a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1015e990 con 0x7f1a1015e1e0 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a0ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1a10143ae0 0x7f1a100cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49480/0 (socket says 172.21.15.53:49480) 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a0ea00640 1 -- 172.21.15.53:0/113394200 learned_addr learned my addr 172.21.15.53:0/113394200 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:52.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a07e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1a100cf9a0 0x7f1a1015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a0f400640 1 --2- 172.21.15.53:0/113394200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a1015e1e0 0x7f1a100702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a0e000640 1 -- 172.21.15.53:0/113394200 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186559439 0 0) 0x7f1a100b1ee0 con 0x7f1a10143ae0 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a07e00640 1 -- 172.21.15.53:0/113394200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a1015e1e0 msgr2=0x7f1a100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a07e00640 1 --2- 172.21.15.53:0/113394200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a1015e1e0 0x7f1a100702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.430+0000 7f1a07e00640 1 -- 172.21.15.53:0/113394200 >> v1:172.21.15.53:6789/0 conn(0x7f1a10143ae0 legacy=0x7f1a100cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.431+0000 7f1a07e00640 1 -- 172.21.15.53:0/113394200 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a100b2130 con 0x7f1a100cf9a0 2024-01-16T15:12:52.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.432+0000 7f1a07e00640 1 --2- 172.21.15.53:0/113394200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015dca0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f19f80097a0 tx=0x7f19f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24810d2517086470 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.432+0000 7f1a0e000640 1 -- 172.21.15.53:0/113394200 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8031830 con 0x7f1a100cf9a0 2024-01-16T15:12:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.432+0000 7f1a0e000640 1 -- 172.21.15.53:0/113394200 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f8031990 con 0x7f1a100cf9a0 2024-01-16T15:12:52.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.432+0000 7f1a0e000640 1 -- 172.21.15.53:0/113394200 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f8031d20 con 0x7f1a100cf9a0 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 -- 172.21.15.53:0/113394200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 msgr2=0x7f1a1015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 --2- 172.21.15.53:0/113394200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015dca0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f19f80097a0 tx=0x7f19f802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 -- 172.21.15.53:0/113394200 shutdown_connections 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 --2- 172.21.15.53:0/113394200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a1015e1e0 0x7f1a100702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 --2- 172.21.15.53:0/113394200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015dca0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 -- 172.21.15.53:0/113394200 >> 172.21.15.53:0/113394200 conn(0x7f1a1006a800 msgr2=0x7f1a1006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 -- 172.21.15.53:0/113394200 shutdown_connections 2024-01-16T15:12:52.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.433+0000 7f1a15a00640 1 -- 172.21.15.53:0/113394200 wait complete. 2024-01-16T15:12:52.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.435+0000 7f1a15a00640 1 Processor -- start 2024-01-16T15:12:52.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.435+0000 7f1a15a00640 1 -- start start 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a10143ae0 0x7f1a1013e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a15a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a1015e1e0 0x7f1a101411f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a15a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100ae360 con 0x7f1a100cf9a0 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a15a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100ad7e0 con 0x7f1a1015e1e0 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a15a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a100adac0 con 0x7f1a10143ae0 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a0ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a0ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57470/0 (socket says 172.21.15.53:57470) 2024-01-16T15:12:52.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a0ea00640 1 -- 172.21.15.53:0/3377785764 learned_addr learned my addr 172.21.15.53:0/3377785764 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a07e00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a10143ae0 0x7f1a1013e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.436+0000 7f1a0f400640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a1015e1e0 0x7f1a101411f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0ea00640 1 -- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a10143ae0 msgr2=0x7f1a1013e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0ea00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a10143ae0 0x7f1a1013e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0ea00640 1 -- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a1015e1e0 msgr2=0x7f1a101411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0ea00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a1015e1e0 0x7f1a101411f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0ea00640 1 -- 172.21.15.53:0/3377785764 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f8009400 con 0x7f1a100cf9a0 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0f400640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a1015e1e0 0x7f1a101411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a0ea00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015bf50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000c4d0 tx=0x7f1a0000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a000074e0 con 0x7f1a100cf9a0 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a00007660 con 0x7f1a100cf9a0 2024-01-16T15:12:52.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a100adda0 con 0x7f1a100cf9a0 2024-01-16T15:12:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a00004a20 con 0x7f1a100cf9a0 2024-01-16T15:12:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.437+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a10141f00 con 0x7f1a100cf9a0 2024-01-16T15:12:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.438+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a100675d0 con 0x7f1a100cf9a0 2024-01-16T15:12:52.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.439+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f1a00004b80 con 0x7f1a100cf9a0 2024-01-16T15:12:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.440+0000 7f1a06a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19e806be90 0x7f19e806e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.441+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1a00087da0 con 0x7f1a100cf9a0 2024-01-16T15:12:52.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.441+0000 7f1a07e00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19e806be90 0x7f19e806e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.442+0000 7f1a07e00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19e806be90 0x7f19e806e350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f19f8036000 tx=0x7f19f8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.445+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a00054400 con 0x7f1a100cf9a0 2024-01-16T15:12:52.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.675+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54143", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a1015e670 con 0x7f19e806be90 2024-01-16T15:12:52.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.699+0000 7f1a06a00640 1 -- 172.21.15.53:0/3377785764 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f19f8031b80 con 0x7f19e806be90 2024-01-16T15:12:52.677 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:52.677 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:52.677 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:12:52.677 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:52.677 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19e806be90 msgr2=0x7f19e806e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19e806be90 0x7f19e806e350 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f19f8036000 tx=0x7f19f8008030 comp rx=0 tx=0).stop 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 msgr2=0x7f1a1015bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015bf50 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000c4d0 tx=0x7f1a0000c990 comp rx=0 tx=0).stop 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 shutdown_connections 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19e806be90 0x7f19e806e350 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a1015e1e0 0x7f1a101411f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a10143ae0 0x7f1a1013e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.701+0000 7f1a15a00640 1 --2- 172.21.15.53:0/3377785764 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a100cf9a0 0x7f1a1015bf50 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.702+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 >> 172.21.15.53:0/3377785764 conn(0x7f1a1006a800 msgr2=0x7f1a1006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.702+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 shutdown_connections 2024-01-16T15:12:52.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.702+0000 7f1a15a00640 1 -- 172.21.15.53:0/3377785764 wait complete. 2024-01-16T15:12:52.698 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:52.699 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54144 2024-01-16T15:12:52.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.949+0000 7f3032a00640 1 Processor -- start 2024-01-16T15:12:52.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.949+0000 7f3032a00640 1 -- start start 2024-01-16T15:12:52.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f3032a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f3032a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c06c470 con 0x7f302c13e360 2024-01-16T15:12:52.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f3032a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302c06c750 con 0x7f302c0cf9a0 2024-01-16T15:12:52.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f3032a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302c06ca30 con 0x7f302c13df80 2024-01-16T15:12:52.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f302be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f302be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57472/0 (socket says 172.21.15.53:57472) 2024-01-16T15:12:52.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.951+0000 7f302be00640 1 -- 172.21.15.53:0/1665264626 learned_addr learned my addr 172.21.15.53:0/1665264626 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:52.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.952+0000 7f302be00640 1 -- 172.21.15.53:0/1665264626 >> v1:172.21.15.146:6789/0 conn(0x7f302c0cf9a0 legacy=0x7f302c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:52.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.952+0000 7f302be00640 1 -- 172.21.15.53:0/1665264626 >> v1:172.21.15.53:6789/0 conn(0x7f302c13e360 legacy=0x7f302c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.952+0000 7f302be00640 1 -- 172.21.15.53:0/1665264626 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c06cd10 con 0x7f302c13df80 2024-01-16T15:12:52.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.952+0000 7f302aa00640 1 -- 172.21.15.53:0/1665264626 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321538323 0 0) 0x7f302c06c470 con 0x7f302c13e360 2024-01-16T15:12:52.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.953+0000 7f302be00640 1 --2- 172.21.15.53:0/1665264626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c13df80 0x7f302c15bee0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f301c009870 tx=0x7f301c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b162f9aceda40a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.953+0000 7f302aa00640 1 -- 172.21.15.53:0/1665264626 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c030850 con 0x7f302c13df80 2024-01-16T15:12:52.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.953+0000 7f302aa00640 1 -- 172.21.15.53:0/1665264626 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c0309b0 con 0x7f302c13df80 2024-01-16T15:12:52.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.953+0000 7f302aa00640 1 -- 172.21.15.53:0/1665264626 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c030d40 con 0x7f302c13df80 2024-01-16T15:12:52.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 -- 172.21.15.53:0/1665264626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c13df80 msgr2=0x7f302c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 --2- 172.21.15.53:0/1665264626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c13df80 0x7f302c15bee0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f301c009870 tx=0x7f301c02b990 comp rx=0 tx=0).stop 2024-01-16T15:12:52.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 -- 172.21.15.53:0/1665264626 shutdown_connections 2024-01-16T15:12:52.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 --2- 172.21.15.53:0/1665264626 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c13df80 0x7f302c15bee0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 -- 172.21.15.53:0/1665264626 >> 172.21.15.53:0/1665264626 conn(0x7f302c06a800 msgr2=0x7f302c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:52.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 -- 172.21.15.53:0/1665264626 shutdown_connections 2024-01-16T15:12:52.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.954+0000 7f3032a00640 1 -- 172.21.15.53:0/1665264626 wait complete. 2024-01-16T15:12:52.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.956+0000 7f3032a00640 1 Processor -- start 2024-01-16T15:12:52.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.956+0000 7f3032a00640 1 -- start start 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.957+0000 7f3032a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c0a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.957+0000 7f3032a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.957+0000 7f3032a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c0a8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.957+0000 7f3032a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0702c0 con 0x7f302c0cf9a0 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.957+0000 7f3032a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302c070510 con 0x7f302c13df80 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.957+0000 7f3032a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302c0707f0 con 0x7f302c13e360 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c0a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c0a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57480/0 (socket says 172.21.15.53:57480) 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 -- 172.21.15.53:0/3774225851 learned_addr learned my addr 172.21.15.53:0/3774225851 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:52.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302b400640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 -- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 msgr2=0x7f302c0a8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c0a8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 -- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 msgr2=0x7f302c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.958+0000 7f302be00640 1 -- 172.21.15.53:0/3774225851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f301c0094d0 con 0x7f302c0cf9a0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.959+0000 7f302be00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c0a84c0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f301c009870 tx=0x7f301c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.959+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c039d60 con 0x7f302c0cf9a0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.959+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f301c043400 con 0x7f302c0cf9a0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.959+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f301c0437b0 con 0x7f302c0cf9a0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.959+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f302c070ad0 con 0x7f302c0cf9a0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.959+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f302c0a8a00 con 0x7f302c0cf9a0 2024-01-16T15:12:52.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.960+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302c0675d0 con 0x7f302c0cf9a0 2024-01-16T15:12:52.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.961+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 27) v1 ==== 90610+0+0 (secure 0 0 0) 0x7f301c02e070 con 0x7f302c0cf9a0 2024-01-16T15:12:52.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.962+0000 7f3023e00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406bf10 0x7f300406e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:52.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.962+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f301c03e070 con 0x7f302c0cf9a0 2024-01-16T15:12:52.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.963+0000 7f302b400640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406bf10 0x7f300406e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:52.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.964+0000 7f302b400640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406bf10 0x7f300406e3d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f302c0b3760 tx=0x7f3014009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:52.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:52.966+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f301c077dc0 con 0x7f302c0cf9a0 2024-01-16T15:12:53.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.194+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7f302c070680 con 0x7f300406bf10 2024-01-16T15:12:53.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.217+0000 7f3023e00640 1 -- 172.21.15.53:0/3774225851 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+46 (secure 0 0 0) 0x7f302c070680 con 0x7f300406bf10 2024-01-16T15:12:53.194 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:53.194 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:53.194 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "complete" 2024-01-16T15:12:53.194 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:53.194 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406bf10 msgr2=0x7f300406e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406bf10 0x7f300406e3d0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f302c0b3760 tx=0x7f3014009040 comp rx=0 tx=0).stop 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 msgr2=0x7f302c0a84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c0a84c0 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f301c009870 tx=0x7f301c004730 comp rx=0 tx=0).stop 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 shutdown_connections 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f300406bf10 0x7f300406e3d0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f302c13e360 0x7f302c0a8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f302c13df80 0x7f302c15a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 --2- 172.21.15.53:0/3774225851 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f302c0cf9a0 0x7f302c0a84c0 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 >> 172.21.15.53:0/3774225851 conn(0x7f302c06a800 msgr2=0x7f302c153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.218+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 shutdown_connections 2024-01-16T15:12:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.219+0000 7f3032a00640 1 -- 172.21.15.53:0/3774225851 wait complete. 2024-01-16T15:12:53.216 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:53.216 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54145 2024-01-16T15:12:53.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.464+0000 7f3eefe00640 1 Processor -- start 2024-01-16T15:12:53.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.464+0000 7f3eefe00640 1 -- start start 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.465+0000 7f3eefe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ee80cd960 0x7f3ee80cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.465+0000 7f3eefe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ee8162f30 0x7f3ee8165320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.465+0000 7f3eefe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ee80ce2a0 con 0x7f3ee815b2a0 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.465+0000 7f3eefe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee80d00a0 con 0x7f3ee8162f30 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.465+0000 7f3eefe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ee80b4010 con 0x7f3ee80cd960 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.466+0000 7f3eed400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3ee815b2a0 0x7f3ee80cf5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49496/0 (socket says 172.21.15.53:49496) 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.466+0000 7f3eed400640 1 -- 172.21.15.53:0/2915528047 learned_addr learned my addr 172.21.15.53:0/2915528047 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.466+0000 7f3eede00640 1 --2- 172.21.15.53:0/2915528047 >> v2:172.21.15.53:3300/0 conn(0x7f3ee8162f30 0x7f3ee8165320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.466+0000 7f3ee7e00640 1 -- 172.21.15.53:0/2915528047 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494856549 0 0) 0x7f3ee80ce2a0 con 0x7f3ee815b2a0 2024-01-16T15:12:53.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.466+0000 7f3ee7e00640 1 -- 172.21.15.53:0/2915528047 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec8003520 con 0x7f3ee815b2a0 2024-01-16T15:12:53.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.466+0000 7f3eeca00640 1 --2- 172.21.15.53:0/2915528047 >> v2:172.21.15.146:3300/0 conn(0x7f3ee80cd960 0x7f3ee80cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.467+0000 7f3eede00640 1 -- 172.21.15.53:0/2915528047 >> v2:172.21.15.146:3300/0 conn(0x7f3ee80cd960 msgr2=0x7f3ee80cdd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:53.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.467+0000 7f3eede00640 1 --2- 172.21.15.53:0/2915528047 >> v2:172.21.15.146:3300/0 conn(0x7f3ee80cd960 0x7f3ee80cdd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.467+0000 7f3eede00640 1 -- 172.21.15.53:0/2915528047 >> v1:172.21.15.53:6789/0 conn(0x7f3ee815b2a0 legacy=0x7f3ee80cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.467+0000 7f3ee7e00640 1 -- 172.21.15.53:0/2915528047 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3438400169 0 0) 0x7f3ec8003520 con 0x7f3ee815b2a0 2024-01-16T15:12:53.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.467+0000 7f3eede00640 1 -- 172.21.15.53:0/2915528047 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee80b42f0 con 0x7f3ee8162f30 2024-01-16T15:12:53.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.468+0000 7f3eede00640 1 --2- 172.21.15.53:0/2915528047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee8162f30 0x7f3ee8165320 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80b2ae0 tx=0x7f3ee002b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b36c079a2520c5bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.468+0000 7f3ee7e00640 1 -- 172.21.15.53:0/2915528047 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8001670 con 0x7f3ee8162f30 2024-01-16T15:12:53.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.468+0000 7f3ee7e00640 1 -- 172.21.15.53:0/2915528047 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee0033440 con 0x7f3ee8162f30 2024-01-16T15:12:53.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.468+0000 7f3ee7e00640 1 -- 172.21.15.53:0/2915528047 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00337d0 con 0x7f3ee8162f30 2024-01-16T15:12:53.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 -- 172.21.15.53:0/2915528047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee8162f30 msgr2=0x7f3ee8165320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2915528047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee8162f30 0x7f3ee8165320 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f3ee80b2ae0 tx=0x7f3ee002b7a0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 -- 172.21.15.53:0/2915528047 shutdown_connections 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2915528047 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee8162f30 0x7f3ee8165320 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2915528047 >> v2:172.21.15.146:3300/0 conn(0x7f3ee80cd960 0x7f3ee80cdd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 -- 172.21.15.53:0/2915528047 >> 172.21.15.53:0/2915528047 conn(0x7f3ee80d1020 msgr2=0x7f3ee80d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 -- 172.21.15.53:0/2915528047 shutdown_connections 2024-01-16T15:12:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.469+0000 7f3eefe00640 1 -- 172.21.15.53:0/2915528047 wait complete. 2024-01-16T15:12:53.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.470+0000 7f3eefe00640 1 Processor -- start 2024-01-16T15:12:53.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.471+0000 7f3eefe00640 1 -- start start 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.471+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 0x7f3ee8155570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ee815b2a0 0x7f3ee8159d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eefe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ee8162f30 0x7f3ee815a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eefe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee815d480 con 0x7f3ee80cd960 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eefe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee81f18b0 con 0x7f3ee815b2a0 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eefe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee815c800 con 0x7f3ee8162f30 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 0x7f3ee8155570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 0x7f3ee8155570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57502/0 (socket says 172.21.15.53:57502) 2024-01-16T15:12:53.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 -- 172.21.15.53:0/2121737792 learned_addr learned my addr 172.21.15.53:0/2121737792 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eeca00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ee815b2a0 0x7f3ee8159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 -- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ee8162f30 msgr2=0x7f3ee815a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ee8162f30 0x7f3ee815a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 -- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ee815b2a0 msgr2=0x7f3ee8159d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ee815b2a0 0x7f3ee8159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.472+0000 7f3eed400640 1 -- 172.21.15.53:0/2121737792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ee0009400 con 0x7f3ee80cd960 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.473+0000 7f3eed400640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 0x7f3ee8155570 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c760 tx=0x7f3ed800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.473+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8005ad0 con 0x7f3ee80cd960 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.473+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed8005c30 con 0x7f3ee80cd960 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.473+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee815cae0 con 0x7f3ee80cd960 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.473+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed8007070 con 0x7f3ee80cd960 2024-01-16T15:12:53.452 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.473+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee815d0a0 con 0x7f3ee80cd960 2024-01-16T15:12:53.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.474+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee815d260 con 0x7f3ee80cd960 2024-01-16T15:12:53.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.476+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f3ed80071d0 con 0x7f3ee80cd960 2024-01-16T15:12:53.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.477+0000 7f3ee5600640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ec006bf20 0x7f3ec006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.477+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ed8088430 con 0x7f3ee80cd960 2024-01-16T15:12:53.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.477+0000 7f3eeca00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ec006bf20 0x7f3ec006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.479+0000 7f3eeca00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ec006bf20 0x7f3ec006e3e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3ee809eaf0 tx=0x7f3ed000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.482+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ed80bde90 con 0x7f3ee80cd960 2024-01-16T15:12:53.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.715+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54145", "target": ["mon-mgr", ""]}) v1 -- 0x7f3ee815e8c0 con 0x7f3ec006bf20 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.739+0000 7f3ee5600640 1 -- 172.21.15.53:0/2121737792 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3ee815e8c0 con 0x7f3ec006bf20 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:53.717 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:53.718 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:53.718 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:53.718 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:53.718 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:53.718 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:53.718 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:53.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.741+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ec006bf20 msgr2=0x7f3ec006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.741+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ec006bf20 0x7f3ec006e3e0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f3ee809eaf0 tx=0x7f3ed000b040 comp rx=0 tx=0).stop 2024-01-16T15:12:53.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.741+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 msgr2=0x7f3ee8155570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.741+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 0x7f3ee8155570 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f3ed800c760 tx=0x7f3ed800cc20 comp rx=0 tx=0).stop 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 shutdown_connections 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ec006bf20 0x7f3ec006e3e0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ee8162f30 0x7f3ee815a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ee815b2a0 0x7f3ee8159d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 --2- 172.21.15.53:0/2121737792 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ee80cd960 0x7f3ee8155570 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 >> 172.21.15.53:0/2121737792 conn(0x7f3ee80d1020 msgr2=0x7f3ee8159350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 shutdown_connections 2024-01-16T15:12:53.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.742+0000 7f3eefe00640 1 -- 172.21.15.53:0/2121737792 wait complete. 2024-01-16T15:12:53.738 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:53.738 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54146 2024-01-16T15:12:53.829 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:53.834 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:12:53.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.978+0000 7f8cfbe00640 1 Processor -- start 2024-01-16T15:12:53.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.978+0000 7f8cfbe00640 1 -- start start 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.980+0000 7f8cfbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8cfc071290 0x7f8cfc071670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.980+0000 7f8cfbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8cfc15f010 0x7f8cfc0b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.980+0000 7f8cfbe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cfc071bb0 con 0x7f8cfc15eb60 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.980+0000 7f8cfbe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cfc0ccb10 con 0x7f8cfc15f010 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.980+0000 7f8cfbe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cfc15f620 con 0x7f8cfc071290 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8cfc15f010 0x7f8cfc0b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cf3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8cfc15eb60 0x7f8cfc0cd520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49510/0 (socket says 172.21.15.53:49510) 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cf3e00640 1 -- 172.21.15.53:0/3198418778 learned_addr learned my addr 172.21.15.53:0/3198418778 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:53.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfaa00640 1 --2- 172.21.15.53:0/3198418778 >> v2:172.21.15.146:3300/0 conn(0x7f8cfc071290 0x7f8cfc071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfb400640 1 -- 172.21.15.53:0/3198418778 >> v2:172.21.15.146:3300/0 conn(0x7f8cfc071290 msgr2=0x7f8cfc071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfb400640 1 --2- 172.21.15.53:0/3198418778 >> v2:172.21.15.146:3300/0 conn(0x7f8cfc071290 0x7f8cfc071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfb400640 1 -- 172.21.15.53:0/3198418778 >> v1:172.21.15.53:6789/0 conn(0x7f8cfc15eb60 legacy=0x7f8cfc0cd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfb400640 1 -- 172.21.15.53:0/3198418778 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc15f900 con 0x7f8cfc15f010 2024-01-16T15:12:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfa000640 1 -- 172.21.15.53:0/3198418778 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397295987 0 0) 0x7f8cfc071bb0 con 0x7f8cfc15eb60 2024-01-16T15:12:53.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.981+0000 7f8cfaa00640 1 --2- 172.21.15.53:0/3198418778 >> v2:172.21.15.146:3300/0 conn(0x7f8cfc071290 0x7f8cfc071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.982+0000 7f8cfb400640 1 --2- 172.21.15.53:0/3198418778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc0b2ac0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc06e3a0 tx=0x7f8cf402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79266bebae8ddd3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfa000640 1 -- 172.21.15.53:0/3198418778 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf402cbf0 con 0x7f8cfc15f010 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfa000640 1 -- 172.21.15.53:0/3198418778 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf402cd50 con 0x7f8cfc15f010 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfa000640 1 -- 172.21.15.53:0/3198418778 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf4034660 con 0x7f8cfc15f010 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfbe00640 1 -- 172.21.15.53:0/3198418778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 msgr2=0x7f8cfc0b2ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/3198418778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc0b2ac0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f8cfc06e3a0 tx=0x7f8cf402be40 comp rx=0 tx=0).stop 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfbe00640 1 -- 172.21.15.53:0/3198418778 shutdown_connections 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/3198418778 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc0b2ac0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/3198418778 >> v2:172.21.15.146:3300/0 conn(0x7f8cfc071290 0x7f8cfc071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.983+0000 7f8cfbe00640 1 -- 172.21.15.53:0/3198418778 >> 172.21.15.53:0/3198418778 conn(0x7f8cfc0b0ed0 msgr2=0x7f8cfc0cec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.984+0000 7f8cfbe00640 1 -- 172.21.15.53:0/3198418778 shutdown_connections 2024-01-16T15:12:53.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.984+0000 7f8cfbe00640 1 -- 172.21.15.53:0/3198418778 wait complete. 2024-01-16T15:12:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.985+0000 7f8cfbe00640 1 Processor -- start 2024-01-16T15:12:53.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.985+0000 7f8cfbe00640 1 -- start start 2024-01-16T15:12:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.986+0000 7f8cfbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cfc071290 0x7f8cfc0ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.986+0000 7f8cfbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cfc15eb60 0x7f8cfc1421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.986+0000 7f8cfbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc15cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.986+0000 7f8cfbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0aba30 con 0x7f8cfc15f010 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.986+0000 7f8cfbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0a8560 con 0x7f8cfc071290 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.986+0000 7f8cfbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cfc0a8840 con 0x7f8cfc15eb60 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc15cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc15cdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57514/0 (socket says 172.21.15.53:57514) 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 -- 172.21.15.53:0/2048003431 learned_addr learned my addr 172.21.15.53:0/2048003431 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfaa00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cfc071290 0x7f8cfc0ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cf3e00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cfc15eb60 0x7f8cfc1421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 -- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cfc15eb60 msgr2=0x7f8cfc1421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cfc15eb60 0x7f8cfc1421f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 -- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cfc071290 msgr2=0x7f8cfc0ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cfc071290 0x7f8cfc0ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:53.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.987+0000 7f8cfb400640 1 -- 172.21.15.53:0/2048003431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf4009a70 con 0x7f8cfc15f010 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.988+0000 7f8cfb400640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc15cdd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f8cf402c450 tx=0x7f8cf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.988+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf403e610 con 0x7f8cfc15f010 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.988+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf403e770 con 0x7f8cfc15f010 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.988+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cf403eb20 con 0x7f8cfc15f010 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.988+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cfc0a8b20 con 0x7f8cfc15f010 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.988+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cfc15d850 con 0x7f8cfc15f010 2024-01-16T15:12:53.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.989+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cfc0a9040 con 0x7f8cfc15f010 2024-01-16T15:12:53.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.991+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8cf403b050 con 0x7f8cfc15f010 2024-01-16T15:12:53.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.991+0000 7f8cf2a00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cd406bf20 0x7f8cd406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:53.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.992+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8cf40aafe0 con 0x7f8cfc15f010 2024-01-16T15:12:53.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.992+0000 7f8cfaa00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cd406bf20 0x7f8cd406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:53.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.993+0000 7f8cfaa00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cd406bf20 0x7f8cd406e3e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8cec006fd0 tx=0x7f8cec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:53.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:53.996+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8cfc0a9040 con 0x7f8cfc15f010 2024-01-16T15:12:54.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.229+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54146", "target": ["mon-mgr", ""]}) v1 -- 0x7f8cfc0cd000 con 0x7f8cd406bf20 2024-01-16T15:12:54.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.254+0000 7f8cf2a00640 1 -- 172.21.15.53:0/2048003431 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f8cec00c590 con 0x7f8cd406bf20 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:54.232 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:54.233 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cd406bf20 msgr2=0x7f8cd406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cd406bf20 0x7f8cd406e3e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8cec006fd0 tx=0x7f8cec008040 comp rx=0 tx=0).stop 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 msgr2=0x7f8cfc15cdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc15cdd0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f8cf402c450 tx=0x7f8cf4004a40 comp rx=0 tx=0).stop 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 shutdown_connections 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cd406bf20 0x7f8cd406e3e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cfc15f010 0x7f8cfc15cdd0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cfc15eb60 0x7f8cfc1421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 --2- 172.21.15.53:0/2048003431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cfc071290 0x7f8cfc0ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 >> 172.21.15.53:0/2048003431 conn(0x7f8cfc0b0ed0 msgr2=0x7f8cfc0ae7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 shutdown_connections 2024-01-16T15:12:54.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.256+0000 7f8cfbe00640 1 -- 172.21.15.53:0/2048003431 wait complete. 2024-01-16T15:12:54.252 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:54.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54147 2024-01-16T15:12:54.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.546+0000 7fba98600640 1 Processor -- start 2024-01-16T15:12:54.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.546+0000 7fba98600640 1 -- start start 2024-01-16T15:12:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba98600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 0x7fba9015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba98600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900cf290 0x7fba900cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba98600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba90070c40 con 0x7fba900ae5a0 2024-01-16T15:12:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba98600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba90071450 con 0x7fba900b3ae0 2024-01-16T15:12:54.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba98600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba900cfc60 con 0x7fba900cf290 2024-01-16T15:12:54.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba95200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 0x7fba9015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba95200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 0x7fba9015e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:40110/0 (socket says 172.21.15.53:40110) 2024-01-16T15:12:54.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba95200640 1 -- 172.21.15.53:0/989720907 learned_addr learned my addr 172.21.15.53:0/989720907 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:54.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.548+0000 7fba96600640 1 --2- 172.21.15.53:0/989720907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900cf290 0x7fba900cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.549+0000 7fba96600640 1 -- 172.21.15.53:0/989720907 >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 msgr2=0x7fba9015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:54.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.549+0000 7fba96600640 1 --2- 172.21.15.53:0/989720907 >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 0x7fba9015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.549+0000 7fba96600640 1 -- 172.21.15.53:0/989720907 >> v1:172.21.15.146:6789/0 conn(0x7fba900ae5a0 legacy=0x7fba9006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.549+0000 7fba96600640 1 -- 172.21.15.53:0/989720907 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba9006c470 con 0x7fba900cf290 2024-01-16T15:12:54.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.549+0000 7fba95200640 1 --2- 172.21.15.53:0/989720907 >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 0x7fba9015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:54.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.550+0000 7fba96600640 1 --2- 172.21.15.53:0/989720907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900cf290 0x7fba900cf720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fba88009d90 tx=0x7fba8802bf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76b3fdbc6b080de server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba8fe00640 1 -- 172.21.15.53:0/989720907 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8802cce0 con 0x7fba900cf290 2024-01-16T15:12:54.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba8fe00640 1 -- 172.21.15.53:0/989720907 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8802ce40 con 0x7fba900cf290 2024-01-16T15:12:54.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba8fe00640 1 -- 172.21.15.53:0/989720907 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba88034660 con 0x7fba900cf290 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 -- 172.21.15.53:0/989720907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900cf290 msgr2=0x7fba900cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 --2- 172.21.15.53:0/989720907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900cf290 0x7fba900cf720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fba88009d90 tx=0x7fba8802bf30 comp rx=0 tx=0).stop 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 -- 172.21.15.53:0/989720907 shutdown_connections 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 --2- 172.21.15.53:0/989720907 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900cf290 0x7fba900cf720 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 --2- 172.21.15.53:0/989720907 >> v2:172.21.15.146:3300/0 conn(0x7fba900b3ae0 0x7fba9015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 -- 172.21.15.53:0/989720907 >> 172.21.15.53:0/989720907 conn(0x7fba9006a800 msgr2=0x7fba9006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:54.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 -- 172.21.15.53:0/989720907 shutdown_connections 2024-01-16T15:12:54.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.552+0000 7fba98600640 1 -- 172.21.15.53:0/989720907 wait complete. 2024-01-16T15:12:54.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.554+0000 7fba98600640 1 Processor -- start 2024-01-16T15:12:54.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.554+0000 7fba98600640 1 -- start start 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.554+0000 7fba98600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba900ae5a0 0x7fba901499b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.554+0000 7fba98600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900b3ae0 0x7fba9014f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba98600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 0x7fba9014f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba98600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba90156fd0 con 0x7fba900cf290 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba98600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba901572b0 con 0x7fba900ae5a0 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba98600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba90157590 con 0x7fba900b3ae0 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 0x7fba9014f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 0x7fba9014f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57524/0 (socket says 172.21.15.53:57524) 2024-01-16T15:12:54.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 -- 172.21.15.53:0/2803936913 learned_addr learned my addr 172.21.15.53:0/2803936913 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba95c00640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba900ae5a0 0x7fba901499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba95200640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900b3ae0 0x7fba9014f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 -- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900b3ae0 msgr2=0x7fba9014f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900b3ae0 0x7fba9014f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 -- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba900ae5a0 msgr2=0x7fba901499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba900ae5a0 0x7fba901499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba96600640 1 -- 172.21.15.53:0/2803936913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba780094d0 con 0x7fba900cf290 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba95200640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900b3ae0 0x7fba9014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.555+0000 7fba95c00640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba900ae5a0 0x7fba901499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.556+0000 7fba96600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 0x7fba9014f7e0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fba88009d90 tx=0x7fba88004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.556+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8803d600 con 0x7fba900cf290 2024-01-16T15:12:54.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.556+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba8803d780 con 0x7fba900cf290 2024-01-16T15:12:54.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.556+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba8803db30 con 0x7fba900cf290 2024-01-16T15:12:54.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.556+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba88009a70 con 0x7fba900cf290 2024-01-16T15:12:54.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.556+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba90157b50 con 0x7fba900cf290 2024-01-16T15:12:54.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.557+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba900675d0 con 0x7fba900cf290 2024-01-16T15:12:54.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.560+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fba8803b040 con 0x7fba900cf290 2024-01-16T15:12:54.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.560+0000 7fba8d600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6806bf20 0x7fba6806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:54.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.560+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fba880aad80 con 0x7fba900cf290 2024-01-16T15:12:54.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.560+0000 7fba95c00640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6806bf20 0x7fba6806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:54.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.561+0000 7fba95c00640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6806bf20 0x7fba6806e3e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba9014f060 tx=0x7fba80009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:54.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.564+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba880774e0 con 0x7fba900cf290 2024-01-16T15:12:54.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.796+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54147", "target": ["mon-mgr", ""]}) v1 -- 0x7fba90071170 con 0x7fba6806bf20 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.820+0000 7fba8d600640 1 -- 172.21.15.53:0/2803936913 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fba800013e0 con 0x7fba6806bf20 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:54.797 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:54.798 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.821+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6806bf20 msgr2=0x7fba6806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.821+0000 7fba98600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6806bf20 0x7fba6806e3e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fba9014f060 tx=0x7fba80009040 comp rx=0 tx=0).stop 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.821+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 msgr2=0x7fba9014f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.821+0000 7fba98600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 0x7fba9014f7e0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7fba88009d90 tx=0x7fba88004a40 comp rx=0 tx=0).stop 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.821+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 shutdown_connections 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.821+0000 7fba98600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba6806bf20 0x7fba6806e3e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.822+0000 7fba98600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba900cf290 0x7fba9014f7e0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.822+0000 7fba98600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba900b3ae0 0x7fba9014f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.822+0000 7fba98600640 1 --2- 172.21.15.53:0/2803936913 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba900ae5a0 0x7fba901499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:54.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.822+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 >> 172.21.15.53:0/2803936913 conn(0x7fba9006a800 msgr2=0x7fba900b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:54.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.822+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 shutdown_connections 2024-01-16T15:12:54.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:54.822+0000 7fba98600640 1 -- 172.21.15.53:0/2803936913 wait complete. 2024-01-16T15:12:54.820 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:54.820 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54148 2024-01-16T15:12:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.069+0000 7f9988200640 1 Processor -- start 2024-01-16T15:12:55.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.070+0000 7f9988200640 1 -- start start 2024-01-16T15:12:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.071+0000 7f9988200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f99800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.071+0000 7f9988200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f998015f9c0 0x7f99800cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.071+0000 7f9988200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998015c5f0 con 0x7f998013df80 2024-01-16T15:12:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.071+0000 7f9988200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998006c470 con 0x7f998015f9c0 2024-01-16T15:12:55.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.071+0000 7f9988200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998006c750 con 0x7f998013e360 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.072+0000 7f9986200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f998015f9c0 0x7f99800cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.072+0000 7f9986200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f998015f9c0 0x7f99800cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57540/0 (socket says 172.21.15.53:57540) 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.072+0000 7f9985800640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f998013df80 0x7f998015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:55830/0 (socket says 172.21.15.53:55830) 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.072+0000 7f9986200640 1 -- 172.21.15.53:0/1547919916 learned_addr learned my addr 172.21.15.53:0/1547919916 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.072+0000 7f9984e00640 1 --2- 172.21.15.53:0/1547919916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f99800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.073+0000 7f9986200640 1 -- 172.21.15.53:0/1547919916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 msgr2=0x7f99800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.073+0000 7f9986200640 1 --2- 172.21.15.53:0/1547919916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f99800cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.073+0000 7f9986200640 1 -- 172.21.15.53:0/1547919916 >> v1:172.21.15.146:6789/0 conn(0x7f998013df80 legacy=0x7f998015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.073+0000 7f9986200640 1 -- 172.21.15.53:0/1547919916 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998006ca30 con 0x7f998015f9c0 2024-01-16T15:12:55.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.074+0000 7f9986200640 1 --2- 172.21.15.53:0/1547919916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99800cf7d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f9978009e50 tx=0x7f997802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec775e476ff768e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f997fe00640 1 -- 172.21.15.53:0/1547919916 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997802cb00 con 0x7f998015f9c0 2024-01-16T15:12:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f997fe00640 1 -- 172.21.15.53:0/1547919916 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997802cc60 con 0x7f998015f9c0 2024-01-16T15:12:55.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f997fe00640 1 -- 172.21.15.53:0/1547919916 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99780343f0 con 0x7f998015f9c0 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f9988200640 1 -- 172.21.15.53:0/1547919916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 msgr2=0x7f99800cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f9988200640 1 --2- 172.21.15.53:0/1547919916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99800cf7d0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f9978009e50 tx=0x7f997802bf60 comp rx=0 tx=0).stop 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f9988200640 1 -- 172.21.15.53:0/1547919916 shutdown_connections 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f9988200640 1 --2- 172.21.15.53:0/1547919916 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99800cf7d0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f9988200640 1 --2- 172.21.15.53:0/1547919916 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f99800cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.075+0000 7f9988200640 1 -- 172.21.15.53:0/1547919916 >> 172.21.15.53:0/1547919916 conn(0x7f998006a800 msgr2=0x7f998006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.076+0000 7f9988200640 1 -- 172.21.15.53:0/1547919916 shutdown_connections 2024-01-16T15:12:55.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.076+0000 7f9988200640 1 -- 172.21.15.53:0/1547919916 wait complete. 2024-01-16T15:12:55.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.077+0000 7f9988200640 1 Processor -- start 2024-01-16T15:12:55.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 -- start start 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f998013df80 0x7f9980150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f998013d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99801f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801f68f0 con 0x7f998015f9c0 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99801f6580 con 0x7f998013df80 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.078+0000 7f9988200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99801f3790 con 0x7f998013e360 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99801f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99801f2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57542/0 (socket says 172.21.15.53:57542) 2024-01-16T15:12:55.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 -- 172.21.15.53:0/1402792974 learned_addr learned my addr 172.21.15.53:0/1402792974 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9985800640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f998013df80 0x7f9980150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9984e00640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f998013d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 -- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 msgr2=0x7f998013d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f998013d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 -- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f998013df80 msgr2=0x7f9980150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f998013df80 0x7f9980150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9986200640 1 -- 172.21.15.53:0/1402792974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9978009b30 con 0x7f998015f9c0 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.079+0000 7f9985800640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f998013df80 0x7f9980150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.080+0000 7f9986200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99801f2ec0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f9978036040 tx=0x7f9978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.080+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9978009510 con 0x7f998015f9c0 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.080+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997803e450 con 0x7f998015f9c0 2024-01-16T15:12:55.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.080+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997803e800 con 0x7f998015f9c0 2024-01-16T15:12:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.080+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99801f3a70 con 0x7f998015f9c0 2024-01-16T15:12:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.080+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99801f3d50 con 0x7f998015f9c0 2024-01-16T15:12:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.081+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99801f6a30 con 0x7f998015f9c0 2024-01-16T15:12:55.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.082+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f997803e960 con 0x7f998015f9c0 2024-01-16T15:12:55.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.083+0000 7f997d600640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f995806bf20 0x7f995806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.083+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f99780ab500 con 0x7f998015f9c0 2024-01-16T15:12:55.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.084+0000 7f9985800640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f995806bf20 0x7f995806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.085+0000 7f9985800640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f995806bf20 0x7f995806e3e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9970000f60 tx=0x7f9970009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.088+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99801f6a30 con 0x7f998015f9c0 2024-01-16T15:12:55.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.321+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54148", "target": ["mon-mgr", ""]}) v1 -- 0x7f998006cff0 con 0x7f995806bf20 2024-01-16T15:12:55.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.345+0000 7f997d600640 1 -- 172.21.15.53:0/1402792974 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f9970001310 con 0x7f995806bf20 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:55.323 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:55.324 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:55.324 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:55.324 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:55.324 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f995806bf20 msgr2=0x7f995806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f995806bf20 0x7f995806e3e0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f9970000f60 tx=0x7f9970009040 comp rx=0 tx=0).stop 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 msgr2=0x7f99801f2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99801f2ec0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7f9978036040 tx=0x7f9978004a40 comp rx=0 tx=0).stop 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 shutdown_connections 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f995806bf20 0x7f995806e3e0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f998015f9c0 0x7f99801f2ec0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f998013e360 0x7f998013d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 --2- 172.21.15.53:0/1402792974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f998013df80 0x7f9980150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 >> 172.21.15.53:0/1402792974 conn(0x7f998006a800 msgr2=0x7f998015caa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.348+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 shutdown_connections 2024-01-16T15:12:55.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.349+0000 7f9988200640 1 -- 172.21.15.53:0/1402792974 wait complete. 2024-01-16T15:12:55.344 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:55.344 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone54149 2024-01-16T15:12:55.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.593+0000 7f3edbe00640 1 Processor -- start 2024-01-16T15:12:55.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.594+0000 7f3edbe00640 1 -- start start 2024-01-16T15:12:55.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.595+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc143810 0x7f3edc071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.595+0000 7f3edbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3edc0ae170 0x7f3edc15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.595+0000 7f3edbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3edc15d770 0x7f3edc15dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.595+0000 7f3edbe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc143bf0 con 0x7f3edc15d770 2024-01-16T15:12:55.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.595+0000 7f3edbe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3edc15e140 con 0x7f3edc0ae170 2024-01-16T15:12:55.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.595+0000 7f3edbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc06c730 con 0x7f3edc143810 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3edc15d770 0x7f3edc15dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3edc15d770 0x7f3edc15dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57544/0 (socket says 172.21.15.53:57544) 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 -- 172.21.15.53:0/3572586034 learned_addr learned my addr 172.21.15.53:0/3572586034 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 -- 172.21.15.53:0/3572586034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc143810 msgr2=0x7f3edc071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 --2- 172.21.15.53:0/3572586034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc143810 0x7f3edc071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 -- 172.21.15.53:0/3572586034 >> v2:172.21.15.146:3300/0 conn(0x7f3edc0ae170 msgr2=0x7f3edc15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:55.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3ed3e00640 1 --2- 172.21.15.53:0/3572586034 >> v2:172.21.15.146:3300/0 conn(0x7f3edc0ae170 0x7f3edc15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 --2- 172.21.15.53:0/3572586034 >> v2:172.21.15.146:3300/0 conn(0x7f3edc0ae170 0x7f3edc15e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.596+0000 7f3edb400640 1 -- 172.21.15.53:0/3572586034 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0ae5f0 con 0x7f3edc15d770 2024-01-16T15:12:55.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.598+0000 7f3edb400640 1 --2- 172.21.15.53:0/3572586034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc15d770 0x7f3edc15dc00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f3edc06e350 tx=0x7f3ed402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=778d61c7e7ec30e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.598+0000 7f3eda000640 1 -- 172.21.15.53:0/3572586034 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed402c9e0 con 0x7f3edc15d770 2024-01-16T15:12:55.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.598+0000 7f3eda000640 1 -- 172.21.15.53:0/3572586034 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ed402cb40 con 0x7f3edc15d770 2024-01-16T15:12:55.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.598+0000 7f3eda000640 1 -- 172.21.15.53:0/3572586034 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ed40343f0 con 0x7f3edc15d770 2024-01-16T15:12:55.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 -- 172.21.15.53:0/3572586034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc15d770 msgr2=0x7f3edc15dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 --2- 172.21.15.53:0/3572586034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc15d770 0x7f3edc15dc00 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7f3edc06e350 tx=0x7f3ed402be40 comp rx=0 tx=0).stop 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 -- 172.21.15.53:0/3572586034 shutdown_connections 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 --2- 172.21.15.53:0/3572586034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc15d770 0x7f3edc15dc00 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 --2- 172.21.15.53:0/3572586034 >> v2:172.21.15.146:3300/0 conn(0x7f3edc0ae170 0x7f3edc15e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 --2- 172.21.15.53:0/3572586034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc143810 0x7f3edc071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 -- 172.21.15.53:0/3572586034 >> 172.21.15.53:0/3572586034 conn(0x7f3edc06a7d0 msgr2=0x7f3edc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 -- 172.21.15.53:0/3572586034 shutdown_connections 2024-01-16T15:12:55.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.599+0000 7f3edbe00640 1 -- 172.21.15.53:0/3572586034 wait complete. 2024-01-16T15:12:55.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.601+0000 7f3edbe00640 1 Processor -- start 2024-01-16T15:12:55.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.602+0000 7f3edbe00640 1 -- start start 2024-01-16T15:12:55.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3edc0ae170 0x7f3edc13ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 0x7f3edc06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3edbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc15d770 0x7f3edc06da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3edbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc14f5e0 con 0x7f3edc143810 2024-01-16T15:12:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3edbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc14f8c0 con 0x7f3edc0ae170 2024-01-16T15:12:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3edbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3edc14fba0 con 0x7f3edc15d770 2024-01-16T15:12:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3ed3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 0x7f3edc06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3ed3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 0x7f3edc06d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57556/0 (socket says 172.21.15.53:57556) 2024-01-16T15:12:55.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.603+0000 7f3ed3e00640 1 -- 172.21.15.53:0/1632022080 learned_addr learned my addr 172.21.15.53:0/1632022080 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3edaa00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3edc0ae170 0x7f3edc13ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3edb400640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc15d770 0x7f3edc06da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed3e00640 1 -- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc15d770 msgr2=0x7f3edc06da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed3e00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc15d770 0x7f3edc06da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed3e00640 1 -- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3edc0ae170 msgr2=0x7f3edc13ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed3e00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3edc0ae170 0x7f3edc13ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed3e00640 1 -- 172.21.15.53:0/1632022080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ed4009b30 con 0x7f3edc143810 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3edaa00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3edc0ae170 0x7f3edc13ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed3e00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 0x7f3edc06d4d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400c7f0 tx=0x7f3ec400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4004220 con 0x7f3edc143810 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec40043a0 con 0x7f3edc143810 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ec4007860 con 0x7f3edc143810 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.604+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc14fe80 con 0x7f3edc143810 2024-01-16T15:12:55.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.605+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc13f220 con 0x7f3edc143810 2024-01-16T15:12:55.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.606+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc1503a0 con 0x7f3edc143810 2024-01-16T15:12:55.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.607+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f3ec4007bd0 con 0x7f3edc143810 2024-01-16T15:12:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.608+0000 7f3ed2a00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3eb406bf50 0x7f3eb406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:55.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.608+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ec4088930 con 0x7f3edc143810 2024-01-16T15:12:55.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.609+0000 7f3edaa00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3eb406bf50 0x7f3eb406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:55.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.610+0000 7f3edaa00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3eb406bf50 0x7f3eb406e410 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc006fd0 tx=0x7f3ecc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:55.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.612+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3edc1503a0 con 0x7f3edc143810 2024-01-16T15:12:55.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.844+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone54149", "target": ["mon-mgr", ""]}) v1 -- 0x7f3edc14fa30 con 0x7f3eb406bf50 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.866+0000 7f3ed2a00640 1 -- 172.21.15.53:0/1632022080 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f3edc14fa30 con 0x7f3eb406bf50 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:55.843 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:55.844 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:55.844 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:55.844 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:55.844 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:55.844 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:55.844 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3eb406bf50 msgr2=0x7f3eb406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3eb406bf50 0x7f3eb406e410 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc006fd0 tx=0x7f3ecc008040 comp rx=0 tx=0).stop 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 msgr2=0x7f3edc06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 0x7f3edc06d4d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f3ec400c7f0 tx=0x7f3ec400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 shutdown_connections 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3eb406bf50 0x7f3eb406e410 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3edc15d770 0x7f3edc06da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3edc143810 0x7f3edc06d4d0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 --2- 172.21.15.53:0/1632022080 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3edc0ae170 0x7f3edc13ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.868+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 >> 172.21.15.53:0/1632022080 conn(0x7f3edc06a7d0 msgr2=0x7f3edc06abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:55.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.869+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 shutdown_connections 2024-01-16T15:12:55.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:55.869+0000 7f3edbe00640 1 -- 172.21.15.53:0/1632022080 wait complete. 2024-01-16T15:12:55.864 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:55.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541410 2024-01-16T15:12:56.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.111+0000 7f10f5600640 1 Processor -- start 2024-01-16T15:12:56.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.112+0000 7f10f5600640 1 -- start start 2024-01-16T15:12:56.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.113+0000 7f10f5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.113+0000 7f10f5600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f006c470 con 0x7f10f013e360 2024-01-16T15:12:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.113+0000 7f10f5600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10f006c750 con 0x7f10f00cf9a0 2024-01-16T15:12:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.113+0000 7f10f5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f006ca30 con 0x7f10f013df80 2024-01-16T15:12:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.114+0000 7f10ee000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f10f013e360 0x7f10f00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49512/0 (socket says 172.21.15.53:49512) 2024-01-16T15:12:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.114+0000 7f10ee000640 1 -- 172.21.15.53:0/1638573020 learned_addr learned my addr 172.21.15.53:0/1638573020 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:56.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.114+0000 7f10eea00640 1 --2- 172.21.15.53:0/1638573020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.115+0000 7f10ed600640 1 -- 172.21.15.53:0/1638573020 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2981818085 0 0) 0x7f10f006c470 con 0x7f10f013e360 2024-01-16T15:12:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.115+0000 7f10ed600640 1 -- 172.21.15.53:0/1638573020 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10d0003520 con 0x7f10f013e360 2024-01-16T15:12:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.115+0000 7f10ed600640 1 -- 172.21.15.53:0/1638573020 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2841212128 0 0) 0x7f10d0003520 con 0x7f10f013e360 2024-01-16T15:12:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.115+0000 7f10eea00640 1 -- 172.21.15.53:0/1638573020 >> v1:172.21.15.146:6789/0 conn(0x7f10f00cf9a0 legacy=0x7f10f00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.115+0000 7f10eea00640 1 -- 172.21.15.53:0/1638573020 >> v1:172.21.15.53:6789/0 conn(0x7f10f013e360 legacy=0x7f10f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.115+0000 7f10eea00640 1 -- 172.21.15.53:0/1638573020 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10f006cd10 con 0x7f10f013df80 2024-01-16T15:12:56.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.116+0000 7f10eea00640 1 --2- 172.21.15.53:0/1638573020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f015bee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f10e00097d0 tx=0x7f10e002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87ed7035cb0fa3b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.117+0000 7f10ed600640 1 -- 172.21.15.53:0/1638573020 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d80019f0 con 0x7f10f013df80 2024-01-16T15:12:56.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.117+0000 7f10ed600640 1 -- 172.21.15.53:0/1638573020 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e002e050 con 0x7f10f013df80 2024-01-16T15:12:56.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.117+0000 7f10ed600640 1 -- 172.21.15.53:0/1638573020 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e0030730 con 0x7f10f013df80 2024-01-16T15:12:56.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.117+0000 7f10f5600640 1 -- 172.21.15.53:0/1638573020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 msgr2=0x7f10f015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.117+0000 7f10f5600640 1 --2- 172.21.15.53:0/1638573020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f015bee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f10e00097d0 tx=0x7f10e002b800 comp rx=0 tx=0).stop 2024-01-16T15:12:56.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.118+0000 7f10f5600640 1 -- 172.21.15.53:0/1638573020 shutdown_connections 2024-01-16T15:12:56.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.118+0000 7f10f5600640 1 --2- 172.21.15.53:0/1638573020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f015bee0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.118+0000 7f10f5600640 1 -- 172.21.15.53:0/1638573020 >> 172.21.15.53:0/1638573020 conn(0x7f10f006a800 msgr2=0x7f10f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.118+0000 7f10f5600640 1 -- 172.21.15.53:0/1638573020 shutdown_connections 2024-01-16T15:12:56.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.118+0000 7f10f5600640 1 -- 172.21.15.53:0/1638573020 wait complete. 2024-01-16T15:12:56.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.120+0000 7f10f5600640 1 Processor -- start 2024-01-16T15:12:56.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.120+0000 7f10f5600640 1 -- start start 2024-01-16T15:12:56.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10f5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10f00cf9a0 0x7f10f01531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10f5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f013cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10f5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 0x7f10f013d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10f5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f013b5d0 con 0x7f10f013e360 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10f5600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10f013b8b0 con 0x7f10f00cf9a0 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10f5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10f013bb90 con 0x7f10f013df80 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10ef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 0x7f10f013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10ef400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 0x7f10f013d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57562/0 (socket says 172.21.15.53:57562) 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10ef400640 1 -- 172.21.15.53:0/2589818005 learned_addr learned my addr 172.21.15.53:0/2589818005 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10ee000640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f013cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.121+0000 7f10eea00640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10f00cf9a0 0x7f10f01531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10ef400640 1 -- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 msgr2=0x7f10f013cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10ef400640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f013cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10ef400640 1 -- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10f00cf9a0 msgr2=0x7f10f01531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10ef400640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10f00cf9a0 0x7f10f01531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10ef400640 1 -- 172.21.15.53:0/2589818005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10e0009400 con 0x7f10f013e360 2024-01-16T15:12:56.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10ef400640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 0x7f10f013d020 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c390 tx=0x7f10e800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e80052d0 con 0x7f10f013e360 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10e8005430 con 0x7f10f013e360 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10e8003810 con 0x7f10f013e360 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.122+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10f013be70 con 0x7f10f013e360 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.123+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10f0153720 con 0x7f10f013e360 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.124+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10f013c390 con 0x7f10f013e360 2024-01-16T15:12:56.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.125+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f10e8003970 con 0x7f10f013e360 2024-01-16T15:12:56.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.126+0000 7f10e6a00640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f10c806bf70 0x7f10c806e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.126+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f10e8087dd0 con 0x7f10f013e360 2024-01-16T15:12:56.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.127+0000 7f10eea00640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f10c806bf70 0x7f10c806e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.128+0000 7f10eea00640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f10c806bf70 0x7f10c806e430 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f10e0034000 tx=0x7f10e0008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.130+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10e8054450 con 0x7f10f013e360 2024-01-16T15:12:56.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.364+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541410", "target": ["mon-mgr", ""]}) v1 -- 0x7f10f013ba20 con 0x7f10c806bf70 2024-01-16T15:12:56.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.389+0000 7f10e6a00640 1 -- 172.21.15.53:0/2589818005 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f10f013ba20 con 0x7f10c806bf70 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:56.367 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:56.368 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:56.368 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:56.368 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:56.368 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f10c806bf70 msgr2=0x7f10c806e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f10c806bf70 0x7f10c806e430 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f10e0034000 tx=0x7f10e0008030 comp rx=0 tx=0).stop 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 msgr2=0x7f10f013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 0x7f10f013d020 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f10e800c390 tx=0x7f10e800c850 comp rx=0 tx=0).stop 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 shutdown_connections 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f10c806bf70 0x7f10c806e430 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10f013e360 0x7f10f013d020 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10f013df80 0x7f10f013cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 --2- 172.21.15.53:0/2589818005 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10f00cf9a0 0x7f10f01531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.392+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 >> 172.21.15.53:0/2589818005 conn(0x7f10f006a800 msgr2=0x7f10f006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.393+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 shutdown_connections 2024-01-16T15:12:56.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.393+0000 7f10f5600640 1 -- 172.21.15.53:0/2589818005 wait complete. 2024-01-16T15:12:56.390 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:56.390 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541411 2024-01-16T15:12:56.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.635+0000 7f65d1a00640 1 Processor -- start 2024-01-16T15:12:56.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.635+0000 7f65d1a00640 1 -- start start 2024-01-16T15:12:56.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65d1a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65cc0adf20 0x7f65cc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65d1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 0x7f65cc143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65d1a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc15f4e0 con 0x7f65cc15bf10 2024-01-16T15:12:56.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65d1a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65cc15f970 con 0x7f65cc0adf20 2024-01-16T15:12:56.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65d1a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc071130 con 0x7f65cc0ccb50 2024-01-16T15:12:56.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65cbe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f65cc0adf20 0x7f65cc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.637+0000 7f65d1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 0x7f65cc143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65d1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 0x7f65cc143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:54314/0 (socket says 172.21.15.53:54314) 2024-01-16T15:12:56.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65d1000640 1 -- 172.21.15.53:0/30261629 learned_addr learned my addr 172.21.15.53:0/30261629 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:56.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65cbe00640 1 -- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 msgr2=0x7f65cc143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:56.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65cbe00640 1 --2- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 0x7f65cc143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65cbe00640 1 -- 172.21.15.53:0/30261629 >> v1:172.21.15.146:6789/0 conn(0x7f65cc15bf10 legacy=0x7f65cc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65cbe00640 1 -- 172.21.15.53:0/30261629 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc15c390 con 0x7f65cc0adf20 2024-01-16T15:12:56.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.638+0000 7f65d1000640 1 --2- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 0x7f65cc143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:56.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.640+0000 7f65cbe00640 1 --2- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc070a30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0097a0 tx=0x7f65bc02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51edb8e7d45785b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.640+0000 7f65cb400640 1 -- 172.21.15.53:0/30261629 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc031780 con 0x7f65cc0adf20 2024-01-16T15:12:56.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65cb400640 1 -- 172.21.15.53:0/30261629 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65bc0318e0 con 0x7f65cc0adf20 2024-01-16T15:12:56.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65cb400640 1 -- 172.21.15.53:0/30261629 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65bc031c70 con 0x7f65cc0adf20 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65d1a00640 1 -- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 msgr2=0x7f65cc070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65d1a00640 1 --2- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc070a30 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f65bc0097a0 tx=0x7f65bc02b820 comp rx=0 tx=0).stop 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65d1a00640 1 -- 172.21.15.53:0/30261629 shutdown_connections 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65d1a00640 1 --2- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc0ccb50 0x7f65cc143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65d1a00640 1 --2- 172.21.15.53:0/30261629 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc070a30 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.641+0000 7f65d1a00640 1 -- 172.21.15.53:0/30261629 >> 172.21.15.53:0/30261629 conn(0x7f65cc06a7d0 msgr2=0x7f65cc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.642+0000 7f65d1a00640 1 -- 172.21.15.53:0/30261629 shutdown_connections 2024-01-16T15:12:56.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.642+0000 7f65d1a00640 1 -- 172.21.15.53:0/30261629 wait complete. 2024-01-16T15:12:56.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.644+0000 7f65d1a00640 1 Processor -- start 2024-01-16T15:12:56.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.644+0000 7f65d1a00640 1 -- start start 2024-01-16T15:12:56.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.645+0000 7f65d1a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc142a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.645+0000 7f65d1a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 0x7f65cc15d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.645+0000 7f65d1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc15bf10 0x7f65cc15db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.645+0000 7f65d1a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc1e9520 con 0x7f65cc0ccb50 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.645+0000 7f65d1a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc142f50 con 0x7f65cc0adf20 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.645+0000 7f65d1a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc1e7680 con 0x7f65cc15bf10 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 0x7f65cc15d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 0x7f65cc15d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57576/0 (socket says 172.21.15.53:57576) 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 -- 172.21.15.53:0/2952013357 learned_addr learned my addr 172.21.15.53:0/2952013357 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65cbe00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc142a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65d1000640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc15bf10 0x7f65cc15db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 -- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc15bf10 msgr2=0x7f65cc15db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc15bf10 0x7f65cc15db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 -- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 msgr2=0x7f65cc142a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc142a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.646+0000 7f65c3e00640 1 -- 172.21.15.53:0/2952013357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c4009b30 con 0x7f65cc0ccb50 2024-01-16T15:12:56.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.647+0000 7f65c3e00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 0x7f65cc15d5d0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f65b400c3f0 tx=0x7f65b400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.647+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b40052d0 con 0x7f65cc0ccb50 2024-01-16T15:12:56.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.647+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b4005450 con 0x7f65cc0ccb50 2024-01-16T15:12:56.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.647+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b4007260 con 0x7f65cc0ccb50 2024-01-16T15:12:56.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.647+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65bc009400 con 0x7f65cc0ccb50 2024-01-16T15:12:56.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.647+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc1e7c40 con 0x7f65cc0ccb50 2024-01-16T15:12:56.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.648+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc067450 con 0x7f65cc0ccb50 2024-01-16T15:12:56.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.650+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f65b40073c0 con 0x7f65cc0ccb50 2024-01-16T15:12:56.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.651+0000 7f65c8c00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65a406bf50 0x7f65a406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:56.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.651+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f65b4087c40 con 0x7f65cc0ccb50 2024-01-16T15:12:56.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.652+0000 7f65cbe00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65a406bf50 0x7f65a406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:56.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.654+0000 7f65cbe00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65a406bf50 0x7f65a406e410 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f65bc036000 tx=0x7f65bc03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:56.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.656+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65b40541f0 con 0x7f65cc0ccb50 2024-01-16T15:12:56.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.878+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541411", "target": ["mon-mgr", ""]}) v1 -- 0x7f65cc15f170 con 0x7f65a406bf50 2024-01-16T15:12:56.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.900+0000 7f65c8c00640 1 -- 172.21.15.53:0/2952013357 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f65bc031a50 con 0x7f65a406bf50 2024-01-16T15:12:56.877 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:56.878 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65a406bf50 msgr2=0x7f65a406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65a406bf50 0x7f65a406e410 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f65bc036000 tx=0x7f65bc03d040 comp rx=0 tx=0).stop 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 msgr2=0x7f65cc15d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 0x7f65cc15d5d0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f65b400c3f0 tx=0x7f65b400c8b0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 shutdown_connections 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65a406bf50 0x7f65a406e410 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65cc15bf10 0x7f65cc15db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65cc0ccb50 0x7f65cc15d5d0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 --2- 172.21.15.53:0/2952013357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65cc0adf20 0x7f65cc142a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:56.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.902+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 >> 172.21.15.53:0/2952013357 conn(0x7f65cc06a7d0 msgr2=0x7f65cc15b3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:56.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.903+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 shutdown_connections 2024-01-16T15:12:56.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:56.903+0000 7f65d1a00640 1 -- 172.21.15.53:0/2952013357 wait complete. 2024-01-16T15:12:56.896 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:56.896 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541412 2024-01-16T15:12:57.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.145+0000 7fb5c0c00640 1 Processor -- start 2024-01-16T15:12:57.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.145+0000 7fb5c0c00640 1 -- start start 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.146+0000 7fb5c0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 0x7fb5bc0d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.146+0000 7fb5c0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb5bc160e10 0x7fb5bc0d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5c0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5bc0d4160 con 0x7fb5bc0cfcf0 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5c0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5bc0ceff0 con 0x7fb5bc160e10 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5c0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0cf410 con 0x7fb5bc0d3910 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5b9e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb5bc0cfcf0 0x7fb5bc160700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49528/0 (socket says 172.21.15.53:49528) 2024-01-16T15:12:57.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5b9e00640 1 -- 172.21.15.53:0/1299122742 learned_addr learned my addr 172.21.15.53:0/1299122742 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:57.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5ba800640 1 --2- 172.21.15.53:0/1299122742 >> v2:172.21.15.146:3300/0 conn(0x7fb5bc160e10 0x7fb5bc0d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5b9400640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 0x7fb5bc0d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194938624 0 0) 0x7fb5bc0d4160 con 0x7fb5bc0cfcf0 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.147+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb59c003520 con 0x7fb5bc0cfcf0 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1625410818 0 0) 0x7fb59c003520 con 0x7fb5bc0cfcf0 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 msgr2=0x7fb5bc0d1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 0x7fb5bc0d1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 >> v2:172.21.15.146:3300/0 conn(0x7fb5bc160e10 msgr2=0x7fb5bc0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 --2- 172.21.15.53:0/1299122742 >> v2:172.21.15.146:3300/0 conn(0x7fb5bc160e10 0x7fb5bc0d1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc0b2420 con 0x7fb5bc0cfcf0 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb5ac0019f0 con 0x7fb5bc0cfcf0 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 >> v1:172.21.15.53:6789/0 conn(0x7fb5bc0cfcf0 legacy=0x7fb5bc160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59c0039a0 0x7fb59c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb59c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 0x7fb59c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0ceff0 con 0x7fb59c0047c0 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0cf410 con 0x7fb59c0039a0 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59c00ba00 con 0x7fb5bc0d3910 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.148+0000 7fb5b9400640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb59c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5b9e00640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59c0039a0 0x7fb59c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5ba800640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 0x7fb59c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5b9400640 1 -- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 msgr2=0x7fb59c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5b9400640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc0d3910 0x7fb59c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5b9400640 1 -- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59c0039a0 msgr2=0x7fb59c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5b9400640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb59c0039a0 0x7fb59c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.149+0000 7fb5b9400640 1 -- 172.21.15.53:0/1299122742 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c003520 con 0x7fb59c0047c0 2024-01-16T15:12:57.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.150+0000 7fb5b9400640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb59c004c40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ac0 tx=0x7fb59c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6124e4ecf5d0ddc0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:57.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.150+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a402c6b0 con 0x7fb59c0047c0 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5a402c810 con 0x7fb59c0047c0 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5b8a00640 1 -- 172.21.15.53:0/1299122742 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5a402cba0 con 0x7fb59c0047c0 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5c0c00640 1 -- 172.21.15.53:0/1299122742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 msgr2=0x7fb59c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/1299122742 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb59c004c40 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ac0 tx=0x7fb59c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5ba800640 1 -- 172.21.15.53:0/1299122742 reap_dead start 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5c0c00640 1 -- 172.21.15.53:0/1299122742 shutdown_connections 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5c0c00640 1 -- 172.21.15.53:0/1299122742 >> 172.21.15.53:0/1299122742 conn(0x7fb5bc09c530 msgr2=0x7fb5bc0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5c0c00640 1 -- 172.21.15.53:0/1299122742 shutdown_connections 2024-01-16T15:12:57.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.151+0000 7fb5c0c00640 1 -- 172.21.15.53:0/1299122742 wait complete. 2024-01-16T15:12:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.153+0000 7fb5c0c00640 1 Processor -- start 2024-01-16T15:12:57.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.153+0000 7fb5c0c00640 1 -- start start 2024-01-16T15:12:57.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.154+0000 7fb5c0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb5bc154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.154+0000 7fb5c0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5bc1e7e00 0x7fb5bc0b8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.154+0000 7fb5c0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc1e7180 0x7fb5bc1e7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.154+0000 7fb5c0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc0c82a0 con 0x7fb59c0047c0 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.154+0000 7fb5c0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc1e7b70 con 0x7fb5bc1e7e00 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.154+0000 7fb5c0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5bc154d80 con 0x7fb5bc1e7180 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb5bc154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb5bc154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57606/0 (socket says 172.21.15.53:57606) 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 -- 172.21.15.53:0/3186332740 learned_addr learned my addr 172.21.15.53:0/3186332740 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5ba800640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc1e7180 0x7fb5bc1e7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9400640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5bc1e7e00 0x7fb5bc0b8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 -- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc1e7180 msgr2=0x7fb5bc1e7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc1e7180 0x7fb5bc1e7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 -- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5bc1e7e00 msgr2=0x7fb5bc0b8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5bc1e7e00 0x7fb5bc0b8870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.155+0000 7fb5b9e00640 1 -- 172.21.15.53:0/3186332740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5bc0c76b0 con 0x7fb59c0047c0 2024-01-16T15:12:57.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.156+0000 7fb5b9e00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb5bc154840 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0079f0 tx=0x7fb5ac00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.156+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac00ee50 con 0x7fb59c0047c0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.156+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5ac014480 con 0x7fb59c0047c0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.156+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5ac014830 con 0x7fb59c0047c0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.156+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5bc0c7900 con 0x7fb59c0047c0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.156+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5bc0c7ec0 con 0x7fb59c0047c0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.157+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5bc0c83e0 con 0x7fb59c0047c0 2024-01-16T15:12:57.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.158+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb5ac012380 con 0x7fb59c0047c0 2024-01-16T15:12:57.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.159+0000 7fb5b2000640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb59406bf50 0x7fb59406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.159+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb5ac088a20 con 0x7fb59c0047c0 2024-01-16T15:12:57.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.160+0000 7fb5b9400640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb59406bf50 0x7fb59406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.161+0000 7fb5b9400640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb59406bf50 0x7fb59406e410 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ac0 tx=0x7fb5a4030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:57.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.164+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5bc0c83e0 con 0x7fb59c0047c0 2024-01-16T15:12:57.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.399+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541412", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5bc0b1fb0 con 0x7fb59406bf50 2024-01-16T15:12:57.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.424+0000 7fb5b2000640 1 -- 172.21.15.53:0/3186332740 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb5bc0b1fb0 con 0x7fb59406bf50 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:57.402 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:57.403 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:57.403 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:57.403 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:57.403 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb59406bf50 msgr2=0x7fb59406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb59406bf50 0x7fb59406e410 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fb5a4009ac0 tx=0x7fb5a4030000 comp rx=0 tx=0).stop 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 msgr2=0x7fb5bc154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb5bc154840 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7fb5ac0079f0 tx=0x7fb5ac00e530 comp rx=0 tx=0).stop 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 shutdown_connections 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb59406bf50 0x7fb59406e410 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5bc1e7180 0x7fb5bc1e7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5bc1e7e00 0x7fb5bc0b8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 --2- 172.21.15.53:0/3186332740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb59c0047c0 0x7fb5bc154840 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 >> 172.21.15.53:0/3186332740 conn(0x7fb5bc09c530 msgr2=0x7fb5bc0d1b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 shutdown_connections 2024-01-16T15:12:57.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.427+0000 7fb5c0c00640 1 -- 172.21.15.53:0/3186332740 wait complete. 2024-01-16T15:12:57.425 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:57.425 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541413 2024-01-16T15:12:57.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.674+0000 7f0218a00640 1 Processor -- start 2024-01-16T15:12:57.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.674+0000 7f0218a00640 1 -- start start 2024-01-16T15:12:57.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.675+0000 7f0218a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f02140cd740 0x7f02140cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.675+0000 7f0218a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021406d2f0 0x7f02140cf450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.675+0000 7f0218a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02140cf990 0x7f02140cfe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.675+0000 7f0218a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02140cefe0 con 0x7f02140cf990 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.675+0000 7f0218a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f021406c930 con 0x7f02140cd740 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.675+0000 7f0218a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02140d0380 con 0x7f021406d2f0 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02140cf990 0x7f02140cfe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f02140cf990 0x7f02140cfe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57620/0 (socket says 172.21.15.53:57620) 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 -- 172.21.15.53:0/337817426 learned_addr learned my addr 172.21.15.53:0/337817426 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0212a00640 1 --2- 172.21.15.53:0/337817426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021406d2f0 0x7f02140cf450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213400640 1 --2- 172.21.15.53:0/337817426 >> v2:172.21.15.146:3300/0 conn(0x7f02140cd740 0x7f02140cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 -- 172.21.15.53:0/337817426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021406d2f0 msgr2=0x7f02140cf450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 --2- 172.21.15.53:0/337817426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021406d2f0 0x7f02140cf450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 -- 172.21.15.53:0/337817426 >> v2:172.21.15.146:3300/0 conn(0x7f02140cd740 msgr2=0x7f02140cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 --2- 172.21.15.53:0/337817426 >> v2:172.21.15.146:3300/0 conn(0x7f02140cd740 0x7f02140cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.676+0000 7f0213e00640 1 -- 172.21.15.53:0/337817426 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02140ce060 con 0x7f02140cf990 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.677+0000 7f0213e00640 1 --2- 172.21.15.53:0/337817426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02140cf990 0x7f02140cfe40 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f021409aa60 tx=0x7f020c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c5a1e402f8be74 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0212000640 1 -- 172.21.15.53:0/337817426 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c031870 con 0x7f02140cf990 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0212000640 1 -- 172.21.15.53:0/337817426 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020c0319d0 con 0x7f02140cf990 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0212000640 1 -- 172.21.15.53:0/337817426 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f020c031d60 con 0x7f02140cf990 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 -- 172.21.15.53:0/337817426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02140cf990 msgr2=0x7f02140cfe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 --2- 172.21.15.53:0/337817426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02140cf990 0x7f02140cfe40 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f021409aa60 tx=0x7f020c02b910 comp rx=0 tx=0).stop 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 -- 172.21.15.53:0/337817426 shutdown_connections 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 --2- 172.21.15.53:0/337817426 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02140cf990 0x7f02140cfe40 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 --2- 172.21.15.53:0/337817426 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f021406d2f0 0x7f02140cf450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 --2- 172.21.15.53:0/337817426 >> v2:172.21.15.146:3300/0 conn(0x7f02140cd740 0x7f02140cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 -- 172.21.15.53:0/337817426 >> 172.21.15.53:0/337817426 conn(0x7f02140d0ec0 msgr2=0x7f02140d12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:57.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 -- 172.21.15.53:0/337817426 shutdown_connections 2024-01-16T15:12:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.678+0000 7f0218a00640 1 -- 172.21.15.53:0/337817426 wait complete. 2024-01-16T15:12:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.680+0000 7f0218a00640 1 Processor -- start 2024-01-16T15:12:57.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.680+0000 7f0218a00640 1 -- start start 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.681+0000 7f0218a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 0x7f02141514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.681+0000 7f0218a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02140cd740 0x7f02140c6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.681+0000 7f0218a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02140cf990 0x7f021415b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.681+0000 7f0218a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f021415c2e0 con 0x7f021406d2f0 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.681+0000 7f0218a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02140c6270 con 0x7f02140cd740 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.681+0000 7f0218a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f021415c000 con 0x7f02140cf990 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 0x7f02141514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 0x7f02141514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57622/0 (socket says 172.21.15.53:57622) 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 -- 172.21.15.53:0/510672020 learned_addr learned my addr 172.21.15.53:0/510672020 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213e00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02140cf990 0x7f021415b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0212a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02140cd740 0x7f02140c6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 -- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02140cf990 msgr2=0x7f021415b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02140cf990 0x7f021415b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 -- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02140cd740 msgr2=0x7f02140c6f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02140cd740 0x7f02140c6f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0213400640 1 -- 172.21.15.53:0/510672020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f020c0094d0 con 0x7f021406d2f0 2024-01-16T15:12:57.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.682+0000 7f0212a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02140cd740 0x7f02140c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:57.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.683+0000 7f0213400640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 0x7f02141514d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0204007e40 tx=0x7f020400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.683+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204014620 con 0x7f021406d2f0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.683+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f020400ee90 con 0x7f021406d2f0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.683+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040149c0 con 0x7f021406d2f0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.683+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02140c6480 con 0x7f021406d2f0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.683+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02140c6a40 con 0x7f021406d2f0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.684+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02140c6c00 con 0x7f021406d2f0 2024-01-16T15:12:57.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.685+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0204014b20 con 0x7f021406d2f0 2024-01-16T15:12:57.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.686+0000 7f020b400640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01ec06bf50 0x7f01ec06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:57.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.686+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0204088f10 con 0x7f021406d2f0 2024-01-16T15:12:57.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.687+0000 7f0212a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01ec06bf50 0x7f01ec06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:57.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.688+0000 7f0212a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01ec06bf50 0x7f01ec06e410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f01fc006fd0 tx=0x7f01fc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:57.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.691+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02040557d0 con 0x7f021406d2f0 2024-01-16T15:12:57.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.920+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541413", "target": ["mon-mgr", ""]}) v1 -- 0x7f021406f4c0 con 0x7f01ec06bf50 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.942+0000 7f020b400640 1 -- 172.21.15.53:0/510672020 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f01fc0094b0 con 0x7f01ec06bf50 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:57.920 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:57.921 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:57.921 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:57.921 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:57.921 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:57.921 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:57.921 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01ec06bf50 msgr2=0x7f01ec06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01ec06bf50 0x7f01ec06e410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f01fc006fd0 tx=0x7f01fc008040 comp rx=0 tx=0).stop 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 msgr2=0x7f02141514d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 0x7f02141514d0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0204007e40 tx=0x7f020400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 shutdown_connections 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f01ec06bf50 0x7f01ec06e410 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02140cf990 0x7f021415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f02140cd740 0x7f02140c6f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 --2- 172.21.15.53:0/510672020 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f021406d2f0 0x7f02141514d0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:57.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.945+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 >> 172.21.15.53:0/510672020 conn(0x7f02140d0ec0 msgr2=0x7f02140b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:57.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.946+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 shutdown_connections 2024-01-16T15:12:57.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:57.946+0000 7f0218a00640 1 -- 172.21.15.53:0/510672020 wait complete. 2024-01-16T15:12:57.944 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:57.944 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541414 2024-01-16T15:12:58.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.181+0000 7fb732400640 1 Processor -- start 2024-01-16T15:12:58.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.182+0000 7fb732400640 1 -- start start 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.183+0000 7fb732400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 0x7fb72c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.183+0000 7fb732400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb72c15bf10 0x7fb72c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.183+0000 7fb732400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72c15f4e0 con 0x7fb72c0adf20 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.183+0000 7fb732400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb72c15f970 con 0x7fb72c15bf10 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.183+0000 7fb732400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c15c3c0 con 0x7fb72c0ccb50 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.184+0000 7fb731000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb72c0adf20 0x7fb72c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49538/0 (socket says 172.21.15.53:49538) 2024-01-16T15:12:58.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.184+0000 7fb731000640 1 -- 172.21.15.53:0/607562682 learned_addr learned my addr 172.21.15.53:0/607562682 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:58.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.184+0000 7fb723e00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 0x7fb72c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.184+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306240628 0 0) 0x7fb72c15f4e0 con 0x7fb72c0adf20 2024-01-16T15:12:58.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c003520 con 0x7fb72c0adf20 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb731a00640 1 --2- 172.21.15.53:0/607562682 >> v2:172.21.15.146:3300/0 conn(0x7fb72c15bf10 0x7fb72c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 857163321 0 0) 0x7fb70c003520 con 0x7fb72c0adf20 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 msgr2=0x7fb72c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 0x7fb72c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 >> v2:172.21.15.146:3300/0 conn(0x7fb72c15bf10 msgr2=0x7fb72c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 --2- 172.21.15.53:0/607562682 >> v2:172.21.15.146:3300/0 conn(0x7fb72c15bf10 0x7fb72c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c071130 con 0x7fb72c0adf20 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb71c001670 con 0x7fb72c0adf20 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb731a00640 1 --2- 172.21.15.53:0/607562682 >> v2:172.21.15.146:3300/0 conn(0x7fb72c15bf10 0x7fb72c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 >> v1:172.21.15.53:6789/0 conn(0x7fb72c0adf20 legacy=0x7fb72c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 0x7fb70c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70c003ef0 0x7fb70c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb70c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c15f970 con 0x7fb70c004d00 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c15c3c0 con 0x7fb70c003ef0 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.185+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70c00ba80 con 0x7fb72c0ccb50 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731a00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb70c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731000640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 0x7fb70c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb723e00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70c003ef0 0x7fb70c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731a00640 1 -- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 msgr2=0x7fb70c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731a00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c0ccb50 0x7fb70c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731a00640 1 -- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70c003ef0 msgr2=0x7fb70c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731a00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70c003ef0 0x7fb70c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb731a00640 1 -- 172.21.15.53:0/607562682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c003520 con 0x7fb70c004d00 2024-01-16T15:12:58.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.186+0000 7fb723e00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb70c003ef0 0x7fb70c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:58.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.187+0000 7fb731a00640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb70c005180 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb724007410 tx=0x7fb70c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83638097e5bd964 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:58.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.187+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb72402cdc0 con 0x7fb70c004d00 2024-01-16T15:12:58.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.187+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb724034440 con 0x7fb70c004d00 2024-01-16T15:12:58.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.187+0000 7fb72be00640 1 -- 172.21.15.53:0/607562682 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7240347b0 con 0x7fb70c004d00 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb732400640 1 -- 172.21.15.53:0/607562682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 msgr2=0x7fb70c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb732400640 1 --2- 172.21.15.53:0/607562682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb70c005180 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7fb724007410 tx=0x7fb70c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb731a00640 1 -- 172.21.15.53:0/607562682 reap_dead start 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb732400640 1 -- 172.21.15.53:0/607562682 shutdown_connections 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb732400640 1 -- 172.21.15.53:0/607562682 >> 172.21.15.53:0/607562682 conn(0x7fb72c06a7d0 msgr2=0x7fb72c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb732400640 1 -- 172.21.15.53:0/607562682 shutdown_connections 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.188+0000 7fb732400640 1 -- 172.21.15.53:0/607562682 wait complete. 2024-01-16T15:12:58.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.190+0000 7fb732400640 1 Processor -- start 2024-01-16T15:12:58.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.190+0000 7fb732400640 1 -- start start 2024-01-16T15:12:58.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.191+0000 7fb732400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb72c150590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.191+0000 7fb732400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c15a9d0 0x7fb72c1401d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.191+0000 7fb732400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb72c15d9e0 0x7fb72c15de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.191+0000 7fb732400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c15e3a0 con 0x7fb70c004d00 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.191+0000 7fb732400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c150ad0 con 0x7fb72c15d9e0 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.191+0000 7fb732400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c1f43a0 con 0x7fb72c15a9d0 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb72c150590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb72c150590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57652/0 (socket says 172.21.15.53:57652) 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 -- 172.21.15.53:0/3907287551 learned_addr learned my addr 172.21.15.53:0/3907287551 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731a00640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb72c15d9e0 0x7fb72c15de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 -- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c15a9d0 msgr2=0x7fb72c1401d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c15a9d0 0x7fb72c1401d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 -- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb72c15d9e0 msgr2=0x7fb72c15de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb72c15d9e0 0x7fb72c15de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 -- 172.21.15.53:0/3907287551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c1f4680 con 0x7fb70c004d00 2024-01-16T15:12:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.192+0000 7fb731000640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb72c150590 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00e7a0 tx=0x7fb71c00ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.193+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c00c870 con 0x7fb70c004d00 2024-01-16T15:12:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.193+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71c00c9d0 con 0x7fb70c004d00 2024-01-16T15:12:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.193+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71c00cd80 con 0x7fb70c004d00 2024-01-16T15:12:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.193+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c1f4960 con 0x7fb70c004d00 2024-01-16T15:12:58.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.193+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c140710 con 0x7fb70c004d00 2024-01-16T15:12:58.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.195+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c1f5160 con 0x7fb70c004d00 2024-01-16T15:12:58.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.195+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fb71c01d460 con 0x7fb70c004d00 2024-01-16T15:12:58.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.196+0000 7fb729600640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb70406bf50 0x7fb70406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.196+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb71c0884f0 con 0x7fb70c004d00 2024-01-16T15:12:58.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.197+0000 7fb723e00640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb70406bf50 0x7fb70406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.198+0000 7fb723e00640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb70406bf50 0x7fb70406e410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb72c142e60 tx=0x7fb714008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:58.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.201+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb72c1f5160 con 0x7fb70c004d00 2024-01-16T15:12:58.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.439+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541414", "target": ["mon-mgr", ""]}) v1 -- 0x7fb72c15f170 con 0x7fb70406bf50 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.461+0000 7fb729600640 1 -- 172.21.15.53:0/3907287551 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fb714009650 con 0x7fb70406bf50 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:58.439 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:58.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:58.440 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:58.440 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb70406bf50 msgr2=0x7fb70406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb70406bf50 0x7fb70406e410 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fb72c142e60 tx=0x7fb714008040 comp rx=0 tx=0).stop 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 msgr2=0x7fb72c150590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb72c150590 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7fb71c00e7a0 tx=0x7fb71c00ec60 comp rx=0 tx=0).stop 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 shutdown_connections 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb70406bf50 0x7fb70406e410 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb72c15d9e0 0x7fb72c15de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb72c15a9d0 0x7fb72c1401d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 --2- 172.21.15.53:0/3907287551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb70c004d00 0x7fb72c150590 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.464+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 >> 172.21.15.53:0/3907287551 conn(0x7fb72c06a7d0 msgr2=0x7fb72c0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.465+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 shutdown_connections 2024-01-16T15:12:58.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.465+0000 7fb732400640 1 -- 172.21.15.53:0/3907287551 wait complete. 2024-01-16T15:12:58.462 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:58.462 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541415 2024-01-16T15:12:58.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.713+0000 7f5553e00640 1 Processor -- start 2024-01-16T15:12:58.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.713+0000 7f5553e00640 1 -- start start 2024-01-16T15:12:58.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.715+0000 7f5553e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.715+0000 7f5553e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f554c13e360 0x7f554c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.715+0000 7f5553e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554c06c470 con 0x7f554c15f9c0 2024-01-16T15:12:58.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.715+0000 7f5553e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f554c06c750 con 0x7f554c13e360 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.715+0000 7f5553e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c06ca30 con 0x7f554c13df80 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f554c13e360 0x7f554c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f554c13e360 0x7f554c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57664/0 (socket says 172.21.15.53:57664) 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 -- 172.21.15.53:0/107104474 learned_addr learned my addr 172.21.15.53:0/107104474 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5551400640 1 --2- 172.21.15.53:0/107104474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 -- 172.21.15.53:0/107104474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 msgr2=0x7f554c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 --2- 172.21.15.53:0/107104474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 -- 172.21.15.53:0/107104474 >> v1:172.21.15.146:6789/0 conn(0x7f554c15f9c0 legacy=0x7f554c1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.716+0000 7f5550a00640 1 -- 172.21.15.53:0/107104474 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f554c06cd10 con 0x7f554c13e360 2024-01-16T15:12:58.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.717+0000 7f5550a00640 1 --2- 172.21.15.53:0/107104474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c13e360 0x7f554c0cf290 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5534009870 tx=0x7f553402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1843f8a5cc0aec1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:58.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f554be00640 1 -- 172.21.15.53:0/107104474 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534031640 con 0x7f554c13e360 2024-01-16T15:12:58.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f554be00640 1 -- 172.21.15.53:0/107104474 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55340317a0 con 0x7f554c13e360 2024-01-16T15:12:58.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f554be00640 1 -- 172.21.15.53:0/107104474 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5534031b30 con 0x7f554c13e360 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 -- 172.21.15.53:0/107104474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c13e360 msgr2=0x7f554c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 --2- 172.21.15.53:0/107104474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c13e360 0x7f554c0cf290 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f5534009870 tx=0x7f553402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 -- 172.21.15.53:0/107104474 shutdown_connections 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 --2- 172.21.15.53:0/107104474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c13e360 0x7f554c0cf290 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 --2- 172.21.15.53:0/107104474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 -- 172.21.15.53:0/107104474 >> 172.21.15.53:0/107104474 conn(0x7f554c06a800 msgr2=0x7f554c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 -- 172.21.15.53:0/107104474 shutdown_connections 2024-01-16T15:12:58.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.718+0000 7f5553e00640 1 -- 172.21.15.53:0/107104474 wait complete. 2024-01-16T15:12:58.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.720+0000 7f5553e00640 1 Processor -- start 2024-01-16T15:12:58.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.720+0000 7f5553e00640 1 -- start start 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.721+0000 7f5553e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c070c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.721+0000 7f5553e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f554c13e360 0x7f554c15a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.721+0000 7f5553e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 0x7f554c13d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.721+0000 7f5553e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c14e4d0 con 0x7f554c15f9c0 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.721+0000 7f5553e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f554c071850 con 0x7f554c13e360 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.721+0000 7f5553e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f554c071290 con 0x7f554c13df80 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 0x7f554c13d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 0x7f554c13d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57668/0 (socket says 172.21.15.53:57668) 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 -- 172.21.15.53:0/3568965427 learned_addr learned my addr 172.21.15.53:0/3568965427 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551400640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c070c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5550a00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f554c13e360 0x7f554c15a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 -- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 msgr2=0x7f554c070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c070c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 -- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f554c13e360 msgr2=0x7f554c15a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f554c13e360 0x7f554c15a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 -- 172.21.15.53:0/3568965427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55340094d0 con 0x7f554c15f9c0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.722+0000 7f5551e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 0x7f554c13d410 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f554400c780 tx=0x7f554400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.723+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5544004b60 con 0x7f554c15f9c0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.723+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f554400ce90 con 0x7f554c15f9c0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.723+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f554400a850 con 0x7f554c15f9c0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.723+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f554c071570 con 0x7f554c15f9c0 2024-01-16T15:12:58.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.723+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f554c14b2d0 con 0x7f554c15f9c0 2024-01-16T15:12:58.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.724+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f554c14b4c0 con 0x7f554c15f9c0 2024-01-16T15:12:58.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.725+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f554400a9b0 con 0x7f554c15f9c0 2024-01-16T15:12:58.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.726+0000 7f5549600640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f552406bf20 0x7f552406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:58.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.726+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5544088760 con 0x7f554c15f9c0 2024-01-16T15:12:58.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.727+0000 7f5551400640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f552406bf20 0x7f552406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:58.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.728+0000 7f5551400640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f552406bf20 0x7f552406e3e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f553c006fd0 tx=0x7f553c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:58.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.731+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5544055080 con 0x7f554c15f9c0 2024-01-16T15:12:58.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.965+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541415", "target": ["mon-mgr", ""]}) v1 -- 0x7f554c0ce850 con 0x7f552406bf20 2024-01-16T15:12:58.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.989+0000 7f5549600640 1 -- 172.21.15.53:0/3568965427 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f553c00f860 con 0x7f552406bf20 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:58.967 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:58.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.991+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f552406bf20 msgr2=0x7f552406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.991+0000 7f5553e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f552406bf20 0x7f552406e3e0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f553c006fd0 tx=0x7f553c008040 comp rx=0 tx=0).stop 2024-01-16T15:12:58.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.991+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 msgr2=0x7f554c13d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:58.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.991+0000 7f5553e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 0x7f554c13d410 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f554400c780 tx=0x7f554400cc40 comp rx=0 tx=0).stop 2024-01-16T15:12:58.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 shutdown_connections 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f552406bf20 0x7f552406e3e0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f554c15f9c0 0x7f554c13d410 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f554c13e360 0x7f554c15a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 --2- 172.21.15.53:0/3568965427 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f554c13df80 0x7f554c070c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 >> 172.21.15.53:0/3568965427 conn(0x7f554c06a800 msgr2=0x7f554c15d540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 shutdown_connections 2024-01-16T15:12:58.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:58.992+0000 7f5553e00640 1 -- 172.21.15.53:0/3568965427 wait complete. 2024-01-16T15:12:58.988 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:58.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541416 2024-01-16T15:12:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.238+0000 7f1e4d800640 1 Processor -- start 2024-01-16T15:12:59.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.239+0000 7f1e4d800640 1 -- start start 2024-01-16T15:12:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.240+0000 7f1e4d800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e480adf20 0x7f1e48070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.240+0000 7f1e4d800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e480ccb50 0x7f1e48143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.240+0000 7f1e4d800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4815f4e0 con 0x7f1e4815bf10 2024-01-16T15:12:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.240+0000 7f1e4d800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4815f970 con 0x7f1e480adf20 2024-01-16T15:12:59.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.240+0000 7f1e4d800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e48071130 con 0x7f1e480ccb50 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e480adf20 0x7f1e48070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e3fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1e4815bf10 0x7f1e480cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49552/0 (socket says 172.21.15.53:49552) 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e3fe00640 1 -- 172.21.15.53:0/209186409 learned_addr learned my addr 172.21.15.53:0/209186409 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e4ce00640 1 --2- 172.21.15.53:0/209186409 >> v2:172.21.15.146:3300/0 conn(0x7f1e480ccb50 0x7f1e48143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47400640 1 -- 172.21.15.53:0/209186409 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186486556 0 0) 0x7f1e4815f4e0 con 0x7f1e4815bf10 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47400640 1 -- 172.21.15.53:0/209186409 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e28003520 con 0x7f1e4815bf10 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47e00640 1 -- 172.21.15.53:0/209186409 >> v2:172.21.15.146:3300/0 conn(0x7f1e480ccb50 msgr2=0x7f1e48143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47e00640 1 --2- 172.21.15.53:0/209186409 >> v2:172.21.15.146:3300/0 conn(0x7f1e480ccb50 0x7f1e48143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47e00640 1 -- 172.21.15.53:0/209186409 >> v1:172.21.15.53:6789/0 conn(0x7f1e4815bf10 legacy=0x7f1e480cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.241+0000 7f1e47e00640 1 -- 172.21.15.53:0/209186409 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4815c390 con 0x7f1e480adf20 2024-01-16T15:12:59.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.243+0000 7f1e47e00640 1 --2- 172.21.15.53:0/209186409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e48070a30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f1e4806ff20 tx=0x7f1e3802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd37f5cbd4f74040 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.243+0000 7f1e47400640 1 -- 172.21.15.53:0/209186409 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3802e070 con 0x7f1e480adf20 2024-01-16T15:12:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.243+0000 7f1e47400640 1 -- 172.21.15.53:0/209186409 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e380315a0 con 0x7f1e480adf20 2024-01-16T15:12:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.243+0000 7f1e47400640 1 -- 172.21.15.53:0/209186409 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38031930 con 0x7f1e480adf20 2024-01-16T15:12:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 -- 172.21.15.53:0/209186409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 msgr2=0x7f1e48070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 --2- 172.21.15.53:0/209186409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e48070a30 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f1e4806ff20 tx=0x7f1e3802b840 comp rx=0 tx=0).stop 2024-01-16T15:12:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 -- 172.21.15.53:0/209186409 shutdown_connections 2024-01-16T15:12:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 --2- 172.21.15.53:0/209186409 >> v2:172.21.15.146:3300/0 conn(0x7f1e480ccb50 0x7f1e48143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 --2- 172.21.15.53:0/209186409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e48070a30 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 -- 172.21.15.53:0/209186409 >> 172.21.15.53:0/209186409 conn(0x7f1e4806a7d0 msgr2=0x7f1e4806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 -- 172.21.15.53:0/209186409 shutdown_connections 2024-01-16T15:12:59.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.244+0000 7f1e4d800640 1 -- 172.21.15.53:0/209186409 wait complete. 2024-01-16T15:12:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.246+0000 7f1e4d800640 1 Processor -- start 2024-01-16T15:12:59.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.246+0000 7f1e4d800640 1 -- start start 2024-01-16T15:12:59.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.247+0000 7f1e4d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e4813b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.247+0000 7f1e4d800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e480ccb50 0x7f1e480b2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.247+0000 7f1e4d800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e4815bf10 0x7f1e48139340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.247+0000 7f1e4d800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e480aea70 con 0x7f1e480adf20 2024-01-16T15:12:59.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.247+0000 7f1e4d800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4813b850 con 0x7f1e4815bf10 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.247+0000 7f1e4d800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e4813bb30 con 0x7f1e480ccb50 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e4813b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e4813b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57680/0 (socket says 172.21.15.53:57680) 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 -- 172.21.15.53:0/1650042281 learned_addr learned my addr 172.21.15.53:0/1650042281 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e4ce00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e4815bf10 0x7f1e48139340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e3fe00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e480ccb50 0x7f1e480b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 -- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e480ccb50 msgr2=0x7f1e480b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e480ccb50 0x7f1e480b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 -- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e4815bf10 msgr2=0x7f1e48139340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e4815bf10 0x7f1e48139340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 -- 172.21.15.53:0/1650042281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38009400 con 0x7f1e480adf20 2024-01-16T15:12:59.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e4ce00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e4815bf10 0x7f1e48139340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:12:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.248+0000 7f1e47e00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e4813b1a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1e38000c00 tx=0x7f1e380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.249+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38032840 con 0x7f1e480adf20 2024-01-16T15:12:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.249+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e380329c0 con 0x7f1e480adf20 2024-01-16T15:12:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.249+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e38032d70 con 0x7f1e480adf20 2024-01-16T15:12:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.249+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4813be10 con 0x7f1e480adf20 2024-01-16T15:12:59.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.249+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e480b28a0 con 0x7f1e480adf20 2024-01-16T15:12:59.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.250+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e48067450 con 0x7f1e480adf20 2024-01-16T15:12:59.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.251+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f1e3804d3f0 con 0x7f1e480adf20 2024-01-16T15:12:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.252+0000 7f1e44c00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e2006bf20 0x7f1e2006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.252+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1e380aa0d0 con 0x7f1e480adf20 2024-01-16T15:12:59.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.252+0000 7f1e3fe00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e2006bf20 0x7f1e2006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.253+0000 7f1e3fe00640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e2006bf20 0x7f1e2006e3e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1e48143270 tx=0x7f1e30009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:59.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.256+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e380b0030 con 0x7f1e480adf20 2024-01-16T15:12:59.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.492+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541416", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e4813bca0 con 0x7f1e2006bf20 2024-01-16T15:12:59.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.514+0000 7f1e44c00640 1 -- 172.21.15.53:0/1650042281 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f1e4813bca0 con 0x7f1e2006bf20 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:12:59.492 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:59.493 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:12:59.493 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e2006bf20 msgr2=0x7f1e2006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e2006bf20 0x7f1e2006e3e0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f1e48143270 tx=0x7f1e30009040 comp rx=0 tx=0).stop 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 msgr2=0x7f1e4813b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e4813b1a0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f1e38000c00 tx=0x7f1e380046c0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 shutdown_connections 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e2006bf20 0x7f1e2006e3e0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e4815bf10 0x7f1e48139340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e480ccb50 0x7f1e480b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 --2- 172.21.15.53:0/1650042281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e480adf20 0x7f1e4813b1a0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 >> 172.21.15.53:0/1650042281 conn(0x7f1e4806a7d0 msgr2=0x7f1e4806abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 shutdown_connections 2024-01-16T15:12:59.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.517+0000 7f1e4d800640 1 -- 172.21.15.53:0/1650042281 wait complete. 2024-01-16T15:12:59.515 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:12:59.515 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541417 2024-01-16T15:12:59.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.767+0000 7f2e30a00640 1 Processor -- start 2024-01-16T15:12:59.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.767+0000 7f2e30a00640 1 -- start start 2024-01-16T15:12:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.769+0000 7f2e30a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 0x7f2e2c0ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.769+0000 7f2e30a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c15dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.769+0000 7f2e30a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c15e140 con 0x7f2e2c143810 2024-01-16T15:12:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.769+0000 7f2e30a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c143bf0 con 0x7f2e2c071e40 2024-01-16T15:12:59.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.769+0000 7f2e30a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0722f0 con 0x7f2e2c15d770 2024-01-16T15:12:59.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.770+0000 7f2e2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c15dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.770+0000 7f2e23e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 0x7f2e2c0ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.770+0000 7f2e23e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 0x7f2e2c0ae170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:40262/0 (socket says 172.21.15.53:40262) 2024-01-16T15:12:59.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.770+0000 7f2e23e00640 1 -- 172.21.15.53:0/940750775 learned_addr learned my addr 172.21.15.53:0/940750775 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:59.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.771+0000 7f2e2b200640 1 -- 172.21.15.53:0/940750775 >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 msgr2=0x7f2e2c0ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:12:59.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.771+0000 7f2e2b200640 1 --2- 172.21.15.53:0/940750775 >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 0x7f2e2c0ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.771+0000 7f2e2b200640 1 -- 172.21.15.53:0/940750775 >> v1:172.21.15.146:6789/0 conn(0x7f2e2c143810 legacy=0x7f2e2c15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.771+0000 7f2e2b200640 1 -- 172.21.15.53:0/940750775 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c06c730 con 0x7f2e2c15d770 2024-01-16T15:12:59.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.771+0000 7f2e23e00640 1 --2- 172.21.15.53:0/940750775 >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 0x7f2e2c0ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:12:59.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.772+0000 7f2e2b200640 1 --2- 172.21.15.53:0/940750775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c15dc00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009d90 tx=0x7f2e2402bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88fc24899f97444d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.773+0000 7f2e29e00640 1 -- 172.21.15.53:0/940750775 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e2402cb30 con 0x7f2e2c15d770 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.773+0000 7f2e29e00640 1 -- 172.21.15.53:0/940750775 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e2402cc90 con 0x7f2e2c15d770 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.773+0000 7f2e29e00640 1 -- 172.21.15.53:0/940750775 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e24034440 con 0x7f2e2c15d770 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 -- 172.21.15.53:0/940750775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 msgr2=0x7f2e2c15dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 --2- 172.21.15.53:0/940750775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c15dc00 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2e24009d90 tx=0x7f2e2402bd80 comp rx=0 tx=0).stop 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 -- 172.21.15.53:0/940750775 shutdown_connections 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 --2- 172.21.15.53:0/940750775 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c15dc00 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 --2- 172.21.15.53:0/940750775 >> v2:172.21.15.146:3300/0 conn(0x7f2e2c071e40 0x7f2e2c0ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 -- 172.21.15.53:0/940750775 >> 172.21.15.53:0/940750775 conn(0x7f2e2c06a7d0 msgr2=0x7f2e2c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:12:59.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 -- 172.21.15.53:0/940750775 shutdown_connections 2024-01-16T15:12:59.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.774+0000 7f2e30a00640 1 -- 172.21.15.53:0/940750775 wait complete. 2024-01-16T15:12:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.775+0000 7f2e30a00640 1 Processor -- start 2024-01-16T15:12:59.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.776+0000 7f2e30a00640 1 -- start start 2024-01-16T15:12:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e30a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2c071e40 0x7f2e2c0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e30a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 0x7f2e2c15c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e30a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c13f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e30a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cb930 con 0x7f2e2c143810 2024-01-16T15:12:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e30a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cbc10 con 0x7f2e2c071e40 2024-01-16T15:12:59.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e30a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c0cbef0 con 0x7f2e2c15d770 2024-01-16T15:12:59.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 0x7f2e2c15c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 0x7f2e2c15c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57686/0 (socket says 172.21.15.53:57686) 2024-01-16T15:12:59.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 -- 172.21.15.53:0/4017142409 learned_addr learned my addr 172.21.15.53:0/4017142409 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:12:59.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e2a800640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2c071e40 0x7f2e2c0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 -- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 msgr2=0x7f2e2c13f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c13f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 -- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2c071e40 msgr2=0x7f2e2c0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2c071e40 0x7f2e2c0af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e23e00640 1 -- 172.21.15.53:0/4017142409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e140094d0 con 0x7f2e2c143810 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.777+0000 7f2e2a800640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2c071e40 0x7f2e2c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.778+0000 7f2e23e00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 0x7f2e2c15c6b0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400e9d0 tx=0x7f2e1400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.778+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e1400ca60 con 0x7f2e2c143810 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.778+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e1400cbc0 con 0x7f2e2c143810 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.778+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e14013470 con 0x7f2e2c143810 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.778+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24009a70 con 0x7f2e2c143810 2024-01-16T15:12:59.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.778+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c0cc4b0 con 0x7f2e2c143810 2024-01-16T15:12:59.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.779+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c067450 con 0x7f2e2c143810 2024-01-16T15:12:59.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.780+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2e140135d0 con 0x7f2e2c143810 2024-01-16T15:12:59.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.781+0000 7f2e22a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e0406bf20 0x7f2e0406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:12:59.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.781+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2e14088b40 con 0x7f2e2c143810 2024-01-16T15:12:59.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.782+0000 7f2e2a800640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e0406bf20 0x7f2e0406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:12:59.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.783+0000 7f2e2a800640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e0406bf20 0x7f2e0406e3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c000f60 tx=0x7f2e1c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:12:59.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:12:59.786+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2e14055180 con 0x7f2e2c143810 2024-01-16T15:12:59.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.017+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541417", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e2c06ccd0 con 0x7f2e0406bf20 2024-01-16T15:13:00.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.040+0000 7f2e22a00640 1 -- 172.21.15.53:0/4017142409 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f2e1c001d30 con 0x7f2e0406bf20 2024-01-16T15:13:00.017 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:00.017 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:00.018 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.042+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e0406bf20 msgr2=0x7f2e0406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.042+0000 7f2e30a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e0406bf20 0x7f2e0406e3e0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f2e1c000f60 tx=0x7f2e1c009040 comp rx=0 tx=0).stop 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.042+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 msgr2=0x7f2e2c15c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.042+0000 7f2e30a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 0x7f2e2c15c6b0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400e9d0 tx=0x7f2e1400ee90 comp rx=0 tx=0).stop 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.042+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 shutdown_connections 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2e0406bf20 0x7f2e0406e3e0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e2c15d770 0x7f2e2c13f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e2c143810 0x7f2e2c15c6b0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 --2- 172.21.15.53:0/4017142409 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e2c071e40 0x7f2e2c0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 >> 172.21.15.53:0/4017142409 conn(0x7f2e2c06a7d0 msgr2=0x7f2e2c140560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:00.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 shutdown_connections 2024-01-16T15:13:00.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.043+0000 7f2e30a00640 1 -- 172.21.15.53:0/4017142409 wait complete. 2024-01-16T15:13:00.040 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:13:00.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541418 2024-01-16T15:13:00.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.285+0000 7f2ccca00640 1 Processor -- start 2024-01-16T15:13:00.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.286+0000 7f2ccca00640 1 -- start start 2024-01-16T15:13:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.287+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc80adf20 0x7f2cc8070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.287+0000 7f2ccca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2cc815bf10 0x7f2cc80cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.287+0000 7f2ccca00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc815f4e0 con 0x7f2cc80cc980 2024-01-16T15:13:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.287+0000 7f2ccca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc815f970 con 0x7f2cc815bf10 2024-01-16T15:13:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.287+0000 7f2ccca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc80cce10 con 0x7f2cc80adf20 2024-01-16T15:13:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.288+0000 7f2cc7200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2cc80cc980 0x7f2cc8143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49558/0 (socket says 172.21.15.53:49558) 2024-01-16T15:13:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.288+0000 7f2cc7200640 1 -- 172.21.15.53:0/3426998296 learned_addr learned my addr 172.21.15.53:0/3426998296 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.288+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/3426998296 >> v2:172.21.15.146:3300/0 conn(0x7f2cc815bf10 0x7f2cc80cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.288+0000 7f2cc6800640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc80adf20 0x7f2cc8070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.288+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2697050444 0 0) 0x7f2cc815f4e0 con 0x7f2cc80cc980 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.288+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca8003520 con 0x7f2cc80cc980 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135406170 0 0) 0x7f2ca8003520 con 0x7f2cc80cc980 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc80adf20 msgr2=0x7f2cc8070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc80adf20 0x7f2cc8070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 >> v2:172.21.15.146:3300/0 conn(0x7f2cc815bf10 msgr2=0x7f2cc80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 --2- 172.21.15.53:0/3426998296 >> v2:172.21.15.146:3300/0 conn(0x7f2cc815bf10 0x7f2cc80cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc8071130 con 0x7f2cc80cc980 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2cc0001670 con 0x7f2cc80cc980 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 >> v1:172.21.15.53:6789/0 conn(0x7f2cc80cc980 legacy=0x7f2cc8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ca8003970 0x7f2ca8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ca80047c0 0x7f2ca8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca8005180 0x7f2ca800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc815f970 con 0x7f2ca80047c0 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc80cce10 con 0x7f2ca8005180 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ca800bff0 con 0x7f2ca8003970 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc6800640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc80adf20 0x7f2cc8070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.289+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ca80047c0 0x7f2ca8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc7200640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca8005180 0x7f2ca800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc5e00640 1 -- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ca8003970 msgr2=0x7f2ca8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc6800640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ca8003970 0x7f2ca8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ca8003970 0x7f2ca8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc5e00640 1 -- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca8005180 msgr2=0x7f2ca800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca8005180 0x7f2ca800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc5e00640 1 -- 172.21.15.53:0/3426998296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca8003520 con 0x7f2ca80047c0 2024-01-16T15:13:00.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.290+0000 7f2cc6800640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ca8003970 0x7f2ca8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:00.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.291+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ca80047c0 0x7f2ca8004c40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009ac0 tx=0x7f2ca80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1505c19305b884b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:00.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.291+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb002c710 con 0x7f2ca80047c0 2024-01-16T15:13:00.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.291+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb002c870 con 0x7f2ca80047c0 2024-01-16T15:13:00.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.291+0000 7f2cc5400640 1 -- 172.21.15.53:0/3426998296 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb002cc00 con 0x7f2ca80047c0 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2ccca00640 1 -- 172.21.15.53:0/3426998296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ca80047c0 msgr2=0x7f2ca8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2ccca00640 1 --2- 172.21.15.53:0/3426998296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ca80047c0 0x7f2ca8004c40 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009ac0 tx=0x7f2ca80057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2cc7200640 1 -- 172.21.15.53:0/3426998296 reap_dead start 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2ccca00640 1 -- 172.21.15.53:0/3426998296 shutdown_connections 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2ccca00640 1 -- 172.21.15.53:0/3426998296 >> 172.21.15.53:0/3426998296 conn(0x7f2cc806a7d0 msgr2=0x7f2cc806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2ccca00640 1 -- 172.21.15.53:0/3426998296 shutdown_connections 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.292+0000 7f2ccca00640 1 -- 172.21.15.53:0/3426998296 wait complete. 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.294+0000 7f2ccca00640 1 Processor -- start 2024-01-16T15:13:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.294+0000 7f2ccca00640 1 -- start start 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca80047c0 0x7f2cc80ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 0x7f2cc81424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc81f1320 0x7f2cc80aedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2ccca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81f0b40 con 0x7f2cc81f4130 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2ccca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81f48a0 con 0x7f2ca80047c0 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2ccca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc81f4b80 con 0x7f2cc81f1320 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 0x7f2cc81424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2cc5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 0x7f2cc81424d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57698/0 (socket says 172.21.15.53:57698) 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2cc5e00640 1 -- 172.21.15.53:0/681589940 learned_addr learned my addr 172.21.15.53:0/681589940 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.295+0000 7f2cc6800640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca80047c0 0x7f2cc80ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc5e00640 1 -- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc81f1320 msgr2=0x7f2cc80aedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc81f1320 0x7f2cc80aedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc5e00640 1 -- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca80047c0 msgr2=0x7f2cc80ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca80047c0 0x7f2cc80ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc5e00640 1 -- 172.21.15.53:0/681589940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc81f4e60 con 0x7f2cc81f4130 2024-01-16T15:13:00.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc6800640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca80047c0 0x7f2cc80ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cc5e00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 0x7f2cc81424d0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009ac0 tx=0x7f2cb0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003810 con 0x7f2cc81f4130 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb002de90 con 0x7f2cc81f4130 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0003b90 con 0x7f2cc81f4130 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.296+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cc81f0e00 con 0x7f2cc81f4130 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.297+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc8142a10 con 0x7f2cc81f4130 2024-01-16T15:13:00.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.298+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc8067450 con 0x7f2cc81f4130 2024-01-16T15:13:00.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.299+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f2cb0003cf0 con 0x7f2cc81f4130 2024-01-16T15:13:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.299+0000 7f2cbea00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ca006bf50 0x7f2ca006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.300+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2cb00aa5a0 con 0x7f2cc81f4130 2024-01-16T15:13:00.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.300+0000 7f2cc6800640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ca006bf50 0x7f2ca006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.302+0000 7f2cc6800640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ca006bf50 0x7f2ca006e410 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2cc815b0a0 tx=0x7f2cb800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:00.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.304+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2cb0076f80 con 0x7f2cc81f4130 2024-01-16T15:13:00.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.540+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541418", "target": ["mon-mgr", ""]}) v1 -- 0x7f2cc815ee90 con 0x7f2ca006bf50 2024-01-16T15:13:00.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.560+0000 7f2cbea00640 1 -- 172.21.15.53:0/681589940 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f2cb8009c50 con 0x7f2ca006bf50 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:00.538 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ca006bf50 msgr2=0x7f2ca006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ca006bf50 0x7f2ca006e410 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2cc815b0a0 tx=0x7f2cb800b040 comp rx=0 tx=0).stop 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 msgr2=0x7f2cc81424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 0x7f2cc81424d0 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f2cb0009ac0 tx=0x7f2cb0004730 comp rx=0 tx=0).stop 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 shutdown_connections 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ca006bf50 0x7f2ca006e410 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2cc81f1320 0x7f2cc80aedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2cc81f4130 0x7f2cc81424d0 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 --2- 172.21.15.53:0/681589940 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ca80047c0 0x7f2cc80ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 >> 172.21.15.53:0/681589940 conn(0x7f2cc806a7d0 msgr2=0x7f2cc80cbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 shutdown_connections 2024-01-16T15:13:00.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.562+0000 7f2ccca00640 1 -- 172.21.15.53:0/681589940 wait complete. 2024-01-16T15:13:00.556 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:13:00.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541419 2024-01-16T15:13:00.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.794+0000 7ff5ae400640 1 Processor -- start 2024-01-16T15:13:00.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.794+0000 7ff5ae400640 1 -- start start 2024-01-16T15:13:00.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.796+0000 7ff5ae400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 0x7ff5a815bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.796+0000 7ff5ae400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a815f4e0 con 0x7ff5a80ccb50 2024-01-16T15:13:00.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.796+0000 7ff5ae400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a8070ee0 con 0x7ff5a80adf20 2024-01-16T15:13:00.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.796+0000 7ff5ae400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a815c450 con 0x7ff5a8070a30 2024-01-16T15:13:00.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.796+0000 7ff5a7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff5a80ccb50 0x7ff5a8143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49562/0 (socket says 172.21.15.53:49562) 2024-01-16T15:13:00.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.796+0000 7ff5a7e00640 1 -- 172.21.15.53:0/1404312129 learned_addr learned my addr 172.21.15.53:0/1404312129 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5ada00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 0x7ff5a815bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 549697490 0 0) 0x7ff5a815f4e0 con 0x7ff5a80ccb50 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff588003520 con 0x7ff5a80ccb50 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2888674481 0 0) 0x7ff588003520 con 0x7ff5a80ccb50 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 msgr2=0x7ff5a815bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 0x7ff5a815bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 >> v1:172.21.15.146:6789/0 conn(0x7ff5a80adf20 legacy=0x7ff5a80cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a8071130 con 0x7ff5a80ccb50 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7ff590001670 con 0x7ff5a80ccb50 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 >> v1:172.21.15.53:6789/0 conn(0x7ff5a80ccb50 legacy=0x7ff5a8143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 0x7ff5880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff588003f70 0x7ff5880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff588005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a815c450 con 0x7ff588004d00 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58800bb70 con 0x7ff588003f70 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58800bd30 con 0x7ff5a8070a30 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.797+0000 7ff5ada00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff588005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5a7e00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff588003f70 0x7ff5880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5ada00640 1 -- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 msgr2=0x7ff5880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5ada00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a8070a30 0x7ff5880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5ada00640 1 -- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff588003f70 msgr2=0x7ff5880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5ada00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff588003f70 0x7ff5880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5ada00640 1 -- 172.21.15.53:0/1404312129 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff588003520 con 0x7ff588004d00 2024-01-16T15:13:00.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.798+0000 7ff5a7e00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff588003f70 0x7ff5880047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.799+0000 7ff5ada00640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff588005180 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009fc0 tx=0x7ff5880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ab1f128a5dc9b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.799+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a002cc60 con 0x7ff588004d00 2024-01-16T15:13:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.799+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a000f040 con 0x7ff588004d00 2024-01-16T15:13:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.799+0000 7ff5a7400640 1 -- 172.21.15.53:0/1404312129 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0034440 con 0x7ff588004d00 2024-01-16T15:13:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ae400640 1 -- 172.21.15.53:0/1404312129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 msgr2=0x7ff588005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ae400640 1 --2- 172.21.15.53:0/1404312129 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff588005180 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009fc0 tx=0x7ff5880057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ada00640 1 -- 172.21.15.53:0/1404312129 reap_dead start 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ae400640 1 -- 172.21.15.53:0/1404312129 shutdown_connections 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ae400640 1 -- 172.21.15.53:0/1404312129 >> 172.21.15.53:0/1404312129 conn(0x7ff5a806a7d0 msgr2=0x7ff5a806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ae400640 1 -- 172.21.15.53:0/1404312129 shutdown_connections 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.800+0000 7ff5ae400640 1 -- 172.21.15.53:0/1404312129 wait complete. 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.802+0000 7ff5ae400640 1 Processor -- start 2024-01-16T15:13:00.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.802+0000 7ff5ae400640 1 -- start start 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ae400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff5a815a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ae400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a80c3d00 0x7ff5a81428c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ae400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5a8134f70 0x7ff5a8135420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ae400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c4da0 con 0x7ff588004d00 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ae400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c4190 con 0x7ff5a8134f70 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ae400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c4470 con 0x7ff5a80c3d00 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ad000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff5a815a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ad000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff5a815a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57724/0 (socket says 172.21.15.53:57724) 2024-01-16T15:13:00.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.803+0000 7ff5ad000640 1 -- 172.21.15.53:0/920458947 learned_addr learned my addr 172.21.15.53:0/920458947 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:00.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5a7e00640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a80c3d00 0x7ff5a81428c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ad000640 1 -- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a80c3d00 msgr2=0x7ff5a81428c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ada00640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5a8134f70 0x7ff5a8135420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ad000640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a80c3d00 0x7ff5a81428c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ad000640 1 -- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5a8134f70 msgr2=0x7ff5a8135420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ad000640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5a8134f70 0x7ff5a8135420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ad000640 1 -- 172.21.15.53:0/920458947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a80c4750 con 0x7ff588004d00 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5a7e00640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a80c3d00 0x7ff5a81428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ad000640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff5a815a810 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff59800c450 tx=0x7ff59800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5980052d0 con 0x7ff588004d00 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff598005430 con 0x7ff588004d00 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a80c4a30 con 0x7ff588004d00 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598012870 con 0x7ff588004d00 2024-01-16T15:13:00.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.804+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a8142e00 con 0x7ff588004d00 2024-01-16T15:13:00.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.806+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a815af30 con 0x7ff588004d00 2024-01-16T15:13:00.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.807+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff5980129d0 con 0x7ff588004d00 2024-01-16T15:13:00.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.808+0000 7ff5a4c00640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff58006bf50 0x7ff58006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:00.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.808+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff598088100 con 0x7ff588004d00 2024-01-16T15:13:00.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.808+0000 7ff5a7e00640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff58006bf50 0x7ff58006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:00.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.810+0000 7ff5a7e00640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff58006bf50 0x7ff58006e410 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff590001eb0 tx=0x7ff590009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:00.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:00.812+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff5a815af30 con 0x7ff588004d00 2024-01-16T15:13:01.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.033+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541419", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5a8143020 con 0x7ff58006bf50 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.055+0000 7ff5a4c00640 1 -- 172.21.15.53:0/920458947 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7ff59000d040 con 0x7ff58006bf50 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:13:01.033 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:01.034 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff58006bf50 msgr2=0x7ff58006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff58006bf50 0x7ff58006e410 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff590001eb0 tx=0x7ff590009040 comp rx=0 tx=0).stop 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 msgr2=0x7ff5a815a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff5a815a810 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7ff59800c450 tx=0x7ff59800c910 comp rx=0 tx=0).stop 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 shutdown_connections 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff58006bf50 0x7ff58006e410 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff5a8134f70 0x7ff5a8135420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff5a80c3d00 0x7ff5a81428c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 --2- 172.21.15.53:0/920458947 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff588004d00 0x7ff5a815a810 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 >> 172.21.15.53:0/920458947 conn(0x7ff5a806a7d0 msgr2=0x7ff5a80ac8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:01.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 shutdown_connections 2024-01-16T15:13:01.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.058+0000 7ff5ae400640 1 -- 172.21.15.53:0/920458947 wait complete. 2024-01-16T15:13:01.055 INFO:tasks.cephfs.test_volumes:expected clone state = ('canceled', 'complete') 2024-01-16T15:13:01.055 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone541420 2024-01-16T15:13:01.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.287+0000 7f32ef200640 1 Processor -- start 2024-01-16T15:13:01.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.287+0000 7f32ef200640 1 -- start start 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ef200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f32f00adf20 0x7f32f0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ef200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f015f4e0 con 0x7f32f00ccb50 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ef200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f0071130 con 0x7f32f015bf10 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ef200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f015f970 con 0x7f32f00adf20 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ede00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f32f00adf20 0x7f32f0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ee800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f32f00ccb50 0x7f32f0143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49578/0 (socket says 172.21.15.53:49578) 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ede00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f32f00adf20 0x7f32f0070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57732/0 (socket says 172.21.15.53:57732) 2024-01-16T15:13:01.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.289+0000 7f32ee800640 1 -- 172.21.15.53:0/793448804 learned_addr learned my addr 172.21.15.53:0/793448804 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:01.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.290+0000 7f32ede00640 1 -- 172.21.15.53:0/793448804 >> v1:172.21.15.146:6789/0 conn(0x7f32f015bf10 legacy=0x7f32f00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:01.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.290+0000 7f32ede00640 1 -- 172.21.15.53:0/793448804 >> v1:172.21.15.53:6789/0 conn(0x7f32f00ccb50 legacy=0x7f32f0143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.290+0000 7f32ede00640 1 -- 172.21.15.53:0/793448804 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f015c390 con 0x7f32f00adf20 2024-01-16T15:13:01.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.290+0000 7f32ed400640 1 -- 172.21.15.53:0/793448804 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130179328 0 0) 0x7f32f015f4e0 con 0x7f32f00ccb50 2024-01-16T15:13:01.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.291+0000 7f32ede00640 1 --2- 172.21.15.53:0/793448804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00adf20 0x7f32f0070a30 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f32e00097a0 tx=0x7f32e002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986234e4cd76e10c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:01.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.291+0000 7f32ed400640 1 -- 172.21.15.53:0/793448804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e002e070 con 0x7f32f00adf20 2024-01-16T15:13:01.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.291+0000 7f32ed400640 1 -- 172.21.15.53:0/793448804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e00315a0 con 0x7f32f00adf20 2024-01-16T15:13:01.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.291+0000 7f32ed400640 1 -- 172.21.15.53:0/793448804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0031930 con 0x7f32f00adf20 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 -- 172.21.15.53:0/793448804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00adf20 msgr2=0x7f32f0070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 --2- 172.21.15.53:0/793448804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00adf20 0x7f32f0070a30 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f32e00097a0 tx=0x7f32e002b840 comp rx=0 tx=0).stop 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 -- 172.21.15.53:0/793448804 shutdown_connections 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 --2- 172.21.15.53:0/793448804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00adf20 0x7f32f0070a30 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 -- 172.21.15.53:0/793448804 >> 172.21.15.53:0/793448804 conn(0x7f32f006a7d0 msgr2=0x7f32f006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 -- 172.21.15.53:0/793448804 shutdown_connections 2024-01-16T15:13:01.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.292+0000 7f32ef200640 1 -- 172.21.15.53:0/793448804 wait complete. 2024-01-16T15:13:01.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.294+0000 7f32ef200640 1 Processor -- start 2024-01-16T15:13:01.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.294+0000 7f32ef200640 1 -- start start 2024-01-16T15:13:01.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32ef200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f00adf20 0x7f32f00b3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32ef200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 0x7f32f00c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32ef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015bf10 0x7f32f00c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32ef200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00b2ef0 con 0x7f32f00ccb50 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32ef200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00b22e0 con 0x7f32f015bf10 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32ef200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f00b25c0 con 0x7f32f00adf20 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 0x7f32f00c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 0x7f32f00c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57744/0 (socket says 172.21.15.53:57744) 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.295+0000 7f32e7e00640 1 -- 172.21.15.53:0/4251146805 learned_addr learned my addr 172.21.15.53:0/4251146805 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32ee800640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015bf10 0x7f32f00c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32ede00640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f00adf20 0x7f32f00b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32e7e00640 1 -- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f00adf20 msgr2=0x7f32f00b3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32e7e00640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f00adf20 0x7f32f00b3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32e7e00640 1 -- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015bf10 msgr2=0x7f32f00c6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32e7e00640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015bf10 0x7f32f00c6b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32e7e00640 1 -- 172.21.15.53:0/4251146805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009400 con 0x7f32f00ccb50 2024-01-16T15:13:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.296+0000 7f32e7e00640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 0x7f32f00c65d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f32d800b3f0 tx=0x7f32d800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:01.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.297+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8011830 con 0x7f32f00ccb50 2024-01-16T15:13:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.297+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8011990 con 0x7f32f00ccb50 2024-01-16T15:13:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.297+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8011d40 con 0x7f32f00ccb50 2024-01-16T15:13:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.297+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f00b28a0 con 0x7f32f00ccb50 2024-01-16T15:13:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.297+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f00c40d0 con 0x7f32f00ccb50 2024-01-16T15:13:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.298+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f0067450 con 0x7f32f00ccb50 2024-01-16T15:13:01.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.299+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f32d801a400 con 0x7f32f00ccb50 2024-01-16T15:13:01.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.301+0000 7f32e6000640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806bf20 0x7f32c806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.301+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f32d80886c0 con 0x7f32f00ccb50 2024-01-16T15:13:01.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.301+0000 7f32ede00640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806bf20 0x7f32c806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.302+0000 7f32ede00640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806bf20 0x7f32c806e3e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f32e0036000 tx=0x7f32e003d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:01.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.305+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f32d8054d00 con 0x7f32f00ccb50 2024-01-16T15:13:01.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.542+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone541420", "target": ["mon-mgr", ""]}) v1 -- 0x7f32f015f170 con 0x7f32c806bf20 2024-01-16T15:13:01.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.564+0000 7f32e6000640 1 -- 172.21.15.53:0/4251146805 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f32e0031b30 con 0x7f32c806bf20 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:13:01.542 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:13:01.543 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:01.543 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:01.543 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:01.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.566+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806bf20 msgr2=0x7f32c806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.566+0000 7f32ef200640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806bf20 0x7f32c806e3e0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f32e0036000 tx=0x7f32e003d040 comp rx=0 tx=0).stop 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.566+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 msgr2=0x7f32f00c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.566+0000 7f32ef200640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 0x7f32f00c65d0 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f32d800b3f0 tx=0x7f32d800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 shutdown_connections 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f32c806bf20 0x7f32c806e3e0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f32f015bf10 0x7f32f00c6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f32f00ccb50 0x7f32f00c65d0 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 --2- 172.21.15.53:0/4251146805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f32f00adf20 0x7f32f00b3380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 >> 172.21.15.53:0/4251146805 conn(0x7f32f006a7d0 msgr2=0x7f32f00ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 shutdown_connections 2024-01-16T15:13:01.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.567+0000 7f32ef200640 1 -- 172.21.15.53:0/4251146805 wait complete. 2024-01-16T15:13:01.564 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:13:01.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.805+0000 7fcd8be00640 1 Processor -- start 2024-01-16T15:13:01.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.806+0000 7fcd8be00640 1 -- start start 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c143c40 0x7fcd8c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd8c0ccc10 0x7fcd8c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c070e50 con 0x7fcd8c0717d0 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8c0723c0 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c144020 con 0x7fcd8c143c40 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd8c0ccc10 0x7fcd8c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd83e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcd8c0717d0 0x7fcd8c0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:49584/0 (socket says 172.21.15.53:49584) 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd83e00640 1 -- 172.21.15.53:0/3745489157 learned_addr learned my addr 172.21.15.53:0/3745489157 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:01.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.807+0000 7fcd8b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcd8c0ccc10 0x7fcd8c06d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57758/0 (socket says 172.21.15.53:57758) 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.808+0000 7fcd8aa00640 1 --2- 172.21.15.53:0/3745489157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c143c40 0x7fcd8c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.808+0000 7fcd8b400640 1 -- 172.21.15.53:0/3745489157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c143c40 msgr2=0x7fcd8c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.808+0000 7fcd8b400640 1 --2- 172.21.15.53:0/3745489157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c143c40 0x7fcd8c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.808+0000 7fcd8b400640 1 -- 172.21.15.53:0/3745489157 >> v1:172.21.15.53:6789/0 conn(0x7fcd8c0717d0 legacy=0x7fcd8c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.808+0000 7fcd8b400640 1 -- 172.21.15.53:0/3745489157 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c071e00 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.808+0000 7fcd8a000640 1 -- 172.21.15.53:0/3745489157 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945339151 0 0) 0x7fcd8c070e50 con 0x7fcd8c0717d0 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.809+0000 7fcd8b400640 1 --2- 172.21.15.53:0/3745489157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c06d160 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009d90 tx=0x7fcd8402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe245c2f91eb776 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:01.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.809+0000 7fcd8a000640 1 -- 172.21.15.53:0/3745489157 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8402cbc0 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.809+0000 7fcd8a000640 1 -- 172.21.15.53:0/3745489157 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8402cd20 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.809+0000 7fcd8a000640 1 -- 172.21.15.53:0/3745489157 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84034660 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 -- 172.21.15.53:0/3745489157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 msgr2=0x7fcd8c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 --2- 172.21.15.53:0/3745489157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c06d160 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fcd84009d90 tx=0x7fcd8402be10 comp rx=0 tx=0).stop 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 -- 172.21.15.53:0/3745489157 shutdown_connections 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 --2- 172.21.15.53:0/3745489157 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c06d160 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 --2- 172.21.15.53:0/3745489157 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c143c40 0x7fcd8c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 -- 172.21.15.53:0/3745489157 >> 172.21.15.53:0/3745489157 conn(0x7fcd8c06a800 msgr2=0x7fcd8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 -- 172.21.15.53:0/3745489157 shutdown_connections 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.810+0000 7fcd8be00640 1 -- 172.21.15.53:0/3745489157 wait complete. 2024-01-16T15:13:01.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.812+0000 7fcd8be00640 1 Processor -- start 2024-01-16T15:13:01.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.812+0000 7fcd8be00640 1 -- start start 2024-01-16T15:13:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.813+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0717d0 0x7fcd8c13e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.813+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c0c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.813+0000 7fcd8be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c143c40 0x7fcd8c0c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.813+0000 7fcd8be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0c4cd0 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.813+0000 7fcd8be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0c7f30 con 0x7fcd8c143c40 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.813+0000 7fcd8be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0ca380 con 0x7fcd8c0717d0 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c0c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c0c79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57768/0 (socket says 172.21.15.53:57768) 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 -- 172.21.15.53:0/2117789163 learned_addr learned my addr 172.21.15.53:0/2117789163 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd8aa00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0717d0 0x7fcd8c13e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd8b400640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c143c40 0x7fcd8c0c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 -- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0717d0 msgr2=0x7fcd8c13e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0717d0 0x7fcd8c13e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 -- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c143c40 msgr2=0x7fcd8c0c9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c143c40 0x7fcd8c0c9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.814+0000 7fcd83e00640 1 -- 172.21.15.53:0/2117789163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd84009a70 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.815+0000 7fcd83e00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c0c79f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c500 tx=0x7fcd7400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.815+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74007560 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.815+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd740047e0 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.815+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c0ca600 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.815+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd74004b90 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.815+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c0cabc0 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.817+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c0c4e40 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.817+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fcd74004cf0 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.818+0000 7fcd83400640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf20 0x7fcd6406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.818+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcd74087c40 con 0x7fcd8c0ccc10 2024-01-16T15:13:01.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.819+0000 7fcd8aa00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf20 0x7fcd6406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:01.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.820+0000 7fcd8aa00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf20 0x7fcd6406e3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c1f6320 tx=0x7fcd7c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:01.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:01.822+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd8c0c4e40 con 0x7fcd8c0ccc10 2024-01-16T15:13:02.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.041+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fcd8c06cc40 con 0x7fcd6406bf20 2024-01-16T15:13:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.045+0000 7fcd83400640 1 -- 172.21.15.53:0/2117789163 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fcd7c0108d0 con 0x7fcd6406bf20 2024-01-16T15:13:02.023 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:13:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:13:02.023 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:02.023 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:13:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf20 msgr2=0x7fcd6406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf20 0x7fcd6406e3e0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c1f6320 tx=0x7fcd7c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 msgr2=0x7fcd8c0c79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c0c79f0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fcd7400c500 tx=0x7fcd7400c9c0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 shutdown_connections 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf20 0x7fcd6406e3e0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd8c143c40 0x7fcd8c0c9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c0ccc10 0x7fcd8c0c79f0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 --2- 172.21.15.53:0/2117789163 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0717d0 0x7fcd8c13e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.047+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 >> 172.21.15.53:0/2117789163 conn(0x7fcd8c06a800 msgr2=0x7fcd8c0cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.048+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 shutdown_connections 2024-01-16T15:13:02.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.048+0000 7fcd8be00640 1 -- 172.21.15.53:0/2117789163 wait complete. 2024-01-16T15:13:02.045 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:13:02.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.293+0000 7f7f49e00640 1 Processor -- start 2024-01-16T15:13:02.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.293+0000 7f7f49e00640 1 -- start start 2024-01-16T15:13:02.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7f441505b0 0x7f7f440b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 0x7f7f4406cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f4406d930 con 0x7f7f4406d500 2024-01-16T15:13:02.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f440b1e30 con 0x7f7f441505b0 2024-01-16T15:13:02.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440cdc10 con 0x7f7f4415a690 2024-01-16T15:13:02.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7f4406d500 0x7f7f4415d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43936/0 (socket says 172.21.15.53:43936) 2024-01-16T15:13:02.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f49400640 1 -- 172.21.15.53:0/4189986759 learned_addr learned my addr 172.21.15.53:0/4189986759 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.295+0000 7f7f48a00640 1 --2- 172.21.15.53:0/4189986759 >> v2:172.21.15.146:3300/0 conn(0x7f7f441505b0 0x7f7f440b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f3be00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 0x7f7f4406cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076909676 0 0) 0x7f7f4406d930 con 0x7f7f4406d500 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f24003520 con 0x7f7f4406d500 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3263405437 0 0) 0x7f7f24003520 con 0x7f7f4406d500 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 msgr2=0x7f7f4406cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 0x7f7f4406cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 >> v2:172.21.15.146:3300/0 conn(0x7f7f441505b0 msgr2=0x7f7f440b3b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 --2- 172.21.15.53:0/4189986759 >> v2:172.21.15.146:3300/0 conn(0x7f7f441505b0 0x7f7f440b3b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f440b20a0 con 0x7f7f4406d500 2024-01-16T15:13:02.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7f3c001a10 con 0x7f7f4406d500 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 >> v1:172.21.15.53:6789/0 conn(0x7f7f4406d500 legacy=0x7f7f4415d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f24003970 0x7f7f24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 0x7f7f24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440b1e30 con 0x7f7f240047c0 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f440cdc10 con 0x7f7f24003970 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.296+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f2400b980 con 0x7f7f4415a690 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f3be00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f48a00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f24003970 0x7f7f24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f49400640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 0x7f7f24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f3be00640 1 -- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 msgr2=0x7f7f24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f3be00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f4415a690 0x7f7f24005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f3be00640 1 -- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f24003970 msgr2=0x7f7f24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f3be00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f24003970 0x7f7f24003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.297+0000 7f7f3be00640 1 -- 172.21.15.53:0/4189986759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f24003520 con 0x7f7f240047c0 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.298+0000 7f7f3be00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f24004c40 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009ac0 tx=0x7f7f240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72286b5e2ea1e083 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.298+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c02c6b0 con 0x7f7f240047c0 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.298+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f2c02c810 con 0x7f7f240047c0 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.298+0000 7f7f43e00640 1 -- 172.21.15.53:0/4189986759 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c02cba0 con 0x7f7f240047c0 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49e00640 1 -- 172.21.15.53:0/4189986759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 msgr2=0x7f7f24004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49e00640 1 --2- 172.21.15.53:0/4189986759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f24004c40 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009ac0 tx=0x7f7f240057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49400640 1 -- 172.21.15.53:0/4189986759 reap_dead start 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49e00640 1 -- 172.21.15.53:0/4189986759 shutdown_connections 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49e00640 1 -- 172.21.15.53:0/4189986759 >> 172.21.15.53:0/4189986759 conn(0x7f7f4406a7d0 msgr2=0x7f7f4406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49e00640 1 -- 172.21.15.53:0/4189986759 shutdown_connections 2024-01-16T15:13:02.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.299+0000 7f7f49e00640 1 -- 172.21.15.53:0/4189986759 wait complete. 2024-01-16T15:13:02.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.301+0000 7f7f49e00640 1 Processor -- start 2024-01-16T15:13:02.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.301+0000 7f7f49e00640 1 -- start start 2024-01-16T15:13:02.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.302+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f44146280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.302+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f44149ba0 0x7f7f44149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.302+0000 7f7f49e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f4414daa0 0x7f7f4414df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.302+0000 7f7f49e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f441e4b60 con 0x7f7f240047c0 2024-01-16T15:13:02.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.302+0000 7f7f49e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f441467c0 con 0x7f7f4414daa0 2024-01-16T15:13:02.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.302+0000 7f7f49e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f4414d610 con 0x7f7f44149ba0 2024-01-16T15:13:02.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f44146280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f44146280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55682/0 (socket says 172.21.15.53:55682) 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 -- 172.21.15.53:0/3732070575 learned_addr learned my addr 172.21.15.53:0/3732070575 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f3be00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f44149ba0 0x7f7f44149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f49400640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f4414daa0 0x7f7f4414df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 -- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f44149ba0 msgr2=0x7f7f44149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f44149ba0 0x7f7f44149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 -- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f4414daa0 msgr2=0x7f7f4414df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f4414daa0 0x7f7f4414df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.303+0000 7f7f48a00640 1 -- 172.21.15.53:0/3732070575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f4414a520 con 0x7f7f240047c0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.304+0000 7f7f48a00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f44146280 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f7f34010a20 tx=0x7f7f34010ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.304+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f3400ec60 con 0x7f7f240047c0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.304+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f3400edc0 con 0x7f7f240047c0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.304+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f440ae760 con 0x7f7f240047c0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.304+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f34012630 con 0x7f7f240047c0 2024-01-16T15:13:02.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.304+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f4414cfc0 con 0x7f7f240047c0 2024-01-16T15:13:02.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.305+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f44067450 con 0x7f7f240047c0 2024-01-16T15:13:02.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.306+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f7f34012790 con 0x7f7f240047c0 2024-01-16T15:13:02.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.307+0000 7f7f41600640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f1c06beb0 0x7f7f1c06e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.307+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7f34016070 con 0x7f7f240047c0 2024-01-16T15:13:02.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.308+0000 7f7f3be00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f1c06beb0 0x7f7f1c06e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.309+0000 7f7f3be00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f1c06beb0 0x7f7f1c06e370 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c036000 tx=0x7f7f2c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:02.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.311+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7f34055760 con 0x7f7f240047c0 2024-01-16T15:13:02.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.540+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7f4415aaf0 con 0x7f7f1c06beb0 2024-01-16T15:13:02.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.579+0000 7f7f41600640 1 -- 172.21.15.53:0/3732070575 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1029 (secure 0 0 0) 0x7f7f2c02c980 con 0x7f7f1c06beb0 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541414" 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54147" 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.557 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54141" 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54142" 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541412" 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541415" 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541417" 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541420" 2024-01-16T15:13:02.558 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541418" 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541416" 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541410" 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54143" 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.559 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541411" 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54148" 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54144" 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541413" 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54146" 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.560 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone541419" 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54145" 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone54149" 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:02.561 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f1c06beb0 msgr2=0x7f7f1c06e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f1c06beb0 0x7f7f1c06e370 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c036000 tx=0x7f7f2c030000 comp rx=0 tx=0).stop 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 msgr2=0x7f7f44146280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f44146280 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7f7f34010a20 tx=0x7f7f34010ee0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 shutdown_connections 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7f1c06beb0 0x7f7f1c06e370 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7f4414daa0 0x7f7f4414df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7f44149ba0 0x7f7f44149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 --2- 172.21.15.53:0/3732070575 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7f240047c0 0x7f7f44146280 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 >> 172.21.15.53:0/3732070575 conn(0x7f7f4406a7d0 msgr2=0x7f7f4415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:02.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 shutdown_connections 2024-01-16T15:13:02.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.582+0000 7f7f49e00640 1 -- 172.21.15.53:0/3732070575 wait complete. 2024-01-16T15:13:02.579 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541414 2024-01-16T15:13:02.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.824+0000 7f0e01600640 1 Processor -- start 2024-01-16T15:13:02.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.824+0000 7f0e01600640 1 -- start start 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0e01600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0dfc159590 0x7f0dfc070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0e01600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 0x7f0dfc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0e01600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dfc06d9e0 con 0x7f0dfc0b3c20 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0e01600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dfc071790 con 0x7f0dfc159590 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0e01600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc071100 con 0x7f0dfc06d050 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0df3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0dfc0b3c20 0x7f0dfc0ccce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43944/0 (socket says 172.21.15.53:43944) 2024-01-16T15:13:02.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.826+0000 7f0df3e00640 1 -- 172.21.15.53:0/1142872244 learned_addr learned my addr 172.21.15.53:0/1142872244 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:02.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfbe00640 1 --2- 172.21.15.53:0/1142872244 >> v2:172.21.15.146:3300/0 conn(0x7f0dfc159590 0x7f0dfc070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0e00c00640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 0x7f0dfc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329352467 0 0) 0x7f0de4001110 con 0x7f0dfc0b3c20 2024-01-16T15:13:02.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc005730 con 0x7f0dfc0b3c20 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3554920061 0 0) 0x7f0ddc005730 con 0x7f0dfc0b3c20 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 msgr2=0x7f0dfc06d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 0x7f0dfc06d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 >> v2:172.21.15.146:3300/0 conn(0x7f0dfc159590 msgr2=0x7f0dfc070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 --2- 172.21.15.53:0/1142872244 >> v2:172.21.15.146:3300/0 conn(0x7f0dfc159590 0x7f0dfc070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc070860 con 0x7f0dfc0b3c20 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.827+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0de4001820 con 0x7f0dfc0b3c20 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 >> v1:172.21.15.53:6789/0 conn(0x7f0dfc0b3c20 legacy=0x7f0dfc0ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 0x7f0ddc005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006150 0x7f0ddc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc006f10 0x7f0ddc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc071790 con 0x7f0ddc006f10 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc071100 con 0x7f0ddc006150 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ddc00dc40 con 0x7f0dfc06d050 2024-01-16T15:13:02.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0e00c00640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc006f10 0x7f0ddc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0df3e00640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006150 0x7f0ddc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0e00c00640 1 -- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 msgr2=0x7f0ddc005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0e00c00640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc06d050 0x7f0ddc005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0e00c00640 1 -- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006150 msgr2=0x7f0ddc0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:02.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0e00c00640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006150 0x7f0ddc0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.828+0000 7f0e00c00640 1 -- 172.21.15.53:0/1142872244 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc005730 con 0x7f0ddc006f10 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0e00c00640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc006f10 0x7f0ddc007390 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009fc0 tx=0x7f0ddc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=336ef61325ff928c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df402cc60 con 0x7f0ddc006f10 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0df400f040 con 0x7f0ddc006f10 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0dfb400640 1 -- 172.21.15.53:0/1142872244 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4034440 con 0x7f0ddc006f10 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0e01600640 1 -- 172.21.15.53:0/1142872244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc006f10 msgr2=0x7f0ddc007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0e01600640 1 --2- 172.21.15.53:0/1142872244 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0ddc006f10 0x7f0ddc007390 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009fc0 tx=0x7f0ddc007a00 comp rx=0 tx=0).stop 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.830+0000 7f0e00c00640 1 -- 172.21.15.53:0/1142872244 reap_dead start 2024-01-16T15:13:02.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.831+0000 7f0e01600640 1 -- 172.21.15.53:0/1142872244 shutdown_connections 2024-01-16T15:13:02.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.831+0000 7f0e01600640 1 -- 172.21.15.53:0/1142872244 >> 172.21.15.53:0/1142872244 conn(0x7f0dfc06a800 msgr2=0x7f0dfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:02.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.831+0000 7f0e01600640 1 -- 172.21.15.53:0/1142872244 shutdown_connections 2024-01-16T15:13:02.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.831+0000 7f0e01600640 1 -- 172.21.15.53:0/1142872244 wait complete. 2024-01-16T15:13:02.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.833+0000 7f0e01600640 1 Processor -- start 2024-01-16T15:13:02.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.833+0000 7f0e01600640 1 -- start start 2024-01-16T15:13:02.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e01600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006f10 0x7f0dfc15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e01600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 0x7f0dfc1eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e01600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc14f2f0 0x7f0dfc154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e01600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc1ef140 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e01600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc155950 con 0x7f0ddc006f10 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e01600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dfc157410 con 0x7f0dfc14f2f0 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 0x7f0dfc1eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 0x7f0dfc1eec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55688/0 (socket says 172.21.15.53:55688) 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 -- 172.21.15.53:0/1287045821 learned_addr learned my addr 172.21.15.53:0/1287045821 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0dfbe00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006f10 0x7f0dfc15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0e00c00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc14f2f0 0x7f0dfc154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 -- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc14f2f0 msgr2=0x7f0dfc154680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc14f2f0 0x7f0dfc154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 -- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006f10 msgr2=0x7f0dfc15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.834+0000 7f0df3e00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006f10 0x7f0dfc15c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0df3e00640 1 -- 172.21.15.53:0/1287045821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dfc155390 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0dfbe00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006f10 0x7f0dfc15c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:02.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0df3e00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 0x7f0dfc1eec00 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f0de400e840 tx=0x7f0de400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:02.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de400c8d0 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0de400ca30 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0de400cdc0 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dfc155670 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.835+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dfc154bc0 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.836+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dfc15a5f0 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.838+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0de401d460 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.838+0000 7f0df8c00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0dd406bf20 0x7f0dd406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:02.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.838+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0de4088430 con 0x7f0dfc14fbd0 2024-01-16T15:13:02.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.839+0000 7f0dfbe00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0dd406bf20 0x7f0dd406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:02.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.840+0000 7f0dfbe00640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0dd406bf20 0x7f0dd406e3e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009a40 tx=0x7f0dec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:02.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:02.843+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0de40bde90 con 0x7f0dfc14fbd0 2024-01-16T15:13:03.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.069+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541414", "target": ["mon-mgr", ""]}) v1 -- 0x7f0dfc0b1f30 con 0x7f0dd406bf20 2024-01-16T15:13:03.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.124+0000 7f0df8c00640 1 -- 172.21.15.53:0/1287045821 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0dec004020 con 0x7f0dd406bf20 2024-01-16T15:13:03.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0dd406bf20 msgr2=0x7f0dd406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0dd406bf20 0x7f0dd406e3e0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f0dec009a40 tx=0x7f0dec008040 comp rx=0 tx=0).stop 2024-01-16T15:13:03.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 msgr2=0x7f0dfc1eec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 0x7f0dfc1eec00 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7f0de400e840 tx=0x7f0de400ed00 comp rx=0 tx=0).stop 2024-01-16T15:13:03.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 shutdown_connections 2024-01-16T15:13:03.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0dd406bf20 0x7f0dd406e3e0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0dfc14f2f0 0x7f0dfc154680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0dfc14fbd0 0x7f0dfc1eec00 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 --2- 172.21.15.53:0/1287045821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0ddc006f10 0x7f0dfc15c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 >> 172.21.15.53:0/1287045821 conn(0x7f0dfc06a800 msgr2=0x7f0dfc150770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:03.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 shutdown_connections 2024-01-16T15:13:03.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.128+0000 7f0e01600640 1 -- 172.21.15.53:0/1287045821 wait complete. 2024-01-16T15:13:03.112 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.133+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.112 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.133+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.112 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.133+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.112 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.133+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.112 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.133+0000 7f6138200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:13:03.135 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.156+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.135 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.157+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.135 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.157+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.135 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.157+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.135 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:03.157+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:03.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.372+0000 7fe3a3400640 1 Processor -- start 2024-01-16T15:13:03.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.372+0000 7fe3a3400640 1 -- start start 2024-01-16T15:13:03.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe39c13cfa0 0x7fe39c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c0cd530 0x7fe39c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a3400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe39c0cdec0 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a3400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe39c0ce1a0 con 0x7fe39c13cfa0 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a3400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c0ce480 con 0x7fe39c0cd530 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a2000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe39c13cfa0 0x7fe39c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a2a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe39c0cd9b0 0x7fe39c15d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43954/0 (socket says 172.21.15.53:43954) 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.374+0000 7fe3a2a00640 1 -- 172.21.15.53:0/2681757108 learned_addr learned my addr 172.21.15.53:0/2681757108 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a1600640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c0cd530 0x7fe39c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a0c00640 1 -- 172.21.15.53:0/2681757108 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295344744 0 0) 0x7fe39c0cdec0 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a0c00640 1 -- 172.21.15.53:0/2681757108 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe37c003520 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a2000640 1 -- 172.21.15.53:0/2681757108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c0cd530 msgr2=0x7fe39c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a2000640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c0cd530 0x7fe39c13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a2000640 1 -- 172.21.15.53:0/2681757108 >> v1:172.21.15.53:6789/0 conn(0x7fe39c0cd9b0 legacy=0x7fe39c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a2000640 1 -- 172.21.15.53:0/2681757108 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39c0ce760 con 0x7fe39c13cfa0 2024-01-16T15:13:03.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.375+0000 7fe3a1600640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c0cd530 0x7fe39c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:03.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.376+0000 7fe3a2000640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c13cfa0 0x7fe39c15afb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009d90 tx=0x7fe38c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c4fb9a77b21853 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.377+0000 7fe3a0c00640 1 -- 172.21.15.53:0/2681757108 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c02cbc0 con 0x7fe39c13cfa0 2024-01-16T15:13:03.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.377+0000 7fe3a0c00640 1 -- 172.21.15.53:0/2681757108 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38c02cd20 con 0x7fe39c13cfa0 2024-01-16T15:13:03.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.377+0000 7fe3a0c00640 1 -- 172.21.15.53:0/2681757108 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38c034660 con 0x7fe39c13cfa0 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.377+0000 7fe3a3400640 1 -- 172.21.15.53:0/2681757108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c13cfa0 msgr2=0x7fe39c15afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.377+0000 7fe3a3400640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c13cfa0 0x7fe39c15afb0 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7fe38c009d90 tx=0x7fe38c02be10 comp rx=0 tx=0).stop 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.378+0000 7fe3a3400640 1 -- 172.21.15.53:0/2681757108 shutdown_connections 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.378+0000 7fe3a3400640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c0cd530 0x7fe39c13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.378+0000 7fe3a3400640 1 --2- 172.21.15.53:0/2681757108 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c13cfa0 0x7fe39c15afb0 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.378+0000 7fe3a3400640 1 -- 172.21.15.53:0/2681757108 >> 172.21.15.53:0/2681757108 conn(0x7fe39c06a800 msgr2=0x7fe39c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.378+0000 7fe3a3400640 1 -- 172.21.15.53:0/2681757108 shutdown_connections 2024-01-16T15:13:03.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.378+0000 7fe3a3400640 1 -- 172.21.15.53:0/2681757108 wait complete. 2024-01-16T15:13:03.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.379+0000 7fe3a3400640 1 Processor -- start 2024-01-16T15:13:03.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.380+0000 7fe3a3400640 1 -- start start 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.380+0000 7fe3a3400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe39c0cd530 0x7fe39c156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.380+0000 7fe3a3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 0x7fe39c14ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a3400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c13cfa0 0x7fe39c1f1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a3400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c150460 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a3400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c150690 con 0x7fe39c0cd530 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a3400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39c150970 con 0x7fe39c13cfa0 2024-01-16T15:13:03.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 0x7fe39c14ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 0x7fe39c14ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55706/0 (socket says 172.21.15.53:55706) 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 -- 172.21.15.53:0/799603852 learned_addr learned my addr 172.21.15.53:0/799603852 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a2a00640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c13cfa0 0x7fe39c1f1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a2000640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe39c0cd530 0x7fe39c156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 -- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c13cfa0 msgr2=0x7fe39c1f1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c13cfa0 0x7fe39c1f1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 -- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe39c0cd530 msgr2=0x7fe39c156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe39c0cd530 0x7fe39c156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a1600640 1 -- 172.21.15.53:0/799603852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38c009a70 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.381+0000 7fe3a2000640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe39c0cd530 0x7fe39c156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.382+0000 7fe3a1600640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 0x7fe39c14ff20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fe38400e7e0 tx=0x7fe38400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.382+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400c8d0 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.382+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe38400ee90 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.382+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38400cc70 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.382+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39c150c50 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.382+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39c151210 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.383+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe39c151580 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.384+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe384013410 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.385+0000 7fe39a000640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe37406bf20 0x7fe37406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.385+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe384088440 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.386+0000 7fe3a2000640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe37406bf20 0x7fe37406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.387+0000 7fe3a2000640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe37406bf20 0x7fe37406e3e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe38c036000 tx=0x7fe38c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.390+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe384054d00 con 0x7fe39c0cd9b0 2024-01-16T15:13:03.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.618+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe39c067450 con 0x7fe37406bf20 2024-01-16T15:13:03.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.658+0000 7fe39a000640 1 -- 172.21.15.53:0/799603852 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe39c067450 con 0x7fe37406bf20 2024-01-16T15:13:03.636 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:03.636 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:13:03.637 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:03.637 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:13:03.637 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:03.637 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:13:03.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe37406bf20 msgr2=0x7fe37406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe37406bf20 0x7fe37406e3e0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fe38c036000 tx=0x7fe38c030000 comp rx=0 tx=0).stop 2024-01-16T15:13:03.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 msgr2=0x7fe39c14ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 0x7fe39c14ff20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7fe38400e7e0 tx=0x7fe38400eca0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 shutdown_connections 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe37406bf20 0x7fe37406e3e0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.661+0000 7fe3a3400640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe39c13cfa0 0x7fe39c1f1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.662+0000 7fe3a3400640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe39c0cd9b0 0x7fe39c14ff20 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.662+0000 7fe3a3400640 1 --2- 172.21.15.53:0/799603852 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe39c0cd530 0x7fe39c156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.662+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 >> 172.21.15.53:0/799603852 conn(0x7fe39c06a800 msgr2=0x7fe39c1582e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.662+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 shutdown_connections 2024-01-16T15:13:03.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.662+0000 7fe3a3400640 1 -- 172.21.15.53:0/799603852 wait complete. 2024-01-16T15:13:03.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:13:03.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.905+0000 7f98e4800640 1 Processor -- start 2024-01-16T15:13:03.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.905+0000 7f98e4800640 1 -- start start 2024-01-16T15:13:03.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.907+0000 7f98e4800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0d1090 0x7f98dc0d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.907+0000 7f98e4800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f98dc0cf100 0x7f98dc0cf590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.907+0000 7f98e4800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98dc0d1a30 con 0x7f98dc146730 2024-01-16T15:13:03.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.907+0000 7f98e4800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98dc0cb6c0 con 0x7f98dc0cf100 2024-01-16T15:13:03.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.907+0000 7f98e4800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0c6860 con 0x7f98dc0d1090 2024-01-16T15:13:03.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.908+0000 7f98e1e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f98dc146730 0x7f98dc0ce2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43970/0 (socket says 172.21.15.53:43970) 2024-01-16T15:13:03.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.908+0000 7f98e1e00640 1 -- 172.21.15.53:0/2541718476 learned_addr learned my addr 172.21.15.53:0/2541718476 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.908+0000 7f98e1400640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0d1090 0x7f98dc0d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.908+0000 7f98e2800640 1 --2- 172.21.15.53:0/2541718476 >> v2:172.21.15.146:3300/0 conn(0x7f98dc0cf100 0x7f98dc0cf590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.908+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1920521308 0 0) 0x7f98dc0d1a30 con 0x7f98dc146730 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.908+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98bc003520 con 0x7f98dc146730 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1736177086 0 0) 0x7f98bc003520 con 0x7f98dc146730 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0d1090 msgr2=0x7f98dc0d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0d1090 0x7f98dc0d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 >> v2:172.21.15.146:3300/0 conn(0x7f98dc0cf100 msgr2=0x7f98dc0cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 --2- 172.21.15.53:0/2541718476 >> v2:172.21.15.146:3300/0 conn(0x7f98dc0cf100 0x7f98dc0cf590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0ce9e0 con 0x7f98dc146730 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f98cc0019f0 con 0x7f98dc146730 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 >> v1:172.21.15.53:6789/0 conn(0x7f98dc146730 legacy=0x7f98dc0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98bc0047c0 0x7f98bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98bc005180 0x7f98bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc00bec0 con 0x7f98bc003970 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc00c000 con 0x7f98bc0047c0 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98bc00c170 con 0x7f98bc005180 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e1400640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0d1090 0x7f98dc0d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.909+0000 7f98e1e00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1400640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98bc0047c0 0x7f98bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e2800640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98bc005180 0x7f98bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1e00640 1 -- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98bc005180 msgr2=0x7f98bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1e00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98bc005180 0x7f98bc00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1e00640 1 -- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98bc0047c0 msgr2=0x7f98bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1e00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98bc0047c0 0x7f98bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1e00640 1 -- 172.21.15.53:0/2541718476 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc003520 con 0x7f98bc003970 2024-01-16T15:13:03.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.910+0000 7f98e1400640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98bc0047c0 0x7f98bc004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.911+0000 7f98e1e00640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98bc003d90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00a860 tx=0x7f98bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53a14b54362d604f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.911+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc034630 con 0x7f98bc003970 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.911+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc02f070 con 0x7f98bc003970 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.911+0000 7f98e0a00640 1 -- 172.21.15.53:0/2541718476 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc034a70 con 0x7f98bc003970 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e4800640 1 -- 172.21.15.53:0/2541718476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 msgr2=0x7f98bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e4800640 1 --2- 172.21.15.53:0/2541718476 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98bc003d90 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00a860 tx=0x7f98bc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e2800640 1 -- 172.21.15.53:0/2541718476 reap_dead start 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e4800640 1 -- 172.21.15.53:0/2541718476 shutdown_connections 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e4800640 1 -- 172.21.15.53:0/2541718476 >> 172.21.15.53:0/2541718476 conn(0x7f98dc066690 msgr2=0x7f98dc06a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e4800640 1 -- 172.21.15.53:0/2541718476 shutdown_connections 2024-01-16T15:13:03.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.912+0000 7f98e4800640 1 -- 172.21.15.53:0/2541718476 wait complete. 2024-01-16T15:13:03.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.914+0000 7f98e4800640 1 Processor -- start 2024-01-16T15:13:03.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.914+0000 7f98e4800640 1 -- start start 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e4800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98dc0b5990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e4800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0cbd40 0x7f98dc1e6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e4800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1e72a0 0x7f98dc1f6750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e4800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc1e7730 con 0x7f98bc003970 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e4800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0add30 con 0x7f98dc1e72a0 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e4800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc0b5ed0 con 0x7f98dc0cbd40 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98dc0b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98dc0b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55722/0 (socket says 172.21.15.53:55722) 2024-01-16T15:13:03.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.915+0000 7f98e1e00640 1 -- 172.21.15.53:0/3549723904 learned_addr learned my addr 172.21.15.53:0/3549723904 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:03.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1400640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0cbd40 0x7f98dc1e6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e2800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1e72a0 0x7f98dc1f6750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1e00640 1 -- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0cbd40 msgr2=0x7f98dc1e6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1e00640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0cbd40 0x7f98dc1e6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1e00640 1 -- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1e72a0 msgr2=0x7f98dc1f6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1e00640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1e72a0 0x7f98dc1f6750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1e00640 1 -- 172.21.15.53:0/3549723904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc0b4120 con 0x7f98bc003970 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1400640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0cbd40 0x7f98dc1e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e1e00640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98dc0b5990 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00a860 tx=0x7f98cc001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc0019c0 con 0x7f98bc003970 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc034850 con 0x7f98bc003970 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98cc034be0 con 0x7f98bc003970 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.916+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc0b4400 con 0x7f98bc003970 2024-01-16T15:13:03.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.917+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc0b49c0 con 0x7f98bc003970 2024-01-16T15:13:03.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.918+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc074a50 con 0x7f98bc003970 2024-01-16T15:13:03.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.919+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f98cc031030 con 0x7f98bc003970 2024-01-16T15:13:03.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.920+0000 7f98da000640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f98b406bf20 0x7f98b406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:03.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.920+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f98cc0aa9c0 con 0x7f98bc003970 2024-01-16T15:13:03.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.920+0000 7f98e1400640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f98b406bf20 0x7f98b406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:03.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.921+0000 7f98e1400640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f98b406bf20 0x7f98b406e3e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f98c4006f00 tx=0x7f98c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:03.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:03.924+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f98cc07abd0 con 0x7f98bc003970 2024-01-16T15:13:04.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.154+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98dc0c62a0 con 0x7f98b406bf20 2024-01-16T15:13:04.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.235+0000 7f98da000640 1 -- 172.21.15.53:0/3549723904 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f98c400c040 con 0x7f98b406bf20 2024-01-16T15:13:04.213 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:04.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f98b406bf20 msgr2=0x7f98b406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f98b406bf20 0x7f98b406e3e0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f98c4006f00 tx=0x7f98c4008040 comp rx=0 tx=0).stop 2024-01-16T15:13:04.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 msgr2=0x7f98dc0b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98dc0b5990 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f98cc00a860 tx=0x7f98cc001d80 comp rx=0 tx=0).stop 2024-01-16T15:13:04.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 shutdown_connections 2024-01-16T15:13:04.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f98b406bf20 0x7f98b406e3e0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98dc1e72a0 0x7f98dc1f6750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98dc0cbd40 0x7f98dc1e6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 --2- 172.21.15.53:0/3549723904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98bc003970 0x7f98dc0b5990 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.238+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 >> 172.21.15.53:0/3549723904 conn(0x7f98dc066690 msgr2=0x7f98dc06cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:04.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.239+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 shutdown_connections 2024-01-16T15:13:04.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.239+0000 7f98e4800640 1 -- 172.21.15.53:0/3549723904 wait complete. 2024-01-16T15:13:04.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:13:04.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.480+0000 7f0d0fa00640 1 Processor -- start 2024-01-16T15:13:04.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.481+0000 7f0d0fa00640 1 -- start start 2024-01-16T15:13:04.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.482+0000 7f0d0fa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0d080d17e0 0x7f0d080d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.482+0000 7f0d0fa00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080d2180 con 0x7f0d080b3890 2024-01-16T15:13:04.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.482+0000 7f0d0fa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d080b3400 con 0x7f0d0815d610 2024-01-16T15:13:04.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.482+0000 7f0d0fa00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d080d1390 con 0x7f0d080d17e0 2024-01-16T15:13:04.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.483+0000 7f0cffc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0d080d17e0 0x7f0d080d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:04.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.483+0000 7f0d0da00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0d080b3890 0x7f0d080b3ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43972/0 (socket says 172.21.15.53:43972) 2024-01-16T15:13:04.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.483+0000 7f0d0da00640 1 -- 172.21.15.53:0/1891684856 learned_addr learned my addr 172.21.15.53:0/1891684856 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:04.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.483+0000 7f0cffc00640 1 -- 172.21.15.53:0/1891684856 >> v1:172.21.15.146:6789/0 conn(0x7f0d0815d610 legacy=0x7f0d080ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:04.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.483+0000 7f0cffc00640 1 -- 172.21.15.53:0/1891684856 >> v1:172.21.15.53:6789/0 conn(0x7f0d080b3890 legacy=0x7f0d080b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.483+0000 7f0cffc00640 1 -- 172.21.15.53:0/1891684856 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d080b27f0 con 0x7f0d080d17e0 2024-01-16T15:13:04.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.485+0000 7f0cffc00640 1 --2- 172.21.15.53:0/1891684856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080d17e0 0x7f0d080d1c40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00097a0 tx=0x7f0cf002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e98818fc2633932 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:04.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.485+0000 7f0d07e00640 1 -- 172.21.15.53:0/1891684856 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0031830 con 0x7f0d080d17e0 2024-01-16T15:13:04.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.485+0000 7f0d07e00640 1 -- 172.21.15.53:0/1891684856 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf0031990 con 0x7f0d080d17e0 2024-01-16T15:13:04.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.485+0000 7f0d07e00640 1 -- 172.21.15.53:0/1891684856 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf0031d20 con 0x7f0d080d17e0 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1891684856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080d17e0 msgr2=0x7f0d080d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1891684856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080d17e0 0x7f0d080d1c40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f0cf00097a0 tx=0x7f0cf002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1891684856 shutdown_connections 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1891684856 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080d17e0 0x7f0d080d1c40 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1891684856 >> 172.21.15.53:0/1891684856 conn(0x7f0d0809c540 msgr2=0x7f0d080d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1891684856 shutdown_connections 2024-01-16T15:13:04.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.486+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1891684856 wait complete. 2024-01-16T15:13:04.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.488+0000 7f0d0fa00640 1 Processor -- start 2024-01-16T15:13:04.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.488+0000 7f0d0fa00640 1 -- start start 2024-01-16T15:13:04.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0fa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 0x7f0d0814f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0fa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d080d17e0 0x7f0d080b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0fa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d0815d610 0x7f0d080b7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0fa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0814ff10 con 0x7f0d080b3890 2024-01-16T15:13:04.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0fa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b60f0 con 0x7f0d080d17e0 2024-01-16T15:13:04.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0fa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d080b63d0 con 0x7f0d0815d610 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 0x7f0d0814f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 0x7f0d0814f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55734/0 (socket says 172.21.15.53:55734) 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0d0d000640 1 -- 172.21.15.53:0/1809317400 learned_addr learned my addr 172.21.15.53:0/1809317400 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.489+0000 7f0cffc00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d080d17e0 0x7f0d080b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0da00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d0815d610 0x7f0d080b7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0d000640 1 -- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d0815d610 msgr2=0x7f0d080b7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0d000640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d0815d610 0x7f0d080b7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0d000640 1 -- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d080d17e0 msgr2=0x7f0d080b8930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0d000640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d080d17e0 0x7f0d080b8930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0d000640 1 -- 172.21.15.53:0/1809317400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf0009400 con 0x7f0d080b3890 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0cffc00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d080d17e0 0x7f0d080b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0d000640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 0x7f0d0814f9d0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800b3f0 tx=0x7f0cf800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011830 con 0x7f0d080b3890 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cf8011990 con 0x7f0d080b3890 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8011d40 con 0x7f0d080b3890 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.490+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d080b66b0 con 0x7f0d080b3890 2024-01-16T15:13:04.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.491+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d080b6c70 con 0x7f0d080b3890 2024-01-16T15:13:04.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.492+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d080675d0 con 0x7f0d080b3890 2024-01-16T15:13:04.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.493+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0cf801a400 con 0x7f0d080b3890 2024-01-16T15:13:04.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.494+0000 7f0d05600640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0ce006bf20 0x7f0ce006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.494+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0cf80886a0 con 0x7f0d080b3890 2024-01-16T15:13:04.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.495+0000 7f0cffc00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0ce006bf20 0x7f0ce006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:04.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.496+0000 7f0cffc00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0ce006bf20 0x7f0ce006e3e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0036000 tx=0x7f0cf0008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:04.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.499+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0cf8054ce0 con 0x7f0d080b3890 2024-01-16T15:13:04.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.719+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d080b3090 con 0x7f0ce006bf20 2024-01-16T15:13:04.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.773+0000 7f0d05600640 1 -- 172.21.15.53:0/1809317400 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0cf0045400 con 0x7f0ce006bf20 2024-01-16T15:13:04.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.775+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0ce006bf20 msgr2=0x7f0ce006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.775+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0ce006bf20 0x7f0ce006e3e0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f0cf0036000 tx=0x7f0cf0008030 comp rx=0 tx=0).stop 2024-01-16T15:13:04.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 msgr2=0x7f0d0814f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 0x7f0d0814f9d0 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800b3f0 tx=0x7f0cf800b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 shutdown_connections 2024-01-16T15:13:04.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0ce006bf20 0x7f0ce006e3e0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d0815d610 0x7f0d080b7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d080d17e0 0x7f0d080b8930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 --2- 172.21.15.53:0/1809317400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d080b3890 0x7f0d0814f9d0 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 >> 172.21.15.53:0/1809317400 conn(0x7f0d0809c540 msgr2=0x7f0d080b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:04.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 shutdown_connections 2024-01-16T15:13:04.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:04.776+0000 7f0d0fa00640 1 -- 172.21.15.53:0/1809317400 wait complete. 2024-01-16T15:13:04.775 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54147 2024-01-16T15:13:04.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.013+0000 7f659fe00640 1 Processor -- start 2024-01-16T15:13:04.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.013+0000 7f659fe00640 1 -- start start 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.014+0000 7f659fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65980d3910 0x7f65980d1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.014+0000 7f659fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6598160e10 0x7f65980d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.015+0000 7f659fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980d4160 con 0x7f65980cfcf0 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.015+0000 7f659fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65980ceff0 con 0x7f65980d3910 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.015+0000 7f659fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65980cf410 con 0x7f6598160e10 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.015+0000 7f659ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65980d3910 0x7f65980d1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.015+0000 7f659ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f65980d3910 0x7f65980d1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55740/0 (socket says 172.21.15.53:55740) 2024-01-16T15:13:04.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.015+0000 7f659ca00640 1 -- 172.21.15.53:0/4057155490 learned_addr learned my addr 172.21.15.53:0/4057155490 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:04.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.016+0000 7f659ca00640 1 -- 172.21.15.53:0/4057155490 >> v2:172.21.15.146:3300/0 conn(0x7f6598160e10 msgr2=0x7f65980d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:04.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.016+0000 7f659ca00640 1 --2- 172.21.15.53:0/4057155490 >> v2:172.21.15.146:3300/0 conn(0x7f6598160e10 0x7f65980d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.016+0000 7f659ca00640 1 -- 172.21.15.53:0/4057155490 >> v1:172.21.15.146:6789/0 conn(0x7f65980cfcf0 legacy=0x7f6598160700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:04.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.016+0000 7f659ca00640 1 -- 172.21.15.53:0/4057155490 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980b2420 con 0x7f65980d3910 2024-01-16T15:13:04.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.017+0000 7f659ca00640 1 --2- 172.21.15.53:0/4057155490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65980d3910 0x7f65980d1100 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f65980cd850 tx=0x7f658002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a2f2bc47508f348 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:04.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.017+0000 7f6597e00640 1 -- 172.21.15.53:0/4057155490 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65800317e0 con 0x7f65980d3910 2024-01-16T15:13:04.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f6597e00640 1 -- 172.21.15.53:0/4057155490 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6580031940 con 0x7f65980d3910 2024-01-16T15:13:04.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f6597e00640 1 -- 172.21.15.53:0/4057155490 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6580031cd0 con 0x7f65980d3910 2024-01-16T15:13:04.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f659fe00640 1 -- 172.21.15.53:0/4057155490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65980d3910 msgr2=0x7f65980d1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:04.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f659fe00640 1 --2- 172.21.15.53:0/4057155490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65980d3910 0x7f65980d1100 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f65980cd850 tx=0x7f658002b880 comp rx=0 tx=0).stop 2024-01-16T15:13:04.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f659fe00640 1 -- 172.21.15.53:0/4057155490 shutdown_connections 2024-01-16T15:13:04.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f659fe00640 1 --2- 172.21.15.53:0/4057155490 >> v2:172.21.15.146:3300/0 conn(0x7f6598160e10 0x7f65980d1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f659fe00640 1 --2- 172.21.15.53:0/4057155490 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f65980d3910 0x7f65980d1100 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:04.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.018+0000 7f659fe00640 1 -- 172.21.15.53:0/4057155490 >> 172.21.15.53:0/4057155490 conn(0x7f659809c530 msgr2=0x7f65980d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:04.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.019+0000 7f659fe00640 1 -- 172.21.15.53:0/4057155490 shutdown_connections 2024-01-16T15:13:04.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.019+0000 7f659fe00640 1 -- 172.21.15.53:0/4057155490 wait complete. 2024-01-16T15:13:04.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.020+0000 7f659fe00640 1 Processor -- start 2024-01-16T15:13:04.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.021+0000 7f659fe00640 1 -- start start 2024-01-16T15:13:05.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.021+0000 7f659fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65980cfcf0 0x7f659814e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.021+0000 7f659fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65980d3910 0x7f65980c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.021+0000 7f659fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 0x7f659815d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980b7bc0 con 0x7f6598160e10 2024-01-16T15:13:05.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65980b7ea0 con 0x7f65980cfcf0 2024-01-16T15:13:05.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65980b8180 con 0x7f65980d3910 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 0x7f659815d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 0x7f659815d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55752/0 (socket says 172.21.15.53:55752) 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 -- 172.21.15.53:0/321121920 learned_addr learned my addr 172.21.15.53:0/321121920 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659d400640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65980cfcf0 0x7f659814e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659ca00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65980d3910 0x7f65980c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 -- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65980d3910 msgr2=0x7f65980c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65980d3910 0x7f65980c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 -- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65980cfcf0 msgr2=0x7f659814e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65980cfcf0 0x7f659814e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659de00640 1 -- 172.21.15.53:0/321121920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65800094d0 con 0x7f6598160e10 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.022+0000 7f659ca00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65980d3910 0x7f65980c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:05.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.023+0000 7f659de00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 0x7f659815d020 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6590009fd0 tx=0x7f659000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.023+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590011830 con 0x7f6598160e10 2024-01-16T15:13:05.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.023+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6590011990 con 0x7f6598160e10 2024-01-16T15:13:05.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.023+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65980b8460 con 0x7f6598160e10 2024-01-16T15:13:05.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.023+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590011d20 con 0x7f6598160e10 2024-01-16T15:13:05.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.023+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f659814eb50 con 0x7f6598160e10 2024-01-16T15:13:05.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.024+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65980b8980 con 0x7f6598160e10 2024-01-16T15:13:05.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.025+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 28) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f659001b460 con 0x7f6598160e10 2024-01-16T15:13:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.026+0000 7f6595600640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f657006bf20 0x7f657006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.026+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f65900883a0 con 0x7f6598160e10 2024-01-16T15:13:05.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.027+0000 7f659d400640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f657006bf20 0x7f657006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.028+0000 7f659d400640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f657006bf20 0x7f657006e3e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6588009f20 tx=0x7f65880094d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.031+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f65900bde90 con 0x7f6598160e10 2024-01-16T15:13:05.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.251+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54147", "target": ["mon-mgr", ""]}) v1 -- 0x7f65980cec10 con 0x7f657006bf20 2024-01-16T15:13:05.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.296+0000 7f6595600640 1 -- 172.21.15.53:0/321121920 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6588009680 con 0x7f657006bf20 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.298+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f657006bf20 msgr2=0x7f657006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.298+0000 7f659fe00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f657006bf20 0x7f657006e3e0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f6588009f20 tx=0x7f65880094d0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.298+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 msgr2=0x7f659815d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 0x7f659815d020 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f6590009fd0 tx=0x7f659000b8e0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 shutdown_connections 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f657006bf20 0x7f657006e3e0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6598160e10 0x7f659815d020 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f65980d3910 0x7f65980c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 --2- 172.21.15.53:0/321121920 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f65980cfcf0 0x7f659814e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 >> 172.21.15.53:0/321121920 conn(0x7f659809c530 msgr2=0x7f65980d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 shutdown_connections 2024-01-16T15:13:05.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.299+0000 7f659fe00640 1 -- 172.21.15.53:0/321121920 wait complete. 2024-01-16T15:13:05.296 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54141 2024-01-16T15:13:05.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.521+0000 7f09ad000640 1 Processor -- start 2024-01-16T15:13:05.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.522+0000 7f09ad000640 1 -- start start 2024-01-16T15:13:05.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09ad000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09a815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09ad000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a806c470 con 0x7f09a813e360 2024-01-16T15:13:05.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09ad000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a806c750 con 0x7f09a80cf9a0 2024-01-16T15:13:05.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09ad000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a806ca30 con 0x7f09a813df80 2024-01-16T15:13:05.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09a5400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f09a813e360 0x7f09a80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43984/0 (socket says 172.21.15.53:43984) 2024-01-16T15:13:05.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09a5400640 1 -- 172.21.15.53:0/1242307041 learned_addr learned my addr 172.21.15.53:0/1242307041 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09a5e00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09a815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966094168 0 0) 0x7f09a806c470 con 0x7f09a813e360 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.524+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0988003520 con 0x7f09a813e360 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2515643919 0 0) 0x7f0988003520 con 0x7f09a813e360 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 msgr2=0x7f09a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09a815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 >> v1:172.21.15.146:6789/0 conn(0x7f09a80cf9a0 legacy=0x7f09a80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a806cd10 con 0x7f09a813e360 2024-01-16T15:13:05.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0990001670 con 0x7f09a813e360 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 >> v1:172.21.15.53:6789/0 conn(0x7f09a813e360 legacy=0x7f09a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0988004020 0x7f09880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0988004d00 0x7f0988005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a806ca30 con 0x7f0988004020 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f098800bbc0 con 0x7f0988004d00 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f098800bd80 con 0x7f09a813df80 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.525+0000 7f09a5400640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0988004020 0x7f09880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5e00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5400640 1 -- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 msgr2=0x7f09880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5400640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09880039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5400640 1 -- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0988004d00 msgr2=0x7f0988005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:05.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a6800640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0988004d00 0x7f0988005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5400640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0988004d00 0x7f0988005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5400640 1 -- 172.21.15.53:0/1242307041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0988003520 con 0x7f0988004020 2024-01-16T15:13:05.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a6800640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0988004d00 0x7f0988005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:05.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.526+0000 7f09a5e00640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813df80 0x7f09880039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.527+0000 7f09a5400640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0988004020 0x7f09880047c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f099000c0e0 tx=0x7f09880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a18ae482c53e411 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.527+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990005ad0 con 0x7f0988004020 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.527+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0990005c30 con 0x7f0988004020 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.527+0000 7f09a4a00640 1 -- 172.21.15.53:0/1242307041 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099002ca30 con 0x7f0988004020 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.527+0000 7f09ad000640 1 -- 172.21.15.53:0/1242307041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0988004020 msgr2=0x7f09880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.527+0000 7f09ad000640 1 --2- 172.21.15.53:0/1242307041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0988004020 0x7f09880047c0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f099000c0e0 tx=0x7f09880057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.528+0000 7f09a6800640 1 -- 172.21.15.53:0/1242307041 reap_dead start 2024-01-16T15:13:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.528+0000 7f09ad000640 1 -- 172.21.15.53:0/1242307041 shutdown_connections 2024-01-16T15:13:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.528+0000 7f09ad000640 1 -- 172.21.15.53:0/1242307041 >> 172.21.15.53:0/1242307041 conn(0x7f09a806a800 msgr2=0x7f09a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.528+0000 7f09ad000640 1 -- 172.21.15.53:0/1242307041 shutdown_connections 2024-01-16T15:13:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.528+0000 7f09ad000640 1 -- 172.21.15.53:0/1242307041 wait complete. 2024-01-16T15:13:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.529+0000 7f09ad000640 1 Processor -- start 2024-01-16T15:13:05.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.530+0000 7f09ad000640 1 -- start start 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.530+0000 7f09ad000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813d1e0 0x7f09a81f1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09ad000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a813d7e0 0x7f09a8153d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09ad000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 0x7f09a80b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09ad000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8155e60 con 0x7f09a80b3290 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09ad000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80b3c60 con 0x7f09a813d7e0 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09ad000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a813d5c0 con 0x7f09a813d1e0 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 0x7f09a80b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 0x7f09a80b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55764/0 (socket says 172.21.15.53:55764) 2024-01-16T15:13:05.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 -- 172.21.15.53:0/3154122431 learned_addr learned my addr 172.21.15.53:0/3154122431 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a5e00640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813d1e0 0x7f09a81f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 -- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813d1e0 msgr2=0x7f09a81f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813d1e0 0x7f09a81f1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 -- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a813d7e0 msgr2=0x7f09a8153d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a813d7e0 0x7f09a8153d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.531+0000 7f09a6800640 1 -- 172.21.15.53:0/3154122431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a81f1550 con 0x7f09a80b3290 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.532+0000 7f09a6800640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 0x7f09a80b3720 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f09a0000f80 tx=0x7f09a000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.532+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0006d50 con 0x7f09a80b3290 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.532+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0003970 con 0x7f09a80b3290 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.532+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a8154650 con 0x7f09a80b3290 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.532+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0003d20 con 0x7f09a80b3290 2024-01-16T15:13:05.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.532+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a8154c10 con 0x7f09a80b3290 2024-01-16T15:13:05.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.533+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a815c2f0 con 0x7f09a80b3290 2024-01-16T15:13:05.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.534+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f09a0005070 con 0x7f09a80b3290 2024-01-16T15:13:05.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.535+0000 7f099e000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f098006c070 0x7f098006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:05.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.535+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f09a00886d0 con 0x7f09a80b3290 2024-01-16T15:13:05.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.536+0000 7f09a5e00640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f098006c070 0x7f098006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:05.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.537+0000 7f09a5e00640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f098006c070 0x7f098006e530 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0998006fd0 tx=0x7f0998008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:05.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.539+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f09a0054ce0 con 0x7f09a80b3290 2024-01-16T15:13:05.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.758+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54141", "target": ["mon-mgr", ""]}) v1 -- 0x7f09a815bee0 con 0x7f098006c070 2024-01-16T15:13:05.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.803+0000 7f099e000640 1 -- 172.21.15.53:0/3154122431 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f09980095b0 con 0x7f098006c070 2024-01-16T15:13:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f098006c070 msgr2=0x7f098006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f098006c070 0x7f098006e530 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0998006fd0 tx=0x7f0998008040 comp rx=0 tx=0).stop 2024-01-16T15:13:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 msgr2=0x7f09a80b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 0x7f09a80b3720 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7f09a0000f80 tx=0x7f09a000dc00 comp rx=0 tx=0).stop 2024-01-16T15:13:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 shutdown_connections 2024-01-16T15:13:05.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f098006c070 0x7f098006e530 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.806+0000 7f09ad000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f09a80b3290 0x7f09a80b3720 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.807+0000 7f09ad000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f09a813d7e0 0x7f09a8153d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.807+0000 7f09ad000640 1 --2- 172.21.15.53:0/3154122431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f09a813d1e0 0x7f09a81f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.807+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 >> 172.21.15.53:0/3154122431 conn(0x7f09a806a800 msgr2=0x7f09a80aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.807+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 shutdown_connections 2024-01-16T15:13:05.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:05.807+0000 7f09ad000640 1 -- 172.21.15.53:0/3154122431 wait complete. 2024-01-16T15:13:05.805 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54142 2024-01-16T15:13:06.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.073+0000 7f06ae600640 1 Processor -- start 2024-01-16T15:13:06.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.073+0000 7f06ae600640 1 -- start start 2024-01-16T15:13:06.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06ae600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80ae5a0 0x7f06a815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06ae600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f06a80cf290 0x7f06a80cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06ae600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a8070c40 con 0x7f06a80b3ae0 2024-01-16T15:13:06.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06ae600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a8071450 con 0x7f06a80cf290 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06ae600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80cfc60 con 0x7f06a80ae5a0 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06a7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f06a80cf290 0x7f06a80cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f069fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f06a80b3ae0 0x7f06a806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:43998/0 (socket says 172.21.15.53:43998) 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f069fe00640 1 -- 172.21.15.53:0/2777480707 learned_addr learned my addr 172.21.15.53:0/2777480707 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.075+0000 7f06a7400640 1 --2- 172.21.15.53:0/2777480707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80ae5a0 0x7f06a815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.076+0000 7f06a7e00640 1 -- 172.21.15.53:0/2777480707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80ae5a0 msgr2=0x7f06a815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.076+0000 7f06a7e00640 1 --2- 172.21.15.53:0/2777480707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80ae5a0 0x7f06a815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.076+0000 7f06a7e00640 1 -- 172.21.15.53:0/2777480707 >> v1:172.21.15.53:6789/0 conn(0x7f06a80b3ae0 legacy=0x7f06a806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.076+0000 7f06a7e00640 1 -- 172.21.15.53:0/2777480707 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a806c470 con 0x7f06a80cf290 2024-01-16T15:13:06.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.076+0000 7f06a6a00640 1 -- 172.21.15.53:0/2777480707 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534403303 0 0) 0x7f06a8070c40 con 0x7f06a80b3ae0 2024-01-16T15:13:06.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.077+0000 7f06a7e00640 1 --2- 172.21.15.53:0/2777480707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a80cf720 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009d90 tx=0x7f06a002bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db72444b67fdd1e9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.077+0000 7f06a6a00640 1 -- 172.21.15.53:0/2777480707 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a002cce0 con 0x7f06a80cf290 2024-01-16T15:13:06.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.077+0000 7f06a6a00640 1 -- 172.21.15.53:0/2777480707 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a002ce40 con 0x7f06a80cf290 2024-01-16T15:13:06.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.077+0000 7f06a6a00640 1 -- 172.21.15.53:0/2777480707 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a0034660 con 0x7f06a80cf290 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.077+0000 7f06ae600640 1 -- 172.21.15.53:0/2777480707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 msgr2=0x7f06a80cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.077+0000 7f06ae600640 1 --2- 172.21.15.53:0/2777480707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a80cf720 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009d90 tx=0x7f06a002bf30 comp rx=0 tx=0).stop 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.078+0000 7f06ae600640 1 -- 172.21.15.53:0/2777480707 shutdown_connections 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.078+0000 7f06ae600640 1 --2- 172.21.15.53:0/2777480707 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a80cf720 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.078+0000 7f06ae600640 1 --2- 172.21.15.53:0/2777480707 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80ae5a0 0x7f06a815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.078+0000 7f06ae600640 1 -- 172.21.15.53:0/2777480707 >> 172.21.15.53:0/2777480707 conn(0x7f06a806a800 msgr2=0x7f06a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.078+0000 7f06ae600640 1 -- 172.21.15.53:0/2777480707 shutdown_connections 2024-01-16T15:13:06.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.078+0000 7f06ae600640 1 -- 172.21.15.53:0/2777480707 wait complete. 2024-01-16T15:13:06.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.079+0000 7f06ae600640 1 Processor -- start 2024-01-16T15:13:06.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.079+0000 7f06ae600640 1 -- start start 2024-01-16T15:13:06.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.080+0000 7f06ae600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a80ae5a0 0x7f06a80cd480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.080+0000 7f06ae600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80b3ae0 0x7f06a8160510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.080+0000 7f06ae600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.080+0000 7f06ae600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8155eb0 con 0x7f06a80cf290 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.080+0000 7f06ae600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a8160a50 con 0x7f06a80ae5a0 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.080+0000 7f06ae600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a80cda60 con 0x7f06a80b3ae0 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a815a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55788/0 (socket says 172.21.15.53:55788) 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 -- 172.21.15.53:0/411383006 learned_addr learned my addr 172.21.15.53:0/411383006 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f069fe00640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80b3ae0 0x7f06a8160510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7400640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a80ae5a0 0x7f06a80cd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 -- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80b3ae0 msgr2=0x7f06a8160510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80b3ae0 0x7f06a8160510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 -- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a80ae5a0 msgr2=0x7f06a80cd480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a80ae5a0 0x7f06a80cd480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 -- 172.21.15.53:0/411383006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a0009a70 con 0x7f06a80cf290 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.081+0000 7f06a7e00640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a815a820 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009d90 tx=0x7f06a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.082+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a003e670 con 0x7f06a80cf290 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.082+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06a003e7f0 con 0x7f06a80cf290 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.082+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06a003eba0 con 0x7f06a80cf290 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.082+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a80cdd40 con 0x7f06a80cf290 2024-01-16T15:13:06.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.082+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a8156150 con 0x7f06a80cf290 2024-01-16T15:13:06.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.083+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a8156340 con 0x7f06a80cf290 2024-01-16T15:13:06.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.084+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f06a003b040 con 0x7f06a80cf290 2024-01-16T15:13:06.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.085+0000 7f069f400640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f068006c040 0x7f068006e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.085+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f06a00ab3a0 con 0x7f06a80cf290 2024-01-16T15:13:06.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.085+0000 7f06a7400640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f068006c040 0x7f068006e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.086+0000 7f06a7400640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f068006c040 0x7f068006e500 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0698006fd0 tx=0x7f0698008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.089+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f06a00779b0 con 0x7f06a80cf290 2024-01-16T15:13:06.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.300+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54142", "target": ["mon-mgr", ""]}) v1 -- 0x7f06a8071170 con 0x7f068006c040 2024-01-16T15:13:06.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.342+0000 7f069f400640 1 -- 172.21.15.53:0/411383006 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f069800b200 con 0x7f068006c040 2024-01-16T15:13:06.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f068006c040 msgr2=0x7f068006e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f068006c040 0x7f068006e500 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f0698006fd0 tx=0x7f0698008040 comp rx=0 tx=0).stop 2024-01-16T15:13:06.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 msgr2=0x7f06a815a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a815a820 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009d90 tx=0x7f06a0004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:06.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 shutdown_connections 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f068006c040 0x7f068006e500 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f06a80cf290 0x7f06a815a820 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f06a80b3ae0 0x7f06a8160510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 --2- 172.21.15.53:0/411383006 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f06a80ae5a0 0x7f06a80cd480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 >> 172.21.15.53:0/411383006 conn(0x7f06a806a800 msgr2=0x7f06a80cb0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 shutdown_connections 2024-01-16T15:13:06.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.345+0000 7f06ae600640 1 -- 172.21.15.53:0/411383006 wait complete. 2024-01-16T15:13:06.342 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541412 2024-01-16T15:13:06.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.590+0000 7f1ce9600640 1 Processor -- start 2024-01-16T15:13:06.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.590+0000 7f1ce9600640 1 -- start start 2024-01-16T15:13:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce413df80 0x7f1ce415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce9600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ce40cf9a0 0x7f1ce40cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce9600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce406c470 con 0x7f1ce413e360 2024-01-16T15:13:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce9600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ce406c750 con 0x7f1ce40cf9a0 2024-01-16T15:13:06.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce406ca30 con 0x7f1ce413df80 2024-01-16T15:13:06.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ce40cf9a0 0x7f1ce40cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce3400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1ce40cf9a0 0x7f1ce40cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55804/0 (socket says 172.21.15.53:55804) 2024-01-16T15:13:06.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.592+0000 7f1ce3400640 1 -- 172.21.15.53:0/3327744260 learned_addr learned my addr 172.21.15.53:0/3327744260 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:06.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.593+0000 7f1ce2a00640 1 --2- 172.21.15.53:0/3327744260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce413df80 0x7f1ce415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.593+0000 7f1ce3400640 1 -- 172.21.15.53:0/3327744260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce413df80 msgr2=0x7f1ce415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:06.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.593+0000 7f1ce3400640 1 --2- 172.21.15.53:0/3327744260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce413df80 0x7f1ce415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.593+0000 7f1ce3400640 1 -- 172.21.15.53:0/3327744260 >> v1:172.21.15.146:6789/0 conn(0x7f1ce413e360 legacy=0x7f1ce40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.593+0000 7f1ce3400640 1 -- 172.21.15.53:0/3327744260 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce406cd10 con 0x7f1ce40cf9a0 2024-01-16T15:13:06.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.594+0000 7f1ce3400640 1 --2- 172.21.15.53:0/3327744260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce40cf9a0 0x7f1ce40cd840 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009e50 tx=0x7f1cdc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7c6a522661c1e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.594+0000 7f1ce2000640 1 -- 172.21.15.53:0/3327744260 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc02cb00 con 0x7f1ce40cf9a0 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.594+0000 7f1ce2000640 1 -- 172.21.15.53:0/3327744260 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc02cc60 con 0x7f1ce40cf9a0 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.594+0000 7f1ce2000640 1 -- 172.21.15.53:0/3327744260 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc0343f0 con 0x7f1ce40cf9a0 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 -- 172.21.15.53:0/3327744260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce40cf9a0 msgr2=0x7f1ce40cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 --2- 172.21.15.53:0/3327744260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce40cf9a0 0x7f1ce40cd840 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009e50 tx=0x7f1cdc02bf60 comp rx=0 tx=0).stop 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 -- 172.21.15.53:0/3327744260 shutdown_connections 2024-01-16T15:13:06.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 --2- 172.21.15.53:0/3327744260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce40cf9a0 0x7f1ce40cd840 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 --2- 172.21.15.53:0/3327744260 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce413df80 0x7f1ce415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 -- 172.21.15.53:0/3327744260 >> 172.21.15.53:0/3327744260 conn(0x7f1ce406a800 msgr2=0x7f1ce406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 -- 172.21.15.53:0/3327744260 shutdown_connections 2024-01-16T15:13:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.595+0000 7f1ce9600640 1 -- 172.21.15.53:0/3327744260 wait complete. 2024-01-16T15:13:06.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.597+0000 7f1ce9600640 1 Processor -- start 2024-01-16T15:13:06.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.597+0000 7f1ce9600640 1 -- start start 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce40cf9a0 0x7f1ce413c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce413df80 0x7f1ce413d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 0x7f1ce41f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce9600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce4070f70 con 0x7f1ce413e360 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce9600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce413d780 con 0x7f1ce413df80 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce9600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ce41f6270 con 0x7f1ce40cf9a0 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 0x7f1ce41f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 0x7f1ce41f5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55808/0 (socket says 172.21.15.53:55808) 2024-01-16T15:13:06.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.598+0000 7f1ce3400640 1 -- 172.21.15.53:0/1583879012 learned_addr learned my addr 172.21.15.53:0/1583879012 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1cdbe00640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce413df80 0x7f1ce413d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce2a00640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce40cf9a0 0x7f1ce413c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce3400640 1 -- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce40cf9a0 msgr2=0x7f1ce413c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce3400640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce40cf9a0 0x7f1ce413c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce3400640 1 -- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce413df80 msgr2=0x7f1ce413d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce3400640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce413df80 0x7f1ce413d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce3400640 1 -- 172.21.15.53:0/1583879012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cdc009b30 con 0x7f1ce413e360 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1cdbe00640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce413df80 0x7f1ce413d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce3400640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 0x7f1ce41f5d30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009e50 tx=0x7f1cdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc009510 con 0x7f1ce413e360 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cdc03e450 con 0x7f1ce413e360 2024-01-16T15:13:06.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc03e800 con 0x7f1ce413e360 2024-01-16T15:13:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce41e7120 con 0x7f1ce413e360 2024-01-16T15:13:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.599+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce40711c0 con 0x7f1ce413e360 2024-01-16T15:13:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.601+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce40675d0 con 0x7f1ce413e360 2024-01-16T15:13:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.601+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f1cdc03e960 con 0x7f1ce413e360 2024-01-16T15:13:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.603+0000 7f1cdaa00640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cbc06c0c0 0x7f1cbc06e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:06.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.603+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1cdc0ab650 con 0x7f1ce413e360 2024-01-16T15:13:06.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.604+0000 7f1ce2a00640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cbc06c0c0 0x7f1cbc06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:06.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.605+0000 7f1ce2a00640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cbc06c0c0 0x7f1cbc06e580 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4006fd0 tx=0x7f1cd4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:06.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.607+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cdc077c60 con 0x7f1ce413e360 2024-01-16T15:13:06.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.823+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541412", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ce415f9c0 con 0x7f1cbc06c0c0 2024-01-16T15:13:06.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.884+0000 7f1cdaa00640 1 -- 172.21.15.53:0/1583879012 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1cd4009d80 con 0x7f1cbc06c0c0 2024-01-16T15:13:06.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.888+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cbc06c0c0 msgr2=0x7f1cbc06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.888+0000 7f1ce9600640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cbc06c0c0 0x7f1cbc06e580 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4006fd0 tx=0x7f1cd4008040 comp rx=0 tx=0).stop 2024-01-16T15:13:06.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.888+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 msgr2=0x7f1ce41f5d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:06.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.888+0000 7f1ce9600640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 0x7f1ce41f5d30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009e50 tx=0x7f1cdc004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:06.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 shutdown_connections 2024-01-16T15:13:06.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1cbc06c0c0 0x7f1cbc06e580 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1ce413e360 0x7f1ce41f5d30 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1ce413df80 0x7f1ce413d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 --2- 172.21.15.53:0/1583879012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1ce40cf9a0 0x7f1ce413c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:06.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 >> 172.21.15.53:0/1583879012 conn(0x7f1ce406a800 msgr2=0x7f1ce415cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:06.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.889+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 shutdown_connections 2024-01-16T15:13:06.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:06.890+0000 7f1ce9600640 1 -- 172.21.15.53:0/1583879012 wait complete. 2024-01-16T15:13:06.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541415 2024-01-16T15:13:07.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.195+0000 7fb55f200640 1 Processor -- start 2024-01-16T15:13:07.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.195+0000 7fb55f200640 1 -- start start 2024-01-16T15:13:07.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.197+0000 7fb55f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 0x7fb55815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.197+0000 7fb55f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb5580cf9a0 0x7fb5580cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.197+0000 7fb55f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb55806c470 con 0x7fb55813e360 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.197+0000 7fb55f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb55806c750 con 0x7fb5580cf9a0 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.197+0000 7fb55f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806ca30 con 0x7fb55813df80 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.198+0000 7fb54f200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb55813e360 0x7fb5580cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44002/0 (socket says 172.21.15.53:44002) 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.198+0000 7fb54f200640 1 -- 172.21.15.53:0/476455607 learned_addr learned my addr 172.21.15.53:0/476455607 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:07.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.198+0000 7fb557e00640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 0x7fb55815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.199+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231541605 0 0) 0x7fb55806c470 con 0x7fb55813e360 2024-01-16T15:13:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.199+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb538003520 con 0x7fb55813e360 2024-01-16T15:13:07.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb55d200640 1 --2- 172.21.15.53:0/476455607 >> v2:172.21.15.146:3300/0 conn(0x7fb5580cf9a0 0x7fb5580cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1912648380 0 0) 0x7fb538003520 con 0x7fb55813e360 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 msgr2=0x7fb55815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 0x7fb55815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 >> v2:172.21.15.146:3300/0 conn(0x7fb5580cf9a0 msgr2=0x7fb5580cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 --2- 172.21.15.53:0/476455607 >> v2:172.21.15.146:3300/0 conn(0x7fb5580cf9a0 0x7fb5580cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55806cd10 con 0x7fb55813e360 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb540001670 con 0x7fb55813e360 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 >> v1:172.21.15.53:6789/0 conn(0x7fb55813e360 legacy=0x7fb5580cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb538003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 0x7fb5380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb538004d00 0x7fb538005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806c750 con 0x7fb538003970 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55806ca30 con 0x7fb538004d00 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53800b980 con 0x7fb55813df80 2024-01-16T15:13:07.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.200+0000 7fb557e00640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb538003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb55d200640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb538004d00 0x7fb538005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb557e00640 1 -- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 msgr2=0x7fb5380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb557e00640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb55813df80 0x7fb5380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb557e00640 1 -- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb538004d00 msgr2=0x7fb538005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb557e00640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb538004d00 0x7fb538005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb557e00640 1 -- 172.21.15.53:0/476455607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55806cd10 con 0x7fb538003970 2024-01-16T15:13:07.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.201+0000 7fb55d200640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb538004d00 0x7fb538005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:07.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.202+0000 7fb557e00640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb538003d90 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb548009af0 tx=0x7fb5380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b731c1856a70664 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.203+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54802c710 con 0x7fb538003970 2024-01-16T15:13:07.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.203+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb54802c870 con 0x7fb538003970 2024-01-16T15:13:07.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.203+0000 7fb557400640 1 -- 172.21.15.53:0/476455607 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb54802cc00 con 0x7fb538003970 2024-01-16T15:13:07.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.204+0000 7fb55f200640 1 -- 172.21.15.53:0/476455607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 msgr2=0x7fb538003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.204+0000 7fb55f200640 1 --2- 172.21.15.53:0/476455607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb538003d90 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7fb548009af0 tx=0x7fb5380057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.204+0000 7fb55d200640 1 -- 172.21.15.53:0/476455607 reap_dead start 2024-01-16T15:13:07.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.204+0000 7fb55f200640 1 -- 172.21.15.53:0/476455607 shutdown_connections 2024-01-16T15:13:07.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.204+0000 7fb55f200640 1 -- 172.21.15.53:0/476455607 >> 172.21.15.53:0/476455607 conn(0x7fb55806a800 msgr2=0x7fb55806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:07.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.205+0000 7fb55f200640 1 -- 172.21.15.53:0/476455607 shutdown_connections 2024-01-16T15:13:07.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.205+0000 7fb55f200640 1 -- 172.21.15.53:0/476455607 wait complete. 2024-01-16T15:13:07.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.209+0000 7fb55f200640 1 Processor -- start 2024-01-16T15:13:07.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.209+0000 7fb55f200640 1 -- start start 2024-01-16T15:13:07.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb5580cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55815d8a0 0x7fb55815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 0x7fb5580c42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55815e260 con 0x7fb538003970 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580c4810 con 0x7fb55815d8a0 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5580cbd30 con 0x7fb5580c3e50 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 0x7fb5580c42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 0x7fb5580c42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:38344/0 (socket says 172.21.15.53:38344) 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb55d200640 1 -- 172.21.15.53:0/1322841219 learned_addr learned my addr 172.21.15.53:0/1322841219 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.212+0000 7fb557e00640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb5580cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb54f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55815d8a0 0x7fb55815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb557e00640 1 -- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 msgr2=0x7fb5580c42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb557e00640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 0x7fb5580c42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb557e00640 1 -- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55815d8a0 msgr2=0x7fb55815dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb557e00640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55815d8a0 0x7fb55815dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb557e00640 1 -- 172.21.15.53:0/1322841219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb558149e20 con 0x7fb538003970 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb54f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55815d8a0 0x7fb55815dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.213+0000 7fb55d200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 0x7fb5580c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:07.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.214+0000 7fb557e00640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb5580cb7f0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb538004700 tx=0x7fb548004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.215+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548003610 con 0x7fb538003970 2024-01-16T15:13:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.215+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb548003770 con 0x7fb538003970 2024-01-16T15:13:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.215+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb548003ae0 con 0x7fb538003970 2024-01-16T15:13:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.215+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5581f3bb0 con 0x7fb538003970 2024-01-16T15:13:07.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.215+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55814a350 con 0x7fb538003970 2024-01-16T15:13:07.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.217+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5580675d0 con 0x7fb538003970 2024-01-16T15:13:07.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.220+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fb54803b070 con 0x7fb538003970 2024-01-16T15:13:07.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.221+0000 7fb554c00640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53006c140 0x7fb53006e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.221+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb5480aa280 con 0x7fb538003970 2024-01-16T15:13:07.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.222+0000 7fb54f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53006c140 0x7fb53006e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.223+0000 7fb54f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53006c140 0x7fb53006e600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb5400017e0 tx=0x7fb540009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.228+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb548076b20 con 0x7fb538003970 2024-01-16T15:13:07.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.502+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541415", "target": ["mon-mgr", ""]}) v1 -- 0x7fb55815bee0 con 0x7fb53006c140 2024-01-16T15:13:07.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.543+0000 7fb554c00640 1 -- 172.21.15.53:0/1322841219 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb540007870 con 0x7fb53006c140 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53006c140 msgr2=0x7fb53006e600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53006c140 0x7fb53006e600 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fb5400017e0 tx=0x7fb540009040 comp rx=0 tx=0).stop 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 msgr2=0x7fb5580cb7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb5580cb7f0 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7fb538004700 tx=0x7fb548004730 comp rx=0 tx=0).stop 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 shutdown_connections 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb53006c140 0x7fb53006e600 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5580c3e50 0x7fb5580c42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb55815d8a0 0x7fb55815dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 --2- 172.21.15.53:0/1322841219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb538003970 0x7fb5580cb7f0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.545+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 >> 172.21.15.53:0/1322841219 conn(0x7fb55806a800 msgr2=0x7fb5580afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:07.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.546+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 shutdown_connections 2024-01-16T15:13:07.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.546+0000 7fb55f200640 1 -- 172.21.15.53:0/1322841219 wait complete. 2024-01-16T15:13:07.541 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541417 2024-01-16T15:13:07.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.772+0000 7f99d0200640 1 Processor -- start 2024-01-16T15:13:07.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.772+0000 7f99d0200640 1 -- start start 2024-01-16T15:13:07.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.773+0000 7f99d0200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f99c80ae5a0 0x7f99c815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.773+0000 7f99d0200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99c80b3ae0 0x7f99c806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.773+0000 7f99d0200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.773+0000 7f99d0200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c8071450 con 0x7f99c80ae5a0 2024-01-16T15:13:07.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99d0200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c8070c40 con 0x7f99c80b3ae0 2024-01-16T15:13:07.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99d0200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c806c470 con 0x7f99c806d120 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99cd800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f99c80ae5a0 0x7f99c815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99cd800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f99c80ae5a0 0x7f99c815e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55842/0 (socket says 172.21.15.53:55842) 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99cd800640 1 -- 172.21.15.53:0/248205822 learned_addr learned my addr 172.21.15.53:0/248205822 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99cce00640 1 --2- 172.21.15.53:0/248205822 >> v2:172.21.15.146:3300/0 conn(0x7f99c80b3ae0 0x7f99c806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.774+0000 7f99ce200640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.775+0000 7f99cd800640 1 -- 172.21.15.53:0/248205822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 msgr2=0x7f99c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.775+0000 7f99cd800640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.775+0000 7f99cd800640 1 -- 172.21.15.53:0/248205822 >> v2:172.21.15.146:3300/0 conn(0x7f99c80b3ae0 msgr2=0x7f99c806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.775+0000 7f99cd800640 1 --2- 172.21.15.53:0/248205822 >> v2:172.21.15.146:3300/0 conn(0x7f99c80b3ae0 0x7f99c806cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.775+0000 7f99cd800640 1 -- 172.21.15.53:0/248205822 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c806c750 con 0x7f99c80ae5a0 2024-01-16T15:13:07.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.775+0000 7f99ce200640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.776+0000 7f99cd800640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c815e430 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f99c80cfea0 tx=0x7f99b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5de8b38139541a4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.776+0000 7f99c7e00640 1 -- 172.21.15.53:0/248205822 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8031870 con 0x7f99c80ae5a0 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.776+0000 7f99c7e00640 1 -- 172.21.15.53:0/248205822 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b80319d0 con 0x7f99c80ae5a0 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.776+0000 7f99c7e00640 1 -- 172.21.15.53:0/248205822 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8031d60 con 0x7f99c80ae5a0 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 -- 172.21.15.53:0/248205822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 msgr2=0x7f99c815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c815e430 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f99c80cfea0 tx=0x7f99b802b910 comp rx=0 tx=0).stop 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 -- 172.21.15.53:0/248205822 shutdown_connections 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 --2- 172.21.15.53:0/248205822 >> v2:172.21.15.146:3300/0 conn(0x7f99c80b3ae0 0x7f99c806cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 --2- 172.21.15.53:0/248205822 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c815e430 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 -- 172.21.15.53:0/248205822 >> 172.21.15.53:0/248205822 conn(0x7f99c806a800 msgr2=0x7f99c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:07.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 -- 172.21.15.53:0/248205822 shutdown_connections 2024-01-16T15:13:07.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.777+0000 7f99d0200640 1 -- 172.21.15.53:0/248205822 wait complete. 2024-01-16T15:13:07.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.779+0000 7f99d0200640 1 Processor -- start 2024-01-16T15:13:07.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.779+0000 7f99d0200640 1 -- start start 2024-01-16T15:13:07.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99d0200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c8149d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99d0200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c80cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99d0200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80b3ae0 0x7f99c8160f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99d0200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c80cd8f0 con 0x7f99c80ae5a0 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99d0200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c814a260 con 0x7f99c80b3ae0 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99d0200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c80ccce0 con 0x7f99c806d120 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99cce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c80cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99cce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c80cc2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55844/0 (socket says 172.21.15.53:55844) 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99cce00640 1 -- 172.21.15.53:0/649710803 learned_addr learned my addr 172.21.15.53:0/649710803 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.781+0000 7f99ce200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80b3ae0 0x7f99c8160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cd800640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c8149d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cce00640 1 -- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 msgr2=0x7f99c8149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cce00640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c8149d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cce00640 1 -- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80b3ae0 msgr2=0x7f99c8160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:07.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cce00640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80b3ae0 0x7f99c8160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:07.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cce00640 1 -- 172.21.15.53:0/649710803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80094d0 con 0x7f99c80ae5a0 2024-01-16T15:13:07.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99cce00640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c80cc2b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f99b000d5f0 tx=0x7f99b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b00125f0 con 0x7f99c80ae5a0 2024-01-16T15:13:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b0012750 con 0x7f99c80ae5a0 2024-01-16T15:13:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c80ccfc0 con 0x7f99c80ae5a0 2024-01-16T15:13:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b0012b00 con 0x7f99c80ae5a0 2024-01-16T15:13:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.782+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c80cd580 con 0x7f99c80ae5a0 2024-01-16T15:13:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.783+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c80cda60 con 0x7f99c80ae5a0 2024-01-16T15:13:07.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.784+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7f99b0012c60 con 0x7f99c80ae5a0 2024-01-16T15:13:07.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.785+0000 7f99c5600640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99a006c070 0x7f99a006e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:07.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.785+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f99b0088d90 con 0x7f99c80ae5a0 2024-01-16T15:13:07.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.786+0000 7f99cd800640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99a006c070 0x7f99a006e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:07.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.787+0000 7f99cd800640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99a006c070 0x7f99a006e530 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f99b8036000 tx=0x7f99b803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:07.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:07.789+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99b00553a0 con 0x7f99c80ae5a0 2024-01-16T15:13:08.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.116+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541417", "target": ["mon-mgr", ""]}) v1 -- 0x7f99c8070550 con 0x7f99a006c070 2024-01-16T15:13:08.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.141+0000 7f99c5600640 1 -- 172.21.15.53:0/649710803 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f99b8031c40 con 0x7f99a006c070 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99a006c070 msgr2=0x7f99a006e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99a006c070 0x7f99a006e530 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f99b8036000 tx=0x7f99b803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 msgr2=0x7f99c80cc2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c80cc2b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f99b000d5f0 tx=0x7f99b000dab0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 shutdown_connections 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99a006c070 0x7f99a006e530 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99c80b3ae0 0x7f99c8160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99c80ae5a0 0x7f99c80cc2b0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 --2- 172.21.15.53:0/649710803 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99c806d120 0x7f99c8149d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 >> 172.21.15.53:0/649710803 conn(0x7f99c806a800 msgr2=0x7f99c806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:08.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.144+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 shutdown_connections 2024-01-16T15:13:08.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.145+0000 7f99d0200640 1 -- 172.21.15.53:0/649710803 wait complete. 2024-01-16T15:13:08.148 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541420 2024-01-16T15:13:08.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.440+0000 7fddbca00640 1 Processor -- start 2024-01-16T15:13:08.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.441+0000 7fddbca00640 1 -- start start 2024-01-16T15:13:08.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.443+0000 7fddbca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddb4150760 0x7fddb4072230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.443+0000 7fddbca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb40b32e0 0x7fddb415d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.443+0000 7fddbca00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb40b3750 con 0x7fddb415d300 2024-01-16T15:13:08.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.443+0000 7fddbca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb415dc80 con 0x7fddb4150760 2024-01-16T15:13:08.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.443+0000 7fddbca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4071190 con 0x7fddb40b32e0 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.443+0000 7fddba000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddb4150760 0x7fddb4072230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddba000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddb4150760 0x7fddb4072230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55856/0 (socket says 172.21.15.53:55856) 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddba000640 1 -- 172.21.15.53:0/2355909677 learned_addr learned my addr 172.21.15.53:0/2355909677 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddbaa00640 1 --2- 172.21.15.53:0/2355909677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb40b32e0 0x7fddb415d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddba000640 1 -- 172.21.15.53:0/2355909677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb40b32e0 msgr2=0x7fddb415d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddba000640 1 --2- 172.21.15.53:0/2355909677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb40b32e0 0x7fddb415d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddba000640 1 -- 172.21.15.53:0/2355909677 >> v1:172.21.15.53:6789/0 conn(0x7fddb415d300 legacy=0x7fddb40b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:08.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.444+0000 7fddba000640 1 -- 172.21.15.53:0/2355909677 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb4071470 con 0x7fddb4150760 2024-01-16T15:13:08.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.445+0000 7fddba000640 1 --2- 172.21.15.53:0/2355909677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb4150760 0x7fddb4072230 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009870 tx=0x7fdda402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58eaa5054646e5e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:08.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.446+0000 7fddb8c00640 1 -- 172.21.15.53:0/2355909677 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031640 con 0x7fddb4150760 2024-01-16T15:13:08.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.446+0000 7fddb8c00640 1 -- 172.21.15.53:0/2355909677 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda40317a0 con 0x7fddb4150760 2024-01-16T15:13:08.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.446+0000 7fddb8c00640 1 -- 172.21.15.53:0/2355909677 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031b30 con 0x7fddb4150760 2024-01-16T15:13:08.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.447+0000 7fddbca00640 1 -- 172.21.15.53:0/2355909677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb4150760 msgr2=0x7fddb4072230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.447+0000 7fddbca00640 1 --2- 172.21.15.53:0/2355909677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb4150760 0x7fddb4072230 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fdda4009870 tx=0x7fdda402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.448+0000 7fddbca00640 1 -- 172.21.15.53:0/2355909677 shutdown_connections 2024-01-16T15:13:08.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.448+0000 7fddbca00640 1 --2- 172.21.15.53:0/2355909677 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb40b32e0 0x7fddb415d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.448+0000 7fddbca00640 1 --2- 172.21.15.53:0/2355909677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb4150760 0x7fddb4072230 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.448+0000 7fddbca00640 1 -- 172.21.15.53:0/2355909677 >> 172.21.15.53:0/2355909677 conn(0x7fddb406a800 msgr2=0x7fddb406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:08.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.448+0000 7fddbca00640 1 -- 172.21.15.53:0/2355909677 shutdown_connections 2024-01-16T15:13:08.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.449+0000 7fddbca00640 1 -- 172.21.15.53:0/2355909677 wait complete. 2024-01-16T15:13:08.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.452+0000 7fddbca00640 1 Processor -- start 2024-01-16T15:13:08.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.452+0000 7fddbca00640 1 -- start start 2024-01-16T15:13:08.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.453+0000 7fddbca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb40b32e0 0x7fddb4142150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddbca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb4150760 0x7fddb415c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddbca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb415d300 0x7fddb41590a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddbca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4148390 con 0x7fddb40b32e0 2024-01-16T15:13:08.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddbca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb4146800 con 0x7fddb4150760 2024-01-16T15:13:08.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddbca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41485a0 con 0x7fddb415d300 2024-01-16T15:13:08.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddb9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb4150760 0x7fddb415c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddb9600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb4150760 0x7fddb415c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:33212/0 (socket says 172.21.15.53:33212) 2024-01-16T15:13:08.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.454+0000 7fddb9600640 1 -- 172.21.15.53:0/2759872329 learned_addr learned my addr 172.21.15.53:0/2759872329 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:08.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.455+0000 7fddba000640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb40b32e0 0x7fddb4142150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.455+0000 7fddbaa00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb415d300 0x7fddb41590a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.456+0000 7fddbaa00640 1 -- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb4150760 msgr2=0x7fddb415c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.456+0000 7fddbaa00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb4150760 0x7fddb415c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.456+0000 7fddbaa00640 1 -- 172.21.15.53:0/2759872329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb40b32e0 msgr2=0x7fddb4142150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.456+0000 7fddbaa00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb40b32e0 0x7fddb4142150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.456+0000 7fddbaa00640 1 -- 172.21.15.53:0/2759872329 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40094d0 con 0x7fddb415d300 2024-01-16T15:13:08.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.457+0000 7fddbaa00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb415d300 0x7fddb41590a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fddac007d90 tx=0x7fddac00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:08.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.458+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac018070 con 0x7fddb415d300 2024-01-16T15:13:08.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.458+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddac00edd0 con 0x7fddb415d300 2024-01-16T15:13:08.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.458+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac003bb0 con 0x7fddb415d300 2024-01-16T15:13:08.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.458+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb4148880 con 0x7fddb415d300 2024-01-16T15:13:08.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.458+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb4146d30 con 0x7fddb415d300 2024-01-16T15:13:08.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.459+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb4146f50 con 0x7fddb415d300 2024-01-16T15:13:08.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.462+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mon.2 v2:172.21.15.146:3301/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fddac014420 con 0x7fddb415d300 2024-01-16T15:13:08.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.466+0000 7fddb2000640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c040 0x7fdd8c06e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:08.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.466+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mon.2 v2:172.21.15.146:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fddac088a00 con 0x7fddb415d300 2024-01-16T15:13:08.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.467+0000 7fddba000640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c040 0x7fdd8c06e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:08.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.468+0000 7fddba000640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c040 0x7fdd8c06e500 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:08.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.471+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mon.2 v2:172.21.15.146:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddac0550c0 con 0x7fddb415d300 2024-01-16T15:13:08.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.720+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541420", "target": ["mon-mgr", ""]}) v1 -- 0x7fddb4071d10 con 0x7fdd8c06c040 2024-01-16T15:13:08.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.757+0000 7fddb2000640 1 -- 172.21.15.53:0/2759872329 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdda4031a00 con 0x7fdd8c06c040 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c040 msgr2=0x7fdd8c06e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c040 0x7fdd8c06e500 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda403c040 comp rx=0 tx=0).stop 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb415d300 msgr2=0x7fddb41590a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb415d300 0x7fddb41590a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fddac007d90 tx=0x7fddac00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 shutdown_connections 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c040 0x7fdd8c06e500 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb415d300 0x7fddb41590a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb4150760 0x7fddb415c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 --2- 172.21.15.53:0/2759872329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb40b32e0 0x7fddb4142150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:08.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.759+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 >> 172.21.15.53:0/2759872329 conn(0x7fddb406a800 msgr2=0x7fddb4159b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:08.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.760+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 shutdown_connections 2024-01-16T15:13:08.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:08.760+0000 7fddbca00640 1 -- 172.21.15.53:0/2759872329 wait complete. 2024-01-16T15:13:08.844 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541418 2024-01-16T15:13:09.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.148+0000 7fb6a4200640 1 Processor -- start 2024-01-16T15:13:09.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.148+0000 7fb6a4200640 1 -- start start 2024-01-16T15:13:09.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.150+0000 7fb6a4200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c14b2c0 0x7fb69c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.150+0000 7fb6a4200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c0cfbf0 con 0x7fb69c071070 2024-01-16T15:13:09.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.150+0000 7fb6a4200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69c071690 con 0x7fb69c0cf7d0 2024-01-16T15:13:09.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.150+0000 7fb6a4200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c14b6a0 con 0x7fb69c14b2c0 2024-01-16T15:13:09.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.151+0000 7fb6a0e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fb69c0cf7d0 0x7fb69c070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:59196/0 (socket says 172.21.15.53:59196) 2024-01-16T15:13:09.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.151+0000 7fb6a0e00640 1 -- 172.21.15.53:0/3804668678 learned_addr learned my addr 172.21.15.53:0/3804668678 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:09.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.151+0000 7fb6a1800640 1 --2- 172.21.15.53:0/3804668678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c14b2c0 0x7fb69c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.151+0000 7fb6a1800640 1 -- 172.21.15.53:0/3804668678 >> v1:172.21.15.146:6789/0 conn(0x7fb69c0cf7d0 legacy=0x7fb69c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.151+0000 7fb6a1800640 1 -- 172.21.15.53:0/3804668678 >> v1:172.21.15.53:6789/0 conn(0x7fb69c071070 legacy=0x7fb69c0cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.151+0000 7fb6a1800640 1 -- 172.21.15.53:0/3804668678 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb69c071970 con 0x7fb69c14b2c0 2024-01-16T15:13:09.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.154+0000 7fb6a1800640 1 --2- 172.21.15.53:0/3804668678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c14b2c0 0x7fb69c0cf290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009dc0 tx=0x7fb68c02bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=663deab13d001b48 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.154+0000 7fb69be00640 1 -- 172.21.15.53:0/3804668678 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c02ccc0 con 0x7fb69c14b2c0 2024-01-16T15:13:09.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.154+0000 7fb69be00640 1 -- 172.21.15.53:0/3804668678 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb68c02ce20 con 0x7fb69c14b2c0 2024-01-16T15:13:09.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.155+0000 7fb69be00640 1 -- 172.21.15.53:0/3804668678 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb68c034660 con 0x7fb69c14b2c0 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 -- 172.21.15.53:0/3804668678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c14b2c0 msgr2=0x7fb69c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 --2- 172.21.15.53:0/3804668678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c14b2c0 0x7fb69c0cf290 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb68c009dc0 tx=0x7fb68c02bf10 comp rx=0 tx=0).stop 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 -- 172.21.15.53:0/3804668678 shutdown_connections 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 --2- 172.21.15.53:0/3804668678 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c14b2c0 0x7fb69c0cf290 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 -- 172.21.15.53:0/3804668678 >> 172.21.15.53:0/3804668678 conn(0x7fb69c06a800 msgr2=0x7fb69c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 -- 172.21.15.53:0/3804668678 shutdown_connections 2024-01-16T15:13:09.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.156+0000 7fb6a4200640 1 -- 172.21.15.53:0/3804668678 wait complete. 2024-01-16T15:13:09.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.159+0000 7fb6a4200640 1 Processor -- start 2024-01-16T15:13:09.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.159+0000 7fb6a4200640 1 -- start start 2024-01-16T15:13:09.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.160+0000 7fb6a4200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 0x7fb69c143dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.160+0000 7fb6a4200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c0cf7d0 0x7fb69c1467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a4200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb69c14b2c0 0x7fb69c146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a0e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c0cf7d0 0x7fb69c1467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a4200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c1527b0 con 0x7fb69c14b2c0 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a4200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c142ba0 con 0x7fb69c071070 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a4200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69c151c10 con 0x7fb69c0cf7d0 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 0x7fb69c143dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 0x7fb69c143dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:33216/0 (socket says 172.21.15.53:33216) 2024-01-16T15:13:09.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a1800640 1 -- 172.21.15.53:0/420597505 learned_addr learned my addr 172.21.15.53:0/420597505 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:09.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.161+0000 7fb6a2200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb69c14b2c0 0x7fb69c146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.162+0000 7fb6a2200640 1 -- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c0cf7d0 msgr2=0x7fb69c1467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.163+0000 7fb6a2200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c0cf7d0 0x7fb69c1467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.163+0000 7fb6a2200640 1 -- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 msgr2=0x7fb69c143dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.163+0000 7fb6a2200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 0x7fb69c143dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.163+0000 7fb6a2200640 1 -- 172.21.15.53:0/420597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c009a70 con 0x7fb69c14b2c0 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.163+0000 7fb6a1800640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 0x7fb69c143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:09.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.163+0000 7fb6a2200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb69c14b2c0 0x7fb69c146d00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb69400c340 tx=0x7fb69400c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.164+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6940052d0 con 0x7fb69c14b2c0 2024-01-16T15:13:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.164+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb69c151e80 con 0x7fb69c14b2c0 2024-01-16T15:13:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.164+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb694005430 con 0x7fb69c14b2c0 2024-01-16T15:13:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.164+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb694003830 con 0x7fb69c14b2c0 2024-01-16T15:13:09.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.164+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb69c152440 con 0x7fb69c14b2c0 2024-01-16T15:13:09.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.167+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69c152960 con 0x7fb69c14b2c0 2024-01-16T15:13:09.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.168+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 29) v1 ==== 90763+0+0 (secure 0 0 0) 0x7fb694003990 con 0x7fb69c14b2c0 2024-01-16T15:13:09.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.170+0000 7fb699600640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb67406c070 0x7fb67406e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.170+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb694087d30 con 0x7fb69c14b2c0 2024-01-16T15:13:09.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.171+0000 7fb6a1800640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb67406c070 0x7fb67406e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.173+0000 7fb6a1800640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb67406c070 0x7fb67406e530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb68c000f80 tx=0x7fb68c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.176+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb6940544b0 con 0x7fb69c14b2c0 2024-01-16T15:13:09.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.381+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fb6940878d0 con 0x7fb69c14b2c0 2024-01-16T15:13:09.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.540+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541418", "target": ["mon-mgr", ""]}) v1 -- 0x7fb69c0cbc40 con 0x7fb67406c070 2024-01-16T15:13:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.553+0000 7fb699600640 1 -- 172.21.15.53:0/420597505 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb68c034730 con 0x7fb67406c070 2024-01-16T15:13:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb67406c070 msgr2=0x7fb67406e530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb67406c070 0x7fb67406e530 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb68c000f80 tx=0x7fb68c030000 comp rx=0 tx=0).stop 2024-01-16T15:13:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb69c14b2c0 msgr2=0x7fb69c146d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb69c14b2c0 0x7fb69c146d00 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb69400c340 tx=0x7fb69400c800 comp rx=0 tx=0).stop 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 shutdown_connections 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb67406c070 0x7fb67406e530 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb69c14b2c0 0x7fb69c146d00 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb69c0cf7d0 0x7fb69c1467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 --2- 172.21.15.53:0/420597505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb69c071070 0x7fb69c143dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 >> 172.21.15.53:0/420597505 conn(0x7fb69c06a800 msgr2=0x7fb69c148da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 shutdown_connections 2024-01-16T15:13:09.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.556+0000 7fb6a4200640 1 -- 172.21.15.53:0/420597505 wait complete. 2024-01-16T15:13:09.552 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541416 2024-01-16T15:13:09.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.756+0000 7f37abe00640 1 Processor -- start 2024-01-16T15:13:09.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.757+0000 7f37abe00640 1 -- start start 2024-01-16T15:13:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.758+0000 7f37abe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f37ac071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.758+0000 7f37abe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f37ac0717d0 0x7f37ac0cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.758+0000 7f37abe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac070e50 con 0x7f37ac0cca40 2024-01-16T15:13:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.758+0000 7f37abe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac0723c0 con 0x7f37ac0717d0 2024-01-16T15:13:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.758+0000 7f37abe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0cce70 con 0x7f37ac143c40 2024-01-16T15:13:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37ab400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f37ac0cca40 0x7f37ac15eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44020/0 (socket says 172.21.15.53:44020) 2024-01-16T15:13:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37ab400640 1 -- 172.21.15.53:0/2365872011 learned_addr learned my addr 172.21.15.53:0/2365872011 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37a3e00640 1 --2- 172.21.15.53:0/2365872011 >> v2:172.21.15.146:3300/0 conn(0x7f37ac0717d0 0x7f37ac0cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37aaa00640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f37ac071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1331624685 0 0) 0x7f37ac070e50 con 0x7f37ac0cca40 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f378c003520 con 0x7f37ac0cca40 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3694839346 0 0) 0x7f378c003520 con 0x7f37ac0cca40 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.759+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 msgr2=0x7f37ac071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f37ac071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 >> v2:172.21.15.146:3300/0 conn(0x7f37ac0717d0 msgr2=0x7f37ac0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:09.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 --2- 172.21.15.53:0/2365872011 >> v2:172.21.15.146:3300/0 conn(0x7f37ac0717d0 0x7f37ac0cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac06e170 con 0x7f37ac0cca40 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f37a40019f0 con 0x7f37ac0cca40 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aaa00640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f37ac071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 >> v1:172.21.15.53:6789/0 conn(0x7f37ac0cca40 legacy=0x7f37ac15eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f378c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c003f00 0x7f378c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004d00 0x7f378c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0723c0 con 0x7f378c004d00 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0cce70 con 0x7f378c003f00 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f378c00ba80 con 0x7f37ac143c40 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37ab400640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004d00 0x7f378c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37a3e00640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c003f00 0x7f378c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.760+0000 7f37aaa00640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f378c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.761+0000 7f37ab400640 1 -- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 msgr2=0x7f378c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:09.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.761+0000 7f37ab400640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac143c40 0x7f378c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.761+0000 7f37ab400640 1 -- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c003f00 msgr2=0x7f378c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:09.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.761+0000 7f37ab400640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c003f00 0x7f378c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.761+0000 7f37ab400640 1 -- 172.21.15.53:0/2365872011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c003520 con 0x7f378c004d00 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.762+0000 7f37ab400640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004d00 0x7f378c005180 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f37a400a790 tx=0x7f378c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad885d1ef11717b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.762+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a4034630 con 0x7f378c004d00 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.762+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37a402f070 con 0x7f378c004d00 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.762+0000 7f37aa000640 1 -- 172.21.15.53:0/2365872011 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37a4034a30 con 0x7f378c004d00 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37abe00640 1 -- 172.21.15.53:0/2365872011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004d00 msgr2=0x7f378c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37abe00640 1 --2- 172.21.15.53:0/2365872011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f378c004d00 0x7f378c005180 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7f37a400a790 tx=0x7f378c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37ab400640 1 -- 172.21.15.53:0/2365872011 reap_dead start 2024-01-16T15:13:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37abe00640 1 -- 172.21.15.53:0/2365872011 shutdown_connections 2024-01-16T15:13:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37abe00640 1 -- 172.21.15.53:0/2365872011 >> 172.21.15.53:0/2365872011 conn(0x7f37ac06a800 msgr2=0x7f37ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37abe00640 1 -- 172.21.15.53:0/2365872011 shutdown_connections 2024-01-16T15:13:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.763+0000 7f37abe00640 1 -- 172.21.15.53:0/2365872011 wait complete. 2024-01-16T15:13:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.764+0000 7f37abe00640 1 Processor -- start 2024-01-16T15:13:09.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 -- start start 2024-01-16T15:13:09.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c004d00 0x7f37ac15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 0x7f37ac15f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac0a9b30 0x7f37ac1f3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac15db10 con 0x7f37ac15dfa0 2024-01-16T15:13:09.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac15e420 con 0x7f378c004d00 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.765+0000 7f37abe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac15cf00 con 0x7f37ac0a9b30 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 0x7f37ac15f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 0x7f37ac15f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55900/0 (socket says 172.21.15.53:55900) 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 -- 172.21.15.53:0/116352904 learned_addr learned my addr 172.21.15.53:0/116352904 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37ab400640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac0a9b30 0x7f37ac1f3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 -- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac0a9b30 msgr2=0x7f37ac1f3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac0a9b30 0x7f37ac1f3e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37aaa00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c004d00 0x7f37ac15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 -- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c004d00 msgr2=0x7f37ac15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c004d00 0x7f37ac15c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 -- 172.21.15.53:0/116352904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac15d1e0 con 0x7f37ac15dfa0 2024-01-16T15:13:09.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.766+0000 7f37a3e00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 0x7f37ac15f380 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f3794011600 tx=0x7f3794011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.767+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400f920 con 0x7f37ac15dfa0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.767+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379400fa80 con 0x7f37ac15dfa0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.767+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f379400fdf0 con 0x7f37ac15dfa0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.767+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac15d4c0 con 0x7f37ac15dfa0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.767+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac15f8c0 con 0x7f37ac15dfa0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.768+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac067450 con 0x7f37ac15dfa0 2024-01-16T15:13:09.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.769+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f3794015460 con 0x7f37ac15dfa0 2024-01-16T15:13:09.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.770+0000 7f37a2a00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f378406bfe0 0x7f378406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:09.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.770+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f3794088ac0 con 0x7f37ac15dfa0 2024-01-16T15:13:09.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.771+0000 7f37aaa00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f378406bfe0 0x7f378406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:09.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.772+0000 7f37aaa00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f378406bfe0 0x7f378406e4a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f379c006fd0 tx=0x7f379c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:09.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:09.775+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3794055380 con 0x7f37ac15dfa0 2024-01-16T15:13:10.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.067+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541416", "target": ["mon-mgr", ""]}) v1 -- 0x7f37ac0720e0 con 0x7f378406bfe0 2024-01-16T15:13:10.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.085+0000 7f37a2a00640 1 -- 172.21.15.53:0/116352904 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f379c00c040 con 0x7f378406bfe0 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f378406bfe0 msgr2=0x7f378406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f378406bfe0 0x7f378406e4a0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f379c006fd0 tx=0x7f379c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 msgr2=0x7f37ac15f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 0x7f37ac15f380 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f3794011600 tx=0x7f3794011ac0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 shutdown_connections 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f378406bfe0 0x7f378406e4a0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f37ac0a9b30 0x7f37ac1f3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f37ac15dfa0 0x7f37ac15f380 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 --2- 172.21.15.53:0/116352904 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f378c004d00 0x7f37ac15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 >> 172.21.15.53:0/116352904 conn(0x7f37ac06a800 msgr2=0x7f37ac141270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 shutdown_connections 2024-01-16T15:13:10.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.088+0000 7f37abe00640 1 -- 172.21.15.53:0/116352904 wait complete. 2024-01-16T15:13:10.083 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541410 2024-01-16T15:13:10.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.310+0000 7f1cbb600640 1 Processor -- start 2024-01-16T15:13:10.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.311+0000 7f1cbb600640 1 -- start start 2024-01-16T15:13:10.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.312+0000 7f1cbb600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 0x7f1cb4164cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.312+0000 7f1cbb600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb4071ae0 con 0x7f1cb40b4280 2024-01-16T15:13:10.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.312+0000 7f1cbb600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb40cdf90 con 0x7f1cb40b3c10 2024-01-16T15:13:10.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.312+0000 7f1cbb600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb4072000 con 0x7f1cb40cf430 2024-01-16T15:13:10.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.312+0000 7f1cb8c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1cb40b4280 0x7f1cb40b4660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44036/0 (socket says 172.21.15.53:44036) 2024-01-16T15:13:10.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb8c00640 1 -- 172.21.15.53:0/3580633894 learned_addr learned my addr 172.21.15.53:0/3580633894 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75813620 0 0) 0x7f1ca4001110 con 0x7f1cb40b4280 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c94003520 con 0x7f1cb40b4280 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb9600640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 0x7f1cb4164cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4156309798 0 0) 0x7f1c94003520 con 0x7f1cb40b4280 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 msgr2=0x7f1cb4164cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 0x7f1cb4164cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 >> v1:172.21.15.146:6789/0 conn(0x7f1cb40b3c10 legacy=0x7f1cb40713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb4071d20 con 0x7f1cb40b4280 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.313+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1ca4001800 con 0x7f1cb40b4280 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 >> v1:172.21.15.53:6789/0 conn(0x7f1cb40b4280 legacy=0x7f1cb40b4660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c940039a0 0x7f1c94003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 0x7f1c940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c94004d00 0x7f1c94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb4072000 con 0x7f1c94004d00 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9400bb90 con 0x7f1c940039a0 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9400df60 con 0x7f1cb40cf430 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb9600640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c94004d00 0x7f1c94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb8c00640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c940039a0 0x7f1c94003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb3e00640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 0x7f1c940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb9600640 1 -- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 msgr2=0x7f1c940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb9600640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb40cf430 0x7f1c940047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb9600640 1 -- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c940039a0 msgr2=0x7f1c94003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb9600640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c940039a0 0x7f1c94003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.314+0000 7f1cb9600640 1 -- 172.21.15.53:0/3580633894 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c94003520 con 0x7f1c94004d00 2024-01-16T15:13:10.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.315+0000 7f1cb9600640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c94004d00 0x7f1c94005180 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f1cac007330 tx=0x7f1c940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74374a21e2ec9348 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac02cca0 con 0x7f1c94004d00 2024-01-16T15:13:10.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac02ce00 con 0x7f1c94004d00 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cb3400640 1 -- 172.21.15.53:0/3580633894 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac034660 con 0x7f1c94004d00 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cbb600640 1 -- 172.21.15.53:0/3580633894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c94004d00 msgr2=0x7f1c94005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cbb600640 1 --2- 172.21.15.53:0/3580633894 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c94004d00 0x7f1c94005180 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f1cac007330 tx=0x7f1c940057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cb9600640 1 -- 172.21.15.53:0/3580633894 reap_dead start 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cbb600640 1 -- 172.21.15.53:0/3580633894 shutdown_connections 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cbb600640 1 -- 172.21.15.53:0/3580633894 >> 172.21.15.53:0/3580633894 conn(0x7f1cb40cfef0 msgr2=0x7f1cb40d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cbb600640 1 -- 172.21.15.53:0/3580633894 shutdown_connections 2024-01-16T15:13:10.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.316+0000 7f1cbb600640 1 -- 172.21.15.53:0/3580633894 wait complete. 2024-01-16T15:13:10.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.318+0000 7f1cbb600640 1 Processor -- start 2024-01-16T15:13:10.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.318+0000 7f1cbb600640 1 -- start start 2024-01-16T15:13:10.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cbb600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c94004d00 0x7f1cb415b670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.298 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cbb600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb4138d30 0x7f1cb413d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cbb600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 0x7f1cb40ce420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cbb600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb41391b0 con 0x7f1cb413fbf0 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cbb600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb4140070 con 0x7f1cb4138d30 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cbb600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb413d850 con 0x7f1c94004d00 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cb9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 0x7f1cb40ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cb9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 0x7f1cb40ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55914/0 (socket says 172.21.15.53:55914) 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.319+0000 7f1cb9600640 1 -- 172.21.15.53:0/200755101 learned_addr learned my addr 172.21.15.53:0/200755101 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb8c00640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c94004d00 0x7f1cb415b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb3e00640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb4138d30 0x7f1cb413d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb9600640 1 -- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c94004d00 msgr2=0x7f1cb415b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb9600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c94004d00 0x7f1cb415b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb9600640 1 -- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb4138d30 msgr2=0x7f1cb413d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb9600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb4138d30 0x7f1cb413d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb9600640 1 -- 172.21.15.53:0/200755101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb415a8b0 con 0x7f1cb413fbf0 2024-01-16T15:13:10.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb8c00640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c94004d00 0x7f1cb415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb9600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 0x7f1cb40ce420 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f1cac02c640 tx=0x7f1cac004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac003b90 con 0x7f1cb413fbf0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac003cf0 con 0x7f1cb413fbf0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac004260 con 0x7f1cb413fbf0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.320+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb415ab90 con 0x7f1cb413fbf0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.321+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb415b150 con 0x7f1cb413fbf0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.322+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb4139110 con 0x7f1cb413fbf0 2024-01-16T15:13:10.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.322+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1cac0043c0 con 0x7f1cb413fbf0 2024-01-16T15:13:10.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.323+0000 7f1cb0c00640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8c06bfb0 0x7f1c8c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.301 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.323+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f1cac0aabd0 con 0x7f1cb413fbf0 2024-01-16T15:13:10.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.324+0000 7f1cb8c00640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8c06bfb0 0x7f1c8c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.325+0000 7f1cb8c00640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8c06bfb0 0x7f1c8c06e470 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4001eb0 tx=0x7f1ca4000f50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.328+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1cac077200 con 0x7f1cb413fbf0 2024-01-16T15:13:10.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.544+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541410", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cb40cdc20 con 0x7f1c8c06bfb0 2024-01-16T15:13:10.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.586+0000 7f1cb0c00640 1 -- 172.21.15.53:0/200755101 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1ca4005e60 con 0x7f1c8c06bfb0 2024-01-16T15:13:10.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8c06bfb0 msgr2=0x7f1c8c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8c06bfb0 0x7f1c8c06e470 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f1ca4001eb0 tx=0x7f1ca4000f50 comp rx=0 tx=0).stop 2024-01-16T15:13:10.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 msgr2=0x7f1cb40ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 0x7f1cb40ce420 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f1cac02c640 tx=0x7f1cac004ab0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 shutdown_connections 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8c06bfb0 0x7f1c8c06e470 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb413fbf0 0x7f1cb40ce420 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb4138d30 0x7f1cb413d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 --2- 172.21.15.53:0/200755101 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c94004d00 0x7f1cb415b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 >> 172.21.15.53:0/200755101 conn(0x7f1cb40cfef0 msgr2=0x7f1cb40afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 shutdown_connections 2024-01-16T15:13:10.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.589+0000 7f1cbb600640 1 -- 172.21.15.53:0/200755101 wait complete. 2024-01-16T15:13:10.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54143 2024-01-16T15:13:10.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.830+0000 7fddd9000640 1 Processor -- start 2024-01-16T15:13:10.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.830+0000 7fddd9000640 1 -- start start 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddd4071200 0x7fddd406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd9000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd415deb0 con 0x7fddd40cd600 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd9000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd415e190 con 0x7fddd4157c30 2024-01-16T15:13:10.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd9000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd415ea30 con 0x7fddd4071200 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd2800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddd4071200 0x7fddd406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd2800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddd4071200 0x7fddd406dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55920/0 (socket says 172.21.15.53:55920) 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddd2800640 1 -- 172.21.15.53:0/1595936131 learned_addr learned my addr 172.21.15.53:0/1595936131 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.832+0000 7fddcbe00640 1 --1- 172.21.15.53:0/1595936131 >> v1:172.21.15.53:6789/0 conn(0x7fddd40cd600 0x7fddd4070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44044/0 (socket says 172.21.15.53:44044) 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.833+0000 7fddd2800640 1 -- 172.21.15.53:0/1595936131 >> v1:172.21.15.146:6789/0 conn(0x7fddd4157c30 legacy=0x7fddd41618f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.833+0000 7fddd2800640 1 -- 172.21.15.53:0/1595936131 >> v1:172.21.15.53:6789/0 conn(0x7fddd40cd600 legacy=0x7fddd4070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.833+0000 7fddd2800640 1 -- 172.21.15.53:0/1595936131 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd415e470 con 0x7fddd4071200 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.834+0000 7fddd2800640 1 --2- 172.21.15.53:0/1595936131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4071200 0x7fddd406dcb0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009e50 tx=0x7fddcc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=195041a962cd131d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.834+0000 7fddd1400640 1 -- 172.21.15.53:0/1595936131 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc02cb20 con 0x7fddd4071200 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.834+0000 7fddd1400640 1 -- 172.21.15.53:0/1595936131 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc02cc80 con 0x7fddd4071200 2024-01-16T15:13:10.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.834+0000 7fddd1400640 1 -- 172.21.15.53:0/1595936131 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc0343f0 con 0x7fddd4071200 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 -- 172.21.15.53:0/1595936131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4071200 msgr2=0x7fddd406dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 --2- 172.21.15.53:0/1595936131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4071200 0x7fddd406dcb0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009e50 tx=0x7fddcc02bf80 comp rx=0 tx=0).stop 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 -- 172.21.15.53:0/1595936131 shutdown_connections 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 --2- 172.21.15.53:0/1595936131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4071200 0x7fddd406dcb0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 -- 172.21.15.53:0/1595936131 >> 172.21.15.53:0/1595936131 conn(0x7fddd40cf700 msgr2=0x7fddd40cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 -- 172.21.15.53:0/1595936131 shutdown_connections 2024-01-16T15:13:10.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.835+0000 7fddd9000640 1 -- 172.21.15.53:0/1595936131 wait complete. 2024-01-16T15:13:10.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.838+0000 7fddd9000640 1 Processor -- start 2024-01-16T15:13:10.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.838+0000 7fddd9000640 1 -- start start 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.839+0000 7fddd9000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd4071200 0x7fddd4155800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.839+0000 7fddd9000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd40cd600 0x7fddd4153b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.839+0000 7fddd9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 0x7fddd4146190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.839+0000 7fddd9000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd4154d20 con 0x7fddd4157c30 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.839+0000 7fddd9000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd41f3430 con 0x7fddd4071200 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.839+0000 7fddd9000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd41554b0 con 0x7fddd40cd600 2024-01-16T15:13:10.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 0x7fddd4146190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 0x7fddd4146190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55934/0 (socket says 172.21.15.53:55934) 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 -- 172.21.15.53:0/3011915352 learned_addr learned my addr 172.21.15.53:0/3011915352 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd1e00640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd4071200 0x7fddd4155800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddcbe00640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd40cd600 0x7fddd4153b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 -- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd40cd600 msgr2=0x7fddd4153b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd40cd600 0x7fddd4153b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 -- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd4071200 msgr2=0x7fddd4155800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd4071200 0x7fddd4155800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 -- 172.21.15.53:0/3011915352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc009b30 con 0x7fddd4157c30 2024-01-16T15:13:10.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.840+0000 7fddd2800640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 0x7fddd4146190 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009e50 tx=0x7fddcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.841+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc009510 con 0x7fddd4157c30 2024-01-16T15:13:10.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.841+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc03e450 con 0x7fddd4157c30 2024-01-16T15:13:10.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.841+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd41466d0 con 0x7fddd4157c30 2024-01-16T15:13:10.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.841+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddcc03e800 con 0x7fddd4157c30 2024-01-16T15:13:10.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.841+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd4154480 con 0x7fddd4157c30 2024-01-16T15:13:10.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.842+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd41f35a0 con 0x7fddd4157c30 2024-01-16T15:13:10.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.843+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7fddcc03e960 con 0x7fddd4157c30 2024-01-16T15:13:10.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.844+0000 7fddca000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddac06bfb0 0x7fddac06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:10.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.844+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7fddcc0ab600 con 0x7fddd4157c30 2024-01-16T15:13:10.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.844+0000 7fddd1e00640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddac06bfb0 0x7fddac06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:10.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.846+0000 7fddd1e00640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddac06bfb0 0x7fddac06e470 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fddc4004ea0 tx=0x7fddc4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:10.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:10.848+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddd41f35a0 con 0x7fddd4157c30 2024-01-16T15:13:11.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.207+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54143", "target": ["mon-mgr", ""]}) v1 -- 0x7fddd41f35a0 con 0x7fddac06bfb0 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.222+0000 7fddca000640 1 -- 172.21.15.53:0/3011915352 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fddd41f35a0 con 0x7fddac06bfb0 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddac06bfb0 msgr2=0x7fddac06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddac06bfb0 0x7fddac06e470 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fddc4004ea0 tx=0x7fddc4009040 comp rx=0 tx=0).stop 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 msgr2=0x7fddd4146190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 0x7fddd4146190 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fddcc009e50 tx=0x7fddcc004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 shutdown_connections 2024-01-16T15:13:11.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddac06bfb0 0x7fddac06e470 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddd4157c30 0x7fddd4146190 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddd40cd600 0x7fddd4153b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 --2- 172.21.15.53:0/3011915352 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddd4071200 0x7fddd4155800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 >> 172.21.15.53:0/3011915352 conn(0x7fddd40cf700 msgr2=0x7fddd4070af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 shutdown_connections 2024-01-16T15:13:11.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.224+0000 7fddd9000640 1 -- 172.21.15.53:0/3011915352 wait complete. 2024-01-16T15:13:11.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541411 2024-01-16T15:13:11.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.436+0000 7f6f8ea00640 1 Processor -- start 2024-01-16T15:13:11.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.437+0000 7f6f8ea00640 1 -- start start 2024-01-16T15:13:11.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6f88160920 0x7f6f880b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8ea00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f880cfea0 0x7f6f88161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8ea00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f880d0a10 con 0x7f6f880cf9f0 2024-01-16T15:13:11.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8ea00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f880d1f50 con 0x7f6f88160920 2024-01-16T15:13:11.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8ea00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f880cf080 con 0x7f6f880cfea0 2024-01-16T15:13:11.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6f880cf9f0 0x7f6f88161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:44046/0 (socket says 172.21.15.53:44046) 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8cc00640 1 -- 172.21.15.53:0/3496104890 learned_addr learned my addr 172.21.15.53:0/3496104890 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.438+0000 7f6f8d600640 1 --2- 172.21.15.53:0/3496104890 >> v2:172.21.15.53:3300/0 conn(0x7f6f88160920 0x7f6f880b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.439+0000 7f6f8e000640 1 --2- 172.21.15.53:0/3496104890 >> v2:172.21.15.146:3300/0 conn(0x7f6f880cfea0 0x7f6f88161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.439+0000 7f6f8d600640 1 -- 172.21.15.53:0/3496104890 >> v2:172.21.15.146:3300/0 conn(0x7f6f880cfea0 msgr2=0x7f6f88161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.439+0000 7f6f8d600640 1 --2- 172.21.15.53:0/3496104890 >> v2:172.21.15.146:3300/0 conn(0x7f6f880cfea0 0x7f6f88161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.439+0000 7f6f8d600640 1 -- 172.21.15.53:0/3496104890 >> v1:172.21.15.53:6789/0 conn(0x7f6f880cf9f0 legacy=0x7f6f88161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.439+0000 7f6f8d600640 1 -- 172.21.15.53:0/3496104890 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f880d0310 con 0x7f6f88160920 2024-01-16T15:13:11.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.440+0000 7f6f8d600640 1 --2- 172.21.15.53:0/3496104890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f88160920 0x7f6f880b3f30 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f6f8808c2c0 tx=0x7f6f7802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d72f264d545aa0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:11.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.440+0000 7f6f87e00640 1 -- 172.21.15.53:0/3496104890 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7802cc70 con 0x7f6f88160920 2024-01-16T15:13:11.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.440+0000 7f6f87e00640 1 -- 172.21.15.53:0/3496104890 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f7802cdd0 con 0x7f6f88160920 2024-01-16T15:13:11.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.440+0000 7f6f87e00640 1 -- 172.21.15.53:0/3496104890 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f78034660 con 0x7f6f88160920 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 -- 172.21.15.53:0/3496104890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f88160920 msgr2=0x7f6f880b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/3496104890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f88160920 0x7f6f880b3f30 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f6f8808c2c0 tx=0x7f6f7802bec0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 -- 172.21.15.53:0/3496104890 shutdown_connections 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/3496104890 >> v2:172.21.15.146:3300/0 conn(0x7f6f880cfea0 0x7f6f88161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/3496104890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f88160920 0x7f6f880b3f30 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 -- 172.21.15.53:0/3496104890 >> 172.21.15.53:0/3496104890 conn(0x7f6f880d3090 msgr2=0x7f6f880d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 -- 172.21.15.53:0/3496104890 shutdown_connections 2024-01-16T15:13:11.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.441+0000 7f6f8ea00640 1 -- 172.21.15.53:0/3496104890 wait complete. 2024-01-16T15:13:11.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.443+0000 7f6f8ea00640 1 Processor -- start 2024-01-16T15:13:11.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.443+0000 7f6f8ea00640 1 -- start start 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8ea00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f880cf9f0 0x7f6f8815a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 0x7f6f880b72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8ea00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f88160920 0x7f6f880b7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8ea00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880cb830 con 0x7f6f880cfea0 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8ea00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f880cbae0 con 0x7f6f88160920 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8ea00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8815d360 con 0x7f6f880cf9f0 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 0x7f6f880b72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 0x7f6f880b72f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55960/0 (socket says 172.21.15.53:55960) 2024-01-16T15:13:11.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 -- 172.21.15.53:0/1699172011 learned_addr learned my addr 172.21.15.53:0/1699172011 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:11.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 -- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f880cf9f0 msgr2=0x7f6f8815a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f880cf9f0 0x7f6f8815a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 -- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f88160920 msgr2=0x7f6f880b7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.444+0000 7f6f8cc00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f88160920 0x7f6f880b7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f8cc00640 1 -- 172.21.15.53:0/1699172011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78009a70 con 0x7f6f880cfea0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f8cc00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 0x7f6f880b72f0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000c480 tx=0x7f6f7000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f70005ce0 con 0x7f6f880cfea0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f70005e40 con 0x7f6f880cfea0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f700072f0 con 0x7f6f880cfea0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8815d640 con 0x7f6f880cfea0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.445+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8815ab30 con 0x7f6f880cfea0 2024-01-16T15:13:11.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.446+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f880cbca0 con 0x7f6f880cfea0 2024-01-16T15:13:11.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.448+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f6f70007510 con 0x7f6f880cfea0 2024-01-16T15:13:11.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.449+0000 7f6f85600640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f6006bfb0 0x7f6f6006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:11.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.449+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f6f70087b00 con 0x7f6f880cfea0 2024-01-16T15:13:11.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.450+0000 7f6f8d600640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f6006bfb0 0x7f6f6006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:11.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.451+0000 7f6f8d600640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f6006bfb0 0x7f6f6006e470 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6f78036000 tx=0x7f6f7802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:11.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.453+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f70054470 con 0x7f6f880cfea0 2024-01-16T15:13:11.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.677+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541411", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f880ceac0 con 0x7f6f6006bfb0 2024-01-16T15:13:11.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.722+0000 7f6f85600640 1 -- 172.21.15.53:0/1699172011 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6f78034440 con 0x7f6f6006bfb0 2024-01-16T15:13:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f6006bfb0 msgr2=0x7f6f6006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f6006bfb0 0x7f6f6006e470 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f6f78036000 tx=0x7f6f7802e000 comp rx=0 tx=0).stop 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 msgr2=0x7f6f880b72f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 0x7f6f880b72f0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f6f7000c480 tx=0x7f6f7000c940 comp rx=0 tx=0).stop 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 shutdown_connections 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f6006bfb0 0x7f6f6006e470 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.724+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f88160920 0x7f6f880b7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.725+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f880cfea0 0x7f6f880b72f0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.725+0000 7f6f8ea00640 1 --2- 172.21.15.53:0/1699172011 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f880cf9f0 0x7f6f8815a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.725+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 >> 172.21.15.53:0/1699172011 conn(0x7f6f880d3090 msgr2=0x7f6f8806c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.725+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 shutdown_connections 2024-01-16T15:13:11.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:11.725+0000 7f6f8ea00640 1 -- 172.21.15.53:0/1699172011 wait complete. 2024-01-16T15:13:11.723 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54148 2024-01-16T15:13:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.022+0000 7f9606e00640 1 Processor -- start 2024-01-16T15:13:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.022+0000 7f9606e00640 1 -- start start 2024-01-16T15:13:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.024+0000 7f9606e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000b15f0 0x7f9600072290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9606e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96000cc7f0 0x7f96000ccc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9606e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9600070160 con 0x7f96001536f0 2024-01-16T15:13:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9606e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f960015a010 con 0x7f96000cc7f0 2024-01-16T15:13:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9606e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f960015a2f0 con 0x7f96000b15f0 2024-01-16T15:13:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9606400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96000cc7f0 0x7f96000ccc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9605000640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f96001536f0 0x7f9600159750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:36140/0 (socket says 172.21.15.53:36140) 2024-01-16T15:13:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.025+0000 7f9605000640 1 -- 172.21.15.53:0/1136505171 learned_addr learned my addr 172.21.15.53:0/1136505171 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.026+0000 7f9605a00640 1 --2- 172.21.15.53:0/1136505171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000b15f0 0x7f9600072290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.026+0000 7f95ffe00640 1 -- 172.21.15.53:0/1136505171 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3334090967 0 0) 0x7f9600070160 con 0x7f96001536f0 2024-01-16T15:13:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.026+0000 7f95ffe00640 1 -- 172.21.15.53:0/1136505171 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e0003520 con 0x7f96001536f0 2024-01-16T15:13:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.027+0000 7f9605a00640 1 -- 172.21.15.53:0/1136505171 >> v2:172.21.15.146:3300/0 conn(0x7f96000cc7f0 msgr2=0x7f96000ccc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.027+0000 7f9605a00640 1 --2- 172.21.15.53:0/1136505171 >> v2:172.21.15.146:3300/0 conn(0x7f96000cc7f0 0x7f96000ccc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.027+0000 7f9605a00640 1 -- 172.21.15.53:0/1136505171 >> v1:172.21.15.146:6789/0 conn(0x7f96001536f0 legacy=0x7f9600159750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.027+0000 7f9605a00640 1 -- 172.21.15.53:0/1136505171 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960015a5d0 con 0x7f96000b15f0 2024-01-16T15:13:12.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.029+0000 7f9605a00640 1 --2- 172.21.15.53:0/1136505171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000b15f0 0x7f9600072290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f95f00097a0 tx=0x7f95f002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6198611aafbe4d83 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.029+0000 7f95ffe00640 1 -- 172.21.15.53:0/1136505171 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f002d040 con 0x7f96000b15f0 2024-01-16T15:13:12.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.029+0000 7f95ffe00640 1 -- 172.21.15.53:0/1136505171 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f0031810 con 0x7f96000b15f0 2024-01-16T15:13:12.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.029+0000 7f95ffe00640 1 -- 172.21.15.53:0/1136505171 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0031ba0 con 0x7f96000b15f0 2024-01-16T15:13:12.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.030+0000 7f9606e00640 1 -- 172.21.15.53:0/1136505171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000b15f0 msgr2=0x7f9600072290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.030+0000 7f9606e00640 1 --2- 172.21.15.53:0/1136505171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000b15f0 0x7f9600072290 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f95f00097a0 tx=0x7f95f002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.031+0000 7f9606e00640 1 -- 172.21.15.53:0/1136505171 shutdown_connections 2024-01-16T15:13:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.031+0000 7f9606e00640 1 --2- 172.21.15.53:0/1136505171 >> v2:172.21.15.146:3300/0 conn(0x7f96000cc7f0 0x7f96000ccc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.031+0000 7f9606e00640 1 --2- 172.21.15.53:0/1136505171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000b15f0 0x7f9600072290 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.031+0000 7f9606e00640 1 -- 172.21.15.53:0/1136505171 >> 172.21.15.53:0/1136505171 conn(0x7f96000b0b00 msgr2=0x7f96000b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.031+0000 7f9606e00640 1 -- 172.21.15.53:0/1136505171 shutdown_connections 2024-01-16T15:13:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.031+0000 7f9606e00640 1 -- 172.21.15.53:0/1136505171 wait complete. 2024-01-16T15:13:12.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.033+0000 7f9606e00640 1 Processor -- start 2024-01-16T15:13:12.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.033+0000 7f9606e00640 1 -- start start 2024-01-16T15:13:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96000b15f0 0x7f960015ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000cc7f0 0x7f9600158940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 0x7f9600156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96000c4d20 con 0x7f96001536f0 2024-01-16T15:13:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96000c40a0 con 0x7f96000b15f0 2024-01-16T15:13:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96000c4380 con 0x7f96000cc7f0 2024-01-16T15:13:12.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 0x7f9600156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9605000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000cc7f0 0x7f9600158940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 0x7f9600156d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50922/0 (socket says 172.21.15.53:50922) 2024-01-16T15:13:12.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9606400640 1 -- 172.21.15.53:0/2561476627 learned_addr learned my addr 172.21.15.53:0/2561476627 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:12.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.034+0000 7f9605a00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96000b15f0 0x7f960015ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9605a00640 1 -- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000cc7f0 msgr2=0x7f9600158940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9605a00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000cc7f0 0x7f9600158940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9605a00640 1 -- 172.21.15.53:0/2561476627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 msgr2=0x7f9600156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9605a00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 0x7f9600156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9605a00640 1 -- 172.21.15.53:0/2561476627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f0009400 con 0x7f96000b15f0 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9606400640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 0x7f9600156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.035+0000 7f9605a00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96000b15f0 0x7f960015ab60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f95f0033000 tx=0x7f95f00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.036+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f003ace0 con 0x7f96000b15f0 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.036+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95f003ae60 con 0x7f96000b15f0 2024-01-16T15:13:12.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.036+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0044640 con 0x7f96000b15f0 2024-01-16T15:13:12.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.036+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96000c4660 con 0x7f96000b15f0 2024-01-16T15:13:12.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.036+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96000c3d70 con 0x7f96000b15f0 2024-01-16T15:13:12.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.037+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96000c4e60 con 0x7f96000b15f0 2024-01-16T15:13:12.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.038+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f95f00447a0 con 0x7f96000b15f0 2024-01-16T15:13:12.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.039+0000 7f95fd600640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95d806c000 0x7f95d806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.039+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f95f00aad80 con 0x7f96000b15f0 2024-01-16T15:13:12.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.039+0000 7f9605000640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95d806c000 0x7f95d806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.040+0000 7f9605000640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95d806c000 0x7f95d806e4c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f960006dea0 tx=0x7f95e8007d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.043+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f95f00776f0 con 0x7f96000b15f0 2024-01-16T15:13:12.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.264+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54148", "target": ["mon-mgr", ""]}) v1 -- 0x7f9600153b70 con 0x7f95d806c000 2024-01-16T15:13:12.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.308+0000 7f95fd600640 1 -- 172.21.15.53:0/2561476627 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f95e80072a0 con 0x7f95d806c000 2024-01-16T15:13:12.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95d806c000 msgr2=0x7f95d806e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95d806c000 0x7f95d806e4c0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f960006dea0 tx=0x7f95e8007d50 comp rx=0 tx=0).stop 2024-01-16T15:13:12.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96000b15f0 msgr2=0x7f960015ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96000b15f0 0x7f960015ab60 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f95f0033000 tx=0x7f95f00046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 shutdown_connections 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95d806c000 0x7f95d806e4c0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96001536f0 0x7f9600156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96000cc7f0 0x7f9600158940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 --2- 172.21.15.53:0/2561476627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96000b15f0 0x7f960015ab60 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 >> 172.21.15.53:0/2561476627 conn(0x7f96000b0b00 msgr2=0x7f96000b0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 shutdown_connections 2024-01-16T15:13:12.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.311+0000 7f9606e00640 1 -- 172.21.15.53:0/2561476627 wait complete. 2024-01-16T15:13:12.311 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54144 2024-01-16T15:13:12.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.559+0000 7f4c7b200640 1 Processor -- start 2024-01-16T15:13:12.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.559+0000 7f4c7b200640 1 -- start start 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c7b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4c7413e360 0x7f4c740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c7b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c7415f9c0 0x7f4c740cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c7b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c7415c5f0 con 0x7f4c7413df80 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c7b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7406c470 con 0x7f4c7413e360 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c7b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7406c750 con 0x7f4c7415f9c0 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c73e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4c7413df80 0x7f4c7415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48352/0 (socket says 172.21.15.53:48352) 2024-01-16T15:13:12.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c73e00640 1 -- 172.21.15.53:0/2338363643 learned_addr learned my addr 172.21.15.53:0/2338363643 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:12.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.561+0000 7f4c6be00640 1 --2- 172.21.15.53:0/2338363643 >> v2:172.21.15.53:3300/0 conn(0x7f4c7413e360 0x7f4c740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c79200640 1 --2- 172.21.15.53:0/2338363643 >> v2:172.21.15.146:3300/0 conn(0x7f4c7415f9c0 0x7f4c740cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c73400640 1 -- 172.21.15.53:0/2338363643 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596063766 0 0) 0x7f4c7415c5f0 con 0x7f4c7413df80 2024-01-16T15:13:12.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c73400640 1 -- 172.21.15.53:0/2338363643 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c54003520 con 0x7f4c7413df80 2024-01-16T15:13:12.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c6be00640 1 -- 172.21.15.53:0/2338363643 >> v2:172.21.15.146:3300/0 conn(0x7f4c7415f9c0 msgr2=0x7f4c740cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:12.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c6be00640 1 --2- 172.21.15.53:0/2338363643 >> v2:172.21.15.146:3300/0 conn(0x7f4c7415f9c0 0x7f4c740cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c6be00640 1 -- 172.21.15.53:0/2338363643 >> v1:172.21.15.53:6789/0 conn(0x7f4c7413df80 legacy=0x7f4c7415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.562+0000 7f4c6be00640 1 -- 172.21.15.53:0/2338363643 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7406ca30 con 0x7f4c7413e360 2024-01-16T15:13:12.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.563+0000 7f4c6be00640 1 --2- 172.21.15.53:0/2338363643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413e360 0x7f4c740cf290 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f4c740ccae0 tx=0x7f4c5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e853538bcbf5780 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.563+0000 7f4c73400640 1 -- 172.21.15.53:0/2338363643 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c031830 con 0x7f4c7413e360 2024-01-16T15:13:12.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.563+0000 7f4c73400640 1 -- 172.21.15.53:0/2338363643 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c031990 con 0x7f4c7413e360 2024-01-16T15:13:12.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.563+0000 7f4c73400640 1 -- 172.21.15.53:0/2338363643 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c031d20 con 0x7f4c7413e360 2024-01-16T15:13:12.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 -- 172.21.15.53:0/2338363643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413e360 msgr2=0x7f4c740cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 --2- 172.21.15.53:0/2338363643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413e360 0x7f4c740cf290 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f4c740ccae0 tx=0x7f4c5c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 -- 172.21.15.53:0/2338363643 shutdown_connections 2024-01-16T15:13:12.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 --2- 172.21.15.53:0/2338363643 >> v2:172.21.15.146:3300/0 conn(0x7f4c7415f9c0 0x7f4c740cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 --2- 172.21.15.53:0/2338363643 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413e360 0x7f4c740cf290 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 -- 172.21.15.53:0/2338363643 >> 172.21.15.53:0/2338363643 conn(0x7f4c7406a800 msgr2=0x7f4c7406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:12.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 -- 172.21.15.53:0/2338363643 shutdown_connections 2024-01-16T15:13:12.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.564+0000 7f4c7b200640 1 -- 172.21.15.53:0/2338363643 wait complete. 2024-01-16T15:13:12.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.566+0000 7f4c7b200640 1 Processor -- start 2024-01-16T15:13:12.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.566+0000 7f4c7b200640 1 -- start start 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c7b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 0x7f4c7413d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c7b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c7413e360 0x7f4c740ab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c7b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c7415f9c0 0x7f4c740abef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c7b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a84c0 con 0x7f4c7413df80 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c7b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a8780 con 0x7f4c7413e360 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c7b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a8a60 con 0x7f4c7415f9c0 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 0x7f4c7413d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 0x7f4c7413d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50940/0 (socket says 172.21.15.53:50940) 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.567+0000 7f4c73e00640 1 -- 172.21.15.53:0/691733328 learned_addr learned my addr 172.21.15.53:0/691733328 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:12.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c6be00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c7413e360 0x7f4c740ab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c73e00640 1 -- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c7415f9c0 msgr2=0x7f4c740abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c73e00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c7415f9c0 0x7f4c740abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c73e00640 1 -- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c7413e360 msgr2=0x7f4c740ab9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c73e00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c7413e360 0x7f4c740ab9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c73e00640 1 -- 172.21.15.53:0/691733328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c009400 con 0x7f4c7413df80 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c6be00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c7413e360 0x7f4c740ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:12.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c73e00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 0x7f4c7413d710 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400c400 tx=0x7f4c6400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c64005ce0 con 0x7f4c7413df80 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c64005e40 con 0x7f4c7413df80 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c640072d0 con 0x7f4c7413df80 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.568+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c740a8d40 con 0x7f4c7413df80 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.569+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c740a9300 con 0x7f4c7413df80 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.570+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c740675d0 con 0x7f4c7413df80 2024-01-16T15:13:12.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.571+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f4c640074b0 con 0x7f4c7413df80 2024-01-16T15:13:12.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.571+0000 7f4c70c00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c4c06bfb0 0x7f4c4c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:12.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.572+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f4c64087b00 con 0x7f4c7413df80 2024-01-16T15:13:12.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.572+0000 7f4c6be00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c4c06bfb0 0x7f4c4c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:12.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.573+0000 7f4c6be00640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c4c06bfb0 0x7f4c4c06e470 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c036000 tx=0x7f4c5c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:12.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.576+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c64054470 con 0x7f4c7413df80 2024-01-16T15:13:12.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.796+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54144", "target": ["mon-mgr", ""]}) v1 -- 0x7f4c7406abe0 con 0x7f4c4c06bfb0 2024-01-16T15:13:12.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.840+0000 7f4c70c00640 1 -- 172.21.15.53:0/691733328 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4c7406abe0 con 0x7f4c4c06bfb0 2024-01-16T15:13:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c4c06bfb0 msgr2=0x7f4c4c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c4c06bfb0 0x7f4c4c06e470 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c036000 tx=0x7f4c5c008030 comp rx=0 tx=0).stop 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 msgr2=0x7f4c7413d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 0x7f4c7413d710 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7f4c6400c400 tx=0x7f4c6400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 shutdown_connections 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c4c06bfb0 0x7f4c4c06e470 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c7415f9c0 0x7f4c740abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c7413e360 0x7f4c740ab9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 --2- 172.21.15.53:0/691733328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c7413df80 0x7f4c7413d710 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.843+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 >> 172.21.15.53:0/691733328 conn(0x7f4c7406a800 msgr2=0x7f4c740b3cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.844+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 shutdown_connections 2024-01-16T15:13:12.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:12.844+0000 7f4c7b200640 1 -- 172.21.15.53:0/691733328 wait complete. 2024-01-16T15:13:12.842 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541413 2024-01-16T15:13:13.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.154+0000 7f63e7200640 1 Processor -- start 2024-01-16T15:13:13.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.155+0000 7f63e7200640 1 -- start start 2024-01-16T15:13:13.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.158+0000 7f63e7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63e0072230 0x7f63e0150760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.158+0000 7f63e7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e00b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.158+0000 7f63e7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63e0071190 con 0x7f63e00cf290 2024-01-16T15:13:13.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.158+0000 7f63e7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63e0072610 con 0x7f63e0072230 2024-01-16T15:13:13.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.158+0000 7f63e7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e0071470 con 0x7f63e015da10 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.158+0000 7f63e5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e00b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.159+0000 7f63e5200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e00b2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:40794/0 (socket says 172.21.15.53:40794) 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.159+0000 7f63e5200640 1 -- 172.21.15.53:0/1222136692 learned_addr learned my addr 172.21.15.53:0/1222136692 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.159+0000 7f63dfe00640 1 --2- 172.21.15.53:0/1222136692 >> v2:172.21.15.146:3300/0 conn(0x7f63e0072230 0x7f63e0150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.159+0000 7f63dfe00640 1 -- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 msgr2=0x7f63e00b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.159+0000 7f63dfe00640 1 --2- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e00b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.159+0000 7f63dfe00640 1 -- 172.21.15.53:0/1222136692 >> v1:172.21.15.53:6789/0 conn(0x7f63e00cf290 legacy=0x7f63e015d300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-01-16T15:13:13.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.160+0000 7f63dfe00640 1 -- 172.21.15.53:0/1222136692 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63e0071750 con 0x7f63e0072230 2024-01-16T15:13:13.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.162+0000 7f63dfe00640 1 --2- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e0150760 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f63d00097a0 tx=0x7f63d002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8ee1d954dfbdf31 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.163+0000 7f63dea00640 1 -- 172.21.15.53:0/1222136692 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d002e070 con 0x7f63e0072230 2024-01-16T15:13:13.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.163+0000 7f63dea00640 1 -- 172.21.15.53:0/1222136692 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00315a0 con 0x7f63e0072230 2024-01-16T15:13:13.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.163+0000 7f63dea00640 1 -- 172.21.15.53:0/1222136692 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0031930 con 0x7f63e0072230 2024-01-16T15:13:13.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.164+0000 7f63e7200640 1 -- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 msgr2=0x7f63e0150760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.164+0000 7f63e7200640 1 --2- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e0150760 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f63d00097a0 tx=0x7f63d002b840 comp rx=0 tx=0).stop 2024-01-16T15:13:13.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.164+0000 7f63e7200640 1 -- 172.21.15.53:0/1222136692 shutdown_connections 2024-01-16T15:13:13.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.164+0000 7f63e7200640 1 --2- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e00b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.164+0000 7f63e7200640 1 --2- 172.21.15.53:0/1222136692 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e0150760 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.164+0000 7f63e7200640 1 -- 172.21.15.53:0/1222136692 >> 172.21.15.53:0/1222136692 conn(0x7f63e006a800 msgr2=0x7f63e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.165+0000 7f63e7200640 1 -- 172.21.15.53:0/1222136692 shutdown_connections 2024-01-16T15:13:13.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.165+0000 7f63e7200640 1 -- 172.21.15.53:0/1222136692 wait complete. 2024-01-16T15:13:13.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.167+0000 7f63e7200640 1 Processor -- start 2024-01-16T15:13:13.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.167+0000 7f63e7200640 1 -- start start 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.170+0000 7f63e7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e01544d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.170+0000 7f63e7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00cf290 0x7f63e0142160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.170+0000 7f63e7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e01ecce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.170+0000 7f63e7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e01403d0 con 0x7f63e00cf290 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.170+0000 7f63e7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63e013fda0 con 0x7f63e0072230 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.170+0000 7f63e7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63e01571d0 con 0x7f63e015da10 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.171+0000 7f63df400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00cf290 0x7f63e0142160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.171+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e01544d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.171+0000 7f63dfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e01544d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:39456/0 (socket says 172.21.15.53:39456) 2024-01-16T15:13:13.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.171+0000 7f63dfe00640 1 -- 172.21.15.53:0/3512842667 learned_addr learned my addr 172.21.15.53:0/3512842667 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:13.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63e5200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e01ecce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63dfe00640 1 -- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 msgr2=0x7f63e01ecce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63dfe00640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e01ecce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63dfe00640 1 -- 172.21.15.53:0/3512842667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00cf290 msgr2=0x7f63e0142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63dfe00640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00cf290 0x7f63e0142160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63dfe00640 1 -- 172.21.15.53:0/3512842667 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0009400 con 0x7f63e0072230 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.172+0000 7f63df400640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00cf290 0x7f63e0142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:13.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.173+0000 7f63dfe00640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e01544d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.173+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0032840 con 0x7f63e0072230 2024-01-16T15:13:13.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.174+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63d00329a0 con 0x7f63e0072230 2024-01-16T15:13:13.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.174+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63e01574b0 con 0x7f63e0072230 2024-01-16T15:13:13.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.174+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63e01426a0 con 0x7f63e0072230 2024-01-16T15:13:13.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.174+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63d0032d50 con 0x7f63e0072230 2024-01-16T15:13:13.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.177+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b0005250 con 0x7f63e0072230 2024-01-16T15:13:13.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.180+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mon.1 v2:172.21.15.146:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f63d004e450 con 0x7f63e0072230 2024-01-16T15:13:13.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.181+0000 7f63d7e00640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63b806c000 0x7f63b806e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.182+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mon.1 v2:172.21.15.146:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f63d00aa4f0 con 0x7f63e0072230 2024-01-16T15:13:13.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.183+0000 7f63df400640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63b806c000 0x7f63b806e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.186+0000 7f63df400640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63b806c000 0x7f63b806e4c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f63e014f960 tx=0x7f63c8007e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.187+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mon.1 v2:172.21.15.146:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63d0076b60 con 0x7f63e0072230 2024-01-16T15:13:13.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.406+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541413", "target": ["mon-mgr", ""]}) v1 -- 0x7f63b0000bc0 con 0x7f63b806c000 2024-01-16T15:13:13.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.448+0000 7f63d7e00640 1 -- 172.21.15.53:0/3512842667 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f63b0000bc0 con 0x7f63b806c000 2024-01-16T15:13:13.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63b806c000 msgr2=0x7f63b806e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63b806c000 0x7f63b806e4c0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f63e014f960 tx=0x7f63c8007e30 comp rx=0 tx=0).stop 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 msgr2=0x7f63e01544d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e01544d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f63d0000c00 tx=0x7f63d00046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 shutdown_connections 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63b806c000 0x7f63b806e4c0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63e015da10 0x7f63e01ecce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63e00cf290 0x7f63e0142160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 --2- 172.21.15.53:0/3512842667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63e0072230 0x7f63e01544d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.450+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 >> 172.21.15.53:0/3512842667 conn(0x7f63e006a800 msgr2=0x7f63e006d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.451+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 shutdown_connections 2024-01-16T15:13:13.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.451+0000 7f63e7200640 1 -- 172.21.15.53:0/3512842667 wait complete. 2024-01-16T15:13:13.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54146 2024-01-16T15:13:13.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.675+0000 7f8be0a00640 1 Processor -- start 2024-01-16T15:13:13.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.675+0000 7f8be0a00640 1 -- start start 2024-01-16T15:13:13.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8be0a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8bdc160920 0x7f8bdc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8be0a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc0d0a10 con 0x7f8bdc0cfea0 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8be0a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bdc161e30 con 0x7f8bdc0cf9f0 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8be0a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bdc0d1f50 con 0x7f8bdc160920 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8bdb200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8bdc0cfea0 0x7f8bdc161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48380/0 (socket says 172.21.15.53:48380) 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8bdb200640 1 -- 172.21.15.53:0/3913355377 learned_addr learned my addr 172.21.15.53:0/3913355377 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550273861 0 0) 0x7f8bdc0d0a10 con 0x7f8bdc0cfea0 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8bda800640 1 --2- 172.21.15.53:0/3913355377 >> v2:172.21.15.146:3300/0 conn(0x7f8bdc160920 0x7f8bdc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.677+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bbc003520 con 0x7f8bdc0cfea0 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1926905580 0 0) 0x7f8bbc003520 con 0x7f8bdc0cfea0 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 >> v2:172.21.15.146:3300/0 conn(0x7f8bdc160920 msgr2=0x7f8bdc0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 --2- 172.21.15.53:0/3913355377 >> v2:172.21.15.146:3300/0 conn(0x7f8bdc160920 0x7f8bdc0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 >> v1:172.21.15.146:6789/0 conn(0x7f8bdc0cf9f0 legacy=0x7f8bdc161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc0cf080 con 0x7f8bdc0cfea0 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8bd40019f0 con 0x7f8bdc0cfea0 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 >> v1:172.21.15.53:6789/0 conn(0x7f8bdc0cfea0 legacy=0x7f8bdc161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bbc0039c0 0x7f8bbc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bbc0047c0 0x7f8bbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc005180 0x7f8bbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0d1f50 con 0x7f8bbc0047c0 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc00c050 con 0x7f8bbc005180 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bbc00c210 con 0x7f8bbc0039c0 2024-01-16T15:13:13.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.678+0000 7f8bd9e00640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bbc0047c0 0x7f8bbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bd9e00640 1 -- 172.21.15.53:0/3913355377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bbc0039c0 msgr2=0x7f8bbc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bd9e00640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bbc0039c0 0x7f8bbc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bd9e00640 1 -- 172.21.15.53:0/3913355377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc005180 msgr2=0x7f8bbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bd9e00640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc005180 0x7f8bbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bd9e00640 1 -- 172.21.15.53:0/3913355377 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bbc003520 con 0x7f8bbc0047c0 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bdb200640 1 -- 172.21.15.53:0/3913355377 reap_dead start 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.679+0000 7f8bd9e00640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bbc0047c0 0x7f8bbc004c40 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4003db0 tx=0x7f8bbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f53e180daa58a42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.680+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc4032600 con 0x7f8bbc0047c0 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.680+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc402e050 con 0x7f8bbc0047c0 2024-01-16T15:13:13.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.680+0000 7f8bd9400640 1 -- 172.21.15.53:0/3913355377 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bc40329e0 con 0x7f8bbc0047c0 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.680+0000 7f8be0a00640 1 -- 172.21.15.53:0/3913355377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bbc0047c0 msgr2=0x7f8bbc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.680+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bbc0047c0 0x7f8bbc004c40 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7f8bc4003db0 tx=0x7f8bbc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.681+0000 7f8be0a00640 1 -- 172.21.15.53:0/3913355377 shutdown_connections 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.681+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3913355377 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bbc0047c0 0x7f8bbc004c40 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.681+0000 7f8be0a00640 1 -- 172.21.15.53:0/3913355377 >> 172.21.15.53:0/3913355377 conn(0x7f8bdc0d3090 msgr2=0x7f8bdc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.681+0000 7f8be0a00640 1 -- 172.21.15.53:0/3913355377 shutdown_connections 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.681+0000 7f8be0a00640 1 -- 172.21.15.53:0/3913355377 wait complete. 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.683+0000 7f8be0a00640 1 Processor -- start 2024-01-16T15:13:13.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.683+0000 7f8be0a00640 1 -- start start 2024-01-16T15:13:13.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8be0a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc0047c0 0x7f8bdc15a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8be0a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bdc1f6640 0x7f8bdc151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8be0a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 0x7f8bdc1516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8be0a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc0b5c50 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8be0a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1f6b70 con 0x7f8bbc0047c0 2024-01-16T15:13:13.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8be0a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bdc1f6e50 con 0x7f8bdc1f6640 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8bdb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 0x7f8bdc1516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8bdb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 0x7f8bdc1516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50958/0 (socket says 172.21.15.53:50958) 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8bdb200640 1 -- 172.21.15.53:0/3176835774 learned_addr learned my addr 172.21.15.53:0/3176835774 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8bda800640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc0047c0 0x7f8bdc15a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.685+0000 7f8bd9e00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bdc1f6640 0x7f8bdc151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bdb200640 1 -- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bdc1f6640 msgr2=0x7f8bdc151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bdb200640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bdc1f6640 0x7f8bdc151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bdb200640 1 -- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc0047c0 msgr2=0x7f8bdc15a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bdb200640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc0047c0 0x7f8bdc15a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bdb200640 1 -- 172.21.15.53:0/3176835774 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc1f7130 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bdb200640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 0x7f8bdc1516b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0d3060 tx=0x7f8bd400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd400ceb0 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd400a610 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd400a9c0 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bdc1f7410 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.686+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bdc15a8b0 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.687+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bdc0b5e00 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.688+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8bd400ab20 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.689+0000 7f8bd2a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bb406bfb0 0x7f8bb406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:13.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.689+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f8bd40881f0 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.690+0000 7f8bda800640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bb406bfb0 0x7f8bb406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:13.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.691+0000 7f8bda800640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bb406bfb0 0x7f8bb406e470 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc067b00 tx=0x7f8bcc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:13.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.694+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8bdc0b5e00 con 0x7f8bdc0ccc60 2024-01-16T15:13:13.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.953+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54146", "target": ["mon-mgr", ""]}) v1 -- 0x7f8bdc0ccab0 con 0x7f8bb406bfb0 2024-01-16T15:13:13.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.987+0000 7f8bd2a00640 1 -- 172.21.15.53:0/3176835774 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8bdc0ccab0 con 0x7f8bb406bfb0 2024-01-16T15:13:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.989+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bb406bfb0 msgr2=0x7f8bb406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.989+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bb406bfb0 0x7f8bb406e470 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc067b00 tx=0x7f8bcc008040 comp rx=0 tx=0).stop 2024-01-16T15:13:13.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.989+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 msgr2=0x7f8bdc1516b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.989+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 0x7f8bdc1516b0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc0d3060 tx=0x7f8bd400cc60 comp rx=0 tx=0).stop 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 shutdown_connections 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bb406bfb0 0x7f8bb406e470 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8bdc0ccc60 0x7f8bdc1516b0 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8bdc1f6640 0x7f8bdc151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 --2- 172.21.15.53:0/3176835774 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8bbc0047c0 0x7f8bdc15a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 >> 172.21.15.53:0/3176835774 conn(0x7f8bdc0d3090 msgr2=0x7f8bdc15fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 shutdown_connections 2024-01-16T15:13:13.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:13.990+0000 7f8be0a00640 1 -- 172.21.15.53:0/3176835774 wait complete. 2024-01-16T15:13:13.993 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone541419 2024-01-16T15:13:14.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.296+0000 7f429ae00640 1 Processor -- start 2024-01-16T15:13:14.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.296+0000 7f429ae00640 1 -- start start 2024-01-16T15:13:14.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.298+0000 7f429ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4294160920 0x7f42940b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.298+0000 7f429ae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42940cf9f0 0x7f42940cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.298+0000 7f429ae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4294161010 0x7f42941614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.298+0000 7f429ae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940d03b0 con 0x7f42940cf9f0 2024-01-16T15:13:14.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.298+0000 7f429ae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42940d0a10 con 0x7f4294161010 2024-01-16T15:13:14.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.298+0000 7f429ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940cf080 con 0x7f4294160920 2024-01-16T15:13:14.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.299+0000 7f4299000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42940cf9f0 0x7f42940cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.299+0000 7f4299000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f42940cf9f0 0x7f42940cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50970/0 (socket says 172.21.15.53:50970) 2024-01-16T15:13:14.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.299+0000 7f4299000640 1 -- 172.21.15.53:0/3376224591 learned_addr learned my addr 172.21.15.53:0/3376224591 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:14.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.299+0000 7f429a400640 1 --2- 172.21.15.53:0/3376224591 >> v2:172.21.15.146:3300/0 conn(0x7f4294161010 0x7f42941614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.300+0000 7f4299a00640 1 --2- 172.21.15.53:0/3376224591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4294160920 0x7f42940b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.300+0000 7f4299000640 1 -- 172.21.15.53:0/3376224591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4294160920 msgr2=0x7f42940b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:14.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.300+0000 7f4299000640 1 --2- 172.21.15.53:0/3376224591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4294160920 0x7f42940b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.300+0000 7f4299000640 1 -- 172.21.15.53:0/3376224591 >> v2:172.21.15.146:3300/0 conn(0x7f4294161010 msgr2=0x7f42941614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:14.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.300+0000 7f4299000640 1 --2- 172.21.15.53:0/3376224591 >> v2:172.21.15.146:3300/0 conn(0x7f4294161010 0x7f42941614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.300+0000 7f4299000640 1 -- 172.21.15.53:0/3376224591 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940c6be0 con 0x7f42940cf9f0 2024-01-16T15:13:14.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.302+0000 7f4299000640 1 --2- 172.21.15.53:0/3376224591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42940cf9f0 0x7f42940cfe70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f42940c6930 tx=0x7f427c02bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e89a748ac64c5043 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.303+0000 7f4293e00640 1 -- 172.21.15.53:0/3376224591 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c02c710 con 0x7f42940cf9f0 2024-01-16T15:13:14.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.303+0000 7f4293e00640 1 -- 172.21.15.53:0/3376224591 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c02c870 con 0x7f42940cf9f0 2024-01-16T15:13:14.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.303+0000 7f4293e00640 1 -- 172.21.15.53:0/3376224591 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c02cc00 con 0x7f42940cf9f0 2024-01-16T15:13:14.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 -- 172.21.15.53:0/3376224591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42940cf9f0 msgr2=0x7f42940cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 --2- 172.21.15.53:0/3376224591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42940cf9f0 0x7f42940cfe70 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7f42940c6930 tx=0x7f427c02bb70 comp rx=0 tx=0).stop 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 -- 172.21.15.53:0/3376224591 shutdown_connections 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 --2- 172.21.15.53:0/3376224591 >> v2:172.21.15.146:3300/0 conn(0x7f4294161010 0x7f42941614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 --2- 172.21.15.53:0/3376224591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42940cf9f0 0x7f42940cfe70 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 --2- 172.21.15.53:0/3376224591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4294160920 0x7f42940b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 -- 172.21.15.53:0/3376224591 >> 172.21.15.53:0/3376224591 conn(0x7f42940d3090 msgr2=0x7f42940d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 -- 172.21.15.53:0/3376224591 shutdown_connections 2024-01-16T15:13:14.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.304+0000 7f429ae00640 1 -- 172.21.15.53:0/3376224591 wait complete. 2024-01-16T15:13:14.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.306+0000 7f429ae00640 1 Processor -- start 2024-01-16T15:13:14.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.307+0000 7f429ae00640 1 -- start start 2024-01-16T15:13:14.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42940cf9f0 0x7f42940ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 0x7f429414a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4294161010 0x7f42940b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429414aa70 con 0x7f4294160920 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42940b6190 con 0x7f4294161010 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42940cc8e0 con 0x7f42940cf9f0 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f4299000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 0x7f429414a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f4299000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 0x7f429414a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50972/0 (socket says 172.21.15.53:50972) 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f4299000640 1 -- 172.21.15.53:0/447559463 learned_addr learned my addr 172.21.15.53:0/447559463 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f4299a00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42940cf9f0 0x7f42940ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.308+0000 7f429a400640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4294161010 0x7f42940b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.309+0000 7f4299000640 1 -- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42940cf9f0 msgr2=0x7f42940ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.309+0000 7f4299000640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42940cf9f0 0x7f42940ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.309+0000 7f4299000640 1 -- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4294161010 msgr2=0x7f42940b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.309+0000 7f4299000640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4294161010 0x7f42940b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.309+0000 7f4299000640 1 -- 172.21.15.53:0/447559463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0094d0 con 0x7f4294160920 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.310+0000 7f4299000640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 0x7f429414a530 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f427c02c0b0 tx=0x7f427c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.310+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c034c00 con 0x7f4294160920 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.310+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c034d60 con 0x7f4294160920 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.310+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42940cbcd0 con 0x7f4294160920 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.310+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42940cc290 con 0x7f4294160920 2024-01-16T15:13:14.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.310+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f427c03f640 con 0x7f4294160920 2024-01-16T15:13:14.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.312+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4264005250 con 0x7f4294160920 2024-01-16T15:13:14.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.316+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f427c03b040 con 0x7f4294160920 2024-01-16T15:13:14.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.317+0000 7f4291600640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f426c06c000 0x7f426c06e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.317+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f427c0aa8b0 con 0x7f4294160920 2024-01-16T15:13:14.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.318+0000 7f4299a00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f426c06c000 0x7f426c06e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.319+0000 7f4299a00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f426c06c000 0x7f426c06e4c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f42940c9be0 tx=0x7f4284009810 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.325+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f427c0772c0 con 0x7f4294160920 2024-01-16T15:13:14.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.623+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone541419", "target": ["mon-mgr", ""]}) v1 -- 0x7f4264000bc0 con 0x7f426c06c000 2024-01-16T15:13:14.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.659+0000 7f4291600640 1 -- 172.21.15.53:0/447559463 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4264000bc0 con 0x7f426c06c000 2024-01-16T15:13:14.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f426c06c000 msgr2=0x7f426c06e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f426c06c000 0x7f426c06e4c0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f42940c9be0 tx=0x7f4284009810 comp rx=0 tx=0).stop 2024-01-16T15:13:14.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 msgr2=0x7f429414a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 0x7f429414a530 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f427c02c0b0 tx=0x7f427c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 shutdown_connections 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f426c06c000 0x7f426c06e4c0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4294161010 0x7f42940b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4294160920 0x7f429414a530 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 --2- 172.21.15.53:0/447559463 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42940cf9f0 0x7f42940ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 >> 172.21.15.53:0/447559463 conn(0x7f42940d3090 msgr2=0x7f42940cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 shutdown_connections 2024-01-16T15:13:14.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.662+0000 7f429ae00640 1 -- 172.21.15.53:0/447559463 wait complete. 2024-01-16T15:13:14.659 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54145 2024-01-16T15:13:14.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.904+0000 7f0c0b600640 1 Processor -- start 2024-01-16T15:13:14.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.904+0000 7f0c0b600640 1 -- start start 2024-01-16T15:13:14.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.906+0000 7f0c0b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 0x7f0c040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.906+0000 7f0c0b600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0c040cf7d0 0x7f0c04070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.907+0000 7f0c0b600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c040cfbf0 con 0x7f0c04070ea0 2024-01-16T15:13:14.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.907+0000 7f0c0b600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c0414b6a0 con 0x7f0c040cf7d0 2024-01-16T15:13:14.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.907+0000 7f0c0b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04071690 con 0x7f0c0414b2c0 2024-01-16T15:13:14.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.907+0000 7f0c08c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 0x7f0c040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.907+0000 7f0c08c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 0x7f0c040cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:40840/0 (socket says 172.21.15.53:40840) 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.907+0000 7f0c08c00640 1 -- 172.21.15.53:0/858012121 learned_addr learned my addr 172.21.15.53:0/858012121 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.908+0000 7f0c08c00640 1 -- 172.21.15.53:0/858012121 >> v2:172.21.15.146:3300/0 conn(0x7f0c040cf7d0 msgr2=0x7f0c04070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.908+0000 7f0c08c00640 1 --2- 172.21.15.53:0/858012121 >> v2:172.21.15.146:3300/0 conn(0x7f0c040cf7d0 0x7f0c04070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.908+0000 7f0c08c00640 1 -- 172.21.15.53:0/858012121 >> v1:172.21.15.53:6789/0 conn(0x7f0c04070ea0 legacy=0x7f0c040b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.908+0000 7f0c03400640 1 -- 172.21.15.53:0/858012121 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3675594893 0 0) 0x7f0c040cfbf0 con 0x7f0c04070ea0 2024-01-16T15:13:14.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.908+0000 7f0c08c00640 1 -- 172.21.15.53:0/858012121 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04071970 con 0x7f0c0414b2c0 2024-01-16T15:13:14.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.911+0000 7f0c08c00640 1 --2- 172.21.15.53:0/858012121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 0x7f0c040cf290 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009870 tx=0x7f0bf402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cdffad3f61a6325 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.912+0000 7f0c03400640 1 -- 172.21.15.53:0/858012121 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4030840 con 0x7f0c0414b2c0 2024-01-16T15:13:14.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.912+0000 7f0c03400640 1 -- 172.21.15.53:0/858012121 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf40309a0 con 0x7f0c0414b2c0 2024-01-16T15:13:14.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.912+0000 7f0c03400640 1 -- 172.21.15.53:0/858012121 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf4030d30 con 0x7f0c0414b2c0 2024-01-16T15:13:14.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 -- 172.21.15.53:0/858012121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 msgr2=0x7f0c040cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 --2- 172.21.15.53:0/858012121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 0x7f0c040cf290 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4009870 tx=0x7f0bf402b980 comp rx=0 tx=0).stop 2024-01-16T15:13:14.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 -- 172.21.15.53:0/858012121 shutdown_connections 2024-01-16T15:13:14.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 --2- 172.21.15.53:0/858012121 >> v2:172.21.15.146:3300/0 conn(0x7f0c040cf7d0 0x7f0c04070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 --2- 172.21.15.53:0/858012121 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c0414b2c0 0x7f0c040cf290 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 -- 172.21.15.53:0/858012121 >> 172.21.15.53:0/858012121 conn(0x7f0c0406a800 msgr2=0x7f0c0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:14.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 -- 172.21.15.53:0/858012121 shutdown_connections 2024-01-16T15:13:14.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.913+0000 7f0c0b600640 1 -- 172.21.15.53:0/858012121 wait complete. 2024-01-16T15:13:14.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.915+0000 7f0c0b600640 1 Processor -- start 2024-01-16T15:13:14.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.915+0000 7f0c0b600640 1 -- start start 2024-01-16T15:13:14.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c0b600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c04070ea0 0x7f0c041438a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c0b600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c040cf7d0 0x7f0c041fa950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c0b600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 0x7f0c041fae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c0b600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c040a8660 con 0x7f0c0414b2c0 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c0b600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04144130 con 0x7f0c04070ea0 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c0b600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c04144410 con 0x7f0c040cf7d0 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 0x7f0c041fae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 0x7f0c041fae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50982/0 (socket says 172.21.15.53:50982) 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c09600640 1 -- 172.21.15.53:0/3343576902 learned_addr learned my addr 172.21.15.53:0/3343576902 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:14.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.916+0000 7f0c08c00640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c04070ea0 0x7f0c041438a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c09600640 1 -- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c040cf7d0 msgr2=0x7f0c041fa950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c09600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c040cf7d0 0x7f0c041fa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c09600640 1 -- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c04070ea0 msgr2=0x7f0c041438a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c09600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c04070ea0 0x7f0c041438a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c09600640 1 -- 172.21.15.53:0/3343576902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bf40094d0 con 0x7f0c0414b2c0 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c08c00640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c04070ea0 0x7f0c041438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:14.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c09600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 0x7f0c041fae90 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc001560 tx=0x7f0bfc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc003da0 con 0x7f0c0414b2c0 2024-01-16T15:13:14.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bfc005070 con 0x7f0c0414b2c0 2024-01-16T15:13:14.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc005420 con 0x7f0c0414b2c0 2024-01-16T15:13:14.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.917+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c041446f0 con 0x7f0c0414b2c0 2024-01-16T15:13:14.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.918+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c04144cb0 con 0x7f0c0414b2c0 2024-01-16T15:13:14.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.919+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c04144ed0 con 0x7f0c0414b2c0 2024-01-16T15:13:14.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.920+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f0bfc005580 con 0x7f0c0414b2c0 2024-01-16T15:13:14.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.921+0000 7f0c00c00640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bdc06bfb0 0x7f0bdc06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:14.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.921+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0bfc087f50 con 0x7f0c0414b2c0 2024-01-16T15:13:14.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.922+0000 7f0c08c00640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bdc06bfb0 0x7f0bdc06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:14.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.924+0000 7f0c08c00640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bdc06bfb0 0x7f0bdc06e470 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4036000 tx=0x7f0bf403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:14.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:14.926+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bfc054780 con 0x7f0c0414b2c0 2024-01-16T15:13:15.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.300+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54145", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c040cbc40 con 0x7f0bdc06bfb0 2024-01-16T15:13:15.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.329+0000 7f0c00c00640 1 -- 172.21.15.53:0/3343576902 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0bf4030dd0 con 0x7f0bdc06bfb0 2024-01-16T15:13:15.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.333+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bdc06bfb0 msgr2=0x7f0bdc06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.333+0000 7f0c0b600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bdc06bfb0 0x7f0bdc06e470 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f0bf4036000 tx=0x7f0bf403b040 comp rx=0 tx=0).stop 2024-01-16T15:13:15.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.333+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 msgr2=0x7f0c041fae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.333+0000 7f0c0b600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 0x7f0c041fae90 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc001560 tx=0x7f0bfc00cc70 comp rx=0 tx=0).stop 2024-01-16T15:13:15.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 shutdown_connections 2024-01-16T15:13:15.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bdc06bfb0 0x7f0bdc06e470 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0c0414b2c0 0x7f0c041fae90 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0c040cf7d0 0x7f0c041fa950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 --2- 172.21.15.53:0/3343576902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0c04070ea0 0x7f0c041438a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 >> 172.21.15.53:0/3343576902 conn(0x7f0c0406a800 msgr2=0x7f0c04148e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 shutdown_connections 2024-01-16T15:13:15.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.334+0000 7f0c0b600640 1 -- 172.21.15.53:0/3343576902 wait complete. 2024-01-16T15:13:15.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone54149 2024-01-16T15:13:15.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.627+0000 7f0e60000640 1 Processor -- start 2024-01-16T15:13:15.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.627+0000 7f0e60000640 1 -- start start 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.629+0000 7f0e60000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0e580cf4c0 0x7f0e580d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.629+0000 7f0e60000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e5815cc90 0x7f0e5815d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.629+0000 7f0e60000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e580b2920 con 0x7f0e581610e0 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.629+0000 7f0e60000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e580cb670 con 0x7f0e580cf4c0 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.629+0000 7f0e60000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e580cb950 con 0x7f0e5815cc90 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5d600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0e581610e0 0x7f0e581614c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:36160/0 (socket says 172.21.15.53:36160) 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5d600640 1 -- 172.21.15.53:0/1697669993 learned_addr learned my addr 172.21.15.53:0/1697669993 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e5815cc90 0x7f0e5815d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5cc00640 1 --2- 172.21.15.53:0/1697669993 >> v2:172.21.15.146:3300/0 conn(0x7f0e580cf4c0 0x7f0e580d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e57e00640 1 -- 172.21.15.53:0/1697669993 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426625848 0 0) 0x7f0e580b2920 con 0x7f0e581610e0 2024-01-16T15:13:15.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e57e00640 1 -- 172.21.15.53:0/1697669993 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e38003520 con 0x7f0e581610e0 2024-01-16T15:13:15.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5cc00640 1 -- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e5815cc90 msgr2=0x7f0e5815d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:15.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5cc00640 1 --2- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e5815cc90 0x7f0e5815d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5cc00640 1 -- 172.21.15.53:0/1697669993 >> v1:172.21.15.146:6789/0 conn(0x7f0e581610e0 legacy=0x7f0e581614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5cc00640 1 -- 172.21.15.53:0/1697669993 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e580cbc30 con 0x7f0e580cf4c0 2024-01-16T15:13:15.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.630+0000 7f0e5e000640 1 --2- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e5815cc90 0x7f0e5815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.631+0000 7f0e5cc00640 1 --2- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580d0340 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0e400097d0 tx=0x7f0e4002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=109e175733a496d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.632+0000 7f0e57e00640 1 -- 172.21.15.53:0/1697669993 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4002d040 con 0x7f0e580cf4c0 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.632+0000 7f0e57e00640 1 -- 172.21.15.53:0/1697669993 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e40031810 con 0x7f0e580cf4c0 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.632+0000 7f0e57e00640 1 -- 172.21.15.53:0/1697669993 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40031ba0 con 0x7f0e580cf4c0 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.632+0000 7f0e60000640 1 -- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 msgr2=0x7f0e580d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.632+0000 7f0e60000640 1 --2- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580d0340 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f0e400097d0 tx=0x7f0e4002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.633+0000 7f0e60000640 1 -- 172.21.15.53:0/1697669993 shutdown_connections 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.633+0000 7f0e60000640 1 --2- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e5815cc90 0x7f0e5815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.633+0000 7f0e60000640 1 --2- 172.21.15.53:0/1697669993 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580d0340 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.633+0000 7f0e60000640 1 -- 172.21.15.53:0/1697669993 >> 172.21.15.53:0/1697669993 conn(0x7f0e580d1020 msgr2=0x7f0e580d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.633+0000 7f0e60000640 1 -- 172.21.15.53:0/1697669993 shutdown_connections 2024-01-16T15:13:15.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.633+0000 7f0e60000640 1 -- 172.21.15.53:0/1697669993 wait complete. 2024-01-16T15:13:15.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.634+0000 7f0e60000640 1 Processor -- start 2024-01-16T15:13:15.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 -- start start 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580cab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 0x7f0e5815a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e581610e0 0x7f0e5814ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e581604e0 con 0x7f0e5815cc90 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e580c4b50 con 0x7f0e580cf4c0 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.635+0000 7f0e60000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e58160730 con 0x7f0e581610e0 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 0x7f0e5815a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 0x7f0e5815a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50984/0 (socket says 172.21.15.53:50984) 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580cab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 -- 172.21.15.53:0/3715031310 learned_addr learned my addr 172.21.15.53:0/3715031310 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5e000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e581610e0 0x7f0e5814ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 -- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e581610e0 msgr2=0x7f0e5814ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e581610e0 0x7f0e5814ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 -- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 msgr2=0x7f0e580cab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580cab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 -- 172.21.15.53:0/3715031310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e48009400 con 0x7f0e5815cc90 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.636+0000 7f0e5cc00640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 0x7f0e5815a880 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0e400098c0 tx=0x7f0e400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.637+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e40006e10 con 0x7f0e5815cc90 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.637+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e40031610 con 0x7f0e5815cc90 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.637+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e400423f0 con 0x7f0e5815cc90 2024-01-16T15:13:15.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.637+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e40009400 con 0x7f0e5815cc90 2024-01-16T15:13:15.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.637+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5815db50 con 0x7f0e5815cc90 2024-01-16T15:13:15.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.639+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e580675d0 con 0x7f0e5815cc90 2024-01-16T15:13:15.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.639+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f0e40042550 con 0x7f0e5815cc90 2024-01-16T15:13:15.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.640+0000 7f0e55600640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e3006bfb0 0x7f0e3006e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:15.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.640+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f0e400aa500 con 0x7f0e5815cc90 2024-01-16T15:13:15.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.640+0000 7f0e5d600640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e3006bfb0 0x7f0e3006e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:15.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.641+0000 7f0e5d600640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e3006bfb0 0x7f0e3006e470 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0e50001560 tx=0x7f0e500019f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:15.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.644+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0e40076be0 con 0x7f0e5815cc90 2024-01-16T15:13:15.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.858+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone54149", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e580cc4d0 con 0x7f0e3006bfb0 2024-01-16T15:13:15.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.904+0000 7f0e55600640 1 -- 172.21.15.53:0/3715031310 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0e50001ec0 con 0x7f0e3006bfb0 2024-01-16T15:13:15.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e3006bfb0 msgr2=0x7f0e3006e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e3006bfb0 0x7f0e3006e470 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f0e50001560 tx=0x7f0e500019f0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 msgr2=0x7f0e5815a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 0x7f0e5815a880 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f0e400098c0 tx=0x7f0e400046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 shutdown_connections 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e3006bfb0 0x7f0e3006e470 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e581610e0 0x7f0e5814ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e5815cc90 0x7f0e5815a880 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 --2- 172.21.15.53:0/3715031310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e580cf4c0 0x7f0e580cab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 >> 172.21.15.53:0/3715031310 conn(0x7f0e580d1020 msgr2=0x7f0e5806e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 shutdown_connections 2024-01-16T15:13:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:15.909+0000 7f0e60000640 1 -- 172.21.15.53:0/3715031310 wait complete. 2024-01-16T15:13:15.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:15.914 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:13:15.985 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-01-16T15:13:15.987 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:13:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.242+0000 7f40c2400640 1 Processor -- start 2024-01-16T15:13:16.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.242+0000 7f40c2400640 1 -- start start 2024-01-16T15:13:16.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40c2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f40bc13df80 0x7f40bc15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40c2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13e360 0x7f40bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40c2400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40bc06c470 con 0x7f40bc15f9c0 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40c2400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40bc06c750 con 0x7f40bc13df80 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40c2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc06ca30 con 0x7f40bc13e360 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40bbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f40bc15f9c0 0x7f40bc1408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:48400/0 (socket says 172.21.15.53:48400) 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.243+0000 7f40bbe00640 1 -- 172.21.15.53:0/3475575081 learned_addr learned my addr 172.21.15.53:0/3475575081 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40baa00640 1 --2- 172.21.15.53:0/3475575081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13e360 0x7f40bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40bb400640 1 --2- 172.21.15.53:0/3475575081 >> v2:172.21.15.146:3300/0 conn(0x7f40bc13df80 0x7f40bc15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40ba000640 1 -- 172.21.15.53:0/3475575081 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1199407339 0 0) 0x7f40bc06c470 con 0x7f40bc15f9c0 2024-01-16T15:13:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40ba000640 1 -- 172.21.15.53:0/3475575081 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c003520 con 0x7f40bc15f9c0 2024-01-16T15:13:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40baa00640 1 -- 172.21.15.53:0/3475575081 >> v2:172.21.15.146:3300/0 conn(0x7f40bc13df80 msgr2=0x7f40bc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40baa00640 1 --2- 172.21.15.53:0/3475575081 >> v2:172.21.15.146:3300/0 conn(0x7f40bc13df80 0x7f40bc15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40baa00640 1 -- 172.21.15.53:0/3475575081 >> v1:172.21.15.53:6789/0 conn(0x7f40bc15f9c0 legacy=0x7f40bc1408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.244+0000 7f40baa00640 1 -- 172.21.15.53:0/3475575081 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40bc06cd10 con 0x7f40bc13e360 2024-01-16T15:13:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.245+0000 7f40baa00640 1 --2- 172.21.15.53:0/3475575081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13e360 0x7f40bc0cf290 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f40a40097a0 tx=0x7f40a402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57746699d827d6d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40ba000640 1 -- 172.21.15.53:0/3475575081 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a402d040 con 0x7f40bc13e360 2024-01-16T15:13:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40ba000640 1 -- 172.21.15.53:0/3475575081 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a4031810 con 0x7f40bc13e360 2024-01-16T15:13:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40ba000640 1 -- 172.21.15.53:0/3475575081 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4031ba0 con 0x7f40bc13e360 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40c2400640 1 -- 172.21.15.53:0/3475575081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13e360 msgr2=0x7f40bc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40c2400640 1 --2- 172.21.15.53:0/3475575081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13e360 0x7f40bc0cf290 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f40a40097a0 tx=0x7f40a402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40c2400640 1 -- 172.21.15.53:0/3475575081 shutdown_connections 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40c2400640 1 --2- 172.21.15.53:0/3475575081 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13e360 0x7f40bc0cf290 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40c2400640 1 --2- 172.21.15.53:0/3475575081 >> v2:172.21.15.146:3300/0 conn(0x7f40bc13df80 0x7f40bc15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.246+0000 7f40c2400640 1 -- 172.21.15.53:0/3475575081 >> 172.21.15.53:0/3475575081 conn(0x7f40bc06a800 msgr2=0x7f40bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.247+0000 7f40c2400640 1 -- 172.21.15.53:0/3475575081 shutdown_connections 2024-01-16T15:13:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.247+0000 7f40c2400640 1 -- 172.21.15.53:0/3475575081 wait complete. 2024-01-16T15:13:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.248+0000 7f40c2400640 1 Processor -- start 2024-01-16T15:13:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.248+0000 7f40c2400640 1 -- start start 2024-01-16T15:13:16.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40c2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13df80 0x7f40bc13cf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40c2400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40bc13e360 0x7f40bc14e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40c2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 0x7f40bc159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40c2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc14bb20 con 0x7f40bc15f9c0 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40c2400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc14af10 con 0x7f40bc13e360 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40c2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40bc14b1f0 con 0x7f40bc13df80 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40bbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 0x7f40bc159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40bbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 0x7f40bc159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50998/0 (socket says 172.21.15.53:50998) 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.249+0000 7f40bbe00640 1 -- 172.21.15.53:0/3393829820 learned_addr learned my addr 172.21.15.53:0/3393829820 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bb400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13df80 0x7f40bc13cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40baa00640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40bc13e360 0x7f40bc14e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bbe00640 1 -- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13df80 msgr2=0x7f40bc13cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bbe00640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13df80 0x7f40bc13cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bbe00640 1 -- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40bc13e360 msgr2=0x7f40bc14e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bbe00640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40bc13e360 0x7f40bc14e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:16.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bbe00640 1 -- 172.21.15.53:0/3393829820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a4009400 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bb400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13df80 0x7f40bc13cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40bbe00640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 0x7f40bc159f80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f40b400cb40 tx=0x7f40b400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b400aca0 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b400ae00 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40bc14b4d0 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.250+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b4012620 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.251+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40bc13d4b0 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.251+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40bc14bc90 con 0x7f40bc15f9c0 2024-01-16T15:13:16.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.252+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 30) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f40b4007070 con 0x7f40bc15f9c0 2024-01-16T15:13:16.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.253+0000 7f40b3400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f409406bfb0 0x7f409406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:16.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.253+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5851+0+0 (secure 0 0 0) 0x7f40b4088600 con 0x7f40bc15f9c0 2024-01-16T15:13:16.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.254+0000 7f40bb400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f409406bfb0 0x7f409406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:16.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.255+0000 7f40bb400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f409406bfb0 0x7f409406e470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009af0 tx=0x7f40ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:16.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.257+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f40b4054f20 con 0x7f40bc15f9c0 2024-01-16T15:13:16.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:16.472+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f40bc06abe0 con 0x7f409406bfb0 2024-01-16T15:13:16.671 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:13:16.691+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:13:16.764 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:13:19.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.390+0000 7f40b3400640 1 -- 172.21.15.53:0/3393829820 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f40bc06abe0 con 0x7f409406bfb0 2024-01-16T15:13:19.369 INFO:teuthology.orchestra.run.smithi053.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:13:19.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.396+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f409406bfb0 msgr2=0x7f409406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:19.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.396+0000 7f40c2400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f409406bfb0 0x7f409406e470 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f40ac009af0 tx=0x7f40ac008040 comp rx=0 tx=0).stop 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.396+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 msgr2=0x7f40bc159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.396+0000 7f40c2400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 0x7f40bc159f80 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f40b400cb40 tx=0x7f40b400a060 comp rx=0 tx=0).stop 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 shutdown_connections 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f409406bfb0 0x7f409406e470 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f40bc15f9c0 0x7f40bc159f80 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f40bc13e360 0x7f40bc14e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 --2- 172.21.15.53:0/3393829820 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f40bc13df80 0x7f40bc13cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 >> 172.21.15.53:0/3393829820 conn(0x7f40bc06a800 msgr2=0x7f40bc1543a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:19.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.397+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 shutdown_connections 2024-01-16T15:13:19.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.398+0000 7f40c2400640 1 -- 172.21.15.53:0/3393829820 wait complete. 2024-01-16T15:13:19.408 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:13:19.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:13:19.503 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:13:19.603 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:13:19.603 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:13:19.603 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:19.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:19.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.901+0000 7f22f2400640 1 Processor -- start 2024-01-16T15:13:19.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.901+0000 7f22f2400640 1 -- start start 2024-01-16T15:13:19.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.902+0000 7f22f2400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22ec162430 0x7f22ec0b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:19.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.902+0000 7f22f2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 0x7f22ec071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:19.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.902+0000 7f22f2400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22ec0702e0 con 0x7f22ec0b3650 2024-01-16T15:13:19.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.902+0000 7f22f2400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22ec0705c0 con 0x7f22ec162430 2024-01-16T15:13:19.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.902+0000 7f22f2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec0708a0 con 0x7f22ec162810 2024-01-16T15:13:19.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.903+0000 7f22e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 0x7f22ec071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:19.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.903+0000 7f22e3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 0x7f22ec071240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:40894/0 (socket says 172.21.15.53:40894) 2024-01-16T15:13:19.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.903+0000 7f22e3e00640 1 -- 172.21.15.53:0/3664095336 learned_addr learned my addr 172.21.15.53:0/3664095336 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:19.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.903+0000 7f22f1000640 1 --2- 172.21.15.53:0/3664095336 >> v2:172.21.15.146:3300/0 conn(0x7f22ec162430 0x7f22ec0b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:19.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.904+0000 7f22e3e00640 1 -- 172.21.15.53:0/3664095336 >> v2:172.21.15.146:3300/0 conn(0x7f22ec162430 msgr2=0x7f22ec0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:19.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.904+0000 7f22e3e00640 1 --2- 172.21.15.53:0/3664095336 >> v2:172.21.15.146:3300/0 conn(0x7f22ec162430 0x7f22ec0b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.904+0000 7f22e3e00640 1 -- 172.21.15.53:0/3664095336 >> v1:172.21.15.146:6789/0 conn(0x7f22ec0b3650 legacy=0x7f22ec1650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:19.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.904+0000 7f22e3e00640 1 -- 172.21.15.53:0/3664095336 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22ec070b80 con 0x7f22ec162810 2024-01-16T15:13:19.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.905+0000 7f22e3e00640 1 --2- 172.21.15.53:0/3664095336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 0x7f22ec071240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009870 tx=0x7f22d402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a11515665b20fc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:19.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22ebe00640 1 -- 172.21.15.53:0/3664095336 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4030840 con 0x7f22ec162810 2024-01-16T15:13:19.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22ebe00640 1 -- 172.21.15.53:0/3664095336 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d40309a0 con 0x7f22ec162810 2024-01-16T15:13:19.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22ebe00640 1 -- 172.21.15.53:0/3664095336 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4030d30 con 0x7f22ec162810 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22f2400640 1 -- 172.21.15.53:0/3664095336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 msgr2=0x7f22ec071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22f2400640 1 --2- 172.21.15.53:0/3664095336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 0x7f22ec071240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f22d4009870 tx=0x7f22d402b980 comp rx=0 tx=0).stop 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22f2400640 1 -- 172.21.15.53:0/3664095336 shutdown_connections 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22f2400640 1 --2- 172.21.15.53:0/3664095336 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec162810 0x7f22ec071240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22f2400640 1 --2- 172.21.15.53:0/3664095336 >> v2:172.21.15.146:3300/0 conn(0x7f22ec162430 0x7f22ec0b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.906+0000 7f22f2400640 1 -- 172.21.15.53:0/3664095336 >> 172.21.15.53:0/3664095336 conn(0x7f22ec06a800 msgr2=0x7f22ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.907+0000 7f22f2400640 1 -- 172.21.15.53:0/3664095336 shutdown_connections 2024-01-16T15:13:19.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.907+0000 7f22f2400640 1 -- 172.21.15.53:0/3664095336 wait complete. 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.908+0000 7f22f2400640 1 Processor -- start 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.909+0000 7f22f2400640 1 -- start start 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.909+0000 7f22f2400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec0b3650 0x7f22ec15b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f2400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 0x7f22ec1fb220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f2400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22ec1fbca0 0x7f22ec143520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f2400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec164800 con 0x7f22ec162430 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f2400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec164ae0 con 0x7f22ec1fbca0 2024-01-16T15:13:19.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f2400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec164dc0 con 0x7f22ec0b3650 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 0x7f22ec1fb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22e3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 0x7f22ec1fb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51000/0 (socket says 172.21.15.53:51000) 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22e3e00640 1 -- 172.21.15.53:0/1288361312 learned_addr learned my addr 172.21.15.53:0/1288361312 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f1000640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec0b3650 0x7f22ec15b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.910+0000 7f22f1a00640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22ec1fbca0 0x7f22ec143520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e3e00640 1 -- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec0b3650 msgr2=0x7f22ec15b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e3e00640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec0b3650 0x7f22ec15b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e3e00640 1 -- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22ec1fbca0 msgr2=0x7f22ec143520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e3e00640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22ec1fbca0 0x7f22ec143520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e3e00640 1 -- 172.21.15.53:0/1288361312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d40094d0 con 0x7f22ec162430 2024-01-16T15:13:19.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22f1a00640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22ec1fbca0 0x7f22ec143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:19.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e3e00640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 0x7f22ec1fb220 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f22d402b9c0 tx=0x7f22d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:19.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.911+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d403e070 con 0x7f22ec162430 2024-01-16T15:13:19.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.912+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d4039a10 con 0x7f22ec162430 2024-01-16T15:13:19.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.912+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec1650a0 con 0x7f22ec162430 2024-01-16T15:13:19.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.912+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22d4039de0 con 0x7f22ec162430 2024-01-16T15:13:19.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.912+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec15bd90 con 0x7f22ec162430 2024-01-16T15:13:19.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.913+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec1655c0 con 0x7f22ec162430 2024-01-16T15:13:19.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.914+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f22d404b400 con 0x7f22ec162430 2024-01-16T15:13:19.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.915+0000 7f22e9600640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22c406bf20 0x7f22c406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:19.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.915+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f22d40aab90 con 0x7f22ec162430 2024-01-16T15:13:19.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.916+0000 7f22f1000640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22c406bf20 0x7f22c406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:19.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.917+0000 7f22f1000640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22c406bf20 0x7f22c406e3e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f22dc006fd0 tx=0x7f22dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:19.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:19.919+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22ec1655c0 con 0x7f22ec162430 2024-01-16T15:13:20.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.191+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22ec1655c0 con 0x7f22ec162430 2024-01-16T15:13:20.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.192+0000 7f22e9600640 1 -- 172.21.15.53:0/1288361312 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f22ec1655c0 con 0x7f22ec162430 2024-01-16T15:13:20.171 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:20.171 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":78,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:19.367883+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/867019165":"2024-01-17T15:13:16.691312+0000","172.21.15.146:6834/867019165":"2024-01-17T15:13:16.691312+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:20.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.194+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22c406bf20 msgr2=0x7f22c406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.194+0000 7f22f2400640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22c406bf20 0x7f22c406e3e0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f22dc006fd0 tx=0x7f22dc008040 comp rx=0 tx=0).stop 2024-01-16T15:13:20.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.194+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 msgr2=0x7f22ec1fb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.194+0000 7f22f2400640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 0x7f22ec1fb220 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f22d402b9c0 tx=0x7f22d40046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 shutdown_connections 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22c406bf20 0x7f22c406e3e0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22ec1fbca0 0x7f22ec143520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22ec162430 0x7f22ec1fb220 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 --2- 172.21.15.53:0/1288361312 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22ec0b3650 0x7f22ec15b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 >> 172.21.15.53:0/1288361312 conn(0x7f22ec06a800 msgr2=0x7f22ec0ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 shutdown_connections 2024-01-16T15:13:20.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.195+0000 7f22f2400640 1 -- 172.21.15.53:0/1288361312 wait complete. 2024-01-16T15:13:20.194 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:13:20.194 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:13:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.429+0000 7ff292000640 1 Processor -- start 2024-01-16T15:13:20.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.429+0000 7ff292000640 1 -- start start 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.431+0000 7ff292000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c15d440 0x7ff28c0c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.431+0000 7ff292000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff28c0c8930 0x7ff28c0c8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.431+0000 7ff292000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff28c160fc0 0x7ff28c161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.431+0000 7ff292000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28c0ce6f0 con 0x7ff28c160fc0 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.431+0000 7ff292000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff28c06b2e0 con 0x7ff28c0c8930 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.431+0000 7ff292000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c0cde90 con 0x7ff28c15d440 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff28c160fc0 0x7ff28c161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff28c160fc0 0x7ff28c161450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51006/0 (socket says 172.21.15.53:51006) 2024-01-16T15:13:20.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 -- 172.21.15.53:0/3163023630 learned_addr learned my addr 172.21.15.53:0/3163023630 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff290c00640 1 --2- 172.21.15.53:0/3163023630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c15d440 0x7ff28c0c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff283e00640 1 --2- 172.21.15.53:0/3163023630 >> v2:172.21.15.146:3300/0 conn(0x7ff28c0c8930 0x7ff28c0c8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 -- 172.21.15.53:0/3163023630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c15d440 msgr2=0x7ff28c0c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 --2- 172.21.15.53:0/3163023630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c15d440 0x7ff28c0c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 -- 172.21.15.53:0/3163023630 >> v2:172.21.15.146:3300/0 conn(0x7ff28c0c8930 msgr2=0x7ff28c0c8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 --2- 172.21.15.53:0/3163023630 >> v2:172.21.15.146:3300/0 conn(0x7ff28c0c8930 0x7ff28c0c8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff291600640 1 -- 172.21.15.53:0/3163023630 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28c0ce130 con 0x7ff28c160fc0 2024-01-16T15:13:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.432+0000 7ff283e00640 1 --2- 172.21.15.53:0/3163023630 >> v2:172.21.15.146:3300/0 conn(0x7ff28c0c8930 0x7ff28c0c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:20.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.433+0000 7ff291600640 1 --2- 172.21.15.53:0/3163023630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c160fc0 0x7ff28c161450 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff284009e50 tx=0x7ff28402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7751d97d5ff5d96a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.433+0000 7ff28be00640 1 -- 172.21.15.53:0/3163023630 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff28402c9e0 con 0x7ff28c160fc0 2024-01-16T15:13:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff28be00640 1 -- 172.21.15.53:0/3163023630 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff28402cb40 con 0x7ff28c160fc0 2024-01-16T15:13:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff28be00640 1 -- 172.21.15.53:0/3163023630 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2840343f0 con 0x7ff28c160fc0 2024-01-16T15:13:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 -- 172.21.15.53:0/3163023630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c160fc0 msgr2=0x7ff28c161450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 --2- 172.21.15.53:0/3163023630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c160fc0 0x7ff28c161450 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7ff284009e50 tx=0x7ff28402be40 comp rx=0 tx=0).stop 2024-01-16T15:13:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 -- 172.21.15.53:0/3163023630 shutdown_connections 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 --2- 172.21.15.53:0/3163023630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c160fc0 0x7ff28c161450 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 --2- 172.21.15.53:0/3163023630 >> v2:172.21.15.146:3300/0 conn(0x7ff28c0c8930 0x7ff28c0c8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 --2- 172.21.15.53:0/3163023630 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c15d440 0x7ff28c0c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 -- 172.21.15.53:0/3163023630 >> 172.21.15.53:0/3163023630 conn(0x7ff28c0b0d70 msgr2=0x7ff28c0b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.434+0000 7ff292000640 1 -- 172.21.15.53:0/3163023630 shutdown_connections 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.435+0000 7ff292000640 1 -- 172.21.15.53:0/3163023630 wait complete. 2024-01-16T15:13:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.436+0000 7ff292000640 1 Processor -- start 2024-01-16T15:13:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.436+0000 7ff292000640 1 -- start start 2024-01-16T15:13:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.437+0000 7ff292000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c0c8930 0x7ff28c0b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.437+0000 7ff292000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 0x7ff28c0cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.437+0000 7ff292000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff28c160fc0 0x7ff28c1f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.437+0000 7ff292000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c1f3810 con 0x7ff28c15d440 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.437+0000 7ff292000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c1f3af0 con 0x7ff28c160fc0 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.437+0000 7ff292000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff28c0b2f70 con 0x7ff28c0c8930 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 0x7ff28c0cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 0x7ff28c0cbbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51022/0 (socket says 172.21.15.53:51022) 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 -- 172.21.15.53:0/1551715286 learned_addr learned my addr 172.21.15.53:0/1551715286 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff290c00640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c0c8930 0x7ff28c0b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff291600640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff28c160fc0 0x7ff28c1f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 -- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c0c8930 msgr2=0x7ff28c0b3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c0c8930 0x7ff28c0b3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 -- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff28c160fc0 msgr2=0x7ff28c1f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff28c160fc0 0x7ff28c1f2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 -- 172.21.15.53:0/1551715286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284009b30 con 0x7ff28c15d440 2024-01-16T15:13:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.438+0000 7ff283e00640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 0x7ff28c0cbbe0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c7f0 tx=0x7ff27400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.439+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274007820 con 0x7ff28c15d440 2024-01-16T15:13:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.439+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff274007980 con 0x7ff28c15d440 2024-01-16T15:13:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.439+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff274007d30 con 0x7ff28c15d440 2024-01-16T15:13:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.439+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff28c0b3250 con 0x7ff28c15d440 2024-01-16T15:13:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.439+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff28c0cc120 con 0x7ff28c15d440 2024-01-16T15:13:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.440+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff28c0b2dd0 con 0x7ff28c15d440 2024-01-16T15:13:20.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.441+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff2740113f0 con 0x7ff28c15d440 2024-01-16T15:13:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.442+0000 7ff289600640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff26406bf20 0x7ff26406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.443+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff274088b80 con 0x7ff28c15d440 2024-01-16T15:13:20.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.443+0000 7ff290c00640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff26406bf20 0x7ff26406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.444+0000 7ff290c00640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff26406bf20 0x7ff26406e3e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff27c006fd0 tx=0x7ff27c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.447+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff28c0b2dd0 con 0x7ff28c15d440 2024-01-16T15:13:20.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.640+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff28c0b3490 con 0x7ff28c15d440 2024-01-16T15:13:20.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.641+0000 7ff289600640 1 -- 172.21.15.53:0/1551715286 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff28c0b3490 con 0x7ff28c15d440 2024-01-16T15:13:20.620 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff26406bf20 msgr2=0x7ff26406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff26406bf20 0x7ff26406e3e0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7ff27c006fd0 tx=0x7ff27c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 msgr2=0x7ff28c0cbbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 0x7ff28c0cbbe0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7ff27400c7f0 tx=0x7ff27400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 shutdown_connections 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff26406bf20 0x7ff26406e3e0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff28c160fc0 0x7ff28c1f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff28c15d440 0x7ff28c0cbbe0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 --2- 172.21.15.53:0/1551715286 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff28c0c8930 0x7ff28c0b3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 >> 172.21.15.53:0/1551715286 conn(0x7ff28c0b0d70 msgr2=0x7ff28c1613d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:20.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 shutdown_connections 2024-01-16T15:13:20.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.644+0000 7ff292000640 1 -- 172.21.15.53:0/1551715286 wait complete. 2024-01-16T15:13:20.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:13:20.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.892+0000 7f46d5800640 1 Processor -- start 2024-01-16T15:13:20.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.893+0000 7f46d5800640 1 -- start start 2024-01-16T15:13:20.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.894+0000 7f46d5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f46d0071410 0x7f46d0146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.894+0000 7f46d5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46d006c7c0 0x7f46d00717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.894+0000 7f46d5800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d006cc50 con 0x7f46d0071d30 2024-01-16T15:13:20.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46d5800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d006cec0 con 0x7f46d006c7c0 2024-01-16T15:13:20.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46d5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d006d1a0 con 0x7f46d0071410 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46d006c7c0 0x7f46d00717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f46d006c7c0 0x7f46d00717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51038/0 (socket says 172.21.15.53:51038) 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 -- 172.21.15.53:0/2513871041 learned_addr learned my addr 172.21.15.53:0/2513871041 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46cfe00640 1 --2- 172.21.15.53:0/2513871041 >> v2:172.21.15.146:3300/0 conn(0x7f46d0071410 0x7f46d0146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 -- 172.21.15.53:0/2513871041 >> v2:172.21.15.146:3300/0 conn(0x7f46d0071410 msgr2=0x7f46d0146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 --2- 172.21.15.53:0/2513871041 >> v2:172.21.15.146:3300/0 conn(0x7f46d0071410 0x7f46d0146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 -- 172.21.15.53:0/2513871041 >> v1:172.21.15.146:6789/0 conn(0x7f46d0071d30 legacy=0x7f46d00b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46c7200640 1 -- 172.21.15.53:0/2513871041 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d006d480 con 0x7f46d006c7c0 2024-01-16T15:13:20.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.895+0000 7f46cfe00640 1 --2- 172.21.15.53:0/2513871041 >> v2:172.21.15.146:3300/0 conn(0x7f46d0071410 0x7f46d0146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:20.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.897+0000 7f46c7200640 1 --2- 172.21.15.53:0/2513871041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00717f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f46d00ad5f0 tx=0x7f46b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4986f4c6d045a6e5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.897+0000 7f46cf400640 1 -- 172.21.15.53:0/2513871041 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8031870 con 0x7f46d006c7c0 2024-01-16T15:13:20.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.897+0000 7f46cf400640 1 -- 172.21.15.53:0/2513871041 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46b80319d0 con 0x7f46d006c7c0 2024-01-16T15:13:20.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.897+0000 7f46cf400640 1 -- 172.21.15.53:0/2513871041 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46b8031d60 con 0x7f46d006c7c0 2024-01-16T15:13:20.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 -- 172.21.15.53:0/2513871041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 msgr2=0x7f46d00717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 --2- 172.21.15.53:0/2513871041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00717f0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f46d00ad5f0 tx=0x7f46b802b910 comp rx=0 tx=0).stop 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 -- 172.21.15.53:0/2513871041 shutdown_connections 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 --2- 172.21.15.53:0/2513871041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00717f0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 --2- 172.21.15.53:0/2513871041 >> v2:172.21.15.146:3300/0 conn(0x7f46d0071410 0x7f46d0146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 -- 172.21.15.53:0/2513871041 >> 172.21.15.53:0/2513871041 conn(0x7f46d006a800 msgr2=0x7f46d006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 -- 172.21.15.53:0/2513871041 shutdown_connections 2024-01-16T15:13:20.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.898+0000 7f46d5800640 1 -- 172.21.15.53:0/2513871041 wait complete. 2024-01-16T15:13:20.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.900+0000 7f46d5800640 1 Processor -- start 2024-01-16T15:13:20.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.900+0000 7f46d5800640 1 -- start start 2024-01-16T15:13:20.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46d5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00b33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46d5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46d0071410 0x7f46d00c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46d5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46d0071d30 0x7f46d015f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46d5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015f170 con 0x7f46d006c7c0 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46d5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015e4f0 con 0x7f46d0071410 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46d5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d015e7d0 con 0x7f46d0071d30 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00b33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00b33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51048/0 (socket says 172.21.15.53:51048) 2024-01-16T15:13:20.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 -- 172.21.15.53:0/2998642261 learned_addr learned my addr 172.21.15.53:0/2998642261 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46c7200640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46d0071410 0x7f46d00c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 -- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46d0071d30 msgr2=0x7f46d015f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46d0071d30 0x7f46d015f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 -- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46d0071410 msgr2=0x7f46d00c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46d0071410 0x7f46d00c3d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.901+0000 7f46cfe00640 1 -- 172.21.15.53:0/2998642261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b80094d0 con 0x7f46d006c7c0 2024-01-16T15:13:20.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.902+0000 7f46cfe00640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00b33f0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c7f0 tx=0x7f46c000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.902+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0007a30 con 0x7f46d006c7c0 2024-01-16T15:13:20.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.902+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c0007b90 con 0x7f46d006c7c0 2024-01-16T15:13:20.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.902+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000ce40 con 0x7f46d006c7c0 2024-01-16T15:13:20.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.902+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d015eab0 con 0x7f46d006c7c0 2024-01-16T15:13:20.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.902+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d015e110 con 0x7f46d006c7c0 2024-01-16T15:13:20.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.904+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d0072140 con 0x7f46d006c7c0 2024-01-16T15:13:20.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.905+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f46c00113f0 con 0x7f46d006c7c0 2024-01-16T15:13:20.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.906+0000 7f46ccc00640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f46a806bf20 0x7f46a806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:20.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.906+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f46c0088a00 con 0x7f46d006c7c0 2024-01-16T15:13:20.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.906+0000 7f46c7200640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f46a806bf20 0x7f46a806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:20.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.908+0000 7f46c7200640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f46a806bf20 0x7f46a806e3e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f46b8036000 tx=0x7f46b803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:20.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:20.911+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f46c0055040 con 0x7f46d006c7c0 2024-01-16T15:13:21.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.128+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f46d0067450 con 0x7f46d006c7c0 2024-01-16T15:13:21.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.129+0000 7f46ccc00640 1 -- 172.21.15.53:0/2998642261 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v78) v1 ==== 80+0+122 (secure 0 0 0) 0x7f46c0058660 con 0x7f46d006c7c0 2024-01-16T15:13:21.107 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6836/867019165 2024-01-17T15:13:16.691312+0000 2024-01-16T15:13:21.107 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6834/867019165 2024-01-17T15:13:16.691312+0000 2024-01-16T15:13:21.108 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f46a806bf20 msgr2=0x7f46a806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f46a806bf20 0x7f46a806e3e0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f46b8036000 tx=0x7f46b803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 msgr2=0x7f46d00b33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00b33f0 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c7f0 tx=0x7f46c000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 shutdown_connections 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f46a806bf20 0x7f46a806e3e0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f46d0071d30 0x7f46d015f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f46d0071410 0x7f46d00c3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 --2- 172.21.15.53:0/2998642261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f46d006c7c0 0x7f46d00b33f0 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.131+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 >> 172.21.15.53:0/2998642261 conn(0x7f46d006a800 msgr2=0x7f46d006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:21.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.132+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 shutdown_connections 2024-01-16T15:13:21.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.132+0000 7f46d5800640 1 -- 172.21.15.53:0/2998642261 wait complete. 2024-01-16T15:13:21.129 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:13:21.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:21.235 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:13:21.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:21.235 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:13:21.249 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[61266]: fuse finished with error 0 and tester_r 0 2024-01-16T15:13:21.448 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:13:21.448 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:21.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:21.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.758+0000 7f0be2200640 1 Processor -- start 2024-01-16T15:13:21.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.758+0000 7f0be2200640 1 -- start start 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.759+0000 7f0be2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.759+0000 7f0be2200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d0f40 0x7f0bdc0d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.759+0000 7f0be2200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc0d0a10 con 0x7f0bdc160920 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.759+0000 7f0be2200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc0cf080 con 0x7f0bdc0d0f40 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.759+0000 7f0be2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0c6be0 con 0x7f0bdc0d0100 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.760+0000 7f0be0e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f0bdc160920 0x7f0bdc0cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:36182/0 (socket says 172.21.15.53:36182) 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.760+0000 7f0be0e00640 1 -- 172.21.15.53:0/2010034149 learned_addr learned my addr 172.21.15.53:0/2010034149 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.760+0000 7f0bd3c00640 1 --2- 172.21.15.53:0/2010034149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:21.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.760+0000 7f0be1800640 1 --2- 172.21.15.53:0/2010034149 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d0f40 0x7f0bdc0d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:21.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.761+0000 7f0bd3c00640 1 -- 172.21.15.53:0/2010034149 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d0f40 msgr2=0x7f0bdc0d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:21.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.761+0000 7f0bd3c00640 1 --2- 172.21.15.53:0/2010034149 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d0f40 0x7f0bdc0d13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.761+0000 7f0bd3c00640 1 -- 172.21.15.53:0/2010034149 >> v1:172.21.15.146:6789/0 conn(0x7f0bdc160920 legacy=0x7f0bdc0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.761+0000 7f0bdbe00640 1 -- 172.21.15.53:0/2010034149 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1811077363 0 0) 0x7f0bdc0d0a10 con 0x7f0bdc160920 2024-01-16T15:13:21.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.761+0000 7f0bd3c00640 1 -- 172.21.15.53:0/2010034149 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc0ce7e0 con 0x7f0bdc0d0100 2024-01-16T15:13:21.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.763+0000 7f0bd3c00640 1 --2- 172.21.15.53:0/2010034149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc0b3f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40097a0 tx=0x7f0bc402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d90a0588ddc44b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:21.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.763+0000 7f0bdbe00640 1 -- 172.21.15.53:0/2010034149 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc402d040 con 0x7f0bdc0d0100 2024-01-16T15:13:21.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.763+0000 7f0bdbe00640 1 -- 172.21.15.53:0/2010034149 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bc4031810 con 0x7f0bdc0d0100 2024-01-16T15:13:21.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.763+0000 7f0bdbe00640 1 -- 172.21.15.53:0/2010034149 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bc4031ba0 con 0x7f0bdc0d0100 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 -- 172.21.15.53:0/2010034149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 msgr2=0x7f0bdc0b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 --2- 172.21.15.53:0/2010034149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc0b3f30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0bc40097a0 tx=0x7f0bc402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 -- 172.21.15.53:0/2010034149 shutdown_connections 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 --2- 172.21.15.53:0/2010034149 >> v2:172.21.15.146:3300/0 conn(0x7f0bdc0d0f40 0x7f0bdc0d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 --2- 172.21.15.53:0/2010034149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc0b3f30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 -- 172.21.15.53:0/2010034149 >> 172.21.15.53:0/2010034149 conn(0x7f0bdc0d3090 msgr2=0x7f0bdc0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 -- 172.21.15.53:0/2010034149 shutdown_connections 2024-01-16T15:13:21.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.764+0000 7f0be2200640 1 -- 172.21.15.53:0/2010034149 wait complete. 2024-01-16T15:13:21.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.766+0000 7f0be2200640 1 Processor -- start 2024-01-16T15:13:21.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.766+0000 7f0be2200640 1 -- start start 2024-01-16T15:13:21.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.767+0000 7f0be2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc15c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:21.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.767+0000 7f0be2200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc0d0f40 0x7f0bdc1f3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:21.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.767+0000 7f0be2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 0x7f0bdc06b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:21.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.767+0000 7f0be2200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b6580 con 0x7f0bdc160920 2024-01-16T15:13:21.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.767+0000 7f0be2200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b8ca0 con 0x7f0bdc0d0f40 2024-01-16T15:13:21.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.767+0000 7f0be2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc0b5c50 con 0x7f0bdc0d0100 2024-01-16T15:13:21.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 0x7f0bdc06b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:21.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 0x7f0bdc06b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:51058/0 (socket says 172.21.15.53:51058) 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 -- 172.21.15.53:0/428830835 learned_addr learned my addr 172.21.15.53:0/428830835 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be0e00640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc15c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0bd3c00640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc0d0f40 0x7f0bdc1f3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 -- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 msgr2=0x7f0bdc15c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc15c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 -- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc0d0f40 msgr2=0x7f0bdc1f3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc0d0f40 0x7f0bdc1f3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.768+0000 7f0be1800640 1 -- 172.21.15.53:0/428830835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc4009400 con 0x7f0bdc160920 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.769+0000 7f0be1800640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 0x7f0bdc06b510 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400efc0 tx=0x7f0bd400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.769+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4014480 con 0x7f0bdc160920 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.769+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd400f040 con 0x7f0bdc160920 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.769+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd4014820 con 0x7f0bdc160920 2024-01-16T15:13:21.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.769+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc0b5f30 con 0x7f0bdc160920 2024-01-16T15:13:21.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.769+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc1f43f0 con 0x7f0bdc160920 2024-01-16T15:13:21.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.770+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc0b66f0 con 0x7f0bdc160920 2024-01-16T15:13:21.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.771+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0bd4014980 con 0x7f0bdc160920 2024-01-16T15:13:21.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.772+0000 7f0bd9600640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bb406bf20 0x7f0bb406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:21.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.772+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f0bd4089160 con 0x7f0bdc160920 2024-01-16T15:13:21.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.773+0000 7f0be0e00640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bb406bf20 0x7f0bb406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:21.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.774+0000 7f0be0e00640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bb406bf20 0x7f0bb406e3e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc000f60 tx=0x7f0bcc007d50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:21.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.777+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bd4055a20 con 0x7f0bdc160920 2024-01-16T15:13:21.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.970+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bdc067450 con 0x7f0bdc160920 2024-01-16T15:13:21.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.972+0000 7f0bd9600640 1 -- 172.21.15.53:0/428830835 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f0bd4018090 con 0x7f0bdc160920 2024-01-16T15:13:21.951 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:21.951 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":78,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:19.367883+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/867019165":"2024-01-17T15:13:16.691312+0000","172.21.15.146:6834/867019165":"2024-01-17T15:13:16.691312+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bb406bf20 msgr2=0x7f0bb406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bb406bf20 0x7f0bb406e3e0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0bcc000f60 tx=0x7f0bcc007d50 comp rx=0 tx=0).stop 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 msgr2=0x7f0bdc06b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 0x7f0bdc06b510 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f0bd400efc0 tx=0x7f0bd400c490 comp rx=0 tx=0).stop 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 shutdown_connections 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0bb406bf20 0x7f0bb406e3e0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bdc160920 0x7f0bdc06b510 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bdc0d0f40 0x7f0bdc1f3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 --2- 172.21.15.53:0/428830835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bdc0d0100 0x7f0bdc15c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 >> 172.21.15.53:0/428830835 conn(0x7f0bdc0d3090 msgr2=0x7f0bdc0c8df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 shutdown_connections 2024-01-16T15:13:21.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:21.975+0000 7f0be2200640 1 -- 172.21.15.53:0/428830835 wait complete. 2024-01-16T15:13:21.975 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:13:21.976 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:13:22.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.217+0000 7f2247200640 1 Processor -- start 2024-01-16T15:13:22.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.218+0000 7f2247200640 1 -- start start 2024-01-16T15:13:22.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.219+0000 7f2247200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 0x7f224813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.219+0000 7f2247200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f224813fae0 0x7f22480cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.219+0000 7f2247200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480ce440 con 0x7f22480cdc20 2024-01-16T15:13:22.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.219+0000 7f2247200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f224813ff60 con 0x7f224813fae0 2024-01-16T15:13:22.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.219+0000 7f2247200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22480ce720 con 0x7f224813cf10 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.220+0000 7f2246800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f22480cdc20 0x7f224815d9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39296/0 (socket says 172.21.15.53:39296) 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.220+0000 7f2246800640 1 -- 172.21.15.53:0/1379436219 learned_addr learned my addr 172.21.15.53:0/1379436219 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.220+0000 7f2245e00640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 0x7f224813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.220+0000 7f223fe00640 1 --2- 172.21.15.53:0/1379436219 >> v2:172.21.15.146:3300/0 conn(0x7f224813fae0 0x7f22480cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.220+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926046571 0 0) 0x7f22480ce440 con 0x7f22480cdc20 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.220+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2228003520 con 0x7f22480cdc20 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2681666905 0 0) 0x7f2228003520 con 0x7f22480cdc20 2024-01-16T15:13:22.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 msgr2=0x7f224813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 0x7f224813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 >> v2:172.21.15.146:3300/0 conn(0x7f224813fae0 msgr2=0x7f22480cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 --2- 172.21.15.53:0/1379436219 >> v2:172.21.15.146:3300/0 conn(0x7f224813fae0 0x7f22480cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480cea00 con 0x7f22480cdc20 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f22400019f0 con 0x7f22480cdc20 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 >> v1:172.21.15.53:6789/0 conn(0x7f22480cdc20 legacy=0x7f224815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2228003970 0x7f2228003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22280047c0 0x7f2228004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 0x7f2228005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224813ff60 con 0x7f22280047c0 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22480ce720 con 0x7f2228003970 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f222800b980 con 0x7f224813cf10 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.221+0000 7f223fe00640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22280047c0 0x7f2228004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.222+0000 7f2245e00640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2228003970 0x7f2228003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.222+0000 7f223fe00640 1 -- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 msgr2=0x7f2228005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:22.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.222+0000 7f223fe00640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f224813cf10 0x7f2228005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.222+0000 7f223fe00640 1 -- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2228003970 msgr2=0x7f2228003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:22.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.222+0000 7f223fe00640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2228003970 0x7f2228003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.222+0000 7f223fe00640 1 -- 172.21.15.53:0/1379436219 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2228003520 con 0x7f22280047c0 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.223+0000 7f223fe00640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22280047c0 0x7f2228004c40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f2230009ac0 tx=0x7f22280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c05c8c427689a6e1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.223+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223002c6d0 con 0x7f22280047c0 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.223+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223002c830 con 0x7f22280047c0 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.223+0000 7f2245400640 1 -- 172.21.15.53:0/1379436219 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223002cbc0 con 0x7f22280047c0 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2247200640 1 -- 172.21.15.53:0/1379436219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22280047c0 msgr2=0x7f2228004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2247200640 1 --2- 172.21.15.53:0/1379436219 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22280047c0 0x7f2228004c40 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f2230009ac0 tx=0x7f22280057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2246800640 1 -- 172.21.15.53:0/1379436219 reap_dead start 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2247200640 1 -- 172.21.15.53:0/1379436219 shutdown_connections 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2247200640 1 -- 172.21.15.53:0/1379436219 >> 172.21.15.53:0/1379436219 conn(0x7f224806a7d0 msgr2=0x7f224806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:22.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2247200640 1 -- 172.21.15.53:0/1379436219 shutdown_connections 2024-01-16T15:13:22.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.224+0000 7f2247200640 1 -- 172.21.15.53:0/1379436219 wait complete. 2024-01-16T15:13:22.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.226+0000 7f2247200640 1 Processor -- start 2024-01-16T15:13:22.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.226+0000 7f2247200640 1 -- start start 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2247200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280047c0 0x7f224815cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2247200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f224815a100 0x7f2248152e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2247200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 0x7f224813e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2247200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22481561d0 con 0x7f2248157fc0 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2247200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22481533c0 con 0x7f224815a100 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2247200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224815a580 con 0x7f22280047c0 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2246800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 0x7f224813e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2246800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 0x7f224813e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41252/0 (socket says 172.21.15.53:41252) 2024-01-16T15:13:22.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2246800640 1 -- 172.21.15.53:0/3258556430 learned_addr learned my addr 172.21.15.53:0/3258556430 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f2245e00640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280047c0 0x7f224815cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.227+0000 7f223fe00640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f224815a100 0x7f2248152e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2246800640 1 -- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280047c0 msgr2=0x7f224815cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2246800640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280047c0 0x7f224815cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2246800640 1 -- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f224815a100 msgr2=0x7f2248152e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2246800640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f224815a100 0x7f2248152e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2246800640 1 -- 172.21.15.53:0/3258556430 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2248156450 con 0x7f2248157fc0 2024-01-16T15:13:22.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f223fe00640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f224815a100 0x7f2248152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2246800640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 0x7f224813e340 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22480b3f10 tx=0x7f224000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224000a610 con 0x7f2248157fc0 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224000a790 con 0x7f2248157fc0 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224000ab40 con 0x7f2248157fc0 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224815d6d0 con 0x7f2248157fc0 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.228+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22481569f0 con 0x7f2248157fc0 2024-01-16T15:13:22.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.230+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2248156be0 con 0x7f2248157fc0 2024-01-16T15:13:22.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.231+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f224000aca0 con 0x7f2248157fc0 2024-01-16T15:13:22.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.232+0000 7f223e000640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f222006bf50 0x7f222006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.232+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2240088240 con 0x7f2248157fc0 2024-01-16T15:13:22.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.232+0000 7f2245e00640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f222006bf50 0x7f222006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.233+0000 7f2245e00640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f222006bf50 0x7f222006e410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2238009930 tx=0x7f2238008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.236+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2240054b00 con 0x7f2248157fc0 2024-01-16T15:13:22.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.431+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f224813f550 con 0x7f2248157fc0 2024-01-16T15:13:22.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.432+0000 7f223e000640 1 -- 172.21.15.53:0/3258556430 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f224001a030 con 0x7f2248157fc0 2024-01-16T15:13:22.410 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:13:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f222006bf50 msgr2=0x7f222006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f222006bf50 0x7f222006e410 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f2238009930 tx=0x7f2238008040 comp rx=0 tx=0).stop 2024-01-16T15:13:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 msgr2=0x7f224813e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 0x7f224813e340 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f22480b3f10 tx=0x7f224000ccd0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 shutdown_connections 2024-01-16T15:13:22.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f222006bf50 0x7f222006e410 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2248157fc0 0x7f224813e340 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f224815a100 0x7f2248152e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.434+0000 7f2247200640 1 --2- 172.21.15.53:0/3258556430 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22280047c0 0x7f224815cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.435+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 >> 172.21.15.53:0/3258556430 conn(0x7f224806a7d0 msgr2=0x7f224815dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.435+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 shutdown_connections 2024-01-16T15:13:22.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.435+0000 7f2247200640 1 -- 172.21.15.53:0/3258556430 wait complete. 2024-01-16T15:13:22.434 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:13:22.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.674+0000 7f0a22200640 1 Processor -- start 2024-01-16T15:13:22.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.674+0000 7f0a22200640 1 -- start start 2024-01-16T15:13:22.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a22200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c143c40 0x7f0a1c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a22200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0a1c0ccc10 0x7f0a1c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a22200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c070e50 con 0x7f0a1c0717d0 2024-01-16T15:13:22.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a22200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a1c0723c0 con 0x7f0a1c0ccc10 2024-01-16T15:13:22.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a22200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c144020 con 0x7f0a1c143c40 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a21800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0a1c0ccc10 0x7f0a1c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a21800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0a1c0ccc10 0x7f0a1c06d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41258/0 (socket says 172.21.15.53:41258) 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.676+0000 7f0a21800640 1 -- 172.21.15.53:0/1370583570 learned_addr learned my addr 172.21.15.53:0/1370583570 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.677+0000 7f0a20e00640 1 --2- 172.21.15.53:0/1370583570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c143c40 0x7f0a1c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.677+0000 7f0a21800640 1 -- 172.21.15.53:0/1370583570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c143c40 msgr2=0x7f0a1c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.677+0000 7f0a21800640 1 --2- 172.21.15.53:0/1370583570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c143c40 0x7f0a1c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.677+0000 7f0a21800640 1 -- 172.21.15.53:0/1370583570 >> v1:172.21.15.146:6789/0 conn(0x7f0a1c0717d0 legacy=0x7f0a1c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.677+0000 7f0a21800640 1 -- 172.21.15.53:0/1370583570 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c071e00 con 0x7f0a1c0ccc10 2024-01-16T15:13:22.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.678+0000 7f0a21800640 1 --2- 172.21.15.53:0/1370583570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c06d160 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009e50 tx=0x7f0a1402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44e3e48487d1ee1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.678+0000 7f0a1b400640 1 -- 172.21.15.53:0/1370583570 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1402ca70 con 0x7f0a1c0ccc10 2024-01-16T15:13:22.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a1b400640 1 -- 172.21.15.53:0/1370583570 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1402cbd0 con 0x7f0a1c0ccc10 2024-01-16T15:13:22.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a1b400640 1 -- 172.21.15.53:0/1370583570 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a140343f0 con 0x7f0a1c0ccc10 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a22200640 1 -- 172.21.15.53:0/1370583570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c0ccc10 msgr2=0x7f0a1c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a22200640 1 --2- 172.21.15.53:0/1370583570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c06d160 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009e50 tx=0x7f0a1402bed0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a22200640 1 -- 172.21.15.53:0/1370583570 shutdown_connections 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a22200640 1 --2- 172.21.15.53:0/1370583570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c06d160 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a22200640 1 --2- 172.21.15.53:0/1370583570 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c143c40 0x7f0a1c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.679+0000 7f0a22200640 1 -- 172.21.15.53:0/1370583570 >> 172.21.15.53:0/1370583570 conn(0x7f0a1c06a800 msgr2=0x7f0a1c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.680+0000 7f0a22200640 1 -- 172.21.15.53:0/1370583570 shutdown_connections 2024-01-16T15:13:22.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.680+0000 7f0a22200640 1 -- 172.21.15.53:0/1370583570 wait complete. 2024-01-16T15:13:22.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.681+0000 7f0a22200640 1 Processor -- start 2024-01-16T15:13:22.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.681+0000 7f0a22200640 1 -- start start 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.682+0000 7f0a22200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c0717d0 0x7f0a1c152a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.682+0000 7f0a22200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c13b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.682+0000 7f0a22200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 0x7f0a1c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.682+0000 7f0a22200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c152fa0 con 0x7f0a1c143c40 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.682+0000 7f0a22200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c13bc40 con 0x7f0a1c0ccc10 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.682+0000 7f0a22200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c1f6440 con 0x7f0a1c0717d0 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 0x7f0a1c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 0x7f0a1c0b2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41272/0 (socket says 172.21.15.53:41272) 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 -- 172.21.15.53:0/3063588075 learned_addr learned my addr 172.21.15.53:0/3063588075 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a20e00640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c0717d0 0x7f0a1c152a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a1be00640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c13b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 -- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c0717d0 msgr2=0x7f0a1c152a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c0717d0 0x7f0a1c152a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 -- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a1c0ccc10 msgr2=0x7f0a1c13b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c13b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 -- 172.21.15.53:0/3063588075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a14009b30 con 0x7f0a1c143c40 2024-01-16T15:13:22.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.683+0000 7f0a21800640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 0x7f0a1c0b2110 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009e50 tx=0x7f0a14004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.684+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14009510 con 0x7f0a1c143c40 2024-01-16T15:13:22.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.684+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1403e450 con 0x7f0a1c143c40 2024-01-16T15:13:22.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.684+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1403e800 con 0x7f0a1c143c40 2024-01-16T15:13:22.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.684+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c1f6720 con 0x7f0a1c143c40 2024-01-16T15:13:22.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.684+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c1f6ce0 con 0x7f0a1c143c40 2024-01-16T15:13:22.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.685+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1c066e70 con 0x7f0a1c143c40 2024-01-16T15:13:22.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.687+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f0a1403e960 con 0x7f0a1c143c40 2024-01-16T15:13:22.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.688+0000 7f0a18c00640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09f406bf20 0x7f09f406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:22.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.688+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f0a140ab5c0 con 0x7f0a1c143c40 2024-01-16T15:13:22.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.688+0000 7f0a20e00640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09f406bf20 0x7f09f406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:22.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.689+0000 7f0a20e00640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09f406bf20 0x7f09f406e3e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c006fd0 tx=0x7f0a0c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:22.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.692+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0a14077c60 con 0x7f0a1c143c40 2024-01-16T15:13:22.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.898+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0a1c0cc310 con 0x7f0a1c143c40 2024-01-16T15:13:22.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.899+0000 7f0a18c00640 1 -- 172.21.15.53:0/3063588075 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v78) v1 ==== 80+0+122 (secure 0 0 0) 0x7f0a1407b280 con 0x7f0a1c143c40 2024-01-16T15:13:22.878 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6836/867019165 2024-01-17T15:13:16.691312+0000 2024-01-16T15:13:22.878 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.146:6834/867019165 2024-01-17T15:13:16.691312+0000 2024-01-16T15:13:22.878 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:13:22.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09f406bf20 msgr2=0x7f09f406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09f406bf20 0x7f09f406e3e0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c006fd0 tx=0x7f0a0c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 msgr2=0x7f0a1c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 0x7f0a1c0b2110 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009e50 tx=0x7f0a14004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 shutdown_connections 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f09f406bf20 0x7f09f406e3e0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0a1c143c40 0x7f0a1c0b2110 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0a1c0ccc10 0x7f0a1c13b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 --2- 172.21.15.53:0/3063588075 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0a1c0717d0 0x7f0a1c152a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 >> 172.21.15.53:0/3063588075 conn(0x7f0a1c06a800 msgr2=0x7f0a1c0cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 shutdown_connections 2024-01-16T15:13:22.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:22.902+0000 7f0a22200640 1 -- 172.21.15.53:0/3063588075 wait complete. 2024-01-16T15:13:22.900 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:13:22.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:22.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:22.995 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[44911]: fuse finished with error 0 and tester_r 0 2024-01-16T15:13:23.000 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:13:23.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:23.000 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:13:23.189 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:13:23.189 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:23.253 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:13:23.253 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:13:23.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:23.253 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:13:23.310 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:13:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:13:23.311 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:13:23.311 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:23.367 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:13:23.367 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:13:23.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:23.367 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:13:23.414 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:13:23.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:13:23.416 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:13:23.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.664+0000 7f1793000640 1 Processor -- start 2024-01-16T15:13:23.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.664+0000 7f1793000640 1 -- start start 2024-01-16T15:13:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.665+0000 7f1793000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 0x7f178c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.665+0000 7f1793000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c0d0a10 con 0x7f178c0cf9f0 2024-01-16T15:13:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.665+0000 7f1793000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f178c161e30 con 0x7f178c0cfe70 2024-01-16T15:13:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.665+0000 7f1793000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0d1f50 con 0x7f178c160920 2024-01-16T15:13:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.666+0000 7f1791200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f178c0cf9f0 0x7f178c161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39302/0 (socket says 172.21.15.53:39302) 2024-01-16T15:13:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.666+0000 7f1791200640 1 -- 172.21.15.53:0/64273736 learned_addr learned my addr 172.21.15.53:0/64273736 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:23.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.666+0000 7f1791c00640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 0x7f178c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.666+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687640118 0 0) 0x7f178c0d0a10 con 0x7f178c0cf9f0 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.666+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f176c003520 con 0x7f178c0cf9f0 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353950641 0 0) 0x7f176c003520 con 0x7f178c0cf9f0 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 msgr2=0x7f178c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 0x7f178c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 >> v1:172.21.15.146:6789/0 conn(0x7f178c0cfe70 legacy=0x7f178c161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0cf080 con 0x7f178c0cf9f0 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1774001670 con 0x7f178c0cf9f0 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 >> v1:172.21.15.53:6789/0 conn(0x7f178c0cf9f0 legacy=0x7f178c161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 0x7f176c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f176c003f70 0x7f176c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f176c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0d1f50 con 0x7f176c004d00 2024-01-16T15:13:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c00bac0 con 0x7f176c003f70 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c00bc80 con 0x7f178c160920 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f1792600640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f176c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.667+0000 7f1791200640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f176c003f70 0x7f176c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.668+0000 7f1791c00640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 0x7f176c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.668+0000 7f1792600640 1 -- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 msgr2=0x7f176c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.668+0000 7f1792600640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c160920 0x7f176c0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.668+0000 7f1792600640 1 -- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f176c003f70 msgr2=0x7f176c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.668+0000 7f1792600640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f176c003f70 0x7f176c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.668+0000 7f1792600640 1 -- 172.21.15.53:0/64273736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c003520 con 0x7f176c004d00 2024-01-16T15:13:23.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.669+0000 7f1792600640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f176c005180 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1784001e50 tx=0x7f176c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96569419c305081 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.669+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784003b70 con 0x7f176c004d00 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.669+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1784003cd0 con 0x7f176c004d00 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.669+0000 7f178be00640 1 -- 172.21.15.53:0/64273736 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1784005310 con 0x7f176c004d00 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1793000640 1 -- 172.21.15.53:0/64273736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 msgr2=0x7f176c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1793000640 1 --2- 172.21.15.53:0/64273736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f176c005180 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f1784001e50 tx=0x7f176c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1792600640 1 -- 172.21.15.53:0/64273736 reap_dead start 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1793000640 1 -- 172.21.15.53:0/64273736 shutdown_connections 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1793000640 1 -- 172.21.15.53:0/64273736 >> 172.21.15.53:0/64273736 conn(0x7f178c0d3090 msgr2=0x7f178c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1793000640 1 -- 172.21.15.53:0/64273736 shutdown_connections 2024-01-16T15:13:23.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.670+0000 7f1793000640 1 -- 172.21.15.53:0/64273736 wait complete. 2024-01-16T15:13:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.672+0000 7f1793000640 1 Processor -- start 2024-01-16T15:13:23.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.672+0000 7f1793000640 1 -- start start 2024-01-16T15:13:23.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1793000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f178c151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1793000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f178c0caa70 0x7f178c0b6df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1793000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c0c8620 0x7f178c0c8ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1793000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0c9010 con 0x7f176c004d00 2024-01-16T15:13:23.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1793000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0b7760 con 0x7f178c0caa70 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1793000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f178c0b7a40 con 0x7f178c0c8620 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1791c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f178c151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1791c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f178c151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41290/0 (socket says 172.21.15.53:41290) 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.673+0000 7f1791c00640 1 -- 172.21.15.53:0/3194914215 learned_addr learned my addr 172.21.15.53:0/3194914215 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791200640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f178c0caa70 0x7f178c0b6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1792600640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c0c8620 0x7f178c0c8ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791c00640 1 -- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c0c8620 msgr2=0x7f178c0c8ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791c00640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c0c8620 0x7f178c0c8ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791c00640 1 -- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f178c0caa70 msgr2=0x7f178c0b6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791c00640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f178c0caa70 0x7f178c0b6df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791c00640 1 -- 172.21.15.53:0/3194914215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f178c0b7d20 con 0x7f176c004d00 2024-01-16T15:13:23.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1791c00640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f178c151510 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f177c010860 tx=0x7f177c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c009610 con 0x7f176c004d00 2024-01-16T15:13:23.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c00eb60 con 0x7f176c004d00 2024-01-16T15:13:23.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.674+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c01c400 con 0x7f176c004d00 2024-01-16T15:13:23.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.675+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f178c0b8000 con 0x7f176c004d00 2024-01-16T15:13:23.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.675+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f178c151a50 con 0x7f176c004d00 2024-01-16T15:13:23.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.676+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f178c067450 con 0x7f176c004d00 2024-01-16T15:13:23.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.677+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f177c01c560 con 0x7f176c004d00 2024-01-16T15:13:23.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.678+0000 7f1789600640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f176406bf50 0x7f176406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:23.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.678+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f177c088b10 con 0x7f176c004d00 2024-01-16T15:13:23.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.679+0000 7f1791200640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f176406bf50 0x7f176406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:23.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.680+0000 7f1791200640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f176406bf50 0x7f176406e410 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f178c0c6870 tx=0x7f1774009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:23.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.683+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f177c055480 con 0x7f176c004d00 2024-01-16T15:13:23.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.912+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f178c08c7e0 con 0x7f176c004d00 2024-01-16T15:13:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.913+0000 7f1789600640 1 -- 172.21.15.53:0/3194914215 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f177c021e90 con 0x7f176c004d00 2024-01-16T15:13:23.893 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:23.893 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/1311204867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1311204867},{"type":"v1","addr":"172.21.15.146:6836","nonce":1311204867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1106083775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1106083775},{"type":"v1","addr":"172.21.15.53:6834","nonce":1106083775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2940561403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2940561403},{"type":"v1","addr":"172.21.15.146:6837","nonce":2940561403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7029,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3081217400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3081217400},{"type":"v1","addr":"172.21.15.53:6835","nonce":3081217400}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2024-01-16T15:13:23.893 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 48 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f176406bf50 msgr2=0x7f176406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f176406bf50 0x7f176406e410 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f178c0c6870 tx=0x7f1774009040 comp rx=0 tx=0).stop 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 msgr2=0x7f178c151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f178c151510 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f177c010860 tx=0x7f177c010d20 comp rx=0 tx=0).stop 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 shutdown_connections 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f176406bf50 0x7f176406e410 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f178c0c8620 0x7f178c0c8ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f178c0caa70 0x7f178c0b6df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 --2- 172.21.15.53:0/3194914215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f176c004d00 0x7f178c151510 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 >> 172.21.15.53:0/3194914215 conn(0x7f178c0d3090 msgr2=0x7f178c06aeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 shutdown_connections 2024-01-16T15:13:23.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:23.916+0000 7f1793000640 1 -- 172.21.15.53:0/3194914215 wait complete. 2024-01-16T15:13:23.916 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:13:23.969 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:23.976 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:24.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.141+0000 7fc7f1000640 1 Processor -- start 2024-01-16T15:13:24.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.141+0000 7fc7f1000640 1 -- start start 2024-01-16T15:13:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.142+0000 7fc7f1000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc7ec0adf20 0x7fc7ec070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.142+0000 7fc7f1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 0x7fc7ec0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.142+0000 7fc7f1000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec15f4e0 con 0x7fc7ec0cc980 2024-01-16T15:13:24.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7f1000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7ec15f970 con 0x7fc7ec0adf20 2024-01-16T15:13:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7f1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec0cce10 con 0x7fc7ec15bf10 2024-01-16T15:13:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7ebe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc7ec0cc980 0x7fc7ec143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39306/0 (socket says 172.21.15.53:39306) 2024-01-16T15:13:24.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7ebe00640 1 -- 172.21.15.53:0/3499086311 learned_addr learned my addr 172.21.15.53:0/3499086311 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7eb400640 1 --2- 172.21.15.53:0/3499086311 >> v2:172.21.15.146:3300/0 conn(0x7fc7ec0adf20 0x7fc7ec070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161995109 0 0) 0x7fc7ec15f4e0 con 0x7fc7ec0cc980 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7cc003520 con 0x7fc7ec0cc980 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.143+0000 7fc7e3e00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 0x7fc7ec0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3647145964 0 0) 0x7fc7cc003520 con 0x7fc7ec0cc980 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 msgr2=0x7fc7ec0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 0x7fc7ec0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 >> v2:172.21.15.146:3300/0 conn(0x7fc7ec0adf20 msgr2=0x7fc7ec070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 --2- 172.21.15.53:0/3499086311 >> v2:172.21.15.146:3300/0 conn(0x7fc7ec0adf20 0x7fc7ec070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec071130 con 0x7fc7ec0cc980 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fc7e40019f0 con 0x7fc7ec0cc980 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 >> v1:172.21.15.53:6789/0 conn(0x7fc7ec0cc980 legacy=0x7fc7ec143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 0x7fc7cc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc003ef0 0x7fc7cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.144+0000 7fc7eaa00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7cc004d00 0x7fc7cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec15f970 con 0x7fc7cc004d00 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec0cce10 con 0x7fc7cc003ef0 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7cc00b9d0 con 0x7fc7ec15bf10 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7cc004d00 0x7fc7cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7e3e00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc003ef0 0x7fc7cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7eb400640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 0x7fc7cc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7ebe00640 1 -- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 msgr2=0x7fc7cc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15bf10 0x7fc7cc003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7ebe00640 1 -- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc003ef0 msgr2=0x7fc7cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc003ef0 0x7fc7cc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.145+0000 7fc7ebe00640 1 -- 172.21.15.53:0/3499086311 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7cc003520 con 0x7fc7cc004d00 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.146+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7cc004d00 0x7fc7cc005180 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400a870 tx=0x7fc7cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f87516193c2b40dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034630 con 0x7fc7cc004d00 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e402f070 con 0x7fc7cc004d00 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7eaa00640 1 -- 172.21.15.53:0/3499086311 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034a50 con 0x7fc7cc004d00 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7f1000640 1 -- 172.21.15.53:0/3499086311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7cc004d00 msgr2=0x7fc7cc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7f1000640 1 --2- 172.21.15.53:0/3499086311 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7cc004d00 0x7fc7cc005180 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400a870 tx=0x7fc7cc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7ebe00640 1 -- 172.21.15.53:0/3499086311 reap_dead start 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7f1000640 1 -- 172.21.15.53:0/3499086311 shutdown_connections 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.147+0000 7fc7f1000640 1 -- 172.21.15.53:0/3499086311 >> 172.21.15.53:0/3499086311 conn(0x7fc7ec06a7d0 msgr2=0x7fc7ec06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.148+0000 7fc7f1000640 1 -- 172.21.15.53:0/3499086311 shutdown_connections 2024-01-16T15:13:24.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.148+0000 7fc7f1000640 1 -- 172.21.15.53:0/3499086311 wait complete. 2024-01-16T15:13:24.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.149+0000 7fc7f1000640 1 Processor -- start 2024-01-16T15:13:24.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.149+0000 7fc7f1000640 1 -- start start 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.150+0000 7fc7f1000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc004d00 0x7fc7ec15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.150+0000 7fc7f1000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15cae0 0x7fc7ec1f5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.150+0000 7fc7f1000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 0x7fc7ec1f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.150+0000 7fc7f1000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec133200 con 0x7fc7ec1f6400 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.150+0000 7fc7f1000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec15cf40 con 0x7fc7cc004d00 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.150+0000 7fc7f1000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7ec15dc10 con 0x7fc7ec15cae0 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 0x7fc7ec1f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 0x7fc7ec1f6880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41304/0 (socket says 172.21.15.53:41304) 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 -- 172.21.15.53:0/4080593879 learned_addr learned my addr 172.21.15.53:0/4080593879 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:24.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7e3e00640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15cae0 0x7fc7ec1f5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7eb400640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc004d00 0x7fc7ec15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 -- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15cae0 msgr2=0x7fc7ec1f5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15cae0 0x7fc7ec1f5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 -- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc004d00 msgr2=0x7fc7ec15c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc004d00 0x7fc7ec15c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.151+0000 7fc7ebe00640 1 -- 172.21.15.53:0/4080593879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec135e70 con 0x7fc7ec1f6400 2024-01-16T15:13:24.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.152+0000 7fc7ebe00640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 0x7fc7ec1f6880 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400a870 tx=0x7fc7ec15f450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.152+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034640 con 0x7fc7ec1f6400 2024-01-16T15:13:24.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.152+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e40347a0 con 0x7fc7ec1f6400 2024-01-16T15:13:24.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.152+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e4034b10 con 0x7fc7ec1f6400 2024-01-16T15:13:24.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.152+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec136150 con 0x7fc7ec1f6400 2024-01-16T15:13:24.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.152+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec0c8490 con 0x7fc7ec1f6400 2024-01-16T15:13:24.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.153+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ec067450 con 0x7fc7ec1f6400 2024-01-16T15:13:24.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.154+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc7e4031030 con 0x7fc7ec1f6400 2024-01-16T15:13:24.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.155+0000 7fc7e3400640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c406bf50 0x7fc7c406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.155+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc7e40aaa00 con 0x7fc7ec1f6400 2024-01-16T15:13:24.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.156+0000 7fc7eb400640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c406bf50 0x7fc7c406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.157+0000 7fc7eb400640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c406bf50 0x7fc7c406e410 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009a90 tx=0x7fc7dc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.160+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7e40773e0 con 0x7fc7ec1f6400 2024-01-16T15:13:24.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.357+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7fc7ec067450 con 0x7fc7ec1f6400 2024-01-16T15:13:24.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.365+0000 7fc7e3400640 1 -- 172.21.15.53:0/4080593879 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v28) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc7e407aa00 con 0x7fc7ec1f6400 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c406bf50 msgr2=0x7fc7c406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c406bf50 0x7fc7c406e410 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc009a90 tx=0x7fc7dc008040 comp rx=0 tx=0).stop 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 msgr2=0x7fc7ec1f6880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 0x7fc7ec1f6880 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fc7e400a870 tx=0x7fc7ec15f450 comp rx=0 tx=0).stop 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 shutdown_connections 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c406bf50 0x7fc7c406e410 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7ec1f6400 0x7fc7ec1f6880 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7ec15cae0 0x7fc7ec1f5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 --2- 172.21.15.53:0/4080593879 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7cc004d00 0x7fc7ec15c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 >> 172.21.15.53:0/4080593879 conn(0x7fc7ec06a7d0 msgr2=0x7fc7ec0cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 shutdown_connections 2024-01-16T15:13:24.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.368+0000 7fc7f1000640 1 -- 172.21.15.53:0/4080593879 wait complete. 2024-01-16T15:13:24.367 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:13:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.610+0000 7f9f93600640 1 Processor -- start 2024-01-16T15:13:24.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.610+0000 7f9f93600640 1 -- start start 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f93600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9f8c0b3ae0 0x7f9f8c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f93600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c070c40 con 0x7f9f8c0cf290 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f93600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c06c470 con 0x7f9f8c0ae5a0 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f93600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c071450 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f83e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9f8c0b3ae0 0x7f9f8c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f83e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9f8c0b3ae0 0x7f9f8c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41316/0 (socket says 172.21.15.53:41316) 2024-01-16T15:13:24.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.612+0000 7f9f83e00640 1 -- 172.21.15.53:0/3787577897 learned_addr learned my addr 172.21.15.53:0/3787577897 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.613+0000 7f9f83e00640 1 -- 172.21.15.53:0/3787577897 >> v1:172.21.15.146:6789/0 conn(0x7f9f8c0ae5a0 legacy=0x7f9f8c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.613+0000 7f9f83e00640 1 -- 172.21.15.53:0/3787577897 >> v1:172.21.15.53:6789/0 conn(0x7f9f8c0cf290 legacy=0x7f9f8c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.613+0000 7f9f83e00640 1 -- 172.21.15.53:0/3787577897 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c06c750 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.614+0000 7f9f83e00640 1 --2- 172.21.15.53:0/3787577897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c15e430 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009870 tx=0x7f9f7402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc52d3b4d2491c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.614+0000 7f9f8be00640 1 -- 172.21.15.53:0/3787577897 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7402f840 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.614+0000 7f9f8be00640 1 -- 172.21.15.53:0/3787577897 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7402f9a0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.614+0000 7f9f8be00640 1 -- 172.21.15.53:0/3787577897 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7402fd30 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 -- 172.21.15.53:0/3787577897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 msgr2=0x7f9f8c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 --2- 172.21.15.53:0/3787577897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c15e430 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009870 tx=0x7f9f7402ba20 comp rx=0 tx=0).stop 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 -- 172.21.15.53:0/3787577897 shutdown_connections 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 --2- 172.21.15.53:0/3787577897 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c15e430 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 -- 172.21.15.53:0/3787577897 >> 172.21.15.53:0/3787577897 conn(0x7f9f8c06a800 msgr2=0x7f9f8c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 -- 172.21.15.53:0/3787577897 shutdown_connections 2024-01-16T15:13:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.615+0000 7f9f93600640 1 -- 172.21.15.53:0/3787577897 wait complete. 2024-01-16T15:13:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.617+0000 7f9f93600640 1 Processor -- start 2024-01-16T15:13:24.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.617+0000 7f9f93600640 1 -- start start 2024-01-16T15:13:24.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.618+0000 7f9f93600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0ae5a0 0x7f9f8c0acfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.618+0000 7f9f93600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c0cd400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.618+0000 7f9f93600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cf290 0x7f9f8c0cb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.618+0000 7f9f93600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0ccfe0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.618+0000 7f9f93600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0cc360 con 0x7f9f8c0ae5a0 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.618+0000 7f9f93600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0cc640 con 0x7f9f8c0cf290 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c0cd400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c0cd400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41322/0 (socket says 172.21.15.53:41322) 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 -- 172.21.15.53:0/4154640858 learned_addr learned my addr 172.21.15.53:0/4154640858 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f91600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cf290 0x7f9f8c0cb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f90c00640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0ae5a0 0x7f9f8c0acfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 -- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cf290 msgr2=0x7f9f8c0cb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cf290 0x7f9f8c0cb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 -- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0ae5a0 msgr2=0x7f9f8c0acfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0ae5a0 0x7f9f8c0acfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 -- 172.21.15.53:0/4154640858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f740094d0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.619+0000 7f9f83e00640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c0cd400 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009870 tx=0x7f9f740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.620+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f7403b040 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.620+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7402f7d0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.620+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f740426c0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.620+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c0cc920 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.620+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c0c40e0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.621+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c0cd120 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.622+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9f74042820 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.623+0000 7f9f89600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f6406bf20 0x7f9f6406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.623+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9f740ab9d0 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.624+0000 7f9f90c00640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f6406bf20 0x7f9f6406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:24.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.625+0000 7f9f90c00640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f6406bf20 0x7f9f6406e3e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c004ea0 tx=0x7f9f7c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:24.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.628+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9f8c0cd120 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.827+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f9f8c0cd120 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.833+0000 7f9f89600640 1 -- 172.21.15.53:0/4154640858 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v29) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9f8c0cd120 con 0x7f9f8c0b3ae0 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f6406bf20 msgr2=0x7f9f6406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f6406bf20 0x7f9f6406e3e0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c004ea0 tx=0x7f9f7c009040 comp rx=0 tx=0).stop 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 msgr2=0x7f9f8c0cd400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c0cd400 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f9f74009870 tx=0x7f9f740046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 shutdown_connections 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9f6406bf20 0x7f9f6406e3e0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9f8c0cf290 0x7f9f8c0cb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9f8c0b3ae0 0x7f9f8c0cd400 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 --2- 172.21.15.53:0/4154640858 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9f8c0ae5a0 0x7f9f8c0acfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 >> 172.21.15.53:0/4154640858 conn(0x7f9f8c06a800 msgr2=0x7f9f8c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:24.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 shutdown_connections 2024-01-16T15:13:24.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:24.836+0000 7f9f93600640 1 -- 172.21.15.53:0/4154640858 wait complete. 2024-01-16T15:13:24.835 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added' 2024-01-16T15:13:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.079+0000 7fd5f6600640 1 Processor -- start 2024-01-16T15:13:25.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.079+0000 7fd5f6600640 1 -- start start 2024-01-16T15:13:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f6600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5f00d0150 0x7fd5f00d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f6600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5f00ceea0 0x7fd5f00b3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f6600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f00b2c70 con 0x7fd5f015d3c0 2024-01-16T15:13:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f6600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f00cf6d0 con 0x7fd5f00ceea0 2024-01-16T15:13:25.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f6600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f00cf3f0 con 0x7fd5f00d0150 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5f00ceea0 0x7fd5f00b3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f5200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd5f015d3c0 0x7fd5f00cfa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39322/0 (socket says 172.21.15.53:39322) 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.081+0000 7fd5f5200640 1 -- 172.21.15.53:0/1188725068 learned_addr learned my addr 172.21.15.53:0/1188725068 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.082+0000 7fd5e7e00640 1 --2- 172.21.15.53:0/1188725068 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00d0150 0x7fd5f00d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.082+0000 7fd5f5c00640 1 -- 172.21.15.53:0/1188725068 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00d0150 msgr2=0x7fd5f00d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.082+0000 7fd5f5c00640 1 --2- 172.21.15.53:0/1188725068 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00d0150 0x7fd5f00d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.082+0000 7fd5f5c00640 1 -- 172.21.15.53:0/1188725068 >> v1:172.21.15.53:6789/0 conn(0x7fd5f015d3c0 legacy=0x7fd5f00cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.082+0000 7fd5f5c00640 1 -- 172.21.15.53:0/1188725068 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00b2620 con 0x7fd5f00ceea0 2024-01-16T15:13:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.083+0000 7fd5f5c00640 1 --2- 172.21.15.53:0/1188725068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00ceea0 0x7fd5f00b3100 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00b5be0 tx=0x7fd5e802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f0a5f39e7bae851 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.083+0000 7fd5efe00640 1 -- 172.21.15.53:0/1188725068 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e802cc70 con 0x7fd5f00ceea0 2024-01-16T15:13:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.083+0000 7fd5efe00640 1 -- 172.21.15.53:0/1188725068 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e802cdd0 con 0x7fd5f00ceea0 2024-01-16T15:13:25.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.083+0000 7fd5efe00640 1 -- 172.21.15.53:0/1188725068 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e8034660 con 0x7fd5f00ceea0 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 -- 172.21.15.53:0/1188725068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00ceea0 msgr2=0x7fd5f00b3100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1188725068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00ceea0 0x7fd5f00b3100 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fd5f00b5be0 tx=0x7fd5e802bec0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 -- 172.21.15.53:0/1188725068 shutdown_connections 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1188725068 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00ceea0 0x7fd5f00b3100 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1188725068 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00d0150 0x7fd5f00d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 -- 172.21.15.53:0/1188725068 >> 172.21.15.53:0/1188725068 conn(0x7fd5f00d3050 msgr2=0x7fd5f00d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 -- 172.21.15.53:0/1188725068 shutdown_connections 2024-01-16T15:13:25.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.084+0000 7fd5f6600640 1 -- 172.21.15.53:0/1188725068 wait complete. 2024-01-16T15:13:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.086+0000 7fd5f6600640 1 Processor -- start 2024-01-16T15:13:25.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.086+0000 7fd5f6600640 1 -- start start 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5f6600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00ceea0 0x7fd5f0151310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5f6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 0x7fd5f01f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5f6600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f015d3c0 0x7fd5f00b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5f6600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f00b7170 con 0x7fd5f00d0150 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5f6600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f01f6d60 con 0x7fd5f015d3c0 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5f6600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f01f7040 con 0x7fd5f00ceea0 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 0x7fd5f01f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 0x7fd5f01f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41340/0 (socket says 172.21.15.53:41340) 2024-01-16T15:13:25.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.087+0000 7fd5e7e00640 1 -- 172.21.15.53:0/1017522535 learned_addr learned my addr 172.21.15.53:0/1017522535 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5f5c00640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f015d3c0 0x7fd5f00b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5f5200640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00ceea0 0x7fd5f0151310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5e7e00640 1 -- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00ceea0 msgr2=0x7fd5f0151310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5e7e00640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00ceea0 0x7fd5f0151310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5e7e00640 1 -- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f015d3c0 msgr2=0x7fd5f00b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5e7e00640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f015d3c0 0x7fd5f00b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5e7e00640 1 -- 172.21.15.53:0/1017522535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e8009a70 con 0x7fd5f00d0150 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5e7e00640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 0x7fd5f01f62a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800d800 tx=0x7fd5d800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d800b6c0 con 0x7fd5f00d0150 2024-01-16T15:13:25.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d800b820 con 0x7fd5f00d0150 2024-01-16T15:13:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.088+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f00b7410 con 0x7fd5f00d0150 2024-01-16T15:13:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.089+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d800bbb0 con 0x7fd5f00d0150 2024-01-16T15:13:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.089+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f00c6dd0 con 0x7fd5f00d0150 2024-01-16T15:13:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.090+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f008ba80 con 0x7fd5f00d0150 2024-01-16T15:13:25.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.091+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fd5d800bd10 con 0x7fd5f00d0150 2024-01-16T15:13:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.092+0000 7fd5ed600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806bf20 0x7fd5c806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.092+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd5d8088b20 con 0x7fd5f00d0150 2024-01-16T15:13:25.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.093+0000 7fd5f5200640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806bf20 0x7fd5c806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.094+0000 7fd5f5200640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806bf20 0x7fd5c806e3e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0005880 tx=0x7fd5e0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.097+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5d8055440 con 0x7fd5f00d0150 2024-01-16T15:13:25.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.369+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]} v 0) v1 -- 0x7fd5f00d0f20 con 0x7fd5f00d0150 2024-01-16T15:13:25.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.420+0000 7fd5ed600640 1 -- 172.21.15.53:0/1017522535 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_progress_drops_when_new_jobs_are_added"]}]=0 v593) v1 ==== 166+0+0 (secure 0 0 0) 0x7fd5d8058a60 con 0x7fd5f00d0150 2024-01-16T15:13:25.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.423+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806bf20 msgr2=0x7fd5c806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.423+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806bf20 0x7fd5c806e3e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0005880 tx=0x7fd5e0009040 comp rx=0 tx=0).stop 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 msgr2=0x7fd5f01f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 0x7fd5f01f62a0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fd5d800d800 tx=0x7fd5d800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 shutdown_connections 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806bf20 0x7fd5c806e3e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f015d3c0 0x7fd5f00b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5f00d0150 0x7fd5f01f62a0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 --2- 172.21.15.53:0/1017522535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00ceea0 0x7fd5f0151310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 >> 172.21.15.53:0/1017522535 conn(0x7fd5f00d3050 msgr2=0x7fd5f006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 shutdown_connections 2024-01-16T15:13:25.405 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.424+0000 7fd5f6600640 1 -- 172.21.15.53:0/1017522535 wait complete. 2024-01-16T15:13:25.424 INFO:tasks.cephfs_test_runner:X = number of clones issued. ... ok 2024-01-16T15:13:25.424 INFO:tasks.cephfs_test_runner:Starting test: test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:13:25.424 INFO:tasks.cephfs_test_runner:test_when_1_clone_is_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:13:25.424 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:25.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.656+0000 7f8f8ce00640 1 Processor -- start 2024-01-16T15:13:25.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.656+0000 7f8f8ce00640 1 -- start start 2024-01-16T15:13:25.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.658+0000 7f8f8ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071410 0x7f8f88146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.658+0000 7f8f8ce00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f8806c7c0 0x7f8f880717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.658+0000 7f8f8ce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f8806cc20 con 0x7f8f88071d30 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.658+0000 7f8f8ce00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f8806cec0 con 0x7f8f8806c7c0 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.658+0000 7f8f8ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8806d1a0 con 0x7f8f88071410 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f8806c7c0 0x7f8f880717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8f8806c7c0 0x7f8f880717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41342/0 (socket says 172.21.15.53:41342) 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 -- 172.21.15.53:0/4202233095 learned_addr learned my addr 172.21.15.53:0/4202233095 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:25.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f87400640 1 --2- 172.21.15.53:0/4202233095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071410 0x7f8f88146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 -- 172.21.15.53:0/4202233095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071410 msgr2=0x7f8f88146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:25.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 --2- 172.21.15.53:0/4202233095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071410 0x7f8f88146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 -- 172.21.15.53:0/4202233095 >> v1:172.21.15.146:6789/0 conn(0x7f8f88071d30 legacy=0x7f8f880b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.659+0000 7f8f86a00640 1 -- 172.21.15.53:0/4202233095 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8806d480 con 0x7f8f8806c7c0 2024-01-16T15:13:25.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.660+0000 7f8f86a00640 1 --2- 172.21.15.53:0/4202233095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8806c7c0 0x7f8f880717f0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009870 tx=0x7f8f7002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f1a7bbeaafbfac6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.661+0000 7f8f86000640 1 -- 172.21.15.53:0/4202233095 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70030840 con 0x7f8f8806c7c0 2024-01-16T15:13:25.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.661+0000 7f8f86000640 1 -- 172.21.15.53:0/4202233095 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f700309a0 con 0x7f8f8806c7c0 2024-01-16T15:13:25.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.661+0000 7f8f86000640 1 -- 172.21.15.53:0/4202233095 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70030d30 con 0x7f8f8806c7c0 2024-01-16T15:13:25.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.661+0000 7f8f8ce00640 1 -- 172.21.15.53:0/4202233095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8806c7c0 msgr2=0x7f8f880717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.661+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/4202233095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8806c7c0 0x7f8f880717f0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7f8f70009870 tx=0x7f8f7002b980 comp rx=0 tx=0).stop 2024-01-16T15:13:25.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.662+0000 7f8f8ce00640 1 -- 172.21.15.53:0/4202233095 shutdown_connections 2024-01-16T15:13:25.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.662+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/4202233095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f8806c7c0 0x7f8f880717f0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.662+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/4202233095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071410 0x7f8f88146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.662+0000 7f8f8ce00640 1 -- 172.21.15.53:0/4202233095 >> 172.21.15.53:0/4202233095 conn(0x7f8f8806a800 msgr2=0x7f8f8806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.662+0000 7f8f8ce00640 1 -- 172.21.15.53:0/4202233095 shutdown_connections 2024-01-16T15:13:25.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.662+0000 7f8f8ce00640 1 -- 172.21.15.53:0/4202233095 wait complete. 2024-01-16T15:13:25.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.663+0000 7f8f8ce00640 1 Processor -- start 2024-01-16T15:13:25.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.663+0000 7f8f8ce00640 1 -- start start 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.664+0000 7f8f8ce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8806c7c0 0x7f8f8813eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.664+0000 7f8f8ce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 0x7f8f881e2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.664+0000 7f8f8ce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071d30 0x7f8f881e75f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.664+0000 7f8f8ce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881e3d60 con 0x7f8f88071410 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.664+0000 7f8f8ce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881e37a0 con 0x7f8f8806c7c0 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.664+0000 7f8f8ce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f881e3a80 con 0x7f8f88071d30 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 0x7f8f881e2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 0x7f8f881e2e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41348/0 (socket says 172.21.15.53:41348) 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 -- 172.21.15.53:0/3047903582 learned_addr learned my addr 172.21.15.53:0/3047903582 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f87400640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8806c7c0 0x7f8f8813eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f87e00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071d30 0x7f8f881e75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 -- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071d30 msgr2=0x7f8f881e75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071d30 0x7f8f881e75f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 -- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8806c7c0 msgr2=0x7f8f8813eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8806c7c0 0x7f8f8813eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 -- 172.21.15.53:0/3047903582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f700094d0 con 0x7f8f88071410 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f87400640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8806c7c0 0x7f8f8813eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:25.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.665+0000 7f8f86a00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 0x7f8f881e2e30 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8f70000c00 tx=0x7f8f70004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.666+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70039c90 con 0x7f8f88071410 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.666+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f70039df0 con 0x7f8f88071410 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.666+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f70043640 con 0x7f8f88071410 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.666+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f881e4e90 con 0x7f8f88071410 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.666+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8813f050 con 0x7f8f88071410 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.667+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f881e53b0 con 0x7f8f88071410 2024-01-16T15:13:25.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.668+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8f700437a0 con 0x7f8f88071410 2024-01-16T15:13:25.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.669+0000 7f8f7f400640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6006bf20 0x7f8f6006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:25.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.669+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f8f7003e070 con 0x7f8f88071410 2024-01-16T15:13:25.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.670+0000 7f8f87400640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6006bf20 0x7f8f6006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:25.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.670+0000 7f8f87400640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6006bf20 0x7f8f6006e3e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8f881e6bd0 tx=0x7f8f78008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:25.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.673+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8f881e53b0 con 0x7f8f88071410 2024-01-16T15:13:25.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.862+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f881e53b0 con 0x7f8f88071410 2024-01-16T15:13:25.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.864+0000 7f8f7f400640 1 -- 172.21.15.53:0/3047903582 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f8f881e53b0 con 0x7f8f88071410 2024-01-16T15:13:25.844 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:25.844 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":78,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:19.367883+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.146:6836/867019165":"2024-01-17T15:13:16.691312+0000","172.21.15.146:6834/867019165":"2024-01-17T15:13:16.691312+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:25.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6006bf20 msgr2=0x7f8f6006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6006bf20 0x7f8f6006e3e0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f8f881e6bd0 tx=0x7f8f78008040 comp rx=0 tx=0).stop 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 msgr2=0x7f8f881e2e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 0x7f8f881e2e30 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f8f70000c00 tx=0x7f8f70004730 comp rx=0 tx=0).stop 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 shutdown_connections 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8f6006bf20 0x7f8f6006e3e0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8f88071d30 0x7f8f881e75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8f88071410 0x7f8f881e2e30 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 --2- 172.21.15.53:0/3047903582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8f8806c7c0 0x7f8f8813eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:25.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 >> 172.21.15.53:0/3047903582 conn(0x7f8f8806a800 msgr2=0x7f8f881465a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:25.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 shutdown_connections 2024-01-16T15:13:25.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:25.867+0000 7f8f8ce00640 1 -- 172.21.15.53:0/3047903582 wait complete. 2024-01-16T15:13:25.865 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:13:25.865 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:13:26.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.106+0000 7f45d5a00640 1 Processor -- start 2024-01-16T15:13:26.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.107+0000 7f45d5a00640 1 -- start start 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.108+0000 7f45d5a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45d015d4b0 0x7f45d00d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.108+0000 7f45d5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d00cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.108+0000 7f45d5a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d00d0320 con 0x7f45d00cf9a0 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.108+0000 7f45d5a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d00b2730 con 0x7f45d015d4b0 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.108+0000 7f45d5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00d1ec0 con 0x7f45d00b41e0 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.108+0000 7f45cfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45d015d4b0 0x7f45d00d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45cfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45d015d4b0 0x7f45d00d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41350/0 (socket says 172.21.15.53:41350) 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45cfe00640 1 -- 172.21.15.53:0/4242403022 learned_addr learned my addr 172.21.15.53:0/4242403022 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45d5000640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d00cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45cfe00640 1 -- 172.21.15.53:0/4242403022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 msgr2=0x7f45d00cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45cfe00640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d00cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45cfe00640 1 -- 172.21.15.53:0/4242403022 >> v1:172.21.15.146:6789/0 conn(0x7f45d00cf9a0 legacy=0x7f45d00cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45cfe00640 1 -- 172.21.15.53:0/4242403022 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d00b29d0 con 0x7f45d015d4b0 2024-01-16T15:13:26.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.109+0000 7f45d5000640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d00cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:26.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.110+0000 7f45cfe00640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00d1550 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009870 tx=0x7f45c002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4889eccd0c0cab43 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.110+0000 7f45cea00640 1 -- 172.21.15.53:0/4242403022 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0031640 con 0x7f45d015d4b0 2024-01-16T15:13:26.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45cea00640 1 -- 172.21.15.53:0/4242403022 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c00317a0 con 0x7f45d015d4b0 2024-01-16T15:13:26.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45cea00640 1 -- 172.21.15.53:0/4242403022 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0031b30 con 0x7f45d015d4b0 2024-01-16T15:13:26.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45d5a00640 1 -- 172.21.15.53:0/4242403022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 msgr2=0x7f45d00d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45d5a00640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00d1550 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009870 tx=0x7f45c002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45d5a00640 1 -- 172.21.15.53:0/4242403022 shutdown_connections 2024-01-16T15:13:26.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45d5a00640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d00cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45d5a00640 1 --2- 172.21.15.53:0/4242403022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00d1550 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.111+0000 7f45d5a00640 1 -- 172.21.15.53:0/4242403022 >> 172.21.15.53:0/4242403022 conn(0x7f45d00d3000 msgr2=0x7f45d00d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:26.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.112+0000 7f45d5a00640 1 -- 172.21.15.53:0/4242403022 shutdown_connections 2024-01-16T15:13:26.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.112+0000 7f45d5a00640 1 -- 172.21.15.53:0/4242403022 wait complete. 2024-01-16T15:13:26.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.114+0000 7f45d5a00640 1 Processor -- start 2024-01-16T15:13:26.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.114+0000 7f45d5a00640 1 -- start start 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.115+0000 7f45d5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d0157db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.115+0000 7f45d5a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d00cf9a0 0x7f45d0151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.115+0000 7f45d5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00ccb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00cbae0 con 0x7f45d015d4b0 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00cbdc0 con 0x7f45d00cf9a0 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d00cc0a0 con 0x7f45d00b41e0 2024-01-16T15:13:26.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00ccb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00ccb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41358/0 (socket says 172.21.15.53:41358) 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 -- 172.21.15.53:0/2640991470 learned_addr learned my addr 172.21.15.53:0/2640991470 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45cfe00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d0157db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45cf400640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d00cf9a0 0x7f45d0151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 -- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 msgr2=0x7f45d0157db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d0157db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 -- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d00cf9a0 msgr2=0x7f45d0151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d00cf9a0 0x7f45d0151350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.116+0000 7f45d5000640 1 -- 172.21.15.53:0/2640991470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c00094d0 con 0x7f45d015d4b0 2024-01-16T15:13:26.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.117+0000 7f45d5000640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00ccb80 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f45c800cea0 tx=0x7f45c800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.117+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8003b90 con 0x7f45d015d4b0 2024-01-16T15:13:26.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.117+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c8003d10 con 0x7f45d015d4b0 2024-01-16T15:13:26.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.117+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d015da10 con 0x7f45d015d4b0 2024-01-16T15:13:26.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.117+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8004260 con 0x7f45d015d4b0 2024-01-16T15:13:26.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.117+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d00cc660 con 0x7f45d015d4b0 2024-01-16T15:13:26.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.118+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d00cc880 con 0x7f45d015d4b0 2024-01-16T15:13:26.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.119+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f45c80043c0 con 0x7f45d015d4b0 2024-01-16T15:13:26.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.120+0000 7f45c7e00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f45a806bf20 0x7f45a806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.120+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f45c8089190 con 0x7f45d015d4b0 2024-01-16T15:13:26.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.121+0000 7f45cfe00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f45a806bf20 0x7f45a806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.122+0000 7f45cfe00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f45a806bf20 0x7f45a806e3e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f45c0036000 tx=0x7f45c003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.124+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f45c80557d0 con 0x7f45d015d4b0 2024-01-16T15:13:26.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.319+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f45d0067450 con 0x7f45d015d4b0 2024-01-16T15:13:26.299 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.320+0000 7f45c7e00640 1 -- 172.21.15.53:0/2640991470 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f45c8058df0 con 0x7f45d015d4b0 2024-01-16T15:13:26.300 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:13:26.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f45a806bf20 msgr2=0x7f45a806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f45a806bf20 0x7f45a806e3e0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f45c0036000 tx=0x7f45c003c040 comp rx=0 tx=0).stop 2024-01-16T15:13:26.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 msgr2=0x7f45d00ccb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.302 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00ccb80 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7f45c800cea0 tx=0x7f45c800cf90 comp rx=0 tx=0).stop 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 shutdown_connections 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f45a806bf20 0x7f45a806e3e0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45d015d4b0 0x7f45d00ccb80 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f45d00cf9a0 0x7f45d0151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 --2- 172.21.15.53:0/2640991470 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45d00b41e0 0x7f45d0157db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 >> 172.21.15.53:0/2640991470 conn(0x7f45d00d3000 msgr2=0x7f45d00b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 shutdown_connections 2024-01-16T15:13:26.303 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.323+0000 7f45d5a00640 1 -- 172.21.15.53:0/2640991470 wait complete. 2024-01-16T15:13:26.323 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:13:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.566+0000 7f67d0c00640 1 Processor -- start 2024-01-16T15:13:26.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.566+0000 7f67d0c00640 1 -- start start 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.567+0000 7f67d0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f67cc15afb0 0x7f67cc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.567+0000 7f67d0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc13fae0 0x7f67cc0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67d0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67cc13ff30 con 0x7f67cc13cfa0 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67d0c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67cc0cdc40 con 0x7f67cc15afb0 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67d0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc0cdec0 con 0x7f67cc13fae0 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f67cc15afb0 0x7f67cc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f67cc15afb0 0x7f67cc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41364/0 (socket says 172.21.15.53:41364) 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 -- 172.21.15.53:0/1987142191 learned_addr learned my addr 172.21.15.53:0/1987142191 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67cbe00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc13fae0 0x7f67cc0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 -- 172.21.15.53:0/1987142191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc13fae0 msgr2=0x7f67cc0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc13fae0 0x7f67cc0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 -- 172.21.15.53:0/1987142191 >> v1:172.21.15.53:6789/0 conn(0x7f67cc13cfa0 legacy=0x7f67cc0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67c3e00640 1 -- 172.21.15.53:0/1987142191 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc0ce1a0 con 0x7f67cc15afb0 2024-01-16T15:13:26.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.568+0000 7f67cbe00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc13fae0 0x7f67cc0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.570+0000 7f67c3e00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc15afb0 0x7f67cc13f5a0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009870 tx=0x7f67b402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c086c2569c69c67a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.570+0000 7f67caa00640 1 -- 172.21.15.53:0/1987142191 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4031640 con 0x7f67cc15afb0 2024-01-16T15:13:26.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.570+0000 7f67caa00640 1 -- 172.21.15.53:0/1987142191 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b40317a0 con 0x7f67cc15afb0 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.570+0000 7f67caa00640 1 -- 172.21.15.53:0/1987142191 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67b4031b30 con 0x7f67cc15afb0 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.570+0000 7f67d0c00640 1 -- 172.21.15.53:0/1987142191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc15afb0 msgr2=0x7f67cc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.570+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc15afb0 0x7f67cc13f5a0 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009870 tx=0x7f67b402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.571+0000 7f67d0c00640 1 -- 172.21.15.53:0/1987142191 shutdown_connections 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.571+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc13fae0 0x7f67cc0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.571+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1987142191 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc15afb0 0x7f67cc13f5a0 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.571+0000 7f67d0c00640 1 -- 172.21.15.53:0/1987142191 >> 172.21.15.53:0/1987142191 conn(0x7f67cc06a800 msgr2=0x7f67cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.571+0000 7f67d0c00640 1 -- 172.21.15.53:0/1987142191 shutdown_connections 2024-01-16T15:13:26.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.571+0000 7f67d0c00640 1 -- 172.21.15.53:0/1987142191 wait complete. 2024-01-16T15:13:26.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.572+0000 7f67d0c00640 1 Processor -- start 2024-01-16T15:13:26.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.572+0000 7f67d0c00640 1 -- start start 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.573+0000 7f67d0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 0x7f67cc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.573+0000 7f67d0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc13fae0 0x7f67cc14c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.573+0000 7f67d0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc15afb0 0x7f67cc13bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.573+0000 7f67d0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc131400 con 0x7f67cc13cfa0 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.573+0000 7f67d0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc132840 con 0x7f67cc13fae0 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.573+0000 7f67d0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67cc167880 con 0x7f67cc15afb0 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 0x7f67cc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 0x7f67cc15bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41380/0 (socket says 172.21.15.53:41380) 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 -- 172.21.15.53:0/1535159724 learned_addr learned my addr 172.21.15.53:0/1535159724 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67c3e00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc13fae0 0x7f67cc14c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cbe00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc15afb0 0x7f67cc13bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 -- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc15afb0 msgr2=0x7f67cc13bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc15afb0 0x7f67cc13bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 -- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc13fae0 msgr2=0x7f67cc14c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc13fae0 0x7f67cc14c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:26.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.574+0000 7f67cb400640 1 -- 172.21.15.53:0/1535159724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b40094d0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.575+0000 7f67cb400640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 0x7f67cc15bd70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00c450 tx=0x7f67bc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.575+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc0052d0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.575+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67bc005430 con 0x7f67cc13cfa0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.575+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67bc012870 con 0x7f67cc13cfa0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.575+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67cc167b60 con 0x7f67cc13cfa0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.575+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc132fe0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.576+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67cc167da0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.577+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f67bc0129d0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.578+0000 7f67c3400640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406bf20 0x7f67a406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:26.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.578+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f67bc087fd0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.579+0000 7f67c3e00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406bf20 0x7f67a406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:26.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.580+0000 7f67c3e00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406bf20 0x7f67a406e3e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f67b4036000 tx=0x7f67b403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:26.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.583+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f67cc167da0 con 0x7f67cc13cfa0 2024-01-16T15:13:26.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:26.845+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7f67cc13ff30 con 0x7f67cc13cfa0 2024-01-16T15:13:27.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.426+0000 7f67c3400640 1 -- 172.21.15.53:0/1535159724 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v595) v1 ==== 156+0+0 (secure 0 0 0) 0x7f67cc13ff30 con 0x7f67cc13cfa0 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406bf20 msgr2=0x7f67a406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406bf20 0x7f67a406e3e0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f67b4036000 tx=0x7f67b403c040 comp rx=0 tx=0).stop 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 msgr2=0x7f67cc15bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 0x7f67cc15bd70 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00c450 tx=0x7f67bc00c910 comp rx=0 tx=0).stop 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 shutdown_connections 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f67a406bf20 0x7f67a406e3e0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f67cc15afb0 0x7f67cc13bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f67cc13fae0 0x7f67cc14c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 --2- 172.21.15.53:0/1535159724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f67cc13cfa0 0x7f67cc15bd70 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 >> 172.21.15.53:0/1535159724 conn(0x7f67cc06a800 msgr2=0x7f67cc15b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 shutdown_connections 2024-01-16T15:13:27.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.430+0000 7f67d0c00640 1 -- 172.21.15.53:0/1535159724 wait complete. 2024-01-16T15:13:27.432 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:13:27.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.678+0000 7febe5a00640 1 Processor -- start 2024-01-16T15:13:27.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.678+0000 7febe5a00640 1 -- start start 2024-01-16T15:13:27.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.679+0000 7febe5a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febe00cf9a0 0x7febe015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:27.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febe5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe015e1e0 0x7febe00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febe5a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00b1ee0 con 0x7febe0143ae0 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febe5a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe00b2780 con 0x7febe00cf9a0 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febe5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe015e990 con 0x7febe015e1e0 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febe00cf9a0 0x7febe015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febe00cf9a0 0x7febe015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41396/0 (socket says 172.21.15.53:41396) 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 -- 172.21.15.53:0/3592833942 learned_addr learned my addr 172.21.15.53:0/3592833942 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:27.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febdf400640 1 --2- 172.21.15.53:0/3592833942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe015e1e0 0x7febe00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:27.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 -- 172.21.15.53:0/3592833942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe015e1e0 msgr2=0x7febe00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:27.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 --2- 172.21.15.53:0/3592833942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe015e1e0 0x7febe00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 -- 172.21.15.53:0/3592833942 >> v1:172.21.15.146:6789/0 conn(0x7febe0143ae0 legacy=0x7febe00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:27.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.680+0000 7febde000640 1 -- 172.21.15.53:0/3592833942 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00b2130 con 0x7febe00cf9a0 2024-01-16T15:13:27.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.682+0000 7febde000640 1 --2- 172.21.15.53:0/3592833942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cf9a0 0x7febe015dca0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7febc800bb10 tx=0x7febc802e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8866ea4417a2db1d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:27.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.682+0000 7febdd600640 1 -- 172.21.15.53:0/3592833942 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc802ee70 con 0x7febe00cf9a0 2024-01-16T15:13:27.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.682+0000 7febdd600640 1 -- 172.21.15.53:0/3592833942 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc8036440 con 0x7febe00cf9a0 2024-01-16T15:13:27.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.682+0000 7febdd600640 1 -- 172.21.15.53:0/3592833942 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc80367d0 con 0x7febe00cf9a0 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 -- 172.21.15.53:0/3592833942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cf9a0 msgr2=0x7febe015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 --2- 172.21.15.53:0/3592833942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cf9a0 0x7febe015dca0 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7febc800bb10 tx=0x7febc802e0c0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 -- 172.21.15.53:0/3592833942 shutdown_connections 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 --2- 172.21.15.53:0/3592833942 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe015e1e0 0x7febe00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 --2- 172.21.15.53:0/3592833942 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cf9a0 0x7febe015dca0 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 -- 172.21.15.53:0/3592833942 >> 172.21.15.53:0/3592833942 conn(0x7febe006a800 msgr2=0x7febe006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 -- 172.21.15.53:0/3592833942 shutdown_connections 2024-01-16T15:13:27.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.683+0000 7febe5a00640 1 -- 172.21.15.53:0/3592833942 wait complete. 2024-01-16T15:13:27.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.685+0000 7febe5a00640 1 Processor -- start 2024-01-16T15:13:27.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.685+0000 7febe5a00640 1 -- start start 2024-01-16T15:13:27.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febe5a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe00cf9a0 0x7febe01f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:27.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febe5a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe0143ae0 0x7febe01405f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:27.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febe5a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 0x7febe0140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febe5a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00cd570 con 0x7febe015e1e0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febe5a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00cd850 con 0x7febe00cf9a0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febe5a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe00cdb30 con 0x7febe0143ae0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 0x7febe0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 0x7febe0140b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41402/0 (socket says 172.21.15.53:41402) 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 -- 172.21.15.53:0/153212987 learned_addr learned my addr 172.21.15.53:0/153212987 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febde000640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe0143ae0 0x7febe01405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdea00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe00cf9a0 0x7febe01f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 -- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe0143ae0 msgr2=0x7febe01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe0143ae0 0x7febe01405f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 -- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe00cf9a0 msgr2=0x7febe01f5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe00cf9a0 0x7febe01f5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.686+0000 7febdf400640 1 -- 172.21.15.53:0/153212987 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc800b770 con 0x7febe015e1e0 2024-01-16T15:13:27.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febdea00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe00cf9a0 0x7febe01f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:27.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febdf400640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 0x7febe0140b30 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7febd800cc30 tx=0x7febd800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:27.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd800ce40 con 0x7febe015e1e0 2024-01-16T15:13:27.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8003990 con 0x7febe015e1e0 2024-01-16T15:13:27.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd8003d20 con 0x7febe015e1e0 2024-01-16T15:13:27.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe00cde10 con 0x7febe015e1e0 2024-01-16T15:13:27.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.687+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe00cc810 con 0x7febe015e1e0 2024-01-16T15:13:27.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.688+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe00ce510 con 0x7febe015e1e0 2024-01-16T15:13:27.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.689+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7febd8003e80 con 0x7febe015e1e0 2024-01-16T15:13:27.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.690+0000 7febd6a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806bf20 0x7febb806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:27.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.690+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7febd8088c30 con 0x7febe015e1e0 2024-01-16T15:13:27.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.691+0000 7febdea00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806bf20 0x7febb806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:27.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.692+0000 7febdea00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806bf20 0x7febb806e3e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7febd0003eb0 tx=0x7febd0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:27.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.695+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febd8055270 con 0x7febe015e1e0 2024-01-16T15:13:27.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.887+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7febe00ce330 con 0x7febe015e1e0 2024-01-16T15:13:27.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.893+0000 7febd6a00640 1 -- 172.21.15.53:0/153212987 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7febe00ce330 con 0x7febe015e1e0 2024-01-16T15:13:27.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806bf20 msgr2=0x7febb806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806bf20 0x7febb806e3e0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7febd0003eb0 tx=0x7febd0009040 comp rx=0 tx=0).stop 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 msgr2=0x7febe0140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 0x7febe0140b30 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7febd800cc30 tx=0x7febd800a5c0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 shutdown_connections 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806bf20 0x7febb806e3e0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe015e1e0 0x7febe0140b30 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe0143ae0 0x7febe01405f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.896+0000 7febe5a00640 1 --2- 172.21.15.53:0/153212987 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe00cf9a0 0x7febe01f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.897+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 >> 172.21.15.53:0/153212987 conn(0x7febe006a800 msgr2=0x7febe00b0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.897+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 shutdown_connections 2024-01-16T15:13:27.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:27.897+0000 7febe5a00640 1 -- 172.21.15.53:0/153212987 wait complete. 2024-01-16T15:13:27.895 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:13:27.895 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:13:28.008 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:13:28.125 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:13:28.126 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:13:28.126 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:28.187 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:13:28.187 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:13:28.187 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:28.243 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:13:28.243 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:13:28.243 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:28.305 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:13:28.305 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:13:28.305 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:28.364 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:13:28.364 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:13:28.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:13:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.614+0000 7f6626200640 1 Processor -- start 2024-01-16T15:13:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.614+0000 7f6626200640 1 -- start start 2024-01-16T15:13:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.615+0000 7f6626200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66200ad820 0x7f6620071160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.615+0000 7f6626200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200716a0 0x7f66200b38d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:28.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.615+0000 7f6626200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662015a500 con 0x7f662015a120 2024-01-16T15:13:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.616+0000 7f6626200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6620071b30 con 0x7f66200ad820 2024-01-16T15:13:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.616+0000 7f6626200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f662006f020 con 0x7f66200716a0 2024-01-16T15:13:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.616+0000 7f661fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66200ad820 0x7f6620071160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.616+0000 7f6625800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200716a0 0x7f66200b38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:28.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.617+0000 7f6624e00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f662015a120 0x7f662006e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:43618/0 (socket says 172.21.15.53:43618) 2024-01-16T15:13:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.617+0000 7f6624e00640 1 -- 172.21.15.53:0/4029738789 learned_addr learned my addr 172.21.15.53:0/4029738789 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.617+0000 7f661f400640 1 -- 172.21.15.53:0/4029738789 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975774396 0 0) 0x7f6610001110 con 0x7f662015a120 2024-01-16T15:13:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.617+0000 7f661f400640 1 -- 172.21.15.53:0/4029738789 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6600003520 con 0x7f662015a120 2024-01-16T15:13:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.618+0000 7f661fe00640 1 -- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200716a0 msgr2=0x7f66200b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.618+0000 7f661fe00640 1 --2- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200716a0 0x7f66200b38d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.618+0000 7f661fe00640 1 -- 172.21.15.53:0/4029738789 >> v1:172.21.15.146:6789/0 conn(0x7f662015a120 legacy=0x7f662006e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.618+0000 7f661fe00640 1 -- 172.21.15.53:0/4029738789 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66200adc80 con 0x7f66200ad820 2024-01-16T15:13:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.618+0000 7f6625800640 1 --2- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200716a0 0x7f66200b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.619+0000 7f661fe00640 1 --2- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66200ad820 0x7f6620071160 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f66080097a0 tx=0x7f660802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e81200a8cb8e2a63 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:28.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.620+0000 7f661f400640 1 -- 172.21.15.53:0/4029738789 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660802d040 con 0x7f66200ad820 2024-01-16T15:13:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.620+0000 7f661f400640 1 -- 172.21.15.53:0/4029738789 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6608031810 con 0x7f66200ad820 2024-01-16T15:13:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.620+0000 7f661f400640 1 -- 172.21.15.53:0/4029738789 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6608031ba0 con 0x7f66200ad820 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 -- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66200ad820 msgr2=0x7f6620071160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 --2- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66200ad820 0x7f6620071160 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f66080097a0 tx=0x7f660802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 -- 172.21.15.53:0/4029738789 shutdown_connections 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 --2- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200716a0 0x7f66200b38d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 --2- 172.21.15.53:0/4029738789 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66200ad820 0x7f6620071160 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 -- 172.21.15.53:0/4029738789 >> 172.21.15.53:0/4029738789 conn(0x7f66200b19c0 msgr2=0x7f66200b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 -- 172.21.15.53:0/4029738789 shutdown_connections 2024-01-16T15:13:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.621+0000 7f6626200640 1 -- 172.21.15.53:0/4029738789 wait complete. 2024-01-16T15:13:28.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.624+0000 7f6626200640 1 Processor -- start 2024-01-16T15:13:28.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.624+0000 7f6626200640 1 -- start start 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6626200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 0x7f66201e3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6626200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200ad820 0x7f66201426a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6626200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662015a120 0x7f66200b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6626200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66201f3b30 con 0x7f66200716a0 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6626200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66201f3f50 con 0x7f662015a120 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6626200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66201f3190 con 0x7f66200ad820 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.625+0000 7f6624e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 0x7f66201e3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 0x7f66201e3bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41418/0 (socket says 172.21.15.53:41418) 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 -- 172.21.15.53:0/1626401148 learned_addr learned my addr 172.21.15.53:0/1626401148 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:28.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f661fe00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200ad820 0x7f66201426a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6625800640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662015a120 0x7f66200b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 -- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200ad820 msgr2=0x7f66201426a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200ad820 0x7f66201426a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 -- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662015a120 msgr2=0x7f66200b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662015a120 0x7f66200b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 -- 172.21.15.53:0/1626401148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6618009a70 con 0x7f66200716a0 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6625800640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662015a120 0x7f66200b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f661fe00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200ad820 0x7f66201426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:28.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.626+0000 7f6624e00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 0x7f66201e3bd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f661000c700 tx=0x7f661000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:28.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.627+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610005ab0 con 0x7f66200716a0 2024-01-16T15:13:28.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.627+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6610005c10 con 0x7f66200716a0 2024-01-16T15:13:28.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.627+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610001d50 con 0x7f66200716a0 2024-01-16T15:13:28.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.627+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6608009400 con 0x7f66200716a0 2024-01-16T15:13:28.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.627+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66201f3750 con 0x7f66200716a0 2024-01-16T15:13:28.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.628+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6620067b60 con 0x7f66200716a0 2024-01-16T15:13:28.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.629+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6610007070 con 0x7f66200716a0 2024-01-16T15:13:28.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.630+0000 7f661cc00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65f806bf20 0x7f65f806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:28.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.630+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6610087c60 con 0x7f66200716a0 2024-01-16T15:13:28.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.631+0000 7f661fe00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65f806bf20 0x7f65f806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:28.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.632+0000 7f661fe00640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65f806bf20 0x7f65f806e3e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6608036000 tx=0x7f660803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:28.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.635+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66100542a0 con 0x7f66200716a0 2024-01-16T15:13:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.874+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f662015a500 con 0x7f66200716a0 2024-01-16T15:13:28.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.875+0000 7f661cc00640 1 -- 172.21.15.53:0/1626401148 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f662015a500 con 0x7f66200716a0 2024-01-16T15:13:28.855 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:28.855 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":48,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6041,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/1311204867","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1311204867},{"type":"v1","addr":"172.21.15.146:6836","nonce":1311204867}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":48},{"gid":6082,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/1106083775","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1106083775},{"type":"v1","addr":"172.21.15.53:6834","nonce":1106083775}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":6088,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/2940561403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":2940561403},{"type":"v1","addr":"172.21.15.146:6837","nonce":2940561403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":40},{"gid":7029,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3081217400","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3081217400},{"type":"v1","addr":"172.21.15.53:6835","nonce":3081217400}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":39}],"filesystems":[]} 2024-01-16T15:13:28.855 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 48 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.877+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65f806bf20 msgr2=0x7f65f806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.877+0000 7f6626200640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65f806bf20 0x7f65f806e3e0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f6608036000 tx=0x7f660803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.877+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 msgr2=0x7f66201e3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.877+0000 7f6626200640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 0x7f66201e3bd0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f661000c700 tx=0x7f661000cbc0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 shutdown_connections 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f65f806bf20 0x7f65f806e3e0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f662015a120 0x7f66200b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66200ad820 0x7f66201426a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 --2- 172.21.15.53:0/1626401148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66200716a0 0x7f66201e3bd0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 >> 172.21.15.53:0/1626401148 conn(0x7f66200b19c0 msgr2=0x7f66200adf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:28.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 shutdown_connections 2024-01-16T15:13:28.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:28.878+0000 7f6626200640 1 -- 172.21.15.53:0/1626401148 wait complete. 2024-01-16T15:13:28.878 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:13:28.878 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:13:28.878 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:13:28.878 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:13:28.879 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:13:28.879 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:13:28.879 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:13:28.879 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:13:28.879 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:13:28.879 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:13:28.879 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:13:28.879 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:13:28.880 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:13:28.880 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:13:28.880 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:13:28.880 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:13:28.897 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:13:28.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:13:28.897 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:13:28.898 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:13:28.900 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:13:28.901 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:13:28.902 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:13:28.903 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:13:28.945 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:13:28.945 DEBUG:teuthology.parallel:result is None 2024-01-16T15:13:28.946 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:13:28.947 DEBUG:teuthology.parallel:result is None 2024-01-16T15:13:28.952 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:13:28.952 DEBUG:teuthology.parallel:result is None 2024-01-16T15:13:28.953 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:13:28.954 DEBUG:teuthology.parallel:result is None 2024-01-16T15:13:28.954 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:13:29.106 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:13:29.107 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:13:29.109 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:13:29.112 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:13:29.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.162+0000 7f1af3200640 1 Processor -- start 2024-01-16T15:13:29.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.162+0000 7f1af3200640 1 -- start start 2024-01-16T15:13:29.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1aec13df80 0x7f1aec15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1aec13e360 0x7f1aec0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec06c470 con 0x7f1aec13e360 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec06c750 con 0x7f1aec13df80 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec06ca30 con 0x7f1aec15f9c0 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1aec13e360 0x7f1aec0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1aec13e360 0x7f1aec0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41478/0 (socket says 172.21.15.53:41478) 2024-01-16T15:13:29.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4013704249 learned_addr learned my addr 172.21.15.53:0/4013704249 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1af1200640 1 --2- 172.21.15.53:0/4013704249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1aebe00640 1 --2- 172.21.15.53:0/4013704249 >> v2:172.21.15.146:3300/0 conn(0x7f1aec13df80 0x7f1aec15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4013704249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 msgr2=0x7f1aec0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 --2- 172.21.15.53:0/4013704249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4013704249 >> v2:172.21.15.146:3300/0 conn(0x7f1aec13df80 msgr2=0x7f1aec15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 --2- 172.21.15.53:0/4013704249 >> v2:172.21.15.146:3300/0 conn(0x7f1aec13df80 0x7f1aec15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.164+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4013704249 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec06cd10 con 0x7f1aec13e360 2024-01-16T15:13:29.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.166+0000 7f1ae3e00640 1 --2- 172.21.15.53:0/4013704249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0cf290 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1aec072770 tx=0x7f1ad402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98768b22bb79890d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:29.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.166+0000 7f1aeb400640 1 -- 172.21.15.53:0/4013704249 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4031870 con 0x7f1aec13e360 2024-01-16T15:13:29.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.166+0000 7f1aeb400640 1 -- 172.21.15.53:0/4013704249 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad40319d0 con 0x7f1aec13e360 2024-01-16T15:13:29.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.166+0000 7f1aeb400640 1 -- 172.21.15.53:0/4013704249 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4031d60 con 0x7f1aec13e360 2024-01-16T15:13:29.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.166+0000 7f1af3200640 1 -- 172.21.15.53:0/4013704249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 msgr2=0x7f1aec0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.166+0000 7f1af3200640 1 --2- 172.21.15.53:0/4013704249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0cf290 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f1aec072770 tx=0x7f1ad402b910 comp rx=0 tx=0).stop 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 -- 172.21.15.53:0/4013704249 shutdown_connections 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 --2- 172.21.15.53:0/4013704249 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 --2- 172.21.15.53:0/4013704249 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0cf290 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 --2- 172.21.15.53:0/4013704249 >> v2:172.21.15.146:3300/0 conn(0x7f1aec13df80 0x7f1aec15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 -- 172.21.15.53:0/4013704249 >> 172.21.15.53:0/4013704249 conn(0x7f1aec06a800 msgr2=0x7f1aec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 -- 172.21.15.53:0/4013704249 shutdown_connections 2024-01-16T15:13:29.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.167+0000 7f1af3200640 1 -- 172.21.15.53:0/4013704249 wait complete. 2024-01-16T15:13:29.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.168+0000 7f1af3200640 1 Processor -- start 2024-01-16T15:13:29.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.169+0000 7f1af3200640 1 -- start start 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.169+0000 7f1af3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec13df80 0x7f1aec0aa140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1af3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0ac5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1af3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec13d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1af3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0aa680 con 0x7f1aec13e360 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1af3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0acb10 con 0x7f1aec13df80 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1af3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0ab150 con 0x7f1aec15f9c0 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0ac5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0ac5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41482/0 (socket says 172.21.15.53:41482) 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4138499572 learned_addr learned my addr 172.21.15.53:0/4138499572 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1af1200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec13d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1aebe00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec13df80 0x7f1aec0aa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 msgr2=0x7f1aec13d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec13d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec13df80 msgr2=0x7f1aec0aa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec13df80 0x7f1aec0aa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.170+0000 7f1ae3e00640 1 -- 172.21.15.53:0/4138499572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad40094d0 con 0x7f1aec13e360 2024-01-16T15:13:29.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.171+0000 7f1ae3e00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0ac5d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4000c00 tx=0x7f1ad40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:29.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.171+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad403aca0 con 0x7f1aec13e360 2024-01-16T15:13:29.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.171+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ad403ae00 con 0x7f1aec13e360 2024-01-16T15:13:29.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.171+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad4044630 con 0x7f1aec13e360 2024-01-16T15:13:29.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.171+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec0aa8b0 con 0x7f1aec13e360 2024-01-16T15:13:29.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.171+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec0aae70 con 0x7f1aec13e360 2024-01-16T15:13:29.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.172+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec0675d0 con 0x7f1aec13e360 2024-01-16T15:13:29.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.173+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f1ad4044790 con 0x7f1aec13e360 2024-01-16T15:13:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.175+0000 7f1ae8c00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac406bf20 0x7f1ac406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.175+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1ad402d040 con 0x7f1aec13e360 2024-01-16T15:13:29.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.175+0000 7f1aebe00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac406bf20 0x7f1ac406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.176+0000 7f1aebe00640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac406bf20 0x7f1ac406e3e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1adc006fd0 tx=0x7f1adc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:29.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.179+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1ad4077480 con 0x7f1aec13e360 2024-01-16T15:13:29.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.389+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1aec0b0950 con 0x7f1aec13e360 2024-01-16T15:13:29.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.456+0000 7f1ae8c00640 1 -- 172.21.15.53:0/4138499572 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v80) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1ad407aaa0 con 0x7f1aec13e360 2024-01-16T15:13:29.437 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:13:29.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac406bf20 msgr2=0x7f1ac406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac406bf20 0x7f1ac406e3e0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1adc006fd0 tx=0x7f1adc008040 comp rx=0 tx=0).stop 2024-01-16T15:13:29.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 msgr2=0x7f1aec0ac5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0ac5d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f1ad4000c00 tx=0x7f1ad40046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 shutdown_connections 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1ac406bf20 0x7f1ac406e3e0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1aec15f9c0 0x7f1aec13d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1aec13e360 0x7f1aec0ac5d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 --2- 172.21.15.53:0/4138499572 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1aec13df80 0x7f1aec0aa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.460+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 >> 172.21.15.53:0/4138499572 conn(0x7f1aec06a800 msgr2=0x7f1aec0af460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.461+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 shutdown_connections 2024-01-16T15:13:29.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.461+0000 7f1af3200640 1 -- 172.21.15.53:0/4138499572 wait complete. 2024-01-16T15:13:29.460 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:13:29.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.682+0000 7f8301e00640 1 Processor -- start 2024-01-16T15:13:29.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.682+0000 7f8301e00640 1 -- start start 2024-01-16T15:13:29.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f8301e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f82fc146850 0x7f82fc0715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f8301e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f82fc071b10 0x7f82fc0ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f8301e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc0adb00 0x7f82fc0700f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f8301e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc071fa0 con 0x7f82fc071b10 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f8301e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc0b3490 con 0x7f82fc146850 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f8301e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc0b3770 con 0x7f82fc0adb00 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f82f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f82fc071b10 0x7f82fc0ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f82f3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f82fc071b10 0x7f82fc0ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41498/0 (socket says 172.21.15.53:41498) 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.684+0000 7f82f3e00640 1 -- 172.21.15.53:0/3459146177 learned_addr learned my addr 172.21.15.53:0/3459146177 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.685+0000 7f8301400640 1 --2- 172.21.15.53:0/3459146177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc0adb00 0x7f82fc0700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.685+0000 7f82f3e00640 1 -- 172.21.15.53:0/3459146177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc0adb00 msgr2=0x7f82fc0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.685+0000 7f82f3e00640 1 --2- 172.21.15.53:0/3459146177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc0adb00 0x7f82fc0700f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.685+0000 7f82f3e00640 1 -- 172.21.15.53:0/3459146177 >> v2:172.21.15.146:3300/0 conn(0x7f82fc146850 msgr2=0x7f82fc0715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.685+0000 7f82f3e00640 1 --2- 172.21.15.53:0/3459146177 >> v2:172.21.15.146:3300/0 conn(0x7f82fc146850 0x7f82fc0715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.685+0000 7f82f3e00640 1 -- 172.21.15.53:0/3459146177 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc0b3a50 con 0x7f82fc071b10 2024-01-16T15:13:29.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.686+0000 7f82f3e00640 1 --2- 172.21.15.53:0/3459146177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc071b10 0x7f82fc0ad5c0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0ad010 tx=0x7f82e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8320d4bdb45cc953 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:29.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.686+0000 7f82fbe00640 1 -- 172.21.15.53:0/3459146177 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4031870 con 0x7f82fc071b10 2024-01-16T15:13:29.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.686+0000 7f82fbe00640 1 -- 172.21.15.53:0/3459146177 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e40319d0 con 0x7f82fc071b10 2024-01-16T15:13:29.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.686+0000 7f82fbe00640 1 -- 172.21.15.53:0/3459146177 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e4031d60 con 0x7f82fc071b10 2024-01-16T15:13:29.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 -- 172.21.15.53:0/3459146177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc071b10 msgr2=0x7f82fc0ad5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 --2- 172.21.15.53:0/3459146177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc071b10 0x7f82fc0ad5c0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7f82fc0ad010 tx=0x7f82e402b910 comp rx=0 tx=0).stop 2024-01-16T15:13:29.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 -- 172.21.15.53:0/3459146177 shutdown_connections 2024-01-16T15:13:29.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 --2- 172.21.15.53:0/3459146177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc0adb00 0x7f82fc0700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 --2- 172.21.15.53:0/3459146177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc071b10 0x7f82fc0ad5c0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 --2- 172.21.15.53:0/3459146177 >> v2:172.21.15.146:3300/0 conn(0x7f82fc146850 0x7f82fc0715d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 -- 172.21.15.53:0/3459146177 >> 172.21.15.53:0/3459146177 conn(0x7f82fc06a3e0 msgr2=0x7f82fc06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:29.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 -- 172.21.15.53:0/3459146177 shutdown_connections 2024-01-16T15:13:29.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.687+0000 7f8301e00640 1 -- 172.21.15.53:0/3459146177 wait complete. 2024-01-16T15:13:29.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.689+0000 7f8301e00640 1 Processor -- start 2024-01-16T15:13:29.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.689+0000 7f8301e00640 1 -- start start 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.690+0000 7f8301e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc071b10 0x7f82fc1438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.690+0000 7f8301e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82fc0adb00 0x7f82fc1f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.690+0000 7f8301e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 0x7f82fc1f6a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.690+0000 7f8301e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc144f90 con 0x7f82fc146850 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.690+0000 7f8301e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc144310 con 0x7f82fc0adb00 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.690+0000 7f8301e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc1445f0 con 0x7f82fc071b10 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 0x7f82fc1f6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 0x7f82fc1f6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41512/0 (socket says 172.21.15.53:41512) 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 -- 172.21.15.53:0/553455090 learned_addr learned my addr 172.21.15.53:0/553455090 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8300a00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc071b10 0x7f82fc1438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f82f3e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82fc0adb00 0x7f82fc1f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 -- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc071b10 msgr2=0x7f82fc1438c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc071b10 0x7f82fc1438c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 -- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82fc0adb00 msgr2=0x7f82fc1f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82fc0adb00 0x7f82fc1f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.691+0000 7f8301400640 1 -- 172.21.15.53:0/553455090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e40094d0 con 0x7f82fc146850 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.692+0000 7f8301400640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 0x7f82fc1f6a80 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f82f4007e60 tx=0x7f82f400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.692+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4014420 con 0x7f82fc146850 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.692+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82f40145a0 con 0x7f82fc146850 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.692+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f4014950 con 0x7f82fc146850 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.692+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc1448d0 con 0x7f82fc146850 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.692+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc13d840 con 0x7f82fc146850 2024-01-16T15:13:29.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.693+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc13d9d0 con 0x7f82fc146850 2024-01-16T15:13:29.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.694+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f82f4014ab0 con 0x7f82fc146850 2024-01-16T15:13:29.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.695+0000 7f82f9600640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f82d406bf20 0x7f82d406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:29.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.696+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f82f4088f30 con 0x7f82fc146850 2024-01-16T15:13:29.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.696+0000 7f8300a00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f82d406bf20 0x7f82d406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:29.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.697+0000 7f8300a00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f82d406bf20 0x7f82d406e3e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009890 tx=0x7f82ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:29.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.700+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f82f40557f0 con 0x7f82fc146850 2024-01-16T15:13:29.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.964+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f82fc0cebc0 con 0x7f82fc146850 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.965+0000 7f82f9600640 1 -- 172.21.15.53:0/553455090 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f82f4058e10 con 0x7f82fc146850 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:13:29.945 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:13:29.946 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:13:29.947 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.948 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:13:29.949 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.950 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.951 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.952 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow r", 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:13:29.954 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:13:29.955 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:13:29.956 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:13:29.957 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:29.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f82d406bf20 msgr2=0x7f82d406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f82d406bf20 0x7f82d406e3e0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009890 tx=0x7f82ec008040 comp rx=0 tx=0).stop 2024-01-16T15:13:29.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 msgr2=0x7f82fc1f6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 0x7f82fc1f6a80 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f82f4007e60 tx=0x7f82f400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 shutdown_connections 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f82d406bf20 0x7f82d406e3e0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f82fc146850 0x7f82fc1f6a80 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f82fc0adb00 0x7f82fc1f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 --2- 172.21.15.53:0/553455090 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f82fc071b10 0x7f82fc1438c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 >> 172.21.15.53:0/553455090 conn(0x7f82fc06a3e0 msgr2=0x7f82fc0b03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 shutdown_connections 2024-01-16T15:13:29.959 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:29.968+0000 7f8301e00640 1 -- 172.21.15.53:0/553455090 wait complete. 2024-01-16T15:13:29.966 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:30.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.193+0000 7f4c44600640 1 Processor -- start 2024-01-16T15:13:30.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.193+0000 7f4c44600640 1 -- start start 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c44600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4c3c13df80 0x7f4c3c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c44600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4c3c0cf9a0 0x7f4c3c0cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c44600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3c06c470 con 0x7f4c3c13e360 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c44600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3c06c750 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c44600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c3c06ca30 con 0x7f4c3c13df80 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c42600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4c3c0cf9a0 0x7f4c3c0cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c41200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4c3c13e360 0x7f4c3c0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39342/0 (socket says 172.21.15.53:39342) 2024-01-16T15:13:30.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.195+0000 7f4c41200640 1 -- 172.21.15.53:0/112197988 learned_addr learned my addr 172.21.15.53:0/112197988 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c42600640 1 -- 172.21.15.53:0/112197988 >> v2:172.21.15.146:3300/0 conn(0x7f4c3c13df80 msgr2=0x7f4c3c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c41c00640 1 --2- 172.21.15.53:0/112197988 >> v2:172.21.15.146:3300/0 conn(0x7f4c3c13df80 0x7f4c3c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c42600640 1 --2- 172.21.15.53:0/112197988 >> v2:172.21.15.146:3300/0 conn(0x7f4c3c13df80 0x7f4c3c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c42600640 1 -- 172.21.15.53:0/112197988 >> v1:172.21.15.53:6789/0 conn(0x7f4c3c13e360 legacy=0x7f4c3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c42600640 1 -- 172.21.15.53:0/112197988 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c3c06cd10 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c3be00640 1 -- 172.21.15.53:0/112197988 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719379715 0 0) 0x7f4c3c06c470 con 0x7f4c3c13e360 2024-01-16T15:13:30.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.196+0000 7f4c41c00640 1 --2- 172.21.15.53:0/112197988 >> v2:172.21.15.146:3300/0 conn(0x7f4c3c13df80 0x7f4c3c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.197+0000 7f4c42600640 1 --2- 172.21.15.53:0/112197988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c0cd840 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c072770 tx=0x7f4c3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a5485fbdd3ddf6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.197+0000 7f4c3be00640 1 -- 172.21.15.53:0/112197988 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34031830 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.197+0000 7f4c3be00640 1 -- 172.21.15.53:0/112197988 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c34031990 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.197+0000 7f4c3be00640 1 -- 172.21.15.53:0/112197988 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c34031d20 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 -- 172.21.15.53:0/112197988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 msgr2=0x7f4c3c0cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 --2- 172.21.15.53:0/112197988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c0cd840 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c072770 tx=0x7f4c3402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 -- 172.21.15.53:0/112197988 shutdown_connections 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 --2- 172.21.15.53:0/112197988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c0cd840 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 --2- 172.21.15.53:0/112197988 >> v2:172.21.15.146:3300/0 conn(0x7f4c3c13df80 0x7f4c3c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 -- 172.21.15.53:0/112197988 >> 172.21.15.53:0/112197988 conn(0x7f4c3c06a800 msgr2=0x7f4c3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 -- 172.21.15.53:0/112197988 shutdown_connections 2024-01-16T15:13:30.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.198+0000 7f4c44600640 1 -- 172.21.15.53:0/112197988 wait complete. 2024-01-16T15:13:30.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.200+0000 7f4c44600640 1 Processor -- start 2024-01-16T15:13:30.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.200+0000 7f4c44600640 1 -- start start 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c44600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c14f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c44600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c3c13df80 0x7f4c3c14df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c44600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c3c13e360 0x7f4c3c071310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c44600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c14e8d0 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c44600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c14ebb0 con 0x7f4c3c13df80 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c44600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3c14ee90 con 0x7f4c3c13e360 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c41c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c41c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c14f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41542/0 (socket says 172.21.15.53:41542) 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.201+0000 7f4c41c00640 1 -- 172.21.15.53:0/727444077 learned_addr learned my addr 172.21.15.53:0/727444077 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41c00640 1 -- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c3c13e360 msgr2=0x7f4c3c071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41c00640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c3c13e360 0x7f4c3c071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41c00640 1 -- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c3c13df80 msgr2=0x7f4c3c14df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41200640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c3c13df80 0x7f4c3c14df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41c00640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c3c13df80 0x7f4c3c14df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41c00640 1 -- 172.21.15.53:0/727444077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c34009400 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41200640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c3c13df80 0x7f4c3c14df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:30.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c41c00640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c14f970 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00cf60 tx=0x7f4c2c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00d840 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c00d9a0 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00dd30 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.202+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3c14f170 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.203+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c3c14e450 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.204+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3c14e670 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.205+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f4c2c015440 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.205+0000 7f4c39600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c1406bf20 0x7f4c1406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.206+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f4c2c0891f0 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.206+0000 7f4c41200640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c1406bf20 0x7f4c1406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.207+0000 7f4c41200640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c1406bf20 0x7f4c1406e3e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4c24000f60 tx=0x7f4c24009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:30.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.210+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4c2c0558e0 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.408+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c3c14f690 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.409+0000 7f4c39600640 1 -- 172.21.15.53:0/727444077 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+12865 (secure 0 0 0) 0x7f4c2c01a070 con 0x7f4c3c0cf9a0 2024-01-16T15:13:30.390 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:30.390 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":82,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:29.471331+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:30.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c1406bf20 msgr2=0x7f4c1406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c1406bf20 0x7f4c1406e3e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f4c24000f60 tx=0x7f4c24009040 comp rx=0 tx=0).stop 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 msgr2=0x7f4c3c14f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c14f970 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00cf60 tx=0x7f4c2c00cf90 comp rx=0 tx=0).stop 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 shutdown_connections 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4c1406bf20 0x7f4c1406e3e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4c3c13e360 0x7f4c3c071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4c3c13df80 0x7f4c3c14df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 --2- 172.21.15.53:0/727444077 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4c3c0cf9a0 0x7f4c3c14f970 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.412+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 >> 172.21.15.53:0/727444077 conn(0x7f4c3c06a800 msgr2=0x7f4c3c15e470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.413+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 shutdown_connections 2024-01-16T15:13:30.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.413+0000 7f4c44600640 1 -- 172.21.15.53:0/727444077 wait complete. 2024-01-16T15:13:30.412 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:13:30.412 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:13:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.652+0000 7f6406600640 1 Processor -- start 2024-01-16T15:13:30.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.653+0000 7f6406600640 1 -- start start 2024-01-16T15:13:30.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.654+0000 7f6406600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f64000ccb50 0x7f6400070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.654+0000 7f6406600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f640015bf10 0x7f6400143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.654+0000 7f6406600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f640015f4e0 con 0x7f64000adf20 2024-01-16T15:13:30.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.654+0000 7f6406600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640015f970 con 0x7f640015bf10 2024-01-16T15:13:30.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.654+0000 7f6406600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6400071130 con 0x7f64000ccb50 2024-01-16T15:13:30.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.655+0000 7f6405c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f640015bf10 0x7f6400143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.655+0000 7f6405c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f640015bf10 0x7f6400143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41552/0 (socket says 172.21.15.53:41552) 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.655+0000 7f6405c00640 1 -- 172.21.15.53:0/653543349 learned_addr learned my addr 172.21.15.53:0/653543349 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.655+0000 7f63ffe00640 1 --2- 172.21.15.53:0/653543349 >> v2:172.21.15.146:3300/0 conn(0x7f64000ccb50 0x7f6400070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.656+0000 7f6405c00640 1 -- 172.21.15.53:0/653543349 >> v2:172.21.15.146:3300/0 conn(0x7f64000ccb50 msgr2=0x7f6400070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.656+0000 7f6405c00640 1 --2- 172.21.15.53:0/653543349 >> v2:172.21.15.146:3300/0 conn(0x7f64000ccb50 0x7f6400070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.656+0000 7f6405c00640 1 -- 172.21.15.53:0/653543349 >> v1:172.21.15.146:6789/0 conn(0x7f64000adf20 legacy=0x7f64000cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:30.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.656+0000 7f6405c00640 1 -- 172.21.15.53:0/653543349 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640015ebb0 con 0x7f640015bf10 2024-01-16T15:13:30.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.657+0000 7f6405c00640 1 --2- 172.21.15.53:0/653543349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f640015bf10 0x7f6400143d30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f640015e9e0 tx=0x7f63f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffdd14d21798b5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:30.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f63ff400640 1 -- 172.21.15.53:0/653543349 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8031870 con 0x7f640015bf10 2024-01-16T15:13:30.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f63ff400640 1 -- 172.21.15.53:0/653543349 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f80319d0 con 0x7f640015bf10 2024-01-16T15:13:30.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f63ff400640 1 -- 172.21.15.53:0/653543349 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f8031d60 con 0x7f640015bf10 2024-01-16T15:13:30.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 -- 172.21.15.53:0/653543349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f640015bf10 msgr2=0x7f6400143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 --2- 172.21.15.53:0/653543349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f640015bf10 0x7f6400143d30 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f640015e9e0 tx=0x7f63f802b910 comp rx=0 tx=0).stop 2024-01-16T15:13:30.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 -- 172.21.15.53:0/653543349 shutdown_connections 2024-01-16T15:13:30.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 --2- 172.21.15.53:0/653543349 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f640015bf10 0x7f6400143d30 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 --2- 172.21.15.53:0/653543349 >> v2:172.21.15.146:3300/0 conn(0x7f64000ccb50 0x7f6400070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 -- 172.21.15.53:0/653543349 >> 172.21.15.53:0/653543349 conn(0x7f640006a7d0 msgr2=0x7f640006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:30.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 -- 172.21.15.53:0/653543349 shutdown_connections 2024-01-16T15:13:30.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.658+0000 7f6406600640 1 -- 172.21.15.53:0/653543349 wait complete. 2024-01-16T15:13:30.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.660+0000 7f6406600640 1 Processor -- start 2024-01-16T15:13:30.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.660+0000 7f6406600640 1 -- start start 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.661+0000 7f6406600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 0x7f640015d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.661+0000 7f6406600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64000ccb50 0x7f64000ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.661+0000 7f6406600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f640015bf10 0x7f6400071a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.661+0000 7f6406600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64001f37b0 con 0x7f64000adf20 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.661+0000 7f6406600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64001f2b30 con 0x7f640015bf10 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.661+0000 7f6406600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64001f2e10 con 0x7f64000ccb50 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 0x7f640015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 0x7f640015d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41568/0 (socket says 172.21.15.53:41568) 2024-01-16T15:13:30.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 -- 172.21.15.53:0/4223449344 learned_addr learned my addr 172.21.15.53:0/4223449344 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405c00640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f640015bf10 0x7f6400071a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 -- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64000ccb50 msgr2=0x7f64000ca6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64000ccb50 0x7f64000ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 -- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f640015bf10 msgr2=0x7f6400071a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f640015bf10 0x7f6400071a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 -- 172.21.15.53:0/4223449344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63f80094d0 con 0x7f64000adf20 2024-01-16T15:13:30.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.662+0000 7f6405200640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 0x7f640015d330 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f63f000c870 tx=0x7f63f000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:30.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.663+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f0003b70 con 0x7f64000adf20 2024-01-16T15:13:30.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.663+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63f0003cf0 con 0x7f64000adf20 2024-01-16T15:13:30.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.663+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63f000aa90 con 0x7f64000adf20 2024-01-16T15:13:30.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.663+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64001f30f0 con 0x7f64000adf20 2024-01-16T15:13:30.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.663+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64001f6ca0 con 0x7f64000adf20 2024-01-16T15:13:30.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.664+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64000c3d00 con 0x7f64000adf20 2024-01-16T15:13:30.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.665+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f63f000abf0 con 0x7f64000adf20 2024-01-16T15:13:30.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.666+0000 7f63fcc00640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63d806bf20 0x7f63d806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:30.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.666+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f63f00884b0 con 0x7f64000adf20 2024-01-16T15:13:30.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.666+0000 7f63ffe00640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63d806bf20 0x7f63d806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:30.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.667+0000 7f63ffe00640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63d806bf20 0x7f63d806e3e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6400072600 tx=0x7f63e8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:30.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.670+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63f0091080 con 0x7f64000adf20 2024-01-16T15:13:30.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.865+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6400067450 con 0x7f64000adf20 2024-01-16T15:13:30.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.866+0000 7f63fcc00640 1 -- 172.21.15.53:0/4223449344 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f63f0003e60 con 0x7f64000adf20 2024-01-16T15:13:30.846 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:13:30.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.869+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63d806bf20 msgr2=0x7f63d806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.869+0000 7f6406600640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63d806bf20 0x7f63d806e3e0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6400072600 tx=0x7f63e8008040 comp rx=0 tx=0).stop 2024-01-16T15:13:30.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.869+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 msgr2=0x7f640015d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:30.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.869+0000 7f6406600640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 0x7f640015d330 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f63f000c870 tx=0x7f63f000cd30 comp rx=0 tx=0).stop 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 shutdown_connections 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63d806bf20 0x7f63d806e3e0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f640015bf10 0x7f6400071a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f64000ccb50 0x7f64000ca6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 --2- 172.21.15.53:0/4223449344 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64000adf20 0x7f640015d330 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 >> 172.21.15.53:0/4223449344 conn(0x7f640006a7d0 msgr2=0x7f64000cc340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 shutdown_connections 2024-01-16T15:13:30.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:30.870+0000 7f6406600640 1 -- 172.21.15.53:0/4223449344 wait complete. 2024-01-16T15:13:30.867 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:13:31.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.102+0000 7f9767e00640 1 Processor -- start 2024-01-16T15:13:31.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.102+0000 7f9767e00640 1 -- start start 2024-01-16T15:13:31.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.103+0000 7f9767e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9768071410 0x7f9768146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9767e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f97680717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9767e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f976806cc20 con 0x7f9768071d30 2024-01-16T15:13:31.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9767e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f976806cec0 con 0x7f9768071410 2024-01-16T15:13:31.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9767e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976806d1a0 con 0x7f976806c7c0 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9766a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9768071410 0x7f9768146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9767400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9768071d30 0x7f97680b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39354/0 (socket says 172.21.15.53:39354) 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9767400640 1 -- 172.21.15.53:0/2913387036 learned_addr learned my addr 172.21.15.53:0/2913387036 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.104+0000 7f9766000640 1 --2- 172.21.15.53:0/2913387036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f97680717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.105+0000 7f9766a00640 1 -- 172.21.15.53:0/2913387036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 msgr2=0x7f97680717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.105+0000 7f9766a00640 1 --2- 172.21.15.53:0/2913387036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f97680717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.105+0000 7f9766a00640 1 -- 172.21.15.53:0/2913387036 >> v1:172.21.15.53:6789/0 conn(0x7f9768071d30 legacy=0x7f97680b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.105+0000 7f9766a00640 1 -- 172.21.15.53:0/2913387036 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976806d480 con 0x7f9768071410 2024-01-16T15:13:31.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.105+0000 7f9765600640 1 -- 172.21.15.53:0/2913387036 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135238851 0 0) 0x7f976806cc20 con 0x7f9768071d30 2024-01-16T15:13:31.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.106+0000 7f9766a00640 1 --2- 172.21.15.53:0/2913387036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768146770 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f97580097a0 tx=0x7f975802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c31c06d7be05f9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:31.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.106+0000 7f9765600640 1 -- 172.21.15.53:0/2913387036 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975802d040 con 0x7f9768071410 2024-01-16T15:13:31.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.106+0000 7f9765600640 1 -- 172.21.15.53:0/2913387036 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9758031810 con 0x7f9768071410 2024-01-16T15:13:31.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.106+0000 7f9765600640 1 -- 172.21.15.53:0/2913387036 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9758031ba0 con 0x7f9768071410 2024-01-16T15:13:31.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 -- 172.21.15.53:0/2913387036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 msgr2=0x7f9768146770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 --2- 172.21.15.53:0/2913387036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768146770 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f97580097a0 tx=0x7f975802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 -- 172.21.15.53:0/2913387036 shutdown_connections 2024-01-16T15:13:31.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 --2- 172.21.15.53:0/2913387036 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f97680717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 --2- 172.21.15.53:0/2913387036 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768146770 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 -- 172.21.15.53:0/2913387036 >> 172.21.15.53:0/2913387036 conn(0x7f976806a800 msgr2=0x7f976806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:31.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 -- 172.21.15.53:0/2913387036 shutdown_connections 2024-01-16T15:13:31.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.107+0000 7f9767e00640 1 -- 172.21.15.53:0/2913387036 wait complete. 2024-01-16T15:13:31.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.109+0000 7f9767e00640 1 Processor -- start 2024-01-16T15:13:31.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.109+0000 7f9767e00640 1 -- start start 2024-01-16T15:13:31.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9767e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f976815c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9767e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768140230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9767e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9768071d30 0x7f976813e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9767e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976813fda0 con 0x7f9768071410 2024-01-16T15:13:31.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9767e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f976813f190 con 0x7f9768071d30 2024-01-16T15:13:31.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9767e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976813f470 con 0x7f976806c7c0 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9766000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768140230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.110+0000 7f9766000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768140230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41590/0 (socket says 172.21.15.53:41590) 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 -- 172.21.15.53:0/3134380487 learned_addr learned my addr 172.21.15.53:0/3134380487 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766a00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f976815c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9767400640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9768071d30 0x7f976813e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 -- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 msgr2=0x7f976815c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f976815c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 -- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9768071d30 msgr2=0x7f976813e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9768071d30 0x7f976813e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 -- 172.21.15.53:0/3134380487 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9758009400 con 0x7f9768071410 2024-01-16T15:13:31.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.111+0000 7f9766000640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768140230 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f9750007ef0 tx=0x7f975000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:31.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.112+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750014620 con 0x7f9768071410 2024-01-16T15:13:31.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.112+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9750014780 con 0x7f9768071410 2024-01-16T15:13:31.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.112+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f976813f750 con 0x7f9768071410 2024-01-16T15:13:31.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.112+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9750014b30 con 0x7f9768071410 2024-01-16T15:13:31.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.112+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9768140770 con 0x7f9768071410 2024-01-16T15:13:31.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.113+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f976813ff50 con 0x7f9768071410 2024-01-16T15:13:31.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.114+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9750014c90 con 0x7f9768071410 2024-01-16T15:13:31.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.115+0000 7f975ea00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f974006bfa0 0x7f974006e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.115+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f9750088e90 con 0x7f9768071410 2024-01-16T15:13:31.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.116+0000 7f9766a00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f974006bfa0 0x7f974006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.117+0000 7f9766a00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f974006bfa0 0x7f974006e460 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9758036000 tx=0x7f975803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:31.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.119+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f976813ff50 con 0x7f9768071410 2024-01-16T15:13:31.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.313+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9768067450 con 0x7f9768071410 2024-01-16T15:13:31.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.314+0000 7f975ea00640 1 -- 172.21.15.53:0/3134380487 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9750055750 con 0x7f9768071410 2024-01-16T15:13:31.293 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:31.294 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:13:31.294 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:31.294 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:13:31.294 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:13:31.294 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:31.294 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f974006bfa0 msgr2=0x7f974006e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f974006bfa0 0x7f974006e460 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9758036000 tx=0x7f975803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 msgr2=0x7f9768140230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768140230 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f9750007ef0 tx=0x7f975000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 shutdown_connections 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f974006bfa0 0x7f974006e460 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9768071d30 0x7f976813e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9768071410 0x7f9768140230 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 --2- 172.21.15.53:0/3134380487 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f976806c7c0 0x7f976815c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 >> 172.21.15.53:0/3134380487 conn(0x7f976806a800 msgr2=0x7f976806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:31.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 shutdown_connections 2024-01-16T15:13:31.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.316+0000 7f9767e00640 1 -- 172.21.15.53:0/3134380487 wait complete. 2024-01-16T15:13:31.315 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:13:31.315 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:13:31.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.556+0000 7f22c7e00640 1 Processor -- start 2024-01-16T15:13:31.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.556+0000 7f22c7e00640 1 -- start start 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.557+0000 7f22c7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22c8160920 0x7f22c80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.557+0000 7f22c7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c80cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.557+0000 7f22c7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22c8161010 0x7f22c81614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.557+0000 7f22c7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c80d03b0 con 0x7f22c8160920 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22c80d0a10 con 0x7f22c8161010 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c80cf080 con 0x7f22c80cf9f0 2024-01-16T15:13:31.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22c8160920 0x7f22c80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22c8160920 0x7f22c80b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41598/0 (socket says 172.21.15.53:41598) 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 -- 172.21.15.53:0/1881883832 learned_addr learned my addr 172.21.15.53:0/1881883832 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 -- 172.21.15.53:0/1881883832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 msgr2=0x7f22c80cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 --2- 172.21.15.53:0/1881883832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c80cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 -- 172.21.15.53:0/1881883832 >> v2:172.21.15.146:3300/0 conn(0x7f22c8161010 msgr2=0x7f22c81614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 --2- 172.21.15.53:0/1881883832 >> v2:172.21.15.146:3300/0 conn(0x7f22c8161010 0x7f22c81614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.558+0000 7f22c6a00640 1 -- 172.21.15.53:0/1881883832 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c80c6be0 con 0x7f22c8160920 2024-01-16T15:13:31.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.559+0000 7f22c6a00640 1 --2- 172.21.15.53:0/1881883832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8160920 0x7f22c80b3f30 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f22c80c6930 tx=0x7f22b802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1188002b29666110 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:31.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.560+0000 7f22c5600640 1 -- 172.21.15.53:0/1881883832 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8031860 con 0x7f22c8160920 2024-01-16T15:13:31.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.560+0000 7f22c5600640 1 -- 172.21.15.53:0/1881883832 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b80319c0 con 0x7f22c8160920 2024-01-16T15:13:31.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.560+0000 7f22c5600640 1 -- 172.21.15.53:0/1881883832 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b8031d50 con 0x7f22c8160920 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.560+0000 7f22c7e00640 1 -- 172.21.15.53:0/1881883832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8160920 msgr2=0x7f22c80b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.560+0000 7f22c7e00640 1 --2- 172.21.15.53:0/1881883832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8160920 0x7f22c80b3f30 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f22c80c6930 tx=0x7f22b802b900 comp rx=0 tx=0).stop 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 -- 172.21.15.53:0/1881883832 shutdown_connections 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 --2- 172.21.15.53:0/1881883832 >> v2:172.21.15.146:3300/0 conn(0x7f22c8161010 0x7f22c81614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 --2- 172.21.15.53:0/1881883832 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c80cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 --2- 172.21.15.53:0/1881883832 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8160920 0x7f22c80b3f30 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 -- 172.21.15.53:0/1881883832 >> 172.21.15.53:0/1881883832 conn(0x7f22c80d3090 msgr2=0x7f22c80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 -- 172.21.15.53:0/1881883832 shutdown_connections 2024-01-16T15:13:31.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.561+0000 7f22c7e00640 1 -- 172.21.15.53:0/1881883832 wait complete. 2024-01-16T15:13:31.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.563+0000 7f22c7e00640 1 Processor -- start 2024-01-16T15:13:31.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.563+0000 7f22c7e00640 1 -- start start 2024-01-16T15:13:31.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.564+0000 7f22c7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c815a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22c8160920 0x7f22c815f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 0x7f22c80b6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c815ce60 con 0x7f22c8161010 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22c815c8a0 con 0x7f22c8160920 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22c815cb80 con 0x7f22c80cf9f0 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 0x7f22c80b6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 0x7f22c80b6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41606/0 (socket says 172.21.15.53:41606) 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7400640 1 -- 172.21.15.53:0/3604233155 learned_addr learned my addr 172.21.15.53:0/3604233155 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c6a00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c815a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c6000640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22c8160920 0x7f22c815f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.565+0000 7f22c7400640 1 -- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 msgr2=0x7f22c815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7400640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c815a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7400640 1 -- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22c8160920 msgr2=0x7f22c815f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7400640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22c8160920 0x7f22c815f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7400640 1 -- 172.21.15.53:0/3604233155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b80094d0 con 0x7f22c8161010 2024-01-16T15:13:31.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c6a00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:31.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7400640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 0x7f22c80b6560 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f22c000b970 tx=0x7f22c000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:31.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c000cb40 con 0x7f22c8161010 2024-01-16T15:13:31.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c000ccc0 con 0x7f22c8161010 2024-01-16T15:13:31.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c0014680 con 0x7f22c8161010 2024-01-16T15:13:31.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c80b5d60 con 0x7f22c8161010 2024-01-16T15:13:31.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.566+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c815a9b0 con 0x7f22c8161010 2024-01-16T15:13:31.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.568+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c80b6280 con 0x7f22c8161010 2024-01-16T15:13:31.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.569+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f22c001d3b0 con 0x7f22c8161010 2024-01-16T15:13:31.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.569+0000 7f22bea00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22a006bf20 0x7f22a006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:31.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.570+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f22c00887e0 con 0x7f22c8161010 2024-01-16T15:13:31.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.570+0000 7f22c6a00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22a006bf20 0x7f22a006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:31.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.571+0000 7f22c6a00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22a006bf20 0x7f22a006e3e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f22b8036000 tx=0x7f22b803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:31.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.574+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22c00bde90 con 0x7f22c8161010 2024-01-16T15:13:31.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:31.769+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f22c8067450 con 0x7f22c8161010 2024-01-16T15:13:32.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.481+0000 7f22bea00640 1 -- 172.21.15.53:0/3604233155 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7f22c0019070 con 0x7f22c8161010 2024-01-16T15:13:32.461 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.484+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22a006bf20 msgr2=0x7f22a006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.484+0000 7f22c7e00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22a006bf20 0x7f22a006e3e0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f22b8036000 tx=0x7f22b803c040 comp rx=0 tx=0).stop 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.484+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 msgr2=0x7f22c80b6560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.484+0000 7f22c7e00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 0x7f22c80b6560 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f22c000b970 tx=0x7f22c000be30 comp rx=0 tx=0).stop 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 shutdown_connections 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22a006bf20 0x7f22a006e3e0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22c8161010 0x7f22c80b6560 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22c8160920 0x7f22c815f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 --2- 172.21.15.53:0/3604233155 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22c80cf9f0 0x7f22c815a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 >> 172.21.15.53:0/3604233155 conn(0x7f22c80d3090 msgr2=0x7f22c80c94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:32.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 shutdown_connections 2024-01-16T15:13:32.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.485+0000 7f22c7e00640 1 -- 172.21.15.53:0/3604233155 wait complete. 2024-01-16T15:13:32.491 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:13:32.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.718+0000 7fbb27e00640 1 Processor -- start 2024-01-16T15:13:32.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.719+0000 7fbb27e00640 1 -- start start 2024-01-16T15:13:32.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.720+0000 7fbb27e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fbb28071f00 0x7fbb28146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.720+0000 7fbb27e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbb2806c7c0 0x7fbb28166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.720+0000 7fbb27e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb280b2e10 con 0x7fbb28071410 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.720+0000 7fbb27e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb2806cc10 con 0x7fbb2806c7c0 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.720+0000 7fbb27e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb2806cec0 con 0x7fbb28071f00 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb26a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbb28071410 0x7fbb280717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54986/0 (socket says 172.21.15.53:54986) 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb26a00640 1 -- 172.21.15.53:0/3256140433 learned_addr learned my addr 172.21.15.53:0/3256140433 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:32.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb27400640 1 --2- 172.21.15.53:0/3256140433 >> v2:172.21.15.53:3300/0 conn(0x7fbb2806c7c0 0x7fbb28166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb1fe00640 1 --2- 172.21.15.53:0/3256140433 >> v2:172.21.15.146:3300/0 conn(0x7fbb28071f00 0x7fbb28146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb26000640 1 -- 172.21.15.53:0/3256140433 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407163399 0 0) 0x7fbb280b2e10 con 0x7fbb28071410 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb26000640 1 -- 172.21.15.53:0/3256140433 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb08003520 con 0x7fbb28071410 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb27400640 1 -- 172.21.15.53:0/3256140433 >> v2:172.21.15.146:3300/0 conn(0x7fbb28071f00 msgr2=0x7fbb28146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb27400640 1 --2- 172.21.15.53:0/3256140433 >> v2:172.21.15.146:3300/0 conn(0x7fbb28071f00 0x7fbb28146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb27400640 1 -- 172.21.15.53:0/3256140433 >> v1:172.21.15.53:6789/0 conn(0x7fbb28071410 legacy=0x7fbb280717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:32.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.721+0000 7fbb27400640 1 -- 172.21.15.53:0/3256140433 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2806d1a0 con 0x7fbb2806c7c0 2024-01-16T15:13:32.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.722+0000 7fbb27400640 1 --2- 172.21.15.53:0/3256140433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb2806c7c0 0x7fbb28166cf0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fbb28146ef0 tx=0x7fbb2002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b073b82838d65c9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:32.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.723+0000 7fbb26000640 1 -- 172.21.15.53:0/3256140433 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2002cbf0 con 0x7fbb2806c7c0 2024-01-16T15:13:32.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.723+0000 7fbb26000640 1 -- 172.21.15.53:0/3256140433 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2002cd50 con 0x7fbb2806c7c0 2024-01-16T15:13:32.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.723+0000 7fbb26000640 1 -- 172.21.15.53:0/3256140433 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb20034640 con 0x7fbb2806c7c0 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.723+0000 7fbb27e00640 1 -- 172.21.15.53:0/3256140433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb2806c7c0 msgr2=0x7fbb28166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.723+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3256140433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb2806c7c0 0x7fbb28166cf0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fbb28146ef0 tx=0x7fbb2002be40 comp rx=0 tx=0).stop 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.724+0000 7fbb27e00640 1 -- 172.21.15.53:0/3256140433 shutdown_connections 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.724+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3256140433 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb2806c7c0 0x7fbb28166cf0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.724+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3256140433 >> v2:172.21.15.146:3300/0 conn(0x7fbb28071f00 0x7fbb28146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.724+0000 7fbb27e00640 1 -- 172.21.15.53:0/3256140433 >> 172.21.15.53:0/3256140433 conn(0x7fbb2806a800 msgr2=0x7fbb2806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.724+0000 7fbb27e00640 1 -- 172.21.15.53:0/3256140433 shutdown_connections 2024-01-16T15:13:32.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.724+0000 7fbb27e00640 1 -- 172.21.15.53:0/3256140433 wait complete. 2024-01-16T15:13:32.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.725+0000 7fbb27e00640 1 Processor -- start 2024-01-16T15:13:32.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.725+0000 7fbb27e00640 1 -- start start 2024-01-16T15:13:32.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.726+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb2806c7c0 0x7fbb2814faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:32.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.726+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb28071410 0x7fbb28140290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:32.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.726+0000 7fbb27e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 0x7fbb28142f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:32.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.726+0000 7fbb27e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb281407d0 con 0x7fbb28071f00 2024-01-16T15:13:32.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.726+0000 7fbb27e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb281409e0 con 0x7fbb28071410 2024-01-16T15:13:32.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.726+0000 7fbb27e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb28142140 con 0x7fbb2806c7c0 2024-01-16T15:13:32.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 0x7fbb28142f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:32.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 0x7fbb28142f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33936/0 (socket says 172.21.15.53:33936) 2024-01-16T15:13:32.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 -- 172.21.15.53:0/3029595830 learned_addr learned my addr 172.21.15.53:0/3029595830 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:32.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb26a00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb2806c7c0 0x7fbb2814faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:32.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb1fe00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb28071410 0x7fbb28140290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:32.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 -- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb2806c7c0 msgr2=0x7fbb2814faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:32.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb2806c7c0 0x7fbb2814faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 -- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb28071410 msgr2=0x7fbb28140290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb28071410 0x7fbb28140290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 -- 172.21.15.53:0/3029595830 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb20009a70 con 0x7fbb28071f00 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb26a00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb2806c7c0 0x7fbb2814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.727+0000 7fbb27400640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 0x7fbb28142f00 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fbb2002c450 tx=0x7fbb20004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.728+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2003e610 con 0x7fbb28071f00 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.728+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb2003e770 con 0x7fbb28071f00 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.728+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb28142420 con 0x7fbb28071f00 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.728+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb2003eb60 con 0x7fbb28071f00 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.728+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb281429e0 con 0x7fbb28071f00 2024-01-16T15:13:32.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.729+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb28142c00 con 0x7fbb28071f00 2024-01-16T15:13:32.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.730+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fbb2003b050 con 0x7fbb28071f00 2024-01-16T15:13:32.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.731+0000 7fbb1ea00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb0006bf20 0x7fbb0006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:32.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.731+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fbb200aaf50 con 0x7fbb28071f00 2024-01-16T15:13:32.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.732+0000 7fbb26a00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb0006bf20 0x7fbb0006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:32.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.733+0000 7fbb26a00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb0006bf20 0x7fbb0006e3e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbb18005880 tx=0x7fbb18009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:32.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.736+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbb20077500 con 0x7fbb28071f00 2024-01-16T15:13:32.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:32.930+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbb28067450 con 0x7fbb28071f00 2024-01-16T15:13:33.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.493+0000 7fbb1ea00640 1 -- 172.21.15.53:0/3029595830 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7fbb2004d070 con 0x7fbb28071f00 2024-01-16T15:13:33.473 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb0006bf20 msgr2=0x7fbb0006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb0006bf20 0x7fbb0006e3e0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fbb18005880 tx=0x7fbb18009040 comp rx=0 tx=0).stop 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 msgr2=0x7fbb28142f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 0x7fbb28142f00 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7fbb2002c450 tx=0x7fbb20004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 shutdown_connections 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbb0006bf20 0x7fbb0006e3e0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbb28071f00 0x7fbb28142f00 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbb28071410 0x7fbb28140290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 --2- 172.21.15.53:0/3029595830 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbb2806c7c0 0x7fbb2814faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.496+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 >> 172.21.15.53:0/3029595830 conn(0x7fbb2806a800 msgr2=0x7fbb2806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.497+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 shutdown_connections 2024-01-16T15:13:33.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.497+0000 7fbb27e00640 1 -- 172.21.15.53:0/3029595830 wait complete. 2024-01-16T15:13:33.511 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:13:33.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.718+0000 7f8532200640 1 Processor -- start 2024-01-16T15:13:33.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.718+0000 7f8532200640 1 -- start start 2024-01-16T15:13:33.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.719+0000 7f8532200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f852c15b2a0 0x7f852c0cf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:33.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.719+0000 7f8532200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f852c0ce070 0x7f852c164f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:33.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.719+0000 7f8532200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852c0d00a0 con 0x7f852c0cfaf0 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.719+0000 7f8532200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c0b4010 con 0x7f852c0ce070 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.719+0000 7f8532200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f852c0b42f0 con 0x7f852c15b2a0 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f852c0ce070 0x7f852c164f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852aa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f852c0cfaf0 0x7f852c0cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54998/0 (socket says 172.21.15.53:54998) 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852aa00640 1 -- 172.21.15.53:0/1684570483 learned_addr learned my addr 172.21.15.53:0/1684570483 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852b400640 1 --2- 172.21.15.53:0/1684570483 >> v2:172.21.15.146:3300/0 conn(0x7f852c15b2a0 0x7f852c0cf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852a000640 1 -- 172.21.15.53:0/1684570483 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13355342 0 0) 0x7f852c0d00a0 con 0x7f852c0cfaf0 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852a000640 1 -- 172.21.15.53:0/1684570483 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f850c003520 con 0x7f852c0cfaf0 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852be00640 1 -- 172.21.15.53:0/1684570483 >> v2:172.21.15.146:3300/0 conn(0x7f852c15b2a0 msgr2=0x7f852c0cf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852be00640 1 --2- 172.21.15.53:0/1684570483 >> v2:172.21.15.146:3300/0 conn(0x7f852c15b2a0 0x7f852c0cf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852be00640 1 -- 172.21.15.53:0/1684570483 >> v1:172.21.15.53:6789/0 conn(0x7f852c0cfaf0 legacy=0x7f852c0cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.720+0000 7f852be00640 1 -- 172.21.15.53:0/1684570483 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0b45d0 con 0x7f852c0ce070 2024-01-16T15:13:33.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.721+0000 7f852b400640 1 --2- 172.21.15.53:0/1684570483 >> v2:172.21.15.146:3300/0 conn(0x7f852c15b2a0 0x7f852c0cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:33.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.722+0000 7f852be00640 1 --2- 172.21.15.53:0/1684570483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c0ce070 0x7f852c164f40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f852c0d0f80 tx=0x7f852402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b5cf10e7844434 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:33.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.722+0000 7f852a000640 1 -- 172.21.15.53:0/1684570483 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852402cc80 con 0x7f852c0ce070 2024-01-16T15:13:33.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.722+0000 7f852a000640 1 -- 172.21.15.53:0/1684570483 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852402cde0 con 0x7f852c0ce070 2024-01-16T15:13:33.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.722+0000 7f852a000640 1 -- 172.21.15.53:0/1684570483 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524034660 con 0x7f852c0ce070 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 -- 172.21.15.53:0/1684570483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c0ce070 msgr2=0x7f852c164f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 --2- 172.21.15.53:0/1684570483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c0ce070 0x7f852c164f40 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f852c0d0f80 tx=0x7f852402bed0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 -- 172.21.15.53:0/1684570483 shutdown_connections 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 --2- 172.21.15.53:0/1684570483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c0ce070 0x7f852c164f40 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 --2- 172.21.15.53:0/1684570483 >> v2:172.21.15.146:3300/0 conn(0x7f852c15b2a0 0x7f852c0cf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 -- 172.21.15.53:0/1684570483 >> 172.21.15.53:0/1684570483 conn(0x7f852c0d1020 msgr2=0x7f852c0d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 -- 172.21.15.53:0/1684570483 shutdown_connections 2024-01-16T15:13:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.723+0000 7f8532200640 1 -- 172.21.15.53:0/1684570483 wait complete. 2024-01-16T15:13:33.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.725+0000 7f8532200640 1 Processor -- start 2024-01-16T15:13:33.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.725+0000 7f8532200640 1 -- start start 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f8532200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f852c0ce070 0x7f852c15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f8532200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f852c0cfaf0 0x7f852c0c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f8532200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 0x7f852c0c9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f8532200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0c8630 con 0x7f852c15b2a0 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f8532200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f852c15b7b0 con 0x7f852c0cfaf0 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f8532200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f852c0c9eb0 con 0x7f852c0ce070 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f852be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 0x7f852c0c9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f852be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 0x7f852c0c9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33956/0 (socket says 172.21.15.53:33956) 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.726+0000 7f852be00640 1 -- 172.21.15.53:0/4284771591 learned_addr learned my addr 172.21.15.53:0/4284771591 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852aa00640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f852c0cfaf0 0x7f852c0c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852b400640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f852c0ce070 0x7f852c15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852be00640 1 -- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f852c0ce070 msgr2=0x7f852c15d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852be00640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f852c0ce070 0x7f852c15d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852be00640 1 -- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f852c0cfaf0 msgr2=0x7f852c0c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852be00640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f852c0cfaf0 0x7f852c0c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852be00640 1 -- 172.21.15.53:0/4284771591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009a70 con 0x7f852c15b2a0 2024-01-16T15:13:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852aa00640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f852c0cfaf0 0x7f852c0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f852be00640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 0x7f852c0c9970 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8524009e80 tx=0x7f8524004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524040650 con 0x7f852c15b2a0 2024-01-16T15:13:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85240407b0 con 0x7f852c15b2a0 2024-01-16T15:13:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524040b60 con 0x7f852c15b2a0 2024-01-16T15:13:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.727+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f852c0c8860 con 0x7f852c15b2a0 2024-01-16T15:13:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.728+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f852c0c8e20 con 0x7f852c15b2a0 2024-01-16T15:13:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.729+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852c15b920 con 0x7f852c15b2a0 2024-01-16T15:13:33.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.730+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f8524040cc0 con 0x7f852c15b2a0 2024-01-16T15:13:33.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.731+0000 7f8523400640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f850406bf50 0x7f850406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:33.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.731+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5637+0+0 (secure 0 0 0) 0x7f85240ab870 con 0x7f852c15b2a0 2024-01-16T15:13:33.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.732+0000 7f852b400640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f850406bf50 0x7f850406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:33.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.733+0000 7f852b400640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f850406bf50 0x7f850406e410 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f851c006fd0 tx=0x7f851c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:33.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.735+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8524078250 con 0x7f852c15b2a0 2024-01-16T15:13:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:33.967+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f852c0675d0 con 0x7f852c15b2a0 2024-01-16T15:13:34.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.497+0000 7f8523400640 1 -- 172.21.15.53:0/4284771591 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v51) v1 ==== 174+0+0 (secure 0 0 0) 0x7f8524009450 con 0x7f852c15b2a0 2024-01-16T15:13:34.478 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 12 and data pool 13 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f850406bf50 msgr2=0x7f850406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f850406bf50 0x7f850406e410 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f851c006fd0 tx=0x7f851c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 msgr2=0x7f852c0c9970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 0x7f852c0c9970 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f8524009e80 tx=0x7f8524004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 shutdown_connections 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f850406bf50 0x7f850406e410 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f852c15b2a0 0x7f852c0c9970 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f852c0cfaf0 0x7f852c0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 --2- 172.21.15.53:0/4284771591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f852c0ce070 0x7f852c15d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.501+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 >> 172.21.15.53:0/4284771591 conn(0x7f852c0d1020 msgr2=0x7f852c0cbef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.502+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 shutdown_connections 2024-01-16T15:13:34.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.502+0000 7f8532200640 1 -- 172.21.15.53:0/4284771591 wait complete. 2024-01-16T15:13:34.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:34.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.731+0000 7fe47c000640 1 Processor -- start 2024-01-16T15:13:34.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.731+0000 7fe47c000640 1 -- start start 2024-01-16T15:13:34.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe47c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe474143ae0 0x7fe4740cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:34.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe47c000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe47415e3b0 0x7fe4740702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:34.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe47c000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4740cfc30 con 0x7fe4740cf7d0 2024-01-16T15:13:34.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe47c000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4740b1ee0 con 0x7fe47415e3b0 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe47c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740b2780 con 0x7fe474143ae0 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe47a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe47415e3b0 0x7fe4740702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe478c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe4740cf7d0 0x7fe47415dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55006/0 (socket says 172.21.15.53:55006) 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.733+0000 7fe478c00640 1 -- 172.21.15.53:0/3286476044 learned_addr learned my addr 172.21.15.53:0/3286476044 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.734+0000 7fe479600640 1 --2- 172.21.15.53:0/3286476044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe474143ae0 0x7fe4740cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.734+0000 7fe47a000640 1 -- 172.21.15.53:0/3286476044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe474143ae0 msgr2=0x7fe4740cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.734+0000 7fe47a000640 1 --2- 172.21.15.53:0/3286476044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe474143ae0 0x7fe4740cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.734+0000 7fe47a000640 1 -- 172.21.15.53:0/3286476044 >> v1:172.21.15.53:6789/0 conn(0x7fe4740cf7d0 legacy=0x7fe47415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.734+0000 7fe473e00640 1 -- 172.21.15.53:0/3286476044 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776120130 0 0) 0x7fe4740cfc30 con 0x7fe4740cf7d0 2024-01-16T15:13:34.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.734+0000 7fe47a000640 1 -- 172.21.15.53:0/3286476044 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47415e990 con 0x7fe47415e3b0 2024-01-16T15:13:34.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.735+0000 7fe47a000640 1 --2- 172.21.15.53:0/3286476044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47415e3b0 0x7fe4740702c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0097a0 tx=0x7fe46c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9a0ffb54b673ab0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:34.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe473e00640 1 -- 172.21.15.53:0/3286476044 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c031830 con 0x7fe47415e3b0 2024-01-16T15:13:34.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe473e00640 1 -- 172.21.15.53:0/3286476044 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c031990 con 0x7fe47415e3b0 2024-01-16T15:13:34.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe473e00640 1 -- 172.21.15.53:0/3286476044 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46c031d20 con 0x7fe47415e3b0 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 -- 172.21.15.53:0/3286476044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47415e3b0 msgr2=0x7fe4740702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 --2- 172.21.15.53:0/3286476044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47415e3b0 0x7fe4740702c0 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0097a0 tx=0x7fe46c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 -- 172.21.15.53:0/3286476044 shutdown_connections 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 --2- 172.21.15.53:0/3286476044 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe47415e3b0 0x7fe4740702c0 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 --2- 172.21.15.53:0/3286476044 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe474143ae0 0x7fe4740cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 -- 172.21.15.53:0/3286476044 >> 172.21.15.53:0/3286476044 conn(0x7fe47406a800 msgr2=0x7fe47406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 -- 172.21.15.53:0/3286476044 shutdown_connections 2024-01-16T15:13:34.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.736+0000 7fe47c000640 1 -- 172.21.15.53:0/3286476044 wait complete. 2024-01-16T15:13:34.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.738+0000 7fe47c000640 1 Processor -- start 2024-01-16T15:13:34.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.738+0000 7fe47c000640 1 -- start start 2024-01-16T15:13:34.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.739+0000 7fe47c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 0x7fe4740ce0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.739+0000 7fe47c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe474143ae0 0x7fe47415ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.739+0000 7fe47c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe47415e3b0 0x7fe474158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.739+0000 7fe47c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740cd130 con 0x7fe4740cf7d0 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.739+0000 7fe47c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740cc790 con 0x7fe474143ae0 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.739+0000 7fe47c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4740cca70 con 0x7fe47415e3b0 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 0x7fe4740ce0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 0x7fe4740ce0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33974/0 (socket says 172.21.15.53:33974) 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 -- 172.21.15.53:0/3868089752 learned_addr learned my addr 172.21.15.53:0/3868089752 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe478c00640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe474143ae0 0x7fe47415ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:34.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe47a000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe47415e3b0 0x7fe474158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:34.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 -- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe47415e3b0 msgr2=0x7fe474158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe47415e3b0 0x7fe474158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 -- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe474143ae0 msgr2=0x7fe47415ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe474143ae0 0x7fe47415ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 -- 172.21.15.53:0/3868089752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c009400 con 0x7fe4740cf7d0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe478c00640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe474143ae0 0x7fe47415ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.740+0000 7fe479600640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 0x7fe4740ce0c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe464007ef0 tx=0x7fe46400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.741+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464014620 con 0x7fe4740cf7d0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.741+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464014780 con 0x7fe4740cf7d0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.741+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464014b30 con 0x7fe4740cf7d0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.741+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4740ccd50 con 0x7fe4740cf7d0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.741+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47415d390 con 0x7fe4740cf7d0 2024-01-16T15:13:34.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.742+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4740cd3a0 con 0x7fe4740cf7d0 2024-01-16T15:13:34.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.743+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe464014c90 con 0x7fe4740cf7d0 2024-01-16T15:13:34.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.744+0000 7fe471600640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe44c06bf50 0x7fe44c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:34.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.744+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe464088fb0 con 0x7fe4740cf7d0 2024-01-16T15:13:34.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.745+0000 7fe478c00640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe44c06bf50 0x7fe44c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:34.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.746+0000 7fe478c00640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe44c06bf50 0x7fe44c06e410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe4740b0b10 tx=0x7fe45c007c10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:34.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.749+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe464055870 con 0x7fe4740cf7d0 2024-01-16T15:13:34.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.946+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe47415d5b0 con 0x7fe4740cf7d0 2024-01-16T15:13:34.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.948+0000 7fe471600640 1 -- 172.21.15.53:0/3868089752 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16580 (secure 0 0 0) 0x7fe464058e90 con 0x7fe4740cf7d0 2024-01-16T15:13:34.928 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:34.929 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":85,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:33.970243+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:13:31.825905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:13:32.988111+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:34.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe44c06bf50 msgr2=0x7fe44c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe44c06bf50 0x7fe44c06e410 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7fe4740b0b10 tx=0x7fe45c007c10 comp rx=0 tx=0).stop 2024-01-16T15:13:34.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 msgr2=0x7fe4740ce0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 0x7fe4740ce0c0 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe464007ef0 tx=0x7fe46400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 shutdown_connections 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe44c06bf50 0x7fe44c06e410 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe47415e3b0 0x7fe474158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe474143ae0 0x7fe47415ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 --2- 172.21.15.53:0/3868089752 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe4740cf7d0 0x7fe4740ce0c0 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 >> 172.21.15.53:0/3868089752 conn(0x7fe47406a800 msgr2=0x7fe47415b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 shutdown_connections 2024-01-16T15:13:34.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:34.951+0000 7fe47c000640 1 -- 172.21.15.53:0/3868089752 wait complete. 2024-01-16T15:13:34.950 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:35.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.182+0000 7ff381400640 1 Processor -- start 2024-01-16T15:13:35.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.182+0000 7ff381400640 1 -- start start 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff381400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff37c143c40 0x7ff37c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff381400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff37c0ccc10 0x7ff37c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff381400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff37c070e50 con 0x7ff37c0717d0 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff381400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37c0723c0 con 0x7ff37c0ccc10 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff381400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff37c144020 con 0x7ff37c143c40 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff380a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff37c0ccc10 0x7ff37c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff380a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff37c0ccc10 0x7ff37c06d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33986/0 (socket says 172.21.15.53:33986) 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff380a00640 1 -- 172.21.15.53:0/4242419586 learned_addr learned my addr 172.21.15.53:0/4242419586 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.184+0000 7ff373200640 1 --1- 172.21.15.53:0/4242419586 >> v1:172.21.15.53:6789/0 conn(0x7ff37c0717d0 0x7ff37c0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55020/0 (socket says 172.21.15.53:55020) 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.185+0000 7ff37be00640 1 --2- 172.21.15.53:0/4242419586 >> v2:172.21.15.146:3300/0 conn(0x7ff37c143c40 0x7ff37c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.185+0000 7ff380a00640 1 -- 172.21.15.53:0/4242419586 >> v2:172.21.15.146:3300/0 conn(0x7ff37c143c40 msgr2=0x7ff37c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.185+0000 7ff380a00640 1 --2- 172.21.15.53:0/4242419586 >> v2:172.21.15.146:3300/0 conn(0x7ff37c143c40 0x7ff37c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.185+0000 7ff380a00640 1 -- 172.21.15.53:0/4242419586 >> v1:172.21.15.53:6789/0 conn(0x7ff37c0717d0 legacy=0x7ff37c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.185+0000 7ff380a00640 1 -- 172.21.15.53:0/4242419586 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff37c071e00 con 0x7ff37c0ccc10 2024-01-16T15:13:35.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.185+0000 7ff37b400640 1 -- 172.21.15.53:0/4242419586 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176580821 0 0) 0x7ff37c070e50 con 0x7ff37c0717d0 2024-01-16T15:13:35.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.186+0000 7ff380a00640 1 --2- 172.21.15.53:0/4242419586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c0ccc10 0x7ff37c06d160 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff37c06e410 tx=0x7ff37402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961ea56393801eca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:35.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.186+0000 7ff37b400640 1 -- 172.21.15.53:0/4242419586 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374031870 con 0x7ff37c0ccc10 2024-01-16T15:13:35.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.186+0000 7ff37b400640 1 -- 172.21.15.53:0/4242419586 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3740319d0 con 0x7ff37c0ccc10 2024-01-16T15:13:35.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.186+0000 7ff37b400640 1 -- 172.21.15.53:0/4242419586 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374031d60 con 0x7ff37c0ccc10 2024-01-16T15:13:35.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 -- 172.21.15.53:0/4242419586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c0ccc10 msgr2=0x7ff37c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 --2- 172.21.15.53:0/4242419586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c0ccc10 0x7ff37c06d160 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7ff37c06e410 tx=0x7ff37402b910 comp rx=0 tx=0).stop 2024-01-16T15:13:35.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 -- 172.21.15.53:0/4242419586 shutdown_connections 2024-01-16T15:13:35.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 --2- 172.21.15.53:0/4242419586 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c0ccc10 0x7ff37c06d160 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 --2- 172.21.15.53:0/4242419586 >> v2:172.21.15.146:3300/0 conn(0x7ff37c143c40 0x7ff37c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 -- 172.21.15.53:0/4242419586 >> 172.21.15.53:0/4242419586 conn(0x7ff37c06a800 msgr2=0x7ff37c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:35.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 -- 172.21.15.53:0/4242419586 shutdown_connections 2024-01-16T15:13:35.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.187+0000 7ff381400640 1 -- 172.21.15.53:0/4242419586 wait complete. 2024-01-16T15:13:35.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.189+0000 7ff381400640 1 Processor -- start 2024-01-16T15:13:35.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.189+0000 7ff381400640 1 -- start start 2024-01-16T15:13:35.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff381400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c0717d0 0x7ff37c139550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff381400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff37c0ccc10 0x7ff37c142470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff381400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 0x7ff37c0adc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff381400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c1413d0 con 0x7ff37c143c40 2024-01-16T15:13:35.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff381400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c1416b0 con 0x7ff37c0717d0 2024-01-16T15:13:35.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff381400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff37c141990 con 0x7ff37c0ccc10 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff380a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 0x7ff37c0adc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff380a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 0x7ff37c0adc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33988/0 (socket says 172.21.15.53:33988) 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.190+0000 7ff380a00640 1 -- 172.21.15.53:0/1786641211 learned_addr learned my addr 172.21.15.53:0/1786641211 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff37be00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c0717d0 0x7ff37c139550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff373200640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff37c0ccc10 0x7ff37c142470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff380a00640 1 -- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff37c0ccc10 msgr2=0x7ff37c142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff380a00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff37c0ccc10 0x7ff37c142470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff380a00640 1 -- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c0717d0 msgr2=0x7ff37c139550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff380a00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c0717d0 0x7ff37c139550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff380a00640 1 -- 172.21.15.53:0/1786641211 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3740094d0 con 0x7ff37c143c40 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff373200640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff37c0ccc10 0x7ff37c142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff380a00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 0x7ff37c0adc60 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff3740090b0 tx=0x7ff3740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:35.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37403ac60 con 0x7ff37c143c40 2024-01-16T15:13:35.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37403adc0 con 0x7ff37c143c40 2024-01-16T15:13:35.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff374044630 con 0x7ff37c143c40 2024-01-16T15:13:35.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.191+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c141c70 con 0x7ff37c143c40 2024-01-16T15:13:35.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.192+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff37c139a90 con 0x7ff37c143c40 2024-01-16T15:13:35.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.193+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff37c142190 con 0x7ff37c143c40 2024-01-16T15:13:35.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.194+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff37404d450 con 0x7ff37c143c40 2024-01-16T15:13:35.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.195+0000 7ff378c00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff35406bf20 0x7ff35406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.195+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff37402d040 con 0x7ff37c143c40 2024-01-16T15:13:35.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.196+0000 7ff37be00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff35406bf20 0x7ff35406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.197+0000 7ff37be00640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff35406bf20 0x7ff35406e3e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff36c006fd0 tx=0x7ff36c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:35.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.200+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff37c142190 con 0x7ff37c143c40 2024-01-16T15:13:35.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.401+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff37c142190 con 0x7ff37c143c40 2024-01-16T15:13:35.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.404+0000 7ff378c00640 1 -- 172.21.15.53:0/1786641211 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+16580 (secure 0 0 0) 0x7ff37c142190 con 0x7ff37c143c40 2024-01-16T15:13:35.384 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:35.384 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":85,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:33.970243+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:13:31.825905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:13:32.988111+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff35406bf20 msgr2=0x7ff35406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff35406bf20 0x7ff35406e3e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff36c006fd0 tx=0x7ff36c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 msgr2=0x7ff37c0adc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 0x7ff37c0adc60 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7ff3740090b0 tx=0x7ff3740046c0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 shutdown_connections 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff35406bf20 0x7ff35406e3e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff37c143c40 0x7ff37c0adc60 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff37c0ccc10 0x7ff37c142470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 --2- 172.21.15.53:0/1786641211 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff37c0717d0 0x7ff37c139550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 >> 172.21.15.53:0/1786641211 conn(0x7ff37c06a800 msgr2=0x7ff37c140a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.407+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 shutdown_connections 2024-01-16T15:13:35.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.408+0000 7ff381400640 1 -- 172.21.15.53:0/1786641211 wait complete. 2024-01-16T15:13:35.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:13:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.650+0000 7fe9e7200640 1 Processor -- start 2024-01-16T15:13:35.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.650+0000 7fe9e7200640 1 -- start start 2024-01-16T15:13:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.651+0000 7fe9e7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe9e815d4b0 0x7fe9e80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.651+0000 7fe9e7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80b2730 con 0x7fe9e80cf9a0 2024-01-16T15:13:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9e7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9e80d1ec0 con 0x7fe9e80b41e0 2024-01-16T15:13:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9e7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9e80b29d0 con 0x7fe9e815d4b0 2024-01-16T15:13:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9dd200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe9e80cf9a0 0x7fe9e80cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55036/0 (socket says 172.21.15.53:55036) 2024-01-16T15:13:35.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9dd200640 1 -- 172.21.15.53:0/3630053693 learned_addr learned my addr 172.21.15.53:0/3630053693 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/3630053693 >> v2:172.21.15.146:3300/0 conn(0x7fe9e815d4b0 0x7fe9e80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845468932 0 0) 0x7fe9e80b2730 con 0x7fe9e80cf9a0 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.652+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c8003520 con 0x7fe9e80cf9a0 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 901150675 0 0) 0x7fe9c8003520 con 0x7fe9e80cf9a0 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 >> v2:172.21.15.146:3300/0 conn(0x7fe9e815d4b0 msgr2=0x7fe9e80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 --2- 172.21.15.53:0/3630053693 >> v2:172.21.15.146:3300/0 conn(0x7fe9e815d4b0 0x7fe9e80d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 >> v1:172.21.15.146:6789/0 conn(0x7fe9e80b41e0 legacy=0x7fe9e80ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e80b2cb0 con 0x7fe9e80cf9a0 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fe9d0001670 con 0x7fe9e80cf9a0 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 >> v1:172.21.15.53:6789/0 conn(0x7fe9e80cf9a0 legacy=0x7fe9e80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9c80039c0 0x7fe9c8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9c80047c0 0x7fe9c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c8005180 0x7fe9c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b29d0 con 0x7fe9c80039c0 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c800c080 con 0x7fe9c80047c0 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9c800c240 con 0x7fe9c8005180 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.653+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9c80039c0 0x7fe9c8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e6800640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c8005180 0x7fe9c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e5e00640 1 -- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c8005180 msgr2=0x7fe9c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9dd200640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9c80047c0 0x7fe9c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c8005180 0x7fe9c800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e5e00640 1 -- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9c80047c0 msgr2=0x7fe9c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9c80047c0 0x7fe9c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e5e00640 1 -- 172.21.15.53:0/3630053693 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c8003520 con 0x7fe9c80039c0 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9e6800640 1 -- 172.21.15.53:0/3630053693 reap_dead start 2024-01-16T15:13:35.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.654+0000 7fe9dd200640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9c80047c0 0x7fe9c8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:35.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.655+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9c80039c0 0x7fe9c8003de0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009fc0 tx=0x7fe9c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23cb39dc575af59b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.655+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d802cc20 con 0x7fe9c80039c0 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.655+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9d800f040 con 0x7fe9c80039c0 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.655+0000 7fe9e5400640 1 -- 172.21.15.53:0/3630053693 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8034440 con 0x7fe9c80039c0 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 -- 172.21.15.53:0/3630053693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9c80039c0 msgr2=0x7fe9c8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9c80039c0 0x7fe9c8003de0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8009fc0 tx=0x7fe9c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 -- 172.21.15.53:0/3630053693 shutdown_connections 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 --2- 172.21.15.53:0/3630053693 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9c80039c0 0x7fe9c8003de0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 -- 172.21.15.53:0/3630053693 >> 172.21.15.53:0/3630053693 conn(0x7fe9e80d3000 msgr2=0x7fe9e80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 -- 172.21.15.53:0/3630053693 shutdown_connections 2024-01-16T15:13:35.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.656+0000 7fe9e7200640 1 -- 172.21.15.53:0/3630053693 wait complete. 2024-01-16T15:13:35.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.658+0000 7fe9e7200640 1 Processor -- start 2024-01-16T15:13:35.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.658+0000 7fe9e7200640 1 -- start start 2024-01-16T15:13:35.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c80039c0 0x7fe9e81f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9e8160090 0x7fe9e80b7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 0x7fe9e80ca970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80ca550 con 0x7fe9e80b8110 2024-01-16T15:13:35.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e80b8590 con 0x7fe9e8160090 2024-01-16T15:13:35.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e81f6b10 con 0x7fe9c80039c0 2024-01-16T15:13:35.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 0x7fe9e80ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 0x7fe9e80ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34004/0 (socket says 172.21.15.53:34004) 2024-01-16T15:13:35.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.659+0000 7fe9e6800640 1 -- 172.21.15.53:0/660721156 learned_addr learned my addr 172.21.15.53:0/660721156 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c80039c0 0x7fe9e81f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e6800640 1 -- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c80039c0 msgr2=0x7fe9e81f62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e6800640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c80039c0 0x7fe9e81f62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e6800640 1 -- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9e8160090 msgr2=0x7fe9e80b7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9dd200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9e8160090 0x7fe9e80b7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e6800640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9e8160090 0x7fe9e80b7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:35.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e6800640 1 -- 172.21.15.53:0/660721156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e815da10 con 0x7fe9e80b8110 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9dd200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9e8160090 0x7fe9e80b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e6800640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 0x7fe9e80ca970 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806ac50 tx=0x7fe9e000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e000b070 con 0x7fe9e80b8110 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e00054e0 con 0x7fe9e80b8110 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e0003830 con 0x7fe9e80b8110 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.660+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e81f6df0 con 0x7fe9e80b8110 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.661+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e80ca170 con 0x7fe9e80b8110 2024-01-16T15:13:35.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.662+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e80ca690 con 0x7fe9e80b8110 2024-01-16T15:13:35.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.663+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fe9e0003990 con 0x7fe9e80b8110 2024-01-16T15:13:35.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.664+0000 7fe9dea00640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe9c006bf20 0x7fe9c006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:35.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.664+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fe9e0087e60 con 0x7fe9e80b8110 2024-01-16T15:13:35.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.664+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe9c006bf20 0x7fe9c006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:35.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.665+0000 7fe9e5e00640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe9c006bf20 0x7fe9c006e3e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8036000 tx=0x7fe9d802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:35.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.668+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe9e80ca690 con 0x7fe9e80b8110 2024-01-16T15:13:35.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:35.907+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe9e80ca690 con 0x7fe9e80b8110 2024-01-16T15:13:36.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.528+0000 7fe9dea00640 1 -- 172.21.15.53:0/660721156 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v54) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe9e80ca690 con 0x7fe9e80b8110 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe9c006bf20 msgr2=0x7fe9c006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe9c006bf20 0x7fe9c006e3e0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8036000 tx=0x7fe9d802e000 comp rx=0 tx=0).stop 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 msgr2=0x7fe9e80ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 0x7fe9e80ca970 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fe9e806ac50 tx=0x7fe9e000d890 comp rx=0 tx=0).stop 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 shutdown_connections 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe9c006bf20 0x7fe9c006e3e0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9e80b8110 0x7fe9e80ca970 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9e8160090 0x7fe9e80b7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 --2- 172.21.15.53:0/660721156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9c80039c0 0x7fe9e81f62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.532+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 >> 172.21.15.53:0/660721156 conn(0x7fe9e80d3000 msgr2=0x7fe9e806c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.533+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 shutdown_connections 2024-01-16T15:13:36.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.533+0000 7fe9e7200640 1 -- 172.21.15.53:0/660721156 wait complete. 2024-01-16T15:13:36.535 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:13:36.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:13:36.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.776+0000 7f193d200640 1 Processor -- start 2024-01-16T15:13:36.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.776+0000 7f193d200640 1 -- start start 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.777+0000 7f193d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 0x7f19380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.777+0000 7f193d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19380cf7d0 0x7f193815dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f193d200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19380cfc30 con 0x7f193815e1e0 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f193d200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19380b1ee0 con 0x7f19380cf7d0 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f193d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b2780 con 0x7f1938143ae0 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f1936800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f193815e1e0 0x7f193815a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55038/0 (socket says 172.21.15.53:55038) 2024-01-16T15:13:36.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f1936800640 1 -- 172.21.15.53:0/1646781451 learned_addr learned my addr 172.21.15.53:0/1646781451 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f1935400640 1 --2- 172.21.15.53:0/1646781451 >> v2:172.21.15.146:3300/0 conn(0x7f19380cf7d0 0x7f193815dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f1935e00640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 0x7f19380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.778+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3475476048 0 0) 0x7f19380cfc30 con 0x7f193815e1e0 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1918003520 con 0x7f193815e1e0 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955870387 0 0) 0x7f1918003520 con 0x7f193815e1e0 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 msgr2=0x7f19380cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 0x7f19380cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 >> v2:172.21.15.146:3300/0 conn(0x7f19380cf7d0 msgr2=0x7f193815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 --2- 172.21.15.53:0/1646781451 >> v2:172.21.15.146:3300/0 conn(0x7f19380cf7d0 0x7f193815dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193815e990 con 0x7f193815e1e0 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f19300019f0 con 0x7f193815e1e0 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 >> v1:172.21.15.53:6789/0 conn(0x7f193815e1e0 legacy=0x7f193815a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 0x7f1918003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1918003f40 0x7f19180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918004d00 0x7f1918005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b1ee0 con 0x7f1918003f40 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19380b2780 con 0x7f1918004d00 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.779+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f191800b980 con 0x7f1938143ae0 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935400640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1918003f40 0x7f19180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1936800640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918004d00 0x7f1918005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935e00640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 0x7f1918003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935400640 1 -- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 msgr2=0x7f1918003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935400640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1938143ae0 0x7f1918003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935400640 1 -- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918004d00 msgr2=0x7f1918005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935400640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918004d00 0x7f1918005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.780+0000 7f1935400640 1 -- 172.21.15.53:0/1646781451 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1918003520 con 0x7f1918003f40 2024-01-16T15:13:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.781+0000 7f1935400640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1918003f40 0x7f19180047c0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1920009af0 tx=0x7f19180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58e1ba7933ad7564 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.781+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920030ae0 con 0x7f1918003f40 2024-01-16T15:13:36.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1920030c40 con 0x7f1918003f40 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f1934a00640 1 -- 172.21.15.53:0/1646781451 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920038440 con 0x7f1918003f40 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f193d200640 1 -- 172.21.15.53:0/1646781451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1918003f40 msgr2=0x7f19180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f193d200640 1 --2- 172.21.15.53:0/1646781451 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1918003f40 0x7f19180047c0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f1920009af0 tx=0x7f19180057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f1936800640 1 -- 172.21.15.53:0/1646781451 reap_dead start 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f193d200640 1 -- 172.21.15.53:0/1646781451 shutdown_connections 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f193d200640 1 -- 172.21.15.53:0/1646781451 >> 172.21.15.53:0/1646781451 conn(0x7f193806a800 msgr2=0x7f193806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f193d200640 1 -- 172.21.15.53:0/1646781451 shutdown_connections 2024-01-16T15:13:36.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.782+0000 7f193d200640 1 -- 172.21.15.53:0/1646781451 wait complete. 2024-01-16T15:13:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.784+0000 7f193d200640 1 Processor -- start 2024-01-16T15:13:36.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.784+0000 7f193d200640 1 -- start start 2024-01-16T15:13:36.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.785+0000 7f193d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918003f40 0x7f193813c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.785+0000 7f193d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 0x7f193806c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.785+0000 7f193d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f193813cfa0 0x7f193806c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.785+0000 7f193d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193813c920 con 0x7f193813cb20 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.785+0000 7f193d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f193813d4e0 con 0x7f1918003f40 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.785+0000 7f193d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193813d7c0 con 0x7f193813cfa0 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 0x7f193806c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 0x7f193806c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34024/0 (socket says 172.21.15.53:34024) 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 -- 172.21.15.53:0/881400099 learned_addr learned my addr 172.21.15.53:0/881400099 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935e00640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918003f40 0x7f193813c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1936800640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f193813cfa0 0x7f193806c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 -- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f193813cfa0 msgr2=0x7f193806c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f193813cfa0 0x7f193806c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 -- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918003f40 msgr2=0x7f193813c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918003f40 0x7f193813c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:36.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1935400640 1 -- 172.21.15.53:0/881400099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193813daa0 con 0x7f193813cb20 2024-01-16T15:13:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.786+0000 7f1936800640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f193813cfa0 0x7f193806c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.787+0000 7f1935400640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 0x7f193806c470 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f192003b040 tx=0x7f1920004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:36.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.787+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920003ae0 con 0x7f193813cb20 2024-01-16T15:13:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.787+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1920003c40 con 0x7f193813cb20 2024-01-16T15:13:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.787+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1920044450 con 0x7f193813cb20 2024-01-16T15:13:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.787+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f193813dd80 con 0x7f193813cb20 2024-01-16T15:13:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.787+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19380ce850 con 0x7f193813cb20 2024-01-16T15:13:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.788+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19380675d0 con 0x7f193813cb20 2024-01-16T15:13:36.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.789+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f19200445b0 con 0x7f193813cb20 2024-01-16T15:13:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.790+0000 7f192e000640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191006bf70 0x7f191006e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:36.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.790+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f19200ae9d0 con 0x7f193813cb20 2024-01-16T15:13:36.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.791+0000 7f1935e00640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191006bf70 0x7f191006e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:36.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.792+0000 7f1935e00640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191006bf70 0x7f191006e430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1928009870 tx=0x7f1928008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:36.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:36.795+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f192007b1d0 con 0x7f193813cb20 2024-01-16T15:13:37.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.030+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f19380ce9e0 con 0x7f193813cb20 2024-01-16T15:13:37.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.547+0000 7f192e000640 1 -- 172.21.15.53:0/881400099 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v55) v1 ==== 122+0+0 (secure 0 0 0) 0x7f192007e7f0 con 0x7f193813cb20 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.550+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191006bf70 msgr2=0x7f191006e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.550+0000 7f193d200640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191006bf70 0x7f191006e430 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f1928009870 tx=0x7f1928008040 comp rx=0 tx=0).stop 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.550+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 msgr2=0x7f193806c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.550+0000 7f193d200640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 0x7f193806c470 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f192003b040 tx=0x7f1920004730 comp rx=0 tx=0).stop 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.550+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 shutdown_connections 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191006bf70 0x7f191006e430 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f193813cfa0 0x7f193806c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f193813cb20 0x7f193806c470 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 --2- 172.21.15.53:0/881400099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1918003f40 0x7f193813c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 >> 172.21.15.53:0/881400099 conn(0x7f193806a800 msgr2=0x7f1938142dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:37.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 shutdown_connections 2024-01-16T15:13:37.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.551+0000 7f193d200640 1 -- 172.21.15.53:0/881400099 wait complete. 2024-01-16T15:13:37.553 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:13:37.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:13:37.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.786+0000 7f93e2200640 1 Processor -- start 2024-01-16T15:13:37.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.786+0000 7f93e2200640 1 -- start start 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.787+0000 7f93e2200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f93dc15afb0 0x7f93dc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e2200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93dc13fae0 0x7f93dc0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e2200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93dc13ff30 con 0x7f93dc13cfa0 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e2200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0cdc40 con 0x7f93dc13fae0 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e2200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93dc0cdec0 con 0x7f93dc15afb0 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93dc13fae0 0x7f93dc0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f93dc13fae0 0x7f93dc0b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34026/0 (socket says 172.21.15.53:34026) 2024-01-16T15:13:37.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 -- 172.21.15.53:0/4134828000 learned_addr learned my addr 172.21.15.53:0/4134828000 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93d3c00640 1 --2- 172.21.15.53:0/4134828000 >> v2:172.21.15.146:3300/0 conn(0x7f93dc15afb0 0x7f93dc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 -- 172.21.15.53:0/4134828000 >> v2:172.21.15.146:3300/0 conn(0x7f93dc15afb0 msgr2=0x7f93dc13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 --2- 172.21.15.53:0/4134828000 >> v2:172.21.15.146:3300/0 conn(0x7f93dc15afb0 0x7f93dc13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 -- 172.21.15.53:0/4134828000 >> v1:172.21.15.146:6789/0 conn(0x7f93dc13cfa0 legacy=0x7f93dc0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93e1800640 1 -- 172.21.15.53:0/4134828000 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93dc0ce1a0 con 0x7f93dc13fae0 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.788+0000 7f93d3c00640 1 --2- 172.21.15.53:0/4134828000 >> v2:172.21.15.146:3300/0 conn(0x7f93dc15afb0 0x7f93dc13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:37.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.790+0000 7f93e1800640 1 --2- 172.21.15.53:0/4134828000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc0b2040 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0b3fd0 tx=0x7f93d402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a63da8a09b1f979b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.790+0000 7f93dbe00640 1 -- 172.21.15.53:0/4134828000 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d402c9f0 con 0x7f93dc13fae0 2024-01-16T15:13:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.790+0000 7f93dbe00640 1 -- 172.21.15.53:0/4134828000 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93d402cb50 con 0x7f93dc13fae0 2024-01-16T15:13:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.790+0000 7f93dbe00640 1 -- 172.21.15.53:0/4134828000 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93d40343f0 con 0x7f93dc13fae0 2024-01-16T15:13:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.790+0000 7f93e2200640 1 -- 172.21.15.53:0/4134828000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 msgr2=0x7f93dc0b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 --2- 172.21.15.53:0/4134828000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc0b2040 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f93dc0b3fd0 tx=0x7f93d402be50 comp rx=0 tx=0).stop 2024-01-16T15:13:37.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 -- 172.21.15.53:0/4134828000 shutdown_connections 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 --2- 172.21.15.53:0/4134828000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc0b2040 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 --2- 172.21.15.53:0/4134828000 >> v2:172.21.15.146:3300/0 conn(0x7f93dc15afb0 0x7f93dc13f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 -- 172.21.15.53:0/4134828000 >> 172.21.15.53:0/4134828000 conn(0x7f93dc06a800 msgr2=0x7f93dc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 -- 172.21.15.53:0/4134828000 shutdown_connections 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.791+0000 7f93e2200640 1 -- 172.21.15.53:0/4134828000 wait complete. 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.792+0000 7f93e2200640 1 Processor -- start 2024-01-16T15:13:37.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.793+0000 7f93e2200640 1 -- start start 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.793+0000 7f93e2200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93dc13cfa0 0x7f93dc153e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.793+0000 7f93e2200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc156890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e2200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93dc15afb0 0x7f93dc1f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e2200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc153280 con 0x7f93dc13fae0 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e2200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc1678f0 con 0x7f93dc13cfa0 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e2200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93dc152fa0 con 0x7f93dc15afb0 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc156890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc156890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34034/0 (socket says 172.21.15.53:34034) 2024-01-16T15:13:37.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 -- 172.21.15.53:0/1876997837 learned_addr learned my addr 172.21.15.53:0/1876997837 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:37.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e1800640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93dc15afb0 0x7f93dc1f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 -- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93dc15afb0 msgr2=0x7f93dc1f7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e0e00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93dc13cfa0 0x7f93dc153e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93dc15afb0 0x7f93dc1f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 -- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93dc13cfa0 msgr2=0x7f93dc153e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93dc13cfa0 0x7f93dc153e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93d3c00640 1 -- 172.21.15.53:0/1876997837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93d4009b30 con 0x7f93dc13fae0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.794+0000 7f93e0e00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93dc13cfa0 0x7f93dc153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.795+0000 7f93d3c00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc156890 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f93c400c7f0 tx=0x7f93c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.795+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4007820 con 0x7f93dc13fae0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.795+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c4007980 con 0x7f93dc13fae0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.795+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4007cf0 con 0x7f93dc13fae0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.795+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93dc153b10 con 0x7f93dc13fae0 2024-01-16T15:13:37.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.795+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93dc152c30 con 0x7f93dc13fae0 2024-01-16T15:13:37.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.797+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93dc067450 con 0x7f93dc13fae0 2024-01-16T15:13:37.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.797+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f93c40113f0 con 0x7f93dc13fae0 2024-01-16T15:13:37.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.798+0000 7f93d9600640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93b406bf50 0x7f93b406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:37.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.799+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f93c4088990 con 0x7f93dc13fae0 2024-01-16T15:13:37.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.799+0000 7f93e0e00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93b406bf50 0x7f93b406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:37.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.801+0000 7f93e0e00640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93b406bf50 0x7f93b406e410 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f93dc071760 tx=0x7f93cc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:37.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:37.803+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f93c4054fd0 con 0x7f93dc13fae0 2024-01-16T15:13:38.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.036+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93dc0cc650 con 0x7f93dc13fae0 2024-01-16T15:13:38.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.037+0000 7f93d9600640 1 -- 172.21.15.53:0/1876997837 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f93c401f090 con 0x7f93dc13fae0 2024-01-16T15:13:38.018 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:38.018 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":55,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6119,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3084871157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3084871157},{"type":"v1","addr":"172.21.15.146:6835","nonce":3084871157}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2008413931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2008413931},{"type":"v1","addr":"172.21.15.53:6835","nonce":2008413931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/614678519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":614678519},{"type":"v1","addr":"172.21.15.146:6837","nonce":614678519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:13:33.971028+0000","modified":"2024-01-16T15:13:37.032876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8040},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/3609060592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3609060592},{"type":"v1","addr":"172.21.15.53:6834","nonce":3609060592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:13:38.018 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 55 2024-01-16T15:13:38.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93b406bf50 msgr2=0x7f93b406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93b406bf50 0x7f93b406e410 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f93dc071760 tx=0x7f93cc009040 comp rx=0 tx=0).stop 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 msgr2=0x7f93dc156890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc156890 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f93c400c7f0 tx=0x7f93c400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 shutdown_connections 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f93b406bf50 0x7f93b406e410 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f93dc15afb0 0x7f93dc1f7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f93dc13fae0 0x7f93dc156890 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 --2- 172.21.15.53:0/1876997837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f93dc13cfa0 0x7f93dc153e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 >> 172.21.15.53:0/1876997837 conn(0x7f93dc06a800 msgr2=0x7f93dc0af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 shutdown_connections 2024-01-16T15:13:38.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.040+0000 7f93e2200640 1 -- 172.21.15.53:0/1876997837 wait complete. 2024-01-16T15:13:38.040 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:13:38.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.279+0000 7faa7fe00640 1 Processor -- start 2024-01-16T15:13:38.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.279+0000 7faa7fe00640 1 -- start start 2024-01-16T15:13:38.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7faa80146770 0x7faa8006c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa800b2e10 con 0x7faa80071410 2024-01-16T15:13:38.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa80146bc0 con 0x7faa80071f00 2024-01-16T15:13:38.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7fe00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8006cec0 con 0x7faa80146770 2024-01-16T15:13:38.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faa80071410 0x7faa800717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55044/0 (socket says 172.21.15.53:55044) 2024-01-16T15:13:38.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7ea00640 1 -- 172.21.15.53:0/312931069 learned_addr learned my addr 172.21.15.53:0/312931069 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:38.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.281+0000 7faa7f400640 1 --2- 172.21.15.53:0/312931069 >> v2:172.21.15.146:3300/0 conn(0x7faa80146770 0x7faa8006c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673031636 0 0) 0x7faa800b2e10 con 0x7faa80071410 2024-01-16T15:13:38.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa60003520 con 0x7faa80071410 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2260761963 0 0) 0x7faa60003520 con 0x7faa80071410 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 >> v2:172.21.15.146:3300/0 conn(0x7faa80146770 msgr2=0x7faa8006c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 --2- 172.21.15.53:0/312931069 >> v2:172.21.15.146:3300/0 conn(0x7faa80146770 0x7faa8006c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 >> v1:172.21.15.146:6789/0 conn(0x7faa80071f00 legacy=0x7faa80164ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8006d1a0 con 0x7faa80071410 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7faa70001670 con 0x7faa80071410 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 >> v1:172.21.15.53:6789/0 conn(0x7faa80071410 legacy=0x7faa800717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa60003970 0x7faa60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa600047c0 0x7faa60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa60005180 0x7faa6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8006cec0 con 0x7faa60003970 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6000c120 con 0x7faa600047c0 2024-01-16T15:13:38.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6000c2e0 con 0x7faa60005180 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.282+0000 7faa7f400640 1 --2- 172.21.15.53:0/312931069 >> v2:172.21.15.146:3300/0 conn(0x7faa80146770 0x7faa8006c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7ea00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa60003970 0x7faa60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa77e00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa600047c0 0x7faa60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7ea00640 1 -- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa60005180 msgr2=0x7faa6000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7ea00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa60005180 0x7faa6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7ea00640 1 -- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa600047c0 msgr2=0x7faa60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7ea00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa600047c0 0x7faa60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7ea00640 1 -- 172.21.15.53:0/312931069 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa60003520 con 0x7faa60003970 2024-01-16T15:13:38.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.283+0000 7faa7f400640 1 -- 172.21.15.53:0/312931069 reap_dead start 2024-01-16T15:13:38.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.284+0000 7faa7ea00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa60003970 0x7faa60003d90 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7faa7000c240 tx=0x7faa600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629f83cca28f6c05 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:38.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.284+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70005ad0 con 0x7faa60003970 2024-01-16T15:13:38.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa70005c30 con 0x7faa60003970 2024-01-16T15:13:38.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7e000640 1 -- 172.21.15.53:0/312931069 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7002cab0 con 0x7faa60003970 2024-01-16T15:13:38.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7fe00640 1 -- 172.21.15.53:0/312931069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa60003970 msgr2=0x7faa60003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7fe00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa60003970 0x7faa60003d90 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7faa7000c240 tx=0x7faa600057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7fe00640 1 -- 172.21.15.53:0/312931069 shutdown_connections 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7fe00640 1 --2- 172.21.15.53:0/312931069 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa60003970 0x7faa60003d90 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7fe00640 1 -- 172.21.15.53:0/312931069 >> 172.21.15.53:0/312931069 conn(0x7faa8006a800 msgr2=0x7faa8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.285+0000 7faa7fe00640 1 -- 172.21.15.53:0/312931069 shutdown_connections 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.286+0000 7faa7fe00640 1 -- 172.21.15.53:0/312931069 wait complete. 2024-01-16T15:13:38.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.288+0000 7faa7fe00640 1 Processor -- start 2024-01-16T15:13:38.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.288+0000 7faa7fe00640 1 -- start start 2024-01-16T15:13:38.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa60003970 0x7faa800cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 0x7faa801f3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa80144090 0x7faa80144510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa7fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa80144a50 con 0x7faa801f3b50 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa7fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa800cdd90 con 0x7faa60003970 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa80142ff0 con 0x7faa80144090 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa77e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 0x7faa801f3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.289+0000 7faa77e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 0x7faa801f3fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34054/0 (socket says 172.21.15.53:34054) 2024-01-16T15:13:38.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 -- 172.21.15.53:0/3137421762 learned_addr learned my addr 172.21.15.53:0/3137421762 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:38.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa7ea00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa60003970 0x7faa800cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa7f400640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa80144090 0x7faa80144510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 -- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa80144090 msgr2=0x7faa80144510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa80144090 0x7faa80144510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 -- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa60003970 msgr2=0x7faa800cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa60003970 0x7faa800cd850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 -- 172.21.15.53:0/3137421762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa801432d0 con 0x7faa801f3b50 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.290+0000 7faa77e00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 0x7faa801f3fd0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7faa6800e4b0 tx=0x7faa6800e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.291+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa680052e0 con 0x7faa801f3b50 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.291+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa68005440 con 0x7faa801f3b50 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.291+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa68003810 con 0x7faa801f3b50 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.291+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa801435b0 con 0x7faa801f3b50 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.291+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa80143b70 con 0x7faa801f3b50 2024-01-16T15:13:38.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.292+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa80143d30 con 0x7faa801f3b50 2024-01-16T15:13:38.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.293+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7faa68003970 con 0x7faa801f3b50 2024-01-16T15:13:38.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.294+0000 7faa76a00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa5806bf50 0x7faa5806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.294+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7faa68088340 con 0x7faa801f3b50 2024-01-16T15:13:38.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.295+0000 7faa7ea00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa5806bf50 0x7faa5806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.297+0000 7faa7ea00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa5806bf50 0x7faa5806e410 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7faa70034000 tx=0x7faa70009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:38.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.299+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faa68054c00 con 0x7faa801f3b50 2024-01-16T15:13:38.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.496+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa80067450 con 0x7faa801f3b50 2024-01-16T15:13:38.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.498+0000 7faa76a00640 1 -- 172.21.15.53:0/3137421762 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+16567 (secure 0 0 0) 0x7faa68058220 con 0x7faa801f3b50 2024-01-16T15:13:38.479 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:38.479 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":86,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:13:35.517353+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:13:31.825905+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2024-01-16T15:13:32.988111+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa5806bf50 msgr2=0x7faa5806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa5806bf50 0x7faa5806e410 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7faa70034000 tx=0x7faa70009030 comp rx=0 tx=0).stop 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 msgr2=0x7faa801f3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 0x7faa801f3fd0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7faa6800e4b0 tx=0x7faa6800e970 comp rx=0 tx=0).stop 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 shutdown_connections 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faa5806bf50 0x7faa5806e410 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faa80144090 0x7faa80144510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faa801f3b50 0x7faa801f3fd0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 --2- 172.21.15.53:0/3137421762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faa60003970 0x7faa800cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 >> 172.21.15.53:0/3137421762 conn(0x7faa8006a800 msgr2=0x7faa800b3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 shutdown_connections 2024-01-16T15:13:38.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.501+0000 7faa7fe00640 1 -- 172.21.15.53:0/3137421762 wait complete. 2024-01-16T15:13:38.501 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:13:38.501 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:13:38.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.747+0000 7fc942200640 1 Processor -- start 2024-01-16T15:13:38.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.747+0000 7fc942200640 1 -- start start 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.748+0000 7fc942200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc93c160920 0x7fc93c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.748+0000 7fc942200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc93c0cf9f0 0x7fc93c0cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.748+0000 7fc942200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c161010 0x7fc93c1614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.748+0000 7fc942200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c0d03e0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.748+0000 7fc942200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc93c0d0a10 con 0x7fc93c160920 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.748+0000 7fc942200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0cf080 con 0x7fc93c161010 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc93c0cf9f0 0x7fc93c0cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc93c0cf9f0 0x7fc93c0cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34068/0 (socket says 172.21.15.53:34068) 2024-01-16T15:13:38.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 -- 172.21.15.53:0/3877759649 learned_addr learned my addr 172.21.15.53:0/3877759649 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc940e00640 1 --2- 172.21.15.53:0/3877759649 >> v2:172.21.15.146:3300/0 conn(0x7fc93c160920 0x7fc93c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc941800640 1 --2- 172.21.15.53:0/3877759649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c161010 0x7fc93c1614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 -- 172.21.15.53:0/3877759649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c161010 msgr2=0x7fc93c1614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 --2- 172.21.15.53:0/3877759649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c161010 0x7fc93c1614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 -- 172.21.15.53:0/3877759649 >> v2:172.21.15.146:3300/0 conn(0x7fc93c160920 msgr2=0x7fc93c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.749+0000 7fc93be00640 1 --2- 172.21.15.53:0/3877759649 >> v2:172.21.15.146:3300/0 conn(0x7fc93c160920 0x7fc93c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.750+0000 7fc93be00640 1 -- 172.21.15.53:0/3877759649 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93c0c6be0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.751+0000 7fc93be00640 1 --2- 172.21.15.53:0/3877759649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0cfea0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fc93c08c060 tx=0x7fc92402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cd5c47c5984969 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:38.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc93b400640 1 -- 172.21.15.53:0/3877759649 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924031860 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc93b400640 1 -- 172.21.15.53:0/3877759649 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9240319c0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc93b400640 1 -- 172.21.15.53:0/3877759649 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc924031d50 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 -- 172.21.15.53:0/3877759649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 msgr2=0x7fc93c0cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 --2- 172.21.15.53:0/3877759649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0cfea0 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fc93c08c060 tx=0x7fc92402b900 comp rx=0 tx=0).stop 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 -- 172.21.15.53:0/3877759649 shutdown_connections 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 --2- 172.21.15.53:0/3877759649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c161010 0x7fc93c1614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 --2- 172.21.15.53:0/3877759649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0cfea0 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 --2- 172.21.15.53:0/3877759649 >> v2:172.21.15.146:3300/0 conn(0x7fc93c160920 0x7fc93c0b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.752+0000 7fc942200640 1 -- 172.21.15.53:0/3877759649 >> 172.21.15.53:0/3877759649 conn(0x7fc93c0d3090 msgr2=0x7fc93c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.753+0000 7fc942200640 1 -- 172.21.15.53:0/3877759649 shutdown_connections 2024-01-16T15:13:38.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.753+0000 7fc942200640 1 -- 172.21.15.53:0/3877759649 wait complete. 2024-01-16T15:13:38.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.755+0000 7fc942200640 1 Processor -- start 2024-01-16T15:13:38.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.756+0000 7fc942200640 1 -- start start 2024-01-16T15:13:38.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc942200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0b6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc942200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c160920 0x7fc93c0cab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc942200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc93c161010 0x7fc93c0cb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc942200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0b8c80 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc942200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0b6a00 con 0x7fc93c161010 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc942200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc93c0b5df0 con 0x7fc93c160920 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc940e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0b6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.757+0000 7fc940e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0b6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34084/0 (socket says 172.21.15.53:34084) 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 -- 172.21.15.53:0/2325442396 learned_addr learned my addr 172.21.15.53:0/2325442396 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc93be00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c160920 0x7fc93c0cab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc941800640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc93c161010 0x7fc93c0cb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 -- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c160920 msgr2=0x7fc93c0cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c160920 0x7fc93c0cab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 -- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc93c161010 msgr2=0x7fc93c0cb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc93c161010 0x7fc93c0cb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 -- 172.21.15.53:0/2325442396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9240094d0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc941800640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc93c161010 0x7fc93c0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.758+0000 7fc940e00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0b6e90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00d800 tx=0x7fc92c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.759+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c0096a0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.759+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc92c009800 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.759+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc92c009990 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.759+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc93c0b60d0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.759+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc93c0b6690 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.760+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc93c067450 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.741 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.761+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fc92c009af0 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.762+0000 7fc938c00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc91406bfa0 0x7fc91406e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.762+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc92c088c70 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.763+0000 7fc93be00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc91406bfa0 0x7fc91406e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:38.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.764+0000 7fc93be00640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc91406bfa0 0x7fc91406e460 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc924036000 tx=0x7fc92403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.767+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc92c055370 con 0x7fc93c0cf9f0 2024-01-16T15:13:38.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.956+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc93c0ceda0 con 0x7fc91406bfa0 2024-01-16T15:13:38.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.985+0000 7fc938c00640 1 -- 172.21.15.53:0/2325442396 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286974 (secure 0 0 0) 0x7fc924044400 con 0x7fc91406bfa0 2024-01-16T15:13:38.969 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:38.970 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc91406bfa0 msgr2=0x7fc91406e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc91406bfa0 0x7fc91406e460 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc924036000 tx=0x7fc92403c040 comp rx=0 tx=0).stop 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 msgr2=0x7fc93c0b6e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0b6e90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fc92c00d800 tx=0x7fc92c00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 shutdown_connections 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc91406bfa0 0x7fc91406e460 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc93c161010 0x7fc93c0cb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc93c160920 0x7fc93c0cab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 --2- 172.21.15.53:0/2325442396 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc93c0cf9f0 0x7fc93c0b6e90 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 >> 172.21.15.53:0/2325442396 conn(0x7fc93c0d3090 msgr2=0x7fc93c06b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:38.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 shutdown_connections 2024-01-16T15:13:38.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:38.990+0000 7fc942200640 1 -- 172.21.15.53:0/2325442396 wait complete. 2024-01-16T15:13:38.989 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-01-16T15:13:38.535993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":567864,"kb_used_data":9344,"kb_used_omap":12,"kb_used_meta":558387,"kb_avail":754406856,"statfs":{"total":773094113280,"available":772512620544,"internally_reserved":0,"allocated":9568256,"data_stored":4746524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":571788848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001871"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767799+0000","last_change":"2024-01-16T15:13:34.545290+0000","last_active":"2024-01-16T15:13:35.767799+0000","last_peered":"2024-01-16T15:13:35.767799+0000","last_clean":"2024-01-16T15:13:35.767799+0000","last_became_active":"2024-01-16T15:13:34.543199+0000","last_became_peered":"2024-01-16T15:13:34.543199+0000","last_unstale":"2024-01-16T15:13:35.767799+0000","last_undegraded":"2024-01-16T15:13:35.767799+0000","last_fullsized":"2024-01-16T15:13:35.767799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:37.806764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.547924+0000","last_change":"2024-01-16T15:13:34.545272+0000","last_active":"2024-01-16T15:13:34.547924+0000","last_peered":"2024-01-16T15:13:34.547924+0000","last_clean":"2024-01-16T15:13:34.547924+0000","last_became_active":"2024-01-16T15:13:34.545000+0000","last_became_peered":"2024-01-16T15:13:34.545000+0000","last_unstale":"2024-01-16T15:13:34.547924+0000","last_undegraded":"2024-01-16T15:13:34.547924+0000","last_fullsized":"2024-01-16T15:13:34.547924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767750+0000","last_change":"2024-01-16T15:13:34.537059+0000","last_active":"2024-01-16T15:13:35.767750+0000","last_peered":"2024-01-16T15:13:35.767750+0000","last_clean":"2024-01-16T15:13:35.767750+0000","last_became_active":"2024-01-16T15:13:34.536563+0000","last_became_peered":"2024-01-16T15:13:34.536563+0000","last_unstale":"2024-01-16T15:13:35.767750+0000","last_undegraded":"2024-01-16T15:13:35.767750+0000","last_fullsized":"2024-01-16T15:13:35.767750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:46.670118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.536459+0000","last_change":"2024-01-16T15:13:34.523132+0000","last_active":"2024-01-16T15:13:34.536459+0000","last_peered":"2024-01-16T15:13:34.536459+0000","last_clean":"2024-01-16T15:13:34.536459+0000","last_became_active":"2024-01-16T15:13:34.522837+0000","last_became_peered":"2024-01-16T15:13:34.522837+0000","last_unstale":"2024-01-16T15:13:34.536459+0000","last_undegraded":"2024-01-16T15:13:34.536459+0000","last_fullsized":"2024-01-16T15:13:34.536459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:28.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534568+0000","last_change":"2024-01-16T15:13:34.540849+0000","last_active":"2024-01-16T15:13:35.534568+0000","last_peered":"2024-01-16T15:13:35.534568+0000","last_clean":"2024-01-16T15:13:35.534568+0000","last_became_active":"2024-01-16T15:13:34.540445+0000","last_became_peered":"2024-01-16T15:13:34.540445+0000","last_unstale":"2024-01-16T15:13:35.534568+0000","last_undegraded":"2024-01-16T15:13:35.534568+0000","last_fullsized":"2024-01-16T15:13:35.534568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:20.893792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541464+0000","last_change":"2024-01-16T15:13:34.539749+0000","last_active":"2024-01-16T15:13:34.541464+0000","last_peered":"2024-01-16T15:13:34.541464+0000","last_clean":"2024-01-16T15:13:34.541464+0000","last_became_active":"2024-01-16T15:13:34.539435+0000","last_became_peered":"2024-01-16T15:13:34.539435+0000","last_unstale":"2024-01-16T15:13:34.541464+0000","last_undegraded":"2024-01-16T15:13:34.541464+0000","last_fullsized":"2024-01-16T15:13:34.541464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:00.822344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.527148+0000","last_change":"2024-01-16T15:13:34.523821+0000","last_active":"2024-01-16T15:13:34.527148+0000","last_peered":"2024-01-16T15:13:34.527148+0000","last_clean":"2024-01-16T15:13:34.527148+0000","last_became_active":"2024-01-16T15:13:34.521967+0000","last_became_peered":"2024-01-16T15:13:34.521967+0000","last_unstale":"2024-01-16T15:13:34.527148+0000","last_undegraded":"2024-01-16T15:13:34.527148+0000","last_fullsized":"2024-01-16T15:13:34.527148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:22.583003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.549233+0000","last_change":"2024-01-16T15:13:34.548663+0000","last_active":"2024-01-16T15:13:34.549233+0000","last_peered":"2024-01-16T15:13:34.549233+0000","last_clean":"2024-01-16T15:13:34.549233+0000","last_became_active":"2024-01-16T15:13:34.548191+0000","last_became_peered":"2024-01-16T15:13:34.548191+0000","last_unstale":"2024-01-16T15:13:34.549233+0000","last_undegraded":"2024-01-16T15:13:34.549233+0000","last_fullsized":"2024-01-16T15:13:34.549233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:33.008030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.546827+0000","last_change":"2024-01-16T15:13:34.543982+0000","last_active":"2024-01-16T15:13:34.546827+0000","last_peered":"2024-01-16T15:13:34.546827+0000","last_clean":"2024-01-16T15:13:34.546827+0000","last_became_active":"2024-01-16T15:13:34.543672+0000","last_became_peered":"2024-01-16T15:13:34.543672+0000","last_unstale":"2024-01-16T15:13:34.546827+0000","last_undegraded":"2024-01-16T15:13:34.546827+0000","last_fullsized":"2024-01-16T15:13:34.546827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:00:53.252197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766374+0000","last_change":"2024-01-16T15:13:34.540086+0000","last_active":"2024-01-16T15:13:35.766374+0000","last_peered":"2024-01-16T15:13:35.766374+0000","last_clean":"2024-01-16T15:13:35.766374+0000","last_became_active":"2024-01-16T15:13:34.539806+0000","last_became_peered":"2024-01-16T15:13:34.539806+0000","last_unstale":"2024-01-16T15:13:35.766374+0000","last_undegraded":"2024-01-16T15:13:35.766374+0000","last_fullsized":"2024-01-16T15:13:35.766374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:29.536665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.535782+0000","last_change":"2024-01-16T15:13:34.525762+0000","last_active":"2024-01-16T15:13:34.535782+0000","last_peered":"2024-01-16T15:13:34.535782+0000","last_clean":"2024-01-16T15:13:34.535782+0000","last_became_active":"2024-01-16T15:13:34.522701+0000","last_became_peered":"2024-01-16T15:13:34.522701+0000","last_unstale":"2024-01-16T15:13:34.535782+0000","last_undegraded":"2024-01-16T15:13:34.535782+0000","last_fullsized":"2024-01-16T15:13:34.535782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:07.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766553+0000","last_change":"2024-01-16T15:13:34.544139+0000","last_active":"2024-01-16T15:13:35.766553+0000","last_peered":"2024-01-16T15:13:35.766553+0000","last_clean":"2024-01-16T15:13:35.766553+0000","last_became_active":"2024-01-16T15:13:34.543821+0000","last_became_peered":"2024-01-16T15:13:34.543821+0000","last_unstale":"2024-01-16T15:13:35.766553+0000","last_undegraded":"2024-01-16T15:13:35.766553+0000","last_fullsized":"2024-01-16T15:13:35.766553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:58.712682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765415+0000","last_change":"2024-01-16T15:13:34.527660+0000","last_active":"2024-01-16T15:13:35.765415+0000","last_peered":"2024-01-16T15:13:35.765415+0000","last_clean":"2024-01-16T15:13:35.765415+0000","last_became_active":"2024-01-16T15:13:34.527307+0000","last_became_peered":"2024-01-16T15:13:34.527307+0000","last_unstale":"2024-01-16T15:13:35.765415+0000","last_undegraded":"2024-01-16T15:13:35.765415+0000","last_fullsized":"2024-01-16T15:13:35.765415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:22.853313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.527523+0000","last_change":"2024-01-16T15:13:34.524323+0000","last_active":"2024-01-16T15:13:34.527523+0000","last_peered":"2024-01-16T15:13:34.527523+0000","last_clean":"2024-01-16T15:13:34.527523+0000","last_became_active":"2024-01-16T15:13:34.523666+0000","last_became_peered":"2024-01-16T15:13:34.523666+0000","last_unstale":"2024-01-16T15:13:34.527523+0000","last_undegraded":"2024-01-16T15:13:34.527523+0000","last_fullsized":"2024-01-16T15:13:34.527523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:05:06.715542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.531383+0000","last_change":"2024-01-16T15:13:34.523713+0000","last_active":"2024-01-16T15:13:34.531383+0000","last_peered":"2024-01-16T15:13:34.531383+0000","last_clean":"2024-01-16T15:13:34.531383+0000","last_became_active":"2024-01-16T15:13:34.523413+0000","last_became_peered":"2024-01-16T15:13:34.523413+0000","last_unstale":"2024-01-16T15:13:34.531383+0000","last_undegraded":"2024-01-16T15:13:34.531383+0000","last_fullsized":"2024-01-16T15:13:34.531383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:50.133400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.533728+0000","last_change":"2024-01-16T15:13:34.534616+0000","last_active":"2024-01-16T15:13:35.533728+0000","last_peered":"2024-01-16T15:13:35.533728+0000","last_clean":"2024-01-16T15:13:35.533728+0000","last_became_active":"2024-01-16T15:13:34.534361+0000","last_became_peered":"2024-01-16T15:13:34.534361+0000","last_unstale":"2024-01-16T15:13:35.533728+0000","last_undegraded":"2024-01-16T15:13:35.533728+0000","last_fullsized":"2024-01-16T15:13:35.533728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:34.778254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535289+0000","last_change":"2024-01-16T15:13:34.532350+0000","last_active":"2024-01-16T15:13:35.535289+0000","last_peered":"2024-01-16T15:13:35.535289+0000","last_clean":"2024-01-16T15:13:35.535289+0000","last_became_active":"2024-01-16T15:13:34.532089+0000","last_became_peered":"2024-01-16T15:13:34.532089+0000","last_unstale":"2024-01-16T15:13:35.535289+0000","last_undegraded":"2024-01-16T15:13:35.535289+0000","last_fullsized":"2024-01-16T15:13:35.535289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:50.133400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.542306+0000","last_change":"2024-01-16T15:13:34.538435+0000","last_active":"2024-01-16T15:13:34.542306+0000","last_peered":"2024-01-16T15:13:34.542306+0000","last_clean":"2024-01-16T15:13:34.542306+0000","last_became_active":"2024-01-16T15:13:34.537395+0000","last_became_peered":"2024-01-16T15:13:34.537395+0000","last_unstale":"2024-01-16T15:13:34.542306+0000","last_undegraded":"2024-01-16T15:13:34.542306+0000","last_fullsized":"2024-01-16T15:13:34.542306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:04.014973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.530101+0000","last_change":"2024-01-16T15:13:34.523287+0000","last_active":"2024-01-16T15:13:34.530101+0000","last_peered":"2024-01-16T15:13:34.530101+0000","last_clean":"2024-01-16T15:13:34.530101+0000","last_became_active":"2024-01-16T15:13:34.522999+0000","last_became_peered":"2024-01-16T15:13:34.522999+0000","last_unstale":"2024-01-16T15:13:34.530101+0000","last_undegraded":"2024-01-16T15:13:34.530101+0000","last_fullsized":"2024-01-16T15:13:34.530101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:30.699519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.538911+0000","last_change":"2024-01-16T15:13:34.528842+0000","last_active":"2024-01-16T15:13:34.538911+0000","last_peered":"2024-01-16T15:13:34.538911+0000","last_clean":"2024-01-16T15:13:34.538911+0000","last_became_active":"2024-01-16T15:13:34.527918+0000","last_became_peered":"2024-01-16T15:13:34.527918+0000","last_unstale":"2024-01-16T15:13:34.538911+0000","last_undegraded":"2024-01-16T15:13:34.538911+0000","last_fullsized":"2024-01-16T15:13:34.538911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:23:59.803289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.537250+0000","last_change":"2024-01-16T15:13:34.525903+0000","last_active":"2024-01-16T15:13:34.537250+0000","last_peered":"2024-01-16T15:13:34.537250+0000","last_clean":"2024-01-16T15:13:34.537250+0000","last_became_active":"2024-01-16T15:13:34.525147+0000","last_became_peered":"2024-01-16T15:13:34.525147+0000","last_unstale":"2024-01-16T15:13:34.537250+0000","last_undegraded":"2024-01-16T15:13:34.537250+0000","last_fullsized":"2024-01-16T15:13:34.537250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:11.404862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.537842+0000","last_change":"2024-01-16T15:13:34.527480+0000","last_active":"2024-01-16T15:13:34.537842+0000","last_peered":"2024-01-16T15:13:34.537842+0000","last_clean":"2024-01-16T15:13:34.537842+0000","last_became_active":"2024-01-16T15:13:34.527212+0000","last_became_peered":"2024-01-16T15:13:34.527212+0000","last_unstale":"2024-01-16T15:13:34.537842+0000","last_undegraded":"2024-01-16T15:13:34.537842+0000","last_fullsized":"2024-01-16T15:13:34.537842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:47.601896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765988+0000","last_change":"2024-01-16T15:13:34.544238+0000","last_active":"2024-01-16T15:13:35.765988+0000","last_peered":"2024-01-16T15:13:35.765988+0000","last_clean":"2024-01-16T15:13:35.765988+0000","last_became_active":"2024-01-16T15:13:34.542808+0000","last_became_peered":"2024-01-16T15:13:34.542808+0000","last_unstale":"2024-01-16T15:13:35.765988+0000","last_undegraded":"2024-01-16T15:13:35.765988+0000","last_fullsized":"2024-01-16T15:13:35.765988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:07.045093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.540618+0000","last_change":"2024-01-16T15:13:34.526545+0000","last_active":"2024-01-16T15:13:34.540618+0000","last_peered":"2024-01-16T15:13:34.540618+0000","last_clean":"2024-01-16T15:13:34.540618+0000","last_became_active":"2024-01-16T15:13:34.526279+0000","last_became_peered":"2024-01-16T15:13:34.526279+0000","last_unstale":"2024-01-16T15:13:34.540618+0000","last_undegraded":"2024-01-16T15:13:34.540618+0000","last_fullsized":"2024-01-16T15:13:34.540618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:31.880133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.547465+0000","last_change":"2024-01-16T15:13:34.545237+0000","last_active":"2024-01-16T15:13:34.547465+0000","last_peered":"2024-01-16T15:13:34.547465+0000","last_clean":"2024-01-16T15:13:34.547465+0000","last_became_active":"2024-01-16T15:13:34.544951+0000","last_became_peered":"2024-01-16T15:13:34.544951+0000","last_unstale":"2024-01-16T15:13:34.547465+0000","last_undegraded":"2024-01-16T15:13:34.547465+0000","last_fullsized":"2024-01-16T15:13:34.547465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:03.318028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767078+0000","last_change":"2024-01-16T15:13:34.525675+0000","last_active":"2024-01-16T15:13:35.767078+0000","last_peered":"2024-01-16T15:13:35.767078+0000","last_clean":"2024-01-16T15:13:35.767078+0000","last_became_active":"2024-01-16T15:13:34.522097+0000","last_became_peered":"2024-01-16T15:13:34.522097+0000","last_unstale":"2024-01-16T15:13:35.767078+0000","last_undegraded":"2024-01-16T15:13:35.767078+0000","last_fullsized":"2024-01-16T15:13:35.767078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:41.963367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535518+0000","last_change":"2024-01-16T15:13:34.532689+0000","last_active":"2024-01-16T15:13:35.535518+0000","last_peered":"2024-01-16T15:13:35.535518+0000","last_clean":"2024-01-16T15:13:35.535518+0000","last_became_active":"2024-01-16T15:13:34.531781+0000","last_became_peered":"2024-01-16T15:13:34.531781+0000","last_unstale":"2024-01-16T15:13:35.535518+0000","last_undegraded":"2024-01-16T15:13:35.535518+0000","last_fullsized":"2024-01-16T15:13:35.535518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:04.049576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766776+0000","last_change":"2024-01-16T15:13:34.541541+0000","last_active":"2024-01-16T15:13:35.766776+0000","last_peered":"2024-01-16T15:13:35.766776+0000","last_clean":"2024-01-16T15:13:35.766776+0000","last_became_active":"2024-01-16T15:13:34.541063+0000","last_became_peered":"2024-01-16T15:13:34.541063+0000","last_unstale":"2024-01-16T15:13:35.766776+0000","last_undegraded":"2024-01-16T15:13:35.766776+0000","last_fullsized":"2024-01-16T15:13:35.766776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:31.164837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.535052+0000","last_change":"2024-01-16T15:13:34.520513+0000","last_active":"2024-01-16T15:13:34.535052+0000","last_peered":"2024-01-16T15:13:34.535052+0000","last_clean":"2024-01-16T15:13:34.535052+0000","last_became_active":"2024-01-16T15:13:34.520242+0000","last_became_peered":"2024-01-16T15:13:34.520242+0000","last_unstale":"2024-01-16T15:13:34.535052+0000","last_undegraded":"2024-01-16T15:13:34.535052+0000","last_fullsized":"2024-01-16T15:13:34.535052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:47.059884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.544039+0000","last_change":"2024-01-16T15:13:34.540319+0000","last_active":"2024-01-16T15:13:34.544039+0000","last_peered":"2024-01-16T15:13:34.544039+0000","last_clean":"2024-01-16T15:13:34.544039+0000","last_became_active":"2024-01-16T15:13:34.538070+0000","last_became_peered":"2024-01-16T15:13:34.538070+0000","last_unstale":"2024-01-16T15:13:34.544039+0000","last_undegraded":"2024-01-16T15:13:34.544039+0000","last_fullsized":"2024-01-16T15:13:34.544039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:00.611894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534213+0000","last_change":"2024-01-16T15:13:34.531319+0000","last_active":"2024-01-16T15:13:35.534213+0000","last_peered":"2024-01-16T15:13:35.534213+0000","last_clean":"2024-01-16T15:13:35.534213+0000","last_became_active":"2024-01-16T15:13:34.528875+0000","last_became_peered":"2024-01-16T15:13:34.528875+0000","last_unstale":"2024-01-16T15:13:35.534213+0000","last_undegraded":"2024-01-16T15:13:35.534213+0000","last_fullsized":"2024-01-16T15:13:35.534213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:21.873064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765582+0000","last_change":"2024-01-16T15:13:34.545148+0000","last_active":"2024-01-16T15:13:35.765582+0000","last_peered":"2024-01-16T15:13:35.765582+0000","last_clean":"2024-01-16T15:13:35.765582+0000","last_became_active":"2024-01-16T15:13:34.544511+0000","last_became_peered":"2024-01-16T15:13:34.544511+0000","last_unstale":"2024-01-16T15:13:35.765582+0000","last_undegraded":"2024-01-16T15:13:35.765582+0000","last_fullsized":"2024-01-16T15:13:35.765582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:41.744618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534956+0000","last_change":"2024-01-16T15:13:34.542671+0000","last_active":"2024-01-16T15:13:35.534956+0000","last_peered":"2024-01-16T15:13:35.534956+0000","last_clean":"2024-01-16T15:13:35.534956+0000","last_became_active":"2024-01-16T15:13:34.542316+0000","last_became_peered":"2024-01-16T15:13:34.542316+0000","last_unstale":"2024-01-16T15:13:35.534956+0000","last_undegraded":"2024-01-16T15:13:35.534956+0000","last_fullsized":"2024-01-16T15:13:35.534956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:57.577177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"85'4","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768452+0000","last_change":"2024-01-16T15:13:33.530176+0000","last_active":"2024-01-16T15:13:35.768452+0000","last_peered":"2024-01-16T15:13:35.768452+0000","last_clean":"2024-01-16T15:13:35.768452+0000","last_became_active":"2024-01-16T15:13:33.529751+0000","last_became_peered":"2024-01-16T15:13:33.529751+0000","last_unstale":"2024-01-16T15:13:35.768452+0000","last_undegraded":"2024-01-16T15:13:35.768452+0000","last_fullsized":"2024-01-16T15:13:35.768452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:53.868622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768380+0000","last_change":"2024-01-16T15:13:34.542801+0000","last_active":"2024-01-16T15:13:35.768380+0000","last_peered":"2024-01-16T15:13:35.768380+0000","last_clean":"2024-01-16T15:13:35.768380+0000","last_became_active":"2024-01-16T15:13:34.541292+0000","last_became_peered":"2024-01-16T15:13:34.541292+0000","last_unstale":"2024-01-16T15:13:35.768380+0000","last_undegraded":"2024-01-16T15:13:35.768380+0000","last_fullsized":"2024-01-16T15:13:35.768380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:01.069273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535556+0000","last_change":"2024-01-16T15:13:33.519828+0000","last_active":"2024-01-16T15:13:35.535556+0000","last_peered":"2024-01-16T15:13:35.535556+0000","last_clean":"2024-01-16T15:13:35.535556+0000","last_became_active":"2024-01-16T15:13:33.518840+0000","last_became_peered":"2024-01-16T15:13:33.518840+0000","last_unstale":"2024-01-16T15:13:35.535556+0000","last_undegraded":"2024-01-16T15:13:35.535556+0000","last_fullsized":"2024-01-16T15:13:35.535556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:20.248152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766743+0000","last_change":"2024-01-16T15:13:34.543173+0000","last_active":"2024-01-16T15:13:35.766743+0000","last_peered":"2024-01-16T15:13:35.766743+0000","last_clean":"2024-01-16T15:13:35.766743+0000","last_became_active":"2024-01-16T15:13:34.542884+0000","last_became_peered":"2024-01-16T15:13:34.542884+0000","last_unstale":"2024-01-16T15:13:35.766743+0000","last_undegraded":"2024-01-16T15:13:35.766743+0000","last_fullsized":"2024-01-16T15:13:35.766743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:07.382368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.543312+0000","last_change":"2024-01-16T15:13:33.521083+0000","last_active":"2024-01-16T15:13:34.543312+0000","last_peered":"2024-01-16T15:13:34.543312+0000","last_clean":"2024-01-16T15:13:34.543312+0000","last_became_active":"2024-01-16T15:13:33.520782+0000","last_became_peered":"2024-01-16T15:13:33.520782+0000","last_unstale":"2024-01-16T15:13:34.543312+0000","last_undegraded":"2024-01-16T15:13:34.543312+0000","last_fullsized":"2024-01-16T15:13:34.543312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:45.120651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767915+0000","last_change":"2024-01-16T15:13:34.529071+0000","last_active":"2024-01-16T15:13:35.767915+0000","last_peered":"2024-01-16T15:13:35.767915+0000","last_clean":"2024-01-16T15:13:35.767915+0000","last_became_active":"2024-01-16T15:13:34.528798+0000","last_became_peered":"2024-01-16T15:13:34.528798+0000","last_unstale":"2024-01-16T15:13:35.767915+0000","last_undegraded":"2024-01-16T15:13:35.767915+0000","last_fullsized":"2024-01-16T15:13:35.767915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:41.488465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.543661+0000","last_change":"2024-01-16T15:13:33.520318+0000","last_active":"2024-01-16T15:13:34.543661+0000","last_peered":"2024-01-16T15:13:34.543661+0000","last_clean":"2024-01-16T15:13:34.543661+0000","last_became_active":"2024-01-16T15:13:33.520021+0000","last_became_peered":"2024-01-16T15:13:33.520021+0000","last_unstale":"2024-01-16T15:13:34.543661+0000","last_undegraded":"2024-01-16T15:13:34.543661+0000","last_fullsized":"2024-01-16T15:13:34.543661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:13.718681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541031+0000","last_change":"2024-01-16T15:13:34.537955+0000","last_active":"2024-01-16T15:13:34.541031+0000","last_peered":"2024-01-16T15:13:34.541031+0000","last_clean":"2024-01-16T15:13:34.541031+0000","last_became_active":"2024-01-16T15:13:34.536276+0000","last_became_peered":"2024-01-16T15:13:34.536276+0000","last_unstale":"2024-01-16T15:13:34.541031+0000","last_undegraded":"2024-01-16T15:13:34.541031+0000","last_fullsized":"2024-01-16T15:13:34.541031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:18.852480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535043+0000","last_change":"2024-01-16T15:13:33.522513+0000","last_active":"2024-01-16T15:13:35.535043+0000","last_peered":"2024-01-16T15:13:35.535043+0000","last_clean":"2024-01-16T15:13:35.535043+0000","last_became_active":"2024-01-16T15:13:33.522250+0000","last_became_peered":"2024-01-16T15:13:33.522250+0000","last_unstale":"2024-01-16T15:13:35.535043+0000","last_undegraded":"2024-01-16T15:13:35.535043+0000","last_fullsized":"2024-01-16T15:13:35.535043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:00.053871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534935+0000","last_change":"2024-01-16T15:13:34.533080+0000","last_active":"2024-01-16T15:13:35.534935+0000","last_peered":"2024-01-16T15:13:35.534935+0000","last_clean":"2024-01-16T15:13:35.534935+0000","last_became_active":"2024-01-16T15:13:34.532798+0000","last_became_peered":"2024-01-16T15:13:34.532798+0000","last_unstale":"2024-01-16T15:13:35.534935+0000","last_undegraded":"2024-01-16T15:13:35.534935+0000","last_fullsized":"2024-01-16T15:13:35.534935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:27.153439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534070+0000","last_change":"2024-01-16T15:13:33.519705+0000","last_active":"2024-01-16T15:13:35.534070+0000","last_peered":"2024-01-16T15:13:35.534070+0000","last_clean":"2024-01-16T15:13:35.534070+0000","last_became_active":"2024-01-16T15:13:33.519228+0000","last_became_peered":"2024-01-16T15:13:33.519228+0000","last_unstale":"2024-01-16T15:13:35.534070+0000","last_undegraded":"2024-01-16T15:13:35.534070+0000","last_fullsized":"2024-01-16T15:13:35.534070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:05.029439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.763745+0000","last_change":"2024-01-16T15:13:34.528368+0000","last_active":"2024-01-16T15:13:35.763745+0000","last_peered":"2024-01-16T15:13:35.763745+0000","last_clean":"2024-01-16T15:13:35.763745+0000","last_became_active":"2024-01-16T15:13:34.523236+0000","last_became_peered":"2024-01-16T15:13:34.523236+0000","last_unstale":"2024-01-16T15:13:35.763745+0000","last_undegraded":"2024-01-16T15:13:35.763745+0000","last_fullsized":"2024-01-16T15:13:35.763745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:05:14.830516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.508824+0000","last_change":"2024-01-16T15:13:33.524369+0000","last_active":"2024-01-16T15:13:34.508824+0000","last_peered":"2024-01-16T15:13:34.508824+0000","last_clean":"2024-01-16T15:13:34.508824+0000","last_became_active":"2024-01-16T15:13:33.524095+0000","last_became_peered":"2024-01-16T15:13:33.524095+0000","last_unstale":"2024-01-16T15:13:34.508824+0000","last_undegraded":"2024-01-16T15:13:34.508824+0000","last_fullsized":"2024-01-16T15:13:34.508824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:04:29.927087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.529829+0000","last_change":"2024-01-16T15:13:34.521799+0000","last_active":"2024-01-16T15:13:34.529829+0000","last_peered":"2024-01-16T15:13:34.529829+0000","last_clean":"2024-01-16T15:13:34.529829+0000","last_became_active":"2024-01-16T15:13:34.521527+0000","last_became_peered":"2024-01-16T15:13:34.521527+0000","last_unstale":"2024-01-16T15:13:34.529829+0000","last_undegraded":"2024-01-16T15:13:34.529829+0000","last_fullsized":"2024-01-16T15:13:34.529829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:57.577177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.520422+0000","last_change":"2024-01-16T15:13:33.521927+0000","last_active":"2024-01-16T15:13:34.520422+0000","last_peered":"2024-01-16T15:13:34.520422+0000","last_clean":"2024-01-16T15:13:34.520422+0000","last_became_active":"2024-01-16T15:13:33.521614+0000","last_became_peered":"2024-01-16T15:13:33.521614+0000","last_unstale":"2024-01-16T15:13:34.520422+0000","last_undegraded":"2024-01-16T15:13:34.520422+0000","last_fullsized":"2024-01-16T15:13:34.520422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:02.543641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.531095+0000","last_change":"2024-01-16T15:13:34.523677+0000","last_active":"2024-01-16T15:13:34.531095+0000","last_peered":"2024-01-16T15:13:34.531095+0000","last_clean":"2024-01-16T15:13:34.531095+0000","last_became_active":"2024-01-16T15:13:34.522372+0000","last_became_peered":"2024-01-16T15:13:34.522372+0000","last_unstale":"2024-01-16T15:13:34.531095+0000","last_undegraded":"2024-01-16T15:13:34.531095+0000","last_fullsized":"2024-01-16T15:13:34.531095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:50.783073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768631+0000","last_change":"2024-01-16T15:13:33.528628+0000","last_active":"2024-01-16T15:13:35.768631+0000","last_peered":"2024-01-16T15:13:35.768631+0000","last_clean":"2024-01-16T15:13:35.768631+0000","last_became_active":"2024-01-16T15:13:33.528296+0000","last_became_peered":"2024-01-16T15:13:33.528296+0000","last_unstale":"2024-01-16T15:13:35.768631+0000","last_undegraded":"2024-01-16T15:13:35.768631+0000","last_fullsized":"2024-01-16T15:13:35.768631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:18:22.191352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.546847+0000","last_change":"2024-01-16T15:13:34.544063+0000","last_active":"2024-01-16T15:13:34.546847+0000","last_peered":"2024-01-16T15:13:34.546847+0000","last_clean":"2024-01-16T15:13:34.546847+0000","last_became_active":"2024-01-16T15:13:34.543766+0000","last_became_peered":"2024-01-16T15:13:34.543766+0000","last_unstale":"2024-01-16T15:13:34.546847+0000","last_undegraded":"2024-01-16T15:13:34.546847+0000","last_fullsized":"2024-01-16T15:13:34.546847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:57:52.593025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766614+0000","last_change":"2024-01-16T15:13:33.528015+0000","last_active":"2024-01-16T15:13:35.766614+0000","last_peered":"2024-01-16T15:13:35.766614+0000","last_clean":"2024-01-16T15:13:35.766614+0000","last_became_active":"2024-01-16T15:13:33.527704+0000","last_became_peered":"2024-01-16T15:13:33.527704+0000","last_unstale":"2024-01-16T15:13:35.766614+0000","last_undegraded":"2024-01-16T15:13:35.766614+0000","last_fullsized":"2024-01-16T15:13:35.766614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:35.320233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541672+0000","last_change":"2024-01-16T15:13:34.536915+0000","last_active":"2024-01-16T15:13:34.541672+0000","last_peered":"2024-01-16T15:13:34.541672+0000","last_clean":"2024-01-16T15:13:34.541672+0000","last_became_active":"2024-01-16T15:13:34.536002+0000","last_became_peered":"2024-01-16T15:13:34.536002+0000","last_unstale":"2024-01-16T15:13:34.541672+0000","last_undegraded":"2024-01-16T15:13:34.541672+0000","last_fullsized":"2024-01-16T15:13:34.541672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:13.879691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"85'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767111+0000","last_change":"2024-01-16T15:13:33.531079+0000","last_active":"2024-01-16T15:13:35.767111+0000","last_peered":"2024-01-16T15:13:35.767111+0000","last_clean":"2024-01-16T15:13:35.767111+0000","last_became_active":"2024-01-16T15:13:33.530816+0000","last_became_peered":"2024-01-16T15:13:33.530816+0000","last_unstale":"2024-01-16T15:13:35.767111+0000","last_undegraded":"2024-01-16T15:13:35.767111+0000","last_fullsized":"2024-01-16T15:13:35.767111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:30.955672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534671+0000","last_change":"2024-01-16T15:13:34.528387+0000","last_active":"2024-01-16T15:13:35.534671+0000","last_peered":"2024-01-16T15:13:35.534671+0000","last_clean":"2024-01-16T15:13:35.534671+0000","last_became_active":"2024-01-16T15:13:34.527521+0000","last_became_peered":"2024-01-16T15:13:34.527521+0000","last_unstale":"2024-01-16T15:13:35.534671+0000","last_undegraded":"2024-01-16T15:13:35.534671+0000","last_fullsized":"2024-01-16T15:13:35.534671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:36.949419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.550039+0000","last_change":"2024-01-16T15:13:33.523798+0000","last_active":"2024-01-16T15:13:34.550039+0000","last_peered":"2024-01-16T15:13:34.550039+0000","last_clean":"2024-01-16T15:13:34.550039+0000","last_became_active":"2024-01-16T15:13:33.523479+0000","last_became_peered":"2024-01-16T15:13:33.523479+0000","last_unstale":"2024-01-16T15:13:34.550039+0000","last_undegraded":"2024-01-16T15:13:34.550039+0000","last_fullsized":"2024-01-16T15:13:34.550039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:46.907808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.534875+0000","last_change":"2024-01-16T15:13:34.527176+0000","last_active":"2024-01-16T15:13:34.534875+0000","last_peered":"2024-01-16T15:13:34.534875+0000","last_clean":"2024-01-16T15:13:34.534875+0000","last_became_active":"2024-01-16T15:13:34.525697+0000","last_became_peered":"2024-01-16T15:13:34.525697+0000","last_unstale":"2024-01-16T15:13:34.534875+0000","last_undegraded":"2024-01-16T15:13:34.534875+0000","last_fullsized":"2024-01-16T15:13:34.534875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:41.528922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.543807+0000","last_change":"2024-01-16T15:13:33.529167+0000","last_active":"2024-01-16T15:13:34.543807+0000","last_peered":"2024-01-16T15:13:34.543807+0000","last_clean":"2024-01-16T15:13:34.543807+0000","last_became_active":"2024-01-16T15:13:33.528376+0000","last_became_peered":"2024-01-16T15:13:33.528376+0000","last_unstale":"2024-01-16T15:13:34.543807+0000","last_undegraded":"2024-01-16T15:13:34.543807+0000","last_fullsized":"2024-01-16T15:13:34.543807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:03.398824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765747+0000","last_change":"2024-01-16T15:13:34.525354+0000","last_active":"2024-01-16T15:13:35.765747+0000","last_peered":"2024-01-16T15:13:35.765747+0000","last_clean":"2024-01-16T15:13:35.765747+0000","last_became_active":"2024-01-16T15:13:34.521214+0000","last_became_peered":"2024-01-16T15:13:34.521214+0000","last_unstale":"2024-01-16T15:13:35.765747+0000","last_undegraded":"2024-01-16T15:13:35.765747+0000","last_fullsized":"2024-01-16T15:13:35.765747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:01.966387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.516322+0000","last_change":"2024-01-16T15:13:33.525041+0000","last_active":"2024-01-16T15:13:34.516322+0000","last_peered":"2024-01-16T15:13:34.516322+0000","last_clean":"2024-01-16T15:13:34.516322+0000","last_became_active":"2024-01-16T15:13:33.524778+0000","last_became_peered":"2024-01-16T15:13:33.524778+0000","last_unstale":"2024-01-16T15:13:34.516322+0000","last_undegraded":"2024-01-16T15:13:34.516322+0000","last_fullsized":"2024-01-16T15:13:34.516322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:14.655944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.542167+0000","last_change":"2024-01-16T15:13:34.537540+0000","last_active":"2024-01-16T15:13:34.542167+0000","last_peered":"2024-01-16T15:13:34.542167+0000","last_clean":"2024-01-16T15:13:34.542167+0000","last_became_active":"2024-01-16T15:13:34.537024+0000","last_became_peered":"2024-01-16T15:13:34.537024+0000","last_unstale":"2024-01-16T15:13:34.542167+0000","last_undegraded":"2024-01-16T15:13:34.542167+0000","last_fullsized":"2024-01-16T15:13:34.542167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:16.188469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.529011+0000","last_change":"2024-01-16T15:13:33.523785+0000","last_active":"2024-01-16T15:13:34.529011+0000","last_peered":"2024-01-16T15:13:34.529011+0000","last_clean":"2024-01-16T15:13:34.529011+0000","last_became_active":"2024-01-16T15:13:33.523508+0000","last_became_peered":"2024-01-16T15:13:33.523508+0000","last_unstale":"2024-01-16T15:13:34.529011+0000","last_undegraded":"2024-01-16T15:13:34.529011+0000","last_fullsized":"2024-01-16T15:13:34.529011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:10.155259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.548357+0000","last_change":"2024-01-16T15:13:34.546690+0000","last_active":"2024-01-16T15:13:34.548357+0000","last_peered":"2024-01-16T15:13:34.548357+0000","last_clean":"2024-01-16T15:13:34.548357+0000","last_became_active":"2024-01-16T15:13:34.546420+0000","last_became_peered":"2024-01-16T15:13:34.546420+0000","last_unstale":"2024-01-16T15:13:34.548357+0000","last_undegraded":"2024-01-16T15:13:34.548357+0000","last_fullsized":"2024-01-16T15:13:34.548357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:45.110834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.542052+0000","last_change":"2024-01-16T15:13:33.527398+0000","last_active":"2024-01-16T15:13:34.542052+0000","last_peered":"2024-01-16T15:13:34.542052+0000","last_clean":"2024-01-16T15:13:34.542052+0000","last_became_active":"2024-01-16T15:13:33.522768+0000","last_became_peered":"2024-01-16T15:13:33.522768+0000","last_unstale":"2024-01-16T15:13:34.542052+0000","last_undegraded":"2024-01-16T15:13:34.542052+0000","last_fullsized":"2024-01-16T15:13:34.542052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:27.998742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767423+0000","last_change":"2024-01-16T15:13:34.532655+0000","last_active":"2024-01-16T15:13:35.767423+0000","last_peered":"2024-01-16T15:13:35.767423+0000","last_clean":"2024-01-16T15:13:35.767423+0000","last_became_active":"2024-01-16T15:13:34.531680+0000","last_became_peered":"2024-01-16T15:13:34.531680+0000","last_unstale":"2024-01-16T15:13:35.767423+0000","last_undegraded":"2024-01-16T15:13:35.767423+0000","last_fullsized":"2024-01-16T15:13:35.767423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:39.654096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768721+0000","last_change":"2024-01-16T15:13:33.527895+0000","last_active":"2024-01-16T15:13:35.768721+0000","last_peered":"2024-01-16T15:13:35.768721+0000","last_clean":"2024-01-16T15:13:35.768721+0000","last_became_active":"2024-01-16T15:13:33.527518+0000","last_became_peered":"2024-01-16T15:13:33.527518+0000","last_unstale":"2024-01-16T15:13:35.768721+0000","last_undegraded":"2024-01-16T15:13:35.768721+0000","last_fullsized":"2024-01-16T15:13:35.768721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:08.292939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768319+0000","last_change":"2024-01-16T15:13:34.538478+0000","last_active":"2024-01-16T15:13:35.768319+0000","last_peered":"2024-01-16T15:13:35.768319+0000","last_clean":"2024-01-16T15:13:35.768319+0000","last_became_active":"2024-01-16T15:13:34.538115+0000","last_became_peered":"2024-01-16T15:13:34.538115+0000","last_unstale":"2024-01-16T15:13:35.768319+0000","last_undegraded":"2024-01-16T15:13:35.768319+0000","last_fullsized":"2024-01-16T15:13:35.768319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:21.263554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541177+0000","last_change":"2024-01-16T15:13:33.521965+0000","last_active":"2024-01-16T15:13:34.541177+0000","last_peered":"2024-01-16T15:13:34.541177+0000","last_clean":"2024-01-16T15:13:34.541177+0000","last_became_active":"2024-01-16T15:13:33.521662+0000","last_became_peered":"2024-01-16T15:13:33.521662+0000","last_unstale":"2024-01-16T15:13:34.541177+0000","last_undegraded":"2024-01-16T15:13:34.541177+0000","last_fullsized":"2024-01-16T15:13:34.541177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:05:42.740807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535050+0000","last_change":"2024-01-16T15:13:34.528570+0000","last_active":"2024-01-16T15:13:35.535050+0000","last_peered":"2024-01-16T15:13:35.535050+0000","last_clean":"2024-01-16T15:13:35.535050+0000","last_became_active":"2024-01-16T15:13:34.527865+0000","last_became_peered":"2024-01-16T15:13:34.527865+0000","last_unstale":"2024-01-16T15:13:35.535050+0000","last_undegraded":"2024-01-16T15:13:35.535050+0000","last_fullsized":"2024-01-16T15:13:35.535050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:50.783073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.509600+0000","last_change":"2024-01-16T15:13:33.525428+0000","last_active":"2024-01-16T15:13:34.509600+0000","last_peered":"2024-01-16T15:13:34.509600+0000","last_clean":"2024-01-16T15:13:34.509600+0000","last_became_active":"2024-01-16T15:13:33.525158+0000","last_became_peered":"2024-01-16T15:13:33.525158+0000","last_unstale":"2024-01-16T15:13:34.509600+0000","last_undegraded":"2024-01-16T15:13:34.509600+0000","last_fullsized":"2024-01-16T15:13:34.509600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:27.406092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.527856+0000","last_change":"2024-01-16T15:13:34.523930+0000","last_active":"2024-01-16T15:13:34.527856+0000","last_peered":"2024-01-16T15:13:34.527856+0000","last_clean":"2024-01-16T15:13:34.527856+0000","last_became_active":"2024-01-16T15:13:34.523288+0000","last_became_peered":"2024-01-16T15:13:34.523288+0000","last_unstale":"2024-01-16T15:13:34.527856+0000","last_undegraded":"2024-01-16T15:13:34.527856+0000","last_fullsized":"2024-01-16T15:13:34.527856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:47:03.133425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.528444+0000","last_change":"2024-01-16T15:13:33.518982+0000","last_active":"2024-01-16T15:13:34.528444+0000","last_peered":"2024-01-16T15:13:34.528444+0000","last_clean":"2024-01-16T15:13:34.528444+0000","last_became_active":"2024-01-16T15:13:33.518206+0000","last_became_peered":"2024-01-16T15:13:33.518206+0000","last_unstale":"2024-01-16T15:13:34.528444+0000","last_undegraded":"2024-01-16T15:13:34.528444+0000","last_fullsized":"2024-01-16T15:13:34.528444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:03.034174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.539645+0000","last_change":"2024-01-16T15:13:34.538866+0000","last_active":"2024-01-16T15:13:34.539645+0000","last_peered":"2024-01-16T15:13:34.539645+0000","last_clean":"2024-01-16T15:13:34.539645+0000","last_became_active":"2024-01-16T15:13:34.538399+0000","last_became_peered":"2024-01-16T15:13:34.538399+0000","last_unstale":"2024-01-16T15:13:34.539645+0000","last_undegraded":"2024-01-16T15:13:34.539645+0000","last_fullsized":"2024-01-16T15:13:34.539645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:22.516934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.516606+0000","last_change":"2024-01-16T15:13:33.521775+0000","last_active":"2024-01-16T15:13:34.516606+0000","last_peered":"2024-01-16T15:13:34.516606+0000","last_clean":"2024-01-16T15:13:34.516606+0000","last_became_active":"2024-01-16T15:13:33.521505+0000","last_became_peered":"2024-01-16T15:13:33.521505+0000","last_unstale":"2024-01-16T15:13:34.516606+0000","last_undegraded":"2024-01-16T15:13:34.516606+0000","last_fullsized":"2024-01-16T15:13:34.516606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:20.467490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.540825+0000","last_change":"2024-01-16T15:13:34.532979+0000","last_active":"2024-01-16T15:13:34.540825+0000","last_peered":"2024-01-16T15:13:34.540825+0000","last_clean":"2024-01-16T15:13:34.540825+0000","last_became_active":"2024-01-16T15:13:34.532168+0000","last_became_peered":"2024-01-16T15:13:34.532168+0000","last_unstale":"2024-01-16T15:13:34.540825+0000","last_undegraded":"2024-01-16T15:13:34.540825+0000","last_fullsized":"2024-01-16T15:13:34.540825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:36.245768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.512876+0000","last_change":"2024-01-16T15:13:33.528732+0000","last_active":"2024-01-16T15:13:34.512876+0000","last_peered":"2024-01-16T15:13:34.512876+0000","last_clean":"2024-01-16T15:13:34.512876+0000","last_became_active":"2024-01-16T15:13:33.528459+0000","last_became_peered":"2024-01-16T15:13:33.528459+0000","last_unstale":"2024-01-16T15:13:34.512876+0000","last_undegraded":"2024-01-16T15:13:34.512876+0000","last_fullsized":"2024-01-16T15:13:34.512876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:14.330799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.544920+0000","last_change":"2024-01-16T15:13:34.542151+0000","last_active":"2024-01-16T15:13:34.544920+0000","last_peered":"2024-01-16T15:13:34.544920+0000","last_clean":"2024-01-16T15:13:34.544920+0000","last_became_active":"2024-01-16T15:13:34.541865+0000","last_became_peered":"2024-01-16T15:13:34.541865+0000","last_unstale":"2024-01-16T15:13:34.544920+0000","last_undegraded":"2024-01-16T15:13:34.544920+0000","last_fullsized":"2024-01-16T15:13:34.544920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:18.116846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.513261+0000","last_change":"2024-01-16T15:13:33.518000+0000","last_active":"2024-01-16T15:13:34.513261+0000","last_peered":"2024-01-16T15:13:34.513261+0000","last_clean":"2024-01-16T15:13:34.513261+0000","last_became_active":"2024-01-16T15:13:33.516415+0000","last_became_peered":"2024-01-16T15:13:33.516415+0000","last_unstale":"2024-01-16T15:13:34.513261+0000","last_undegraded":"2024-01-16T15:13:34.513261+0000","last_fullsized":"2024-01-16T15:13:34.513261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:25.356804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535313+0000","last_change":"2024-01-16T15:13:34.534481+0000","last_active":"2024-01-16T15:13:35.535313+0000","last_peered":"2024-01-16T15:13:35.535313+0000","last_clean":"2024-01-16T15:13:35.535313+0000","last_became_active":"2024-01-16T15:13:34.534203+0000","last_became_peered":"2024-01-16T15:13:34.534203+0000","last_unstale":"2024-01-16T15:13:35.535313+0000","last_undegraded":"2024-01-16T15:13:35.535313+0000","last_fullsized":"2024-01-16T15:13:35.535313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:26.372207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.539469+0000","last_change":"2024-01-16T15:13:33.525384+0000","last_active":"2024-01-16T15:13:34.539469+0000","last_peered":"2024-01-16T15:13:34.539469+0000","last_clean":"2024-01-16T15:13:34.539469+0000","last_became_active":"2024-01-16T15:13:33.525117+0000","last_became_peered":"2024-01-16T15:13:33.525117+0000","last_unstale":"2024-01-16T15:13:34.539469+0000","last_undegraded":"2024-01-16T15:13:34.539469+0000","last_fullsized":"2024-01-16T15:13:34.539469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:24.897403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766709+0000","last_change":"2024-01-16T15:13:34.538260+0000","last_active":"2024-01-16T15:13:35.766709+0000","last_peered":"2024-01-16T15:13:35.766709+0000","last_clean":"2024-01-16T15:13:35.766709+0000","last_became_active":"2024-01-16T15:13:34.537908+0000","last_became_peered":"2024-01-16T15:13:34.537908+0000","last_unstale":"2024-01-16T15:13:35.766709+0000","last_undegraded":"2024-01-16T15:13:35.766709+0000","last_fullsized":"2024-01-16T15:13:35.766709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:26.900872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.521218+0000","last_change":"2024-01-16T15:13:33.528870+0000","last_active":"2024-01-16T15:13:34.521218+0000","last_peered":"2024-01-16T15:13:34.521218+0000","last_clean":"2024-01-16T15:13:34.521218+0000","last_became_active":"2024-01-16T15:13:33.528585+0000","last_became_peered":"2024-01-16T15:13:33.528585+0000","last_unstale":"2024-01-16T15:13:34.521218+0000","last_undegraded":"2024-01-16T15:13:34.521218+0000","last_fullsized":"2024-01-16T15:13:34.521218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:23.019368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766943+0000","last_change":"2024-01-16T15:13:34.543512+0000","last_active":"2024-01-16T15:13:35.766943+0000","last_peered":"2024-01-16T15:13:35.766943+0000","last_clean":"2024-01-16T15:13:35.766943+0000","last_became_active":"2024-01-16T15:13:34.541471+0000","last_became_peered":"2024-01-16T15:13:34.541471+0000","last_unstale":"2024-01-16T15:13:35.766943+0000","last_undegraded":"2024-01-16T15:13:35.766943+0000","last_fullsized":"2024-01-16T15:13:35.766943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:06.044841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767260+0000","last_change":"2024-01-16T15:13:33.527678+0000","last_active":"2024-01-16T15:13:35.767260+0000","last_peered":"2024-01-16T15:13:35.767260+0000","last_clean":"2024-01-16T15:13:35.767260+0000","last_became_active":"2024-01-16T15:13:33.527371+0000","last_became_peered":"2024-01-16T15:13:33.527371+0000","last_unstale":"2024-01-16T15:13:35.767260+0000","last_undegraded":"2024-01-16T15:13:35.767260+0000","last_fullsized":"2024-01-16T15:13:35.767260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.383745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535421+0000","last_change":"2024-01-16T15:13:34.528192+0000","last_active":"2024-01-16T15:13:35.535421+0000","last_peered":"2024-01-16T15:13:35.535421+0000","last_clean":"2024-01-16T15:13:35.535421+0000","last_became_active":"2024-01-16T15:13:34.527080+0000","last_became_peered":"2024-01-16T15:13:34.527080+0000","last_unstale":"2024-01-16T15:13:35.535421+0000","last_undegraded":"2024-01-16T15:13:35.535421+0000","last_fullsized":"2024-01-16T15:13:35.535421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:15.671346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767005+0000","last_change":"2024-01-16T15:13:33.523600+0000","last_active":"2024-01-16T15:13:35.767005+0000","last_peered":"2024-01-16T15:13:35.767005+0000","last_clean":"2024-01-16T15:13:35.767005+0000","last_became_active":"2024-01-16T15:13:33.523325+0000","last_became_peered":"2024-01-16T15:13:33.523325+0000","last_unstale":"2024-01-16T15:13:35.767005+0000","last_undegraded":"2024-01-16T15:13:35.767005+0000","last_fullsized":"2024-01-16T15:13:35.767005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:08.442154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.538655+0000","last_change":"2024-01-16T15:13:34.535307+0000","last_active":"2024-01-16T15:13:34.538655+0000","last_peered":"2024-01-16T15:13:34.538655+0000","last_clean":"2024-01-16T15:13:34.538655+0000","last_became_active":"2024-01-16T15:13:34.535022+0000","last_became_peered":"2024-01-16T15:13:34.535022+0000","last_unstale":"2024-01-16T15:13:34.538655+0000","last_undegraded":"2024-01-16T15:13:34.538655+0000","last_fullsized":"2024-01-16T15:13:34.538655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:40:20.618842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.508823+0000","last_change":"2024-01-16T15:13:33.523790+0000","last_active":"2024-01-16T15:13:34.508823+0000","last_peered":"2024-01-16T15:13:34.508823+0000","last_clean":"2024-01-16T15:13:34.508823+0000","last_became_active":"2024-01-16T15:13:33.523145+0000","last_became_peered":"2024-01-16T15:13:33.523145+0000","last_unstale":"2024-01-16T15:13:34.508823+0000","last_undegraded":"2024-01-16T15:13:34.508823+0000","last_fullsized":"2024-01-16T15:13:34.508823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:31.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.540462+0000","last_change":"2024-01-16T15:13:34.533505+0000","last_active":"2024-01-16T15:13:34.540462+0000","last_peered":"2024-01-16T15:13:34.540462+0000","last_clean":"2024-01-16T15:13:34.540462+0000","last_became_active":"2024-01-16T15:13:34.533229+0000","last_became_peered":"2024-01-16T15:13:34.533229+0000","last_unstale":"2024-01-16T15:13:34.540462+0000","last_undegraded":"2024-01-16T15:13:34.540462+0000","last_fullsized":"2024-01-16T15:13:34.540462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:57:27.301534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534260+0000","last_change":"2024-01-16T15:13:33.520685+0000","last_active":"2024-01-16T15:13:35.534260+0000","last_peered":"2024-01-16T15:13:35.534260+0000","last_clean":"2024-01-16T15:13:35.534260+0000","last_became_active":"2024-01-16T15:13:33.517416+0000","last_became_peered":"2024-01-16T15:13:33.517416+0000","last_unstale":"2024-01-16T15:13:35.534260+0000","last_undegraded":"2024-01-16T15:13:35.534260+0000","last_fullsized":"2024-01-16T15:13:35.534260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:36.791362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.763901+0000","last_change":"2024-01-16T15:13:34.523371+0000","last_active":"2024-01-16T15:13:35.763901+0000","last_peered":"2024-01-16T15:13:35.763901+0000","last_clean":"2024-01-16T15:13:35.763901+0000","last_became_active":"2024-01-16T15:13:34.522911+0000","last_became_peered":"2024-01-16T15:13:34.522911+0000","last_unstale":"2024-01-16T15:13:35.763901+0000","last_undegraded":"2024-01-16T15:13:35.763901+0000","last_fullsized":"2024-01-16T15:13:35.763901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:59.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766387+0000","last_change":"2024-01-16T15:13:33.523448+0000","last_active":"2024-01-16T15:13:35.766387+0000","last_peered":"2024-01-16T15:13:35.766387+0000","last_clean":"2024-01-16T15:13:35.766387+0000","last_became_active":"2024-01-16T15:13:33.523155+0000","last_became_peered":"2024-01-16T15:13:33.523155+0000","last_unstale":"2024-01-16T15:13:35.766387+0000","last_undegraded":"2024-01-16T15:13:35.766387+0000","last_fullsized":"2024-01-16T15:13:35.766387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:37.663483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766311+0000","last_change":"2024-01-16T15:13:34.546455+0000","last_active":"2024-01-16T15:13:35.766311+0000","last_peered":"2024-01-16T15:13:35.766311+0000","last_clean":"2024-01-16T15:13:35.766311+0000","last_became_active":"2024-01-16T15:13:34.545469+0000","last_became_peered":"2024-01-16T15:13:34.545469+0000","last_unstale":"2024-01-16T15:13:35.766311+0000","last_undegraded":"2024-01-16T15:13:35.766311+0000","last_fullsized":"2024-01-16T15:13:35.766311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:50.768273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.533639+0000","last_change":"2024-01-16T15:13:33.525656+0000","last_active":"2024-01-16T15:13:35.533639+0000","last_peered":"2024-01-16T15:13:35.533639+0000","last_clean":"2024-01-16T15:13:35.533639+0000","last_became_active":"2024-01-16T15:13:33.525375+0000","last_became_peered":"2024-01-16T15:13:33.525375+0000","last_unstale":"2024-01-16T15:13:35.533639+0000","last_undegraded":"2024-01-16T15:13:35.533639+0000","last_fullsized":"2024-01-16T15:13:35.533639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:57.697280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.547144+0000","last_change":"2024-01-16T15:13:34.541437+0000","last_active":"2024-01-16T15:13:34.547144+0000","last_peered":"2024-01-16T15:13:34.547144+0000","last_clean":"2024-01-16T15:13:34.547144+0000","last_became_active":"2024-01-16T15:13:34.541164+0000","last_became_peered":"2024-01-16T15:13:34.541164+0000","last_unstale":"2024-01-16T15:13:34.547144+0000","last_undegraded":"2024-01-16T15:13:34.547144+0000","last_fullsized":"2024-01-16T15:13:34.547144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:41.380286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767966+0000","last_change":"2024-01-16T15:13:33.522356+0000","last_active":"2024-01-16T15:13:35.767966+0000","last_peered":"2024-01-16T15:13:35.767966+0000","last_clean":"2024-01-16T15:13:35.767966+0000","last_became_active":"2024-01-16T15:13:33.522036+0000","last_became_peered":"2024-01-16T15:13:33.522036+0000","last_unstale":"2024-01-16T15:13:35.767966+0000","last_undegraded":"2024-01-16T15:13:35.767966+0000","last_fullsized":"2024-01-16T15:13:35.767966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:55.397809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":228,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.508239+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:13:34.508239+0000","last_peered":"2024-01-16T15:13:34.508239+0000","last_clean":"2024-01-16T15:13:34.508239+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:13:34.508239+0000","last_undegraded":"2024-01-16T15:13:34.508239+0000","last_fullsized":"2024-01-16T15:13:34.508239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509608,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68248,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303592,"statfs":{"total":96636764160,"available":96566878208,"internally_reserved":0,"allocated":1060864,"data_stored":450561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.6619999999999999,"15min":1.605},"min":{"1min":0.71399999999999997,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":8.8070000000000004,"5min":13.084,"15min":27.853999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":2.8559999999999999,"5min":2.327,"15min":1.704},"min":{"1min":0.70599999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":28.603999999999999,"5min":34.939999999999998,"15min":34.939999999999998},"last":1.278}]},{"osd":1,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.5620000000000001,"15min":1.4670000000000001},"min":{"1min":0.86299999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.5760000000000001,"5min":13.859,"15min":13.859},"last":1.1859999999999999},{"interface":"front","average":{"1min":3.198,"5min":2.0590000000000002,"15min":1.6319999999999999},"min":{"1min":0.81699999999999995,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.135}]},{"osd":2,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":1.8140000000000001,"15min":1.595},"min":{"1min":0.94299999999999995,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":9.6679999999999993,"5min":13.050000000000001,"15min":13.050000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":2.2149999999999999,"15min":1.7509999999999999},"min":{"1min":0.82799999999999996,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":1.972,"5min":35.011000000000003,"15min":35.011000000000003},"last":1.5349999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.7310000000000001,"15min":1.667},"min":{"1min":0.76500000000000001,"5min":0.439,"15min":0.374},"max":{"1min":1.579,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.867,"5min":1.9119999999999999,"15min":1.732},"min":{"1min":0.98499999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":9.7780000000000005,"5min":14.273999999999999,"15min":14.273999999999999},"last":1.583}]},{"osd":4,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":2.379,"15min":1.859},"min":{"1min":0.82299999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7649999999999999,"5min":34.972999999999999,"15min":34.972999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.8939999999999999,"15min":1.8200000000000001},"min":{"1min":0.875,"5min":0.627,"15min":0.40400000000000003},"max":{"1min":1.7290000000000001,"5min":16.003,"15min":16.003},"last":1.6930000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":2.1779999999999999,"15min":2.0329999999999999},"min":{"1min":1.1739999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.9059999999999999,"5min":23.760999999999999,"15min":23.760999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.954,"5min":2.2719999999999998,"15min":1.9159999999999999},"min":{"1min":0.90500000000000003,"5min":0.79300000000000004,"15min":0.52400000000000002},"max":{"1min":9.4120000000000008,"5min":23.716999999999999,"15min":23.716999999999999},"last":1.484}]},{"osd":7,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":2.532,"15min":2.0739999999999998},"min":{"1min":1.1739999999999999,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.831,"5min":34.889000000000003,"15min":34.889000000000003},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.569,"5min":2.1040000000000001,"15min":2.2010000000000001},"min":{"1min":1.2230000000000001,"5min":0.74099999999999999,"15min":0.47599999999999998},"max":{"1min":1.976,"5min":23.242999999999999,"15min":46.305},"last":1.748}]}]},{"osd":4,"up_from":15,"seq":64424509610,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64080,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307760,"statfs":{"total":96636764160,"available":96571146240,"internally_reserved":0,"allocated":1048576,"data_stored":449867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.5840000000000001,"15min":1.536},"min":{"1min":0.72799999999999998,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":3.617,"5min":22.757999999999999,"15min":22.757999999999999},"last":1.01},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.3200000000000001,"15min":1.282},"min":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.8640000000000001,"5min":14.034000000000001,"15min":14.034000000000001},"last":1.1060000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.5409999999999999,"15min":1.5700000000000001},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.7989999999999999,"5min":17.811,"15min":18.347000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.482,"15min":1.492},"min":{"1min":0.877,"5min":0.66500000000000004,"15min":0.498},"max":{"1min":2.3199999999999998,"5min":10.670999999999999,"15min":18.858000000000001},"last":1.3260000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.6950000000000001,"15min":1.675},"min":{"1min":0.59799999999999998,"5min":0.58299999999999996,"15min":0.56499999999999995},"max":{"1min":5.907,"5min":17.651,"15min":17.651},"last":1.843},{"interface":"front","average":{"1min":1.337,"5min":2.113,"15min":1.726},"min":{"1min":0.85499999999999998,"5min":0.68500000000000005,"15min":0.57699999999999996},"max":{"1min":1.8029999999999999,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.4630000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.419,"15min":1.5549999999999999},"min":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":2.1070000000000002,"5min":4.7329999999999997,"15min":26.741},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.627,"15min":1.796},"min":{"1min":0.82099999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.0379999999999998,"5min":17.715,"15min":26.701000000000001},"last":2.016}]},{"osd":5,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.702,"15min":1.734},"min":{"1min":1.0009999999999999,"5min":0.497,"15min":0.441},"max":{"1min":5.3170000000000002,"5min":10.763999999999999,"15min":18.919},"last":2.0680000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.542,"15min":1.673},"min":{"1min":0.89500000000000002,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":5.3520000000000003,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.9179999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":2.347,"15min":1.9410000000000001},"min":{"1min":0.92500000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.863,"5min":32.649999999999999,"15min":32.649999999999999},"last":2.323},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.867,"15min":1.8300000000000001},"min":{"1min":1.012,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":3.4289999999999998,"5min":24.094999999999999,"15min":24.094999999999999},"last":2.169}]},{"osd":7,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.8260000000000001,"15min":1.7689999999999999},"min":{"1min":1.117,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":5.835,"5min":17.446000000000002,"15min":26.683},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.8380000000000001,"5min":2.073,"15min":1.976},"min":{"1min":0.82599999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0079999999999991,"5min":17.765000000000001,"15min":17.765000000000001},"last":2.21}]}]},{"osd":7,"up_from":14,"seq":60129542312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68808,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303032,"statfs":{"total":96636764160,"available":96566304768,"internally_reserved":0,"allocated":1634304,"data_stored":1033977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5649999999999999,"15min":1.494},"min":{"1min":0.72499999999999998,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.6559999999999999,"5min":12.35,"15min":15.661},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.587,"5min":1.486,"15min":1.528},"min":{"1min":0.80500000000000005,"5min":0.73499999999999999,"15min":0.40300000000000002},"max":{"1min":12.461,"5min":12.461,"15min":30.126000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.708,"15min":1.4379999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.6659999999999999,"5min":24.338000000000001,"15min":24.338000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.754,"5min":1.5349999999999999,"15min":1.605},"min":{"1min":0.95799999999999996,"5min":0.66100000000000003,"15min":0.46200000000000002},"max":{"1min":12.526,"5min":12.526,"15min":21.298999999999999},"last":1.236}]},{"osd":2,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.5549999999999999,"15min":1.4950000000000001},"min":{"1min":0.82099999999999995,"5min":0.68899999999999995,"15min":0.51500000000000001},"max":{"1min":1.5720000000000001,"5min":12.474,"15min":12.474},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.224,"5min":2.089,"15min":1.643},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7210000000000001,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.355}]},{"osd":3,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.619,"15min":1.6000000000000001},"min":{"1min":1.006,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.8819999999999999,"5min":12.58,"15min":12.58},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":2.2349999999999999,"15min":1.768},"min":{"1min":0.629,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":1.7030000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":1.1319999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.51,"15min":1.597},"min":{"1min":0.93799999999999994,"5min":0.626,"15min":0.42799999999999999},"max":{"1min":4.1390000000000002,"5min":12.539,"15min":14.659000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3220000000000001,"5min":2.339,"15min":1.8140000000000001},"min":{"1min":0.58899999999999997,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":1.6870000000000001,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.486}]},{"osd":5,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.6599999999999999,"15min":1.8420000000000001},"min":{"1min":0.80800000000000005,"5min":0.66800000000000004,"15min":0.36699999999999999},"max":{"1min":12.609999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.748},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.8280000000000001,"15min":1.8400000000000001},"min":{"1min":0.76700000000000002,"5min":0.57999999999999996,"15min":0.41099999999999998},"max":{"1min":3.4159999999999999,"5min":12.836,"15min":21.312000000000001},"last":1.583}]},{"osd":6,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":2.29,"15min":1.861},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":1.8460000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.641},{"interface":"front","average":{"1min":1.976,"5min":1.7569999999999999,"15min":1.9630000000000001},"min":{"1min":0.80400000000000005,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":12.567,"5min":12.567,"15min":29.991},"last":1.845}]}]},{"osd":5,"up_from":14,"seq":60129542313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76560,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295280,"statfs":{"total":96636764160,"available":96558366720,"internally_reserved":0,"allocated":1052672,"data_stored":445173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.8460000000000001,"15min":1.534},"min":{"1min":0.68700000000000006,"5min":0.48799999999999999,"15min":0.42499999999999999},"max":{"1min":6.8380000000000001,"5min":19.382999999999999,"15min":19.382999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.2649999999999999,"15min":1.319},"min":{"1min":0.67500000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.649,"5min":3.698,"15min":11.109},"last":2.4790000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.8620000000000001,"15min":1.6040000000000001},"min":{"1min":0.64400000000000002,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":8.0909999999999993,"5min":17.052,"15min":17.693999999999999},"last":4.2450000000000001},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.9590000000000001,"15min":1.627},"min":{"1min":0.753,"5min":0.72599999999999998,"15min":0.432},"max":{"1min":2.3300000000000001,"5min":20.645,"15min":20.645},"last":0.56699999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":2.0249999999999999,"15min":1.667},"min":{"1min":0.70999999999999996,"5min":0.66500000000000004,"15min":0.48099999999999998},"max":{"1min":2.1320000000000001,"5min":21.138000000000002,"15min":21.138000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.4870000000000001,"15min":1.4610000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":2.21,"5min":4.0380000000000003,"15min":11.321},"last":2.2200000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.891,"15min":1.8029999999999999},"min":{"1min":0.69499999999999995,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":7.984,"5min":17.100999999999999,"15min":23.109000000000002},"last":1.329},{"interface":"front","average":{"1min":1.4330000000000001,"5min":2.0699999999999998,"15min":1.7909999999999999},"min":{"1min":0.73999999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0779999999999998,"5min":19.303000000000001,"15min":19.303000000000001},"last":0.65700000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.881,"15min":1.8160000000000001},"min":{"1min":0.72399999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":1.8080000000000001,"5min":17.143999999999998,"15min":24.436},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.968,"15min":1.744},"min":{"1min":0.59899999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":2.4569999999999999,"5min":19.126000000000001,"15min":19.126000000000001},"last":0.872}]},{"osd":6,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.478,"15min":1.5940000000000001},"min":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":2.524,"5min":6.5880000000000001,"15min":21.192},"last":2.2650000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":2.1920000000000002,"15min":1.9610000000000001},"min":{"1min":0.68500000000000005,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":1.7410000000000001,"5min":27.042000000000002,"15min":27.042000000000002},"last":2.105}]},{"osd":7,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.8819999999999999,"15min":1.907},"min":{"1min":0.88,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":2.4180000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":2.1579999999999999},{"interface":"front","average":{"1min":1.8819999999999999,"5min":2.1309999999999998,"15min":2.0990000000000002},"min":{"1min":1.0049999999999999,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":8.0380000000000003,"5min":16.939,"15min":24.367000000000001},"last":0.90100000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76556,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295284,"statfs":{"total":96636764160,"available":96558370816,"internally_reserved":0,"allocated":1048576,"data_stored":445151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.5069999999999999,"15min":1.248},"min":{"1min":0.44700000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":5.7400000000000002,"5min":28.437999999999999,"15min":31.562999999999999},"last":2.02},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.1679999999999999,"15min":1.133},"min":{"1min":0.40600000000000003,"5min":0.40100000000000002,"15min":0.39600000000000002},"max":{"1min":1.6879999999999999,"5min":11.361000000000001,"15min":11.361000000000001},"last":7.1429999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1319999999999999,"15min":1.089},"min":{"1min":0.53800000000000003,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.373,"5min":5.931,"15min":15.569000000000001},"last":10.856999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.7669999999999999,"15min":1.4079999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.36599999999999999},"max":{"1min":1.8129999999999999,"5min":28.352,"15min":28.352},"last":0.81100000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":3.294,"5min":2.3039999999999998,"15min":1.619},"min":{"1min":0.60699999999999998,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":39.265999999999998,"5min":39.265999999999998,"15min":39.265999999999998},"last":7.8159999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.4650000000000001,"15min":1.341},"min":{"1min":0.498,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":1.4590000000000001,"5min":13.526,"15min":14.577999999999999},"last":7.9160000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.5049999999999999,"15min":1.534},"min":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.53800000000000003},"max":{"1min":5.6890000000000001,"5min":6.0629999999999997,"15min":20.978999999999999},"last":4.5570000000000004},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.4390000000000001,"15min":1.8320000000000001},"min":{"1min":0.93400000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3330000000000002,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.976}]},{"osd":5,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9279999999999999,"15min":1.7509999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":6.3869999999999996,"5min":17.093,"15min":20.663},"last":4.5039999999999996},{"interface":"front","average":{"1min":2.0139999999999998,"5min":2.431,"15min":1.925},"min":{"1min":1.012,"5min":0.98499999999999999,"15min":0.76000000000000001},"max":{"1min":6.0679999999999996,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.925}]},{"osd":6,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.7869999999999999,"15min":1.7210000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":3.2149999999999999,"5min":13.691000000000001,"15min":13.691000000000001},"last":7.0060000000000002},{"interface":"front","average":{"1min":1.381,"5min":2.0710000000000002,"15min":1.8779999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":1.99,"5min":26.698,"15min":26.698},"last":4.6500000000000004}]},{"osd":7,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.9630000000000001,"15min":1.7250000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.829,"5min":13.622,"15min":13.622},"last":7.7750000000000004},{"interface":"front","average":{"1min":1.627,"5min":2.0819999999999999,"15min":2.133},"min":{"1min":0.88300000000000001,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.3180000000000001,"5min":15.372999999999999,"15min":49.811},"last":4.6029999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542312,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72396,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299444,"statfs":{"total":96636764160,"available":96562630656,"internally_reserved":0,"allocated":1040384,"data_stored":444121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.091,"15min":1.006},"min":{"1min":0.66600000000000004,"5min":0.42299999999999999,"15min":0.34200000000000003},"max":{"1min":1.9079999999999999,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.137,"15min":1.0780000000000001},"min":{"1min":0.73299999999999998,"5min":0.44500000000000001,"15min":0.28599999999999998},"max":{"1min":1.8240000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.623}]},{"osd":1,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.429,"15min":1.175},"min":{"1min":0.75600000000000001,"5min":0.50900000000000001,"15min":0.40300000000000002},"max":{"1min":3.056,"5min":23.135999999999999,"15min":23.135999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.2789999999999999,"15min":1.1870000000000001},"min":{"1min":0.82099999999999995,"5min":0.41499999999999998,"15min":0.41199999999999998},"max":{"1min":1.49,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":0.89700000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1180000000000001,"15min":1.071},"min":{"1min":0.66400000000000003,"5min":0.56699999999999995,"15min":0.34300000000000003},"max":{"1min":1.7070000000000001,"5min":4.0199999999999996,"15min":9.1530000000000005},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.486,"15min":1.391},"min":{"1min":0.92200000000000004,"5min":0.54300000000000004,"15min":0.45800000000000002},"max":{"1min":1.776,"5min":4.8380000000000001,"15min":4.9610000000000003},"last":1.389}]},{"osd":4,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.573,"15min":1.4850000000000001},"min":{"1min":0.79900000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":3.8180000000000001,"5min":21.837,"15min":21.837},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.742,"15min":1.5509999999999999},"min":{"1min":0.99399999999999999,"5min":0.79300000000000004,"15min":0.59199999999999997},"max":{"1min":2.9609999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.6830000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.6080000000000001,"15min":1.53},"min":{"1min":1.04,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":6.1879999999999997,"5min":17.841999999999999,"15min":17.841999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.597,"5min":1.5840000000000001,"15min":1.4990000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":5.9130000000000003,"5min":5.9130000000000003,"15min":9.423},"last":1.538}]},{"osd":6,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.875,"15min":1.6439999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.1059999999999999,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6319999999999999,"15min":1.716},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":21.564},"last":1.637}]},{"osd":7,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":2.0910000000000002,"15min":1.7130000000000001},"min":{"1min":1.0549999999999999,"5min":0.68300000000000005,"15min":0.64100000000000001},"max":{"1min":3.9020000000000001,"5min":22.834,"15min":22.834},"last":1.589},{"interface":"front","average":{"1min":1.5640000000000001,"5min":2.5249999999999999,"15min":2.0739999999999998},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.82799999999999996},"max":{"1min":2.4620000000000002,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.4990000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72404,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299436,"statfs":{"total":96636764160,"available":96562622464,"internally_reserved":0,"allocated":1048576,"data_stored":443733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":2.028,"15min":1.3049999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.619,"5min":31.178999999999998,"15min":31.178999999999998},"last":1.091},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.49,"15min":1.254},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.5029999999999999,"5min":26.859999999999999,"15min":26.859999999999999},"last":0.70199999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.121,"15min":1.111},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.377,"5min":3.4420000000000002,"15min":6.6120000000000001},"last":1.165},{"interface":"front","average":{"1min":1.234,"5min":1.8899999999999999,"15min":1.5169999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.0489999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":1.3660000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.9630000000000001,"15min":1.6279999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.837,"5min":27.609999999999999,"15min":27.609999999999999},"last":1.508},{"interface":"front","average":{"1min":1.1619999999999999,"5min":2.1960000000000002,"15min":1.629},"min":{"1min":0.51100000000000001,"5min":0.48999999999999999,"15min":0.36399999999999999},"max":{"1min":1.6719999999999999,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.657}]},{"osd":4,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4710000000000001,"15min":1.446},"min":{"1min":0.78400000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.2410000000000001,"5min":5.7590000000000003,"15min":10.577999999999999},"last":2.2879999999999998},{"interface":"front","average":{"1min":1.413,"5min":2.1480000000000001,"15min":1.736},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.1899999999999999,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.7390000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.4510000000000001,"15min":1.528},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.865,"5min":2.7879999999999998,"15min":17.469000000000001},"last":1.798},{"interface":"front","average":{"1min":1.47,"5min":1.6319999999999999,"15min":1.6040000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1539999999999999,"5min":5.8090000000000002,"15min":8.2989999999999995},"last":1.595}]},{"osd":6,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.6060000000000001,"15min":1.696},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0550000000000002,"5min":4.3780000000000001,"15min":17.428999999999998},"last":1.288},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.923,"15min":1.855},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1190000000000002,"5min":18.512,"15min":18.850999999999999},"last":1.869}]},{"osd":7,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.873,"15min":1.881},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.1150000000000002,"5min":15.885,"15min":30.834},"last":2.0179999999999998},{"interface":"front","average":{"1min":1.5489999999999999,"5min":2.492,"15min":2.0409999999999999},"min":{"1min":0.81100000000000005,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":2.3029999999999999,"5min":27.716999999999999,"15min":27.716999999999999},"last":2.2200000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542312,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68812,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303028,"statfs":{"total":96636764160,"available":96566300672,"internally_reserved":0,"allocated":1634304,"data_stored":1033941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.5680000000000001,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.434},{"interface":"front","average":{"1min":1.083,"5min":1.5229999999999999,"15min":1.4970000000000001},"min":{"1min":0.48699999999999999,"5min":0.40000000000000002,"15min":0.33900000000000002},"max":{"1min":1.5289999999999999,"5min":26.459,"15min":50.429000000000002},"last":0.61599999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.496,"15min":1.454},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.38},"max":{"1min":1.4470000000000001,"5min":20.041,"15min":22.716999999999999},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.722,"15min":1.607},"min":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.38},"max":{"1min":1.5640000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":0.77100000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.26,"15min":1.2170000000000001},"min":{"1min":0.52900000000000003,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":3.085,"5min":5.8719999999999999,"15min":10.795},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.5269999999999999,"15min":1.5760000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.837,"5min":9.484,"15min":22.254999999999999},"last":0.56499999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.7490000000000001,"15min":1.7929999999999999},"min":{"1min":0.79300000000000004,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.9319999999999999,"5min":21.300999999999998,"15min":22.407},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.873,"15min":1.907},"min":{"1min":0.72199999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.0459999999999998,"5min":26.57,"15min":50.497},"last":0.84299999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.5680000000000001,"15min":1.623},"min":{"1min":0.70699999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.8400000000000001,"5min":9.9860000000000007,"15min":14.112},"last":0.871},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.77,"15min":1.841},"min":{"1min":0.90800000000000003,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.8169999999999999,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.212}]},{"osd":6,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.639,"15min":1.7749999999999999},"min":{"1min":0.73399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.2970000000000002,"5min":8.2349999999999994,"15min":31.387},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.889,"15min":1.948},"min":{"1min":0.93799999999999994,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":1.948,"5min":9.2840000000000007,"15min":23.547000000000001},"last":0.97199999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8440000000000001,"5min":2.1419999999999999,"15min":2.0590000000000002},"min":{"1min":0.85299999999999998,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":9.6980000000000004,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.02},{"interface":"front","average":{"1min":1.516,"5min":1.9390000000000001,"15min":1.8340000000000001},"min":{"1min":0.76300000000000001,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.044,"5min":12.455,"15min":15.747},"last":0.94899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:13:38.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:13:39.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.277+0000 7ff0b1400640 1 Processor -- start 2024-01-16T15:13:39.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.277+0000 7ff0b1400640 1 -- start start 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.278+0000 7ff0b1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac0cc500 0x7ff0ac0cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.278+0000 7ff0b1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff0ac06d160 0x7ff0ac06d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.278+0000 7ff0b1400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0ac070e50 con 0x7ff0ac143c40 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.278+0000 7ff0b1400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0ac0723c0 con 0x7ff0ac06d160 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.278+0000 7ff0b1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac144020 con 0x7ff0ac0cc500 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff0ac06d160 0x7ff0ac06d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff0ac06d160 0x7ff0ac06d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34096/0 (socket says 172.21.15.53:34096) 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 -- 172.21.15.53:0/238308416 learned_addr learned my addr 172.21.15.53:0/238308416 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:39.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0abe00640 1 --1- 172.21.15.53:0/238308416 >> v1:172.21.15.53:6789/0 conn(0x7ff0ac143c40 0x7ff0ac071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55056/0 (socket says 172.21.15.53:55056) 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0ab400640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac0cc500 0x7ff0ac0cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 -- 172.21.15.53:0/238308416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac0cc500 msgr2=0x7ff0ac0cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac0cc500 0x7ff0ac0cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 -- 172.21.15.53:0/238308416 >> v1:172.21.15.53:6789/0 conn(0x7ff0ac143c40 legacy=0x7ff0ac071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0aaa00640 1 -- 172.21.15.53:0/238308416 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056135710 0 0) 0x7ff0ac070e50 con 0x7ff0ac143c40 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0b0a00640 1 -- 172.21.15.53:0/238308416 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac0ccea0 con 0x7ff0ac06d160 2024-01-16T15:13:39.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.279+0000 7ff0ab400640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac0cc500 0x7ff0ac0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:39.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.280+0000 7ff0b0a00640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac06d590 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009e50 tx=0x7ff0a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d7155cec1ac7d29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:39.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0aaa00640 1 -- 172.21.15.53:0/238308416 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a402ca70 con 0x7ff0ac06d160 2024-01-16T15:13:39.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0aaa00640 1 -- 172.21.15.53:0/238308416 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a402cbd0 con 0x7ff0ac06d160 2024-01-16T15:13:39.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0aaa00640 1 -- 172.21.15.53:0/238308416 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a40343f0 con 0x7ff0ac06d160 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0b1400640 1 -- 172.21.15.53:0/238308416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 msgr2=0x7ff0ac06d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0b1400640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac06d590 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009e50 tx=0x7ff0a402bed0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0b1400640 1 -- 172.21.15.53:0/238308416 shutdown_connections 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0b1400640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac06d590 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0b1400640 1 --2- 172.21.15.53:0/238308416 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac0cc500 0x7ff0ac0cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.281+0000 7ff0b1400640 1 -- 172.21.15.53:0/238308416 >> 172.21.15.53:0/238308416 conn(0x7ff0ac06a800 msgr2=0x7ff0ac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.282+0000 7ff0b1400640 1 -- 172.21.15.53:0/238308416 shutdown_connections 2024-01-16T15:13:39.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.282+0000 7ff0b1400640 1 -- 172.21.15.53:0/238308416 wait complete. 2024-01-16T15:13:39.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.283+0000 7ff0b1400640 1 Processor -- start 2024-01-16T15:13:39.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.283+0000 7ff0b1400640 1 -- start start 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.284+0000 7ff0b1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac13ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.284+0000 7ff0b1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ac0cc500 0x7ff0ac1edca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.284+0000 7ff0b1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac143c40 0x7ff0ac1eeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.284+0000 7ff0b1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac1ef9a0 con 0x7ff0ac06d160 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.284+0000 7ff0b1400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac1efc80 con 0x7ff0ac0cc500 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.284+0000 7ff0b1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0ac1eff60 con 0x7ff0ac143c40 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac13ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac13ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34106/0 (socket says 172.21.15.53:34106) 2024-01-16T15:13:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 -- 172.21.15.53:0/3646132364 learned_addr learned my addr 172.21.15.53:0/3646132364 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:39.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0b0a00640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac143c40 0x7ff0ac1eeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0ab400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ac0cc500 0x7ff0ac1edca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 -- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac143c40 msgr2=0x7ff0ac1eeea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac143c40 0x7ff0ac1eeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 -- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ac0cc500 msgr2=0x7ff0ac1edca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ac0cc500 0x7ff0ac1edca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 -- 172.21.15.53:0/3646132364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a4009b30 con 0x7ff0ac06d160 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0ab400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ac0cc500 0x7ff0ac1edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.285+0000 7ff0abe00640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac13ada0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00c4d0 tx=0x7ff09c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.286+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c0074e0 con 0x7ff0ac06d160 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.286+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff09c007660 con 0x7ff0ac06d160 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.286+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c004a20 con 0x7ff0ac06d160 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.286+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0ac1f0240 con 0x7ff0ac06d160 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.286+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0ac0a90f0 con 0x7ff0ac06d160 2024-01-16T15:13:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.287+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0ac1f0760 con 0x7ff0ac06d160 2024-01-16T15:13:39.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.288+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7ff09c004b80 con 0x7ff0ac06d160 2024-01-16T15:13:39.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.289+0000 7ff0a3e00640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff08406bf20 0x7ff08406e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.289+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7ff09c0879f0 con 0x7ff0ac06d160 2024-01-16T15:13:39.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.290+0000 7ff0ab400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff08406bf20 0x7ff08406e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.291+0000 7ff0ab400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff08406bf20 0x7ff08406e3e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff094006fd0 tx=0x7ff094008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.293+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff09c054090 con 0x7ff0ac06d160 2024-01-16T15:13:39.429 INFO:tasks.ceph.mds.c.smithi053.stderr:2024-01-16T15:13:39.448+0000 7fa363400640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:13:39.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.482+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0ac1efdf0 con 0x7ff08406bf20 2024-01-16T15:13:39.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.508+0000 7ff0a3e00640 1 -- 172.21.15.53:0/3646132364 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286974 (secure 0 0 0) 0x7ff0ac1efdf0 con 0x7ff08406bf20 2024-01-16T15:13:39.493 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:39.494 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:13:39.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff08406bf20 msgr2=0x7ff08406e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff08406bf20 0x7ff08406e3e0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7ff094006fd0 tx=0x7ff094008040 comp rx=0 tx=0).stop 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 msgr2=0x7ff0ac13ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac13ada0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00c4d0 tx=0x7ff09c00c990 comp rx=0 tx=0).stop 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 shutdown_connections 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff08406bf20 0x7ff08406e3e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff0ac143c40 0x7ff0ac1eeea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff0ac0cc500 0x7ff0ac1edca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 --2- 172.21.15.53:0/3646132364 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff0ac06d160 0x7ff0ac13ada0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.514+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 >> 172.21.15.53:0/3646132364 conn(0x7ff0ac06a800 msgr2=0x7ff0ac0cbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.515+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 shutdown_connections 2024-01-16T15:13:39.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.515+0000 7ff0b1400640 1 -- 172.21.15.53:0/3646132364 wait complete. 2024-01-16T15:13:39.513 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":496,"stamp":"2024-01-16T15:13:38.535993+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":96,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":567864,"kb_used_data":9344,"kb_used_omap":12,"kb_used_meta":558387,"kb_avail":754406856,"statfs":{"total":773094113280,"available":772512620544,"internally_reserved":0,"allocated":9568256,"data_stored":4746524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":571788848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001871"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767799+0000","last_change":"2024-01-16T15:13:34.545290+0000","last_active":"2024-01-16T15:13:35.767799+0000","last_peered":"2024-01-16T15:13:35.767799+0000","last_clean":"2024-01-16T15:13:35.767799+0000","last_became_active":"2024-01-16T15:13:34.543199+0000","last_became_peered":"2024-01-16T15:13:34.543199+0000","last_unstale":"2024-01-16T15:13:35.767799+0000","last_undegraded":"2024-01-16T15:13:35.767799+0000","last_fullsized":"2024-01-16T15:13:35.767799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:37.806764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.547924+0000","last_change":"2024-01-16T15:13:34.545272+0000","last_active":"2024-01-16T15:13:34.547924+0000","last_peered":"2024-01-16T15:13:34.547924+0000","last_clean":"2024-01-16T15:13:34.547924+0000","last_became_active":"2024-01-16T15:13:34.545000+0000","last_became_peered":"2024-01-16T15:13:34.545000+0000","last_unstale":"2024-01-16T15:13:34.547924+0000","last_undegraded":"2024-01-16T15:13:34.547924+0000","last_fullsized":"2024-01-16T15:13:34.547924+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.946806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767750+0000","last_change":"2024-01-16T15:13:34.537059+0000","last_active":"2024-01-16T15:13:35.767750+0000","last_peered":"2024-01-16T15:13:35.767750+0000","last_clean":"2024-01-16T15:13:35.767750+0000","last_became_active":"2024-01-16T15:13:34.536563+0000","last_became_peered":"2024-01-16T15:13:34.536563+0000","last_unstale":"2024-01-16T15:13:35.767750+0000","last_undegraded":"2024-01-16T15:13:35.767750+0000","last_fullsized":"2024-01-16T15:13:35.767750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:46.670118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.536459+0000","last_change":"2024-01-16T15:13:34.523132+0000","last_active":"2024-01-16T15:13:34.536459+0000","last_peered":"2024-01-16T15:13:34.536459+0000","last_clean":"2024-01-16T15:13:34.536459+0000","last_became_active":"2024-01-16T15:13:34.522837+0000","last_became_peered":"2024-01-16T15:13:34.522837+0000","last_unstale":"2024-01-16T15:13:34.536459+0000","last_undegraded":"2024-01-16T15:13:34.536459+0000","last_fullsized":"2024-01-16T15:13:34.536459+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:28.204740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534568+0000","last_change":"2024-01-16T15:13:34.540849+0000","last_active":"2024-01-16T15:13:35.534568+0000","last_peered":"2024-01-16T15:13:35.534568+0000","last_clean":"2024-01-16T15:13:35.534568+0000","last_became_active":"2024-01-16T15:13:34.540445+0000","last_became_peered":"2024-01-16T15:13:34.540445+0000","last_unstale":"2024-01-16T15:13:35.534568+0000","last_undegraded":"2024-01-16T15:13:35.534568+0000","last_fullsized":"2024-01-16T15:13:35.534568+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:20.893792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541464+0000","last_change":"2024-01-16T15:13:34.539749+0000","last_active":"2024-01-16T15:13:34.541464+0000","last_peered":"2024-01-16T15:13:34.541464+0000","last_clean":"2024-01-16T15:13:34.541464+0000","last_became_active":"2024-01-16T15:13:34.539435+0000","last_became_peered":"2024-01-16T15:13:34.539435+0000","last_unstale":"2024-01-16T15:13:34.541464+0000","last_undegraded":"2024-01-16T15:13:34.541464+0000","last_fullsized":"2024-01-16T15:13:34.541464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:00.822344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.527148+0000","last_change":"2024-01-16T15:13:34.523821+0000","last_active":"2024-01-16T15:13:34.527148+0000","last_peered":"2024-01-16T15:13:34.527148+0000","last_clean":"2024-01-16T15:13:34.527148+0000","last_became_active":"2024-01-16T15:13:34.521967+0000","last_became_peered":"2024-01-16T15:13:34.521967+0000","last_unstale":"2024-01-16T15:13:34.527148+0000","last_undegraded":"2024-01-16T15:13:34.527148+0000","last_fullsized":"2024-01-16T15:13:34.527148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:03:22.583003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.549233+0000","last_change":"2024-01-16T15:13:34.548663+0000","last_active":"2024-01-16T15:13:34.549233+0000","last_peered":"2024-01-16T15:13:34.549233+0000","last_clean":"2024-01-16T15:13:34.549233+0000","last_became_active":"2024-01-16T15:13:34.548191+0000","last_became_peered":"2024-01-16T15:13:34.548191+0000","last_unstale":"2024-01-16T15:13:34.549233+0000","last_undegraded":"2024-01-16T15:13:34.549233+0000","last_fullsized":"2024-01-16T15:13:34.549233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:56:33.008030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.546827+0000","last_change":"2024-01-16T15:13:34.543982+0000","last_active":"2024-01-16T15:13:34.546827+0000","last_peered":"2024-01-16T15:13:34.546827+0000","last_clean":"2024-01-16T15:13:34.546827+0000","last_became_active":"2024-01-16T15:13:34.543672+0000","last_became_peered":"2024-01-16T15:13:34.543672+0000","last_unstale":"2024-01-16T15:13:34.546827+0000","last_undegraded":"2024-01-16T15:13:34.546827+0000","last_fullsized":"2024-01-16T15:13:34.546827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:00:53.252197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766374+0000","last_change":"2024-01-16T15:13:34.540086+0000","last_active":"2024-01-16T15:13:35.766374+0000","last_peered":"2024-01-16T15:13:35.766374+0000","last_clean":"2024-01-16T15:13:35.766374+0000","last_became_active":"2024-01-16T15:13:34.539806+0000","last_became_peered":"2024-01-16T15:13:34.539806+0000","last_unstale":"2024-01-16T15:13:35.766374+0000","last_undegraded":"2024-01-16T15:13:35.766374+0000","last_fullsized":"2024-01-16T15:13:35.766374+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:53:29.536665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.535782+0000","last_change":"2024-01-16T15:13:34.525762+0000","last_active":"2024-01-16T15:13:34.535782+0000","last_peered":"2024-01-16T15:13:34.535782+0000","last_clean":"2024-01-16T15:13:34.535782+0000","last_became_active":"2024-01-16T15:13:34.522701+0000","last_became_peered":"2024-01-16T15:13:34.522701+0000","last_unstale":"2024-01-16T15:13:34.535782+0000","last_undegraded":"2024-01-16T15:13:34.535782+0000","last_fullsized":"2024-01-16T15:13:34.535782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:38:07.449909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766553+0000","last_change":"2024-01-16T15:13:34.544139+0000","last_active":"2024-01-16T15:13:35.766553+0000","last_peered":"2024-01-16T15:13:35.766553+0000","last_clean":"2024-01-16T15:13:35.766553+0000","last_became_active":"2024-01-16T15:13:34.543821+0000","last_became_peered":"2024-01-16T15:13:34.543821+0000","last_unstale":"2024-01-16T15:13:35.766553+0000","last_undegraded":"2024-01-16T15:13:35.766553+0000","last_fullsized":"2024-01-16T15:13:35.766553+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:58.712682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765415+0000","last_change":"2024-01-16T15:13:34.527660+0000","last_active":"2024-01-16T15:13:35.765415+0000","last_peered":"2024-01-16T15:13:35.765415+0000","last_clean":"2024-01-16T15:13:35.765415+0000","last_became_active":"2024-01-16T15:13:34.527307+0000","last_became_peered":"2024-01-16T15:13:34.527307+0000","last_unstale":"2024-01-16T15:13:35.765415+0000","last_undegraded":"2024-01-16T15:13:35.765415+0000","last_fullsized":"2024-01-16T15:13:35.765415+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:22.853313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.527523+0000","last_change":"2024-01-16T15:13:34.524323+0000","last_active":"2024-01-16T15:13:34.527523+0000","last_peered":"2024-01-16T15:13:34.527523+0000","last_clean":"2024-01-16T15:13:34.527523+0000","last_became_active":"2024-01-16T15:13:34.523666+0000","last_became_peered":"2024-01-16T15:13:34.523666+0000","last_unstale":"2024-01-16T15:13:34.527523+0000","last_undegraded":"2024-01-16T15:13:34.527523+0000","last_fullsized":"2024-01-16T15:13:34.527523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:05:06.715542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.531383+0000","last_change":"2024-01-16T15:13:34.523713+0000","last_active":"2024-01-16T15:13:34.531383+0000","last_peered":"2024-01-16T15:13:34.531383+0000","last_clean":"2024-01-16T15:13:34.531383+0000","last_became_active":"2024-01-16T15:13:34.523413+0000","last_became_peered":"2024-01-16T15:13:34.523413+0000","last_unstale":"2024-01-16T15:13:34.531383+0000","last_undegraded":"2024-01-16T15:13:34.531383+0000","last_fullsized":"2024-01-16T15:13:34.531383+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:50.133400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.533728+0000","last_change":"2024-01-16T15:13:34.534616+0000","last_active":"2024-01-16T15:13:35.533728+0000","last_peered":"2024-01-16T15:13:35.533728+0000","last_clean":"2024-01-16T15:13:35.533728+0000","last_became_active":"2024-01-16T15:13:34.534361+0000","last_became_peered":"2024-01-16T15:13:34.534361+0000","last_unstale":"2024-01-16T15:13:35.533728+0000","last_undegraded":"2024-01-16T15:13:35.533728+0000","last_fullsized":"2024-01-16T15:13:35.533728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:38:34.778254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535289+0000","last_change":"2024-01-16T15:13:34.532350+0000","last_active":"2024-01-16T15:13:35.535289+0000","last_peered":"2024-01-16T15:13:35.535289+0000","last_clean":"2024-01-16T15:13:35.535289+0000","last_became_active":"2024-01-16T15:13:34.532089+0000","last_became_peered":"2024-01-16T15:13:34.532089+0000","last_unstale":"2024-01-16T15:13:35.535289+0000","last_undegraded":"2024-01-16T15:13:35.535289+0000","last_fullsized":"2024-01-16T15:13:35.535289+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:50.133400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.542306+0000","last_change":"2024-01-16T15:13:34.538435+0000","last_active":"2024-01-16T15:13:34.542306+0000","last_peered":"2024-01-16T15:13:34.542306+0000","last_clean":"2024-01-16T15:13:34.542306+0000","last_became_active":"2024-01-16T15:13:34.537395+0000","last_became_peered":"2024-01-16T15:13:34.537395+0000","last_unstale":"2024-01-16T15:13:34.542306+0000","last_undegraded":"2024-01-16T15:13:34.542306+0000","last_fullsized":"2024-01-16T15:13:34.542306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:04.014973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.530101+0000","last_change":"2024-01-16T15:13:34.523287+0000","last_active":"2024-01-16T15:13:34.530101+0000","last_peered":"2024-01-16T15:13:34.530101+0000","last_clean":"2024-01-16T15:13:34.530101+0000","last_became_active":"2024-01-16T15:13:34.522999+0000","last_became_peered":"2024-01-16T15:13:34.522999+0000","last_unstale":"2024-01-16T15:13:34.530101+0000","last_undegraded":"2024-01-16T15:13:34.530101+0000","last_fullsized":"2024-01-16T15:13:34.530101+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:30.699519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.538911+0000","last_change":"2024-01-16T15:13:34.528842+0000","last_active":"2024-01-16T15:13:34.538911+0000","last_peered":"2024-01-16T15:13:34.538911+0000","last_clean":"2024-01-16T15:13:34.538911+0000","last_became_active":"2024-01-16T15:13:34.527918+0000","last_became_peered":"2024-01-16T15:13:34.527918+0000","last_unstale":"2024-01-16T15:13:34.538911+0000","last_undegraded":"2024-01-16T15:13:34.538911+0000","last_fullsized":"2024-01-16T15:13:34.538911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:23:59.803289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.537250+0000","last_change":"2024-01-16T15:13:34.525903+0000","last_active":"2024-01-16T15:13:34.537250+0000","last_peered":"2024-01-16T15:13:34.537250+0000","last_clean":"2024-01-16T15:13:34.537250+0000","last_became_active":"2024-01-16T15:13:34.525147+0000","last_became_peered":"2024-01-16T15:13:34.525147+0000","last_unstale":"2024-01-16T15:13:34.537250+0000","last_undegraded":"2024-01-16T15:13:34.537250+0000","last_fullsized":"2024-01-16T15:13:34.537250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:07:11.404862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.537842+0000","last_change":"2024-01-16T15:13:34.527480+0000","last_active":"2024-01-16T15:13:34.537842+0000","last_peered":"2024-01-16T15:13:34.537842+0000","last_clean":"2024-01-16T15:13:34.537842+0000","last_became_active":"2024-01-16T15:13:34.527212+0000","last_became_peered":"2024-01-16T15:13:34.527212+0000","last_unstale":"2024-01-16T15:13:34.537842+0000","last_undegraded":"2024-01-16T15:13:34.537842+0000","last_fullsized":"2024-01-16T15:13:34.537842+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:50:47.601896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765988+0000","last_change":"2024-01-16T15:13:34.544238+0000","last_active":"2024-01-16T15:13:35.765988+0000","last_peered":"2024-01-16T15:13:35.765988+0000","last_clean":"2024-01-16T15:13:35.765988+0000","last_became_active":"2024-01-16T15:13:34.542808+0000","last_became_peered":"2024-01-16T15:13:34.542808+0000","last_unstale":"2024-01-16T15:13:35.765988+0000","last_undegraded":"2024-01-16T15:13:35.765988+0000","last_fullsized":"2024-01-16T15:13:35.765988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:07.045093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.540618+0000","last_change":"2024-01-16T15:13:34.526545+0000","last_active":"2024-01-16T15:13:34.540618+0000","last_peered":"2024-01-16T15:13:34.540618+0000","last_clean":"2024-01-16T15:13:34.540618+0000","last_became_active":"2024-01-16T15:13:34.526279+0000","last_became_peered":"2024-01-16T15:13:34.526279+0000","last_unstale":"2024-01-16T15:13:34.540618+0000","last_undegraded":"2024-01-16T15:13:34.540618+0000","last_fullsized":"2024-01-16T15:13:34.540618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:33:31.880133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.547465+0000","last_change":"2024-01-16T15:13:34.545237+0000","last_active":"2024-01-16T15:13:34.547465+0000","last_peered":"2024-01-16T15:13:34.547465+0000","last_clean":"2024-01-16T15:13:34.547465+0000","last_became_active":"2024-01-16T15:13:34.544951+0000","last_became_peered":"2024-01-16T15:13:34.544951+0000","last_unstale":"2024-01-16T15:13:34.547465+0000","last_undegraded":"2024-01-16T15:13:34.547465+0000","last_fullsized":"2024-01-16T15:13:34.547465+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:26:03.318028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767078+0000","last_change":"2024-01-16T15:13:34.525675+0000","last_active":"2024-01-16T15:13:35.767078+0000","last_peered":"2024-01-16T15:13:35.767078+0000","last_clean":"2024-01-16T15:13:35.767078+0000","last_became_active":"2024-01-16T15:13:34.522097+0000","last_became_peered":"2024-01-16T15:13:34.522097+0000","last_unstale":"2024-01-16T15:13:35.767078+0000","last_undegraded":"2024-01-16T15:13:35.767078+0000","last_fullsized":"2024-01-16T15:13:35.767078+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:50:41.963367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535518+0000","last_change":"2024-01-16T15:13:34.532689+0000","last_active":"2024-01-16T15:13:35.535518+0000","last_peered":"2024-01-16T15:13:35.535518+0000","last_clean":"2024-01-16T15:13:35.535518+0000","last_became_active":"2024-01-16T15:13:34.531781+0000","last_became_peered":"2024-01-16T15:13:34.531781+0000","last_unstale":"2024-01-16T15:13:35.535518+0000","last_undegraded":"2024-01-16T15:13:35.535518+0000","last_fullsized":"2024-01-16T15:13:35.535518+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:04.049576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766776+0000","last_change":"2024-01-16T15:13:34.541541+0000","last_active":"2024-01-16T15:13:35.766776+0000","last_peered":"2024-01-16T15:13:35.766776+0000","last_clean":"2024-01-16T15:13:35.766776+0000","last_became_active":"2024-01-16T15:13:34.541063+0000","last_became_peered":"2024-01-16T15:13:34.541063+0000","last_unstale":"2024-01-16T15:13:35.766776+0000","last_undegraded":"2024-01-16T15:13:35.766776+0000","last_fullsized":"2024-01-16T15:13:35.766776+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:31.164837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.535052+0000","last_change":"2024-01-16T15:13:34.520513+0000","last_active":"2024-01-16T15:13:34.535052+0000","last_peered":"2024-01-16T15:13:34.535052+0000","last_clean":"2024-01-16T15:13:34.535052+0000","last_became_active":"2024-01-16T15:13:34.520242+0000","last_became_peered":"2024-01-16T15:13:34.520242+0000","last_unstale":"2024-01-16T15:13:34.535052+0000","last_undegraded":"2024-01-16T15:13:34.535052+0000","last_fullsized":"2024-01-16T15:13:34.535052+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:43:47.059884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.544039+0000","last_change":"2024-01-16T15:13:34.540319+0000","last_active":"2024-01-16T15:13:34.544039+0000","last_peered":"2024-01-16T15:13:34.544039+0000","last_clean":"2024-01-16T15:13:34.544039+0000","last_became_active":"2024-01-16T15:13:34.538070+0000","last_became_peered":"2024-01-16T15:13:34.538070+0000","last_unstale":"2024-01-16T15:13:34.544039+0000","last_undegraded":"2024-01-16T15:13:34.544039+0000","last_fullsized":"2024-01-16T15:13:34.544039+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:01:00.611894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534213+0000","last_change":"2024-01-16T15:13:34.531319+0000","last_active":"2024-01-16T15:13:35.534213+0000","last_peered":"2024-01-16T15:13:35.534213+0000","last_clean":"2024-01-16T15:13:35.534213+0000","last_became_active":"2024-01-16T15:13:34.528875+0000","last_became_peered":"2024-01-16T15:13:34.528875+0000","last_unstale":"2024-01-16T15:13:35.534213+0000","last_undegraded":"2024-01-16T15:13:35.534213+0000","last_fullsized":"2024-01-16T15:13:35.534213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:21.873064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765582+0000","last_change":"2024-01-16T15:13:34.545148+0000","last_active":"2024-01-16T15:13:35.765582+0000","last_peered":"2024-01-16T15:13:35.765582+0000","last_clean":"2024-01-16T15:13:35.765582+0000","last_became_active":"2024-01-16T15:13:34.544511+0000","last_became_peered":"2024-01-16T15:13:34.544511+0000","last_unstale":"2024-01-16T15:13:35.765582+0000","last_undegraded":"2024-01-16T15:13:35.765582+0000","last_fullsized":"2024-01-16T15:13:35.765582+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:21:41.744618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534956+0000","last_change":"2024-01-16T15:13:34.542671+0000","last_active":"2024-01-16T15:13:35.534956+0000","last_peered":"2024-01-16T15:13:35.534956+0000","last_clean":"2024-01-16T15:13:35.534956+0000","last_became_active":"2024-01-16T15:13:34.542316+0000","last_became_peered":"2024-01-16T15:13:34.542316+0000","last_unstale":"2024-01-16T15:13:35.534956+0000","last_undegraded":"2024-01-16T15:13:35.534956+0000","last_fullsized":"2024-01-16T15:13:35.534956+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:57.577177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"85'4","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768452+0000","last_change":"2024-01-16T15:13:33.530176+0000","last_active":"2024-01-16T15:13:35.768452+0000","last_peered":"2024-01-16T15:13:35.768452+0000","last_clean":"2024-01-16T15:13:35.768452+0000","last_became_active":"2024-01-16T15:13:33.529751+0000","last_became_peered":"2024-01-16T15:13:33.529751+0000","last_unstale":"2024-01-16T15:13:35.768452+0000","last_undegraded":"2024-01-16T15:13:35.768452+0000","last_fullsized":"2024-01-16T15:13:35.768452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:53.868622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768380+0000","last_change":"2024-01-16T15:13:34.542801+0000","last_active":"2024-01-16T15:13:35.768380+0000","last_peered":"2024-01-16T15:13:35.768380+0000","last_clean":"2024-01-16T15:13:35.768380+0000","last_became_active":"2024-01-16T15:13:34.541292+0000","last_became_peered":"2024-01-16T15:13:34.541292+0000","last_unstale":"2024-01-16T15:13:35.768380+0000","last_undegraded":"2024-01-16T15:13:35.768380+0000","last_fullsized":"2024-01-16T15:13:35.768380+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:01.069273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535556+0000","last_change":"2024-01-16T15:13:33.519828+0000","last_active":"2024-01-16T15:13:35.535556+0000","last_peered":"2024-01-16T15:13:35.535556+0000","last_clean":"2024-01-16T15:13:35.535556+0000","last_became_active":"2024-01-16T15:13:33.518840+0000","last_became_peered":"2024-01-16T15:13:33.518840+0000","last_unstale":"2024-01-16T15:13:35.535556+0000","last_undegraded":"2024-01-16T15:13:35.535556+0000","last_fullsized":"2024-01-16T15:13:35.535556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:20.248152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766743+0000","last_change":"2024-01-16T15:13:34.543173+0000","last_active":"2024-01-16T15:13:35.766743+0000","last_peered":"2024-01-16T15:13:35.766743+0000","last_clean":"2024-01-16T15:13:35.766743+0000","last_became_active":"2024-01-16T15:13:34.542884+0000","last_became_peered":"2024-01-16T15:13:34.542884+0000","last_unstale":"2024-01-16T15:13:35.766743+0000","last_undegraded":"2024-01-16T15:13:35.766743+0000","last_fullsized":"2024-01-16T15:13:35.766743+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:29:07.382368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.543312+0000","last_change":"2024-01-16T15:13:33.521083+0000","last_active":"2024-01-16T15:13:34.543312+0000","last_peered":"2024-01-16T15:13:34.543312+0000","last_clean":"2024-01-16T15:13:34.543312+0000","last_became_active":"2024-01-16T15:13:33.520782+0000","last_became_peered":"2024-01-16T15:13:33.520782+0000","last_unstale":"2024-01-16T15:13:34.543312+0000","last_undegraded":"2024-01-16T15:13:34.543312+0000","last_fullsized":"2024-01-16T15:13:34.543312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:22:45.120651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767915+0000","last_change":"2024-01-16T15:13:34.529071+0000","last_active":"2024-01-16T15:13:35.767915+0000","last_peered":"2024-01-16T15:13:35.767915+0000","last_clean":"2024-01-16T15:13:35.767915+0000","last_became_active":"2024-01-16T15:13:34.528798+0000","last_became_peered":"2024-01-16T15:13:34.528798+0000","last_unstale":"2024-01-16T15:13:35.767915+0000","last_undegraded":"2024-01-16T15:13:35.767915+0000","last_fullsized":"2024-01-16T15:13:35.767915+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:41.488465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.543661+0000","last_change":"2024-01-16T15:13:33.520318+0000","last_active":"2024-01-16T15:13:34.543661+0000","last_peered":"2024-01-16T15:13:34.543661+0000","last_clean":"2024-01-16T15:13:34.543661+0000","last_became_active":"2024-01-16T15:13:33.520021+0000","last_became_peered":"2024-01-16T15:13:33.520021+0000","last_unstale":"2024-01-16T15:13:34.543661+0000","last_undegraded":"2024-01-16T15:13:34.543661+0000","last_fullsized":"2024-01-16T15:13:34.543661+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:59:13.718681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541031+0000","last_change":"2024-01-16T15:13:34.537955+0000","last_active":"2024-01-16T15:13:34.541031+0000","last_peered":"2024-01-16T15:13:34.541031+0000","last_clean":"2024-01-16T15:13:34.541031+0000","last_became_active":"2024-01-16T15:13:34.536276+0000","last_became_peered":"2024-01-16T15:13:34.536276+0000","last_unstale":"2024-01-16T15:13:34.541031+0000","last_undegraded":"2024-01-16T15:13:34.541031+0000","last_fullsized":"2024-01-16T15:13:34.541031+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:18.852480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535043+0000","last_change":"2024-01-16T15:13:33.522513+0000","last_active":"2024-01-16T15:13:35.535043+0000","last_peered":"2024-01-16T15:13:35.535043+0000","last_clean":"2024-01-16T15:13:35.535043+0000","last_became_active":"2024-01-16T15:13:33.522250+0000","last_became_peered":"2024-01-16T15:13:33.522250+0000","last_unstale":"2024-01-16T15:13:35.535043+0000","last_undegraded":"2024-01-16T15:13:35.535043+0000","last_fullsized":"2024-01-16T15:13:35.535043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:35:00.053871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534935+0000","last_change":"2024-01-16T15:13:34.533080+0000","last_active":"2024-01-16T15:13:35.534935+0000","last_peered":"2024-01-16T15:13:35.534935+0000","last_clean":"2024-01-16T15:13:35.534935+0000","last_became_active":"2024-01-16T15:13:34.532798+0000","last_became_peered":"2024-01-16T15:13:34.532798+0000","last_unstale":"2024-01-16T15:13:35.534935+0000","last_undegraded":"2024-01-16T15:13:35.534935+0000","last_fullsized":"2024-01-16T15:13:35.534935+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:08:27.153439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534070+0000","last_change":"2024-01-16T15:13:33.519705+0000","last_active":"2024-01-16T15:13:35.534070+0000","last_peered":"2024-01-16T15:13:35.534070+0000","last_clean":"2024-01-16T15:13:35.534070+0000","last_became_active":"2024-01-16T15:13:33.519228+0000","last_became_peered":"2024-01-16T15:13:33.519228+0000","last_unstale":"2024-01-16T15:13:35.534070+0000","last_undegraded":"2024-01-16T15:13:35.534070+0000","last_fullsized":"2024-01-16T15:13:35.534070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:05.029439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.763745+0000","last_change":"2024-01-16T15:13:34.528368+0000","last_active":"2024-01-16T15:13:35.763745+0000","last_peered":"2024-01-16T15:13:35.763745+0000","last_clean":"2024-01-16T15:13:35.763745+0000","last_became_active":"2024-01-16T15:13:34.523236+0000","last_became_peered":"2024-01-16T15:13:34.523236+0000","last_unstale":"2024-01-16T15:13:35.763745+0000","last_undegraded":"2024-01-16T15:13:35.763745+0000","last_fullsized":"2024-01-16T15:13:35.763745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:05:14.830516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.508824+0000","last_change":"2024-01-16T15:13:33.524369+0000","last_active":"2024-01-16T15:13:34.508824+0000","last_peered":"2024-01-16T15:13:34.508824+0000","last_clean":"2024-01-16T15:13:34.508824+0000","last_became_active":"2024-01-16T15:13:33.524095+0000","last_became_peered":"2024-01-16T15:13:33.524095+0000","last_unstale":"2024-01-16T15:13:34.508824+0000","last_undegraded":"2024-01-16T15:13:34.508824+0000","last_fullsized":"2024-01-16T15:13:34.508824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:04:29.927087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.529829+0000","last_change":"2024-01-16T15:13:34.521799+0000","last_active":"2024-01-16T15:13:34.529829+0000","last_peered":"2024-01-16T15:13:34.529829+0000","last_clean":"2024-01-16T15:13:34.529829+0000","last_became_active":"2024-01-16T15:13:34.521527+0000","last_became_peered":"2024-01-16T15:13:34.521527+0000","last_unstale":"2024-01-16T15:13:34.529829+0000","last_undegraded":"2024-01-16T15:13:34.529829+0000","last_fullsized":"2024-01-16T15:13:34.529829+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:57.577177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.520422+0000","last_change":"2024-01-16T15:13:33.521927+0000","last_active":"2024-01-16T15:13:34.520422+0000","last_peered":"2024-01-16T15:13:34.520422+0000","last_clean":"2024-01-16T15:13:34.520422+0000","last_became_active":"2024-01-16T15:13:33.521614+0000","last_became_peered":"2024-01-16T15:13:33.521614+0000","last_unstale":"2024-01-16T15:13:34.520422+0000","last_undegraded":"2024-01-16T15:13:34.520422+0000","last_fullsized":"2024-01-16T15:13:34.520422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:46:02.543641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.531095+0000","last_change":"2024-01-16T15:13:34.523677+0000","last_active":"2024-01-16T15:13:34.531095+0000","last_peered":"2024-01-16T15:13:34.531095+0000","last_clean":"2024-01-16T15:13:34.531095+0000","last_became_active":"2024-01-16T15:13:34.522372+0000","last_became_peered":"2024-01-16T15:13:34.522372+0000","last_unstale":"2024-01-16T15:13:34.531095+0000","last_undegraded":"2024-01-16T15:13:34.531095+0000","last_fullsized":"2024-01-16T15:13:34.531095+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:50.783073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768631+0000","last_change":"2024-01-16T15:13:33.528628+0000","last_active":"2024-01-16T15:13:35.768631+0000","last_peered":"2024-01-16T15:13:35.768631+0000","last_clean":"2024-01-16T15:13:35.768631+0000","last_became_active":"2024-01-16T15:13:33.528296+0000","last_became_peered":"2024-01-16T15:13:33.528296+0000","last_unstale":"2024-01-16T15:13:35.768631+0000","last_undegraded":"2024-01-16T15:13:35.768631+0000","last_fullsized":"2024-01-16T15:13:35.768631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:18:22.191352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.546847+0000","last_change":"2024-01-16T15:13:34.544063+0000","last_active":"2024-01-16T15:13:34.546847+0000","last_peered":"2024-01-16T15:13:34.546847+0000","last_clean":"2024-01-16T15:13:34.546847+0000","last_became_active":"2024-01-16T15:13:34.543766+0000","last_became_peered":"2024-01-16T15:13:34.543766+0000","last_unstale":"2024-01-16T15:13:34.546847+0000","last_undegraded":"2024-01-16T15:13:34.546847+0000","last_fullsized":"2024-01-16T15:13:34.546847+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:57:52.593025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766614+0000","last_change":"2024-01-16T15:13:33.528015+0000","last_active":"2024-01-16T15:13:35.766614+0000","last_peered":"2024-01-16T15:13:35.766614+0000","last_clean":"2024-01-16T15:13:35.766614+0000","last_became_active":"2024-01-16T15:13:33.527704+0000","last_became_peered":"2024-01-16T15:13:33.527704+0000","last_unstale":"2024-01-16T15:13:35.766614+0000","last_undegraded":"2024-01-16T15:13:35.766614+0000","last_fullsized":"2024-01-16T15:13:35.766614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:29:35.320233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541672+0000","last_change":"2024-01-16T15:13:34.536915+0000","last_active":"2024-01-16T15:13:34.541672+0000","last_peered":"2024-01-16T15:13:34.541672+0000","last_clean":"2024-01-16T15:13:34.541672+0000","last_became_active":"2024-01-16T15:13:34.536002+0000","last_became_peered":"2024-01-16T15:13:34.536002+0000","last_unstale":"2024-01-16T15:13:34.541672+0000","last_undegraded":"2024-01-16T15:13:34.541672+0000","last_fullsized":"2024-01-16T15:13:34.541672+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:13.879691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"85'3","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767111+0000","last_change":"2024-01-16T15:13:33.531079+0000","last_active":"2024-01-16T15:13:35.767111+0000","last_peered":"2024-01-16T15:13:35.767111+0000","last_clean":"2024-01-16T15:13:35.767111+0000","last_became_active":"2024-01-16T15:13:33.530816+0000","last_became_peered":"2024-01-16T15:13:33.530816+0000","last_unstale":"2024-01-16T15:13:35.767111+0000","last_undegraded":"2024-01-16T15:13:35.767111+0000","last_fullsized":"2024-01-16T15:13:35.767111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:04:30.955672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534671+0000","last_change":"2024-01-16T15:13:34.528387+0000","last_active":"2024-01-16T15:13:35.534671+0000","last_peered":"2024-01-16T15:13:35.534671+0000","last_clean":"2024-01-16T15:13:35.534671+0000","last_became_active":"2024-01-16T15:13:34.527521+0000","last_became_peered":"2024-01-16T15:13:34.527521+0000","last_unstale":"2024-01-16T15:13:35.534671+0000","last_undegraded":"2024-01-16T15:13:35.534671+0000","last_fullsized":"2024-01-16T15:13:35.534671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:03:36.949419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.550039+0000","last_change":"2024-01-16T15:13:33.523798+0000","last_active":"2024-01-16T15:13:34.550039+0000","last_peered":"2024-01-16T15:13:34.550039+0000","last_clean":"2024-01-16T15:13:34.550039+0000","last_became_active":"2024-01-16T15:13:33.523479+0000","last_became_peered":"2024-01-16T15:13:33.523479+0000","last_unstale":"2024-01-16T15:13:34.550039+0000","last_undegraded":"2024-01-16T15:13:34.550039+0000","last_fullsized":"2024-01-16T15:13:34.550039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:05:46.907808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.534875+0000","last_change":"2024-01-16T15:13:34.527176+0000","last_active":"2024-01-16T15:13:34.534875+0000","last_peered":"2024-01-16T15:13:34.534875+0000","last_clean":"2024-01-16T15:13:34.534875+0000","last_became_active":"2024-01-16T15:13:34.525697+0000","last_became_peered":"2024-01-16T15:13:34.525697+0000","last_unstale":"2024-01-16T15:13:34.534875+0000","last_undegraded":"2024-01-16T15:13:34.534875+0000","last_fullsized":"2024-01-16T15:13:34.534875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:12:41.528922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.543807+0000","last_change":"2024-01-16T15:13:33.529167+0000","last_active":"2024-01-16T15:13:34.543807+0000","last_peered":"2024-01-16T15:13:34.543807+0000","last_clean":"2024-01-16T15:13:34.543807+0000","last_became_active":"2024-01-16T15:13:33.528376+0000","last_became_peered":"2024-01-16T15:13:33.528376+0000","last_unstale":"2024-01-16T15:13:34.543807+0000","last_undegraded":"2024-01-16T15:13:34.543807+0000","last_fullsized":"2024-01-16T15:13:34.543807+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:42:03.398824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.765747+0000","last_change":"2024-01-16T15:13:34.525354+0000","last_active":"2024-01-16T15:13:35.765747+0000","last_peered":"2024-01-16T15:13:35.765747+0000","last_clean":"2024-01-16T15:13:35.765747+0000","last_became_active":"2024-01-16T15:13:34.521214+0000","last_became_peered":"2024-01-16T15:13:34.521214+0000","last_unstale":"2024-01-16T15:13:35.765747+0000","last_undegraded":"2024-01-16T15:13:35.765747+0000","last_fullsized":"2024-01-16T15:13:35.765747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:01.966387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.516322+0000","last_change":"2024-01-16T15:13:33.525041+0000","last_active":"2024-01-16T15:13:34.516322+0000","last_peered":"2024-01-16T15:13:34.516322+0000","last_clean":"2024-01-16T15:13:34.516322+0000","last_became_active":"2024-01-16T15:13:33.524778+0000","last_became_peered":"2024-01-16T15:13:33.524778+0000","last_unstale":"2024-01-16T15:13:34.516322+0000","last_undegraded":"2024-01-16T15:13:34.516322+0000","last_fullsized":"2024-01-16T15:13:34.516322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:14.655944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.542167+0000","last_change":"2024-01-16T15:13:34.537540+0000","last_active":"2024-01-16T15:13:34.542167+0000","last_peered":"2024-01-16T15:13:34.542167+0000","last_clean":"2024-01-16T15:13:34.542167+0000","last_became_active":"2024-01-16T15:13:34.537024+0000","last_became_peered":"2024-01-16T15:13:34.537024+0000","last_unstale":"2024-01-16T15:13:34.542167+0000","last_undegraded":"2024-01-16T15:13:34.542167+0000","last_fullsized":"2024-01-16T15:13:34.542167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:16.188469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.529011+0000","last_change":"2024-01-16T15:13:33.523785+0000","last_active":"2024-01-16T15:13:34.529011+0000","last_peered":"2024-01-16T15:13:34.529011+0000","last_clean":"2024-01-16T15:13:34.529011+0000","last_became_active":"2024-01-16T15:13:33.523508+0000","last_became_peered":"2024-01-16T15:13:33.523508+0000","last_unstale":"2024-01-16T15:13:34.529011+0000","last_undegraded":"2024-01-16T15:13:34.529011+0000","last_fullsized":"2024-01-16T15:13:34.529011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:46:10.155259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.548357+0000","last_change":"2024-01-16T15:13:34.546690+0000","last_active":"2024-01-16T15:13:34.548357+0000","last_peered":"2024-01-16T15:13:34.548357+0000","last_clean":"2024-01-16T15:13:34.548357+0000","last_became_active":"2024-01-16T15:13:34.546420+0000","last_became_peered":"2024-01-16T15:13:34.546420+0000","last_unstale":"2024-01-16T15:13:34.548357+0000","last_undegraded":"2024-01-16T15:13:34.548357+0000","last_fullsized":"2024-01-16T15:13:34.548357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:45.110834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.542052+0000","last_change":"2024-01-16T15:13:33.527398+0000","last_active":"2024-01-16T15:13:34.542052+0000","last_peered":"2024-01-16T15:13:34.542052+0000","last_clean":"2024-01-16T15:13:34.542052+0000","last_became_active":"2024-01-16T15:13:33.522768+0000","last_became_peered":"2024-01-16T15:13:33.522768+0000","last_unstale":"2024-01-16T15:13:34.542052+0000","last_undegraded":"2024-01-16T15:13:34.542052+0000","last_fullsized":"2024-01-16T15:13:34.542052+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:17:27.998742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767423+0000","last_change":"2024-01-16T15:13:34.532655+0000","last_active":"2024-01-16T15:13:35.767423+0000","last_peered":"2024-01-16T15:13:35.767423+0000","last_clean":"2024-01-16T15:13:35.767423+0000","last_became_active":"2024-01-16T15:13:34.531680+0000","last_became_peered":"2024-01-16T15:13:34.531680+0000","last_unstale":"2024-01-16T15:13:35.767423+0000","last_undegraded":"2024-01-16T15:13:35.767423+0000","last_fullsized":"2024-01-16T15:13:35.767423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:39.654096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768721+0000","last_change":"2024-01-16T15:13:33.527895+0000","last_active":"2024-01-16T15:13:35.768721+0000","last_peered":"2024-01-16T15:13:35.768721+0000","last_clean":"2024-01-16T15:13:35.768721+0000","last_became_active":"2024-01-16T15:13:33.527518+0000","last_became_peered":"2024-01-16T15:13:33.527518+0000","last_unstale":"2024-01-16T15:13:35.768721+0000","last_undegraded":"2024-01-16T15:13:35.768721+0000","last_fullsized":"2024-01-16T15:13:35.768721+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:21:08.292939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.768319+0000","last_change":"2024-01-16T15:13:34.538478+0000","last_active":"2024-01-16T15:13:35.768319+0000","last_peered":"2024-01-16T15:13:35.768319+0000","last_clean":"2024-01-16T15:13:35.768319+0000","last_became_active":"2024-01-16T15:13:34.538115+0000","last_became_peered":"2024-01-16T15:13:34.538115+0000","last_unstale":"2024-01-16T15:13:35.768319+0000","last_undegraded":"2024-01-16T15:13:35.768319+0000","last_fullsized":"2024-01-16T15:13:35.768319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:17:21.263554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.541177+0000","last_change":"2024-01-16T15:13:33.521965+0000","last_active":"2024-01-16T15:13:34.541177+0000","last_peered":"2024-01-16T15:13:34.541177+0000","last_clean":"2024-01-16T15:13:34.541177+0000","last_became_active":"2024-01-16T15:13:33.521662+0000","last_became_peered":"2024-01-16T15:13:33.521662+0000","last_unstale":"2024-01-16T15:13:34.541177+0000","last_undegraded":"2024-01-16T15:13:34.541177+0000","last_fullsized":"2024-01-16T15:13:34.541177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:05:42.740807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535050+0000","last_change":"2024-01-16T15:13:34.528570+0000","last_active":"2024-01-16T15:13:35.535050+0000","last_peered":"2024-01-16T15:13:35.535050+0000","last_clean":"2024-01-16T15:13:35.535050+0000","last_became_active":"2024-01-16T15:13:34.527865+0000","last_became_peered":"2024-01-16T15:13:34.527865+0000","last_unstale":"2024-01-16T15:13:35.535050+0000","last_undegraded":"2024-01-16T15:13:35.535050+0000","last_fullsized":"2024-01-16T15:13:35.535050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:43:50.783073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.509600+0000","last_change":"2024-01-16T15:13:33.525428+0000","last_active":"2024-01-16T15:13:34.509600+0000","last_peered":"2024-01-16T15:13:34.509600+0000","last_clean":"2024-01-16T15:13:34.509600+0000","last_became_active":"2024-01-16T15:13:33.525158+0000","last_became_peered":"2024-01-16T15:13:33.525158+0000","last_unstale":"2024-01-16T15:13:34.509600+0000","last_undegraded":"2024-01-16T15:13:34.509600+0000","last_fullsized":"2024-01-16T15:13:34.509600+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:49:27.406092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.527856+0000","last_change":"2024-01-16T15:13:34.523930+0000","last_active":"2024-01-16T15:13:34.527856+0000","last_peered":"2024-01-16T15:13:34.527856+0000","last_clean":"2024-01-16T15:13:34.527856+0000","last_became_active":"2024-01-16T15:13:34.523288+0000","last_became_peered":"2024-01-16T15:13:34.523288+0000","last_unstale":"2024-01-16T15:13:34.527856+0000","last_undegraded":"2024-01-16T15:13:34.527856+0000","last_fullsized":"2024-01-16T15:13:34.527856+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:47:03.133425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.528444+0000","last_change":"2024-01-16T15:13:33.518982+0000","last_active":"2024-01-16T15:13:34.528444+0000","last_peered":"2024-01-16T15:13:34.528444+0000","last_clean":"2024-01-16T15:13:34.528444+0000","last_became_active":"2024-01-16T15:13:33.518206+0000","last_became_peered":"2024-01-16T15:13:33.518206+0000","last_unstale":"2024-01-16T15:13:34.528444+0000","last_undegraded":"2024-01-16T15:13:34.528444+0000","last_fullsized":"2024-01-16T15:13:34.528444+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:18:03.034174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.539645+0000","last_change":"2024-01-16T15:13:34.538866+0000","last_active":"2024-01-16T15:13:34.539645+0000","last_peered":"2024-01-16T15:13:34.539645+0000","last_clean":"2024-01-16T15:13:34.539645+0000","last_became_active":"2024-01-16T15:13:34.538399+0000","last_became_peered":"2024-01-16T15:13:34.538399+0000","last_unstale":"2024-01-16T15:13:34.539645+0000","last_undegraded":"2024-01-16T15:13:34.539645+0000","last_fullsized":"2024-01-16T15:13:34.539645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:22.516934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.516606+0000","last_change":"2024-01-16T15:13:33.521775+0000","last_active":"2024-01-16T15:13:34.516606+0000","last_peered":"2024-01-16T15:13:34.516606+0000","last_clean":"2024-01-16T15:13:34.516606+0000","last_became_active":"2024-01-16T15:13:33.521505+0000","last_became_peered":"2024-01-16T15:13:33.521505+0000","last_unstale":"2024-01-16T15:13:34.516606+0000","last_undegraded":"2024-01-16T15:13:34.516606+0000","last_fullsized":"2024-01-16T15:13:34.516606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:51:20.467490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.540825+0000","last_change":"2024-01-16T15:13:34.532979+0000","last_active":"2024-01-16T15:13:34.540825+0000","last_peered":"2024-01-16T15:13:34.540825+0000","last_clean":"2024-01-16T15:13:34.540825+0000","last_became_active":"2024-01-16T15:13:34.532168+0000","last_became_peered":"2024-01-16T15:13:34.532168+0000","last_unstale":"2024-01-16T15:13:34.540825+0000","last_undegraded":"2024-01-16T15:13:34.540825+0000","last_fullsized":"2024-01-16T15:13:34.540825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:38:36.245768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.512876+0000","last_change":"2024-01-16T15:13:33.528732+0000","last_active":"2024-01-16T15:13:34.512876+0000","last_peered":"2024-01-16T15:13:34.512876+0000","last_clean":"2024-01-16T15:13:34.512876+0000","last_became_active":"2024-01-16T15:13:33.528459+0000","last_became_peered":"2024-01-16T15:13:33.528459+0000","last_unstale":"2024-01-16T15:13:34.512876+0000","last_undegraded":"2024-01-16T15:13:34.512876+0000","last_fullsized":"2024-01-16T15:13:34.512876+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:21:14.330799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.544920+0000","last_change":"2024-01-16T15:13:34.542151+0000","last_active":"2024-01-16T15:13:34.544920+0000","last_peered":"2024-01-16T15:13:34.544920+0000","last_clean":"2024-01-16T15:13:34.544920+0000","last_became_active":"2024-01-16T15:13:34.541865+0000","last_became_peered":"2024-01-16T15:13:34.541865+0000","last_unstale":"2024-01-16T15:13:34.544920+0000","last_undegraded":"2024-01-16T15:13:34.544920+0000","last_fullsized":"2024-01-16T15:13:34.544920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:38:18.116846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.513261+0000","last_change":"2024-01-16T15:13:33.518000+0000","last_active":"2024-01-16T15:13:34.513261+0000","last_peered":"2024-01-16T15:13:34.513261+0000","last_clean":"2024-01-16T15:13:34.513261+0000","last_became_active":"2024-01-16T15:13:33.516415+0000","last_became_peered":"2024-01-16T15:13:33.516415+0000","last_unstale":"2024-01-16T15:13:34.513261+0000","last_undegraded":"2024-01-16T15:13:34.513261+0000","last_fullsized":"2024-01-16T15:13:34.513261+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:25.356804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535313+0000","last_change":"2024-01-16T15:13:34.534481+0000","last_active":"2024-01-16T15:13:35.535313+0000","last_peered":"2024-01-16T15:13:35.535313+0000","last_clean":"2024-01-16T15:13:35.535313+0000","last_became_active":"2024-01-16T15:13:34.534203+0000","last_became_peered":"2024-01-16T15:13:34.534203+0000","last_unstale":"2024-01-16T15:13:35.535313+0000","last_undegraded":"2024-01-16T15:13:35.535313+0000","last_fullsized":"2024-01-16T15:13:35.535313+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:09:26.372207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"85'2","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.539469+0000","last_change":"2024-01-16T15:13:33.525384+0000","last_active":"2024-01-16T15:13:34.539469+0000","last_peered":"2024-01-16T15:13:34.539469+0000","last_clean":"2024-01-16T15:13:34.539469+0000","last_became_active":"2024-01-16T15:13:33.525117+0000","last_became_peered":"2024-01-16T15:13:33.525117+0000","last_unstale":"2024-01-16T15:13:34.539469+0000","last_undegraded":"2024-01-16T15:13:34.539469+0000","last_fullsized":"2024-01-16T15:13:34.539469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:24.897403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766709+0000","last_change":"2024-01-16T15:13:34.538260+0000","last_active":"2024-01-16T15:13:35.766709+0000","last_peered":"2024-01-16T15:13:35.766709+0000","last_clean":"2024-01-16T15:13:35.766709+0000","last_became_active":"2024-01-16T15:13:34.537908+0000","last_became_peered":"2024-01-16T15:13:34.537908+0000","last_unstale":"2024-01-16T15:13:35.766709+0000","last_undegraded":"2024-01-16T15:13:35.766709+0000","last_fullsized":"2024-01-16T15:13:35.766709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:40:26.900872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.521218+0000","last_change":"2024-01-16T15:13:33.528870+0000","last_active":"2024-01-16T15:13:34.521218+0000","last_peered":"2024-01-16T15:13:34.521218+0000","last_clean":"2024-01-16T15:13:34.521218+0000","last_became_active":"2024-01-16T15:13:33.528585+0000","last_became_peered":"2024-01-16T15:13:33.528585+0000","last_unstale":"2024-01-16T15:13:34.521218+0000","last_undegraded":"2024-01-16T15:13:34.521218+0000","last_fullsized":"2024-01-16T15:13:34.521218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:19:23.019368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766943+0000","last_change":"2024-01-16T15:13:34.543512+0000","last_active":"2024-01-16T15:13:35.766943+0000","last_peered":"2024-01-16T15:13:35.766943+0000","last_clean":"2024-01-16T15:13:35.766943+0000","last_became_active":"2024-01-16T15:13:34.541471+0000","last_became_peered":"2024-01-16T15:13:34.541471+0000","last_unstale":"2024-01-16T15:13:35.766943+0000","last_undegraded":"2024-01-16T15:13:35.766943+0000","last_fullsized":"2024-01-16T15:13:35.766943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:59:06.044841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767260+0000","last_change":"2024-01-16T15:13:33.527678+0000","last_active":"2024-01-16T15:13:35.767260+0000","last_peered":"2024-01-16T15:13:35.767260+0000","last_clean":"2024-01-16T15:13:35.767260+0000","last_became_active":"2024-01-16T15:13:33.527371+0000","last_became_peered":"2024-01-16T15:13:33.527371+0000","last_unstale":"2024-01-16T15:13:35.767260+0000","last_undegraded":"2024-01-16T15:13:35.767260+0000","last_fullsized":"2024-01-16T15:13:35.767260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:55:46.383745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.535421+0000","last_change":"2024-01-16T15:13:34.528192+0000","last_active":"2024-01-16T15:13:35.535421+0000","last_peered":"2024-01-16T15:13:35.535421+0000","last_clean":"2024-01-16T15:13:35.535421+0000","last_became_active":"2024-01-16T15:13:34.527080+0000","last_became_peered":"2024-01-16T15:13:34.527080+0000","last_unstale":"2024-01-16T15:13:35.535421+0000","last_undegraded":"2024-01-16T15:13:35.535421+0000","last_fullsized":"2024-01-16T15:13:35.535421+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:15.671346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767005+0000","last_change":"2024-01-16T15:13:33.523600+0000","last_active":"2024-01-16T15:13:35.767005+0000","last_peered":"2024-01-16T15:13:35.767005+0000","last_clean":"2024-01-16T15:13:35.767005+0000","last_became_active":"2024-01-16T15:13:33.523325+0000","last_became_peered":"2024-01-16T15:13:33.523325+0000","last_unstale":"2024-01-16T15:13:35.767005+0000","last_undegraded":"2024-01-16T15:13:35.767005+0000","last_fullsized":"2024-01-16T15:13:35.767005+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:08.442154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.538655+0000","last_change":"2024-01-16T15:13:34.535307+0000","last_active":"2024-01-16T15:13:34.538655+0000","last_peered":"2024-01-16T15:13:34.538655+0000","last_clean":"2024-01-16T15:13:34.538655+0000","last_became_active":"2024-01-16T15:13:34.535022+0000","last_became_peered":"2024-01-16T15:13:34.535022+0000","last_unstale":"2024-01-16T15:13:34.538655+0000","last_undegraded":"2024-01-16T15:13:34.538655+0000","last_fullsized":"2024-01-16T15:13:34.538655+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:40:20.618842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.508823+0000","last_change":"2024-01-16T15:13:33.523790+0000","last_active":"2024-01-16T15:13:34.508823+0000","last_peered":"2024-01-16T15:13:34.508823+0000","last_clean":"2024-01-16T15:13:34.508823+0000","last_became_active":"2024-01-16T15:13:33.523145+0000","last_became_peered":"2024-01-16T15:13:33.523145+0000","last_unstale":"2024-01-16T15:13:34.508823+0000","last_undegraded":"2024-01-16T15:13:34.508823+0000","last_fullsized":"2024-01-16T15:13:34.508823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:25:31.929357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.540462+0000","last_change":"2024-01-16T15:13:34.533505+0000","last_active":"2024-01-16T15:13:34.540462+0000","last_peered":"2024-01-16T15:13:34.540462+0000","last_clean":"2024-01-16T15:13:34.540462+0000","last_became_active":"2024-01-16T15:13:34.533229+0000","last_became_peered":"2024-01-16T15:13:34.533229+0000","last_unstale":"2024-01-16T15:13:34.540462+0000","last_undegraded":"2024-01-16T15:13:34.540462+0000","last_fullsized":"2024-01-16T15:13:34.540462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:57:27.301534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.534260+0000","last_change":"2024-01-16T15:13:33.520685+0000","last_active":"2024-01-16T15:13:35.534260+0000","last_peered":"2024-01-16T15:13:35.534260+0000","last_clean":"2024-01-16T15:13:35.534260+0000","last_became_active":"2024-01-16T15:13:33.517416+0000","last_became_peered":"2024-01-16T15:13:33.517416+0000","last_unstale":"2024-01-16T15:13:35.534260+0000","last_undegraded":"2024-01-16T15:13:35.534260+0000","last_fullsized":"2024-01-16T15:13:35.534260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:36.791362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.763901+0000","last_change":"2024-01-16T15:13:34.523371+0000","last_active":"2024-01-16T15:13:35.763901+0000","last_peered":"2024-01-16T15:13:35.763901+0000","last_clean":"2024-01-16T15:13:35.763901+0000","last_became_active":"2024-01-16T15:13:34.522911+0000","last_became_peered":"2024-01-16T15:13:34.522911+0000","last_unstale":"2024-01-16T15:13:35.763901+0000","last_undegraded":"2024-01-16T15:13:35.763901+0000","last_fullsized":"2024-01-16T15:13:35.763901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:44:59.561501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766387+0000","last_change":"2024-01-16T15:13:33.523448+0000","last_active":"2024-01-16T15:13:35.766387+0000","last_peered":"2024-01-16T15:13:35.766387+0000","last_clean":"2024-01-16T15:13:35.766387+0000","last_became_active":"2024-01-16T15:13:33.523155+0000","last_became_peered":"2024-01-16T15:13:33.523155+0000","last_unstale":"2024-01-16T15:13:35.766387+0000","last_undegraded":"2024-01-16T15:13:35.766387+0000","last_fullsized":"2024-01-16T15:13:35.766387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:37.663483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.766311+0000","last_change":"2024-01-16T15:13:34.546455+0000","last_active":"2024-01-16T15:13:35.766311+0000","last_peered":"2024-01-16T15:13:35.766311+0000","last_clean":"2024-01-16T15:13:35.766311+0000","last_became_active":"2024-01-16T15:13:34.545469+0000","last_became_peered":"2024-01-16T15:13:34.545469+0000","last_unstale":"2024-01-16T15:13:35.766311+0000","last_undegraded":"2024-01-16T15:13:35.766311+0000","last_fullsized":"2024-01-16T15:13:35.766311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:24:50.768273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.533639+0000","last_change":"2024-01-16T15:13:33.525656+0000","last_active":"2024-01-16T15:13:35.533639+0000","last_peered":"2024-01-16T15:13:35.533639+0000","last_clean":"2024-01-16T15:13:35.533639+0000","last_became_active":"2024-01-16T15:13:33.525375+0000","last_became_peered":"2024-01-16T15:13:33.525375+0000","last_unstale":"2024-01-16T15:13:35.533639+0000","last_undegraded":"2024-01-16T15:13:35.533639+0000","last_fullsized":"2024-01-16T15:13:35.533639+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:34:57.697280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":9,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.547144+0000","last_change":"2024-01-16T15:13:34.541437+0000","last_active":"2024-01-16T15:13:34.547144+0000","last_peered":"2024-01-16T15:13:34.547144+0000","last_clean":"2024-01-16T15:13:34.547144+0000","last_became_active":"2024-01-16T15:13:34.541164+0000","last_became_peered":"2024-01-16T15:13:34.541164+0000","last_unstale":"2024-01-16T15:13:34.547144+0000","last_undegraded":"2024-01-16T15:13:34.547144+0000","last_fullsized":"2024-01-16T15:13:34.547144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:33.482952+0000","last_clean_scrub_stamp":"2024-01-16T15:13:33.482952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:18:41.380286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"85'1","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-01-16T15:13:35.767966+0000","last_change":"2024-01-16T15:13:33.522356+0000","last_active":"2024-01-16T15:13:35.767966+0000","last_peered":"2024-01-16T15:13:35.767966+0000","last_clean":"2024-01-16T15:13:35.767966+0000","last_became_active":"2024-01-16T15:13:33.522036+0000","last_became_peered":"2024-01-16T15:13:33.522036+0000","last_unstale":"2024-01-16T15:13:35.767966+0000","last_undegraded":"2024-01-16T15:13:35.767966+0000","last_fullsized":"2024-01-16T15:13:35.767966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:13:32.467549+0000","last_clean_scrub_stamp":"2024-01-16T15:13:32.467549+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:55.397809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":228,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-01-16T15:13:34.508239+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:13:34.508239+0000","last_peered":"2024-01-16T15:13:34.508239+0000","last_clean":"2024-01-16T15:13:34.508239+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:13:34.508239+0000","last_undegraded":"2024-01-16T15:13:34.508239+0000","last_fullsized":"2024-01-16T15:13:34.508239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509608,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68248,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303592,"statfs":{"total":96636764160,"available":96566878208,"internally_reserved":0,"allocated":1060864,"data_stored":450561,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5880000000000001,"5min":1.6619999999999999,"15min":1.605},"min":{"1min":0.71399999999999997,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":8.8070000000000004,"5min":13.084,"15min":27.853999999999999},"last":1.2430000000000001},{"interface":"front","average":{"1min":2.8559999999999999,"5min":2.327,"15min":1.704},"min":{"1min":0.70599999999999996,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":28.603999999999999,"5min":34.939999999999998,"15min":34.939999999999998},"last":1.278}]},{"osd":1,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.5620000000000001,"15min":1.4670000000000001},"min":{"1min":0.86299999999999999,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.5760000000000001,"5min":13.859,"15min":13.859},"last":1.1859999999999999},{"interface":"front","average":{"1min":3.198,"5min":2.0590000000000002,"15min":1.6319999999999999},"min":{"1min":0.81699999999999995,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":33.225000000000001,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.135}]},{"osd":2,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.7749999999999999,"5min":1.8140000000000001,"15min":1.595},"min":{"1min":0.94299999999999995,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":9.6679999999999993,"5min":13.050000000000001,"15min":13.050000000000001},"last":1.3360000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":2.2149999999999999,"15min":1.7509999999999999},"min":{"1min":0.82799999999999996,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":1.972,"5min":35.011000000000003,"15min":35.011000000000003},"last":1.5349999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.7310000000000001,"15min":1.667},"min":{"1min":0.76500000000000001,"5min":0.439,"15min":0.374},"max":{"1min":1.579,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.867,"5min":1.9119999999999999,"15min":1.732},"min":{"1min":0.98499999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":9.7780000000000005,"5min":14.273999999999999,"15min":14.273999999999999},"last":1.583}]},{"osd":4,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":2.379,"15min":1.859},"min":{"1min":0.82299999999999995,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7649999999999999,"5min":34.972999999999999,"15min":34.972999999999999},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.353,"5min":1.8939999999999999,"15min":1.8200000000000001},"min":{"1min":0.875,"5min":0.627,"15min":0.40400000000000003},"max":{"1min":1.7290000000000001,"5min":16.003,"15min":16.003},"last":1.6930000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":2.1779999999999999,"15min":2.0329999999999999},"min":{"1min":1.1739999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.9059999999999999,"5min":23.760999999999999,"15min":23.760999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.954,"5min":2.2719999999999998,"15min":1.9159999999999999},"min":{"1min":0.90500000000000003,"5min":0.79300000000000004,"15min":0.52400000000000002},"max":{"1min":9.4120000000000008,"5min":23.716999999999999,"15min":23.716999999999999},"last":1.484}]},{"osd":7,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":2.532,"15min":2.0739999999999998},"min":{"1min":1.1739999999999999,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.831,"5min":34.889000000000003,"15min":34.889000000000003},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.569,"5min":2.1040000000000001,"15min":2.2010000000000001},"min":{"1min":1.2230000000000001,"5min":0.74099999999999999,"15min":0.47599999999999998},"max":{"1min":1.976,"5min":23.242999999999999,"15min":46.305},"last":1.748}]}]},{"osd":4,"up_from":15,"seq":64424509610,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64080,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307760,"statfs":{"total":96636764160,"available":96571146240,"internally_reserved":0,"allocated":1048576,"data_stored":449867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.5840000000000001,"15min":1.536},"min":{"1min":0.72799999999999998,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":3.617,"5min":22.757999999999999,"15min":22.757999999999999},"last":1.01},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.3200000000000001,"15min":1.282},"min":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.8640000000000001,"5min":14.034000000000001,"15min":14.034000000000001},"last":1.1060000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1990000000000001,"5min":1.5409999999999999,"15min":1.5700000000000001},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.7989999999999999,"5min":17.811,"15min":18.347000000000001},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.482,"15min":1.492},"min":{"1min":0.877,"5min":0.66500000000000004,"15min":0.498},"max":{"1min":2.3199999999999998,"5min":10.670999999999999,"15min":18.858000000000001},"last":1.3260000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.6950000000000001,"15min":1.675},"min":{"1min":0.59799999999999998,"5min":0.58299999999999996,"15min":0.56499999999999995},"max":{"1min":5.907,"5min":17.651,"15min":17.651},"last":1.843},{"interface":"front","average":{"1min":1.337,"5min":2.113,"15min":1.726},"min":{"1min":0.85499999999999998,"5min":0.68500000000000005,"15min":0.57699999999999996},"max":{"1min":1.8029999999999999,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.4630000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.419,"15min":1.5549999999999999},"min":{"1min":0.78500000000000003,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":2.1070000000000002,"5min":4.7329999999999997,"15min":26.741},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.627,"15min":1.796},"min":{"1min":0.82099999999999995,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":2.0379999999999998,"5min":17.715,"15min":26.701000000000001},"last":2.016}]},{"osd":5,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6799999999999999,"5min":1.702,"15min":1.734},"min":{"1min":1.0009999999999999,"5min":0.497,"15min":0.441},"max":{"1min":5.3170000000000002,"5min":10.763999999999999,"15min":18.919},"last":2.0680000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.542,"15min":1.673},"min":{"1min":0.89500000000000002,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":5.3520000000000003,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.9179999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":2.347,"15min":1.9410000000000001},"min":{"1min":0.92500000000000004,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.863,"5min":32.649999999999999,"15min":32.649999999999999},"last":2.323},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.867,"15min":1.8300000000000001},"min":{"1min":1.012,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":3.4289999999999998,"5min":24.094999999999999,"15min":24.094999999999999},"last":2.169}]},{"osd":7,"last update":"Tue Jan 16 15:13:13 2024","interfaces":[{"interface":"back","average":{"1min":1.6180000000000001,"5min":1.8260000000000001,"15min":1.7689999999999999},"min":{"1min":1.117,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":5.835,"5min":17.446000000000002,"15min":26.683},"last":2.1259999999999999},{"interface":"front","average":{"1min":1.8380000000000001,"5min":2.073,"15min":1.976},"min":{"1min":0.82599999999999996,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":8.0079999999999991,"5min":17.765000000000001,"15min":17.765000000000001},"last":2.21}]}]},{"osd":7,"up_from":14,"seq":60129542312,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68808,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303032,"statfs":{"total":96636764160,"available":96566304768,"internally_reserved":0,"allocated":1634304,"data_stored":1033977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.5649999999999999,"15min":1.494},"min":{"1min":0.72499999999999998,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.6559999999999999,"5min":12.35,"15min":15.661},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.587,"5min":1.486,"15min":1.528},"min":{"1min":0.80500000000000005,"5min":0.73499999999999999,"15min":0.40300000000000002},"max":{"1min":12.461,"5min":12.461,"15min":30.126000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.708,"15min":1.4379999999999999},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.6659999999999999,"5min":24.338000000000001,"15min":24.338000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":1.754,"5min":1.5349999999999999,"15min":1.605},"min":{"1min":0.95799999999999996,"5min":0.66100000000000003,"15min":0.46200000000000002},"max":{"1min":12.526,"5min":12.526,"15min":21.298999999999999},"last":1.236}]},{"osd":2,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.5549999999999999,"15min":1.4950000000000001},"min":{"1min":0.82099999999999995,"5min":0.68899999999999995,"15min":0.51500000000000001},"max":{"1min":1.5720000000000001,"5min":12.474,"15min":12.474},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.224,"5min":2.089,"15min":1.643},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.7210000000000001,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.355}]},{"osd":3,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.619,"15min":1.6000000000000001},"min":{"1min":1.006,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.8819999999999999,"5min":12.58,"15min":12.58},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":2.2349999999999999,"15min":1.768},"min":{"1min":0.629,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":1.7030000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":1.1319999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.51,"15min":1.597},"min":{"1min":0.93799999999999994,"5min":0.626,"15min":0.42799999999999999},"max":{"1min":4.1390000000000002,"5min":12.539,"15min":14.659000000000001},"last":1.296},{"interface":"front","average":{"1min":1.3220000000000001,"5min":2.339,"15min":1.8140000000000001},"min":{"1min":0.58899999999999997,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":1.6870000000000001,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.486}]},{"osd":5,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.8180000000000001,"5min":1.6599999999999999,"15min":1.8420000000000001},"min":{"1min":0.80800000000000005,"5min":0.66800000000000004,"15min":0.36699999999999999},"max":{"1min":12.609999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.748},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.8280000000000001,"15min":1.8400000000000001},"min":{"1min":0.76700000000000002,"5min":0.57999999999999996,"15min":0.41099999999999998},"max":{"1min":3.4159999999999999,"5min":12.836,"15min":21.312000000000001},"last":1.583}]},{"osd":6,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":2.29,"15min":1.861},"min":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":1.8460000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.641},{"interface":"front","average":{"1min":1.976,"5min":1.7569999999999999,"15min":1.9630000000000001},"min":{"1min":0.80400000000000005,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":12.567,"5min":12.567,"15min":29.991},"last":1.845}]}]},{"osd":5,"up_from":14,"seq":60129542313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76560,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295280,"statfs":{"total":96636764160,"available":96558366720,"internally_reserved":0,"allocated":1052672,"data_stored":445173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":1.8460000000000001,"15min":1.534},"min":{"1min":0.68700000000000006,"5min":0.48799999999999999,"15min":0.42499999999999999},"max":{"1min":6.8380000000000001,"5min":19.382999999999999,"15min":19.382999999999999},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.179,"5min":1.2649999999999999,"15min":1.319},"min":{"1min":0.67500000000000004,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.649,"5min":3.698,"15min":11.109},"last":2.4790000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.8620000000000001,"15min":1.6040000000000001},"min":{"1min":0.64400000000000002,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":8.0909999999999993,"5min":17.052,"15min":17.693999999999999},"last":4.2450000000000001},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.9590000000000001,"15min":1.627},"min":{"1min":0.753,"5min":0.72599999999999998,"15min":0.432},"max":{"1min":2.3300000000000001,"5min":20.645,"15min":20.645},"last":0.56699999999999995}]},{"osd":2,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.448,"5min":2.0249999999999999,"15min":1.667},"min":{"1min":0.70999999999999996,"5min":0.66500000000000004,"15min":0.48099999999999998},"max":{"1min":2.1320000000000001,"5min":21.138000000000002,"15min":21.138000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.4870000000000001,"15min":1.4610000000000001},"min":{"1min":0.626,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":2.21,"5min":4.0380000000000003,"15min":11.321},"last":2.2200000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.891,"15min":1.8029999999999999},"min":{"1min":0.69499999999999995,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":7.984,"5min":17.100999999999999,"15min":23.109000000000002},"last":1.329},{"interface":"front","average":{"1min":1.4330000000000001,"5min":2.0699999999999998,"15min":1.7909999999999999},"min":{"1min":0.73999999999999999,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0779999999999998,"5min":19.303000000000001,"15min":19.303000000000001},"last":0.65700000000000003}]},{"osd":4,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.881,"15min":1.8160000000000001},"min":{"1min":0.72399999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":1.8080000000000001,"5min":17.143999999999998,"15min":24.436},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.968,"15min":1.744},"min":{"1min":0.59899999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":2.4569999999999999,"5min":19.126000000000001,"15min":19.126000000000001},"last":0.872}]},{"osd":6,"last update":"Tue Jan 16 15:13:18 2024","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.478,"15min":1.5940000000000001},"min":{"1min":0.51900000000000002,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":2.524,"5min":6.5880000000000001,"15min":21.192},"last":2.2650000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":2.1920000000000002,"15min":1.9610000000000001},"min":{"1min":0.68500000000000005,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":1.7410000000000001,"5min":27.042000000000002,"15min":27.042000000000002},"last":2.105}]},{"osd":7,"last update":"Tue Jan 16 15:13:06 2024","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.8819999999999999,"15min":1.907},"min":{"1min":0.88,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":2.4180000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":2.1579999999999999},{"interface":"front","average":{"1min":1.8819999999999999,"5min":2.1309999999999998,"15min":2.0990000000000002},"min":{"1min":1.0049999999999999,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":8.0380000000000003,"5min":16.939,"15min":24.367000000000001},"last":0.90100000000000002}]}]},{"osd":3,"up_from":14,"seq":60129542313,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76556,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295284,"statfs":{"total":96636764160,"available":96558370816,"internally_reserved":0,"allocated":1048576,"data_stored":445151,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.1950000000000001,"5min":1.5069999999999999,"15min":1.248},"min":{"1min":0.44700000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":5.7400000000000002,"5min":28.437999999999999,"15min":31.562999999999999},"last":2.02},{"interface":"front","average":{"1min":0.98899999999999999,"5min":1.1679999999999999,"15min":1.133},"min":{"1min":0.40600000000000003,"5min":0.40100000000000002,"15min":0.39600000000000002},"max":{"1min":1.6879999999999999,"5min":11.361000000000001,"15min":11.361000000000001},"last":7.1429999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.1319999999999999,"15min":1.089},"min":{"1min":0.53800000000000003,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.373,"5min":5.931,"15min":15.569000000000001},"last":10.856999999999999},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.7669999999999999,"15min":1.4079999999999999},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.36599999999999999},"max":{"1min":1.8129999999999999,"5min":28.352,"15min":28.352},"last":0.81100000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":3.294,"5min":2.3039999999999998,"15min":1.619},"min":{"1min":0.60699999999999998,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":39.265999999999998,"5min":39.265999999999998,"15min":39.265999999999998},"last":7.8159999999999998},{"interface":"front","average":{"1min":1.117,"5min":1.4650000000000001,"15min":1.341},"min":{"1min":0.498,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":1.4590000000000001,"5min":13.526,"15min":14.577999999999999},"last":7.9160000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.5049999999999999,"15min":1.534},"min":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.53800000000000003},"max":{"1min":5.6890000000000001,"5min":6.0629999999999997,"15min":20.978999999999999},"last":4.5570000000000004},{"interface":"front","average":{"1min":1.5149999999999999,"5min":2.4390000000000001,"15min":1.8320000000000001},"min":{"1min":0.93400000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":2.3330000000000002,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.976}]},{"osd":5,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.6899999999999999,"5min":1.9279999999999999,"15min":1.7509999999999999},"min":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":6.3869999999999996,"5min":17.093,"15min":20.663},"last":4.5039999999999996},{"interface":"front","average":{"1min":2.0139999999999998,"5min":2.431,"15min":1.925},"min":{"1min":1.012,"5min":0.98499999999999999,"15min":0.76000000000000001},"max":{"1min":6.0679999999999996,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.925}]},{"osd":6,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.7869999999999999,"15min":1.7210000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":3.2149999999999999,"5min":13.691000000000001,"15min":13.691000000000001},"last":7.0060000000000002},{"interface":"front","average":{"1min":1.381,"5min":2.0710000000000002,"15min":1.8779999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":1.99,"5min":26.698,"15min":26.698},"last":4.6500000000000004}]},{"osd":7,"last update":"Tue Jan 16 15:13:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.9630000000000001,"15min":1.7250000000000001},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.829,"5min":13.622,"15min":13.622},"last":7.7750000000000004},{"interface":"front","average":{"1min":1.627,"5min":2.0819999999999999,"15min":2.133},"min":{"1min":0.88300000000000001,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.3180000000000001,"5min":15.372999999999999,"15min":49.811},"last":4.6029999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542312,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72396,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299444,"statfs":{"total":96636764160,"available":96562630656,"internally_reserved":0,"allocated":1040384,"data_stored":444121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":1.091,"15min":1.006},"min":{"1min":0.66600000000000004,"5min":0.42299999999999999,"15min":0.34200000000000003},"max":{"1min":1.9079999999999999,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.137,"15min":1.0780000000000001},"min":{"1min":0.73299999999999998,"5min":0.44500000000000001,"15min":0.28599999999999998},"max":{"1min":1.8240000000000001,"5min":4.7859999999999996,"15min":4.7859999999999996},"last":0.623}]},{"osd":1,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.429,"15min":1.175},"min":{"1min":0.75600000000000001,"5min":0.50900000000000001,"15min":0.40300000000000002},"max":{"1min":3.056,"5min":23.135999999999999,"15min":23.135999999999999},"last":1.0209999999999999},{"interface":"front","average":{"1min":1.095,"5min":1.2789999999999999,"15min":1.1870000000000001},"min":{"1min":0.82099999999999995,"5min":0.41499999999999998,"15min":0.41199999999999998},"max":{"1min":1.49,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":0.89700000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":1.1180000000000001,"15min":1.071},"min":{"1min":0.66400000000000003,"5min":0.56699999999999995,"15min":0.34300000000000003},"max":{"1min":1.7070000000000001,"5min":4.0199999999999996,"15min":9.1530000000000005},"last":1.7729999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.486,"15min":1.391},"min":{"1min":0.92200000000000004,"5min":0.54300000000000004,"15min":0.45800000000000002},"max":{"1min":1.776,"5min":4.8380000000000001,"15min":4.9610000000000003},"last":1.389}]},{"osd":4,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.573,"15min":1.4850000000000001},"min":{"1min":0.79900000000000004,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":3.8180000000000001,"5min":21.837,"15min":21.837},"last":1.0900000000000001},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.742,"15min":1.5509999999999999},"min":{"1min":0.99399999999999999,"5min":0.79300000000000004,"15min":0.59199999999999997},"max":{"1min":2.9609999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.6830000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.6080000000000001,"15min":1.53},"min":{"1min":1.04,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":6.1879999999999997,"5min":17.841999999999999,"15min":17.841999999999999},"last":1.3300000000000001},{"interface":"front","average":{"1min":1.597,"5min":1.5840000000000001,"15min":1.4990000000000001},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":5.9130000000000003,"5min":5.9130000000000003,"15min":9.423},"last":1.538}]},{"osd":6,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.875,"15min":1.6439999999999999},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":2.1059999999999999,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.6279999999999999,"5min":1.6319999999999999,"15min":1.716},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":4.8620000000000001,"5min":4.8620000000000001,"15min":21.564},"last":1.637}]},{"osd":7,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":2.0910000000000002,"15min":1.7130000000000001},"min":{"1min":1.0549999999999999,"5min":0.68300000000000005,"15min":0.64100000000000001},"max":{"1min":3.9020000000000001,"5min":22.834,"15min":22.834},"last":1.589},{"interface":"front","average":{"1min":1.5640000000000001,"5min":2.5249999999999999,"15min":2.0739999999999998},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.82799999999999996},"max":{"1min":2.4620000000000002,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.4990000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542314,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72404,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299436,"statfs":{"total":96636764160,"available":96562622464,"internally_reserved":0,"allocated":1048576,"data_stored":443733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":2.028,"15min":1.3049999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.619,"5min":31.178999999999998,"15min":31.178999999999998},"last":1.091},{"interface":"front","average":{"1min":0.92600000000000005,"5min":1.49,"15min":1.254},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":1.5029999999999999,"5min":26.859999999999999,"15min":26.859999999999999},"last":0.70199999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.121,"15min":1.111},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":1.377,"5min":3.4420000000000002,"15min":6.6120000000000001},"last":1.165},{"interface":"front","average":{"1min":1.234,"5min":1.8899999999999999,"15min":1.5169999999999999},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":2.0489999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":1.3660000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.1279999999999999,"5min":1.9630000000000001,"15min":1.6279999999999999},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.837,"5min":27.609999999999999,"15min":27.609999999999999},"last":1.508},{"interface":"front","average":{"1min":1.1619999999999999,"5min":2.1960000000000002,"15min":1.629},"min":{"1min":0.51100000000000001,"5min":0.48999999999999999,"15min":0.36399999999999999},"max":{"1min":1.6719999999999999,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.657}]},{"osd":4,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4710000000000001,"15min":1.446},"min":{"1min":0.78400000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":2.2410000000000001,"5min":5.7590000000000003,"15min":10.577999999999999},"last":2.2879999999999998},{"interface":"front","average":{"1min":1.413,"5min":2.1480000000000001,"15min":1.736},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.1899999999999999,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.7390000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.4510000000000001,"15min":1.528},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.865,"5min":2.7879999999999998,"15min":17.469000000000001},"last":1.798},{"interface":"front","average":{"1min":1.47,"5min":1.6319999999999999,"15min":1.6040000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1539999999999999,"5min":5.8090000000000002,"15min":8.2989999999999995},"last":1.595}]},{"osd":6,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.6060000000000001,"15min":1.696},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0550000000000002,"5min":4.3780000000000001,"15min":17.428999999999998},"last":1.288},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.923,"15min":1.855},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.1190000000000002,"5min":18.512,"15min":18.850999999999999},"last":1.869}]},{"osd":7,"last update":"Tue Jan 16 15:13:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":1.873,"15min":1.881},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.1150000000000002,"5min":15.885,"15min":30.834},"last":2.0179999999999998},{"interface":"front","average":{"1min":1.5489999999999999,"5min":2.492,"15min":2.0409999999999999},"min":{"1min":0.81100000000000005,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":2.3029999999999999,"5min":27.716999999999999,"15min":27.716999999999999},"last":2.2200000000000002}]}]},{"osd":0,"up_from":14,"seq":60129542312,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68812,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303028,"statfs":{"total":96636764160,"available":96566300672,"internally_reserved":0,"allocated":1634304,"data_stored":1033941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":1.212,"15min":1.1899999999999999},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.5680000000000001,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.434},{"interface":"front","average":{"1min":1.083,"5min":1.5229999999999999,"15min":1.4970000000000001},"min":{"1min":0.48699999999999999,"5min":0.40000000000000002,"15min":0.33900000000000002},"max":{"1min":1.5289999999999999,"5min":26.459,"15min":50.429000000000002},"last":0.61599999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.496,"15min":1.454},"min":{"1min":0.57299999999999995,"5min":0.434,"15min":0.38},"max":{"1min":1.4470000000000001,"5min":20.041,"15min":22.716999999999999},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.722,"15min":1.607},"min":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.38},"max":{"1min":1.5640000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":0.77100000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.26,"15min":1.2170000000000001},"min":{"1min":0.52900000000000003,"5min":0.45800000000000002,"15min":0.372},"max":{"1min":3.085,"5min":5.8719999999999999,"15min":10.795},"last":0.48799999999999999},{"interface":"front","average":{"1min":1.26,"5min":1.5269999999999999,"15min":1.5760000000000001},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.837,"5min":9.484,"15min":22.254999999999999},"last":0.56499999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.7490000000000001,"15min":1.7929999999999999},"min":{"1min":0.79300000000000004,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.9319999999999999,"5min":21.300999999999998,"15min":22.407},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.873,"15min":1.907},"min":{"1min":0.72199999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.0459999999999998,"5min":26.57,"15min":50.497},"last":0.84299999999999997}]},{"osd":5,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.5680000000000001,"15min":1.623},"min":{"1min":0.70699999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.8400000000000001,"5min":9.9860000000000007,"15min":14.112},"last":0.871},{"interface":"front","average":{"1min":1.4610000000000001,"5min":1.77,"15min":1.841},"min":{"1min":0.90800000000000003,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.8169999999999999,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.212}]},{"osd":6,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.639,"15min":1.7749999999999999},"min":{"1min":0.73399999999999999,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":2.2970000000000002,"5min":8.2349999999999994,"15min":31.387},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.889,"15min":1.948},"min":{"1min":0.93799999999999994,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":1.948,"5min":9.2840000000000007,"15min":23.547000000000001},"last":0.97199999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:13:10 2024","interfaces":[{"interface":"back","average":{"1min":1.8440000000000001,"5min":2.1419999999999999,"15min":2.0590000000000002},"min":{"1min":0.85299999999999998,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":9.6980000000000004,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.02},{"interface":"front","average":{"1min":1.516,"5min":1.9390000000000001,"15min":1.8340000000000001},"min":{"1min":0.76300000000000001,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.044,"5min":12.455,"15min":15.747},"last":0.94899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:13:39.518 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:13:39.518 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:13:39.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.801+0000 7fdeea600640 1 Processor -- start 2024-01-16T15:13:39.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.801+0000 7fdeea600640 1 -- start start 2024-01-16T15:13:39.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdeea600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 0x7fdee40cd780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdeea600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fdee40cdcc0 0x7fdee406c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdeea600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdee413e440 con 0x7fdee415ed20 2024-01-16T15:13:39.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdeea600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdee41417a0 con 0x7fdee40cdcc0 2024-01-16T15:13:39.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdeea600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee40ce1d0 con 0x7fdee4141360 2024-01-16T15:13:39.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdee3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fdee415ed20 0x7fdee415f100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55066/0 (socket says 172.21.15.53:55066) 2024-01-16T15:13:39.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.803+0000 7fdee3400640 1 -- 172.21.15.53:0/3582275329 learned_addr learned my addr 172.21.15.53:0/3582275329 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee3e00640 1 --2- 172.21.15.53:0/3582275329 >> v2:172.21.15.146:3300/0 conn(0x7fdee40cdcc0 0x7fdee406c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdedbe00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 0x7fdee40cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910482019 0 0) 0x7fdee413e440 con 0x7fdee415ed20 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec4003520 con 0x7fdee415ed20 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 854107119 0 0) 0x7fdec4003520 con 0x7fdee415ed20 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 msgr2=0x7fdee40cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 0x7fdee40cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 >> v2:172.21.15.146:3300/0 conn(0x7fdee40cdcc0 msgr2=0x7fdee406c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 --2- 172.21.15.53:0/3582275329 >> v2:172.21.15.146:3300/0 conn(0x7fdee40cdcc0 0x7fdee406c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee40ce4b0 con 0x7fdee415ed20 2024-01-16T15:13:39.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fded4001670 con 0x7fdee415ed20 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 >> v1:172.21.15.53:6789/0 conn(0x7fdee415ed20 legacy=0x7fdee415f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdec4003970 0x7fdec4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 0x7fdec40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee40ce1d0 con 0x7fdec4003970 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400b980 con 0x7fdec4004d00 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.804+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400bb40 con 0x7fdee4141360 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3400640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdec4003970 0x7fdec4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdedbe00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 0x7fdec40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3400640 1 -- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 msgr2=0x7fdec40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3400640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee4141360 0x7fdec40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3e00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3400640 1 -- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4004d00 msgr2=0x7fdec4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:39.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3400640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3400640 1 -- 172.21.15.53:0/3582275329 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4003520 con 0x7fdec4003970 2024-01-16T15:13:39.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.805+0000 7fdee3e00640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.806+0000 7fdee3400640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdec4003970 0x7fdec4003d90 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fded400c160 tx=0x7fdec40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f2ffb703189b213 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded4005ad0 con 0x7fdec4003970 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded4005c30 con 0x7fdec4003970 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdee2a00640 1 -- 172.21.15.53:0/3582275329 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded402c810 con 0x7fdec4003970 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdeea600640 1 -- 172.21.15.53:0/3582275329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdec4003970 msgr2=0x7fdec4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdeea600640 1 --2- 172.21.15.53:0/3582275329 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdec4003970 0x7fdec4003d90 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7fded400c160 tx=0x7fdec40057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdee3e00640 1 -- 172.21.15.53:0/3582275329 reap_dead start 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdeea600640 1 -- 172.21.15.53:0/3582275329 shutdown_connections 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdeea600640 1 -- 172.21.15.53:0/3582275329 >> 172.21.15.53:0/3582275329 conn(0x7fdee40b0be0 msgr2=0x7fdee40b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.807+0000 7fdeea600640 1 -- 172.21.15.53:0/3582275329 shutdown_connections 2024-01-16T15:13:39.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.808+0000 7fdeea600640 1 -- 172.21.15.53:0/3582275329 wait complete. 2024-01-16T15:13:39.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.809+0000 7fdeea600640 1 Processor -- start 2024-01-16T15:13:39.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.809+0000 7fdeea600640 1 -- start start 2024-01-16T15:13:39.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.810+0000 7fdeea600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4003970 0x7fdee40ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.810+0000 7fdeea600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 0x7fdee41f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.810+0000 7fdeea600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee41f44a0 0x7fdee41f4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.810+0000 7fdeea600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee40adcd0 con 0x7fdee40ad850 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.810+0000 7fdeea600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdee40aaa90 con 0x7fdec4003970 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.810+0000 7fdeea600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdee40aad30 con 0x7fdee41f44a0 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 0x7fdee41f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 0x7fdee41f3f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34128/0 (socket says 172.21.15.53:34128) 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 -- 172.21.15.53:0/3970064008 learned_addr learned my addr 172.21.15.53:0/3970064008 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdee3400640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4003970 0x7fdee40ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdee3e00640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee41f44a0 0x7fdee41f4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 -- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee41f44a0 msgr2=0x7fdee41f4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee41f44a0 0x7fdee41f4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 -- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4003970 msgr2=0x7fdee40ad310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:39.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4003970 0x7fdee40ad310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:39.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 -- 172.21.15.53:0/3970064008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdee40ab010 con 0x7fdee40ad850 2024-01-16T15:13:39.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdee3e00640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee41f44a0 0x7fdee41f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:39.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.811+0000 7fdedbe00640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 0x7fdee41f3f60 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fdecc010890 tx=0x7fdecc010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:39.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.812+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc00eb60 con 0x7fdee40ad850 2024-01-16T15:13:39.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.812+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc00ece0 con 0x7fdee40ad850 2024-01-16T15:13:39.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.812+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdecc01c640 con 0x7fdee40ad850 2024-01-16T15:13:39.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.812+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdee40ab2f0 con 0x7fdee40ad850 2024-01-16T15:13:39.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.812+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdee40a8c80 con 0x7fdee40ad850 2024-01-16T15:13:39.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.813+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdee4067670 con 0x7fdee40ad850 2024-01-16T15:13:39.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.814+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fdecc0143f0 con 0x7fdee40ad850 2024-01-16T15:13:39.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.815+0000 7fdedb400640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdebc06bf50 0x7fdebc06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:39.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.815+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fdecc0886f0 con 0x7fdee40ad850 2024-01-16T15:13:39.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.816+0000 7fdee3400640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdebc06bf50 0x7fdebc06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:39.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.817+0000 7fdee3400640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdebc06bf50 0x7fdebc06e410 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fded4001050 tx=0x7fded4007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:39.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:39.819+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdecc055010 con 0x7fdee40ad850 2024-01-16T15:13:40.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.090+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fdee406dce0 con 0x7fdee40ad850 2024-01-16T15:13:40.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.094+0000 7fdedb400640 1 -- 172.21.15.53:0/3970064008 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7fdecc058630 con 0x7fdee40ad850 2024-01-16T15:13:40.074 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:13:40.075 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:13:40.075 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:13:40.075 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:13:40.075 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:13:40.075 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:13:40.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdebc06bf50 msgr2=0x7fdebc06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdebc06bf50 0x7fdebc06e410 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fded4001050 tx=0x7fded4007000 comp rx=0 tx=0).stop 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 msgr2=0x7fdee41f3f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 0x7fdee41f3f60 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7fdecc010890 tx=0x7fdecc010d50 comp rx=0 tx=0).stop 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 shutdown_connections 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdebc06bf50 0x7fdebc06e410 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdee41f44a0 0x7fdee41f4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdee40ad850 0x7fdee41f3f60 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 --2- 172.21.15.53:0/3970064008 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdec4003970 0x7fdee40ad310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 >> 172.21.15.53:0/3970064008 conn(0x7fdee40b0be0 msgr2=0x7fdee40af5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 shutdown_connections 2024-01-16T15:13:40.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.097+0000 7fdeea600640 1 -- 172.21.15.53:0/3970064008 wait complete. 2024-01-16T15:13:40.098 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:13:40.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.340+0000 7f9625a00640 1 Processor -- start 2024-01-16T15:13:40.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.340+0000 7f9625a00640 1 -- start start 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.342+0000 7f9625a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f962013df80 0x7f962015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.342+0000 7f9625a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96200cf9a0 0x7f96200cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.342+0000 7f9625a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962006c470 con 0x7f962013e360 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.342+0000 7f9625a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962006c750 con 0x7f96200cf9a0 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.342+0000 7f9625a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962006ca30 con 0x7f962013df80 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.342+0000 7f961f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96200cf9a0 0x7f96200cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96200cf9a0 0x7f96200cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34136/0 (socket says 172.21.15.53:34136) 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961f400640 1 -- 172.21.15.53:0/3024112413 learned_addr learned my addr 172.21.15.53:0/3024112413 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:40.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961e000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f962013e360 0x7f96200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55068/0 (socket says 172.21.15.53:55068) 2024-01-16T15:13:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961ea00640 1 --2- 172.21.15.53:0/3024112413 >> v2:172.21.15.146:3300/0 conn(0x7f962013df80 0x7f962015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961f400640 1 -- 172.21.15.53:0/3024112413 >> v2:172.21.15.146:3300/0 conn(0x7f962013df80 msgr2=0x7f962015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961f400640 1 --2- 172.21.15.53:0/3024112413 >> v2:172.21.15.146:3300/0 conn(0x7f962013df80 0x7f962015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961f400640 1 -- 172.21.15.53:0/3024112413 >> v1:172.21.15.53:6789/0 conn(0x7f962013e360 legacy=0x7f96200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.343+0000 7f961f400640 1 -- 172.21.15.53:0/3024112413 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962006cd10 con 0x7f96200cf9a0 2024-01-16T15:13:40.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.344+0000 7f961f400640 1 --2- 172.21.15.53:0/3024112413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96200cf9a0 0x7f96200cd840 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9620072770 tx=0x7f961802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1367ec5ec9ff4605 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:40.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.345+0000 7f961d600640 1 -- 172.21.15.53:0/3024112413 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618030860 con 0x7f96200cf9a0 2024-01-16T15:13:40.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.345+0000 7f961d600640 1 -- 172.21.15.53:0/3024112413 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96180309c0 con 0x7f96200cf9a0 2024-01-16T15:13:40.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.345+0000 7f961d600640 1 -- 172.21.15.53:0/3024112413 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618030d50 con 0x7f96200cf9a0 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 -- 172.21.15.53:0/3024112413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96200cf9a0 msgr2=0x7f96200cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 --2- 172.21.15.53:0/3024112413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96200cf9a0 0x7f96200cd840 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f9620072770 tx=0x7f961802b9a0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 -- 172.21.15.53:0/3024112413 shutdown_connections 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 --2- 172.21.15.53:0/3024112413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96200cf9a0 0x7f96200cd840 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 --2- 172.21.15.53:0/3024112413 >> v2:172.21.15.146:3300/0 conn(0x7f962013df80 0x7f962015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 -- 172.21.15.53:0/3024112413 >> 172.21.15.53:0/3024112413 conn(0x7f962006a800 msgr2=0x7f962006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 -- 172.21.15.53:0/3024112413 shutdown_connections 2024-01-16T15:13:40.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.346+0000 7f9625a00640 1 -- 172.21.15.53:0/3024112413 wait complete. 2024-01-16T15:13:40.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.348+0000 7f9625a00640 1 Processor -- start 2024-01-16T15:13:40.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.348+0000 7f9625a00640 1 -- start start 2024-01-16T15:13:40.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f9625a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96200cf9a0 0x7f9620153300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:40.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f9625a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f962013df80 0x7f96201e6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:40.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f9625a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 0x7f962015d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:40.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f9625a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b2ec0 con 0x7f962013e360 2024-01-16T15:13:40.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f9625a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9620153840 con 0x7f962013df80 2024-01-16T15:13:40.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f9625a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96200b1e20 con 0x7f96200cf9a0 2024-01-16T15:13:40.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 0x7f962015d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:40.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 0x7f962015d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34148/0 (socket says 172.21.15.53:34148) 2024-01-16T15:13:40.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 -- 172.21.15.53:0/1289327457 learned_addr learned my addr 172.21.15.53:0/1289327457 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961e000640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f962013df80 0x7f96201e6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 -- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96200cf9a0 msgr2=0x7f9620153300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96200cf9a0 0x7f9620153300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 -- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f962013df80 msgr2=0x7f96201e6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f962013df80 0x7f96201e6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961f400640 1 -- 172.21.15.53:0/1289327457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96180094d0 con 0x7f962013e360 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.349+0000 7f961e000640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f962013df80 0x7f96201e6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.350+0000 7f961f400640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 0x7f962015d8a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f961802b9e0 tx=0x7f9618004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.350+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618039a90 con 0x7f962013e360 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.350+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9618039bf0 con 0x7f962013e360 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.350+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96200b2100 con 0x7f962013e360 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.350+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9618043410 con 0x7f962013e360 2024-01-16T15:13:40.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.350+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96200b26c0 con 0x7f962013e360 2024-01-16T15:13:40.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.352+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96200675d0 con 0x7f962013e360 2024-01-16T15:13:40.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.353+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f961804c3b0 con 0x7f962013e360 2024-01-16T15:13:40.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.353+0000 7f9616a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f806bf20 0x7f95f806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:40.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.354+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f96180aaea0 con 0x7f962013e360 2024-01-16T15:13:40.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.354+0000 7f961ea00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f806bf20 0x7f95f806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:40.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.356+0000 7f961ea00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f806bf20 0x7f95f806e3e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f96100098f0 tx=0x7f9610008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:40.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.360+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96180774e0 con 0x7f962013e360 2024-01-16T15:13:40.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.597+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f962006df00 con 0x7f962013e360 2024-01-16T15:13:40.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.598+0000 7f9616a00640 1 -- 172.21.15.53:0/1289327457 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f9618048020 con 0x7f962013e360 2024-01-16T15:13:40.578 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:13:40.579 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":55,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6119,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3084871157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3084871157},{"type":"v1","addr":"172.21.15.146:6835","nonce":3084871157}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2008413931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2008413931},{"type":"v1","addr":"172.21.15.53:6835","nonce":2008413931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/614678519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":614678519},{"type":"v1","addr":"172.21.15.146:6837","nonce":614678519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50}],"filesystems":[{"mdsmap":{"epoch":55,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:13:33.971028+0000","modified":"2024-01-16T15:13:37.032876+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8040},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8040":{"gid":8040,"name":"c","rank":0,"incarnation":51,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/3609060592","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3609060592},{"type":"v1","addr":"172.21.15.53:6834","nonce":3609060592}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-01-16T15:13:40.579 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 55 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.600+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f806bf20 msgr2=0x7f95f806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.600+0000 7f9625a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f806bf20 0x7f95f806e3e0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f96100098f0 tx=0x7f9610008040 comp rx=0 tx=0).stop 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.600+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 msgr2=0x7f962015d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.600+0000 7f9625a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 0x7f962015d8a0 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f961802b9e0 tx=0x7f9618004730 comp rx=0 tx=0).stop 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 shutdown_connections 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f95f806bf20 0x7f95f806e3e0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f962013e360 0x7f962015d8a0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f962013df80 0x7f96201e6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 --2- 172.21.15.53:0/1289327457 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96200cf9a0 0x7f9620153300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:40.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 >> 172.21.15.53:0/1289327457 conn(0x7f962006a800 msgr2=0x7f96200b0130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:40.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 shutdown_connections 2024-01-16T15:13:40.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:40.601+0000 7f9625a00640 1 -- 172.21.15.53:0/1289327457 wait complete. 2024-01-16T15:13:41.003 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 55, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:13:33.971028+0000', 'modified': '2024-01-16T15:13:37.032876+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8040}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8040': {'gid': 8040, 'name': 'c', 'rank': 0, 'incarnation': 51, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6834/3609060592', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6832', 'nonce': 3609060592}, {'type': 'v1', 'addr': '172.21.15.53:6834', 'nonce': 3609060592}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:13:41.134 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:13:41.134 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:13:41.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.389+0000 7f6895a00640 1 Processor -- start 2024-01-16T15:13:41.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.390+0000 7f6895a00640 1 -- start start 2024-01-16T15:13:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.391+0000 7f6895a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6890071f00 0x7f6890146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.391+0000 7f6895a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68900b2e10 con 0x7f689006c7c0 2024-01-16T15:13:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.391+0000 7f6895a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689006cc10 con 0x7f6890071410 2024-01-16T15:13:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.391+0000 7f6895a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689006cec0 con 0x7f6890071f00 2024-01-16T15:13:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.392+0000 7f6895000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f689006c7c0 0x7f6890166cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55074/0 (socket says 172.21.15.53:55074) 2024-01-16T15:13:41.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.392+0000 7f6895000640 1 -- 172.21.15.53:0/2048641660 learned_addr learned my addr 172.21.15.53:0/2048641660 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:41.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.392+0000 7f6887e00640 1 --2- 172.21.15.53:0/2048641660 >> v2:172.21.15.146:3300/0 conn(0x7f6890071f00 0x7f6890146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.392+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2905577978 0 0) 0x7f68900b2e10 con 0x7f689006c7c0 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.392+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6870003520 con 0x7f689006c7c0 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305356364 0 0) 0x7f6870003520 con 0x7f689006c7c0 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 >> v2:172.21.15.146:3300/0 conn(0x7f6890071f00 msgr2=0x7f6890146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 --2- 172.21.15.53:0/2048641660 >> v2:172.21.15.146:3300/0 conn(0x7f6890071f00 0x7f6890146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 >> v1:172.21.15.146:6789/0 conn(0x7f6890071410 legacy=0x7f68900717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689006d1a0 con 0x7f689006c7c0 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f68880019f0 con 0x7f689006c7c0 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 >> v1:172.21.15.53:6789/0 conn(0x7f689006c7c0 legacy=0x7f6890166cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68700039c0 0x7f6870003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68700047c0 0x7f6870004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700051d0 0x7f687000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689006cec0 con 0x7f68700047c0 2024-01-16T15:13:41.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f687000c120 con 0x7f68700051d0 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687000c2e0 con 0x7f68700039c0 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f6887e00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68700047c0 0x7f6870004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f688fe00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68700039c0 0x7f6870003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.393+0000 7f6887e00640 1 -- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68700039c0 msgr2=0x7f6870003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6895000640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700051d0 0x7f687000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6887e00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68700039c0 0x7f6870003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6887e00640 1 -- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700051d0 msgr2=0x7f687000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6887e00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700051d0 0x7f687000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6887e00640 1 -- 172.21.15.53:0/2048641660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6870003520 con 0x7f68700047c0 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f688fe00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68700039c0 0x7f6870003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6895000640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700051d0 0x7f687000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:41.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.394+0000 7f6895000640 1 -- 172.21.15.53:0/2048641660 reap_dead start 2024-01-16T15:13:41.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.395+0000 7f6887e00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68700047c0 0x7f6870004c40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f6878009a90 tx=0x7f68700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd08aaaa93daac1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.395+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687802c680 con 0x7f68700047c0 2024-01-16T15:13:41.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.395+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687802c7e0 con 0x7f68700047c0 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.395+0000 7f688f400640 1 -- 172.21.15.53:0/2048641660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687802cb70 con 0x7f68700047c0 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 -- 172.21.15.53:0/2048641660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68700047c0 msgr2=0x7f6870004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68700047c0 0x7f6870004c40 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f6878009a90 tx=0x7f68700057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 -- 172.21.15.53:0/2048641660 shutdown_connections 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 --2- 172.21.15.53:0/2048641660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68700047c0 0x7f6870004c40 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 -- 172.21.15.53:0/2048641660 >> 172.21.15.53:0/2048641660 conn(0x7f689006a800 msgr2=0x7f689006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 -- 172.21.15.53:0/2048641660 shutdown_connections 2024-01-16T15:13:41.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.396+0000 7f6895a00640 1 -- 172.21.15.53:0/2048641660 wait complete. 2024-01-16T15:13:41.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.398+0000 7f6895a00640 1 Processor -- start 2024-01-16T15:13:41.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.398+0000 7f6895a00640 1 -- start start 2024-01-16T15:13:41.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6895a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700047c0 0x7f689015c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6895a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 0x7f6890144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6895a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68901f62b0 0x7f6890144870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6895a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68901f6db0 con 0x7f68901f5e30 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6895a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68900cd630 con 0x7f68700047c0 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6895a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68901f67f0 con 0x7f68901f62b0 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6887e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 0x7f6890144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6887e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 0x7f6890144330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34166/0 (socket says 172.21.15.53:34166) 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f6887e00640 1 -- 172.21.15.53:0/2742831535 learned_addr learned my addr 172.21.15.53:0/2742831535 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.399+0000 7f688fe00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700047c0 0x7f689015c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6895000640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68901f62b0 0x7f6890144870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6887e00640 1 -- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68901f62b0 msgr2=0x7f6890144870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6887e00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68901f62b0 0x7f6890144870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6887e00640 1 -- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700047c0 msgr2=0x7f689015c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6887e00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700047c0 0x7f689015c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6887e00640 1 -- 172.21.15.53:0/2742831535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68901f6ad0 con 0x7f68901f5e30 2024-01-16T15:13:41.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6887e00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 0x7f6890144330 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f6878000c00 tx=0x7f6878004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f688cc00640 1 -- 172.21.15.53:0/2742831535 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878003810 con 0x7f68901f5e30 2024-01-16T15:13:41.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f688cc00640 1 -- 172.21.15.53:0/2742831535 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6878003970 con 0x7f68901f5e30 2024-01-16T15:13:41.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f688cc00640 1 -- 172.21.15.53:0/2742831535 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878003ce0 con 0x7f68901f5e30 2024-01-16T15:13:41.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.400+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68900cd2c0 con 0x7f68901f5e30 2024-01-16T15:13:41.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.401+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689015ce30 con 0x7f68901f5e30 2024-01-16T15:13:41.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.403+0000 7f688cc00640 1 -- 172.21.15.53:0/2742831535 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6878034440 con 0x7f68901f5e30 2024-01-16T15:13:41.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.404+0000 7f688cc00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f686806bf50 0x7f686806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.404+0000 7f688cc00640 1 -- 172.21.15.53:0/2742831535 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f68780aa770 con 0x7f68901f5e30 2024-01-16T15:13:41.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.405+0000 7f688fe00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f686806bf50 0x7f686806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.405+0000 7f6899c46740 1 Processor -- start 2024-01-16T15:13:41.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.405+0000 7f6899c46740 1 -- start start 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e1070 0x55c8372e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f688fe00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f686806bf50 0x7f686806e410 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6890067b00 tx=0x7f6880009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6899c46740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c8372e0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e07b0 0x55c837324730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6899c46740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83723f880 con 0x55c8372dbd70 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83724eae0 con 0x55c8372e07b0 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c83724c1d0 con 0x55c8372e1070 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c8372e0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c8372e0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34180/0 (socket says 172.21.15.53:34180) 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 -- 172.21.15.53:0/829440376 learned_addr learned my addr 172.21.15.53:0/829440376 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f688fe00640 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e1070 0x55c8372e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 -- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e1070 msgr2=0x55c8372e1450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e1070 0x55c8372e1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 -- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e07b0 msgr2=0x55c837324730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e07b0 0x55c837324730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.406+0000 7f6887e00640 1 -- 172.21.15.53:0/829440376 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c83720f860 con 0x55c8372dbd70 2024-01-16T15:13:41.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.407+0000 7f6887e00640 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c8372e0270 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f687807abd0 tx=0x7f68780045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0722426a983dbd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.407+0000 7f6884c00640 1 -- 172.21.15.53:0/829440376 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687807ece0 con 0x55c8372dbd70 2024-01-16T15:13:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.407+0000 7f6884c00640 1 -- 172.21.15.53:0/829440376 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6878082e90 con 0x55c8372dbd70 2024-01-16T15:13:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.407+0000 7f6884c00640 1 -- 172.21.15.53:0/829440376 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878077350 con 0x55c8372dbd70 2024-01-16T15:13:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.407+0000 7f6899c46740 1 -- 172.21.15.53:0/829440376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 msgr2=0x55c8372e0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.407+0000 7f6899c46740 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c8372e0270 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f687807abd0 tx=0x7f68780045a0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 -- 172.21.15.53:0/829440376 shutdown_connections 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e07b0 0x55c837324730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c8372e0270 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 --2- 172.21.15.53:0/829440376 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e1070 0x55c8372e1450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 -- 172.21.15.53:0/829440376 >> 172.21.15.53:0/829440376 conn(0x55c8372a2c70 msgr2=0x55c837326b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 -- 172.21.15.53:0/829440376 shutdown_connections 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.408+0000 7f6899c46740 1 -- 172.21.15.53:0/829440376 wait complete. 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.409+0000 7f6899c46740 1 -- start start 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.409+0000 7f6899c46740 1 Processor -- start 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372dbd70 0x55c837301320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c83732b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 0x55c8373253d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c837247ac0 con 0x55c8372e1070 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8371e42f0 con 0x55c8372dbd70 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c837201c70 con 0x55c8372e07b0 2024-01-16T15:13:41.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6899c46740 10 client.0 _ll_register_callbacks cb 0x55c8372063b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6895000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 0x55c8373253d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6895000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 0x55c8373253d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34190/0 (socket says 172.21.15.53:34190) 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f6895000640 1 -- 172.21.15.53:0/2132443287 learned_addr learned my addr 172.21.15.53:0/2132443287 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.410+0000 7f688fe00640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372dbd70 0x55c837301320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6887e00640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c83732b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6895000640 1 -- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 msgr2=0x55c83732b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6895000640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c83732b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6895000640 1 -- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372dbd70 msgr2=0x55c837301320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6895000640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372dbd70 0x55c837301320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6895000640 1 -- 172.21.15.53:0/2132443287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689015ce30 con 0x55c8372e1070 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f688fe00640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372dbd70 0x55c837301320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6895000640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 0x55c8373253d0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f68900722e0 tx=0x7f6888007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5518425602f68cbe server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f687d600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f687d600640 1 -- 172.21.15.53:0/2132443287 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68880079d0 con 0x55c8372e1070 2024-01-16T15:13:41.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f687d600640 1 -- 172.21.15.53:0/2132443287 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f6888004d70 con 0x55c8372e1070 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c837240aa0 con 0x55c8372e1070 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f687d600640 1 -- 172.21.15.53:0/2132443287 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f688800f6a0 con 0x55c8372e1070 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6899c46740 10 client.8193 fetch_fsmap learned FSMap version 55 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.411+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x55c837246140 con 0x55c8372e1070 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f687d600640 1 -- 172.21.15.53:0/2132443287 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c837240aa0 con 0x55c8372e1070 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f687d600640 1 -- 172.21.15.53:0/2132443287 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 55) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6888012070 con 0x55c8372e1070 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f6899c46740 10 client.8193 fetch_fsmap finished waiting for FSMap version 55 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f6899c46740 10 client.8193 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f6899c46740 20 client.8193 populate_metadata read hostname 'smithi053' 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f6899c46740 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c837341f70 0x55c837344360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f6899c46740 4 client.8193 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 --> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8372407c0 con 0x55c837341f70 2024-01-16T15:13:41.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.412+0000 7f688fe00640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c837341f70 0x55c837344360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.413+0000 7f688fe00640 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c837341f70 0x55c837344360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16ea208b6e5523fb server_cookie=9ffff3c4b9bcb59c in_seq=0 out_seq=0 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.413+0000 7f687d600640 10 client.8193 ms_handle_connect on v2:172.21.15.53:6832/3609060592 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f687d600640 1 -- 172.21.15.53:0/2132443287 <== mds.0 v2:172.21.15.53:6832/3609060592 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c8372407c0 con 0x55c837341f70 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f687d600640 10 client.8193 handle_command_reply: tid=0 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 client.8193 shutdown 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 msgr2=0x55c8373253d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 0x55c8373253d0 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f68900722e0 tx=0x7f6888007780 comp rx=0 tx=0).stop 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 shutdown_connections 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c837341f70 0x55c837344360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e1070 0x55c8373253d0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c83732b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 --2- 172.21.15.53:0/2132443287 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372dbd70 0x55c837301320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f687d600640 0 client.8193 ms_handle_reset on v2:172.21.15.53:6832/3609060592 2024-01-16T15:13:41.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 >> 172.21.15.53:0/2132443287 conn(0x55c8372a2c70 msgr2=0x55c8372dadd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 shutdown_connections 2024-01-16T15:13:41.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.414+0000 7f6899c46740 1 -- 172.21.15.53:0/2132443287 wait complete. 2024-01-16T15:13:41.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.415+0000 7f6899c46740 20 client.8193 trim_cache size 0 max 16384 2024-01-16T15:13:41.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.432+0000 7f6899c46740 1 Processor -- start 2024-01-16T15:13:41.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.432+0000 7f6899c46740 1 -- start start 2024-01-16T15:13:41.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f6899c46740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c837326b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c8373270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372df960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f6899c46740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c836faaf20 con 0x55c8372dbd70 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8371fbe30 con 0x55c8372e1070 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c8373268a0 con 0x55c8372e07b0 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f688fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c837326b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f688fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c837326b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34196/0 (socket says 172.21.15.53:34196) 2024-01-16T15:13:41.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.433+0000 7f688fe00640 1 -- 172.21.15.53:0/3536977425 learned_addr learned my addr 172.21.15.53:0/3536977425 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f688fe00640 1 -- 172.21.15.53:0/3536977425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 msgr2=0x55c8373270a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f688fe00640 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c8373270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f688fe00640 1 -- 172.21.15.53:0/3536977425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 msgr2=0x55c8372df960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f688fe00640 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f688fe00640 1 -- 172.21.15.53:0/3536977425 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8371e3b80 con 0x55c8372dbd70 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f688fe00640 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c837326b60 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x55c837303dc0 tx=0x7f6880012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3b85693e7c0935 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f6885600640 1 -- 172.21.15.53:0/3536977425 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f688000fd40 con 0x55c8372dbd70 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f6885600640 1 -- 172.21.15.53:0/3536977425 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f688000d050 con 0x55c8372dbd70 2024-01-16T15:13:41.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.434+0000 7f6885600640 1 -- 172.21.15.53:0/3536977425 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68800242d0 con 0x55c8372dbd70 2024-01-16T15:13:41.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 -- 172.21.15.53:0/3536977425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 msgr2=0x55c837326b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c837326b60 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x55c837303dc0 tx=0x7f6880012000 comp rx=0 tx=0).stop 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 -- 172.21.15.53:0/3536977425 shutdown_connections 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372df960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372e07b0 0x55c8373270a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 --2- 172.21.15.53:0/3536977425 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372dbd70 0x55c837326b60 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 -- 172.21.15.53:0/3536977425 >> 172.21.15.53:0/3536977425 conn(0x55c8372a2c70 msgr2=0x55c837323b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 -- 172.21.15.53:0/3536977425 shutdown_connections 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.435+0000 7f6899c46740 1 -- 172.21.15.53:0/3536977425 wait complete. 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.436+0000 7f6899c46740 1 -- start start 2024-01-16T15:13:41.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.436+0000 7f6899c46740 1 Processor -- start 2024-01-16T15:13:41.419 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372dbd70 0x55c837391040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 0x55c8372df960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372dfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c837327070 con 0x55c8372e07b0 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c83732aa50 con 0x55c8372e1070 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c837391880 con 0x55c8372dbd70 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6899c46740 10 client.0 _ll_register_callbacks cb 0x7f68880018f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6887e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 0x55c8372df960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6887e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 0x55c8372df960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34206/0 (socket says 172.21.15.53:34206) 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.437+0000 7f6887e00640 1 -- 172.21.15.53:0/3177425702 learned_addr learned my addr 172.21.15.53:0/3177425702 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f688fe00640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372dbd70 0x55c837391040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6895000640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372dfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6887e00640 1 -- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372dbd70 msgr2=0x55c837391040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6887e00640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372dbd70 0x55c837391040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6887e00640 1 -- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 msgr2=0x55c8372dfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6887e00640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372dfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6887e00640 1 -- 172.21.15.53:0/3177425702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689006cc10 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6887e00640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 0x55c8372df960 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f6878082960 tx=0x7f6878009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0f743f98f7cec5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f687d600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f687d600640 1 -- 172.21.15.53:0/3177425702 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687807b0a0 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f687d600640 1 -- 172.21.15.53:0/3177425702 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f687807ece0 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.438+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6878082e90 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f687d600640 1 -- 172.21.15.53:0/3177425702 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f687807a130 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 10 client.8199 fetch_fsmap learned FSMap version 55 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=2+}) v3 -- 0x55c837209170 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f687d600640 1 -- 172.21.15.53:0/3177425702 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6878082e90 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f687d600640 1 -- 172.21.15.53:0/3177425702 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 55) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6878082a00 con 0x55c8372e07b0 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 10 client.8199 fetch_fsmap finished waiting for FSMap version 55 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 10 client.8199 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:13:41.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 20 client.8199 populate_metadata read hostname 'smithi053' 2024-01-16T15:13:41.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c8373483d0 0x55c8373487b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 4 client.8199 mds_command: new command op to 8040 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:13:41.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 --> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c836f486c0 con 0x55c8373483d0 2024-01-16T15:13:41.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.439+0000 7f688fe00640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c8373483d0 0x55c8373487b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f688fe00640 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c8373483d0 0x55c8373487b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f281f20c988ad37 server_cookie=4f969c9113ab5db in_seq=0 out_seq=0 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f687d600640 10 client.8199 ms_handle_connect on v2:172.21.15.53:6832/3609060592 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f687d600640 1 -- 172.21.15.53:0/3177425702 <== mds.0 v2:172.21.15.53:6832/3609060592 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55c836f486c0 con 0x55c8373483d0 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f687d600640 10 client.8199 handle_command_reply: tid=0 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f6899c46740 1 client.8199 shutdown 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 msgr2=0x55c8372df960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f6899c46740 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 0x55c8372df960 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f6878082960 tx=0x7f6878009190 comp rx=0 tx=0).stop 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 shutdown_connections 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.440+0000 7f6899c46740 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:6832/3609060592,v1:172.21.15.53:6834/3609060592] conn(0x55c8373483d0 0x55c8373487b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55c8372e1070 0x55c8372dfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55c8372e07b0 0x55c8372df960 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 1 --2- 172.21.15.53:0/3177425702 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55c8372dbd70 0x55c837391040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f687d600640 0 client.8199 ms_handle_reset on v2:172.21.15.53:6832/3609060592 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 >> 172.21.15.53:0/3177425702 conn(0x55c8372a2c70 msgr2=0x55c8372dc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 shutdown_connections 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 1 -- 172.21.15.53:0/3177425702 wait complete. 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6899c46740 20 client.8199 trim_cache size 0 max 16384 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:13:41.424 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 8040, 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 6.9414572970000004, 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 55, 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 88, 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 86, 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 12.309516478000001 2024-01-16T15:13:41.425 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f686806bf50 msgr2=0x7f686806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6895a00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f686806bf50 0x7f686806e410 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f6890067b00 tx=0x7f6880009040 comp rx=0 tx=0).stop 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 msgr2=0x7f6890144330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.441+0000 7f6895a00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 0x7f6890144330 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f6878000c00 tx=0x7f6878004730 comp rx=0 tx=0).stop 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 shutdown_connections 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f686806bf50 0x7f686806e410 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f68901f62b0 0x7f6890144870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f68901f5e30 0x7f6890144330 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 --2- 172.21.15.53:0/2742831535 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f68700047c0 0x7f689015c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 >> 172.21.15.53:0/2742831535 conn(0x7f689006a800 msgr2=0x7f68900afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 shutdown_connections 2024-01-16T15:13:41.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.442+0000 7f6895a00640 1 -- 172.21.15.53:0/2742831535 wait complete. 2024-01-16T15:13:41.439 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:13:41.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.667+0000 7fa4ed800640 1 Processor -- start 2024-01-16T15:13:41.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.667+0000 7fa4ed800640 1 -- start start 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.669+0000 7fa4ed800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa4e8162430 0x7fa4e80b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.669+0000 7fa4ed800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa4e8162810 0x7fa4e8071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.669+0000 7fa4ed800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e80702e0 con 0x7fa4e80b3650 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.669+0000 7fa4ed800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e80705c0 con 0x7fa4e8162810 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.669+0000 7fa4ed800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4e80708a0 con 0x7fa4e8162430 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.669+0000 7fa4ece00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa4e80b3650 0x7fa4e81650c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55078/0 (socket says 172.21.15.53:55078) 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4ece00640 1 -- 172.21.15.53:0/286136052 learned_addr learned my addr 172.21.15.53:0/286136052 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:41.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4dfe00640 1 --2- 172.21.15.53:0/286136052 >> v2:172.21.15.53:3300/0 conn(0x7fa4e8162810 0x7fa4e8071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4e7e00640 1 --2- 172.21.15.53:0/286136052 >> v2:172.21.15.146:3300/0 conn(0x7fa4e8162430 0x7fa4e80b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4e7400640 1 -- 172.21.15.53:0/286136052 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671217000 0 0) 0x7fa4e80702e0 con 0x7fa4e80b3650 2024-01-16T15:13:41.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4e7400640 1 -- 172.21.15.53:0/286136052 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4c8003520 con 0x7fa4e80b3650 2024-01-16T15:13:41.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4dfe00640 1 -- 172.21.15.53:0/286136052 >> v2:172.21.15.146:3300/0 conn(0x7fa4e8162430 msgr2=0x7fa4e80b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:41.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4dfe00640 1 --2- 172.21.15.53:0/286136052 >> v2:172.21.15.146:3300/0 conn(0x7fa4e8162430 0x7fa4e80b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4dfe00640 1 -- 172.21.15.53:0/286136052 >> v1:172.21.15.53:6789/0 conn(0x7fa4e80b3650 legacy=0x7fa4e81650c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.670+0000 7fa4dfe00640 1 -- 172.21.15.53:0/286136052 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8070b80 con 0x7fa4e8162810 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.671+0000 7fa4dfe00640 1 --2- 172.21.15.53:0/286136052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e8162810 0x7fa4e8071240 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80b2c20 tx=0x7fa4d002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116b191ed6d0737b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4e7400640 1 -- 172.21.15.53:0/286136052 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0031830 con 0x7fa4e8162810 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4e7400640 1 -- 172.21.15.53:0/286136052 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4d0031990 con 0x7fa4e8162810 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4e7400640 1 -- 172.21.15.53:0/286136052 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4d0031d20 con 0x7fa4e8162810 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 -- 172.21.15.53:0/286136052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e8162810 msgr2=0x7fa4e8071240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 --2- 172.21.15.53:0/286136052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e8162810 0x7fa4e8071240 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7fa4e80b2c20 tx=0x7fa4d002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 -- 172.21.15.53:0/286136052 shutdown_connections 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 --2- 172.21.15.53:0/286136052 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e8162810 0x7fa4e8071240 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 --2- 172.21.15.53:0/286136052 >> v2:172.21.15.146:3300/0 conn(0x7fa4e8162430 0x7fa4e80b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 -- 172.21.15.53:0/286136052 >> 172.21.15.53:0/286136052 conn(0x7fa4e806a800 msgr2=0x7fa4e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 -- 172.21.15.53:0/286136052 shutdown_connections 2024-01-16T15:13:41.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.672+0000 7fa4ed800640 1 -- 172.21.15.53:0/286136052 wait complete. 2024-01-16T15:13:41.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.675+0000 7fa4ed800640 1 Processor -- start 2024-01-16T15:13:41.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.675+0000 7fa4ed800640 1 -- start start 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.676+0000 7fa4ed800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4e80b3650 0x7fa4e81eb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.676+0000 7fa4ed800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e8162430 0x7fa4e814bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.676+0000 7fa4ed800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 0x7fa4e8152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.676+0000 7fa4ed800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e814d100 con 0x7fa4e81523d0 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.676+0000 7fa4ed800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e814dd10 con 0x7fa4e8162430 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.676+0000 7fa4ed800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e81ebf60 con 0x7fa4e80b3650 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 0x7fa4e8152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 0x7fa4e8152860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:34234/0 (socket says 172.21.15.53:34234) 2024-01-16T15:13:41.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 -- 172.21.15.53:0/16040544 learned_addr learned my addr 172.21.15.53:0/16040544 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:41.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4e7e00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4e80b3650 0x7fa4e81eb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4dfe00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e8162430 0x7fa4e814bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 -- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4e80b3650 msgr2=0x7fa4e81eb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4e80b3650 0x7fa4e81eb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 -- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e8162430 msgr2=0x7fa4e814bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.677+0000 7fa4ece00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e8162430 0x7fa4e814bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4ece00640 1 -- 172.21.15.53:0/16040544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d0009400 con 0x7fa4e81523d0 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4e7e00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4e80b3650 0x7fa4e81eb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4ece00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 0x7fa4e8152860 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000cbb0 tx=0x7fa4e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e000ad40 con 0x7fa4e81523d0 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4e0012400 con 0x7fa4e81523d0 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e000aea0 con 0x7fa4e81523d0 2024-01-16T15:13:41.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.678+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e81ec240 con 0x7fa4e81523d0 2024-01-16T15:13:41.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.679+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e81ec800 con 0x7fa4e81523d0 2024-01-16T15:13:41.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.680+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e8158710 con 0x7fa4e81523d0 2024-01-16T15:13:41.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.681+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fa4e0007070 con 0x7fa4e81523d0 2024-01-16T15:13:41.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.682+0000 7fa4e4c00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa4c006bf50 0x7fa4c006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:41.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.682+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fa4e00886f0 con 0x7fa4e81523d0 2024-01-16T15:13:41.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.682+0000 7fa4e7e00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa4c006bf50 0x7fa4c006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:41.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.683+0000 7fa4e7e00640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa4c006bf50 0x7fa4c006e410 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8006fd0 tx=0x7fa4d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:41.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.686+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4e0054de0 con 0x7fa4e81523d0 2024-01-16T15:13:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.926+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa4e8067450 con 0x7fa4e81523d0 2024-01-16T15:13:41.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.927+0000 7fa4e4c00640 1 -- 172.21.15.53:0/16040544 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa4e0017020 con 0x7fa4e81523d0 2024-01-16T15:13:41.908 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:13:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa4c006bf50 msgr2=0x7fa4c006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa4c006bf50 0x7fa4c006e410 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa4d8006fd0 tx=0x7fa4d8008040 comp rx=0 tx=0).stop 2024-01-16T15:13:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 msgr2=0x7fa4e8152860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 0x7fa4e8152860 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7fa4e000cbb0 tx=0x7fa4e000a430 comp rx=0 tx=0).stop 2024-01-16T15:13:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 shutdown_connections 2024-01-16T15:13:41.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa4c006bf50 0x7fa4c006e410 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4e81523d0 0x7fa4e8152860 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa4e8162430 0x7fa4e814bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 --2- 172.21.15.53:0/16040544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa4e80b3650 0x7fa4e81eb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 >> 172.21.15.53:0/16040544 conn(0x7fa4e806a800 msgr2=0x7fa4e806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.930+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 shutdown_connections 2024-01-16T15:13:41.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:41.931+0000 7fa4ed800640 1 -- 172.21.15.53:0/16040544 wait complete. 2024-01-16T15:13:41.930 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:13:41.930 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:13:41.930 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:13:41.930 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:13:41.930 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:41.931 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:13:41.931 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:13:41.931 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:13:41.931 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:13:41.931 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:41.931 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:13:41.978 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:13:41.978 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:13:41.978 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:13:41.978 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 41828sec preferred_lft 41828sec 2024-01-16T15:13:41.979 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.980 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:41.981 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:41.981 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:13:41.981 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:41.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:41.982 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:13:42.072 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:13:42.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:42.077 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:13:42.130 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:13:42.130 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:13:42.130 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:13:42.130 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:13:42.130 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:13:42.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:42.131 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:13:42.131 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:13:42.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:13:42.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:13:42.132 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:13:42.132 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:13:42.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:42.329 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:13:42.384 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:13:42.385 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:13:42.385 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:13:42.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:42.386 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:13:42.484 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:13:42.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:42.484 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:42.544 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:13:42.544 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:42.544 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:42.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:13:42.694 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:42.754 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:13:42.754 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:13:42.754 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:42.755 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:42.756 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:42.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:42.757 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:13:42.812 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:13:42.812 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:13:42.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:13:42.890 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:13:42.933 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:13:42.934 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:42.935 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:42.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:42.936 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:13:43.010 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:13:43.030+0000 7ff81b452480 -1 init, newargv = 0x559e27942940 newargc=15 2024-01-16T15:13:43.011 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[68494]: starting ceph client 2024-01-16T15:13:43.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[68494]: starting fuse 2024-01-16T15:13:44.120 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:13:44.233 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:13:44.292 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:44.293 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:13:44.294 INFO:teuthology.orchestra.run.smithi053.stdout:/home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:44.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:44.295 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:13:44.353 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:13:44.353 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:13:44.353 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:13:44.353 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:13:44.353 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:13:44.353 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:13:44.354 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:13:44.355 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:13:44.546 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.68494.asok 2024-01-16T15:13:44.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:13:44.564 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.68494.asok 2024-01-16T15:13:44.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:44.565 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.68494.asok status 2024-01-16T15:13:44.810 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:13:44.810 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:13:44.810 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:13:44.810 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "68494", 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 8214, 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 8214 2024-01-16T15:13:44.811 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 389445808 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 389445808 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.8214 192.168.0.1:0/389445808", 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/389445808", 2024-01-16T15:13:44.812 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:13:44.813 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 55, 2024-01-16T15:13:44.813 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 88, 2024-01-16T15:13:44.813 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:13:44.813 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:13:44.813 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:13:44.813 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:13:44.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:44.825 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:13:44.825 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:44.880 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:13:44.880 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:44.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:44.880 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:13:44.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:13:45.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.225+0000 7f6baaa00640 1 Processor -- start 2024-01-16T15:13:45.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.225+0000 7f6baaa00640 1 -- start start 2024-01-16T15:13:45.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.227+0000 7f6baaa00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6ba4143c40 0x7f6ba4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:45.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.227+0000 7f6baaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba40ccc10 0x7f6ba406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:45.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.227+0000 7f6baaa00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ba4070e50 con 0x7f6ba40717d0 2024-01-16T15:13:45.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.227+0000 7f6baaa00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ba40723c0 con 0x7f6ba4143c40 2024-01-16T15:13:45.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.227+0000 7f6baaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba4144020 con 0x7f6ba40ccc10 2024-01-16T15:13:45.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.228+0000 7f6ba9600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6ba4143c40 0x7f6ba4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:45.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.228+0000 7f6baa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba40ccc10 0x7f6ba406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:45.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.228+0000 7f6ba9600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6ba4143c40 0x7f6ba4071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:35584/0 (socket says 172.21.15.53:35584) 2024-01-16T15:13:45.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.228+0000 7f6ba9600640 1 -- 172.21.15.53:0/1029982396 learned_addr learned my addr 172.21.15.53:0/1029982396 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:45.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.229+0000 7f6baa000640 1 -- 172.21.15.53:0/1029982396 >> v2:172.21.15.146:3300/0 conn(0x7f6ba4143c40 msgr2=0x7f6ba4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:45.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.229+0000 7f6baa000640 1 --2- 172.21.15.53:0/1029982396 >> v2:172.21.15.146:3300/0 conn(0x7f6ba4143c40 0x7f6ba4071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.229+0000 7f6baa000640 1 -- 172.21.15.53:0/1029982396 >> v1:172.21.15.146:6789/0 conn(0x7f6ba40717d0 legacy=0x7f6ba40cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:45.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.229+0000 7f6baa000640 1 -- 172.21.15.53:0/1029982396 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba4071e00 con 0x7f6ba40ccc10 2024-01-16T15:13:45.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.229+0000 7f6ba3e00640 1 -- 172.21.15.53:0/1029982396 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916066446 0 0) 0x7f6ba4070e50 con 0x7f6ba40717d0 2024-01-16T15:13:45.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.230+0000 7f6baa000640 1 --2- 172.21.15.53:0/1029982396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba40ccc10 0x7f6ba406d160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009d90 tx=0x7f6b9c02be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83d8a2eff8023c29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:45.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.231+0000 7f6ba3e00640 1 -- 172.21.15.53:0/1029982396 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c02cbd0 con 0x7f6ba40ccc10 2024-01-16T15:13:45.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.231+0000 7f6ba3e00640 1 -- 172.21.15.53:0/1029982396 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9c02cd30 con 0x7f6ba40ccc10 2024-01-16T15:13:45.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.231+0000 7f6ba3e00640 1 -- 172.21.15.53:0/1029982396 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9c02cea0 con 0x7f6ba40ccc10 2024-01-16T15:13:45.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.231+0000 7f6baaa00640 1 -- 172.21.15.53:0/1029982396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba40ccc10 msgr2=0x7f6ba406d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:45.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.231+0000 7f6baaa00640 1 --2- 172.21.15.53:0/1029982396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba40ccc10 0x7f6ba406d160 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c009d90 tx=0x7f6b9c02be20 comp rx=0 tx=0).stop 2024-01-16T15:13:45.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.232+0000 7f6baaa00640 1 -- 172.21.15.53:0/1029982396 shutdown_connections 2024-01-16T15:13:45.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.232+0000 7f6baaa00640 1 --2- 172.21.15.53:0/1029982396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba40ccc10 0x7f6ba406d160 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.232+0000 7f6baaa00640 1 --2- 172.21.15.53:0/1029982396 >> v2:172.21.15.146:3300/0 conn(0x7f6ba4143c40 0x7f6ba4071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.232+0000 7f6baaa00640 1 -- 172.21.15.53:0/1029982396 >> 172.21.15.53:0/1029982396 conn(0x7f6ba406a800 msgr2=0x7f6ba406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:45.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.232+0000 7f6baaa00640 1 -- 172.21.15.53:0/1029982396 shutdown_connections 2024-01-16T15:13:45.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.232+0000 7f6baaa00640 1 -- 172.21.15.53:0/1029982396 wait complete. 2024-01-16T15:13:45.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.234+0000 7f6baaa00640 1 Processor -- start 2024-01-16T15:13:45.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.234+0000 7f6baaa00640 1 -- start start 2024-01-16T15:13:45.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 0x7f6ba4159de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:45.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baaa00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ba40ccc10 0x7f6ba40b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baaa00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba4143c40 0x7f6ba413bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baaa00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba415a320 con 0x7f6ba40717d0 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baaa00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40b36d0 con 0x7f6ba40ccc10 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baaa00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ba40b2ac0 con 0x7f6ba4143c40 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6ba9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 0x7f6ba4159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6ba9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 0x7f6ba4159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56064/0 (socket says 172.21.15.53:56064) 2024-01-16T15:13:45.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6ba9600640 1 -- 172.21.15.53:0/2506799882 learned_addr learned my addr 172.21.15.53:0/2506799882 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:45.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6baa000640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba4143c40 0x7f6ba413bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:45.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.235+0000 7f6ba8c00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ba40ccc10 0x7f6ba40b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:45.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba9600640 1 -- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba4143c40 msgr2=0x7f6ba413bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:45.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba9600640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba4143c40 0x7f6ba413bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba9600640 1 -- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ba40ccc10 msgr2=0x7f6ba40b2020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba9600640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ba40ccc10 0x7f6ba40b2020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba9600640 1 -- 172.21.15.53:0/2506799882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9c009a70 con 0x7f6ba40717d0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba8c00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ba40ccc10 0x7f6ba40b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba9600640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 0x7f6ba4159de0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009fd0 tx=0x7f6b9400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400c940 con 0x7f6ba40717d0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b9400caa0 con 0x7f6ba40717d0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba40b2da0 con 0x7f6ba40717d0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b9400ce30 con 0x7f6ba40717d0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.236+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ba40b3360 con 0x7f6ba40717d0 2024-01-16T15:13:45.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.237+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ba4067450 con 0x7f6ba40717d0 2024-01-16T15:13:45.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.239+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f6b94013470 con 0x7f6ba40717d0 2024-01-16T15:13:45.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.240+0000 7f6ba1600640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b7c06bf50 0x7f6b7c06e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:45.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.240+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f6b94088a60 con 0x7f6ba40717d0 2024-01-16T15:13:45.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.241+0000 7f6ba8c00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b7c06bf50 0x7f6b7c06e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:45.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.242+0000 7f6ba8c00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b7c06bf50 0x7f6b7c06e410 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40707a0 tx=0x7f6b8c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:45.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.245+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b94055070 con 0x7f6ba40717d0 2024-01-16T15:13:45.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.480+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6ba40b0840 con 0x7f6ba40717d0 2024-01-16T15:13:45.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.481+0000 7f6ba1600640 1 -- 172.21.15.53:0/2506799882 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v55) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6b94017070 con 0x7f6ba40717d0 2024-01-16T15:13:45.462 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b7c06bf50 msgr2=0x7f6b7c06e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b7c06bf50 0x7f6b7c06e410 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6ba40707a0 tx=0x7f6b8c009040 comp rx=0 tx=0).stop 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 msgr2=0x7f6ba4159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 0x7f6ba4159de0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009fd0 tx=0x7f6b9400ed40 comp rx=0 tx=0).stop 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 shutdown_connections 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b7c06bf50 0x7f6b7c06e410 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ba4143c40 0x7f6ba413bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ba40ccc10 0x7f6ba40b2020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 --2- 172.21.15.53:0/2506799882 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ba40717d0 0x7f6ba4159de0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 >> 172.21.15.53:0/2506799882 conn(0x7f6ba406a800 msgr2=0x7f6ba40cc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.484+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 shutdown_connections 2024-01-16T15:13:45.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:45.485+0000 7f6baaa00640 1 -- 172.21.15.53:0/2506799882 wait complete. 2024-01-16T15:13:45.484 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:13:45.484 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:13:45.484 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:13:45.484 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:13:45.485 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:45.485 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:13:45.485 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:13:45.485 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:13:45.485 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:13:45.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:45.485 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:13:45.542 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:13:45.542 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:13:45.542 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:13:45.542 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.542 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 41704sec preferred_lft 41704sec 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:13:45.543 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:13:45.544 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:13:45.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:45.545 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:13:45.632 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:13:45.636 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:45.636 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:13:45.690 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:13:45.690 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:13:45.690 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:13:45.691 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:13:45.691 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:13:45.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:45.691 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:13:45.692 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:13:45.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:13:45.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:13:45.692 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:13:45.692 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:13:45.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:45.871 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:13:45.919 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:13:45.919 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:13:45.920 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:13:45.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:45.920 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:13:46.014 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:13:46.014 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:46.014 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:46.068 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:13:46.068 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:13:46.068 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:46.121 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:13:46.214 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:46.273 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:13:46.273 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:13:46.273 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:13:46.274 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:13:46.274 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:13:46.274 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:13:46.274 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:13:46.274 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:13:46.274 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:13:46.275 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:13:46.275 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:13:46.275 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:46.275 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:13:46.275 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:13:46.275 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:46.276 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:13:46.277 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:13:46.278 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:46.278 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:46.278 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:46.278 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:46.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:46.280 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:13:46.328 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:13:46.329 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:13:46.382 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:13:46.467 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:46.527 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:13:46.528 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:46.529 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:46.530 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:46.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:46.531 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:13:46.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:13:46.599+0000 7f79ed3f9480 -1 init, newargv = 0x5607689b8920 newargc=15 2024-01-16T15:13:46.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[45986]: starting ceph client 2024-01-16T15:13:46.590 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[45986]: starting fuse 2024-01-16T15:13:47.681 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:13:47.783 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:13:47.837 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:13:47.838 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:13:47.839 INFO:teuthology.orchestra.run.smithi146.stdout:/home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:47.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:47.841 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:13:47.895 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:13:47.895 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:13:47.895 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:13:47.895 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:13:47.895 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:13:47.895 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:13:47.896 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:13:47.897 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:13:48.075 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.45986.asok 2024-01-16T15:13:48.079 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:13:48 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:13:48.093 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.45986.asok 2024-01-16T15:13:48.093 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:48.093 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.45986.asok status 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:13:48.329 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "45986", 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6847, 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 6847 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:13:48.330 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 91976145 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 91976145 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.6847 192.168.0.1:0/91976145", 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/91976145", 2024-01-16T15:13:48.331 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:13:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 55, 2024-01-16T15:13:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 88, 2024-01-16T15:13:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:13:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:13:48.332 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:13:48.332 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:13:48.344 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:48.344 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:13:48.345 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:48.397 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:13:48.397 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:48.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:48.397 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:13:48.485 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:13:48.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.739+0000 7f5b2d400640 1 Processor -- start 2024-01-16T15:13:48.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.739+0000 7f5b2d400640 1 -- start start 2024-01-16T15:13:48.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.740+0000 7f5b2d400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5b28143c40 0x7f5b28071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.740+0000 7f5b2d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b280ccc10 0x7f5b2806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b28070e50 con 0x7f5b280717d0 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2d400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b280723c0 con 0x7f5b280ccc10 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2d400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b28144020 con 0x7f5b28143c40 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b280ccc10 0x7f5b2806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b280ccc10 0x7f5b2806d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56070/0 (socket says 172.21.15.53:56070) 2024-01-16T15:13:48.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2ca00640 1 -- 172.21.15.53:0/2960364677 learned_addr learned my addr 172.21.15.53:0/2960364677 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:48.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b27e00640 1 --2- 172.21.15.53:0/2960364677 >> v2:172.21.15.146:3300/0 conn(0x7f5b28143c40 0x7f5b28071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:48.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2ca00640 1 -- 172.21.15.53:0/2960364677 >> v2:172.21.15.146:3300/0 conn(0x7f5b28143c40 msgr2=0x7f5b28071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:48.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.741+0000 7f5b2ca00640 1 --2- 172.21.15.53:0/2960364677 >> v2:172.21.15.146:3300/0 conn(0x7f5b28143c40 0x7f5b28071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:48.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.742+0000 7f5b2ca00640 1 -- 172.21.15.53:0/2960364677 >> v1:172.21.15.146:6789/0 conn(0x7f5b280717d0 legacy=0x7f5b280cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:48.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.742+0000 7f5b2ca00640 1 -- 172.21.15.53:0/2960364677 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b28071e00 con 0x7f5b280ccc10 2024-01-16T15:13:48.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.743+0000 7f5b2ca00640 1 --2- 172.21.15.53:0/2960364677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280ccc10 0x7f5b2806d160 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f5b2806e410 tx=0x7f5b2002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b681fedb4700c659 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:48.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.743+0000 7f5b26a00640 1 -- 172.21.15.53:0/2960364677 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200317e0 con 0x7f5b280ccc10 2024-01-16T15:13:48.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.743+0000 7f5b26a00640 1 -- 172.21.15.53:0/2960364677 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b20031940 con 0x7f5b280ccc10 2024-01-16T15:13:48.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.743+0000 7f5b26a00640 1 -- 172.21.15.53:0/2960364677 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20031cd0 con 0x7f5b280ccc10 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 -- 172.21.15.53:0/2960364677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280ccc10 msgr2=0x7f5b2806d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 --2- 172.21.15.53:0/2960364677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280ccc10 0x7f5b2806d160 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f5b2806e410 tx=0x7f5b2002b880 comp rx=0 tx=0).stop 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 -- 172.21.15.53:0/2960364677 shutdown_connections 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 --2- 172.21.15.53:0/2960364677 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280ccc10 0x7f5b2806d160 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 --2- 172.21.15.53:0/2960364677 >> v2:172.21.15.146:3300/0 conn(0x7f5b28143c40 0x7f5b28071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 -- 172.21.15.53:0/2960364677 >> 172.21.15.53:0/2960364677 conn(0x7f5b2806a800 msgr2=0x7f5b2806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 -- 172.21.15.53:0/2960364677 shutdown_connections 2024-01-16T15:13:48.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.744+0000 7f5b2d400640 1 -- 172.21.15.53:0/2960364677 wait complete. 2024-01-16T15:13:48.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.746+0000 7f5b2d400640 1 Processor -- start 2024-01-16T15:13:48.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.747+0000 7f5b2d400640 1 -- start start 2024-01-16T15:13:48.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b2d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 0x7f5b28142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:48.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b2d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b280ccc10 0x7f5b28156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b2d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b28143c40 0x7f5b280af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b2d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b281384d0 con 0x7f5b280717d0 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b2d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b28141610 con 0x7f5b280ccc10 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b2d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b281418f0 con 0x7f5b28143c40 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b27e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 0x7f5b28142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b27e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 0x7f5b28142ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56082/0 (socket says 172.21.15.53:56082) 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b27e00640 1 -- 172.21.15.53:0/671227171 learned_addr learned my addr 172.21.15.53:0/671227171 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.748+0000 7f5b27400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b280ccc10 0x7f5b28156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b2ca00640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b28143c40 0x7f5b280af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27e00640 1 -- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b28143c40 msgr2=0x7f5b280af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27e00640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b28143c40 0x7f5b280af410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27e00640 1 -- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b280ccc10 msgr2=0x7f5b28156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27e00640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b280ccc10 0x7f5b28156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:48.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27e00640 1 -- 172.21.15.53:0/671227171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b200094d0 con 0x7f5b280717d0 2024-01-16T15:13:48.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b280ccc10 0x7f5b28156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:48.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b27e00640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 0x7f5b28142ef0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f5b18007ef0 tx=0x7f5b1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:48.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18014620 con 0x7f5b280717d0 2024-01-16T15:13:48.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b18014780 con 0x7f5b280717d0 2024-01-16T15:13:48.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b28141bd0 con 0x7f5b280717d0 2024-01-16T15:13:48.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.749+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b18014b30 con 0x7f5b280717d0 2024-01-16T15:13:48.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.750+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b28142190 con 0x7f5b280717d0 2024-01-16T15:13:48.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.751+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b28142320 con 0x7f5b280717d0 2024-01-16T15:13:48.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.752+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f5b18014c90 con 0x7f5b280717d0 2024-01-16T15:13:48.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.753+0000 7f5b1fe00640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b0006bf20 0x7f5b0006e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:48.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.753+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f5b18088fe0 con 0x7f5b280717d0 2024-01-16T15:13:48.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.753+0000 7f5b27400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b0006bf20 0x7f5b0006e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:48.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.755+0000 7f5b27400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b0006bf20 0x7f5b0006e3e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5b2813cd90 tx=0x7f5b10008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:48.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.760+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b180558a0 con 0x7f5b280717d0 2024-01-16T15:13:48.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:48.999+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5b28067450 con 0x7f5b280717d0 2024-01-16T15:13:49.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.407+0000 7f5b1fe00640 1 -- 172.21.15.53:0/671227171 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v56) v1 ==== 136+0+0 (secure 0 0 0) 0x7f5b18058ec0 con 0x7f5b280717d0 2024-01-16T15:13:49.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b0006bf20 msgr2=0x7f5b0006e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b0006bf20 0x7f5b0006e3e0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f5b2813cd90 tx=0x7f5b10008040 comp rx=0 tx=0).stop 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 msgr2=0x7f5b28142ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 0x7f5b28142ef0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f5b18007ef0 tx=0x7f5b1800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 shutdown_connections 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b0006bf20 0x7f5b0006e3e0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b28143c40 0x7f5b280af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b280ccc10 0x7f5b28156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.410+0000 7f5b2d400640 1 --2- 172.21.15.53:0/671227171 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b280717d0 0x7f5b28142ef0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.411+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 >> 172.21.15.53:0/671227171 conn(0x7f5b2806a800 msgr2=0x7f5b2806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.411+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 shutdown_connections 2024-01-16T15:13:49.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.411+0000 7f5b2d400640 1 -- 172.21.15.53:0/671227171 wait complete. 2024-01-16T15:13:49.411 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:13:49.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.655+0000 7fde07400640 1 Processor -- start 2024-01-16T15:13:49.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.655+0000 7fde07400640 1 -- start start 2024-01-16T15:13:49.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde07400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fde00166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde07400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0015c5f0 con 0x7fde0013df80 2024-01-16T15:13:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde07400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde0006c470 con 0x7fde0013e360 2024-01-16T15:13:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde07400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde0006c750 con 0x7fde000cf9a0 2024-01-16T15:13:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde04a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fde0013df80 0x7fde0015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47294/0 (socket says 172.21.15.53:47294) 2024-01-16T15:13:49.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde04a00640 1 -- 172.21.15.53:0/2582467632 learned_addr learned my addr 172.21.15.53:0/2582467632 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.657+0000 7fde05400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fde00166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173148245 0 0) 0x7fde0015c5f0 con 0x7fde0013df80 2024-01-16T15:13:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde0003520 con 0x7fde0013df80 2024-01-16T15:13:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3984000403 0 0) 0x7fdde0003520 con 0x7fde0013df80 2024-01-16T15:13:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 msgr2=0x7fde00166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:49.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fde00166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 >> v1:172.21.15.146:6789/0 conn(0x7fde0013e360 legacy=0x7fde000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0006ca30 con 0x7fde0013df80 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fddf0001670 con 0x7fde0013df80 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 >> v1:172.21.15.53:6789/0 conn(0x7fde0013df80 legacy=0x7fde0015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fdde00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fdde00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdde0004d00 0x7fdde0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0006c750 con 0x7fdde0003ee0 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdde000b9e0 con 0x7fdde0004d00 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.658+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde000bba0 con 0x7fde000cf9a0 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fddffe00640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fdde00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:49.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fde04a00640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fdde00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fde05400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdde0004d00 0x7fdde0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fddffe00640 1 -- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 msgr2=0x7fdde00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fddffe00640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fdde00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fddffe00640 1 -- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdde0004d00 msgr2=0x7fdde0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fddffe00640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdde0004d00 0x7fdde0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fddffe00640 1 -- 172.21.15.53:0/2582467632 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde0003520 con 0x7fdde0003ee0 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fde05400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdde0004d00 0x7fdde0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:49.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.659+0000 7fde04a00640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde000cf9a0 0x7fdde00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:49.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.660+0000 7fddffe00640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fdde00047c0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001830 tx=0x7fdde00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90d315e9e165b69 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:49.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde8032600 con 0x7fdde0003ee0 2024-01-16T15:13:49.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde802e050 con 0x7fdde0003ee0 2024-01-16T15:13:49.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fddff400640 1 -- 172.21.15.53:0/2582467632 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde80329c0 con 0x7fdde0003ee0 2024-01-16T15:13:49.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fde07400640 1 -- 172.21.15.53:0/2582467632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 msgr2=0x7fdde00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fde07400640 1 --2- 172.21.15.53:0/2582467632 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fdde00047c0 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7fdde8001830 tx=0x7fdde00057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fde05400640 1 -- 172.21.15.53:0/2582467632 reap_dead start 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fde07400640 1 -- 172.21.15.53:0/2582467632 shutdown_connections 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.661+0000 7fde07400640 1 -- 172.21.15.53:0/2582467632 >> 172.21.15.53:0/2582467632 conn(0x7fde0006a800 msgr2=0x7fde0006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.662+0000 7fde07400640 1 -- 172.21.15.53:0/2582467632 shutdown_connections 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.662+0000 7fde07400640 1 -- 172.21.15.53:0/2582467632 wait complete. 2024-01-16T15:13:49.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.663+0000 7fde07400640 1 Processor -- start 2024-01-16T15:13:49.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.664+0000 7fde07400640 1 -- start start 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde07400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fde001f0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde07400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0015ae30 0x7fde00150740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde07400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde00150c80 0x7fde000b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde07400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0014e6e0 con 0x7fdde0003ee0 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde07400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0014f1d0 con 0x7fde0015ae30 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde07400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde001f0970 con 0x7fde00150c80 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fde001f0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fde001f0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56100/0 (socket says 172.21.15.53:56100) 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 -- 172.21.15.53:0/2294244048 learned_addr learned my addr 172.21.15.53:0/2294244048 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 -- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde00150c80 msgr2=0x7fde000b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde00150c80 0x7fde000b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 -- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0015ae30 msgr2=0x7fde00150740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0015ae30 0x7fde00150740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.665+0000 7fde04a00640 1 -- 172.21.15.53:0/2294244048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0014e930 con 0x7fdde0003ee0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.666+0000 7fde04a00640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fde001f0430 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fddf000e780 tx=0x7fddf000ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.666+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf000c5e0 con 0x7fdde0003ee0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.666+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddf000c740 con 0x7fdde0003ee0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.666+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf000caf0 con 0x7fdde0003ee0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.666+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde0014ec10 con 0x7fdde0003ee0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.666+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde000b3230 con 0x7fdde0003ee0 2024-01-16T15:13:49.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.667+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde000675d0 con 0x7fdde0003ee0 2024-01-16T15:13:49.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.668+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fddf000cc50 con 0x7fdde0003ee0 2024-01-16T15:13:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.669+0000 7fddfcc00640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddd806bf50 0x7fddd806e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.669+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fddf0088450 con 0x7fdde0003ee0 2024-01-16T15:13:49.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.670+0000 7fddffe00640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddd806bf50 0x7fddd806e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:49.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.671+0000 7fddffe00640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddd806bf50 0x7fddd806e410 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdde8035000 tx=0x7fdde8009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:49.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.674+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fddf00bde90 con 0x7fdde0003ee0 2024-01-16T15:13:49.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.893+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fde0006cff0 con 0x7fddd806bf50 2024-01-16T15:13:49.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.896+0000 7fddfcc00640 1 -- 172.21.15.53:0/2294244048 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdde8032400 con 0x7fddd806bf50 2024-01-16T15:13:49.876 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:13:49.876 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:13:49.876 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:13:49.876 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:13:49.876 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.898+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddd806bf50 msgr2=0x7fddd806e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.898+0000 7fde07400640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddd806bf50 0x7fddd806e410 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fdde8035000 tx=0x7fdde8009030 comp rx=0 tx=0).stop 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.898+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 msgr2=0x7fde001f0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.898+0000 7fde07400640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fde001f0430 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7fddf000e780 tx=0x7fddf000ec40 comp rx=0 tx=0).stop 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 shutdown_connections 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fddd806bf50 0x7fddd806e410 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fde00150c80 0x7fde000b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fde0015ae30 0x7fde00150740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 --2- 172.21.15.53:0/2294244048 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdde0003ee0 0x7fde001f0430 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 >> 172.21.15.53:0/2294244048 conn(0x7fde0006a800 msgr2=0x7fde0015c850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:49.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 shutdown_connections 2024-01-16T15:13:49.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:49.899+0000 7fde07400640 1 -- 172.21.15.53:0/2294244048 wait complete. 2024-01-16T15:13:49.897 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:13:50.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.140+0000 7f19f7a00640 1 Processor -- start 2024-01-16T15:13:50.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.141+0000 7f19f7a00640 1 -- start start 2024-01-16T15:13:50.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.142+0000 7f19f7a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f19f013df80 0x7f19f015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.142+0000 7f19f7a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f013e360 0x7f19f00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.142+0000 7f19f7a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f19f015f9c0 0x7f19f00cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.142+0000 7f19f7a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f006c470 con 0x7f19f013df80 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f7a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f006c750 con 0x7f19f015f9c0 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f7a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f006ca30 con 0x7f19f013e360 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f19f013df80 0x7f19f015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f5000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f19f013df80 0x7f19f015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56116/0 (socket says 172.21.15.53:56116) 2024-01-16T15:13:50.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f5000640 1 -- 172.21.15.53:0/1604949818 learned_addr learned my addr 172.21.15.53:0/1604949818 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:50.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f5a00640 1 --2- 172.21.15.53:0/1604949818 >> v2:172.21.15.146:3300/0 conn(0x7f19f015f9c0 0x7f19f00cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19e7e00640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f013e360 0x7f19f00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.143+0000 7f19f5000640 1 -- 172.21.15.53:0/1604949818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f013e360 msgr2=0x7f19f00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:50.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.144+0000 7f19f5000640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f013e360 0x7f19f00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.144+0000 7f19f5000640 1 -- 172.21.15.53:0/1604949818 >> v2:172.21.15.146:3300/0 conn(0x7f19f015f9c0 msgr2=0x7f19f00cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:50.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.144+0000 7f19f5000640 1 --2- 172.21.15.53:0/1604949818 >> v2:172.21.15.146:3300/0 conn(0x7f19f015f9c0 0x7f19f00cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.144+0000 7f19f5000640 1 -- 172.21.15.53:0/1604949818 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f006cd10 con 0x7f19f013df80 2024-01-16T15:13:50.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.144+0000 7f19f5a00640 1 --2- 172.21.15.53:0/1604949818 >> v2:172.21.15.146:3300/0 conn(0x7f19f015f9c0 0x7f19f00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:13:50.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.144+0000 7f19e7e00640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f013e360 0x7f19f00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:50.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.145+0000 7f19f5000640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013df80 0x7f19f015bee0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f19f006e6d0 tx=0x7f19e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeeb036952b0b99c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:50.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.145+0000 7f19efe00640 1 -- 172.21.15.53:0/1604949818 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0031870 con 0x7f19f013df80 2024-01-16T15:13:50.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.145+0000 7f19efe00640 1 -- 172.21.15.53:0/1604949818 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e00319d0 con 0x7f19f013df80 2024-01-16T15:13:50.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.145+0000 7f19efe00640 1 -- 172.21.15.53:0/1604949818 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0031d60 con 0x7f19f013df80 2024-01-16T15:13:50.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 -- 172.21.15.53:0/1604949818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013df80 msgr2=0x7f19f015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013df80 0x7f19f015bee0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f19f006e6d0 tx=0x7f19e002b910 comp rx=0 tx=0).stop 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 -- 172.21.15.53:0/1604949818 shutdown_connections 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 --2- 172.21.15.53:0/1604949818 >> v2:172.21.15.146:3300/0 conn(0x7f19f015f9c0 0x7f19f00cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f013e360 0x7f19f00cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 --2- 172.21.15.53:0/1604949818 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013df80 0x7f19f015bee0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 -- 172.21.15.53:0/1604949818 >> 172.21.15.53:0/1604949818 conn(0x7f19f006a800 msgr2=0x7f19f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 -- 172.21.15.53:0/1604949818 shutdown_connections 2024-01-16T15:13:50.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.146+0000 7f19f7a00640 1 -- 172.21.15.53:0/1604949818 wait complete. 2024-01-16T15:13:50.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.148+0000 7f19f7a00640 1 Processor -- start 2024-01-16T15:13:50.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.148+0000 7f19f7a00640 1 -- start start 2024-01-16T15:13:50.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f7a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19f013df80 0x7f19f00a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f7a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 0x7f19f00a9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f7a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f015f9c0 0x7f19f01f31b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f7a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f012efd0 con 0x7f19f013e360 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f7a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f012e350 con 0x7f19f013df80 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f7a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f012e630 con 0x7f19f015f9c0 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 0x7f19f00a9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19e7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 0x7f19f00a9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56122/0 (socket says 172.21.15.53:56122) 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19e7e00640 1 -- 172.21.15.53:0/3872166434 learned_addr learned my addr 172.21.15.53:0/3872166434 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f5000640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19f013df80 0x7f19f00a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.149+0000 7f19f5a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f015f9c0 0x7f19f01f31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19e7e00640 1 -- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f015f9c0 msgr2=0x7f19f01f31b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19e7e00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f015f9c0 0x7f19f01f31b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19e7e00640 1 -- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19f013df80 msgr2=0x7f19f00a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19e7e00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19f013df80 0x7f19f00a84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19e7e00640 1 -- 172.21.15.53:0/3872166434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00094d0 con 0x7f19f013e360 2024-01-16T15:13:50.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19f5000640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19f013df80 0x7f19f00a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:50.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19e7e00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 0x7f19f00a9f60 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c7f0 tx=0x7f19d800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8007a30 con 0x7f19f013e360 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8007b90 con 0x7f19f013e360 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d800ce40 con 0x7f19f013e360 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f012e910 con 0x7f19f013e360 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.150+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f00aa4a0 con 0x7f19f013e360 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.152+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f012f110 con 0x7f19f013e360 2024-01-16T15:13:50.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.152+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f19d80113f0 con 0x7f19f013e360 2024-01-16T15:13:50.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.154+0000 7f19ed600640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19c806bf20 0x7f19c806e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.154+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f19d80889b0 con 0x7f19f013e360 2024-01-16T15:13:50.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.154+0000 7f19f5000640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19c806bf20 0x7f19c806e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.156+0000 7f19f5000640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19c806bf20 0x7f19c806e3e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:50.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.158+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19f012f110 con 0x7f19f013e360 2024-01-16T15:13:50.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.352+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f19f012f110 con 0x7f19f013e360 2024-01-16T15:13:50.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.353+0000 7f19ed600640 1 -- 172.21.15.53:0/3872166434 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v30)=0 v30) v1 ==== 122+0+0 (secure 0 0 0) 0x7f19f012f110 con 0x7f19f013e360 2024-01-16T15:13:50.336 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19c806bf20 msgr2=0x7f19c806e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19c806bf20 0x7f19c806e3e0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f19e0036000 tx=0x7f19e003c040 comp rx=0 tx=0).stop 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 msgr2=0x7f19f00a9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 0x7f19f00a9f60 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f19d800c7f0 tx=0x7f19d800cbf0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 shutdown_connections 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19c806bf20 0x7f19c806e3e0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19f015f9c0 0x7f19f01f31b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19f013e360 0x7f19f00a9f60 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 --2- 172.21.15.53:0/3872166434 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19f013df80 0x7f19f00a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 >> 172.21.15.53:0/3872166434 conn(0x7f19f006a800 msgr2=0x7f19f006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 shutdown_connections 2024-01-16T15:13:50.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.356+0000 7f19f7a00640 1 -- 172.21.15.53:0/3872166434 wait complete. 2024-01-16T15:13:50.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:13:50.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.601+0000 7f882e200640 1 Processor -- start 2024-01-16T15:13:50.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.601+0000 7f882e200640 1 -- start start 2024-01-16T15:13:50.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882e200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88280ccb50 0x7f8828070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882e200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f882815bf10 0x7f8828143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882e200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882815f4e0 con 0x7f88280adf20 2024-01-16T15:13:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882e200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f882815f970 con 0x7f882815bf10 2024-01-16T15:13:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882e200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8828071130 con 0x7f88280ccb50 2024-01-16T15:13:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882d800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f882815bf10 0x7f8828143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882d800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f882815bf10 0x7f8828143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56124/0 (socket says 172.21.15.53:56124) 2024-01-16T15:13:50.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.603+0000 7f882d800640 1 -- 172.21.15.53:0/42304570 learned_addr learned my addr 172.21.15.53:0/42304570 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:50.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.604+0000 7f882d800640 1 -- 172.21.15.53:0/42304570 >> v2:172.21.15.146:3300/0 conn(0x7f88280ccb50 msgr2=0x7f8828070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:50.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.604+0000 7f882d800640 1 --2- 172.21.15.53:0/42304570 >> v2:172.21.15.146:3300/0 conn(0x7f88280ccb50 0x7f8828070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.604+0000 7f882d800640 1 -- 172.21.15.53:0/42304570 >> v1:172.21.15.146:6789/0 conn(0x7f88280adf20 legacy=0x7f88280cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:50.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.604+0000 7f882d800640 1 -- 172.21.15.53:0/42304570 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f882815ebb0 con 0x7f882815bf10 2024-01-16T15:13:50.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.605+0000 7f882d800640 1 --2- 172.21.15.53:0/42304570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828143d30 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f882815e9e0 tx=0x7f882002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac31507e02747c3d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:50.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.605+0000 7f8827e00640 1 -- 172.21.15.53:0/42304570 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882002c9f0 con 0x7f882815bf10 2024-01-16T15:13:50.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.605+0000 7f8827e00640 1 -- 172.21.15.53:0/42304570 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882002cb50 con 0x7f882815bf10 2024-01-16T15:13:50.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.605+0000 7f8827e00640 1 -- 172.21.15.53:0/42304570 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88200343f0 con 0x7f882815bf10 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 -- 172.21.15.53:0/42304570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 msgr2=0x7f8828143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 --2- 172.21.15.53:0/42304570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828143d30 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f882815e9e0 tx=0x7f882002be50 comp rx=0 tx=0).stop 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 -- 172.21.15.53:0/42304570 shutdown_connections 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 --2- 172.21.15.53:0/42304570 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828143d30 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 --2- 172.21.15.53:0/42304570 >> v2:172.21.15.146:3300/0 conn(0x7f88280ccb50 0x7f8828070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 -- 172.21.15.53:0/42304570 >> 172.21.15.53:0/42304570 conn(0x7f882806a7d0 msgr2=0x7f882806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 -- 172.21.15.53:0/42304570 shutdown_connections 2024-01-16T15:13:50.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.606+0000 7f882e200640 1 -- 172.21.15.53:0/42304570 wait complete. 2024-01-16T15:13:50.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.608+0000 7f882e200640 1 Processor -- start 2024-01-16T15:13:50.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.608+0000 7f882e200640 1 -- start start 2024-01-16T15:13:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882e200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88280adf20 0x7f8828141060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882e200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88280ccb50 0x7f882813fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882e200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828159b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882e200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88280b3a80 con 0x7f882815bf10 2024-01-16T15:13:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882e200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88281415a0 con 0x7f88280ccb50 2024-01-16T15:13:50.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882e200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88280b34c0 con 0x7f88280adf20 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882d800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56136/0 (socket says 172.21.15.53:56136) 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.609+0000 7f882d800640 1 -- 172.21.15.53:0/4215157454 learned_addr learned my addr 172.21.15.53:0/4215157454 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882ce00640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88280adf20 0x7f8828141060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f881fe00640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88280ccb50 0x7f882813fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882d800640 1 -- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88280adf20 msgr2=0x7f8828141060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882d800640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88280adf20 0x7f8828141060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882d800640 1 -- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88280ccb50 msgr2=0x7f882813fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882d800640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88280ccb50 0x7f882813fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882d800640 1 -- 172.21.15.53:0/4215157454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8820009b30 con 0x7f882815bf10 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882ce00640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88280adf20 0x7f8828141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f881fe00640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88280ccb50 0x7f882813fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882d800640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828159b90 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f8820036040 tx=0x7f8820004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:50.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882003e450 con 0x7f882815bf10 2024-01-16T15:13:50.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f882003e5d0 con 0x7f882815bf10 2024-01-16T15:13:50.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f882003e980 con 0x7f882815bf10 2024-01-16T15:13:50.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.610+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88280b37a0 con 0x7f882815bf10 2024-01-16T15:13:50.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.611+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f882813f7d0 con 0x7f882815bf10 2024-01-16T15:13:50.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.612+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f882813f350 con 0x7f882815bf10 2024-01-16T15:13:50.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.613+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f882003eae0 con 0x7f882815bf10 2024-01-16T15:13:50.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.613+0000 7f8825600640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f880006bf50 0x7f880006e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:50.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.614+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f88200ab370 con 0x7f882815bf10 2024-01-16T15:13:50.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.614+0000 7f882ce00640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f880006bf50 0x7f880006e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:50.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.615+0000 7f882ce00640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f880006bf50 0x7f880006e410 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8818007fd0 tx=0x7f8818009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:50.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.618+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f882813f350 con 0x7f882815bf10 2024-01-16T15:13:50.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.811+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f882813f350 con 0x7f882815bf10 2024-01-16T15:13:50.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.817+0000 7f8825600640 1 -- 172.21.15.53:0/4215157454 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v31)=0 v31) v1 ==== 131+0+0 (secure 0 0 0) 0x7f882813f350 con 0x7f882815bf10 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f880006bf50 msgr2=0x7f880006e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f880006bf50 0x7f880006e410 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8818007fd0 tx=0x7f8818009040 comp rx=0 tx=0).stop 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 msgr2=0x7f8828159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828159b90 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f8820036040 tx=0x7f8820004a40 comp rx=0 tx=0).stop 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 shutdown_connections 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f880006bf50 0x7f880006e410 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f882815bf10 0x7f8828159b90 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88280ccb50 0x7f882813fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 --2- 172.21.15.53:0/4215157454 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88280adf20 0x7f8828141060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:50.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 >> 172.21.15.53:0/4215157454 conn(0x7f882806a7d0 msgr2=0x7f882806c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:50.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 shutdown_connections 2024-01-16T15:13:50.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:50.820+0000 7f882e200640 1 -- 172.21.15.53:0/4215157454 wait complete. 2024-01-16T15:13:50.817 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:13:51.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.059+0000 7fcd93400640 1 Processor -- start 2024-01-16T15:13:51.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.060+0000 7fcd93400640 1 -- start start 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd93400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcd8c15d4b0 0x7fcd8c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd93400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 0x7fcd8c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd93400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd8c0d0320 con 0x7fcd8c0cddf0 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd93400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd8c0b2730 con 0x7fcd8c15d4b0 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd93400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0d1ec0 con 0x7fcd8c0cf9a0 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd92a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcd8c0cddf0 0x7fcd8c0ce280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:47298/0 (socket says 172.21.15.53:47298) 2024-01-16T15:13:51.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.061+0000 7fcd92a00640 1 -- 172.21.15.53:0/1458153577 learned_addr learned my addr 172.21.15.53:0/1458153577 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:51.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd92000640 1 --2- 172.21.15.53:0/1458153577 >> v2:172.21.15.146:3300/0 conn(0x7fcd8c15d4b0 0x7fcd8c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd91600640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 0x7fcd8c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800263468 0 0) 0x7fcd8c0d0320 con 0x7fcd8c0cddf0 2024-01-16T15:13:51.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd6c003520 con 0x7fcd8c0cddf0 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578909556 0 0) 0x7fcd6c003520 con 0x7fcd8c0cddf0 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 msgr2=0x7fcd8c0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 0x7fcd8c0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 >> v2:172.21.15.146:3300/0 conn(0x7fcd8c15d4b0 msgr2=0x7fcd8c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 --2- 172.21.15.53:0/1458153577 >> v2:172.21.15.146:3300/0 conn(0x7fcd8c15d4b0 0x7fcd8c0d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c0b29d0 con 0x7fcd8c0cddf0 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fcd840019f0 con 0x7fcd8c0cddf0 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 >> v1:172.21.15.53:6789/0 conn(0x7fcd8c0cddf0 legacy=0x7fcd8c0ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 0x7fcd6c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c003ef0 0x7fcd6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.062+0000 7fcd90c00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0b2730 con 0x7fcd6c004d00 2024-01-16T15:13:51.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0d1ec0 con 0x7fcd6c003ef0 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c00b980 con 0x7fcd8c0cf9a0 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92a00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92000640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 0x7fcd6c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd91600640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c003ef0 0x7fcd6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92a00640 1 -- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 msgr2=0x7fcd6c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92a00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c0cf9a0 0x7fcd6c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92a00640 1 -- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c003ef0 msgr2=0x7fcd6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92a00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c003ef0 0x7fcd6c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.063+0000 7fcd92a00640 1 -- 172.21.15.53:0/1458153577 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c003520 con 0x7fcd6c004d00 2024-01-16T15:13:51.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.064+0000 7fcd92a00640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400a830 tx=0x7fcd6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d69ee265d7cb101f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:51.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd8402cea0 con 0x7fcd6c004d00 2024-01-16T15:13:51.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd8402f070 con 0x7fcd6c004d00 2024-01-16T15:13:51.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd90c00640 1 -- 172.21.15.53:0/1458153577 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84007530 con 0x7fcd6c004d00 2024-01-16T15:13:51.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd93400640 1 -- 172.21.15.53:0/1458153577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd6c004d00 msgr2=0x7fcd6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd93400640 1 --2- 172.21.15.53:0/1458153577 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd6c004d00 0x7fcd6c005180 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7fcd8400a830 tx=0x7fcd6c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd92a00640 1 -- 172.21.15.53:0/1458153577 reap_dead start 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd93400640 1 -- 172.21.15.53:0/1458153577 shutdown_connections 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd93400640 1 -- 172.21.15.53:0/1458153577 >> 172.21.15.53:0/1458153577 conn(0x7fcd8c0d3000 msgr2=0x7fcd8c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd93400640 1 -- 172.21.15.53:0/1458153577 shutdown_connections 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.065+0000 7fcd93400640 1 -- 172.21.15.53:0/1458153577 wait complete. 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.067+0000 7fcd93400640 1 Processor -- start 2024-01-16T15:13:51.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.067+0000 7fcd93400640 1 -- start start 2024-01-16T15:13:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd93400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c004d00 0x7fcd8c15a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd93400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c1545b0 0x7fcd8c09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 0x7fcd8c15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd93400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c15be50 con 0x7fcd8c15b460 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd93400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c154a30 con 0x7fcd6c004d00 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd93400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd8c0ca530 con 0x7fcd8c1545b0 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd92a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 0x7fcd8c15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd92a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 0x7fcd8c15b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56154/0 (socket says 172.21.15.53:56154) 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.068+0000 7fcd92a00640 1 -- 172.21.15.53:0/197803065 learned_addr learned my addr 172.21.15.53:0/197803065 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92000640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c004d00 0x7fcd8c15a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd91600640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c1545b0 0x7fcd8c09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92a00640 1 -- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c1545b0 msgr2=0x7fcd8c09cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92a00640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c1545b0 0x7fcd8c09cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92a00640 1 -- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c004d00 msgr2=0x7fcd8c15a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92a00640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c004d00 0x7fcd8c15a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92a00640 1 -- 172.21.15.53:0/197803065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd8c15da10 con 0x7fcd8c15b460 2024-01-16T15:13:51.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd92a00640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 0x7fcd8c15b910 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcd8402cbc0 tx=0x7fcd84001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.069+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd84001930 con 0x7fcd8c15b460 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.070+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd84004b60 con 0x7fcd8c15b460 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.070+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd840070d0 con 0x7fcd8c15b460 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.070+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd8c0ca200 con 0x7fcd8c15b460 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.070+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd8c09d7f0 con 0x7fcd8c15b460 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.071+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd8c067450 con 0x7fcd8c15b460 2024-01-16T15:13:51.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.072+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7fcd84031030 con 0x7fcd8c15b460 2024-01-16T15:13:51.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.073+0000 7fcd8a000640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf50 0x7fcd6406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.073+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fcd840aa090 con 0x7fcd8c15b460 2024-01-16T15:13:51.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.074+0000 7fcd92000640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf50 0x7fcd6406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.075+0000 7fcd92000640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf50 0x7fcd6406e410 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0b19b0 tx=0x7fcd7c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:51.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.077+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcd84076950 con 0x7fcd8c15b460 2024-01-16T15:13:51.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.271+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7fcd8c067450 con 0x7fcd8c15b460 2024-01-16T15:13:51.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.278+0000 7fcd8a000640 1 -- 172.21.15.53:0/197803065 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v32)=0 v32) v1 ==== 137+0+0 (secure 0 0 0) 0x7fcd84079f70 con 0x7fcd8c15b460 2024-01-16T15:13:51.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf50 msgr2=0x7fcd6406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf50 0x7fcd6406e410 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fcd8c0b19b0 tx=0x7fcd7c008040 comp rx=0 tx=0).stop 2024-01-16T15:13:51.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 msgr2=0x7fcd8c15b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 0x7fcd8c15b910 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fcd8402cbc0 tx=0x7fcd84001d80 comp rx=0 tx=0).stop 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 shutdown_connections 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcd6406bf50 0x7fcd6406e410 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcd8c15b460 0x7fcd8c15b910 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcd8c1545b0 0x7fcd8c09cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 --2- 172.21.15.53:0/197803065 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcd6c004d00 0x7fcd8c15a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 >> 172.21.15.53:0/197803065 conn(0x7fcd8c0d3000 msgr2=0x7fcd8c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.281+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 shutdown_connections 2024-01-16T15:13:51.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.282+0000 7fcd93400640 1 -- 172.21.15.53:0/197803065 wait complete. 2024-01-16T15:13:51.281 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:13:51.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.528+0000 7f79b9a00640 1 Processor -- start 2024-01-16T15:13:51.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.528+0000 7f79b9a00640 1 -- start start 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79b9a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f79b40b3ae0 0x7f79b415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79b9a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b4070c40 con 0x7f79b40cf290 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79b9a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b406c470 con 0x7f79b40ae5a0 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79b9a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b4071450 con 0x7f79b40b3ae0 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79abe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f79b40b3ae0 0x7f79b415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79abe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f79b40b3ae0 0x7f79b415e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56158/0 (socket says 172.21.15.53:56158) 2024-01-16T15:13:51.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.530+0000 7f79abe00640 1 -- 172.21.15.53:0/1077837178 learned_addr learned my addr 172.21.15.53:0/1077837178 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:51.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.531+0000 7f79abe00640 1 -- 172.21.15.53:0/1077837178 >> v1:172.21.15.146:6789/0 conn(0x7f79b40ae5a0 legacy=0x7f79b406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:51.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.531+0000 7f79abe00640 1 -- 172.21.15.53:0/1077837178 >> v1:172.21.15.53:6789/0 conn(0x7f79b40cf290 legacy=0x7f79b40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.531+0000 7f79abe00640 1 -- 172.21.15.53:0/1077837178 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b406c750 con 0x7f79b40b3ae0 2024-01-16T15:13:51.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.532+0000 7f79abe00640 1 --2- 172.21.15.53:0/1077837178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40b3ae0 0x7f79b415e430 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f799c009870 tx=0x7f799c02ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e756e5425a091a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:51.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.532+0000 7f79b2000640 1 -- 172.21.15.53:0/1077837178 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c02f850 con 0x7f79b40b3ae0 2024-01-16T15:13:51.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.532+0000 7f79b2000640 1 -- 172.21.15.53:0/1077837178 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f799c02f9b0 con 0x7f79b40b3ae0 2024-01-16T15:13:51.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.532+0000 7f79b2000640 1 -- 172.21.15.53:0/1077837178 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f799c02fd40 con 0x7f79b40b3ae0 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 -- 172.21.15.53:0/1077837178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40b3ae0 msgr2=0x7f79b415e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1077837178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40b3ae0 0x7f79b415e430 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f799c009870 tx=0x7f799c02ba30 comp rx=0 tx=0).stop 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 -- 172.21.15.53:0/1077837178 shutdown_connections 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1077837178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40b3ae0 0x7f79b415e430 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 -- 172.21.15.53:0/1077837178 >> 172.21.15.53:0/1077837178 conn(0x7f79b406a800 msgr2=0x7f79b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 -- 172.21.15.53:0/1077837178 shutdown_connections 2024-01-16T15:13:51.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.533+0000 7f79b9a00640 1 -- 172.21.15.53:0/1077837178 wait complete. 2024-01-16T15:13:51.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.535+0000 7f79b9a00640 1 Processor -- start 2024-01-16T15:13:51.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.535+0000 7f79b9a00640 1 -- start start 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 0x7f79b41525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b40b3ae0 0x7f79b41597a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b9a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b40cf290 0x7f79b4159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b9a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4152b10 con 0x7f79b40ae5a0 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b9a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4155080 con 0x7f79b40cf290 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b9a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4153fe0 con 0x7f79b40b3ae0 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 0x7f79b41525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 0x7f79b41525d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56174/0 (socket says 172.21.15.53:56174) 2024-01-16T15:13:51.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.536+0000 7f79b2a00640 1 -- 172.21.15.53:0/1910529050 learned_addr learned my addr 172.21.15.53:0/1910529050 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b3400640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b40cf290 0x7f79b4159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79abe00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b40b3ae0 0x7f79b41597a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b2a00640 1 -- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b40b3ae0 msgr2=0x7f79b41597a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b2a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b40b3ae0 0x7f79b41597a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b2a00640 1 -- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b40cf290 msgr2=0x7f79b4159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b2a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b40cf290 0x7f79b4159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b2a00640 1 -- 172.21.15.53:0/1910529050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f799c0094d0 con 0x7f79b40ae5a0 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79b2a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 0x7f79b41525d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f79a400c430 tx=0x7f79a400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4011790 con 0x7f79b40ae5a0 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a40118f0 con 0x7f79b40ae5a0 2024-01-16T15:13:51.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.537+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a4011ca0 con 0x7f79b40ae5a0 2024-01-16T15:13:51.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.538+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b41542c0 con 0x7f79b40ae5a0 2024-01-16T15:13:51.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.538+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b4154880 con 0x7f79b40ae5a0 2024-01-16T15:13:51.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.539+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b40cf700 con 0x7f79b40ae5a0 2024-01-16T15:13:51.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.540+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f79a401c460 con 0x7f79b40ae5a0 2024-01-16T15:13:51.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.541+0000 7f79aaa00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f798c06bf20 0x7f798c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:51.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.541+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f79a4088530 con 0x7f79b40ae5a0 2024-01-16T15:13:51.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.542+0000 7f79abe00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f798c06bf20 0x7f798c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:51.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.543+0000 7f79abe00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f798c06bf20 0x7f798c06e3e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f799c036000 tx=0x7f799c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:51.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.545+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79a4054df0 con 0x7f79b40ae5a0 2024-01-16T15:13:51.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.768+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f79b4070550 con 0x7f798c06bf20 2024-01-16T15:13:51.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:51.778+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:51.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:51.778+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:51.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:51.778+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:51.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:51.778+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:51.758 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:13:51.778+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:13:51.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.853+0000 7f79aaa00640 1 -- 172.21.15.53:0/1910529050 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f799c02f7f0 con 0x7f798c06bf20 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f798c06bf20 msgr2=0x7f798c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f798c06bf20 0x7f798c06e3e0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f799c036000 tx=0x7f799c008030 comp rx=0 tx=0).stop 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 msgr2=0x7f79b41525d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 0x7f79b41525d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7f79a400c430 tx=0x7f79a400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 shutdown_connections 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f798c06bf20 0x7f798c06e3e0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79b40cf290 0x7f79b4159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b40b3ae0 0x7f79b41597a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.856+0000 7f79b9a00640 1 --2- 172.21.15.53:0/1910529050 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b40ae5a0 0x7f79b41525d0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.857+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 >> 172.21.15.53:0/1910529050 conn(0x7f79b406a800 msgr2=0x7f79b40b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.857+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 shutdown_connections 2024-01-16T15:13:51.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:51.857+0000 7f79b9a00640 1 -- 172.21.15.53:0/1910529050 wait complete. 2024-01-16T15:13:51.859 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:13:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.100+0000 7f552b200640 1 Processor -- start 2024-01-16T15:13:52.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.100+0000 7f552b200640 1 -- start start 2024-01-16T15:13:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f552c0ccb50 0x7f552c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f552c15bf10 0x7f552c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552c15f4e0 con 0x7f552c0adf20 2024-01-16T15:13:52.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552c15f970 con 0x7f552c15bf10 2024-01-16T15:13:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f552c071130 con 0x7f552c0ccb50 2024-01-16T15:13:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f552c15bf10 0x7f552c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f552c15bf10 0x7f552c143d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54744/0 (socket says 172.21.15.53:54744) 2024-01-16T15:13:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 -- 172.21.15.53:0/1546349940 learned_addr learned my addr 172.21.15.53:0/1546349940 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f5529400640 1 --2- 172.21.15.53:0/1546349940 >> v2:172.21.15.146:3300/0 conn(0x7f552c0ccb50 0x7f552c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 -- 172.21.15.53:0/1546349940 >> v2:172.21.15.146:3300/0 conn(0x7f552c0ccb50 msgr2=0x7f552c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 --2- 172.21.15.53:0/1546349940 >> v2:172.21.15.146:3300/0 conn(0x7f552c0ccb50 0x7f552c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 -- 172.21.15.53:0/1546349940 >> v1:172.21.15.146:6789/0 conn(0x7f552c0adf20 legacy=0x7f552c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.102+0000 7f552a800640 1 -- 172.21.15.53:0/1546349940 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f552c15ebb0 con 0x7f552c15bf10 2024-01-16T15:13:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.103+0000 7f5529400640 1 --2- 172.21.15.53:0/1546349940 >> v2:172.21.15.146:3300/0 conn(0x7f552c0ccb50 0x7f552c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:52.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.104+0000 7f552a800640 1 --2- 172.21.15.53:0/1546349940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c15bf10 0x7f552c143d30 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f552c15e9e0 tx=0x7f552402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e59439ee3681466c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:52.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.104+0000 7f5528a00640 1 -- 172.21.15.53:0/1546349940 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552402c9f0 con 0x7f552c15bf10 2024-01-16T15:13:52.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.104+0000 7f5528a00640 1 -- 172.21.15.53:0/1546349940 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552402cb50 con 0x7f552c15bf10 2024-01-16T15:13:52.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.104+0000 7f5528a00640 1 -- 172.21.15.53:0/1546349940 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55240343f0 con 0x7f552c15bf10 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 -- 172.21.15.53:0/1546349940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c15bf10 msgr2=0x7f552c143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 --2- 172.21.15.53:0/1546349940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c15bf10 0x7f552c143d30 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7f552c15e9e0 tx=0x7f552402be50 comp rx=0 tx=0).stop 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 -- 172.21.15.53:0/1546349940 shutdown_connections 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 --2- 172.21.15.53:0/1546349940 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c15bf10 0x7f552c143d30 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 --2- 172.21.15.53:0/1546349940 >> v2:172.21.15.146:3300/0 conn(0x7f552c0ccb50 0x7f552c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 -- 172.21.15.53:0/1546349940 >> 172.21.15.53:0/1546349940 conn(0x7f552c06a7d0 msgr2=0x7f552c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 -- 172.21.15.53:0/1546349940 shutdown_connections 2024-01-16T15:13:52.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.105+0000 7f552b200640 1 -- 172.21.15.53:0/1546349940 wait complete. 2024-01-16T15:13:52.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.107+0000 7f552b200640 1 Processor -- start 2024-01-16T15:13:52.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.107+0000 7f552b200640 1 -- start start 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f552b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 0x7f552c15cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f552b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f552c0ccb50 0x7f552c15d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f552b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f552c15bf10 0x7f552c14c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f552b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c15dd20 con 0x7f552c0adf20 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f552b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552c15e000 con 0x7f552c0ccb50 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f552b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f552c14a1c0 con 0x7f552c15bf10 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f5529e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 0x7f552c15cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f5529e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 0x7f552c15cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54754/0 (socket says 172.21.15.53:54754) 2024-01-16T15:13:52.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.108+0000 7f5529e00640 1 -- 172.21.15.53:0/598291654 learned_addr learned my addr 172.21.15.53:0/598291654 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:52.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529400640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f552c0ccb50 0x7f552c15d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529e00640 1 -- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f552c15bf10 msgr2=0x7f552c14c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529e00640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f552c15bf10 0x7f552c14c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529e00640 1 -- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f552c0ccb50 msgr2=0x7f552c15d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529e00640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f552c0ccb50 0x7f552c15d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529e00640 1 -- 172.21.15.53:0/598291654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5524009b30 con 0x7f552c0adf20 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5529e00640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 0x7f552c15cfa0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c410 tx=0x7f551c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c0035f0 con 0x7f552c0adf20 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f551c003750 con 0x7f552c0adf20 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f551c003b40 con 0x7f552c0adf20 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552c14a4a0 con 0x7f552c0adf20 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.109+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552c15da20 con 0x7f552c0adf20 2024-01-16T15:13:52.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.111+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f552c067450 con 0x7f552c0adf20 2024-01-16T15:13:52.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.111+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f551c003ca0 con 0x7f552c0adf20 2024-01-16T15:13:52.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.113+0000 7f5522000640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550406bf50 0x7f550406e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.113+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f551c0879f0 con 0x7f552c0adf20 2024-01-16T15:13:52.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.114+0000 7f5529400640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550406bf50 0x7f550406e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.115+0000 7f5529400640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550406bf50 0x7f550406e410 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f552c0cc300 tx=0x7f5514008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:52.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.117+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f551c054090 con 0x7f552c0adf20 2024-01-16T15:13:52.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.343+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f552c15c380 con 0x7f550406bf50 2024-01-16T15:13:52.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.354+0000 7f5522000640 1 -- 172.21.15.53:0/598291654 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f551400c040 con 0x7f550406bf50 2024-01-16T15:13:52.334 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550406bf50 msgr2=0x7f550406e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550406bf50 0x7f550406e410 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f552c0cc300 tx=0x7f5514008040 comp rx=0 tx=0).stop 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 msgr2=0x7f552c15cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 0x7f552c15cfa0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7f551c00c410 tx=0x7f551c00c8d0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 shutdown_connections 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f550406bf50 0x7f550406e410 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f552c15bf10 0x7f552c14c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f552c0ccb50 0x7f552c15d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 --2- 172.21.15.53:0/598291654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f552c0adf20 0x7f552c15cfa0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 >> 172.21.15.53:0/598291654 conn(0x7f552c06a7d0 msgr2=0x7f552c06cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:52.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 shutdown_connections 2024-01-16T15:13:52.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.357+0000 7f552b200640 1 -- 172.21.15.53:0/598291654 wait complete. 2024-01-16T15:13:52.355 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:13:52.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.600+0000 7f9cb8e00640 1 Processor -- start 2024-01-16T15:13:52.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.600+0000 7f9cb8e00640 1 -- start start 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb8e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9cb40cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb8e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9cb4071200 0x7f9cb406dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb8e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb415deb0 con 0x7f9cb41618f0 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb8e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb415ea30 con 0x7f9cb4071200 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb8e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb406d860 con 0x7f9cb4157c30 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb1400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9cb41618f0 0x7f9cb4070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41230/0 (socket says 172.21.15.53:41230) 2024-01-16T15:13:52.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.602+0000 7f9cb1400640 1 -- 172.21.15.53:0/3924020760 learned_addr learned my addr 172.21.15.53:0/3924020760 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:13:52.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb2800640 1 --2- 172.21.15.53:0/3924020760 >> v2:172.21.15.146:3300/0 conn(0x7f9cb4071200 0x7f9cb406dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9cb40cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965710600 0 0) 0x7f9cb415deb0 con 0x7f9cb41618f0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c94003520 con 0x7f9cb41618f0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978208352 0 0) 0x7f9c94003520 con 0x7f9cb41618f0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 msgr2=0x7f9cb40cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9cb40cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 >> v2:172.21.15.146:3300/0 conn(0x7f9cb4071200 msgr2=0x7f9cb406dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 --2- 172.21.15.53:0/3924020760 >> v2:172.21.15.146:3300/0 conn(0x7f9cb4071200 0x7f9cb406dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb415e190 con 0x7f9cb41618f0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.603+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9c9c001690 con 0x7f9cb41618f0 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 >> v1:172.21.15.53:6789/0 conn(0x7f9cb41618f0 legacy=0x7f9cb4070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9c94003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9c940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb415ea30 con 0x7f9c94003970 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb406d860 con 0x7f9c94004d00 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9400b980 con 0x7f9cb4157c30 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9c94003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1400640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9c940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb2800640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1e00640 1 -- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 msgr2=0x7f9c940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9c940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1e00640 1 -- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c94004d00 msgr2=0x7f9c94005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1e00640 1 -- 172.21.15.53:0/3924020760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94003520 con 0x7f9c94003970 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb1400640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb4157c30 0x7f9c940047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:52.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.604+0000 7f9cb2800640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c94004d00 0x7f9c94005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:13:52.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9c94003d90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009ac0 tx=0x7f9c940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f7b4ce6f1ff3430 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:52.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca402c6b0 con 0x7f9c94003970 2024-01-16T15:13:52.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca402c810 con 0x7f9c94003970 2024-01-16T15:13:52.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb0a00640 1 -- 172.21.15.53:0/3924020760 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca402cba0 con 0x7f9c94003970 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb8e00640 1 -- 172.21.15.53:0/3924020760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 msgr2=0x7f9c94003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/3924020760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9c94003d90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4009ac0 tx=0x7f9c940057f0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb2800640 1 -- 172.21.15.53:0/3924020760 reap_dead start 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb8e00640 1 -- 172.21.15.53:0/3924020760 shutdown_connections 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.606+0000 7f9cb8e00640 1 -- 172.21.15.53:0/3924020760 >> 172.21.15.53:0/3924020760 conn(0x7f9cb40cf700 msgr2=0x7f9cb40cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.607+0000 7f9cb8e00640 1 -- 172.21.15.53:0/3924020760 shutdown_connections 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.607+0000 7f9cb8e00640 1 -- 172.21.15.53:0/3924020760 wait complete. 2024-01-16T15:13:52.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.608+0000 7f9cb8e00640 1 Processor -- start 2024-01-16T15:13:52.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.609+0000 7f9cb8e00640 1 -- start start 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.609+0000 7f9cb8e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9cb4145b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb8e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb40b3290 0x7f9cb40b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb8e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cb415c5e0 0x7f9cb415ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb8e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb41462e0 con 0x7f9c94003970 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb8e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb4146580 con 0x7f9cb415c5e0 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb8e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb4146860 con 0x7f9cb40b3290 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9cb4145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9cb4145b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:54774/0 (socket says 172.21.15.53:54774) 2024-01-16T15:13:52.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 -- 172.21.15.53:0/843924974 learned_addr learned my addr 172.21.15.53:0/843924974 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb2800640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cb415c5e0 0x7f9cb415ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1400640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb40b3290 0x7f9cb40b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 -- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb40b3290 msgr2=0x7f9cb40b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb40b3290 0x7f9cb40b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 -- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cb415c5e0 msgr2=0x7f9cb415ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cb415c5e0 0x7f9cb415ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.610+0000 7f9cb1e00640 1 -- 172.21.15.53:0/843924974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb4146b40 con 0x7f9c94003970 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.611+0000 7f9cb1e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9cb4145b10 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f9ca402c250 tx=0x7f9ca4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:52.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.611+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4003810 con 0x7f9c94003970 2024-01-16T15:13:52.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.611+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca4003970 con 0x7f9c94003970 2024-01-16T15:13:52.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.611+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4003ce0 con 0x7f9c94003970 2024-01-16T15:13:52.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.611+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb415cfa0 con 0x7f9c94003970 2024-01-16T15:13:52.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.611+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb40b2560 con 0x7f9c94003970 2024-01-16T15:13:52.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.612+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb40675d0 con 0x7f9c94003970 2024-01-16T15:13:52.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.614+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 31) v1 ==== 90661+0+0 (secure 0 0 0) 0x7f9ca4034440 con 0x7f9c94003970 2024-01-16T15:13:52.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.615+0000 7f9caa000640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c8c06bf20 0x7f9c8c06e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:13:52.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.615+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f9ca40aa9e0 con 0x7f9c94003970 2024-01-16T15:13:52.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.616+0000 7f9cb1400640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c8c06bf20 0x7f9c8c06e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:13:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.617+0000 7f9cb1400640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c8c06bf20 0x7f9c8c06e3e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40aee10 tx=0x7f9c9c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:13:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.620+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ca4077250 con 0x7f9c94003970 2024-01-16T15:13:52.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.841+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f9ca4077620 con 0x7f9c94003970 2024-01-16T15:13:52.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.846+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cb415e750 con 0x7f9c8c06bf20 2024-01-16T15:13:52.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.858+0000 7f9caa000640 1 -- 172.21.15.53:0/843924974 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f9c9c00d040 con 0x7f9c8c06bf20 2024-01-16T15:13:52.837 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622 2024-01-16T15:13:52.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c8c06bf20 msgr2=0x7f9c8c06e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c8c06bf20 0x7f9c8c06e3e0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40aee10 tx=0x7f9c9c009040 comp rx=0 tx=0).stop 2024-01-16T15:13:52.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 msgr2=0x7f9cb4145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:13:52.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9cb4145b10 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f9ca402c250 tx=0x7f9ca4004730 comp rx=0 tx=0).stop 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 shutdown_connections 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c8c06bf20 0x7f9c8c06e3e0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cb415c5e0 0x7f9cb415ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cb40b3290 0x7f9cb40b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 --2- 172.21.15.53:0/843924974 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c94003970 0x7f9cb4145b10 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 >> 172.21.15.53:0/843924974 conn(0x7f9cb40cf700 msgr2=0x7f9cb4157120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 shutdown_connections 2024-01-16T15:13:52.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:13:52.860+0000 7f9cb8e00640 1 -- 172.21.15.53:0/843924974 wait complete. 2024-01-16T15:13:52.860 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622 2024-01-16T15:13:52.860 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:13:52.860 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:13:52.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:13:52.910 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:13:54.100 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:13:54.108 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:10.145 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:14:10.146 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:14:10.146 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 17.1555 s, 62.6 MB/s 2024-01-16T15:14:10.147 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:14:10.148 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:10.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:10.205 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:14:24.136 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:14:24.136 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:14:24.136 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 13.867 s, 77.4 MB/s 2024-01-16T15:14:24.138 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:14:24.138 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:14:24.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:24.194 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:14:24.250 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:24.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:38.496 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:14:38.497 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:14:38.497 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.2375 s, 75.4 MB/s 2024-01-16T15:14:38.498 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:14:38.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.752+0000 7f8d5be00640 1 Processor -- start 2024-01-16T15:14:38.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.752+0000 7f8d5be00640 1 -- start start 2024-01-16T15:14:38.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 0x7f8d5c15e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c15e6a0 con 0x7f8d5c06b3f0 2024-01-16T15:14:38.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c071810 con 0x7f8d5c15f3b0 2024-01-16T15:14:38.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c071af0 con 0x7f8d5c15af30 2024-01-16T15:14:38.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5a000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8d5c06b3f0 0x7f8d5c06b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:52726/0 (socket says 172.21.15.53:52726) 2024-01-16T15:14:38.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5a000640 1 -- 172.21.15.53:0/321172397 learned_addr learned my addr 172.21.15.53:0/321172397 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.754+0000 7f8d5aa00640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 0x7f8d5c15e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569104670 0 0) 0x7f8d5c15e6a0 con 0x7f8d5c06b3f0 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d3c003520 con 0x7f8d5c06b3f0 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3042359196 0 0) 0x7f8d3c003520 con 0x7f8d5c06b3f0 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 msgr2=0x7f8d5c15e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 0x7f8d5c15e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 >> v1:172.21.15.146:6789/0 conn(0x7f8d5c15f3b0 legacy=0x7f8d5c15f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c15e8d0 con 0x7f8d5c06b3f0 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8d44001670 con 0x7f8d5c06b3f0 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 >> v1:172.21.15.53:6789/0 conn(0x7f8d5c06b3f0 legacy=0x7f8d5c06b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0039a0 0x7f8d3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c0047c0 0x7f8d3c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 0x7f8d3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c071af0 con 0x7f8d3c0047c0 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c00ba60 con 0x7f8d3c0039a0 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.755+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c00bc20 con 0x7f8d5c15af30 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5a000640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c0047c0 0x7f8d3c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5aa00640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0039a0 0x7f8d3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5b400640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 0x7f8d3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5a000640 1 -- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 msgr2=0x7f8d3c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5a000640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c15af30 0x7f8d3c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5a000640 1 -- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0039a0 msgr2=0x7f8d3c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5a000640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0039a0 0x7f8d3c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.756+0000 7f8d5a000640 1 -- 172.21.15.53:0/321172397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d3c003520 con 0x7f8d3c0047c0 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.757+0000 7f8d5a000640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c0047c0 0x7f8d3c004c40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8d44001eb0 tx=0x7f8d3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1398f7a2f4a569 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d44005ad0 con 0x7f8d3c0047c0 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d44005c30 con 0x7f8d3c0047c0 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d59600640 1 -- 172.21.15.53:0/321172397 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4402ca00 con 0x7f8d3c0047c0 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5be00640 1 -- 172.21.15.53:0/321172397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c0047c0 msgr2=0x7f8d3c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5be00640 1 --2- 172.21.15.53:0/321172397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d3c0047c0 0x7f8d3c004c40 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f8d44001eb0 tx=0x7f8d3c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5b400640 1 -- 172.21.15.53:0/321172397 reap_dead start 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5be00640 1 -- 172.21.15.53:0/321172397 shutdown_connections 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5be00640 1 -- 172.21.15.53:0/321172397 >> 172.21.15.53:0/321172397 conn(0x7f8d5c070950 msgr2=0x7f8d5c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5be00640 1 -- 172.21.15.53:0/321172397 shutdown_connections 2024-01-16T15:14:38.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.758+0000 7f8d5be00640 1 -- 172.21.15.53:0/321172397 wait complete. 2024-01-16T15:14:38.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.760+0000 7f8d5be00640 1 Processor -- start 2024-01-16T15:14:38.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 -- start start 2024-01-16T15:14:38.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0047c0 0x7f8d5c1580a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c0b42b0 0x7f8d5c1f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 0x7f8d5c0c6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c7050 con 0x7f8d5c0c6680 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1585e0 con 0x7f8d3c0047c0 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.761+0000 7f8d5be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c1f3e20 con 0x7f8d5c0b42b0 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 0x7f8d5c0c6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 0x7f8d5c0c6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48110/0 (socket says 172.21.15.53:48110) 2024-01-16T15:14:38.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 -- 172.21.15.53:0/1535210776 learned_addr learned my addr 172.21.15.53:0/1535210776 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:38.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5aa00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0047c0 0x7f8d5c1580a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5a000640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c0b42b0 0x7f8d5c1f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 -- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c0b42b0 msgr2=0x7f8d5c1f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c0b42b0 0x7f8d5c1f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 -- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0047c0 msgr2=0x7f8d5c1580a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0047c0 0x7f8d5c1580a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5b400640 1 -- 172.21.15.53:0/1535210776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c0c4a00 con 0x7f8d5c0c6680 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.762+0000 7f8d5a000640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c0b42b0 0x7f8d5c1f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.763+0000 7f8d5b400640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 0x7f8d5c0c6b10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c003970 tx=0x7f8d4c00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.763+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c003970 con 0x7f8d5c0c6680 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.763+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d4c003ad0 con 0x7f8d5c0c6680 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.763+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d4c003e80 con 0x7f8d5c0c6680 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.763+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c0c4ce0 con 0x7f8d5c0c6680 2024-01-16T15:14:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.763+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c0c52a0 con 0x7f8d5c0c6680 2024-01-16T15:14:38.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.764+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c073b70 con 0x7f8d5c0c6680 2024-01-16T15:14:38.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.766+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8d4c0050d0 con 0x7f8d5c0c6680 2024-01-16T15:14:38.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.766+0000 7f8d52a00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d3406bfb0 0x7f8d3406e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:38.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.766+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f8d4c088660 con 0x7f8d5c0c6680 2024-01-16T15:14:38.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.767+0000 7f8d5aa00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d3406bfb0 0x7f8d3406e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:38.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.769+0000 7f8d5aa00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d3406bfb0 0x7f8d3406e470 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c15fe20 tx=0x7f8d54008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:38.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:38.772+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8d4c054c90 con 0x7f8d5c0c6680 2024-01-16T15:14:38.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.000+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d5c15ee90 con 0x7f8d3406bfb0 2024-01-16T15:14:39.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.035+0000 7f8d52a00640 1 -- 172.21.15.53:0/1535210776 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8d54012950 con 0x7f8d3406bfb0 2024-01-16T15:14:39.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d3406bfb0 msgr2=0x7f8d3406e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d3406bfb0 0x7f8d3406e470 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c15fe20 tx=0x7f8d54008040 comp rx=0 tx=0).stop 2024-01-16T15:14:39.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 msgr2=0x7f8d5c0c6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 0x7f8d5c0c6b10 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c003970 tx=0x7f8d4c00ec10 comp rx=0 tx=0).stop 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 shutdown_connections 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8d3406bfb0 0x7f8d3406e470 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8d5c0c6680 0x7f8d5c0c6b10 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8d5c0b42b0 0x7f8d5c1f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 --2- 172.21.15.53:0/1535210776 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8d3c0047c0 0x7f8d5c1580a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 >> 172.21.15.53:0/1535210776 conn(0x7f8d5c070950 msgr2=0x7f8d5c15b650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 shutdown_connections 2024-01-16T15:14:39.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.038+0000 7f8d5be00640 1 -- 172.21.15.53:0/1535210776 wait complete. 2024-01-16T15:14:39.039 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:14:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.283+0000 7f6aab200640 1 Processor -- start 2024-01-16T15:14:39.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.283+0000 7f6aab200640 1 -- start start 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.284+0000 7f6aab200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6aac0cf9a0 0x7f6aac0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.284+0000 7f6aab200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6aac0b31d0 0x7f6aac0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.284+0000 7f6aab200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aac0d0320 con 0x7f6aac15d4b0 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.284+0000 7f6aab200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aac0d1c60 con 0x7f6aac0b31d0 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.284+0000 7f6aab200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aac0b2730 con 0x7f6aac0cf9a0 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6aac0b31d0 0x7f6aac0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6aac0b31d0 0x7f6aac0b3660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48116/0 (socket says 172.21.15.53:48116) 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 -- 172.21.15.53:0/1500485228 learned_addr learned my addr 172.21.15.53:0/1500485228 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aa9400640 1 --2- 172.21.15.53:0/1500485228 >> v2:172.21.15.146:3300/0 conn(0x7f6aac0cf9a0 0x7f6aac0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 -- 172.21.15.53:0/1500485228 >> v2:172.21.15.146:3300/0 conn(0x7f6aac0cf9a0 msgr2=0x7f6aac0cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:39.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 --2- 172.21.15.53:0/1500485228 >> v2:172.21.15.146:3300/0 conn(0x7f6aac0cf9a0 0x7f6aac0cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 -- 172.21.15.53:0/1500485228 >> v1:172.21.15.146:6789/0 conn(0x7f6aac15d4b0 legacy=0x7f6aac0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:39.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.285+0000 7f6aaa800640 1 -- 172.21.15.53:0/1500485228 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aac0d1ec0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.286+0000 7f6aaa800640 1 --2- 172.21.15.53:0/1500485228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac0b3660 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f6aac08b580 tx=0x7f6aa402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf810f12663fcd8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:39.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.287+0000 7f6aa8a00640 1 -- 172.21.15.53:0/1500485228 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4031870 con 0x7f6aac0b31d0 2024-01-16T15:14:39.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.287+0000 7f6aa8a00640 1 -- 172.21.15.53:0/1500485228 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aa40319d0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.287+0000 7f6aa8a00640 1 -- 172.21.15.53:0/1500485228 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aa4031d60 con 0x7f6aac0b31d0 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.287+0000 7f6aab200640 1 -- 172.21.15.53:0/1500485228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 msgr2=0x7f6aac0b3660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.287+0000 7f6aab200640 1 --2- 172.21.15.53:0/1500485228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac0b3660 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f6aac08b580 tx=0x7f6aa402b910 comp rx=0 tx=0).stop 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.288+0000 7f6aab200640 1 -- 172.21.15.53:0/1500485228 shutdown_connections 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.288+0000 7f6aab200640 1 --2- 172.21.15.53:0/1500485228 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac0b3660 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.288+0000 7f6aab200640 1 --2- 172.21.15.53:0/1500485228 >> v2:172.21.15.146:3300/0 conn(0x7f6aac0cf9a0 0x7f6aac0cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.288+0000 7f6aab200640 1 -- 172.21.15.53:0/1500485228 >> 172.21.15.53:0/1500485228 conn(0x7f6aac0d3000 msgr2=0x7f6aac0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.288+0000 7f6aab200640 1 -- 172.21.15.53:0/1500485228 shutdown_connections 2024-01-16T15:14:39.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.288+0000 7f6aab200640 1 -- 172.21.15.53:0/1500485228 wait complete. 2024-01-16T15:14:39.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.289+0000 7f6aab200640 1 Processor -- start 2024-01-16T15:14:39.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.290+0000 7f6aab200640 1 -- start start 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aab200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aab200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6aac0cf9a0 0x7f6aac09d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aab200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aac15d4b0 0x7f6aac0b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aab200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac0a0da0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aab200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac09dd00 con 0x7f6aac15d4b0 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aab200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aac09dfe0 con 0x7f6aac0cf9a0 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac157e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48122/0 (socket says 172.21.15.53:48122) 2024-01-16T15:14:39.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 -- 172.21.15.53:0/1900548653 learned_addr learned my addr 172.21.15.53:0/1900548653 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:39.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aaa800640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aac15d4b0 0x7f6aac0b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9400640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6aac0cf9a0 0x7f6aac09d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 -- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6aac0cf9a0 msgr2=0x7f6aac09d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6aac0cf9a0 0x7f6aac09d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 -- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aac15d4b0 msgr2=0x7f6aac0b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aac15d4b0 0x7f6aac0b0250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.291+0000 7f6aa9e00640 1 -- 172.21.15.53:0/1900548653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa40094d0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.292+0000 7f6aa9e00640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac157e70 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400b3e0 tx=0x7f6a9400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.292+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94011830 con 0x7f6aac0b31d0 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.292+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a940119b0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.292+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a94011d60 con 0x7f6aac0b31d0 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.292+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aac09e2c0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.292+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aac0cbae0 con 0x7f6aac0b31d0 2024-01-16T15:14:39.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.293+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aac06cc90 con 0x7f6aac0b31d0 2024-01-16T15:14:39.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.294+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f6a9401a400 con 0x7f6aac0b31d0 2024-01-16T15:14:39.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.295+0000 7f6aa2000640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8406bfe0 0x7f6a8406e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.295+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f6a94088780 con 0x7f6aac0b31d0 2024-01-16T15:14:39.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.296+0000 7f6aa9400640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8406bfe0 0x7f6a8406e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.297+0000 7f6aa9400640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8406bfe0 0x7f6a8406e4a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6aac1e96f0 tx=0x7f6a9c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:39.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.300+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6a94055000 con 0x7f6aac0b31d0 2024-01-16T15:14:39.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.526+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6aac0d21a0 con 0x7f6a8406bfe0 2024-01-16T15:14:39.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.537+0000 7f6aa2000640 1 -- 172.21.15.53:0/1900548653 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f6a9c009a80 con 0x7f6a8406bfe0 2024-01-16T15:14:39.519 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622 2024-01-16T15:14:39.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.539+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8406bfe0 msgr2=0x7f6a8406e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.539+0000 7f6aab200640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8406bfe0 0x7f6a8406e4a0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f6aac1e96f0 tx=0x7f6a9c00b040 comp rx=0 tx=0).stop 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.539+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 msgr2=0x7f6aac157e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.539+0000 7f6aab200640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac157e70 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f6a9400b3e0 tx=0x7f6a9400b8a0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 shutdown_connections 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8406bfe0 0x7f6a8406e4a0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6aac15d4b0 0x7f6aac0b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6aac0cf9a0 0x7f6aac09d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 --2- 172.21.15.53:0/1900548653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6aac0b31d0 0x7f6aac157e70 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 >> 172.21.15.53:0/1900548653 conn(0x7f6aac0d3000 msgr2=0x7f6aac0cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 shutdown_connections 2024-01-16T15:14:39.522 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.540+0000 7f6aab200640 1 -- 172.21.15.53:0/1900548653 wait complete. 2024-01-16T15:14:39.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:39.541 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622) 2024-01-16T15:14:39.598 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622 2024-01-16T15:14:39.598 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:14:39.598 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:14:39.599 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:14:39.599 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 ss1clone1 2024-01-16T15:14:39.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.849+0000 7f1d43800640 1 Processor -- start 2024-01-16T15:14:39.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.849+0000 7f1d43800640 1 -- start start 2024-01-16T15:14:39.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.851+0000 7f1d43800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1d3c13e360 0x7f1d3c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.851+0000 7f1d43800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c15c5f0 con 0x7f1d3c15f9c0 2024-01-16T15:14:39.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.851+0000 7f1d43800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c06c470 con 0x7f1d3c13df80 2024-01-16T15:14:39.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.851+0000 7f1d43800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d3c06c750 con 0x7f1d3c13e360 2024-01-16T15:14:39.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.851+0000 7f1d41800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1d3c15f9c0 0x7f1d3c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:52740/0 (socket says 172.21.15.53:52740) 2024-01-16T15:14:39.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.851+0000 7f1d41800640 1 -- 172.21.15.53:0/4142507392 learned_addr learned my addr 172.21.15.53:0/4142507392 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3be00640 1 --2- 172.21.15.53:0/4142507392 >> v2:172.21.15.146:3300/0 conn(0x7f1d3c13e360 0x7f1d3c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3579339285 0 0) 0x7f1d3c15c5f0 con 0x7f1d3c15f9c0 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d1c003520 con 0x7f1d3c15f9c0 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1556934782 0 0) 0x7f1d1c003520 con 0x7f1d3c15f9c0 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 >> v2:172.21.15.146:3300/0 conn(0x7f1d3c13e360 msgr2=0x7f1d3c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 --2- 172.21.15.53:0/4142507392 >> v2:172.21.15.146:3300/0 conn(0x7f1d3c13e360 0x7f1d3c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 >> v1:172.21.15.146:6789/0 conn(0x7f1d3c13df80 legacy=0x7f1d3c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c06ca30 con 0x7f1d3c15f9c0 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1d34001a10 con 0x7f1d3c15f9c0 2024-01-16T15:14:39.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 >> v1:172.21.15.53:6789/0 conn(0x7f1d3c15f9c0 legacy=0x7f1d3c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d1c0047c0 0x7f1d1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d1c005180 0x7f1d1c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.852+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c06c750 con 0x7f1d1c003970 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c00c020 con 0x7f1d1c005180 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d1c00c1e0 con 0x7f1d1c0047c0 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d40e00640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d40e00640 1 -- 172.21.15.53:0/4142507392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d1c0047c0 msgr2=0x7f1d1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d40e00640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d1c0047c0 0x7f1d1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d40e00640 1 -- 172.21.15.53:0/4142507392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d1c005180 msgr2=0x7f1d1c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:39.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d40e00640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d1c005180 0x7f1d1c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d40e00640 1 -- 172.21.15.53:0/4142507392 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d1c003520 con 0x7f1d1c003970 2024-01-16T15:14:39.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.853+0000 7f1d41800640 1 -- 172.21.15.53:0/4142507392 reap_dead start 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.854+0000 7f1d40e00640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d1c003d90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c001750 tx=0x7f1d1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d55745d753fae21e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0052d0 con 0x7f1d1c003970 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c02e050 con 0x7f1d1c003970 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d3b400640 1 -- 172.21.15.53:0/4142507392 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c003830 con 0x7f1d1c003970 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 -- 172.21.15.53:0/4142507392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 msgr2=0x7f1d1c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d1c003d90 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c001750 tx=0x7f1d1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 -- 172.21.15.53:0/4142507392 shutdown_connections 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 --2- 172.21.15.53:0/4142507392 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d1c003d90 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 -- 172.21.15.53:0/4142507392 >> 172.21.15.53:0/4142507392 conn(0x7f1d3c06a800 msgr2=0x7f1d3c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 -- 172.21.15.53:0/4142507392 shutdown_connections 2024-01-16T15:14:39.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.855+0000 7f1d43800640 1 -- 172.21.15.53:0/4142507392 wait complete. 2024-01-16T15:14:39.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.857+0000 7f1d43800640 1 Processor -- start 2024-01-16T15:14:39.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.857+0000 7f1d43800640 1 -- start start 2024-01-16T15:14:39.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.858+0000 7f1d43800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d3c159510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.858+0000 7f1d43800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c158690 0x7f1d3c139250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.858+0000 7f1d43800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c151250 0x7f1d3c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.858+0000 7f1d43800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c159a50 con 0x7f1d1c003970 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.858+0000 7f1d43800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0b1ca0 con 0x7f1d3c158690 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.858+0000 7f1d43800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c139790 con 0x7f1d3c151250 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d3c159510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d3c159510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48132/0 (socket says 172.21.15.53:48132) 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 -- 172.21.15.53:0/437703094 learned_addr learned my addr 172.21.15.53:0/437703094 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d41800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c151250 0x7f1d3c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d3be00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c158690 0x7f1d3c139250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 -- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c151250 msgr2=0x7f1d3c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c151250 0x7f1d3c0702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 -- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c158690 msgr2=0x7f1d3c139250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c158690 0x7f1d3c139250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.859+0000 7f1d40e00640 1 -- 172.21.15.53:0/437703094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0b1ef0 con 0x7f1d1c003970 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.860+0000 7f1d40e00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d3c159510 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c02bc00 tx=0x7f1d2c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:39.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.860+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0076c0 con 0x7f1d1c003970 2024-01-16T15:14:39.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.860+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c007a30 con 0x7f1d1c003970 2024-01-16T15:14:39.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.860+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c0b21d0 con 0x7f1d1c003970 2024-01-16T15:14:39.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.860+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c006be0 con 0x7f1d1c003970 2024-01-16T15:14:39.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.860+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c0b2790 con 0x7f1d1c003970 2024-01-16T15:14:39.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.861+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c0675d0 con 0x7f1d1c003970 2024-01-16T15:14:39.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.862+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f1d2c006d40 con 0x7f1d1c003970 2024-01-16T15:14:39.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.863+0000 7f1d38c00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d1406bf40 0x7f1d1406e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:39.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.863+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f1d2c0aa220 con 0x7f1d1c003970 2024-01-16T15:14:39.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.864+0000 7f1d3be00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d1406bf40 0x7f1d1406e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:39.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.865+0000 7f1d3be00640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d1406bf40 0x7f1d1406e400 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c158ad0 tx=0x7f1d24008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:39.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:39.868+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1d2c076ae0 con 0x7f1d1c003970 2024-01-16T15:14:40.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.098+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1d3c06cff0 con 0x7f1d1406bf40 2024-01-16T15:14:40.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.257+0000 7f1d38c00640 1 -- 172.21.15.53:0/437703094 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1d24009640 con 0x7f1d1406bf40 2024-01-16T15:14:40.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d1406bf40 msgr2=0x7f1d1406e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d1406bf40 0x7f1d1406e400 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c158ad0 tx=0x7f1d24008040 comp rx=0 tx=0).stop 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 msgr2=0x7f1d3c159510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d3c159510 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c02bc00 tx=0x7f1d2c001120 comp rx=0 tx=0).stop 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 shutdown_connections 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1d1406bf40 0x7f1d1406e400 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1d3c151250 0x7f1d3c0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1d3c158690 0x7f1d3c139250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 --2- 172.21.15.53:0/437703094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1d1c003970 0x7f1d3c159510 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.260+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 >> 172.21.15.53:0/437703094 conn(0x7f1d3c06a800 msgr2=0x7f1d3c15d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.261+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 shutdown_connections 2024-01-16T15:14:40.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.261+0000 7f1d43800640 1 -- 172.21.15.53:0/437703094 wait complete. 2024-01-16T15:14:40.246 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.263+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.246 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.263+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.246 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.263+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.246 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.263+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.246 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.264+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.264 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:14:40.265 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:14:40.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.290+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.290+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.290+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.290+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:14:40.290+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:14:40.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.499+0000 7f4513e00640 1 Processor -- start 2024-01-16T15:14:40.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.499+0000 7f4513e00640 1 -- start start 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f451415d4b0 0x7f45140d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45140b41e0 0x7f45140cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45140d0320 con 0x7f45140cf9a0 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140b2730 con 0x7f45140b41e0 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45140d1ec0 con 0x7f451415d4b0 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45140b41e0 0x7f45140cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f45140b41e0 0x7f45140cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48134/0 (socket says 172.21.15.53:48134) 2024-01-16T15:14:40.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.500+0000 7f4513400640 1 -- 172.21.15.53:0/3335959418 learned_addr learned my addr 172.21.15.53:0/3335959418 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:40.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.501+0000 7f4512a00640 1 --2- 172.21.15.53:0/3335959418 >> v2:172.21.15.146:3300/0 conn(0x7f451415d4b0 0x7f45140d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.501+0000 7f4513400640 1 -- 172.21.15.53:0/3335959418 >> v2:172.21.15.146:3300/0 conn(0x7f451415d4b0 msgr2=0x7f45140d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:40.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.501+0000 7f4513400640 1 --2- 172.21.15.53:0/3335959418 >> v2:172.21.15.146:3300/0 conn(0x7f451415d4b0 0x7f45140d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.501+0000 7f4513400640 1 -- 172.21.15.53:0/3335959418 >> v1:172.21.15.53:6789/0 conn(0x7f45140cf9a0 legacy=0x7f45140cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.501+0000 7f4513400640 1 -- 172.21.15.53:0/3335959418 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140b29d0 con 0x7f45140b41e0 2024-01-16T15:14:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.502+0000 7f4513400640 1 --2- 172.21.15.53:0/3335959418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140b41e0 0x7f45140cfde0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f451408b5b0 tx=0x7f450c02bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8774de0a7569e506 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.502+0000 7f4511600640 1 -- 172.21.15.53:0/3335959418 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c02cb10 con 0x7f45140b41e0 2024-01-16T15:14:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.502+0000 7f4511600640 1 -- 172.21.15.53:0/3335959418 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f450c02cc70 con 0x7f45140b41e0 2024-01-16T15:14:40.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.502+0000 7f4511600640 1 -- 172.21.15.53:0/3335959418 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f450c0343f0 con 0x7f45140b41e0 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 -- 172.21.15.53:0/3335959418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140b41e0 msgr2=0x7f45140cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 --2- 172.21.15.53:0/3335959418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140b41e0 0x7f45140cfde0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f451408b5b0 tx=0x7f450c02bf70 comp rx=0 tx=0).stop 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 -- 172.21.15.53:0/3335959418 shutdown_connections 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 --2- 172.21.15.53:0/3335959418 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140b41e0 0x7f45140cfde0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 --2- 172.21.15.53:0/3335959418 >> v2:172.21.15.146:3300/0 conn(0x7f451415d4b0 0x7f45140d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 -- 172.21.15.53:0/3335959418 >> 172.21.15.53:0/3335959418 conn(0x7f45140d3000 msgr2=0x7f45140d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 -- 172.21.15.53:0/3335959418 shutdown_connections 2024-01-16T15:14:40.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.503+0000 7f4513e00640 1 -- 172.21.15.53:0/3335959418 wait complete. 2024-01-16T15:14:40.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.505+0000 7f4513e00640 1 Processor -- start 2024-01-16T15:14:40.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.505+0000 7f4513e00640 1 -- start start 2024-01-16T15:14:40.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4513e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45140b41e0 0x7f45140b8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4513e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 0x7f451415c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4513e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f451415d4b0 0x7f451409f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4513e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451409f7a0 con 0x7f45140cf9a0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4513e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b88b0 con 0x7f451415d4b0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4513e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45140b5d50 con 0x7f45140b41e0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4512000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 0x7f451415c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4512000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 0x7f451415c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48138/0 (socket says 172.21.15.53:48138) 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.506+0000 7f4512000640 1 -- 172.21.15.53:0/3437325821 learned_addr learned my addr 172.21.15.53:0/3437325821 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512a00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45140b41e0 0x7f45140b8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4513400640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f451415d4b0 0x7f451409f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512000640 1 -- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45140b41e0 msgr2=0x7f45140b8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512000640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45140b41e0 0x7f45140b8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512000640 1 -- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f451415d4b0 msgr2=0x7f451409f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512000640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f451415d4b0 0x7f451409f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512000640 1 -- 172.21.15.53:0/3437325821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c009b30 con 0x7f45140cf9a0 2024-01-16T15:14:40.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4513400640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f451415d4b0 0x7f451409f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4512000640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 0x7f451415c8a0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00c450 tx=0x7f44fc00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc0052d0 con 0x7f45140cf9a0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44fc005430 con 0x7f45140cf9a0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44fc012870 con 0x7f45140cf9a0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.507+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451415da10 con 0x7f45140cf9a0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.508+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45140b6310 con 0x7f45140cf9a0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.509+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4514067450 con 0x7f45140cf9a0 2024-01-16T15:14:40.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.510+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f44fc0129d0 con 0x7f45140cf9a0 2024-01-16T15:14:40.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.511+0000 7f450aa00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f44ec06bfb0 0x7f44ec06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:40.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.511+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f44fc0880f0 con 0x7f45140cf9a0 2024-01-16T15:14:40.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.511+0000 7f4512a00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f44ec06bfb0 0x7f44ec06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:40.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.513+0000 7f4512a00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f44ec06bfb0 0x7f44ec06e470 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4504006fd0 tx=0x7f4504008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:40.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.515+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f44fc054a10 con 0x7f45140cf9a0 2024-01-16T15:14:40.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.747+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f45140cfe50 con 0x7f44ec06bfb0 2024-01-16T15:14:40.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.766+0000 7f450aa00640 1 -- 172.21.15.53:0/3437325821 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f45140cfe50 con 0x7f44ec06bfb0 2024-01-16T15:14:40.748 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:14:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:14:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:14:40.748 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:14:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:14:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:14:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:14:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:40.749 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:40.749 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:14:40.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f44ec06bfb0 msgr2=0x7f44ec06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f44ec06bfb0 0x7f44ec06e470 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4504006fd0 tx=0x7f4504008040 comp rx=0 tx=0).stop 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 msgr2=0x7f451415c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 0x7f451415c8a0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7f44fc00c450 tx=0x7f44fc00c910 comp rx=0 tx=0).stop 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 shutdown_connections 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f44ec06bfb0 0x7f44ec06e470 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f451415d4b0 0x7f451409f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f45140cf9a0 0x7f451415c8a0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 --2- 172.21.15.53:0/3437325821 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f45140b41e0 0x7f45140b8a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.769+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 >> 172.21.15.53:0/3437325821 conn(0x7f45140d3000 msgr2=0x7f45140b49d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.770+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 shutdown_connections 2024-01-16T15:14:40.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:40.770+0000 7f4513e00640 1 -- 172.21.15.53:0/3437325821 wait complete. 2024-01-16T15:14:40.770 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs ss1clone1 2024-01-16T15:14:40.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.016+0000 7f8e8ac00640 1 Processor -- start 2024-01-16T15:14:40.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.016+0000 7f8e8ac00640 1 -- start start 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8406c730 0x7f8e840b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8ac00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e8406cb10 0x7f8e8406cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8ac00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e840708d0 con 0x7f8e84070160 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8ac00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e84070bb0 con 0x7f8e8406cb10 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84070e90 con 0x7f8e8406c730 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e8406cb10 0x7f8e8406cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e8406cb10 0x7f8e8406cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48146/0 (socket says 172.21.15.53:48146) 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.018+0000 7f8e8a200640 1 -- 172.21.15.53:0/2371640884 learned_addr learned my addr 172.21.15.53:0/2371640884 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.019+0000 7f8e89800640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8406c730 0x7f8e840b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.019+0000 7f8e8a200640 1 -- 172.21.15.53:0/2371640884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8406c730 msgr2=0x7f8e840b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.019+0000 7f8e8a200640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8406c730 0x7f8e840b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.019+0000 7f8e8a200640 1 -- 172.21.15.53:0/2371640884 >> v1:172.21.15.146:6789/0 conn(0x7f8e84070160 legacy=0x7f8e84071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:41.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.019+0000 7f8e8a200640 1 -- 172.21.15.53:0/2371640884 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e84071170 con 0x7f8e8406cb10 2024-01-16T15:14:41.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.019+0000 7f8e89800640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8406c730 0x7f8e840b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.020+0000 7f8e8a200640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e8406cf80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009e50 tx=0x7f8e7c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8d69dd9a0d3ae57 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.021+0000 7f8e83e00640 1 -- 172.21.15.53:0/2371640884 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c02ca70 con 0x7f8e8406cb10 2024-01-16T15:14:41.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.021+0000 7f8e83e00640 1 -- 172.21.15.53:0/2371640884 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7c02cbd0 con 0x7f8e8406cb10 2024-01-16T15:14:41.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.021+0000 7f8e83e00640 1 -- 172.21.15.53:0/2371640884 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7c0343f0 con 0x7f8e8406cb10 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.021+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2371640884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 msgr2=0x7f8e8406cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.021+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e8406cf80 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7f8e7c009e50 tx=0x7f8e7c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.022+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2371640884 shutdown_connections 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.022+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e8406cf80 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.022+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2371640884 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e8406c730 0x7f8e840b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.022+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2371640884 >> 172.21.15.53:0/2371640884 conn(0x7f8e8406a7d0 msgr2=0x7f8e8406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.022+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2371640884 shutdown_connections 2024-01-16T15:14:41.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.022+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2371640884 wait complete. 2024-01-16T15:14:41.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.024+0000 7f8e8ac00640 1 Processor -- start 2024-01-16T15:14:41.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.024+0000 7f8e8ac00640 1 -- start start 2024-01-16T15:14:41.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.025+0000 7f8e8ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e8406c730 0x7f8e840aaea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.025+0000 7f8e8ac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e84143e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.025+0000 7f8e8ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e84070160 0x7f8e840ae7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.025+0000 7f8e8ac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e840a94b0 con 0x7f8e8406cb10 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.025+0000 7f8e8ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e84144370 con 0x7f8e8406c730 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.025+0000 7f8e8ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e840a88a0 con 0x7f8e84070160 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e84143e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e84143e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48158/0 (socket says 172.21.15.53:48158) 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 -- 172.21.15.53:0/2237667812 learned_addr learned my addr 172.21.15.53:0/2237667812 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e89800640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e8406c730 0x7f8e840aaea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e8a200640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e84070160 0x7f8e840ae7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 -- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e84070160 msgr2=0x7f8e840ae7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e84070160 0x7f8e840ae7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 -- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e8406c730 msgr2=0x7f8e840aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e8406c730 0x7f8e840aaea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:41.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e88e00640 1 -- 172.21.15.53:0/2237667812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e7c009b30 con 0x7f8e8406cb10 2024-01-16T15:14:41.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.026+0000 7f8e8a200640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e84070160 0x7f8e840ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:41.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.027+0000 7f8e88e00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e84143e30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00b3f0 tx=0x7f8e6c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.027+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c011830 con 0x7f8e8406cb10 2024-01-16T15:14:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.027+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e6c011990 con 0x7f8e8406cb10 2024-01-16T15:14:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.027+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c011d40 con 0x7f8e8406cb10 2024-01-16T15:14:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.027+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e840a8b80 con 0x7f8e8406cb10 2024-01-16T15:14:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.027+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e840a9140 con 0x7f8e8406cb10 2024-01-16T15:14:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.029+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e840a9660 con 0x7f8e8406cb10 2024-01-16T15:14:41.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.030+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 32) v1 ==== 90712+0+0 (secure 0 0 0) 0x7f8e6c01a400 con 0x7f8e8406cb10 2024-01-16T15:14:41.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.031+0000 7f8e81600640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5c06bfb0 0x7f8e5c06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:41.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.031+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f8e6c088410 con 0x7f8e8406cb10 2024-01-16T15:14:41.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.031+0000 7f8e89800640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5c06bfb0 0x7f8e5c06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:41.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.033+0000 7f8e89800640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5c06bfb0 0x7f8e5c06e470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400a990 tx=0x7f8e74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:41.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.035+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e6c0bde90 con 0x7f8e8406cb10 2024-01-16T15:14:41.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.273+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e8406d4c0 con 0x7f8e5c06bfb0 2024-01-16T15:14:41.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:41.487+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f8e6c0550a0 con 0x7f8e8406cb10 2024-01-16T15:14:54.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.309+0000 7f8e81600640 1 -- 172.21.15.53:0/2237667812 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8e7400a660 con 0x7f8e5c06bfb0 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5c06bfb0 msgr2=0x7f8e5c06e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5c06bfb0 0x7f8e5c06e470 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f8e7400a990 tx=0x7f8e74008040 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 msgr2=0x7f8e84143e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e84143e30 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00b3f0 tx=0x7f8e6c00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 shutdown_connections 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e5c06bfb0 0x7f8e5c06e470 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e84070160 0x7f8e840ae7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e8406cb10 0x7f8e84143e30 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 --2- 172.21.15.53:0/2237667812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e8406c730 0x7f8e840aaea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 >> 172.21.15.53:0/2237667812 conn(0x7f8e8406a7d0 msgr2=0x7f8e84144b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 shutdown_connections 2024-01-16T15:14:54.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:54.313+0000 7f8e8ac00640 1 -- 172.21.15.53:0/2237667812 wait complete. 2024-01-16T15:14:54.381 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:54.390 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:14:56.319 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs ss1clone1 2024-01-16T15:14:56.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.571+0000 7fddb3e00640 1 Processor -- start 2024-01-16T15:14:56.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.571+0000 7fddb3e00640 1 -- start start 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddb406c730 0x7fddb40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fddb406cb10 0x7fddb406cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb40708d0 con 0x7fddb4070160 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb4070bb0 con 0x7fddb406c730 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb4070e90 con 0x7fddb406cb10 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddb406c730 0x7fddb40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fddb406c730 0x7fddb40b2e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39568/0 (socket says 172.21.15.53:39568) 2024-01-16T15:14:56.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.573+0000 7fddb2a00640 1 -- 172.21.15.53:0/2182367799 learned_addr learned my addr 172.21.15.53:0/2182367799 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:56.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.574+0000 7fddb3400640 1 --2- 172.21.15.53:0/2182367799 >> v2:172.21.15.146:3300/0 conn(0x7fddb406cb10 0x7fddb406cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.574+0000 7fddb2a00640 1 -- 172.21.15.53:0/2182367799 >> v2:172.21.15.146:3300/0 conn(0x7fddb406cb10 msgr2=0x7fddb406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.574+0000 7fddb2a00640 1 --2- 172.21.15.53:0/2182367799 >> v2:172.21.15.146:3300/0 conn(0x7fddb406cb10 0x7fddb406cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.574+0000 7fddb2a00640 1 -- 172.21.15.53:0/2182367799 >> v1:172.21.15.53:6789/0 conn(0x7fddb4070160 legacy=0x7fddb4071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:56.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.574+0000 7fddb2a00640 1 -- 172.21.15.53:0/2182367799 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb4071170 con 0x7fddb406c730 2024-01-16T15:14:56.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.574+0000 7fddb3400640 1 --2- 172.21.15.53:0/2182367799 >> v2:172.21.15.146:3300/0 conn(0x7fddb406cb10 0x7fddb406cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:14:56.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.575+0000 7fddb2a00640 1 --2- 172.21.15.53:0/2182367799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406c730 0x7fddb40b2e00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fddb406e530 tx=0x7fdda402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=401c771c192f0a74 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb2000640 1 -- 172.21.15.53:0/2182367799 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031890 con 0x7fddb406c730 2024-01-16T15:14:56.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb2000640 1 -- 172.21.15.53:0/2182367799 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda40319f0 con 0x7fddb406c730 2024-01-16T15:14:56.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb2000640 1 -- 172.21.15.53:0/2182367799 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda4031d80 con 0x7fddb406c730 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 -- 172.21.15.53:0/2182367799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406c730 msgr2=0x7fddb40b2e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 --2- 172.21.15.53:0/2182367799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406c730 0x7fddb40b2e00 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fddb406e530 tx=0x7fdda402b930 comp rx=0 tx=0).stop 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 -- 172.21.15.53:0/2182367799 shutdown_connections 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 --2- 172.21.15.53:0/2182367799 >> v2:172.21.15.146:3300/0 conn(0x7fddb406cb10 0x7fddb406cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 --2- 172.21.15.53:0/2182367799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406c730 0x7fddb40b2e00 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 -- 172.21.15.53:0/2182367799 >> 172.21.15.53:0/2182367799 conn(0x7fddb406a7d0 msgr2=0x7fddb406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 -- 172.21.15.53:0/2182367799 shutdown_connections 2024-01-16T15:14:56.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.576+0000 7fddb3e00640 1 -- 172.21.15.53:0/2182367799 wait complete. 2024-01-16T15:14:56.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.578+0000 7fddb3e00640 1 Processor -- start 2024-01-16T15:14:56.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.578+0000 7fddb3e00640 1 -- start start 2024-01-16T15:14:56.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.579+0000 7fddb3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb406c730 0x7fddb41454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.579+0000 7fddb3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 0x7fddb4145cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.579+0000 7fddb3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb4070160 0x7fddb4146210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.579+0000 7fddb3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb40add40 con 0x7fddb406cb10 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.579+0000 7fddb3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41e8830 con 0x7fddb406c730 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.579+0000 7fddb3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb41e9290 con 0x7fddb4070160 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 0x7fddb4145cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 0x7fddb4145cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39580/0 (socket says 172.21.15.53:39580) 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 -- 172.21.15.53:0/862588250 learned_addr learned my addr 172.21.15.53:0/862588250 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fddb2a00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb406c730 0x7fddb41454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fddb3400640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb4070160 0x7fddb4146210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 -- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb4070160 msgr2=0x7fddb4146210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb4070160 0x7fddb4146210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 -- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb406c730 msgr2=0x7fddb41454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb406c730 0x7fddb41454d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:56.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.580+0000 7fdda9e00640 1 -- 172.21.15.53:0/862588250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40094d0 con 0x7fddb406cb10 2024-01-16T15:14:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.581+0000 7fdda9e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 0x7fddb4145cd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00b3f0 tx=0x7fdd9c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.581+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c011830 con 0x7fddb406cb10 2024-01-16T15:14:56.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.581+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd9c011990 con 0x7fddb406cb10 2024-01-16T15:14:56.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.581+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb40adf20 con 0x7fddb406cb10 2024-01-16T15:14:56.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.581+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c011d40 con 0x7fddb406cb10 2024-01-16T15:14:56.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.581+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb41e8a90 con 0x7fddb406cb10 2024-01-16T15:14:56.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.582+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb4067450 con 0x7fddb406cb10 2024-01-16T15:14:56.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.583+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fdd9c01a400 con 0x7fddb406cb10 2024-01-16T15:14:56.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.584+0000 7fddab400640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c0d0 0x7fdd8c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:56.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.584+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fdd9c088790 con 0x7fddb406cb10 2024-01-16T15:14:56.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.585+0000 7fddb2a00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c0d0 0x7fdd8c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:56.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.586+0000 7fddb2a00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c0d0 0x7fdd8c06e590 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:56.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.589+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fdd9c054d90 con 0x7fddb406cb10 2024-01-16T15:14:56.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:56.824+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7fddb406d480 con 0x7fdd8c06c0d0 2024-01-16T15:14:59.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.159+0000 7fddab400640 1 -- 172.21.15.53:0/862588250 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7fdda4031d30 con 0x7fdd8c06c0d0 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:14:59.143 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.144 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.144 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:14:59.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c0d0 msgr2=0x7fdd8c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c0d0 0x7fdd8c06e590 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fdda4036000 tx=0x7fdda403c040 comp rx=0 tx=0).stop 2024-01-16T15:14:59.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 msgr2=0x7fddb4145cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 0x7fddb4145cd0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00b3f0 tx=0x7fdd9c00b8b0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 shutdown_connections 2024-01-16T15:14:59.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fdd8c06c0d0 0x7fdd8c06e590 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fddb4070160 0x7fddb4146210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fddb406cb10 0x7fddb4145cd0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 --2- 172.21.15.53:0/862588250 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fddb406c730 0x7fddb41454d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.163+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 >> 172.21.15.53:0/862588250 conn(0x7fddb406a7d0 msgr2=0x7fddb40af030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.164+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 shutdown_connections 2024-01-16T15:14:59.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.164+0000 7fddb3e00640 1 -- 172.21.15.53:0/862588250 wait complete. 2024-01-16T15:14:59.168 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:14:59.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.401+0000 7f2436c00640 1 Processor -- start 2024-01-16T15:14:59.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.401+0000 7f2436c00640 1 -- start start 2024-01-16T15:14:59.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.402+0000 7f2436c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24300cd600 0x7f2430070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.402+0000 7f2436c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24300b36f0 con 0x7f243015e8d0 2024-01-16T15:14:59.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.402+0000 7f2436c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2430071480 con 0x7f2430071030 2024-01-16T15:14:59.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.402+0000 7f2436c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24300cda60 con 0x7f24300cd600 2024-01-16T15:14:59.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.403+0000 7f242fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f243015e8d0 0x7f243006a3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40420/0 (socket says 172.21.15.53:40420) 2024-01-16T15:14:59.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.403+0000 7f242fe00640 1 -- 172.21.15.53:0/3783650997 learned_addr learned my addr 172.21.15.53:0/3783650997 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.403+0000 7f242f400640 1 --2- 172.21.15.53:0/3783650997 >> v2:172.21.15.146:3300/0 conn(0x7f24300cd600 0x7f2430070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.403+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725640232 0 0) 0x7f24300b36f0 con 0x7f243015e8d0 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.403+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2410003520 con 0x7f243015e8d0 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2436198316 0 0) 0x7f2410003520 con 0x7f243015e8d0 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 >> v2:172.21.15.146:3300/0 conn(0x7f24300cd600 msgr2=0x7f2430070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 --2- 172.21.15.53:0/3783650997 >> v2:172.21.15.146:3300/0 conn(0x7f24300cd600 0x7f2430070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 >> v1:172.21.15.146:6789/0 conn(0x7f2430071030 legacy=0x7f24300b3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24300aec70 con 0x7f243015e8d0 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2420001670 con 0x7f243015e8d0 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 >> v1:172.21.15.53:6789/0 conn(0x7f243015e8d0 legacy=0x7f243006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2410003970 0x7f2410003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24100047c0 0x7f2410004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410005180 0x7f241000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24300cda60 con 0x7f2410003970 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241000c020 con 0x7f2410005180 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241000c1e0 con 0x7f24100047c0 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.404+0000 7f242fe00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2410003970 0x7f2410003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f242f400640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24100047c0 0x7f2410004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f2434c00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410005180 0x7f241000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f242fe00640 1 -- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24100047c0 msgr2=0x7f2410004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f242fe00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24100047c0 0x7f2410004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f242fe00640 1 -- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410005180 msgr2=0x7f241000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f242fe00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410005180 0x7f241000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f242fe00640 1 -- 172.21.15.53:0/3783650997 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2410003520 con 0x7f2410003970 2024-01-16T15:14:59.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.405+0000 7f2434c00640 1 -- 172.21.15.53:0/3783650997 reap_dead start 2024-01-16T15:14:59.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.406+0000 7f242fe00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2410003970 0x7f2410003d90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f242000c240 tx=0x7f24100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ceba1b48fddd3a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.406+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2420005ad0 con 0x7f2410003970 2024-01-16T15:14:59.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.406+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2420005c30 con 0x7f2410003970 2024-01-16T15:14:59.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.406+0000 7f242ea00640 1 -- 172.21.15.53:0/3783650997 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242002caf0 con 0x7f2410003970 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 -- 172.21.15.53:0/3783650997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2410003970 msgr2=0x7f2410003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2410003970 0x7f2410003d90 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f242000c240 tx=0x7f24100057f0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 -- 172.21.15.53:0/3783650997 shutdown_connections 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 --2- 172.21.15.53:0/3783650997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2410003970 0x7f2410003d90 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 -- 172.21.15.53:0/3783650997 >> 172.21.15.53:0/3783650997 conn(0x7f24300cf700 msgr2=0x7f24300cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 -- 172.21.15.53:0/3783650997 shutdown_connections 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.407+0000 7f2436c00640 1 -- 172.21.15.53:0/3783650997 wait complete. 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.409+0000 7f2436c00640 1 Processor -- start 2024-01-16T15:14:59.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.409+0000 7f2436c00640 1 -- start start 2024-01-16T15:14:59.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f2436c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410003970 0x7f24300b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f2436c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 0x7f24301458a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f2436c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2430145de0 0x7f2430148850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f2436c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2430146260 con 0x7f24301e5840 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f2436c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24301464f0 con 0x7f2410003970 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f2436c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2430146740 con 0x7f2430145de0 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f242f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 0x7f24301458a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f242f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 0x7f24301458a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39606/0 (socket says 172.21.15.53:39606) 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.410+0000 7f242f400640 1 -- 172.21.15.53:0/3424119372 learned_addr learned my addr 172.21.15.53:0/3424119372 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242fe00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410003970 0x7f24300b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242f400640 1 -- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2430145de0 msgr2=0x7f2430148850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242f400640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2430145de0 0x7f2430148850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242f400640 1 -- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410003970 msgr2=0x7f24300b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242f400640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410003970 0x7f24300b3fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242f400640 1 -- 172.21.15.53:0/3424119372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2430146a20 con 0x7f24301e5840 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f242f400640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 0x7f24301458a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f2418010790 tx=0x7f2418010c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418009630 con 0x7f24301e5840 2024-01-16T15:14:59.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241800eb60 con 0x7f24301e5840 2024-01-16T15:14:59.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2418010ea0 con 0x7f24301e5840 2024-01-16T15:14:59.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.411+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2430146d00 con 0x7f24301e5840 2024-01-16T15:14:59.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.412+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24300b44f0 con 0x7f24301e5840 2024-01-16T15:14:59.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.413+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24300675d0 con 0x7f24301e5840 2024-01-16T15:14:59.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.414+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f241801c400 con 0x7f24301e5840 2024-01-16T15:14:59.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.415+0000 7f2427e00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240806c100 0x7f240806e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.415+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f2418088cb0 con 0x7f24301e5840 2024-01-16T15:14:59.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.416+0000 7f242fe00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240806c100 0x7f240806e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.417+0000 7f242fe00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240806c100 0x7f240806e5c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2420034000 tx=0x7f2420009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.420+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2418055310 con 0x7f24301e5840 2024-01-16T15:14:59.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.686+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f2430071b20 con 0x7f24301e5840 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.687+0000 7f2427e00640 1 -- 172.21.15.53:0/3424119372 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+2305 (secure 0 0 0) 0x7f241805de60 con 0x7f24301e5840 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.670 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 926, 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:14:59.671 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 88, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:14:59.672 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 799, 2024-01-16T15:14:59.673 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3222164164, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 7029141504, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 766064971776, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 511, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 7420, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 0, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 2 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 56, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 6, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:14:59.674 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 6, 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "c", 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 8040 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:14:59.675 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 17, 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:14:40.563540+0000", 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "services": { 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": { 2024-01-16T15:14:59.676 INFO:teuthology.orchestra.run.smithi053.stdout: "daemons": { 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "summary": "", 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "y": { 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "start_epoch": 0, 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "start_stamp": "0.000000", 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 0, 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "(unrecognized address family 0)/0", 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": {}, 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: "task_status": {} 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:14:59.677 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:14:59.678 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": {} 2024-01-16T15:14:59.678 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240806c100 msgr2=0x7f240806e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240806c100 0x7f240806e5c0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2420034000 tx=0x7f2420009030 comp rx=0 tx=0).stop 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 msgr2=0x7f24301458a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 0x7f24301458a0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f2418010790 tx=0x7f2418010c50 comp rx=0 tx=0).stop 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 shutdown_connections 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f240806c100 0x7f240806e5c0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2430145de0 0x7f2430148850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24301e5840 0x7f24301458a0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 --2- 172.21.15.53:0/3424119372 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2410003970 0x7f24300b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 >> 172.21.15.53:0/3424119372 conn(0x7f24300cf700 msgr2=0x7f243015b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 shutdown_connections 2024-01-16T15:14:59.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.690+0000 7f2436c00640 1 -- 172.21.15.53:0/3424119372 wait complete. 2024-01-16T15:14:59.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:14:59.693 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/ss1clone1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622) 2024-01-16T15:14:59.751 INFO:teuthology.orchestra.run.smithi053.stderr:ls: cannot access 'volumes/_nogroup/ss1clone1/a3bc7693-6bdb-4fb8-bdf1-2aa0d5a1f622': No such file or directory 2024-01-16T15:14:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:14:59.753 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:14:59.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.994+0000 7ffba1800640 1 Processor -- start 2024-01-16T15:14:59.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.994+0000 7ffba1800640 1 -- start start 2024-01-16T15:14:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffba1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ffb9c0c9870 0x7ffb9c0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffba1800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffb9c0d0560 0x7ffb9c161030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffba1800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb9c0cf4c0 con 0x7ffb9c0d2260 2024-01-16T15:14:59.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffba1800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c0d2ef0 con 0x7ffb9c0d0560 2024-01-16T15:14:59.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffba1800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb9c0c9c50 con 0x7ffb9c0c9870 2024-01-16T15:14:59.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffb9a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffb9c0d0560 0x7ffb9c161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffb9a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ffb9c0d0560 0x7ffb9c161030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39612/0 (socket says 172.21.15.53:39612) 2024-01-16T15:14:59.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffb9a000640 1 -- 172.21.15.53:0/1543409281 learned_addr learned my addr 172.21.15.53:0/1543409281 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.996+0000 7ffb9aa00640 1 --2- 172.21.15.53:0/1543409281 >> v2:172.21.15.146:3300/0 conn(0x7ffb9c0c9870 0x7ffb9c0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.997+0000 7ffb9a000640 1 -- 172.21.15.53:0/1543409281 >> v2:172.21.15.146:3300/0 conn(0x7ffb9c0c9870 msgr2=0x7ffb9c0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:14:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.997+0000 7ffb9a000640 1 --2- 172.21.15.53:0/1543409281 >> v2:172.21.15.146:3300/0 conn(0x7ffb9c0c9870 0x7ffb9c0ceb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.997+0000 7ffb9a000640 1 -- 172.21.15.53:0/1543409281 >> v1:172.21.15.53:6789/0 conn(0x7ffb9c0d2260 legacy=0x7ffb9c0d0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.997+0000 7ffb9a000640 1 -- 172.21.15.53:0/1543409281 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb9c0cf7a0 con 0x7ffb9c0d0560 2024-01-16T15:14:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.998+0000 7ffb9a000640 1 --2- 172.21.15.53:0/1543409281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c161030 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c08d8f0 tx=0x7ffb8402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902afa382a795b9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.998+0000 7ffb99600640 1 -- 172.21.15.53:0/1543409281 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8402cb10 con 0x7ffb9c0d0560 2024-01-16T15:14:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.998+0000 7ffb99600640 1 -- 172.21.15.53:0/1543409281 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8402cc70 con 0x7ffb9c0d0560 2024-01-16T15:14:59.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.998+0000 7ffb99600640 1 -- 172.21.15.53:0/1543409281 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb840343f0 con 0x7ffb9c0d0560 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 -- 172.21.15.53:0/1543409281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 msgr2=0x7ffb9c161030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 --2- 172.21.15.53:0/1543409281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c161030 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7ffb9c08d8f0 tx=0x7ffb8402bf70 comp rx=0 tx=0).stop 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 -- 172.21.15.53:0/1543409281 shutdown_connections 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 --2- 172.21.15.53:0/1543409281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c161030 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 --2- 172.21.15.53:0/1543409281 >> v2:172.21.15.146:3300/0 conn(0x7ffb9c0c9870 0x7ffb9c0ceb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 -- 172.21.15.53:0/1543409281 >> 172.21.15.53:0/1543409281 conn(0x7ffb9c09c560 msgr2=0x7ffb9c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 -- 172.21.15.53:0/1543409281 shutdown_connections 2024-01-16T15:14:59.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:14:59.999+0000 7ffba1800640 1 -- 172.21.15.53:0/1543409281 wait complete. 2024-01-16T15:14:59.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.001+0000 7ffba1800640 1 Processor -- start 2024-01-16T15:14:59.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.001+0000 7ffba1800640 1 -- start start 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffba1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb9c0c9870 0x7ffb9c0afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffba1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c150650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffba1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb9c0d2260 0x7ffb9c0cc330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffba1800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c0b0460 con 0x7ffb9c0d0560 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffba1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c151340 con 0x7ffb9c0c9870 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffba1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9c151620 con 0x7ffb9c0d2260 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffb9a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c150650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffb9a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c150650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39614/0 (socket says 172.21.15.53:39614) 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.002+0000 7ffb9a000640 1 -- 172.21.15.53:0/85399544 learned_addr learned my addr 172.21.15.53:0/85399544 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:14:59.985 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9aa00640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb9c0c9870 0x7ffb9c0afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9b400640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb9c0d2260 0x7ffb9c0cc330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9a000640 1 -- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb9c0d2260 msgr2=0x7ffb9c0cc330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9a000640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb9c0d2260 0x7ffb9c0cc330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9a000640 1 -- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb9c0c9870 msgr2=0x7ffb9c0afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9a000640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb9c0c9870 0x7ffb9c0afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9a000640 1 -- 172.21.15.53:0/85399544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84009b30 con 0x7ffb9c0d0560 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.003+0000 7ffb9a000640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c150650 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7ffb84000f80 tx=0x7ffb84004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.004+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb84009510 con 0x7ffb9c0d0560 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.004+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8403e450 con 0x7ffb9c0d0560 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.004+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb8403e800 con 0x7ffb9c0d0560 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.004+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb9c151900 con 0x7ffb9c0d0560 2024-01-16T15:14:59.987 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.004+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb9c150b90 con 0x7ffb9c0d0560 2024-01-16T15:14:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.005+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb9c0c8c60 con 0x7ffb9c0d0560 2024-01-16T15:14:59.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.006+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7ffb8403e960 con 0x7ffb9c0d0560 2024-01-16T15:14:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.007+0000 7ffb92a00640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb7406c0d0 0x7ffb7406e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:14:59.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.007+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7ffb840ab760 con 0x7ffb9c0d0560 2024-01-16T15:14:59.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.007+0000 7ffb9aa00640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb7406c0d0 0x7ffb7406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:14:59.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.008+0000 7ffb9aa00640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb7406c0d0 0x7ffb7406e590 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c006fd0 tx=0x7ffb8c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:14:59.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.011+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ffb84077e80 con 0x7ffb9c0d0560 2024-01-16T15:15:00.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.230+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb9c1514b0 con 0x7ffb7406c0d0 2024-01-16T15:15:00.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.233+0000 7ffb92a00640 1 -- 172.21.15.53:0/85399544 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ffb9c1514b0 con 0x7ffb7406c0d0 2024-01-16T15:15:00.216 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:15:00.216 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:00.216 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:15:00.216 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:00.216 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:15:00.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb7406c0d0 msgr2=0x7ffb7406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb7406c0d0 0x7ffb7406e590 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c006fd0 tx=0x7ffb8c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 msgr2=0x7ffb9c150650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c150650 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7ffb84000f80 tx=0x7ffb84004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 shutdown_connections 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ffb7406c0d0 0x7ffb7406e590 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ffb9c0d2260 0x7ffb9c0cc330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ffb9c0d0560 0x7ffb9c150650 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 --2- 172.21.15.53:0/85399544 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ffb9c0c9870 0x7ffb9c0afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 >> 172.21.15.53:0/85399544 conn(0x7ffb9c09c560 msgr2=0x7ffb9c0ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 shutdown_connections 2024-01-16T15:15:00.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.236+0000 7ffba1800640 1 -- 172.21.15.53:0/85399544 wait complete. 2024-01-16T15:15:00.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:15:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.472+0000 7f424d200640 1 Processor -- start 2024-01-16T15:15:00.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.472+0000 7f424d200640 1 -- start start 2024-01-16T15:15:00.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.473+0000 7f424d200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f424815d4b0 0x7f42480d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.473+0000 7f424d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42480cf9a0 0x7f42480cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.473+0000 7f424d200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42480d0320 con 0x7f42480cddf0 2024-01-16T15:15:00.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.473+0000 7f424d200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42480b2730 con 0x7f424815d4b0 2024-01-16T15:15:00.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.473+0000 7f424d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42480d1ec0 con 0x7f42480cf9a0 2024-01-16T15:15:00.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.474+0000 7f4247400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f424815d4b0 0x7f42480d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.474+0000 7f423fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42480cf9a0 0x7f42480cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.474+0000 7f4247400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f424815d4b0 0x7f42480d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:37164/0 (socket says 172.21.15.53:37164) 2024-01-16T15:15:00.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.474+0000 7f4247400640 1 -- 172.21.15.53:0/3478105589 learned_addr learned my addr 172.21.15.53:0/3478105589 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:00.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.475+0000 7f4247400640 1 -- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42480cf9a0 msgr2=0x7f42480cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.475+0000 7f4247400640 1 --2- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42480cf9a0 0x7f42480cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.475+0000 7f4247400640 1 -- 172.21.15.53:0/3478105589 >> v1:172.21.15.146:6789/0 conn(0x7f42480cddf0 legacy=0x7f42480ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.475+0000 7f4247400640 1 -- 172.21.15.53:0/3478105589 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42480b29d0 con 0x7f424815d4b0 2024-01-16T15:15:00.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.475+0000 7f423fe00640 1 --2- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42480cf9a0 0x7f42480cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:00.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.477+0000 7f4247400640 1 --2- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424815d4b0 0x7f42480d1550 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f42380098a0 tx=0x7f423802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b43adc943801f6e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.477+0000 7f4246a00640 1 -- 172.21.15.53:0/3478105589 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238030840 con 0x7f424815d4b0 2024-01-16T15:15:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f4246a00640 1 -- 172.21.15.53:0/3478105589 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42380309a0 con 0x7f424815d4b0 2024-01-16T15:15:00.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f4246a00640 1 -- 172.21.15.53:0/3478105589 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238030d30 con 0x7f424815d4b0 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 -- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424815d4b0 msgr2=0x7f42480d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 --2- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424815d4b0 0x7f42480d1550 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f42380098a0 tx=0x7f423802b980 comp rx=0 tx=0).stop 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 -- 172.21.15.53:0/3478105589 shutdown_connections 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 --2- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42480cf9a0 0x7f42480cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 --2- 172.21.15.53:0/3478105589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f424815d4b0 0x7f42480d1550 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 -- 172.21.15.53:0/3478105589 >> 172.21.15.53:0/3478105589 conn(0x7f42480d3000 msgr2=0x7f42480d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 -- 172.21.15.53:0/3478105589 shutdown_connections 2024-01-16T15:15:00.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.478+0000 7f424d200640 1 -- 172.21.15.53:0/3478105589 wait complete. 2024-01-16T15:15:00.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.480+0000 7f424d200640 1 Processor -- start 2024-01-16T15:15:00.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.480+0000 7f424d200640 1 -- start start 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.481+0000 7f424d200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42480cddf0 0x7f42480cad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.481+0000 7f424d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 0x7f424815e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f424d200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f424815d4b0 0x7f424815e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f424d200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42480cb270 con 0x7f42480cf9a0 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f424d200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f424809dec0 con 0x7f42480cddf0 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f424d200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42480cb480 con 0x7f424815d4b0 2024-01-16T15:15:00.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 0x7f424815e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 0x7f424815e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39626/0 (socket says 172.21.15.53:39626) 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 -- 172.21.15.53:0/3588571638 learned_addr learned my addr 172.21.15.53:0/3588571638 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f4247400640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42480cddf0 0x7f42480cad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f4247e00640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f424815d4b0 0x7f424815e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 -- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f424815d4b0 msgr2=0x7f424815e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f424815d4b0 0x7f424815e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 -- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42480cddf0 msgr2=0x7f42480cad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42480cddf0 0x7f42480cad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.482+0000 7f423fe00640 1 -- 172.21.15.53:0/3588571638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4230009400 con 0x7f42480cf9a0 2024-01-16T15:15:00.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.483+0000 7f423fe00640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 0x7f424815e070 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f423000ebe0 tx=0x7f423000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.483+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4230009760 con 0x7f42480cf9a0 2024-01-16T15:15:00.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.483+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423000cc30 con 0x7f42480cf9a0 2024-01-16T15:15:00.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.483+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42380094d0 con 0x7f42480cf9a0 2024-01-16T15:15:00.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.483+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423000eee0 con 0x7f42480cf9a0 2024-01-16T15:15:00.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.483+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f424815da10 con 0x7f42480cf9a0 2024-01-16T15:15:00.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.484+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4248067450 con 0x7f42480cf9a0 2024-01-16T15:15:00.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.485+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f4230013470 con 0x7f42480cf9a0 2024-01-16T15:15:00.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.486+0000 7f423f400640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f422006c0d0 0x7f422006e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.486+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f4230088c20 con 0x7f42480cf9a0 2024-01-16T15:15:00.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.487+0000 7f4247400640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f422006c0d0 0x7f422006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.488+0000 7f4247400640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f422006c0d0 0x7f422006e590 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4238036000 tx=0x7f423803b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.491+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4230055220 con 0x7f42480cf9a0 2024-01-16T15:15:00.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.710+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f424815d890 con 0x7f422006c0d0 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.722+0000 7f423f400640 1 -- 172.21.15.53:0/3588571638 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+79 (secure 0 0 0) 0x7f4238030b90 con 0x7f422006c0d0 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1clone1" 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:00.705 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.724+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f422006c0d0 msgr2=0x7f422006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.724+0000 7f424d200640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f422006c0d0 0x7f422006e590 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f4238036000 tx=0x7f423803b040 comp rx=0 tx=0).stop 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 msgr2=0x7f424815e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 0x7f424815e070 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7f423000ebe0 tx=0x7f423000c520 comp rx=0 tx=0).stop 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 shutdown_connections 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f422006c0d0 0x7f422006e590 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f424815d4b0 0x7f424815e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42480cf9a0 0x7f424815e070 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 --2- 172.21.15.53:0/3588571638 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42480cddf0 0x7f42480cad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 >> 172.21.15.53:0/3588571638 conn(0x7f42480d3000 msgr2=0x7f42480b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 shutdown_connections 2024-01-16T15:15:00.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.725+0000 7f424d200640 1 -- 172.21.15.53:0/3588571638 wait complete. 2024-01-16T15:15:00.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:15:00.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.975+0000 7f691ac00640 1 Processor -- start 2024-01-16T15:15:00.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.975+0000 7f691ac00640 1 -- start start 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.976+0000 7f691ac00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6914160890 0x7f69140d3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691ac00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69140c6be0 0x7f69140cfb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691ac00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140cf6a0 con 0x7f6914160c70 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691ac00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69140d1430 con 0x7f69140c6be0 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691ac00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69140b4370 con 0x7f6914160890 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69140c6be0 0x7f69140cfb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69140c6be0 0x7f69140cfb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39642/0 (socket says 172.21.15.53:39642) 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f691a200640 1 -- 172.21.15.53:0/149589760 learned_addr learned my addr 172.21.15.53:0/149589760 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:00.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.977+0000 7f6919800640 1 --2- 172.21.15.53:0/149589760 >> v2:172.21.15.146:3300/0 conn(0x7f6914160890 0x7f69140d3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.978+0000 7f691a200640 1 -- 172.21.15.53:0/149589760 >> v2:172.21.15.146:3300/0 conn(0x7f6914160890 msgr2=0x7f69140d3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:00.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.978+0000 7f691a200640 1 --2- 172.21.15.53:0/149589760 >> v2:172.21.15.146:3300/0 conn(0x7f6914160890 0x7f69140d3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.978+0000 7f691a200640 1 -- 172.21.15.53:0/149589760 >> v1:172.21.15.53:6789/0 conn(0x7f6914160c70 legacy=0x7f69140d1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.978+0000 7f691a200640 1 -- 172.21.15.53:0/149589760 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140ce7c0 con 0x7f69140c6be0 2024-01-16T15:15:00.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.979+0000 7f691a200640 1 --2- 172.21.15.53:0/149589760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f69140cfb30 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f69140b5c10 tx=0x7f690c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c795476994b4fe35 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.979+0000 7f6913e00640 1 -- 172.21.15.53:0/149589760 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c0317e0 con 0x7f69140c6be0 2024-01-16T15:15:00.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.979+0000 7f6913e00640 1 -- 172.21.15.53:0/149589760 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690c031940 con 0x7f69140c6be0 2024-01-16T15:15:00.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.979+0000 7f6913e00640 1 -- 172.21.15.53:0/149589760 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f690c031cd0 con 0x7f69140c6be0 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 -- 172.21.15.53:0/149589760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 msgr2=0x7f69140cfb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 --2- 172.21.15.53:0/149589760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f69140cfb30 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7f69140b5c10 tx=0x7f690c02b880 comp rx=0 tx=0).stop 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 -- 172.21.15.53:0/149589760 shutdown_connections 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 --2- 172.21.15.53:0/149589760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f69140cfb30 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 --2- 172.21.15.53:0/149589760 >> v2:172.21.15.146:3300/0 conn(0x7f6914160890 0x7f69140d3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 -- 172.21.15.53:0/149589760 >> 172.21.15.53:0/149589760 conn(0x7f69140d3090 msgr2=0x7f69140d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 -- 172.21.15.53:0/149589760 shutdown_connections 2024-01-16T15:15:00.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.980+0000 7f691ac00640 1 -- 172.21.15.53:0/149589760 wait complete. 2024-01-16T15:15:00.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.982+0000 7f691ac00640 1 Processor -- start 2024-01-16T15:15:00.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.982+0000 7f691ac00640 1 -- start start 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f691ac00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f6914152f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f691ac00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6914160890 0x7f691415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f691ac00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6914160c70 0x7f69140b6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f691ac00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6914154430 con 0x7f69140c6be0 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f691ac00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69141534a0 con 0x7f6914160890 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f691ac00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6914152df0 con 0x7f6914160c70 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.983+0000 7f6919800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f6914152f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f6914152f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39646/0 (socket says 172.21.15.53:39646) 2024-01-16T15:15:00.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 -- 172.21.15.53:0/2723537038 learned_addr learned my addr 172.21.15.53:0/2723537038 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:00.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f691a200640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6914160c70 0x7f69140b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6918e00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6914160890 0x7f691415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 -- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6914160c70 msgr2=0x7f69140b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6914160c70 0x7f69140b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 -- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6914160890 msgr2=0x7f691415faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6914160890 0x7f691415faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.984+0000 7f6919800640 1 -- 172.21.15.53:0/2723537038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690c0094d0 con 0x7f69140c6be0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.985+0000 7f6919800640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f6914152f60 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6904007e40 tx=0x7f690400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.985+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6904014620 con 0x7f69140c6be0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.985+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f690400ee90 con 0x7f69140c6be0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.985+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6914153e70 con 0x7f69140c6be0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.985+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69040149c0 con 0x7f69140c6be0 2024-01-16T15:15:00.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.985+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69140b5e50 con 0x7f69140c6be0 2024-01-16T15:15:00.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.986+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6914067450 con 0x7f69140c6be0 2024-01-16T15:15:00.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.987+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f6904014b20 con 0x7f69140c6be0 2024-01-16T15:15:00.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.989+0000 7f6911600640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68ec06c0d0 0x7f68ec06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:00.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.989+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f69040892b0 con 0x7f69140c6be0 2024-01-16T15:15:00.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.990+0000 7f6918e00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68ec06c0d0 0x7f68ec06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:00.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.991+0000 7f6918e00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68ec06c0d0 0x7f68ec06e590 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f69140cf600 tx=0x7f68fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:00.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:00.994+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f69040558b0 con 0x7f69140c6be0 2024-01-16T15:15:01.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.218+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f69140ced80 con 0x7f68ec06c0d0 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.250+0000 7f6911600640 1 -- 172.21.15.53:0/2723537038 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f68fc001c70 con 0x7f68ec06c0d0 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:01.233 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.252+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68ec06c0d0 msgr2=0x7f68ec06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.252+0000 7f691ac00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68ec06c0d0 0x7f68ec06e590 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f69140cf600 tx=0x7f68fc009040 comp rx=0 tx=0).stop 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.252+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 msgr2=0x7f6914152f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.252+0000 7f691ac00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f6914152f60 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7f6904007e40 tx=0x7f690400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 shutdown_connections 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f68ec06c0d0 0x7f68ec06e590 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6914160c70 0x7f69140b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6914160890 0x7f691415faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 --2- 172.21.15.53:0/2723537038 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69140c6be0 0x7f6914152f60 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 >> 172.21.15.53:0/2723537038 conn(0x7f69140d3090 msgr2=0x7f69140c7030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 shutdown_connections 2024-01-16T15:15:01.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.253+0000 7f691ac00640 1 -- 172.21.15.53:0/2723537038 wait complete. 2024-01-16T15:15:01.256 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.507+0000 7f15ed400640 1 Processor -- start 2024-01-16T15:15:01.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.508+0000 7f15ed400640 1 -- start start 2024-01-16T15:15:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.509+0000 7f15ed400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e813df80 0x7f15e815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.509+0000 7f15ed400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e806c470 con 0x7f15e80cf9a0 2024-01-16T15:15:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.509+0000 7f15ed400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15e806c750 con 0x7f15e813e360 2024-01-16T15:15:01.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.509+0000 7f15ed400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e806ca30 con 0x7f15e813df80 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.510+0000 7f15e7200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f15e80cf9a0 0x7f15e80cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40438/0 (socket says 172.21.15.53:40438) 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.510+0000 7f15e7200640 1 -- 172.21.15.53:0/3433743280 learned_addr learned my addr 172.21.15.53:0/3433743280 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.510+0000 7f15e6800640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e813df80 0x7f15e815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.510+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269588881 0 0) 0x7f15e806c470 con 0x7f15e80cf9a0 2024-01-16T15:15:01.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.510+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c8003520 con 0x7f15e80cf9a0 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2966121880 0 0) 0x7f15c8003520 con 0x7f15e80cf9a0 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e813df80 msgr2=0x7f15e815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e813df80 0x7f15e815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 >> v1:172.21.15.146:6789/0 conn(0x7f15e813e360 legacy=0x7f15e80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e806cd10 con 0x7f15e80cf9a0 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f15e00019f0 con 0x7f15e80cf9a0 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 >> v1:172.21.15.53:6789/0 conn(0x7f15e80cf9a0 legacy=0x7f15e80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15c8003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15c8004350 0x7f15c8004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15c8005020 0x7f15c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e806ca30 con 0x7f15c80039a0 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c800bef0 con 0x7f15c8004350 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15c800c0b0 con 0x7f15c8005020 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e6800640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e813df80 0x7f15e815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.511+0000 7f15e6800640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15c8003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15e7200640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15c8005020 0x7f15c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15dfe00640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15c8004350 0x7f15c8004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15e6800640 1 -- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15c8005020 msgr2=0x7f15c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15e6800640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15c8005020 0x7f15c800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15e6800640 1 -- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15c8004350 msgr2=0x7f15c8004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15e6800640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15c8004350 0x7f15c8004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15e6800640 1 -- 172.21.15.53:0/3433743280 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8003520 con 0x7f15c80039a0 2024-01-16T15:15:01.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.512+0000 7f15dfe00640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15c8004350 0x7f15c8004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.513+0000 7f15e6800640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15c8003d80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f15e80b1bf0 tx=0x7f15c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e43d68fd667d8c62 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.513+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8004030 con 0x7f15c80039a0 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.513+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d8004190 con 0x7f15c80039a0 2024-01-16T15:15:01.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.513+0000 7f15e5e00640 1 -- 172.21.15.53:0/3433743280 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8004520 con 0x7f15c80039a0 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15ed400640 1 -- 172.21.15.53:0/3433743280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 msgr2=0x7f15c8003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15ed400640 1 --2- 172.21.15.53:0/3433743280 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15c8003d80 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f15e80b1bf0 tx=0x7f15c80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15e7200640 1 -- 172.21.15.53:0/3433743280 reap_dead start 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15ed400640 1 -- 172.21.15.53:0/3433743280 shutdown_connections 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15ed400640 1 -- 172.21.15.53:0/3433743280 >> 172.21.15.53:0/3433743280 conn(0x7f15e806a800 msgr2=0x7f15e806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15ed400640 1 -- 172.21.15.53:0/3433743280 shutdown_connections 2024-01-16T15:15:01.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.514+0000 7f15ed400640 1 -- 172.21.15.53:0/3433743280 wait complete. 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.516+0000 7f15ed400640 1 Processor -- start 2024-01-16T15:15:01.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.516+0000 7f15ed400640 1 -- start start 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.517+0000 7f15ed400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15e8153500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.517+0000 7f15ed400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15e81f29c0 0x7f15e81f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.517+0000 7f15ed400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e80af310 0x7f15e80af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.517+0000 7f15ed400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80afcd0 con 0x7f15c80039a0 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.517+0000 7f15ed400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e813a7b0 con 0x7f15e81f29c0 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.517+0000 7f15ed400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15e80b3b70 con 0x7f15e80af310 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15e8153500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15e8153500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39666/0 (socket says 172.21.15.53:39666) 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 -- 172.21.15.53:0/13019666 learned_addr learned my addr 172.21.15.53:0/13019666 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15dfe00640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15e81f29c0 0x7f15e81f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e7200640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e80af310 0x7f15e80af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 -- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e80af310 msgr2=0x7f15e80af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e80af310 0x7f15e80af790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 -- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15e81f29c0 msgr2=0x7f15e81f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15e81f29c0 0x7f15e81f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e6800640 1 -- 172.21.15.53:0/13019666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e813aa00 con 0x7f15c80039a0 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.518+0000 7f15e7200640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e80af310 0x7f15e80af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:01.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.519+0000 7f15e6800640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15e8153500 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f15d802c380 tx=0x7f15d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.519+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d802cb20 con 0x7f15c80039a0 2024-01-16T15:15:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.519+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d802cc80 con 0x7f15c80039a0 2024-01-16T15:15:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.519+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8003610 con 0x7f15c80039a0 2024-01-16T15:15:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.519+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e813ace0 con 0x7f15c80039a0 2024-01-16T15:15:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.519+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e8153a40 con 0x7f15c80039a0 2024-01-16T15:15:01.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.520+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e80675d0 con 0x7f15c80039a0 2024-01-16T15:15:01.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.522+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f15d8003770 con 0x7f15c80039a0 2024-01-16T15:15:01.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.522+0000 7f15dea00640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f15c006c100 0x7f15c006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:01.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.523+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f15d80aad80 con 0x7f15c80039a0 2024-01-16T15:15:01.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.523+0000 7f15dfe00640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f15c006c100 0x7f15c006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:01.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.524+0000 7f15dfe00640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f15c006c100 0x7f15c006e5c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f15d0007fd0 tx=0x7f15d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:01.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.528+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f15d80774a0 con 0x7f15c80039a0 2024-01-16T15:15:01.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.757+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f15e80afcd0 con 0x7f15c006c100 2024-01-16T15:15:01.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.817+0000 7f15dea00640 1 -- 172.21.15.53:0/13019666 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f15e80afcd0 con 0x7f15c006c100 2024-01-16T15:15:01.800 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:01.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.821+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f15c006c100 msgr2=0x7f15c006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.821+0000 7f15ed400640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f15c006c100 0x7f15c006e5c0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f15d0007fd0 tx=0x7f15d0009040 comp rx=0 tx=0).stop 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.821+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 msgr2=0x7f15e8153500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.821+0000 7f15ed400640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15e8153500 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f15d802c380 tx=0x7f15d8004730 comp rx=0 tx=0).stop 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 shutdown_connections 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f15c006c100 0x7f15c006e5c0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f15e80af310 0x7f15e80af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f15e81f29c0 0x7f15e81f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 --2- 172.21.15.53:0/13019666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f15c80039a0 0x7f15e8153500 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 >> 172.21.15.53:0/13019666 conn(0x7f15e806a800 msgr2=0x7f15e80b1410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 shutdown_connections 2024-01-16T15:15:01.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:01.822+0000 7f15ed400640 1 -- 172.21.15.53:0/13019666 wait complete. 2024-01-16T15:15:01.826 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:15:02.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.071+0000 7f13cd800640 1 Processor -- start 2024-01-16T15:15:02.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.071+0000 7f13cd800640 1 -- start start 2024-01-16T15:15:02.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.072+0000 7f13cd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 0x7f13c8099ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.072+0000 7f13cd800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80cfa70 con 0x7f13c815e880 2024-01-16T15:15:02.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.072+0000 7f13cd800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c809a3e0 con 0x7f13c806e7c0 2024-01-16T15:15:02.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.072+0000 7f13cd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80cff00 con 0x7f13c8099a60 2024-01-16T15:15:02.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.073+0000 7f13c6a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f13c815e880 0x7f13c806ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57520/0 (socket says 172.21.15.53:57520) 2024-01-16T15:15:02.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.073+0000 7f13c6a00640 1 -- 172.21.15.53:0/2248772367 learned_addr learned my addr 172.21.15.53:0/2248772367 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:02.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.073+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 154462746 0 0) 0x7f13c80cfa70 con 0x7f13c815e880 2024-01-16T15:15:02.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.073+0000 7f13c6000640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 0x7f13c8099ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.073+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8003520 con 0x7f13c815e880 2024-01-16T15:15:02.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058091359 0 0) 0x7f13a8003520 con 0x7f13c815e880 2024-01-16T15:15:02.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 msgr2=0x7f13c8099ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 0x7f13c8099ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 >> v1:172.21.15.146:6789/0 conn(0x7f13c806e7c0 legacy=0x7f13c806ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c806fc70 con 0x7f13c815e880 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f13b0001670 con 0x7f13c815e880 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 >> v1:172.21.15.53:6789/0 conn(0x7f13c815e880 legacy=0x7f13c806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13a8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13a80047c0 0x7f13a8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 0x7f13a80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80cff00 con 0x7f13a80039a0 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13a800bb40 con 0x7f13a80047c0 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13a800bd00 con 0x7f13c8099a60 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13a8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.074+0000 7f13c6000640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13a80047c0 0x7f13a8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.075+0000 7f13c7400640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 0x7f13a80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.075+0000 7f13c6a00640 1 -- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 msgr2=0x7f13a80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.075+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c8099a60 0x7f13a80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.075+0000 7f13c6a00640 1 -- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13a80047c0 msgr2=0x7f13a8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.075+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13a80047c0 0x7f13a8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.075+0000 7f13c6a00640 1 -- 172.21.15.53:0/2248772367 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13a8003520 con 0x7f13a80039a0 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.076+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13a8003dc0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c240 tx=0x7f13a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f2733dd30e7386 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.076+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0005ad0 con 0x7f13a80039a0 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.076+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b0005c30 con 0x7f13a80039a0 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.076+0000 7f13c5600640 1 -- 172.21.15.53:0/2248772367 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b002c8b0 con 0x7f13a80039a0 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13cd800640 1 -- 172.21.15.53:0/2248772367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 msgr2=0x7f13a8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13cd800640 1 --2- 172.21.15.53:0/2248772367 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13a8003dc0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f13b000c240 tx=0x7f13a80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13c7400640 1 -- 172.21.15.53:0/2248772367 reap_dead start 2024-01-16T15:15:02.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13cd800640 1 -- 172.21.15.53:0/2248772367 shutdown_connections 2024-01-16T15:15:02.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13cd800640 1 -- 172.21.15.53:0/2248772367 >> 172.21.15.53:0/2248772367 conn(0x7f13c80b2d00 msgr2=0x7f13c80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13cd800640 1 -- 172.21.15.53:0/2248772367 shutdown_connections 2024-01-16T15:15:02.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.077+0000 7f13cd800640 1 -- 172.21.15.53:0/2248772367 wait complete. 2024-01-16T15:15:02.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.079+0000 7f13cd800640 1 Processor -- start 2024-01-16T15:15:02.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.079+0000 7f13cd800640 1 -- start start 2024-01-16T15:15:02.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13cd800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13c81e6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13cd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c815ad10 0x7f13c81481d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13cd800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c8148b40 0x7f13c8148fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13cd800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80b6f50 con 0x7f13a80039a0 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13cd800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c81e61d0 con 0x7f13c8148b40 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13cd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c80b6c70 con 0x7f13c815ad10 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13c6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13c81e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13c6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13c81e6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52864/0 (socket says 172.21.15.53:52864) 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.080+0000 7f13c6a00640 1 -- 172.21.15.53:0/2875075393 learned_addr learned my addr 172.21.15.53:0/2875075393 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c7400640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c8148b40 0x7f13c8148fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6000640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c815ad10 0x7f13c81481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6a00640 1 -- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c815ad10 msgr2=0x7f13c81481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c815ad10 0x7f13c81481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6a00640 1 -- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c8148b40 msgr2=0x7f13c8148fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c8148b40 0x7f13c8148fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6a00640 1 -- 172.21.15.53:0/2875075393 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c81e5b80 con 0x7f13a80039a0 2024-01-16T15:15:02.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6000640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c815ad10 0x7f13c81481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:02.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13c6a00640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13c81e6660 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f13b0001930 tx=0x7f13b0001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.081+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b0039070 con 0x7f13a80039a0 2024-01-16T15:15:02.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.082+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b002dd50 con 0x7f13a80039a0 2024-01-16T15:15:02.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.082+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c81e5e60 con 0x7f13a80039a0 2024-01-16T15:15:02.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.082+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b003c040 con 0x7f13a80039a0 2024-01-16T15:15:02.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.082+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c8149500 con 0x7f13a80039a0 2024-01-16T15:15:02.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.083+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c80675d0 con 0x7f13a80039a0 2024-01-16T15:15:02.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.084+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f13b0032070 con 0x7f13a80039a0 2024-01-16T15:15:02.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.085+0000 7f13bea00640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13a006c100 0x7f13a006e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.085+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f13b00a9e70 con 0x7f13a80039a0 2024-01-16T15:15:02.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.086+0000 7f13c6000640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13a006c100 0x7f13a006e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.087+0000 7f13c6000640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13a006c100 0x7f13a006e5c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f13c80c7e80 tx=0x7f13b800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.090+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f13b0076590 con 0x7f13a80039a0 2024-01-16T15:15:02.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.311+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f13c80cf700 con 0x7f13a006c100 2024-01-16T15:15:02.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.352+0000 7f13bea00640 1 -- 172.21.15.53:0/2875075393 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f13b8003ad0 con 0x7f13a006c100 2024-01-16T15:15:02.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13a006c100 msgr2=0x7f13a006e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13a006c100 0x7f13a006e5c0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f13c80c7e80 tx=0x7f13b800b040 comp rx=0 tx=0).stop 2024-01-16T15:15:02.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 msgr2=0x7f13c81e6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13c81e6660 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7f13b0001930 tx=0x7f13b0001a00 comp rx=0 tx=0).stop 2024-01-16T15:15:02.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 shutdown_connections 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f13a006c100 0x7f13a006e5c0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f13c8148b40 0x7f13c8148fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f13c815ad10 0x7f13c81481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 --2- 172.21.15.53:0/2875075393 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13a80039a0 0x7f13c81e6660 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.355+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 >> 172.21.15.53:0/2875075393 conn(0x7f13c80b2d00 msgr2=0x7f13c806b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.356+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 shutdown_connections 2024-01-16T15:15:02.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.356+0000 7f13cd800640 1 -- 172.21.15.53:0/2875075393 wait complete. 2024-01-16T15:15:02.342 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.360+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.343 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.360+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.343 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.360+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.343 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.360+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.343 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.360+0000 7f6136e00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.363 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs ss1clone1 2024-01-16T15:15:02.366 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.384+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.366 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.384+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.366 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.384+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.366 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.384+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.366 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:02.384+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:02.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.597+0000 7f8416000640 1 Processor -- start 2024-01-16T15:15:02.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.598+0000 7f8416000640 1 -- start start 2024-01-16T15:15:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.599+0000 7f8416000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84100d17e0 0x7f84100d1c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.599+0000 7f8416000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100b3890 0x7f84100b3d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.599+0000 7f8416000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84100d2180 con 0x7f841015d610 2024-01-16T15:15:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.599+0000 7f8416000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84100d1390 con 0x7f84100d17e0 2024-01-16T15:15:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.599+0000 7f8416000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84100b3400 con 0x7f84100b3890 2024-01-16T15:15:02.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.599+0000 7f8407e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84100d17e0 0x7f84100d1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f8407e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f84100d17e0 0x7f84100d1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52872/0 (socket says 172.21.15.53:52872) 2024-01-16T15:15:02.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f8407e00640 1 -- 172.21.15.53:0/2407451783 learned_addr learned my addr 172.21.15.53:0/2407451783 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:02.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f840fe00640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100b3890 0x7f84100b3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f8407e00640 1 -- 172.21.15.53:0/2407451783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100b3890 msgr2=0x7f84100b3d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f8407e00640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100b3890 0x7f84100b3d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f8407e00640 1 -- 172.21.15.53:0/2407451783 >> v1:172.21.15.146:6789/0 conn(0x7f841015d610 legacy=0x7f84100ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:02.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f8407e00640 1 -- 172.21.15.53:0/2407451783 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84100b27f0 con 0x7f84100d17e0 2024-01-16T15:15:02.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.600+0000 7f840fe00640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100b3890 0x7f84100b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.601+0000 7f8407e00640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84100d17e0 0x7f84100d1c40 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009870 tx=0x7f83f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d366000a4d95771 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f840ea00640 1 -- 172.21.15.53:0/2407451783 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8031640 con 0x7f84100d17e0 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f840ea00640 1 -- 172.21.15.53:0/2407451783 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83f80317a0 con 0x7f84100d17e0 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f840ea00640 1 -- 172.21.15.53:0/2407451783 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83f8031b30 con 0x7f84100d17e0 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 -- 172.21.15.53:0/2407451783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84100d17e0 msgr2=0x7f84100d1c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84100d17e0 0x7f84100d1c40 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009870 tx=0x7f83f802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 -- 172.21.15.53:0/2407451783 shutdown_connections 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100b3890 0x7f84100b3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 --2- 172.21.15.53:0/2407451783 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f84100d17e0 0x7f84100d1c40 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 -- 172.21.15.53:0/2407451783 >> 172.21.15.53:0/2407451783 conn(0x7f841009c540 msgr2=0x7f84100d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 -- 172.21.15.53:0/2407451783 shutdown_connections 2024-01-16T15:15:02.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.602+0000 7f8416000640 1 -- 172.21.15.53:0/2407451783 wait complete. 2024-01-16T15:15:02.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.604+0000 7f8416000640 1 Processor -- start 2024-01-16T15:15:02.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.604+0000 7f8416000640 1 -- start start 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.605+0000 7f8416000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84100b3890 0x7f84100cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.605+0000 7f8416000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100d17e0 0x7f8410151660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.605+0000 7f8416000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 0x7f84100b77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.605+0000 7f8416000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84100cbe10 con 0x7f841015d610 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.605+0000 7f8416000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8410151ba0 con 0x7f84100b3890 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.605+0000 7f8416000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84100b9020 con 0x7f84100d17e0 2024-01-16T15:15:02.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 0x7f84100b77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 0x7f84100b77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52880/0 (socket says 172.21.15.53:52880) 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 -- 172.21.15.53:0/197624550 learned_addr learned my addr 172.21.15.53:0/197624550 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840f400640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84100b3890 0x7f84100cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f8407e00640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100d17e0 0x7f8410151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 -- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100d17e0 msgr2=0x7f8410151660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100d17e0 0x7f8410151660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 -- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84100b3890 msgr2=0x7f84100cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84100b3890 0x7f84100cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.606+0000 7f840fe00640 1 -- 172.21.15.53:0/197624550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f80094d0 con 0x7f841015d610 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.607+0000 7f840fe00640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 0x7f84100b77a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f840800cef0 tx=0x7f840800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.607+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840800d840 con 0x7f841015d610 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.607+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f840800d9c0 con 0x7f841015d610 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.607+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840800dd70 con 0x7f841015d610 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.607+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84100b7ce0 con 0x7f841015d610 2024-01-16T15:15:02.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.607+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84100b8260 con 0x7f841015d610 2024-01-16T15:15:02.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.609+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84100675d0 con 0x7f841015d610 2024-01-16T15:15:02.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.609+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7f840801e3b0 con 0x7f841015d610 2024-01-16T15:15:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.610+0000 7f8407400640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83e806c0d0 0x7f83e806e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.610+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7f8408089410 con 0x7f841015d610 2024-01-16T15:15:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.611+0000 7f840f400640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83e806c0d0 0x7f83e806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:02.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.612+0000 7f840f400640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83e806c0d0 0x7f83e806e590 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8400006fd0 tx=0x7f8400008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:02.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.615+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8408055ad0 con 0x7f841015d610 2024-01-16T15:15:02.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.840+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "ss1clone1", "target": ["mon-mgr", ""]}) v1 -- 0x7f84100b3090 con 0x7f83e806c0d0 2024-01-16T15:15:02.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.881+0000 7f8407400640 1 -- 172.21.15.53:0/197624550 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f840000bda0 con 0x7f83e806c0d0 2024-01-16T15:15:02.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83e806c0d0 msgr2=0x7f83e806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83e806c0d0 0x7f83e806e590 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8400006fd0 tx=0x7f8400008040 comp rx=0 tx=0).stop 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 msgr2=0x7f84100b77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 0x7f84100b77a0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f840800cef0 tx=0x7f840800a710 comp rx=0 tx=0).stop 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 shutdown_connections 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83e806c0d0 0x7f83e806e590 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841015d610 0x7f84100b77a0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84100d17e0 0x7f8410151660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.884+0000 7f8416000640 1 --2- 172.21.15.53:0/197624550 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f84100b3890 0x7f84100cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.885+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 >> 172.21.15.53:0/197624550 conn(0x7f841009c540 msgr2=0x7f841006c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.885+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 shutdown_connections 2024-01-16T15:15:02.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:02.885+0000 7f8416000640 1 -- 172.21.15.53:0/197624550 wait complete. 2024-01-16T15:15:02.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:02.886 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:15:02.935 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-01-16T15:15:02.937 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:15:03.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.135+0000 7fa169a00640 1 Processor -- start 2024-01-16T15:15:03.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.136+0000 7fa169a00640 1 -- start start 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.137+0000 7fa169a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa16415d4b0 0x7fa1640d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.137+0000 7fa169a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640b2730 con 0x7fa1640cf9a0 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.137+0000 7fa169a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1640d1ec0 con 0x7fa1640b41e0 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.137+0000 7fa169a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1640b29d0 con 0x7fa16415d4b0 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.137+0000 7fa163e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa16415d4b0 0x7fa1640d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.138+0000 7fa163e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa16415d4b0 0x7fa1640d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52892/0 (socket says 172.21.15.53:52892) 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.138+0000 7fa163e00640 1 -- 172.21.15.53:0/2237687571 learned_addr learned my addr 172.21.15.53:0/2237687571 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.138+0000 7fa163e00640 1 -- 172.21.15.53:0/2237687571 >> v1:172.21.15.146:6789/0 conn(0x7fa1640b41e0 legacy=0x7fa1640ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.138+0000 7fa163e00640 1 -- 172.21.15.53:0/2237687571 >> v1:172.21.15.53:6789/0 conn(0x7fa1640cf9a0 legacy=0x7fa1640cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.138+0000 7fa163e00640 1 -- 172.21.15.53:0/2237687571 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1640b2cb0 con 0x7fa16415d4b0 2024-01-16T15:15:03.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.138+0000 7fa163400640 1 -- 172.21.15.53:0/2237687571 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033260100 0 0) 0x7fa1640b2730 con 0x7fa1640cf9a0 2024-01-16T15:15:03.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.139+0000 7fa163e00640 1 --2- 172.21.15.53:0/2237687571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa16415d4b0 0x7fa1640d1550 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fa154009870 tx=0x7fa15402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b5d51f809edf42f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa163400640 1 -- 172.21.15.53:0/2237687571 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154030860 con 0x7fa16415d4b0 2024-01-16T15:15:03.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa163400640 1 -- 172.21.15.53:0/2237687571 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1540309c0 con 0x7fa16415d4b0 2024-01-16T15:15:03.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa163400640 1 -- 172.21.15.53:0/2237687571 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa154030d50 con 0x7fa16415d4b0 2024-01-16T15:15:03.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 -- 172.21.15.53:0/2237687571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa16415d4b0 msgr2=0x7fa1640d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 --2- 172.21.15.53:0/2237687571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa16415d4b0 0x7fa1640d1550 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fa154009870 tx=0x7fa15402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 -- 172.21.15.53:0/2237687571 shutdown_connections 2024-01-16T15:15:03.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 --2- 172.21.15.53:0/2237687571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa16415d4b0 0x7fa1640d1550 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 -- 172.21.15.53:0/2237687571 >> 172.21.15.53:0/2237687571 conn(0x7fa1640d3000 msgr2=0x7fa1640d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:03.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 -- 172.21.15.53:0/2237687571 shutdown_connections 2024-01-16T15:15:03.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.140+0000 7fa169a00640 1 -- 172.21.15.53:0/2237687571 wait complete. 2024-01-16T15:15:03.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.142+0000 7fa169a00640 1 Processor -- start 2024-01-16T15:15:03.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.142+0000 7fa169a00640 1 -- start start 2024-01-16T15:15:03.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.143+0000 7fa169a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 0x7fa1640cad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.143+0000 7fa169a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1640cf9a0 0x7fa16415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.143+0000 7fa169a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa16415d4b0 0x7fa16409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.143+0000 7fa169a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640cb2c0 con 0x7fa1640b41e0 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.143+0000 7fa169a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa16409fca0 con 0x7fa16415d4b0 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.143+0000 7fa169a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1640cb4d0 con 0x7fa1640cf9a0 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 0x7fa1640cad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 0x7fa1640cad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52906/0 (socket says 172.21.15.53:52906) 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 -- 172.21.15.53:0/1601201919 learned_addr learned my addr 172.21.15.53:0/1601201919 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 -- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1640cf9a0 msgr2=0x7fa16415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1640cf9a0 0x7fa16415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 -- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa16415d4b0 msgr2=0x7fa16409ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa169000640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa16415d4b0 0x7fa16409ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa16415d4b0 0x7fa16409ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:03.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 -- 172.21.15.53:0/1601201919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1540094d0 con 0x7fa1640b41e0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.144+0000 7fa163e00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 0x7fa1640cad80 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fa154009870 tx=0x7fa1540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.145+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15403e070 con 0x7fa1640b41e0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.145+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa154004030 con 0x7fa1640b41e0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.145+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1540043e0 con 0x7fa1640b41e0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.145+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16415da10 con 0x7fa1640b41e0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.145+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1640cba90 con 0x7fa1640b41e0 2024-01-16T15:15:03.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.146+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa164067450 con 0x7fa1640b41e0 2024-01-16T15:15:03.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.147+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 33) v1 ==== 90814+0+0 (secure 0 0 0) 0x7fa154042070 con 0x7fa1640b41e0 2024-01-16T15:15:03.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.149+0000 7fa160c00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa13c06c0d0 0x7fa13c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:03.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.149+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5766+0+0 (secure 0 0 0) 0x7fa1540aa7a0 con 0x7fa1640b41e0 2024-01-16T15:15:03.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.149+0000 7fa15b200640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa13c06c0d0 0x7fa13c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:03.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.151+0000 7fa15b200640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa13c06c0d0 0x7fa13c06e590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa16406cdd0 tx=0x7fa14c007910 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:03.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.153+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa154077060 con 0x7fa1640b41e0 2024-01-16T15:15:03.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.378+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7fa1640d1530 con 0x7fa13c06c0d0 2024-01-16T15:15:03.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:03.426+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 34) v1 ==== 90916+0+0 (secure 0 0 0) 0x7fa154077430 con 0x7fa1640b41e0 2024-01-16T15:15:03.845 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:15:03.862+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:15:03.933 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:15:05.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:05.443+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa15407fbb0 con 0x7fa1640b41e0 2024-01-16T15:15:06.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.893+0000 7fa160c00640 1 -- 172.21.15.53:0/1601201919 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7fa1640d1530 con 0x7fa13c06c0d0 2024-01-16T15:15:06.877 INFO:teuthology.orchestra.run.smithi053.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:15:06.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.898+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa13c06c0d0 msgr2=0x7fa13c06e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.898+0000 7fa169a00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa13c06c0d0 0x7fa13c06e590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fa16406cdd0 tx=0x7fa14c007910 comp rx=0 tx=0).stop 2024-01-16T15:15:06.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.898+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 msgr2=0x7fa1640cad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:06.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.898+0000 7fa169a00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 0x7fa1640cad80 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fa154009870 tx=0x7fa1540046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.899+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 shutdown_connections 2024-01-16T15:15:06.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.900+0000 7fa169a00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa13c06c0d0 0x7fa13c06e590 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.900+0000 7fa169a00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa16415d4b0 0x7fa16409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.900+0000 7fa169a00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa1640cf9a0 0x7fa16415c150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.900+0000 7fa169a00640 1 --2- 172.21.15.53:0/1601201919 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa1640b41e0 0x7fa1640cad80 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:06.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.900+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 >> 172.21.15.53:0/1601201919 conn(0x7fa1640d3000 msgr2=0x7fa16406b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:06.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.900+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 shutdown_connections 2024-01-16T15:15:06.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:06.901+0000 7fa169a00640 1 -- 172.21.15.53:0/1601201919 wait complete. 2024-01-16T15:15:06.916 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:15:06.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:07.010 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:07.107 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:15:07.107 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:15:07.107 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:07.153 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:07.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.415+0000 7f7bf6a00640 1 Processor -- start 2024-01-16T15:15:07.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.415+0000 7f7bf6a00640 1 -- start start 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7bf0099ea0 0x7f7bf006eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf6a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf00cfa70 con 0x7f7bf015e880 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf6a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf00cff00 con 0x7f7bf0099a60 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf6a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf006fc70 con 0x7f7bf0099ea0 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7befe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7bf015e880 0x7f7bf006ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57536/0 (socket says 172.21.15.53:57536) 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7befe00640 1 -- 172.21.15.53:0/1535461637 learned_addr learned my addr 172.21.15.53:0/1535461637 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:07.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf4a00640 1 --2- 172.21.15.53:0/1535461637 >> v2:172.21.15.53:3300/0 conn(0x7f7bf0099ea0 0x7f7bf006eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf4a00640 1 -- 172.21.15.53:0/1535461637 >> v1:172.21.15.146:6789/0 conn(0x7f7bf0099a60 legacy=0x7f7bf006e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf4a00640 1 -- 172.21.15.53:0/1535461637 >> v1:172.21.15.53:6789/0 conn(0x7f7bf015e880 legacy=0x7f7bf006ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.417+0000 7f7bf4a00640 1 -- 172.21.15.53:0/1535461637 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf00cf420 con 0x7f7bf0099ea0 2024-01-16T15:15:07.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.418+0000 7f7bf4a00640 1 --2- 172.21.15.53:0/1535461637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099ea0 0x7f7bf006eed0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7be80097a0 tx=0x7f7be802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87555d624d787ab0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.419+0000 7f7beea00640 1 -- 172.21.15.53:0/1535461637 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8030610 con 0x7f7bf0099ea0 2024-01-16T15:15:07.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.419+0000 7f7beea00640 1 -- 172.21.15.53:0/1535461637 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8030770 con 0x7f7bf0099ea0 2024-01-16T15:15:07.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.419+0000 7f7beea00640 1 -- 172.21.15.53:0/1535461637 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8030b00 con 0x7f7bf0099ea0 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.419+0000 7f7bf6a00640 1 -- 172.21.15.53:0/1535461637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099ea0 msgr2=0x7f7bf006eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.419+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/1535461637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099ea0 0x7f7bf006eed0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f7be80097a0 tx=0x7f7be802b960 comp rx=0 tx=0).stop 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.420+0000 7f7bf6a00640 1 -- 172.21.15.53:0/1535461637 shutdown_connections 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.420+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/1535461637 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099ea0 0x7f7bf006eed0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.420+0000 7f7bf6a00640 1 -- 172.21.15.53:0/1535461637 >> 172.21.15.53:0/1535461637 conn(0x7f7bf00b2d00 msgr2=0x7f7bf00b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.420+0000 7f7bf6a00640 1 -- 172.21.15.53:0/1535461637 shutdown_connections 2024-01-16T15:15:07.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.420+0000 7f7bf6a00640 1 -- 172.21.15.53:0/1535461637 wait complete. 2024-01-16T15:15:07.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.421+0000 7f7bf6a00640 1 Processor -- start 2024-01-16T15:15:07.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.422+0000 7f7bf6a00640 1 -- start start 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7bf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 0x7f7bf00c5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7bf6a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bf0099ea0 0x7f7bf014d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7bf6a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bf015e880 0x7f7bf014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7bf6a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf014dd30 con 0x7f7bf0099a60 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7bf6a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00ca2a0 con 0x7f7bf015e880 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7bf6a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf00ca580 con 0x7f7bf0099ea0 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 0x7f7bf00c5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 0x7f7bf00c5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52936/0 (socket says 172.21.15.53:52936) 2024-01-16T15:15:07.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 -- 172.21.15.53:0/3936382225 learned_addr learned my addr 172.21.15.53:0/3936382225 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:07.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 -- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bf0099ea0 msgr2=0x7f7bf014d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bf0099ea0 0x7f7bf014d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 -- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bf015e880 msgr2=0x7f7bf014d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bf015e880 0x7f7bf014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.423+0000 7f7befe00640 1 -- 172.21.15.53:0/3936382225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be8009400 con 0x7f7bf0099a60 2024-01-16T15:15:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.424+0000 7f7befe00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 0x7f7bf00c5b10 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f7be000cbc0 tx=0x7f7be000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.424+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be000aca0 con 0x7f7bf0099a60 2024-01-16T15:15:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.424+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be000ae00 con 0x7f7bf0099a60 2024-01-16T15:15:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.424+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be0012600 con 0x7f7bf0099a60 2024-01-16T15:15:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.424+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf00ca860 con 0x7f7bf0099a60 2024-01-16T15:15:07.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.424+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf00cae20 con 0x7f7bf0099a60 2024-01-16T15:15:07.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.425+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf009a2d0 con 0x7f7bf0099a60 2024-01-16T15:15:07.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.426+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f7be0007070 con 0x7f7bf0099a60 2024-01-16T15:15:07.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.427+0000 7f7be7e00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7bc806c160 0x7f7bc806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.427+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f7be0088780 con 0x7f7bf0099a60 2024-01-16T15:15:07.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.428+0000 7f7bef400640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7bc806c160 0x7f7bc806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.429+0000 7f7bef400640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7bc806c160 0x7f7bc806e620 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7bf015e5e0 tx=0x7f7bd80079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.432+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7bf009a2d0 con 0x7f7bf0099a60 2024-01-16T15:15:07.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.629+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7bf009a2d0 con 0x7f7bf0099a60 2024-01-16T15:15:07.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.631+0000 7f7be7e00640 1 -- 172.21.15.53:0/3936382225 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f7bf009a2d0 con 0x7f7bf0099a60 2024-01-16T15:15:07.616 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:07.616 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":91,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:06.876072+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/3609060592":"2024-01-17T15:15:03.862542+0000","172.21.15.53:6832/3609060592":"2024-01-17T15:15:03.862542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.634+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7bc806c160 msgr2=0x7f7bc806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.634+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7bc806c160 0x7f7bc806e620 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f7bf015e5e0 tx=0x7f7bd80079b0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.634+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 msgr2=0x7f7bf00c5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.634+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 0x7f7bf00c5b10 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f7be000cbc0 tx=0x7f7be000a060 comp rx=0 tx=0).stop 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 shutdown_connections 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7bc806c160 0x7f7bc806e620 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7bf015e880 0x7f7bf014d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7bf0099ea0 0x7f7bf014d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 --2- 172.21.15.53:0/3936382225 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7bf0099a60 0x7f7bf00c5b10 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 >> 172.21.15.53:0/3936382225 conn(0x7f7bf00b2d00 msgr2=0x7f7bf00cbae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:07.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 shutdown_connections 2024-01-16T15:15:07.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.635+0000 7f7bf6a00640 1 -- 172.21.15.53:0/3936382225 wait complete. 2024-01-16T15:15:07.643 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:07.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:07.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.904+0000 7fcc31200640 1 Processor -- start 2024-01-16T15:15:07.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.904+0000 7fcc31200640 1 -- start start 2024-01-16T15:15:07.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.906+0000 7fcc31200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc2c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.906+0000 7fcc31200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c070c40 con 0x7fcc2c0b3ae0 2024-01-16T15:15:07.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.906+0000 7fcc31200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc2c06c470 con 0x7fcc2c0cf290 2024-01-16T15:15:07.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.906+0000 7fcc31200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c071450 con 0x7fcc2c0ae5a0 2024-01-16T15:15:07.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.906+0000 7fcc21200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcc2c0b3ae0 0x7fcc2c06cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57540/0 (socket says 172.21.15.53:57540) 2024-01-16T15:15:07.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.906+0000 7fcc21200640 1 -- 172.21.15.53:0/163789474 learned_addr learned my addr 172.21.15.53:0/163789474 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929118772 0 0) 0x7fcc2c070c40 con 0x7fcc2c0b3ae0 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc0c003520 con 0x7fcc2c0b3ae0 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29e00640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc2c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2790072210 0 0) 0x7fcc0c003520 con 0x7fcc2c0b3ae0 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 msgr2=0x7fcc2c15e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc2c15e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 >> v1:172.21.15.146:6789/0 conn(0x7fcc2c0cf290 legacy=0x7fcc2c0b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c06c750 con 0x7fcc2c0b3ae0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fcc14001670 con 0x7fcc2c0b3ae0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29e00640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc2c15e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 >> v1:172.21.15.53:6789/0 conn(0x7fcc2c0b3ae0 legacy=0x7fcc2c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc0c0039a0 0x7fcc0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc0c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc0c004d00 0x7fcc0c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c071450 con 0x7fcc0c004d00 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c00bba0 con 0x7fcc0c0039a0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.907+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c00bd30 con 0x7fcc2c0ae5a0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc2a800640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc0c004d00 0x7fcc0c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc29e00640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc0c0039a0 0x7fcc0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc21200640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc0c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc2a800640 1 -- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 msgr2=0x7fcc0c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc2a800640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc0c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc2a800640 1 -- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc0c0039a0 msgr2=0x7fcc0c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:07.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc2a800640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc0c0039a0 0x7fcc0c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc2a800640 1 -- 172.21.15.53:0/163789474 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c003520 con 0x7fcc0c004d00 2024-01-16T15:15:07.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.908+0000 7fcc21200640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc2c0ae5a0 0x7fcc0c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.909+0000 7fcc2a800640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc0c004d00 0x7fcc0c005180 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fcc24001750 tx=0x7fcc0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7492c590cdbb9a2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.909+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240052d0 con 0x7fcc0c004d00 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.909+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2402e050 con 0x7fcc0c004d00 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.909+0000 7fcc29400640 1 -- 172.21.15.53:0/163789474 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc240035f0 con 0x7fcc0c004d00 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc31200640 1 -- 172.21.15.53:0/163789474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc0c004d00 msgr2=0x7fcc0c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc31200640 1 --2- 172.21.15.53:0/163789474 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc0c004d00 0x7fcc0c005180 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7fcc24001750 tx=0x7fcc0c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc2a800640 1 -- 172.21.15.53:0/163789474 reap_dead start 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc31200640 1 -- 172.21.15.53:0/163789474 shutdown_connections 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc31200640 1 -- 172.21.15.53:0/163789474 >> 172.21.15.53:0/163789474 conn(0x7fcc2c06a800 msgr2=0x7fcc2c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc31200640 1 -- 172.21.15.53:0/163789474 shutdown_connections 2024-01-16T15:15:07.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.910+0000 7fcc31200640 1 -- 172.21.15.53:0/163789474 wait complete. 2024-01-16T15:15:07.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.912+0000 7fcc31200640 1 Processor -- start 2024-01-16T15:15:07.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.912+0000 7fcc31200640 1 -- start start 2024-01-16T15:15:07.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.913+0000 7fcc31200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc0c004d00 0x7fcc2c144480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.913+0000 7fcc31200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 0x7fcc2c0ad1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.913+0000 7fcc31200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2c0cd870 0x7fcc2c0cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.913+0000 7fcc31200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0b2580 con 0x7fcc2c0acd40 2024-01-16T15:15:07.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.913+0000 7fcc31200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c147790 con 0x7fcc2c0cd870 2024-01-16T15:15:07.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.913+0000 7fcc31200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c0ad6e0 con 0x7fcc0c004d00 2024-01-16T15:15:07.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 0x7fcc2c0ad1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 0x7fcc2c0ad1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52964/0 (socket says 172.21.15.53:52964) 2024-01-16T15:15:07.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 -- 172.21.15.53:0/968635019 learned_addr learned my addr 172.21.15.53:0/968635019 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc29e00640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc0c004d00 0x7fcc2c144480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc2a800640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2c0cd870 0x7fcc2c0cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 -- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc0c004d00 msgr2=0x7fcc2c144480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc0c004d00 0x7fcc2c144480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 -- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2c0cd870 msgr2=0x7fcc2c0cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2c0cd870 0x7fcc2c0cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:07.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 -- 172.21.15.53:0/968635019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc2c0ce5f0 con 0x7fcc2c0acd40 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc29e00640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc0c004d00 0x7fcc2c144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc2a800640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2c0cd870 0x7fcc2c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.914+0000 7fcc21200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 0x7fcc2c0ad1a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400d890 tx=0x7fcc1400dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.915+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc140074e0 con 0x7fcc2c0acd40 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.915+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc14007640 con 0x7fcc2c0acd40 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.915+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc140049e0 con 0x7fcc2c0acd40 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.915+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c147140 con 0x7fcc2c0acd40 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.915+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c146de0 con 0x7fcc2c0acd40 2024-01-16T15:15:07.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.916+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c147940 con 0x7fcc2c0acd40 2024-01-16T15:15:07.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.917+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fcc14004b40 con 0x7fcc2c0acd40 2024-01-16T15:15:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.918+0000 7fcc22a00640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc0406c160 0x7fcc0406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.918+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fcc14088310 con 0x7fcc2c0acd40 2024-01-16T15:15:07.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.919+0000 7fcc29e00640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc0406c160 0x7fcc0406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:07.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.920+0000 7fcc29e00640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc0406c160 0x7fcc0406e620 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0098e0 tx=0x7fcc1c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:07.907 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:07.923+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcc2c147940 con 0x7fcc2c0acd40 2024-01-16T15:15:08.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.114+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc2c147940 con 0x7fcc2c0acd40 2024-01-16T15:15:08.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.115+0000 7fcc22a00640 1 -- 172.21.15.53:0/968635019 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc2c147940 con 0x7fcc2c0acd40 2024-01-16T15:15:08.099 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:15:08.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc0406c160 msgr2=0x7fcc0406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc0406c160 0x7fcc0406e620 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c0098e0 tx=0x7fcc1c00b040 comp rx=0 tx=0).stop 2024-01-16T15:15:08.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 msgr2=0x7fcc2c0ad1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 0x7fcc2c0ad1a0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7fcc1400d890 tx=0x7fcc1400dd50 comp rx=0 tx=0).stop 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 shutdown_connections 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcc0406c160 0x7fcc0406e620 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcc2c0cd870 0x7fcc2c0cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcc2c0acd40 0x7fcc2c0ad1a0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 --2- 172.21.15.53:0/968635019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcc0c004d00 0x7fcc2c144480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.118+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 >> 172.21.15.53:0/968635019 conn(0x7fcc2c06a800 msgr2=0x7fcc2c0b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.119+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 shutdown_connections 2024-01-16T15:15:08.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.119+0000 7fcc31200640 1 -- 172.21.15.53:0/968635019 wait complete. 2024-01-16T15:15:08.122 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:15:08.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.349+0000 7f79b7e00640 1 Processor -- start 2024-01-16T15:15:08.333 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.349+0000 7f79b7e00640 1 -- start start 2024-01-16T15:15:08.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.350+0000 7f79b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f79b8143c40 0x7f79b8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.350+0000 7f79b7e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b8070e50 con 0x7f79b80ccc10 2024-01-16T15:15:08.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b8144020 con 0x7f79b80717d0 2024-01-16T15:15:08.334 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b80723c0 con 0x7f79b8143c40 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b7400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f79b80ccc10 0x7f79b806d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57542/0 (socket says 172.21.15.53:57542) 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b7400640 1 -- 172.21.15.53:0/1460753331 learned_addr learned my addr 172.21.15.53:0/1460753331 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1460753331 >> v2:172.21.15.146:3300/0 conn(0x7f79b8143c40 0x7f79b8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302503384 0 0) 0x7f79b8070e50 con 0x7f79b80ccc10 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.351+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7998003520 con 0x7f79b80ccc10 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 71981387 0 0) 0x7f7998003520 con 0x7f79b80ccc10 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 >> v2:172.21.15.146:3300/0 conn(0x7f79b8143c40 msgr2=0x7f79b8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 --2- 172.21.15.53:0/1460753331 >> v2:172.21.15.146:3300/0 conn(0x7f79b8143c40 0x7f79b8071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 >> v1:172.21.15.146:6789/0 conn(0x7f79b80717d0 legacy=0x7f79b80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b8071e00 con 0x7f79b80ccc10 2024-01-16T15:15:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f79b00019f0 con 0x7f79b80ccc10 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 >> v1:172.21.15.53:6789/0 conn(0x7f79b80ccc10 legacy=0x7f79b806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79980039c0 0x7f7998003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980047c0 0x7f7998004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980051d0 0x7f799800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b80723c0 con 0x7f79980039c0 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799800c120 con 0x7f79980051d0 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f799800c2e0 con 0x7f79980047c0 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.352+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79980039c0 0x7f7998003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79afe00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980047c0 0x7f7998004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b6a00640 1 -- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980047c0 msgr2=0x7f7998004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79980047c0 0x7f7998004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b6a00640 1 -- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980051d0 msgr2=0x7f799800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b7400640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980051d0 0x7f799800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980051d0 0x7f799800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b6a00640 1 -- 172.21.15.53:0/1460753331 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7998003520 con 0x7f79980039c0 2024-01-16T15:15:08.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b7400640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980051d0 0x7f799800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:08.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.353+0000 7f79b7400640 1 -- 172.21.15.53:0/1460753331 reap_dead start 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.354+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79980039c0 0x7f7998003de0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009a90 tx=0x7f79980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4fef42cc8a862db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.354+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a802c680 con 0x7f79980039c0 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.354+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a802c7e0 con 0x7f79980039c0 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.354+0000 7f79b6000640 1 -- 172.21.15.53:0/1460753331 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a802cb70 con 0x7f79980039c0 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 -- 172.21.15.53:0/1460753331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79980039c0 msgr2=0x7f7998003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79980039c0 0x7f7998003de0 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f79a8009a90 tx=0x7f79980057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 -- 172.21.15.53:0/1460753331 shutdown_connections 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1460753331 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79980039c0 0x7f7998003de0 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 -- 172.21.15.53:0/1460753331 >> 172.21.15.53:0/1460753331 conn(0x7f79b806a800 msgr2=0x7f79b806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 -- 172.21.15.53:0/1460753331 shutdown_connections 2024-01-16T15:15:08.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.355+0000 7f79b7e00640 1 -- 172.21.15.53:0/1460753331 wait complete. 2024-01-16T15:15:08.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.357+0000 7f79b7e00640 1 Processor -- start 2024-01-16T15:15:08.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.357+0000 7f79b7e00640 1 -- start start 2024-01-16T15:15:08.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.358+0000 7f79b7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980039c0 0x7f79b815e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.358+0000 7f79b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 0x7f79b815c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.358+0000 7f79b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b8135cc0 0x7f79b8136140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.358+0000 7f79b7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b815cfa0 con 0x7f79b815c270 2024-01-16T15:15:08.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.358+0000 7f79b7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b815d280 con 0x7f79980039c0 2024-01-16T15:15:08.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.358+0000 7f79b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b815d560 con 0x7f79b8135cc0 2024-01-16T15:15:08.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 0x7f79b815c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 0x7f79b815c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52978/0 (socket says 172.21.15.53:52978) 2024-01-16T15:15:08.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 -- 172.21.15.53:0/1683822798 learned_addr learned my addr 172.21.15.53:0/1683822798 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 -- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b8135cc0 msgr2=0x7f79b8136140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b8135cc0 0x7f79b8136140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 -- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980039c0 msgr2=0x7f79b815e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980039c0 0x7f79b815e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 -- 172.21.15.53:0/1683822798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b815d840 con 0x7f79b815c270 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79afe00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 0x7f79b815c6f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f79a000d570 tx=0x7f79a000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.359+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0005480 con 0x7f79b815c270 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.360+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0003610 con 0x7f79b815c270 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.360+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79a0003980 con 0x7f79b815c270 2024-01-16T15:15:08.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.360+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b815db20 con 0x7f79b815c270 2024-01-16T15:15:08.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.360+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b8136680 con 0x7f79b815c270 2024-01-16T15:15:08.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.361+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b8067450 con 0x7f79b815c270 2024-01-16T15:15:08.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.362+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f79a0003ae0 con 0x7f79b815c270 2024-01-16T15:15:08.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.363+0000 7f79aea00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f799006c190 0x7f799006e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:08.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.363+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f79a0088800 con 0x7f79b815c270 2024-01-16T15:15:08.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.363+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f799006c190 0x7f799006e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:08.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.365+0000 7f79b6a00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f799006c190 0x7f799006e650 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f79a8036000 tx=0x7f79a802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:08.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.368+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f79a0054de0 con 0x7f79b815c270 2024-01-16T15:15:08.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.581+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f79b80ceb30 con 0x7f79b815c270 2024-01-16T15:15:08.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.582+0000 7f79aea00640 1 -- 172.21.15.53:0/1683822798 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v91) v1 ==== 80+0+122 (secure 0 0 0) 0x7f79a0001230 con 0x7f79b815c270 2024-01-16T15:15:08.566 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6834/3609060592 2024-01-17T15:15:03.862542+0000 2024-01-16T15:15:08.567 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6832/3609060592 2024-01-17T15:15:03.862542+0000 2024-01-16T15:15:08.567 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:15:08.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f799006c190 msgr2=0x7f799006e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f799006c190 0x7f799006e650 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f79a8036000 tx=0x7f79a802e000 comp rx=0 tx=0).stop 2024-01-16T15:15:08.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 msgr2=0x7f79b815c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 0x7f79b815c6f0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f79a000d570 tx=0x7f79a000da30 comp rx=0 tx=0).stop 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 shutdown_connections 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f799006c190 0x7f799006e650 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f79b8135cc0 0x7f79b8136140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f79b815c270 0x7f79b815c6f0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 --2- 172.21.15.53:0/1683822798 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f79980039c0 0x7f79b815e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 >> 172.21.15.53:0/1683822798 conn(0x7f79b806a800 msgr2=0x7f79b80b0360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 shutdown_connections 2024-01-16T15:15:08.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:08.585+0000 7f79b7e00640 1 -- 172.21.15.53:0/1683822798 wait complete. 2024-01-16T15:15:08.589 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:15:08.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:08.590 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:08.692 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:15:08.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:08.692 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:15:08.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[68494]: fuse finished with error 0 and tester_r 0 2024-01-16T15:15:08.948 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:15:08.948 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:09.006 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:09.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.253+0000 7f1a7fe00640 1 Processor -- start 2024-01-16T15:15:09.237 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.253+0000 7f1a7fe00640 1 -- start start 2024-01-16T15:15:09.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.255+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a800cc500 0x7f1a800cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.255+0000 7f1a7fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a80070e50 con 0x7f1a80143c40 2024-01-16T15:15:09.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.255+0000 7f1a7fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a800ccea0 con 0x7f1a8006d160 2024-01-16T15:15:09.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.255+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a800723c0 con 0x7f1a800cc500 2024-01-16T15:15:09.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a7ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1a80143c40 0x7f1a80071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57548/0 (socket says 172.21.15.53:57548) 2024-01-16T15:15:09.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a7ea00640 1 -- 172.21.15.53:0/525862927 learned_addr learned my addr 172.21.15.53:0/525862927 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:09.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a75e00640 1 --2- 172.21.15.53:0/525862927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a800cc500 0x7f1a800cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412831294 0 0) 0x7f1a80070e50 con 0x7f1a80143c40 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a60003520 con 0x7f1a80143c40 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227182229 0 0) 0x7f1a800ccea0 con 0x7f1a8006d160 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.256+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a80070e50 con 0x7f1a8006d160 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.257+0000 7f1a75e00640 1 -- 172.21.15.53:0/525862927 >> v1:172.21.15.146:6789/0 conn(0x7f1a8006d160 legacy=0x7f1a8006d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.257+0000 7f1a75e00640 1 -- 172.21.15.53:0/525862927 >> v1:172.21.15.53:6789/0 conn(0x7f1a80143c40 legacy=0x7f1a80071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.257+0000 7f1a75e00640 1 -- 172.21.15.53:0/525862927 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a80144020 con 0x7f1a800cc500 2024-01-16T15:15:09.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.258+0000 7f1a75e00640 1 --2- 172.21.15.53:0/525862927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a800cc500 0x7f1a800cc960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a680097d0 tx=0x7f1a6802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32932c9eee60fa23 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6802d040 con 0x7f1a800cc500 2024-01-16T15:15:09.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a68031810 con 0x7f1a800cc500 2024-01-16T15:15:09.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7e000640 1 -- 172.21.15.53:0/525862927 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68031ba0 con 0x7f1a800cc500 2024-01-16T15:15:09.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7fe00640 1 -- 172.21.15.53:0/525862927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a800cc500 msgr2=0x7f1a800cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/525862927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a800cc500 0x7f1a800cc960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1a680097d0 tx=0x7f1a6802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7fe00640 1 -- 172.21.15.53:0/525862927 shutdown_connections 2024-01-16T15:15:09.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/525862927 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a800cc500 0x7f1a800cc960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7fe00640 1 -- 172.21.15.53:0/525862927 >> 172.21.15.53:0/525862927 conn(0x7f1a8006a800 msgr2=0x7f1a8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.259+0000 7f1a7fe00640 1 -- 172.21.15.53:0/525862927 shutdown_connections 2024-01-16T15:15:09.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.260+0000 7f1a7fe00640 1 -- 172.21.15.53:0/525862927 wait complete. 2024-01-16T15:15:09.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.261+0000 7f1a7fe00640 1 Processor -- start 2024-01-16T15:15:09.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.261+0000 7f1a7fe00640 1 -- start start 2024-01-16T15:15:09.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.262+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a8006d160 0x7f1a800b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.262+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 0x7f1a8013c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.262+0000 7f1a7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a80143c40 0x7f1a800b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.262+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8013cbc0 con 0x7f1a800cc500 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.262+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a800b2b30 con 0x7f1a8006d160 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.262+0000 7f1a7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a800b2e10 con 0x7f1a80143c40 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 0x7f1a8013c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 0x7f1a8013c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52984/0 (socket says 172.21.15.53:52984) 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 -- 172.21.15.53:0/1782851697 learned_addr learned my addr 172.21.15.53:0/1782851697 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a7ea00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a8006d160 0x7f1a800b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 -- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a80143c40 msgr2=0x7f1a800b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a80143c40 0x7f1a800b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 -- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a8006d160 msgr2=0x7f1a800b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a8006d160 0x7f1a800b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a75e00640 1 -- 172.21.15.53:0/1782851697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a68009400 con 0x7f1a800cc500 2024-01-16T15:15:09.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.263+0000 7f1a7ea00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a8006d160 0x7f1a800b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:09.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.264+0000 7f1a75e00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 0x7f1a8013c680 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f1a68000c00 tx=0x7f1a680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.264+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a6803ace0 con 0x7f1a800cc500 2024-01-16T15:15:09.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.264+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a6803ae60 con 0x7f1a800cc500 2024-01-16T15:15:09.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.264+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a800b30f0 con 0x7f1a800cc500 2024-01-16T15:15:09.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.264+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a68044640 con 0x7f1a800cc500 2024-01-16T15:15:09.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.264+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a800b36b0 con 0x7f1a800cc500 2024-01-16T15:15:09.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.265+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a80070b10 con 0x7f1a800cc500 2024-01-16T15:15:09.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.266+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f1a680447a0 con 0x7f1a800cc500 2024-01-16T15:15:09.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.267+0000 7f1a77400640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a5806c160 0x7f1a5806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.267+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1a680aaa90 con 0x7f1a800cc500 2024-01-16T15:15:09.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.268+0000 7f1a7ea00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a5806c160 0x7f1a5806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.269+0000 7f1a7ea00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a5806c160 0x7f1a5806e620 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1a70007fd0 tx=0x7f1a70009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.256 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.272+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1a68077470 con 0x7f1a800cc500 2024-01-16T15:15:09.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.469+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a800b38a0 con 0x7f1a800cc500 2024-01-16T15:15:09.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.471+0000 7f1a77400640 1 -- 172.21.15.53:0/1782851697 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f1a6807aa90 con 0x7f1a800cc500 2024-01-16T15:15:09.455 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:09.455 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":91,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:06.876072+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/3609060592":"2024-01-17T15:15:03.862542+0000","172.21.15.53:6832/3609060592":"2024-01-17T15:15:03.862542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:09.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a5806c160 msgr2=0x7f1a5806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a5806c160 0x7f1a5806e620 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f1a70007fd0 tx=0x7f1a70009040 comp rx=0 tx=0).stop 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 msgr2=0x7f1a8013c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 0x7f1a8013c680 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f1a68000c00 tx=0x7f1a680046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 shutdown_connections 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1a5806c160 0x7f1a5806e620 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1a80143c40 0x7f1a800b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1a800cc500 0x7f1a8013c680 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 --2- 172.21.15.53:0/1782851697 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1a8006d160 0x7f1a800b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.473+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 >> 172.21.15.53:0/1782851697 conn(0x7f1a8006a800 msgr2=0x7f1a800aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.474+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 shutdown_connections 2024-01-16T15:15:09.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.474+0000 7f1a7fe00640 1 -- 172.21.15.53:0/1782851697 wait complete. 2024-01-16T15:15:09.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:09.478 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:09.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.724+0000 7fa8dce00640 1 Processor -- start 2024-01-16T15:15:09.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.724+0000 7fa8dce00640 1 -- start start 2024-01-16T15:15:09.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.726+0000 7fa8dce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d813df80 0x7fa8d815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.726+0000 7fa8dce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fa8d80cf9a0 0x7fa8d80cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.726+0000 7fa8dce00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d806c470 con 0x7fa8d813e360 2024-01-16T15:15:09.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.726+0000 7fa8dce00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d806c750 con 0x7fa8d80cf9a0 2024-01-16T15:15:09.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.726+0000 7fa8dce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d806ca30 con 0x7fa8d813df80 2024-01-16T15:15:09.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.727+0000 7fa8d5e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d813df80 0x7fa8d815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.727+0000 7fa8cd200640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fa8d813e360 0x7fa8d80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:39486/0 (socket says 172.21.15.53:39486) 2024-01-16T15:15:09.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.727+0000 7fa8cd200640 1 -- 172.21.15.53:0/2799919298 learned_addr learned my addr 172.21.15.53:0/2799919298 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:09.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.727+0000 7fa8d6800640 1 --2- 172.21.15.53:0/2799919298 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80cf9a0 0x7fa8d80cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.728+0000 7fa8d5e00640 1 -- 172.21.15.53:0/2799919298 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80cf9a0 msgr2=0x7fa8d80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:09.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.728+0000 7fa8d5e00640 1 --2- 172.21.15.53:0/2799919298 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80cf9a0 0x7fa8d80cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.728+0000 7fa8d5400640 1 -- 172.21.15.53:0/2799919298 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876088636 0 0) 0x7fa8d806c470 con 0x7fa8d813e360 2024-01-16T15:15:09.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.728+0000 7fa8d5e00640 1 -- 172.21.15.53:0/2799919298 >> v1:172.21.15.146:6789/0 conn(0x7fa8d813e360 legacy=0x7fa8d80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.728+0000 7fa8d5e00640 1 -- 172.21.15.53:0/2799919298 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d806cd10 con 0x7fa8d813df80 2024-01-16T15:15:09.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.729+0000 7fa8d5e00640 1 --2- 172.21.15.53:0/2799919298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d813df80 0x7fa8d815bee0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009d90 tx=0x7fa8c802bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a178898edb3e6182 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8d5400640 1 -- 172.21.15.53:0/2799919298 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c802cc50 con 0x7fa8d813df80 2024-01-16T15:15:09.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8d5400640 1 -- 172.21.15.53:0/2799919298 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c802cdb0 con 0x7fa8d813df80 2024-01-16T15:15:09.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8d5400640 1 -- 172.21.15.53:0/2799919298 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8c8034660 con 0x7fa8d813df80 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8dce00640 1 -- 172.21.15.53:0/2799919298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d813df80 msgr2=0x7fa8d815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8dce00640 1 --2- 172.21.15.53:0/2799919298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d813df80 0x7fa8d815bee0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8009d90 tx=0x7fa8c802bea0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8dce00640 1 -- 172.21.15.53:0/2799919298 shutdown_connections 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8dce00640 1 --2- 172.21.15.53:0/2799919298 >> v2:172.21.15.146:3300/0 conn(0x7fa8d80cf9a0 0x7fa8d80cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.730+0000 7fa8dce00640 1 --2- 172.21.15.53:0/2799919298 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d813df80 0x7fa8d815bee0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.731+0000 7fa8dce00640 1 -- 172.21.15.53:0/2799919298 >> 172.21.15.53:0/2799919298 conn(0x7fa8d806a800 msgr2=0x7fa8d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.731+0000 7fa8dce00640 1 -- 172.21.15.53:0/2799919298 shutdown_connections 2024-01-16T15:15:09.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.731+0000 7fa8dce00640 1 -- 172.21.15.53:0/2799919298 wait complete. 2024-01-16T15:15:09.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.733+0000 7fa8dce00640 1 Processor -- start 2024-01-16T15:15:09.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.733+0000 7fa8dce00640 1 -- start start 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.734+0000 7fa8dce00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80cf9a0 0x7fa8d81531e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.734+0000 7fa8dce00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d813df80 0x7fa8d80a9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.734+0000 7fa8dce00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 0x7fa8d8159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.734+0000 7fa8dce00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d8071ca0 con 0x7fa8d813e360 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8dce00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d8153720 con 0x7fa8d813df80 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8dce00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80a9d10 con 0x7fa8d80cf9a0 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 0x7fa8d8159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 0x7fa8d8159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:52994/0 (socket says 172.21.15.53:52994) 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 -- 172.21.15.53:0/3713703674 learned_addr learned my addr 172.21.15.53:0/3713703674 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d5e00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80cf9a0 0x7fa8d81531e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8cd200640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d813df80 0x7fa8d80a9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 -- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80cf9a0 msgr2=0x7fa8d81531e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80cf9a0 0x7fa8d81531e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 -- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d813df80 msgr2=0x7fa8d80a9270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d813df80 0x7fa8d80a9270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.735+0000 7fa8d6800640 1 -- 172.21.15.53:0/3713703674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8009a70 con 0x7fa8d813e360 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.736+0000 7fa8d6800640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 0x7fa8d8159a30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000e820 tx=0x7fa8d000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.736+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0009510 con 0x7fa8d813e360 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.736+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d000ee70 con 0x7fa8d813e360 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.736+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d8071370 con 0x7fa8d813e360 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.736+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d000cb80 con 0x7fa8d813e360 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.736+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d8071930 con 0x7fa8d813e360 2024-01-16T15:15:09.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.737+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d8071e10 con 0x7fa8d813e360 2024-01-16T15:15:09.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.738+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fa8d000cce0 con 0x7fa8d813e360 2024-01-16T15:15:09.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.740+0000 7fa8cea00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8b006c160 0x7fa8b006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:09.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.740+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fa8d0088750 con 0x7fa8d813e360 2024-01-16T15:15:09.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.740+0000 7fa8d5e00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8b006c160 0x7fa8b006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:09.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.742+0000 7fa8d5e00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8b006c160 0x7fa8b006e620 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8036000 tx=0x7fa8c8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:09.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.744+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa8d0055010 con 0x7fa8d813e360 2024-01-16T15:15:09.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.948+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa8d80675d0 con 0x7fa8d813e360 2024-01-16T15:15:09.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.949+0000 7fa8cea00640 1 -- 172.21.15.53:0/3713703674 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa8d0058630 con 0x7fa8d813e360 2024-01-16T15:15:09.933 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8b006c160 msgr2=0x7fa8b006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8b006c160 0x7fa8b006e620 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7fa8c8036000 tx=0x7fa8c8030000 comp rx=0 tx=0).stop 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 msgr2=0x7fa8d8159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 0x7fa8d8159a30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000e820 tx=0x7fa8d000ec20 comp rx=0 tx=0).stop 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 shutdown_connections 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa8b006c160 0x7fa8b006e620 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8d813e360 0x7fa8d8159a30 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa8d813df80 0x7fa8d80a9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 --2- 172.21.15.53:0/3713703674 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8d80cf9a0 0x7fa8d81531e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:09.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 >> 172.21.15.53:0/3713703674 conn(0x7fa8d806a800 msgr2=0x7fa8d815dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:09.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 shutdown_connections 2024-01-16T15:15:09.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:09.952+0000 7fa8dce00640 1 -- 172.21.15.53:0/3713703674 wait complete. 2024-01-16T15:15:09.957 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:15:10.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.191+0000 7fcbf7e00640 1 Processor -- start 2024-01-16T15:15:10.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.192+0000 7fcbf7e00640 1 -- start start 2024-01-16T15:15:10.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.193+0000 7fcbf7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbf80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.193+0000 7fcbf7e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf80d0a10 con 0x7fcbf80cfe70 2024-01-16T15:15:10.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.193+0000 7fcbf7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf8161e30 con 0x7fcbf80cf9f0 2024-01-16T15:15:10.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.193+0000 7fcbf7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80d1f50 con 0x7fcbf8160920 2024-01-16T15:15:10.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.194+0000 7fcbf7400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcbf80cfe70 0x7fcbf8161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57550/0 (socket says 172.21.15.53:57550) 2024-01-16T15:15:10.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.194+0000 7fcbf7400640 1 -- 172.21.15.53:0/206546528 learned_addr learned my addr 172.21.15.53:0/206546528 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.194+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbf80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.194+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568367713 0 0) 0x7fcbf80d0a10 con 0x7fcbf80cfe70 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.194+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbd8003520 con 0x7fcbf80cfe70 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 199337419 0 0) 0x7fcbd8003520 con 0x7fcbf80cfe70 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 msgr2=0x7fcbf80b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbf80b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 >> v1:172.21.15.146:6789/0 conn(0x7fcbf80cf9f0 legacy=0x7fcbf8161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf80cf080 con 0x7fcbf80cfe70 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fcbf00019f0 con 0x7fcbf80cfe70 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbf80b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 >> v1:172.21.15.53:6789/0 conn(0x7fcbf80cfe70 legacy=0x7fcbf8161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbd8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbd80047c0 0x7fcbd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbd80051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80d1f50 con 0x7fcbd80039a0 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd800bb10 con 0x7fcbd80047c0 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbd800bc80 con 0x7fcbf8160920 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbd8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf7400640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbd80051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.195+0000 7fcbf6000640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbd80047c0 0x7fcbd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.196+0000 7fcbf6a00640 1 -- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 msgr2=0x7fcbd80051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.196+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8160920 0x7fcbd80051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.196+0000 7fcbf6a00640 1 -- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbd80047c0 msgr2=0x7fcbd8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:10.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.196+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbd80047c0 0x7fcbd8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.196+0000 7fcbf6a00640 1 -- 172.21.15.53:0/206546528 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd8003520 con 0x7fcbd80039a0 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.197+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbd8003dc0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80098a0 tx=0x7fcbd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d8e4c6abc451aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.197+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8004230 con 0x7fcbd80039a0 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.197+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe8004390 con 0x7fcbd80039a0 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.197+0000 7fcbf5600640 1 -- 172.21.15.53:0/206546528 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8003830 con 0x7fcbd80039a0 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7e00640 1 -- 172.21.15.53:0/206546528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 msgr2=0x7fcbd8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/206546528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbd8003dc0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80098a0 tx=0x7fcbd80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7400640 1 -- 172.21.15.53:0/206546528 reap_dead start 2024-01-16T15:15:10.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7e00640 1 -- 172.21.15.53:0/206546528 shutdown_connections 2024-01-16T15:15:10.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7e00640 1 -- 172.21.15.53:0/206546528 >> 172.21.15.53:0/206546528 conn(0x7fcbf80d3090 msgr2=0x7fcbf80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7e00640 1 -- 172.21.15.53:0/206546528 shutdown_connections 2024-01-16T15:15:10.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.198+0000 7fcbf7e00640 1 -- 172.21.15.53:0/206546528 wait complete. 2024-01-16T15:15:10.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.201+0000 7fcbf7e00640 1 Processor -- start 2024-01-16T15:15:10.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.201+0000 7fcbf7e00640 1 -- start start 2024-01-16T15:15:10.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.202+0000 7fcbf7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbf80b8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.202+0000 7fcbf7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8151610 0x7fcbf81f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.202+0000 7fcbf7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbf81f6ac0 0x7fcbf81f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.202+0000 7fcbf7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf8154c30 con 0x7fcbd80039a0 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.202+0000 7fcbf7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80cab70 con 0x7fcbf81f6ac0 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.202+0000 7fcbf7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80b5c50 con 0x7fcbf8151610 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbf80b8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbf80b8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53008/0 (socket says 172.21.15.53:53008) 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 -- 172.21.15.53:0/2800476563 learned_addr learned my addr 172.21.15.53:0/2800476563 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf7400640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbf81f6ac0 0x7fcbf81f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6000640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8151610 0x7fcbf81f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 -- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8151610 msgr2=0x7fcbf81f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8151610 0x7fcbf81f6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 -- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbf81f6ac0 msgr2=0x7fcbf81f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbf81f6ac0 0x7fcbf81f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.203+0000 7fcbf6a00640 1 -- 172.21.15.53:0/2800476563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf8151a70 con 0x7fcbd80039a0 2024-01-16T15:15:10.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.204+0000 7fcbf6a00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbf80b8550 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80098a0 tx=0x7fcbe8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.204+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8030630 con 0x7fcbd80039a0 2024-01-16T15:15:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.204+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe8030790 con 0x7fcbd80039a0 2024-01-16T15:15:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.204+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8030b20 con 0x7fcbd80039a0 2024-01-16T15:15:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.204+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf80cadc0 con 0x7fcbd80039a0 2024-01-16T15:15:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.204+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf80cb380 con 0x7fcbd80039a0 2024-01-16T15:15:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.205+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8154bc0 con 0x7fcbd80039a0 2024-01-16T15:15:10.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.207+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fcbe8030c80 con 0x7fcbd80039a0 2024-01-16T15:15:10.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.207+0000 7fcbeea00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcbd006c160 0x7fcbd006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:10.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.208+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fcbe80aa440 con 0x7fcbd80039a0 2024-01-16T15:15:10.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.208+0000 7fcbf6000640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcbd006c160 0x7fcbd006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:10.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.209+0000 7fcbf6000640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcbd006c160 0x7fcbd006e620 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0007fd0 tx=0x7fcbe0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:10.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.214+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcbe8076a20 con 0x7fcbd80039a0 2024-01-16T15:15:10.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.422+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcbf8067450 con 0x7fcbd80039a0 2024-01-16T15:15:10.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.423+0000 7fcbeea00640 1 -- 172.21.15.53:0/2800476563 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v91) v1 ==== 80+0+122 (secure 0 0 0) 0x7fcbe8048020 con 0x7fcbd80039a0 2024-01-16T15:15:10.407 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6834/3609060592 2024-01-17T15:15:03.862542+0000 2024-01-16T15:15:10.407 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6832/3609060592 2024-01-17T15:15:03.862542+0000 2024-01-16T15:15:10.407 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.425+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcbd006c160 msgr2=0x7fcbd006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.425+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcbd006c160 0x7fcbd006e620 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0007fd0 tx=0x7fcbe0009040 comp rx=0 tx=0).stop 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.425+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 msgr2=0x7fcbf80b8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.425+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbf80b8550 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7fcbe80098a0 tx=0x7fcbe8004730 comp rx=0 tx=0).stop 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 shutdown_connections 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcbd006c160 0x7fcbd006e620 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcbf81f6ac0 0x7fcbf81f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcbf8151610 0x7fcbf81f6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 --2- 172.21.15.53:0/2800476563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcbd80039a0 0x7fcbf80b8550 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:10.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 >> 172.21.15.53:0/2800476563 conn(0x7fcbf80d3090 msgr2=0x7fcbf80c9460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:10.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 shutdown_connections 2024-01-16T15:15:10.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:10.426+0000 7fcbf7e00640 1 -- 172.21.15.53:0/2800476563 wait complete. 2024-01-16T15:15:10.429 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:15:10.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:10.429 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:10.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[45986]: fuse finished with error 0 and tester_r 0 2024-01-16T15:15:10.529 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:15:10.529 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:10.530 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:15:10.710 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:15:10.710 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:10.770 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:10.770 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:15:10.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:10.771 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:15:10.828 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:15:10.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:15:10.829 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:15:10.829 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:10.887 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:10.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:15:10.888 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:10.888 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:15:10.939 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:15:10.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:15:10.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:11.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.189+0000 7fb238c00640 1 Processor -- start 2024-01-16T15:15:11.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.190+0000 7fb238c00640 1 -- start start 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb238c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb23415e3b0 0x7fb234164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb238c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340b1ee0 con 0x7fb234143ae0 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb238c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2340b2780 con 0x7fb2340cf9a0 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb238c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb23415e990 con 0x7fb23415e3b0 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb232800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb23415e3b0 0x7fb234164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb232800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb23415e3b0 0x7fb234164de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53024/0 (socket says 172.21.15.53:53024) 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb232800640 1 -- 172.21.15.53:0/3309987256 learned_addr learned my addr 172.21.15.53:0/3309987256 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:11.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.192+0000 7fb231e00640 1 --1- 172.21.15.53:0/3309987256 >> v1:172.21.15.53:6789/0 conn(0x7fb234143ae0 0x7fb2340cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57554/0 (socket says 172.21.15.53:57554) 2024-01-16T15:15:11.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.193+0000 7fb232800640 1 -- 172.21.15.53:0/3309987256 >> v1:172.21.15.146:6789/0 conn(0x7fb2340cf9a0 legacy=0x7fb23415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.193+0000 7fb232800640 1 -- 172.21.15.53:0/3309987256 >> v1:172.21.15.53:6789/0 conn(0x7fb234143ae0 legacy=0x7fb2340cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.193+0000 7fb232800640 1 -- 172.21.15.53:0/3309987256 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2340b2130 con 0x7fb23415e3b0 2024-01-16T15:15:11.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.193+0000 7fb230a00640 1 -- 172.21.15.53:0/3309987256 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834939391 0 0) 0x7fb2340b1ee0 con 0x7fb234143ae0 2024-01-16T15:15:11.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.194+0000 7fb232800640 1 --2- 172.21.15.53:0/3309987256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb23415e3b0 0x7fb234164de0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00bb10 tx=0x7fb22c02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81a0eb3d50e68724 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.194+0000 7fb230a00640 1 -- 172.21.15.53:0/3309987256 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c036440 con 0x7fb23415e3b0 2024-01-16T15:15:11.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.194+0000 7fb230a00640 1 -- 172.21.15.53:0/3309987256 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22c0365a0 con 0x7fb23415e3b0 2024-01-16T15:15:11.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.194+0000 7fb230a00640 1 -- 172.21.15.53:0/3309987256 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22c036930 con 0x7fb23415e3b0 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 -- 172.21.15.53:0/3309987256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb23415e3b0 msgr2=0x7fb234164de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 --2- 172.21.15.53:0/3309987256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb23415e3b0 0x7fb234164de0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7fb22c00bb10 tx=0x7fb22c02e160 comp rx=0 tx=0).stop 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 -- 172.21.15.53:0/3309987256 shutdown_connections 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 --2- 172.21.15.53:0/3309987256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb23415e3b0 0x7fb234164de0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 -- 172.21.15.53:0/3309987256 >> 172.21.15.53:0/3309987256 conn(0x7fb23406a800 msgr2=0x7fb23406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 -- 172.21.15.53:0/3309987256 shutdown_connections 2024-01-16T15:15:11.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.195+0000 7fb238c00640 1 -- 172.21.15.53:0/3309987256 wait complete. 2024-01-16T15:15:11.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.196+0000 7fb238c00640 1 Processor -- start 2024-01-16T15:15:11.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.197+0000 7fb238c00640 1 -- start start 2024-01-16T15:15:11.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb238c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 0x7fb2340ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb238c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb234143ae0 0x7fb234141c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb238c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23415e3b0 0x7fb2340c6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb238c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340ac8d0 con 0x7fb2340cf9a0 2024-01-16T15:15:11.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb238c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340adb40 con 0x7fb234143ae0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb238c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2340c6b70 con 0x7fb23415e3b0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 0x7fb2340ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 0x7fb2340ad600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53030/0 (socket says 172.21.15.53:53030) 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 -- 172.21.15.53:0/2399893701 learned_addr learned my addr 172.21.15.53:0/2399893701 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 -- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23415e3b0 msgr2=0x7fb2340c6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23415e3b0 0x7fb2340c6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 -- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb234143ae0 msgr2=0x7fb234141c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb234143ae0 0x7fb234141c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.198+0000 7fb231e00640 1 -- 172.21.15.53:0/2399893701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb22c00b770 con 0x7fb2340cf9a0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.199+0000 7fb231e00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 0x7fb2340ad600 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c890 tx=0x7fb22400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.199+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22400a640 con 0x7fb2340cf9a0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.199+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22400a7c0 con 0x7fb2340cf9a0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.199+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2340acb20 con 0x7fb2340cf9a0 2024-01-16T15:15:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.199+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22400ab70 con 0x7fb2340cf9a0 2024-01-16T15:15:11.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.199+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2340ad0e0 con 0x7fb2340cf9a0 2024-01-16T15:15:11.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.200+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2340ad300 con 0x7fb2340cf9a0 2024-01-16T15:15:11.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.201+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fb22400acd0 con 0x7fb2340cf9a0 2024-01-16T15:15:11.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.202+0000 7fb22a000640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb20c06c160 0x7fb20c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.202+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb224088260 con 0x7fb2340cf9a0 2024-01-16T15:15:11.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.203+0000 7fb231400640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb20c06c160 0x7fb20c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.204+0000 7fb231400640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb20c06c160 0x7fb20c06e620 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb2340affb0 tx=0x7fb21c0079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.207+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb224054840 con 0x7fb2340cf9a0 2024-01-16T15:15:11.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.446+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb23406c3f0 con 0x7fb2340cf9a0 2024-01-16T15:15:11.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.447+0000 7fb22a000640 1 -- 172.21.15.53:0/2399893701 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fb224018020 con 0x7fb2340cf9a0 2024-01-16T15:15:11.431 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:11.431 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6119,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3084871157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3084871157},{"type":"v1","addr":"172.21.15.146:6835","nonce":3084871157}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2008413931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2008413931},{"type":"v1","addr":"172.21.15.53:6835","nonce":2008413931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/614678519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":614678519},{"type":"v1","addr":"172.21.15.146:6837","nonce":614678519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/2844987177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2844987177},{"type":"v1","addr":"172.21.15.53:6834","nonce":2844987177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58}],"filesystems":[]} 2024-01-16T15:15:11.431 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 58 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.449+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb20c06c160 msgr2=0x7fb20c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.449+0000 7fb238c00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb20c06c160 0x7fb20c06e620 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fb2340affb0 tx=0x7fb21c0079b0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.449+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 msgr2=0x7fb2340ad600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.449+0000 7fb238c00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 0x7fb2340ad600 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7fb22400c890 tx=0x7fb22400cd50 comp rx=0 tx=0).stop 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 shutdown_connections 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb20c06c160 0x7fb20c06e620 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb23415e3b0 0x7fb2340c6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb234143ae0 0x7fb234141c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 --2- 172.21.15.53:0/2399893701 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb2340cf9a0 0x7fb2340ad600 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 >> 172.21.15.53:0/2399893701 conn(0x7fb23406a800 msgr2=0x7fb23406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 shutdown_connections 2024-01-16T15:15:11.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.450+0000 7fb238c00640 1 -- 172.21.15.53:0/2399893701 wait complete. 2024-01-16T15:15:11.453 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:15:11.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.689+0000 7f76e9c00640 1 Processor -- start 2024-01-16T15:15:11.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.689+0000 7f76e9c00640 1 -- start start 2024-01-16T15:15:11.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.691+0000 7f76e9c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f76e4143810 0x7f76e4071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.691+0000 7f76e9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e415e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.691+0000 7f76e9c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76e415d770 0x7f76e415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.691+0000 7f76e9c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e4143bf0 con 0x7f76e4143810 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.691+0000 7f76e9c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76e415e140 con 0x7f76e415d770 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.691+0000 7f76e9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e406c730 con 0x7f76e40ae170 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f76e4143810 0x7f76e4071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f76e4143810 0x7f76e4071e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53044/0 (socket says 172.21.15.53:53044) 2024-01-16T15:15:11.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 -- 172.21.15.53:0/268186237 learned_addr learned my addr 172.21.15.53:0/268186237 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e9200640 1 --2- 172.21.15.53:0/268186237 >> v2:172.21.15.146:3300/0 conn(0x7f76e415d770 0x7f76e415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3400640 1 --2- 172.21.15.53:0/268186237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e415e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 -- 172.21.15.53:0/268186237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 msgr2=0x7f76e415e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 --2- 172.21.15.53:0/268186237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e415e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 -- 172.21.15.53:0/268186237 >> v2:172.21.15.146:3300/0 conn(0x7f76e415d770 msgr2=0x7f76e415dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 --2- 172.21.15.53:0/268186237 >> v2:172.21.15.146:3300/0 conn(0x7f76e415d770 0x7f76e415dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.692+0000 7f76e3e00640 1 -- 172.21.15.53:0/268186237 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e40ae5f0 con 0x7f76e4143810 2024-01-16T15:15:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.694+0000 7f76e3e00640 1 --2- 172.21.15.53:0/268186237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e4071e40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f76e40acc20 tx=0x7f76d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6240b34182aa7a5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.694+0000 7f76e2a00640 1 -- 172.21.15.53:0/268186237 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d40317e0 con 0x7f76e4143810 2024-01-16T15:15:11.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e2a00640 1 -- 172.21.15.53:0/268186237 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76d4031940 con 0x7f76e4143810 2024-01-16T15:15:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e2a00640 1 -- 172.21.15.53:0/268186237 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76d4031cd0 con 0x7f76e4143810 2024-01-16T15:15:11.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 -- 172.21.15.53:0/268186237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 msgr2=0x7f76e4071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 --2- 172.21.15.53:0/268186237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e4071e40 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f76e40acc20 tx=0x7f76d402b880 comp rx=0 tx=0).stop 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 -- 172.21.15.53:0/268186237 shutdown_connections 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 --2- 172.21.15.53:0/268186237 >> v2:172.21.15.146:3300/0 conn(0x7f76e415d770 0x7f76e415dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 --2- 172.21.15.53:0/268186237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e415e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 --2- 172.21.15.53:0/268186237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e4071e40 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 -- 172.21.15.53:0/268186237 >> 172.21.15.53:0/268186237 conn(0x7f76e406a7d0 msgr2=0x7f76e406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 -- 172.21.15.53:0/268186237 shutdown_connections 2024-01-16T15:15:11.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.695+0000 7f76e9c00640 1 -- 172.21.15.53:0/268186237 wait complete. 2024-01-16T15:15:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.697+0000 7f76e9c00640 1 Processor -- start 2024-01-16T15:15:11.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.697+0000 7f76e9c00640 1 -- start start 2024-01-16T15:15:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.698+0000 7f76e9c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e4145f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.698+0000 7f76e9c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e413cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.698+0000 7f76e9c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76e415d770 0x7f76e415c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.698+0000 7f76e9c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e413df00 con 0x7f76e4143810 2024-01-16T15:15:11.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.698+0000 7f76e9c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76e40c3d00 con 0x7f76e415d770 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.698+0000 7f76e9c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76e4146440 con 0x7f76e40ae170 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e413cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e413cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53058/0 (socket says 172.21.15.53:53058) 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 -- 172.21.15.53:0/3184775400 learned_addr learned my addr 172.21.15.53:0/3184775400 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e9200640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76e415d770 0x7f76e415c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3e00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e4145f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 -- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 msgr2=0x7f76e4145f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e4145f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 -- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76e415d770 msgr2=0x7f76e415c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76e415d770 0x7f76e415c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.699+0000 7f76e3400640 1 -- 172.21.15.53:0/3184775400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76d40094d0 con 0x7f76e4143810 2024-01-16T15:15:11.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.700+0000 7f76e3400640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e413cb00 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00d800 tx=0x7f76cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.700+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc0096a0 con 0x7f76e4143810 2024-01-16T15:15:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.700+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76cc009800 con 0x7f76e4143810 2024-01-16T15:15:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.700+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76cc009b90 con 0x7f76e4143810 2024-01-16T15:15:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.700+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76e413e320 con 0x7f76e4143810 2024-01-16T15:15:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.700+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76e413d330 con 0x7f76e4143810 2024-01-16T15:15:11.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.701+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76e4067450 con 0x7f76e4143810 2024-01-16T15:15:11.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.702+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f76cc009cf0 con 0x7f76e4143810 2024-01-16T15:15:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.704+0000 7f76dbe00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f76bc06c160 0x7f76bc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.704+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f76cc088b30 con 0x7f76e4143810 2024-01-16T15:15:11.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.704+0000 7f76e3e00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f76bc06c160 0x7f76bc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:11.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.706+0000 7f76e3e00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f76bc06c160 0x7f76bc06e620 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f76d4036000 tx=0x7f76d403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:11.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.708+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f76cc0553f0 con 0x7f76e4143810 2024-01-16T15:15:11.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.900+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f76e4070b10 con 0x7f76e4143810 2024-01-16T15:15:11.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.906+0000 7f76dbe00640 1 -- 172.21.15.53:0/3184775400 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v33) v1 ==== 120+0+0 (secure 0 0 0) 0x7f76cc00b440 con 0x7f76e4143810 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f76bc06c160 msgr2=0x7f76bc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f76bc06c160 0x7f76bc06e620 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f76d4036000 tx=0x7f76d403d040 comp rx=0 tx=0).stop 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 msgr2=0x7f76e413cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e413cb00 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f76cc00d800 tx=0x7f76cc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 shutdown_connections 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f76bc06c160 0x7f76bc06e620 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.909+0000 7f76e9c00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f76e415d770 0x7f76e415c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.910+0000 7f76e9c00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76e4143810 0x7f76e413cb00 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.910+0000 7f76e9c00640 1 --2- 172.21.15.53:0/3184775400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76e40ae170 0x7f76e4145f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.910+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 >> 172.21.15.53:0/3184775400 conn(0x7f76e406a7d0 msgr2=0x7f76e40b3640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.910+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 shutdown_connections 2024-01-16T15:15:11.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:11.910+0000 7f76e9c00640 1 -- 172.21.15.53:0/3184775400 wait complete. 2024-01-16T15:15:11.915 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:15:12.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.146+0000 7f5c83800640 1 Processor -- start 2024-01-16T15:15:12.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.147+0000 7f5c83800640 1 -- start start 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.148+0000 7f5c83800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c7c0c9870 0x7f5c7c0ceb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.148+0000 7f5c83800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c7c161740 0x7f5c7c0c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.148+0000 7f5c83800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c7c0cf4c0 con 0x7f5c7c0d0560 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.148+0000 7f5c83800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7c0d2ef0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.148+0000 7f5c83800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c7c0c9c50 con 0x7f5c7c161740 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c7c0c9870 0x7f5c7c0ceb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c7c0c9870 0x7f5c7c0ceb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49852/0 (socket says 172.21.15.53:49852) 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 -- 172.21.15.53:0/3260996983 learned_addr learned my addr 172.21.15.53:0/3260996983 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:12.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c81800640 1 --2- 172.21.15.53:0/3260996983 >> v2:172.21.15.146:3300/0 conn(0x7f5c7c161740 0x7f5c7c0c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 -- 172.21.15.53:0/3260996983 >> v2:172.21.15.146:3300/0 conn(0x7f5c7c161740 msgr2=0x7f5c7c0c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 --2- 172.21.15.53:0/3260996983 >> v2:172.21.15.146:3300/0 conn(0x7f5c7c161740 0x7f5c7c0c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 -- 172.21.15.53:0/3260996983 >> v1:172.21.15.146:6789/0 conn(0x7f5c7c0d0560 legacy=0x7f5c7c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.149+0000 7f5c80e00640 1 -- 172.21.15.53:0/3260996983 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7c0cf7a0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.151+0000 7f5c80e00640 1 --2- 172.21.15.53:0/3260996983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c0ceb50 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c08d8f0 tx=0x7f5c6c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183f75e17baad235 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.151+0000 7f5c7be00640 1 -- 172.21.15.53:0/3260996983 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c031830 con 0x7f5c7c0c9870 2024-01-16T15:15:12.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.151+0000 7f5c7be00640 1 -- 172.21.15.53:0/3260996983 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c031990 con 0x7f5c7c0c9870 2024-01-16T15:15:12.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.151+0000 7f5c7be00640 1 -- 172.21.15.53:0/3260996983 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c031d20 con 0x7f5c7c0c9870 2024-01-16T15:15:12.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 -- 172.21.15.53:0/3260996983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 msgr2=0x7f5c7c0ceb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 --2- 172.21.15.53:0/3260996983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c0ceb50 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c08d8f0 tx=0x7f5c6c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 -- 172.21.15.53:0/3260996983 shutdown_connections 2024-01-16T15:15:12.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 --2- 172.21.15.53:0/3260996983 >> v2:172.21.15.146:3300/0 conn(0x7f5c7c161740 0x7f5c7c0c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 --2- 172.21.15.53:0/3260996983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c0ceb50 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 -- 172.21.15.53:0/3260996983 >> 172.21.15.53:0/3260996983 conn(0x7f5c7c09c560 msgr2=0x7f5c7c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 -- 172.21.15.53:0/3260996983 shutdown_connections 2024-01-16T15:15:12.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.152+0000 7f5c83800640 1 -- 172.21.15.53:0/3260996983 wait complete. 2024-01-16T15:15:12.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.154+0000 7f5c83800640 1 Processor -- start 2024-01-16T15:15:12.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.154+0000 7f5c83800640 1 -- start start 2024-01-16T15:15:12.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.155+0000 7f5c83800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c15ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.155+0000 7f5c83800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c7c0d0560 0x7f5c7c15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.155+0000 7f5c83800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c7c161740 0x7f5c7c15b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.155+0000 7f5c83800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c15bb40 con 0x7f5c7c0c9870 2024-01-16T15:15:12.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.155+0000 7f5c83800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0b71f0 con 0x7f5c7c0d0560 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.155+0000 7f5c83800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7c0b8150 con 0x7f5c7c161740 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c15ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c15ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49860/0 (socket says 172.21.15.53:49860) 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 -- 172.21.15.53:0/4186481644 learned_addr learned my addr 172.21.15.53:0/4186481644 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c73e00640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c7c0d0560 0x7f5c7c15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c81800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c7c161740 0x7f5c7c15b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 -- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c7c161740 msgr2=0x7f5c7c15b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c7c161740 0x7f5c7c15b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 -- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c7c0d0560 msgr2=0x7f5c7c15b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c7c0d0560 0x7f5c7c15b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.156+0000 7f5c80e00640 1 -- 172.21.15.53:0/4186481644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c0094d0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.157+0000 7f5c80e00640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c15ab80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009960 tx=0x7f5c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.157+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c03a8f0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.157+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c03aa50 con 0x7f5c7c0c9870 2024-01-16T15:15:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.157+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c7c0b8430 con 0x7f5c7c0c9870 2024-01-16T15:15:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.157+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c6c03ae00 con 0x7f5c7c0c9870 2024-01-16T15:15:12.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.157+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7c0b89f0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.159+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c7c0675d0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.160+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f5c6c04d450 con 0x7f5c7c0c9870 2024-01-16T15:15:12.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.161+0000 7f5c79600640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5406c190 0x7f5c5406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.161+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5c6c0aaa00 con 0x7f5c7c0c9870 2024-01-16T15:15:12.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.162+0000 7f5c73e00640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5406c190 0x7f5c5406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.163+0000 7f5c73e00640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5406c190 0x7f5c5406e650 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0b92c0 tx=0x7f5c64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.166+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c6c076fe0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.360+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7f5c7c0cd570 con 0x7f5c7c0c9870 2024-01-16T15:15:12.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.366+0000 7f5c79600640 1 -- 172.21.15.53:0/4186481644 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v34) v1 ==== 115+0+0 (secure 0 0 0) 0x7f5c6c0350b0 con 0x7f5c7c0c9870 2024-01-16T15:15:12.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5406c190 msgr2=0x7f5c5406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5406c190 0x7f5c5406e650 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f5c7c0b92c0 tx=0x7f5c64009040 comp rx=0 tx=0).stop 2024-01-16T15:15:12.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 msgr2=0x7f5c7c15ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c15ab80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c009960 tx=0x7f5c6c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 shutdown_connections 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5406c190 0x7f5c5406e650 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c7c161740 0x7f5c7c15b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c7c0d0560 0x7f5c7c15b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 --2- 172.21.15.53:0/4186481644 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c7c0c9870 0x7f5c7c15ab80 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 >> 172.21.15.53:0/4186481644 conn(0x7f5c7c09c560 msgr2=0x7f5c7c0d1450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 shutdown_connections 2024-01-16T15:15:12.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.369+0000 7f5c83800640 1 -- 172.21.15.53:0/4186481644 wait complete. 2024-01-16T15:15:12.373 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:15:12.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.615+0000 7fb819000640 1 Processor -- start 2024-01-16T15:15:12.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.615+0000 7fb819000640 1 -- start start 2024-01-16T15:15:12.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.616+0000 7fb819000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb814160770 0x7fb8140ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.616+0000 7fb819000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406e5a0 0x7fb814070d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb819000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb814066430 con 0x7fb8140b2490 2024-01-16T15:15:12.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb819000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81406a560 con 0x7fb814160770 2024-01-16T15:15:12.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb819000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81406a820 con 0x7fb81406e5a0 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb813e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb8140b2490 0x7fb814160e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51556/0 (socket says 172.21.15.53:51556) 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb813400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb814160770 0x7fb8140ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb813e00640 1 -- 172.21.15.53:0/464865846 learned_addr learned my addr 172.21.15.53:0/464865846 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb812a00640 1 --2- 172.21.15.53:0/464865846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406e5a0 0x7fb814070d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb813400640 1 -- 172.21.15.53:0/464865846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406e5a0 msgr2=0x7fb814070d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb813400640 1 --2- 172.21.15.53:0/464865846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406e5a0 0x7fb814070d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.617+0000 7fb813400640 1 -- 172.21.15.53:0/464865846 >> v1:172.21.15.53:6789/0 conn(0x7fb8140b2490 legacy=0x7fb814160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.618+0000 7fb813400640 1 -- 172.21.15.53:0/464865846 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81406ab00 con 0x7fb814160770 2024-01-16T15:15:12.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.618+0000 7fb812000640 1 -- 172.21.15.53:0/464865846 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788388896 0 0) 0x7fb814066430 con 0x7fb8140b2490 2024-01-16T15:15:12.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.619+0000 7fb813400640 1 --2- 172.21.15.53:0/464865846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb814160770 0x7fb8140ac8d0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb8040097a0 tx=0x7fb80402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8756f4569e584fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.619+0000 7fb812000640 1 -- 172.21.15.53:0/464865846 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb80402d040 con 0x7fb814160770 2024-01-16T15:15:12.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.619+0000 7fb812000640 1 -- 172.21.15.53:0/464865846 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb804031810 con 0x7fb814160770 2024-01-16T15:15:12.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.619+0000 7fb812000640 1 -- 172.21.15.53:0/464865846 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb804031ba0 con 0x7fb814160770 2024-01-16T15:15:12.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 -- 172.21.15.53:0/464865846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb814160770 msgr2=0x7fb8140ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 --2- 172.21.15.53:0/464865846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb814160770 0x7fb8140ac8d0 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7fb8040097a0 tx=0x7fb80402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 -- 172.21.15.53:0/464865846 shutdown_connections 2024-01-16T15:15:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 --2- 172.21.15.53:0/464865846 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb81406e5a0 0x7fb814070d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 --2- 172.21.15.53:0/464865846 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb814160770 0x7fb8140ac8d0 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 -- 172.21.15.53:0/464865846 >> 172.21.15.53:0/464865846 conn(0x7fb8140b15b0 msgr2=0x7fb8140b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 -- 172.21.15.53:0/464865846 shutdown_connections 2024-01-16T15:15:12.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.620+0000 7fb819000640 1 -- 172.21.15.53:0/464865846 wait complete. 2024-01-16T15:15:12.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.622+0000 7fb819000640 1 Processor -- start 2024-01-16T15:15:12.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.622+0000 7fb819000640 1 -- start start 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb819000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81406e5a0 0x7fb8140cade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb819000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 0x7fb81406cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb819000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb814160770 0x7fb81406d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb819000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb814146a80 con 0x7fb8140b2490 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb819000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8141474e0 con 0x7fb81406e5a0 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb819000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb814149ad0 con 0x7fb814160770 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb812a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 0x7fb81406cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb812a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 0x7fb81406cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49890/0 (socket says 172.21.15.53:49890) 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.623+0000 7fb812a00640 1 -- 172.21.15.53:0/2265722910 learned_addr learned my addr 172.21.15.53:0/2265722910 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb813e00640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb814160770 0x7fb81406d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb813400640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81406e5a0 0x7fb8140cade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb812a00640 1 -- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb814160770 msgr2=0x7fb81406d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb812a00640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb814160770 0x7fb81406d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb812a00640 1 -- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81406e5a0 msgr2=0x7fb8140cade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb812a00640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81406e5a0 0x7fb8140cade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb812a00640 1 -- 172.21.15.53:0/2265722910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb804009400 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb812a00640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 0x7fb81406cea0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00d800 tx=0x7fb7fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.624+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc00b6c0 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.625+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7fc00b820 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.625+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7fc00bbb0 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.625+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb814146e90 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.625+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb814148da0 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.626+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb814147690 con 0x7fb8140b2490 2024-01-16T15:15:12.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.627+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fb7fc00bd10 con 0x7fb8140b2490 2024-01-16T15:15:12.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.628+0000 7fb80b400640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb7ec06c160 0x7fb7ec06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:12.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.628+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fb7fc088ef0 con 0x7fb8140b2490 2024-01-16T15:15:12.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.629+0000 7fb813400640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb7ec06c160 0x7fb7ec06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:12.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.630+0000 7fb813400640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb7ec06c160 0x7fb7ec06e620 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb804036000 tx=0x7fb80403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:12.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.633+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb814147690 con 0x7fb8140b2490 2024-01-16T15:15:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.826+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fb814147650 con 0x7fb8140b2490 2024-01-16T15:15:12.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.834+0000 7fb80b400640 1 -- 172.21.15.53:0/2265722910 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v35) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb7fc0554d0 con 0x7fb8140b2490 2024-01-16T15:15:12.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb7ec06c160 msgr2=0x7fb7ec06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb7ec06c160 0x7fb7ec06e620 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fb804036000 tx=0x7fb80403c040 comp rx=0 tx=0).stop 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 msgr2=0x7fb81406cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 0x7fb81406cea0 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc00d800 tx=0x7fb7fc00dcc0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 shutdown_connections 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb7ec06c160 0x7fb7ec06e620 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb814160770 0x7fb81406d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb8140b2490 0x7fb81406cea0 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 --2- 172.21.15.53:0/2265722910 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb81406e5a0 0x7fb8140cade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 >> 172.21.15.53:0/2265722910 conn(0x7fb8140b15b0 msgr2=0x7fb8140ae5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 shutdown_connections 2024-01-16T15:15:12.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:12.836+0000 7fb819000640 1 -- 172.21.15.53:0/2265722910 wait complete. 2024-01-16T15:15:12.839 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled' 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.069+0000 7ff3c7c00640 1 Processor -- start 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.069+0000 7ff3c7c00640 1 -- start start 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c7c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff3c00cf9a0 0x7ff3c015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c7c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3c015e1e0 0x7ff3c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c7c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00b1ee0 con 0x7ff3c0143ae0 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c7c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00b2780 con 0x7ff3c015e1e0 2024-01-16T15:15:13.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c7c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c015e990 con 0x7ff3c00cf9a0 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3c015e1e0 0x7ff3c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff3c015e1e0 0x7ff3c00702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49902/0 (socket says 172.21.15.53:49902) 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.071+0000 7ff3c5c00640 1 -- 172.21.15.53:0/2913894230 learned_addr learned my addr 172.21.15.53:0/2913894230 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.072+0000 7ff3b7c00640 1 --2- 172.21.15.53:0/2913894230 >> v2:172.21.15.146:3300/0 conn(0x7ff3c00cf9a0 0x7ff3c015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.072+0000 7ff3c5c00640 1 -- 172.21.15.53:0/2913894230 >> v2:172.21.15.146:3300/0 conn(0x7ff3c00cf9a0 msgr2=0x7ff3c015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.072+0000 7ff3c5c00640 1 --2- 172.21.15.53:0/2913894230 >> v2:172.21.15.146:3300/0 conn(0x7ff3c00cf9a0 0x7ff3c015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.072+0000 7ff3c5c00640 1 -- 172.21.15.53:0/2913894230 >> v1:172.21.15.146:6789/0 conn(0x7ff3c0143ae0 legacy=0x7ff3c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.072+0000 7ff3c5c00640 1 -- 172.21.15.53:0/2913894230 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c00b2130 con 0x7ff3c015e1e0 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.072+0000 7ff3b7c00640 1 --2- 172.21.15.53:0/2913894230 >> v2:172.21.15.146:3300/0 conn(0x7ff3c00cf9a0 0x7ff3c015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:13.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.073+0000 7ff3c5c00640 1 --2- 172.21.15.53:0/2913894230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c00702c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00cb830 tx=0x7ff3b802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea0eae372ba2795 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.073+0000 7ff3bfe00640 1 -- 172.21.15.53:0/2913894230 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b802ca80 con 0x7ff3c015e1e0 2024-01-16T15:15:13.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.073+0000 7ff3bfe00640 1 -- 172.21.15.53:0/2913894230 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b802cbe0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.073+0000 7ff3bfe00640 1 -- 172.21.15.53:0/2913894230 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b80343f0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 -- 172.21.15.53:0/2913894230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 msgr2=0x7ff3c00702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/2913894230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c00702c0 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7ff3c00cb830 tx=0x7ff3b802bee0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 -- 172.21.15.53:0/2913894230 shutdown_connections 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/2913894230 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c00702c0 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/2913894230 >> v2:172.21.15.146:3300/0 conn(0x7ff3c00cf9a0 0x7ff3c015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 -- 172.21.15.53:0/2913894230 >> 172.21.15.53:0/2913894230 conn(0x7ff3c006a800 msgr2=0x7ff3c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 -- 172.21.15.53:0/2913894230 shutdown_connections 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.074+0000 7ff3c7c00640 1 -- 172.21.15.53:0/2913894230 wait complete. 2024-01-16T15:15:13.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.076+0000 7ff3c7c00640 1 Processor -- start 2024-01-16T15:15:13.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.076+0000 7ff3c7c00640 1 -- start start 2024-01-16T15:15:13.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.077+0000 7ff3c7c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3c00cf9a0 0x7ff3c0159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.077+0000 7ff3c7c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3c0143ae0 0x7ff3c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.077+0000 7ff3c7c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c0070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.077+0000 7ff3c7c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00c7090 con 0x7ff3c015e1e0 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.077+0000 7ff3c7c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00c84c0 con 0x7ff3c0143ae0 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.077+0000 7ff3c7c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00c3ef0 con 0x7ff3c00cf9a0 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c0070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c0070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49908/0 (socket says 172.21.15.53:49908) 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 -- 172.21.15.53:0/3127929510 learned_addr learned my addr 172.21.15.53:0/3127929510 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3b7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3c0143ae0 0x7ff3c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5200640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3c00cf9a0 0x7ff3c0159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 -- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3c00cf9a0 msgr2=0x7ff3c0159b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3c00cf9a0 0x7ff3c0159b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 -- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3c0143ae0 msgr2=0x7ff3c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3c0143ae0 0x7ff3c00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 -- 172.21.15.53:0/3127929510 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b8009b30 con 0x7ff3c015e1e0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5200640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3c00cf9a0 0x7ff3c0159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.078+0000 7ff3c5c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c0070800 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8036040 tx=0x7ff3b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.079+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b803e450 con 0x7ff3c015e1e0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.079+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b803e5d0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.079+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b803e980 con 0x7ff3c015e1e0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.079+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c00c4190 con 0x7ff3c015e1e0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.079+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c00c4750 con 0x7ff3c015e1e0 2024-01-16T15:15:13.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.080+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c015a060 con 0x7ff3c015e1e0 2024-01-16T15:15:13.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.081+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7ff3b802cd50 con 0x7ff3c015e1e0 2024-01-16T15:15:13.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.082+0000 7ff3bd600640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff39806c160 0x7ff39806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.083+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff3b80ab2e0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.083+0000 7ff3c5200640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff39806c160 0x7ff39806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.084+0000 7ff3c5200640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff39806c160 0x7ff39806e620 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0007fd0 tx=0x7ff3b0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.087+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff3b8077ba0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.355+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]} v 0) v1 -- 0x7ff3c00675d0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.412+0000 7ff3bd600640 1 -- 172.21.15.53:0/3127929510 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_1_clone_is_cancelled"]}]=0 v664) v1 ==== 153+0+0 (secure 0 0 0) 0x7ff3b80430e0 con 0x7ff3c015e1e0 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff39806c160 msgr2=0x7ff39806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff39806c160 0x7ff39806e620 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff3b0007fd0 tx=0x7ff3b0009040 comp rx=0 tx=0).stop 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 msgr2=0x7ff3c0070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c0070800 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8036040 tx=0x7ff3b8004a40 comp rx=0 tx=0).stop 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 shutdown_connections 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff39806c160 0x7ff39806e620 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff3c015e1e0 0x7ff3c0070800 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff3c0143ae0 0x7ff3c00702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 --2- 172.21.15.53:0/3127929510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff3c00cf9a0 0x7ff3c0159b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.415+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 >> 172.21.15.53:0/3127929510 conn(0x7ff3c006a800 msgr2=0x7ff3c006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.400 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.416+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 shutdown_connections 2024-01-16T15:15:13.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.416+0000 7ff3c7c00640 1 -- 172.21.15.53:0/3127929510 wait complete. 2024-01-16T15:15:13.419 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:15:13.420 INFO:tasks.cephfs_test_runner:Starting test: test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that the progress bars for ongoing clones is removed from the 2024-01-16T15:15:13.420 INFO:tasks.cephfs_test_runner:test_when_3_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:15:13.420 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:13.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.655+0000 7f66de800640 1 Processor -- start 2024-01-16T15:15:13.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.655+0000 7f66de800640 1 -- start start 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66de800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66d80cd4a0 0x7f66d80b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66de800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f66d80ad760 0x7f66d815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66de800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d815ea30 con 0x7f66d815b930 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66de800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d80adc40 con 0x7f66d80cd4a0 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66de800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d80adf20 con 0x7f66d80ad760 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66dd400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66d80cd4a0 0x7f66d80b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66dd400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66d80cd4a0 0x7f66d80b3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49916/0 (socket says 172.21.15.53:49916) 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.657+0000 7f66dd400640 1 -- 172.21.15.53:0/890431217 learned_addr learned my addr 172.21.15.53:0/890431217 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:13.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.658+0000 7f66dca00640 1 --2- 172.21.15.53:0/890431217 >> v2:172.21.15.146:3300/0 conn(0x7f66d80ad760 0x7f66d815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.658+0000 7f66dd400640 1 -- 172.21.15.53:0/890431217 >> v2:172.21.15.146:3300/0 conn(0x7f66d80ad760 msgr2=0x7f66d815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.658+0000 7f66dd400640 1 --2- 172.21.15.53:0/890431217 >> v2:172.21.15.146:3300/0 conn(0x7f66d80ad760 0x7f66d815b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.658+0000 7f66dd400640 1 -- 172.21.15.53:0/890431217 >> v1:172.21.15.146:6789/0 conn(0x7f66d815b930 legacy=0x7f66d80cf030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:13.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.658+0000 7f66dd400640 1 -- 172.21.15.53:0/890431217 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d80ae200 con 0x7f66d80cd4a0 2024-01-16T15:15:13.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.659+0000 7f66dd400640 1 --2- 172.21.15.53:0/890431217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d80cd4a0 0x7f66d80b3a50 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f66d80cebe0 tx=0x7f66c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=569b5fe18afb7b1e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66d7e00640 1 -- 172.21.15.53:0/890431217 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c80317e0 con 0x7f66d80cd4a0 2024-01-16T15:15:13.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66d7e00640 1 -- 172.21.15.53:0/890431217 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c8031940 con 0x7f66d80cd4a0 2024-01-16T15:15:13.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66d7e00640 1 -- 172.21.15.53:0/890431217 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c8031cd0 con 0x7f66d80cd4a0 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 -- 172.21.15.53:0/890431217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d80cd4a0 msgr2=0x7f66d80b3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 --2- 172.21.15.53:0/890431217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d80cd4a0 0x7f66d80b3a50 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f66d80cebe0 tx=0x7f66c802b880 comp rx=0 tx=0).stop 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 -- 172.21.15.53:0/890431217 shutdown_connections 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 --2- 172.21.15.53:0/890431217 >> v2:172.21.15.146:3300/0 conn(0x7f66d80ad760 0x7f66d815b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 --2- 172.21.15.53:0/890431217 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d80cd4a0 0x7f66d80b3a50 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 -- 172.21.15.53:0/890431217 >> 172.21.15.53:0/890431217 conn(0x7f66d806a800 msgr2=0x7f66d806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 -- 172.21.15.53:0/890431217 shutdown_connections 2024-01-16T15:15:13.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.660+0000 7f66de800640 1 -- 172.21.15.53:0/890431217 wait complete. 2024-01-16T15:15:13.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.662+0000 7f66de800640 1 Processor -- start 2024-01-16T15:15:13.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.662+0000 7f66de800640 1 -- start start 2024-01-16T15:15:13.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66de800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d80ad760 0x7f66d80c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66de800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d80cd4a0 0x7f66d80c8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66de800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 0x7f66d80afc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66de800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d814a240 con 0x7f66d815b930 2024-01-16T15:15:13.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66de800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d80b08d0 con 0x7f66d80cd4a0 2024-01-16T15:15:13.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66de800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d814a550 con 0x7f66d80ad760 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.663+0000 7f66dde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 0x7f66d80afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 0x7f66d80afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49932/0 (socket says 172.21.15.53:49932) 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 -- 172.21.15.53:0/2422156360 learned_addr learned my addr 172.21.15.53:0/2422156360 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dca00640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d80cd4a0 0x7f66d80c8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dd400640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d80ad760 0x7f66d80c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 -- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d80ad760 msgr2=0x7f66d80c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d80ad760 0x7f66d80c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 -- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d80cd4a0 msgr2=0x7f66d80c8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d80cd4a0 0x7f66d80c8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 -- 172.21.15.53:0/2422156360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c80094d0 con 0x7f66d815b930 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dca00640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d80cd4a0 0x7f66d80c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:13.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.664+0000 7f66dde00640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 0x7f66d80afc30 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f66d000b9e0 tx=0x7f66d000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.665+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d000cb20 con 0x7f66d815b930 2024-01-16T15:15:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.665+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66d000cc80 con 0x7f66d815b930 2024-01-16T15:15:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.665+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66d0014680 con 0x7f66d815b930 2024-01-16T15:15:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.665+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d814a830 con 0x7f66d815b930 2024-01-16T15:15:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.665+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d80b0bb0 con 0x7f66d815b930 2024-01-16T15:15:13.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.666+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d814ad50 con 0x7f66d815b930 2024-01-16T15:15:13.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.667+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f66d001d3b0 con 0x7f66d815b930 2024-01-16T15:15:13.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.668+0000 7f66d5600640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66b006c160 0x7f66b006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:13.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.669+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f66d0088970 con 0x7f66d815b930 2024-01-16T15:15:13.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.669+0000 7f66dd400640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66b006c160 0x7f66b006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:13.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.670+0000 7f66dd400640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66b006c160 0x7f66b006e620 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f66c8036000 tx=0x7f66c803d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:13.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.673+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66d814ad50 con 0x7f66d815b930 2024-01-16T15:15:13.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.869+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66d814ad50 con 0x7f66d815b930 2024-01-16T15:15:13.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.871+0000 7f66d5600640 1 -- 172.21.15.53:0/2422156360 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+12994 (secure 0 0 0) 0x7f66d814ad50 con 0x7f66d815b930 2024-01-16T15:15:13.855 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:13.855 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":91,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:06.876072+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/3609060592":"2024-01-17T15:15:03.862542+0000","172.21.15.53:6832/3609060592":"2024-01-17T15:15:03.862542+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66b006c160 msgr2=0x7f66b006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66b006c160 0x7f66b006e620 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f66c8036000 tx=0x7f66c803d040 comp rx=0 tx=0).stop 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 msgr2=0x7f66d80afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 0x7f66d80afc30 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f66d000b9e0 tx=0x7f66d000bea0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 shutdown_connections 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66b006c160 0x7f66b006e620 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d815b930 0x7f66d80afc30 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d80cd4a0 0x7f66d80c8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 --2- 172.21.15.53:0/2422156360 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d80ad760 0x7f66d80c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:13.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 >> 172.21.15.53:0/2422156360 conn(0x7f66d806a800 msgr2=0x7f66d80cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:13.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 shutdown_connections 2024-01-16T15:15:13.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:13.874+0000 7f66de800640 1 -- 172.21.15.53:0/2422156360 wait complete. 2024-01-16T15:15:13.879 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:13.879 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:14.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.125+0000 7fe896e00640 1 Processor -- start 2024-01-16T15:15:14.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.125+0000 7fe896e00640 1 -- start start 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe896e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe896e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe89015e1e0 0x7fe8900702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe896e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8900b1ee0 con 0x7fe890143ae0 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe896e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8900b2780 con 0x7fe89015e1e0 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe896e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe89015e990 con 0x7fe8900cf9a0 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe88f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe88f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:43514/0 (socket says 172.21.15.53:43514) 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe88f400640 1 -- 172.21.15.53:0/2146408591 learned_addr learned my addr 172.21.15.53:0/2146408591 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:14.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.127+0000 7fe894e00640 1 --2- 172.21.15.53:0/2146408591 >> v2:172.21.15.146:3300/0 conn(0x7fe89015e1e0 0x7fe8900702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.128+0000 7fe88f400640 1 -- 172.21.15.53:0/2146408591 >> v2:172.21.15.146:3300/0 conn(0x7fe89015e1e0 msgr2=0x7fe8900702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.128+0000 7fe88f400640 1 --2- 172.21.15.53:0/2146408591 >> v2:172.21.15.146:3300/0 conn(0x7fe89015e1e0 0x7fe8900702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.128+0000 7fe88f400640 1 -- 172.21.15.53:0/2146408591 >> v1:172.21.15.146:6789/0 conn(0x7fe890143ae0 legacy=0x7fe8900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.128+0000 7fe88f400640 1 -- 172.21.15.53:0/2146408591 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8900b2130 con 0x7fe8900cf9a0 2024-01-16T15:15:14.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.129+0000 7fe88f400640 1 --2- 172.21.15.53:0/2146408591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89015dca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe878009870 tx=0x7fe87802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb23a92fd7e64b9e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.130+0000 7fe88ea00640 1 -- 172.21.15.53:0/2146408591 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878031640 con 0x7fe8900cf9a0 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.130+0000 7fe88ea00640 1 -- 172.21.15.53:0/2146408591 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8780317a0 con 0x7fe8900cf9a0 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.130+0000 7fe88ea00640 1 -- 172.21.15.53:0/2146408591 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe878031b30 con 0x7fe8900cf9a0 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 -- 172.21.15.53:0/2146408591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 msgr2=0x7fe89015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 --2- 172.21.15.53:0/2146408591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89015dca0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fe878009870 tx=0x7fe87802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 -- 172.21.15.53:0/2146408591 shutdown_connections 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 --2- 172.21.15.53:0/2146408591 >> v2:172.21.15.146:3300/0 conn(0x7fe89015e1e0 0x7fe8900702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 --2- 172.21.15.53:0/2146408591 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89015dca0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 -- 172.21.15.53:0/2146408591 >> 172.21.15.53:0/2146408591 conn(0x7fe89006a800 msgr2=0x7fe89006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 -- 172.21.15.53:0/2146408591 shutdown_connections 2024-01-16T15:15:14.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.131+0000 7fe896e00640 1 -- 172.21.15.53:0/2146408591 wait complete. 2024-01-16T15:15:14.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.134+0000 7fe896e00640 1 Processor -- start 2024-01-16T15:15:14.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.134+0000 7fe896e00640 1 -- start start 2024-01-16T15:15:14.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.135+0000 7fe896e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89013e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.135+0000 7fe896e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 0x7fe890141030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.135+0000 7fe896e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe89015e1e0 0x7fe890141570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.135+0000 7fe896e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900af980 con 0x7fe890143ae0 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.135+0000 7fe896e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900aed70 con 0x7fe89015e1e0 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.135+0000 7fe896e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8900af050 con 0x7fe8900cf9a0 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 0x7fe890141030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 0x7fe890141030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49940/0 (socket says 172.21.15.53:49940) 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 -- 172.21.15.53:0/829503759 learned_addr learned my addr 172.21.15.53:0/829503759 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:14.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88fe00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89013e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe894e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe89015e1e0 0x7fe890141570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 -- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 msgr2=0x7fe89013e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89013e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 -- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe89015e1e0 msgr2=0x7fe890141570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe89015e1e0 0x7fe890141570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.136+0000 7fe88f400640 1 -- 172.21.15.53:0/829503759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8780094d0 con 0x7fe890143ae0 2024-01-16T15:15:14.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.137+0000 7fe88f400640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 0x7fe890141030 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fe878009870 tx=0x7fe8780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.137+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87803a930 con 0x7fe890143ae0 2024-01-16T15:15:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.137+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe87803aa90 con 0x7fe890143ae0 2024-01-16T15:15:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.137+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe87803ae40 con 0x7fe890143ae0 2024-01-16T15:15:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.137+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8900af330 con 0x7fe890143ae0 2024-01-16T15:15:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.138+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe89013edb0 con 0x7fe890143ae0 2024-01-16T15:15:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.139+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8900675d0 con 0x7fe890143ae0 2024-01-16T15:15:14.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.140+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fe87802f050 con 0x7fe890143ae0 2024-01-16T15:15:14.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.141+0000 7fe887e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe86806c160 0x7fe86806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.141+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fe8780aa860 con 0x7fe890143ae0 2024-01-16T15:15:14.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.142+0000 7fe88fe00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe86806c160 0x7fe86806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.143+0000 7fe88fe00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe86806c160 0x7fe86806e620 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe880004ea0 tx=0x7fe880009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.146+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe878076e40 con 0x7fe890143ae0 2024-01-16T15:15:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.340+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8900afac0 con 0x7fe890143ae0 2024-01-16T15:15:14.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.340+0000 7fe887e00640 1 -- 172.21.15.53:0/829503759 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe87807a460 con 0x7fe890143ae0 2024-01-16T15:15:14.325 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:15:14.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe86806c160 msgr2=0x7fe86806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe86806c160 0x7fe86806e620 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe880004ea0 tx=0x7fe880009040 comp rx=0 tx=0).stop 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 msgr2=0x7fe890141030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 0x7fe890141030 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7fe878009870 tx=0x7fe8780046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 shutdown_connections 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe86806c160 0x7fe86806e620 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe89015e1e0 0x7fe890141570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe890143ae0 0x7fe890141030 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 --2- 172.21.15.53:0/829503759 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8900cf9a0 0x7fe89013e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 >> 172.21.15.53:0/829503759 conn(0x7fe89006a800 msgr2=0x7fe89006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 shutdown_connections 2024-01-16T15:15:14.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.343+0000 7fe896e00640 1 -- 172.21.15.53:0/829503759 wait complete. 2024-01-16T15:15:14.348 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:15:14.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.590+0000 7f3055a00640 1 Processor -- start 2024-01-16T15:15:14.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.590+0000 7f3055a00640 1 -- start start 2024-01-16T15:15:14.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3050160920 0x7f30500b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f30500cf9f0 0x7f30500cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050161010 0x7f30501614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30500d03e0 con 0x7f30500cf9f0 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30500d0a10 con 0x7f3050160920 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30500cf080 con 0x7f3050161010 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3047200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f30500cf9f0 0x7f30500cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3047200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f30500cf9f0 0x7f30500cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49952/0 (socket says 172.21.15.53:49952) 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3047200640 1 -- 172.21.15.53:0/1091185369 learned_addr learned my addr 172.21.15.53:0/1091185369 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f304fe00640 1 --2- 172.21.15.53:0/1091185369 >> v2:172.21.15.146:3300/0 conn(0x7f3050160920 0x7f30500b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.592+0000 7f3055000640 1 --2- 172.21.15.53:0/1091185369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050161010 0x7f30501614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.593+0000 7f3047200640 1 -- 172.21.15.53:0/1091185369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050161010 msgr2=0x7f30501614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.593+0000 7f3047200640 1 --2- 172.21.15.53:0/1091185369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050161010 0x7f30501614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.593+0000 7f3047200640 1 -- 172.21.15.53:0/1091185369 >> v2:172.21.15.146:3300/0 conn(0x7f3050160920 msgr2=0x7f30500b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:14.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.593+0000 7f3047200640 1 --2- 172.21.15.53:0/1091185369 >> v2:172.21.15.146:3300/0 conn(0x7f3050160920 0x7f30500b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.593+0000 7f3047200640 1 -- 172.21.15.53:0/1091185369 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30500c6be0 con 0x7f30500cf9f0 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.594+0000 7f3047200640 1 --2- 172.21.15.53:0/1091185369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30500cf9f0 0x7f30500cfea0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f305008c060 tx=0x7f303802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1238859030e4b54 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.594+0000 7f304f400640 1 -- 172.21.15.53:0/1091185369 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038031870 con 0x7f30500cf9f0 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.594+0000 7f304f400640 1 -- 172.21.15.53:0/1091185369 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30380319d0 con 0x7f30500cf9f0 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.594+0000 7f304f400640 1 -- 172.21.15.53:0/1091185369 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3038031d60 con 0x7f30500cf9f0 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 -- 172.21.15.53:0/1091185369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30500cf9f0 msgr2=0x7f30500cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 --2- 172.21.15.53:0/1091185369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30500cf9f0 0x7f30500cfea0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7f305008c060 tx=0x7f303802b910 comp rx=0 tx=0).stop 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 -- 172.21.15.53:0/1091185369 shutdown_connections 2024-01-16T15:15:14.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 --2- 172.21.15.53:0/1091185369 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050161010 0x7f30501614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 --2- 172.21.15.53:0/1091185369 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30500cf9f0 0x7f30500cfea0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 --2- 172.21.15.53:0/1091185369 >> v2:172.21.15.146:3300/0 conn(0x7f3050160920 0x7f30500b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 -- 172.21.15.53:0/1091185369 >> 172.21.15.53:0/1091185369 conn(0x7f30500d3090 msgr2=0x7f30500d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 -- 172.21.15.53:0/1091185369 shutdown_connections 2024-01-16T15:15:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.595+0000 7f3055a00640 1 -- 172.21.15.53:0/1091185369 wait complete. 2024-01-16T15:15:14.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.597+0000 7f3055a00640 1 Processor -- start 2024-01-16T15:15:14.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.597+0000 7f3055a00640 1 -- start start 2024-01-16T15:15:14.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.598+0000 7f3055a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30500cf9f0 0x7f30501f24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.598+0000 7f3055a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050160920 0x7f30500ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.598+0000 7f3055a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 0x7f30500cbc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.598+0000 7f3055a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500b5c50 con 0x7f3050161010 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.598+0000 7f3055a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30500cabe0 con 0x7f30500cf9f0 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.598+0000 7f3055a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30500caec0 con 0x7f3050160920 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 0x7f30500cbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 0x7f30500cbc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49956/0 (socket says 172.21.15.53:49956) 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 -- 172.21.15.53:0/1635734012 learned_addr learned my addr 172.21.15.53:0/1635734012 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3047200640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050160920 0x7f30500ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f304fe00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30500cf9f0 0x7f30501f24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 -- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050160920 msgr2=0x7f30500ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050160920 0x7f30500ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 -- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30500cf9f0 msgr2=0x7f30501f24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30500cf9f0 0x7f30501f24f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:14.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3055000640 1 -- 172.21.15.53:0/1635734012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30380094d0 con 0x7f3050161010 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.599+0000 7f3047200640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050160920 0x7f30500ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.600+0000 7f3055000640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 0x7f30500cbc50 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f3048007e40 tx=0x7f304800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.600+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048014620 con 0x7f3050161010 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.600+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304800ee90 con 0x7f3050161010 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.600+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480149e0 con 0x7f3050161010 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.600+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30500cb1a0 con 0x7f3050161010 2024-01-16T15:15:14.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.600+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3050151580 con 0x7f3050161010 2024-01-16T15:15:14.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.601+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3050067450 con 0x7f3050161010 2024-01-16T15:15:14.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.602+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3048014b40 con 0x7f3050161010 2024-01-16T15:15:14.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.603+0000 7f304cc00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f302806c160 0x7f302806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:14.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.603+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f30480893b0 con 0x7f3050161010 2024-01-16T15:15:14.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.604+0000 7f304fe00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f302806c160 0x7f302806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:14.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.605+0000 7f304fe00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f302806c160 0x7f302806e620 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3040006fd0 tx=0x7f3040008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:14.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.608+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3048055990 con 0x7f3050161010 2024-01-16T15:15:14.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:14.874+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f30500cba30 con 0x7f3050161010 2024-01-16T15:15:15.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.427+0000 7f304cc00640 1 -- 172.21.15.53:0/1635734012 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v666) v1 ==== 158+0+0 (secure 0 0 0) 0x7f3048058fb0 con 0x7f3050161010 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.430+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f302806c160 msgr2=0x7f302806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.430+0000 7f3055a00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f302806c160 0x7f302806e620 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f3040006fd0 tx=0x7f3040008040 comp rx=0 tx=0).stop 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.430+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 msgr2=0x7f30500cbc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.430+0000 7f3055a00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 0x7f30500cbc50 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f3048007e40 tx=0x7f304800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 shutdown_connections 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f302806c160 0x7f302806e620 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3050161010 0x7f30500cbc50 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3050160920 0x7f30500ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 --2- 172.21.15.53:0/1635734012 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30500cf9f0 0x7f30501f24f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 >> 172.21.15.53:0/1635734012 conn(0x7f30500d3090 msgr2=0x7f305006c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:15.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 shutdown_connections 2024-01-16T15:15:15.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.431+0000 7f3055a00640 1 -- 172.21.15.53:0/1635734012 wait complete. 2024-01-16T15:15:15.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:15.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.684+0000 7fc7f5800640 1 Processor -- start 2024-01-16T15:15:15.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.684+0000 7fc7f5800640 1 -- start start 2024-01-16T15:15:15.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7f5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 0x7fc7f00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7f5800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f00d0a10 con 0x7fc7f00cf9f0 2024-01-16T15:15:15.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7f5800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7f0161e30 con 0x7fc7f00cfe70 2024-01-16T15:15:15.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7f5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f00d1f50 con 0x7fc7f0160920 2024-01-16T15:15:15.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7ef400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc7f00cf9f0 0x7fc7f0161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51566/0 (socket says 172.21.15.53:51566) 2024-01-16T15:15:15.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7ef400640 1 -- 172.21.15.53:0/1068501285 learned_addr learned my addr 172.21.15.53:0/1068501285 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:15.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.686+0000 7fc7efe00640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 0x7fc7f00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548543584 0 0) 0x7fc7f00d0a10 con 0x7fc7f00cf9f0 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d0003520 con 0x7fc7f00cf9f0 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3040404776 0 0) 0x7fc7d0003520 con 0x7fc7f00cf9f0 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 msgr2=0x7fc7f00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 0x7fc7f00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 >> v1:172.21.15.146:6789/0 conn(0x7fc7f00cfe70 legacy=0x7fc7f0161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f00cf080 con 0x7fc7f00cf9f0 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fc7d8001670 con 0x7fc7f00cf9f0 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 >> v1:172.21.15.53:6789/0 conn(0x7fc7f00cf9f0 legacy=0x7fc7f0161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 0x7fc7d00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d0003f70 0x7fc7d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0004d00 0x7fc7d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f00d1f50 con 0x7fc7d0003f70 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d000bac0 con 0x7fc7d0004d00 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.687+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d000bc80 con 0x7fc7f0160920 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7ef400640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d0003f70 0x7fc7d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7f4e00640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0004d00 0x7fc7d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7ef400640 1 -- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 msgr2=0x7fc7d00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7ef400640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0160920 0x7fc7d00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7ef400640 1 -- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0004d00 msgr2=0x7fc7d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7ef400640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0004d00 0x7fc7d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.688+0000 7fc7ef400640 1 -- 172.21.15.53:0/1068501285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d0003520 con 0x7fc7d0003f70 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.689+0000 7fc7ef400640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d0003f70 0x7fc7d00047c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800c0e0 tx=0x7fc7d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be7ae47ff2993dd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.689+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8005ad0 con 0x7fc7d0003f70 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.689+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d8005c30 con 0x7fc7d0003f70 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.689+0000 7fc7eea00640 1 -- 172.21.15.53:0/1068501285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80072b0 con 0x7fc7d0003f70 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f5800640 1 -- 172.21.15.53:0/1068501285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d0003f70 msgr2=0x7fc7d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f5800640 1 --2- 172.21.15.53:0/1068501285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7d0003f70 0x7fc7d00047c0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fc7d800c0e0 tx=0x7fc7d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f4e00640 1 -- 172.21.15.53:0/1068501285 reap_dead start 2024-01-16T15:15:15.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f5800640 1 -- 172.21.15.53:0/1068501285 shutdown_connections 2024-01-16T15:15:15.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f5800640 1 -- 172.21.15.53:0/1068501285 >> 172.21.15.53:0/1068501285 conn(0x7fc7f00d3090 msgr2=0x7fc7f00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:15.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f5800640 1 -- 172.21.15.53:0/1068501285 shutdown_connections 2024-01-16T15:15:15.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.690+0000 7fc7f5800640 1 -- 172.21.15.53:0/1068501285 wait complete. 2024-01-16T15:15:15.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.692+0000 7fc7f5800640 1 Processor -- start 2024-01-16T15:15:15.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.692+0000 7fc7f5800640 1 -- start start 2024-01-16T15:15:15.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.693+0000 7fc7f5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0003f70 0x7fc7f01514d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.693+0000 7fc7f5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0152e70 0x7fc7f00c8940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.693+0000 7fc7f5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 0x7fc7f01f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.693+0000 7fc7f5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f00c9330 con 0x7fc7f00c8e80 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.693+0000 7fc7f5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f01f74a0 con 0x7fc7d0003f70 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.693+0000 7fc7f5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7f01520b0 con 0x7fc7f0152e70 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 0x7fc7f01f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 0x7fc7f01f6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49960/0 (socket says 172.21.15.53:49960) 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 -- 172.21.15.53:0/2592058682 learned_addr learned my addr 172.21.15.53:0/2592058682 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7efe00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0003f70 0x7fc7f01514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7ef400640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0152e70 0x7fc7f00c8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 -- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0152e70 msgr2=0x7fc7f00c8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0152e70 0x7fc7f00c8940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 -- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0003f70 msgr2=0x7fc7f01514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0003f70 0x7fc7f01514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.694+0000 7fc7f4e00640 1 -- 172.21.15.53:0/2592058682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f0152390 con 0x7fc7f00c8e80 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.695+0000 7fc7f4e00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 0x7fc7f01f6770 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fc7f006a740 tx=0x7fc7e800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.695+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e800ee90 con 0x7fc7f00c8e80 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.695+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7e8003970 con 0x7fc7f00c8e80 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.695+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8003d20 con 0x7fc7f00c8e80 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.695+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7f0152670 con 0x7fc7f00c8e80 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.695+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7f0151a10 con 0x7fc7f00c8e80 2024-01-16T15:15:15.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.696+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7f0152b90 con 0x7fc7f00c8e80 2024-01-16T15:15:15.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.697+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fc7e80050d0 con 0x7fc7f00c8e80 2024-01-16T15:15:15.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.698+0000 7fc7e7e00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c806c160 0x7fc7c806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:15.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.698+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc7e80886d0 con 0x7fc7f00c8e80 2024-01-16T15:15:15.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.699+0000 7fc7efe00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c806c160 0x7fc7c806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:15.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.700+0000 7fc7efe00640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c806c160 0x7fc7c806e620 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009ae0 tx=0x7fc7e00094a0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:15.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.703+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc7e8054cb0 con 0x7fc7f00c8e80 2024-01-16T15:15:15.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.900+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc7f0067450 con 0x7fc7f00c8e80 2024-01-16T15:15:15.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.907+0000 7fc7e7e00640 1 -- 172.21.15.53:0/2592058682 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc7e80582d0 con 0x7fc7f00c8e80 2024-01-16T15:15:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c806c160 msgr2=0x7fc7c806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c806c160 0x7fc7c806e620 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc7e0009ae0 tx=0x7fc7e00094a0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 msgr2=0x7fc7f01f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:15.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 0x7fc7f01f6770 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fc7f006a740 tx=0x7fc7e800eb50 comp rx=0 tx=0).stop 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 shutdown_connections 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc7c806c160 0x7fc7c806e620 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc7f00c8e80 0x7fc7f01f6770 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7f0152e70 0x7fc7f00c8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 --2- 172.21.15.53:0/2592058682 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc7d0003f70 0x7fc7f01514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.910+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 >> 172.21.15.53:0/2592058682 conn(0x7fc7f00d3090 msgr2=0x7fc7f006c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.911+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 shutdown_connections 2024-01-16T15:15:15.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:15.911+0000 7fc7f5800640 1 -- 172.21.15.53:0/2592058682 wait complete. 2024-01-16T15:15:15.916 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:15:15.917 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:16.039 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:15:16.164 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:15:16.164 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:15:16.164 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:16.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:16.226 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:15:16.226 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:16.283 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:16.283 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:15:16.283 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:16.344 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:15:16.345 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:15:16.345 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:16.402 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:15:16.402 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:15:16.403 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.650+0000 7f6b1da00640 1 Processor -- start 2024-01-16T15:15:16.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.650+0000 7f6b1da00640 1 -- start start 2024-01-16T15:15:16.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.652+0000 7f6b1da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cd4a0 0x7f6b180b3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.652+0000 7f6b1da00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6b180cf030 0x7f6b180cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.652+0000 7f6b1da00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1815ea30 con 0x7f6b180ad760 2024-01-16T15:15:16.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.652+0000 7f6b1da00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b180cf9c0 con 0x7f6b180cf030 2024-01-16T15:15:16.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.652+0000 7f6b1da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180adc40 con 0x7f6b180cd4a0 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6b180cf030 0x7f6b180cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6b180cf030 0x7f6b180cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49966/0 (socket says 172.21.15.53:49966) 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 -- 172.21.15.53:0/3365734421 learned_addr learned my addr 172.21.15.53:0/3365734421 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b17e00640 1 --2- 172.21.15.53:0/3365734421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cd4a0 0x7f6b180b3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 -- 172.21.15.53:0/3365734421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cd4a0 msgr2=0x7f6b180b3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 --2- 172.21.15.53:0/3365734421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cd4a0 0x7f6b180b3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 -- 172.21.15.53:0/3365734421 >> v1:172.21.15.146:6789/0 conn(0x7f6b180ad760 legacy=0x7f6b1815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.653+0000 7f6b1d000640 1 -- 172.21.15.53:0/3365734421 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b180adf20 con 0x7f6b180cf030 2024-01-16T15:15:16.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.654+0000 7f6b1d000640 1 --2- 172.21.15.53:0/3365734421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cf030 0x7f6b180cf480 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009e50 tx=0x7f6b1002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f9c07e55b5d753 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:16.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.655+0000 7f6b17400640 1 -- 172.21.15.53:0/3365734421 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b1002ca70 con 0x7f6b180cf030 2024-01-16T15:15:16.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.655+0000 7f6b17400640 1 -- 172.21.15.53:0/3365734421 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b1002cbd0 con 0x7f6b180cf030 2024-01-16T15:15:16.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.655+0000 7f6b17400640 1 -- 172.21.15.53:0/3365734421 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b100343f0 con 0x7f6b180cf030 2024-01-16T15:15:16.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.655+0000 7f6b1da00640 1 -- 172.21.15.53:0/3365734421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cf030 msgr2=0x7f6b180cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.655+0000 7f6b1da00640 1 --2- 172.21.15.53:0/3365734421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cf030 0x7f6b180cf480 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f6b10009e50 tx=0x7f6b1002bed0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.656+0000 7f6b1da00640 1 -- 172.21.15.53:0/3365734421 shutdown_connections 2024-01-16T15:15:16.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.656+0000 7f6b1da00640 1 --2- 172.21.15.53:0/3365734421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cf030 0x7f6b180cf480 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.656+0000 7f6b1da00640 1 --2- 172.21.15.53:0/3365734421 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cd4a0 0x7f6b180b3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.656+0000 7f6b1da00640 1 -- 172.21.15.53:0/3365734421 >> 172.21.15.53:0/3365734421 conn(0x7f6b1806a800 msgr2=0x7f6b1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:16.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.656+0000 7f6b1da00640 1 -- 172.21.15.53:0/3365734421 shutdown_connections 2024-01-16T15:15:16.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.656+0000 7f6b1da00640 1 -- 172.21.15.53:0/3365734421 wait complete. 2024-01-16T15:15:16.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.657+0000 7f6b1da00640 1 Processor -- start 2024-01-16T15:15:16.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.658+0000 7f6b1da00640 1 -- start start 2024-01-16T15:15:16.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1da00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b180ad760 0x7f6b181f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1da00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 0x7f6b180cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1da00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cf030 0x7f6b1809c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1da00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180b0f40 con 0x7f6b180cd4a0 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1da00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b181f35d0 con 0x7f6b180ad760 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1da00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b180b00e0 con 0x7f6b180cf030 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 0x7f6b180cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 0x7f6b180cc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49980/0 (socket says 172.21.15.53:49980) 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 -- 172.21.15.53:0/2894181646 learned_addr learned my addr 172.21.15.53:0/2894181646 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b17e00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b180ad760 0x7f6b181f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1d000640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cf030 0x7f6b1809c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 -- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cf030 msgr2=0x7f6b1809c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cf030 0x7f6b1809c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 -- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b180ad760 msgr2=0x7f6b181f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b180ad760 0x7f6b181f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b0fe00640 1 -- 172.21.15.53:0/2894181646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b10009b30 con 0x7f6b180cd4a0 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.659+0000 7f6b1d000640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cf030 0x7f6b1809c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.660+0000 7f6b0fe00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 0x7f6b180cc7c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c430 tx=0x7f6b0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.660+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00011790 con 0x7f6b180cd4a0 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.660+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b000118f0 con 0x7f6b180cd4a0 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.660+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b180b03c0 con 0x7f6b180cd4a0 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.660+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b00011ce0 con 0x7f6b180cd4a0 2024-01-16T15:15:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.660+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b180b0980 con 0x7f6b180cd4a0 2024-01-16T15:15:16.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.661+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b18067450 con 0x7f6b180cd4a0 2024-01-16T15:15:16.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.663+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f6b0001c460 con 0x7f6b180cd4a0 2024-01-16T15:15:16.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.664+0000 7f6b14c00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006c160 0x7f6af006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:16.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.664+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f6b000888e0 con 0x7f6b180cd4a0 2024-01-16T15:15:16.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.664+0000 7f6b17e00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006c160 0x7f6af006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:16.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.665+0000 7f6b17e00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006c160 0x7f6af006e620 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6b08006fd0 tx=0x7f6b08008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:16.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.668+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b00054ec0 con 0x7f6b180cd4a0 2024-01-16T15:15:16.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.901+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b180cd010 con 0x7f6b180cd4a0 2024-01-16T15:15:16.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.902+0000 7f6b14c00640 1 -- 172.21.15.53:0/2894181646 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f6b000584e0 con 0x7f6b180cd4a0 2024-01-16T15:15:16.887 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:16.887 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6119,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/3084871157","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":3084871157},{"type":"v1","addr":"172.21.15.146:6835","nonce":3084871157}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6742,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2008413931","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2008413931},{"type":"v1","addr":"172.21.15.53:6835","nonce":2008413931}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":6748,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/614678519","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":614678519},{"type":"v1","addr":"172.21.15.146:6837","nonce":614678519}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":50},{"gid":8394,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/2844987177","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":2844987177},{"type":"v1","addr":"172.21.15.53:6834","nonce":2844987177}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":58}],"filesystems":[]} 2024-01-16T15:15:16.887 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 58 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006c160 msgr2=0x7f6af006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006c160 0x7f6af006e620 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6b08006fd0 tx=0x7f6b08008040 comp rx=0 tx=0).stop 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 msgr2=0x7f6b180cc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 0x7f6b180cc7c0 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f6b0000c430 tx=0x7f6b0000c8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 shutdown_connections 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6af006c160 0x7f6af006e620 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b180cf030 0x7f6b1809c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b180cd4a0 0x7f6b180cc7c0 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 --2- 172.21.15.53:0/2894181646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b180ad760 0x7f6b181f3090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:16.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 >> 172.21.15.53:0/2894181646 conn(0x7f6b1806a800 msgr2=0x7f6b1815c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:16.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 shutdown_connections 2024-01-16T15:15:16.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:16.905+0000 7f6b1da00640 1 -- 172.21.15.53:0/2894181646 wait complete. 2024-01-16T15:15:16.909 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:15:16.909 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:15:16.909 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:15:16.909 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:16.910 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:15:16.910 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:15:16.910 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:15:16.910 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:16.910 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:15:16.910 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:15:16.911 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:15:16.911 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:16.911 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:15:16.911 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:15:16.911 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:15:16.911 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:15:16.930 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:15:16.930 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:15:16.931 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:15:16.931 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:15:16.932 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:15:16.932 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:15:16.940 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:15:16.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:15:16.978 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:15:16.978 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:16.984 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:15:16.985 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:16.987 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:15:16.987 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:16.994 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:15:16.994 DEBUG:teuthology.parallel:result is None 2024-01-16T15:15:16.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:15:17.143 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:15:17.145 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:15:17.151 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:15:17.152 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:15:17.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.200+0000 7f550de00640 1 Processor -- start 2024-01-16T15:15:17.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.200+0000 7f550de00640 1 -- start start 2024-01-16T15:15:17.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.201+0000 7f550de00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5508160920 0x7f55080b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.201+0000 7f550de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55080cf9f0 0x7f55080cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.201+0000 7f550de00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55080d0a10 con 0x7f5508161010 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.201+0000 7f550de00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55080d03e0 con 0x7f55080cf9f0 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.201+0000 7f550de00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55080d1f50 con 0x7f5508160920 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f5507e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55080cf9f0 0x7f55080cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5508160920 0x7f55080b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550d400640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5508161010 0x7f5508161480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:45466/0 (socket says 172.21.15.53:45466) 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f5507e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f55080cf9f0 0x7f55080cfea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50016/0 (socket says 172.21.15.53:50016) 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550d400640 1 -- 172.21.15.53:0/1460956931 learned_addr learned my addr 172.21.15.53:0/1460956931 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550ca00640 1 -- 172.21.15.53:0/1460956931 >> v2:172.21.15.53:3300/0 conn(0x7f55080cf9f0 msgr2=0x7f55080cfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550ca00640 1 --2- 172.21.15.53:0/1460956931 >> v2:172.21.15.53:3300/0 conn(0x7f55080cf9f0 0x7f55080cfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550ca00640 1 -- 172.21.15.53:0/1460956931 >> v1:172.21.15.146:6789/0 conn(0x7f5508161010 legacy=0x7f5508161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.202+0000 7f550ca00640 1 -- 172.21.15.53:0/1460956931 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080cf080 con 0x7f5508160920 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.203+0000 7f550ca00640 1 --2- 172.21.15.53:0/1460956931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5508160920 0x7f55080b3f30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f550808c060 tx=0x7f54f802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7d827c380d7f6e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.204+0000 7f5507400640 1 -- 172.21.15.53:0/1460956931 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8031830 con 0x7f5508160920 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.204+0000 7f5507400640 1 -- 172.21.15.53:0/1460956931 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54f8031990 con 0x7f5508160920 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.204+0000 7f5507400640 1 -- 172.21.15.53:0/1460956931 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f8031d20 con 0x7f5508160920 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.204+0000 7f550de00640 1 -- 172.21.15.53:0/1460956931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5508160920 msgr2=0x7f55080b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.204+0000 7f550de00640 1 --2- 172.21.15.53:0/1460956931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5508160920 0x7f55080b3f30 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f550808c060 tx=0x7f54f802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.205+0000 7f550de00640 1 -- 172.21.15.53:0/1460956931 shutdown_connections 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.205+0000 7f550de00640 1 --2- 172.21.15.53:0/1460956931 >> v2:172.21.15.53:3300/0 conn(0x7f55080cf9f0 0x7f55080cfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.205+0000 7f550de00640 1 --2- 172.21.15.53:0/1460956931 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5508160920 0x7f55080b3f30 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.205+0000 7f550de00640 1 -- 172.21.15.53:0/1460956931 >> 172.21.15.53:0/1460956931 conn(0x7f55080d3090 msgr2=0x7f55080d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:17.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.205+0000 7f550de00640 1 -- 172.21.15.53:0/1460956931 shutdown_connections 2024-01-16T15:15:17.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.205+0000 7f550de00640 1 -- 172.21.15.53:0/1460956931 wait complete. 2024-01-16T15:15:17.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.206+0000 7f550de00640 1 Processor -- start 2024-01-16T15:15:17.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.206+0000 7f550de00640 1 -- start start 2024-01-16T15:15:17.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.207+0000 7f550de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55080cf9f0 0x7f550814c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.207+0000 7f550de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5508160920 0x7f550815c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.207+0000 7f550de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 0x7f550815cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.207+0000 7f550de00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550814cae0 con 0x7f5508161010 2024-01-16T15:15:17.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.207+0000 7f550de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080b6630 con 0x7f55080cf9f0 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.207+0000 7f550de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55080b6910 con 0x7f5508160920 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 0x7f550815cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 0x7f550815cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50032/0 (socket says 172.21.15.53:50032) 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 -- 172.21.15.53:0/2158798112 learned_addr learned my addr 172.21.15.53:0/2158798112 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550ca00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55080cf9f0 0x7f550814c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f5507e00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5508160920 0x7f550815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 -- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5508160920 msgr2=0x7f550815c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5508160920 0x7f550815c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 -- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55080cf9f0 msgr2=0x7f550814c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55080cf9f0 0x7f550814c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.208+0000 7f550d400640 1 -- 172.21.15.53:0/2158798112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f8009400 con 0x7f5508161010 2024-01-16T15:15:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.209+0000 7f550d400640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 0x7f550815cbb0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f550000cbc0 tx=0x7f550000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.209+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000ad40 con 0x7f5508161010 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.209+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5500012400 con 0x7f5508161010 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.209+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55080b6bf0 con 0x7f5508161010 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.209+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550000aea0 con 0x7f5508161010 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.209+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55080cca60 con 0x7f5508161010 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.210+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55080b7110 con 0x7f5508161010 2024-01-16T15:15:17.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.211+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f5500007070 con 0x7f5508161010 2024-01-16T15:15:17.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.212+0000 7f5504c00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f54e006c160 0x7f54e006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.212+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5500088850 con 0x7f5508161010 2024-01-16T15:15:17.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.213+0000 7f550ca00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f54e006c160 0x7f54e006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.214+0000 7f550ca00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f54e006c160 0x7f54e006e620 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f54f8036000 tx=0x7f54f803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:17.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.217+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55080b7110 con 0x7f5508161010 2024-01-16T15:15:17.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.425+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f55080b7110 con 0x7f5508161010 2024-01-16T15:15:17.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.452+0000 7f5504c00640 1 -- 172.21.15.53:0/2158798112 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v93) v1 ==== 97+0+0 (secure 0 0 0) 0x7f55080b7110 con 0x7f5508161010 2024-01-16T15:15:17.436 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:15:17.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f54e006c160 msgr2=0x7f54e006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f54e006c160 0x7f54e006e620 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f54f8036000 tx=0x7f54f803c040 comp rx=0 tx=0).stop 2024-01-16T15:15:17.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 msgr2=0x7f550815cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 0x7f550815cbb0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f550000cbc0 tx=0x7f550000a430 comp rx=0 tx=0).stop 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 shutdown_connections 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f54e006c160 0x7f54e006e620 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5508161010 0x7f550815cbb0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5508160920 0x7f550815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 --2- 172.21.15.53:0/2158798112 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55080cf9f0 0x7f550814c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 >> 172.21.15.53:0/2158798112 conn(0x7f55080d3090 msgr2=0x7f550806c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 shutdown_connections 2024-01-16T15:15:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.455+0000 7f550de00640 1 -- 172.21.15.53:0/2158798112 wait complete. 2024-01-16T15:15:17.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:15:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.695+0000 7fe902a00640 1 Processor -- start 2024-01-16T15:15:17.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.695+0000 7fe902a00640 1 -- start start 2024-01-16T15:15:17.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.697+0000 7fe902a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe8fc13cda0 0x7fe8fc15da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.697+0000 7fe902a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 0x7fe8fc070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.697+0000 7fe902a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8fc0700f0 con 0x7fe8fc15dfa0 2024-01-16T15:15:17.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.697+0000 7fe902a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8fc0ce530 con 0x7fe8fc13cda0 2024-01-16T15:15:17.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.697+0000 7fe902a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc15e3e0 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.697+0000 7fe902000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 0x7fe8fc070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.698+0000 7fe902000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 0x7fe8fc070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:43630/0 (socket says 172.21.15.53:43630) 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.698+0000 7fe902000640 1 -- 172.21.15.53:0/4101118211 learned_addr learned my addr 172.21.15.53:0/4101118211 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.698+0000 7fe901600640 1 --2- 172.21.15.53:0/4101118211 >> v2:172.21.15.146:3300/0 conn(0x7fe8fc13cda0 0x7fe8fc15da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.698+0000 7fe902000640 1 -- 172.21.15.53:0/4101118211 >> v2:172.21.15.146:3300/0 conn(0x7fe8fc13cda0 msgr2=0x7fe8fc15da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.698+0000 7fe902000640 1 --2- 172.21.15.53:0/4101118211 >> v2:172.21.15.146:3300/0 conn(0x7fe8fc13cda0 0x7fe8fc15da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.699+0000 7fe902000640 1 -- 172.21.15.53:0/4101118211 >> v1:172.21.15.146:6789/0 conn(0x7fe8fc15dfa0 legacy=0x7fe8fc0cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.699+0000 7fe902000640 1 -- 172.21.15.53:0/4101118211 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8fc0ce720 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.700+0000 7fe902000640 1 --2- 172.21.15.53:0/4101118211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 0x7fe8fc070b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400d150 tx=0x7fe8f402e0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7155d953b140b3d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.701+0000 7fe8fbe00640 1 -- 172.21.15.53:0/4101118211 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f402ea80 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.701+0000 7fe8fbe00640 1 -- 172.21.15.53:0/4101118211 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f402ebe0 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.701+0000 7fe8fbe00640 1 -- 172.21.15.53:0/4101118211 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8f4009600 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.701+0000 7fe902a00640 1 -- 172.21.15.53:0/4101118211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 msgr2=0x7fe8fc070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.701+0000 7fe902a00640 1 --2- 172.21.15.53:0/4101118211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 0x7fe8fc070b40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe8f400d150 tx=0x7fe8f402e0e0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.702+0000 7fe902a00640 1 -- 172.21.15.53:0/4101118211 shutdown_connections 2024-01-16T15:15:17.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.702+0000 7fe902a00640 1 --2- 172.21.15.53:0/4101118211 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc0ce0c0 0x7fe8fc070b40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.702+0000 7fe902a00640 1 --2- 172.21.15.53:0/4101118211 >> v2:172.21.15.146:3300/0 conn(0x7fe8fc13cda0 0x7fe8fc15da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.702+0000 7fe902a00640 1 -- 172.21.15.53:0/4101118211 >> 172.21.15.53:0/4101118211 conn(0x7fe8fc06a3e0 msgr2=0x7fe8fc06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.702+0000 7fe902a00640 1 -- 172.21.15.53:0/4101118211 shutdown_connections 2024-01-16T15:15:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.702+0000 7fe902a00640 1 -- 172.21.15.53:0/4101118211 wait complete. 2024-01-16T15:15:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.703+0000 7fe902a00640 1 Processor -- start 2024-01-16T15:15:17.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 -- start start 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 0x7fe8fc157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8fc13cda0 0x7fe8fc150710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc15dfa0 0x7fe8fc153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc156cf0 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc1595f0 con 0x7fe8fc13cda0 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.704+0000 7fe902a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8fc1598d0 con 0x7fe8fc15dfa0 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.705+0000 7fe901600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 0x7fe8fc157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.705+0000 7fe901600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 0x7fe8fc157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50034/0 (socket says 172.21.15.53:50034) 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.705+0000 7fe901600640 1 -- 172.21.15.53:0/1534952799 learned_addr learned my addr 172.21.15.53:0/1534952799 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.705+0000 7fe902000640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc15dfa0 0x7fe8fc153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.705+0000 7fe900c00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8fc13cda0 0x7fe8fc150710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.705+0000 7fe901600640 1 -- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc15dfa0 msgr2=0x7fe8fc153330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe901600640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc15dfa0 0x7fe8fc153330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe901600640 1 -- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8fc13cda0 msgr2=0x7fe8fc150710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe901600640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8fc13cda0 0x7fe8fc150710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe901600640 1 -- 172.21.15.53:0/1534952799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8f400bd40 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe900c00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8fc13cda0 0x7fe8fc150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:17.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe902000640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc15dfa0 0x7fe8fc153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe901600640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 0x7fe8fc157110 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00e9b0 tx=0x7fe8ec00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec009670 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8ec00c9a0 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8fc156630 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8ec00cd50 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.706+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8fc150c50 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.707+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8fc156f60 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.710+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fe8ec013470 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.711+0000 7fe8f9600640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8d406c160 0x7fe8d406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:17.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.711+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fe8ec088900 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.711+0000 7fe900c00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8d406c160 0x7fe8d406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:17.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.712+0000 7fe900c00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8d406c160 0x7fe8d406e620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4007fd0 tx=0x7fe8e4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:17.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.715+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe8ec0551c0 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.984+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fe8fc067450 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.985+0000 7fe8f9600640 1 -- 172.21.15.53:0/1534952799 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fe8ec0587e0 con 0x7fe8fc0ce0c0 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.970 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.971 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.972 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.973 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.974 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.975 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.976 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.977 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.978 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:15:17.979 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:15:17.980 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.981 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow r", 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:15:17.982 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:15:17.983 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.984 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.985 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:15:17.986 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:15:17.987 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:15:17.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8d406c160 msgr2=0x7fe8d406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8d406c160 0x7fe8d406e620 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fe8e4007fd0 tx=0x7fe8e4009040 comp rx=0 tx=0).stop 2024-01-16T15:15:17.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 msgr2=0x7fe8fc157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:17.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 0x7fe8fc157110 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec00e9b0 tx=0x7fe8ec00ee70 comp rx=0 tx=0).stop 2024-01-16T15:15:17.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 shutdown_connections 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe8d406c160 0x7fe8d406e620 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe8fc15dfa0 0x7fe8fc153330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe8fc13cda0 0x7fe8fc150710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 --2- 172.21.15.53:0/1534952799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe8fc0ce0c0 0x7fe8fc157110 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 >> 172.21.15.53:0/1534952799 conn(0x7fe8fc06a3e0 msgr2=0x7fe8fc06c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.987+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 shutdown_connections 2024-01-16T15:15:17.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:17.988+0000 7fe902a00640 1 -- 172.21.15.53:0/1534952799 wait complete. 2024-01-16T15:15:17.991 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:18.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.214+0000 7f99a5800640 1 Processor -- start 2024-01-16T15:15:18.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.214+0000 7f99a5800640 1 -- start start 2024-01-16T15:15:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a5800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f99a0143c40 0x7f99a0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00717d0 0x7f99a00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a5800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99a0070e50 con 0x7f99a00cca40 2024-01-16T15:15:18.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a5800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99a00723c0 con 0x7f99a0143c40 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00cce70 con 0x7f99a00717d0 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a4e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f99a00cca40 0x7f99a015eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51578/0 (socket says 172.21.15.53:51578) 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.216+0000 7f99a4e00640 1 -- 172.21.15.53:0/1953525802 learned_addr learned my addr 172.21.15.53:0/1953525802 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.217+0000 7f999f400640 1 --2- 172.21.15.53:0/1953525802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00717d0 0x7f99a00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.217+0000 7f999fe00640 1 --2- 172.21.15.53:0/1953525802 >> v2:172.21.15.146:3300/0 conn(0x7f99a0143c40 0x7f99a0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.217+0000 7f999ea00640 1 -- 172.21.15.53:0/1953525802 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998554376 0 0) 0x7f99a0070e50 con 0x7f99a00cca40 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.217+0000 7f999ea00640 1 -- 172.21.15.53:0/1953525802 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9980003520 con 0x7f99a00cca40 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.217+0000 7f999f400640 1 -- 172.21.15.53:0/1953525802 >> v2:172.21.15.146:3300/0 conn(0x7f99a0143c40 msgr2=0x7f99a0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.217+0000 7f999f400640 1 --2- 172.21.15.53:0/1953525802 >> v2:172.21.15.146:3300/0 conn(0x7f99a0143c40 0x7f99a0071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.218+0000 7f999f400640 1 -- 172.21.15.53:0/1953525802 >> v1:172.21.15.53:6789/0 conn(0x7f99a00cca40 legacy=0x7f99a015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.218+0000 7f999ea00640 1 -- 172.21.15.53:0/1953525802 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4277327914 0 0) 0x7f9980003520 con 0x7f99a00cca40 2024-01-16T15:15:18.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.218+0000 7f999f400640 1 -- 172.21.15.53:0/1953525802 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a006e170 con 0x7f99a00717d0 2024-01-16T15:15:18.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.219+0000 7f999f400640 1 --2- 172.21.15.53:0/1953525802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00717d0 0x7f99a00cc500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f99880097a0 tx=0x7f998802b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdf0cba39fd2db2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.219+0000 7f999ea00640 1 -- 172.21.15.53:0/1953525802 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99980019f0 con 0x7f99a00717d0 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.219+0000 7f999ea00640 1 -- 172.21.15.53:0/1953525802 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f998802e050 con 0x7f99a00717d0 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.219+0000 7f999ea00640 1 -- 172.21.15.53:0/1953525802 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988030710 con 0x7f99a00717d0 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 -- 172.21.15.53:0/1953525802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00717d0 msgr2=0x7f99a00cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 --2- 172.21.15.53:0/1953525802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00717d0 0x7f99a00cc500 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f99880097a0 tx=0x7f998802b710 comp rx=0 tx=0).stop 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 -- 172.21.15.53:0/1953525802 shutdown_connections 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 --2- 172.21.15.53:0/1953525802 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a00717d0 0x7f99a00cc500 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 --2- 172.21.15.53:0/1953525802 >> v2:172.21.15.146:3300/0 conn(0x7f99a0143c40 0x7f99a0071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 -- 172.21.15.53:0/1953525802 >> 172.21.15.53:0/1953525802 conn(0x7f99a006a800 msgr2=0x7f99a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 -- 172.21.15.53:0/1953525802 shutdown_connections 2024-01-16T15:15:18.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.220+0000 7f99a5800640 1 -- 172.21.15.53:0/1953525802 wait complete. 2024-01-16T15:15:18.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.222+0000 7f99a5800640 1 Processor -- start 2024-01-16T15:15:18.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.222+0000 7f99a5800640 1 -- start start 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f99a5800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00717d0 0x7f99a015a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f99a5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 0x7f99a0138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f99a5800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a0143c40 0x7f99a00b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f99a5800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a015ada0 con 0x7f99a00cca40 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f99a5800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00b3910 con 0x7f99a00717d0 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f99a5800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99a00b2b10 con 0x7f99a0143c40 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f999f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 0x7f99a0138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.223+0000 7f999f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 0x7f99a0138350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50046/0 (socket says 172.21.15.53:50046) 2024-01-16T15:15:18.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 -- 172.21.15.53:0/3752262609 learned_addr learned my addr 172.21.15.53:0/3752262609 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999fe00640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00717d0 0x7f99a015a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f99a4e00640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a0143c40 0x7f99a00b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 -- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a0143c40 msgr2=0x7f99a00b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a0143c40 0x7f99a00b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 -- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00717d0 msgr2=0x7f99a015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00717d0 0x7f99a015a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 -- 172.21.15.53:0/3752262609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9988009400 con 0x7f99a00cca40 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.224+0000 7f999f400640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 0x7f99a0138350 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f99880098c0 tx=0x7f99880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.225+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988003ea0 con 0x7f99a00cca40 2024-01-16T15:15:18.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.225+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99880044c0 con 0x7f99a00cca40 2024-01-16T15:15:18.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.225+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9988030750 con 0x7f99a00cca40 2024-01-16T15:15:18.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.225+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99a0138890 con 0x7f99a00cca40 2024-01-16T15:15:18.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.225+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99a00b24c0 con 0x7f99a00cca40 2024-01-16T15:15:18.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.226+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99a0067450 con 0x7f99a00cca40 2024-01-16T15:15:18.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.227+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f9988039070 con 0x7f99a00cca40 2024-01-16T15:15:18.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.228+0000 7f9997e00640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f997806c190 0x7f997806e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.228+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f99880aa0d0 con 0x7f99a00cca40 2024-01-16T15:15:18.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.228+0000 7f999fe00640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f997806c190 0x7f997806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.229+0000 7f999fe00640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f997806c190 0x7f997806e650 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9990009a30 tx=0x7f9990008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.232+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f99880766b0 con 0x7f99a00cca40 2024-01-16T15:15:18.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.427+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99a00ceb30 con 0x7f99a00cca40 2024-01-16T15:15:18.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.429+0000 7f9997e00640 1 -- 172.21.15.53:0/3752262609 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+12865 (secure 0 0 0) 0x7f9988040090 con 0x7f99a00cca40 2024-01-16T15:15:18.414 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:18.414 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":95,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:17.469572+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.431+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f997806c190 msgr2=0x7f997806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.431+0000 7f99a5800640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f997806c190 0x7f997806e650 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f9990009a30 tx=0x7f9990008040 comp rx=0 tx=0).stop 2024-01-16T15:15:18.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.431+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 msgr2=0x7f99a0138350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.431+0000 7f99a5800640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 0x7f99a0138350 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f99880098c0 tx=0x7f99880046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 shutdown_connections 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f997806c190 0x7f997806e650 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99a0143c40 0x7f99a00b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99a00cca40 0x7f99a0138350 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 --2- 172.21.15.53:0/3752262609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99a00717d0 0x7f99a015a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 >> 172.21.15.53:0/3752262609 conn(0x7f99a006a800 msgr2=0x7f99a013f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 shutdown_connections 2024-01-16T15:15:18.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.432+0000 7f99a5800640 1 -- 172.21.15.53:0/3752262609 wait complete. 2024-01-16T15:15:18.435 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:15:18.436 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:15:18.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.668+0000 7f7b49000640 1 Processor -- start 2024-01-16T15:15:18.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.668+0000 7f7b49000640 1 -- start start 2024-01-16T15:15:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.670+0000 7f7b49000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 0x7f7b44070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.670+0000 7f7b49000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f7b4415bf10 0x7f7b440cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.670+0000 7f7b49000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4415f4e0 con 0x7f7b440cc980 2024-01-16T15:15:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.670+0000 7f7b49000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4415f970 con 0x7f7b4415bf10 2024-01-16T15:15:18.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.670+0000 7f7b49000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b440cce10 con 0x7f7b440adf20 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.670+0000 7f7b43e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7b440cc980 0x7f7b44143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51582/0 (socket says 172.21.15.53:51582) 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b43e00640 1 -- 172.21.15.53:0/1899843740 learned_addr learned my addr 172.21.15.53:0/1899843740 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b43400640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 0x7f7b44070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42a00640 1 --2- 172.21.15.53:0/1899843740 >> v2:172.21.15.146:3300/0 conn(0x7f7b4415bf10 0x7f7b440cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779993668 0 0) 0x7f7b4415f4e0 con 0x7f7b440cc980 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b24003520 con 0x7f7b440cc980 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3182702479 0 0) 0x7f7b24003520 con 0x7f7b440cc980 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 msgr2=0x7f7b44070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 0x7f7b44070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 >> v2:172.21.15.146:3300/0 conn(0x7f7b4415bf10 msgr2=0x7f7b440cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 --2- 172.21.15.53:0/1899843740 >> v2:172.21.15.146:3300/0 conn(0x7f7b4415bf10 0x7f7b440cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44071130 con 0x7f7b440cc980 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7b3c0019f0 con 0x7f7b440cc980 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.671+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 >> v1:172.21.15.53:6789/0 conn(0x7f7b440cc980 legacy=0x7f7b44143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42000640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b24003970 0x7f7b24003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42000640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b240047c0 0x7f7b24004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42000640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 0x7f7b24005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4415f970 con 0x7f7b240047c0 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b440cce10 con 0x7f7b24003970 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b2400b980 con 0x7f7b440adf20 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42a00640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b240047c0 0x7f7b24004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42a00640 1 -- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 msgr2=0x7f7b24005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42a00640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440adf20 0x7f7b24005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42a00640 1 -- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b24003970 msgr2=0x7f7b24003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42a00640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b24003970 0x7f7b24003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.672+0000 7f7b42a00640 1 -- 172.21.15.53:0/1899843740 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b24003520 con 0x7f7b240047c0 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.673+0000 7f7b42a00640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b240047c0 0x7f7b24004c40 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009af0 tx=0x7f7b240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e00670ce0dbde5f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.674+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c02c710 con 0x7f7b240047c0 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.674+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2c02c870 con 0x7f7b240047c0 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.674+0000 7f7b42000640 1 -- 172.21.15.53:0/1899843740 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b2c02cc00 con 0x7f7b240047c0 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.674+0000 7f7b49000640 1 -- 172.21.15.53:0/1899843740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b240047c0 msgr2=0x7f7b24004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.674+0000 7f7b49000640 1 --2- 172.21.15.53:0/1899843740 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b240047c0 0x7f7b24004c40 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f7b2c009af0 tx=0x7f7b240057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.674+0000 7f7b43e00640 1 -- 172.21.15.53:0/1899843740 reap_dead start 2024-01-16T15:15:18.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.675+0000 7f7b49000640 1 -- 172.21.15.53:0/1899843740 shutdown_connections 2024-01-16T15:15:18.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.675+0000 7f7b49000640 1 -- 172.21.15.53:0/1899843740 >> 172.21.15.53:0/1899843740 conn(0x7f7b4406a7d0 msgr2=0x7f7b4406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.675+0000 7f7b49000640 1 -- 172.21.15.53:0/1899843740 shutdown_connections 2024-01-16T15:15:18.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.675+0000 7f7b49000640 1 -- 172.21.15.53:0/1899843740 wait complete. 2024-01-16T15:15:18.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.677+0000 7f7b49000640 1 Processor -- start 2024-01-16T15:15:18.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.677+0000 7f7b49000640 1 -- start start 2024-01-16T15:15:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b49000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b240047c0 0x7f7b440ca8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b49000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440c6dd0 0x7f7b440c7230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b49000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 0x7f7b440b2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b49000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b440c7a90 con 0x7f7b440b28a0 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b49000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b440b3870 con 0x7f7b240047c0 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b49000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b440c7770 con 0x7f7b440c6dd0 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 0x7f7b440b2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b43e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 0x7f7b440b2d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50060/0 (socket says 172.21.15.53:50060) 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.678+0000 7f7b43e00640 1 -- 172.21.15.53:0/3804710291 learned_addr learned my addr 172.21.15.53:0/3804710291 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43e00640 1 -- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440c6dd0 msgr2=0x7f7b440c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43e00640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440c6dd0 0x7f7b440c7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43e00640 1 -- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b240047c0 msgr2=0x7f7b440ca8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43400640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b240047c0 0x7f7b440ca8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43e00640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b240047c0 0x7f7b440ca8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43e00640 1 -- 172.21.15.53:0/3804710291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b440b3590 con 0x7f7b440b28a0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43400640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b240047c0 0x7f7b440ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b43e00640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 0x7f7b440b2d50 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f7b440ae7d0 tx=0x7f7b3c00cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00a6d0 con 0x7f7b440b28a0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b3c00a850 con 0x7f7b440b28a0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b44134190 con 0x7f7b440b28a0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b3c00ac00 con 0x7f7b440b28a0 2024-01-16T15:15:18.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.679+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b44134750 con 0x7f7b440b28a0 2024-01-16T15:15:18.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.681+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44133ff0 con 0x7f7b440b28a0 2024-01-16T15:15:18.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.682+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f7b3c00ad60 con 0x7f7b440b28a0 2024-01-16T15:15:18.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.683+0000 7f7b3b400640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7b1c06c1b0 0x7f7b1c06e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:18.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.683+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f7b3c0882c0 con 0x7f7b440b28a0 2024-01-16T15:15:18.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.684+0000 7f7b43400640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7b1c06c1b0 0x7f7b1c06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:18.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.685+0000 7f7b43400640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7b1c06c1b0 0x7f7b1c06e670 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b34004770 tx=0x7f7b34008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:18.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.688+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7b44133ff0 con 0x7f7b440b28a0 2024-01-16T15:15:18.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.880+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7b44133ff0 con 0x7f7b440b28a0 2024-01-16T15:15:18.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.881+0000 7f7b3b400640 1 -- 172.21.15.53:0/3804710291 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7b44133ff0 con 0x7f7b440b28a0 2024-01-16T15:15:18.865 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7b1c06c1b0 msgr2=0x7f7b1c06e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7b1c06c1b0 0x7f7b1c06e670 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f7b34004770 tx=0x7f7b34008040 comp rx=0 tx=0).stop 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 msgr2=0x7f7b440b2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 0x7f7b440b2d50 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f7b440ae7d0 tx=0x7f7b3c00cf00 comp rx=0 tx=0).stop 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 shutdown_connections 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7b1c06c1b0 0x7f7b1c06e670 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7b440b28a0 0x7f7b440b2d50 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7b440c6dd0 0x7f7b440c7230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 --2- 172.21.15.53:0/3804710291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7b240047c0 0x7f7b440ca8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.884+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 >> 172.21.15.53:0/3804710291 conn(0x7f7b4406a7d0 msgr2=0x7f7b4406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.885+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 shutdown_connections 2024-01-16T15:15:18.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:18.885+0000 7f7b49000640 1 -- 172.21.15.53:0/3804710291 wait complete. 2024-01-16T15:15:18.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:15:19.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.123+0000 7f7169200640 1 Processor -- start 2024-01-16T15:15:19.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.123+0000 7f7169200640 1 -- start start 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7169200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716413df80 0x7f716415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7169200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f716413e360 0x7f71640cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7169200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f716415f9c0 0x7f71640cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7169200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716406c470 con 0x7f716415f9c0 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7169200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f716406c750 con 0x7f716413e360 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7169200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f716406ca30 con 0x7f716413df80 2024-01-16T15:15:19.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7162800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f716415f9c0 0x7f71640cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7162800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f716415f9c0 0x7f71640cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50074/0 (socket says 172.21.15.53:50074) 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.125+0000 7f7162800640 1 -- 172.21.15.53:0/11979200 learned_addr learned my addr 172.21.15.53:0/11979200 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7161e00640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716413df80 0x7f716415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f715be00640 1 --2- 172.21.15.53:0/11979200 >> v2:172.21.15.146:3300/0 conn(0x7f716413e360 0x7f71640cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7162800640 1 -- 172.21.15.53:0/11979200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716413df80 msgr2=0x7f716415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7162800640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716413df80 0x7f716415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7162800640 1 -- 172.21.15.53:0/11979200 >> v2:172.21.15.146:3300/0 conn(0x7f716413e360 msgr2=0x7f71640cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7162800640 1 --2- 172.21.15.53:0/11979200 >> v2:172.21.15.146:3300/0 conn(0x7f716413e360 0x7f71640cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7162800640 1 -- 172.21.15.53:0/11979200 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716406cd10 con 0x7f716415f9c0 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f715be00640 1 --2- 172.21.15.53:0/11979200 >> v2:172.21.15.146:3300/0 conn(0x7f716413e360 0x7f71640cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:19.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.126+0000 7f7161e00640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716413df80 0x7f716415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:19.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.127+0000 7f7162800640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716415f9c0 0x7f71640cf7d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f71640cfed0 tx=0x7f715c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5812a7890120e98a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.127+0000 7f7161400640 1 -- 172.21.15.53:0/11979200 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c02ca70 con 0x7f716415f9c0 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.127+0000 7f7161400640 1 -- 172.21.15.53:0/11979200 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f715c02cbd0 con 0x7f716415f9c0 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.127+0000 7f7161400640 1 -- 172.21.15.53:0/11979200 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715c0343f0 con 0x7f716415f9c0 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 -- 172.21.15.53:0/11979200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716415f9c0 msgr2=0x7f71640cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716415f9c0 0x7f71640cf7d0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f71640cfed0 tx=0x7f715c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 -- 172.21.15.53:0/11979200 shutdown_connections 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716415f9c0 0x7f71640cf7d0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 --2- 172.21.15.53:0/11979200 >> v2:172.21.15.146:3300/0 conn(0x7f716413e360 0x7f71640cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 --2- 172.21.15.53:0/11979200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716413df80 0x7f716415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 -- 172.21.15.53:0/11979200 >> 172.21.15.53:0/11979200 conn(0x7f716406a800 msgr2=0x7f716406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 -- 172.21.15.53:0/11979200 shutdown_connections 2024-01-16T15:15:19.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.128+0000 7f7169200640 1 -- 172.21.15.53:0/11979200 wait complete. 2024-01-16T15:15:19.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.130+0000 7f7169200640 1 Processor -- start 2024-01-16T15:15:19.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.130+0000 7f7169200640 1 -- start start 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f7169200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716413df80 0x7f71640b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f7169200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 0x7f71640a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f7169200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716415f9c0 0x7f71641547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f7169200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b2ec0 con 0x7f716413e360 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f7169200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71640b3d00 con 0x7f716413df80 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f7169200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71640a9440 con 0x7f716415f9c0 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f715be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 0x7f71640a8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f715be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 0x7f71640a8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50076/0 (socket says 172.21.15.53:50076) 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.131+0000 7f715be00640 1 -- 172.21.15.53:0/3740931139 learned_addr learned my addr 172.21.15.53:0/3740931139 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f7161e00640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716413df80 0x7f71640b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f7162800640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716415f9c0 0x7f71641547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715be00640 1 -- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716415f9c0 msgr2=0x7f71641547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715be00640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716415f9c0 0x7f71641547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715be00640 1 -- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716413df80 msgr2=0x7f71640b2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715be00640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716413df80 0x7f71640b2980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715be00640 1 -- 172.21.15.53:0/3740931139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c009b30 con 0x7f716413e360 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715be00640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 0x7f71640a8f00 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f714c00c7f0 tx=0x7f714c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c007a30 con 0x7f716413e360 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714c007b90 con 0x7f716413e360 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714c00ce40 con 0x7f716413e360 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.132+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71640b30f0 con 0x7f716413e360 2024-01-16T15:15:19.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.133+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71640b36b0 con 0x7f716413e360 2024-01-16T15:15:19.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.134+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71640b3e90 con 0x7f716413e360 2024-01-16T15:15:19.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.135+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f714c0113f0 con 0x7f716413e360 2024-01-16T15:15:19.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.135+0000 7f715a000640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f713c06c160 0x7f713c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.136+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f714c088ad0 con 0x7f716413e360 2024-01-16T15:15:19.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.136+0000 7f7161e00640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f713c06c160 0x7f713c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.137+0000 7f7161e00640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f713c06c160 0x7f713c06e620 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f71640b1f00 tx=0x7f7154008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.140+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f714c0550b0 con 0x7f716413e360 2024-01-16T15:15:19.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.335+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f71640675d0 con 0x7f716413e360 2024-01-16T15:15:19.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.336+0000 7f715a000640 1 -- 172.21.15.53:0/3740931139 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v95) v1 ==== 84+0+65 (secure 0 0 0) 0x7f714c0586d0 con 0x7f716413e360 2024-01-16T15:15:19.320 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:19.321 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:15:19.321 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:19.321 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:15:19.321 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:15:19.321 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:19.321 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:15:19.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.340+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f713c06c160 msgr2=0x7f713c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.340+0000 7f7169200640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f713c06c160 0x7f713c06e620 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f71640b1f00 tx=0x7f7154008040 comp rx=0 tx=0).stop 2024-01-16T15:15:19.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.340+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 msgr2=0x7f71640a8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.340+0000 7f7169200640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 0x7f71640a8f00 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f714c00c7f0 tx=0x7f714c00cbf0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.340+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 shutdown_connections 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.340+0000 7f7169200640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f713c06c160 0x7f713c06e620 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.341+0000 7f7169200640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f716415f9c0 0x7f71641547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.341+0000 7f7169200640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f716413e360 0x7f71640a8f00 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.341+0000 7f7169200640 1 --2- 172.21.15.53:0/3740931139 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f716413df80 0x7f71640b2980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.341+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 >> 172.21.15.53:0/3740931139 conn(0x7f716406a800 msgr2=0x7f716413c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.341+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 shutdown_connections 2024-01-16T15:15:19.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.341+0000 7f7169200640 1 -- 172.21.15.53:0/3740931139 wait complete. 2024-01-16T15:15:19.345 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:15:19.345 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:15:19.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.586+0000 7fabe4a00640 1 Processor -- start 2024-01-16T15:15:19.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.587+0000 7fabe4a00640 1 -- start start 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.588+0000 7fabe4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fabe0143c40 0x7fabe0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.588+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.588+0000 7fabe4a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0070e50 con 0x7fabe00cca40 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.588+0000 7fabe4a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabe00723c0 con 0x7fabe0143c40 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.588+0000 7fabe4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe00cce70 con 0x7fabe00717d0 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fabe0143c40 0x7fabe0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fabe0143c40 0x7fabe0071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50086/0 (socket says 172.21.15.53:50086) 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 -- 172.21.15.53:0/998786149 learned_addr learned my addr 172.21.15.53:0/998786149 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabd7e00640 1 --2- 172.21.15.53:0/998786149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 -- 172.21.15.53:0/998786149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 msgr2=0x7fabe00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 --2- 172.21.15.53:0/998786149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe00cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 -- 172.21.15.53:0/998786149 >> v1:172.21.15.146:6789/0 conn(0x7fabe00cca40 legacy=0x7fabe015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.589+0000 7fabde800640 1 -- 172.21.15.53:0/998786149 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe006e170 con 0x7fabe0143c40 2024-01-16T15:15:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.591+0000 7fabde800640 1 --2- 172.21.15.53:0/998786149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe0143c40 0x7fabe0071290 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009870 tx=0x7fabd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fff68513236a101 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.591+0000 7fabdde00640 1 -- 172.21.15.53:0/998786149 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0031640 con 0x7fabe0143c40 2024-01-16T15:15:19.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.591+0000 7fabdde00640 1 -- 172.21.15.53:0/998786149 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabd00317a0 con 0x7fabe0143c40 2024-01-16T15:15:19.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.591+0000 7fabdde00640 1 -- 172.21.15.53:0/998786149 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd0031b30 con 0x7fabe0143c40 2024-01-16T15:15:19.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 -- 172.21.15.53:0/998786149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe0143c40 msgr2=0x7fabe0071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 --2- 172.21.15.53:0/998786149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe0143c40 0x7fabe0071290 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7fabd0009870 tx=0x7fabd002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 -- 172.21.15.53:0/998786149 shutdown_connections 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 --2- 172.21.15.53:0/998786149 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 --2- 172.21.15.53:0/998786149 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe0143c40 0x7fabe0071290 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 -- 172.21.15.53:0/998786149 >> 172.21.15.53:0/998786149 conn(0x7fabe006a800 msgr2=0x7fabe006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 -- 172.21.15.53:0/998786149 shutdown_connections 2024-01-16T15:15:19.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.592+0000 7fabe4a00640 1 -- 172.21.15.53:0/998786149 wait complete. 2024-01-16T15:15:19.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.594+0000 7fabe4a00640 1 Processor -- start 2024-01-16T15:15:19.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.594+0000 7fabe4a00640 1 -- start start 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe01395e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 0x7fabe01577e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabe4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe0143c40 0x7fabe00afac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabe4a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe013b9f0 con 0x7fabe00cca40 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabe4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabe0139b20 con 0x7fabe0143c40 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabe4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabe01581e0 con 0x7fabe00717d0 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabd7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 0x7fabe01577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabd7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 0x7fabe01577e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50098/0 (socket says 172.21.15.53:50098) 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.595+0000 7fabd7e00640 1 -- 172.21.15.53:0/827746578 learned_addr learned my addr 172.21.15.53:0/827746578 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd7e00640 1 -- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 msgr2=0x7fabe01395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd7e00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe01395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd7e00640 1 -- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe0143c40 msgr2=0x7fabe00afac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabdf200640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe0143c40 0x7fabe00afac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd7e00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe0143c40 0x7fabe00afac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd7e00640 1 -- 172.21.15.53:0/827746578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd00094d0 con 0x7fabe00cca40 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabdf200640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe0143c40 0x7fabe00afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd7e00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 0x7fabe01577e0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fabc800d800 tx=0x7fabc800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc800b6c0 con 0x7fabe00cca40 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc800b840 con 0x7fabe00cca40 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc800bbf0 con 0x7fabe00cca40 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe01584c0 con 0x7fabe00cca40 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.596+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe0158a80 con 0x7fabe00cca40 2024-01-16T15:15:19.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.597+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe0067450 con 0x7fabe00cca40 2024-01-16T15:15:19.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.598+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fabc800bd50 con 0x7fabe00cca40 2024-01-16T15:15:19.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.599+0000 7fabd6a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabb806c160 0x7fabb806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:19.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.600+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fabc8089030 con 0x7fabe00cca40 2024-01-16T15:15:19.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.600+0000 7fabde800640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabb806c160 0x7fabb806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:19.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.601+0000 7fabde800640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabb806c160 0x7fabb806e620 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fabd0036000 tx=0x7fabd0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:19.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.604+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fabc8055610 con 0x7fabe00cca40 2024-01-16T15:15:19.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:19.789+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fabe013bb30 con 0x7fabe00cca40 2024-01-16T15:15:20.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.479+0000 7fabd6a00640 1 -- 172.21.15.53:0/827746578 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v96) v1 ==== 138+0+0 (secure 0 0 0) 0x7fabc8058c30 con 0x7fabe00cca40 2024-01-16T15:15:20.464 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:15:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabb806c160 msgr2=0x7fabb806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabb806c160 0x7fabb806e620 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fabd0036000 tx=0x7fabd0030000 comp rx=0 tx=0).stop 2024-01-16T15:15:20.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 msgr2=0x7fabe01577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 0x7fabe01577e0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7fabc800d800 tx=0x7fabc800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 shutdown_connections 2024-01-16T15:15:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabb806c160 0x7fabb806e620 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabe0143c40 0x7fabe00afac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabe00cca40 0x7fabe01577e0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 --2- 172.21.15.53:0/827746578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabe00717d0 0x7fabe01395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.483+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 >> 172.21.15.53:0/827746578 conn(0x7fabe006a800 msgr2=0x7fabe015a770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.484+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 shutdown_connections 2024-01-16T15:15:20.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.484+0000 7fabe4a00640 1 -- 172.21.15.53:0/827746578 wait complete. 2024-01-16T15:15:20.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:15:20.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.703+0000 7fd983200640 1 Processor -- start 2024-01-16T15:15:20.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.704+0000 7fd983200640 1 -- start start 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.705+0000 7fd983200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd98415afb0 0x7fd98413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.705+0000 7fd983200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd98413fae0 0x7fd9840b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.705+0000 7fd983200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98413ff30 con 0x7fd98413cfa0 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.705+0000 7fd983200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9840cdc40 con 0x7fd98413fae0 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.705+0000 7fd983200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9840cdec0 con 0x7fd98415afb0 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd98413fae0 0x7fd9840b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd98413fae0 0x7fd9840b2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50106/0 (socket says 172.21.15.53:50106) 2024-01-16T15:15:20.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 -- 172.21.15.53:0/2463220439 learned_addr learned my addr 172.21.15.53:0/2463220439 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:20.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd97be00640 1 --2- 172.21.15.53:0/2463220439 >> v2:172.21.15.146:3300/0 conn(0x7fd98415afb0 0x7fd98413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 -- 172.21.15.53:0/2463220439 >> v2:172.21.15.146:3300/0 conn(0x7fd98415afb0 msgr2=0x7fd98413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:20.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 --2- 172.21.15.53:0/2463220439 >> v2:172.21.15.146:3300/0 conn(0x7fd98415afb0 0x7fd98413f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 -- 172.21.15.53:0/2463220439 >> v1:172.21.15.53:6789/0 conn(0x7fd98413cfa0 legacy=0x7fd9840cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.706+0000 7fd982800640 1 -- 172.21.15.53:0/2463220439 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9840ce1a0 con 0x7fd98413fae0 2024-01-16T15:15:20.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.707+0000 7fd982800640 1 --2- 172.21.15.53:0/2463220439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd9840b2040 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd9840b3fd0 tx=0x7fd97c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d29f0b9044ef21a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd981400640 1 -- 172.21.15.53:0/2463220439 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c02ca80 con 0x7fd98413fae0 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd981400640 1 -- 172.21.15.53:0/2463220439 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c02cbe0 con 0x7fd98413fae0 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd981400640 1 -- 172.21.15.53:0/2463220439 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0343f0 con 0x7fd98413fae0 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 -- 172.21.15.53:0/2463220439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 msgr2=0x7fd9840b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 --2- 172.21.15.53:0/2463220439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd9840b2040 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fd9840b3fd0 tx=0x7fd97c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 -- 172.21.15.53:0/2463220439 shutdown_connections 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 --2- 172.21.15.53:0/2463220439 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd9840b2040 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 --2- 172.21.15.53:0/2463220439 >> v2:172.21.15.146:3300/0 conn(0x7fd98415afb0 0x7fd98413f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 -- 172.21.15.53:0/2463220439 >> 172.21.15.53:0/2463220439 conn(0x7fd98406a800 msgr2=0x7fd98406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 -- 172.21.15.53:0/2463220439 shutdown_connections 2024-01-16T15:15:20.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.708+0000 7fd983200640 1 -- 172.21.15.53:0/2463220439 wait complete. 2024-01-16T15:15:20.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.710+0000 7fd983200640 1 Processor -- start 2024-01-16T15:15:20.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.710+0000 7fd983200640 1 -- start start 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd983200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd98413cfa0 0x7fd9841539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd983200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd984139be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd983200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd98415afb0 0x7fd98413a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd983200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd984154a90 con 0x7fd98413fae0 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd983200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98413d740 con 0x7fd98415afb0 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd983200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98413da20 con 0x7fd98413cfa0 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd97be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd984139be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd97be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd984139be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50120/0 (socket says 172.21.15.53:50120) 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.711+0000 7fd97be00640 1 -- 172.21.15.53:0/1963560824 learned_addr learned my addr 172.21.15.53:0/1963560824 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:20.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd982800640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd98415afb0 0x7fd98413a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97be00640 1 -- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd98413cfa0 msgr2=0x7fd9841539c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd981e00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd98413cfa0 0x7fd9841539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97be00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd98413cfa0 0x7fd9841539c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97be00640 1 -- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd98415afb0 msgr2=0x7fd98413a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97be00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd98415afb0 0x7fd98413a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97be00640 1 -- 172.21.15.53:0/1963560824 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c009b30 con 0x7fd98413fae0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd982800640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd98415afb0 0x7fd98413a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd981e00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd98413cfa0 0x7fd9841539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97be00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd984139be0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00d5f0 tx=0x7fd96c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.712+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c0125f0 con 0x7fd98413fae0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.713+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96c012750 con 0x7fd98413fae0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.713+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98413dd00 con 0x7fd98413fae0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.713+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd96c012b00 con 0x7fd98413fae0 2024-01-16T15:15:20.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.713+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98413e2c0 con 0x7fd98413fae0 2024-01-16T15:15:20.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.714+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98413e480 con 0x7fd98413fae0 2024-01-16T15:15:20.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.715+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fd96c012c60 con 0x7fd98413fae0 2024-01-16T15:15:20.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.716+0000 7fd97a000640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95c06c160 0x7fd95c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:20.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.716+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5255+0+0 (secure 0 0 0) 0x7fd96c089960 con 0x7fd98413fae0 2024-01-16T15:15:20.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.717+0000 7fd981e00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95c06c160 0x7fd95c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:20.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.718+0000 7fd981e00640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95c06c160 0x7fd95c06e620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd974000f60 tx=0x7fd974009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:20.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.721+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd96c089e60 con 0x7fd98413fae0 2024-01-16T15:15:20.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:20.917+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fd984067450 con 0x7fd98413fae0 2024-01-16T15:15:21.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.494+0000 7fd97a000640 1 -- 172.21.15.53:0/1963560824 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v97) v1 ==== 170+0+0 (secure 0 0 0) 0x7fd96c055f40 con 0x7fd98413fae0 2024-01-16T15:15:21.479 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:15:21.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95c06c160 msgr2=0x7fd95c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95c06c160 0x7fd95c06e620 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd974000f60 tx=0x7fd974009040 comp rx=0 tx=0).stop 2024-01-16T15:15:21.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 msgr2=0x7fd984139be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd984139be0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fd96c00d5f0 tx=0x7fd96c00dab0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 shutdown_connections 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd95c06c160 0x7fd95c06e620 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd98415afb0 0x7fd98413a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd98413fae0 0x7fd984139be0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 --2- 172.21.15.53:0/1963560824 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd98413cfa0 0x7fd9841539c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 >> 172.21.15.53:0/1963560824 conn(0x7fd98406a800 msgr2=0x7fd9841581b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 shutdown_connections 2024-01-16T15:15:21.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.498+0000 7fd983200640 1 -- 172.21.15.53:0/1963560824 wait complete. 2024-01-16T15:15:21.520 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:15:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.717+0000 7ff03b200640 1 Processor -- start 2024-01-16T15:15:21.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.717+0000 7ff03b200640 1 -- start start 2024-01-16T15:15:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff03b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 0x7ff03c15bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff03b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03c15f4e0 con 0x7ff03c0ccb50 2024-01-16T15:15:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff03b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03c070ee0 con 0x7ff03c0adf20 2024-01-16T15:15:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff03b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c15c450 con 0x7ff03c070a30 2024-01-16T15:15:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff033e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff03c0ccb50 0x7ff03c143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:51608/0 (socket says 172.21.15.53:51608) 2024-01-16T15:15:21.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff033e00640 1 -- 172.21.15.53:0/460252799 learned_addr learned my addr 172.21.15.53:0/460252799 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:21.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073201327 0 0) 0x7ff03c15f4e0 con 0x7ff03c0ccb50 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff01c003520 con 0x7ff03c0ccb50 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.719+0000 7ff03a800640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 0x7ff03c15bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3476291765 0 0) 0x7ff01c003520 con 0x7ff03c0ccb50 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 msgr2=0x7ff03c15bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 0x7ff03c15bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 >> v1:172.21.15.146:6789/0 conn(0x7ff03c0adf20 legacy=0x7ff03c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c071130 con 0x7ff03c0ccb50 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7ff024001670 con 0x7ff03c0ccb50 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 >> v1:172.21.15.53:6789/0 conn(0x7ff03c0ccb50 legacy=0x7ff03c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff01c0039a0 0x7ff01c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff01c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 0x7ff01c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c15c450 con 0x7ff01c0047c0 2024-01-16T15:15:21.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c0054b0 con 0x7ff01c0039a0 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c005670 con 0x7ff03c070a30 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.720+0000 7ff033e00640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff01c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.721+0000 7ff033e00640 1 -- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 msgr2=0x7ff01c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.721+0000 7ff033e00640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c070a30 0x7ff01c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.721+0000 7ff033e00640 1 -- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff01c0039a0 msgr2=0x7ff01c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.721+0000 7ff039e00640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff01c0039a0 0x7ff01c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.721+0000 7ff033e00640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff01c0039a0 0x7ff01c003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.721+0000 7ff033e00640 1 -- 172.21.15.53:0/460252799 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c003520 con 0x7ff01c0047c0 2024-01-16T15:15:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.722+0000 7ff033e00640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff01c004c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7ff024007f90 tx=0x7ff01c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48313dab90cbe6b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.722+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff024005ad0 con 0x7ff01c0047c0 2024-01-16T15:15:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.722+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff024005c30 con 0x7ff01c0047c0 2024-01-16T15:15:21.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.722+0000 7ff039400640 1 -- 172.21.15.53:0/460252799 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02402c790 con 0x7ff01c0047c0 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.722+0000 7ff03b200640 1 -- 172.21.15.53:0/460252799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 msgr2=0x7ff01c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.722+0000 7ff03b200640 1 --2- 172.21.15.53:0/460252799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff01c004c40 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7ff024007f90 tx=0x7ff01c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.723+0000 7ff03a800640 1 -- 172.21.15.53:0/460252799 reap_dead start 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.723+0000 7ff03b200640 1 -- 172.21.15.53:0/460252799 shutdown_connections 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.723+0000 7ff03b200640 1 -- 172.21.15.53:0/460252799 >> 172.21.15.53:0/460252799 conn(0x7ff03c06a7d0 msgr2=0x7ff03c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.723+0000 7ff03b200640 1 -- 172.21.15.53:0/460252799 shutdown_connections 2024-01-16T15:15:21.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.723+0000 7ff03b200640 1 -- 172.21.15.53:0/460252799 wait complete. 2024-01-16T15:15:21.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.725+0000 7ff03b200640 1 Processor -- start 2024-01-16T15:15:21.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.725+0000 7ff03b200640 1 -- start start 2024-01-16T15:15:21.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff03b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff03c1360d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff03b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff03c1f68d0 0x7ff03c1f21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff03b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c0ac8d0 0x7ff03c0acd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff03b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c0c70b0 con 0x7ff01c0047c0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff03b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c1341b0 con 0x7ff03c1f68d0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff03b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff03c0ad290 con 0x7ff03c0ac8d0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff03c1360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff03c1360d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50140/0 (socket says 172.21.15.53:50140) 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 -- 172.21.15.53:0/4082551872 learned_addr learned my addr 172.21.15.53:0/4082551872 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 -- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c0ac8d0 msgr2=0x7ff03c0acd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c0ac8d0 0x7ff03c0acd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 -- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff03c1f68d0 msgr2=0x7ff03c1f21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff033e00640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff03c1f68d0 0x7ff03c1f21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff03c1f68d0 0x7ff03c1f21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.726+0000 7ff039e00640 1 -- 172.21.15.53:0/4082551872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff03c1f6d30 con 0x7ff01c0047c0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.727+0000 7ff039e00640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff03c1360d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00d570 tx=0x7ff02c00da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.727+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c005480 con 0x7ff01c0047c0 2024-01-16T15:15:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.727+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c003610 con 0x7ff01c0047c0 2024-01-16T15:15:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.727+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c003980 con 0x7ff01c0047c0 2024-01-16T15:15:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.727+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff03c1366f0 con 0x7ff01c0047c0 2024-01-16T15:15:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.727+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff03c136cb0 con 0x7ff01c0047c0 2024-01-16T15:15:21.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.728+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03c067450 con 0x7ff01c0047c0 2024-01-16T15:15:21.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.730+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7ff02c003ae0 con 0x7ff01c0047c0 2024-01-16T15:15:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.730+0000 7ff032000640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff01406c190 0x7ff01406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.730+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5637+0+0 (secure 0 0 0) 0x7ff02c0892a0 con 0x7ff01c0047c0 2024-01-16T15:15:21.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.731+0000 7ff033e00640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff01406c190 0x7ff01406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:21.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.732+0000 7ff033e00640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff01406c190 0x7ff01406e650 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff024034000 tx=0x7ff024037040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:21.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.734+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff02c0558c0 con 0x7ff01c0047c0 2024-01-16T15:15:21.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:21.956+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7ff03c06e110 con 0x7ff01c0047c0 2024-01-16T15:15:22.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.498+0000 7ff032000640 1 -- 172.21.15.53:0/4082551872 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v61) v1 ==== 174+0+0 (secure 0 0 0) 0x7ff02c001230 con 0x7ff01c0047c0 2024-01-16T15:15:22.483 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 14 and data pool 15 2024-01-16T15:15:22.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.504+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff01406c190 msgr2=0x7ff01406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.504+0000 7ff03b200640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff01406c190 0x7ff01406e650 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7ff024034000 tx=0x7ff024037040 comp rx=0 tx=0).stop 2024-01-16T15:15:22.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.504+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 msgr2=0x7ff03c1360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.504+0000 7ff03b200640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff03c1360d0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7ff02c00d570 tx=0x7ff02c00da30 comp rx=0 tx=0).stop 2024-01-16T15:15:22.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.513+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 shutdown_connections 2024-01-16T15:15:22.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.513+0000 7ff03b200640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff01406c190 0x7ff01406e650 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.513+0000 7ff03b200640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff03c0ac8d0 0x7ff03c0acd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.513+0000 7ff03b200640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff03c1f68d0 0x7ff03c1f21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.513+0000 7ff03b200640 1 --2- 172.21.15.53:0/4082551872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff01c0047c0 0x7ff03c1360d0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.513+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 >> 172.21.15.53:0/4082551872 conn(0x7ff03c06a7d0 msgr2=0x7ff03c0702b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.514+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 shutdown_connections 2024-01-16T15:15:22.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.514+0000 7ff03b200640 1 -- 172.21.15.53:0/4082551872 wait complete. 2024-01-16T15:15:22.539 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:22.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.730+0000 7f989f600640 1 Processor -- start 2024-01-16T15:15:22.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.731+0000 7f989f600640 1 -- start start 2024-01-16T15:15:22.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.732+0000 7f989f600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f98981610e0 0x7f98980d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.732+0000 7f989f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 0x7f98981614c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.732+0000 7f989f600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98980b2920 con 0x7f989815cc90 2024-01-16T15:15:22.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.732+0000 7f989f600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9898161a00 con 0x7f98981610e0 2024-01-16T15:15:22.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.732+0000 7f989f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98980cb670 con 0x7f98980cf4c0 2024-01-16T15:15:22.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.733+0000 7f9897e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 0x7f98981614c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.733+0000 7f989d600640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f989815cc90 0x7f989815d120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:43004/0 (socket says 172.21.15.53:43004) 2024-01-16T15:15:22.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.733+0000 7f989d600640 1 -- 172.21.15.53:0/12891099 learned_addr learned my addr 172.21.15.53:0/12891099 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:22.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.733+0000 7f9897e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 0x7f98981614c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:37448/0 (socket says 172.21.15.53:37448) 2024-01-16T15:15:22.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.733+0000 7f989cc00640 1 --2- 172.21.15.53:0/12891099 >> v2:172.21.15.146:3300/0 conn(0x7f98981610e0 0x7f98980d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.734+0000 7f9897e00640 1 -- 172.21.15.53:0/12891099 >> v2:172.21.15.146:3300/0 conn(0x7f98981610e0 msgr2=0x7f98980d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:22.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.734+0000 7f9897e00640 1 --2- 172.21.15.53:0/12891099 >> v2:172.21.15.146:3300/0 conn(0x7f98981610e0 0x7f98980d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.734+0000 7f9897e00640 1 -- 172.21.15.53:0/12891099 >> v1:172.21.15.146:6789/0 conn(0x7f989815cc90 legacy=0x7f989815d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.734+0000 7f9897e00640 1 -- 172.21.15.53:0/12891099 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98980cb950 con 0x7f98980cf4c0 2024-01-16T15:15:22.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.734+0000 7f989cc00640 1 --2- 172.21.15.53:0/12891099 >> v2:172.21.15.146:3300/0 conn(0x7f98981610e0 0x7f98980d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:22.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.735+0000 7f9897e00640 1 --2- 172.21.15.53:0/12891099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 0x7f98981614c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f98800097a0 tx=0x7f988002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31e0e8f8023af626 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.736+0000 7f9897400640 1 -- 172.21.15.53:0/12891099 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f988002d040 con 0x7f98980cf4c0 2024-01-16T15:15:22.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.736+0000 7f9897400640 1 -- 172.21.15.53:0/12891099 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9880031810 con 0x7f98980cf4c0 2024-01-16T15:15:22.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.736+0000 7f9897400640 1 -- 172.21.15.53:0/12891099 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9880031ba0 con 0x7f98980cf4c0 2024-01-16T15:15:22.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 -- 172.21.15.53:0/12891099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 msgr2=0x7f98981614c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 --2- 172.21.15.53:0/12891099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 0x7f98981614c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f98800097a0 tx=0x7f988002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 -- 172.21.15.53:0/12891099 shutdown_connections 2024-01-16T15:15:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 --2- 172.21.15.53:0/12891099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f98980cf4c0 0x7f98981614c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 --2- 172.21.15.53:0/12891099 >> v2:172.21.15.146:3300/0 conn(0x7f98981610e0 0x7f98980d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 -- 172.21.15.53:0/12891099 >> 172.21.15.53:0/12891099 conn(0x7f98980d1020 msgr2=0x7f98980d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 -- 172.21.15.53:0/12891099 shutdown_connections 2024-01-16T15:15:22.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.737+0000 7f989f600640 1 -- 172.21.15.53:0/12891099 wait complete. 2024-01-16T15:15:22.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.739+0000 7f989f600640 1 Processor -- start 2024-01-16T15:15:22.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.739+0000 7f989f600640 1 -- start start 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.740+0000 7f989f600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98980cf4c0 0x7f98980b6af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.740+0000 7f989f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f989815cc90 0x7f9898158270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.740+0000 7f989f600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 0x7f98981f2e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.740+0000 7f989f600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98981f3360 con 0x7f98981610e0 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.740+0000 7f989f600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f989815e5f0 con 0x7f98980cf4c0 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.740+0000 7f989f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f989815e8d0 con 0x7f989815cc90 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 0x7f98981f2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 0x7f98981f2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35856/0 (socket says 172.21.15.53:35856) 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 -- 172.21.15.53:0/933873127 learned_addr learned my addr 172.21.15.53:0/933873127 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989cc00640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98980cf4c0 0x7f98980b6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f9897e00640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f989815cc90 0x7f9898158270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 -- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f989815cc90 msgr2=0x7f9898158270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f989815cc90 0x7f9898158270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 -- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98980cf4c0 msgr2=0x7f98980b6af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98980cf4c0 0x7f98980b6af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f989d600640 1 -- 172.21.15.53:0/933873127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9888009400 con 0x7f98981610e0 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.741+0000 7f9897e00640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f989815cc90 0x7f9898158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:22.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.742+0000 7f989d600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 0x7f98981f2e20 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f989000cbc0 tx=0x7f989000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.742+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989000ad40 con 0x7f98981610e0 2024-01-16T15:15:22.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.742+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9890012400 con 0x7f98981610e0 2024-01-16T15:15:22.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.742+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f989000aea0 con 0x7f98981610e0 2024-01-16T15:15:22.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.742+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9880009400 con 0x7f98981610e0 2024-01-16T15:15:22.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.742+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f989815ee90 con 0x7f98981610e0 2024-01-16T15:15:22.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.743+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98980675d0 con 0x7f98981610e0 2024-01-16T15:15:22.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.744+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f9890007070 con 0x7f98981610e0 2024-01-16T15:15:22.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.745+0000 7f9894c00640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f987006c160 0x7f987006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:22.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.745+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f98900887b0 con 0x7f98981610e0 2024-01-16T15:15:22.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.746+0000 7f989cc00640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f987006c160 0x7f987006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:22.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.747+0000 7f989cc00640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f987006c160 0x7f987006e620 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f98980cea80 tx=0x7f9888008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:22.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.750+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9890054e80 con 0x7f98981610e0 2024-01-16T15:15:22.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.934+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9898160c00 con 0x7f98981610e0 2024-01-16T15:15:22.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.937+0000 7f9894c00640 1 -- 172.21.15.53:0/933873127 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16528 (secure 0 0 0) 0x7f98900870a0 con 0x7f98981610e0 2024-01-16T15:15:22.922 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:22.922 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":98,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:21.958607+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:19.844933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:20.976647+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:22.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f987006c160 msgr2=0x7f987006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f987006c160 0x7f987006e620 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f98980cea80 tx=0x7f9888008040 comp rx=0 tx=0).stop 2024-01-16T15:15:22.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 msgr2=0x7f98981f2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:22.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 0x7f98981f2e20 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f989000cbc0 tx=0x7f989000a430 comp rx=0 tx=0).stop 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 shutdown_connections 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f987006c160 0x7f987006e620 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f98981610e0 0x7f98981f2e20 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f989815cc90 0x7f9898158270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 --2- 172.21.15.53:0/933873127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f98980cf4c0 0x7f98980b6af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.939+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 >> 172.21.15.53:0/933873127 conn(0x7f98980d1020 msgr2=0x7f989806fd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.940+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 shutdown_connections 2024-01-16T15:15:22.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:22.940+0000 7f989f600640 1 -- 172.21.15.53:0/933873127 wait complete. 2024-01-16T15:15:22.941 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:23.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.173+0000 7fd5f8200640 1 Processor -- start 2024-01-16T15:15:23.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.174+0000 7fd5f8200640 1 -- start start 2024-01-16T15:15:23.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.175+0000 7fd5f8200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5f00723e0 0x7fd5f0166170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.175+0000 7fd5f8200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f00ae8a0 con 0x7fd5f015b3f0 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.175+0000 7fd5f8200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5f015fd40 con 0x7fd5f00ae3f0 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.175+0000 7fd5f8200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5f0071930 con 0x7fd5f00723e0 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.176+0000 7fd5f5800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd5f015b3f0 0x7fd5f00adce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55498/0 (socket says 172.21.15.53:55498) 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.176+0000 7fd5f5800640 1 -- 172.21.15.53:0/2592969564 learned_addr learned my addr 172.21.15.53:0/2592969564 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.176+0000 7fd5f6200640 1 --2- 172.21.15.53:0/2592969564 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00723e0 0x7fd5f0166170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.176+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943827110 0 0) 0x7fd5f00ae8a0 con 0x7fd5f015b3f0 2024-01-16T15:15:23.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.176+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5d0003520 con 0x7fd5f015b3f0 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3484837060 0 0) 0x7fd5d0003520 con 0x7fd5f015b3f0 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00723e0 msgr2=0x7fd5f0166170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 --2- 172.21.15.53:0/2592969564 >> v2:172.21.15.146:3300/0 conn(0x7fd5f00723e0 0x7fd5f0166170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 >> v1:172.21.15.146:6789/0 conn(0x7fd5f00ae3f0 legacy=0x7fd5f0071cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f015ef80 con 0x7fd5f015b3f0 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd5e0001670 con 0x7fd5f015b3f0 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 >> v1:172.21.15.53:6789/0 conn(0x7fd5f015b3f0 legacy=0x7fd5f00adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00039c0 0x7fd5d0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d0005180 0x7fd5d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f0071930 con 0x7fd5d00047c0 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d000c080 con 0x7fd5d0005180 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d000c240 con 0x7fd5d00039c0 2024-01-16T15:15:23.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.177+0000 7fd5f4e00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f6200640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d0005180 0x7fd5d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f4e00640 1 -- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00039c0 msgr2=0x7fd5d0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f4e00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5d00039c0 0x7fd5d0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f4e00640 1 -- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d0005180 msgr2=0x7fd5d000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f4e00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5d0005180 0x7fd5d000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f4e00640 1 -- 172.21.15.53:0/2592969564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d0003520 con 0x7fd5d00047c0 2024-01-16T15:15:23.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.178+0000 7fd5f6200640 1 -- 172.21.15.53:0/2592969564 reap_dead start 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.179+0000 7fd5f4e00640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5d0004c40 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0073ca0 tx=0x7fd5d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01a526affcbe978 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.179+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d80052d0 con 0x7fd5d00047c0 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.179+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d802e050 con 0x7fd5d00047c0 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.179+0000 7fd5efe00640 1 -- 172.21.15.53:0/2592969564 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8003830 con 0x7fd5d00047c0 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 -- 172.21.15.53:0/2592969564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 msgr2=0x7fd5d0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5d0004c40 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fd5f0073ca0 tx=0x7fd5d00057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 -- 172.21.15.53:0/2592969564 shutdown_connections 2024-01-16T15:15:23.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 --2- 172.21.15.53:0/2592969564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5d0004c40 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 -- 172.21.15.53:0/2592969564 >> 172.21.15.53:0/2592969564 conn(0x7fd5f00b0c00 msgr2=0x7fd5f00b1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 -- 172.21.15.53:0/2592969564 shutdown_connections 2024-01-16T15:15:23.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.180+0000 7fd5f8200640 1 -- 172.21.15.53:0/2592969564 wait complete. 2024-01-16T15:15:23.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.182+0000 7fd5f8200640 1 Processor -- start 2024-01-16T15:15:23.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.182+0000 7fd5f8200640 1 -- start start 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f8200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5f00c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f8200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00cad70 0x7fd5f009a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f8200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f00cb1f0 0x7fd5f009a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f8200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f00cd2c0 con 0x7fd5d00047c0 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f8200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f00af880 con 0x7fd5f00cb1f0 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f8200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5f00aeee0 con 0x7fd5f00cad70 2024-01-16T15:15:23.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5f00c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5f00c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35874/0 (socket says 172.21.15.53:35874) 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.183+0000 7fd5f5800640 1 -- 172.21.15.53:0/1520021581 learned_addr learned my addr 172.21.15.53:0/1520021581 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f5800640 1 -- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00cad70 msgr2=0x7fd5f009a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f5800640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00cad70 0x7fd5f009a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f5800640 1 -- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f00cb1f0 msgr2=0x7fd5f009a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f6200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f00cb1f0 0x7fd5f009a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f5800640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f00cb1f0 0x7fd5f009a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f5800640 1 -- 172.21.15.53:0/1520021581 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5f00af1c0 con 0x7fd5d00047c0 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f6200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f00cb1f0 0x7fd5f009a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:23.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f5800640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5f00c8280 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c700 tx=0x7fd5e000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00072d0 con 0x7fd5d00047c0 2024-01-16T15:15:23.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e0007430 con 0x7fd5d00047c0 2024-01-16T15:15:23.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00047e0 con 0x7fd5d00047c0 2024-01-16T15:15:23.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5f00af4a0 con 0x7fd5d00047c0 2024-01-16T15:15:23.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.184+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5f00cc6d0 con 0x7fd5d00047c0 2024-01-16T15:15:23.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.186+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5f00af9c0 con 0x7fd5d00047c0 2024-01-16T15:15:23.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.187+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fd5e0004940 con 0x7fd5d00047c0 2024-01-16T15:15:23.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.187+0000 7fd5ed600640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806c160 0x7fd5c806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.188+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd5e0087c70 con 0x7fd5d00047c0 2024-01-16T15:15:23.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.188+0000 7fd5f4e00640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806c160 0x7fd5c806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.189+0000 7fd5f4e00640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806c160 0x7fd5c806e620 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8033000 tx=0x7fd5d802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.192+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5f00af9c0 con 0x7fd5d00047c0 2024-01-16T15:15:23.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.388+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5f00af9c0 con 0x7fd5d00047c0 2024-01-16T15:15:23.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.393+0000 7fd5ed600640 1 -- 172.21.15.53:0/1520021581 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+16528 (secure 0 0 0) 0x7fd5f00af9c0 con 0x7fd5d00047c0 2024-01-16T15:15:23.378 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:23.378 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":98,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:21.958607+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:19.844933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:20.976647+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.395+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806c160 msgr2=0x7fd5c806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.395+0000 7fd5f8200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806c160 0x7fd5c806e620 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8033000 tx=0x7fd5d802f000 comp rx=0 tx=0).stop 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 msgr2=0x7fd5f00c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5f00c8280 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fd5e000c700 tx=0x7fd5e000cbc0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 shutdown_connections 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c806c160 0x7fd5c806e620 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5f00cb1f0 0x7fd5f009a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5f00cad70 0x7fd5f009a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 --2- 172.21.15.53:0/1520021581 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5d00047c0 0x7fd5f00c8280 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 >> 172.21.15.53:0/1520021581 conn(0x7fd5f00b0c00 msgr2=0x7fd5f006da30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 shutdown_connections 2024-01-16T15:15:23.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.396+0000 7fd5f8200640 1 -- 172.21.15.53:0/1520021581 wait complete. 2024-01-16T15:15:23.402 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:15:23.449 INFO:tasks.ceph.mds.c.smithi053.stderr:2024-01-16T15:15:23.465+0000 7fb60ea00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:15:23.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.645+0000 7f10c8a00640 1 Processor -- start 2024-01-16T15:15:23.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.645+0000 7f10c8a00640 1 -- start start 2024-01-16T15:15:23.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 0x7f10c00cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f10c015e3b0 0x7f10c00702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c00cfc30 con 0x7f10c00cf7d0 2024-01-16T15:15:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c8a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c00b1ee0 con 0x7f10c015e3b0 2024-01-16T15:15:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00b2780 con 0x7f10c0143ae0 2024-01-16T15:15:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c5600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f10c00cf7d0 0x7f10c015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55508/0 (socket says 172.21.15.53:55508) 2024-01-16T15:15:23.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.647+0000 7f10c5600640 1 -- 172.21.15.53:0/3957251817 learned_addr learned my addr 172.21.15.53:0/3957251817 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c6000640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 0x7f10c00cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c6a00640 1 --2- 172.21.15.53:0/3957251817 >> v2:172.21.15.146:3300/0 conn(0x7f10c015e3b0 0x7f10c00702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813408241 0 0) 0x7f10c00cfc30 con 0x7f10c00cf7d0 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10a0003520 con 0x7f10c00cf7d0 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 83660029 0 0) 0x7f10a0003520 con 0x7f10c00cf7d0 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 msgr2=0x7f10c00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 0x7f10c00cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 >> v2:172.21.15.146:3300/0 conn(0x7f10c015e3b0 msgr2=0x7f10c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 --2- 172.21.15.53:0/3957251817 >> v2:172.21.15.146:3300/0 conn(0x7f10c015e3b0 0x7f10c00702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c015e990 con 0x7f10c00cf7d0 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f10a8001670 con 0x7f10c00cf7d0 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 >> v1:172.21.15.53:6789/0 conn(0x7f10c00cf7d0 legacy=0x7f10c015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0003970 0x7f10a0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 0x7f10a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10a0004d00 0x7f10a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00b1ee0 con 0x7f10a0004d00 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00b2780 con 0x7f10a0003970 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.648+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a000db90 con 0x7f10c0143ae0 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.649+0000 7f10c6a00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10a0004d00 0x7f10a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.649+0000 7f10c6a00640 1 -- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 msgr2=0x7f10a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.649+0000 7f10c6a00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c0143ae0 0x7f10a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.649+0000 7f10c6a00640 1 -- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0003970 msgr2=0x7f10a0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:23.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.649+0000 7f10c6a00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0003970 0x7f10a0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.649+0000 7f10c6a00640 1 -- 172.21.15.53:0/3957251817 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a0003520 con 0x7f10a0004d00 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.650+0000 7f10c6a00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10a0004d00 0x7f10a0005180 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009fc0 tx=0x7f10a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=861f31f53b9724f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.650+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b802cc60 con 0x7f10a0004d00 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.650+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b800f040 con 0x7f10a0004d00 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.650+0000 7f10c4c00640 1 -- 172.21.15.53:0/3957251817 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8034440 con 0x7f10a0004d00 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c8a00640 1 -- 172.21.15.53:0/3957251817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10a0004d00 msgr2=0x7f10a0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c8a00640 1 --2- 172.21.15.53:0/3957251817 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10a0004d00 0x7f10a0005180 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009fc0 tx=0x7f10a00057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c6a00640 1 -- 172.21.15.53:0/3957251817 reap_dead start 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c8a00640 1 -- 172.21.15.53:0/3957251817 shutdown_connections 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c8a00640 1 -- 172.21.15.53:0/3957251817 >> 172.21.15.53:0/3957251817 conn(0x7f10c006a800 msgr2=0x7f10c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c8a00640 1 -- 172.21.15.53:0/3957251817 shutdown_connections 2024-01-16T15:15:23.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.651+0000 7f10c8a00640 1 -- 172.21.15.53:0/3957251817 wait complete. 2024-01-16T15:15:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.653+0000 7f10c8a00640 1 Processor -- start 2024-01-16T15:15:23.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.653+0000 7f10c8a00640 1 -- start start 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0004d00 0x7f10c015d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c00c8260 0x7f10c0159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 0x7f10c01407d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c0140d10 con 0x7f10c0140350 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c0159ca0 con 0x7f10a0004d00 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c00c95e0 con 0x7f10c00c8260 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 0x7f10c01407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 0x7f10c01407d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35888/0 (socket says 172.21.15.53:35888) 2024-01-16T15:15:23.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.654+0000 7f10c6a00640 1 -- 172.21.15.53:0/4036210812 learned_addr learned my addr 172.21.15.53:0/4036210812 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c6a00640 1 -- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c00c8260 msgr2=0x7f10c0159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c6a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c00c8260 0x7f10c0159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c6a00640 1 -- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0004d00 msgr2=0x7f10c015d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c6a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0004d00 0x7f10c015d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c6a00640 1 -- 172.21.15.53:0/4036210812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c00c89d0 con 0x7f10c0140350 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c6a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 0x7f10c01407d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f10b802c6d0 tx=0x7f10b8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8034d50 con 0x7f10c0140350 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b8003990 con 0x7f10c0140350 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c00c8cb0 con 0x7f10c0140350 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b8003d40 con 0x7f10c0140350 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.655+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c00c9270 con 0x7f10c0140350 2024-01-16T15:15:23.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.656+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c00675d0 con 0x7f10c0140350 2024-01-16T15:15:23.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.657+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f10b8004030 con 0x7f10c0140350 2024-01-16T15:15:23.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.658+0000 7f10be000640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f109806c160 0x7f109806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.658+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f10b80aa4a0 con 0x7f10c0140350 2024-01-16T15:15:23.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.659+0000 7f10c6000640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f109806c160 0x7f109806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:23.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.660+0000 7f10c6000640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f109806c160 0x7f109806e620 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009a40 tx=0x7f10b0009500 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:23.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.663+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f10b8076b30 con 0x7f10c0140350 2024-01-16T15:15:23.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:23.898+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f10c00c9430 con 0x7f10c0140350 2024-01-16T15:15:24.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:24.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.538+0000 7f10be000640 1 -- 172.21.15.53:0/4036210812 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v63) v1 ==== 118+0+0 (secure 0 0 0) 0x7f10b807a150 con 0x7f10c0140350 2024-01-16T15:15:24.532 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.541+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f109806c160 msgr2=0x7f109806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.541+0000 7f10c8a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f109806c160 0x7f109806e620 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f10b0009a40 tx=0x7f10b0009500 comp rx=0 tx=0).stop 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.541+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 msgr2=0x7f10c01407d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.541+0000 7f10c8a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 0x7f10c01407d0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f10b802c6d0 tx=0x7f10b8009750 comp rx=0 tx=0).stop 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 shutdown_connections 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f109806c160 0x7f109806e620 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f10c0140350 0x7f10c01407d0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f10c00c8260 0x7f10c0159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 --2- 172.21.15.53:0/4036210812 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f10a0004d00 0x7f10c015d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 >> 172.21.15.53:0/4036210812 conn(0x7f10c006a800 msgr2=0x7f10c0142f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:24.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 shutdown_connections 2024-01-16T15:15:24.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.542+0000 7f10c8a00640 1 -- 172.21.15.53:0/4036210812 wait complete. 2024-01-16T15:15:24.548 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:15:24.548 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:15:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.815+0000 7fd3f0c00640 1 Processor -- start 2024-01-16T15:15:24.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.815+0000 7fd3f0c00640 1 -- start start 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.817+0000 7fd3f0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3ec143810 0x7fd3ec071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.817+0000 7fd3f0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3ec0ae170 0x7fd3ec15e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.817+0000 7fd3f0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3ec0ae620 con 0x7fd3ec15d770 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.817+0000 7fd3f0c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec143bf0 con 0x7fd3ec143810 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.817+0000 7fd3f0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3ec06c730 con 0x7fd3ec0ae170 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3eb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd3ec143810 0x7fd3ec071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3ebe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd3ec15d770 0x7fd3ec15dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55514/0 (socket says 172.21.15.53:55514) 2024-01-16T15:15:24.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3ebe00640 1 -- 172.21.15.53:0/1768399273 learned_addr learned my addr 172.21.15.53:0/1768399273 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3e3e00640 1 --2- 172.21.15.53:0/1768399273 >> v2:172.21.15.146:3300/0 conn(0x7fd3ec0ae170 0x7fd3ec15e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3eb400640 1 -- 172.21.15.53:0/1768399273 >> v2:172.21.15.146:3300/0 conn(0x7fd3ec0ae170 msgr2=0x7fd3ec15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3eb400640 1 --2- 172.21.15.53:0/1768399273 >> v2:172.21.15.146:3300/0 conn(0x7fd3ec0ae170 0x7fd3ec15e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3eb400640 1 -- 172.21.15.53:0/1768399273 >> v1:172.21.15.53:6789/0 conn(0x7fd3ec15d770 legacy=0x7fd3ec15dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3eb400640 1 -- 172.21.15.53:0/1768399273 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ec0724d0 con 0x7fd3ec143810 2024-01-16T15:15:24.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.818+0000 7fd3eaa00640 1 -- 172.21.15.53:0/1768399273 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746974365 0 0) 0x7fd3ec0ae620 con 0x7fd3ec15d770 2024-01-16T15:15:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.820+0000 7fd3eb400640 1 --2- 172.21.15.53:0/1768399273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec143810 0x7fd3ec071e40 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0b0a40 tx=0x7fd3dc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4cee64823a82f77 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.820+0000 7fd3eaa00640 1 -- 172.21.15.53:0/1768399273 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc02e070 con 0x7fd3ec143810 2024-01-16T15:15:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.820+0000 7fd3eaa00640 1 -- 172.21.15.53:0/1768399273 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3dc0315a0 con 0x7fd3ec143810 2024-01-16T15:15:24.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.820+0000 7fd3eaa00640 1 -- 172.21.15.53:0/1768399273 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3dc031930 con 0x7fd3ec143810 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 -- 172.21.15.53:0/1768399273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec143810 msgr2=0x7fd3ec071e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/1768399273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec143810 0x7fd3ec071e40 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd3ec0b0a40 tx=0x7fd3dc02b840 comp rx=0 tx=0).stop 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 -- 172.21.15.53:0/1768399273 shutdown_connections 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/1768399273 >> v2:172.21.15.146:3300/0 conn(0x7fd3ec0ae170 0x7fd3ec15e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/1768399273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec143810 0x7fd3ec071e40 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 -- 172.21.15.53:0/1768399273 >> 172.21.15.53:0/1768399273 conn(0x7fd3ec06a7d0 msgr2=0x7fd3ec06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 -- 172.21.15.53:0/1768399273 shutdown_connections 2024-01-16T15:15:24.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.821+0000 7fd3f0c00640 1 -- 172.21.15.53:0/1768399273 wait complete. 2024-01-16T15:15:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.823+0000 7fd3f0c00640 1 Processor -- start 2024-01-16T15:15:24.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.823+0000 7fd3f0c00640 1 -- start start 2024-01-16T15:15:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3f0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3ec0ae170 0x7fd3ec1f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3f0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3ec143810 0x7fd3ec06d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3f0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 0x7fd3ec06d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3f0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec13be40 con 0x7fd3ec15d770 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3f0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec1440f0 con 0x7fd3ec0ae170 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3f0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3ec144e10 con 0x7fd3ec143810 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 0x7fd3ec06d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 0x7fd3ec06d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35906/0 (socket says 172.21.15.53:35906) 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.824+0000 7fd3ebe00640 1 -- 172.21.15.53:0/4103831782 learned_addr learned my addr 172.21.15.53:0/4103831782 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3ebe00640 1 -- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3ec143810 msgr2=0x7fd3ec06d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3ebe00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3ec143810 0x7fd3ec06d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3ebe00640 1 -- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3ec0ae170 msgr2=0x7fd3ec1f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3ebe00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3ec0ae170 0x7fd3ec1f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3ebe00640 1 -- 172.21.15.53:0/4103831782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3dc009400 con 0x7fd3ec15d770 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3ebe00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 0x7fd3ec06d860 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400c7b0 tx=0x7fd3e400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e400a640 con 0x7fd3ec15d770 2024-01-16T15:15:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3e400a7c0 con 0x7fd3ec15d770 2024-01-16T15:15:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3ec13c1e0 con 0x7fd3ec15d770 2024-01-16T15:15:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.825+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3e400ab70 con 0x7fd3ec15d770 2024-01-16T15:15:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.826+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3ec144b30 con 0x7fd3ec15d770 2024-01-16T15:15:24.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.827+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3ec13bfb0 con 0x7fd3ec15d770 2024-01-16T15:15:24.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.828+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fd3e400acd0 con 0x7fd3ec15d770 2024-01-16T15:15:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.829+0000 7fd3e3400640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c406c160 0x7fd3c406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.829+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd3e4087de0 con 0x7fd3ec15d770 2024-01-16T15:15:24.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.830+0000 7fd3eb400640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c406c160 0x7fd3c406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:24.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.831+0000 7fd3eb400640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c406c160 0x7fd3c406e620 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc036000 tx=0x7fd3dc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:24.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:24.834+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3ec13bfb0 con 0x7fd3ec15d770 2024-01-16T15:15:25.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.067+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fd3ec13bfb0 con 0x7fd3ec15d770 2024-01-16T15:15:25.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.549+0000 7fd3e3400640 1 -- 172.21.15.53:0/4103831782 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd3ec13bfb0 con 0x7fd3ec15d770 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.552+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c406c160 msgr2=0x7fd3c406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.552+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c406c160 0x7fd3c406e620 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fd3dc036000 tx=0x7fd3dc030000 comp rx=0 tx=0).stop 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.552+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 msgr2=0x7fd3ec06d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.552+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 0x7fd3ec06d860 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fd3e400c7b0 tx=0x7fd3e400cc70 comp rx=0 tx=0).stop 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 shutdown_connections 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3c406c160 0x7fd3c406e620 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3ec15d770 0x7fd3ec06d860 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3ec143810 0x7fd3ec06d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 --2- 172.21.15.53:0/4103831782 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3ec0ae170 0x7fd3ec1f2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 >> 172.21.15.53:0/4103831782 conn(0x7fd3ec06a7d0 msgr2=0x7fd3ec141ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:25.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 shutdown_connections 2024-01-16T15:15:25.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.553+0000 7fd3f0c00640 1 -- 172.21.15.53:0/4103831782 wait complete. 2024-01-16T15:15:25.560 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:15:25.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:25.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.792+0000 7f580d600640 1 Processor -- start 2024-01-16T15:15:25.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.792+0000 7f580d600640 1 -- start start 2024-01-16T15:15:25.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f580d600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f58080cc500 0x7f58080cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f580d600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5808070e50 con 0x7f5808143c40 2024-01-16T15:15:25.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f580d600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5808144020 con 0x7f580806d160 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f580d600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58080723c0 con 0x7f58080cc500 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f5807400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f58080cc500 0x7f58080cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f5807e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5808143c40 0x7f5808071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55520/0 (socket says 172.21.15.53:55520) 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.794+0000 7f5807e00640 1 -- 172.21.15.53:0/4104267630 learned_addr learned my addr 172.21.15.53:0/4104267630 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.795+0000 7f5807400640 1 -- 172.21.15.53:0/4104267630 >> v1:172.21.15.146:6789/0 conn(0x7f580806d160 legacy=0x7f580806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.795+0000 7f5807400640 1 -- 172.21.15.53:0/4104267630 >> v1:172.21.15.53:6789/0 conn(0x7f5808143c40 legacy=0x7f5808071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.795+0000 7f5807400640 1 -- 172.21.15.53:0/4104267630 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5808071e00 con 0x7f58080cc500 2024-01-16T15:15:25.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.796+0000 7f5807400640 1 --2- 172.21.15.53:0/4104267630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58080cc990 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f57f00097a0 tx=0x7f57f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f80a5de9012886f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:25.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.796+0000 7f5806a00640 1 -- 172.21.15.53:0/4104267630 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f002e070 con 0x7f58080cc500 2024-01-16T15:15:25.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.796+0000 7f5806a00640 1 -- 172.21.15.53:0/4104267630 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f00315a0 con 0x7f58080cc500 2024-01-16T15:15:25.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.796+0000 7f5806a00640 1 -- 172.21.15.53:0/4104267630 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0031930 con 0x7f58080cc500 2024-01-16T15:15:25.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.796+0000 7f580d600640 1 -- 172.21.15.53:0/4104267630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 msgr2=0x7f58080cc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.796+0000 7f580d600640 1 --2- 172.21.15.53:0/4104267630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58080cc990 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f57f00097a0 tx=0x7f57f002b840 comp rx=0 tx=0).stop 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.797+0000 7f580d600640 1 -- 172.21.15.53:0/4104267630 shutdown_connections 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.797+0000 7f580d600640 1 --2- 172.21.15.53:0/4104267630 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58080cc990 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.797+0000 7f580d600640 1 -- 172.21.15.53:0/4104267630 >> 172.21.15.53:0/4104267630 conn(0x7f580806a800 msgr2=0x7f580806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.797+0000 7f580d600640 1 -- 172.21.15.53:0/4104267630 shutdown_connections 2024-01-16T15:15:25.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.797+0000 7f580d600640 1 -- 172.21.15.53:0/4104267630 wait complete. 2024-01-16T15:15:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.798+0000 7f580d600640 1 Processor -- start 2024-01-16T15:15:25.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.799+0000 7f580d600640 1 -- start start 2024-01-16T15:15:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f580d600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580806d160 0x7f58081f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f580d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58081f6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f580d600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5808143c40 0x7f5808139ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f580d600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580813b570 con 0x7f58080cc500 2024-01-16T15:15:25.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f580d600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f580813d6b0 con 0x7f580806d160 2024-01-16T15:15:25.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f580d600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f580813acd0 con 0x7f5808143c40 2024-01-16T15:15:25.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58081f6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58081f6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35930/0 (socket says 172.21.15.53:35930) 2024-01-16T15:15:25.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 -- 172.21.15.53:0/3450395458 learned_addr learned my addr 172.21.15.53:0/3450395458 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 -- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5808143c40 msgr2=0x7f5808139ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5808143c40 0x7f5808139ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807e00640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580806d160 0x7f58081f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 -- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580806d160 msgr2=0x7f58081f61f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580806d160 0x7f58081f61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.800+0000 7f5807400640 1 -- 172.21.15.53:0/3450395458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f0009400 con 0x7f58080cc500 2024-01-16T15:15:25.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.801+0000 7f5807400640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58081f6730 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f57f00098c0 tx=0x7f57f00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:25.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.801+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0032900 con 0x7f58080cc500 2024-01-16T15:15:25.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.801+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f0032a80 con 0x7f58080cc500 2024-01-16T15:15:25.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.801+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f580813afb0 con 0x7f58080cc500 2024-01-16T15:15:25.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.801+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f0032e30 con 0x7f58080cc500 2024-01-16T15:15:25.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.801+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5808141360 con 0x7f58080cc500 2024-01-16T15:15:25.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.802+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5808067450 con 0x7f58080cc500 2024-01-16T15:15:25.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.803+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f57f003b3f0 con 0x7f58080cc500 2024-01-16T15:15:25.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.804+0000 7f57ffe00640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f57e006c160 0x7f57e006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:25.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.804+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f57f00aa9e0 con 0x7f58080cc500 2024-01-16T15:15:25.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.805+0000 7f5807e00640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f57e006c160 0x7f57e006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:25.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.806+0000 7f5807e00640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f57e006c160 0x7f57e006e620 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f57f8005880 tx=0x7f57f8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:25.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:25.809+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f57f0076fc0 con 0x7f58080cc500 2024-01-16T15:15:26.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.041+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58080ceb30 con 0x7f58080cc500 2024-01-16T15:15:26.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.042+0000 7f57ffe00640 1 -- 172.21.15.53:0/3450395458 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f57f007a5e0 con 0x7f58080cc500 2024-01-16T15:15:26.027 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:26.027 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1983839723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1983839723},{"type":"v1","addr":"172.21.15.146:6835","nonce":1983839723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":7048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1926242315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1926242315},{"type":"v1","addr":"172.21.15.146:6837","nonce":1926242315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2218545626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2218545626},{"type":"v1","addr":"172.21.15.53:6835","nonce":2218545626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:21.959064+0000","modified":"2024-01-16T15:15:25.069971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8508":{"gid":8508,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/1250261984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1250261984},{"type":"v1","addr":"172.21.15.53:6834","nonce":1250261984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:15:26.028 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 64 2024-01-16T15:15:26.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.045+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f57e006c160 msgr2=0x7f57e006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.045+0000 7f580d600640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f57e006c160 0x7f57e006e620 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f57f8005880 tx=0x7f57f8009040 comp rx=0 tx=0).stop 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.045+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 msgr2=0x7f58081f6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.045+0000 7f580d600640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58081f6730 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f57f00098c0 tx=0x7f57f00046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 shutdown_connections 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f57e006c160 0x7f57e006e620 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5808143c40 0x7f5808139ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58080cc500 0x7f58081f6730 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 --2- 172.21.15.53:0/3450395458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f580806d160 0x7f58081f61f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 >> 172.21.15.53:0/3450395458 conn(0x7f580806a800 msgr2=0x7f5808144230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 shutdown_connections 2024-01-16T15:15:26.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.046+0000 7f580d600640 1 -- 172.21.15.53:0/3450395458 wait complete. 2024-01-16T15:15:26.049 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:15:26.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.286+0000 7f88f3200640 1 Processor -- start 2024-01-16T15:15:26.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.286+0000 7f88f3200640 1 -- start start 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f88ec143ae0 0x7f88ec0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f3200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f88ec0cf7d0 0x7f88ec15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f3200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88ec0cfc30 con 0x7f88ec15e1e0 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0b1ee0 con 0x7f88ec143ae0 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f3200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88ec0b2780 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88ebe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f88ec143ae0 0x7f88ec0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f1200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f88ec15e1e0 0x7f88ec15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55526/0 (socket says 172.21.15.53:55526) 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.287+0000 7f88f1200640 1 -- 172.21.15.53:0/3153156921 learned_addr learned my addr 172.21.15.53:0/3153156921 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:26.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.288+0000 7f88e3e00640 1 --2- 172.21.15.53:0/3153156921 >> v2:172.21.15.146:3300/0 conn(0x7f88ec0cf7d0 0x7f88ec15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.288+0000 7f88ebe00640 1 -- 172.21.15.53:0/3153156921 >> v2:172.21.15.146:3300/0 conn(0x7f88ec0cf7d0 msgr2=0x7f88ec15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.288+0000 7f88ebe00640 1 --2- 172.21.15.53:0/3153156921 >> v2:172.21.15.146:3300/0 conn(0x7f88ec0cf7d0 0x7f88ec15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.288+0000 7f88ebe00640 1 -- 172.21.15.53:0/3153156921 >> v1:172.21.15.53:6789/0 conn(0x7f88ec15e1e0 legacy=0x7f88ec15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.288+0000 7f88ebe00640 1 -- 172.21.15.53:0/3153156921 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88ec15e990 con 0x7f88ec143ae0 2024-01-16T15:15:26.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.289+0000 7f88ebe00640 1 --2- 172.21.15.53:0/3153156921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec143ae0 0x7f88ec0cf290 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0cb830 tx=0x7f88dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731a4277b59c5d00 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.289+0000 7f88eb400640 1 -- 172.21.15.53:0/3153156921 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc031830 con 0x7f88ec143ae0 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.289+0000 7f88eb400640 1 -- 172.21.15.53:0/3153156921 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc031990 con 0x7f88ec143ae0 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.289+0000 7f88eb400640 1 -- 172.21.15.53:0/3153156921 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc031d20 con 0x7f88ec143ae0 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 -- 172.21.15.53:0/3153156921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec143ae0 msgr2=0x7f88ec0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 --2- 172.21.15.53:0/3153156921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec143ae0 0x7f88ec0cf290 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0cb830 tx=0x7f88dc02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 -- 172.21.15.53:0/3153156921 shutdown_connections 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 --2- 172.21.15.53:0/3153156921 >> v2:172.21.15.146:3300/0 conn(0x7f88ec0cf7d0 0x7f88ec15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 --2- 172.21.15.53:0/3153156921 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec143ae0 0x7f88ec0cf290 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 -- 172.21.15.53:0/3153156921 >> 172.21.15.53:0/3153156921 conn(0x7f88ec06a800 msgr2=0x7f88ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 -- 172.21.15.53:0/3153156921 shutdown_connections 2024-01-16T15:15:26.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.290+0000 7f88f3200640 1 -- 172.21.15.53:0/3153156921 wait complete. 2024-01-16T15:15:26.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.292+0000 7f88f3200640 1 Processor -- start 2024-01-16T15:15:26.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.292+0000 7f88f3200640 1 -- start start 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.293+0000 7f88f3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 0x7f88ec15d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.293+0000 7f88f3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88ec143ae0 0x7f88ec141610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.293+0000 7f88f3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88ec15e1e0 0x7f88ec1f1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.293+0000 7f88f3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec15d5d0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.293+0000 7f88f3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec15b020 con 0x7f88ec15e1e0 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.293+0000 7f88f3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec15b300 con 0x7f88ec143ae0 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 0x7f88ec15d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 0x7f88ec15d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35946/0 (socket says 172.21.15.53:35946) 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 -- 172.21.15.53:0/1530067690 learned_addr learned my addr 172.21.15.53:0/1530067690 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88e3e00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88ec143ae0 0x7f88ec141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 -- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88ec143ae0 msgr2=0x7f88ec141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88ec143ae0 0x7f88ec141610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 -- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88ec15e1e0 msgr2=0x7f88ec1f1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88ec15e1e0 0x7f88ec1f1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 -- 172.21.15.53:0/1530067690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc009400 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.294+0000 7f88ebe00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 0x7f88ec15d090 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009890 tx=0x7f88dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.295+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc03ba70 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.295+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88dc03bbf0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.295+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88dc045400 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.295+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec15b5e0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.295+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec15bba0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.296+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec15bd90 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.297+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f88dc04e440 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.299+0000 7f88e8c00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88c406c160 0x7f88c406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.299+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f88dc0aa8d0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.300+0000 7f88e3e00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88c406c160 0x7f88c406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.301+0000 7f88e3e00640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88c406c160 0x7f88c406e620 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f88d4006fd0 tx=0x7f88d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.304+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f88dc0772b0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.498+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88ec0675d0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.500+0000 7f88e8c00640 1 -- 172.21.15.53:0/1530067690 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+16515 (secure 0 0 0) 0x7f88dc07a8d0 con 0x7f88ec0cf7d0 2024-01-16T15:15:26.485 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:26.485 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":99,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:15:23.499947+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:15:19.844933+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2024-01-16T15:15:20.976647+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"99","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:15:26.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.502+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88c406c160 msgr2=0x7f88c406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.502+0000 7f88f3200640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88c406c160 0x7f88c406e620 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f88d4006fd0 tx=0x7f88d4008040 comp rx=0 tx=0).stop 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.502+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 msgr2=0x7f88ec15d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.502+0000 7f88f3200640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 0x7f88ec15d090 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f88dc009890 tx=0x7f88dc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 shutdown_connections 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f88c406c160 0x7f88c406e620 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f88ec15e1e0 0x7f88ec1f1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f88ec143ae0 0x7f88ec141610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 --2- 172.21.15.53:0/1530067690 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f88ec0cf7d0 0x7f88ec15d090 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 >> 172.21.15.53:0/1530067690 conn(0x7f88ec06a800 msgr2=0x7f88ec0ad3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 shutdown_connections 2024-01-16T15:15:26.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.503+0000 7f88f3200640 1 -- 172.21.15.53:0/1530067690 wait complete. 2024-01-16T15:15:26.508 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:15:26.508 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:26.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.733+0000 7fad83e00640 1 Processor -- start 2024-01-16T15:15:26.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.733+0000 7fad83e00640 1 -- start start 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.735+0000 7fad83e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad84071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.735+0000 7fad83e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fad840ccc10 0x7fad8406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.735+0000 7fad83e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad84070e50 con 0x7fad840717d0 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.735+0000 7fad83e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad840723c0 con 0x7fad840ccc10 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.735+0000 7fad83e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad84144020 con 0x7fad84143c40 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.736+0000 7fad82000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fad840717d0 0x7fad840cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55538/0 (socket says 172.21.15.53:55538) 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.736+0000 7fad82000640 1 -- 172.21.15.53:0/3545655389 learned_addr learned my addr 172.21.15.53:0/3545655389 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.736+0000 7fad83400640 1 --2- 172.21.15.53:0/3545655389 >> v2:172.21.15.146:3300/0 conn(0x7fad840ccc10 0x7fad8406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.736+0000 7fad82a00640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad84071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.736+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900820361 0 0) 0x7fad84070e50 con 0x7fad840717d0 2024-01-16T15:15:26.722 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.736+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad64003520 con 0x7fad840717d0 2024-01-16T15:15:26.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2086759336 0 0) 0x7fad64003520 con 0x7fad840717d0 2024-01-16T15:15:26.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 msgr2=0x7fad84071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad84071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 >> v2:172.21.15.146:3300/0 conn(0x7fad840ccc10 msgr2=0x7fad8406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 --2- 172.21.15.53:0/3545655389 >> v2:172.21.15.146:3300/0 conn(0x7fad840ccc10 0x7fad8406d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad84071e00 con 0x7fad840717d0 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fad6c001670 con 0x7fad840717d0 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 >> v1:172.21.15.53:6789/0 conn(0x7fad840717d0 legacy=0x7fad840cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad64003970 0x7fad64003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad640047c0 0x7fad64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840723c0 con 0x7fad640047c0 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad84144020 con 0x7fad64003970 2024-01-16T15:15:26.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6400b9d0 con 0x7fad84143c40 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad82000640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad640047c0 0x7fad64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad82a00640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad64003970 0x7fad64003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.737+0000 7fad83400640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.738+0000 7fad82000640 1 -- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 msgr2=0x7fad64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.738+0000 7fad82000640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad64005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.738+0000 7fad82000640 1 -- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad64003970 msgr2=0x7fad64003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.738+0000 7fad82000640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad64003970 0x7fad64003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.738+0000 7fad82000640 1 -- 172.21.15.53:0/3545655389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64003520 con 0x7fad640047c0 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.738+0000 7fad83400640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad84143c40 0x7fad64005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.739+0000 7fad82000640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad640047c0 0x7fad64004c40 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00c140 tx=0x7fad640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dbacbe18e55b09b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.739+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c005ad0 con 0x7fad640047c0 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.739+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6c005c50 con 0x7fad640047c0 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.739+0000 7fad81600640 1 -- 172.21.15.53:0/3545655389 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6c02c810 con 0x7fad640047c0 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.739+0000 7fad83e00640 1 -- 172.21.15.53:0/3545655389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad640047c0 msgr2=0x7fad64004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.739+0000 7fad83e00640 1 --2- 172.21.15.53:0/3545655389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad640047c0 0x7fad64004c40 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7fad6c00c140 tx=0x7fad640057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.740+0000 7fad83400640 1 -- 172.21.15.53:0/3545655389 reap_dead start 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.740+0000 7fad83e00640 1 -- 172.21.15.53:0/3545655389 shutdown_connections 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.740+0000 7fad83e00640 1 -- 172.21.15.53:0/3545655389 >> 172.21.15.53:0/3545655389 conn(0x7fad8406a800 msgr2=0x7fad8406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.740+0000 7fad83e00640 1 -- 172.21.15.53:0/3545655389 shutdown_connections 2024-01-16T15:15:26.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.740+0000 7fad83e00640 1 -- 172.21.15.53:0/3545655389 wait complete. 2024-01-16T15:15:26.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.742+0000 7fad83e00640 1 Processor -- start 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.742+0000 7fad83e00640 1 -- start start 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad640047c0 0x7fad8415a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad840b2f50 0x7fad840b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 0x7fad840ca000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840b38f0 con 0x7fad840c9b80 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad840cb470 con 0x7fad640047c0 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad8413ed80 con 0x7fad840b2f50 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 0x7fad840ca000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 0x7fad840ca000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35966/0 (socket says 172.21.15.53:35966) 2024-01-16T15:15:26.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 -- 172.21.15.53:0/3287017951 learned_addr learned my addr 172.21.15.53:0/3287017951 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad82000640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad840b2f50 0x7fad840b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 -- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad840b2f50 msgr2=0x7fad840b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad840b2f50 0x7fad840b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 -- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad640047c0 msgr2=0x7fad8415a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad640047c0 0x7fad8415a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad83400640 1 -- 172.21.15.53:0/3287017951 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad8415ae80 con 0x7fad840c9b80 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.743+0000 7fad82000640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad840b2f50 0x7fad840b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.744+0000 7fad83400640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 0x7fad840ca000 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fad840cbcd0 tx=0x7fad7c00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.744+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c017400 con 0x7fad840c9b80 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.744+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad7c011040 con 0x7fad840c9b80 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.744+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad840ca860 con 0x7fad840c9b80 2024-01-16T15:15:26.731 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.744+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad7c017780 con 0x7fad840c9b80 2024-01-16T15:15:26.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.744+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad840cae20 con 0x7fad840c9b80 2024-01-16T15:15:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.745+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad84067450 con 0x7fad840c9b80 2024-01-16T15:15:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.746+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fad7c0203b0 con 0x7fad840c9b80 2024-01-16T15:15:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.747+0000 7fad7b400640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad5c06c160 0x7fad5c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.747+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fad7c088990 con 0x7fad840c9b80 2024-01-16T15:15:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.748+0000 7fad82a00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad5c06c160 0x7fad5c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:26.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.749+0000 7fad82a00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad5c06c160 0x7fad5c06e620 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fad740097b0 tx=0x7fad74008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:26.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.752+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fad7c055250 con 0x7fad840c9b80 2024-01-16T15:15:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.940+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad8406cc40 con 0x7fad5c06c160 2024-01-16T15:15:26.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.969+0000 7fad7b400640 1 -- 172.21.15.53:0/3287017951 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285572 (secure 0 0 0) 0x7fad74009d90 con 0x7fad5c06c160 2024-01-16T15:15:26.959 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:26.961 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:15:26.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad5c06c160 msgr2=0x7fad5c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad5c06c160 0x7fad5c06e620 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fad740097b0 tx=0x7fad74008040 comp rx=0 tx=0).stop 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 msgr2=0x7fad840ca000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 0x7fad840ca000 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fad840cbcd0 tx=0x7fad7c00f4a0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 shutdown_connections 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fad5c06c160 0x7fad5c06e620 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fad840c9b80 0x7fad840ca000 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fad840b2f50 0x7fad840b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 --2- 172.21.15.53:0/3287017951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fad640047c0 0x7fad8415a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 >> 172.21.15.53:0/3287017951 conn(0x7fad8406a800 msgr2=0x7fad840ae7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 shutdown_connections 2024-01-16T15:15:26.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:26.976+0000 7fad83e00640 1 -- 172.21.15.53:0/3287017951 wait complete. 2024-01-16T15:15:26.980 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2024-01-16T15:15:26.579691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591096,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581640,"kb_used_data":10640,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":754393080,"statfs":{"total":773094113280,"available":772498513920,"internally_reserved":0,"allocated":10895360,"data_stored":5353955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":584568368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001591"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691428+0000","last_change":"2024-01-16T15:15:22.537519+0000","last_active":"2024-01-16T15:15:23.691428+0000","last_peered":"2024-01-16T15:15:23.691428+0000","last_clean":"2024-01-16T15:15:23.691428+0000","last_became_active":"2024-01-16T15:15:22.535630+0000","last_became_peered":"2024-01-16T15:15:22.535630+0000","last_unstale":"2024-01-16T15:15:23.691428+0000","last_undegraded":"2024-01-16T15:15:23.691428+0000","last_fullsized":"2024-01-16T15:15:23.691428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:53.424772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740792+0000","last_change":"2024-01-16T15:15:22.532497+0000","last_active":"2024-01-16T15:15:23.740792+0000","last_peered":"2024-01-16T15:15:23.740792+0000","last_clean":"2024-01-16T15:15:23.740792+0000","last_became_active":"2024-01-16T15:15:22.531043+0000","last_became_peered":"2024-01-16T15:15:22.531043+0000","last_unstale":"2024-01-16T15:15:23.740792+0000","last_undegraded":"2024-01-16T15:15:23.740792+0000","last_fullsized":"2024-01-16T15:15:23.740792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691119+0000","last_change":"2024-01-16T15:15:22.535023+0000","last_active":"2024-01-16T15:15:23.691119+0000","last_peered":"2024-01-16T15:15:23.691119+0000","last_clean":"2024-01-16T15:15:23.691119+0000","last_became_active":"2024-01-16T15:15:22.534684+0000","last_became_peered":"2024-01-16T15:15:22.534684+0000","last_unstale":"2024-01-16T15:15:23.691119+0000","last_undegraded":"2024-01-16T15:15:23.691119+0000","last_fullsized":"2024-01-16T15:15:23.691119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:00.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509733+0000","last_change":"2024-01-16T15:15:22.524150+0000","last_active":"2024-01-16T15:15:23.509733+0000","last_peered":"2024-01-16T15:15:23.509733+0000","last_clean":"2024-01-16T15:15:23.509733+0000","last_became_active":"2024-01-16T15:15:22.522946+0000","last_became_peered":"2024-01-16T15:15:22.522946+0000","last_unstale":"2024-01-16T15:15:23.509733+0000","last_undegraded":"2024-01-16T15:15:23.509733+0000","last_fullsized":"2024-01-16T15:15:23.509733+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:13.223095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.519050+0000","last_change":"2024-01-16T15:15:21.519050+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.519050+0000","last_undegraded":"2024-01-16T15:15:21.519050+0000","last_fullsized":"2024-01-16T15:15:21.519050+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005648+0000","last_change":"2024-01-16T15:15:22.536600+0000","last_active":"2024-01-16T15:15:24.005648+0000","last_peered":"2024-01-16T15:15:24.005648+0000","last_clean":"2024-01-16T15:15:24.005648+0000","last_became_active":"2024-01-16T15:15:22.536327+0000","last_became_peered":"2024-01-16T15:15:22.536327+0000","last_unstale":"2024-01-16T15:15:24.005648+0000","last_undegraded":"2024-01-16T15:15:24.005648+0000","last_fullsized":"2024-01-16T15:15:24.005648+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005398+0000","last_change":"2024-01-16T15:15:22.535575+0000","last_active":"2024-01-16T15:15:24.005398+0000","last_peered":"2024-01-16T15:15:24.005398+0000","last_clean":"2024-01-16T15:15:24.005398+0000","last_became_active":"2024-01-16T15:15:22.535292+0000","last_became_peered":"2024-01-16T15:15:22.535292+0000","last_unstale":"2024-01-16T15:15:24.005398+0000","last_undegraded":"2024-01-16T15:15:24.005398+0000","last_fullsized":"2024-01-16T15:15:24.005398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:06.443796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740110+0000","last_change":"2024-01-16T15:15:22.521459+0000","last_active":"2024-01-16T15:15:23.740110+0000","last_peered":"2024-01-16T15:15:23.740110+0000","last_clean":"2024-01-16T15:15:23.740110+0000","last_became_active":"2024-01-16T15:15:22.520884+0000","last_became_peered":"2024-01-16T15:15:22.520884+0000","last_unstale":"2024-01-16T15:15:23.740110+0000","last_undegraded":"2024-01-16T15:15:23.740110+0000","last_fullsized":"2024-01-16T15:15:23.740110+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.513402+0000","last_change":"2024-01-16T15:15:21.513402+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.513402+0000","last_undegraded":"2024-01-16T15:15:21.513402+0000","last_fullsized":"2024-01-16T15:15:21.513402+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.520840+0000","last_change":"2024-01-16T15:15:21.520840+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.520840+0000","last_undegraded":"2024-01-16T15:15:21.520840+0000","last_fullsized":"2024-01-16T15:15:21.520840+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005800+0000","last_change":"2024-01-16T15:15:22.533081+0000","last_active":"2024-01-16T15:15:24.005800+0000","last_peered":"2024-01-16T15:15:24.005800+0000","last_clean":"2024-01-16T15:15:24.005800+0000","last_became_active":"2024-01-16T15:15:22.529012+0000","last_became_peered":"2024-01-16T15:15:22.529012+0000","last_unstale":"2024-01-16T15:15:24.005800+0000","last_undegraded":"2024-01-16T15:15:24.005800+0000","last_fullsized":"2024-01-16T15:15:24.005800+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.510864+0000","last_change":"2024-01-16T15:15:21.510864+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.510864+0000","last_undegraded":"2024-01-16T15:15:21.510864+0000","last_fullsized":"2024-01-16T15:15:21.510864+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509163+0000","last_change":"2024-01-16T15:15:22.532050+0000","last_active":"2024-01-16T15:15:23.509163+0000","last_peered":"2024-01-16T15:15:23.509163+0000","last_clean":"2024-01-16T15:15:23.509163+0000","last_became_active":"2024-01-16T15:15:22.531307+0000","last_became_peered":"2024-01-16T15:15:22.531307+0000","last_unstale":"2024-01-16T15:15:23.509163+0000","last_undegraded":"2024-01-16T15:15:23.509163+0000","last_fullsized":"2024-01-16T15:15:23.509163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:34.860510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609381+0000","last_change":"2024-01-16T15:15:22.533322+0000","last_active":"2024-01-16T15:15:23.609381+0000","last_peered":"2024-01-16T15:15:23.609381+0000","last_clean":"2024-01-16T15:15:23.609381+0000","last_became_active":"2024-01-16T15:15:22.532960+0000","last_became_peered":"2024-01-16T15:15:22.532960+0000","last_unstale":"2024-01-16T15:15:23.609381+0000","last_undegraded":"2024-01-16T15:15:23.609381+0000","last_fullsized":"2024-01-16T15:15:23.609381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:47.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508639+0000","last_change":"2024-01-16T15:15:22.532776+0000","last_active":"2024-01-16T15:15:23.508639+0000","last_peered":"2024-01-16T15:15:23.508639+0000","last_clean":"2024-01-16T15:15:23.508639+0000","last_became_active":"2024-01-16T15:15:22.532494+0000","last_became_peered":"2024-01-16T15:15:22.532494+0000","last_unstale":"2024-01-16T15:15:23.508639+0000","last_undegraded":"2024-01-16T15:15:23.508639+0000","last_fullsized":"2024-01-16T15:15:23.508639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005108+0000","last_change":"2024-01-16T15:15:22.530744+0000","last_active":"2024-01-16T15:15:24.005108+0000","last_peered":"2024-01-16T15:15:24.005108+0000","last_clean":"2024-01-16T15:15:24.005108+0000","last_became_active":"2024-01-16T15:15:22.525764+0000","last_became_peered":"2024-01-16T15:15:22.525764+0000","last_unstale":"2024-01-16T15:15:24.005108+0000","last_undegraded":"2024-01-16T15:15:24.005108+0000","last_fullsized":"2024-01-16T15:15:24.005108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:51.206294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741415+0000","last_change":"2024-01-16T15:15:22.523512+0000","last_active":"2024-01-16T15:15:23.741415+0000","last_peered":"2024-01-16T15:15:23.741415+0000","last_clean":"2024-01-16T15:15:23.741415+0000","last_became_active":"2024-01-16T15:15:22.521121+0000","last_became_peered":"2024-01-16T15:15:22.521121+0000","last_unstale":"2024-01-16T15:15:23.741415+0000","last_undegraded":"2024-01-16T15:15:23.741415+0000","last_fullsized":"2024-01-16T15:15:23.741415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:35.046638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509378+0000","last_change":"2024-01-16T15:15:22.531166+0000","last_active":"2024-01-16T15:15:23.509378+0000","last_peered":"2024-01-16T15:15:23.509378+0000","last_clean":"2024-01-16T15:15:23.509378+0000","last_became_active":"2024-01-16T15:15:22.530595+0000","last_became_peered":"2024-01-16T15:15:22.530595+0000","last_unstale":"2024-01-16T15:15:23.509378+0000","last_undegraded":"2024-01-16T15:15:23.509378+0000","last_fullsized":"2024-01-16T15:15:23.509378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:01.070526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610445+0000","last_change":"2024-01-16T15:15:22.524237+0000","last_active":"2024-01-16T15:15:23.610445+0000","last_peered":"2024-01-16T15:15:23.610445+0000","last_clean":"2024-01-16T15:15:23.610445+0000","last_became_active":"2024-01-16T15:15:22.523982+0000","last_became_peered":"2024-01-16T15:15:22.523982+0000","last_unstale":"2024-01-16T15:15:23.610445+0000","last_undegraded":"2024-01-16T15:15:23.610445+0000","last_fullsized":"2024-01-16T15:15:23.610445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:34.668807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.508179+0000","last_change":"2024-01-16T15:15:21.508179+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.508179+0000","last_undegraded":"2024-01-16T15:15:21.508179+0000","last_fullsized":"2024-01-16T15:15:21.508179+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609784+0000","last_change":"2024-01-16T15:15:22.529446+0000","last_active":"2024-01-16T15:15:23.609784+0000","last_peered":"2024-01-16T15:15:23.609784+0000","last_clean":"2024-01-16T15:15:23.609784+0000","last_became_active":"2024-01-16T15:15:22.529124+0000","last_became_peered":"2024-01-16T15:15:22.529124+0000","last_unstale":"2024-01-16T15:15:23.609784+0000","last_undegraded":"2024-01-16T15:15:23.609784+0000","last_fullsized":"2024-01-16T15:15:23.609784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:14.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740836+0000","last_change":"2024-01-16T15:15:22.541357+0000","last_active":"2024-01-16T15:15:23.740836+0000","last_peered":"2024-01-16T15:15:23.740836+0000","last_clean":"2024-01-16T15:15:23.740836+0000","last_became_active":"2024-01-16T15:15:22.539470+0000","last_became_peered":"2024-01-16T15:15:22.539470+0000","last_unstale":"2024-01-16T15:15:23.740836+0000","last_undegraded":"2024-01-16T15:15:23.740836+0000","last_fullsized":"2024-01-16T15:15:23.740836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740324+0000","last_change":"2024-01-16T15:15:22.528944+0000","last_active":"2024-01-16T15:15:23.740324+0000","last_peered":"2024-01-16T15:15:23.740324+0000","last_clean":"2024-01-16T15:15:23.740324+0000","last_became_active":"2024-01-16T15:15:22.528668+0000","last_became_peered":"2024-01-16T15:15:22.528668+0000","last_unstale":"2024-01-16T15:15:23.740324+0000","last_undegraded":"2024-01-16T15:15:23.740324+0000","last_fullsized":"2024-01-16T15:15:23.740324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005396+0000","last_change":"2024-01-16T15:15:22.527598+0000","last_active":"2024-01-16T15:15:24.005396+0000","last_peered":"2024-01-16T15:15:24.005396+0000","last_clean":"2024-01-16T15:15:24.005396+0000","last_became_active":"2024-01-16T15:15:22.526241+0000","last_became_peered":"2024-01-16T15:15:22.526241+0000","last_unstale":"2024-01-16T15:15:24.005396+0000","last_undegraded":"2024-01-16T15:15:24.005396+0000","last_fullsized":"2024-01-16T15:15:24.005396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:41.199256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.522680+0000","last_change":"2024-01-16T15:15:21.522680+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.522680+0000","last_undegraded":"2024-01-16T15:15:21.522680+0000","last_fullsized":"2024-01-16T15:15:21.522680+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741073+0000","last_change":"2024-01-16T15:15:22.518878+0000","last_active":"2024-01-16T15:15:23.741073+0000","last_peered":"2024-01-16T15:15:23.741073+0000","last_clean":"2024-01-16T15:15:23.741073+0000","last_became_active":"2024-01-16T15:15:22.518594+0000","last_became_peered":"2024-01-16T15:15:22.518594+0000","last_unstale":"2024-01-16T15:15:23.741073+0000","last_undegraded":"2024-01-16T15:15:23.741073+0000","last_fullsized":"2024-01-16T15:15:23.741073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006005+0000","last_change":"2024-01-16T15:15:22.529986+0000","last_active":"2024-01-16T15:15:24.006005+0000","last_peered":"2024-01-16T15:15:24.006005+0000","last_clean":"2024-01-16T15:15:24.006005+0000","last_became_active":"2024-01-16T15:15:22.525092+0000","last_became_peered":"2024-01-16T15:15:22.525092+0000","last_unstale":"2024-01-16T15:15:24.006005+0000","last_undegraded":"2024-01-16T15:15:24.006005+0000","last_fullsized":"2024-01-16T15:15:24.006005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.522614+0000","last_change":"2024-01-16T15:15:21.522614+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.522614+0000","last_undegraded":"2024-01-16T15:15:21.522614+0000","last_fullsized":"2024-01-16T15:15:21.522614+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740772+0000","last_change":"2024-01-16T15:15:22.518786+0000","last_active":"2024-01-16T15:15:23.740772+0000","last_peered":"2024-01-16T15:15:23.740772+0000","last_clean":"2024-01-16T15:15:23.740772+0000","last_became_active":"2024-01-16T15:15:22.512965+0000","last_became_peered":"2024-01-16T15:15:22.512965+0000","last_unstale":"2024-01-16T15:15:23.740772+0000","last_undegraded":"2024-01-16T15:15:23.740772+0000","last_fullsized":"2024-01-16T15:15:23.740772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609350+0000","last_change":"2024-01-16T15:15:22.521137+0000","last_active":"2024-01-16T15:15:23.609350+0000","last_peered":"2024-01-16T15:15:23.609350+0000","last_clean":"2024-01-16T15:15:23.609350+0000","last_became_active":"2024-01-16T15:15:22.520842+0000","last_became_peered":"2024-01-16T15:15:22.520842+0000","last_unstale":"2024-01-16T15:15:23.609350+0000","last_undegraded":"2024-01-16T15:15:23.609350+0000","last_fullsized":"2024-01-16T15:15:23.609350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:02.829205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508652+0000","last_change":"2024-01-16T15:15:22.541257+0000","last_active":"2024-01-16T15:15:23.508652+0000","last_peered":"2024-01-16T15:15:23.508652+0000","last_clean":"2024-01-16T15:15:23.508652+0000","last_became_active":"2024-01-16T15:15:22.540754+0000","last_became_peered":"2024-01-16T15:15:22.540754+0000","last_unstale":"2024-01-16T15:15:23.508652+0000","last_undegraded":"2024-01-16T15:15:23.508652+0000","last_fullsized":"2024-01-16T15:15:23.508652+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:10.664582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690290+0000","last_change":"2024-01-16T15:15:22.516532+0000","last_active":"2024-01-16T15:15:23.690290+0000","last_peered":"2024-01-16T15:15:23.690290+0000","last_clean":"2024-01-16T15:15:23.690290+0000","last_became_active":"2024-01-16T15:15:22.515440+0000","last_became_peered":"2024-01-16T15:15:22.515440+0000","last_unstale":"2024-01-16T15:15:23.690290+0000","last_undegraded":"2024-01-16T15:15:23.690290+0000","last_fullsized":"2024-01-16T15:15:23.690290+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:42.093044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610359+0000","last_change":"2024-01-16T15:15:22.521967+0000","last_active":"2024-01-16T15:15:23.610359+0000","last_peered":"2024-01-16T15:15:23.610359+0000","last_clean":"2024-01-16T15:15:23.610359+0000","last_became_active":"2024-01-16T15:15:22.520663+0000","last_became_peered":"2024-01-16T15:15:22.520663+0000","last_unstale":"2024-01-16T15:15:23.610359+0000","last_undegraded":"2024-01-16T15:15:23.610359+0000","last_fullsized":"2024-01-16T15:15:23.610359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:27.652785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741472+0000","last_change":"2024-01-16T15:15:21.529234+0000","last_active":"2024-01-16T15:15:23.741472+0000","last_peered":"2024-01-16T15:15:23.741472+0000","last_clean":"2024-01-16T15:15:23.741472+0000","last_became_active":"2024-01-16T15:15:21.528711+0000","last_became_peered":"2024-01-16T15:15:21.528711+0000","last_unstale":"2024-01-16T15:15:23.741472+0000","last_undegraded":"2024-01-16T15:15:23.741472+0000","last_fullsized":"2024-01-16T15:15:23.741472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:34.638361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509939+0000","last_change":"2024-01-16T15:15:22.540312+0000","last_active":"2024-01-16T15:15:23.509939+0000","last_peered":"2024-01-16T15:15:23.509939+0000","last_clean":"2024-01-16T15:15:23.509939+0000","last_became_active":"2024-01-16T15:15:22.538619+0000","last_became_peered":"2024-01-16T15:15:22.538619+0000","last_unstale":"2024-01-16T15:15:23.509939+0000","last_undegraded":"2024-01-16T15:15:23.509939+0000","last_fullsized":"2024-01-16T15:15:23.509939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:02.521266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.542686+0000","last_change":"2024-01-16T15:15:21.516982+0000","last_active":"2024-01-16T15:15:22.542686+0000","last_peered":"2024-01-16T15:15:22.542686+0000","last_clean":"2024-01-16T15:15:22.542686+0000","last_became_active":"2024-01-16T15:15:21.516459+0000","last_became_peered":"2024-01-16T15:15:21.516459+0000","last_unstale":"2024-01-16T15:15:22.542686+0000","last_undegraded":"2024-01-16T15:15:22.542686+0000","last_fullsized":"2024-01-16T15:15:22.542686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690957+0000","last_change":"2024-01-16T15:15:22.522691+0000","last_active":"2024-01-16T15:15:23.690957+0000","last_peered":"2024-01-16T15:15:23.690957+0000","last_clean":"2024-01-16T15:15:23.690957+0000","last_became_active":"2024-01-16T15:15:22.522375+0000","last_became_peered":"2024-01-16T15:15:22.522375+0000","last_unstale":"2024-01-16T15:15:23.690957+0000","last_undegraded":"2024-01-16T15:15:23.690957+0000","last_fullsized":"2024-01-16T15:15:23.690957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:12.190032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740678+0000","last_change":"2024-01-16T15:15:21.529660+0000","last_active":"2024-01-16T15:15:23.740678+0000","last_peered":"2024-01-16T15:15:23.740678+0000","last_clean":"2024-01-16T15:15:23.740678+0000","last_became_active":"2024-01-16T15:15:21.529009+0000","last_became_peered":"2024-01-16T15:15:21.529009+0000","last_unstale":"2024-01-16T15:15:23.740678+0000","last_undegraded":"2024-01-16T15:15:23.740678+0000","last_fullsized":"2024-01-16T15:15:23.740678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:49.081309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.508270+0000","last_change":"2024-01-16T15:15:21.508270+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.508270+0000","last_undegraded":"2024-01-16T15:15:21.508270+0000","last_fullsized":"2024-01-16T15:15:21.508270+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740604+0000","last_change":"2024-01-16T15:15:21.521824+0000","last_active":"2024-01-16T15:15:23.740604+0000","last_peered":"2024-01-16T15:15:23.740604+0000","last_clean":"2024-01-16T15:15:23.740604+0000","last_became_active":"2024-01-16T15:15:21.521406+0000","last_became_peered":"2024-01-16T15:15:21.521406+0000","last_unstale":"2024-01-16T15:15:23.740604+0000","last_undegraded":"2024-01-16T15:15:23.740604+0000","last_fullsized":"2024-01-16T15:15:23.740604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:39.695221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004361+0000","last_change":"2024-01-16T15:15:22.540055+0000","last_active":"2024-01-16T15:15:24.004361+0000","last_peered":"2024-01-16T15:15:24.004361+0000","last_clean":"2024-01-16T15:15:24.004361+0000","last_became_active":"2024-01-16T15:15:22.538674+0000","last_became_peered":"2024-01-16T15:15:22.538674+0000","last_unstale":"2024-01-16T15:15:24.004361+0000","last_undegraded":"2024-01-16T15:15:24.004361+0000","last_fullsized":"2024-01-16T15:15:24.004361+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:37.034062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690792+0000","last_change":"2024-01-16T15:15:21.530799+0000","last_active":"2024-01-16T15:15:23.690792+0000","last_peered":"2024-01-16T15:15:23.690792+0000","last_clean":"2024-01-16T15:15:23.690792+0000","last_became_active":"2024-01-16T15:15:21.530451+0000","last_became_peered":"2024-01-16T15:15:21.530451+0000","last_unstale":"2024-01-16T15:15:23.690792+0000","last_undegraded":"2024-01-16T15:15:23.690792+0000","last_fullsized":"2024-01-16T15:15:23.690792+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:26.637253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005884+0000","last_change":"2024-01-16T15:15:22.533064+0000","last_active":"2024-01-16T15:15:24.005884+0000","last_peered":"2024-01-16T15:15:24.005884+0000","last_clean":"2024-01-16T15:15:24.005884+0000","last_became_active":"2024-01-16T15:15:22.532644+0000","last_became_peered":"2024-01-16T15:15:22.532644+0000","last_unstale":"2024-01-16T15:15:24.005884+0000","last_undegraded":"2024-01-16T15:15:24.005884+0000","last_fullsized":"2024-01-16T15:15:24.005884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740860+0000","last_change":"2024-01-16T15:15:21.525811+0000","last_active":"2024-01-16T15:15:23.740860+0000","last_peered":"2024-01-16T15:15:23.740860+0000","last_clean":"2024-01-16T15:15:23.740860+0000","last_became_active":"2024-01-16T15:15:21.525152+0000","last_became_peered":"2024-01-16T15:15:21.525152+0000","last_unstale":"2024-01-16T15:15:23.740860+0000","last_undegraded":"2024-01-16T15:15:23.740860+0000","last_fullsized":"2024-01-16T15:15:23.740860+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:38.101810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508468+0000","last_change":"2024-01-16T15:15:22.524402+0000","last_active":"2024-01-16T15:15:23.508468+0000","last_peered":"2024-01-16T15:15:23.508468+0000","last_clean":"2024-01-16T15:15:23.508468+0000","last_became_active":"2024-01-16T15:15:22.524129+0000","last_became_peered":"2024-01-16T15:15:22.524129+0000","last_unstale":"2024-01-16T15:15:23.508468+0000","last_undegraded":"2024-01-16T15:15:23.508468+0000","last_fullsized":"2024-01-16T15:15:23.508468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.533761+0000","last_change":"2024-01-16T15:15:21.532465+0000","last_active":"2024-01-16T15:15:21.533761+0000","last_peered":"2024-01-16T15:15:21.533761+0000","last_clean":"2024-01-16T15:15:21.533761+0000","last_became_active":"2024-01-16T15:15:21.532238+0000","last_became_peered":"2024-01-16T15:15:21.532238+0000","last_unstale":"2024-01-16T15:15:21.533761+0000","last_undegraded":"2024-01-16T15:15:21.533761+0000","last_fullsized":"2024-01-16T15:15:21.533761+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:49.415856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505378+0000","last_change":"2024-01-16T15:15:21.505378+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505378+0000","last_undegraded":"2024-01-16T15:15:21.505378+0000","last_fullsized":"2024-01-16T15:15:21.505378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690271+0000","last_change":"2024-01-16T15:15:21.524375+0000","last_active":"2024-01-16T15:15:23.690271+0000","last_peered":"2024-01-16T15:15:23.690271+0000","last_clean":"2024-01-16T15:15:23.690271+0000","last_became_active":"2024-01-16T15:15:21.523731+0000","last_became_peered":"2024-01-16T15:15:21.523731+0000","last_unstale":"2024-01-16T15:15:23.690271+0000","last_undegraded":"2024-01-16T15:15:23.690271+0000","last_fullsized":"2024-01-16T15:15:23.690271+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:13.884029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741391+0000","last_change":"2024-01-16T15:15:22.528073+0000","last_active":"2024-01-16T15:15:23.741391+0000","last_peered":"2024-01-16T15:15:23.741391+0000","last_clean":"2024-01-16T15:15:23.741391+0000","last_became_active":"2024-01-16T15:15:22.526190+0000","last_became_peered":"2024-01-16T15:15:22.526190+0000","last_unstale":"2024-01-16T15:15:23.741391+0000","last_undegraded":"2024-01-16T15:15:23.741391+0000","last_fullsized":"2024-01-16T15:15:23.741391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:55.442576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691500+0000","last_change":"2024-01-16T15:15:21.520386+0000","last_active":"2024-01-16T15:15:23.691500+0000","last_peered":"2024-01-16T15:15:23.691500+0000","last_clean":"2024-01-16T15:15:23.691500+0000","last_became_active":"2024-01-16T15:15:21.520059+0000","last_became_peered":"2024-01-16T15:15:21.520059+0000","last_unstale":"2024-01-16T15:15:23.691500+0000","last_undegraded":"2024-01-16T15:15:23.691500+0000","last_fullsized":"2024-01-16T15:15:23.691500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:33.653275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.504957+0000","last_change":"2024-01-16T15:15:21.504957+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.504957+0000","last_undegraded":"2024-01-16T15:15:21.504957+0000","last_fullsized":"2024-01-16T15:15:21.504957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509435+0000","last_change":"2024-01-16T15:15:21.534208+0000","last_active":"2024-01-16T15:15:23.509435+0000","last_peered":"2024-01-16T15:15:23.509435+0000","last_clean":"2024-01-16T15:15:23.509435+0000","last_became_active":"2024-01-16T15:15:21.533545+0000","last_became_peered":"2024-01-16T15:15:21.533545+0000","last_unstale":"2024-01-16T15:15:23.509435+0000","last_undegraded":"2024-01-16T15:15:23.509435+0000","last_fullsized":"2024-01-16T15:15:23.509435+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518791+0000","last_change":"2024-01-16T15:15:21.518791+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518791+0000","last_undegraded":"2024-01-16T15:15:21.518791+0000","last_fullsized":"2024-01-16T15:15:21.518791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.547699+0000","last_change":"2024-01-16T15:15:21.521285+0000","last_active":"2024-01-16T15:15:22.547699+0000","last_peered":"2024-01-16T15:15:22.547699+0000","last_clean":"2024-01-16T15:15:22.547699+0000","last_became_active":"2024-01-16T15:15:21.520717+0000","last_became_peered":"2024-01-16T15:15:21.520717+0000","last_unstale":"2024-01-16T15:15:22.547699+0000","last_undegraded":"2024-01-16T15:15:22.547699+0000","last_fullsized":"2024-01-16T15:15:22.547699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:15.674482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006046+0000","last_change":"2024-01-16T15:15:22.529757+0000","last_active":"2024-01-16T15:15:24.006046+0000","last_peered":"2024-01-16T15:15:24.006046+0000","last_clean":"2024-01-16T15:15:24.006046+0000","last_became_active":"2024-01-16T15:15:22.524620+0000","last_became_peered":"2024-01-16T15:15:22.524620+0000","last_unstale":"2024-01-16T15:15:24.006046+0000","last_undegraded":"2024-01-16T15:15:24.006046+0000","last_fullsized":"2024-01-16T15:15:24.006046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.520270+0000","last_change":"2024-01-16T15:15:21.518507+0000","last_active":"2024-01-16T15:15:21.520270+0000","last_peered":"2024-01-16T15:15:21.520270+0000","last_clean":"2024-01-16T15:15:21.520270+0000","last_became_active":"2024-01-16T15:15:21.518075+0000","last_became_peered":"2024-01-16T15:15:21.518075+0000","last_unstale":"2024-01-16T15:15:21.520270+0000","last_undegraded":"2024-01-16T15:15:21.520270+0000","last_fullsized":"2024-01-16T15:15:21.520270+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:25.234717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509009+0000","last_change":"2024-01-16T15:15:22.521859+0000","last_active":"2024-01-16T15:15:23.509009+0000","last_peered":"2024-01-16T15:15:23.509009+0000","last_clean":"2024-01-16T15:15:23.509009+0000","last_became_active":"2024-01-16T15:15:22.521561+0000","last_became_peered":"2024-01-16T15:15:22.521561+0000","last_unstale":"2024-01-16T15:15:23.509009+0000","last_undegraded":"2024-01-16T15:15:23.509009+0000","last_fullsized":"2024-01-16T15:15:23.509009+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:14.797953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609905+0000","last_change":"2024-01-16T15:15:21.521158+0000","last_active":"2024-01-16T15:15:23.609905+0000","last_peered":"2024-01-16T15:15:23.609905+0000","last_clean":"2024-01-16T15:15:23.609905+0000","last_became_active":"2024-01-16T15:15:21.520851+0000","last_became_peered":"2024-01-16T15:15:21.520851+0000","last_unstale":"2024-01-16T15:15:23.609905+0000","last_undegraded":"2024-01-16T15:15:23.609905+0000","last_fullsized":"2024-01-16T15:15:23.609905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:47.111965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690791+0000","last_change":"2024-01-16T15:15:22.530599+0000","last_active":"2024-01-16T15:15:23.690791+0000","last_peered":"2024-01-16T15:15:23.690791+0000","last_clean":"2024-01-16T15:15:23.690791+0000","last_became_active":"2024-01-16T15:15:22.526185+0000","last_became_peered":"2024-01-16T15:15:22.526185+0000","last_unstale":"2024-01-16T15:15:23.690791+0000","last_undegraded":"2024-01-16T15:15:23.690791+0000","last_fullsized":"2024-01-16T15:15:23.690791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:11.722014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609612+0000","last_change":"2024-01-16T15:15:21.526311+0000","last_active":"2024-01-16T15:15:23.609612+0000","last_peered":"2024-01-16T15:15:23.609612+0000","last_clean":"2024-01-16T15:15:23.609612+0000","last_became_active":"2024-01-16T15:15:21.526019+0000","last_became_peered":"2024-01-16T15:15:21.526019+0000","last_unstale":"2024-01-16T15:15:23.609612+0000","last_undegraded":"2024-01-16T15:15:23.609612+0000","last_fullsized":"2024-01-16T15:15:23.609612+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:30.739366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505003+0000","last_change":"2024-01-16T15:15:21.505003+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505003+0000","last_undegraded":"2024-01-16T15:15:21.505003+0000","last_fullsized":"2024-01-16T15:15:21.505003+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.530670+0000","last_change":"2024-01-16T15:15:21.528755+0000","last_active":"2024-01-16T15:15:21.530670+0000","last_peered":"2024-01-16T15:15:21.530670+0000","last_clean":"2024-01-16T15:15:21.530670+0000","last_became_active":"2024-01-16T15:15:21.528462+0000","last_became_peered":"2024-01-16T15:15:21.528462+0000","last_unstale":"2024-01-16T15:15:21.530670+0000","last_undegraded":"2024-01-16T15:15:21.530670+0000","last_fullsized":"2024-01-16T15:15:21.530670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:46.120683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609164+0000","last_change":"2024-01-16T15:15:22.529246+0000","last_active":"2024-01-16T15:15:23.609164+0000","last_peered":"2024-01-16T15:15:23.609164+0000","last_clean":"2024-01-16T15:15:23.609164+0000","last_became_active":"2024-01-16T15:15:22.528966+0000","last_became_peered":"2024-01-16T15:15:22.528966+0000","last_unstale":"2024-01-16T15:15:23.609164+0000","last_undegraded":"2024-01-16T15:15:23.609164+0000","last_fullsized":"2024-01-16T15:15:23.609164+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:46.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690545+0000","last_change":"2024-01-16T15:15:21.525650+0000","last_active":"2024-01-16T15:15:23.690545+0000","last_peered":"2024-01-16T15:15:23.690545+0000","last_clean":"2024-01-16T15:15:23.690545+0000","last_became_active":"2024-01-16T15:15:21.524160+0000","last_became_peered":"2024-01-16T15:15:21.524160+0000","last_unstale":"2024-01-16T15:15:23.690545+0000","last_undegraded":"2024-01-16T15:15:23.690545+0000","last_fullsized":"2024-01-16T15:15:23.690545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:28.946524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.521709+0000","last_change":"2024-01-16T15:15:21.521709+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.521709+0000","last_undegraded":"2024-01-16T15:15:21.521709+0000","last_fullsized":"2024-01-16T15:15:21.521709+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.520782+0000","last_change":"2024-01-16T15:15:21.520262+0000","last_active":"2024-01-16T15:15:21.520782+0000","last_peered":"2024-01-16T15:15:21.520782+0000","last_clean":"2024-01-16T15:15:21.520782+0000","last_became_active":"2024-01-16T15:15:21.519869+0000","last_became_peered":"2024-01-16T15:15:21.519869+0000","last_unstale":"2024-01-16T15:15:21.520782+0000","last_undegraded":"2024-01-16T15:15:21.520782+0000","last_fullsized":"2024-01-16T15:15:21.520782+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:13.821755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004344+0000","last_change":"2024-01-16T15:15:22.543578+0000","last_active":"2024-01-16T15:15:24.004344+0000","last_peered":"2024-01-16T15:15:24.004344+0000","last_clean":"2024-01-16T15:15:24.004344+0000","last_became_active":"2024-01-16T15:15:22.543283+0000","last_became_peered":"2024-01-16T15:15:22.543283+0000","last_unstale":"2024-01-16T15:15:24.004344+0000","last_undegraded":"2024-01-16T15:15:24.004344+0000","last_fullsized":"2024-01-16T15:15:24.004344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.530844+0000","last_change":"2024-01-16T15:15:21.528614+0000","last_active":"2024-01-16T15:15:21.530844+0000","last_peered":"2024-01-16T15:15:21.530844+0000","last_clean":"2024-01-16T15:15:21.530844+0000","last_became_active":"2024-01-16T15:15:21.528325+0000","last_became_peered":"2024-01-16T15:15:21.528325+0000","last_unstale":"2024-01-16T15:15:21.530844+0000","last_undegraded":"2024-01-16T15:15:21.530844+0000","last_fullsized":"2024-01-16T15:15:21.530844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:58.124662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007346+0000","last_change":"2024-01-16T15:15:22.530224+0000","last_active":"2024-01-16T15:15:24.007346+0000","last_peered":"2024-01-16T15:15:24.007346+0000","last_clean":"2024-01-16T15:15:24.007346+0000","last_became_active":"2024-01-16T15:15:22.528654+0000","last_became_peered":"2024-01-16T15:15:22.528654+0000","last_unstale":"2024-01-16T15:15:24.007346+0000","last_undegraded":"2024-01-16T15:15:24.007346+0000","last_fullsized":"2024-01-16T15:15:24.007346+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740462+0000","last_change":"2024-01-16T15:15:21.521658+0000","last_active":"2024-01-16T15:15:23.740462+0000","last_peered":"2024-01-16T15:15:23.740462+0000","last_clean":"2024-01-16T15:15:23.740462+0000","last_became_active":"2024-01-16T15:15:21.521051+0000","last_became_peered":"2024-01-16T15:15:21.521051+0000","last_unstale":"2024-01-16T15:15:23.740462+0000","last_undegraded":"2024-01-16T15:15:23.740462+0000","last_fullsized":"2024-01-16T15:15:23.740462+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609501+0000","last_change":"2024-01-16T15:15:22.525393+0000","last_active":"2024-01-16T15:15:23.609501+0000","last_peered":"2024-01-16T15:15:23.609501+0000","last_clean":"2024-01-16T15:15:23.609501+0000","last_became_active":"2024-01-16T15:15:22.524494+0000","last_became_peered":"2024-01-16T15:15:22.524494+0000","last_unstale":"2024-01-16T15:15:23.609501+0000","last_undegraded":"2024-01-16T15:15:23.609501+0000","last_fullsized":"2024-01-16T15:15:23.609501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:10.852002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740975+0000","last_change":"2024-01-16T15:15:21.531548+0000","last_active":"2024-01-16T15:15:23.740975+0000","last_peered":"2024-01-16T15:15:23.740975+0000","last_clean":"2024-01-16T15:15:23.740975+0000","last_became_active":"2024-01-16T15:15:21.531267+0000","last_became_peered":"2024-01-16T15:15:21.531267+0000","last_unstale":"2024-01-16T15:15:23.740975+0000","last_undegraded":"2024-01-16T15:15:23.740975+0000","last_fullsized":"2024-01-16T15:15:23.740975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:42.610622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609997+0000","last_change":"2024-01-16T15:15:22.522512+0000","last_active":"2024-01-16T15:15:23.609997+0000","last_peered":"2024-01-16T15:15:23.609997+0000","last_clean":"2024-01-16T15:15:23.609997+0000","last_became_active":"2024-01-16T15:15:22.522206+0000","last_became_peered":"2024-01-16T15:15:22.522206+0000","last_unstale":"2024-01-16T15:15:23.609997+0000","last_undegraded":"2024-01-16T15:15:23.609997+0000","last_fullsized":"2024-01-16T15:15:23.609997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:29.962056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004895+0000","last_change":"2024-01-16T15:15:21.524797+0000","last_active":"2024-01-16T15:15:24.004895+0000","last_peered":"2024-01-16T15:15:24.004895+0000","last_clean":"2024-01-16T15:15:24.004895+0000","last_became_active":"2024-01-16T15:15:21.524502+0000","last_became_peered":"2024-01-16T15:15:21.524502+0000","last_unstale":"2024-01-16T15:15:24.004895+0000","last_undegraded":"2024-01-16T15:15:24.004895+0000","last_fullsized":"2024-01-16T15:15:24.004895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:46.648168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609344+0000","last_change":"2024-01-16T15:15:22.523225+0000","last_active":"2024-01-16T15:15:23.609344+0000","last_peered":"2024-01-16T15:15:23.609344+0000","last_clean":"2024-01-16T15:15:23.609344+0000","last_became_active":"2024-01-16T15:15:22.522951+0000","last_became_peered":"2024-01-16T15:15:22.522951+0000","last_unstale":"2024-01-16T15:15:23.609344+0000","last_undegraded":"2024-01-16T15:15:23.609344+0000","last_fullsized":"2024-01-16T15:15:23.609344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:21.818020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004192+0000","last_change":"2024-01-16T15:15:21.519991+0000","last_active":"2024-01-16T15:15:24.004192+0000","last_peered":"2024-01-16T15:15:24.004192+0000","last_clean":"2024-01-16T15:15:24.004192+0000","last_became_active":"2024-01-16T15:15:21.519565+0000","last_became_peered":"2024-01-16T15:15:21.519565+0000","last_unstale":"2024-01-16T15:15:24.004192+0000","last_undegraded":"2024-01-16T15:15:24.004192+0000","last_fullsized":"2024-01-16T15:15:24.004192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:50.723441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518797+0000","last_change":"2024-01-16T15:15:21.518797+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518797+0000","last_undegraded":"2024-01-16T15:15:21.518797+0000","last_fullsized":"2024-01-16T15:15:21.518797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740258+0000","last_change":"2024-01-16T15:15:21.519702+0000","last_active":"2024-01-16T15:15:23.740258+0000","last_peered":"2024-01-16T15:15:23.740258+0000","last_clean":"2024-01-16T15:15:23.740258+0000","last_became_active":"2024-01-16T15:15:21.518926+0000","last_became_peered":"2024-01-16T15:15:21.518926+0000","last_unstale":"2024-01-16T15:15:23.740258+0000","last_undegraded":"2024-01-16T15:15:23.740258+0000","last_fullsized":"2024-01-16T15:15:23.740258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:31.707777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741243+0000","last_change":"2024-01-16T15:15:22.536514+0000","last_active":"2024-01-16T15:15:23.741243+0000","last_peered":"2024-01-16T15:15:23.741243+0000","last_clean":"2024-01-16T15:15:23.741243+0000","last_became_active":"2024-01-16T15:15:22.536248+0000","last_became_peered":"2024-01-16T15:15:22.536248+0000","last_unstale":"2024-01-16T15:15:23.741243+0000","last_undegraded":"2024-01-16T15:15:23.741243+0000","last_fullsized":"2024-01-16T15:15:23.741243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:12:10.024161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005379+0000","last_change":"2024-01-16T15:15:21.522416+0000","last_active":"2024-01-16T15:15:24.005379+0000","last_peered":"2024-01-16T15:15:24.005379+0000","last_clean":"2024-01-16T15:15:24.005379+0000","last_became_active":"2024-01-16T15:15:21.522118+0000","last_became_peered":"2024-01-16T15:15:21.522118+0000","last_unstale":"2024-01-16T15:15:24.005379+0000","last_undegraded":"2024-01-16T15:15:24.005379+0000","last_fullsized":"2024-01-16T15:15:24.005379+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.762652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691386+0000","last_change":"2024-01-16T15:15:22.531377+0000","last_active":"2024-01-16T15:15:23.691386+0000","last_peered":"2024-01-16T15:15:23.691386+0000","last_clean":"2024-01-16T15:15:23.691386+0000","last_became_active":"2024-01-16T15:15:22.529798+0000","last_became_peered":"2024-01-16T15:15:22.529798+0000","last_unstale":"2024-01-16T15:15:23.691386+0000","last_undegraded":"2024-01-16T15:15:23.691386+0000","last_fullsized":"2024-01-16T15:15:23.691386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.721245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741334+0000","last_change":"2024-01-16T15:15:21.532318+0000","last_active":"2024-01-16T15:15:23.741334+0000","last_peered":"2024-01-16T15:15:23.741334+0000","last_clean":"2024-01-16T15:15:23.741334+0000","last_became_active":"2024-01-16T15:15:21.531869+0000","last_became_peered":"2024-01-16T15:15:21.531869+0000","last_unstale":"2024-01-16T15:15:23.741334+0000","last_undegraded":"2024-01-16T15:15:23.741334+0000","last_fullsized":"2024-01-16T15:15:23.741334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:40.183724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690994+0000","last_change":"2024-01-16T15:15:22.535293+0000","last_active":"2024-01-16T15:15:23.690994+0000","last_peered":"2024-01-16T15:15:23.690994+0000","last_clean":"2024-01-16T15:15:23.690994+0000","last_became_active":"2024-01-16T15:15:22.534929+0000","last_became_peered":"2024-01-16T15:15:22.534929+0000","last_unstale":"2024-01-16T15:15:23.690994+0000","last_undegraded":"2024-01-16T15:15:23.690994+0000","last_fullsized":"2024-01-16T15:15:23.690994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.606169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740931+0000","last_change":"2024-01-16T15:15:21.520890+0000","last_active":"2024-01-16T15:15:23.740931+0000","last_peered":"2024-01-16T15:15:23.740931+0000","last_clean":"2024-01-16T15:15:23.740931+0000","last_became_active":"2024-01-16T15:15:21.520488+0000","last_became_peered":"2024-01-16T15:15:21.520488+0000","last_unstale":"2024-01-16T15:15:23.740931+0000","last_undegraded":"2024-01-16T15:15:23.740931+0000","last_fullsized":"2024-01-16T15:15:23.740931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:08.456302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.521585+0000","last_change":"2024-01-16T15:15:21.521585+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.521585+0000","last_undegraded":"2024-01-16T15:15:21.521585+0000","last_fullsized":"2024-01-16T15:15:21.521585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.528795+0000","last_change":"2024-01-16T15:15:21.527345+0000","last_active":"2024-01-16T15:15:21.528795+0000","last_peered":"2024-01-16T15:15:21.528795+0000","last_clean":"2024-01-16T15:15:21.528795+0000","last_became_active":"2024-01-16T15:15:21.527018+0000","last_became_peered":"2024-01-16T15:15:21.527018+0000","last_unstale":"2024-01-16T15:15:21.528795+0000","last_undegraded":"2024-01-16T15:15:21.528795+0000","last_fullsized":"2024-01-16T15:15:21.528795+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:59.366965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609781+0000","last_change":"2024-01-16T15:15:22.525600+0000","last_active":"2024-01-16T15:15:23.609781+0000","last_peered":"2024-01-16T15:15:23.609781+0000","last_clean":"2024-01-16T15:15:23.609781+0000","last_became_active":"2024-01-16T15:15:22.525112+0000","last_became_peered":"2024-01-16T15:15:22.525112+0000","last_unstale":"2024-01-16T15:15:23.609781+0000","last_undegraded":"2024-01-16T15:15:23.609781+0000","last_fullsized":"2024-01-16T15:15:23.609781+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:49.965076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741004+0000","last_change":"2024-01-16T15:15:21.526060+0000","last_active":"2024-01-16T15:15:23.741004+0000","last_peered":"2024-01-16T15:15:23.741004+0000","last_clean":"2024-01-16T15:15:23.741004+0000","last_became_active":"2024-01-16T15:15:21.525792+0000","last_became_peered":"2024-01-16T15:15:21.525792+0000","last_unstale":"2024-01-16T15:15:23.741004+0000","last_undegraded":"2024-01-16T15:15:23.741004+0000","last_fullsized":"2024-01-16T15:15:23.741004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505027+0000","last_change":"2024-01-16T15:15:21.505027+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505027+0000","last_undegraded":"2024-01-16T15:15:21.505027+0000","last_fullsized":"2024-01-16T15:15:21.505027+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.533268+0000","last_change":"2024-01-16T15:15:21.530401+0000","last_active":"2024-01-16T15:15:21.533268+0000","last_peered":"2024-01-16T15:15:21.533268+0000","last_clean":"2024-01-16T15:15:21.533268+0000","last_became_active":"2024-01-16T15:15:21.530083+0000","last_became_peered":"2024-01-16T15:15:21.530083+0000","last_unstale":"2024-01-16T15:15:21.533268+0000","last_undegraded":"2024-01-16T15:15:21.533268+0000","last_fullsized":"2024-01-16T15:15:21.533268+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:56.829359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609251+0000","last_change":"2024-01-16T15:15:22.530971+0000","last_active":"2024-01-16T15:15:23.609251+0000","last_peered":"2024-01-16T15:15:23.609251+0000","last_clean":"2024-01-16T15:15:23.609251+0000","last_became_active":"2024-01-16T15:15:22.530694+0000","last_became_peered":"2024-01-16T15:15:22.530694+0000","last_unstale":"2024-01-16T15:15:23.609251+0000","last_undegraded":"2024-01-16T15:15:23.609251+0000","last_fullsized":"2024-01-16T15:15:23.609251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:02.926295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508734+0000","last_change":"2024-01-16T15:15:21.524472+0000","last_active":"2024-01-16T15:15:23.508734+0000","last_peered":"2024-01-16T15:15:23.508734+0000","last_clean":"2024-01-16T15:15:23.508734+0000","last_became_active":"2024-01-16T15:15:21.524175+0000","last_became_peered":"2024-01-16T15:15:21.524175+0000","last_unstale":"2024-01-16T15:15:23.508734+0000","last_undegraded":"2024-01-16T15:15:23.508734+0000","last_fullsized":"2024-01-16T15:15:23.508734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:43.337353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518265+0000","last_change":"2024-01-16T15:15:21.518265+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518265+0000","last_undegraded":"2024-01-16T15:15:21.518265+0000","last_fullsized":"2024-01-16T15:15:21.518265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609465+0000","last_change":"2024-01-16T15:15:21.533877+0000","last_active":"2024-01-16T15:15:23.609465+0000","last_peered":"2024-01-16T15:15:23.609465+0000","last_clean":"2024-01-16T15:15:23.609465+0000","last_became_active":"2024-01-16T15:15:21.533489+0000","last_became_peered":"2024-01-16T15:15:21.533489+0000","last_unstale":"2024-01-16T15:15:23.609465+0000","last_undegraded":"2024-01-16T15:15:23.609465+0000","last_fullsized":"2024-01-16T15:15:23.609465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:15.404651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610593+0000","last_change":"2024-01-16T15:15:22.521850+0000","last_active":"2024-01-16T15:15:23.610593+0000","last_peered":"2024-01-16T15:15:23.610593+0000","last_clean":"2024-01-16T15:15:23.610593+0000","last_became_active":"2024-01-16T15:15:22.521581+0000","last_became_peered":"2024-01-16T15:15:22.521581+0000","last_unstale":"2024-01-16T15:15:23.610593+0000","last_undegraded":"2024-01-16T15:15:23.610593+0000","last_fullsized":"2024-01-16T15:15:23.610593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:29.487154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.545734+0000","last_change":"2024-01-16T15:15:21.522571+0000","last_active":"2024-01-16T15:15:22.545734+0000","last_peered":"2024-01-16T15:15:22.545734+0000","last_clean":"2024-01-16T15:15:22.545734+0000","last_became_active":"2024-01-16T15:15:21.522205+0000","last_became_peered":"2024-01-16T15:15:21.522205+0000","last_unstale":"2024-01-16T15:15:22.545734+0000","last_undegraded":"2024-01-16T15:15:22.545734+0000","last_fullsized":"2024-01-16T15:15:22.545734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":241,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516932+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:15:22.516932+0000","last_peered":"2024-01-16T15:15:22.516932+0000","last_clean":"2024-01-16T15:15:22.516932+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:15:22.516932+0000","last_undegraded":"2024-01-16T15:15:22.516932+0000","last_fullsized":"2024-01-16T15:15:22.516932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509629,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68364,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303476,"statfs":{"total":96636764160,"available":96566759424,"internally_reserved":0,"allocated":1179648,"data_stored":505985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3140000000000001,"15min":1.615},"min":{"1min":0.75,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":2.5030000000000001,"5min":8.8070000000000004,"15min":27.853999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.663,"15min":1.708},"min":{"1min":0.93000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6399999999999999,"5min":28.603999999999999,"15min":34.939999999999998},"last":0.875}]},{"osd":1,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.292,"15min":1.4830000000000001},"min":{"1min":0.94899999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.528,"5min":8.2650000000000006,"15min":13.859},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.754,"15min":1.651},"min":{"1min":0.85599999999999998,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":2.702,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.2589999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4330000000000001,"15min":1.5860000000000001},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":2.6230000000000002,"5min":9.6679999999999993,"15min":13.050000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.69499999999999995,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":2.6309999999999998,"5min":6.782,"15min":35.011000000000003},"last":1.3100000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.486,"15min":1.657},"min":{"1min":0.94799999999999995,"5min":0.439,"15min":0.374},"max":{"1min":1.8720000000000001,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.7170000000000001},"min":{"1min":0.999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.758,"5min":9.7780000000000005,"15min":14.273999999999999},"last":1.1519999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3919999999999999,"15min":1.827},"min":{"1min":0.90900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7090000000000001,"5min":10.952999999999999,"15min":34.972999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.4319999999999999,"15min":1.798},"min":{"1min":0.85899999999999999,"5min":0.58299999999999996,"15min":0.40400000000000003},"max":{"1min":1.6519999999999999,"5min":5.2300000000000004,"15min":16.003},"last":1.411}]},{"osd":5,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.591,"15min":2.0099999999999998},"min":{"1min":1.1339999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.798,"5min":4.4550000000000001,"15min":23.760999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6259999999999999,"15min":1.9059999999999999},"min":{"1min":0.96299999999999997,"5min":0.70699999999999996,"15min":0.52400000000000002},"max":{"1min":3.496,"5min":9.4120000000000008,"15min":23.716999999999999},"last":1.363}]},{"osd":7,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.5069999999999999,"15min":2.0430000000000001},"min":{"1min":0.90700000000000003,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":6.9219999999999997,"15min":34.889000000000003},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.79,"15min":2.1760000000000002},"min":{"1min":1.2470000000000001,"5min":0.65100000000000002,"15min":0.47599999999999998},"max":{"1min":2.0499999999999998,"5min":23.242999999999999,"15min":46.305},"last":1.548}]}]},{"osd":4,"up_from":15,"seq":64424509631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68380,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303460,"statfs":{"total":96636764160,"available":96566743040,"internally_reserved":0,"allocated":1191936,"data_stored":512288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.226,"15min":1.55},"min":{"1min":0.76900000000000002,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.1399999999999997,"15min":22.757999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.125,"15min":1.2729999999999999},"min":{"1min":0.69799999999999995,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.474,"5min":4.0270000000000001,"15min":14.034000000000001},"last":1.623}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.413,"15min":1.577},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.851,"5min":17.811,"15min":18.347000000000001},"last":1.002},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3140000000000001,"15min":1.49},"min":{"1min":0.90700000000000003,"5min":0.59799999999999998,"15min":0.498},"max":{"1min":1.5600000000000001,"5min":4.0449999999999999,"15min":18.858000000000001},"last":1.6679999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.544,"15min":1.679},"min":{"1min":0.83799999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.768,"5min":17.651,"15min":17.651},"last":1.204},{"interface":"front","average":{"1min":1.327,"5min":1.754,"15min":1.722},"min":{"1min":0.83499999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8180000000000001,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.77}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3320000000000001,"15min":1.554},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":1.6659999999999999,"5min":3.972,"15min":26.741},"last":1.458},{"interface":"front","average":{"1min":1.284,"5min":1.571,"15min":1.7929999999999999},"min":{"1min":0.86899999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.762,"5min":17.715,"15min":26.701000000000001},"last":1.3149999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5660000000000001,"15min":1.7370000000000001},"min":{"1min":1.0029999999999999,"5min":0.51400000000000001,"15min":0.441},"max":{"1min":1.988,"5min":5.3170000000000002,"15min":18.919},"last":1.367},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5029999999999999,"15min":1.6759999999999999},"min":{"1min":0.95599999999999996,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":2.0270000000000001,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.8280000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":2.0739999999999998,"15min":1.9419999999999999},"min":{"1min":0.995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.0750000000000002,"5min":32.649999999999999,"15min":32.649999999999999},"last":1.514},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5469999999999999,"15min":1.823},"min":{"1min":1.1799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0630000000000002,"5min":4.3129999999999997,"15min":24.094999999999999},"last":1.55}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.6890000000000001,"15min":1.7649999999999999},"min":{"1min":1.056,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":1.8899999999999999,"5min":17.446000000000002,"15min":26.683},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.871,"15min":1.9650000000000001},"min":{"1min":1.248,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9350000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":1.407}]}]},{"osd":7,"up_from":14,"seq":60129542334,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68988,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302852,"statfs":{"total":96636764160,"available":96566120448,"internally_reserved":0,"allocated":1818624,"data_stored":1115899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2649999999999999,"15min":1.5089999999999999},"min":{"1min":0.68100000000000005,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.7310000000000001,"5min":4.3680000000000003,"15min":15.661},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.3240000000000001,"15min":1.5449999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":2.1080000000000001,"5min":12.461,"15min":30.126000000000001},"last":1.226}]},{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.423,"15min":1.4530000000000001},"min":{"1min":0.63200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.085,"5min":20.327000000000002,"15min":24.338000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.395,"15min":1.6180000000000001},"min":{"1min":0.81399999999999995,"5min":0.77400000000000002,"15min":0.46200000000000002},"max":{"1min":1.6419999999999999,"5min":12.526,"15min":21.298999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.57199999999999995,"15min":0.51500000000000001},"max":{"1min":1.9219999999999999,"5min":4.3380000000000001,"15min":12.474},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.702,"15min":1.651},"min":{"1min":0.69299999999999995,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8999999999999999,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4039999999999999,"15min":1.6020000000000001},"min":{"1min":0.88800000000000001,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.7150000000000001,"5min":4.2530000000000001,"15min":12.58},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.9430000000000001,"15min":1.7869999999999999},"min":{"1min":0.78600000000000003,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":2.4630000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3520000000000001,"15min":1.591},"min":{"1min":0.99099999999999999,"5min":0.54100000000000004,"15min":0.42799999999999999},"max":{"1min":2.0219999999999998,"5min":4.1390000000000002,"15min":14.659000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.831,"15min":1.825},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":2.028,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.3600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.486,"15min":1.8360000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.6459999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.113},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.8400000000000001},"min":{"1min":1.1870000000000001,"5min":0.66000000000000003,"15min":0.41099999999999998},"max":{"1min":1.8400000000000001,"5min":7.4649999999999999,"15min":21.312000000000001},"last":1.383}]},{"osd":6,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.9790000000000001,"15min":1.8640000000000001},"min":{"1min":0.97199999999999998,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":2.0880000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.597,"15min":1.956},"min":{"1min":0.81699999999999995,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":1.8300000000000001,"5min":12.567,"15min":29.991},"last":1.3300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542335,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76728,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295112,"statfs":{"total":96636764160,"available":96558194688,"internally_reserved":0,"allocated":1220608,"data_stored":525489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.405,"15min":1.5389999999999999},"min":{"1min":0.79700000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.6579999999999999,"5min":6.8380000000000001,"15min":19.382999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.3740000000000001,"15min":1.3899999999999999},"min":{"1min":0.77000000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":13.127000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":1.008}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.5409999999999999,"15min":1.6339999999999999},"min":{"1min":0.70699999999999996,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":3.8109999999999999,"5min":13.571,"15min":17.693999999999999},"last":1.214},{"interface":"front","average":{"1min":1.996,"5min":1.6419999999999999,"15min":1.6950000000000001},"min":{"1min":0.95099999999999996,"5min":0.51600000000000001,"15min":0.432},"max":{"1min":17.062000000000001,"5min":17.062000000000001,"15min":20.645},"last":1.395}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.5740000000000001,"15min":1.6879999999999999},"min":{"1min":0.79100000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.77,"5min":6.4039999999999999,"15min":21.138000000000002},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.462,"15min":1.4910000000000001},"min":{"1min":0.995,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":4.0720000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.5349999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4930000000000001,"15min":1.8},"min":{"1min":0.95499999999999996,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":2.4769999999999999,"5min":7.984,"15min":23.109000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.6060000000000001,"15min":1.7949999999999999},"min":{"1min":1.0169999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.95,"5min":6.4610000000000003,"15min":19.303000000000001},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5449999999999999,"15min":1.849},"min":{"1min":0.85799999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":9.3309999999999995,"5min":10.831,"15min":24.436},"last":1.617},{"interface":"front","average":{"1min":1.329,"5min":1.48,"15min":1.744},"min":{"1min":0.84699999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":1.6140000000000001,"5min":5.6820000000000004,"15min":19.126000000000001},"last":1.5800000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.526,"15min":1.635},"min":{"1min":0.95599999999999996,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":1.718,"5min":12.914,"15min":21.192},"last":1.341},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5609999999999999,"15min":1.978},"min":{"1min":0.97299999999999998,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":2.0299999999999998,"5min":7.8609999999999998,"15min":27.042000000000002},"last":1.764}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7989999999999999,"15min":1.9350000000000001},"min":{"1min":0.92600000000000005,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":13.034000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":1.663},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.7410000000000001,"15min":2.0840000000000001},"min":{"1min":1.216,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":2.1429999999999998,"5min":10.756,"15min":24.367000000000001},"last":1.8440000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542335,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":527053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.5,"15min":1.278},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":1.403,"5min":28.437999999999999,"15min":31.562999999999999},"last":1.238},{"interface":"front","average":{"1min":1.117,"5min":1.26,"15min":1.173},"min":{"1min":0.65900000000000003,"5min":0.40600000000000003,"15min":0.39600000000000002},"max":{"1min":1.8440000000000001,"5min":11.361000000000001,"15min":11.361000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.236,"15min":1.145},"min":{"1min":0.59999999999999998,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.5940000000000001,"5min":10.856999999999999,"15min":15.569000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.7050000000000001,"15min":1.429},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.643,"5min":28.352,"15min":28.352},"last":1.103}]},{"osd":2,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":2.379,"15min":1.6879999999999999},"min":{"1min":0.71999999999999997,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":1.8109999999999999,"5min":39.265999999999998,"15min":39.265999999999998},"last":1.351},{"interface":"front","average":{"1min":1.264,"5min":1.4970000000000001,"15min":1.385},"min":{"1min":0.67800000000000005,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":3.911,"5min":13.526,"15min":14.577999999999999},"last":1.1679999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.516,"15min":1.5620000000000001},"min":{"1min":0.86499999999999999,"5min":0.64900000000000002,"15min":0.53800000000000003},"max":{"1min":2.1019999999999999,"5min":6.0629999999999997,"15min":20.978999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.448,"5min":2.2879999999999998,"15min":1.8480000000000001},"min":{"1min":1.034,"5min":0.93100000000000005,"15min":0.72999999999999998},"max":{"1min":2.0030000000000001,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.472}]},{"osd":5,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.746,"15min":1.766},"min":{"1min":0.81699999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":1.911,"5min":10.988,"15min":20.663},"last":1.512},{"interface":"front","average":{"1min":1.575,"5min":2.3090000000000002,"15min":1.9490000000000001},"min":{"1min":0.96299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1019999999999999,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.637}]},{"osd":6,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.77,"15min":1.754},"min":{"1min":1.0069999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.2170000000000001,"5min":13.691000000000001,"15min":13.691000000000001},"last":1.321},{"interface":"front","average":{"1min":2.0910000000000002,"5min":2.0819999999999999,"15min":1.9510000000000001},"min":{"1min":0.95099999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":15.114000000000001,"5min":26.698,"15min":26.698},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.905,"15min":1.776},"min":{"1min":1.0589999999999999,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.9630000000000001,"5min":13.622,"15min":13.622},"last":1.431},{"interface":"front","average":{"1min":1.5289999999999999,"5min":2.0459999999999998,"15min":2.1659999999999999},"min":{"1min":0.90400000000000003,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.2789999999999999,"5min":15.372999999999999,"15min":49.811},"last":1.671}]}]},{"osd":2,"up_from":14,"seq":60129542334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76732,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295108,"statfs":{"total":96636764160,"available":96558190592,"internally_reserved":0,"allocated":1220608,"data_stored":525453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.284,"15min":1.0449999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.7130000000000001,"5min":18.289999999999999,"15min":18.289999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.2210000000000001,"15min":1.099},"min":{"1min":0.65100000000000002,"5min":0.40699999999999997,"15min":0.28599999999999998},"max":{"1min":1.6000000000000001,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.4650000000000001,"15min":1.1850000000000001},"min":{"1min":0.55900000000000005,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":1.8979999999999999,"5min":23.135999999999999,"15min":23.135999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.278,"15min":1.2050000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.6699999999999999,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":1.016}]},{"osd":3,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3859999999999999,"15min":1.1839999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":7.306,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.421,"5min":1.647,"15min":1.4510000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.839,"5min":19.375,"15min":19.375},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.8140000000000001,"15min":1.5620000000000001},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.59999999999999998},"max":{"1min":2.0249999999999999,"5min":23.225000000000001,"15min":23.225000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.651,"15min":1.5600000000000001},"min":{"1min":1.036,"5min":0.72699999999999998,"15min":0.59199999999999997},"max":{"1min":1.7509999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.335}]},{"osd":5,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.8580000000000001,"15min":1.6200000000000001},"min":{"1min":1.0740000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":7.7640000000000002,"5min":20.157,"15min":20.157},"last":1.2},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8089999999999999,"15min":1.601},"min":{"1min":0.84299999999999997,"5min":0.623,"15min":0.623},"max":{"1min":7.9820000000000002,"5min":25.004000000000001,"15min":25.004000000000001},"last":1.7090000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":2.1040000000000001,"15min":1.7390000000000001},"min":{"1min":0.89400000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":7.6630000000000003,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.948,"15min":1.839},"min":{"1min":0.81100000000000005,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":7.8680000000000003,"5min":28.760999999999999,"15min":28.760999999999999},"last":1.49}]},{"osd":7,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":2.2919999999999998,"15min":1.772},"min":{"1min":1.05,"5min":0.82099999999999995,"15min":0.64100000000000001},"max":{"1min":3.1219999999999999,"5min":22.834,"15min":22.834},"last":1.554},{"interface":"front","average":{"1min":1.53,"5min":2.2879999999999998,"15min":2.069},"min":{"1min":0.91000000000000003,"5min":0.872,"15min":0.82799999999999996},"max":{"1min":1.931,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.4279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72560,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299280,"statfs":{"total":96636764160,"available":96562462720,"internally_reserved":0,"allocated":1216512,"data_stored":525419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.9850000000000001,"15min":1.3260000000000001},"min":{"1min":0.48599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.256,"5min":31.178999999999998,"15min":31.178999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.4470000000000001,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.901,"5min":26.859999999999999,"15min":26.859999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1439999999999999,"15min":1.143},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":3.5830000000000002,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.9239999999999999,"15min":1.5529999999999999},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":3.4929999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":0.84599999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.879,"15min":1.651},"min":{"1min":0.56299999999999994,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.8049999999999999,"5min":27.609999999999999,"15min":27.609999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.1709999999999998,"15min":1.6559999999999999},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":2.9750000000000001,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.3069999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4390000000000001,"15min":1.466},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.58099999999999996},"max":{"1min":2.3069999999999999,"5min":4.7249999999999996,"15min":10.577999999999999},"last":1.46},{"interface":"front","average":{"1min":1.411,"5min":2.0760000000000001,"15min":1.74},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.0310000000000001,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.411,"15min":1.5449999999999999},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.032,"5min":3.032,"15min":17.469000000000001},"last":1.508},{"interface":"front","average":{"1min":1.377,"5min":1.522,"15min":1.6060000000000001},"min":{"1min":1.0229999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1859999999999999,"5min":4.6120000000000001,"15min":8.2989999999999995},"last":1.077}]},{"osd":6,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5569999999999999,"15min":1.7150000000000001},"min":{"1min":0.85999999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.9009999999999998,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.611},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5549999999999999,"15min":1.873},"min":{"1min":1.0800000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5760000000000001,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.2669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6419999999999999,"15min":1.899},"min":{"1min":0.94799999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4249999999999998,"5min":4.6989999999999998,"15min":30.834},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.2589999999999999,"15min":2.0609999999999999},"min":{"1min":0.96799999999999997,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":1.9890000000000001,"5min":27.716999999999999,"15min":27.716999999999999},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542334,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73152,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298688,"statfs":{"total":96636764160,"available":96561856512,"internally_reserved":0,"allocated":1814528,"data_stored":1116369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.103,"15min":1.1879999999999999},"min":{"1min":0.52100000000000002,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.73,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.353,"15min":1.4970000000000001},"min":{"1min":0.82299999999999995,"5min":0.42499999999999999,"15min":0.33900000000000002},"max":{"1min":1.623,"5min":26.459,"15min":50.429000000000002},"last":0.93500000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":0.66100000000000003,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.8009999999999999,"5min":20.041,"15min":22.716999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.452,"15min":1.6160000000000001},"min":{"1min":0.78400000000000003,"5min":0.503,"15min":0.38},"max":{"1min":1.7430000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":1.252}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2410000000000001,"15min":1.218},"min":{"1min":0.749,"5min":0.40000000000000002,"15min":0.372},"max":{"1min":1.8129999999999999,"5min":5.8719999999999999,"15min":10.795},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.425,"15min":1.5820000000000001},"min":{"1min":0.70799999999999996,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.8280000000000001,"5min":9.484,"15min":22.254999999999999},"last":1.462}]},{"osd":4,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5,"15min":1.806},"min":{"1min":0.78800000000000003,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.986,"5min":3.7749999999999999,"15min":22.407},"last":1.536},{"interface":"front","average":{"1min":1.48,"5min":1.6930000000000001,"15min":1.919},"min":{"1min":1.1970000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.907,"5min":26.57,"15min":50.497},"last":1.365}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.401,"15min":1.6220000000000001},"min":{"1min":0.83199999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.0550000000000002,"5min":4.7359999999999998,"15min":14.112},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.7,"15min":1.847},"min":{"1min":1.05,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.9430000000000001,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.5920000000000001,"15min":1.7769999999999999},"min":{"1min":0.871,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.968,"5min":8.2349999999999994,"15min":31.387},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.841,"15min":1.9650000000000001},"min":{"1min":1.01,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":3.5209999999999999,"5min":9.2840000000000007,"15min":23.547000000000001},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":2.0190000000000001,"15min":2.0680000000000001},"min":{"1min":1.292,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.21,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.679},{"interface":"front","average":{"1min":1.639,"5min":1.8660000000000001,"15min":1.835},"min":{"1min":1.0349999999999999,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.1859999999999999,"5min":12.455,"15min":15.747},"last":1.5760000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:26.985 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:27.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.267+0000 7fc08be00640 1 Processor -- start 2024-01-16T15:15:27.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.267+0000 7fc08be00640 1 -- start start 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc08be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc08c143c40 0x7fc08c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc08be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc08c0ccc10 0x7fc08c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc08be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc08c070e50 con 0x7fc08c0717d0 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc08be00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c0723c0 con 0x7fc08c0ccc10 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc08be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc08c144020 con 0x7fc08c143c40 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc083e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc08c0717d0 0x7fc08c0cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55546/0 (socket says 172.21.15.53:55546) 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc08b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc08c0ccc10 0x7fc08c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.269+0000 7fc083e00640 1 -- 172.21.15.53:0/2313204268 learned_addr learned my addr 172.21.15.53:0/2313204268 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:27.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08aa00640 1 --2- 172.21.15.53:0/2313204268 >> v2:172.21.15.146:3300/0 conn(0x7fc08c143c40 0x7fc08c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08b400640 1 -- 172.21.15.53:0/2313204268 >> v2:172.21.15.146:3300/0 conn(0x7fc08c143c40 msgr2=0x7fc08c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08b400640 1 --2- 172.21.15.53:0/2313204268 >> v2:172.21.15.146:3300/0 conn(0x7fc08c143c40 0x7fc08c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08a000640 1 -- 172.21.15.53:0/2313204268 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881110122 0 0) 0x7fc08c070e50 con 0x7fc08c0717d0 2024-01-16T15:15:27.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08b400640 1 -- 172.21.15.53:0/2313204268 >> v1:172.21.15.53:6789/0 conn(0x7fc08c0717d0 legacy=0x7fc08c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08b400640 1 -- 172.21.15.53:0/2313204268 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc08c071e00 con 0x7fc08c0ccc10 2024-01-16T15:15:27.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.270+0000 7fc08aa00640 1 --2- 172.21.15.53:0/2313204268 >> v2:172.21.15.146:3300/0 conn(0x7fc08c143c40 0x7fc08c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.271+0000 7fc08b400640 1 --2- 172.21.15.53:0/2313204268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0ccc10 0x7fc08c06d160 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fc08c06e410 tx=0x7fc08402be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d740b70a567ea98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.271+0000 7fc08a000640 1 -- 172.21.15.53:0/2313204268 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08402cbe0 con 0x7fc08c0ccc10 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.271+0000 7fc08a000640 1 -- 172.21.15.53:0/2313204268 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc08402cd40 con 0x7fc08c0ccc10 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.271+0000 7fc08a000640 1 -- 172.21.15.53:0/2313204268 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc084034660 con 0x7fc08c0ccc10 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 -- 172.21.15.53:0/2313204268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0ccc10 msgr2=0x7fc08c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 --2- 172.21.15.53:0/2313204268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0ccc10 0x7fc08c06d160 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fc08c06e410 tx=0x7fc08402be30 comp rx=0 tx=0).stop 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 -- 172.21.15.53:0/2313204268 shutdown_connections 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 --2- 172.21.15.53:0/2313204268 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0ccc10 0x7fc08c06d160 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 --2- 172.21.15.53:0/2313204268 >> v2:172.21.15.146:3300/0 conn(0x7fc08c143c40 0x7fc08c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 -- 172.21.15.53:0/2313204268 >> 172.21.15.53:0/2313204268 conn(0x7fc08c06a800 msgr2=0x7fc08c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 -- 172.21.15.53:0/2313204268 shutdown_connections 2024-01-16T15:15:27.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.272+0000 7fc08be00640 1 -- 172.21.15.53:0/2313204268 wait complete. 2024-01-16T15:15:27.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.274+0000 7fc08be00640 1 Processor -- start 2024-01-16T15:15:27.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.274+0000 7fc08be00640 1 -- start start 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.275+0000 7fc08be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 0x7fc08c15a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.275+0000 7fc08be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08c0ccc10 0x7fc08c0ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.275+0000 7fc08be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08c143c40 0x7fc08c13d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.275+0000 7fc08be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c1f0cc0 con 0x7fc08c0717d0 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.275+0000 7fc08be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0a8dd0 con 0x7fc08c143c40 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.275+0000 7fc08be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc08c0a84c0 con 0x7fc08c0ccc10 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 0x7fc08c15a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 0x7fc08c15a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35986/0 (socket says 172.21.15.53:35986) 2024-01-16T15:15:27.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 -- 172.21.15.53:0/4060861860 learned_addr learned my addr 172.21.15.53:0/4060861860 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc083e00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08c0ccc10 0x7fc08c0ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08b400640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08c143c40 0x7fc08c13d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 -- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08c0ccc10 msgr2=0x7fc08c0ab290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08c0ccc10 0x7fc08c0ab290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 -- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08c143c40 msgr2=0x7fc08c13d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08c143c40 0x7fc08c13d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.276+0000 7fc08aa00640 1 -- 172.21.15.53:0/4060861860 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc084009a70 con 0x7fc08c0717d0 2024-01-16T15:15:27.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.277+0000 7fc08aa00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 0x7fc08c15a780 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00e730 tx=0x7fc07c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.277+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c004230 con 0x7fc08c0717d0 2024-01-16T15:15:27.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.277+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc07c004390 con 0x7fc08c0717d0 2024-01-16T15:15:27.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.277+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c00c8f0 con 0x7fc08c0717d0 2024-01-16T15:15:27.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.277+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc08c0abd10 con 0x7fc08c0717d0 2024-01-16T15:15:27.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.277+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc08c0a8a60 con 0x7fc08c0717d0 2024-01-16T15:15:27.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.278+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc08c067450 con 0x7fc08c0717d0 2024-01-16T15:15:27.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.279+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fc07c00ca50 con 0x7fc08c0717d0 2024-01-16T15:15:27.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.280+0000 7fc083400640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc06406c160 0x7fc06406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.280+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fc07c088570 con 0x7fc08c0717d0 2024-01-16T15:15:27.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.281+0000 7fc083e00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc06406c160 0x7fc06406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.282+0000 7fc083e00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc06406c160 0x7fc06406e620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc08c140bc0 tx=0x7fc074007de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.269 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.284+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc07c054b70 con 0x7fc08c0717d0 2024-01-16T15:15:27.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.480+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc08c06cc40 con 0x7fc06406c160 2024-01-16T15:15:27.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.506+0000 7fc083400640 1 -- 172.21.15.53:0/4060861860 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285572 (secure 0 0 0) 0x7fc074004c70 con 0x7fc06406c160 2024-01-16T15:15:27.495 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:27.496 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:15:27.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.512+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc06406c160 msgr2=0x7fc06406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.512+0000 7fc08be00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc06406c160 0x7fc06406e620 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fc08c140bc0 tx=0x7fc074007de0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.512+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 msgr2=0x7fc08c15a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.512+0000 7fc08be00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 0x7fc08c15a780 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fc07c00e730 tx=0x7fc07c00ebf0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 shutdown_connections 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc06406c160 0x7fc06406e620 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc08c143c40 0x7fc08c13d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc08c0ccc10 0x7fc08c0ab290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 --2- 172.21.15.53:0/4060861860 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc08c0717d0 0x7fc08c15a780 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 >> 172.21.15.53:0/4060861860 conn(0x7fc08c06a800 msgr2=0x7fc08c140070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 shutdown_connections 2024-01-16T15:15:27.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.513+0000 7fc08be00640 1 -- 172.21.15.53:0/4060861860 wait complete. 2024-01-16T15:15:27.517 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2024-01-16T15:15:26.579691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591096,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581640,"kb_used_data":10640,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":754393080,"statfs":{"total":773094113280,"available":772498513920,"internally_reserved":0,"allocated":10895360,"data_stored":5353955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":584568368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001591"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691428+0000","last_change":"2024-01-16T15:15:22.537519+0000","last_active":"2024-01-16T15:15:23.691428+0000","last_peered":"2024-01-16T15:15:23.691428+0000","last_clean":"2024-01-16T15:15:23.691428+0000","last_became_active":"2024-01-16T15:15:22.535630+0000","last_became_peered":"2024-01-16T15:15:22.535630+0000","last_unstale":"2024-01-16T15:15:23.691428+0000","last_undegraded":"2024-01-16T15:15:23.691428+0000","last_fullsized":"2024-01-16T15:15:23.691428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:53.424772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740792+0000","last_change":"2024-01-16T15:15:22.532497+0000","last_active":"2024-01-16T15:15:23.740792+0000","last_peered":"2024-01-16T15:15:23.740792+0000","last_clean":"2024-01-16T15:15:23.740792+0000","last_became_active":"2024-01-16T15:15:22.531043+0000","last_became_peered":"2024-01-16T15:15:22.531043+0000","last_unstale":"2024-01-16T15:15:23.740792+0000","last_undegraded":"2024-01-16T15:15:23.740792+0000","last_fullsized":"2024-01-16T15:15:23.740792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691119+0000","last_change":"2024-01-16T15:15:22.535023+0000","last_active":"2024-01-16T15:15:23.691119+0000","last_peered":"2024-01-16T15:15:23.691119+0000","last_clean":"2024-01-16T15:15:23.691119+0000","last_became_active":"2024-01-16T15:15:22.534684+0000","last_became_peered":"2024-01-16T15:15:22.534684+0000","last_unstale":"2024-01-16T15:15:23.691119+0000","last_undegraded":"2024-01-16T15:15:23.691119+0000","last_fullsized":"2024-01-16T15:15:23.691119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:00.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509733+0000","last_change":"2024-01-16T15:15:22.524150+0000","last_active":"2024-01-16T15:15:23.509733+0000","last_peered":"2024-01-16T15:15:23.509733+0000","last_clean":"2024-01-16T15:15:23.509733+0000","last_became_active":"2024-01-16T15:15:22.522946+0000","last_became_peered":"2024-01-16T15:15:22.522946+0000","last_unstale":"2024-01-16T15:15:23.509733+0000","last_undegraded":"2024-01-16T15:15:23.509733+0000","last_fullsized":"2024-01-16T15:15:23.509733+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:13.223095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.519050+0000","last_change":"2024-01-16T15:15:21.519050+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.519050+0000","last_undegraded":"2024-01-16T15:15:21.519050+0000","last_fullsized":"2024-01-16T15:15:21.519050+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005648+0000","last_change":"2024-01-16T15:15:22.536600+0000","last_active":"2024-01-16T15:15:24.005648+0000","last_peered":"2024-01-16T15:15:24.005648+0000","last_clean":"2024-01-16T15:15:24.005648+0000","last_became_active":"2024-01-16T15:15:22.536327+0000","last_became_peered":"2024-01-16T15:15:22.536327+0000","last_unstale":"2024-01-16T15:15:24.005648+0000","last_undegraded":"2024-01-16T15:15:24.005648+0000","last_fullsized":"2024-01-16T15:15:24.005648+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005398+0000","last_change":"2024-01-16T15:15:22.535575+0000","last_active":"2024-01-16T15:15:24.005398+0000","last_peered":"2024-01-16T15:15:24.005398+0000","last_clean":"2024-01-16T15:15:24.005398+0000","last_became_active":"2024-01-16T15:15:22.535292+0000","last_became_peered":"2024-01-16T15:15:22.535292+0000","last_unstale":"2024-01-16T15:15:24.005398+0000","last_undegraded":"2024-01-16T15:15:24.005398+0000","last_fullsized":"2024-01-16T15:15:24.005398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:06.443796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740110+0000","last_change":"2024-01-16T15:15:22.521459+0000","last_active":"2024-01-16T15:15:23.740110+0000","last_peered":"2024-01-16T15:15:23.740110+0000","last_clean":"2024-01-16T15:15:23.740110+0000","last_became_active":"2024-01-16T15:15:22.520884+0000","last_became_peered":"2024-01-16T15:15:22.520884+0000","last_unstale":"2024-01-16T15:15:23.740110+0000","last_undegraded":"2024-01-16T15:15:23.740110+0000","last_fullsized":"2024-01-16T15:15:23.740110+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.513402+0000","last_change":"2024-01-16T15:15:21.513402+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.513402+0000","last_undegraded":"2024-01-16T15:15:21.513402+0000","last_fullsized":"2024-01-16T15:15:21.513402+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.520840+0000","last_change":"2024-01-16T15:15:21.520840+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.520840+0000","last_undegraded":"2024-01-16T15:15:21.520840+0000","last_fullsized":"2024-01-16T15:15:21.520840+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005800+0000","last_change":"2024-01-16T15:15:22.533081+0000","last_active":"2024-01-16T15:15:24.005800+0000","last_peered":"2024-01-16T15:15:24.005800+0000","last_clean":"2024-01-16T15:15:24.005800+0000","last_became_active":"2024-01-16T15:15:22.529012+0000","last_became_peered":"2024-01-16T15:15:22.529012+0000","last_unstale":"2024-01-16T15:15:24.005800+0000","last_undegraded":"2024-01-16T15:15:24.005800+0000","last_fullsized":"2024-01-16T15:15:24.005800+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.510864+0000","last_change":"2024-01-16T15:15:21.510864+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.510864+0000","last_undegraded":"2024-01-16T15:15:21.510864+0000","last_fullsized":"2024-01-16T15:15:21.510864+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509163+0000","last_change":"2024-01-16T15:15:22.532050+0000","last_active":"2024-01-16T15:15:23.509163+0000","last_peered":"2024-01-16T15:15:23.509163+0000","last_clean":"2024-01-16T15:15:23.509163+0000","last_became_active":"2024-01-16T15:15:22.531307+0000","last_became_peered":"2024-01-16T15:15:22.531307+0000","last_unstale":"2024-01-16T15:15:23.509163+0000","last_undegraded":"2024-01-16T15:15:23.509163+0000","last_fullsized":"2024-01-16T15:15:23.509163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:34.860510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609381+0000","last_change":"2024-01-16T15:15:22.533322+0000","last_active":"2024-01-16T15:15:23.609381+0000","last_peered":"2024-01-16T15:15:23.609381+0000","last_clean":"2024-01-16T15:15:23.609381+0000","last_became_active":"2024-01-16T15:15:22.532960+0000","last_became_peered":"2024-01-16T15:15:22.532960+0000","last_unstale":"2024-01-16T15:15:23.609381+0000","last_undegraded":"2024-01-16T15:15:23.609381+0000","last_fullsized":"2024-01-16T15:15:23.609381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:47.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508639+0000","last_change":"2024-01-16T15:15:22.532776+0000","last_active":"2024-01-16T15:15:23.508639+0000","last_peered":"2024-01-16T15:15:23.508639+0000","last_clean":"2024-01-16T15:15:23.508639+0000","last_became_active":"2024-01-16T15:15:22.532494+0000","last_became_peered":"2024-01-16T15:15:22.532494+0000","last_unstale":"2024-01-16T15:15:23.508639+0000","last_undegraded":"2024-01-16T15:15:23.508639+0000","last_fullsized":"2024-01-16T15:15:23.508639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005108+0000","last_change":"2024-01-16T15:15:22.530744+0000","last_active":"2024-01-16T15:15:24.005108+0000","last_peered":"2024-01-16T15:15:24.005108+0000","last_clean":"2024-01-16T15:15:24.005108+0000","last_became_active":"2024-01-16T15:15:22.525764+0000","last_became_peered":"2024-01-16T15:15:22.525764+0000","last_unstale":"2024-01-16T15:15:24.005108+0000","last_undegraded":"2024-01-16T15:15:24.005108+0000","last_fullsized":"2024-01-16T15:15:24.005108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:51.206294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741415+0000","last_change":"2024-01-16T15:15:22.523512+0000","last_active":"2024-01-16T15:15:23.741415+0000","last_peered":"2024-01-16T15:15:23.741415+0000","last_clean":"2024-01-16T15:15:23.741415+0000","last_became_active":"2024-01-16T15:15:22.521121+0000","last_became_peered":"2024-01-16T15:15:22.521121+0000","last_unstale":"2024-01-16T15:15:23.741415+0000","last_undegraded":"2024-01-16T15:15:23.741415+0000","last_fullsized":"2024-01-16T15:15:23.741415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:35.046638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509378+0000","last_change":"2024-01-16T15:15:22.531166+0000","last_active":"2024-01-16T15:15:23.509378+0000","last_peered":"2024-01-16T15:15:23.509378+0000","last_clean":"2024-01-16T15:15:23.509378+0000","last_became_active":"2024-01-16T15:15:22.530595+0000","last_became_peered":"2024-01-16T15:15:22.530595+0000","last_unstale":"2024-01-16T15:15:23.509378+0000","last_undegraded":"2024-01-16T15:15:23.509378+0000","last_fullsized":"2024-01-16T15:15:23.509378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:01.070526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610445+0000","last_change":"2024-01-16T15:15:22.524237+0000","last_active":"2024-01-16T15:15:23.610445+0000","last_peered":"2024-01-16T15:15:23.610445+0000","last_clean":"2024-01-16T15:15:23.610445+0000","last_became_active":"2024-01-16T15:15:22.523982+0000","last_became_peered":"2024-01-16T15:15:22.523982+0000","last_unstale":"2024-01-16T15:15:23.610445+0000","last_undegraded":"2024-01-16T15:15:23.610445+0000","last_fullsized":"2024-01-16T15:15:23.610445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:34.668807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.508179+0000","last_change":"2024-01-16T15:15:21.508179+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.508179+0000","last_undegraded":"2024-01-16T15:15:21.508179+0000","last_fullsized":"2024-01-16T15:15:21.508179+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609784+0000","last_change":"2024-01-16T15:15:22.529446+0000","last_active":"2024-01-16T15:15:23.609784+0000","last_peered":"2024-01-16T15:15:23.609784+0000","last_clean":"2024-01-16T15:15:23.609784+0000","last_became_active":"2024-01-16T15:15:22.529124+0000","last_became_peered":"2024-01-16T15:15:22.529124+0000","last_unstale":"2024-01-16T15:15:23.609784+0000","last_undegraded":"2024-01-16T15:15:23.609784+0000","last_fullsized":"2024-01-16T15:15:23.609784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:14.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740836+0000","last_change":"2024-01-16T15:15:22.541357+0000","last_active":"2024-01-16T15:15:23.740836+0000","last_peered":"2024-01-16T15:15:23.740836+0000","last_clean":"2024-01-16T15:15:23.740836+0000","last_became_active":"2024-01-16T15:15:22.539470+0000","last_became_peered":"2024-01-16T15:15:22.539470+0000","last_unstale":"2024-01-16T15:15:23.740836+0000","last_undegraded":"2024-01-16T15:15:23.740836+0000","last_fullsized":"2024-01-16T15:15:23.740836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740324+0000","last_change":"2024-01-16T15:15:22.528944+0000","last_active":"2024-01-16T15:15:23.740324+0000","last_peered":"2024-01-16T15:15:23.740324+0000","last_clean":"2024-01-16T15:15:23.740324+0000","last_became_active":"2024-01-16T15:15:22.528668+0000","last_became_peered":"2024-01-16T15:15:22.528668+0000","last_unstale":"2024-01-16T15:15:23.740324+0000","last_undegraded":"2024-01-16T15:15:23.740324+0000","last_fullsized":"2024-01-16T15:15:23.740324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005396+0000","last_change":"2024-01-16T15:15:22.527598+0000","last_active":"2024-01-16T15:15:24.005396+0000","last_peered":"2024-01-16T15:15:24.005396+0000","last_clean":"2024-01-16T15:15:24.005396+0000","last_became_active":"2024-01-16T15:15:22.526241+0000","last_became_peered":"2024-01-16T15:15:22.526241+0000","last_unstale":"2024-01-16T15:15:24.005396+0000","last_undegraded":"2024-01-16T15:15:24.005396+0000","last_fullsized":"2024-01-16T15:15:24.005396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:41.199256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.522680+0000","last_change":"2024-01-16T15:15:21.522680+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.522680+0000","last_undegraded":"2024-01-16T15:15:21.522680+0000","last_fullsized":"2024-01-16T15:15:21.522680+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741073+0000","last_change":"2024-01-16T15:15:22.518878+0000","last_active":"2024-01-16T15:15:23.741073+0000","last_peered":"2024-01-16T15:15:23.741073+0000","last_clean":"2024-01-16T15:15:23.741073+0000","last_became_active":"2024-01-16T15:15:22.518594+0000","last_became_peered":"2024-01-16T15:15:22.518594+0000","last_unstale":"2024-01-16T15:15:23.741073+0000","last_undegraded":"2024-01-16T15:15:23.741073+0000","last_fullsized":"2024-01-16T15:15:23.741073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006005+0000","last_change":"2024-01-16T15:15:22.529986+0000","last_active":"2024-01-16T15:15:24.006005+0000","last_peered":"2024-01-16T15:15:24.006005+0000","last_clean":"2024-01-16T15:15:24.006005+0000","last_became_active":"2024-01-16T15:15:22.525092+0000","last_became_peered":"2024-01-16T15:15:22.525092+0000","last_unstale":"2024-01-16T15:15:24.006005+0000","last_undegraded":"2024-01-16T15:15:24.006005+0000","last_fullsized":"2024-01-16T15:15:24.006005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.522614+0000","last_change":"2024-01-16T15:15:21.522614+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.522614+0000","last_undegraded":"2024-01-16T15:15:21.522614+0000","last_fullsized":"2024-01-16T15:15:21.522614+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740772+0000","last_change":"2024-01-16T15:15:22.518786+0000","last_active":"2024-01-16T15:15:23.740772+0000","last_peered":"2024-01-16T15:15:23.740772+0000","last_clean":"2024-01-16T15:15:23.740772+0000","last_became_active":"2024-01-16T15:15:22.512965+0000","last_became_peered":"2024-01-16T15:15:22.512965+0000","last_unstale":"2024-01-16T15:15:23.740772+0000","last_undegraded":"2024-01-16T15:15:23.740772+0000","last_fullsized":"2024-01-16T15:15:23.740772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609350+0000","last_change":"2024-01-16T15:15:22.521137+0000","last_active":"2024-01-16T15:15:23.609350+0000","last_peered":"2024-01-16T15:15:23.609350+0000","last_clean":"2024-01-16T15:15:23.609350+0000","last_became_active":"2024-01-16T15:15:22.520842+0000","last_became_peered":"2024-01-16T15:15:22.520842+0000","last_unstale":"2024-01-16T15:15:23.609350+0000","last_undegraded":"2024-01-16T15:15:23.609350+0000","last_fullsized":"2024-01-16T15:15:23.609350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:02.829205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508652+0000","last_change":"2024-01-16T15:15:22.541257+0000","last_active":"2024-01-16T15:15:23.508652+0000","last_peered":"2024-01-16T15:15:23.508652+0000","last_clean":"2024-01-16T15:15:23.508652+0000","last_became_active":"2024-01-16T15:15:22.540754+0000","last_became_peered":"2024-01-16T15:15:22.540754+0000","last_unstale":"2024-01-16T15:15:23.508652+0000","last_undegraded":"2024-01-16T15:15:23.508652+0000","last_fullsized":"2024-01-16T15:15:23.508652+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:10.664582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690290+0000","last_change":"2024-01-16T15:15:22.516532+0000","last_active":"2024-01-16T15:15:23.690290+0000","last_peered":"2024-01-16T15:15:23.690290+0000","last_clean":"2024-01-16T15:15:23.690290+0000","last_became_active":"2024-01-16T15:15:22.515440+0000","last_became_peered":"2024-01-16T15:15:22.515440+0000","last_unstale":"2024-01-16T15:15:23.690290+0000","last_undegraded":"2024-01-16T15:15:23.690290+0000","last_fullsized":"2024-01-16T15:15:23.690290+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:42.093044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610359+0000","last_change":"2024-01-16T15:15:22.521967+0000","last_active":"2024-01-16T15:15:23.610359+0000","last_peered":"2024-01-16T15:15:23.610359+0000","last_clean":"2024-01-16T15:15:23.610359+0000","last_became_active":"2024-01-16T15:15:22.520663+0000","last_became_peered":"2024-01-16T15:15:22.520663+0000","last_unstale":"2024-01-16T15:15:23.610359+0000","last_undegraded":"2024-01-16T15:15:23.610359+0000","last_fullsized":"2024-01-16T15:15:23.610359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:27.652785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741472+0000","last_change":"2024-01-16T15:15:21.529234+0000","last_active":"2024-01-16T15:15:23.741472+0000","last_peered":"2024-01-16T15:15:23.741472+0000","last_clean":"2024-01-16T15:15:23.741472+0000","last_became_active":"2024-01-16T15:15:21.528711+0000","last_became_peered":"2024-01-16T15:15:21.528711+0000","last_unstale":"2024-01-16T15:15:23.741472+0000","last_undegraded":"2024-01-16T15:15:23.741472+0000","last_fullsized":"2024-01-16T15:15:23.741472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:34.638361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509939+0000","last_change":"2024-01-16T15:15:22.540312+0000","last_active":"2024-01-16T15:15:23.509939+0000","last_peered":"2024-01-16T15:15:23.509939+0000","last_clean":"2024-01-16T15:15:23.509939+0000","last_became_active":"2024-01-16T15:15:22.538619+0000","last_became_peered":"2024-01-16T15:15:22.538619+0000","last_unstale":"2024-01-16T15:15:23.509939+0000","last_undegraded":"2024-01-16T15:15:23.509939+0000","last_fullsized":"2024-01-16T15:15:23.509939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:02.521266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.542686+0000","last_change":"2024-01-16T15:15:21.516982+0000","last_active":"2024-01-16T15:15:22.542686+0000","last_peered":"2024-01-16T15:15:22.542686+0000","last_clean":"2024-01-16T15:15:22.542686+0000","last_became_active":"2024-01-16T15:15:21.516459+0000","last_became_peered":"2024-01-16T15:15:21.516459+0000","last_unstale":"2024-01-16T15:15:22.542686+0000","last_undegraded":"2024-01-16T15:15:22.542686+0000","last_fullsized":"2024-01-16T15:15:22.542686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690957+0000","last_change":"2024-01-16T15:15:22.522691+0000","last_active":"2024-01-16T15:15:23.690957+0000","last_peered":"2024-01-16T15:15:23.690957+0000","last_clean":"2024-01-16T15:15:23.690957+0000","last_became_active":"2024-01-16T15:15:22.522375+0000","last_became_peered":"2024-01-16T15:15:22.522375+0000","last_unstale":"2024-01-16T15:15:23.690957+0000","last_undegraded":"2024-01-16T15:15:23.690957+0000","last_fullsized":"2024-01-16T15:15:23.690957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:12.190032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740678+0000","last_change":"2024-01-16T15:15:21.529660+0000","last_active":"2024-01-16T15:15:23.740678+0000","last_peered":"2024-01-16T15:15:23.740678+0000","last_clean":"2024-01-16T15:15:23.740678+0000","last_became_active":"2024-01-16T15:15:21.529009+0000","last_became_peered":"2024-01-16T15:15:21.529009+0000","last_unstale":"2024-01-16T15:15:23.740678+0000","last_undegraded":"2024-01-16T15:15:23.740678+0000","last_fullsized":"2024-01-16T15:15:23.740678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:49.081309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.508270+0000","last_change":"2024-01-16T15:15:21.508270+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.508270+0000","last_undegraded":"2024-01-16T15:15:21.508270+0000","last_fullsized":"2024-01-16T15:15:21.508270+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740604+0000","last_change":"2024-01-16T15:15:21.521824+0000","last_active":"2024-01-16T15:15:23.740604+0000","last_peered":"2024-01-16T15:15:23.740604+0000","last_clean":"2024-01-16T15:15:23.740604+0000","last_became_active":"2024-01-16T15:15:21.521406+0000","last_became_peered":"2024-01-16T15:15:21.521406+0000","last_unstale":"2024-01-16T15:15:23.740604+0000","last_undegraded":"2024-01-16T15:15:23.740604+0000","last_fullsized":"2024-01-16T15:15:23.740604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:39.695221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004361+0000","last_change":"2024-01-16T15:15:22.540055+0000","last_active":"2024-01-16T15:15:24.004361+0000","last_peered":"2024-01-16T15:15:24.004361+0000","last_clean":"2024-01-16T15:15:24.004361+0000","last_became_active":"2024-01-16T15:15:22.538674+0000","last_became_peered":"2024-01-16T15:15:22.538674+0000","last_unstale":"2024-01-16T15:15:24.004361+0000","last_undegraded":"2024-01-16T15:15:24.004361+0000","last_fullsized":"2024-01-16T15:15:24.004361+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:37.034062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690792+0000","last_change":"2024-01-16T15:15:21.530799+0000","last_active":"2024-01-16T15:15:23.690792+0000","last_peered":"2024-01-16T15:15:23.690792+0000","last_clean":"2024-01-16T15:15:23.690792+0000","last_became_active":"2024-01-16T15:15:21.530451+0000","last_became_peered":"2024-01-16T15:15:21.530451+0000","last_unstale":"2024-01-16T15:15:23.690792+0000","last_undegraded":"2024-01-16T15:15:23.690792+0000","last_fullsized":"2024-01-16T15:15:23.690792+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:26.637253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005884+0000","last_change":"2024-01-16T15:15:22.533064+0000","last_active":"2024-01-16T15:15:24.005884+0000","last_peered":"2024-01-16T15:15:24.005884+0000","last_clean":"2024-01-16T15:15:24.005884+0000","last_became_active":"2024-01-16T15:15:22.532644+0000","last_became_peered":"2024-01-16T15:15:22.532644+0000","last_unstale":"2024-01-16T15:15:24.005884+0000","last_undegraded":"2024-01-16T15:15:24.005884+0000","last_fullsized":"2024-01-16T15:15:24.005884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740860+0000","last_change":"2024-01-16T15:15:21.525811+0000","last_active":"2024-01-16T15:15:23.740860+0000","last_peered":"2024-01-16T15:15:23.740860+0000","last_clean":"2024-01-16T15:15:23.740860+0000","last_became_active":"2024-01-16T15:15:21.525152+0000","last_became_peered":"2024-01-16T15:15:21.525152+0000","last_unstale":"2024-01-16T15:15:23.740860+0000","last_undegraded":"2024-01-16T15:15:23.740860+0000","last_fullsized":"2024-01-16T15:15:23.740860+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:38.101810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508468+0000","last_change":"2024-01-16T15:15:22.524402+0000","last_active":"2024-01-16T15:15:23.508468+0000","last_peered":"2024-01-16T15:15:23.508468+0000","last_clean":"2024-01-16T15:15:23.508468+0000","last_became_active":"2024-01-16T15:15:22.524129+0000","last_became_peered":"2024-01-16T15:15:22.524129+0000","last_unstale":"2024-01-16T15:15:23.508468+0000","last_undegraded":"2024-01-16T15:15:23.508468+0000","last_fullsized":"2024-01-16T15:15:23.508468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.533761+0000","last_change":"2024-01-16T15:15:21.532465+0000","last_active":"2024-01-16T15:15:21.533761+0000","last_peered":"2024-01-16T15:15:21.533761+0000","last_clean":"2024-01-16T15:15:21.533761+0000","last_became_active":"2024-01-16T15:15:21.532238+0000","last_became_peered":"2024-01-16T15:15:21.532238+0000","last_unstale":"2024-01-16T15:15:21.533761+0000","last_undegraded":"2024-01-16T15:15:21.533761+0000","last_fullsized":"2024-01-16T15:15:21.533761+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:49.415856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505378+0000","last_change":"2024-01-16T15:15:21.505378+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505378+0000","last_undegraded":"2024-01-16T15:15:21.505378+0000","last_fullsized":"2024-01-16T15:15:21.505378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690271+0000","last_change":"2024-01-16T15:15:21.524375+0000","last_active":"2024-01-16T15:15:23.690271+0000","last_peered":"2024-01-16T15:15:23.690271+0000","last_clean":"2024-01-16T15:15:23.690271+0000","last_became_active":"2024-01-16T15:15:21.523731+0000","last_became_peered":"2024-01-16T15:15:21.523731+0000","last_unstale":"2024-01-16T15:15:23.690271+0000","last_undegraded":"2024-01-16T15:15:23.690271+0000","last_fullsized":"2024-01-16T15:15:23.690271+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:13.884029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741391+0000","last_change":"2024-01-16T15:15:22.528073+0000","last_active":"2024-01-16T15:15:23.741391+0000","last_peered":"2024-01-16T15:15:23.741391+0000","last_clean":"2024-01-16T15:15:23.741391+0000","last_became_active":"2024-01-16T15:15:22.526190+0000","last_became_peered":"2024-01-16T15:15:22.526190+0000","last_unstale":"2024-01-16T15:15:23.741391+0000","last_undegraded":"2024-01-16T15:15:23.741391+0000","last_fullsized":"2024-01-16T15:15:23.741391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:55.442576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691500+0000","last_change":"2024-01-16T15:15:21.520386+0000","last_active":"2024-01-16T15:15:23.691500+0000","last_peered":"2024-01-16T15:15:23.691500+0000","last_clean":"2024-01-16T15:15:23.691500+0000","last_became_active":"2024-01-16T15:15:21.520059+0000","last_became_peered":"2024-01-16T15:15:21.520059+0000","last_unstale":"2024-01-16T15:15:23.691500+0000","last_undegraded":"2024-01-16T15:15:23.691500+0000","last_fullsized":"2024-01-16T15:15:23.691500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:33.653275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.504957+0000","last_change":"2024-01-16T15:15:21.504957+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.504957+0000","last_undegraded":"2024-01-16T15:15:21.504957+0000","last_fullsized":"2024-01-16T15:15:21.504957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509435+0000","last_change":"2024-01-16T15:15:21.534208+0000","last_active":"2024-01-16T15:15:23.509435+0000","last_peered":"2024-01-16T15:15:23.509435+0000","last_clean":"2024-01-16T15:15:23.509435+0000","last_became_active":"2024-01-16T15:15:21.533545+0000","last_became_peered":"2024-01-16T15:15:21.533545+0000","last_unstale":"2024-01-16T15:15:23.509435+0000","last_undegraded":"2024-01-16T15:15:23.509435+0000","last_fullsized":"2024-01-16T15:15:23.509435+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518791+0000","last_change":"2024-01-16T15:15:21.518791+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518791+0000","last_undegraded":"2024-01-16T15:15:21.518791+0000","last_fullsized":"2024-01-16T15:15:21.518791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.547699+0000","last_change":"2024-01-16T15:15:21.521285+0000","last_active":"2024-01-16T15:15:22.547699+0000","last_peered":"2024-01-16T15:15:22.547699+0000","last_clean":"2024-01-16T15:15:22.547699+0000","last_became_active":"2024-01-16T15:15:21.520717+0000","last_became_peered":"2024-01-16T15:15:21.520717+0000","last_unstale":"2024-01-16T15:15:22.547699+0000","last_undegraded":"2024-01-16T15:15:22.547699+0000","last_fullsized":"2024-01-16T15:15:22.547699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:15.674482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006046+0000","last_change":"2024-01-16T15:15:22.529757+0000","last_active":"2024-01-16T15:15:24.006046+0000","last_peered":"2024-01-16T15:15:24.006046+0000","last_clean":"2024-01-16T15:15:24.006046+0000","last_became_active":"2024-01-16T15:15:22.524620+0000","last_became_peered":"2024-01-16T15:15:22.524620+0000","last_unstale":"2024-01-16T15:15:24.006046+0000","last_undegraded":"2024-01-16T15:15:24.006046+0000","last_fullsized":"2024-01-16T15:15:24.006046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.520270+0000","last_change":"2024-01-16T15:15:21.518507+0000","last_active":"2024-01-16T15:15:21.520270+0000","last_peered":"2024-01-16T15:15:21.520270+0000","last_clean":"2024-01-16T15:15:21.520270+0000","last_became_active":"2024-01-16T15:15:21.518075+0000","last_became_peered":"2024-01-16T15:15:21.518075+0000","last_unstale":"2024-01-16T15:15:21.520270+0000","last_undegraded":"2024-01-16T15:15:21.520270+0000","last_fullsized":"2024-01-16T15:15:21.520270+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:25.234717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509009+0000","last_change":"2024-01-16T15:15:22.521859+0000","last_active":"2024-01-16T15:15:23.509009+0000","last_peered":"2024-01-16T15:15:23.509009+0000","last_clean":"2024-01-16T15:15:23.509009+0000","last_became_active":"2024-01-16T15:15:22.521561+0000","last_became_peered":"2024-01-16T15:15:22.521561+0000","last_unstale":"2024-01-16T15:15:23.509009+0000","last_undegraded":"2024-01-16T15:15:23.509009+0000","last_fullsized":"2024-01-16T15:15:23.509009+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:14.797953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609905+0000","last_change":"2024-01-16T15:15:21.521158+0000","last_active":"2024-01-16T15:15:23.609905+0000","last_peered":"2024-01-16T15:15:23.609905+0000","last_clean":"2024-01-16T15:15:23.609905+0000","last_became_active":"2024-01-16T15:15:21.520851+0000","last_became_peered":"2024-01-16T15:15:21.520851+0000","last_unstale":"2024-01-16T15:15:23.609905+0000","last_undegraded":"2024-01-16T15:15:23.609905+0000","last_fullsized":"2024-01-16T15:15:23.609905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:47.111965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690791+0000","last_change":"2024-01-16T15:15:22.530599+0000","last_active":"2024-01-16T15:15:23.690791+0000","last_peered":"2024-01-16T15:15:23.690791+0000","last_clean":"2024-01-16T15:15:23.690791+0000","last_became_active":"2024-01-16T15:15:22.526185+0000","last_became_peered":"2024-01-16T15:15:22.526185+0000","last_unstale":"2024-01-16T15:15:23.690791+0000","last_undegraded":"2024-01-16T15:15:23.690791+0000","last_fullsized":"2024-01-16T15:15:23.690791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:11.722014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609612+0000","last_change":"2024-01-16T15:15:21.526311+0000","last_active":"2024-01-16T15:15:23.609612+0000","last_peered":"2024-01-16T15:15:23.609612+0000","last_clean":"2024-01-16T15:15:23.609612+0000","last_became_active":"2024-01-16T15:15:21.526019+0000","last_became_peered":"2024-01-16T15:15:21.526019+0000","last_unstale":"2024-01-16T15:15:23.609612+0000","last_undegraded":"2024-01-16T15:15:23.609612+0000","last_fullsized":"2024-01-16T15:15:23.609612+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:30.739366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505003+0000","last_change":"2024-01-16T15:15:21.505003+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505003+0000","last_undegraded":"2024-01-16T15:15:21.505003+0000","last_fullsized":"2024-01-16T15:15:21.505003+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.530670+0000","last_change":"2024-01-16T15:15:21.528755+0000","last_active":"2024-01-16T15:15:21.530670+0000","last_peered":"2024-01-16T15:15:21.530670+0000","last_clean":"2024-01-16T15:15:21.530670+0000","last_became_active":"2024-01-16T15:15:21.528462+0000","last_became_peered":"2024-01-16T15:15:21.528462+0000","last_unstale":"2024-01-16T15:15:21.530670+0000","last_undegraded":"2024-01-16T15:15:21.530670+0000","last_fullsized":"2024-01-16T15:15:21.530670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:46.120683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609164+0000","last_change":"2024-01-16T15:15:22.529246+0000","last_active":"2024-01-16T15:15:23.609164+0000","last_peered":"2024-01-16T15:15:23.609164+0000","last_clean":"2024-01-16T15:15:23.609164+0000","last_became_active":"2024-01-16T15:15:22.528966+0000","last_became_peered":"2024-01-16T15:15:22.528966+0000","last_unstale":"2024-01-16T15:15:23.609164+0000","last_undegraded":"2024-01-16T15:15:23.609164+0000","last_fullsized":"2024-01-16T15:15:23.609164+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:46.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690545+0000","last_change":"2024-01-16T15:15:21.525650+0000","last_active":"2024-01-16T15:15:23.690545+0000","last_peered":"2024-01-16T15:15:23.690545+0000","last_clean":"2024-01-16T15:15:23.690545+0000","last_became_active":"2024-01-16T15:15:21.524160+0000","last_became_peered":"2024-01-16T15:15:21.524160+0000","last_unstale":"2024-01-16T15:15:23.690545+0000","last_undegraded":"2024-01-16T15:15:23.690545+0000","last_fullsized":"2024-01-16T15:15:23.690545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:28.946524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.521709+0000","last_change":"2024-01-16T15:15:21.521709+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.521709+0000","last_undegraded":"2024-01-16T15:15:21.521709+0000","last_fullsized":"2024-01-16T15:15:21.521709+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.520782+0000","last_change":"2024-01-16T15:15:21.520262+0000","last_active":"2024-01-16T15:15:21.520782+0000","last_peered":"2024-01-16T15:15:21.520782+0000","last_clean":"2024-01-16T15:15:21.520782+0000","last_became_active":"2024-01-16T15:15:21.519869+0000","last_became_peered":"2024-01-16T15:15:21.519869+0000","last_unstale":"2024-01-16T15:15:21.520782+0000","last_undegraded":"2024-01-16T15:15:21.520782+0000","last_fullsized":"2024-01-16T15:15:21.520782+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:13.821755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004344+0000","last_change":"2024-01-16T15:15:22.543578+0000","last_active":"2024-01-16T15:15:24.004344+0000","last_peered":"2024-01-16T15:15:24.004344+0000","last_clean":"2024-01-16T15:15:24.004344+0000","last_became_active":"2024-01-16T15:15:22.543283+0000","last_became_peered":"2024-01-16T15:15:22.543283+0000","last_unstale":"2024-01-16T15:15:24.004344+0000","last_undegraded":"2024-01-16T15:15:24.004344+0000","last_fullsized":"2024-01-16T15:15:24.004344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.530844+0000","last_change":"2024-01-16T15:15:21.528614+0000","last_active":"2024-01-16T15:15:21.530844+0000","last_peered":"2024-01-16T15:15:21.530844+0000","last_clean":"2024-01-16T15:15:21.530844+0000","last_became_active":"2024-01-16T15:15:21.528325+0000","last_became_peered":"2024-01-16T15:15:21.528325+0000","last_unstale":"2024-01-16T15:15:21.530844+0000","last_undegraded":"2024-01-16T15:15:21.530844+0000","last_fullsized":"2024-01-16T15:15:21.530844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:58.124662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007346+0000","last_change":"2024-01-16T15:15:22.530224+0000","last_active":"2024-01-16T15:15:24.007346+0000","last_peered":"2024-01-16T15:15:24.007346+0000","last_clean":"2024-01-16T15:15:24.007346+0000","last_became_active":"2024-01-16T15:15:22.528654+0000","last_became_peered":"2024-01-16T15:15:22.528654+0000","last_unstale":"2024-01-16T15:15:24.007346+0000","last_undegraded":"2024-01-16T15:15:24.007346+0000","last_fullsized":"2024-01-16T15:15:24.007346+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740462+0000","last_change":"2024-01-16T15:15:21.521658+0000","last_active":"2024-01-16T15:15:23.740462+0000","last_peered":"2024-01-16T15:15:23.740462+0000","last_clean":"2024-01-16T15:15:23.740462+0000","last_became_active":"2024-01-16T15:15:21.521051+0000","last_became_peered":"2024-01-16T15:15:21.521051+0000","last_unstale":"2024-01-16T15:15:23.740462+0000","last_undegraded":"2024-01-16T15:15:23.740462+0000","last_fullsized":"2024-01-16T15:15:23.740462+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609501+0000","last_change":"2024-01-16T15:15:22.525393+0000","last_active":"2024-01-16T15:15:23.609501+0000","last_peered":"2024-01-16T15:15:23.609501+0000","last_clean":"2024-01-16T15:15:23.609501+0000","last_became_active":"2024-01-16T15:15:22.524494+0000","last_became_peered":"2024-01-16T15:15:22.524494+0000","last_unstale":"2024-01-16T15:15:23.609501+0000","last_undegraded":"2024-01-16T15:15:23.609501+0000","last_fullsized":"2024-01-16T15:15:23.609501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:10.852002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740975+0000","last_change":"2024-01-16T15:15:21.531548+0000","last_active":"2024-01-16T15:15:23.740975+0000","last_peered":"2024-01-16T15:15:23.740975+0000","last_clean":"2024-01-16T15:15:23.740975+0000","last_became_active":"2024-01-16T15:15:21.531267+0000","last_became_peered":"2024-01-16T15:15:21.531267+0000","last_unstale":"2024-01-16T15:15:23.740975+0000","last_undegraded":"2024-01-16T15:15:23.740975+0000","last_fullsized":"2024-01-16T15:15:23.740975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:42.610622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609997+0000","last_change":"2024-01-16T15:15:22.522512+0000","last_active":"2024-01-16T15:15:23.609997+0000","last_peered":"2024-01-16T15:15:23.609997+0000","last_clean":"2024-01-16T15:15:23.609997+0000","last_became_active":"2024-01-16T15:15:22.522206+0000","last_became_peered":"2024-01-16T15:15:22.522206+0000","last_unstale":"2024-01-16T15:15:23.609997+0000","last_undegraded":"2024-01-16T15:15:23.609997+0000","last_fullsized":"2024-01-16T15:15:23.609997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:29.962056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004895+0000","last_change":"2024-01-16T15:15:21.524797+0000","last_active":"2024-01-16T15:15:24.004895+0000","last_peered":"2024-01-16T15:15:24.004895+0000","last_clean":"2024-01-16T15:15:24.004895+0000","last_became_active":"2024-01-16T15:15:21.524502+0000","last_became_peered":"2024-01-16T15:15:21.524502+0000","last_unstale":"2024-01-16T15:15:24.004895+0000","last_undegraded":"2024-01-16T15:15:24.004895+0000","last_fullsized":"2024-01-16T15:15:24.004895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:46.648168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609344+0000","last_change":"2024-01-16T15:15:22.523225+0000","last_active":"2024-01-16T15:15:23.609344+0000","last_peered":"2024-01-16T15:15:23.609344+0000","last_clean":"2024-01-16T15:15:23.609344+0000","last_became_active":"2024-01-16T15:15:22.522951+0000","last_became_peered":"2024-01-16T15:15:22.522951+0000","last_unstale":"2024-01-16T15:15:23.609344+0000","last_undegraded":"2024-01-16T15:15:23.609344+0000","last_fullsized":"2024-01-16T15:15:23.609344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:21.818020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004192+0000","last_change":"2024-01-16T15:15:21.519991+0000","last_active":"2024-01-16T15:15:24.004192+0000","last_peered":"2024-01-16T15:15:24.004192+0000","last_clean":"2024-01-16T15:15:24.004192+0000","last_became_active":"2024-01-16T15:15:21.519565+0000","last_became_peered":"2024-01-16T15:15:21.519565+0000","last_unstale":"2024-01-16T15:15:24.004192+0000","last_undegraded":"2024-01-16T15:15:24.004192+0000","last_fullsized":"2024-01-16T15:15:24.004192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:50.723441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518797+0000","last_change":"2024-01-16T15:15:21.518797+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518797+0000","last_undegraded":"2024-01-16T15:15:21.518797+0000","last_fullsized":"2024-01-16T15:15:21.518797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740258+0000","last_change":"2024-01-16T15:15:21.519702+0000","last_active":"2024-01-16T15:15:23.740258+0000","last_peered":"2024-01-16T15:15:23.740258+0000","last_clean":"2024-01-16T15:15:23.740258+0000","last_became_active":"2024-01-16T15:15:21.518926+0000","last_became_peered":"2024-01-16T15:15:21.518926+0000","last_unstale":"2024-01-16T15:15:23.740258+0000","last_undegraded":"2024-01-16T15:15:23.740258+0000","last_fullsized":"2024-01-16T15:15:23.740258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:31.707777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741243+0000","last_change":"2024-01-16T15:15:22.536514+0000","last_active":"2024-01-16T15:15:23.741243+0000","last_peered":"2024-01-16T15:15:23.741243+0000","last_clean":"2024-01-16T15:15:23.741243+0000","last_became_active":"2024-01-16T15:15:22.536248+0000","last_became_peered":"2024-01-16T15:15:22.536248+0000","last_unstale":"2024-01-16T15:15:23.741243+0000","last_undegraded":"2024-01-16T15:15:23.741243+0000","last_fullsized":"2024-01-16T15:15:23.741243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:12:10.024161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005379+0000","last_change":"2024-01-16T15:15:21.522416+0000","last_active":"2024-01-16T15:15:24.005379+0000","last_peered":"2024-01-16T15:15:24.005379+0000","last_clean":"2024-01-16T15:15:24.005379+0000","last_became_active":"2024-01-16T15:15:21.522118+0000","last_became_peered":"2024-01-16T15:15:21.522118+0000","last_unstale":"2024-01-16T15:15:24.005379+0000","last_undegraded":"2024-01-16T15:15:24.005379+0000","last_fullsized":"2024-01-16T15:15:24.005379+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.762652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691386+0000","last_change":"2024-01-16T15:15:22.531377+0000","last_active":"2024-01-16T15:15:23.691386+0000","last_peered":"2024-01-16T15:15:23.691386+0000","last_clean":"2024-01-16T15:15:23.691386+0000","last_became_active":"2024-01-16T15:15:22.529798+0000","last_became_peered":"2024-01-16T15:15:22.529798+0000","last_unstale":"2024-01-16T15:15:23.691386+0000","last_undegraded":"2024-01-16T15:15:23.691386+0000","last_fullsized":"2024-01-16T15:15:23.691386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.721245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741334+0000","last_change":"2024-01-16T15:15:21.532318+0000","last_active":"2024-01-16T15:15:23.741334+0000","last_peered":"2024-01-16T15:15:23.741334+0000","last_clean":"2024-01-16T15:15:23.741334+0000","last_became_active":"2024-01-16T15:15:21.531869+0000","last_became_peered":"2024-01-16T15:15:21.531869+0000","last_unstale":"2024-01-16T15:15:23.741334+0000","last_undegraded":"2024-01-16T15:15:23.741334+0000","last_fullsized":"2024-01-16T15:15:23.741334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:40.183724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690994+0000","last_change":"2024-01-16T15:15:22.535293+0000","last_active":"2024-01-16T15:15:23.690994+0000","last_peered":"2024-01-16T15:15:23.690994+0000","last_clean":"2024-01-16T15:15:23.690994+0000","last_became_active":"2024-01-16T15:15:22.534929+0000","last_became_peered":"2024-01-16T15:15:22.534929+0000","last_unstale":"2024-01-16T15:15:23.690994+0000","last_undegraded":"2024-01-16T15:15:23.690994+0000","last_fullsized":"2024-01-16T15:15:23.690994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.606169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740931+0000","last_change":"2024-01-16T15:15:21.520890+0000","last_active":"2024-01-16T15:15:23.740931+0000","last_peered":"2024-01-16T15:15:23.740931+0000","last_clean":"2024-01-16T15:15:23.740931+0000","last_became_active":"2024-01-16T15:15:21.520488+0000","last_became_peered":"2024-01-16T15:15:21.520488+0000","last_unstale":"2024-01-16T15:15:23.740931+0000","last_undegraded":"2024-01-16T15:15:23.740931+0000","last_fullsized":"2024-01-16T15:15:23.740931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:08.456302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.521585+0000","last_change":"2024-01-16T15:15:21.521585+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.521585+0000","last_undegraded":"2024-01-16T15:15:21.521585+0000","last_fullsized":"2024-01-16T15:15:21.521585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.528795+0000","last_change":"2024-01-16T15:15:21.527345+0000","last_active":"2024-01-16T15:15:21.528795+0000","last_peered":"2024-01-16T15:15:21.528795+0000","last_clean":"2024-01-16T15:15:21.528795+0000","last_became_active":"2024-01-16T15:15:21.527018+0000","last_became_peered":"2024-01-16T15:15:21.527018+0000","last_unstale":"2024-01-16T15:15:21.528795+0000","last_undegraded":"2024-01-16T15:15:21.528795+0000","last_fullsized":"2024-01-16T15:15:21.528795+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:59.366965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609781+0000","last_change":"2024-01-16T15:15:22.525600+0000","last_active":"2024-01-16T15:15:23.609781+0000","last_peered":"2024-01-16T15:15:23.609781+0000","last_clean":"2024-01-16T15:15:23.609781+0000","last_became_active":"2024-01-16T15:15:22.525112+0000","last_became_peered":"2024-01-16T15:15:22.525112+0000","last_unstale":"2024-01-16T15:15:23.609781+0000","last_undegraded":"2024-01-16T15:15:23.609781+0000","last_fullsized":"2024-01-16T15:15:23.609781+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:49.965076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741004+0000","last_change":"2024-01-16T15:15:21.526060+0000","last_active":"2024-01-16T15:15:23.741004+0000","last_peered":"2024-01-16T15:15:23.741004+0000","last_clean":"2024-01-16T15:15:23.741004+0000","last_became_active":"2024-01-16T15:15:21.525792+0000","last_became_peered":"2024-01-16T15:15:21.525792+0000","last_unstale":"2024-01-16T15:15:23.741004+0000","last_undegraded":"2024-01-16T15:15:23.741004+0000","last_fullsized":"2024-01-16T15:15:23.741004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505027+0000","last_change":"2024-01-16T15:15:21.505027+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505027+0000","last_undegraded":"2024-01-16T15:15:21.505027+0000","last_fullsized":"2024-01-16T15:15:21.505027+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.533268+0000","last_change":"2024-01-16T15:15:21.530401+0000","last_active":"2024-01-16T15:15:21.533268+0000","last_peered":"2024-01-16T15:15:21.533268+0000","last_clean":"2024-01-16T15:15:21.533268+0000","last_became_active":"2024-01-16T15:15:21.530083+0000","last_became_peered":"2024-01-16T15:15:21.530083+0000","last_unstale":"2024-01-16T15:15:21.533268+0000","last_undegraded":"2024-01-16T15:15:21.533268+0000","last_fullsized":"2024-01-16T15:15:21.533268+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:56.829359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609251+0000","last_change":"2024-01-16T15:15:22.530971+0000","last_active":"2024-01-16T15:15:23.609251+0000","last_peered":"2024-01-16T15:15:23.609251+0000","last_clean":"2024-01-16T15:15:23.609251+0000","last_became_active":"2024-01-16T15:15:22.530694+0000","last_became_peered":"2024-01-16T15:15:22.530694+0000","last_unstale":"2024-01-16T15:15:23.609251+0000","last_undegraded":"2024-01-16T15:15:23.609251+0000","last_fullsized":"2024-01-16T15:15:23.609251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:02.926295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508734+0000","last_change":"2024-01-16T15:15:21.524472+0000","last_active":"2024-01-16T15:15:23.508734+0000","last_peered":"2024-01-16T15:15:23.508734+0000","last_clean":"2024-01-16T15:15:23.508734+0000","last_became_active":"2024-01-16T15:15:21.524175+0000","last_became_peered":"2024-01-16T15:15:21.524175+0000","last_unstale":"2024-01-16T15:15:23.508734+0000","last_undegraded":"2024-01-16T15:15:23.508734+0000","last_fullsized":"2024-01-16T15:15:23.508734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:43.337353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518265+0000","last_change":"2024-01-16T15:15:21.518265+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518265+0000","last_undegraded":"2024-01-16T15:15:21.518265+0000","last_fullsized":"2024-01-16T15:15:21.518265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609465+0000","last_change":"2024-01-16T15:15:21.533877+0000","last_active":"2024-01-16T15:15:23.609465+0000","last_peered":"2024-01-16T15:15:23.609465+0000","last_clean":"2024-01-16T15:15:23.609465+0000","last_became_active":"2024-01-16T15:15:21.533489+0000","last_became_peered":"2024-01-16T15:15:21.533489+0000","last_unstale":"2024-01-16T15:15:23.609465+0000","last_undegraded":"2024-01-16T15:15:23.609465+0000","last_fullsized":"2024-01-16T15:15:23.609465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:15.404651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610593+0000","last_change":"2024-01-16T15:15:22.521850+0000","last_active":"2024-01-16T15:15:23.610593+0000","last_peered":"2024-01-16T15:15:23.610593+0000","last_clean":"2024-01-16T15:15:23.610593+0000","last_became_active":"2024-01-16T15:15:22.521581+0000","last_became_peered":"2024-01-16T15:15:22.521581+0000","last_unstale":"2024-01-16T15:15:23.610593+0000","last_undegraded":"2024-01-16T15:15:23.610593+0000","last_fullsized":"2024-01-16T15:15:23.610593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:29.487154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.545734+0000","last_change":"2024-01-16T15:15:21.522571+0000","last_active":"2024-01-16T15:15:22.545734+0000","last_peered":"2024-01-16T15:15:22.545734+0000","last_clean":"2024-01-16T15:15:22.545734+0000","last_became_active":"2024-01-16T15:15:21.522205+0000","last_became_peered":"2024-01-16T15:15:21.522205+0000","last_unstale":"2024-01-16T15:15:22.545734+0000","last_undegraded":"2024-01-16T15:15:22.545734+0000","last_fullsized":"2024-01-16T15:15:22.545734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":241,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516932+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:15:22.516932+0000","last_peered":"2024-01-16T15:15:22.516932+0000","last_clean":"2024-01-16T15:15:22.516932+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:15:22.516932+0000","last_undegraded":"2024-01-16T15:15:22.516932+0000","last_fullsized":"2024-01-16T15:15:22.516932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509629,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68364,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303476,"statfs":{"total":96636764160,"available":96566759424,"internally_reserved":0,"allocated":1179648,"data_stored":505985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3140000000000001,"15min":1.615},"min":{"1min":0.75,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":2.5030000000000001,"5min":8.8070000000000004,"15min":27.853999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.663,"15min":1.708},"min":{"1min":0.93000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6399999999999999,"5min":28.603999999999999,"15min":34.939999999999998},"last":0.875}]},{"osd":1,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.292,"15min":1.4830000000000001},"min":{"1min":0.94899999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.528,"5min":8.2650000000000006,"15min":13.859},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.754,"15min":1.651},"min":{"1min":0.85599999999999998,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":2.702,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.2589999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4330000000000001,"15min":1.5860000000000001},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":2.6230000000000002,"5min":9.6679999999999993,"15min":13.050000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.69499999999999995,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":2.6309999999999998,"5min":6.782,"15min":35.011000000000003},"last":1.3100000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.486,"15min":1.657},"min":{"1min":0.94799999999999995,"5min":0.439,"15min":0.374},"max":{"1min":1.8720000000000001,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.7170000000000001},"min":{"1min":0.999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.758,"5min":9.7780000000000005,"15min":14.273999999999999},"last":1.1519999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3919999999999999,"15min":1.827},"min":{"1min":0.90900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7090000000000001,"5min":10.952999999999999,"15min":34.972999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.4319999999999999,"15min":1.798},"min":{"1min":0.85899999999999999,"5min":0.58299999999999996,"15min":0.40400000000000003},"max":{"1min":1.6519999999999999,"5min":5.2300000000000004,"15min":16.003},"last":1.411}]},{"osd":5,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.591,"15min":2.0099999999999998},"min":{"1min":1.1339999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.798,"5min":4.4550000000000001,"15min":23.760999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6259999999999999,"15min":1.9059999999999999},"min":{"1min":0.96299999999999997,"5min":0.70699999999999996,"15min":0.52400000000000002},"max":{"1min":3.496,"5min":9.4120000000000008,"15min":23.716999999999999},"last":1.363}]},{"osd":7,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.5069999999999999,"15min":2.0430000000000001},"min":{"1min":0.90700000000000003,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":6.9219999999999997,"15min":34.889000000000003},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.79,"15min":2.1760000000000002},"min":{"1min":1.2470000000000001,"5min":0.65100000000000002,"15min":0.47599999999999998},"max":{"1min":2.0499999999999998,"5min":23.242999999999999,"15min":46.305},"last":1.548}]}]},{"osd":4,"up_from":15,"seq":64424509631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68380,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303460,"statfs":{"total":96636764160,"available":96566743040,"internally_reserved":0,"allocated":1191936,"data_stored":512288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.226,"15min":1.55},"min":{"1min":0.76900000000000002,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.1399999999999997,"15min":22.757999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.125,"15min":1.2729999999999999},"min":{"1min":0.69799999999999995,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.474,"5min":4.0270000000000001,"15min":14.034000000000001},"last":1.623}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.413,"15min":1.577},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.851,"5min":17.811,"15min":18.347000000000001},"last":1.002},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3140000000000001,"15min":1.49},"min":{"1min":0.90700000000000003,"5min":0.59799999999999998,"15min":0.498},"max":{"1min":1.5600000000000001,"5min":4.0449999999999999,"15min":18.858000000000001},"last":1.6679999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.544,"15min":1.679},"min":{"1min":0.83799999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.768,"5min":17.651,"15min":17.651},"last":1.204},{"interface":"front","average":{"1min":1.327,"5min":1.754,"15min":1.722},"min":{"1min":0.83499999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8180000000000001,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.77}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3320000000000001,"15min":1.554},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":1.6659999999999999,"5min":3.972,"15min":26.741},"last":1.458},{"interface":"front","average":{"1min":1.284,"5min":1.571,"15min":1.7929999999999999},"min":{"1min":0.86899999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.762,"5min":17.715,"15min":26.701000000000001},"last":1.3149999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5660000000000001,"15min":1.7370000000000001},"min":{"1min":1.0029999999999999,"5min":0.51400000000000001,"15min":0.441},"max":{"1min":1.988,"5min":5.3170000000000002,"15min":18.919},"last":1.367},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5029999999999999,"15min":1.6759999999999999},"min":{"1min":0.95599999999999996,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":2.0270000000000001,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.8280000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":2.0739999999999998,"15min":1.9419999999999999},"min":{"1min":0.995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.0750000000000002,"5min":32.649999999999999,"15min":32.649999999999999},"last":1.514},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5469999999999999,"15min":1.823},"min":{"1min":1.1799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0630000000000002,"5min":4.3129999999999997,"15min":24.094999999999999},"last":1.55}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.6890000000000001,"15min":1.7649999999999999},"min":{"1min":1.056,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":1.8899999999999999,"5min":17.446000000000002,"15min":26.683},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.871,"15min":1.9650000000000001},"min":{"1min":1.248,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9350000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":1.407}]}]},{"osd":7,"up_from":14,"seq":60129542334,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68988,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302852,"statfs":{"total":96636764160,"available":96566120448,"internally_reserved":0,"allocated":1818624,"data_stored":1115899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2649999999999999,"15min":1.5089999999999999},"min":{"1min":0.68100000000000005,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.7310000000000001,"5min":4.3680000000000003,"15min":15.661},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.3240000000000001,"15min":1.5449999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":2.1080000000000001,"5min":12.461,"15min":30.126000000000001},"last":1.226}]},{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.423,"15min":1.4530000000000001},"min":{"1min":0.63200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.085,"5min":20.327000000000002,"15min":24.338000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.395,"15min":1.6180000000000001},"min":{"1min":0.81399999999999995,"5min":0.77400000000000002,"15min":0.46200000000000002},"max":{"1min":1.6419999999999999,"5min":12.526,"15min":21.298999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.57199999999999995,"15min":0.51500000000000001},"max":{"1min":1.9219999999999999,"5min":4.3380000000000001,"15min":12.474},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.702,"15min":1.651},"min":{"1min":0.69299999999999995,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8999999999999999,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4039999999999999,"15min":1.6020000000000001},"min":{"1min":0.88800000000000001,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.7150000000000001,"5min":4.2530000000000001,"15min":12.58},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.9430000000000001,"15min":1.7869999999999999},"min":{"1min":0.78600000000000003,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":2.4630000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3520000000000001,"15min":1.591},"min":{"1min":0.99099999999999999,"5min":0.54100000000000004,"15min":0.42799999999999999},"max":{"1min":2.0219999999999998,"5min":4.1390000000000002,"15min":14.659000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.831,"15min":1.825},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":2.028,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.3600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.486,"15min":1.8360000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.6459999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.113},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.8400000000000001},"min":{"1min":1.1870000000000001,"5min":0.66000000000000003,"15min":0.41099999999999998},"max":{"1min":1.8400000000000001,"5min":7.4649999999999999,"15min":21.312000000000001},"last":1.383}]},{"osd":6,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.9790000000000001,"15min":1.8640000000000001},"min":{"1min":0.97199999999999998,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":2.0880000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.597,"15min":1.956},"min":{"1min":0.81699999999999995,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":1.8300000000000001,"5min":12.567,"15min":29.991},"last":1.3300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542335,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76728,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295112,"statfs":{"total":96636764160,"available":96558194688,"internally_reserved":0,"allocated":1220608,"data_stored":525489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.405,"15min":1.5389999999999999},"min":{"1min":0.79700000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.6579999999999999,"5min":6.8380000000000001,"15min":19.382999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.3740000000000001,"15min":1.3899999999999999},"min":{"1min":0.77000000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":13.127000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":1.008}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.5409999999999999,"15min":1.6339999999999999},"min":{"1min":0.70699999999999996,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":3.8109999999999999,"5min":13.571,"15min":17.693999999999999},"last":1.214},{"interface":"front","average":{"1min":1.996,"5min":1.6419999999999999,"15min":1.6950000000000001},"min":{"1min":0.95099999999999996,"5min":0.51600000000000001,"15min":0.432},"max":{"1min":17.062000000000001,"5min":17.062000000000001,"15min":20.645},"last":1.395}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.5740000000000001,"15min":1.6879999999999999},"min":{"1min":0.79100000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.77,"5min":6.4039999999999999,"15min":21.138000000000002},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.462,"15min":1.4910000000000001},"min":{"1min":0.995,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":4.0720000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.5349999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4930000000000001,"15min":1.8},"min":{"1min":0.95499999999999996,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":2.4769999999999999,"5min":7.984,"15min":23.109000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.6060000000000001,"15min":1.7949999999999999},"min":{"1min":1.0169999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.95,"5min":6.4610000000000003,"15min":19.303000000000001},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5449999999999999,"15min":1.849},"min":{"1min":0.85799999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":9.3309999999999995,"5min":10.831,"15min":24.436},"last":1.617},{"interface":"front","average":{"1min":1.329,"5min":1.48,"15min":1.744},"min":{"1min":0.84699999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":1.6140000000000001,"5min":5.6820000000000004,"15min":19.126000000000001},"last":1.5800000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.526,"15min":1.635},"min":{"1min":0.95599999999999996,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":1.718,"5min":12.914,"15min":21.192},"last":1.341},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5609999999999999,"15min":1.978},"min":{"1min":0.97299999999999998,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":2.0299999999999998,"5min":7.8609999999999998,"15min":27.042000000000002},"last":1.764}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7989999999999999,"15min":1.9350000000000001},"min":{"1min":0.92600000000000005,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":13.034000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":1.663},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.7410000000000001,"15min":2.0840000000000001},"min":{"1min":1.216,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":2.1429999999999998,"5min":10.756,"15min":24.367000000000001},"last":1.8440000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542335,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":527053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.5,"15min":1.278},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":1.403,"5min":28.437999999999999,"15min":31.562999999999999},"last":1.238},{"interface":"front","average":{"1min":1.117,"5min":1.26,"15min":1.173},"min":{"1min":0.65900000000000003,"5min":0.40600000000000003,"15min":0.39600000000000002},"max":{"1min":1.8440000000000001,"5min":11.361000000000001,"15min":11.361000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.236,"15min":1.145},"min":{"1min":0.59999999999999998,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.5940000000000001,"5min":10.856999999999999,"15min":15.569000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.7050000000000001,"15min":1.429},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.643,"5min":28.352,"15min":28.352},"last":1.103}]},{"osd":2,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":2.379,"15min":1.6879999999999999},"min":{"1min":0.71999999999999997,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":1.8109999999999999,"5min":39.265999999999998,"15min":39.265999999999998},"last":1.351},{"interface":"front","average":{"1min":1.264,"5min":1.4970000000000001,"15min":1.385},"min":{"1min":0.67800000000000005,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":3.911,"5min":13.526,"15min":14.577999999999999},"last":1.1679999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.516,"15min":1.5620000000000001},"min":{"1min":0.86499999999999999,"5min":0.64900000000000002,"15min":0.53800000000000003},"max":{"1min":2.1019999999999999,"5min":6.0629999999999997,"15min":20.978999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.448,"5min":2.2879999999999998,"15min":1.8480000000000001},"min":{"1min":1.034,"5min":0.93100000000000005,"15min":0.72999999999999998},"max":{"1min":2.0030000000000001,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.472}]},{"osd":5,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.746,"15min":1.766},"min":{"1min":0.81699999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":1.911,"5min":10.988,"15min":20.663},"last":1.512},{"interface":"front","average":{"1min":1.575,"5min":2.3090000000000002,"15min":1.9490000000000001},"min":{"1min":0.96299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1019999999999999,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.637}]},{"osd":6,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.77,"15min":1.754},"min":{"1min":1.0069999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.2170000000000001,"5min":13.691000000000001,"15min":13.691000000000001},"last":1.321},{"interface":"front","average":{"1min":2.0910000000000002,"5min":2.0819999999999999,"15min":1.9510000000000001},"min":{"1min":0.95099999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":15.114000000000001,"5min":26.698,"15min":26.698},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.905,"15min":1.776},"min":{"1min":1.0589999999999999,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.9630000000000001,"5min":13.622,"15min":13.622},"last":1.431},{"interface":"front","average":{"1min":1.5289999999999999,"5min":2.0459999999999998,"15min":2.1659999999999999},"min":{"1min":0.90400000000000003,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.2789999999999999,"5min":15.372999999999999,"15min":49.811},"last":1.671}]}]},{"osd":2,"up_from":14,"seq":60129542334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76732,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295108,"statfs":{"total":96636764160,"available":96558190592,"internally_reserved":0,"allocated":1220608,"data_stored":525453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.284,"15min":1.0449999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.7130000000000001,"5min":18.289999999999999,"15min":18.289999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.2210000000000001,"15min":1.099},"min":{"1min":0.65100000000000002,"5min":0.40699999999999997,"15min":0.28599999999999998},"max":{"1min":1.6000000000000001,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.4650000000000001,"15min":1.1850000000000001},"min":{"1min":0.55900000000000005,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":1.8979999999999999,"5min":23.135999999999999,"15min":23.135999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.278,"15min":1.2050000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.6699999999999999,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":1.016}]},{"osd":3,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3859999999999999,"15min":1.1839999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":7.306,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.421,"5min":1.647,"15min":1.4510000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.839,"5min":19.375,"15min":19.375},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.8140000000000001,"15min":1.5620000000000001},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.59999999999999998},"max":{"1min":2.0249999999999999,"5min":23.225000000000001,"15min":23.225000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.651,"15min":1.5600000000000001},"min":{"1min":1.036,"5min":0.72699999999999998,"15min":0.59199999999999997},"max":{"1min":1.7509999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.335}]},{"osd":5,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.8580000000000001,"15min":1.6200000000000001},"min":{"1min":1.0740000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":7.7640000000000002,"5min":20.157,"15min":20.157},"last":1.2},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8089999999999999,"15min":1.601},"min":{"1min":0.84299999999999997,"5min":0.623,"15min":0.623},"max":{"1min":7.9820000000000002,"5min":25.004000000000001,"15min":25.004000000000001},"last":1.7090000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":2.1040000000000001,"15min":1.7390000000000001},"min":{"1min":0.89400000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":7.6630000000000003,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.948,"15min":1.839},"min":{"1min":0.81100000000000005,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":7.8680000000000003,"5min":28.760999999999999,"15min":28.760999999999999},"last":1.49}]},{"osd":7,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":2.2919999999999998,"15min":1.772},"min":{"1min":1.05,"5min":0.82099999999999995,"15min":0.64100000000000001},"max":{"1min":3.1219999999999999,"5min":22.834,"15min":22.834},"last":1.554},{"interface":"front","average":{"1min":1.53,"5min":2.2879999999999998,"15min":2.069},"min":{"1min":0.91000000000000003,"5min":0.872,"15min":0.82799999999999996},"max":{"1min":1.931,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.4279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72560,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299280,"statfs":{"total":96636764160,"available":96562462720,"internally_reserved":0,"allocated":1216512,"data_stored":525419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.9850000000000001,"15min":1.3260000000000001},"min":{"1min":0.48599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.256,"5min":31.178999999999998,"15min":31.178999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.4470000000000001,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.901,"5min":26.859999999999999,"15min":26.859999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1439999999999999,"15min":1.143},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":3.5830000000000002,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.9239999999999999,"15min":1.5529999999999999},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":3.4929999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":0.84599999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.879,"15min":1.651},"min":{"1min":0.56299999999999994,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.8049999999999999,"5min":27.609999999999999,"15min":27.609999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.1709999999999998,"15min":1.6559999999999999},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":2.9750000000000001,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.3069999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4390000000000001,"15min":1.466},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.58099999999999996},"max":{"1min":2.3069999999999999,"5min":4.7249999999999996,"15min":10.577999999999999},"last":1.46},{"interface":"front","average":{"1min":1.411,"5min":2.0760000000000001,"15min":1.74},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.0310000000000001,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.411,"15min":1.5449999999999999},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.032,"5min":3.032,"15min":17.469000000000001},"last":1.508},{"interface":"front","average":{"1min":1.377,"5min":1.522,"15min":1.6060000000000001},"min":{"1min":1.0229999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1859999999999999,"5min":4.6120000000000001,"15min":8.2989999999999995},"last":1.077}]},{"osd":6,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5569999999999999,"15min":1.7150000000000001},"min":{"1min":0.85999999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.9009999999999998,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.611},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5549999999999999,"15min":1.873},"min":{"1min":1.0800000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5760000000000001,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.2669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6419999999999999,"15min":1.899},"min":{"1min":0.94799999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4249999999999998,"5min":4.6989999999999998,"15min":30.834},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.2589999999999999,"15min":2.0609999999999999},"min":{"1min":0.96799999999999997,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":1.9890000000000001,"5min":27.716999999999999,"15min":27.716999999999999},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542334,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73152,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298688,"statfs":{"total":96636764160,"available":96561856512,"internally_reserved":0,"allocated":1814528,"data_stored":1116369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.103,"15min":1.1879999999999999},"min":{"1min":0.52100000000000002,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.73,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.353,"15min":1.4970000000000001},"min":{"1min":0.82299999999999995,"5min":0.42499999999999999,"15min":0.33900000000000002},"max":{"1min":1.623,"5min":26.459,"15min":50.429000000000002},"last":0.93500000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":0.66100000000000003,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.8009999999999999,"5min":20.041,"15min":22.716999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.452,"15min":1.6160000000000001},"min":{"1min":0.78400000000000003,"5min":0.503,"15min":0.38},"max":{"1min":1.7430000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":1.252}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2410000000000001,"15min":1.218},"min":{"1min":0.749,"5min":0.40000000000000002,"15min":0.372},"max":{"1min":1.8129999999999999,"5min":5.8719999999999999,"15min":10.795},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.425,"15min":1.5820000000000001},"min":{"1min":0.70799999999999996,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.8280000000000001,"5min":9.484,"15min":22.254999999999999},"last":1.462}]},{"osd":4,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5,"15min":1.806},"min":{"1min":0.78800000000000003,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.986,"5min":3.7749999999999999,"15min":22.407},"last":1.536},{"interface":"front","average":{"1min":1.48,"5min":1.6930000000000001,"15min":1.919},"min":{"1min":1.1970000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.907,"5min":26.57,"15min":50.497},"last":1.365}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.401,"15min":1.6220000000000001},"min":{"1min":0.83199999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.0550000000000002,"5min":4.7359999999999998,"15min":14.112},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.7,"15min":1.847},"min":{"1min":1.05,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.9430000000000001,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.5920000000000001,"15min":1.7769999999999999},"min":{"1min":0.871,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.968,"5min":8.2349999999999994,"15min":31.387},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.841,"15min":1.9650000000000001},"min":{"1min":1.01,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":3.5209999999999999,"5min":9.2840000000000007,"15min":23.547000000000001},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":2.0190000000000001,"15min":2.0680000000000001},"min":{"1min":1.292,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.21,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.679},{"interface":"front","average":{"1min":1.639,"5min":1.8660000000000001,"15min":1.835},"min":{"1min":1.0349999999999999,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.1859999999999999,"5min":12.455,"15min":15.747},"last":1.5760000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:27.522 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:27.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.804+0000 7f53c5600640 1 Processor -- start 2024-01-16T15:15:27.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.804+0000 7f53c5600640 1 -- start start 2024-01-16T15:15:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.805+0000 7f53c5600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53c0143c40 0x7f53c0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.805+0000 7f53c5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.805+0000 7f53c5600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f53c00cca40 0x7f53c015eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.805+0000 7f53c5600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00723c0 con 0x7f53c0143c40 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.805+0000 7f53c5600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c0070e50 con 0x7f53c00cca40 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.805+0000 7f53c5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00cce70 con 0x7f53c00717d0 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53c0143c40 0x7f53c0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f53c0143c40 0x7f53c0071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:35990/0 (socket says 172.21.15.53:35990) 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 -- 172.21.15.53:0/1947922723 learned_addr learned my addr 172.21.15.53:0/1947922723 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 -- 172.21.15.53:0/1947922723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 msgr2=0x7f53c00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 --2- 172.21.15.53:0/1947922723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 -- 172.21.15.53:0/1947922723 >> v2:172.21.15.146:3300/0 conn(0x7f53c00cca40 msgr2=0x7f53c015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:27.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 --2- 172.21.15.53:0/1947922723 >> v2:172.21.15.146:3300/0 conn(0x7f53c00cca40 0x7f53c015eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.806+0000 7f53bfe00640 1 -- 172.21.15.53:0/1947922723 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c0144020 con 0x7f53c0143c40 2024-01-16T15:15:27.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.807+0000 7f53bfe00640 1 --2- 172.21.15.53:0/1947922723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c0143c40 0x7f53c0071290 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f53c00cc3f0 tx=0x7f53b002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f926962343ff6277 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.808+0000 7f53bea00640 1 -- 172.21.15.53:0/1947922723 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b00317e0 con 0x7f53c0143c40 2024-01-16T15:15:27.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.808+0000 7f53bea00640 1 -- 172.21.15.53:0/1947922723 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b0031940 con 0x7f53c0143c40 2024-01-16T15:15:27.793 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.808+0000 7f53bea00640 1 -- 172.21.15.53:0/1947922723 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b0031cd0 con 0x7f53c0143c40 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.808+0000 7f53c5600640 1 -- 172.21.15.53:0/1947922723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c0143c40 msgr2=0x7f53c0071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.808+0000 7f53c5600640 1 --2- 172.21.15.53:0/1947922723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c0143c40 0x7f53c0071290 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f53c00cc3f0 tx=0x7f53b002b880 comp rx=0 tx=0).stop 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 -- 172.21.15.53:0/1947922723 shutdown_connections 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 --2- 172.21.15.53:0/1947922723 >> v2:172.21.15.146:3300/0 conn(0x7f53c00cca40 0x7f53c015eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 --2- 172.21.15.53:0/1947922723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 --2- 172.21.15.53:0/1947922723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c0143c40 0x7f53c0071290 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 -- 172.21.15.53:0/1947922723 >> 172.21.15.53:0/1947922723 conn(0x7f53c006a800 msgr2=0x7f53c006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 -- 172.21.15.53:0/1947922723 shutdown_connections 2024-01-16T15:15:27.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.809+0000 7f53c5600640 1 -- 172.21.15.53:0/1947922723 wait complete. 2024-01-16T15:15:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.811+0000 7f53c5600640 1 Processor -- start 2024-01-16T15:15:27.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.812+0000 7f53c5600640 1 -- start start 2024-01-16T15:15:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53c5600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00b2e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53c5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 0x7f53c013e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53c5600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53c0143c40 0x7f53c0138bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53c5600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c013d550 con 0x7f53c00cca40 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53c5600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c013d830 con 0x7f53c0143c40 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53c5600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c013db10 con 0x7f53c00717d0 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53bf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 0x7f53c013e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53bf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 0x7f53c013e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36004/0 (socket says 172.21.15.53:36004) 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.813+0000 7f53bf400640 1 -- 172.21.15.53:0/4129597229 learned_addr learned my addr 172.21.15.53:0/4129597229 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bfe00640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00b2e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bf400640 1 -- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 msgr2=0x7f53c00b2e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bf400640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00b2e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bf400640 1 -- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53c0143c40 msgr2=0x7f53c0138bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bf400640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53c0143c40 0x7f53c0138bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bf400640 1 -- 172.21.15.53:0/4129597229 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b00094d0 con 0x7f53c00cca40 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bfe00640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:27.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53bf400640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 0x7f53c013e310 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f53a800b410 tx=0x7f53a800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8011830 con 0x7f53c00cca40 2024-01-16T15:15:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a80119b0 con 0x7f53c00cca40 2024-01-16T15:15:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c013ddf0 con 0x7f53c00cca40 2024-01-16T15:15:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.814+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53a8011d60 con 0x7f53c00cca40 2024-01-16T15:15:27.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.815+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c013d120 con 0x7f53c00cca40 2024-01-16T15:15:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.816+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c01437a0 con 0x7f53c00cca40 2024-01-16T15:15:27.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.817+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f53a801a400 con 0x7f53c00cca40 2024-01-16T15:15:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.818+0000 7f53b7e00640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f539806c160 0x7f539806e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:27.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.818+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f53a8088b10 con 0x7f53c00cca40 2024-01-16T15:15:27.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.819+0000 7f53bfe00640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f539806c160 0x7f539806e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:27.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.820+0000 7f53bfe00640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f539806c160 0x7f539806e620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f53b0036000 tx=0x7f53b003d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:27.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:27.824+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f53a80550f0 con 0x7f53c00cca40 2024-01-16T15:15:27.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.013+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f53c006c960 con 0x7f539806c160 2024-01-16T15:15:28.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.036+0000 7f53b7e00640 1 -- 172.21.15.53:0/4129597229 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+285572 (secure 0 0 0) 0x7f53b0031c30 con 0x7f539806c160 2024-01-16T15:15:28.025 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:28.026 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:15:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f539806c160 msgr2=0x7f539806e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f539806c160 0x7f539806e620 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f53b0036000 tx=0x7f53b003d040 comp rx=0 tx=0).stop 2024-01-16T15:15:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 msgr2=0x7f53c013e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 0x7f53c013e310 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f53a800b410 tx=0x7f53a800b8d0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 shutdown_connections 2024-01-16T15:15:28.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f539806c160 0x7f539806e620 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f53c0143c40 0x7f53c0138bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53c00cca40 0x7f53c013e310 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 --2- 172.21.15.53:0/4129597229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f53c00717d0 0x7f53c00b2e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.041+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 >> 172.21.15.53:0/4129597229 conn(0x7f53c006a800 msgr2=0x7f53c0142800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:28.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.042+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 shutdown_connections 2024-01-16T15:15:28.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.042+0000 7f53c5600640 1 -- 172.21.15.53:0/4129597229 wait complete. 2024-01-16T15:15:28.045 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":563,"stamp":"2024-01-16T15:15:26.579691+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":591096,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":193,"num_write_kb":2426,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146,"ondisk_log_size":146,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581640,"kb_used_data":10640,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":754393080,"statfs":{"total":773094113280,"available":772498513920,"internally_reserved":0,"allocated":10895360,"data_stored":5353955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":584568368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":22,"apply_latency_ms":22,"commit_latency_ns":22000000,"apply_latency_ns":22000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001591"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691428+0000","last_change":"2024-01-16T15:15:22.537519+0000","last_active":"2024-01-16T15:15:23.691428+0000","last_peered":"2024-01-16T15:15:23.691428+0000","last_clean":"2024-01-16T15:15:23.691428+0000","last_became_active":"2024-01-16T15:15:22.535630+0000","last_became_peered":"2024-01-16T15:15:22.535630+0000","last_unstale":"2024-01-16T15:15:23.691428+0000","last_undegraded":"2024-01-16T15:15:23.691428+0000","last_fullsized":"2024-01-16T15:15:23.691428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:53.424772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740792+0000","last_change":"2024-01-16T15:15:22.532497+0000","last_active":"2024-01-16T15:15:23.740792+0000","last_peered":"2024-01-16T15:15:23.740792+0000","last_clean":"2024-01-16T15:15:23.740792+0000","last_became_active":"2024-01-16T15:15:22.531043+0000","last_became_peered":"2024-01-16T15:15:22.531043+0000","last_unstale":"2024-01-16T15:15:23.740792+0000","last_undegraded":"2024-01-16T15:15:23.740792+0000","last_fullsized":"2024-01-16T15:15:23.740792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691119+0000","last_change":"2024-01-16T15:15:22.535023+0000","last_active":"2024-01-16T15:15:23.691119+0000","last_peered":"2024-01-16T15:15:23.691119+0000","last_clean":"2024-01-16T15:15:23.691119+0000","last_became_active":"2024-01-16T15:15:22.534684+0000","last_became_peered":"2024-01-16T15:15:22.534684+0000","last_unstale":"2024-01-16T15:15:23.691119+0000","last_undegraded":"2024-01-16T15:15:23.691119+0000","last_fullsized":"2024-01-16T15:15:23.691119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:00.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509733+0000","last_change":"2024-01-16T15:15:22.524150+0000","last_active":"2024-01-16T15:15:23.509733+0000","last_peered":"2024-01-16T15:15:23.509733+0000","last_clean":"2024-01-16T15:15:23.509733+0000","last_became_active":"2024-01-16T15:15:22.522946+0000","last_became_peered":"2024-01-16T15:15:22.522946+0000","last_unstale":"2024-01-16T15:15:23.509733+0000","last_undegraded":"2024-01-16T15:15:23.509733+0000","last_fullsized":"2024-01-16T15:15:23.509733+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:13.223095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.519050+0000","last_change":"2024-01-16T15:15:21.519050+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.519050+0000","last_undegraded":"2024-01-16T15:15:21.519050+0000","last_fullsized":"2024-01-16T15:15:21.519050+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005648+0000","last_change":"2024-01-16T15:15:22.536600+0000","last_active":"2024-01-16T15:15:24.005648+0000","last_peered":"2024-01-16T15:15:24.005648+0000","last_clean":"2024-01-16T15:15:24.005648+0000","last_became_active":"2024-01-16T15:15:22.536327+0000","last_became_peered":"2024-01-16T15:15:22.536327+0000","last_unstale":"2024-01-16T15:15:24.005648+0000","last_undegraded":"2024-01-16T15:15:24.005648+0000","last_fullsized":"2024-01-16T15:15:24.005648+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005398+0000","last_change":"2024-01-16T15:15:22.535575+0000","last_active":"2024-01-16T15:15:24.005398+0000","last_peered":"2024-01-16T15:15:24.005398+0000","last_clean":"2024-01-16T15:15:24.005398+0000","last_became_active":"2024-01-16T15:15:22.535292+0000","last_became_peered":"2024-01-16T15:15:22.535292+0000","last_unstale":"2024-01-16T15:15:24.005398+0000","last_undegraded":"2024-01-16T15:15:24.005398+0000","last_fullsized":"2024-01-16T15:15:24.005398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:06.443796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740110+0000","last_change":"2024-01-16T15:15:22.521459+0000","last_active":"2024-01-16T15:15:23.740110+0000","last_peered":"2024-01-16T15:15:23.740110+0000","last_clean":"2024-01-16T15:15:23.740110+0000","last_became_active":"2024-01-16T15:15:22.520884+0000","last_became_peered":"2024-01-16T15:15:22.520884+0000","last_unstale":"2024-01-16T15:15:23.740110+0000","last_undegraded":"2024-01-16T15:15:23.740110+0000","last_fullsized":"2024-01-16T15:15:23.740110+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.513402+0000","last_change":"2024-01-16T15:15:21.513402+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.513402+0000","last_undegraded":"2024-01-16T15:15:21.513402+0000","last_fullsized":"2024-01-16T15:15:21.513402+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.520840+0000","last_change":"2024-01-16T15:15:21.520840+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.520840+0000","last_undegraded":"2024-01-16T15:15:21.520840+0000","last_fullsized":"2024-01-16T15:15:21.520840+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005800+0000","last_change":"2024-01-16T15:15:22.533081+0000","last_active":"2024-01-16T15:15:24.005800+0000","last_peered":"2024-01-16T15:15:24.005800+0000","last_clean":"2024-01-16T15:15:24.005800+0000","last_became_active":"2024-01-16T15:15:22.529012+0000","last_became_peered":"2024-01-16T15:15:22.529012+0000","last_unstale":"2024-01-16T15:15:24.005800+0000","last_undegraded":"2024-01-16T15:15:24.005800+0000","last_fullsized":"2024-01-16T15:15:24.005800+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.510864+0000","last_change":"2024-01-16T15:15:21.510864+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.510864+0000","last_undegraded":"2024-01-16T15:15:21.510864+0000","last_fullsized":"2024-01-16T15:15:21.510864+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509163+0000","last_change":"2024-01-16T15:15:22.532050+0000","last_active":"2024-01-16T15:15:23.509163+0000","last_peered":"2024-01-16T15:15:23.509163+0000","last_clean":"2024-01-16T15:15:23.509163+0000","last_became_active":"2024-01-16T15:15:22.531307+0000","last_became_peered":"2024-01-16T15:15:22.531307+0000","last_unstale":"2024-01-16T15:15:23.509163+0000","last_undegraded":"2024-01-16T15:15:23.509163+0000","last_fullsized":"2024-01-16T15:15:23.509163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:34.860510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609381+0000","last_change":"2024-01-16T15:15:22.533322+0000","last_active":"2024-01-16T15:15:23.609381+0000","last_peered":"2024-01-16T15:15:23.609381+0000","last_clean":"2024-01-16T15:15:23.609381+0000","last_became_active":"2024-01-16T15:15:22.532960+0000","last_became_peered":"2024-01-16T15:15:22.532960+0000","last_unstale":"2024-01-16T15:15:23.609381+0000","last_undegraded":"2024-01-16T15:15:23.609381+0000","last_fullsized":"2024-01-16T15:15:23.609381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:47.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508639+0000","last_change":"2024-01-16T15:15:22.532776+0000","last_active":"2024-01-16T15:15:23.508639+0000","last_peered":"2024-01-16T15:15:23.508639+0000","last_clean":"2024-01-16T15:15:23.508639+0000","last_became_active":"2024-01-16T15:15:22.532494+0000","last_became_peered":"2024-01-16T15:15:22.532494+0000","last_unstale":"2024-01-16T15:15:23.508639+0000","last_undegraded":"2024-01-16T15:15:23.508639+0000","last_fullsized":"2024-01-16T15:15:23.508639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005108+0000","last_change":"2024-01-16T15:15:22.530744+0000","last_active":"2024-01-16T15:15:24.005108+0000","last_peered":"2024-01-16T15:15:24.005108+0000","last_clean":"2024-01-16T15:15:24.005108+0000","last_became_active":"2024-01-16T15:15:22.525764+0000","last_became_peered":"2024-01-16T15:15:22.525764+0000","last_unstale":"2024-01-16T15:15:24.005108+0000","last_undegraded":"2024-01-16T15:15:24.005108+0000","last_fullsized":"2024-01-16T15:15:24.005108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:51.206294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741415+0000","last_change":"2024-01-16T15:15:22.523512+0000","last_active":"2024-01-16T15:15:23.741415+0000","last_peered":"2024-01-16T15:15:23.741415+0000","last_clean":"2024-01-16T15:15:23.741415+0000","last_became_active":"2024-01-16T15:15:22.521121+0000","last_became_peered":"2024-01-16T15:15:22.521121+0000","last_unstale":"2024-01-16T15:15:23.741415+0000","last_undegraded":"2024-01-16T15:15:23.741415+0000","last_fullsized":"2024-01-16T15:15:23.741415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:35.046638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509378+0000","last_change":"2024-01-16T15:15:22.531166+0000","last_active":"2024-01-16T15:15:23.509378+0000","last_peered":"2024-01-16T15:15:23.509378+0000","last_clean":"2024-01-16T15:15:23.509378+0000","last_became_active":"2024-01-16T15:15:22.530595+0000","last_became_peered":"2024-01-16T15:15:22.530595+0000","last_unstale":"2024-01-16T15:15:23.509378+0000","last_undegraded":"2024-01-16T15:15:23.509378+0000","last_fullsized":"2024-01-16T15:15:23.509378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:01.070526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610445+0000","last_change":"2024-01-16T15:15:22.524237+0000","last_active":"2024-01-16T15:15:23.610445+0000","last_peered":"2024-01-16T15:15:23.610445+0000","last_clean":"2024-01-16T15:15:23.610445+0000","last_became_active":"2024-01-16T15:15:22.523982+0000","last_became_peered":"2024-01-16T15:15:22.523982+0000","last_unstale":"2024-01-16T15:15:23.610445+0000","last_undegraded":"2024-01-16T15:15:23.610445+0000","last_fullsized":"2024-01-16T15:15:23.610445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:34.668807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.508179+0000","last_change":"2024-01-16T15:15:21.508179+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.508179+0000","last_undegraded":"2024-01-16T15:15:21.508179+0000","last_fullsized":"2024-01-16T15:15:21.508179+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609784+0000","last_change":"2024-01-16T15:15:22.529446+0000","last_active":"2024-01-16T15:15:23.609784+0000","last_peered":"2024-01-16T15:15:23.609784+0000","last_clean":"2024-01-16T15:15:23.609784+0000","last_became_active":"2024-01-16T15:15:22.529124+0000","last_became_peered":"2024-01-16T15:15:22.529124+0000","last_unstale":"2024-01-16T15:15:23.609784+0000","last_undegraded":"2024-01-16T15:15:23.609784+0000","last_fullsized":"2024-01-16T15:15:23.609784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:14.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740836+0000","last_change":"2024-01-16T15:15:22.541357+0000","last_active":"2024-01-16T15:15:23.740836+0000","last_peered":"2024-01-16T15:15:23.740836+0000","last_clean":"2024-01-16T15:15:23.740836+0000","last_became_active":"2024-01-16T15:15:22.539470+0000","last_became_peered":"2024-01-16T15:15:22.539470+0000","last_unstale":"2024-01-16T15:15:23.740836+0000","last_undegraded":"2024-01-16T15:15:23.740836+0000","last_fullsized":"2024-01-16T15:15:23.740836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740324+0000","last_change":"2024-01-16T15:15:22.528944+0000","last_active":"2024-01-16T15:15:23.740324+0000","last_peered":"2024-01-16T15:15:23.740324+0000","last_clean":"2024-01-16T15:15:23.740324+0000","last_became_active":"2024-01-16T15:15:22.528668+0000","last_became_peered":"2024-01-16T15:15:22.528668+0000","last_unstale":"2024-01-16T15:15:23.740324+0000","last_undegraded":"2024-01-16T15:15:23.740324+0000","last_fullsized":"2024-01-16T15:15:23.740324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005396+0000","last_change":"2024-01-16T15:15:22.527598+0000","last_active":"2024-01-16T15:15:24.005396+0000","last_peered":"2024-01-16T15:15:24.005396+0000","last_clean":"2024-01-16T15:15:24.005396+0000","last_became_active":"2024-01-16T15:15:22.526241+0000","last_became_peered":"2024-01-16T15:15:22.526241+0000","last_unstale":"2024-01-16T15:15:24.005396+0000","last_undegraded":"2024-01-16T15:15:24.005396+0000","last_fullsized":"2024-01-16T15:15:24.005396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:41.199256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.522680+0000","last_change":"2024-01-16T15:15:21.522680+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.522680+0000","last_undegraded":"2024-01-16T15:15:21.522680+0000","last_fullsized":"2024-01-16T15:15:21.522680+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741073+0000","last_change":"2024-01-16T15:15:22.518878+0000","last_active":"2024-01-16T15:15:23.741073+0000","last_peered":"2024-01-16T15:15:23.741073+0000","last_clean":"2024-01-16T15:15:23.741073+0000","last_became_active":"2024-01-16T15:15:22.518594+0000","last_became_peered":"2024-01-16T15:15:22.518594+0000","last_unstale":"2024-01-16T15:15:23.741073+0000","last_undegraded":"2024-01-16T15:15:23.741073+0000","last_fullsized":"2024-01-16T15:15:23.741073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006005+0000","last_change":"2024-01-16T15:15:22.529986+0000","last_active":"2024-01-16T15:15:24.006005+0000","last_peered":"2024-01-16T15:15:24.006005+0000","last_clean":"2024-01-16T15:15:24.006005+0000","last_became_active":"2024-01-16T15:15:22.525092+0000","last_became_peered":"2024-01-16T15:15:22.525092+0000","last_unstale":"2024-01-16T15:15:24.006005+0000","last_undegraded":"2024-01-16T15:15:24.006005+0000","last_fullsized":"2024-01-16T15:15:24.006005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.522614+0000","last_change":"2024-01-16T15:15:21.522614+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.522614+0000","last_undegraded":"2024-01-16T15:15:21.522614+0000","last_fullsized":"2024-01-16T15:15:21.522614+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740772+0000","last_change":"2024-01-16T15:15:22.518786+0000","last_active":"2024-01-16T15:15:23.740772+0000","last_peered":"2024-01-16T15:15:23.740772+0000","last_clean":"2024-01-16T15:15:23.740772+0000","last_became_active":"2024-01-16T15:15:22.512965+0000","last_became_peered":"2024-01-16T15:15:22.512965+0000","last_unstale":"2024-01-16T15:15:23.740772+0000","last_undegraded":"2024-01-16T15:15:23.740772+0000","last_fullsized":"2024-01-16T15:15:23.740772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609350+0000","last_change":"2024-01-16T15:15:22.521137+0000","last_active":"2024-01-16T15:15:23.609350+0000","last_peered":"2024-01-16T15:15:23.609350+0000","last_clean":"2024-01-16T15:15:23.609350+0000","last_became_active":"2024-01-16T15:15:22.520842+0000","last_became_peered":"2024-01-16T15:15:22.520842+0000","last_unstale":"2024-01-16T15:15:23.609350+0000","last_undegraded":"2024-01-16T15:15:23.609350+0000","last_fullsized":"2024-01-16T15:15:23.609350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:02.829205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508652+0000","last_change":"2024-01-16T15:15:22.541257+0000","last_active":"2024-01-16T15:15:23.508652+0000","last_peered":"2024-01-16T15:15:23.508652+0000","last_clean":"2024-01-16T15:15:23.508652+0000","last_became_active":"2024-01-16T15:15:22.540754+0000","last_became_peered":"2024-01-16T15:15:22.540754+0000","last_unstale":"2024-01-16T15:15:23.508652+0000","last_undegraded":"2024-01-16T15:15:23.508652+0000","last_fullsized":"2024-01-16T15:15:23.508652+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:10.664582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690290+0000","last_change":"2024-01-16T15:15:22.516532+0000","last_active":"2024-01-16T15:15:23.690290+0000","last_peered":"2024-01-16T15:15:23.690290+0000","last_clean":"2024-01-16T15:15:23.690290+0000","last_became_active":"2024-01-16T15:15:22.515440+0000","last_became_peered":"2024-01-16T15:15:22.515440+0000","last_unstale":"2024-01-16T15:15:23.690290+0000","last_undegraded":"2024-01-16T15:15:23.690290+0000","last_fullsized":"2024-01-16T15:15:23.690290+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:42.093044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610359+0000","last_change":"2024-01-16T15:15:22.521967+0000","last_active":"2024-01-16T15:15:23.610359+0000","last_peered":"2024-01-16T15:15:23.610359+0000","last_clean":"2024-01-16T15:15:23.610359+0000","last_became_active":"2024-01-16T15:15:22.520663+0000","last_became_peered":"2024-01-16T15:15:22.520663+0000","last_unstale":"2024-01-16T15:15:23.610359+0000","last_undegraded":"2024-01-16T15:15:23.610359+0000","last_fullsized":"2024-01-16T15:15:23.610359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:27.652785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741472+0000","last_change":"2024-01-16T15:15:21.529234+0000","last_active":"2024-01-16T15:15:23.741472+0000","last_peered":"2024-01-16T15:15:23.741472+0000","last_clean":"2024-01-16T15:15:23.741472+0000","last_became_active":"2024-01-16T15:15:21.528711+0000","last_became_peered":"2024-01-16T15:15:21.528711+0000","last_unstale":"2024-01-16T15:15:23.741472+0000","last_undegraded":"2024-01-16T15:15:23.741472+0000","last_fullsized":"2024-01-16T15:15:23.741472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:34.638361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509939+0000","last_change":"2024-01-16T15:15:22.540312+0000","last_active":"2024-01-16T15:15:23.509939+0000","last_peered":"2024-01-16T15:15:23.509939+0000","last_clean":"2024-01-16T15:15:23.509939+0000","last_became_active":"2024-01-16T15:15:22.538619+0000","last_became_peered":"2024-01-16T15:15:22.538619+0000","last_unstale":"2024-01-16T15:15:23.509939+0000","last_undegraded":"2024-01-16T15:15:23.509939+0000","last_fullsized":"2024-01-16T15:15:23.509939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:02.521266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.542686+0000","last_change":"2024-01-16T15:15:21.516982+0000","last_active":"2024-01-16T15:15:22.542686+0000","last_peered":"2024-01-16T15:15:22.542686+0000","last_clean":"2024-01-16T15:15:22.542686+0000","last_became_active":"2024-01-16T15:15:21.516459+0000","last_became_peered":"2024-01-16T15:15:21.516459+0000","last_unstale":"2024-01-16T15:15:22.542686+0000","last_undegraded":"2024-01-16T15:15:22.542686+0000","last_fullsized":"2024-01-16T15:15:22.542686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690957+0000","last_change":"2024-01-16T15:15:22.522691+0000","last_active":"2024-01-16T15:15:23.690957+0000","last_peered":"2024-01-16T15:15:23.690957+0000","last_clean":"2024-01-16T15:15:23.690957+0000","last_became_active":"2024-01-16T15:15:22.522375+0000","last_became_peered":"2024-01-16T15:15:22.522375+0000","last_unstale":"2024-01-16T15:15:23.690957+0000","last_undegraded":"2024-01-16T15:15:23.690957+0000","last_fullsized":"2024-01-16T15:15:23.690957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:12.190032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740678+0000","last_change":"2024-01-16T15:15:21.529660+0000","last_active":"2024-01-16T15:15:23.740678+0000","last_peered":"2024-01-16T15:15:23.740678+0000","last_clean":"2024-01-16T15:15:23.740678+0000","last_became_active":"2024-01-16T15:15:21.529009+0000","last_became_peered":"2024-01-16T15:15:21.529009+0000","last_unstale":"2024-01-16T15:15:23.740678+0000","last_undegraded":"2024-01-16T15:15:23.740678+0000","last_fullsized":"2024-01-16T15:15:23.740678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:49.081309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.508270+0000","last_change":"2024-01-16T15:15:21.508270+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.508270+0000","last_undegraded":"2024-01-16T15:15:21.508270+0000","last_fullsized":"2024-01-16T15:15:21.508270+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740604+0000","last_change":"2024-01-16T15:15:21.521824+0000","last_active":"2024-01-16T15:15:23.740604+0000","last_peered":"2024-01-16T15:15:23.740604+0000","last_clean":"2024-01-16T15:15:23.740604+0000","last_became_active":"2024-01-16T15:15:21.521406+0000","last_became_peered":"2024-01-16T15:15:21.521406+0000","last_unstale":"2024-01-16T15:15:23.740604+0000","last_undegraded":"2024-01-16T15:15:23.740604+0000","last_fullsized":"2024-01-16T15:15:23.740604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:39.695221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004361+0000","last_change":"2024-01-16T15:15:22.540055+0000","last_active":"2024-01-16T15:15:24.004361+0000","last_peered":"2024-01-16T15:15:24.004361+0000","last_clean":"2024-01-16T15:15:24.004361+0000","last_became_active":"2024-01-16T15:15:22.538674+0000","last_became_peered":"2024-01-16T15:15:22.538674+0000","last_unstale":"2024-01-16T15:15:24.004361+0000","last_undegraded":"2024-01-16T15:15:24.004361+0000","last_fullsized":"2024-01-16T15:15:24.004361+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:37.034062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690792+0000","last_change":"2024-01-16T15:15:21.530799+0000","last_active":"2024-01-16T15:15:23.690792+0000","last_peered":"2024-01-16T15:15:23.690792+0000","last_clean":"2024-01-16T15:15:23.690792+0000","last_became_active":"2024-01-16T15:15:21.530451+0000","last_became_peered":"2024-01-16T15:15:21.530451+0000","last_unstale":"2024-01-16T15:15:23.690792+0000","last_undegraded":"2024-01-16T15:15:23.690792+0000","last_fullsized":"2024-01-16T15:15:23.690792+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:26.637253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005884+0000","last_change":"2024-01-16T15:15:22.533064+0000","last_active":"2024-01-16T15:15:24.005884+0000","last_peered":"2024-01-16T15:15:24.005884+0000","last_clean":"2024-01-16T15:15:24.005884+0000","last_became_active":"2024-01-16T15:15:22.532644+0000","last_became_peered":"2024-01-16T15:15:22.532644+0000","last_unstale":"2024-01-16T15:15:24.005884+0000","last_undegraded":"2024-01-16T15:15:24.005884+0000","last_fullsized":"2024-01-16T15:15:24.005884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740860+0000","last_change":"2024-01-16T15:15:21.525811+0000","last_active":"2024-01-16T15:15:23.740860+0000","last_peered":"2024-01-16T15:15:23.740860+0000","last_clean":"2024-01-16T15:15:23.740860+0000","last_became_active":"2024-01-16T15:15:21.525152+0000","last_became_peered":"2024-01-16T15:15:21.525152+0000","last_unstale":"2024-01-16T15:15:23.740860+0000","last_undegraded":"2024-01-16T15:15:23.740860+0000","last_fullsized":"2024-01-16T15:15:23.740860+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:38.101810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508468+0000","last_change":"2024-01-16T15:15:22.524402+0000","last_active":"2024-01-16T15:15:23.508468+0000","last_peered":"2024-01-16T15:15:23.508468+0000","last_clean":"2024-01-16T15:15:23.508468+0000","last_became_active":"2024-01-16T15:15:22.524129+0000","last_became_peered":"2024-01-16T15:15:22.524129+0000","last_unstale":"2024-01-16T15:15:23.508468+0000","last_undegraded":"2024-01-16T15:15:23.508468+0000","last_fullsized":"2024-01-16T15:15:23.508468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.533761+0000","last_change":"2024-01-16T15:15:21.532465+0000","last_active":"2024-01-16T15:15:21.533761+0000","last_peered":"2024-01-16T15:15:21.533761+0000","last_clean":"2024-01-16T15:15:21.533761+0000","last_became_active":"2024-01-16T15:15:21.532238+0000","last_became_peered":"2024-01-16T15:15:21.532238+0000","last_unstale":"2024-01-16T15:15:21.533761+0000","last_undegraded":"2024-01-16T15:15:21.533761+0000","last_fullsized":"2024-01-16T15:15:21.533761+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:49:49.415856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505378+0000","last_change":"2024-01-16T15:15:21.505378+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505378+0000","last_undegraded":"2024-01-16T15:15:21.505378+0000","last_fullsized":"2024-01-16T15:15:21.505378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690271+0000","last_change":"2024-01-16T15:15:21.524375+0000","last_active":"2024-01-16T15:15:23.690271+0000","last_peered":"2024-01-16T15:15:23.690271+0000","last_clean":"2024-01-16T15:15:23.690271+0000","last_became_active":"2024-01-16T15:15:21.523731+0000","last_became_peered":"2024-01-16T15:15:21.523731+0000","last_unstale":"2024-01-16T15:15:23.690271+0000","last_undegraded":"2024-01-16T15:15:23.690271+0000","last_fullsized":"2024-01-16T15:15:23.690271+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:13.884029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741391+0000","last_change":"2024-01-16T15:15:22.528073+0000","last_active":"2024-01-16T15:15:23.741391+0000","last_peered":"2024-01-16T15:15:23.741391+0000","last_clean":"2024-01-16T15:15:23.741391+0000","last_became_active":"2024-01-16T15:15:22.526190+0000","last_became_peered":"2024-01-16T15:15:22.526190+0000","last_unstale":"2024-01-16T15:15:23.741391+0000","last_undegraded":"2024-01-16T15:15:23.741391+0000","last_fullsized":"2024-01-16T15:15:23.741391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:55.442576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691500+0000","last_change":"2024-01-16T15:15:21.520386+0000","last_active":"2024-01-16T15:15:23.691500+0000","last_peered":"2024-01-16T15:15:23.691500+0000","last_clean":"2024-01-16T15:15:23.691500+0000","last_became_active":"2024-01-16T15:15:21.520059+0000","last_became_peered":"2024-01-16T15:15:21.520059+0000","last_unstale":"2024-01-16T15:15:23.691500+0000","last_undegraded":"2024-01-16T15:15:23.691500+0000","last_fullsized":"2024-01-16T15:15:23.691500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:33.653275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.504957+0000","last_change":"2024-01-16T15:15:21.504957+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.504957+0000","last_undegraded":"2024-01-16T15:15:21.504957+0000","last_fullsized":"2024-01-16T15:15:21.504957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509435+0000","last_change":"2024-01-16T15:15:21.534208+0000","last_active":"2024-01-16T15:15:23.509435+0000","last_peered":"2024-01-16T15:15:23.509435+0000","last_clean":"2024-01-16T15:15:23.509435+0000","last_became_active":"2024-01-16T15:15:21.533545+0000","last_became_peered":"2024-01-16T15:15:21.533545+0000","last_unstale":"2024-01-16T15:15:23.509435+0000","last_undegraded":"2024-01-16T15:15:23.509435+0000","last_fullsized":"2024-01-16T15:15:23.509435+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518791+0000","last_change":"2024-01-16T15:15:21.518791+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518791+0000","last_undegraded":"2024-01-16T15:15:21.518791+0000","last_fullsized":"2024-01-16T15:15:21.518791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.547699+0000","last_change":"2024-01-16T15:15:21.521285+0000","last_active":"2024-01-16T15:15:22.547699+0000","last_peered":"2024-01-16T15:15:22.547699+0000","last_clean":"2024-01-16T15:15:22.547699+0000","last_became_active":"2024-01-16T15:15:21.520717+0000","last_became_peered":"2024-01-16T15:15:21.520717+0000","last_unstale":"2024-01-16T15:15:22.547699+0000","last_undegraded":"2024-01-16T15:15:22.547699+0000","last_fullsized":"2024-01-16T15:15:22.547699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:15.674482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006046+0000","last_change":"2024-01-16T15:15:22.529757+0000","last_active":"2024-01-16T15:15:24.006046+0000","last_peered":"2024-01-16T15:15:24.006046+0000","last_clean":"2024-01-16T15:15:24.006046+0000","last_became_active":"2024-01-16T15:15:22.524620+0000","last_became_peered":"2024-01-16T15:15:22.524620+0000","last_unstale":"2024-01-16T15:15:24.006046+0000","last_undegraded":"2024-01-16T15:15:24.006046+0000","last_fullsized":"2024-01-16T15:15:24.006046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.520270+0000","last_change":"2024-01-16T15:15:21.518507+0000","last_active":"2024-01-16T15:15:21.520270+0000","last_peered":"2024-01-16T15:15:21.520270+0000","last_clean":"2024-01-16T15:15:21.520270+0000","last_became_active":"2024-01-16T15:15:21.518075+0000","last_became_peered":"2024-01-16T15:15:21.518075+0000","last_unstale":"2024-01-16T15:15:21.520270+0000","last_undegraded":"2024-01-16T15:15:21.520270+0000","last_fullsized":"2024-01-16T15:15:21.520270+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:25.234717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509009+0000","last_change":"2024-01-16T15:15:22.521859+0000","last_active":"2024-01-16T15:15:23.509009+0000","last_peered":"2024-01-16T15:15:23.509009+0000","last_clean":"2024-01-16T15:15:23.509009+0000","last_became_active":"2024-01-16T15:15:22.521561+0000","last_became_peered":"2024-01-16T15:15:22.521561+0000","last_unstale":"2024-01-16T15:15:23.509009+0000","last_undegraded":"2024-01-16T15:15:23.509009+0000","last_fullsized":"2024-01-16T15:15:23.509009+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:14.797953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609905+0000","last_change":"2024-01-16T15:15:21.521158+0000","last_active":"2024-01-16T15:15:23.609905+0000","last_peered":"2024-01-16T15:15:23.609905+0000","last_clean":"2024-01-16T15:15:23.609905+0000","last_became_active":"2024-01-16T15:15:21.520851+0000","last_became_peered":"2024-01-16T15:15:21.520851+0000","last_unstale":"2024-01-16T15:15:23.609905+0000","last_undegraded":"2024-01-16T15:15:23.609905+0000","last_fullsized":"2024-01-16T15:15:23.609905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:47.111965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690791+0000","last_change":"2024-01-16T15:15:22.530599+0000","last_active":"2024-01-16T15:15:23.690791+0000","last_peered":"2024-01-16T15:15:23.690791+0000","last_clean":"2024-01-16T15:15:23.690791+0000","last_became_active":"2024-01-16T15:15:22.526185+0000","last_became_peered":"2024-01-16T15:15:22.526185+0000","last_unstale":"2024-01-16T15:15:23.690791+0000","last_undegraded":"2024-01-16T15:15:23.690791+0000","last_fullsized":"2024-01-16T15:15:23.690791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:11.722014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609612+0000","last_change":"2024-01-16T15:15:21.526311+0000","last_active":"2024-01-16T15:15:23.609612+0000","last_peered":"2024-01-16T15:15:23.609612+0000","last_clean":"2024-01-16T15:15:23.609612+0000","last_became_active":"2024-01-16T15:15:21.526019+0000","last_became_peered":"2024-01-16T15:15:21.526019+0000","last_unstale":"2024-01-16T15:15:23.609612+0000","last_undegraded":"2024-01-16T15:15:23.609612+0000","last_fullsized":"2024-01-16T15:15:23.609612+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:30.739366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505003+0000","last_change":"2024-01-16T15:15:21.505003+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505003+0000","last_undegraded":"2024-01-16T15:15:21.505003+0000","last_fullsized":"2024-01-16T15:15:21.505003+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.530670+0000","last_change":"2024-01-16T15:15:21.528755+0000","last_active":"2024-01-16T15:15:21.530670+0000","last_peered":"2024-01-16T15:15:21.530670+0000","last_clean":"2024-01-16T15:15:21.530670+0000","last_became_active":"2024-01-16T15:15:21.528462+0000","last_became_peered":"2024-01-16T15:15:21.528462+0000","last_unstale":"2024-01-16T15:15:21.530670+0000","last_undegraded":"2024-01-16T15:15:21.530670+0000","last_fullsized":"2024-01-16T15:15:21.530670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:46.120683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609164+0000","last_change":"2024-01-16T15:15:22.529246+0000","last_active":"2024-01-16T15:15:23.609164+0000","last_peered":"2024-01-16T15:15:23.609164+0000","last_clean":"2024-01-16T15:15:23.609164+0000","last_became_active":"2024-01-16T15:15:22.528966+0000","last_became_peered":"2024-01-16T15:15:22.528966+0000","last_unstale":"2024-01-16T15:15:23.609164+0000","last_undegraded":"2024-01-16T15:15:23.609164+0000","last_fullsized":"2024-01-16T15:15:23.609164+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:46.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690545+0000","last_change":"2024-01-16T15:15:21.525650+0000","last_active":"2024-01-16T15:15:23.690545+0000","last_peered":"2024-01-16T15:15:23.690545+0000","last_clean":"2024-01-16T15:15:23.690545+0000","last_became_active":"2024-01-16T15:15:21.524160+0000","last_became_peered":"2024-01-16T15:15:21.524160+0000","last_unstale":"2024-01-16T15:15:23.690545+0000","last_undegraded":"2024-01-16T15:15:23.690545+0000","last_fullsized":"2024-01-16T15:15:23.690545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:28.946524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.521709+0000","last_change":"2024-01-16T15:15:21.521709+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.521709+0000","last_undegraded":"2024-01-16T15:15:21.521709+0000","last_fullsized":"2024-01-16T15:15:21.521709+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.520782+0000","last_change":"2024-01-16T15:15:21.520262+0000","last_active":"2024-01-16T15:15:21.520782+0000","last_peered":"2024-01-16T15:15:21.520782+0000","last_clean":"2024-01-16T15:15:21.520782+0000","last_became_active":"2024-01-16T15:15:21.519869+0000","last_became_peered":"2024-01-16T15:15:21.519869+0000","last_unstale":"2024-01-16T15:15:21.520782+0000","last_undegraded":"2024-01-16T15:15:21.520782+0000","last_fullsized":"2024-01-16T15:15:21.520782+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:13.821755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004344+0000","last_change":"2024-01-16T15:15:22.543578+0000","last_active":"2024-01-16T15:15:24.004344+0000","last_peered":"2024-01-16T15:15:24.004344+0000","last_clean":"2024-01-16T15:15:24.004344+0000","last_became_active":"2024-01-16T15:15:22.543283+0000","last_became_peered":"2024-01-16T15:15:22.543283+0000","last_unstale":"2024-01-16T15:15:24.004344+0000","last_undegraded":"2024-01-16T15:15:24.004344+0000","last_fullsized":"2024-01-16T15:15:24.004344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.530844+0000","last_change":"2024-01-16T15:15:21.528614+0000","last_active":"2024-01-16T15:15:21.530844+0000","last_peered":"2024-01-16T15:15:21.530844+0000","last_clean":"2024-01-16T15:15:21.530844+0000","last_became_active":"2024-01-16T15:15:21.528325+0000","last_became_peered":"2024-01-16T15:15:21.528325+0000","last_unstale":"2024-01-16T15:15:21.530844+0000","last_undegraded":"2024-01-16T15:15:21.530844+0000","last_fullsized":"2024-01-16T15:15:21.530844+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:58.124662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007346+0000","last_change":"2024-01-16T15:15:22.530224+0000","last_active":"2024-01-16T15:15:24.007346+0000","last_peered":"2024-01-16T15:15:24.007346+0000","last_clean":"2024-01-16T15:15:24.007346+0000","last_became_active":"2024-01-16T15:15:22.528654+0000","last_became_peered":"2024-01-16T15:15:22.528654+0000","last_unstale":"2024-01-16T15:15:24.007346+0000","last_undegraded":"2024-01-16T15:15:24.007346+0000","last_fullsized":"2024-01-16T15:15:24.007346+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740462+0000","last_change":"2024-01-16T15:15:21.521658+0000","last_active":"2024-01-16T15:15:23.740462+0000","last_peered":"2024-01-16T15:15:23.740462+0000","last_clean":"2024-01-16T15:15:23.740462+0000","last_became_active":"2024-01-16T15:15:21.521051+0000","last_became_peered":"2024-01-16T15:15:21.521051+0000","last_unstale":"2024-01-16T15:15:23.740462+0000","last_undegraded":"2024-01-16T15:15:23.740462+0000","last_fullsized":"2024-01-16T15:15:23.740462+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609501+0000","last_change":"2024-01-16T15:15:22.525393+0000","last_active":"2024-01-16T15:15:23.609501+0000","last_peered":"2024-01-16T15:15:23.609501+0000","last_clean":"2024-01-16T15:15:23.609501+0000","last_became_active":"2024-01-16T15:15:22.524494+0000","last_became_peered":"2024-01-16T15:15:22.524494+0000","last_unstale":"2024-01-16T15:15:23.609501+0000","last_undegraded":"2024-01-16T15:15:23.609501+0000","last_fullsized":"2024-01-16T15:15:23.609501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:10.852002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740975+0000","last_change":"2024-01-16T15:15:21.531548+0000","last_active":"2024-01-16T15:15:23.740975+0000","last_peered":"2024-01-16T15:15:23.740975+0000","last_clean":"2024-01-16T15:15:23.740975+0000","last_became_active":"2024-01-16T15:15:21.531267+0000","last_became_peered":"2024-01-16T15:15:21.531267+0000","last_unstale":"2024-01-16T15:15:23.740975+0000","last_undegraded":"2024-01-16T15:15:23.740975+0000","last_fullsized":"2024-01-16T15:15:23.740975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:42.610622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609997+0000","last_change":"2024-01-16T15:15:22.522512+0000","last_active":"2024-01-16T15:15:23.609997+0000","last_peered":"2024-01-16T15:15:23.609997+0000","last_clean":"2024-01-16T15:15:23.609997+0000","last_became_active":"2024-01-16T15:15:22.522206+0000","last_became_peered":"2024-01-16T15:15:22.522206+0000","last_unstale":"2024-01-16T15:15:23.609997+0000","last_undegraded":"2024-01-16T15:15:23.609997+0000","last_fullsized":"2024-01-16T15:15:23.609997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:29.962056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004895+0000","last_change":"2024-01-16T15:15:21.524797+0000","last_active":"2024-01-16T15:15:24.004895+0000","last_peered":"2024-01-16T15:15:24.004895+0000","last_clean":"2024-01-16T15:15:24.004895+0000","last_became_active":"2024-01-16T15:15:21.524502+0000","last_became_peered":"2024-01-16T15:15:21.524502+0000","last_unstale":"2024-01-16T15:15:24.004895+0000","last_undegraded":"2024-01-16T15:15:24.004895+0000","last_fullsized":"2024-01-16T15:15:24.004895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:46.648168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609344+0000","last_change":"2024-01-16T15:15:22.523225+0000","last_active":"2024-01-16T15:15:23.609344+0000","last_peered":"2024-01-16T15:15:23.609344+0000","last_clean":"2024-01-16T15:15:23.609344+0000","last_became_active":"2024-01-16T15:15:22.522951+0000","last_became_peered":"2024-01-16T15:15:22.522951+0000","last_unstale":"2024-01-16T15:15:23.609344+0000","last_undegraded":"2024-01-16T15:15:23.609344+0000","last_fullsized":"2024-01-16T15:15:23.609344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:21.818020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004192+0000","last_change":"2024-01-16T15:15:21.519991+0000","last_active":"2024-01-16T15:15:24.004192+0000","last_peered":"2024-01-16T15:15:24.004192+0000","last_clean":"2024-01-16T15:15:24.004192+0000","last_became_active":"2024-01-16T15:15:21.519565+0000","last_became_peered":"2024-01-16T15:15:21.519565+0000","last_unstale":"2024-01-16T15:15:24.004192+0000","last_undegraded":"2024-01-16T15:15:24.004192+0000","last_fullsized":"2024-01-16T15:15:24.004192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:50.723441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518797+0000","last_change":"2024-01-16T15:15:21.518797+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518797+0000","last_undegraded":"2024-01-16T15:15:21.518797+0000","last_fullsized":"2024-01-16T15:15:21.518797+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740258+0000","last_change":"2024-01-16T15:15:21.519702+0000","last_active":"2024-01-16T15:15:23.740258+0000","last_peered":"2024-01-16T15:15:23.740258+0000","last_clean":"2024-01-16T15:15:23.740258+0000","last_became_active":"2024-01-16T15:15:21.518926+0000","last_became_peered":"2024-01-16T15:15:21.518926+0000","last_unstale":"2024-01-16T15:15:23.740258+0000","last_undegraded":"2024-01-16T15:15:23.740258+0000","last_fullsized":"2024-01-16T15:15:23.740258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:31.707777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741243+0000","last_change":"2024-01-16T15:15:22.536514+0000","last_active":"2024-01-16T15:15:23.741243+0000","last_peered":"2024-01-16T15:15:23.741243+0000","last_clean":"2024-01-16T15:15:23.741243+0000","last_became_active":"2024-01-16T15:15:22.536248+0000","last_became_peered":"2024-01-16T15:15:22.536248+0000","last_unstale":"2024-01-16T15:15:23.741243+0000","last_undegraded":"2024-01-16T15:15:23.741243+0000","last_fullsized":"2024-01-16T15:15:23.741243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:12:10.024161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005379+0000","last_change":"2024-01-16T15:15:21.522416+0000","last_active":"2024-01-16T15:15:24.005379+0000","last_peered":"2024-01-16T15:15:24.005379+0000","last_clean":"2024-01-16T15:15:24.005379+0000","last_became_active":"2024-01-16T15:15:21.522118+0000","last_became_peered":"2024-01-16T15:15:21.522118+0000","last_unstale":"2024-01-16T15:15:24.005379+0000","last_undegraded":"2024-01-16T15:15:24.005379+0000","last_fullsized":"2024-01-16T15:15:24.005379+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.762652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691386+0000","last_change":"2024-01-16T15:15:22.531377+0000","last_active":"2024-01-16T15:15:23.691386+0000","last_peered":"2024-01-16T15:15:23.691386+0000","last_clean":"2024-01-16T15:15:23.691386+0000","last_became_active":"2024-01-16T15:15:22.529798+0000","last_became_peered":"2024-01-16T15:15:22.529798+0000","last_unstale":"2024-01-16T15:15:23.691386+0000","last_undegraded":"2024-01-16T15:15:23.691386+0000","last_fullsized":"2024-01-16T15:15:23.691386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.721245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741334+0000","last_change":"2024-01-16T15:15:21.532318+0000","last_active":"2024-01-16T15:15:23.741334+0000","last_peered":"2024-01-16T15:15:23.741334+0000","last_clean":"2024-01-16T15:15:23.741334+0000","last_became_active":"2024-01-16T15:15:21.531869+0000","last_became_peered":"2024-01-16T15:15:21.531869+0000","last_unstale":"2024-01-16T15:15:23.741334+0000","last_undegraded":"2024-01-16T15:15:23.741334+0000","last_fullsized":"2024-01-16T15:15:23.741334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:40.183724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690994+0000","last_change":"2024-01-16T15:15:22.535293+0000","last_active":"2024-01-16T15:15:23.690994+0000","last_peered":"2024-01-16T15:15:23.690994+0000","last_clean":"2024-01-16T15:15:23.690994+0000","last_became_active":"2024-01-16T15:15:22.534929+0000","last_became_peered":"2024-01-16T15:15:22.534929+0000","last_unstale":"2024-01-16T15:15:23.690994+0000","last_undegraded":"2024-01-16T15:15:23.690994+0000","last_fullsized":"2024-01-16T15:15:23.690994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.606169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740931+0000","last_change":"2024-01-16T15:15:21.520890+0000","last_active":"2024-01-16T15:15:23.740931+0000","last_peered":"2024-01-16T15:15:23.740931+0000","last_clean":"2024-01-16T15:15:23.740931+0000","last_became_active":"2024-01-16T15:15:21.520488+0000","last_became_peered":"2024-01-16T15:15:21.520488+0000","last_unstale":"2024-01-16T15:15:23.740931+0000","last_undegraded":"2024-01-16T15:15:23.740931+0000","last_fullsized":"2024-01-16T15:15:23.740931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:08.456302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.521585+0000","last_change":"2024-01-16T15:15:21.521585+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.521585+0000","last_undegraded":"2024-01-16T15:15:21.521585+0000","last_fullsized":"2024-01-16T15:15:21.521585+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.528795+0000","last_change":"2024-01-16T15:15:21.527345+0000","last_active":"2024-01-16T15:15:21.528795+0000","last_peered":"2024-01-16T15:15:21.528795+0000","last_clean":"2024-01-16T15:15:21.528795+0000","last_became_active":"2024-01-16T15:15:21.527018+0000","last_became_peered":"2024-01-16T15:15:21.527018+0000","last_unstale":"2024-01-16T15:15:21.528795+0000","last_undegraded":"2024-01-16T15:15:21.528795+0000","last_fullsized":"2024-01-16T15:15:21.528795+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:32:59.366965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609781+0000","last_change":"2024-01-16T15:15:22.525600+0000","last_active":"2024-01-16T15:15:23.609781+0000","last_peered":"2024-01-16T15:15:23.609781+0000","last_clean":"2024-01-16T15:15:23.609781+0000","last_became_active":"2024-01-16T15:15:22.525112+0000","last_became_peered":"2024-01-16T15:15:22.525112+0000","last_unstale":"2024-01-16T15:15:23.609781+0000","last_undegraded":"2024-01-16T15:15:23.609781+0000","last_fullsized":"2024-01-16T15:15:23.609781+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:49.965076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741004+0000","last_change":"2024-01-16T15:15:21.526060+0000","last_active":"2024-01-16T15:15:23.741004+0000","last_peered":"2024-01-16T15:15:23.741004+0000","last_clean":"2024-01-16T15:15:23.741004+0000","last_became_active":"2024-01-16T15:15:21.525792+0000","last_became_peered":"2024-01-16T15:15:21.525792+0000","last_unstale":"2024-01-16T15:15:23.741004+0000","last_undegraded":"2024-01-16T15:15:23.741004+0000","last_fullsized":"2024-01-16T15:15:23.741004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.505027+0000","last_change":"2024-01-16T15:15:21.505027+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.505027+0000","last_undegraded":"2024-01-16T15:15:21.505027+0000","last_fullsized":"2024-01-16T15:15:21.505027+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":9,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-01-16T15:15:21.533268+0000","last_change":"2024-01-16T15:15:21.530401+0000","last_active":"2024-01-16T15:15:21.533268+0000","last_peered":"2024-01-16T15:15:21.533268+0000","last_clean":"2024-01-16T15:15:21.533268+0000","last_became_active":"2024-01-16T15:15:21.530083+0000","last_became_peered":"2024-01-16T15:15:21.530083+0000","last_unstale":"2024-01-16T15:15:21.533268+0000","last_undegraded":"2024-01-16T15:15:21.533268+0000","last_fullsized":"2024-01-16T15:15:21.533268+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:15:56.829359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609251+0000","last_change":"2024-01-16T15:15:22.530971+0000","last_active":"2024-01-16T15:15:23.609251+0000","last_peered":"2024-01-16T15:15:23.609251+0000","last_clean":"2024-01-16T15:15:23.609251+0000","last_became_active":"2024-01-16T15:15:22.530694+0000","last_became_peered":"2024-01-16T15:15:22.530694+0000","last_unstale":"2024-01-16T15:15:23.609251+0000","last_undegraded":"2024-01-16T15:15:23.609251+0000","last_fullsized":"2024-01-16T15:15:23.609251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:02.926295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508734+0000","last_change":"2024-01-16T15:15:21.524472+0000","last_active":"2024-01-16T15:15:23.508734+0000","last_peered":"2024-01-16T15:15:23.508734+0000","last_clean":"2024-01-16T15:15:23.508734+0000","last_became_active":"2024-01-16T15:15:21.524175+0000","last_became_peered":"2024-01-16T15:15:21.524175+0000","last_unstale":"2024-01-16T15:15:23.508734+0000","last_undegraded":"2024-01-16T15:15:23.508734+0000","last_fullsized":"2024-01-16T15:15:23.508734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:43.337353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":1,"reported_epoch":97,"state":"creating+peering","last_fresh":"2024-01-16T15:15:21.518265+0000","last_change":"2024-01-16T15:15:21.518265+0000","last_active":"2024-01-16T15:15:21.481640+0000","last_peered":"2024-01-16T15:15:21.481640+0000","last_clean":"2024-01-16T15:15:21.481640+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-01-16T15:15:21.518265+0000","last_undegraded":"2024-01-16T15:15:21.518265+0000","last_fullsized":"2024-01-16T15:15:21.518265+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609465+0000","last_change":"2024-01-16T15:15:21.533877+0000","last_active":"2024-01-16T15:15:23.609465+0000","last_peered":"2024-01-16T15:15:23.609465+0000","last_clean":"2024-01-16T15:15:23.609465+0000","last_became_active":"2024-01-16T15:15:21.533489+0000","last_became_peered":"2024-01-16T15:15:21.533489+0000","last_unstale":"2024-01-16T15:15:23.609465+0000","last_undegraded":"2024-01-16T15:15:23.609465+0000","last_fullsized":"2024-01-16T15:15:23.609465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:15.404651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610593+0000","last_change":"2024-01-16T15:15:22.521850+0000","last_active":"2024-01-16T15:15:23.610593+0000","last_peered":"2024-01-16T15:15:23.610593+0000","last_clean":"2024-01-16T15:15:23.610593+0000","last_became_active":"2024-01-16T15:15:22.521581+0000","last_became_peered":"2024-01-16T15:15:22.521581+0000","last_unstale":"2024-01-16T15:15:23.610593+0000","last_undegraded":"2024-01-16T15:15:23.610593+0000","last_fullsized":"2024-01-16T15:15:23.610593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:29.487154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.545734+0000","last_change":"2024-01-16T15:15:21.522571+0000","last_active":"2024-01-16T15:15:22.545734+0000","last_peered":"2024-01-16T15:15:22.545734+0000","last_clean":"2024-01-16T15:15:22.545734+0000","last_became_active":"2024-01-16T15:15:21.522205+0000","last_became_peered":"2024-01-16T15:15:21.522205+0000","last_unstale":"2024-01-16T15:15:22.545734+0000","last_undegraded":"2024-01-16T15:15:22.545734+0000","last_fullsized":"2024-01-16T15:15:22.545734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":241,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516932+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:15:22.516932+0000","last_peered":"2024-01-16T15:15:22.516932+0000","last_clean":"2024-01-16T15:15:22.516932+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:15:22.516932+0000","last_undegraded":"2024-01-16T15:15:22.516932+0000","last_fullsized":"2024-01-16T15:15:22.516932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":728,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":29,"ondisk_log_size":29,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509629,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68364,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303476,"statfs":{"total":96636764160,"available":96566759424,"internally_reserved":0,"allocated":1179648,"data_stored":505985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3140000000000001,"15min":1.615},"min":{"1min":0.75,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":2.5030000000000001,"5min":8.8070000000000004,"15min":27.853999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.663,"15min":1.708},"min":{"1min":0.93000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6399999999999999,"5min":28.603999999999999,"15min":34.939999999999998},"last":0.875}]},{"osd":1,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.292,"15min":1.4830000000000001},"min":{"1min":0.94899999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.528,"5min":8.2650000000000006,"15min":13.859},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.754,"15min":1.651},"min":{"1min":0.85599999999999998,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":2.702,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.2589999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4330000000000001,"15min":1.5860000000000001},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":2.6230000000000002,"5min":9.6679999999999993,"15min":13.050000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.69499999999999995,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":2.6309999999999998,"5min":6.782,"15min":35.011000000000003},"last":1.3100000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.486,"15min":1.657},"min":{"1min":0.94799999999999995,"5min":0.439,"15min":0.374},"max":{"1min":1.8720000000000001,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.7170000000000001},"min":{"1min":0.999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.758,"5min":9.7780000000000005,"15min":14.273999999999999},"last":1.1519999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3919999999999999,"15min":1.827},"min":{"1min":0.90900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7090000000000001,"5min":10.952999999999999,"15min":34.972999999999999},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.4319999999999999,"15min":1.798},"min":{"1min":0.85899999999999999,"5min":0.58299999999999996,"15min":0.40400000000000003},"max":{"1min":1.6519999999999999,"5min":5.2300000000000004,"15min":16.003},"last":1.411}]},{"osd":5,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.591,"15min":2.0099999999999998},"min":{"1min":1.1339999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.798,"5min":4.4550000000000001,"15min":23.760999999999999},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6259999999999999,"15min":1.9059999999999999},"min":{"1min":0.96299999999999997,"5min":0.70699999999999996,"15min":0.52400000000000002},"max":{"1min":3.496,"5min":9.4120000000000008,"15min":23.716999999999999},"last":1.363}]},{"osd":7,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.5069999999999999,"15min":2.0430000000000001},"min":{"1min":0.90700000000000003,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":6.9219999999999997,"15min":34.889000000000003},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.79,"15min":2.1760000000000002},"min":{"1min":1.2470000000000001,"5min":0.65100000000000002,"15min":0.47599999999999998},"max":{"1min":2.0499999999999998,"5min":23.242999999999999,"15min":46.305},"last":1.548}]}]},{"osd":4,"up_from":15,"seq":64424509631,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68380,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303460,"statfs":{"total":96636764160,"available":96566743040,"internally_reserved":0,"allocated":1191936,"data_stored":512288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.226,"15min":1.55},"min":{"1min":0.76900000000000002,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.1399999999999997,"15min":22.757999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.125,"15min":1.2729999999999999},"min":{"1min":0.69799999999999995,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.474,"5min":4.0270000000000001,"15min":14.034000000000001},"last":1.623}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.413,"15min":1.577},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.851,"5min":17.811,"15min":18.347000000000001},"last":1.002},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3140000000000001,"15min":1.49},"min":{"1min":0.90700000000000003,"5min":0.59799999999999998,"15min":0.498},"max":{"1min":1.5600000000000001,"5min":4.0449999999999999,"15min":18.858000000000001},"last":1.6679999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.544,"15min":1.679},"min":{"1min":0.83799999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.768,"5min":17.651,"15min":17.651},"last":1.204},{"interface":"front","average":{"1min":1.327,"5min":1.754,"15min":1.722},"min":{"1min":0.83499999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8180000000000001,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.77}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3320000000000001,"15min":1.554},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":1.6659999999999999,"5min":3.972,"15min":26.741},"last":1.458},{"interface":"front","average":{"1min":1.284,"5min":1.571,"15min":1.7929999999999999},"min":{"1min":0.86899999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.762,"5min":17.715,"15min":26.701000000000001},"last":1.3149999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5660000000000001,"15min":1.7370000000000001},"min":{"1min":1.0029999999999999,"5min":0.51400000000000001,"15min":0.441},"max":{"1min":1.988,"5min":5.3170000000000002,"15min":18.919},"last":1.367},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5029999999999999,"15min":1.6759999999999999},"min":{"1min":0.95599999999999996,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":2.0270000000000001,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.8280000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":2.0739999999999998,"15min":1.9419999999999999},"min":{"1min":0.995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.0750000000000002,"5min":32.649999999999999,"15min":32.649999999999999},"last":1.514},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5469999999999999,"15min":1.823},"min":{"1min":1.1799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0630000000000002,"5min":4.3129999999999997,"15min":24.094999999999999},"last":1.55}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.6890000000000001,"15min":1.7649999999999999},"min":{"1min":1.056,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":1.8899999999999999,"5min":17.446000000000002,"15min":26.683},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.871,"15min":1.9650000000000001},"min":{"1min":1.248,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9350000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":1.407}]}]},{"osd":7,"up_from":14,"seq":60129542334,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68988,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302852,"statfs":{"total":96636764160,"available":96566120448,"internally_reserved":0,"allocated":1818624,"data_stored":1115899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2649999999999999,"15min":1.5089999999999999},"min":{"1min":0.68100000000000005,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.7310000000000001,"5min":4.3680000000000003,"15min":15.661},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.3240000000000001,"15min":1.5449999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":2.1080000000000001,"5min":12.461,"15min":30.126000000000001},"last":1.226}]},{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.423,"15min":1.4530000000000001},"min":{"1min":0.63200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.085,"5min":20.327000000000002,"15min":24.338000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.395,"15min":1.6180000000000001},"min":{"1min":0.81399999999999995,"5min":0.77400000000000002,"15min":0.46200000000000002},"max":{"1min":1.6419999999999999,"5min":12.526,"15min":21.298999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.57199999999999995,"15min":0.51500000000000001},"max":{"1min":1.9219999999999999,"5min":4.3380000000000001,"15min":12.474},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.702,"15min":1.651},"min":{"1min":0.69299999999999995,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8999999999999999,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4039999999999999,"15min":1.6020000000000001},"min":{"1min":0.88800000000000001,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.7150000000000001,"5min":4.2530000000000001,"15min":12.58},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.9430000000000001,"15min":1.7869999999999999},"min":{"1min":0.78600000000000003,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":2.4630000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3520000000000001,"15min":1.591},"min":{"1min":0.99099999999999999,"5min":0.54100000000000004,"15min":0.42799999999999999},"max":{"1min":2.0219999999999998,"5min":4.1390000000000002,"15min":14.659000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.831,"15min":1.825},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":2.028,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.3600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.486,"15min":1.8360000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.6459999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.113},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.8400000000000001},"min":{"1min":1.1870000000000001,"5min":0.66000000000000003,"15min":0.41099999999999998},"max":{"1min":1.8400000000000001,"5min":7.4649999999999999,"15min":21.312000000000001},"last":1.383}]},{"osd":6,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.9790000000000001,"15min":1.8640000000000001},"min":{"1min":0.97199999999999998,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":2.0880000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.597,"15min":1.956},"min":{"1min":0.81699999999999995,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":1.8300000000000001,"5min":12.567,"15min":29.991},"last":1.3300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542335,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76728,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295112,"statfs":{"total":96636764160,"available":96558194688,"internally_reserved":0,"allocated":1220608,"data_stored":525489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.405,"15min":1.5389999999999999},"min":{"1min":0.79700000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.6579999999999999,"5min":6.8380000000000001,"15min":19.382999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.3740000000000001,"15min":1.3899999999999999},"min":{"1min":0.77000000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":13.127000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":1.008}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.5409999999999999,"15min":1.6339999999999999},"min":{"1min":0.70699999999999996,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":3.8109999999999999,"5min":13.571,"15min":17.693999999999999},"last":1.214},{"interface":"front","average":{"1min":1.996,"5min":1.6419999999999999,"15min":1.6950000000000001},"min":{"1min":0.95099999999999996,"5min":0.51600000000000001,"15min":0.432},"max":{"1min":17.062000000000001,"5min":17.062000000000001,"15min":20.645},"last":1.395}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.5740000000000001,"15min":1.6879999999999999},"min":{"1min":0.79100000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.77,"5min":6.4039999999999999,"15min":21.138000000000002},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.462,"15min":1.4910000000000001},"min":{"1min":0.995,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":4.0720000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.5349999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4930000000000001,"15min":1.8},"min":{"1min":0.95499999999999996,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":2.4769999999999999,"5min":7.984,"15min":23.109000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.6060000000000001,"15min":1.7949999999999999},"min":{"1min":1.0169999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.95,"5min":6.4610000000000003,"15min":19.303000000000001},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5449999999999999,"15min":1.849},"min":{"1min":0.85799999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":9.3309999999999995,"5min":10.831,"15min":24.436},"last":1.617},{"interface":"front","average":{"1min":1.329,"5min":1.48,"15min":1.744},"min":{"1min":0.84699999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":1.6140000000000001,"5min":5.6820000000000004,"15min":19.126000000000001},"last":1.5800000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.526,"15min":1.635},"min":{"1min":0.95599999999999996,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":1.718,"5min":12.914,"15min":21.192},"last":1.341},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5609999999999999,"15min":1.978},"min":{"1min":0.97299999999999998,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":2.0299999999999998,"5min":7.8609999999999998,"15min":27.042000000000002},"last":1.764}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7989999999999999,"15min":1.9350000000000001},"min":{"1min":0.92600000000000005,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":13.034000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":1.663},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.7410000000000001,"15min":2.0840000000000001},"min":{"1min":1.216,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":2.1429999999999998,"5min":10.756,"15min":24.367000000000001},"last":1.8440000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542335,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":527053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.5,"15min":1.278},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":1.403,"5min":28.437999999999999,"15min":31.562999999999999},"last":1.238},{"interface":"front","average":{"1min":1.117,"5min":1.26,"15min":1.173},"min":{"1min":0.65900000000000003,"5min":0.40600000000000003,"15min":0.39600000000000002},"max":{"1min":1.8440000000000001,"5min":11.361000000000001,"15min":11.361000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.236,"15min":1.145},"min":{"1min":0.59999999999999998,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.5940000000000001,"5min":10.856999999999999,"15min":15.569000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.7050000000000001,"15min":1.429},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.643,"5min":28.352,"15min":28.352},"last":1.103}]},{"osd":2,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":2.379,"15min":1.6879999999999999},"min":{"1min":0.71999999999999997,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":1.8109999999999999,"5min":39.265999999999998,"15min":39.265999999999998},"last":1.351},{"interface":"front","average":{"1min":1.264,"5min":1.4970000000000001,"15min":1.385},"min":{"1min":0.67800000000000005,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":3.911,"5min":13.526,"15min":14.577999999999999},"last":1.1679999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.516,"15min":1.5620000000000001},"min":{"1min":0.86499999999999999,"5min":0.64900000000000002,"15min":0.53800000000000003},"max":{"1min":2.1019999999999999,"5min":6.0629999999999997,"15min":20.978999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.448,"5min":2.2879999999999998,"15min":1.8480000000000001},"min":{"1min":1.034,"5min":0.93100000000000005,"15min":0.72999999999999998},"max":{"1min":2.0030000000000001,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.472}]},{"osd":5,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.746,"15min":1.766},"min":{"1min":0.81699999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":1.911,"5min":10.988,"15min":20.663},"last":1.512},{"interface":"front","average":{"1min":1.575,"5min":2.3090000000000002,"15min":1.9490000000000001},"min":{"1min":0.96299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1019999999999999,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.637}]},{"osd":6,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.77,"15min":1.754},"min":{"1min":1.0069999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.2170000000000001,"5min":13.691000000000001,"15min":13.691000000000001},"last":1.321},{"interface":"front","average":{"1min":2.0910000000000002,"5min":2.0819999999999999,"15min":1.9510000000000001},"min":{"1min":0.95099999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":15.114000000000001,"5min":26.698,"15min":26.698},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.905,"15min":1.776},"min":{"1min":1.0589999999999999,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.9630000000000001,"5min":13.622,"15min":13.622},"last":1.431},{"interface":"front","average":{"1min":1.5289999999999999,"5min":2.0459999999999998,"15min":2.1659999999999999},"min":{"1min":0.90400000000000003,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.2789999999999999,"5min":15.372999999999999,"15min":49.811},"last":1.671}]}]},{"osd":2,"up_from":14,"seq":60129542334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76732,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295108,"statfs":{"total":96636764160,"available":96558190592,"internally_reserved":0,"allocated":1220608,"data_stored":525453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.284,"15min":1.0449999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.7130000000000001,"5min":18.289999999999999,"15min":18.289999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.2210000000000001,"15min":1.099},"min":{"1min":0.65100000000000002,"5min":0.40699999999999997,"15min":0.28599999999999998},"max":{"1min":1.6000000000000001,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.4650000000000001,"15min":1.1850000000000001},"min":{"1min":0.55900000000000005,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":1.8979999999999999,"5min":23.135999999999999,"15min":23.135999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.278,"15min":1.2050000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.6699999999999999,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":1.016}]},{"osd":3,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3859999999999999,"15min":1.1839999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":7.306,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.421,"5min":1.647,"15min":1.4510000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.839,"5min":19.375,"15min":19.375},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.8140000000000001,"15min":1.5620000000000001},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.59999999999999998},"max":{"1min":2.0249999999999999,"5min":23.225000000000001,"15min":23.225000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.651,"15min":1.5600000000000001},"min":{"1min":1.036,"5min":0.72699999999999998,"15min":0.59199999999999997},"max":{"1min":1.7509999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.335}]},{"osd":5,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.8580000000000001,"15min":1.6200000000000001},"min":{"1min":1.0740000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":7.7640000000000002,"5min":20.157,"15min":20.157},"last":1.2},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8089999999999999,"15min":1.601},"min":{"1min":0.84299999999999997,"5min":0.623,"15min":0.623},"max":{"1min":7.9820000000000002,"5min":25.004000000000001,"15min":25.004000000000001},"last":1.7090000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":2.1040000000000001,"15min":1.7390000000000001},"min":{"1min":0.89400000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":7.6630000000000003,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.948,"15min":1.839},"min":{"1min":0.81100000000000005,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":7.8680000000000003,"5min":28.760999999999999,"15min":28.760999999999999},"last":1.49}]},{"osd":7,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":2.2919999999999998,"15min":1.772},"min":{"1min":1.05,"5min":0.82099999999999995,"15min":0.64100000000000001},"max":{"1min":3.1219999999999999,"5min":22.834,"15min":22.834},"last":1.554},{"interface":"front","average":{"1min":1.53,"5min":2.2879999999999998,"15min":2.069},"min":{"1min":0.91000000000000003,"5min":0.872,"15min":0.82799999999999996},"max":{"1min":1.931,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.4279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72560,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299280,"statfs":{"total":96636764160,"available":96562462720,"internally_reserved":0,"allocated":1216512,"data_stored":525419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.9850000000000001,"15min":1.3260000000000001},"min":{"1min":0.48599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.256,"5min":31.178999999999998,"15min":31.178999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.4470000000000001,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.901,"5min":26.859999999999999,"15min":26.859999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1439999999999999,"15min":1.143},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":3.5830000000000002,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.9239999999999999,"15min":1.5529999999999999},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":3.4929999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":0.84599999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.879,"15min":1.651},"min":{"1min":0.56299999999999994,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.8049999999999999,"5min":27.609999999999999,"15min":27.609999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.1709999999999998,"15min":1.6559999999999999},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":2.9750000000000001,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.3069999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4390000000000001,"15min":1.466},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.58099999999999996},"max":{"1min":2.3069999999999999,"5min":4.7249999999999996,"15min":10.577999999999999},"last":1.46},{"interface":"front","average":{"1min":1.411,"5min":2.0760000000000001,"15min":1.74},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.0310000000000001,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.411,"15min":1.5449999999999999},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.032,"5min":3.032,"15min":17.469000000000001},"last":1.508},{"interface":"front","average":{"1min":1.377,"5min":1.522,"15min":1.6060000000000001},"min":{"1min":1.0229999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1859999999999999,"5min":4.6120000000000001,"15min":8.2989999999999995},"last":1.077}]},{"osd":6,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5569999999999999,"15min":1.7150000000000001},"min":{"1min":0.85999999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.9009999999999998,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.611},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5549999999999999,"15min":1.873},"min":{"1min":1.0800000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5760000000000001,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.2669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6419999999999999,"15min":1.899},"min":{"1min":0.94799999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4249999999999998,"5min":4.6989999999999998,"15min":30.834},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.2589999999999999,"15min":2.0609999999999999},"min":{"1min":0.96799999999999997,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":1.9890000000000001,"5min":27.716999999999999,"15min":27.716999999999999},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542334,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73152,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298688,"statfs":{"total":96636764160,"available":96561856512,"internally_reserved":0,"allocated":1814528,"data_stored":1116369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.103,"15min":1.1879999999999999},"min":{"1min":0.52100000000000002,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.73,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.353,"15min":1.4970000000000001},"min":{"1min":0.82299999999999995,"5min":0.42499999999999999,"15min":0.33900000000000002},"max":{"1min":1.623,"5min":26.459,"15min":50.429000000000002},"last":0.93500000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":0.66100000000000003,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.8009999999999999,"5min":20.041,"15min":22.716999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.452,"15min":1.6160000000000001},"min":{"1min":0.78400000000000003,"5min":0.503,"15min":0.38},"max":{"1min":1.7430000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":1.252}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2410000000000001,"15min":1.218},"min":{"1min":0.749,"5min":0.40000000000000002,"15min":0.372},"max":{"1min":1.8129999999999999,"5min":5.8719999999999999,"15min":10.795},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.425,"15min":1.5820000000000001},"min":{"1min":0.70799999999999996,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.8280000000000001,"5min":9.484,"15min":22.254999999999999},"last":1.462}]},{"osd":4,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5,"15min":1.806},"min":{"1min":0.78800000000000003,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.986,"5min":3.7749999999999999,"15min":22.407},"last":1.536},{"interface":"front","average":{"1min":1.48,"5min":1.6930000000000001,"15min":1.919},"min":{"1min":1.1970000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.907,"5min":26.57,"15min":50.497},"last":1.365}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.401,"15min":1.6220000000000001},"min":{"1min":0.83199999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.0550000000000002,"5min":4.7359999999999998,"15min":14.112},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.7,"15min":1.847},"min":{"1min":1.05,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.9430000000000001,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.5920000000000001,"15min":1.7769999999999999},"min":{"1min":0.871,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.968,"5min":8.2349999999999994,"15min":31.387},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.841,"15min":1.9650000000000001},"min":{"1min":1.01,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":3.5209999999999999,"5min":9.2840000000000007,"15min":23.547000000000001},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":2.0190000000000001,"15min":2.0680000000000001},"min":{"1min":1.292,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.21,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.679},{"interface":"front","average":{"1min":1.639,"5min":1.8660000000000001,"15min":1.835},"min":{"1min":1.0349999999999999,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.1859999999999999,"5min":12.455,"15min":15.747},"last":1.5760000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:28.049 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3b is not active+clean 2024-01-16T15:15:28.049 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.519050+0000', 'last_change': '2024-01-16T15:15:21.519050+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.519050+0000', 'last_undegraded': '2024-01-16T15:15:21.519050+0000', 'last_fullsized': '2024-01-16T15:15:21.519050+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.37 is not active+clean 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.513402+0000', 'last_change': '2024-01-16T15:15:21.513402+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.513402+0000', 'last_undegraded': '2024-01-16T15:15:21.513402+0000', 'last_fullsized': '2024-01-16T15:15:21.513402+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.36 is not active+clean 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.520840+0000', 'last_change': '2024-01-16T15:15:21.520840+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.520840+0000', 'last_undegraded': '2024-01-16T15:15:21.520840+0000', 'last_fullsized': '2024-01-16T15:15:21.520840+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.34 is not active+clean 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.510864+0000', 'last_change': '2024-01-16T15:15:21.510864+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.510864+0000', 'last_undegraded': '2024-01-16T15:15:21.510864+0000', 'last_fullsized': '2024-01-16T15:15:21.510864+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.2c is not active+clean 2024-01-16T15:15:28.050 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.508179+0000', 'last_change': '2024-01-16T15:15:21.508179+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.508179+0000', 'last_undegraded': '2024-01-16T15:15:21.508179+0000', 'last_fullsized': '2024-01-16T15:15:21.508179+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.27 is not active+clean 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.522680+0000', 'last_change': '2024-01-16T15:15:21.522680+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.522680+0000', 'last_undegraded': '2024-01-16T15:15:21.522680+0000', 'last_fullsized': '2024-01-16T15:15:21.522680+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.24 is not active+clean 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.522614+0000', 'last_change': '2024-01-16T15:15:21.522614+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.522614+0000', 'last_undegraded': '2024-01-16T15:15:21.522614+0000', 'last_fullsized': '2024-01-16T15:15:21.522614+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1d is not active+clean 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.508270+0000', 'last_change': '2024-01-16T15:15:21.508270+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.508270+0000', 'last_undegraded': '2024-01-16T15:15:21.508270+0000', 'last_fullsized': '2024-01-16T15:15:21.508270+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.19 is not active+clean 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.505378+0000', 'last_change': '2024-01-16T15:15:21.505378+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.505378+0000', 'last_undegraded': '2024-01-16T15:15:21.505378+0000', 'last_fullsized': '2024-01-16T15:15:21.505378+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.17 is not active+clean 2024-01-16T15:15:28.051 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.504957+0000', 'last_change': '2024-01-16T15:15:21.504957+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.504957+0000', 'last_undegraded': '2024-01-16T15:15:21.504957+0000', 'last_fullsized': '2024-01-16T15:15:21.504957+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.14 is not active+clean 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.518791+0000', 'last_change': '2024-01-16T15:15:21.518791+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.518791+0000', 'last_undegraded': '2024-01-16T15:15:21.518791+0000', 'last_fullsized': '2024-01-16T15:15:21.518791+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.10 is not active+clean 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.505003+0000', 'last_change': '2024-01-16T15:15:21.505003+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.505003+0000', 'last_undegraded': '2024-01-16T15:15:21.505003+0000', 'last_fullsized': '2024-01-16T15:15:21.505003+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.f is not active+clean 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.521709+0000', 'last_change': '2024-01-16T15:15:21.521709+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.521709+0000', 'last_undegraded': '2024-01-16T15:15:21.521709+0000', 'last_fullsized': '2024-01-16T15:15:21.521709+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.9 is not active+clean 2024-01-16T15:15:28.052 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.518797+0000', 'last_change': '2024-01-16T15:15:21.518797+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.518797+0000', 'last_undegraded': '2024-01-16T15:15:21.518797+0000', 'last_fullsized': '2024-01-16T15:15:21.518797+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.5 is not active+clean 2024-01-16T15:15:28.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.521585+0000', 'last_change': '2024-01-16T15:15:21.521585+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.521585+0000', 'last_undegraded': '2024-01-16T15:15:21.521585+0000', 'last_fullsized': '2024-01-16T15:15:21.521585+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.3 is not active+clean 2024-01-16T15:15:28.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.505027+0000', 'last_change': '2024-01-16T15:15:21.505027+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.505027+0000', 'last_undegraded': '2024-01-16T15:15:21.505027+0000', 'last_fullsized': '2024-01-16T15:15:21.505027+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-01-16T15:15:28.053 INFO:tasks.cephfs.filesystem.ceph_manager:PG 15.1 is not active+clean 2024-01-16T15:15:28.053 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '15.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 97, 'state': 'creating+peering', 'last_fresh': '2024-01-16T15:15:21.518265+0000', 'last_change': '2024-01-16T15:15:21.518265+0000', 'last_active': '2024-01-16T15:15:21.481640+0000', 'last_peered': '2024-01-16T15:15:21.481640+0000', 'last_clean': '2024-01-16T15:15:21.481640+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-01-16T15:15:21.518265+0000', 'last_undegraded': '2024-01-16T15:15:21.518265+0000', 'last_fullsized': '2024-01-16T15:15:21.518265+0000', 'mapping_epoch': 97, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 97, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'last_clean_scrub_stamp': '2024-01-16T15:15:21.481640+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 3], 'acting': [6, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-01-16T15:15:28.054 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-01-16T15:15:28.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.330+0000 7f3a99600640 1 Processor -- start 2024-01-16T15:15:28.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.330+0000 7f3a99600640 1 -- start start 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a99600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3a9413cfa0 0x7f3a9415afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a99600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3a9413fcb0 0x7f3a940b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a99600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a940cdec0 con 0x7f3a940cd530 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a99600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a940ce1a0 con 0x7f3a9413fcb0 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a99600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a940ce480 con 0x7f3a9413cfa0 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a93400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3a940cd530 0x7f3a9413f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55562/0 (socket says 172.21.15.53:55562) 2024-01-16T15:15:28.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.332+0000 7f3a93400640 1 -- 172.21.15.53:0/2616271099 learned_addr learned my addr 172.21.15.53:0/2616271099 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:28.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a98c00640 1 --2- 172.21.15.53:0/2616271099 >> v2:172.21.15.53:3300/0 conn(0x7f3a9413fcb0 0x7f3a940b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a93e00640 1 --2- 172.21.15.53:0/2616271099 >> v2:172.21.15.146:3300/0 conn(0x7f3a9413cfa0 0x7f3a9415afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a92a00640 1 -- 172.21.15.53:0/2616271099 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995574186 0 0) 0x7f3a940cdec0 con 0x7f3a940cd530 2024-01-16T15:15:28.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a92a00640 1 -- 172.21.15.53:0/2616271099 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a74003520 con 0x7f3a940cd530 2024-01-16T15:15:28.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a98c00640 1 -- 172.21.15.53:0/2616271099 >> v2:172.21.15.146:3300/0 conn(0x7f3a9413cfa0 msgr2=0x7f3a9415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a98c00640 1 --2- 172.21.15.53:0/2616271099 >> v2:172.21.15.146:3300/0 conn(0x7f3a9413cfa0 0x7f3a9415afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a98c00640 1 -- 172.21.15.53:0/2616271099 >> v1:172.21.15.53:6789/0 conn(0x7f3a940cd530 legacy=0x7f3a9413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.333+0000 7f3a98c00640 1 -- 172.21.15.53:0/2616271099 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a940ce760 con 0x7f3a9413fcb0 2024-01-16T15:15:28.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.334+0000 7f3a98c00640 1 --2- 172.21.15.53:0/2616271099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413fcb0 0x7f3a940b2040 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f3a9406cf00 tx=0x7f3a8c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fb9935de20e77d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a92a00640 1 -- 172.21.15.53:0/2616271099 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c02cbe0 con 0x7f3a9413fcb0 2024-01-16T15:15:28.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a92a00640 1 -- 172.21.15.53:0/2616271099 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a8c02cd40 con 0x7f3a9413fcb0 2024-01-16T15:15:28.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a92a00640 1 -- 172.21.15.53:0/2616271099 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a8c034640 con 0x7f3a9413fcb0 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a99600640 1 -- 172.21.15.53:0/2616271099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413fcb0 msgr2=0x7f3a940b2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a99600640 1 --2- 172.21.15.53:0/2616271099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413fcb0 0x7f3a940b2040 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f3a9406cf00 tx=0x7f3a8c02be30 comp rx=0 tx=0).stop 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a99600640 1 -- 172.21.15.53:0/2616271099 shutdown_connections 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a99600640 1 --2- 172.21.15.53:0/2616271099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413fcb0 0x7f3a940b2040 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.335+0000 7f3a99600640 1 --2- 172.21.15.53:0/2616271099 >> v2:172.21.15.146:3300/0 conn(0x7f3a9413cfa0 0x7f3a9415afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.336+0000 7f3a99600640 1 -- 172.21.15.53:0/2616271099 >> 172.21.15.53:0/2616271099 conn(0x7f3a9406a800 msgr2=0x7f3a9406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.336+0000 7f3a99600640 1 -- 172.21.15.53:0/2616271099 shutdown_connections 2024-01-16T15:15:28.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.336+0000 7f3a99600640 1 -- 172.21.15.53:0/2616271099 wait complete. 2024-01-16T15:15:28.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.337+0000 7f3a99600640 1 Processor -- start 2024-01-16T15:15:28.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.338+0000 7f3a99600640 1 -- start start 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a99600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a940cd530 0x7f3a94071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a99600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 0x7f3a94156fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a99600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a9413fcb0 0x7f3a94157500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a99600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9413a360 con 0x7f3a9413cfa0 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a99600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9413a640 con 0x7f3a940cd530 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a99600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a9413a920 con 0x7f3a9413fcb0 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 0x7f3a94156fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 0x7f3a94156fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36018/0 (socket says 172.21.15.53:36018) 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 -- 172.21.15.53:0/854247804 learned_addr learned my addr 172.21.15.53:0/854247804 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a98c00640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a9413fcb0 0x7f3a94157500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 -- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a9413fcb0 msgr2=0x7f3a94157500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a9413fcb0 0x7f3a94157500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 -- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a940cd530 msgr2=0x7f3a94071550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93e00640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a940cd530 0x7f3a94071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a940cd530 0x7f3a94071550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.339+0000 7f3a93400640 1 -- 172.21.15.53:0/854247804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c009a70 con 0x7f3a9413cfa0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.340+0000 7f3a93400640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 0x7f3a94156fc0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00c430 tx=0x7f3a7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.340+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c005ce0 con 0x7f3a9413cfa0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.340+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7c005e40 con 0x7f3a9413cfa0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.340+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a7c0072f0 con 0x7f3a9413cfa0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.340+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a9413ac00 con 0x7f3a9413cfa0 2024-01-16T15:15:28.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.340+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a94154b90 con 0x7f3a9413cfa0 2024-01-16T15:15:28.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.342+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a9413b120 con 0x7f3a9413cfa0 2024-01-16T15:15:28.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.342+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3a7c007450 con 0x7f3a9413cfa0 2024-01-16T15:15:28.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.343+0000 7f3a8be00640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3a6c06c160 0x7f3a6c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.343+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3a7c087fc0 con 0x7f3a9413cfa0 2024-01-16T15:15:28.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.344+0000 7f3a93e00640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3a6c06c160 0x7f3a6c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.345+0000 7f3a93e00640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3a6c06c160 0x7f3a6c06e620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3a84009a10 tx=0x7f3a84008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.348+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3a9413b120 con 0x7f3a9413cfa0 2024-01-16T15:15:28.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.616+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f3a9413b120 con 0x7f3a9413cfa0 2024-01-16T15:15:28.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.617+0000 7f3a8be00640 1 -- 172.21.15.53:0/854247804 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1260 (secure 0 0 0) 0x7f3a9413b120 con 0x7f3a9413cfa0 2024-01-16T15:15:28.601 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:28.602 INFO:teuthology.orchestra.run.smithi053.stdout:{"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":955,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":99,"num_osds":8,"num_up_osds":8,"osd_up_since":1705417185,"num_in_osds":8,"osd_in_since":1705417176,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":80},{"state_name":"creating+peering","count":17}],"num_pgs":97,"num_pools":3,"num_objects":17,"data_bytes":591096,"bytes_used":595599360,"bytes_avail":772498513920,"bytes_total":773094113280,"inactive_pgs_ratio":0.17525772750377655,"write_bytes_sec":1706,"read_op_per_sec":0,"write_op_per_sec":5},"fsmap":{"epoch":64,"id":7,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":7,"rank":0,"name":"c","status":"up:active","gid":8508}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":17,"modified":"2024-01-16T15:14:40.563540+0000","services":{"mgr":{"daemons":{"summary":"","y":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-01-16T15:15:28.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3a6c06c160 msgr2=0x7f3a6c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3a6c06c160 0x7f3a6c06e620 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f3a84009a10 tx=0x7f3a84008040 comp rx=0 tx=0).stop 2024-01-16T15:15:28.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 msgr2=0x7f3a94156fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 0x7f3a94156fc0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f3a7c00c430 tx=0x7f3a7c00c8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 shutdown_connections 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3a6c06c160 0x7f3a6c06e620 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3a9413fcb0 0x7f3a94157500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3a9413cfa0 0x7f3a94156fc0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 --2- 172.21.15.53:0/854247804 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3a940cd530 0x7f3a94071550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.619+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 >> 172.21.15.53:0/854247804 conn(0x7f3a9406a800 msgr2=0x7f3a94153bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.620+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 shutdown_connections 2024-01-16T15:15:28.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.620+0000 7f3a99600640 1 -- 172.21.15.53:0/854247804 wait complete. 2024-01-16T15:15:28.622 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-01-16T15:15:28.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:28.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.857+0000 7f36c7200640 1 Processor -- start 2024-01-16T15:15:28.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.857+0000 7f36c7200640 1 -- start start 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 0x7f36c815afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f36c813fcb0 0x7f36c80b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c80cdec0 con 0x7f36c80cd530 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c80ce1a0 con 0x7f36c813fcb0 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80ce480 con 0x7f36c813cfa0 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c5400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f36c80cd530 0x7f36c813f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55576/0 (socket says 172.21.15.53:55576) 2024-01-16T15:15:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.859+0000 7f36c5400640 1 -- 172.21.15.53:0/3192167660 learned_addr learned my addr 172.21.15.53:0/3192167660 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:28.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c5e00640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 0x7f36c815afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c6800640 1 --2- 172.21.15.53:0/3192167660 >> v2:172.21.15.146:3300/0 conn(0x7f36c813fcb0 0x7f36c80b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632110083 0 0) 0x7f36c80cdec0 con 0x7f36c80cd530 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a8003520 con 0x7f36c80cd530 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877557700 0 0) 0x7f36a8003520 con 0x7f36c80cd530 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 msgr2=0x7f36c815afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 0x7f36c815afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 >> v2:172.21.15.146:3300/0 conn(0x7f36c813fcb0 msgr2=0x7f36c80b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 --2- 172.21.15.53:0/3192167660 >> v2:172.21.15.146:3300/0 conn(0x7f36c813fcb0 0x7f36c80b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c80ce760 con 0x7f36c80cd530 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f36b0001670 con 0x7f36c80cd530 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 >> v1:172.21.15.53:6789/0 conn(0x7f36c80cd530 legacy=0x7f36c813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.860+0000 7f36c4a00640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 0x7f36a8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c4a00640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36a8003f40 0x7f36a80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c4a00640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a8004d00 0x7f36a8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80ce1a0 con 0x7f36a8004d00 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80ce480 con 0x7f36a8003f40 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36a800ba30 con 0x7f36c813cfa0 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c6800640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a8004d00 0x7f36a8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c5e00640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 0x7f36a8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c6800640 1 -- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 msgr2=0x7f36a8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c6800640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c813cfa0 0x7f36a8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c6800640 1 -- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36a8003f40 msgr2=0x7f36a80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c6800640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36a8003f40 0x7f36a80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.861+0000 7f36c6800640 1 -- 172.21.15.53:0/3192167660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36a8003520 con 0x7f36a8004d00 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.862+0000 7f36c6800640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a8004d00 0x7f36a8005180 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009fc0 tx=0x7f36a80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd87426880177b4e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c002cc60 con 0x7f36a8004d00 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c000f040 con 0x7f36a8004d00 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c4a00640 1 -- 172.21.15.53:0/3192167660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0034440 con 0x7f36a8004d00 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c7200640 1 -- 172.21.15.53:0/3192167660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a8004d00 msgr2=0x7f36a8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c7200640 1 --2- 172.21.15.53:0/3192167660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36a8004d00 0x7f36a8005180 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009fc0 tx=0x7f36a80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c6800640 1 -- 172.21.15.53:0/3192167660 reap_dead start 2024-01-16T15:15:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c7200640 1 -- 172.21.15.53:0/3192167660 shutdown_connections 2024-01-16T15:15:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.863+0000 7f36c7200640 1 -- 172.21.15.53:0/3192167660 >> 172.21.15.53:0/3192167660 conn(0x7f36c806a800 msgr2=0x7f36c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.864+0000 7f36c7200640 1 -- 172.21.15.53:0/3192167660 shutdown_connections 2024-01-16T15:15:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.864+0000 7f36c7200640 1 -- 172.21.15.53:0/3192167660 wait complete. 2024-01-16T15:15:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.865+0000 7f36c7200640 1 Processor -- start 2024-01-16T15:15:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.866+0000 7f36c7200640 1 -- start start 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.866+0000 7f36c7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36a8004d00 0x7f36c814c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c814cb80 0x7f36c80aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 0x7f36c814fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c80af660 con 0x7f36c80af1e0 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c813d410 con 0x7f36c814cb80 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c813d6f0 con 0x7f36a8004d00 2024-01-16T15:15:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 0x7f36c814fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 0x7f36c814fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36030/0 (socket says 172.21.15.53:36030) 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 -- 172.21.15.53:0/2030720723 learned_addr learned my addr 172.21.15.53:0/2030720723 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c5e00640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36a8004d00 0x7f36c814c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 -- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36a8004d00 msgr2=0x7f36c814c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36a8004d00 0x7f36c814c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 -- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c814cb80 msgr2=0x7f36c80aeca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c814cb80 0x7f36c80aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.867+0000 7f36c6800640 1 -- 172.21.15.53:0/2030720723 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c813d9d0 con 0x7f36c80af1e0 2024-01-16T15:15:28.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.868+0000 7f36c6800640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 0x7f36c814fdf0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f36c0037040 tx=0x7f36c0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.868+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0034d50 con 0x7f36c80af1e0 2024-01-16T15:15:28.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.868+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0003990 con 0x7f36c80af1e0 2024-01-16T15:15:28.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.868+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0003d40 con 0x7f36c80af1e0 2024-01-16T15:15:28.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.868+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c813dcb0 con 0x7f36c80af1e0 2024-01-16T15:15:28.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.868+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c813e270 con 0x7f36c80af1e0 2024-01-16T15:15:28.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.870+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c813e460 con 0x7f36c80af1e0 2024-01-16T15:15:28.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.871+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f36c0004030 con 0x7f36c80af1e0 2024-01-16T15:15:28.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.871+0000 7f36be000640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36a006c1e0 0x7f36a006e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:28.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.872+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f36c00aa460 con 0x7f36c80af1e0 2024-01-16T15:15:28.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.872+0000 7f36c5e00640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36a006c1e0 0x7f36a006e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:28.858 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.873+0000 7f36c5e00640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36a006c1e0 0x7f36a006e6a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f36c80ae820 tx=0x7f36b8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:28.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:28.876+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f36c0076b30 con 0x7f36c80af1e0 2024-01-16T15:15:29.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.069+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f36c815afb0 con 0x7f36a006c1e0 2024-01-16T15:15:29.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.092+0000 7f36be000640 1 -- 172.21.15.53:0/2030720723 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286954 (secure 0 0 0) 0x7f36b80095c0 con 0x7f36a006c1e0 2024-01-16T15:15:29.080 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:29.082 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36a006c1e0 msgr2=0x7f36a006e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36a006c1e0 0x7f36a006e6a0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f36c80ae820 tx=0x7f36b8008040 comp rx=0 tx=0).stop 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 msgr2=0x7f36c814fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 0x7f36c814fdf0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f36c0037040 tx=0x7f36c0009750 comp rx=0 tx=0).stop 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 shutdown_connections 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36a006c1e0 0x7f36a006e6a0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c80af1e0 0x7f36c814fdf0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c814cb80 0x7f36c80aeca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 --2- 172.21.15.53:0/2030720723 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36a8004d00 0x7f36c814c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 >> 172.21.15.53:0/2030720723 conn(0x7f36c806a800 msgr2=0x7f36c8071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 shutdown_connections 2024-01-16T15:15:29.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:29.098+0000 7f36c7200640 1 -- 172.21.15.53:0/2030720723 wait complete. 2024-01-16T15:15:29.104 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":564,"stamp":"2024-01-16T15:15:28.580634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581724,"kb_used_data":10728,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":754392996,"statfs":{"total":773094113280,"available":772498427904,"internally_reserved":0,"allocated":10985472,"data_stored":5388908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":584568368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.081697"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691428+0000","last_change":"2024-01-16T15:15:22.537519+0000","last_active":"2024-01-16T15:15:23.691428+0000","last_peered":"2024-01-16T15:15:23.691428+0000","last_clean":"2024-01-16T15:15:23.691428+0000","last_became_active":"2024-01-16T15:15:22.535630+0000","last_became_peered":"2024-01-16T15:15:22.535630+0000","last_unstale":"2024-01-16T15:15:23.691428+0000","last_undegraded":"2024-01-16T15:15:23.691428+0000","last_fullsized":"2024-01-16T15:15:23.691428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:53.424772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740792+0000","last_change":"2024-01-16T15:15:22.532497+0000","last_active":"2024-01-16T15:15:23.740792+0000","last_peered":"2024-01-16T15:15:23.740792+0000","last_clean":"2024-01-16T15:15:23.740792+0000","last_became_active":"2024-01-16T15:15:22.531043+0000","last_became_peered":"2024-01-16T15:15:22.531043+0000","last_unstale":"2024-01-16T15:15:23.740792+0000","last_undegraded":"2024-01-16T15:15:23.740792+0000","last_fullsized":"2024-01-16T15:15:23.740792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691119+0000","last_change":"2024-01-16T15:15:22.535023+0000","last_active":"2024-01-16T15:15:23.691119+0000","last_peered":"2024-01-16T15:15:23.691119+0000","last_clean":"2024-01-16T15:15:23.691119+0000","last_became_active":"2024-01-16T15:15:22.534684+0000","last_became_peered":"2024-01-16T15:15:22.534684+0000","last_unstale":"2024-01-16T15:15:23.691119+0000","last_undegraded":"2024-01-16T15:15:23.691119+0000","last_fullsized":"2024-01-16T15:15:23.691119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:00.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509733+0000","last_change":"2024-01-16T15:15:22.524150+0000","last_active":"2024-01-16T15:15:23.509733+0000","last_peered":"2024-01-16T15:15:23.509733+0000","last_clean":"2024-01-16T15:15:23.509733+0000","last_became_active":"2024-01-16T15:15:22.522946+0000","last_became_peered":"2024-01-16T15:15:22.522946+0000","last_unstale":"2024-01-16T15:15:23.509733+0000","last_undegraded":"2024-01-16T15:15:23.509733+0000","last_fullsized":"2024-01-16T15:15:23.509733+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:13.223095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006320+0000","last_change":"2024-01-16T15:15:22.545386+0000","last_active":"2024-01-16T15:15:24.006320+0000","last_peered":"2024-01-16T15:15:24.006320+0000","last_clean":"2024-01-16T15:15:24.006320+0000","last_became_active":"2024-01-16T15:15:22.544057+0000","last_became_peered":"2024-01-16T15:15:22.544057+0000","last_unstale":"2024-01-16T15:15:24.006320+0000","last_undegraded":"2024-01-16T15:15:24.006320+0000","last_fullsized":"2024-01-16T15:15:24.006320+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:30:46.298808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005648+0000","last_change":"2024-01-16T15:15:22.536600+0000","last_active":"2024-01-16T15:15:24.005648+0000","last_peered":"2024-01-16T15:15:24.005648+0000","last_clean":"2024-01-16T15:15:24.005648+0000","last_became_active":"2024-01-16T15:15:22.536327+0000","last_became_peered":"2024-01-16T15:15:22.536327+0000","last_unstale":"2024-01-16T15:15:24.005648+0000","last_undegraded":"2024-01-16T15:15:24.005648+0000","last_fullsized":"2024-01-16T15:15:24.005648+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005398+0000","last_change":"2024-01-16T15:15:22.535575+0000","last_active":"2024-01-16T15:15:24.005398+0000","last_peered":"2024-01-16T15:15:24.005398+0000","last_clean":"2024-01-16T15:15:24.005398+0000","last_became_active":"2024-01-16T15:15:22.535292+0000","last_became_peered":"2024-01-16T15:15:22.535292+0000","last_unstale":"2024-01-16T15:15:24.005398+0000","last_undegraded":"2024-01-16T15:15:24.005398+0000","last_fullsized":"2024-01-16T15:15:24.005398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:06.443796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740110+0000","last_change":"2024-01-16T15:15:22.521459+0000","last_active":"2024-01-16T15:15:23.740110+0000","last_peered":"2024-01-16T15:15:23.740110+0000","last_clean":"2024-01-16T15:15:23.740110+0000","last_became_active":"2024-01-16T15:15:22.520884+0000","last_became_peered":"2024-01-16T15:15:22.520884+0000","last_unstale":"2024-01-16T15:15:23.740110+0000","last_undegraded":"2024-01-16T15:15:23.740110+0000","last_fullsized":"2024-01-16T15:15:23.740110+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511641+0000","last_change":"2024-01-16T15:15:22.526500+0000","last_active":"2024-01-16T15:15:23.511641+0000","last_peered":"2024-01-16T15:15:23.511641+0000","last_clean":"2024-01-16T15:15:23.511641+0000","last_became_active":"2024-01-16T15:15:22.526166+0000","last_became_peered":"2024-01-16T15:15:22.526166+0000","last_unstale":"2024-01-16T15:15:23.511641+0000","last_undegraded":"2024-01-16T15:15:23.511641+0000","last_fullsized":"2024-01-16T15:15:23.511641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:06.510364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007360+0000","last_change":"2024-01-16T15:15:22.542938+0000","last_active":"2024-01-16T15:15:24.007360+0000","last_peered":"2024-01-16T15:15:24.007360+0000","last_clean":"2024-01-16T15:15:24.007360+0000","last_became_active":"2024-01-16T15:15:22.542687+0000","last_became_peered":"2024-01-16T15:15:22.542687+0000","last_unstale":"2024-01-16T15:15:24.007360+0000","last_undegraded":"2024-01-16T15:15:24.007360+0000","last_fullsized":"2024-01-16T15:15:24.007360+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:12.904233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005800+0000","last_change":"2024-01-16T15:15:22.533081+0000","last_active":"2024-01-16T15:15:24.005800+0000","last_peered":"2024-01-16T15:15:24.005800+0000","last_clean":"2024-01-16T15:15:24.005800+0000","last_became_active":"2024-01-16T15:15:22.529012+0000","last_became_peered":"2024-01-16T15:15:22.529012+0000","last_unstale":"2024-01-16T15:15:24.005800+0000","last_undegraded":"2024-01-16T15:15:24.005800+0000","last_fullsized":"2024-01-16T15:15:24.005800+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511625+0000","last_change":"2024-01-16T15:15:22.522860+0000","last_active":"2024-01-16T15:15:23.511625+0000","last_peered":"2024-01-16T15:15:23.511625+0000","last_clean":"2024-01-16T15:15:23.511625+0000","last_became_active":"2024-01-16T15:15:22.521459+0000","last_became_peered":"2024-01-16T15:15:22.521459+0000","last_unstale":"2024-01-16T15:15:23.511625+0000","last_undegraded":"2024-01-16T15:15:23.511625+0000","last_fullsized":"2024-01-16T15:15:23.511625+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509163+0000","last_change":"2024-01-16T15:15:22.532050+0000","last_active":"2024-01-16T15:15:23.509163+0000","last_peered":"2024-01-16T15:15:23.509163+0000","last_clean":"2024-01-16T15:15:23.509163+0000","last_became_active":"2024-01-16T15:15:22.531307+0000","last_became_peered":"2024-01-16T15:15:22.531307+0000","last_unstale":"2024-01-16T15:15:23.509163+0000","last_undegraded":"2024-01-16T15:15:23.509163+0000","last_fullsized":"2024-01-16T15:15:23.509163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:34.860510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609381+0000","last_change":"2024-01-16T15:15:22.533322+0000","last_active":"2024-01-16T15:15:23.609381+0000","last_peered":"2024-01-16T15:15:23.609381+0000","last_clean":"2024-01-16T15:15:23.609381+0000","last_became_active":"2024-01-16T15:15:22.532960+0000","last_became_peered":"2024-01-16T15:15:22.532960+0000","last_unstale":"2024-01-16T15:15:23.609381+0000","last_undegraded":"2024-01-16T15:15:23.609381+0000","last_fullsized":"2024-01-16T15:15:23.609381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:47.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508639+0000","last_change":"2024-01-16T15:15:22.532776+0000","last_active":"2024-01-16T15:15:23.508639+0000","last_peered":"2024-01-16T15:15:23.508639+0000","last_clean":"2024-01-16T15:15:23.508639+0000","last_became_active":"2024-01-16T15:15:22.532494+0000","last_became_peered":"2024-01-16T15:15:22.532494+0000","last_unstale":"2024-01-16T15:15:23.508639+0000","last_undegraded":"2024-01-16T15:15:23.508639+0000","last_fullsized":"2024-01-16T15:15:23.508639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005108+0000","last_change":"2024-01-16T15:15:22.530744+0000","last_active":"2024-01-16T15:15:24.005108+0000","last_peered":"2024-01-16T15:15:24.005108+0000","last_clean":"2024-01-16T15:15:24.005108+0000","last_became_active":"2024-01-16T15:15:22.525764+0000","last_became_peered":"2024-01-16T15:15:22.525764+0000","last_unstale":"2024-01-16T15:15:24.005108+0000","last_undegraded":"2024-01-16T15:15:24.005108+0000","last_fullsized":"2024-01-16T15:15:24.005108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:51.206294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741415+0000","last_change":"2024-01-16T15:15:22.523512+0000","last_active":"2024-01-16T15:15:23.741415+0000","last_peered":"2024-01-16T15:15:23.741415+0000","last_clean":"2024-01-16T15:15:23.741415+0000","last_became_active":"2024-01-16T15:15:22.521121+0000","last_became_peered":"2024-01-16T15:15:22.521121+0000","last_unstale":"2024-01-16T15:15:23.741415+0000","last_undegraded":"2024-01-16T15:15:23.741415+0000","last_fullsized":"2024-01-16T15:15:23.741415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:35.046638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509378+0000","last_change":"2024-01-16T15:15:22.531166+0000","last_active":"2024-01-16T15:15:23.509378+0000","last_peered":"2024-01-16T15:15:23.509378+0000","last_clean":"2024-01-16T15:15:23.509378+0000","last_became_active":"2024-01-16T15:15:22.530595+0000","last_became_peered":"2024-01-16T15:15:22.530595+0000","last_unstale":"2024-01-16T15:15:23.509378+0000","last_undegraded":"2024-01-16T15:15:23.509378+0000","last_fullsized":"2024-01-16T15:15:23.509378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:01.070526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610445+0000","last_change":"2024-01-16T15:15:22.524237+0000","last_active":"2024-01-16T15:15:23.610445+0000","last_peered":"2024-01-16T15:15:23.610445+0000","last_clean":"2024-01-16T15:15:23.610445+0000","last_became_active":"2024-01-16T15:15:22.523982+0000","last_became_peered":"2024-01-16T15:15:22.523982+0000","last_unstale":"2024-01-16T15:15:23.610445+0000","last_undegraded":"2024-01-16T15:15:23.610445+0000","last_fullsized":"2024-01-16T15:15:23.610445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:34.668807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511753+0000","last_change":"2024-01-16T15:15:22.523069+0000","last_active":"2024-01-16T15:15:23.511753+0000","last_peered":"2024-01-16T15:15:23.511753+0000","last_clean":"2024-01-16T15:15:23.511753+0000","last_became_active":"2024-01-16T15:15:22.522232+0000","last_became_peered":"2024-01-16T15:15:22.522232+0000","last_unstale":"2024-01-16T15:15:23.511753+0000","last_undegraded":"2024-01-16T15:15:23.511753+0000","last_fullsized":"2024-01-16T15:15:23.511753+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609784+0000","last_change":"2024-01-16T15:15:22.529446+0000","last_active":"2024-01-16T15:15:23.609784+0000","last_peered":"2024-01-16T15:15:23.609784+0000","last_clean":"2024-01-16T15:15:23.609784+0000","last_became_active":"2024-01-16T15:15:22.529124+0000","last_became_peered":"2024-01-16T15:15:22.529124+0000","last_unstale":"2024-01-16T15:15:23.609784+0000","last_undegraded":"2024-01-16T15:15:23.609784+0000","last_fullsized":"2024-01-16T15:15:23.609784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:14.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740836+0000","last_change":"2024-01-16T15:15:22.541357+0000","last_active":"2024-01-16T15:15:23.740836+0000","last_peered":"2024-01-16T15:15:23.740836+0000","last_clean":"2024-01-16T15:15:23.740836+0000","last_became_active":"2024-01-16T15:15:22.539470+0000","last_became_peered":"2024-01-16T15:15:22.539470+0000","last_unstale":"2024-01-16T15:15:23.740836+0000","last_undegraded":"2024-01-16T15:15:23.740836+0000","last_fullsized":"2024-01-16T15:15:23.740836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740324+0000","last_change":"2024-01-16T15:15:22.528944+0000","last_active":"2024-01-16T15:15:23.740324+0000","last_peered":"2024-01-16T15:15:23.740324+0000","last_clean":"2024-01-16T15:15:23.740324+0000","last_became_active":"2024-01-16T15:15:22.528668+0000","last_became_peered":"2024-01-16T15:15:22.528668+0000","last_unstale":"2024-01-16T15:15:23.740324+0000","last_undegraded":"2024-01-16T15:15:23.740324+0000","last_fullsized":"2024-01-16T15:15:23.740324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005396+0000","last_change":"2024-01-16T15:15:22.527598+0000","last_active":"2024-01-16T15:15:24.005396+0000","last_peered":"2024-01-16T15:15:24.005396+0000","last_clean":"2024-01-16T15:15:24.005396+0000","last_became_active":"2024-01-16T15:15:22.526241+0000","last_became_peered":"2024-01-16T15:15:22.526241+0000","last_unstale":"2024-01-16T15:15:24.005396+0000","last_undegraded":"2024-01-16T15:15:24.005396+0000","last_fullsized":"2024-01-16T15:15:24.005396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:41.199256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007216+0000","last_change":"2024-01-16T15:15:22.537798+0000","last_active":"2024-01-16T15:15:24.007216+0000","last_peered":"2024-01-16T15:15:24.007216+0000","last_clean":"2024-01-16T15:15:24.007216+0000","last_became_active":"2024-01-16T15:15:22.536669+0000","last_became_peered":"2024-01-16T15:15:22.536669+0000","last_unstale":"2024-01-16T15:15:24.007216+0000","last_undegraded":"2024-01-16T15:15:24.007216+0000","last_fullsized":"2024-01-16T15:15:24.007216+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:06:49.471608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741073+0000","last_change":"2024-01-16T15:15:22.518878+0000","last_active":"2024-01-16T15:15:23.741073+0000","last_peered":"2024-01-16T15:15:23.741073+0000","last_clean":"2024-01-16T15:15:23.741073+0000","last_became_active":"2024-01-16T15:15:22.518594+0000","last_became_peered":"2024-01-16T15:15:22.518594+0000","last_unstale":"2024-01-16T15:15:23.741073+0000","last_undegraded":"2024-01-16T15:15:23.741073+0000","last_fullsized":"2024-01-16T15:15:23.741073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006005+0000","last_change":"2024-01-16T15:15:22.529986+0000","last_active":"2024-01-16T15:15:24.006005+0000","last_peered":"2024-01-16T15:15:24.006005+0000","last_clean":"2024-01-16T15:15:24.006005+0000","last_became_active":"2024-01-16T15:15:22.525092+0000","last_became_peered":"2024-01-16T15:15:22.525092+0000","last_unstale":"2024-01-16T15:15:24.006005+0000","last_undegraded":"2024-01-16T15:15:24.006005+0000","last_fullsized":"2024-01-16T15:15:24.006005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006703+0000","last_change":"2024-01-16T15:15:22.531986+0000","last_active":"2024-01-16T15:15:24.006703+0000","last_peered":"2024-01-16T15:15:24.006703+0000","last_clean":"2024-01-16T15:15:24.006703+0000","last_became_active":"2024-01-16T15:15:22.531247+0000","last_became_peered":"2024-01-16T15:15:22.531247+0000","last_unstale":"2024-01-16T15:15:24.006703+0000","last_undegraded":"2024-01-16T15:15:24.006703+0000","last_fullsized":"2024-01-16T15:15:24.006703+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:21.901076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740772+0000","last_change":"2024-01-16T15:15:22.518786+0000","last_active":"2024-01-16T15:15:23.740772+0000","last_peered":"2024-01-16T15:15:23.740772+0000","last_clean":"2024-01-16T15:15:23.740772+0000","last_became_active":"2024-01-16T15:15:22.512965+0000","last_became_peered":"2024-01-16T15:15:22.512965+0000","last_unstale":"2024-01-16T15:15:23.740772+0000","last_undegraded":"2024-01-16T15:15:23.740772+0000","last_fullsized":"2024-01-16T15:15:23.740772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609350+0000","last_change":"2024-01-16T15:15:22.521137+0000","last_active":"2024-01-16T15:15:23.609350+0000","last_peered":"2024-01-16T15:15:23.609350+0000","last_clean":"2024-01-16T15:15:23.609350+0000","last_became_active":"2024-01-16T15:15:22.520842+0000","last_became_peered":"2024-01-16T15:15:22.520842+0000","last_unstale":"2024-01-16T15:15:23.609350+0000","last_undegraded":"2024-01-16T15:15:23.609350+0000","last_fullsized":"2024-01-16T15:15:23.609350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:02.829205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508652+0000","last_change":"2024-01-16T15:15:22.541257+0000","last_active":"2024-01-16T15:15:23.508652+0000","last_peered":"2024-01-16T15:15:23.508652+0000","last_clean":"2024-01-16T15:15:23.508652+0000","last_became_active":"2024-01-16T15:15:22.540754+0000","last_became_peered":"2024-01-16T15:15:22.540754+0000","last_unstale":"2024-01-16T15:15:23.508652+0000","last_undegraded":"2024-01-16T15:15:23.508652+0000","last_fullsized":"2024-01-16T15:15:23.508652+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:10.664582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690290+0000","last_change":"2024-01-16T15:15:22.516532+0000","last_active":"2024-01-16T15:15:23.690290+0000","last_peered":"2024-01-16T15:15:23.690290+0000","last_clean":"2024-01-16T15:15:23.690290+0000","last_became_active":"2024-01-16T15:15:22.515440+0000","last_became_peered":"2024-01-16T15:15:22.515440+0000","last_unstale":"2024-01-16T15:15:23.690290+0000","last_undegraded":"2024-01-16T15:15:23.690290+0000","last_fullsized":"2024-01-16T15:15:23.690290+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:42.093044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610359+0000","last_change":"2024-01-16T15:15:22.521967+0000","last_active":"2024-01-16T15:15:23.610359+0000","last_peered":"2024-01-16T15:15:23.610359+0000","last_clean":"2024-01-16T15:15:23.610359+0000","last_became_active":"2024-01-16T15:15:22.520663+0000","last_became_peered":"2024-01-16T15:15:22.520663+0000","last_unstale":"2024-01-16T15:15:23.610359+0000","last_undegraded":"2024-01-16T15:15:23.610359+0000","last_fullsized":"2024-01-16T15:15:23.610359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:27.652785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741472+0000","last_change":"2024-01-16T15:15:21.529234+0000","last_active":"2024-01-16T15:15:23.741472+0000","last_peered":"2024-01-16T15:15:23.741472+0000","last_clean":"2024-01-16T15:15:23.741472+0000","last_became_active":"2024-01-16T15:15:21.528711+0000","last_became_peered":"2024-01-16T15:15:21.528711+0000","last_unstale":"2024-01-16T15:15:23.741472+0000","last_undegraded":"2024-01-16T15:15:23.741472+0000","last_fullsized":"2024-01-16T15:15:23.741472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:34.638361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509939+0000","last_change":"2024-01-16T15:15:22.540312+0000","last_active":"2024-01-16T15:15:23.509939+0000","last_peered":"2024-01-16T15:15:23.509939+0000","last_clean":"2024-01-16T15:15:23.509939+0000","last_became_active":"2024-01-16T15:15:22.538619+0000","last_became_peered":"2024-01-16T15:15:22.538619+0000","last_unstale":"2024-01-16T15:15:23.509939+0000","last_undegraded":"2024-01-16T15:15:23.509939+0000","last_fullsized":"2024-01-16T15:15:23.509939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:02.521266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.542686+0000","last_change":"2024-01-16T15:15:21.516982+0000","last_active":"2024-01-16T15:15:22.542686+0000","last_peered":"2024-01-16T15:15:22.542686+0000","last_clean":"2024-01-16T15:15:22.542686+0000","last_became_active":"2024-01-16T15:15:21.516459+0000","last_became_peered":"2024-01-16T15:15:21.516459+0000","last_unstale":"2024-01-16T15:15:22.542686+0000","last_undegraded":"2024-01-16T15:15:22.542686+0000","last_fullsized":"2024-01-16T15:15:22.542686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690957+0000","last_change":"2024-01-16T15:15:22.522691+0000","last_active":"2024-01-16T15:15:23.690957+0000","last_peered":"2024-01-16T15:15:23.690957+0000","last_clean":"2024-01-16T15:15:23.690957+0000","last_became_active":"2024-01-16T15:15:22.522375+0000","last_became_peered":"2024-01-16T15:15:22.522375+0000","last_unstale":"2024-01-16T15:15:23.690957+0000","last_undegraded":"2024-01-16T15:15:23.690957+0000","last_fullsized":"2024-01-16T15:15:23.690957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:12.190032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740678+0000","last_change":"2024-01-16T15:15:21.529660+0000","last_active":"2024-01-16T15:15:23.740678+0000","last_peered":"2024-01-16T15:15:23.740678+0000","last_clean":"2024-01-16T15:15:23.740678+0000","last_became_active":"2024-01-16T15:15:21.529009+0000","last_became_peered":"2024-01-16T15:15:21.529009+0000","last_unstale":"2024-01-16T15:15:23.740678+0000","last_undegraded":"2024-01-16T15:15:23.740678+0000","last_fullsized":"2024-01-16T15:15:23.740678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:49.081309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511419+0000","last_change":"2024-01-16T15:15:22.539245+0000","last_active":"2024-01-16T15:15:23.511419+0000","last_peered":"2024-01-16T15:15:23.511419+0000","last_clean":"2024-01-16T15:15:23.511419+0000","last_became_active":"2024-01-16T15:15:22.538962+0000","last_became_peered":"2024-01-16T15:15:22.538962+0000","last_unstale":"2024-01-16T15:15:23.511419+0000","last_undegraded":"2024-01-16T15:15:23.511419+0000","last_fullsized":"2024-01-16T15:15:23.511419+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:56.014379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740604+0000","last_change":"2024-01-16T15:15:21.521824+0000","last_active":"2024-01-16T15:15:23.740604+0000","last_peered":"2024-01-16T15:15:23.740604+0000","last_clean":"2024-01-16T15:15:23.740604+0000","last_became_active":"2024-01-16T15:15:21.521406+0000","last_became_peered":"2024-01-16T15:15:21.521406+0000","last_unstale":"2024-01-16T15:15:23.740604+0000","last_undegraded":"2024-01-16T15:15:23.740604+0000","last_fullsized":"2024-01-16T15:15:23.740604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:39.695221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004361+0000","last_change":"2024-01-16T15:15:22.540055+0000","last_active":"2024-01-16T15:15:24.004361+0000","last_peered":"2024-01-16T15:15:24.004361+0000","last_clean":"2024-01-16T15:15:24.004361+0000","last_became_active":"2024-01-16T15:15:22.538674+0000","last_became_peered":"2024-01-16T15:15:22.538674+0000","last_unstale":"2024-01-16T15:15:24.004361+0000","last_undegraded":"2024-01-16T15:15:24.004361+0000","last_fullsized":"2024-01-16T15:15:24.004361+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:37.034062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690792+0000","last_change":"2024-01-16T15:15:21.530799+0000","last_active":"2024-01-16T15:15:23.690792+0000","last_peered":"2024-01-16T15:15:23.690792+0000","last_clean":"2024-01-16T15:15:23.690792+0000","last_became_active":"2024-01-16T15:15:21.530451+0000","last_became_peered":"2024-01-16T15:15:21.530451+0000","last_unstale":"2024-01-16T15:15:23.690792+0000","last_undegraded":"2024-01-16T15:15:23.690792+0000","last_fullsized":"2024-01-16T15:15:23.690792+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:26.637253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005884+0000","last_change":"2024-01-16T15:15:22.533064+0000","last_active":"2024-01-16T15:15:24.005884+0000","last_peered":"2024-01-16T15:15:24.005884+0000","last_clean":"2024-01-16T15:15:24.005884+0000","last_became_active":"2024-01-16T15:15:22.532644+0000","last_became_peered":"2024-01-16T15:15:22.532644+0000","last_unstale":"2024-01-16T15:15:24.005884+0000","last_undegraded":"2024-01-16T15:15:24.005884+0000","last_fullsized":"2024-01-16T15:15:24.005884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740860+0000","last_change":"2024-01-16T15:15:21.525811+0000","last_active":"2024-01-16T15:15:23.740860+0000","last_peered":"2024-01-16T15:15:23.740860+0000","last_clean":"2024-01-16T15:15:23.740860+0000","last_became_active":"2024-01-16T15:15:21.525152+0000","last_became_peered":"2024-01-16T15:15:21.525152+0000","last_unstale":"2024-01-16T15:15:23.740860+0000","last_undegraded":"2024-01-16T15:15:23.740860+0000","last_fullsized":"2024-01-16T15:15:23.740860+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:38.101810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508468+0000","last_change":"2024-01-16T15:15:22.524402+0000","last_active":"2024-01-16T15:15:23.508468+0000","last_peered":"2024-01-16T15:15:23.508468+0000","last_clean":"2024-01-16T15:15:23.508468+0000","last_became_active":"2024-01-16T15:15:22.524129+0000","last_became_peered":"2024-01-16T15:15:22.524129+0000","last_unstale":"2024-01-16T15:15:23.508468+0000","last_undegraded":"2024-01-16T15:15:23.508468+0000","last_fullsized":"2024-01-16T15:15:23.508468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.532301+0000","last_change":"2024-01-16T15:15:21.532465+0000","last_active":"2024-01-16T15:15:22.532301+0000","last_peered":"2024-01-16T15:15:22.532301+0000","last_clean":"2024-01-16T15:15:22.532301+0000","last_became_active":"2024-01-16T15:15:21.532238+0000","last_became_peered":"2024-01-16T15:15:21.532238+0000","last_unstale":"2024-01-16T15:15:22.532301+0000","last_undegraded":"2024-01-16T15:15:22.532301+0000","last_fullsized":"2024-01-16T15:15:22.532301+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:46.120683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511165+0000","last_change":"2024-01-16T15:15:22.530629+0000","last_active":"2024-01-16T15:15:23.511165+0000","last_peered":"2024-01-16T15:15:23.511165+0000","last_clean":"2024-01-16T15:15:23.511165+0000","last_became_active":"2024-01-16T15:15:22.530363+0000","last_became_peered":"2024-01-16T15:15:22.530363+0000","last_unstale":"2024-01-16T15:15:23.511165+0000","last_undegraded":"2024-01-16T15:15:23.511165+0000","last_fullsized":"2024-01-16T15:15:23.511165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:37.560637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690271+0000","last_change":"2024-01-16T15:15:21.524375+0000","last_active":"2024-01-16T15:15:23.690271+0000","last_peered":"2024-01-16T15:15:23.690271+0000","last_clean":"2024-01-16T15:15:23.690271+0000","last_became_active":"2024-01-16T15:15:21.523731+0000","last_became_peered":"2024-01-16T15:15:21.523731+0000","last_unstale":"2024-01-16T15:15:23.690271+0000","last_undegraded":"2024-01-16T15:15:23.690271+0000","last_fullsized":"2024-01-16T15:15:23.690271+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:13.884029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741391+0000","last_change":"2024-01-16T15:15:22.528073+0000","last_active":"2024-01-16T15:15:23.741391+0000","last_peered":"2024-01-16T15:15:23.741391+0000","last_clean":"2024-01-16T15:15:23.741391+0000","last_became_active":"2024-01-16T15:15:22.526190+0000","last_became_peered":"2024-01-16T15:15:22.526190+0000","last_unstale":"2024-01-16T15:15:23.741391+0000","last_undegraded":"2024-01-16T15:15:23.741391+0000","last_fullsized":"2024-01-16T15:15:23.741391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:55.442576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691500+0000","last_change":"2024-01-16T15:15:21.520386+0000","last_active":"2024-01-16T15:15:23.691500+0000","last_peered":"2024-01-16T15:15:23.691500+0000","last_clean":"2024-01-16T15:15:23.691500+0000","last_became_active":"2024-01-16T15:15:21.520059+0000","last_became_peered":"2024-01-16T15:15:21.520059+0000","last_unstale":"2024-01-16T15:15:23.691500+0000","last_undegraded":"2024-01-16T15:15:23.691500+0000","last_fullsized":"2024-01-16T15:15:23.691500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:33.653275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511649+0000","last_change":"2024-01-16T15:15:22.539263+0000","last_active":"2024-01-16T15:15:23.511649+0000","last_peered":"2024-01-16T15:15:23.511649+0000","last_clean":"2024-01-16T15:15:23.511649+0000","last_became_active":"2024-01-16T15:15:22.538976+0000","last_became_peered":"2024-01-16T15:15:22.538976+0000","last_unstale":"2024-01-16T15:15:23.511649+0000","last_undegraded":"2024-01-16T15:15:23.511649+0000","last_fullsized":"2024-01-16T15:15:23.511649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:44.352885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509435+0000","last_change":"2024-01-16T15:15:21.534208+0000","last_active":"2024-01-16T15:15:23.509435+0000","last_peered":"2024-01-16T15:15:23.509435+0000","last_clean":"2024-01-16T15:15:23.509435+0000","last_became_active":"2024-01-16T15:15:21.533545+0000","last_became_peered":"2024-01-16T15:15:21.533545+0000","last_unstale":"2024-01-16T15:15:23.509435+0000","last_undegraded":"2024-01-16T15:15:23.509435+0000","last_fullsized":"2024-01-16T15:15:23.509435+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006816+0000","last_change":"2024-01-16T15:15:22.534236+0000","last_active":"2024-01-16T15:15:24.006816+0000","last_peered":"2024-01-16T15:15:24.006816+0000","last_clean":"2024-01-16T15:15:24.006816+0000","last_became_active":"2024-01-16T15:15:22.533922+0000","last_became_peered":"2024-01-16T15:15:22.533922+0000","last_unstale":"2024-01-16T15:15:24.006816+0000","last_undegraded":"2024-01-16T15:15:24.006816+0000","last_fullsized":"2024-01-16T15:15:24.006816+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:01.861870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.547699+0000","last_change":"2024-01-16T15:15:21.521285+0000","last_active":"2024-01-16T15:15:22.547699+0000","last_peered":"2024-01-16T15:15:22.547699+0000","last_clean":"2024-01-16T15:15:22.547699+0000","last_became_active":"2024-01-16T15:15:21.520717+0000","last_became_peered":"2024-01-16T15:15:21.520717+0000","last_unstale":"2024-01-16T15:15:22.547699+0000","last_undegraded":"2024-01-16T15:15:22.547699+0000","last_fullsized":"2024-01-16T15:15:22.547699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:15.674482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006046+0000","last_change":"2024-01-16T15:15:22.529757+0000","last_active":"2024-01-16T15:15:24.006046+0000","last_peered":"2024-01-16T15:15:24.006046+0000","last_clean":"2024-01-16T15:15:24.006046+0000","last_became_active":"2024-01-16T15:15:22.524620+0000","last_became_peered":"2024-01-16T15:15:22.524620+0000","last_unstale":"2024-01-16T15:15:24.006046+0000","last_undegraded":"2024-01-16T15:15:24.006046+0000","last_fullsized":"2024-01-16T15:15:24.006046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"98'6","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.552074+0000","last_change":"2024-01-16T15:15:21.518507+0000","last_active":"2024-01-16T15:15:22.552074+0000","last_peered":"2024-01-16T15:15:22.552074+0000","last_clean":"2024-01-16T15:15:22.552074+0000","last_became_active":"2024-01-16T15:15:21.518075+0000","last_became_peered":"2024-01-16T15:15:21.518075+0000","last_unstale":"2024-01-16T15:15:22.552074+0000","last_undegraded":"2024-01-16T15:15:22.552074+0000","last_fullsized":"2024-01-16T15:15:22.552074+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:55.375960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509009+0000","last_change":"2024-01-16T15:15:22.521859+0000","last_active":"2024-01-16T15:15:23.509009+0000","last_peered":"2024-01-16T15:15:23.509009+0000","last_clean":"2024-01-16T15:15:23.509009+0000","last_became_active":"2024-01-16T15:15:22.521561+0000","last_became_peered":"2024-01-16T15:15:22.521561+0000","last_unstale":"2024-01-16T15:15:23.509009+0000","last_undegraded":"2024-01-16T15:15:23.509009+0000","last_fullsized":"2024-01-16T15:15:23.509009+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:14.797953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609905+0000","last_change":"2024-01-16T15:15:21.521158+0000","last_active":"2024-01-16T15:15:23.609905+0000","last_peered":"2024-01-16T15:15:23.609905+0000","last_clean":"2024-01-16T15:15:23.609905+0000","last_became_active":"2024-01-16T15:15:21.520851+0000","last_became_peered":"2024-01-16T15:15:21.520851+0000","last_unstale":"2024-01-16T15:15:23.609905+0000","last_undegraded":"2024-01-16T15:15:23.609905+0000","last_fullsized":"2024-01-16T15:15:23.609905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:47.111965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690791+0000","last_change":"2024-01-16T15:15:22.530599+0000","last_active":"2024-01-16T15:15:23.690791+0000","last_peered":"2024-01-16T15:15:23.690791+0000","last_clean":"2024-01-16T15:15:23.690791+0000","last_became_active":"2024-01-16T15:15:22.526185+0000","last_became_peered":"2024-01-16T15:15:22.526185+0000","last_unstale":"2024-01-16T15:15:23.690791+0000","last_undegraded":"2024-01-16T15:15:23.690791+0000","last_fullsized":"2024-01-16T15:15:23.690791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:11.722014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609612+0000","last_change":"2024-01-16T15:15:21.526311+0000","last_active":"2024-01-16T15:15:23.609612+0000","last_peered":"2024-01-16T15:15:23.609612+0000","last_clean":"2024-01-16T15:15:23.609612+0000","last_became_active":"2024-01-16T15:15:21.526019+0000","last_became_peered":"2024-01-16T15:15:21.526019+0000","last_unstale":"2024-01-16T15:15:23.609612+0000","last_undegraded":"2024-01-16T15:15:23.609612+0000","last_fullsized":"2024-01-16T15:15:23.609612+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:30.739366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.510461+0000","last_change":"2024-01-16T15:15:22.530528+0000","last_active":"2024-01-16T15:15:23.510461+0000","last_peered":"2024-01-16T15:15:23.510461+0000","last_clean":"2024-01-16T15:15:23.510461+0000","last_became_active":"2024-01-16T15:15:22.530244+0000","last_became_peered":"2024-01-16T15:15:22.530244+0000","last_unstale":"2024-01-16T15:15:23.510461+0000","last_undegraded":"2024-01-16T15:15:23.510461+0000","last_fullsized":"2024-01-16T15:15:23.510461+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:19.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516805+0000","last_change":"2024-01-16T15:15:21.528755+0000","last_active":"2024-01-16T15:15:22.516805+0000","last_peered":"2024-01-16T15:15:22.516805+0000","last_clean":"2024-01-16T15:15:22.516805+0000","last_became_active":"2024-01-16T15:15:21.528462+0000","last_became_peered":"2024-01-16T15:15:21.528462+0000","last_unstale":"2024-01-16T15:15:22.516805+0000","last_undegraded":"2024-01-16T15:15:22.516805+0000","last_fullsized":"2024-01-16T15:15:22.516805+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:14.369274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609164+0000","last_change":"2024-01-16T15:15:22.529246+0000","last_active":"2024-01-16T15:15:23.609164+0000","last_peered":"2024-01-16T15:15:23.609164+0000","last_clean":"2024-01-16T15:15:23.609164+0000","last_became_active":"2024-01-16T15:15:22.528966+0000","last_became_peered":"2024-01-16T15:15:22.528966+0000","last_unstale":"2024-01-16T15:15:23.609164+0000","last_undegraded":"2024-01-16T15:15:23.609164+0000","last_fullsized":"2024-01-16T15:15:23.609164+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:46.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690545+0000","last_change":"2024-01-16T15:15:21.525650+0000","last_active":"2024-01-16T15:15:23.690545+0000","last_peered":"2024-01-16T15:15:23.690545+0000","last_clean":"2024-01-16T15:15:23.690545+0000","last_became_active":"2024-01-16T15:15:21.524160+0000","last_became_peered":"2024-01-16T15:15:21.524160+0000","last_unstale":"2024-01-16T15:15:23.690545+0000","last_undegraded":"2024-01-16T15:15:23.690545+0000","last_fullsized":"2024-01-16T15:15:23.690545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:28.946524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007397+0000","last_change":"2024-01-16T15:15:22.532714+0000","last_active":"2024-01-16T15:15:24.007397+0000","last_peered":"2024-01-16T15:15:24.007397+0000","last_clean":"2024-01-16T15:15:24.007397+0000","last_became_active":"2024-01-16T15:15:22.532449+0000","last_became_peered":"2024-01-16T15:15:22.532449+0000","last_unstale":"2024-01-16T15:15:24.007397+0000","last_undegraded":"2024-01-16T15:15:24.007397+0000","last_fullsized":"2024-01-16T15:15:24.007397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:15.384806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.544469+0000","last_change":"2024-01-16T15:15:21.520262+0000","last_active":"2024-01-16T15:15:22.544469+0000","last_peered":"2024-01-16T15:15:22.544469+0000","last_clean":"2024-01-16T15:15:22.544469+0000","last_became_active":"2024-01-16T15:15:21.519869+0000","last_became_peered":"2024-01-16T15:15:21.519869+0000","last_unstale":"2024-01-16T15:15:22.544469+0000","last_undegraded":"2024-01-16T15:15:22.544469+0000","last_fullsized":"2024-01-16T15:15:22.544469+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004344+0000","last_change":"2024-01-16T15:15:22.543578+0000","last_active":"2024-01-16T15:15:24.004344+0000","last_peered":"2024-01-16T15:15:24.004344+0000","last_clean":"2024-01-16T15:15:24.004344+0000","last_became_active":"2024-01-16T15:15:22.543283+0000","last_became_peered":"2024-01-16T15:15:22.543283+0000","last_unstale":"2024-01-16T15:15:24.004344+0000","last_undegraded":"2024-01-16T15:15:24.004344+0000","last_fullsized":"2024-01-16T15:15:24.004344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.549526+0000","last_change":"2024-01-16T15:15:21.528614+0000","last_active":"2024-01-16T15:15:22.549526+0000","last_peered":"2024-01-16T15:15:22.549526+0000","last_clean":"2024-01-16T15:15:22.549526+0000","last_became_active":"2024-01-16T15:15:21.528325+0000","last_became_peered":"2024-01-16T15:15:21.528325+0000","last_unstale":"2024-01-16T15:15:22.549526+0000","last_undegraded":"2024-01-16T15:15:22.549526+0000","last_fullsized":"2024-01-16T15:15:22.549526+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:25.234717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007346+0000","last_change":"2024-01-16T15:15:22.530224+0000","last_active":"2024-01-16T15:15:24.007346+0000","last_peered":"2024-01-16T15:15:24.007346+0000","last_clean":"2024-01-16T15:15:24.007346+0000","last_became_active":"2024-01-16T15:15:22.528654+0000","last_became_peered":"2024-01-16T15:15:22.528654+0000","last_unstale":"2024-01-16T15:15:24.007346+0000","last_undegraded":"2024-01-16T15:15:24.007346+0000","last_fullsized":"2024-01-16T15:15:24.007346+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740462+0000","last_change":"2024-01-16T15:15:21.521658+0000","last_active":"2024-01-16T15:15:23.740462+0000","last_peered":"2024-01-16T15:15:23.740462+0000","last_clean":"2024-01-16T15:15:23.740462+0000","last_became_active":"2024-01-16T15:15:21.521051+0000","last_became_peered":"2024-01-16T15:15:21.521051+0000","last_unstale":"2024-01-16T15:15:23.740462+0000","last_undegraded":"2024-01-16T15:15:23.740462+0000","last_fullsized":"2024-01-16T15:15:23.740462+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609501+0000","last_change":"2024-01-16T15:15:22.525393+0000","last_active":"2024-01-16T15:15:23.609501+0000","last_peered":"2024-01-16T15:15:23.609501+0000","last_clean":"2024-01-16T15:15:23.609501+0000","last_became_active":"2024-01-16T15:15:22.524494+0000","last_became_peered":"2024-01-16T15:15:22.524494+0000","last_unstale":"2024-01-16T15:15:23.609501+0000","last_undegraded":"2024-01-16T15:15:23.609501+0000","last_fullsized":"2024-01-16T15:15:23.609501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:10.852002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740975+0000","last_change":"2024-01-16T15:15:21.531548+0000","last_active":"2024-01-16T15:15:23.740975+0000","last_peered":"2024-01-16T15:15:23.740975+0000","last_clean":"2024-01-16T15:15:23.740975+0000","last_became_active":"2024-01-16T15:15:21.531267+0000","last_became_peered":"2024-01-16T15:15:21.531267+0000","last_unstale":"2024-01-16T15:15:23.740975+0000","last_undegraded":"2024-01-16T15:15:23.740975+0000","last_fullsized":"2024-01-16T15:15:23.740975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:42.610622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609997+0000","last_change":"2024-01-16T15:15:22.522512+0000","last_active":"2024-01-16T15:15:23.609997+0000","last_peered":"2024-01-16T15:15:23.609997+0000","last_clean":"2024-01-16T15:15:23.609997+0000","last_became_active":"2024-01-16T15:15:22.522206+0000","last_became_peered":"2024-01-16T15:15:22.522206+0000","last_unstale":"2024-01-16T15:15:23.609997+0000","last_undegraded":"2024-01-16T15:15:23.609997+0000","last_fullsized":"2024-01-16T15:15:23.609997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:29.962056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004895+0000","last_change":"2024-01-16T15:15:21.524797+0000","last_active":"2024-01-16T15:15:24.004895+0000","last_peered":"2024-01-16T15:15:24.004895+0000","last_clean":"2024-01-16T15:15:24.004895+0000","last_became_active":"2024-01-16T15:15:21.524502+0000","last_became_peered":"2024-01-16T15:15:21.524502+0000","last_unstale":"2024-01-16T15:15:24.004895+0000","last_undegraded":"2024-01-16T15:15:24.004895+0000","last_fullsized":"2024-01-16T15:15:24.004895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:46.648168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609344+0000","last_change":"2024-01-16T15:15:22.523225+0000","last_active":"2024-01-16T15:15:23.609344+0000","last_peered":"2024-01-16T15:15:23.609344+0000","last_clean":"2024-01-16T15:15:23.609344+0000","last_became_active":"2024-01-16T15:15:22.522951+0000","last_became_peered":"2024-01-16T15:15:22.522951+0000","last_unstale":"2024-01-16T15:15:23.609344+0000","last_undegraded":"2024-01-16T15:15:23.609344+0000","last_fullsized":"2024-01-16T15:15:23.609344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:21.818020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004192+0000","last_change":"2024-01-16T15:15:21.519991+0000","last_active":"2024-01-16T15:15:24.004192+0000","last_peered":"2024-01-16T15:15:24.004192+0000","last_clean":"2024-01-16T15:15:24.004192+0000","last_became_active":"2024-01-16T15:15:21.519565+0000","last_became_peered":"2024-01-16T15:15:21.519565+0000","last_unstale":"2024-01-16T15:15:24.004192+0000","last_undegraded":"2024-01-16T15:15:24.004192+0000","last_fullsized":"2024-01-16T15:15:24.004192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:50.723441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006770+0000","last_change":"2024-01-16T15:15:22.535998+0000","last_active":"2024-01-16T15:15:24.006770+0000","last_peered":"2024-01-16T15:15:24.006770+0000","last_clean":"2024-01-16T15:15:24.006770+0000","last_became_active":"2024-01-16T15:15:22.535745+0000","last_became_peered":"2024-01-16T15:15:22.535745+0000","last_unstale":"2024-01-16T15:15:24.006770+0000","last_undegraded":"2024-01-16T15:15:24.006770+0000","last_fullsized":"2024-01-16T15:15:24.006770+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:08.616887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740258+0000","last_change":"2024-01-16T15:15:21.519702+0000","last_active":"2024-01-16T15:15:23.740258+0000","last_peered":"2024-01-16T15:15:23.740258+0000","last_clean":"2024-01-16T15:15:23.740258+0000","last_became_active":"2024-01-16T15:15:21.518926+0000","last_became_peered":"2024-01-16T15:15:21.518926+0000","last_unstale":"2024-01-16T15:15:23.740258+0000","last_undegraded":"2024-01-16T15:15:23.740258+0000","last_fullsized":"2024-01-16T15:15:23.740258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:31.707777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741243+0000","last_change":"2024-01-16T15:15:22.536514+0000","last_active":"2024-01-16T15:15:23.741243+0000","last_peered":"2024-01-16T15:15:23.741243+0000","last_clean":"2024-01-16T15:15:23.741243+0000","last_became_active":"2024-01-16T15:15:22.536248+0000","last_became_peered":"2024-01-16T15:15:22.536248+0000","last_unstale":"2024-01-16T15:15:23.741243+0000","last_undegraded":"2024-01-16T15:15:23.741243+0000","last_fullsized":"2024-01-16T15:15:23.741243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:12:10.024161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005379+0000","last_change":"2024-01-16T15:15:21.522416+0000","last_active":"2024-01-16T15:15:24.005379+0000","last_peered":"2024-01-16T15:15:24.005379+0000","last_clean":"2024-01-16T15:15:24.005379+0000","last_became_active":"2024-01-16T15:15:21.522118+0000","last_became_peered":"2024-01-16T15:15:21.522118+0000","last_unstale":"2024-01-16T15:15:24.005379+0000","last_undegraded":"2024-01-16T15:15:24.005379+0000","last_fullsized":"2024-01-16T15:15:24.005379+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.762652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691386+0000","last_change":"2024-01-16T15:15:22.531377+0000","last_active":"2024-01-16T15:15:23.691386+0000","last_peered":"2024-01-16T15:15:23.691386+0000","last_clean":"2024-01-16T15:15:23.691386+0000","last_became_active":"2024-01-16T15:15:22.529798+0000","last_became_peered":"2024-01-16T15:15:22.529798+0000","last_unstale":"2024-01-16T15:15:23.691386+0000","last_undegraded":"2024-01-16T15:15:23.691386+0000","last_fullsized":"2024-01-16T15:15:23.691386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.721245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741334+0000","last_change":"2024-01-16T15:15:21.532318+0000","last_active":"2024-01-16T15:15:23.741334+0000","last_peered":"2024-01-16T15:15:23.741334+0000","last_clean":"2024-01-16T15:15:23.741334+0000","last_became_active":"2024-01-16T15:15:21.531869+0000","last_became_peered":"2024-01-16T15:15:21.531869+0000","last_unstale":"2024-01-16T15:15:23.741334+0000","last_undegraded":"2024-01-16T15:15:23.741334+0000","last_fullsized":"2024-01-16T15:15:23.741334+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:40.183724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690994+0000","last_change":"2024-01-16T15:15:22.535293+0000","last_active":"2024-01-16T15:15:23.690994+0000","last_peered":"2024-01-16T15:15:23.690994+0000","last_clean":"2024-01-16T15:15:23.690994+0000","last_became_active":"2024-01-16T15:15:22.534929+0000","last_became_peered":"2024-01-16T15:15:22.534929+0000","last_unstale":"2024-01-16T15:15:23.690994+0000","last_undegraded":"2024-01-16T15:15:23.690994+0000","last_fullsized":"2024-01-16T15:15:23.690994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.606169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740931+0000","last_change":"2024-01-16T15:15:21.520890+0000","last_active":"2024-01-16T15:15:23.740931+0000","last_peered":"2024-01-16T15:15:23.740931+0000","last_clean":"2024-01-16T15:15:23.740931+0000","last_became_active":"2024-01-16T15:15:21.520488+0000","last_became_peered":"2024-01-16T15:15:21.520488+0000","last_unstale":"2024-01-16T15:15:23.740931+0000","last_undegraded":"2024-01-16T15:15:23.740931+0000","last_fullsized":"2024-01-16T15:15:23.740931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:08.456302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006786+0000","last_change":"2024-01-16T15:15:22.531273+0000","last_active":"2024-01-16T15:15:24.006786+0000","last_peered":"2024-01-16T15:15:24.006786+0000","last_clean":"2024-01-16T15:15:24.006786+0000","last_became_active":"2024-01-16T15:15:22.530828+0000","last_became_peered":"2024-01-16T15:15:22.530828+0000","last_unstale":"2024-01-16T15:15:24.006786+0000","last_undegraded":"2024-01-16T15:15:24.006786+0000","last_fullsized":"2024-01-16T15:15:24.006786+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:25.470687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.531826+0000","last_change":"2024-01-16T15:15:21.527345+0000","last_active":"2024-01-16T15:15:22.531826+0000","last_peered":"2024-01-16T15:15:22.531826+0000","last_clean":"2024-01-16T15:15:22.531826+0000","last_became_active":"2024-01-16T15:15:21.527018+0000","last_became_peered":"2024-01-16T15:15:21.527018+0000","last_unstale":"2024-01-16T15:15:22.531826+0000","last_undegraded":"2024-01-16T15:15:22.531826+0000","last_fullsized":"2024-01-16T15:15:22.531826+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:13.821755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609781+0000","last_change":"2024-01-16T15:15:22.525600+0000","last_active":"2024-01-16T15:15:23.609781+0000","last_peered":"2024-01-16T15:15:23.609781+0000","last_clean":"2024-01-16T15:15:23.609781+0000","last_became_active":"2024-01-16T15:15:22.525112+0000","last_became_peered":"2024-01-16T15:15:22.525112+0000","last_unstale":"2024-01-16T15:15:23.609781+0000","last_undegraded":"2024-01-16T15:15:23.609781+0000","last_fullsized":"2024-01-16T15:15:23.609781+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:49.965076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741004+0000","last_change":"2024-01-16T15:15:21.526060+0000","last_active":"2024-01-16T15:15:23.741004+0000","last_peered":"2024-01-16T15:15:23.741004+0000","last_clean":"2024-01-16T15:15:23.741004+0000","last_became_active":"2024-01-16T15:15:21.525792+0000","last_became_peered":"2024-01-16T15:15:21.525792+0000","last_unstale":"2024-01-16T15:15:23.741004+0000","last_undegraded":"2024-01-16T15:15:23.741004+0000","last_fullsized":"2024-01-16T15:15:23.741004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511167+0000","last_change":"2024-01-16T15:15:22.541120+0000","last_active":"2024-01-16T15:15:23.511167+0000","last_peered":"2024-01-16T15:15:23.511167+0000","last_clean":"2024-01-16T15:15:23.511167+0000","last_became_active":"2024-01-16T15:15:22.537720+0000","last_became_peered":"2024-01-16T15:15:22.537720+0000","last_unstale":"2024-01-16T15:15:23.511167+0000","last_undegraded":"2024-01-16T15:15:23.511167+0000","last_fullsized":"2024-01-16T15:15:23.511167+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:05.452028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.531464+0000","last_change":"2024-01-16T15:15:21.530401+0000","last_active":"2024-01-16T15:15:22.531464+0000","last_peered":"2024-01-16T15:15:22.531464+0000","last_clean":"2024-01-16T15:15:22.531464+0000","last_became_active":"2024-01-16T15:15:21.530083+0000","last_became_peered":"2024-01-16T15:15:21.530083+0000","last_unstale":"2024-01-16T15:15:22.531464+0000","last_undegraded":"2024-01-16T15:15:22.531464+0000","last_fullsized":"2024-01-16T15:15:22.531464+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:18.693488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609251+0000","last_change":"2024-01-16T15:15:22.530971+0000","last_active":"2024-01-16T15:15:23.609251+0000","last_peered":"2024-01-16T15:15:23.609251+0000","last_clean":"2024-01-16T15:15:23.609251+0000","last_became_active":"2024-01-16T15:15:22.530694+0000","last_became_peered":"2024-01-16T15:15:22.530694+0000","last_unstale":"2024-01-16T15:15:23.609251+0000","last_undegraded":"2024-01-16T15:15:23.609251+0000","last_fullsized":"2024-01-16T15:15:23.609251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:02.926295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508734+0000","last_change":"2024-01-16T15:15:21.524472+0000","last_active":"2024-01-16T15:15:23.508734+0000","last_peered":"2024-01-16T15:15:23.508734+0000","last_clean":"2024-01-16T15:15:23.508734+0000","last_became_active":"2024-01-16T15:15:21.524175+0000","last_became_peered":"2024-01-16T15:15:21.524175+0000","last_unstale":"2024-01-16T15:15:23.508734+0000","last_undegraded":"2024-01-16T15:15:23.508734+0000","last_fullsized":"2024-01-16T15:15:23.508734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:43.337353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007607+0000","last_change":"2024-01-16T15:15:22.533778+0000","last_active":"2024-01-16T15:15:24.007607+0000","last_peered":"2024-01-16T15:15:24.007607+0000","last_clean":"2024-01-16T15:15:24.007607+0000","last_became_active":"2024-01-16T15:15:22.533426+0000","last_became_peered":"2024-01-16T15:15:22.533426+0000","last_unstale":"2024-01-16T15:15:24.007607+0000","last_undegraded":"2024-01-16T15:15:24.007607+0000","last_fullsized":"2024-01-16T15:15:24.007607+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:47.611650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609465+0000","last_change":"2024-01-16T15:15:21.533877+0000","last_active":"2024-01-16T15:15:23.609465+0000","last_peered":"2024-01-16T15:15:23.609465+0000","last_clean":"2024-01-16T15:15:23.609465+0000","last_became_active":"2024-01-16T15:15:21.533489+0000","last_became_peered":"2024-01-16T15:15:21.533489+0000","last_unstale":"2024-01-16T15:15:23.609465+0000","last_undegraded":"2024-01-16T15:15:23.609465+0000","last_fullsized":"2024-01-16T15:15:23.609465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:15.404651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610593+0000","last_change":"2024-01-16T15:15:22.521850+0000","last_active":"2024-01-16T15:15:23.610593+0000","last_peered":"2024-01-16T15:15:23.610593+0000","last_clean":"2024-01-16T15:15:23.610593+0000","last_became_active":"2024-01-16T15:15:22.521581+0000","last_became_peered":"2024-01-16T15:15:22.521581+0000","last_unstale":"2024-01-16T15:15:23.610593+0000","last_undegraded":"2024-01-16T15:15:23.610593+0000","last_fullsized":"2024-01-16T15:15:23.610593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:29.487154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.545734+0000","last_change":"2024-01-16T15:15:21.522571+0000","last_active":"2024-01-16T15:15:22.545734+0000","last_peered":"2024-01-16T15:15:22.545734+0000","last_clean":"2024-01-16T15:15:22.545734+0000","last_became_active":"2024-01-16T15:15:21.522205+0000","last_became_peered":"2024-01-16T15:15:21.522205+0000","last_unstale":"2024-01-16T15:15:22.545734+0000","last_undegraded":"2024-01-16T15:15:22.545734+0000","last_fullsized":"2024-01-16T15:15:22.545734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":241,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516932+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:15:22.516932+0000","last_peered":"2024-01-16T15:15:22.516932+0000","last_clean":"2024-01-16T15:15:22.516932+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:15:22.516932+0000","last_undegraded":"2024-01-16T15:15:22.516932+0000","last_fullsized":"2024-01-16T15:15:22.516932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509630,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68408,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303432,"statfs":{"total":96636764160,"available":96566714368,"internally_reserved":0,"allocated":1224704,"data_stored":526071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3140000000000001,"15min":1.615},"min":{"1min":0.75,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":2.5030000000000001,"5min":8.8070000000000004,"15min":27.853999999999999},"last":1.105},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.663,"15min":1.708},"min":{"1min":0.93000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6399999999999999,"5min":28.603999999999999,"15min":34.939999999999998},"last":1.359}]},{"osd":1,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.292,"15min":1.4830000000000001},"min":{"1min":0.94899999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.528,"5min":8.2650000000000006,"15min":13.859},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.754,"15min":1.651},"min":{"1min":0.85599999999999998,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":2.702,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.3080000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4330000000000001,"15min":1.5860000000000001},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":2.6230000000000002,"5min":9.6679999999999993,"15min":13.050000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.69499999999999995,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":2.6309999999999998,"5min":6.782,"15min":35.011000000000003},"last":0.998}]},{"osd":3,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.486,"15min":1.657},"min":{"1min":0.94799999999999995,"5min":0.439,"15min":0.374},"max":{"1min":1.8720000000000001,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.393},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.7170000000000001},"min":{"1min":0.999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.758,"5min":9.7780000000000005,"15min":14.273999999999999},"last":1.212}]},{"osd":4,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3919999999999999,"15min":1.827},"min":{"1min":0.90900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7090000000000001,"5min":10.952999999999999,"15min":34.972999999999999},"last":0.873},{"interface":"front","average":{"1min":1.331,"5min":1.4319999999999999,"15min":1.798},"min":{"1min":0.85899999999999999,"5min":0.58299999999999996,"15min":0.40400000000000003},"max":{"1min":1.6519999999999999,"5min":5.2300000000000004,"15min":16.003},"last":1.165}]},{"osd":5,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.591,"15min":2.0099999999999998},"min":{"1min":1.1339999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.798,"5min":4.4550000000000001,"15min":23.760999999999999},"last":1.454},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6259999999999999,"15min":1.9059999999999999},"min":{"1min":0.96299999999999997,"5min":0.70699999999999996,"15min":0.52400000000000002},"max":{"1min":3.496,"5min":9.4120000000000008,"15min":23.716999999999999},"last":1.415}]},{"osd":7,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.5069999999999999,"15min":2.0430000000000001},"min":{"1min":0.90700000000000003,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":6.9219999999999997,"15min":34.889000000000003},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.79,"15min":2.1760000000000002},"min":{"1min":1.2470000000000001,"5min":0.65100000000000002,"15min":0.47599999999999998},"max":{"1min":2.0499999999999998,"5min":23.242999999999999,"15min":46.305},"last":1.5}]}]},{"osd":4,"up_from":15,"seq":64424509632,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68420,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303420,"statfs":{"total":96636764160,"available":96566702080,"internally_reserved":0,"allocated":1236992,"data_stored":527155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.226,"15min":1.55},"min":{"1min":0.76900000000000002,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.1399999999999997,"15min":22.757999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.125,"15min":1.2729999999999999},"min":{"1min":0.69799999999999995,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.474,"5min":4.0270000000000001,"15min":14.034000000000001},"last":1.0329999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.413,"15min":1.577},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.851,"5min":17.811,"15min":18.347000000000001},"last":1.248},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3140000000000001,"15min":1.49},"min":{"1min":0.90700000000000003,"5min":0.59799999999999998,"15min":0.498},"max":{"1min":1.5600000000000001,"5min":4.0449999999999999,"15min":18.858000000000001},"last":1.5149999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.544,"15min":1.679},"min":{"1min":0.83799999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.768,"5min":17.651,"15min":17.651},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.327,"5min":1.754,"15min":1.722},"min":{"1min":0.83499999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8180000000000001,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.1020000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3320000000000001,"15min":1.554},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":1.6659999999999999,"5min":3.972,"15min":26.741},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.571,"15min":1.7929999999999999},"min":{"1min":0.86899999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.762,"5min":17.715,"15min":26.701000000000001},"last":1.1639999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5660000000000001,"15min":1.7370000000000001},"min":{"1min":1.0029999999999999,"5min":0.51400000000000001,"15min":0.441},"max":{"1min":1.988,"5min":5.3170000000000002,"15min":18.919},"last":1.292},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5029999999999999,"15min":1.6759999999999999},"min":{"1min":0.95599999999999996,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":2.0270000000000001,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.583}]},{"osd":6,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":2.0739999999999998,"15min":1.9419999999999999},"min":{"1min":0.995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.0750000000000002,"5min":32.649999999999999,"15min":32.649999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5469999999999999,"15min":1.823},"min":{"1min":1.1799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0630000000000002,"5min":4.3129999999999997,"15min":24.094999999999999},"last":1.4159999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.6890000000000001,"15min":1.7649999999999999},"min":{"1min":1.056,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":1.8899999999999999,"5min":17.446000000000002,"15min":26.683},"last":1.369},{"interface":"front","average":{"1min":1.52,"5min":1.871,"15min":1.9650000000000001},"min":{"1min":1.248,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9350000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":1.458}]}]},{"osd":7,"up_from":14,"seq":60129542334,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68988,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302852,"statfs":{"total":96636764160,"available":96566120448,"internally_reserved":0,"allocated":1818624,"data_stored":1115899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2649999999999999,"15min":1.5089999999999999},"min":{"1min":0.68100000000000005,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.7310000000000001,"5min":4.3680000000000003,"15min":15.661},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.3240000000000001,"15min":1.5449999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":2.1080000000000001,"5min":12.461,"15min":30.126000000000001},"last":1.226}]},{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.423,"15min":1.4530000000000001},"min":{"1min":0.63200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.085,"5min":20.327000000000002,"15min":24.338000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.395,"15min":1.6180000000000001},"min":{"1min":0.81399999999999995,"5min":0.77400000000000002,"15min":0.46200000000000002},"max":{"1min":1.6419999999999999,"5min":12.526,"15min":21.298999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.57199999999999995,"15min":0.51500000000000001},"max":{"1min":1.9219999999999999,"5min":4.3380000000000001,"15min":12.474},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.702,"15min":1.651},"min":{"1min":0.69299999999999995,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8999999999999999,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4039999999999999,"15min":1.6020000000000001},"min":{"1min":0.88800000000000001,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.7150000000000001,"5min":4.2530000000000001,"15min":12.58},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.9430000000000001,"15min":1.7869999999999999},"min":{"1min":0.78600000000000003,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":2.4630000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3520000000000001,"15min":1.591},"min":{"1min":0.99099999999999999,"5min":0.54100000000000004,"15min":0.42799999999999999},"max":{"1min":2.0219999999999998,"5min":4.1390000000000002,"15min":14.659000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.831,"15min":1.825},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":2.028,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.3600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.486,"15min":1.8360000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.6459999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.113},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.8400000000000001},"min":{"1min":1.1870000000000001,"5min":0.66000000000000003,"15min":0.41099999999999998},"max":{"1min":1.8400000000000001,"5min":7.4649999999999999,"15min":21.312000000000001},"last":1.383}]},{"osd":6,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.9790000000000001,"15min":1.8640000000000001},"min":{"1min":0.97199999999999998,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":2.0880000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.597,"15min":1.956},"min":{"1min":0.81699999999999995,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":1.8300000000000001,"5min":12.567,"15min":29.991},"last":1.3300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542335,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76728,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295112,"statfs":{"total":96636764160,"available":96558194688,"internally_reserved":0,"allocated":1220608,"data_stored":525489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.405,"15min":1.5389999999999999},"min":{"1min":0.79700000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.6579999999999999,"5min":6.8380000000000001,"15min":19.382999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.3740000000000001,"15min":1.3899999999999999},"min":{"1min":0.77000000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":13.127000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":1.008}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.5409999999999999,"15min":1.6339999999999999},"min":{"1min":0.70699999999999996,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":3.8109999999999999,"5min":13.571,"15min":17.693999999999999},"last":1.214},{"interface":"front","average":{"1min":1.996,"5min":1.6419999999999999,"15min":1.6950000000000001},"min":{"1min":0.95099999999999996,"5min":0.51600000000000001,"15min":0.432},"max":{"1min":17.062000000000001,"5min":17.062000000000001,"15min":20.645},"last":1.395}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.5740000000000001,"15min":1.6879999999999999},"min":{"1min":0.79100000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.77,"5min":6.4039999999999999,"15min":21.138000000000002},"last":1.7190000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.462,"15min":1.4910000000000001},"min":{"1min":0.995,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":4.0720000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.5349999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4930000000000001,"15min":1.8},"min":{"1min":0.95499999999999996,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":2.4769999999999999,"5min":7.984,"15min":23.109000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.6060000000000001,"15min":1.7949999999999999},"min":{"1min":1.0169999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.95,"5min":6.4610000000000003,"15min":19.303000000000001},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5449999999999999,"15min":1.849},"min":{"1min":0.85799999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":9.3309999999999995,"5min":10.831,"15min":24.436},"last":1.617},{"interface":"front","average":{"1min":1.329,"5min":1.48,"15min":1.744},"min":{"1min":0.84699999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":1.6140000000000001,"5min":5.6820000000000004,"15min":19.126000000000001},"last":1.5800000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.526,"15min":1.635},"min":{"1min":0.95599999999999996,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":1.718,"5min":12.914,"15min":21.192},"last":1.341},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5609999999999999,"15min":1.978},"min":{"1min":0.97299999999999998,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":2.0299999999999998,"5min":7.8609999999999998,"15min":27.042000000000002},"last":1.764}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7989999999999999,"15min":1.9350000000000001},"min":{"1min":0.92600000000000005,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":13.034000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":1.663},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.7410000000000001,"15min":2.0840000000000001},"min":{"1min":1.216,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":2.1429999999999998,"5min":10.756,"15min":24.367000000000001},"last":1.8440000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542335,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":527053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.5,"15min":1.278},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":1.403,"5min":28.437999999999999,"15min":31.562999999999999},"last":1.238},{"interface":"front","average":{"1min":1.117,"5min":1.26,"15min":1.173},"min":{"1min":0.65900000000000003,"5min":0.40600000000000003,"15min":0.39600000000000002},"max":{"1min":1.8440000000000001,"5min":11.361000000000001,"15min":11.361000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.236,"15min":1.145},"min":{"1min":0.59999999999999998,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.5940000000000001,"5min":10.856999999999999,"15min":15.569000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.7050000000000001,"15min":1.429},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.643,"5min":28.352,"15min":28.352},"last":1.103}]},{"osd":2,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":2.379,"15min":1.6879999999999999},"min":{"1min":0.71999999999999997,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":1.8109999999999999,"5min":39.265999999999998,"15min":39.265999999999998},"last":1.351},{"interface":"front","average":{"1min":1.264,"5min":1.4970000000000001,"15min":1.385},"min":{"1min":0.67800000000000005,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":3.911,"5min":13.526,"15min":14.577999999999999},"last":1.1679999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.516,"15min":1.5620000000000001},"min":{"1min":0.86499999999999999,"5min":0.64900000000000002,"15min":0.53800000000000003},"max":{"1min":2.1019999999999999,"5min":6.0629999999999997,"15min":20.978999999999999},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.448,"5min":2.2879999999999998,"15min":1.8480000000000001},"min":{"1min":1.034,"5min":0.93100000000000005,"15min":0.72999999999999998},"max":{"1min":2.0030000000000001,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.472}]},{"osd":5,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.746,"15min":1.766},"min":{"1min":0.81699999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":1.911,"5min":10.988,"15min":20.663},"last":1.512},{"interface":"front","average":{"1min":1.575,"5min":2.3090000000000002,"15min":1.9490000000000001},"min":{"1min":0.96299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1019999999999999,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.637}]},{"osd":6,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.77,"15min":1.754},"min":{"1min":1.0069999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.2170000000000001,"5min":13.691000000000001,"15min":13.691000000000001},"last":1.321},{"interface":"front","average":{"1min":2.0910000000000002,"5min":2.0819999999999999,"15min":1.9510000000000001},"min":{"1min":0.95099999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":15.114000000000001,"5min":26.698,"15min":26.698},"last":1.6000000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.905,"15min":1.776},"min":{"1min":1.0589999999999999,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.9630000000000001,"5min":13.622,"15min":13.622},"last":1.431},{"interface":"front","average":{"1min":1.5289999999999999,"5min":2.0459999999999998,"15min":2.1659999999999999},"min":{"1min":0.90400000000000003,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.2789999999999999,"5min":15.372999999999999,"15min":49.811},"last":1.671}]}]},{"osd":2,"up_from":14,"seq":60129542334,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76732,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295108,"statfs":{"total":96636764160,"available":96558190592,"internally_reserved":0,"allocated":1220608,"data_stored":525453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.284,"15min":1.0449999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.7130000000000001,"5min":18.289999999999999,"15min":18.289999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.2210000000000001,"15min":1.099},"min":{"1min":0.65100000000000002,"5min":0.40699999999999997,"15min":0.28599999999999998},"max":{"1min":1.6000000000000001,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":0.80100000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.4650000000000001,"15min":1.1850000000000001},"min":{"1min":0.55900000000000005,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":1.8979999999999999,"5min":23.135999999999999,"15min":23.135999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.278,"15min":1.2050000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.6699999999999999,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":1.016}]},{"osd":3,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3859999999999999,"15min":1.1839999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":7.306,"5min":18.838000000000001,"15min":18.838000000000001},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.421,"5min":1.647,"15min":1.4510000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.839,"5min":19.375,"15min":19.375},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.8140000000000001,"15min":1.5620000000000001},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.59999999999999998},"max":{"1min":2.0249999999999999,"5min":23.225000000000001,"15min":23.225000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.651,"15min":1.5600000000000001},"min":{"1min":1.036,"5min":0.72699999999999998,"15min":0.59199999999999997},"max":{"1min":1.7509999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.335}]},{"osd":5,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.8580000000000001,"15min":1.6200000000000001},"min":{"1min":1.0740000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":7.7640000000000002,"5min":20.157,"15min":20.157},"last":1.2},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8089999999999999,"15min":1.601},"min":{"1min":0.84299999999999997,"5min":0.623,"15min":0.623},"max":{"1min":7.9820000000000002,"5min":25.004000000000001,"15min":25.004000000000001},"last":1.7090000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":2.1040000000000001,"15min":1.7390000000000001},"min":{"1min":0.89400000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":7.6630000000000003,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.948,"15min":1.839},"min":{"1min":0.81100000000000005,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":7.8680000000000003,"5min":28.760999999999999,"15min":28.760999999999999},"last":1.49}]},{"osd":7,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":2.2919999999999998,"15min":1.772},"min":{"1min":1.05,"5min":0.82099999999999995,"15min":0.64100000000000001},"max":{"1min":3.1219999999999999,"5min":22.834,"15min":22.834},"last":1.554},{"interface":"front","average":{"1min":1.53,"5min":2.2879999999999998,"15min":2.069},"min":{"1min":0.91000000000000003,"5min":0.872,"15min":0.82799999999999996},"max":{"1min":1.931,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.4279999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542336,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72560,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299280,"statfs":{"total":96636764160,"available":96562462720,"internally_reserved":0,"allocated":1216512,"data_stored":525419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.9850000000000001,"15min":1.3260000000000001},"min":{"1min":0.48599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.256,"5min":31.178999999999998,"15min":31.178999999999998},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.4470000000000001,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.901,"5min":26.859999999999999,"15min":26.859999999999999},"last":0.67200000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1439999999999999,"15min":1.143},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":3.5830000000000002,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.9239999999999999,"15min":1.5529999999999999},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":3.4929999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":0.84599999999999997}]},{"osd":3,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.879,"15min":1.651},"min":{"1min":0.56299999999999994,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.8049999999999999,"5min":27.609999999999999,"15min":27.609999999999999},"last":0.89600000000000002},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.1709999999999998,"15min":1.6559999999999999},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":2.9750000000000001,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.3069999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4390000000000001,"15min":1.466},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.58099999999999996},"max":{"1min":2.3069999999999999,"5min":4.7249999999999996,"15min":10.577999999999999},"last":1.46},{"interface":"front","average":{"1min":1.411,"5min":2.0760000000000001,"15min":1.74},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.0310000000000001,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.2170000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.411,"15min":1.5449999999999999},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.032,"5min":3.032,"15min":17.469000000000001},"last":1.508},{"interface":"front","average":{"1min":1.377,"5min":1.522,"15min":1.6060000000000001},"min":{"1min":1.0229999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1859999999999999,"5min":4.6120000000000001,"15min":8.2989999999999995},"last":1.077}]},{"osd":6,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5569999999999999,"15min":1.7150000000000001},"min":{"1min":0.85999999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.9009999999999998,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.611},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5549999999999999,"15min":1.873},"min":{"1min":1.0800000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5760000000000001,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.2669999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6419999999999999,"15min":1.899},"min":{"1min":0.94799999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4249999999999998,"5min":4.6989999999999998,"15min":30.834},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.2589999999999999,"15min":2.0609999999999999},"min":{"1min":0.96799999999999997,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":1.9890000000000001,"5min":27.716999999999999,"15min":27.716999999999999},"last":1.4099999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542334,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73152,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298688,"statfs":{"total":96636764160,"available":96561856512,"internally_reserved":0,"allocated":1814528,"data_stored":1116369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.103,"15min":1.1879999999999999},"min":{"1min":0.52100000000000002,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.73,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.353,"15min":1.4970000000000001},"min":{"1min":0.82299999999999995,"5min":0.42499999999999999,"15min":0.33900000000000002},"max":{"1min":1.623,"5min":26.459,"15min":50.429000000000002},"last":0.93500000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":0.66100000000000003,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.8009999999999999,"5min":20.041,"15min":22.716999999999999},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.452,"15min":1.6160000000000001},"min":{"1min":0.78400000000000003,"5min":0.503,"15min":0.38},"max":{"1min":1.7430000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":1.252}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2410000000000001,"15min":1.218},"min":{"1min":0.749,"5min":0.40000000000000002,"15min":0.372},"max":{"1min":1.8129999999999999,"5min":5.8719999999999999,"15min":10.795},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.425,"15min":1.5820000000000001},"min":{"1min":0.70799999999999996,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.8280000000000001,"5min":9.484,"15min":22.254999999999999},"last":1.462}]},{"osd":4,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5,"15min":1.806},"min":{"1min":0.78800000000000003,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.986,"5min":3.7749999999999999,"15min":22.407},"last":1.536},{"interface":"front","average":{"1min":1.48,"5min":1.6930000000000001,"15min":1.919},"min":{"1min":1.1970000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.907,"5min":26.57,"15min":50.497},"last":1.365}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.401,"15min":1.6220000000000001},"min":{"1min":0.83199999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.0550000000000002,"5min":4.7359999999999998,"15min":14.112},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.7,"15min":1.847},"min":{"1min":1.05,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.9430000000000001,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.5920000000000001,"15min":1.7769999999999999},"min":{"1min":0.871,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.968,"5min":8.2349999999999994,"15min":31.387},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.782,"5min":1.841,"15min":1.9650000000000001},"min":{"1min":1.01,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":3.5209999999999999,"5min":9.2840000000000007,"15min":23.547000000000001},"last":1.659}]},{"osd":7,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":2.0190000000000001,"15min":2.0680000000000001},"min":{"1min":1.292,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.21,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.679},{"interface":"front","average":{"1min":1.639,"5min":1.8660000000000001,"15min":1.835},"min":{"1min":1.0349999999999999,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.1859999999999999,"5min":12.455,"15min":15.747},"last":1.5760000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:32.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:15:32.344 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.361+0000 7f4f0de00640 1 Processor -- start 2024-01-16T15:15:32.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.361+0000 7f4f0de00640 1 -- start start 2024-01-16T15:15:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.362+0000 7f4f0de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 0x7f4f080b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.362+0000 7f4f0de00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0806cea0 con 0x7f4f080cd240 2024-01-16T15:15:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f0de00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0806cbc0 con 0x7f4f0809a170 2024-01-16T15:15:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f0de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0806f350 con 0x7f4f0806c6b0 2024-01-16T15:15:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f06a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4f080cd240 0x7f4f08099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54580/0 (socket says 172.21.15.53:54580) 2024-01-16T15:15:32.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f06a00640 1 -- 172.21.15.53:0/1450318609 learned_addr learned my addr 172.21.15.53:0/1450318609 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f07400640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 0x7f4f080b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051544740 0 0) 0x7f4f0806cea0 con 0x7f4f080cd240 2024-01-16T15:15:32.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.363+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ee8003520 con 0x7f4f080cd240 2024-01-16T15:15:32.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755484805 0 0) 0x7f4ee8003520 con 0x7f4f080cd240 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 msgr2=0x7f4f080b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 0x7f4f080b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 >> v1:172.21.15.146:6789/0 conn(0x7f4f0809a170 legacy=0x7f4f080cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0806f630 con 0x7f4f080cd240 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4ef0001670 con 0x7f4f080cd240 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 >> v1:172.21.15.53:6789/0 conn(0x7f4f080cd240 legacy=0x7f4f08099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ee80039a0 0x7f4ee8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4ee8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 0x7f4ee8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0806f350 con 0x7f4ee80047c0 2024-01-16T15:15:32.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee800ba60 con 0x7f4ee80039a0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee800bc20 con 0x7f4f0806c6b0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.364+0000 7f4f06000640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4ee8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f07400640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 0x7f4ee8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06a00640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ee80039a0 0x7f4ee8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06000640 1 -- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 msgr2=0x7f4ee8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06000640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f0806c6b0 0x7f4ee8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06000640 1 -- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ee80039a0 msgr2=0x7f4ee8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06000640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ee80039a0 0x7f4ee8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06000640 1 -- 172.21.15.53:0/1450318609 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8003520 con 0x7f4ee80047c0 2024-01-16T15:15:32.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.365+0000 7f4f06a00640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ee80039a0 0x7f4ee8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.366+0000 7f4f06000640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4ee8004c40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8001e20 tx=0x7f4ee80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405ccda2180e9f72 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8003b70 con 0x7f4ee80047c0 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef8003cd0 con 0x7f4ee80047c0 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f05600640 1 -- 172.21.15.53:0/1450318609 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8005310 con 0x7f4ee80047c0 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f0de00640 1 -- 172.21.15.53:0/1450318609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 msgr2=0x7f4ee8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f0de00640 1 --2- 172.21.15.53:0/1450318609 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4ee8004c40 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8001e20 tx=0x7f4ee80057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f07400640 1 -- 172.21.15.53:0/1450318609 reap_dead start 2024-01-16T15:15:32.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f0de00640 1 -- 172.21.15.53:0/1450318609 shutdown_connections 2024-01-16T15:15:32.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f0de00640 1 -- 172.21.15.53:0/1450318609 >> 172.21.15.53:0/1450318609 conn(0x7f4f080b2d00 msgr2=0x7f4f080b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:32.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f0de00640 1 -- 172.21.15.53:0/1450318609 shutdown_connections 2024-01-16T15:15:32.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.367+0000 7f4f0de00640 1 -- 172.21.15.53:0/1450318609 wait complete. 2024-01-16T15:15:32.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.369+0000 7f4f0de00640 1 Processor -- start 2024-01-16T15:15:32.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.369+0000 7f4f0de00640 1 -- start start 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.370+0000 7f4f0de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4f080b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.370+0000 7f4f0de00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0815a620 0x7f4f080b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.370+0000 7f4f0de00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f080b4ab0 0x7f4f080b4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.370+0000 7f4f0de00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0815aec0 con 0x7f4ee80047c0 2024-01-16T15:15:32.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.370+0000 7f4f0de00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080b4060 con 0x7f4f0815a620 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.370+0000 7f4f0de00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080b4620 con 0x7f4f080b4ab0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4f080b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4f080b5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39512/0 (socket says 172.21.15.53:39512) 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 -- 172.21.15.53:0/2706737589 learned_addr learned my addr 172.21.15.53:0/2706737589 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f07400640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f080b4ab0 0x7f4f080b4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06000640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0815a620 0x7f4f080b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 -- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f080b4ab0 msgr2=0x7f4f080b4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f080b4ab0 0x7f4f080b4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 -- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0815a620 msgr2=0x7f4f080b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0815a620 0x7f4f080b1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 -- 172.21.15.53:0/2706737589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080b6510 con 0x7f4ee80047c0 2024-01-16T15:15:32.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.371+0000 7f4f06a00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4f080b5e50 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000d810 tx=0x7f4ef000dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.372+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef00074e0 con 0x7f4ee80047c0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.372+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef0007640 con 0x7f4ee80047c0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.372+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef0004a00 con 0x7f4ee80047c0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.372+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f080b67f0 con 0x7f4ee80047c0 2024-01-16T15:15:32.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.372+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f080b1f40 con 0x7f4ee80047c0 2024-01-16T15:15:32.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.373+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f080b2160 con 0x7f4ee80047c0 2024-01-16T15:15:32.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.374+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4ef0004b60 con 0x7f4ee80047c0 2024-01-16T15:15:32.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.375+0000 7f4eff400640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4ee006c160 0x7f4ee006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.375+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4ef00883a0 con 0x7f4ee80047c0 2024-01-16T15:15:32.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.376+0000 7f4f06000640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4ee006c160 0x7f4ee006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.377+0000 7f4f06000640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4ee006c160 0x7f4ee006e620 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8034000 tx=0x7f4ef8009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.379+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ef0054980 con 0x7f4ee80047c0 2024-01-16T15:15:32.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.574+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f080cc8e0 con 0x7f4ee006c160 2024-01-16T15:15:32.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.597+0000 7f4eff400640 1 -- 172.21.15.53:0/2706737589 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286900 (secure 0 0 0) 0x7f4f080cc8e0 con 0x7f4ee006c160 2024-01-16T15:15:32.585 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:32.587 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:15:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4ee006c160 msgr2=0x7f4ee006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4ee006c160 0x7f4ee006e620 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8034000 tx=0x7f4ef8009030 comp rx=0 tx=0).stop 2024-01-16T15:15:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 msgr2=0x7f4f080b5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4f080b5e50 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f4ef000d810 tx=0x7f4ef000dcd0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 shutdown_connections 2024-01-16T15:15:32.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4ee006c160 0x7f4ee006e620 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f080b4ab0 0x7f4f080b4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f0815a620 0x7f4f080b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 --2- 172.21.15.53:0/2706737589 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ee80047c0 0x7f4f080b5e50 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.603+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 >> 172.21.15.53:0/2706737589 conn(0x7f4f080b2d00 msgr2=0x7f4f080b0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:32.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.604+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 shutdown_connections 2024-01-16T15:15:32.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.604+0000 7f4f0de00640 1 -- 172.21.15.53:0/2706737589 wait complete. 2024-01-16T15:15:32.608 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":565,"stamp":"2024-01-16T15:15:30.581371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":581724,"kb_used_data":10728,"kb_used_omap":12,"kb_used_meta":570867,"kb_avail":754392996,"statfs":{"total":773094113280,"available":772498427904,"internally_reserved":0,"allocated":10985472,"data_stored":5388908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":584568368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.076948"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691428+0000","last_change":"2024-01-16T15:15:22.537519+0000","last_active":"2024-01-16T15:15:23.691428+0000","last_peered":"2024-01-16T15:15:23.691428+0000","last_clean":"2024-01-16T15:15:23.691428+0000","last_became_active":"2024-01-16T15:15:22.535630+0000","last_became_peered":"2024-01-16T15:15:22.535630+0000","last_unstale":"2024-01-16T15:15:23.691428+0000","last_undegraded":"2024-01-16T15:15:23.691428+0000","last_fullsized":"2024-01-16T15:15:23.691428+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:53.424772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740792+0000","last_change":"2024-01-16T15:15:22.532497+0000","last_active":"2024-01-16T15:15:23.740792+0000","last_peered":"2024-01-16T15:15:23.740792+0000","last_clean":"2024-01-16T15:15:23.740792+0000","last_became_active":"2024-01-16T15:15:22.531043+0000","last_became_peered":"2024-01-16T15:15:22.531043+0000","last_unstale":"2024-01-16T15:15:23.740792+0000","last_undegraded":"2024-01-16T15:15:23.740792+0000","last_fullsized":"2024-01-16T15:15:23.740792+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691119+0000","last_change":"2024-01-16T15:15:22.535023+0000","last_active":"2024-01-16T15:15:23.691119+0000","last_peered":"2024-01-16T15:15:23.691119+0000","last_clean":"2024-01-16T15:15:23.691119+0000","last_became_active":"2024-01-16T15:15:22.534684+0000","last_became_peered":"2024-01-16T15:15:22.534684+0000","last_unstale":"2024-01-16T15:15:23.691119+0000","last_undegraded":"2024-01-16T15:15:23.691119+0000","last_fullsized":"2024-01-16T15:15:23.691119+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:46:00.780930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509733+0000","last_change":"2024-01-16T15:15:22.524150+0000","last_active":"2024-01-16T15:15:23.509733+0000","last_peered":"2024-01-16T15:15:23.509733+0000","last_clean":"2024-01-16T15:15:23.509733+0000","last_became_active":"2024-01-16T15:15:22.522946+0000","last_became_peered":"2024-01-16T15:15:22.522946+0000","last_unstale":"2024-01-16T15:15:23.509733+0000","last_undegraded":"2024-01-16T15:15:23.509733+0000","last_fullsized":"2024-01-16T15:15:23.509733+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:26:13.223095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006320+0000","last_change":"2024-01-16T15:15:22.545386+0000","last_active":"2024-01-16T15:15:24.006320+0000","last_peered":"2024-01-16T15:15:24.006320+0000","last_clean":"2024-01-16T15:15:24.006320+0000","last_became_active":"2024-01-16T15:15:22.544057+0000","last_became_peered":"2024-01-16T15:15:22.544057+0000","last_unstale":"2024-01-16T15:15:24.006320+0000","last_undegraded":"2024-01-16T15:15:24.006320+0000","last_fullsized":"2024-01-16T15:15:24.006320+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:30:46.298808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005648+0000","last_change":"2024-01-16T15:15:22.536600+0000","last_active":"2024-01-16T15:15:24.005648+0000","last_peered":"2024-01-16T15:15:24.005648+0000","last_clean":"2024-01-16T15:15:24.005648+0000","last_became_active":"2024-01-16T15:15:22.536327+0000","last_became_peered":"2024-01-16T15:15:22.536327+0000","last_unstale":"2024-01-16T15:15:24.005648+0000","last_undegraded":"2024-01-16T15:15:24.005648+0000","last_fullsized":"2024-01-16T15:15:24.005648+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005398+0000","last_change":"2024-01-16T15:15:22.535575+0000","last_active":"2024-01-16T15:15:24.005398+0000","last_peered":"2024-01-16T15:15:24.005398+0000","last_clean":"2024-01-16T15:15:24.005398+0000","last_became_active":"2024-01-16T15:15:22.535292+0000","last_became_peered":"2024-01-16T15:15:22.535292+0000","last_unstale":"2024-01-16T15:15:24.005398+0000","last_undegraded":"2024-01-16T15:15:24.005398+0000","last_fullsized":"2024-01-16T15:15:24.005398+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:04:06.443796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740110+0000","last_change":"2024-01-16T15:15:22.521459+0000","last_active":"2024-01-16T15:15:23.740110+0000","last_peered":"2024-01-16T15:15:23.740110+0000","last_clean":"2024-01-16T15:15:23.740110+0000","last_became_active":"2024-01-16T15:15:22.520884+0000","last_became_peered":"2024-01-16T15:15:22.520884+0000","last_unstale":"2024-01-16T15:15:23.740110+0000","last_undegraded":"2024-01-16T15:15:23.740110+0000","last_fullsized":"2024-01-16T15:15:23.740110+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:00:15.239110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511641+0000","last_change":"2024-01-16T15:15:22.526500+0000","last_active":"2024-01-16T15:15:23.511641+0000","last_peered":"2024-01-16T15:15:23.511641+0000","last_clean":"2024-01-16T15:15:23.511641+0000","last_became_active":"2024-01-16T15:15:22.526166+0000","last_became_peered":"2024-01-16T15:15:22.526166+0000","last_unstale":"2024-01-16T15:15:23.511641+0000","last_undegraded":"2024-01-16T15:15:23.511641+0000","last_fullsized":"2024-01-16T15:15:23.511641+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:49:06.510364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007360+0000","last_change":"2024-01-16T15:15:22.542938+0000","last_active":"2024-01-16T15:15:24.007360+0000","last_peered":"2024-01-16T15:15:24.007360+0000","last_clean":"2024-01-16T15:15:24.007360+0000","last_became_active":"2024-01-16T15:15:22.542687+0000","last_became_peered":"2024-01-16T15:15:22.542687+0000","last_unstale":"2024-01-16T15:15:24.007360+0000","last_undegraded":"2024-01-16T15:15:24.007360+0000","last_fullsized":"2024-01-16T15:15:24.007360+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:30:12.904233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005800+0000","last_change":"2024-01-16T15:15:22.533081+0000","last_active":"2024-01-16T15:15:24.005800+0000","last_peered":"2024-01-16T15:15:24.005800+0000","last_clean":"2024-01-16T15:15:24.005800+0000","last_became_active":"2024-01-16T15:15:22.529012+0000","last_became_peered":"2024-01-16T15:15:22.529012+0000","last_unstale":"2024-01-16T15:15:24.005800+0000","last_undegraded":"2024-01-16T15:15:24.005800+0000","last_fullsized":"2024-01-16T15:15:24.005800+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511625+0000","last_change":"2024-01-16T15:15:22.522860+0000","last_active":"2024-01-16T15:15:23.511625+0000","last_peered":"2024-01-16T15:15:23.511625+0000","last_clean":"2024-01-16T15:15:23.511625+0000","last_became_active":"2024-01-16T15:15:22.521459+0000","last_became_peered":"2024-01-16T15:15:22.521459+0000","last_unstale":"2024-01-16T15:15:23.511625+0000","last_undegraded":"2024-01-16T15:15:23.511625+0000","last_fullsized":"2024-01-16T15:15:23.511625+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509163+0000","last_change":"2024-01-16T15:15:22.532050+0000","last_active":"2024-01-16T15:15:23.509163+0000","last_peered":"2024-01-16T15:15:23.509163+0000","last_clean":"2024-01-16T15:15:23.509163+0000","last_became_active":"2024-01-16T15:15:22.531307+0000","last_became_peered":"2024-01-16T15:15:22.531307+0000","last_unstale":"2024-01-16T15:15:23.509163+0000","last_undegraded":"2024-01-16T15:15:23.509163+0000","last_fullsized":"2024-01-16T15:15:23.509163+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:42:34.860510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609381+0000","last_change":"2024-01-16T15:15:22.533322+0000","last_active":"2024-01-16T15:15:23.609381+0000","last_peered":"2024-01-16T15:15:23.609381+0000","last_clean":"2024-01-16T15:15:23.609381+0000","last_became_active":"2024-01-16T15:15:22.532960+0000","last_became_peered":"2024-01-16T15:15:22.532960+0000","last_unstale":"2024-01-16T15:15:23.609381+0000","last_undegraded":"2024-01-16T15:15:23.609381+0000","last_fullsized":"2024-01-16T15:15:23.609381+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:46:47.560189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508639+0000","last_change":"2024-01-16T15:15:22.532776+0000","last_active":"2024-01-16T15:15:23.508639+0000","last_peered":"2024-01-16T15:15:23.508639+0000","last_clean":"2024-01-16T15:15:23.508639+0000","last_became_active":"2024-01-16T15:15:22.532494+0000","last_became_peered":"2024-01-16T15:15:22.532494+0000","last_unstale":"2024-01-16T15:15:23.508639+0000","last_undegraded":"2024-01-16T15:15:23.508639+0000","last_fullsized":"2024-01-16T15:15:23.508639+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005108+0000","last_change":"2024-01-16T15:15:22.530744+0000","last_active":"2024-01-16T15:15:24.005108+0000","last_peered":"2024-01-16T15:15:24.005108+0000","last_clean":"2024-01-16T15:15:24.005108+0000","last_became_active":"2024-01-16T15:15:22.525764+0000","last_became_peered":"2024-01-16T15:15:22.525764+0000","last_unstale":"2024-01-16T15:15:24.005108+0000","last_undegraded":"2024-01-16T15:15:24.005108+0000","last_fullsized":"2024-01-16T15:15:24.005108+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:51.206294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741415+0000","last_change":"2024-01-16T15:15:22.523512+0000","last_active":"2024-01-16T15:15:23.741415+0000","last_peered":"2024-01-16T15:15:23.741415+0000","last_clean":"2024-01-16T15:15:23.741415+0000","last_became_active":"2024-01-16T15:15:22.521121+0000","last_became_peered":"2024-01-16T15:15:22.521121+0000","last_unstale":"2024-01-16T15:15:23.741415+0000","last_undegraded":"2024-01-16T15:15:23.741415+0000","last_fullsized":"2024-01-16T15:15:23.741415+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:20:35.046638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509378+0000","last_change":"2024-01-16T15:15:22.531166+0000","last_active":"2024-01-16T15:15:23.509378+0000","last_peered":"2024-01-16T15:15:23.509378+0000","last_clean":"2024-01-16T15:15:23.509378+0000","last_became_active":"2024-01-16T15:15:22.530595+0000","last_became_peered":"2024-01-16T15:15:22.530595+0000","last_unstale":"2024-01-16T15:15:23.509378+0000","last_undegraded":"2024-01-16T15:15:23.509378+0000","last_fullsized":"2024-01-16T15:15:23.509378+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:01.070526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610445+0000","last_change":"2024-01-16T15:15:22.524237+0000","last_active":"2024-01-16T15:15:23.610445+0000","last_peered":"2024-01-16T15:15:23.610445+0000","last_clean":"2024-01-16T15:15:23.610445+0000","last_became_active":"2024-01-16T15:15:22.523982+0000","last_became_peered":"2024-01-16T15:15:22.523982+0000","last_unstale":"2024-01-16T15:15:23.610445+0000","last_undegraded":"2024-01-16T15:15:23.610445+0000","last_fullsized":"2024-01-16T15:15:23.610445+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:34.668807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511753+0000","last_change":"2024-01-16T15:15:22.523069+0000","last_active":"2024-01-16T15:15:23.511753+0000","last_peered":"2024-01-16T15:15:23.511753+0000","last_clean":"2024-01-16T15:15:23.511753+0000","last_became_active":"2024-01-16T15:15:22.522232+0000","last_became_peered":"2024-01-16T15:15:22.522232+0000","last_unstale":"2024-01-16T15:15:23.511753+0000","last_undegraded":"2024-01-16T15:15:23.511753+0000","last_fullsized":"2024-01-16T15:15:23.511753+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:48:23.850204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609784+0000","last_change":"2024-01-16T15:15:22.529446+0000","last_active":"2024-01-16T15:15:23.609784+0000","last_peered":"2024-01-16T15:15:23.609784+0000","last_clean":"2024-01-16T15:15:23.609784+0000","last_became_active":"2024-01-16T15:15:22.529124+0000","last_became_peered":"2024-01-16T15:15:22.529124+0000","last_unstale":"2024-01-16T15:15:23.609784+0000","last_undegraded":"2024-01-16T15:15:23.609784+0000","last_fullsized":"2024-01-16T15:15:23.609784+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:14.899561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740836+0000","last_change":"2024-01-16T15:15:22.541357+0000","last_active":"2024-01-16T15:15:23.740836+0000","last_peered":"2024-01-16T15:15:23.740836+0000","last_clean":"2024-01-16T15:15:23.740836+0000","last_became_active":"2024-01-16T15:15:22.539470+0000","last_became_peered":"2024-01-16T15:15:22.539470+0000","last_unstale":"2024-01-16T15:15:23.740836+0000","last_undegraded":"2024-01-16T15:15:23.740836+0000","last_fullsized":"2024-01-16T15:15:23.740836+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740324+0000","last_change":"2024-01-16T15:15:22.528944+0000","last_active":"2024-01-16T15:15:23.740324+0000","last_peered":"2024-01-16T15:15:23.740324+0000","last_clean":"2024-01-16T15:15:23.740324+0000","last_became_active":"2024-01-16T15:15:22.528668+0000","last_became_peered":"2024-01-16T15:15:22.528668+0000","last_unstale":"2024-01-16T15:15:23.740324+0000","last_undegraded":"2024-01-16T15:15:23.740324+0000","last_fullsized":"2024-01-16T15:15:23.740324+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:59.497924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005396+0000","last_change":"2024-01-16T15:15:22.527598+0000","last_active":"2024-01-16T15:15:24.005396+0000","last_peered":"2024-01-16T15:15:24.005396+0000","last_clean":"2024-01-16T15:15:24.005396+0000","last_became_active":"2024-01-16T15:15:22.526241+0000","last_became_peered":"2024-01-16T15:15:22.526241+0000","last_unstale":"2024-01-16T15:15:24.005396+0000","last_undegraded":"2024-01-16T15:15:24.005396+0000","last_fullsized":"2024-01-16T15:15:24.005396+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:41.199256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007216+0000","last_change":"2024-01-16T15:15:22.537798+0000","last_active":"2024-01-16T15:15:24.007216+0000","last_peered":"2024-01-16T15:15:24.007216+0000","last_clean":"2024-01-16T15:15:24.007216+0000","last_became_active":"2024-01-16T15:15:22.536669+0000","last_became_peered":"2024-01-16T15:15:22.536669+0000","last_unstale":"2024-01-16T15:15:24.007216+0000","last_undegraded":"2024-01-16T15:15:24.007216+0000","last_fullsized":"2024-01-16T15:15:24.007216+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:06:49.471608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741073+0000","last_change":"2024-01-16T15:15:22.518878+0000","last_active":"2024-01-16T15:15:23.741073+0000","last_peered":"2024-01-16T15:15:23.741073+0000","last_clean":"2024-01-16T15:15:23.741073+0000","last_became_active":"2024-01-16T15:15:22.518594+0000","last_became_peered":"2024-01-16T15:15:22.518594+0000","last_unstale":"2024-01-16T15:15:23.741073+0000","last_undegraded":"2024-01-16T15:15:23.741073+0000","last_fullsized":"2024-01-16T15:15:23.741073+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006005+0000","last_change":"2024-01-16T15:15:22.529986+0000","last_active":"2024-01-16T15:15:24.006005+0000","last_peered":"2024-01-16T15:15:24.006005+0000","last_clean":"2024-01-16T15:15:24.006005+0000","last_became_active":"2024-01-16T15:15:22.525092+0000","last_became_peered":"2024-01-16T15:15:22.525092+0000","last_unstale":"2024-01-16T15:15:24.006005+0000","last_undegraded":"2024-01-16T15:15:24.006005+0000","last_fullsized":"2024-01-16T15:15:24.006005+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:47.277440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006703+0000","last_change":"2024-01-16T15:15:22.531986+0000","last_active":"2024-01-16T15:15:24.006703+0000","last_peered":"2024-01-16T15:15:24.006703+0000","last_clean":"2024-01-16T15:15:24.006703+0000","last_became_active":"2024-01-16T15:15:22.531247+0000","last_became_peered":"2024-01-16T15:15:22.531247+0000","last_unstale":"2024-01-16T15:15:24.006703+0000","last_undegraded":"2024-01-16T15:15:24.006703+0000","last_fullsized":"2024-01-16T15:15:24.006703+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:58:21.901076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740772+0000","last_change":"2024-01-16T15:15:22.518786+0000","last_active":"2024-01-16T15:15:23.740772+0000","last_peered":"2024-01-16T15:15:23.740772+0000","last_clean":"2024-01-16T15:15:23.740772+0000","last_became_active":"2024-01-16T15:15:22.512965+0000","last_became_peered":"2024-01-16T15:15:22.512965+0000","last_unstale":"2024-01-16T15:15:23.740772+0000","last_undegraded":"2024-01-16T15:15:23.740772+0000","last_fullsized":"2024-01-16T15:15:23.740772+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609350+0000","last_change":"2024-01-16T15:15:22.521137+0000","last_active":"2024-01-16T15:15:23.609350+0000","last_peered":"2024-01-16T15:15:23.609350+0000","last_clean":"2024-01-16T15:15:23.609350+0000","last_became_active":"2024-01-16T15:15:22.520842+0000","last_became_peered":"2024-01-16T15:15:22.520842+0000","last_unstale":"2024-01-16T15:15:23.609350+0000","last_undegraded":"2024-01-16T15:15:23.609350+0000","last_fullsized":"2024-01-16T15:15:23.609350+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:02.829205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508652+0000","last_change":"2024-01-16T15:15:22.541257+0000","last_active":"2024-01-16T15:15:23.508652+0000","last_peered":"2024-01-16T15:15:23.508652+0000","last_clean":"2024-01-16T15:15:23.508652+0000","last_became_active":"2024-01-16T15:15:22.540754+0000","last_became_peered":"2024-01-16T15:15:22.540754+0000","last_unstale":"2024-01-16T15:15:23.508652+0000","last_undegraded":"2024-01-16T15:15:23.508652+0000","last_fullsized":"2024-01-16T15:15:23.508652+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:10.664582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690290+0000","last_change":"2024-01-16T15:15:22.516532+0000","last_active":"2024-01-16T15:15:23.690290+0000","last_peered":"2024-01-16T15:15:23.690290+0000","last_clean":"2024-01-16T15:15:23.690290+0000","last_became_active":"2024-01-16T15:15:22.515440+0000","last_became_peered":"2024-01-16T15:15:22.515440+0000","last_unstale":"2024-01-16T15:15:23.690290+0000","last_undegraded":"2024-01-16T15:15:23.690290+0000","last_fullsized":"2024-01-16T15:15:23.690290+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:42.093044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610359+0000","last_change":"2024-01-16T15:15:22.521967+0000","last_active":"2024-01-16T15:15:23.610359+0000","last_peered":"2024-01-16T15:15:23.610359+0000","last_clean":"2024-01-16T15:15:23.610359+0000","last_became_active":"2024-01-16T15:15:22.520663+0000","last_became_peered":"2024-01-16T15:15:22.520663+0000","last_unstale":"2024-01-16T15:15:23.610359+0000","last_undegraded":"2024-01-16T15:15:23.610359+0000","last_fullsized":"2024-01-16T15:15:23.610359+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:27.652785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"98'4","reported_seq":16,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741472+0000","last_change":"2024-01-16T15:15:21.529234+0000","last_active":"2024-01-16T15:15:23.741472+0000","last_peered":"2024-01-16T15:15:23.741472+0000","last_clean":"2024-01-16T15:15:23.741472+0000","last_became_active":"2024-01-16T15:15:21.528711+0000","last_became_peered":"2024-01-16T15:15:21.528711+0000","last_unstale":"2024-01-16T15:15:23.741472+0000","last_undegraded":"2024-01-16T15:15:23.741472+0000","last_fullsized":"2024-01-16T15:15:23.741472+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:32:34.638361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509939+0000","last_change":"2024-01-16T15:15:22.540312+0000","last_active":"2024-01-16T15:15:23.509939+0000","last_peered":"2024-01-16T15:15:23.509939+0000","last_clean":"2024-01-16T15:15:23.509939+0000","last_became_active":"2024-01-16T15:15:22.538619+0000","last_became_peered":"2024-01-16T15:15:22.538619+0000","last_unstale":"2024-01-16T15:15:23.509939+0000","last_undegraded":"2024-01-16T15:15:23.509939+0000","last_fullsized":"2024-01-16T15:15:23.509939+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:02.521266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.542686+0000","last_change":"2024-01-16T15:15:21.516982+0000","last_active":"2024-01-16T15:15:22.542686+0000","last_peered":"2024-01-16T15:15:22.542686+0000","last_clean":"2024-01-16T15:15:22.542686+0000","last_became_active":"2024-01-16T15:15:21.516459+0000","last_became_peered":"2024-01-16T15:15:21.516459+0000","last_unstale":"2024-01-16T15:15:22.542686+0000","last_undegraded":"2024-01-16T15:15:22.542686+0000","last_fullsized":"2024-01-16T15:15:22.542686+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690957+0000","last_change":"2024-01-16T15:15:22.522691+0000","last_active":"2024-01-16T15:15:23.690957+0000","last_peered":"2024-01-16T15:15:23.690957+0000","last_clean":"2024-01-16T15:15:23.690957+0000","last_became_active":"2024-01-16T15:15:22.522375+0000","last_became_peered":"2024-01-16T15:15:22.522375+0000","last_unstale":"2024-01-16T15:15:23.690957+0000","last_undegraded":"2024-01-16T15:15:23.690957+0000","last_fullsized":"2024-01-16T15:15:23.690957+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:12.190032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740678+0000","last_change":"2024-01-16T15:15:21.529660+0000","last_active":"2024-01-16T15:15:23.740678+0000","last_peered":"2024-01-16T15:15:23.740678+0000","last_clean":"2024-01-16T15:15:23.740678+0000","last_became_active":"2024-01-16T15:15:21.529009+0000","last_became_peered":"2024-01-16T15:15:21.529009+0000","last_unstale":"2024-01-16T15:15:23.740678+0000","last_undegraded":"2024-01-16T15:15:23.740678+0000","last_fullsized":"2024-01-16T15:15:23.740678+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:41:49.081309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511419+0000","last_change":"2024-01-16T15:15:22.539245+0000","last_active":"2024-01-16T15:15:23.511419+0000","last_peered":"2024-01-16T15:15:23.511419+0000","last_clean":"2024-01-16T15:15:23.511419+0000","last_became_active":"2024-01-16T15:15:22.538962+0000","last_became_peered":"2024-01-16T15:15:22.538962+0000","last_unstale":"2024-01-16T15:15:23.511419+0000","last_undegraded":"2024-01-16T15:15:23.511419+0000","last_fullsized":"2024-01-16T15:15:23.511419+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:56.014379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740604+0000","last_change":"2024-01-16T15:15:21.521824+0000","last_active":"2024-01-16T15:15:23.740604+0000","last_peered":"2024-01-16T15:15:23.740604+0000","last_clean":"2024-01-16T15:15:23.740604+0000","last_became_active":"2024-01-16T15:15:21.521406+0000","last_became_peered":"2024-01-16T15:15:21.521406+0000","last_unstale":"2024-01-16T15:15:23.740604+0000","last_undegraded":"2024-01-16T15:15:23.740604+0000","last_fullsized":"2024-01-16T15:15:23.740604+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:18:39.695221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004361+0000","last_change":"2024-01-16T15:15:22.540055+0000","last_active":"2024-01-16T15:15:24.004361+0000","last_peered":"2024-01-16T15:15:24.004361+0000","last_clean":"2024-01-16T15:15:24.004361+0000","last_became_active":"2024-01-16T15:15:22.538674+0000","last_became_peered":"2024-01-16T15:15:22.538674+0000","last_unstale":"2024-01-16T15:15:24.004361+0000","last_undegraded":"2024-01-16T15:15:24.004361+0000","last_fullsized":"2024-01-16T15:15:24.004361+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:10:37.034062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690792+0000","last_change":"2024-01-16T15:15:21.530799+0000","last_active":"2024-01-16T15:15:23.690792+0000","last_peered":"2024-01-16T15:15:23.690792+0000","last_clean":"2024-01-16T15:15:23.690792+0000","last_became_active":"2024-01-16T15:15:21.530451+0000","last_became_peered":"2024-01-16T15:15:21.530451+0000","last_unstale":"2024-01-16T15:15:23.690792+0000","last_undegraded":"2024-01-16T15:15:23.690792+0000","last_fullsized":"2024-01-16T15:15:23.690792+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:06:26.637253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005884+0000","last_change":"2024-01-16T15:15:22.533064+0000","last_active":"2024-01-16T15:15:24.005884+0000","last_peered":"2024-01-16T15:15:24.005884+0000","last_clean":"2024-01-16T15:15:24.005884+0000","last_became_active":"2024-01-16T15:15:22.532644+0000","last_became_peered":"2024-01-16T15:15:22.532644+0000","last_unstale":"2024-01-16T15:15:24.005884+0000","last_undegraded":"2024-01-16T15:15:24.005884+0000","last_fullsized":"2024-01-16T15:15:24.005884+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:54:09.384744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"98'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740860+0000","last_change":"2024-01-16T15:15:21.525811+0000","last_active":"2024-01-16T15:15:23.740860+0000","last_peered":"2024-01-16T15:15:23.740860+0000","last_clean":"2024-01-16T15:15:23.740860+0000","last_became_active":"2024-01-16T15:15:21.525152+0000","last_became_peered":"2024-01-16T15:15:21.525152+0000","last_unstale":"2024-01-16T15:15:23.740860+0000","last_undegraded":"2024-01-16T15:15:23.740860+0000","last_fullsized":"2024-01-16T15:15:23.740860+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:38.101810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508468+0000","last_change":"2024-01-16T15:15:22.524402+0000","last_active":"2024-01-16T15:15:23.508468+0000","last_peered":"2024-01-16T15:15:23.508468+0000","last_clean":"2024-01-16T15:15:23.508468+0000","last_became_active":"2024-01-16T15:15:22.524129+0000","last_became_peered":"2024-01-16T15:15:22.524129+0000","last_unstale":"2024-01-16T15:15:23.508468+0000","last_undegraded":"2024-01-16T15:15:23.508468+0000","last_fullsized":"2024-01-16T15:15:23.508468+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:24:33.672912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.532301+0000","last_change":"2024-01-16T15:15:21.532465+0000","last_active":"2024-01-16T15:15:22.532301+0000","last_peered":"2024-01-16T15:15:22.532301+0000","last_clean":"2024-01-16T15:15:22.532301+0000","last_became_active":"2024-01-16T15:15:21.532238+0000","last_became_peered":"2024-01-16T15:15:21.532238+0000","last_unstale":"2024-01-16T15:15:22.532301+0000","last_undegraded":"2024-01-16T15:15:22.532301+0000","last_fullsized":"2024-01-16T15:15:22.532301+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:50:46.120683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511165+0000","last_change":"2024-01-16T15:15:22.530629+0000","last_active":"2024-01-16T15:15:23.511165+0000","last_peered":"2024-01-16T15:15:23.511165+0000","last_clean":"2024-01-16T15:15:23.511165+0000","last_became_active":"2024-01-16T15:15:22.530363+0000","last_became_peered":"2024-01-16T15:15:22.530363+0000","last_unstale":"2024-01-16T15:15:23.511165+0000","last_undegraded":"2024-01-16T15:15:23.511165+0000","last_fullsized":"2024-01-16T15:15:23.511165+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:14:37.560637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690271+0000","last_change":"2024-01-16T15:15:21.524375+0000","last_active":"2024-01-16T15:15:23.690271+0000","last_peered":"2024-01-16T15:15:23.690271+0000","last_clean":"2024-01-16T15:15:23.690271+0000","last_became_active":"2024-01-16T15:15:21.523731+0000","last_became_peered":"2024-01-16T15:15:21.523731+0000","last_unstale":"2024-01-16T15:15:23.690271+0000","last_undegraded":"2024-01-16T15:15:23.690271+0000","last_fullsized":"2024-01-16T15:15:23.690271+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:42:13.884029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741391+0000","last_change":"2024-01-16T15:15:22.528073+0000","last_active":"2024-01-16T15:15:23.741391+0000","last_peered":"2024-01-16T15:15:23.741391+0000","last_clean":"2024-01-16T15:15:23.741391+0000","last_became_active":"2024-01-16T15:15:22.526190+0000","last_became_peered":"2024-01-16T15:15:22.526190+0000","last_unstale":"2024-01-16T15:15:23.741391+0000","last_undegraded":"2024-01-16T15:15:23.741391+0000","last_fullsized":"2024-01-16T15:15:23.741391+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:49:55.442576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691500+0000","last_change":"2024-01-16T15:15:21.520386+0000","last_active":"2024-01-16T15:15:23.691500+0000","last_peered":"2024-01-16T15:15:23.691500+0000","last_clean":"2024-01-16T15:15:23.691500+0000","last_became_active":"2024-01-16T15:15:21.520059+0000","last_became_peered":"2024-01-16T15:15:21.520059+0000","last_unstale":"2024-01-16T15:15:23.691500+0000","last_undegraded":"2024-01-16T15:15:23.691500+0000","last_fullsized":"2024-01-16T15:15:23.691500+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:29:33.653275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511649+0000","last_change":"2024-01-16T15:15:22.539263+0000","last_active":"2024-01-16T15:15:23.511649+0000","last_peered":"2024-01-16T15:15:23.511649+0000","last_clean":"2024-01-16T15:15:23.511649+0000","last_became_active":"2024-01-16T15:15:22.538976+0000","last_became_peered":"2024-01-16T15:15:22.538976+0000","last_unstale":"2024-01-16T15:15:23.511649+0000","last_undegraded":"2024-01-16T15:15:23.511649+0000","last_fullsized":"2024-01-16T15:15:23.511649+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:44.352885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509435+0000","last_change":"2024-01-16T15:15:21.534208+0000","last_active":"2024-01-16T15:15:23.509435+0000","last_peered":"2024-01-16T15:15:23.509435+0000","last_clean":"2024-01-16T15:15:23.509435+0000","last_became_active":"2024-01-16T15:15:21.533545+0000","last_became_peered":"2024-01-16T15:15:21.533545+0000","last_unstale":"2024-01-16T15:15:23.509435+0000","last_undegraded":"2024-01-16T15:15:23.509435+0000","last_fullsized":"2024-01-16T15:15:23.509435+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006816+0000","last_change":"2024-01-16T15:15:22.534236+0000","last_active":"2024-01-16T15:15:24.006816+0000","last_peered":"2024-01-16T15:15:24.006816+0000","last_clean":"2024-01-16T15:15:24.006816+0000","last_became_active":"2024-01-16T15:15:22.533922+0000","last_became_peered":"2024-01-16T15:15:22.533922+0000","last_unstale":"2024-01-16T15:15:24.006816+0000","last_undegraded":"2024-01-16T15:15:24.006816+0000","last_fullsized":"2024-01-16T15:15:24.006816+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:12:01.861870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"98'3","reported_seq":15,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.547699+0000","last_change":"2024-01-16T15:15:21.521285+0000","last_active":"2024-01-16T15:15:22.547699+0000","last_peered":"2024-01-16T15:15:22.547699+0000","last_clean":"2024-01-16T15:15:22.547699+0000","last_became_active":"2024-01-16T15:15:21.520717+0000","last_became_peered":"2024-01-16T15:15:21.520717+0000","last_unstale":"2024-01-16T15:15:22.547699+0000","last_undegraded":"2024-01-16T15:15:22.547699+0000","last_fullsized":"2024-01-16T15:15:22.547699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:15.674482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006046+0000","last_change":"2024-01-16T15:15:22.529757+0000","last_active":"2024-01-16T15:15:24.006046+0000","last_peered":"2024-01-16T15:15:24.006046+0000","last_clean":"2024-01-16T15:15:24.006046+0000","last_became_active":"2024-01-16T15:15:22.524620+0000","last_became_peered":"2024-01-16T15:15:22.524620+0000","last_unstale":"2024-01-16T15:15:24.006046+0000","last_undegraded":"2024-01-16T15:15:24.006046+0000","last_fullsized":"2024-01-16T15:15:24.006046+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:05:38.407996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"98'6","reported_seq":18,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.552074+0000","last_change":"2024-01-16T15:15:21.518507+0000","last_active":"2024-01-16T15:15:22.552074+0000","last_peered":"2024-01-16T15:15:22.552074+0000","last_clean":"2024-01-16T15:15:22.552074+0000","last_became_active":"2024-01-16T15:15:21.518075+0000","last_became_peered":"2024-01-16T15:15:21.518075+0000","last_unstale":"2024-01-16T15:15:22.552074+0000","last_undegraded":"2024-01-16T15:15:22.552074+0000","last_fullsized":"2024-01-16T15:15:22.552074+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:43:55.375960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.509009+0000","last_change":"2024-01-16T15:15:22.521859+0000","last_active":"2024-01-16T15:15:23.509009+0000","last_peered":"2024-01-16T15:15:23.509009+0000","last_clean":"2024-01-16T15:15:23.509009+0000","last_became_active":"2024-01-16T15:15:22.521561+0000","last_became_peered":"2024-01-16T15:15:22.521561+0000","last_unstale":"2024-01-16T15:15:23.509009+0000","last_undegraded":"2024-01-16T15:15:23.509009+0000","last_fullsized":"2024-01-16T15:15:23.509009+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:14.797953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609905+0000","last_change":"2024-01-16T15:15:21.521158+0000","last_active":"2024-01-16T15:15:23.609905+0000","last_peered":"2024-01-16T15:15:23.609905+0000","last_clean":"2024-01-16T15:15:23.609905+0000","last_became_active":"2024-01-16T15:15:21.520851+0000","last_became_peered":"2024-01-16T15:15:21.520851+0000","last_unstale":"2024-01-16T15:15:23.609905+0000","last_undegraded":"2024-01-16T15:15:23.609905+0000","last_fullsized":"2024-01-16T15:15:23.609905+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:09:47.111965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690791+0000","last_change":"2024-01-16T15:15:22.530599+0000","last_active":"2024-01-16T15:15:23.690791+0000","last_peered":"2024-01-16T15:15:23.690791+0000","last_clean":"2024-01-16T15:15:23.690791+0000","last_became_active":"2024-01-16T15:15:22.526185+0000","last_became_peered":"2024-01-16T15:15:22.526185+0000","last_unstale":"2024-01-16T15:15:23.690791+0000","last_undegraded":"2024-01-16T15:15:23.690791+0000","last_fullsized":"2024-01-16T15:15:23.690791+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:23:11.722014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609612+0000","last_change":"2024-01-16T15:15:21.526311+0000","last_active":"2024-01-16T15:15:23.609612+0000","last_peered":"2024-01-16T15:15:23.609612+0000","last_clean":"2024-01-16T15:15:23.609612+0000","last_became_active":"2024-01-16T15:15:21.526019+0000","last_became_peered":"2024-01-16T15:15:21.526019+0000","last_unstale":"2024-01-16T15:15:23.609612+0000","last_undegraded":"2024-01-16T15:15:23.609612+0000","last_fullsized":"2024-01-16T15:15:23.609612+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:07:30.739366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.510461+0000","last_change":"2024-01-16T15:15:22.530528+0000","last_active":"2024-01-16T15:15:23.510461+0000","last_peered":"2024-01-16T15:15:23.510461+0000","last_clean":"2024-01-16T15:15:23.510461+0000","last_became_active":"2024-01-16T15:15:22.530244+0000","last_became_peered":"2024-01-16T15:15:22.530244+0000","last_unstale":"2024-01-16T15:15:23.510461+0000","last_undegraded":"2024-01-16T15:15:23.510461+0000","last_fullsized":"2024-01-16T15:15:23.510461+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:19.651108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516805+0000","last_change":"2024-01-16T15:15:21.528755+0000","last_active":"2024-01-16T15:15:22.516805+0000","last_peered":"2024-01-16T15:15:22.516805+0000","last_clean":"2024-01-16T15:15:22.516805+0000","last_became_active":"2024-01-16T15:15:21.528462+0000","last_became_peered":"2024-01-16T15:15:21.528462+0000","last_unstale":"2024-01-16T15:15:22.516805+0000","last_undegraded":"2024-01-16T15:15:22.516805+0000","last_fullsized":"2024-01-16T15:15:22.516805+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:14.369274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609164+0000","last_change":"2024-01-16T15:15:22.529246+0000","last_active":"2024-01-16T15:15:23.609164+0000","last_peered":"2024-01-16T15:15:23.609164+0000","last_clean":"2024-01-16T15:15:23.609164+0000","last_became_active":"2024-01-16T15:15:22.528966+0000","last_became_peered":"2024-01-16T15:15:22.528966+0000","last_unstale":"2024-01-16T15:15:23.609164+0000","last_undegraded":"2024-01-16T15:15:23.609164+0000","last_fullsized":"2024-01-16T15:15:23.609164+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:46.624569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"98'8","reported_seq":20,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690545+0000","last_change":"2024-01-16T15:15:21.525650+0000","last_active":"2024-01-16T15:15:23.690545+0000","last_peered":"2024-01-16T15:15:23.690545+0000","last_clean":"2024-01-16T15:15:23.690545+0000","last_became_active":"2024-01-16T15:15:21.524160+0000","last_became_peered":"2024-01-16T15:15:21.524160+0000","last_unstale":"2024-01-16T15:15:23.690545+0000","last_undegraded":"2024-01-16T15:15:23.690545+0000","last_fullsized":"2024-01-16T15:15:23.690545+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:28.946524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007397+0000","last_change":"2024-01-16T15:15:22.532714+0000","last_active":"2024-01-16T15:15:24.007397+0000","last_peered":"2024-01-16T15:15:24.007397+0000","last_clean":"2024-01-16T15:15:24.007397+0000","last_became_active":"2024-01-16T15:15:22.532449+0000","last_became_peered":"2024-01-16T15:15:22.532449+0000","last_unstale":"2024-01-16T15:15:24.007397+0000","last_undegraded":"2024-01-16T15:15:24.007397+0000","last_fullsized":"2024-01-16T15:15:24.007397+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:01:15.384806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"98'2","reported_seq":14,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.544469+0000","last_change":"2024-01-16T15:15:21.520262+0000","last_active":"2024-01-16T15:15:22.544469+0000","last_peered":"2024-01-16T15:15:22.544469+0000","last_clean":"2024-01-16T15:15:22.544469+0000","last_became_active":"2024-01-16T15:15:21.519869+0000","last_became_peered":"2024-01-16T15:15:21.519869+0000","last_unstale":"2024-01-16T15:15:22.544469+0000","last_undegraded":"2024-01-16T15:15:22.544469+0000","last_fullsized":"2024-01-16T15:15:22.544469+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:53.121940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004344+0000","last_change":"2024-01-16T15:15:22.543578+0000","last_active":"2024-01-16T15:15:24.004344+0000","last_peered":"2024-01-16T15:15:24.004344+0000","last_clean":"2024-01-16T15:15:24.004344+0000","last_became_active":"2024-01-16T15:15:22.543283+0000","last_became_peered":"2024-01-16T15:15:22.543283+0000","last_unstale":"2024-01-16T15:15:24.004344+0000","last_undegraded":"2024-01-16T15:15:24.004344+0000","last_fullsized":"2024-01-16T15:15:24.004344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:24:17.817501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.549526+0000","last_change":"2024-01-16T15:15:21.528614+0000","last_active":"2024-01-16T15:15:22.549526+0000","last_peered":"2024-01-16T15:15:22.549526+0000","last_clean":"2024-01-16T15:15:22.549526+0000","last_became_active":"2024-01-16T15:15:21.528325+0000","last_became_peered":"2024-01-16T15:15:21.528325+0000","last_unstale":"2024-01-16T15:15:22.549526+0000","last_undegraded":"2024-01-16T15:15:22.549526+0000","last_fullsized":"2024-01-16T15:15:22.549526+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:44:25.234717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007346+0000","last_change":"2024-01-16T15:15:22.530224+0000","last_active":"2024-01-16T15:15:24.007346+0000","last_peered":"2024-01-16T15:15:24.007346+0000","last_clean":"2024-01-16T15:15:24.007346+0000","last_became_active":"2024-01-16T15:15:22.528654+0000","last_became_peered":"2024-01-16T15:15:22.528654+0000","last_unstale":"2024-01-16T15:15:24.007346+0000","last_undegraded":"2024-01-16T15:15:24.007346+0000","last_fullsized":"2024-01-16T15:15:24.007346+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:02.566046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740462+0000","last_change":"2024-01-16T15:15:21.521658+0000","last_active":"2024-01-16T15:15:23.740462+0000","last_peered":"2024-01-16T15:15:23.740462+0000","last_clean":"2024-01-16T15:15:23.740462+0000","last_became_active":"2024-01-16T15:15:21.521051+0000","last_became_peered":"2024-01-16T15:15:21.521051+0000","last_unstale":"2024-01-16T15:15:23.740462+0000","last_undegraded":"2024-01-16T15:15:23.740462+0000","last_fullsized":"2024-01-16T15:15:23.740462+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609501+0000","last_change":"2024-01-16T15:15:22.525393+0000","last_active":"2024-01-16T15:15:23.609501+0000","last_peered":"2024-01-16T15:15:23.609501+0000","last_clean":"2024-01-16T15:15:23.609501+0000","last_became_active":"2024-01-16T15:15:22.524494+0000","last_became_peered":"2024-01-16T15:15:22.524494+0000","last_unstale":"2024-01-16T15:15:23.609501+0000","last_undegraded":"2024-01-16T15:15:23.609501+0000","last_fullsized":"2024-01-16T15:15:23.609501+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:08:10.852002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740975+0000","last_change":"2024-01-16T15:15:21.531548+0000","last_active":"2024-01-16T15:15:23.740975+0000","last_peered":"2024-01-16T15:15:23.740975+0000","last_clean":"2024-01-16T15:15:23.740975+0000","last_became_active":"2024-01-16T15:15:21.531267+0000","last_became_peered":"2024-01-16T15:15:21.531267+0000","last_unstale":"2024-01-16T15:15:23.740975+0000","last_undegraded":"2024-01-16T15:15:23.740975+0000","last_fullsized":"2024-01-16T15:15:23.740975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:42.610622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609997+0000","last_change":"2024-01-16T15:15:22.522512+0000","last_active":"2024-01-16T15:15:23.609997+0000","last_peered":"2024-01-16T15:15:23.609997+0000","last_clean":"2024-01-16T15:15:23.609997+0000","last_became_active":"2024-01-16T15:15:22.522206+0000","last_became_peered":"2024-01-16T15:15:22.522206+0000","last_unstale":"2024-01-16T15:15:23.609997+0000","last_undegraded":"2024-01-16T15:15:23.609997+0000","last_fullsized":"2024-01-16T15:15:23.609997+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:29.962056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004895+0000","last_change":"2024-01-16T15:15:21.524797+0000","last_active":"2024-01-16T15:15:24.004895+0000","last_peered":"2024-01-16T15:15:24.004895+0000","last_clean":"2024-01-16T15:15:24.004895+0000","last_became_active":"2024-01-16T15:15:21.524502+0000","last_became_peered":"2024-01-16T15:15:21.524502+0000","last_unstale":"2024-01-16T15:15:24.004895+0000","last_undegraded":"2024-01-16T15:15:24.004895+0000","last_fullsized":"2024-01-16T15:15:24.004895+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:17:46.648168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609344+0000","last_change":"2024-01-16T15:15:22.523225+0000","last_active":"2024-01-16T15:15:23.609344+0000","last_peered":"2024-01-16T15:15:23.609344+0000","last_clean":"2024-01-16T15:15:23.609344+0000","last_became_active":"2024-01-16T15:15:22.522951+0000","last_became_peered":"2024-01-16T15:15:22.522951+0000","last_unstale":"2024-01-16T15:15:23.609344+0000","last_undegraded":"2024-01-16T15:15:23.609344+0000","last_fullsized":"2024-01-16T15:15:23.609344+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:21.818020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.004192+0000","last_change":"2024-01-16T15:15:21.519991+0000","last_active":"2024-01-16T15:15:24.004192+0000","last_peered":"2024-01-16T15:15:24.004192+0000","last_clean":"2024-01-16T15:15:24.004192+0000","last_became_active":"2024-01-16T15:15:21.519565+0000","last_became_peered":"2024-01-16T15:15:21.519565+0000","last_unstale":"2024-01-16T15:15:24.004192+0000","last_undegraded":"2024-01-16T15:15:24.004192+0000","last_fullsized":"2024-01-16T15:15:24.004192+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:52:50.723441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006770+0000","last_change":"2024-01-16T15:15:22.535998+0000","last_active":"2024-01-16T15:15:24.006770+0000","last_peered":"2024-01-16T15:15:24.006770+0000","last_clean":"2024-01-16T15:15:24.006770+0000","last_became_active":"2024-01-16T15:15:22.535745+0000","last_became_peered":"2024-01-16T15:15:22.535745+0000","last_unstale":"2024-01-16T15:15:24.006770+0000","last_undegraded":"2024-01-16T15:15:24.006770+0000","last_fullsized":"2024-01-16T15:15:24.006770+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:00:08.616887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740258+0000","last_change":"2024-01-16T15:15:21.519702+0000","last_active":"2024-01-16T15:15:23.740258+0000","last_peered":"2024-01-16T15:15:23.740258+0000","last_clean":"2024-01-16T15:15:23.740258+0000","last_became_active":"2024-01-16T15:15:21.518926+0000","last_became_peered":"2024-01-16T15:15:21.518926+0000","last_unstale":"2024-01-16T15:15:23.740258+0000","last_undegraded":"2024-01-16T15:15:23.740258+0000","last_fullsized":"2024-01-16T15:15:23.740258+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:31.707777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741243+0000","last_change":"2024-01-16T15:15:22.536514+0000","last_active":"2024-01-16T15:15:23.741243+0000","last_peered":"2024-01-16T15:15:23.741243+0000","last_clean":"2024-01-16T15:15:23.741243+0000","last_became_active":"2024-01-16T15:15:22.536248+0000","last_became_peered":"2024-01-16T15:15:22.536248+0000","last_unstale":"2024-01-16T15:15:23.741243+0000","last_undegraded":"2024-01-16T15:15:23.741243+0000","last_fullsized":"2024-01-16T15:15:23.741243+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:12:10.024161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"98'2","reported_seq":14,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.005379+0000","last_change":"2024-01-16T15:15:21.522416+0000","last_active":"2024-01-16T15:15:24.005379+0000","last_peered":"2024-01-16T15:15:24.005379+0000","last_clean":"2024-01-16T15:15:24.005379+0000","last_became_active":"2024-01-16T15:15:21.522118+0000","last_became_peered":"2024-01-16T15:15:21.522118+0000","last_unstale":"2024-01-16T15:15:24.005379+0000","last_undegraded":"2024-01-16T15:15:24.005379+0000","last_fullsized":"2024-01-16T15:15:24.005379+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:09:22.762652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.691386+0000","last_change":"2024-01-16T15:15:22.531377+0000","last_active":"2024-01-16T15:15:23.691386+0000","last_peered":"2024-01-16T15:15:23.691386+0000","last_clean":"2024-01-16T15:15:23.691386+0000","last_became_active":"2024-01-16T15:15:22.529798+0000","last_became_peered":"2024-01-16T15:15:22.529798+0000","last_unstale":"2024-01-16T15:15:23.691386+0000","last_undegraded":"2024-01-16T15:15:23.691386+0000","last_fullsized":"2024-01-16T15:15:23.691386+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:46:07.721245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"99'1","reported_seq":13,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:25.470635+0000","last_change":"2024-01-16T15:15:21.532318+0000","last_active":"2024-01-16T15:15:25.470635+0000","last_peered":"2024-01-16T15:15:25.470635+0000","last_clean":"2024-01-16T15:15:25.470635+0000","last_became_active":"2024-01-16T15:15:21.531869+0000","last_became_peered":"2024-01-16T15:15:21.531869+0000","last_unstale":"2024-01-16T15:15:25.470635+0000","last_undegraded":"2024-01-16T15:15:25.470635+0000","last_fullsized":"2024-01-16T15:15:25.470635+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:31:40.183724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.690994+0000","last_change":"2024-01-16T15:15:22.535293+0000","last_active":"2024-01-16T15:15:23.690994+0000","last_peered":"2024-01-16T15:15:23.690994+0000","last_clean":"2024-01-16T15:15:23.690994+0000","last_became_active":"2024-01-16T15:15:22.534929+0000","last_became_peered":"2024-01-16T15:15:22.534929+0000","last_unstale":"2024-01-16T15:15:23.690994+0000","last_undegraded":"2024-01-16T15:15:23.690994+0000","last_fullsized":"2024-01-16T15:15:23.690994+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:20:59.606169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.740931+0000","last_change":"2024-01-16T15:15:21.520890+0000","last_active":"2024-01-16T15:15:23.740931+0000","last_peered":"2024-01-16T15:15:23.740931+0000","last_clean":"2024-01-16T15:15:23.740931+0000","last_became_active":"2024-01-16T15:15:21.520488+0000","last_became_peered":"2024-01-16T15:15:21.520488+0000","last_unstale":"2024-01-16T15:15:23.740931+0000","last_undegraded":"2024-01-16T15:15:23.740931+0000","last_fullsized":"2024-01-16T15:15:23.740931+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:30:08.456302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.006786+0000","last_change":"2024-01-16T15:15:22.531273+0000","last_active":"2024-01-16T15:15:24.006786+0000","last_peered":"2024-01-16T15:15:24.006786+0000","last_clean":"2024-01-16T15:15:24.006786+0000","last_became_active":"2024-01-16T15:15:22.530828+0000","last_became_peered":"2024-01-16T15:15:22.530828+0000","last_unstale":"2024-01-16T15:15:24.006786+0000","last_undegraded":"2024-01-16T15:15:24.006786+0000","last_fullsized":"2024-01-16T15:15:24.006786+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:56:25.470687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.531826+0000","last_change":"2024-01-16T15:15:21.527345+0000","last_active":"2024-01-16T15:15:22.531826+0000","last_peered":"2024-01-16T15:15:22.531826+0000","last_clean":"2024-01-16T15:15:22.531826+0000","last_became_active":"2024-01-16T15:15:21.527018+0000","last_became_peered":"2024-01-16T15:15:21.527018+0000","last_unstale":"2024-01-16T15:15:22.531826+0000","last_undegraded":"2024-01-16T15:15:22.531826+0000","last_fullsized":"2024-01-16T15:15:22.531826+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:13.821755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609781+0000","last_change":"2024-01-16T15:15:22.525600+0000","last_active":"2024-01-16T15:15:23.609781+0000","last_peered":"2024-01-16T15:15:23.609781+0000","last_clean":"2024-01-16T15:15:23.609781+0000","last_became_active":"2024-01-16T15:15:22.525112+0000","last_became_peered":"2024-01-16T15:15:22.525112+0000","last_unstale":"2024-01-16T15:15:23.609781+0000","last_undegraded":"2024-01-16T15:15:23.609781+0000","last_fullsized":"2024-01-16T15:15:23.609781+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:33:49.965076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.741004+0000","last_change":"2024-01-16T15:15:21.526060+0000","last_active":"2024-01-16T15:15:23.741004+0000","last_peered":"2024-01-16T15:15:23.741004+0000","last_clean":"2024-01-16T15:15:23.741004+0000","last_became_active":"2024-01-16T15:15:21.525792+0000","last_became_peered":"2024-01-16T15:15:21.525792+0000","last_unstale":"2024-01-16T15:15:23.741004+0000","last_undegraded":"2024-01-16T15:15:23.741004+0000","last_fullsized":"2024-01-16T15:15:23.741004+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:47:42.115589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.511167+0000","last_change":"2024-01-16T15:15:22.541120+0000","last_active":"2024-01-16T15:15:23.511167+0000","last_peered":"2024-01-16T15:15:23.511167+0000","last_clean":"2024-01-16T15:15:23.511167+0000","last_became_active":"2024-01-16T15:15:22.537720+0000","last_became_peered":"2024-01-16T15:15:22.537720+0000","last_unstale":"2024-01-16T15:15:23.511167+0000","last_undegraded":"2024-01-16T15:15:23.511167+0000","last_fullsized":"2024-01-16T15:15:23.511167+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:08:05.452028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.531464+0000","last_change":"2024-01-16T15:15:21.530401+0000","last_active":"2024-01-16T15:15:22.531464+0000","last_peered":"2024-01-16T15:15:22.531464+0000","last_clean":"2024-01-16T15:15:22.531464+0000","last_became_active":"2024-01-16T15:15:21.530083+0000","last_became_peered":"2024-01-16T15:15:21.530083+0000","last_unstale":"2024-01-16T15:15:22.531464+0000","last_undegraded":"2024-01-16T15:15:22.531464+0000","last_fullsized":"2024-01-16T15:15:22.531464+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:18.693488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609251+0000","last_change":"2024-01-16T15:15:22.530971+0000","last_active":"2024-01-16T15:15:23.609251+0000","last_peered":"2024-01-16T15:15:23.609251+0000","last_clean":"2024-01-16T15:15:23.609251+0000","last_became_active":"2024-01-16T15:15:22.530694+0000","last_became_peered":"2024-01-16T15:15:22.530694+0000","last_unstale":"2024-01-16T15:15:23.609251+0000","last_undegraded":"2024-01-16T15:15:23.609251+0000","last_fullsized":"2024-01-16T15:15:23.609251+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:22:02.926295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.508734+0000","last_change":"2024-01-16T15:15:21.524472+0000","last_active":"2024-01-16T15:15:23.508734+0000","last_peered":"2024-01-16T15:15:23.508734+0000","last_clean":"2024-01-16T15:15:23.508734+0000","last_became_active":"2024-01-16T15:15:21.524175+0000","last_became_peered":"2024-01-16T15:15:21.524175+0000","last_unstale":"2024-01-16T15:15:23.508734+0000","last_undegraded":"2024-01-16T15:15:23.508734+0000","last_fullsized":"2024-01-16T15:15:23.508734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:25:43.337353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:24.007607+0000","last_change":"2024-01-16T15:15:22.533778+0000","last_active":"2024-01-16T15:15:24.007607+0000","last_peered":"2024-01-16T15:15:24.007607+0000","last_clean":"2024-01-16T15:15:24.007607+0000","last_became_active":"2024-01-16T15:15:22.533426+0000","last_became_peered":"2024-01-16T15:15:22.533426+0000","last_unstale":"2024-01-16T15:15:24.007607+0000","last_undegraded":"2024-01-16T15:15:24.007607+0000","last_fullsized":"2024-01-16T15:15:24.007607+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:24:47.611650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.609465+0000","last_change":"2024-01-16T15:15:21.533877+0000","last_active":"2024-01-16T15:15:23.609465+0000","last_peered":"2024-01-16T15:15:23.609465+0000","last_clean":"2024-01-16T15:15:23.609465+0000","last_became_active":"2024-01-16T15:15:21.533489+0000","last_became_peered":"2024-01-16T15:15:21.533489+0000","last_unstale":"2024-01-16T15:15:23.609465+0000","last_undegraded":"2024-01-16T15:15:23.609465+0000","last_fullsized":"2024-01-16T15:15:23.609465+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:51:15.404651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-01-16T15:15:23.610593+0000","last_change":"2024-01-16T15:15:22.521850+0000","last_active":"2024-01-16T15:15:23.610593+0000","last_peered":"2024-01-16T15:15:23.610593+0000","last_clean":"2024-01-16T15:15:23.610593+0000","last_became_active":"2024-01-16T15:15:22.521581+0000","last_became_peered":"2024-01-16T15:15:22.521581+0000","last_unstale":"2024-01-16T15:15:23.610593+0000","last_undegraded":"2024-01-16T15:15:23.610593+0000","last_fullsized":"2024-01-16T15:15:23.610593+0000","mapping_epoch":97,"log_start":"0'0","ondisk_log_start":"0'0","created":97,"last_epoch_clean":98,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:21.481640+0000","last_clean_scrub_stamp":"2024-01-16T15:15:21.481640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:35:29.487154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"98'1","reported_seq":13,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.545734+0000","last_change":"2024-01-16T15:15:21.522571+0000","last_active":"2024-01-16T15:15:22.545734+0000","last_peered":"2024-01-16T15:15:22.545734+0000","last_clean":"2024-01-16T15:15:22.545734+0000","last_became_active":"2024-01-16T15:15:21.522205+0000","last_became_peered":"2024-01-16T15:15:21.522205+0000","last_unstale":"2024-01-16T15:15:22.545734+0000","last_undegraded":"2024-01-16T15:15:22.545734+0000","last_fullsized":"2024-01-16T15:15:22.545734+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":96,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:15:20.466108+0000","last_clean_scrub_stamp":"2024-01-16T15:15:20.466108+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:04.961973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":241,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-01-16T15:15:22.516932+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:15:22.516932+0000","last_peered":"2024-01-16T15:15:22.516932+0000","last_clean":"2024-01-16T15:15:22.516932+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:15:22.516932+0000","last_undegraded":"2024-01-16T15:15:22.516932+0000","last_fullsized":"2024-01-16T15:15:22.516932+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509630,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68408,"kb_used_data":1196,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303432,"statfs":{"total":96636764160,"available":96566714368,"internally_reserved":0,"allocated":1224704,"data_stored":526071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.3140000000000001,"15min":1.615},"min":{"1min":0.75,"5min":0.57099999999999995,"15min":0.55100000000000005},"max":{"1min":2.5030000000000001,"5min":8.8070000000000004,"15min":27.853999999999999},"last":1.105},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.663,"15min":1.708},"min":{"1min":0.93000000000000005,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.6399999999999999,"5min":28.603999999999999,"15min":34.939999999999998},"last":1.359}]},{"osd":1,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.292,"15min":1.4830000000000001},"min":{"1min":0.94899999999999995,"5min":0.374,"15min":0.374},"max":{"1min":1.528,"5min":8.2650000000000006,"15min":13.859},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.754,"15min":1.651},"min":{"1min":0.85599999999999998,"5min":0.69999999999999996,"15min":0.502},"max":{"1min":2.702,"5min":33.225000000000001,"15min":33.225000000000001},"last":1.3080000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4339999999999999,"5min":1.4330000000000001,"15min":1.5860000000000001},"min":{"1min":0.879,"5min":0.71899999999999997,"15min":0.55900000000000005},"max":{"1min":2.6230000000000002,"5min":9.6679999999999993,"15min":13.050000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.371,"15min":1.742},"min":{"1min":0.69499999999999995,"5min":0.59099999999999997,"15min":0.53100000000000003},"max":{"1min":2.6309999999999998,"5min":6.782,"15min":35.011000000000003},"last":0.998}]},{"osd":3,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.486,"15min":1.657},"min":{"1min":0.94799999999999995,"5min":0.439,"15min":0.374},"max":{"1min":1.8720000000000001,"5min":15.076000000000001,"15min":16.353000000000002},"last":1.393},{"interface":"front","average":{"1min":1.5509999999999999,"5min":1.5549999999999999,"15min":1.7170000000000001},"min":{"1min":0.999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.758,"5min":9.7780000000000005,"15min":14.273999999999999},"last":1.212}]},{"osd":4,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3919999999999999,"15min":1.827},"min":{"1min":0.90900000000000003,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.7090000000000001,"5min":10.952999999999999,"15min":34.972999999999999},"last":0.873},{"interface":"front","average":{"1min":1.331,"5min":1.4319999999999999,"15min":1.798},"min":{"1min":0.85899999999999999,"5min":0.58299999999999996,"15min":0.40400000000000003},"max":{"1min":1.6519999999999999,"5min":5.2300000000000004,"15min":16.003},"last":1.165}]},{"osd":5,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.591,"15min":2.0099999999999998},"min":{"1min":1.1339999999999999,"5min":0.69199999999999995,"15min":0.58099999999999996},"max":{"1min":1.798,"5min":4.4550000000000001,"15min":23.760999999999999},"last":1.454},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.6259999999999999,"15min":1.9059999999999999},"min":{"1min":0.96299999999999997,"5min":0.70699999999999996,"15min":0.52400000000000002},"max":{"1min":3.496,"5min":9.4120000000000008,"15min":23.716999999999999},"last":1.415}]},{"osd":7,"last update":"Tue Jan 16 15:15:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.5069999999999999,"15min":2.0430000000000001},"min":{"1min":0.90700000000000003,"5min":0.54200000000000004,"15min":0.48099999999999998},"max":{"1min":1.696,"5min":6.9219999999999997,"15min":34.889000000000003},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.512,"5min":1.79,"15min":2.1760000000000002},"min":{"1min":1.2470000000000001,"5min":0.65100000000000002,"15min":0.47599999999999998},"max":{"1min":2.0499999999999998,"5min":23.242999999999999,"15min":46.305},"last":1.5}]}]},{"osd":4,"up_from":15,"seq":64424509632,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68420,"kb_used_data":1208,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303420,"statfs":{"total":96636764160,"available":96566702080,"internally_reserved":0,"allocated":1236992,"data_stored":527155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.226,"15min":1.55},"min":{"1min":0.76900000000000002,"5min":0.44400000000000001,"15min":0.38700000000000001},"max":{"1min":1.899,"5min":4.1399999999999997,"15min":22.757999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.125,"15min":1.2729999999999999},"min":{"1min":0.69799999999999995,"5min":0.59699999999999998,"15min":0.432},"max":{"1min":1.474,"5min":4.0270000000000001,"15min":14.034000000000001},"last":1.0329999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.413,"15min":1.577},"min":{"1min":0.72599999999999998,"5min":0.52800000000000002,"15min":0.51600000000000001},"max":{"1min":1.851,"5min":17.811,"15min":18.347000000000001},"last":1.248},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.3140000000000001,"15min":1.49},"min":{"1min":0.90700000000000003,"5min":0.59799999999999998,"15min":0.498},"max":{"1min":1.5600000000000001,"5min":4.0449999999999999,"15min":18.858000000000001},"last":1.5149999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.544,"15min":1.679},"min":{"1min":0.83799999999999997,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.768,"5min":17.651,"15min":17.651},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.327,"5min":1.754,"15min":1.722},"min":{"1min":0.83499999999999996,"5min":0.52600000000000002,"15min":0.52600000000000002},"max":{"1min":1.8180000000000001,"5min":29.161000000000001,"15min":29.161000000000001},"last":1.1020000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.3320000000000001,"15min":1.554},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.41699999999999998},"max":{"1min":1.6659999999999999,"5min":3.972,"15min":26.741},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.284,"5min":1.571,"15min":1.7929999999999999},"min":{"1min":0.86899999999999999,"5min":0.54800000000000004,"15min":0.54800000000000004},"max":{"1min":1.762,"5min":17.715,"15min":26.701000000000001},"last":1.1639999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5660000000000001,"15min":1.7370000000000001},"min":{"1min":1.0029999999999999,"5min":0.51400000000000001,"15min":0.441},"max":{"1min":1.988,"5min":5.3170000000000002,"15min":18.919},"last":1.292},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.5029999999999999,"15min":1.6759999999999999},"min":{"1min":0.95599999999999996,"5min":0.39100000000000001,"15min":0.36399999999999999},"max":{"1min":2.0270000000000001,"5min":5.3520000000000003,"15min":26.597000000000001},"last":1.583}]},{"osd":6,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.544,"5min":2.0739999999999998,"15min":1.9419999999999999},"min":{"1min":0.995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":5.0750000000000002,"5min":32.649999999999999,"15min":32.649999999999999},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5469999999999999,"15min":1.823},"min":{"1min":1.1799999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.0630000000000002,"5min":4.3129999999999997,"15min":24.094999999999999},"last":1.4159999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.6890000000000001,"15min":1.7649999999999999},"min":{"1min":1.056,"5min":0.437,"15min":0.41699999999999998},"max":{"1min":1.8899999999999999,"5min":17.446000000000002,"15min":26.683},"last":1.369},{"interface":"front","average":{"1min":1.52,"5min":1.871,"15min":1.9650000000000001},"min":{"1min":1.248,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.9350000000000001,"5min":17.765000000000001,"15min":17.765000000000001},"last":1.458}]}]},{"osd":7,"up_from":14,"seq":60129542334,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68988,"kb_used_data":1776,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302852,"statfs":{"total":96636764160,"available":96566120448,"internally_reserved":0,"allocated":1818624,"data_stored":1115899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68811210},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.2649999999999999,"15min":1.5089999999999999},"min":{"1min":0.68100000000000005,"5min":0.63900000000000001,"15min":0.36899999999999999},"max":{"1min":1.7310000000000001,"5min":4.3680000000000003,"15min":15.661},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.196,"5min":1.3240000000000001,"15min":1.5449999999999999},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":2.1080000000000001,"5min":12.461,"15min":30.126000000000001},"last":1.226}]},{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.423,"15min":1.4530000000000001},"min":{"1min":0.63200000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":2.085,"5min":20.327000000000002,"15min":24.338000000000001},"last":1.0169999999999999},{"interface":"front","average":{"1min":1.234,"5min":1.395,"15min":1.6180000000000001},"min":{"1min":0.81399999999999995,"5min":0.77400000000000002,"15min":0.46200000000000002},"max":{"1min":1.6419999999999999,"5min":12.526,"15min":21.298999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.3440000000000001,"15min":1.4970000000000001},"min":{"1min":0.71899999999999997,"5min":0.57199999999999995,"15min":0.51500000000000001},"max":{"1min":1.9219999999999999,"5min":4.3380000000000001,"15min":12.474},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.702,"15min":1.651},"min":{"1min":0.69299999999999995,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":1.8999999999999999,"5min":42.165999999999997,"15min":42.165999999999997},"last":1.4219999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4039999999999999,"15min":1.6020000000000001},"min":{"1min":0.88800000000000001,"5min":0.72899999999999998,"15min":0.47699999999999998},"max":{"1min":1.7150000000000001,"5min":4.2530000000000001,"15min":12.58},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.9430000000000001,"15min":1.7869999999999999},"min":{"1min":0.78600000000000003,"5min":0.629,"15min":0.54600000000000004},"max":{"1min":2.4630000000000001,"5min":42.222000000000001,"15min":42.222000000000001},"last":0.92400000000000004}]},{"osd":4,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3520000000000001,"15min":1.591},"min":{"1min":0.99099999999999999,"5min":0.54100000000000004,"15min":0.42799999999999999},"max":{"1min":2.0219999999999998,"5min":4.1390000000000002,"15min":14.659000000000001},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.831,"15min":1.825},"min":{"1min":0.57199999999999995,"5min":0.504,"15min":0.38800000000000001},"max":{"1min":2.028,"5min":42.276000000000003,"15min":42.276000000000003},"last":1.3600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.486,"15min":1.8360000000000001},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.6459999999999999,"5min":12.609999999999999,"15min":26.23},"last":1.113},{"interface":"front","average":{"1min":1.5309999999999999,"5min":1.6160000000000001,"15min":1.8400000000000001},"min":{"1min":1.1870000000000001,"5min":0.66000000000000003,"15min":0.41099999999999998},"max":{"1min":1.8400000000000001,"5min":7.4649999999999999,"15min":21.312000000000001},"last":1.383}]},{"osd":6,"last update":"Tue Jan 16 15:15:22 2024","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.9790000000000001,"15min":1.8640000000000001},"min":{"1min":0.97199999999999998,"5min":0.66000000000000003,"15min":0.501},"max":{"1min":2.0880000000000001,"5min":42.090000000000003,"15min":42.090000000000003},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.482,"5min":1.597,"15min":1.956},"min":{"1min":0.81699999999999995,"5min":0.67500000000000004,"15min":0.40799999999999997},"max":{"1min":1.8300000000000001,"5min":12.567,"15min":29.991},"last":1.3300000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542336,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76728,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295112,"statfs":{"total":96636764160,"available":96558194688,"internally_reserved":0,"allocated":1220608,"data_stored":525489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.405,"15min":1.5389999999999999},"min":{"1min":0.79700000000000004,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.6579999999999999,"5min":6.8380000000000001,"15min":19.382999999999999},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.9730000000000001,"5min":1.3740000000000001,"15min":1.3899999999999999},"min":{"1min":0.77000000000000002,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":13.127000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":1.127}]},{"osd":1,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.5409999999999999,"15min":1.6339999999999999},"min":{"1min":0.70699999999999996,"5min":0.56100000000000005,"15min":0.52000000000000002},"max":{"1min":3.8109999999999999,"5min":13.571,"15min":17.693999999999999},"last":1.484},{"interface":"front","average":{"1min":1.996,"5min":1.6419999999999999,"15min":1.6950000000000001},"min":{"1min":0.95099999999999996,"5min":0.51600000000000001,"15min":0.432},"max":{"1min":17.062000000000001,"5min":17.062000000000001,"15min":20.645},"last":1.27}]},{"osd":2,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.5740000000000001,"15min":1.6879999999999999},"min":{"1min":0.79100000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.77,"5min":6.4039999999999999,"15min":21.138000000000002},"last":1.591},{"interface":"front","average":{"1min":1.528,"5min":1.462,"15min":1.4910000000000001},"min":{"1min":0.995,"5min":0.626,"15min":0.58099999999999996},"max":{"1min":4.0720000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.45}]},{"osd":3,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.4930000000000001,"15min":1.8},"min":{"1min":0.95499999999999996,"5min":0.60099999999999998,"15min":0.59799999999999998},"max":{"1min":2.4769999999999999,"5min":7.984,"15min":23.109000000000002},"last":1.3080000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.6060000000000001,"15min":1.7949999999999999},"min":{"1min":1.0169999999999999,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.95,"5min":6.4610000000000003,"15min":19.303000000000001},"last":1.5049999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.643,"5min":1.5449999999999999,"15min":1.849},"min":{"1min":0.85799999999999998,"5min":0.55000000000000004,"15min":0.442},"max":{"1min":9.3309999999999995,"5min":10.831,"15min":24.436},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.48,"15min":1.744},"min":{"1min":0.84699999999999998,"5min":0.53500000000000003,"15min":0.434},"max":{"1min":1.6140000000000001,"5min":5.6820000000000004,"15min":19.126000000000001},"last":1.3979999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:15:24 2024","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.526,"15min":1.635},"min":{"1min":0.95599999999999996,"5min":0.48699999999999999,"15min":0.46500000000000002},"max":{"1min":1.718,"5min":12.914,"15min":21.192},"last":1.004},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.5609999999999999,"15min":1.978},"min":{"1min":0.97299999999999998,"5min":0.51000000000000001,"15min":0.48599999999999999},"max":{"1min":2.0299999999999998,"5min":7.8609999999999998,"15min":27.042000000000002},"last":1.3600000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.968,"5min":1.7989999999999999,"15min":1.9350000000000001},"min":{"1min":0.92600000000000005,"5min":0.59299999999999997,"15min":0.51200000000000001},"max":{"1min":13.034000000000001,"5min":18.585999999999999,"15min":32.405999999999999},"last":1.329},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.7410000000000001,"15min":2.0840000000000001},"min":{"1min":1.216,"5min":0.70699999999999996,"15min":0.56000000000000005},"max":{"1min":2.1429999999999998,"5min":10.756,"15min":24.367000000000001},"last":1.554}]}]},{"osd":3,"up_from":14,"seq":60129542336,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76736,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295104,"statfs":{"total":96636764160,"available":96558186496,"internally_reserved":0,"allocated":1232896,"data_stored":527053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77330887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93899999999999995,"5min":1.5,"15min":1.278},"min":{"1min":0.38800000000000001,"5min":0.38500000000000001,"15min":0.28999999999999998},"max":{"1min":1.403,"5min":28.437999999999999,"15min":31.562999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.117,"5min":1.26,"15min":1.173},"min":{"1min":0.65900000000000003,"5min":0.40600000000000003,"15min":0.39600000000000002},"max":{"1min":1.8440000000000001,"5min":11.361000000000001,"15min":11.361000000000001},"last":0.89900000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":1.236,"15min":1.145},"min":{"1min":0.59999999999999998,"5min":0.42999999999999999,"15min":0.33300000000000002},"max":{"1min":1.5940000000000001,"5min":10.856999999999999,"15min":15.569000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.7050000000000001,"15min":1.429},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.643,"5min":28.352,"15min":28.352},"last":0.66800000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":2.379,"15min":1.6879999999999999},"min":{"1min":0.71999999999999997,"5min":0.47799999999999998,"15min":0.36399999999999999},"max":{"1min":1.8109999999999999,"5min":39.265999999999998,"15min":39.265999999999998},"last":1.5009999999999999},{"interface":"front","average":{"1min":1.264,"5min":1.4970000000000001,"15min":1.385},"min":{"1min":0.67800000000000005,"5min":0.498,"15min":0.46200000000000002},"max":{"1min":3.911,"5min":13.526,"15min":14.577999999999999},"last":1.097}]},{"osd":4,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.516,"15min":1.5620000000000001},"min":{"1min":0.86499999999999999,"5min":0.64900000000000002,"15min":0.53800000000000003},"max":{"1min":2.1019999999999999,"5min":6.0629999999999997,"15min":20.978999999999999},"last":0.995},{"interface":"front","average":{"1min":1.448,"5min":2.2879999999999998,"15min":1.8480000000000001},"min":{"1min":1.034,"5min":0.93100000000000005,"15min":0.72999999999999998},"max":{"1min":2.0030000000000001,"5min":28.547999999999998,"15min":28.547999999999998},"last":1.2130000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.353,"5min":1.746,"15min":1.766},"min":{"1min":0.81699999999999995,"5min":0.69199999999999995,"15min":0.67000000000000004},"max":{"1min":1.911,"5min":10.988,"15min":20.663},"last":1.306},{"interface":"front","average":{"1min":1.575,"5min":2.3090000000000002,"15min":1.9490000000000001},"min":{"1min":0.96299999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1019999999999999,"5min":28.588000000000001,"15min":28.588000000000001},"last":1.399}]},{"osd":6,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.77,"15min":1.754},"min":{"1min":1.0069999999999999,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.2170000000000001,"5min":13.691000000000001,"15min":13.691000000000001},"last":1.345},{"interface":"front","average":{"1min":2.0910000000000002,"5min":2.0819999999999999,"15min":1.9510000000000001},"min":{"1min":0.95099999999999996,"5min":0.77300000000000002,"15min":0.76400000000000001},"max":{"1min":15.114000000000001,"5min":26.698,"15min":26.698},"last":1.464}]},{"osd":7,"last update":"Tue Jan 16 15:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5289999999999999,"5min":1.905,"15min":1.776},"min":{"1min":1.0589999999999999,"5min":0.80400000000000005,"15min":0.60499999999999998},"max":{"1min":1.9630000000000001,"5min":13.622,"15min":13.622},"last":1.528},{"interface":"front","average":{"1min":1.5289999999999999,"5min":2.0459999999999998,"15min":2.1659999999999999},"min":{"1min":0.90400000000000003,"5min":0.87,"15min":0.80400000000000005},"max":{"1min":2.2789999999999999,"5min":15.372999999999999,"15min":49.811},"last":1.575}]}]},{"osd":2,"up_from":14,"seq":60129542335,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76732,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94295108,"statfs":{"total":96636764160,"available":96558190592,"internally_reserved":0,"allocated":1220608,"data_stored":525453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.284,"15min":1.0449999999999999},"min":{"1min":0.372,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.7130000000000001,"5min":18.289999999999999,"15min":18.289999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.2210000000000001,"15min":1.099},"min":{"1min":0.65100000000000002,"5min":0.40699999999999997,"15min":0.28599999999999998},"max":{"1min":1.6000000000000001,"5min":7.9960000000000004,"15min":7.9960000000000004},"last":1.7529999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.4650000000000001,"15min":1.1850000000000001},"min":{"1min":0.55900000000000005,"5min":0.45900000000000002,"15min":0.40300000000000002},"max":{"1min":1.8979999999999999,"5min":23.135999999999999,"15min":23.135999999999999},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.2050000000000001,"5min":1.278,"15min":1.2050000000000001},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.6699999999999999,"5min":4.7439999999999998,"15min":4.8849999999999998},"last":1.127}]},{"osd":3,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.3859999999999999,"15min":1.1839999999999999},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":7.306,"5min":18.838000000000001,"15min":18.838000000000001},"last":1.5},{"interface":"front","average":{"1min":1.421,"5min":1.647,"15min":1.4510000000000001},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.839,"5min":19.375,"15min":19.375},"last":1.54}]},{"osd":4,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.8140000000000001,"15min":1.5620000000000001},"min":{"1min":0.79300000000000004,"5min":0.70499999999999996,"15min":0.59999999999999998},"max":{"1min":2.0249999999999999,"5min":23.225000000000001,"15min":23.225000000000001},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.651,"15min":1.5600000000000001},"min":{"1min":1.036,"5min":0.72699999999999998,"15min":0.59199999999999997},"max":{"1min":1.7509999999999999,"5min":9.8960000000000008,"15min":9.8960000000000008},"last":1.381}]},{"osd":5,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.7809999999999999,"5min":1.8580000000000001,"15min":1.6200000000000001},"min":{"1min":1.0740000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"max":{"1min":7.7640000000000002,"5min":20.157,"15min":20.157},"last":1.7},{"interface":"front","average":{"1min":1.8169999999999999,"5min":1.8089999999999999,"15min":1.601},"min":{"1min":0.84299999999999997,"5min":0.623,"15min":0.623},"max":{"1min":7.9820000000000002,"5min":25.004000000000001,"15min":25.004000000000001},"last":1.3080000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.8120000000000001,"5min":2.1040000000000001,"15min":1.7390000000000001},"min":{"1min":0.89400000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":7.6630000000000003,"5min":34.523000000000003,"15min":34.523000000000003},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.9550000000000001,"5min":1.948,"15min":1.839},"min":{"1min":0.81100000000000005,"5min":0.74099999999999999,"15min":0.72999999999999998},"max":{"1min":7.8680000000000003,"5min":28.760999999999999,"15min":28.760999999999999},"last":1.597}]},{"osd":7,"last update":"Tue Jan 16 15:15:12 2024","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":2.2919999999999998,"15min":1.772},"min":{"1min":1.05,"5min":0.82099999999999995,"15min":0.64100000000000001},"max":{"1min":3.1219999999999999,"5min":22.834,"15min":22.834},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.53,"5min":2.2879999999999998,"15min":2.069},"min":{"1min":0.91000000000000003,"5min":0.872,"15min":0.82799999999999996},"max":{"1min":1.931,"5min":30.251000000000001,"15min":39.685000000000002},"last":1.653}]}]},{"osd":1,"up_from":14,"seq":60129542337,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72560,"kb_used_data":1188,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94299280,"statfs":{"total":96636764160,"available":96562462720,"internally_reserved":0,"allocated":1216512,"data_stored":525419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":1.9850000000000001,"15min":1.3260000000000001},"min":{"1min":0.48599999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.256,"5min":31.178999999999998,"15min":31.178999999999998},"last":1.1739999999999999},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.4470000000000001,"15min":1.2789999999999999},"min":{"1min":0.47299999999999998,"5min":0.247,"15min":0.247},"max":{"1min":1.901,"5min":26.859999999999999,"15min":26.859999999999999},"last":1.0169999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.1439999999999999,"15min":1.143},"min":{"1min":0.45400000000000001,"5min":0.37,"15min":0.37},"max":{"1min":3.5830000000000002,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.3520000000000001,"5min":1.9239999999999999,"15min":1.5529999999999999},"min":{"1min":0.40100000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":3.4929999999999999,"5min":27.408000000000001,"15min":27.408000000000001},"last":1.5509999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.879,"15min":1.651},"min":{"1min":0.56299999999999994,"5min":0.46300000000000002,"15min":0.443},"max":{"1min":1.8049999999999999,"5min":27.609999999999999,"15min":27.609999999999999},"last":1.669},{"interface":"front","average":{"1min":1.2609999999999999,"5min":2.1709999999999998,"15min":1.6559999999999999},"min":{"1min":0.53100000000000003,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":2.9750000000000001,"5min":75.016000000000005,"15min":75.016000000000005},"last":1.2609999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.3839999999999999,"5min":1.4390000000000001,"15min":1.466},"min":{"1min":0.98599999999999999,"5min":0.71599999999999997,"15min":0.58099999999999996},"max":{"1min":2.3069999999999999,"5min":4.7249999999999996,"15min":10.577999999999999},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.411,"5min":2.0760000000000001,"15min":1.74},"min":{"1min":0.78400000000000003,"5min":0.72199999999999998,"15min":0.67000000000000004},"max":{"1min":2.0310000000000001,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.788}]},{"osd":5,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.498,"5min":1.411,"15min":1.5449999999999999},"min":{"1min":0.76600000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":3.032,"5min":3.032,"15min":17.469000000000001},"last":1.49},{"interface":"front","average":{"1min":1.377,"5min":1.522,"15min":1.6060000000000001},"min":{"1min":1.0229999999999999,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.1859999999999999,"5min":4.6120000000000001,"15min":8.2989999999999995},"last":1.526}]},{"osd":6,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.613,"5min":1.5569999999999999,"15min":1.7150000000000001},"min":{"1min":0.85999999999999999,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.9009999999999998,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.403},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.5549999999999999,"15min":1.873},"min":{"1min":1.0800000000000001,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":2.5760000000000001,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.3129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6419999999999999,"15min":1.899},"min":{"1min":0.94799999999999995,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":2.4249999999999998,"5min":4.6989999999999998,"15min":30.834},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":2.2589999999999999,"15min":2.0609999999999999},"min":{"1min":0.96799999999999997,"5min":0.80700000000000005,"15min":0.69099999999999995},"max":{"1min":1.9890000000000001,"5min":27.716999999999999,"15min":27.716999999999999},"last":1.8999999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542335,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73152,"kb_used_data":1772,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298688,"statfs":{"total":96636764160,"available":96561856512,"internally_reserved":0,"allocated":1814528,"data_stored":1116369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.103,"15min":1.1879999999999999},"min":{"1min":0.52100000000000002,"5min":0.35199999999999998,"15min":0.32300000000000001},"max":{"1min":1.73,"5min":11.188000000000001,"15min":22.132999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.353,"15min":1.4970000000000001},"min":{"1min":0.82299999999999995,"5min":0.42499999999999999,"15min":0.33900000000000002},"max":{"1min":1.623,"5min":26.459,"15min":50.429000000000002},"last":0.71799999999999997}]},{"osd":2,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.234,"5min":1.4510000000000001,"15min":1.466},"min":{"1min":0.66100000000000003,"5min":0.42399999999999999,"15min":0.38},"max":{"1min":1.8009999999999999,"5min":20.041,"15min":22.716999999999999},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.264,"5min":1.452,"15min":1.6160000000000001},"min":{"1min":0.78400000000000003,"5min":0.503,"15min":0.38},"max":{"1min":1.7430000000000001,"5min":26.710000000000001,"15min":50.581000000000003},"last":1.0620000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2410000000000001,"15min":1.218},"min":{"1min":0.749,"5min":0.40000000000000002,"15min":0.372},"max":{"1min":1.8129999999999999,"5min":5.8719999999999999,"15min":10.795},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.425,"15min":1.5820000000000001},"min":{"1min":0.70799999999999996,"5min":0.47399999999999998,"15min":0.46700000000000003},"max":{"1min":1.8280000000000001,"5min":9.484,"15min":22.254999999999999},"last":1.462}]},{"osd":4,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.5,"15min":1.806},"min":{"1min":0.78800000000000003,"5min":0.70399999999999996,"15min":0.625},"max":{"1min":1.986,"5min":3.7749999999999999,"15min":22.407},"last":1.617},{"interface":"front","average":{"1min":1.48,"5min":1.6930000000000001,"15min":1.919},"min":{"1min":1.1970000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.907,"5min":26.57,"15min":50.497},"last":1.5780000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.401,"15min":1.6220000000000001},"min":{"1min":0.83199999999999996,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":2.0550000000000002,"5min":4.7359999999999998,"15min":14.112},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.7,"15min":1.847},"min":{"1min":1.05,"5min":0.71699999999999997,"15min":0.71399999999999997},"max":{"1min":1.9430000000000001,"5min":7.2599999999999998,"15min":22.882999999999999},"last":1.7270000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.5920000000000001,"15min":1.7769999999999999},"min":{"1min":0.871,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.968,"5min":8.2349999999999994,"15min":31.387},"last":1.671},{"interface":"front","average":{"1min":1.782,"5min":1.841,"15min":1.9650000000000001},"min":{"1min":1.01,"5min":0.79200000000000004,"15min":0.64900000000000002},"max":{"1min":3.5209999999999999,"5min":9.2840000000000007,"15min":23.547000000000001},"last":1.829}]},{"osd":7,"last update":"Tue Jan 16 15:15:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":2.0190000000000001,"15min":2.0680000000000001},"min":{"1min":1.292,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":2.21,"5min":26.649999999999999,"15min":50.542000000000002},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.8660000000000001,"15min":1.835},"min":{"1min":1.0349999999999999,"5min":0.63700000000000001,"15min":0.58399999999999996},"max":{"1min":2.1859999999999999,"5min":12.455,"15min":15.747},"last":1.778}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:15:32.613 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:15:32.613 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:15:32.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.901+0000 7f6f73e00640 1 Processor -- start 2024-01-16T15:15:32.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.901+0000 7f6f73e00640 1 -- start start 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.903+0000 7f6f73e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f7415f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.903+0000 7f6f73e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6f740adfc0 0x7f6f740cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.903+0000 7f6f73e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f740ceb80 con 0x7f6f740b47d0 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.903+0000 7f6f73e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f7415fa60 con 0x7f6f740adfc0 2024-01-16T15:15:32.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.903+0000 7f6f73e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f7415bac0 con 0x7f6f740b43f0 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.904+0000 7f6f72a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f7415f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.904+0000 7f6f72a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f7415f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:44454/0 (socket says 172.21.15.53:44454) 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.904+0000 7f6f72a00640 1 -- 172.21.15.53:0/2232680437 learned_addr learned my addr 172.21.15.53:0/2232680437 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.904+0000 7f6f73400640 1 --2- 172.21.15.53:0/2232680437 >> v2:172.21.15.146:3300/0 conn(0x7f6f740adfc0 0x7f6f740cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.905+0000 7f6f72a00640 1 -- 172.21.15.53:0/2232680437 >> v2:172.21.15.146:3300/0 conn(0x7f6f740adfc0 msgr2=0x7f6f740cc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.905+0000 7f6f72a00640 1 --2- 172.21.15.53:0/2232680437 >> v2:172.21.15.146:3300/0 conn(0x7f6f740adfc0 0x7f6f740cc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.905+0000 7f6f72a00640 1 -- 172.21.15.53:0/2232680437 >> v1:172.21.15.146:6789/0 conn(0x7f6f740b47d0 legacy=0x7f6f740ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.905+0000 7f6f72a00640 1 -- 172.21.15.53:0/2232680437 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7415bda0 con 0x7f6f740b43f0 2024-01-16T15:15:32.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.905+0000 7f6f72000640 1 -- 172.21.15.53:0/2232680437 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1690361964 0 0) 0x7f6f740ceb80 con 0x7f6f740b47d0 2024-01-16T15:15:32.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.906+0000 7f6f72a00640 1 --2- 172.21.15.53:0/2232680437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f7415f520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009870 tx=0x7f6f6402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a41a82470070db3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.907+0000 7f6f72000640 1 -- 172.21.15.53:0/2232680437 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64030840 con 0x7f6f740b43f0 2024-01-16T15:15:32.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.907+0000 7f6f72000640 1 -- 172.21.15.53:0/2232680437 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f640309a0 con 0x7f6f740b43f0 2024-01-16T15:15:32.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.907+0000 7f6f72000640 1 -- 172.21.15.53:0/2232680437 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f64030d30 con 0x7f6f740b43f0 2024-01-16T15:15:32.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 -- 172.21.15.53:0/2232680437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 msgr2=0x7f6f7415f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 --2- 172.21.15.53:0/2232680437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f7415f520 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6f64009870 tx=0x7f6f6402b980 comp rx=0 tx=0).stop 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 -- 172.21.15.53:0/2232680437 shutdown_connections 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 --2- 172.21.15.53:0/2232680437 >> v2:172.21.15.146:3300/0 conn(0x7f6f740adfc0 0x7f6f740cc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 --2- 172.21.15.53:0/2232680437 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f7415f520 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 -- 172.21.15.53:0/2232680437 >> 172.21.15.53:0/2232680437 conn(0x7f6f740b0b70 msgr2=0x7f6f740b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 -- 172.21.15.53:0/2232680437 shutdown_connections 2024-01-16T15:15:32.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.908+0000 7f6f73e00640 1 -- 172.21.15.53:0/2232680437 wait complete. 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.910+0000 7f6f73e00640 1 Processor -- start 2024-01-16T15:15:32.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.910+0000 7f6f73e00640 1 -- start start 2024-01-16T15:15:32.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.911+0000 7f6f73e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f740adfc0 0x7f6f7415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.911+0000 7f6f73e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f74155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.911+0000 7f6f73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 0x7f6f74156100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.911+0000 7f6f73e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74156640 con 0x7f6f740b47d0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.911+0000 7f6f73e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f74157190 con 0x7f6f740adfc0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.911+0000 7f6f73e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f741568f0 con 0x7f6f740b43f0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 0x7f6f74156100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 0x7f6f74156100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39528/0 (socket says 172.21.15.53:39528) 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 -- 172.21.15.53:0/3010513019 learned_addr learned my addr 172.21.15.53:0/3010513019 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f69e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f74155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f72a00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f740adfc0 0x7f6f7415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 -- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 msgr2=0x7f6f74155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f74155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 -- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f740adfc0 msgr2=0x7f6f7415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:32.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f740adfc0 0x7f6f7415d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 -- 172.21.15.53:0/3010513019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f640094d0 con 0x7f6f740b47d0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.912+0000 7f6f73400640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 0x7f6f74156100 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c007e30 tx=0x7f6f6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.913+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c014620 con 0x7f6f740b47d0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.913+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f6c00ee90 con 0x7f6f740b47d0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.913+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c0149c0 con 0x7f6f740b47d0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.913+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f74156bd0 con 0x7f6f740b47d0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.913+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f741f33e0 con 0x7f6f740b47d0 2024-01-16T15:15:32.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.914+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f74067450 con 0x7f6f740b47d0 2024-01-16T15:15:32.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.915+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f6f6c014b20 con 0x7f6f740b47d0 2024-01-16T15:15:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.916+0000 7f6f6b400640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f4c06c160 0x7f6f4c06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.916+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6f6c089380 con 0x7f6f740b47d0 2024-01-16T15:15:32.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.917+0000 7f6f72a00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f4c06c160 0x7f6f4c06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:32.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.918+0000 7f6f72a00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f4c06c160 0x7f6f4c06e620 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6f64036000 tx=0x7f6f6403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:32.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:32.920+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f6c055960 con 0x7f6f740b47d0 2024-01-16T15:15:33.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.190+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f6f7406fff0 con 0x7f6f740b47d0 2024-01-16T15:15:33.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.195+0000 7f6f6b400640 1 -- 172.21.15.53:0/3010513019 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f6f6c058f80 con 0x7f6f740b47d0 2024-01-16T15:15:33.179 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:15:33.179 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:15:33.179 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:15:33.179 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:15:33.180 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:15:33.180 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f4c06c160 msgr2=0x7f6f4c06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f4c06c160 0x7f6f4c06e620 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f6f64036000 tx=0x7f6f6403b040 comp rx=0 tx=0).stop 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 msgr2=0x7f6f74156100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 0x7f6f74156100 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c007e30 tx=0x7f6f6c00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 shutdown_connections 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f4c06c160 0x7f6f4c06e620 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f740b47d0 0x7f6f74156100 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6f740b43f0 0x7f6f74155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 --2- 172.21.15.53:0/3010513019 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f740adfc0 0x7f6f7415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 >> 172.21.15.53:0/3010513019 conn(0x7f6f740b0b70 msgr2=0x7f6f7406cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:33.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 shutdown_connections 2024-01-16T15:15:33.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.199+0000 7f6f73e00640 1 -- 172.21.15.53:0/3010513019 wait complete. 2024-01-16T15:15:33.205 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:15:33.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.441+0000 7f4282a00640 1 Processor -- start 2024-01-16T15:15:33.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.442+0000 7f4282a00640 1 -- start start 2024-01-16T15:15:33.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f4282a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f427c15d610 0x7f427c0ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:33.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f4282a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0b3400 con 0x7f427c0ced60 2024-01-16T15:15:33.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f4282a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c0b27f0 con 0x7f427c0d1ef0 2024-01-16T15:15:33.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f4282a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c0d1390 con 0x7f427c15d610 2024-01-16T15:15:33.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f427be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f427c15d610 0x7f427c0ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:33.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f427b400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f427c0ced60 0x7f427c0d17e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54582/0 (socket says 172.21.15.53:54582) 2024-01-16T15:15:33.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.443+0000 7f427b400640 1 -- 172.21.15.53:0/778763455 learned_addr learned my addr 172.21.15.53:0/778763455 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:33.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.444+0000 7f427be00640 1 -- 172.21.15.53:0/778763455 >> v1:172.21.15.146:6789/0 conn(0x7f427c0d1ef0 legacy=0x7f427c0b3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:33.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.444+0000 7f427be00640 1 -- 172.21.15.53:0/778763455 >> v1:172.21.15.53:6789/0 conn(0x7f427c0ced60 legacy=0x7f427c0d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.444+0000 7f427be00640 1 -- 172.21.15.53:0/778763455 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c0b2ad0 con 0x7f427c15d610 2024-01-16T15:15:33.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.444+0000 7f427aa00640 1 -- 172.21.15.53:0/778763455 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801931631 0 0) 0x7f427c0b3400 con 0x7f427c0ced60 2024-01-16T15:15:33.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.445+0000 7f427be00640 1 --2- 172.21.15.53:0/778763455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c15d610 0x7f427c0ce820 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f426c009d90 tx=0x7f426c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a26e0d01c47a3abc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:33.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.445+0000 7f427aa00640 1 -- 172.21.15.53:0/778763455 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c02cc60 con 0x7f427c15d610 2024-01-16T15:15:33.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.445+0000 7f427aa00640 1 -- 172.21.15.53:0/778763455 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f426c02cdc0 con 0x7f427c15d610 2024-01-16T15:15:33.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.445+0000 7f427aa00640 1 -- 172.21.15.53:0/778763455 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f426c034660 con 0x7f427c15d610 2024-01-16T15:15:33.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 -- 172.21.15.53:0/778763455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c15d610 msgr2=0x7f427c0ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 --2- 172.21.15.53:0/778763455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c15d610 0x7f427c0ce820 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f426c009d90 tx=0x7f426c02beb0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 -- 172.21.15.53:0/778763455 shutdown_connections 2024-01-16T15:15:33.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 --2- 172.21.15.53:0/778763455 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c15d610 0x7f427c0ce820 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 -- 172.21.15.53:0/778763455 >> 172.21.15.53:0/778763455 conn(0x7f427c09c540 msgr2=0x7f427c0d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:33.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 -- 172.21.15.53:0/778763455 shutdown_connections 2024-01-16T15:15:33.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.446+0000 7f4282a00640 1 -- 172.21.15.53:0/778763455 wait complete. 2024-01-16T15:15:33.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.448+0000 7f4282a00640 1 Processor -- start 2024-01-16T15:15:33.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.449+0000 7f4282a00640 1 -- start start 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f4282a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c0ced60 0x7f427c0ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f4282a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 0x7f427c1f6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f4282a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c15d610 0x7f427c0b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f4282a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1f2ff0 con 0x7f427c0d1ef0 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f4282a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1f6850 con 0x7f427c15d610 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f4282a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1f2cd0 con 0x7f427c0ced60 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f427b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 0x7f427c1f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f427b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 0x7f427c1f6310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39542/0 (socket says 172.21.15.53:39542) 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f427b400640 1 -- 172.21.15.53:0/1708023306 learned_addr learned my addr 172.21.15.53:0/1708023306 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.450+0000 7f427be00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c0ced60 0x7f427c0ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f4280a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c15d610 0x7f427c0b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:33.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f427b400640 1 -- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c0ced60 msgr2=0x7f427c0ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f427b400640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c0ced60 0x7f427c0ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f427b400640 1 -- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c15d610 msgr2=0x7f427c0b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f427b400640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c15d610 0x7f427c0b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f427b400640 1 -- 172.21.15.53:0/1708023306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f426c009a70 con 0x7f427c0d1ef0 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f4280a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c15d610 0x7f427c0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f427b400640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 0x7f427c1f6310 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f426400c500 tx=0x7f426400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264007560 con 0x7f427c0d1ef0 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42640047e0 con 0x7f427c0d1ef0 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4264004b90 con 0x7f427c0d1ef0 2024-01-16T15:15:33.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.451+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427c0cb8b0 con 0x7f427c0d1ef0 2024-01-16T15:15:33.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.452+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c1f6d60 con 0x7f427c0d1ef0 2024-01-16T15:15:33.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.453+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427c1f3130 con 0x7f427c0d1ef0 2024-01-16T15:15:33.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.454+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4264004cf0 con 0x7f427c0d1ef0 2024-01-16T15:15:33.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.455+0000 7f4273e00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f425406c160 0x7f425406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.455+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4264087ae0 con 0x7f427c0d1ef0 2024-01-16T15:15:33.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.456+0000 7f427be00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f425406c160 0x7f425406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:33.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.457+0000 7f427be00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f425406c160 0x7f425406e620 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f426c036000 tx=0x7f426c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:33.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.460+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f427c1f3130 con 0x7f427c0d1ef0 2024-01-16T15:15:33.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.692+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f427c1f3130 con 0x7f427c0d1ef0 2024-01-16T15:15:33.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.693+0000 7f4273e00640 1 -- 172.21.15.53:0/1708023306 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 64 v64) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f427c1f3130 con 0x7f427c0d1ef0 2024-01-16T15:15:33.678 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:15:33.678 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":64,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1983839723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1983839723},{"type":"v1","addr":"172.21.15.146:6835","nonce":1983839723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":7048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1926242315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1926242315},{"type":"v1","addr":"172.21.15.146:6837","nonce":1926242315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2218545626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2218545626},{"type":"v1","addr":"172.21.15.53:6835","nonce":2218545626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60}],"filesystems":[{"mdsmap":{"epoch":64,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:15:21.959064+0000","modified":"2024-01-16T15:15:25.069971+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8508},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8508":{"gid":8508,"name":"c","rank":0,"incarnation":61,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/1250261984","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":1250261984},{"type":"v1","addr":"172.21.15.53:6834","nonce":1250261984}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-01-16T15:15:33.678 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 64 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.696+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f425406c160 msgr2=0x7f425406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.696+0000 7f4282a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f425406c160 0x7f425406e620 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f426c036000 tx=0x7f426c030000 comp rx=0 tx=0).stop 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.696+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 msgr2=0x7f427c1f6310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.696+0000 7f4282a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 0x7f427c1f6310 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f426400c500 tx=0x7f426400c9c0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 shutdown_connections 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f425406c160 0x7f425406e620 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f427c15d610 0x7f427c0b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f427c0d1ef0 0x7f427c1f6310 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 --2- 172.21.15.53:0/1708023306 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f427c0ced60 0x7f427c0ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:33.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 >> 172.21.15.53:0/1708023306 conn(0x7f427c09c540 msgr2=0x7f427c06c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:33.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 shutdown_connections 2024-01-16T15:15:33.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:33.697+0000 7f4282a00640 1 -- 172.21.15.53:0/1708023306 wait complete. 2024-01-16T15:15:34.102 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 64, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:15:21.959064+0000', 'modified': '2024-01-16T15:15:25.069971+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8508}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8508': {'gid': 8508, 'name': 'c', 'rank': 0, 'incarnation': 61, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6834/1250261984', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6832', 'nonce': 1250261984}, {'type': 'v1', 'addr': '172.21.15.53:6834', 'nonce': 1250261984}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:15:34.103 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:15:34.103 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:15:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.351+0000 7f9a0b400640 1 Processor -- start 2024-01-16T15:15:34.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.351+0000 7f9a0b400640 1 -- start start 2024-01-16T15:15:34.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.353+0000 7f9a0b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a0406d6b0 0x7f9a0406a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.353+0000 7f9a0b400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a0406aad0 0x7f9a040ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.353+0000 7f9a0b400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a04072080 con 0x7f9a04144650 2024-01-16T15:15:34.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.353+0000 7f9a0b400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a04071220 con 0x7f9a0406aad0 2024-01-16T15:15:34.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.353+0000 7f9a0b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0406af30 con 0x7f9a0406d6b0 2024-01-16T15:15:34.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.353+0000 7f9a0a000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9a04144650 0x7f9a0406cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54588/0 (socket says 172.21.15.53:54588) 2024-01-16T15:15:34.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.354+0000 7f9a0a000640 1 -- 172.21.15.53:0/628869751 learned_addr learned my addr 172.21.15.53:0/628869751 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:34.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.354+0000 7f9a09600640 1 --2- 172.21.15.53:0/628869751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a0406d6b0 0x7f9a0406a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.354+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/628869751 >> v2:172.21.15.146:3300/0 conn(0x7f9a0406aad0 0x7f9a040ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.354+0000 7f9a08c00640 1 -- 172.21.15.53:0/628869751 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968058033 0 0) 0x7f9a04072080 con 0x7f9a04144650 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.354+0000 7f9a08c00640 1 -- 172.21.15.53:0/628869751 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e4003520 con 0x7f9a04144650 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.355+0000 7f9a09600640 1 -- 172.21.15.53:0/628869751 >> v2:172.21.15.146:3300/0 conn(0x7f9a0406aad0 msgr2=0x7f9a040ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.355+0000 7f9a09600640 1 --2- 172.21.15.53:0/628869751 >> v2:172.21.15.146:3300/0 conn(0x7f9a0406aad0 0x7f9a040ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.355+0000 7f9a09600640 1 -- 172.21.15.53:0/628869751 >> v1:172.21.15.53:6789/0 conn(0x7f9a04144650 legacy=0x7f9a0406cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.355+0000 7f9a09600640 1 -- 172.21.15.53:0/628869751 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a04071470 con 0x7f9a0406d6b0 2024-01-16T15:15:34.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.356+0000 7f9a09600640 1 --2- 172.21.15.53:0/628869751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a0406d6b0 0x7f9a0406a590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13ae459a5dfe8400 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.356+0000 7f9a08c00640 1 -- 172.21.15.53:0/628869751 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec02d040 con 0x7f9a0406d6b0 2024-01-16T15:15:34.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.356+0000 7f9a08c00640 1 -- 172.21.15.53:0/628869751 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec031810 con 0x7f9a0406d6b0 2024-01-16T15:15:34.340 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.356+0000 7f9a08c00640 1 -- 172.21.15.53:0/628869751 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec031b80 con 0x7f9a0406d6b0 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 -- 172.21.15.53:0/628869751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a0406d6b0 msgr2=0x7f9a0406a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 --2- 172.21.15.53:0/628869751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a0406d6b0 0x7f9a0406a590 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec02b8b0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 -- 172.21.15.53:0/628869751 shutdown_connections 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 --2- 172.21.15.53:0/628869751 >> v2:172.21.15.146:3300/0 conn(0x7f9a0406aad0 0x7f9a040ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 --2- 172.21.15.53:0/628869751 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a0406d6b0 0x7f9a0406a590 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 -- 172.21.15.53:0/628869751 >> 172.21.15.53:0/628869751 conn(0x7f9a040b1a50 msgr2=0x7f9a040b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 -- 172.21.15.53:0/628869751 shutdown_connections 2024-01-16T15:15:34.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.357+0000 7f9a0b400640 1 -- 172.21.15.53:0/628869751 wait complete. 2024-01-16T15:15:34.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.359+0000 7f9a0b400640 1 Processor -- start 2024-01-16T15:15:34.343 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.359+0000 7f9a0b400640 1 -- start start 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a0b400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a0406aad0 0x7f9a0409a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a0b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 0x7f9a0413e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a0b400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a04144650 0x7f9a0415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a0b400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0409c4d0 con 0x7f9a0406d6b0 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a0b400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0409c720 con 0x7f9a0406aad0 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a0b400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0409ca00 con 0x7f9a04144650 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 0x7f9a0413e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 0x7f9a0413e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39548/0 (socket says 172.21.15.53:39548) 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.360+0000 7f9a09600640 1 -- 172.21.15.53:0/3164547845 learned_addr learned my addr 172.21.15.53:0/3164547845 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a04144650 0x7f9a0415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a09600640 1 -- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a04144650 msgr2=0x7f9a0415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a0a000640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a0406aad0 0x7f9a0409a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a09600640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a04144650 0x7f9a0415a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a09600640 1 -- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a0406aad0 msgr2=0x7f9a0409a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a09600640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a0406aad0 0x7f9a0409a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a09600640 1 -- 172.21.15.53:0/3164547845 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec009400 con 0x7f9a0406d6b0 2024-01-16T15:15:34.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a04144650 0x7f9a0415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:34.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a09600640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 0x7f9a0413e090 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f99ec000c00 tx=0x7f99ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a02000640 1 -- 172.21.15.53:0/3164547845 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec003820 con 0x7f9a0406d6b0 2024-01-16T15:15:34.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a02000640 1 -- 172.21.15.53:0/3164547845 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec03f070 con 0x7f9a0406d6b0 2024-01-16T15:15:34.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.361+0000 7f9a02000640 1 -- 172.21.15.53:0/3164547845 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec003bc0 con 0x7f9a0406d6b0 2024-01-16T15:15:34.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.362+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0409cce0 con 0x7f9a0406d6b0 2024-01-16T15:15:34.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.362+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0409d2a0 con 0x7f9a0406d6b0 2024-01-16T15:15:34.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.364+0000 7f9a02000640 1 -- 172.21.15.53:0/3164547845 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f99ec003d20 con 0x7f9a0406d6b0 2024-01-16T15:15:34.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.365+0000 7f9a02000640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99dc06c190 0x7f99dc06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.365+0000 7f9a02000640 1 -- 172.21.15.53:0/3164547845 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99ec041070 con 0x7f9a0406d6b0 2024-01-16T15:15:34.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.366+0000 7f9a0a000640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99dc06c190 0x7f99dc06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.366+0000 7f9a0f7f9740 1 Processor -- start 2024-01-16T15:15:34.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.366+0000 7f9a0f7f9740 1 -- start start 2024-01-16T15:15:34.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517f8130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0a000640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99dc06c190 0x7f99dc06e650 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f99fc0012e0 tx=0x7f99fc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 0x561e517cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ce1b0 0x561e51810730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e5172b880 con 0x561e517c7d70 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e5173aae0 con 0x561e517ce1b0 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e517381d0 con 0x561e517f7d50 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 0x561e517cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 0x561e517cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39564/0 (socket says 172.21.15.53:39564) 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 -- 172.21.15.53:0/855120481 learned_addr learned my addr 172.21.15.53:0/855120481 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0a000640 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517f8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 -- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 msgr2=0x561e517f8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517f8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ce1b0 0x561e51810730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 -- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ce1b0 msgr2=0x561e51810730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.367+0000 7f9a09600640 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ce1b0 0x561e51810730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.368+0000 7f9a09600640 1 -- 172.21.15.53:0/855120481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e516fb860 con 0x561e517c7d70 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.368+0000 7f9a09600640 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 0x561e517cdc70 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec07ad90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e32770dad892cb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.368+0000 7f99faa00640 1 -- 172.21.15.53:0/855120481 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec07b190 con 0x561e517c7d70 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.368+0000 7f99faa00640 1 -- 172.21.15.53:0/855120481 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec07f2e0 con 0x561e517c7d70 2024-01-16T15:15:34.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.368+0000 7f99faa00640 1 -- 172.21.15.53:0/855120481 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec083490 con 0x561e517c7d70 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/855120481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 msgr2=0x561e517cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 0x561e517cdc70 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f99ec0097a0 tx=0x7f99ec07ad90 comp rx=0 tx=0).stop 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/855120481 shutdown_connections 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ce1b0 0x561e51810730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517c7d70 0x561e517cdc70 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/855120481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517f8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/855120481 >> 172.21.15.53:0/855120481 conn(0x561e5178ec70 msgr2=0x561e51812b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/855120481 shutdown_connections 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.369+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/855120481 wait complete. 2024-01-16T15:15:34.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.370+0000 7f9a0f7f9740 1 -- start start 2024-01-16T15:15:34.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.370+0000 7f9a0f7f9740 1 Processor -- start 2024-01-16T15:15:34.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e517e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517ea150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517ecfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e51733ac0 con 0x561e517ecb20 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e516d02f0 con 0x561e517c7d70 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e516edc70 con 0x561e517f7d50 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0f7f9740 10 client.0 _ll_register_callbacks cb 0x561e516f23b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517ecfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517ecfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39566/0 (socket says 172.21.15.53:39566) 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a0aa00640 1 -- 172.21.15.53:0/401938400 learned_addr learned my addr 172.21.15.53:0/401938400 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.371+0000 7f9a09600640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517ea150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0a000640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e517e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0aa00640 1 -- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 msgr2=0x561e517ea150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517ea150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0aa00640 1 -- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 msgr2=0x561e517e8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e517e8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0aa00640 1 -- 172.21.15.53:0/401938400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0409d2a0 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a09600640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517ea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517ecfa0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009f90 tx=0x7f99f40116f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74845a38686ebd98 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.372+0000 7f99f3400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f99f3400640 1 -- 172.21.15.53:0/401938400 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4015870 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561e5172caa0 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f99f3400640 1 -- 172.21.15.53:0/401938400 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99f40159d0 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f99f3400640 1 -- 172.21.15.53:0/401938400 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99f4011ea0 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 10 client.8667 fetch_fsmap learned FSMap version 64 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x561e51732140 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f99f3400640 1 -- 172.21.15.53:0/401938400 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561e5172caa0 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f99f3400640 1 -- 172.21.15.53:0/401938400 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f99f4015b90 con 0x561e517ecb20 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 10 client.8667 fetch_fsmap finished waiting for FSMap version 64 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 10 client.8667 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 20 client.8667 populate_metadata read hostname 'smithi053' 2024-01-16T15:15:34.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e518325b0 0x561e518349a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 4 client.8667 mds_command: new command op to 8508 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:15:34.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.373+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 --> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561e5172c7c0 con 0x561e518325b0 2024-01-16T15:15:34.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.374+0000 7f9a0a000640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e518325b0 0x561e518349a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.374+0000 7f9a0a000640 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e518325b0 0x561e518349a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccc091e3ba54f8c6 server_cookie=5711151ee8c057de in_seq=0 out_seq=0 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.374+0000 7f99f3400640 10 client.8667 ms_handle_connect on v2:172.21.15.53:6832/1250261984 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f99f3400640 1 -- 172.21.15.53:0/401938400 <== mds.0 v2:172.21.15.53:6832/1250261984 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561e5172c7c0 con 0x561e518325b0 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f99f3400640 10 client.8667 handle_command_reply: tid=0 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 client.8667 shutdown 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 msgr2=0x561e517ecfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517ecfa0 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009f90 tx=0x7f99f40116f0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 shutdown_connections 2024-01-16T15:15:34.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e518325b0 0x561e518349a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517ecfa0 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e517ea150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.375+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/401938400 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e517e8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.376+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 >> 172.21.15.53:0/401938400 conn(0x561e5178ec70 msgr2=0x561e517ccc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.376+0000 7f99f3400640 0 client.8667 ms_handle_reset on v2:172.21.15.53:6832/1250261984 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.376+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 shutdown_connections 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.376+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/401938400 wait complete. 2024-01-16T15:15:34.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.376+0000 7f9a0f7f9740 20 client.8667 trim_cache size 0 max 16384 2024-01-16T15:15:34.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.393+0000 7f9a0f7f9740 1 Processor -- start 2024-01-16T15:15:34.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.393+0000 7f9a0f7f9740 1 -- start start 2024-01-16T15:15:34.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517c7d70 0x561e517f51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ecb20 0x561e5180c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 0x561e517f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e51496f20 con 0x561e517f7d50 2024-01-16T15:15:34.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e517c40b0 con 0x561e517ecb20 2024-01-16T15:15:34.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e516e7e30 con 0x561e517c7d70 2024-01-16T15:15:34.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.394+0000 7f9a0aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 0x561e517f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 0x561e517f00d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39572/0 (socket says 172.21.15.53:39572) 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 -- 172.21.15.53:0/3604557582 learned_addr learned my addr 172.21.15.53:0/3604557582 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0a000640 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517c7d70 0x561e517f51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 -- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517c7d70 msgr2=0x561e517f51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517c7d70 0x561e517f51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 -- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ecb20 msgr2=0x561e5180c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ecb20 0x561e5180c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 -- 172.21.15.53:0/3604557582 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561e516cfb80 con 0x561e517f7d50 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0a000640 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517c7d70 0x561e517f51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.395+0000 7f9a0aa00640 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 0x561e517f00d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f99f4000c00 tx=0x7f99f4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a5e04073b3f6a8e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.396+0000 7f99fb400640 1 -- 172.21.15.53:0/3604557582 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4009aa0 con 0x561e517f7d50 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.396+0000 7f99fb400640 1 -- 172.21.15.53:0/3604557582 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f4011ea0 con 0x561e517f7d50 2024-01-16T15:15:34.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.396+0000 7f99fb400640 1 -- 172.21.15.53:0/3604557582 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f4004280 con 0x561e517f7d50 2024-01-16T15:15:34.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.396+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/3604557582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 msgr2=0x561e517f00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.396+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 0x561e517f00d0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f99f4000c00 tx=0x7f99f4016000 comp rx=0 tx=0).stop 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/3604557582 shutdown_connections 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517f7d50 0x561e517f00d0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517ecb20 0x561e5180c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/3604557582 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517c7d70 0x561e517f51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/3604557582 >> 172.21.15.53:0/3604557582 conn(0x561e5178ec70 msgr2=0x561e517e9f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/3604557582 shutdown_connections 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/3604557582 wait complete. 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.397+0000 7f9a0f7f9740 1 -- start start 2024-01-16T15:15:34.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.398+0000 7f9a0f7f9740 1 Processor -- start 2024-01-16T15:15:34.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e518819c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517f43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e5181e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e517cc270 con 0x561e517ecb20 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x561e51838d70 con 0x561e517c7d70 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x561e51838fc0 con 0x561e517f7d50 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a0f7f9740 10 client.0 _ll_register_callbacks cb 0x7f99f40094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517f43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517f43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39584/0 (socket says 172.21.15.53:39584) 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 -- 172.21.15.53:0/2847258296 learned_addr learned my addr 172.21.15.53:0/2847258296 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 -- 172.21.15.53:0/2847258296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 msgr2=0x561e5181e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e5181e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 -- 172.21.15.53:0/2847258296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 msgr2=0x561e518819c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e518819c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 -- 172.21.15.53:0/2847258296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561e51732140 con 0x561e517ecb20 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f9a09600640 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517f43a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f99ec077640 tx=0x7f99ec006c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b8b259c0a79072e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.399+0000 7f99f3400640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:15:34.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f99f3400640 1 -- 172.21.15.53:0/2847258296 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec07a880 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f99f3400640 1 -- 172.21.15.53:0/2847258296 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f99ec07b190 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99ec07f2e0 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f99f3400640 1 -- 172.21.15.53:0/2847258296 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=64) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99ec0ab180 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 10 client.8673 fetch_fsmap learned FSMap version 64 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=64,monmap=2+}) v3 -- 0x561e516f5170 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f99f3400640 1 -- 172.21.15.53:0/2847258296 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec07f2e0 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f99f3400640 1 -- 172.21.15.53:0/2847258296 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 64) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f99ec0aaa60 con 0x561e517ecb20 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 10 client.8673 fetch_fsmap finished waiting for FSMap version 64 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 10 client.8673 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 20 client.8673 populate_metadata read hostname 'smithi053' 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e517fa800 0x561e517fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 4 client.8673 mds_command: new command op to 8508 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.400+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 --> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561e514346c0 con 0x561e517fa800 2024-01-16T15:15:34.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.401+0000 7f9a0a000640 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e517fa800 0x561e517fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.401+0000 7f9a0a000640 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e517fa800 0x561e517fabe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eada054f73da1c1 server_cookie=840dd4ccbfe67b9c in_seq=0 out_seq=0 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.401+0000 7f99f3400640 10 client.8673 ms_handle_connect on v2:172.21.15.53:6832/1250261984 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.401+0000 7f99f3400640 1 -- 172.21.15.53:0/2847258296 <== mds.0 v2:172.21.15.53:6832/1250261984 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x561e514346c0 con 0x561e517fa800 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.401+0000 7f99f3400640 10 client.8673 handle_command_reply: tid=0 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.401+0000 7f9a0f7f9740 1 client.8673 shutdown 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 msgr2=0x561e517f43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517f43a0 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f99ec077640 tx=0x7f99ec006c30 comp rx=0 tx=0).stop 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 shutdown_connections 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:6832/1250261984,v1:172.21.15.53:6834/1250261984] conn(0x561e517fa800 0x561e517fabe0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x561e517f7d50 0x561e5181e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x561e517ecb20 0x561e517f43a0 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 --2- 172.21.15.53:0/2847258296 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x561e517c7d70 0x561e518819c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f99f3400640 0 client.8673 ms_handle_reset on v2:172.21.15.53:6832/1250261984 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 >> 172.21.15.53:0/2847258296 conn(0x561e5178ec70 msgr2=0x561e517f50f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 shutdown_connections 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 1 -- 172.21.15.53:0/2847258296 wait complete. 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.402+0000 7f9a0f7f9740 20 client.8673 trim_cache size 0 max 16384 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 8508, 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:15:34.389 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:15:34.390 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 11.900354194, 2024-01-16T15:15:34.390 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 64, 2024-01-16T15:15:34.390 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 99, 2024-01-16T15:15:34.390 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 99, 2024-01-16T15:15:34.390 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 17.228892206000001 2024-01-16T15:15:34.390 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99dc06c190 msgr2=0x7f99dc06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99dc06c190 0x7f99dc06e650 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f99fc0012e0 tx=0x7f99fc009040 comp rx=0 tx=0).stop 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 msgr2=0x7f9a0413e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 0x7f9a0413e090 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f99ec000c00 tx=0x7f99ec0046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 shutdown_connections 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99dc06c190 0x7f99dc06e650 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a04144650 0x7f9a0415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a0406d6b0 0x7f9a0413e090 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 --2- 172.21.15.53:0/3164547845 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a0406aad0 0x7f9a0409a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 >> 172.21.15.53:0/3164547845 conn(0x7f9a040b1a50 msgr2=0x7f9a0406f600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 shutdown_connections 2024-01-16T15:15:34.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.403+0000 7f9a0b400640 1 -- 172.21.15.53:0/3164547845 wait complete. 2024-01-16T15:15:34.404 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:34.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.632+0000 7f3af1600640 1 Processor -- start 2024-01-16T15:15:34.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.633+0000 7f3af1600640 1 -- start start 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.634+0000 7f3af1600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 0x7f3aec0b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.634+0000 7f3af1600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3aec0b4140 0x7f3aec0d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.634+0000 7f3af1600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aec0c7bf0 con 0x7f3aec0d1b90 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.634+0000 7f3af1600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aec0d2020 con 0x7f3aec0b4140 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.634+0000 7f3af1600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0d0420 con 0x7f3aec0b34e0 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.634+0000 7f3af0c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3aec0d1b90 0x7f3aec0cdc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54602/0 (socket says 172.21.15.53:54602) 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3af0c00640 1 -- 172.21.15.53:0/2701457055 learned_addr learned my addr 172.21.15.53:0/2701457055 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeb400640 1 --2- 172.21.15.53:0/2701457055 >> v2:172.21.15.146:3300/0 conn(0x7f3aec0b4140 0x7f3aec0d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aebe00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 0x7f3aec0b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879417817 0 0) 0x7f3aec0c7bf0 con 0x7f3aec0d1b90 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3acc003520 con 0x7f3aec0d1b90 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773995848 0 0) 0x7f3acc003520 con 0x7f3aec0d1b90 2024-01-16T15:15:34.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 msgr2=0x7f3aec0b3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 0x7f3aec0b3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 >> v2:172.21.15.146:3300/0 conn(0x7f3aec0b4140 msgr2=0x7f3aec0d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 --2- 172.21.15.53:0/2701457055 >> v2:172.21.15.146:3300/0 conn(0x7f3aec0b4140 0x7f3aec0d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.635+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec0d0e70 con 0x7f3aec0d1b90 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3ae4001670 con 0x7f3aec0d1b90 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 >> v1:172.21.15.53:6789/0 conn(0x7f3aec0d1b90 legacy=0x7f3aec0cdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 0x7f3acc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3acc003f40 0x7f3acc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3acc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0d2020 con 0x7f3acc004d00 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec0d0420 con 0x7f3acc003f40 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3acc00b980 con 0x7f3aec0b34e0 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3af0c00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3acc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.636+0000 7f3aebe00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 0x7f3acc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.637+0000 7f3aeb400640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3acc003f40 0x7f3acc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.637+0000 7f3af0c00640 1 -- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 msgr2=0x7f3acc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.637+0000 7f3af0c00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec0b34e0 0x7f3acc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.637+0000 7f3af0c00640 1 -- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3acc003f40 msgr2=0x7f3acc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.637+0000 7f3af0c00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3acc003f40 0x7f3acc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.637+0000 7f3af0c00640 1 -- 172.21.15.53:0/2701457055 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3acc003520 con 0x7f3acc004d00 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.638+0000 7f3af0c00640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3acc005180 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c140 tx=0x7f3acc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f71e65327c95e7b7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.638+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4005ad0 con 0x7f3acc004d00 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.638+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4005c50 con 0x7f3acc004d00 2024-01-16T15:15:34.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.638+0000 7f3aeaa00640 1 -- 172.21.15.53:0/2701457055 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae402c810 con 0x7f3acc004d00 2024-01-16T15:15:34.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af1600640 1 -- 172.21.15.53:0/2701457055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 msgr2=0x7f3acc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af1600640 1 --2- 172.21.15.53:0/2701457055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3acc005180 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c140 tx=0x7f3acc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af0c00640 1 -- 172.21.15.53:0/2701457055 reap_dead start 2024-01-16T15:15:34.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af1600640 1 -- 172.21.15.53:0/2701457055 shutdown_connections 2024-01-16T15:15:34.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af1600640 1 -- 172.21.15.53:0/2701457055 >> 172.21.15.53:0/2701457055 conn(0x7f3aec0d3090 msgr2=0x7f3aec0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af1600640 1 -- 172.21.15.53:0/2701457055 shutdown_connections 2024-01-16T15:15:34.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.639+0000 7f3af1600640 1 -- 172.21.15.53:0/2701457055 wait complete. 2024-01-16T15:15:34.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.641+0000 7f3af1600640 1 Processor -- start 2024-01-16T15:15:34.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.641+0000 7f3af1600640 1 -- start start 2024-01-16T15:15:34.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3af1600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3aec1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3af1600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3aec1526b0 0x7f3aec15db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3af1600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec15c800 0x7f3aec15cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3af1600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15d1c0 con 0x7f3acc004d00 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3af1600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15e0d0 con 0x7f3aec1526b0 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3af1600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec15e370 con 0x7f3aec15c800 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3aebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3aec1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3aebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3aec1601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39594/0 (socket says 172.21.15.53:39594) 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.642+0000 7f3aebe00640 1 -- 172.21.15.53:0/1848652905 learned_addr learned my addr 172.21.15.53:0/1848652905 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3aebe00640 1 -- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec15c800 msgr2=0x7f3aec15cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3af0c00640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec15c800 0x7f3aec15cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3aebe00640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec15c800 0x7f3aec15cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3aebe00640 1 -- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3aec1526b0 msgr2=0x7f3aec15db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3aebe00640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3aec1526b0 0x7f3aec15db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3aebe00640 1 -- 172.21.15.53:0/1848652905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec15e650 con 0x7f3acc004d00 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3aebe00640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3aec1601f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3adc011600 tx=0x7f3adc011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc00f920 con 0x7f3acc004d00 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3adc00fa80 con 0x7f3acc004d00 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3adc00fdf0 con 0x7f3acc004d00 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.643+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec15e930 con 0x7f3acc004d00 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.644+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec157da0 con 0x7f3acc004d00 2024-01-16T15:15:34.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.645+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec067450 con 0x7f3acc004d00 2024-01-16T15:15:34.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.646+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f3adc015460 con 0x7f3acc004d00 2024-01-16T15:15:34.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.647+0000 7f3ae3e00640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac406c190 0x7f3ac406e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:34.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.647+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3adc088b60 con 0x7f3acc004d00 2024-01-16T15:15:34.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.647+0000 7f3aeb400640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac406c190 0x7f3ac406e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:34.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.649+0000 7f3aeb400640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac406c190 0x7f3ac406e650 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3aec15d7c0 tx=0x7f3ad4009750 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:34.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.651+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3adc055420 con 0x7f3acc004d00 2024-01-16T15:15:34.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.885+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3aec066720 con 0x7f3acc004d00 2024-01-16T15:15:34.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.886+0000 7f3ae3e00640 1 -- 172.21.15.53:0/1848652905 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7f3adc058a40 con 0x7f3acc004d00 2024-01-16T15:15:34.870 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:34.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac406c190 msgr2=0x7f3ac406e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac406c190 0x7f3ac406e650 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3aec15d7c0 tx=0x7f3ad4009750 comp rx=0 tx=0).stop 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 msgr2=0x7f3aec1601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3aec1601f0 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f3adc011600 tx=0x7f3adc011ac0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 shutdown_connections 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac406c190 0x7f3ac406e650 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3aec15c800 0x7f3aec15cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3aec1526b0 0x7f3aec15db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 --2- 172.21.15.53:0/1848652905 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3acc004d00 0x7f3aec1601f0 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 >> 172.21.15.53:0/1848652905 conn(0x7f3aec0d3090 msgr2=0x7f3aec0c6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.889+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 shutdown_connections 2024-01-16T15:15:34.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:34.890+0000 7f3af1600640 1 -- 172.21.15.53:0/1848652905 wait complete. 2024-01-16T15:15:34.892 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:34.892 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:34.892 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:15:34.893 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:15:34.893 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:34.893 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:34.893 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:34.893 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:34.893 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:34.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:34.893 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:15:34.949 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:34.949 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:34.949 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:34.949 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.949 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 41715sec preferred_lft 41715sec 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:15:34.950 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:34.951 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:34.952 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:15:34.952 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:34.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:34.953 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:35.048 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:15:35.053 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:35.053 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:15:35.108 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:15:35.109 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:35.109 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:15:35.109 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:15:35.109 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:35.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:35.109 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:15:35.109 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:15:35.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:15:35.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:15:35.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:15:35.110 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:15:35.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:35.306 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:15:35.360 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:15:35.360 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:15:35.361 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:35.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:35.362 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:15:35.463 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:15:35.463 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:35.463 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:35.518 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:15:35.518 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:35.518 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:35.567 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:15:35.658 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:35.716 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:15:35.716 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:15:35.716 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:15:35.716 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:15:35.716 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:35.717 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:35.718 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:35.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:35.719 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:15:35.776 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:35.776 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:15:35.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:15:35.863 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:15:35.904 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:15:35.905 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:35.906 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:15:35.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:35.907 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:15:35.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:15:35.992+0000 7f04bb042480 -1 init, newargv = 0x564200371960 newargc=15 2024-01-16T15:15:35.976 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[72083]: starting ceph client 2024-01-16T15:15:35.986 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[72083]: starting fuse 2024-01-16T15:15:35.994 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:15:35.995 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:15:35.995 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:15:35.996 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:15:36.198 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.72083.asok 2024-01-16T15:15:36.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:36.217 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.72083.asok 2024-01-16T15:15:36.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:36.218 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.72083.asok status 2024-01-16T15:15:36.463 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:15:36.463 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:15:36.463 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:36.463 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:36.463 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "72083", 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 8685, 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 8685 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:15:36.464 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 995874811 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 995874811 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.8685 192.168.0.1:0/995874811", 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/995874811", 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 64, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 99, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:15:36.465 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:15:36.466 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:15:36.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:36.480 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:15:36.480 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:36.538 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:15:36.538 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:36.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:36.539 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:15:36.641 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:15:36.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.878+0000 7f6997c00640 1 Processor -- start 2024-01-16T15:15:36.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.878+0000 7f6997c00640 1 -- start start 2024-01-16T15:15:36.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.880+0000 7f6997c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 0x7f69900cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.880+0000 7f6997c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699015deb0 con 0x7f6990071200 2024-01-16T15:15:36.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.881+0000 7f6997c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699015e190 con 0x7f69901618f0 2024-01-16T15:15:36.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.881+0000 7f6997c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699015ea30 con 0x7f6990157c30 2024-01-16T15:15:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.881+0000 7f6995c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6990071200 0x7f699006dcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54608/0 (socket says 172.21.15.53:54608) 2024-01-16T15:15:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.881+0000 7f6995c00640 1 -- 172.21.15.53:0/1338736780 learned_addr learned my addr 172.21.15.53:0/1338736780 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.881+0000 7f6995200640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 0x7f69900cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180065572 0 0) 0x7f699015deb0 con 0x7f6990071200 2024-01-16T15:15:36.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6970003520 con 0x7f6990071200 2024-01-16T15:15:36.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 336502152 0 0) 0x7f6970003520 con 0x7f6990071200 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 msgr2=0x7f69900cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 0x7f69900cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 >> v1:172.21.15.146:6789/0 conn(0x7f69901618f0 legacy=0x7f6990070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699015e470 con 0x7f6990071200 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6988001690 con 0x7f6990071200 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 >> v1:172.21.15.53:6789/0 conn(0x7f6990071200 legacy=0x7f699006dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 0x7f69700039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6970003f70 0x7f69700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f6970005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699015ea30 con 0x7f6970004d00 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f697000bac0 con 0x7f6970003f70 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.882+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f697000bc80 con 0x7f6990157c30 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995c00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f6970005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6987c00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6970003f70 0x7f69700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995200640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 0x7f69700039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995c00640 1 -- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 msgr2=0x7f69700039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995c00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6990157c30 0x7f69700039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995c00640 1 -- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6970003f70 msgr2=0x7f69700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995c00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6970003f70 0x7f69700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.883+0000 7f6995c00640 1 -- 172.21.15.53:0/1338736780 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6970003520 con 0x7f6970004d00 2024-01-16T15:15:36.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.885+0000 7f6995c00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f6970005180 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f69900cbb90 tx=0x7f69700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7398ba7692e967 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:36.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.885+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988005ad0 con 0x7f6970004d00 2024-01-16T15:15:36.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.885+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6988005c30 con 0x7f6970004d00 2024-01-16T15:15:36.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.885+0000 7f698fe00640 1 -- 172.21.15.53:0/1338736780 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698802c790 con 0x7f6970004d00 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6997c00640 1 -- 172.21.15.53:0/1338736780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 msgr2=0x7f6970005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6997c00640 1 --2- 172.21.15.53:0/1338736780 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f6970005180 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f69900cbb90 tx=0x7f69700057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6995c00640 1 -- 172.21.15.53:0/1338736780 reap_dead start 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6997c00640 1 -- 172.21.15.53:0/1338736780 shutdown_connections 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6997c00640 1 -- 172.21.15.53:0/1338736780 >> 172.21.15.53:0/1338736780 conn(0x7f69900cf700 msgr2=0x7f69900cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6997c00640 1 -- 172.21.15.53:0/1338736780 shutdown_connections 2024-01-16T15:15:36.870 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.886+0000 7f6997c00640 1 -- 172.21.15.53:0/1338736780 wait complete. 2024-01-16T15:15:36.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.888+0000 7f6997c00640 1 Processor -- start 2024-01-16T15:15:36.871 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.888+0000 7f6997c00640 1 -- start start 2024-01-16T15:15:36.873 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6997c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f699015b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6997c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f699014a240 0x7f6990158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6997c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69901f38a0 0x7f69900cd600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6997c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f699015c230 con 0x7f6970004d00 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6997c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6990158ff0 con 0x7f699014a240 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6997c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f699014a6c0 con 0x7f69901f38a0 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6995200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f699015b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6995200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f699015b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39616/0 (socket says 172.21.15.53:39616) 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.889+0000 7f6995200640 1 -- 172.21.15.53:0/3167604247 learned_addr learned my addr 172.21.15.53:0/3167604247 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69901f38a0 0x7f69900cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6987c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f699014a240 0x7f6990158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995200640 1 -- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69901f38a0 msgr2=0x7f69900cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995200640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69901f38a0 0x7f69900cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995200640 1 -- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f699014a240 msgr2=0x7f6990158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:36.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995200640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f699014a240 0x7f6990158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995200640 1 -- 172.21.15.53:0/3167604247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699015b620 con 0x7f6970004d00 2024-01-16T15:15:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69901f38a0 0x7f69900cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6987c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f699014a240 0x7f6990158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6995200640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f699015b040 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f6980009e10 tx=0x7f698000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698000e040 con 0x7f6970004d00 2024-01-16T15:15:36.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698000f040 con 0x7f6970004d00 2024-01-16T15:15:36.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6980015870 con 0x7f6970004d00 2024-01-16T15:15:36.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.890+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f699015b900 con 0x7f6970004d00 2024-01-16T15:15:36.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.891+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f699015bec0 con 0x7f6970004d00 2024-01-16T15:15:36.876 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.892+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f699006d3d0 con 0x7f6970004d00 2024-01-16T15:15:36.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.893+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f69800159d0 con 0x7f6970004d00 2024-01-16T15:15:36.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.894+0000 7f698d600640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f696806c190 0x7f696806e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:36.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.894+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f69800895a0 con 0x7f6970004d00 2024-01-16T15:15:36.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.895+0000 7f6987c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f696806c190 0x7f696806e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:36.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.896+0000 7f6987c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f696806c190 0x7f696806e650 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f69901f6da0 tx=0x7f6978009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:36.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:36.899+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6980055da0 con 0x7f6970004d00 2024-01-16T15:15:37.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.133+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f69900675d0 con 0x7f6970004d00 2024-01-16T15:15:37.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.134+0000 7f698d600640 1 -- 172.21.15.53:0/3167604247 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v64) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6980022060 con 0x7f6970004d00 2024-01-16T15:15:37.118 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f696806c190 msgr2=0x7f696806e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f696806c190 0x7f696806e650 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f69901f6da0 tx=0x7f6978009040 comp rx=0 tx=0).stop 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 msgr2=0x7f699015b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f699015b040 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f6980009e10 tx=0x7f698000f580 comp rx=0 tx=0).stop 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 shutdown_connections 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f696806c190 0x7f696806e650 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69901f38a0 0x7f69900cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f699014a240 0x7f6990158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 --2- 172.21.15.53:0/3167604247 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6970004d00 0x7f699015b040 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 >> 172.21.15.53:0/3167604247 conn(0x7f69900cf700 msgr2=0x7f69900cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.136+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 shutdown_connections 2024-01-16T15:15:37.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:37.137+0000 7f6997c00640 1 -- 172.21.15.53:0/3167604247 wait complete. 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:15:37.139 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:37.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.140 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:37.196 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 41593sec preferred_lft 41593sec 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:15:37.197 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:15:37.198 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:15:37.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.199 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:15:37.287 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:15:37.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.291 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:15:37.341 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:15:37.341 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:15:37.341 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:15:37.341 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:15:37.341 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:15:37.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.342 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:15:37.343 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:15:37.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:15:37.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:15:37.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:15:37.343 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:15:37.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.522 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:15:37.571 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:15:37.571 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:15:37.572 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:37.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.573 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:15:37.659 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:15:37.660 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:37.660 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:37.712 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:15:37.712 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:15:37.712 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:37.759 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:15:37.899 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:37.954 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:15:37.954 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:15:37.954 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:15:37.954 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:15:37.954 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:15:37.955 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:15:37.956 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:37.957 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:37.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:37.959 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:15:38.008 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:15:38.008 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:15:38.013 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:15:38.089 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:15:38.129 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:15:38.130 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:15:38.131 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:38.132 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:15:38.133 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:38.133 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:15:38.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:38.134 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:15:38.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:15:38.210+0000 7f40b3cb0480 -1 init, newargv = 0x560eab4579d0 newargc=15 2024-01-16T15:15:38.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[46780]: starting ceph client 2024-01-16T15:15:38.206 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[46780]: starting fuse 2024-01-16T15:15:38.218 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:15:38.219 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:15:38.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:15:38.219 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:15:38.219 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:15:38.219 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:15:38.219 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:15:38.219 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:15:38.220 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:15:38.221 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:15:38.416 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.46780.asok 2024-01-16T15:15:38.420 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:15:38 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:15:38.433 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.46780.asok 2024-01-16T15:15:38.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:38.433 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.46780.asok status 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "46780", 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:15:38.667 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 7162, 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 7162 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2288027818 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:15:38.668 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2288027818 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.7162 192.168.0.1:0/2288027818", 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/2288027818", 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 64, 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 99, 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:15:38.669 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:15:38.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:38.682 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:15:38.682 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:38.729 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:15:38.729 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:38.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:38.730 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:15:38.868 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:15:39.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.121+0000 7f6b6f800640 1 Processor -- start 2024-01-16T15:15:39.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.121+0000 7f6b6f800640 1 -- start start 2024-01-16T15:15:39.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.123+0000 7f6b6f800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6b6814b2c0 0x7f6b680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.123+0000 7f6b6f800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b680cf7d0 0x7f6b68070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.123+0000 7f6b6f800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b680cfbf0 con 0x7f6b68070ea0 2024-01-16T15:15:39.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.123+0000 7f6b6f800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b6814b6a0 con 0x7f6b6814b2c0 2024-01-16T15:15:39.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.123+0000 7f6b6f800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68071690 con 0x7f6b680cf7d0 2024-01-16T15:15:39.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.124+0000 7f6b6ce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6b6814b2c0 0x7f6b680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.124+0000 7f6b6ce00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6b6814b2c0 0x7f6b680cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:52620/0 (socket says 172.21.15.53:52620) 2024-01-16T15:15:39.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.124+0000 7f6b6ce00640 1 -- 172.21.15.53:0/3159933104 learned_addr learned my addr 172.21.15.53:0/3159933104 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:39.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.124+0000 7f6b67e00640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b680cf7d0 0x7f6b68070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.125+0000 7f6b6ce00640 1 -- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b680cf7d0 msgr2=0x7f6b68070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.125+0000 7f6b6ce00640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b680cf7d0 0x7f6b68070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.125+0000 7f6b6ce00640 1 -- 172.21.15.53:0/3159933104 >> v1:172.21.15.146:6789/0 conn(0x7f6b68070ea0 legacy=0x7f6b680b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.125+0000 7f6b6ce00640 1 -- 172.21.15.53:0/3159933104 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b68071970 con 0x7f6b6814b2c0 2024-01-16T15:15:39.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.125+0000 7f6b67e00640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b680cf7d0 0x7f6b68070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:15:39.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.126+0000 7f6b6ce00640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b6814b2c0 0x7f6b680cf290 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009870 tx=0x7f6b5802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6162080b50ad461 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b67400640 1 -- 172.21.15.53:0/3159933104 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58030840 con 0x7f6b6814b2c0 2024-01-16T15:15:39.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b67400640 1 -- 172.21.15.53:0/3159933104 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b580309a0 con 0x7f6b6814b2c0 2024-01-16T15:15:39.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b67400640 1 -- 172.21.15.53:0/3159933104 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b58030d30 con 0x7f6b6814b2c0 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b6f800640 1 -- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b6814b2c0 msgr2=0x7f6b680cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b6f800640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b6814b2c0 0x7f6b680cf290 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f6b58009870 tx=0x7f6b5802b980 comp rx=0 tx=0).stop 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b6f800640 1 -- 172.21.15.53:0/3159933104 shutdown_connections 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b6f800640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b680cf7d0 0x7f6b68070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b6f800640 1 --2- 172.21.15.53:0/3159933104 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b6814b2c0 0x7f6b680cf290 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.127+0000 7f6b6f800640 1 -- 172.21.15.53:0/3159933104 >> 172.21.15.53:0/3159933104 conn(0x7f6b6806a800 msgr2=0x7f6b6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.128+0000 7f6b6f800640 1 -- 172.21.15.53:0/3159933104 shutdown_connections 2024-01-16T15:15:39.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.128+0000 7f6b6f800640 1 -- 172.21.15.53:0/3159933104 wait complete. 2024-01-16T15:15:39.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.129+0000 7f6b6f800640 1 Processor -- start 2024-01-16T15:15:39.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.130+0000 7f6b6f800640 1 -- start start 2024-01-16T15:15:39.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6f800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b68070ea0 0x7f6b68142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6f800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 0x7f6b680ce230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6f800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b6814b2c0 0x7f6b680ce770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6f800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68142790 con 0x7f6b680cf7d0 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6f800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68164760 con 0x7f6b68070ea0 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6f800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b68163b50 con 0x7f6b6814b2c0 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 0x7f6b680ce230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 0x7f6b680ce230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39628/0 (socket says 172.21.15.53:39628) 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 -- 172.21.15.53:0/1254553357 learned_addr learned my addr 172.21.15.53:0/1254553357 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6d800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b6814b2c0 0x7f6b680ce770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 -- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b6814b2c0 msgr2=0x7f6b680ce770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b6ce00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b68070ea0 0x7f6b68142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b6814b2c0 0x7f6b680ce770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 -- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b68070ea0 msgr2=0x7f6b68142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b68070ea0 0x7f6b68142250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.131+0000 7f6b67e00640 1 -- 172.21.15.53:0/1254553357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b50009400 con 0x7f6b680cf7d0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.132+0000 7f6b67e00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 0x7f6b680ce230 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000ebe0 tx=0x7f6b5000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.132+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b50009760 con 0x7f6b680cf7d0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.132+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b5000cc30 con 0x7f6b680cf7d0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.132+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b580094d0 con 0x7f6b680cf7d0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.132+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b5000eee0 con 0x7f6b680cf7d0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.132+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b68164110 con 0x7f6b680cf7d0 2024-01-16T15:15:39.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.133+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b680675d0 con 0x7f6b680cf7d0 2024-01-16T15:15:39.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.134+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f6b50013470 con 0x7f6b680cf7d0 2024-01-16T15:15:39.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.135+0000 7f6b64c00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b4006c160 0x7f6b4006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.119 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.136+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6b50088a00 con 0x7f6b680cf7d0 2024-01-16T15:15:39.120 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.136+0000 7f6b6ce00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b4006c160 0x7f6b4006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.137+0000 7f6b6ce00640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b4006c160 0x7f6b4006e620 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6b58036000 tx=0x7f6b5803b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.140+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6b500552c0 con 0x7f6b680cf7d0 2024-01-16T15:15:39.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.372+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6b6814a980 con 0x7f6b680cf7d0 2024-01-16T15:15:39.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.415+0000 7f6b64c00640 1 -- 172.21.15.53:0/1254553357 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v65) v1 ==== 136+0+0 (secure 0 0 0) 0x7f6b500588e0 con 0x7f6b680cf7d0 2024-01-16T15:15:39.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b4006c160 msgr2=0x7f6b4006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b4006c160 0x7f6b4006e620 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f6b58036000 tx=0x7f6b5803b040 comp rx=0 tx=0).stop 2024-01-16T15:15:39.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 msgr2=0x7f6b680ce230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 0x7f6b680ce230 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f6b5000ebe0 tx=0x7f6b5000c520 comp rx=0 tx=0).stop 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 shutdown_connections 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6b4006c160 0x7f6b4006e620 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6b6814b2c0 0x7f6b680ce770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6b680cf7d0 0x7f6b680ce230 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 --2- 172.21.15.53:0/1254553357 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6b68070ea0 0x7f6b68142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 >> 172.21.15.53:0/1254553357 conn(0x7f6b6806a800 msgr2=0x7f6b680b2320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 shutdown_connections 2024-01-16T15:15:39.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.418+0000 7f6b6f800640 1 -- 172.21.15.53:0/1254553357 wait complete. 2024-01-16T15:15:39.423 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:15:39.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.664+0000 7f25d0c00640 1 Processor -- start 2024-01-16T15:15:39.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.665+0000 7f25d0c00640 1 -- start start 2024-01-16T15:15:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.666+0000 7f25d0c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25cc15afb0 0x7f25cc13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.666+0000 7f25d0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f25cc13fae0 0x7f25cc0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.666+0000 7f25d0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25cc13ff30 con 0x7f25cc13cfa0 2024-01-16T15:15:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.666+0000 7f25d0c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc0cdc40 con 0x7f25cc15afb0 2024-01-16T15:15:39.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.666+0000 7f25d0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25cc0cdec0 con 0x7f25cc13fae0 2024-01-16T15:15:39.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.666+0000 7f25caa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25cc15afb0 0x7f25cc13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25caa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f25cc15afb0 0x7f25cc13f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39640/0 (socket says 172.21.15.53:39640) 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25caa00640 1 -- 172.21.15.53:0/1637519904 learned_addr learned my addr 172.21.15.53:0/1637519904 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25cbe00640 1 --2- 172.21.15.53:0/1637519904 >> v2:172.21.15.146:3300/0 conn(0x7f25cc13fae0 0x7f25cc0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25caa00640 1 -- 172.21.15.53:0/1637519904 >> v2:172.21.15.146:3300/0 conn(0x7f25cc13fae0 msgr2=0x7f25cc0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25caa00640 1 --2- 172.21.15.53:0/1637519904 >> v2:172.21.15.146:3300/0 conn(0x7f25cc13fae0 0x7f25cc0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25caa00640 1 -- 172.21.15.53:0/1637519904 >> v1:172.21.15.146:6789/0 conn(0x7f25cc13cfa0 legacy=0x7f25cc0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.667+0000 7f25caa00640 1 -- 172.21.15.53:0/1637519904 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc0ce1a0 con 0x7f25cc15afb0 2024-01-16T15:15:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.669+0000 7f25caa00640 1 --2- 172.21.15.53:0/1637519904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc15afb0 0x7f25cc13f5a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f25cc154a90 tx=0x7f25b402e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=702ead5550329688 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.669+0000 7f25ca000640 1 -- 172.21.15.53:0/1637519904 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b402ee90 con 0x7f25cc15afb0 2024-01-16T15:15:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.669+0000 7f25ca000640 1 -- 172.21.15.53:0/1637519904 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b4036440 con 0x7f25cc15afb0 2024-01-16T15:15:39.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.669+0000 7f25ca000640 1 -- 172.21.15.53:0/1637519904 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b40367d0 con 0x7f25cc15afb0 2024-01-16T15:15:39.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 -- 172.21.15.53:0/1637519904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc15afb0 msgr2=0x7f25cc13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 --2- 172.21.15.53:0/1637519904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc15afb0 0x7f25cc13f5a0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f25cc154a90 tx=0x7f25b402e0e0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 -- 172.21.15.53:0/1637519904 shutdown_connections 2024-01-16T15:15:39.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 --2- 172.21.15.53:0/1637519904 >> v2:172.21.15.146:3300/0 conn(0x7f25cc13fae0 0x7f25cc0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 --2- 172.21.15.53:0/1637519904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc15afb0 0x7f25cc13f5a0 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 -- 172.21.15.53:0/1637519904 >> 172.21.15.53:0/1637519904 conn(0x7f25cc06a800 msgr2=0x7f25cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 -- 172.21.15.53:0/1637519904 shutdown_connections 2024-01-16T15:15:39.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.670+0000 7f25d0c00640 1 -- 172.21.15.53:0/1637519904 wait complete. 2024-01-16T15:15:39.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.672+0000 7f25d0c00640 1 Processor -- start 2024-01-16T15:15:39.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.672+0000 7f25d0c00640 1 -- start start 2024-01-16T15:15:39.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25d0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 0x7f25cc138960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25d0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25cc13fae0 0x7f25cc13a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25d0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25cc15afb0 0x7f25cc13e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25d0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc153d90 con 0x7f25cc13cfa0 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25d0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc138ea0 con 0x7f25cc15afb0 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25d0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25cc152e30 con 0x7f25cc13fae0 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25cb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 0x7f25cc138960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25cb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 0x7f25cc138960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39654/0 (socket says 172.21.15.53:39654) 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.673+0000 7f25cb400640 1 -- 172.21.15.53:0/2472804317 learned_addr learned my addr 172.21.15.53:0/2472804317 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cb400640 1 -- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25cc13fae0 msgr2=0x7f25cc13a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cb400640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25cc13fae0 0x7f25cc13a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cb400640 1 -- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25cc15afb0 msgr2=0x7f25cc13e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cbe00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25cc15afb0 0x7f25cc13e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cb400640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25cc15afb0 0x7f25cc13e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cb400640 1 -- 172.21.15.53:0/2472804317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b400b770 con 0x7f25cc13cfa0 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25cb400640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 0x7f25cc138960 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00c790 tx=0x7f25bc00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc00cea0 con 0x7f25cc13cfa0 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25bc006d50 con 0x7f25cc13cfa0 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.674+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25cc153110 con 0x7f25cc13cfa0 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.675+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25bc003bb0 con 0x7f25cc13cfa0 2024-01-16T15:15:39.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.675+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25cc1536d0 con 0x7f25cc13cfa0 2024-01-16T15:15:39.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.676+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25cc067450 con 0x7f25cc13cfa0 2024-01-16T15:15:39.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.677+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f25bc003d10 con 0x7f25cc13cfa0 2024-01-16T15:15:39.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.678+0000 7f25c3400640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a406c160 0x7f25a406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:39.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.678+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f25bc088150 con 0x7f25cc13cfa0 2024-01-16T15:15:39.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.678+0000 7f25caa00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a406c160 0x7f25a406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:39.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.679+0000 7f25caa00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a406c160 0x7f25a406e620 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f25b4038000 tx=0x7f25b4030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:39.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.682+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f25bc0584a0 con 0x7f25cc13cfa0 2024-01-16T15:15:39.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.899+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f25cc1e3ce0 con 0x7f25a406c160 2024-01-16T15:15:39.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.902+0000 7f25c3400640 1 -- 172.21.15.53:0/2472804317 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f25cc1e3ce0 con 0x7f25a406c160 2024-01-16T15:15:39.886 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:15:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:15:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:15:39.887 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:15:39.887 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a406c160 msgr2=0x7f25a406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a406c160 0x7f25a406e620 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f25b4038000 tx=0x7f25b4030000 comp rx=0 tx=0).stop 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 msgr2=0x7f25cc138960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 0x7f25cc138960 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00c790 tx=0x7f25bc00cc50 comp rx=0 tx=0).stop 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 shutdown_connections 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f25a406c160 0x7f25a406e620 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f25cc15afb0 0x7f25cc13e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f25cc13fae0 0x7f25cc13a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 --2- 172.21.15.53:0/2472804317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f25cc13cfa0 0x7f25cc138960 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 >> 172.21.15.53:0/2472804317 conn(0x7f25cc06a800 msgr2=0x7f25cc13bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 shutdown_connections 2024-01-16T15:15:39.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:39.905+0000 7f25d0c00640 1 -- 172.21.15.53:0/2472804317 wait complete. 2024-01-16T15:15:39.909 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:15:40.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.148+0000 7f0392000640 1 Processor -- start 2024-01-16T15:15:40.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.149+0000 7f0392000640 1 -- start start 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.150+0000 7f0392000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f038c146850 0x7f038c0715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0392000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c0ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0392000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f038c06d480 con 0x7f038c0adb00 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0392000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f038c071fa0 con 0x7f038c146850 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0392000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c0b3490 con 0x7f038c071b10 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f038c146850 0x7f038c0715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f038c146850 0x7f038c0715d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39668/0 (socket says 172.21.15.53:39668) 2024-01-16T15:15:40.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 -- 172.21.15.53:0/1494330833 learned_addr learned my addr 172.21.15.53:0/1494330833 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f038be00640 1 --2- 172.21.15.53:0/1494330833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c0ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 -- 172.21.15.53:0/1494330833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 msgr2=0x7f038c0ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 --2- 172.21.15.53:0/1494330833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c0ad5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 -- 172.21.15.53:0/1494330833 >> v1:172.21.15.146:6789/0 conn(0x7f038c0adb00 legacy=0x7f038c0700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.151+0000 7f0390c00640 1 -- 172.21.15.53:0/1494330833 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f038c0b3770 con 0x7f038c146850 2024-01-16T15:15:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.153+0000 7f0390c00640 1 --2- 172.21.15.53:0/1494330833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c146850 0x7f038c0715d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f037c009870 tx=0x7f037c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73741d17ca756f9c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.153+0000 7f038b400640 1 -- 172.21.15.53:0/1494330833 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c031640 con 0x7f038c146850 2024-01-16T15:15:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.153+0000 7f038b400640 1 -- 172.21.15.53:0/1494330833 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f037c0317a0 con 0x7f038c146850 2024-01-16T15:15:40.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.153+0000 7f038b400640 1 -- 172.21.15.53:0/1494330833 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f037c031b30 con 0x7f038c146850 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 -- 172.21.15.53:0/1494330833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c146850 msgr2=0x7f038c0715d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 --2- 172.21.15.53:0/1494330833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c146850 0x7f038c0715d0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f037c009870 tx=0x7f037c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 -- 172.21.15.53:0/1494330833 shutdown_connections 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 --2- 172.21.15.53:0/1494330833 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c0ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 --2- 172.21.15.53:0/1494330833 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c146850 0x7f038c0715d0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 -- 172.21.15.53:0/1494330833 >> 172.21.15.53:0/1494330833 conn(0x7f038c06a3e0 msgr2=0x7f038c06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:40.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 -- 172.21.15.53:0/1494330833 shutdown_connections 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.154+0000 7f0392000640 1 -- 172.21.15.53:0/1494330833 wait complete. 2024-01-16T15:15:40.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.156+0000 7f0392000640 1 Processor -- start 2024-01-16T15:15:40.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.156+0000 7f0392000640 1 -- start start 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0392000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c156790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0392000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 0x7f038c0c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0392000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c146850 0x7f038c0af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0392000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c137030 con 0x7f038c0adb00 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0392000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f038c156cd0 con 0x7f038c146850 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0392000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f038c1363b0 con 0x7f038c071b10 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f038be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 0x7f038c0c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f038be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 0x7f038c0c40b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39682/0 (socket says 172.21.15.53:39682) 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f038be00640 1 -- 172.21.15.53:0/2675120676 learned_addr learned my addr 172.21.15.53:0/2675120676 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f038be00640 1 -- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 msgr2=0x7f038c156790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.157+0000 7f0391600640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c146850 0x7f038c0af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f038be00640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c156790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f038be00640 1 -- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c146850 msgr2=0x7f038c0af160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f038be00640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c146850 0x7f038c0af160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f038be00640 1 -- 172.21.15.53:0/2675120676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f037c0094d0 con 0x7f038c0adb00 2024-01-16T15:15:40.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f0391600640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c146850 0x7f038c0af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:40.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f038be00640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 0x7f038c0c40b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f037400c7f0 tx=0x7f037400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374004220 con 0x7f038c0adb00 2024-01-16T15:15:40.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03740043a0 con 0x7f038c0adb00 2024-01-16T15:15:40.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0374007860 con 0x7f038c0adb00 2024-01-16T15:15:40.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f038c136690 con 0x7f038c0adb00 2024-01-16T15:15:40.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.158+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f038c136c50 con 0x7f038c0adb00 2024-01-16T15:15:40.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.160+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f038c137170 con 0x7f038c0adb00 2024-01-16T15:15:40.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.161+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f0374007bd0 con 0x7f038c0adb00 2024-01-16T15:15:40.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.162+0000 7f0388c00640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f036406c160 0x7f036406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.162+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0374088960 con 0x7f038c0adb00 2024-01-16T15:15:40.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.163+0000 7f0390c00640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f036406c160 0x7f036406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.164+0000 7f0390c00640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f036406c160 0x7f036406e620 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f037c036000 tx=0x7f037c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.167+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f038c137170 con 0x7f038c0adb00 2024-01-16T15:15:40.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.365+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f038c137170 con 0x7f038c0adb00 2024-01-16T15:15:40.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.365+0000 7f0388c00640 1 -- 172.21.15.53:0/2675120676 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v36)=0 v36) v1 ==== 122+0+0 (secure 0 0 0) 0x7f038c137170 con 0x7f038c0adb00 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f036406c160 msgr2=0x7f036406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f036406c160 0x7f036406e620 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f037c036000 tx=0x7f037c030000 comp rx=0 tx=0).stop 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 msgr2=0x7f038c0c40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 0x7f038c0c40b0 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f037400c7f0 tx=0x7f037400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 shutdown_connections 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f036406c160 0x7f036406e620 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f038c146850 0x7f038c0af160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f038c0adb00 0x7f038c0c40b0 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 --2- 172.21.15.53:0/2675120676 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f038c071b10 0x7f038c156790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 >> 172.21.15.53:0/2675120676 conn(0x7f038c06a3e0 msgr2=0x7f038c0700f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:40.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 shutdown_connections 2024-01-16T15:15:40.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.368+0000 7f0392000640 1 -- 172.21.15.53:0/2675120676 wait complete. 2024-01-16T15:15:40.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 10 2024-01-16T15:15:40.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.613+0000 7f4afa200640 1 Processor -- start 2024-01-16T15:15:40.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.613+0000 7f4afa200640 1 -- start start 2024-01-16T15:15:40.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.615+0000 7f4afa200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4af406c730 0x7f4af40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.615+0000 7f4afa200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af40708d0 con 0x7f4af406cb10 2024-01-16T15:15:40.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.615+0000 7f4afa200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4af4070bb0 con 0x7f4af4070160 2024-01-16T15:15:40.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.615+0000 7f4afa200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4af4070e90 con 0x7f4af406c730 2024-01-16T15:15:40.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.615+0000 7f4af9800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4af406cb10 0x7f4af406cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54618/0 (socket says 172.21.15.53:54618) 2024-01-16T15:15:40.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af9800640 1 -- 172.21.15.53:0/3496433988 learned_addr learned my addr 172.21.15.53:0/3496433988 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af8e00640 1 --2- 172.21.15.53:0/3496433988 >> v2:172.21.15.146:3300/0 conn(0x7f4af406c730 0x7f4af40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985381082 0 0) 0x7f4af40708d0 con 0x7f4af406cb10 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ad4003520 con 0x7f4af406cb10 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 542483604 0 0) 0x7f4ad4003520 con 0x7f4af406cb10 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 >> v2:172.21.15.146:3300/0 conn(0x7f4af406c730 msgr2=0x7f4af40b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 --2- 172.21.15.53:0/3496433988 >> v2:172.21.15.146:3300/0 conn(0x7f4af406c730 0x7f4af40b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 >> v1:172.21.15.146:6789/0 conn(0x7f4af4070160 legacy=0x7f4af4071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4071170 con 0x7f4af406cb10 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4aec001670 con 0x7f4af406cb10 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 >> v1:172.21.15.53:6789/0 conn(0x7f4af406cb10 legacy=0x7f4af406cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.616+0000 7f4af3e00640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad40039c0 0x7f4ad4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af3e00640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40047c0 0x7f4ad4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af3e00640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4005180 0x7f4ad400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af4070e90 con 0x7f4ad4005180 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad400c080 con 0x7f4ad40047c0 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad400c240 con 0x7f4ad40039c0 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af9800640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4005180 0x7f4ad400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af9800640 1 -- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad40039c0 msgr2=0x7f4ad4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af9800640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ad40039c0 0x7f4ad4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4aebc00640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40047c0 0x7f4ad4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af9800640 1 -- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40047c0 msgr2=0x7f4ad4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.617+0000 7f4af9800640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40047c0 0x7f4ad4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.618+0000 7f4af9800640 1 -- 172.21.15.53:0/3496433988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4003520 con 0x7f4ad4005180 2024-01-16T15:15:40.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.618+0000 7f4aebc00640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad40047c0 0x7f4ad4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:40.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.618+0000 7f4af9800640 1 -- 172.21.15.53:0/3496433988 reap_dead start 2024-01-16T15:15:40.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.619+0000 7f4af9800640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4005180 0x7f4ad400b980 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00c0e0 tx=0x7f4ad40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a7f8c1b6d92fc78 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.619+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec005ad0 con 0x7f4ad4005180 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.619+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aec005c70 con 0x7f4ad4005180 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.619+0000 7f4af3e00640 1 -- 172.21.15.53:0/3496433988 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aec02c810 con 0x7f4ad4005180 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.619+0000 7f4afa200640 1 -- 172.21.15.53:0/3496433988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4005180 msgr2=0x7f4ad400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.619+0000 7f4afa200640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4005180 0x7f4ad400b980 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7f4aec00c0e0 tx=0x7f4ad40057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.620+0000 7f4afa200640 1 -- 172.21.15.53:0/3496433988 shutdown_connections 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.620+0000 7f4afa200640 1 --2- 172.21.15.53:0/3496433988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ad4005180 0x7f4ad400b980 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.620+0000 7f4afa200640 1 -- 172.21.15.53:0/3496433988 >> 172.21.15.53:0/3496433988 conn(0x7f4af406a7d0 msgr2=0x7f4af406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.620+0000 7f4afa200640 1 -- 172.21.15.53:0/3496433988 shutdown_connections 2024-01-16T15:15:40.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.620+0000 7f4afa200640 1 -- 172.21.15.53:0/3496433988 wait complete. 2024-01-16T15:15:40.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.622+0000 7f4afa200640 1 Processor -- start 2024-01-16T15:15:40.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.622+0000 7f4afa200640 1 -- start start 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4afa200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad4005180 0x7f4af41fa310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4afa200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 0x7f4af40ad290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4afa200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4af40ad7d0 0x7f4af40af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4afa200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41fb270 con 0x7f4af41613f0 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4afa200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41facb0 con 0x7f4ad4005180 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4afa200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4af41faf90 con 0x7f4af40ad7d0 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 0x7f4af40ad290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 0x7f4af40ad290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39708/0 (socket says 172.21.15.53:39708) 2024-01-16T15:15:40.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 -- 172.21.15.53:0/3975857607 learned_addr learned my addr 172.21.15.53:0/3975857607 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:40.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 -- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4af40ad7d0 msgr2=0x7f4af40af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4af40ad7d0 0x7f4af40af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 -- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad4005180 msgr2=0x7f4af41fa310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad4005180 0x7f4af41fa310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.623+0000 7f4aebc00640 1 -- 172.21.15.53:0/3975857607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af40aa750 con 0x7f4af41613f0 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.624+0000 7f4aebc00640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 0x7f4af40ad290 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00d770 tx=0x7f4adc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.624+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc005270 con 0x7f4af41613f0 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.624+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc0053d0 con 0x7f4af41613f0 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.624+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af40aaa30 con 0x7f4af41613f0 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.624+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc00b870 con 0x7f4af41613f0 2024-01-16T15:15:40.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.624+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af41faa30 con 0x7f4af41613f0 2024-01-16T15:15:40.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.625+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4af40aa5b0 con 0x7f4af41613f0 2024-01-16T15:15:40.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.626+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f4adc00b9d0 con 0x7f4af41613f0 2024-01-16T15:15:40.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.627+0000 7f4af1600640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4acc06c160 0x7f4acc06e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:40.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.627+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4adc088880 con 0x7f4af41613f0 2024-01-16T15:15:40.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.628+0000 7f4af8e00640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4acc06c160 0x7f4acc06e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:40.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.629+0000 7f4af8e00640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4acc06c160 0x7f4acc06e620 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40070b0 tx=0x7f4ae400d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.632+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4af40aa5b0 con 0x7f4af41613f0 2024-01-16T15:15:40.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.825+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f4af40aa5b0 con 0x7f4af41613f0 2024-01-16T15:15:40.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.831+0000 7f4af1600640 1 -- 172.21.15.53:0/3975857607 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v37)=0 v37) v1 ==== 131+0+0 (secure 0 0 0) 0x7f4af40aa5b0 con 0x7f4af41613f0 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4acc06c160 msgr2=0x7f4acc06e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4acc06c160 0x7f4acc06e620 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f4ae40070b0 tx=0x7f4ae400d040 comp rx=0 tx=0).stop 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 msgr2=0x7f4af40ad290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 0x7f4af40ad290 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7f4adc00d770 tx=0x7f4adc00dc30 comp rx=0 tx=0).stop 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 shutdown_connections 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4acc06c160 0x7f4acc06e620 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4af40ad7d0 0x7f4af40af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4af41613f0 0x7f4af40ad290 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 --2- 172.21.15.53:0/3975857607 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ad4005180 0x7f4af41fa310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:40.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 >> 172.21.15.53:0/3975857607 conn(0x7f4af406a7d0 msgr2=0x7f4af406cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:40.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 shutdown_connections 2024-01-16T15:15:40.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:40.834+0000 7f4afa200640 1 -- 172.21.15.53:0/3975857607 wait complete. 2024-01-16T15:15:40.838 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:15:41.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.079+0000 7f21cbe00640 1 Processor -- start 2024-01-16T15:15:41.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.080+0000 7f21cbe00640 1 -- start start 2024-01-16T15:15:41.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cbe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f21cc1608b0 0x7f21cc0d3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc0d1690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cbe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0cf6c0 con 0x7f21cc0c6c00 2024-01-16T15:15:41.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cbe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc0d1450 con 0x7f21cc1608b0 2024-01-16T15:15:41.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0b4390 con 0x7f21cc160c90 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21caa00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f21cc1608b0 0x7f21cc0d3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cb400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f21cc0c6c00 0x7f21cc161130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54632/0 (socket says 172.21.15.53:54632) 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.081+0000 7f21cb400640 1 -- 172.21.15.53:0/3242238150 learned_addr learned my addr 172.21.15.53:0/3242238150 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.082+0000 7f21c3e00640 1 --2- 172.21.15.53:0/3242238150 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc0d1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.082+0000 7f21caa00640 1 -- 172.21.15.53:0/3242238150 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 msgr2=0x7f21cc0d1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.082+0000 7f21caa00640 1 --2- 172.21.15.53:0/3242238150 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc0d1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.082+0000 7f21caa00640 1 -- 172.21.15.53:0/3242238150 >> v1:172.21.15.53:6789/0 conn(0x7f21cc0c6c00 legacy=0x7f21cc161130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.082+0000 7f21caa00640 1 -- 172.21.15.53:0/3242238150 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc0ce7e0 con 0x7f21cc1608b0 2024-01-16T15:15:41.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.083+0000 7f21caa00640 1 --2- 172.21.15.53:0/3242238150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0d3bd0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097a0 tx=0x7f21bc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2edf506252b0575a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.083+0000 7f21ca000640 1 -- 172.21.15.53:0/3242238150 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0316f0 con 0x7f21cc1608b0 2024-01-16T15:15:41.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.083+0000 7f21ca000640 1 -- 172.21.15.53:0/3242238150 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc031850 con 0x7f21cc1608b0 2024-01-16T15:15:41.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.083+0000 7f21ca000640 1 -- 172.21.15.53:0/3242238150 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc031be0 con 0x7f21cc1608b0 2024-01-16T15:15:41.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 -- 172.21.15.53:0/3242238150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 msgr2=0x7f21cc0d3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 --2- 172.21.15.53:0/3242238150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0d3bd0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f21bc0097a0 tx=0x7f21bc02b790 comp rx=0 tx=0).stop 2024-01-16T15:15:41.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 -- 172.21.15.53:0/3242238150 shutdown_connections 2024-01-16T15:15:41.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 --2- 172.21.15.53:0/3242238150 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc0d1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 --2- 172.21.15.53:0/3242238150 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0d3bd0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 -- 172.21.15.53:0/3242238150 >> 172.21.15.53:0/3242238150 conn(0x7f21cc0d30b0 msgr2=0x7f21cc0d34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:41.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 -- 172.21.15.53:0/3242238150 shutdown_connections 2024-01-16T15:15:41.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.084+0000 7f21cbe00640 1 -- 172.21.15.53:0/3242238150 wait complete. 2024-01-16T15:15:41.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.086+0000 7f21cbe00640 1 Processor -- start 2024-01-16T15:15:41.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.086+0000 7f21cbe00640 1 -- start start 2024-01-16T15:15:41.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21cbe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21cc0c6c00 0x7f21cc1489d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21cbe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0b5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21cbe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc144fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21cbe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc1f62b0 con 0x7f21cc1608b0 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21cbe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0b64d0 con 0x7f21cc0c6c00 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21cbe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0b85e0 con 0x7f21cc160c90 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0b5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21c3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0b5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39720/0 (socket says 172.21.15.53:39720) 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.087+0000 7f21c3e00640 1 -- 172.21.15.53:0/1352267527 learned_addr learned my addr 172.21.15.53:0/1352267527 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21caa00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21cc0c6c00 0x7f21cc1489d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21cb400640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc144fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c3e00640 1 -- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 msgr2=0x7f21cc144fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c3e00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc144fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c3e00640 1 -- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21cc0c6c00 msgr2=0x7f21cc1489d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c3e00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21cc0c6c00 0x7f21cc1489d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c3e00640 1 -- 172.21.15.53:0/1352267527 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc009400 con 0x7f21cc1608b0 2024-01-16T15:15:41.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c3e00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0b5ca0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f21b400c7f0 tx=0x7f21b400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.088+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b4004220 con 0x7f21cc1608b0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.089+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21b4004380 con 0x7f21cc1608b0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.089+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc0b88c0 con 0x7f21cc1608b0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.089+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21b4007840 con 0x7f21cc1608b0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.089+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc0b61e0 con 0x7f21cc1608b0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.090+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc08d110 con 0x7f21cc1608b0 2024-01-16T15:15:41.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.091+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f21b4007bb0 con 0x7f21cc1608b0 2024-01-16T15:15:41.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.092+0000 7f21c2a00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f21a406c160 0x7f21a406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.092+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f21b4088880 con 0x7f21cc1608b0 2024-01-16T15:15:41.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.092+0000 7f21caa00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f21a406c160 0x7f21a406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.093+0000 7f21caa00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f21a406c160 0x7f21a406e620 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f21bc036000 tx=0x7f21bc008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.096+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21b4054e60 con 0x7f21cc1608b0 2024-01-16T15:15:41.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.292+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f21cc067450 con 0x7f21cc1608b0 2024-01-16T15:15:41.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.297+0000 7f21c2a00640 1 -- 172.21.15.53:0/1352267527 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v38)=0 v38) v1 ==== 137+0+0 (secure 0 0 0) 0x7f21b401e0d0 con 0x7f21cc1608b0 2024-01-16T15:15:41.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f21a406c160 msgr2=0x7f21a406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f21a406c160 0x7f21a406e620 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f21bc036000 tx=0x7f21bc008030 comp rx=0 tx=0).stop 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 msgr2=0x7f21cc0b5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0b5ca0 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f21b400c7f0 tx=0x7f21b400ccb0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 shutdown_connections 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f21a406c160 0x7f21a406e620 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21cc160c90 0x7f21cc144fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21cc1608b0 0x7f21cc0b5ca0 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 --2- 172.21.15.53:0/1352267527 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21cc0c6c00 0x7f21cc1489d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 >> 172.21.15.53:0/1352267527 conn(0x7f21cc0d30b0 msgr2=0x7f21cc0c7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 shutdown_connections 2024-01-16T15:15:41.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.300+0000 7f21cbe00640 1 -- 172.21.15.53:0/1352267527 wait complete. 2024-01-16T15:15:41.303 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:15:41.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.541+0000 7fcface00640 1 Processor -- start 2024-01-16T15:15:41.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.542+0000 7fcface00640 1 -- start start 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcface00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcfa8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcface00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fcfa80717d0 0x7fcfa80cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcface00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa8070e50 con 0x7fcfa80cca40 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcface00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfa80723c0 con 0x7fcfa80717d0 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcface00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80cce70 con 0x7fcfa8143c40 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcfa7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fcfa80cca40 0x7fcfa815eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54640/0 (socket says 172.21.15.53:54640) 2024-01-16T15:15:41.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcfa7e00640 1 -- 172.21.15.53:0/3279805873 learned_addr learned my addr 172.21.15.53:0/3279805873 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:41.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.544+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345633127 0 0) 0x7fcfa8070e50 con 0x7fcfa80cca40 2024-01-16T15:15:41.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf88003520 con 0x7fcfa80cca40 2024-01-16T15:15:41.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/3279805873 >> v2:172.21.15.146:3300/0 conn(0x7fcfa80717d0 0x7fcfa80cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa7400640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcfa8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4053999030 0 0) 0x7fcf88003520 con 0x7fcfa80cca40 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 msgr2=0x7fcfa8071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcfa8071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 >> v2:172.21.15.146:3300/0 conn(0x7fcfa80717d0 msgr2=0x7fcfa80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 --2- 172.21.15.53:0/3279805873 >> v2:172.21.15.146:3300/0 conn(0x7fcfa80717d0 0x7fcfa80cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa806e170 con 0x7fcfa80cca40 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fcfa00019f0 con 0x7fcfa80cca40 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 >> v1:172.21.15.53:6789/0 conn(0x7fcfa80cca40 legacy=0x7fcfa815eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcf88003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcf880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80723c0 con 0x7fcf88003ff0 2024-01-16T15:15:41.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa80cce70 con 0x7fcf88004d00 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf8800bb90 con 0x7fcfa8143c40 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/3279805873 >> v2:172.21.15.146:3300/0 conn(0x7fcfa80717d0 0x7fcfa80cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.545+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcf880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa7400640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcf88003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa7e00640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa6a00640 1 -- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 msgr2=0x7fcf88003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcf88003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa6a00640 1 -- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf88004d00 msgr2=0x7fcf88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcf88004d00 0x7fcf88005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa6a00640 1 -- 172.21.15.53:0/3279805873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf88003520 con 0x7fcf88003ff0 2024-01-16T15:15:41.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.546+0000 7fcfa7400640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa8143c40 0x7fcf88003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:41.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.547+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcf880047c0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098a0 tx=0x7fcf880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca15b4398582e4e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.547+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90037600 con 0x7fcf88003ff0 2024-01-16T15:15:41.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.547+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf90037760 con 0x7fcf88003ff0 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.547+0000 7fcfa6000640 1 -- 172.21.15.53:0/3279805873 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90037ad0 con 0x7fcf88003ff0 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcface00640 1 -- 172.21.15.53:0/3279805873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 msgr2=0x7fcf880047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcface00640 1 --2- 172.21.15.53:0/3279805873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcf880047c0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fcf900098a0 tx=0x7fcf880057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcfa7e00640 1 -- 172.21.15.53:0/3279805873 reap_dead start 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcface00640 1 -- 172.21.15.53:0/3279805873 shutdown_connections 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcface00640 1 -- 172.21.15.53:0/3279805873 >> 172.21.15.53:0/3279805873 conn(0x7fcfa806a800 msgr2=0x7fcfa806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcface00640 1 -- 172.21.15.53:0/3279805873 shutdown_connections 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.548+0000 7fcface00640 1 -- 172.21.15.53:0/3279805873 wait complete. 2024-01-16T15:15:41.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.550+0000 7fcface00640 1 Processor -- start 2024-01-16T15:15:41.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.550+0000 7fcface00640 1 -- start start 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcface00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcfa81f7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcface00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa81e65f0 0x7fcfa815a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcface00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfa812dd40 0x7fcfa8142490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcface00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81e6db0 con 0x7fcf88003ff0 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcface00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81e7090 con 0x7fcfa812dd40 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcface00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81e7370 con 0x7fcfa81e65f0 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.551+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcfa81f7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcfa81f7290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:39740/0 (socket says 172.21.15.53:39740) 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 -- 172.21.15.53:0/4034575363 learned_addr learned my addr 172.21.15.53:0/4034575363 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa81e65f0 0x7fcfa815a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7e00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfa812dd40 0x7fcfa8142490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 -- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa81e65f0 msgr2=0x7fcfa815a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa81e65f0 0x7fcfa815a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 -- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfa812dd40 msgr2=0x7fcfa8142490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfa812dd40 0x7fcfa8142490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 -- 172.21.15.53:0/4034575363 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa81e7650 con 0x7fcf88003ff0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.552+0000 7fcfa7400640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcfa81f7290 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fcf98006a90 tx=0x7fcf98004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.553+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98014870 con 0x7fcf88003ff0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.553+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf980149d0 con 0x7fcf88003ff0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.553+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98014d60 con 0x7fcf88003ff0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.553+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa81e7930 con 0x7fcf88003ff0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.553+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa81e6a70 con 0x7fcf88003ff0 2024-01-16T15:15:41.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.554+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa8067450 con 0x7fcf88003ff0 2024-01-16T15:15:41.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.555+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7fcf98015410 con 0x7fcf88003ff0 2024-01-16T15:15:41.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.556+0000 7fcf9f400640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf8006c160 0x7fcf8006e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:41.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.556+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fcf980892b0 con 0x7fcf88003ff0 2024-01-16T15:15:41.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.556+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf8006c160 0x7fcf8006e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:41.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.558+0000 7fcfa6a00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf8006c160 0x7fcf8006e620 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcf90036000 tx=0x7fcf90031000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:41.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.560+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcf98055890 con 0x7fcf88003ff0 2024-01-16T15:15:41.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.775+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfa80cc8a0 con 0x7fcf8006c160 2024-01-16T15:15:41.772 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:41.787+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:41.772 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:41.787+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:41.772 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:41.787+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:41.772 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:41.787+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:41.772 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:15:41.787+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:15:41.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.862+0000 7fcf9f400640 1 -- 172.21.15.53:0/4034575363 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcfa80cc8a0 con 0x7fcf8006c160 2024-01-16T15:15:41.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf8006c160 msgr2=0x7fcf8006e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf8006c160 0x7fcf8006e620 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fcf90036000 tx=0x7fcf90031000 comp rx=0 tx=0).stop 2024-01-16T15:15:41.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 msgr2=0x7fcfa81f7290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:41.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcfa81f7290 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fcf98006a90 tx=0x7fcf98004610 comp rx=0 tx=0).stop 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 shutdown_connections 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcf8006c160 0x7fcf8006e620 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcfa812dd40 0x7fcfa8142490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcfa81e65f0 0x7fcfa815a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 --2- 172.21.15.53:0/4034575363 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcf88003ff0 0x7fcfa81f7290 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 >> 172.21.15.53:0/4034575363 conn(0x7fcfa806a800 msgr2=0x7fcfa806d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 shutdown_connections 2024-01-16T15:15:41.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:41.865+0000 7fcface00640 1 -- 172.21.15.53:0/4034575363 wait complete. 2024-01-16T15:15:41.870 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:42.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.110+0000 7f2b61c00640 1 Processor -- start 2024-01-16T15:15:42.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.110+0000 7f2b61c00640 1 -- start start 2024-01-16T15:15:42.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b61c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2b5c0adf20 0x7f2b5c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b61c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b5c15bf10 0x7f2b5c0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b61c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c0cc980 0x7f2b5c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b61c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c15f970 con 0x7f2b5c15bf10 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b61c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c15f4e0 con 0x7f2b5c0adf20 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b61c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0cce10 con 0x7f2b5c0cc980 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.112+0000 7f2b5b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b5c15bf10 0x7f2b5c0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b5c15bf10 0x7f2b5c0cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46024/0 (socket says 172.21.15.53:46024) 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 -- 172.21.15.53:0/416576060 learned_addr learned my addr 172.21.15.53:0/416576060 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5be00640 1 --2- 172.21.15.53:0/416576060 >> v2:172.21.15.146:3300/0 conn(0x7f2b5c0adf20 0x7f2b5c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b61200640 1 --2- 172.21.15.53:0/416576060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c0cc980 0x7f2b5c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 -- 172.21.15.53:0/416576060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c0cc980 msgr2=0x7f2b5c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 --2- 172.21.15.53:0/416576060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c0cc980 0x7f2b5c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 -- 172.21.15.53:0/416576060 >> v2:172.21.15.146:3300/0 conn(0x7f2b5c0adf20 msgr2=0x7f2b5c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 --2- 172.21.15.53:0/416576060 >> v2:172.21.15.146:3300/0 conn(0x7f2b5c0adf20 0x7f2b5c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5b400640 1 -- 172.21.15.53:0/416576060 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c071130 con 0x7f2b5c15bf10 2024-01-16T15:15:42.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.113+0000 7f2b5be00640 1 --2- 172.21.15.53:0/416576060 >> v2:172.21.15.146:3300/0 conn(0x7f2b5c0adf20 0x7f2b5c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:42.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.116+0000 7f2b5b400640 1 --2- 172.21.15.53:0/416576060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c15bf10 0x7f2b5c0cc440 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0b4150 tx=0x7f2b4402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61ebc995f67d8df server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.117+0000 7f2b5aa00640 1 -- 172.21.15.53:0/416576060 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b440317e0 con 0x7f2b5c15bf10 2024-01-16T15:15:42.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.117+0000 7f2b5aa00640 1 -- 172.21.15.53:0/416576060 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b44031940 con 0x7f2b5c15bf10 2024-01-16T15:15:42.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.117+0000 7f2b5aa00640 1 -- 172.21.15.53:0/416576060 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44031cd0 con 0x7f2b5c15bf10 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.117+0000 7f2b61c00640 1 -- 172.21.15.53:0/416576060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c15bf10 msgr2=0x7f2b5c0cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.117+0000 7f2b61c00640 1 --2- 172.21.15.53:0/416576060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c15bf10 0x7f2b5c0cc440 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0b4150 tx=0x7f2b4402b880 comp rx=0 tx=0).stop 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 -- 172.21.15.53:0/416576060 shutdown_connections 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 --2- 172.21.15.53:0/416576060 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c0cc980 0x7f2b5c143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 --2- 172.21.15.53:0/416576060 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c15bf10 0x7f2b5c0cc440 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 --2- 172.21.15.53:0/416576060 >> v2:172.21.15.146:3300/0 conn(0x7f2b5c0adf20 0x7f2b5c070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 -- 172.21.15.53:0/416576060 >> 172.21.15.53:0/416576060 conn(0x7f2b5c06a7d0 msgr2=0x7f2b5c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 -- 172.21.15.53:0/416576060 shutdown_connections 2024-01-16T15:15:42.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.118+0000 7f2b61c00640 1 -- 172.21.15.53:0/416576060 wait complete. 2024-01-16T15:15:42.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.120+0000 7f2b61c00640 1 Processor -- start 2024-01-16T15:15:42.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.120+0000 7f2b61c00640 1 -- start start 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b5c0adf20 0x7f2b5c15a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 0x7f2b5c15c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c15bf10 0x7f2b5c15cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c15aa90 con 0x7f2b5c0cc980 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c13a050 con 0x7f2b5c0adf20 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0b3220 con 0x7f2b5c15bf10 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 0x7f2b5c15c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b5b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 0x7f2b5c15c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46038/0 (socket says 172.21.15.53:46038) 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b5b400640 1 -- 172.21.15.53:0/2651260966 learned_addr learned my addr 172.21.15.53:0/2651260966 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.121+0000 7f2b61200640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c15bf10 0x7f2b5c15cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5be00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b5c0adf20 0x7f2b5c15a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5b400640 1 -- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c15bf10 msgr2=0x7f2b5c15cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5b400640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c15bf10 0x7f2b5c15cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5b400640 1 -- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b5c0adf20 msgr2=0x7f2b5c15a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5b400640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b5c0adf20 0x7f2b5c15a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5b400640 1 -- 172.21.15.53:0/2651260966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b440094d0 con 0x7f2b5c0cc980 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b5b400640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 0x7f2b5c15c5a0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f2b440090b0 tx=0x7f2b440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.122+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44032880 con 0x7f2b5c0cc980 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.123+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b44032a00 con 0x7f2b5c0cc980 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.123+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b44032db0 con 0x7f2b5c0cc980 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.123+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c0b3500 con 0x7f2b5c0cc980 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.123+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c0b3ac0 con 0x7f2b5c0cc980 2024-01-16T15:15:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.124+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c13a1c0 con 0x7f2b5c0cc980 2024-01-16T15:15:42.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.125+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f2b4404e450 con 0x7f2b5c0cc980 2024-01-16T15:15:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.126+0000 7f2b53e00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b3406c160 0x7f2b3406e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.126+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2b440aab90 con 0x7f2b5c0cc980 2024-01-16T15:15:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.127+0000 7f2b5be00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b3406c160 0x7f2b3406e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.128+0000 7f2b5be00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b3406c160 0x7f2b3406e620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006fd0 tx=0x7f2b4c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.130+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b44077170 con 0x7f2b5c0cc980 2024-01-16T15:15:42.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.353+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b5c15ee90 con 0x7f2b3406c160 2024-01-16T15:15:42.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.364+0000 7f2b53e00640 1 -- 172.21.15.53:0/2651260966 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f2b4c00f860 con 0x7f2b3406c160 2024-01-16T15:15:42.349 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b3406c160 msgr2=0x7f2b3406e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b3406c160 0x7f2b3406e620 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c006fd0 tx=0x7f2b4c008040 comp rx=0 tx=0).stop 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 msgr2=0x7f2b5c15c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 0x7f2b5c15c5a0 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f2b440090b0 tx=0x7f2b440046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 shutdown_connections 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b3406c160 0x7f2b3406e620 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b5c15bf10 0x7f2b5c15cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b5c0cc980 0x7f2b5c15c5a0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 --2- 172.21.15.53:0/2651260966 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b5c0adf20 0x7f2b5c15a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.367+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 >> 172.21.15.53:0/2651260966 conn(0x7f2b5c06a7d0 msgr2=0x7f2b5c0aec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.368+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 shutdown_connections 2024-01-16T15:15:42.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.368+0000 7f2b61c00640 1 -- 172.21.15.53:0/2651260966 wait complete. 2024-01-16T15:15:42.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:15:42.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.614+0000 7f78f3e00640 1 Processor -- start 2024-01-16T15:15:42.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.614+0000 7f78f3e00640 1 -- start start 2024-01-16T15:15:42.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.615+0000 7f78f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f40cf9a0 0x7f78f40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.615+0000 7f78f3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40d0320 con 0x7f78f415d4b0 2024-01-16T15:15:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.615+0000 7f78f3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78f40d1c60 con 0x7f78f40b31d0 2024-01-16T15:15:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.615+0000 7f78f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f40b2730 con 0x7f78f40cf9a0 2024-01-16T15:15:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.616+0000 7f78f2a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f78f415d4b0 0x7f78f40d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:56356/0 (socket says 172.21.15.53:56356) 2024-01-16T15:15:42.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.616+0000 7f78f2a00640 1 -- 172.21.15.53:0/3603639841 learned_addr learned my addr 172.21.15.53:0/3603639841 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.616+0000 7f78f2000640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f40cf9a0 0x7f78f40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.616+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144082487 0 0) 0x7f78f40d0320 con 0x7f78f415d4b0 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.616+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78d4003520 con 0x7f78f415d4b0 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.616+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2332902859 0 0) 0x7f78d4003520 con 0x7f78f415d4b0 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f40cf9a0 msgr2=0x7f78f40cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f40cf9a0 0x7f78f40cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 >> v1:172.21.15.146:6789/0 conn(0x7f78f40b31d0 legacy=0x7f78f40b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f40d1ec0 con 0x7f78f415d4b0 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f78e4001670 con 0x7f78f415d4b0 2024-01-16T15:15:42.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 >> v1:172.21.15.53:6789/0 conn(0x7f78f415d4b0 legacy=0x7f78f40d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40039a0 0x7f78d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d40047c0 0x7f78d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d4005180 0x7f78d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f40b2730 con 0x7f78d40047c0 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d400c080 con 0x7f78d4005180 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d400c240 con 0x7f78d40039a0 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f2000640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78f40cf9a0 0x7f78f40cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f2000640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d40047c0 0x7f78d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.617+0000 7f78f3400640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d4005180 0x7f78d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.618+0000 7f78f2a00640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40039a0 0x7f78d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.618+0000 7f78f2000640 1 -- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40039a0 msgr2=0x7f78d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.618+0000 7f78f2000640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40039a0 0x7f78d4003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.618+0000 7f78f2000640 1 -- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d4005180 msgr2=0x7f78d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.618+0000 7f78f2000640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78d4005180 0x7f78d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.618+0000 7f78f2000640 1 -- 172.21.15.53:0/3603639841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d4003520 con 0x7f78d40047c0 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.619+0000 7f78f2000640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d40047c0 0x7f78d4004c40 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f78dc006fd0 tx=0x7f78d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f8d5960b39a441f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.619+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc033600 con 0x7f78d40047c0 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.619+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc02f050 con 0x7f78d40047c0 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.619+0000 7f78f1600640 1 -- 172.21.15.53:0/3603639841 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc033980 con 0x7f78d40047c0 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3e00640 1 -- 172.21.15.53:0/3603639841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d40047c0 msgr2=0x7f78d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3e00640 1 --2- 172.21.15.53:0/3603639841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78d40047c0 0x7f78d4004c40 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f78dc006fd0 tx=0x7f78d40057f0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3400640 1 -- 172.21.15.53:0/3603639841 reap_dead start 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3e00640 1 -- 172.21.15.53:0/3603639841 shutdown_connections 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3e00640 1 -- 172.21.15.53:0/3603639841 >> 172.21.15.53:0/3603639841 conn(0x7f78f40d3000 msgr2=0x7f78f40d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3e00640 1 -- 172.21.15.53:0/3603639841 shutdown_connections 2024-01-16T15:15:42.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.620+0000 7f78f3e00640 1 -- 172.21.15.53:0/3603639841 wait complete. 2024-01-16T15:15:42.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.622+0000 7f78f3e00640 1 Processor -- start 2024-01-16T15:15:42.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.623+0000 7f78f3e00640 1 -- start start 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40047c0 0x7f78f40ca0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 0x7f78f40cdfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4151070 0x7f78f41514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f4151a30 con 0x7f78f4154540 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f40ca610 con 0x7f78f4151070 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78f41534a0 con 0x7f78d40047c0 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 0x7f78f40cdfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 0x7f78f40cdfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46060/0 (socket says 172.21.15.53:46060) 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f2000640 1 -- 172.21.15.53:0/142591733 learned_addr learned my addr 172.21.15.53:0/142591733 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f3400640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4151070 0x7f78f41514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.624+0000 7f78f2a00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40047c0 0x7f78f40ca0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2000640 1 -- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40047c0 msgr2=0x7f78f40ca0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2000640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40047c0 0x7f78f40ca0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2000640 1 -- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4151070 msgr2=0x7f78f41514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2000640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4151070 0x7f78f41514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2000640 1 -- 172.21.15.53:0/142591733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f4153780 con 0x7f78f4154540 2024-01-16T15:15:42.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2a00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40047c0 0x7f78f40ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:15:42.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f2000640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 0x7f78f40cdfb0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f78f40b5b90 tx=0x7f78dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc03d710 con 0x7f78f4154540 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78dc03d870 con 0x7f78f4154540 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78dc03dc00 con 0x7f78f4154540 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f4153a60 con 0x7f78f4154540 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.625+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f4154020 con 0x7f78f4154540 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.627+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f4067450 con 0x7f78f4154540 2024-01-16T15:15:42.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.628+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 35) v1 ==== 90865+0+0 (secure 0 0 0) 0x7f78dc032420 con 0x7f78f4154540 2024-01-16T15:15:42.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.628+0000 7f78eb400640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f78cc06c190 0x7f78cc06e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:15:42.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.629+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f78dc0aa1f0 con 0x7f78f4154540 2024-01-16T15:15:42.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.629+0000 7f78f2a00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f78cc06c190 0x7f78cc06e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:15:42.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.631+0000 7f78f2a00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f78cc06c190 0x7f78cc06e650 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f78e4007910 tx=0x7f78e4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:15:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.634+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f78dc076ab0 con 0x7f78f4154540 2024-01-16T15:15:42.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.858+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f78f40d21a0 con 0x7f78cc06c190 2024-01-16T15:15:42.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.869+0000 7f78eb400640 1 -- 172.21.15.53:0/142591733 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f78e400d040 con 0x7f78cc06c190 2024-01-16T15:15:42.853 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.871+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f78cc06c190 msgr2=0x7f78cc06e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.871+0000 7f78f3e00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f78cc06c190 0x7f78cc06e650 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f78e4007910 tx=0x7f78e4009040 comp rx=0 tx=0).stop 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.871+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 msgr2=0x7f78f40cdfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.871+0000 7f78f3e00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 0x7f78f40cdfb0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f78f40b5b90 tx=0x7f78dc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 shutdown_connections 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f78cc06c190 0x7f78cc06e650 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f78f4151070 0x7f78f41514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f78f4154540 0x7f78f40cdfb0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 --2- 172.21.15.53:0/142591733 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f78d40047c0 0x7f78f40ca0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 >> 172.21.15.53:0/142591733 conn(0x7f78f40d3000 msgr2=0x7f78f406b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 shutdown_connections 2024-01-16T15:15:42.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:15:42.872+0000 7f78f3e00640 1 -- 172.21.15.53:0/142591733 wait complete. 2024-01-16T15:15:42.874 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:15:42.875 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:15:42.875 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:42.930 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:42.930 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:15:54.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:54.670 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:15:58.364 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:15:58.364 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:15:58.364 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 15.3552 s, 69.9 MB/s 2024-01-16T15:15:58.366 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:15:58.366 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:15:58.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:15:58.422 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:16:13.147 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:16:13.147 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:16:13.147 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.6613 s, 73.2 MB/s 2024-01-16T15:16:13.149 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:16:13.149 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:16:13.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:13.206 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:16:24.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:24.811 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:27.389 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:16:27.389 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:16:27.389 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.1236 s, 76.0 MB/s 2024-01-16T15:16:27.391 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:16:27.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.635+0000 7f49a7800640 1 Processor -- start 2024-01-16T15:16:27.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.635+0000 7f49a7800640 1 -- start start 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f49a7800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49a00ae5a0 0x7f49a015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f49a7800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 0x7f49a00cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f49a7800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49a0070c40 con 0x7f49a00b3ae0 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f49a7800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49a0071450 con 0x7f49a00ae5a0 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f49a7800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00cfc60 con 0x7f49a00cf290 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f499fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f49a00b3ae0 0x7f49a006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:38066/0 (socket says 172.21.15.53:38066) 2024-01-16T15:16:27.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.637+0000 7f499fe00640 1 -- 172.21.15.53:0/2138640540 learned_addr learned my addr 172.21.15.53:0/2138640540 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:16:27.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f49a5800640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 0x7f49a00cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f49a4e00640 1 --2- 172.21.15.53:0/2138640540 >> v2:172.21.15.146:3300/0 conn(0x7f49a00ae5a0 0x7f49a015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579001261 0 0) 0x7f49a0070c40 con 0x7f49a00b3ae0 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4980003520 con 0x7f49a00b3ae0 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2449762290 0 0) 0x7f4980003520 con 0x7f49a00b3ae0 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 msgr2=0x7f49a00cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 0x7f49a00cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 >> v2:172.21.15.146:3300/0 conn(0x7f49a00ae5a0 msgr2=0x7f49a015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 --2- 172.21.15.53:0/2138640540 >> v2:172.21.15.146:3300/0 conn(0x7f49a00ae5a0 0x7f49a015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.638+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a006c470 con 0x7f49a00b3ae0 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4988001670 con 0x7f49a00b3ae0 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 >> v1:172.21.15.53:6789/0 conn(0x7f49a00b3ae0 legacy=0x7f49a006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 0x7f4980003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4980003ef0 0x7f49800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4980004d00 0x7f4980005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a0071450 con 0x7f4980004d00 2024-01-16T15:16:27.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00cfc60 con 0x7f4980003ef0 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f498000b9b0 con 0x7f49a00cf290 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f49a5800640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4980004d00 0x7f4980005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f49a4e00640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 0x7f4980003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.639+0000 7f499fe00640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4980003ef0 0x7f49800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.640+0000 7f49a5800640 1 -- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 msgr2=0x7f4980003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.640+0000 7f49a5800640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49a00cf290 0x7f4980003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.640+0000 7f49a5800640 1 -- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4980003ef0 msgr2=0x7f49800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.640+0000 7f49a5800640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4980003ef0 0x7f49800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.640+0000 7f49a5800640 1 -- 172.21.15.53:0/2138640540 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4980003520 con 0x7f4980004d00 2024-01-16T15:16:27.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.641+0000 7f49a5800640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4980004d00 0x7f4980005180 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f4998009fc0 tx=0x7f49800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46dcdbf08064f44e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:27.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.641+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499802cc20 con 0x7f4980004d00 2024-01-16T15:16:27.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.641+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499800f040 con 0x7f4980004d00 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.641+0000 7f499f400640 1 -- 172.21.15.53:0/2138640540 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4998034440 con 0x7f4980004d00 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a7800640 1 -- 172.21.15.53:0/2138640540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4980004d00 msgr2=0x7f4980005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a7800640 1 --2- 172.21.15.53:0/2138640540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4980004d00 0x7f4980005180 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f4998009fc0 tx=0x7f49800057f0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a5800640 1 -- 172.21.15.53:0/2138640540 reap_dead start 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a7800640 1 -- 172.21.15.53:0/2138640540 shutdown_connections 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a7800640 1 -- 172.21.15.53:0/2138640540 >> 172.21.15.53:0/2138640540 conn(0x7f49a006a800 msgr2=0x7f49a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a7800640 1 -- 172.21.15.53:0/2138640540 shutdown_connections 2024-01-16T15:16:27.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.642+0000 7f49a7800640 1 -- 172.21.15.53:0/2138640540 wait complete. 2024-01-16T15:16:27.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.644+0000 7f49a7800640 1 Processor -- start 2024-01-16T15:16:27.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.644+0000 7f49a7800640 1 -- start start 2024-01-16T15:16:27.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.645+0000 7f49a7800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4980004d00 0x7f49a0148b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.645+0000 7f49a7800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 0x7f49a015fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.645+0000 7f49a7800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00cc8b0 0x7f49a00ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.645+0000 7f49a7800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a00cd680 con 0x7f49a015fb70 2024-01-16T15:16:27.631 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.645+0000 7f49a7800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49a0157ce0 con 0x7f49a00cc8b0 2024-01-16T15:16:27.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.645+0000 7f49a7800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49a0160530 con 0x7f4980004d00 2024-01-16T15:16:27.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 0x7f49a015fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 0x7f49a015fff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36050/0 (socket says 172.21.15.53:36050) 2024-01-16T15:16:27.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 -- 172.21.15.53:0/2188809571 learned_addr learned my addr 172.21.15.53:0/2188809571 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f49a5800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00cc8b0 0x7f49a00ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 -- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4980004d00 msgr2=0x7f49a0148b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4980004d00 0x7f49a0148b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 -- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00cc8b0 msgr2=0x7f49a00ccd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00cc8b0 0x7f49a00ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.646+0000 7f499fe00640 1 -- 172.21.15.53:0/2188809571 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0157340 con 0x7f49a015fb70 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.647+0000 7f499fe00640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 0x7f49a015fff0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f498800e870 tx=0x7f498800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.647+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800c8d0 con 0x7f49a015fb70 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.647+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800ca30 con 0x7f49a015fb70 2024-01-16T15:16:27.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.647+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49a0157620 con 0x7f49a015fb70 2024-01-16T15:16:27.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.647+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800cde0 con 0x7f49a015fb70 2024-01-16T15:16:27.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.647+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49a0149070 con 0x7f49a015fb70 2024-01-16T15:16:27.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.648+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49a00675d0 con 0x7f49a015fb70 2024-01-16T15:16:27.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.649+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 36) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f498801d460 con 0x7f49a015fb70 2024-01-16T15:16:27.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.650+0000 7f499cc00640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f497806c1f0 0x7f497806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:27.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.650+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4988088a90 con 0x7f49a015fb70 2024-01-16T15:16:27.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.651+0000 7f49a4e00640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f497806c1f0 0x7f497806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:27.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.652+0000 7f49a4e00640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f497806c1f0 0x7f497806e6b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4990009990 tx=0x7f4990008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:27.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.656+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4988055060 con 0x7f49a015fb70 2024-01-16T15:16:27.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.880+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f49a0071170 con 0x7f497806c1f0 2024-01-16T15:16:27.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.916+0000 7f499cc00640 1 -- 172.21.15.53:0/2188809571 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4990011860 con 0x7f497806c1f0 2024-01-16T15:16:27.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.919+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f497806c1f0 msgr2=0x7f497806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.919+0000 7f49a7800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f497806c1f0 0x7f497806e6b0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4990009990 tx=0x7f4990008040 comp rx=0 tx=0).stop 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.919+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 msgr2=0x7f49a015fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.919+0000 7f49a7800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 0x7f49a015fff0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f498800e870 tx=0x7f498800ed30 comp rx=0 tx=0).stop 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 shutdown_connections 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f497806c1f0 0x7f497806e6b0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49a00cc8b0 0x7f49a00ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49a015fb70 0x7f49a015fff0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 --2- 172.21.15.53:0/2188809571 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4980004d00 0x7f49a0148b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 >> 172.21.15.53:0/2188809571 conn(0x7f49a006a800 msgr2=0x7f49a0160f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 shutdown_connections 2024-01-16T15:16:27.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:27.920+0000 7f49a7800640 1 -- 172.21.15.53:0/2188809571 wait complete. 2024-01-16T15:16:27.924 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:16:28.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.163+0000 7f1e6ee00640 1 Processor -- start 2024-01-16T15:16:28.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.163+0000 7f1e6ee00640 1 -- start start 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.164+0000 7f1e6ee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e6814b2c0 0x7f1e680cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e6ee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e68071070 0x7f1e680cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e6ee00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e680cfbf0 con 0x7f1e680cf7d0 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e6ee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e6814b6a0 con 0x7f1e6814b2c0 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e6ee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e68071690 con 0x7f1e68071070 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e67e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e6814b2c0 0x7f1e680cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e67e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e6814b2c0 0x7f1e680cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36056/0 (socket says 172.21.15.53:36056) 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e67e00640 1 -- 172.21.15.53:0/2892301276 learned_addr learned my addr 172.21.15.53:0/2892301276 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:28.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.165+0000 7f1e6ce00640 1 --2- 172.21.15.53:0/2892301276 >> v2:172.21.15.146:3300/0 conn(0x7f1e68071070 0x7f1e680cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:28.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.166+0000 7f1e67e00640 1 -- 172.21.15.53:0/2892301276 >> v2:172.21.15.146:3300/0 conn(0x7f1e68071070 msgr2=0x7f1e680cc160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:28.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.166+0000 7f1e67e00640 1 --2- 172.21.15.53:0/2892301276 >> v2:172.21.15.146:3300/0 conn(0x7f1e68071070 0x7f1e680cc160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.166+0000 7f1e67e00640 1 -- 172.21.15.53:0/2892301276 >> v1:172.21.15.146:6789/0 conn(0x7f1e680cf7d0 legacy=0x7f1e68070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:28.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.166+0000 7f1e67e00640 1 -- 172.21.15.53:0/2892301276 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e68071970 con 0x7f1e6814b2c0 2024-01-16T15:16:28.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.166+0000 7f1e6ce00640 1 --2- 172.21.15.53:0/2892301276 >> v2:172.21.15.146:3300/0 conn(0x7f1e68071070 0x7f1e680cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:28.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.167+0000 7f1e67e00640 1 --2- 172.21.15.53:0/2892301276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e6814b2c0 0x7f1e680cf290 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1e68165330 tx=0x7f1e5802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b9f901c62ebd0a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:28.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.167+0000 7f1e67400640 1 -- 172.21.15.53:0/2892301276 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58031890 con 0x7f1e6814b2c0 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.167+0000 7f1e67400640 1 -- 172.21.15.53:0/2892301276 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e580319f0 con 0x7f1e6814b2c0 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.167+0000 7f1e67400640 1 -- 172.21.15.53:0/2892301276 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58031d80 con 0x7f1e6814b2c0 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 -- 172.21.15.53:0/2892301276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e6814b2c0 msgr2=0x7f1e680cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/2892301276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e6814b2c0 0x7f1e680cf290 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f1e68165330 tx=0x7f1e5802b930 comp rx=0 tx=0).stop 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 -- 172.21.15.53:0/2892301276 shutdown_connections 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/2892301276 >> v2:172.21.15.146:3300/0 conn(0x7f1e68071070 0x7f1e680cc160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/2892301276 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e6814b2c0 0x7f1e680cf290 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 -- 172.21.15.53:0/2892301276 >> 172.21.15.53:0/2892301276 conn(0x7f1e6806a800 msgr2=0x7f1e6806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 -- 172.21.15.53:0/2892301276 shutdown_connections 2024-01-16T15:16:28.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.168+0000 7f1e6ee00640 1 -- 172.21.15.53:0/2892301276 wait complete. 2024-01-16T15:16:28.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.170+0000 7f1e6ee00640 1 Processor -- start 2024-01-16T15:16:28.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.170+0000 7f1e6ee00640 1 -- start start 2024-01-16T15:16:28.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e6ee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 0x7f1e68161e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:28.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e6ee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e680cf7d0 0x7f1e68163560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e6ee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e6814b2c0 0x7f1e680a9f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e6ee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e68160e00 con 0x7f1e68071070 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e6ee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e68142510 con 0x7f1e680cf7d0 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e6ee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e681624c0 con 0x7f1e6814b2c0 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 0x7f1e68161e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e67e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 0x7f1e68161e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36072/0 (socket says 172.21.15.53:36072) 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.171+0000 7f1e67e00640 1 -- 172.21.15.53:0/532250538 learned_addr learned my addr 172.21.15.53:0/532250538 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e67e00640 1 -- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e6814b2c0 msgr2=0x7f1e680a9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e67e00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e6814b2c0 0x7f1e680a9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e67e00640 1 -- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e680cf7d0 msgr2=0x7f1e68163560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e67e00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e680cf7d0 0x7f1e68163560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e67e00640 1 -- 172.21.15.53:0/532250538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e580094d0 con 0x7f1e68071070 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e67e00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 0x7f1e68161e70 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f1e58000c00 tx=0x7f1e580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e58031840 con 0x7f1e68071070 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.172+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e580443f0 con 0x7f1e68071070 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.173+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e580447a0 con 0x7f1e68071070 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.173+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e681627a0 con 0x7f1e68071070 2024-01-16T15:16:28.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.173+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e68162d60 con 0x7f1e68071070 2024-01-16T15:16:28.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.174+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e680675d0 con 0x7f1e68071070 2024-01-16T15:16:28.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.175+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 36) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f1e5804d450 con 0x7f1e68071070 2024-01-16T15:16:28.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.176+0000 7f1e64c00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4006c1f0 0x7f1e4006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:28.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.176+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f1e580aaf30 con 0x7f1e68071070 2024-01-16T15:16:28.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.176+0000 7f1e5fe00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4006c1f0 0x7f1e4006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:28.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.178+0000 7f1e5fe00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4006c1f0 0x7f1e4006e6b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1e681fa500 tx=0x7f1e50007990 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:28.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.180+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e580777f0 con 0x7f1e68071070 2024-01-16T15:16:28.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.402+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e680cbc40 con 0x7f1e4006c1f0 2024-01-16T15:16:28.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.413+0000 7f1e64c00640 1 -- 172.21.15.53:0/532250538 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f1e500050d0 con 0x7f1e4006c1f0 2024-01-16T15:16:28.399 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.415+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4006c1f0 msgr2=0x7f1e4006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.415+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4006c1f0 0x7f1e4006e6b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f1e681fa500 tx=0x7f1e50007990 comp rx=0 tx=0).stop 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.415+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 msgr2=0x7f1e68161e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.415+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 0x7f1e68161e70 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f1e58000c00 tx=0x7f1e580046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 shutdown_connections 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4006c1f0 0x7f1e4006e6b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e6814b2c0 0x7f1e680a9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e680cf7d0 0x7f1e68163560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 --2- 172.21.15.53:0/532250538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e68071070 0x7f1e68161e70 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:28.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 >> 172.21.15.53:0/532250538 conn(0x7f1e6806a800 msgr2=0x7f1e6806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:28.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 shutdown_connections 2024-01-16T15:16:28.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:28.416+0000 7f1e6ee00640 1 -- 172.21.15.53:0/532250538 wait complete. 2024-01-16T15:16:28.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:28.422 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a) 2024-01-16T15:16:28.481 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:16:28.481 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:16:28.482 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:16:28.483 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:16:30.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:30.483 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a) 2024-01-16T15:16:30.549 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:16:30.549 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:16:30.550 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:16:30.550 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:16:32.552 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:16:32.552 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a) 2024-01-16T15:16:32.621 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/625670eb-e79e-42b9-ad02-aa932888c11a 2024-01-16T15:16:32.622 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:16:32.622 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:16:32.623 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:16:32.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66971 2024-01-16T15:16:32.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.867+0000 7f50e5400640 1 Processor -- start 2024-01-16T15:16:32.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.867+0000 7f50e5400640 1 -- start start 2024-01-16T15:16:32.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50e5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 0x7f50e015afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50e5400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f50e013fcb0 0x7f50e00b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50e5400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50e00cdec0 con 0x7f50e00cd530 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50e5400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50e00ce1a0 con 0x7f50e013fcb0 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50e5400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00ce480 con 0x7f50e013cfa0 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50df400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f50e00cd530 0x7f50e013f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39140/0 (socket says 172.21.15.53:39140) 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50df400640 1 -- 172.21.15.53:0/111849361 learned_addr learned my addr 172.21.15.53:0/111849361 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50dfe00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 0x7f50e015afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.869+0000 7f50e4a00640 1 --2- 172.21.15.53:0/111849361 >> v2:172.21.15.146:3300/0 conn(0x7f50e013fcb0 0x7f50e00b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955471613 0 0) 0x7f50e00cdec0 con 0x7f50e00cd530 2024-01-16T15:16:32.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c0005730 con 0x7f50e00cd530 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291692041 0 0) 0x7f50c0005730 con 0x7f50e00cd530 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 msgr2=0x7f50e015afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 0x7f50e015afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 >> v2:172.21.15.146:3300/0 conn(0x7f50e013fcb0 msgr2=0x7f50e00b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 --2- 172.21.15.53:0/111849361 >> v2:172.21.15.146:3300/0 conn(0x7f50e013fcb0 0x7f50e00b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e00ce760 con 0x7f50e00cd530 2024-01-16T15:16:32.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f50c8001670 con 0x7f50e00cd530 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 >> v1:172.21.15.53:6789/0 conn(0x7f50e00cd530 legacy=0x7f50e013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50c0005b80 0x7f50c0005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c00069d0 0x7f50c0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 0x7f50c0007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00ce1a0 con 0x7f50c0005b80 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e00ce480 con 0x7f50c00069d0 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.870+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c000dbe0 con 0x7f50e013cfa0 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50dfe00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50c0005b80 0x7f50c0005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50df400640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c00069d0 0x7f50c0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50dfe00640 1 -- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 msgr2=0x7f50c0007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50e4a00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 0x7f50c0007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50dfe00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50e013cfa0 0x7f50c0007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50dfe00640 1 -- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c00069d0 msgr2=0x7f50c0006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50dfe00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c00069d0 0x7f50c0006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50dfe00640 1 -- 172.21.15.53:0/111849361 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0005730 con 0x7f50c0005b80 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.871+0000 7f50df400640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50c00069d0 0x7f50c0006e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.872+0000 7f50dfe00640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50c0005b80 0x7f50c0005fa0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009af0 tx=0x7f50c0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d993059da062e957 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.873+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d002c6b0 con 0x7f50c0005b80 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.873+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d002c810 con 0x7f50c0005b80 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.873+0000 7f50dea00640 1 -- 172.21.15.53:0/111849361 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d002cba0 con 0x7f50c0005b80 2024-01-16T15:16:32.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.873+0000 7f50e5400640 1 -- 172.21.15.53:0/111849361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50c0005b80 msgr2=0x7f50c0005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.873+0000 7f50e5400640 1 --2- 172.21.15.53:0/111849361 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50c0005b80 0x7f50c0005fa0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f50d0009af0 tx=0x7f50c0007a00 comp rx=0 tx=0).stop 2024-01-16T15:16:32.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.873+0000 7f50e4a00640 1 -- 172.21.15.53:0/111849361 reap_dead start 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.874+0000 7f50e5400640 1 -- 172.21.15.53:0/111849361 shutdown_connections 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.874+0000 7f50e5400640 1 -- 172.21.15.53:0/111849361 >> 172.21.15.53:0/111849361 conn(0x7f50e006a800 msgr2=0x7f50e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.874+0000 7f50e5400640 1 -- 172.21.15.53:0/111849361 shutdown_connections 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.874+0000 7f50e5400640 1 -- 172.21.15.53:0/111849361 wait complete. 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.875+0000 7f50e5400640 1 Processor -- start 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.876+0000 7f50e5400640 1 -- start start 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.876+0000 7f50e5400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c0005b80 0x7f50e01572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50e5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 0x7f50e013a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50e5400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50e013b590 0x7f50e013ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50e5400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e01f76f0 con 0x7f50e014e100 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50e5400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e013bf60 con 0x7f50e013b590 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50e5400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50e0157810 con 0x7f50c0005b80 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 0x7f50e013a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 0x7f50e013a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53314/0 (socket says 172.21.15.53:53314) 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 -- 172.21.15.53:0/4031340218 learned_addr learned my addr 172.21.15.53:0/4031340218 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50dfe00640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c0005b80 0x7f50e01572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50e4a00640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50e013b590 0x7f50e013ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 -- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c0005b80 msgr2=0x7f50e01572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c0005b80 0x7f50e01572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 -- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50e013b590 msgr2=0x7f50e013ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50e013b590 0x7f50e013ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.877+0000 7f50df400640 1 -- 172.21.15.53:0/4031340218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e01f6a70 con 0x7f50e014e100 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.878+0000 7f50df400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 0x7f50e013a8d0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f50c800d770 tx=0x7f50c800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.878+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80072d0 con 0x7f50e014e100 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.878+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c8007430 con 0x7f50e014e100 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.878+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e01f6d50 con 0x7f50e014e100 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.878+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c80047e0 con 0x7f50e014e100 2024-01-16T15:16:32.867 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.878+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e01f7310 con 0x7f50e014e100 2024-01-16T15:16:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.880+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e0067450 con 0x7f50e014e100 2024-01-16T15:16:32.868 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.881+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 36) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f50c8004940 con 0x7f50e014e100 2024-01-16T15:16:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.882+0000 7f50d7e00640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50b806c1f0 0x7f50b806e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.882+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f50c8088350 con 0x7f50e014e100 2024-01-16T15:16:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.882+0000 7f50dfe00640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50b806c1f0 0x7f50b806e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:32.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.884+0000 7f50dfe00640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50b806c1f0 0x7f50b806e6b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f50d0036000 tx=0x7f50d0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:32.872 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:32.887+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f50c80586a0 con 0x7f50e014e100 2024-01-16T15:16:33.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.113+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66971", "target": ["mon-mgr", ""]}) v1 -- 0x7f50e006ac10 con 0x7f50b806c1f0 2024-01-16T15:16:33.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.264+0000 7f50d7e00640 1 -- 172.21.15.53:0/4031340218 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f50e006ac10 con 0x7f50b806c1f0 2024-01-16T15:16:33.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.267+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50b806c1f0 msgr2=0x7f50b806e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.267+0000 7f50e5400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50b806c1f0 0x7f50b806e6b0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f50d0036000 tx=0x7f50d0030000 comp rx=0 tx=0).stop 2024-01-16T15:16:33.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.267+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 msgr2=0x7f50e013a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.267+0000 7f50e5400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 0x7f50e013a8d0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f50c800d770 tx=0x7f50c800dc30 comp rx=0 tx=0).stop 2024-01-16T15:16:33.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 shutdown_connections 2024-01-16T15:16:33.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f50b806c1f0 0x7f50b806e6b0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f50e013b590 0x7f50e013ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f50e014e100 0x7f50e013a8d0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 --2- 172.21.15.53:0/4031340218 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f50c0005b80 0x7f50e01572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 >> 172.21.15.53:0/4031340218 conn(0x7f50e006a800 msgr2=0x7f50e00b0020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 shutdown_connections 2024-01-16T15:16:33.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.268+0000 7f50e5400640 1 -- 172.21.15.53:0/4031340218 wait complete. 2024-01-16T15:16:33.257 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.271+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.258 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.271+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.258 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.271+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.258 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.271+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.258 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.271+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66972 2024-01-16T15:16:33.295 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.308+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.295 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.309+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.296 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.309+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.296 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.309+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.296 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:33.309+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.504+0000 7f2bc9800640 1 Processor -- start 2024-01-16T15:16:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.504+0000 7f2bc9800640 1 -- start start 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc9800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2bc4143c40 0x7f2bc4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc9800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc40ccc10 0x7f2bc406d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc9800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bc4070e50 con 0x7f2bc40717d0 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc9800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bc40723c0 con 0x7f2bc4143c40 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc9800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc4144020 con 0x7f2bc40ccc10 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2bc4143c40 0x7f2bc4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2bc4143c40 0x7f2bc4071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53328/0 (socket says 172.21.15.53:53328) 2024-01-16T15:16:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.506+0000 7f2bc3e00640 1 -- 172.21.15.53:0/1659799510 learned_addr learned my addr 172.21.15.53:0/1659799510 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:33.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.507+0000 7f2bc3e00640 1 -- 172.21.15.53:0/1659799510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc40ccc10 msgr2=0x7f2bc406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:33.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.507+0000 7f2bc8e00640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc40ccc10 0x7f2bc406d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.507+0000 7f2bc3e00640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc40ccc10 0x7f2bc406d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.507+0000 7f2bc3e00640 1 -- 172.21.15.53:0/1659799510 >> v1:172.21.15.146:6789/0 conn(0x7f2bc40717d0 legacy=0x7f2bc40cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:33.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.507+0000 7f2bc3e00640 1 -- 172.21.15.53:0/1659799510 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bc4071e00 con 0x7f2bc4143c40 2024-01-16T15:16:33.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.507+0000 7f2bc8e00640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc40ccc10 0x7f2bc406d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.508+0000 7f2bc3e00640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc4143c40 0x7f2bc4071290 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009870 tx=0x7f2bb402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c653a92ea3c50edd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.508+0000 7f2bc2a00640 1 -- 172.21.15.53:0/1659799510 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40317c0 con 0x7f2bc4143c40 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.508+0000 7f2bc2a00640 1 -- 172.21.15.53:0/1659799510 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb4031920 con 0x7f2bc4143c40 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.508+0000 7f2bc2a00640 1 -- 172.21.15.53:0/1659799510 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb4031cb0 con 0x7f2bc4143c40 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 -- 172.21.15.53:0/1659799510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc4143c40 msgr2=0x7f2bc4071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc4143c40 0x7f2bc4071290 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4009870 tx=0x7f2bb402b860 comp rx=0 tx=0).stop 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 -- 172.21.15.53:0/1659799510 shutdown_connections 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc40ccc10 0x7f2bc406d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 --2- 172.21.15.53:0/1659799510 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc4143c40 0x7f2bc4071290 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 -- 172.21.15.53:0/1659799510 >> 172.21.15.53:0/1659799510 conn(0x7f2bc406a800 msgr2=0x7f2bc406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 -- 172.21.15.53:0/1659799510 shutdown_connections 2024-01-16T15:16:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.509+0000 7f2bc9800640 1 -- 172.21.15.53:0/1659799510 wait complete. 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.511+0000 7f2bc9800640 1 Processor -- start 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.511+0000 7f2bc9800640 1 -- start start 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc9800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 0x7f2bc412eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc9800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc40ccc10 0x7f2bc40ca2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc9800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc4143c40 0x7f2bc40ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc9800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc412b5d0 con 0x7f2bc40717d0 2024-01-16T15:16:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc9800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc412b8b0 con 0x7f2bc40ccc10 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc9800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bc412bb90 con 0x7f2bc4143c40 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 0x7f2bc412eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 0x7f2bc412eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53334/0 (socket says 172.21.15.53:53334) 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc3e00640 1 -- 172.21.15.53:0/2574034988 learned_addr learned my addr 172.21.15.53:0/2574034988 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc8e00640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc4143c40 0x7f2bc40ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc3e00640 1 -- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc4143c40 msgr2=0x7f2bc40ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.512+0000 7f2bc3e00640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc4143c40 0x7f2bc40ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc3e00640 1 -- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc40ccc10 msgr2=0x7f2bc40ca2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc3400640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc40ccc10 0x7f2bc40ca2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc3e00640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc40ccc10 0x7f2bc40ca2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc3e00640 1 -- 172.21.15.53:0/2574034988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb40094d0 con 0x7f2bc40717d0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc3400640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc40ccc10 0x7f2bc40ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc3e00640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 0x7f2bc412eac0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4000c00 tx=0x7f2bb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40329a0 con 0x7f2bc40717d0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bb4032b20 con 0x7f2bc40717d0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bb40453f0 con 0x7f2bc40717d0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.513+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bc412be70 con 0x7f2bc40717d0 2024-01-16T15:16:33.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.514+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bc412c430 con 0x7f2bc40717d0 2024-01-16T15:16:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.515+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bc4067450 con 0x7f2bc40717d0 2024-01-16T15:16:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.516+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 36) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f2bb404d3f0 con 0x7f2bc40717d0 2024-01-16T15:16:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.517+0000 7f2bbbe00640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b9c06c1f0 0x7f2b9c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.517+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f2bb40aa680 con 0x7f2bc40717d0 2024-01-16T15:16:33.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.517+0000 7f2bc3400640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b9c06c1f0 0x7f2b9c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:33.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.519+0000 7f2bc3400640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b9c06c1f0 0x7f2b9c06e6b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2bc41f6490 tx=0x7f2bac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:33.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.522+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2bb4076f40 con 0x7f2bc40717d0 2024-01-16T15:16:33.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.763+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66972", "target": ["mon-mgr", ""]}) v1 -- 0x7f2bc412bd00 con 0x7f2b9c06c1f0 2024-01-16T15:16:33.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.923+0000 7f2bbbe00640 1 -- 172.21.15.53:0/2574034988 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2bc412bd00 con 0x7f2b9c06c1f0 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b9c06c1f0 msgr2=0x7f2b9c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b9c06c1f0 0x7f2b9c06e6b0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f2bc41f6490 tx=0x7f2bac008040 comp rx=0 tx=0).stop 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 msgr2=0x7f2bc412eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 0x7f2bc412eac0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f2bb4000c00 tx=0x7f2bb40046c0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 shutdown_connections 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2b9c06c1f0 0x7f2b9c06e6b0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2bc4143c40 0x7f2bc40ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2bc40ccc10 0x7f2bc40ca2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 --2- 172.21.15.53:0/2574034988 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2bc40717d0 0x7f2bc412eac0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:33.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 >> 172.21.15.53:0/2574034988 conn(0x7f2bc406a800 msgr2=0x7f2bc40af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:33.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 shutdown_connections 2024-01-16T15:16:33.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:33.927+0000 7f2bc9800640 1 -- 172.21.15.53:0/2574034988 wait complete. 2024-01-16T15:16:33.936 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone66973 2024-01-16T15:16:34.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.175+0000 7f30ebe00640 1 Processor -- start 2024-01-16T15:16:34.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.175+0000 7f30ebe00640 1 -- start start 2024-01-16T15:16:34.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.176+0000 7f30ebe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f30ec15afb0 0x7f30ec13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.176+0000 7f30ebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec13fae0 0x7f30ec0b2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.176+0000 7f30ebe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30ec13ff30 con 0x7f30ec13cfa0 2024-01-16T15:16:34.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.176+0000 7f30ebe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30ec0cdc40 con 0x7f30ec15afb0 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.176+0000 7f30ebe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec0cdec0 con 0x7f30ec13fae0 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30eaa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f30ec13cfa0 0x7f30ec0cd530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39154/0 (socket says 172.21.15.53:39154) 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30eaa00640 1 -- 172.21.15.53:0/2393122725 learned_addr learned my addr 172.21.15.53:0/2393122725 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30ea000640 1 --2- 172.21.15.53:0/2393122725 >> v2:172.21.15.53:3300/0 conn(0x7f30ec15afb0 0x7f30ec13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30eb400640 1 --2- 172.21.15.53:0/2393122725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec13fae0 0x7f30ec0b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30e9600640 1 -- 172.21.15.53:0/2393122725 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408943353 0 0) 0x7f30dc001490 con 0x7f30ec13cfa0 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30e9600640 1 -- 172.21.15.53:0/2393122725 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30cc003520 con 0x7f30ec13cfa0 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30ea000640 1 -- 172.21.15.53:0/2393122725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec13fae0 msgr2=0x7f30ec0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30ea000640 1 --2- 172.21.15.53:0/2393122725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec13fae0 0x7f30ec0b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30ea000640 1 -- 172.21.15.53:0/2393122725 >> v1:172.21.15.53:6789/0 conn(0x7f30ec13cfa0 legacy=0x7f30ec0cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.177+0000 7f30ea000640 1 -- 172.21.15.53:0/2393122725 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30ec0ce1a0 con 0x7f30ec15afb0 2024-01-16T15:16:34.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.179+0000 7f30ea000640 1 --2- 172.21.15.53:0/2393122725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec15afb0 0x7f30ec13f5a0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f30d40097a0 tx=0x7f30d402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5c053787484dbb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:34.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.179+0000 7f30e9600640 1 -- 172.21.15.53:0/2393122725 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402e070 con 0x7f30ec15afb0 2024-01-16T15:16:34.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.179+0000 7f30e9600640 1 -- 172.21.15.53:0/2393122725 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d40315a0 con 0x7f30ec15afb0 2024-01-16T15:16:34.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.179+0000 7f30e9600640 1 -- 172.21.15.53:0/2393122725 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d4031910 con 0x7f30ec15afb0 2024-01-16T15:16:34.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.179+0000 7f30ebe00640 1 -- 172.21.15.53:0/2393122725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec15afb0 msgr2=0x7f30ec13f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.179+0000 7f30ebe00640 1 --2- 172.21.15.53:0/2393122725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec15afb0 0x7f30ec13f5a0 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f30d40097a0 tx=0x7f30d402b840 comp rx=0 tx=0).stop 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.180+0000 7f30ebe00640 1 -- 172.21.15.53:0/2393122725 shutdown_connections 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.180+0000 7f30ebe00640 1 --2- 172.21.15.53:0/2393122725 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec13fae0 0x7f30ec0b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.180+0000 7f30ebe00640 1 --2- 172.21.15.53:0/2393122725 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec15afb0 0x7f30ec13f5a0 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.180+0000 7f30ebe00640 1 -- 172.21.15.53:0/2393122725 >> 172.21.15.53:0/2393122725 conn(0x7f30ec06a800 msgr2=0x7f30ec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.180+0000 7f30ebe00640 1 -- 172.21.15.53:0/2393122725 shutdown_connections 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.180+0000 7f30ebe00640 1 -- 172.21.15.53:0/2393122725 wait complete. 2024-01-16T15:16:34.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.181+0000 7f30ebe00640 1 Processor -- start 2024-01-16T15:16:34.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.182+0000 7f30ebe00640 1 -- start start 2024-01-16T15:16:34.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.182+0000 7f30ebe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 0x7f30ec153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.182+0000 7f30ebe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30ec13fae0 0x7f30ec138b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.182+0000 7f30ebe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec15afb0 0x7f30ec13a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30ebe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec13a520 con 0x7f30ec13cfa0 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30ebe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec13e400 con 0x7f30ec13fae0 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30ebe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30ec139c50 con 0x7f30ec15afb0 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 0x7f30ec153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 0x7f30ec153d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53350/0 (socket says 172.21.15.53:53350) 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 -- 172.21.15.53:0/878460264 learned_addr learned my addr 172.21.15.53:0/878460264 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30ea000640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30ec13fae0 0x7f30ec138b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eb400640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec15afb0 0x7f30ec13a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 -- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec15afb0 msgr2=0x7f30ec13a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec15afb0 0x7f30ec13a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 -- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30ec13fae0 msgr2=0x7f30ec138b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30ec13fae0 0x7f30ec138b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.183+0000 7f30eaa00640 1 -- 172.21.15.53:0/878460264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d4009400 con 0x7f30ec13cfa0 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.184+0000 7f30eaa00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 0x7f30ec153d90 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00cb40 tx=0x7f30dc00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.184+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc00aca0 con 0x7f30ec13cfa0 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.184+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30dc00ae20 con 0x7f30ec13cfa0 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.184+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30dc012640 con 0x7f30ec13cfa0 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.184+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30ec139060 con 0x7f30ec13cfa0 2024-01-16T15:16:34.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.184+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30ec139e60 con 0x7f30ec13cfa0 2024-01-16T15:16:34.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.185+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30ec13a790 con 0x7f30ec13cfa0 2024-01-16T15:16:34.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.186+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 36) v1 ==== 90916+0+0 (secure 0 0 0) 0x7f30dc007070 con 0x7f30ec13cfa0 2024-01-16T15:16:34.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.187+0000 7f30e2a00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30c406c1f0 0x7f30c406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:34.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.187+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f30dc0886b0 con 0x7f30ec13cfa0 2024-01-16T15:16:34.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.188+0000 7f30ea000640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30c406c1f0 0x7f30c406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:34.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.189+0000 7f30ea000640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30c406c1f0 0x7f30c406e6b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f30d4036000 tx=0x7f30d403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:34.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.192+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f30dc054e70 con 0x7f30ec13cfa0 2024-01-16T15:16:34.406 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:34.420+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone66973", "target": ["mon-mgr", ""]}) v1 -- 0x7f30ec0cea40 con 0x7f30c406c1f0 2024-01-16T15:16:35.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:35.451+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 37) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f30dc055240 con 0x7f30ec13cfa0 2024-01-16T15:16:37.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.376+0000 7f30e2a00640 1 -- 172.21.15.53:0/878460264 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f30d4031a70 con 0x7f30c406c1f0 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.379+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30c406c1f0 msgr2=0x7f30c406e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.379+0000 7f30ebe00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30c406c1f0 0x7f30c406e6b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f30d4036000 tx=0x7f30d403d040 comp rx=0 tx=0).stop 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.379+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 msgr2=0x7f30ec153d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.379+0000 7f30ebe00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 0x7f30ec153d90 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f30dc00cb40 tx=0x7f30dc00a060 comp rx=0 tx=0).stop 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 shutdown_connections 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f30c406c1f0 0x7f30c406e6b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f30ec15afb0 0x7f30ec13a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f30ec13fae0 0x7f30ec138b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 --2- 172.21.15.53:0/878460264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f30ec13cfa0 0x7f30ec153d90 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 >> 172.21.15.53:0/878460264 conn(0x7f30ec06a800 msgr2=0x7f30ec0aea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:37.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 shutdown_connections 2024-01-16T15:16:37.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.380+0000 7f30ebe00640 1 -- 172.21.15.53:0/878460264 wait complete. 2024-01-16T15:16:37.387 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:16:37.387 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66973 2024-01-16T15:16:37.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.624+0000 7f24b4a00640 1 Processor -- start 2024-01-16T15:16:37.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.625+0000 7f24b4a00640 1 -- start start 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.626+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.626+0000 7f24b4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24b015bf10 0x7f24b00cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.626+0000 7f24b4a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b015f4e0 con 0x7f24b00cc980 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.626+0000 7f24b4a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b015f970 con 0x7f24b015bf10 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.626+0000 7f24b4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00cce10 con 0x7f24b00adf20 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ade00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24b015bf10 0x7f24b00cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ade00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f24b015bf10 0x7f24b00cc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53354/0 (socket says 172.21.15.53:53354) 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ade00640 1 -- 172.21.15.53:0/2553514997 learned_addr learned my addr 172.21.15.53:0/2553514997 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:37.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ae800640 1 --2- 172.21.15.53:0/2553514997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ade00640 1 -- 172.21.15.53:0/2553514997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 msgr2=0x7f24b0070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ade00640 1 --2- 172.21.15.53:0/2553514997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b0070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.627+0000 7f24ade00640 1 -- 172.21.15.53:0/2553514997 >> v1:172.21.15.146:6789/0 conn(0x7f24b00cc980 legacy=0x7f24b0143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.628+0000 7f24ade00640 1 -- 172.21.15.53:0/2553514997 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0071130 con 0x7f24b015bf10 2024-01-16T15:16:37.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.629+0000 7f24ade00640 1 --2- 172.21.15.53:0/2553514997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b00cc440 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009870 tx=0x7f24a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf47c007a22862f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:37.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.630+0000 7f24ad400640 1 -- 172.21.15.53:0/2553514997 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0031640 con 0x7f24b015bf10 2024-01-16T15:16:37.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.630+0000 7f24ad400640 1 -- 172.21.15.53:0/2553514997 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a00317a0 con 0x7f24b015bf10 2024-01-16T15:16:37.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.630+0000 7f24ad400640 1 -- 172.21.15.53:0/2553514997 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a0031b30 con 0x7f24b015bf10 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.630+0000 7f24b4a00640 1 -- 172.21.15.53:0/2553514997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 msgr2=0x7f24b00cc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.630+0000 7f24b4a00640 1 --2- 172.21.15.53:0/2553514997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b00cc440 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f24a0009870 tx=0x7f24a002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.631+0000 7f24b4a00640 1 -- 172.21.15.53:0/2553514997 shutdown_connections 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.631+0000 7f24b4a00640 1 --2- 172.21.15.53:0/2553514997 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b00cc440 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.631+0000 7f24b4a00640 1 --2- 172.21.15.53:0/2553514997 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b0070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.631+0000 7f24b4a00640 1 -- 172.21.15.53:0/2553514997 >> 172.21.15.53:0/2553514997 conn(0x7f24b006a7d0 msgr2=0x7f24b006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:37.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.631+0000 7f24b4a00640 1 -- 172.21.15.53:0/2553514997 shutdown_connections 2024-01-16T15:16:37.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.631+0000 7f24b4a00640 1 -- 172.21.15.53:0/2553514997 wait complete. 2024-01-16T15:16:37.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.633+0000 7f24b4a00640 1 Processor -- start 2024-01-16T15:16:37.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.633+0000 7f24b4a00640 1 -- start start 2024-01-16T15:16:37.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b00af990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:37.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b00cc980 0x7f24b015a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24b4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b013f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24b4a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b013e750 con 0x7f24b015bf10 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24b4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b013dad0 con 0x7f24b00cc980 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24b4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b013ddb0 con 0x7f24b00adf20 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24af200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b013f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24af200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b013f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:53366/0 (socket says 172.21.15.53:53366) 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24af200640 1 -- 172.21.15.53:0/3230679505 learned_addr learned my addr 172.21.15.53:0/3230679505 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.634+0000 7f24ae800640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b00af990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24ade00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b00cc980 0x7f24b015a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24af200640 1 -- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 msgr2=0x7f24b00af990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24af200640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b00af990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24af200640 1 -- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b00cc980 msgr2=0x7f24b015a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24af200640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b00cc980 0x7f24b015a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24af200640 1 -- 172.21.15.53:0/3230679505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a00094d0 con 0x7f24b015bf10 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24ade00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b00cc980 0x7f24b015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24af200640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b013f1b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c6b0 tx=0x7f24a800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8003b70 con 0x7f24b015bf10 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a8003cf0 con 0x7f24b015bf10 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b013e090 con 0x7f24b015bf10 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.635+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a8005310 con 0x7f24b015bf10 2024-01-16T15:16:37.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.636+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b013d6f0 con 0x7f24b015bf10 2024-01-16T15:16:37.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.637+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b015aa00 con 0x7f24b015bf10 2024-01-16T15:16:37.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.638+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 37) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f24a8005470 con 0x7f24b015bf10 2024-01-16T15:16:37.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.639+0000 7f24a6a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f248806c310 0x7f248806e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:37.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.639+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f24a8088800 con 0x7f24b015bf10 2024-01-16T15:16:37.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.640+0000 7f24ae800640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f248806c310 0x7f248806e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:37.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.641+0000 7f24ae800640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f248806c310 0x7f248806e7d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f24b00b1c80 tx=0x7f2498008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:37.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.644+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24b015aa00 con 0x7f24b015bf10 2024-01-16T15:16:37.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:37.880+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66973", "target": ["mon-mgr", ""]}) v1 -- 0x7f24b015ee90 con 0x7f248806c310 2024-01-16T15:16:42.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.220+0000 7f24a6a00640 1 -- 172.21.15.53:0/3230679505 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f24980095e0 con 0x7f248806c310 2024-01-16T15:16:42.207 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:16:42.207 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:16:42.207 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:16:42.207 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:16:42.208 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:16:42.208 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:16:42.208 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:16:42.208 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:16:42.208 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:16:42.208 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:16:42.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f248806c310 msgr2=0x7f248806e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:42.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f248806c310 0x7f248806e7d0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f24b00b1c80 tx=0x7f2498008040 comp rx=0 tx=0).stop 2024-01-16T15:16:42.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 msgr2=0x7f24b013f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b013f1b0 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f24a800c6b0 tx=0x7f24a800cb70 comp rx=0 tx=0).stop 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 shutdown_connections 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f248806c310 0x7f248806e7d0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24b015bf10 0x7f24b013f1b0 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24b00cc980 0x7f24b015a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 --2- 172.21.15.53:0/3230679505 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24b00adf20 0x7f24b00af990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.225+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 >> 172.21.15.53:0/3230679505 conn(0x7f24b006a7d0 msgr2=0x7f24b006db60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.226+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 shutdown_connections 2024-01-16T15:16:42.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.226+0000 7f24b4a00640 1 -- 172.21.15.53:0/3230679505 wait complete. 2024-01-16T15:16:42.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone66971 2024-01-16T15:16:42.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.481+0000 7f96b7e00640 1 Processor -- start 2024-01-16T15:16:42.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.481+0000 7f96b7e00640 1 -- start start 2024-01-16T15:16:42.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.482+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8071e40 0x7f96b80ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:42.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.482+0000 7f96b7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 0x7f96b815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:42.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.482+0000 7f96b7e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96b815e140 con 0x7f96b8143810 2024-01-16T15:16:42.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.482+0000 7f96b7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96b8143bf0 con 0x7f96b815d770 2024-01-16T15:16:42.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.482+0000 7f96b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b80722c0 con 0x7f96b8071e40 2024-01-16T15:16:42.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.483+0000 7f96b7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 0x7f96b815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:42.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.483+0000 7f96b6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8071e40 0x7f96b80ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:42.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.483+0000 7f96b7400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 0x7f96b815dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:60688/0 (socket says 172.21.15.53:60688) 2024-01-16T15:16:42.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.483+0000 7f96b7400640 1 -- 172.21.15.53:0/1445950441 learned_addr learned my addr 172.21.15.53:0/1445950441 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:42.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.484+0000 7f96b6000640 1 -- 172.21.15.53:0/1445950441 >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 msgr2=0x7f96b815dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:42.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.484+0000 7f96b6000640 1 --2- 172.21.15.53:0/1445950441 >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 0x7f96b815dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.484+0000 7f96b6000640 1 -- 172.21.15.53:0/1445950441 >> v1:172.21.15.146:6789/0 conn(0x7f96b8143810 legacy=0x7f96b815e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:42.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.484+0000 7f96b6000640 1 -- 172.21.15.53:0/1445950441 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b806c730 con 0x7f96b8071e40 2024-01-16T15:16:42.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.484+0000 7f96b7400640 1 --2- 172.21.15.53:0/1445950441 >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 0x7f96b815dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:16:42.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.486+0000 7f96b6000640 1 --2- 172.21.15.53:0/1445950441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8071e40 0x7f96b80ae170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f96a00097a0 tx=0x7f96a002b7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e897fb0b017e3bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:42.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.486+0000 7f96b5600640 1 -- 172.21.15.53:0/1445950441 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a002f050 con 0x7f96b8071e40 2024-01-16T15:16:42.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.486+0000 7f96b5600640 1 -- 172.21.15.53:0/1445950441 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a0031700 con 0x7f96b8071e40 2024-01-16T15:16:42.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.486+0000 7f96b5600640 1 -- 172.21.15.53:0/1445950441 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0031a90 con 0x7f96b8071e40 2024-01-16T15:16:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.487+0000 7f96b7e00640 1 -- 172.21.15.53:0/1445950441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8071e40 msgr2=0x7f96b80ae170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.487+0000 7f96b7e00640 1 --2- 172.21.15.53:0/1445950441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8071e40 0x7f96b80ae170 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f96a00097a0 tx=0x7f96a002b7a0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.487+0000 7f96b7e00640 1 -- 172.21.15.53:0/1445950441 shutdown_connections 2024-01-16T15:16:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.487+0000 7f96b7e00640 1 --2- 172.21.15.53:0/1445950441 >> v2:172.21.15.146:3300/0 conn(0x7f96b815d770 0x7f96b815dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.487+0000 7f96b7e00640 1 --2- 172.21.15.53:0/1445950441 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8071e40 0x7f96b80ae170 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.487+0000 7f96b7e00640 1 -- 172.21.15.53:0/1445950441 >> 172.21.15.53:0/1445950441 conn(0x7f96b806a7d0 msgr2=0x7f96b806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:42.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.488+0000 7f96b7e00640 1 -- 172.21.15.53:0/1445950441 shutdown_connections 2024-01-16T15:16:42.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.488+0000 7f96b7e00640 1 -- 172.21.15.53:0/1445950441 wait complete. 2024-01-16T15:16:42.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.489+0000 7f96b7e00640 1 Processor -- start 2024-01-16T15:16:42.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.489+0000 7f96b7e00640 1 -- start start 2024-01-16T15:16:42.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.490+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 0x7f96b806d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:42.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.490+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8143810 0x7f96b80b0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:42.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.490+0000 7f96b7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b815d770 0x7f96b80ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:42.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.490+0000 7f96b7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8144a90 con 0x7f96b8071e40 2024-01-16T15:16:42.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.490+0000 7f96b7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8144d70 con 0x7f96b815d770 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.490+0000 7f96b7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b8145050 con 0x7f96b8143810 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 0x7f96b806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 0x7f96b806d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43588/0 (socket says 172.21.15.53:43588) 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 -- 172.21.15.53:0/2874717691 learned_addr learned my addr 172.21.15.53:0/2874717691 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b7400640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b815d770 0x7f96b80ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 -- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8143810 msgr2=0x7f96b80b0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8143810 0x7f96b80b0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 -- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b815d770 msgr2=0x7f96b80ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b815d770 0x7f96b80ae9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.491+0000 7f96b6a00640 1 -- 172.21.15.53:0/2874717691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b0009b30 con 0x7f96b8071e40 2024-01-16T15:16:42.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.492+0000 7f96b6a00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 0x7f96b806d590 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f96a800c450 tx=0x7f96a800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:42.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.492+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a80052d0 con 0x7f96b8071e40 2024-01-16T15:16:42.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.492+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96a8005430 con 0x7f96b8071e40 2024-01-16T15:16:42.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.492+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a8012870 con 0x7f96b8071e40 2024-01-16T15:16:42.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.492+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a0009400 con 0x7f96b8071e40 2024-01-16T15:16:42.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.492+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96b81453e0 con 0x7f96b8071e40 2024-01-16T15:16:42.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.493+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96b8145bf0 con 0x7f96b8071e40 2024-01-16T15:16:42.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.494+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 37) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f96a80129d0 con 0x7f96b8071e40 2024-01-16T15:16:42.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.495+0000 7f96af400640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f969006c310 0x7f969006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:42.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.495+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f96a80885a0 con 0x7f96b8071e40 2024-01-16T15:16:42.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.496+0000 7f96b6000640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f969006c310 0x7f969006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:42.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.497+0000 7f96b6000640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f969006c310 0x7f969006e7d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f96a0035000 tx=0x7f96a0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:42.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.500+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96a8054b40 con 0x7f96b8071e40 2024-01-16T15:16:42.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:42.743+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone66971", "target": ["mon-mgr", ""]}) v1 -- 0x7f96b806b150 con 0x7f969006c310 2024-01-16T15:16:47.208 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:47.221+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:47.208 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:47.221+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:47.208 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:47.221+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:47.208 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:47.221+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:47.208 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:16:47.221+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:16:47.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.457+0000 7f96af400640 1 -- 172.21.15.53:0/2874717691 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f96b806b150 con 0x7f969006c310 2024-01-16T15:16:47.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f969006c310 msgr2=0x7f969006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:47.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f969006c310 0x7f969006e7d0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f96a0035000 tx=0x7f96a0030000 comp rx=0 tx=0).stop 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 msgr2=0x7f96b806d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 0x7f96b806d590 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f96a800c450 tx=0x7f96a800c910 comp rx=0 tx=0).stop 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 shutdown_connections 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f969006c310 0x7f969006e7d0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96b815d770 0x7f96b80ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96b8143810 0x7f96b80b0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 --2- 172.21.15.53:0/2874717691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96b8071e40 0x7f96b806d590 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.460+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 >> 172.21.15.53:0/2874717691 conn(0x7f96b806a7d0 msgr2=0x7f96b80ae170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:47.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.461+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 shutdown_connections 2024-01-16T15:16:47.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:47.461+0000 7f96b7e00640 1 -- 172.21.15.53:0/2874717691 wait complete. 2024-01-16T15:16:49.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone66972 2024-01-16T15:16:49.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.713+0000 7f5ad9400640 1 Processor -- start 2024-01-16T15:16:49.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.713+0000 7f5ad9400640 1 -- start start 2024-01-16T15:16:49.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad9400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5ad413e360 0x7f5ad40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad9400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ad415f9c0 0x7f5ad40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad9400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad415c5f0 con 0x7f5ad413df80 2024-01-16T15:16:49.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad9400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad406c470 con 0x7f5ad415f9c0 2024-01-16T15:16:49.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad9400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad406c750 con 0x7f5ad413e360 2024-01-16T15:16:49.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad2800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5ad413df80 0x7f5ad415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55112/0 (socket says 172.21.15.53:55112) 2024-01-16T15:16:49.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad2800640 1 -- 172.21.15.53:0/79964872 learned_addr learned my addr 172.21.15.53:0/79964872 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:16:49.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.715+0000 7f5ad3200640 1 --2- 172.21.15.53:0/79964872 >> v2:172.21.15.53:3300/0 conn(0x7f5ad415f9c0 0x7f5ad40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad1e00640 1 --2- 172.21.15.53:0/79964872 >> v2:172.21.15.146:3300/0 conn(0x7f5ad413e360 0x7f5ad40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad1400640 1 -- 172.21.15.53:0/79964872 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592111593 0 0) 0x7f5ad415c5f0 con 0x7f5ad413df80 2024-01-16T15:16:49.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad1400640 1 -- 172.21.15.53:0/79964872 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4003520 con 0x7f5ad413df80 2024-01-16T15:16:49.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad3200640 1 -- 172.21.15.53:0/79964872 >> v2:172.21.15.146:3300/0 conn(0x7f5ad413e360 msgr2=0x7f5ad40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:16:49.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad3200640 1 --2- 172.21.15.53:0/79964872 >> v2:172.21.15.146:3300/0 conn(0x7f5ad413e360 0x7f5ad40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad3200640 1 -- 172.21.15.53:0/79964872 >> v1:172.21.15.53:6789/0 conn(0x7f5ad413df80 legacy=0x7f5ad415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.716+0000 7f5ad3200640 1 -- 172.21.15.53:0/79964872 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad406ca30 con 0x7f5ad415f9c0 2024-01-16T15:16:49.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.717+0000 7f5ad3200640 1 --2- 172.21.15.53:0/79964872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad40cf7d0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f5ad406ded0 tx=0x7f5acc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70421647c4830435 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:49.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.717+0000 7f5ad1400640 1 -- 172.21.15.53:0/79964872 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc02cc80 con 0x7f5ad415f9c0 2024-01-16T15:16:49.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.717+0000 7f5ad1400640 1 -- 172.21.15.53:0/79964872 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc02cde0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.717+0000 7f5ad1400640 1 -- 172.21.15.53:0/79964872 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc034660 con 0x7f5ad415f9c0 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 -- 172.21.15.53:0/79964872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 msgr2=0x7f5ad40cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 --2- 172.21.15.53:0/79964872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad40cf7d0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f5ad406ded0 tx=0x7f5acc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 -- 172.21.15.53:0/79964872 shutdown_connections 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 --2- 172.21.15.53:0/79964872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad40cf7d0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 --2- 172.21.15.53:0/79964872 >> v2:172.21.15.146:3300/0 conn(0x7f5ad413e360 0x7f5ad40cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 -- 172.21.15.53:0/79964872 >> 172.21.15.53:0/79964872 conn(0x7f5ad406a800 msgr2=0x7f5ad406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 -- 172.21.15.53:0/79964872 shutdown_connections 2024-01-16T15:16:49.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.718+0000 7f5ad9400640 1 -- 172.21.15.53:0/79964872 wait complete. 2024-01-16T15:16:49.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.720+0000 7f5ad9400640 1 Processor -- start 2024-01-16T15:16:49.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.720+0000 7f5ad9400640 1 -- start start 2024-01-16T15:16:49.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad9400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad413df80 0x7f5ad415df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad9400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad413e360 0x7f5ad415a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad9400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad4157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad9400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4159e60 con 0x7f5ad415f9c0 2024-01-16T15:16:49.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad9400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4159550 con 0x7f5ad413df80 2024-01-16T15:16:49.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad9400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad4156b40 con 0x7f5ad413e360 2024-01-16T15:16:49.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad4157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad4157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43596/0 (socket says 172.21.15.53:43596) 2024-01-16T15:16:49.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad3200640 1 -- 172.21.15.53:0/3474167585 learned_addr learned my addr 172.21.15.53:0/3474167585 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:16:49.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad3200640 1 -- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad413e360 msgr2=0x7f5ad415a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.721+0000 7f5ad3200640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad413e360 0x7f5ad415a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5ad3200640 1 -- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad413df80 msgr2=0x7f5ad415df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:16:49.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5ad3200640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad413df80 0x7f5ad415df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:16:49.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5ad3200640 1 -- 172.21.15.53:0/3474167585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc009a70 con 0x7f5ad415f9c0 2024-01-16T15:16:49.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5ad3200640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad4157580 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009e80 tx=0x7f5acc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:49.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc040650 con 0x7f5ad415f9c0 2024-01-16T15:16:49.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5acc0407b0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5acc040b60 con 0x7f5ad415f9c0 2024-01-16T15:16:49.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad41597a0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.722+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad415e4b0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.724+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad4159fa0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.725+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 38) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f5acc040cc0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.726+0000 7f5acaa00640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aac06c3a0 0x7f5aac06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:16:49.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.726+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5acc0ab3f0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.726+0000 7f5ad2800640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aac06c3a0 0x7f5aac06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:16:49.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.728+0000 7f5ad2800640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aac06c3a0 0x7f5aac06e860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5ad415c780 tx=0x7f5ac4007930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:16:49.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.731+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ad4159fa0 con 0x7f5ad415f9c0 2024-01-16T15:16:49.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:16:49.962+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone66972", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ad406cff0 con 0x7f5aac06c3a0 2024-01-16T15:16:54.940 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:16:54.949 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:25.090 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:25.099 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:55.557 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:17:55.566 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:11.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:11.743+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 39) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f5acc0780e0 con 0x7f5ad415f9c0 2024-01-16T15:18:25.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:25.746 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:47.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.745+0000 7f5acaa00640 1 -- 172.21.15.53:0/3474167585 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5ac40058b0 con 0x7f5aac06c3a0 2024-01-16T15:18:47.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aac06c3a0 msgr2=0x7f5aac06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aac06c3a0 0x7f5aac06e860 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5ad415c780 tx=0x7f5ac4007930 comp rx=0 tx=0).stop 2024-01-16T15:18:47.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 msgr2=0x7f5ad4157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:47.739 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad4157580 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5acc009e80 tx=0x7f5acc004a40 comp rx=0 tx=0).stop 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 shutdown_connections 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aac06c3a0 0x7f5aac06e860 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad415f9c0 0x7f5ad4157580 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad413e360 0x7f5ad415a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 --2- 172.21.15.53:0/3474167585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad413df80 0x7f5ad415df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.748+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 >> 172.21.15.53:0/3474167585 conn(0x7f5ad406a800 msgr2=0x7f5ad40b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.749+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 shutdown_connections 2024-01-16T15:18:47.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:47.749+0000 7f5ad9400640 1 -- 172.21.15.53:0/3474167585 wait complete. 2024-01-16T15:18:49.760 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone66973 2024-01-16T15:18:49.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.006+0000 7f1a01c00640 1 Processor -- start 2024-01-16T15:18:49.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.006+0000 7f1a01c00640 1 -- start start 2024-01-16T15:18:49.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.008+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc14b2c0 0x7f19fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.008+0000 7f1a01c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f19fc071070 0x7f19fc0cc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:49.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.008+0000 7f1a01c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19fc0cfbf0 con 0x7f19fc0cf7d0 2024-01-16T15:18:49.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.008+0000 7f1a01c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19fc14b6a0 con 0x7f19fc071070 2024-01-16T15:18:49.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.008+0000 7f1a01c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc071690 con 0x7f19fc14b2c0 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f19fc071070 0x7f19fc0cc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f19fc071070 0x7f19fc0cc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55934/0 (socket says 172.21.15.53:55934) 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 -- 172.21.15.53:0/1186805900 learned_addr learned my addr 172.21.15.53:0/1186805900 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19faa00640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc14b2c0 0x7f19fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 -- 172.21.15.53:0/1186805900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc14b2c0 msgr2=0x7f19fc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc14b2c0 0x7f19fc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:49.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 -- 172.21.15.53:0/1186805900 >> v1:172.21.15.146:6789/0 conn(0x7f19fc0cf7d0 legacy=0x7f19fc070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:50.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19fb400640 1 -- 172.21.15.53:0/1186805900 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc071970 con 0x7f19fc071070 2024-01-16T15:18:50.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.009+0000 7f19faa00640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc14b2c0 0x7f19fc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:50.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.010+0000 7f19fb400640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc071070 0x7f19fc0cc160 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009870 tx=0x7f19f402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef464cdda3ab5819 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:50.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.011+0000 7f19f9600640 1 -- 172.21.15.53:0/1186805900 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4031640 con 0x7f19fc071070 2024-01-16T15:18:50.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.011+0000 7f19f9600640 1 -- 172.21.15.53:0/1186805900 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19f40317a0 con 0x7f19fc071070 2024-01-16T15:18:50.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.011+0000 7f19f9600640 1 -- 172.21.15.53:0/1186805900 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19f4031b30 con 0x7f19fc071070 2024-01-16T15:18:50.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.011+0000 7f1a01c00640 1 -- 172.21.15.53:0/1186805900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc071070 msgr2=0x7f19fc0cc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.011+0000 7f1a01c00640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc071070 0x7f19fc0cc160 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f19f4009870 tx=0x7f19f402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.012+0000 7f1a01c00640 1 -- 172.21.15.53:0/1186805900 shutdown_connections 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.012+0000 7f1a01c00640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc071070 0x7f19fc0cc160 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.012+0000 7f1a01c00640 1 --2- 172.21.15.53:0/1186805900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc14b2c0 0x7f19fc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.012+0000 7f1a01c00640 1 -- 172.21.15.53:0/1186805900 >> 172.21.15.53:0/1186805900 conn(0x7f19fc06a800 msgr2=0x7f19fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.012+0000 7f1a01c00640 1 -- 172.21.15.53:0/1186805900 shutdown_connections 2024-01-16T15:18:50.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.012+0000 7f1a01c00640 1 -- 172.21.15.53:0/1186805900 wait complete. 2024-01-16T15:18:50.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.013+0000 7f1a01c00640 1 Processor -- start 2024-01-16T15:18:50.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.014+0000 7f1a01c00640 1 -- start start 2024-01-16T15:18:50.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.014+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc071070 0x7f19fc0b0090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.014+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 0x7f19fc164480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.014+0000 7f1a01c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19fc14b2c0 0x7f19fc1649c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f1a01c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1f7390 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f1a01c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1f75a0 con 0x7f19fc14b2c0 2024-01-16T15:18:50.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f1a01c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc1f7880 con 0x7f19fc071070 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f19fa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 0x7f19fc164480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f19fa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 0x7f19fc164480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55940/0 (socket says 172.21.15.53:55940) 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f19fa000640 1 -- 172.21.15.53:0/2546164900 learned_addr learned my addr 172.21.15.53:0/2546164900 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f19faa00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc071070 0x7f19fc0b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.015+0000 7f19fb400640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19fc14b2c0 0x7f19fc1649c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19fa000640 1 -- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc071070 msgr2=0x7f19fc0b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19fa000640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc071070 0x7f19fc0b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19fa000640 1 -- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19fc14b2c0 msgr2=0x7f19fc1649c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19fa000640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19fc14b2c0 0x7f19fc1649c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19fa000640 1 -- 172.21.15.53:0/2546164900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f40094d0 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19fa000640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 0x7f19fc164480 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f19e400c7c0 tx=0x7f19e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e400aa50 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e400abb0 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc1f7b60 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e4012420 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.016+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc1f8120 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.018+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc0675d0 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.019+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 39) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f19e4012580 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.019+0000 7f19f2a00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19d406c310 0x7f19d406e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:50.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.020+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f19e4088970 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.020+0000 7f19faa00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19d406c310 0x7f19d406e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:50.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.021+0000 7f19faa00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19d406c310 0x7f19d406e7d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00a950 tx=0x7f19ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:50.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.025+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f19e4054f70 con 0x7f19fc0cf7d0 2024-01-16T15:18:50.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:50.262+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone66973", "target": ["mon-mgr", ""]}) v1 -- 0x7f19fc0cbc40 con 0x7f19d406c310 2024-01-16T15:18:52.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.680+0000 7f19f2a00640 1 -- 172.21.15.53:0/2546164900 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f19ec00c050 con 0x7f19d406c310 2024-01-16T15:18:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19d406c310 msgr2=0x7f19d406e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19d406c310 0x7f19d406e7d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00a950 tx=0x7f19ec008040 comp rx=0 tx=0).stop 2024-01-16T15:18:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 msgr2=0x7f19fc164480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:52.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 0x7f19fc164480 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f19e400c7c0 tx=0x7f19e400cc80 comp rx=0 tx=0).stop 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 shutdown_connections 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f19d406c310 0x7f19d406e7d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f19fc14b2c0 0x7f19fc1649c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19fc0cf7d0 0x7f19fc164480 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 --2- 172.21.15.53:0/2546164900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19fc071070 0x7f19fc0b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 >> 172.21.15.53:0/2546164900 conn(0x7f19fc06a800 msgr2=0x7f19fc06c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.683+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 shutdown_connections 2024-01-16T15:18:52.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:52.684+0000 7f1a01c00640 1 -- 172.21.15.53:0/2546164900 wait complete. 2024-01-16T15:18:54.698 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66971 2024-01-16T15:18:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.947+0000 7f5850a00640 1 Processor -- start 2024-01-16T15:18:54.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.948+0000 7f5850a00640 1 -- start start 2024-01-16T15:18:54.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.949+0000 7f5850a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f58480ae5a0 0x7f584815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.949+0000 7f5850a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5848070c40 con 0x7f58480cf290 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f5850a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f584806c470 con 0x7f58480b3ae0 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f5850a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5848071450 con 0x7f58480ae5a0 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f58480ae5a0 0x7f584815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584ea00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f58480cf290 0x7f58480b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33280/0 (socket says 172.21.15.53:33280) 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584ea00640 1 -- 172.21.15.53:0/32360495 learned_addr learned my addr 172.21.15.53:0/32360495 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584e000640 1 -- 172.21.15.53:0/32360495 >> v1:172.21.15.146:6789/0 conn(0x7f58480b3ae0 legacy=0x7f584806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:54.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584e000640 1 -- 172.21.15.53:0/32360495 >> v1:172.21.15.53:6789/0 conn(0x7f58480cf290 legacy=0x7f58480b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584e000640 1 -- 172.21.15.53:0/32360495 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584806c750 con 0x7f58480ae5a0 2024-01-16T15:18:54.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.950+0000 7f584cc00640 1 -- 172.21.15.53:0/32360495 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3223432337 0 0) 0x7f5848070c40 con 0x7f58480cf290 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.952+0000 7f584e000640 1 --2- 172.21.15.53:0/32360495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480ae5a0 0x7f584815e430 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f58380097a0 tx=0x7f583802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b945c1f9406f89db server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.952+0000 7f584cc00640 1 -- 172.21.15.53:0/32360495 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583802f810 con 0x7f58480ae5a0 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.952+0000 7f584cc00640 1 -- 172.21.15.53:0/32360495 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f583802f970 con 0x7f58480ae5a0 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.952+0000 7f584cc00640 1 -- 172.21.15.53:0/32360495 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583802fd00 con 0x7f58480ae5a0 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.952+0000 7f5850a00640 1 -- 172.21.15.53:0/32360495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480ae5a0 msgr2=0x7f584815e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.952+0000 7f5850a00640 1 --2- 172.21.15.53:0/32360495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480ae5a0 0x7f584815e430 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f58380097a0 tx=0x7f583802b9f0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.953+0000 7f5850a00640 1 -- 172.21.15.53:0/32360495 shutdown_connections 2024-01-16T15:18:54.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.953+0000 7f5850a00640 1 --2- 172.21.15.53:0/32360495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480ae5a0 0x7f584815e430 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.953+0000 7f5850a00640 1 -- 172.21.15.53:0/32360495 >> 172.21.15.53:0/32360495 conn(0x7f584806a800 msgr2=0x7f584806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:54.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.953+0000 7f5850a00640 1 -- 172.21.15.53:0/32360495 shutdown_connections 2024-01-16T15:18:54.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.953+0000 7f5850a00640 1 -- 172.21.15.53:0/32360495 wait complete. 2024-01-16T15:18:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.954+0000 7f5850a00640 1 Processor -- start 2024-01-16T15:18:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.955+0000 7f5850a00640 1 -- start start 2024-01-16T15:18:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f5850a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58480ae5a0 0x7f58481f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f5850a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 0x7f584815b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f5850a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58480cf290 0x7f584815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f5850a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58481f2b10 con 0x7f58480b3ae0 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f5850a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f584815c110 con 0x7f58480cf290 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f5850a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f584815c3f0 con 0x7f58480ae5a0 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 0x7f584815b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 0x7f584815b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41372/0 (socket says 172.21.15.53:41372) 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 -- 172.21.15.53:0/1280901479 learned_addr learned my addr 172.21.15.53:0/1280901479 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 -- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58480ae5a0 msgr2=0x7f58481f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584ea00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58480cf290 0x7f584815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58480ae5a0 0x7f58481f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 -- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58480cf290 msgr2=0x7f584815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.956+0000 7f584d600640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58480cf290 0x7f584815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f584d600640 1 -- 172.21.15.53:0/1280901479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838009400 con 0x7f58480b3ae0 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f584ea00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58480cf290 0x7f584815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f584d600640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 0x7f584815b5a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f583000c870 tx=0x7f583000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5830003b70 con 0x7f58480b3ae0 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5830003cf0 con 0x7f58480b3ae0 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584815c6d0 con 0x7f58480b3ae0 2024-01-16T15:18:54.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f583000aa90 con 0x7f58480b3ae0 2024-01-16T15:18:54.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.957+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584815cc90 con 0x7f58480b3ae0 2024-01-16T15:18:54.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.959+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58480675d0 con 0x7f58480b3ae0 2024-01-16T15:18:54.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.959+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 39) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f583000abf0 con 0x7f58480b3ae0 2024-01-16T15:18:54.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.960+0000 7f5846000640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f582006c310 0x7f582006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:54.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.960+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f58300888d0 con 0x7f58480b3ae0 2024-01-16T15:18:54.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.961+0000 7f584e000640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f582006c310 0x7f582006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:54.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.962+0000 7f584e000640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f582006c310 0x7f582006e7d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f5838036000 tx=0x7f5838039040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:54.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:54.965+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f58300bee90 con 0x7f58480b3ae0 2024-01-16T15:18:55.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.201+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66971", "target": ["mon-mgr", ""]}) v1 -- 0x7f5848070550 con 0x7f582006c310 2024-01-16T15:18:55.203 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:18:55.213+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:55.203 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:18:55.213+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:55.204 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:18:55.213+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:55.204 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:18:55.213+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:55.204 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:18:55.213+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:18:55.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.249+0000 7f5846000640 1 -- 172.21.15.53:0/1280901479 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f583802fb60 con 0x7f582006c310 2024-01-16T15:18:55.240 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:18:55.241 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:55.242 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:18:55.242 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:18:55.242 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:18:55.242 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:55.242 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:55.242 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f582006c310 msgr2=0x7f582006e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f582006c310 0x7f582006e7d0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f5838036000 tx=0x7f5838039040 comp rx=0 tx=0).stop 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 msgr2=0x7f584815b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 0x7f584815b5a0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f583000c870 tx=0x7f583000cd30 comp rx=0 tx=0).stop 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 shutdown_connections 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f582006c310 0x7f582006e7d0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f58480cf290 0x7f584815bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f58480b3ae0 0x7f584815b5a0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 --2- 172.21.15.53:0/1280901479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f58480ae5a0 0x7f58481f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 >> 172.21.15.53:0/1280901479 conn(0x7f584806a800 msgr2=0x7f584806cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 shutdown_connections 2024-01-16T15:18:55.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.252+0000 7f5850a00640 1 -- 172.21.15.53:0/1280901479 wait complete. 2024-01-16T15:18:55.264 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66972 2024-01-16T15:18:55.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.505+0000 7f7020e00640 1 Processor -- start 2024-01-16T15:18:55.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.505+0000 7f7020e00640 1 -- start start 2024-01-16T15:18:55.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.506+0000 7f7020e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f701c071e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f7020e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c06c730 con 0x7f701c0ae170 2024-01-16T15:18:55.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f7020e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f701c0ae5f0 con 0x7f701c15d770 2024-01-16T15:18:55.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f7020e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c143bf0 con 0x7f701c143810 2024-01-16T15:18:55.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f7013e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f701c0ae170 0x7f701c15e8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33296/0 (socket says 172.21.15.53:33296) 2024-01-16T15:18:55.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f7013e00640 1 -- 172.21.15.53:0/1993153760 learned_addr learned my addr 172.21.15.53:0/1993153760 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:18:55.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f701b400640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f701c071e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3243998477 0 0) 0x7f701c06c730 con 0x7f701c0ae170 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.507+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc003520 con 0x7f701c0ae170 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3935135618 0 0) 0x7f6ffc003520 con 0x7f701c0ae170 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 msgr2=0x7f701c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f701c071e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 >> v1:172.21.15.146:6789/0 conn(0x7f701c15d770 legacy=0x7f701c15dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c0724d0 con 0x7f701c0ae170 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7004001670 con 0x7f701c0ae170 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701b400640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f701c071e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 >> v1:172.21.15.53:6789/0 conn(0x7f701c0ae170 legacy=0x7f701c15e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ffc0039a0 0x7f6ffc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0047c0 0x7f6ffc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f6ffc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c143bf0 con 0x7f6ffc0047c0 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc005580 con 0x7f6ffc0039a0 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ffc00bec0 con 0x7f701c143810 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.508+0000 7f7013e00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0047c0 0x7f6ffc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f701b400640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ffc0039a0 0x7f6ffc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f701be00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f6ffc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f7013e00640 1 -- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 msgr2=0x7f6ffc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:55.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f7013e00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f701c143810 0x7f6ffc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f7013e00640 1 -- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ffc0039a0 msgr2=0x7f6ffc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f7013e00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ffc0039a0 0x7f6ffc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.509+0000 7f7013e00640 1 -- 172.21.15.53:0/1993153760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc003520 con 0x7f6ffc0047c0 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.510+0000 7f7013e00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0047c0 0x7f6ffc004c40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f700400c0e0 tx=0x7f6ffc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc74a694cb0e4c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.510+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004005ad0 con 0x7f6ffc0047c0 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.510+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7004005c50 con 0x7f6ffc0047c0 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.510+0000 7f701aa00640 1 -- 172.21.15.53:0/1993153760 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700402c7d0 con 0x7f6ffc0047c0 2024-01-16T15:18:55.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f7020e00640 1 -- 172.21.15.53:0/1993153760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0047c0 msgr2=0x7f6ffc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f7020e00640 1 --2- 172.21.15.53:0/1993153760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ffc0047c0 0x7f6ffc004c40 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7f700400c0e0 tx=0x7f6ffc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f701be00640 1 -- 172.21.15.53:0/1993153760 reap_dead start 2024-01-16T15:18:55.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f7020e00640 1 -- 172.21.15.53:0/1993153760 shutdown_connections 2024-01-16T15:18:55.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f7020e00640 1 -- 172.21.15.53:0/1993153760 >> 172.21.15.53:0/1993153760 conn(0x7f701c06a7d0 msgr2=0x7f701c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f7020e00640 1 -- 172.21.15.53:0/1993153760 shutdown_connections 2024-01-16T15:18:55.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.511+0000 7f7020e00640 1 -- 172.21.15.53:0/1993153760 wait complete. 2024-01-16T15:18:55.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.513+0000 7f7020e00640 1 Processor -- start 2024-01-16T15:18:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.513+0000 7f7020e00640 1 -- start start 2024-01-16T15:18:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.514+0000 7f7020e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ffc0047c0 0x7f701c0cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.514+0000 7f7020e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 0x7f701c13c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.514+0000 7f7020e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701c1f6cc0 0x7f701c13c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.514+0000 7f7020e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c13cbf0 con 0x7f701c1f6860 2024-01-16T15:18:55.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.514+0000 7f7020e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701c13dda0 con 0x7f701c1f6cc0 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.514+0000 7f7020e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701c0cc210 con 0x7f6ffc0047c0 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 0x7f701c13c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 0x7f701c13c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41398/0 (socket says 172.21.15.53:41398) 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 -- 172.21.15.53:0/1593918122 learned_addr learned my addr 172.21.15.53:0/1593918122 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f701b400640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ffc0047c0 0x7f701c0cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f701be00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701c1f6cc0 0x7f701c13c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 -- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ffc0047c0 msgr2=0x7f701c0cbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ffc0047c0 0x7f701c0cbcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 -- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701c1f6cc0 msgr2=0x7f701c13c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701c1f6cc0 0x7f701c13c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f7013e00640 1 -- 172.21.15.53:0/1593918122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f701c13dff0 con 0x7f701c1f6860 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.515+0000 7f701be00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701c1f6cc0 0x7f701c13c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.516+0000 7f7013e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 0x7f701c13c170 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f700402c440 tx=0x7f701c15e7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.516+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700403c040 con 0x7f701c1f6860 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.516+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700402cbe0 con 0x7f701c1f6860 2024-01-16T15:18:55.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.516+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7004007070 con 0x7f701c1f6860 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.516+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f701c13e2d0 con 0x7f701c1f6860 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.516+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f701c13e890 con 0x7f701c1f6860 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.518+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f701c067450 con 0x7f701c1f6860 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.518+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 39) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f7004032070 con 0x7f701c1f6860 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.519+0000 7f7013400640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ff406c360 0x7f6ff406e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.519+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f70040a9ec0 con 0x7f701c1f6860 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.520+0000 7f701b400640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ff406c360 0x7f6ff406e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:55.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.521+0000 7f701b400640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ff406c360 0x7f6ff406e820 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:55.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.524+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7004076460 con 0x7f701c1f6860 2024-01-16T15:18:55.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.757+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66972", "target": ["mon-mgr", ""]}) v1 -- 0x7f701c06cfb0 con 0x7f6ff406c360 2024-01-16T15:18:55.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.781+0000 7f7013400640 1 -- 172.21.15.53:0/1593918122 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f700c00e530 con 0x7f6ff406c360 2024-01-16T15:18:55.771 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:18:55.771 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:55.772 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:55.773 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ff406c360 msgr2=0x7f6ff406e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ff406c360 0x7f6ff406e820 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f700c006fd0 tx=0x7f700c008040 comp rx=0 tx=0).stop 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 msgr2=0x7f701c13c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 0x7f701c13c170 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f700402c440 tx=0x7f701c15e7f0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 shutdown_connections 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6ff406c360 0x7f6ff406e820 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f701c1f6cc0 0x7f701c13c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f701c1f6860 0x7f701c13c170 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 --2- 172.21.15.53:0/1593918122 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ffc0047c0 0x7f701c0cbcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:55.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.784+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 >> 172.21.15.53:0/1593918122 conn(0x7f701c06a7d0 msgr2=0x7f701c0b1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:55.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.785+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 shutdown_connections 2024-01-16T15:18:55.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:55.785+0000 7f7020e00640 1 -- 172.21.15.53:0/1593918122 wait complete. 2024-01-16T15:18:55.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone66973 2024-01-16T15:18:56.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.033+0000 7f9e7a800640 1 Processor -- start 2024-01-16T15:18:56.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.033+0000 7f9e7a800640 1 -- start start 2024-01-16T15:18:56.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e7a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9e740cda00 0x7f9e740d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e7a800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740b2920 con 0x7f9e741610e0 2024-01-16T15:18:56.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e7a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740cb670 con 0x7f9e740cf4c0 2024-01-16T15:18:56.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e7a800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e740cb950 con 0x7f9e740cda00 2024-01-16T15:18:56.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e73400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9e741610e0 0x7f9e741614c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33304/0 (socket says 172.21.15.53:33304) 2024-01-16T15:18:56.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e73400640 1 -- 172.21.15.53:0/2316697618 learned_addr learned my addr 172.21.15.53:0/2316697618 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e73e00640 1 --2- 172.21.15.53:0/2316697618 >> v2:172.21.15.146:3300/0 conn(0x7f9e740cda00 0x7f9e740d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283449338 0 0) 0x7f9e740b2920 con 0x7f9e741610e0 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.035+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54003520 con 0x7f9e741610e0 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3199404348 0 0) 0x7f9e54003520 con 0x7f9e741610e0 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 >> v2:172.21.15.146:3300/0 conn(0x7f9e740cda00 msgr2=0x7f9e740d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 --2- 172.21.15.53:0/2316697618 >> v2:172.21.15.146:3300/0 conn(0x7f9e740cda00 0x7f9e740d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 >> v1:172.21.15.146:6789/0 conn(0x7f9e740cf4c0 legacy=0x7f9e7415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e740cbc30 con 0x7f9e741610e0 2024-01-16T15:18:56.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9e64001670 con 0x7f9e741610e0 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 >> v1:172.21.15.53:6789/0 conn(0x7f9e741610e0 legacy=0x7f9e741614c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540039c0 0x7f9e54003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e540047c0 0x7f9e54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e54005180 0x7f9e5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740cb950 con 0x7f9e540047c0 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400c080 con 0x7f9e540039c0 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400c240 con 0x7f9e54005180 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.036+0000 7f9e72a00640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e540047c0 0x7f9e54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e73400640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540039c0 0x7f9e54003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e72a00640 1 -- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e54005180 msgr2=0x7f9e5400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e73e00640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e54005180 0x7f9e5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e72a00640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e54005180 0x7f9e5400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e72a00640 1 -- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540039c0 msgr2=0x7f9e54003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e72a00640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540039c0 0x7f9e54003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e72a00640 1 -- 172.21.15.53:0/2316697618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e54003520 con 0x7f9e540047c0 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.037+0000 7f9e73e00640 1 -- 172.21.15.53:0/2316697618 reap_dead start 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.038+0000 7f9e72a00640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e540047c0 0x7f9e54004c40 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c001750 tx=0x7f9e540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29408b67f3ff31ad server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.038+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c0052d0 con 0x7f9e540047c0 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.038+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5c02e050 con 0x7f9e540047c0 2024-01-16T15:18:56.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.038+0000 7f9e72000640 1 -- 172.21.15.53:0/2316697618 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5c003830 con 0x7f9e540047c0 2024-01-16T15:18:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 -- 172.21.15.53:0/2316697618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e540047c0 msgr2=0x7f9e54004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e540047c0 0x7f9e54004c40 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7f9e5c001750 tx=0x7f9e540057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 -- 172.21.15.53:0/2316697618 shutdown_connections 2024-01-16T15:18:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 --2- 172.21.15.53:0/2316697618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e540047c0 0x7f9e54004c40 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 -- 172.21.15.53:0/2316697618 >> 172.21.15.53:0/2316697618 conn(0x7f9e740d1020 msgr2=0x7f9e740d1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:56.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 -- 172.21.15.53:0/2316697618 shutdown_connections 2024-01-16T15:18:56.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.039+0000 7f9e7a800640 1 -- 172.21.15.53:0/2316697618 wait complete. 2024-01-16T15:18:56.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.041+0000 7f9e7a800640 1 Processor -- start 2024-01-16T15:18:56.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.041+0000 7f9e7a800640 1 -- start start 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e7a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540047c0 0x7f9e741e6920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e7a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e740c4fd0 0x7f9e740c5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e7a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 0x7f9e7415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e7a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7415c7b0 con 0x7f9e7415bdf0 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e7a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e740c5950 con 0x7f9e540047c0 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e7a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e741e7b60 con 0x7f9e740c4fd0 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 0x7f9e7415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 0x7f9e7415c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41412/0 (socket says 172.21.15.53:41412) 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.042+0000 7f9e73e00640 1 -- 172.21.15.53:0/1644322786 learned_addr learned my addr 172.21.15.53:0/1644322786 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73400640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540047c0 0x7f9e741e6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e72a00640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e740c4fd0 0x7f9e740c5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73e00640 1 -- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e740c4fd0 msgr2=0x7f9e740c5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73e00640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e740c4fd0 0x7f9e740c5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73e00640 1 -- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540047c0 msgr2=0x7f9e741e6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73e00640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540047c0 0x7f9e741e6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73e00640 1 -- 172.21.15.53:0/1644322786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e741e6f50 con 0x7f9e7415bdf0 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e72a00640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e740c4fd0 0x7f9e740c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e73e00640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 0x7f9e7415c270 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9e740cde90 tx=0x7f9e6c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c016420 con 0x7f9e7415bdf0 2024-01-16T15:18:56.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c0165a0 con 0x7f9e7415bdf0 2024-01-16T15:18:56.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6c016950 con 0x7f9e7415bdf0 2024-01-16T15:18:56.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.043+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e741e7230 con 0x7f9e7415bdf0 2024-01-16T15:18:56.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.044+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e741e77f0 con 0x7f9e7415bdf0 2024-01-16T15:18:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.045+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e741e79b0 con 0x7f9e7415bdf0 2024-01-16T15:18:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.045+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 39) v1 ==== 91018+0+0 (secure 0 0 0) 0x7f9e6c016ab0 con 0x7f9e7415bdf0 2024-01-16T15:18:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.046+0000 7f9e6b400640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9e4c06c340 0x7f9e4c06e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.047+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9e6c0896f0 con 0x7f9e7415bdf0 2024-01-16T15:18:56.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.047+0000 7f9e73400640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9e4c06c340 0x7f9e4c06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:56.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.048+0000 7f9e73400640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9e4c06c340 0x7f9e4c06e800 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e740cebf0 tx=0x7f9e64000ed0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:56.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.051+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9e6c055c90 con 0x7f9e7415bdf0 2024-01-16T15:18:56.128 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:56.136 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:18:56.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:56.284+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone66973", "target": ["mon-mgr", ""]}) v1 -- 0x7f9e740cc4d0 con 0x7f9e4c06c340 2024-01-16T15:18:57.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.208+0000 7f9e6b400640 1 -- 172.21.15.53:0/1644322786 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+243 (secure 0 0 0) 0x7f9e64001c70 con 0x7f9e4c06c340 2024-01-16T15:18:57.198 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:18:57.198 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:18:57.198 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "canceled", 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "failure": { 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "errno": "4", 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: "error_msg": "user interrupted clone operation" 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:57.199 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.211+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9e4c06c340 msgr2=0x7f9e4c06e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.211+0000 7f9e7a800640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9e4c06c340 0x7f9e4c06e800 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f9e740cebf0 tx=0x7f9e64000ed0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.211+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 msgr2=0x7f9e7415c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.211+0000 7f9e7a800640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 0x7f9e7415c270 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f9e740cde90 tx=0x7f9e6c00f520 comp rx=0 tx=0).stop 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 shutdown_connections 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9e4c06c340 0x7f9e4c06e800 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9e7415bdf0 0x7f9e7415c270 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9e740c4fd0 0x7f9e740c5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 --2- 172.21.15.53:0/1644322786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9e540047c0 0x7f9e741e6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 >> 172.21.15.53:0/1644322786 conn(0x7f9e740d1020 msgr2=0x7f9e740cc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:57.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 shutdown_connections 2024-01-16T15:18:57.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.212+0000 7f9e7a800640 1 -- 172.21.15.53:0/1644322786 wait complete. 2024-01-16T15:18:57.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format json-pretty 2024-01-16T15:18:57.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.502+0000 7feaf9e00640 1 Processor -- start 2024-01-16T15:18:57.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.502+0000 7feaf9e00640 1 -- start start 2024-01-16T15:18:57.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.504+0000 7feaf9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 0x7feaf415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.504+0000 7feaf9e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf4070c40 con 0x7feaf40cf290 2024-01-16T15:18:57.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.504+0000 7feaf9e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feaf406c470 con 0x7feaf40b3ae0 2024-01-16T15:18:57.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.504+0000 7feaf9e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf4071450 con 0x7feaf40ae5a0 2024-01-16T15:18:57.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.504+0000 7feaf3400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7feaf40cf290 0x7feaf40b1fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33306/0 (socket says 172.21.15.53:33306) 2024-01-16T15:18:57.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.504+0000 7feaf3400640 1 -- 172.21.15.53:0/3306052210 learned_addr learned my addr 172.21.15.53:0/3306052210 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:18:57.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 0x7feaf415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197526268 0 0) 0x7feaf4070c40 con 0x7feaf40cf290 2024-01-16T15:18:57.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fead4003520 con 0x7feaf40cf290 2024-01-16T15:18:57.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2570447554 0 0) 0x7fead4003520 con 0x7feaf40cf290 2024-01-16T15:18:57.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 msgr2=0x7feaf415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:57.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 0x7feaf415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 >> v1:172.21.15.146:6789/0 conn(0x7feaf40b3ae0 legacy=0x7feaf406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf406c750 con 0x7feaf40cf290 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7feaec0019f0 con 0x7feaf40cf290 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 >> v1:172.21.15.53:6789/0 conn(0x7feaf40cf290 legacy=0x7feaf40b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fead40039a0 0x7fead4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40047c0 0x7fead4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 0x7fead4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf4071450 con 0x7fead40039a0 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bac0 con 0x7fead40047c0 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.505+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fead400bc80 con 0x7feaf40ae5a0 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fead40039a0 0x7fead4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf3400640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 0x7fead4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feae9e00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40047c0 0x7fead4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf2a00640 1 -- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 msgr2=0x7fead4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:57.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf40ae5a0 0x7fead4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf2a00640 1 -- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40047c0 msgr2=0x7fead4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:57.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40047c0 0x7fead4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.506+0000 7feaf2a00640 1 -- 172.21.15.53:0/3306052210 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead4003520 con 0x7fead40039a0 2024-01-16T15:18:57.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.508+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fead40039a0 0x7fead4003dc0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7feae4009a90 tx=0x7fead40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c34d67ad1070d7d4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:57.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.509+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae402c680 con 0x7fead40039a0 2024-01-16T15:18:57.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.509+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feae402c7e0 con 0x7fead40039a0 2024-01-16T15:18:57.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.509+0000 7feaf2000640 1 -- 172.21.15.53:0/3306052210 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feae402cb70 con 0x7fead40039a0 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.509+0000 7feaf9e00640 1 -- 172.21.15.53:0/3306052210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fead40039a0 msgr2=0x7fead4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.509+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3306052210 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fead40039a0 0x7fead4003dc0 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7feae4009a90 tx=0x7fead40057f0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.509+0000 7feaf3400640 1 -- 172.21.15.53:0/3306052210 reap_dead start 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.510+0000 7feaf9e00640 1 -- 172.21.15.53:0/3306052210 shutdown_connections 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.510+0000 7feaf9e00640 1 -- 172.21.15.53:0/3306052210 >> 172.21.15.53:0/3306052210 conn(0x7feaf406a800 msgr2=0x7feaf406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.510+0000 7feaf9e00640 1 -- 172.21.15.53:0/3306052210 shutdown_connections 2024-01-16T15:18:57.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.510+0000 7feaf9e00640 1 -- 172.21.15.53:0/3306052210 wait complete. 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.512+0000 7feaf9e00640 1 Processor -- start 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.512+0000 7feaf9e00640 1 -- start start 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf9e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40039a0 0x7feaf4147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 0x7feaf41f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf9e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf415f500 0x7feaf415f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf9e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf41f6f70 con 0x7feaf41f6090 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf9e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaf41f6a50 con 0x7fead40039a0 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf9e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feaf415fef0 con 0x7feaf415f500 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 0x7feaf41f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 0x7feaf41f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41422/0 (socket says 172.21.15.53:41422) 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 -- 172.21.15.53:0/3762771760 learned_addr learned my addr 172.21.15.53:0/3762771760 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40039a0 0x7feaf4147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 -- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf415f500 msgr2=0x7feaf415f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf415f500 0x7feaf415f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 -- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40039a0 msgr2=0x7feaf4147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40039a0 0x7feaf4147010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.513+0000 7feae9e00640 1 -- 172.21.15.53:0/3762771760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4148010 con 0x7feaf41f6090 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.514+0000 7feae9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 0x7feaf41f6510 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7feadc00e550 tx=0x7feadc00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.514+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc0052e0 con 0x7feaf41f6090 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.514+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feadc005440 con 0x7feaf41f6090 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.514+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaf4160110 con 0x7feaf41f6090 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.514+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc003810 con 0x7feaf41f6090 2024-01-16T15:18:57.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.514+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaf41479c0 con 0x7feaf41f6090 2024-01-16T15:18:57.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.515+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaf41481a0 con 0x7feaf41f6090 2024-01-16T15:18:57.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.516+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7feadc003970 con 0x7feaf41f6090 2024-01-16T15:18:57.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.517+0000 7feaebe00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feacc06c3d0 0x7feacc06e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:57.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.517+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7feadc0884f0 con 0x7feaf41f6090 2024-01-16T15:18:57.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.518+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feacc06c3d0 0x7feacc06e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:57.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.519+0000 7feaf2a00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feacc06c3d0 0x7feacc06e890 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7feae4036000 tx=0x7feae4030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:57.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.522+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feadc054db0 con 0x7feaf41f6090 2024-01-16T15:18:57.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.792+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feaf40675d0 con 0x7feaf41f6090 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.793+0000 7feaebe00640 1 -- 172.21.15.53:0/3762771760 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1853 (secure 0 0 0) 0x7feadc0583d0 con 0x7feaf41f6090 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: "fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: "health": { 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "HEALTH_OK", 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: "checks": {}, 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: "mutes": [] 2024-01-16T15:18:57.783 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "election_epoch": 4, 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum": [ 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: 0, 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: 1, 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: 2 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_names": [ 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "a", 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "b", 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "c" 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "quorum_age": 1164, 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "monmap": { 2024-01-16T15:18:57.784 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 1, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "min_mon_release_name": "reef", 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "num_mons": 3 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap": { 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 99, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "num_osds": 8, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "num_up_osds": 8, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_up_since": 1705417185, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "num_in_osds": 8, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_in_since": 1705417176, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "num_remapped_pgs": 0 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.785 INFO:teuthology.orchestra.run.smithi053.stdout: "pgmap": { 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "pgs_by_state": [ 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "state_name": "active+clean", 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "count": 97 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pgs": 97, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "num_pools": 3, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "num_objects": 799, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "data_bytes": 3222346151, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_used": 7052025856, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_avail": 766042087424, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "bytes_total": 773094113280, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "read_bytes_sec": 767, 2024-01-16T15:18:57.786 INFO:teuthology.orchestra.run.smithi053.stdout: "write_bytes_sec": 5970, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "read_op_per_sec": 0, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "write_op_per_sec": 2 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "fsmap": { 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 65, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 7, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "up": 1, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "in": 1, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "max": 1, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "by_rank": [ 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "filesystem_id": 7, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "rank": 0, 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "c", 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "status": "up:active", 2024-01-16T15:18:57.787 INFO:teuthology.orchestra.run.smithi053.stdout: "gid": 8508 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "up:standby": 3 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "mgrmap": { 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "available": true, 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "num_standbys": 1, 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "modules": [ 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "iostat", 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "nfs", 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "restful" 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: ], 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:18:57.788 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout: "servicemap": { 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout: "epoch": 21, 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout: "modified": "2024-01-16T15:16:28.620434+0000", 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout: "services": {} 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout: "progress_events": {} 2024-01-16T15:18:57.789 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.795+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feacc06c3d0 msgr2=0x7feacc06e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.795+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feacc06c3d0 0x7feacc06e890 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7feae4036000 tx=0x7feae4030000 comp rx=0 tx=0).stop 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.795+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 msgr2=0x7feaf41f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.795+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 0x7feaf41f6510 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7feadc00e550 tx=0x7feadc00ea10 comp rx=0 tx=0).stop 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 shutdown_connections 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7feacc06c3d0 0x7feacc06e890 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feaf415f500 0x7feaf415f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feaf41f6090 0x7feaf41f6510 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 --2- 172.21.15.53:0/3762771760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fead40039a0 0x7feaf4147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:57.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 >> 172.21.15.53:0/3762771760 conn(0x7feaf406a800 msgr2=0x7feaf415da20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:57.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 shutdown_connections 2024-01-16T15:18:57.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:57.796+0000 7feaf9e00640 1 -- 172.21.15.53:0/3762771760 wait complete. 2024-01-16T15:18:57.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:18:57.806 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls -lh volumes/_nogroup/subvol_clone66971/625670eb-e79e-42b9-ad02-aa932888c11a) 2024-01-16T15:18:57.867 INFO:teuthology.orchestra.run.smithi053.stderr:ls: cannot access 'volumes/_nogroup/subvol_clone66971/625670eb-e79e-42b9-ad02-aa932888c11a': No such file or directory 2024-01-16T15:18:57.868 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-01-16T15:18:57.869 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:18:58.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.097+0000 7f0da0800640 1 Processor -- start 2024-01-16T15:18:58.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.098+0000 7f0da0800640 1 -- start start 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.099+0000 7f0da0800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0d980cd170 0x7f0d98071070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.099+0000 7f0da0800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d98071690 con 0x7f0d980cf9a0 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.099+0000 7f0da0800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d98071970 con 0x7f0d9814b2c0 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.099+0000 7f0da0800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9814b6a0 con 0x7f0d980cd170 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.100+0000 7f0d9d400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0d980cf9a0 0x7f0d98070960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33322/0 (socket says 172.21.15.53:33322) 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.100+0000 7f0d9d400640 1 -- 172.21.15.53:0/3060754086 learned_addr learned my addr 172.21.15.53:0/3060754086 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:18:58.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.100+0000 7f0d9e800640 1 --2- 172.21.15.53:0/3060754086 >> v2:172.21.15.146:3300/0 conn(0x7f0d980cd170 0x7f0d98071070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.100+0000 7f0d9ca00640 1 -- 172.21.15.53:0/3060754086 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115966897 0 0) 0x7f0d98071690 con 0x7f0d980cf9a0 2024-01-16T15:18:58.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.100+0000 7f0d9ca00640 1 -- 172.21.15.53:0/3060754086 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d78003520 con 0x7f0d980cf9a0 2024-01-16T15:18:58.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.101+0000 7f0d9ca00640 1 -- 172.21.15.53:0/3060754086 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523920974 0 0) 0x7f0d78003520 con 0x7f0d980cf9a0 2024-01-16T15:18:58.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.101+0000 7f0d9e800640 1 -- 172.21.15.53:0/3060754086 >> v1:172.21.15.146:6789/0 conn(0x7f0d9814b2c0 legacy=0x7f0d980cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.101+0000 7f0d9e800640 1 -- 172.21.15.53:0/3060754086 >> v1:172.21.15.53:6789/0 conn(0x7f0d980cf9a0 legacy=0x7f0d98070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.101+0000 7f0d9e800640 1 -- 172.21.15.53:0/3060754086 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d98071c50 con 0x7f0d980cd170 2024-01-16T15:18:58.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.102+0000 7f0d9e800640 1 --2- 172.21.15.53:0/3060754086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cd170 0x7f0d98071070 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009d90 tx=0x7f0d9002be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5ccaf64aaedd7cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:58.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0d9ca00640 1 -- 172.21.15.53:0/3060754086 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80001670 con 0x7f0d980cd170 2024-01-16T15:18:58.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0d9ca00640 1 -- 172.21.15.53:0/3060754086 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9002c940 con 0x7f0d980cd170 2024-01-16T15:18:58.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0d9ca00640 1 -- 172.21.15.53:0/3060754086 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9002ccb0 con 0x7f0d980cd170 2024-01-16T15:18:58.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0da0800640 1 -- 172.21.15.53:0/3060754086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cd170 msgr2=0x7f0d98071070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0da0800640 1 --2- 172.21.15.53:0/3060754086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cd170 0x7f0d98071070 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009d90 tx=0x7f0d9002be10 comp rx=0 tx=0).stop 2024-01-16T15:18:58.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0da0800640 1 -- 172.21.15.53:0/3060754086 shutdown_connections 2024-01-16T15:18:58.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0da0800640 1 --2- 172.21.15.53:0/3060754086 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cd170 0x7f0d98071070 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0da0800640 1 -- 172.21.15.53:0/3060754086 >> 172.21.15.53:0/3060754086 conn(0x7f0d9806a800 msgr2=0x7f0d9806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:58.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.103+0000 7f0da0800640 1 -- 172.21.15.53:0/3060754086 shutdown_connections 2024-01-16T15:18:58.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.104+0000 7f0da0800640 1 -- 172.21.15.53:0/3060754086 wait complete. 2024-01-16T15:18:58.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.105+0000 7f0da0800640 1 Processor -- start 2024-01-16T15:18:58.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.105+0000 7f0da0800640 1 -- start start 2024-01-16T15:18:58.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.106+0000 7f0da0800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 0x7f0d9815c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.106+0000 7f0da0800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cf9a0 0x7f0d98163ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.106+0000 7f0da0800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d9814b2c0 0x7f0d9813f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.106+0000 7f0da0800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d98164c70 con 0x7f0d980cd170 2024-01-16T15:18:58.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.106+0000 7f0da0800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981612a0 con 0x7f0d980cf9a0 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.106+0000 7f0da0800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d98162f20 con 0x7f0d9814b2c0 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 0x7f0d9815c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 0x7f0d9815c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41434/0 (socket says 172.21.15.53:41434) 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 -- 172.21.15.53:0/2720758410 learned_addr learned my addr 172.21.15.53:0/2720758410 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9d400640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cf9a0 0x7f0d98163ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9e800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d9814b2c0 0x7f0d9813f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 -- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d9814b2c0 msgr2=0x7f0d9813f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d9814b2c0 0x7f0d9813f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 -- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cf9a0 msgr2=0x7f0d98163ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cf9a0 0x7f0d98163ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.107+0000 7f0d9de00640 1 -- 172.21.15.53:0/2720758410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d90009a70 con 0x7f0d980cd170 2024-01-16T15:18:58.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.108+0000 7f0d9de00640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 0x7f0d9815c2c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800c400 tx=0x7f0d8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:58.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.108+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d880052d0 con 0x7f0d980cd170 2024-01-16T15:18:58.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.108+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d88005430 con 0x7f0d980cd170 2024-01-16T15:18:58.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.108+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88003830 con 0x7f0d980cd170 2024-01-16T15:18:58.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.108+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d98163200 con 0x7f0d980cd170 2024-01-16T15:18:58.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.108+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d981637c0 con 0x7f0d980cd170 2024-01-16T15:18:58.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.109+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d980675d0 con 0x7f0d980cd170 2024-01-16T15:18:58.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.110+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f0d88003990 con 0x7f0d980cd170 2024-01-16T15:18:58.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.111+0000 7f0d96000640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d7006c3d0 0x7f0d7006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.111+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f0d88087fd0 con 0x7f0d980cd170 2024-01-16T15:18:58.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.112+0000 7f0d9d400640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d7006c3d0 0x7f0d7006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.113+0000 7f0d9d400640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d7006c3d0 0x7f0d7006e890 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0d9815cc10 tx=0x7f0d80009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:58.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.116+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d88054550 con 0x7f0d980cd170 2024-01-16T15:18:58.332 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.342+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d980cbc40 con 0x7f0d7006c3d0 2024-01-16T15:18:58.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.345+0000 7f0d96000640 1 -- 172.21.15.53:0/2720758410 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f0d80001510 con 0x7f0d7006c3d0 2024-01-16T15:18:58.335 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:18:58.335 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:58.335 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:18:58.336 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:58.336 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d7006c3d0 msgr2=0x7f0d7006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d7006c3d0 0x7f0d7006e890 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0d9815cc10 tx=0x7f0d80009040 comp rx=0 tx=0).stop 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 msgr2=0x7f0d9815c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 0x7f0d9815c2c0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800c400 tx=0x7f0d8800c8c0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 shutdown_connections 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0d7006c3d0 0x7f0d7006e890 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d9814b2c0 0x7f0d9813f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d980cf9a0 0x7f0d98163ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 --2- 172.21.15.53:0/2720758410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d980cd170 0x7f0d9815c2c0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 >> 172.21.15.53:0/2720758410 conn(0x7f0d9806a800 msgr2=0x7f0d9806cbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:58.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 shutdown_connections 2024-01-16T15:18:58.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.348+0000 7f0da0800640 1 -- 172.21.15.53:0/2720758410 wait complete. 2024-01-16T15:18:58.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:18:58.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.593+0000 7fabf7e00640 1 Processor -- start 2024-01-16T15:18:58.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.594+0000 7fabf7e00640 1 -- start start 2024-01-16T15:18:58.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.595+0000 7fabf7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.595+0000 7fabf7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fabf80cddf0 0x7fabf80b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.595+0000 7fabf7e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabf80ce440 con 0x7fabf813fae0 2024-01-16T15:18:58.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.595+0000 7fabf7e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabf813ff60 con 0x7fabf80cddf0 2024-01-16T15:18:58.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.595+0000 7fabf7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf80ce720 con 0x7fabf813cf10 2024-01-16T15:18:58.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf6000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fabf813fae0 0x7fabf80cd6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33334/0 (socket says 172.21.15.53:33334) 2024-01-16T15:18:58.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf6000640 1 -- 172.21.15.53:0/3967591614 learned_addr learned my addr 172.21.15.53:0/3967591614 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:18:58.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf7400640 1 --2- 172.21.15.53:0/3967591614 >> v2:172.21.15.53:3300/0 conn(0x7fabf80cddf0 0x7fabf80b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf6a00640 1 --2- 172.21.15.53:0/3967591614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf5600640 1 -- 172.21.15.53:0/3967591614 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53832199 0 0) 0x7fabf80ce440 con 0x7fabf813fae0 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf5600640 1 -- 172.21.15.53:0/3967591614 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd8003520 con 0x7fabf813fae0 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf7400640 1 -- 172.21.15.53:0/3967591614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 msgr2=0x7fabf813f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf7400640 1 --2- 172.21.15.53:0/3967591614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf7400640 1 -- 172.21.15.53:0/3967591614 >> v1:172.21.15.53:6789/0 conn(0x7fabf813fae0 legacy=0x7fabf80cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.596+0000 7fabf7400640 1 -- 172.21.15.53:0/3967591614 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf80cea00 con 0x7fabf80cddf0 2024-01-16T15:18:58.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.597+0000 7fabf7400640 1 --2- 172.21.15.53:0/3967591614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf80b2590 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009d90 tx=0x7fabf002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3afa16cd06cfb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:58.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.598+0000 7fabf5600640 1 -- 172.21.15.53:0/3967591614 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf002cc50 con 0x7fabf80cddf0 2024-01-16T15:18:58.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.598+0000 7fabf5600640 1 -- 172.21.15.53:0/3967591614 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf002cdb0 con 0x7fabf80cddf0 2024-01-16T15:18:58.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.598+0000 7fabf5600640 1 -- 172.21.15.53:0/3967591614 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabf0034660 con 0x7fabf80cddf0 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.598+0000 7fabf7e00640 1 -- 172.21.15.53:0/3967591614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 msgr2=0x7fabf80b2590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.598+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3967591614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf80b2590 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fabf0009d90 tx=0x7fabf002bea0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.599+0000 7fabf7e00640 1 -- 172.21.15.53:0/3967591614 shutdown_connections 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.599+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3967591614 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf80b2590 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.599+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3967591614 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.599+0000 7fabf7e00640 1 -- 172.21.15.53:0/3967591614 >> 172.21.15.53:0/3967591614 conn(0x7fabf806a7d0 msgr2=0x7fabf806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.599+0000 7fabf7e00640 1 -- 172.21.15.53:0/3967591614 shutdown_connections 2024-01-16T15:18:58.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.599+0000 7fabf7e00640 1 -- 172.21.15.53:0/3967591614 wait complete. 2024-01-16T15:18:58.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.601+0000 7fabf7e00640 1 Processor -- start 2024-01-16T15:18:58.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.601+0000 7fabf7e00640 1 -- start start 2024-01-16T15:18:58.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf81365d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf813fae0 0x7fabf80711b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8136b10 con 0x7fabf80cddf0 2024-01-16T15:18:58.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf8154df0 con 0x7fabf813fae0 2024-01-16T15:18:58.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabf81550d0 con 0x7fabf813cf10 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf81365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf81365d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41446/0 (socket says 172.21.15.53:41446) 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 -- 172.21.15.53:0/3843865442 learned_addr learned my addr 172.21.15.53:0/3843865442 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 -- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 msgr2=0x7fabf813f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 -- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf813fae0 msgr2=0x7fabf80711b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf813fae0 0x7fabf80711b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.602+0000 7fabf6a00640 1 -- 172.21.15.53:0/3843865442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf0009a70 con 0x7fabf80cddf0 2024-01-16T15:18:58.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.603+0000 7fabf6a00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf81365d0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d800 tx=0x7fabe800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:58.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.603+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe800b6c0 con 0x7fabf80cddf0 2024-01-16T15:18:58.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.603+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe800b820 con 0x7fabf80cddf0 2024-01-16T15:18:58.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.603+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe800bbb0 con 0x7fabf80cddf0 2024-01-16T15:18:58.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.603+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf81553b0 con 0x7fabf80cddf0 2024-01-16T15:18:58.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.603+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf8155970 con 0x7fabf80cddf0 2024-01-16T15:18:58.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.605+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf8155b30 con 0x7fabf80cddf0 2024-01-16T15:18:58.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.606+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7fabe800bd10 con 0x7fabf80cddf0 2024-01-16T15:18:58.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.607+0000 7fabeea00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabd006c3a0 0x7fabd006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:58.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.607+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fabe8088cd0 con 0x7fabf80cddf0 2024-01-16T15:18:58.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.607+0000 7fabf6000640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabd006c3a0 0x7fabd006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:58.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.609+0000 7fabf6000640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabd006c3a0 0x7fabd006e860 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fabf8071dd0 tx=0x7fabe0007930 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:58.601 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.611+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fabe8055590 con 0x7fabf80cddf0 2024-01-16T15:18:58.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.833+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fabf80b2070 con 0x7fabd006c3a0 2024-01-16T15:18:58.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.849+0000 7fabeea00640 1 -- 172.21.15.53:0/3843865442 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+185 (secure 0 0 0) 0x7fabe00058b0 con 0x7fabd006c3a0 2024-01-16T15:18:58.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:18:58.839 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:18:58.839 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:58.839 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:18:58.839 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone66971" 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone66973" 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone66972" 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:18:58.840 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:18:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabd006c3a0 msgr2=0x7fabd006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabd006c3a0 0x7fabd006e860 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fabf8071dd0 tx=0x7fabe0007930 comp rx=0 tx=0).stop 2024-01-16T15:18:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 msgr2=0x7fabf81365d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf81365d0 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fabe800d800 tx=0x7fabe800dcc0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 shutdown_connections 2024-01-16T15:18:58.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fabd006c3a0 0x7fabd006e860 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fabf813fae0 0x7fabf80711b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fabf813cf10 0x7fabf813f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 --2- 172.21.15.53:0/3843865442 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fabf80cddf0 0x7fabf81365d0 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:58.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.851+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 >> 172.21.15.53:0/3843865442 conn(0x7fabf806a7d0 msgr2=0x7fabf80af410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:58.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.852+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 shutdown_connections 2024-01-16T15:18:58.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:58.852+0000 7fabf7e00640 1 -- 172.21.15.53:0/3843865442 wait complete. 2024-01-16T15:18:58.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:18:59.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.105+0000 7f4acd000640 1 Processor -- start 2024-01-16T15:18:59.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.105+0000 7f4acd000640 1 -- start start 2024-01-16T15:18:59.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4acd000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ac8143c40 0x7f4ac8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:59.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4acd000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 0x7f4ac806d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:59.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4acd000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ac8070e50 con 0x7f4ac80717d0 2024-01-16T15:18:59.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4acd000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ac80723c0 con 0x7f4ac8143c40 2024-01-16T15:18:59.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4acd000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac8144020 con 0x7f4ac80ccc10 2024-01-16T15:18:59.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4ac7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ac8143c40 0x7f4ac8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:59.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4ac7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ac8143c40 0x7f4ac8071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41454/0 (socket says 172.21.15.53:41454) 2024-01-16T15:18:59.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.107+0000 7f4ac7400640 1 -- 172.21.15.53:0/427399436 learned_addr learned my addr 172.21.15.53:0/427399436 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:59.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.108+0000 7f4ac7400640 1 -- 172.21.15.53:0/427399436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 msgr2=0x7f4ac806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:59.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.108+0000 7f4ac7400640 1 --2- 172.21.15.53:0/427399436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 0x7f4ac806d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:59.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.108+0000 7f4ac7400640 1 -- 172.21.15.53:0/427399436 >> v1:172.21.15.146:6789/0 conn(0x7f4ac80717d0 legacy=0x7f4ac80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:18:59.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.108+0000 7f4ac7400640 1 -- 172.21.15.53:0/427399436 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ac8071e00 con 0x7f4ac8143c40 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.109+0000 7f4ac7400640 1 --2- 172.21.15.53:0/427399436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac8143c40 0x7f4ac8071290 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009870 tx=0x7f4ab802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4c477feca48f3f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.110+0000 7f4ac6000640 1 -- 172.21.15.53:0/427399436 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab80317c0 con 0x7f4ac8143c40 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.110+0000 7f4ac6000640 1 -- 172.21.15.53:0/427399436 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab8031920 con 0x7f4ac8143c40 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.110+0000 7f4ac6000640 1 -- 172.21.15.53:0/427399436 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8031cb0 con 0x7f4ac8143c40 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.110+0000 7f4acd000640 1 -- 172.21.15.53:0/427399436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac8143c40 msgr2=0x7f4ac8071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.110+0000 7f4acd000640 1 --2- 172.21.15.53:0/427399436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac8143c40 0x7f4ac8071290 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8009870 tx=0x7f4ab802b860 comp rx=0 tx=0).stop 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.111+0000 7f4acd000640 1 -- 172.21.15.53:0/427399436 shutdown_connections 2024-01-16T15:18:59.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.111+0000 7f4acd000640 1 --2- 172.21.15.53:0/427399436 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 0x7f4ac806d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:59.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.111+0000 7f4acd000640 1 --2- 172.21.15.53:0/427399436 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac8143c40 0x7f4ac8071290 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:59.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.111+0000 7f4acd000640 1 -- 172.21.15.53:0/427399436 >> 172.21.15.53:0/427399436 conn(0x7f4ac806a800 msgr2=0x7f4ac806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:18:59.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.111+0000 7f4acd000640 1 -- 172.21.15.53:0/427399436 shutdown_connections 2024-01-16T15:18:59.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.111+0000 7f4acd000640 1 -- 172.21.15.53:0/427399436 wait complete. 2024-01-16T15:18:59.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.112+0000 7f4acd000640 1 Processor -- start 2024-01-16T15:18:59.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.113+0000 7f4acd000640 1 -- start start 2024-01-16T15:18:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4acd000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 0x7f4ac80aece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4acd000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 0x7f4ac8135470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:59.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4acd000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ac8143c40 0x7f4ac81359b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4acd000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac8157110 con 0x7f4ac80717d0 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4acd000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac81572f0 con 0x7f4ac8143c40 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4acd000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ac81575d0 con 0x7f4ac80ccc10 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 0x7f4ac80aece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 0x7f4ac80aece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:41464/0 (socket says 172.21.15.53:41464) 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 -- 172.21.15.53:0/2857222494 learned_addr learned my addr 172.21.15.53:0/2857222494 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 -- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 msgr2=0x7f4ac8135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 0x7f4ac8135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 -- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ac8143c40 msgr2=0x7f4ac81359b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:18:59.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ac8143c40 0x7f4ac81359b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.114+0000 7f4ac7400640 1 -- 172.21.15.53:0/2857222494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab80094d0 con 0x7f4ac80717d0 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.115+0000 7f4ac7400640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 0x7f4ac80aece0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8000c00 tx=0x7f4ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.115+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8032860 con 0x7f4ac80717d0 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.115+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab80329e0 con 0x7f4ac80717d0 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.115+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab8032d90 con 0x7f4ac80717d0 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.115+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ac81578b0 con 0x7f4ac80717d0 2024-01-16T15:18:59.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.115+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ac8157e70 con 0x7f4ac80717d0 2024-01-16T15:18:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.116+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ac8067450 con 0x7f4ac80717d0 2024-01-16T15:18:59.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.118+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f4ab804e450 con 0x7f4ac80717d0 2024-01-16T15:18:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.119+0000 7f4abf400640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aa006c3d0 0x7f4aa006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:18:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.119+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4ab80aaa10 con 0x7f4ac80717d0 2024-01-16T15:18:59.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.119+0000 7f4ac6a00640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aa006c3d0 0x7f4aa006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:18:59.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.121+0000 7f4ac6a00640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aa006c3d0 0x7f4aa006e890 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0006fd0 tx=0x7f4ab0006bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:18:59.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.124+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4ab80772d0 con 0x7f4ac80717d0 2024-01-16T15:18:59.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:18:59.348+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ac806cc40 con 0x7f4aa006c3d0 2024-01-16T15:19:02.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.212+0000 7f4abf400640 1 -- 172.21.15.53:0/2857222494 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f4ab0013430 con 0x7f4aa006c3d0 2024-01-16T15:19:02.203 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:02.203 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:19:02.203 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:02.203 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:19:02.203 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:02.204 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aa006c3d0 msgr2=0x7f4aa006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aa006c3d0 0x7f4aa006e890 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f4ab0006fd0 tx=0x7f4ab0006bc0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 msgr2=0x7f4ac80aece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 0x7f4ac80aece0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f4ab8000c00 tx=0x7f4ab80046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 shutdown_connections 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4aa006c3d0 0x7f4aa006e890 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ac8143c40 0x7f4ac81359b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ac80ccc10 0x7f4ac8135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 --2- 172.21.15.53:0/2857222494 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ac80717d0 0x7f4ac80aece0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.216+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 >> 172.21.15.53:0/2857222494 conn(0x7f4ac806a800 msgr2=0x7f4ac8139700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.217+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 shutdown_connections 2024-01-16T15:19:02.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.217+0000 7f4acd000640 1 -- 172.21.15.53:0/2857222494 wait complete. 2024-01-16T15:19:02.230 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:19:02.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.474+0000 7f62cf200640 1 Processor -- start 2024-01-16T15:19:02.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.475+0000 7f62cf200640 1 -- start start 2024-01-16T15:19:02.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.476+0000 7f62cf200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f62c8160920 0x7f62c80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.477+0000 7f62cf200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80d0a10 con 0x7f62c80cfea0 2024-01-16T15:19:02.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.477+0000 7f62cf200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c8161e30 con 0x7f62c80cf9f0 2024-01-16T15:19:02.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.477+0000 7f62cf200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c80d1f50 con 0x7f62c8160920 2024-01-16T15:19:02.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.477+0000 7f62ce800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f62c80cfea0 0x7f62c8161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50220/0 (socket says 172.21.15.53:50220) 2024-01-16T15:19:02.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.477+0000 7f62ce800640 1 -- 172.21.15.53:0/2954607460 learned_addr learned my addr 172.21.15.53:0/2954607460 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:02.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.477+0000 7f62cde00640 1 --2- 172.21.15.53:0/2954607460 >> v2:172.21.15.146:3300/0 conn(0x7f62c8160920 0x7f62c80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.478+0000 7f62cca00640 1 -- 172.21.15.53:0/2954607460 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530453799 0 0) 0x7f62c80d0a10 con 0x7f62c80cfea0 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.478+0000 7f62cca00640 1 -- 172.21.15.53:0/2954607460 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62a8003520 con 0x7f62c80cfea0 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.478+0000 7f62cde00640 1 -- 172.21.15.53:0/2954607460 >> v1:172.21.15.146:6789/0 conn(0x7f62c80cf9f0 legacy=0x7f62c8161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.478+0000 7f62cde00640 1 -- 172.21.15.53:0/2954607460 >> v1:172.21.15.53:6789/0 conn(0x7f62c80cfea0 legacy=0x7f62c8161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.478+0000 7f62cca00640 1 -- 172.21.15.53:0/2954607460 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1528743177 0 0) 0x7f62a8003520 con 0x7f62c80cfea0 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.478+0000 7f62cde00640 1 -- 172.21.15.53:0/2954607460 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c80cf080 con 0x7f62c8160920 2024-01-16T15:19:02.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.480+0000 7f62cde00640 1 --2- 172.21.15.53:0/2954607460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c80b3f30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f62b80097a0 tx=0x7f62b802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=755bdba16df2e1cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.480+0000 7f62cca00640 1 -- 172.21.15.53:0/2954607460 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8030610 con 0x7f62c8160920 2024-01-16T15:19:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.480+0000 7f62cca00640 1 -- 172.21.15.53:0/2954607460 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b8030770 con 0x7f62c8160920 2024-01-16T15:19:02.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.480+0000 7f62cca00640 1 -- 172.21.15.53:0/2954607460 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b8030b00 con 0x7f62c8160920 2024-01-16T15:19:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 -- 172.21.15.53:0/2954607460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 msgr2=0x7f62c80b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 --2- 172.21.15.53:0/2954607460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c80b3f30 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f62b80097a0 tx=0x7f62b802b960 comp rx=0 tx=0).stop 2024-01-16T15:19:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 -- 172.21.15.53:0/2954607460 shutdown_connections 2024-01-16T15:19:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 --2- 172.21.15.53:0/2954607460 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c80b3f30 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 -- 172.21.15.53:0/2954607460 >> 172.21.15.53:0/2954607460 conn(0x7f62c80d3090 msgr2=0x7f62c80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 -- 172.21.15.53:0/2954607460 shutdown_connections 2024-01-16T15:19:02.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.481+0000 7f62cf200640 1 -- 172.21.15.53:0/2954607460 wait complete. 2024-01-16T15:19:02.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.483+0000 7f62cf200640 1 Processor -- start 2024-01-16T15:19:02.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.483+0000 7f62cf200640 1 -- start start 2024-01-16T15:19:02.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62c80cf9f0 0x7f62c80cab40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cf200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 0x7f62c81f6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c81f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cf200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c815c6c0 con 0x7f62c80cfea0 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80cb4e0 con 0x7f62c8160920 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80cb7c0 con 0x7f62c80cf9f0 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 0x7f62c81f6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cd400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 0x7f62c81f6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49848/0 (socket says 172.21.15.53:49848) 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cd400640 1 -- 172.21.15.53:0/3809451098 learned_addr learned my addr 172.21.15.53:0/3809451098 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62cde00640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62c80cf9f0 0x7f62c80cab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.484+0000 7f62ce800640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c81f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cd400640 1 -- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62c80cf9f0 msgr2=0x7f62c80cab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cd400640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62c80cf9f0 0x7f62c80cab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cd400640 1 -- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 msgr2=0x7f62c81f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cd400640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c81f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cd400640 1 -- 172.21.15.53:0/3809451098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62b8009400 con 0x7f62c80cfea0 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62ce800640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c81f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cd400640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 0x7f62c81f6620 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f62b000c730 tx=0x7f62b000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0003b70 con 0x7f62c80cfea0 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b0003cf0 con 0x7f62c80cfea0 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c80cbaa0 con 0x7f62c80cfea0 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0003ea0 con 0x7f62c80cfea0 2024-01-16T15:19:02.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.485+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c80cbd80 con 0x7f62c80cfea0 2024-01-16T15:19:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.487+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c81f4540 con 0x7f62c80cfea0 2024-01-16T15:19:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.488+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f62b000aa90 con 0x7f62c80cfea0 2024-01-16T15:19:02.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.489+0000 7f62c6000640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62a006c3a0 0x7f62a006e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:02.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.489+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f62b0088970 con 0x7f62c80cfea0 2024-01-16T15:19:02.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.489+0000 7f62cde00640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62a006c3a0 0x7f62a006e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:02.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.491+0000 7f62cde00640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62a006c3a0 0x7f62a006e860 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f62b8036000 tx=0x7f62b8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:02.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.493+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f62c81f4540 con 0x7f62c80cfea0 2024-01-16T15:19:02.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.722+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f62c80ce7e0 con 0x7f62a006c3a0 2024-01-16T15:19:02.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.792+0000 7f62c6000640 1 -- 172.21.15.53:0/3809451098 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f62b80309f0 con 0x7f62a006c3a0 2024-01-16T15:19:02.783 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62a006c3a0 msgr2=0x7f62a006e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62a006c3a0 0x7f62a006e860 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f62b8036000 tx=0x7f62b8008030 comp rx=0 tx=0).stop 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 msgr2=0x7f62c81f6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 0x7f62c81f6620 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f62b000c730 tx=0x7f62b000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 shutdown_connections 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f62a006c3a0 0x7f62a006e860 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f62c8160920 0x7f62c81f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f62c80cfea0 0x7f62c81f6620 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 --2- 172.21.15.53:0/3809451098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f62c80cf9f0 0x7f62c80cab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 >> 172.21.15.53:0/3809451098 conn(0x7f62c80d3090 msgr2=0x7f62c8160d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 shutdown_connections 2024-01-16T15:19:02.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:02.796+0000 7f62cf200640 1 -- 172.21.15.53:0/3809451098 wait complete. 2024-01-16T15:19:02.808 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm cephfs sv1 2024-01-16T15:19:03.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.036+0000 7f2406600640 1 Processor -- start 2024-01-16T15:19:03.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.037+0000 7f2406600640 1 -- start start 2024-01-16T15:19:03.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.039+0000 7f2406600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240015d4b0 0x7f24000d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.039+0000 7f2406600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24000b41e0 0x7f24000cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.039+0000 7f2406600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24000d0320 con 0x7f24000cf9a0 2024-01-16T15:19:03.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.039+0000 7f2406600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24000b2730 con 0x7f24000b41e0 2024-01-16T15:19:03.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.039+0000 7f2406600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000d1ec0 con 0x7f240015d4b0 2024-01-16T15:19:03.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.039+0000 7f2405c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24000b41e0 0x7f24000cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f24000b41e0 0x7f24000cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:54986/0 (socket says 172.21.15.53:54986) 2024-01-16T15:19:03.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405c00640 1 -- 172.21.15.53:0/995431452 learned_addr learned my addr 172.21.15.53:0/995431452 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:03.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405200640 1 --2- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240015d4b0 0x7f24000d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405c00640 1 -- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240015d4b0 msgr2=0x7f24000d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:03.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405c00640 1 --2- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240015d4b0 0x7f24000d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405c00640 1 -- 172.21.15.53:0/995431452 >> v1:172.21.15.146:6789/0 conn(0x7f24000cf9a0 legacy=0x7f24000cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.040+0000 7f2405c00640 1 -- 172.21.15.53:0/995431452 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24000b29d0 con 0x7f24000b41e0 2024-01-16T15:19:03.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.042+0000 7f2405c00640 1 --2- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24000b41e0 0x7f24000cfde0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009e80 tx=0x7f23f802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73c1b461954dc7aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:03.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.043+0000 7f23ff400640 1 -- 172.21.15.53:0/995431452 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f802caf0 con 0x7f24000b41e0 2024-01-16T15:19:03.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.043+0000 7f23ff400640 1 -- 172.21.15.53:0/995431452 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f802cc50 con 0x7f24000b41e0 2024-01-16T15:19:03.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.043+0000 7f23ff400640 1 -- 172.21.15.53:0/995431452 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f80343f0 con 0x7f24000b41e0 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.043+0000 7f2406600640 1 -- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24000b41e0 msgr2=0x7f24000cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.043+0000 7f2406600640 1 --2- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24000b41e0 0x7f24000cfde0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009e80 tx=0x7f23f802bf50 comp rx=0 tx=0).stop 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.044+0000 7f2406600640 1 -- 172.21.15.53:0/995431452 shutdown_connections 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.044+0000 7f2406600640 1 --2- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24000b41e0 0x7f24000cfde0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.044+0000 7f2406600640 1 --2- 172.21.15.53:0/995431452 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f240015d4b0 0x7f24000d1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.044+0000 7f2406600640 1 -- 172.21.15.53:0/995431452 >> 172.21.15.53:0/995431452 conn(0x7f24000d3000 msgr2=0x7f24000d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.044+0000 7f2406600640 1 -- 172.21.15.53:0/995431452 shutdown_connections 2024-01-16T15:19:03.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.044+0000 7f2406600640 1 -- 172.21.15.53:0/995431452 wait complete. 2024-01-16T15:19:03.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.046+0000 7f2406600640 1 Processor -- start 2024-01-16T15:19:03.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.046+0000 7f2406600640 1 -- start start 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f2406600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000b41e0 0x7f2400157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f2406600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 0x7f24001604f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f2406600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240015d4b0 0x7f2400160a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f2406600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000b5f90 con 0x7f24000cf9a0 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f2406600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24000b6270 con 0x7f240015d4b0 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f2406600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24000b6550 con 0x7f24000b41e0 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f23ffe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 0x7f24001604f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f23ffe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 0x7f24001604f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49858/0 (socket says 172.21.15.53:49858) 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.047+0000 7f23ffe00640 1 -- 172.21.15.53:0/3221910760 learned_addr learned my addr 172.21.15.53:0/3221910760 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f2405c00640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240015d4b0 0x7f2400160a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f2405200640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000b41e0 0x7f2400157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23ffe00640 1 -- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000b41e0 msgr2=0x7f2400157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23ffe00640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000b41e0 0x7f2400157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23ffe00640 1 -- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240015d4b0 msgr2=0x7f2400160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23ffe00640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240015d4b0 0x7f2400160a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23ffe00640 1 -- 172.21.15.53:0/3221910760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009b30 con 0x7f24000cf9a0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23ffe00640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 0x7f24001604f0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f23e800c450 tx=0x7f23e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e80052d0 con 0x7f24000cf9a0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23e8005430 con 0x7f24000cf9a0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e8012870 con 0x7f24000cf9a0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.048+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240015da10 con 0x7f24000cf9a0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.049+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24000b6b10 con 0x7f24000cf9a0 2024-01-16T15:19:03.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.050+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240004ed60 con 0x7f24000cf9a0 2024-01-16T15:19:03.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.051+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f23e80129d0 con 0x7f24000cf9a0 2024-01-16T15:19:03.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.052+0000 7f23fcc00640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f23d806c3a0 0x7f23d806e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.052+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f23e8088580 con 0x7f24000cf9a0 2024-01-16T15:19:03.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.052+0000 7f2405200640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f23d806c3a0 0x7f23d806e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.054+0000 7f2405200640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f23d806c3a0 0x7f23d806e860 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009a30 tx=0x7f23f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:03.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.056+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f23e8090070 con 0x7f24000cf9a0 2024-01-16T15:19:03.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.275+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f240015f100 con 0x7f23d806c3a0 2024-01-16T15:19:03.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.321+0000 7f23fcc00640 1 -- 172.21.15.53:0/3221910760 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f240015f100 con 0x7f23d806c3a0 2024-01-16T15:19:03.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.323+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f23d806c3a0 msgr2=0x7f23d806e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.323+0000 7f2406600640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f23d806c3a0 0x7f23d806e860 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f23f0009a30 tx=0x7f23f0008040 comp rx=0 tx=0).stop 2024-01-16T15:19:03.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.323+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 msgr2=0x7f24001604f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.323+0000 7f2406600640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 0x7f24001604f0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f23e800c450 tx=0x7f23e800c910 comp rx=0 tx=0).stop 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 shutdown_connections 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f23d806c3a0 0x7f23d806e860 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f240015d4b0 0x7f2400160a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24000cf9a0 0x7f24001604f0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 --2- 172.21.15.53:0/3221910760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24000b41e0 0x7f2400157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 >> 172.21.15.53:0/3221910760 conn(0x7f24000d3000 msgr2=0x7f240006c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 shutdown_connections 2024-01-16T15:19:03.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.324+0000 7f2406600640 1 -- 172.21.15.53:0/3221910760 wait complete. 2024-01-16T15:19:03.319 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.328+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.319 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.328+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.319 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.328+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.319 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.328+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.320 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.328+0000 7f6138c00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66971 2024-01-16T15:19:03.348 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.357+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.348 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.357+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.348 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.357+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.348 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.357+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.348 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:03.357+0000 7f6137800640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:03.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.567+0000 7f9c59a00640 1 Processor -- start 2024-01-16T15:19:03.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.567+0000 7f9c59a00640 1 -- start start 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.568+0000 7f9c59a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c54143c40 0x7f9c54071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c59a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9c540717d0 0x7f9c540cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540cca40 0x7f9c5415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c59a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c540723c0 con 0x7f9c54143c40 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c59a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c54070e50 con 0x7f9c540717d0 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c59a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540cce40 con 0x7f9c540cca40 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c54143c40 0x7f9c54071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9c54143c40 0x7f9c54071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49866/0 (socket says 172.21.15.53:49866) 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 -- 172.21.15.53:0/4051943055 learned_addr learned my addr 172.21.15.53:0/4051943055 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:03.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c59000640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540cca40 0x7f9c5415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 -- 172.21.15.53:0/4051943055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540cca40 msgr2=0x7f9c5415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53400640 1 --2- 172.21.15.53:0/4051943055 >> v2:172.21.15.146:3300/0 conn(0x7f9c540717d0 0x7f9c540cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540cca40 0x7f9c5415eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 -- 172.21.15.53:0/4051943055 >> v2:172.21.15.146:3300/0 conn(0x7f9c540717d0 msgr2=0x7f9c540cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 --2- 172.21.15.53:0/4051943055 >> v2:172.21.15.146:3300/0 conn(0x7f9c540717d0 0x7f9c540cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.569+0000 7f9c53e00640 1 -- 172.21.15.53:0/4051943055 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c54144020 con 0x7f9c54143c40 2024-01-16T15:19:03.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.570+0000 7f9c59000640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540cca40 0x7f9c5415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:03.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.571+0000 7f9c53e00640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c54071290 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f9c540b44a0 tx=0x7f9c4c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfcdeb60f0fc12af server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.571+0000 7f9c52a00640 1 -- 172.21.15.53:0/4051943055 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c02c9e0 con 0x7f9c54143c40 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.571+0000 7f9c52a00640 1 -- 172.21.15.53:0/4051943055 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c4c02cb40 con 0x7f9c54143c40 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.571+0000 7f9c52a00640 1 -- 172.21.15.53:0/4051943055 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c4c0343f0 con 0x7f9c54143c40 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 -- 172.21.15.53:0/4051943055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 msgr2=0x7f9c54071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c54071290 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f9c540b44a0 tx=0x7f9c4c02be40 comp rx=0 tx=0).stop 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 -- 172.21.15.53:0/4051943055 shutdown_connections 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540cca40 0x7f9c5415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 --2- 172.21.15.53:0/4051943055 >> v2:172.21.15.146:3300/0 conn(0x7f9c540717d0 0x7f9c540cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 --2- 172.21.15.53:0/4051943055 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c54071290 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 -- 172.21.15.53:0/4051943055 >> 172.21.15.53:0/4051943055 conn(0x7f9c5406a800 msgr2=0x7f9c5406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 -- 172.21.15.53:0/4051943055 shutdown_connections 2024-01-16T15:19:03.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.572+0000 7f9c59a00640 1 -- 172.21.15.53:0/4051943055 wait complete. 2024-01-16T15:19:03.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.574+0000 7f9c59a00640 1 Processor -- start 2024-01-16T15:19:03.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.574+0000 7f9c59a00640 1 -- start start 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540717d0 0x7f9c5415a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c540cca40 0x7f9c540ab050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c540acc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540c88e0 con 0x7f9c54143c40 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540ad960 con 0x7f9c540cca40 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c540adc40 con 0x7f9c540717d0 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.575+0000 7f9c59000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c540acc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c540acc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49880/0 (socket says 172.21.15.53:49880) 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 -- 172.21.15.53:0/644169909 learned_addr learned my addr 172.21.15.53:0/644169909 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:03.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c53e00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540717d0 0x7f9c5415a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c53400640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c540cca40 0x7f9c540ab050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 -- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540717d0 msgr2=0x7f9c5415a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540717d0 0x7f9c5415a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 -- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c540cca40 msgr2=0x7f9c540ab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c540cca40 0x7f9c540ab050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 -- 172.21.15.53:0/644169909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c4c009b30 con 0x7f9c54143c40 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c53400640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c540cca40 0x7f9c540ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.576+0000 7f9c59000640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c540acc00 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400c7c0 tx=0x7f9c4400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.577+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44004230 con 0x7f9c54143c40 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.577+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c44004390 con 0x7f9c54143c40 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.577+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c44007860 con 0x7f9c54143c40 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.577+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c540adf20 con 0x7f9c54143c40 2024-01-16T15:19:03.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.577+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c5415ae30 con 0x7f9c54143c40 2024-01-16T15:19:03.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.578+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c540c8a50 con 0x7f9c54143c40 2024-01-16T15:19:03.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.579+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f9c440079c0 con 0x7f9c54143c40 2024-01-16T15:19:03.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.580+0000 7f9c4be00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c2c06c3a0 0x7f9c2c06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:03.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.580+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9c44088970 con 0x7f9c54143c40 2024-01-16T15:19:03.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.581+0000 7f9c53e00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c2c06c3a0 0x7f9c2c06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:03.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.582+0000 7f9c53e00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c2c06c3a0 0x7f9c2c06e860 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c035000 tx=0x7f9c4c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:03.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.585+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9c540c8a50 con 0x7f9c54143c40 2024-01-16T15:19:03.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.812+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66971", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c540ae440 con 0x7f9c2c06c3a0 2024-01-16T15:19:03.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.851+0000 7f9c4be00640 1 -- 172.21.15.53:0/644169909 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9c540ae440 con 0x7f9c2c06c3a0 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.854+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c2c06c3a0 msgr2=0x7f9c2c06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.854+0000 7f9c59a00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c2c06c3a0 0x7f9c2c06e860 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f9c4c035000 tx=0x7f9c4c030000 comp rx=0 tx=0).stop 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.854+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 msgr2=0x7f9c540acc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.854+0000 7f9c59a00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c540acc00 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f9c4400c7c0 tx=0x7f9c4400cc80 comp rx=0 tx=0).stop 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 shutdown_connections 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9c2c06c3a0 0x7f9c2c06e860 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9c54143c40 0x7f9c540acc00 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9c540cca40 0x7f9c540ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 --2- 172.21.15.53:0/644169909 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9c540717d0 0x7f9c5415a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:03.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 >> 172.21.15.53:0/644169909 conn(0x7f9c5406a800 msgr2=0x7f9c540aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:03.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 shutdown_connections 2024-01-16T15:19:03.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:03.855+0000 7f9c59a00640 1 -- 172.21.15.53:0/644169909 wait complete. 2024-01-16T15:19:03.864 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66973 2024-01-16T15:19:04.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.081+0000 7f612ca00640 1 Processor -- start 2024-01-16T15:19:04.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.081+0000 7f612ca00640 1 -- start start 2024-01-16T15:19:04.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f612ca00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6128159590 0x7f6128070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f612ca00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f612806d050 0x7f612806d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f612ca00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6128071790 con 0x7f61280b3c20 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f612ca00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f612806d990 con 0x7f6128159590 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f612ca00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6128071100 con 0x7f612806d050 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f6126800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6128159590 0x7f6128070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f6126800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6128159590 0x7f6128070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49896/0 (socket says 172.21.15.53:49896) 2024-01-16T15:19:04.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.083+0000 7f6126800640 1 -- 172.21.15.53:0/3566456478 learned_addr learned my addr 172.21.15.53:0/3566456478 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.084+0000 7f6127200640 1 --2- 172.21.15.53:0/3566456478 >> v2:172.21.15.146:3300/0 conn(0x7f612806d050 0x7f612806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.084+0000 7f6126800640 1 -- 172.21.15.53:0/3566456478 >> v2:172.21.15.146:3300/0 conn(0x7f612806d050 msgr2=0x7f612806d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.084+0000 7f6126800640 1 --2- 172.21.15.53:0/3566456478 >> v2:172.21.15.146:3300/0 conn(0x7f612806d050 0x7f612806d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.084+0000 7f6126800640 1 -- 172.21.15.53:0/3566456478 >> v1:172.21.15.146:6789/0 conn(0x7f61280b3c20 legacy=0x7f61280ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.084+0000 7f6126800640 1 -- 172.21.15.53:0/3566456478 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6128070860 con 0x7f6128159590 2024-01-16T15:19:04.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.085+0000 7f6126800640 1 --2- 172.21.15.53:0/3566456478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6128159590 0x7f6128070160 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f61280b3770 tx=0x7f611802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb88e5d5c332c602 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.085+0000 7f6125e00640 1 -- 172.21.15.53:0/3566456478 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61180317e0 con 0x7f6128159590 2024-01-16T15:19:04.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.085+0000 7f6125e00640 1 -- 172.21.15.53:0/3566456478 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6118031940 con 0x7f6128159590 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.085+0000 7f6125e00640 1 -- 172.21.15.53:0/3566456478 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6118031cd0 con 0x7f6128159590 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 -- 172.21.15.53:0/3566456478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6128159590 msgr2=0x7f6128070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 --2- 172.21.15.53:0/3566456478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6128159590 0x7f6128070160 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f61280b3770 tx=0x7f611802b880 comp rx=0 tx=0).stop 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 -- 172.21.15.53:0/3566456478 shutdown_connections 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 --2- 172.21.15.53:0/3566456478 >> v2:172.21.15.146:3300/0 conn(0x7f612806d050 0x7f612806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 --2- 172.21.15.53:0/3566456478 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6128159590 0x7f6128070160 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 -- 172.21.15.53:0/3566456478 >> 172.21.15.53:0/3566456478 conn(0x7f612806a800 msgr2=0x7f612806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 -- 172.21.15.53:0/3566456478 shutdown_connections 2024-01-16T15:19:04.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.086+0000 7f612ca00640 1 -- 172.21.15.53:0/3566456478 wait complete. 2024-01-16T15:19:04.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.088+0000 7f612ca00640 1 Processor -- start 2024-01-16T15:19:04.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.088+0000 7f612ca00640 1 -- start start 2024-01-16T15:19:04.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f612ca00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f612806d050 0x7f61280adf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f612ca00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 0x7f6128155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f612ca00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6128159590 0x7f61280af9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f612ca00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61280ae470 con 0x7f61280b3c20 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f612ca00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6128155ed0 con 0x7f612806d050 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f612ca00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61281f3090 con 0x7f6128159590 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f611dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 0x7f6128155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f611dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 0x7f6128155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49904/0 (socket says 172.21.15.53:49904) 2024-01-16T15:19:04.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.089+0000 7f611dc00640 1 -- 172.21.15.53:0/589173438 learned_addr learned my addr 172.21.15.53:0/589173438 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:04.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f6126800640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f612806d050 0x7f61280adf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f6127200640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6128159590 0x7f61280af9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611dc00640 1 -- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6128159590 msgr2=0x7f61280af9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611dc00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6128159590 0x7f61280af9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611dc00640 1 -- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f612806d050 msgr2=0x7f61280adf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611dc00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f612806d050 0x7f61280adf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611dc00640 1 -- 172.21.15.53:0/589173438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61180094d0 con 0x7f61280b3c20 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f6126800640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f612806d050 0x7f61280adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611dc00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 0x7f6128155990 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f611000b3e0 tx=0x7f611000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.090+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110011830 con 0x7f61280b3c20 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.091+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61100119b0 con 0x7f61280b3c20 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.091+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110011d60 con 0x7f61280b3c20 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.091+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61281578f0 con 0x7f61280b3c20 2024-01-16T15:19:04.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.091+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6128157050 con 0x7f61280b3c20 2024-01-16T15:19:04.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.092+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6128067450 con 0x7f61280b3c20 2024-01-16T15:19:04.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.093+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f611001a400 con 0x7f61280b3c20 2024-01-16T15:19:04.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.094+0000 7f611f400640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f610006c3d0 0x7f610006e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.094+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f61100888a0 con 0x7f61280b3c20 2024-01-16T15:19:04.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.094+0000 7f6126800640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f610006c3d0 0x7f610006e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.096+0000 7f6126800640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f610006c3d0 0x7f610006e890 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6118036000 tx=0x7f611803d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.099+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6110055160 con 0x7f61280b3c20 2024-01-16T15:19:04.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.310+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66973", "target": ["mon-mgr", ""]}) v1 -- 0x7f61280b1f30 con 0x7f610006c3d0 2024-01-16T15:19:04.342 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.352+0000 7f611f400640 1 -- 172.21.15.53:0/589173438 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6118031d20 con 0x7f610006c3d0 2024-01-16T15:19:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.354+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f610006c3d0 msgr2=0x7f610006e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.354+0000 7f612ca00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f610006c3d0 0x7f610006e890 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f6118036000 tx=0x7f611803d040 comp rx=0 tx=0).stop 2024-01-16T15:19:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.354+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 msgr2=0x7f6128155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.354+0000 7f612ca00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 0x7f6128155990 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f611000b3e0 tx=0x7f611000b8a0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 shutdown_connections 2024-01-16T15:19:04.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f610006c3d0 0x7f610006e890 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6128159590 0x7f61280af9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61280b3c20 0x7f6128155990 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 --2- 172.21.15.53:0/589173438 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f612806d050 0x7f61280adf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 >> 172.21.15.53:0/589173438 conn(0x7f612806a800 msgr2=0x7f612814f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 shutdown_connections 2024-01-16T15:19:04.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.355+0000 7f612ca00640 1 -- 172.21.15.53:0/589173438 wait complete. 2024-01-16T15:19:04.365 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume rm --force cephfs subvol_clone66972 2024-01-16T15:19:04.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.584+0000 7f9d01c00640 1 Processor -- start 2024-01-16T15:19:04.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.584+0000 7f9d01c00640 1 -- start start 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9d01c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9cfc162810 0x7f9cfc071240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9d01c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cfc06c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9d01c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cfc071c10 con 0x7f9cfc162430 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9d01c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cfc0702e0 con 0x7f9cfc162810 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9d01c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0705c0 con 0x7f9cfc071780 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9cfbe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9cfc162430 0x7f9cfc0b3110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50236/0 (socket says 172.21.15.53:50236) 2024-01-16T15:19:04.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9cfbe00640 1 -- 172.21.15.53:0/3495937610 learned_addr learned my addr 172.21.15.53:0/3495937610 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9cfb400640 1 --2- 172.21.15.53:0/3495937610 >> v2:172.21.15.146:3300/0 conn(0x7f9cfc162810 0x7f9cfc071240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9d01200640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cfc06c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487671655 0 0) 0x7f9cfc071c10 con 0x7f9cfc162430 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.586+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cdc003520 con 0x7f9cfc162430 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1624031042 0 0) 0x7f9cdc003520 con 0x7f9cfc162430 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 msgr2=0x7f9cfc06c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cfc06c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 >> v2:172.21.15.146:3300/0 conn(0x7f9cfc162810 msgr2=0x7f9cfc071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 --2- 172.21.15.53:0/3495937610 >> v2:172.21.15.146:3300/0 conn(0x7f9cfc162810 0x7f9cfc071240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc0708a0 con 0x7f9cfc162430 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9cec0019f0 con 0x7f9cfc162430 2024-01-16T15:19:04.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 >> v1:172.21.15.53:6789/0 conn(0x7f9cfc162430 legacy=0x7f9cfc0b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cdc003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cdc003f40 0x7f9cdc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0702e0 con 0x7f9cdc003f40 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc0705c0 con 0x7f9cdc004d00 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc00b980 con 0x7f9cfc071780 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.587+0000 7f9cfb400640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cdc003f40 0x7f9cdc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfbe00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cdc003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9d01200640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfb400640 1 -- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 msgr2=0x7f9cdc003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfb400640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cdc003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfb400640 1 -- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cdc004d00 msgr2=0x7f9cdc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfb400640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfb400640 1 -- 172.21.15.53:0/3495937610 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc003520 con 0x7f9cdc003f40 2024-01-16T15:19:04.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9d01200640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cdc004d00 0x7f9cdc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:04.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.588+0000 7f9cfbe00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cfc071780 0x7f9cdc003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:04.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.589+0000 7f9cfb400640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cdc003f40 0x7f9cdc0047c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009ac0 tx=0x7f9cdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3608de9228afbb8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.589+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce402c6b0 con 0x7f9cdc003f40 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.589+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce402c810 con 0x7f9cdc003f40 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.589+0000 7f9cfaa00640 1 -- 172.21.15.53:0/3495937610 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce402cbc0 con 0x7f9cdc003f40 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01c00640 1 -- 172.21.15.53:0/3495937610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cdc003f40 msgr2=0x7f9cdc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01c00640 1 --2- 172.21.15.53:0/3495937610 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cdc003f40 0x7f9cdc0047c0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009ac0 tx=0x7f9cdc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01200640 1 -- 172.21.15.53:0/3495937610 reap_dead start 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01c00640 1 -- 172.21.15.53:0/3495937610 shutdown_connections 2024-01-16T15:19:04.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01c00640 1 -- 172.21.15.53:0/3495937610 >> 172.21.15.53:0/3495937610 conn(0x7f9cfc06a800 msgr2=0x7f9cfc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:04.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01c00640 1 -- 172.21.15.53:0/3495937610 shutdown_connections 2024-01-16T15:19:04.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.590+0000 7f9d01c00640 1 -- 172.21.15.53:0/3495937610 wait complete. 2024-01-16T15:19:04.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.592+0000 7f9d01c00640 1 Processor -- start 2024-01-16T15:19:04.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.592+0000 7f9d01c00640 1 -- start start 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9d01c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cdc003f40 0x7f9cfc163fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9d01c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 0x7f9cfc15b170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9d01c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc15b6b0 0x7f9cfc157560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9d01c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc162f10 con 0x7f9cfc142e00 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9d01c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc1631f0 con 0x7f9cfc15b6b0 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9d01c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cfc1634d0 con 0x7f9cdc003f40 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9cfb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 0x7f9cfc15b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9cfb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 0x7f9cfc15b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49918/0 (socket says 172.21.15.53:49918) 2024-01-16T15:19:04.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9cfb400640 1 -- 172.21.15.53:0/4055624147 learned_addr learned my addr 172.21.15.53:0/4055624147 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.593+0000 7f9cfbe00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cdc003f40 0x7f9cfc163fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9d01200640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc15b6b0 0x7f9cfc157560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cfb400640 1 -- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cdc003f40 msgr2=0x7f9cfc163fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cfb400640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cdc003f40 0x7f9cfc163fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cfb400640 1 -- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc15b6b0 msgr2=0x7f9cfc157560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cfb400640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc15b6b0 0x7f9cfc157560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cfb400640 1 -- 172.21.15.53:0/4055624147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc1637b0 con 0x7f9cfc142e00 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cfb400640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 0x7f9cfc15b170 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009be0 tx=0x7f9ce4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4034a80 con 0x7f9cfc142e00 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce4034be0 con 0x7f9cfc142e00 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce403e440 con 0x7f9cfc142e00 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.594+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cfc163a90 con 0x7f9cfc142e00 2024-01-16T15:19:04.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.595+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc143260 con 0x7f9cfc142e00 2024-01-16T15:19:04.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.595+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cfc163c20 con 0x7f9cfc142e00 2024-01-16T15:19:04.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.597+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f9ce403e5a0 con 0x7f9cfc142e00 2024-01-16T15:19:04.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.597+0000 7f9cf3e00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9cd406c3d0 0x7f9cd406e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:04.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.598+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f9ce40aa220 con 0x7f9cfc142e00 2024-01-16T15:19:04.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.598+0000 7f9cfbe00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9cd406c3d0 0x7f9cd406e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:04.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.600+0000 7f9cfbe00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9cd406c3d0 0x7f9cd406e890 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0013d0 tx=0x7f9cec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:04.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.602+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9ce4076ae0 con 0x7f9cfc142e00 2024-01-16T15:19:04.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.828+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume rm", "force": true, "vol_name": "cephfs", "sub_name": "subvol_clone66972", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cfc0b3820 con 0x7f9cd406c3d0 2024-01-16T15:19:04.857 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.867+0000 7f9cf3e00640 1 -- 172.21.15.53:0/4055624147 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f9cec00d070 con 0x7f9cd406c3d0 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9cd406c3d0 msgr2=0x7f9cd406e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9cd406c3d0 0x7f9cd406e890 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f9cec0013d0 tx=0x7f9cec009040 comp rx=0 tx=0).stop 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 msgr2=0x7f9cfc15b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 0x7f9cfc15b170 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4009be0 tx=0x7f9ce4004730 comp rx=0 tx=0).stop 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 shutdown_connections 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9cd406c3d0 0x7f9cd406e890 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9cfc15b6b0 0x7f9cfc157560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9cfc142e00 0x7f9cfc15b170 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 --2- 172.21.15.53:0/4055624147 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9cdc003f40 0x7f9cfc163fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 >> 172.21.15.53:0/4055624147 conn(0x7f9cfc06a800 msgr2=0x7f9cfc0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.870+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 shutdown_connections 2024-01-16T15:19:04.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:04.871+0000 7f9d01c00640 1 -- 172.21.15.53:0/4055624147 wait complete. 2024-01-16T15:19:04.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:04.881 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'stat -c %h /home/ubuntu/cephtest/mnt.0/./volumes/_deleting') 2024-01-16T15:19:04.947 INFO:teuthology.orchestra.run.smithi053.stdout:2 2024-01-16T15:19:04.949 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume rm cephfs --yes-i-really-mean-it 2024-01-16T15:19:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.173+0000 7f66d3e00640 1 Processor -- start 2024-01-16T15:19:05.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.174+0000 7f66d3e00640 1 -- start start 2024-01-16T15:19:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.175+0000 7f66d3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66d41505b0 0x7f66d40b3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.175+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d415a690 0x7f66d406cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66d406d930 con 0x7f66d406d500 2024-01-16T15:19:05.166 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66d40b1e30 con 0x7f66d41505b0 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d40cdc10 con 0x7f66d415a690 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66d41505b0 0x7f66d40b3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f66d41505b0 0x7f66d40b3b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49928/0 (socket says 172.21.15.53:49928) 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 -- 172.21.15.53:0/350057499 learned_addr learned my addr 172.21.15.53:0/350057499 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 -- 172.21.15.53:0/350057499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d415a690 msgr2=0x7f66d406cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2000640 1 --2- 172.21.15.53:0/350057499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d415a690 0x7f66d406cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 --2- 172.21.15.53:0/350057499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d415a690 0x7f66d406cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 -- 172.21.15.53:0/350057499 >> v1:172.21.15.146:6789/0 conn(0x7f66d406d500 legacy=0x7f66d415d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.176+0000 7f66d2a00640 1 -- 172.21.15.53:0/350057499 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66d40b20a0 con 0x7f66d41505b0 2024-01-16T15:19:05.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.178+0000 7f66d2a00640 1 --2- 172.21.15.53:0/350057499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d41505b0 0x7f66d40b3b20 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f66d406e370 tx=0x7f66c402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629d7d4818a3bc36 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:05.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.178+0000 7f66d1600640 1 -- 172.21.15.53:0/350057499 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4031640 con 0x7f66d41505b0 2024-01-16T15:19:05.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.178+0000 7f66d1600640 1 -- 172.21.15.53:0/350057499 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66c40317a0 con 0x7f66d41505b0 2024-01-16T15:19:05.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.178+0000 7f66d1600640 1 -- 172.21.15.53:0/350057499 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66c4031b30 con 0x7f66d41505b0 2024-01-16T15:19:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 -- 172.21.15.53:0/350057499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d41505b0 msgr2=0x7f66d40b3b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 --2- 172.21.15.53:0/350057499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d41505b0 0x7f66d40b3b20 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f66d406e370 tx=0x7f66c402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 -- 172.21.15.53:0/350057499 shutdown_connections 2024-01-16T15:19:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 --2- 172.21.15.53:0/350057499 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d415a690 0x7f66d406cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 --2- 172.21.15.53:0/350057499 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d41505b0 0x7f66d40b3b20 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 -- 172.21.15.53:0/350057499 >> 172.21.15.53:0/350057499 conn(0x7f66d406a7d0 msgr2=0x7f66d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:05.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 -- 172.21.15.53:0/350057499 shutdown_connections 2024-01-16T15:19:05.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.179+0000 7f66d3e00640 1 -- 172.21.15.53:0/350057499 wait complete. 2024-01-16T15:19:05.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.181+0000 7f66d3e00640 1 Processor -- start 2024-01-16T15:19:05.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.181+0000 7f66d3e00640 1 -- start start 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d406d500 0x7f66d414a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d41505b0 0x7f66d40ae820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 0x7f66d413c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d414c800 con 0x7f66d415a690 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d414cae0 con 0x7f66d41505b0 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66d414cdc0 con 0x7f66d406d500 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 0x7f66d413c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 0x7f66d413c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49934/0 (socket says 172.21.15.53:49934) 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.182+0000 7f66d3400640 1 -- 172.21.15.53:0/1064751039 learned_addr learned my addr 172.21.15.53:0/1064751039 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:05.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d2a00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d406d500 0x7f66d414a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d2000640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d41505b0 0x7f66d40ae820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d3400640 1 -- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d406d500 msgr2=0x7f66d414a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d3400640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d406d500 0x7f66d414a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d3400640 1 -- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d41505b0 msgr2=0x7f66d40ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d3400640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d41505b0 0x7f66d40ae820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d3400640 1 -- 172.21.15.53:0/1064751039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c40094d0 con 0x7f66d415a690 2024-01-16T15:19:05.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d2000640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d41505b0 0x7f66d40ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.183+0000 7f66d3400640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 0x7f66d413c170 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00b490 tx=0x7f66cc00b950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.184+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc011830 con 0x7f66d415a690 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.184+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66cc0119b0 con 0x7f66d415a690 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.184+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66cc011d60 con 0x7f66d415a690 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.184+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d414d0a0 con 0x7f66d415a690 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.184+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d415cda0 con 0x7f66d415a690 2024-01-16T15:19:05.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.185+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d414d5c0 con 0x7f66d415a690 2024-01-16T15:19:05.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.186+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 40) v1 ==== 91069+0+0 (secure 0 0 0) 0x7f66cc01a400 con 0x7f66d415a690 2024-01-16T15:19:05.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.187+0000 7f66cb400640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66ac06c3a0 0x7f66ac06e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.188+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f66cc088850 con 0x7f66d415a690 2024-01-16T15:19:05.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.188+0000 7f66d2a00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66ac06c3a0 0x7f66ac06e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:05.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.189+0000 7f66d2a00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66ac06c3a0 0x7f66ac06e860 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f66c4036000 tx=0x7f66c403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:05.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.192+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f66d414d5c0 con 0x7f66d415a690 2024-01-16T15:19:05.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.412+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume rm", "vol_name": "cephfs", "yes-i-really-mean-it": "--yes-i-really-mean-it", "target": ["mon-mgr", ""]}) v1 -- 0x7f66d414cc50 con 0x7f66ac06c3a0 2024-01-16T15:19:05.421 INFO:tasks.ceph.mon.a.smithi053.stderr:2024-01-16T15:19:05.431+0000 7f60dc800640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-01-16T15:19:05.525 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:19:05.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:05.786+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 41) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f66cc0554e0 con 0x7f66d415a690 2024-01-16T15:19:07.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:07.794+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f66cc0172c0 con 0x7f66d415a690 2024-01-16T15:19:08.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.477+0000 7f66cb400640 1 -- 172.21.15.53:0/1064751039 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7f66d414cc50 con 0x7f66ac06c3a0 2024-01-16T15:19:08.468 INFO:teuthology.orchestra.run.smithi053.stdout:metadata pool: cephfs_metadata data pool: ['cephfs_data'] removed 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66ac06c3a0 msgr2=0x7f66ac06e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66ac06c3a0 0x7f66ac06e860 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f66c4036000 tx=0x7f66c403c040 comp rx=0 tx=0).stop 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 msgr2=0x7f66d413c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 0x7f66d413c170 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00b490 tx=0x7f66cc00b950 comp rx=0 tx=0).stop 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 shutdown_connections 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f66ac06c3a0 0x7f66ac06e860 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66d415a690 0x7f66d413c170 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f66d41505b0 0x7f66d40ae820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 --2- 172.21.15.53:0/1064751039 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f66d406d500 0x7f66d414a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.481+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 >> 172.21.15.53:0/1064751039 conn(0x7f66d406a7d0 msgr2=0x7f66d414f810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.482+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 shutdown_connections 2024-01-16T15:19:08.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:08.482+0000 7f66d3e00640 1 -- 172.21.15.53:0/1064751039 wait complete. 2024-01-16T15:19:08.495 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:19:08.496 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:19:08.613 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:19:08.700 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:19:08.700 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:19:08.700 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:08.762 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:09.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.027+0000 7f4b54000640 1 Processor -- start 2024-01-16T15:19:09.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.027+0000 7f4b54000640 1 -- start start 2024-01-16T15:19:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b54000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 0x7f4b4c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b54000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4b4c15e1e0 0x7f4b4c0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b54000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4c0b1ee0 con 0x7f4b4c143ae0 2024-01-16T15:19:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b54000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b4c0b2780 con 0x7f4b4c15e1e0 2024-01-16T15:19:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b54000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c15e990 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b52000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4b4c15e1e0 0x7f4b4c0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.029+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 0x7f4b4c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b50c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 0x7f4b4c15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:45254/0 (socket says 172.21.15.53:45254) 2024-01-16T15:19:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b50c00640 1 -- 172.21.15.53:0/489261390 learned_addr learned my addr 172.21.15.53:0/489261390 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b52000640 1 -- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 msgr2=0x7f4b4c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b52000640 1 --2- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 0x7f4b4c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b52000640 1 -- 172.21.15.53:0/489261390 >> v1:172.21.15.146:6789/0 conn(0x7f4b4c143ae0 legacy=0x7f4b4c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b52000640 1 -- 172.21.15.53:0/489261390 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4c0b2130 con 0x7f4b4c15e1e0 2024-01-16T15:19:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.030+0000 7f4b50c00640 1 --2- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 0x7f4b4c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:19:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.032+0000 7f4b52000640 1 --2- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c15e1e0 0x7f4b4c0702c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009d90 tx=0x7f4b4402e0c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f60473cb64aeb99 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.032+0000 7f4b4be00640 1 -- 172.21.15.53:0/489261390 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b4402ee70 con 0x7f4b4c15e1e0 2024-01-16T15:19:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.032+0000 7f4b4be00640 1 -- 172.21.15.53:0/489261390 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b44036440 con 0x7f4b4c15e1e0 2024-01-16T15:19:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.032+0000 7f4b4be00640 1 -- 172.21.15.53:0/489261390 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b440367d0 con 0x7f4b4c15e1e0 2024-01-16T15:19:09.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 -- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c15e1e0 msgr2=0x7f4b4c0702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 --2- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c15e1e0 0x7f4b4c0702c0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f4b44009d90 tx=0x7f4b4402e0c0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 -- 172.21.15.53:0/489261390 shutdown_connections 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 --2- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c15e1e0 0x7f4b4c0702c0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 --2- 172.21.15.53:0/489261390 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c0cf9a0 0x7f4b4c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 -- 172.21.15.53:0/489261390 >> 172.21.15.53:0/489261390 conn(0x7f4b4c06a800 msgr2=0x7f4b4c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 -- 172.21.15.53:0/489261390 shutdown_connections 2024-01-16T15:19:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.033+0000 7f4b54000640 1 -- 172.21.15.53:0/489261390 wait complete. 2024-01-16T15:19:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.035+0000 7f4b54000640 1 Processor -- start 2024-01-16T15:19:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.035+0000 7f4b54000640 1 -- start start 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b54000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 0x7f4b4c0aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b54000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c143ae0 0x7f4b4c159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b54000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c15e1e0 0x7f4b4c0c4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b54000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c159eb0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b54000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0c4640 con 0x7f4b4c143ae0 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b54000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4c0af070 con 0x7f4b4c15e1e0 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b51600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 0x7f4b4c0aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b51600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 0x7f4b4c0aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49952/0 (socket says 172.21.15.53:49952) 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b51600640 1 -- 172.21.15.53:0/3687936302 learned_addr learned my addr 172.21.15.53:0/3687936302 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b50c00640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c143ae0 0x7f4b4c159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.036+0000 7f4b52000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c15e1e0 0x7f4b4c0c4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b51600640 1 -- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c15e1e0 msgr2=0x7f4b4c0c4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b51600640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c15e1e0 0x7f4b4c0c4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b51600640 1 -- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c143ae0 msgr2=0x7f4b4c159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b51600640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c143ae0 0x7f4b4c159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b51600640 1 -- 172.21.15.53:0/3687936302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b340094d0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b52000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c15e1e0 0x7f4b4c0c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b51600640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 0x7f4b4c0aea20 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c400 tx=0x7f4b3c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c0052d0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c005450 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c007260 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b44009a70 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.037+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4c0af630 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.039+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4c0675d0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.039+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4b3c0073c0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.040+0000 7f4b49600640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b2406c460 0x7f4b2406e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.040+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f4b3c087ec0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.041+0000 7f4b50c00640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b2406c460 0x7f4b2406e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.042+0000 7f4b50c00640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b2406c460 0x7f4b2406e920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c15a9a0 tx=0x7f4b34008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.045+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b3c054490 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.324+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b4c15a6f0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.325+0000 7f4b49600640 1 -- 172.21.15.53:0/3687936302 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+12995 (secure 0 0 0) 0x7f4b3c057ab0 con 0x7f4b4c0cf9a0 2024-01-16T15:19:09.315 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:09.315 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":102,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:08.449618+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/1250261984":"2024-01-17T15:19:05.431906+0000","172.21.15.53:6832/1250261984":"2024-01-17T15:19:05.431906+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:09.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b2406c460 msgr2=0x7f4b2406e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b2406c460 0x7f4b2406e920 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c15a9a0 tx=0x7f4b34008040 comp rx=0 tx=0).stop 2024-01-16T15:19:09.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 msgr2=0x7f4b4c0aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 0x7f4b4c0aea20 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00c400 tx=0x7f4b3c00c8c0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 shutdown_connections 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b2406c460 0x7f4b2406e920 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4b4c15e1e0 0x7f4b4c0c4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4b4c143ae0 0x7f4b4c159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 --2- 172.21.15.53:0/3687936302 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4b4c0cf9a0 0x7f4b4c0aea20 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.327+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 >> 172.21.15.53:0/3687936302 conn(0x7f4b4c06a800 msgr2=0x7f4b4c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.328+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 shutdown_connections 2024-01-16T15:19:09.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.328+0000 7f4b54000640 1 -- 172.21.15.53:0/3687936302 wait complete. 2024-01-16T15:19:09.336 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:19:09.336 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:19:09.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.537+0000 7fc443200640 1 Processor -- start 2024-01-16T15:19:09.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.537+0000 7fc443200640 1 -- start start 2024-01-16T15:19:09.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.538+0000 7fc443200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc43c13e360 0x7fc43c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.538+0000 7fc443200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc43c15f9c0 0x7fc43c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.538+0000 7fc443200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc43c15c5f0 con 0x7fc43c13df80 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.538+0000 7fc443200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c06c470 con 0x7fc43c15f9c0 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.538+0000 7fc443200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c06c750 con 0x7fc43c13e360 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc43be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc43c13df80 0x7fc43c15bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50248/0 (socket says 172.21.15.53:50248) 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc43be00640 1 -- 172.21.15.53:0/2058493917 learned_addr learned my addr 172.21.15.53:0/2058493917 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc441200640 1 --2- 172.21.15.53:0/2058493917 >> v2:172.21.15.53:3300/0 conn(0x7fc43c15f9c0 0x7fc43c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc441200640 1 -- 172.21.15.53:0/2058493917 >> v2:172.21.15.146:3300/0 conn(0x7fc43c13e360 msgr2=0x7fc43c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.530 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc43b400640 1 -- 172.21.15.53:0/2058493917 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58162676 0 0) 0x7fc43c15c5f0 con 0x7fc43c13df80 2024-01-16T15:19:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc441200640 1 --2- 172.21.15.53:0/2058493917 >> v2:172.21.15.146:3300/0 conn(0x7fc43c13e360 0x7fc43c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc441200640 1 -- 172.21.15.53:0/2058493917 >> v1:172.21.15.53:6789/0 conn(0x7fc43c13df80 legacy=0x7fc43c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.531 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.539+0000 7fc441200640 1 -- 172.21.15.53:0/2058493917 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c06ca30 con 0x7fc43c15f9c0 2024-01-16T15:19:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.541+0000 7fc441200640 1 --2- 172.21.15.53:0/2058493917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c15f9c0 0x7fc43c0cf7d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fc43c06ded0 tx=0x7fc43402bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527391d9f90c1230 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.541+0000 7fc43b400640 1 -- 172.21.15.53:0/2058493917 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc43402cc70 con 0x7fc43c15f9c0 2024-01-16T15:19:09.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.541+0000 7fc43b400640 1 -- 172.21.15.53:0/2058493917 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc43402cdd0 con 0x7fc43c15f9c0 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.541+0000 7fc43b400640 1 -- 172.21.15.53:0/2058493917 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc434034660 con 0x7fc43c15f9c0 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 -- 172.21.15.53:0/2058493917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c15f9c0 msgr2=0x7fc43c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 --2- 172.21.15.53:0/2058493917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c15f9c0 0x7fc43c0cf7d0 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fc43c06ded0 tx=0x7fc43402bec0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 -- 172.21.15.53:0/2058493917 shutdown_connections 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 --2- 172.21.15.53:0/2058493917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c15f9c0 0x7fc43c0cf7d0 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 --2- 172.21.15.53:0/2058493917 >> v2:172.21.15.146:3300/0 conn(0x7fc43c13e360 0x7fc43c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 -- 172.21.15.53:0/2058493917 >> 172.21.15.53:0/2058493917 conn(0x7fc43c06a800 msgr2=0x7fc43c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 -- 172.21.15.53:0/2058493917 shutdown_connections 2024-01-16T15:19:09.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.542+0000 7fc443200640 1 -- 172.21.15.53:0/2058493917 wait complete. 2024-01-16T15:19:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.544+0000 7fc443200640 1 Processor -- start 2024-01-16T15:19:09.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.544+0000 7fc443200640 1 -- start start 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc443200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc43c13df80 0x7fc43c14d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc443200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 0x7fc43c133830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc443200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc43c15f9c0 0x7fc43c14f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc443200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c150b80 con 0x7fc43c13e360 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc443200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c14aaf0 con 0x7fc43c13df80 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc443200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c14ff70 con 0x7fc43c15f9c0 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 0x7fc43c133830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 0x7fc43c133830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49968/0 (socket says 172.21.15.53:49968) 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 -- 172.21.15.53:0/2414813201 learned_addr learned my addr 172.21.15.53:0/2414813201 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:09.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 -- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc43c15f9c0 msgr2=0x7fc43c14f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc43c15f9c0 0x7fc43c14f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 -- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc43c13df80 msgr2=0x7fc43c14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc43c13df80 0x7fc43c14d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.545+0000 7fc433200640 1 -- 172.21.15.53:0/2414813201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc434009a70 con 0x7fc43c13e360 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.546+0000 7fc433200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 0x7fc43c133830 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc42400b450 tx=0x7fc42400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.546+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424011890 con 0x7fc43c13e360 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.546+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4240119f0 con 0x7fc43c13e360 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.546+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc424011da0 con 0x7fc43c13e360 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.546+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c150250 con 0x7fc43c13e360 2024-01-16T15:19:09.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.546+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c150810 con 0x7fc43c13e360 2024-01-16T15:19:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.547+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c0675d0 con 0x7fc43c13e360 2024-01-16T15:19:09.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.548+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc42401a400 con 0x7fc43c13e360 2024-01-16T15:19:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.549+0000 7fc438c00640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc41406c430 0x7fc41406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.549+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc424088eb0 con 0x7fc43c13e360 2024-01-16T15:19:09.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.550+0000 7fc43be00640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc41406c430 0x7fc41406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.551+0000 7fc43be00640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc41406c430 0x7fc41406e8f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc42c005880 tx=0x7fc42c007a00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.554+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc424055420 con 0x7fc43c13e360 2024-01-16T15:19:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.747+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc43c06df00 con 0x7fc43c13e360 2024-01-16T15:19:09.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.748+0000 7fc438c00640 1 -- 172.21.15.53:0/2414813201 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc424015020 con 0x7fc43c13e360 2024-01-16T15:19:09.739 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc41406c430 msgr2=0x7fc41406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc41406c430 0x7fc41406e8f0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fc42c005880 tx=0x7fc42c007a00 comp rx=0 tx=0).stop 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 msgr2=0x7fc43c133830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 0x7fc43c133830 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7fc42400b450 tx=0x7fc42400b910 comp rx=0 tx=0).stop 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 shutdown_connections 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc41406c430 0x7fc41406e8f0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc43c15f9c0 0x7fc43c14f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.742 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc43c13e360 0x7fc43c133830 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 --2- 172.21.15.53:0/2414813201 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc43c13df80 0x7fc43c14d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 >> 172.21.15.53:0/2414813201 conn(0x7fc43c06a800 msgr2=0x7fc43c15dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 shutdown_connections 2024-01-16T15:19:09.743 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.751+0000 7fc443200640 1 -- 172.21.15.53:0/2414813201 wait complete. 2024-01-16T15:19:09.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:19:09.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.995+0000 7f16ad200640 1 Processor -- start 2024-01-16T15:19:09.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.996+0000 7f16ad200640 1 -- start start 2024-01-16T15:19:09.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16ad200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16a80cd240 0x7f16a80b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16ad200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a809a170 0x7f16a80cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16ad200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a806cea0 con 0x7f16a806c6b0 2024-01-16T15:19:09.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16ad200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a806cbc0 con 0x7f16a80cd240 2024-01-16T15:19:09.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16ad200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a806f350 con 0x7f16a809a170 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16a5e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16a80cd240 0x7f16a80b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16a5e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16a80cd240 0x7f16a80b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49984/0 (socket says 172.21.15.53:49984) 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16a5e00640 1 -- 172.21.15.53:0/3682301177 learned_addr learned my addr 172.21.15.53:0/3682301177 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.998+0000 7f16a6800640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a809a170 0x7f16a80cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.999+0000 7f16a5e00640 1 -- 172.21.15.53:0/3682301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a809a170 msgr2=0x7f16a80cf640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.999+0000 7f16a5e00640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a809a170 0x7f16a80cf640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.999+0000 7f16a5e00640 1 -- 172.21.15.53:0/3682301177 >> v1:172.21.15.146:6789/0 conn(0x7f16a806c6b0 legacy=0x7f16a8099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.999+0000 7f16a5e00640 1 -- 172.21.15.53:0/3682301177 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a806f630 con 0x7f16a80cd240 2024-01-16T15:19:09.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:09.999+0000 7f16a6800640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a809a170 0x7f16a80cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:09.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.000+0000 7f16a5e00640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80b3820 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f1698009870 tx=0x7f169802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b084dd5203abb1f5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.000+0000 7f16a4a00640 1 -- 172.21.15.53:0/3682301177 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698031640 con 0x7f16a80cd240 2024-01-16T15:19:09.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.000+0000 7f16a4a00640 1 -- 172.21.15.53:0/3682301177 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16980317a0 con 0x7f16a80cd240 2024-01-16T15:19:09.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.000+0000 7f16a4a00640 1 -- 172.21.15.53:0/3682301177 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1698031b30 con 0x7f16a80cd240 2024-01-16T15:19:09.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 -- 172.21.15.53:0/3682301177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 msgr2=0x7f16a80b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80b3820 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f1698009870 tx=0x7f169802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 -- 172.21.15.53:0/3682301177 shutdown_connections 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a809a170 0x7f16a80cf640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 --2- 172.21.15.53:0/3682301177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80b3820 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 -- 172.21.15.53:0/3682301177 >> 172.21.15.53:0/3682301177 conn(0x7f16a80b2d00 msgr2=0x7f16a80b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 -- 172.21.15.53:0/3682301177 shutdown_connections 2024-01-16T15:19:09.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.001+0000 7f16ad200640 1 -- 172.21.15.53:0/3682301177 wait complete. 2024-01-16T15:19:09.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.003+0000 7f16ad200640 1 Processor -- start 2024-01-16T15:19:09.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.003+0000 7f16ad200640 1 -- start start 2024-01-16T15:19:09.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.004+0000 7f16ad200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a806c6b0 0x7f16a815a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.004+0000 7f16ad200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a809a170 0x7f16a81f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.004+0000 7f16ad200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.004+0000 7f16ad200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c5a20 con 0x7f16a80cd240 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.004+0000 7f16ad200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a81f7280 con 0x7f16a809a170 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.004+0000 7f16ad200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a80c4da0 con 0x7f16a806c6b0 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80c89e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49990/0 (socket says 172.21.15.53:49990) 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 -- 172.21.15.53:0/4245570493 learned_addr learned my addr 172.21.15.53:0/4245570493 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a5e00640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a806c6b0 0x7f16a815a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a5400640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a809a170 0x7f16a81f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 -- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a806c6b0 msgr2=0x7f16a815a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a806c6b0 0x7f16a815a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 -- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a809a170 msgr2=0x7f16a81f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a809a170 0x7f16a81f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 -- 172.21.15.53:0/4245570493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980094d0 con 0x7f16a80cd240 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.005+0000 7f16a6800640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80c89e0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c7f0 tx=0x7f16a000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.006+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0007820 con 0x7f16a80cd240 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.006+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a0007980 con 0x7f16a80cd240 2024-01-16T15:19:09.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.006+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16a0007d70 con 0x7f16a80cd240 2024-01-16T15:19:09.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.006+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a80c5080 con 0x7f16a80cd240 2024-01-16T15:19:09.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.006+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a80c5640 con 0x7f16a80cd240 2024-01-16T15:19:10.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.007+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a80c5b60 con 0x7f16a80cd240 2024-01-16T15:19:10.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.008+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f16a00113f0 con 0x7f16a80cd240 2024-01-16T15:19:10.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.009+0000 7f169e000640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168006c430 0x7f168006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.009+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f16a0088e20 con 0x7f16a80cd240 2024-01-16T15:19:10.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.010+0000 7f16a5e00640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168006c430 0x7f168006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.011+0000 7f16a5e00640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168006c430 0x7f168006e8f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1698036000 tx=0x7f169803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.013+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f16a80c5b60 con 0x7f16a80cd240 2024-01-16T15:19:10.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.222+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f16a80c5b60 con 0x7f16a80cd240 2024-01-16T15:19:10.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.223+0000 7f169e000640 1 -- 172.21.15.53:0/4245570493 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v102) v1 ==== 80+0+122 (secure 0 0 0) 0x7f16a80c5b60 con 0x7f16a80cd240 2024-01-16T15:19:10.214 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6834/1250261984 2024-01-17T15:19:05.431906+0000 2024-01-16T15:19:10.214 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6832/1250261984 2024-01-17T15:19:05.431906+0000 2024-01-16T15:19:10.214 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:19:10.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.225+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168006c430 msgr2=0x7f168006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.225+0000 7f16ad200640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168006c430 0x7f168006e8f0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f1698036000 tx=0x7f169803c040 comp rx=0 tx=0).stop 2024-01-16T15:19:10.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.225+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 msgr2=0x7f16a80c89e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.225+0000 7f16ad200640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80c89e0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7f16a000c7f0 tx=0x7f16a000ccb0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 shutdown_connections 2024-01-16T15:19:10.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168006c430 0x7f168006e8f0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16a80cd240 0x7f16a80c89e0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16a809a170 0x7f16a81f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 --2- 172.21.15.53:0/4245570493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16a806c6b0 0x7f16a815a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 >> 172.21.15.53:0/4245570493 conn(0x7f16a80b2d00 msgr2=0x7f16a8099a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 shutdown_connections 2024-01-16T15:19:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.226+0000 7f16ad200640 1 -- 172.21.15.53:0/4245570493 wait complete. 2024-01-16T15:19:10.237 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:19:10.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:10.237 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:10.342 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:19:10.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:10.342 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:19:10.343 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[72083]: fuse finished with error 0 and tester_r 0 2024-01-16T15:19:10.592 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:19:10.592 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:10.651 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:10.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.890+0000 7ff990600640 1 Processor -- start 2024-01-16T15:19:10.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.890+0000 7ff990600640 1 -- start start 2024-01-16T15:19:10.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff990600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff98815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff990600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff9880cf9a0 0x7ff9880cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff990600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff98806c470 con 0x7ff98813e360 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff990600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff98806c750 con 0x7ff9880cf9a0 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff990600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806ca30 con 0x7ff98813df80 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff98d200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff98813e360 0x7ff9880cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50254/0 (socket says 172.21.15.53:50254) 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.892+0000 7ff98d200640 1 -- 172.21.15.53:0/1039475074 learned_addr learned my addr 172.21.15.53:0/1039475074 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff98e600640 1 --2- 172.21.15.53:0/1039475074 >> v2:172.21.15.146:3300/0 conn(0x7ff9880cf9a0 0x7ff9880cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff98dc00640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff98815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054370670 0 0) 0x7ff98806c470 con 0x7ff98813e360 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff968003520 con 0x7ff98813e360 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3454317871 0 0) 0x7ff968003520 con 0x7ff98813e360 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 msgr2=0x7ff98815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff98815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 >> v2:172.21.15.146:3300/0 conn(0x7ff9880cf9a0 msgr2=0x7ff9880cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 --2- 172.21.15.53:0/1039475074 >> v2:172.21.15.146:3300/0 conn(0x7ff9880cf9a0 0x7ff9880cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98806cd10 con 0x7ff98813e360 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7ff970001670 con 0x7ff98813e360 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 >> v1:172.21.15.53:6789/0 conn(0x7ff98813e360 legacy=0x7ff9880cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.893+0000 7ff987e00640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff968003970 0x7ff968003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff987e00640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9680047c0 0x7ff968004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff987e00640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff968005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806c750 con 0x7ff9680047c0 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98806ca30 con 0x7ff968003970 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff96800b980 con 0x7ff98813df80 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98d200640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9680047c0 0x7ff968004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98dc00640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff968003970 0x7ff968003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98e600640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff968005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98d200640 1 -- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 msgr2=0x7ff968005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98d200640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff968005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98d200640 1 -- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff968003970 msgr2=0x7ff968003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:10.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98d200640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff968003970 0x7ff968003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98d200640 1 -- 172.21.15.53:0/1039475074 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff968003520 con 0x7ff9680047c0 2024-01-16T15:19:10.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.894+0000 7ff98e600640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff98813df80 0x7ff968005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff98d200640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9680047c0 0x7ff968004c40 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff97000c290 tx=0x7ff9680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4d87f8ecfde7727 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff970005ad0 con 0x7ff9680047c0 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff970005c30 con 0x7ff9680047c0 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff987e00640 1 -- 172.21.15.53:0/1039475074 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97002c8d0 con 0x7ff9680047c0 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff990600640 1 -- 172.21.15.53:0/1039475074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9680047c0 msgr2=0x7ff968004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff990600640 1 --2- 172.21.15.53:0/1039475074 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff9680047c0 0x7ff968004c40 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7ff97000c290 tx=0x7ff9680057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff98e600640 1 -- 172.21.15.53:0/1039475074 reap_dead start 2024-01-16T15:19:10.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff990600640 1 -- 172.21.15.53:0/1039475074 shutdown_connections 2024-01-16T15:19:10.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.896+0000 7ff990600640 1 -- 172.21.15.53:0/1039475074 >> 172.21.15.53:0/1039475074 conn(0x7ff98806a800 msgr2=0x7ff98806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:10.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.897+0000 7ff990600640 1 -- 172.21.15.53:0/1039475074 shutdown_connections 2024-01-16T15:19:10.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.897+0000 7ff990600640 1 -- 172.21.15.53:0/1039475074 wait complete. 2024-01-16T15:19:10.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.898+0000 7ff990600640 1 Processor -- start 2024-01-16T15:19:10.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.898+0000 7ff990600640 1 -- start start 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.899+0000 7ff990600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9680047c0 0x7ff9881f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.899+0000 7ff990600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff988154080 0x7ff988158360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.899+0000 7ff990600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 0x7ff98813d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.899+0000 7ff990600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98813de20 con 0x7ff98813d430 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.899+0000 7ff990600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881f4ea0 con 0x7ff9680047c0 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.899+0000 7ff990600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff988158fe0 con 0x7ff988154080 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 0x7ff98813d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 0x7ff98813d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:49996/0 (socket says 172.21.15.53:49996) 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 -- 172.21.15.53:0/1390329853 learned_addr learned my addr 172.21.15.53:0/1390329853 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98d200640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff988154080 0x7ff988158360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98dc00640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9680047c0 0x7ff9881f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 -- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff988154080 msgr2=0x7ff988158360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff988154080 0x7ff988158360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 -- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9680047c0 msgr2=0x7ff9881f4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:10.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9680047c0 0x7ff9881f4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:10.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 -- 172.21.15.53:0/1390329853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff988151ae0 con 0x7ff98813d430 2024-01-16T15:19:10.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.900+0000 7ff98e600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 0x7ff98813d8e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7ff9880cc770 tx=0x7ff98000f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.901+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980017400 con 0x7ff98813d430 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.901+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff980011040 con 0x7ff98813d430 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.901+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff980017780 con 0x7ff98813d430 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.901+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff988154600 con 0x7ff98813d430 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.901+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff988154bc0 con 0x7ff98813d430 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.902+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9880675d0 con 0x7ff98813d430 2024-01-16T15:19:10.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.903+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7ff9800203b0 con 0x7ff98813d430 2024-01-16T15:19:10.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.904+0000 7ff985600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96006c430 0x7ff96006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:10.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.905+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7ff980088db0 con 0x7ff98813d430 2024-01-16T15:19:10.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.905+0000 7ff98dc00640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96006c430 0x7ff96006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:10.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.906+0000 7ff98dc00640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96006c430 0x7ff96006e8f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff9880cc0d0 tx=0x7ff978008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:10.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:10.909+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff980055320 con 0x7ff98813d430 2024-01-16T15:19:11.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.104+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff988151c70 con 0x7ff98813d430 2024-01-16T15:19:11.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.106+0000 7ff985600640 1 -- 172.21.15.53:0/1390329853 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+12995 (secure 0 0 0) 0x7ff98001c090 con 0x7ff98813d430 2024-01-16T15:19:11.097 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:11.097 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":102,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:08.449618+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/1250261984":"2024-01-17T15:19:05.431906+0000","172.21.15.53:6832/1250261984":"2024-01-17T15:19:05.431906+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:11.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96006c430 msgr2=0x7ff96006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96006c430 0x7ff96006e8f0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7ff9880cc0d0 tx=0x7ff978008040 comp rx=0 tx=0).stop 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 msgr2=0x7ff98813d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 0x7ff98813d8e0 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7ff9880cc770 tx=0x7ff98000f4a0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 shutdown_connections 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff96006c430 0x7ff96006e8f0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff98813d430 0x7ff98813d8e0 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff988154080 0x7ff988158360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 --2- 172.21.15.53:0/1390329853 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff9680047c0 0x7ff9881f4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 >> 172.21.15.53:0/1390329853 conn(0x7ff98806a800 msgr2=0x7ff98806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 shutdown_connections 2024-01-16T15:19:11.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.109+0000 7ff990600640 1 -- 172.21.15.53:0/1390329853 wait complete. 2024-01-16T15:19:11.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:19:11.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:19:11.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.354+0000 7f8a5c200640 1 Processor -- start 2024-01-16T15:19:11.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.354+0000 7f8a5c200640 1 -- start start 2024-01-16T15:19:11.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.356+0000 7f8a5c200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8a54071200 0x7f8a54161a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.356+0000 7f8a5c200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a54161580 con 0x7f8a54160980 2024-01-16T15:19:11.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.356+0000 7f8a5c200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a54160d60 con 0x7f8a5406ba30 2024-01-16T15:19:11.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.356+0000 7f8a5c200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a540cf1a0 con 0x7f8a54071200 2024-01-16T15:19:11.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.357+0000 7f8a59800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8a54160980 0x7f8a540ce3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50270/0 (socket says 172.21.15.53:50270) 2024-01-16T15:19:11.347 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.357+0000 7f8a59800640 1 -- 172.21.15.53:0/3894021124 learned_addr learned my addr 172.21.15.53:0/3894021124 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:11.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.357+0000 7f8a5a200640 1 --2- 172.21.15.53:0/3894021124 >> v2:172.21.15.146:3300/0 conn(0x7f8a54071200 0x7f8a54161a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.357+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497353919 0 0) 0x7f8a54161580 con 0x7f8a54160980 2024-01-16T15:19:11.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.357+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a34003520 con 0x7f8a54160980 2024-01-16T15:19:11.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 808141038 0 0) 0x7f8a34003520 con 0x7f8a54160980 2024-01-16T15:19:11.349 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 >> v2:172.21.15.146:3300/0 conn(0x7f8a54071200 msgr2=0x7f8a54161a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 --2- 172.21.15.53:0/3894021124 >> v2:172.21.15.146:3300/0 conn(0x7f8a54071200 0x7f8a54161a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 >> v1:172.21.15.146:6789/0 conn(0x7f8a5406ba30 legacy=0x7f8a54070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a540cebe0 con 0x7f8a54160980 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8a440019f0 con 0x7f8a54160980 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 >> v1:172.21.15.53:6789/0 conn(0x7f8a54160980 legacy=0x7f8a540ce3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a34003970 0x7f8a34003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340047c0 0x7f8a34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a34005180 0x7f8a3400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3400bff0 con 0x7f8a34003970 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3400c160 con 0x7f8a34005180 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3400c320 con 0x7f8a340047c0 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a59800640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a34003970 0x7f8a34003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.358+0000 7f8a5a200640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a34005180 0x7f8a3400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a58e00640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340047c0 0x7f8a34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a59800640 1 -- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340047c0 msgr2=0x7f8a34004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a59800640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340047c0 0x7f8a34004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a59800640 1 -- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a34005180 msgr2=0x7f8a3400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a59800640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a34005180 0x7f8a3400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a59800640 1 -- 172.21.15.53:0/3894021124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a34003520 con 0x7f8a34003970 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a5a200640 1 -- 172.21.15.53:0/3894021124 reap_dead start 2024-01-16T15:19:11.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.359+0000 7f8a58e00640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a340047c0 0x7f8a34004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.360+0000 7f8a59800640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a34003970 0x7f8a34003d90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400a830 tx=0x7f8a340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ccaf2a72d13427 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.360+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a440072d0 con 0x7f8a34003970 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.360+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4402f070 con 0x7f8a34003970 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.360+0000 7f8a53e00640 1 -- 172.21.15.53:0/3894021124 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a44007730 con 0x7f8a34003970 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 -- 172.21.15.53:0/3894021124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a34003970 msgr2=0x7f8a34003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a34003970 0x7f8a34003d90 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7f8a4400a830 tx=0x7f8a340057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 -- 172.21.15.53:0/3894021124 shutdown_connections 2024-01-16T15:19:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 --2- 172.21.15.53:0/3894021124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a34003970 0x7f8a34003d90 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 -- 172.21.15.53:0/3894021124 >> 172.21.15.53:0/3894021124 conn(0x7f8a540cf700 msgr2=0x7f8a540cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 -- 172.21.15.53:0/3894021124 shutdown_connections 2024-01-16T15:19:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.361+0000 7f8a5c200640 1 -- 172.21.15.53:0/3894021124 wait complete. 2024-01-16T15:19:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.363+0000 7f8a5c200640 1 Processor -- start 2024-01-16T15:19:11.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.363+0000 7f8a5c200640 1 -- start start 2024-01-16T15:19:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5c200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a34003970 0x7f8a5415f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5c200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a5415e140 0x7f8a54144730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5c200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 0x7f8a54145520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5c200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5415f310 con 0x7f8a541450a0 2024-01-16T15:19:11.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5c200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5415e700 con 0x7f8a5415e140 2024-01-16T15:19:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5c200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5415e9e0 con 0x7f8a34003970 2024-01-16T15:19:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 0x7f8a54145520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5a200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 0x7f8a54145520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50004/0 (socket says 172.21.15.53:50004) 2024-01-16T15:19:11.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a5a200640 1 -- 172.21.15.53:0/4272296013 learned_addr learned my addr 172.21.15.53:0/4272296013 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:11.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.364+0000 7f8a58e00640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a5415e140 0x7f8a54144730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a59800640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a34003970 0x7f8a5415f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5a200640 1 -- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a34003970 msgr2=0x7f8a5415f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5a200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a34003970 0x7f8a5415f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5a200640 1 -- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a5415e140 msgr2=0x7f8a54144730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5a200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a5415e140 0x7f8a54144730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5a200640 1 -- 172.21.15.53:0/4272296013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a5415ecc0 con 0x7f8a541450a0 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a59800640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a34003970 0x7f8a5415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5a200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 0x7f8a54145520 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f8a540cbd30 tx=0x7f8a4c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c004500 con 0x7f8a541450a0 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a4c003610 con 0x7f8a541450a0 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0039c0 con 0x7f8a541450a0 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.365+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5415efa0 con 0x7f8a541450a0 2024-01-16T15:19:11.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.366+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a54145a60 con 0x7f8a541450a0 2024-01-16T15:19:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.367+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5415f4c0 con 0x7f8a541450a0 2024-01-16T15:19:11.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.368+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8a4c003b20 con 0x7f8a541450a0 2024-01-16T15:19:11.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.369+0000 7f8a51600640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a2c06c460 0x7f8a2c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.369+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f8a4c0887f0 con 0x7f8a541450a0 2024-01-16T15:19:11.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.369+0000 7f8a59800640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a2c06c460 0x7f8a2c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.370+0000 7f8a59800640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a2c06c460 0x7f8a2c06e920 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8a44034000 tx=0x7f8a44007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:11.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.373+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8a5415f4c0 con 0x7f8a541450a0 2024-01-16T15:19:11.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.569+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a5406cf20 con 0x7f8a541450a0 2024-01-16T15:19:11.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.570+0000 7f8a51600640 1 -- 172.21.15.53:0/4272296013 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a4c054d30 con 0x7f8a541450a0 2024-01-16T15:19:11.560 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:19:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.572+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a2c06c460 msgr2=0x7f8a2c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.572+0000 7f8a5c200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a2c06c460 0x7f8a2c06e920 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8a44034000 tx=0x7f8a44007000 comp rx=0 tx=0).stop 2024-01-16T15:19:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.572+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 msgr2=0x7f8a54145520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.572+0000 7f8a5c200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 0x7f8a54145520 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7f8a540cbd30 tx=0x7f8a4c011920 comp rx=0 tx=0).stop 2024-01-16T15:19:11.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 shutdown_connections 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8a2c06c460 0x7f8a2c06e920 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8a541450a0 0x7f8a54145520 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8a5415e140 0x7f8a54144730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 --2- 172.21.15.53:0/4272296013 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8a34003970 0x7f8a5415f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 >> 172.21.15.53:0/4272296013 conn(0x7f8a540cf700 msgr2=0x7f8a540cbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 shutdown_connections 2024-01-16T15:19:11.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.573+0000 7f8a5c200640 1 -- 172.21.15.53:0/4272296013 wait complete. 2024-01-16T15:19:11.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:19:11.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.813+0000 7f9a78a00640 1 Processor -- start 2024-01-16T15:19:11.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.813+0000 7f9a78a00640 1 -- start start 2024-01-16T15:19:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a78a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a74160920 0x7f9a740b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a78a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9a740cf9f0 0x7f9a740cfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a78a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740d0a10 con 0x7f9a74161010 2024-01-16T15:19:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a78a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740d03e0 con 0x7f9a740cf9f0 2024-01-16T15:19:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a78a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a740d1f50 con 0x7f9a74160920 2024-01-16T15:19:11.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a6be00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9a740cf9f0 0x7f9a740cfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a73200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9a74161010 0x7f9a74161480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:50282/0 (socket says 172.21.15.53:50282) 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.815+0000 7f9a73200640 1 -- 172.21.15.53:0/1490132123 learned_addr learned my addr 172.21.15.53:0/1490132123 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.816+0000 7f9a72800640 1 --2- 172.21.15.53:0/1490132123 >> v2:172.21.15.146:3300/0 conn(0x7f9a74160920 0x7f9a740b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.816+0000 7f9a6be00640 1 -- 172.21.15.53:0/1490132123 >> v2:172.21.15.146:3300/0 conn(0x7f9a74160920 msgr2=0x7f9a740b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.816+0000 7f9a6be00640 1 --2- 172.21.15.53:0/1490132123 >> v2:172.21.15.146:3300/0 conn(0x7f9a74160920 0x7f9a740b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.816+0000 7f9a6be00640 1 -- 172.21.15.53:0/1490132123 >> v1:172.21.15.53:6789/0 conn(0x7f9a74161010 legacy=0x7f9a74161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.816+0000 7f9a6be00640 1 -- 172.21.15.53:0/1490132123 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740cf080 con 0x7f9a740cf9f0 2024-01-16T15:19:11.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.816+0000 7f9a71e00640 1 -- 172.21.15.53:0/1490132123 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1628036512 0 0) 0x7f9a740d0a10 con 0x7f9a74161010 2024-01-16T15:19:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.817+0000 7f9a6be00640 1 --2- 172.21.15.53:0/1490132123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a740cfea0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f9a7408c060 tx=0x7f9a5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c2962cc719e003 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:11.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.817+0000 7f9a71e00640 1 -- 172.21.15.53:0/1490132123 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c031830 con 0x7f9a740cf9f0 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.817+0000 7f9a71e00640 1 -- 172.21.15.53:0/1490132123 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c031990 con 0x7f9a740cf9f0 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.817+0000 7f9a71e00640 1 -- 172.21.15.53:0/1490132123 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c031d20 con 0x7f9a740cf9f0 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 -- 172.21.15.53:0/1490132123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 msgr2=0x7f9a740cfea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 --2- 172.21.15.53:0/1490132123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a740cfea0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f9a7408c060 tx=0x7f9a5c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 -- 172.21.15.53:0/1490132123 shutdown_connections 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 --2- 172.21.15.53:0/1490132123 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a740cfea0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 --2- 172.21.15.53:0/1490132123 >> v2:172.21.15.146:3300/0 conn(0x7f9a74160920 0x7f9a740b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 -- 172.21.15.53:0/1490132123 >> 172.21.15.53:0/1490132123 conn(0x7f9a740d3090 msgr2=0x7f9a740d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 -- 172.21.15.53:0/1490132123 shutdown_connections 2024-01-16T15:19:11.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.818+0000 7f9a78a00640 1 -- 172.21.15.53:0/1490132123 wait complete. 2024-01-16T15:19:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.820+0000 7f9a78a00640 1 Processor -- start 2024-01-16T15:19:11.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.820+0000 7f9a78a00640 1 -- start start 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a7414a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a74160920 0x7f9a74149820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a78a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a74161010 0x7f9a740ca7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a78a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414d130 con 0x7f9a740cf9f0 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a78a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a740b7c20 con 0x7f9a74160920 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a78a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7414a5f0 con 0x7f9a74161010 2024-01-16T15:19:11.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a72800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a7414a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a72800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a7414a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50026/0 (socket says 172.21.15.53:50026) 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.821+0000 7f9a72800640 1 -- 172.21.15.53:0/717494414 learned_addr learned my addr 172.21.15.53:0/717494414 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a6be00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a74160920 0x7f9a74149820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a72800640 1 -- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a74161010 msgr2=0x7f9a740ca7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a72800640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a74161010 0x7f9a740ca7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a72800640 1 -- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a74160920 msgr2=0x7f9a74149820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a72800640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a74160920 0x7f9a74149820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a72800640 1 -- 172.21.15.53:0/717494414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5c009400 con 0x7f9a740cf9f0 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a6be00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a74160920 0x7f9a74149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:11.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a72800640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a7414a0b0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400d800 tx=0x7f9a6400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400b6c0 con 0x7f9a740cf9f0 2024-01-16T15:19:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6400b820 con 0x7f9a740cf9f0 2024-01-16T15:19:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6400bbb0 con 0x7f9a740cf9f0 2024-01-16T15:19:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.822+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7414c520 con 0x7f9a740cf9f0 2024-01-16T15:19:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.823+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7414cae0 con 0x7f9a740cf9f0 2024-01-16T15:19:11.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.824+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74067450 con 0x7f9a740cf9f0 2024-01-16T15:19:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.824+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f9a6400bd10 con 0x7f9a740cf9f0 2024-01-16T15:19:11.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.825+0000 7f9a6aa00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9a4c06c460 0x7f9a4c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.826+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f9a64088ff0 con 0x7f9a740cf9f0 2024-01-16T15:19:11.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.826+0000 7f9a6be00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9a4c06c460 0x7f9a4c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:11.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.827+0000 7f9a6be00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9a4c06c460 0x7f9a4c06e920 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c036000 tx=0x7f9a5c03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:11.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:11.830+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a640555a0 con 0x7f9a740cf9f0 2024-01-16T15:19:12.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.039+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9a7408c7e0 con 0x7f9a740cf9f0 2024-01-16T15:19:12.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.040+0000 7f9a6aa00640 1 -- 172.21.15.53:0/717494414 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v102) v1 ==== 80+0+122 (secure 0 0 0) 0x7f9a64058bc0 con 0x7f9a740cf9f0 2024-01-16T15:19:12.030 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6834/1250261984 2024-01-17T15:19:05.431906+0000 2024-01-16T15:19:12.030 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.15.53:6832/1250261984 2024-01-17T15:19:05.431906+0000 2024-01-16T15:19:12.031 INFO:teuthology.orchestra.run.smithi053.stderr:listed 2 entries 2024-01-16T15:19:12.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.042+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9a4c06c460 msgr2=0x7f9a4c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:12.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.042+0000 7f9a78a00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9a4c06c460 0x7f9a4c06e920 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c036000 tx=0x7f9a5c03c040 comp rx=0 tx=0).stop 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 msgr2=0x7f9a7414a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a7414a0b0 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x7f9a6400d800 tx=0x7f9a6400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 shutdown_connections 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9a4c06c460 0x7f9a4c06e920 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9a74161010 0x7f9a740ca7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a74160920 0x7f9a74149820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 --2- 172.21.15.53:0/717494414 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a740cf9f0 0x7f9a7414a0b0 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 >> 172.21.15.53:0/717494414 conn(0x7f9a740d3090 msgr2=0x7f9a7406b750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 shutdown_connections 2024-01-16T15:19:12.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.043+0000 7f9a78a00640 1 -- 172.21.15.53:0/717494414 wait complete. 2024-01-16T15:19:12.052 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:19:12.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:12.052 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:12.145 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[46780]: fuse finished with error 0 and tester_r 0 2024-01-16T15:19:12.150 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:19:12.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:12.150 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:19:12.333 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:19:12.333 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:12.396 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:19:12.396 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:19:12.396 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:12.396 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-01-16T15:19:12.453 INFO:teuthology.orchestra.run.smithi053.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-01-16T15:19:12.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:19:12.455 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:19:12.455 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:12.512 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:19:12.512 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:19:12.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:12.513 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-01-16T15:19:12.562 INFO:teuthology.orchestra.run.smithi146.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-01-16T15:19:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:19:12.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:19:12.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.809+0000 7f5b87600640 1 Processor -- start 2024-01-16T15:19:12.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.809+0000 7f5b87600640 1 -- start start 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b87600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800ae5a0 0x7f5b8015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b87600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b800cf290 0x7f5b800cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b87600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b80070c40 con 0x7f5b800b3ae0 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b87600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b80071450 con 0x7f5b800cf290 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b87600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b800cfc60 con 0x7f5b800ae5a0 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b85600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b800cf290 0x7f5b800cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b85600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5b800cf290 0x7f5b800cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33668/0 (socket says 172.21.15.53:33668) 2024-01-16T15:19:12.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.811+0000 7f5b85600640 1 -- 172.21.15.53:0/3388920340 learned_addr learned my addr 172.21.15.53:0/3388920340 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.812+0000 7f5b84c00640 1 --2- 172.21.15.53:0/3388920340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800ae5a0 0x7f5b8015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.812+0000 7f5b85600640 1 -- 172.21.15.53:0/3388920340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800ae5a0 msgr2=0x7f5b8015e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.812+0000 7f5b85600640 1 --2- 172.21.15.53:0/3388920340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800ae5a0 0x7f5b8015e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.812+0000 7f5b85600640 1 -- 172.21.15.53:0/3388920340 >> v1:172.21.15.146:6789/0 conn(0x7f5b800b3ae0 legacy=0x7f5b8006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.812+0000 7f5b85600640 1 -- 172.21.15.53:0/3388920340 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8006c470 con 0x7f5b800cf290 2024-01-16T15:19:12.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.813+0000 7f5b85600640 1 --2- 172.21.15.53:0/3388920340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800cf290 0x7f5b800cf720 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009e50 tx=0x7f5b7802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28304c44d4fdc5d2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:12.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.813+0000 7f5b7f400640 1 -- 172.21.15.53:0/3388920340 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7802cb00 con 0x7f5b800cf290 2024-01-16T15:19:12.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.813+0000 7f5b7f400640 1 -- 172.21.15.53:0/3388920340 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7802cc60 con 0x7f5b800cf290 2024-01-16T15:19:12.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.813+0000 7f5b7f400640 1 -- 172.21.15.53:0/3388920340 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b780343f0 con 0x7f5b800cf290 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 -- 172.21.15.53:0/3388920340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800cf290 msgr2=0x7f5b800cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 --2- 172.21.15.53:0/3388920340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800cf290 0x7f5b800cf720 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f5b78009e50 tx=0x7f5b7802bf60 comp rx=0 tx=0).stop 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 -- 172.21.15.53:0/3388920340 shutdown_connections 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 --2- 172.21.15.53:0/3388920340 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800cf290 0x7f5b800cf720 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 --2- 172.21.15.53:0/3388920340 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800ae5a0 0x7f5b8015e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 -- 172.21.15.53:0/3388920340 >> 172.21.15.53:0/3388920340 conn(0x7f5b8006a800 msgr2=0x7f5b8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 -- 172.21.15.53:0/3388920340 shutdown_connections 2024-01-16T15:19:12.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.814+0000 7f5b87600640 1 -- 172.21.15.53:0/3388920340 wait complete. 2024-01-16T15:19:12.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.816+0000 7f5b87600640 1 Processor -- start 2024-01-16T15:19:12.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.816+0000 7f5b87600640 1 -- start start 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b87600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 0x7f5b8014c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b87600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800b3ae0 0x7f5b8014e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b87600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b800cf290 0x7f5b8015fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b87600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b801f76c0 con 0x7f5b800ae5a0 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b87600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b801f6a40 con 0x7f5b800cf290 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b87600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b801f6d20 con 0x7f5b800b3ae0 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b84c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 0x7f5b8014c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b84c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 0x7f5b8014c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33680/0 (socket says 172.21.15.53:33680) 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.817+0000 7f5b84c00640 1 -- 172.21.15.53:0/712401261 learned_addr learned my addr 172.21.15.53:0/712401261 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b84c00640 1 -- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800b3ae0 msgr2=0x7f5b8014e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b84c00640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800b3ae0 0x7f5b8014e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b84c00640 1 -- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b800cf290 msgr2=0x7f5b8015fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b84c00640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b800cf290 0x7f5b8015fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b84c00640 1 -- 172.21.15.53:0/712401261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b78009b30 con 0x7f5b800ae5a0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b84c00640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 0x7f5b8014c3e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000d800 tx=0x7f5b7000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7000b6c0 con 0x7f5b800ae5a0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7000b820 con 0x7f5b800ae5a0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7000bbb0 con 0x7f5b800ae5a0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b801f7000 con 0x7f5b800ae5a0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.818+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8014cc40 con 0x7f5b800ae5a0 2024-01-16T15:19:12.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.820+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b800675d0 con 0x7f5b800ae5a0 2024-01-16T15:19:12.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.821+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f5b7000bd10 con 0x7f5b800ae5a0 2024-01-16T15:19:12.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.822+0000 7f5b7cc00640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b5806c430 0x7f5b5806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:12.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.822+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f5b70089060 con 0x7f5b800ae5a0 2024-01-16T15:19:12.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.822+0000 7f5b7fe00640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b5806c430 0x7f5b5806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:12.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.824+0000 7f5b7fe00640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b5806c430 0x7f5b5806e8f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5b8006b2d0 tx=0x7f5b680079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:12.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:12.826+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5b700555d0 con 0x7f5b800ae5a0 2024-01-16T15:19:13.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.058+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b800cb620 con 0x7f5b800ae5a0 2024-01-16T15:19:13.049 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.059+0000 7f5b7cc00640 1 -- 172.21.15.53:0/712401261 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f5b70058bf0 con 0x7f5b800ae5a0 2024-01-16T15:19:13.049 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:13.049 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1983839723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1983839723},{"type":"v1","addr":"172.21.15.146:6835","nonce":1983839723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":7048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1926242315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1926242315},{"type":"v1","addr":"172.21.15.146:6837","nonce":1926242315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2218545626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2218545626},{"type":"v1","addr":"172.21.15.53:6835","nonce":2218545626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8916,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/3613190207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3613190207},{"type":"v1","addr":"172.21.15.53:6834","nonce":3613190207}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[]} 2024-01-16T15:19:13.050 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 67 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b5806c430 msgr2=0x7f5b5806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b5806c430 0x7f5b5806e8f0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f5b8006b2d0 tx=0x7f5b680079b0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 msgr2=0x7f5b8014c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 0x7f5b8014c3e0 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f5b7000d800 tx=0x7f5b7000dcc0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 shutdown_connections 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b5806c430 0x7f5b5806e8f0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.052 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5b800cf290 0x7f5b8015fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5b800b3ae0 0x7f5b8014e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 --2- 172.21.15.53:0/712401261 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5b800ae5a0 0x7f5b8014c3e0 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 >> 172.21.15.53:0/712401261 conn(0x7f5b8006a800 msgr2=0x7f5b8015a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:13.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 shutdown_connections 2024-01-16T15:19:13.053 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.061+0000 7f5b87600640 1 -- 172.21.15.53:0/712401261 wait complete. 2024-01-16T15:19:13.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/clone_delay_every_regfile 2024-01-16T15:19:13.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.319+0000 7f2b0dc00640 1 Processor -- start 2024-01-16T15:19:13.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.319+0000 7f2b0dc00640 1 -- start start 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b0dc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2b080b2e00 0x7f2b08070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b0dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b0806cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b0dc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b080706a0 con 0x7f2b0806c730 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b0dc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b080b3290 con 0x7f2b080b2e00 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b0dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080708d0 con 0x7f2b0806cb10 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b07e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2b0806c730 0x7f2b08071970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40624/0 (socket says 172.21.15.53:40624) 2024-01-16T15:19:13.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.321+0000 7f2b07e00640 1 -- 172.21.15.53:0/612537719 learned_addr learned my addr 172.21.15.53:0/612537719 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2affe00640 1 --2- 172.21.15.53:0/612537719 >> v2:172.21.15.53:3300/0 conn(0x7f2b080b2e00 0x7f2b08070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2b0d200640 1 --2- 172.21.15.53:0/612537719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b0806cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2b07400640 1 -- 172.21.15.53:0/612537719 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130844188 0 0) 0x7f2b080706a0 con 0x7f2b0806c730 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2b07400640 1 -- 172.21.15.53:0/612537719 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae8003520 con 0x7f2b0806c730 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2affe00640 1 -- 172.21.15.53:0/612537719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 msgr2=0x7f2b0806cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2affe00640 1 --2- 172.21.15.53:0/612537719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b0806cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2affe00640 1 -- 172.21.15.53:0/612537719 >> v1:172.21.15.53:6789/0 conn(0x7f2b0806c730 legacy=0x7f2b08071970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.322+0000 7f2affe00640 1 -- 172.21.15.53:0/612537719 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b08070bb0 con 0x7f2b080b2e00 2024-01-16T15:19:13.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.323+0000 7f2affe00640 1 --2- 172.21.15.53:0/612537719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b08070160 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f2af00097a0 tx=0x7f2af002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ac8fd8c57f10ab server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:13.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b07400640 1 -- 172.21.15.53:0/612537719 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af002d040 con 0x7f2b080b2e00 2024-01-16T15:19:13.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b07400640 1 -- 172.21.15.53:0/612537719 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2af0031810 con 0x7f2b080b2e00 2024-01-16T15:19:13.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b07400640 1 -- 172.21.15.53:0/612537719 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2af0031ba0 con 0x7f2b080b2e00 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 -- 172.21.15.53:0/612537719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 msgr2=0x7f2b08070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/612537719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b08070160 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f2af00097a0 tx=0x7f2af002b8b0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 -- 172.21.15.53:0/612537719 shutdown_connections 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/612537719 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b0806cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/612537719 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b08070160 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 -- 172.21.15.53:0/612537719 >> 172.21.15.53:0/612537719 conn(0x7f2b0806a7d0 msgr2=0x7f2b0806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.324+0000 7f2b0dc00640 1 -- 172.21.15.53:0/612537719 shutdown_connections 2024-01-16T15:19:13.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.325+0000 7f2b0dc00640 1 -- 172.21.15.53:0/612537719 wait complete. 2024-01-16T15:19:13.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.327+0000 7f2b0dc00640 1 Processor -- start 2024-01-16T15:19:13.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.327+0000 7f2b0dc00640 1 -- start start 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.328+0000 7f2b0dc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b0806c730 0x7f2b080cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.328+0000 7f2b0dc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b080ad5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.328+0000 7f2b0dc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b0815db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.328+0000 7f2b0dc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b0812fad0 con 0x7f2b080b2e00 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.328+0000 7f2b0dc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080adff0 con 0x7f2b0806c730 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.328+0000 7f2b0dc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b080ae2d0 con 0x7f2b0806cb10 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b0815db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b0815db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33692/0 (socket says 172.21.15.53:33692) 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 -- 172.21.15.53:0/1548674649 learned_addr learned my addr 172.21.15.53:0/1548674649 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2affe00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b080ad5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b07e00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b0806c730 0x7f2b080cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 -- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 msgr2=0x7f2b080ad5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b080ad5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 -- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b0806c730 msgr2=0x7f2b080cd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b0806c730 0x7f2b080cd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b0d200640 1 -- 172.21.15.53:0/1548674649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af0009400 con 0x7f2b080b2e00 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.329+0000 7f2b07e00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b0806c730 0x7f2b080cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.330+0000 7f2b0d200640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b0815db80 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f2b00007e40 tx=0x7f2b0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.330+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b00014620 con 0x7f2b080b2e00 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.330+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b0000ee90 con 0x7f2b080b2e00 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.330+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b000149e0 con 0x7f2b080b2e00 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.330+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b0812f410 con 0x7f2b080b2e00 2024-01-16T15:19:13.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.330+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b08144e60 con 0x7f2b080b2e00 2024-01-16T15:19:13.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.331+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08144ff0 con 0x7f2b080b2e00 2024-01-16T15:19:13.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.333+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2b00014b40 con 0x7f2b080b2e00 2024-01-16T15:19:13.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.334+0000 7f2b04c00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ae006c430 0x7f2ae006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.334+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2b00089470 con 0x7f2b080b2e00 2024-01-16T15:19:13.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.335+0000 7f2b07e00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ae006c430 0x7f2ae006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.336+0000 7f2b07e00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ae006c430 0x7f2ae006e8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2af8005880 tx=0x7f2af8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:13.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.339+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2b000559e0 con 0x7f2b080b2e00 2024-01-16T15:19:13.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.536+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"} v 0) v1 -- 0x7f2b0812fc10 con 0x7f2b080b2e00 2024-01-16T15:19:13.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.542+0000 7f2b04c00640 1 -- 172.21.15.53:0/1548674649 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/clone_delay_every_regfile"}]=0 v39) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2b0812fc10 con 0x7f2b080b2e00 2024-01-16T15:19:13.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ae006c430 msgr2=0x7f2ae006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ae006c430 0x7f2ae006e8f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2af8005880 tx=0x7f2af8009040 comp rx=0 tx=0).stop 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 msgr2=0x7f2b0815db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b0815db80 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f2b00007e40 tx=0x7f2b0000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 shutdown_connections 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ae006c430 0x7f2ae006e8f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2b080b2e00 0x7f2b0815db80 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2b0806cb10 0x7f2b080ad5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 --2- 172.21.15.53:0/1548674649 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2b0806c730 0x7f2b080cd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 >> 172.21.15.53:0/1548674649 conn(0x7f2b0806a7d0 msgr2=0x7f2b080ae830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 shutdown_connections 2024-01-16T15:19:13.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.545+0000 7f2b0dc00640 1 -- 172.21.15.53:0/1548674649 wait complete. 2024-01-16T15:19:13.555 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/volumes/snapshot_clone_delay 2024-01-16T15:19:13.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.782+0000 7fd0dae00640 1 Processor -- start 2024-01-16T15:19:13.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.782+0000 7fd0dae00640 1 -- start start 2024-01-16T15:19:13.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.783+0000 7fd0dae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0d413cf10 0x7fd0d413f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0dae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd0d40cddf0 0x7fd0d40b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0dae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0d40ce440 con 0x7fd0d413fae0 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0dae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d413ff90 con 0x7fd0d413cf10 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0dae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0d40ce220 con 0x7fd0d40cddf0 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0d413cf10 0x7fd0d413f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd0d413cf10 0x7fd0d413f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33694/0 (socket says 172.21.15.53:33694) 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 -- 172.21.15.53:0/1161526421 learned_addr learned my addr 172.21.15.53:0/1161526421 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 -- 172.21.15.53:0/1161526421 >> v2:172.21.15.146:3300/0 conn(0x7fd0d40cddf0 msgr2=0x7fd0d40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 --2- 172.21.15.53:0/1161526421 >> v2:172.21.15.146:3300/0 conn(0x7fd0d40cddf0 0x7fd0d40b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 -- 172.21.15.53:0/1161526421 >> v1:172.21.15.146:6789/0 conn(0x7fd0d413fae0 legacy=0x7fd0d40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:13.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.784+0000 7fd0d9a00640 1 -- 172.21.15.53:0/1161526421 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0d40ce720 con 0x7fd0d413cf10 2024-01-16T15:19:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.785+0000 7fd0d9a00640 1 --2- 172.21.15.53:0/1161526421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d413f5a0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4157f90 tx=0x7fd0c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d55514c8350c76 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0d3e00640 1 -- 172.21.15.53:0/1161526421 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4031870 con 0x7fd0d413cf10 2024-01-16T15:19:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0d3e00640 1 -- 172.21.15.53:0/1161526421 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0c40319d0 con 0x7fd0d413cf10 2024-01-16T15:19:13.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0d3e00640 1 -- 172.21.15.53:0/1161526421 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0c4031d60 con 0x7fd0d413cf10 2024-01-16T15:19:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0dae00640 1 -- 172.21.15.53:0/1161526421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 msgr2=0x7fd0d413f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0dae00640 1 --2- 172.21.15.53:0/1161526421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d413f5a0 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fd0d4157f90 tx=0x7fd0c402b910 comp rx=0 tx=0).stop 2024-01-16T15:19:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0dae00640 1 -- 172.21.15.53:0/1161526421 shutdown_connections 2024-01-16T15:19:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0dae00640 1 --2- 172.21.15.53:0/1161526421 >> v2:172.21.15.146:3300/0 conn(0x7fd0d40cddf0 0x7fd0d40b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0dae00640 1 --2- 172.21.15.53:0/1161526421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d413f5a0 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.786+0000 7fd0dae00640 1 -- 172.21.15.53:0/1161526421 >> 172.21.15.53:0/1161526421 conn(0x7fd0d406a7d0 msgr2=0x7fd0d406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:13.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.787+0000 7fd0dae00640 1 -- 172.21.15.53:0/1161526421 shutdown_connections 2024-01-16T15:19:13.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.787+0000 7fd0dae00640 1 -- 172.21.15.53:0/1161526421 wait complete. 2024-01-16T15:19:13.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.788+0000 7fd0dae00640 1 Processor -- start 2024-01-16T15:19:13.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.788+0000 7fd0dae00640 1 -- start start 2024-01-16T15:19:13.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.789+0000 7fd0dae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d40cddf0 0x7fd0d4138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.789+0000 7fd0dae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d4153b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.789+0000 7fd0dae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d413fae0 0x7fd0d41540c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.789+0000 7fd0dae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d413c330 con 0x7fd0d413cf10 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.789+0000 7fd0dae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d414fc50 con 0x7fd0d413fae0 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.789+0000 7fd0dae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0d414ff30 con 0x7fd0d40cddf0 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d4153b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d4153b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33696/0 (socket says 172.21.15.53:33696) 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 -- 172.21.15.53:0/2242223926 learned_addr learned my addr 172.21.15.53:0/2242223926 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9a00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d40cddf0 0x7fd0d4138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 -- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d40cddf0 msgr2=0x7fd0d4138cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d40cddf0 0x7fd0d4138cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 -- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d413fae0 msgr2=0x7fd0d41540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d413fae0 0x7fd0d41540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 -- 172.21.15.53:0/2242223926 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0c40094d0 con 0x7fd0d413cf10 2024-01-16T15:19:13.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.790+0000 7fd0d9000640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d4153b80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00b3e0 tx=0x7fd0bc00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:13.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.791+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc011830 con 0x7fd0d413cf10 2024-01-16T15:19:13.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.791+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0bc0119b0 con 0x7fd0d413cf10 2024-01-16T15:19:13.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.791+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0bc011d60 con 0x7fd0d413cf10 2024-01-16T15:19:13.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.791+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0d4150210 con 0x7fd0d413cf10 2024-01-16T15:19:13.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.791+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0d4155240 con 0x7fd0d413cf10 2024-01-16T15:19:13.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.792+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0d414fd90 con 0x7fd0d413cf10 2024-01-16T15:19:13.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.793+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd0bc01a400 con 0x7fd0d413cf10 2024-01-16T15:19:13.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.795+0000 7fd0d1600640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd0ac06c430 0x7fd0ac06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:13.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.795+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fd0bc088cb0 con 0x7fd0d413cf10 2024-01-16T15:19:13.785 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.795+0000 7fd0d9a00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd0ac06c430 0x7fd0ac06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:13.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.796+0000 7fd0d9a00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd0ac06c430 0x7fd0ac06e8f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4036000 tx=0x7fd0c403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:13.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.800+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd0d414fd90 con 0x7fd0d413cf10 2024-01-16T15:19:13.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.992+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"} v 0) v1 -- 0x7fd0d4150730 con 0x7fd0d413cf10 2024-01-16T15:19:13.988 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:13.998+0000 7fd0d1600640 1 -- 172.21.15.53:0/2242223926 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/volumes/snapshot_clone_delay"}]=0 v40) v1 ==== 115+0+0 (secure 0 0 0) 0x7fd0d4150730 con 0x7fd0d413cf10 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd0ac06c430 msgr2=0x7fd0ac06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd0ac06c430 0x7fd0ac06e8f0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fd0c4036000 tx=0x7fd0c403c040 comp rx=0 tx=0).stop 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 msgr2=0x7fd0d4153b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d4153b80 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc00b3e0 tx=0x7fd0bc00b8a0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 shutdown_connections 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd0ac06c430 0x7fd0ac06e8f0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd0d413fae0 0x7fd0d41540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd0d413cf10 0x7fd0d4153b80 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 --2- 172.21.15.53:0/2242223926 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd0d40cddf0 0x7fd0d4138cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:13.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 >> 172.21.15.53:0/2242223926 conn(0x7fd0d406a7d0 msgr2=0x7fd0d413f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:13.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 shutdown_connections 2024-01-16T15:19:13.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.001+0000 7fd0dae00640 1 -- 172.21.15.53:0/2242223926 wait complete. 2024-01-16T15:19:14.012 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-01-16T15:19:14.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.245+0000 7f87d3e00640 1 Processor -- start 2024-01-16T15:19:14.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.246+0000 7f87d3e00640 1 -- start start 2024-01-16T15:19:14.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f87d415d4b0 0x7f87d40d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d40b41e0 0x7f87d40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d3e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d40d0320 con 0x7f87d40cf9a0 2024-01-16T15:19:14.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d40b2730 con 0x7f87d415d4b0 2024-01-16T15:19:14.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d40d1ec0 con 0x7f87d40b41e0 2024-01-16T15:19:14.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f87d415d4b0 0x7f87d40d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d2000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f87d40cf9a0 0x7f87d40cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40630/0 (socket says 172.21.15.53:40630) 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.247+0000 7f87d2000640 1 -- 172.21.15.53:0/1478665845 learned_addr learned my addr 172.21.15.53:0/1478665845 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.248+0000 7f87d2a00640 1 -- 172.21.15.53:0/1478665845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d40b41e0 msgr2=0x7f87d40cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.248+0000 7f87d3400640 1 --2- 172.21.15.53:0/1478665845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d40b41e0 0x7f87d40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.248+0000 7f87d2a00640 1 --2- 172.21.15.53:0/1478665845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d40b41e0 0x7f87d40cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.248+0000 7f87d2a00640 1 -- 172.21.15.53:0/1478665845 >> v1:172.21.15.53:6789/0 conn(0x7f87d40cf9a0 legacy=0x7f87d40cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.248+0000 7f87d2a00640 1 -- 172.21.15.53:0/1478665845 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d40b29d0 con 0x7f87d415d4b0 2024-01-16T15:19:14.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.248+0000 7f87d1600640 1 -- 172.21.15.53:0/1478665845 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562574250 0 0) 0x7f87d40d0320 con 0x7f87d40cf9a0 2024-01-16T15:19:14.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.249+0000 7f87d2a00640 1 --2- 172.21.15.53:0/1478665845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d415d4b0 0x7f87d40d1550 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f87c40097a0 tx=0x7f87c402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=416f0e5bbe7442aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:14.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.249+0000 7f87d1600640 1 -- 172.21.15.53:0/1478665845 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c402d040 con 0x7f87d415d4b0 2024-01-16T15:19:14.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.249+0000 7f87d1600640 1 -- 172.21.15.53:0/1478665845 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c4031810 con 0x7f87d415d4b0 2024-01-16T15:19:14.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.249+0000 7f87d1600640 1 -- 172.21.15.53:0/1478665845 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4031ba0 con 0x7f87d415d4b0 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 -- 172.21.15.53:0/1478665845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d415d4b0 msgr2=0x7f87d40d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 --2- 172.21.15.53:0/1478665845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d415d4b0 0x7f87d40d1550 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f87c40097a0 tx=0x7f87c402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 -- 172.21.15.53:0/1478665845 shutdown_connections 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 --2- 172.21.15.53:0/1478665845 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d40b41e0 0x7f87d40cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 --2- 172.21.15.53:0/1478665845 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d415d4b0 0x7f87d40d1550 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 -- 172.21.15.53:0/1478665845 >> 172.21.15.53:0/1478665845 conn(0x7f87d40d3000 msgr2=0x7f87d40d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 -- 172.21.15.53:0/1478665845 shutdown_connections 2024-01-16T15:19:14.241 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.250+0000 7f87d3e00640 1 -- 172.21.15.53:0/1478665845 wait complete. 2024-01-16T15:19:14.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.252+0000 7f87d3e00640 1 Processor -- start 2024-01-16T15:19:14.242 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.252+0000 7f87d3e00640 1 -- start start 2024-01-16T15:19:14.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 0x7f87d406c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87d40cf9a0 0x7f87d40a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d415d4b0 0x7f87d40a0910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.243 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d406b030 con 0x7f87d40b41e0 2024-01-16T15:19:14.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d4154a00 con 0x7f87d40cf9a0 2024-01-16T15:19:14.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d406b280 con 0x7f87d415d4b0 2024-01-16T15:19:14.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 0x7f87d406c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 0x7f87d406c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33722/0 (socket says 172.21.15.53:33722) 2024-01-16T15:19:14.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 -- 172.21.15.53:0/3691106542 learned_addr learned my addr 172.21.15.53:0/3691106542 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 -- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d415d4b0 msgr2=0x7f87d40a0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d415d4b0 0x7f87d40a0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 -- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87d40cf9a0 msgr2=0x7f87d40a03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87d40cf9a0 0x7f87d40a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.253+0000 7f87d2a00640 1 -- 172.21.15.53:0/3691106542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c4009400 con 0x7f87d40b41e0 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.254+0000 7f87d2a00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 0x7f87d406c320 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f87c40097a0 tx=0x7f87c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.254+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c403aad0 con 0x7f87d40b41e0 2024-01-16T15:19:14.245 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.254+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c403ac30 con 0x7f87d40b41e0 2024-01-16T15:19:14.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.254+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d415da10 con 0x7f87d40b41e0 2024-01-16T15:19:14.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.254+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c4004030 con 0x7f87d40b41e0 2024-01-16T15:19:14.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.254+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d406b840 con 0x7f87d40b41e0 2024-01-16T15:19:14.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.256+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d4067450 con 0x7f87d40b41e0 2024-01-16T15:19:14.246 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.256+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f87c4042070 con 0x7f87d40b41e0 2024-01-16T15:19:14.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.257+0000 7f87caa00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87ac06c430 0x7f87ac06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.247 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.257+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f87c40aa670 con 0x7f87d40b41e0 2024-01-16T15:19:14.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.258+0000 7f87d2000640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87ac06c430 0x7f87ac06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.249 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.259+0000 7f87d2000640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87ac06c430 0x7f87ac06e8f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f87bc001560 tx=0x7f87bc007ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:14.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.262+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87c4077050 con 0x7f87d40b41e0 2024-01-16T15:19:14.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.446+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f87d40cfff0 con 0x7f87d40b41e0 2024-01-16T15:19:14.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.451+0000 7f87caa00640 1 -- 172.21.15.53:0/3691106542 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v41) v1 ==== 104+0+0 (secure 0 0 0) 0x7f87c40350a0 con 0x7f87d40b41e0 2024-01-16T15:19:14.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87ac06c430 msgr2=0x7f87ac06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87ac06c430 0x7f87ac06e8f0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f87bc001560 tx=0x7f87bc007ca0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 msgr2=0x7f87d406c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 0x7f87d406c320 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f87c40097a0 tx=0x7f87c40046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 shutdown_connections 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87ac06c430 0x7f87ac06e8f0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87d415d4b0 0x7f87d40a0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87d40cf9a0 0x7f87d40a03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 --2- 172.21.15.53:0/3691106542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87d40b41e0 0x7f87d406c320 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 >> 172.21.15.53:0/3691106542 conn(0x7f87d40d3000 msgr2=0x7f87d40b5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 shutdown_connections 2024-01-16T15:19:14.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.454+0000 7f87d3e00640 1 -- 172.21.15.53:0/3691106542 wait complete. 2024-01-16T15:19:14.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled' 2024-01-16T15:19:14.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.693+0000 7f7510600640 1 Processor -- start 2024-01-16T15:19:14.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.693+0000 7f7510600640 1 -- start start 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f7510600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f750813e360 0x7f75080cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f7510600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f750815f9c0 0x7f75080cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f7510600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750815c5f0 con 0x7f750813df80 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f7510600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750806c470 con 0x7f750815f9c0 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f7510600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750806c750 con 0x7f750813e360 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f750e600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f750815f9c0 0x7f75080cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f750e600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f750815f9c0 0x7f75080cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33734/0 (socket says 172.21.15.53:33734) 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.695+0000 7f750e600640 1 -- 172.21.15.53:0/595435790 learned_addr learned my addr 172.21.15.53:0/595435790 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:14.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.696+0000 7f750d200640 1 --2- 172.21.15.53:0/595435790 >> v2:172.21.15.146:3300/0 conn(0x7f750813e360 0x7f75080cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.696+0000 7f750e600640 1 -- 172.21.15.53:0/595435790 >> v2:172.21.15.146:3300/0 conn(0x7f750813e360 msgr2=0x7f75080cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:14.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.696+0000 7f750e600640 1 --2- 172.21.15.53:0/595435790 >> v2:172.21.15.146:3300/0 conn(0x7f750813e360 0x7f75080cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.696+0000 7f750e600640 1 -- 172.21.15.53:0/595435790 >> v1:172.21.15.146:6789/0 conn(0x7f750813df80 legacy=0x7f750815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:14.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.696+0000 7f750e600640 1 -- 172.21.15.53:0/595435790 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750806ca30 con 0x7f750815f9c0 2024-01-16T15:19:14.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.697+0000 7f750e600640 1 --2- 172.21.15.53:0/595435790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750815f9c0 0x7f75080cf7d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f75080ccae0 tx=0x7f750002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b62143cdab6d770 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7507e00640 1 -- 172.21.15.53:0/595435790 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750002ca50 con 0x7f750815f9c0 2024-01-16T15:19:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7507e00640 1 -- 172.21.15.53:0/595435790 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750002cbb0 con 0x7f750815f9c0 2024-01-16T15:19:14.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7507e00640 1 -- 172.21.15.53:0/595435790 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75000343f0 con 0x7f750815f9c0 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7510600640 1 -- 172.21.15.53:0/595435790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750815f9c0 msgr2=0x7f75080cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7510600640 1 --2- 172.21.15.53:0/595435790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750815f9c0 0x7f75080cf7d0 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7f75080ccae0 tx=0x7f750002beb0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7510600640 1 -- 172.21.15.53:0/595435790 shutdown_connections 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.698+0000 7f7510600640 1 --2- 172.21.15.53:0/595435790 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750815f9c0 0x7f75080cf7d0 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.699+0000 7f7510600640 1 --2- 172.21.15.53:0/595435790 >> v2:172.21.15.146:3300/0 conn(0x7f750813e360 0x7f75080cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.699+0000 7f7510600640 1 -- 172.21.15.53:0/595435790 >> 172.21.15.53:0/595435790 conn(0x7f750806a800 msgr2=0x7f750806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.699+0000 7f7510600640 1 -- 172.21.15.53:0/595435790 shutdown_connections 2024-01-16T15:19:14.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.699+0000 7f7510600640 1 -- 172.21.15.53:0/595435790 wait complete. 2024-01-16T15:19:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.701+0000 7f7510600640 1 Processor -- start 2024-01-16T15:19:14.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.701+0000 7f7510600640 1 -- start start 2024-01-16T15:19:14.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.702+0000 7f7510600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f750813df80 0x7f75080b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.702+0000 7f7510600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 0x7f750814aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f7510600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f750815f9c0 0x7f750814b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f7510600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750814c440 con 0x7f750813e360 2024-01-16T15:19:14.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f7510600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750814b830 con 0x7f750815f9c0 2024-01-16T15:19:14.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f7510600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750814bb10 con 0x7f750813df80 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 0x7f750814aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 0x7f750814aac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33738/0 (socket says 172.21.15.53:33738) 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 -- 172.21.15.53:0/941013379 learned_addr learned my addr 172.21.15.53:0/941013379 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750dc00640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f750813df80 0x7f75080b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750e600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f750815f9c0 0x7f750814b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 -- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f750813df80 msgr2=0x7f75080b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f750813df80 0x7f75080b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 -- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f750815f9c0 msgr2=0x7f750814b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f750815f9c0 0x7f750814b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.703+0000 7f750d200640 1 -- 172.21.15.53:0/941013379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7500009b30 con 0x7f750813e360 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.704+0000 7f750d200640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 0x7f750814aac0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f74f000d7d0 tx=0x7f74f000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.704+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f0018070 con 0x7f750813e360 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.704+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f0004020 con 0x7f750813e360 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.704+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f00041b0 con 0x7f750813e360 2024-01-16T15:19:14.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.704+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750814bdf0 con 0x7f750813e360 2024-01-16T15:19:14.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.704+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750814b540 con 0x7f750813e360 2024-01-16T15:19:14.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.705+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750814c5b0 con 0x7f750813e360 2024-01-16T15:19:14.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.706+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f74f0004310 con 0x7f750813e360 2024-01-16T15:19:14.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.708+0000 7f7505600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f74e006c430 0x7f74e006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:14.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.708+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f74f0088980 con 0x7f750813e360 2024-01-16T15:19:14.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.709+0000 7f750dc00640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f74e006c430 0x7f74e006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:14.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.710+0000 7f750dc00640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f74e006c430 0x7f74e006e8f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f74f800a750 tx=0x7f74f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:14.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.713+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f74f0054ef0 con 0x7f750813e360 2024-01-16T15:19:14.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:14.982+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]} v 0) v1 -- 0x7f75080675d0 con 0x7f750813e360 2024-01-16T15:19:15.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.509+0000 7f7505600640 1 -- 172.21.15.53:0/941013379 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_volumes.TestCloneProgress.test_when_3_clones_are_cancelled"]}]=0 v808) v1 ==== 155+0+0 (secure 0 0 0) 0x7f74f0058510 con 0x7f750813e360 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f74e006c430 msgr2=0x7f74e006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f74e006c430 0x7f74e006e8f0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f74f800a750 tx=0x7f74f8008040 comp rx=0 tx=0).stop 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 msgr2=0x7f750814aac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 0x7f750814aac0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f74f000d7d0 tx=0x7f74f000dc90 comp rx=0 tx=0).stop 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 shutdown_connections 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f74e006c430 0x7f74e006e8f0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f750815f9c0 0x7f750814b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f750813e360 0x7f750814aac0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 --2- 172.21.15.53:0/941013379 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f750813df80 0x7f75080b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 >> 172.21.15.53:0/941013379 conn(0x7f750806a800 msgr2=0x7f750815cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 shutdown_connections 2024-01-16T15:19:15.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.513+0000 7f7510600640 1 -- 172.21.15.53:0/941013379 wait complete. 2024-01-16T15:19:15.529 INFO:tasks.cephfs_test_runner:Test that the progress bars for ongoing clones is removed from the ... ok 2024-01-16T15:19:15.529 INFO:tasks.cephfs_test_runner:Starting test: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:19:15.529 INFO:tasks.cephfs_test_runner:test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:19:15.530 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:15.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.758+0000 7f80ba000640 1 Processor -- start 2024-01-16T15:19:15.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.758+0000 7f80ba000640 1 -- start start 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80ba000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f80b40cc990 0x7f80b406d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80ba000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b4070e50 con 0x7f80b40cc500 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80ba000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80b4144020 con 0x7f80b4143c40 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80ba000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80b40723c0 con 0x7f80b40cc990 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80b3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f80b40cc500 0x7f80b406d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40646/0 (socket says 172.21.15.53:40646) 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80b3e00640 1 -- 172.21.15.53:0/3343792164 learned_addr learned my addr 172.21.15.53:0/3343792164 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:15.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.760+0000 7f80b9600640 1 --2- 172.21.15.53:0/3343792164 >> v2:172.21.15.53:3300/0 conn(0x7f80b40cc990 0x7f80b406d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.761+0000 7f80b3400640 1 -- 172.21.15.53:0/3343792164 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957393885 0 0) 0x7f80b4070e50 con 0x7f80b40cc500 2024-01-16T15:19:15.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.761+0000 7f80b3400640 1 -- 172.21.15.53:0/3343792164 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8094003520 con 0x7f80b40cc500 2024-01-16T15:19:15.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.761+0000 7f80b9600640 1 -- 172.21.15.53:0/3343792164 >> v1:172.21.15.146:6789/0 conn(0x7f80b4143c40 legacy=0x7f80b4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.761+0000 7f80b9600640 1 -- 172.21.15.53:0/3343792164 >> v1:172.21.15.53:6789/0 conn(0x7f80b40cc500 legacy=0x7f80b406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.761+0000 7f80b9600640 1 -- 172.21.15.53:0/3343792164 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80b4071e00 con 0x7f80b40cc990 2024-01-16T15:19:15.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.762+0000 7f80b9600640 1 --2- 172.21.15.53:0/3343792164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b40cc990 0x7f80b406d870 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0e89a6c2a5a346 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:15.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80b3400640 1 -- 172.21.15.53:0/3343792164 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac02cc60 con 0x7f80b40cc990 2024-01-16T15:19:15.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80b3400640 1 -- 172.21.15.53:0/3343792164 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac02cdc0 con 0x7f80b40cc990 2024-01-16T15:19:15.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80b3400640 1 -- 172.21.15.53:0/3343792164 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac034640 con 0x7f80b40cc990 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 -- 172.21.15.53:0/3343792164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b40cc990 msgr2=0x7f80b406d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 --2- 172.21.15.53:0/3343792164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b40cc990 0x7f80b406d870 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac02beb0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 -- 172.21.15.53:0/3343792164 shutdown_connections 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 --2- 172.21.15.53:0/3343792164 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b40cc990 0x7f80b406d870 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 -- 172.21.15.53:0/3343792164 >> 172.21.15.53:0/3343792164 conn(0x7f80b406a800 msgr2=0x7f80b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 -- 172.21.15.53:0/3343792164 shutdown_connections 2024-01-16T15:19:15.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.763+0000 7f80ba000640 1 -- 172.21.15.53:0/3343792164 wait complete. 2024-01-16T15:19:15.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.765+0000 7f80ba000640 1 Processor -- start 2024-01-16T15:19:15.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.765+0000 7f80ba000640 1 -- start start 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.766+0000 7f80ba000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80b40cc500 0x7f80b40b2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.766+0000 7f80ba000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80b40cc990 0x7f80b415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.766+0000 7f80ba000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 0x7f80b415aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.766+0000 7f80ba000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40c7dc0 con 0x7f80b4143c40 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.766+0000 7f80ba000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40c3ef0 con 0x7f80b40cc990 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.766+0000 7f80ba000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80b40c41d0 con 0x7f80b40cc500 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 0x7f80b415aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 0x7f80b415aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33764/0 (socket says 172.21.15.53:33764) 2024-01-16T15:19:15.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 -- 172.21.15.53:0/647535545 learned_addr learned my addr 172.21.15.53:0/647535545 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:15.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b3e00640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80b40cc990 0x7f80b415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b8c00640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80b40cc500 0x7f80b40b2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 -- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80b40cc500 msgr2=0x7f80b40b2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80b40cc500 0x7f80b40b2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 -- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80b40cc990 msgr2=0x7f80b415a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80b40cc990 0x7f80b415a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 -- 172.21.15.53:0/647535545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac009a70 con 0x7f80b4143c40 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.767+0000 7f80b9600640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 0x7f80b415aa70 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.768+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac003b90 con 0x7f80b4143c40 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.768+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80ac003d10 con 0x7f80b4143c40 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.768+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80ac004270 con 0x7f80b4143c40 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.768+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80b40c44b0 con 0x7f80b4143c40 2024-01-16T15:19:15.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.768+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80b40b33f0 con 0x7f80b4143c40 2024-01-16T15:19:15.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.769+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80b40c49d0 con 0x7f80b4143c40 2024-01-16T15:19:15.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.770+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f80ac03b050 con 0x7f80b4143c40 2024-01-16T15:19:15.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.771+0000 7f80b0c00640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f808c06c430 0x7f808c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:15.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.771+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f80ac0ab010 con 0x7f80b4143c40 2024-01-16T15:19:15.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.772+0000 7f80b8c00640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f808c06c430 0x7f808c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:15.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.773+0000 7f80b8c00640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f808c06c430 0x7f808c06e8f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000f60 tx=0x7f80a4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:15.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.776+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80b40c49d0 con 0x7f80b4143c40 2024-01-16T15:19:15.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.978+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80b40c49d0 con 0x7f80b4143c40 2024-01-16T15:19:15.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.980+0000 7f80b0c00640 1 -- 172.21.15.53:0/647535545 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+12995 (secure 0 0 0) 0x7f80b40c49d0 con 0x7f80b4143c40 2024-01-16T15:19:15.971 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:15.971 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":102,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:08.449618+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.53:6834/1250261984":"2024-01-17T15:19:05.431906+0000","172.21.15.53:6832/1250261984":"2024-01-17T15:19:05.431906+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:15.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f808c06c430 msgr2=0x7f808c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f808c06c430 0x7f808c06e8f0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f80a4000f60 tx=0x7f80a4009040 comp rx=0 tx=0).stop 2024-01-16T15:19:15.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 msgr2=0x7f80b415aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 0x7f80b415aa70 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f80ac009d90 tx=0x7f80ac004a40 comp rx=0 tx=0).stop 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 shutdown_connections 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f808c06c430 0x7f808c06e8f0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f80b4143c40 0x7f80b415aa70 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f80b40cc990 0x7f80b415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 --2- 172.21.15.53:0/647535545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f80b40cc500 0x7f80b40b2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 >> 172.21.15.53:0/647535545 conn(0x7f80b406a800 msgr2=0x7f80b413fdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 shutdown_connections 2024-01-16T15:19:15.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:15.983+0000 7f80ba000640 1 -- 172.21.15.53:0/647535545 wait complete. 2024-01-16T15:19:15.992 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:19:15.992 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:19:16.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.222+0000 7f2e16000640 1 Processor -- start 2024-01-16T15:19:16.212 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.222+0000 7f2e16000640 1 -- start start 2024-01-16T15:19:16.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.224+0000 7f2e16000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 0x7f2e1015bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.224+0000 7f2e16000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e1015f4e0 con 0x7f2e100adf20 2024-01-16T15:19:16.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.224+0000 7f2e16000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e10070ee0 con 0x7f2e100ccb50 2024-01-16T15:19:16.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.224+0000 7f2e16000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1015c450 con 0x7f2e10070a30 2024-01-16T15:19:16.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.224+0000 7f2e14c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2e100adf20 0x7f2e100cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40650/0 (socket says 172.21.15.53:40650) 2024-01-16T15:19:16.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.224+0000 7f2e14c00640 1 -- 172.21.15.53:0/872115980 learned_addr learned my addr 172.21.15.53:0/872115980 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311175597 0 0) 0x7f2e1015f4e0 con 0x7f2e100adf20 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df0003520 con 0x7f2e100adf20 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 308974490 0 0) 0x7f2df0003520 con 0x7f2e100adf20 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 msgr2=0x7f2e1015bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 0x7f2e1015bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 >> v1:172.21.15.146:6789/0 conn(0x7f2e100ccb50 legacy=0x7f2e10143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e10071130 con 0x7f2e100adf20 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2e00001670 con 0x7f2e100adf20 2024-01-16T15:19:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 >> v1:172.21.15.53:6789/0 conn(0x7f2e100adf20 legacy=0x7f2e100cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2df00039a0 0x7f2df0003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df00047c0 0x7f2df0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 0x7f2df000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1015c450 con 0x7f2df00047c0 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df00055c0 con 0x7f2df00039a0 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.225+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2df000bec0 con 0x7f2e10070a30 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e07e00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df00047c0 0x7f2df0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e15600640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 0x7f2df000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e14c00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2df00039a0 0x7f2df0003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e07e00640 1 -- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 msgr2=0x7f2df000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e07e00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 0x7f2df000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e07e00640 1 -- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2df00039a0 msgr2=0x7f2df0003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e07e00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2df00039a0 0x7f2df0003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e07e00640 1 -- 172.21.15.53:0/872115980 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0003520 con 0x7f2df00047c0 2024-01-16T15:19:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e14c00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2df00039a0 0x7f2df0003da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:16.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.226+0000 7f2e15600640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2e10070a30 0x7f2df000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.227+0000 7f2e07e00640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df00047c0 0x7f2df0004c40 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2df8003d80 tx=0x7f2df00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d6faadc071672 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.227+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df8032600 con 0x7f2df00047c0 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df802e050 con 0x7f2df00047c0 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e0fe00640 1 -- 172.21.15.53:0/872115980 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df80329a0 con 0x7f2df00047c0 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e16000640 1 -- 172.21.15.53:0/872115980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df00047c0 msgr2=0x7f2df0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e16000640 1 --2- 172.21.15.53:0/872115980 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2df00047c0 0x7f2df0004c40 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f2df8003d80 tx=0x7f2df00057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e15600640 1 -- 172.21.15.53:0/872115980 reap_dead start 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e16000640 1 -- 172.21.15.53:0/872115980 shutdown_connections 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e16000640 1 -- 172.21.15.53:0/872115980 >> 172.21.15.53:0/872115980 conn(0x7f2e1006a7d0 msgr2=0x7f2e1006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e16000640 1 -- 172.21.15.53:0/872115980 shutdown_connections 2024-01-16T15:19:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.228+0000 7f2e16000640 1 -- 172.21.15.53:0/872115980 wait complete. 2024-01-16T15:19:16.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.230+0000 7f2e16000640 1 Processor -- start 2024-01-16T15:19:16.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.231+0000 7f2e16000640 1 -- start start 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.231+0000 7f2e16000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2df00047c0 0x7f2e1015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e16000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 0x7f2e1013b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e16000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1013b540 0x7f2e100af440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e16000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1015ad10 con 0x7f2e1013cc10 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e16000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1013e7f0 con 0x7f2e1013b540 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e16000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1013bb70 con 0x7f2df00047c0 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 0x7f2e1013b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 0x7f2e1013b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33780/0 (socket says 172.21.15.53:33780) 2024-01-16T15:19:16.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 -- 172.21.15.53:0/652259148 learned_addr learned my addr 172.21.15.53:0/652259148 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e14c00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2df00047c0 0x7f2e1015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e15600640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1013b540 0x7f2e100af440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 -- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2df00047c0 msgr2=0x7f2e1015a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2df00047c0 0x7f2e1015a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 -- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1013b540 msgr2=0x7f2e100af440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1013b540 0x7f2e100af440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e07e00640 1 -- 172.21.15.53:0/652259148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e1013be50 con 0x7f2e1013cc10 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.232+0000 7f2e14c00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2df00047c0 0x7f2e1015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.233+0000 7f2e07e00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 0x7f2e1013b000 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f2df802bb10 tx=0x7f2df80036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.233+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df803d420 con 0x7f2e1013cc10 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.233+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df803d580 con 0x7f2e1013cc10 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.233+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df803d8f0 con 0x7f2e1013cc10 2024-01-16T15:19:16.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.233+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1013c130 con 0x7f2e1013cc10 2024-01-16T15:19:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.233+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1013c6f0 con 0x7f2e1013cc10 2024-01-16T15:19:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.234+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e1013b920 con 0x7f2e1013cc10 2024-01-16T15:19:16.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.235+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f2df803da50 con 0x7f2e1013cc10 2024-01-16T15:19:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.236+0000 7f2e0d600640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2de806c460 0x7f2de806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.236+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f2df80aa6c0 con 0x7f2e1013cc10 2024-01-16T15:19:16.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.237+0000 7f2e14c00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2de806c460 0x7f2de806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.238+0000 7f2e14c00640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2de806c460 0x7f2de806e920 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2e000017a0 tx=0x7f2e00009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:16.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.241+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2df8076ce0 con 0x7f2e1013cc10 2024-01-16T15:19:16.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.435+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2e1013c880 con 0x7f2e1013cc10 2024-01-16T15:19:16.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.436+0000 7f2e0d600640 1 -- 172.21.15.53:0/652259148 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2df8037020 con 0x7f2e1013cc10 2024-01-16T15:19:16.426 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:19:16.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.438+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2de806c460 msgr2=0x7f2de806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2de806c460 0x7f2de806e920 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f2e000017a0 tx=0x7f2e00009040 comp rx=0 tx=0).stop 2024-01-16T15:19:16.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 msgr2=0x7f2e1013b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 0x7f2e1013b000 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f2df802bb10 tx=0x7f2df80036f0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 shutdown_connections 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2de806c460 0x7f2de806e920 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2e1013b540 0x7f2e100af440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2e1013cc10 0x7f2e1013b000 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 --2- 172.21.15.53:0/652259148 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2df00047c0 0x7f2e1015a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 >> 172.21.15.53:0/652259148 conn(0x7f2e1006a7d0 msgr2=0x7f2e1015cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 shutdown_connections 2024-01-16T15:19:16.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.439+0000 7f2e16000640 1 -- 172.21.15.53:0/652259148 wait complete. 2024-01-16T15:19:16.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled' 2024-01-16T15:19:16.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.680+0000 7fc204c00640 1 Processor -- start 2024-01-16T15:19:16.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.680+0000 7fc204c00640 1 -- start start 2024-01-16T15:19:16.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.682+0000 7fc204c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc2000ae5a0 0x7fc20015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.682+0000 7fc204c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000cf290 0x7fc2000cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.682+0000 7fc204c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc200070c40 con 0x7fc2000b3ae0 2024-01-16T15:19:16.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.682+0000 7fc204c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc200071450 con 0x7fc2000ae5a0 2024-01-16T15:19:16.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.682+0000 7fc204c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2000cfc60 con 0x7fc2000cf290 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.682+0000 7fc1fd400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc2000b3ae0 0x7fc20006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40652/0 (socket says 172.21.15.53:40652) 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fd400640 1 -- 172.21.15.53:0/2951613407 learned_addr learned my addr 172.21.15.53:0/2951613407 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fde00640 1 --2- 172.21.15.53:0/2951613407 >> v2:172.21.15.53:3300/0 conn(0x7fc2000ae5a0 0x7fc20015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fe800640 1 --2- 172.21.15.53:0/2951613407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000cf290 0x7fc2000cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fca00640 1 -- 172.21.15.53:0/2951613407 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790489058 0 0) 0x7fc200070c40 con 0x7fc2000b3ae0 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fde00640 1 -- 172.21.15.53:0/2951613407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000cf290 msgr2=0x7fc2000cf720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fde00640 1 --2- 172.21.15.53:0/2951613407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000cf290 0x7fc2000cf720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fde00640 1 -- 172.21.15.53:0/2951613407 >> v1:172.21.15.53:6789/0 conn(0x7fc2000b3ae0 legacy=0x7fc20006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.683+0000 7fc1fde00640 1 -- 172.21.15.53:0/2951613407 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20006c470 con 0x7fc2000ae5a0 2024-01-16T15:19:16.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.684+0000 7fc1fde00640 1 --2- 172.21.15.53:0/2951613407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000ae5a0 0x7fc20015e430 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80097a0 tx=0x7fc1e802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d9202bae3a361e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:16.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc1fca00640 1 -- 172.21.15.53:0/2951613407 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8030610 con 0x7fc2000ae5a0 2024-01-16T15:19:16.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc1fca00640 1 -- 172.21.15.53:0/2951613407 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1e8030770 con 0x7fc2000ae5a0 2024-01-16T15:19:16.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc1fca00640 1 -- 172.21.15.53:0/2951613407 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e8030b00 con 0x7fc2000ae5a0 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 -- 172.21.15.53:0/2951613407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000ae5a0 msgr2=0x7fc20015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 --2- 172.21.15.53:0/2951613407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000ae5a0 0x7fc20015e430 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7fc1e80097a0 tx=0x7fc1e802b960 comp rx=0 tx=0).stop 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 -- 172.21.15.53:0/2951613407 shutdown_connections 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 --2- 172.21.15.53:0/2951613407 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000cf290 0x7fc2000cf720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 --2- 172.21.15.53:0/2951613407 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000ae5a0 0x7fc20015e430 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 -- 172.21.15.53:0/2951613407 >> 172.21.15.53:0/2951613407 conn(0x7fc20006a800 msgr2=0x7fc20006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 -- 172.21.15.53:0/2951613407 shutdown_connections 2024-01-16T15:19:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.685+0000 7fc204c00640 1 -- 172.21.15.53:0/2951613407 wait complete. 2024-01-16T15:19:16.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.687+0000 7fc204c00640 1 Processor -- start 2024-01-16T15:19:16.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.687+0000 7fc204c00640 1 -- start start 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.688+0000 7fc204c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000ae5a0 0x7fc200156c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.688+0000 7fc204c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000b3ae0 0x7fc2001450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.688+0000 7fc204c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 0x7fc2000cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.688+0000 7fc204c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2001455f0 con 0x7fc2000cf290 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.688+0000 7fc204c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc20015f9b0 con 0x7fc2000b3ae0 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.688+0000 7fc204c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc200148880 con 0x7fc2000ae5a0 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 0x7fc2000cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 0x7fc2000cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33788/0 (socket says 172.21.15.53:33788) 2024-01-16T15:19:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 -- 172.21.15.53:0/219424488 learned_addr learned my addr 172.21.15.53:0/219424488 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:16.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fde00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000ae5a0 0x7fc200156c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.680 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fd400640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000b3ae0 0x7fc2001450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 -- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000ae5a0 msgr2=0x7fc200156c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000ae5a0 0x7fc200156c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 -- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000b3ae0 msgr2=0x7fc2001450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000b3ae0 0x7fc2001450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.689+0000 7fc1fe800640 1 -- 172.21.15.53:0/219424488 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1e8009400 con 0x7fc2000cf290 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.690+0000 7fc1fe800640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 0x7fc2000cc8e0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800d5d0 tx=0x7fc1f800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.690+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f80125f0 con 0x7fc2000cf290 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.690+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1f8012770 con 0x7fc2000cf290 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.690+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2001485a0 con 0x7fc2000cf290 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.690+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1f8012b20 con 0x7fc2000cf290 2024-01-16T15:19:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.690+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20015f640 con 0x7fc2000cf290 2024-01-16T15:19:16.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.691+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2000675d0 con 0x7fc2000cf290 2024-01-16T15:19:16.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.692+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc1f8012c80 con 0x7fc2000cf290 2024-01-16T15:19:16.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.693+0000 7fc1f6000640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1d806c480 0x7fc1d806e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:16.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.693+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fc1f8088f30 con 0x7fc2000cf290 2024-01-16T15:19:16.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.694+0000 7fc1fde00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1d806c480 0x7fc1d806e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:16.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.695+0000 7fc1fde00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1d806c480 0x7fc1d806e940 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8036000 tx=0x7fc1e8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:16.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.698+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc1f80554a0 con 0x7fc2000cf290 2024-01-16T15:19:16.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:16.966+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]} v 0) v1 -- 0x7fc20015faf0 con 0x7fc2000cf290 2024-01-16T15:19:17.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.520+0000 7fc1f6000640 1 -- 172.21.15.53:0/219424488 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_volumes.TestCloneProgress.test_when_7_clones_are_cancelled"]}]=0 v810) v1 ==== 158+0+0 (secure 0 0 0) 0x7fc1f8058ac0 con 0x7fc2000cf290 2024-01-16T15:19:17.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1d806c480 msgr2=0x7fc1d806e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1d806c480 0x7fc1d806e940 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fc1e8036000 tx=0x7fc1e8008030 comp rx=0 tx=0).stop 2024-01-16T15:19:17.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 msgr2=0x7fc2000cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 0x7fc2000cc8e0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800d5d0 tx=0x7fc1f800da90 comp rx=0 tx=0).stop 2024-01-16T15:19:17.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 shutdown_connections 2024-01-16T15:19:17.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc1d806c480 0x7fc1d806e940 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc2000cf290 0x7fc2000cc8e0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc2000b3ae0 0x7fc2001450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.523+0000 7fc204c00640 1 --2- 172.21.15.53:0/219424488 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc2000ae5a0 0x7fc200156c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.524+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 >> 172.21.15.53:0/219424488 conn(0x7fc20006a800 msgr2=0x7fc20015d820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.524+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 shutdown_connections 2024-01-16T15:19:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.524+0000 7fc204c00640 1 -- 172.21.15.53:0/219424488 wait complete. 2024-01-16T15:19:17.536 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:19:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.770+0000 7f4baa000640 1 Processor -- start 2024-01-16T15:19:17.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.771+0000 7f4baa000640 1 -- start start 2024-01-16T15:19:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.772+0000 7f4baa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4ba41447f0 0x7f4ba415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4baa000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ba40cccf0 0x7f4ba415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4baa000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ba415c5c0 con 0x7f4ba406c470 2024-01-16T15:19:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4baa000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba406d690 con 0x7f4ba40cccf0 2024-01-16T15:19:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4baa000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ba40cebb0 con 0x7f4ba41447f0 2024-01-16T15:19:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4ba9600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ba40cccf0 0x7f4ba415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4ba9600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4ba40cccf0 0x7f4ba415c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33792/0 (socket says 172.21.15.53:33792) 2024-01-16T15:19:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4ba9600640 1 -- 172.21.15.53:0/4067906892 learned_addr learned my addr 172.21.15.53:0/4067906892 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.773+0000 7f4ba8c00640 1 --2- 172.21.15.53:0/4067906892 >> v2:172.21.15.146:3300/0 conn(0x7f4ba41447f0 0x7f4ba415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.774+0000 7f4ba9600640 1 -- 172.21.15.53:0/4067906892 >> v2:172.21.15.146:3300/0 conn(0x7f4ba41447f0 msgr2=0x7f4ba415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:17.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.774+0000 7f4ba9600640 1 --2- 172.21.15.53:0/4067906892 >> v2:172.21.15.146:3300/0 conn(0x7f4ba41447f0 0x7f4ba415c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.774+0000 7f4ba9600640 1 -- 172.21.15.53:0/4067906892 >> v1:172.21.15.146:6789/0 conn(0x7f4ba406c470 legacy=0x7f4ba40cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:17.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.774+0000 7f4ba9600640 1 -- 172.21.15.53:0/4067906892 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba406ca10 con 0x7f4ba40cccf0 2024-01-16T15:19:17.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.775+0000 7f4ba9600640 1 --2- 172.21.15.53:0/4067906892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba40cccf0 0x7f4ba415c7c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4072520 tx=0x7f4b9c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38988b40eb5b4124 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:17.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.775+0000 7f4ba3400640 1 -- 172.21.15.53:0/4067906892 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c02ca80 con 0x7f4ba40cccf0 2024-01-16T15:19:17.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.775+0000 7f4ba3400640 1 -- 172.21.15.53:0/4067906892 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9c02cbe0 con 0x7f4ba40cccf0 2024-01-16T15:19:17.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.775+0000 7f4ba3400640 1 -- 172.21.15.53:0/4067906892 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0343f0 con 0x7f4ba40cccf0 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 -- 172.21.15.53:0/4067906892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba40cccf0 msgr2=0x7f4ba415c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 --2- 172.21.15.53:0/4067906892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba40cccf0 0x7f4ba415c7c0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4072520 tx=0x7f4b9c02bee0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 -- 172.21.15.53:0/4067906892 shutdown_connections 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 --2- 172.21.15.53:0/4067906892 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba40cccf0 0x7f4ba415c7c0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 --2- 172.21.15.53:0/4067906892 >> v2:172.21.15.146:3300/0 conn(0x7f4ba41447f0 0x7f4ba415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 -- 172.21.15.53:0/4067906892 >> 172.21.15.53:0/4067906892 conn(0x7f4ba406a3e0 msgr2=0x7f4ba406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 -- 172.21.15.53:0/4067906892 shutdown_connections 2024-01-16T15:19:17.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.776+0000 7f4baa000640 1 -- 172.21.15.53:0/4067906892 wait complete. 2024-01-16T15:19:17.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.778+0000 7f4baa000640 1 Processor -- start 2024-01-16T15:19:17.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.778+0000 7f4baa000640 1 -- start start 2024-01-16T15:19:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.779+0000 7f4baa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 0x7f4ba41f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.779+0000 7f4baa000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba40cccf0 0x7f4ba4140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.779+0000 7f4baa000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba41447f0 0x7f4ba40b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.779+0000 7f4baa000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba41f7820 con 0x7f4ba406c470 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.779+0000 7f4baa000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba41f7b00 con 0x7f4ba40cccf0 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.779+0000 7f4baa000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba40ae7d0 con 0x7f4ba41447f0 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 0x7f4ba41f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 0x7f4ba41f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33808/0 (socket says 172.21.15.53:33808) 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 -- 172.21.15.53:0/3268986095 learned_addr learned my addr 172.21.15.53:0/3268986095 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba3e00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba40cccf0 0x7f4ba4140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 -- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba41447f0 msgr2=0x7f4ba40b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba41447f0 0x7f4ba40b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 -- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba40cccf0 msgr2=0x7f4ba4140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba40cccf0 0x7f4ba4140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 -- 172.21.15.53:0/3268986095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c009b30 con 0x7f4ba406c470 2024-01-16T15:19:17.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.780+0000 7f4ba8c00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 0x7f4ba41f6f60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400d800 tx=0x7f4b9400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:17.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.781+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400b6c0 con 0x7f4ba406c470 2024-01-16T15:19:17.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.781+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b9400b820 con 0x7f4ba406c470 2024-01-16T15:19:17.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.781+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba40aeab0 con 0x7f4ba406c470 2024-01-16T15:19:17.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.781+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b9400bbb0 con 0x7f4ba406c470 2024-01-16T15:19:17.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.781+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba4141020 con 0x7f4ba406c470 2024-01-16T15:19:17.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.782+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba4067450 con 0x7f4ba406c470 2024-01-16T15:19:17.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.783+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4b9400bd10 con 0x7f4ba406c470 2024-01-16T15:19:17.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.784+0000 7f4ba0c00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7c06c460 0x7f4b7c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:17.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.784+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f4b94088fa0 con 0x7f4ba406c470 2024-01-16T15:19:17.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.785+0000 7f4ba3e00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7c06c460 0x7f4b7c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:17.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.786+0000 7f4ba3e00640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7c06c460 0x7f4b7c06e920 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4ba415e650 tx=0x7f4b8c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:17.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.788+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4b94055570 con 0x7f4ba406c470 2024-01-16T15:19:17.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.980+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4ba4142aa0 con 0x7f4ba406c470 2024-01-16T15:19:17.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.987+0000 7f4ba0c00640 1 -- 172.21.15.53:0/3268986095 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4b94058b90 con 0x7f4ba406c470 2024-01-16T15:19:17.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.989+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7c06c460 msgr2=0x7f4b7c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.989+0000 7f4baa000640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7c06c460 0x7f4b7c06e920 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f4ba415e650 tx=0x7f4b8c009040 comp rx=0 tx=0).stop 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.989+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 msgr2=0x7f4ba41f6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.989+0000 7f4baa000640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 0x7f4ba41f6f60 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f4b9400d800 tx=0x7f4b9400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 shutdown_connections 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4b7c06c460 0x7f4b7c06e920 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4ba41447f0 0x7f4ba40b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4ba40cccf0 0x7f4ba4140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 --2- 172.21.15.53:0/3268986095 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4ba406c470 0x7f4ba41f6f60 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 >> 172.21.15.53:0/3268986095 conn(0x7f4ba406a3e0 msgr2=0x7f4ba415cd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 shutdown_connections 2024-01-16T15:19:17.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:17.990+0000 7f4baa000640 1 -- 172.21.15.53:0/3268986095 wait complete. 2024-01-16T15:19:18.001 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-01-16T15:19:18.002 DEBUG:teuthology.orchestra.run.smithi053:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:19:18.120 DEBUG:teuthology.orchestra.run.smithi146:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-01-16T15:19:18.236 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-01-16T15:19:18.236 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:19:18.236 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:18.299 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:19:18.299 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:19:18.299 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:18.355 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:19:18.356 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:19:18.356 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:18.415 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:19:18.415 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:19:18.415 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:18.472 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:19:18.473 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-01-16T15:19:18.473 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:19:18.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.713+0000 7fbd2b200640 1 Processor -- start 2024-01-16T15:19:18.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.714+0000 7fbd2b200640 1 -- start start 2024-01-16T15:19:18.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.715+0000 7fbd2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd240cc500 0x7fbd240cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.715+0000 7fbd2b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd2406d160 0x7fbd2406d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd2b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd24070e50 con 0x7fbd24143c40 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd2b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd240723c0 con 0x7fbd2406d160 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24144020 con 0x7fbd240cc500 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd2a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fbd2406d160 0x7fbd2406d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd29e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fbd24143c40 0x7fbd24071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40664/0 (socket says 172.21.15.53:40664) 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd29e00640 1 -- 172.21.15.53:0/3911285915 learned_addr learned my addr 172.21.15.53:0/3911285915 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:18.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.716+0000 7fbd29400640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd240cc500 0x7fbd240cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:18.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd28a00640 1 -- 172.21.15.53:0/3911285915 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615827345 0 0) 0x7fbd24070e50 con 0x7fbd24143c40 2024-01-16T15:19:18.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd28a00640 1 -- 172.21.15.53:0/3911285915 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04003520 con 0x7fbd24143c40 2024-01-16T15:19:18.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd2a800640 1 -- 172.21.15.53:0/3911285915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd240cc500 msgr2=0x7fbd240cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:18.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd2a800640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd240cc500 0x7fbd240cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd2a800640 1 -- 172.21.15.53:0/3911285915 >> v1:172.21.15.53:6789/0 conn(0x7fbd24143c40 legacy=0x7fbd24071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd2a800640 1 -- 172.21.15.53:0/3911285915 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd240ccea0 con 0x7fbd2406d160 2024-01-16T15:19:18.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.717+0000 7fbd29400640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd240cc500 0x7fbd240cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:18.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.718+0000 7fbd2a800640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2406d160 0x7fbd2406d590 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009d90 tx=0x7fbd1c02be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b29b0adaf7679d0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:18.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd28a00640 1 -- 172.21.15.53:0/3911285915 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c02cbc0 con 0x7fbd2406d160 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd28a00640 1 -- 172.21.15.53:0/3911285915 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd1c02cd20 con 0x7fbd2406d160 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd28a00640 1 -- 172.21.15.53:0/3911285915 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c034660 con 0x7fbd2406d160 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 -- 172.21.15.53:0/3911285915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2406d160 msgr2=0x7fbd2406d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2406d160 0x7fbd2406d590 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c009d90 tx=0x7fbd1c02be10 comp rx=0 tx=0).stop 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 -- 172.21.15.53:0/3911285915 shutdown_connections 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd2406d160 0x7fbd2406d590 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 --2- 172.21.15.53:0/3911285915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd240cc500 0x7fbd240cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 -- 172.21.15.53:0/3911285915 >> 172.21.15.53:0/3911285915 conn(0x7fbd2406a800 msgr2=0x7fbd2406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 -- 172.21.15.53:0/3911285915 shutdown_connections 2024-01-16T15:19:18.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.719+0000 7fbd2b200640 1 -- 172.21.15.53:0/3911285915 wait complete. 2024-01-16T15:19:18.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.721+0000 7fbd2b200640 1 Processor -- start 2024-01-16T15:19:18.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.721+0000 7fbd2b200640 1 -- start start 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.722+0000 7fbd2b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2406d160 0x7fbd2415a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.722+0000 7fbd2b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 0x7fbd240c9860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.722+0000 7fbd2b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd24143c40 0x7fbd240c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.722+0000 7fbd2b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd24133cb0 con 0x7fbd240cc500 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.722+0000 7fbd2b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd241f2cb0 con 0x7fbd24143c40 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.722+0000 7fbd2b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd2415aa70 con 0x7fbd2406d160 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 0x7fbd240c9860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 0x7fbd240c9860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33818/0 (socket says 172.21.15.53:33818) 2024-01-16T15:19:18.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 -- 172.21.15.53:0/4199678657 learned_addr learned my addr 172.21.15.53:0/4199678657 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29e00640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2406d160 0x7fbd2415a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd2a800640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd24143c40 0x7fbd240c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 -- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2406d160 msgr2=0x7fbd2415a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2406d160 0x7fbd2415a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 -- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd24143c40 msgr2=0x7fbd240c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd24143c40 0x7fbd240c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 -- 172.21.15.53:0/4199678657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c009a70 con 0x7fbd240cc500 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.723+0000 7fbd29400640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 0x7fbd240c9860 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00e730 tx=0x7fbd0c00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.724+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c004230 con 0x7fbd240cc500 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.724+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd0c004390 con 0x7fbd240cc500 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.724+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c00c8f0 con 0x7fbd240cc500 2024-01-16T15:19:18.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.724+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd241336f0 con 0x7fbd240cc500 2024-01-16T15:19:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.724+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd241f2ec0 con 0x7fbd240cc500 2024-01-16T15:19:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.725+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd24067450 con 0x7fbd240cc500 2024-01-16T15:19:18.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.726+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fbd0c00ca50 con 0x7fbd240cc500 2024-01-16T15:19:18.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.727+0000 7fbd22000640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06c430 0x7fbcfc06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:18.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.727+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7fbd0c0886c0 con 0x7fbd240cc500 2024-01-16T15:19:18.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.728+0000 7fbd29e00640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06c430 0x7fbcfc06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:18.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.729+0000 7fbd29e00640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06c430 0x7fbcfc06e8f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbd14005880 tx=0x7fbd14009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:18.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.732+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fbd0c054c30 con 0x7fbd240cc500 2024-01-16T15:19:18.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.965+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd240ceb30 con 0x7fbd240cc500 2024-01-16T15:19:18.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.966+0000 7fbd22000640 1 -- 172.21.15.53:0/4199678657 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fbd0c020020 con 0x7fbd240cc500 2024-01-16T15:19:18.958 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:18.958 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":67,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6835/1983839723","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":1983839723},{"type":"v1","addr":"172.21.15.146:6835","nonce":1983839723}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59},{"gid":7048,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1926242315","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6836","nonce":1926242315},{"type":"v1","addr":"172.21.15.146:6837","nonce":1926242315}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":7054,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/2218545626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":2218545626},{"type":"v1","addr":"172.21.15.53:6835","nonce":2218545626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":60},{"gid":8916,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6834/3613190207","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3613190207},{"type":"v1","addr":"172.21.15.53:6834","nonce":3613190207}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":67}],"filesystems":[]} 2024-01-16T15:19:18.958 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 67 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06c430 msgr2=0x7fbcfc06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06c430 0x7fbcfc06e8f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fbd14005880 tx=0x7fbd14009040 comp rx=0 tx=0).stop 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 msgr2=0x7fbd240c9860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 0x7fbd240c9860 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00e730 tx=0x7fbd0c00ebf0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 shutdown_connections 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fbcfc06c430 0x7fbcfc06e8f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fbd24143c40 0x7fbd240c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fbd240cc500 0x7fbd240c9860 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 --2- 172.21.15.53:0/4199678657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fbd2406d160 0x7fbd2415a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 >> 172.21.15.53:0/4199678657 conn(0x7fbd2406a800 msgr2=0x7fbd2406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 shutdown_connections 2024-01-16T15:19:18.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:18.969+0000 7fbd2b200640 1 -- 172.21.15.53:0/4199678657 wait complete. 2024-01-16T15:19:18.979 INFO:tasks.ceph.mds.a:Restarting daemon 2024-01-16T15:19:18.979 INFO:tasks.ceph.mds.a:Stopping old one... 2024-01-16T15:19:18.979 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:19:18.979 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:19:18.979 INFO:tasks.ceph.mds.c:Restarting daemon 2024-01-16T15:19:18.980 INFO:tasks.ceph.mds.c:Stopping old one... 2024-01-16T15:19:18.980 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:19:18.980 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:19:18.980 INFO:tasks.ceph.mds.b:Restarting daemon 2024-01-16T15:19:18.980 INFO:tasks.ceph.mds.b:Stopping old one... 2024-01-16T15:19:18.980 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:19:18.980 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:19:18.981 INFO:tasks.ceph.mds.d:Restarting daemon 2024-01-16T15:19:18.981 INFO:tasks.ceph.mds.d:Stopping old one... 2024-01-16T15:19:18.981 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:19:18.981 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:19:18.998 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:19:18.998 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-01-16T15:19:19.003 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:19:19.004 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-01-16T15:19:19.006 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:19:19.006 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-01-16T15:19:19.009 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:19:19.009 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-01-16T15:19:19.054 INFO:tasks.ceph.mds.b:Started 2024-01-16T15:19:19.054 DEBUG:teuthology.parallel:result is None 2024-01-16T15:19:19.056 INFO:tasks.ceph.mds.c:Started 2024-01-16T15:19:19.057 DEBUG:teuthology.parallel:result is None 2024-01-16T15:19:19.060 INFO:tasks.ceph.mds.d:Started 2024-01-16T15:19:19.060 DEBUG:teuthology.parallel:result is None 2024-01-16T15:19:19.063 INFO:tasks.ceph.mds.a:Started 2024-01-16T15:19:19.063 DEBUG:teuthology.parallel:result is None 2024-01-16T15:19:19.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-01-16T15:19:19.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2024-01-16T15:19:19.208 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d is failed for ~0s 2024-01-16T15:19:19.213 INFO:tasks.ceph.mds.d.smithi146.stdout:starting mds.d at 2024-01-16T15:19:19.214 INFO:tasks.ceph.mds.b.smithi146.stdout:starting mds.b at 2024-01-16T15:19:19.220 INFO:tasks.ceph.mds.a.smithi053.stdout:starting mds.a at 2024-01-16T15:19:19.221 INFO:tasks.ceph.mds.c.smithi053.stdout:starting mds.c at 2024-01-16T15:19:19.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.271+0000 7f1cb6000640 1 Processor -- start 2024-01-16T15:19:19.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.272+0000 7f1cb6000640 1 -- start start 2024-01-16T15:19:19.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.273+0000 7f1cb6000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1cb013df80 0x7f1cb015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.273+0000 7f1cb6000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb006c470 con 0x7f1cb00cf9a0 2024-01-16T15:19:19.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.273+0000 7f1cb6000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb006c750 con 0x7f1cb013e360 2024-01-16T15:19:19.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.273+0000 7f1cb6000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb006ca30 con 0x7f1cb013df80 2024-01-16T15:19:19.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.273+0000 7f1cafe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1cb00cf9a0 0x7f1cb00cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40676/0 (socket says 172.21.15.53:40676) 2024-01-16T15:19:19.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.273+0000 7f1cafe00640 1 -- 172.21.15.53:0/2976578189 learned_addr learned my addr 172.21.15.53:0/2976578189 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caf400640 1 --2- 172.21.15.53:0/2976578189 >> v2:172.21.15.53:3300/0 conn(0x7f1cb013df80 0x7f1cb015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096643588 0 0) 0x7f1cb006c470 con 0x7f1cb00cf9a0 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c90003520 con 0x7f1cb00cf9a0 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812967465 0 0) 0x7f1cb006c750 con 0x7f1cb013e360 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cb006c470 con 0x7f1cb013e360 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caf400640 1 -- 172.21.15.53:0/2976578189 >> v1:172.21.15.146:6789/0 conn(0x7f1cb013e360 legacy=0x7f1cb00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caf400640 1 -- 172.21.15.53:0/2976578189 >> v1:172.21.15.53:6789/0 conn(0x7f1cb00cf9a0 legacy=0x7f1cb00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259995065 0 0) 0x7f1c90003520 con 0x7f1cb00cf9a0 2024-01-16T15:19:19.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.274+0000 7f1caf400640 1 -- 172.21.15.53:0/2976578189 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb006cd10 con 0x7f1cb013df80 2024-01-16T15:19:19.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.275+0000 7f1caf400640 1 --2- 172.21.15.53:0/2976578189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb015bee0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00097a0 tx=0x7f1ca002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e05b4d1112d7b22f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:19.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca002f050 con 0x7f1cb013df80 2024-01-16T15:19:19.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0031500 con 0x7f1cb013df80 2024-01-16T15:19:19.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1caea00640 1 -- 172.21.15.53:0/2976578189 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca80019f0 con 0x7f1cb013df80 2024-01-16T15:19:19.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1cb6000640 1 -- 172.21.15.53:0/2976578189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 msgr2=0x7f1cb015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1cb6000640 1 --2- 172.21.15.53:0/2976578189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb015bee0 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f1ca00097a0 tx=0x7f1ca002b820 comp rx=0 tx=0).stop 2024-01-16T15:19:19.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1cb6000640 1 -- 172.21.15.53:0/2976578189 shutdown_connections 2024-01-16T15:19:19.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1cb6000640 1 --2- 172.21.15.53:0/2976578189 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb015bee0 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1cb6000640 1 -- 172.21.15.53:0/2976578189 >> 172.21.15.53:0/2976578189 conn(0x7f1cb006a800 msgr2=0x7f1cb006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:19.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.276+0000 7f1cb6000640 1 -- 172.21.15.53:0/2976578189 shutdown_connections 2024-01-16T15:19:19.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.277+0000 7f1cb6000640 1 -- 172.21.15.53:0/2976578189 wait complete. 2024-01-16T15:19:19.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.278+0000 7f1cb6000640 1 Processor -- start 2024-01-16T15:19:19.270 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.278+0000 7f1cb6000640 1 -- start start 2024-01-16T15:19:19.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.279+0000 7f1cb6000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb00cf9a0 0x7f1cb0150560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.279+0000 7f1cb6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb00b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.279+0000 7f1cb6000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb013e360 0x7f1cb00b37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.279+0000 7f1cb6000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb012bbd0 con 0x7f1cb013df80 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.279+0000 7f1cb6000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb012c060 con 0x7f1cb00cf9a0 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.279+0000 7f1cb6000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb012afc0 con 0x7f1cb013e360 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb00b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb00b3290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33866/0 (socket says 172.21.15.53:33866) 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 -- 172.21.15.53:0/9495273 learned_addr learned my addr 172.21.15.53:0/9495273 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1caf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb00cf9a0 0x7f1cb0150560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1cafe00640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb013e360 0x7f1cb00b37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 -- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb013e360 msgr2=0x7f1cb00b37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb013e360 0x7f1cb00b37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 -- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb00cf9a0 msgr2=0x7f1cb0150560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.272 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb00cf9a0 0x7f1cb0150560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 -- 172.21.15.53:0/9495273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0009400 con 0x7f1cb013df80 2024-01-16T15:19:19.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1cafe00640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb013e360 0x7f1cb00b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:19.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.280+0000 7f1ca6800640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb00b3290 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c450 tx=0x7f1c9800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:19.273 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.281+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c980052d0 con 0x7f1cb013df80 2024-01-16T15:19:19.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.281+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c98005430 con 0x7f1cb013df80 2024-01-16T15:19:19.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.281+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c98012870 con 0x7f1cb013df80 2024-01-16T15:19:19.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.281+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb012b2a0 con 0x7f1cb013df80 2024-01-16T15:19:19.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.281+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb012b860 con 0x7f1cb013df80 2024-01-16T15:19:19.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.282+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cb00675d0 con 0x7f1cb013df80 2024-01-16T15:19:19.274 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.283+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f1c980129d0 con 0x7f1cb013df80 2024-01-16T15:19:19.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.284+0000 7f1ca7e00640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8806c460 0x7f1c8806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.284+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 4971+0+0 (secure 0 0 0) 0x7f1c98088560 con 0x7f1cb013df80 2024-01-16T15:19:19.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.285+0000 7f1caf400640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8806c460 0x7f1c8806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.286+0000 7f1caf400640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8806c460 0x7f1c8806e920 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0035000 tx=0x7f1ca0008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:19.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.288+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c98054b90 con 0x7f1cb013df80 2024-01-16T15:19:19.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.495+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1cb006df00 con 0x7f1cb013df80 2024-01-16T15:19:19.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.542+0000 7f1ca7e00640 1 -- 172.21.15.53:0/9495273 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v104) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1c980581b0 con 0x7f1cb013df80 2024-01-16T15:19:19.534 INFO:teuthology.orchestra.run.smithi053.stderr: removed all blocklist entries 2024-01-16T15:19:19.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.545+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8806c460 msgr2=0x7f1c8806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.545+0000 7f1cb6000640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8806c460 0x7f1c8806e920 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1ca0035000 tx=0x7f1ca0008030 comp rx=0 tx=0).stop 2024-01-16T15:19:19.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.545+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 msgr2=0x7f1cb00b3290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.545+0000 7f1cb6000640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb00b3290 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f1c9800c450 tx=0x7f1c9800c910 comp rx=0 tx=0).stop 2024-01-16T15:19:19.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 shutdown_connections 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c8806c460 0x7f1c8806e920 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1cb013e360 0x7f1cb00b37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1cb013df80 0x7f1cb00b3290 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 --2- 172.21.15.53:0/9495273 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1cb00cf9a0 0x7f1cb0150560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 >> 172.21.15.53:0/9495273 conn(0x7f1cb006a800 msgr2=0x7f1cb00b01a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 shutdown_connections 2024-01-16T15:19:19.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.546+0000 7f1cb6000640 1 -- 172.21.15.53:0/9495273 wait complete. 2024-01-16T15:19:19.558 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-01-16T15:19:19.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.784+0000 7f2494600640 1 Processor -- start 2024-01-16T15:19:19.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.784+0000 7f2494600640 1 -- start start 2024-01-16T15:19:19.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.786+0000 7f2494600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f248c13df80 0x7f248c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.786+0000 7f2494600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c06c470 con 0x7f248c0cf9a0 2024-01-16T15:19:19.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.786+0000 7f2494600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f248c06c750 con 0x7f248c13e360 2024-01-16T15:19:19.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.786+0000 7f2494600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f248c06ca30 con 0x7f248c13df80 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.786+0000 7f2492600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f248c0cf9a0 0x7f248c0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40690/0 (socket says 172.21.15.53:40690) 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.786+0000 7f2492600640 1 -- 172.21.15.53:0/848006059 learned_addr learned my addr 172.21.15.53:0/848006059 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f2491c00640 1 --2- 172.21.15.53:0/848006059 >> v2:172.21.15.146:3300/0 conn(0x7f248c13df80 0x7f248c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202507523 0 0) 0x7f248c06c470 con 0x7f248c0cf9a0 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f246c003520 con 0x7f248c0cf9a0 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3194018181 0 0) 0x7f246c003520 con 0x7f248c0cf9a0 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 >> v2:172.21.15.146:3300/0 conn(0x7f248c13df80 msgr2=0x7f248c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 --2- 172.21.15.53:0/848006059 >> v2:172.21.15.146:3300/0 conn(0x7f248c13df80 0x7f248c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 >> v1:172.21.15.146:6789/0 conn(0x7f248c13e360 legacy=0x7f248c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c06cd10 con 0x7f248c0cf9a0 2024-01-16T15:19:19.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2484001670 con 0x7f248c0cf9a0 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 >> v1:172.21.15.53:6789/0 conn(0x7f248c0cf9a0 legacy=0x7f248c0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246c0039c0 0x7f246c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.787+0000 7f248be00640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f246c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f248be00640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246c005180 0x7f246c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c06ca30 con 0x7f246c0047c0 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c00c080 con 0x7f246c0039c0 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c00c240 con 0x7f246c005180 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491200640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f246c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491c00640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246c0039c0 0x7f246c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491200640 1 -- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246c005180 msgr2=0x7f246c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491200640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f246c005180 0x7f246c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491200640 1 -- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246c0039c0 msgr2=0x7f246c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491200640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f246c0039c0 0x7f246c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2491200640 1 -- 172.21.15.53:0/848006059 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c003520 con 0x7f246c0047c0 2024-01-16T15:19:19.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.788+0000 7f2492600640 1 -- 172.21.15.53:0/848006059 reap_dead start 2024-01-16T15:19:19.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.789+0000 7f2491200640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f246c004c40 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f2474001720 tx=0x7f246c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4f260c8f5bc094 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.790+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24740052d0 con 0x7f246c0047c0 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.790+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247402e050 con 0x7f246c0047c0 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.790+0000 7f248be00640 1 -- 172.21.15.53:0/848006059 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2474003830 con 0x7f246c0047c0 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.790+0000 7f2494600640 1 -- 172.21.15.53:0/848006059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 msgr2=0x7f246c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.790+0000 7f2494600640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f246c004c40 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f2474001720 tx=0x7f246c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.791+0000 7f2494600640 1 -- 172.21.15.53:0/848006059 shutdown_connections 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.791+0000 7f2494600640 1 --2- 172.21.15.53:0/848006059 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f246c004c40 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.791+0000 7f2494600640 1 -- 172.21.15.53:0/848006059 >> 172.21.15.53:0/848006059 conn(0x7f248c06a800 msgr2=0x7f248c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.791+0000 7f2494600640 1 -- 172.21.15.53:0/848006059 shutdown_connections 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.791+0000 7f2494600640 1 -- 172.21.15.53:0/848006059 wait complete. 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.792+0000 7f2494600640 1 Processor -- start 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.793+0000 7f2494600640 1 -- start start 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.793+0000 7f2494600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f248c070dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2494600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f248c071310 0x7f248c0b2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2494600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f248c0b2e70 0x7f248c0b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2494600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c153cc0 con 0x7f246c0047c0 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2494600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248c071b70 con 0x7f248c0b2e70 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2494600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248c153f10 con 0x7f248c071310 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f248c070dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f248c070dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33878/0 (socket says 172.21.15.53:33878) 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 -- 172.21.15.53:0/4129234636 learned_addr learned my addr 172.21.15.53:0/4129234636 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2492600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f248c0b2e70 0x7f248c0b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491200640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f248c071310 0x7f248c0b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 -- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f248c071310 msgr2=0x7f248c0b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f248c071310 0x7f248c0b2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 -- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f248c0b2e70 msgr2=0x7f248c0b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:19.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f248c0b2e70 0x7f248c0b32f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491c00640 1 -- 172.21.15.53:0/4129234636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c1541f0 con 0x7f246c0047c0 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.794+0000 7f2491200640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f248c071310 0x7f248c0b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.795+0000 7f2491c00640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f248c070dd0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f247c007f30 tx=0x7f247c011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.795+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c00e030 con 0x7f246c0047c0 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.795+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f247c016470 con 0x7f246c0047c0 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.795+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f247c016820 con 0x7f246c0047c0 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.795+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f248c1544d0 con 0x7f246c0047c0 2024-01-16T15:19:19.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.795+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f248c1f32a0 con 0x7f246c0047c0 2024-01-16T15:19:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.796+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f248c154660 con 0x7f246c0047c0 2024-01-16T15:19:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.798+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f247c016980 con 0x7f246c0047c0 2024-01-16T15:19:19.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.799+0000 7f2489600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f246406c430 0x7f246406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:19.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.799+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f247c00f070 con 0x7f246c0047c0 2024-01-16T15:19:19.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.800+0000 7f2491200640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f246406c430 0x7f246406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:19.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.801+0000 7f2491200640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f246406c430 0x7f246406e8f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2474033000 tx=0x7f2474009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:19.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:19.804+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f247c056060 con 0x7f246c0047c0 2024-01-16T15:19:20.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.069+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f248c0675d0 con 0x7f246c0047c0 2024-01-16T15:19:20.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.070+0000 7f2489600640 1 -- 172.21.15.53:0/4129234636 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f247c059680 con 0x7f246c0047c0 2024-01-16T15:19:20.062 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:20.062 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:19:20.062 INFO:teuthology.orchestra.run.smithi053.stdout: "auth_dump": [ 2024-01-16T15:19:20.062 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.062 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.a", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlmutVGxAAN4xdX/IG1NyYp3/c8l27IQ==", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.b", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZl4c4dAxAAXGc31epWMxYXZlq8yh9N8w==", 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.063 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.c", 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlwTANLRAAYC6tvYUfOuNlISVcWzP8JA==", 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.064 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mds.d", 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlH/cvExAAECNLNJVB/L7otxyr1D+suA==", 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.065 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.0", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCxmaZluHOTBRAA9kkHY4QW3BtJhwTiyoGYhg==", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.066 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.1", 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC0maZlx53EHxAA4sPXG4kyKJnqDTkjbWUtHA==", 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.067 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.2", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC3maZllDLcOBAAZgB3oCjjji2bWPKUBL+7Lg==", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.3", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQC7maZlh5R+BxAAWwHUBys2FxuXh/xJga7Dtw==", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.068 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.4", 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDCmaZlqZPnFhAANPJ6Lhjuzywo30LNWDBqBA==", 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.069 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.5", 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDFmaZlrU3TLxAAcS7UzsUYrMDqDKpJY65ecg==", 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.6", 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDImaZlivuDOhAAeyqDp5QHJ9PLrpHSN+rZKg==", 2024-01-16T15:19:20.070 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "osd.7", 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQDMmaZlteJcCBAACG0sFgIo4yEHzD4VD7UMbQ==", 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.071 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow profile osd", 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile osd", 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.0", 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ==", 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow r", 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.072 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.1", 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlWNtoOBAAND1Ov5BjVHzdp2PiJpBvzA==", 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.073 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.2", 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCumaZlbmP1LBAANwRbriLTwuERN4QLaxU5bw==", 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.3", 2024-01-16T15:19:20.074 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCvmaZlGStiBxAAwMd4OXh6zop8DY8mPFqKWw==", 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow", 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow r", 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow rw", 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow rwx" 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "client.admin", 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlU6iWCRAAwYmNWAYM2g1Kv+F5kK0fXg==", 2024-01-16T15:19:20.075 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "mgr": "allow *", 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow *", 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.x", 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCtmaZlyrl+DhAAKBCxmfl2R5sHmCuGt6x5Ww==", 2024-01-16T15:19:20.076 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "entity": "mgr.y", 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "key": "AQCsmaZlpX+VOxAAX/y/2Qrzl6W9mHCTxHYldQ==", 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "caps": { 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "mds": "allow *", 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "mon": "allow profile mgr", 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: "osd": "allow *" 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:20.077 INFO:teuthology.orchestra.run.smithi053.stdout: ] 2024-01-16T15:19:20.078 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f246406c430 msgr2=0x7f246406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f246406c430 0x7f246406e8f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f2474033000 tx=0x7f2474009030 comp rx=0 tx=0).stop 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 msgr2=0x7f248c070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f248c070dd0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f247c007f30 tx=0x7f247c011660 comp rx=0 tx=0).stop 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 shutdown_connections 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f246406c430 0x7f246406e8f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f248c0b2e70 0x7f248c0b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f248c071310 0x7f248c0b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 --2- 172.21.15.53:0/4129234636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f246c0047c0 0x7f248c070dd0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 >> 172.21.15.53:0/4129234636 conn(0x7f248c06a800 msgr2=0x7f248c15e6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 shutdown_connections 2024-01-16T15:19:20.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.073+0000 7f2494600640 1 -- 172.21.15.53:0/4129234636 wait complete. 2024-01-16T15:19:20.085 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:20.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.317+0000 7f9931c00640 1 Processor -- start 2024-01-16T15:19:20.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.317+0000 7f9931c00640 1 -- start start 2024-01-16T15:19:20.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.319+0000 7f9931c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f992c13cfa0 0x7f992c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.319+0000 7f9931c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c0cdec0 con 0x7f992c13fcb0 2024-01-16T15:19:20.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.319+0000 7f9931c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992c0ce1a0 con 0x7f992c0cd530 2024-01-16T15:19:20.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.319+0000 7f9931c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992c0ce480 con 0x7f992c13cfa0 2024-01-16T15:19:20.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.319+0000 7f9931200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f992c13fcb0 0x7f992c0b2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40704/0 (socket says 172.21.15.53:40704) 2024-01-16T15:19:20.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.319+0000 7f9931200640 1 -- 172.21.15.53:0/1477860825 learned_addr learned my addr 172.21.15.53:0/1477860825 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992be00640 1 --2- 172.21.15.53:0/1477860825 >> v2:172.21.15.146:3300/0 conn(0x7f992c13cfa0 0x7f992c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2986352382 0 0) 0x7f992c0cdec0 con 0x7f992c13fcb0 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f990c003520 con 0x7f992c13fcb0 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3133352437 0 0) 0x7f990c003520 con 0x7f992c13fcb0 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 >> v2:172.21.15.146:3300/0 conn(0x7f992c13cfa0 msgr2=0x7f992c15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 --2- 172.21.15.53:0/1477860825 >> v2:172.21.15.146:3300/0 conn(0x7f992c13cfa0 0x7f992c15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 >> v1:172.21.15.146:6789/0 conn(0x7f992c0cd530 legacy=0x7f992c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c0ce760 con 0x7f992c13fcb0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f99240019f0 con 0x7f992c13fcb0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 >> v1:172.21.15.53:6789/0 conn(0x7f992c13fcb0 legacy=0x7f992c0b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990c0051d0 0x7f990c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c0ce480 con 0x7f990c0047c0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f990c00e330 con 0x7f990c0039c0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f990c00e4f0 con 0x7f990c0051d0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.320+0000 7f9923e00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f992be00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9923e00640 1 -- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990c0051d0 msgr2=0x7f990c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:20.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9931200640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990c0051d0 0x7f990c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9923e00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990c0051d0 0x7f990c00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9923e00640 1 -- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990c0039c0 msgr2=0x7f990c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9923e00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f990c0039c0 0x7f990c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9923e00640 1 -- 172.21.15.53:0/1477860825 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f990c003520 con 0x7f990c0047c0 2024-01-16T15:19:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9931200640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f990c0051d0 0x7f990c00db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:20.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.321+0000 7f9931200640 1 -- 172.21.15.53:0/1477860825 reap_dead start 2024-01-16T15:19:20.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.322+0000 7f9923e00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f9914001720 tx=0x7f990c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e14ecc37734ae624 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99140052d0 con 0x7f990c0047c0 2024-01-16T15:19:20.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991402e050 con 0x7f990c0047c0 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f992b400640 1 -- 172.21.15.53:0/1477860825 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9914003830 con 0x7f990c0047c0 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 -- 172.21.15.53:0/1477860825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 msgr2=0x7f990c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f9914001720 tx=0x7f990c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 -- 172.21.15.53:0/1477860825 shutdown_connections 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 --2- 172.21.15.53:0/1477860825 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f990c004c40 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 -- 172.21.15.53:0/1477860825 >> 172.21.15.53:0/1477860825 conn(0x7f992c06a800 msgr2=0x7f992c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 -- 172.21.15.53:0/1477860825 shutdown_connections 2024-01-16T15:19:20.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.323+0000 7f9931c00640 1 -- 172.21.15.53:0/1477860825 wait complete. 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.325+0000 7f9931c00640 1 Processor -- start 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.325+0000 7f9931c00640 1 -- start start 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.326+0000 7f9931c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f992c0ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.326+0000 7f9931c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f992c1e43d0 0x7f992c1f6d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.326+0000 7f9931c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f992c1e4850 0x7f992c1f7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.326+0000 7f9931c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c13e920 con 0x7f990c0047c0 2024-01-16T15:19:20.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.326+0000 7f9931c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f992c13ec00 con 0x7f992c1e43d0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.326+0000 7f9931c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992c13eee0 con 0x7f992c1e4850 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f992c0ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f992c0ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33898/0 (socket says 172.21.15.53:33898) 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 -- 172.21.15.53:0/595903946 learned_addr learned my addr 172.21.15.53:0/595903946 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f9923e00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f992c1e43d0 0x7f992c1f6d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f9931200640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f992c1e4850 0x7f992c1f7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 -- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f992c1e4850 msgr2=0x7f992c1f7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f992c1e4850 0x7f992c1f7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 -- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f992c1e43d0 msgr2=0x7f992c1f6d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f992c1e43d0 0x7f992c1f6d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.327+0000 7f992be00640 1 -- 172.21.15.53:0/595903946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992c13f1c0 con 0x7f990c0047c0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.328+0000 7f992be00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f992c0ad8b0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f991c0116d0 tx=0x7f991c011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.328+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c00f9c0 con 0x7f990c0047c0 2024-01-16T15:19:20.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.328+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c00fb20 con 0x7f990c0047c0 2024-01-16T15:19:20.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.328+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992c13f4a0 con 0x7f990c0047c0 2024-01-16T15:19:20.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.328+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c00fe90 con 0x7f990c0047c0 2024-01-16T15:19:20.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.328+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992c13f9c0 con 0x7f990c0047c0 2024-01-16T15:19:20.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.329+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992c067450 con 0x7f990c0047c0 2024-01-16T15:19:20.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.331+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f991c015460 con 0x7f990c0047c0 2024-01-16T15:19:20.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.332+0000 7f9928c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f990406c430 0x7f990406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.332+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f991c089330 con 0x7f990c0047c0 2024-01-16T15:19:20.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.332+0000 7f9923e00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f990406c430 0x7f990406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.334+0000 7f9923e00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f990406c430 0x7f990406e8f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9914033000 tx=0x7f9914009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.336+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f991c0558a0 con 0x7f990c0047c0 2024-01-16T15:19:20.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.532+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f992c0cbff0 con 0x7f990c0047c0 2024-01-16T15:19:20.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.533+0000 7f9928c00640 1 -- 172.21.15.53:0/595903946 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+12866 (secure 0 0 0) 0x7f991c058ec0 con 0x7f990c0047c0 2024-01-16T15:19:20.524 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:20.525 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":106,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:19.557262+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f990406c430 msgr2=0x7f990406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f990406c430 0x7f990406e8f0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9914033000 tx=0x7f9914009030 comp rx=0 tx=0).stop 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 msgr2=0x7f992c0ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f992c0ad8b0 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f991c0116d0 tx=0x7f991c011b90 comp rx=0 tx=0).stop 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 shutdown_connections 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f990406c430 0x7f990406e8f0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f992c1e4850 0x7f992c1f7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f992c1e43d0 0x7f992c1f6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.528 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 --2- 172.21.15.53:0/595903946 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f990c0047c0 0x7f992c0ad8b0 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 >> 172.21.15.53:0/595903946 conn(0x7f992c06a800 msgr2=0x7f992c06cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:20.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 shutdown_connections 2024-01-16T15:19:20.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.536+0000 7f9931c00640 1 -- 172.21.15.53:0/595903946 wait complete. 2024-01-16T15:19:20.549 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:19:20.549 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:19:20.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.769+0000 7fc42d400640 1 Processor -- start 2024-01-16T15:19:20.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.769+0000 7fc42d400640 1 -- start start 2024-01-16T15:19:20.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.770+0000 7fc42d400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc42814b2c0 0x7fc4280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc42d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4280cf7d0 0x7fc428070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc42d400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4280cfbf0 con 0x7fc428070ea0 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc42d400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc42814b6a0 con 0x7fc42814b2c0 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc42d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc428071690 con 0x7fc4280cf7d0 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc426800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc42814b2c0 0x7fc4280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc426800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc42814b2c0 0x7fc4280cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33902/0 (socket says 172.21.15.53:33902) 2024-01-16T15:19:20.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc426800640 1 -- 172.21.15.53:0/2288046559 learned_addr learned my addr 172.21.15.53:0/2288046559 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:20.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.771+0000 7fc425e00640 1 --2- 172.21.15.53:0/2288046559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4280cf7d0 0x7fc428070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.772+0000 7fc426800640 1 -- 172.21.15.53:0/2288046559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4280cf7d0 msgr2=0x7fc428070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:20.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.772+0000 7fc426800640 1 --2- 172.21.15.53:0/2288046559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4280cf7d0 0x7fc428070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.772+0000 7fc426800640 1 -- 172.21.15.53:0/2288046559 >> v1:172.21.15.146:6789/0 conn(0x7fc428070ea0 legacy=0x7fc4280b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:20.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.772+0000 7fc426800640 1 -- 172.21.15.53:0/2288046559 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428071970 con 0x7fc42814b2c0 2024-01-16T15:19:20.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.773+0000 7fc426800640 1 --2- 172.21.15.53:0/2288046559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280cf290 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc418009870 tx=0x7fc41802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbea7507cbacc0da server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc425400640 1 -- 172.21.15.53:0/2288046559 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418031640 con 0x7fc42814b2c0 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc425400640 1 -- 172.21.15.53:0/2288046559 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4180317a0 con 0x7fc42814b2c0 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc425400640 1 -- 172.21.15.53:0/2288046559 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418031b30 con 0x7fc42814b2c0 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc42d400640 1 -- 172.21.15.53:0/2288046559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 msgr2=0x7fc4280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc42d400640 1 --2- 172.21.15.53:0/2288046559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280cf290 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7fc418009870 tx=0x7fc41802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc42d400640 1 -- 172.21.15.53:0/2288046559 shutdown_connections 2024-01-16T15:19:20.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc42d400640 1 --2- 172.21.15.53:0/2288046559 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4280cf7d0 0x7fc428070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc42d400640 1 --2- 172.21.15.53:0/2288046559 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280cf290 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.774+0000 7fc42d400640 1 -- 172.21.15.53:0/2288046559 >> 172.21.15.53:0/2288046559 conn(0x7fc42806a800 msgr2=0x7fc42806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:20.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.775+0000 7fc42d400640 1 -- 172.21.15.53:0/2288046559 shutdown_connections 2024-01-16T15:19:20.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.775+0000 7fc42d400640 1 -- 172.21.15.53:0/2288046559 wait complete. 2024-01-16T15:19:20.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.776+0000 7fc42d400640 1 Processor -- start 2024-01-16T15:19:20.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.777+0000 7fc42d400640 1 -- start start 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc42d400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc428070ea0 0x7fc428143d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc42d400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4280cf7d0 0x7fc42815c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc42d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc42d400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc428163ef0 con 0x7fc42814b2c0 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc42d400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281632e0 con 0x7fc4280cf7d0 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc42d400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4281635c0 con 0x7fc428070ea0 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280ce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33906/0 (socket says 172.21.15.53:33906) 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 -- 172.21.15.53:0/4078547034 learned_addr learned my addr 172.21.15.53:0/4078547034 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc425e00640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4280cf7d0 0x7fc42815c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc426800640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc428070ea0 0x7fc428143d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 -- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc428070ea0 msgr2=0x7fc428143d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc428070ea0 0x7fc428143d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 -- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4280cf7d0 msgr2=0x7fc42815c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4280cf7d0 0x7fc42815c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.778+0000 7fc427200640 1 -- 172.21.15.53:0/4078547034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4180094d0 con 0x7fc42814b2c0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.779+0000 7fc427200640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280ce4a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc42000c780 tx=0x7fc42000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.779+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc420004b60 con 0x7fc42814b2c0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.779+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42000ce90 con 0x7fc42814b2c0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.779+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc42000a850 con 0x7fc42814b2c0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.779+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4281638a0 con 0x7fc42814b2c0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.779+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc428144280 con 0x7fc42814b2c0 2024-01-16T15:19:20.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.781+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4281640a0 con 0x7fc42814b2c0 2024-01-16T15:19:20.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.782+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fc42000a9b0 con 0x7fc42814b2c0 2024-01-16T15:19:20.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.783+0000 7fc41ea00640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc40006c430 0x7fc40006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:20.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.783+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fc420088c40 con 0x7fc42814b2c0 2024-01-16T15:19:20.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.784+0000 7fc426800640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc40006c430 0x7fc40006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:20.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.785+0000 7fc426800640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc40006c430 0x7fc40006e8f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc418036000 tx=0x7fc41803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:20.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.788+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4281640a0 con 0x7fc42814b2c0 2024-01-16T15:19:20.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.979+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4280675d0 con 0x7fc42814b2c0 2024-01-16T15:19:20.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.980+0000 7fc41ea00640 1 -- 172.21.15.53:0/4078547034 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4200551b0 con 0x7fc42814b2c0 2024-01-16T15:19:20.972 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:19:20.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.982+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc40006c430 msgr2=0x7fc40006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.982+0000 7fc42d400640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc40006c430 0x7fc40006e8f0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc418036000 tx=0x7fc41803c040 comp rx=0 tx=0).stop 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.982+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 msgr2=0x7fc4280ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.982+0000 7fc42d400640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280ce4a0 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7fc42000c780 tx=0x7fc42000cc40 comp rx=0 tx=0).stop 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 shutdown_connections 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc40006c430 0x7fc40006e8f0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc42814b2c0 0x7fc4280ce4a0 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4280cf7d0 0x7fc42815c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 --2- 172.21.15.53:0/4078547034 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc428070ea0 0x7fc428143d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 >> 172.21.15.53:0/4078547034 conn(0x7fc42806a800 msgr2=0x7fc42806ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 shutdown_connections 2024-01-16T15:19:20.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:20.983+0000 7fc42d400640 1 -- 172.21.15.53:0/4078547034 wait complete. 2024-01-16T15:19:20.994 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-01-16T15:19:21.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.220+0000 7f841e800640 1 Processor -- start 2024-01-16T15:19:21.211 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.220+0000 7f841e800640 1 -- start start 2024-01-16T15:19:21.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841e800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f84180b3c20 0x7f8418070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841e800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f841806d050 0x7f841806d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841e800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8418071790 con 0x7f8418159590 2024-01-16T15:19:21.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841e800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f841806d990 con 0x7f841806d050 2024-01-16T15:19:21.213 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841e800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8418071100 con 0x7f84180b3c20 2024-01-16T15:19:21.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f841806d050 0x7f841806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841de00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f841806d050 0x7f841806d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33914/0 (socket says 172.21.15.53:33914) 2024-01-16T15:19:21.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841de00640 1 -- 172.21.15.53:0/1461822880 learned_addr learned my addr 172.21.15.53:0/1461822880 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:21.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.222+0000 7f841ca00640 1 --2- 172.21.15.53:0/1461822880 >> v2:172.21.15.146:3300/0 conn(0x7f84180b3c20 0x7f8418070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.223+0000 7f841de00640 1 -- 172.21.15.53:0/1461822880 >> v2:172.21.15.146:3300/0 conn(0x7f84180b3c20 msgr2=0x7f8418070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:21.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.223+0000 7f841de00640 1 --2- 172.21.15.53:0/1461822880 >> v2:172.21.15.146:3300/0 conn(0x7f84180b3c20 0x7f8418070160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.223+0000 7f841de00640 1 -- 172.21.15.53:0/1461822880 >> v1:172.21.15.146:6789/0 conn(0x7f8418159590 legacy=0x7f84180ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:21.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.223+0000 7f841de00640 1 -- 172.21.15.53:0/1461822880 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418070860 con 0x7f841806d050 2024-01-16T15:19:21.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.224+0000 7f841de00640 1 --2- 172.21.15.53:0/1461822880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841806d050 0x7f841806d450 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f84180ce820 tx=0x7f841002e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b47e2fdabf0ba1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:21.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.224+0000 7f8417e00640 1 -- 172.21.15.53:0/1461822880 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841002ee00 con 0x7f841806d050 2024-01-16T15:19:21.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.224+0000 7f8417e00640 1 -- 172.21.15.53:0/1461822880 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8410036440 con 0x7f841806d050 2024-01-16T15:19:21.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.224+0000 7f8417e00640 1 -- 172.21.15.53:0/1461822880 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84100367d0 con 0x7f841806d050 2024-01-16T15:19:21.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 -- 172.21.15.53:0/1461822880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841806d050 msgr2=0x7f841806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 --2- 172.21.15.53:0/1461822880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841806d050 0x7f841806d450 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f84180ce820 tx=0x7f841002e050 comp rx=0 tx=0).stop 2024-01-16T15:19:21.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 -- 172.21.15.53:0/1461822880 shutdown_connections 2024-01-16T15:19:21.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 --2- 172.21.15.53:0/1461822880 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f841806d050 0x7f841806d450 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 --2- 172.21.15.53:0/1461822880 >> v2:172.21.15.146:3300/0 conn(0x7f84180b3c20 0x7f8418070160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 -- 172.21.15.53:0/1461822880 >> 172.21.15.53:0/1461822880 conn(0x7f841806a800 msgr2=0x7f841806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:21.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 -- 172.21.15.53:0/1461822880 shutdown_connections 2024-01-16T15:19:21.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.225+0000 7f841e800640 1 -- 172.21.15.53:0/1461822880 wait complete. 2024-01-16T15:19:21.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.227+0000 7f841e800640 1 Processor -- start 2024-01-16T15:19:21.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.227+0000 7f841e800640 1 -- start start 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.228+0000 7f841e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841806d050 0x7f8418142800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.228+0000 7f841e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84180b3c20 0x7f841815c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.228+0000 7f841e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 0x7f841813f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.228+0000 7f841e800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418152650 con 0x7f8418159590 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.228+0000 7f841e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418152930 con 0x7f841806d050 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.228+0000 7f841e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8418152c10 con 0x7f84180b3c20 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 0x7f841813f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 0x7f841813f890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33918/0 (socket says 172.21.15.53:33918) 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 -- 172.21.15.53:0/3031392658 learned_addr learned my addr 172.21.15.53:0/3031392658 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841d400640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841806d050 0x7f8418142800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841ca00640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84180b3c20 0x7f841815c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 -- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84180b3c20 msgr2=0x7f841815c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84180b3c20 0x7f841815c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 -- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841806d050 msgr2=0x7f8418142800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841806d050 0x7f8418142800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 -- 172.21.15.53:0/3031392658 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841000b770 con 0x7f8418159590 2024-01-16T15:19:21.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.229+0000 7f841de00640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 0x7f841813f890 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f841000b2c0 tx=0x7f84100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.230+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410034850 con 0x7f8418159590 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.230+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84100349b0 con 0x7f8418159590 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.230+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410034d60 con 0x7f8418159590 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.230+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8418152ef0 con 0x7f8418159590 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.230+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8418142d40 con 0x7f8418159590 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.231+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84180acbb0 con 0x7f8418159590 2024-01-16T15:19:21.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.232+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f84100403e0 con 0x7f8418159590 2024-01-16T15:19:21.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.233+0000 7f8415600640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83f006c480 0x7f83f006e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.233+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4885+0+0 (secure 0 0 0) 0x7f84100ab100 con 0x7f8418159590 2024-01-16T15:19:21.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.234+0000 7f841d400640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83f006c480 0x7f83f006e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.235+0000 7f841d400640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83f006c480 0x7f83f006e940 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8408009f20 tx=0x7f84080094d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:21.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.237+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f84180acbb0 con 0x7f8418159590 2024-01-16T15:19:21.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.434+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f84180acbb0 con 0x7f8418159590 2024-01-16T15:19:21.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.435+0000 7f8415600640 1 -- 172.21.15.53:0/3031392658 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v106) v1 ==== 84+0+65 (secure 0 0 0) 0x7f84180acbb0 con 0x7f8418159590 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout: "poolnum": 1, 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout: "poolname": ".mgr" 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:21.427 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:19:21.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83f006c480 msgr2=0x7f83f006e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83f006c480 0x7f83f006e940 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f8408009f20 tx=0x7f84080094d0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 msgr2=0x7f841813f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 0x7f841813f890 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f841000b2c0 tx=0x7f84100046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 shutdown_connections 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f83f006c480 0x7f83f006e940 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8418159590 0x7f841813f890 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f84180b3c20 0x7f841815c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 --2- 172.21.15.53:0/3031392658 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f841806d050 0x7f8418142800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 >> 172.21.15.53:0/3031392658 conn(0x7f841806a800 msgr2=0x7f84180ae950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:21.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 shutdown_connections 2024-01-16T15:19:21.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.438+0000 7f841e800640 1 -- 172.21.15.53:0/3031392658 wait complete. 2024-01-16T15:19:21.447 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-01-16T15:19:21.447 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-01-16T15:19:21.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.681+0000 7fd6bf200640 1 Processor -- start 2024-01-16T15:19:21.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.681+0000 7fd6bf200640 1 -- start start 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6bf200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6b815d610 0x7fd6b80ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6bf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b80ced60 0x7fd6b80d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6bf200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b80b3400 con 0x7fd6b80d1d20 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6bf200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6b80d1390 con 0x7fd6b815d610 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6bf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80d21b0 con 0x7fd6b80ced60 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6b7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6b815d610 0x7fd6b80ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6b7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd6b815d610 0x7fd6b80ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33926/0 (socket says 172.21.15.53:33926) 2024-01-16T15:19:21.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.683+0000 7fd6b7e00640 1 -- 172.21.15.53:0/3671668542 learned_addr learned my addr 172.21.15.53:0/3671668542 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:21.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.684+0000 7fd6afe00640 1 --2- 172.21.15.53:0/3671668542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b80ced60 0x7fd6b80d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.684+0000 7fd6b7e00640 1 -- 172.21.15.53:0/3671668542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b80ced60 msgr2=0x7fd6b80d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:21.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.684+0000 7fd6b7e00640 1 --2- 172.21.15.53:0/3671668542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b80ced60 0x7fd6b80d17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.684+0000 7fd6b7e00640 1 -- 172.21.15.53:0/3671668542 >> v1:172.21.15.53:6789/0 conn(0x7fd6b80d1d20 legacy=0x7fd6b80cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.684+0000 7fd6b7e00640 1 -- 172.21.15.53:0/3671668542 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b80b27f0 con 0x7fd6b815d610 2024-01-16T15:19:21.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.685+0000 7fd6b7e00640 1 --2- 172.21.15.53:0/3671668542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b815d610 0x7fd6b80ce820 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009870 tx=0x7fd6a802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d04a6c7c4b2fa6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:21.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.685+0000 7fd6b7400640 1 -- 172.21.15.53:0/3671668542 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8030840 con 0x7fd6b815d610 2024-01-16T15:19:21.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.685+0000 7fd6b7400640 1 -- 172.21.15.53:0/3671668542 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a80309a0 con 0x7fd6b815d610 2024-01-16T15:19:21.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.685+0000 7fd6b7400640 1 -- 172.21.15.53:0/3671668542 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8030d30 con 0x7fd6b815d610 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 -- 172.21.15.53:0/3671668542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b815d610 msgr2=0x7fd6b80ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 --2- 172.21.15.53:0/3671668542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b815d610 0x7fd6b80ce820 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009870 tx=0x7fd6a802b980 comp rx=0 tx=0).stop 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 -- 172.21.15.53:0/3671668542 shutdown_connections 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 --2- 172.21.15.53:0/3671668542 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b80ced60 0x7fd6b80d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 --2- 172.21.15.53:0/3671668542 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b815d610 0x7fd6b80ce820 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 -- 172.21.15.53:0/3671668542 >> 172.21.15.53:0/3671668542 conn(0x7fd6b809c540 msgr2=0x7fd6b80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 -- 172.21.15.53:0/3671668542 shutdown_connections 2024-01-16T15:19:21.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.686+0000 7fd6bf200640 1 -- 172.21.15.53:0/3671668542 wait complete. 2024-01-16T15:19:21.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.688+0000 7fd6bf200640 1 Processor -- start 2024-01-16T15:19:21.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.688+0000 7fd6bf200640 1 -- start start 2024-01-16T15:19:21.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6bf200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 0x7fd6b8160230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6bf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b80d1d20 0x7fd6b81f5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6bf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b815d610 0x7fd6b814c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6bf200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80b90c0 con 0x7fd6b80ced60 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6bf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b81f6450 con 0x7fd6b80d1d20 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6bf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b80ca6b0 con 0x7fd6b815d610 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.689+0000 7fd6b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 0x7fd6b8160230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 0x7fd6b8160230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33932/0 (socket says 172.21.15.53:33932) 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 -- 172.21.15.53:0/772276664 learned_addr learned my addr 172.21.15.53:0/772276664 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6afe00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b80d1d20 0x7fd6b81f5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6bd200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b815d610 0x7fd6b814c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 -- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b815d610 msgr2=0x7fd6b814c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b815d610 0x7fd6b814c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 -- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b80d1d20 msgr2=0x7fd6b81f5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b80d1d20 0x7fd6b81f5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 -- 172.21.15.53:0/772276664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a80094d0 con 0x7fd6b80ced60 2024-01-16T15:19:21.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6bd200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b815d610 0x7fd6b814c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:21.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.690+0000 7fd6b7e00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 0x7fd6b8160230 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009870 tx=0x7fd6a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:21.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.691+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a8039a90 con 0x7fd6b80ced60 2024-01-16T15:19:21.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.691+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a8039bf0 con 0x7fd6b80ced60 2024-01-16T15:19:21.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.691+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a80433f0 con 0x7fd6b80ced60 2024-01-16T15:19:21.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.691+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b80ca990 con 0x7fd6b80ced60 2024-01-16T15:19:21.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.691+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b815fa30 con 0x7fd6b80ced60 2024-01-16T15:19:21.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.692+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b80675d0 con 0x7fd6b80ced60 2024-01-16T15:19:21.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.693+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd6a8042390 con 0x7fd6b80ced60 2024-01-16T15:19:21.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.694+0000 7fd6b4c00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd69006c430 0x7fd69006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:21.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.695+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 4885+0+0 (secure 0 0 0) 0x7fd6a80aadb0 con 0x7fd6b80ced60 2024-01-16T15:19:21.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.695+0000 7fd6afe00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd69006c430 0x7fd69006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:21.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.696+0000 7fd6afe00640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd69006c430 0x7fd69006e8f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd6b815f680 tx=0x7fd6a0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:21.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.700+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd6a8077290 con 0x7fd6b80ced60 2024-01-16T15:19:21.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:21.893+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd6b80b13a0 con 0x7fd6b80ced60 2024-01-16T15:19:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.575+0000 7fd6b4c00640 1 -- 172.21.15.53:0/772276664 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v107) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd6a8048020 con 0x7fd6b80ced60 2024-01-16T15:19:22.567 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_metadata' created 2024-01-16T15:19:22.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd69006c430 msgr2=0x7fd69006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd69006c430 0x7fd69006e8f0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fd6b815f680 tx=0x7fd6a0008040 comp rx=0 tx=0).stop 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 msgr2=0x7fd6b8160230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 0x7fd6b8160230 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7fd6a8009870 tx=0x7fd6a8004730 comp rx=0 tx=0).stop 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 shutdown_connections 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd69006c430 0x7fd69006e8f0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd6b815d610 0x7fd6b814c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd6b80d1d20 0x7fd6b81f5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 --2- 172.21.15.53:0/772276664 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd6b80ced60 0x7fd6b8160230 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 >> 172.21.15.53:0/772276664 conn(0x7fd6b809c540 msgr2=0x7fd6b80d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 shutdown_connections 2024-01-16T15:19:22.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.579+0000 7fd6bf200640 1 -- 172.21.15.53:0/772276664 wait complete. 2024-01-16T15:19:22.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-01-16T15:19:22.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.817+0000 7faed3200640 1 Processor -- start 2024-01-16T15:19:22.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.817+0000 7faed3200640 1 -- start start 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed3200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faed4160920 0x7faed40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed40cfea0 0x7faed4161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed3200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed40d0a10 con 0x7faed40cf9f0 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed3200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40d0330 con 0x7faed4160920 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed40d1f50 con 0x7faed40cfea0 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faed4160920 0x7faed40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed1400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7faed40cf9f0 0x7faed4161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55680/0 (socket says 172.21.15.53:55680) 2024-01-16T15:19:22.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.819+0000 7faed1400640 1 -- 172.21.15.53:0/3757413182 learned_addr learned my addr 172.21.15.53:0/3757413182 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed2800640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed40cfea0 0x7faed4161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed1e00640 1 -- 172.21.15.53:0/3757413182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed40cfea0 msgr2=0x7faed4161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed1e00640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed40cfea0 0x7faed4161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed1e00640 1 -- 172.21.15.53:0/3757413182 >> v1:172.21.15.53:6789/0 conn(0x7faed40cf9f0 legacy=0x7faed4161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed1e00640 1 -- 172.21.15.53:0/3757413182 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed40cf080 con 0x7faed4160920 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed0a00640 1 -- 172.21.15.53:0/3757413182 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734728943 0 0) 0x7faed40d0a10 con 0x7faed40cf9f0 2024-01-16T15:19:22.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.820+0000 7faed2800640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed40cfea0 0x7faed4161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:22.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.821+0000 7faed1e00640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed4160920 0x7faed40b3f30 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7faec40097a0 tx=0x7faec402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434eb66e3c1c516f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:22.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.821+0000 7faed0a00640 1 -- 172.21.15.53:0/3757413182 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec402d040 con 0x7faed4160920 2024-01-16T15:19:22.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed0a00640 1 -- 172.21.15.53:0/3757413182 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec4031810 con 0x7faed4160920 2024-01-16T15:19:22.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed0a00640 1 -- 172.21.15.53:0/3757413182 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4031ba0 con 0x7faed4160920 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 -- 172.21.15.53:0/3757413182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed4160920 msgr2=0x7faed40b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed4160920 0x7faed40b3f30 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7faec40097a0 tx=0x7faec402b8b0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 -- 172.21.15.53:0/3757413182 shutdown_connections 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed40cfea0 0x7faed4161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 --2- 172.21.15.53:0/3757413182 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed4160920 0x7faed40b3f30 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 -- 172.21.15.53:0/3757413182 >> 172.21.15.53:0/3757413182 conn(0x7faed40d3090 msgr2=0x7faed40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:22.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 -- 172.21.15.53:0/3757413182 shutdown_connections 2024-01-16T15:19:22.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.822+0000 7faed3200640 1 -- 172.21.15.53:0/3757413182 wait complete. 2024-01-16T15:19:22.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.824+0000 7faed3200640 1 Processor -- start 2024-01-16T15:19:22.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.824+0000 7faed3200640 1 -- start start 2024-01-16T15:19:22.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.825+0000 7faed3200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 0x7faed41f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:22.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.825+0000 7faed3200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faed40cfea0 0x7faed4150e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:22.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed3200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed4160920 0x7faed41f75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed3200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed41513d0 con 0x7faed40cf9f0 2024-01-16T15:19:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed3200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed40ca370 con 0x7faed40cfea0 2024-01-16T15:19:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed3200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41f7b10 con 0x7faed4160920 2024-01-16T15:19:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 0x7faed41f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 0x7faed41f35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55764/0 (socket says 172.21.15.53:55764) 2024-01-16T15:19:22.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 -- 172.21.15.53:0/3636369461 learned_addr learned my addr 172.21.15.53:0/3636369461 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:22.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 -- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed4160920 msgr2=0x7faed41f75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:22.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed2800640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed4160920 0x7faed41f75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:22.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed4160920 0x7faed41f75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 -- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faed40cfea0 msgr2=0x7faed4150e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1400640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faed40cfea0 0x7faed4150e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faed40cfea0 0x7faed4150e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.826+0000 7faed1e00640 1 -- 172.21.15.53:0/3636369461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec4009400 con 0x7faed40cf9f0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.827+0000 7faed1e00640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 0x7faed41f35d0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7faec40098c0 tx=0x7faec40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.827+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4044400 con 0x7faed40cf9f0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.827+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec4044580 con 0x7faed40cf9f0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.827+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed41f3be0 con 0x7faed40cf9f0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.827+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faec4044930 con 0x7faed40cf9f0 2024-01-16T15:19:22.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.827+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed41f41a0 con 0x7faed40cf9f0 2024-01-16T15:19:22.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.828+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed4067450 con 0x7faed40cf9f0 2024-01-16T15:19:22.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.830+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7faec404d440 con 0x7faed40cf9f0 2024-01-16T15:19:22.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.831+0000 7faeca000640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faeac06c430 0x7faeac06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:22.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.831+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5255+0+0 (secure 0 0 0) 0x7faec40ab350 con 0x7faed40cf9f0 2024-01-16T15:19:22.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.832+0000 7faed1400640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faeac06c430 0x7faeac06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:22.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.833+0000 7faed1400640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faeac06c430 0x7faeac06e8f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faed40b7110 tx=0x7faebc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:22.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:22.836+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7faec4077a20 con 0x7faed40cf9f0 2024-01-16T15:19:23.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.031+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7faed40b3fc0 con 0x7faed40cf9f0 2024-01-16T15:19:23.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.591+0000 7faeca000640 1 -- 172.21.15.53:0/3636369461 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v108) v1 ==== 170+0+0 (secure 0 0 0) 0x7faec4035070 con 0x7faed40cf9f0 2024-01-16T15:19:23.583 INFO:teuthology.orchestra.run.smithi053.stderr:pool 'cephfs_data' created 2024-01-16T15:19:23.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faeac06c430 msgr2=0x7faeac06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:23.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faeac06c430 0x7faeac06e8f0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7faed40b7110 tx=0x7faebc009040 comp rx=0 tx=0).stop 2024-01-16T15:19:23.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 msgr2=0x7faed41f35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:23.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 0x7faed41f35d0 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7faec40098c0 tx=0x7faec40046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 shutdown_connections 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7faeac06c430 0x7faeac06e8f0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faed4160920 0x7faed41f75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faed40cfea0 0x7faed4150e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 --2- 172.21.15.53:0/3636369461 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faed40cf9f0 0x7faed41f35d0 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.596+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 >> 172.21.15.53:0/3636369461 conn(0x7faed40d3090 msgr2=0x7faed406bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.597+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 shutdown_connections 2024-01-16T15:19:23.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.598+0000 7faed3200640 1 -- 172.21.15.53:0/3636369461 wait complete. 2024-01-16T15:19:23.625 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-01-16T15:19:23.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.820+0000 7fb204c00640 1 Processor -- start 2024-01-16T15:19:23.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.820+0000 7fb204c00640 1 -- start start 2024-01-16T15:19:23.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb204c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb20013df80 0x7fb20015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb204c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2000cf9a0 0x7fb2000cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb204c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb20006c470 con 0x7fb20013e360 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb204c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb20006c750 con 0x7fb20013df80 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb204c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20006ca30 con 0x7fb2000cf9a0 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb1fde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb20013df80 0x7fb20015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb1fde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb20013df80 0x7fb20015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55770/0 (socket says 172.21.15.53:55770) 2024-01-16T15:19:23.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb1fde00640 1 -- 172.21.15.53:0/302077185 learned_addr learned my addr 172.21.15.53:0/302077185 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:23.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.822+0000 7fb1fe800640 1 --2- 172.21.15.53:0/302077185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2000cf9a0 0x7fb2000cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:23.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.823+0000 7fb1fde00640 1 -- 172.21.15.53:0/302077185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2000cf9a0 msgr2=0x7fb2000cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:23.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.823+0000 7fb1fde00640 1 --2- 172.21.15.53:0/302077185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2000cf9a0 0x7fb2000cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.823+0000 7fb1fde00640 1 -- 172.21.15.53:0/302077185 >> v1:172.21.15.146:6789/0 conn(0x7fb20013e360 legacy=0x7fb2000cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:23.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.823+0000 7fb1fde00640 1 -- 172.21.15.53:0/302077185 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20006cd10 con 0x7fb20013df80 2024-01-16T15:19:23.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.824+0000 7fb1fde00640 1 --2- 172.21.15.53:0/302077185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013df80 0x7fb20015bee0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009870 tx=0x7fb1f002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbd7e816486ae65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:23.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.824+0000 7fb1fd400640 1 -- 172.21.15.53:0/302077185 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0031640 con 0x7fb20013df80 2024-01-16T15:19:23.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.824+0000 7fb1fd400640 1 -- 172.21.15.53:0/302077185 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f00317a0 con 0x7fb20013df80 2024-01-16T15:19:23.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.824+0000 7fb1fd400640 1 -- 172.21.15.53:0/302077185 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f0031b30 con 0x7fb20013df80 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 -- 172.21.15.53:0/302077185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013df80 msgr2=0x7fb20015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 --2- 172.21.15.53:0/302077185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013df80 0x7fb20015bee0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0009870 tx=0x7fb1f002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 -- 172.21.15.53:0/302077185 shutdown_connections 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 --2- 172.21.15.53:0/302077185 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb2000cf9a0 0x7fb2000cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 --2- 172.21.15.53:0/302077185 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013df80 0x7fb20015bee0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 -- 172.21.15.53:0/302077185 >> 172.21.15.53:0/302077185 conn(0x7fb20006a800 msgr2=0x7fb20006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:23.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 -- 172.21.15.53:0/302077185 shutdown_connections 2024-01-16T15:19:23.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.825+0000 7fb204c00640 1 -- 172.21.15.53:0/302077185 wait complete. 2024-01-16T15:19:23.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.827+0000 7fb204c00640 1 Processor -- start 2024-01-16T15:19:23.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.827+0000 7fb204c00640 1 -- start start 2024-01-16T15:19:23.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb204c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2000cf9a0 0x7fb20013d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:23.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb204c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb20013df80 0x7fb200158680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:23.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb204c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 0x7fb2001f8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:23.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb204c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200157ae0 con 0x7fb20013e360 2024-01-16T15:19:23.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb204c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb200071240 con 0x7fb2000cf9a0 2024-01-16T15:19:23.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb204c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb20013d720 con 0x7fb20013df80 2024-01-16T15:19:23.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb1fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 0x7fb2001f8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:23.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.828+0000 7fb1fe800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 0x7fb2001f8310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55786/0 (socket says 172.21.15.53:55786) 2024-01-16T15:19:23.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 -- 172.21.15.53:0/404764200 learned_addr learned my addr 172.21.15.53:0/404764200 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:23.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1f7e00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb20013df80 0x7fb200158680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fde00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2000cf9a0 0x7fb20013d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 -- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb20013df80 msgr2=0x7fb200158680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb20013df80 0x7fb200158680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 -- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2000cf9a0 msgr2=0x7fb20013d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2000cf9a0 0x7fb20013d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 -- 172.21.15.53:0/404764200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f00094d0 con 0x7fb20013e360 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.829+0000 7fb1fe800640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 0x7fb2001f8310 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8007d80 tx=0x7fb1f800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.830+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8014420 con 0x7fb20013e360 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.830+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f8014580 con 0x7fb20013e360 2024-01-16T15:19:23.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.830+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8014930 con 0x7fb20013e360 2024-01-16T15:19:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.830+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb200070630 con 0x7fb20013e360 2024-01-16T15:19:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.830+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb200070bf0 con 0x7fb20013e360 2024-01-16T15:19:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.831+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2000713f0 con 0x7fb20013e360 2024-01-16T15:19:23.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.832+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fb1f8014a90 con 0x7fb20013e360 2024-01-16T15:19:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.834+0000 7fb1f6000640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1d806c430 0x7fb1d806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:23.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.834+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5637+0+0 (secure 0 0 0) 0x7fb1f8089290 con 0x7fb20013e360 2024-01-16T15:19:23.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.834+0000 7fb1fde00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1d806c430 0x7fb1d806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:23.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.835+0000 7fb1fde00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1d806c430 0x7fb1d806e8f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0036000 tx=0x7fb1f003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:23.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:23.839+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb2000713f0 con 0x7fb20013e360 2024-01-16T15:19:24.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.071+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fb2000713f0 con 0x7fb20013e360 2024-01-16T15:19:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.602+0000 7fb1f6000640 1 -- 172.21.15.53:0/404764200 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v70) v1 ==== 174+0+0 (secure 0 0 0) 0x7fb2000713f0 con 0x7fb20013e360 2024-01-16T15:19:24.595 INFO:teuthology.orchestra.run.smithi053.stderr:new fs with metadata pool 16 and data pool 17 2024-01-16T15:19:24.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.613+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1d806c430 msgr2=0x7fb1d806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.613+0000 7fb204c00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1d806c430 0x7fb1d806e8f0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fb1f0036000 tx=0x7fb1f003c040 comp rx=0 tx=0).stop 2024-01-16T15:19:24.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.613+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 msgr2=0x7fb2001f8310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.613+0000 7fb204c00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 0x7fb2001f8310 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fb1f8007d80 tx=0x7fb1f800e4b0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.615+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 shutdown_connections 2024-01-16T15:19:24.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.615+0000 7fb204c00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb1d806c430 0x7fb1d806e8f0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.615+0000 7fb204c00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb20013e360 0x7fb2001f8310 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.615+0000 7fb204c00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb20013df80 0x7fb200158680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.615+0000 7fb204c00640 1 --2- 172.21.15.53:0/404764200 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb2000cf9a0 0x7fb20013d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.615+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 >> 172.21.15.53:0/404764200 conn(0x7fb20006a800 msgr2=0x7fb20006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.616+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 shutdown_connections 2024-01-16T15:19:24.608 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.616+0000 7fb204c00640 1 -- 172.21.15.53:0/404764200 wait complete. 2024-01-16T15:19:24.643 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:24.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.832+0000 7f6faba00640 1 Processor -- start 2024-01-16T15:19:24.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.832+0000 7f6faba00640 1 -- start start 2024-01-16T15:19:24.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.834+0000 7f6faba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 0x7f6fa415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.834+0000 7f6faba00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40b1ee0 con 0x7f6fa415e1e0 2024-01-16T15:19:24.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.834+0000 7f6faba00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fa40b2780 con 0x7f6fa4143ae0 2024-01-16T15:19:24.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.834+0000 7f6faba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa415e990 con 0x7f6fa40cf9a0 2024-01-16T15:19:24.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.834+0000 7f6fa9a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6fa415e1e0 0x7f6fa40702c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55688/0 (socket says 172.21.15.53:55688) 2024-01-16T15:19:24.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.834+0000 7f6fa9a00640 1 -- 172.21.15.53:0/2250406380 learned_addr learned my addr 172.21.15.53:0/2250406380 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:24.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6f9bc00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 0x7f6fa415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 591972821 0 0) 0x7f6fa40b1ee0 con 0x7f6fa415e1e0 2024-01-16T15:19:24.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f84005730 con 0x7f6fa415e1e0 2024-01-16T15:19:24.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1333217798 0 0) 0x7f6f84005730 con 0x7f6fa415e1e0 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 msgr2=0x7f6fa415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 0x7f6fa415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 >> v1:172.21.15.146:6789/0 conn(0x7f6fa4143ae0 legacy=0x7f6fa40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa40b2130 con 0x7f6fa415e1e0 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6f9c001670 con 0x7f6fa415e1e0 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 >> v1:172.21.15.53:6789/0 conn(0x7f6fa415e1e0 legacy=0x7f6fa40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f84005bb0 0x7f6f84005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840069d0 0x7f6f84006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.835+0000 7f6fa3e00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 0x7f6f84007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa415e990 con 0x7f6f84005bb0 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8400dc70 con 0x7f6f840069d0 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8400de30 con 0x7f6fa40cf9a0 2024-01-16T15:19:24.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f84005bb0 0x7f6f84005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa9000640 1 -- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 msgr2=0x7f6f84007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cf9a0 0x7f6f84007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa9000640 1 -- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840069d0 msgr2=0x7f6f84006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6f9bc00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840069d0 0x7f6f84006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f840069d0 0x7f6f84006e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.836+0000 7f6fa9000640 1 -- 172.21.15.53:0/2250406380 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f84005730 con 0x7f6f84005bb0 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.837+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f84005bb0 0x7f6f84005fd0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f6f94001720 tx=0x7f6f84007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44ed0acb390c4cfa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f940052d0 con 0x7f6f84005bb0 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9402e050 con 0x7f6f84005bb0 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6fa3e00640 1 -- 172.21.15.53:0/2250406380 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f940035f0 con 0x7f6f84005bb0 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6faba00640 1 -- 172.21.15.53:0/2250406380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f84005bb0 msgr2=0x7f6f84005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6faba00640 1 --2- 172.21.15.53:0/2250406380 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6f84005bb0 0x7f6f84005fd0 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f6f94001720 tx=0x7f6f84007a00 comp rx=0 tx=0).stop 2024-01-16T15:19:24.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6fa9a00640 1 -- 172.21.15.53:0/2250406380 reap_dead start 2024-01-16T15:19:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6faba00640 1 -- 172.21.15.53:0/2250406380 shutdown_connections 2024-01-16T15:19:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6faba00640 1 -- 172.21.15.53:0/2250406380 >> 172.21.15.53:0/2250406380 conn(0x7f6fa406a800 msgr2=0x7f6fa406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6faba00640 1 -- 172.21.15.53:0/2250406380 shutdown_connections 2024-01-16T15:19:24.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.838+0000 7f6faba00640 1 -- 172.21.15.53:0/2250406380 wait complete. 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.840+0000 7f6faba00640 1 Processor -- start 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.840+0000 7f6faba00640 1 -- start start 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6faba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f84005bb0 0x7f6fa413e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6faba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 0x7f6fa40cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6faba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cdde0 0x7f6fa40ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6faba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa413e730 con 0x7f6fa40affd0 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6faba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40ce260 con 0x7f6f84005bb0 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6faba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa40a9cd0 con 0x7f6fa40cdde0 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.841+0000 7f6f9bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 0x7f6fa40cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 0x7f6fa40cd8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55808/0 (socket says 172.21.15.53:55808) 2024-01-16T15:19:24.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 -- 172.21.15.53:0/2735046262 learned_addr learned my addr 172.21.15.53:0/2735046262 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f84005bb0 0x7f6fa413e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6fa9a00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cdde0 0x7f6fa40ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 -- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cdde0 msgr2=0x7f6fa40ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cdde0 0x7f6fa40ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 -- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f84005bb0 msgr2=0x7f6fa413e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f84005bb0 0x7f6fa413e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 -- 172.21.15.53:0/2735046262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fa41f1e80 con 0x7f6fa40affd0 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f84005bb0 0x7f6fa413e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6fa9a00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cdde0 0x7f6fa40ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.842+0000 7f6f9bc00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 0x7f6fa40cd8a0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00f350 tx=0x7f6f8c00f810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:24.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.843+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c017600 con 0x7f6fa40affd0 2024-01-16T15:19:24.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.843+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8c011040 con 0x7f6fa40affd0 2024-01-16T15:19:24.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.843+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8c0179c0 con 0x7f6fa40affd0 2024-01-16T15:19:24.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.843+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa41f2160 con 0x7f6fa40affd0 2024-01-16T15:19:24.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.843+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa41f2720 con 0x7f6fa40affd0 2024-01-16T15:19:24.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.844+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa40675d0 con 0x7f6fa40affd0 2024-01-16T15:19:24.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.845+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6f8c017b20 con 0x7f6fa40affd0 2024-01-16T15:19:24.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.846+0000 7f6fa1600640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f7c06c430 0x7f6f7c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:24.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.846+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6f8c089070 con 0x7f6fa40affd0 2024-01-16T15:19:24.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.847+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f7c06c430 0x7f6f7c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:24.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.848+0000 7f6fa9000640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f7c06c430 0x7f6f7c06e8f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6f94033000 tx=0x7f6f94009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:24.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:24.851+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6f8c0556a0 con 0x7f6fa40affd0 2024-01-16T15:19:25.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.047+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6fa415c690 con 0x7f6fa40affd0 2024-01-16T15:19:25.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.049+0000 7f6fa1600640 1 -- 172.21.15.53:0/2735046262 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16547 (secure 0 0 0) 0x7f6f8c01c070 con 0x7f6fa40affd0 2024-01-16T15:19:25.041 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:25.041 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":109,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:24.073908+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f7c06c430 msgr2=0x7f6f7c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f7c06c430 0x7f6f7c06e8f0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f6f94033000 tx=0x7f6f94009030 comp rx=0 tx=0).stop 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 msgr2=0x7f6fa40cd8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 0x7f6fa40cd8a0 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f6f8c00f350 tx=0x7f6f8c00f810 comp rx=0 tx=0).stop 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 shutdown_connections 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6f7c06c430 0x7f6f7c06e8f0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6fa40cdde0 0x7f6fa40ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6fa40affd0 0x7f6fa40cd8a0 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 --2- 172.21.15.53:0/2735046262 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6f84005bb0 0x7f6fa413e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 >> 172.21.15.53:0/2735046262 conn(0x7f6fa406a800 msgr2=0x7f6fa40702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.051+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 shutdown_connections 2024-01-16T15:19:25.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.052+0000 7f6faba00640 1 -- 172.21.15.53:0/2735046262 wait complete. 2024-01-16T15:19:25.064 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:25.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.287+0000 7fd34f200640 1 Processor -- start 2024-01-16T15:19:25.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.287+0000 7fd34f200640 1 -- start start 2024-01-16T15:19:25.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd350164d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3500cdc40 con 0x7fd35015afb0 2024-01-16T15:19:25.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34f200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3500cdec0 con 0x7fd35013cfa0 2024-01-16T15:19:25.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500ce1a0 con 0x7fd35013fcb0 2024-01-16T15:19:25.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd347e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd35015afb0 0x7fd35013f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55700/0 (socket says 172.21.15.53:55700) 2024-01-16T15:19:25.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd347e00640 1 -- 172.21.15.53:0/1168129786 learned_addr learned my addr 172.21.15.53:0/1168129786 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34e800640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd350164d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950162917 0 0) 0x7fd3500cdc40 con 0x7fd35015afb0 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.289+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd330003520 con 0x7fd35015afb0 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3651590275 0 0) 0x7fd330003520 con 0x7fd35015afb0 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 msgr2=0x7fd350164d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd350164d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 >> v1:172.21.15.146:6789/0 conn(0x7fd35013cfa0 legacy=0x7fd3500cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3500ce480 con 0x7fd35015afb0 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd338001670 con 0x7fd35015afb0 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 >> v1:172.21.15.53:6789/0 conn(0x7fd35015afb0 legacy=0x7fd35013f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3300039a0 0x7fd330003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd3300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd330004d00 0x7fd330005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500ce1a0 con 0x7fd330004d00 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33000bc30 con 0x7fd3300039a0 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33000e000 con 0x7fd35013fcb0 2024-01-16T15:19:25.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.290+0000 7fd34e800640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd330004d00 0x7fd330005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd34de00640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3300039a0 0x7fd330003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd34e800640 1 -- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 msgr2=0x7fd3300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd347e00640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd3300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd34e800640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd3300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd34e800640 1 -- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3300039a0 msgr2=0x7fd330003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd34e800640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3300039a0 0x7fd330003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd34e800640 1 -- 172.21.15.53:0/1168129786 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330003520 con 0x7fd330004d00 2024-01-16T15:19:25.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.291+0000 7fd347e00640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd35013fcb0 0x7fd3300047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:25.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.292+0000 7fd34e800640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd330004d00 0x7fd330005180 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd348007360 tx=0x7fd3300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21656d248e2bdf9b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.292+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34802ccb0 con 0x7fd330004d00 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.292+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd34802ce10 con 0x7fd330004d00 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.292+0000 7fd34d400640 1 -- 172.21.15.53:0/1168129786 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348034660 con 0x7fd330004d00 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34f200640 1 -- 172.21.15.53:0/1168129786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd330004d00 msgr2=0x7fd330005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34f200640 1 --2- 172.21.15.53:0/1168129786 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd330004d00 0x7fd330005180 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fd348007360 tx=0x7fd3300057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34e800640 1 -- 172.21.15.53:0/1168129786 reap_dead start 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34f200640 1 -- 172.21.15.53:0/1168129786 shutdown_connections 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34f200640 1 -- 172.21.15.53:0/1168129786 >> 172.21.15.53:0/1168129786 conn(0x7fd35006a800 msgr2=0x7fd35006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34f200640 1 -- 172.21.15.53:0/1168129786 shutdown_connections 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.293+0000 7fd34f200640 1 -- 172.21.15.53:0/1168129786 wait complete. 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.295+0000 7fd34f200640 1 Processor -- start 2024-01-16T15:19:25.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.295+0000 7fd34f200640 1 -- start start 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.296+0000 7fd34f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd330004d00 0x7fd3501f3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.296+0000 7fd34f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 0x7fd3500addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.296+0000 7fd34f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3501ec9c0 0x7fd3501ece70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.296+0000 7fd34f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd350154ac0 con 0x7fd3501f6630 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.296+0000 7fd34f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3501f6ab0 con 0x7fd3501ec9c0 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.296+0000 7fd34f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3501f6d80 con 0x7fd330004d00 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 0x7fd3500addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 0x7fd3500addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55834/0 (socket says 172.21.15.53:55834) 2024-01-16T15:19:25.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 -- 172.21.15.53:0/57784403 learned_addr learned my addr 172.21.15.53:0/57784403 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd34e800640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3501ec9c0 0x7fd3501ece70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd34de00640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd330004d00 0x7fd3501f3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 -- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd330004d00 msgr2=0x7fd3501f3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd330004d00 0x7fd3501f3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 -- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3501ec9c0 msgr2=0x7fd3501ece70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3501ec9c0 0x7fd3501ece70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.297+0000 7fd347e00640 1 -- 172.21.15.53:0/57784403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3501f7060 con 0x7fd3501f6630 2024-01-16T15:19:25.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.298+0000 7fd347e00640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 0x7fd3500addf0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fd33800d810 tx=0x7fd33800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:25.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.298+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3380074e0 con 0x7fd3501f6630 2024-01-16T15:19:25.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.298+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd338007640 con 0x7fd3501f6630 2024-01-16T15:19:25.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.298+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3380049e0 con 0x7fd3501f6630 2024-01-16T15:19:25.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.298+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3501f7340 con 0x7fd3501f6630 2024-01-16T15:19:25.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.298+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3501ed3b0 con 0x7fd3501f6630 2024-01-16T15:19:25.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.299+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd350067450 con 0x7fd3501f6630 2024-01-16T15:19:25.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.300+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fd338004b40 con 0x7fd3501f6630 2024-01-16T15:19:25.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.301+0000 7fd346000640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd32806c430 0x7fd32806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.301+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fd338088580 con 0x7fd3501f6630 2024-01-16T15:19:25.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.302+0000 7fd34de00640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd32806c430 0x7fd32806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.303+0000 7fd34de00640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd32806c430 0x7fd32806e8f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd35013f3b0 tx=0x7fd340009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:25.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.306+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd338054af0 con 0x7fd3501f6630 2024-01-16T15:19:25.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.500+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3500b4000 con 0x7fd3501f6630 2024-01-16T15:19:25.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.503+0000 7fd346000640 1 -- 172.21.15.53:0/57784403 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16547 (secure 0 0 0) 0x7fd338019070 con 0x7fd3501f6630 2024-01-16T15:19:25.495 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:25.495 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":109,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:24.073908+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:25.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd32806c430 msgr2=0x7fd32806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd32806c430 0x7fd32806e8f0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fd35013f3b0 tx=0x7fd340009040 comp rx=0 tx=0).stop 2024-01-16T15:19:25.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 msgr2=0x7fd3500addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 0x7fd3500addf0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fd33800d810 tx=0x7fd33800dcd0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 shutdown_connections 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd32806c430 0x7fd32806e8f0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3501ec9c0 0x7fd3501ece70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3501f6630 0x7fd3500addf0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 --2- 172.21.15.53:0/57784403 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd330004d00 0x7fd3501f3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 >> 172.21.15.53:0/57784403 conn(0x7fd35006a800 msgr2=0x7fd350071620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 shutdown_connections 2024-01-16T15:19:25.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.506+0000 7fd34f200640 1 -- 172.21.15.53:0/57784403 wait complete. 2024-01-16T15:19:25.519 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-01-16T15:19:25.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.754+0000 7f5c88600640 1 Processor -- start 2024-01-16T15:19:25.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.755+0000 7f5c88600640 1 -- start start 2024-01-16T15:19:25.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.756+0000 7f5c88600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c8013df80 0x7f5c8015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.756+0000 7f5c88600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5c8013e360 0x7f5c800cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.756+0000 7f5c88600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8015f9c0 0x7f5c800cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.756+0000 7f5c88600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8006c470 con 0x7f5c8013df80 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.756+0000 7f5c88600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c8006c750 con 0x7f5c8013e360 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.756+0000 7f5c88600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c8006ca30 con 0x7f5c8015f9c0 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c8013df80 0x7f5c8015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5c8013df80 0x7f5c8015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55846/0 (socket says 172.21.15.53:55846) 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 -- 172.21.15.53:0/920025804 learned_addr learned my addr 172.21.15.53:0/920025804 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c86600640 1 --2- 172.21.15.53:0/920025804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8015f9c0 0x7f5c800cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85200640 1 --2- 172.21.15.53:0/920025804 >> v2:172.21.15.146:3300/0 conn(0x7f5c8013e360 0x7f5c800cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 -- 172.21.15.53:0/920025804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8015f9c0 msgr2=0x7f5c800cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:25.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 --2- 172.21.15.53:0/920025804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8015f9c0 0x7f5c800cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 -- 172.21.15.53:0/920025804 >> v2:172.21.15.146:3300/0 conn(0x7f5c8013e360 msgr2=0x7f5c800cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 --2- 172.21.15.53:0/920025804 >> v2:172.21.15.146:3300/0 conn(0x7f5c8013e360 0x7f5c800cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.757+0000 7f5c85c00640 1 -- 172.21.15.53:0/920025804 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8006cd10 con 0x7f5c8013df80 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.758+0000 7f5c85c00640 1 --2- 172.21.15.53:0/920025804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8013df80 0x7f5c8015bee0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f5c8006ded0 tx=0x7f5c7002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73927a8edd105921 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.759+0000 7f5c7fe00640 1 -- 172.21.15.53:0/920025804 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c7002ca70 con 0x7f5c8013df80 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.759+0000 7f5c7fe00640 1 -- 172.21.15.53:0/920025804 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c7002cbd0 con 0x7f5c8013df80 2024-01-16T15:19:25.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.759+0000 7f5c7fe00640 1 -- 172.21.15.53:0/920025804 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c700343f0 con 0x7f5c8013df80 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 -- 172.21.15.53:0/920025804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8013df80 msgr2=0x7f5c8015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 --2- 172.21.15.53:0/920025804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8013df80 0x7f5c8015bee0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f5c8006ded0 tx=0x7f5c7002bed0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 -- 172.21.15.53:0/920025804 shutdown_connections 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 --2- 172.21.15.53:0/920025804 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8015f9c0 0x7f5c800cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 --2- 172.21.15.53:0/920025804 >> v2:172.21.15.146:3300/0 conn(0x7f5c8013e360 0x7f5c800cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 --2- 172.21.15.53:0/920025804 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8013df80 0x7f5c8015bee0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 -- 172.21.15.53:0/920025804 >> 172.21.15.53:0/920025804 conn(0x7f5c8006a800 msgr2=0x7f5c8006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:25.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 -- 172.21.15.53:0/920025804 shutdown_connections 2024-01-16T15:19:25.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.760+0000 7f5c88600640 1 -- 172.21.15.53:0/920025804 wait complete. 2024-01-16T15:19:25.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.762+0000 7f5c88600640 1 Processor -- start 2024-01-16T15:19:25.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.762+0000 7f5c88600640 1 -- start start 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c88600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c8013df80 0x7f5c801549d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c88600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8013e360 0x7f5c80136700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c88600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 0x7f5c800b3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c88600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c80071ce0 con 0x7f5c8015f9c0 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c88600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c80071160 con 0x7f5c8013df80 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c88600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c80071440 con 0x7f5c8013e360 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c86600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 0x7f5c800b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c86600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 0x7f5c800b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55858/0 (socket says 172.21.15.53:55858) 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.763+0000 7f5c86600640 1 -- 172.21.15.53:0/1325679226 learned_addr learned my addr 172.21.15.53:0/1325679226 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c85200640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8013e360 0x7f5c80136700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c85c00640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c8013df80 0x7f5c801549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c86600640 1 -- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8013e360 msgr2=0x7f5c80136700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c86600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8013e360 0x7f5c80136700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c86600640 1 -- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c8013df80 msgr2=0x7f5c801549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:25.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c86600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c8013df80 0x7f5c801549d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:25.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c86600640 1 -- 172.21.15.53:0/1325679226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c70009b30 con 0x7f5c8015f9c0 2024-01-16T15:19:25.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c86600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 0x7f5c800b3770 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800d5d0 tx=0x7f5c7800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:25.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c780125f0 con 0x7f5c8015f9c0 2024-01-16T15:19:25.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c78012750 con 0x7f5c8015f9c0 2024-01-16T15:19:25.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c78012b00 con 0x7f5c8015f9c0 2024-01-16T15:19:25.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.764+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c80071720 con 0x7f5c8015f9c0 2024-01-16T15:19:25.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.765+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c80154f10 con 0x7f5c8015f9c0 2024-01-16T15:19:25.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.766+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c800675d0 con 0x7f5c8015f9c0 2024-01-16T15:19:25.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.767+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f5c78012c60 con 0x7f5c8015f9c0 2024-01-16T15:19:25.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.768+0000 7f5c7d600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5806c430 0x7f5c5806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:25.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.768+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f5c78089020 con 0x7f5c8015f9c0 2024-01-16T15:19:25.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.768+0000 7f5c85c00640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5806c430 0x7f5c5806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:25.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.770+0000 7f5c85c00640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5806c430 0x7f5c5806e8f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5c70035000 tx=0x7f5c70030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:25.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:25.772+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5c78055500 con 0x7f5c8015f9c0 2024-01-16T15:19:25.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2024-01-16T15:19:25.919 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.d has been restored 2024-01-16T15:19:25.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.003+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f5c8015bf70 con 0x7f5c8015f9c0 2024-01-16T15:19:26.273 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:26.282 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:26.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.629+0000 7f5c7d600640 1 -- 172.21.15.53:0/1325679226 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) v1 ==== 118+0+0 (secure 0 0 0) 0x7f5c78058b20 con 0x7f5c8015f9c0 2024-01-16T15:19:26.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5806c430 msgr2=0x7f5c5806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:26.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5806c430 0x7f5c5806e8f0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f5c70035000 tx=0x7f5c70030000 comp rx=0 tx=0).stop 2024-01-16T15:19:26.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 msgr2=0x7f5c800b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:26.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 0x7f5c800b3770 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f5c7800d5d0 tx=0x7f5c7800da90 comp rx=0 tx=0).stop 2024-01-16T15:19:26.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 shutdown_connections 2024-01-16T15:19:26.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5c5806c430 0x7f5c5806e8f0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5c8015f9c0 0x7f5c800b3770 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5c8013e360 0x7f5c80136700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 --2- 172.21.15.53:0/1325679226 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5c8013df80 0x7f5c801549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 >> 172.21.15.53:0/1325679226 conn(0x7f5c8006a800 msgr2=0x7f5c8013ee00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:26.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 shutdown_connections 2024-01-16T15:19:26.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.633+0000 7f5c88600640 1 -- 172.21.15.53:0/1325679226 wait complete. 2024-01-16T15:19:26.648 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-01-16T15:19:26.648 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-01-16T15:19:26.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.883+0000 7f22c1a00640 1 Processor -- start 2024-01-16T15:19:26.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.883+0000 7f22c1a00640 1 -- start start 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.885+0000 7f22c1a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22bc13e360 0x7f22bc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.885+0000 7f22c1a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f22bc15f9c0 0x7f22bc0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.885+0000 7f22c1a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22bc15c5f0 con 0x7f22bc13df80 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.885+0000 7f22c1a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc06c470 con 0x7f22bc13e360 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.885+0000 7f22c1a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22bc06c750 con 0x7f22bc15f9c0 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22bc13e360 0x7f22bc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22bc13e360 0x7f22bc0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55866/0 (socket says 172.21.15.53:55866) 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 -- 172.21.15.53:0/3880618328 learned_addr learned my addr 172.21.15.53:0/3880618328 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:26.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22bb400640 1 --2- 172.21.15.53:0/3880618328 >> v2:172.21.15.146:3300/0 conn(0x7f22bc15f9c0 0x7f22bc0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:26.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 -- 172.21.15.53:0/3880618328 >> v2:172.21.15.146:3300/0 conn(0x7f22bc15f9c0 msgr2=0x7f22bc0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:26.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 --2- 172.21.15.53:0/3880618328 >> v2:172.21.15.146:3300/0 conn(0x7f22bc15f9c0 0x7f22bc0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 -- 172.21.15.53:0/3880618328 >> v1:172.21.15.146:6789/0 conn(0x7f22bc13df80 legacy=0x7f22bc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:26.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.886+0000 7f22ba000640 1 -- 172.21.15.53:0/3880618328 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22bc06ca30 con 0x7f22bc13e360 2024-01-16T15:19:26.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.888+0000 7f22ba000640 1 --2- 172.21.15.53:0/3880618328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc13e360 0x7f22bc0cf290 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f22bc06ded0 tx=0x7f22a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b79aea03588f9bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:26.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.888+0000 7f22b9600640 1 -- 172.21.15.53:0/3880618328 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031870 con 0x7f22bc13e360 2024-01-16T15:19:26.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.888+0000 7f22b9600640 1 -- 172.21.15.53:0/3880618328 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a40319d0 con 0x7f22bc13e360 2024-01-16T15:19:26.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.888+0000 7f22b9600640 1 -- 172.21.15.53:0/3880618328 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22a4031d60 con 0x7f22bc13e360 2024-01-16T15:19:26.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 -- 172.21.15.53:0/3880618328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc13e360 msgr2=0x7f22bc0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:26.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3880618328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc13e360 0x7f22bc0cf290 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f22bc06ded0 tx=0x7f22a402b910 comp rx=0 tx=0).stop 2024-01-16T15:19:26.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 -- 172.21.15.53:0/3880618328 shutdown_connections 2024-01-16T15:19:26.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3880618328 >> v2:172.21.15.146:3300/0 conn(0x7f22bc15f9c0 0x7f22bc0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3880618328 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc13e360 0x7f22bc0cf290 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 -- 172.21.15.53:0/3880618328 >> 172.21.15.53:0/3880618328 conn(0x7f22bc06a800 msgr2=0x7f22bc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:26.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 -- 172.21.15.53:0/3880618328 shutdown_connections 2024-01-16T15:19:26.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.889+0000 7f22c1a00640 1 -- 172.21.15.53:0/3880618328 wait complete. 2024-01-16T15:19:26.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.891+0000 7f22c1a00640 1 Processor -- start 2024-01-16T15:19:26.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.891+0000 7f22c1a00640 1 -- start start 2024-01-16T15:19:26.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22c1a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22bc13df80 0x7f22bc1f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:26.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22c1a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22bc13e360 0x7f22bc15d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22c1a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 0x7f22bc1f6f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22c1a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc15e270 con 0x7f22bc15f9c0 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22c1a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc1f49b0 con 0x7f22bc13df80 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22c1a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22bc13a0d0 con 0x7f22bc13e360 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22bb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 0x7f22bc1f6f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22bb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 0x7f22bc1f6f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55880/0 (socket says 172.21.15.53:55880) 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22bb400640 1 -- 172.21.15.53:0/3292480041 learned_addr learned my addr 172.21.15.53:0/3292480041 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22baa00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22bc13df80 0x7f22bc1f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.893+0000 7f22ba000640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22bc13e360 0x7f22bc15d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22bb400640 1 -- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22bc13e360 msgr2=0x7f22bc15d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22bb400640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22bc13e360 0x7f22bc15d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22bb400640 1 -- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22bc13df80 msgr2=0x7f22bc1f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22bb400640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22bc13df80 0x7f22bc1f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22bb400640 1 -- 172.21.15.53:0/3292480041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a40094d0 con 0x7f22bc15f9c0 2024-01-16T15:19:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22ba000640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22bc13e360 0x7f22bc15d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22bb400640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 0x7f22bc1f6f60 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f22b4007e60 tx=0x7f22b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:26.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4014420 con 0x7f22bc15f9c0 2024-01-16T15:19:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22b4014580 con 0x7f22bc15f9c0 2024-01-16T15:19:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4014930 con 0x7f22bc15f9c0 2024-01-16T15:19:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.894+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22bc1440f0 con 0x7f22bc15f9c0 2024-01-16T15:19:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.895+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22bc15df90 con 0x7f22bc15f9c0 2024-01-16T15:19:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.896+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22bc144280 con 0x7f22bc15f9c0 2024-01-16T15:19:26.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.897+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f22b4014a90 con 0x7f22bc15f9c0 2024-01-16T15:19:26.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.898+0000 7f22b2a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f229406c430 0x7f229406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.898+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f22b4089410 con 0x7f22bc15f9c0 2024-01-16T15:19:26.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.899+0000 7f22baa00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f229406c430 0x7f229406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:26.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.900+0000 7f22baa00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f229406c430 0x7f229406e8f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f22ac006fd0 tx=0x7f22ac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:26.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:26.904+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22b4055980 con 0x7f22bc15f9c0 2024-01-16T15:19:27.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.145+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f22bc0cf490 con 0x7f22bc15f9c0 2024-01-16T15:19:27.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.642+0000 7f22b2a00640 1 -- 172.21.15.53:0/3292480041 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v74) v1 ==== 122+0+0 (secure 0 0 0) 0x7f22b4058fa0 con 0x7f22bc15f9c0 2024-01-16T15:19:27.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f229406c430 msgr2=0x7f229406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:27.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f229406c430 0x7f229406e8f0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f22ac006fd0 tx=0x7f22ac008040 comp rx=0 tx=0).stop 2024-01-16T15:19:27.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 msgr2=0x7f22bc1f6f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:27.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 0x7f22bc1f6f60 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7f22b4007e60 tx=0x7f22b400e4b0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 shutdown_connections 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f229406c430 0x7f229406e8f0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22bc15f9c0 0x7f22bc1f6f60 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22bc13e360 0x7f22bc15d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 --2- 172.21.15.53:0/3292480041 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22bc13df80 0x7f22bc1f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 >> 172.21.15.53:0/3292480041 conn(0x7f22bc06a800 msgr2=0x7f22bc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.646+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 shutdown_connections 2024-01-16T15:19:27.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.647+0000 7f22c1a00640 1 -- 172.21.15.53:0/3292480041 wait complete. 2024-01-16T15:19:27.660 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-01-16T15:19:27.660 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:19:27.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.891+0000 7faea6c00640 1 Processor -- start 2024-01-16T15:19:27.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.892+0000 7faea6c00640 1 -- start start 2024-01-16T15:19:27.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.893+0000 7faea6c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faea013ff20 0x7faea015e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:27.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.893+0000 7faea6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea0140300 0x7faea006a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:27.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.893+0000 7faea6c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea00b5440 con 0x7faea006a920 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.893+0000 7faea6c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea006adb0 con 0x7faea013ff20 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.893+0000 7faea6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea0161c80 con 0x7faea0140300 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faea013ff20 0x7faea015e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7faea013ff20 0x7faea015e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55882/0 (socket says 172.21.15.53:55882) 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 -- 172.21.15.53:0/953240741 learned_addr learned my addr 172.21.15.53:0/953240741 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea4e00640 1 --2- 172.21.15.53:0/953240741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea0140300 0x7faea006a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:27.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 -- 172.21.15.53:0/953240741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea0140300 msgr2=0x7faea006a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 --2- 172.21.15.53:0/953240741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea0140300 0x7faea006a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 -- 172.21.15.53:0/953240741 >> v1:172.21.15.53:6789/0 conn(0x7faea006a920 legacy=0x7faea00b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.894+0000 7faea5800640 1 -- 172.21.15.53:0/953240741 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea015e6f0 con 0x7faea013ff20 2024-01-16T15:19:27.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.895+0000 7faea5800640 1 --2- 172.21.15.53:0/953240741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea013ff20 0x7faea015e1b0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fae90009870 tx=0x7fae9002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=741606cbd3da1a4a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7fae9fe00640 1 -- 172.21.15.53:0/953240741 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90031640 con 0x7faea013ff20 2024-01-16T15:19:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7fae9fe00640 1 -- 172.21.15.53:0/953240741 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae900317a0 con 0x7faea013ff20 2024-01-16T15:19:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7fae9fe00640 1 -- 172.21.15.53:0/953240741 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90031b30 con 0x7faea013ff20 2024-01-16T15:19:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7faea6c00640 1 -- 172.21.15.53:0/953240741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea013ff20 msgr2=0x7faea015e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:27.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7faea6c00640 1 --2- 172.21.15.53:0/953240741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea013ff20 0x7faea015e1b0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fae90009870 tx=0x7fae9002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7faea6c00640 1 -- 172.21.15.53:0/953240741 shutdown_connections 2024-01-16T15:19:27.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7faea6c00640 1 --2- 172.21.15.53:0/953240741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea0140300 0x7faea006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7faea6c00640 1 --2- 172.21.15.53:0/953240741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea013ff20 0x7faea015e1b0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.896+0000 7faea6c00640 1 -- 172.21.15.53:0/953240741 >> 172.21.15.53:0/953240741 conn(0x7faea00b1cd0 msgr2=0x7faea00b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:27.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.897+0000 7faea6c00640 1 -- 172.21.15.53:0/953240741 shutdown_connections 2024-01-16T15:19:27.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.897+0000 7faea6c00640 1 -- 172.21.15.53:0/953240741 wait complete. 2024-01-16T15:19:27.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.898+0000 7faea6c00640 1 Processor -- start 2024-01-16T15:19:27.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.898+0000 7faea6c00640 1 -- start start 2024-01-16T15:19:27.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.899+0000 7faea6c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 0x7faea0150480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.899+0000 7faea6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea013ff20 0x7faea00cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.899+0000 7faea6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea0140300 0x7faea015e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.899+0000 7faea6c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea00ce9e0 con 0x7faea006a920 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.899+0000 7faea6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea01509c0 con 0x7faea0140300 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.899+0000 7faea6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea00cdd60 con 0x7faea013ff20 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 0x7faea0150480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 0x7faea0150480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55896/0 (socket says 172.21.15.53:55896) 2024-01-16T15:19:27.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 -- 172.21.15.53:0/3123967259 learned_addr learned my addr 172.21.15.53:0/3123967259 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 -- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea013ff20 msgr2=0x7faea00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea4e00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea013ff20 0x7faea00cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea6200640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea0140300 0x7faea015e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea013ff20 0x7faea00cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 -- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea0140300 msgr2=0x7faea015e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea0140300 0x7faea015e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea5800640 1 -- 172.21.15.53:0/3123967259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900094d0 con 0x7faea006a920 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.900+0000 7faea4e00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea013ff20 0x7faea00cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.901+0000 7faea5800640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 0x7faea0150480 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fae90000c00 tx=0x7fae900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:27.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.901+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9003ac60 con 0x7faea006a920 2024-01-16T15:19:27.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.901+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9003ade0 con 0x7faea006a920 2024-01-16T15:19:27.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.901+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae90044630 con 0x7faea006a920 2024-01-16T15:19:27.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.901+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea00ce040 con 0x7faea006a920 2024-01-16T15:19:27.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.901+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea00ce600 con 0x7faea006a920 2024-01-16T15:19:27.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.902+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea0067450 con 0x7faea006a920 2024-01-16T15:19:27.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.903+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fae9002f050 con 0x7faea006a920 2024-01-16T15:19:27.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.904+0000 7fae9d600640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae7806c430 0x7fae7806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:27.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.904+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fae900aabc0 con 0x7faea006a920 2024-01-16T15:19:27.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.905+0000 7faea4e00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae7806c430 0x7fae7806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:27.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.906+0000 7faea4e00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae7806c430 0x7fae7806e8f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7faea006ee60 tx=0x7fae88008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:27.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:27.909+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fae90077130 con 0x7faea006a920 2024-01-16T15:19:28.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.143+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faea00b4300 con 0x7faea006a920 2024-01-16T15:19:28.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.144+0000 7fae9d600640 1 -- 172.21.15.53:0/3123967259 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fae90032040 con 0x7faea006a920 2024-01-16T15:19:28.136 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:28.137 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1519219735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1519219735},{"type":"v1","addr":"172.21.15.146:6837","nonce":1519219735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2058874467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2058874467},{"type":"v1","addr":"172.21.15.146:6836","nonce":2058874467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3227938262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3227938262},{"type":"v1","addr":"172.21.15.53:6835","nonce":3227938262}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:19:24.074687+0000","modified":"2024-01-16T15:19:27.148085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/1996992482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1996992482},{"type":"v1","addr":"172.21.15.53:6834","nonce":1996992482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:19:28.137 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 74 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae7806c430 msgr2=0x7fae7806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae7806c430 0x7fae7806e8f0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7faea006ee60 tx=0x7fae88008040 comp rx=0 tx=0).stop 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 msgr2=0x7faea0150480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 0x7faea0150480 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fae90000c00 tx=0x7fae900046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 shutdown_connections 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fae7806c430 0x7fae7806e8f0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7faea0140300 0x7faea015e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.147+0000 7faea6c00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7faea013ff20 0x7faea00cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.148+0000 7faea6c00640 1 --2- 172.21.15.53:0/3123967259 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7faea006a920 0x7faea0150480 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.148+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 >> 172.21.15.53:0/3123967259 conn(0x7faea00b1cd0 msgr2=0x7faea00b2360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:28.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.148+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 shutdown_connections 2024-01-16T15:19:28.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.148+0000 7faea6c00640 1 -- 172.21.15.53:0/3123967259 wait complete. 2024-01-16T15:19:28.158 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:19:28.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.384+0000 7f7609e00640 1 Processor -- start 2024-01-16T15:19:28.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.384+0000 7f7609e00640 1 -- start start 2024-01-16T15:19:28.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7609e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7604143c40 0x7f7604071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7609e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040717d0 0x7f76040cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7609e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f76040cca40 0x7f760415eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7609e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76040723c0 con 0x7f7604143c40 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7609e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7604070e50 con 0x7f76040cca40 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7609e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76040cce70 con 0x7f76040717d0 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.386+0000 7f7608a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7604143c40 0x7f7604071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7604143c40 0x7f7604071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55906/0 (socket says 172.21.15.53:55906) 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 -- 172.21.15.53:0/1920660781 learned_addr learned my addr 172.21.15.53:0/1920660781 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7603e00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040717d0 0x7f76040cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7609400640 1 --2- 172.21.15.53:0/1920660781 >> v2:172.21.15.146:3300/0 conn(0x7f76040cca40 0x7f760415eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 -- 172.21.15.53:0/1920660781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040717d0 msgr2=0x7f76040cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040717d0 0x7f76040cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 -- 172.21.15.53:0/1920660781 >> v2:172.21.15.146:3300/0 conn(0x7f76040cca40 msgr2=0x7f760415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:28.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 --2- 172.21.15.53:0/1920660781 >> v2:172.21.15.146:3300/0 conn(0x7f76040cca40 0x7f760415eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7608a00640 1 -- 172.21.15.53:0/1920660781 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7604144020 con 0x7f7604143c40 2024-01-16T15:19:28.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.387+0000 7f7603e00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040717d0 0x7f76040cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.388+0000 7f7608a00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7604143c40 0x7f7604071290 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f76040cc3f0 tx=0x7f75f402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6664fcd54693c5d8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.388+0000 7f7603400640 1 -- 172.21.15.53:0/1920660781 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40317d0 con 0x7f7604143c40 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.388+0000 7f7603400640 1 -- 172.21.15.53:0/1920660781 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f4031930 con 0x7f7604143c40 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.388+0000 7f7603400640 1 -- 172.21.15.53:0/1920660781 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4031cc0 con 0x7f7604143c40 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 -- 172.21.15.53:0/1920660781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7604143c40 msgr2=0x7f7604071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7604143c40 0x7f7604071290 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7f76040cc3f0 tx=0x7f75f402b870 comp rx=0 tx=0).stop 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 -- 172.21.15.53:0/1920660781 shutdown_connections 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 --2- 172.21.15.53:0/1920660781 >> v2:172.21.15.146:3300/0 conn(0x7f76040cca40 0x7f760415eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040717d0 0x7f76040cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 --2- 172.21.15.53:0/1920660781 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7604143c40 0x7f7604071290 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 -- 172.21.15.53:0/1920660781 >> 172.21.15.53:0/1920660781 conn(0x7f760406a800 msgr2=0x7f760406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:28.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 -- 172.21.15.53:0/1920660781 shutdown_connections 2024-01-16T15:19:28.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.389+0000 7f7609e00640 1 -- 172.21.15.53:0/1920660781 wait complete. 2024-01-16T15:19:28.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.391+0000 7f7609e00640 1 Processor -- start 2024-01-16T15:19:28.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.391+0000 7f7609e00640 1 -- start start 2024-01-16T15:19:28.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7609e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 0x7f76041f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7609e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040cca40 0x7f76040c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7609e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7604143c40 0x7f76040b3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7609e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76041f65f0 con 0x7f76040717d0 2024-01-16T15:19:28.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7609e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76040ca870 con 0x7f7604143c40 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7609e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76040c3e10 con 0x7f76040cca40 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7608a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 0x7f76041f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7608a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 0x7f76041f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55910/0 (socket says 172.21.15.53:55910) 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.392+0000 7f7608a00640 1 -- 172.21.15.53:0/3600030091 learned_addr learned my addr 172.21.15.53:0/3600030091 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7603e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040cca40 0x7f76040c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7609400640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7604143c40 0x7f76040b3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7608a00640 1 -- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040cca40 msgr2=0x7f76040c4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7608a00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040cca40 0x7f76040c4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7608a00640 1 -- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7604143c40 msgr2=0x7f76040b3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7608a00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7604143c40 0x7f76040b3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7608a00640 1 -- 172.21.15.53:0/3600030091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f40094d0 con 0x7f76040717d0 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7609400640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7604143c40 0x7f76040b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7608a00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 0x7f76041f60b0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f75f4000c00 tx=0x7f75f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.393+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f40328f0 con 0x7f76040717d0 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.394+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f4032a70 con 0x7f76040717d0 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.394+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76040c40f0 con 0x7f76040717d0 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.394+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f4032e20 con 0x7f76040717d0 2024-01-16T15:19:28.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.394+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76040c46b0 con 0x7f76040717d0 2024-01-16T15:19:28.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.395+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7604067450 con 0x7f76040717d0 2024-01-16T15:19:28.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.396+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f75f4030070 con 0x7f76040717d0 2024-01-16T15:19:28.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.397+0000 7f7600c00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f75dc06c430 0x7f75dc06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.397+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f75f40aa780 con 0x7f76040717d0 2024-01-16T15:19:28.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.397+0000 7f7603e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f75dc06c430 0x7f75dc06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.399+0000 7f7603e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f75dc06c430 0x7f75dc06e8f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f76040c57d0 tx=0x7f75ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:28.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.401+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f75f40770f0 con 0x7f76040717d0 2024-01-16T15:19:28.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.599+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f76040b3ec0 con 0x7f76040717d0 2024-01-16T15:19:28.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.602+0000 7f7600c00640 1 -- 172.21.15.53:0/3600030091 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16534 (secure 0 0 0) 0x7f75f407a710 con 0x7f76040717d0 2024-01-16T15:19:28.594 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:28.595 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":110,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:25.603808+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:19:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f75dc06c430 msgr2=0x7f75dc06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f75dc06c430 0x7f75dc06e8f0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f76040c57d0 tx=0x7f75ec008040 comp rx=0 tx=0).stop 2024-01-16T15:19:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 msgr2=0x7f76041f60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 0x7f76041f60b0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f75f4000c00 tx=0x7f75f40046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 shutdown_connections 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f75dc06c430 0x7f75dc06e8f0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7604143c40 0x7f76040b3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f76040cca40 0x7f76040c4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 --2- 172.21.15.53:0/3600030091 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f76040717d0 0x7f76041f60b0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 >> 172.21.15.53:0/3600030091 conn(0x7f760406a800 msgr2=0x7f7604140200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 shutdown_connections 2024-01-16T15:19:28.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.605+0000 7f7609e00640 1 -- 172.21.15.53:0/3600030091 wait complete. 2024-01-16T15:19:28.616 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-01-16T15:19:28.616 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:19:28.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.846+0000 7f7753c00640 1 Processor -- start 2024-01-16T15:19:28.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.846+0000 7f7753c00640 1 -- start start 2024-01-16T15:19:28.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.847+0000 7f7753c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f774c161740 0x7f774c168380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.847+0000 7f7753c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774c0cf260 con 0x7f774c0d0560 2024-01-16T15:19:28.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.847+0000 7f7753c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774c0cf4c0 con 0x7f774c0c9870 2024-01-16T15:19:28.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.847+0000 7f7753c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774c0d2ef0 con 0x7f774c161740 2024-01-16T15:19:28.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.848+0000 7f774be00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f774c0d0560 0x7f774c161030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55726/0 (socket says 172.21.15.53:55726) 2024-01-16T15:19:28.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.848+0000 7f774be00640 1 -- 172.21.15.53:0/1312343216 learned_addr learned my addr 172.21.15.53:0/1312343216 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:28.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.848+0000 7f7751c00640 1 --2- 172.21.15.53:0/1312343216 >> v2:172.21.15.53:3300/0 conn(0x7f774c161740 0x7f774c168380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.848+0000 7f774b400640 1 -- 172.21.15.53:0/1312343216 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133590145 0 0) 0x7f774c0cf260 con 0x7f774c0d0560 2024-01-16T15:19:28.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.848+0000 7f774b400640 1 -- 172.21.15.53:0/1312343216 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f772c003520 con 0x7f774c0d0560 2024-01-16T15:19:28.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.849+0000 7f7751c00640 1 -- 172.21.15.53:0/1312343216 >> v1:172.21.15.146:6789/0 conn(0x7f774c0c9870 legacy=0x7f774c0ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.849+0000 7f7751c00640 1 -- 172.21.15.53:0/1312343216 >> v1:172.21.15.53:6789/0 conn(0x7f774c0d0560 legacy=0x7f774c161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.849+0000 7f7751c00640 1 -- 172.21.15.53:0/1312343216 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c0cf7a0 con 0x7f774c161740 2024-01-16T15:19:28.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.850+0000 7f7751c00640 1 --2- 172.21.15.53:0/1312343216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c168380 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f7744009d90 tx=0x7f774402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2117a9b9acb1a15d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:28.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.850+0000 7f774b400640 1 -- 172.21.15.53:0/1312343216 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774402cd00 con 0x7f774c161740 2024-01-16T15:19:28.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.850+0000 7f774b400640 1 -- 172.21.15.53:0/1312343216 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774402ce60 con 0x7f774c161740 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.850+0000 7f774b400640 1 -- 172.21.15.53:0/1312343216 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7744034660 con 0x7f774c161740 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.850+0000 7f7753c00640 1 -- 172.21.15.53:0/1312343216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 msgr2=0x7f774c168380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.850+0000 7f7753c00640 1 --2- 172.21.15.53:0/1312343216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c168380 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f7744009d90 tx=0x7f774402bf50 comp rx=0 tx=0).stop 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.851+0000 7f7753c00640 1 -- 172.21.15.53:0/1312343216 shutdown_connections 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.851+0000 7f7753c00640 1 --2- 172.21.15.53:0/1312343216 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c168380 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.851+0000 7f7753c00640 1 -- 172.21.15.53:0/1312343216 >> 172.21.15.53:0/1312343216 conn(0x7f774c09c560 msgr2=0x7f774c0d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.851+0000 7f7753c00640 1 -- 172.21.15.53:0/1312343216 shutdown_connections 2024-01-16T15:19:28.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.851+0000 7f7753c00640 1 -- 172.21.15.53:0/1312343216 wait complete. 2024-01-16T15:19:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.853+0000 7f7753c00640 1 Processor -- start 2024-01-16T15:19:28.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.853+0000 7f7753c00640 1 -- start start 2024-01-16T15:19:28.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7753c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774c0c9870 0x7f774c1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7753c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f774c0d0560 0x7f774c0b8240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7753c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c0b5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7753c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c1f08f0 con 0x7f774c161740 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7753c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c0b7ad0 con 0x7f774c0d0560 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7753c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f774c1f36e0 con 0x7f774c0c9870 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7751c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c0b5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7751c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c0b5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55928/0 (socket says 172.21.15.53:55928) 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7751c00640 1 -- 172.21.15.53:0/2147683082 learned_addr learned my addr 172.21.15.53:0/2147683082 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f774be00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f774c0d0560 0x7f774c0b8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7751c00640 1 -- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774c0c9870 msgr2=0x7f774c1f2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.854+0000 7f7751200640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774c0c9870 0x7f774c1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7751c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774c0c9870 0x7f774c1f2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7751c00640 1 -- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f774c0d0560 msgr2=0x7f774c0b8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7751c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f774c0d0560 0x7f774c0b8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7751c00640 1 -- 172.21.15.53:0/2147683082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7744009a70 con 0x7f774c161740 2024-01-16T15:19:28.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f774be00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f774c0d0560 0x7f774c0b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7751200640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774c0c9870 0x7f774c1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:28.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7751c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c0b5e70 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f7744009d90 tx=0x7f7744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774403d600 con 0x7f774c161740 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f774403d780 con 0x7f774c161740 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f774403db30 con 0x7f774c161740 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774c1f39c0 con 0x7f774c161740 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.855+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774c0b7d20 con 0x7f774c161740 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.857+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774c0675d0 con 0x7f774c161740 2024-01-16T15:19:28.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.858+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f774403b040 con 0x7f774c161740 2024-01-16T15:19:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.858+0000 7f7748c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f772406c430 0x7f772406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:28.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.859+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f77440ab060 con 0x7f774c161740 2024-01-16T15:19:28.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.859+0000 7f7751200640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f772406c430 0x7f772406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:28.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.860+0000 7f7751200640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f772406c430 0x7f772406e8f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f773c000f60 tx=0x7f773c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:28.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:28.863+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f77440776f0 con 0x7f774c161740 2024-01-16T15:19:29.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.054+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f774c0cfd60 con 0x7f772406c430 2024-01-16T15:19:29.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.082+0000 7f7748c00640 1 -- 172.21.15.53:0/2147683082 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287189 (secure 0 0 0) 0x7f773c001cd0 con 0x7f772406c430 2024-01-16T15:19:29.079 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:29.081 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:19:29.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f772406c430 msgr2=0x7f772406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f772406c430 0x7f772406e8f0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f773c000f60 tx=0x7f773c009040 comp rx=0 tx=0).stop 2024-01-16T15:19:29.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 msgr2=0x7f774c0b5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c0b5e70 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f7744009d90 tx=0x7f7744004a40 comp rx=0 tx=0).stop 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 shutdown_connections 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f772406c430 0x7f772406e8f0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f774c161740 0x7f774c0b5e70 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f774c0d0560 0x7f774c0b8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 --2- 172.21.15.53:0/2147683082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f774c0c9870 0x7f774c1f2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 >> 172.21.15.53:0/2147683082 conn(0x7f774c09c560 msgr2=0x7f774c06b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 shutdown_connections 2024-01-16T15:19:29.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.089+0000 7f7753c00640 1 -- 172.21.15.53:0/2147683082 wait complete. 2024-01-16T15:19:29.104 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2024-01-16T15:19:28.692976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":566132,"kb_used_data":11776,"kb_used_omap":12,"kb_used_meta":554227,"kb_avail":754408588,"statfs":{"total":773094113280,"available":772514394112,"internally_reserved":0,"allocated":12058624,"data_stored":5859004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567529008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001915"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637642+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:19:24.637642+0000","last_peered":"2024-01-16T15:19:24.637642+0000","last_clean":"2024-01-16T15:19:24.637642+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:19:24.637642+0000","last_undegraded":"2024-01-16T15:19:24.637642+0000","last_fullsized":"2024-01-16T15:19:24.637642+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:32:27.826457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168771+0000","last_change":"2024-01-16T15:19:24.622288+0000","last_active":"2024-01-16T15:19:26.168771+0000","last_peered":"2024-01-16T15:19:26.168771+0000","last_clean":"2024-01-16T15:19:26.168771+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:19:26.168771+0000","last_undegraded":"2024-01-16T15:19:26.168771+0000","last_fullsized":"2024-01-16T15:19:26.168771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:20.176733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626908+0000","last_change":"2024-01-16T15:19:24.618862+0000","last_active":"2024-01-16T15:19:24.626908+0000","last_peered":"2024-01-16T15:19:24.626908+0000","last_clean":"2024-01-16T15:19:24.626908+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:19:24.626908+0000","last_undegraded":"2024-01-16T15:19:24.626908+0000","last_fullsized":"2024-01-16T15:19:24.626908+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:53.058982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642476+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:19:24.642476+0000","last_peered":"2024-01-16T15:19:24.642476+0000","last_clean":"2024-01-16T15:19:24.642476+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:19:24.642476+0000","last_undegraded":"2024-01-16T15:19:24.642476+0000","last_fullsized":"2024-01-16T15:19:24.642476+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:11.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631057+0000","last_change":"2024-01-16T15:19:24.626093+0000","last_active":"2024-01-16T15:19:24.631057+0000","last_peered":"2024-01-16T15:19:24.631057+0000","last_clean":"2024-01-16T15:19:24.631057+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:19:24.631057+0000","last_undegraded":"2024-01-16T15:19:24.631057+0000","last_fullsized":"2024-01-16T15:19:24.631057+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:47.875736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.630172+0000","last_change":"2024-01-16T15:19:24.625809+0000","last_active":"2024-01-16T15:19:24.630172+0000","last_peered":"2024-01-16T15:19:24.630172+0000","last_clean":"2024-01-16T15:19:24.630172+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:19:24.630172+0000","last_undegraded":"2024-01-16T15:19:24.630172+0000","last_fullsized":"2024-01-16T15:19:24.630172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:40:11.125336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627850+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:19:24.627850+0000","last_peered":"2024-01-16T15:19:24.627850+0000","last_clean":"2024-01-16T15:19:24.627850+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:19:24.627850+0000","last_undegraded":"2024-01-16T15:19:24.627850+0000","last_fullsized":"2024-01-16T15:19:24.627850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:02.507029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.623313+0000","last_change":"2024-01-16T15:19:24.616757+0000","last_active":"2024-01-16T15:19:24.623313+0000","last_peered":"2024-01-16T15:19:24.623313+0000","last_clean":"2024-01-16T15:19:24.623313+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:19:24.623313+0000","last_undegraded":"2024-01-16T15:19:24.623313+0000","last_fullsized":"2024-01-16T15:19:24.623313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:44.065214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642075+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:19:24.642075+0000","last_peered":"2024-01-16T15:19:24.642075+0000","last_clean":"2024-01-16T15:19:24.642075+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:19:24.642075+0000","last_undegraded":"2024-01-16T15:19:24.642075+0000","last_fullsized":"2024-01-16T15:19:24.642075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:30.175114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636104+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:19:24.636104+0000","last_peered":"2024-01-16T15:19:24.636104+0000","last_clean":"2024-01-16T15:19:24.636104+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:19:24.636104+0000","last_undegraded":"2024-01-16T15:19:24.636104+0000","last_fullsized":"2024-01-16T15:19:24.636104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:13.468787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168713+0000","last_change":"2024-01-16T15:19:24.625137+0000","last_active":"2024-01-16T15:19:26.168713+0000","last_peered":"2024-01-16T15:19:26.168713+0000","last_clean":"2024-01-16T15:19:26.168713+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:19:26.168713+0000","last_undegraded":"2024-01-16T15:19:26.168713+0000","last_fullsized":"2024-01-16T15:19:26.168713+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628098+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:19:24.628098+0000","last_peered":"2024-01-16T15:19:24.628098+0000","last_clean":"2024-01-16T15:19:24.628098+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:19:24.628098+0000","last_undegraded":"2024-01-16T15:19:24.628098+0000","last_fullsized":"2024-01-16T15:19:24.628098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:31.587040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627199+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:19:24.627199+0000","last_peered":"2024-01-16T15:19:24.627199+0000","last_clean":"2024-01-16T15:19:24.627199+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:19:24.627199+0000","last_undegraded":"2024-01-16T15:19:24.627199+0000","last_fullsized":"2024-01-16T15:19:24.627199+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:32.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628574+0000","last_change":"2024-01-16T15:19:24.624351+0000","last_active":"2024-01-16T15:19:24.628574+0000","last_peered":"2024-01-16T15:19:24.628574+0000","last_clean":"2024-01-16T15:19:24.628574+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:19:24.628574+0000","last_undegraded":"2024-01-16T15:19:24.628574+0000","last_fullsized":"2024-01-16T15:19:24.628574+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:59.580723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.618778+0000","last_change":"2024-01-16T15:19:24.613444+0000","last_active":"2024-01-16T15:19:24.618778+0000","last_peered":"2024-01-16T15:19:24.618778+0000","last_clean":"2024-01-16T15:19:24.618778+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:19:24.618778+0000","last_undegraded":"2024-01-16T15:19:24.618778+0000","last_fullsized":"2024-01-16T15:19:24.618778+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:06.791309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.620617+0000","last_change":"2024-01-16T15:19:24.614694+0000","last_active":"2024-01-16T15:19:24.620617+0000","last_peered":"2024-01-16T15:19:24.620617+0000","last_clean":"2024-01-16T15:19:24.620617+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:19:24.620617+0000","last_undegraded":"2024-01-16T15:19:24.620617+0000","last_fullsized":"2024-01-16T15:19:24.620617+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:58.872650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.169003+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:19:26.169003+0000","last_peered":"2024-01-16T15:19:26.169003+0000","last_clean":"2024-01-16T15:19:26.169003+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:19:26.169003+0000","last_undegraded":"2024-01-16T15:19:26.169003+0000","last_fullsized":"2024-01-16T15:19:26.169003+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626504+0000","last_change":"2024-01-16T15:19:24.629022+0000","last_active":"2024-01-16T15:19:25.626504+0000","last_peered":"2024-01-16T15:19:25.626504+0000","last_clean":"2024-01-16T15:19:25.626504+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:19:25.626504+0000","last_undegraded":"2024-01-16T15:19:25.626504+0000","last_fullsized":"2024-01-16T15:19:25.626504+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:22.444157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626946+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:19:24.626946+0000","last_peered":"2024-01-16T15:19:24.626946+0000","last_clean":"2024-01-16T15:19:24.626946+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:19:24.626946+0000","last_undegraded":"2024-01-16T15:19:24.626946+0000","last_fullsized":"2024-01-16T15:19:24.626946+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:01.542383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168872+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:19:26.168872+0000","last_peered":"2024-01-16T15:19:26.168872+0000","last_clean":"2024-01-16T15:19:26.168872+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:19:26.168872+0000","last_undegraded":"2024-01-16T15:19:26.168872+0000","last_fullsized":"2024-01-16T15:19:26.168872+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.169055+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:19:26.169055+0000","last_peered":"2024-01-16T15:19:26.169055+0000","last_clean":"2024-01-16T15:19:26.169055+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:19:26.169055+0000","last_undegraded":"2024-01-16T15:19:26.169055+0000","last_fullsized":"2024-01-16T15:19:26.169055+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628396+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:19:24.628396+0000","last_peered":"2024-01-16T15:19:24.628396+0000","last_clean":"2024-01-16T15:19:24.628396+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:19:24.628396+0000","last_undegraded":"2024-01-16T15:19:24.628396+0000","last_fullsized":"2024-01-16T15:19:24.628396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:11.461387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168099+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:19:26.168099+0000","last_peered":"2024-01-16T15:19:26.168099+0000","last_clean":"2024-01-16T15:19:26.168099+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:19:26.168099+0000","last_undegraded":"2024-01-16T15:19:26.168099+0000","last_fullsized":"2024-01-16T15:19:26.168099+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:07.051272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.632017+0000","last_change":"2024-01-16T15:19:24.626920+0000","last_active":"2024-01-16T15:19:24.632017+0000","last_peered":"2024-01-16T15:19:24.632017+0000","last_clean":"2024-01-16T15:19:24.632017+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:19:24.632017+0000","last_undegraded":"2024-01-16T15:19:24.632017+0000","last_fullsized":"2024-01-16T15:19:24.632017+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:45:17.563783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.169091+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:19:26.169091+0000","last_peered":"2024-01-16T15:19:26.169091+0000","last_clean":"2024-01-16T15:19:26.169091+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:19:26.169091+0000","last_undegraded":"2024-01-16T15:19:26.169091+0000","last_fullsized":"2024-01-16T15:19:26.169091+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.638727+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:19:24.638727+0000","last_peered":"2024-01-16T15:19:24.638727+0000","last_clean":"2024-01-16T15:19:24.638727+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:19:24.638727+0000","last_undegraded":"2024-01-16T15:19:24.638727+0000","last_fullsized":"2024-01-16T15:19:24.638727+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:22:30.752763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627851+0000","last_change":"2024-01-16T15:19:24.623489+0000","last_active":"2024-01-16T15:19:24.627851+0000","last_peered":"2024-01-16T15:19:24.627851+0000","last_clean":"2024-01-16T15:19:24.627851+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:19:24.627851+0000","last_undegraded":"2024-01-16T15:19:24.627851+0000","last_fullsized":"2024-01-16T15:19:24.627851+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:47.875736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.632768+0000","last_change":"2024-01-16T15:19:24.628737+0000","last_active":"2024-01-16T15:19:24.632768+0000","last_peered":"2024-01-16T15:19:24.632768+0000","last_clean":"2024-01-16T15:19:24.632768+0000","last_became_active":"2024-01-16T15:19:24.628439+0000","last_became_peered":"2024-01-16T15:19:24.628439+0000","last_unstale":"2024-01-16T15:19:24.632768+0000","last_undegraded":"2024-01-16T15:19:24.632768+0000","last_fullsized":"2024-01-16T15:19:24.632768+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168304+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:19:26.168304+0000","last_peered":"2024-01-16T15:19:26.168304+0000","last_clean":"2024-01-16T15:19:26.168304+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:19:26.168304+0000","last_undegraded":"2024-01-16T15:19:26.168304+0000","last_fullsized":"2024-01-16T15:19:26.168304+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:40.862803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.633416+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:19:24.633416+0000","last_peered":"2024-01-16T15:19:24.633416+0000","last_clean":"2024-01-16T15:19:24.633416+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:19:24.633416+0000","last_undegraded":"2024-01-16T15:19:24.633416+0000","last_fullsized":"2024-01-16T15:19:24.633416+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:55:44.343940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637840+0000","last_change":"2024-01-16T15:19:24.629910+0000","last_active":"2024-01-16T15:19:24.637840+0000","last_peered":"2024-01-16T15:19:24.637840+0000","last_clean":"2024-01-16T15:19:24.637840+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:19:24.637840+0000","last_undegraded":"2024-01-16T15:19:24.637840+0000","last_fullsized":"2024-01-16T15:19:24.637840+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:30.687663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.634847+0000","last_change":"2024-01-16T15:19:24.633567+0000","last_active":"2024-01-16T15:19:24.634847+0000","last_peered":"2024-01-16T15:19:24.634847+0000","last_clean":"2024-01-16T15:19:24.634847+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:19:24.634847+0000","last_undegraded":"2024-01-16T15:19:24.634847+0000","last_fullsized":"2024-01-16T15:19:24.634847+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:54.199001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627260+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:19:24.627260+0000","last_peered":"2024-01-16T15:19:24.627260+0000","last_clean":"2024-01-16T15:19:24.627260+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:19:24.627260+0000","last_undegraded":"2024-01-16T15:19:24.627260+0000","last_fullsized":"2024-01-16T15:19:24.627260+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:34.368248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636605+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:19:24.636605+0000","last_peered":"2024-01-16T15:19:24.636605+0000","last_clean":"2024-01-16T15:19:24.636605+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:19:24.636605+0000","last_undegraded":"2024-01-16T15:19:24.636605+0000","last_fullsized":"2024-01-16T15:19:24.636605+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.038677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.627087+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:19:25.627087+0000","last_peered":"2024-01-16T15:19:25.627087+0000","last_clean":"2024-01-16T15:19:25.627087+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:19:25.627087+0000","last_undegraded":"2024-01-16T15:19:25.627087+0000","last_fullsized":"2024-01-16T15:19:25.627087+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:25.905229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.633095+0000","last_change":"2024-01-16T15:19:23.621310+0000","last_active":"2024-01-16T15:19:24.633095+0000","last_peered":"2024-01-16T15:19:24.633095+0000","last_clean":"2024-01-16T15:19:24.633095+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:19:24.633095+0000","last_undegraded":"2024-01-16T15:19:24.633095+0000","last_fullsized":"2024-01-16T15:19:24.633095+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:19.958286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631382+0000","last_change":"2024-01-16T15:19:24.630161+0000","last_active":"2024-01-16T15:19:24.631382+0000","last_peered":"2024-01-16T15:19:24.631382+0000","last_clean":"2024-01-16T15:19:24.631382+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:19:24.631382+0000","last_undegraded":"2024-01-16T15:19:24.631382+0000","last_fullsized":"2024-01-16T15:19:24.631382+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:01.440713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636973+0000","last_change":"2024-01-16T15:19:23.630011+0000","last_active":"2024-01-16T15:19:24.636973+0000","last_peered":"2024-01-16T15:19:24.636973+0000","last_clean":"2024-01-16T15:19:24.636973+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:19:24.636973+0000","last_undegraded":"2024-01-16T15:19:24.636973+0000","last_fullsized":"2024-01-16T15:19:24.636973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:21.485125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626963+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:19:24.626963+0000","last_peered":"2024-01-16T15:19:24.626963+0000","last_clean":"2024-01-16T15:19:24.626963+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:19:24.626963+0000","last_undegraded":"2024-01-16T15:19:24.626963+0000","last_fullsized":"2024-01-16T15:19:24.626963+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:59.238148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.638258+0000","last_change":"2024-01-16T15:19:23.618597+0000","last_active":"2024-01-16T15:19:24.638258+0000","last_peered":"2024-01-16T15:19:24.638258+0000","last_clean":"2024-01-16T15:19:24.638258+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:19:24.638258+0000","last_undegraded":"2024-01-16T15:19:24.638258+0000","last_fullsized":"2024-01-16T15:19:24.638258+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:05.775312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642236+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:19:24.642236+0000","last_peered":"2024-01-16T15:19:24.642236+0000","last_clean":"2024-01-16T15:19:24.642236+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:19:24.642236+0000","last_undegraded":"2024-01-16T15:19:24.642236+0000","last_fullsized":"2024-01-16T15:19:24.642236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:45.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.635883+0000","last_change":"2024-01-16T15:19:23.618024+0000","last_active":"2024-01-16T15:19:24.635883+0000","last_peered":"2024-01-16T15:19:24.635883+0000","last_clean":"2024-01-16T15:19:24.635883+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:19:24.635883+0000","last_undegraded":"2024-01-16T15:19:24.635883+0000","last_fullsized":"2024-01-16T15:19:24.635883+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:50.476809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168310+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:19:26.168310+0000","last_peered":"2024-01-16T15:19:26.168310+0000","last_clean":"2024-01-16T15:19:26.168310+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:19:26.168310+0000","last_undegraded":"2024-01-16T15:19:26.168310+0000","last_fullsized":"2024-01-16T15:19:26.168310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637764+0000","last_change":"2024-01-16T15:19:23.622641+0000","last_active":"2024-01-16T15:19:24.637764+0000","last_peered":"2024-01-16T15:19:24.637764+0000","last_clean":"2024-01-16T15:19:24.637764+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:19:24.637764+0000","last_undegraded":"2024-01-16T15:19:24.637764+0000","last_fullsized":"2024-01-16T15:19:24.637764+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:39.846805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626373+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:19:24.626373+0000","last_peered":"2024-01-16T15:19:24.626373+0000","last_clean":"2024-01-16T15:19:24.626373+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:19:24.626373+0000","last_undegraded":"2024-01-16T15:19:24.626373+0000","last_fullsized":"2024-01-16T15:19:24.626373+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:32.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168172+0000","last_change":"2024-01-16T15:19:23.620941+0000","last_active":"2024-01-16T15:19:26.168172+0000","last_peered":"2024-01-16T15:19:26.168172+0000","last_clean":"2024-01-16T15:19:26.168172+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:19:26.168172+0000","last_undegraded":"2024-01-16T15:19:26.168172+0000","last_fullsized":"2024-01-16T15:19:26.168172+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:27.971375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627832+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:19:24.627832+0000","last_peered":"2024-01-16T15:19:24.627832+0000","last_clean":"2024-01-16T15:19:24.627832+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:19:24.627832+0000","last_undegraded":"2024-01-16T15:19:24.627832+0000","last_fullsized":"2024-01-16T15:19:24.627832+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:48:36.352635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.601586+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:19:24.601586+0000","last_peered":"2024-01-16T15:19:24.601586+0000","last_clean":"2024-01-16T15:19:24.601586+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:19:24.601586+0000","last_undegraded":"2024-01-16T15:19:24.601586+0000","last_fullsized":"2024-01-16T15:19:24.601586+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:07.254538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.633796+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:19:24.633796+0000","last_peered":"2024-01-16T15:19:24.633796+0000","last_clean":"2024-01-16T15:19:24.633796+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:19:24.633796+0000","last_undegraded":"2024-01-16T15:19:24.633796+0000","last_fullsized":"2024-01-16T15:19:24.633796+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:08:27.804532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.591363+0000","last_change":"2024-01-16T15:19:23.621670+0000","last_active":"2024-01-16T15:19:24.591363+0000","last_peered":"2024-01-16T15:19:24.591363+0000","last_clean":"2024-01-16T15:19:24.591363+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:19:24.591363+0000","last_undegraded":"2024-01-16T15:19:24.591363+0000","last_fullsized":"2024-01-16T15:19:24.591363+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:53.183004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.627124+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:19:25.627124+0000","last_peered":"2024-01-16T15:19:25.627124+0000","last_clean":"2024-01-16T15:19:25.627124+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:19:25.627124+0000","last_undegraded":"2024-01-16T15:19:25.627124+0000","last_fullsized":"2024-01-16T15:19:25.627124+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:56.499150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.603442+0000","last_change":"2024-01-16T15:19:23.624332+0000","last_active":"2024-01-16T15:19:24.603442+0000","last_peered":"2024-01-16T15:19:24.603442+0000","last_clean":"2024-01-16T15:19:24.603442+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:19:24.603442+0000","last_undegraded":"2024-01-16T15:19:24.603442+0000","last_fullsized":"2024-01-16T15:19:24.603442+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:56.995452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.632742+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:19:24.632742+0000","last_peered":"2024-01-16T15:19:24.632742+0000","last_clean":"2024-01-16T15:19:24.632742+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:19:24.632742+0000","last_undegraded":"2024-01-16T15:19:24.632742+0000","last_fullsized":"2024-01-16T15:19:24.632742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"109'3","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.646105+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:19:24.646105+0000","last_peered":"2024-01-16T15:19:24.646105+0000","last_clean":"2024-01-16T15:19:24.646105+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:19:24.646105+0000","last_undegraded":"2024-01-16T15:19:24.646105+0000","last_fullsized":"2024-01-16T15:19:24.646105+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:48.585086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631616+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:19:24.631616+0000","last_peered":"2024-01-16T15:19:24.631616+0000","last_clean":"2024-01-16T15:19:24.631616+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:19:24.631616+0000","last_undegraded":"2024-01-16T15:19:24.631616+0000","last_fullsized":"2024-01-16T15:19:24.631616+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:38:05.613201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"109'6","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.648369+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:19:24.648369+0000","last_peered":"2024-01-16T15:19:24.648369+0000","last_clean":"2024-01-16T15:19:24.648369+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:19:24.648369+0000","last_undegraded":"2024-01-16T15:19:24.648369+0000","last_fullsized":"2024-01-16T15:19:24.648369+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:52.253914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.635949+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:19:24.635949+0000","last_peered":"2024-01-16T15:19:24.635949+0000","last_clean":"2024-01-16T15:19:24.635949+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:19:24.635949+0000","last_undegraded":"2024-01-16T15:19:24.635949+0000","last_fullsized":"2024-01-16T15:19:24.635949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:12.009819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.641122+0000","last_change":"2024-01-16T15:19:23.617724+0000","last_active":"2024-01-16T15:19:24.641122+0000","last_peered":"2024-01-16T15:19:24.641122+0000","last_clean":"2024-01-16T15:19:24.641122+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:19:24.641122+0000","last_undegraded":"2024-01-16T15:19:24.641122+0000","last_fullsized":"2024-01-16T15:19:24.641122+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:35.823425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631437+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:19:24.631437+0000","last_peered":"2024-01-16T15:19:24.631437+0000","last_clean":"2024-01-16T15:19:24.631437+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:19:24.631437+0000","last_undegraded":"2024-01-16T15:19:24.631437+0000","last_fullsized":"2024-01-16T15:19:24.631437+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626138+0000","last_change":"2024-01-16T15:19:23.624683+0000","last_active":"2024-01-16T15:19:25.626138+0000","last_peered":"2024-01-16T15:19:25.626138+0000","last_clean":"2024-01-16T15:19:25.626138+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:19:25.626138+0000","last_undegraded":"2024-01-16T15:19:25.626138+0000","last_fullsized":"2024-01-16T15:19:25.626138+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:38.379463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628502+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:19:24.628502+0000","last_peered":"2024-01-16T15:19:24.628502+0000","last_clean":"2024-01-16T15:19:24.628502+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:19:24.628502+0000","last_undegraded":"2024-01-16T15:19:24.628502+0000","last_fullsized":"2024-01-16T15:19:24.628502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:07.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.645127+0000","last_change":"2024-01-16T15:19:23.625591+0000","last_active":"2024-01-16T15:19:24.645127+0000","last_peered":"2024-01-16T15:19:24.645127+0000","last_clean":"2024-01-16T15:19:24.645127+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:19:24.645127+0000","last_undegraded":"2024-01-16T15:19:24.645127+0000","last_fullsized":"2024-01-16T15:19:24.645127+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:27.727635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.620265+0000","last_change":"2024-01-16T15:19:24.617249+0000","last_active":"2024-01-16T15:19:24.620265+0000","last_peered":"2024-01-16T15:19:24.620265+0000","last_clean":"2024-01-16T15:19:24.620265+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:19:24.620265+0000","last_undegraded":"2024-01-16T15:19:24.620265+0000","last_fullsized":"2024-01-16T15:19:24.620265+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:40:11.125336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.645029+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:19:24.645029+0000","last_peered":"2024-01-16T15:19:24.645029+0000","last_clean":"2024-01-16T15:19:24.645029+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:19:24.645029+0000","last_undegraded":"2024-01-16T15:19:24.645029+0000","last_fullsized":"2024-01-16T15:19:24.645029+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:27.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626866+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:19:24.626866+0000","last_peered":"2024-01-16T15:19:24.626866+0000","last_clean":"2024-01-16T15:19:24.626866+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:19:24.626866+0000","last_undegraded":"2024-01-16T15:19:24.626866+0000","last_fullsized":"2024-01-16T15:19:24.626866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:08.024918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636769+0000","last_change":"2024-01-16T15:19:23.622230+0000","last_active":"2024-01-16T15:19:24.636769+0000","last_peered":"2024-01-16T15:19:24.636769+0000","last_clean":"2024-01-16T15:19:24.636769+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:19:24.636769+0000","last_undegraded":"2024-01-16T15:19:24.636769+0000","last_fullsized":"2024-01-16T15:19:24.636769+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626560+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:19:24.626560+0000","last_peered":"2024-01-16T15:19:24.626560+0000","last_clean":"2024-01-16T15:19:24.626560+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:19:24.626560+0000","last_undegraded":"2024-01-16T15:19:24.626560+0000","last_fullsized":"2024-01-16T15:19:24.626560+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:45.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626574+0000","last_change":"2024-01-16T15:19:23.626903+0000","last_active":"2024-01-16T15:19:25.626574+0000","last_peered":"2024-01-16T15:19:25.626574+0000","last_clean":"2024-01-16T15:19:25.626574+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:19:25.626574+0000","last_undegraded":"2024-01-16T15:19:25.626574+0000","last_fullsized":"2024-01-16T15:19:25.626574+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:58.751536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.629627+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:19:24.629627+0000","last_peered":"2024-01-16T15:19:24.629627+0000","last_clean":"2024-01-16T15:19:24.629627+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:19:24.629627+0000","last_undegraded":"2024-01-16T15:19:24.629627+0000","last_fullsized":"2024-01-16T15:19:24.629627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:24.185677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.624043+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:19:24.624043+0000","last_peered":"2024-01-16T15:19:24.624043+0000","last_clean":"2024-01-16T15:19:24.624043+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:19:24.624043+0000","last_undegraded":"2024-01-16T15:19:24.624043+0000","last_fullsized":"2024-01-16T15:19:24.624043+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:00.825374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.623270+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:19:24.623270+0000","last_peered":"2024-01-16T15:19:24.623270+0000","last_clean":"2024-01-16T15:19:24.623270+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:19:24.623270+0000","last_undegraded":"2024-01-16T15:19:24.623270+0000","last_fullsized":"2024-01-16T15:19:24.623270+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.807749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642249+0000","last_change":"2024-01-16T15:19:23.621285+0000","last_active":"2024-01-16T15:19:24.642249+0000","last_peered":"2024-01-16T15:19:24.642249+0000","last_clean":"2024-01-16T15:19:24.642249+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:19:24.642249+0000","last_undegraded":"2024-01-16T15:19:24.642249+0000","last_fullsized":"2024-01-16T15:19:24.642249+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:44.070274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.635611+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:19:24.635611+0000","last_peered":"2024-01-16T15:19:24.635611+0000","last_clean":"2024-01-16T15:19:24.635611+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:19:24.635611+0000","last_undegraded":"2024-01-16T15:19:24.635611+0000","last_fullsized":"2024-01-16T15:19:24.635611+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:32.314064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.591226+0000","last_change":"2024-01-16T15:19:23.619619+0000","last_active":"2024-01-16T15:19:24.591226+0000","last_peered":"2024-01-16T15:19:24.591226+0000","last_clean":"2024-01-16T15:19:24.591226+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:19:24.591226+0000","last_undegraded":"2024-01-16T15:19:24.591226+0000","last_fullsized":"2024-01-16T15:19:24.591226+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:15.166720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626053+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:19:24.626053+0000","last_peered":"2024-01-16T15:19:24.626053+0000","last_clean":"2024-01-16T15:19:24.626053+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:19:24.626053+0000","last_undegraded":"2024-01-16T15:19:24.626053+0000","last_fullsized":"2024-01-16T15:19:24.626053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:40.922371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.604218+0000","last_change":"2024-01-16T15:19:23.620635+0000","last_active":"2024-01-16T15:19:24.604218+0000","last_peered":"2024-01-16T15:19:24.604218+0000","last_clean":"2024-01-16T15:19:24.604218+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:19:24.604218+0000","last_undegraded":"2024-01-16T15:19:24.604218+0000","last_fullsized":"2024-01-16T15:19:24.604218+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628521+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:19:24.628521+0000","last_peered":"2024-01-16T15:19:24.628521+0000","last_clean":"2024-01-16T15:19:24.628521+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:19:24.628521+0000","last_undegraded":"2024-01-16T15:19:24.628521+0000","last_fullsized":"2024-01-16T15:19:24.628521+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:28.987372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.601964+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:19:24.601964+0000","last_peered":"2024-01-16T15:19:24.601964+0000","last_clean":"2024-01-16T15:19:24.601964+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:19:24.601964+0000","last_undegraded":"2024-01-16T15:19:24.601964+0000","last_fullsized":"2024-01-16T15:19:24.601964+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:48.621454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626679+0000","last_change":"2024-01-16T15:19:24.638752+0000","last_active":"2024-01-16T15:19:25.626679+0000","last_peered":"2024-01-16T15:19:25.626679+0000","last_clean":"2024-01-16T15:19:25.626679+0000","last_became_active":"2024-01-16T15:19:24.638466+0000","last_became_peered":"2024-01-16T15:19:24.638466+0000","last_unstale":"2024-01-16T15:19:25.626679+0000","last_undegraded":"2024-01-16T15:19:25.626679+0000","last_fullsized":"2024-01-16T15:19:25.626679+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:17.797680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.643355+0000","last_change":"2024-01-16T15:19:23.630170+0000","last_active":"2024-01-16T15:19:24.643355+0000","last_peered":"2024-01-16T15:19:24.643355+0000","last_clean":"2024-01-16T15:19:24.643355+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:19:24.643355+0000","last_undegraded":"2024-01-16T15:19:24.643355+0000","last_fullsized":"2024-01-16T15:19:24.643355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:06.035275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642865+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:19:24.642865+0000","last_peered":"2024-01-16T15:19:24.642865+0000","last_clean":"2024-01-16T15:19:24.642865+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:19:24.642865+0000","last_undegraded":"2024-01-16T15:19:24.642865+0000","last_fullsized":"2024-01-16T15:19:24.642865+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:19:49.999663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626900+0000","last_change":"2024-01-16T15:19:23.623778+0000","last_active":"2024-01-16T15:19:25.626900+0000","last_peered":"2024-01-16T15:19:25.626900+0000","last_clean":"2024-01-16T15:19:25.626900+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:19:25.626900+0000","last_undegraded":"2024-01-16T15:19:25.626900+0000","last_fullsized":"2024-01-16T15:19:25.626900+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:53.783590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637131+0000","last_change":"2024-01-16T15:19:24.628361+0000","last_active":"2024-01-16T15:19:24.637131+0000","last_peered":"2024-01-16T15:19:24.637131+0000","last_clean":"2024-01-16T15:19:24.637131+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:19:24.637131+0000","last_undegraded":"2024-01-16T15:19:24.637131+0000","last_fullsized":"2024-01-16T15:19:24.637131+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:32.038856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.601994+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:19:24.601994+0000","last_peered":"2024-01-16T15:19:24.601994+0000","last_clean":"2024-01-16T15:19:24.601994+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:19:24.601994+0000","last_undegraded":"2024-01-16T15:19:24.601994+0000","last_fullsized":"2024-01-16T15:19:24.601994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:40.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.625901+0000","last_change":"2024-01-16T15:19:24.623591+0000","last_active":"2024-01-16T15:19:25.625901+0000","last_peered":"2024-01-16T15:19:25.625901+0000","last_clean":"2024-01-16T15:19:25.625901+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:19:25.625901+0000","last_undegraded":"2024-01-16T15:19:25.625901+0000","last_fullsized":"2024-01-16T15:19:25.625901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:23:57.706980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.614526+0000","last_change":"2024-01-16T15:19:23.612616+0000","last_active":"2024-01-16T15:19:24.614526+0000","last_peered":"2024-01-16T15:19:24.614526+0000","last_clean":"2024-01-16T15:19:24.614526+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:19:24.614526+0000","last_undegraded":"2024-01-16T15:19:24.614526+0000","last_fullsized":"2024-01-16T15:19:24.614526+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:19.160736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168410+0000","last_change":"2024-01-16T15:19:24.626321+0000","last_active":"2024-01-16T15:19:26.168410+0000","last_peered":"2024-01-16T15:19:26.168410+0000","last_clean":"2024-01-16T15:19:26.168410+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:19:26.168410+0000","last_undegraded":"2024-01-16T15:19:26.168410+0000","last_fullsized":"2024-01-16T15:19:26.168410+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.624234+0000","last_change":"2024-01-16T15:19:23.624071+0000","last_active":"2024-01-16T15:19:24.624234+0000","last_peered":"2024-01-16T15:19:24.624234+0000","last_clean":"2024-01-16T15:19:24.624234+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:19:24.624234+0000","last_undegraded":"2024-01-16T15:19:24.624234+0000","last_fullsized":"2024-01-16T15:19:24.624234+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:45.204823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168798+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:19:26.168798+0000","last_peered":"2024-01-16T15:19:26.168798+0000","last_clean":"2024-01-16T15:19:26.168798+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:19:26.168798+0000","last_undegraded":"2024-01-16T15:19:26.168798+0000","last_fullsized":"2024-01-16T15:19:26.168798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168926+0000","last_change":"2024-01-16T15:19:23.619253+0000","last_active":"2024-01-16T15:19:26.168926+0000","last_peered":"2024-01-16T15:19:26.168926+0000","last_clean":"2024-01-16T15:19:26.168926+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:19:26.168926+0000","last_undegraded":"2024-01-16T15:19:26.168926+0000","last_fullsized":"2024-01-16T15:19:26.168926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:16.781683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.629292+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:19:24.629292+0000","last_peered":"2024-01-16T15:19:24.629292+0000","last_clean":"2024-01-16T15:19:24.629292+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:19:24.629292+0000","last_undegraded":"2024-01-16T15:19:24.629292+0000","last_fullsized":"2024-01-16T15:19:24.629292+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:19:10.642184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.591234+0000","last_change":"2024-01-16T15:19:23.629871+0000","last_active":"2024-01-16T15:19:24.591234+0000","last_peered":"2024-01-16T15:19:24.591234+0000","last_clean":"2024-01-16T15:19:24.591234+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:19:24.591234+0000","last_undegraded":"2024-01-16T15:19:24.591234+0000","last_fullsized":"2024-01-16T15:19:24.591234+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:05:17.872229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.624488+0000","last_change":"2024-01-16T15:19:24.621286+0000","last_active":"2024-01-16T15:19:24.624488+0000","last_peered":"2024-01-16T15:19:24.624488+0000","last_clean":"2024-01-16T15:19:24.624488+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:19:24.624488+0000","last_undegraded":"2024-01-16T15:19:24.624488+0000","last_fullsized":"2024-01-16T15:19:24.624488+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:10.983492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.602334+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:19:24.602334+0000","last_peered":"2024-01-16T15:19:24.602334+0000","last_clean":"2024-01-16T15:19:24.602334+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:19:24.602334+0000","last_undegraded":"2024-01-16T15:19:24.602334+0000","last_fullsized":"2024-01-16T15:19:24.602334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:01.099875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627015+0000","last_change":"2024-01-16T15:19:24.621294+0000","last_active":"2024-01-16T15:19:24.627015+0000","last_peered":"2024-01-16T15:19:24.627015+0000","last_clean":"2024-01-16T15:19:24.627015+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:19:24.627015+0000","last_undegraded":"2024-01-16T15:19:24.627015+0000","last_fullsized":"2024-01-16T15:19:24.627015+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:58.723198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.623375+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:19:24.623375+0000","last_peered":"2024-01-16T15:19:24.623375+0000","last_clean":"2024-01-16T15:19:24.623375+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:19:24.623375+0000","last_undegraded":"2024-01-16T15:19:24.623375+0000","last_fullsized":"2024-01-16T15:19:24.623375+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:59.676130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":253,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-16T15:19:23.597516+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:19:23.597516+0000","last_peered":"2024-01-16T15:19:23.597516+0000","last_clean":"2024-01-16T15:19:23.597516+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:19:23.597516+0000","last_undegraded":"2024-01-16T15:19:23.597516+0000","last_fullsized":"2024-01-16T15:19:23.597516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509678,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68544,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303296,"statfs":{"total":96636764160,"available":96566575104,"internally_reserved":0,"allocated":1363968,"data_stored":588997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.3340000000000001,"15min":1.6559999999999999},"min":{"1min":0.80200000000000005,"5min":0.65600000000000003,"15min":0.57099999999999995},"max":{"1min":1.8,"5min":19.224,"15min":27.853999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.161,"15min":1.7030000000000001},"min":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.47199999999999998},"max":{"1min":1.742,"5min":1.742,"15min":34.939999999999998},"last":1.056}]},{"osd":1,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.218,"15min":1.4630000000000001},"min":{"1min":0.91800000000000004,"5min":0.79200000000000004,"15min":0.374},"max":{"1min":1.724,"5min":2.5419999999999998,"15min":13.859},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.496,"15min":1.7250000000000001},"min":{"1min":0.96199999999999997,"5min":0.68400000000000005,"15min":0.502},"max":{"1min":1.5589999999999999,"5min":19.178000000000001,"15min":33.225000000000001},"last":0.996}]},{"osd":2,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.526,"15min":1.647},"min":{"1min":0.85299999999999998,"5min":0.74299999999999999,"15min":0.55900000000000005},"max":{"1min":1.679,"5min":19.126000000000001,"15min":19.126000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.321,"15min":1.7549999999999999},"min":{"1min":0.89000000000000001,"5min":0.69499999999999995,"15min":0.53100000000000003},"max":{"1min":1.6850000000000001,"5min":2.9750000000000001,"15min":35.011000000000003},"last":0.86699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3240000000000001,"15min":1.6359999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.374},"max":{"1min":1.8240000000000001,"5min":2.5339999999999998,"15min":16.353000000000002},"last":1.103},{"interface":"front","average":{"1min":1.47,"5min":1.6970000000000001,"15min":1.8009999999999999},"min":{"1min":1.0009999999999999,"5min":0.88900000000000001,"15min":0.57899999999999996},"max":{"1min":1.9750000000000001,"5min":19.268999999999998,"15min":19.268999999999998},"last":1.2270000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2769999999999999,"15min":1.8109999999999999},"min":{"1min":0.752,"5min":0.69099999999999995,"15min":0.40799999999999997},"max":{"1min":1.669,"5min":1.984,"15min":34.972999999999999},"last":1.272},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3400000000000001,"15min":1.7609999999999999},"min":{"1min":0.85699999999999998,"5min":0.71399999999999997,"15min":0.40400000000000003},"max":{"1min":1.9239999999999999,"5min":2.649,"15min":16.003},"last":1.173}]},{"osd":5,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5269999999999999,"15min":1.9710000000000001},"min":{"1min":1.048,"5min":0.79400000000000004,"15min":0.58099999999999996},"max":{"1min":1.825,"5min":2.851,"15min":23.760999999999999},"last":1.373},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.6950000000000001,"15min":1.9370000000000001},"min":{"1min":1.175,"5min":0.55100000000000005,"15min":0.52400000000000002},"max":{"1min":2.1680000000000001,"5min":19.036999999999999,"15min":23.716999999999999},"last":1.4339999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":2.0019999999999998},"min":{"1min":0.92600000000000005,"5min":0.747,"15min":0.48099999999999998},"max":{"1min":1.8149999999999999,"5min":2.1539999999999999,"15min":34.889000000000003},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.5760000000000001,"15min":2.153},"min":{"1min":1.169,"5min":0.86599999999999999,"15min":0.47599999999999998},"max":{"1min":1.962,"5min":2.7549999999999999,"15min":46.305},"last":1.589}]}]},{"osd":4,"up_from":15,"seq":64424509680,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68540,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303300,"statfs":{"total":96636764160,"available":96566579200,"internally_reserved":0,"allocated":1359872,"data_stored":588927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.208,"15min":1.5069999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.38700000000000001},"max":{"1min":1.7989999999999999,"5min":2.0350000000000001,"15min":22.757999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2190000000000001,"15min":1.2969999999999999},"min":{"1min":0.81999999999999995,"5min":0.69799999999999995,"15min":0.432},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":14.034000000000001},"last":0.873}]},{"osd":1,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1739999999999999,"15min":1.548},"min":{"1min":0.875,"5min":0.71899999999999997,"15min":0.51600000000000001},"max":{"1min":1.7150000000000001,"5min":1.851,"15min":18.347000000000001},"last":1.02},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2769999999999999,"15min":1.484},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.498},"max":{"1min":1.7110000000000001,"5min":2.673,"15min":18.858000000000001},"last":1.1000000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.258,"15min":1.653},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.45600000000000002},"max":{"1min":1.6879999999999999,"5min":1.768,"15min":17.651},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.371,"15min":1.726},"min":{"1min":1.0429999999999999,"5min":0.83499999999999996,"15min":0.52600000000000002},"max":{"1min":2.0019999999999998,"5min":2.7280000000000002,"15min":29.161000000000001},"last":1.208}]},{"osd":3,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.377,"15min":1.5740000000000001},"min":{"1min":0.83699999999999997,"5min":0.76200000000000001,"15min":0.41699999999999998},"max":{"1min":1.7629999999999999,"5min":2.6989999999999998,"15min":26.741},"last":1.361},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3340000000000001,"15min":1.7350000000000001},"min":{"1min":1.093,"5min":0.78500000000000003,"15min":0.54800000000000004},"max":{"1min":1.905,"5min":1.905,"15min":26.701000000000001},"last":1.1779999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4359999999999999,"15min":1.6930000000000001},"min":{"1min":0.96299999999999997,"5min":0.88500000000000001,"15min":0.441},"max":{"1min":1.6950000000000001,"5min":2.7730000000000001,"15min":18.919},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.4710000000000001,"15min":1.6479999999999999},"min":{"1min":1.054,"5min":0.78900000000000003,"15min":0.36399999999999999},"max":{"1min":1.782,"5min":2.0270000000000001,"15min":26.597000000000001},"last":1.254}]},{"osd":6,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.381,"15min":1.8959999999999999},"min":{"1min":1.0129999999999999,"5min":0.98899999999999999,"15min":0.41499999999999998},"max":{"1min":1.6359999999999999,"5min":5.0750000000000002,"15min":32.649999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.518,"15min":1.7989999999999999},"min":{"1min":1.085,"5min":0.78200000000000003,"15min":0.48099999999999998},"max":{"1min":1.716,"5min":2.8170000000000002,"15min":24.094999999999999},"last":1.716}]},{"osd":7,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4810000000000001,"15min":1.762},"min":{"1min":1.2250000000000001,"5min":1.0489999999999999,"15min":0.41699999999999998},"max":{"1min":1.667,"5min":1.8899999999999999,"15min":26.683},"last":1.667},{"interface":"front","average":{"1min":1.464,"5min":1.5169999999999999,"15min":1.853},"min":{"1min":1.143,"5min":1.093,"15min":0.46500000000000002},"max":{"1min":1.6579999999999999,"5min":2.2669999999999999,"15min":17.765000000000001},"last":1.6200000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73276,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298564,"statfs":{"total":96636764160,"available":96561729536,"internally_reserved":0,"allocated":1949696,"data_stored":1173583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.232,"15min":1.538},"min":{"1min":0.70999999999999996,"5min":0.65400000000000003,"15min":0.36899999999999999},"max":{"1min":1.6599999999999999,"5min":2.1779999999999999,"15min":15.661},"last":1.675},{"interface":"front","average":{"1min":1.242,"5min":1.23,"15min":1.5820000000000001},"min":{"1min":0.97399999999999998,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":1.4330000000000001,"5min":2.29,"15min":30.126000000000001},"last":1.4910000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2450000000000001,"15min":1.464},"min":{"1min":0.94999999999999996,"5min":0.63200000000000001,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.085,"15min":24.338000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.302,"5min":1.2629999999999999,"15min":1.6140000000000001},"min":{"1min":0.93300000000000005,"5min":0.745,"15min":0.46200000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":21.298999999999999},"last":1.0369999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.286,"15min":1.496},"min":{"1min":0.97299999999999998,"5min":0.71899999999999997,"15min":0.51500000000000001},"max":{"1min":1.5669999999999999,"5min":3.726,"15min":12.474},"last":1.538},{"interface":"front","average":{"1min":1.387,"5min":1.55,"15min":1.7230000000000001},"min":{"1min":0.79900000000000004,"5min":0.69299999999999995,"15min":0.54700000000000004},"max":{"1min":1.8240000000000001,"5min":19.763999999999999,"15min":42.165999999999997},"last":0.95999999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.337,"15min":1.581},"min":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.47699999999999998},"max":{"1min":1.6499999999999999,"5min":1.887,"15min":12.58},"last":1.375},{"interface":"front","average":{"1min":1.492,"5min":1.508,"15min":1.8100000000000001},"min":{"1min":1.1160000000000001,"5min":0.78600000000000003,"15min":0.54600000000000004},"max":{"1min":2.3180000000000001,"5min":6.6029999999999998,"15min":42.222000000000001},"last":1.24}]},{"osd":4,"last update":"Tue Jan 16 15:19:25 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3080000000000001,"15min":1.5600000000000001},"min":{"1min":0.90800000000000003,"5min":0.79700000000000004,"15min":0.42799999999999999},"max":{"1min":1.8799999999999999,"5min":2.0219999999999998,"15min":14.659000000000001},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.419,"15min":1.7929999999999999},"min":{"1min":0.95699999999999996,"5min":0.57199999999999995,"15min":0.38800000000000001},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":42.276000000000003},"last":1.4430000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.284,"15min":1.7989999999999999},"min":{"1min":0.96199999999999997,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":26.23},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.48,"15min":1.8089999999999999},"min":{"1min":1.1319999999999999,"5min":0.89200000000000002,"15min":0.41099999999999998},"max":{"1min":1.954,"5min":1.9630000000000001,"15min":21.312000000000001},"last":1.7709999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:19:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.54,"15min":1.855},"min":{"1min":1.1579999999999999,"5min":0.67100000000000004,"15min":0.501},"max":{"1min":2.653,"5min":2.653,"15min":42.090000000000003},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4970000000000001,"15min":1.9319999999999999},"min":{"1min":1.125,"5min":0.59099999999999997,"15min":0.40799999999999997},"max":{"1min":2.1709999999999998,"5min":2.2050000000000001,"15min":29.991},"last":1.819}]}]},{"osd":5,"up_from":14,"seq":60129542383,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64380,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307460,"statfs":{"total":96636764160,"available":96570839040,"internally_reserved":0,"allocated":1355776,"data_stored":583203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1779999999999999,"15min":1.5609999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.32900000000000001},"max":{"1min":1.742,"5min":6.0720000000000001,"15min":19.382999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.3160000000000001,"15min":1.363},"min":{"1min":0.76100000000000001,"5min":0.66700000000000004,"15min":0.57099999999999995},"max":{"1min":1.5760000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2,"15min":1.6399999999999999},"min":{"1min":0.84199999999999997,"5min":0.70699999999999996,"15min":0.52000000000000002},"max":{"1min":1.53,"5min":3.8109999999999999,"15min":17.693999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.4510000000000001,"15min":1.7250000000000001},"min":{"1min":0.82199999999999995,"5min":0.79600000000000004,"15min":0.432},"max":{"1min":1.647,"5min":17.062000000000001,"15min":20.645},"last":1.2230000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.425,"15min":1.7310000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.42499999999999999},"max":{"1min":1.825,"5min":5.8810000000000002,"15min":21.138000000000002},"last":1.407},{"interface":"front","average":{"1min":1.341,"5min":1.4019999999999999,"15min":1.5169999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.58099999999999996},"max":{"1min":1.7050000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.2749999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.341,"15min":1.8029999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.59799999999999998},"max":{"1min":1.748,"5min":2.4769999999999999,"15min":23.109000000000002},"last":1.038},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4350000000000001,"15min":1.766},"min":{"1min":0.747,"5min":0.72899999999999998,"15min":0.47399999999999998},"max":{"1min":1.6910000000000001,"5min":5.9489999999999998,"15min":19.303000000000001},"last":0.99199999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5069999999999999,"15min":1.8779999999999999},"min":{"1min":0.91900000000000004,"5min":0.85799999999999998,"15min":0.442},"max":{"1min":1.5880000000000001,"5min":15.975,"15min":24.436},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.4279999999999999,"15min":1.7509999999999999},"min":{"1min":0.98299999999999998,"5min":0.746,"15min":0.434},"max":{"1min":1.669,"5min":5.6349999999999998,"15min":19.126000000000001},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.446,"15min":1.6299999999999999},"min":{"1min":0.84799999999999998,"5min":0.77000000000000002,"15min":0.46500000000000002},"max":{"1min":1.7090000000000001,"5min":12.914,"15min":21.192},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.6120000000000001,"15min":2.0249999999999999},"min":{"1min":1.151,"5min":0.85999999999999999,"15min":0.48599999999999999},"max":{"1min":1.7789999999999999,"5min":16.552,"15min":27.042000000000002},"last":1.1819999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.55,"15min":1.9179999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.51200000000000001},"max":{"1min":1.7769999999999999,"5min":13.034000000000001,"15min":32.405999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5509999999999999,"15min":2.0659999999999998},"min":{"1min":0.71799999999999997,"5min":0.70899999999999996,"15min":0.56000000000000005},"max":{"1min":1.8169999999999999,"5min":2.1429999999999998,"15min":24.367000000000001},"last":1.5029999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542383,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64388,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307452,"statfs":{"total":96636764160,"available":96570830848,"internally_reserved":0,"allocated":1363968,"data_stored":584281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.98899999999999999,"15min":1.29},"min":{"1min":0.69799999999999995,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":1.5249999999999999,"5min":2.02,"15min":31.562999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.1930000000000001,"15min":1.1950000000000001},"min":{"1min":0.73299999999999998,"5min":0.504,"15min":0.39600000000000002},"max":{"1min":1.6359999999999999,"5min":7.1429999999999998,"15min":11.361000000000001},"last":1.1479999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.1160000000000001,"15min":1.1479999999999999},"min":{"1min":0.71399999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.282,"5min":10.856999999999999,"15min":15.569000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.4470000000000001},"min":{"1min":0.86299999999999999,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.7,"5min":2.0089999999999999,"15min":28.352},"last":1.252}]},{"osd":2,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.488,"15min":1.7230000000000001},"min":{"1min":1.004,"5min":0.57399999999999995,"15min":0.47799999999999998},"max":{"1min":1.732,"5min":7.8159999999999998,"15min":39.265999999999998},"last":1.363},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2849999999999999,"15min":1.341},"min":{"1min":0.90900000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5760000000000001,"5min":7.9160000000000004,"15min":13.526},"last":1.3899999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3859999999999999,"15min":1.546},"min":{"1min":1.099,"5min":0.73799999999999999,"15min":0.53800000000000003},"max":{"1min":1.9179999999999999,"5min":4.5570000000000004,"15min":20.978999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.514,"15min":1.863},"min":{"1min":1.2909999999999999,"5min":0.93999999999999995,"15min":0.72999999999999998},"max":{"1min":2.165,"5min":2.165,"15min":28.547999999999998},"last":1.5780000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.603,"15min":1.798},"min":{"1min":1.0209999999999999,"5min":0.77800000000000002,"15min":0.67000000000000004},"max":{"1min":1.8009999999999999,"5min":14.003,"15min":20.663},"last":1.095},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.591,"15min":1.9610000000000001},"min":{"1min":1.056,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":28.588000000000001},"last":1.4410000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.544,"15min":1.736},"min":{"1min":1.115,"5min":0.85299999999999998,"15min":0.69799999999999995},"max":{"1min":2.0870000000000002,"5min":7.0060000000000002,"15min":13.691000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.6899999999999999,"15min":1.9419999999999999},"min":{"1min":1.21,"5min":0.83399999999999996,"15min":0.76400000000000001},"max":{"1min":2.0619999999999998,"5min":15.114000000000001,"15min":26.698},"last":1.208}]},{"osd":7,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.7909999999999999,"15min":1.8149999999999999},"min":{"1min":1.1899999999999999,"5min":0.81899999999999995,"15min":0.60499999999999998},"max":{"1min":2.1829999999999998,"5min":12.378,"15min":13.622},"last":1.609},{"interface":"front","average":{"1min":1.655,"5min":1.677,"15min":2.165},"min":{"1min":1.1519999999999999,"5min":0.90400000000000003,"15min":0.80400000000000005},"max":{"1min":2.0009999999999999,"5min":4.6029999999999998,"15min":49.811},"last":1.4810000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542382,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76856,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294984,"statfs":{"total":96636764160,"available":96558063616,"internally_reserved":0,"allocated":1347584,"data_stored":582599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.93899999999999995,"15min":1.038},"min":{"1min":0.59799999999999998,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.4470000000000001,"5min":1.982,"15min":18.289999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.087,"5min":1.0840000000000001,"15min":1.1180000000000001},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.28599999999999998},"max":{"1min":1.5489999999999999,"5min":2.4790000000000001,"15min":7.9960000000000004},"last":1.165}]},{"osd":1,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.1140000000000001,"15min":1.1910000000000001},"min":{"1min":0.68100000000000005,"5min":0.501,"15min":0.40300000000000002},"max":{"1min":1.4650000000000001,"5min":1.931,"15min":23.135999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.214,"15min":1.23},"min":{"1min":0.70399999999999996,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.7629999999999999,"5min":3.012,"15min":4.8849999999999998},"last":1.262}]},{"osd":3,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.218,"15min":1.1890000000000001},"min":{"1min":0.64900000000000002,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":1.5720000000000001,"5min":7.306,"15min":18.838000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.73699999999999999,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.7549999999999999,"5min":2.6030000000000002,"15min":19.375},"last":0.98499999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.397,"15min":1.587},"min":{"1min":0.84699999999999998,"5min":0.79300000000000004,"15min":0.59999999999999998},"max":{"1min":1.8,"5min":5.9530000000000003,"15min":23.225000000000001},"last":1.228},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.474,"15min":1.5800000000000001},"min":{"1min":1.125,"5min":0.97399999999999998,"15min":0.59199999999999997},"max":{"1min":2.2080000000000002,"5min":3.4460000000000002,"15min":9.8960000000000008},"last":1.3480000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4790000000000001,"15min":1.619},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.70999999999999996},"max":{"1min":2.113,"5min":7.7640000000000002,"15min":20.157},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5329999999999999,"15min":1.593},"min":{"1min":0.95499999999999996,"5min":0.84299999999999997,"15min":0.623},"max":{"1min":2.2320000000000002,"5min":7.9820000000000002,"15min":25.004000000000001},"last":1.633}]},{"osd":6,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5740000000000001,"15min":1.736},"min":{"1min":0.92000000000000004,"5min":0.89400000000000002,"15min":0.70499999999999996},"max":{"1min":2.1709999999999998,"5min":7.6630000000000003,"15min":34.523000000000003},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.663,"15min":1.8300000000000001},"min":{"1min":1.1080000000000001,"5min":0.81100000000000005,"15min":0.74099999999999999},"max":{"1min":2.0409999999999999,"5min":7.8680000000000003,"15min":28.760999999999999},"last":1.7330000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.534,"15min":1.784},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.64100000000000001},"max":{"1min":1.984,"5min":3.1219999999999999,"15min":22.834},"last":1.548},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5960000000000001,"15min":2.0990000000000002},"min":{"1min":1.095,"5min":0.91000000000000003,"15min":0.82799999999999996},"max":{"1min":1.871,"5min":2.9350000000000001,"15min":39.685000000000002},"last":1.4059999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76852,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294988,"statfs":{"total":96636764160,"available":96558067712,"internally_reserved":0,"allocated":1355776,"data_stored":582723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.95099999999999996,"15min":1.3009999999999999},"min":{"1min":0.67600000000000005,"5min":0.48599999999999999,"15min":0.28100000000000003},"max":{"1min":1.474,"5min":2.024,"15min":31.178999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.125,"15min":1.306},"min":{"1min":0.79000000000000004,"5min":0.47299999999999998,"15min":0.247},"max":{"1min":1.4910000000000001,"5min":2.7679999999999998,"15min":26.859999999999999},"last":1.284}]},{"osd":2,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.147,"15min":1.163},"min":{"1min":0.72699999999999998,"5min":0.45400000000000001,"15min":0.37},"max":{"1min":1.6899999999999999,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":1.173},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3819999999999999,"15min":1.538},"min":{"1min":1.032,"5min":0.40100000000000002,"15min":0.39000000000000001},"max":{"1min":1.587,"5min":3.5600000000000001,"15min":27.408000000000001},"last":1.7330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3180000000000001,"15min":1.655},"min":{"1min":0.98599999999999999,"5min":0.56299999999999994,"15min":0.443},"max":{"1min":1.7849999999999999,"5min":2.238,"15min":27.609999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.24,"15min":1.6539999999999999},"min":{"1min":0.86199999999999999,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":1.627,"5min":2.9750000000000001,"15min":75.016000000000005},"last":1.605}]},{"osd":4,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.4079999999999999,"15min":1.496},"min":{"1min":1.0129999999999999,"5min":0.90500000000000003,"15min":0.58099999999999996},"max":{"1min":1.6599999999999999,"5min":3.3420000000000001,"15min":10.577999999999999},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5149999999999999,"15min":1.718},"min":{"1min":1.101,"5min":0.78400000000000003,"15min":0.67000000000000004},"max":{"1min":1.7809999999999999,"5min":3.6920000000000002,"15min":27.920999999999999},"last":1.8400000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4830000000000001,"15min":1.5620000000000001},"min":{"1min":1.161,"5min":0.76600000000000001,"15min":0.63700000000000001},"max":{"1min":1.7250000000000001,"5min":3.032,"15min":17.469000000000001},"last":1.698},{"interface":"front","average":{"1min":1.407,"5min":1.4710000000000001,"15min":1.6120000000000001},"min":{"1min":1.0609999999999999,"5min":0.84499999999999997,"15min":0.53300000000000003},"max":{"1min":1.75,"5min":3.3940000000000001,"15min":8.2989999999999995},"last":1.966}]},{"osd":6,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.573,"15min":1.726},"min":{"1min":1.155,"5min":0.85199999999999998,"15min":0.56999999999999995},"max":{"1min":2.0550000000000002,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5820000000000001,"15min":1.8939999999999999},"min":{"1min":1.2130000000000001,"5min":1.0009999999999999,"15min":0.65600000000000003},"max":{"1min":1.9039999999999999,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.651}]},{"osd":7,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.6919999999999999,"15min":1.9370000000000001},"min":{"1min":1.2689999999999999,"5min":0.68899999999999995,"15min":0.48099999999999998},"max":{"1min":1.9330000000000001,"5min":7.0499999999999998,"15min":30.834},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.677,"5min":1.6679999999999999,"15min":2.0550000000000002},"min":{"1min":1.2849999999999999,"5min":0.89900000000000002,"15min":0.69099999999999995},"max":{"1min":2.1600000000000001,"5min":3.7839999999999998,"15min":27.716999999999999},"last":1.9179999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542382,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73296,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298544,"statfs":{"total":96636764160,"available":96561709056,"internally_reserved":0,"allocated":1961984,"data_stored":1174691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.032,"15min":1.206},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.32300000000000001},"max":{"1min":1.2969999999999999,"5min":10.113,"15min":22.132999999999999},"last":1.246},{"interface":"front","average":{"1min":1.014,"5min":1.0660000000000001,"15min":1.4970000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.33900000000000002},"max":{"1min":1.5780000000000001,"5min":1.732,"15min":50.429000000000002},"last":1.3859999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.1599999999999999,"15min":1.425},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.38},"max":{"1min":1.6970000000000001,"5min":1.9370000000000001,"15min":22.716999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.077,"5min":1.2110000000000001,"15min":1.613},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.38},"max":{"1min":1.6599999999999999,"5min":1.889,"15min":50.581000000000003},"last":1.169}]},{"osd":3,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.1559999999999999,"15min":1.1839999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.372},"max":{"1min":1.371,"5min":5.3280000000000003,"15min":5.8719999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.375,"15min":1.5609999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.46700000000000003},"max":{"1min":1.841,"5min":2.5859999999999999,"15min":22.254999999999999},"last":1.3440000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.4850000000000001,"15min":1.8100000000000001},"min":{"1min":0.95599999999999996,"5min":0.78800000000000003,"15min":0.70399999999999996},"max":{"1min":1.804,"5min":2.8079999999999998,"15min":22.407},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.423,"15min":1.893},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.57899999999999996},"max":{"1min":1.639,"5min":2.7770000000000001,"15min":50.497},"last":1.635}]},{"osd":5,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.4330000000000001,"15min":1.591},"min":{"1min":0.97599999999999998,"5min":0.83199999999999996,"15min":0.51900000000000002},"max":{"1min":1.7250000000000001,"5min":5.5609999999999999,"15min":14.112},"last":1.028},{"interface":"front","average":{"1min":1.373,"5min":1.5680000000000001,"15min":1.8600000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.71699999999999997},"max":{"1min":1.887,"5min":2.6840000000000002,"15min":22.882999999999999},"last":1.5309999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4790000000000001,"15min":1.7549999999999999},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.55600000000000005},"max":{"1min":1.647,"5min":5.4509999999999996,"15min":31.387},"last":1.296},{"interface":"front","average":{"1min":1.498,"5min":1.6639999999999999,"15min":1.984},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.64900000000000002},"max":{"1min":1.974,"5min":3.5209999999999999,"15min":23.547000000000001},"last":1.6659999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.569,"15min":2.069},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.67900000000000005},"max":{"1min":1.764,"5min":2.8679999999999999,"15min":50.542000000000002},"last":1.917},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.6040000000000001,"15min":1.8080000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.58399999999999996},"max":{"1min":1.952,"5min":5.5060000000000002,"15min":15.747},"last":1.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:19:29.109 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:19:29.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.381+0000 7f8759a00640 1 Processor -- start 2024-01-16T15:19:29.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.381+0000 7f8759a00640 1 -- start start 2024-01-16T15:19:29.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.383+0000 7f8759a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f875415e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.383+0000 7f8759a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 0x7f875406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.383+0000 7f8759a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87540b5440 con 0x7f875406a920 2024-01-16T15:19:29.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.383+0000 7f8759a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f875406adb0 con 0x7f8754140300 2024-01-16T15:19:29.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.383+0000 7f8759a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8754161c80 con 0x7f875413ff20 2024-01-16T15:19:29.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.384+0000 7f874be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 0x7f875406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.384+0000 7f8753e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f875415e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.384+0000 7f874be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 0x7f875406a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:47964/0 (socket says 172.21.15.53:47964) 2024-01-16T15:19:29.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.384+0000 7f874be00640 1 -- 172.21.15.53:0/3233161399 learned_addr learned my addr 172.21.15.53:0/3233161399 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:29.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.385+0000 7f8753e00640 1 -- 172.21.15.53:0/3233161399 >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 msgr2=0x7f875406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:29.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.385+0000 7f8753e00640 1 --2- 172.21.15.53:0/3233161399 >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 0x7f875406a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.385+0000 7f8753e00640 1 -- 172.21.15.53:0/3233161399 >> v1:172.21.15.146:6789/0 conn(0x7f875406a920 legacy=0x7f87540b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.385+0000 7f8753e00640 1 -- 172.21.15.53:0/3233161399 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875415e6f0 con 0x7f875413ff20 2024-01-16T15:19:29.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.385+0000 7f874be00640 1 --2- 172.21.15.53:0/3233161399 >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 0x7f875406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:19:29.378 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.386+0000 7f8753e00640 1 --2- 172.21.15.53:0/3233161399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f875415e1b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f87440097a0 tx=0x7f874402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8492b27963bdd581 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:29.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.387+0000 7f8753400640 1 -- 172.21.15.53:0/3233161399 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874402e070 con 0x7f875413ff20 2024-01-16T15:19:29.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.387+0000 7f8753400640 1 -- 172.21.15.53:0/3233161399 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87440315a0 con 0x7f875413ff20 2024-01-16T15:19:29.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.387+0000 7f8753400640 1 -- 172.21.15.53:0/3233161399 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8744031930 con 0x7f875413ff20 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 -- 172.21.15.53:0/3233161399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 msgr2=0x7f875415e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 --2- 172.21.15.53:0/3233161399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f875415e1b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f87440097a0 tx=0x7f874402b840 comp rx=0 tx=0).stop 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 -- 172.21.15.53:0/3233161399 shutdown_connections 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 --2- 172.21.15.53:0/3233161399 >> v2:172.21.15.146:3300/0 conn(0x7f8754140300 0x7f875406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 --2- 172.21.15.53:0/3233161399 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f875415e1b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 -- 172.21.15.53:0/3233161399 >> 172.21.15.53:0/3233161399 conn(0x7f87540b1cd0 msgr2=0x7f87540b20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 -- 172.21.15.53:0/3233161399 shutdown_connections 2024-01-16T15:19:29.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.388+0000 7f8759a00640 1 -- 172.21.15.53:0/3233161399 wait complete. 2024-01-16T15:19:29.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.390+0000 7f8759a00640 1 Processor -- start 2024-01-16T15:19:29.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.391+0000 7f8759a00640 1 -- start start 2024-01-16T15:19:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f875406a920 0x7f875415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f87540ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 0x7f87540cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f875415d070 con 0x7f8754140300 2024-01-16T15:19:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87540adc40 con 0x7f875406a920 2024-01-16T15:19:29.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87540cd4a0 con 0x7f875413ff20 2024-01-16T15:19:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.392+0000 7f8759000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 0x7f87540cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 0x7f87540cecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55932/0 (socket says 172.21.15.53:55932) 2024-01-16T15:19:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 -- 172.21.15.53:0/2264720083 learned_addr learned my addr 172.21.15.53:0/2264720083 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8753e00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f875406a920 0x7f875415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f874be00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f87540ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 -- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 msgr2=0x7f87540ccc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f87540ccc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 -- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f875406a920 msgr2=0x7f875415cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f875406a920 0x7f875415cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 -- 172.21.15.53:0/2264720083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c0094d0 con 0x7f8754140300 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.393+0000 7f8759000640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 0x7f87540cecb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f874c00c7a0 tx=0x7f874c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.394+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c003b70 con 0x7f8754140300 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.394+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f874c003cf0 con 0x7f8754140300 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.394+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c003ea0 con 0x7f8754140300 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.394+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8744009400 con 0x7f8754140300 2024-01-16T15:19:29.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.394+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87540ad8d0 con 0x7f8754140300 2024-01-16T15:19:29.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.395+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87540addb0 con 0x7f8754140300 2024-01-16T15:19:29.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.396+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f874c00aa90 con 0x7f8754140300 2024-01-16T15:19:29.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.398+0000 7f8750c00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f872c06c430 0x7f872c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.398+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f874c0887d0 con 0x7f8754140300 2024-01-16T15:19:29.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.398+0000 7f8753e00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f872c06c430 0x7f872c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.400+0000 7f8753e00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f872c06c430 0x7f872c06e8f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8744036000 tx=0x7f874403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:29.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.402+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f874c0bee90 con 0x7f8754140300 2024-01-16T15:19:29.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.594+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87540b4df0 con 0x7f872c06c430 2024-01-16T15:19:29.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.619+0000 7f8750c00640 1 -- 172.21.15.53:0/2264720083 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287189 (secure 0 0 0) 0x7f87440319d0 con 0x7f872c06c430 2024-01-16T15:19:29.616 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:29.618 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:19:29.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f872c06c430 msgr2=0x7f872c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f872c06c430 0x7f872c06e8f0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f8744036000 tx=0x7f874403d040 comp rx=0 tx=0).stop 2024-01-16T15:19:29.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 msgr2=0x7f87540cecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 0x7f87540cecb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f874c00c7a0 tx=0x7f874c00cc60 comp rx=0 tx=0).stop 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 shutdown_connections 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f872c06c430 0x7f872c06e8f0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8754140300 0x7f87540cecb0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f875413ff20 0x7f87540ccc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 --2- 172.21.15.53:0/2264720083 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f875406a920 0x7f875415cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 >> 172.21.15.53:0/2264720083 conn(0x7f87540b1cd0 msgr2=0x7f87540b23d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 shutdown_connections 2024-01-16T15:19:29.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.626+0000 7f8759a00640 1 -- 172.21.15.53:0/2264720083 wait complete. 2024-01-16T15:19:29.639 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":695,"stamp":"2024-01-16T15:19:28.692976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":138,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":566132,"kb_used_data":11776,"kb_used_omap":12,"kb_used_meta":554227,"kb_avail":754408588,"statfs":{"total":773094113280,"available":772514394112,"internally_reserved":0,"allocated":12058624,"data_stored":5859004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":567529008},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001915"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637642+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:19:24.637642+0000","last_peered":"2024-01-16T15:19:24.637642+0000","last_clean":"2024-01-16T15:19:24.637642+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:19:24.637642+0000","last_undegraded":"2024-01-16T15:19:24.637642+0000","last_fullsized":"2024-01-16T15:19:24.637642+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:32:27.826457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168771+0000","last_change":"2024-01-16T15:19:24.622288+0000","last_active":"2024-01-16T15:19:26.168771+0000","last_peered":"2024-01-16T15:19:26.168771+0000","last_clean":"2024-01-16T15:19:26.168771+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:19:26.168771+0000","last_undegraded":"2024-01-16T15:19:26.168771+0000","last_fullsized":"2024-01-16T15:19:26.168771+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:20.176733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626908+0000","last_change":"2024-01-16T15:19:24.618862+0000","last_active":"2024-01-16T15:19:24.626908+0000","last_peered":"2024-01-16T15:19:24.626908+0000","last_clean":"2024-01-16T15:19:24.626908+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:19:24.626908+0000","last_undegraded":"2024-01-16T15:19:24.626908+0000","last_fullsized":"2024-01-16T15:19:24.626908+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:53.058982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642476+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:19:24.642476+0000","last_peered":"2024-01-16T15:19:24.642476+0000","last_clean":"2024-01-16T15:19:24.642476+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:19:24.642476+0000","last_undegraded":"2024-01-16T15:19:24.642476+0000","last_fullsized":"2024-01-16T15:19:24.642476+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:11.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631057+0000","last_change":"2024-01-16T15:19:24.626093+0000","last_active":"2024-01-16T15:19:24.631057+0000","last_peered":"2024-01-16T15:19:24.631057+0000","last_clean":"2024-01-16T15:19:24.631057+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:19:24.631057+0000","last_undegraded":"2024-01-16T15:19:24.631057+0000","last_fullsized":"2024-01-16T15:19:24.631057+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:47.875736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.630172+0000","last_change":"2024-01-16T15:19:24.625809+0000","last_active":"2024-01-16T15:19:24.630172+0000","last_peered":"2024-01-16T15:19:24.630172+0000","last_clean":"2024-01-16T15:19:24.630172+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:19:24.630172+0000","last_undegraded":"2024-01-16T15:19:24.630172+0000","last_fullsized":"2024-01-16T15:19:24.630172+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:40:11.125336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627850+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:19:24.627850+0000","last_peered":"2024-01-16T15:19:24.627850+0000","last_clean":"2024-01-16T15:19:24.627850+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:19:24.627850+0000","last_undegraded":"2024-01-16T15:19:24.627850+0000","last_fullsized":"2024-01-16T15:19:24.627850+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:18:02.507029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.623313+0000","last_change":"2024-01-16T15:19:24.616757+0000","last_active":"2024-01-16T15:19:24.623313+0000","last_peered":"2024-01-16T15:19:24.623313+0000","last_clean":"2024-01-16T15:19:24.623313+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:19:24.623313+0000","last_undegraded":"2024-01-16T15:19:24.623313+0000","last_fullsized":"2024-01-16T15:19:24.623313+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:58:44.065214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642075+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:19:24.642075+0000","last_peered":"2024-01-16T15:19:24.642075+0000","last_clean":"2024-01-16T15:19:24.642075+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:19:24.642075+0000","last_undegraded":"2024-01-16T15:19:24.642075+0000","last_fullsized":"2024-01-16T15:19:24.642075+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:04:30.175114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636104+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:19:24.636104+0000","last_peered":"2024-01-16T15:19:24.636104+0000","last_clean":"2024-01-16T15:19:24.636104+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:19:24.636104+0000","last_undegraded":"2024-01-16T15:19:24.636104+0000","last_fullsized":"2024-01-16T15:19:24.636104+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:32:13.468787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168713+0000","last_change":"2024-01-16T15:19:24.625137+0000","last_active":"2024-01-16T15:19:26.168713+0000","last_peered":"2024-01-16T15:19:26.168713+0000","last_clean":"2024-01-16T15:19:26.168713+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:19:26.168713+0000","last_undegraded":"2024-01-16T15:19:26.168713+0000","last_fullsized":"2024-01-16T15:19:26.168713+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628098+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:19:24.628098+0000","last_peered":"2024-01-16T15:19:24.628098+0000","last_clean":"2024-01-16T15:19:24.628098+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:19:24.628098+0000","last_undegraded":"2024-01-16T15:19:24.628098+0000","last_fullsized":"2024-01-16T15:19:24.628098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:31.587040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627199+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:19:24.627199+0000","last_peered":"2024-01-16T15:19:24.627199+0000","last_clean":"2024-01-16T15:19:24.627199+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:19:24.627199+0000","last_undegraded":"2024-01-16T15:19:24.627199+0000","last_fullsized":"2024-01-16T15:19:24.627199+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:32.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628574+0000","last_change":"2024-01-16T15:19:24.624351+0000","last_active":"2024-01-16T15:19:24.628574+0000","last_peered":"2024-01-16T15:19:24.628574+0000","last_clean":"2024-01-16T15:19:24.628574+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:19:24.628574+0000","last_undegraded":"2024-01-16T15:19:24.628574+0000","last_fullsized":"2024-01-16T15:19:24.628574+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:59.580723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.618778+0000","last_change":"2024-01-16T15:19:24.613444+0000","last_active":"2024-01-16T15:19:24.618778+0000","last_peered":"2024-01-16T15:19:24.618778+0000","last_clean":"2024-01-16T15:19:24.618778+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:19:24.618778+0000","last_undegraded":"2024-01-16T15:19:24.618778+0000","last_fullsized":"2024-01-16T15:19:24.618778+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:06.791309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.620617+0000","last_change":"2024-01-16T15:19:24.614694+0000","last_active":"2024-01-16T15:19:24.620617+0000","last_peered":"2024-01-16T15:19:24.620617+0000","last_clean":"2024-01-16T15:19:24.620617+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:19:24.620617+0000","last_undegraded":"2024-01-16T15:19:24.620617+0000","last_fullsized":"2024-01-16T15:19:24.620617+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:58.872650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.169003+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:19:26.169003+0000","last_peered":"2024-01-16T15:19:26.169003+0000","last_clean":"2024-01-16T15:19:26.169003+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:19:26.169003+0000","last_undegraded":"2024-01-16T15:19:26.169003+0000","last_fullsized":"2024-01-16T15:19:26.169003+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626504+0000","last_change":"2024-01-16T15:19:24.629022+0000","last_active":"2024-01-16T15:19:25.626504+0000","last_peered":"2024-01-16T15:19:25.626504+0000","last_clean":"2024-01-16T15:19:25.626504+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:19:25.626504+0000","last_undegraded":"2024-01-16T15:19:25.626504+0000","last_fullsized":"2024-01-16T15:19:25.626504+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:22.444157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626946+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:19:24.626946+0000","last_peered":"2024-01-16T15:19:24.626946+0000","last_clean":"2024-01-16T15:19:24.626946+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:19:24.626946+0000","last_undegraded":"2024-01-16T15:19:24.626946+0000","last_fullsized":"2024-01-16T15:19:24.626946+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:01.542383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168872+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:19:26.168872+0000","last_peered":"2024-01-16T15:19:26.168872+0000","last_clean":"2024-01-16T15:19:26.168872+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:19:26.168872+0000","last_undegraded":"2024-01-16T15:19:26.168872+0000","last_fullsized":"2024-01-16T15:19:26.168872+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.169055+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:19:26.169055+0000","last_peered":"2024-01-16T15:19:26.169055+0000","last_clean":"2024-01-16T15:19:26.169055+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:19:26.169055+0000","last_undegraded":"2024-01-16T15:19:26.169055+0000","last_fullsized":"2024-01-16T15:19:26.169055+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628396+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:19:24.628396+0000","last_peered":"2024-01-16T15:19:24.628396+0000","last_clean":"2024-01-16T15:19:24.628396+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:19:24.628396+0000","last_undegraded":"2024-01-16T15:19:24.628396+0000","last_fullsized":"2024-01-16T15:19:24.628396+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:11.461387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168099+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:19:26.168099+0000","last_peered":"2024-01-16T15:19:26.168099+0000","last_clean":"2024-01-16T15:19:26.168099+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:19:26.168099+0000","last_undegraded":"2024-01-16T15:19:26.168099+0000","last_fullsized":"2024-01-16T15:19:26.168099+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:07.051272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.632017+0000","last_change":"2024-01-16T15:19:24.626920+0000","last_active":"2024-01-16T15:19:24.632017+0000","last_peered":"2024-01-16T15:19:24.632017+0000","last_clean":"2024-01-16T15:19:24.632017+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:19:24.632017+0000","last_undegraded":"2024-01-16T15:19:24.632017+0000","last_fullsized":"2024-01-16T15:19:24.632017+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:45:17.563783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.169091+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:19:26.169091+0000","last_peered":"2024-01-16T15:19:26.169091+0000","last_clean":"2024-01-16T15:19:26.169091+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:19:26.169091+0000","last_undegraded":"2024-01-16T15:19:26.169091+0000","last_fullsized":"2024-01-16T15:19:26.169091+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.638727+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:19:24.638727+0000","last_peered":"2024-01-16T15:19:24.638727+0000","last_clean":"2024-01-16T15:19:24.638727+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:19:24.638727+0000","last_undegraded":"2024-01-16T15:19:24.638727+0000","last_fullsized":"2024-01-16T15:19:24.638727+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:22:30.752763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627851+0000","last_change":"2024-01-16T15:19:24.623489+0000","last_active":"2024-01-16T15:19:24.627851+0000","last_peered":"2024-01-16T15:19:24.627851+0000","last_clean":"2024-01-16T15:19:24.627851+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:19:24.627851+0000","last_undegraded":"2024-01-16T15:19:24.627851+0000","last_fullsized":"2024-01-16T15:19:24.627851+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:20:47.875736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.632768+0000","last_change":"2024-01-16T15:19:24.628737+0000","last_active":"2024-01-16T15:19:24.632768+0000","last_peered":"2024-01-16T15:19:24.632768+0000","last_clean":"2024-01-16T15:19:24.632768+0000","last_became_active":"2024-01-16T15:19:24.628439+0000","last_became_peered":"2024-01-16T15:19:24.628439+0000","last_unstale":"2024-01-16T15:19:24.632768+0000","last_undegraded":"2024-01-16T15:19:24.632768+0000","last_fullsized":"2024-01-16T15:19:24.632768+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168304+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:19:26.168304+0000","last_peered":"2024-01-16T15:19:26.168304+0000","last_clean":"2024-01-16T15:19:26.168304+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:19:26.168304+0000","last_undegraded":"2024-01-16T15:19:26.168304+0000","last_fullsized":"2024-01-16T15:19:26.168304+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:40.862803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.633416+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:19:24.633416+0000","last_peered":"2024-01-16T15:19:24.633416+0000","last_clean":"2024-01-16T15:19:24.633416+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:19:24.633416+0000","last_undegraded":"2024-01-16T15:19:24.633416+0000","last_fullsized":"2024-01-16T15:19:24.633416+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:55:44.343940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637840+0000","last_change":"2024-01-16T15:19:24.629910+0000","last_active":"2024-01-16T15:19:24.637840+0000","last_peered":"2024-01-16T15:19:24.637840+0000","last_clean":"2024-01-16T15:19:24.637840+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:19:24.637840+0000","last_undegraded":"2024-01-16T15:19:24.637840+0000","last_fullsized":"2024-01-16T15:19:24.637840+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:30.687663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.634847+0000","last_change":"2024-01-16T15:19:24.633567+0000","last_active":"2024-01-16T15:19:24.634847+0000","last_peered":"2024-01-16T15:19:24.634847+0000","last_clean":"2024-01-16T15:19:24.634847+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:19:24.634847+0000","last_undegraded":"2024-01-16T15:19:24.634847+0000","last_fullsized":"2024-01-16T15:19:24.634847+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:54.199001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627260+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:19:24.627260+0000","last_peered":"2024-01-16T15:19:24.627260+0000","last_clean":"2024-01-16T15:19:24.627260+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:19:24.627260+0000","last_undegraded":"2024-01-16T15:19:24.627260+0000","last_fullsized":"2024-01-16T15:19:24.627260+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:34.368248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636605+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:19:24.636605+0000","last_peered":"2024-01-16T15:19:24.636605+0000","last_clean":"2024-01-16T15:19:24.636605+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:19:24.636605+0000","last_undegraded":"2024-01-16T15:19:24.636605+0000","last_fullsized":"2024-01-16T15:19:24.636605+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.038677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.627087+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:19:25.627087+0000","last_peered":"2024-01-16T15:19:25.627087+0000","last_clean":"2024-01-16T15:19:25.627087+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:19:25.627087+0000","last_undegraded":"2024-01-16T15:19:25.627087+0000","last_fullsized":"2024-01-16T15:19:25.627087+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:25.905229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.633095+0000","last_change":"2024-01-16T15:19:23.621310+0000","last_active":"2024-01-16T15:19:24.633095+0000","last_peered":"2024-01-16T15:19:24.633095+0000","last_clean":"2024-01-16T15:19:24.633095+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:19:24.633095+0000","last_undegraded":"2024-01-16T15:19:24.633095+0000","last_fullsized":"2024-01-16T15:19:24.633095+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:19.958286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631382+0000","last_change":"2024-01-16T15:19:24.630161+0000","last_active":"2024-01-16T15:19:24.631382+0000","last_peered":"2024-01-16T15:19:24.631382+0000","last_clean":"2024-01-16T15:19:24.631382+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:19:24.631382+0000","last_undegraded":"2024-01-16T15:19:24.631382+0000","last_fullsized":"2024-01-16T15:19:24.631382+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:01.440713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636973+0000","last_change":"2024-01-16T15:19:23.630011+0000","last_active":"2024-01-16T15:19:24.636973+0000","last_peered":"2024-01-16T15:19:24.636973+0000","last_clean":"2024-01-16T15:19:24.636973+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:19:24.636973+0000","last_undegraded":"2024-01-16T15:19:24.636973+0000","last_fullsized":"2024-01-16T15:19:24.636973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:21.485125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626963+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:19:24.626963+0000","last_peered":"2024-01-16T15:19:24.626963+0000","last_clean":"2024-01-16T15:19:24.626963+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:19:24.626963+0000","last_undegraded":"2024-01-16T15:19:24.626963+0000","last_fullsized":"2024-01-16T15:19:24.626963+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:44:59.238148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.638258+0000","last_change":"2024-01-16T15:19:23.618597+0000","last_active":"2024-01-16T15:19:24.638258+0000","last_peered":"2024-01-16T15:19:24.638258+0000","last_clean":"2024-01-16T15:19:24.638258+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:19:24.638258+0000","last_undegraded":"2024-01-16T15:19:24.638258+0000","last_fullsized":"2024-01-16T15:19:24.638258+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:05.775312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642236+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:19:24.642236+0000","last_peered":"2024-01-16T15:19:24.642236+0000","last_clean":"2024-01-16T15:19:24.642236+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:19:24.642236+0000","last_undegraded":"2024-01-16T15:19:24.642236+0000","last_fullsized":"2024-01-16T15:19:24.642236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:45.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.635883+0000","last_change":"2024-01-16T15:19:23.618024+0000","last_active":"2024-01-16T15:19:24.635883+0000","last_peered":"2024-01-16T15:19:24.635883+0000","last_clean":"2024-01-16T15:19:24.635883+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:19:24.635883+0000","last_undegraded":"2024-01-16T15:19:24.635883+0000","last_fullsized":"2024-01-16T15:19:24.635883+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:50.476809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168310+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:19:26.168310+0000","last_peered":"2024-01-16T15:19:26.168310+0000","last_clean":"2024-01-16T15:19:26.168310+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:19:26.168310+0000","last_undegraded":"2024-01-16T15:19:26.168310+0000","last_fullsized":"2024-01-16T15:19:26.168310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637764+0000","last_change":"2024-01-16T15:19:23.622641+0000","last_active":"2024-01-16T15:19:24.637764+0000","last_peered":"2024-01-16T15:19:24.637764+0000","last_clean":"2024-01-16T15:19:24.637764+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:19:24.637764+0000","last_undegraded":"2024-01-16T15:19:24.637764+0000","last_fullsized":"2024-01-16T15:19:24.637764+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:39.846805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626373+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:19:24.626373+0000","last_peered":"2024-01-16T15:19:24.626373+0000","last_clean":"2024-01-16T15:19:24.626373+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:19:24.626373+0000","last_undegraded":"2024-01-16T15:19:24.626373+0000","last_fullsized":"2024-01-16T15:19:24.626373+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:41:32.812714+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168172+0000","last_change":"2024-01-16T15:19:23.620941+0000","last_active":"2024-01-16T15:19:26.168172+0000","last_peered":"2024-01-16T15:19:26.168172+0000","last_clean":"2024-01-16T15:19:26.168172+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:19:26.168172+0000","last_undegraded":"2024-01-16T15:19:26.168172+0000","last_fullsized":"2024-01-16T15:19:26.168172+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:27.971375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627832+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:19:24.627832+0000","last_peered":"2024-01-16T15:19:24.627832+0000","last_clean":"2024-01-16T15:19:24.627832+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:19:24.627832+0000","last_undegraded":"2024-01-16T15:19:24.627832+0000","last_fullsized":"2024-01-16T15:19:24.627832+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:48:36.352635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.601586+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:19:24.601586+0000","last_peered":"2024-01-16T15:19:24.601586+0000","last_clean":"2024-01-16T15:19:24.601586+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:19:24.601586+0000","last_undegraded":"2024-01-16T15:19:24.601586+0000","last_fullsized":"2024-01-16T15:19:24.601586+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:07.254538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.633796+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:19:24.633796+0000","last_peered":"2024-01-16T15:19:24.633796+0000","last_clean":"2024-01-16T15:19:24.633796+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:19:24.633796+0000","last_undegraded":"2024-01-16T15:19:24.633796+0000","last_fullsized":"2024-01-16T15:19:24.633796+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:08:27.804532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.591363+0000","last_change":"2024-01-16T15:19:23.621670+0000","last_active":"2024-01-16T15:19:24.591363+0000","last_peered":"2024-01-16T15:19:24.591363+0000","last_clean":"2024-01-16T15:19:24.591363+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:19:24.591363+0000","last_undegraded":"2024-01-16T15:19:24.591363+0000","last_fullsized":"2024-01-16T15:19:24.591363+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:53.183004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.627124+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:19:25.627124+0000","last_peered":"2024-01-16T15:19:25.627124+0000","last_clean":"2024-01-16T15:19:25.627124+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:19:25.627124+0000","last_undegraded":"2024-01-16T15:19:25.627124+0000","last_fullsized":"2024-01-16T15:19:25.627124+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:56.499150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.603442+0000","last_change":"2024-01-16T15:19:23.624332+0000","last_active":"2024-01-16T15:19:24.603442+0000","last_peered":"2024-01-16T15:19:24.603442+0000","last_clean":"2024-01-16T15:19:24.603442+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:19:24.603442+0000","last_undegraded":"2024-01-16T15:19:24.603442+0000","last_fullsized":"2024-01-16T15:19:24.603442+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:56.995452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.632742+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:19:24.632742+0000","last_peered":"2024-01-16T15:19:24.632742+0000","last_clean":"2024-01-16T15:19:24.632742+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:19:24.632742+0000","last_undegraded":"2024-01-16T15:19:24.632742+0000","last_fullsized":"2024-01-16T15:19:24.632742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"109'3","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.646105+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:19:24.646105+0000","last_peered":"2024-01-16T15:19:24.646105+0000","last_clean":"2024-01-16T15:19:24.646105+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:19:24.646105+0000","last_undegraded":"2024-01-16T15:19:24.646105+0000","last_fullsized":"2024-01-16T15:19:24.646105+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:48.585086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631616+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:19:24.631616+0000","last_peered":"2024-01-16T15:19:24.631616+0000","last_clean":"2024-01-16T15:19:24.631616+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:19:24.631616+0000","last_undegraded":"2024-01-16T15:19:24.631616+0000","last_fullsized":"2024-01-16T15:19:24.631616+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:38:05.613201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"109'6","reported_seq":17,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.648369+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:19:24.648369+0000","last_peered":"2024-01-16T15:19:24.648369+0000","last_clean":"2024-01-16T15:19:24.648369+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:19:24.648369+0000","last_undegraded":"2024-01-16T15:19:24.648369+0000","last_fullsized":"2024-01-16T15:19:24.648369+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:52.253914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.635949+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:19:24.635949+0000","last_peered":"2024-01-16T15:19:24.635949+0000","last_clean":"2024-01-16T15:19:24.635949+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:19:24.635949+0000","last_undegraded":"2024-01-16T15:19:24.635949+0000","last_fullsized":"2024-01-16T15:19:24.635949+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:34:12.009819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.641122+0000","last_change":"2024-01-16T15:19:23.617724+0000","last_active":"2024-01-16T15:19:24.641122+0000","last_peered":"2024-01-16T15:19:24.641122+0000","last_clean":"2024-01-16T15:19:24.641122+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:19:24.641122+0000","last_undegraded":"2024-01-16T15:19:24.641122+0000","last_fullsized":"2024-01-16T15:19:24.641122+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:35.823425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.631437+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:19:24.631437+0000","last_peered":"2024-01-16T15:19:24.631437+0000","last_clean":"2024-01-16T15:19:24.631437+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:19:24.631437+0000","last_undegraded":"2024-01-16T15:19:24.631437+0000","last_fullsized":"2024-01-16T15:19:24.631437+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626138+0000","last_change":"2024-01-16T15:19:23.624683+0000","last_active":"2024-01-16T15:19:25.626138+0000","last_peered":"2024-01-16T15:19:25.626138+0000","last_clean":"2024-01-16T15:19:25.626138+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:19:25.626138+0000","last_undegraded":"2024-01-16T15:19:25.626138+0000","last_fullsized":"2024-01-16T15:19:25.626138+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:38.379463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628502+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:19:24.628502+0000","last_peered":"2024-01-16T15:19:24.628502+0000","last_clean":"2024-01-16T15:19:24.628502+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:19:24.628502+0000","last_undegraded":"2024-01-16T15:19:24.628502+0000","last_fullsized":"2024-01-16T15:19:24.628502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:07.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":19,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.645127+0000","last_change":"2024-01-16T15:19:23.625591+0000","last_active":"2024-01-16T15:19:24.645127+0000","last_peered":"2024-01-16T15:19:24.645127+0000","last_clean":"2024-01-16T15:19:24.645127+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:19:24.645127+0000","last_undegraded":"2024-01-16T15:19:24.645127+0000","last_fullsized":"2024-01-16T15:19:24.645127+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:27.727635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.620265+0000","last_change":"2024-01-16T15:19:24.617249+0000","last_active":"2024-01-16T15:19:24.620265+0000","last_peered":"2024-01-16T15:19:24.620265+0000","last_clean":"2024-01-16T15:19:24.620265+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:19:24.620265+0000","last_undegraded":"2024-01-16T15:19:24.620265+0000","last_fullsized":"2024-01-16T15:19:24.620265+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:40:11.125336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.645029+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:19:24.645029+0000","last_peered":"2024-01-16T15:19:24.645029+0000","last_clean":"2024-01-16T15:19:24.645029+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:19:24.645029+0000","last_undegraded":"2024-01-16T15:19:24.645029+0000","last_fullsized":"2024-01-16T15:19:24.645029+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:27.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626866+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:19:24.626866+0000","last_peered":"2024-01-16T15:19:24.626866+0000","last_clean":"2024-01-16T15:19:24.626866+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:19:24.626866+0000","last_undegraded":"2024-01-16T15:19:24.626866+0000","last_fullsized":"2024-01-16T15:19:24.626866+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:23:08.024918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.636769+0000","last_change":"2024-01-16T15:19:23.622230+0000","last_active":"2024-01-16T15:19:24.636769+0000","last_peered":"2024-01-16T15:19:24.636769+0000","last_clean":"2024-01-16T15:19:24.636769+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:19:24.636769+0000","last_undegraded":"2024-01-16T15:19:24.636769+0000","last_fullsized":"2024-01-16T15:19:24.636769+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626560+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:19:24.626560+0000","last_peered":"2024-01-16T15:19:24.626560+0000","last_clean":"2024-01-16T15:19:24.626560+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:19:24.626560+0000","last_undegraded":"2024-01-16T15:19:24.626560+0000","last_fullsized":"2024-01-16T15:19:24.626560+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:14:45.752569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":13,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626574+0000","last_change":"2024-01-16T15:19:23.626903+0000","last_active":"2024-01-16T15:19:25.626574+0000","last_peered":"2024-01-16T15:19:25.626574+0000","last_clean":"2024-01-16T15:19:25.626574+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:19:25.626574+0000","last_undegraded":"2024-01-16T15:19:25.626574+0000","last_fullsized":"2024-01-16T15:19:25.626574+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:58.751536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.629627+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:19:24.629627+0000","last_peered":"2024-01-16T15:19:24.629627+0000","last_clean":"2024-01-16T15:19:24.629627+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:19:24.629627+0000","last_undegraded":"2024-01-16T15:19:24.629627+0000","last_fullsized":"2024-01-16T15:19:24.629627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:47:24.185677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.624043+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:19:24.624043+0000","last_peered":"2024-01-16T15:19:24.624043+0000","last_clean":"2024-01-16T15:19:24.624043+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:19:24.624043+0000","last_undegraded":"2024-01-16T15:19:24.624043+0000","last_fullsized":"2024-01-16T15:19:24.624043+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:00.825374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.623270+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:19:24.623270+0000","last_peered":"2024-01-16T15:19:24.623270+0000","last_clean":"2024-01-16T15:19:24.623270+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:19:24.623270+0000","last_undegraded":"2024-01-16T15:19:24.623270+0000","last_fullsized":"2024-01-16T15:19:24.623270+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:17:03.807749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642249+0000","last_change":"2024-01-16T15:19:23.621285+0000","last_active":"2024-01-16T15:19:24.642249+0000","last_peered":"2024-01-16T15:19:24.642249+0000","last_clean":"2024-01-16T15:19:24.642249+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:19:24.642249+0000","last_undegraded":"2024-01-16T15:19:24.642249+0000","last_fullsized":"2024-01-16T15:19:24.642249+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:44.070274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.635611+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:19:24.635611+0000","last_peered":"2024-01-16T15:19:24.635611+0000","last_clean":"2024-01-16T15:19:24.635611+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:19:24.635611+0000","last_undegraded":"2024-01-16T15:19:24.635611+0000","last_fullsized":"2024-01-16T15:19:24.635611+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:32.314064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.591226+0000","last_change":"2024-01-16T15:19:23.619619+0000","last_active":"2024-01-16T15:19:24.591226+0000","last_peered":"2024-01-16T15:19:24.591226+0000","last_clean":"2024-01-16T15:19:24.591226+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:19:24.591226+0000","last_undegraded":"2024-01-16T15:19:24.591226+0000","last_fullsized":"2024-01-16T15:19:24.591226+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:15.166720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.626053+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:19:24.626053+0000","last_peered":"2024-01-16T15:19:24.626053+0000","last_clean":"2024-01-16T15:19:24.626053+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:19:24.626053+0000","last_undegraded":"2024-01-16T15:19:24.626053+0000","last_fullsized":"2024-01-16T15:19:24.626053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:40.922371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.604218+0000","last_change":"2024-01-16T15:19:23.620635+0000","last_active":"2024-01-16T15:19:24.604218+0000","last_peered":"2024-01-16T15:19:24.604218+0000","last_clean":"2024-01-16T15:19:24.604218+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:19:24.604218+0000","last_undegraded":"2024-01-16T15:19:24.604218+0000","last_fullsized":"2024-01-16T15:19:24.604218+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.628521+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:19:24.628521+0000","last_peered":"2024-01-16T15:19:24.628521+0000","last_clean":"2024-01-16T15:19:24.628521+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:19:24.628521+0000","last_undegraded":"2024-01-16T15:19:24.628521+0000","last_fullsized":"2024-01-16T15:19:24.628521+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:28.987372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.601964+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:19:24.601964+0000","last_peered":"2024-01-16T15:19:24.601964+0000","last_clean":"2024-01-16T15:19:24.601964+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:19:24.601964+0000","last_undegraded":"2024-01-16T15:19:24.601964+0000","last_fullsized":"2024-01-16T15:19:24.601964+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:48.621454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626679+0000","last_change":"2024-01-16T15:19:24.638752+0000","last_active":"2024-01-16T15:19:25.626679+0000","last_peered":"2024-01-16T15:19:25.626679+0000","last_clean":"2024-01-16T15:19:25.626679+0000","last_became_active":"2024-01-16T15:19:24.638466+0000","last_became_peered":"2024-01-16T15:19:24.638466+0000","last_unstale":"2024-01-16T15:19:25.626679+0000","last_undegraded":"2024-01-16T15:19:25.626679+0000","last_fullsized":"2024-01-16T15:19:25.626679+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:17.797680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.643355+0000","last_change":"2024-01-16T15:19:23.630170+0000","last_active":"2024-01-16T15:19:24.643355+0000","last_peered":"2024-01-16T15:19:24.643355+0000","last_clean":"2024-01-16T15:19:24.643355+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:19:24.643355+0000","last_undegraded":"2024-01-16T15:19:24.643355+0000","last_fullsized":"2024-01-16T15:19:24.643355+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:06.035275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.642865+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:19:24.642865+0000","last_peered":"2024-01-16T15:19:24.642865+0000","last_clean":"2024-01-16T15:19:24.642865+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:19:24.642865+0000","last_undegraded":"2024-01-16T15:19:24.642865+0000","last_fullsized":"2024-01-16T15:19:24.642865+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:19:49.999663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.626900+0000","last_change":"2024-01-16T15:19:23.623778+0000","last_active":"2024-01-16T15:19:25.626900+0000","last_peered":"2024-01-16T15:19:25.626900+0000","last_clean":"2024-01-16T15:19:25.626900+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:19:25.626900+0000","last_undegraded":"2024-01-16T15:19:25.626900+0000","last_fullsized":"2024-01-16T15:19:25.626900+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:53.783590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.637131+0000","last_change":"2024-01-16T15:19:24.628361+0000","last_active":"2024-01-16T15:19:24.637131+0000","last_peered":"2024-01-16T15:19:24.637131+0000","last_clean":"2024-01-16T15:19:24.637131+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:19:24.637131+0000","last_undegraded":"2024-01-16T15:19:24.637131+0000","last_fullsized":"2024-01-16T15:19:24.637131+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:32.038856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.601994+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:19:24.601994+0000","last_peered":"2024-01-16T15:19:24.601994+0000","last_clean":"2024-01-16T15:19:24.601994+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:19:24.601994+0000","last_undegraded":"2024-01-16T15:19:24.601994+0000","last_fullsized":"2024-01-16T15:19:24.601994+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:40.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:25.625901+0000","last_change":"2024-01-16T15:19:24.623591+0000","last_active":"2024-01-16T15:19:25.625901+0000","last_peered":"2024-01-16T15:19:25.625901+0000","last_clean":"2024-01-16T15:19:25.625901+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:19:25.625901+0000","last_undegraded":"2024-01-16T15:19:25.625901+0000","last_fullsized":"2024-01-16T15:19:25.625901+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:23:57.706980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.614526+0000","last_change":"2024-01-16T15:19:23.612616+0000","last_active":"2024-01-16T15:19:24.614526+0000","last_peered":"2024-01-16T15:19:24.614526+0000","last_clean":"2024-01-16T15:19:24.614526+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:19:24.614526+0000","last_undegraded":"2024-01-16T15:19:24.614526+0000","last_fullsized":"2024-01-16T15:19:24.614526+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:19.160736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168410+0000","last_change":"2024-01-16T15:19:24.626321+0000","last_active":"2024-01-16T15:19:26.168410+0000","last_peered":"2024-01-16T15:19:26.168410+0000","last_clean":"2024-01-16T15:19:26.168410+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:19:26.168410+0000","last_undegraded":"2024-01-16T15:19:26.168410+0000","last_fullsized":"2024-01-16T15:19:26.168410+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.624234+0000","last_change":"2024-01-16T15:19:23.624071+0000","last_active":"2024-01-16T15:19:24.624234+0000","last_peered":"2024-01-16T15:19:24.624234+0000","last_clean":"2024-01-16T15:19:24.624234+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:19:24.624234+0000","last_undegraded":"2024-01-16T15:19:24.624234+0000","last_fullsized":"2024-01-16T15:19:24.624234+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:45.204823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168798+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:19:26.168798+0000","last_peered":"2024-01-16T15:19:26.168798+0000","last_clean":"2024-01-16T15:19:26.168798+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:19:26.168798+0000","last_undegraded":"2024-01-16T15:19:26.168798+0000","last_fullsized":"2024-01-16T15:19:26.168798+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-01-16T15:19:26.168926+0000","last_change":"2024-01-16T15:19:23.619253+0000","last_active":"2024-01-16T15:19:26.168926+0000","last_peered":"2024-01-16T15:19:26.168926+0000","last_clean":"2024-01-16T15:19:26.168926+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:19:26.168926+0000","last_undegraded":"2024-01-16T15:19:26.168926+0000","last_fullsized":"2024-01-16T15:19:26.168926+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:16.781683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.629292+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:19:24.629292+0000","last_peered":"2024-01-16T15:19:24.629292+0000","last_clean":"2024-01-16T15:19:24.629292+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:19:24.629292+0000","last_undegraded":"2024-01-16T15:19:24.629292+0000","last_fullsized":"2024-01-16T15:19:24.629292+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:19:10.642184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.591234+0000","last_change":"2024-01-16T15:19:23.629871+0000","last_active":"2024-01-16T15:19:24.591234+0000","last_peered":"2024-01-16T15:19:24.591234+0000","last_clean":"2024-01-16T15:19:24.591234+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:19:24.591234+0000","last_undegraded":"2024-01-16T15:19:24.591234+0000","last_fullsized":"2024-01-16T15:19:24.591234+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:05:17.872229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.624488+0000","last_change":"2024-01-16T15:19:24.621286+0000","last_active":"2024-01-16T15:19:24.624488+0000","last_peered":"2024-01-16T15:19:24.624488+0000","last_clean":"2024-01-16T15:19:24.624488+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:19:24.624488+0000","last_undegraded":"2024-01-16T15:19:24.624488+0000","last_fullsized":"2024-01-16T15:19:24.624488+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:00:10.983492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.602334+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:19:24.602334+0000","last_peered":"2024-01-16T15:19:24.602334+0000","last_clean":"2024-01-16T15:19:24.602334+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:19:24.602334+0000","last_undegraded":"2024-01-16T15:19:24.602334+0000","last_fullsized":"2024-01-16T15:19:24.602334+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:01.099875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":9,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.627015+0000","last_change":"2024-01-16T15:19:24.621294+0000","last_active":"2024-01-16T15:19:24.627015+0000","last_peered":"2024-01-16T15:19:24.627015+0000","last_clean":"2024-01-16T15:19:24.627015+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:19:24.627015+0000","last_undegraded":"2024-01-16T15:19:24.627015+0000","last_fullsized":"2024-01-16T15:19:24.627015+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:18:58.723198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-01-16T15:19:24.623375+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:19:24.623375+0000","last_peered":"2024-01-16T15:19:24.623375+0000","last_clean":"2024-01-16T15:19:24.623375+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:19:24.623375+0000","last_undegraded":"2024-01-16T15:19:24.623375+0000","last_fullsized":"2024-01-16T15:19:24.623375+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:59.676130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":253,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-01-16T15:19:23.597516+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:19:23.597516+0000","last_peered":"2024-01-16T15:19:23.597516+0000","last_clean":"2024-01-16T15:19:23.597516+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:19:23.597516+0000","last_undegraded":"2024-01-16T15:19:23.597516+0000","last_fullsized":"2024-01-16T15:19:23.597516+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509678,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68544,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303296,"statfs":{"total":96636764160,"available":96566575104,"internally_reserved":0,"allocated":1363968,"data_stored":588997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.3340000000000001,"15min":1.6559999999999999},"min":{"1min":0.80200000000000005,"5min":0.65600000000000003,"15min":0.57099999999999995},"max":{"1min":1.8,"5min":19.224,"15min":27.853999999999999},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.149,"5min":1.161,"15min":1.7030000000000001},"min":{"1min":0.61699999999999999,"5min":0.57399999999999995,"15min":0.47199999999999998},"max":{"1min":1.742,"5min":1.742,"15min":34.939999999999998},"last":1.056}]},{"osd":1,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.218,"15min":1.4630000000000001},"min":{"1min":0.91800000000000004,"5min":0.79200000000000004,"15min":0.374},"max":{"1min":1.724,"5min":2.5419999999999998,"15min":13.859},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2509999999999999,"5min":1.496,"15min":1.7250000000000001},"min":{"1min":0.96199999999999997,"5min":0.68400000000000005,"15min":0.502},"max":{"1min":1.5589999999999999,"5min":19.178000000000001,"15min":33.225000000000001},"last":0.996}]},{"osd":2,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":1.526,"15min":1.647},"min":{"1min":0.85299999999999998,"5min":0.74299999999999999,"15min":0.55900000000000005},"max":{"1min":1.679,"5min":19.126000000000001,"15min":19.126000000000001},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.321,"15min":1.7549999999999999},"min":{"1min":0.89000000000000001,"5min":0.69499999999999995,"15min":0.53100000000000003},"max":{"1min":1.6850000000000001,"5min":2.9750000000000001,"15min":35.011000000000003},"last":0.86699999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3240000000000001,"15min":1.6359999999999999},"min":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.374},"max":{"1min":1.8240000000000001,"5min":2.5339999999999998,"15min":16.353000000000002},"last":1.103},{"interface":"front","average":{"1min":1.47,"5min":1.6970000000000001,"15min":1.8009999999999999},"min":{"1min":1.0009999999999999,"5min":0.88900000000000001,"15min":0.57899999999999996},"max":{"1min":1.9750000000000001,"5min":19.268999999999998,"15min":19.268999999999998},"last":1.2270000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.254,"5min":1.2769999999999999,"15min":1.8109999999999999},"min":{"1min":0.752,"5min":0.69099999999999995,"15min":0.40799999999999997},"max":{"1min":1.669,"5min":1.984,"15min":34.972999999999999},"last":1.272},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.3400000000000001,"15min":1.7609999999999999},"min":{"1min":0.85699999999999998,"5min":0.71399999999999997,"15min":0.40400000000000003},"max":{"1min":1.9239999999999999,"5min":2.649,"15min":16.003},"last":1.173}]},{"osd":5,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5269999999999999,"15min":1.9710000000000001},"min":{"1min":1.048,"5min":0.79400000000000004,"15min":0.58099999999999996},"max":{"1min":1.825,"5min":2.851,"15min":23.760999999999999},"last":1.373},{"interface":"front","average":{"1min":1.5549999999999999,"5min":1.6950000000000001,"15min":1.9370000000000001},"min":{"1min":1.175,"5min":0.55100000000000005,"15min":0.52400000000000002},"max":{"1min":2.1680000000000001,"5min":19.036999999999999,"15min":23.716999999999999},"last":1.4339999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:19:21 2024","interfaces":[{"interface":"back","average":{"1min":1.401,"5min":1.401,"15min":2.0019999999999998},"min":{"1min":0.92600000000000005,"5min":0.747,"15min":0.48099999999999998},"max":{"1min":1.8149999999999999,"5min":2.1539999999999999,"15min":34.889000000000003},"last":1.6699999999999999},{"interface":"front","average":{"1min":1.6379999999999999,"5min":1.5760000000000001,"15min":2.153},"min":{"1min":1.169,"5min":0.86599999999999999,"15min":0.47599999999999998},"max":{"1min":1.962,"5min":2.7549999999999999,"15min":46.305},"last":1.589}]}]},{"osd":4,"up_from":15,"seq":64424509680,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68540,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94303300,"statfs":{"total":96636764160,"available":96566579200,"internally_reserved":0,"allocated":1359872,"data_stored":588927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68811205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.208,"15min":1.5069999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.38700000000000001},"max":{"1min":1.7989999999999999,"5min":2.0350000000000001,"15min":22.757999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2190000000000001,"15min":1.2969999999999999},"min":{"1min":0.81999999999999995,"5min":0.69799999999999995,"15min":0.432},"max":{"1min":1.9530000000000001,"5min":1.9530000000000001,"15min":14.034000000000001},"last":0.873}]},{"osd":1,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1739999999999999,"15min":1.548},"min":{"1min":0.875,"5min":0.71899999999999997,"15min":0.51600000000000001},"max":{"1min":1.7150000000000001,"5min":1.851,"15min":18.347000000000001},"last":1.02},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.2769999999999999,"15min":1.484},"min":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.498},"max":{"1min":1.7110000000000001,"5min":2.673,"15min":18.858000000000001},"last":1.1000000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.258,"15min":1.653},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.45600000000000002},"max":{"1min":1.6879999999999999,"5min":1.768,"15min":17.651},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.371,"15min":1.726},"min":{"1min":1.0429999999999999,"5min":0.83499999999999996,"15min":0.52600000000000002},"max":{"1min":2.0019999999999998,"5min":2.7280000000000002,"15min":29.161000000000001},"last":1.208}]},{"osd":3,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.377,"15min":1.5740000000000001},"min":{"1min":0.83699999999999997,"5min":0.76200000000000001,"15min":0.41699999999999998},"max":{"1min":1.7629999999999999,"5min":2.6989999999999998,"15min":26.741},"last":1.361},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.3340000000000001,"15min":1.7350000000000001},"min":{"1min":1.093,"5min":0.78500000000000003,"15min":0.54800000000000004},"max":{"1min":1.905,"5min":1.905,"15min":26.701000000000001},"last":1.1779999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4359999999999999,"15min":1.6930000000000001},"min":{"1min":0.96299999999999997,"5min":0.88500000000000001,"15min":0.441},"max":{"1min":1.6950000000000001,"5min":2.7730000000000001,"15min":18.919},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.393,"5min":1.4710000000000001,"15min":1.6479999999999999},"min":{"1min":1.054,"5min":0.78900000000000003,"15min":0.36399999999999999},"max":{"1min":1.782,"5min":2.0270000000000001,"15min":26.597000000000001},"last":1.254}]},{"osd":6,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.381,"15min":1.8959999999999999},"min":{"1min":1.0129999999999999,"5min":0.98899999999999999,"15min":0.41499999999999998},"max":{"1min":1.6359999999999999,"5min":5.0750000000000002,"15min":32.649999999999999},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.518,"15min":1.7989999999999999},"min":{"1min":1.085,"5min":0.78200000000000003,"15min":0.48099999999999998},"max":{"1min":1.716,"5min":2.8170000000000002,"15min":24.094999999999999},"last":1.716}]},{"osd":7,"last update":"Tue Jan 16 15:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.4810000000000001,"15min":1.762},"min":{"1min":1.2250000000000001,"5min":1.0489999999999999,"15min":0.41699999999999998},"max":{"1min":1.667,"5min":1.8899999999999999,"15min":26.683},"last":1.667},{"interface":"front","average":{"1min":1.464,"5min":1.5169999999999999,"15min":1.853},"min":{"1min":1.143,"5min":1.093,"15min":0.46500000000000002},"max":{"1min":1.6579999999999999,"5min":2.2669999999999999,"15min":17.765000000000001},"last":1.6200000000000001}]}]},{"osd":7,"up_from":14,"seq":60129542382,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73276,"kb_used_data":1904,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298564,"statfs":{"total":96636764160,"available":96561729536,"internally_reserved":0,"allocated":1949696,"data_stored":1173583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.232,"15min":1.538},"min":{"1min":0.70999999999999996,"5min":0.65400000000000003,"15min":0.36899999999999999},"max":{"1min":1.6599999999999999,"5min":2.1779999999999999,"15min":15.661},"last":1.675},{"interface":"front","average":{"1min":1.242,"5min":1.23,"15min":1.5820000000000001},"min":{"1min":0.97399999999999998,"5min":0.67900000000000005,"15min":0.40300000000000002},"max":{"1min":1.4330000000000001,"5min":2.29,"15min":30.126000000000001},"last":1.4910000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.2450000000000001,"15min":1.464},"min":{"1min":0.94999999999999996,"5min":0.63200000000000001,"15min":0.48599999999999999},"max":{"1min":1.732,"5min":2.085,"15min":24.338000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.302,"5min":1.2629999999999999,"15min":1.6140000000000001},"min":{"1min":0.93300000000000005,"5min":0.745,"15min":0.46200000000000002},"max":{"1min":1.7050000000000001,"5min":1.7050000000000001,"15min":21.298999999999999},"last":1.0369999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.286,"15min":1.496},"min":{"1min":0.97299999999999998,"5min":0.71899999999999997,"15min":0.51500000000000001},"max":{"1min":1.5669999999999999,"5min":3.726,"15min":12.474},"last":1.538},{"interface":"front","average":{"1min":1.387,"5min":1.55,"15min":1.7230000000000001},"min":{"1min":0.79900000000000004,"5min":0.69299999999999995,"15min":0.54700000000000004},"max":{"1min":1.8240000000000001,"5min":19.763999999999999,"15min":42.165999999999997},"last":0.95999999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.337,"15min":1.581},"min":{"1min":0.86299999999999999,"5min":0.85699999999999998,"15min":0.47699999999999998},"max":{"1min":1.6499999999999999,"5min":1.887,"15min":12.58},"last":1.375},{"interface":"front","average":{"1min":1.492,"5min":1.508,"15min":1.8100000000000001},"min":{"1min":1.1160000000000001,"5min":0.78600000000000003,"15min":0.54600000000000004},"max":{"1min":2.3180000000000001,"5min":6.6029999999999998,"15min":42.222000000000001},"last":1.24}]},{"osd":4,"last update":"Tue Jan 16 15:19:25 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.3080000000000001,"15min":1.5600000000000001},"min":{"1min":0.90800000000000003,"5min":0.79700000000000004,"15min":0.42799999999999999},"max":{"1min":1.8799999999999999,"5min":2.0219999999999998,"15min":14.659000000000001},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.419,"15min":1.7929999999999999},"min":{"1min":0.95699999999999996,"5min":0.57199999999999995,"15min":0.38800000000000001},"max":{"1min":2.4780000000000002,"5min":2.4780000000000002,"15min":42.276000000000003},"last":1.4430000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.284,"15min":1.7989999999999999},"min":{"1min":0.96199999999999997,"5min":0.59999999999999998,"15min":0.36699999999999999},"max":{"1min":1.7809999999999999,"5min":1.7809999999999999,"15min":26.23},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.48,"15min":1.8089999999999999},"min":{"1min":1.1319999999999999,"5min":0.89200000000000002,"15min":0.41099999999999998},"max":{"1min":1.954,"5min":1.9630000000000001,"15min":21.312000000000001},"last":1.7709999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:19:25 2024","interfaces":[{"interface":"back","average":{"1min":1.5740000000000001,"5min":1.54,"15min":1.855},"min":{"1min":1.1579999999999999,"5min":0.67100000000000004,"15min":0.501},"max":{"1min":2.653,"5min":2.653,"15min":42.090000000000003},"last":1.5960000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.4970000000000001,"15min":1.9319999999999999},"min":{"1min":1.125,"5min":0.59099999999999997,"15min":0.40799999999999997},"max":{"1min":2.1709999999999998,"5min":2.2050000000000001,"15min":29.991},"last":1.819}]}]},{"osd":5,"up_from":14,"seq":60129542383,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64380,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307460,"statfs":{"total":96636764160,"available":96570839040,"internally_reserved":0,"allocated":1355776,"data_stored":583203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.1779999999999999,"15min":1.5609999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.32900000000000001},"max":{"1min":1.742,"5min":6.0720000000000001,"15min":19.382999999999999},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.3160000000000001,"15min":1.363},"min":{"1min":0.76100000000000001,"5min":0.66700000000000004,"15min":0.57099999999999995},"max":{"1min":1.5760000000000001,"5min":13.127000000000001,"15min":13.127000000000001},"last":0.76200000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.2,"15min":1.6399999999999999},"min":{"1min":0.84199999999999997,"5min":0.70699999999999996,"15min":0.52000000000000002},"max":{"1min":1.53,"5min":3.8109999999999999,"15min":17.693999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.4510000000000001,"15min":1.7250000000000001},"min":{"1min":0.82199999999999995,"5min":0.79600000000000004,"15min":0.432},"max":{"1min":1.647,"5min":17.062000000000001,"15min":20.645},"last":1.2230000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.425,"15min":1.7310000000000001},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.42499999999999999},"max":{"1min":1.825,"5min":5.8810000000000002,"15min":21.138000000000002},"last":1.407},{"interface":"front","average":{"1min":1.341,"5min":1.4019999999999999,"15min":1.5169999999999999},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.58099999999999996},"max":{"1min":1.7050000000000001,"5min":4.0720000000000001,"15min":11.321},"last":1.2749999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.341,"15min":1.8029999999999999},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.59799999999999998},"max":{"1min":1.748,"5min":2.4769999999999999,"15min":23.109000000000002},"last":1.038},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.4350000000000001,"15min":1.766},"min":{"1min":0.747,"5min":0.72899999999999998,"15min":0.47399999999999998},"max":{"1min":1.6910000000000001,"5min":5.9489999999999998,"15min":19.303000000000001},"last":0.99199999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.1399999999999999,"5min":1.5069999999999999,"15min":1.8779999999999999},"min":{"1min":0.91900000000000004,"5min":0.85799999999999998,"15min":0.442},"max":{"1min":1.5880000000000001,"5min":15.975,"15min":24.436},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.359,"5min":1.4279999999999999,"15min":1.7509999999999999},"min":{"1min":0.98299999999999998,"5min":0.746,"15min":0.434},"max":{"1min":1.669,"5min":5.6349999999999998,"15min":19.126000000000001},"last":1.3320000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:30 2024","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.446,"15min":1.6299999999999999},"min":{"1min":0.84799999999999998,"5min":0.77000000000000002,"15min":0.46500000000000002},"max":{"1min":1.7090000000000001,"5min":12.914,"15min":21.192},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.421,"5min":1.6120000000000001,"15min":2.0249999999999999},"min":{"1min":1.151,"5min":0.85999999999999999,"15min":0.48599999999999999},"max":{"1min":1.7789999999999999,"5min":16.552,"15min":27.042000000000002},"last":1.1819999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":1.55,"15min":1.9179999999999999},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.51200000000000001},"max":{"1min":1.7769999999999999,"5min":13.034000000000001,"15min":32.405999999999999},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5509999999999999,"15min":2.0659999999999998},"min":{"1min":0.71799999999999997,"5min":0.70899999999999996,"15min":0.56000000000000005},"max":{"1min":1.8169999999999999,"5min":2.1429999999999998,"15min":24.367000000000001},"last":1.5029999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542383,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64388,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94307452,"statfs":{"total":96636764160,"available":96570830848,"internally_reserved":0,"allocated":1363968,"data_stored":584281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":64551367},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":0.98899999999999999,"15min":1.29},"min":{"1min":0.69799999999999995,"5min":0.315,"15min":0.28999999999999998},"max":{"1min":1.5249999999999999,"5min":2.02,"15min":31.562999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.099,"5min":1.1930000000000001,"15min":1.1950000000000001},"min":{"1min":0.73299999999999998,"5min":0.504,"15min":0.39600000000000002},"max":{"1min":1.6359999999999999,"5min":7.1429999999999998,"15min":11.361000000000001},"last":1.1479999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.1160000000000001,"15min":1.1479999999999999},"min":{"1min":0.71399999999999997,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.282,"5min":10.856999999999999,"15min":15.569000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.218,"15min":1.4470000000000001},"min":{"1min":0.86299999999999999,"5min":0.45000000000000001,"15min":0.36599999999999999},"max":{"1min":1.7,"5min":2.0089999999999999,"15min":28.352},"last":1.252}]},{"osd":2,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.488,"15min":1.7230000000000001},"min":{"1min":1.004,"5min":0.57399999999999995,"15min":0.47799999999999998},"max":{"1min":1.732,"5min":7.8159999999999998,"15min":39.265999999999998},"last":1.363},{"interface":"front","average":{"1min":1.1950000000000001,"5min":1.2849999999999999,"15min":1.341},"min":{"1min":0.90900000000000003,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":1.5760000000000001,"5min":7.9160000000000004,"15min":13.526},"last":1.3899999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.3859999999999999,"15min":1.546},"min":{"1min":1.099,"5min":0.73799999999999999,"15min":0.53800000000000003},"max":{"1min":1.9179999999999999,"5min":4.5570000000000004,"15min":20.978999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.514,"15min":1.863},"min":{"1min":1.2909999999999999,"5min":0.93999999999999995,"15min":0.72999999999999998},"max":{"1min":2.165,"5min":2.165,"15min":28.547999999999998},"last":1.5780000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.603,"15min":1.798},"min":{"1min":1.0209999999999999,"5min":0.77800000000000002,"15min":0.67000000000000004},"max":{"1min":1.8009999999999999,"5min":14.003,"15min":20.663},"last":1.095},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.591,"15min":1.9610000000000001},"min":{"1min":1.056,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":28.588000000000001},"last":1.4410000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.536,"5min":1.544,"15min":1.736},"min":{"1min":1.115,"5min":0.85299999999999998,"15min":0.69799999999999995},"max":{"1min":2.0870000000000002,"5min":7.0060000000000002,"15min":13.691000000000001},"last":1.5369999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.6899999999999999,"15min":1.9419999999999999},"min":{"1min":1.21,"5min":0.83399999999999996,"15min":0.76400000000000001},"max":{"1min":2.0619999999999998,"5min":15.114000000000001,"15min":26.698},"last":1.208}]},{"osd":7,"last update":"Tue Jan 16 15:18:28 2024","interfaces":[{"interface":"back","average":{"1min":1.6259999999999999,"5min":1.7909999999999999,"15min":1.8149999999999999},"min":{"1min":1.1899999999999999,"5min":0.81899999999999995,"15min":0.60499999999999998},"max":{"1min":2.1829999999999998,"5min":12.378,"15min":13.622},"last":1.609},{"interface":"front","average":{"1min":1.655,"5min":1.677,"15min":2.165},"min":{"1min":1.1519999999999999,"5min":0.90400000000000003,"15min":0.80400000000000005},"max":{"1min":2.0009999999999999,"5min":4.6029999999999998,"15min":49.811},"last":1.4810000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542382,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76856,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294984,"statfs":{"total":96636764160,"available":96558063616,"internally_reserved":0,"allocated":1347584,"data_stored":582599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.93899999999999995,"15min":1.038},"min":{"1min":0.59799999999999998,"5min":0.372,"15min":0.34200000000000003},"max":{"1min":1.4470000000000001,"5min":1.982,"15min":18.289999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":1.087,"5min":1.0840000000000001,"15min":1.1180000000000001},"min":{"1min":0.61899999999999999,"5min":0.51000000000000001,"15min":0.28599999999999998},"max":{"1min":1.5489999999999999,"5min":2.4790000000000001,"15min":7.9960000000000004},"last":1.165}]},{"osd":1,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":1.1140000000000001,"15min":1.1910000000000001},"min":{"1min":0.68100000000000005,"5min":0.501,"15min":0.40300000000000002},"max":{"1min":1.4650000000000001,"5min":1.931,"15min":23.135999999999999},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.248,"5min":1.214,"15min":1.23},"min":{"1min":0.70399999999999996,"5min":0.48399999999999999,"15min":0.41199999999999998},"max":{"1min":1.7629999999999999,"5min":3.012,"15min":4.8849999999999998},"last":1.262}]},{"osd":3,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.218,"15min":1.1890000000000001},"min":{"1min":0.64900000000000002,"5min":0.44700000000000001,"15min":0.34300000000000003},"max":{"1min":1.5720000000000001,"5min":7.306,"15min":18.838000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.4950000000000001,"15min":1.4950000000000001},"min":{"1min":0.73699999999999999,"5min":0.58599999999999997,"15min":0.45800000000000002},"max":{"1min":1.7549999999999999,"5min":2.6030000000000002,"15min":19.375},"last":0.98499999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.397,"15min":1.587},"min":{"1min":0.84699999999999998,"5min":0.79300000000000004,"15min":0.59999999999999998},"max":{"1min":1.8,"5min":5.9530000000000003,"15min":23.225000000000001},"last":1.228},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.474,"15min":1.5800000000000001},"min":{"1min":1.125,"5min":0.97399999999999998,"15min":0.59199999999999997},"max":{"1min":2.2080000000000002,"5min":3.4460000000000002,"15min":9.8960000000000008},"last":1.3480000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.4790000000000001,"15min":1.619},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.70999999999999996},"max":{"1min":2.113,"5min":7.7640000000000002,"15min":20.157},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.5329999999999999,"15min":1.593},"min":{"1min":0.95499999999999996,"5min":0.84299999999999997,"15min":0.623},"max":{"1min":2.2320000000000002,"5min":7.9820000000000002,"15min":25.004000000000001},"last":1.633}]},{"osd":6,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.5740000000000001,"15min":1.736},"min":{"1min":0.92000000000000004,"5min":0.89400000000000002,"15min":0.70499999999999996},"max":{"1min":2.1709999999999998,"5min":7.6630000000000003,"15min":34.523000000000003},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.663,"15min":1.8300000000000001},"min":{"1min":1.1080000000000001,"5min":0.81100000000000005,"15min":0.74099999999999999},"max":{"1min":2.0409999999999999,"5min":7.8680000000000003,"15min":28.760999999999999},"last":1.7330000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:19:17 2024","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.534,"15min":1.784},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.64100000000000001},"max":{"1min":1.984,"5min":3.1219999999999999,"15min":22.834},"last":1.548},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.5960000000000001,"15min":2.0990000000000002},"min":{"1min":1.095,"5min":0.91000000000000003,"15min":0.82799999999999996},"max":{"1min":1.871,"5min":2.9350000000000001,"15min":39.685000000000002},"last":1.4059999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542384,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76852,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":94294988,"statfs":{"total":96636764160,"available":96558067712,"internally_reserved":0,"allocated":1355776,"data_stored":582723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.95099999999999996,"15min":1.3009999999999999},"min":{"1min":0.67600000000000005,"5min":0.48599999999999999,"15min":0.28100000000000003},"max":{"1min":1.474,"5min":2.024,"15min":31.178999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.147,"5min":1.125,"15min":1.306},"min":{"1min":0.79000000000000004,"5min":0.47299999999999998,"15min":0.247},"max":{"1min":1.4910000000000001,"5min":2.7679999999999998,"15min":26.859999999999999},"last":1.284}]},{"osd":2,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.147,"15min":1.163},"min":{"1min":0.72699999999999998,"5min":0.45400000000000001,"15min":0.37},"max":{"1min":1.6899999999999999,"5min":3.6819999999999999,"15min":6.6120000000000001},"last":1.173},{"interface":"front","average":{"1min":1.3540000000000001,"5min":1.3819999999999999,"15min":1.538},"min":{"1min":1.032,"5min":0.40100000000000002,"15min":0.39000000000000001},"max":{"1min":1.587,"5min":3.5600000000000001,"15min":27.408000000000001},"last":1.7330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.3180000000000001,"15min":1.655},"min":{"1min":0.98599999999999999,"5min":0.56299999999999994,"15min":0.443},"max":{"1min":1.7849999999999999,"5min":2.238,"15min":27.609999999999999},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.1899999999999999,"5min":1.24,"15min":1.6539999999999999},"min":{"1min":0.86199999999999999,"5min":0.45700000000000002,"15min":0.36399999999999999},"max":{"1min":1.627,"5min":2.9750000000000001,"15min":75.016000000000005},"last":1.605}]},{"osd":4,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.4079999999999999,"15min":1.496},"min":{"1min":1.0129999999999999,"5min":0.90500000000000003,"15min":0.58099999999999996},"max":{"1min":1.6599999999999999,"5min":3.3420000000000001,"15min":10.577999999999999},"last":1.4139999999999999},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.5149999999999999,"15min":1.718},"min":{"1min":1.101,"5min":0.78400000000000003,"15min":0.67000000000000004},"max":{"1min":1.7809999999999999,"5min":3.6920000000000002,"15min":27.920999999999999},"last":1.8400000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4830000000000001,"15min":1.5620000000000001},"min":{"1min":1.161,"5min":0.76600000000000001,"15min":0.63700000000000001},"max":{"1min":1.7250000000000001,"5min":3.032,"15min":17.469000000000001},"last":1.698},{"interface":"front","average":{"1min":1.407,"5min":1.4710000000000001,"15min":1.6120000000000001},"min":{"1min":1.0609999999999999,"5min":0.84499999999999997,"15min":0.53300000000000003},"max":{"1min":1.75,"5min":3.3940000000000001,"15min":8.2989999999999995},"last":1.966}]},{"osd":6,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.573,"15min":1.726},"min":{"1min":1.155,"5min":0.85199999999999998,"15min":0.56999999999999995},"max":{"1min":2.0550000000000002,"5min":3.9870000000000001,"15min":17.428999999999998},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5820000000000001,"15min":1.8939999999999999},"min":{"1min":1.2130000000000001,"5min":1.0009999999999999,"15min":0.65600000000000003},"max":{"1min":1.9039999999999999,"5min":3.8180000000000001,"15min":18.850999999999999},"last":1.651}]},{"osd":7,"last update":"Tue Jan 16 15:18:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.6919999999999999,"15min":1.9370000000000001},"min":{"1min":1.2689999999999999,"5min":0.68899999999999995,"15min":0.48099999999999998},"max":{"1min":1.9330000000000001,"5min":7.0499999999999998,"15min":30.834},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.677,"5min":1.6679999999999999,"15min":2.0550000000000002},"min":{"1min":1.2849999999999999,"5min":0.89900000000000002,"15min":0.69099999999999995},"max":{"1min":2.1600000000000001,"5min":3.7839999999999998,"15min":27.716999999999999},"last":1.9179999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542382,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73296,"kb_used_data":1916,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298544,"statfs":{"total":96636764160,"available":96561709056,"internally_reserved":0,"allocated":1961984,"data_stored":1174691,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":1.032,"15min":1.206},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.32300000000000001},"max":{"1min":1.2969999999999999,"5min":10.113,"15min":22.132999999999999},"last":1.246},{"interface":"front","average":{"1min":1.014,"5min":1.0660000000000001,"15min":1.4970000000000001},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.33900000000000002},"max":{"1min":1.5780000000000001,"5min":1.732,"15min":50.429000000000002},"last":1.3859999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.1599999999999999,"15min":1.425},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.38},"max":{"1min":1.6970000000000001,"5min":1.9370000000000001,"15min":22.716999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.077,"5min":1.2110000000000001,"15min":1.613},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.38},"max":{"1min":1.6599999999999999,"5min":1.889,"15min":50.581000000000003},"last":1.169}]},{"osd":3,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.1559999999999999,"15min":1.1839999999999999},"min":{"1min":0.45900000000000002,"5min":0.45900000000000002,"15min":0.372},"max":{"1min":1.371,"5min":5.3280000000000003,"15min":5.8719999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.1819999999999999,"5min":1.375,"15min":1.5609999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.46700000000000003},"max":{"1min":1.841,"5min":2.5859999999999999,"15min":22.254999999999999},"last":1.3440000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.4850000000000001,"15min":1.8100000000000001},"min":{"1min":0.95599999999999996,"5min":0.78800000000000003,"15min":0.70399999999999996},"max":{"1min":1.804,"5min":2.8079999999999998,"15min":22.407},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.423,"15min":1.893},"min":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.57899999999999996},"max":{"1min":1.639,"5min":2.7770000000000001,"15min":50.497},"last":1.635}]},{"osd":5,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.4330000000000001,"15min":1.591},"min":{"1min":0.97599999999999998,"5min":0.83199999999999996,"15min":0.51900000000000002},"max":{"1min":1.7250000000000001,"5min":5.5609999999999999,"15min":14.112},"last":1.028},{"interface":"front","average":{"1min":1.373,"5min":1.5680000000000001,"15min":1.8600000000000001},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.71699999999999997},"max":{"1min":1.887,"5min":2.6840000000000002,"15min":22.882999999999999},"last":1.5309999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.304,"5min":1.4790000000000001,"15min":1.7549999999999999},"min":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.55600000000000005},"max":{"1min":1.647,"5min":5.4509999999999996,"15min":31.387},"last":1.296},{"interface":"front","average":{"1min":1.498,"5min":1.6639999999999999,"15min":1.984},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.64900000000000002},"max":{"1min":1.974,"5min":3.5209999999999999,"15min":23.547000000000001},"last":1.6659999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:19:14 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.569,"15min":2.069},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.67900000000000005},"max":{"1min":1.764,"5min":2.8679999999999999,"15min":50.542000000000002},"last":1.917},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.6040000000000001,"15min":1.8080000000000001},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.58399999999999996},"max":{"1min":1.952,"5min":5.5060000000000002,"15min":15.747},"last":1.488}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:19:29.644 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-01-16T15:19:29.645 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-01-16T15:19:29.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.925+0000 7fa0fb200640 1 Processor -- start 2024-01-16T15:19:29.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.925+0000 7fa0fb200640 1 -- start start 2024-01-16T15:19:29.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.926+0000 7fa0fb200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0fc13ce60 0x7fa0fc15e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.926+0000 7fa0fb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc15e5f0 0x7fa0fc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.926+0000 7fa0fb200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc13f6b0 con 0x7fa0fc06d900 2024-01-16T15:19:29.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.926+0000 7fa0fb200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0fc1402f0 con 0x7fa0fc13ce60 2024-01-16T15:19:29.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.926+0000 7fa0fb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc06dd90 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f9e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa0fc13ce60 0x7fa0fc15e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0fa800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fa0fc06d900 0x7fa0fc071200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55734/0 (socket says 172.21.15.53:55734) 2024-01-16T15:19:29.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0fa800640 1 -- 172.21.15.53:0/2190344700 learned_addr learned my addr 172.21.15.53:0/2190344700 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f9e00640 1 -- 172.21.15.53:0/2190344700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc15e5f0 msgr2=0x7fa0fc06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f3e00640 1 --2- 172.21.15.53:0/2190344700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc15e5f0 0x7fa0fc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f9e00640 1 --2- 172.21.15.53:0/2190344700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc15e5f0 0x7fa0fc06d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f9e00640 1 -- 172.21.15.53:0/2190344700 >> v1:172.21.15.53:6789/0 conn(0x7fa0fc06d900 legacy=0x7fa0fc071200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f9e00640 1 -- 172.21.15.53:0/2190344700 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc13d240 con 0x7fa0fc13ce60 2024-01-16T15:19:29.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.927+0000 7fa0f9400640 1 -- 172.21.15.53:0/2190344700 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713910407 0 0) 0x7fa0fc13f6b0 con 0x7fa0fc06d900 2024-01-16T15:19:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.929+0000 7fa0f9e00640 1 --2- 172.21.15.53:0/2190344700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc13ce60 0x7fa0fc15e0b0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0097a0 tx=0x7fa0ec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b02d5de225f71c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.929+0000 7fa0f9400640 1 -- 172.21.15.53:0/2190344700 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec031830 con 0x7fa0fc13ce60 2024-01-16T15:19:29.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.929+0000 7fa0f9400640 1 -- 172.21.15.53:0/2190344700 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ec031990 con 0x7fa0fc13ce60 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.929+0000 7fa0f9400640 1 -- 172.21.15.53:0/2190344700 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ec031d20 con 0x7fa0fc13ce60 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.929+0000 7fa0fb200640 1 -- 172.21.15.53:0/2190344700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc13ce60 msgr2=0x7fa0fc15e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.929+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2190344700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc13ce60 0x7fa0fc15e0b0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec0097a0 tx=0x7fa0ec02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.930+0000 7fa0fb200640 1 -- 172.21.15.53:0/2190344700 shutdown_connections 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.930+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2190344700 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc15e5f0 0x7fa0fc06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.930+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2190344700 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc13ce60 0x7fa0fc15e0b0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.930+0000 7fa0fb200640 1 -- 172.21.15.53:0/2190344700 >> 172.21.15.53:0/2190344700 conn(0x7fa0fc0b0b70 msgr2=0x7fa0fc0b0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.930+0000 7fa0fb200640 1 -- 172.21.15.53:0/2190344700 shutdown_connections 2024-01-16T15:19:29.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.930+0000 7fa0fb200640 1 -- 172.21.15.53:0/2190344700 wait complete. 2024-01-16T15:19:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.931+0000 7fa0fb200640 1 Processor -- start 2024-01-16T15:19:29.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.932+0000 7fa0fb200640 1 -- start start 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fb200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0fc06d900 0x7fa0fc135680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fb200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc13ce60 0x7fa0fc138da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 0x7fa0fc137a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fb200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc06fff0 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fb200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc138170 con 0x7fa0fc06d900 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fb200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0fc070240 con 0x7fa0fc13ce60 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fa800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 0x7fa0fc137a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fa800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 0x7fa0fc137a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55938/0 (socket says 172.21.15.53:55938) 2024-01-16T15:19:29.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fa800640 1 -- 172.21.15.53:0/2637220448 learned_addr learned my addr 172.21.15.53:0/2637220448 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:29.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fa800640 1 -- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc13ce60 msgr2=0x7fa0fc138da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fa800640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc13ce60 0x7fa0fc138da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0fa800640 1 -- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0fc06d900 msgr2=0x7fa0fc135680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.933+0000 7fa0f9e00640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0fc06d900 0x7fa0fc135680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0fa800640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0fc06d900 0x7fa0fc135680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0fa800640 1 -- 172.21.15.53:0/2637220448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec009400 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0f9e00640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0fc06d900 0x7fa0fc135680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0fa800640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 0x7fa0fc137a70 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fa0f400c780 tx=0x7fa0f400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400ce90 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f400a640 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0f400a9d0 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0fc070520 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.934+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0fc13a5e0 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.936+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0fc070a40 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.937+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fa0f400ab30 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.937+0000 7fa0f2000640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa0d406c430 0x7fa0d406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:29.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.938+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fa0f4088030 con 0x7fa0fc15e5f0 2024-01-16T15:19:29.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.938+0000 7fa0f9e00640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa0d406c430 0x7fa0d406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:29.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.940+0000 7fa0f9e00640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa0d406c430 0x7fa0d406e8f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec036000 tx=0x7fa0ec008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:29.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:29.942+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa0fc070a40 con 0x7fa0fc15e5f0 2024-01-16T15:19:30.198 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.206+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa0fc070a40 con 0x7fa0fc15e5f0 2024-01-16T15:19:30.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.211+0000 7fa0f2000640 1 -- 172.21.15.53:0/2637220448 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa0fc070a40 con 0x7fa0fc15e5f0 2024-01-16T15:19:30.203 INFO:teuthology.orchestra.run.smithi053.stdout:[client.0] 2024-01-16T15:19:30.203 INFO:teuthology.orchestra.run.smithi053.stdout: key = AQCumaZlmoVdIRAA9PbFJqtudhpIgY2e9s3YwQ== 2024-01-16T15:19:30.203 INFO:teuthology.orchestra.run.smithi053.stdout: caps mds = "allow" 2024-01-16T15:19:30.203 INFO:teuthology.orchestra.run.smithi053.stdout: caps mon = "allow r" 2024-01-16T15:19:30.203 INFO:teuthology.orchestra.run.smithi053.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-01-16T15:19:30.203 INFO:teuthology.orchestra.run.smithi053.stderr:updated caps for client.0 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.213+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa0d406c430 msgr2=0x7fa0d406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.213+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa0d406c430 0x7fa0d406e8f0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fa0ec036000 tx=0x7fa0ec008030 comp rx=0 tx=0).stop 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.213+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 msgr2=0x7fa0fc137a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.213+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 0x7fa0fc137a70 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fa0f400c780 tx=0x7fa0f400cc40 comp rx=0 tx=0).stop 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 shutdown_connections 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa0d406c430 0x7fa0d406e8f0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa0fc15e5f0 0x7fa0fc137a70 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa0fc13ce60 0x7fa0fc138da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 --2- 172.21.15.53:0/2637220448 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa0fc06d900 0x7fa0fc135680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 >> 172.21.15.53:0/2637220448 conn(0x7fa0fc0b0b70 msgr2=0x7fa0fc15b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 shutdown_connections 2024-01-16T15:19:30.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.214+0000 7fa0fb200640 1 -- 172.21.15.53:0/2637220448 wait complete. 2024-01-16T15:19:30.224 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:19:30.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.445+0000 7f3b26c00640 1 Processor -- start 2024-01-16T15:19:30.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.445+0000 7f3b26c00640 1 -- start start 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b26c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3b20143ae0 0x7f3b200cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b26c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b200cf7d0 0x7f3b2015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b26c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b200cfc30 con 0x7f3b2015e1e0 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b26c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b200b1ee0 con 0x7f3b20143ae0 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b26c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200b2780 con 0x7f3b200cf7d0 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b1fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3b20143ae0 0x7f3b200cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:30.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b24c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3b2015e1e0 0x7f3b2015a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55744/0 (socket says 172.21.15.53:55744) 2024-01-16T15:19:30.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b24c00640 1 -- 172.21.15.53:0/905739180 learned_addr learned my addr 172.21.15.53:0/905739180 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:30.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.447+0000 7f3b17200640 1 --2- 172.21.15.53:0/905739180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b200cf7d0 0x7f3b2015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:30.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.448+0000 7f3b1fe00640 1 -- 172.21.15.53:0/905739180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b200cf7d0 msgr2=0x7f3b2015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:30.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.448+0000 7f3b1fe00640 1 --2- 172.21.15.53:0/905739180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b200cf7d0 0x7f3b2015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.448+0000 7f3b1fe00640 1 -- 172.21.15.53:0/905739180 >> v1:172.21.15.53:6789/0 conn(0x7f3b2015e1e0 legacy=0x7f3b2015a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.448+0000 7f3b1fe00640 1 -- 172.21.15.53:0/905739180 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2015e990 con 0x7f3b20143ae0 2024-01-16T15:19:30.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.448+0000 7f3b1f400640 1 -- 172.21.15.53:0/905739180 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640427372 0 0) 0x7f3b200cfc30 con 0x7f3b2015e1e0 2024-01-16T15:19:30.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.449+0000 7f3b1fe00640 1 --2- 172.21.15.53:0/905739180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b20143ae0 0x7f3b200cf290 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f3b100097a0 tx=0x7f3b1002dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27eefb64d5b9c4cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:30.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.450+0000 7f3b1f400640 1 -- 172.21.15.53:0/905739180 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1002f8a0 con 0x7f3b20143ae0 2024-01-16T15:19:30.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.450+0000 7f3b1f400640 1 -- 172.21.15.53:0/905739180 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1002fa00 con 0x7f3b20143ae0 2024-01-16T15:19:30.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.450+0000 7f3b1f400640 1 -- 172.21.15.53:0/905739180 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b1002fd90 con 0x7f3b20143ae0 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.450+0000 7f3b26c00640 1 -- 172.21.15.53:0/905739180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b20143ae0 msgr2=0x7f3b200cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.450+0000 7f3b26c00640 1 --2- 172.21.15.53:0/905739180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b20143ae0 0x7f3b200cf290 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7f3b100097a0 tx=0x7f3b1002dae0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.451+0000 7f3b26c00640 1 -- 172.21.15.53:0/905739180 shutdown_connections 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.451+0000 7f3b26c00640 1 --2- 172.21.15.53:0/905739180 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b200cf7d0 0x7f3b2015dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.451+0000 7f3b26c00640 1 --2- 172.21.15.53:0/905739180 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b20143ae0 0x7f3b200cf290 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.451+0000 7f3b26c00640 1 -- 172.21.15.53:0/905739180 >> 172.21.15.53:0/905739180 conn(0x7f3b2006a800 msgr2=0x7f3b2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.451+0000 7f3b26c00640 1 -- 172.21.15.53:0/905739180 shutdown_connections 2024-01-16T15:19:30.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.451+0000 7f3b26c00640 1 -- 172.21.15.53:0/905739180 wait complete. 2024-01-16T15:19:30.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.453+0000 7f3b26c00640 1 Processor -- start 2024-01-16T15:19:30.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.453+0000 7f3b26c00640 1 -- start start 2024-01-16T15:19:30.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b26c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b200cf7d0 0x7f3b2013ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b26c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b20143ae0 0x7f3b2015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b26c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 0x7f3b2015b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b26c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2015a830 con 0x7f3b2015e1e0 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b26c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200ccfe0 con 0x7f3b200cf7d0 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b26c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b200cc6b0 con 0x7f3b20143ae0 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b24c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 0x7f3b2015b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b24c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 0x7f3b2015b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55970/0 (socket says 172.21.15.53:55970) 2024-01-16T15:19:30.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b24c00640 1 -- 172.21.15.53:0/275129806 learned_addr learned my addr 172.21.15.53:0/275129806 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:30.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b1fe00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b200cf7d0 0x7f3b2013ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:30.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.455+0000 7f3b17200640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b20143ae0 0x7f3b2015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b24c00640 1 -- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b20143ae0 msgr2=0x7f3b2015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b24c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b20143ae0 0x7f3b2015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b24c00640 1 -- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b200cf7d0 msgr2=0x7f3b2013ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b24c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b200cf7d0 0x7f3b2013ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b24c00640 1 -- 172.21.15.53:0/275129806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b10009400 con 0x7f3b2015e1e0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b24c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 0x7f3b2015b910 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c410 tx=0x7f3b1800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b180074e0 con 0x7f3b2015e1e0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b18007640 con 0x7f3b2015e1e0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18004a20 con 0x7f3b2015e1e0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.456+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b200cc990 con 0x7f3b2015e1e0 2024-01-16T15:19:30.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.457+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2013bfa0 con 0x7f3b2015e1e0 2024-01-16T15:19:30.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.458+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b200b2bf0 con 0x7f3b2015e1e0 2024-01-16T15:19:30.450 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.458+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f3b18004b80 con 0x7f3b2015e1e0 2024-01-16T15:19:30.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.459+0000 7f3b1cc00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3af806c460 0x7f3af806e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:30.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.459+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f3b18088850 con 0x7f3b2015e1e0 2024-01-16T15:19:30.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.460+0000 7f3b1fe00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3af806c460 0x7f3af806e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:30.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.461+0000 7f3b1fe00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3af806c460 0x7f3af806e920 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3b10038000 tx=0x7f3b10008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:30.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.464+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3b200b2bf0 con 0x7f3b2015e1e0 2024-01-16T15:19:30.553 INFO:tasks.ceph.mds.c.smithi053.stderr:2024-01-16T15:19:30.562+0000 7fd0ddc00640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-01-16T15:19:30.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.698+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b200b2bf0 con 0x7f3b2015e1e0 2024-01-16T15:19:30.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.699+0000 7f3b1cc00640 1 -- 172.21.15.53:0/275129806 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f3b200b2bf0 con 0x7f3b2015e1e0 2024-01-16T15:19:30.691 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:19:30.692 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":74,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1519219735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1519219735},{"type":"v1","addr":"172.21.15.146:6837","nonce":1519219735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2058874467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2058874467},{"type":"v1","addr":"172.21.15.146:6836","nonce":2058874467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3227938262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3227938262},{"type":"v1","addr":"172.21.15.53:6835","nonce":3227938262}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:19:24.074687+0000","modified":"2024-01-16T15:19:27.148085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/1996992482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1996992482},{"type":"v1","addr":"172.21.15.53:6834","nonce":1996992482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:19:30.692 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 74 2024-01-16T15:19:30.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3af806c460 msgr2=0x7f3af806e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3af806c460 0x7f3af806e920 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f3b10038000 tx=0x7f3b10008030 comp rx=0 tx=0).stop 2024-01-16T15:19:30.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 msgr2=0x7f3b2015b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 0x7f3b2015b910 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f3b1800c410 tx=0x7f3b1800c8d0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 shutdown_connections 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3af806c460 0x7f3af806e920 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.702+0000 7f3b26c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3b2015e1e0 0x7f3b2015b910 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.703+0000 7f3b26c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3b20143ae0 0x7f3b2015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.703+0000 7f3b26c00640 1 --2- 172.21.15.53:0/275129806 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3b200cf7d0 0x7f3b2013ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.703+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 >> 172.21.15.53:0/275129806 conn(0x7f3b2006a800 msgr2=0x7f3b200702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.703+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 shutdown_connections 2024-01-16T15:19:30.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:30.703+0000 7f3b26c00640 1 -- 172.21.15.53:0/275129806 wait complete. 2024-01-16T15:19:31.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-01-16T15:19:24.074687+0000', 'modified': '2024-01-16T15:19:27.148085+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9054}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9054': {'gid': 9054, 'name': 'c', 'rank': 0, 'incarnation': 70, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.53:6834/1996992482', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.53:6833', 'nonce': 1996992482}, {'type': 'v1', 'addr': '172.21.15.53:6834', 'nonce': 1996992482}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-01-16T15:19:31.117 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-01-16T15:19:31.117 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-01-16T15:19:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.354+0000 7f6851c00640 1 Processor -- start 2024-01-16T15:19:31.346 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.354+0000 7f6851c00640 1 -- start start 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f684c15afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f684c0cd530 0x7f684c13f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684c0cdec0 con 0x7f684c0cd9b0 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f684c0ce1a0 con 0x7f684c0cd530 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0ce480 con 0x7f684c13cfa0 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f684c0cd9b0 0x7f684c15d9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55758/0 (socket says 172.21.15.53:55758) 2024-01-16T15:19:31.348 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.356+0000 7f6851200640 1 -- 172.21.15.53:0/3123179458 learned_addr learned my addr 172.21.15.53:0/3123179458 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f6843e00640 1 --2- 172.21.15.53:0/3123179458 >> v2:172.21.15.146:3300/0 conn(0x7f684c0cd530 0x7f684c13f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684be00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f684c15afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280407739 0 0) 0x7f684c0cdec0 con 0x7f684c0cd9b0 2024-01-16T15:19:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f682c003520 con 0x7f684c0cd9b0 2024-01-16T15:19:31.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1605834193 0 0) 0x7f682c003520 con 0x7f684c0cd9b0 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 msgr2=0x7f684c15afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f684c15afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 >> v2:172.21.15.146:3300/0 conn(0x7f684c0cd530 msgr2=0x7f684c13f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 --2- 172.21.15.53:0/3123179458 >> v2:172.21.15.146:3300/0 conn(0x7f684c0cd530 0x7f684c13f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c0ce760 con 0x7f684c0cd9b0 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6844001a10 con 0x7f684c0cd9b0 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 >> v1:172.21.15.53:6789/0 conn(0x7f684c0cd9b0 legacy=0x7f684c15d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f682c003970 0x7f682c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f682c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0ce1a0 con 0x7f682c003970 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c0ce480 con 0x7f682c0047c0 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.357+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f682c00dbe0 con 0x7f684c13cfa0 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f684be00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f682c003970 0x7f682c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f6843e00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f6851200640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f682c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f684be00640 1 -- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 msgr2=0x7f682c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f684be00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f682c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f684be00640 1 -- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682c0047c0 msgr2=0x7f682c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f684be00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f682c0047c0 0x7f682c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f684be00640 1 -- 172.21.15.53:0/3123179458 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f682c003520 con 0x7f682c003970 2024-01-16T15:19:31.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.358+0000 7f6851200640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f684c13cfa0 0x7f682c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.359+0000 7f684be00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f682c003970 0x7f682c003d90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f683c009af0 tx=0x7f682c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5352cabf742c56 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.359+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02c6b0 con 0x7f682c003970 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c02c810 con 0x7f682c003970 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f684b400640 1 -- 172.21.15.53:0/3123179458 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02cba0 con 0x7f682c003970 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851c00640 1 -- 172.21.15.53:0/3123179458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f682c003970 msgr2=0x7f682c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851c00640 1 --2- 172.21.15.53:0/3123179458 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f682c003970 0x7f682c003d90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f683c009af0 tx=0x7f682c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851200640 1 -- 172.21.15.53:0/3123179458 reap_dead start 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851c00640 1 -- 172.21.15.53:0/3123179458 shutdown_connections 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851c00640 1 -- 172.21.15.53:0/3123179458 >> 172.21.15.53:0/3123179458 conn(0x7f684c06a800 msgr2=0x7f684c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851c00640 1 -- 172.21.15.53:0/3123179458 shutdown_connections 2024-01-16T15:19:31.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.360+0000 7f6851c00640 1 -- 172.21.15.53:0/3123179458 wait complete. 2024-01-16T15:19:31.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.362+0000 7f6851c00640 1 Processor -- start 2024-01-16T15:19:31.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.362+0000 7f6851c00640 1 -- start start 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.363+0000 7f6851c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682c003970 0x7f684c1f3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.363+0000 7f6851c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 0x7f684c13bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.363+0000 7f6851c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684c15cce0 0x7f684c13c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.363+0000 7f6851c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c155b20 con 0x7f684c15c860 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.363+0000 7f6851c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f684c155d10 con 0x7f684c15cce0 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.363+0000 7f6851c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f684c1f4210 con 0x7f682c003970 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 0x7f684c13bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 0x7f684c13bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55992/0 (socket says 172.21.15.53:55992) 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 -- 172.21.15.53:0/2841852524 learned_addr learned my addr 172.21.15.53:0/2841852524 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:31.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 -- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682c003970 msgr2=0x7f684c1f3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682c003970 0x7f684c1f3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 -- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684c15cce0 msgr2=0x7f684c13c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684c15cce0 0x7f684c13c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 -- 172.21.15.53:0/2841852524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c154f10 con 0x7f684c15c860 2024-01-16T15:19:31.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.364+0000 7f6843e00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 0x7f684c13bf30 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f68340116d0 tx=0x7f6834011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.365+0000 7f6848c00640 1 -- 172.21.15.53:0/2841852524 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683400f9c0 con 0x7f684c15c860 2024-01-16T15:19:31.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.365+0000 7f6848c00640 1 -- 172.21.15.53:0/2841852524 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683400fb20 con 0x7f684c15c860 2024-01-16T15:19:31.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.365+0000 7f6848c00640 1 -- 172.21.15.53:0/2841852524 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683400fe90 con 0x7f684c15c860 2024-01-16T15:19:31.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.365+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684c1551f0 con 0x7f684c15c860 2024-01-16T15:19:31.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.365+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c1557b0 con 0x7f684c15c860 2024-01-16T15:19:31.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.367+0000 7f6848c00640 1 -- 172.21.15.53:0/2841852524 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6834015460 con 0x7f684c15c860 2024-01-16T15:19:31.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.368+0000 7f6848c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f682406c430 0x7f682406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.368+0000 7f6848c00640 1 -- 172.21.15.53:0/2841852524 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6834088f50 con 0x7f684c15c860 2024-01-16T15:19:31.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.369+0000 7f684be00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f682406c430 0x7f682406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.369+0000 7f6855de4740 1 Processor -- start 2024-01-16T15:19:31.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.369+0000 7f6855de4740 1 -- start start 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f684be00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f682406c430 0x7f682406e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f683c036000 tx=0x7f683c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024269070 0x55a024269450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024263d70 0x55a024268270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f6855de4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a0242ac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f6855de4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0241c7880 con 0x55a0242687b0 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0241d6ae0 con 0x55a024269070 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.370+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0241d41d0 con 0x55a024263d70 2024-01-16T15:19:31.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a0242ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a0242ac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56006/0 (socket says 172.21.15.53:56006) 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 -- 172.21.15.53:0/3579989995 learned_addr learned my addr 172.21.15.53:0/3579989995 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f684be00640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024269070 0x55a024269450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6843e00640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024263d70 0x55a024268270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 -- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024263d70 msgr2=0x55a024268270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024263d70 0x55a024268270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 -- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024269070 msgr2=0x55a024269450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024269070 0x55a024269450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 -- 172.21.15.53:0/3579989995 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a024197860 con 0x55a0242687b0 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f684be00640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024269070 0x55a024269450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6843e00640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024263d70 0x55a024268270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.371+0000 7f6851200640 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a0242ac730 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f6844001fd0 tx=0x7f684400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137b6f644baef630 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6840c00640 1 -- 172.21.15.53:0/3579989995 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6844004b60 con 0x55a0242687b0 2024-01-16T15:19:31.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6840c00640 1 -- 172.21.15.53:0/3579989995 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844004cc0 con 0x55a0242687b0 2024-01-16T15:19:31.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6840c00640 1 -- 172.21.15.53:0/3579989995 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68440072b0 con 0x55a0242687b0 2024-01-16T15:19:31.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 -- 172.21.15.53:0/3579989995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 msgr2=0x55a0242ac730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a0242ac730 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f6844001fd0 tx=0x7f684400d520 comp rx=0 tx=0).stop 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 -- 172.21.15.53:0/3579989995 shutdown_connections 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a0242ac730 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024263d70 0x55a024268270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 --2- 172.21.15.53:0/3579989995 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024269070 0x55a024269450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 -- 172.21.15.53:0/3579989995 >> 172.21.15.53:0/3579989995 conn(0x55a02422ac70 msgr2=0x55a0242aeb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 -- 172.21.15.53:0/3579989995 shutdown_connections 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.372+0000 7f6855de4740 1 -- 172.21.15.53:0/3579989995 wait complete. 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.373+0000 7f6855de4740 1 -- start start 2024-01-16T15:19:31.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.373+0000 7f6855de4740 1 Processor -- start 2024-01-16T15:19:31.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.374+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024263d70 0x55a02428d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.374+0000 7f6855de4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a02428e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.374+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a0242b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.374+0000 7f6855de4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0241cfac0 con 0x55a0242687b0 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.374+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02416c2f0 con 0x55a024263d70 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.374+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a024189c70 con 0x55a024269070 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6855de4740 10 client.0 _ll_register_callbacks cb 0x55a02418e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a02428e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a02428e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56016/0 (socket says 172.21.15.53:56016) 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 -- 172.21.15.53:0/2920909389 learned_addr learned my addr 172.21.15.53:0/2920909389 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 -- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 msgr2=0x55a0242b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f684be00640 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024263d70 0x55a02428d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a0242b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 -- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024263d70 msgr2=0x55a02428d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024263d70 0x55a02428d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 -- 172.21.15.53:0/2920909389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f684c0ce760 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6843e00640 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a02428e800 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f68340116a0 tx=0x7f68340592f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5585eaae40049c2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.375+0000 7f6839600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6839600640 1 -- 172.21.15.53:0/2920909389 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68340611f0 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6839600640 1 -- 172.21.15.53:0/2920909389 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f683405f080 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0241c8aa0 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6839600640 1 -- 172.21.15.53:0/2920909389 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f683405f280 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 10 client.9195 fetch_fsmap learned FSMap version 74 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x55a0241ce140 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6839600640 1 -- 172.21.15.53:0/2920909389 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6834074030 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6839600640 1 -- 172.21.15.53:0/2920909389 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 74) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f683401b020 con 0x55a0242687b0 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 10 client.9195 fetch_fsmap finished waiting for FSMap version 74 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 10 client.9195 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:19:31.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 20 client.9195 populate_metadata read hostname 'smithi053' 2024-01-16T15:19:31.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a0242932f0 0x55a0242b6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 4 client.9195 mds_command: new command op to 9054 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-01-16T15:19:31.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.376+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 --> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0241c87c0 con 0x55a0242932f0 2024-01-16T15:19:31.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.377+0000 7f684be00640 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a0242932f0 0x55a0242b6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.377+0000 7f684be00640 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a0242932f0 0x55a0242b6360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f9ead405688856b server_cookie=3ad045bf93e5e77b in_seq=0 out_seq=0 2024-01-16T15:19:31.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.377+0000 7f6839600640 10 client.9195 ms_handle_connect on v2:172.21.15.53:6833/1996992482 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.378+0000 7f6839600640 1 -- 172.21.15.53:0/2920909389 <== mds.0 v2:172.21.15.53:6833/1996992482 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a0241c87c0 con 0x55a0242932f0 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.378+0000 7f6839600640 10 client.9195 handle_command_reply: tid=0 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.378+0000 7f6855de4740 1 client.9195 shutdown 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 msgr2=0x55a02428e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a02428e800 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f68340116a0 tx=0x7f68340592f0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 shutdown_connections 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a0242932f0 0x55a0242b6360 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a0242b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a0242687b0 0x55a02428e800 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6839600640 0 client.9195 ms_handle_reset on v2:172.21.15.53:6833/1996992482 2024-01-16T15:19:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 --2- 172.21.15.53:0/2920909389 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a024263d70 0x55a02428d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 >> 172.21.15.53:0/2920909389 conn(0x55a02422ac70 msgr2=0x55a024262dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 shutdown_connections 2024-01-16T15:19:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 1 -- 172.21.15.53:0/2920909389 wait complete. 2024-01-16T15:19:31.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.379+0000 7f6855de4740 20 client.9195 trim_cache size 0 max 16384 2024-01-16T15:19:31.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.396+0000 7f6855de4740 1 Processor -- start 2024-01-16T15:19:31.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.396+0000 7f6855de4740 1 -- start start 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f6855de4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a0242b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a0242d0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a0242ad8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f6855de4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a023f32f20 con 0x55a024263d70 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0242c1870 con 0x55a0242687b0 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0242ae650 con 0x55a024269070 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f684be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a0242b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f684be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a0242b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56032/0 (socket says 172.21.15.53:56032) 2024-01-16T15:19:31.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.397+0000 7f684be00640 1 -- 172.21.15.53:0/1526918161 learned_addr learned my addr 172.21.15.53:0/1526918161 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f684be00640 1 -- 172.21.15.53:0/1526918161 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 msgr2=0x55a0242ad8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f684be00640 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a0242ad8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f684be00640 1 -- 172.21.15.53:0/1526918161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 msgr2=0x55a0242d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f684be00640 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a0242d0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f684be00640 1 -- 172.21.15.53:0/1526918161 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a02416bb80 con 0x55a024263d70 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f684be00640 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a0242b20c0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f683c037040 tx=0x7f683c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360e87498f6e727e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f6841600640 1 -- 172.21.15.53:0/1526918161 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c02cd00 con 0x55a024263d70 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f6841600640 1 -- 172.21.15.53:0/1526918161 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f683c044040 con 0x55a024263d70 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.398+0000 7f6841600640 1 -- 172.21.15.53:0/1526918161 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c04b230 con 0x55a024263d70 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 -- 172.21.15.53:0/1526918161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 msgr2=0x55a0242b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a0242b20c0 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f683c037040 tx=0x7f683c008030 comp rx=0 tx=0).stop 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 -- 172.21.15.53:0/1526918161 shutdown_connections 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a0242ad8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a0242d0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 --2- 172.21.15.53:0/1526918161 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a0242b20c0 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 -- 172.21.15.53:0/1526918161 >> 172.21.15.53:0/1526918161 conn(0x55a02422ac70 msgr2=0x55a02428b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 -- 172.21.15.53:0/1526918161 shutdown_connections 2024-01-16T15:19:31.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.399+0000 7f6855de4740 1 -- 172.21.15.53:0/1526918161 wait complete. 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.400+0000 7f6855de4740 1 -- start start 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.400+0000 7f6855de4740 1 Processor -- start 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a024290100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a024290640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a024262dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0242ae4c0 con 0x55a024263d70 2024-01-16T15:19:31.393 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a02431c920 con 0x55a0242687b0 2024-01-16T15:19:31.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0242adbb0 con 0x55a024269070 2024-01-16T15:19:31.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f6855de4740 10 client.0 _ll_register_callbacks cb 0x7f6834055570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-01-16T15:19:31.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.401+0000 7f684be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a024290100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a024290100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56034/0 (socket says 172.21.15.53:56034) 2024-01-16T15:19:31.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 -- 172.21.15.53:0/1677366636 learned_addr learned my addr 172.21.15.53:0/1677366636 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f6851200640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a024262dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f6843e00640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a024290640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 -- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 msgr2=0x55a024262dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a024262dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 -- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 msgr2=0x55a024290640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a024290640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 -- 172.21.15.53:0/1677366636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682c003520 con 0x55a024263d70 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f684be00640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a024290100 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ac0 tx=0x7f683c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5682d784f61cbac8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.402+0000 7f6839600640 10 client.0 ms_handle_connect on v2:172.21.15.53:3300/0 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6839600640 1 -- 172.21.15.53:0/1677366636 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f683c04b230 con 0x55a024263d70 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6839600640 1 -- 172.21.15.53:0/1677366636 <== mon.0 v2:172.21.15.53:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f683c04b390 con 0x55a024263d70 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6844004cc0 con 0x55a024263d70 2024-01-16T15:19:31.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6839600640 1 -- 172.21.15.53:0/1677366636 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7f683c04b550 con 0x55a024263d70 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 10 client.9201 fetch_fsmap learned FSMap version 74 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x55a024191170 con 0x55a024263d70 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6839600640 1 -- 172.21.15.53:0/1677366636 <== mon.0 v2:172.21.15.53:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6844004cc0 con 0x55a024263d70 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6839600640 1 -- 172.21.15.53:0/1677366636 <== mon.0 v2:172.21.15.53:3300/0 5 ==== fsmap(e 74) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f683c04b6b0 con 0x55a024263d70 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 10 client.9201 fetch_fsmap finished waiting for FSMap version 74 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 10 client.9201 resolve_mds: resolved name 'c' to daemon mds.c 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 20 client.9201 populate_metadata read hostname 'smithi053' 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a024318960 0x55a024263310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 4 client.9201 mds_command: new command op to 9054 tid=0 multi_id=0 [{"prefix": "status"}] 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.403+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 --> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a023ed06c0 con 0x55a024318960 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.404+0000 7f6843e00640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a024318960 0x55a024263310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.404+0000 7f6843e00640 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a024318960 0x55a024263310 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=888aeea09047737f server_cookie=c2748fa65b36cfef in_seq=0 out_seq=0 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.404+0000 7f6839600640 10 client.9201 ms_handle_connect on v2:172.21.15.53:6833/1996992482 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.404+0000 7f6839600640 1 -- 172.21.15.53:0/1677366636 <== mds.0 v2:172.21.15.53:6833/1996992482 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x7f6834074160 con 0x55a024318960 2024-01-16T15:19:31.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.404+0000 7f6839600640 10 client.9201 handle_command_reply: tid=0 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.404+0000 7f6855de4740 1 client.9201 shutdown 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 msgr2=0x55a024290100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a024290100 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f683c009ac0 tx=0x7f683c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 shutdown_connections 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:6833/1996992482,v1:172.21.15.53:6834/1996992482] conn(0x55a024318960 0x55a024263310 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x55a024269070 0x55a024262dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x55a0242687b0 0x55a024290640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 --2- 172.21.15.53:0/1677366636 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x55a024263d70 0x55a024290100 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6839600640 0 client.9201 ms_handle_reset on v2:172.21.15.53:6833/1996992482 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 >> 172.21.15.53:0/1677366636 conn(0x55a02422ac70 msgr2=0x55a0242b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 shutdown_connections 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 1 -- 172.21.15.53:0/1677366636 wait complete. 2024-01-16T15:19:31.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.405+0000 7f6855de4740 20 client.9201 trim_cache size 0 max 16384 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "cluster_fsid": "c2138e65-ef75-44e8-8327-90d7ebaa10cd", 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "whoami": 0, 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 9054, 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "want_state": "up:active", 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "up:active", 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs", 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "rank_uptime": 6.7986509709999998, 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "mdsmap_epoch": 74, 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch": 110, 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "osdmap_epoch_barrier": 110, 2024-01-16T15:19:31.400 INFO:teuthology.orchestra.run.smithi053.stdout: "uptime": 12.170341649999999 2024-01-16T15:19:31.401 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:19:31.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f682406c430 msgr2=0x7f682406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f682406c430 0x7f682406e8f0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f683c036000 tx=0x7f683c02e000 comp rx=0 tx=0).stop 2024-01-16T15:19:31.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 msgr2=0x7f684c13bf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 0x7f684c13bf30 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f68340116d0 tx=0x7f6834011b90 comp rx=0 tx=0).stop 2024-01-16T15:19:31.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 shutdown_connections 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f682406c430 0x7f682406e8f0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f684c15cce0 0x7f684c13c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f684c15c860 0x7f684c13bf30 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 --2- 172.21.15.53:0/2841852524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f682c003970 0x7f684c1f3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 >> 172.21.15.53:0/2841852524 conn(0x7f684c06a800 msgr2=0x7f684c153dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 shutdown_connections 2024-01-16T15:19:31.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.406+0000 7f6851c00640 1 -- 172.21.15.53:0/2841852524 wait complete. 2024-01-16T15:19:31.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:19:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.642+0000 7fba09c00640 1 Processor -- start 2024-01-16T15:19:31.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.642+0000 7fba09c00640 1 -- start start 2024-01-16T15:19:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba09c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0413e360 0x7fba040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba09c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba0415f9c0 0x7fba040cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba09c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0415c5f0 con 0x7fba0413df80 2024-01-16T15:19:31.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba09c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0406c470 con 0x7fba0415f9c0 2024-01-16T15:19:31.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba09c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0406c750 con 0x7fba0413e360 2024-01-16T15:19:31.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba03400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba0415f9c0 0x7fba040cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba02a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fba0413df80 0x7fba0415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55764/0 (socket says 172.21.15.53:55764) 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba02a00640 1 -- 172.21.15.53:0/3927635229 learned_addr learned my addr 172.21.15.53:0/3927635229 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.644+0000 7fba02000640 1 --2- 172.21.15.53:0/3927635229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0413e360 0x7fba040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.645+0000 7fba03400640 1 -- 172.21.15.53:0/3927635229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0413e360 msgr2=0x7fba040cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.645+0000 7fba03400640 1 --2- 172.21.15.53:0/3927635229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0413e360 0x7fba040cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.645+0000 7fba03400640 1 -- 172.21.15.53:0/3927635229 >> v1:172.21.15.53:6789/0 conn(0x7fba0413df80 legacy=0x7fba0415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.645+0000 7fba01600640 1 -- 172.21.15.53:0/3927635229 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861709204 0 0) 0x7fba0415c5f0 con 0x7fba0413df80 2024-01-16T15:19:31.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.645+0000 7fba03400640 1 -- 172.21.15.53:0/3927635229 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0406ca30 con 0x7fba0415f9c0 2024-01-16T15:19:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.646+0000 7fba03400640 1 --2- 172.21.15.53:0/3927635229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0415f9c0 0x7fba040cf7d0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009d90 tx=0x7fb9fc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38deaf58647a65 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.646+0000 7fba01600640 1 -- 172.21.15.53:0/3927635229 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc02cc60 con 0x7fba0415f9c0 2024-01-16T15:19:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.646+0000 7fba01600640 1 -- 172.21.15.53:0/3927635229 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9fc02cdc0 con 0x7fba0415f9c0 2024-01-16T15:19:31.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.646+0000 7fba01600640 1 -- 172.21.15.53:0/3927635229 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9fc034660 con 0x7fba0415f9c0 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 -- 172.21.15.53:0/3927635229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0415f9c0 msgr2=0x7fba040cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 --2- 172.21.15.53:0/3927635229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0415f9c0 0x7fba040cf7d0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fb9fc009d90 tx=0x7fb9fc02beb0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 -- 172.21.15.53:0/3927635229 shutdown_connections 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 --2- 172.21.15.53:0/3927635229 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0415f9c0 0x7fba040cf7d0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 --2- 172.21.15.53:0/3927635229 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0413e360 0x7fba040cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 -- 172.21.15.53:0/3927635229 >> 172.21.15.53:0/3927635229 conn(0x7fba0406a800 msgr2=0x7fba0406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 -- 172.21.15.53:0/3927635229 shutdown_connections 2024-01-16T15:19:31.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.647+0000 7fba09c00640 1 -- 172.21.15.53:0/3927635229 wait complete. 2024-01-16T15:19:31.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.649+0000 7fba09c00640 1 Processor -- start 2024-01-16T15:19:31.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.649+0000 7fba09c00640 1 -- start start 2024-01-16T15:19:31.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba09c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 0x7fba040b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba09c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0413e360 0x7fba0413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba09c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0415f9c0 0x7fba041f64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba09c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0414e880 con 0x7fba0413df80 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba09c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0414eb60 con 0x7fba0413e360 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba09c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba04071de0 con 0x7fba0415f9c0 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 0x7fba040b3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 0x7fba040b3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:56044/0 (socket says 172.21.15.53:56044) 2024-01-16T15:19:31.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 -- 172.21.15.53:0/1128575807 learned_addr learned my addr 172.21.15.53:0/1128575807 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 -- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0415f9c0 msgr2=0x7fba041f64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0415f9c0 0x7fba041f64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 -- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0413e360 msgr2=0x7fba0413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0413e360 0x7fba0413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.650+0000 7fba02a00640 1 -- 172.21.15.53:0/1128575807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9fc009a70 con 0x7fba0413df80 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.651+0000 7fba02a00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 0x7fba040b3e70 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400c430 tx=0x7fb9f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.651+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f40074e0 con 0x7fba0413df80 2024-01-16T15:19:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.651+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f4007660 con 0x7fba0413df80 2024-01-16T15:19:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.651+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4004a20 con 0x7fba0413df80 2024-01-16T15:19:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.651+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba040720c0 con 0x7fba0413df80 2024-01-16T15:19:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.651+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0413dcf0 con 0x7fba0413df80 2024-01-16T15:19:31.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.653+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba040722e0 con 0x7fba0413df80 2024-01-16T15:19:31.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.654+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7fb9f4004b80 con 0x7fba0413df80 2024-01-16T15:19:31.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.654+0000 7fb9faa00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9dc06c460 0x7fb9dc06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:31.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.655+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7fb9f40889a0 con 0x7fba0413df80 2024-01-16T15:19:31.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.655+0000 7fba02000640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9dc06c460 0x7fb9dc06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:31.648 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.656+0000 7fba02000640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9dc06c460 0x7fb9dc06e920 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fba041f6430 tx=0x7fb9ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:31.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.659+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb9f4054ee0 con 0x7fba0413df80 2024-01-16T15:19:31.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.895+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fba0414ed20 con 0x7fba0413df80 2024-01-16T15:19:31.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.896+0000 7fb9faa00640 1 -- 172.21.15.53:0/1128575807 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb9f4058500 con 0x7fba0413df80 2024-01-16T15:19:31.888 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9dc06c460 msgr2=0x7fb9dc06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9dc06c460 0x7fb9dc06e920 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fba041f6430 tx=0x7fb9ec008040 comp rx=0 tx=0).stop 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 msgr2=0x7fba040b3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 0x7fba040b3e70 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400c430 tx=0x7fb9f400c8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 shutdown_connections 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb9dc06c460 0x7fb9dc06e920 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba0415f9c0 0x7fba041f64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba0413e360 0x7fba0413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 --2- 172.21.15.53:0/1128575807 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba0413df80 0x7fba040b3e70 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:31.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 >> 172.21.15.53:0/1128575807 conn(0x7fba0406a800 msgr2=0x7fba0415e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:31.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 shutdown_connections 2024-01-16T15:19:31.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:31.899+0000 7fba09c00640 1 -- 172.21.15.53:0/1128575807 wait complete. 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi053.front.sepia.ceph.com 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:19:31.912 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:19:31.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:31.913 DEBUG:teuthology.orchestra.run.smithi053:> ip addr 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 ::1/128 scope host 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:da brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f0 2024-01-16T15:19:31.967 INFO:teuthology.orchestra.run.smithi053.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 0c:c4:7a:6b:fd:db brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp7s0f1 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:18 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f0 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: inet 172.21.15.53/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft 41478sec preferred_lft 41478sec 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::ae1f:6bff:fef8:3718/64 scope link noprefixroute 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether ac:1f:6b:f8:37:19 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: altname enp1s0f1 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:19:31.968 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::7c57:68ff:fe2b:33e2/64 scope link 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8a:bd:4c:5c:d9:d4 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::88bd:4cff:fe5c:d9d4/64 scope link 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: link/ether 8e:8f:47:bd:96:49 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: inet6 fe80::8c8f:47ff:febd:9649/64 scope link 2024-01-16T15:19:31.969 INFO:teuthology.orchestra.run.smithi053.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:31.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:31.970 DEBUG:teuthology.orchestra.run.smithi053:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:19:32.066 INFO:teuthology.orchestra.run.smithi053.stdout:1 2024-01-16T15:19:32.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:32.071 DEBUG:teuthology.orchestra.run.smithi053:> route 2024-01-16T15:19:32.129 INFO:teuthology.orchestra.run.smithi053.stdout:Kernel IP routing table 2024-01-16T15:19:32.129 INFO:teuthology.orchestra.run.smithi053.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:19:32.129 INFO:teuthology.orchestra.run.smithi053.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-01-16T15:19:32.129 INFO:teuthology.orchestra.run.smithi053.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-01-16T15:19:32.129 INFO:teuthology.orchestra.run.smithi053.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:19:32.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:32.130 DEBUG:teuthology.orchestra.run.smithi053:> (cd / && exec bash -c ' 2024-01-16T15:19:32.130 DEBUG:teuthology.orchestra.run.smithi053:> set -e 2024-01-16T15:19:32.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-01-16T15:19:32.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-01-16T15:19:32.131 DEBUG:teuthology.orchestra.run.smithi053:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-01-16T15:19:32.131 DEBUG:teuthology.orchestra.run.smithi053:> ') 2024-01-16T15:19:32.327 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:32.327 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:19:32.378 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:19:32.378 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:19:32.379 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:19:32.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:32.380 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link set brx.0 up 2024-01-16T15:19:32.473 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-01-16T15:19:32.473 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:32.473 DEBUG:teuthology.orchestra.run.smithi053:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:32.530 INFO:teuthology.orchestra.run.smithi053.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-01-16T15:19:32.530 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:32.530 DEBUG:teuthology.orchestra.run.smithi053:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:32.583 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:19:32.681 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:32.738 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:19:32.738 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:19:32.738 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:19:32.738 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:19:32.738 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:32.739 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:32.740 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:32.741 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:32.742 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:19:32.797 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:19:32.797 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-01-16T15:19:32.802 DEBUG:teuthology.orchestra.run.smithi053:> sudo modprobe fuse 2024-01-16T15:19:32.882 DEBUG:teuthology.orchestra.run.smithi053:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/proc 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/sys 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/dev 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/security 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/shm 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/pts 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/run 2024-01-16T15:19:32.924 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/cgroup 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/pstore 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/ 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/selinux 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/mqueue 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/dev/hugepages 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/debug 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/tracing 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/sys/kernel/config 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/run/user/1000 2024-01-16T15:19:32.925 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-0 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-1 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-2 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/var/lib/ceph/osd/ceph-3 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:32.926 INFO:teuthology.orchestra.run.smithi053.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:19:32.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:32.927 DEBUG:teuthology.orchestra.run.smithi053:> ls /sys/fs/fuse/connections 2024-01-16T15:19:32.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:2024-01-16T15:19:33.000+0000 7f034b43d480 -1 init, newargv = 0x560a4a2f3650 newargc=15 2024-01-16T15:19:32.992 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[75897]: starting ceph client 2024-01-16T15:19:33.001 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[75897]: starting fuse 2024-01-16T15:19:33.015 INFO:teuthology.orchestra.run.smithi053.stdout:39 2024-01-16T15:19:33.015 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [39] 2024-01-16T15:19:33.015 DEBUG:teuthology.orchestra.run.smithi053:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:19:33.015 DEBUG:teuthology.orchestra.run.smithi053:> import glob 2024-01-16T15:19:33.015 DEBUG:teuthology.orchestra.run.smithi053:> import re 2024-01-16T15:19:33.015 DEBUG:teuthology.orchestra.run.smithi053:> import os 2024-01-16T15:19:33.015 DEBUG:teuthology.orchestra.run.smithi053:> import subprocess 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> def _find_admin_socket(client_name): 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> files = glob.glob(asok_path) 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> # Given a non-glob path, it better be there 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> if "*" not in asok_path: 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> assert(len(files) == 1) 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> return files[0] 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> for f in files: 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:19:33.016 DEBUG:teuthology.orchestra.run.smithi053:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> contents = proc_f.read() 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> if mountpoint in contents: 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> return f 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> print(_find_admin_socket("client.0")) 2024-01-16T15:19:33.017 DEBUG:teuthology.orchestra.run.smithi053:> ' 2024-01-16T15:19:33.222 INFO:teuthology.orchestra.run.smithi053.stdout:/var/run/ceph/ceph-client.0.75897.asok 2024-01-16T15:19:33.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:19:33.240 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.75897.asok 2024-01-16T15:19:33.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:33.240 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.75897.asok status 2024-01-16T15:19:33.488 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:19:33.488 INFO:teuthology.orchestra.run.smithi053.stdout: "metadata": { 2024-01-16T15:19:33.488 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:19:33.488 INFO:teuthology.orchestra.run.smithi053.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:19:33.488 INFO:teuthology.orchestra.run.smithi053.stdout: "entity_id": "0", 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "hostname": "smithi053", 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "pid": "75897", 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "root": "/" 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_count": 0, 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "dentry_pinned_count": 0, 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "id": 9219, 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "inst": { 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "name": { 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "client", 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "num": 9219 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:19:33.489 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 533294070 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": { 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "type": "v1", 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "nonce": 533294070 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "inst_str": "client.9219 192.168.0.1:0/533294070", 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "addr_str": "192.168.0.1:0/533294070", 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "inode_count": 1, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "mds_epoch": 74, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch": 110, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "blocklisted": false, 2024-01-16T15:19:33.490 INFO:teuthology.orchestra.run.smithi053.stdout: "fs_name": "cephfs" 2024-01-16T15:19:33.491 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:19:33.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:33.505 DEBUG:teuthology.orchestra.run.smithi053:> stat --file-system '--printf=%T 2024-01-16T15:19:33.505 DEBUG:teuthology.orchestra.run.smithi053:> ' -- /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:33.562 INFO:teuthology.orchestra.run.smithi053.stdout:fuseblk 2024-01-16T15:19:33.562 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:33.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:33.563 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-01-16T15:19:33.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-01-16T15:19:33.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.892+0000 7f4a3be00640 1 Processor -- start 2024-01-16T15:19:33.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.893+0000 7f4a3be00640 1 -- start start 2024-01-16T15:19:33.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.894+0000 7f4a3be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4a3c0d0100 0x7f4a3c0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.894+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 0x7f4a3c0d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.894+0000 7f4a3be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a3c0d0a10 con 0x7f4a3c160920 2024-01-16T15:19:33.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.894+0000 7f4a3be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a3c0cf080 con 0x7f4a3c0d0100 2024-01-16T15:19:33.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.894+0000 7f4a3be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0c6be0 con 0x7f4a3c0d0f40 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.895+0000 7f4a3aa00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4a3c160920 0x7f4a3c0cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39394/0 (socket says 172.21.15.53:39394) 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.895+0000 7f4a3aa00640 1 -- 172.21.15.53:0/4002447223 learned_addr learned my addr 172.21.15.53:0/4002447223 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.895+0000 7f4a3b400640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 0x7f4a3c0d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.895+0000 7f4a3a000640 1 --2- 172.21.15.53:0/4002447223 >> v2:172.21.15.146:3300/0 conn(0x7f4a3c0d0100 0x7f4a3c0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.895+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450540508 0 0) 0x7f4a3c0d0a10 con 0x7f4a3c160920 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.895+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a1c003520 con 0x7f4a3c160920 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360434856 0 0) 0x7f4a1c003520 con 0x7f4a3c160920 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 msgr2=0x7f4a3c0d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 0x7f4a3c0d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 >> v2:172.21.15.146:3300/0 conn(0x7f4a3c0d0100 msgr2=0x7f4a3c0b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 --2- 172.21.15.53:0/4002447223 >> v2:172.21.15.146:3300/0 conn(0x7f4a3c0d0100 0x7f4a3c0b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0ce7e0 con 0x7f4a3c160920 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4a2c001670 con 0x7f4a3c160920 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 >> v1:172.21.15.53:6789/0 conn(0x7f4a3c160920 legacy=0x7f4a3c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a1c003970 0x7f4a1c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 0x7f4a1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1c004d00 0x7f4a1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0cf080 con 0x7f4a1c003970 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0c6be0 con 0x7f4a1c004d00 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a1c00b980 con 0x7f4a3c0d0f40 2024-01-16T15:19:33.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.896+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a1c003970 0x7f4a1c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3a000640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 0x7f4a1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3b400640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1c004d00 0x7f4a1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3aa00640 1 -- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 msgr2=0x7f4a1c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a3c0d0f40 0x7f4a1c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3aa00640 1 -- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1c004d00 msgr2=0x7f4a1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a1c004d00 0x7f4a1c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.897+0000 7f4a3aa00640 1 -- 172.21.15.53:0/4002447223 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a1c003520 con 0x7f4a1c003970 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.898+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a1c003970 0x7f4a1c003d90 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c110 tx=0x7f4a1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97b4f684f477b89a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:33.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.898+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c005ad0 con 0x7f4a1c003970 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.898+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a2c005c50 con 0x7f4a1c003970 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.898+0000 7f4a39600640 1 -- 172.21.15.53:0/4002447223 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2c02ca70 con 0x7f4a1c003970 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3be00640 1 -- 172.21.15.53:0/4002447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a1c003970 msgr2=0x7f4a1c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3be00640 1 --2- 172.21.15.53:0/4002447223 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a1c003970 0x7f4a1c003d90 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c00c110 tx=0x7f4a1c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3b400640 1 -- 172.21.15.53:0/4002447223 reap_dead start 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3be00640 1 -- 172.21.15.53:0/4002447223 shutdown_connections 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3be00640 1 -- 172.21.15.53:0/4002447223 >> 172.21.15.53:0/4002447223 conn(0x7f4a3c0d3090 msgr2=0x7f4a3c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3be00640 1 -- 172.21.15.53:0/4002447223 shutdown_connections 2024-01-16T15:19:33.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.899+0000 7f4a3be00640 1 -- 172.21.15.53:0/4002447223 wait complete. 2024-01-16T15:19:33.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.901+0000 7f4a3be00640 1 Processor -- start 2024-01-16T15:19:33.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.901+0000 7f4a3be00640 1 -- start start 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.902+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a1c003970 0x7f4a3c1f38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.902+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3c0ca160 0x7f4a3c0b8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.902+0000 7f4a3be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 0x7f4a3c0cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.902+0000 7f4a3be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0cccf0 con 0x7f4a3c0cc300 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.902+0000 7f4a3be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0ca5f0 con 0x7f4a3c0ca160 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.902+0000 7f4a3be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3c0ca8d0 con 0x7f4a1c003970 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 0x7f4a3c0cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 0x7f4a3c0cc7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59246/0 (socket says 172.21.15.53:59246) 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 -- 172.21.15.53:0/412042985 learned_addr learned my addr 172.21.15.53:0/412042985 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:33.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a1c003970 0x7f4a3c1f38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3a000640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3c0ca160 0x7f4a3c0b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 -- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a1c003970 msgr2=0x7f4a3c1f38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a1c003970 0x7f4a3c1f38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 -- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3c0ca160 msgr2=0x7f4a3c0b8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3c0ca160 0x7f4a3c0b8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 -- 172.21.15.53:0/412042985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0cabb0 con 0x7f4a3c0cc300 2024-01-16T15:19:33.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.903+0000 7f4a3b400640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 0x7f4a3c0cc7b0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0d3060 tx=0x7f4a3400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:33.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.904+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a34017400 con 0x7f4a3c0cc300 2024-01-16T15:19:33.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.904+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a34011040 con 0x7f4a3c0cc300 2024-01-16T15:19:33.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.904+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a340177c0 con 0x7f4a3c0cc300 2024-01-16T15:19:33.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.904+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a3c0cae90 con 0x7f4a3c0cc300 2024-01-16T15:19:33.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.904+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3c1f3de0 con 0x7f4a3c0cc300 2024-01-16T15:19:33.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.905+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a3c067450 con 0x7f4a3c0cc300 2024-01-16T15:19:33.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.906+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4a34017920 con 0x7f4a3c0cc300 2024-01-16T15:19:33.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.907+0000 7f4a32a00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1406c430 0x7f4a1406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:33.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.907+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4a34088f80 con 0x7f4a3c0cc300 2024-01-16T15:19:33.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.908+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1406c430 0x7f4a1406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:33.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.909+0000 7f4a3aa00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1406c430 0x7f4a1406e8f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c034000 tx=0x7f4a2c007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:33.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:33.912+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4a340554f0 con 0x7f4a3c0cc300 2024-01-16T15:19:34.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.145+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4a3c08b1c0 con 0x7f4a3c0cc300 2024-01-16T15:19:34.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.145+0000 7f4a32a00640 1 -- 172.21.15.53:0/412042985 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4a3c08b1c0 con 0x7f4a3c0cc300 2024-01-16T15:19:34.137 INFO:teuthology.orchestra.run.smithi053.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1406c430 msgr2=0x7f4a1406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1406c430 0x7f4a1406e8f0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f4a2c034000 tx=0x7f4a2c007000 comp rx=0 tx=0).stop 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 msgr2=0x7f4a3c0cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 0x7f4a3c0cc7b0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0d3060 tx=0x7f4a3400f4a0 comp rx=0 tx=0).stop 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 shutdown_connections 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4a1406c430 0x7f4a1406e8f0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4a3c0cc300 0x7f4a3c0cc7b0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4a3c0ca160 0x7f4a3c0b8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:34.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 --2- 172.21.15.53:0/412042985 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4a1c003970 0x7f4a3c1f38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:34.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 >> 172.21.15.53:0/412042985 conn(0x7f4a3c0d3090 msgr2=0x7f4a3c0cdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:34.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 shutdown_connections 2024-01-16T15:19:34.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:34.148+0000 7f4a3be00640 1 -- 172.21.15.53:0/412042985 wait complete. 2024-01-16T15:19:34.160 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-01-16T15:19:34.160 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-01-16T15:19:34.160 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi146.front.sepia.ceph.com 2024-01-16T15:19:34.161 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-01-16T15:19:34.161 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:34.161 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-01-16T15:19:34.161 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-01-16T15:19:34.161 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-01-16T15:19:34.161 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:19:34.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.161 DEBUG:teuthology.orchestra.run.smithi146:> ip addr 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: inet 127.0.0.1/8 scope host lo 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 ::1/128 scope host 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6e brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f0 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:34.219 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:d9:42:6f brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: altname enp7s0f1 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fc brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 0c:c4:7a:88:76:fd brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: inet 172.21.15.146/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft 41356sec preferred_lft 41356sec 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::ec4:7aff:fe88:76fd/64 scope link 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-01-16T15:19:34.220 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::7885:33ff:feaa:4abd/64 scope link 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 06:27:6b:29:d2:fa brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::427:6bff:fe29:d2fa/64 scope link 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: link/ether 6a:97:5d:65:3f:d1 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: inet6 fe80::6897:5dff:fe65:3fd1/64 scope link 2024-01-16T15:19:34.221 INFO:teuthology.orchestra.run.smithi146.stdout: valid_lft forever preferred_lft forever 2024-01-16T15:19:34.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.222 DEBUG:teuthology.orchestra.run.smithi146:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-01-16T15:19:34.311 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-01-16T15:19:34.315 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.315 DEBUG:teuthology.orchestra.run.smithi146:> route 2024-01-16T15:19:34.366 INFO:teuthology.orchestra.run.smithi146.stdout:Kernel IP routing table 2024-01-16T15:19:34.366 INFO:teuthology.orchestra.run.smithi146.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-01-16T15:19:34.366 INFO:teuthology.orchestra.run.smithi146.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-01-16T15:19:34.366 INFO:teuthology.orchestra.run.smithi146.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-01-16T15:19:34.366 INFO:teuthology.orchestra.run.smithi146.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-01-16T15:19:34.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.367 DEBUG:teuthology.orchestra.run.smithi146:> (cd / && exec bash -c ' 2024-01-16T15:19:34.367 DEBUG:teuthology.orchestra.run.smithi146:> set -e 2024-01-16T15:19:34.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-01-16T15:19:34.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-01-16T15:19:34.367 DEBUG:teuthology.orchestra.run.smithi146:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-01-16T15:19:34.367 DEBUG:teuthology.orchestra.run.smithi146:> ') 2024-01-16T15:19:34.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.548 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:19:34.598 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:19:34.599 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:19:34.599 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.1' netns for '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:19:34.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.600 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link set brx.0 up 2024-01-16T15:19:34.689 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-01-16T15:19:34.690 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:34.690 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:34.739 INFO:teuthology.orchestra.run.smithi146.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-01-16T15:19:34.739 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-01-16T15:19:34.739 DEBUG:teuthology.orchestra.run.smithi146:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:34.787 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:19:34.878 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:34.935 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:34.936 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:19:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:19:34.938 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:19:34.938 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:34.938 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:34.938 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:34.938 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:34.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:34.939 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:19:34.987 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-01-16T15:19:34.987 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-01-16T15:19:34.991 DEBUG:teuthology.orchestra.run.smithi146:> sudo modprobe fuse 2024-01-16T15:19:35.066 DEBUG:teuthology.orchestra.run.smithi146:> cat /proc/self/mounts | awk '{print $2}' 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/proc 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/sys 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/dev 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/security 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/shm 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/pts 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/run 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/cgroup 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/pstore 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/ 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/selinux 2024-01-16T15:19:35.111 INFO:teuthology.orchestra.run.smithi146.stdout:/proc/sys/fs/binfmt_misc 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/mqueue 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/dev/hugepages 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/debug 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/tracing 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/fs/fuse/connections 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/sys/kernel/config 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-sysctl.service 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/nfs/rpc_pipefs 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/run/user/1000 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph 2024-01-16T15:19:35.112 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-4 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-5 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-6 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/var/lib/ceph/osd/ceph-7 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:35.113 INFO:teuthology.orchestra.run.smithi146.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:19:35.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:35.114 DEBUG:teuthology.orchestra.run.smithi146:> ls /sys/fs/fuse/connections 2024-01-16T15:19:35.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:2024-01-16T15:19:35.181+0000 7fb9da840480 -1 init, newargv = 0x557f23ee8b50 newargc=15 2024-01-16T15:19:35.172 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[47621]: starting ceph client 2024-01-16T15:19:35.183 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[47621]: starting fuse 2024-01-16T15:19:35.196 INFO:teuthology.orchestra.run.smithi146.stdout:41 2024-01-16T15:19:35.197 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> sudo stdin-killer -- python3 -c ' 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> import glob 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> import re 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> import os 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> import subprocess 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> def _find_admin_socket(client_name): 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> files = glob.glob(asok_path) 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:19:35.197 DEBUG:teuthology.orchestra.run.smithi146:> # Given a non-glob path, it better be there 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> if "*" not in asok_path: 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> assert(len(files) == 1) 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> return files[0] 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> for f in files: 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> if os.path.exists("/proc/{0}".format(pid)): 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> contents = proc_f.read() 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> if mountpoint in contents: 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> return f 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> print(_find_admin_socket("client.1")) 2024-01-16T15:19:35.198 DEBUG:teuthology.orchestra.run.smithi146:> ' 2024-01-16T15:19:35.398 INFO:teuthology.orchestra.run.smithi146.stdout:/var/run/ceph/ceph-client.1.47621.asok 2024-01-16T15:19:35.403 INFO:teuthology.orchestra.run.smithi146.stderr:2024-01-16T15:19:35 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-01-16T15:19:35.416 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.47621.asok 2024-01-16T15:19:35.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:35.416 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.47621.asok status 2024-01-16T15:19:35.651 INFO:teuthology.orchestra.run.smithi146.stdout:{ 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "metadata": { 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_sha1": "73a4532fc4d97adee0b985d4d42874ea351a1c86", 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "ceph_version": "ceph version 18.0.0-7629-g73a4532f (73a4532fc4d97adee0b985d4d42874ea351a1c86) reef (dev)", 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "entity_id": "1", 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "hostname": "smithi146", 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "pid": "47621", 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "root": "/" 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_count": 0, 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "dentry_pinned_count": 0, 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "id": 6677, 2024-01-16T15:19:35.652 INFO:teuthology.orchestra.run.smithi146.stdout: "inst": { 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "name": { 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "client", 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "num": 6677 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2391631557 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: } 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": { 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "type": "v1", 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "addr": "192.168.0.1:0", 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "nonce": 2391631557 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: }, 2024-01-16T15:19:35.653 INFO:teuthology.orchestra.run.smithi146.stdout: "inst_str": "client.6677 192.168.0.1:0/2391631557", 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "addr_str": "192.168.0.1:0/2391631557", 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "inode_count": 1, 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "mds_epoch": 74, 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch": 110, 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "osd_epoch_barrier": 0, 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "blocklisted": false, 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout: "fs_name": "cephfs" 2024-01-16T15:19:35.654 INFO:teuthology.orchestra.run.smithi146.stdout:} 2024-01-16T15:19:35.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:35.666 DEBUG:teuthology.orchestra.run.smithi146:> stat --file-system '--printf=%T 2024-01-16T15:19:35.666 DEBUG:teuthology.orchestra.run.smithi146:> ' -- /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:35.716 INFO:teuthology.orchestra.run.smithi146.stdout:fuseblk 2024-01-16T15:19:35.716 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:35.716 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:35.716 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-01-16T15:19:35.856 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2024-01-16T15:19:36.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.097+0000 7f0253e00640 1 Processor -- start 2024-01-16T15:19:36.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.097+0000 7f0253e00640 1 -- start start 2024-01-16T15:19:36.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.099+0000 7f0253e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 0x7f02540d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.099+0000 7f0253e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02540b2c70 con 0x7f02540b4110 2024-01-16T15:19:36.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.099+0000 7f0253e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02540cf6d0 con 0x7f02540cfa40 2024-01-16T15:19:36.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.099+0000 7f0253e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02540cf3f0 con 0x7f025415d3c0 2024-01-16T15:19:36.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.100+0000 7f0253400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f02540b4110 0x7f02540ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39408/0 (socket says 172.21.15.53:39408) 2024-01-16T15:19:36.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.100+0000 7f0253400640 1 -- 172.21.15.53:0/2971444291 learned_addr learned my addr 172.21.15.53:0/2971444291 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:36.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.100+0000 7f0252a00640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 0x7f02540d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.100+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 275931596 0 0) 0x7f02540b2c70 con 0x7f02540b4110 2024-01-16T15:19:36.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.100+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0234003520 con 0x7f02540b4110 2024-01-16T15:19:36.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3238506038 0 0) 0x7f0234003520 con 0x7f02540b4110 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 msgr2=0x7f02540d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 0x7f02540d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 >> v1:172.21.15.146:6789/0 conn(0x7f02540cfa40 legacy=0x7f02540cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02540b2620 con 0x7f02540b4110 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f024c0019f0 con 0x7f02540b4110 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 >> v1:172.21.15.53:6789/0 conn(0x7f02540b4110 legacy=0x7f02540ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 0x7f02340039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234003f70 0x7f02340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0234004d00 0x7f0234005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02540cf3f0 con 0x7f0234004d00 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023400bac0 con 0x7f0234003f70 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f023400bc80 con 0x7f025415d3c0 2024-01-16T15:19:36.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.101+0000 7f0253400640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0234004d00 0x7f0234005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.102+0000 7f0253400640 1 -- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 msgr2=0x7f02340039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:36.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.102+0000 7f0253400640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f025415d3c0 0x7f02340039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.102+0000 7f0253400640 1 -- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234003f70 msgr2=0x7f02340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:36.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.102+0000 7f0253400640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234003f70 0x7f02340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.102+0000 7f0253400640 1 -- 172.21.15.53:0/2971444291 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0234003520 con 0x7f0234004d00 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.103+0000 7f0253400640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0234004d00 0x7f0234005180 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f024c00a750 tx=0x7f02340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdadc856c1c4447f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.103+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c034630 con 0x7f0234004d00 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.103+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c02f070 con 0x7f0234004d00 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.103+0000 7f0252000640 1 -- 172.21.15.53:0/2971444291 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c034a30 con 0x7f0234004d00 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253e00640 1 -- 172.21.15.53:0/2971444291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0234004d00 msgr2=0x7f0234005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253e00640 1 --2- 172.21.15.53:0/2971444291 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0234004d00 0x7f0234005180 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f024c00a750 tx=0x7f02340057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253400640 1 -- 172.21.15.53:0/2971444291 reap_dead start 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253e00640 1 -- 172.21.15.53:0/2971444291 shutdown_connections 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253e00640 1 -- 172.21.15.53:0/2971444291 >> 172.21.15.53:0/2971444291 conn(0x7f02540d3050 msgr2=0x7f02540d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:36.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253e00640 1 -- 172.21.15.53:0/2971444291 shutdown_connections 2024-01-16T15:19:36.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.104+0000 7f0253e00640 1 -- 172.21.15.53:0/2971444291 wait complete. 2024-01-16T15:19:36.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.106+0000 7f0253e00640 1 Processor -- start 2024-01-16T15:19:36.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.106+0000 7f0253e00640 1 -- start start 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234004d00 0x7f02540b7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02540b5e10 0x7f02540cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 0x7f02540b0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0254157810 con 0x7f02540cbf80 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02540c6d20 con 0x7f0234004d00 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02540c7000 con 0x7f02540b5e10 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 0x7f02540b0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 0x7f02540b0b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59270/0 (socket says 172.21.15.53:59270) 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.107+0000 7f0253400640 1 -- 172.21.15.53:0/1757464557 learned_addr learned my addr 172.21.15.53:0/1757464557 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0252a00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234004d00 0x7f02540b7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f024be00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02540b5e10 0x7f02540cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253400640 1 -- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02540b5e10 msgr2=0x7f02540cba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253400640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02540b5e10 0x7f02540cba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253400640 1 -- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234004d00 msgr2=0x7f02540b7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253400640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234004d00 0x7f02540b7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253400640 1 -- 172.21.15.53:0/1757464557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02540c72e0 con 0x7f02540cbf80 2024-01-16T15:19:36.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253400640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 0x7f02540b0b40 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f024c00a750 tx=0x7f025408b0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c034630 con 0x7f02540cbf80 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f024c034790 con 0x7f02540cbf80 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f024c034b00 con 0x7f02540cbf80 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.108+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02540c75c0 con 0x7f02540cbf80 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.109+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02540b8190 con 0x7f02540cbf80 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.110+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02540c7ae0 con 0x7f02540cbf80 2024-01-16T15:19:36.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.111+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f024c031030 con 0x7f02540cbf80 2024-01-16T15:19:36.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.112+0000 7f024aa00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f022c06c460 0x7f022c06e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.112+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f024c0aad70 con 0x7f02540cbf80 2024-01-16T15:19:36.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.112+0000 7f0252a00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f022c06c460 0x7f022c06e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.114+0000 7f0252a00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f022c06c460 0x7f022c06e920 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0244004770 tx=0x7f0244008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:36.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.116+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f02540c7ae0 con 0x7f02540cbf80 2024-01-16T15:19:36.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.347+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f025408ba80 con 0x7f02540cbf80 2024-01-16T15:19:36.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.707+0000 7f024aa00640 1 -- 172.21.15.53:0/1757464557 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v75) v1 ==== 136+0+0 (secure 0 0 0) 0x7f024c077340 con 0x7f02540cbf80 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.711+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f022c06c460 msgr2=0x7f022c06e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.711+0000 7f0253e00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f022c06c460 0x7f022c06e920 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f0244004770 tx=0x7f0244008040 comp rx=0 tx=0).stop 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.711+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 msgr2=0x7f02540b0b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.711+0000 7f0253e00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 0x7f02540b0b40 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f024c00a750 tx=0x7f025408b0a0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 shutdown_connections 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f022c06c460 0x7f022c06e920 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f02540cbf80 0x7f02540b0b40 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f02540b5e10 0x7f02540cba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 --2- 172.21.15.53:0/1757464557 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0234004d00 0x7f02540b7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 >> 172.21.15.53:0/1757464557 conn(0x7f02540d3050 msgr2=0x7f02540d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:36.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 shutdown_connections 2024-01-16T15:19:36.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.712+0000 7f0253e00640 1 -- 172.21.15.53:0/1757464557 wait complete. 2024-01-16T15:19:36.724 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:19:36.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.946+0000 7f8e6c000640 1 Processor -- start 2024-01-16T15:19:36.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.947+0000 7f8e6c000640 1 -- start start 2024-01-16T15:19:36.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.948+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e6415d430 0x7f8e640afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.948+0000 7f8e6c000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e640d0e80 0x7f8e640d3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.948+0000 7f8e6c000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e640d3b30 con 0x7f8e640d36a0 2024-01-16T15:19:36.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.948+0000 7f8e6c000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e640b0730 con 0x7f8e640d0e80 2024-01-16T15:19:36.940 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.948+0000 7f8e6c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e640b0a10 con 0x7f8e6415d430 2024-01-16T15:19:36.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e68c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e640d0e80 0x7f8e640d3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e68c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8e640d0e80 0x7f8e640d3160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59278/0 (socket says 172.21.15.53:59278) 2024-01-16T15:19:36.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e68c00640 1 -- 172.21.15.53:0/3378266102 learned_addr learned my addr 172.21.15.53:0/3378266102 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:36.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e69600640 1 --2- 172.21.15.53:0/3378266102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e6415d430 0x7f8e640afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e68c00640 1 -- 172.21.15.53:0/3378266102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e6415d430 msgr2=0x7f8e640afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:36.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e68c00640 1 --2- 172.21.15.53:0/3378266102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e6415d430 0x7f8e640afee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.949+0000 7f8e68c00640 1 -- 172.21.15.53:0/3378266102 >> v1:172.21.15.146:6789/0 conn(0x7f8e640d36a0 legacy=0x7f8e640b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.950+0000 7f8e68c00640 1 -- 172.21.15.53:0/3378266102 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e640b0cf0 con 0x7f8e640d0e80 2024-01-16T15:19:36.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.951+0000 7f8e68c00640 1 --2- 172.21.15.53:0/3378266102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e640d0e80 0x7f8e640d3160 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009870 tx=0x7f8e4c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ce7b3974bfbedf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.951+0000 7f8e63e00640 1 -- 172.21.15.53:0/3378266102 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c030840 con 0x7f8e640d0e80 2024-01-16T15:19:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.951+0000 7f8e63e00640 1 -- 172.21.15.53:0/3378266102 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c0309a0 con 0x7f8e640d0e80 2024-01-16T15:19:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.951+0000 7f8e63e00640 1 -- 172.21.15.53:0/3378266102 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c030d30 con 0x7f8e640d0e80 2024-01-16T15:19:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.951+0000 7f8e6c000640 1 -- 172.21.15.53:0/3378266102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e640d0e80 msgr2=0x7f8e640d3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.951+0000 7f8e6c000640 1 --2- 172.21.15.53:0/3378266102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e640d0e80 0x7f8e640d3160 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009870 tx=0x7f8e4c02b980 comp rx=0 tx=0).stop 2024-01-16T15:19:36.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.952+0000 7f8e6c000640 1 -- 172.21.15.53:0/3378266102 shutdown_connections 2024-01-16T15:19:36.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.952+0000 7f8e6c000640 1 --2- 172.21.15.53:0/3378266102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e640d0e80 0x7f8e640d3160 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.952+0000 7f8e6c000640 1 --2- 172.21.15.53:0/3378266102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e6415d430 0x7f8e640afee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.952+0000 7f8e6c000640 1 -- 172.21.15.53:0/3378266102 >> 172.21.15.53:0/3378266102 conn(0x7f8e6408ccd0 msgr2=0x7f8e6409c5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:36.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.952+0000 7f8e6c000640 1 -- 172.21.15.53:0/3378266102 shutdown_connections 2024-01-16T15:19:36.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.952+0000 7f8e6c000640 1 -- 172.21.15.53:0/3378266102 wait complete. 2024-01-16T15:19:36.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.953+0000 7f8e6c000640 1 Processor -- start 2024-01-16T15:19:36.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.954+0000 7f8e6c000640 1 -- start start 2024-01-16T15:19:36.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e640d0e80 0x7f8e640ca380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e640d36a0 0x7f8e6414f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 0x7f8e64150de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e6414fd40 con 0x7f8e6415d430 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e64150020 con 0x7f8e640d36a0 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e64150300 con 0x7f8e640d0e80 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 0x7f8e64150de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 0x7f8e64150de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59294/0 (socket says 172.21.15.53:59294) 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6a000640 1 -- 172.21.15.53:0/1946083543 learned_addr learned my addr 172.21.15.53:0/1946083543 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e69600640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e640d0e80 0x7f8e640ca380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e68c00640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e640d36a0 0x7f8e6414f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6a000640 1 -- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e640d0e80 msgr2=0x7f8e640ca380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.955+0000 7f8e6a000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e640d0e80 0x7f8e640ca380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e6a000640 1 -- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e640d36a0 msgr2=0x7f8e6414f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e6a000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e640d36a0 0x7f8e6414f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:36.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e6a000640 1 -- 172.21.15.53:0/1946083543 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0094d0 con 0x7f8e6415d430 2024-01-16T15:19:36.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e6a000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 0x7f8e64150de0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00c7a0 tx=0x7f8e5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:36.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c003b70 con 0x7f8e6415d430 2024-01-16T15:19:36.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e5c003cf0 con 0x7f8e6415d430 2024-01-16T15:19:36.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c003ea0 con 0x7f8e6415d430 2024-01-16T15:19:36.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e641505e0 con 0x7f8e6415d430 2024-01-16T15:19:36.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.956+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e640ca8c0 con 0x7f8e6415d430 2024-01-16T15:19:36.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.958+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e64150b00 con 0x7f8e6415d430 2024-01-16T15:19:36.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.959+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f8e5c00aa90 con 0x7f8e6415d430 2024-01-16T15:19:36.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.959+0000 7f8e61600640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e3c06c430 0x7f8e3c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:36.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.960+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f8e5c0889b0 con 0x7f8e6415d430 2024-01-16T15:19:36.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.960+0000 7f8e69600640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e3c06c430 0x7f8e3c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:36.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.961+0000 7f8e69600640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e3c06c430 0x7f8e3c06e8f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8e54006fd0 tx=0x7f8e54008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:36.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:36.964+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8e64150b00 con 0x7f8e6415d430 2024-01-16T15:19:37.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.182+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e640b0420 con 0x7f8e3c06c430 2024-01-16T15:19:37.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.187+0000 7f8e61600640 1 -- 172.21.15.53:0/1946083543 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f8e5400f860 con 0x7f8e3c06c430 2024-01-16T15:19:37.178 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:19:37.178 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:19:37.178 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:19:37.179 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:19:37.179 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e3c06c430 msgr2=0x7f8e3c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e3c06c430 0x7f8e3c06e8f0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f8e54006fd0 tx=0x7f8e54008040 comp rx=0 tx=0).stop 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 msgr2=0x7f8e64150de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 0x7f8e64150de0 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00c7a0 tx=0x7f8e5c00cc60 comp rx=0 tx=0).stop 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 shutdown_connections 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8e3c06c430 0x7f8e3c06e8f0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8e6415d430 0x7f8e64150de0 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8e640d36a0 0x7f8e6414f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 --2- 172.21.15.53:0/1946083543 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8e640d0e80 0x7f8e640ca380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 >> 172.21.15.53:0/1946083543 conn(0x7f8e6408ccd0 msgr2=0x7f8e640b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:37.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 shutdown_connections 2024-01-16T15:19:37.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.189+0000 7f8e6c000640 1 -- 172.21.15.53:0/1946083543 wait complete. 2024-01-16T15:19:37.201 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-01-16T15:19:37.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.434+0000 7f4dd7a00640 1 Processor -- start 2024-01-16T15:19:37.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.434+0000 7f4dd7a00640 1 -- start start 2024-01-16T15:19:37.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.435+0000 7f4dd7a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0157c30 0x7f4dd00cd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.435+0000 7f4dd7a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4dd0071200 0x7f4dd006dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.435+0000 7f4dd7a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd015deb0 con 0x7f4dd01618f0 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.435+0000 7f4dd7a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd015ea30 con 0x7f4dd0071200 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.435+0000 7f4dd7a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd006d860 con 0x7f4dd0157c30 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4dd0071200 0x7f4dd006dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4dd0071200 0x7f4dd006dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59300/0 (socket says 172.21.15.53:59300) 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 -- 172.21.15.53:0/2755193828 learned_addr learned my addr 172.21.15.53:0/2755193828 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dc7e00640 1 --1- 172.21.15.53:0/2755193828 >> v1:172.21.15.53:6789/0 conn(0x7f4dd01618f0 0x7f4dd0070af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39414/0 (socket says 172.21.15.53:39414) 2024-01-16T15:19:37.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5000640 1 --2- 172.21.15.53:0/2755193828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0157c30 0x7f4dd00cd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 -- 172.21.15.53:0/2755193828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0157c30 msgr2=0x7f4dd00cd600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:37.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 --2- 172.21.15.53:0/2755193828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0157c30 0x7f4dd00cd600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 -- 172.21.15.53:0/2755193828 >> v1:172.21.15.53:6789/0 conn(0x7f4dd01618f0 legacy=0x7f4dd0070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.436+0000 7f4dd5a00640 1 -- 172.21.15.53:0/2755193828 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd015e190 con 0x7f4dd0071200 2024-01-16T15:19:37.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.437+0000 7f4dd5a00640 1 --2- 172.21.15.53:0/2755193828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0071200 0x7f4dd006dcb0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009e50 tx=0x7f4dc802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f2dc82950d4c05 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:37.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.438+0000 7f4dcfe00640 1 -- 172.21.15.53:0/2755193828 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc802cb00 con 0x7f4dd0071200 2024-01-16T15:19:37.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.438+0000 7f4dcfe00640 1 -- 172.21.15.53:0/2755193828 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc802cc60 con 0x7f4dd0071200 2024-01-16T15:19:37.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.438+0000 7f4dcfe00640 1 -- 172.21.15.53:0/2755193828 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80343f0 con 0x7f4dd0071200 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 -- 172.21.15.53:0/2755193828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0071200 msgr2=0x7f4dd006dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/2755193828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0071200 0x7f4dd006dcb0 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f4dc8009e50 tx=0x7f4dc802bf60 comp rx=0 tx=0).stop 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 -- 172.21.15.53:0/2755193828 shutdown_connections 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/2755193828 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0071200 0x7f4dd006dcb0 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/2755193828 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0157c30 0x7f4dd00cd600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 -- 172.21.15.53:0/2755193828 >> 172.21.15.53:0/2755193828 conn(0x7f4dd00cf700 msgr2=0x7f4dd00cfae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 -- 172.21.15.53:0/2755193828 shutdown_connections 2024-01-16T15:19:37.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.439+0000 7f4dd7a00640 1 -- 172.21.15.53:0/2755193828 wait complete. 2024-01-16T15:19:37.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.441+0000 7f4dd7a00640 1 Processor -- start 2024-01-16T15:19:37.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.441+0000 7f4dd7a00640 1 -- start start 2024-01-16T15:19:37.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dd7a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0071200 0x7f4dd00b43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dd7a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 0x7f4dd00b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dd7a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd01618f0 0x7f4dd01f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dd7a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd01e8d40 con 0x7f4dd0157c30 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dd7a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd01e91d0 con 0x7f4dd01618f0 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dd7a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00b4900 con 0x7f4dd0071200 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dc7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 0x7f4dd00b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dc7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 0x7f4dd00b37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59304/0 (socket says 172.21.15.53:59304) 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.442+0000 7f4dc7e00640 1 -- 172.21.15.53:0/3970463177 learned_addr learned my addr 172.21.15.53:0/3970463177 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dd5000640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0071200 0x7f4dd00b43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dd5a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd01618f0 0x7f4dd01f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dc7e00640 1 -- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0071200 msgr2=0x7f4dd00b43c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dc7e00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0071200 0x7f4dd00b43c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dc7e00640 1 -- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd01618f0 msgr2=0x7f4dd01f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dc7e00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd01618f0 0x7f4dd01f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dc7e00640 1 -- 172.21.15.53:0/3970463177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc8009b30 con 0x7f4dd0157c30 2024-01-16T15:19:37.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dd5a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd01618f0 0x7f4dd01f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:37.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dc7e00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 0x7f4dd00b37a0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4db800c450 tx=0x7f4db800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8007480 con 0x7f4dd0157c30 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4db8007600 con 0x7f4dd0157c30 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4db8004a20 con 0x7f4dd0157c30 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.443+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd01e8130 con 0x7f4dd0157c30 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.444+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd01e86f0 con 0x7f4dd0157c30 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.445+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd01e8eb0 con 0x7f4dd0157c30 2024-01-16T15:19:37.437 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.446+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f4db8004b80 con 0x7f4dd0157c30 2024-01-16T15:19:37.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.446+0000 7f4dcd600640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4da806c430 0x7f4da806e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.438 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.447+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f4db8087b80 con 0x7f4dd0157c30 2024-01-16T15:19:37.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.447+0000 7f4dd5000640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4da806c430 0x7f4da806e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.448+0000 7f4dd5000640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4da806c430 0x7f4da806e8f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0006fd0 tx=0x7f4dc0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:37.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.451+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4db80540f0 con 0x7f4dd0157c30 2024-01-16T15:19:37.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.646+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f4dd00675d0 con 0x7f4dd0157c30 2024-01-16T15:19:37.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.647+0000 7f4dcd600640 1 -- 172.21.15.53:0/3970463177 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v42)=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f4db80180c0 con 0x7f4dd0157c30 2024-01-16T15:19:37.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4da806c430 msgr2=0x7f4da806e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4da806c430 0x7f4da806e8f0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0006fd0 tx=0x7f4dc0008040 comp rx=0 tx=0).stop 2024-01-16T15:19:37.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 msgr2=0x7f4dd00b37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 0x7f4dd00b37a0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f4db800c450 tx=0x7f4db800c910 comp rx=0 tx=0).stop 2024-01-16T15:19:37.642 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 shutdown_connections 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4da806c430 0x7f4da806e8f0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4dd01618f0 0x7f4dd01f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4dd0157c30 0x7f4dd00b37a0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 --2- 172.21.15.53:0/3970463177 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4dd0071200 0x7f4dd00b43c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 >> 172.21.15.53:0/3970463177 conn(0x7f4dd00cf700 msgr2=0x7f4dd00ae8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 shutdown_connections 2024-01-16T15:19:37.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.650+0000 7f4dd7a00640 1 -- 172.21.15.53:0/3970463177 wait complete. 2024-01-16T15:19:37.662 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/snapshot_clone_delay 30 2024-01-16T15:19:37.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.889+0000 7f00fd800640 1 Processor -- start 2024-01-16T15:19:37.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.889+0000 7f00fd800640 1 -- start start 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00fd800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00f813df80 0x7f00f815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00fd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00fd800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00f806c470 con 0x7f00f815f9c0 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00fd800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00f806c750 con 0x7f00f813df80 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00fd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f806ca30 con 0x7f00f813e360 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00f813df80 0x7f00f815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00f813df80 0x7f00f815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59308/0 (socket says 172.21.15.53:59308) 2024-01-16T15:19:37.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 -- 172.21.15.53:0/4277641917 learned_addr learned my addr 172.21.15.53:0/4277641917 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:37.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 -- 172.21.15.53:0/4277641917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 msgr2=0x7f00f80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:37.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 --2- 172.21.15.53:0/4277641917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 -- 172.21.15.53:0/4277641917 >> v1:172.21.15.146:6789/0 conn(0x7f00f815f9c0 legacy=0x7f00f81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:37.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.891+0000 7f00f6a00640 1 -- 172.21.15.53:0/4277641917 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f806cd10 con 0x7f00f813df80 2024-01-16T15:19:37.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.893+0000 7f00f6a00640 1 --2- 172.21.15.53:0/4277641917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f815bee0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758d86ab66d470e6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:37.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.893+0000 7f00f5600640 1 -- 172.21.15.53:0/4277641917 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8031640 con 0x7f00f813df80 2024-01-16T15:19:37.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.893+0000 7f00f5600640 1 -- 172.21.15.53:0/4277641917 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e80317a0 con 0x7f00f813df80 2024-01-16T15:19:37.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.893+0000 7f00f5600640 1 -- 172.21.15.53:0/4277641917 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8031b30 con 0x7f00f813df80 2024-01-16T15:19:37.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 -- 172.21.15.53:0/4277641917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 msgr2=0x7f00f815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 --2- 172.21.15.53:0/4277641917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f815bee0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 -- 172.21.15.53:0/4277641917 shutdown_connections 2024-01-16T15:19:37.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 --2- 172.21.15.53:0/4277641917 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f80cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 --2- 172.21.15.53:0/4277641917 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f815bee0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 -- 172.21.15.53:0/4277641917 >> 172.21.15.53:0/4277641917 conn(0x7f00f806a800 msgr2=0x7f00f806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:37.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 -- 172.21.15.53:0/4277641917 shutdown_connections 2024-01-16T15:19:37.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.894+0000 7f00fd800640 1 -- 172.21.15.53:0/4277641917 wait complete. 2024-01-16T15:19:37.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 Processor -- start 2024-01-16T15:19:37.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 -- start start 2024-01-16T15:19:37.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f813a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f8071de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00f815f9c0 0x7f00f8070560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f813a7f0 con 0x7f00f813df80 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8071950 con 0x7f00f815f9c0 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.898+0000 7f00fd800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00f8072320 con 0x7f00f813e360 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f813a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f813a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59310/0 (socket says 172.21.15.53:59310) 2024-01-16T15:19:37.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 -- 172.21.15.53:0/4095988538 learned_addr learned my addr 172.21.15.53:0/4095988538 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6000640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f8071de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f7400640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00f815f9c0 0x7f00f8070560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 -- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 msgr2=0x7f00f8071de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f8071de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 -- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00f815f9c0 msgr2=0x7f00f8070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.899+0000 7f00f6a00640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00f815f9c0 0x7f00f8070560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00f6a00640 1 -- 172.21.15.53:0/4095988538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00e80094d0 con 0x7f00f813df80 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00f7400640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00f815f9c0 0x7f00f8070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00f6a00640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f813a2b0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e803ac60 con 0x7f00f813df80 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00e803ade0 con 0x7f00f813df80 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00e8044630 con 0x7f00f813df80 2024-01-16T15:19:37.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00f8070aa0 con 0x7f00f813df80 2024-01-16T15:19:37.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.900+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00f8071020 con 0x7f00f813df80 2024-01-16T15:19:37.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.903+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00f80675d0 con 0x7f00f813df80 2024-01-16T15:19:37.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.903+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f00e802f050 con 0x7f00f813df80 2024-01-16T15:19:37.895 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.904+0000 7f00eea00640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d006c430 0x7f00d006e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:37.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.904+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f00e80aabe0 con 0x7f00f813df80 2024-01-16T15:19:37.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.906+0000 7f00f6000640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d006c430 0x7f00d006e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:37.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.907+0000 7f00f6000640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d006c430 0x7f00d006e8f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f00f80704f0 tx=0x7f00e0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:37.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:37.909+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00e8077150 con 0x7f00f813df80 2024-01-16T15:19:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.103+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}] v 0) v1 -- 0x7f00f806df00 con 0x7f00f813df80 2024-01-16T15:19:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.109+0000 7f00eea00640 1 -- 172.21.15.53:0/4095988538 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/snapshot_clone_delay}]=0 v43)=0 v43) v1 ==== 131+0+0 (secure 0 0 0) 0x7f00e807a770 con 0x7f00f813df80 2024-01-16T15:19:38.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d006c430 msgr2=0x7f00d006e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d006c430 0x7f00d006e8f0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f00f80704f0 tx=0x7f00e0008040 comp rx=0 tx=0).stop 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 msgr2=0x7f00f813a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f813a2b0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009870 tx=0x7f00e80046c0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 shutdown_connections 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f00d006c430 0x7f00d006e8f0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00f815f9c0 0x7f00f8070560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00f813e360 0x7f00f8071de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 --2- 172.21.15.53:0/4095988538 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00f813df80 0x7f00f813a2b0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 >> 172.21.15.53:0/4095988538 conn(0x7f00f806a800 msgr2=0x7f00f8154e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.112+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 shutdown_connections 2024-01-16T15:19:38.105 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.113+0000 7f00fd800640 1 -- 172.21.15.53:0/4095988538 wait complete. 2024-01-16T15:19:38.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/volumes/clone_delay_every_regfile 120 2024-01-16T15:19:38.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.359+0000 7f42b2a00640 1 Processor -- start 2024-01-16T15:19:38.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.359+0000 7f42b2a00640 1 -- start start 2024-01-16T15:19:38.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b2a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42ac0cf9a0 0x7f42ac0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b2a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ac0d0320 con 0x7f42ac15d4b0 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b2a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42ac0d1c60 con 0x7f42ac0cf9a0 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b2a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac0b2730 con 0x7f42ac0b31d0 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42ac0cf9a0 0x7f42ac0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b2000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f42ac0cf9a0 0x7f42ac0cfde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:59944/0 (socket says 172.21.15.53:59944) 2024-01-16T15:19:38.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.360+0000 7f42b0c00640 1 -- 172.21.15.53:0/1398624243 learned_addr learned my addr 172.21.15.53:0/1398624243 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:38.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.361+0000 7f42b0c00640 1 -- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 msgr2=0x7f42ac0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:38.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.361+0000 7f42b0c00640 1 --2- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.361+0000 7f42b0c00640 1 -- 172.21.15.53:0/1398624243 >> v1:172.21.15.146:6789/0 conn(0x7f42ac15d4b0 legacy=0x7f42ac0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.361+0000 7f42b0c00640 1 -- 172.21.15.53:0/1398624243 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ac0d1ec0 con 0x7f42ac0cf9a0 2024-01-16T15:19:38.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.364+0000 7f42b0c00640 1 --2- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cfde0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef536781dc173934 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:38.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.364+0000 7f42abe00640 1 -- 172.21.15.53:0/1398624243 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294030840 con 0x7f42ac0cf9a0 2024-01-16T15:19:38.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.364+0000 7f42abe00640 1 -- 172.21.15.53:0/1398624243 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42940309a0 con 0x7f42ac0cf9a0 2024-01-16T15:19:38.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.364+0000 7f42abe00640 1 -- 172.21.15.53:0/1398624243 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4294030d30 con 0x7f42ac0cf9a0 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 -- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 msgr2=0x7f42ac0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 --2- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cfde0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4294009870 tx=0x7f429402b980 comp rx=0 tx=0).stop 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 -- 172.21.15.53:0/1398624243 shutdown_connections 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 --2- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 --2- 172.21.15.53:0/1398624243 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cfde0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 -- 172.21.15.53:0/1398624243 >> 172.21.15.53:0/1398624243 conn(0x7f42ac0d3000 msgr2=0x7f42ac0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 -- 172.21.15.53:0/1398624243 shutdown_connections 2024-01-16T15:19:38.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.365+0000 7f42b2a00640 1 -- 172.21.15.53:0/1398624243 wait complete. 2024-01-16T15:19:38.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.367+0000 7f42b2a00640 1 Processor -- start 2024-01-16T15:19:38.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.367+0000 7f42b2a00640 1 -- start start 2024-01-16T15:19:38.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cbf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 0x7f42ac09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac159070 con 0x7f42ac15d4b0 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac1592c0 con 0x7f42ac0cf9a0 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ac1595a0 con 0x7f42ac0b31d0 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 0x7f42ac09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 0x7f42ac09cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59312/0 (socket says 172.21.15.53:59312) 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 -- 172.21.15.53:0/441836665 learned_addr learned my addr 172.21.15.53:0/441836665 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b0c00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b1600640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 -- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 msgr2=0x7f42ac157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 -- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 msgr2=0x7f42ac0cbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b2000640 1 -- 172.21.15.53:0/441836665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42940094d0 con 0x7f42ac15d4b0 2024-01-16T15:19:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.369+0000 7f42b1600640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:38.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.371+0000 7f42b2000640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 0x7f42ac09cce0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f42a400efc0 tx=0x7f42a400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.371+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a4009ba0 con 0x7f42ac15d4b0 2024-01-16T15:19:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.371+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a400f040 con 0x7f42ac15d4b0 2024-01-16T15:19:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.371+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a40146c0 con 0x7f42ac15d4b0 2024-01-16T15:19:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.371+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ac159880 con 0x7f42ac15d4b0 2024-01-16T15:19:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.371+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ac159b60 con 0x7f42ac15d4b0 2024-01-16T15:19:38.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.373+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ac158200 con 0x7f42ac15d4b0 2024-01-16T15:19:38.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.374+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f42a4014820 con 0x7f42ac15d4b0 2024-01-16T15:19:38.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.374+0000 7f42a9600640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f428406c430 0x7f428406e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.375+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f42a40895b0 con 0x7f42ac15d4b0 2024-01-16T15:19:38.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.375+0000 7f42b1600640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f428406c430 0x7f428406e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.376+0000 7f42b1600640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f428406c430 0x7f428406e8f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f429c000f60 tx=0x7f429c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:38.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.379+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f42ac158200 con 0x7f42ac15d4b0 2024-01-16T15:19:38.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.585+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}] v 0) v1 -- 0x7f42ac158200 con 0x7f42ac15d4b0 2024-01-16T15:19:38.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.592+0000 7f42a9600640 1 -- 172.21.15.53:0/441836665 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/volumes/clone_delay_every_regfile}]=0 v44)=0 v44) v1 ==== 137+0+0 (secure 0 0 0) 0x7f42ac158200 con 0x7f42ac15d4b0 2024-01-16T15:19:38.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f428406c430 msgr2=0x7f428406e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f428406c430 0x7f428406e8f0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f429c000f60 tx=0x7f429c009040 comp rx=0 tx=0).stop 2024-01-16T15:19:38.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 msgr2=0x7f42ac09cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 0x7f42ac09cce0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f42a400efc0 tx=0x7f42a400c490 comp rx=0 tx=0).stop 2024-01-16T15:19:38.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 shutdown_connections 2024-01-16T15:19:38.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f428406c430 0x7f428406e8f0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f42ac15d4b0 0x7f42ac09cce0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f42ac0cf9a0 0x7f42ac0cbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 --2- 172.21.15.53:0/441836665 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f42ac0b31d0 0x7f42ac157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 >> 172.21.15.53:0/441836665 conn(0x7f42ac0d3000 msgr2=0x7f42ac0b0570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:38.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 shutdown_connections 2024-01-16T15:19:38.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.595+0000 7f42b2a00640 1 -- 172.21.15.53:0/441836665 wait complete. 2024-01-16T15:19:38.608 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv1 --mode=777 2024-01-16T15:19:38.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.852+0000 7f6e60a00640 1 Processor -- start 2024-01-16T15:19:38.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.852+0000 7f6e60a00640 1 -- start start 2024-01-16T15:19:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.854+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 0x7f6e5c0d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.854+0000 7f6e60a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0d0a10 con 0x7f6e5c0d0100 2024-01-16T15:19:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.854+0000 7f6e60a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e5c0cf080 con 0x7f6e5c160920 2024-01-16T15:19:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.854+0000 7f6e60a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c6be0 con 0x7f6e5c0b3f30 2024-01-16T15:19:38.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.854+0000 7f6e59e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6e5c0d0100 0x7f6e5c0d0f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39430/0 (socket says 172.21.15.53:39430) 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.854+0000 7f6e59e00640 1 -- 172.21.15.53:0/4094806504 learned_addr learned my addr 172.21.15.53:0/4094806504 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e5b200640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 0x7f6e5c0d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412932620 0 0) 0x7f6e5c0d0a10 con 0x7f6e5c0d0100 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e3c003520 con 0x7f6e5c0d0100 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 424082560 0 0) 0x7f6e3c003520 con 0x7f6e5c0d0100 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 msgr2=0x7f6e5c0d1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:38.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 0x7f6e5c0d1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 >> v1:172.21.15.146:6789/0 conn(0x7f6e5c160920 legacy=0x7f6e5c0cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c0ce7e0 con 0x7f6e5c0d0100 2024-01-16T15:19:38.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.855+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6e44001670 con 0x7f6e5c0d0100 2024-01-16T15:19:38.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 >> v1:172.21.15.53:6789/0 conn(0x7f6e5c0d0100 legacy=0x7f6e5c0d0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c0039a0 0x7f6e3c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 0x7f6e3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e3c004d00 0x7f6e3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c0c6be0 con 0x7f6e3c004d00 2024-01-16T15:19:38.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c00ba30 con 0x7f6e3c0039a0 2024-01-16T15:19:38.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e3c00bbf0 con 0x7f6e5c0b3f30 2024-01-16T15:19:38.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5b200640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e3c004d00 0x7f6e3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e59e00640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 0x7f6e3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5a800640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c0039a0 0x7f6e3c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5b200640 1 -- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 msgr2=0x7f6e3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:38.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5b200640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0b3f30 0x7f6e3c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5b200640 1 -- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c0039a0 msgr2=0x7f6e3c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:38.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5b200640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c0039a0 0x7f6e3c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5b200640 1 -- 172.21.15.53:0/4094806504 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e3c003520 con 0x7f6e3c004d00 2024-01-16T15:19:38.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.856+0000 7f6e5a800640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c0039a0 0x7f6e3c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:38.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e5b200640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e3c004d00 0x7f6e3c005180 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009a90 tx=0x7f6e3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42d09dbc11385aa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:38.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5402c680 con 0x7f6e3c004d00 2024-01-16T15:19:38.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5402c7e0 con 0x7f6e3c004d00 2024-01-16T15:19:38.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e59400640 1 -- 172.21.15.53:0/4094806504 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5402cb70 con 0x7f6e3c004d00 2024-01-16T15:19:38.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e60a00640 1 -- 172.21.15.53:0/4094806504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e3c004d00 msgr2=0x7f6e3c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e60a00640 1 --2- 172.21.15.53:0/4094806504 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e3c004d00 0x7f6e3c005180 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009a90 tx=0x7f6e3c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e5b200640 1 -- 172.21.15.53:0/4094806504 reap_dead start 2024-01-16T15:19:38.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.858+0000 7f6e60a00640 1 -- 172.21.15.53:0/4094806504 shutdown_connections 2024-01-16T15:19:38.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.859+0000 7f6e60a00640 1 -- 172.21.15.53:0/4094806504 >> 172.21.15.53:0/4094806504 conn(0x7f6e5c0d3090 msgr2=0x7f6e5c0d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:38.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.859+0000 7f6e60a00640 1 -- 172.21.15.53:0/4094806504 shutdown_connections 2024-01-16T15:19:38.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.859+0000 7f6e60a00640 1 -- 172.21.15.53:0/4094806504 wait complete. 2024-01-16T15:19:38.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.861+0000 7f6e60a00640 1 Processor -- start 2024-01-16T15:19:38.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.861+0000 7f6e60a00640 1 -- start start 2024-01-16T15:19:38.861 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c004d00 0x7f6e5c1547d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0cb610 0x7f6e5c1f2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e60a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 0x7f6e5c0c8410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e60a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1f6880 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e60a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c154d10 con 0x7f6e3c004d00 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e60a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c1f3790 con 0x7f6e5c0cb610 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 0x7f6e5c0c8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.862 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 0x7f6e5c0c8410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59324/0 (socket says 172.21.15.53:59324) 2024-01-16T15:19:38.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 -- 172.21.15.53:0/3777029314 learned_addr learned my addr 172.21.15.53:0/3777029314 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:38.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 -- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0cb610 msgr2=0x7f6e5c1f2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0cb610 0x7f6e5c1f2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 -- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c004d00 msgr2=0x7f6e5c1547d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:38.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c004d00 0x7f6e5c1547d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:38.863 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.862+0000 7f6e5b200640 1 -- 172.21.15.53:0/3777029314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c1f3a70 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.863+0000 7f6e5b200640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 0x7f6e5c0c8410 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009a90 tx=0x7f6e54004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:38.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.863+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5402cd60 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.863+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e54003610 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.863+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e540039c0 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.863+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c1f3d50 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.864 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.863+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c1f3480 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.864+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c067450 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.865+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6e54003b20 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.866+0000 7f6e52a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e3406c480 0x7f6e3406e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:38.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.866+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6e540a98c0 con 0x7f6e5c1f6d10 2024-01-16T15:19:38.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.867+0000 7f6e5a800640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e3406c480 0x7f6e3406e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:38.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.868+0000 7f6e5a800640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e3406c480 0x7f6e3406e940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c007fd0 tx=0x7f6e4c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:38.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:38.871+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e54075e30 con 0x7f6e5c1f6d10 2024-01-16T15:19:39.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.093+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv1", "mode": "777", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e5c1f3ee0 con 0x7f6e3406c480 2024-01-16T15:19:39.097 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:39.105+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:39.097 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:39.105+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:39.097 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:39.105+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:39.097 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:39.105+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:39.097 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:19:39.105+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:19:39.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.187+0000 7f6e52a00640 1 -- 172.21.15.53:0/3777029314 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6e5c1f3ee0 con 0x7f6e3406c480 2024-01-16T15:19:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e3406c480 msgr2=0x7f6e3406e940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e3406c480 0x7f6e3406e940 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c007fd0 tx=0x7f6e4c009040 comp rx=0 tx=0).stop 2024-01-16T15:19:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 msgr2=0x7f6e5c0c8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 0x7f6e5c0c8410 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009a90 tx=0x7f6e54004730 comp rx=0 tx=0).stop 2024-01-16T15:19:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 shutdown_connections 2024-01-16T15:19:39.181 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e3406c480 0x7f6e3406e940 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e5c1f6d10 0x7f6e5c0c8410 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6e5c0cb610 0x7f6e5c1f2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 --2- 172.21.15.53:0/3777029314 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e3c004d00 0x7f6e5c1547d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 >> 172.21.15.53:0/3777029314 conn(0x7f6e5c0d3090 msgr2=0x7f6e5c0d0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 shutdown_connections 2024-01-16T15:19:39.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.190+0000 7f6e60a00640 1 -- 172.21.15.53:0/3777029314 wait complete. 2024-01-16T15:19:39.199 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:19:39.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.418+0000 7efcb0200640 1 Processor -- start 2024-01-16T15:19:39.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.419+0000 7efcb0200640 1 -- start start 2024-01-16T15:19:39.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.420+0000 7efcb0200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 0x7efca815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.420+0000 7efcb0200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca806c470 con 0x7efca813e360 2024-01-16T15:19:39.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.420+0000 7efcb0200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca806c750 con 0x7efca80cf9a0 2024-01-16T15:19:39.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.420+0000 7efcb0200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca806ca30 con 0x7efca813df80 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.421+0000 7efcace00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7efca813e360 0x7efca80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39444/0 (socket says 172.21.15.53:39444) 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.421+0000 7efcace00640 1 -- 172.21.15.53:0/3000972382 learned_addr learned my addr 172.21.15.53:0/3000972382 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.421+0000 7efcad800640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 0x7efca815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.421+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 469457955 0 0) 0x7efca806c470 con 0x7efca813e360 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.421+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc88003520 con 0x7efca813e360 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4088254265 0 0) 0x7efc88003520 con 0x7efca813e360 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 msgr2=0x7efca815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 0x7efca815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 >> v1:172.21.15.146:6789/0 conn(0x7efca80cf9a0 legacy=0x7efca80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca806cd10 con 0x7efca813e360 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7efc90001670 con 0x7efca813e360 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 >> v1:172.21.15.53:6789/0 conn(0x7efca813e360 legacy=0x7efca80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc880039a0 0x7efc88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc880047c0 0x7efc88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 0x7efc88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca806ca30 con 0x7efc880047c0 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc8800ba30 con 0x7efc880039a0 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc8800bbf0 con 0x7efca813df80 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efcace00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc880047c0 0x7efc88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.422+0000 7efcad800640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc880039a0 0x7efc88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcae200640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 0x7efc88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcace00640 1 -- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 msgr2=0x7efc88005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcace00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efca813df80 0x7efc88005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcace00640 1 -- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc880039a0 msgr2=0x7efc88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:39.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcace00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc880039a0 0x7efc88003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcace00640 1 -- 172.21.15.53:0/3000972382 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc88003520 con 0x7efc880047c0 2024-01-16T15:19:39.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.423+0000 7efcad800640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efc880039a0 0x7efc88003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efcace00640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc880047c0 0x7efc88004c40 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7efc9000c210 tx=0x7efc880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=457fa068e5fb2597 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc90005ad0 con 0x7efc880047c0 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc90005c30 con 0x7efc880047c0 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efca7e00640 1 -- 172.21.15.53:0/3000972382 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9002c870 con 0x7efc880047c0 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efcb0200640 1 -- 172.21.15.53:0/3000972382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc880047c0 msgr2=0x7efc88004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efcb0200640 1 --2- 172.21.15.53:0/3000972382 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efc880047c0 0x7efc88004c40 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7efc9000c210 tx=0x7efc880057f0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.424+0000 7efcae200640 1 -- 172.21.15.53:0/3000972382 reap_dead start 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.425+0000 7efcb0200640 1 -- 172.21.15.53:0/3000972382 shutdown_connections 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.425+0000 7efcb0200640 1 -- 172.21.15.53:0/3000972382 >> 172.21.15.53:0/3000972382 conn(0x7efca806a800 msgr2=0x7efca806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.425+0000 7efcb0200640 1 -- 172.21.15.53:0/3000972382 shutdown_connections 2024-01-16T15:19:39.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.425+0000 7efcb0200640 1 -- 172.21.15.53:0/3000972382 wait complete. 2024-01-16T15:19:39.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.426+0000 7efcb0200640 1 Processor -- start 2024-01-16T15:19:39.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.427+0000 7efcb0200640 1 -- start start 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcb0200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc880047c0 0x7efca81f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcb0200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca81f7010 0x7efca813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcb0200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 0x7efca80b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcb0200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca81f6630 con 0x7efca80b2cf0 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcb0200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca80b36b0 con 0x7efca81f7010 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcb0200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca80c5100 con 0x7efc880047c0 2024-01-16T15:19:39.420 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 0x7efca80b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 0x7efca80b3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59338/0 (socket says 172.21.15.53:59338) 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 -- 172.21.15.53:0/569999082 learned_addr learned my addr 172.21.15.53:0/569999082 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcace00640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca81f7010 0x7efca813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcad800640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc880047c0 0x7efca81f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 -- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc880047c0 msgr2=0x7efca81f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc880047c0 0x7efca81f3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 -- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca81f7010 msgr2=0x7efca813d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca81f7010 0x7efca813d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcae200640 1 -- 172.21.15.53:0/569999082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca81f3ff0 con 0x7efca80b2cf0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.428+0000 7efcace00640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca81f7010 0x7efca813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.429+0000 7efcae200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 0x7efca80b3170 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7efca80cfea0 tx=0x7efca000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.429+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca0003b70 con 0x7efca80b2cf0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.429+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efca0003cd0 con 0x7efca80b2cf0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.429+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca00052b0 con 0x7efca80b2cf0 2024-01-16T15:19:39.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.429+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca80c4860 con 0x7efca80b2cf0 2024-01-16T15:19:39.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.429+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca80c4e20 con 0x7efca80b2cf0 2024-01-16T15:19:39.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.430+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca80675d0 con 0x7efca80b2cf0 2024-01-16T15:19:39.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.431+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7efca0005410 con 0x7efca80b2cf0 2024-01-16T15:19:39.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.432+0000 7efca5600640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efc8006c460 0x7efc8006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.432+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7efca00883b0 con 0x7efca80b2cf0 2024-01-16T15:19:39.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.433+0000 7efcad800640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efc8006c460 0x7efc8006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.434+0000 7efcad800640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efc8006c460 0x7efc8006e920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7efc980099c0 tx=0x7efc98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:39.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.436+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7efca0054920 con 0x7efca80b2cf0 2024-01-16T15:19:39.661 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.670+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7efca815bee0 con 0x7efc8006c460 2024-01-16T15:19:39.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.681+0000 7efca5600640 1 -- 172.21.15.53:0/569999082 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7efc980118e0 con 0x7efc8006c460 2024-01-16T15:19:39.673 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710 2024-01-16T15:19:39.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.684+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efc8006c460 msgr2=0x7efc8006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.684+0000 7efcb0200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efc8006c460 0x7efc8006e920 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7efc980099c0 tx=0x7efc98008040 comp rx=0 tx=0).stop 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.684+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 msgr2=0x7efca80b3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.684+0000 7efcb0200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 0x7efca80b3170 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7efca80cfea0 tx=0x7efca000dc00 comp rx=0 tx=0).stop 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 shutdown_connections 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7efc8006c460 0x7efc8006e920 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7efca80b2cf0 0x7efca80b3170 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7efca81f7010 0x7efca813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 --2- 172.21.15.53:0/569999082 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7efc880047c0 0x7efca81f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 >> 172.21.15.53:0/569999082 conn(0x7efca806a800 msgr2=0x7efca80afac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 shutdown_connections 2024-01-16T15:19:39.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.685+0000 7efcb0200640 1 -- 172.21.15.53:0/569999082 wait complete. 2024-01-16T15:19:39.694 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:19:39.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.931+0000 7f6ab8a00640 1 Processor -- start 2024-01-16T15:19:39.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.932+0000 7f6ab8a00640 1 -- start start 2024-01-16T15:19:39.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.933+0000 7f6ab8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ab413fcb0 0x7f6ab40cd6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.933+0000 7f6ab8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab40ce440 con 0x7f6ab413cf10 2024-01-16T15:19:39.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.933+0000 7f6ab8a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab40ce720 con 0x7f6ab40cdc20 2024-01-16T15:19:39.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.933+0000 7f6ab8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab40cea00 con 0x7f6ab413fcb0 2024-01-16T15:19:39.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.934+0000 7f6ab2800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6ab413cf10 0x7f6ab413f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39458/0 (socket says 172.21.15.53:39458) 2024-01-16T15:19:39.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.934+0000 7f6ab2800640 1 -- 172.21.15.53:0/3245655089 learned_addr learned my addr 172.21.15.53:0/3245655089 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:19:39.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.934+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473795476 0 0) 0x7f6ab40ce440 con 0x7f6ab413cf10 2024-01-16T15:19:39.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.934+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a94005730 con 0x7f6ab413cf10 2024-01-16T15:19:39.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6aabe00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ab413fcb0 0x7f6ab40cd6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661504396 0 0) 0x7f6a94005730 con 0x7f6ab413cf10 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ab413fcb0 msgr2=0x7f6ab40cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ab413fcb0 0x7f6ab40cd6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 >> v1:172.21.15.146:6789/0 conn(0x7f6ab40cdc20 legacy=0x7f6ab40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab40b1d90 con 0x7f6ab413cf10 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6aa4001670 con 0x7f6ab413cf10 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 >> v1:172.21.15.53:6789/0 conn(0x7f6ab413cf10 legacy=0x7f6ab413f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a94005bb0 0x7f6a94005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a940069d0 0x7f6a94006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a940074b0 0x7f6a9400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab40cea00 con 0x7f6a940069d0 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9400e380 con 0x7f6a940074b0 2024-01-16T15:19:39.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9400e540 con 0x7f6a94005bb0 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6aabe00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ab413fcb0 0x7f6ab40cd6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.935+0000 7f6aabe00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a940069d0 0x7f6a94006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6ab2800640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a94005bb0 0x7f6a94005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6ab3200640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a940074b0 0x7f6a9400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6aabe00640 1 -- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a94005bb0 msgr2=0x7f6a94005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6aabe00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a94005bb0 0x7f6a94005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6aabe00640 1 -- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a940074b0 msgr2=0x7f6a9400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6aabe00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6a940074b0 0x7f6a9400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.936+0000 7f6aabe00640 1 -- 172.21.15.53:0/3245655089 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a94005730 con 0x7f6a940069d0 2024-01-16T15:19:39.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.937+0000 7f6aabe00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a940069d0 0x7f6a94006e50 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00a860 tx=0x7f6a94007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ce0dd856a78710 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:39.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.937+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c032600 con 0x7f6a940069d0 2024-01-16T15:19:39.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.937+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c032780 con 0x7f6a940069d0 2024-01-16T15:19:39.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.937+0000 7f6ab1e00640 1 -- 172.21.15.53:0/3245655089 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c032b70 con 0x7f6a940069d0 2024-01-16T15:19:39.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab8a00640 1 -- 172.21.15.53:0/3245655089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a940069d0 msgr2=0x7f6a94006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/3245655089 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6a940069d0 0x7f6a94006e50 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c00a860 tx=0x7f6a94007a00 comp rx=0 tx=0).stop 2024-01-16T15:19:39.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab3200640 1 -- 172.21.15.53:0/3245655089 reap_dead start 2024-01-16T15:19:39.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab8a00640 1 -- 172.21.15.53:0/3245655089 shutdown_connections 2024-01-16T15:19:39.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab8a00640 1 -- 172.21.15.53:0/3245655089 >> 172.21.15.53:0/3245655089 conn(0x7f6ab406a7d0 msgr2=0x7f6ab406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:39.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab8a00640 1 -- 172.21.15.53:0/3245655089 shutdown_connections 2024-01-16T15:19:39.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.938+0000 7f6ab8a00640 1 -- 172.21.15.53:0/3245655089 wait complete. 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.940+0000 7f6ab8a00640 1 Processor -- start 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.940+0000 7f6ab8a00640 1 -- start start 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6ab8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a940069d0 0x7f6ab4148e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6ab8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 0x7f6ab415bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6ab8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ab4153840 0x7f6ab4153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6ab8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab4154210 con 0x7f6ab40701d0 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6ab8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab415d020 con 0x7f6ab4153840 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6ab8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ab41493d0 con 0x7f6a940069d0 2024-01-16T15:19:39.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6aabe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 0x7f6ab415bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6aabe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 0x7f6ab415bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59364/0 (socket says 172.21.15.53:59364) 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.941+0000 7f6aabe00640 1 -- 172.21.15.53:0/1093301184 learned_addr learned my addr 172.21.15.53:0/1093301184 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6ab2800640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a940069d0 0x7f6ab4148e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6ab3200640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ab4153840 0x7f6ab4153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6aabe00640 1 -- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a940069d0 msgr2=0x7f6ab4148e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6aabe00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a940069d0 0x7f6ab4148e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6aabe00640 1 -- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ab4153840 msgr2=0x7f6ab4153cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6aabe00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ab4153840 0x7f6ab4153cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6aabe00640 1 -- 172.21.15.53:0/1093301184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab415d4b0 con 0x7f6ab40701d0 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6ab3200640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ab4153840 0x7f6ab4153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6ab2800640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a940069d0 0x7f6ab4148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:19:39.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.942+0000 7f6aabe00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 0x7f6ab415bb10 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c034040 tx=0x7f6a9c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:39.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.943+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c044410 con 0x7f6ab40701d0 2024-01-16T15:19:39.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.943+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9c0445b0 con 0x7f6ab40701d0 2024-01-16T15:19:39.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.943+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9c044960 con 0x7f6ab40701d0 2024-01-16T15:19:39.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.943+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab415c410 con 0x7f6ab40701d0 2024-01-16T15:19:39.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.943+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab415c9d0 con 0x7f6ab40701d0 2024-01-16T15:19:39.935 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.944+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ab415d1d0 con 0x7f6ab40701d0 2024-01-16T15:19:39.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.945+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 42) v1 ==== 91120+0+0 (secure 0 0 0) 0x7f6a9c044ac0 con 0x7f6ab40701d0 2024-01-16T15:19:39.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.946+0000 7f6aaaa00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8c06c430 0x7f6a8c06e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:19:39.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.946+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 5737+0+0 (secure 0 0 0) 0x7f6a9c0a9fe0 con 0x7f6ab40701d0 2024-01-16T15:19:39.937 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.946+0000 7f6ab2800640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8c06c430 0x7f6a8c06e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:19:39.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.948+0000 7f6ab2800640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8c06c430 0x7f6a8c06e8f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6ab413a280 tx=0x7f6aa4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:19:39.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:39.950+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6ab415d1d0 con 0x7f6ab40701d0 2024-01-16T15:19:40.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.173+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ab40ce0b0 con 0x7f6a8c06c430 2024-01-16T15:19:40.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.184+0000 7f6aaaa00640 1 -- 172.21.15.53:0/1093301184 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f6aa4010920 con 0x7f6a8c06c430 2024-01-16T15:19:40.175 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.186+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8c06c430 msgr2=0x7f6a8c06e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.186+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8c06c430 0x7f6a8c06e8f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f6ab413a280 tx=0x7f6aa4009040 comp rx=0 tx=0).stop 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.186+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 msgr2=0x7f6ab415bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.186+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 0x7f6ab415bb10 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f6a9c034040 tx=0x7f6a9c004730 comp rx=0 tx=0).stop 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 shutdown_connections 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6a8c06c430 0x7f6a8c06e8f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ab4153840 0x7f6ab4153cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6ab40701d0 0x7f6ab415bb10 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 --2- 172.21.15.53:0/1093301184 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6a940069d0 0x7f6ab4148e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:19:40.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 >> 172.21.15.53:0/1093301184 conn(0x7f6ab406a7d0 msgr2=0x7f6ab406cfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:19:40.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 shutdown_connections 2024-01-16T15:19:40.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:19:40.187+0000 7f6ab8a00640 1 -- 172.21.15.53:0/1093301184 wait complete. 2024-01-16T15:19:40.197 DEBUG:tasks.cephfs.test_volumes:filling subvolume sv1 with 3 files each 1024MB size under directory volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710 2024-01-16T15:19:40.198 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:19:40.198 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:40.250 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:40.251 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710/subvolume_file.0 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:19:56.421 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:56.430 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:19:57.356 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:19:57.356 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:19:57.357 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 17.0316 s, 63.0 MB/s 2024-01-16T15:19:57.358 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:19:57.358 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:19:57.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:19:57.416 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710/subvolume_file.1 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:20:11.614 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:20:11.615 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:20:11.615 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.1369 s, 76.0 MB/s 2024-01-16T15:20:11.616 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:20:11.616 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:20:11.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:20:11.673 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710/subvolume_file.2 bs=1M conv=fdatasync count=1024 seek=0) 2024-01-16T15:20:26.124 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records in 2024-01-16T15:20:26.124 INFO:teuthology.orchestra.run.smithi053.stderr:1024+0 records out 2024-01-16T15:20:26.125 INFO:teuthology.orchestra.run.smithi053.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 14.3893 s, 74.6 MB/s 2024-01-16T15:20:26.126 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot create cephfs sv1 ss1 2024-01-16T15:20:26.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.366+0000 7f855a800640 1 Processor -- start 2024-01-16T15:20:26.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.367+0000 7f855a800640 1 -- start start 2024-01-16T15:20:26.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.368+0000 7f855a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8554143c40 0x7f8554071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.368+0000 7f855a800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8554070e50 con 0x7f85540ccc10 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.368+0000 7f855a800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8554144020 con 0x7f85540717d0 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.368+0000 7f855a800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85540723c0 con 0x7f8554143c40 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8554143c40 0x7f8554071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f85540ccc10 0x7f855406d160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54248/0 (socket says 172.21.15.53:54248) 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8554143c40 0x7f8554071290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47608/0 (socket says 172.21.15.53:47608) 2024-01-16T15:20:26.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559e00640 1 -- 172.21.15.53:0/322879989 learned_addr learned my addr 172.21.15.53:0/322879989 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:26.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559400640 1 -- 172.21.15.53:0/322879989 >> v1:172.21.15.146:6789/0 conn(0x7f85540717d0 legacy=0x7f85540cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:26.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559400640 1 -- 172.21.15.53:0/322879989 >> v1:172.21.15.53:6789/0 conn(0x7f85540ccc10 legacy=0x7f855406d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.369+0000 7f8559400640 1 -- 172.21.15.53:0/322879989 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554071e00 con 0x7f8554143c40 2024-01-16T15:20:26.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.370+0000 7f8559400640 1 --2- 172.21.15.53:0/322879989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554071290 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f853c0097a0 tx=0x7f853c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38094470f0b45871 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:26.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.371+0000 7f8553e00640 1 -- 172.21.15.53:0/322879989 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c02e070 con 0x7f8554143c40 2024-01-16T15:20:26.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.371+0000 7f8553e00640 1 -- 172.21.15.53:0/322879989 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c0315a0 con 0x7f8554143c40 2024-01-16T15:20:26.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.371+0000 7f8553e00640 1 -- 172.21.15.53:0/322879989 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c031930 con 0x7f8554143c40 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.371+0000 7f855a800640 1 -- 172.21.15.53:0/322879989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 msgr2=0x7f8554071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.371+0000 7f855a800640 1 --2- 172.21.15.53:0/322879989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554071290 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f853c0097a0 tx=0x7f853c02b840 comp rx=0 tx=0).stop 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.372+0000 7f855a800640 1 -- 172.21.15.53:0/322879989 shutdown_connections 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.372+0000 7f855a800640 1 --2- 172.21.15.53:0/322879989 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554071290 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.372+0000 7f855a800640 1 -- 172.21.15.53:0/322879989 >> 172.21.15.53:0/322879989 conn(0x7f855406a800 msgr2=0x7f855406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.372+0000 7f855a800640 1 -- 172.21.15.53:0/322879989 shutdown_connections 2024-01-16T15:20:26.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.372+0000 7f855a800640 1 -- 172.21.15.53:0/322879989 wait complete. 2024-01-16T15:20:26.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.374+0000 7f855a800640 1 Processor -- start 2024-01-16T15:20:26.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.374+0000 7f855a800640 1 -- start start 2024-01-16T15:20:26.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f855a800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540717d0 0x7f85540c9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f855a800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85540ccc10 0x7f85540c9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f855a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554135cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f855a800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855415afd0 con 0x7f8554143c40 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f855a800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554139d40 con 0x7f85540717d0 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f855a800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85540c4ef0 con 0x7f85540ccc10 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f8559e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554135cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f8559e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554135cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47610/0 (socket says 172.21.15.53:47610) 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f8559e00640 1 -- 172.21.15.53:0/2927167235 learned_addr learned my addr 172.21.15.53:0/2927167235 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f8559400640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540717d0 0x7f85540c9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.375+0000 7f8558a00640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85540ccc10 0x7f85540c9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8559e00640 1 -- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85540ccc10 msgr2=0x7f85540c9ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8559e00640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85540ccc10 0x7f85540c9ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8559e00640 1 -- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540717d0 msgr2=0x7f85540c9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8559e00640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540717d0 0x7f85540c9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8559e00640 1 -- 172.21.15.53:0/2927167235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c009400 con 0x7f8554143c40 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8559e00640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554135cc0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f854c00c420 tx=0x7f854c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c005ce0 con 0x7f8554143c40 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f854c005e40 con 0x7f8554143c40 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854c0072d0 con 0x7f8554143c40 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85540c51d0 con 0x7f8554143c40 2024-01-16T15:20:26.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.376+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85540c5790 con 0x7f8554143c40 2024-01-16T15:20:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.378+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8554067450 con 0x7f8554143c40 2024-01-16T15:20:26.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.379+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 43) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f854c0074f0 con 0x7f8554143c40 2024-01-16T15:20:26.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.380+0000 7f8551600640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f852c06c4c0 0x7f852c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.380+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f854c087f30 con 0x7f8554143c40 2024-01-16T15:20:26.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.381+0000 7f8559400640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f852c06c4c0 0x7f852c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.382+0000 7f8559400640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f852c06c4c0 0x7f852c06e980 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f853c036000 tx=0x7f853c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:26.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.385+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f854c054590 con 0x7f8554143c40 2024-01-16T15:20:26.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.611+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot create", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""]}) v1 -- 0x7f855406cc40 con 0x7f852c06c4c0 2024-01-16T15:20:26.621 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:26.628 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:26.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.645+0000 7f8551600640 1 -- 172.21.15.53:0/2927167235 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f853c031b10 con 0x7f852c06c4c0 2024-01-16T15:20:26.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f852c06c4c0 msgr2=0x7f852c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f852c06c4c0 0x7f852c06e980 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f853c036000 tx=0x7f853c03d040 comp rx=0 tx=0).stop 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 msgr2=0x7f8554135cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554135cc0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f854c00c420 tx=0x7f854c00c8e0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 shutdown_connections 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f852c06c4c0 0x7f852c06e980 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8554143c40 0x7f8554135cc0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85540ccc10 0x7f85540c9ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 --2- 172.21.15.53:0/2927167235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85540717d0 0x7f85540c9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 >> 172.21.15.53:0/2927167235 conn(0x7f855406a800 msgr2=0x7f85540afcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.648+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 shutdown_connections 2024-01-16T15:20:26.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.649+0000 7f855a800640 1 -- 172.21.15.53:0/2927167235 wait complete. 2024-01-16T15:20:26.658 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv1 2024-01-16T15:20:26.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.922+0000 7f63d7200640 1 Processor -- start 2024-01-16T15:20:26.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.922+0000 7f63d7200640 1 -- start start 2024-01-16T15:20:26.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.923+0000 7f63d7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63d015b320 0x7f63d00ce060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.923+0000 7f63d7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63d00ce5a0 0x7f63d00ceae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.923+0000 7f63d7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d00cc730 con 0x7f63d00ccbc0 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.923+0000 7f63d7200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00cbe00 con 0x7f63d00ce5a0 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d00cc0e0 con 0x7f63d015b320 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d5400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f63d00ccbc0 0x7f63d015b700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54252/0 (socket says 172.21.15.53:54252) 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d5400640 1 -- 172.21.15.53:0/2765060982 learned_addr learned my addr 172.21.15.53:0/2765060982 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d6800640 1 --2- 172.21.15.53:0/2765060982 >> v2:172.21.15.53:3300/0 conn(0x7f63d00ce5a0 0x7f63d00ceae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d5e00640 1 --2- 172.21.15.53:0/2765060982 >> v2:172.21.15.146:3300/0 conn(0x7f63d015b320 0x7f63d00ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d4a00640 1 -- 172.21.15.53:0/2765060982 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372144656 0 0) 0x7f63d00cc730 con 0x7f63d00ccbc0 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d4a00640 1 -- 172.21.15.53:0/2765060982 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b0003520 con 0x7f63d00ccbc0 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d6800640 1 -- 172.21.15.53:0/2765060982 >> v2:172.21.15.146:3300/0 conn(0x7f63d015b320 msgr2=0x7f63d00ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:26.917 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d6800640 1 --2- 172.21.15.53:0/2765060982 >> v2:172.21.15.146:3300/0 conn(0x7f63d015b320 0x7f63d00ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.924+0000 7f63d6800640 1 -- 172.21.15.53:0/2765060982 >> v1:172.21.15.53:6789/0 conn(0x7f63d00ccbc0 legacy=0x7f63d015b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.925+0000 7f63d6800640 1 -- 172.21.15.53:0/2765060982 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d00cc3c0 con 0x7f63d00ce5a0 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.926+0000 7f63d6800640 1 --2- 172.21.15.53:0/2765060982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d00ce5a0 0x7f63d00ceae0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f63d00714d0 tx=0x7f63c802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37a2bb3a5b0e5cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.926+0000 7f63d4a00640 1 -- 172.21.15.53:0/2765060982 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c802cc80 con 0x7f63d00ce5a0 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.926+0000 7f63d4a00640 1 -- 172.21.15.53:0/2765060982 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c802cde0 con 0x7f63d00ce5a0 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.926+0000 7f63d4a00640 1 -- 172.21.15.53:0/2765060982 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8034660 con 0x7f63d00ce5a0 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.926+0000 7f63d7200640 1 -- 172.21.15.53:0/2765060982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d00ce5a0 msgr2=0x7f63d00ceae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.926+0000 7f63d7200640 1 --2- 172.21.15.53:0/2765060982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d00ce5a0 0x7f63d00ceae0 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f63d00714d0 tx=0x7f63c802bed0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.927+0000 7f63d7200640 1 -- 172.21.15.53:0/2765060982 shutdown_connections 2024-01-16T15:20:26.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.927+0000 7f63d7200640 1 --2- 172.21.15.53:0/2765060982 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d00ce5a0 0x7f63d00ceae0 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.927+0000 7f63d7200640 1 --2- 172.21.15.53:0/2765060982 >> v2:172.21.15.146:3300/0 conn(0x7f63d015b320 0x7f63d00ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.927+0000 7f63d7200640 1 -- 172.21.15.53:0/2765060982 >> 172.21.15.53:0/2765060982 conn(0x7f63d00b2a60 msgr2=0x7f63d00b2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:26.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.927+0000 7f63d7200640 1 -- 172.21.15.53:0/2765060982 shutdown_connections 2024-01-16T15:20:26.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.927+0000 7f63d7200640 1 -- 172.21.15.53:0/2765060982 wait complete. 2024-01-16T15:20:26.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.929+0000 7f63d7200640 1 Processor -- start 2024-01-16T15:20:26.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.929+0000 7f63d7200640 1 -- start start 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63d00ccbc0 0x7f63d009c580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63d00ce5a0 0x7f63d0151440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 0x7f63d006cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d009b1a0 con 0x7f63d015b320 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d009cac0 con 0x7f63d00ccbc0 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0151980 con 0x7f63d00ce5a0 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 0x7f63d006cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d6800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 0x7f63d006cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47636/0 (socket says 172.21.15.53:47636) 2024-01-16T15:20:26.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d6800640 1 -- 172.21.15.53:0/124592819 learned_addr learned my addr 172.21.15.53:0/124592819 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d5400640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63d00ce5a0 0x7f63d0151440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d5e00640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63d00ccbc0 0x7f63d009c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.930+0000 7f63d6800640 1 -- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63d00ce5a0 msgr2=0x7f63d0151440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d6800640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63d00ce5a0 0x7f63d0151440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d6800640 1 -- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63d00ccbc0 msgr2=0x7f63d009c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d6800640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63d00ccbc0 0x7f63d009c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d6800640 1 -- 172.21.15.53:0/124592819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c8009a70 con 0x7f63d015b320 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d5e00640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63d00ccbc0 0x7f63d009c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d5400640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63d00ce5a0 0x7f63d0151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d6800640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 0x7f63d006cd10 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f63c802c4e0 tx=0x7f63c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:26.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8040650 con 0x7f63d015b320 2024-01-16T15:20:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c80407b0 con 0x7f63d015b320 2024-01-16T15:20:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d014ed20 con 0x7f63d015b320 2024-01-16T15:20:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c8040b60 con 0x7f63d015b320 2024-01-16T15:20:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.931+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d014f2e0 con 0x7f63d015b320 2024-01-16T15:20:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.933+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d0067b60 con 0x7f63d015b320 2024-01-16T15:20:26.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.933+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 43) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f63c8040cc0 con 0x7f63d015b320 2024-01-16T15:20:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.935+0000 7f63ce000640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63a806c4c0 0x7f63a806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.935+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f63c80ab4c0 con 0x7f63d015b320 2024-01-16T15:20:26.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.936+0000 7f63d5e00640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63a806c4c0 0x7f63a806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:26.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.937+0000 7f63d5e00640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63a806c4c0 0x7f63a806e980 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f63c0006fd0 tx=0x7f63c0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:26.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:26.940+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f63c8077d80 con 0x7f63d015b320 2024-01-16T15:20:27.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.163+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""]}) v1 -- 0x7f63d015be10 con 0x7f63a806c4c0 2024-01-16T15:20:27.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.178+0000 7f63ce000640 1 -- 172.21.15.53:0/124592819 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f63c0009530 con 0x7f63a806c4c0 2024-01-16T15:20:27.170 INFO:teuthology.orchestra.run.smithi053.stdout:/volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63a806c4c0 msgr2=0x7f63a806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63a806c4c0 0x7f63a806e980 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f63c0006fd0 tx=0x7f63c0008040 comp rx=0 tx=0).stop 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 msgr2=0x7f63d006cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 0x7f63d006cd10 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7f63c802c4e0 tx=0x7f63c8004a40 comp rx=0 tx=0).stop 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 shutdown_connections 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f63a806c4c0 0x7f63a806e980 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:27.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63d015b320 0x7f63d006cd10 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:27.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63d00ce5a0 0x7f63d0151440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:27.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 --2- 172.21.15.53:0/124592819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63d00ccbc0 0x7f63d009c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:27.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 >> 172.21.15.53:0/124592819 conn(0x7f63d00b2a60 msgr2=0x7f63d015a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:27.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 shutdown_connections 2024-01-16T15:20:27.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:27.181+0000 7f63d7200640 1 -- 172.21.15.53:0/124592819 wait complete. 2024-01-16T15:20:27.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:20:27.193 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710) 2024-01-16T15:20:27.252 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710 2024-01-16T15:20:27.252 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3138256896" 2024-01-16T15:20:27.252 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:20:27.253 INFO:tasks.cephfs.test_volumes:r_size = 3138256896 2024-01-16T15:20:29.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:20:29.254 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr -n ceph.dir.rbytes volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710) 2024-01-16T15:20:29.324 INFO:teuthology.orchestra.run.smithi053.stdout:# file: volumes/_nogroup/sv1/da5803fc-3129-4aaa-a66c-5adbba715710 2024-01-16T15:20:29.324 INFO:teuthology.orchestra.run.smithi053.stdout:ceph.dir.rbytes="3221225472" 2024-01-16T15:20:29.324 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:20:29.325 INFO:tasks.cephfs.test_volumes:r_size = 3221225472 2024-01-16T15:20:29.326 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52241 2024-01-16T15:20:29.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.570+0000 7f282f400640 1 Processor -- start 2024-01-16T15:20:29.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.571+0000 7f282f400640 1 -- start start 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.572+0000 7f282f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f282813e360 0x7f28280cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.572+0000 7f282f400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282815c5f0 con 0x7f282815f9c0 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.572+0000 7f282f400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f282806c470 con 0x7f282813df80 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.572+0000 7f282f400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f282806c750 con 0x7f282813e360 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f282d400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f282815f9c0 0x7f28280cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54262/0 (socket says 172.21.15.53:54262) 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f2827e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f282813e360 0x7f28280cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:29.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f282d400640 1 -- 172.21.15.53:0/3628442592 learned_addr learned my addr 172.21.15.53:0/3628442592 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:29.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f2827e00640 1 -- 172.21.15.53:0/3628442592 >> v1:172.21.15.146:6789/0 conn(0x7f282813df80 legacy=0x7f282815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:29.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f2827e00640 1 -- 172.21.15.53:0/3628442592 >> v1:172.21.15.53:6789/0 conn(0x7f282815f9c0 legacy=0x7f28280cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:29.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f2827e00640 1 -- 172.21.15.53:0/3628442592 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f282806ca30 con 0x7f282813e360 2024-01-16T15:20:29.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.573+0000 7f2827400640 1 -- 172.21.15.53:0/3628442592 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164499825 0 0) 0x7f282815c5f0 con 0x7f282815f9c0 2024-01-16T15:20:29.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.575+0000 7f2827e00640 1 --2- 172.21.15.53:0/3628442592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282813e360 0x7f28280cf290 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2810009d90 tx=0x7f281002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fcb96a77f17eb4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:29.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.575+0000 7f2827400640 1 -- 172.21.15.53:0/3628442592 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f281002cc60 con 0x7f282813e360 2024-01-16T15:20:29.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.575+0000 7f2827400640 1 -- 172.21.15.53:0/3628442592 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f281002cdc0 con 0x7f282813e360 2024-01-16T15:20:29.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.575+0000 7f2827400640 1 -- 172.21.15.53:0/3628442592 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2810034660 con 0x7f282813e360 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.575+0000 7f282f400640 1 -- 172.21.15.53:0/3628442592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282813e360 msgr2=0x7f28280cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.575+0000 7f282f400640 1 --2- 172.21.15.53:0/3628442592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282813e360 0x7f28280cf290 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7f2810009d90 tx=0x7f281002beb0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.576+0000 7f282f400640 1 -- 172.21.15.53:0/3628442592 shutdown_connections 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.576+0000 7f282f400640 1 --2- 172.21.15.53:0/3628442592 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282813e360 0x7f28280cf290 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.576+0000 7f282f400640 1 -- 172.21.15.53:0/3628442592 >> 172.21.15.53:0/3628442592 conn(0x7f282806a800 msgr2=0x7f282806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.576+0000 7f282f400640 1 -- 172.21.15.53:0/3628442592 shutdown_connections 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.576+0000 7f282f400640 1 -- 172.21.15.53:0/3628442592 wait complete. 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.577+0000 7f282f400640 1 Processor -- start 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 -- start start 2024-01-16T15:20:29.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f282813df80 0x7f282813cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f282813e360 0x7f28280b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 0x7f28280b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2828151f00 con 0x7f282815f9c0 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28281521e0 con 0x7f282813df80 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.578+0000 7f282f400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28281524c0 con 0x7f282813e360 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 0x7f28280b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 0x7f28280b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47662/0 (socket says 172.21.15.53:47662) 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 -- 172.21.15.53:0/1513237972 learned_addr learned my addr 172.21.15.53:0/1513237972 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282ca00640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f282813df80 0x7f282813cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 -- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f282813e360 msgr2=0x7f28280b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f282813e360 0x7f28280b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 -- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f282813df80 msgr2=0x7f282813cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f282813df80 0x7f282813cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282d400640 1 -- 172.21.15.53:0/1513237972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2810009a70 con 0x7f282815f9c0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.579+0000 7f282ca00640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f282813df80 0x7f282813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.580+0000 7f282d400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 0x7f28280b3230 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f282000c450 tx=0x7f282000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.580+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820005ce0 con 0x7f282815f9c0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.580+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2820005e60 con 0x7f282815f9c0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.580+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2820007310 con 0x7f282815f9c0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.580+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28281527a0 con 0x7f282815f9c0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.580+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f282813d1c0 con 0x7f282815f9c0 2024-01-16T15:20:29.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.581+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2828152cc0 con 0x7f282815f9c0 2024-01-16T15:20:29.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.582+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 43) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2820007470 con 0x7f282815f9c0 2024-01-16T15:20:29.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.583+0000 7f2824c00640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f280006c4c0 0x7f280006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:29.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.583+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2820088160 con 0x7f282815f9c0 2024-01-16T15:20:29.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.584+0000 7f282ca00640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f280006c4c0 0x7f280006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:29.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.584+0000 7f282ca00640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f280006c4c0 0x7f280006e980 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f281800a830 tx=0x7f2818008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:29.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.588+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f28200546b0 con 0x7f282815f9c0 2024-01-16T15:20:29.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.820+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52241", "target": ["mon-mgr", ""]}) v1 -- 0x7f2828152350 con 0x7f280006c4c0 2024-01-16T15:20:29.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.975+0000 7f2824c00640 1 -- 172.21.15.53:0/1513237972 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2828152350 con 0x7f280006c4c0 2024-01-16T15:20:29.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f280006c4c0 msgr2=0x7f280006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:29.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f280006c4c0 0x7f280006e980 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f281800a830 tx=0x7f2818008040 comp rx=0 tx=0).stop 2024-01-16T15:20:29.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 msgr2=0x7f28280b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:29.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 0x7f28280b3230 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f282000c450 tx=0x7f282000c910 comp rx=0 tx=0).stop 2024-01-16T15:20:29.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 shutdown_connections 2024-01-16T15:20:29.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f280006c4c0 0x7f280006e980 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f282815f9c0 0x7f28280b3230 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f282813e360 0x7f28280b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 --2- 172.21.15.53:0/1513237972 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f282813df80 0x7f282813cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:29.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 >> 172.21.15.53:0/1513237972 conn(0x7f282806a800 msgr2=0x7f2828070660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:29.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 shutdown_connections 2024-01-16T15:20:29.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:29.979+0000 7f282f400640 1 -- 172.21.15.53:0/1513237972 wait complete. 2024-01-16T15:20:29.975 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:29.982+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:29.975 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:29.982+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:29.976 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:29.982+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:29.976 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:29.982+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:29.976 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:29.982+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:30.000 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52242 2024-01-16T15:20:30.008 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:30.014+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:30.009 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:30.014+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:30.009 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:30.014+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:30.009 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:30.014+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:30.009 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:20:30.014+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:20:30.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.228+0000 7f6c09c00640 1 Processor -- start 2024-01-16T15:20:30.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.229+0000 7f6c09c00640 1 -- start start 2024-01-16T15:20:30.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.230+0000 7f6c09c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6c040d0100 0x7f6c040b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.230+0000 7f6c09c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c040d0f40 0x7f6c040d1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.230+0000 7f6c09c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c040d0a10 con 0x7f6c04160920 2024-01-16T15:20:30.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.230+0000 7f6c09c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040cf080 con 0x7f6c040d0f40 2024-01-16T15:20:30.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.230+0000 7f6c09c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c040c6be0 con 0x7f6c040d0100 2024-01-16T15:20:30.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c040d0f40 0x7f6c040d1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6c040d0f40 0x7f6c040d1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47676/0 (socket says 172.21.15.53:47676) 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 -- 172.21.15.53:0/2988388212 learned_addr learned my addr 172.21.15.53:0/2988388212 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c03400640 1 --2- 172.21.15.53:0/2988388212 >> v2:172.21.15.146:3300/0 conn(0x7f6c040d0100 0x7f6c040b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 -- 172.21.15.53:0/2988388212 >> v2:172.21.15.146:3300/0 conn(0x7f6c040d0100 msgr2=0x7f6c040b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 --2- 172.21.15.53:0/2988388212 >> v2:172.21.15.146:3300/0 conn(0x7f6c040d0100 0x7f6c040b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 -- 172.21.15.53:0/2988388212 >> v1:172.21.15.146:6789/0 conn(0x7f6c04160920 legacy=0x7f6c040cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.231+0000 7f6c09200640 1 -- 172.21.15.53:0/2988388212 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c040ce7e0 con 0x7f6c040d0f40 2024-01-16T15:20:30.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.232+0000 7f6c09200640 1 --2- 172.21.15.53:0/2988388212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c040d0f40 0x7f6c040d1390 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6c0408c1f0 tx=0x7f6bfc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc84f6aca8acf5a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:30.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.233+0000 7f6c02a00640 1 -- 172.21.15.53:0/2988388212 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc031870 con 0x7f6c040d0f40 2024-01-16T15:20:30.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.233+0000 7f6c02a00640 1 -- 172.21.15.53:0/2988388212 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc0319d0 con 0x7f6c040d0f40 2024-01-16T15:20:30.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.233+0000 7f6c02a00640 1 -- 172.21.15.53:0/2988388212 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc031d60 con 0x7f6c040d0f40 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 -- 172.21.15.53:0/2988388212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c040d0f40 msgr2=0x7f6c040d1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 --2- 172.21.15.53:0/2988388212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c040d0f40 0x7f6c040d1390 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f6c0408c1f0 tx=0x7f6bfc02b910 comp rx=0 tx=0).stop 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 -- 172.21.15.53:0/2988388212 shutdown_connections 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 --2- 172.21.15.53:0/2988388212 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c040d0f40 0x7f6c040d1390 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 --2- 172.21.15.53:0/2988388212 >> v2:172.21.15.146:3300/0 conn(0x7f6c040d0100 0x7f6c040b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 -- 172.21.15.53:0/2988388212 >> 172.21.15.53:0/2988388212 conn(0x7f6c040d3090 msgr2=0x7f6c040d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 -- 172.21.15.53:0/2988388212 shutdown_connections 2024-01-16T15:20:30.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.234+0000 7f6c09c00640 1 -- 172.21.15.53:0/2988388212 wait complete. 2024-01-16T15:20:30.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.235+0000 7f6c09c00640 1 Processor -- start 2024-01-16T15:20:30.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.236+0000 7f6c09c00640 1 -- start start 2024-01-16T15:20:30.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c040d0100 0x7f6c0415b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c040d0f40 0x7f6c040ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 0x7f6c040b6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040b5c50 con 0x7f6c04160920 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040b5f30 con 0x7f6c040d0f40 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c040b6210 con 0x7f6c040d0100 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 0x7f6c040b6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 0x7f6c040b6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47692/0 (socket says 172.21.15.53:47692) 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c09200640 1 -- 172.21.15.53:0/1561771850 learned_addr learned my addr 172.21.15.53:0/1561771850 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c03400640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c040d0f40 0x7f6c040ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.237+0000 7f6c03e00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c040d0100 0x7f6c0415b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09200640 1 -- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c040d0100 msgr2=0x7f6c0415b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09200640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c040d0100 0x7f6c0415b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09200640 1 -- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c040d0f40 msgr2=0x7f6c040ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09200640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c040d0f40 0x7f6c040ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09200640 1 -- 172.21.15.53:0/1561771850 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bfc0094d0 con 0x7f6c04160920 2024-01-16T15:20:30.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09200640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 0x7f6c040b6cf0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0090b0 tx=0x7f6bfc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:30.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc03ac60 con 0x7f6c04160920 2024-01-16T15:20:30.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bfc03adc0 con 0x7f6c04160920 2024-01-16T15:20:30.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bfc044630 con 0x7f6c04160920 2024-01-16T15:20:30.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.238+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c040b64f0 con 0x7f6c04160920 2024-01-16T15:20:30.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.239+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0415baf0 con 0x7f6c04160920 2024-01-16T15:20:30.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.240+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c040b6a10 con 0x7f6c04160920 2024-01-16T15:20:30.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.241+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 43) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f6bfc04d450 con 0x7f6c04160920 2024-01-16T15:20:30.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.241+0000 7f6bfbe00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6bdc06c4c0 0x7f6bdc06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.242+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6bfc02d040 con 0x7f6c04160920 2024-01-16T15:20:30.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.242+0000 7f6c03e00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6bdc06c4c0 0x7f6bdc06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.244+0000 7f6c03e00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6bdc06c4c0 0x7f6bdc06e980 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009f20 tx=0x7f6bf40094d0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:30.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.248+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6c040b6a10 con 0x7f6c04160920 2024-01-16T15:20:30.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.480+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52242", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c04160d00 con 0x7f6bdc06c4c0 2024-01-16T15:20:30.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.635+0000 7f6bfbe00640 1 -- 172.21.15.53:0/1561771850 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6bf4009680 con 0x7f6bdc06c4c0 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6bdc06c4c0 msgr2=0x7f6bdc06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6bdc06c4c0 0x7f6bdc06e980 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f6bf4009f20 tx=0x7f6bf40094d0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 msgr2=0x7f6c040b6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 0x7f6c040b6cf0 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f6bfc0090b0 tx=0x7f6bfc0046c0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 shutdown_connections 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6bdc06c4c0 0x7f6bdc06e980 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6c04160920 0x7f6c040b6cf0 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6c040d0f40 0x7f6c040ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 --2- 172.21.15.53:0/1561771850 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6c040d0100 0x7f6c0415b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.639+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 >> 172.21.15.53:0/1561771850 conn(0x7f6c040d3090 msgr2=0x7f6c0415d580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.640+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 shutdown_connections 2024-01-16T15:20:30.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.640+0000 7f6c09c00640 1 -- 172.21.15.53:0/1561771850 wait complete. 2024-01-16T15:20:30.656 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52243 2024-01-16T15:20:30.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.885+0000 7f202d000640 1 Processor -- start 2024-01-16T15:20:30.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.885+0000 7f202d000640 1 -- start start 2024-01-16T15:20:30.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.887+0000 7f202d000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20280cc500 0x7f20280cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.887+0000 7f202d000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2028070e50 con 0x7f2028143c40 2024-01-16T15:20:30.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.887+0000 7f202d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2028144020 con 0x7f202806d160 2024-01-16T15:20:30.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.887+0000 7f202d000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20280723c0 con 0x7f20280cc500 2024-01-16T15:20:30.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.888+0000 7f2027400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2028143c40 0x7f2028071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:54272/0 (socket says 172.21.15.53:54272) 2024-01-16T15:20:30.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.888+0000 7f2027400640 1 -- 172.21.15.53:0/338711580 learned_addr learned my addr 172.21.15.53:0/338711580 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.888+0000 7f201fe00640 1 --2- 172.21.15.53:0/338711580 >> v2:172.21.15.146:3300/0 conn(0x7f20280cc500 0x7f20280cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.888+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4004832134 0 0) 0x7f2028070e50 con 0x7f2028143c40 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.888+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2008003520 con 0x7f2028143c40 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420446066 0 0) 0x7f2008003520 con 0x7f2028143c40 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 >> v2:172.21.15.146:3300/0 conn(0x7f20280cc500 msgr2=0x7f20280cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 --2- 172.21.15.53:0/338711580 >> v2:172.21.15.146:3300/0 conn(0x7f20280cc500 0x7f20280cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 >> v1:172.21.15.146:6789/0 conn(0x7f202806d160 legacy=0x7f202806d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2028071e00 con 0x7f2028143c40 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2018001670 con 0x7f2028143c40 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 >> v1:172.21.15.53:6789/0 conn(0x7f2028143c40 legacy=0x7f2028071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008003970 0x7f2008003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20080047c0 0x7f2008004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2008005180 0x7f200800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20280723c0 con 0x7f2008005180 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f200800c120 con 0x7f2008003970 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f200800c2e0 con 0x7f20080047c0 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2027e00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2008005180 0x7f200800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2027400640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008003970 0x7f2008003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f201fe00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20080047c0 0x7f2008004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.889+0000 7f2027e00640 1 -- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20080047c0 msgr2=0x7f2008004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.890+0000 7f2027e00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20080047c0 0x7f2008004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.890+0000 7f2027e00640 1 -- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008003970 msgr2=0x7f2008003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:30.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.890+0000 7f2027e00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008003970 0x7f2008003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.890+0000 7f2027e00640 1 -- 172.21.15.53:0/338711580 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008003520 con 0x7f2008005180 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.890+0000 7f2027e00640 1 -- 172.21.15.53:0/338711580 reap_dead start 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.891+0000 7f2027e00640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2008005180 0x7f200800b980 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f2020001e50 tx=0x7f20080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75f73a739c61c122 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.891+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020003b70 con 0x7f2008005180 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.891+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2020003d10 con 0x7f2008005180 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.891+0000 7f2026a00640 1 -- 172.21.15.53:0/338711580 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2020005310 con 0x7f2008005180 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.891+0000 7f202d000640 1 -- 172.21.15.53:0/338711580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2008005180 msgr2=0x7f200800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.891+0000 7f202d000640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2008005180 0x7f200800b980 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f2020001e50 tx=0x7f20080057f0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.892+0000 7f202d000640 1 -- 172.21.15.53:0/338711580 shutdown_connections 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.892+0000 7f202d000640 1 --2- 172.21.15.53:0/338711580 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2008005180 0x7f200800b980 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.892+0000 7f202d000640 1 -- 172.21.15.53:0/338711580 >> 172.21.15.53:0/338711580 conn(0x7f202806a800 msgr2=0x7f202806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:30.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.892+0000 7f202d000640 1 -- 172.21.15.53:0/338711580 shutdown_connections 2024-01-16T15:20:30.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.892+0000 7f202d000640 1 -- 172.21.15.53:0/338711580 wait complete. 2024-01-16T15:20:30.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.893+0000 7f202d000640 1 Processor -- start 2024-01-16T15:20:30.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.894+0000 7f202d000640 1 -- start start 2024-01-16T15:20:30.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f202d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008005180 0x7f202815a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.887 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f202d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 0x7f20280c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f202d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2028142790 0x7f2028142c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f202d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2028143160 con 0x7f20280c46f0 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f202d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f202813eff0 con 0x7f2008005180 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f202d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f202813f2d0 con 0x7f2028142790 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 0x7f20280c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 0x7f20280c4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47714/0 (socket says 172.21.15.53:47714) 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 -- 172.21.15.53:0/4206092106 learned_addr learned my addr 172.21.15.53:0/4206092106 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 -- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2028142790 msgr2=0x7f2028142c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2028142790 0x7f2028142c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 -- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008005180 msgr2=0x7f202815a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f2027400640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008005180 0x7f202815a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008005180 0x7f202815a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:30.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.895+0000 7f201fe00640 1 -- 172.21.15.53:0/4206092106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f202813f5b0 con 0x7f20280c46f0 2024-01-16T15:20:30.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.896+0000 7f201fe00640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 0x7f20280c4b50 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f2010010860 tx=0x7f2010010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:30.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.896+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2010009610 con 0x7f20280c46f0 2024-01-16T15:20:30.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.896+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f201000eb60 con 0x7f20280c46f0 2024-01-16T15:20:30.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.896+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f201001c400 con 0x7f20280c46f0 2024-01-16T15:20:30.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.896+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f202815d6d0 con 0x7f20280c46f0 2024-01-16T15:20:30.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.896+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20280cb470 con 0x7f20280c46f0 2024-01-16T15:20:30.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.897+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2028067450 con 0x7f20280c46f0 2024-01-16T15:20:30.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.898+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 43) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f201001c560 con 0x7f20280c46f0 2024-01-16T15:20:30.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.900+0000 7f201f400640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f200006c4c0 0x7f200006e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:30.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.900+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f20100890f0 con 0x7f20280c46f0 2024-01-16T15:20:30.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.900+0000 7f2027400640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f200006c4c0 0x7f200006e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:30.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.901+0000 7f2027400640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f200006c4c0 0x7f200006e980 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2018001200 tx=0x7f2018001150 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:30.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:30.904+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20100556f0 con 0x7f20280c46f0 2024-01-16T15:20:31.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:31.133+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52243", "target": ["mon-mgr", ""]}) v1 -- 0x7f202813ede0 con 0x7f200006c4c0 2024-01-16T15:20:31.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:31.570+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2010055ac0 con 0x7f20280c46f0 2024-01-16T15:20:34.428 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.435+0000 7f201f400640 1 -- 172.21.15.53:0/4206092106 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f202813ede0 con 0x7f200006c4c0 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f200006c4c0 msgr2=0x7f200006e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f200006c4c0 0x7f200006e980 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2018001200 tx=0x7f2018001150 comp rx=0 tx=0).stop 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 msgr2=0x7f20280c4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 0x7f20280c4b50 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f2010010860 tx=0x7f2010010d20 comp rx=0 tx=0).stop 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 shutdown_connections 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f200006c4c0 0x7f200006e980 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2028142790 0x7f2028142c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20280c46f0 0x7f20280c4b50 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 --2- 172.21.15.53:0/4206092106 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2008005180 0x7f202815a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.439+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 >> 172.21.15.53:0/4206092106 conn(0x7f202806a800 msgr2=0x7f202806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:34.432 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.440+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 shutdown_connections 2024-01-16T15:20:34.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.440+0000 7f202d000640 1 -- 172.21.15.53:0/4206092106 wait complete. 2024-01-16T15:20:34.454 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52244 2024-01-16T15:20:34.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.670+0000 7f49dd200640 1 Processor -- start 2024-01-16T15:20:34.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.670+0000 7f49dd200640 1 -- start start 2024-01-16T15:20:34.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.671+0000 7f49dd200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f49d815d3c0 0x7f49d80d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:34.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.671+0000 7f49dd200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49d80cfa40 0x7f49d80cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:34.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49dd200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d80b2c70 con 0x7f49d80cde90 2024-01-16T15:20:34.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49dd200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d80d03f0 con 0x7f49d80cfa40 2024-01-16T15:20:34.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49dd200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d80cf6d0 con 0x7f49d815d3c0 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49cfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f49d80cfa40 0x7f49d80cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49d7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f49d80cde90 0x7f49d80ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59438/0 (socket says 172.21.15.53:59438) 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49d7e00640 1 -- 172.21.15.53:0/2376610105 learned_addr learned my addr 172.21.15.53:0/2376610105 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.672+0000 7f49d7400640 1 --2- 172.21.15.53:0/2376610105 >> v2:172.21.15.146:3300/0 conn(0x7f49d815d3c0 0x7f49d80d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.673+0000 7f49cfe00640 1 -- 172.21.15.53:0/2376610105 >> v2:172.21.15.146:3300/0 conn(0x7f49d815d3c0 msgr2=0x7f49d80d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.673+0000 7f49cfe00640 1 --2- 172.21.15.53:0/2376610105 >> v2:172.21.15.146:3300/0 conn(0x7f49d815d3c0 0x7f49d80d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.673+0000 7f49cfe00640 1 -- 172.21.15.53:0/2376610105 >> v1:172.21.15.53:6789/0 conn(0x7f49d80cde90 legacy=0x7f49d80ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.673+0000 7f49cfe00640 1 -- 172.21.15.53:0/2376610105 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80cf3f0 con 0x7f49d80cfa40 2024-01-16T15:20:34.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.674+0000 7f49cfe00640 1 --2- 172.21.15.53:0/2376610105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d80cfeb0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f49d808b6c0 tx=0x7f49c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9f0da19ff59de1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:34.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.674+0000 7f49d6a00640 1 -- 172.21.15.53:0/2376610105 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0031830 con 0x7f49d80cfa40 2024-01-16T15:20:34.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.674+0000 7f49d6a00640 1 -- 172.21.15.53:0/2376610105 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c0031990 con 0x7f49d80cfa40 2024-01-16T15:20:34.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.674+0000 7f49d6a00640 1 -- 172.21.15.53:0/2376610105 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c0031d20 con 0x7f49d80cfa40 2024-01-16T15:20:34.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 -- 172.21.15.53:0/2376610105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 msgr2=0x7f49d80cfeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 --2- 172.21.15.53:0/2376610105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d80cfeb0 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f49d808b6c0 tx=0x7f49c002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 -- 172.21.15.53:0/2376610105 shutdown_connections 2024-01-16T15:20:34.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 --2- 172.21.15.53:0/2376610105 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d80cfeb0 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 --2- 172.21.15.53:0/2376610105 >> v2:172.21.15.146:3300/0 conn(0x7f49d815d3c0 0x7f49d80d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 -- 172.21.15.53:0/2376610105 >> 172.21.15.53:0/2376610105 conn(0x7f49d80d3050 msgr2=0x7f49d80d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:34.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 -- 172.21.15.53:0/2376610105 shutdown_connections 2024-01-16T15:20:34.668 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.675+0000 7f49dd200640 1 -- 172.21.15.53:0/2376610105 wait complete. 2024-01-16T15:20:34.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.678+0000 7f49dd200640 1 Processor -- start 2024-01-16T15:20:34.670 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.678+0000 7f49dd200640 1 -- start start 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49dd200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49d80cde90 0x7f49d8156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49dd200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d815df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49dd200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49d815d3c0 0x7f49d815e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49dd200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80caee0 con 0x7f49d80cfa40 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49dd200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80b8580 con 0x7f49d80cde90 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49dd200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d80cb130 con 0x7f49d815d3c0 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49cfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d815df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:34.672 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49cfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d815df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42952/0 (socket says 172.21.15.53:42952) 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.679+0000 7f49cfe00640 1 -- 172.21.15.53:0/1163983902 learned_addr learned my addr 172.21.15.53:0/1163983902 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49d7400640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49d80cde90 0x7f49d8156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49d7e00640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49d815d3c0 0x7f49d815e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49cfe00640 1 -- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49d815d3c0 msgr2=0x7f49d815e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49cfe00640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49d815d3c0 0x7f49d815e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49cfe00640 1 -- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49d80cde90 msgr2=0x7f49d8156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49cfe00640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49d80cde90 0x7f49d8156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:34.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49cfe00640 1 -- 172.21.15.53:0/1163983902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c0009400 con 0x7f49d80cfa40 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.680+0000 7f49cfe00640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d815df70 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009890 tx=0x7f49c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.681+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c003a6f0 con 0x7f49d80cfa40 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.681+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c003a870 con 0x7f49d80cfa40 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.681+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d80cb410 con 0x7f49d80cfa40 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.681+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49c003ac20 con 0x7f49d80cfa40 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.681+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d81569d0 con 0x7f49d80cfa40 2024-01-16T15:20:34.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.682+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d808ba80 con 0x7f49d80cfa40 2024-01-16T15:20:34.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.683+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f49c0043410 con 0x7f49d80cfa40 2024-01-16T15:20:34.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.684+0000 7f49cf400640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b006c5e0 0x7f49b006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:34.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.685+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f49c00aa380 con 0x7f49d80cfa40 2024-01-16T15:20:34.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.685+0000 7f49d7400640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b006c5e0 0x7f49b006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:34.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.686+0000 7f49d7400640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b006c5e0 0x7f49b006eaa0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009a30 tx=0x7f49c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:34.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.690+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f49c00e0e90 con 0x7f49d80cfa40 2024-01-16T15:20:34.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:34.919+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52244", "target": ["mon-mgr", ""]}) v1 -- 0x7f49d80b2900 con 0x7f49b006c5e0 2024-01-16T15:20:39.435 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.443+0000 7f49cf400640 1 -- 172.21.15.53:0/1163983902 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f49c8009680 con 0x7f49b006c5e0 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b006c5e0 msgr2=0x7f49b006eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b006c5e0 0x7f49b006eaa0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f49c8009a30 tx=0x7f49c8008040 comp rx=0 tx=0).stop 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 msgr2=0x7f49d815df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d815df70 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009890 tx=0x7f49c00046c0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 shutdown_connections 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f49b006c5e0 0x7f49b006eaa0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f49d815d3c0 0x7f49d815e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f49d80cfa40 0x7f49d815df70 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.446+0000 7f49dd200640 1 --2- 172.21.15.53:0/1163983902 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f49d80cde90 0x7f49d8156490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.439 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.447+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 >> 172.21.15.53:0/1163983902 conn(0x7f49d80d3050 msgr2=0x7f49d80b1d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:39.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.447+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 shutdown_connections 2024-01-16T15:20:39.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.447+0000 7f49dd200640 1 -- 172.21.15.53:0/1163983902 wait complete. 2024-01-16T15:20:39.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52245 2024-01-16T15:20:39.695 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.703+0000 7fc715400640 1 Processor -- start 2024-01-16T15:20:39.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.703+0000 7fc715400640 1 -- start start 2024-01-16T15:20:39.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc715400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc710162430 0x7fc7100b3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:39.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc715400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc71006c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:39.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc715400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7100702e0 con 0x7fc710162810 2024-01-16T15:20:39.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc715400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7100705c0 con 0x7fc710162430 2024-01-16T15:20:39.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc715400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100708a0 con 0x7fc7100b3650 2024-01-16T15:20:39.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc70fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc710162430 0x7fc7100b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:39.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc70fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fc710162430 0x7fc7100b3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42962/0 (socket says 172.21.15.53:42962) 2024-01-16T15:20:39.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.705+0000 7fc70fe00640 1 -- 172.21.15.53:0/3850905756 learned_addr learned my addr 172.21.15.53:0/3850905756 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.706+0000 7fc70fe00640 1 -- 172.21.15.53:0/3850905756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 msgr2=0x7fc71006c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.706+0000 7fc70fe00640 1 --2- 172.21.15.53:0/3850905756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc71006c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.706+0000 7fc70fe00640 1 -- 172.21.15.53:0/3850905756 >> v1:172.21.15.146:6789/0 conn(0x7fc710162810 legacy=0x7fc710071240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.706+0000 7fc70fe00640 1 -- 172.21.15.53:0/3850905756 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc710070b80 con 0x7fc710162430 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.707+0000 7fc70fe00640 1 --2- 172.21.15.53:0/3850905756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710162430 0x7fc7100b3110 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fc700009870 tx=0x7fc70002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654fa10008693a2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.707+0000 7fc70f400640 1 -- 172.21.15.53:0/3850905756 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700031640 con 0x7fc710162430 2024-01-16T15:20:39.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.707+0000 7fc70f400640 1 -- 172.21.15.53:0/3850905756 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7000317a0 con 0x7fc710162430 2024-01-16T15:20:39.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.707+0000 7fc70f400640 1 -- 172.21.15.53:0/3850905756 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc700031b30 con 0x7fc710162430 2024-01-16T15:20:39.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 -- 172.21.15.53:0/3850905756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710162430 msgr2=0x7fc7100b3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:39.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 --2- 172.21.15.53:0/3850905756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710162430 0x7fc7100b3110 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fc700009870 tx=0x7fc70002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 -- 172.21.15.53:0/3850905756 shutdown_connections 2024-01-16T15:20:39.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 --2- 172.21.15.53:0/3850905756 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc71006c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 --2- 172.21.15.53:0/3850905756 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc710162430 0x7fc7100b3110 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 -- 172.21.15.53:0/3850905756 >> 172.21.15.53:0/3850905756 conn(0x7fc71006a800 msgr2=0x7fc71006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:39.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 -- 172.21.15.53:0/3850905756 shutdown_connections 2024-01-16T15:20:39.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.708+0000 7fc715400640 1 -- 172.21.15.53:0/3850905756 wait complete. 2024-01-16T15:20:39.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.710+0000 7fc715400640 1 Processor -- start 2024-01-16T15:20:39.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.710+0000 7fc715400640 1 -- start start 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc715400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc7101571c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc715400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc710162430 0x7fc710147890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc715400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 0x7fc71015c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc715400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015ca80 con 0x7fc71015c0c0 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc715400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015fc10 con 0x7fc710162430 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc715400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015f000 con 0x7fc7100b3650 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc714a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 0x7fc71015c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc714a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 0x7fc71015c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42966/0 (socket says 172.21.15.53:42966) 2024-01-16T15:20:39.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.711+0000 7fc714a00640 1 -- 172.21.15.53:0/3509987285 learned_addr learned my addr 172.21.15.53:0/3509987285 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:39.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc70fe00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc7101571c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:39.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc714a00640 1 -- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 msgr2=0x7fc7101571c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:39.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc707e00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc710162430 0x7fc710147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc714a00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc7101571c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc714a00640 1 -- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc710162430 msgr2=0x7fc710147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc714a00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc710162430 0x7fc710147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc714a00640 1 -- 172.21.15.53:0/3509987285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000094d0 con 0x7fc71015c0c0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc707e00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc710162430 0x7fc710147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.712+0000 7fc714a00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 0x7fc71015c540 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc70800b9e0 tx=0x7fc70800bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.713+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70800cb20 con 0x7fc71015c0c0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.713+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70800cc80 con 0x7fc71015c0c0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.713+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc708014680 con 0x7fc71015c0c0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.713+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc71015f2e0 con 0x7fc71015c0c0 2024-01-16T15:20:39.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.713+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71015f8a0 con 0x7fc71015c0c0 2024-01-16T15:20:39.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.714+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc710067450 con 0x7fc71015c0c0 2024-01-16T15:20:39.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.715+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fc70801d3b0 con 0x7fc71015c0c0 2024-01-16T15:20:39.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.716+0000 7fc70cc00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6e806c5e0 0x7fc6e806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:39.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.717+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc708088b60 con 0x7fc71015c0c0 2024-01-16T15:20:39.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.717+0000 7fc70fe00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6e806c5e0 0x7fc6e806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:39.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.718+0000 7fc70fe00640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6e806c5e0 0x7fc6e806eaa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc700036000 tx=0x7fc70003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:39.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.721+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc708055080 con 0x7fc71015c0c0 2024-01-16T15:20:39.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:39.949+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52245", "target": ["mon-mgr", ""]}) v1 -- 0x7fc7100b2ef0 con 0x7fc6e806c5e0 2024-01-16T15:20:44.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.434+0000 7fc70cc00640 1 -- 172.21.15.53:0/3509987285 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc7100b2ef0 con 0x7fc6e806c5e0 2024-01-16T15:20:44.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.437+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6e806c5e0 msgr2=0x7fc6e806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:44.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.437+0000 7fc715400640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6e806c5e0 0x7fc6e806eaa0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc700036000 tx=0x7fc70003c040 comp rx=0 tx=0).stop 2024-01-16T15:20:44.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.437+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 msgr2=0x7fc71015c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:44.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.437+0000 7fc715400640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 0x7fc71015c540 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fc70800b9e0 tx=0x7fc70800bea0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 shutdown_connections 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc6e806c5e0 0x7fc6e806eaa0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc71015c0c0 0x7fc71015c540 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc710162430 0x7fc710147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 --2- 172.21.15.53:0/3509987285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc7100b3650 0x7fc7101571c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 >> 172.21.15.53:0/3509987285 conn(0x7fc71006a800 msgr2=0x7fc7100b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 shutdown_connections 2024-01-16T15:20:44.431 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.438+0000 7fc715400640 1 -- 172.21.15.53:0/3509987285 wait complete. 2024-01-16T15:20:44.452 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52246 2024-01-16T15:20:44.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.694+0000 7f18ac400640 1 Processor -- start 2024-01-16T15:20:44.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.694+0000 7f18ac400640 1 -- start start 2024-01-16T15:20:44.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.696+0000 7f18ac400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18a413e360 0x7f18a40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:44.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.696+0000 7f18ac400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a415f9c0 0x7f18a40cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.696+0000 7f18ac400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18a415c5f0 con 0x7f18a413df80 2024-01-16T15:20:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.696+0000 7f18ac400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18a406c470 con 0x7f18a413e360 2024-01-16T15:20:44.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.696+0000 7f18ac400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a406c750 con 0x7f18a415f9c0 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.696+0000 7f18a9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18a413e360 0x7f18a40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f18a413e360 0x7f18a40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50072/0 (socket says 172.21.15.53:50072) 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a9000640 1 -- 172.21.15.53:0/761651032 learned_addr learned my addr 172.21.15.53:0/761651032 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18aa400640 1 --2- 172.21.15.53:0/761651032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a415f9c0 0x7f18a40cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a9000640 1 -- 172.21.15.53:0/761651032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a415f9c0 msgr2=0x7f18a40cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a9000640 1 --2- 172.21.15.53:0/761651032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a415f9c0 0x7f18a40cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a9000640 1 -- 172.21.15.53:0/761651032 >> v1:172.21.15.53:6789/0 conn(0x7f18a413df80 legacy=0x7f18a415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a9000640 1 -- 172.21.15.53:0/761651032 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18a406ca30 con 0x7f18a413e360 2024-01-16T15:20:44.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.697+0000 7f18a3e00640 1 -- 172.21.15.53:0/761651032 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055001689 0 0) 0x7f18a415c5f0 con 0x7f18a413df80 2024-01-16T15:20:44.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.699+0000 7f18a9000640 1 --2- 172.21.15.53:0/761651032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a413e360 0x7f18a40cf290 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f188c009870 tx=0x7f188c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb234a47a73a248e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:44.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.699+0000 7f18a3e00640 1 -- 172.21.15.53:0/761651032 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c030840 con 0x7f18a413e360 2024-01-16T15:20:44.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.699+0000 7f18a3e00640 1 -- 172.21.15.53:0/761651032 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f188c0309a0 con 0x7f18a413e360 2024-01-16T15:20:44.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.699+0000 7f18a3e00640 1 -- 172.21.15.53:0/761651032 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f188c030d30 con 0x7f18a413e360 2024-01-16T15:20:44.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 -- 172.21.15.53:0/761651032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a413e360 msgr2=0x7f18a40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:44.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 --2- 172.21.15.53:0/761651032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a413e360 0x7f18a40cf290 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f188c009870 tx=0x7f188c02b980 comp rx=0 tx=0).stop 2024-01-16T15:20:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 -- 172.21.15.53:0/761651032 shutdown_connections 2024-01-16T15:20:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 --2- 172.21.15.53:0/761651032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a415f9c0 0x7f18a40cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 --2- 172.21.15.53:0/761651032 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a413e360 0x7f18a40cf290 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 -- 172.21.15.53:0/761651032 >> 172.21.15.53:0/761651032 conn(0x7f18a406a800 msgr2=0x7f18a406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 -- 172.21.15.53:0/761651032 shutdown_connections 2024-01-16T15:20:44.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.700+0000 7f18ac400640 1 -- 172.21.15.53:0/761651032 wait complete. 2024-01-16T15:20:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.702+0000 7f18ac400640 1 Processor -- start 2024-01-16T15:20:44.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.702+0000 7f18ac400640 1 -- start start 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.703+0000 7f18ac400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a413df80 0x7f18a415a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.703+0000 7f18ac400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18a413e360 0x7f18a415a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.703+0000 7f18ac400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 0x7f18a414ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.703+0000 7f18ac400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a414b7c0 con 0x7f18a415f9c0 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.703+0000 7f18ac400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18a414baa0 con 0x7f18a413e360 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.703+0000 7f18ac400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18a414bd80 con 0x7f18a413df80 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 0x7f18a414ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 0x7f18a414ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50086/0 (socket says 172.21.15.53:50086) 2024-01-16T15:20:44.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 -- 172.21.15.53:0/568639949 learned_addr learned my addr 172.21.15.53:0/568639949 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18a9a00640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a413df80 0x7f18a415a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18a9000640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18a413e360 0x7f18a415a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 -- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a413df80 msgr2=0x7f18a415a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a413df80 0x7f18a415a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 -- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18a413e360 msgr2=0x7f18a415a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18a413e360 0x7f18a415a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 -- 172.21.15.53:0/568639949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f188c0094d0 con 0x7f18a415f9c0 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18a9000640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18a413e360 0x7f18a415a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.704+0000 7f18aa400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 0x7f18a414ab00 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f189c007e30 tx=0x7f189c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.705+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c014620 con 0x7f18a415f9c0 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.705+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f189c00ee90 con 0x7f18a415f9c0 2024-01-16T15:20:44.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.705+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f189c0149c0 con 0x7f18a415f9c0 2024-01-16T15:20:44.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.705+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18a414c060 con 0x7f18a415f9c0 2024-01-16T15:20:44.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.705+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18a414c860 con 0x7f18a415f9c0 2024-01-16T15:20:44.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.706+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18a414deb0 con 0x7f18a415f9c0 2024-01-16T15:20:44.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.707+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f189c014b20 con 0x7f18a415f9c0 2024-01-16T15:20:44.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.708+0000 7f18a1600640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f187c06c5e0 0x7f187c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:44.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.708+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f189c0895a0 con 0x7f18a415f9c0 2024-01-16T15:20:44.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.709+0000 7f18a9a00640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f187c06c5e0 0x7f187c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:44.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.710+0000 7f18a9a00640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f187c06c5e0 0x7f187c06eaa0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1894007fd0 tx=0x7f1894009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:44.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.713+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f189c055ac0 con 0x7f18a415f9c0 2024-01-16T15:20:44.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:44.943+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52246", "target": ["mon-mgr", ""]}) v1 -- 0x7f18a414bc10 con 0x7f187c06c5e0 2024-01-16T15:20:45.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.104+0000 7f18a1600640 1 -- 172.21.15.53:0/568639949 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f18a414bc10 con 0x7f187c06c5e0 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f187c06c5e0 msgr2=0x7f187c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f187c06c5e0 0x7f187c06eaa0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f1894007fd0 tx=0x7f1894009040 comp rx=0 tx=0).stop 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 msgr2=0x7f18a414ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 0x7f18a414ab00 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f189c007e30 tx=0x7f189c00e4b0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 shutdown_connections 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f187c06c5e0 0x7f187c06eaa0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f18a415f9c0 0x7f18a414ab00 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f18a413e360 0x7f18a415a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 --2- 172.21.15.53:0/568639949 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f18a413df80 0x7f18a415a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 >> 172.21.15.53:0/568639949 conn(0x7f18a406a800 msgr2=0x7f18a40cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 shutdown_connections 2024-01-16T15:20:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.108+0000 7f18ac400640 1 -- 172.21.15.53:0/568639949 wait complete. 2024-01-16T15:20:45.121 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot clone cephfs sv1 ss1 subvol_clone52247 2024-01-16T15:20:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.360+0000 7fc588e00640 1 Processor -- start 2024-01-16T15:20:45.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.361+0000 7fc588e00640 1 -- start start 2024-01-16T15:20:45.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.362+0000 7fc588e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc5840cc500 0x7fc5840cc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc588e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc584070e50 con 0x7fc584143c40 2024-01-16T15:20:45.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc588e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc584144020 con 0x7fc58406d160 2024-01-16T15:20:45.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc588e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5840723c0 con 0x7fc5840cc500 2024-01-16T15:20:45.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc583400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fc584143c40 0x7fc584071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55342/0 (socket says 172.21.15.53:55342) 2024-01-16T15:20:45.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc583400640 1 -- 172.21.15.53:0/4161475564 learned_addr learned my addr 172.21.15.53:0/4161475564 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc582a00640 1 --2- 172.21.15.53:0/4161475564 >> v2:172.21.15.146:3300/0 conn(0x7fc5840cc500 0x7fc5840cc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2097558402 0 0) 0x7fc584070e50 con 0x7fc584143c40 2024-01-16T15:20:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.363+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc564003520 con 0x7fc584143c40 2024-01-16T15:20:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2603125292 0 0) 0x7fc564003520 con 0x7fc584143c40 2024-01-16T15:20:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 >> v2:172.21.15.146:3300/0 conn(0x7fc5840cc500 msgr2=0x7fc5840cc990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:45.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 --2- 172.21.15.53:0/4161475564 >> v2:172.21.15.146:3300/0 conn(0x7fc5840cc500 0x7fc5840cc990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 >> v1:172.21.15.146:6789/0 conn(0x7fc58406d160 legacy=0x7fc58406d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc584071e00 con 0x7fc584143c40 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fc574001670 con 0x7fc584143c40 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 >> v1:172.21.15.53:6789/0 conn(0x7fc584143c40 legacy=0x7fc584071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5640039c0 0x7fc564003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5640047c0 0x7fc564004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640051d0 0x7fc56400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5840723c0 con 0x7fc5640047c0 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400c120 con 0x7fc5640039c0 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56400c2e0 con 0x7fc5640051d0 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.364+0000 7fc582a00640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5640047c0 0x7fc564004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc583400640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5640039c0 0x7fc564003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc582a00640 1 -- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640051d0 msgr2=0x7fc56400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc582a00640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640051d0 0x7fc56400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc582a00640 1 -- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5640039c0 msgr2=0x7fc564003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:45.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc582a00640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5640039c0 0x7fc564003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc582a00640 1 -- 172.21.15.53:0/4161475564 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc564003520 con 0x7fc5640047c0 2024-01-16T15:20:45.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc583e00640 1 -- 172.21.15.53:0/4161475564 reap_dead start 2024-01-16T15:20:45.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.365+0000 7fc583400640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc5640039c0 0x7fc564003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.366+0000 7fc582a00640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5640047c0 0x7fc564004c40 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009a90 tx=0x7fc5640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=435dd89efefb46a9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.366+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c02c6b0 con 0x7fc5640047c0 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.366+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc56c02c810 con 0x7fc5640047c0 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.366+0000 7fc582000640 1 -- 172.21.15.53:0/4161475564 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56c02cba0 con 0x7fc5640047c0 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 -- 172.21.15.53:0/4161475564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5640047c0 msgr2=0x7fc564004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5640047c0 0x7fc564004c40 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7fc56c009a90 tx=0x7fc5640057f0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 -- 172.21.15.53:0/4161475564 shutdown_connections 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 --2- 172.21.15.53:0/4161475564 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc5640047c0 0x7fc564004c40 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 -- 172.21.15.53:0/4161475564 >> 172.21.15.53:0/4161475564 conn(0x7fc58406a800 msgr2=0x7fc58406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 -- 172.21.15.53:0/4161475564 shutdown_connections 2024-01-16T15:20:45.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.367+0000 7fc588e00640 1 -- 172.21.15.53:0/4161475564 wait complete. 2024-01-16T15:20:45.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.369+0000 7fc588e00640 1 Processor -- start 2024-01-16T15:20:45.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.369+0000 7fc588e00640 1 -- start start 2024-01-16T15:20:45.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc588e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640047c0 0x7fc58415a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc588e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc58415e380 0x7fc58415b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc588e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 0x7fc58415bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc588e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58415a570 con 0x7fc58415a730 2024-01-16T15:20:45.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc588e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58415f4e0 con 0x7fc58415e380 2024-01-16T15:20:45.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc588e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58415e8d0 con 0x7fc5640047c0 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.370+0000 7fc583e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 0x7fc58415bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 0x7fc58415bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50100/0 (socket says 172.21.15.53:50100) 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 -- 172.21.15.53:0/640240131 learned_addr learned my addr 172.21.15.53:0/640240131 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc582a00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc58415e380 0x7fc58415b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583400640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640047c0 0x7fc58415a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 -- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640047c0 msgr2=0x7fc58415a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640047c0 0x7fc58415a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 -- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc58415e380 msgr2=0x7fc58415b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc58415e380 0x7fc58415b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 -- 172.21.15.53:0/640240131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58415ebb0 con 0x7fc58415a730 2024-01-16T15:20:45.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc583e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 0x7fc58415bda0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc584144ca0 tx=0x7fc57c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:45.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.371+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c005d40 con 0x7fc58415a730 2024-01-16T15:20:45.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.372+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57c00ced0 con 0x7fc58415a730 2024-01-16T15:20:45.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.372+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc57c00a7f0 con 0x7fc58415a730 2024-01-16T15:20:45.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.372+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58415ee90 con 0x7fc58415a730 2024-01-16T15:20:45.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.372+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc584142710 con 0x7fc58415a730 2024-01-16T15:20:45.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.373+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc584067450 con 0x7fc58415a730 2024-01-16T15:20:45.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.374+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fc57c00a950 con 0x7fc58415a730 2024-01-16T15:20:45.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.375+0000 7fc57b400640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc55c06c5e0 0x7fc55c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:45.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.375+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc57c087a80 con 0x7fc58415a730 2024-01-16T15:20:45.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.376+0000 7fc583400640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc55c06c5e0 0x7fc55c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:45.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.377+0000 7fc583400640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc55c06c5e0 0x7fc55c06eaa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc574000c00 tx=0x7fc574009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:45.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.379+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc57c053fa0 con 0x7fc58415a730 2024-01-16T15:20:45.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.610+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot clone", "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target_sub_name": "subvol_clone52247", "target": ["mon-mgr", ""]}) v1 -- 0x7fc584142930 con 0x7fc55c06c5e0 2024-01-16T15:20:45.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.786+0000 7fc57b400640 1 -- 172.21.15.53:0/640240131 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc57400d070 con 0x7fc55c06c5e0 2024-01-16T15:20:45.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.789+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc55c06c5e0 msgr2=0x7fc55c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.789+0000 7fc588e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc55c06c5e0 0x7fc55c06eaa0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc574000c00 tx=0x7fc574009040 comp rx=0 tx=0).stop 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.789+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 msgr2=0x7fc58415bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 0x7fc58415bda0 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7fc584144ca0 tx=0x7fc57c00cb90 comp rx=0 tx=0).stop 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 shutdown_connections 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc55c06c5e0 0x7fc55c06eaa0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc58415a730 0x7fc58415bda0 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc58415e380 0x7fc58415b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 --2- 172.21.15.53:0/640240131 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc5640047c0 0x7fc58415a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 >> 172.21.15.53:0/640240131 conn(0x7fc58406a800 msgr2=0x7fc5840715a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 shutdown_connections 2024-01-16T15:20:45.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:45.790+0000 7fc588e00640 1 -- 172.21.15.53:0/640240131 wait complete. 2024-01-16T15:20:45.802 INFO:tasks.cephfs.test_volumes:expected clone state = ('pending', 'in-progress') 2024-01-16T15:20:45.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone status cephfs subvol_clone52247 2024-01-16T15:20:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.034+0000 7f0f7b200640 1 Processor -- start 2024-01-16T15:20:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.035+0000 7f0f7b200640 1 -- start start 2024-01-16T15:20:46.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.036+0000 7f0f7b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0f7c15d4b0 0x7f0f7c0d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.036+0000 7f0f7b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c0b2730 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.036+0000 7f0f7b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7c0d1ec0 con 0x7f0f7c0b41e0 2024-01-16T15:20:46.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.036+0000 7f0f7b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7c0b29d0 con 0x7f0f7c15d4b0 2024-01-16T15:20:46.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.037+0000 7f0f73e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0f7c0cf9a0 0x7f0f7c0cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55356/0 (socket says 172.21.15.53:55356) 2024-01-16T15:20:46.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.037+0000 7f0f73e00640 1 -- 172.21.15.53:0/638846198 learned_addr learned my addr 172.21.15.53:0/638846198 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.037+0000 7f0f79e00640 1 --2- 172.21.15.53:0/638846198 >> v2:172.21.15.146:3300/0 conn(0x7f0f7c15d4b0 0x7f0f7c0d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.037+0000 7f0f79400640 1 -- 172.21.15.53:0/638846198 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197303873 0 0) 0x7f0f7c0b2730 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.038+0000 7f0f79400640 1 -- 172.21.15.53:0/638846198 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f5c003520 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.038+0000 7f0f79e00640 1 -- 172.21.15.53:0/638846198 >> v1:172.21.15.146:6789/0 conn(0x7f0f7c0b41e0 legacy=0x7f0f7c0ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.038+0000 7f0f79400640 1 -- 172.21.15.53:0/638846198 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615552402 0 0) 0x7f0f5c003520 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.038+0000 7f0f79e00640 1 -- 172.21.15.53:0/638846198 >> v1:172.21.15.53:6789/0 conn(0x7f0f7c0cf9a0 legacy=0x7f0f7c0cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.038+0000 7f0f79e00640 1 -- 172.21.15.53:0/638846198 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7c0b2cb0 con 0x7f0f7c15d4b0 2024-01-16T15:20:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.039+0000 7f0f79e00640 1 --2- 172.21.15.53:0/638846198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c15d4b0 0x7f0f7c0d1550 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0097a0 tx=0x7f0f6c02b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe201c8763e56c79 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.040+0000 7f0f79400640 1 -- 172.21.15.53:0/638846198 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f6c02f050 con 0x7f0f7c15d4b0 2024-01-16T15:20:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.040+0000 7f0f79400640 1 -- 172.21.15.53:0/638846198 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f6c031500 con 0x7f0f7c15d4b0 2024-01-16T15:20:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.040+0000 7f0f79400640 1 -- 172.21.15.53:0/638846198 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64001670 con 0x7f0f7c15d4b0 2024-01-16T15:20:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 -- 172.21.15.53:0/638846198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c15d4b0 msgr2=0x7f0f7c0d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 --2- 172.21.15.53:0/638846198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c15d4b0 0x7f0f7c0d1550 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c0097a0 tx=0x7f0f6c02b820 comp rx=0 tx=0).stop 2024-01-16T15:20:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 -- 172.21.15.53:0/638846198 shutdown_connections 2024-01-16T15:20:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 --2- 172.21.15.53:0/638846198 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c15d4b0 0x7f0f7c0d1550 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 -- 172.21.15.53:0/638846198 >> 172.21.15.53:0/638846198 conn(0x7f0f7c0d3000 msgr2=0x7f0f7c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:46.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 -- 172.21.15.53:0/638846198 shutdown_connections 2024-01-16T15:20:46.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.041+0000 7f0f7b200640 1 -- 172.21.15.53:0/638846198 wait complete. 2024-01-16T15:20:46.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.042+0000 7f0f7b200640 1 Processor -- start 2024-01-16T15:20:46.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.043+0000 7f0f7b200640 1 -- start start 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f7b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c0b41e0 0x7f0f7c154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f7b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 0x7f0f7c09d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f7b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f7c15d4b0 0x7f0f7c0b7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f7b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c1f4050 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f7b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c154a60 con 0x7f0f7c0b41e0 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f7b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7c15a870 con 0x7f0f7c15d4b0 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 0x7f0f7c09d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 0x7f0f7c09d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50108/0 (socket says 172.21.15.53:50108) 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 -- 172.21.15.53:0/1635633819 learned_addr learned my addr 172.21.15.53:0/1635633819 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f79e00640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c0b41e0 0x7f0f7c154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 -- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f7c15d4b0 msgr2=0x7f0f7c0b7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f7c15d4b0 0x7f0f7c0b7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 -- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c0b41e0 msgr2=0x7f0f7c154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c0b41e0 0x7f0f7c154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.044+0000 7f0f73e00640 1 -- 172.21.15.53:0/1635633819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c009400 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.045+0000 7f0f73e00640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 0x7f0f7c09d4d0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400c760 tx=0x7f0f6400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.045+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64005ad0 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.045+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f64005c30 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.045+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f64007070 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.045+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f7c15da10 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.045+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f7c1f36b0 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.046+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7c1f4190 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.047+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f0f640071d0 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.048+0000 7f0f72000640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f5406c5e0 0x7f0f5406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.048+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0f64088910 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.049+0000 7f0f79e00640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f5406c5e0 0x7f0f5406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.050+0000 7f0f79e00640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f5406c5e0 0x7f0f5406eaa0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c035000 tx=0x7f0f6c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:46.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.053+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0f7c1f4190 con 0x7f0f7c0cf9a0 2024-01-16T15:20:46.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.284+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone status", "vol_name": "cephfs", "clone_name": "subvol_clone52247", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f7c15d890 con 0x7f0f5406c5e0 2024-01-16T15:20:46.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.299+0000 7f0f72000640 1 -- 172.21.15.53:0/1635633819 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+144 (secure 0 0 0) 0x7f0f6c042070 con 0x7f0f5406c5e0 2024-01-16T15:20:46.291 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: "status": { 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: "state": "pending", 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: "source": { 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: "volume": "cephfs", 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: "subvolume": "sv1", 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: "snapshot": "ss1" 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:20:46.292 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f5406c5e0 msgr2=0x7f0f5406eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f5406c5e0 0x7f0f5406eaa0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c035000 tx=0x7f0f6c008030 comp rx=0 tx=0).stop 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 msgr2=0x7f0f7c09d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 0x7f0f7c09d4d0 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f0f6400c760 tx=0x7f0f6400cc20 comp rx=0 tx=0).stop 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 shutdown_connections 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f5406c5e0 0x7f0f5406eaa0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0f7c15d4b0 0x7f0f7c0b7c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0f7c0cf9a0 0x7f0f7c09d4d0 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 --2- 172.21.15.53:0/1635633819 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0f7c0b41e0 0x7f0f7c154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.301+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 >> 172.21.15.53:0/1635633819 conn(0x7f0f7c0d3000 msgr2=0x7f0f7c0cddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:46.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.302+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 shutdown_connections 2024-01-16T15:20:46.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.302+0000 7f0f7b200640 1 -- 172.21.15.53:0/1635633819 wait complete. 2024-01-16T15:20:46.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52241 2024-01-16T15:20:46.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.550+0000 7fece7000640 1 Processor -- start 2024-01-16T15:20:46.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.550+0000 7fece7000640 1 -- start start 2024-01-16T15:20:46.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.552+0000 7fece7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 0x7fece0070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.552+0000 7fece7000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece015f4e0 con 0x7fece015bf10 2024-01-16T15:20:46.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.552+0000 7fece7000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fece0071130 con 0x7fece00ccb50 2024-01-16T15:20:46.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.552+0000 7fece7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece015f970 con 0x7fece00adf20 2024-01-16T15:20:46.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.552+0000 7fece5200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fece015bf10 0x7fece00cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55364/0 (socket says 172.21.15.53:55364) 2024-01-16T15:20:46.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fece5200640 1 -- 172.21.15.53:0/4174324307 learned_addr learned my addr 172.21.15.53:0/4174324307 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545375947 0 0) 0x7fece015f4e0 con 0x7fece015bf10 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fece5c00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 0x7fece0070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc0003520 con 0x7fece015bf10 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2900933522 0 0) 0x7fecc0003520 con 0x7fece015bf10 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 msgr2=0x7fece0070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 0x7fece0070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 >> v1:172.21.15.146:6789/0 conn(0x7fece00ccb50 legacy=0x7fece0143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece015c390 con 0x7fece015bf10 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.553+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fecc8001670 con 0x7fece015bf10 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 >> v1:172.21.15.53:6789/0 conn(0x7fece015bf10 legacy=0x7fece00cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.547 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecc00039a0 0x7fecc0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 0x7fecc00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004d00 0x7fecc0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece015f970 con 0x7fecc00039a0 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000bb10 con 0x7fecc0004d00 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fecc000bcd0 con 0x7fece00adf20 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fece5c00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecc00039a0 0x7fecc0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fece5c00640 1 -- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 msgr2=0x7fecc00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fece5c00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fece00adf20 0x7fecc00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fece5c00640 1 -- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004d00 msgr2=0x7fecc0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fece5c00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fecc0004d00 0x7fecc0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.554+0000 7fece5c00640 1 -- 172.21.15.53:0/4174324307 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0003520 con 0x7fecc00039a0 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.555+0000 7fece5c00640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecc00039a0 0x7fecc0003dc0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009ac0 tx=0x7fecc00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d9d6c8bfe3b7cd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd002c710 con 0x7fecc00039a0 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd002c870 con 0x7fecc00039a0 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fecdfe00640 1 -- 172.21.15.53:0/4174324307 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd002cc00 con 0x7fecc00039a0 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fece7000640 1 -- 172.21.15.53:0/4174324307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecc00039a0 msgr2=0x7fecc0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fece7000640 1 --2- 172.21.15.53:0/4174324307 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fecc00039a0 0x7fecc0003dc0 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fecd0009ac0 tx=0x7fecc00057f0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fece6600640 1 -- 172.21.15.53:0/4174324307 reap_dead start 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fece7000640 1 -- 172.21.15.53:0/4174324307 shutdown_connections 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.556+0000 7fece7000640 1 -- 172.21.15.53:0/4174324307 >> 172.21.15.53:0/4174324307 conn(0x7fece006a7d0 msgr2=0x7fece006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.557+0000 7fece7000640 1 -- 172.21.15.53:0/4174324307 shutdown_connections 2024-01-16T15:20:46.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.557+0000 7fece7000640 1 -- 172.21.15.53:0/4174324307 wait complete. 2024-01-16T15:20:46.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.558+0000 7fece7000640 1 Processor -- start 2024-01-16T15:20:46.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.558+0000 7fece7000640 1 -- start start 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.559+0000 7fece7000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc00039a0 0x7fece013f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.559+0000 7fece7000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fece01e5cb0 0x7fece01e7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.559+0000 7fece7000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 0x7fece00b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.559+0000 7fece7000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece01e4c10 con 0x7fece01e8410 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.559+0000 7fece7000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece01e4ef0 con 0x7fece01e5cb0 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.559+0000 7fece7000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece01e51d0 con 0x7fecc00039a0 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 0x7fece00b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 0x7fece00b3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50132/0 (socket says 172.21.15.53:50132) 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 -- 172.21.15.53:0/3491025124 learned_addr learned my addr 172.21.15.53:0/3491025124 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:20:46.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece5c00640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc00039a0 0x7fece013f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece5200640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fece01e5cb0 0x7fece01e7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 -- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc00039a0 msgr2=0x7fece013f9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc00039a0 0x7fece013f9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 -- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fece01e5cb0 msgr2=0x7fece01e7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fece01e5cb0 0x7fece01e7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece6600640 1 -- 172.21.15.53:0/3491025124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece01e54b0 con 0x7fece01e8410 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece5200640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fece01e5cb0 0x7fece01e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.560+0000 7fece5c00640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc00039a0 0x7fece013f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:20:46.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.561+0000 7fece6600640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 0x7fece00b3010 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fece00ae7d0 tx=0x7fecd800cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:46.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.561+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd8005d40 con 0x7fece01e8410 2024-01-16T15:20:46.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.561+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecd800a610 con 0x7fece01e8410 2024-01-16T15:20:46.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.561+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecd800a9c0 con 0x7fece01e8410 2024-01-16T15:20:46.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.561+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece01e5790 con 0x7fece01e8410 2024-01-16T15:20:46.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.561+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece00b3550 con 0x7fece01e8410 2024-01-16T15:20:46.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.562+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece0067450 con 0x7fece01e8410 2024-01-16T15:20:46.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.563+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 44) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fecd800ab20 con 0x7fece01e8410 2024-01-16T15:20:46.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.564+0000 7fecdd600640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecb806c5e0 0x7fecb806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:20:46.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.564+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fecd8087c00 con 0x7fece01e8410 2024-01-16T15:20:46.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.565+0000 7fece5c00640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecb806c5e0 0x7fecb806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:20:46.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.566+0000 7fece5c00640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecb806c5e0 0x7fecb806eaa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fecd0036000 tx=0x7fecd0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:20:46.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.569+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fecd8054120 con 0x7fece01e8410 2024-01-16T15:20:46.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:20:46.803+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone52241", "target": ["mon-mgr", ""]}) v1 -- 0x7fece01e5060 con 0x7fecb806c5e0 2024-01-16T15:20:56.761 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:20:56.770 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:04.286 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:04.292+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:04.286 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:04.292+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:04.286 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:04.292+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:04.286 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:04.292+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:04.286 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:04.292+0000 7f613d200640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:04.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.525+0000 7fecdd600640 1 -- 172.21.15.53:0/3491025124 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fece01e5060 con 0x7fecb806c5e0 2024-01-16T15:21:04.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecb806c5e0 msgr2=0x7fecb806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:04.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecb806c5e0 0x7fecb806eaa0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fecd0036000 tx=0x7fecd0030000 comp rx=0 tx=0).stop 2024-01-16T15:21:04.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 msgr2=0x7fece00b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:04.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 0x7fece00b3010 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fece00ae7d0 tx=0x7fecd800cb90 comp rx=0 tx=0).stop 2024-01-16T15:21:04.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 shutdown_connections 2024-01-16T15:21:04.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecb806c5e0 0x7fecb806eaa0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:04.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fece01e8410 0x7fece00b3010 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:04.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fece01e5cb0 0x7fece01e7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:04.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 --2- 172.21.15.53:0/3491025124 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fecc00039a0 0x7fece013f9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:04.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 >> 172.21.15.53:0/3491025124 conn(0x7fece006a7d0 msgr2=0x7fece015b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:04.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 shutdown_connections 2024-01-16T15:21:04.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:04.529+0000 7fece7000640 1 -- 172.21.15.53:0/3491025124 wait complete. 2024-01-16T15:21:06.546 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52242 2024-01-16T15:21:06.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.783+0000 7fe9ab400640 1 Processor -- start 2024-01-16T15:21:06.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.784+0000 7fe9ab400640 1 -- start start 2024-01-16T15:21:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.785+0000 7fe9ab400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe9a415d4b0 0x7fe9a40d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.785+0000 7fe9ab400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fe9a40cf9a0 0x7fe9a40cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.785+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a40cddf0 0x7fe9a40ce280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.785+0000 7fe9ab400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a40d0320 con 0x7fe9a415d4b0 2024-01-16T15:21:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.785+0000 7fe9ab400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a40b2730 con 0x7fe9a40cf9a0 2024-01-16T15:21:06.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.785+0000 7fe9ab400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40d1ec0 con 0x7fe9a40cddf0 2024-01-16T15:21:06.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe9a415d4b0 0x7fe9a40d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe9a415d4b0 0x7fe9a40d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42446/0 (socket says 172.21.15.53:42446) 2024-01-16T15:21:06.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 -- 172.21.15.53:0/2244098524 learned_addr learned my addr 172.21.15.53:0/2244098524 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:21:06.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9a9600640 1 --2- 172.21.15.53:0/2244098524 >> v2:172.21.15.146:3300/0 conn(0x7fe9a40cf9a0 0x7fe9a40cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aaa00640 1 --2- 172.21.15.53:0/2244098524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a40cddf0 0x7fe9a40ce280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 -- 172.21.15.53:0/2244098524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a40cddf0 msgr2=0x7fe9a40ce280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 --2- 172.21.15.53:0/2244098524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a40cddf0 0x7fe9a40ce280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 -- 172.21.15.53:0/2244098524 >> v2:172.21.15.146:3300/0 conn(0x7fe9a40cf9a0 msgr2=0x7fe9a40cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 --2- 172.21.15.53:0/2244098524 >> v2:172.21.15.146:3300/0 conn(0x7fe9a40cf9a0 0x7fe9a40cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.786+0000 7fe9aa000640 1 -- 172.21.15.53:0/2244098524 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40b29d0 con 0x7fe9a415d4b0 2024-01-16T15:21:06.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.787+0000 7fe9a9600640 1 --2- 172.21.15.53:0/2244098524 >> v2:172.21.15.146:3300/0 conn(0x7fe9a40cf9a0 0x7fe9a40cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:21:06.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.788+0000 7fe9aa000640 1 --2- 172.21.15.53:0/2244098524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a415d4b0 0x7fe9a40d1550 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe9a408b580 tx=0x7fe99402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e834b69418ae431a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:06.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.789+0000 7fe9a8c00640 1 -- 172.21.15.53:0/2244098524 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994031870 con 0x7fe9a415d4b0 2024-01-16T15:21:06.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.789+0000 7fe9a8c00640 1 -- 172.21.15.53:0/2244098524 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9940319d0 con 0x7fe9a415d4b0 2024-01-16T15:21:06.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.789+0000 7fe9a8c00640 1 -- 172.21.15.53:0/2244098524 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994031d60 con 0x7fe9a415d4b0 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.789+0000 7fe9ab400640 1 -- 172.21.15.53:0/2244098524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a415d4b0 msgr2=0x7fe9a40d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.789+0000 7fe9ab400640 1 --2- 172.21.15.53:0/2244098524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a415d4b0 0x7fe9a40d1550 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7fe9a408b580 tx=0x7fe99402b910 comp rx=0 tx=0).stop 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 -- 172.21.15.53:0/2244098524 shutdown_connections 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 --2- 172.21.15.53:0/2244098524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a40cddf0 0x7fe9a40ce280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 --2- 172.21.15.53:0/2244098524 >> v2:172.21.15.146:3300/0 conn(0x7fe9a40cf9a0 0x7fe9a40cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 --2- 172.21.15.53:0/2244098524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a415d4b0 0x7fe9a40d1550 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 -- 172.21.15.53:0/2244098524 >> 172.21.15.53:0/2244098524 conn(0x7fe9a40d3000 msgr2=0x7fe9a40d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 -- 172.21.15.53:0/2244098524 shutdown_connections 2024-01-16T15:21:06.784 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.790+0000 7fe9ab400640 1 -- 172.21.15.53:0/2244098524 wait complete. 2024-01-16T15:21:06.786 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.793+0000 7fe9ab400640 1 Processor -- start 2024-01-16T15:21:06.787 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.793+0000 7fe9ab400640 1 -- start start 2024-01-16T15:21:06.788 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.794+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 0x7fe9a415c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40cf9a0 0x7fe9a40cb600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9ab400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a415d4b0 0x7fe9a414ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9ab400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40cb170 con 0x7fe9a40cddf0 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9ab400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40ca560 con 0x7fe9a40cf9a0 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9ab400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40ca840 con 0x7fe9a415d4b0 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 0x7fe9a415c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 0x7fe9a415c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:42458/0 (socket says 172.21.15.53:42458) 2024-01-16T15:21:06.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 -- 172.21.15.53:0/4241669445 learned_addr learned my addr 172.21.15.53:0/4241669445 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:21:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aaa00640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a415d4b0 0x7fe9a414ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 -- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a415d4b0 msgr2=0x7fe9a414ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9a9600640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40cf9a0 0x7fe9a40cb600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a415d4b0 0x7fe9a414ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.790 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 -- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40cf9a0 msgr2=0x7fe9a40cb600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40cf9a0 0x7fe9a40cb600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.795+0000 7fe9aa000640 1 -- 172.21.15.53:0/4241669445 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9940094d0 con 0x7fe9a40cddf0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.796+0000 7fe9aa000640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 0x7fe9a415c2a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe9940097c0 tx=0x7fe9940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.796+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99403ac60 con 0x7fe9a40cddf0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.796+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99403adc0 con 0x7fe9a40cddf0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.796+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe994044610 con 0x7fe9a40cddf0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.796+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a415da10 con 0x7fe9a40cddf0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.796+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a40cae00 con 0x7fe9a40cddf0 2024-01-16T15:21:06.791 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.797+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a40d1b20 con 0x7fe9a40cddf0 2024-01-16T15:21:06.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.799+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 45) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fe99404d450 con 0x7fe9a40cddf0 2024-01-16T15:21:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.800+0000 7fe9a2000640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe97c06c670 0x7fe97c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:21:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.800+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe99402d040 con 0x7fe9a40cddf0 2024-01-16T15:21:06.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.801+0000 7fe9a9600640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe97c06c670 0x7fe97c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:21:06.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.802+0000 7fe9a9600640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe97c06c670 0x7fe97c06eb30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe9a408c960 tx=0x7fe98c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:21:06.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:06.804+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe994077670 con 0x7fe9a40cddf0 2024-01-16T15:21:07.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:07.036+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone52242", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9a40cf290 con 0x7fe97c06c670 2024-01-16T15:21:14.272 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:14.278+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:14.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:14.279+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:14.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:14.279+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:14.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:14.279+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:14.273 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:14.279+0000 7f613be00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:15.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:15.829+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mgrmap(e 46) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fe9940ab460 con 0x7fe9a40cddf0 2024-01-16T15:21:26.905 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:26.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:35.591 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:35.597+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:35.591 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:35.597+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:35.591 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:35.597+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:35.591 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:35.597+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:35.592 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:21:35.597+0000 7f613aa00640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:21:37.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:21:37.844+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 8 ==== mgrmap(e 47) v1 ==== 91426+0+0 (secure 0 0 0) 0x7fe994077a40 con 0x7fe9a40cddf0 2024-01-16T15:21:57.071 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:21:57.080 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:11.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:22:11.839+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mon.0 v2:172.21.15.53:3300/0 9 ==== mgrmap(e 48) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fe99407f0f0 con 0x7fe9a40cddf0 2024-01-16T15:22:27.209 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:27.218 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:57.343 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:22:57.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:04.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.801+0000 7fe9a2000640 1 -- 172.21.15.53:0/4241669445 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe9a40cf290 con 0x7fe97c06c670 2024-01-16T15:23:04.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe97c06c670 msgr2=0x7fe97c06eb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe97c06c670 0x7fe97c06eb30 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe9a408c960 tx=0x7fe98c008040 comp rx=0 tx=0).stop 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 msgr2=0x7fe9a415c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 0x7fe9a415c2a0 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7fe9940097c0 tx=0x7fe9940046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 shutdown_connections 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe97c06c670 0x7fe97c06eb30 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe9a415d4b0 0x7fe9a414ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe9a40cf9a0 0x7fe9a40cb600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 --2- 172.21.15.53:0/4241669445 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe9a40cddf0 0x7fe9a415c2a0 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 >> 172.21.15.53:0/4241669445 conn(0x7fe9a40d3000 msgr2=0x7fe9a40b39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 shutdown_connections 2024-01-16T15:23:04.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:04.804+0000 7fe9ab400640 1 -- 172.21.15.53:0/4241669445 wait complete. 2024-01-16T15:23:06.823 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52243 2024-01-16T15:23:07.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.063+0000 7fba6f400640 1 Processor -- start 2024-01-16T15:23:07.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.063+0000 7fba6f400640 1 -- start start 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.064+0000 7fba6f400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba6813cf10 0x7fba6813f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.064+0000 7fba6f400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680ce440 con 0x7fba6813fae0 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6f400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba680ce720 con 0x7fba680cddf0 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6f400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba6813ff90 con 0x7fba6813cf10 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba6813cf10 0x7fba6813f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6e000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fba6813cf10 0x7fba6813f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36958/0 (socket says 172.21.15.53:36958) 2024-01-16T15:23:07.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6e000640 1 -- 172.21.15.53:0/2118670176 learned_addr learned my addr 172.21.15.53:0/2118670176 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:23:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6e000640 1 -- 172.21.15.53:0/2118670176 >> v1:172.21.15.146:6789/0 conn(0x7fba680cddf0 legacy=0x7fba680b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6e000640 1 -- 172.21.15.53:0/2118670176 >> v1:172.21.15.53:6789/0 conn(0x7fba6813fae0 legacy=0x7fba680cd6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6e000640 1 -- 172.21.15.53:0/2118670176 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba680cea00 con 0x7fba6813cf10 2024-01-16T15:23:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.065+0000 7fba6cc00640 1 -- 172.21.15.53:0/2118670176 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734583362 0 0) 0x7fba680ce440 con 0x7fba6813fae0 2024-01-16T15:23:07.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.067+0000 7fba6e000640 1 --2- 172.21.15.53:0/2118670176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6813cf10 0x7fba6813f5a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fba5800bb10 tx=0x7fba5802e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe864ebb4d208bea server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.067+0000 7fba6cc00640 1 -- 172.21.15.53:0/2118670176 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58036440 con 0x7fba6813cf10 2024-01-16T15:23:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.067+0000 7fba6cc00640 1 -- 172.21.15.53:0/2118670176 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba580365a0 con 0x7fba6813cf10 2024-01-16T15:23:07.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.067+0000 7fba6cc00640 1 -- 172.21.15.53:0/2118670176 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58036930 con 0x7fba6813cf10 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 -- 172.21.15.53:0/2118670176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6813cf10 msgr2=0x7fba6813f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 --2- 172.21.15.53:0/2118670176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6813cf10 0x7fba6813f5a0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fba5800bb10 tx=0x7fba5802e170 comp rx=0 tx=0).stop 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 -- 172.21.15.53:0/2118670176 shutdown_connections 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 --2- 172.21.15.53:0/2118670176 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba6813cf10 0x7fba6813f5a0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 -- 172.21.15.53:0/2118670176 >> 172.21.15.53:0/2118670176 conn(0x7fba6806a7d0 msgr2=0x7fba6806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 -- 172.21.15.53:0/2118670176 shutdown_connections 2024-01-16T15:23:07.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.068+0000 7fba6f400640 1 -- 172.21.15.53:0/2118670176 wait complete. 2024-01-16T15:23:07.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.070+0000 7fba6f400640 1 Processor -- start 2024-01-16T15:23:07.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.070+0000 7fba6f400640 1 -- start start 2024-01-16T15:23:07.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 0x7fba68070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6813cf10 0x7fba681572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6f400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6813fae0 0x7fba68153350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6f400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba68157800 con 0x7fba680cddf0 2024-01-16T15:23:07.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6f400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba680ac8d0 con 0x7fba6813fae0 2024-01-16T15:23:07.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6f400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba680acb20 con 0x7fba6813cf10 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 0x7fba68070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 0x7fba68070b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:36968/0 (socket says 172.21.15.53:36968) 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 -- 172.21.15.53:0/3494831899 learned_addr learned my addr 172.21.15.53:0/3494831899 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6ea00640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6813fae0 0x7fba68153350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6d600640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6813cf10 0x7fba681572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 -- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6813cf10 msgr2=0x7fba681572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6813cf10 0x7fba681572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 -- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6813fae0 msgr2=0x7fba68153350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.071+0000 7fba6e000640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6813fae0 0x7fba68153350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba6e000640 1 -- 172.21.15.53:0/3494831899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba5800b770 con 0x7fba680cddf0 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba6d600640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6813cf10 0x7fba681572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:23:07.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba6e000640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 0x7fba68070b70 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fba5800bc30 tx=0x7fba580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba58036440 con 0x7fba680cddf0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba58034b20 con 0x7fba680cddf0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba680ace00 con 0x7fba680cddf0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba5803f440 con 0x7fba680cddf0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.072+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba680ad3c0 con 0x7fba680cddf0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.073+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba680ad5b0 con 0x7fba680cddf0 2024-01-16T15:23:07.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.075+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 48) v1 ==== 91375+0+0 (secure 0 0 0) 0x7fba5803d070 con 0x7fba680cddf0 2024-01-16T15:23:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.075+0000 7fba66000640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba4006c700 0x7fba4006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:07.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.076+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fba580ab000 con 0x7fba680cddf0 2024-01-16T15:23:07.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.076+0000 7fba6d600640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba4006c700 0x7fba4006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:07.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.077+0000 7fba6d600640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba4006c700 0x7fba4006ebc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fba6815a9b0 tx=0x7fba50009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:07.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.080+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fba58077920 con 0x7fba680cddf0 2024-01-16T15:23:07.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:07.313+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone52243", "target": ["mon-mgr", ""]}) v1 -- 0x7fba680b2070 con 0x7fba4006c700 2024-01-16T15:23:09.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.737+0000 7fba66000640 1 -- 172.21.15.53:0/3494831899 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fba50001320 con 0x7fba4006c700 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.740+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba4006c700 msgr2=0x7fba4006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.740+0000 7fba6f400640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba4006c700 0x7fba4006ebc0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fba6815a9b0 tx=0x7fba50009040 comp rx=0 tx=0).stop 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.740+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 msgr2=0x7fba68070b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.740+0000 7fba6f400640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 0x7fba68070b70 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fba5800bc30 tx=0x7fba580046c0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 shutdown_connections 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fba4006c700 0x7fba4006ebc0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fba6813fae0 0x7fba68153350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fba6813cf10 0x7fba681572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 --2- 172.21.15.53:0/3494831899 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fba680cddf0 0x7fba68070b70 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 >> 172.21.15.53:0/3494831899 conn(0x7fba6806a7d0 msgr2=0x7fba68159d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 shutdown_connections 2024-01-16T15:23:09.737 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:09.741+0000 7fba6f400640 1 -- 172.21.15.53:0/3494831899 wait complete. 2024-01-16T15:23:11.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52244 2024-01-16T15:23:11.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.000+0000 7f96cf400640 1 Processor -- start 2024-01-16T15:23:11.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.000+0000 7f96cf400640 1 -- start start 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.001+0000 7f96cf400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96c80cfde0 0x7f96c80b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.001+0000 7f96cf400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c80d1c60 con 0x7f96c815d4b0 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.001+0000 7f96cf400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c80b2730 con 0x7f96c80cf9a0 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.001+0000 7f96cf400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96c80d1ec0 con 0x7f96c80cfde0 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96cea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f96c80cfde0 0x7f96c80b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96ce000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f96c815d4b0 0x7f96c80d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42336/0 (socket says 172.21.15.53:42336) 2024-01-16T15:23:11.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96ce000640 1 -- 172.21.15.53:0/1862472112 learned_addr learned my addr 172.21.15.53:0/1862472112 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96cea00640 1 -- 172.21.15.53:0/1862472112 >> v1:172.21.15.146:6789/0 conn(0x7f96c80cf9a0 legacy=0x7f96c80b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96cea00640 1 -- 172.21.15.53:0/1862472112 >> v1:172.21.15.53:6789/0 conn(0x7f96c815d4b0 legacy=0x7f96c80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96ccc00640 1 -- 172.21.15.53:0/1862472112 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3536771307 0 0) 0x7f96c80d1c60 con 0x7f96c815d4b0 2024-01-16T15:23:11.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.002+0000 7f96cea00640 1 -- 172.21.15.53:0/1862472112 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c80b29d0 con 0x7f96c80cfde0 2024-01-16T15:23:11.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.003+0000 7f96cea00640 1 --2- 172.21.15.53:0/1862472112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c80cfde0 0x7f96c80b38e0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009d90 tx=0x7f96c002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81882f2da1843c10 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:11.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96ccc00640 1 -- 172.21.15.53:0/1862472112 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c002cc60 con 0x7f96c80cfde0 2024-01-16T15:23:11.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96ccc00640 1 -- 172.21.15.53:0/1862472112 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c002cdc0 con 0x7f96c80cfde0 2024-01-16T15:23:11.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96ccc00640 1 -- 172.21.15.53:0/1862472112 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0034660 con 0x7f96c80cfde0 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96cf400640 1 -- 172.21.15.53:0/1862472112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c80cfde0 msgr2=0x7f96c80b38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96cf400640 1 --2- 172.21.15.53:0/1862472112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c80cfde0 0x7f96c80b38e0 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009d90 tx=0x7f96c002beb0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96cf400640 1 -- 172.21.15.53:0/1862472112 shutdown_connections 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96cf400640 1 --2- 172.21.15.53:0/1862472112 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c80cfde0 0x7f96c80b38e0 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.004+0000 7f96cf400640 1 -- 172.21.15.53:0/1862472112 >> 172.21.15.53:0/1862472112 conn(0x7f96c80d3000 msgr2=0x7f96c80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.005+0000 7f96cf400640 1 -- 172.21.15.53:0/1862472112 shutdown_connections 2024-01-16T15:23:12.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.005+0000 7f96cf400640 1 -- 172.21.15.53:0/1862472112 wait complete. 2024-01-16T15:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.006+0000 7f96cf400640 1 Processor -- start 2024-01-16T15:23:12.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.006+0000 7f96cf400640 1 -- start start 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.007+0000 7f96cf400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96c80cf9a0 0x7f96c81e9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.007+0000 7f96cf400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96c80cfde0 0x7f96c809d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.007+0000 7f96cf400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 0x7f96c814a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.007+0000 7f96cf400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c80c94c0 con 0x7f96c815d4b0 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.007+0000 7f96cf400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96c814a100 con 0x7f96c80cf9a0 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.007+0000 7f96cf400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c81494f0 con 0x7f96c80cfde0 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 0x7f96c814a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 0x7f96c814a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44612/0 (socket says 172.21.15.53:44612) 2024-01-16T15:23:12.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 -- 172.21.15.53:0/3770580493 learned_addr learned my addr 172.21.15.53:0/3770580493 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cd600640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96c80cfde0 0x7f96c809d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96ce000640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96c80cf9a0 0x7f96c81e9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 -- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96c80cfde0 msgr2=0x7f96c809d120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96c80cfde0 0x7f96c809d120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 -- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96c80cf9a0 msgr2=0x7f96c81e9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96c80cf9a0 0x7f96c81e9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 -- 172.21.15.53:0/3770580493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c0009a70 con 0x7f96c815d4b0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.008+0000 7f96cea00640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 0x7f96c814a590 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009d90 tx=0x7f96c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.009+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0040450 con 0x7f96c815d4b0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.009+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c00405d0 con 0x7f96c815d4b0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.009+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0040980 con 0x7f96c815d4b0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.009+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c81497d0 con 0x7f96c815d4b0 2024-01-16T15:23:12.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.009+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c8149d90 con 0x7f96c815d4b0 2024-01-16T15:23:12.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.010+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c80d1110 con 0x7f96c815d4b0 2024-01-16T15:23:12.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.011+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 48) v1 ==== 91375+0+0 (secure 0 0 0) 0x7f96c0040ae0 con 0x7f96c815d4b0 2024-01-16T15:23:12.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.012+0000 7f96c6000640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f96a006c700 0x7f96a006ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:12.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.012+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f96c00ab470 con 0x7f96c815d4b0 2024-01-16T15:23:12.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.013+0000 7f96ce000640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f96a006c700 0x7f96a006ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:12.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.014+0000 7f96ce000640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f96a006c700 0x7f96a006ebc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f96b8000f60 tx=0x7f96b8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:12.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.017+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96c0077d30 con 0x7f96c815d4b0 2024-01-16T15:23:12.244 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:12.249+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone52244", "target": ["mon-mgr", ""]}) v1 -- 0x7f96c8067f00 con 0x7f96a006c700 2024-01-16T15:23:14.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.748+0000 7f96c6000640 1 -- 172.21.15.53:0/3770580493 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f96c8067f00 con 0x7f96a006c700 2024-01-16T15:23:14.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f96a006c700 msgr2=0x7f96a006ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f96a006c700 0x7f96a006ebc0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f96b8000f60 tx=0x7f96b8009040 comp rx=0 tx=0).stop 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 msgr2=0x7f96c814a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 0x7f96c814a590 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f96c0009d90 tx=0x7f96c0004a40 comp rx=0 tx=0).stop 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 shutdown_connections 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f96a006c700 0x7f96a006ebc0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96c815d4b0 0x7f96c814a590 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96c80cfde0 0x7f96c809d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 --2- 172.21.15.53:0/3770580493 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96c80cf9a0 0x7f96c81e9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 >> 172.21.15.53:0/3770580493 conn(0x7f96c80d3000 msgr2=0x7f96c806c120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 shutdown_connections 2024-01-16T15:23:14.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:14.752+0000 7f96cf400640 1 -- 172.21.15.53:0/3770580493 wait complete. 2024-01-16T15:23:16.767 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52245 2024-01-16T15:23:17.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.006+0000 7ff247c00640 1 Processor -- start 2024-01-16T15:23:17.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.006+0000 7ff247c00640 1 -- start start 2024-01-16T15:23:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.008+0000 7ff247c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 0x7ff24015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.008+0000 7ff247c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff24015e1e0 0x7ff2400702c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.008+0000 7ff247c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2400b1ee0 con 0x7ff240143ae0 2024-01-16T15:23:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.008+0000 7ff247c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2400b2780 con 0x7ff24015e1e0 2024-01-16T15:23:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.008+0000 7ff247c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24015e990 con 0x7ff2400cf9a0 2024-01-16T15:23:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 0x7ff24015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff245c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff24015e1e0 0x7ff2400702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 0x7ff24015dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:48560/0 (socket says 172.21.15.53:48560) 2024-01-16T15:23:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 -- 172.21.15.53:0/3679642993 learned_addr learned my addr 172.21.15.53:0/3679642993 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:23:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 -- 172.21.15.53:0/3679642993 >> v2:172.21.15.146:3300/0 conn(0x7ff24015e1e0 msgr2=0x7ff2400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 --2- 172.21.15.53:0/3679642993 >> v2:172.21.15.146:3300/0 conn(0x7ff24015e1e0 0x7ff2400702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 -- 172.21.15.53:0/3679642993 >> v1:172.21.15.146:6789/0 conn(0x7ff240143ae0 legacy=0x7ff2400cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.009+0000 7ff237c00640 1 -- 172.21.15.53:0/3679642993 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2400b2130 con 0x7ff2400cf9a0 2024-01-16T15:23:17.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.012+0000 7ff237c00640 1 --2- 172.21.15.53:0/3679642993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 0x7ff24015dca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff228009870 tx=0x7ff22802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=822bec689aae20c6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:17.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.013+0000 7ff23fe00640 1 -- 172.21.15.53:0/3679642993 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228030840 con 0x7ff2400cf9a0 2024-01-16T15:23:17.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.013+0000 7ff23fe00640 1 -- 172.21.15.53:0/3679642993 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2280309a0 con 0x7ff2400cf9a0 2024-01-16T15:23:17.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.013+0000 7ff23fe00640 1 -- 172.21.15.53:0/3679642993 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228030d30 con 0x7ff2400cf9a0 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 -- 172.21.15.53:0/3679642993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 msgr2=0x7ff24015dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 --2- 172.21.15.53:0/3679642993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 0x7ff24015dca0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff228009870 tx=0x7ff22802b980 comp rx=0 tx=0).stop 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 -- 172.21.15.53:0/3679642993 shutdown_connections 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 --2- 172.21.15.53:0/3679642993 >> v2:172.21.15.146:3300/0 conn(0x7ff24015e1e0 0x7ff2400702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 --2- 172.21.15.53:0/3679642993 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff2400cf9a0 0x7ff24015dca0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 -- 172.21.15.53:0/3679642993 >> 172.21.15.53:0/3679642993 conn(0x7ff24006a800 msgr2=0x7ff24006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 -- 172.21.15.53:0/3679642993 shutdown_connections 2024-01-16T15:23:17.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.014+0000 7ff247c00640 1 -- 172.21.15.53:0/3679642993 wait complete. 2024-01-16T15:23:17.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.015+0000 7ff247c00640 1 Processor -- start 2024-01-16T15:23:17.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.016+0000 7ff247c00640 1 -- start start 2024-01-16T15:23:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff247c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2400cf9a0 0x7ff2401418a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff247c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 0x7ff2400c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff247c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff24015e1e0 0x7ff2400c4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff247c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400c6360 con 0x7ff240143ae0 2024-01-16T15:23:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff247c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400c5750 con 0x7ff2400cf9a0 2024-01-16T15:23:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff247c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400c5a30 con 0x7ff24015e1e0 2024-01-16T15:23:17.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff237c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 0x7ff2400c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff237c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 0x7ff2400c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44614/0 (socket says 172.21.15.53:44614) 2024-01-16T15:23:17.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff237c00640 1 -- 172.21.15.53:0/4245151796 learned_addr learned my addr 172.21.15.53:0/4245151796 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:23:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff245200640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2400cf9a0 0x7ff2401418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.017+0000 7ff245c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff24015e1e0 0x7ff2400c4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.018+0000 7ff237c00640 1 -- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff24015e1e0 msgr2=0x7ff2400c4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.018+0000 7ff237c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff24015e1e0 0x7ff2400c4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.018+0000 7ff237c00640 1 -- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2400cf9a0 msgr2=0x7ff2401418a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.018+0000 7ff237c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2400cf9a0 0x7ff2401418a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.018+0000 7ff237c00640 1 -- 172.21.15.53:0/4245151796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2280094d0 con 0x7ff240143ae0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.018+0000 7ff237c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 0x7ff2400c3e50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7ff228000c00 tx=0x7ff228004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.019+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228039c90 con 0x7ff240143ae0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.019+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228039df0 con 0x7ff240143ae0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.019+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228043640 con 0x7ff240143ae0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.019+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2400c5d10 con 0x7ff240143ae0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.019+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff240141de0 con 0x7ff240143ae0 2024-01-16T15:23:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.020+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2400c6510 con 0x7ff240143ae0 2024-01-16T15:23:17.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.021+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 48) v1 ==== 91375+0+0 (secure 0 0 0) 0x7ff2280437a0 con 0x7ff240143ae0 2024-01-16T15:23:17.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.022+0000 7ff23d600640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff21806c700 0x7ff21806ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:17.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.022+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff22803e070 con 0x7ff240143ae0 2024-01-16T15:23:17.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.023+0000 7ff245200640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff21806c700 0x7ff21806ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:17.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.024+0000 7ff245200640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff21806c700 0x7ff21806ebc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff2300012e0 tx=0x7ff230009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:17.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.027+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff2400c6510 con 0x7ff240143ae0 2024-01-16T15:23:17.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:17.260+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone52245", "target": ["mon-mgr", ""]}) v1 -- 0x7ff240141fa0 con 0x7ff21806c700 2024-01-16T15:23:27.482 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:27.492 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:38.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.719+0000 7ff23d600640 1 -- 172.21.15.53:0/4245151796 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff240141fa0 con 0x7ff21806c700 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff21806c700 msgr2=0x7ff21806ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff21806c700 0x7ff21806ebc0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7ff2300012e0 tx=0x7ff230009040 comp rx=0 tx=0).stop 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 msgr2=0x7ff2400c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 0x7ff2400c3e50 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7ff228000c00 tx=0x7ff228004730 comp rx=0 tx=0).stop 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 shutdown_connections 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff21806c700 0x7ff21806ebc0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff24015e1e0 0x7ff2400c4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff240143ae0 0x7ff2400c3e50 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 --2- 172.21.15.53:0/4245151796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff2400cf9a0 0x7ff2401418a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 >> 172.21.15.53:0/4245151796 conn(0x7ff24006a800 msgr2=0x7ff24006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 shutdown_connections 2024-01-16T15:23:38.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:38.722+0000 7ff247c00640 1 -- 172.21.15.53:0/4245151796 wait complete. 2024-01-16T15:23:40.737 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52246 2024-01-16T15:23:40.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.973+0000 7f22e2c00640 1 Processor -- start 2024-01-16T15:23:40.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.973+0000 7f22e2c00640 1 -- start start 2024-01-16T15:23:40.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e2c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22dc0adf20 0x7f22dc070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e2c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22dc15f4e0 con 0x7f22dc0ccb50 2024-01-16T15:23:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e2c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22dc071130 con 0x7f22dc15bf10 2024-01-16T15:23:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e2c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22dc15f970 con 0x7f22dc0adf20 2024-01-16T15:23:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e1800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f22dc0adf20 0x7f22dc070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e2200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f22dc0ccb50 0x7f22dc143d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:35216/0 (socket says 172.21.15.53:35216) 2024-01-16T15:23:40.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.975+0000 7f22e2200640 1 -- 172.21.15.53:0/2380462983 learned_addr learned my addr 172.21.15.53:0/2380462983 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:23:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.976+0000 7f22dbe00640 1 -- 172.21.15.53:0/2380462983 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520497043 0 0) 0x7f22dc15f4e0 con 0x7f22dc0ccb50 2024-01-16T15:23:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.976+0000 7f22dbe00640 1 -- 172.21.15.53:0/2380462983 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22bc003520 con 0x7f22dc0ccb50 2024-01-16T15:23:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.976+0000 7f22e1800640 1 -- 172.21.15.53:0/2380462983 >> v1:172.21.15.146:6789/0 conn(0x7f22dc15bf10 legacy=0x7f22dc0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:23:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.976+0000 7f22e1800640 1 -- 172.21.15.53:0/2380462983 >> v1:172.21.15.53:6789/0 conn(0x7f22dc0ccb50 legacy=0x7f22dc143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:40.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.976+0000 7f22e1800640 1 -- 172.21.15.53:0/2380462983 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc15c390 con 0x7f22dc0adf20 2024-01-16T15:23:40.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.977+0000 7f22e1800640 1 --2- 172.21.15.53:0/2380462983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0adf20 0x7f22dc070a30 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0097a0 tx=0x7f22cc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9eed2f10cfa7bc7a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:40.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.977+0000 7f22dbe00640 1 -- 172.21.15.53:0/2380462983 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc02e070 con 0x7f22dc0adf20 2024-01-16T15:23:40.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.977+0000 7f22dbe00640 1 -- 172.21.15.53:0/2380462983 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22cc0315a0 con 0x7f22dc0adf20 2024-01-16T15:23:40.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.977+0000 7f22dbe00640 1 -- 172.21.15.53:0/2380462983 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22cc031930 con 0x7f22dc0adf20 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 -- 172.21.15.53:0/2380462983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0adf20 msgr2=0x7f22dc070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 --2- 172.21.15.53:0/2380462983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0adf20 0x7f22dc070a30 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f22cc0097a0 tx=0x7f22cc02b840 comp rx=0 tx=0).stop 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 -- 172.21.15.53:0/2380462983 shutdown_connections 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 --2- 172.21.15.53:0/2380462983 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0adf20 0x7f22dc070a30 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 -- 172.21.15.53:0/2380462983 >> 172.21.15.53:0/2380462983 conn(0x7f22dc06a7d0 msgr2=0x7f22dc06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 -- 172.21.15.53:0/2380462983 shutdown_connections 2024-01-16T15:23:40.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.978+0000 7f22e2c00640 1 -- 172.21.15.53:0/2380462983 wait complete. 2024-01-16T15:23:40.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.980+0000 7f22e2c00640 1 Processor -- start 2024-01-16T15:23:40.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.980+0000 7f22e2c00640 1 -- start start 2024-01-16T15:23:40.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e2c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22dc0adf20 0x7f22dc15aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:40.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e2c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0ccb50 0x7f22dc0c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e2c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22dc15bf10 0x7f22dc0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e2c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0c4360 con 0x7f22dc0ccb50 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e2c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc1f4170 con 0x7f22dc0adf20 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e2c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc1f4450 con 0x7f22dc15bf10 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0ccb50 0x7f22dc0c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0ccb50 0x7f22dc0c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:50152/0 (socket says 172.21.15.53:50152) 2024-01-16T15:23:40.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.981+0000 7f22e0e00640 1 -- 172.21.15.53:0/2476864160 learned_addr learned my addr 172.21.15.53:0/2476864160 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:23:40.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e0e00640 1 -- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22dc15bf10 msgr2=0x7f22dc0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:40.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e0e00640 1 --2- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f22dc15bf10 0x7f22dc0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:40.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e0e00640 1 -- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22dc0adf20 msgr2=0x7f22dc15aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:23:40.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e0e00640 1 --2- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f22dc0adf20 0x7f22dc15aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:23:40.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e0e00640 1 -- 172.21.15.53:0/2476864160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22cc009400 con 0x7f22dc0ccb50 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e0e00640 1 --2- 172.21.15.53:0/2476864160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f22dc0ccb50 0x7f22dc0c3e20 secure :-1 s=READY pgs=1833 cs=0 l=1 rev1=1 crypto rx=0x7f22c400b3f0 tx=0x7f22c400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22d9600640 1 -- 172.21.15.53:0/2476864160 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c4011830 con 0x7f22dc0ccb50 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22d9600640 1 -- 172.21.15.53:0/2476864160 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22c4011990 con 0x7f22dc0ccb50 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22d9600640 1 -- 172.21.15.53:0/2476864160 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22c4011d40 con 0x7f22dc0ccb50 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e2c00640 1 -- 172.21.15.53:0/2476864160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22dc1f4730 con 0x7f22dc0ccb50 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.982+0000 7f22e2c00640 1 -- 172.21.15.53:0/2476864160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22dc1f4cf0 con 0x7f22dc0ccb50 2024-01-16T15:23:40.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.984+0000 7f22e2c00640 1 -- 172.21.15.53:0/2476864160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22dc067450 con 0x7f22dc0ccb50 2024-01-16T15:23:40.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.985+0000 7f22d9600640 1 -- 172.21.15.53:0/2476864160 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f22c401a400 con 0x7f22dc0ccb50 2024-01-16T15:23:40.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.986+0000 7f22d9600640 1 --2- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22b406c630 0x7f22b406eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:23:40.981 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.986+0000 7f22d9600640 1 -- 172.21.15.53:0/2476864160 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f22c4088a90 con 0x7f22dc0ccb50 2024-01-16T15:23:40.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.986+0000 7f22e1800640 1 --2- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22b406c630 0x7f22b406eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:23:40.984 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.988+0000 7f22e1800640 1 --2- 172.21.15.53:0/2476864160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f22b406c630 0x7f22b406eaf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f22cc036000 tx=0x7f22cc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:23:40.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:40.991+0000 7f22d9600640 1 -- 172.21.15.53:0/2476864160 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f22c4054fb0 con 0x7f22dc0ccb50 2024-01-16T15:23:41.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:23:41.225+0000 7f22e2c00640 1 -- 172.21.15.53:0/2476864160 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs clone cancel", "vol_name": "cephfs", "clone_name": "subvol_clone52246", "target": ["mon-mgr", ""]}) v1 -- 0x7f22dc15f170 con 0x7f22b406c630 2024-01-16T15:23:57.630 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:23:57.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:27.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:27.780 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:58.133 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:24:58.142 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:28.275 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:28.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-01-16T15:25:40.863 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2024-01-16T15:25:41.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.082+0000 7f2195e00640 1 Processor -- start 2024-01-16T15:25:41.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.082+0000 7f2195e00640 1 -- start start 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.084+0000 7f2195e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f219014b2c0 0x7f21900cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.084+0000 7f2195e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 0x7f2190070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.084+0000 7f2195e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21900cfbf0 con 0x7f2190070ea0 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.084+0000 7f2195e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219014b6a0 con 0x7f219014b2c0 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.084+0000 7f2195e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2190071690 con 0x7f21900cf7d0 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.085+0000 7f218f400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2190070ea0 0x7f21900b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42674/0 (socket says 172.21.15.53:42674) 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.085+0000 7f218f400640 1 -- 172.21.15.53:0/763313413 learned_addr learned my addr 172.21.15.53:0/763313413 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.085+0000 7f218ea00640 1 --2- 172.21.15.53:0/763313413 >> v2:172.21.15.146:3300/0 conn(0x7f219014b2c0 0x7f21900cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.085+0000 7f2187e00640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 0x7f2190070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.085+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298388004 0 0) 0x7f21900cfbf0 con 0x7f2190070ea0 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.085+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2170003520 con 0x7f2190070ea0 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189163818 0 0) 0x7f2170003520 con 0x7f2190070ea0 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 msgr2=0x7f2190070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 0x7f2190070960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 >> v2:172.21.15.146:3300/0 conn(0x7f219014b2c0 msgr2=0x7f21900cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 --2- 172.21.15.53:0/763313413 >> v2:172.21.15.146:3300/0 conn(0x7f219014b2c0 0x7f21900cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190071970 con 0x7f2190070ea0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f21880019f0 con 0x7f2190070ea0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 >> v1:172.21.15.53:6789/0 conn(0x7f2190070ea0 legacy=0x7f21900b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 0x7f2170003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2170003ef0 0x7f21700047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2170004d00 0x7f2170005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f219014b6a0 con 0x7f2170004d00 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2190071690 con 0x7f2170003ef0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f217000b980 con 0x7f21900cf7d0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218ea00640 1 --2- 172.21.15.53:0/763313413 >> v2:172.21.15.146:3300/0 conn(0x7f219014b2c0 0x7f21900cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.086+0000 7f218f400640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2170004d00 0x7f2170005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f2187e00640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2170003ef0 0x7f21700047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f218ea00640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 0x7f2170003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f218f400640 1 -- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 msgr2=0x7f2170003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f218f400640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21900cf7d0 0x7f2170003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f218f400640 1 -- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2170003ef0 msgr2=0x7f21700047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f218f400640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2170003ef0 0x7f21700047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.087+0000 7f218f400640 1 -- 172.21.15.53:0/763313413 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2170003520 con 0x7f2170004d00 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.088+0000 7f218f400640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2170004d00 0x7f2170005180 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f218800a860 tx=0x7f21700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a362d6a6ce6bf469 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.088+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188034630 con 0x7f2170004d00 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.088+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218802f070 con 0x7f2170004d00 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.088+0000 7f218e000640 1 -- 172.21.15.53:0/763313413 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188034a30 con 0x7f2170004d00 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f2195e00640 1 -- 172.21.15.53:0/763313413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2170004d00 msgr2=0x7f2170005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f2195e00640 1 --2- 172.21.15.53:0/763313413 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2170004d00 0x7f2170005180 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7f218800a860 tx=0x7f21700057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f218f400640 1 -- 172.21.15.53:0/763313413 reap_dead start 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f2195e00640 1 -- 172.21.15.53:0/763313413 shutdown_connections 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f2195e00640 1 -- 172.21.15.53:0/763313413 >> 172.21.15.53:0/763313413 conn(0x7f219006a800 msgr2=0x7f219006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f2195e00640 1 -- 172.21.15.53:0/763313413 shutdown_connections 2024-01-16T15:25:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.089+0000 7f2195e00640 1 -- 172.21.15.53:0/763313413 wait complete. 2024-01-16T15:25:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.091+0000 7f2195e00640 1 Processor -- start 2024-01-16T15:25:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.091+0000 7f2195e00640 1 -- start start 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f2195e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2170004d00 0x7f21900ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f2195e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f219013ebc0 0x7f21901633a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f2195e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 0x7f21901f9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f2195e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2190164e00 con 0x7f21901638e0 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f2195e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2190163d60 con 0x7f219013ebc0 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f2195e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2190164040 con 0x7f2170004d00 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f218f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 0x7f21901f9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f218f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 0x7f21901f9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43748/0 (socket says 172.21.15.53:43748) 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.092+0000 7f218f400640 1 -- 172.21.15.53:0/1690458232 learned_addr learned my addr 172.21.15.53:0/1690458232 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f2187e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f219013ebc0 0x7f21901633a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218ea00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2170004d00 0x7f21900ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218f400640 1 -- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2170004d00 msgr2=0x7f21900ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218f400640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2170004d00 0x7f21900ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218f400640 1 -- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f219013ebc0 msgr2=0x7f21901633a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218f400640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f219013ebc0 0x7f21901633a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218f400640 1 -- 172.21.15.53:0/1690458232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2190164320 con 0x7f21901638e0 2024-01-16T15:25:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f218f400640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 0x7f21901f9f00 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f218800a860 tx=0x7f2188001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:41.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188034640 con 0x7f21901638e0 2024-01-16T15:25:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21880347a0 con 0x7f21901638e0 2024-01-16T15:25:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2190164600 con 0x7f21901638e0 2024-01-16T15:25:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.093+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2188034b10 con 0x7f21901638e0 2024-01-16T15:25:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.094+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21900cd430 con 0x7f21901638e0 2024-01-16T15:25:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.095+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21900cd430 con 0x7f21901638e0 2024-01-16T15:25:41.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.096+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2188031030 con 0x7f21901638e0 2024-01-16T15:25:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.097+0000 7f2187400640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f216806c5e0 0x7f216806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.097+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f21880aad90 con 0x7f21901638e0 2024-01-16T15:25:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.097+0000 7f218ea00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f216806c5e0 0x7f216806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.099+0000 7f218ea00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f216806c5e0 0x7f216806eaa0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2180009c20 tx=0x7f2180008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:41.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.104+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21880772b0 con 0x7f21901638e0 2024-01-16T15:25:41.331 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.333+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f21900cbc40 con 0x7f216806c5e0 2024-01-16T15:25:41.335 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.338+0000 7f2187400640 1 -- 172.21.15.53:0/1690458232 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2180012ea0 con 0x7f216806c5e0 2024-01-16T15:25:41.335 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:25:41.335 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.336 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "cephfs" 2024-01-16T15:25:41.336 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:25:41.336 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:25:41.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f216806c5e0 msgr2=0x7f216806eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.338 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f216806c5e0 0x7f216806eaa0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2180009c20 tx=0x7f2180008040 comp rx=0 tx=0).stop 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 msgr2=0x7f21901f9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 0x7f21901f9f00 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7f218800a860 tx=0x7f2188001d80 comp rx=0 tx=0).stop 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 shutdown_connections 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f216806c5e0 0x7f216806eaa0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21901638e0 0x7f21901f9f00 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f219013ebc0 0x7f21901633a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 --2- 172.21.15.53:0/1690458232 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2170004d00 0x7f21900ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 >> 172.21.15.53:0/1690458232 conn(0x7f219006a800 msgr2=0x7f219006c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 shutdown_connections 2024-01-16T15:25:41.339 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.341+0000 7f2195e00640 1 -- 172.21.15.53:0/1690458232 wait complete. 2024-01-16T15:25:41.357 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume ls cephfs --format json 2024-01-16T15:25:41.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.583+0000 7f4911e00640 1 Processor -- start 2024-01-16T15:25:41.581 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.583+0000 7f4911e00640 1 -- start start 2024-01-16T15:25:41.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f4911e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f490c13df80 0x7f490c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f4911e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f4911e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f490c15f9c0 0x7f490c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f4911e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490c06c470 con 0x7f490c15f9c0 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f4911e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f490c06c750 con 0x7f490c13df80 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f4911e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490c06ca30 con 0x7f490c13e360 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f490b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f490c15f9c0 0x7f490c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f490b400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f490c15f9c0 0x7f490c0cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43750/0 (socket says 172.21.15.53:43750) 2024-01-16T15:25:41.583 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.585+0000 7f490b400640 1 -- 172.21.15.53:0/2518243885 learned_addr learned my addr 172.21.15.53:0/2518243885 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f4903e00640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490aa00640 1 --2- 172.21.15.53:0/2518243885 >> v2:172.21.15.146:3300/0 conn(0x7f490c13df80 0x7f490c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490b400640 1 -- 172.21.15.53:0/2518243885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 msgr2=0x7f490c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490b400640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490b400640 1 -- 172.21.15.53:0/2518243885 >> v2:172.21.15.146:3300/0 conn(0x7f490c13df80 msgr2=0x7f490c15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490b400640 1 --2- 172.21.15.53:0/2518243885 >> v2:172.21.15.146:3300/0 conn(0x7f490c13df80 0x7f490c15bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490b400640 1 -- 172.21.15.53:0/2518243885 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490c06cd10 con 0x7f490c15f9c0 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f4903e00640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:41.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.586+0000 7f490aa00640 1 --2- 172.21.15.53:0/2518243885 >> v2:172.21.15.146:3300/0 conn(0x7f490c13df80 0x7f490c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:41.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.587+0000 7f490b400640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c15f9c0 0x7f490c0cf7d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f490c072770 tx=0x7f490402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42a573c56e1290e7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:41.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.587+0000 7f490a000640 1 -- 172.21.15.53:0/2518243885 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490402ca80 con 0x7f490c15f9c0 2024-01-16T15:25:41.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f490a000640 1 -- 172.21.15.53:0/2518243885 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490402cbe0 con 0x7f490c15f9c0 2024-01-16T15:25:41.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f490a000640 1 -- 172.21.15.53:0/2518243885 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49040343f0 con 0x7f490c15f9c0 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 -- 172.21.15.53:0/2518243885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c15f9c0 msgr2=0x7f490c0cf7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c15f9c0 0x7f490c0cf7d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7f490c072770 tx=0x7f490402bee0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 -- 172.21.15.53:0/2518243885 shutdown_connections 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c15f9c0 0x7f490c0cf7d0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 --2- 172.21.15.53:0/2518243885 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 --2- 172.21.15.53:0/2518243885 >> v2:172.21.15.146:3300/0 conn(0x7f490c13df80 0x7f490c15bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 -- 172.21.15.53:0/2518243885 >> 172.21.15.53:0/2518243885 conn(0x7f490c06a800 msgr2=0x7f490c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.588+0000 7f4911e00640 1 -- 172.21.15.53:0/2518243885 shutdown_connections 2024-01-16T15:25:41.586 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.589+0000 7f4911e00640 1 -- 172.21.15.53:0/2518243885 wait complete. 2024-01-16T15:25:41.587 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.590+0000 7f4911e00640 1 Processor -- start 2024-01-16T15:25:41.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.591+0000 7f4911e00640 1 -- start start 2024-01-16T15:25:41.590 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4911e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 0x7f490c0b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4911e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0aac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4911e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f490c15f9c0 0x7f490c0ab190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4911e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0b3840 con 0x7f490c13df80 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4911e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0a98b0 con 0x7f490c15f9c0 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4911e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f490c0a8810 con 0x7f490c13e360 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f490aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 0x7f490c0b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f490aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 0x7f490c0b3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:43752/0 (socket says 172.21.15.53:43752) 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f490aa00640 1 -- 172.21.15.53:0/3306622613 learned_addr learned my addr 172.21.15.53:0/3306622613 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f4903e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0aac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.592+0000 7f490b400640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f490c15f9c0 0x7f490c0ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490aa00640 1 -- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 msgr2=0x7f490c0aac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490aa00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0aac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490aa00640 1 -- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f490c15f9c0 msgr2=0x7f490c0ab190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490aa00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f490c15f9c0 0x7f490c0ab190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490aa00640 1 -- 172.21.15.53:0/3306622613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4904009b30 con 0x7f490c13df80 2024-01-16T15:25:41.591 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490b400640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f490c15f9c0 0x7f490c0ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f4903e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f490aa00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 0x7f490c0b3300 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c7f0 tx=0x7f48fc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc007820 con 0x7f490c13df80 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc007980 con 0x7f490c13df80 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc007cf0 con 0x7f490c13df80 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490c0a8af0 con 0x7f490c13df80 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.593+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490c0a90b0 con 0x7f490c13df80 2024-01-16T15:25:41.592 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.594+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490c0675d0 con 0x7f490c13df80 2024-01-16T15:25:41.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.596+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f48fc0113f0 con 0x7f490c13df80 2024-01-16T15:25:41.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.596+0000 7f4903400640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e406c630 0x7f48e406eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:41.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.597+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f48fc088f50 con 0x7f490c13df80 2024-01-16T15:25:41.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.597+0000 7f4903e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e406c630 0x7f48e406eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:41.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.598+0000 7f4903e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e406c630 0x7f48e406eaf0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f490c15aa00 tx=0x7f48f4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:41.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.601+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f48fc055470 con 0x7f490c13df80 2024-01-16T15:25:41.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.823+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume ls", "vol_name": "cephfs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f490c1408d0 con 0x7f48e406c630 2024-01-16T15:25:41.830 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:41.833+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:41.830 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:41.833+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:41.831 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:41.833+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:41.831 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:41.833+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:41.831 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:41.833+0000 7f6135000640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:25:41.865 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.868+0000 7f4903400640 1 -- 172.21.15.53:0/3306622613 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+381 (secure 0 0 0) 0x7f48f400f830 con 0x7f48e406c630 2024-01-16T15:25:41.865 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:41.865 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:25:41.865 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "sv1" 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52241" 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52247" 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52246" 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52245" 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.866 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52243" 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52242" 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: }, 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "subvol_clone52244" 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:25:41.867 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e406c630 msgr2=0x7f48e406eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e406c630 0x7f48e406eaf0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f490c15aa00 tx=0x7f48f4008040 comp rx=0 tx=0).stop 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 msgr2=0x7f490c0b3300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 0x7f490c0b3300 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f48fc00c7f0 tx=0x7f48fc00ccb0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 shutdown_connections 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f48e406c630 0x7f48e406eaf0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f490c15f9c0 0x7f490c0ab190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f490c13e360 0x7f490c0aac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 --2- 172.21.15.53:0/3306622613 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f490c13df80 0x7f490c0b3300 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 >> 172.21.15.53:0/3306622613 conn(0x7f490c06a800 msgr2=0x7f490c0afe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 shutdown_connections 2024-01-16T15:25:41.869 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:41.871+0000 7f4911e00640 1 -- 172.21.15.53:0/3306622613 wait complete. 2024-01-16T15:25:41.888 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot ls cephfs sv1 --format json 2024-01-16T15:25:42.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.107+0000 7f825b200640 1 Processor -- start 2024-01-16T15:25:42.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.107+0000 7f825b200640 1 -- start start 2024-01-16T15:25:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f825b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f825413e360 0x7f82540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f825b200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f825415f9c0 0x7f82540cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.106 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f825b200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f825415c5f0 con 0x7f825413df80 2024-01-16T15:25:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f825b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825406c470 con 0x7f825413e360 2024-01-16T15:25:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f825b200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f825406c750 con 0x7f825415f9c0 2024-01-16T15:25:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f8253e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f825413df80 0x7f825415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39872/0 (socket says 172.21.15.53:39872) 2024-01-16T15:25:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f8253e00640 1 -- 172.21.15.53:0/843643094 learned_addr learned my addr 172.21.15.53:0/843643094 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:25:42.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.109+0000 7f8253400640 1 --2- 172.21.15.53:0/843643094 >> v2:172.21.15.53:3300/0 conn(0x7f825413e360 0x7f82540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.110+0000 7f8259200640 1 --2- 172.21.15.53:0/843643094 >> v2:172.21.15.146:3300/0 conn(0x7f825415f9c0 0x7f82540cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.110+0000 7f8253400640 1 -- 172.21.15.53:0/843643094 >> v2:172.21.15.146:3300/0 conn(0x7f825415f9c0 msgr2=0x7f82540cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.110+0000 7f8253400640 1 --2- 172.21.15.53:0/843643094 >> v2:172.21.15.146:3300/0 conn(0x7f825415f9c0 0x7f82540cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.110+0000 7f8253400640 1 -- 172.21.15.53:0/843643094 >> v1:172.21.15.53:6789/0 conn(0x7f825413df80 legacy=0x7f825415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.110+0000 7f8253400640 1 -- 172.21.15.53:0/843643094 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825406ca30 con 0x7f825413e360 2024-01-16T15:25:42.108 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.110+0000 7f8252a00640 1 -- 172.21.15.53:0/843643094 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228041902 0 0) 0x7f825415c5f0 con 0x7f825413df80 2024-01-16T15:25:42.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.111+0000 7f8253400640 1 --2- 172.21.15.53:0/843643094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413e360 0x7f82540cf290 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f82540ccae0 tx=0x7f823c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ec2dd339b98a6a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:42.109 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.111+0000 7f8252a00640 1 -- 172.21.15.53:0/843643094 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c031830 con 0x7f825413e360 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.111+0000 7f8252a00640 1 -- 172.21.15.53:0/843643094 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c031990 con 0x7f825413e360 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.111+0000 7f8252a00640 1 -- 172.21.15.53:0/843643094 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c031d20 con 0x7f825413e360 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 -- 172.21.15.53:0/843643094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413e360 msgr2=0x7f82540cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 --2- 172.21.15.53:0/843643094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413e360 0x7f82540cf290 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f82540ccae0 tx=0x7f823c02b8d0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 -- 172.21.15.53:0/843643094 shutdown_connections 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 --2- 172.21.15.53:0/843643094 >> v2:172.21.15.146:3300/0 conn(0x7f825415f9c0 0x7f82540cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 --2- 172.21.15.53:0/843643094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413e360 0x7f82540cf290 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 -- 172.21.15.53:0/843643094 >> 172.21.15.53:0/843643094 conn(0x7f825406a800 msgr2=0x7f825406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 -- 172.21.15.53:0/843643094 shutdown_connections 2024-01-16T15:25:42.110 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.112+0000 7f825b200640 1 -- 172.21.15.53:0/843643094 wait complete. 2024-01-16T15:25:42.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.114+0000 7f825b200640 1 Processor -- start 2024-01-16T15:25:42.111 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.114+0000 7f825b200640 1 -- start start 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f825b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 0x7f82540af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f825b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f825413e360 0x7f8254150790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f825b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f825415f9c0 0x7f82541f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f825b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82541f1880 con 0x7f825413df80 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f825b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82541f0c90 con 0x7f825413e360 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f825b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82541f0f50 con 0x7f825415f9c0 2024-01-16T15:25:42.113 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f8253e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 0x7f82540af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f8253e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 0x7f82540af1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46306/0 (socket says 172.21.15.53:46306) 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.115+0000 7f8253e00640 1 -- 172.21.15.53:0/2895396501 learned_addr learned my addr 172.21.15.53:0/2895396501 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f8253e00640 1 -- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f825415f9c0 msgr2=0x7f82541f36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f8253e00640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f825415f9c0 0x7f82541f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f8253e00640 1 -- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f825413e360 msgr2=0x7f8254150790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f8253e00640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f825413e360 0x7f8254150790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f8253e00640 1 -- 172.21.15.53:0/2895396501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c009400 con 0x7f825413df80 2024-01-16T15:25:42.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f8253e00640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 0x7f82540af1d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f824400c400 tx=0x7f824400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8244005ce0 con 0x7f825413df80 2024-01-16T15:25:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8244005e40 con 0x7f825413df80 2024-01-16T15:25:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82541f1230 con 0x7f825413df80 2024-01-16T15:25:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.116+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82440072d0 con 0x7f825413df80 2024-01-16T15:25:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.117+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82540af710 con 0x7f825413df80 2024-01-16T15:25:42.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.118+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82540af8a0 con 0x7f825413df80 2024-01-16T15:25:42.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.119+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f82440074f0 con 0x7f825413df80 2024-01-16T15:25:42.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.119+0000 7f824be00640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f822c06c5e0 0x7f822c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.120+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8244088240 con 0x7f825413df80 2024-01-16T15:25:42.117 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.120+0000 7f8253400640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f822c06c5e0 0x7f822c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.118 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.121+0000 7f8253400640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f822c06c5e0 0x7f822c06eaa0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f823c036000 tx=0x7f823c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:42.121 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.124+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8244054760 con 0x7f825413df80 2024-01-16T15:25:42.345 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.347+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot ls", "vol_name": "cephfs", "sub_name": "sv1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f825406cff0 con 0x7f822c06c5e0 2024-01-16T15:25:42.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.379+0000 7f824be00640 1 -- 172.21.15.53:0/2895396501 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f823c031b00 con 0x7f822c06c5e0 2024-01-16T15:25:42.376 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:42.376 INFO:teuthology.orchestra.run.smithi053.stdout:[ 2024-01-16T15:25:42.377 INFO:teuthology.orchestra.run.smithi053.stdout: { 2024-01-16T15:25:42.377 INFO:teuthology.orchestra.run.smithi053.stdout: "name": "ss1" 2024-01-16T15:25:42.377 INFO:teuthology.orchestra.run.smithi053.stdout: } 2024-01-16T15:25:42.377 INFO:teuthology.orchestra.run.smithi053.stdout:] 2024-01-16T15:25:42.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f822c06c5e0 msgr2=0x7f822c06eaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f822c06c5e0 0x7f822c06eaa0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f823c036000 tx=0x7f823c030000 comp rx=0 tx=0).stop 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 msgr2=0x7f82540af1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 0x7f82540af1d0 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f824400c400 tx=0x7f824400c8c0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 shutdown_connections 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f822c06c5e0 0x7f822c06eaa0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f825415f9c0 0x7f82541f36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f825413e360 0x7f8254150790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 --2- 172.21.15.53:0/2895396501 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f825413df80 0x7f82540af1d0 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 >> 172.21.15.53:0/2895396501 conn(0x7f825406a800 msgr2=0x7f82540cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 shutdown_connections 2024-01-16T15:25:42.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.382+0000 7f825b200640 1 -- 172.21.15.53:0/2895396501 wait complete. 2024-01-16T15:25:42.397 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:42.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.619+0000 7febdfe00640 1 Processor -- start 2024-01-16T15:25:42.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.619+0000 7febdfe00640 1 -- start start 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.620+0000 7febdfe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febe01447f0 0x7febe015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe00cccf0 0x7febe015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdfe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe015c5c0 con 0x7febe006c470 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdfe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe006d690 con 0x7febe01447f0 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe00cebb0 con 0x7febe00cccf0 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febe01447f0 0x7febe015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7febe01447f0 0x7febe015c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46308/0 (socket says 172.21.15.53:46308) 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febde000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7febe006c470 0x7febe00cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39880/0 (socket says 172.21.15.53:39880) 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 -- 172.21.15.53:0/4231584515 learned_addr learned my addr 172.21.15.53:0/4231584515 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 -- 172.21.15.53:0/4231584515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe00cccf0 msgr2=0x7febe015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 --2- 172.21.15.53:0/4231584515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe00cccf0 0x7febe015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 -- 172.21.15.53:0/4231584515 >> v1:172.21.15.53:6789/0 conn(0x7febe006c470 legacy=0x7febe00cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdea00640 1 -- 172.21.15.53:0/4231584515 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe006ca10 con 0x7febe01447f0 2024-01-16T15:25:42.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.621+0000 7febdd600640 1 -- 172.21.15.53:0/4231584515 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534916604 0 0) 0x7febe015c5c0 con 0x7febe006c470 2024-01-16T15:25:42.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.622+0000 7febdea00640 1 --2- 172.21.15.53:0/4231584515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe01447f0 0x7febe015c080 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7febe00cecf0 tx=0x7febd002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f87790d7042430 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:42.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdd600640 1 -- 172.21.15.53:0/4231584515 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0031640 con 0x7febe01447f0 2024-01-16T15:25:42.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdd600640 1 -- 172.21.15.53:0/4231584515 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd00317a0 con 0x7febe01447f0 2024-01-16T15:25:42.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdd600640 1 -- 172.21.15.53:0/4231584515 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0031b30 con 0x7febe01447f0 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdfe00640 1 -- 172.21.15.53:0/4231584515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe01447f0 msgr2=0x7febe015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdfe00640 1 --2- 172.21.15.53:0/4231584515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe01447f0 0x7febe015c080 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7febe00cecf0 tx=0x7febd002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdfe00640 1 -- 172.21.15.53:0/4231584515 shutdown_connections 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdfe00640 1 --2- 172.21.15.53:0/4231584515 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe00cccf0 0x7febe015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdfe00640 1 --2- 172.21.15.53:0/4231584515 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe01447f0 0x7febe015c080 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.623+0000 7febdfe00640 1 -- 172.21.15.53:0/4231584515 >> 172.21.15.53:0/4231584515 conn(0x7febe006a3e0 msgr2=0x7febe006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.624+0000 7febdfe00640 1 -- 172.21.15.53:0/4231584515 shutdown_connections 2024-01-16T15:25:42.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.624+0000 7febdfe00640 1 -- 172.21.15.53:0/4231584515 wait complete. 2024-01-16T15:25:42.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.625+0000 7febdfe00640 1 Processor -- start 2024-01-16T15:25:42.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.625+0000 7febdfe00640 1 -- start start 2024-01-16T15:25:42.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febdfe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe006c470 0x7febe01f6c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febdfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cccf0 0x7febe013d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febdfe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe01447f0 0x7febe013df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febdfe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe013e490 con 0x7febe00cccf0 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febdfe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe01f61f0 con 0x7febe01447f0 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febdfe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe013e6c0 con 0x7febe006c470 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.626+0000 7febde000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cccf0 0x7febe013d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cccf0 0x7febe013d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46318/0 (socket says 172.21.15.53:46318) 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 -- 172.21.15.53:0/1781696590 learned_addr learned my addr 172.21.15.53:0/1781696590 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febdea00640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe006c470 0x7febe01f6c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febdf400640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe01447f0 0x7febe013df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 -- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe006c470 msgr2=0x7febe01f6c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe006c470 0x7febe01f6c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 -- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe01447f0 msgr2=0x7febe013df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe01447f0 0x7febe013df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 -- 172.21.15.53:0/1781696590 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd00094d0 con 0x7febe00cccf0 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febdea00640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7febe006c470 0x7febe01f6c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febdf400640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7febe01447f0 0x7febe013df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:42.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febde000640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7febe00cccf0 0x7febe013d860 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7febc800c500 tx=0x7febc800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.627+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8007560 con 0x7febe00cccf0 2024-01-16T15:25:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.628+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febc80047e0 con 0x7febe00cccf0 2024-01-16T15:25:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.628+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febc8004b90 con 0x7febe00cccf0 2024-01-16T15:25:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.628+0000 7febdfe00640 1 -- 172.21.15.53:0/1781696590 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe01f6460 con 0x7febe00cccf0 2024-01-16T15:25:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.628+0000 7febdfe00640 1 -- 172.21.15.53:0/1781696590 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe013e9a0 con 0x7febe00cccf0 2024-01-16T15:25:42.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.629+0000 7febdfe00640 1 -- 172.21.15.53:0/1781696590 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe01f6980 con 0x7febe00cccf0 2024-01-16T15:25:42.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.630+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 49) v1 ==== 91273+0+0 (secure 0 0 0) 0x7febc8004cf0 con 0x7febe00cccf0 2024-01-16T15:25:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.631+0000 7febd6a00640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806c610 0x7febb806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.631+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7febc8087d00 con 0x7febe00cccf0 2024-01-16T15:25:42.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.631+0000 7febdea00640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806c610 0x7febb806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:42.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.632+0000 7febdea00640 1 --2- 172.21.15.53:0/1781696590 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7febb806c610 0x7febb806ead0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7febd0036000 tx=0x7febd003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:42.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.635+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7febc8054220 con 0x7febe00cccf0 2024-01-16T15:25:42.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.855+0000 7febdfe00640 1 -- 172.21.15.53:0/1781696590 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7febe006d2b0 con 0x7febb806c610 2024-01-16T15:25:42.866 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:42.868+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:42.866 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:42.868+0000 7febd6a00640 1 -- 172.21.15.53:0/1781696590 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7febd0031b20 con 0x7febb806c610 2024-01-16T15:25:42.866 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:42.867 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:42.885 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:44.886 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:45.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.124+0000 7f4348a00640 1 Processor -- start 2024-01-16T15:25:45.123 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.124+0000 7f4348a00640 1 -- start start 2024-01-16T15:25:45.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.126+0000 7f4348a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f434415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.124 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.126+0000 7f4348a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434415c5c0 con 0x7f43440cccf0 2024-01-16T15:25:45.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.126+0000 7f4348a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f434406ca10 con 0x7f434406c470 2024-01-16T15:25:45.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.126+0000 7f4348a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f434406d690 con 0x7f43441447f0 2024-01-16T15:25:45.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.127+0000 7f4343e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f43440cccf0 0x7f434415c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39886/0 (socket says 172.21.15.53:39886) 2024-01-16T15:25:45.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.127+0000 7f4343e00640 1 -- 172.21.15.53:0/4003274563 learned_addr learned my addr 172.21.15.53:0/4003274563 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:25:45.125 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.127+0000 7f4343400640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f434415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.127+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739856340 0 0) 0x7f434415c5c0 con 0x7f43440cccf0 2024-01-16T15:25:45.126 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.127+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4324003520 con 0x7f43440cccf0 2024-01-16T15:25:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2131884950 0 0) 0x7f4324003520 con 0x7f43440cccf0 2024-01-16T15:25:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 msgr2=0x7f434415c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f434415c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 >> v1:172.21.15.146:6789/0 conn(0x7f434406c470 legacy=0x7f43440cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f434406ccf0 con 0x7f43440cccf0 2024-01-16T15:25:45.127 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f433c0019f0 con 0x7f43440cccf0 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4343400640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f434415c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 >> v1:172.21.15.53:6789/0 conn(0x7f43440cccf0 legacy=0x7f434415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f43240039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f43240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4324004d00 0x7f4324005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434406d690 con 0x7f4324003f30 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f432400bb40 con 0x7f4324004d00 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f432400bd00 con 0x7f43441447f0 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f433a800640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f43240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f4343400640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f43240039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.128+0000 7f433a800640 1 -- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 msgr2=0x7f43240039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.129+0000 7f4343e00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4324004d00 0x7f4324005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.129+0000 7f433a800640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441447f0 0x7f43240039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.128 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.129+0000 7f433a800640 1 -- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4324004d00 msgr2=0x7f4324005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:45.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.129+0000 7f433a800640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4324004d00 0x7f4324005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.129+0000 7f433a800640 1 -- 172.21.15.53:0/4003274563 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4334009400 con 0x7f4324003f30 2024-01-16T15:25:45.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.129+0000 7f4343e00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4324004d00 0x7f4324005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.130+0000 7f433a800640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f43240047c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f432c001750 tx=0x7f43240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49a9acc59e7653bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.130+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c0052d0 con 0x7f4324003f30 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.130+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f432c02e050 con 0x7f4324003f30 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.130+0000 7f4342a00640 1 -- 172.21.15.53:0/4003274563 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f432c003830 con 0x7f4324003f30 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4348a00640 1 -- 172.21.15.53:0/4003274563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 msgr2=0x7f43240047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4348a00640 1 --2- 172.21.15.53:0/4003274563 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f43240047c0 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7f432c001750 tx=0x7f43240057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4343e00640 1 -- 172.21.15.53:0/4003274563 reap_dead start 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4348a00640 1 -- 172.21.15.53:0/4003274563 shutdown_connections 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4348a00640 1 -- 172.21.15.53:0/4003274563 >> 172.21.15.53:0/4003274563 conn(0x7f434406a3e0 msgr2=0x7f434406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4348a00640 1 -- 172.21.15.53:0/4003274563 shutdown_connections 2024-01-16T15:25:45.130 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.131+0000 7f4348a00640 1 -- 172.21.15.53:0/4003274563 wait complete. 2024-01-16T15:25:45.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.133+0000 7f4348a00640 1 Processor -- start 2024-01-16T15:25:45.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.133+0000 7f4348a00640 1 -- start start 2024-01-16T15:25:45.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4348a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f434415b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4348a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43441f3720 0x7f4344070980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4348a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441f3c40 0x7f43440ad450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4348a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f434415bde0 con 0x7f4324003f30 2024-01-16T15:25:45.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4348a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4344070ec0 con 0x7f43441f3720 2024-01-16T15:25:45.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4348a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4344071190 con 0x7f43441f3c40 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4343400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f434415b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4343400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f434415b8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46330/0 (socket says 172.21.15.53:46330) 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.134+0000 7f4343400640 1 -- 172.21.15.53:0/4152411481 learned_addr learned my addr 172.21.15.53:0/4152411481 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f433a800640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43441f3720 0x7f4344070980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343e00640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441f3c40 0x7f43440ad450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343400640 1 -- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441f3c40 msgr2=0x7f43440ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343400640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441f3c40 0x7f43440ad450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343400640 1 -- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43441f3720 msgr2=0x7f4344070980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343400640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43441f3720 0x7f4344070980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343400640 1 -- 172.21.15.53:0/4152411481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4344071470 con 0x7f4324003f30 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f433a800640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f43441f3720 0x7f4344070980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343e00640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f43441f3c40 0x7f43440ad450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:45.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4343400640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4324003f30 0x7f434415b8a0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7f4334011900 tx=0x7f4334011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:45.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f433400fbd0 con 0x7f4324003f30 2024-01-16T15:25:45.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f433400fd30 con 0x7f4324003f30 2024-01-16T15:25:45.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f4348a00640 1 -- 172.21.15.53:0/4152411481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43440ac970 con 0x7f4324003f30 2024-01-16T15:25:45.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.135+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4334015660 con 0x7f4324003f30 2024-01-16T15:25:45.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.136+0000 7f4348a00640 1 -- 172.21.15.53:0/4152411481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43440acf30 con 0x7f4324003f30 2024-01-16T15:25:45.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.137+0000 7f4348a00640 1 -- 172.21.15.53:0/4152411481 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4344067450 con 0x7f4324003f30 2024-01-16T15:25:45.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.138+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f43340157c0 con 0x7f4324003f30 2024-01-16T15:25:45.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.139+0000 7f433be00640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431c06c6a0 0x7f431c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:45.137 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.139+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f43340896c0 con 0x7f4324003f30 2024-01-16T15:25:45.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.140+0000 7f433a800640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431c06c6a0 0x7f431c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:45.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.141+0000 7f433a800640 1 --2- 172.21.15.53:0/4152411481 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f431c06c6a0 0x7f431c06eb60 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f432c033000 tx=0x7f432c009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:45.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.143+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4334055bd0 con 0x7f4324003f30 2024-01-16T15:25:45.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.371+0000 7f4348a00640 1 -- 172.21.15.53:0/4152411481 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4344144bd0 con 0x7f431c06c6a0 2024-01-16T15:25:45.385 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:45.388+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:45.386 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:45.388+0000 7f433be00640 1 -- 172.21.15.53:0/4152411481 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f432c005490 con 0x7f431c06c6a0 2024-01-16T15:25:45.386 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:45.386 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:45.406 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:47.408 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:47.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.653+0000 7fea7c800640 1 Processor -- start 2024-01-16T15:25:47.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.654+0000 7fea7c800640 1 -- start start 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.655+0000 7fea7c800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fea7415d610 0x7fea740ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.655+0000 7fea7c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea740d17e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.655+0000 7fea7c800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea740b3400 con 0x7fea740d1d20 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.655+0000 7fea7c800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea740d1390 con 0x7fea7415d610 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.655+0000 7fea7c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea740d21b0 con 0x7fea740ced60 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.655+0000 7fea79e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fea7415d610 0x7fea740ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.656+0000 7fea79e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fea7415d610 0x7fea740ce820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46336/0 (socket says 172.21.15.53:46336) 2024-01-16T15:25:47.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.656+0000 7fea79e00640 1 -- 172.21.15.53:0/2948058722 learned_addr learned my addr 172.21.15.53:0/2948058722 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:47.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.656+0000 7fea79e00640 1 -- 172.21.15.53:0/2948058722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 msgr2=0x7fea740d17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:47.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.656+0000 7fea79e00640 1 --2- 172.21.15.53:0/2948058722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea740d17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:47.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.656+0000 7fea79e00640 1 -- 172.21.15.53:0/2948058722 >> v1:172.21.15.146:6789/0 conn(0x7fea740d1d20 legacy=0x7fea740cf830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:47.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.656+0000 7fea79e00640 1 -- 172.21.15.53:0/2948058722 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea740b27f0 con 0x7fea7415d610 2024-01-16T15:25:47.655 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.657+0000 7fea79e00640 1 --2- 172.21.15.53:0/2948058722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea7415d610 0x7fea740ce820 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fea64009870 tx=0x7fea6402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7946c70cc652f567 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:47.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea78a00640 1 -- 172.21.15.53:0/2948058722 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64031640 con 0x7fea7415d610 2024-01-16T15:25:47.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea78a00640 1 -- 172.21.15.53:0/2948058722 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea640317a0 con 0x7fea7415d610 2024-01-16T15:25:47.656 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea78a00640 1 -- 172.21.15.53:0/2948058722 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea64031b30 con 0x7fea7415d610 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 -- 172.21.15.53:0/2948058722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea7415d610 msgr2=0x7fea740ce820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 --2- 172.21.15.53:0/2948058722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea7415d610 0x7fea740ce820 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fea64009870 tx=0x7fea6402b8f0 comp rx=0 tx=0).stop 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 -- 172.21.15.53:0/2948058722 shutdown_connections 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 --2- 172.21.15.53:0/2948058722 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea740d17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 --2- 172.21.15.53:0/2948058722 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea7415d610 0x7fea740ce820 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 -- 172.21.15.53:0/2948058722 >> 172.21.15.53:0/2948058722 conn(0x7fea7409c540 msgr2=0x7fea740d3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.658+0000 7fea7c800640 1 -- 172.21.15.53:0/2948058722 shutdown_connections 2024-01-16T15:25:47.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.659+0000 7fea7c800640 1 -- 172.21.15.53:0/2948058722 wait complete. 2024-01-16T15:25:47.658 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.660+0000 7fea7c800640 1 Processor -- start 2024-01-16T15:25:47.659 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.660+0000 7fea7c800640 1 -- start start 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.661+0000 7fea7c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea74159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.661+0000 7fea7c800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea740d1d20 0x7fea740b7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.661+0000 7fea7c800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea7415d610 0x7fea740c96b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.661+0000 7fea7c800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740cd6f0 con 0x7fea740d1d20 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.661+0000 7fea7c800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea740ca630 con 0x7fea7415d610 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.661+0000 7fea7c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea74159cb0 con 0x7fea740ced60 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea740d1d20 0x7fea740b7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea740d1d20 0x7fea740b7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46352/0 (socket says 172.21.15.53:46352) 2024-01-16T15:25:47.660 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 -- 172.21.15.53:0/2046672094 learned_addr learned my addr 172.21.15.53:0/2046672094 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:47.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79e00640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea74159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:47.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea7a800640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea7415d610 0x7fea740c96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:47.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 -- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 msgr2=0x7fea74159770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:47.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea74159770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:47.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 -- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea7415d610 msgr2=0x7fea740c96b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:47.662 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea7415d610 0x7fea740c96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79400640 1 -- 172.21.15.53:0/2046672094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea640094d0 con 0x7fea740d1d20 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.662+0000 7fea79e00640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea740ced60 0x7fea74159770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.663+0000 7fea79400640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea740d1d20 0x7fea740b7180 secure :-1 s=READY pgs=1849 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00b4a0 tx=0x7fea5c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.663+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c010610 con 0x7fea740d1d20 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.663+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c010770 con 0x7fea740d1d20 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.663+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c010b20 con 0x7fea740d1d20 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.663+0000 7fea7c800640 1 -- 172.21.15.53:0/2046672094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea7415eed0 con 0x7fea740d1d20 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.663+0000 7fea7c800640 1 -- 172.21.15.53:0/2046672094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7415f490 con 0x7fea740d1d20 2024-01-16T15:25:47.663 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.664+0000 7fea7c800640 1 -- 172.21.15.53:0/2046672094 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea740675d0 con 0x7fea740d1d20 2024-01-16T15:25:47.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.665+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fea5c010c80 con 0x7fea740d1d20 2024-01-16T15:25:47.664 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.666+0000 7fea72000640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4c06c6a0 0x7fea4c06eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:47.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.666+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fea5c088880 con 0x7fea740d1d20 2024-01-16T15:25:47.665 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.667+0000 7fea79e00640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4c06c6a0 0x7fea4c06eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:47.666 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.668+0000 7fea79e00640 1 --2- 172.21.15.53:0/2046672094 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4c06c6a0 0x7fea4c06eb60 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fea64036000 tx=0x7fea6403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:47.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.671+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea5c054d90 con 0x7fea740d1d20 2024-01-16T15:25:47.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.903+0000 7fea7c800640 1 -- 172.21.15.53:0/2046672094 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fea7408d240 con 0x7fea4c06c6a0 2024-01-16T15:25:47.915 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:47.917+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:47.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:47.917+0000 7fea72000640 1 -- 172.21.15.53:0/2046672094 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea7408d240 con 0x7fea4c06c6a0 2024-01-16T15:25:47.915 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:47.916 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:49.934 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:50.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.172+0000 7f9667200640 1 Processor -- start 2024-01-16T15:25:50.171 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.173+0000 7f9667200640 1 -- start start 2024-01-16T15:25:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.174+0000 7f9667200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966806c470 0x7f96680cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f9667200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966806ca10 con 0x7f96681447f0 2024-01-16T15:25:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f9667200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f966806ccf0 con 0x7f966806dab0 2024-01-16T15:25:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f9667200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966806d690 con 0x7f966806c470 2024-01-16T15:25:50.173 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f9665e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f96681447f0 0x7f966815c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39888/0 (socket says 172.21.15.53:39888) 2024-01-16T15:25:50.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f9665e00640 1 -- 172.21.15.53:0/2380142153 learned_addr learned my addr 172.21.15.53:0/2380142153 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:25:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f965fe00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966806c470 0x7f96680cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.175+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4277495495 0 0) 0x7f966806ca10 con 0x7f96681447f0 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9648003520 con 0x7f96681447f0 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 311730782 0 0) 0x7f9648003520 con 0x7f96681447f0 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966806c470 msgr2=0x7f96680cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966806c470 0x7f96680cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 >> v1:172.21.15.146:6789/0 conn(0x7f966806dab0 legacy=0x7f966815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966806cfd0 con 0x7f96681447f0 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9658001670 con 0x7f96681447f0 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 >> v1:172.21.15.53:6789/0 conn(0x7f96681447f0 legacy=0x7f966815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96480039a0 0x7f9648003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96480047c0 0x7f9648004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96480051d0 0x7f964800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966806d690 con 0x7f96480047c0 2024-01-16T15:25:50.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964800c0d0 con 0x7f96480051d0 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964800c290 con 0x7f96480039a0 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.176+0000 7f965fe00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966806c470 0x7f96680cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f965fe00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96480047c0 0x7f9648004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f9666800640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96480051d0 0x7f964800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f9665e00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96480039a0 0x7f9648003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f965fe00640 1 -- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96480039a0 msgr2=0x7f9648003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f965fe00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f96480039a0 0x7f9648003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f965fe00640 1 -- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96480051d0 msgr2=0x7f964800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f965fe00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f96480051d0 0x7f964800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.177 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.177+0000 7f965fe00640 1 -- 172.21.15.53:0/2380142153 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648003520 con 0x7f96480047c0 2024-01-16T15:25:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.178+0000 7f965fe00640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96480047c0 0x7f9648004c40 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f9650006fd0 tx=0x7f96480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecab4d4492122833 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650033600 con 0x7f96480047c0 2024-01-16T15:25:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f965002f050 con 0x7f96480047c0 2024-01-16T15:25:50.178 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9665400640 1 -- 172.21.15.53:0/2380142153 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650033980 con 0x7f96480047c0 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9667200640 1 -- 172.21.15.53:0/2380142153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96480047c0 msgr2=0x7f9648004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9667200640 1 --2- 172.21.15.53:0/2380142153 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96480047c0 0x7f9648004c40 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f9650006fd0 tx=0x7f96480057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9666800640 1 -- 172.21.15.53:0/2380142153 reap_dead start 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9667200640 1 -- 172.21.15.53:0/2380142153 shutdown_connections 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.179+0000 7f9667200640 1 -- 172.21.15.53:0/2380142153 >> 172.21.15.53:0/2380142153 conn(0x7f966806a3e0 msgr2=0x7f966806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.180+0000 7f9667200640 1 -- 172.21.15.53:0/2380142153 shutdown_connections 2024-01-16T15:25:50.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.180+0000 7f9667200640 1 -- 172.21.15.53:0/2380142153 wait complete. 2024-01-16T15:25:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.182+0000 7f9667200640 1 Processor -- start 2024-01-16T15:25:50.180 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.182+0000 7f9667200640 1 -- start start 2024-01-16T15:25:50.182 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9667200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f966815b350 0x7f9668140420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9667200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966813d3e0 0x7f96680b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9667200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96680b13f0 0x7f966813b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9667200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96680b1880 con 0x7f96680b13f0 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9667200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f966813ec10 con 0x7f966815b350 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9667200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f966813eef0 con 0x7f966813d3e0 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9666800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96680b13f0 0x7f966813b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9666800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96680b13f0 0x7f966813b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46368/0 (socket says 172.21.15.53:46368) 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9666800640 1 -- 172.21.15.53:0/1737465805 learned_addr learned my addr 172.21.15.53:0/1737465805 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f965fe00640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966813d3e0 0x7f96680b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.183+0000 7f9665e00640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f966815b350 0x7f9668140420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9666800640 1 -- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966813d3e0 msgr2=0x7f96680b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9666800640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966813d3e0 0x7f96680b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.183 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9666800640 1 -- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f966815b350 msgr2=0x7f9668140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9666800640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f966815b350 0x7f9668140420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9666800640 1 -- 172.21.15.53:0/1737465805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f966813f1d0 con 0x7f96680b13f0 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f965fe00640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f966813d3e0 0x7f96680b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9665e00640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f966815b350 0x7f9668140420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9666800640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f96680b13f0 0x7f966813b320 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f9660007fd0 tx=0x7f966000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660003970 con 0x7f96680b13f0 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9660003af0 con 0x7f96680b13f0 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9660003ea0 con 0x7f96680b13f0 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9667200640 1 -- 172.21.15.53:0/1737465805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f966813f4b0 con 0x7f96680b13f0 2024-01-16T15:25:50.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.184+0000 7f9667200640 1 -- 172.21.15.53:0/1737465805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f966813e8a0 con 0x7f96680b13f0 2024-01-16T15:25:50.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.186+0000 7f9667200640 1 -- 172.21.15.53:0/1737465805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96681f2e90 con 0x7f96680b13f0 2024-01-16T15:25:50.185 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.187+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f96600050d0 con 0x7f96680b13f0 2024-01-16T15:25:50.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.187+0000 7f965e000640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f964006c6a0 0x7f964006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:50.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.188+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9660088610 con 0x7f96680b13f0 2024-01-16T15:25:50.186 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.188+0000 7f9665e00640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f964006c6a0 0x7f964006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:50.188 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.189+0000 7f9665e00640 1 --2- 172.21.15.53:0/1737465805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f964006c6a0 0x7f964006eb60 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f9658000c00 tx=0x7f9658001150 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:50.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.192+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f96681f2e90 con 0x7f96680b13f0 2024-01-16T15:25:50.429 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.430+0000 7f9667200640 1 -- 172.21.15.53:0/1737465805 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9668144bd0 con 0x7f964006c6a0 2024-01-16T15:25:50.442 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:50.444+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:50.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:50.444+0000 7f965e000640 1 -- 172.21.15.53:0/1737465805 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9658005e60 con 0x7f964006c6a0 2024-01-16T15:25:50.443 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:50.443 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:50.460 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:52.461 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:52.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.704+0000 7f4f49e00640 1 Processor -- start 2024-01-16T15:25:52.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.704+0000 7f4f49e00640 1 -- start start 2024-01-16T15:25:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.706+0000 7f4f49e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4f4413e360 0x7f4f440cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.706+0000 7f4f49e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4f4415f9c0 0x7f4f440cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.706+0000 7f4f49e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f4415c5f0 con 0x7f4f4413df80 2024-01-16T15:25:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.706+0000 7f4f49e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4406c470 con 0x7f4f4413e360 2024-01-16T15:25:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.706+0000 7f4f49e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f4406c750 con 0x7f4f4415f9c0 2024-01-16T15:25:52.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4f4413e360 0x7f4f440cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f4f4413e360 0x7f4f440cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57146/0 (socket says 172.21.15.53:57146) 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 -- 172.21.15.53:0/174392319 learned_addr learned my addr 172.21.15.53:0/174392319 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f43400640 1 --2- 172.21.15.53:0/174392319 >> v2:172.21.15.146:3300/0 conn(0x7f4f4415f9c0 0x7f4f440cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 -- 172.21.15.53:0/174392319 >> v2:172.21.15.146:3300/0 conn(0x7f4f4415f9c0 msgr2=0x7f4f440cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 --2- 172.21.15.53:0/174392319 >> v2:172.21.15.146:3300/0 conn(0x7f4f4415f9c0 0x7f4f440cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 -- 172.21.15.53:0/174392319 >> v1:172.21.15.146:6789/0 conn(0x7f4f4413df80 legacy=0x7f4f4415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:52.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.707+0000 7f4f42000640 1 -- 172.21.15.53:0/174392319 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4406ca30 con 0x7f4f4413e360 2024-01-16T15:25:52.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.709+0000 7f4f42000640 1 --2- 172.21.15.53:0/174392319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4413e360 0x7f4f440cf290 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f4f440ccae0 tx=0x7f4f2c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16249a3b21811e6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:52.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.709+0000 7f4f41600640 1 -- 172.21.15.53:0/174392319 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c030860 con 0x7f4f4413e360 2024-01-16T15:25:52.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.709+0000 7f4f41600640 1 -- 172.21.15.53:0/174392319 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f2c0309c0 con 0x7f4f4413e360 2024-01-16T15:25:52.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.709+0000 7f4f41600640 1 -- 172.21.15.53:0/174392319 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c030d50 con 0x7f4f4413e360 2024-01-16T15:25:52.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 -- 172.21.15.53:0/174392319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4413e360 msgr2=0x7f4f440cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:52.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 --2- 172.21.15.53:0/174392319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4413e360 0x7f4f440cf290 secure :-1 s=READY pgs=1853 cs=0 l=1 rev1=1 crypto rx=0x7f4f440ccae0 tx=0x7f4f2c02b9a0 comp rx=0 tx=0).stop 2024-01-16T15:25:52.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 -- 172.21.15.53:0/174392319 shutdown_connections 2024-01-16T15:25:52.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 --2- 172.21.15.53:0/174392319 >> v2:172.21.15.146:3300/0 conn(0x7f4f4415f9c0 0x7f4f440cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:52.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 --2- 172.21.15.53:0/174392319 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4413e360 0x7f4f440cf290 unknown :-1 s=CLOSED pgs=1853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:52.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 -- 172.21.15.53:0/174392319 >> 172.21.15.53:0/174392319 conn(0x7f4f4406a800 msgr2=0x7f4f4406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:52.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 -- 172.21.15.53:0/174392319 shutdown_connections 2024-01-16T15:25:52.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.710+0000 7f4f49e00640 1 -- 172.21.15.53:0/174392319 wait complete. 2024-01-16T15:25:52.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.712+0000 7f4f49e00640 1 Processor -- start 2024-01-16T15:25:52.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.712+0000 7f4f49e00640 1 -- start start 2024-01-16T15:25:52.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f49e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f4413df80 0x7f4f4414fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f49e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f4413e360 0x7f4f44155d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f49e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4415f9c0 0x7f4f441562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f49e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44158fe0 con 0x7f4f4415f9c0 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f49e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44155260 con 0x7f4f4413e360 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f49e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f44158a20 con 0x7f4f4413df80 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f43400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4415f9c0 0x7f4f441562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f43400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4415f9c0 0x7f4f441562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57160/0 (socket says 172.21.15.53:57160) 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f43400640 1 -- 172.21.15.53:0/3882348724 learned_addr learned my addr 172.21.15.53:0/3882348724 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f42000640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f4413e360 0x7f4f44155d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:52.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.713+0000 7f4f42a00640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f4413df80 0x7f4f4414fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:52.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f43400640 1 -- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f4413df80 msgr2=0x7f4f4414fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:52.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f43400640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f4413df80 0x7f4f4414fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:52.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f43400640 1 -- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f4413e360 msgr2=0x7f4f44155d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f43400640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f4413e360 0x7f4f44155d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f43400640 1 -- 172.21.15.53:0/3882348724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0094d0 con 0x7f4f4415f9c0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f42000640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4f4413e360 0x7f4f44155d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f42a00640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4f4413df80 0x7f4f4414fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f43400640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4f4415f9c0 0x7f4f441562b0 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c007e30 tx=0x7f4f3c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.714+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c014620 con 0x7f4f4415f9c0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.715+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f3c00ee90 con 0x7f4f4415f9c0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.715+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3c0149c0 con 0x7f4f4415f9c0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.715+0000 7f4f49e00640 1 -- 172.21.15.53:0/3882348724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f44158d00 con 0x7f4f4415f9c0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.715+0000 7f4f49e00640 1 -- 172.21.15.53:0/3882348724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f44154e80 con 0x7f4f4415f9c0 2024-01-16T15:25:52.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.716+0000 7f4f49e00640 1 -- 172.21.15.53:0/3882348724 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f44150980 con 0x7f4f4415f9c0 2024-01-16T15:25:52.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.717+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f4f3c014b20 con 0x7f4f4415f9c0 2024-01-16T15:25:52.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.718+0000 7f4f3b400640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4f1c06c670 0x7f4f1c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:52.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.718+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4f3c0891d0 con 0x7f4f4415f9c0 2024-01-16T15:25:52.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.719+0000 7f4f42a00640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4f1c06c670 0x7f4f1c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:52.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.720+0000 7f4f42a00640 1 --2- 172.21.15.53:0/3882348724 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4f1c06c670 0x7f4f1c06eb30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f4f34000f60 tx=0x7f4f34009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:52.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.723+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4f44150980 con 0x7f4f4415f9c0 2024-01-16T15:25:52.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.952+0000 7f4f49e00640 1 -- 172.21.15.53:0/3882348724 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4f4415f060 con 0x7f4f1c06c670 2024-01-16T15:25:52.964 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:52.966+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:52.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:52.966+0000 7f4f3b400640 1 -- 172.21.15.53:0/3882348724 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4f4415f060 con 0x7f4f1c06c670 2024-01-16T15:25:52.964 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:52.965 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:54.981 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:55.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.217+0000 7f39bc400640 1 Processor -- start 2024-01-16T15:25:55.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.217+0000 7f39bc400640 1 -- start start 2024-01-16T15:25:55.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39bc400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 0x7f39b40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39bc400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b415c5f0 con 0x7f39b413df80 2024-01-16T15:25:55.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39bc400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39b406c470 con 0x7f39b415f9c0 2024-01-16T15:25:55.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39bc400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b406c750 con 0x7f39b413e360 2024-01-16T15:25:55.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39b9a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f39b413df80 0x7f39b415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41364/0 (socket says 172.21.15.53:41364) 2024-01-16T15:25:55.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39b9a00640 1 -- 172.21.15.53:0/3194559578 learned_addr learned my addr 172.21.15.53:0/3194559578 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.219+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132196133 0 0) 0x7f39b415c5f0 con 0x7f39b413df80 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994003520 con 0x7f39b413df80 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b9000640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 0x7f39b40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1429751046 0 0) 0x7f3994003520 con 0x7f39b413df80 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 msgr2=0x7f39b40cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 0x7f39b40cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 >> v1:172.21.15.146:6789/0 conn(0x7f39b415f9c0 legacy=0x7f39b40cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b406ca30 con 0x7f39b413df80 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f39a4001670 con 0x7f39b413df80 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 >> v1:172.21.15.53:6789/0 conn(0x7f39b413df80 legacy=0x7f39b415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39940039a0 0x7f3994003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940047c0 0x7f3994004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 0x7f39940051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b406c750 con 0x7f39940039a0 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f399400bae0 con 0x7f39940047c0 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f399400bca0 con 0x7f39b413e360 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.220+0000 7f39b9a00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39940039a0 0x7f3994003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9000640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940047c0 0x7f3994004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9a00640 1 -- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 msgr2=0x7f39940051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39ba400640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 0x7f39940051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9a00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b413e360 0x7f39940051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9a00640 1 -- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940047c0 msgr2=0x7f3994004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9a00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940047c0 0x7f3994004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9a00640 1 -- 172.21.15.53:0/3194559578 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3994003520 con 0x7f39940039a0 2024-01-16T15:25:55.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.221+0000 7f39b9000640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940047c0 0x7f3994004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.222+0000 7f39b9a00640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39940039a0 0x7f3994003dc0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c240 tx=0x7f39940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a255a6f3c1339097 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.222+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a4005ad0 con 0x7f39940039a0 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.222+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a4005c30 con 0x7f39940039a0 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.222+0000 7f39b3e00640 1 -- 172.21.15.53:0/3194559578 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a402c8b0 con 0x7f39940039a0 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39bc400640 1 -- 172.21.15.53:0/3194559578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39940039a0 msgr2=0x7f3994003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39bc400640 1 --2- 172.21.15.53:0/3194559578 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39940039a0 0x7f3994003dc0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7f39a400c240 tx=0x7f39940057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39ba400640 1 -- 172.21.15.53:0/3194559578 reap_dead start 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39bc400640 1 -- 172.21.15.53:0/3194559578 shutdown_connections 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39bc400640 1 -- 172.21.15.53:0/3194559578 >> 172.21.15.53:0/3194559578 conn(0x7f39b406a800 msgr2=0x7f39b406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39bc400640 1 -- 172.21.15.53:0/3194559578 shutdown_connections 2024-01-16T15:25:55.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.223+0000 7f39bc400640 1 -- 172.21.15.53:0/3194559578 wait complete. 2024-01-16T15:25:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.226+0000 7f39bc400640 1 Processor -- start 2024-01-16T15:25:55.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.226+0000 7f39bc400640 1 -- start start 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.227+0000 7f39bc400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940039a0 0x7f39b414d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.227+0000 7f39bc400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39b414ed90 0x7f39b413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.227+0000 7f39bc400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b40702c0 0x7f39b40b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.227+0000 7f39bc400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b40721a0 con 0x7f39b414ed90 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.227+0000 7f39bc400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b4070860 con 0x7f39940039a0 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.227+0000 7f39bc400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39b4070b40 con 0x7f39b40702c0 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39b414ed90 0x7f39b413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39b414ed90 0x7f39b413d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57192/0 (socket says 172.21.15.53:57192) 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 -- 172.21.15.53:0/1491875660 learned_addr learned my addr 172.21.15.53:0/1491875660 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9a00640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940039a0 0x7f39b414d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39ba400640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b40702c0 0x7f39b40b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 -- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b40702c0 msgr2=0x7f39b40b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b40702c0 0x7f39b40b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 -- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940039a0 msgr2=0x7f39b414d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f39940039a0 0x7f39b414d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39b9000640 1 -- 172.21.15.53:0/1491875660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b4070e20 con 0x7f39b414ed90 2024-01-16T15:25:55.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.228+0000 7f39ba400640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f39b40702c0 0x7f39b40b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.229+0000 7f39b9000640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f39b414ed90 0x7f39b413d7f0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f399c00e7e0 tx=0x7f399c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.229+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00c6d0 con 0x7f39b414ed90 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.229+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c00c830 con 0x7f39b414ed90 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.229+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c00cbe0 con 0x7f39b414ed90 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.229+0000 7f39bc400640 1 -- 172.21.15.53:0/1491875660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b4071100 con 0x7f39b414ed90 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.229+0000 7f39bc400640 1 -- 172.21.15.53:0/1491875660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b40b3620 con 0x7f39b414ed90 2024-01-16T15:25:55.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.231+0000 7f39bc400640 1 -- 172.21.15.53:0/1491875660 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b40712f0 con 0x7f39b414ed90 2024-01-16T15:25:55.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.232+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f399c00cd40 con 0x7f39b414ed90 2024-01-16T15:25:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.232+0000 7f39b1600640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f398c06c670 0x7f398c06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.233+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f399c088930 con 0x7f39b414ed90 2024-01-16T15:25:55.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.233+0000 7f39b9a00640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f398c06c670 0x7f398c06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:55.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.234+0000 7f39b9a00640 1 --2- 172.21.15.53:0/1491875660 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f398c06c670 0x7f398c06eb30 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f39a4034000 tx=0x7f39a4009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:55.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.237+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f399c054e40 con 0x7f39b414ed90 2024-01-16T15:25:55.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.464+0000 7f39bc400640 1 -- 172.21.15.53:0/1491875660 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f39b406cff0 con 0x7f398c06c670 2024-01-16T15:25:55.478 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:55.481+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:55.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:55.481+0000 7f39b1600640 1 -- 172.21.15.53:0/1491875660 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f39a40058f0 con 0x7f398c06c670 2024-01-16T15:25:55.479 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:55.480 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:55.493 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:57.494 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:25:57.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.723+0000 7f9a1f600640 1 Processor -- start 2024-01-16T15:25:57.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.723+0000 7f9a1f600640 1 -- start start 2024-01-16T15:25:57.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.725+0000 7f9a1f600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9a1813e360 0x7f9a180cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.725+0000 7f9a1f600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1815c5f0 con 0x7f9a1813df80 2024-01-16T15:25:57.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.725+0000 7f9a1f600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1806c470 con 0x7f9a1815f9c0 2024-01-16T15:25:57.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.725+0000 7f9a1f600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1806c750 con 0x7f9a1813e360 2024-01-16T15:25:57.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.725+0000 7f9a1cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9a1813df80 0x7f9a1815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41374/0 (socket says 172.21.15.53:41374) 2024-01-16T15:25:57.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.725+0000 7f9a1cc00640 1 -- 172.21.15.53:0/150510500 learned_addr learned my addr 172.21.15.53:0/150510500 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:25:57.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17e00640 1 --2- 172.21.15.53:0/150510500 >> v2:172.21.15.146:3300/0 conn(0x7f9a1813e360 0x7f9a180cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773260526 0 0) 0x7f9a1815c5f0 con 0x7f9a1813df80 2024-01-16T15:25:57.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f8003520 con 0x7f9a1813df80 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1057197495 0 0) 0x7f99f8003520 con 0x7f9a1813df80 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 >> v2:172.21.15.146:3300/0 conn(0x7f9a1813e360 msgr2=0x7f9a180cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 --2- 172.21.15.53:0/150510500 >> v2:172.21.15.146:3300/0 conn(0x7f9a1813e360 0x7f9a180cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 >> v1:172.21.15.146:6789/0 conn(0x7f9a1815f9c0 legacy=0x7f9a180cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1806ca30 con 0x7f9a1813df80 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f9a08001690 con 0x7f9a1813df80 2024-01-16T15:25:57.725 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 >> v1:172.21.15.53:6789/0 conn(0x7f9a1813df80 legacy=0x7f9a1815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.726+0000 7f9a17400640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80039c0 0x7f99f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a17400640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f80047c0 0x7f99f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a17400640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99f8005180 0x7f99f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1806c750 con 0x7f99f8005180 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f800c080 con 0x7f99f80039c0 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f800c240 con 0x7f99f80047c0 2024-01-16T15:25:57.726 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1d600640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99f8005180 0x7f99f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a17e00640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f80047c0 0x7f99f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1cc00640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80039c0 0x7f99f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1d600640 1 -- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f80047c0 msgr2=0x7f99f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1d600640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f80047c0 0x7f99f8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1d600640 1 -- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80039c0 msgr2=0x7f99f8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1d600640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80039c0 0x7f99f8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1d600640 1 -- 172.21.15.53:0/150510500 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f8003520 con 0x7f99f8005180 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.727+0000 7f9a1cc00640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f99f80039c0 0x7f99f8003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:25:57.727 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.728+0000 7f9a1d600640 1 -- 172.21.15.53:0/150510500 reap_dead start 2024-01-16T15:25:57.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.728+0000 7f9a1d600640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99f8005180 0x7f99f800b980 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f9a10001e20 tx=0x7f99f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b9bc471b096f9f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:57.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.729+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10003b70 con 0x7f99f8005180 2024-01-16T15:25:57.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.729+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10003cd0 con 0x7f99f8005180 2024-01-16T15:25:57.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.729+0000 7f9a17400640 1 -- 172.21.15.53:0/150510500 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a100052f0 con 0x7f99f8005180 2024-01-16T15:25:57.728 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.729+0000 7f9a1f600640 1 -- 172.21.15.53:0/150510500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99f8005180 msgr2=0x7f99f800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:57.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.729+0000 7f9a1f600640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99f8005180 0x7f99f800b980 secure :-1 s=READY pgs=1859 cs=0 l=1 rev1=1 crypto rx=0x7f9a10001e20 tx=0x7f99f80057f0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.730+0000 7f9a1f600640 1 -- 172.21.15.53:0/150510500 shutdown_connections 2024-01-16T15:25:57.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.730+0000 7f9a1f600640 1 --2- 172.21.15.53:0/150510500 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f99f8005180 0x7f99f800b980 unknown :-1 s=CLOSED pgs=1859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.730+0000 7f9a1f600640 1 -- 172.21.15.53:0/150510500 >> 172.21.15.53:0/150510500 conn(0x7f9a1806a800 msgr2=0x7f9a1806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:25:57.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.730+0000 7f9a1f600640 1 -- 172.21.15.53:0/150510500 shutdown_connections 2024-01-16T15:25:57.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.730+0000 7f9a1f600640 1 -- 172.21.15.53:0/150510500 wait complete. 2024-01-16T15:25:57.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.731+0000 7f9a1f600640 1 Processor -- start 2024-01-16T15:25:57.730 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.732+0000 7f9a1f600640 1 -- start start 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1f600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f8005180 0x7f9a1814d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1f600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a1814e810 0x7f9a181f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1f600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a180af790 0x7f9a1813d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1f600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1814f2b0 con 0x7f9a180af790 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1f600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1814ecf0 con 0x7f9a1814e810 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1f600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a1814efd0 con 0x7f99f8005180 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a180af790 0x7f9a1813d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1d600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a180af790 0x7f9a1813d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57208/0 (socket says 172.21.15.53:57208) 2024-01-16T15:25:57.732 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1d600640 1 -- 172.21.15.53:0/2293961240 learned_addr learned my addr 172.21.15.53:0/2293961240 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1cc00640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f8005180 0x7f9a1814d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a17e00640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a1814e810 0x7f9a181f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.733+0000 7f9a1d600640 1 -- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f8005180 msgr2=0x7f9a1814d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1d600640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f99f8005180 0x7f9a1814d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1d600640 1 -- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a1814e810 msgr2=0x7f9a181f0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1d600640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9a1814e810 0x7f9a181f0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1d600640 1 -- 172.21.15.53:0/2293961240 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1814aa30 con 0x7f9a180af790 2024-01-16T15:25:57.733 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1d600640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9a180af790 0x7f9a1813d450 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f9a10001f40 tx=0x7f9a100014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:57.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10006d50 con 0x7f9a180af790 2024-01-16T15:25:57.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a10007020 con 0x7f9a180af790 2024-01-16T15:25:57.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a10007390 con 0x7f9a180af790 2024-01-16T15:25:57.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1f600640 1 -- 172.21.15.53:0/2293961240 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a1814ad10 con 0x7f9a180af790 2024-01-16T15:25:57.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.734+0000 7f9a1f600640 1 -- 172.21.15.53:0/2293961240 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a181f1490 con 0x7f9a180af790 2024-01-16T15:25:57.734 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.736+0000 7f9a1f600640 1 -- 172.21.15.53:0/2293961240 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a180ccb50 con 0x7f9a180af790 2024-01-16T15:25:57.735 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.736+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f9a1003f030 con 0x7f9a180af790 2024-01-16T15:25:57.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.737+0000 7f9a14c00640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99f006c6a0 0x7f99f006eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:25:57.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.738+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9a10003e40 con 0x7f9a180af790 2024-01-16T15:25:57.736 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.738+0000 7f9a1cc00640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99f006c6a0 0x7f99f006eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:25:57.738 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.739+0000 7f9a1cc00640 1 --2- 172.21.15.53:0/2293961240 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f99f006c6a0 0x7f99f006eb60 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f9a08001010 tx=0x7f9a08009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:25:57.740 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.742+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9a10076fb0 con 0x7f9a180af790 2024-01-16T15:25:57.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.974+0000 7f9a1f600640 1 -- 172.21.15.53:0/2293961240 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9a1814d7c0 con 0x7f99f006c6a0 2024-01-16T15:25:57.986 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:25:57.988+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:25:57.986 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:25:57.988+0000 7f9a14c00640 1 -- 172.21.15.53:0/2293961240 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f9a1814d7c0 con 0x7f99f006c6a0 2024-01-16T15:25:57.987 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:25:57.987 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:25:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:25:58.686 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:25:58.695 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:00.011 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:00.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.253+0000 7f3cf0c00640 1 Processor -- start 2024-01-16T15:26:00.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.254+0000 7f3cf0c00640 1 -- start start 2024-01-16T15:26:00.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.255+0000 7f3cf0c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3cec143ae0 0x7f3cec0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.255+0000 7f3cf0c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cec0cfc30 con 0x7f3cec0cf7d0 2024-01-16T15:26:00.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.255+0000 7f3cf0c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cec0b1ee0 con 0x7f3cec15e3b0 2024-01-16T15:26:00.254 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.255+0000 7f3cf0c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cec0b2780 con 0x7f3cec143ae0 2024-01-16T15:26:00.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.256+0000 7f3ce3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f3cec0cf7d0 0x7f3cec15dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:41380/0 (socket says 172.21.15.53:41380) 2024-01-16T15:26:00.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.256+0000 7f3ce3e00640 1 -- 172.21.15.53:0/1277878099 learned_addr learned my addr 172.21.15.53:0/1277878099 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:00.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.256+0000 7f3ce9e00640 1 --2- 172.21.15.53:0/1277878099 >> v2:172.21.15.146:3300/0 conn(0x7f3cec143ae0 0x7f3cec0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.256+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372968087 0 0) 0x7f3cec0cfc30 con 0x7f3cec0cf7d0 2024-01-16T15:26:00.255 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ccc003520 con 0x7f3cec0cf7d0 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171243852 0 0) 0x7f3ccc003520 con 0x7f3cec0cf7d0 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 >> v2:172.21.15.146:3300/0 conn(0x7f3cec143ae0 msgr2=0x7f3cec0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 --2- 172.21.15.53:0/1277878099 >> v2:172.21.15.146:3300/0 conn(0x7f3cec143ae0 0x7f3cec0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 >> v1:172.21.15.146:6789/0 conn(0x7f3cec15e3b0 legacy=0x7f3cec0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec15e990 con 0x7f3cec0cf7d0 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f3cd4001670 con 0x7f3cec0cf7d0 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 >> v1:172.21.15.53:6789/0 conn(0x7f3cec0cf7d0 legacy=0x7f3cec15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc003970 0x7f3ccc003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc004370 0x7f3ccc004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc005040 0x7f3ccc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0b2780 con 0x7f3ccc005040 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00e1b0 con 0x7f3ccc004370 2024-01-16T15:26:00.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc00e370 con 0x7f3ccc003970 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.257+0000 7f3cea800640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc005040 0x7f3ccc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3ce9e00640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc003970 0x7f3ccc003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3ce3e00640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc004370 0x7f3ccc004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3cea800640 1 -- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc003970 msgr2=0x7f3ccc003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3cea800640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc003970 0x7f3ccc003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3cea800640 1 -- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc004370 msgr2=0x7f3ccc004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3cea800640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc004370 0x7f3ccc004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3cea800640 1 -- 172.21.15.53:0/1277878099 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc003520 con 0x7f3ccc005040 2024-01-16T15:26:00.258 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.258+0000 7f3ce3e00640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ccc004370 0x7f3ccc004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:00.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.259+0000 7f3cea800640 1 -- 172.21.15.53:0/1277878099 reap_dead start 2024-01-16T15:26:00.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.259+0000 7f3cea800640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc005040 0x7f3ccc00db90 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4001f60 tx=0x7f3ccc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62a90fe888044027 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:00.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce402d6f0 con 0x7f3ccc005040 2024-01-16T15:26:00.259 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce402d850 con 0x7f3ccc005040 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3ce9400640 1 -- 172.21.15.53:0/1277878099 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ce402dc00 con 0x7f3ccc005040 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 -- 172.21.15.53:0/1277878099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc005040 msgr2=0x7f3ccc00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc005040 0x7f3ccc00db90 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4001f60 tx=0x7f3ccc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 -- 172.21.15.53:0/1277878099 shutdown_connections 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 --2- 172.21.15.53:0/1277878099 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ccc005040 0x7f3ccc00db90 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 -- 172.21.15.53:0/1277878099 >> 172.21.15.53:0/1277878099 conn(0x7f3cec06a800 msgr2=0x7f3cec06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 -- 172.21.15.53:0/1277878099 shutdown_connections 2024-01-16T15:26:00.260 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.260+0000 7f3cf0c00640 1 -- 172.21.15.53:0/1277878099 wait complete. 2024-01-16T15:26:00.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.262+0000 7f3cf0c00640 1 Processor -- start 2024-01-16T15:26:00.261 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.262+0000 7f3cf0c00640 1 -- start start 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3cf0c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005040 0x7f3cec070d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3cf0c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0c4d90 0x7f3cec13fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3cf0c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0ce1a0 0x7f3cec140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3cf0c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0c3fd0 con 0x7f3cec0c4d90 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3cf0c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0c42b0 con 0x7f3cec0ce1a0 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3cf0c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec0c4590 con 0x7f3ccc005040 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.263+0000 7f3ce3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0c4d90 0x7f3cec13fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0c4d90 0x7f3cec13fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:57240/0 (socket says 172.21.15.53:57240) 2024-01-16T15:26:00.262 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 -- 172.21.15.53:0/830628930 learned_addr learned my addr 172.21.15.53:0/830628930 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:00.263 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce9e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005040 0x7f3cec070d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3cea800640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0ce1a0 0x7f3cec140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 -- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005040 msgr2=0x7f3cec070d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005040 0x7f3cec070d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 -- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0ce1a0 msgr2=0x7f3cec140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0ce1a0 0x7f3cec140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 -- 172.21.15.53:0/830628930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec0c4870 con 0x7f3cec0c4d90 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3cea800640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3cec0ce1a0 0x7f3cec140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce9e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ccc005040 0x7f3cec070d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.264+0000 7f3ce3e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3cec0c4d90 0x7f3cec13fa80 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f3cd400d7c0 tx=0x7f3cd400dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.265+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40074e0 con 0x7f3cec0c4d90 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.265+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cd4007640 con 0x7f3cec0c4d90 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.265+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4004a00 con 0x7f3cec0c4d90 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.265+0000 7f3cf0c00640 1 -- 172.21.15.53:0/830628930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec0ce630 con 0x7f3cec0c4d90 2024-01-16T15:26:00.264 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.265+0000 7f3cf0c00640 1 -- 172.21.15.53:0/830628930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec140aa0 con 0x7f3cec0c4d90 2024-01-16T15:26:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.266+0000 7f3cf0c00640 1 -- 172.21.15.53:0/830628930 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cec140d50 con 0x7f3cec0c4d90 2024-01-16T15:26:00.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.267+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f3cd4004b60 con 0x7f3cec0c4d90 2024-01-16T15:26:00.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.268+0000 7f3ce2000640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406c670 0x7f3cc406eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:00.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.268+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3cd4088590 con 0x7f3cec0c4d90 2024-01-16T15:26:00.267 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.269+0000 7f3ce9e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406c670 0x7f3cc406eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:00.268 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.270+0000 7f3ce9e00640 1 --2- 172.21.15.53:0/830628930 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3cc406c670 0x7f3cc406eb30 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f3cec140f40 tx=0x7f3cdc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:00.271 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.273+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3cd4054aa0 con 0x7f3cec0c4d90 2024-01-16T15:26:00.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.501+0000 7f3cf0c00640 1 -- 172.21.15.53:0/830628930 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3cec0b2410 con 0x7f3cc406c670 2024-01-16T15:26:00.513 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:00.515+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:00.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:00.515+0000 7f3ce2000640 1 -- 172.21.15.53:0/830628930 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f3cdc012950 con 0x7f3cc406c670 2024-01-16T15:26:00.514 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:00.514 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:02.526 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:02.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.766+0000 7f6664800640 1 Processor -- start 2024-01-16T15:26:02.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.767+0000 7f6664800640 1 -- start start 2024-01-16T15:26:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.768+0000 7f6664800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f665c09a170 0x7f665c0b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.768+0000 7f6664800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f665c06c6b0 0x7f665c0cf640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6664800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f665c06cea0 con 0x7f665c0cd240 2024-01-16T15:26:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6664800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c06cbc0 con 0x7f665c06c6b0 2024-01-16T15:26:02.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6664800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f665c06f350 con 0x7f665c09a170 2024-01-16T15:26:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6662800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f665c06c6b0 0x7f665c0cf640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6661e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f665c0cd240 0x7f665c099a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33012/0 (socket says 172.21.15.53:33012) 2024-01-16T15:26:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6661e00640 1 -- 172.21.15.53:0/2285495509 learned_addr learned my addr 172.21.15.53:0/2285495509 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.769+0000 7f6661400640 1 --2- 172.21.15.53:0/2285495509 >> v2:172.21.15.146:3300/0 conn(0x7f665c09a170 0x7f665c0b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:02.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.770+0000 7f6662800640 1 -- 172.21.15.53:0/2285495509 >> v2:172.21.15.146:3300/0 conn(0x7f665c09a170 msgr2=0x7f665c0b3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:02.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.770+0000 7f6662800640 1 --2- 172.21.15.53:0/2285495509 >> v2:172.21.15.146:3300/0 conn(0x7f665c09a170 0x7f665c0b3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:02.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.770+0000 7f6662800640 1 -- 172.21.15.53:0/2285495509 >> v1:172.21.15.53:6789/0 conn(0x7f665c0cd240 legacy=0x7f665c099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:02.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.770+0000 7f6660a00640 1 -- 172.21.15.53:0/2285495509 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722690020 0 0) 0x7f665c06cea0 con 0x7f665c0cd240 2024-01-16T15:26:02.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.770+0000 7f6662800640 1 -- 172.21.15.53:0/2285495509 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f665c06f630 con 0x7f665c06c6b0 2024-01-16T15:26:02.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.771+0000 7f6662800640 1 --2- 172.21.15.53:0/2285495509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c06c6b0 0x7f665c0cf640 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f665c071800 tx=0x7f665402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9934089ee97643a7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:02.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.771+0000 7f6660a00640 1 -- 172.21.15.53:0/2285495509 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6654031830 con 0x7f665c06c6b0 2024-01-16T15:26:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.771+0000 7f6660a00640 1 -- 172.21.15.53:0/2285495509 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6654031990 con 0x7f665c06c6b0 2024-01-16T15:26:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.771+0000 7f6660a00640 1 -- 172.21.15.53:0/2285495509 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6654031d20 con 0x7f665c06c6b0 2024-01-16T15:26:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 -- 172.21.15.53:0/2285495509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c06c6b0 msgr2=0x7f665c0cf640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 --2- 172.21.15.53:0/2285495509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c06c6b0 0x7f665c0cf640 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7f665c071800 tx=0x7f665402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:26:02.770 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 -- 172.21.15.53:0/2285495509 shutdown_connections 2024-01-16T15:26:02.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 --2- 172.21.15.53:0/2285495509 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c06c6b0 0x7f665c0cf640 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:02.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 --2- 172.21.15.53:0/2285495509 >> v2:172.21.15.146:3300/0 conn(0x7f665c09a170 0x7f665c0b3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:02.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 -- 172.21.15.53:0/2285495509 >> 172.21.15.53:0/2285495509 conn(0x7f665c0b2d00 msgr2=0x7f665c0b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:02.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 -- 172.21.15.53:0/2285495509 shutdown_connections 2024-01-16T15:26:02.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.772+0000 7f6664800640 1 -- 172.21.15.53:0/2285495509 wait complete. 2024-01-16T15:26:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.774+0000 7f6664800640 1 Processor -- start 2024-01-16T15:26:02.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.774+0000 7f6664800640 1 -- start start 2024-01-16T15:26:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6664800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c06c6b0 0x7f665c0b4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6664800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c09a170 0x7f665c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:02.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6664800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c0cd240 0x7f665c1f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6664800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c1f3d90 con 0x7f665c0cd240 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6664800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c0cb4e0 con 0x7f665c09a170 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6664800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f665c1f4000 con 0x7f665c06c6b0 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6662800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c0cd240 0x7f665c1f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6662800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c0cd240 0x7f665c1f2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60584/0 (socket says 172.21.15.53:60584) 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.775+0000 7f6662800640 1 -- 172.21.15.53:0/2451541398 learned_addr learned my addr 172.21.15.53:0/2451541398 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6661400640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c09a170 0x7f665c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6661e00640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c06c6b0 0x7f665c0b4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6662800640 1 -- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c06c6b0 msgr2=0x7f665c0b4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6662800640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c06c6b0 0x7f665c0b4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6662800640 1 -- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c09a170 msgr2=0x7f665c0b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6662800640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c09a170 0x7f665c0b4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:02.775 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6662800640 1 -- 172.21.15.53:0/2451541398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6654009400 con 0x7f665c0cd240 2024-01-16T15:26:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6661e00640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f665c06c6b0 0x7f665c0b4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6661400640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f665c09a170 0x7f665c0b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6662800640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f665c0cd240 0x7f665c1f2bd0 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f6654009890 tx=0x7f66540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:02.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66540317d0 con 0x7f665c0cd240 2024-01-16T15:26:02.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665403a9e0 con 0x7f665c0cd240 2024-01-16T15:26:02.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665403ad90 con 0x7f665c0cd240 2024-01-16T15:26:02.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.776+0000 7f6664800640 1 -- 172.21.15.53:0/2451541398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c0b5ed0 con 0x7f665c0cd240 2024-01-16T15:26:02.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.777+0000 7f6664800640 1 -- 172.21.15.53:0/2451541398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c0b5910 con 0x7f665c0cd240 2024-01-16T15:26:02.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.778+0000 7f6664800640 1 -- 172.21.15.53:0/2451541398 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c06d0d0 con 0x7f665c0cd240 2024-01-16T15:26:02.777 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.779+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f665404b3b0 con 0x7f665c0cd240 2024-01-16T15:26:02.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.780+0000 7f665a000640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f663406c6a0 0x7f663406eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:02.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.780+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f66540aaa00 con 0x7f665c0cd240 2024-01-16T15:26:02.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.781+0000 7f6661e00640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f663406c6a0 0x7f663406eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:02.780 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.782+0000 7f6661e00640 1 --2- 172.21.15.53:0/2451541398 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f663406c6a0 0x7f663406eb60 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f664c0012e0 tx=0x7f664c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:02.783 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:02.785+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6654077030 con 0x7f665c0cd240 2024-01-16T15:26:03.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:03.016+0000 7f6664800640 1 -- 172.21.15.53:0/2451541398 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f665c0cd620 con 0x7f663406c6a0 2024-01-16T15:26:03.029 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:03.031+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:03.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:03.032+0000 7f665a000640 1 -- 172.21.15.53:0/2451541398 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f664c00d950 con 0x7f663406c6a0 2024-01-16T15:26:03.030 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:03.030 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:03.045 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:05.046 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.280+0000 7f87f9200640 1 Processor -- start 2024-01-16T15:26:05.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.280+0000 7f87f9200640 1 -- start start 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f9200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f87f414b2c0 0x7f87f40cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40cf7d0 0x7f87f4070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f40cfbf0 con 0x7f87f4070ea0 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f9200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f414b6a0 con 0x7f87f414b2c0 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4071690 con 0x7f87f40cf7d0 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f87f414b2c0 0x7f87f40cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f1e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f87f414b2c0 0x7f87f40cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60600/0 (socket says 172.21.15.53:60600) 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.282+0000 7f87f1e00640 1 -- 172.21.15.53:0/3096157503 learned_addr learned my addr 172.21.15.53:0/3096157503 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.283+0000 7f87f1400640 1 --2- 172.21.15.53:0/3096157503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40cf7d0 0x7f87f4070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.283+0000 7f87f1e00640 1 -- 172.21.15.53:0/3096157503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40cf7d0 msgr2=0x7f87f4070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.283+0000 7f87f1e00640 1 --2- 172.21.15.53:0/3096157503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40cf7d0 0x7f87f4070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.283+0000 7f87f1e00640 1 -- 172.21.15.53:0/3096157503 >> v1:172.21.15.146:6789/0 conn(0x7f87f4070ea0 legacy=0x7f87f40b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.283+0000 7f87f1e00640 1 -- 172.21.15.53:0/3096157503 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f4071970 con 0x7f87f414b2c0 2024-01-16T15:26:05.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.284+0000 7f87f1e00640 1 --2- 172.21.15.53:0/3096157503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f414b2c0 0x7f87f40cf290 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009e50 tx=0x7f87e402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f2dac160283d1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.284+0000 7f87f0a00640 1 -- 172.21.15.53:0/3096157503 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e402ca70 con 0x7f87f414b2c0 2024-01-16T15:26:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.284+0000 7f87f0a00640 1 -- 172.21.15.53:0/3096157503 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e402cbd0 con 0x7f87f414b2c0 2024-01-16T15:26:05.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.284+0000 7f87f0a00640 1 -- 172.21.15.53:0/3096157503 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e40343f0 con 0x7f87f414b2c0 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 -- 172.21.15.53:0/3096157503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f414b2c0 msgr2=0x7f87f40cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 --2- 172.21.15.53:0/3096157503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f414b2c0 0x7f87f40cf290 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f87e4009e50 tx=0x7f87e402bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 -- 172.21.15.53:0/3096157503 shutdown_connections 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 --2- 172.21.15.53:0/3096157503 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f40cf7d0 0x7f87f4070960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 --2- 172.21.15.53:0/3096157503 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f414b2c0 0x7f87f40cf290 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 -- 172.21.15.53:0/3096157503 >> 172.21.15.53:0/3096157503 conn(0x7f87f406a800 msgr2=0x7f87f406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 -- 172.21.15.53:0/3096157503 shutdown_connections 2024-01-16T15:26:05.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.285+0000 7f87f9200640 1 -- 172.21.15.53:0/3096157503 wait complete. 2024-01-16T15:26:05.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.287+0000 7f87f9200640 1 Processor -- start 2024-01-16T15:26:05.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.287+0000 7f87f9200640 1 -- start start 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.288+0000 7f87f9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f4070ea0 0x7f87f4164990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.288+0000 7f87f9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f40cf7d0 0x7f87f41463f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.288+0000 7f87f9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f414b2c0 0x7f87f4143740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.288+0000 7f87f9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4147f50 con 0x7f87f40cf7d0 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.288+0000 7f87f9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4146eb0 con 0x7f87f414b2c0 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.288+0000 7f87f9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f4147190 con 0x7f87f4070ea0 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f40cf7d0 0x7f87f41463f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f40cf7d0 0x7f87f41463f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60608/0 (socket says 172.21.15.53:60608) 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 -- 172.21.15.53:0/664057805 learned_addr learned my addr 172.21.15.53:0/664057805 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1e00640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f4070ea0 0x7f87f4164990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 -- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f4070ea0 msgr2=0x7f87f4164990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:05.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f87f4070ea0 0x7f87f4164990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 -- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f414b2c0 msgr2=0x7f87f4143740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f87f414b2c0 0x7f87f4143740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 -- 172.21.15.53:0/664057805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e4009b30 con 0x7f87f40cf7d0 2024-01-16T15:26:05.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.289+0000 7f87f1400640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f87f40cf7d0 0x7f87f41463f0 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00c7f0 tx=0x7f87dc00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.290+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc004220 con 0x7f87f40cf7d0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.290+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc0043a0 con 0x7f87f40cf7d0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.290+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc007860 con 0x7f87f40cf7d0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.290+0000 7f87f9200640 1 -- 172.21.15.53:0/664057805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f4147470 con 0x7f87f40cf7d0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.290+0000 7f87f9200640 1 -- 172.21.15.53:0/664057805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f4147a30 con 0x7f87f40cf7d0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.291+0000 7f87f9200640 1 -- 172.21.15.53:0/664057805 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f4147c20 con 0x7f87f40cf7d0 2024-01-16T15:26:05.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.292+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f87dc007bd0 con 0x7f87f40cf7d0 2024-01-16T15:26:05.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.293+0000 7f87ea000640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87cc06c670 0x7f87cc06eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.293+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f87dc088ba0 con 0x7f87f40cf7d0 2024-01-16T15:26:05.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.294+0000 7f87f1e00640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87cc06c670 0x7f87cc06eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:05.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.295+0000 7f87f1e00640 1 --2- 172.21.15.53:0/664057805 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f87cc06c670 0x7f87cc06eb30 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f87e4035000 tx=0x7f87e4030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:05.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.298+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f87dc0550b0 con 0x7f87f40cf7d0 2024-01-16T15:26:05.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.526+0000 7f87f9200640 1 -- 172.21.15.53:0/664057805 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f87f4147300 con 0x7f87cc06c670 2024-01-16T15:26:05.537 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:05.539+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:05.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:05.540+0000 7f87ea000640 1 -- 172.21.15.53:0/664057805 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f87f4147300 con 0x7f87cc06c670 2024-01-16T15:26:05.538 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:05.538 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:07.554 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:07.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.794+0000 7f00ada00640 1 Processor -- start 2024-01-16T15:26:07.792 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.795+0000 7f00ada00640 1 -- start start 2024-01-16T15:26:07.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.796+0000 7f00ada00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00a80cf9a0 0x7f00a8166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:07.794 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.796+0000 7f00ada00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a815c5f0 con 0x7f00a813df80 2024-01-16T15:26:07.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.796+0000 7f00ada00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a806c470 con 0x7f00a813e360 2024-01-16T15:26:07.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.796+0000 7f00ada00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a806c750 con 0x7f00a80cf9a0 2024-01-16T15:26:07.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.797+0000 7f00a7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00a80cf9a0 0x7f00a8166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:07.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.797+0000 7f00a7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f00a80cf9a0 0x7f00a8166030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60620/0 (socket says 172.21.15.53:60620) 2024-01-16T15:26:07.795 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.797+0000 7f00a7400640 1 -- 172.21.15.53:0/1036061966 learned_addr learned my addr 172.21.15.53:0/1036061966 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:07.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.798+0000 7f00a7400640 1 -- 172.21.15.53:0/1036061966 >> v1:172.21.15.146:6789/0 conn(0x7f00a813e360 legacy=0x7f00a80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:07.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.798+0000 7f00a7400640 1 -- 172.21.15.53:0/1036061966 >> v1:172.21.15.53:6789/0 conn(0x7f00a813df80 legacy=0x7f00a815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:07.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.798+0000 7f00a7400640 1 -- 172.21.15.53:0/1036061966 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a806ca30 con 0x7f00a80cf9a0 2024-01-16T15:26:07.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.798+0000 7f00a6000640 1 -- 172.21.15.53:0/1036061966 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416757592 0 0) 0x7f00a815c5f0 con 0x7f00a813df80 2024-01-16T15:26:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.799+0000 7f00a7400640 1 --2- 172.21.15.53:0/1036061966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a80cf9a0 0x7f00a8166030 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009e50 tx=0x7f00a002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=815639ea0291cbda server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.799+0000 7f00a6000640 1 -- 172.21.15.53:0/1036061966 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a002cbb0 con 0x7f00a80cf9a0 2024-01-16T15:26:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.799+0000 7f00a6000640 1 -- 172.21.15.53:0/1036061966 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a002cd10 con 0x7f00a80cf9a0 2024-01-16T15:26:07.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.799+0000 7f00a6000640 1 -- 172.21.15.53:0/1036061966 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0034610 con 0x7f00a80cf9a0 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 -- 172.21.15.53:0/1036061966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a80cf9a0 msgr2=0x7f00a8166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 --2- 172.21.15.53:0/1036061966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a80cf9a0 0x7f00a8166030 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7f00a0009e50 tx=0x7f00a002c010 comp rx=0 tx=0).stop 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 -- 172.21.15.53:0/1036061966 shutdown_connections 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 --2- 172.21.15.53:0/1036061966 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a80cf9a0 0x7f00a8166030 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 -- 172.21.15.53:0/1036061966 >> 172.21.15.53:0/1036061966 conn(0x7f00a806a800 msgr2=0x7f00a806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 -- 172.21.15.53:0/1036061966 shutdown_connections 2024-01-16T15:26:07.798 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.800+0000 7f00ada00640 1 -- 172.21.15.53:0/1036061966 wait complete. 2024-01-16T15:26:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.802+0000 7f00ada00640 1 Processor -- start 2024-01-16T15:26:07.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.802+0000 7f00ada00640 1 -- start start 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00ada00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cf9a0 0x7f00a814ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00ada00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a813df80 0x7f00a81321f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00ada00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a813e360 0x7f00a814ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00ada00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a814f6c0 con 0x7f00a813e360 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00ada00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a814f9a0 con 0x7f00a813df80 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00ada00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a814fc80 con 0x7f00a80cf9a0 2024-01-16T15:26:07.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a813e360 0x7f00a814ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00a7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a813e360 0x7f00a814ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60634/0 (socket says 172.21.15.53:60634) 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.803+0000 7f00a7400640 1 -- 172.21.15.53:0/2934823310 learned_addr learned my addr 172.21.15.53:0/2934823310 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a6a00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cf9a0 0x7f00a814ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a7400640 1 -- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cf9a0 msgr2=0x7f00a814ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f009fe00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a813df80 0x7f00a81321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a7400640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cf9a0 0x7f00a814ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a7400640 1 -- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a813df80 msgr2=0x7f00a81321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a7400640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a813df80 0x7f00a81321f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a7400640 1 -- 172.21.15.53:0/2934823310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0009b30 con 0x7f00a813e360 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a6a00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f00a80cf9a0 0x7f00a814ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f009fe00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f00a813df80 0x7f00a81321f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f00a7400640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f00a813e360 0x7f00a814ee00 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f00a0036040 tx=0x7f00a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0003990 con 0x7f00a813e360 2024-01-16T15:26:07.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.804+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a0003af0 con 0x7f00a813e360 2024-01-16T15:26:07.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.805+0000 7f00ada00640 1 -- 172.21.15.53:0/2934823310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a814ff60 con 0x7f00a813e360 2024-01-16T15:26:07.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.805+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0003c80 con 0x7f00a813e360 2024-01-16T15:26:07.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.805+0000 7f00ada00640 1 -- 172.21.15.53:0/2934823310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a814f340 con 0x7f00a813e360 2024-01-16T15:26:07.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.806+0000 7f00ada00640 1 -- 172.21.15.53:0/2934823310 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a80675d0 con 0x7f00a813e360 2024-01-16T15:26:07.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.807+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7f00a0048460 con 0x7f00a813e360 2024-01-16T15:26:07.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.808+0000 7f009ea00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f008006c670 0x7f008006eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:07.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.808+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00a00ab330 con 0x7f00a813e360 2024-01-16T15:26:07.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.808+0000 7f00a6a00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f008006c670 0x7f008006eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:07.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.810+0000 7f00a6a00640 1 --2- 172.21.15.53:0/2934823310 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f008006c670 0x7f008006eb30 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f0098000f60 tx=0x7f0098009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:07.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:07.812+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f00a0077840 con 0x7f00a813e360 2024-01-16T15:26:08.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:08.038+0000 7f00ada00640 1 -- 172.21.15.53:0/2934823310 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00a814fb10 con 0x7f008006c670 2024-01-16T15:26:08.049 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:08.051+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:08.050 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:08.052+0000 7f009ea00640 1 -- 172.21.15.53:0/2934823310 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f00a814fb10 con 0x7f008006c670 2024-01-16T15:26:08.050 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:08.050 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:08.072 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:10.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:10.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.313+0000 7fb1b8a00640 1 Processor -- start 2024-01-16T15:26:10.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.313+0000 7fb1b8a00640 1 -- start start 2024-01-16T15:26:10.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.315+0000 7fb1b8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb1b015e3b0 0x7fb1b0164de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.315+0000 7fb1b8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00b1ee0 con 0x7fb1b00cf9a0 2024-01-16T15:26:10.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.315+0000 7fb1b8a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1b00b2780 con 0x7fb1b0143ae0 2024-01-16T15:26:10.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.315+0000 7fb1b8a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1b015e990 con 0x7fb1b015e3b0 2024-01-16T15:26:10.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.315+0000 7fb1b5600640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb1b00cf9a0 0x7fb1b015dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:33016/0 (socket says 172.21.15.53:33016) 2024-01-16T15:26:10.313 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.315+0000 7fb1b5600640 1 -- 172.21.15.53:0/2399561491 learned_addr learned my addr 172.21.15.53:0/2399561491 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731835136 0 0) 0x7fb1b00b1ee0 con 0x7fb1b00cf9a0 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190003520 con 0x7fb1b00cf9a0 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b6a00640 1 --2- 172.21.15.53:0/2399561491 >> v2:172.21.15.146:3300/0 conn(0x7fb1b015e3b0 0x7fb1b0164de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3187386876 0 0) 0x7fb190003520 con 0x7fb1b00cf9a0 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 >> v2:172.21.15.146:3300/0 conn(0x7fb1b015e3b0 msgr2=0x7fb1b0164de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 --2- 172.21.15.53:0/2399561491 >> v2:172.21.15.146:3300/0 conn(0x7fb1b015e3b0 0x7fb1b0164de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 >> v1:172.21.15.146:6789/0 conn(0x7fb1b0143ae0 legacy=0x7fb1b00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b00b2130 con 0x7fb1b00cf9a0 2024-01-16T15:26:10.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb198001670 con 0x7fb1b00cf9a0 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 >> v1:172.21.15.53:6789/0 conn(0x7fb1b00cf9a0 legacy=0x7fb1b015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.316+0000 7fb1b4c00640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb190003970 0x7fb190003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b4c00640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1900047c0 0x7fb190004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b4c00640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb190005180 0x7fb19000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b015e990 con 0x7fb190003970 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19000c0d0 con 0x7fb190005180 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19000c290 con 0x7fb1900047c0 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb190003970 0x7fb190003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6000640 1 -- 172.21.15.53:0/2399561491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1900047c0 msgr2=0x7fb190004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb1900047c0 0x7fb190004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6000640 1 -- 172.21.15.53:0/2399561491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb190005180 msgr2=0x7fb19000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb190005180 0x7fb19000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6000640 1 -- 172.21.15.53:0/2399561491 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb190003520 con 0x7fb190003970 2024-01-16T15:26:10.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.317+0000 7fb1b6a00640 1 -- 172.21.15.53:0/2399561491 reap_dead start 2024-01-16T15:26:10.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.318+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb190003970 0x7fb190003d90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0001750 tx=0x7fb1900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2de651fe20895fcd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a00052d0 con 0x7fb190003970 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a002e050 con 0x7fb190003970 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b4c00640 1 -- 172.21.15.53:0/2399561491 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a0003830 con 0x7fb190003970 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2399561491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb190003970 msgr2=0x7fb190003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b8a00640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb190003970 0x7fb190003d90 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0001750 tx=0x7fb1900057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2399561491 shutdown_connections 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b8a00640 1 --2- 172.21.15.53:0/2399561491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb190003970 0x7fb190003d90 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.319+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2399561491 >> 172.21.15.53:0/2399561491 conn(0x7fb1b006a800 msgr2=0x7fb1b006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.320+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2399561491 shutdown_connections 2024-01-16T15:26:10.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.320+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2399561491 wait complete. 2024-01-16T15:26:10.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.321+0000 7fb1b8a00640 1 Processor -- start 2024-01-16T15:26:10.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.322+0000 7fb1b8a00640 1 -- start start 2024-01-16T15:26:10.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.322+0000 7fb1b8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb190003970 0x7fb1b00c9720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.322+0000 7fb1b8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b0159900 0x7fb1b01f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.322+0000 7fb1b8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b01f37b0 0x7fb1b00cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.322+0000 7fb1b8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b0135600 con 0x7fb1b0159900 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b01358e0 con 0x7fb1b01f37b0 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b0135bc0 con 0x7fb190003970 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b0159900 0x7fb1b01f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b0159900 0x7fb1b01f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:60642/0 (socket says 172.21.15.53:60642) 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 -- 172.21.15.53:0/2147950492 learned_addr learned my addr 172.21.15.53:0/2147950492 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb190003970 0x7fb1b00c9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 -- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb190003970 msgr2=0x7fb1b00c9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb190003970 0x7fb1b00c9720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b6a00640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b01f37b0 0x7fb1b00cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 -- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b01f37b0 msgr2=0x7fb1b00cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb1b01f37b0 0x7fb1b00cd660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.323+0000 7fb1b5600640 1 -- 172.21.15.53:0/2147950492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b0135ea0 con 0x7fb1b0159900 2024-01-16T15:26:10.322 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.324+0000 7fb1b5600640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb1b0159900 0x7fb1b01f3270 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fb19800c680 tx=0x7fb19800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.324+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1980072d0 con 0x7fb1b0159900 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.324+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb198007450 con 0x7fb1b0159900 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.324+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1980047e0 con 0x7fb1b0159900 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.324+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2147950492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b0136180 con 0x7fb1b0159900 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.324+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2147950492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b01f2c40 con 0x7fb1b0159900 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.325+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2147950492 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b01363c0 con 0x7fb1b0159900 2024-01-16T15:26:10.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.326+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 50) v1 ==== 91324+0+0 (secure 0 0 0) 0x7fb198004960 con 0x7fb1b0159900 2024-01-16T15:26:10.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.328+0000 7fb1ae000640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb18806c6a0 0x7fb18806eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:10.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.328+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb198087fe0 con 0x7fb1b0159900 2024-01-16T15:26:10.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.328+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb18806c6a0 0x7fb18806eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:10.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.329+0000 7fb1b6000640 1 --2- 172.21.15.53:0/2147950492 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb18806c6a0 0x7fb18806eb60 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7fb1a0033000 tx=0x7fb1a0009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:10.330 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.333+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb1b01363c0 con 0x7fb1b0159900 2024-01-16T15:26:10.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.564+0000 7fb1b8a00640 1 -- 172.21.15.53:0/2147950492 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb1b0135a50 con 0x7fb18806c6a0 2024-01-16T15:26:10.575 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:10.577+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:10.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:10.578+0000 7fb1ae000640 1 -- 172.21.15.53:0/2147950492 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb1b0135a50 con 0x7fb18806c6a0 2024-01-16T15:26:10.576 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:10.576 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:10.592 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:12.594 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:12.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.834+0000 7f0bb3e00640 1 Processor -- start 2024-01-16T15:26:12.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.834+0000 7f0bb3e00640 1 -- start start 2024-01-16T15:26:12.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0bb41447f0 0x7f0bb415c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0bb40cccf0 0x7f0bb415c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb3e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bb415c5c0 con 0x7f0bb406c470 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb3e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb406d690 con 0x7f0bb41447f0 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb3e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bb40cebb0 con 0x7f0bb40cccf0 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0bb41447f0 0x7f0bb415c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb2a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0bb41447f0 0x7f0bb415c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48792/0 (socket says 172.21.15.53:48792) 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.836+0000 7f0bb2a00640 1 -- 172.21.15.53:0/730692042 learned_addr learned my addr 172.21.15.53:0/730692042 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:12.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.837+0000 7f0bb2a00640 1 -- 172.21.15.53:0/730692042 >> v2:172.21.15.146:3300/0 conn(0x7f0bb40cccf0 msgr2=0x7f0bb415c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:12.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.837+0000 7f0bb2a00640 1 --2- 172.21.15.53:0/730692042 >> v2:172.21.15.146:3300/0 conn(0x7f0bb40cccf0 0x7f0bb415c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:12.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.837+0000 7f0bb2a00640 1 -- 172.21.15.53:0/730692042 >> v1:172.21.15.146:6789/0 conn(0x7f0bb406c470 legacy=0x7f0bb40cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:12.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.837+0000 7f0bb2a00640 1 -- 172.21.15.53:0/730692042 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bb406ca10 con 0x7f0bb41447f0 2024-01-16T15:26:12.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.838+0000 7f0bb2a00640 1 --2- 172.21.15.53:0/730692042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb415c080 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40aea60 tx=0x7f0ba402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7535a87556b892f0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:12.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.838+0000 7f0bb1600640 1 -- 172.21.15.53:0/730692042 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba40317e0 con 0x7f0bb41447f0 2024-01-16T15:26:12.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.838+0000 7f0bb1600640 1 -- 172.21.15.53:0/730692042 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ba4031940 con 0x7f0bb41447f0 2024-01-16T15:26:12.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.838+0000 7f0bb1600640 1 -- 172.21.15.53:0/730692042 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4031cd0 con 0x7f0bb41447f0 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 -- 172.21.15.53:0/730692042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 msgr2=0x7f0bb415c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 --2- 172.21.15.53:0/730692042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb415c080 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f0bb40aea60 tx=0x7f0ba402b880 comp rx=0 tx=0).stop 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 -- 172.21.15.53:0/730692042 shutdown_connections 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 --2- 172.21.15.53:0/730692042 >> v2:172.21.15.146:3300/0 conn(0x7f0bb40cccf0 0x7f0bb415c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 --2- 172.21.15.53:0/730692042 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb415c080 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 -- 172.21.15.53:0/730692042 >> 172.21.15.53:0/730692042 conn(0x7f0bb406a3e0 msgr2=0x7f0bb406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 -- 172.21.15.53:0/730692042 shutdown_connections 2024-01-16T15:26:12.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.839+0000 7f0bb3e00640 1 -- 172.21.15.53:0/730692042 wait complete. 2024-01-16T15:26:12.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.841+0000 7f0bb3e00640 1 Processor -- start 2024-01-16T15:26:12.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.841+0000 7f0bb3e00640 1 -- start start 2024-01-16T15:26:12.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.842+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb406c470 0x7f0bb40ad2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:12.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.842+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb40cccf0 0x7f0bb40cb950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:12.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.842+0000 7f0bb3e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb40c3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:12.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.842+0000 7f0bb3e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb4070220 con 0x7f0bb41447f0 2024-01-16T15:26:12.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.842+0000 7f0bb3e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41f6370 con 0x7f0bb40cccf0 2024-01-16T15:26:12.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.842+0000 7f0bb3e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41f6650 con 0x7f0bb406c470 2024-01-16T15:26:12.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb40c3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:12.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb40c3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48806/0 (socket says 172.21.15.53:48806) 2024-01-16T15:26:12.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 -- 172.21.15.53:0/2287395281 learned_addr learned my addr 172.21.15.53:0/2287395281 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:12.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb2000640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb40cccf0 0x7f0bb40cb950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:12.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb2a00640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb406c470 0x7f0bb40ad2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 -- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb406c470 msgr2=0x7f0bb40ad2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb406c470 0x7f0bb40ad2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 -- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb40cccf0 msgr2=0x7f0bb40cb950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0bb40cccf0 0x7f0bb40cb950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 -- 172.21.15.53:0/2287395281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba40094d0 con 0x7f0bb41447f0 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb2a00640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0bb406c470 0x7f0bb40ad2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.843+0000 7f0bb3400640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0bb41447f0 0x7f0bb40c3de0 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00b9e0 tx=0x7f0bac00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.844+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00cb20 con 0x7f0bb41447f0 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.844+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bac00cca0 con 0x7f0bb41447f0 2024-01-16T15:26:12.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.844+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac014680 con 0x7f0bb41447f0 2024-01-16T15:26:12.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.844+0000 7f0bb3e00640 1 -- 172.21.15.53:0/2287395281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb41f6930 con 0x7f0bb41447f0 2024-01-16T15:26:12.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.844+0000 7f0bb3e00640 1 -- 172.21.15.53:0/2287395281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb40cbe90 con 0x7f0bb41447f0 2024-01-16T15:26:12.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.845+0000 7f0bb3e00640 1 -- 172.21.15.53:0/2287395281 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb4070390 con 0x7f0bb41447f0 2024-01-16T15:26:12.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.846+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f0bac01d3b0 con 0x7f0bb41447f0 2024-01-16T15:26:12.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.847+0000 7f0baaa00640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0b8c06c5e0 0x7f0b8c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:12.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.847+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0bac088b00 con 0x7f0bb41447f0 2024-01-16T15:26:12.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.848+0000 7f0bb2a00640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0b8c06c5e0 0x7f0b8c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:12.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.849+0000 7f0bb2a00640 1 --2- 172.21.15.53:0/2287395281 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0b8c06c5e0 0x7f0b8c06eaa0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4036000 tx=0x7f0ba403d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:12.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:12.852+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0bac055020 con 0x7f0bb41447f0 2024-01-16T15:26:13.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:13.080+0000 7f0bb3e00640 1 -- 172.21.15.53:0/2287395281 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0bb406d2b0 con 0x7f0b8c06c5e0 2024-01-16T15:26:13.090 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:13.092+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:13.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:13.093+0000 7f0baaa00640 1 -- 172.21.15.53:0/2287395281 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0ba4031d40 con 0x7f0b8c06c5e0 2024-01-16T15:26:13.091 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:13.091 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:13.112 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:15.113 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:15.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.356+0000 7f0d2a000640 1 Processor -- start 2024-01-16T15:26:15.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.356+0000 7f0d2a000640 1 -- start start 2024-01-16T15:26:15.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d2a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0d240af640 0x7f0d240713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d2a000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0d240aef60 0x7f0d24071930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d2a000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d240b2f60 con 0x7f0d2415d5e0 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d2a000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d240cfd80 con 0x7f0d240af640 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d2a000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d240b3de0 con 0x7f0d240aef60 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d23400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0d2415d5e0 0x7f0d240b36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37000/0 (socket says 172.21.15.53:37000) 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d23400640 1 -- 172.21.15.53:0/779361551 learned_addr learned my addr 172.21.15.53:0/779361551 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.358+0000 7f0d22a00640 1 --2- 172.21.15.53:0/779361551 >> v2:172.21.15.53:3300/0 conn(0x7f0d240af640 0x7f0d240713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d23e00640 1 --2- 172.21.15.53:0/779361551 >> v2:172.21.15.146:3300/0 conn(0x7f0d240aef60 0x7f0d24071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d22000640 1 -- 172.21.15.53:0/779361551 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004132134 0 0) 0x7f0d240b2f60 con 0x7f0d2415d5e0 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d22000640 1 -- 172.21.15.53:0/779361551 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d04003520 con 0x7f0d2415d5e0 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d22a00640 1 -- 172.21.15.53:0/779361551 >> v2:172.21.15.146:3300/0 conn(0x7f0d240aef60 msgr2=0x7f0d24071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d22a00640 1 --2- 172.21.15.53:0/779361551 >> v2:172.21.15.146:3300/0 conn(0x7f0d240aef60 0x7f0d24071930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d22a00640 1 -- 172.21.15.53:0/779361551 >> v1:172.21.15.53:6789/0 conn(0x7f0d2415d5e0 legacy=0x7f0d240b36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.359+0000 7f0d22a00640 1 -- 172.21.15.53:0/779361551 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2406fd00 con 0x7f0d240af640 2024-01-16T15:26:15.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.360+0000 7f0d22a00640 1 --2- 172.21.15.53:0/779361551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240af640 0x7f0d240713f0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f0d240cc490 tx=0x7f0d0c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b636f611cbd11f4d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:15.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.360+0000 7f0d22000640 1 -- 172.21.15.53:0/779361551 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c02cc70 con 0x7f0d240af640 2024-01-16T15:26:15.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.360+0000 7f0d22000640 1 -- 172.21.15.53:0/779361551 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c02cdd0 con 0x7f0d240af640 2024-01-16T15:26:15.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.360+0000 7f0d22000640 1 -- 172.21.15.53:0/779361551 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d0c034660 con 0x7f0d240af640 2024-01-16T15:26:15.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 -- 172.21.15.53:0/779361551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240af640 msgr2=0x7f0d240713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 --2- 172.21.15.53:0/779361551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240af640 0x7f0d240713f0 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7f0d240cc490 tx=0x7f0d0c02bec0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 -- 172.21.15.53:0/779361551 shutdown_connections 2024-01-16T15:26:15.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 --2- 172.21.15.53:0/779361551 >> v2:172.21.15.146:3300/0 conn(0x7f0d240aef60 0x7f0d24071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 --2- 172.21.15.53:0/779361551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240af640 0x7f0d240713f0 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 -- 172.21.15.53:0/779361551 >> 172.21.15.53:0/779361551 conn(0x7f0d240b1c60 msgr2=0x7f0d240b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:15.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 -- 172.21.15.53:0/779361551 shutdown_connections 2024-01-16T15:26:15.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.361+0000 7f0d2a000640 1 -- 172.21.15.53:0/779361551 wait complete. 2024-01-16T15:26:15.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.363+0000 7f0d2a000640 1 Processor -- start 2024-01-16T15:26:15.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.363+0000 7f0d2a000640 1 -- start start 2024-01-16T15:26:15.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d2a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240aef60 0x7f0d2415da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d2a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d240af640 0x7f0d2414f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d2a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d2415d5e0 0x7f0d2415bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d2a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240cea80 con 0x7f0d240aef60 2024-01-16T15:26:15.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d2a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d2415f470 con 0x7f0d2415d5e0 2024-01-16T15:26:15.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d2a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d240cef70 con 0x7f0d240af640 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.364+0000 7f0d23400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240aef60 0x7f0d2415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240aef60 0x7f0d2415da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48814/0 (socket says 172.21.15.53:48814) 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 -- 172.21.15.53:0/2826397486 learned_addr learned my addr 172.21.15.53:0/2826397486 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d22a00640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d240af640 0x7f0d2414f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 -- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d240af640 msgr2=0x7f0d2414f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23e00640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d2415d5e0 0x7f0d2415bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0d240af640 0x7f0d2414f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 -- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d2415d5e0 msgr2=0x7f0d2415bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0d2415d5e0 0x7f0d2415bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 -- 172.21.15.53:0/2826397486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c009a70 con 0x7f0d240aef60 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.365+0000 7f0d23400640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0d240aef60 0x7f0d2415da60 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f0d1400c410 tx=0x7f0d1400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.366+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d14005ce0 con 0x7f0d240aef60 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.366+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d14005e40 con 0x7f0d240aef60 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.366+0000 7f0d2a000640 1 -- 172.21.15.53:0/2826397486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d2415f680 con 0x7f0d240aef60 2024-01-16T15:26:15.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.366+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d140072f0 con 0x7f0d240aef60 2024-01-16T15:26:15.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.366+0000 7f0d2a000640 1 -- 172.21.15.53:0/2826397486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d2415fc40 con 0x7f0d240aef60 2024-01-16T15:26:15.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.367+0000 7f0d2a000640 1 -- 172.21.15.53:0/2826397486 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d240b44e0 con 0x7f0d240aef60 2024-01-16T15:26:15.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.368+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f0d14007510 con 0x7f0d240aef60 2024-01-16T15:26:15.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.369+0000 7f0d1b400640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0cfc06c5e0 0x7f0cfc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:15.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.369+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0d140882b0 con 0x7f0d240aef60 2024-01-16T15:26:15.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.369+0000 7f0d22a00640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0cfc06c5e0 0x7f0cfc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:15.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.371+0000 7f0d22a00640 1 --2- 172.21.15.53:0/2826397486 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0cfc06c5e0 0x7f0cfc06eaa0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f0d0c036000 tx=0x7f0d0c030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:15.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.373+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0d14054890 con 0x7f0d240aef60 2024-01-16T15:26:15.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.607+0000 7f0d2a000640 1 -- 172.21.15.53:0/2826397486 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d24161cf0 con 0x7f0cfc06c5e0 2024-01-16T15:26:15.617 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:15.619+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:15.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:15.620+0000 7f0d1b400640 1 -- 172.21.15.53:0/2826397486 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0d0c02ca70 con 0x7f0cfc06c5e0 2024-01-16T15:26:15.618 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:15.618 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:17.634 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:17.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.879+0000 7f6ead200640 1 Processor -- start 2024-01-16T15:26:17.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.880+0000 7f6ead200640 1 -- start start 2024-01-16T15:26:17.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.881+0000 7f6ead200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6ea80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.881+0000 7f6ead200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80cfc30 con 0x7f6ea80cf7d0 2024-01-16T15:26:17.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.881+0000 7f6ead200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80b1ee0 con 0x7f6ea815e3b0 2024-01-16T15:26:17.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.881+0000 7f6ead200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80b2780 con 0x7f6ea8143ae0 2024-01-16T15:26:17.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.882+0000 7f6e9d200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6ea80cf7d0 0x7f6ea815dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37014/0 (socket says 172.21.15.53:37014) 2024-01-16T15:26:17.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.882+0000 7f6e9d200640 1 -- 172.21.15.53:0/3017441332 learned_addr learned my addr 172.21.15.53:0/3017441332 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.882+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6ea80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.882+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025993520 0 0) 0x7f6ea80cfc30 con 0x7f6ea80cf7d0 2024-01-16T15:26:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.882+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e88005730 con 0x7f6ea80cf7d0 2024-01-16T15:26:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.882+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731021457 0 0) 0x7f6e88005730 con 0x7f6ea80cf7d0 2024-01-16T15:26:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 msgr2=0x7f6ea80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6ea80cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 >> v1:172.21.15.146:6789/0 conn(0x7f6ea815e3b0 legacy=0x7f6ea80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea815e990 con 0x7f6ea80cf7d0 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6e90001670 con 0x7f6ea80cf7d0 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 >> v1:172.21.15.53:6789/0 conn(0x7f6ea80cf7d0 legacy=0x7f6ea815dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6e88005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6e880069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e88006f10 0x7f6e88007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea80b2780 con 0x7f6e88005bb0 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8800dc40 con 0x7f6e88006f10 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8800de00 con 0x7f6ea8143ae0 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6e88005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.883+0000 7f6e9d200640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6e880069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6ea6800640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e88006f10 0x7f6e88007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6ea5e00640 1 -- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 msgr2=0x7f6e880069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6e880069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6ea5e00640 1 -- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e88006f10 msgr2=0x7f6e88007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:17.884 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6e88006f10 0x7f6e88007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:17.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6ea5e00640 1 -- 172.21.15.53:0/3017441332 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e88005730 con 0x7f6e88005bb0 2024-01-16T15:26:17.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.884+0000 7f6e9d200640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea8143ae0 0x7f6e880069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.885+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6e88005fd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009f90 tx=0x7f6e88007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b9eb79e13d9486e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.885+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9802cbd0 con 0x7f6e88005bb0 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.885+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9800f040 con 0x7f6e88005bb0 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.885+0000 7f6ea5400640 1 -- 172.21.15.53:0/3017441332 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98034440 con 0x7f6e88005bb0 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ead200640 1 -- 172.21.15.53:0/3017441332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 msgr2=0x7f6e88005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ead200640 1 --2- 172.21.15.53:0/3017441332 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6e88005fd0 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f6e98009f90 tx=0x7f6e88007a00 comp rx=0 tx=0).stop 2024-01-16T15:26:17.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ea6800640 1 -- 172.21.15.53:0/3017441332 reap_dead start 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ead200640 1 -- 172.21.15.53:0/3017441332 shutdown_connections 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ead200640 1 -- 172.21.15.53:0/3017441332 >> 172.21.15.53:0/3017441332 conn(0x7f6ea806a800 msgr2=0x7f6ea806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ead200640 1 -- 172.21.15.53:0/3017441332 shutdown_connections 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.886+0000 7f6ead200640 1 -- 172.21.15.53:0/3017441332 wait complete. 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.888+0000 7f6ead200640 1 Processor -- start 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.888+0000 7f6ead200640 1 -- start start 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ead200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6ea80caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ead200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b12f0 0x7f6ea81f6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ead200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea80b1750 0x7f6ea81f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ead200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea8140cc0 con 0x7f6e88005bb0 2024-01-16T15:26:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ead200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea81f6ea0 con 0x7f6ea80b1750 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ead200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea813fd60 con 0x7f6ea80b12f0 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ea5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6ea80caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ea5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6ea80caea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48834/0 (socket says 172.21.15.53:48834) 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.889+0000 7f6ea5e00640 1 -- 172.21.15.53:0/2822689022 learned_addr learned my addr 172.21.15.53:0/2822689022 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6e9d200640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b12f0 0x7f6ea81f6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea6800640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea80b1750 0x7f6ea81f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea5e00640 1 -- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b12f0 msgr2=0x7f6ea81f6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6ea80b12f0 0x7f6ea81f6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea5e00640 1 -- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea80b1750 msgr2=0x7f6ea81f65a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6ea80b1750 0x7f6ea81f65a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea5e00640 1 -- 172.21.15.53:0/2822689022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea8140040 con 0x7f6e88005bb0 2024-01-16T15:26:17.891 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.890+0000 7f6ea5e00640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6e88005bb0 0x7f6ea80caea0 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x7f6e9802c570 tx=0x7f6e98009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:17.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.891+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98034e50 con 0x7f6e88005bb0 2024-01-16T15:26:17.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.891+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e98040450 con 0x7f6e88005bb0 2024-01-16T15:26:17.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.891+0000 7f6ead200640 1 -- 172.21.15.53:0/2822689022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea8140320 con 0x7f6e88005bb0 2024-01-16T15:26:17.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.891+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e98040800 con 0x7f6e88005bb0 2024-01-16T15:26:17.892 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.891+0000 7f6ead200640 1 -- 172.21.15.53:0/2822689022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea81408e0 con 0x7f6e88005bb0 2024-01-16T15:26:17.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.892+0000 7f6ead200640 1 -- 172.21.15.53:0/2822689022 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea80675d0 con 0x7f6e88005bb0 2024-01-16T15:26:17.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.894+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f6e98040960 con 0x7f6e88005bb0 2024-01-16T15:26:17.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.894+0000 7f6e9ea00640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e8006c5e0 0x7f6e8006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:17.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.895+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6e980ab290 con 0x7f6e88005bb0 2024-01-16T15:26:17.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.895+0000 7f6e9d200640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e8006c5e0 0x7f6e8006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:17.894 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.897+0000 7f6e9d200640 1 --2- 172.21.15.53:0/2822689022 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f6e8006c5e0 0x7f6e8006eaa0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f6ea80ae2d0 tx=0x7f6e90009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:17.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:17.906+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6e98077810 con 0x7f6e88005bb0 2024-01-16T15:26:18.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:18.142+0000 7f6ead200640 1 -- 172.21.15.53:0/2822689022 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6ea80b2410 con 0x7f6e8006c5e0 2024-01-16T15:26:18.152 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:18.154+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:18.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:18.155+0000 7f6e9ea00640 1 -- 172.21.15.53:0/2822689022 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6e9000d050 con 0x7f6e8006c5e0 2024-01-16T15:26:18.153 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:18.153 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:20.170 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.412+0000 7fd250400640 1 Processor -- start 2024-01-16T15:26:20.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.413+0000 7fd250400640 1 -- start start 2024-01-16T15:26:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.414+0000 7fd250400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd24813df80 0x7fd24815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:20.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd250400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd24813e360 0x7fd2480cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd250400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd24806c470 con 0x7fd24815f9c0 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd250400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24806c750 con 0x7fd24813e360 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd250400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd24806ca30 con 0x7fd24813df80 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd24813e360 0x7fd2480cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd24813e360 0x7fd2480cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48840/0 (socket says 172.21.15.53:48840) 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 -- 172.21.15.53:0/3366794853 learned_addr learned my addr 172.21.15.53:0/3366794853 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 -- 172.21.15.53:0/3366794853 >> v2:172.21.15.146:3300/0 conn(0x7fd24813df80 msgr2=0x7fd24815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:20.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 --2- 172.21.15.53:0/3366794853 >> v2:172.21.15.146:3300/0 conn(0x7fd24813df80 0x7fd24815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 -- 172.21.15.53:0/3366794853 >> v1:172.21.15.146:6789/0 conn(0x7fd24815f9c0 legacy=0x7fd2481408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:20.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.415+0000 7fd24d000640 1 -- 172.21.15.53:0/3366794853 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24806cd10 con 0x7fd24813e360 2024-01-16T15:26:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.417+0000 7fd24d000640 1 --2- 172.21.15.53:0/3366794853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd2480cf290 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fd248072770 tx=0x7fd23002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788040be7635f66a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.417+0000 7fd247e00640 1 -- 172.21.15.53:0/3366794853 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230031870 con 0x7fd24813e360 2024-01-16T15:26:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.417+0000 7fd247e00640 1 -- 172.21.15.53:0/3366794853 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2300319d0 con 0x7fd24813e360 2024-01-16T15:26:20.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.417+0000 7fd247e00640 1 -- 172.21.15.53:0/3366794853 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230031d60 con 0x7fd24813e360 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 -- 172.21.15.53:0/3366794853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 msgr2=0x7fd2480cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 --2- 172.21.15.53:0/3366794853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd2480cf290 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7fd248072770 tx=0x7fd23002b910 comp rx=0 tx=0).stop 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 -- 172.21.15.53:0/3366794853 shutdown_connections 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 --2- 172.21.15.53:0/3366794853 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd2480cf290 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 --2- 172.21.15.53:0/3366794853 >> v2:172.21.15.146:3300/0 conn(0x7fd24813df80 0x7fd24815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 -- 172.21.15.53:0/3366794853 >> 172.21.15.53:0/3366794853 conn(0x7fd24806a800 msgr2=0x7fd24806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 -- 172.21.15.53:0/3366794853 shutdown_connections 2024-01-16T15:26:20.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.418+0000 7fd250400640 1 -- 172.21.15.53:0/3366794853 wait complete. 2024-01-16T15:26:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.420+0000 7fd250400640 1 Processor -- start 2024-01-16T15:26:20.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.421+0000 7fd250400640 1 -- start start 2024-01-16T15:26:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd250400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24813df80 0x7fd24813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd250400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd24814ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd250400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24815f9c0 0x7fd2480b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd250400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24814bbd0 con 0x7fd24813e360 2024-01-16T15:26:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd250400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd24814b610 con 0x7fd24815f9c0 2024-01-16T15:26:20.422 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd250400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd24814b8f0 con 0x7fd24813df80 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.422+0000 7fd24d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd24814ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd24814ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48844/0 (socket says 172.21.15.53:48844) 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 -- 172.21.15.53:0/3811379343 learned_addr learned my addr 172.21.15.53:0/3811379343 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24e400640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24815f9c0 0x7fd2480b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24da00640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24813df80 0x7fd24813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 -- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24813df80 msgr2=0x7fd24813d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24813df80 0x7fd24813d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 -- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24815f9c0 msgr2=0x7fd2480b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd24815f9c0 0x7fd2480b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 -- 172.21.15.53:0/3811379343 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2300094d0 con 0x7fd24813e360 2024-01-16T15:26:20.423 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24da00640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd24813df80 0x7fd24813d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:20.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.423+0000 7fd24d000640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd24813e360 0x7fd24814ac50 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7fd230000c00 tx=0x7fd2300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:20.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.424+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23003ad10 con 0x7fd24813e360 2024-01-16T15:26:20.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.424+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd23003ae70 con 0x7fd24813e360 2024-01-16T15:26:20.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.424+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230044610 con 0x7fd24813e360 2024-01-16T15:26:20.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.424+0000 7fd250400640 1 -- 172.21.15.53:0/3811379343 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd24814c590 con 0x7fd24813e360 2024-01-16T15:26:20.424 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.424+0000 7fd250400640 1 -- 172.21.15.53:0/3811379343 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd248151070 con 0x7fd24813e360 2024-01-16T15:26:20.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.425+0000 7fd250400640 1 -- 172.21.15.53:0/3811379343 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd24814cab0 con 0x7fd24813e360 2024-01-16T15:26:20.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.426+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd23004d450 con 0x7fd24813e360 2024-01-16T15:26:20.425 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.428+0000 7fd245600640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd22006c5e0 0x7fd22006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:20.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.428+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd23002d040 con 0x7fd24813e360 2024-01-16T15:26:20.426 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.428+0000 7fd24da00640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd22006c5e0 0x7fd22006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:20.427 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.429+0000 7fd24da00640 1 --2- 172.21.15.53:0/3811379343 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd22006c5e0 0x7fd22006eaa0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fd238007eb0 tx=0x7fd2380073c0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:20.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.432+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd24814cab0 con 0x7fd24813e360 2024-01-16T15:26:20.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.673+0000 7fd250400640 1 -- 172.21.15.53:0/3811379343 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd24814ba60 con 0x7fd22006c5e0 2024-01-16T15:26:20.684 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:20.685+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:20.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:20.686+0000 7fd245600640 1 -- 172.21.15.53:0/3811379343 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd24814ba60 con 0x7fd22006c5e0 2024-01-16T15:26:20.685 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:20.685 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:20.695 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:22.697 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:22.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.941+0000 7fe109800640 1 Processor -- start 2024-01-16T15:26:22.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.941+0000 7fe109800640 1 -- start start 2024-01-16T15:26:22.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe109800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe104143ae0 0x7fe1040cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:22.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe109800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe10415dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:22.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe109800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1040cfc30 con 0x7fe10415e1e0 2024-01-16T15:26:22.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe109800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1040b1ee0 con 0x7fe104143ae0 2024-01-16T15:26:22.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe109800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1040b2780 con 0x7fe1040cf7d0 2024-01-16T15:26:22.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe102a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe104143ae0 0x7fe1040cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe102a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe104143ae0 0x7fe1040cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48572/0 (socket says 172.21.15.53:48572) 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe102a00640 1 -- 172.21.15.53:0/128431772 learned_addr learned my addr 172.21.15.53:0/128431772 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.943+0000 7fe102000640 1 --2- 172.21.15.53:0/128431772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe10415dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.944+0000 7fe102a00640 1 -- 172.21.15.53:0/128431772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 msgr2=0x7fe10415dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.944+0000 7fe102a00640 1 --2- 172.21.15.53:0/128431772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe10415dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.944+0000 7fe102a00640 1 -- 172.21.15.53:0/128431772 >> v1:172.21.15.146:6789/0 conn(0x7fe10415e1e0 legacy=0x7fe10415a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.944+0000 7fe102a00640 1 -- 172.21.15.53:0/128431772 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10415e990 con 0x7fe104143ae0 2024-01-16T15:26:22.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.945+0000 7fe102a00640 1 --2- 172.21.15.53:0/128431772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe104143ae0 0x7fe1040cf290 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009e50 tx=0x7fe0f402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60453de3c2b0f941 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:22.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.945+0000 7fe101600640 1 -- 172.21.15.53:0/128431772 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f402ca70 con 0x7fe104143ae0 2024-01-16T15:26:22.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.945+0000 7fe101600640 1 -- 172.21.15.53:0/128431772 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0f402cbd0 con 0x7fe104143ae0 2024-01-16T15:26:22.943 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.945+0000 7fe101600640 1 -- 172.21.15.53:0/128431772 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0f40343f0 con 0x7fe104143ae0 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 -- 172.21.15.53:0/128431772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe104143ae0 msgr2=0x7fe1040cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 --2- 172.21.15.53:0/128431772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe104143ae0 0x7fe1040cf290 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fe0f4009e50 tx=0x7fe0f402bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 -- 172.21.15.53:0/128431772 shutdown_connections 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 --2- 172.21.15.53:0/128431772 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe10415dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 --2- 172.21.15.53:0/128431772 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe104143ae0 0x7fe1040cf290 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 -- 172.21.15.53:0/128431772 >> 172.21.15.53:0/128431772 conn(0x7fe10406a800 msgr2=0x7fe10406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 -- 172.21.15.53:0/128431772 shutdown_connections 2024-01-16T15:26:22.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.946+0000 7fe109800640 1 -- 172.21.15.53:0/128431772 wait complete. 2024-01-16T15:26:22.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.949+0000 7fe109800640 1 Processor -- start 2024-01-16T15:26:22.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.949+0000 7fe109800640 1 -- start start 2024-01-16T15:26:22.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.950+0000 7fe109800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe104140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:22.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.950+0000 7fe109800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe104143ae0 0x7fe104159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:22.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.950+0000 7fe109800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10415e1e0 0x7fe10413f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:22.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.950+0000 7fe109800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe104166140 con 0x7fe10415e1e0 2024-01-16T15:26:22.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.950+0000 7fe109800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe104166420 con 0x7fe104143ae0 2024-01-16T15:26:22.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.950+0000 7fe109800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe104166700 con 0x7fe1040cf7d0 2024-01-16T15:26:22.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.951+0000 7fe102a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe104140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:22.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.951+0000 7fe103400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10415e1e0 0x7fe10413f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.951+0000 7fe102000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe104143ae0 0x7fe104159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.951+0000 7fe102a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe104140e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:46020/0 (socket says 172.21.15.53:46020) 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.951+0000 7fe102a00640 1 -- 172.21.15.53:0/275005070 learned_addr learned my addr 172.21.15.53:0/275005070 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe103400640 1 -- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 msgr2=0x7fe104140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe103400640 1 --2- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe1040cf7d0 0x7fe104140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe103400640 1 -- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe104143ae0 msgr2=0x7fe104159280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe103400640 1 --2- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe104143ae0 0x7fe104159280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe103400640 1 -- 172.21.15.53:0/275005070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f4009b30 con 0x7fe10415e1e0 2024-01-16T15:26:22.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe103400640 1 --2- 172.21.15.53:0/275005070 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe10415e1e0 0x7fe10413f1b0 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fe0fc00b410 tx=0x7fe0fc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:22.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc011830 con 0x7fe10415e1e0 2024-01-16T15:26:22.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0fc0119b0 con 0x7fe10415e1e0 2024-01-16T15:26:22.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0fc011d60 con 0x7fe10415e1e0 2024-01-16T15:26:22.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.952+0000 7fe109800640 1 -- 172.21.15.53:0/275005070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1041669e0 con 0x7fe10415e1e0 2024-01-16T15:26:22.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.953+0000 7fe109800640 1 -- 172.21.15.53:0/275005070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe104165df0 con 0x7fe10415e1e0 2024-01-16T15:26:22.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.954+0000 7fe109800640 1 -- 172.21.15.53:0/275005070 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1040b2bf0 con 0x7fe10415e1e0 2024-01-16T15:26:22.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.955+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fe0fc01a400 con 0x7fe10415e1e0 2024-01-16T15:26:22.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.956+0000 7fe0faa00640 1 --2- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0dc06c5e0 0x7fe0dc06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:22.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.956+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe0fc088a70 con 0x7fe10415e1e0 2024-01-16T15:26:22.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.956+0000 7fe102a00640 1 --2- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0dc06c5e0 0x7fe0dc06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:22.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.958+0000 7fe102a00640 1 --2- 172.21.15.53:0/275005070 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe0dc06c5e0 0x7fe0dc06eaa0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fe0f402bf10 tx=0x7fe0f4030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:22.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:22.961+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe1040b2bf0 con 0x7fe10415e1e0 2024-01-16T15:26:23.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:23.193+0000 7fe109800640 1 -- 172.21.15.53:0/275005070 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe104166590 con 0x7fe0dc06c5e0 2024-01-16T15:26:23.204 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:23.206+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:23.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:23.207+0000 7fe0faa00640 1 -- 172.21.15.53:0/275005070 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe104166590 con 0x7fe0dc06c5e0 2024-01-16T15:26:23.205 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:23.206 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:23.218 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:25.219 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:25.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.466+0000 7f21a7e00640 1 Processor -- start 2024-01-16T15:26:25.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.466+0000 7f21a7e00640 1 -- start start 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.467+0000 7f21a7e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f21a8160920 0x7f21a80b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.467+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cfea0 0x7f21a8161720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.467+0000 7f21a7e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a80d0a10 con 0x7f21a80cf9f0 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.467+0000 7f21a7e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80d0330 con 0x7f21a8160920 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.467+0000 7f21a7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80d1f50 con 0x7f21a80cfea0 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.468+0000 7f21a6000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f21a80cf9f0 0x7f21a8161010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39642/0 (socket says 172.21.15.53:39642) 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.468+0000 7f21a6000640 1 -- 172.21.15.53:0/508187396 learned_addr learned my addr 172.21.15.53:0/508187396 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:25.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.468+0000 7f21a6a00640 1 --2- 172.21.15.53:0/508187396 >> v2:172.21.15.146:3300/0 conn(0x7f21a8160920 0x7f21a80b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.468+0000 7f21a7400640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cfea0 0x7f21a8161720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.468+0000 7f21a5600640 1 -- 172.21.15.53:0/508187396 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942067707 0 0) 0x7f21a80d0a10 con 0x7f21a80cf9f0 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.469+0000 7f21a5600640 1 -- 172.21.15.53:0/508187396 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188003520 con 0x7f21a80cf9f0 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.469+0000 7f21a6a00640 1 -- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cfea0 msgr2=0x7f21a8161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.469+0000 7f21a6a00640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cfea0 0x7f21a8161720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.469+0000 7f21a6a00640 1 -- 172.21.15.53:0/508187396 >> v1:172.21.15.53:6789/0 conn(0x7f21a80cf9f0 legacy=0x7f21a8161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.469+0000 7f21a6a00640 1 -- 172.21.15.53:0/508187396 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a80cf080 con 0x7f21a8160920 2024-01-16T15:26:25.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.469+0000 7f21a7400640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cfea0 0x7f21a8161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:25.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.470+0000 7f21a6a00640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 0x7f21a80b3f30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f21980097a0 tx=0x7f219802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4726c54f68639fa server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:25.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.471+0000 7f21a5600640 1 -- 172.21.15.53:0/508187396 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198031830 con 0x7f21a8160920 2024-01-16T15:26:25.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.471+0000 7f21a5600640 1 -- 172.21.15.53:0/508187396 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2198031990 con 0x7f21a8160920 2024-01-16T15:26:25.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.471+0000 7f21a5600640 1 -- 172.21.15.53:0/508187396 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2198031d20 con 0x7f21a8160920 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 -- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 msgr2=0x7f21a80b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 0x7f21a80b3f30 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f21980097a0 tx=0x7f219802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 -- 172.21.15.53:0/508187396 shutdown_connections 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cfea0 0x7f21a8161720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 --2- 172.21.15.53:0/508187396 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 0x7f21a80b3f30 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 -- 172.21.15.53:0/508187396 >> 172.21.15.53:0/508187396 conn(0x7f21a80d3090 msgr2=0x7f21a80d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 -- 172.21.15.53:0/508187396 shutdown_connections 2024-01-16T15:26:25.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.472+0000 7f21a7e00640 1 -- 172.21.15.53:0/508187396 wait complete. 2024-01-16T15:26:25.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.474+0000 7f21a7e00640 1 Processor -- start 2024-01-16T15:26:25.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.474+0000 7f21a7e00640 1 -- start start 2024-01-16T15:26:25.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cf9f0 0x7f21a8150c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:25.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80cfea0 0x7f21a815e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:25.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 0x7f21a80b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a80b6670 con 0x7f21a80cfea0 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21a81511b0 con 0x7f21a8160920 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21a815f260 con 0x7f21a80cf9f0 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80cfea0 0x7f21a815e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a6000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80cfea0 0x7f21a815e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48594/0 (socket says 172.21.15.53:48594) 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a6000640 1 -- 172.21.15.53:0/2865665796 learned_addr learned my addr 172.21.15.53:0/2865665796 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a6a00640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cf9f0 0x7f21a8150c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.475+0000 7f21a7400640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 0x7f21a80b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a6000640 1 -- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cf9f0 msgr2=0x7f21a8150c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a6000640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f21a80cf9f0 0x7f21a8150c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a6000640 1 -- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 msgr2=0x7f21a80b4c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a6000640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f21a8160920 0x7f21a80b4c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:25.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a6000640 1 -- 172.21.15.53:0/2865665796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2198009400 con 0x7f21a80cfea0 2024-01-16T15:26:25.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a6000640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f21a80cfea0 0x7f21a815e810 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f219000caa0 tx=0x7f219000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:25.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f219000ac00 con 0x7f21a80cfea0 2024-01-16T15:26:25.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f219000ad60 con 0x7f21a80cfea0 2024-01-16T15:26:25.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2190012620 con 0x7f21a80cfea0 2024-01-16T15:26:25.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.476+0000 7f21a7e00640 1 -- 172.21.15.53:0/2865665796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a80b5d80 con 0x7f21a80cfea0 2024-01-16T15:26:25.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.477+0000 7f21a7e00640 1 -- 172.21.15.53:0/2865665796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a80b6300 con 0x7f21a80cfea0 2024-01-16T15:26:25.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.478+0000 7f21a7e00640 1 -- 172.21.15.53:0/2865665796 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a8067450 con 0x7f21a80cfea0 2024-01-16T15:26:25.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.479+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f2190007070 con 0x7f21a80cfea0 2024-01-16T15:26:25.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.480+0000 7f219ea00640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f218006c5e0 0x7f218006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:25.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.480+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2190088b80 con 0x7f21a80cfea0 2024-01-16T15:26:25.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.480+0000 7f21a6a00640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f218006c5e0 0x7f218006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:25.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.482+0000 7f21a6a00640 1 --2- 172.21.15.53:0/2865665796 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f218006c5e0 0x7f218006eaa0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f2198036000 tx=0x7f219803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:25.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.485+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f21900550a0 con 0x7f21a80cfea0 2024-01-16T15:26:25.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.714+0000 7f21a7e00640 1 -- 172.21.15.53:0/2865665796 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f21a80ceac0 con 0x7f218006c5e0 2024-01-16T15:26:25.728 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:25.731+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:25.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:25.731+0000 7f219ea00640 1 -- 172.21.15.53:0/2865665796 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f219803a420 con 0x7f218006c5e0 2024-01-16T15:26:25.729 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:25.729 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:25.744 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:27.745 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:27.989 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.991+0000 7f642f200640 1 Processor -- start 2024-01-16T15:26:27.990 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.992+0000 7f642f200640 1 -- start start 2024-01-16T15:26:27.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.993+0000 7f642f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f64300d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:27.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.993+0000 7f642f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f64300b4110 0x7f64300cfe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:27.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.993+0000 7f642f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64300b2c70 con 0x7f64300cfa40 2024-01-16T15:26:27.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.993+0000 7f642f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64300d03c0 con 0x7f64300b4110 2024-01-16T15:26:27.991 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.993+0000 7f642f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64300cf6d0 con 0x7f643015d3c0 2024-01-16T15:26:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.993+0000 7f6427e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f64300cfa40 0x7f64300cde90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39654/0 (socket says 172.21.15.53:39654) 2024-01-16T15:26:27.992 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f6427e00640 1 -- 172.21.15.53:0/857012127 learned_addr learned my addr 172.21.15.53:0/857012127 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642de00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f64300d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:27.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642e800640 1 --2- 172.21.15.53:0/857012127 >> v2:172.21.15.146:3300/0 conn(0x7f64300b4110 0x7f64300cfe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533562144 0 0) 0x7f64300b2c70 con 0x7f64300cfa40 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6410003520 con 0x7f64300cfa40 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 58462402 0 0) 0x7f6410003520 con 0x7f64300cfa40 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 msgr2=0x7f64300d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f64300d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 >> v2:172.21.15.146:3300/0 conn(0x7f64300b4110 msgr2=0x7f64300cfe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 --2- 172.21.15.53:0/857012127 >> v2:172.21.15.146:3300/0 conn(0x7f64300b4110 0x7f64300cfe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64300cf3f0 con 0x7f64300cfa40 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.994+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f6418001670 con 0x7f64300cfa40 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 >> v1:172.21.15.53:6789/0 conn(0x7f64300cfa40 legacy=0x7f64300cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f6410003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6410003f40 0x7f64100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6410004d00 0x7f6410005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:27.994 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64300d03c0 con 0x7f6410003f40 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64300cf6d0 con 0x7f6410004d00 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f641000b980 con 0x7f643015d3c0 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f6427e00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6410003f40 0x7f64100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642de00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f6410003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f6427e00640 1 -- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 msgr2=0x7f6410003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f6427e00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f6410003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642e800640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6410004d00 0x7f6410005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f6427e00640 1 -- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6410004d00 msgr2=0x7f6410005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f6427e00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6410004d00 0x7f6410005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f6427e00640 1 -- 172.21.15.53:0/857012127 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6410003520 con 0x7f6410003f40 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642de00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f643015d3c0 0x7f6410003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:27.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.995+0000 7f642e800640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6410004d00 0x7f6410005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:27.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.996+0000 7f6427e00640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6410003f40 0x7f64100047c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f641800c1c0 tx=0x7f64100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87d98c3797df3a0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:27.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.997+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6418005ad0 con 0x7f6410003f40 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.997+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6418005c50 con 0x7f6410003f40 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.997+0000 7f642d400640 1 -- 172.21.15.53:0/857012127 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641802c810 con 0x7f6410003f40 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.997+0000 7f642f200640 1 -- 172.21.15.53:0/857012127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6410003f40 msgr2=0x7f64100047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.997+0000 7f642f200640 1 --2- 172.21.15.53:0/857012127 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6410003f40 0x7f64100047c0 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f641800c1c0 tx=0x7f64100057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.997+0000 7f642e800640 1 -- 172.21.15.53:0/857012127 reap_dead start 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.998+0000 7f642f200640 1 -- 172.21.15.53:0/857012127 shutdown_connections 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.998+0000 7f642f200640 1 -- 172.21.15.53:0/857012127 >> 172.21.15.53:0/857012127 conn(0x7f64300d3050 msgr2=0x7f64300d3460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.998+0000 7f642f200640 1 -- 172.21.15.53:0/857012127 shutdown_connections 2024-01-16T15:26:27.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:27.998+0000 7f642f200640 1 -- 172.21.15.53:0/857012127 wait complete. 2024-01-16T15:26:27.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.000+0000 7f642f200640 1 Processor -- start 2024-01-16T15:26:27.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.000+0000 7f642f200640 1 -- start start 2024-01-16T15:26:28.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6410003f40 0x7f6430147dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64300b7e00 0x7f6430148610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64300b8280 0x7f6430148b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64300c9e70 con 0x7f64300b8280 2024-01-16T15:26:28.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64300b7690 con 0x7f64300b7e00 2024-01-16T15:26:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64300b7970 con 0x7f6410003f40 2024-01-16T15:26:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64300b8280 0x7f6430148b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64300b8280 0x7f6430148b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48612/0 (socket says 172.21.15.53:48612) 2024-01-16T15:26:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.002+0000 7f642e800640 1 -- 172.21.15.53:0/3417248667 learned_addr learned my addr 172.21.15.53:0/3417248667 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642de00640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6410003f40 0x7f6430147dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f6427e00640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64300b7e00 0x7f6430148610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642e800640 1 -- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6410003f40 msgr2=0x7f6430147dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642e800640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6410003f40 0x7f6430147dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642e800640 1 -- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64300b7e00 msgr2=0x7f6430148610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642e800640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f64300b7e00 0x7f6430148610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642e800640 1 -- 172.21.15.53:0/3417248667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6430148300 con 0x7f64300b8280 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.003+0000 7f642e800640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f64300b8280 0x7f6430148b50 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f64300cf810 tx=0x7f6428010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.004+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642800e8a0 con 0x7f64300b8280 2024-01-16T15:26:28.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.004+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f642800ea00 con 0x7f64300b8280 2024-01-16T15:26:28.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.004+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642800ed90 con 0x7f64300b8280 2024-01-16T15:26:28.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.004+0000 7f642f200640 1 -- 172.21.15.53:0/3417248667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64300c6d00 con 0x7f64300b8280 2024-01-16T15:26:28.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.004+0000 7f642f200640 1 -- 172.21.15.53:0/3417248667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f643014a6a0 con 0x7f64300b8280 2024-01-16T15:26:28.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.005+0000 7f642f200640 1 -- 172.21.15.53:0/3417248667 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f643008ba80 con 0x7f64300b8280 2024-01-16T15:26:28.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.006+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f64280143f0 con 0x7f64300b8280 2024-01-16T15:26:28.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.007+0000 7f6426000640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f640806c610 0x7f640806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:28.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.007+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6428088360 con 0x7f64300b8280 2024-01-16T15:26:28.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.008+0000 7f642de00640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f640806c610 0x7f640806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:28.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.009+0000 7f642de00640 1 --2- 172.21.15.53:0/3417248667 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f640806c610 0x7f640806ead0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f6420003b60 tx=0x7f642000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:28.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.011+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6428054c20 con 0x7f64300b8280 2024-01-16T15:26:28.240 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.242+0000 7f642f200640 1 -- 172.21.15.53:0/3417248667 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64300b2900 con 0x7f640806c610 2024-01-16T15:26:28.252 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:28.255+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:28.253 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:28.255+0000 7f6426000640 1 -- 172.21.15.53:0/3417248667 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f64200047c0 con 0x7f640806c610 2024-01-16T15:26:28.253 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:28.253 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:28.268 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:28.830 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:28.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:30.269 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:30.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.514+0000 7f2915000640 1 Processor -- start 2024-01-16T15:26:30.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.515+0000 7f2915000640 1 -- start start 2024-01-16T15:26:30.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.516+0000 7f2915000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:30.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.516+0000 7f2915000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f291006c470 0x7f29100cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:30.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.516+0000 7f2915000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291015c5c0 con 0x7f291006dab0 2024-01-16T15:26:30.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.516+0000 7f2915000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291006d690 con 0x7f291006c470 2024-01-16T15:26:30.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.516+0000 7f2915000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29100cebb0 con 0x7f29101447f0 2024-01-16T15:26:30.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.517+0000 7f290f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:30.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.517+0000 7f290f400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:46074/0 (socket says 172.21.15.53:46074) 2024-01-16T15:26:30.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.517+0000 7f290f400640 1 -- 172.21.15.53:0/2973475818 learned_addr learned my addr 172.21.15.53:0/2973475818 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:30.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.517+0000 7f290fe00640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f291006dab0 0x7f291015c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.53:44028/0 (socket says 172.21.15.53:44028) 2024-01-16T15:26:30.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.517+0000 7f2907e00640 1 --2- 172.21.15.53:0/2973475818 >> v2:172.21.15.146:3300/0 conn(0x7f291006c470 0x7f29100cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:30.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.518+0000 7f290f400640 1 -- 172.21.15.53:0/2973475818 >> v2:172.21.15.146:3300/0 conn(0x7f291006c470 msgr2=0x7f29100cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:30.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.518+0000 7f290ea00640 1 -- 172.21.15.53:0/2973475818 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377502052 0 0) 0x7f291015c5c0 con 0x7f291006dab0 2024-01-16T15:26:30.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.518+0000 7f290f400640 1 --2- 172.21.15.53:0/2973475818 >> v2:172.21.15.146:3300/0 conn(0x7f291006c470 0x7f29100cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:30.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.518+0000 7f290f400640 1 -- 172.21.15.53:0/2973475818 >> v1:172.21.15.146:6789/0 conn(0x7f291006dab0 legacy=0x7f291015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:30.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.518+0000 7f290f400640 1 -- 172.21.15.53:0/2973475818 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291006ca10 con 0x7f29101447f0 2024-01-16T15:26:30.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.520+0000 7f290f400640 1 --2- 172.21.15.53:0/2973475818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015c080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2900009870 tx=0x7f290002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe17ea712fe5b015 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:30.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.520+0000 7f290ea00640 1 -- 172.21.15.53:0/2973475818 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900031640 con 0x7f29101447f0 2024-01-16T15:26:30.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.520+0000 7f290ea00640 1 -- 172.21.15.53:0/2973475818 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29000317a0 con 0x7f29101447f0 2024-01-16T15:26:30.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.520+0000 7f290ea00640 1 -- 172.21.15.53:0/2973475818 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900031b30 con 0x7f29101447f0 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 -- 172.21.15.53:0/2973475818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 msgr2=0x7f291015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 --2- 172.21.15.53:0/2973475818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015c080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2900009870 tx=0x7f290002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 -- 172.21.15.53:0/2973475818 shutdown_connections 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 --2- 172.21.15.53:0/2973475818 >> v2:172.21.15.146:3300/0 conn(0x7f291006c470 0x7f29100cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 --2- 172.21.15.53:0/2973475818 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015c080 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 -- 172.21.15.53:0/2973475818 >> 172.21.15.53:0/2973475818 conn(0x7f291006a3e0 msgr2=0x7f291006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 -- 172.21.15.53:0/2973475818 shutdown_connections 2024-01-16T15:26:30.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.521+0000 7f2915000640 1 -- 172.21.15.53:0/2973475818 wait complete. 2024-01-16T15:26:30.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.523+0000 7f2915000640 1 Processor -- start 2024-01-16T15:26:30.521 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.523+0000 7f2915000640 1 -- start start 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f2915000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f291006c470 0x7f291013a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f2915000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f291006dab0 0x7f29100cb000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f2915000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f2915000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c4410 con 0x7f291006c470 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f2915000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29100c3de0 con 0x7f291006dab0 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f2915000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29100cb9a0 con 0x7f29101447f0 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f290f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f291006c470 0x7f291013a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:30.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.524+0000 7f290f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f291006c470 0x7f291013a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:48624/0 (socket says 172.21.15.53:48624) 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 -- 172.21.15.53:0/4268809533 learned_addr learned my addr 172.21.15.53:0/4268809533 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f2907e00640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f291006dab0 0x7f29100cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290fe00640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 -- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 msgr2=0x7f291015a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 -- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f291006dab0 msgr2=0x7f29100cb000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f291006dab0 0x7f29100cb000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 -- 172.21.15.53:0/4268809533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29000094d0 con 0x7f291006c470 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290fe00640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29101447f0 0x7f291015a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:30.524 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.525+0000 7f290f400640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f291006c470 0x7f291013a950 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f2900000c00 tx=0x7f29000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:30.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.526+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290003a930 con 0x7f291006c470 2024-01-16T15:26:30.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.526+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290003aa90 con 0x7f291006c470 2024-01-16T15:26:30.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.526+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290003ae40 con 0x7f291006c470 2024-01-16T15:26:30.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.526+0000 7f2915000640 1 -- 172.21.15.53:0/4268809533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29100cbc80 con 0x7f291006c470 2024-01-16T15:26:30.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.526+0000 7f2915000640 1 -- 172.21.15.53:0/4268809533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f291013ae90 con 0x7f291006c470 2024-01-16T15:26:30.525 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.527+0000 7f2915000640 1 -- 172.21.15.53:0/4268809533 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291013b080 con 0x7f291006c470 2024-01-16T15:26:30.526 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.528+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f290002f050 con 0x7f291006c470 2024-01-16T15:26:30.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.529+0000 7f2907400640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f28e806c5e0 0x7f28e806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:30.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.529+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f29000aaa40 con 0x7f291006c470 2024-01-16T15:26:30.527 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.530+0000 7f2907e00640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f28e806c5e0 0x7f28e806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:30.529 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.531+0000 7f2907e00640 1 --2- 172.21.15.53:0/4268809533 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f28e806c5e0 0x7f28e806eaa0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f2910142d10 tx=0x7f28f8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:30.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.534+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2900076f60 con 0x7f291006c470 2024-01-16T15:26:30.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.765+0000 7f2915000640 1 -- 172.21.15.53:0/4268809533 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f291015c5b0 con 0x7f28e806c5e0 2024-01-16T15:26:30.776 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:30.778+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:30.776 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:30.778+0000 7f2907400640 1 -- 172.21.15.53:0/4268809533 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f291015c5b0 con 0x7f28e806c5e0 2024-01-16T15:26:30.777 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:30.777 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:32.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:33.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.034+0000 7f8ce2800640 1 Processor -- start 2024-01-16T15:26:33.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.035+0000 7f8ce2800640 1 -- start start 2024-01-16T15:26:33.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce2800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8cdc06c470 0x7f8cdc0cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc06dab0 0x7f8cdc0ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce2800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cdc06ca10 con 0x7f8cdc1447f0 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce2800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cdc06d690 con 0x7f8cdc06c470 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc0cebb0 con 0x7f8cdc06dab0 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce1400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8cdc1447f0 0x7f8cdc15c080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55678/0 (socket says 172.21.15.53:55678) 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.036+0000 7f8ce1400640 1 -- 172.21.15.53:0/1922089285 learned_addr learned my addr 172.21.15.53:0/1922089285 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8ce1e00640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc06dab0 0x7f8cdc0ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778746376 0 0) 0x7f8cdc06ca10 con 0x7f8cdc1447f0 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8ce0a00640 1 --2- 172.21.15.53:0/1922089285 >> v2:172.21.15.146:3300/0 conn(0x7f8cdc06c470 0x7f8cdc0cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cbc003520 con 0x7f8cdc1447f0 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2463312501 0 0) 0x7f8cbc003520 con 0x7f8cdc1447f0 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc06dab0 msgr2=0x7f8cdc0ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc06dab0 0x7f8cdc0ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 >> v2:172.21.15.146:3300/0 conn(0x7f8cdc06c470 msgr2=0x7f8cdc0cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 --2- 172.21.15.53:0/1922089285 >> v2:172.21.15.146:3300/0 conn(0x7f8cdc06c470 0x7f8cdc0cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc06ccf0 con 0x7f8cdc1447f0 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f8ccc001670 con 0x7f8cdc1447f0 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 >> v1:172.21.15.53:6789/0 conn(0x7f8cdc1447f0 legacy=0x7f8cdc15c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cbc0047c0 0x7f8cbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cbc005180 0x7f8cbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc06d690 con 0x7f8cbc003970 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc0cebb0 con 0x7f8cbc0047c0 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cbc00c020 con 0x7f8cbc005180 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.037+0000 7f8ce1e00640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc06dab0 0x7f8cdc0ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.038+0000 7f8ce1400640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.038+0000 7f8ce1400640 1 -- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cbc005180 msgr2=0x7f8cbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.038+0000 7f8ce1400640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cbc005180 0x7f8cbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.038+0000 7f8ce1400640 1 -- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cbc0047c0 msgr2=0x7f8cbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.038+0000 7f8ce1400640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cbc0047c0 0x7f8cbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.038+0000 7f8ce1400640 1 -- 172.21.15.53:0/1922089285 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc003520 con 0x7f8cbc003970 2024-01-16T15:26:33.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.039+0000 7f8ce1400640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cbc003d90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00c240 tx=0x7f8cbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1142496ae8bb4d49 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:33.040 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.039+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc02cb00 con 0x7f8cbc003970 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.039+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc02cc60 con 0x7f8cbc003970 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8cdbe00640 1 -- 172.21.15.53:0/1922089285 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc005af0 con 0x7f8cbc003970 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce2800640 1 -- 172.21.15.53:0/1922089285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 msgr2=0x7f8cbc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce2800640 1 --2- 172.21.15.53:0/1922089285 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cbc003d90 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00c240 tx=0x7f8cbc0057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce1e00640 1 -- 172.21.15.53:0/1922089285 reap_dead start 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce2800640 1 -- 172.21.15.53:0/1922089285 shutdown_connections 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce2800640 1 -- 172.21.15.53:0/1922089285 >> 172.21.15.53:0/1922089285 conn(0x7f8cdc06a3e0 msgr2=0x7f8cdc06a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce2800640 1 -- 172.21.15.53:0/1922089285 shutdown_connections 2024-01-16T15:26:33.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.040+0000 7f8ce2800640 1 -- 172.21.15.53:0/1922089285 wait complete. 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.042+0000 7f8ce2800640 1 Processor -- start 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.042+0000 7f8ce2800640 1 -- start start 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cdc153600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc14d890 0x7f8cdc1f6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cdc1f65c0 0x7f8cdc12dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce2800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc1273a0 con 0x7f8cbc003970 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc129920 con 0x7f8cdc1f65c0 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cdc153b40 con 0x7f8cdc14d890 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.043+0000 7f8ce1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cdc153600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cdc153600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45450/0 (socket says 172.21.15.53:45450) 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 -- 172.21.15.53:0/457908120 learned_addr learned my addr 172.21.15.53:0/457908120 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:33.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce0a00640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc14d890 0x7f8cdc1f6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1e00640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cdc1f65c0 0x7f8cdc12dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 -- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc14d890 msgr2=0x7f8cdc1f6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8cdc14d890 0x7f8cdc1f6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 -- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cdc1f65c0 msgr2=0x7f8cdc12dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cdc1f65c0 0x7f8cdc12dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 -- 172.21.15.53:0/457908120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cdc1f6c00 con 0x7f8cbc003970 2024-01-16T15:26:33.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1e00640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8cdc1f65c0 0x7f8cdc12dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.044+0000 7f8ce1400640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8cbc003970 0x7f8cdc153600 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f8ccc00c240 tx=0x7f8ccc001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.045+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc03c040 con 0x7f8cbc003970 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.045+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc001630 con 0x7f8cbc003970 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.045+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ccc007270 con 0x7f8cbc003970 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.045+0000 7f8ce2800640 1 -- 172.21.15.53:0/457908120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cdc1f6ee0 con 0x7f8cbc003970 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.045+0000 7f8ce2800640 1 -- 172.21.15.53:0/457908120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cdc126fc0 con 0x7f8cbc003970 2024-01-16T15:26:33.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.046+0000 7f8ce2800640 1 -- 172.21.15.53:0/457908120 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cdc1f69a0 con 0x7f8cbc003970 2024-01-16T15:26:33.046 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.047+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f8ccc032070 con 0x7f8cbc003970 2024-01-16T15:26:33.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.048+0000 7f8cd9600640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cb406c5e0 0x7f8cb406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:33.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.048+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8ccc0a9ce0 con 0x7f8cbc003970 2024-01-16T15:26:33.047 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.048+0000 7f8ce0a00640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cb406c5e0 0x7f8cb406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:33.048 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.050+0000 7f8ce0a00640 1 --2- 172.21.15.53:0/457908120 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8cb406c5e0 0x7f8cb406eaa0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f8cdc12bfc0 tx=0x7f8cc4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:33.051 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.052+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8ccc076650 con 0x7f8cbc003970 2024-01-16T15:26:33.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.290+0000 7f8ce2800640 1 -- 172.21.15.53:0/457908120 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8cdc0cea40 con 0x7f8cb406c5e0 2024-01-16T15:26:33.305 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:33.306+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:33.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:33.306+0000 7f8cd9600640 1 -- 172.21.15.53:0/457908120 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f8cdc0cea40 con 0x7f8cb406c5e0 2024-01-16T15:26:33.305 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:33.306 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:33.319 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:35.321 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:35.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.564+0000 7f83b4c00640 1 Processor -- start 2024-01-16T15:26:35.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.564+0000 7f83b4c00640 1 -- start start 2024-01-16T15:26:35.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.566+0000 7f83b4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:35.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.566+0000 7f83b4c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00d0a10 con 0x7f83b00cfe70 2024-01-16T15:26:35.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.566+0000 7f83b4c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b0161e30 con 0x7f83b00cf9f0 2024-01-16T15:26:35.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.566+0000 7f83b4c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00d1f50 con 0x7f83b0160920 2024-01-16T15:26:35.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.566+0000 7f83afe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f83b00cfe70 0x7f83b0161720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55690/0 (socket says 172.21.15.53:55690) 2024-01-16T15:26:35.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.566+0000 7f83afe00640 1 -- 172.21.15.53:0/2647624241 learned_addr learned my addr 172.21.15.53:0/2647624241 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:35.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83af400640 1 --2- 172.21.15.53:0/2647624241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:35.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424997964 0 0) 0x7f83b00d0a10 con 0x7f83b00cfe70 2024-01-16T15:26:35.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8390003520 con 0x7f83b00cfe70 2024-01-16T15:26:35.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 547274859 0 0) 0x7f83b0161e30 con 0x7f83b00cf9f0 2024-01-16T15:26:35.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b00d0a10 con 0x7f83b00cf9f0 2024-01-16T15:26:35.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186769376 0 0) 0x7f8390003520 con 0x7f83b00cfe70 2024-01-16T15:26:35.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83af400640 1 -- 172.21.15.53:0/2647624241 >> v1:172.21.15.146:6789/0 conn(0x7f83b00cf9f0 legacy=0x7f83b0161010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:35.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83af400640 1 -- 172.21.15.53:0/2647624241 >> v1:172.21.15.53:6789/0 conn(0x7f83b00cfe70 legacy=0x7f83b0161720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:35.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.567+0000 7f83af400640 1 -- 172.21.15.53:0/2647624241 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00cf080 con 0x7f83b0160920 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.569+0000 7f83af400640 1 --2- 172.21.15.53:0/2647624241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b3f30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009dc0 tx=0x7f83a002bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b99392382885ec8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.569+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a8001670 con 0x7f83b0160920 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.569+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83a002c910 con 0x7f83b0160920 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83ae000640 1 -- 172.21.15.53:0/2647624241 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83a002cca0 con 0x7f83b0160920 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 -- 172.21.15.53:0/2647624241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 msgr2=0x7f83b00b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 --2- 172.21.15.53:0/2647624241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b3f30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f83a0009dc0 tx=0x7f83a002bde0 comp rx=0 tx=0).stop 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 -- 172.21.15.53:0/2647624241 shutdown_connections 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 --2- 172.21.15.53:0/2647624241 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b3f30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 -- 172.21.15.53:0/2647624241 >> 172.21.15.53:0/2647624241 conn(0x7f83b00d3090 msgr2=0x7f83b00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:35.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 -- 172.21.15.53:0/2647624241 shutdown_connections 2024-01-16T15:26:35.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.570+0000 7f83b4c00640 1 -- 172.21.15.53:0/2647624241 wait complete. 2024-01-16T15:26:35.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.572+0000 7f83b4c00640 1 Processor -- start 2024-01-16T15:26:35.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.573+0000 7f83b4c00640 1 -- start start 2024-01-16T15:26:35.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83b4c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83b00cf9f0 0x7f83b015a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:35.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83b4c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b00cfe70 0x7f83b00cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:35.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83b4c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:35.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83b4c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b015a890 con 0x7f83b00cfe70 2024-01-16T15:26:35.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83b4c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00cbb10 con 0x7f83b00cf9f0 2024-01-16T15:26:35.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83b4c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83b00cb7d0 con 0x7f83b0160920 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b00cfe70 0x7f83b00cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83aea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b00cfe70 0x7f83b00cb290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45454/0 (socket says 172.21.15.53:45454) 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83aea00640 1 -- 172.21.15.53:0/2969858321 learned_addr learned my addr 172.21.15.53:0/2969858321 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.574+0000 7f83af400640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83b00cf9f0 0x7f83b015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83afe00640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83aea00640 1 -- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 msgr2=0x7f83b00b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83aea00640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f83b0160920 0x7f83b00b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83aea00640 1 -- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83b00cf9f0 msgr2=0x7f83b015a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83aea00640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83b00cf9f0 0x7f83b015a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83aea00640 1 -- 172.21.15.53:0/2969858321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83a0009a70 con 0x7f83b00cfe70 2024-01-16T15:26:35.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83af400640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f83b00cf9f0 0x7f83b015a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:26:35.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83aea00640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f83b00cfe70 0x7f83b00cb290 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7f839800c430 tx=0x7f839800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83980052d0 con 0x7f83b00cfe70 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8398005450 con 0x7f83b00cfe70 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.575+0000 7f83b4c00640 1 -- 172.21.15.53:0/2969858321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b01f2de0 con 0x7f83b00cfe70 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.576+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8398003830 con 0x7f83b00cfe70 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.576+0000 7f83b4c00640 1 -- 172.21.15.53:0/2969858321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b01f3380 con 0x7f83b00cfe70 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.577+0000 7f83b4c00640 1 -- 172.21.15.53:0/2969858321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b0067450 con 0x7f83b00cfe70 2024-01-16T15:26:35.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.578+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f8398003990 con 0x7f83b00cfe70 2024-01-16T15:26:35.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.579+0000 7f83a7400640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f838806c5e0 0x7f838806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:35.578 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.579+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8398088270 con 0x7f83b00cfe70 2024-01-16T15:26:35.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.579+0000 7f83af400640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f838806c5e0 0x7f838806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:35.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.581+0000 7f83af400640 1 --2- 172.21.15.53:0/2969858321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f838806c5e0 0x7f838806eaa0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f83a0036000 tx=0x7f83a0030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:35.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.584+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8398054790 con 0x7f83b00cfe70 2024-01-16T15:26:35.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.821+0000 7f83b4c00640 1 -- 172.21.15.53:0/2969858321 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f83b00ce7e0 con 0x7f838806c5e0 2024-01-16T15:26:35.833 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:35.834+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:35.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:35.834+0000 7f83a7400640 1 -- 172.21.15.53:0/2969858321 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f83a002ce50 con 0x7f838806c5e0 2024-01-16T15:26:35.834 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:35.834 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:37.849 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:38.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.090+0000 7f5a11400640 1 Processor -- start 2024-01-16T15:26:38.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.090+0000 7f5a11400640 1 -- start start 2024-01-16T15:26:38.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a11400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5a0c13df80 0x7f5a0c15bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a11400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c13e360 0x7f5a0c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a11400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f5a0c15f9c0 0x7f5a0c0cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a11400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c06c470 con 0x7f5a0c13df80 2024-01-16T15:26:38.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a11400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0c06c750 con 0x7f5a0c15f9c0 2024-01-16T15:26:38.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a11400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c06ca30 con 0x7f5a0c13e360 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a0a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5a0c13df80 0x7f5a0c15bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a0a800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5a0c13df80 0x7f5a0c15bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45460/0 (socket says 172.21.15.53:45460) 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.092+0000 7f5a0a800640 1 -- 172.21.15.53:0/1048449874 learned_addr learned my addr 172.21.15.53:0/1048449874 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a0b200640 1 --2- 172.21.15.53:0/1048449874 >> v2:172.21.15.146:3300/0 conn(0x7f5a0c15f9c0 0x7f5a0c0cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a09e00640 1 --2- 172.21.15.53:0/1048449874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c13e360 0x7f5a0c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a0a800640 1 -- 172.21.15.53:0/1048449874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c13e360 msgr2=0x7f5a0c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a0a800640 1 --2- 172.21.15.53:0/1048449874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c13e360 0x7f5a0c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a0a800640 1 -- 172.21.15.53:0/1048449874 >> v2:172.21.15.146:3300/0 conn(0x7f5a0c15f9c0 msgr2=0x7f5a0c0cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a0a800640 1 --2- 172.21.15.53:0/1048449874 >> v2:172.21.15.146:3300/0 conn(0x7f5a0c15f9c0 0x7f5a0c0cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.093+0000 7f5a0a800640 1 -- 172.21.15.53:0/1048449874 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0c06cd10 con 0x7f5a0c13df80 2024-01-16T15:26:38.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.095+0000 7f5a0a800640 1 --2- 172.21.15.53:0/1048449874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13df80 0x7f5a0c15bee0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c06e6d0 tx=0x7f59fc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e173e41f7b7ba6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:38.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.095+0000 7f5a09400640 1 -- 172.21.15.53:0/1048449874 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc02ca70 con 0x7f5a0c13df80 2024-01-16T15:26:38.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.095+0000 7f5a09400640 1 -- 172.21.15.53:0/1048449874 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59fc02cbd0 con 0x7f5a0c13df80 2024-01-16T15:26:38.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.095+0000 7f5a09400640 1 -- 172.21.15.53:0/1048449874 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59fc0343f0 con 0x7f5a0c13df80 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 -- 172.21.15.53:0/1048449874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13df80 msgr2=0x7f5a0c15bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 --2- 172.21.15.53:0/1048449874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13df80 0x7f5a0c15bee0 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7f5a0c06e6d0 tx=0x7f59fc02bed0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 -- 172.21.15.53:0/1048449874 shutdown_connections 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 --2- 172.21.15.53:0/1048449874 >> v2:172.21.15.146:3300/0 conn(0x7f5a0c15f9c0 0x7f5a0c0cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 --2- 172.21.15.53:0/1048449874 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c13e360 0x7f5a0c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 --2- 172.21.15.53:0/1048449874 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13df80 0x7f5a0c15bee0 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 -- 172.21.15.53:0/1048449874 >> 172.21.15.53:0/1048449874 conn(0x7f5a0c06a800 msgr2=0x7f5a0c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:38.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 -- 172.21.15.53:0/1048449874 shutdown_connections 2024-01-16T15:26:38.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.096+0000 7f5a11400640 1 -- 172.21.15.53:0/1048449874 wait complete. 2024-01-16T15:26:38.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.098+0000 7f5a11400640 1 Processor -- start 2024-01-16T15:26:38.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.098+0000 7f5a11400640 1 -- start start 2024-01-16T15:26:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.099+0000 7f5a11400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a0c13df80 0x7f5a0c1f00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.099+0000 7f5a11400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13e360 0x7f5a0c0afc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.099+0000 7f5a11400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c15f9c0 0x7f5a0c0b0140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.099+0000 7f5a11400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c12df40 con 0x7f5a0c13e360 2024-01-16T15:26:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.099+0000 7f5a11400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c12d330 con 0x7f5a0c13df80 2024-01-16T15:26:38.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.099+0000 7f5a11400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0c12d610 con 0x7f5a0c15f9c0 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13e360 0x7f5a0c0afc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13e360 0x7f5a0c0afc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45462/0 (socket says 172.21.15.53:45462) 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 -- 172.21.15.53:0/3349805491 learned_addr learned my addr 172.21.15.53:0/3349805491 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a0a800640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a0c13df80 0x7f5a0c1f00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a0b200640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c15f9c0 0x7f5a0c0b0140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 -- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c15f9c0 msgr2=0x7f5a0c0b0140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5a0c15f9c0 0x7f5a0c0b0140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 -- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a0c13df80 msgr2=0x7f5a0c1f00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a0c13df80 0x7f5a0c1f00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 -- 172.21.15.53:0/3349805491 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc009b30 con 0x7f5a0c13e360 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a0a800640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5a0c13df80 0x7f5a0c1f00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:38.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.100+0000 7f5a09e00640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5a0c13e360 0x7f5a0c0afc00 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7f59f400e9b0 tx=0x7f59f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.101+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f4009670 con 0x7f5a0c13e360 2024-01-16T15:26:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.101+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59f400c9a0 con 0x7f5a0c13e360 2024-01-16T15:26:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.101+0000 7f5a11400640 1 -- 172.21.15.53:0/3349805491 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0c12d8f0 con 0x7f5a0c13e360 2024-01-16T15:26:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.101+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59f400cd50 con 0x7f5a0c13e360 2024-01-16T15:26:38.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.101+0000 7f5a11400640 1 -- 172.21.15.53:0/3349805491 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0c1f0620 con 0x7f5a0c13e360 2024-01-16T15:26:38.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.102+0000 7f5a11400640 1 -- 172.21.15.53:0/3349805491 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0c0675d0 con 0x7f5a0c13e360 2024-01-16T15:26:38.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.103+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f59f4013470 con 0x7f5a0c13e360 2024-01-16T15:26:38.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.104+0000 7f5a02a00640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f59e406c610 0x7f59e406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:38.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.104+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f59f4088e20 con 0x7f5a0c13e360 2024-01-16T15:26:38.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.105+0000 7f5a0a800640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f59e406c610 0x7f59e406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:38.104 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.106+0000 7f5a0a800640 1 --2- 172.21.15.53:0/3349805491 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f59e406c610 0x7f59e406ead0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f59fc035000 tx=0x7f59fc030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:38.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.109+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f59f4055340 con 0x7f5a0c13e360 2024-01-16T15:26:38.341 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.342+0000 7f5a11400640 1 -- 172.21.15.53:0/3349805491 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a0c12d780 con 0x7f59e406c610 2024-01-16T15:26:38.354 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:38.356+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:38.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:38.356+0000 7f5a02a00640 1 -- 172.21.15.53:0/3349805491 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5a0c12d780 con 0x7f59e406c610 2024-01-16T15:26:38.355 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:38.356 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:40.371 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:40.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.615+0000 7fdeb6c00640 1 Processor -- start 2024-01-16T15:26:40.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.615+0000 7fdeb6c00640 1 -- start start 2024-01-16T15:26:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb6c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdeb01447f0 0x7fdeb015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb6c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb015c5c0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb6c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdeb006ca10 con 0x7fdeb006c470 2024-01-16T15:26:40.616 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb6c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb006d690 con 0x7fdeb01447f0 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdeb01447f0 0x7fdeb015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb5800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fdeb01447f0 0x7fdeb015c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45470/0 (socket says 172.21.15.53:45470) 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.617+0000 7fdeb5800640 1 -- 172.21.15.53:0/1479238260 learned_addr learned my addr 172.21.15.53:0/1479238260 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.618+0000 7fdeb5800640 1 -- 172.21.15.53:0/1479238260 >> v1:172.21.15.146:6789/0 conn(0x7fdeb006c470 legacy=0x7fdeb00cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.618+0000 7fdeb5800640 1 -- 172.21.15.53:0/1479238260 >> v1:172.21.15.53:6789/0 conn(0x7fdeb00cccf0 legacy=0x7fdeb015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.618+0000 7fdeb5800640 1 -- 172.21.15.53:0/1479238260 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb006ccf0 con 0x7fdeb01447f0 2024-01-16T15:26:40.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.619+0000 7fdeb5800640 1 --2- 172.21.15.53:0/1479238260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb01447f0 0x7fdeb015c080 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009870 tx=0x7fdea002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be8b34fd505dda6b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.619+0000 7fdeafe00640 1 -- 172.21.15.53:0/1479238260 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031860 con 0x7fdeb01447f0 2024-01-16T15:26:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.619+0000 7fdeafe00640 1 -- 172.21.15.53:0/1479238260 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea00319c0 con 0x7fdeb01447f0 2024-01-16T15:26:40.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.619+0000 7fdeafe00640 1 -- 172.21.15.53:0/1479238260 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0031d50 con 0x7fdeb01447f0 2024-01-16T15:26:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1479238260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb01447f0 msgr2=0x7fdeb015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 --2- 172.21.15.53:0/1479238260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb01447f0 0x7fdeb015c080 secure :-1 s=READY pgs=1901 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009870 tx=0x7fdea002b900 comp rx=0 tx=0).stop 2024-01-16T15:26:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1479238260 shutdown_connections 2024-01-16T15:26:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 --2- 172.21.15.53:0/1479238260 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb01447f0 0x7fdeb015c080 unknown :-1 s=CLOSED pgs=1901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:40.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1479238260 >> 172.21.15.53:0/1479238260 conn(0x7fdeb006a3e0 msgr2=0x7fdeb006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1479238260 shutdown_connections 2024-01-16T15:26:40.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.620+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1479238260 wait complete. 2024-01-16T15:26:40.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.622+0000 7fdeb6c00640 1 Processor -- start 2024-01-16T15:26:40.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.622+0000 7fdeb6c00640 1 -- start start 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb006c470 0x7fdeb01f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb6c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb00cccf0 0x7fdeb0139d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb01447f0 0x7fdeb00ad100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb6c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00ac970 con 0x7fdeb00cccf0 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb01f6dd0 con 0x7fdeb01447f0 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb01f70b0 con 0x7fdeb006c470 2024-01-16T15:26:40.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.623+0000 7fdeb4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb00cccf0 0x7fdeb0139d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb00cccf0 0x7fdeb0139d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:45484/0 (socket says 172.21.15.53:45484) 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 -- 172.21.15.53:0/1158158736 learned_addr learned my addr 172.21.15.53:0/1158158736 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb5800640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb006c470 0x7fdeb01f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 -- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb006c470 msgr2=0x7fdeb01f6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fdeb006c470 0x7fdeb01f6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 -- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb01447f0 msgr2=0x7fdeb00ad100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb6200640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb01447f0 0x7fdeb00ad100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb01447f0 0x7fdeb00ad100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 -- 172.21.15.53:0/1158158736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea00094d0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb6200640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fdeb01447f0 0x7fdeb00ad100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:40.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.624+0000 7fdeb4e00640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fdeb00cccf0 0x7fdeb0139d70 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fde9800c430 tx=0x7fde9800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.625+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98011790 con 0x7fdeb00cccf0 2024-01-16T15:26:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.625+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde980118f0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.625+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98011ce0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.625+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1158158736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb00acbe0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.625+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1158158736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb01f7390 con 0x7fdeb00cccf0 2024-01-16T15:26:40.625 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.626+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1158158736 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb00ace00 con 0x7fdeb00cccf0 2024-01-16T15:26:40.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.627+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fde9801c460 con 0x7fdeb00cccf0 2024-01-16T15:26:40.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.628+0000 7fdead600640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde8806c5e0 0x7fde8806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:40.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.628+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fde98088ba0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.629+0000 7fdeb5800640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde8806c5e0 0x7fde8806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:40.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.630+0000 7fdeb5800640 1 --2- 172.21.15.53:0/1158158736 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fde8806c5e0 0x7fde8806eaa0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fdea0036000 tx=0x7fdea003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:40.632 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.633+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fde980550c0 con 0x7fdeb00cccf0 2024-01-16T15:26:40.860 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.861+0000 7fdeb6c00640 1 -- 172.21.15.53:0/1158158736 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdeb0144bd0 con 0x7fde8806c5e0 2024-01-16T15:26:40.874 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:40.875+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:40.874 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:40.875+0000 7fdead600640 1 -- 172.21.15.53:0/1158158736 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdea0031dc0 con 0x7fde8806c5e0 2024-01-16T15:26:40.875 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:40.875 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:40.889 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:42.891 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:43.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.130+0000 7f6673200640 1 Processor -- start 2024-01-16T15:26:43.129 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.130+0000 7f6673200640 1 -- start start 2024-01-16T15:26:43.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6673200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6674146850 0x7f66740715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6673200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f6674071b10 0x7f66740ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6673200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667406d480 con 0x7f66740adb00 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6673200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6674071fa0 con 0x7f6674146850 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6673200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66740b3490 con 0x7f6674071b10 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6674146850 0x7f66740715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f6674146850 0x7f66740715d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58864/0 (socket says 172.21.15.53:58864) 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 -- 172.21.15.53:0/628128517 learned_addr learned my addr 172.21.15.53:0/628128517 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 -- 172.21.15.53:0/628128517 >> v2:172.21.15.146:3300/0 conn(0x7f6674071b10 msgr2=0x7f66740ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 --2- 172.21.15.53:0/628128517 >> v2:172.21.15.146:3300/0 conn(0x7f6674071b10 0x7f66740ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 -- 172.21.15.53:0/628128517 >> v1:172.21.15.146:6789/0 conn(0x7f66740adb00 legacy=0x7f66740700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:43.133 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.132+0000 7f6671e00640 1 -- 172.21.15.53:0/628128517 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66740b3770 con 0x7f6674146850 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.134+0000 7f6671e00640 1 --2- 172.21.15.53:0/628128517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6674146850 0x7f66740715d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f6674160cc0 tx=0x7f666402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e4cd18f31ae39d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.134+0000 7f6671400640 1 -- 172.21.15.53:0/628128517 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6664031870 con 0x7f6674146850 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.134+0000 7f6671400640 1 -- 172.21.15.53:0/628128517 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66640319d0 con 0x7f6674146850 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.134+0000 7f6671400640 1 -- 172.21.15.53:0/628128517 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6664031d60 con 0x7f6674146850 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.134+0000 7f6673200640 1 -- 172.21.15.53:0/628128517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6674146850 msgr2=0x7f66740715d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.134+0000 7f6673200640 1 --2- 172.21.15.53:0/628128517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6674146850 0x7f66740715d0 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f6674160cc0 tx=0x7f666402b910 comp rx=0 tx=0).stop 2024-01-16T15:26:43.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.135+0000 7f6673200640 1 -- 172.21.15.53:0/628128517 shutdown_connections 2024-01-16T15:26:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.135+0000 7f6673200640 1 --2- 172.21.15.53:0/628128517 >> v2:172.21.15.146:3300/0 conn(0x7f6674071b10 0x7f66740ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.135+0000 7f6673200640 1 --2- 172.21.15.53:0/628128517 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6674146850 0x7f66740715d0 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.135+0000 7f6673200640 1 -- 172.21.15.53:0/628128517 >> 172.21.15.53:0/628128517 conn(0x7f667406a3e0 msgr2=0x7f667406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.135+0000 7f6673200640 1 -- 172.21.15.53:0/628128517 shutdown_connections 2024-01-16T15:26:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.135+0000 7f6673200640 1 -- 172.21.15.53:0/628128517 wait complete. 2024-01-16T15:26:43.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.136+0000 7f6673200640 1 Processor -- start 2024-01-16T15:26:43.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.137+0000 7f6673200640 1 -- start start 2024-01-16T15:26:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.137+0000 7f6673200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6674071b10 0x7f66741437e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6673200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66740adb00 0x7f66741e8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6673200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6674146850 0x7f66741e66b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6673200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66741e7bf0 con 0x7f66740adb00 2024-01-16T15:26:43.138 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6673200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66741e8550 con 0x7f6674146850 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6673200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66741e6f70 con 0x7f6674071b10 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66740adb00 0x7f66741e8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66740adb00 0x7f66741e8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58880/0 (socket says 172.21.15.53:58880) 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 -- 172.21.15.53:0/1909540100 learned_addr learned my addr 172.21.15.53:0/1909540100 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6672800640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6674146850 0x7f66741e66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 -- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6674071b10 msgr2=0x7f66741437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6674071b10 0x7f66741437e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 -- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6674146850 msgr2=0x7f66741e66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6674146850 0x7f66741e66b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f666be00640 1 -- 172.21.15.53:0/1909540100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66640094d0 con 0x7f66740adb00 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.138+0000 7f6672800640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6674146850 0x7f66741e66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.139+0000 7f666be00640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f66740adb00 0x7f66741e8010 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f665c00b4d0 tx=0x7f665c00b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:43.139 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.139+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c010870 con 0x7f66740adb00 2024-01-16T15:26:43.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.139+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f665c0109d0 con 0x7f66740adb00 2024-01-16T15:26:43.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.139+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c010d80 con 0x7f66740adb00 2024-01-16T15:26:43.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.139+0000 7f6673200640 1 -- 172.21.15.53:0/1909540100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66741e7250 con 0x7f66740adb00 2024-01-16T15:26:43.140 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.139+0000 7f6673200640 1 -- 172.21.15.53:0/1909540100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66741e7810 con 0x7f66740adb00 2024-01-16T15:26:43.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.140+0000 7f6673200640 1 -- 172.21.15.53:0/1909540100 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6674067450 con 0x7f66740adb00 2024-01-16T15:26:43.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.141+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f665c00f3c0 con 0x7f66740adb00 2024-01-16T15:26:43.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.142+0000 7f666a000640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f664c06c610 0x7f664c06ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:43.141 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.142+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f665c088b50 con 0x7f66740adb00 2024-01-16T15:26:43.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.143+0000 7f6671e00640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f664c06c610 0x7f664c06ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:43.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.144+0000 7f6671e00640 1 --2- 172.21.15.53:0/1909540100 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f664c06c610 0x7f664c06ead0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f6664036000 tx=0x7f6664030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:43.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.147+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f665c055070 con 0x7f66740adb00 2024-01-16T15:26:43.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.376+0000 7f6673200640 1 -- 172.21.15.53:0/1909540100 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f66740af600 con 0x7f664c06c610 2024-01-16T15:26:43.388 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:43.389+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:43.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:43.389+0000 7f666a000640 1 -- 172.21.15.53:0/1909540100 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f66740af600 con 0x7f664c06c610 2024-01-16T15:26:43.389 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:43.389 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:43.403 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:45.405 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:45.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.639+0000 7ff747200640 1 Processor -- start 2024-01-16T15:26:45.639 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.639+0000 7ff747200640 1 -- start start 2024-01-16T15:26:45.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.641+0000 7ff747200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff7400d0b60 0x7ff7400ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.641+0000 7ff747200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff740071de0 con 0x7ff74006a3e0 2024-01-16T15:26:45.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.641+0000 7ff747200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7400af270 con 0x7ff7400aede0 2024-01-16T15:26:45.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.641+0000 7ff747200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff740072030 con 0x7ff7400d0b60 2024-01-16T15:26:45.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.642+0000 7ff73fe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff74006a3e0 0x7ff74006a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:53844/0 (socket says 172.21.15.53:53844) 2024-01-16T15:26:45.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.642+0000 7ff73fe00640 1 -- 172.21.15.53:0/1853985057 learned_addr learned my addr 172.21.15.53:0/1853985057 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:26:45.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.642+0000 7ff73f400640 1 --2- 172.21.15.53:0/1853985057 >> v2:172.21.15.146:3300/0 conn(0x7ff7400d0b60 0x7ff7400ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.642+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310363311 0 0) 0x7ff740071de0 con 0x7ff74006a3e0 2024-01-16T15:26:45.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.642+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff720003520 con 0x7ff74006a3e0 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3732788837 0 0) 0x7ff720003520 con 0x7ff74006a3e0 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 >> v2:172.21.15.146:3300/0 conn(0x7ff7400d0b60 msgr2=0x7ff7400ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 --2- 172.21.15.53:0/1853985057 >> v2:172.21.15.146:3300/0 conn(0x7ff7400d0b60 0x7ff7400ac8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 >> v1:172.21.15.146:6789/0 conn(0x7ff7400aede0 legacy=0x7ff74006f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740072310 con 0x7ff74006a3e0 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7ff730001670 con 0x7ff74006a3e0 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 >> v1:172.21.15.53:6789/0 conn(0x7ff74006a3e0 legacy=0x7ff74006a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720003970 0x7ff720003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.643 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff720005180 0x7ff72000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740072030 con 0x7ff720005180 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72000c020 con 0x7ff7200047c0 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72000c1e0 con 0x7ff720003970 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.643+0000 7ff745200640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff720005180 0x7ff72000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff73fe00640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720003970 0x7ff720003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff73f400640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff745200640 1 -- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720003970 msgr2=0x7ff720003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff745200640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720003970 0x7ff720003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff745200640 1 -- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7200047c0 msgr2=0x7ff720004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff745200640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff745200640 1 -- 172.21.15.53:0/1853985057 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff720003520 con 0x7ff720005180 2024-01-16T15:26:45.644 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.644+0000 7ff73f400640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff7200047c0 0x7ff720004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:45.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.645+0000 7ff745200640 1 -- 172.21.15.53:0/1853985057 reap_dead start 2024-01-16T15:26:45.645 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.645+0000 7ff745200640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff720005180 0x7ff72000b980 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ff738001e50 tx=0x7ff7200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b542173b371ce48 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.645+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738003b70 con 0x7ff720005180 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.645+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff738003d10 con 0x7ff720005180 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.645+0000 7ff73ea00640 1 -- 172.21.15.53:0/1853985057 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738005310 con 0x7ff720005180 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 -- 172.21.15.53:0/1853985057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff720005180 msgr2=0x7ff72000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff720005180 0x7ff72000b980 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7ff738001e50 tx=0x7ff7200057f0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 -- 172.21.15.53:0/1853985057 shutdown_connections 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 --2- 172.21.15.53:0/1853985057 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff720005180 0x7ff72000b980 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 -- 172.21.15.53:0/1853985057 >> 172.21.15.53:0/1853985057 conn(0x7ff7400cfef0 msgr2=0x7ff7400d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 -- 172.21.15.53:0/1853985057 shutdown_connections 2024-01-16T15:26:45.646 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.646+0000 7ff747200640 1 -- 172.21.15.53:0/1853985057 wait complete. 2024-01-16T15:26:45.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.648+0000 7ff747200640 1 Processor -- start 2024-01-16T15:26:45.647 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.648+0000 7ff747200640 1 -- start start 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff747200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720005180 0x7ff7401602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff747200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff74014fb20 0x7ff7400cc940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff747200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff74015e3a0 0x7ff740146430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff747200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff740152ca0 con 0x7ff74015e3a0 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff747200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7401607f0 con 0x7ff74014fb20 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff747200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff740151d40 con 0x7ff720005180 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff745200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff74015e3a0 0x7ff740146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff745200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff74015e3a0 0x7ff740146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58888/0 (socket says 172.21.15.53:58888) 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.649+0000 7ff745200640 1 -- 172.21.15.53:0/3925710495 learned_addr learned my addr 172.21.15.53:0/3925710495 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:45.649 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff73fe00640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720005180 0x7ff7401602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff73f400640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff74014fb20 0x7ff7400cc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff745200640 1 -- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720005180 msgr2=0x7ff7401602b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff745200640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff720005180 0x7ff7401602b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff745200640 1 -- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff74014fb20 msgr2=0x7ff7400cc940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff745200640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff74014fb20 0x7ff7400cc940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff745200640 1 -- 172.21.15.53:0/3925710495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff740152020 con 0x7ff74015e3a0 2024-01-16T15:26:45.650 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff745200640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff74015e3a0 0x7ff740146430 secure :-1 s=READY pgs=1907 cs=0 l=1 rev1=1 crypto rx=0x7ff73802c060 tx=0x7ff7380014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:45.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738006d50 con 0x7ff74015e3a0 2024-01-16T15:26:45.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff738003b80 con 0x7ff74015e3a0 2024-01-16T15:26:45.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.650+0000 7ff747200640 1 -- 172.21.15.53:0/3925710495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff740152300 con 0x7ff74015e3a0 2024-01-16T15:26:45.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.651+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738007230 con 0x7ff74015e3a0 2024-01-16T15:26:45.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.651+0000 7ff747200640 1 -- 172.21.15.53:0/3925710495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7401528c0 con 0x7ff74015e3a0 2024-01-16T15:26:45.651 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.652+0000 7ff747200640 1 -- 172.21.15.53:0/3925710495 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7400675d0 con 0x7ff74015e3a0 2024-01-16T15:26:45.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.653+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7ff738007390 con 0x7ff74015e3a0 2024-01-16T15:26:45.652 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.654+0000 7ff737e00640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff71806c5e0 0x7ff71806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:45.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.654+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff7380aa660 con 0x7ff74015e3a0 2024-01-16T15:26:45.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.654+0000 7ff73fe00640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff71806c5e0 0x7ff71806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:45.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.655+0000 7ff73fe00640 1 --2- 172.21.15.53:0/3925710495 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff71806c5e0 0x7ff71806eaa0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7ff730001050 tx=0x7ff730009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:45.657 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.658+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff738076af0 con 0x7ff74015e3a0 2024-01-16T15:26:45.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.890+0000 7ff747200640 1 -- 172.21.15.53:0/3925710495 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff74006eb60 con 0x7ff71806c5e0 2024-01-16T15:26:45.903 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:45.904+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:45.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:45.905+0000 7ff737e00640 1 -- 172.21.15.53:0/3925710495 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff7300015d0 con 0x7ff71806c5e0 2024-01-16T15:26:45.903 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:45.904 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:47.920 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:48.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.159+0000 7fd5eee00640 1 Processor -- start 2024-01-16T15:26:48.158 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.159+0000 7fd5eee00640 1 -- start start 2024-01-16T15:26:48.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.160+0000 7fd5eee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e81447f0 0x7fd5e815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.160+0000 7fd5eee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd5e806c470 0x7fd5e80cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.160+0000 7fd5eee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5e806dab0 0x7fd5e80ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.160+0000 7fd5eee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e806d690 con 0x7fd5e806dab0 2024-01-16T15:26:48.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.160+0000 7fd5eee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5e80cebb0 con 0x7fd5e806c470 2024-01-16T15:26:48.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.160+0000 7fd5eee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e815c5c0 con 0x7fd5e81447f0 2024-01-16T15:26:48.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.161+0000 7fd5ee400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5e806dab0 0x7fd5e80ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.161+0000 7fd5ee400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd5e806dab0 0x7fd5e80ccb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58900/0 (socket says 172.21.15.53:58900) 2024-01-16T15:26:48.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.161+0000 7fd5ee400640 1 -- 172.21.15.53:0/3327994741 learned_addr learned my addr 172.21.15.53:0/3327994741 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:48.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.161+0000 7fd5ed000640 1 --2- 172.21.15.53:0/3327994741 >> v2:172.21.15.146:3300/0 conn(0x7fd5e806c470 0x7fd5e80cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.161 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.161+0000 7fd5eda00640 1 --2- 172.21.15.53:0/3327994741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e81447f0 0x7fd5e815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.162+0000 7fd5ee400640 1 -- 172.21.15.53:0/3327994741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e81447f0 msgr2=0x7fd5e815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.162+0000 7fd5ee400640 1 --2- 172.21.15.53:0/3327994741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e81447f0 0x7fd5e815c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.162+0000 7fd5ee400640 1 -- 172.21.15.53:0/3327994741 >> v2:172.21.15.146:3300/0 conn(0x7fd5e806c470 msgr2=0x7fd5e80cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.162+0000 7fd5ee400640 1 --2- 172.21.15.53:0/3327994741 >> v2:172.21.15.146:3300/0 conn(0x7fd5e806c470 0x7fd5e80cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.162+0000 7fd5ee400640 1 -- 172.21.15.53:0/3327994741 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e806ca10 con 0x7fd5e806dab0 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.163+0000 7fd5ee400640 1 --2- 172.21.15.53:0/3327994741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80ccb20 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fd5e806e0c0 tx=0x7fd5e002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba508d9393be3370 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:48.162 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.163+0000 7fd5e7e00640 1 -- 172.21.15.53:0/3327994741 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e002c9f0 con 0x7fd5e806dab0 2024-01-16T15:26:48.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.163+0000 7fd5e7e00640 1 -- 172.21.15.53:0/3327994741 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5e002cb50 con 0x7fd5e806dab0 2024-01-16T15:26:48.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.163+0000 7fd5e7e00640 1 -- 172.21.15.53:0/3327994741 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5e00343f0 con 0x7fd5e806dab0 2024-01-16T15:26:48.163 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 -- 172.21.15.53:0/3327994741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 msgr2=0x7fd5e80ccb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 --2- 172.21.15.53:0/3327994741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80ccb20 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fd5e806e0c0 tx=0x7fd5e002be50 comp rx=0 tx=0).stop 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 -- 172.21.15.53:0/3327994741 shutdown_connections 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 --2- 172.21.15.53:0/3327994741 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80ccb20 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 --2- 172.21.15.53:0/3327994741 >> v2:172.21.15.146:3300/0 conn(0x7fd5e806c470 0x7fd5e80cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 --2- 172.21.15.53:0/3327994741 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e81447f0 0x7fd5e815c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 -- 172.21.15.53:0/3327994741 >> 172.21.15.53:0/3327994741 conn(0x7fd5e806a3e0 msgr2=0x7fd5e806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 -- 172.21.15.53:0/3327994741 shutdown_connections 2024-01-16T15:26:48.164 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.164+0000 7fd5eee00640 1 -- 172.21.15.53:0/3327994741 wait complete. 2024-01-16T15:26:48.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.166+0000 7fd5eee00640 1 Processor -- start 2024-01-16T15:26:48.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.166+0000 7fd5eee00640 1 -- start start 2024-01-16T15:26:48.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.167+0000 7fd5eee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e806c470 0x7fd5e8157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.167+0000 7fd5eee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80aea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.167+0000 7fd5eee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e81447f0 0x7fd5e813c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.167+0000 7fd5eee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80afc20 con 0x7fd5e806dab0 2024-01-16T15:26:48.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.167+0000 7fd5eee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80af500 con 0x7fd5e81447f0 2024-01-16T15:26:48.167 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.167+0000 7fd5eee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5e80af7e0 con 0x7fd5e806c470 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80aea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80aea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58914/0 (socket says 172.21.15.53:58914) 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 -- 172.21.15.53:0/4205216760 learned_addr learned my addr 172.21.15.53:0/4205216760 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5eda00640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e806c470 0x7fd5e8157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ee400640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e81447f0 0x7fd5e813c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 -- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e806c470 msgr2=0x7fd5e8157e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd5e806c470 0x7fd5e8157e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 -- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e81447f0 msgr2=0x7fd5e813c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e81447f0 0x7fd5e813c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ed000640 1 -- 172.21.15.53:0/4205216760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5e0009b30 con 0x7fd5e806dab0 2024-01-16T15:26:48.168 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.168+0000 7fd5ee400640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd5e81447f0 0x7fd5e813c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:48.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.169+0000 7fd5ed000640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd5e806dab0 0x7fd5e80aea80 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fd5d000d800 tx=0x7fd5d000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:48.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.169+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d00096a0 con 0x7fd5e806dab0 2024-01-16T15:26:48.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.169+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5d0009800 con 0x7fd5e806dab0 2024-01-16T15:26:48.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.169+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5d0009990 con 0x7fd5e806dab0 2024-01-16T15:26:48.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.169+0000 7fd5eee00640 1 -- 172.21.15.53:0/4205216760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5e80afe70 con 0x7fd5e806dab0 2024-01-16T15:26:48.169 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.169+0000 7fd5eee00640 1 -- 172.21.15.53:0/4205216760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5e8140560 con 0x7fd5e806dab0 2024-01-16T15:26:48.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.170+0000 7fd5eee00640 1 -- 172.21.15.53:0/4205216760 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5e8067450 con 0x7fd5e806dab0 2024-01-16T15:26:48.170 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.171+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fd5d0009af0 con 0x7fd5e806dab0 2024-01-16T15:26:48.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.173+0000 7fd5e5600640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c006c5e0 0x7fd5c006eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:48.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.173+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd5d0088cc0 con 0x7fd5e806dab0 2024-01-16T15:26:48.172 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.173+0000 7fd5eda00640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c006c5e0 0x7fd5c006eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:48.174 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.175+0000 7fd5eda00640 1 --2- 172.21.15.53:0/4205216760 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd5c006c5e0 0x7fd5c006eaa0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8006fd0 tx=0x7fd5d8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:48.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.177+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd5d0055580 con 0x7fd5e806dab0 2024-01-16T15:26:48.402 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.402+0000 7fd5eee00640 1 -- 172.21.15.53:0/4205216760 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd5e806d2b0 con 0x7fd5c006c5e0 2024-01-16T15:26:48.416 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:48.417+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:48.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:48.418+0000 7fd5e5600640 1 -- 172.21.15.53:0/4205216760 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd5d800b200 con 0x7fd5c006c5e0 2024-01-16T15:26:48.416 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:48.417 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:48.429 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:50.430 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:50.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.672+0000 7f5cd3800640 1 Processor -- start 2024-01-16T15:26:50.671 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.672+0000 7f5cd3800640 1 -- start start 2024-01-16T15:26:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.674+0000 7f5cd3800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc143ae0 0x7f5ccc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:50.673 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.674+0000 7f5cd3800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ccc0cf7d0 0x7f5ccc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.674+0000 7f5cd3800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0cfc30 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.674+0000 7f5cd3800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b1ee0 con 0x7f5ccc0cf7d0 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.674+0000 7f5cd3800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0b2780 con 0x7f5ccc143ae0 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.674+0000 7f5cc3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ccc0cf7d0 0x7f5ccc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.675+0000 7f5cc3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ccc0cf7d0 0x7f5ccc15dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58928/0 (socket says 172.21.15.53:58928) 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.675+0000 7f5cc3e00640 1 -- 172.21.15.53:0/3117785469 learned_addr learned my addr 172.21.15.53:0/3117785469 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:50.674 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.675+0000 7f5cc3e00640 1 -- 172.21.15.53:0/3117785469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc143ae0 msgr2=0x7f5ccc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.675+0000 7f5cc3e00640 1 --2- 172.21.15.53:0/3117785469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc143ae0 0x7f5ccc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.675+0000 7f5cc3e00640 1 -- 172.21.15.53:0/3117785469 >> v1:172.21.15.53:6789/0 conn(0x7f5ccc15e1e0 legacy=0x7f5ccc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.675+0000 7f5cc3e00640 1 -- 172.21.15.53:0/3117785469 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc15e990 con 0x7f5ccc0cf7d0 2024-01-16T15:26:50.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.676+0000 7f5cc3e00640 1 --2- 172.21.15.53:0/3117785469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc0cf7d0 0x7f5ccc15dca0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009870 tx=0x7f5cb402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1fbfba984a1d2f7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5ccbe00640 1 -- 172.21.15.53:0/3117785469 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4030840 con 0x7f5ccc0cf7d0 2024-01-16T15:26:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5ccbe00640 1 -- 172.21.15.53:0/3117785469 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb40309a0 con 0x7f5ccc0cf7d0 2024-01-16T15:26:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5ccbe00640 1 -- 172.21.15.53:0/3117785469 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4030d30 con 0x7f5ccc0cf7d0 2024-01-16T15:26:50.676 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5cd3800640 1 -- 172.21.15.53:0/3117785469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc0cf7d0 msgr2=0x7f5ccc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5cd3800640 1 --2- 172.21.15.53:0/3117785469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc0cf7d0 0x7f5ccc15dca0 secure :-1 s=READY pgs=1911 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009870 tx=0x7f5cb402b980 comp rx=0 tx=0).stop 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5cd3800640 1 -- 172.21.15.53:0/3117785469 shutdown_connections 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5cd3800640 1 --2- 172.21.15.53:0/3117785469 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc0cf7d0 0x7f5ccc15dca0 unknown :-1 s=CLOSED pgs=1911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5cd3800640 1 --2- 172.21.15.53:0/3117785469 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc143ae0 0x7f5ccc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.677+0000 7f5cd3800640 1 -- 172.21.15.53:0/3117785469 >> 172.21.15.53:0/3117785469 conn(0x7f5ccc06a800 msgr2=0x7f5ccc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.678+0000 7f5cd3800640 1 -- 172.21.15.53:0/3117785469 shutdown_connections 2024-01-16T15:26:50.677 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.678+0000 7f5cd3800640 1 -- 172.21.15.53:0/3117785469 wait complete. 2024-01-16T15:26:50.678 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.679+0000 7f5cd3800640 1 Processor -- start 2024-01-16T15:26:50.679 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 -- start start 2024-01-16T15:26:50.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc0cf7d0 0x7f5ccc0aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:50.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ccc143ae0 0x7f5ccc0b0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:50.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc15e1e0 0x7f5ccc15ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:50.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc13a970 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.681 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc13ac10 con 0x7f5ccc143ae0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.680+0000 7f5cd3800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc13aef0 con 0x7f5ccc0cf7d0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc15e1e0 0x7f5ccc15ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc15e1e0 0x7f5ccc15ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58934/0 (socket says 172.21.15.53:58934) 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 -- 172.21.15.53:0/1083744703 learned_addr learned my addr 172.21.15.53:0/1083744703 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd0e00640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc0cf7d0 0x7f5ccc0aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cc3e00640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ccc143ae0 0x7f5ccc0b0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 -- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc0cf7d0 msgr2=0x7f5ccc0aea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ccc0cf7d0 0x7f5ccc0aea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 -- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ccc143ae0 msgr2=0x7f5ccc0b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ccc143ae0 0x7f5ccc0b0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.681+0000 7f5cd1800640 1 -- 172.21.15.53:0/1083744703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cb40094d0 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.682+0000 7f5cd1800640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ccc15e1e0 0x7f5ccc15ca60 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f5cc400c730 tx=0x7f5cc400cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.682+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4006d50 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.682 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.682+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cc4003970 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.682+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cc4003d20 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.682+0000 7f5cd3800640 1 -- 172.21.15.53:0/1083744703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ccc13b1d0 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.682+0000 7f5cd3800640 1 -- 172.21.15.53:0/1083744703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc15cfa0 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.683+0000 7f5cd3800640 1 -- 172.21.15.53:0/1083744703 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc0675d0 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.683 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.684+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f5cc40050d0 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.685+0000 7f5cc9600640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ca406c5e0 0x7f5ca406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:50.684 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.685+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5cc4088710 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.686+0000 7f5cd0e00640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ca406c5e0 0x7f5ca406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:50.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.687+0000 7f5cd0e00640 1 --2- 172.21.15.53:0/1083744703 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5ca406c5e0 0x7f5ca406eaa0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc06ad00 tx=0x7f5cbc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:50.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.690+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5cc4054c30 con 0x7f5ccc15e1e0 2024-01-16T15:26:50.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.923+0000 7f5cd3800640 1 -- 172.21.15.53:0/1083744703 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ccc15e640 con 0x7f5ca406c5e0 2024-01-16T15:26:50.935 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:50.936+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:50.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:50.937+0000 7f5cc9600640 1 -- 172.21.15.53:0/1083744703 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5cbc009e40 con 0x7f5ca406c5e0 2024-01-16T15:26:50.936 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:50.936 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:52.951 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:53.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.192+0000 7f51efe00640 1 Processor -- start 2024-01-16T15:26:53.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.192+0000 7f51efe00640 1 -- start start 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.193+0000 7f51efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.193+0000 7f51efe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51f0160b80 0x7f51f01617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.193+0000 7f51efe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f00b2b90 con 0x7f51f00b3eb0 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.193+0000 7f51efe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00d21d0 con 0x7f51f0160b80 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.193+0000 7f51efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00b2e30 con 0x7f51f01607a0 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51ef400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51f0160b80 0x7f51f01617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51ef400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f51f0160b80 0x7f51f01617d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47702/0 (socket says 172.21.15.53:47702) 2024-01-16T15:26:53.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51ef400640 1 -- 172.21.15.53:0/1495580572 learned_addr learned my addr 172.21.15.53:0/1495580572 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51eea00640 1 --2- 172.21.15.53:0/1495580572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51ef400640 1 -- 172.21.15.53:0/1495580572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 msgr2=0x7f51f00b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51ef400640 1 --2- 172.21.15.53:0/1495580572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.194+0000 7f51ef400640 1 -- 172.21.15.53:0/1495580572 >> v1:172.21.15.146:6789/0 conn(0x7f51f00b3eb0 legacy=0x7f51f01610c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.193 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.195+0000 7f51ef400640 1 -- 172.21.15.53:0/1495580572 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f00b3110 con 0x7f51f0160b80 2024-01-16T15:26:53.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.196+0000 7f51ef400640 1 --2- 172.21.15.53:0/1495580572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f0160b80 0x7f51f01617d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009e50 tx=0x7f51e802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed79ece0098154e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:53.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.196+0000 7f51ed600640 1 -- 172.21.15.53:0/1495580572 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e802cb00 con 0x7f51f0160b80 2024-01-16T15:26:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.196+0000 7f51ed600640 1 -- 172.21.15.53:0/1495580572 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e802cc60 con 0x7f51f0160b80 2024-01-16T15:26:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.196+0000 7f51ed600640 1 -- 172.21.15.53:0/1495580572 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e80343f0 con 0x7f51f0160b80 2024-01-16T15:26:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 -- 172.21.15.53:0/1495580572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f0160b80 msgr2=0x7f51f01617d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 --2- 172.21.15.53:0/1495580572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f0160b80 0x7f51f01617d0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f51e8009e50 tx=0x7f51e802bf60 comp rx=0 tx=0).stop 2024-01-16T15:26:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 -- 172.21.15.53:0/1495580572 shutdown_connections 2024-01-16T15:26:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 --2- 172.21.15.53:0/1495580572 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f0160b80 0x7f51f01617d0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 --2- 172.21.15.53:0/1495580572 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 -- 172.21.15.53:0/1495580572 >> 172.21.15.53:0/1495580572 conn(0x7f51f009c590 msgr2=0x7f51f00d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 -- 172.21.15.53:0/1495580572 shutdown_connections 2024-01-16T15:26:53.196 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.197+0000 7f51efe00640 1 -- 172.21.15.53:0/1495580572 wait complete. 2024-01-16T15:26:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.199+0000 7f51efe00640 1 Processor -- start 2024-01-16T15:26:53.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.199+0000 7f51efe00640 1 -- start start 2024-01-16T15:26:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.200+0000 7f51efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f00b3eb0 0x7f51f015b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.200+0000 7f51efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.200+0000 7f51efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f0160b80 0x7f51f00c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.200+0000 7f51efe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00cc450 con 0x7f51f00b3eb0 2024-01-16T15:26:53.199 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.200+0000 7f51efe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00cb330 con 0x7f51f0160b80 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.200+0000 7f51efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f015bd00 con 0x7f51f01607a0 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f00b3eb0 0x7f51f015b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f00b3eb0 0x7f51f015b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47706/0 (socket says 172.21.15.53:47706) 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 -- 172.21.15.53:0/1375704355 learned_addr learned my addr 172.21.15.53:0/1375704355 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51ee000640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51ef400640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f0160b80 0x7f51f00c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 -- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 msgr2=0x7f51f00b85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f51f01607a0 0x7f51f00b85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.200 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 -- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f0160b80 msgr2=0x7f51f00c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f51f0160b80 0x7f51f00c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.201+0000 7f51eea00640 1 -- 172.21.15.53:0/1375704355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e8009b30 con 0x7f51f00b3eb0 2024-01-16T15:26:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.202+0000 7f51eea00640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f51f00b3eb0 0x7f51f015b7c0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f51e000d800 tx=0x7f51e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:53.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.202+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e000b6c0 con 0x7f51f00b3eb0 2024-01-16T15:26:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.202+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e000b820 con 0x7f51f00b3eb0 2024-01-16T15:26:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.202+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e000bbb0 con 0x7f51f00b3eb0 2024-01-16T15:26:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.202+0000 7f51efe00640 1 -- 172.21.15.53:0/1375704355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f00cb7d0 con 0x7f51f00b3eb0 2024-01-16T15:26:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.202+0000 7f51efe00640 1 -- 172.21.15.53:0/1375704355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f00cbd90 con 0x7f51f00b3eb0 2024-01-16T15:26:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.203+0000 7f51efe00640 1 -- 172.21.15.53:0/1375704355 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f00cc6c0 con 0x7f51f00b3eb0 2024-01-16T15:26:53.202 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.204+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f51e000bd10 con 0x7f51f00b3eb0 2024-01-16T15:26:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.205+0000 7f51e6a00640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51c806c5e0 0x7f51c806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:53.203 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.205+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f51e0089120 con 0x7f51f00b3eb0 2024-01-16T15:26:53.204 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.206+0000 7f51ee000640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51c806c5e0 0x7f51c806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:53.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.207+0000 7f51ee000640 1 --2- 172.21.15.53:0/1375704355 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f51c806c5e0 0x7f51c806eaa0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f51d8000f60 tx=0x7f51d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:53.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.210+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f51e0055640 con 0x7f51f00b3eb0 2024-01-16T15:26:53.433 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.435+0000 7f51efe00640 1 -- 172.21.15.53:0/1375704355 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f51f00b39b0 con 0x7f51c806c5e0 2024-01-16T15:26:53.447 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:53.448+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:53.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:53.449+0000 7f51e6a00640 1 -- 172.21.15.53:0/1375704355 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f51d8001360 con 0x7f51c806c5e0 2024-01-16T15:26:53.447 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:53.448 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:53.464 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:55.465 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:55.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.707+0000 7f61cae00640 1 Processor -- start 2024-01-16T15:26:55.705 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.707+0000 7f61cae00640 1 -- start start 2024-01-16T15:26:55.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61cae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f61c415e170 0x7f61c40cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:55.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61cae00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f61c40cdef0 0x7f61c4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:55.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61cae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61c40700f0 con 0x7f61c413cda0 2024-01-16T15:26:55.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61cae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c40ce720 con 0x7f61c415e170 2024-01-16T15:26:55.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61cae00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61c40cea00 con 0x7f61c40cdef0 2024-01-16T15:26:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61c9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f61c415e170 0x7f61c40cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61c9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f61c415e170 0x7f61c40cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47716/0 (socket says 172.21.15.53:47716) 2024-01-16T15:26:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61c9000640 1 -- 172.21.15.53:0/3032522820 learned_addr learned my addr 172.21.15.53:0/3032522820 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:55.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.709+0000 7f61ca400640 1 --2- 172.21.15.53:0/3032522820 >> v2:172.21.15.146:3300/0 conn(0x7f61c40cdef0 0x7f61c4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:55.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.710+0000 7f61c9000640 1 -- 172.21.15.53:0/3032522820 >> v2:172.21.15.146:3300/0 conn(0x7f61c40cdef0 msgr2=0x7f61c4070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:55.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.710+0000 7f61c9000640 1 --2- 172.21.15.53:0/3032522820 >> v2:172.21.15.146:3300/0 conn(0x7f61c40cdef0 0x7f61c4070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:55.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.710+0000 7f61c9000640 1 -- 172.21.15.53:0/3032522820 >> v1:172.21.15.146:6789/0 conn(0x7f61c413cda0 legacy=0x7f61c415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:55.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.710+0000 7f61c9000640 1 -- 172.21.15.53:0/3032522820 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c4070340 con 0x7f61c415e170 2024-01-16T15:26:55.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.711+0000 7f61c9000640 1 --2- 172.21.15.53:0/3032522820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c415e170 0x7f61c40cd9b0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f61c40aea90 tx=0x7f61ac02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554e7e325345b033 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:55.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.711+0000 7f61c3e00640 1 -- 172.21.15.53:0/3032522820 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac031890 con 0x7f61c415e170 2024-01-16T15:26:55.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.711+0000 7f61c3e00640 1 -- 172.21.15.53:0/3032522820 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61ac0319f0 con 0x7f61c415e170 2024-01-16T15:26:55.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.711+0000 7f61c3e00640 1 -- 172.21.15.53:0/3032522820 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac031d80 con 0x7f61c415e170 2024-01-16T15:26:55.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 -- 172.21.15.53:0/3032522820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c415e170 msgr2=0x7f61c40cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:55.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 --2- 172.21.15.53:0/3032522820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c415e170 0x7f61c40cd9b0 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f61c40aea90 tx=0x7f61ac02b930 comp rx=0 tx=0).stop 2024-01-16T15:26:55.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 -- 172.21.15.53:0/3032522820 shutdown_connections 2024-01-16T15:26:55.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 --2- 172.21.15.53:0/3032522820 >> v2:172.21.15.146:3300/0 conn(0x7f61c40cdef0 0x7f61c4070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:55.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 --2- 172.21.15.53:0/3032522820 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c415e170 0x7f61c40cd9b0 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:55.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 -- 172.21.15.53:0/3032522820 >> 172.21.15.53:0/3032522820 conn(0x7f61c406a3e0 msgr2=0x7f61c406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:55.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 -- 172.21.15.53:0/3032522820 shutdown_connections 2024-01-16T15:26:55.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.712+0000 7f61cae00640 1 -- 172.21.15.53:0/3032522820 wait complete. 2024-01-16T15:26:55.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.714+0000 7f61cae00640 1 Processor -- start 2024-01-16T15:26:55.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.714+0000 7f61cae00640 1 -- start start 2024-01-16T15:26:55.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61cae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c40cdef0 0x7f61c40ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:55.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61cae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61c413cda0 0x7f61c4070cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:55.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61cae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61c415e170 0x7f61c413b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:55.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61cae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4153930 con 0x7f61c40cdef0 2024-01-16T15:26:55.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61cae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c4153040 con 0x7f61c415e170 2024-01-16T15:26:55.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61cae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c41579b0 con 0x7f61c413cda0 2024-01-16T15:26:55.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61c9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c40cdef0 0x7f61c40ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:55.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61c9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c40cdef0 0x7f61c40ade80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47718/0 (socket says 172.21.15.53:47718) 2024-01-16T15:26:55.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.715+0000 7f61c9a00640 1 -- 172.21.15.53:0/387950432 learned_addr learned my addr 172.21.15.53:0/387950432 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:55.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9000640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61c413cda0 0x7f61c4070cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:55.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9a00640 1 -- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61c413cda0 msgr2=0x7f61c4070cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:55.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9a00640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61c413cda0 0x7f61c4070cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:55.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9a00640 1 -- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61c415e170 msgr2=0x7f61c413b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:55.715 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9a00640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f61c415e170 0x7f61c413b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9a00640 1 -- 172.21.15.53:0/387950432 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0094d0 con 0x7f61c40cdef0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9000640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f61c413cda0 0x7f61c4070cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c9a00640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f61c40cdef0 0x7f61c40ade80 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f61b400c400 tx=0x7f61b400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b40052d0 con 0x7f61c40cdef0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61b4005430 con 0x7f61c40cdef0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61cae00640 1 -- 172.21.15.53:0/387950432 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61c41576d0 con 0x7f61c40cdef0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.716+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61b4003830 con 0x7f61c40cdef0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.717+0000 7f61cae00640 1 -- 172.21.15.53:0/387950432 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61c4153550 con 0x7f61c40cdef0 2024-01-16T15:26:55.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.718+0000 7f61cae00640 1 -- 172.21.15.53:0/387950432 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61c4067450 con 0x7f61c40cdef0 2024-01-16T15:26:55.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.719+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f61b4003990 con 0x7f61c40cdef0 2024-01-16T15:26:55.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.720+0000 7f61c1600640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f619c06c5e0 0x7f619c06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:55.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.721+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f61b4088160 con 0x7f61c40cdef0 2024-01-16T15:26:55.719 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.721+0000 7f61c9000640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f619c06c5e0 0x7f619c06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:55.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.722+0000 7f61c9000640 1 --2- 172.21.15.53:0/387950432 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f619c06c5e0 0x7f619c06eaa0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f61ac036000 tx=0x7f61ac008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:55.723 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.726+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f61b4054680 con 0x7f61c40cdef0 2024-01-16T15:26:55.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.950+0000 7f61cae00640 1 -- 172.21.15.53:0/387950432 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61c4070860 con 0x7f619c06c5e0 2024-01-16T15:26:55.961 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:55.963+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:55.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:55.964+0000 7f61c1600640 1 -- 172.21.15.53:0/387950432 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f61c4070860 con 0x7f619c06c5e0 2024-01-16T15:26:55.962 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:55.962 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:57.977 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:26:58.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.218+0000 7f0fa7200640 1 Processor -- start 2024-01-16T15:26:58.217 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.219+0000 7f0fa7200640 1 -- start start 2024-01-16T15:26:58.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.220+0000 7f0fa7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa0160920 0x7f0fa00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.220+0000 7f0fa7200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fa00cf9f0 0x7f0fa00cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.220+0000 7f0fa7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0fa0161010 0x7f0fa01614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.220+0000 7f0fa7200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa00d03b0 con 0x7f0fa00cf9f0 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.220+0000 7f0fa7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fa00d0a10 con 0x7f0fa0161010 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.220+0000 7f0fa7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00cf080 con 0x7f0fa0160920 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fa00cf9f0 0x7f0fa00cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fa00cf9f0 0x7f0fa00cfe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47734/0 (socket says 172.21.15.53:47734) 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 -- 172.21.15.53:0/843566174 learned_addr learned my addr 172.21.15.53:0/843566174 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5e00640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa0160920 0x7f0fa00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa6800640 1 --2- 172.21.15.53:0/843566174 >> v2:172.21.15.146:3300/0 conn(0x7f0fa0161010 0x7f0fa01614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 -- 172.21.15.53:0/843566174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa0160920 msgr2=0x7f0fa00b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa0160920 0x7f0fa00b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 -- 172.21.15.53:0/843566174 >> v2:172.21.15.146:3300/0 conn(0x7f0fa0161010 msgr2=0x7f0fa01614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 --2- 172.21.15.53:0/843566174 >> v2:172.21.15.146:3300/0 conn(0x7f0fa0161010 0x7f0fa01614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5400640 1 -- 172.21.15.53:0/843566174 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa00c6be0 con 0x7f0fa00cf9f0 2024-01-16T15:26:58.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa5e00640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa0160920 0x7f0fa00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:58.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.221+0000 7f0fa6800640 1 --2- 172.21.15.53:0/843566174 >> v2:172.21.15.146:3300/0 conn(0x7f0fa0161010 0x7f0fa01614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:58.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.222+0000 7f0fa5400640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa00cf9f0 0x7f0fa00cfe70 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0fa00c6930 tx=0x7f0f8802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cacfa84a6328258f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.223+0000 7f0fa4a00640 1 -- 172.21.15.53:0/843566174 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031860 con 0x7f0fa00cf9f0 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.223+0000 7f0fa4a00640 1 -- 172.21.15.53:0/843566174 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880319c0 con 0x7f0fa00cf9f0 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.223+0000 7f0fa4a00640 1 -- 172.21.15.53:0/843566174 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031d50 con 0x7f0fa00cf9f0 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 -- 172.21.15.53:0/843566174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa00cf9f0 msgr2=0x7f0fa00cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa00cf9f0 0x7f0fa00cfe70 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f0fa00c6930 tx=0x7f0f8802b900 comp rx=0 tx=0).stop 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 -- 172.21.15.53:0/843566174 shutdown_connections 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 --2- 172.21.15.53:0/843566174 >> v2:172.21.15.146:3300/0 conn(0x7f0fa0161010 0x7f0fa01614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa00cf9f0 0x7f0fa00cfe70 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 --2- 172.21.15.53:0/843566174 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa0160920 0x7f0fa00b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 -- 172.21.15.53:0/843566174 >> 172.21.15.53:0/843566174 conn(0x7f0fa00d3090 msgr2=0x7f0fa00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 -- 172.21.15.53:0/843566174 shutdown_connections 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.224+0000 7f0fa7200640 1 -- 172.21.15.53:0/843566174 wait complete. 2024-01-16T15:26:58.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.226+0000 7f0fa7200640 1 Processor -- start 2024-01-16T15:26:58.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.226+0000 7f0fa7200640 1 -- start start 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa00cf9f0 0x7f0fa0159a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa0160920 0x7f0fa00b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fa0161010 0x7f0fa00b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa0159f90 con 0x7f0fa0160920 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00ca6b0 con 0x7f0fa0161010 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fa00ca990 con 0x7f0fa00cf9f0 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa0160920 0x7f0fa00b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa0160920 0x7f0fa00b77e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47750/0 (socket says 172.21.15.53:47750) 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 -- 172.21.15.53:0/781730497 learned_addr learned my addr 172.21.15.53:0/781730497 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5e00640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa00cf9f0 0x7f0fa0159a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 -- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa00cf9f0 msgr2=0x7f0fa0159a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa00cf9f0 0x7f0fa0159a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 -- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fa0161010 msgr2=0x7f0fa00b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fa0161010 0x7f0fa00b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.228+0000 7f0fa5400640 1 -- 172.21.15.53:0/781730497 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f880094d0 con 0x7f0fa0160920 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0fa5e00640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fa00cf9f0 0x7f0fa0159a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0fa5400640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fa0160920 0x7f0fa00b77e0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f0f88000c00 tx=0x7f0f880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88031830 con 0x7f0fa0160920 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f880443f0 con 0x7f0fa0160920 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f880447a0 con 0x7f0fa0160920 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0fa7200640 1 -- 172.21.15.53:0/781730497 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fa00cac70 con 0x7f0fa0160920 2024-01-16T15:26:58.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.229+0000 7f0fa7200640 1 -- 172.21.15.53:0/781730497 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fa00cb230 con 0x7f0fa0160920 2024-01-16T15:26:58.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.231+0000 7f0fa7200640 1 -- 172.21.15.53:0/781730497 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fa00cb420 con 0x7f0fa0160920 2024-01-16T15:26:58.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.232+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f0f8804d450 con 0x7f0fa0160920 2024-01-16T15:26:58.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.233+0000 7f0f9e000640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f7806c610 0x7f0f7806ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:26:58.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.233+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0f880ab2f0 con 0x7f0fa0160920 2024-01-16T15:26:58.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.234+0000 7f0fa5e00640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f7806c610 0x7f0f7806ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:26:58.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.235+0000 7f0fa5e00640 1 --2- 172.21.15.53:0/781730497 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0f7806c610 0x7f0f7806ead0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f0f90008fc0 tx=0x7f0f90008510 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:26:58.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.238+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fa00cb420 con 0x7f0fa0160920 2024-01-16T15:26:58.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.474+0000 7f0fa7200640 1 -- 172.21.15.53:0/781730497 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0fa015e5f0 con 0x7f0f7806c610 2024-01-16T15:26:58.488 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:26:58.489+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:26:58.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:26:58.490+0000 7f0f9e000640 1 -- 172.21.15.53:0/781730497 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0fa015e5f0 con 0x7f0f7806c610 2024-01-16T15:26:58.489 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:26:58.489 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:26:58.504 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:26:58.980 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:26:58.989 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:00.505 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:00.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.752+0000 7f8093a00640 1 Processor -- start 2024-01-16T15:27:00.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.752+0000 7f8093a00640 1 -- start start 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8093a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f808c15f560 0x7f808c166020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8093a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808c1405a0 con 0x7f808c155900 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8093a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808c13e810 con 0x7f808c06f160 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8093a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f808c13f170 con 0x7f808c15f560 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8091000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f808c155900 0x7f808c06ea50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:35456/0 (socket says 172.21.15.53:35456) 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8091000640 1 -- 172.21.15.53:0/2085738262 learned_addr learned my addr 172.21.15.53:0/2085738262 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:00.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f8091a00640 1 --2- 172.21.15.53:0/2085738262 >> v2:172.21.15.53:3300/0 conn(0x7f808c15f560 0x7f808c166020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:00.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f808b400640 1 -- 172.21.15.53:0/2085738262 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 84946059 0 0) 0x7f808c1405a0 con 0x7f808c155900 2024-01-16T15:27:00.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.754+0000 7f808b400640 1 -- 172.21.15.53:0/2085738262 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f806c003520 con 0x7f808c155900 2024-01-16T15:27:00.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.755+0000 7f8091a00640 1 -- 172.21.15.53:0/2085738262 >> v1:172.21.15.146:6789/0 conn(0x7f808c06f160 legacy=0x7f808c15ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:00.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.755+0000 7f8091a00640 1 -- 172.21.15.53:0/2085738262 >> v1:172.21.15.53:6789/0 conn(0x7f808c155900 legacy=0x7f808c06ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:00.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.755+0000 7f8091a00640 1 -- 172.21.15.53:0/2085738262 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808c13eab0 con 0x7f808c15f560 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.756+0000 7f8091a00640 1 --2- 172.21.15.53:0/2085738262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c15f560 0x7f808c166020 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f8084009d90 tx=0x7f808402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e0508ea4104f3bb server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.756+0000 7f808b400640 1 -- 172.21.15.53:0/2085738262 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808402cc60 con 0x7f808c15f560 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.756+0000 7f808b400640 1 -- 172.21.15.53:0/2085738262 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f808402cdc0 con 0x7f808c15f560 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.756+0000 7f808b400640 1 -- 172.21.15.53:0/2085738262 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8084034660 con 0x7f808c15f560 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 -- 172.21.15.53:0/2085738262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c15f560 msgr2=0x7f808c166020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 --2- 172.21.15.53:0/2085738262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c15f560 0x7f808c166020 secure :-1 s=READY pgs=1920 cs=0 l=1 rev1=1 crypto rx=0x7f8084009d90 tx=0x7f808402beb0 comp rx=0 tx=0).stop 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 -- 172.21.15.53:0/2085738262 shutdown_connections 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 --2- 172.21.15.53:0/2085738262 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c15f560 0x7f808c166020 unknown :-1 s=CLOSED pgs=1920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 -- 172.21.15.53:0/2085738262 >> 172.21.15.53:0/2085738262 conn(0x7f808c06a5c0 msgr2=0x7f808c06a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 -- 172.21.15.53:0/2085738262 shutdown_connections 2024-01-16T15:27:00.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.757+0000 7f8093a00640 1 -- 172.21.15.53:0/2085738262 wait complete. 2024-01-16T15:27:00.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.758+0000 7f8093a00640 1 Processor -- start 2024-01-16T15:27:00.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.759+0000 7f8093a00640 1 -- start start 2024-01-16T15:27:00.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8093a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f808c06f160 0x7f808c1543e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:00.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8093a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c155900 0x7f808c13f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:00.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8093a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c15f560 0x7f808c1f1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:00.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8093a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808c159c90 con 0x7f808c155900 2024-01-16T15:27:00.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8093a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808c154920 con 0x7f808c15f560 2024-01-16T15:27:00.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8093a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808c158d30 con 0x7f808c06f160 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c155900 0x7f808c13f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c155900 0x7f808c13f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47758/0 (socket says 172.21.15.53:47758) 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 -- 172.21.15.53:0/346518156 learned_addr learned my addr 172.21.15.53:0/346518156 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8091000640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f808c06f160 0x7f808c1543e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 -- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f808c06f160 msgr2=0x7f808c1543e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f808c06f160 0x7f808c1543e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 -- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c15f560 msgr2=0x7f808c1f1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f808c15f560 0x7f808c1f1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f808be00640 1 -- 172.21.15.53:0/346518156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8084009a70 con 0x7f808c155900 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.760+0000 7f8091000640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f808c06f160 0x7f808c1543e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.761+0000 7f808be00640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f808c155900 0x7f808c13f590 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f807400c430 tx=0x7f807400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.761+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074011790 con 0x7f808c155900 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.761+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80740118f0 con 0x7f808c155900 2024-01-16T15:27:00.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.761+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8074011ce0 con 0x7f808c155900 2024-01-16T15:27:00.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.761+0000 7f8093a00640 1 -- 172.21.15.53:0/346518156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f808c159010 con 0x7f808c155900 2024-01-16T15:27:00.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.761+0000 7f8093a00640 1 -- 172.21.15.53:0/346518156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f808c1595d0 con 0x7f808c155900 2024-01-16T15:27:00.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.762+0000 7f8093a00640 1 -- 172.21.15.53:0/346518156 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f808c0675d0 con 0x7f808c155900 2024-01-16T15:27:00.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.763+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f807401c460 con 0x7f808c155900 2024-01-16T15:27:00.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.764+0000 7f8088c00640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f806406c5e0 0x7f806406eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:00.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.764+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8074088ba0 con 0x7f808c155900 2024-01-16T15:27:00.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.765+0000 7f8091000640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f806406c5e0 0x7f806406eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:00.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.766+0000 7f8091000640 1 --2- 172.21.15.53:0/346518156 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f806406c5e0 0x7f806406eaa0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f807c0011e0 tx=0x7f807c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:00.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.770+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f80740550c0 con 0x7f808c155900 2024-01-16T15:27:00.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:00.999+0000 7f8093a00640 1 -- 172.21.15.53:0/346518156 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f808c15fb80 con 0x7f806406c5e0 2024-01-16T15:27:01.011 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:01.012+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:01.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:01.013+0000 7f8088c00640 1 -- 172.21.15.53:0/346518156 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f807c0013f0 con 0x7f806406c5e0 2024-01-16T15:27:01.011 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:01.011 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:03.031 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:03.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.277+0000 7f3354c00640 1 Processor -- start 2024-01-16T15:27:03.275 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.277+0000 7f3354c00640 1 -- start start 2024-01-16T15:27:03.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f3354c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f33500b4760 0x7f3350160e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f3354c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335015e510 0x7f33501613c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f3354c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33500b2e40 0x7f33500b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f3354c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33500b3830 con 0x7f33500b2e40 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f3354c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3350161d20 con 0x7f33500b4760 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f3354c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3350070a10 con 0x7f335015e510 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.278+0000 7f334e800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33500b2e40 0x7f33500b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f33500b2e40 0x7f33500b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32902/0 (socket says 172.21.15.53:32902) 2024-01-16T15:27:03.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 -- 172.21.15.53:0/4101256524 learned_addr learned my addr 172.21.15.53:0/4101256524 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334d400640 1 --2- 172.21.15.53:0/4101256524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335015e510 0x7f33501613c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334de00640 1 --2- 172.21.15.53:0/4101256524 >> v2:172.21.15.146:3300/0 conn(0x7f33500b4760 0x7f3350160e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 -- 172.21.15.53:0/4101256524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335015e510 msgr2=0x7f33501613c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 --2- 172.21.15.53:0/4101256524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335015e510 0x7f33501613c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 -- 172.21.15.53:0/4101256524 >> v2:172.21.15.146:3300/0 conn(0x7f33500b4760 msgr2=0x7f3350160e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 --2- 172.21.15.53:0/4101256524 >> v2:172.21.15.146:3300/0 conn(0x7f33500b4760 0x7f3350160e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.278 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.279+0000 7f334e800640 1 -- 172.21.15.53:0/4101256524 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33500b4b40 con 0x7f33500b2e40 2024-01-16T15:27:03.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.280+0000 7f334e800640 1 --2- 172.21.15.53:0/4101256524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33500b2e40 0x7f33500b32f0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f335006f6f0 tx=0x7f334802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6431889ccf3b41c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:03.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.280+0000 7f334ca00640 1 -- 172.21.15.53:0/4101256524 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348031870 con 0x7f33500b2e40 2024-01-16T15:27:03.279 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.280+0000 7f334ca00640 1 -- 172.21.15.53:0/4101256524 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33480319d0 con 0x7f33500b2e40 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f334ca00640 1 -- 172.21.15.53:0/4101256524 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348031d60 con 0x7f33500b2e40 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 -- 172.21.15.53:0/4101256524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33500b2e40 msgr2=0x7f33500b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 --2- 172.21.15.53:0/4101256524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33500b2e40 0x7f33500b32f0 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f335006f6f0 tx=0x7f334802b910 comp rx=0 tx=0).stop 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 -- 172.21.15.53:0/4101256524 shutdown_connections 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 --2- 172.21.15.53:0/4101256524 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f33500b2e40 0x7f33500b32f0 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 --2- 172.21.15.53:0/4101256524 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f335015e510 0x7f33501613c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 --2- 172.21.15.53:0/4101256524 >> v2:172.21.15.146:3300/0 conn(0x7f33500b4760 0x7f3350160e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 -- 172.21.15.53:0/4101256524 >> 172.21.15.53:0/4101256524 conn(0x7f335006a5c0 msgr2=0x7f335006a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 -- 172.21.15.53:0/4101256524 shutdown_connections 2024-01-16T15:27:03.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.281+0000 7f3354c00640 1 -- 172.21.15.53:0/4101256524 wait complete. 2024-01-16T15:27:03.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.283+0000 7f3354c00640 1 Processor -- start 2024-01-16T15:27:03.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.283+0000 7f3354c00640 1 -- start start 2024-01-16T15:27:03.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f3354c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33500b2e40 0x7f33501f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f3354c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33500b4760 0x7f33500ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f3354c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335015e510 0x7f33501506e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f3354c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33500cacd0 con 0x7f335015e510 2024-01-16T15:27:03.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f3354c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33500cafb0 con 0x7f33500b4760 2024-01-16T15:27:03.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f3354c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33500cb290 con 0x7f33500b2e40 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f334e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335015e510 0x7f33501506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f334e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335015e510 0x7f33501506e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32912/0 (socket says 172.21.15.53:32912) 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.284+0000 7f334e800640 1 -- 172.21.15.53:0/1905724653 learned_addr learned my addr 172.21.15.53:0/1905724653 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334de00640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33500b2e40 0x7f33501f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334d400640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33500b4760 0x7f33500ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334e800640 1 -- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33500b2e40 msgr2=0x7f33501f6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334e800640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33500b2e40 0x7f33501f6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334e800640 1 -- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33500b4760 msgr2=0x7f33500ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334e800640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f33500b4760 0x7f33500ca280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334e800640 1 -- 172.21.15.53:0/1905724653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33480094d0 con 0x7f335015e510 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334de00640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f33500b2e40 0x7f33501f6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:03.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.285+0000 7f334e800640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f335015e510 0x7f33501506e0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7f33480090b0 tx=0x7f33480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:03.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.286+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334803abe0 con 0x7f335015e510 2024-01-16T15:27:03.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.286+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334803ad40 con 0x7f335015e510 2024-01-16T15:27:03.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.286+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348044630 con 0x7f335015e510 2024-01-16T15:27:03.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.286+0000 7f3354c00640 1 -- 172.21.15.53:0/1905724653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33500cb570 con 0x7f335015e510 2024-01-16T15:27:03.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.286+0000 7f3354c00640 1 -- 172.21.15.53:0/1905724653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33501f6f30 con 0x7f335015e510 2024-01-16T15:27:03.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.287+0000 7f3354c00640 1 -- 172.21.15.53:0/1905724653 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33500cba90 con 0x7f335015e510 2024-01-16T15:27:03.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.288+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f334804d450 con 0x7f335015e510 2024-01-16T15:27:03.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.289+0000 7f3346000640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f332806c5e0 0x7f332806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:03.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.289+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f334802d040 con 0x7f335015e510 2024-01-16T15:27:03.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.290+0000 7f334de00640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f332806c5e0 0x7f332806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:03.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.291+0000 7f334de00640 1 --2- 172.21.15.53:0/1905724653 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f332806c5e0 0x7f332806eaa0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f3340006fd0 tx=0x7f3340008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:03.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.294+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f33480775b0 con 0x7f335015e510 2024-01-16T15:27:03.523 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.525+0000 7f3354c00640 1 -- 172.21.15.53:0/1905724653 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33500b5680 con 0x7f332806c5e0 2024-01-16T15:27:03.536 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:03.537+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:03.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:03.538+0000 7f3346000640 1 -- 172.21.15.53:0/1905724653 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f33500b5680 con 0x7f332806c5e0 2024-01-16T15:27:03.537 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:03.537 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:03.559 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:05.560 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:05.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.799+0000 7fb5d6200640 1 Processor -- start 2024-01-16T15:27:05.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.799+0000 7fb5d6200640 1 -- start start 2024-01-16T15:27:05.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d6200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb5d0160920 0x7fb5d00b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:05.799 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d00cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:05.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d6200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d00d0a10 con 0x7fb5d0161010 2024-01-16T15:27:05.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d6200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d00d03b0 con 0x7fb5d0160920 2024-01-16T15:27:05.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d00d1f50 con 0x7fb5d00cf9f0 2024-01-16T15:27:05.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d4e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb5d0160920 0x7fb5d00b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:05.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d4e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb5d0160920 0x7fb5d00b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32916/0 (socket says 172.21.15.53:32916) 2024-01-16T15:27:05.800 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.801+0000 7fb5d4e00640 1 -- 172.21.15.53:0/2191019717 learned_addr learned my addr 172.21.15.53:0/2191019717 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.802+0000 7fb5d4e00640 1 -- 172.21.15.53:0/2191019717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 msgr2=0x7fb5d00cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.802+0000 7fb5cfe00640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d00cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.802+0000 7fb5d4e00640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d00cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.802+0000 7fb5d4e00640 1 -- 172.21.15.53:0/2191019717 >> v1:172.21.15.53:6789/0 conn(0x7fb5d0161010 legacy=0x7fb5d01614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.802+0000 7fb5d4e00640 1 -- 172.21.15.53:0/2191019717 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d00cf080 con 0x7fb5d0160920 2024-01-16T15:27:05.801 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.802+0000 7fb5cfe00640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d00cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:05.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.803+0000 7fb5d4e00640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0160920 0x7fb5d00b3f30 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009870 tx=0x7fb5b802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67fa21c30dc8c31 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:05.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.803+0000 7fb5cf400640 1 -- 172.21.15.53:0/2191019717 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8030840 con 0x7fb5d0160920 2024-01-16T15:27:05.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.803+0000 7fb5cf400640 1 -- 172.21.15.53:0/2191019717 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b80309a0 con 0x7fb5d0160920 2024-01-16T15:27:05.802 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5cf400640 1 -- 172.21.15.53:0/2191019717 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8030d30 con 0x7fb5d0160920 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 -- 172.21.15.53:0/2191019717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0160920 msgr2=0x7fb5d00b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0160920 0x7fb5d00b3f30 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8009870 tx=0x7fb5b802b980 comp rx=0 tx=0).stop 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 -- 172.21.15.53:0/2191019717 shutdown_connections 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d00cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 --2- 172.21.15.53:0/2191019717 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0160920 0x7fb5d00b3f30 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 -- 172.21.15.53:0/2191019717 >> 172.21.15.53:0/2191019717 conn(0x7fb5d00d3090 msgr2=0x7fb5d00d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 -- 172.21.15.53:0/2191019717 shutdown_connections 2024-01-16T15:27:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.804+0000 7fb5d6200640 1 -- 172.21.15.53:0/2191019717 wait complete. 2024-01-16T15:27:05.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.806+0000 7fb5d6200640 1 Processor -- start 2024-01-16T15:27:05.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.806+0000 7fb5d6200640 1 -- start start 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.807+0000 7fb5d6200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d015b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.807+0000 7fb5d6200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d0160920 0x7fb5d006bf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.807+0000 7fb5d6200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0161010 0x7fb5d006c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.807+0000 7fb5d6200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d015bcf0 con 0x7fb5d0161010 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.807+0000 7fb5d6200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d015c5e0 con 0x7fb5d0160920 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.807+0000 7fb5d6200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d00b88a0 con 0x7fb5d00cf9f0 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0161010 0x7fb5d006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d4e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d015b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0161010 0x7fb5d006c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32930/0 (socket says 172.21.15.53:32930) 2024-01-16T15:27:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 -- 172.21.15.53:0/1381695915 learned_addr learned my addr 172.21.15.53:0/1381695915 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5cfe00640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d0160920 0x7fb5d006bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 -- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 msgr2=0x7fb5d015b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5d00cf9f0 0x7fb5d015b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 -- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d0160920 msgr2=0x7fb5d006bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.808+0000 7fb5d5800640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5d0160920 0x7fb5d006bf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.809+0000 7fb5d5800640 1 -- 172.21.15.53:0/1381695915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b80094d0 con 0x7fb5d0161010 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.809+0000 7fb5d5800640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5d0161010 0x7fb5d006c440 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fb5c800c3c0 tx=0x7fb5c800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.809+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c80052d0 con 0x7fb5d0161010 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.809+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c8005430 con 0x7fb5d0161010 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.809+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8003810 con 0x7fb5d0161010 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.809+0000 7fb5d6200640 1 -- 172.21.15.53:0/1381695915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d015c830 con 0x7fb5d0161010 2024-01-16T15:27:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.810+0000 7fb5d6200640 1 -- 172.21.15.53:0/1381695915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d015cdf0 con 0x7fb5d0161010 2024-01-16T15:27:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.811+0000 7fb5d6200640 1 -- 172.21.15.53:0/1381695915 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d0067450 con 0x7fb5d0161010 2024-01-16T15:27:05.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.812+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7fb5c8003970 con 0x7fb5d0161010 2024-01-16T15:27:05.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.813+0000 7fb5ccc00640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a806c5e0 0x7fb5a806eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:05.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.813+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb5c8087fc0 con 0x7fb5d0161010 2024-01-16T15:27:05.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.814+0000 7fb5d4e00640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a806c5e0 0x7fb5a806eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:05.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.815+0000 7fb5d4e00640 1 --2- 172.21.15.53:0/1381695915 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a806c5e0 0x7fb5a806eaa0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb5b8036000 tx=0x7fb5b8008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:05.818+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5c80545f0 con 0x7fb5d0161010 2024-01-16T15:27:06.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:06.047+0000 7fb5d6200640 1 -- 172.21.15.53:0/1381695915 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5d00ceac0 con 0x7fb5a806c5e0 2024-01-16T15:27:06.059 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:06.060+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:06.061+0000 7fb5ccc00640 1 -- 172.21.15.53:0/1381695915 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5b8030ce0 con 0x7fb5a806c5e0 2024-01-16T15:27:06.060 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:06.060 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:06.073 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:08.073 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:08.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.314+0000 7feeb7200640 1 Processor -- start 2024-01-16T15:27:08.312 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.314+0000 7feeb7200640 1 -- start start 2024-01-16T15:27:08.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.316+0000 7feeb7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 0x7feeb80cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.316+0000 7feeb7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb806ca10 con 0x7feeb806dab0 2024-01-16T15:27:08.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.316+0000 7feeb7200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeb806ccf0 con 0x7feeb81447f0 2024-01-16T15:27:08.314 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.316+0000 7feeb7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb806d690 con 0x7feeb806c470 2024-01-16T15:27:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.316+0000 7feeb6800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7feeb806dab0 0x7feeb815c7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55048/0 (socket says 172.21.15.53:55048) 2024-01-16T15:27:08.315 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.316+0000 7feeb6800640 1 -- 172.21.15.53:0/2589103256 learned_addr learned my addr 172.21.15.53:0/2589103256 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeafe00640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 0x7feeb80cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528590771 0 0) 0x7feeb806ca10 con 0x7feeb806dab0 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98003520 con 0x7feeb806dab0 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1503982715 0 0) 0x7fee98003520 con 0x7feeb806dab0 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 msgr2=0x7feeb80cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 0x7feeb80cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 >> v1:172.21.15.146:6789/0 conn(0x7feeb81447f0 legacy=0x7feeb815c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb806cfd0 con 0x7feeb806dab0 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7feeb00019f0 con 0x7feeb806dab0 2024-01-16T15:27:08.316 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 >> v1:172.21.15.53:6789/0 conn(0x7feeb806dab0 legacy=0x7feeb815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee980039a0 0x7fee98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 0x7fee980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.317+0000 7feeb5400640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7fee98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb806d690 con 0x7fee98004d00 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9800bb40 con 0x7fee980039a0 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9800bd00 con 0x7feeb806c470 2024-01-16T15:27:08.317 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb6800640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7fee98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeafe00640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 0x7fee980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb5e00640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee980039a0 0x7fee98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb6800640 1 -- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 msgr2=0x7fee980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb6800640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb806c470 0x7fee980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb6800640 1 -- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee980039a0 msgr2=0x7fee98003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb6800640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fee980039a0 0x7fee98003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.318+0000 7feeb6800640 1 -- 172.21.15.53:0/2589103256 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee98003520 con 0x7fee98004d00 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.320+0000 7feeb6800640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7fee98005180 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7feeb000a760 tx=0x7fee980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2f6ddcf6f19b3c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.320+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0034630 con 0x7fee98004d00 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.320+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feeb002f070 con 0x7fee98004d00 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.320+0000 7feeb5400640 1 -- 172.21.15.53:0/2589103256 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0034a10 con 0x7fee98004d00 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb7200640 1 -- 172.21.15.53:0/2589103256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 msgr2=0x7fee98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb7200640 1 --2- 172.21.15.53:0/2589103256 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7fee98005180 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7feeb000a760 tx=0x7fee980057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb6800640 1 -- 172.21.15.53:0/2589103256 reap_dead start 2024-01-16T15:27:08.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb7200640 1 -- 172.21.15.53:0/2589103256 shutdown_connections 2024-01-16T15:27:08.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb7200640 1 -- 172.21.15.53:0/2589103256 >> 172.21.15.53:0/2589103256 conn(0x7feeb806a3e0 msgr2=0x7feeb806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:08.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb7200640 1 -- 172.21.15.53:0/2589103256 shutdown_connections 2024-01-16T15:27:08.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.321+0000 7feeb7200640 1 -- 172.21.15.53:0/2589103256 wait complete. 2024-01-16T15:27:08.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.323+0000 7feeb7200640 1 Processor -- start 2024-01-16T15:27:08.321 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.323+0000 7feeb7200640 1 -- start start 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.324+0000 7feeb7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7feeb80acb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.324+0000 7feeb7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80ad080 0x7feeb814e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.324+0000 7feeb7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb814eca0 0x7feeb813b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.324+0000 7feeb7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80ae240 con 0x7fee98004d00 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.324+0000 7feeb7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb814f130 con 0x7feeb80ad080 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.324+0000 7feeb7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeb80ad5c0 con 0x7feeb814eca0 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7feeb80acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7feeb80acb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32950/0 (socket says 172.21.15.53:32950) 2024-01-16T15:27:08.323 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 -- 172.21.15.53:0/3461764611 learned_addr learned my addr 172.21.15.53:0/3461764611 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb6800640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb814eca0 0x7feeb813b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 -- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb814eca0 msgr2=0x7feeb813b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb814eca0 0x7feeb813b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.324 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 -- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80ad080 msgr2=0x7feeb814e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feeb80ad080 0x7feeb814e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 -- 172.21.15.53:0/3461764611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeb80ad8a0 con 0x7fee98004d00 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb6800640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feeb814eca0 0x7feeb813b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.325+0000 7feeb5e00640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fee98004d00 0x7feeb80acb40 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7feea800e580 tx=0x7feea800ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.326+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80052e0 con 0x7fee98004d00 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.326+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea8005440 con 0x7fee98004d00 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.326+0000 7feeb7200640 1 -- 172.21.15.53:0/3461764611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb80adb80 con 0x7fee98004d00 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.326+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8003810 con 0x7fee98004d00 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.326+0000 7feeb7200640 1 -- 172.21.15.53:0/3461764611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb8159e90 con 0x7fee98004d00 2024-01-16T15:27:08.325 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.327+0000 7feeb7200640 1 -- 172.21.15.53:0/3461764611 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb8067450 con 0x7fee98004d00 2024-01-16T15:27:08.326 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.328+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7feea8003970 con 0x7fee98004d00 2024-01-16T15:27:08.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.329+0000 7feeae000640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fee9006c610 0x7fee9006ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:08.327 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.329+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feea8088990 con 0x7fee98004d00 2024-01-16T15:27:08.328 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.330+0000 7feeafe00640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fee9006c610 0x7fee9006ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:08.329 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.331+0000 7feeafe00640 1 --2- 172.21.15.53:0/3461764611 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fee9006c610 0x7fee9006ead0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7feeb813d840 tx=0x7feea0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:08.337 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.339+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feea8054e30 con 0x7fee98004d00 2024-01-16T15:27:08.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.570+0000 7feeb7200640 1 -- 172.21.15.53:0/3461764611 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feeb8144bd0 con 0x7fee9006c610 2024-01-16T15:27:08.581 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:08.583+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:08.582 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:08.584+0000 7feeae000640 1 -- 172.21.15.53:0/3461764611 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7feea0011860 con 0x7fee9006c610 2024-01-16T15:27:08.582 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:08.582 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:08.596 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:10.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:10.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.841+0000 7f1c18c00640 1 Processor -- start 2024-01-16T15:27:10.840 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.842+0000 7f1c18c00640 1 -- start start 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.843+0000 7f1c18c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c14144410 0x7f1c1406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.843+0000 7f1c18c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 0x7f1c1406a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.843+0000 7f1c18c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c140acda0 con 0x7f1c14144870 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.843+0000 7f1c18c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140af740 con 0x7f1c14144410 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.843+0000 7f1c18c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140ad5b0 con 0x7f1c140b2600 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.844+0000 7f1c13e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1c14144870 0x7f1c14071160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55052/0 (socket says 172.21.15.53:55052) 2024-01-16T15:27:10.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.844+0000 7f1c13e00640 1 -- 172.21.15.53:0/1926198166 learned_addr learned my addr 172.21.15.53:0/1926198166 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:10.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.844+0000 7f1c13400640 1 --2- 172.21.15.53:0/1926198166 >> v2:172.21.15.146:3300/0 conn(0x7f1c14144410 0x7f1c1406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.844+0000 7f1c12a00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 0x7f1c1406a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.844+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209714119 0 0) 0x7f1c140acda0 con 0x7f1c14144870 2024-01-16T15:27:10.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.844+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf4003520 con 0x7f1c14144870 2024-01-16T15:27:10.844 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1381261506 0 0) 0x7f1bf4003520 con 0x7f1c14144870 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 msgr2=0x7f1c1406a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 0x7f1c1406a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 >> v2:172.21.15.146:3300/0 conn(0x7f1c14144410 msgr2=0x7f1c1406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 --2- 172.21.15.53:0/1926198166 >> v2:172.21.15.146:3300/0 conn(0x7f1c14144410 0x7f1c1406a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140b3240 con 0x7f1c14144870 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f1c0c0019f0 con 0x7f1c14144870 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 >> v1:172.21.15.53:6789/0 conn(0x7f1c14144870 legacy=0x7f1c14071160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 0x7f1bf4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4003f40 0x7f1bf40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bf4004d00 0x7f1bf4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140af740 con 0x7f1bf4004d00 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140ad5b0 con 0x7f1bf4003f40 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf400b980 con 0x7f1c140b2600 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.845+0000 7f1c13e00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bf4004d00 0x7f1bf4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c13400640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 0x7f1bf4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.845 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c12a00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4003f40 0x7f1bf40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c13e00640 1 -- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 msgr2=0x7f1bf4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:10.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c13e00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c140b2600 0x7f1bf4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c13e00640 1 -- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4003f40 msgr2=0x7f1bf40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:10.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c13e00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4003f40 0x7f1bf40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c13e00640 1 -- 172.21.15.53:0/1926198166 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4003520 con 0x7f1bf4004d00 2024-01-16T15:27:10.846 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.846+0000 7f1c12a00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1bf4003f40 0x7f1bf40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.847+0000 7f1c13e00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bf4004d00 0x7f1bf4005180 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00a7c0 tx=0x7f1bf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8558017c943577cf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.847+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c034630 con 0x7f1bf4004d00 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.847+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c02f070 con 0x7f1bf4004d00 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.847+0000 7f1c12000640 1 -- 172.21.15.53:0/1926198166 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c034a30 con 0x7f1bf4004d00 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c18c00640 1 -- 172.21.15.53:0/1926198166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bf4004d00 msgr2=0x7f1bf4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c18c00640 1 --2- 172.21.15.53:0/1926198166 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1bf4004d00 0x7f1bf4005180 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c00a7c0 tx=0x7f1bf40057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c13e00640 1 -- 172.21.15.53:0/1926198166 reap_dead start 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c18c00640 1 -- 172.21.15.53:0/1926198166 shutdown_connections 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c18c00640 1 -- 172.21.15.53:0/1926198166 >> 172.21.15.53:0/1926198166 conn(0x7f1c140b19c0 msgr2=0x7f1c140b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c18c00640 1 -- 172.21.15.53:0/1926198166 shutdown_connections 2024-01-16T15:27:10.847 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.848+0000 7f1c18c00640 1 -- 172.21.15.53:0/1926198166 wait complete. 2024-01-16T15:27:10.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.850+0000 7f1c18c00640 1 Processor -- start 2024-01-16T15:27:10.848 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.850+0000 7f1c18c00640 1 -- start start 2024-01-16T15:27:10.849 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c18c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf4004d00 0x7f1c141395f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c18c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c14155630 0x7f1c14157bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c18c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c14158110 0x7f1c14071820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c18c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c141597d0 con 0x7f1c14158110 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c18c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14155ac0 con 0x7f1c14155630 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c18c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14155da0 con 0x7f1bf4004d00 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c14158110 0x7f1c14071820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c14158110 0x7f1c14071820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:32962/0 (socket says 172.21.15.53:32962) 2024-01-16T15:27:10.850 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.851+0000 7f1c13e00640 1 -- 172.21.15.53:0/2739886076 learned_addr learned my addr 172.21.15.53:0/2739886076 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:10.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13400640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf4004d00 0x7f1c141395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c12a00640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c14155630 0x7f1c14157bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13e00640 1 -- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf4004d00 msgr2=0x7f1c141395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:10.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13e00640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf4004d00 0x7f1c141395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.851 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13e00640 1 -- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c14155630 msgr2=0x7f1c14157bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13e00640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c14155630 0x7f1c14157bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13e00640 1 -- 172.21.15.53:0/2739886076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c14156080 con 0x7f1c14158110 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13400640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1bf4004d00 0x7f1c141395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c13e00640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c14158110 0x7f1c14071820 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f1c0c02cad0 tx=0x7f1c0c001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c001f00 con 0x7f1c14158110 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0c034430 con 0x7f1c14158110 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c18c00640 1 -- 172.21.15.53:0/2739886076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c14156360 con 0x7f1c14158110 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.852+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c0c0347a0 con 0x7f1c14158110 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.853+0000 7f1c18c00640 1 -- 172.21.15.53:0/2739886076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c14071d60 con 0x7f1c14158110 2024-01-16T15:27:10.852 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.854+0000 7f1c18c00640 1 -- 172.21.15.53:0/2739886076 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c14159920 con 0x7f1c14158110 2024-01-16T15:27:10.853 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.855+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 51) v1 ==== 91273+0+0 (secure 0 0 0) 0x7f1c0c031030 con 0x7f1c14158110 2024-01-16T15:27:10.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.856+0000 7f1c0b400640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bec06c5e0 0x7f1bec06eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:10.854 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.856+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1c0c0aa1d0 con 0x7f1c14158110 2024-01-16T15:27:10.855 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.857+0000 7f1c13400640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bec06c5e0 0x7f1bec06eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:10.856 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.858+0000 7f1c13400640 1 --2- 172.21.15.53:0/2739886076 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1bec06c5e0 0x7f1bec06eaa0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009a80 tx=0x7f1c04008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:10.859 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:10.861+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c14159920 con 0x7f1c14158110 2024-01-16T15:27:11.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:11.089+0000 7f1c18c00640 1 -- 172.21.15.53:0/2739886076 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1c140b2ed0 con 0x7f1bec06c5e0 2024-01-16T15:27:11.101 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:11.102+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:11.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:11.104+0000 7f1c0b400640 1 -- 172.21.15.53:0/2739886076 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1c04012a10 con 0x7f1bec06c5e0 2024-01-16T15:27:11.102 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:11.102 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:11.117 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:13.118 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:13.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.359+0000 7f29bee00640 1 Processor -- start 2024-01-16T15:27:13.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.359+0000 7f29bee00640 1 -- start start 2024-01-16T15:27:13.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bee00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f29b81447f0 0x7f29b815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bee00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f29b806c470 0x7f29b80cc5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b80ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bee00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b806d690 con 0x7f29b81447f0 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bee00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29b80cebb0 con 0x7f29b806c470 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b815c5c0 con 0x7f29b806dab0 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.361+0000 7f29bda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f29b81447f0 0x7f29b815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f29b81447f0 0x7f29b815c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46336/0 (socket says 172.21.15.53:46336) 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 -- 172.21.15.53:0/1591991143 learned_addr learned my addr 172.21.15.53:0/1591991143 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bd000640 1 --2- 172.21.15.53:0/1591991143 >> v2:172.21.15.146:3300/0 conn(0x7f29b806c470 0x7f29b80cc5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 -- 172.21.15.53:0/1591991143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 msgr2=0x7f29b80ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29be400640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b80ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b80ccb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 -- 172.21.15.53:0/1591991143 >> v2:172.21.15.146:3300/0 conn(0x7f29b806c470 msgr2=0x7f29b80cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 --2- 172.21.15.53:0/1591991143 >> v2:172.21.15.146:3300/0 conn(0x7f29b806c470 0x7f29b80cc5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.361 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bda00640 1 -- 172.21.15.53:0/1591991143 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b806ca10 con 0x7f29b81447f0 2024-01-16T15:27:13.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29bd000640 1 --2- 172.21.15.53:0/1591991143 >> v2:172.21.15.146:3300/0 conn(0x7f29b806c470 0x7f29b80cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:13.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.362+0000 7f29be400640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b80ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.363+0000 7f29bda00640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b81447f0 0x7f29b815c080 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f29b806c900 tx=0x7f29a802b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63b51d40808f1fd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.363+0000 7f29b7e00640 1 -- 172.21.15.53:0/1591991143 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8031750 con 0x7f29b81447f0 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29b7e00640 1 -- 172.21.15.53:0/1591991143 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a80318b0 con 0x7f29b81447f0 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29b7e00640 1 -- 172.21.15.53:0/1591991143 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8031c40 con 0x7f29b81447f0 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 -- 172.21.15.53:0/1591991143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b81447f0 msgr2=0x7f29b815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b81447f0 0x7f29b815c080 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7f29b806c900 tx=0x7f29a802b7f0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 -- 172.21.15.53:0/1591991143 shutdown_connections 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b80ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 --2- 172.21.15.53:0/1591991143 >> v2:172.21.15.146:3300/0 conn(0x7f29b806c470 0x7f29b80cc5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 --2- 172.21.15.53:0/1591991143 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b81447f0 0x7f29b815c080 unknown :-1 s=CLOSED pgs=1933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 -- 172.21.15.53:0/1591991143 >> 172.21.15.53:0/1591991143 conn(0x7f29b806a3e0 msgr2=0x7f29b806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 -- 172.21.15.53:0/1591991143 shutdown_connections 2024-01-16T15:27:13.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.364+0000 7f29bee00640 1 -- 172.21.15.53:0/1591991143 wait complete. 2024-01-16T15:27:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.367+0000 7f29bee00640 1 Processor -- start 2024-01-16T15:27:13.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.367+0000 7f29bee00640 1 -- start start 2024-01-16T15:27:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bee00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b806c470 0x7f29b80adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bee00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b8140340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bee00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29b81447f0 0x7f29b80b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bee00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b814c560 con 0x7f29b806c470 2024-01-16T15:27:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bee00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b814c800 con 0x7f29b81447f0 2024-01-16T15:27:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bee00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29b814cae0 con 0x7f29b806dab0 2024-01-16T15:27:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b806c470 0x7f29b80adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bda00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b806c470 0x7f29b80adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46350/0 (socket says 172.21.15.53:46350) 2024-01-16T15:27:13.367 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bda00640 1 -- 172.21.15.53:0/1098182292 learned_addr learned my addr 172.21.15.53:0/1098182292 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.368+0000 7f29bd000640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b8140340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29be400640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29b81447f0 0x7f29b80b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bda00640 1 -- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 msgr2=0x7f29b8140340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bda00640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b8140340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bda00640 1 -- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29b81447f0 msgr2=0x7f29b80b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bda00640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f29b81447f0 0x7f29b80b19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:13.368 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bda00640 1 -- 172.21.15.53:0/1098182292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a80094d0 con 0x7f29b806c470 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bd000640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f29b806dab0 0x7f29b8140340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bda00640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f29b806c470 0x7f29b80adc70 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7f29a8000c00 tx=0x7f29a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a8033c80 con 0x7f29b806c470 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29a8033de0 con 0x7f29b806c470 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29a803c640 con 0x7f29b806c470 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.369+0000 7f29bee00640 1 -- 172.21.15.53:0/1098182292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b814cdc0 con 0x7f29b806c470 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.370+0000 7f29bee00640 1 -- 172.21.15.53:0/1098182292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b80ae1b0 con 0x7f29b806c470 2024-01-16T15:27:13.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.371+0000 7f29bee00640 1 -- 172.21.15.53:0/1098182292 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b80ae340 con 0x7f29b806c470 2024-01-16T15:27:13.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.372+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f29a803a040 con 0x7f29b806c470 2024-01-16T15:27:13.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.373+0000 7f29b5600640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f299006c550 0x7f299006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:13.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.373+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f29a80aa940 con 0x7f29b806c470 2024-01-16T15:27:13.372 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.374+0000 7f29bd000640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f299006c550 0x7f299006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:13.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.375+0000 7f29bd000640 1 --2- 172.21.15.53:0/1098182292 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f299006c550 0x7f299006ea10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f29b80cc060 tx=0x7f29a0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:13.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.378+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29a8076fa0 con 0x7f29b806c470 2024-01-16T15:27:13.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.612+0000 7f29bee00640 1 -- 172.21.15.53:0/1098182292 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29b806d2b0 con 0x7f299006c550 2024-01-16T15:27:13.624 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:13.625+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:13.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:13.626+0000 7f29b5600640 1 -- 172.21.15.53:0/1098182292 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f29a000c040 con 0x7f299006c550 2024-01-16T15:27:13.624 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:13.625 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:13.635 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:15.636 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:15.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.876+0000 7f7a7fe00640 1 Processor -- start 2024-01-16T15:27:15.875 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.876+0000 7f7a7fe00640 1 -- start start 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f7a801447f0 0x7f7a8015c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a800cccf0 0x7f7a8015c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7fe00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8015c5c0 con 0x7f7a8006c470 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a8006d690 con 0x7f7a801447f0 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a800cebb0 con 0x7f7a800cccf0 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7e000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f7a8006c470 0x7f7a800cc5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55454/0 (socket says 172.21.15.53:55454) 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7e000640 1 -- 172.21.15.53:0/3233244078 learned_addr learned my addr 172.21.15.53:0/3233244078 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:15.877 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.878+0000 7f7a7ea00640 1 --2- 172.21.15.53:0/3233244078 >> v2:172.21.15.53:3300/0 conn(0x7f7a801447f0 0x7f7a8015c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.879+0000 7f7a7f400640 1 --2- 172.21.15.53:0/3233244078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a800cccf0 0x7f7a8015c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.879+0000 7f7a7ea00640 1 -- 172.21.15.53:0/3233244078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a800cccf0 msgr2=0x7f7a8015c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:15.878 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.879+0000 7f7a7ea00640 1 --2- 172.21.15.53:0/3233244078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a800cccf0 0x7f7a8015c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.879+0000 7f7a7ea00640 1 -- 172.21.15.53:0/3233244078 >> v1:172.21.15.53:6789/0 conn(0x7f7a8006c470 legacy=0x7f7a800cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.879+0000 7f7a7ea00640 1 -- 172.21.15.53:0/3233244078 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8006ca10 con 0x7f7a801447f0 2024-01-16T15:27:15.879 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.880+0000 7f7a7ea00640 1 --2- 172.21.15.53:0/3233244078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a801447f0 0x7f7a8015c080 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f7a800cecf0 tx=0x7f7a7002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8937a1c8a1e0b5c0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7d600640 1 -- 172.21.15.53:0/3233244078 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70031780 con 0x7f7a801447f0 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7d600640 1 -- 172.21.15.53:0/3233244078 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a700318e0 con 0x7f7a801447f0 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7d600640 1 -- 172.21.15.53:0/3233244078 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70031c70 con 0x7f7a801447f0 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 -- 172.21.15.53:0/3233244078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a801447f0 msgr2=0x7f7a8015c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 --2- 172.21.15.53:0/3233244078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a801447f0 0x7f7a8015c080 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f7a800cecf0 tx=0x7f7a7002b820 comp rx=0 tx=0).stop 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 -- 172.21.15.53:0/3233244078 shutdown_connections 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 --2- 172.21.15.53:0/3233244078 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a800cccf0 0x7f7a8015c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 --2- 172.21.15.53:0/3233244078 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a801447f0 0x7f7a8015c080 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 -- 172.21.15.53:0/3233244078 >> 172.21.15.53:0/3233244078 conn(0x7f7a8006a3e0 msgr2=0x7f7a8006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:15.880 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.881+0000 7f7a7fe00640 1 -- 172.21.15.53:0/3233244078 shutdown_connections 2024-01-16T15:27:15.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.882+0000 7f7a7fe00640 1 -- 172.21.15.53:0/3233244078 wait complete. 2024-01-16T15:27:15.881 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.883+0000 7f7a7fe00640 1 Processor -- start 2024-01-16T15:27:15.882 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.883+0000 7f7a7fe00640 1 -- start start 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.884+0000 7f7a7fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a8006c470 0x7f7a801f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.884+0000 7f7a7fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a800cccf0 0x7f7a8013f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.884+0000 7f7a7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a801447f0 0x7f7a8013c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.884+0000 7f7a7fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8013b560 con 0x7f7a8006c470 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.884+0000 7f7a7fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8013b840 con 0x7f7a800cccf0 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.884+0000 7f7a7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a8013bb20 con 0x7f7a801447f0 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a8006c470 0x7f7a801f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a8006c470 0x7f7a801f0ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46368/0 (socket says 172.21.15.53:46368) 2024-01-16T15:27:15.883 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 -- 172.21.15.53:0/2727200841 learned_addr learned my addr 172.21.15.53:0/2727200841 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7f400640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a801447f0 0x7f7a8013c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7e000640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a800cccf0 0x7f7a8013f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 -- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a801447f0 msgr2=0x7f7a8013c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7a801447f0 0x7f7a8013c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 -- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a800cccf0 msgr2=0x7f7a8013f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a800cccf0 0x7f7a8013f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 -- 172.21.15.53:0/2727200841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a70009400 con 0x7f7a8006c470 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7e000640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7a800cccf0 0x7f7a8013f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:15.885 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.885+0000 7f7a7ea00640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7a8006c470 0x7f7a801f0ed0 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f7a70000c00 tx=0x7f7a700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.886+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a700328e0 con 0x7f7a8006c470 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.886+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a70032a60 con 0x7f7a8006c470 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.886+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a70032e10 con 0x7f7a8006c470 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.886+0000 7f7a7fe00640 1 -- 172.21.15.53:0/2727200841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8013be00 con 0x7f7a8006c470 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.886+0000 7f7a7fe00640 1 -- 172.21.15.53:0/2727200841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a801f1410 con 0x7f7a8006c470 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.887+0000 7f7a7fe00640 1 -- 172.21.15.53:0/2727200841 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a8013c320 con 0x7f7a8006c470 2024-01-16T15:27:15.886 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.888+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f7a7004e450 con 0x7f7a8006c470 2024-01-16T15:27:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.889+0000 7f7a76a00640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a5806c550 0x7f7a5806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.889+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f7a700aa910 con 0x7f7a8006c470 2024-01-16T15:27:15.888 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.890+0000 7f7a7e000640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a5806c550 0x7f7a5806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:15.889 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.891+0000 7f7a7e000640 1 --2- 172.21.15.53:0/2727200841 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f7a5806c550 0x7f7a5806ea10 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7a80143870 tx=0x7f7a68009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:15.893 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:15.894+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f7a700772f0 con 0x7f7a8006c470 2024-01-16T15:27:16.122 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:16.123+0000 7f7a7fe00640 1 -- 172.21.15.53:0/2727200841 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7a8006d2b0 con 0x7f7a5806c550 2024-01-16T15:27:16.136 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:16.137+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:16.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:16.138+0000 7f7a76a00640 1 -- 172.21.15.53:0/2727200841 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f7a68001ce0 con 0x7f7a5806c550 2024-01-16T15:27:16.136 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:16.137 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:18.147 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:18.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.386+0000 7f5ad6c00640 1 Processor -- start 2024-01-16T15:27:18.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.386+0000 7f5ad6c00640 1 -- start start 2024-01-16T15:27:18.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.388+0000 7f5ad6c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5ad00b3ae0 0x7f5ad015e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.388+0000 7f5ad6c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad0070c40 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.388+0000 7f5ad6c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad006c470 con 0x7f5ad00cf290 2024-01-16T15:27:18.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.388+0000 7f5ad6c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad0071450 con 0x7f5ad00b3ae0 2024-01-16T15:27:18.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5acfe00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5ad00ae5a0 0x7f5ad006cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:55466/0 (socket says 172.21.15.53:55466) 2024-01-16T15:27:18.387 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5acfe00640 1 -- 172.21.15.53:0/3136392317 learned_addr learned my addr 172.21.15.53:0/3136392317 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:18.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5ac7e00640 1 --2- 172.21.15.53:0/3136392317 >> v2:172.21.15.53:3300/0 conn(0x7f5ad00b3ae0 0x7f5ad015e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5acf400640 1 -- 172.21.15.53:0/3136392317 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160331597 0 0) 0x7f5ad0070c40 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5acf400640 1 -- 172.21.15.53:0/3136392317 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab0003520 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5ac7e00640 1 -- 172.21.15.53:0/3136392317 >> v1:172.21.15.146:6789/0 conn(0x7f5ad00cf290 legacy=0x7f5ad00b1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5ac7e00640 1 -- 172.21.15.53:0/3136392317 >> v1:172.21.15.53:6789/0 conn(0x7f5ad00ae5a0 legacy=0x7f5ad006cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.389+0000 7f5ac7e00640 1 -- 172.21.15.53:0/3136392317 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad006c750 con 0x7f5ad00b3ae0 2024-01-16T15:27:18.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.391+0000 7f5ac7e00640 1 --2- 172.21.15.53:0/3136392317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00b3ae0 0x7f5ad015e430 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80097a0 tx=0x7f5ab802b9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17534dd45f163bf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:18.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.391+0000 7f5acf400640 1 -- 172.21.15.53:0/3136392317 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab802f810 con 0x7f5ad00b3ae0 2024-01-16T15:27:18.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.391+0000 7f5acf400640 1 -- 172.21.15.53:0/3136392317 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab802f970 con 0x7f5ad00b3ae0 2024-01-16T15:27:18.389 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.391+0000 7f5acf400640 1 -- 172.21.15.53:0/3136392317 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab802fd00 con 0x7f5ad00b3ae0 2024-01-16T15:27:18.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.391+0000 7f5ad6c00640 1 -- 172.21.15.53:0/3136392317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00b3ae0 msgr2=0x7f5ad015e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.391+0000 7f5ad6c00640 1 --2- 172.21.15.53:0/3136392317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00b3ae0 0x7f5ad015e430 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80097a0 tx=0x7f5ab802b9f0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.392+0000 7f5ad6c00640 1 -- 172.21.15.53:0/3136392317 shutdown_connections 2024-01-16T15:27:18.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.392+0000 7f5ad6c00640 1 --2- 172.21.15.53:0/3136392317 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00b3ae0 0x7f5ad015e430 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.392+0000 7f5ad6c00640 1 -- 172.21.15.53:0/3136392317 >> 172.21.15.53:0/3136392317 conn(0x7f5ad006a800 msgr2=0x7f5ad006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:18.390 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.392+0000 7f5ad6c00640 1 -- 172.21.15.53:0/3136392317 shutdown_connections 2024-01-16T15:27:18.391 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.392+0000 7f5ad6c00640 1 -- 172.21.15.53:0/3136392317 wait complete. 2024-01-16T15:27:18.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.393+0000 7f5ad6c00640 1 Processor -- start 2024-01-16T15:27:18.392 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.394+0000 7f5ad6c00640 1 -- start start 2024-01-16T15:27:18.394 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.394+0000 7f5ad6c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00ae5a0 0x7f5ad014a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad6c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad00b3ae0 0x7f5ad00b2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad6c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad00cf290 0x7f5ad0147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad6c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00cc040 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad6c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad0148530 con 0x7f5ad00cf290 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad6c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00cc1f0 con 0x7f5ad00b3ae0 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00ae5a0 0x7f5ad014a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00ae5a0 0x7f5ad014a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46398/0 (socket says 172.21.15.53:46398) 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 -- 172.21.15.53:0/1895652456 learned_addr learned my addr 172.21.15.53:0/1895652456 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ac7e00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad00b3ae0 0x7f5ad00b2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad4c00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad00cf290 0x7f5ad0147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 -- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad00b3ae0 msgr2=0x7f5ad00b2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5ad00b3ae0 0x7f5ad00b2680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 -- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad00cf290 msgr2=0x7f5ad0147090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad00cf290 0x7f5ad0147090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:18.395 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5acfe00640 1 -- 172.21.15.53:0/1895652456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab8009400 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.395+0000 7f5ad4c00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5ad00cf290 0x7f5ad0147090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.396+0000 7f5acfe00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5ad00ae5a0 0x7f5ad014a710 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000c530 tx=0x7f5ac000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.396+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0005ce0 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.396+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0005e40 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.396+0000 7f5ad6c00640 1 -- 172.21.15.53:0/1895652456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad00c41c0 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.396+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac00072b0 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.396+0000 7f5ad6c00640 1 -- 172.21.15.53:0/1895652456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad0147e70 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.396 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.397+0000 7f5ad6c00640 1 -- 172.21.15.53:0/1895652456 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad0148670 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.397 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.399+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f5ac0007410 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.398 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.400+0000 7f5accc00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aa806c550 0x7f5aa806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:18.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.400+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5ac0088000 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.399 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.401+0000 7f5ac7e00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aa806c550 0x7f5aa806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:18.401 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.402+0000 7f5ac7e00640 1 --2- 172.21.15.53:0/1895652456 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5aa806c550 0x7f5aa806ea10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8036000 tx=0x7f5ab8039040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:18.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.405+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5ad0148670 con 0x7f5ad00ae5a0 2024-01-16T15:27:18.640 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.641+0000 7f5ad6c00640 1 -- 172.21.15.53:0/1895652456 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5ad0070550 con 0x7f5aa806c550 2024-01-16T15:27:18.653 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:18.654+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:18.656+0000 7f5accc00640 1 -- 172.21.15.53:0/1895652456 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5ab802fae0 con 0x7f5aa806c550 2024-01-16T15:27:18.654 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:18.654 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:20.669 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:20.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.899+0000 7f08bae00640 1 Processor -- start 2024-01-16T15:27:20.898 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.899+0000 7f08bae00640 1 -- start start 2024-01-16T15:27:20.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.900+0000 7f08bae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b413cda0 0x7f08b415da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.900+0000 7f08bae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f08b40ce0c0 0x7f08b4070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08bae00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b40700f0 con 0x7f08b415dfa0 2024-01-16T15:27:20.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08bae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b415e3e0 con 0x7f08b40ce0c0 2024-01-16T15:27:20.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08bae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40ce720 con 0x7f08b413cda0 2024-01-16T15:27:20.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08ba400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f08b40ce0c0 0x7f08b4070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b413cda0 0x7f08b415da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08ba400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f08b40ce0c0 0x7f08b4070b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46400/0 (socket says 172.21.15.53:46400) 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.901+0000 7f08ba400640 1 -- 172.21.15.53:0/1409913106 learned_addr learned my addr 172.21.15.53:0/1409913106 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.902+0000 7f08ba400640 1 -- 172.21.15.53:0/1409913106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b413cda0 msgr2=0x7f08b415da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.902+0000 7f08ba400640 1 --2- 172.21.15.53:0/1409913106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b413cda0 0x7f08b415da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.902+0000 7f08ba400640 1 -- 172.21.15.53:0/1409913106 >> v1:172.21.15.146:6789/0 conn(0x7f08b415dfa0 legacy=0x7f08b40cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.902+0000 7f08ba400640 1 -- 172.21.15.53:0/1409913106 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b40cea00 con 0x7f08b40ce0c0 2024-01-16T15:27:20.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.903+0000 7f08ba400640 1 --2- 172.21.15.53:0/1409913106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b4070b40 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009e50 tx=0x7f08ac02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d3dfba90bed844 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.904+0000 7f08b3e00640 1 -- 172.21.15.53:0/1409913106 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac02ca70 con 0x7f08b40ce0c0 2024-01-16T15:27:20.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.904+0000 7f08b3e00640 1 -- 172.21.15.53:0/1409913106 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac02cbd0 con 0x7f08b40ce0c0 2024-01-16T15:27:20.903 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.904+0000 7f08b3e00640 1 -- 172.21.15.53:0/1409913106 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08ac0343f0 con 0x7f08b40ce0c0 2024-01-16T15:27:20.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.904+0000 7f08bae00640 1 -- 172.21.15.53:0/1409913106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 msgr2=0x7f08b4070b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.904+0000 7f08bae00640 1 --2- 172.21.15.53:0/1409913106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b4070b40 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f08ac009e50 tx=0x7f08ac02bed0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.905+0000 7f08bae00640 1 -- 172.21.15.53:0/1409913106 shutdown_connections 2024-01-16T15:27:20.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.905+0000 7f08bae00640 1 --2- 172.21.15.53:0/1409913106 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b4070b40 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.905+0000 7f08bae00640 1 --2- 172.21.15.53:0/1409913106 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b413cda0 0x7f08b415da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.904 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.905+0000 7f08bae00640 1 -- 172.21.15.53:0/1409913106 >> 172.21.15.53:0/1409913106 conn(0x7f08b406a3e0 msgr2=0x7f08b406a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:20.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.905+0000 7f08bae00640 1 -- 172.21.15.53:0/1409913106 shutdown_connections 2024-01-16T15:27:20.905 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.905+0000 7f08bae00640 1 -- 172.21.15.53:0/1409913106 wait complete. 2024-01-16T15:27:20.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.906+0000 7f08bae00640 1 Processor -- start 2024-01-16T15:27:20.906 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.907+0000 7f08bae00640 1 -- start start 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08bae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b413c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08bae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08b413cda0 0x7f08b4154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08bae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b415dfa0 0x7f08b40b2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08bae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b0e90 con 0x7f08b40ce0c0 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08bae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b32f0 con 0x7f08b413cda0 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08bae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b40b35d0 con 0x7f08b415dfa0 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b413c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b413c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:46408/0 (socket says 172.21.15.53:46408) 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 -- 172.21.15.53:0/413386528 learned_addr learned my addr 172.21.15.53:0/413386528 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08ba400640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b415dfa0 0x7f08b40b2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9000640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08b413cda0 0x7f08b4154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.909 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 -- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b415dfa0 msgr2=0x7f08b40b2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b415dfa0 0x7f08b40b2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 -- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08b413cda0 msgr2=0x7f08b4154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08b413cda0 0x7f08b4154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9a00640 1 -- 172.21.15.53:0/413386528 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac009b30 con 0x7f08b40ce0c0 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08ba400640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f08b415dfa0 0x7f08b40b2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.908+0000 7f08b9000640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f08b413cda0 0x7f08b4154d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.909+0000 7f08b9a00640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f08b40ce0c0 0x7f08b413c780 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7f08a400e930 tx=0x7f08a400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.909+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a4009690 con 0x7f08b40ce0c0 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.909+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a400ca00 con 0x7f08b40ce0c0 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.909+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08a400cdb0 con 0x7f08b40ce0c0 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.909+0000 7f08bae00640 1 -- 172.21.15.53:0/413386528 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b40b38b0 con 0x7f08b40ce0c0 2024-01-16T15:27:20.910 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.909+0000 7f08bae00640 1 -- 172.21.15.53:0/413386528 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b41f29d0 con 0x7f08b40ce0c0 2024-01-16T15:27:20.911 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.910+0000 7f08bae00640 1 -- 172.21.15.53:0/413386528 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b4067450 con 0x7f08b40ce0c0 2024-01-16T15:27:20.912 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.912+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f08a4013470 con 0x7f08b40ce0c0 2024-01-16T15:27:20.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.914+0000 7f08b1600640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f088c06c550 0x7f088c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:20.913 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.914+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f08a4089270 con 0x7f08b40ce0c0 2024-01-16T15:27:20.914 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.915+0000 7f08b9000640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f088c06c550 0x7f088c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:20.915 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.916+0000 7f08b9000640 1 --2- 172.21.15.53:0/413386528 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f088c06c550 0x7f088c06ea10 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f08b413f860 tx=0x7f089c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:20.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:20.918+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f08a4055bf0 con 0x7f08b40ce0c0 2024-01-16T15:27:21.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:21.142+0000 7f08bae00640 1 -- 172.21.15.53:0/413386528 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f08b4070620 con 0x7f088c06c550 2024-01-16T15:27:21.158 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:21.159+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:21.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:21.160+0000 7f08b1600640 1 -- 172.21.15.53:0/413386528 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f089c0013e0 con 0x7f088c06c550 2024-01-16T15:27:21.159 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:21.159 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:23.172 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.403+0000 7f20cb400640 1 Processor -- start 2024-01-16T15:27:23.403 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.404+0000 7f20cb400640 1 -- start start 2024-01-16T15:27:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20cb400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f20c40d0100 0x7f20c40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20cb400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 0x7f20c40d13d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20cb400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20c40d0a10 con 0x7f20c4160920 2024-01-16T15:27:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20cb400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20c40cf080 con 0x7f20c40d0100 2024-01-16T15:27:23.404 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20cb400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40c6be0 con 0x7f20c40d0f40 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20ca000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f20c4160920 0x7f20c40cf9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42580/0 (socket says 172.21.15.53:42580) 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.405+0000 7f20ca000640 1 -- 172.21.15.53:0/2224126551 learned_addr learned my addr 172.21.15.53:0/2224126551 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c9600640 1 --2- 172.21.15.53:0/2224126551 >> v2:172.21.15.146:3300/0 conn(0x7f20c40d0100 0x7f20c40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20caa00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 0x7f20c40d13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460423956 0 0) 0x7f20c40d0a10 con 0x7f20c4160920 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20a4003520 con 0x7f20c4160920 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1897766784 0 0) 0x7f20a4003520 con 0x7f20c4160920 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 msgr2=0x7f20c40d13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 0x7f20c40d13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 >> v2:172.21.15.146:3300/0 conn(0x7f20c40d0100 msgr2=0x7f20c40b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 --2- 172.21.15.53:0/2224126551 >> v2:172.21.15.146:3300/0 conn(0x7f20c40d0100 0x7f20c40b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40ce7e0 con 0x7f20c4160920 2024-01-16T15:27:23.407 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f20bc001670 con 0x7f20c4160920 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 >> v1:172.21.15.53:6789/0 conn(0x7f20c4160920 legacy=0x7f20c40cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20a4003970 0x7f20a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 0x7f20a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.406+0000 7f20c8c00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20a4004d00 0x7f20a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40cf080 con 0x7f20a4004d00 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40c6be0 con 0x7f20a4003970 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a400b980 con 0x7f20c40d0f40 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20caa00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20a4004d00 0x7f20a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20ca000640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20a4003970 0x7f20a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20caa00640 1 -- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 msgr2=0x7f20a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20c9600640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 0x7f20a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20caa00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20c40d0f40 0x7f20a40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20caa00640 1 -- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20a4003970 msgr2=0x7f20a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:23.408 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20caa00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20a4003970 0x7f20a4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.409 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.407+0000 7f20caa00640 1 -- 172.21.15.53:0/2224126551 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a4003520 con 0x7f20a4004d00 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.408+0000 7f20caa00640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20a4004d00 0x7f20a4005180 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009ac0 tx=0x7f20a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e41e9ab79d99cc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b402c6b0 con 0x7f20a4004d00 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b402c810 con 0x7f20a4004d00 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20c8c00640 1 -- 172.21.15.53:0/2224126551 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b402cba0 con 0x7f20a4004d00 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20cb400640 1 -- 172.21.15.53:0/2224126551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20a4004d00 msgr2=0x7f20a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20cb400640 1 --2- 172.21.15.53:0/2224126551 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20a4004d00 0x7f20a4005180 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009ac0 tx=0x7f20a40057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20caa00640 1 -- 172.21.15.53:0/2224126551 reap_dead start 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20cb400640 1 -- 172.21.15.53:0/2224126551 shutdown_connections 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20cb400640 1 -- 172.21.15.53:0/2224126551 >> 172.21.15.53:0/2224126551 conn(0x7f20c40d3090 msgr2=0x7f20c40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.409+0000 7f20cb400640 1 -- 172.21.15.53:0/2224126551 shutdown_connections 2024-01-16T15:27:23.410 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.410+0000 7f20cb400640 1 -- 172.21.15.53:0/2224126551 wait complete. 2024-01-16T15:27:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.411+0000 7f20cb400640 1 Processor -- start 2024-01-16T15:27:23.411 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.411+0000 7f20cb400640 1 -- start start 2024-01-16T15:27:23.412 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.412+0000 7f20cb400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20a4004d00 0x7f20c41596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.412+0000 7f20cb400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20c414d5d0 0x7f20c4159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.412+0000 7f20cb400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20c40b6d10 0x7f20c40b7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.412+0000 7f20cb400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b85a0 con 0x7f20c40b6d10 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.412+0000 7f20cb400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b7990 con 0x7f20c414d5d0 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.412+0000 7f20cb400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20c40b7c70 con 0x7f20a4004d00 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20c40b6d10 0x7f20c40b7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20c40b6d10 0x7f20c40b7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47712/0 (socket says 172.21.15.53:47712) 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 -- 172.21.15.53:0/2662487887 learned_addr learned my addr 172.21.15.53:0/2662487887 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20c9600640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20c414d5d0 0x7f20c4159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20ca000640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20a4004d00 0x7f20c41596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 -- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20a4004d00 msgr2=0x7f20c41596d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f20a4004d00 0x7f20c41596d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.413 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 -- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20c414d5d0 msgr2=0x7f20c4159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f20c414d5d0 0x7f20c4159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.413+0000 7f20caa00640 1 -- 172.21.15.53:0/2662487887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20c40b7f50 con 0x7f20c40b6d10 2024-01-16T15:27:23.414 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.414+0000 7f20caa00640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f20c40b6d10 0x7f20c40b7170 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x7f20b4009be0 tx=0x7f20b4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.414+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b4004440 con 0x7f20c40b6d10 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.414+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20b402de10 con 0x7f20c40b6d10 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.414+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20b40358c0 con 0x7f20c40b6d10 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.414+0000 7f20cb400640 1 -- 172.21.15.53:0/2662487887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20c40b8230 con 0x7f20c40b6d10 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.414+0000 7f20cb400640 1 -- 172.21.15.53:0/2662487887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20c40b8a30 con 0x7f20c40b6d10 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.415+0000 7f20cb400640 1 -- 172.21.15.53:0/2662487887 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20c40b83c0 con 0x7f20c40b6d10 2024-01-16T15:27:23.415 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.416+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f20b403b050 con 0x7f20c40b6d10 2024-01-16T15:27:23.416 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.417+0000 7f20c2000640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f209c06c580 0x7f209c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.417+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f20b4000e40 con 0x7f20c40b6d10 2024-01-16T15:27:23.417 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.418+0000 7f20ca000640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f209c06c580 0x7f209c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:23.418 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.419+0000 7f20ca000640 1 --2- 172.21.15.53:0/2662487887 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f209c06c580 0x7f209c06ea40 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f20bc000fb0 tx=0x7f20bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:23.421 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.422+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f20b4076100 con 0x7f20c40b6d10 2024-01-16T15:27:23.653 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.653+0000 7f20cb400640 1 -- 172.21.15.53:0/2662487887 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f20c40b7de0 con 0x7f209c06c580 2024-01-16T15:27:23.667 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:23.669+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:23.667 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:23.668+0000 7f20c2000640 1 -- 172.21.15.53:0/2662487887 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f20c40b7de0 con 0x7f209c06c580 2024-01-16T15:27:23.667 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:23.667 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:23.679 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:25.681 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:25.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.916+0000 7fa807200640 1 Processor -- start 2024-01-16T15:27:25.916 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.917+0000 7fa807200640 1 -- start start 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.918+0000 7fa807200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8081447f0 0x7fa80815c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.918+0000 7fa807200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8080cccf0 0x7fa80815c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.918+0000 7fa807200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa80815c5c0 con 0x7fa80806c470 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.918+0000 7fa807200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa80806d690 con 0x7fa8081447f0 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.918+0000 7fa807200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8080cebb0 con 0x7fa8080cccf0 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8081447f0 0x7fa80815c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa8081447f0 0x7fa80815c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47722/0 (socket says 172.21.15.53:47722) 2024-01-16T15:27:25.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 -- 172.21.15.53:0/3092531823 learned_addr learned my addr 172.21.15.53:0/3092531823 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa806800640 1 --2- 172.21.15.53:0/3092531823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8080cccf0 0x7fa80815c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 -- 172.21.15.53:0/3092531823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8080cccf0 msgr2=0x7fa80815c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 --2- 172.21.15.53:0/3092531823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8080cccf0 0x7fa80815c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 -- 172.21.15.53:0/3092531823 >> v1:172.21.15.146:6789/0 conn(0x7fa80806c470 legacy=0x7fa8080cc5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:25.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.919+0000 7fa805e00640 1 -- 172.21.15.53:0/3092531823 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80806ca10 con 0x7fa8081447f0 2024-01-16T15:27:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.920+0000 7fa805e00640 1 --2- 172.21.15.53:0/3092531823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8081447f0 0x7fa80815c080 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa8080cecf0 tx=0x7fa7f802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=965cc00ced189089 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.921+0000 7fa805400640 1 -- 172.21.15.53:0/3092531823 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f802c9c0 con 0x7fa8081447f0 2024-01-16T15:27:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.921+0000 7fa805400640 1 -- 172.21.15.53:0/3092531823 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f802cb20 con 0x7fa8081447f0 2024-01-16T15:27:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.921+0000 7fa805400640 1 -- 172.21.15.53:0/3092531823 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f80343f0 con 0x7fa8081447f0 2024-01-16T15:27:25.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.921+0000 7fa807200640 1 -- 172.21.15.53:0/3092531823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8081447f0 msgr2=0x7fa80815c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.921+0000 7fa807200640 1 --2- 172.21.15.53:0/3092531823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8081447f0 0x7fa80815c080 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x7fa8080cecf0 tx=0x7fa7f802be20 comp rx=0 tx=0).stop 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.922+0000 7fa807200640 1 -- 172.21.15.53:0/3092531823 shutdown_connections 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.922+0000 7fa807200640 1 --2- 172.21.15.53:0/3092531823 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8080cccf0 0x7fa80815c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.922+0000 7fa807200640 1 --2- 172.21.15.53:0/3092531823 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8081447f0 0x7fa80815c080 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.922+0000 7fa807200640 1 -- 172.21.15.53:0/3092531823 >> 172.21.15.53:0/3092531823 conn(0x7fa80806a3e0 msgr2=0x7fa80806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.922+0000 7fa807200640 1 -- 172.21.15.53:0/3092531823 shutdown_connections 2024-01-16T15:27:25.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.922+0000 7fa807200640 1 -- 172.21.15.53:0/3092531823 wait complete. 2024-01-16T15:27:25.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.923+0000 7fa807200640 1 Processor -- start 2024-01-16T15:27:25.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.924+0000 7fa807200640 1 -- start start 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.924+0000 7fa807200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa80806c470 0x7fa80815b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa807200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8080cccf0 0x7fa80813e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa807200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8081447f0 0x7fa80813a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa807200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8081397e0 con 0x7fa8080cccf0 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa807200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa808139220 con 0x7fa80806c470 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa807200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa808139500 con 0x7fa8081447f0 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8080cccf0 0x7fa80813e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8080cccf0 0x7fa80813e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47736/0 (socket says 172.21.15.53:47736) 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 -- 172.21.15.53:0/2538816663 learned_addr learned my addr 172.21.15.53:0/2538816663 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa806800640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8081447f0 0x7fa80813a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa805e00640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa80806c470 0x7fa80815b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 -- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8081447f0 msgr2=0x7fa80813a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa8081447f0 0x7fa80813a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 -- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa80806c470 msgr2=0x7fa80815b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:25.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa80806c470 0x7fa80815b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:25.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa7fd200640 1 -- 172.21.15.53:0/2538816663 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f8009b30 con 0x7fa8080cccf0 2024-01-16T15:27:25.926 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.925+0000 7fa805e00640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa80806c470 0x7fa80815b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.926+0000 7fa7fd200640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa8080cccf0 0x7fa80813e570 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000c430 tx=0x7fa7f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.926+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0011790 con 0x7fa8080cccf0 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.926+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f00118f0 con 0x7fa8080cccf0 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.926+0000 7fa807200640 1 -- 172.21.15.53:0/2538816663 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa80813bb70 con 0x7fa8080cccf0 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.926+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0011ce0 con 0x7fa8080cccf0 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.926+0000 7fa807200640 1 -- 172.21.15.53:0/2538816663 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa808138e60 con 0x7fa8080cccf0 2024-01-16T15:27:25.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.927+0000 7fa807200640 1 -- 172.21.15.53:0/2538816663 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa80813b9d0 con 0x7fa8080cccf0 2024-01-16T15:27:25.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.929+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fa7f001c460 con 0x7fa8080cccf0 2024-01-16T15:27:25.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.929+0000 7fa7fea00640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7e006c550 0x7fa7e006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:25.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.930+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa7f0088b50 con 0x7fa8080cccf0 2024-01-16T15:27:25.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.930+0000 7fa805e00640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7e006c550 0x7fa7e006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:25.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.931+0000 7fa805e00640 1 --2- 172.21.15.53:0/2538816663 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa7e006c550 0x7fa7e006ea10 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fa7f8035000 tx=0x7fa7f8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:25.934 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:25.934+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa80813b9d0 con 0x7fa8080cccf0 2024-01-16T15:27:26.165 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:26.166+0000 7fa807200640 1 -- 172.21.15.53:0/2538816663 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa80806d2b0 con 0x7fa7e006c550 2024-01-16T15:27:26.178 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:26.180+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:26.179 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:26.180+0000 7fa7fea00640 1 -- 172.21.15.53:0/2538816663 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fa7f8040450 con 0x7fa7e006c550 2024-01-16T15:27:26.179 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:26.180 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:26.201 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:28.203 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:28.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.442+0000 7fab75a00640 1 Processor -- start 2024-01-16T15:27:28.442 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.442+0000 7fab75a00640 1 -- start start 2024-01-16T15:27:28.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.444+0000 7fab75a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab7013cda0 0x7fab7015da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.444+0000 7fab75a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab700ce0c0 0x7fab70070b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.444+0000 7fab75a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab700700f0 con 0x7fab7015dfa0 2024-01-16T15:27:28.443 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.444+0000 7fab75a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab700ce530 con 0x7fab7013cda0 2024-01-16T15:27:28.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.444+0000 7fab75a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7015e3e0 con 0x7fab700ce0c0 2024-01-16T15:27:28.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab7013cda0 0x7fab7015da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab75000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab700ce0c0 0x7fab70070b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fab7013cda0 0x7fab7015da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:54436/0 (socket says 172.21.15.53:54436) 2024-01-16T15:27:28.444 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 -- 172.21.15.53:0/2699731735 learned_addr learned my addr 172.21.15.53:0/2699731735 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:28.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 -- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab700ce0c0 msgr2=0x7fab70070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 --2- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab700ce0c0 0x7fab70070b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 -- 172.21.15.53:0/2699731735 >> v1:172.21.15.146:6789/0 conn(0x7fab7015dfa0 legacy=0x7fab700cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:28.445 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.445+0000 7fab6fe00640 1 -- 172.21.15.53:0/2699731735 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab700ce720 con 0x7fab7013cda0 2024-01-16T15:27:28.446 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.447+0000 7fab6fe00640 1 --2- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab7013cda0 0x7fab7015da60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fab60009870 tx=0x7fab6002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e63a2b6c5131be9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.448+0000 7fab6ea00640 1 -- 172.21.15.53:0/2699731735 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600317c0 con 0x7fab7013cda0 2024-01-16T15:27:28.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.448+0000 7fab6ea00640 1 -- 172.21.15.53:0/2699731735 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab60031920 con 0x7fab7013cda0 2024-01-16T15:27:28.447 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.448+0000 7fab6ea00640 1 -- 172.21.15.53:0/2699731735 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60031cb0 con 0x7fab7013cda0 2024-01-16T15:27:28.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.448+0000 7fab75a00640 1 -- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab7013cda0 msgr2=0x7fab7015da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.448+0000 7fab75a00640 1 --2- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab7013cda0 0x7fab7015da60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fab60009870 tx=0x7fab6002b860 comp rx=0 tx=0).stop 2024-01-16T15:27:28.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.449+0000 7fab75a00640 1 -- 172.21.15.53:0/2699731735 shutdown_connections 2024-01-16T15:27:28.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.449+0000 7fab75a00640 1 --2- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab700ce0c0 0x7fab70070b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.449+0000 7fab75a00640 1 --2- 172.21.15.53:0/2699731735 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab7013cda0 0x7fab7015da60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.449+0000 7fab75a00640 1 -- 172.21.15.53:0/2699731735 >> 172.21.15.53:0/2699731735 conn(0x7fab7006a3e0 msgr2=0x7fab7006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:28.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.449+0000 7fab75a00640 1 -- 172.21.15.53:0/2699731735 shutdown_connections 2024-01-16T15:27:28.449 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.449+0000 7fab75a00640 1 -- 172.21.15.53:0/2699731735 wait complete. 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.451+0000 7fab75a00640 1 Processor -- start 2024-01-16T15:27:28.451 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.451+0000 7fab75a00640 1 -- start start 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab700ce0c0 0x7fab7013bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7013cda0 0x7fab7013f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab7015dfa0 0x7fab701500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7013fc20 con 0x7fab7013cda0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7013f9a0 con 0x7fab700ce0c0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab70153bd0 con 0x7fab7015dfa0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab6f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7013cda0 0x7fab7013f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab6f400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7013cda0 0x7fab7013f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47744/0 (socket says 172.21.15.53:47744) 2024-01-16T15:27:28.453 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab6f400640 1 -- 172.21.15.53:0/4099069206 learned_addr learned my addr 172.21.15.53:0/4099069206 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab6fe00640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab700ce0c0 0x7fab7013bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.453+0000 7fab75000640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab7015dfa0 0x7fab701500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab6f400640 1 -- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab7015dfa0 msgr2=0x7fab701500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab6f400640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fab7015dfa0 0x7fab701500e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab6f400640 1 -- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab700ce0c0 msgr2=0x7fab7013bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab6f400640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fab700ce0c0 0x7fab7013bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:28.454 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab6f400640 1 -- 172.21.15.53:0/4099069206 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab600094d0 con 0x7fab7013cda0 2024-01-16T15:27:28.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab6f400640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fab7013cda0 0x7fab7013f460 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x7fab5800c430 tx=0x7fab5800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58011790 con 0x7fab7013cda0 2024-01-16T15:27:28.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab580118f0 con 0x7fab7013cda0 2024-01-16T15:27:28.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab75a00640 1 -- 172.21.15.53:0/4099069206 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab70153eb0 con 0x7fab7013cda0 2024-01-16T15:27:28.455 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.454+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab58011ce0 con 0x7fab7013cda0 2024-01-16T15:27:28.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.455+0000 7fab75a00640 1 -- 172.21.15.53:0/4099069206 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab70150620 con 0x7fab7013cda0 2024-01-16T15:27:28.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.456+0000 7fab75a00640 1 -- 172.21.15.53:0/4099069206 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab70067450 con 0x7fab7013cda0 2024-01-16T15:27:28.456 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.457+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fab5801c460 con 0x7fab7013cda0 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.458+0000 7fab67e00640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fab4806c550 0x7fab4806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:28.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.458+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fab58088b40 con 0x7fab7013cda0 2024-01-16T15:27:28.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.459+0000 7fab6fe00640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fab4806c550 0x7fab4806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:28.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.460+0000 7fab6fe00640 1 --2- 172.21.15.53:0/4099069206 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fab4806c550 0x7fab4806ea10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fab60036000 tx=0x7fab6003d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:28.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.462+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fab58055080 con 0x7fab7013cda0 2024-01-16T15:27:28.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.693+0000 7fab75a00640 1 -- 172.21.15.53:0/4099069206 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab70070620 con 0x7fab4806c550 2024-01-16T15:27:28.706 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:28.707+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:28.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:28.707+0000 7fab67e00640 1 -- 172.21.15.53:0/4099069206 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7fab60031a90 con 0x7fab4806c550 2024-01-16T15:27:28.707 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:28.707 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:28.720 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:29.136 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:29.144 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:30.721 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:30.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.963+0000 7f297fc00640 1 Processor -- start 2024-01-16T15:27:30.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.964+0000 7f297fc00640 1 -- start start 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f297fc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f297813e360 0x7f29780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f297fc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297815c5f0 con 0x7f297813df80 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f297fc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f297806c470 con 0x7f297815f9c0 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f297fc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f297806c750 con 0x7f297813e360 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f2977e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f297813e360 0x7f29780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f297d200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f297813df80 0x7f297815bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:42584/0 (socket says 172.21.15.53:42584) 2024-01-16T15:27:30.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.965+0000 7f297d200640 1 -- 172.21.15.53:0/761583523 learned_addr learned my addr 172.21.15.53:0/761583523 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:30.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.966+0000 7f2977e00640 1 -- 172.21.15.53:0/761583523 >> v1:172.21.15.146:6789/0 conn(0x7f297815f9c0 legacy=0x7f29780cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:30.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.966+0000 7f2977e00640 1 -- 172.21.15.53:0/761583523 >> v1:172.21.15.53:6789/0 conn(0x7f297813df80 legacy=0x7f297815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.966+0000 7f2977e00640 1 -- 172.21.15.53:0/761583523 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297806ca30 con 0x7f297813e360 2024-01-16T15:27:30.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.967+0000 7f2977e00640 1 --2- 172.21.15.53:0/761583523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780cf290 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f29600097a0 tx=0x7f296002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce0c7ea97470bd57 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.967+0000 7f2977400640 1 -- 172.21.15.53:0/761583523 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2960031830 con 0x7f297813e360 2024-01-16T15:27:30.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.967+0000 7f2977400640 1 -- 172.21.15.53:0/761583523 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2960031990 con 0x7f297813e360 2024-01-16T15:27:30.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.967+0000 7f2977400640 1 -- 172.21.15.53:0/761583523 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2960031d20 con 0x7f297813e360 2024-01-16T15:27:30.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 -- 172.21.15.53:0/761583523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 msgr2=0x7f29780cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 --2- 172.21.15.53:0/761583523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780cf290 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x7f29600097a0 tx=0x7f296002b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 -- 172.21.15.53:0/761583523 shutdown_connections 2024-01-16T15:27:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 --2- 172.21.15.53:0/761583523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780cf290 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 -- 172.21.15.53:0/761583523 >> 172.21.15.53:0/761583523 conn(0x7f297806a800 msgr2=0x7f297806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 -- 172.21.15.53:0/761583523 shutdown_connections 2024-01-16T15:27:30.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.968+0000 7f297fc00640 1 -- 172.21.15.53:0/761583523 wait complete. 2024-01-16T15:27:30.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.970+0000 7f297fc00640 1 Processor -- start 2024-01-16T15:27:30.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.970+0000 7f297fc00640 1 -- start start 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297fc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297813df80 0x7f2978152c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297fc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297fc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f297815f9c0 0x7f2978152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297fc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781e6160 con 0x7f297813e360 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297fc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781534e0 con 0x7f297813df80 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297fc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2978152aa0 con 0x7f297815f9c0 2024-01-16T15:27:30.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f2977e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f2977e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:47760/0 (socket says 172.21.15.53:47760) 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f2977e00640 1 -- 172.21.15.53:0/1495441027 learned_addr learned my addr 172.21.15.53:0/1495441027 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f297d200640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297813df80 0x7f2978152c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.971+0000 7f2977e00640 1 -- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f297815f9c0 msgr2=0x7f2978152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2977e00640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f297815f9c0 0x7f2978152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2977e00640 1 -- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297813df80 msgr2=0x7f2978152c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2977e00640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f297813df80 0x7f2978152c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:30.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2977e00640 1 -- 172.21.15.53:0/1495441027 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960009400 con 0x7f297813e360 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2977e00640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f297813e360 0x7f29780b2cf0 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f296002bee0 tx=0x7f29600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f296003ba70 con 0x7f297813e360 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f296003bbf0 con 0x7f297813e360 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29600433f0 con 0x7f297813e360 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.972+0000 7f297fc00640 1 -- 172.21.15.53:0/1495441027 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29780b33f0 con 0x7f297813e360 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.973+0000 7f297fc00640 1 -- 172.21.15.53:0/1495441027 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29781e5df0 con 0x7f297813e360 2024-01-16T15:27:30.973 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.974+0000 7f297fc00640 1 -- 172.21.15.53:0/1495441027 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29781e5fb0 con 0x7f297813e360 2024-01-16T15:27:30.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.975+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f296004c3b0 con 0x7f297813e360 2024-01-16T15:27:30.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.975+0000 7f2974c00640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f295006c550 0x7f295006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:30.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.976+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f29600aa7b0 con 0x7f297813e360 2024-01-16T15:27:30.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.976+0000 7f297d200640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f295006c550 0x7f295006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:30.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.977+0000 7f297d200640 1 --2- 172.21.15.53:0/1495441027 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f295006c550 0x7f295006ea10 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f29680012e0 tx=0x7f2968009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:30.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:30.980+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f29600770d0 con 0x7f297813e360 2024-01-16T15:27:31.207 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:31.208+0000 7f297fc00640 1 -- 172.21.15.53:0/1495441027 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f297806cff0 con 0x7f295006c550 2024-01-16T15:27:31.221 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:31.223+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:31.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:31.223+0000 7f2974c00640 1 -- 172.21.15.53:0/1495441027 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2968001d50 con 0x7f295006c550 2024-01-16T15:27:31.222 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:31.222 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:31.237 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:33.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:33.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.477+0000 7ff693a00640 1 Processor -- start 2024-01-16T15:27:33.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.477+0000 7ff693a00640 1 -- start start 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff693a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff68c143ae0 0x7ff68c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff693a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c0cf7d0 0x7ff68c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff693a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff68c15e1e0 0x7ff68c15a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff693a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c0cfc30 con 0x7ff68c15e1e0 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff693a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68c0b1ee0 con 0x7ff68c143ae0 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff693a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0b2780 con 0x7ff68c0cf7d0 2024-01-16T15:27:33.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff691a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff68c15e1e0 0x7ff68c15a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff691a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff68c15e1e0 0x7ff68c15a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37278/0 (socket says 172.21.15.53:37278) 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff691a00640 1 -- 172.21.15.53:0/2730302852 learned_addr learned my addr 172.21.15.53:0/2730302852 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff68be00640 1 --2- 172.21.15.53:0/2730302852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c0cf7d0 0x7ff68c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.479+0000 7ff691000640 1 --2- 172.21.15.53:0/2730302852 >> v2:172.21.15.146:3300/0 conn(0x7ff68c143ae0 0x7ff68c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.480+0000 7ff691a00640 1 -- 172.21.15.53:0/2730302852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c0cf7d0 msgr2=0x7ff68c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.480+0000 7ff691a00640 1 --2- 172.21.15.53:0/2730302852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c0cf7d0 0x7ff68c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.480+0000 7ff691a00640 1 -- 172.21.15.53:0/2730302852 >> v2:172.21.15.146:3300/0 conn(0x7ff68c143ae0 msgr2=0x7ff68c0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.480+0000 7ff691a00640 1 --2- 172.21.15.53:0/2730302852 >> v2:172.21.15.146:3300/0 conn(0x7ff68c143ae0 0x7ff68c0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.480+0000 7ff691a00640 1 -- 172.21.15.53:0/2730302852 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff68c15e990 con 0x7ff68c15e1e0 2024-01-16T15:27:33.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.481+0000 7ff691a00640 1 --2- 172.21.15.53:0/2730302852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c15e1e0 0x7ff68c15a670 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0cfea0 tx=0x7ff68402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d3132bf9d3251b1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:33.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.481+0000 7ff68b400640 1 -- 172.21.15.53:0/2730302852 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68402ca70 con 0x7ff68c15e1e0 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.481+0000 7ff68b400640 1 -- 172.21.15.53:0/2730302852 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68402cbd0 con 0x7ff68c15e1e0 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.481+0000 7ff68b400640 1 -- 172.21.15.53:0/2730302852 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6840343f0 con 0x7ff68c15e1e0 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 -- 172.21.15.53:0/2730302852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c15e1e0 msgr2=0x7ff68c15a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 --2- 172.21.15.53:0/2730302852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c15e1e0 0x7ff68c15a670 secure :-1 s=READY pgs=1952 cs=0 l=1 rev1=1 crypto rx=0x7ff68c0cfea0 tx=0x7ff68402bed0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 -- 172.21.15.53:0/2730302852 shutdown_connections 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 --2- 172.21.15.53:0/2730302852 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c15e1e0 0x7ff68c15a670 unknown :-1 s=CLOSED pgs=1952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 --2- 172.21.15.53:0/2730302852 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c0cf7d0 0x7ff68c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 --2- 172.21.15.53:0/2730302852 >> v2:172.21.15.146:3300/0 conn(0x7ff68c143ae0 0x7ff68c0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 -- 172.21.15.53:0/2730302852 >> 172.21.15.53:0/2730302852 conn(0x7ff68c06a800 msgr2=0x7ff68c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 -- 172.21.15.53:0/2730302852 shutdown_connections 2024-01-16T15:27:33.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.482+0000 7ff693a00640 1 -- 172.21.15.53:0/2730302852 wait complete. 2024-01-16T15:27:33.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.484+0000 7ff693a00640 1 Processor -- start 2024-01-16T15:27:33.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.484+0000 7ff693a00640 1 -- start start 2024-01-16T15:27:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff693a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff68c0cf7d0 0x7ff68c141bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff693a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c143ae0 0x7ff68c1f33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff693a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c15e1e0 0x7ff68c0c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff693a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c142100 con 0x7ff68c143ae0 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff693a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0c43f0 con 0x7ff68c0cf7d0 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff693a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68c0c46d0 con 0x7ff68c15e1e0 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff68be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c143ae0 0x7ff68c1f33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff68be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c143ae0 0x7ff68c1f33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37292/0 (socket says 172.21.15.53:37292) 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff68be00640 1 -- 172.21.15.53:0/1806889073 learned_addr learned my addr 172.21.15.53:0/1806889073 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.485+0000 7ff691000640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff68c0cf7d0 0x7ff68c141bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff691a00640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c15e1e0 0x7ff68c0c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff68be00640 1 -- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c15e1e0 msgr2=0x7ff68c0c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff68be00640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c15e1e0 0x7ff68c0c5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff68be00640 1 -- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff68c0cf7d0 msgr2=0x7ff68c141bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff68be00640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff68c0cf7d0 0x7ff68c141bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:33.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff68be00640 1 -- 172.21.15.53:0/1806889073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684009b30 con 0x7ff68c143ae0 2024-01-16T15:27:33.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff691a00640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff68c15e1e0 0x7ff68c0c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:33.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff68be00640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff68c143ae0 0x7ff68c1f33e0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7ff67400d800 tx=0x7ff67400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6740096a0 con 0x7ff68c143ae0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff674009800 con 0x7ff68c143ae0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff674009b90 con 0x7ff68c143ae0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.486+0000 7ff693a00640 1 -- 172.21.15.53:0/1806889073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff68c0c49b0 con 0x7ff68c143ae0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.487+0000 7ff693a00640 1 -- 172.21.15.53:0/1806889073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68c0c4f70 con 0x7ff68c143ae0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.488+0000 7ff693a00640 1 -- 172.21.15.53:0/1806889073 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68c0c5130 con 0x7ff68c143ae0 2024-01-16T15:27:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.489+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7ff674009cf0 con 0x7ff68c143ae0 2024-01-16T15:27:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.490+0000 7ff688c00640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff66406c550 0x7ff66406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.490+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff674089010 con 0x7ff68c143ae0 2024-01-16T15:27:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.490+0000 7ff691000640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff66406c550 0x7ff66406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.491+0000 7ff691000640 1 --2- 172.21.15.53:0/1806889073 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff66406c550 0x7ff66406ea10 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7ff67c006fd0 tx=0x7ff67c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:33.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.494+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff674055550 con 0x7ff68c143ae0 2024-01-16T15:27:33.729 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.729+0000 7ff693a00640 1 -- 172.21.15.53:0/1806889073 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff68c15e640 con 0x7ff66406c550 2024-01-16T15:27:33.744 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:33.745+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:33.744 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:33.745+0000 7ff688c00640 1 -- 172.21.15.53:0/1806889073 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7ff67c00b600 con 0x7ff66406c550 2024-01-16T15:27:33.745 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:33.745 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:35.759 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:35.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.994+0000 7f129c800640 1 Processor -- start 2024-01-16T15:27:35.993 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.994+0000 7f129c800640 1 -- start start 2024-01-16T15:27:35.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.995+0000 7f129c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940ae5a0 0x7f129415e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:35.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.995+0000 7f129c800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12940cf290 0x7f12940cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:35.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.995+0000 7f129c800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1294070c40 con 0x7f12940b3ae0 2024-01-16T15:27:35.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.995+0000 7f129c800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1294071450 con 0x7f12940cf290 2024-01-16T15:27:35.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.995+0000 7f129c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12940cfc60 con 0x7f12940ae5a0 2024-01-16T15:27:35.995 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.996+0000 7f129a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12940cf290 0x7f12940cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.996+0000 7f129a800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f12940cf290 0x7f12940cf720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:33750/0 (socket says 172.21.15.53:33750) 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.996+0000 7f129a800640 1 -- 172.21.15.53:0/3216565130 learned_addr learned my addr 172.21.15.53:0/3216565130 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.996+0000 7f1299e00640 1 --2- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940ae5a0 0x7f129415e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.997+0000 7f129a800640 1 -- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940ae5a0 msgr2=0x7f129415e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.997+0000 7f129a800640 1 --2- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940ae5a0 0x7f129415e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.997+0000 7f129a800640 1 -- 172.21.15.53:0/3216565130 >> v1:172.21.15.146:6789/0 conn(0x7f12940b3ae0 legacy=0x7f129406cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:35.996 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.997+0000 7f129a800640 1 -- 172.21.15.53:0/3216565130 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129406c470 con 0x7f12940cf290 2024-01-16T15:27:35.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.998+0000 7f129a800640 1 --2- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940cf290 0x7f12940cf720 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f128c009e50 tx=0x7f128c02bff0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c409a8d19c52f2a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:35.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f1298a00640 1 -- 172.21.15.53:0/3216565130 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c02cb90 con 0x7f12940cf290 2024-01-16T15:27:35.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f1298a00640 1 -- 172.21.15.53:0/3216565130 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c02ccf0 con 0x7f12940cf290 2024-01-16T15:27:35.998 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f1298a00640 1 -- 172.21.15.53:0/3216565130 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c034610 con 0x7f12940cf290 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 -- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940cf290 msgr2=0x7f12940cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 --2- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940cf290 0x7f12940cf720 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f128c009e50 tx=0x7f128c02bff0 comp rx=0 tx=0).stop 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 -- 172.21.15.53:0/3216565130 shutdown_connections 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 --2- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940cf290 0x7f12940cf720 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 --2- 172.21.15.53:0/3216565130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940ae5a0 0x7f129415e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 -- 172.21.15.53:0/3216565130 >> 172.21.15.53:0/3216565130 conn(0x7f129406a800 msgr2=0x7f129406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:35.999+0000 7f129c800640 1 -- 172.21.15.53:0/3216565130 shutdown_connections 2024-01-16T15:27:35.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.000+0000 7f129c800640 1 -- 172.21.15.53:0/3216565130 wait complete. 2024-01-16T15:27:36.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.001+0000 7f129c800640 1 Processor -- start 2024-01-16T15:27:36.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.001+0000 7f129c800640 1 -- start start 2024-01-16T15:27:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.002+0000 7f129c800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12940ae5a0 0x7f12941f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.002+0000 7f129c800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940b3ae0 0x7f12941f6440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.002+0000 7f129c800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940cf290 0x7f12941f6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.002+0000 7f129c800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12940cc020 con 0x7f12940ae5a0 2024-01-16T15:27:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.002+0000 7f129c800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12941604d0 con 0x7f12940b3ae0 2024-01-16T15:27:36.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.002+0000 7f129c800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12941601f0 con 0x7f12940cf290 2024-01-16T15:27:36.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12940ae5a0 0x7f12941f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12940ae5a0 0x7f12941f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37294/0 (socket says 172.21.15.53:37294) 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 -- 172.21.15.53:0/221835912 learned_addr learned my addr 172.21.15.53:0/221835912 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299400640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940b3ae0 0x7f12941f6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f129a800640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940cf290 0x7f12941f6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 -- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940cf290 msgr2=0x7f12941f6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f12940cf290 0x7f12941f6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 -- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940b3ae0 msgr2=0x7f12941f6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f12940b3ae0 0x7f12941f6440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.003+0000 7f1299e00640 1 -- 172.21.15.53:0/221835912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c009b30 con 0x7f12940ae5a0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.004+0000 7f1299e00640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f12940ae5a0 0x7f12941f2660 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f128400e9b0 tx=0x7f128400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.004+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284009670 con 0x7f12940ae5a0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.004+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128400c9a0 con 0x7f12940ae5a0 2024-01-16T15:27:36.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.004+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128400cd50 con 0x7f12940ae5a0 2024-01-16T15:27:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.004+0000 7f129c800640 1 -- 172.21.15.53:0/221835912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12940cb9d0 con 0x7f12940ae5a0 2024-01-16T15:27:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.004+0000 7f129c800640 1 -- 172.21.15.53:0/221835912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12941f2ba0 con 0x7f12940ae5a0 2024-01-16T15:27:36.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.005+0000 7f129c800640 1 -- 172.21.15.53:0/221835912 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12940675d0 con 0x7f12940ae5a0 2024-01-16T15:27:36.006 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.006+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1284013470 con 0x7f12940ae5a0 2024-01-16T15:27:36.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.008+0000 7f1292000640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f126c06c550 0x7f126c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:36.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.008+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1284088a20 con 0x7f12940ae5a0 2024-01-16T15:27:36.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.008+0000 7f1299400640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f126c06c550 0x7f126c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:36.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.010+0000 7f1299400640 1 --2- 172.21.15.53:0/221835912 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f126c06c550 0x7f126c06ea10 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f12940cea20 tx=0x7f127c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:36.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.013+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f12840552e0 con 0x7f12940ae5a0 2024-01-16T15:27:36.239 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.240+0000 7f129c800640 1 -- 172.21.15.53:0/221835912 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1294071170 con 0x7f126c06c550 2024-01-16T15:27:36.252 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:36.253+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:36.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:36.253+0000 7f1292000640 1 -- 172.21.15.53:0/221835912 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f127c001360 con 0x7f126c06c550 2024-01-16T15:27:36.253 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:36.253 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:36.266 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:38.267 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:38.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.501+0000 7f695c000640 1 Processor -- start 2024-01-16T15:27:38.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.501+0000 7f695c000640 1 -- start start 2024-01-16T15:27:38.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.502+0000 7f695c000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f695413df80 0x7f695415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.502+0000 7f695c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 0x7f69540cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.502+0000 7f695c000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f695406c470 con 0x7f695415f9c0 2024-01-16T15:27:38.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.502+0000 7f695c000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f695406c750 con 0x7f695413df80 2024-01-16T15:27:38.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.502+0000 7f695c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f695406ca30 con 0x7f695413e360 2024-01-16T15:27:38.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.503+0000 7f695a000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f695415f9c0 0x7f69541408d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37800/0 (socket says 172.21.15.53:37800) 2024-01-16T15:27:38.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.503+0000 7f695a000640 1 -- 172.21.15.53:0/190950547 learned_addr learned my addr 172.21.15.53:0/190950547 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.503+0000 7f6959600640 1 --2- 172.21.15.53:0/190950547 >> v2:172.21.15.146:3300/0 conn(0x7f695413df80 0x7f695415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.503+0000 7f6958c00640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 0x7f69540cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.503+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201120295 0 0) 0x7f695406c470 con 0x7f695415f9c0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.503+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6934003520 con 0x7f695415f9c0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2950141643 0 0) 0x7f6934003520 con 0x7f695415f9c0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 msgr2=0x7f69540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 0x7f69540cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 >> v2:172.21.15.146:3300/0 conn(0x7f695413df80 msgr2=0x7f695415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 --2- 172.21.15.53:0/190950547 >> v2:172.21.15.146:3300/0 conn(0x7f695413df80 0x7f695415bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695406cd10 con 0x7f695415f9c0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f694c001670 con 0x7f695415f9c0 2024-01-16T15:27:38.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 >> v1:172.21.15.53:6789/0 conn(0x7f695415f9c0 legacy=0x7f69541408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934003970 0x7f6934003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 0x7f69340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6934004d00 0x7f6934005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695406c750 con 0x7f6934004d00 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695406ca30 con 0x7f6934003970 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693400b980 con 0x7f695413e360 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.504+0000 7f695a000640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6934004d00 0x7f6934005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f6958c00640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 0x7f69340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f6959600640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934003970 0x7f6934003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f695a000640 1 -- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 msgr2=0x7f69340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f695a000640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f695413e360 0x7f69340047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f695a000640 1 -- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934003970 msgr2=0x7f6934003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:38.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f695a000640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934003970 0x7f6934003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f695a000640 1 -- 172.21.15.53:0/190950547 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6934003520 con 0x7f6934004d00 2024-01-16T15:27:38.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.505+0000 7f6959600640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934003970 0x7f6934003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.506+0000 7f695a000640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6934004d00 0x7f6934005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f694c00c190 tx=0x7f69340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b4e6d73a6ae6d9b server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.506+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c005ad0 con 0x7f6934004d00 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.506+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f694c005c50 con 0x7f6934004d00 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.506+0000 7f6953e00640 1 -- 172.21.15.53:0/190950547 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f694c02ca50 con 0x7f6934004d00 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695c000640 1 -- 172.21.15.53:0/190950547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6934004d00 msgr2=0x7f6934005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695c000640 1 --2- 172.21.15.53:0/190950547 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6934004d00 0x7f6934005180 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f694c00c190 tx=0x7f69340057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695a000640 1 -- 172.21.15.53:0/190950547 reap_dead start 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695c000640 1 -- 172.21.15.53:0/190950547 shutdown_connections 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695c000640 1 -- 172.21.15.53:0/190950547 >> 172.21.15.53:0/190950547 conn(0x7f695406a800 msgr2=0x7f695406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695c000640 1 -- 172.21.15.53:0/190950547 shutdown_connections 2024-01-16T15:27:38.508 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.507+0000 7f695c000640 1 -- 172.21.15.53:0/190950547 wait complete. 2024-01-16T15:27:38.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.509+0000 7f695c000640 1 Processor -- start 2024-01-16T15:27:38.509 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.509+0000 7f695c000640 1 -- start start 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f695c000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934004d00 0x7f695413d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f695c000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f695415d8a0 0x7f695415dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f695c000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6954150630 0x7f6954150ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f695c000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f695415e260 con 0x7f695415d8a0 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f695c000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6954150ff0 con 0x7f6934004d00 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f695c000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69541f62e0 con 0x7f6954150630 2024-01-16T15:27:38.511 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f6958c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f695415d8a0 0x7f695415dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f6958c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f695415d8a0 0x7f695415dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37306/0 (socket says 172.21.15.53:37306) 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.510+0000 7f6958c00640 1 -- 172.21.15.53:0/3793279999 learned_addr learned my addr 172.21.15.53:0/3793279999 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6959600640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934004d00 0x7f695413d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f695a000640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6954150630 0x7f6954150ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6958c00640 1 -- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6954150630 msgr2=0x7f6954150ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6958c00640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6954150630 0x7f6954150ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6958c00640 1 -- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934004d00 msgr2=0x7f695413d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6958c00640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934004d00 0x7f695413d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6958c00640 1 -- 172.21.15.53:0/3793279999 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695413dd80 con 0x7f695415d8a0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6959600640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6934004d00 0x7f695413d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6958c00640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f695415d8a0 0x7f695415dd20 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x7f693c011660 tx=0x7f693c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00f960 con 0x7f695415d8a0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.511+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f693c00fac0 con 0x7f695415d8a0 2024-01-16T15:27:38.512 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.512+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c00fe30 con 0x7f695415d8a0 2024-01-16T15:27:38.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.512+0000 7f695c000640 1 -- 172.21.15.53:0/3793279999 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69541f6530 con 0x7f695415d8a0 2024-01-16T15:27:38.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.512+0000 7f695c000640 1 -- 172.21.15.53:0/3793279999 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69541f6af0 con 0x7f695415d8a0 2024-01-16T15:27:38.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.513+0000 7f695c000640 1 -- 172.21.15.53:0/3793279999 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69540675d0 con 0x7f695415d8a0 2024-01-16T15:27:38.513 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.514+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f693c015460 con 0x7f695415d8a0 2024-01-16T15:27:38.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.515+0000 7f6951600640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f692c06c550 0x7f692c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:38.514 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.515+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f693c089010 con 0x7f695415d8a0 2024-01-16T15:27:38.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.515+0000 7f6959600640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f692c06c550 0x7f692c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:38.516 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.517+0000 7f6959600640 1 --2- 172.21.15.53:0/3793279999 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f692c06c550 0x7f692c06ea10 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f6944007520 tx=0x7f694400d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:38.518 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.519+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f693c055600 con 0x7f695415d8a0 2024-01-16T15:27:38.747 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.747+0000 7f695c000640 1 -- 172.21.15.53:0/3793279999 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f69540ce850 con 0x7f692c06c550 2024-01-16T15:27:38.760 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:38.761+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:38.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:38.761+0000 7f6951600640 1 -- 172.21.15.53:0/3793279999 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6944011040 con 0x7f692c06c550 2024-01-16T15:27:38.761 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:38.761 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:38.780 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:40.781 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:41.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.080+0000 7f0602a00640 1 Processor -- start 2024-01-16T15:27:41.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.081+0000 7f0602a00640 1 -- start start 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.083+0000 7f0602a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f05fc143ae0 0x7f05fc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.083+0000 7f0602a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f05fc0cf7d0 0x7f05fc15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.083+0000 7f0602a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05fc0cfc30 con 0x7f05fc15e1e0 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.083+0000 7f0602a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc0b1ee0 con 0x7f05fc0cf7d0 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.083+0000 7f0602a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05fc0b2780 con 0x7f05fc143ae0 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f0600a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f05fc15e1e0 0x7f05fc15a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:37808/0 (socket says 172.21.15.53:37808) 2024-01-16T15:27:41.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f0600a00640 1 -- 172.21.15.53:0/790363523 learned_addr learned my addr 172.21.15.53:0/790363523 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05f3e00640 1 --2- 172.21.15.53:0/790363523 >> v2:172.21.15.53:3300/0 conn(0x7f05fc0cf7d0 0x7f05fc15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05fb400640 1 -- 172.21.15.53:0/790363523 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847810903 0 0) 0x7f05fc0cfc30 con 0x7f05fc15e1e0 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05fb400640 1 -- 172.21.15.53:0/790363523 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05dc003520 con 0x7f05fc15e1e0 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05f3e00640 1 -- 172.21.15.53:0/790363523 >> v2:172.21.15.146:3300/0 conn(0x7f05fc143ae0 msgr2=0x7f05fc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05f3e00640 1 --2- 172.21.15.53:0/790363523 >> v2:172.21.15.146:3300/0 conn(0x7f05fc143ae0 0x7f05fc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05f3e00640 1 -- 172.21.15.53:0/790363523 >> v1:172.21.15.53:6789/0 conn(0x7f05fc15e1e0 legacy=0x7f05fc15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:41.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.084+0000 7f05f3e00640 1 -- 172.21.15.53:0/790363523 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05fc15e990 con 0x7f05fc0cf7d0 2024-01-16T15:27:41.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.086+0000 7f05f3e00640 1 --2- 172.21.15.53:0/790363523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc0cf7d0 0x7f05fc15dca0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0cfea0 tx=0x7f05e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3acf2cad0653c1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.087+0000 7f05fb400640 1 -- 172.21.15.53:0/790363523 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4031830 con 0x7f05fc0cf7d0 2024-01-16T15:27:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.087+0000 7f05fb400640 1 -- 172.21.15.53:0/790363523 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05e4031990 con 0x7f05fc0cf7d0 2024-01-16T15:27:41.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.087+0000 7f05fb400640 1 -- 172.21.15.53:0/790363523 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05e4031d20 con 0x7f05fc0cf7d0 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.088+0000 7f0602a00640 1 -- 172.21.15.53:0/790363523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc0cf7d0 msgr2=0x7f05fc15dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.088+0000 7f0602a00640 1 --2- 172.21.15.53:0/790363523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc0cf7d0 0x7f05fc15dca0 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0cfea0 tx=0x7f05e402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.089+0000 7f0602a00640 1 -- 172.21.15.53:0/790363523 shutdown_connections 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.089+0000 7f0602a00640 1 --2- 172.21.15.53:0/790363523 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc0cf7d0 0x7f05fc15dca0 secure :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7f05fc0cfea0 tx=0x7f05e402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.089+0000 7f0602a00640 1 --2- 172.21.15.53:0/790363523 >> v2:172.21.15.146:3300/0 conn(0x7f05fc143ae0 0x7f05fc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.089+0000 7f0602a00640 1 -- 172.21.15.53:0/790363523 >> 172.21.15.53:0/790363523 conn(0x7f05fc06a800 msgr2=0x7f05fc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.089+0000 7f0602a00640 1 -- 172.21.15.53:0/790363523 shutdown_connections 2024-01-16T15:27:41.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.089+0000 7f0602a00640 1 -- 172.21.15.53:0/790363523 wait complete. 2024-01-16T15:27:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.094+0000 7f0602a00640 1 Processor -- start 2024-01-16T15:27:41.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.094+0000 7f0602a00640 1 -- start start 2024-01-16T15:27:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0602a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc143ae0 0x7f05fc141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0602a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15e1e0 0x7f05fc142b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0602a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc1f0d50 0x7f05fc141bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0602a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1f15a0 con 0x7f05fc1f0d50 2024-01-16T15:27:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0602a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1f1880 con 0x7f05fc15e1e0 2024-01-16T15:27:41.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0602a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05fc1f1b60 con 0x7f05fc143ae0 2024-01-16T15:27:41.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.095+0000 7f0600a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc1f0d50 0x7f05fc141bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:41.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f0600a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc1f0d50 0x7f05fc141bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:37322/0 (socket says 172.21.15.53:37322) 2024-01-16T15:27:41.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f0600a00640 1 -- 172.21.15.53:0/1914617687 learned_addr learned my addr 172.21.15.53:0/1914617687 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:41.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f05fbe00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc143ae0 0x7f05fc141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:41.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f05f3e00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15e1e0 0x7f05fc142b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:41.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f0600a00640 1 -- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc143ae0 msgr2=0x7f05fc141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f0600a00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f05fc143ae0 0x7f05fc141600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f0600a00640 1 -- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15e1e0 msgr2=0x7f05fc142b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.096+0000 7f0600a00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f05fc15e1e0 0x7f05fc142b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.097+0000 7f0600a00640 1 -- 172.21.15.53:0/1914617687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05e4009400 con 0x7f05fc1f0d50 2024-01-16T15:27:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.097+0000 7f0600a00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f05fc1f0d50 0x7f05fc141bd0 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7f05f400cb40 tx=0x7f05f400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:41.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.098+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f4015e90 con 0x7f05fc1f0d50 2024-01-16T15:27:41.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.098+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05f400a770 con 0x7f05fc1f0d50 2024-01-16T15:27:41.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.098+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05f400ab40 con 0x7f05fc1f0d50 2024-01-16T15:27:41.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.098+0000 7f0602a00640 1 -- 172.21.15.53:0/1914617687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05fc1f1e40 con 0x7f05fc1f0d50 2024-01-16T15:27:41.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.098+0000 7f0602a00640 1 -- 172.21.15.53:0/1914617687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05fc1f1200 con 0x7f05fc1f0d50 2024-01-16T15:27:41.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.100+0000 7f0602a00640 1 -- 172.21.15.53:0/1914617687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05fc0675d0 con 0x7f05fc1f0d50 2024-01-16T15:27:41.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.102+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f05f401b070 con 0x7f05fc1f0d50 2024-01-16T15:27:41.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.104+0000 7f05f8c00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406c5d0 0x7f05d406ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:41.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.104+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f05f4091370 con 0x7f05fc1f0d50 2024-01-16T15:27:41.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.108+0000 7f05fbe00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406c5d0 0x7f05d406ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:41.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.113+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f05f405d8f0 con 0x7f05fc1f0d50 2024-01-16T15:27:41.116 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.117+0000 7f05fbe00640 1 --2- 172.21.15.53:0/1914617687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f05d406c5d0 0x7f05d406ea90 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f05ec009890 tx=0x7f05ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:41.430 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.431+0000 7f0602a00640 1 -- 172.21.15.53:0/1914617687 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05fc1f19f0 con 0x7f05d406c5d0 2024-01-16T15:27:41.446 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:41.446+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:41.449+0000 7f05f8c00640 1 -- 172.21.15.53:0/1914617687 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f05fc1f19f0 con 0x7f05d406c5d0 2024-01-16T15:27:41.448 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:41.448 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:41.467 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:43.468 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:43.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.707+0000 7f2f60a00640 1 Processor -- start 2024-01-16T15:27:43.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.707+0000 7f2f60a00640 1 -- start start 2024-01-16T15:27:43.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.708+0000 7f2f60a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f580d1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.708+0000 7f2f60a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2f5806ed10 0x7f2f5815acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.708+0000 7f2f60a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f580cb6f0 con 0x7f2f5806e930 2024-01-16T15:27:43.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f60a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f580d1f60 con 0x7f2f5806ed10 2024-01-16T15:27:43.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f60a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580cbf00 con 0x7f2f580cd2f0 2024-01-16T15:27:43.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f5e000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2f5806e930 0x7f2f580b3f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40134/0 (socket says 172.21.15.53:40134) 2024-01-16T15:27:43.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f5e000640 1 -- 172.21.15.53:0/54816203 learned_addr learned my addr 172.21.15.53:0/54816203 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f5ea00640 1 --2- 172.21.15.53:0/54816203 >> v2:172.21.15.146:3300/0 conn(0x7f2f5806ed10 0x7f2f5815acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f5d600640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f580d1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.709+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937308641 0 0) 0x7f2f580cb6f0 con 0x7f2f5806e930 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f38003520 con 0x7f2f5806e930 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3707240777 0 0) 0x7f2f38003520 con 0x7f2f5806e930 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 msgr2=0x7f2f580d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f580d1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 >> v2:172.21.15.146:3300/0 conn(0x7f2f5806ed10 msgr2=0x7f2f5815acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 --2- 172.21.15.53:0/54816203 >> v2:172.21.15.146:3300/0 conn(0x7f2f5806ed10 0x7f2f5815acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580cd840 con 0x7f2f5806e930 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2f500019f0 con 0x7f2f5806e930 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 >> v1:172.21.15.53:6789/0 conn(0x7f2f5806e930 legacy=0x7f2f580b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:43.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f38003970 0x7f2f38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f380047c0 0x7f2f38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580cbf00 con 0x7f2f380047c0 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3800b980 con 0x7f2f38003970 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3800baf0 con 0x7f2f580cd2f0 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.710+0000 7f2f5d600640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f380047c0 0x7f2f38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5e000640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f38003970 0x7f2f38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5d600640 1 -- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 msgr2=0x7f2f38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5ea00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5d600640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f38005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5d600640 1 -- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f38003970 msgr2=0x7f2f38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5d600640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f38003970 0x7f2f38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5d600640 1 -- 172.21.15.53:0/54816203 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38003520 con 0x7f2f380047c0 2024-01-16T15:27:43.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.711+0000 7f2f5ea00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f580cd2f0 0x7f2f38005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:43.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.712+0000 7f2f5d600640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f380047c0 0x7f2f38004c40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009ac0 tx=0x7f2f380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf155e6b2b811ec5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.712+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4002c6b0 con 0x7f2f380047c0 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.712+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4002c810 con 0x7f2f380047c0 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.712+0000 7f2f5cc00640 1 -- 172.21.15.53:0/54816203 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4002cba0 con 0x7f2f380047c0 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f60a00640 1 -- 172.21.15.53:0/54816203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f380047c0 msgr2=0x7f2f38004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f60a00640 1 --2- 172.21.15.53:0/54816203 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f380047c0 0x7f2f38004c40 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009ac0 tx=0x7f2f380057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f5ea00640 1 -- 172.21.15.53:0/54816203 reap_dead start 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f60a00640 1 -- 172.21.15.53:0/54816203 shutdown_connections 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f60a00640 1 -- 172.21.15.53:0/54816203 >> 172.21.15.53:0/54816203 conn(0x7f2f5806a3e0 msgr2=0x7f2f5806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f60a00640 1 -- 172.21.15.53:0/54816203 shutdown_connections 2024-01-16T15:27:43.713 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.713+0000 7f2f60a00640 1 -- 172.21.15.53:0/54816203 wait complete. 2024-01-16T15:27:43.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.715+0000 7f2f60a00640 1 Processor -- start 2024-01-16T15:27:43.714 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.715+0000 7f2f60a00640 1 -- start start 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.716+0000 7f2f60a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f380047c0 0x7f2f58150700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.716+0000 7f2f60a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f58158690 0x7f2f5814c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.716+0000 7f2f60a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f580c9830 0x7f2f580c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.716+0000 7f2f60a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580ca1f0 con 0x7f2f580c9830 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.716+0000 7f2f60a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f581f62b0 con 0x7f2f58158690 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.716+0000 7f2f60a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f58158b10 con 0x7f2f380047c0 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f580c9830 0x7f2f580c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f580c9830 0x7f2f580c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33062/0 (socket says 172.21.15.53:33062) 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 -- 172.21.15.53:0/372094766 learned_addr learned my addr 172.21.15.53:0/372094766 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5e000640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f380047c0 0x7f2f58150700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5d600640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f58158690 0x7f2f5814c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 -- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f380047c0 msgr2=0x7f2f58150700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2f380047c0 0x7f2f58150700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.716 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 -- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f58158690 msgr2=0x7f2f5814c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:43.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2f58158690 0x7f2f5814c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:43.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 -- 172.21.15.53:0/372094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f58158dc0 con 0x7f2f580c9830 2024-01-16T15:27:43.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.717+0000 7f2f5ea00640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2f580c9830 0x7f2f580c9cb0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f2f5806b3c0 tx=0x7f2f48010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:43.717 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.718+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4800e870 con 0x7f2f580c9830 2024-01-16T15:27:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.718+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f4800e9d0 con 0x7f2f580c9830 2024-01-16T15:27:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.718+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4800ed60 con 0x7f2f580c9830 2024-01-16T15:27:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.718+0000 7f2f60a00640 1 -- 172.21.15.53:0/372094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f581f6500 con 0x7f2f580c9830 2024-01-16T15:27:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.718+0000 7f2f60a00640 1 -- 172.21.15.53:0/372094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f581f6ac0 con 0x7f2f580c9830 2024-01-16T15:27:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.719+0000 7f2f60a00640 1 -- 172.21.15.53:0/372094766 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f580675d0 con 0x7f2f580c9830 2024-01-16T15:27:43.718 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.720+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2f4801c400 con 0x7f2f580c9830 2024-01-16T15:27:43.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.721+0000 7f2f56000640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3006c550 0x7f2f3006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:43.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.721+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2f48088b00 con 0x7f2f580c9830 2024-01-16T15:27:43.720 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.722+0000 7f2f5e000640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3006c550 0x7f2f3006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:43.721 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.723+0000 7f2f5e000640 1 --2- 172.21.15.53:0/372094766 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2f3006c550 0x7f2f3006ea10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f2f50001330 tx=0x7f2f50009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:43.724 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.726+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2f480553c0 con 0x7f2f580c9830 2024-01-16T15:27:43.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.963+0000 7f2f60a00640 1 -- 172.21.15.53:0/372094766 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "fs subvolume snapshot rm", "force": true, "vol_name": "cephfs", "sub_name": "sv1", "snap_name": "ss1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2f580ce950 con 0x7f2f3006c550 2024-01-16T15:27:43.975 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:27:43.976+0000 7f6135000640 -1 mgr.server reply reply (11) Resource temporarily unavailable snapshot 'ss1' has pending clones 2024-01-16T15:27:43.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:43.977+0000 7f2f56000640 1 -- 172.21.15.53:0/372094766 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: -11 snapshot 'ss1' has pending clones) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2f50001930 con 0x7f2f3006c550 2024-01-16T15:27:43.976 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:43.977 INFO:teuthology.orchestra.run.smithi053.stderr:Error EAGAIN: snapshot 'ss1' has pending clones 2024-01-16T15:27:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 11 2024-01-16T15:27:43.993 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for ... ERROR 2024-01-16T15:27:43.993 INFO:tasks.cephfs_test_runner:ERROR 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:27:43.994 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52246' 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:43.995 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:Ran 7 tests in 1625.987s 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:43.996 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 8224, in test_when_7_clones_are_cancelled 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: self._cancel_clones_and_assert(c) 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7986, in _cancel_clones_and_assert 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: self._cancel_clones(clones, check_status=True) 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7966, in _cancel_clones 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: self.run_ceph_cmd(f'fs clone cancel {v} {i}', 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: return self.mon_manager.run_cluster_cmd(**kwargs) 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/ceph_manager.py", line 1602, in run_cluster_cmd 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: return self.controller.run(**kwargs) 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/remote.py", line 523, in run 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: r = self._runner(client=self.ssh, name=self.shortname, **kwargs) 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 455, in run 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: r.wait() 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 161, in wait 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-01-16T15:27:43.997 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi053 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs clone cancel cephfs subvol_clone52246' 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:====================================================================== 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:ERROR: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:Test that both progress bars, one for ongoing clones and other for 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7597, in tearDown 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner: self._loop_cmd('fs subvolume snapshot rm --force --format ' 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs/test_volumes.py", line 7566, in _loop_cmd 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner: raise RuntimeError(f'This cmd failed 50 times: {cmdargs}') 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner:RuntimeError: This cmd failed 50 times: fs subvolume snapshot rm --force --format json cephfs sv1 ss1 2024-01-16T15:27:43.998 INFO:tasks.cephfs_test_runner: 2024-01-16T15:27:43.999 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:44.256 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6b7e28a498e34ae7baf49d210efb213d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_cd45576300487d997e5a85abed65500b9f5d143b/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_73a4532fc4d97adee0b985d4d42874ea351a1c86/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress) 2024-01-16T15:27:44.258 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-01-16T15:27:44.267 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-01-16T15:27:44.276 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-01-16T15:27:44.276 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:27:44.276 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:44.339 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:27:44.339 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:44.396 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:44.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.624+0000 7f1e7c200640 1 Processor -- start 2024-01-16T15:27:44.624 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.625+0000 7f1e7c200640 1 -- start start 2024-01-16T15:27:44.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.626+0000 7f1e7c200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1e7415eee0 0x7f1e7406a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:44.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.626+0000 7f1e7c200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e7406a920 0x7f1e740cce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:44.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.626+0000 7f1e7c200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e740725b0 con 0x7f1e7415b960 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.626+0000 7f1e7c200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7415f340 con 0x7f1e7406a920 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.626+0000 7f1e7c200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7406ad90 con 0x7f1e7415eee0 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.627+0000 7f1e7a200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1e7406a920 0x7f1e740cce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.627+0000 7f1e79800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1e7415b960 0x7f1e740715e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40140/0 (socket says 172.21.15.53:40140) 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.627+0000 7f1e79800640 1 -- 172.21.15.53:0/4022010421 learned_addr learned my addr 172.21.15.53:0/4022010421 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.627+0000 7f1e78e00640 1 --2- 172.21.15.53:0/4022010421 >> v2:172.21.15.146:3300/0 conn(0x7f1e7415eee0 0x7f1e7406a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.627+0000 7f1e7a200640 1 -- 172.21.15.53:0/4022010421 >> v2:172.21.15.146:3300/0 conn(0x7f1e7415eee0 msgr2=0x7f1e7406a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:44.627 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.627+0000 7f1e7a200640 1 --2- 172.21.15.53:0/4022010421 >> v2:172.21.15.146:3300/0 conn(0x7f1e7415eee0 0x7f1e7406a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.628+0000 7f1e7a200640 1 -- 172.21.15.53:0/4022010421 >> v1:172.21.15.53:6789/0 conn(0x7f1e7415b960 legacy=0x7f1e740715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:44.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.628+0000 7f1e73e00640 1 -- 172.21.15.53:0/4022010421 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439922997 0 0) 0x7f1e740725b0 con 0x7f1e7415b960 2024-01-16T15:27:44.628 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.628+0000 7f1e7a200640 1 -- 172.21.15.53:0/4022010421 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7415fe00 con 0x7f1e7406a920 2024-01-16T15:27:44.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.629+0000 7f1e7a200640 1 --2- 172.21.15.53:0/4022010421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7406a920 0x7f1e740cce80 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f1e7406c4d0 tx=0x7f1e6c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fb7825692d39b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:44.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.629+0000 7f1e73e00640 1 -- 172.21.15.53:0/4022010421 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c02cc80 con 0x7f1e7406a920 2024-01-16T15:27:44.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.629+0000 7f1e73e00640 1 -- 172.21.15.53:0/4022010421 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6c02cde0 con 0x7f1e7406a920 2024-01-16T15:27:44.629 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.629+0000 7f1e73e00640 1 -- 172.21.15.53:0/4022010421 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6c034660 con 0x7f1e7406a920 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 -- 172.21.15.53:0/4022010421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7406a920 msgr2=0x7f1e740cce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 --2- 172.21.15.53:0/4022010421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7406a920 0x7f1e740cce80 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f1e7406c4d0 tx=0x7f1e6c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 -- 172.21.15.53:0/4022010421 shutdown_connections 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 --2- 172.21.15.53:0/4022010421 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7406a920 0x7f1e740cce80 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 --2- 172.21.15.53:0/4022010421 >> v2:172.21.15.146:3300/0 conn(0x7f1e7415eee0 0x7f1e7406a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 -- 172.21.15.53:0/4022010421 >> 172.21.15.53:0/4022010421 conn(0x7f1e740cff90 msgr2=0x7f1e740d0370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 -- 172.21.15.53:0/4022010421 shutdown_connections 2024-01-16T15:27:44.630 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.630+0000 7f1e7c200640 1 -- 172.21.15.53:0/4022010421 wait complete. 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.632+0000 7f1e7c200640 1 Processor -- start 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.632+0000 7f1e7c200640 1 -- start start 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e7c200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e7406a920 0x7f1e74150c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e7c200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 0x7f1e7413abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e7c200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e7415eee0 0x7f1e7413c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e7c200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7413b530 con 0x7f1e7415b960 2024-01-16T15:27:44.633 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e7c200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7413b810 con 0x7f1e7406a920 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e7c200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7413baf0 con 0x7f1e7415eee0 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e78e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 0x7f1e7413abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e78e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 0x7f1e7413abc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33076/0 (socket says 172.21.15.53:33076) 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e78e00640 1 -- 172.21.15.53:0/2383180762 learned_addr learned my addr 172.21.15.53:0/2383180762 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e79800640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e7406a920 0x7f1e74150c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e78e00640 1 -- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e7415eee0 msgr2=0x7f1e7413c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e78e00640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e7415eee0 0x7f1e7413c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.633+0000 7f1e78e00640 1 -- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e7406a920 msgr2=0x7f1e74150c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:44.634 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e78e00640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e7406a920 0x7f1e74150c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e78e00640 1 -- 172.21.15.53:0/2383180762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c009a70 con 0x7f1e7415b960 2024-01-16T15:27:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e78e00640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 0x7f1e7413abc0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c00d5f0 tx=0x7f1e5c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c0125f0 con 0x7f1e7415b960 2024-01-16T15:27:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e5c012750 con 0x7f1e7415b960 2024-01-16T15:27:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e5c012b00 con 0x7f1e7415b960 2024-01-16T15:27:44.635 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7413bdd0 con 0x7f1e7415b960 2024-01-16T15:27:44.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.634+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e74151170 con 0x7f1e7415b960 2024-01-16T15:27:44.636 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.635+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e740675d0 con 0x7f1e7415b960 2024-01-16T15:27:44.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.636+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1e5c012c60 con 0x7f1e7415b960 2024-01-16T15:27:44.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.637+0000 7f1e71600640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4c06c550 0x7f1e4c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:44.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.637+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1e5c089020 con 0x7f1e7415b960 2024-01-16T15:27:44.637 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.638+0000 7f1e79800640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4c06c550 0x7f1e4c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:44.638 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.639+0000 7f1e79800640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4c06c550 0x7f1e4c06ea10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1e64007fd0 tx=0x7f1e64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:44.641 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.643+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1e5c0555c0 con 0x7f1e7415b960 2024-01-16T15:27:44.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.837+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e740713c0 con 0x7f1e7415b960 2024-01-16T15:27:44.839 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.840+0000 7f1e71600640 1 -- 172.21.15.53:0/2383180762 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f1e5c058be0 con 0x7f1e7415b960 2024-01-16T15:27:44.839 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:44.840 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":112,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:42.134132+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.24","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:44.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4c06c550 msgr2=0x7f1e4c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:44.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4c06c550 0x7f1e4c06ea10 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f1e64007fd0 tx=0x7f1e64009040 comp rx=0 tx=0).stop 2024-01-16T15:27:44.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 msgr2=0x7f1e7413abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:44.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 0x7f1e7413abc0 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f1e5c00d5f0 tx=0x7f1e5c00dab0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 shutdown_connections 2024-01-16T15:27:44.842 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1e4c06c550 0x7f1e4c06ea10 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1e7415eee0 0x7f1e7413c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1e7415b960 0x7f1e7413abc0 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 --2- 172.21.15.53:0/2383180762 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1e7406a920 0x7f1e74150c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:44.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 >> 172.21.15.53:0/2383180762 conn(0x7f1e740cff90 msgr2=0x7f1e741550c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:44.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 shutdown_connections 2024-01-16T15:27:44.843 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:44.843+0000 7f1e7c200640 1 -- 172.21.15.53:0/2383180762 wait complete. 2024-01-16T15:27:44.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:19:21.949075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:19:23.088954+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:44.862 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:45.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.088+0000 7fd153a00640 1 Processor -- start 2024-01-16T15:27:45.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.088+0000 7fd153a00640 1 -- start start 2024-01-16T15:27:45.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.090+0000 7fd153a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.090+0000 7fd153a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd14c0cf7d0 0x7fd14c15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.090+0000 7fd153a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd14c0cfc30 con 0x7fd14c15e1e0 2024-01-16T15:27:45.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.090+0000 7fd153a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd14c0b1ee0 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.090+0000 7fd153a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c0b2780 con 0x7fd14c143ae0 2024-01-16T15:27:45.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.091+0000 7fd151000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.091+0000 7fd151000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c0cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:33342/0 (socket says 172.21.15.53:33342) 2024-01-16T15:27:45.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.091+0000 7fd151000640 1 -- 172.21.15.53:0/2577885619 learned_addr learned my addr 172.21.15.53:0/2577885619 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:45.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.091+0000 7fd14be00640 1 --2- 172.21.15.53:0/2577885619 >> v2:172.21.15.146:3300/0 conn(0x7fd14c0cf7d0 0x7fd14c15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.092+0000 7fd151000640 1 -- 172.21.15.53:0/2577885619 >> v2:172.21.15.146:3300/0 conn(0x7fd14c0cf7d0 msgr2=0x7fd14c15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:45.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.092+0000 7fd151000640 1 --2- 172.21.15.53:0/2577885619 >> v2:172.21.15.146:3300/0 conn(0x7fd14c0cf7d0 0x7fd14c15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.092+0000 7fd151000640 1 -- 172.21.15.53:0/2577885619 >> v1:172.21.15.146:6789/0 conn(0x7fd14c15e1e0 legacy=0x7fd14c15a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.092+0000 7fd151000640 1 -- 172.21.15.53:0/2577885619 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14c15e990 con 0x7fd14c143ae0 2024-01-16T15:27:45.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.092+0000 7fd14be00640 1 --2- 172.21.15.53:0/2577885619 >> v2:172.21.15.146:3300/0 conn(0x7fd14c0cf7d0 0x7fd14c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:27:45.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.093+0000 7fd151000640 1 --2- 172.21.15.53:0/2577885619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c0cf290 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009870 tx=0x7fd13c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=901341237fe85e17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:45.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd14b400640 1 -- 172.21.15.53:0/2577885619 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c030840 con 0x7fd14c143ae0 2024-01-16T15:27:45.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd14b400640 1 -- 172.21.15.53:0/2577885619 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c0309a0 con 0x7fd14c143ae0 2024-01-16T15:27:45.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd14b400640 1 -- 172.21.15.53:0/2577885619 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c030d30 con 0x7fd14c143ae0 2024-01-16T15:27:45.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 -- 172.21.15.53:0/2577885619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 msgr2=0x7fd14c0cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 --2- 172.21.15.53:0/2577885619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c0cf290 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd13c009870 tx=0x7fd13c02b980 comp rx=0 tx=0).stop 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 -- 172.21.15.53:0/2577885619 shutdown_connections 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 --2- 172.21.15.53:0/2577885619 >> v2:172.21.15.146:3300/0 conn(0x7fd14c0cf7d0 0x7fd14c15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 --2- 172.21.15.53:0/2577885619 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c0cf290 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 -- 172.21.15.53:0/2577885619 >> 172.21.15.53:0/2577885619 conn(0x7fd14c06a800 msgr2=0x7fd14c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 -- 172.21.15.53:0/2577885619 shutdown_connections 2024-01-16T15:27:45.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.094+0000 7fd153a00640 1 -- 172.21.15.53:0/2577885619 wait complete. 2024-01-16T15:27:45.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.096+0000 7fd153a00640 1 Processor -- start 2024-01-16T15:27:45.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.097+0000 7fd153a00640 1 -- start start 2024-01-16T15:27:45.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd153a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 0x7fd14c154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd153a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c158bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd153a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd14c15e1e0 0x7fd14c0cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd153a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c159600 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd153a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c1598e0 con 0x7fd14c15e1e0 2024-01-16T15:27:45.097 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd153a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14c159bc0 con 0x7fd14c143ae0 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd151000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 0x7fd14c154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd151000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 0x7fd14c154e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33086/0 (socket says 172.21.15.53:33086) 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd151000640 1 -- 172.21.15.53:0/462196531 learned_addr learned my addr 172.21.15.53:0/462196531 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd14be00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c158bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.098+0000 7fd151a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd14c15e1e0 0x7fd14c0cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd151000640 1 -- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 msgr2=0x7fd14c158bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd151000640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c158bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd151000640 1 -- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd14c15e1e0 msgr2=0x7fd14c0cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd151000640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd14c15e1e0 0x7fd14c0cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd151000640 1 -- 172.21.15.53:0/462196531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd134009400 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd151000640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 0x7fd14c154e70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c039c90 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd13c039df0 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.100 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.099+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13c0094d0 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.100+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd13c043640 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.100+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd14c0aeac0 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.101 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.101+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd14c06ad20 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.102+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd13c0437a0 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.103+0000 7fd148c00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd12406c580 0x7fd12406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.103+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd13c03e070 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.102 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.103+0000 7fd14be00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd12406c580 0x7fd12406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.103 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.104+0000 7fd14be00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd12406c580 0x7fd12406ea40 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd14c15d010 tx=0x7fd134008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:45.107 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.109+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd13c078130 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.305 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.306+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd14c0675d0 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.306 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.307+0000 7fd148c00640 1 -- 172.21.15.53:0/462196531 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd13c034020 con 0x7fd14c0cf7d0 2024-01-16T15:27:45.306 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:27:45.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd12406c580 msgr2=0x7fd12406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd12406c580 0x7fd12406ea40 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fd14c15d010 tx=0x7fd134008040 comp rx=0 tx=0).stop 2024-01-16T15:27:45.309 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 msgr2=0x7fd14c154e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 0x7fd14c154e70 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7fd13c000c00 tx=0x7fd13c004730 comp rx=0 tx=0).stop 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 shutdown_connections 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd12406c580 0x7fd12406ea40 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd14c15e1e0 0x7fd14c0cd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd14c143ae0 0x7fd14c158bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 --2- 172.21.15.53:0/462196531 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd14c0cf7d0 0x7fd14c154e70 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 >> 172.21.15.53:0/462196531 conn(0x7fd14c06a800 msgr2=0x7fd14c0702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.310+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 shutdown_connections 2024-01-16T15:27:45.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.311+0000 7fd153a00640 1 -- 172.21.15.53:0/462196531 wait complete. 2024-01-16T15:27:45.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:45.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.554+0000 7f245be00640 1 Processor -- start 2024-01-16T15:27:45.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.554+0000 7f245be00640 1 -- start start 2024-01-16T15:27:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.556+0000 7f245be00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f245c0adf20 0x7f245c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.555 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.556+0000 7f245be00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245c15f4e0 con 0x7f245c15bf10 2024-01-16T15:27:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.556+0000 7f245be00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245c071130 con 0x7f245c0ccb50 2024-01-16T15:27:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.556+0000 7f245be00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245c15f970 con 0x7f245c0adf20 2024-01-16T15:27:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.556+0000 7f2453e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f245c15bf10 0x7f245c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40154/0 (socket says 172.21.15.53:40154) 2024-01-16T15:27:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.556+0000 7f2453e00640 1 -- 172.21.15.53:0/1973031072 learned_addr learned my addr 172.21.15.53:0/1973031072 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245aa00640 1 --2- 172.21.15.53:0/1973031072 >> v2:172.21.15.146:3300/0 conn(0x7f245c0adf20 0x7f245c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677647841 0 0) 0x7f245c15f4e0 con 0x7f245c15bf10 2024-01-16T15:27:45.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f243c003520 con 0x7f245c15bf10 2024-01-16T15:27:45.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2152816925 0 0) 0x7f243c003520 con 0x7f245c15bf10 2024-01-16T15:27:45.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 >> v2:172.21.15.146:3300/0 conn(0x7f245c0adf20 msgr2=0x7f245c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:45.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 --2- 172.21.15.53:0/1973031072 >> v2:172.21.15.146:3300/0 conn(0x7f245c0adf20 0x7f245c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 >> v1:172.21.15.146:6789/0 conn(0x7f245c0ccb50 legacy=0x7f245c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c15c390 con 0x7f245c15bf10 2024-01-16T15:27:45.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2444001670 con 0x7f245c15bf10 2024-01-16T15:27:45.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 >> v1:172.21.15.53:6789/0 conn(0x7f245c15bf10 legacy=0x7f245c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.557+0000 7f245a000640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243c0039c0 0x7f243c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245a000640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243c0047c0 0x7f243c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245a000640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f243c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245c15f970 con 0x7f243c0051d0 2024-01-16T15:27:45.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243c00c120 con 0x7f243c0039c0 2024-01-16T15:27:45.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f243c00c2e0 con 0x7f243c0047c0 2024-01-16T15:27:45.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245b400640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f243c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f2453e00640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243c0047c0 0x7f243c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245b400640 1 -- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243c0047c0 msgr2=0x7f243c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:45.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245aa00640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243c0039c0 0x7f243c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245b400640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f243c0047c0 0x7f243c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245b400640 1 -- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243c0039c0 msgr2=0x7f243c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:45.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245b400640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f243c0039c0 0x7f243c003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.558+0000 7f245b400640 1 -- 172.21.15.53:0/1973031072 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f243c003520 con 0x7f243c0051d0 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.559+0000 7f245b400640 1 -- 172.21.15.53:0/1973031072 reap_dead start 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.559+0000 7f245b400640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f243c00b980 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f2454001e20 tx=0x7f243c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48607b844b571ca server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454003b70 con 0x7f243c0051d0 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454003cd0 con 0x7f243c0051d0 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245a000640 1 -- 172.21.15.53:0/1973031072 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2454005310 con 0x7f243c0051d0 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245be00640 1 -- 172.21.15.53:0/1973031072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 msgr2=0x7f243c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245be00640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f243c00b980 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f2454001e20 tx=0x7f243c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245be00640 1 -- 172.21.15.53:0/1973031072 shutdown_connections 2024-01-16T15:27:45.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245be00640 1 --2- 172.21.15.53:0/1973031072 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f243c00b980 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.560+0000 7f245be00640 1 -- 172.21.15.53:0/1973031072 >> 172.21.15.53:0/1973031072 conn(0x7f245c06a7d0 msgr2=0x7f245c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:45.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.561+0000 7f245be00640 1 -- 172.21.15.53:0/1973031072 shutdown_connections 2024-01-16T15:27:45.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.561+0000 7f245be00640 1 -- 172.21.15.53:0/1973031072 wait complete. 2024-01-16T15:27:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.563+0000 7f245be00640 1 Processor -- start 2024-01-16T15:27:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.563+0000 7f245be00640 1 -- start start 2024-01-16T15:27:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245be00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f245c0c90c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245be00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f245c0accf0 0x7f245c15a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245be00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f245c0ad170 0x7f245c15a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245be00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245c0c4ed0 con 0x7f243c0051d0 2024-01-16T15:27:45.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245be00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f245c0c9600 con 0x7f245c0ad170 2024-01-16T15:27:45.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245be00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245c1f71b0 con 0x7f245c0accf0 2024-01-16T15:27:45.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f245c0c90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245aa00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f245c0c90c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33104/0 (socket says 172.21.15.53:33104) 2024-01-16T15:27:45.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f245aa00640 1 -- 172.21.15.53:0/2798855351 learned_addr learned my addr 172.21.15.53:0/2798855351 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:45.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245aa00640 1 -- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f245c0accf0 msgr2=0x7f245c15a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.564+0000 7f2453e00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f245c0accf0 0x7f245c15a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245b400640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f245c0ad170 0x7f245c15a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245aa00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f245c0accf0 0x7f245c15a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245aa00640 1 -- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f245c0ad170 msgr2=0x7f245c15a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245aa00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f245c0ad170 0x7f245c15a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245aa00640 1 -- 172.21.15.53:0/2798855351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c0c5190 con 0x7f243c0051d0 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f2453e00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f245c0accf0 0x7f245c15a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:45.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245b400640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f245c0ad170 0x7f245c15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:45.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245aa00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f245c0c90c0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f244c010860 tx=0x7f244c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:45.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c009610 con 0x7f243c0051d0 2024-01-16T15:27:45.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c00eb60 con 0x7f243c0051d0 2024-01-16T15:27:45.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f244c01c400 con 0x7f243c0051d0 2024-01-16T15:27:45.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f245c0c5470 con 0x7f243c0051d0 2024-01-16T15:27:45.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.565+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f245c1f66a0 con 0x7f243c0051d0 2024-01-16T15:27:45.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.567+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245c067450 con 0x7f243c0051d0 2024-01-16T15:27:45.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.568+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f244c01c560 con 0x7f243c0051d0 2024-01-16T15:27:45.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.568+0000 7f2452a00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f243406c550 0x7f243406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:45.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.569+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f244c088d30 con 0x7f243c0051d0 2024-01-16T15:27:45.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.569+0000 7f2453e00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f243406c550 0x7f243406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:45.574 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.570+0000 7f2453e00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f243406c550 0x7f243406ea10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2444001200 tx=0x7f2444009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:45.575 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.573+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f244c055700 con 0x7f243c0051d0 2024-01-16T15:27:45.767 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.768+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f245c06ac20 con 0x7f243c0051d0 2024-01-16T15:27:45.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.769+0000 7f2452a00640 1 -- 172.21.15.53:0/2798855351 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f244c058d20 con 0x7f243c0051d0 2024-01-16T15:27:45.768 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:27:45.771 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f243406c550 msgr2=0x7f243406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f243406c550 0x7f243406ea10 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f2444001200 tx=0x7f2444009040 comp rx=0 tx=0).stop 2024-01-16T15:27:45.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 msgr2=0x7f245c0c90c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:45.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f245c0c90c0 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f244c010860 tx=0x7f244c010d20 comp rx=0 tx=0).stop 2024-01-16T15:27:45.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 shutdown_connections 2024-01-16T15:27:45.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f243406c550 0x7f243406ea10 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f245c0ad170 0x7f245c15a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f245c0accf0 0x7f245c15a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 --2- 172.21.15.53:0/2798855351 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f243c0051d0 0x7f245c0c90c0 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:45.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 >> 172.21.15.53:0/2798855351 conn(0x7f245c06a7d0 msgr2=0x7f245c144560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:45.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 shutdown_connections 2024-01-16T15:27:45.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:45.772+0000 7f245be00640 1 -- 172.21.15.53:0/2798855351 wait complete. 2024-01-16T15:27:45.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:46.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.021+0000 7f5bc1400640 1 Processor -- start 2024-01-16T15:27:46.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.021+0000 7f5bc1400640 1 -- start start 2024-01-16T15:27:46.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bc1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc14b2c0 0x7f5bbc0cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bc1400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5bbc0cf7d0 0x7f5bbc070960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bc1400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bbc0cfbf0 con 0x7f5bbc070ea0 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bc1400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bbc14b6a0 con 0x7f5bbc0cf7d0 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bc1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc071690 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bb3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5bbc0cf7d0 0x7f5bbc070960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bb3e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f5bbc0cf7d0 0x7f5bbc070960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33116/0 (socket says 172.21.15.53:33116) 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bb3e00640 1 -- 172.21.15.53:0/2892508794 learned_addr learned my addr 172.21.15.53:0/2892508794 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:46.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.023+0000 7f5bbb200640 1 --1- 172.21.15.53:0/2892508794 >> v1:172.21.15.53:6789/0 conn(0x7f5bbc070ea0 0x7f5bbc0b1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40168/0 (socket says 172.21.15.53:40168) 2024-01-16T15:27:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.024+0000 7f5bba800640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc14b2c0 0x7f5bbc0cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.024+0000 7f5bb3e00640 1 -- 172.21.15.53:0/2892508794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc14b2c0 msgr2=0x7f5bbc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.024+0000 7f5bb3e00640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc14b2c0 0x7f5bbc0cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.024+0000 7f5bb3e00640 1 -- 172.21.15.53:0/2892508794 >> v1:172.21.15.53:6789/0 conn(0x7f5bbc070ea0 legacy=0x7f5bbc0b1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.024+0000 7f5bb3e00640 1 -- 172.21.15.53:0/2892508794 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc071970 con 0x7f5bbc0cf7d0 2024-01-16T15:27:46.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.024+0000 7f5bba800640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc14b2c0 0x7f5bbc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.025+0000 7f5bb3e00640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc0cf7d0 0x7f5bbc070960 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009870 tx=0x7f5ba402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8f7ccaf4562cabf server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.025+0000 7f5bb9e00640 1 -- 172.21.15.53:0/2892508794 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4030840 con 0x7f5bbc0cf7d0 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.025+0000 7f5bb9e00640 1 -- 172.21.15.53:0/2892508794 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba40309a0 con 0x7f5bbc0cf7d0 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.025+0000 7f5bb9e00640 1 -- 172.21.15.53:0/2892508794 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4030d30 con 0x7f5bbc0cf7d0 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 -- 172.21.15.53:0/2892508794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc0cf7d0 msgr2=0x7f5bbc070960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc0cf7d0 0x7f5bbc070960 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f5ba4009870 tx=0x7f5ba402b980 comp rx=0 tx=0).stop 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 -- 172.21.15.53:0/2892508794 shutdown_connections 2024-01-16T15:27:46.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc0cf7d0 0x7f5bbc070960 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 --2- 172.21.15.53:0/2892508794 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc14b2c0 0x7f5bbc0cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 -- 172.21.15.53:0/2892508794 >> 172.21.15.53:0/2892508794 conn(0x7f5bbc06a800 msgr2=0x7f5bbc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:46.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 -- 172.21.15.53:0/2892508794 shutdown_connections 2024-01-16T15:27:46.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.026+0000 7f5bc1400640 1 -- 172.21.15.53:0/2892508794 wait complete. 2024-01-16T15:27:46.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.028+0000 7f5bc1400640 1 Processor -- start 2024-01-16T15:27:46.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.028+0000 7f5bc1400640 1 -- start start 2024-01-16T15:27:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bc1400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc070ea0 0x7f5bbc1f6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bc1400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0cf7d0 0x7f5bbc162880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bc1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 0x7f5bbc145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bc1400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc162dc0 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bc1400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc1f7c60 con 0x7f5bbc070ea0 2024-01-16T15:27:46.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bc1400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc153a90 con 0x7f5bbc0cf7d0 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.029+0000 7f5bbb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 0x7f5bbc145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 0x7f5bbc145100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33128/0 (socket says 172.21.15.53:33128) 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 -- 172.21.15.53:0/3621266837 learned_addr learned my addr 172.21.15.53:0/3621266837 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bb3e00640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0cf7d0 0x7f5bbc162880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bba800640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc070ea0 0x7f5bbc1f6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 -- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0cf7d0 msgr2=0x7f5bbc162880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0cf7d0 0x7f5bbc162880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 -- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc070ea0 msgr2=0x7f5bbc1f6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc070ea0 0x7f5bbc1f6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 -- 172.21.15.53:0/3621266837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba40094d0 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.030+0000 7f5bbb200640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 0x7f5bbc145100 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400c780 tx=0x7f5bb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.031+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb400ce90 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.031+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb400a640 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.031+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bbc145640 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.031+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb400a9f0 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.031+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bbc1f76a0 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.032+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bbc0675d0 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.033+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f5bb400ab50 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.034+0000 7f5bb2a00640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9406c550 0x7f5b9406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.034+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5bb4088280 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.035+0000 7f5bba800640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9406c550 0x7f5b9406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.036+0000 7f5bba800640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9406c550 0x7f5b9406ea10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00a950 tx=0x7f5bac008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:46.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.038+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5bb4054870 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.232+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5bbc153c00 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.233+0000 7f5bb2a00640 1 -- 172.21.15.53:0/3621266837 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5bb4017070 con 0x7f5bbc14b2c0 2024-01-16T15:27:46.232 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:27:46.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9406c550 msgr2=0x7f5b9406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9406c550 0x7f5b9406ea10 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5bac00a950 tx=0x7f5bac008040 comp rx=0 tx=0).stop 2024-01-16T15:27:46.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 msgr2=0x7f5bbc145100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 0x7f5bbc145100 secure :-1 s=READY pgs=1973 cs=0 l=1 rev1=1 crypto rx=0x7f5bb400c780 tx=0x7f5bb400cc40 comp rx=0 tx=0).stop 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 shutdown_connections 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f5b9406c550 0x7f5b9406ea10 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5bbc14b2c0 0x7f5bbc145100 unknown :-1 s=CLOSED pgs=1973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5bbc0cf7d0 0x7f5bbc162880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 --2- 172.21.15.53:0/3621266837 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5bbc070ea0 0x7f5bbc1f6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 >> 172.21.15.53:0/3621266837 conn(0x7f5bbc06a800 msgr2=0x7f5bbc0ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.236+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 shutdown_connections 2024-01-16T15:27:46.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.237+0000 7f5bc1400640 1 -- 172.21.15.53:0/3621266837 wait complete. 2024-01-16T15:27:46.253 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:46.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.479+0000 7f194a000640 1 Processor -- start 2024-01-16T15:27:46.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.480+0000 7f194a000640 1 -- start start 2024-01-16T15:27:46.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.481+0000 7f194a000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1944160920 0x7f19440b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.481+0000 7f194a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19440cf9f0 0x7f19440cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.481+0000 7f194a000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1944161010 0x7f19441614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.481+0000 7f194a000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19440d03b0 con 0x7f1944161010 2024-01-16T15:27:46.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.481+0000 7f194a000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19440d0a10 con 0x7f1944160920 2024-01-16T15:27:46.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.481+0000 7f194a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19440cf080 con 0x7f19440cf9f0 2024-01-16T15:27:46.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1944161010 0x7f19441614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1944161010 0x7f19441614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33142/0 (socket says 172.21.15.53:33142) 2024-01-16T15:27:46.481 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 -- 172.21.15.53:0/492882183 learned_addr learned my addr 172.21.15.53:0/492882183 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:46.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f193be00640 1 --2- 172.21.15.53:0/492882183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19440cf9f0 0x7f19440cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1948c00640 1 --2- 172.21.15.53:0/492882183 >> v2:172.21.15.146:3300/0 conn(0x7f1944160920 0x7f19440b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 -- 172.21.15.53:0/492882183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19440cf9f0 msgr2=0x7f19440cfe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:46.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 --2- 172.21.15.53:0/492882183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19440cf9f0 0x7f19440cfe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 -- 172.21.15.53:0/492882183 >> v2:172.21.15.146:3300/0 conn(0x7f1944160920 msgr2=0x7f19440b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 --2- 172.21.15.53:0/492882183 >> v2:172.21.15.146:3300/0 conn(0x7f1944160920 0x7f19440b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.482+0000 7f1949600640 1 -- 172.21.15.53:0/492882183 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19440c6be0 con 0x7f1944161010 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.483+0000 7f1949600640 1 --2- 172.21.15.53:0/492882183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1944161010 0x7f19441614a0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f194408c060 tx=0x7f193c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8cc3b086d6414f server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f1943e00640 1 -- 172.21.15.53:0/492882183 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c02ca70 con 0x7f1944161010 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f1943e00640 1 -- 172.21.15.53:0/492882183 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193c02cbd0 con 0x7f1944161010 2024-01-16T15:27:46.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f1943e00640 1 -- 172.21.15.53:0/492882183 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193c0343f0 con 0x7f1944161010 2024-01-16T15:27:46.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 -- 172.21.15.53:0/492882183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1944161010 msgr2=0x7f19441614a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 --2- 172.21.15.53:0/492882183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1944161010 0x7f19441614a0 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f194408c060 tx=0x7f193c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 -- 172.21.15.53:0/492882183 shutdown_connections 2024-01-16T15:27:46.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 --2- 172.21.15.53:0/492882183 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1944161010 0x7f19441614a0 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 --2- 172.21.15.53:0/492882183 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f19440cf9f0 0x7f19440cfe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 --2- 172.21.15.53:0/492882183 >> v2:172.21.15.146:3300/0 conn(0x7f1944160920 0x7f19440b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 -- 172.21.15.53:0/492882183 >> 172.21.15.53:0/492882183 conn(0x7f19440d3090 msgr2=0x7f19440d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:46.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 -- 172.21.15.53:0/492882183 shutdown_connections 2024-01-16T15:27:46.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.484+0000 7f194a000640 1 -- 172.21.15.53:0/492882183 wait complete. 2024-01-16T15:27:46.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.486+0000 7f194a000640 1 Processor -- start 2024-01-16T15:27:46.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.486+0000 7f194a000640 1 -- start start 2024-01-16T15:27:46.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.487+0000 7f194a000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 0x7f194414d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.487+0000 7f194a000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1944160920 0x7f19440caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.487+0000 7f194a000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1944161010 0x7f19440c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.487+0000 7f194a000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194414da10 con 0x7f19440cf9f0 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.487+0000 7f194a000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19440b7b90 con 0x7f1944161010 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.487+0000 7f194a000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19440b7e70 con 0x7f1944160920 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 0x7f194414d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 0x7f194414d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33150/0 (socket says 172.21.15.53:33150) 2024-01-16T15:27:46.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 -- 172.21.15.53:0/239298021 learned_addr learned my addr 172.21.15.53:0/239298021 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f193be00640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1944160920 0x7f19440caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1949600640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1944161010 0x7f19440c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 -- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1944160920 msgr2=0x7f19440caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1944160920 0x7f19440caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 -- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1944161010 msgr2=0x7f19440c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1944161010 0x7f19440c8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.488+0000 7f1948c00640 1 -- 172.21.15.53:0/239298021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f193c009b30 con 0x7f19440cf9f0 2024-01-16T15:27:46.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.489+0000 7f1948c00640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 0x7f194414d4d0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f193400d800 tx=0x7f193400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:46.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.489+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193400b6c0 con 0x7f19440cf9f0 2024-01-16T15:27:46.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.489+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f193400b820 con 0x7f19440cf9f0 2024-01-16T15:27:46.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.489+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19440b8150 con 0x7f19440cf9f0 2024-01-16T15:27:46.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.489+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f193400bbb0 con 0x7f19440cf9f0 2024-01-16T15:27:46.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.489+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19440ca380 con 0x7f19440cf9f0 2024-01-16T15:27:46.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.490+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1944067450 con 0x7f19440cf9f0 2024-01-16T15:27:46.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.491+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f193400bd10 con 0x7f19440cf9f0 2024-01-16T15:27:46.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.492+0000 7f1941600640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191c06c550 0x7f191c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:46.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.492+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f19340890f0 con 0x7f19440cf9f0 2024-01-16T15:27:46.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.493+0000 7f193be00640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191c06c550 0x7f191c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:46.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.494+0000 7f193be00640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191c06c550 0x7f191c06ea10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f192c006fd0 tx=0x7f192c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:46.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.496+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1934055630 con 0x7f19440cf9f0 2024-01-16T15:27:46.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.709+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f194408c7e0 con 0x7f19440cf9f0 2024-01-16T15:27:46.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.709+0000 7f1941600640 1 -- 172.21.15.53:0/239298021 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v112) v1 ==== 80+0+0 (secure 0 0 0) 0x7f1934058c50 con 0x7f19440cf9f0 2024-01-16T15:27:46.708 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:27:46.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191c06c550 msgr2=0x7f191c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191c06c550 0x7f191c06ea10 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f192c006fd0 tx=0x7f192c008040 comp rx=0 tx=0).stop 2024-01-16T15:27:46.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 msgr2=0x7f194414d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:46.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 0x7f194414d4d0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f193400d800 tx=0x7f193400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 shutdown_connections 2024-01-16T15:27:46.711 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f191c06c550 0x7f191c06ea10 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1944161010 0x7f19440c8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1944160920 0x7f19440caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 --2- 172.21.15.53:0/239298021 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f19440cf9f0 0x7f194414d4d0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:46.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 >> 172.21.15.53:0/239298021 conn(0x7f19440d3090 msgr2=0x7f194406c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:46.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 shutdown_connections 2024-01-16T15:27:46.712 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:46.712+0000 7f194a000640 1 -- 172.21.15.53:0/239298021 wait complete. 2024-01-16T15:27:46.730 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:27:46.730 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:46.784 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:47.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.010+0000 7f2ab5c00640 1 Processor -- start 2024-01-16T15:27:47.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.010+0000 7f2ab5c00640 1 -- start start 2024-01-16T15:27:47.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2ab5c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f2ab013df80 0x7f2ab015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2ab5c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2ab00cf9a0 0x7f2ab00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2ab5c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab006c470 con 0x7f2ab013e360 2024-01-16T15:27:47.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2ab5c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab006c750 con 0x7f2ab00cf9a0 2024-01-16T15:27:47.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2ab5c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab006ca30 con 0x7f2ab013df80 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2aaf400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2ab00cf9a0 0x7f2ab00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2aaf400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2ab00cf9a0 0x7f2ab00cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33158/0 (socket says 172.21.15.53:33158) 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2aaf400640 1 -- 172.21.15.53:0/2285510996 learned_addr learned my addr 172.21.15.53:0/2285510996 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.012+0000 7f2aae000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2ab013e360 0x7f2ab00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40182/0 (socket says 172.21.15.53:40182) 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.013+0000 7f2aaea00640 1 --2- 172.21.15.53:0/2285510996 >> v2:172.21.15.146:3300/0 conn(0x7f2ab013df80 0x7f2ab015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.013+0000 7f2aaf400640 1 -- 172.21.15.53:0/2285510996 >> v2:172.21.15.146:3300/0 conn(0x7f2ab013df80 msgr2=0x7f2ab015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.013+0000 7f2aaf400640 1 --2- 172.21.15.53:0/2285510996 >> v2:172.21.15.146:3300/0 conn(0x7f2ab013df80 0x7f2ab015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.013+0000 7f2aaf400640 1 -- 172.21.15.53:0/2285510996 >> v1:172.21.15.53:6789/0 conn(0x7f2ab013e360 legacy=0x7f2ab00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.013+0000 7f2aaf400640 1 -- 172.21.15.53:0/2285510996 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab006cd10 con 0x7f2ab00cf9a0 2024-01-16T15:27:47.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.014+0000 7f2aaf400640 1 --2- 172.21.15.53:0/2285510996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00cd840 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0072770 tx=0x7f2aa802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc520e832d28eeff server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.014+0000 7f2aad600640 1 -- 172.21.15.53:0/2285510996 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa802cb10 con 0x7f2ab00cf9a0 2024-01-16T15:27:47.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.014+0000 7f2aad600640 1 -- 172.21.15.53:0/2285510996 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aa802cc70 con 0x7f2ab00cf9a0 2024-01-16T15:27:47.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.014+0000 7f2aad600640 1 -- 172.21.15.53:0/2285510996 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aa80343f0 con 0x7f2ab00cf9a0 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2285510996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00cf9a0 msgr2=0x7f2ab00cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2285510996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00cd840 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0072770 tx=0x7f2aa802bf70 comp rx=0 tx=0).stop 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2285510996 shutdown_connections 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2285510996 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00cd840 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2285510996 >> v2:172.21.15.146:3300/0 conn(0x7f2ab013df80 0x7f2ab015bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2285510996 >> 172.21.15.53:0/2285510996 conn(0x7f2ab006a800 msgr2=0x7f2ab006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:47.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2285510996 shutdown_connections 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.015+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2285510996 wait complete. 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.017+0000 7f2ab5c00640 1 Processor -- start 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.017+0000 7f2ab5c00640 1 -- start start 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2ab5c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2ab5c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 0x7f2ab0153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2ab5c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ab013e360 0x7f2ab00a8c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2ab5c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab01f3090 con 0x7f2ab013df80 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2ab5c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab0153870 con 0x7f2ab00cf9a0 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2ab5c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab01f31f0 con 0x7f2ab013e360 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2aae000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 0x7f2ab0153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2aae000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 0x7f2ab0153330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33164/0 (socket says 172.21.15.53:33164) 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.018+0000 7f2aae000640 1 -- 172.21.15.53:0/2627734102 learned_addr learned my addr 172.21.15.53:0/2627734102 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aaea00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aaf400640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ab013e360 0x7f2ab00a8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aae000640 1 -- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ab013e360 msgr2=0x7f2ab00a8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aae000640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ab013e360 0x7f2ab00a8c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aae000640 1 -- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ab00cf9a0 msgr2=0x7f2ab00b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aae000640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aae000640 1 -- 172.21.15.53:0/2627734102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2aa8009b30 con 0x7f2ab013df80 2024-01-16T15:27:47.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aae000640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 0x7f2ab0153330 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c450 tx=0x7f2a9800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.019+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98007480 con 0x7f2ab013df80 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.020+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a980075e0 con 0x7f2ab013df80 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.020+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a98004a00 con 0x7f2ab013df80 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.020+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab00b3640 con 0x7f2ab013df80 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.020+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab00b3c00 con 0x7f2ab013df80 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.021+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab00675d0 con 0x7f2ab013df80 2024-01-16T15:27:47.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.022+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2a98004b60 con 0x7f2ab013df80 2024-01-16T15:27:47.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.023+0000 7f2aa6a00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2a8806c550 0x7f2a8806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.023+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2a98087d20 con 0x7f2ab013df80 2024-01-16T15:27:47.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.024+0000 7f2aaea00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2a8806c550 0x7f2a8806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.025+0000 7f2aaea00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2a8806c550 0x7f2a8806ea10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0006fd0 tx=0x7f2aa0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.027+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2a98054260 con 0x7f2ab013df80 2024-01-16T15:27:47.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.222+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ab006df00 con 0x7f2ab013df80 2024-01-16T15:27:47.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.224+0000 7f2aa6a00640 1 -- 172.21.15.53:0/2627734102 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f2a98018020 con 0x7f2ab013df80 2024-01-16T15:27:47.224 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:47.224 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":112,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:42.134132+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.24","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:47.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2a8806c550 msgr2=0x7f2a8806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2a8806c550 0x7f2a8806ea10 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f2aa0006fd0 tx=0x7f2aa0008040 comp rx=0 tx=0).stop 2024-01-16T15:27:47.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 msgr2=0x7f2ab0153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 0x7f2ab0153330 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f2a9800c450 tx=0x7f2a9800c910 comp rx=0 tx=0).stop 2024-01-16T15:27:47.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 shutdown_connections 2024-01-16T15:27:47.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2a8806c550 0x7f2a8806ea10 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ab013e360 0x7f2ab00a8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ab013df80 0x7f2ab0153330 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 --2- 172.21.15.53:0/2627734102 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ab00cf9a0 0x7f2ab00b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 >> 172.21.15.53:0/2627734102 conn(0x7f2ab006a800 msgr2=0x7f2ab00b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:47.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 shutdown_connections 2024-01-16T15:27:47.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.228+0000 7f2ab5c00640 1 -- 172.21.15.53:0/2627734102 wait complete. 2024-01-16T15:27:47.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:19:21.949075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:19:23.088954+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:47.247 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:47.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.473+0000 7fe55b200640 1 Processor -- start 2024-01-16T15:27:47.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.474+0000 7fe55b200640 1 -- start start 2024-01-16T15:27:47.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.475+0000 7fe55b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe55415d4b0 0x7fe5540d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.475+0000 7fe55b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540b41e0 0x7fe5540cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.475+0000 7fe55b200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5540d0320 con 0x7fe5540cf9a0 2024-01-16T15:27:47.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.475+0000 7fe55b200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5540b2730 con 0x7fe55415d4b0 2024-01-16T15:27:47.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.475+0000 7fe55b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5540d1ec0 con 0x7fe5540b41e0 2024-01-16T15:27:47.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe55415d4b0 0x7fe5540d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe55415d4b0 0x7fe5540d1550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33174/0 (socket says 172.21.15.53:33174) 2024-01-16T15:27:47.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 -- 172.21.15.53:0/3062843835 learned_addr learned my addr 172.21.15.53:0/3062843835 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:47.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe55a800640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540b41e0 0x7fe5540cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 -- 172.21.15.53:0/3062843835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540b41e0 msgr2=0x7fe5540cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:47.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540b41e0 0x7fe5540cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 -- 172.21.15.53:0/3062843835 >> v1:172.21.15.146:6789/0 conn(0x7fe5540cf9a0 legacy=0x7fe5540cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:47.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe559e00640 1 -- 172.21.15.53:0/3062843835 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5540b29d0 con 0x7fe55415d4b0 2024-01-16T15:27:47.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.476+0000 7fe55a800640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540b41e0 0x7fe5540cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:47.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.477+0000 7fe559e00640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe55415d4b0 0x7fe5540d1550 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009870 tx=0x7fe53c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1764b7234cfd7c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.478+0000 7fe558a00640 1 -- 172.21.15.53:0/3062843835 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c031640 con 0x7fe55415d4b0 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.478+0000 7fe558a00640 1 -- 172.21.15.53:0/3062843835 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c0317a0 con 0x7fe55415d4b0 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.478+0000 7fe558a00640 1 -- 172.21.15.53:0/3062843835 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c031b30 con 0x7fe55415d4b0 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.478+0000 7fe55b200640 1 -- 172.21.15.53:0/3062843835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe55415d4b0 msgr2=0x7fe5540d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.478+0000 7fe55b200640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe55415d4b0 0x7fe5540d1550 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7fe53c009870 tx=0x7fe53c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.479+0000 7fe55b200640 1 -- 172.21.15.53:0/3062843835 shutdown_connections 2024-01-16T15:27:47.478 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.479+0000 7fe55b200640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540b41e0 0x7fe5540cfde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.479+0000 7fe55b200640 1 --2- 172.21.15.53:0/3062843835 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe55415d4b0 0x7fe5540d1550 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.479+0000 7fe55b200640 1 -- 172.21.15.53:0/3062843835 >> 172.21.15.53:0/3062843835 conn(0x7fe5540d3000 msgr2=0x7fe5540d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:47.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.479+0000 7fe55b200640 1 -- 172.21.15.53:0/3062843835 shutdown_connections 2024-01-16T15:27:47.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.479+0000 7fe55b200640 1 -- 172.21.15.53:0/3062843835 wait complete. 2024-01-16T15:27:47.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.480+0000 7fe55b200640 1 Processor -- start 2024-01-16T15:27:47.480 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.481+0000 7fe55b200640 1 -- start start 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.481+0000 7fe55b200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 0x7fe554148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.481+0000 7fe55b200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540cf9a0 0x7fe5540a0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe55b200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe55415d4b0 0x7fe5541479c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe55b200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55415ad80 con 0x7fe5540b41e0 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe55b200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe554149380 con 0x7fe55415d4b0 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe55b200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe554146920 con 0x7fe5540cf9a0 2024-01-16T15:27:47.482 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 0x7fe554148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 0x7fe554148e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33184/0 (socket says 172.21.15.53:33184) 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 -- 172.21.15.53:0/1574972269 learned_addr learned my addr 172.21.15.53:0/1574972269 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 -- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540cf9a0 msgr2=0x7fe5540a0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540cf9a0 0x7fe5540a0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 -- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe55415d4b0 msgr2=0x7fe5541479c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe55415d4b0 0x7fe5541479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.482+0000 7fe559e00640 1 -- 172.21.15.53:0/1574972269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c0094d0 con 0x7fe5540b41e0 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.483+0000 7fe559e00640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 0x7fe554148e40 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0097c0 tx=0x7fe53c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.483+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c03ac60 con 0x7fe5540b41e0 2024-01-16T15:27:47.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.483+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe53c03adc0 con 0x7fe5540b41e0 2024-01-16T15:27:47.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.483+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53c044630 con 0x7fe5540b41e0 2024-01-16T15:27:47.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.483+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55415da10 con 0x7fe5540b41e0 2024-01-16T15:27:47.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.483+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe554146ee0 con 0x7fe5540b41e0 2024-01-16T15:27:47.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.484+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5540d2cd0 con 0x7fe5540b41e0 2024-01-16T15:27:47.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.485+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe53c02f050 con 0x7fe5540b41e0 2024-01-16T15:27:47.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.486+0000 7fe552000640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe52c06c5a0 0x7fe52c06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.486+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe53c0aa840 con 0x7fe5540b41e0 2024-01-16T15:27:47.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.487+0000 7fe559400640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe52c06c5a0 0x7fe52c06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.486 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.488+0000 7fe559400640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe52c06c5a0 0x7fe52c06ea60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe5541462c0 tx=0x7fe544007d30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.491+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe53c077100 con 0x7fe5540b41e0 2024-01-16T15:27:47.685 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.687+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe554067450 con 0x7fe5540b41e0 2024-01-16T15:27:47.686 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.687+0000 7fe552000640 1 -- 172.21.15.53:0/1574972269 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe53c0350a0 con 0x7fe5540b41e0 2024-01-16T15:27:47.686 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:27:47.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe52c06c5a0 msgr2=0x7fe52c06ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe52c06c5a0 0x7fe52c06ea60 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fe5541462c0 tx=0x7fe544007d30 comp rx=0 tx=0).stop 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 msgr2=0x7fe554148e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 0x7fe554148e40 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7fe53c0097c0 tx=0x7fe53c0046c0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 shutdown_connections 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe52c06c5a0 0x7fe52c06ea60 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe55415d4b0 0x7fe5541479c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe5540cf9a0 0x7fe5540a0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 --2- 172.21.15.53:0/1574972269 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe5540b41e0 0x7fe554148e40 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.690+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 >> 172.21.15.53:0/1574972269 conn(0x7fe5540d3000 msgr2=0x7fe5540cdd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:47.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.691+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 shutdown_connections 2024-01-16T15:27:47.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.691+0000 7fe55b200640 1 -- 172.21.15.53:0/1574972269 wait complete. 2024-01-16T15:27:47.707 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:47.918 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.920+0000 7fca7fe00640 1 Processor -- start 2024-01-16T15:27:47.919 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.920+0000 7fca7fe00640 1 -- start start 2024-01-16T15:27:47.920 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca801409d0 0x7fca800cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca800cec80 0x7fca800ae520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7fe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca800ce7f0 con 0x7fca80140db0 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7fe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca800cdec0 con 0x7fca801409d0 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800ce1a0 con 0x7fca800cec80 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca801409d0 0x7fca800cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fca801409d0 0x7fca800cbeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33194/0 (socket says 172.21.15.53:33194) 2024-01-16T15:27:47.921 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7ea00640 1 -- 172.21.15.53:0/3535559521 learned_addr learned my addr 172.21.15.53:0/3535559521 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7e000640 1 --2- 172.21.15.53:0/3535559521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca800cec80 0x7fca800ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.922+0000 7fca7ea00640 1 -- 172.21.15.53:0/3535559521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca800cec80 msgr2=0x7fca800ae520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.923+0000 7fca7ea00640 1 --2- 172.21.15.53:0/3535559521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca800cec80 0x7fca800ae520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.922 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.923+0000 7fca7ea00640 1 -- 172.21.15.53:0/3535559521 >> v1:172.21.15.146:6789/0 conn(0x7fca80140db0 legacy=0x7fca8015eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:47.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.923+0000 7fca7ea00640 1 -- 172.21.15.53:0/3535559521 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca800ce480 con 0x7fca801409d0 2024-01-16T15:27:47.923 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.924+0000 7fca7ea00640 1 --2- 172.21.15.53:0/3535559521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca801409d0 0x7fca800cbeb0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fca70009870 tx=0x7fca7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fd30e3d534d513a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.924+0000 7fca7d600640 1 -- 172.21.15.53:0/3535559521 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70031640 con 0x7fca801409d0 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.924+0000 7fca7d600640 1 -- 172.21.15.53:0/3535559521 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca700317a0 con 0x7fca801409d0 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.924+0000 7fca7d600640 1 -- 172.21.15.53:0/3535559521 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70031b30 con 0x7fca801409d0 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 -- 172.21.15.53:0/3535559521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca801409d0 msgr2=0x7fca800cbeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 --2- 172.21.15.53:0/3535559521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca801409d0 0x7fca800cbeb0 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fca70009870 tx=0x7fca7002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 -- 172.21.15.53:0/3535559521 shutdown_connections 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 --2- 172.21.15.53:0/3535559521 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca800cec80 0x7fca800ae520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.924 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 --2- 172.21.15.53:0/3535559521 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca801409d0 0x7fca800cbeb0 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 -- 172.21.15.53:0/3535559521 >> 172.21.15.53:0/3535559521 conn(0x7fca800cf4a0 msgr2=0x7fca800cf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:47.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 -- 172.21.15.53:0/3535559521 shutdown_connections 2024-01-16T15:27:47.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.925+0000 7fca7fe00640 1 -- 172.21.15.53:0/3535559521 wait complete. 2024-01-16T15:27:47.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.927+0000 7fca7fe00640 1 Processor -- start 2024-01-16T15:27:47.925 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.927+0000 7fca7fe00640 1 -- start start 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 0x7fca800b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7fe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca801409d0 0x7fca801f3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7fe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca80140db0 0x7fca800afea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7fe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca801f3860 con 0x7fca800cec80 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7fe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca800b38f0 con 0x7fca801409d0 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7fe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca800cb590 con 0x7fca80140db0 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 0x7fca800b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.927 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7ea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 0x7fca800b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33202/0 (socket says 172.21.15.53:33202) 2024-01-16T15:27:47.928 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.928+0000 7fca7ea00640 1 -- 172.21.15.53:0/1348154922 learned_addr learned my addr 172.21.15.53:0/1348154922 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:47.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7f400640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca80140db0 0x7fca800afea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7e000640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca801409d0 0x7fca801f3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7ea00640 1 -- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca80140db0 msgr2=0x7fca800afea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7ea00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca80140db0 0x7fca800afea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.929 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7ea00640 1 -- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca801409d0 msgr2=0x7fca801f3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7ea00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca801409d0 0x7fca801f3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7ea00640 1 -- 172.21.15.53:0/1348154922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700094d0 con 0x7fca800cec80 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7f400640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca80140db0 0x7fca800afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7e000640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca801409d0 0x7fca801f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7ea00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 0x7fca800b33b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fca70009870 tx=0x7fca700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca7003ac60 con 0x7fca800cec80 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca7003ade0 con 0x7fca800cec80 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70044630 con 0x7fca800cec80 2024-01-16T15:27:47.930 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.929+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca800cb870 con 0x7fca800cec80 2024-01-16T15:27:47.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.930+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca801f3480 con 0x7fca800cec80 2024-01-16T15:27:47.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.931+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca801f3ad0 con 0x7fca800cec80 2024-01-16T15:27:47.931 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.932+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fca7002f050 con 0x7fca800cec80 2024-01-16T15:27:47.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.933+0000 7fca76a00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca5806c550 0x7fca5806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:47.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.933+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fca700aa720 con 0x7fca800cec80 2024-01-16T15:27:47.932 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.933+0000 7fca7e000640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca5806c550 0x7fca5806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:47.933 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.935+0000 7fca7e000640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca5806c550 0x7fca5806ea10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fca800cff40 tx=0x7fca68008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:47.936 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:47.937+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fca70076fe0 con 0x7fca800cec80 2024-01-16T15:27:48.131 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.132+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fca80067450 con 0x7fca800cec80 2024-01-16T15:27:48.132 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.133+0000 7fca76a00640 1 -- 172.21.15.53:0/1348154922 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7fca70031e60 con 0x7fca800cec80 2024-01-16T15:27:48.132 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:27:48.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca5806c550 msgr2=0x7fca5806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.134 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca5806c550 0x7fca5806ea10 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fca800cff40 tx=0x7fca68008040 comp rx=0 tx=0).stop 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 msgr2=0x7fca800b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 0x7fca800b33b0 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fca70009870 tx=0x7fca700046c0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 shutdown_connections 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fca5806c550 0x7fca5806ea10 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fca80140db0 0x7fca800afea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fca801409d0 0x7fca801f3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.135 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 --2- 172.21.15.53:0/1348154922 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fca800cec80 0x7fca800b33b0 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 >> 172.21.15.53:0/1348154922 conn(0x7fca800cf4a0 msgr2=0x7fca80159e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:48.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 shutdown_connections 2024-01-16T15:27:48.136 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.135+0000 7fca7fe00640 1 -- 172.21.15.53:0/1348154922 wait complete. 2024-01-16T15:27:48.154 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:48.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.370+0000 7f0072c00640 1 Processor -- start 2024-01-16T15:27:48.370 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.370+0000 7f0072c00640 1 -- start start 2024-01-16T15:27:48.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.372+0000 7f0072c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f006c070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.372+0000 7f0072c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f006c15bf10 0x7f006c143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.372+0000 7f0072c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006c15f4e0 con 0x7f006c0adf20 2024-01-16T15:27:48.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.372+0000 7f0072c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006c15f970 con 0x7f006c15bf10 2024-01-16T15:27:48.371 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.372+0000 7f0072c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c15c3c0 con 0x7f006c0ccb50 2024-01-16T15:27:48.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.372+0000 7f0071800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f006c0adf20 0x7f006c0cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40196/0 (socket says 172.21.15.53:40196) 2024-01-16T15:27:48.373 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.373+0000 7f0071800640 1 -- 172.21.15.53:0/1231244795 learned_addr learned my addr 172.21.15.53:0/1231244795 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.373+0000 7f0072200640 1 --2- 172.21.15.53:0/1231244795 >> v2:172.21.15.146:3300/0 conn(0x7f006c15bf10 0x7f006c143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.373+0000 7f0070e00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f006c070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.373+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594932780 0 0) 0x7f006c15f4e0 con 0x7f006c0adf20 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.373+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f004c003520 con 0x7f006c0adf20 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2213854803 0 0) 0x7f004c003520 con 0x7f006c0adf20 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 msgr2=0x7f006c070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f006c070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 >> v2:172.21.15.146:3300/0 conn(0x7f006c15bf10 msgr2=0x7f006c143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:48.374 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 --2- 172.21.15.53:0/1231244795 >> v2:172.21.15.146:3300/0 conn(0x7f006c15bf10 0x7f006c143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c071130 con 0x7f006c0adf20 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f005c001670 con 0x7f006c0adf20 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 >> v1:172.21.15.53:6789/0 conn(0x7f006c0adf20 legacy=0x7f006c0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c003970 0x7f004c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c0047c0 0x7f004c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f004c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c15f970 con 0x7f004c003970 2024-01-16T15:27:48.375 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c15c3c0 con 0x7f004c0047c0 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f004c00fd90 con 0x7f006c0ccb50 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f0071800640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c003970 0x7f004c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f0070e00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c0047c0 0x7f004c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.374+0000 7f0072200640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f004c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.375+0000 7f0071800640 1 -- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 msgr2=0x7f004c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.375+0000 7f0071800640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f004c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.375+0000 7f0071800640 1 -- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c0047c0 msgr2=0x7f004c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:48.376 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.375+0000 7f0071800640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c0047c0 0x7f004c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.375+0000 7f0071800640 1 -- 172.21.15.53:0/1231244795 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f004c003520 con 0x7f004c003970 2024-01-16T15:27:48.377 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.375+0000 7f0072200640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c0ccb50 0x7f004c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.376+0000 7f0071800640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c003970 0x7f004c003d90 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c270 tx=0x7f004c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc6abf2f16ee1a3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.376+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c005ad0 con 0x7f004c003970 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.376+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c005c50 con 0x7f004c003970 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.376+0000 7f006be00640 1 -- 172.21.15.53:0/1231244795 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c02c8b0 con 0x7f004c003970 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072c00640 1 -- 172.21.15.53:0/1231244795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c003970 msgr2=0x7f004c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072c00640 1 --2- 172.21.15.53:0/1231244795 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f004c003970 0x7f004c003d90 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7f005c00c270 tx=0x7f004c0057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072200640 1 -- 172.21.15.53:0/1231244795 reap_dead start 2024-01-16T15:27:48.379 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072c00640 1 -- 172.21.15.53:0/1231244795 shutdown_connections 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072c00640 1 -- 172.21.15.53:0/1231244795 >> 172.21.15.53:0/1231244795 conn(0x7f006c06a7d0 msgr2=0x7f006c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072c00640 1 -- 172.21.15.53:0/1231244795 shutdown_connections 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.377+0000 7f0072c00640 1 -- 172.21.15.53:0/1231244795 wait complete. 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.379+0000 7f0072c00640 1 Processor -- start 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.379+0000 7f0072c00640 1 -- start start 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c003970 0x7f006c0b3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c1375c0 0x7f006c137d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 0x7f006c06d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.380 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c06d670 con 0x7f006c06ccb0 2024-01-16T15:27:48.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006c1384e0 con 0x7f004c003970 2024-01-16T15:27:48.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006c13af50 con 0x7f006c1375c0 2024-01-16T15:27:48.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 0x7f006c06d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 0x7f006c06d130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33222/0 (socket says 172.21.15.53:33222) 2024-01-16T15:27:48.381 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 -- 172.21.15.53:0/1107504556 learned_addr learned my addr 172.21.15.53:0/1107504556 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:48.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 -- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c1375c0 msgr2=0x7f006c137d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.382 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c1375c0 0x7f006c137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 -- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c003970 msgr2=0x7f006c0b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c003970 0x7f006c0b3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.380+0000 7f0072200640 1 -- 172.21.15.53:0/1107504556 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006c138240 con 0x7f006c06ccb0 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.381+0000 7f0072200640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 0x7f006c06d130 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f006c144cf0 tx=0x7f006400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.381+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064016620 con 0x7f006c06ccb0 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.381+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0064016780 con 0x7f006c06ccb0 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.381+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0064016b10 con 0x7f006c06ccb0 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.381+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f006c13a620 con 0x7f006c06ccb0 2024-01-16T15:27:48.383 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.381+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006c13abe0 con 0x7f006c06ccb0 2024-01-16T15:27:48.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.382+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006c067450 con 0x7f006c06ccb0 2024-01-16T15:27:48.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.384+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0064016c70 con 0x7f006c06ccb0 2024-01-16T15:27:48.384 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.385+0000 7f0069600640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f004406c550 0x7f004406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.385+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f00640898b0 con 0x7f006c06ccb0 2024-01-16T15:27:48.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.386+0000 7f0071800640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f004406c550 0x7f004406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.385 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.387+0000 7f0071800640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f004406c550 0x7f004406ea10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f005c034000 tx=0x7f005c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:48.388 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.390+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0064055ea0 con 0x7f006c06ccb0 2024-01-16T15:27:48.584 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.585+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f006c13ada0 con 0x7f006c06ccb0 2024-01-16T15:27:48.585 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.586+0000 7f0069600640 1 -- 172.21.15.53:0/1107504556 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f00640594c0 con 0x7f006c06ccb0 2024-01-16T15:27:48.585 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:27:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f004406c550 msgr2=0x7f004406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f004406c550 0x7f004406ea10 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f005c034000 tx=0x7f005c02e000 comp rx=0 tx=0).stop 2024-01-16T15:27:48.588 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 msgr2=0x7f006c06d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 0x7f006c06d130 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7f006c144cf0 tx=0x7f006400f520 comp rx=0 tx=0).stop 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 shutdown_connections 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f004406c550 0x7f004406ea10 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f006c06ccb0 0x7f006c06d130 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f006c1375c0 0x7f006c137d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 --2- 172.21.15.53:0/1107504556 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f004c003970 0x7f006c0b3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 >> 172.21.15.53:0/1107504556 conn(0x7f006c06a7d0 msgr2=0x7f006c06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 shutdown_connections 2024-01-16T15:27:48.589 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.589+0000 7f0072c00640 1 -- 172.21.15.53:0/1107504556 wait complete. 2024-01-16T15:27:48.607 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:48.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.823+0000 7fb43e600640 1 Processor -- start 2024-01-16T15:27:48.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.824+0000 7fb43e600640 1 -- start start 2024-01-16T15:27:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.825+0000 7fb43e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43813df80 0x7fb43815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.825+0000 7fb43e600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb43813e360 0x7fb4380cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.825+0000 7fb43e600640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb43815f9c0 0x7fb4380cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.825+0000 7fb43e600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43806c470 con 0x7fb43813e360 2024-01-16T15:27:48.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.825+0000 7fb43e600640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43806c750 con 0x7fb43815f9c0 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.825+0000 7fb43e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43806ca30 con 0x7fb43813df80 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb43813e360 0x7fb4380cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb43813e360 0x7fb4380cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33228/0 (socket says 172.21.15.53:33228) 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 -- 172.21.15.53:0/790629540 learned_addr learned my addr 172.21.15.53:0/790629540 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb437400640 1 --2- 172.21.15.53:0/790629540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43813df80 0x7fb43815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb437e00640 1 --2- 172.21.15.53:0/790629540 >> v2:172.21.15.146:3300/0 conn(0x7fb43815f9c0 0x7fb4380cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 -- 172.21.15.53:0/790629540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43813df80 msgr2=0x7fb43815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:48.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 --2- 172.21.15.53:0/790629540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43813df80 0x7fb43815bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 -- 172.21.15.53:0/790629540 >> v2:172.21.15.146:3300/0 conn(0x7fb43815f9c0 msgr2=0x7fb4380cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:48.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 --2- 172.21.15.53:0/790629540 >> v2:172.21.15.146:3300/0 conn(0x7fb43815f9c0 0x7fb4380cf7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.826+0000 7fb436a00640 1 -- 172.21.15.53:0/790629540 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43806cd10 con 0x7fb43813e360 2024-01-16T15:27:48.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.827+0000 7fb436a00640 1 --2- 172.21.15.53:0/790629540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813e360 0x7fb4380cf290 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fb43806e6d0 tx=0x7fb42002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eccf3d73805da2ee server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:48.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.828+0000 7fb436000640 1 -- 172.21.15.53:0/790629540 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420031860 con 0x7fb43813e360 2024-01-16T15:27:48.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.828+0000 7fb436000640 1 -- 172.21.15.53:0/790629540 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4200319c0 con 0x7fb43813e360 2024-01-16T15:27:48.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.828+0000 7fb436000640 1 -- 172.21.15.53:0/790629540 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb420031d50 con 0x7fb43813e360 2024-01-16T15:27:48.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 -- 172.21.15.53:0/790629540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813e360 msgr2=0x7fb4380cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 --2- 172.21.15.53:0/790629540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813e360 0x7fb4380cf290 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fb43806e6d0 tx=0x7fb42002b900 comp rx=0 tx=0).stop 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 -- 172.21.15.53:0/790629540 shutdown_connections 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 --2- 172.21.15.53:0/790629540 >> v2:172.21.15.146:3300/0 conn(0x7fb43815f9c0 0x7fb4380cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 --2- 172.21.15.53:0/790629540 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813e360 0x7fb4380cf290 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 --2- 172.21.15.53:0/790629540 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43813df80 0x7fb43815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 -- 172.21.15.53:0/790629540 >> 172.21.15.53:0/790629540 conn(0x7fb43806a800 msgr2=0x7fb43806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 -- 172.21.15.53:0/790629540 shutdown_connections 2024-01-16T15:27:48.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.829+0000 7fb43e600640 1 -- 172.21.15.53:0/790629540 wait complete. 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.831+0000 7fb43e600640 1 Processor -- start 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.831+0000 7fb43e600640 1 -- start start 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb43e600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 0x7fb43813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb43e600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb43813e360 0x7fb4380aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb43e600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43815f9c0 0x7fb4380aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb43e600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380702c0 con 0x7fb43813df80 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb43e600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb438070510 con 0x7fb43813e360 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb43e600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4380707f0 con 0x7fb43815f9c0 2024-01-16T15:27:48.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 0x7fb43813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 0x7fb43813d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33238/0 (socket says 172.21.15.53:33238) 2024-01-16T15:27:48.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 -- 172.21.15.53:0/2092789555 learned_addr learned my addr 172.21.15.53:0/2092789555 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:48.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 -- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43815f9c0 msgr2=0x7fb4380aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43815f9c0 0x7fb4380aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 -- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb43813e360 msgr2=0x7fb4380aa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:48.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb43813e360 0x7fb4380aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:48.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.832+0000 7fb437400640 1 -- 172.21.15.53:0/2092789555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4200094d0 con 0x7fb43813df80 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.833+0000 7fb437400640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 0x7fb43813d730 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fb42800ccd0 tx=0x7fb42800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.833+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428003990 con 0x7fb43813df80 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.833+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb428003af0 con 0x7fb43813df80 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.833+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb428004020 con 0x7fb43813df80 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.833+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb438070ad0 con 0x7fb43813df80 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.833+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb438071090 con 0x7fb43813df80 2024-01-16T15:27:48.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.834+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4380675d0 con 0x7fb43813df80 2024-01-16T15:27:48.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.835+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb428004180 con 0x7fb43813df80 2024-01-16T15:27:48.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.837+0000 7fb42f400640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb41006c550 0x7fb41006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:48.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.837+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb428088be0 con 0x7fb43813df80 2024-01-16T15:27:48.836 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.838+0000 7fb436a00640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb41006c550 0x7fb41006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:48.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.839+0000 7fb436a00640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb41006c550 0x7fb41006ea10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb420036000 tx=0x7fb420030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:48.841 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:48.842+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb4280554a0 con 0x7fb43813df80 2024-01-16T15:27:49.054 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.055+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb43806df00 con 0x7fb43813df80 2024-01-16T15:27:49.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.056+0000 7fb42f400640 1 -- 172.21.15.53:0/2092789555 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v112) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb42801d030 con 0x7fb43813df80 2024-01-16T15:27:49.055 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:27:49.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb41006c550 msgr2=0x7fb41006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:49.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb41006c550 0x7fb41006ea10 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb420036000 tx=0x7fb420030000 comp rx=0 tx=0).stop 2024-01-16T15:27:49.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 msgr2=0x7fb43813d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:49.059 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 0x7fb43813d730 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fb42800ccd0 tx=0x7fb42800a5c0 comp rx=0 tx=0).stop 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 shutdown_connections 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb41006c550 0x7fb41006ea10 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb43815f9c0 0x7fb4380aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb43813e360 0x7fb4380aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 --2- 172.21.15.53:0/2092789555 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb43813df80 0x7fb43813d730 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 >> 172.21.15.53:0/2092789555 conn(0x7fb43806a800 msgr2=0x7fb4380cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 shutdown_connections 2024-01-16T15:27:49.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:49.059+0000 7fb43e600640 1 -- 172.21.15.53:0/2092789555 wait complete. 2024-01-16T15:27:49.076 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi053.front.sepia.ceph.com... 2024-01-16T15:27:49.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:49.076 DEBUG:teuthology.orchestra.run.smithi053:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-01-16T15:27:49.164 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi053.stderr:ceph-fuse[75897]: fuse finished with error 0 and tester_r 0 2024-01-16T15:27:49.791 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:27:49.791 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:27:49.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:49.792 DEBUG:teuthology.orchestra.run.smithi053:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-01-16T15:27:49.839 DEBUG:teuthology.orchestra.run.smithi053:> set -ex 2024-01-16T15:27:49.839 DEBUG:teuthology.orchestra.run.smithi053:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:49.881 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054633,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250024k,nr_inodes=812506,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500052k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-01-16T15:27:49.881 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:27:49.881 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:49.940 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:27:49.940 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:49.995 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:50.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.222+0000 7f7388c00640 1 Processor -- start 2024-01-16T15:27:50.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.222+0000 7f7388c00640 1 -- start start 2024-01-16T15:27:50.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.223+0000 7f7388c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73840cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.223+0000 7f7388c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f738415f9c0 0x7f73840cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.223+0000 7f7388c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738415c5f0 con 0x7f738413df80 2024-01-16T15:27:50.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.223+0000 7f7388c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f738406c470 con 0x7f738415f9c0 2024-01-16T15:27:50.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.223+0000 7f7388c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f738406c750 con 0x7f738413e360 2024-01-16T15:27:50.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.224+0000 7f7381e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f738413df80 0x7f738415bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40210/0 (socket says 172.21.15.53:40210) 2024-01-16T15:27:50.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.224+0000 7f7381e00640 1 -- 172.21.15.53:0/2353227651 learned_addr learned my addr 172.21.15.53:0/2353227651 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.224+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023166501 0 0) 0x7f738415c5f0 con 0x7f738413df80 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.224+0000 7f7381400640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73840cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.224+0000 7f7382800640 1 --2- 172.21.15.53:0/2353227651 >> v2:172.21.15.146:3300/0 conn(0x7f738415f9c0 0x7f73840cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.224+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7364003520 con 0x7f738413df80 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2962380079 0 0) 0x7f7364003520 con 0x7f738413df80 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 msgr2=0x7f73840cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73840cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 >> v2:172.21.15.146:3300/0 conn(0x7f738415f9c0 msgr2=0x7f73840cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:50.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 --2- 172.21.15.53:0/2353227651 >> v2:172.21.15.146:3300/0 conn(0x7f738415f9c0 0x7f73840cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738406ca30 con 0x7f738413df80 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f7374001670 con 0x7f738413df80 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7382800640 1 --2- 172.21.15.53:0/2353227651 >> v2:172.21.15.146:3300/0 conn(0x7f738415f9c0 0x7f73840cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 >> v1:172.21.15.53:6789/0 conn(0x7f738413df80 legacy=0x7f738415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7364003970 0x7f7364003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73640047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7364004d00 0x7f7364005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738406c470 con 0x7f7364003970 2024-01-16T15:27:50.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f738406c750 con 0x7f7364004d00 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f736400bae0 con 0x7f738413e360 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.225+0000 7f7381e00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7364003970 0x7f7364003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381400640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73640047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7382800640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7364004d00 0x7f7364005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381e00640 1 -- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 msgr2=0x7f73640047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381e00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73640047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381e00640 1 -- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7364004d00 msgr2=0x7f7364005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381e00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7364004d00 0x7f7364005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381e00640 1 -- 172.21.15.53:0/2353227651 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7364003520 con 0x7f7364003970 2024-01-16T15:27:50.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7382800640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f7364004d00 0x7f7364005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:50.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.226+0000 7f7381400640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f738413e360 0x7f73640047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:50.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.227+0000 7f7381e00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7364003970 0x7f7364003d90 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f737400c110 tx=0x7f73640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f623f8318e76ad4 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.227+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374005ad0 con 0x7f7364003970 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.227+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7374005c50 con 0x7f7364003970 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.227+0000 7f7380a00640 1 -- 172.21.15.53:0/2353227651 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737402c830 con 0x7f7364003970 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7388c00640 1 -- 172.21.15.53:0/2353227651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7364003970 msgr2=0x7f7364003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7388c00640 1 --2- 172.21.15.53:0/2353227651 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f7364003970 0x7f7364003d90 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7f737400c110 tx=0x7f73640057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7382800640 1 -- 172.21.15.53:0/2353227651 reap_dead start 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7388c00640 1 -- 172.21.15.53:0/2353227651 shutdown_connections 2024-01-16T15:27:50.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7388c00640 1 -- 172.21.15.53:0/2353227651 >> 172.21.15.53:0/2353227651 conn(0x7f738406a800 msgr2=0x7f738406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7388c00640 1 -- 172.21.15.53:0/2353227651 shutdown_connections 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.228+0000 7f7388c00640 1 -- 172.21.15.53:0/2353227651 wait complete. 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.230+0000 7f7388c00640 1 Processor -- start 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.230+0000 7f7388c00640 1 -- start start 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7388c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7364003970 0x7f73840aaa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7388c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f738413d840 0x7f73840b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7388c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 0x7f7384151500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7388c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73840aaf70 con 0x7f73840b3230 2024-01-16T15:27:50.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7388c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73840b36b0 con 0x7f738413d840 2024-01-16T15:27:50.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7388c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7384152d10 con 0x7f7364003970 2024-01-16T15:27:50.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 0x7f7384151500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 0x7f7384151500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33260/0 (socket says 172.21.15.53:33260) 2024-01-16T15:27:50.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 -- 172.21.15.53:0/879813138 learned_addr learned my addr 172.21.15.53:0/879813138 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:50.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7381400640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f738413d840 0x7f73840b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7381e00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7364003970 0x7f73840aaa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 -- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7364003970 msgr2=0x7f73840aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7364003970 0x7f73840aaa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 -- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f738413d840 msgr2=0x7f73840b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f738413d840 0x7f73840b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7382800640 1 -- 172.21.15.53:0/879813138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7384152170 con 0x7f73840b3230 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.231+0000 7f7381400640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f738413d840 0x7f73840b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.232+0000 7f7382800640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 0x7f7384151500 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f738415b6b0 tx=0x7f737c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.232+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c007cf0 con 0x7f73840b3230 2024-01-16T15:27:50.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.232+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c007e50 con 0x7f73840b3230 2024-01-16T15:27:50.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.232+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73840a8e40 con 0x7f73840b3230 2024-01-16T15:27:50.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.232+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737c014640 con 0x7f73840b3230 2024-01-16T15:27:50.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.232+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73841526c0 con 0x7f73840b3230 2024-01-16T15:27:50.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.233+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73840675d0 con 0x7f73840b3230 2024-01-16T15:27:50.235 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.234+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f737c01d3b0 con 0x7f73840b3230 2024-01-16T15:27:50.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.235+0000 7f737a000640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f735c06c550 0x7f735c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.235+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f737c0888a0 con 0x7f73840b3230 2024-01-16T15:27:50.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.236+0000 7f7381e00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f735c06c550 0x7f735c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.236 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.237+0000 7f7381e00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f735c06c550 0x7f735c06ea10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7374034000 tx=0x7f7374007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:50.238 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.240+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f737c054de0 con 0x7f73840b3230 2024-01-16T15:27:50.434 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.436+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f738406df00 con 0x7f73840b3230 2024-01-16T15:27:50.436 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.438+0000 7f737a000640 1 -- 172.21.15.53:0/879813138 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f737c019090 con 0x7f73840b3230 2024-01-16T15:27:50.437 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:50.437 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":112,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:42.134132+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.24","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:50.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f735c06c550 msgr2=0x7f735c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f735c06c550 0x7f735c06ea10 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f7374034000 tx=0x7f7374007000 comp rx=0 tx=0).stop 2024-01-16T15:27:50.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 msgr2=0x7f7384151500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 0x7f7384151500 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7f738415b6b0 tx=0x7f737c004770 comp rx=0 tx=0).stop 2024-01-16T15:27:50.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 shutdown_connections 2024-01-16T15:27:50.440 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f735c06c550 0x7f735c06ea10 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f73840b3230 0x7f7384151500 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f738413d840 0x7f73840b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 --2- 172.21.15.53:0/879813138 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f7364003970 0x7f73840aaa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 >> 172.21.15.53:0/879813138 conn(0x7f738406a800 msgr2=0x7f738415a140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:50.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 shutdown_connections 2024-01-16T15:27:50.441 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.441+0000 7f7388c00640 1 -- 172.21.15.53:0/879813138 wait complete. 2024-01-16T15:27:50.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:19:21.949075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:19:23.088954+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:50.459 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:50.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.688+0000 7f9d53200640 1 Processor -- start 2024-01-16T15:27:50.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.689+0000 7f9d53200640 1 -- start start 2024-01-16T15:27:50.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d53200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0cf9a0 0x7f9d4c0cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d53200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f9d4c0b31d0 0x7f9d4c0b3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d53200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d4c0d0320 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.689 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d53200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0d1c60 con 0x7f9d4c0cf9a0 2024-01-16T15:27:50.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d53200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d4c0b2730 con 0x7f9d4c0b31d0 2024-01-16T15:27:50.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d51e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f9d4c15d4b0 0x7f9d4c0d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40226/0 (socket says 172.21.15.53:40226) 2024-01-16T15:27:50.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.690+0000 7f9d51e00640 1 -- 172.21.15.53:0/3944756202 learned_addr learned my addr 172.21.15.53:0/3944756202 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:50.690 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d51400640 1 --2- 172.21.15.53:0/3944756202 >> v2:172.21.15.53:3300/0 conn(0x7f9d4c0cf9a0 0x7f9d4c0cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d52800640 1 --2- 172.21.15.53:0/3944756202 >> v2:172.21.15.146:3300/0 conn(0x7f9d4c0b31d0 0x7f9d4c0b3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d50a00640 1 -- 172.21.15.53:0/3944756202 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783901754 0 0) 0x7f9d4c0d0320 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d50a00640 1 -- 172.21.15.53:0/3944756202 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d2c003520 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d51400640 1 -- 172.21.15.53:0/3944756202 >> v2:172.21.15.146:3300/0 conn(0x7f9d4c0b31d0 msgr2=0x7f9d4c0b3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:50.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d51400640 1 --2- 172.21.15.53:0/3944756202 >> v2:172.21.15.146:3300/0 conn(0x7f9d4c0b31d0 0x7f9d4c0b3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d51400640 1 -- 172.21.15.53:0/3944756202 >> v1:172.21.15.53:6789/0 conn(0x7f9d4c15d4b0 legacy=0x7f9d4c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.691+0000 7f9d51400640 1 -- 172.21.15.53:0/3944756202 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c0d1ec0 con 0x7f9d4c0cf9a0 2024-01-16T15:27:50.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.692+0000 7f9d51400640 1 --2- 172.21.15.53:0/3944756202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0cfde0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c072a30 tx=0x7f9d3c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86dabb7c692d3d5c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:50.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d50a00640 1 -- 172.21.15.53:0/3944756202 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c02cc80 con 0x7f9d4c0cf9a0 2024-01-16T15:27:50.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d50a00640 1 -- 172.21.15.53:0/3944756202 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3c02cde0 con 0x7f9d4c0cf9a0 2024-01-16T15:27:50.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d50a00640 1 -- 172.21.15.53:0/3944756202 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c034660 con 0x7f9d4c0cf9a0 2024-01-16T15:27:50.693 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d53200640 1 -- 172.21.15.53:0/3944756202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0cf9a0 msgr2=0x7f9d4c0cfde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d53200640 1 --2- 172.21.15.53:0/3944756202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0cfde0 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c072a30 tx=0x7f9d3c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d53200640 1 -- 172.21.15.53:0/3944756202 shutdown_connections 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d53200640 1 --2- 172.21.15.53:0/3944756202 >> v2:172.21.15.146:3300/0 conn(0x7f9d4c0b31d0 0x7f9d4c0b3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d53200640 1 --2- 172.21.15.53:0/3944756202 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0cfde0 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.693+0000 7f9d53200640 1 -- 172.21.15.53:0/3944756202 >> 172.21.15.53:0/3944756202 conn(0x7f9d4c0d3000 msgr2=0x7f9d4c0d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.694+0000 7f9d53200640 1 -- 172.21.15.53:0/3944756202 shutdown_connections 2024-01-16T15:27:50.694 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.694+0000 7f9d53200640 1 -- 172.21.15.53:0/3944756202 wait complete. 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.695+0000 7f9d53200640 1 Processor -- start 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.695+0000 7f9d53200640 1 -- start start 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.696+0000 7f9d53200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d4c0b31d0 0x7f9d4c15c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.696+0000 7f9d53200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0b83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.696+0000 7f9d53200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 0x7f9d4c0b8930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.696+0000 7f9d53200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c15c790 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.697 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.696+0000 7f9d53200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c09cce0 con 0x7f9d4c0b31d0 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.696+0000 7f9d53200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d4c09cf30 con 0x7f9d4c0cf9a0 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 0x7f9d4c0b8930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 0x7f9d4c0b8930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33276/0 (socket says 172.21.15.53:33276) 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 -- 172.21.15.53:0/1463310473 learned_addr learned my addr 172.21.15.53:0/1463310473 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d51e00640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d4c0b31d0 0x7f9d4c15c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d51400640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0b83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 -- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d4c0cf9a0 msgr2=0x7f9d4c0b83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0b83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 -- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d4c0b31d0 msgr2=0x7f9d4c15c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d4c0b31d0 0x7f9d4c15c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d52800640 1 -- 172.21.15.53:0/1463310473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c009a70 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d51e00640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d4c0b31d0 0x7f9d4c15c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.697+0000 7f9d51400640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.698+0000 7f9d52800640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 0x7f9d4c0b8930 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400d5d0 tx=0x7f9d4400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.698+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440125f0 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.698+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d44012770 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.698+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44012b20 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.698+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d4c09d210 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.698+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4c09d7d0 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.699+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d4c067450 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.700+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f9d44012c80 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.701+0000 7f9d4a000640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d2406c550 0x7f9d2406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:50.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.701+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f9d440890f0 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.701 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.702+0000 7f9d51e00640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d2406c550 0x7f9d2406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:50.702 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.703+0000 7f9d51e00640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d2406c550 0x7f9d2406ea10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9d34007fd0 tx=0x7f9d34009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:50.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.706+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f9d440555b0 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.896 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.898+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9d4c0b43b0 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.897 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.898+0000 7f9d4a000640 1 -- 172.21.15.53:0/1463310473 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9d44058bd0 con 0x7f9d4c15d4b0 2024-01-16T15:27:50.897 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:27:50.899 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d2406c550 msgr2=0x7f9d2406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d2406c550 0x7f9d2406ea10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9d34007fd0 tx=0x7f9d34009040 comp rx=0 tx=0).stop 2024-01-16T15:27:50.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 msgr2=0x7f9d4c0b8930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:50.900 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 0x7f9d4c0b8930 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400d5d0 tx=0x7f9d4400da90 comp rx=0 tx=0).stop 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 shutdown_connections 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f9d2406c550 0x7f9d2406ea10 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f9d4c15d4b0 0x7f9d4c0b8930 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f9d4c0cf9a0 0x7f9d4c0b83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 --2- 172.21.15.53:0/1463310473 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f9d4c0b31d0 0x7f9d4c15c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 >> 172.21.15.53:0/1463310473 conn(0x7f9d4c0d3000 msgr2=0x7f9d4c06c470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:50.901 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 shutdown_connections 2024-01-16T15:27:50.902 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:50.901+0000 7f9d53200640 1 -- 172.21.15.53:0/1463310473 wait complete. 2024-01-16T15:27:50.919 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:51.142 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.144+0000 7f85cfc00640 1 Processor -- start 2024-01-16T15:27:51.143 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.144+0000 7f85cfc00640 1 -- start start 2024-01-16T15:27:51.144 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.145+0000 7f85cfc00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f85c813df80 0x7f85c815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85cfc00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c806c470 con 0x7f85c813e360 2024-01-16T15:27:51.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85cfc00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85c806c750 con 0x7f85c80cf9a0 2024-01-16T15:27:51.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85cfc00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c806ca30 con 0x7f85c813df80 2024-01-16T15:27:51.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85cd200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f85c813df80 0x7f85c815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85c7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f85c813e360 0x7f85c80cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40238/0 (socket says 172.21.15.53:40238) 2024-01-16T15:27:51.145 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85c7e00640 1 -- 172.21.15.53:0/208934000 learned_addr learned my addr 172.21.15.53:0/208934000 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:51.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85cd200640 1 -- 172.21.15.53:0/208934000 >> v1:172.21.15.146:6789/0 conn(0x7f85c80cf9a0 legacy=0x7f85c80cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:51.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.146+0000 7f85cd200640 1 -- 172.21.15.53:0/208934000 >> v1:172.21.15.53:6789/0 conn(0x7f85c813e360 legacy=0x7f85c80cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.146 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.147+0000 7f85cd200640 1 -- 172.21.15.53:0/208934000 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c806cd10 con 0x7f85c813df80 2024-01-16T15:27:51.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.148+0000 7f85cd200640 1 --2- 172.21.15.53:0/208934000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813df80 0x7f85c815bee0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f85b80097a0 tx=0x7f85b802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d176182c7a8529b5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:51.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.148+0000 7f85c7400640 1 -- 172.21.15.53:0/208934000 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8031830 con 0x7f85c813df80 2024-01-16T15:27:51.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.148+0000 7f85c7400640 1 -- 172.21.15.53:0/208934000 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8031990 con 0x7f85c813df80 2024-01-16T15:27:51.147 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.148+0000 7f85c7400640 1 -- 172.21.15.53:0/208934000 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85b8031d20 con 0x7f85c813df80 2024-01-16T15:27:51.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 -- 172.21.15.53:0/208934000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813df80 msgr2=0x7f85c815bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 --2- 172.21.15.53:0/208934000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813df80 0x7f85c815bee0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f85b80097a0 tx=0x7f85b802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 -- 172.21.15.53:0/208934000 shutdown_connections 2024-01-16T15:27:51.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 --2- 172.21.15.53:0/208934000 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813df80 0x7f85c815bee0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.148 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 -- 172.21.15.53:0/208934000 >> 172.21.15.53:0/208934000 conn(0x7f85c806a800 msgr2=0x7f85c806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:51.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 -- 172.21.15.53:0/208934000 shutdown_connections 2024-01-16T15:27:51.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.149+0000 7f85cfc00640 1 -- 172.21.15.53:0/208934000 wait complete. 2024-01-16T15:27:51.149 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.151+0000 7f85cfc00640 1 Processor -- start 2024-01-16T15:27:51.150 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.151+0000 7f85cfc00640 1 -- start start 2024-01-16T15:27:51.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cfc00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c80cf9a0 0x7f85c8153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cfc00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c813df80 0x7f85c80b3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cfc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 0x7f85c80ab2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cfc00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80a88a0 con 0x7f85c813e360 2024-01-16T15:27:51.151 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cfc00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80a8b80 con 0x7f85c80cf9a0 2024-01-16T15:27:51.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cfc00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c80a8e60 con 0x7f85c813df80 2024-01-16T15:27:51.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cdc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 0x7f85c80ab2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cdc00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 0x7f85c80ab2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33298/0 (socket says 172.21.15.53:33298) 2024-01-16T15:27:51.152 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.152+0000 7f85cdc00640 1 -- 172.21.15.53:0/3408162160 learned_addr learned my addr 172.21.15.53:0/3408162160 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cd200640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c80cf9a0 0x7f85c8153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85c7e00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c813df80 0x7f85c80b3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.153 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cdc00640 1 -- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c813df80 msgr2=0x7f85c80b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cdc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c813df80 0x7f85c80b3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cdc00640 1 -- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c80cf9a0 msgr2=0x7f85c8153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cdc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c80cf9a0 0x7f85c8153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cdc00640 1 -- 172.21.15.53:0/3408162160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b8009400 con 0x7f85c813e360 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85c7e00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c813df80 0x7f85c80b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.153+0000 7f85cdc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 0x7f85c80ab2f0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f85c000b9e0 tx=0x7f85c000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.154+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c0003b70 con 0x7f85c813e360 2024-01-16T15:27:51.154 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.154+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c0003cd0 con 0x7f85c813e360 2024-01-16T15:27:51.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.154+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c000cb60 con 0x7f85c813e360 2024-01-16T15:27:51.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.154+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c80a9140 con 0x7f85c813e360 2024-01-16T15:27:51.155 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.154+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c80a84c0 con 0x7f85c813e360 2024-01-16T15:27:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.155+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c80a9660 con 0x7f85c813e360 2024-01-16T15:27:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.156+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f85c000ccc0 con 0x7f85c813e360 2024-01-16T15:27:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.157+0000 7f85c4c00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f85a006c550 0x7f85a006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.157+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f85c00888e0 con 0x7f85c813e360 2024-01-16T15:27:51.156 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.157+0000 7f85cd200640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f85a006c550 0x7f85a006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.157 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.159+0000 7f85cd200640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f85a006c550 0x7f85a006ea10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f85b8036000 tx=0x7f85b803c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:51.159 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.161+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f85c80a9660 con 0x7f85c813e360 2024-01-16T15:27:51.353 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.354+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85c80a9660 con 0x7f85c813e360 2024-01-16T15:27:51.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.356+0000 7f85c4c00640 1 -- 172.21.15.53:0/3408162160 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f85c80a9660 con 0x7f85c813e360 2024-01-16T15:27:51.354 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:27:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f85a006c550 msgr2=0x7f85a006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f85a006c550 0x7f85a006ea10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f85b8036000 tx=0x7f85b803c040 comp rx=0 tx=0).stop 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 msgr2=0x7f85c80ab2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 0x7f85c80ab2f0 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f85c000b9e0 tx=0x7f85c000bea0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 shutdown_connections 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f85a006c550 0x7f85a006ea10 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f85c813e360 0x7f85c80ab2f0 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f85c813df80 0x7f85c80b3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 --2- 172.21.15.53:0/3408162160 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f85c80cf9a0 0x7f85c8153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.358+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 >> 172.21.15.53:0/3408162160 conn(0x7f85c806a800 msgr2=0x7f85c80ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.359+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 shutdown_connections 2024-01-16T15:27:51.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.359+0000 7f85cfc00640 1 -- 172.21.15.53:0/3408162160 wait complete. 2024-01-16T15:27:51.377 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:51.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.601+0000 7f1c68400640 1 Processor -- start 2024-01-16T15:27:51.600 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.601+0000 7f1c68400640 1 -- start start 2024-01-16T15:27:51.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c68400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f1c6013e360 0x7f1c600cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c68400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f1c6015f9c0 0x7f1c600cf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c68400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6015c5f0 con 0x7f1c6013df80 2024-01-16T15:27:51.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c68400640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6006c470 con 0x7f1c6013e360 2024-01-16T15:27:51.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c68400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c6006c750 con 0x7f1c6015f9c0 2024-01-16T15:27:51.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c65a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f1c6013df80 0x7f1c6015bee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:40252/0 (socket says 172.21.15.53:40252) 2024-01-16T15:27:51.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.603+0000 7f1c65a00640 1 -- 172.21.15.53:0/4233791192 learned_addr learned my addr 172.21.15.53:0/4233791192 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:51.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c65000640 1 --2- 172.21.15.53:0/4233791192 >> v2:172.21.15.53:3300/0 conn(0x7f1c6013e360 0x7f1c600cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c66400640 1 --2- 172.21.15.53:0/4233791192 >> v2:172.21.15.146:3300/0 conn(0x7f1c6015f9c0 0x7f1c600cf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c5fe00640 1 -- 172.21.15.53:0/4233791192 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018058585 0 0) 0x7f1c6015c5f0 con 0x7f1c6013df80 2024-01-16T15:27:51.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c5fe00640 1 -- 172.21.15.53:0/4233791192 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c40003520 con 0x7f1c6013df80 2024-01-16T15:27:51.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c65000640 1 -- 172.21.15.53:0/4233791192 >> v2:172.21.15.146:3300/0 conn(0x7f1c6015f9c0 msgr2=0x7f1c600cf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:51.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c65000640 1 --2- 172.21.15.53:0/4233791192 >> v2:172.21.15.146:3300/0 conn(0x7f1c6015f9c0 0x7f1c600cf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c65000640 1 -- 172.21.15.53:0/4233791192 >> v1:172.21.15.53:6789/0 conn(0x7f1c6013df80 legacy=0x7f1c6015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.604+0000 7f1c65000640 1 -- 172.21.15.53:0/4233791192 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c6006ca30 con 0x7f1c6013e360 2024-01-16T15:27:51.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.605+0000 7f1c65000640 1 --2- 172.21.15.53:0/4233791192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6013e360 0x7f1c600cf290 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f1c600ccae0 tx=0x7f1c4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72eca7d4d24c40b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c5fe00640 1 -- 172.21.15.53:0/4233791192 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48031830 con 0x7f1c6013e360 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c5fe00640 1 -- 172.21.15.53:0/4233791192 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c48031990 con 0x7f1c6013e360 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c5fe00640 1 -- 172.21.15.53:0/4233791192 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c48031d00 con 0x7f1c6013e360 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c68400640 1 -- 172.21.15.53:0/4233791192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6013e360 msgr2=0x7f1c600cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c68400640 1 --2- 172.21.15.53:0/4233791192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6013e360 0x7f1c600cf290 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f1c600ccae0 tx=0x7f1c4802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c68400640 1 -- 172.21.15.53:0/4233791192 shutdown_connections 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c68400640 1 --2- 172.21.15.53:0/4233791192 >> v2:172.21.15.146:3300/0 conn(0x7f1c6015f9c0 0x7f1c600cf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c68400640 1 --2- 172.21.15.53:0/4233791192 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6013e360 0x7f1c600cf290 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.606+0000 7f1c68400640 1 -- 172.21.15.53:0/4233791192 >> 172.21.15.53:0/4233791192 conn(0x7f1c6006a800 msgr2=0x7f1c6006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.607+0000 7f1c68400640 1 -- 172.21.15.53:0/4233791192 shutdown_connections 2024-01-16T15:27:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.607+0000 7f1c68400640 1 -- 172.21.15.53:0/4233791192 wait complete. 2024-01-16T15:27:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.608+0000 7f1c68400640 1 Processor -- start 2024-01-16T15:27:51.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.608+0000 7f1c68400640 1 -- start start 2024-01-16T15:27:51.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.609+0000 7f1c68400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c6013df80 0x7f1c60153890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.609+0000 7f1c68400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c6013e360 0x7f1c600b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.609+0000 7f1c68400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 0x7f1c600b3230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.609+0000 7f1c68400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c60135520 con 0x7f1c6015f9c0 2024-01-16T15:27:51.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.609+0000 7f1c68400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c601325e0 con 0x7f1c6013e360 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.609+0000 7f1c68400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c601328c0 con 0x7f1c6013df80 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 0x7f1c600b3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 0x7f1c600b3230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33310/0 (socket says 172.21.15.53:33310) 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 -- 172.21.15.53:0/4036575666 learned_addr learned my addr 172.21.15.53:0/4036575666 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c65000640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c6013e360 0x7f1c600b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c65a00640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c6013df80 0x7f1c60153890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 -- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c6013df80 msgr2=0x7f1c60153890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c6013df80 0x7f1c60153890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 -- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c6013e360 msgr2=0x7f1c600b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c6013e360 0x7f1c600b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.610+0000 7f1c66400640 1 -- 172.21.15.53:0/4036575666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48009400 con 0x7f1c6015f9c0 2024-01-16T15:27:51.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.611+0000 7f1c66400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 0x7f1c600b3230 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800e850 tx=0x7f1c5800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.611+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c58009630 con 0x7f1c6015f9c0 2024-01-16T15:27:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.611+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5800c940 con 0x7f1c6015f9c0 2024-01-16T15:27:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.611+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5800ccf0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.611+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c60132ba0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.611+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c601f3bb0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.612+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c601330c0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.613+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f1c58013470 con 0x7f1c6015f9c0 2024-01-16T15:27:51.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.614+0000 7f1c5d600640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c3806c550 0x7f1c3806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:51.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.614+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1c58088a20 con 0x7f1c6015f9c0 2024-01-16T15:27:51.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.615+0000 7f1c65a00640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c3806c550 0x7f1c3806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:51.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.616+0000 7f1c65a00640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c3806c550 0x7f1c3806ea10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f1c6013df50 tx=0x7f1c50009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:51.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.619+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1c580552e0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.814+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1c600675d0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.816+0000 7f1c5d600640 1 -- 172.21.15.53:0/4036575666 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1c580170a0 con 0x7f1c6015f9c0 2024-01-16T15:27:51.814 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:27:51.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c3806c550 msgr2=0x7f1c3806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c3806c550 0x7f1c3806ea10 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f1c6013df50 tx=0x7f1c50009040 comp rx=0 tx=0).stop 2024-01-16T15:27:51.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 msgr2=0x7f1c600b3230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:51.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 0x7f1c600b3230 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f1c5800e850 tx=0x7f1c5800ed10 comp rx=0 tx=0).stop 2024-01-16T15:27:51.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 shutdown_connections 2024-01-16T15:27:51.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f1c3806c550 0x7f1c3806ea10 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f1c6015f9c0 0x7f1c600b3230 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f1c6013e360 0x7f1c600b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 --2- 172.21.15.53:0/4036575666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f1c6013df80 0x7f1c60153890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:51.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.818+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 >> 172.21.15.53:0/4036575666 conn(0x7f1c6006a800 msgr2=0x7f1c600afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:51.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.819+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 shutdown_connections 2024-01-16T15:27:51.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:51.819+0000 7f1c68400640 1 -- 172.21.15.53:0/4036575666 wait complete. 2024-01-16T15:27:51.837 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:52.055 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.056+0000 7fd3e6400640 1 Processor -- start 2024-01-16T15:27:52.056 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.057+0000 7fd3e6400640 1 -- start start 2024-01-16T15:27:52.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.058+0000 7fd3e6400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fd3e00cf9a0 0x7fd3e015dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.058+0000 7fd3e6400640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00b1ee0 con 0x7fd3e0143ae0 2024-01-16T15:27:52.057 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.058+0000 7fd3e6400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3e00b2780 con 0x7fd3e015e1e0 2024-01-16T15:27:52.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.058+0000 7fd3e6400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3e015e990 con 0x7fd3e00cf9a0 2024-01-16T15:27:52.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.058+0000 7fd3df400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fd3e0143ae0 0x7fd3e00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:52052/0 (socket says 172.21.15.53:52052) 2024-01-16T15:27:52.058 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.058+0000 7fd3df400640 1 -- 172.21.15.53:0/3664652345 learned_addr learned my addr 172.21.15.53:0/3664652345 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:52.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.059+0000 7fd3dea00640 1 --2- 172.21.15.53:0/3664652345 >> v2:172.21.15.146:3300/0 conn(0x7fd3e00cf9a0 0x7fd3e015dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.059+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366629297 0 0) 0x7fd3e00b1ee0 con 0x7fd3e0143ae0 2024-01-16T15:27:52.060 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.059+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3c0003520 con 0x7fd3e0143ae0 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.059+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2190245459 0 0) 0x7fd3c0003520 con 0x7fd3e0143ae0 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.059+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 >> v2:172.21.15.146:3300/0 conn(0x7fd3e00cf9a0 msgr2=0x7fd3e015dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.059+0000 7fd3de000640 1 --2- 172.21.15.53:0/3664652345 >> v2:172.21.15.146:3300/0 conn(0x7fd3e00cf9a0 0x7fd3e015dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 >> v1:172.21.15.146:6789/0 conn(0x7fd3e015e1e0 legacy=0x7fd3e00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e00b2130 con 0x7fd3e0143ae0 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fd3d0001670 con 0x7fd3e0143ae0 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 >> v1:172.21.15.53:6789/0 conn(0x7fd3e0143ae0 legacy=0x7fd3e00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.061 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3c00039c0 0x7fd3c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c00047c0 0x7fd3c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c0005180 0x7fd3c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e015e990 con 0x7fd3c00039c0 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c000c080 con 0x7fd3c0005180 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3c000c240 con 0x7fd3c00047c0 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3df400640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3c00039c0 0x7fd3c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3df400640 1 -- 172.21.15.53:0/3664652345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c00047c0 msgr2=0x7fd3c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3df400640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3c00047c0 0x7fd3c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.062 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3df400640 1 -- 172.21.15.53:0/3664652345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c0005180 msgr2=0x7fd3c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:52.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3df400640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c0005180 0x7fd3c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3df400640 1 -- 172.21.15.53:0/3664652345 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c0003520 con 0x7fd3c00039c0 2024-01-16T15:27:52.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.060+0000 7fd3dfe00640 1 -- 172.21.15.53:0/3664652345 reap_dead start 2024-01-16T15:27:52.063 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.062+0000 7fd3df400640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3c00039c0 0x7fd3c0003de0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000c0e0 tx=0x7fd3c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3da55e6976fc9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.062+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d0005ad0 con 0x7fd3c00039c0 2024-01-16T15:27:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.062+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3d0005c30 con 0x7fd3c00039c0 2024-01-16T15:27:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.062+0000 7fd3de000640 1 -- 172.21.15.53:0/3664652345 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3d002ca30 con 0x7fd3c00039c0 2024-01-16T15:27:52.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 -- 172.21.15.53:0/3664652345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3c00039c0 msgr2=0x7fd3c0003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3c00039c0 0x7fd3c0003de0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7fd3d000c0e0 tx=0x7fd3c00057f0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 -- 172.21.15.53:0/3664652345 shutdown_connections 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3664652345 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3c00039c0 0x7fd3c0003de0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 -- 172.21.15.53:0/3664652345 >> 172.21.15.53:0/3664652345 conn(0x7fd3e006a800 msgr2=0x7fd3e006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 -- 172.21.15.53:0/3664652345 shutdown_connections 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.063+0000 7fd3e6400640 1 -- 172.21.15.53:0/3664652345 wait complete. 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.065+0000 7fd3e6400640 1 Processor -- start 2024-01-16T15:27:52.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.065+0000 7fd3e6400640 1 -- start start 2024-01-16T15:27:52.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.066+0000 7fd3e6400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c00039c0 0x7fd3e013a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.066+0000 7fd3e6400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 0x7fd3e013bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.066+0000 7fd3e6400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3e013c510 0x7fd3e01395b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.066+0000 7fd3e6400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e013d550 con 0x7fd3e0159cb0 2024-01-16T15:27:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.066+0000 7fd3e6400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e013c9d0 con 0x7fd3c00039c0 2024-01-16T15:27:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.066+0000 7fd3e6400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e013ccb0 con 0x7fd3e013c510 2024-01-16T15:27:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 0x7fd3e013bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 0x7fd3e013bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58936/0 (socket says 172.21.15.53:58936) 2024-01-16T15:27:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 -- 172.21.15.53:0/3837175130 learned_addr learned my addr 172.21.15.53:0/3837175130 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:52.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3df400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c00039c0 0x7fd3e013a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 -- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3e013c510 msgr2=0x7fd3e01395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3e013c510 0x7fd3e01395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 -- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c00039c0 msgr2=0x7fd3e013a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c00039c0 0x7fd3e013a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 -- 172.21.15.53:0/3837175130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3e013cf90 con 0x7fd3e0159cb0 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.067+0000 7fd3dea00640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 0x7fd3e013bfd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800e770 tx=0x7fd3c800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.068+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800c730 con 0x7fd3e0159cb0 2024-01-16T15:27:52.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.068+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c800c890 con 0x7fd3e0159cb0 2024-01-16T15:27:52.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.068+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c800cc40 con 0x7fd3e0159cb0 2024-01-16T15:27:52.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.068+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e013d270 con 0x7fd3e0159cb0 2024-01-16T15:27:52.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.068+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e013aaf0 con 0x7fd3e0159cb0 2024-01-16T15:27:52.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.069+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e013d6e0 con 0x7fd3e0159cb0 2024-01-16T15:27:52.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.070+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd3c8013470 con 0x7fd3e0159cb0 2024-01-16T15:27:52.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.071+0000 7fd3d7400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3b806c580 0x7fd3b806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.071+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd3c80887c0 con 0x7fd3e0159cb0 2024-01-16T15:27:52.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.072+0000 7fd3df400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3b806c580 0x7fd3b806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.073+0000 7fd3df400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3b806c580 0x7fd3b806ea40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0034000 tx=0x7fd3d0009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:52.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.076+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fd3c8055080 con 0x7fd3e0159cb0 2024-01-16T15:27:52.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.284+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3e013aaf0 con 0x7fd3e0159cb0 2024-01-16T15:27:52.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.285+0000 7fd3d7400640 1 -- 172.21.15.53:0/3837175130 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v112) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd3e013aaf0 con 0x7fd3e0159cb0 2024-01-16T15:27:52.283 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:27:52.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3b806c580 msgr2=0x7fd3b806ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3b806c580 0x7fd3b806ea40 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fd3d0034000 tx=0x7fd3d0009030 comp rx=0 tx=0).stop 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 msgr2=0x7fd3e013bfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 0x7fd3e013bfd0 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fd3c800e770 tx=0x7fd3c800ec30 comp rx=0 tx=0).stop 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 shutdown_connections 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd3b806c580 0x7fd3b806ea40 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd3e013c510 0x7fd3e01395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd3e0159cb0 0x7fd3e013bfd0 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.287+0000 7fd3e6400640 1 --2- 172.21.15.53:0/3837175130 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd3c00039c0 0x7fd3e013a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.288+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 >> 172.21.15.53:0/3837175130 conn(0x7fd3e006a800 msgr2=0x7fd3e00aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.288+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 shutdown_connections 2024-01-16T15:27:52.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.288+0000 7fd3e6400640 1 -- 172.21.15.53:0/3837175130 wait complete. 2024-01-16T15:27:52.307 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:27:52.307 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:27:52.364 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:27:52.593 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.594+0000 7f6936e00640 1 Processor -- start 2024-01-16T15:27:52.594 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.594+0000 7f6936e00640 1 -- start start 2024-01-16T15:27:52.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.595+0000 7f6936e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930160920 0x7f69300b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.595+0000 7f6936e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69300cf9f0 0x7f69300cfe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6936e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69300d0a10 con 0x7f6930161010 2024-01-16T15:27:52.595 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6936e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69300d03b0 con 0x7f69300cf9f0 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6936e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69300d1f50 con 0x7f6930160920 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6935000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f69300cf9f0 0x7f69300cfe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6936400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f6930161010 0x7f69301614a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:52062/0 (socket says 172.21.15.53:52062) 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6936400640 1 -- 172.21.15.53:0/1114882796 learned_addr learned my addr 172.21.15.53:0/1114882796 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6935a00640 1 --2- 172.21.15.53:0/1114882796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930160920 0x7f69300b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6935000640 1 -- 172.21.15.53:0/1114882796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930160920 msgr2=0x7f69300b3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6935000640 1 --2- 172.21.15.53:0/1114882796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930160920 0x7f69300b3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.596 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6935000640 1 -- 172.21.15.53:0/1114882796 >> v1:172.21.15.53:6789/0 conn(0x7f6930161010 legacy=0x7f69301614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.597 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.596+0000 7f6935000640 1 -- 172.21.15.53:0/1114882796 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69300cf080 con 0x7f69300cf9f0 2024-01-16T15:27:52.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.598+0000 7f6935000640 1 --2- 172.21.15.53:0/1114882796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69300cf9f0 0x7f69300cfe70 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f69180097a0 tx=0x7f691802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=643ce2b2a19c8e29 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:52.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.598+0000 7f692fe00640 1 -- 172.21.15.53:0/1114882796 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691802d040 con 0x7f69300cf9f0 2024-01-16T15:27:52.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.598+0000 7f692fe00640 1 -- 172.21.15.53:0/1114882796 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6918031810 con 0x7f69300cf9f0 2024-01-16T15:27:52.598 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.598+0000 7f692fe00640 1 -- 172.21.15.53:0/1114882796 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6918031ba0 con 0x7f69300cf9f0 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.598+0000 7f6936e00640 1 -- 172.21.15.53:0/1114882796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69300cf9f0 msgr2=0x7f69300cfe70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.598+0000 7f6936e00640 1 --2- 172.21.15.53:0/1114882796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69300cf9f0 0x7f69300cfe70 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f69180097a0 tx=0x7f691802b8b0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.599+0000 7f6936e00640 1 -- 172.21.15.53:0/1114882796 shutdown_connections 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.599+0000 7f6936e00640 1 --2- 172.21.15.53:0/1114882796 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f69300cf9f0 0x7f69300cfe70 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.599+0000 7f6936e00640 1 --2- 172.21.15.53:0/1114882796 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f6930160920 0x7f69300b3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.599+0000 7f6936e00640 1 -- 172.21.15.53:0/1114882796 >> 172.21.15.53:0/1114882796 conn(0x7f69300d3090 msgr2=0x7f69300d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.599+0000 7f6936e00640 1 -- 172.21.15.53:0/1114882796 shutdown_connections 2024-01-16T15:27:52.599 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.599+0000 7f6936e00640 1 -- 172.21.15.53:0/1114882796 wait complete. 2024-01-16T15:27:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.600+0000 7f6936e00640 1 Processor -- start 2024-01-16T15:27:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.601+0000 7f6936e00640 1 -- start start 2024-01-16T15:27:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300cf9f0 0x7f69300b7900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6930160920 0x7f693014b280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.602 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 0x7f6930157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6930155930 con 0x7f6930161010 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b7e40 con 0x7f6930160920 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69300b6860 con 0x7f69300cf9f0 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 0x7f6930157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 0x7f6930157010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58960/0 (socket says 172.21.15.53:58960) 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936400640 1 -- 172.21.15.53:0/31060215 learned_addr learned my addr 172.21.15.53:0/31060215 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:52.603 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6935000640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6930160920 0x7f693014b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6936400640 1 -- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300cf9f0 msgr2=0x7f69300b7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.602+0000 7f6935a00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300cf9f0 0x7f69300b7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936400640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300cf9f0 0x7f69300b7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936400640 1 -- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6930160920 msgr2=0x7f693014b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936400640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6930160920 0x7f693014b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936400640 1 -- 172.21.15.53:0/31060215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6918009400 con 0x7f6930161010 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6935000640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6930160920 0x7f693014b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:52.604 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6935a00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300cf9f0 0x7f69300b7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936400640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 0x7f6930157010 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f692800cbc0 tx=0x7f692800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692800ad40 con 0x7f6930161010 2024-01-16T15:27:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6928012400 con 0x7f6930161010 2024-01-16T15:27:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f692800aea0 con 0x7f6930161010 2024-01-16T15:27:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69300b6b40 con 0x7f6930161010 2024-01-16T15:27:52.605 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.603+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69300b7100 con 0x7f6930161010 2024-01-16T15:27:52.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.605+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6930067450 con 0x7f6930161010 2024-01-16T15:27:52.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.606+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f6928007070 con 0x7f6930161010 2024-01-16T15:27:52.606 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.606+0000 7f692d600640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f690806c550 0x7f690806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:52.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.607+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f6928088a40 con 0x7f6930161010 2024-01-16T15:27:52.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.607+0000 7f6935a00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f690806c550 0x7f690806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:52.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.608+0000 7f6935a00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f690806c550 0x7f690806ea10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6920006fd0 tx=0x7f6920008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:52.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.611+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f6928054fa0 con 0x7f6930161010 2024-01-16T15:27:52.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.806+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f693008c7e0 con 0x7f6930161010 2024-01-16T15:27:52.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.809+0000 7f692d600640 1 -- 172.21.15.53:0/31060215 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16631 (secure 0 0 0) 0x7f6928017030 con 0x7f6930161010 2024-01-16T15:27:52.808 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:27:52.809 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":112,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:42.134132+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.24","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:27:52.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f690806c550 msgr2=0x7f690806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f690806c550 0x7f690806ea10 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f6920006fd0 tx=0x7f6920008040 comp rx=0 tx=0).stop 2024-01-16T15:27:52.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 msgr2=0x7f6930157010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:52.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 0x7f6930157010 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7f692800cbc0 tx=0x7f692800a430 comp rx=0 tx=0).stop 2024-01-16T15:27:52.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 shutdown_connections 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f690806c550 0x7f690806ea10 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f6930161010 0x7f6930157010 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f6930160920 0x7f693014b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 --2- 172.21.15.53:0/31060215 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f69300cf9f0 0x7f69300b7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 >> 172.21.15.53:0/31060215 conn(0x7f69300d3090 msgr2=0x7f693006c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 shutdown_connections 2024-01-16T15:27:52.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:52.812+0000 7f6936e00640 1 -- 172.21.15.53:0/31060215 wait complete. 2024-01-16T15:27:52.832 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-01-16T14:59:46.242799+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2024-01-16T15:19:21.949075+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2024-01-16T15:19:23.088954+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-01-16T15:27:52.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-01-16T15:27:53.064 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.065+0000 7fc4ab400640 1 Processor -- start 2024-01-16T15:27:53.065 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.066+0000 7fc4ab400640 1 -- start start 2024-01-16T15:27:53.066 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4ab400640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc4a406c730 0x7fc4a40b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4ab400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a4071970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4ab400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a40708d0 con 0x7fc4a406cb10 2024-01-16T15:27:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4ab400640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a4070bb0 con 0x7fc4a406c730 2024-01-16T15:27:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4ab400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a4070e90 con 0x7fc4a4070160 2024-01-16T15:27:53.067 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4a9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a4071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.068+0000 7fc4aa000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fc4a406c730 0x7fc4a40b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4a9600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a4071970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:41650/0 (socket says 172.21.15.53:41650) 2024-01-16T15:27:53.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4a9600640 1 -- 172.21.15.53:0/1832044431 learned_addr learned my addr 172.21.15.53:0/1832044431 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:53.068 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4a9600640 1 -- 172.21.15.53:0/1832044431 >> v2:172.21.15.146:3300/0 conn(0x7fc4a406c730 msgr2=0x7fc4a40b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:53.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4a9600640 1 --2- 172.21.15.53:0/1832044431 >> v2:172.21.15.146:3300/0 conn(0x7fc4a406c730 0x7fc4a40b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4a9600640 1 -- 172.21.15.53:0/1832044431 >> v1:172.21.15.146:6789/0 conn(0x7fc4a406cb10 legacy=0x7fc4a406cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4a9600640 1 -- 172.21.15.53:0/1832044431 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a4071170 con 0x7fc4a4070160 2024-01-16T15:27:53.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.069+0000 7fc4aa000640 1 --2- 172.21.15.53:0/1832044431 >> v2:172.21.15.146:3300/0 conn(0x7fc4a406c730 0x7fc4a40b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-01-16T15:27:53.069 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.071+0000 7fc4a9600640 1 --2- 172.21.15.53:0/1832044431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a4071970 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009870 tx=0x7fc48c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eaaedefb52aa66d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.071+0000 7fc4a8c00640 1 -- 172.21.15.53:0/1832044431 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c031640 con 0x7fc4a4070160 2024-01-16T15:27:53.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.071+0000 7fc4a8c00640 1 -- 172.21.15.53:0/1832044431 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c0317a0 con 0x7fc4a4070160 2024-01-16T15:27:53.070 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.071+0000 7fc4a8c00640 1 -- 172.21.15.53:0/1832044431 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c031b30 con 0x7fc4a4070160 2024-01-16T15:27:53.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 -- 172.21.15.53:0/1832044431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 msgr2=0x7fc4a4071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 --2- 172.21.15.53:0/1832044431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a4071970 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009870 tx=0x7fc48c02b8f0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 -- 172.21.15.53:0/1832044431 shutdown_connections 2024-01-16T15:27:53.071 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 --2- 172.21.15.53:0/1832044431 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a4071970 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 --2- 172.21.15.53:0/1832044431 >> v2:172.21.15.146:3300/0 conn(0x7fc4a406c730 0x7fc4a40b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 -- 172.21.15.53:0/1832044431 >> 172.21.15.53:0/1832044431 conn(0x7fc4a406a7d0 msgr2=0x7fc4a406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:53.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 -- 172.21.15.53:0/1832044431 shutdown_connections 2024-01-16T15:27:53.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.072+0000 7fc4ab400640 1 -- 172.21.15.53:0/1832044431 wait complete. 2024-01-16T15:27:53.072 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.074+0000 7fc4ab400640 1 Processor -- start 2024-01-16T15:27:53.073 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.074+0000 7fc4ab400640 1 -- start start 2024-01-16T15:27:53.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4ab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 0x7fc4a41fa100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4ab400640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a406cb10 0x7fc4a40afa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4ab400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a40affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.074 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4ab400640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a41f74c0 con 0x7fc4a406c730 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4ab400640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a41f77a0 con 0x7fc4a406cb10 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4ab400640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a41f7a80 con 0x7fc4a4070160 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4aa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 0x7fc4a41fa100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4aa000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 0x7fc4a41fa100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58970/0 (socket says 172.21.15.53:58970) 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.075+0000 7fc4aa000640 1 -- 172.21.15.53:0/4258238483 learned_addr learned my addr 172.21.15.53:0/4258238483 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4a9600640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a406cb10 0x7fc4a40afa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.075 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aaa00640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a40affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aa000640 1 -- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 msgr2=0x7fc4a40affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.076 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aa000640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a40affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aa000640 1 -- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a406cb10 msgr2=0x7fc4a40afa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aa000640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a406cb10 0x7fc4a40afa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aa000640 1 -- 172.21.15.53:0/4258238483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc494009400 con 0x7fc4a406c730 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.076+0000 7fc4aa000640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 0x7fc4a41fa100 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fc49400ebe0 tx=0x7fc49400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.077+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc494009760 con 0x7fc4a406c730 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.077+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc49400cc30 con 0x7fc4a406c730 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.077+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49400eee0 con 0x7fc4a406c730 2024-01-16T15:27:53.077 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.077+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc48c0094d0 con 0x7fc4a406c730 2024-01-16T15:27:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.077+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a41f8040 con 0x7fc4a406c730 2024-01-16T15:27:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.078+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a4067450 con 0x7fc4a406c730 2024-01-16T15:27:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.079+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fc494013470 con 0x7fc4a406c730 2024-01-16T15:27:53.078 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.080+0000 7fc4a2000640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc47c06c550 0x7fc47c06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.080+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fc494088b30 con 0x7fc4a406c730 2024-01-16T15:27:53.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.080+0000 7fc4a9600640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc47c06c550 0x7fc47c06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.082+0000 7fc4a9600640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc47c06c550 0x7fc47c06ea10 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc48c036000 tx=0x7fc48c03c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.084+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fc4940553c0 con 0x7fc4a406c730 2024-01-16T15:27:53.276 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.277+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4a40af570 con 0x7fc4a406c730 2024-01-16T15:27:53.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.278+0000 7fc4a2000640 1 -- 172.21.15.53:0/4258238483 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc494021020 con 0x7fc4a406c730 2024-01-16T15:27:53.277 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 1 2024-01-16T15:27:53.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc47c06c550 msgr2=0x7fc47c06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc47c06c550 0x7fc47c06ea10 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fc48c036000 tx=0x7fc48c03c040 comp rx=0 tx=0).stop 2024-01-16T15:27:53.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 msgr2=0x7fc4a41fa100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 0x7fc4a41fa100 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fc49400ebe0 tx=0x7fc49400c520 comp rx=0 tx=0).stop 2024-01-16T15:27:53.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 shutdown_connections 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fc47c06c550 0x7fc47c06ea10 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fc4a4070160 0x7fc4a40affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fc4a406cb10 0x7fc4a40afa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 --2- 172.21.15.53:0/4258238483 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fc4a406c730 0x7fc4a41fa100 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 >> 172.21.15.53:0/4258238483 conn(0x7fc4a406a7d0 msgr2=0x7fc4a40cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 shutdown_connections 2024-01-16T15:27:53.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.281+0000 7fc4ab400640 1 -- 172.21.15.53:0/4258238483 wait complete. 2024-01-16T15:27:53.300 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-01-16T15:27:53.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.533+0000 7f8c1f200640 1 Processor -- start 2024-01-16T15:27:53.532 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.533+0000 7f8c1f200640 1 -- start start 2024-01-16T15:27:53.533 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.534+0000 7f8c1f200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f8c200ccb50 0x7f8c20070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.534+0000 7f8c1f200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f8c2015bf10 0x7f8c20143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.534+0000 7f8c1f200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c2015f4e0 con 0x7f8c200adf20 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.534+0000 7f8c1f200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c2015f970 con 0x7f8c2015bf10 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.534+0000 7f8c1f200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c20071130 con 0x7f8c200ccb50 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1de00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f8c200adf20 0x7f8c200cc440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:52064/0 (socket says 172.21.15.53:52064) 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1de00640 1 -- 172.21.15.53:0/34307357 learned_addr learned my addr 172.21.15.53:0/34307357 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:53.534 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1e800640 1 --2- 172.21.15.53:0/34307357 >> v2:172.21.15.53:3300/0 conn(0x7f8c2015bf10 0x7f8c20143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c15200640 1 --2- 172.21.15.53:0/34307357 >> v2:172.21.15.146:3300/0 conn(0x7f8c200ccb50 0x7f8c20070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1d400640 1 -- 172.21.15.53:0/34307357 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774254757 0 0) 0x7f8c2015f4e0 con 0x7f8c200adf20 2024-01-16T15:27:53.535 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1d400640 1 -- 172.21.15.53:0/34307357 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c00003520 con 0x7f8c200adf20 2024-01-16T15:27:53.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1e800640 1 -- 172.21.15.53:0/34307357 >> v2:172.21.15.146:3300/0 conn(0x7f8c200ccb50 msgr2=0x7f8c20070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:53.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1e800640 1 --2- 172.21.15.53:0/34307357 >> v2:172.21.15.146:3300/0 conn(0x7f8c200ccb50 0x7f8c20070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1e800640 1 -- 172.21.15.53:0/34307357 >> v1:172.21.15.53:6789/0 conn(0x7f8c200adf20 legacy=0x7f8c200cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.535+0000 7f8c1e800640 1 -- 172.21.15.53:0/34307357 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c2015ebb0 con 0x7f8c2015bf10 2024-01-16T15:27:53.536 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.537+0000 7f8c1e800640 1 --2- 172.21.15.53:0/34307357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c2015bf10 0x7f8c20143d30 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f8c2015e9e0 tx=0x7f8c1802be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e423029a5c774aa9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.537+0000 7f8c1d400640 1 -- 172.21.15.53:0/34307357 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1802cbf0 con 0x7f8c2015bf10 2024-01-16T15:27:53.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.537+0000 7f8c1d400640 1 -- 172.21.15.53:0/34307357 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1802cd50 con 0x7f8c2015bf10 2024-01-16T15:27:53.537 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.537+0000 7f8c1d400640 1 -- 172.21.15.53:0/34307357 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c18034660 con 0x7f8c2015bf10 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.537+0000 7f8c1f200640 1 -- 172.21.15.53:0/34307357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c2015bf10 msgr2=0x7f8c20143d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.537+0000 7f8c1f200640 1 --2- 172.21.15.53:0/34307357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c2015bf10 0x7f8c20143d30 secure :-1 s=READY pgs=2008 cs=0 l=1 rev1=1 crypto rx=0x7f8c2015e9e0 tx=0x7f8c1802be40 comp rx=0 tx=0).stop 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.538+0000 7f8c1f200640 1 -- 172.21.15.53:0/34307357 shutdown_connections 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.538+0000 7f8c1f200640 1 --2- 172.21.15.53:0/34307357 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c2015bf10 0x7f8c20143d30 unknown :-1 s=CLOSED pgs=2008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.538+0000 7f8c1f200640 1 --2- 172.21.15.53:0/34307357 >> v2:172.21.15.146:3300/0 conn(0x7f8c200ccb50 0x7f8c20070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.538+0000 7f8c1f200640 1 -- 172.21.15.53:0/34307357 >> 172.21.15.53:0/34307357 conn(0x7f8c2006a7d0 msgr2=0x7f8c2006abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.538+0000 7f8c1f200640 1 -- 172.21.15.53:0/34307357 shutdown_connections 2024-01-16T15:27:53.538 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.538+0000 7f8c1f200640 1 -- 172.21.15.53:0/34307357 wait complete. 2024-01-16T15:27:53.539 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.540+0000 7f8c1f200640 1 Processor -- start 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.540+0000 7f8c1f200640 1 -- start start 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1f200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 0x7f8c200b3840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1f200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c200ccb50 0x7f8c200ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1f200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c2015bf10 0x7f8c200ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1f200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200b3d80 con 0x7f8c200adf20 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1f200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200c4ab0 con 0x7f8c200ccb50 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1f200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c200c3ea0 con 0x7f8c2015bf10 2024-01-16T15:27:53.540 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 0x7f8c200b3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 0x7f8c200b3840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:58994/0 (socket says 172.21.15.53:58994) 2024-01-16T15:27:53.541 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 -- 172.21.15.53:0/3734897397 learned_addr learned my addr 172.21.15.53:0/3734897397 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:53.542 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 -- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c2015bf10 msgr2=0x7f8c200ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c2015bf10 0x7f8c200ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 -- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c200ccb50 msgr2=0x7f8c200ac8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c200ccb50 0x7f8c200ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.541+0000 7f8c1de00640 1 -- 172.21.15.53:0/3734897397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c18009a70 con 0x7f8c200adf20 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.542+0000 7f8c1de00640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 0x7f8c200b3840 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000c530 tx=0x7f8c1000c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.542+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10005ce0 con 0x7f8c200adf20 2024-01-16T15:27:53.543 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.542+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c10005e40 con 0x7f8c200adf20 2024-01-16T15:27:53.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.542+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c100072b0 con 0x7f8c200adf20 2024-01-16T15:27:53.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.542+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c200c4180 con 0x7f8c200adf20 2024-01-16T15:27:53.544 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.542+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c200c4740 con 0x7f8c200adf20 2024-01-16T15:27:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.544+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c20067450 con 0x7f8c200adf20 2024-01-16T15:27:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.545+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f8c10007410 con 0x7f8c200adf20 2024-01-16T15:27:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.546+0000 7f8c16a00640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bf806c550 0x7f8bf806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.546+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f8c10088150 con 0x7f8c200adf20 2024-01-16T15:27:53.545 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.546+0000 7f8c15200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bf806c550 0x7f8bf806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:53.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.548+0000 7f8c15200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bf806c550 0x7f8bf806ea10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8c200ad470 tx=0x7f8c08008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:53.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.550+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f8c10054750 con 0x7f8c200adf20 2024-01-16T15:27:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.746+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8c20070370 con 0x7f8c200adf20 2024-01-16T15:27:53.745 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.746+0000 7f8c16a00640 1 -- 172.21.15.53:0/3734897397 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8c10017020 con 0x7f8c200adf20 2024-01-16T15:27:53.745 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 32 2024-01-16T15:27:53.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bf806c550 msgr2=0x7f8bf806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bf806c550 0x7f8bf806ea10 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8c200ad470 tx=0x7f8c08008040 comp rx=0 tx=0).stop 2024-01-16T15:27:53.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 msgr2=0x7f8c200b3840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 0x7f8c200b3840 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f8c1000c530 tx=0x7f8c1000c9f0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 shutdown_connections 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f8bf806c550 0x7f8bf806ea10 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f8c2015bf10 0x7f8c200ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f8c200ccb50 0x7f8c200ac8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 --2- 172.21.15.53:0/3734897397 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f8c200adf20 0x7f8c200b3840 unknown :-1 s=CLOSED pgs=2009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.749+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 >> 172.21.15.53:0/3734897397 conn(0x7f8c2006a7d0 msgr2=0x7f8c2006d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:53.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.750+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 shutdown_connections 2024-01-16T15:27:53.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.750+0000 7f8c1f200640 1 -- 172.21.15.53:0/3734897397 wait complete. 2024-01-16T15:27:53.769 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-01-16T15:27:53.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.998+0000 7f16b1800640 1 Processor -- start 2024-01-16T15:27:53.997 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:53.998+0000 7f16b1800640 1 -- start start 2024-01-16T15:27:53.999 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16b1800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16ac0adf20 0x7f16ac070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16b1800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f16ac15bf10 0x7f16ac0cc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16b1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac0cc980 0x7f16ac143d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16b1800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac15f970 con 0x7f16ac0adf20 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16b1800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ac15f4e0 con 0x7f16ac15bf10 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16b1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0cce10 con 0x7f16ac0cc980 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16abe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16ac0adf20 0x7f16ac070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16abe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f16ac0adf20 0x7f16ac070a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59000/0 (socket says 172.21.15.53:59000) 2024-01-16T15:27:54.000 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.000+0000 7f16abe00640 1 -- 172.21.15.53:0/372074669 learned_addr learned my addr 172.21.15.53:0/372074669 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16ab400640 1 --2- 172.21.15.53:0/372074669 >> v2:172.21.15.146:3300/0 conn(0x7f16ac15bf10 0x7f16ac0cc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16b0e00640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac0cc980 0x7f16ac143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16abe00640 1 -- 172.21.15.53:0/372074669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac0cc980 msgr2=0x7f16ac143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16abe00640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac0cc980 0x7f16ac143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16abe00640 1 -- 172.21.15.53:0/372074669 >> v2:172.21.15.146:3300/0 conn(0x7f16ac15bf10 msgr2=0x7f16ac0cc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16abe00640 1 --2- 172.21.15.53:0/372074669 >> v2:172.21.15.146:3300/0 conn(0x7f16ac15bf10 0x7f16ac0cc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16abe00640 1 -- 172.21.15.53:0/372074669 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac071130 con 0x7f16ac0adf20 2024-01-16T15:27:54.001 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.001+0000 7f16b0e00640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac0cc980 0x7f16ac143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:54.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.003+0000 7f16abe00640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0adf20 0x7f16ac070a30 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f16ac15e9e0 tx=0x7f169c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a8bf51c4b0acdd server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:54.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.003+0000 7f16aaa00640 1 -- 172.21.15.53:0/372074669 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c0317e0 con 0x7f16ac0adf20 2024-01-16T15:27:54.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.003+0000 7f16aaa00640 1 -- 172.21.15.53:0/372074669 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c031940 con 0x7f16ac0adf20 2024-01-16T15:27:54.002 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.003+0000 7f16aaa00640 1 -- 172.21.15.53:0/372074669 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c031cd0 con 0x7f16ac0adf20 2024-01-16T15:27:54.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 -- 172.21.15.53:0/372074669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0adf20 msgr2=0x7f16ac070a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0adf20 0x7f16ac070a30 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f16ac15e9e0 tx=0x7f169c02b880 comp rx=0 tx=0).stop 2024-01-16T15:27:54.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 -- 172.21.15.53:0/372074669 shutdown_connections 2024-01-16T15:27:54.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac0cc980 0x7f16ac143d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.003 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 --2- 172.21.15.53:0/372074669 >> v2:172.21.15.146:3300/0 conn(0x7f16ac15bf10 0x7f16ac0cc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 --2- 172.21.15.53:0/372074669 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0adf20 0x7f16ac070a30 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 -- 172.21.15.53:0/372074669 >> 172.21.15.53:0/372074669 conn(0x7f16ac06a7d0 msgr2=0x7f16ac06abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:54.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 -- 172.21.15.53:0/372074669 shutdown_connections 2024-01-16T15:27:54.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.004+0000 7f16b1800640 1 -- 172.21.15.53:0/372074669 wait complete. 2024-01-16T15:27:54.004 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.006+0000 7f16b1800640 1 Processor -- start 2024-01-16T15:27:54.005 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.006+0000 7f16b1800640 1 -- start start 2024-01-16T15:27:54.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16b1800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16ac0adf20 0x7f16ac0cb550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16b1800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 0x7f16ac1f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16b1800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac15bf10 0x7f16ac15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16b1800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac1f3680 con 0x7f16ac0cc980 2024-01-16T15:27:54.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16b1800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0ca4b0 con 0x7f16ac0adf20 2024-01-16T15:27:54.007 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16b1800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ac0ca790 con 0x7f16ac15bf10 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16ab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 0x7f16ac1f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16ab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 0x7f16ac1f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59002/0 (socket says 172.21.15.53:59002) 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.007+0000 7f16ab400640 1 -- 172.21.15.53:0/1276254098 learned_addr learned my addr 172.21.15.53:0/1276254098 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16abe00640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16ac0adf20 0x7f16ac0cb550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16b0e00640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac15bf10 0x7f16ac15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16ab400640 1 -- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac15bf10 msgr2=0x7f16ac15a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16ab400640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac15bf10 0x7f16ac15a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16ab400640 1 -- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16ac0adf20 msgr2=0x7f16ac0cb550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.008 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16ab400640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16ac0adf20 0x7f16ac0cb550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16ab400640 1 -- 172.21.15.53:0/1276254098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169c0094d0 con 0x7f16ac0cc980 2024-01-16T15:27:54.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16abe00640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16ac0adf20 0x7f16ac0cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:27:54.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.008+0000 7f16ab400640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 0x7f16ac1f3140 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f169400d800 tx=0x7f169400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.009+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169400b6c0 con 0x7f16ac0cc980 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.009+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169400b820 con 0x7f16ac0cc980 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.009+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169400bbb0 con 0x7f16ac0cc980 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.009+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac0caa70 con 0x7f16ac0cc980 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.009+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac0cb030 con 0x7f16ac0cc980 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.010+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac0cb1f0 con 0x7f16ac0cc980 2024-01-16T15:27:54.010 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.011+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f169400bd10 con 0x7f16ac0cc980 2024-01-16T15:27:54.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.012+0000 7f16a3e00640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168406c550 0x7f168406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.012+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f1694089090 con 0x7f16ac0cc980 2024-01-16T15:27:54.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.013+0000 7f16abe00640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168406c550 0x7f168406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.014+0000 7f16abe00640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168406c550 0x7f168406ea10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f169c036000 tx=0x7f169c03d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:54.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.018+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f1694055630 con 0x7f16ac0cc980 2024-01-16T15:27:54.214 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.215+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f16ac067450 con 0x7f16ac0cc980 2024-01-16T15:27:54.215 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.216+0000 7f16a3e00640 1 -- 172.21.15.53:0/1276254098 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1694058c50 con 0x7f16ac0cc980 2024-01-16T15:27:54.215 INFO:teuthology.orchestra.run.smithi053.stdout:pg_num: 64 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.219+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168406c550 msgr2=0x7f168406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.219+0000 7f16b1800640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168406c550 0x7f168406ea10 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f169c036000 tx=0x7f169c03d040 comp rx=0 tx=0).stop 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.219+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 msgr2=0x7f16ac1f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.219+0000 7f16b1800640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 0x7f16ac1f3140 secure :-1 s=READY pgs=2011 cs=0 l=1 rev1=1 crypto rx=0x7f169400d800 tx=0x7f169400dcc0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 shutdown_connections 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f168406c550 0x7f168406ea10 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f16ac15bf10 0x7f16ac15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f16ac0cc980 0x7f16ac1f3140 unknown :-1 s=CLOSED pgs=2011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 --2- 172.21.15.53:0/1276254098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f16ac0adf20 0x7f16ac0cb550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 >> 172.21.15.53:0/1276254098 conn(0x7f16ac06a7d0 msgr2=0x7f16ac06d6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 shutdown_connections 2024-01-16T15:27:54.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.220+0000 7f16b1800640 1 -- 172.21.15.53:0/1276254098 wait complete. 2024-01-16T15:27:54.238 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-01-16T15:27:54.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.461+0000 7f4eb2800640 1 Processor -- start 2024-01-16T15:27:54.461 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.461+0000 7f4eb2800640 1 -- start start 2024-01-16T15:27:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.463+0000 7f4eb2800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f4eac0cf9a0 0x7f4eac15dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.463+0000 7f4eb2800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac0b1ee0 con 0x7f4eac143ae0 2024-01-16T15:27:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.463+0000 7f4eb2800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac0b2780 con 0x7f4eac15e1e0 2024-01-16T15:27:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.463+0000 7f4eb2800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac15e990 con 0x7f4eac0cf9a0 2024-01-16T15:27:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.463+0000 7f4eab400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f4eac143ae0 0x7f4eac0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:52074/0 (socket says 172.21.15.53:52074) 2024-01-16T15:27:54.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.463+0000 7f4eab400640 1 -- 172.21.15.53:0/3958823098 learned_addr learned my addr 172.21.15.53:0/3958823098 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:27:54.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/3958823098 >> v2:172.21.15.146:3300/0 conn(0x7f4eac0cf9a0 0x7f4eac15dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1587197574 0 0) 0x7f4eac0b1ee0 con 0x7f4eac143ae0 2024-01-16T15:27:54.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e8c005730 con 0x7f4eac143ae0 2024-01-16T15:27:54.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263174074 0 0) 0x7f4e8c005730 con 0x7f4eac143ae0 2024-01-16T15:27:54.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 >> v2:172.21.15.146:3300/0 conn(0x7f4eac0cf9a0 msgr2=0x7f4eac15dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:54.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 --2- 172.21.15.53:0/3958823098 >> v2:172.21.15.146:3300/0 conn(0x7f4eac0cf9a0 0x7f4eac15dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 >> v1:172.21.15.146:6789/0 conn(0x7f4eac15e1e0 legacy=0x7f4eac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac0b2130 con 0x7f4eac143ae0 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f4e9c001670 con 0x7f4eac143ae0 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/3958823098 >> v2:172.21.15.146:3300/0 conn(0x7f4eac0cf9a0 0x7f4eac15dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 >> v1:172.21.15.53:6789/0 conn(0x7f4eac143ae0 legacy=0x7f4eac0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e8c005bd0 0x7f4e8c005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4e8c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e8c0073e0 0x7f4e8c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac15e990 con 0x7f4e8c0069d0 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c00e330 con 0x7f4e8c005bd0 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8c00e4f0 con 0x7f4e8c0073e0 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.464+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4e8c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4eab400640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e8c005bd0 0x7f4e8c005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4eabe00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e8c0073e0 0x7f4e8c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4ea3e00640 1 -- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e8c0073e0 msgr2=0x7f4e8c00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4e8c0073e0 0x7f4e8c00db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4ea3e00640 1 -- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e8c005bd0 msgr2=0x7f4e8c005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:27:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4e8c005bd0 0x7f4e8c005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4ea3e00640 1 -- 172.21.15.53:0/3958823098 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e8c005730 con 0x7f4e8c0069d0 2024-01-16T15:27:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.465+0000 7f4eabe00640 1 -- 172.21.15.53:0/3958823098 reap_dead start 2024-01-16T15:27:54.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.466+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4e8c006e50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f4e940098a0 tx=0x7f4e8c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3de0a2de5647b296 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94004230 con 0x7f4e8c0069d0 2024-01-16T15:27:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94004390 con 0x7f4e8c0069d0 2024-01-16T15:27:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eaaa00640 1 -- 172.21.15.53:0/3958823098 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e94003830 con 0x7f4e8c0069d0 2024-01-16T15:27:54.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 -- 172.21.15.53:0/3958823098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 msgr2=0x7f4e8c006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4e8c006e50 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7f4e940098a0 tx=0x7f4e8c007a00 comp rx=0 tx=0).stop 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 -- 172.21.15.53:0/3958823098 shutdown_connections 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 --2- 172.21.15.53:0/3958823098 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4e8c006e50 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 -- 172.21.15.53:0/3958823098 >> 172.21.15.53:0/3958823098 conn(0x7f4eac06a800 msgr2=0x7f4eac06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 -- 172.21.15.53:0/3958823098 shutdown_connections 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.467+0000 7f4eb2800640 1 -- 172.21.15.53:0/3958823098 wait complete. 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.469+0000 7f4eb2800640 1 Processor -- start 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.469+0000 7f4eb2800640 1 -- start start 2024-01-16T15:27:54.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.470+0000 7f4eb2800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4eac15b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.470+0000 7f4eb2800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4eac15a4e0 0x7f4eac0702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eb2800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eac070800 0x7f4eac0cd730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eb2800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac070c80 con 0x7f4e8c0069d0 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eb2800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac15bc50 con 0x7f4eac15a4e0 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eb2800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac15bf30 con 0x7f4eac070800 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4eac15b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4eac15b3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59020/0 (socket says 172.21.15.53:59020) 2024-01-16T15:27:54.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 -- 172.21.15.53:0/355990646 learned_addr learned my addr 172.21.15.53:0/355990646 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 -- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eac070800 msgr2=0x7f4eac0cd730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eac070800 0x7f4eac0cd730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 -- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4eac15a4e0 msgr2=0x7f4eac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4eac15a4e0 0x7f4eac0702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.471+0000 7f4eab400640 1 -- 172.21.15.53:0/355990646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac15c210 con 0x7f4e8c0069d0 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.472+0000 7f4eab400640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4eac15b3d0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e7b0 tx=0x7f4e9c00ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.472+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00ee60 con 0x7f4e8c0069d0 2024-01-16T15:27:54.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.472+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9c00c8d0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.472+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac15c4f0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.472+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9c00cc80 con 0x7f4e8c0069d0 2024-01-16T15:27:54.475 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.472+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac1f2ea0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.473+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac0675d0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.474+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f4e9c01d460 con 0x7f4e8c0069d0 2024-01-16T15:27:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.475+0000 7f4ea3400640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e8406c580 0x7f4e8406ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:27:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.475+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f4e9c088ae0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.476+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e8406c580 0x7f4e8406ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:27:54.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.477+0000 7f4ea3e00640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e8406c580 0x7f4e8406ea40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4e94035000 tx=0x7f4e94030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:27:54.479 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.480+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f4e9c055080 con 0x7f4e8c0069d0 2024-01-16T15:27:54.687 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.688+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4eac1411f0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.688 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.689+0000 7f4ea3400640 1 -- 172.21.15.53:0/355990646 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v112) v1 ==== 80+0+0 (secure 0 0 0) 0x7f4e9c0586a0 con 0x7f4e8c0069d0 2024-01-16T15:27:54.688 INFO:teuthology.orchestra.run.smithi053.stderr:listed 0 entries 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.691+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e8406c580 msgr2=0x7f4e8406ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.691+0000 7f4eb2800640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e8406c580 0x7f4e8406ea40 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f4e94035000 tx=0x7f4e94030000 comp rx=0 tx=0).stop 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.691+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 msgr2=0x7f4eac15b3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4eac15b3d0 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c00e7b0 tx=0x7f4e9c00ec70 comp rx=0 tx=0).stop 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 shutdown_connections 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f4e8406c580 0x7f4e8406ea40 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f4eac070800 0x7f4eac0cd730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f4eac15a4e0 0x7f4eac0702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 --2- 172.21.15.53:0/355990646 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f4e8c0069d0 0x7f4eac15b3d0 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:27:54.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 >> 172.21.15.53:0/355990646 conn(0x7f4eac06a800 msgr2=0x7f4eac142b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:27:54.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 shutdown_connections 2024-01-16T15:27:54.692 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:27:54.692+0000 7f4eb2800640 1 -- 172.21.15.53:0/355990646 wait complete. 2024-01-16T15:27:54.711 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi146.front.sepia.ceph.com... 2024-01-16T15:27:54.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:27:54.711 DEBUG:teuthology.orchestra.run.smithi146:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-01-16T15:27:54.805 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi146.stderr:ceph-fuse[47621]: fuse finished with error 0 and tester_r 0 2024-01-16T15:27:54.810 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:27:59.274 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:27:59.283 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:00.912 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:28:00.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:00.912 DEBUG:teuthology.orchestra.run.smithi146:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-01-16T15:28:00.971 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-01-16T15:28:00.971 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/proc/self/mounts of=/dev/stdout 2024-01-16T15:28:01.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053860,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249404k,nr_inodes=812351,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498816k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-01-16T15:28:01.019 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.019 DEBUG:teuthology.orchestra.run.smithi053:> ip netns list 2024-01-16T15:28:01.078 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-01-16T15:28:01.078 INFO:teuthology.orchestra.run.smithi053.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-01-16T15:28:01.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.080 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-01-16T15:28:01.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.183 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-01-16T15:28:01.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.285 DEBUG:teuthology.orchestra.run.smithi053:> sudo ip link delete ceph-brx 2024-01-16T15:28:01.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.416 DEBUG:teuthology.orchestra.run.smithi146:> ip netns list 2024-01-16T15:28:01.471 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-01-16T15:28:01.472 INFO:teuthology.orchestra.run.smithi146.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-01-16T15:28:01.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-01-16T15:28:01.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.572 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-01-16T15:28:01.652 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-01-16T15:28:01.652 DEBUG:teuthology.orchestra.run.smithi146:> sudo ip link delete ceph-brx 2024-01-16T15:28:01.782 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-01-16T15:28:01.791 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-01-16T15:28:01.791 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:02.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.023+0000 7fed28000640 1 Processor -- start 2024-01-16T15:28:02.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.023+0000 7fed28000640 1 -- start start 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.025+0000 7fed28000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fed2013df80 0x7fed2015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed28000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 0x7fed200cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed28000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2006c470 con 0x7fed2013e360 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed28000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed2006c750 con 0x7fed2013df80 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed28000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed2006ca30 con 0x7fed200cf9a0 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed24c00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fed2013e360 0x7fed200cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57580/0 (socket says 172.21.15.53:57580) 2024-01-16T15:28:02.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed24c00640 1 -- 172.21.15.53:0/3363018585 learned_addr learned my addr 172.21.15.53:0/3363018585 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.026+0000 7fed25600640 1 --2- 172.21.15.53:0/3363018585 >> v2:172.21.15.146:3300/0 conn(0x7fed2013df80 0x7fed2015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed26000640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 0x7fed200cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376838798 0 0) 0x7fed2006c470 con 0x7fed2013e360 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed00003520 con 0x7fed2013e360 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1654730162 0 0) 0x7fed00003520 con 0x7fed2013e360 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 msgr2=0x7fed200cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 0x7fed200cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 >> v2:172.21.15.146:3300/0 conn(0x7fed2013df80 msgr2=0x7fed2015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 --2- 172.21.15.53:0/3363018585 >> v2:172.21.15.146:3300/0 conn(0x7fed2013df80 0x7fed2015bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2006cd10 con 0x7fed2013e360 2024-01-16T15:28:02.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fed080019f0 con 0x7fed2013e360 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 >> v1:172.21.15.53:6789/0 conn(0x7fed2013e360 legacy=0x7fed200cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed00003970 0x7fed00003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 0x7fed000047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed00005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2006c750 con 0x7fed00004d00 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed2006ca30 con 0x7fed00003970 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.027+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0000b980 con 0x7fed200cf9a0 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.028+0000 7fed26000640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed00005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.028+0000 7fed26000640 1 -- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 msgr2=0x7fed000047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:02.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.028+0000 7fed26000640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed200cf9a0 0x7fed000047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.028+0000 7fed26000640 1 -- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed00003970 msgr2=0x7fed00003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:02.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.028+0000 7fed26000640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed00003970 0x7fed00003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.028+0000 7fed26000640 1 -- 172.21.15.53:0/3363018585 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed00003520 con 0x7fed00004d00 2024-01-16T15:28:02.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.029+0000 7fed26000640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed00005180 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fed18009ac0 tx=0x7fed000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41f1cc24cb5cfa4a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:02.031 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1802c6b0 con 0x7fed00004d00 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1802c810 con 0x7fed00004d00 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed1fe00640 1 -- 172.21.15.53:0/3363018585 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1802cba0 con 0x7fed00004d00 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed28000640 1 -- 172.21.15.53:0/3363018585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 msgr2=0x7fed00005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed28000640 1 --2- 172.21.15.53:0/3363018585 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed00005180 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fed18009ac0 tx=0x7fed000057f0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed26000640 1 -- 172.21.15.53:0/3363018585 reap_dead start 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed28000640 1 -- 172.21.15.53:0/3363018585 shutdown_connections 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed28000640 1 -- 172.21.15.53:0/3363018585 >> 172.21.15.53:0/3363018585 conn(0x7fed2006a800 msgr2=0x7fed2006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:02.032 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.030+0000 7fed28000640 1 -- 172.21.15.53:0/3363018585 shutdown_connections 2024-01-16T15:28:02.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.031+0000 7fed28000640 1 -- 172.21.15.53:0/3363018585 wait complete. 2024-01-16T15:28:02.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.032+0000 7fed28000640 1 Processor -- start 2024-01-16T15:28:02.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.033+0000 7fed28000640 1 -- start start 2024-01-16T15:28:02.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.033+0000 7fed28000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed200ac3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.033+0000 7fed28000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed201f02e0 0x7fed201f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.033+0000 7fed28000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed200a84c0 0x7fed200b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed28000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed200a9c00 con 0x7fed00004d00 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed28000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed201f0760 con 0x7fed200a84c0 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed28000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed201f3540 con 0x7fed201f02e0 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed200ac3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.035 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed200ac3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55476/0 (socket says 172.21.15.53:55476) 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 -- 172.21.15.53:0/858018235 learned_addr learned my addr 172.21.15.53:0/858018235 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed26000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed200a84c0 0x7fed200b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 -- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed201f02e0 msgr2=0x7fed201f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed201f02e0 0x7fed201f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 -- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed200a84c0 msgr2=0x7fed200b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed200a84c0 0x7fed200b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.034+0000 7fed25600640 1 -- 172.21.15.53:0/858018235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed201f3820 con 0x7fed00004d00 2024-01-16T15:28:02.036 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.035+0000 7fed25600640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed200ac3c0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fed1000e7f0 tx=0x7fed1000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:02.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.035+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000c940 con 0x7fed00004d00 2024-01-16T15:28:02.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.035+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1000caa0 con 0x7fed00004d00 2024-01-16T15:28:02.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.035+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000ce50 con 0x7fed00004d00 2024-01-16T15:28:02.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.035+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed201f3b00 con 0x7fed00004d00 2024-01-16T15:28:02.037 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.035+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed201f3250 con 0x7fed00004d00 2024-01-16T15:28:02.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.036+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed200675d0 con 0x7fed00004d00 2024-01-16T15:28:02.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.037+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fed10013470 con 0x7fed00004d00 2024-01-16T15:28:02.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.038+0000 7fed1d600640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecf806c5d0 0x7fecf806ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.038+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fed100888c0 con 0x7fed00004d00 2024-01-16T15:28:02.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.039+0000 7fed24c00640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecf806c5d0 0x7fecf806ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.040+0000 7fed24c00640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecf806c5d0 0x7fecf806ea90 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed200b4240 tx=0x7fed08001940 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:02.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.043+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fed10055180 con 0x7fed00004d00 2024-01-16T15:28:02.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.231+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed201f3d40 con 0x7fecf806c5d0 2024-01-16T15:28:02.257 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.258+0000 7fed1d600640 1 -- 172.21.15.53:0/858018235 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288968 (secure 0 0 0) 0x7fed201f3d40 con 0x7fecf806c5d0 2024-01-16T15:28:02.259 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:02.263 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:28:02.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecf806c5d0 msgr2=0x7fecf806ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecf806c5d0 0x7fecf806ea90 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fed200b4240 tx=0x7fed08001940 comp rx=0 tx=0).stop 2024-01-16T15:28:02.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 msgr2=0x7fed200ac3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed200ac3c0 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7fed1000e7f0 tx=0x7fed1000ecb0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 shutdown_connections 2024-01-16T15:28:02.265 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fecf806c5d0 0x7fecf806ea90 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fed200a84c0 0x7fed200b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fed201f02e0 0x7fed201f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 --2- 172.21.15.53:0/858018235 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fed00004d00 0x7fed200ac3c0 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 >> 172.21.15.53:0/858018235 conn(0x7fed2006a800 msgr2=0x7fed200b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:02.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 shutdown_connections 2024-01-16T15:28:02.266 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.266+0000 7fed28000640 1 -- 172.21.15.53:0/858018235 wait complete. 2024-01-16T15:28:02.290 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":953,"stamp":"2024-01-16T15:28:00.906556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4907,"ondisk_log_size":4907,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11125304,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":625011,"kb_avail":743849416,"statfs":{"total":773094113280,"available":761701801984,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":640011824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12773,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004945"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256079+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.256079+0000","last_peered":"2024-01-16T15:27:49.256079+0000","last_clean":"2024-01-16T15:27:49.256079+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.256079+0000","last_undegraded":"2024-01-16T15:27:49.256079+0000","last_fullsized":"2024-01-16T15:27:49.256079+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:08:35.875409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'73","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237782+0000","last_change":"2024-01-16T15:19:24.622288+0000","last_active":"2024-01-16T15:27:49.237782+0000","last_peered":"2024-01-16T15:27:49.237782+0000","last_clean":"2024-01-16T15:27:49.237782+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:27:49.237782+0000","last_undegraded":"2024-01-16T15:27:49.237782+0000","last_fullsized":"2024-01-16T15:27:49.237782+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:20.176733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251487+0000","last_change":"2024-01-16T15:19:24.618862+0000","last_active":"2024-01-16T15:27:49.251487+0000","last_peered":"2024-01-16T15:27:49.251487+0000","last_clean":"2024-01-16T15:27:49.251487+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:27:49.251487+0000","last_undegraded":"2024-01-16T15:27:49.251487+0000","last_fullsized":"2024-01-16T15:27:49.251487+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:02.115872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'56","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.231727+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:27:49.231727+0000","last_peered":"2024-01-16T15:27:49.231727+0000","last_clean":"2024-01-16T15:27:49.231727+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:27:49.231727+0000","last_undegraded":"2024-01-16T15:27:49.231727+0000","last_fullsized":"2024-01-16T15:27:49.231727+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:32.713991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'62","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.232748+0000","last_change":"2024-01-16T15:19:24.626093+0000","last_active":"2024-01-16T15:27:49.232748+0000","last_peered":"2024-01-16T15:27:49.232748+0000","last_clean":"2024-01-16T15:27:49.232748+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:27:49.232748+0000","last_undegraded":"2024-01-16T15:27:49.232748+0000","last_fullsized":"2024-01-16T15:27:49.232748+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:45.086271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'76","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239918+0000","last_change":"2024-01-16T15:19:24.625809+0000","last_active":"2024-01-16T15:27:49.239918+0000","last_peered":"2024-01-16T15:27:49.239918+0000","last_clean":"2024-01-16T15:27:49.239918+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:27:49.239918+0000","last_undegraded":"2024-01-16T15:27:49.239918+0000","last_fullsized":"2024-01-16T15:27:49.239918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:21.453533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'67","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.241483+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:27:49.241483+0000","last_peered":"2024-01-16T15:27:49.241483+0000","last_clean":"2024-01-16T15:27:49.241483+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:27:49.241483+0000","last_undegraded":"2024-01-16T15:27:49.241483+0000","last_fullsized":"2024-01-16T15:27:49.241483+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:15:57.152091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'58","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248998+0000","last_change":"2024-01-16T15:19:24.616757+0000","last_active":"2024-01-16T15:27:49.248998+0000","last_peered":"2024-01-16T15:27:49.248998+0000","last_clean":"2024-01-16T15:27:49.248998+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:27:49.248998+0000","last_undegraded":"2024-01-16T15:27:49.248998+0000","last_fullsized":"2024-01-16T15:27:49.248998+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:30.687663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.255855+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:27:49.255855+0000","last_peered":"2024-01-16T15:27:49.255855+0000","last_clean":"2024-01-16T15:27:49.255855+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:27:49.255855+0000","last_undegraded":"2024-01-16T15:27:49.255855+0000","last_fullsized":"2024-01-16T15:27:49.255855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:07.972777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257125+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:27:49.257125+0000","last_peered":"2024-01-16T15:27:49.257125+0000","last_clean":"2024-01-16T15:27:49.257125+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:27:49.257125+0000","last_undegraded":"2024-01-16T15:27:49.257125+0000","last_fullsized":"2024-01-16T15:27:49.257125+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:41.949199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'70","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.227737+0000","last_change":"2024-01-16T15:19:24.625137+0000","last_active":"2024-01-16T15:27:49.227737+0000","last_peered":"2024-01-16T15:27:49.227737+0000","last_clean":"2024-01-16T15:27:49.227737+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:27:49.227737+0000","last_undegraded":"2024-01-16T15:27:49.227737+0000","last_fullsized":"2024-01-16T15:27:49.227737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'64","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236182+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:27:49.236182+0000","last_peered":"2024-01-16T15:27:49.236182+0000","last_clean":"2024-01-16T15:27:49.236182+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:27:49.236182+0000","last_undegraded":"2024-01-16T15:27:49.236182+0000","last_fullsized":"2024-01-16T15:27:49.236182+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:27.512415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'65","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242745+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:27:49.242745+0000","last_peered":"2024-01-16T15:27:49.242745+0000","last_clean":"2024-01-16T15:27:49.242745+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:27:49.242745+0000","last_undegraded":"2024-01-16T15:27:49.242745+0000","last_fullsized":"2024-01-16T15:27:49.242745+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:08.270535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'72","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.245113+0000","last_change":"2024-01-16T15:19:24.624351+0000","last_active":"2024-01-16T15:27:49.245113+0000","last_peered":"2024-01-16T15:27:49.245113+0000","last_clean":"2024-01-16T15:27:49.245113+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:27:49.245113+0000","last_undegraded":"2024-01-16T15:27:49.245113+0000","last_fullsized":"2024-01-16T15:27:49.245113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:41.925967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'59","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246430+0000","last_change":"2024-01-16T15:19:24.613444+0000","last_active":"2024-01-16T15:27:49.246430+0000","last_peered":"2024-01-16T15:27:49.246430+0000","last_clean":"2024-01-16T15:27:49.246430+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:27:49.246430+0000","last_undegraded":"2024-01-16T15:27:49.246430+0000","last_fullsized":"2024-01-16T15:27:49.246430+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:59.930379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'74","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248683+0000","last_change":"2024-01-16T15:19:24.614694+0000","last_active":"2024-01-16T15:27:49.248683+0000","last_peered":"2024-01-16T15:27:49.248683+0000","last_clean":"2024-01-16T15:27:49.248683+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:27:49.248683+0000","last_undegraded":"2024-01-16T15:27:49.248683+0000","last_fullsized":"2024-01-16T15:27:49.248683+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:59.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'52","reported_seq":73,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.228316+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:27:49.228316+0000","last_peered":"2024-01-16T15:27:49.228316+0000","last_clean":"2024-01-16T15:27:49.228316+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:27:49.228316+0000","last_undegraded":"2024-01-16T15:27:49.228316+0000","last_fullsized":"2024-01-16T15:27:49.228316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.235929+0000","last_change":"2024-01-16T15:19:24.629022+0000","last_active":"2024-01-16T15:27:49.235929+0000","last_peered":"2024-01-16T15:27:49.235929+0000","last_clean":"2024-01-16T15:27:49.235929+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:27:49.235929+0000","last_undegraded":"2024-01-16T15:27:49.235929+0000","last_fullsized":"2024-01-16T15:27:49.235929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:22.444157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'74","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250216+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:27:49.250216+0000","last_peered":"2024-01-16T15:27:49.250216+0000","last_clean":"2024-01-16T15:27:49.250216+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:27:49.250216+0000","last_undegraded":"2024-01-16T15:27:49.250216+0000","last_fullsized":"2024-01-16T15:27:49.250216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:11.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247569+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:27:49.247569+0000","last_peered":"2024-01-16T15:27:49.247569+0000","last_clean":"2024-01-16T15:27:49.247569+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:27:49.247569+0000","last_undegraded":"2024-01-16T15:27:49.247569+0000","last_fullsized":"2024-01-16T15:27:49.247569+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'73","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248711+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:27:49.248711+0000","last_peered":"2024-01-16T15:27:49.248711+0000","last_clean":"2024-01-16T15:27:49.248711+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:27:49.248711+0000","last_undegraded":"2024-01-16T15:27:49.248711+0000","last_fullsized":"2024-01-16T15:27:49.248711+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'71","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233752+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:27:49.233752+0000","last_peered":"2024-01-16T15:27:49.233752+0000","last_clean":"2024-01-16T15:27:49.233752+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:27:49.233752+0000","last_undegraded":"2024-01-16T15:27:49.233752+0000","last_fullsized":"2024-01-16T15:27:49.233752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:26.924342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'63","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247674+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:27:49.247674+0000","last_peered":"2024-01-16T15:27:49.247674+0000","last_clean":"2024-01-16T15:27:49.247674+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:27:49.247674+0000","last_undegraded":"2024-01-16T15:27:49.247674+0000","last_fullsized":"2024-01-16T15:27:49.247674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:07.051272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'71","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249786+0000","last_change":"2024-01-16T15:19:24.626920+0000","last_active":"2024-01-16T15:27:49.249786+0000","last_peered":"2024-01-16T15:27:49.249786+0000","last_clean":"2024-01-16T15:27:49.249786+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:27:49.249786+0000","last_undegraded":"2024-01-16T15:27:49.249786+0000","last_fullsized":"2024-01-16T15:27:49.249786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'69","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233273+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:27:49.233273+0000","last_peered":"2024-01-16T15:27:49.233273+0000","last_clean":"2024-01-16T15:27:49.233273+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:27:49.233273+0000","last_undegraded":"2024-01-16T15:27:49.233273+0000","last_fullsized":"2024-01-16T15:27:49.233273+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251042+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:27:49.251042+0000","last_peered":"2024-01-16T15:27:49.251042+0000","last_clean":"2024-01-16T15:27:49.251042+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:27:49.251042+0000","last_undegraded":"2024-01-16T15:27:49.251042+0000","last_fullsized":"2024-01-16T15:27:49.251042+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:49.687478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'65","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.226338+0000","last_change":"2024-01-16T15:19:24.623489+0000","last_active":"2024-01-16T15:27:49.226338+0000","last_peered":"2024-01-16T15:27:49.226338+0000","last_clean":"2024-01-16T15:27:49.226338+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:27:49.226338+0000","last_undegraded":"2024-01-16T15:27:49.226338+0000","last_fullsized":"2024-01-16T15:27:49.226338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:11.064893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'50","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.241685+0000","last_change":"2024-01-16T15:19:42.218585+0000","last_active":"2024-01-16T15:27:49.241685+0000","last_peered":"2024-01-16T15:27:49.241685+0000","last_clean":"2024-01-16T15:27:49.241685+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:27:49.241685+0000","last_undegraded":"2024-01-16T15:27:49.241685+0000","last_fullsized":"2024-01-16T15:27:49.241685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:02.749154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'62","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248953+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:27:49.248953+0000","last_peered":"2024-01-16T15:27:49.248953+0000","last_clean":"2024-01-16T15:27:49.248953+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:27:49.248953+0000","last_undegraded":"2024-01-16T15:27:49.248953+0000","last_fullsized":"2024-01-16T15:27:49.248953+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:40.862803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'78","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246500+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:27:49.246500+0000","last_peered":"2024-01-16T15:27:49.246500+0000","last_clean":"2024-01-16T15:27:49.246500+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:27:49.246500+0000","last_undegraded":"2024-01-16T15:27:49.246500+0000","last_fullsized":"2024-01-16T15:27:49.246500+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:45.144417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'91","reported_seq":120,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252053+0000","last_change":"2024-01-16T15:19:24.629910+0000","last_active":"2024-01-16T15:27:49.252053+0000","last_peered":"2024-01-16T15:27:49.252053+0000","last_clean":"2024-01-16T15:27:49.252053+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:27:49.252053+0000","last_undegraded":"2024-01-16T15:27:49.252053+0000","last_fullsized":"2024-01-16T15:27:49.252053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:44.059899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'78","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246839+0000","last_change":"2024-01-16T15:19:24.633567+0000","last_active":"2024-01-16T15:27:49.246839+0000","last_peered":"2024-01-16T15:27:49.246839+0000","last_clean":"2024-01-16T15:27:49.246839+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:27:49.246839+0000","last_undegraded":"2024-01-16T15:27:49.246839+0000","last_fullsized":"2024-01-16T15:27:49.246839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:24.582502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'60","reported_seq":83,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.219133+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:27:49.219133+0000","last_peered":"2024-01-16T15:27:49.219133+0000","last_clean":"2024-01-16T15:27:49.219133+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:27:49.219133+0000","last_undegraded":"2024-01-16T15:27:49.219133+0000","last_fullsized":"2024-01-16T15:27:49.219133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:01.155551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":22,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.591516+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:25:40.591516+0000","last_peered":"2024-01-16T15:25:40.591516+0000","last_clean":"2024-01-16T15:25:40.591516+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:25:40.591516+0000","last_undegraded":"2024-01-16T15:25:40.591516+0000","last_fullsized":"2024-01-16T15:25:40.591516+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.038677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'67","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247582+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:27:49.247582+0000","last_peered":"2024-01-16T15:27:49.247582+0000","last_clean":"2024-01-16T15:27:49.247582+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:27:49.247582+0000","last_undegraded":"2024-01-16T15:27:49.247582+0000","last_fullsized":"2024-01-16T15:27:49.247582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:25.905229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.933509+0000","last_change":"2024-01-16T15:19:23.621310+0000","last_active":"2024-01-16T15:25:25.933509+0000","last_peered":"2024-01-16T15:25:25.933509+0000","last_clean":"2024-01-16T15:25:25.933509+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:25:25.933509+0000","last_undegraded":"2024-01-16T15:25:25.933509+0000","last_fullsized":"2024-01-16T15:25:25.933509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:19.958286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'66","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246028+0000","last_change":"2024-01-16T15:19:24.630161+0000","last_active":"2024-01-16T15:27:49.246028+0000","last_peered":"2024-01-16T15:27:49.246028+0000","last_clean":"2024-01-16T15:27:49.246028+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:27:49.246028+0000","last_undegraded":"2024-01-16T15:27:49.246028+0000","last_fullsized":"2024-01-16T15:27:49.246028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:54:32.432478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'3","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.565625+0000","last_change":"2024-01-16T15:19:23.630011+0000","last_active":"2024-01-16T15:24:25.565625+0000","last_peered":"2024-01-16T15:24:25.565625+0000","last_clean":"2024-01-16T15:24:25.565625+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:24:25.565625+0000","last_undegraded":"2024-01-16T15:24:25.565625+0000","last_fullsized":"2024-01-16T15:24:25.565625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:21.485125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'84","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250676+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:27:49.250676+0000","last_peered":"2024-01-16T15:27:49.250676+0000","last_clean":"2024-01-16T15:27:49.250676+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:27:49.250676+0000","last_undegraded":"2024-01-16T15:27:49.250676+0000","last_fullsized":"2024-01-16T15:27:49.250676+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:53.269911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.565587+0000","last_change":"2024-01-16T15:19:23.618597+0000","last_active":"2024-01-16T15:24:25.565587+0000","last_peered":"2024-01-16T15:24:25.565587+0000","last_clean":"2024-01-16T15:24:25.565587+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:24:25.565587+0000","last_undegraded":"2024-01-16T15:24:25.565587+0000","last_fullsized":"2024-01-16T15:24:25.565587+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:05.775312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'78","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250467+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:27:49.250467+0000","last_peered":"2024-01-16T15:27:49.250467+0000","last_clean":"2024-01-16T15:27:49.250467+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:27:49.250467+0000","last_undegraded":"2024-01-16T15:27:49.250467+0000","last_fullsized":"2024-01-16T15:27:49.250467+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:01.404745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.578781+0000","last_change":"2024-01-16T15:19:23.618024+0000","last_active":"2024-01-16T15:24:25.578781+0000","last_peered":"2024-01-16T15:24:25.578781+0000","last_clean":"2024-01-16T15:24:25.578781+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:24:25.578781+0000","last_undegraded":"2024-01-16T15:24:25.578781+0000","last_fullsized":"2024-01-16T15:24:25.578781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:50.476809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'82","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234990+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:27:49.234990+0000","last_peered":"2024-01-16T15:27:49.234990+0000","last_clean":"2024-01-16T15:27:49.234990+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:27:49.234990+0000","last_undegraded":"2024-01-16T15:27:49.234990+0000","last_fullsized":"2024-01-16T15:27:49.234990+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.321825+0000","last_change":"2024-01-16T15:19:23.622641+0000","last_active":"2024-01-16T15:24:25.321825+0000","last_peered":"2024-01-16T15:24:25.321825+0000","last_clean":"2024-01-16T15:24:25.321825+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:24:25.321825+0000","last_undegraded":"2024-01-16T15:24:25.321825+0000","last_fullsized":"2024-01-16T15:24:25.321825+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:39.846805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'80","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250148+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:27:49.250148+0000","last_peered":"2024-01-16T15:27:49.250148+0000","last_clean":"2024-01-16T15:27:49.250148+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:27:49.250148+0000","last_undegraded":"2024-01-16T15:27:49.250148+0000","last_fullsized":"2024-01-16T15:27:49.250148+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:19:48.939386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.744702+0000","last_change":"2024-01-16T15:19:23.620941+0000","last_active":"2024-01-16T15:24:16.744702+0000","last_peered":"2024-01-16T15:24:16.744702+0000","last_clean":"2024-01-16T15:24:16.744702+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:24:16.744702+0000","last_undegraded":"2024-01-16T15:24:16.744702+0000","last_fullsized":"2024-01-16T15:24:16.744702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:27.971375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'68","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234098+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:27:49.234098+0000","last_peered":"2024-01-16T15:27:49.234098+0000","last_clean":"2024-01-16T15:27:49.234098+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:27:49.234098+0000","last_undegraded":"2024-01-16T15:27:49.234098+0000","last_fullsized":"2024-01-16T15:27:49.234098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:53.297338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.587238+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:24:15.587238+0000","last_peered":"2024-01-16T15:24:15.587238+0000","last_clean":"2024-01-16T15:24:15.587238+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:24:15.587238+0000","last_undegraded":"2024-01-16T15:24:15.587238+0000","last_fullsized":"2024-01-16T15:24:15.587238+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:07.254538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'66","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242411+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:27:49.242411+0000","last_peered":"2024-01-16T15:27:49.242411+0000","last_clean":"2024-01-16T15:27:49.242411+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:27:49.242411+0000","last_undegraded":"2024-01-16T15:27:49.242411+0000","last_fullsized":"2024-01-16T15:27:49.242411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:40.365875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.563703+0000","last_change":"2024-01-16T15:19:23.621670+0000","last_active":"2024-01-16T15:24:15.563703+0000","last_peered":"2024-01-16T15:24:15.563703+0000","last_clean":"2024-01-16T15:24:15.563703+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:24:15.563703+0000","last_undegraded":"2024-01-16T15:24:15.563703+0000","last_fullsized":"2024-01-16T15:24:15.563703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:53.183004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'64","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237836+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:27:49.237836+0000","last_peered":"2024-01-16T15:27:49.237836+0000","last_clean":"2024-01-16T15:27:49.237836+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:27:49.237836+0000","last_undegraded":"2024-01-16T15:27:49.237836+0000","last_fullsized":"2024-01-16T15:27:49.237836+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:56.499150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.319813+0000","last_change":"2024-01-16T15:19:23.624332+0000","last_active":"2024-01-16T15:24:15.319813+0000","last_peered":"2024-01-16T15:24:15.319813+0000","last_clean":"2024-01-16T15:24:15.319813+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:24:15.319813+0000","last_undegraded":"2024-01-16T15:24:15.319813+0000","last_fullsized":"2024-01-16T15:24:15.319813+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:56.995452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'60","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243544+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:27:49.243544+0000","last_peered":"2024-01-16T15:27:49.243544+0000","last_clean":"2024-01-16T15:27:49.243544+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:27:49.243544+0000","last_undegraded":"2024-01-16T15:27:49.243544+0000","last_fullsized":"2024-01-16T15:27:49.243544+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:06.684073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'31","reported_seq":48,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:43.246740+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:25:43.246740+0000","last_peered":"2024-01-16T15:25:43.246740+0000","last_clean":"2024-01-16T15:25:43.246740+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:25:43.246740+0000","last_undegraded":"2024-01-16T15:25:43.246740+0000","last_fullsized":"2024-01-16T15:25:43.246740+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:48.585086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'69","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251340+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:27:49.251340+0000","last_peered":"2024-01-16T15:27:49.251340+0000","last_clean":"2024-01-16T15:27:49.251340+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:27:49.251340+0000","last_undegraded":"2024-01-16T15:27:49.251340+0000","last_fullsized":"2024-01-16T15:27:49.251340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:23.918834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'307","reported_seq":324,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.260197+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:27:59.260197+0000","last_peered":"2024-01-16T15:27:59.260197+0000","last_clean":"2024-01-16T15:27:59.260197+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:27:59.260197+0000","last_undegraded":"2024-01-16T15:27:59.260197+0000","last_fullsized":"2024-01-16T15:27:59.260197+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:52.253914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'63","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242837+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:27:49.242837+0000","last_peered":"2024-01-16T15:27:49.242837+0000","last_clean":"2024-01-16T15:27:49.242837+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:27:49.242837+0000","last_undegraded":"2024-01-16T15:27:49.242837+0000","last_fullsized":"2024-01-16T15:27:49.242837+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:37.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.569761+0000","last_change":"2024-01-16T15:19:23.617724+0000","last_active":"2024-01-16T15:24:05.569761+0000","last_peered":"2024-01-16T15:24:05.569761+0000","last_clean":"2024-01-16T15:24:05.569761+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:24:05.569761+0000","last_undegraded":"2024-01-16T15:24:05.569761+0000","last_fullsized":"2024-01-16T15:24:05.569761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:35.823425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'80","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249807+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:27:49.249807+0000","last_peered":"2024-01-16T15:27:49.249807+0000","last_clean":"2024-01-16T15:27:49.249807+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:27:49.249807+0000","last_undegraded":"2024-01-16T15:27:49.249807+0000","last_fullsized":"2024-01-16T15:27:49.249807+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.195535+0000","last_change":"2024-01-16T15:19:23.624683+0000","last_active":"2024-01-16T15:24:07.195535+0000","last_peered":"2024-01-16T15:24:07.195535+0000","last_clean":"2024-01-16T15:24:07.195535+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:24:07.195535+0000","last_undegraded":"2024-01-16T15:24:07.195535+0000","last_fullsized":"2024-01-16T15:24:07.195535+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:38.379463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'73","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244605+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:27:49.244605+0000","last_peered":"2024-01-16T15:27:49.244605+0000","last_clean":"2024-01-16T15:27:49.244605+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:27:49.244605+0000","last_undegraded":"2024-01-16T15:27:49.244605+0000","last_fullsized":"2024-01-16T15:27:49.244605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":25,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.561655+0000","last_change":"2024-01-16T15:19:23.625591+0000","last_active":"2024-01-16T15:24:05.561655+0000","last_peered":"2024-01-16T15:24:05.561655+0000","last_clean":"2024-01-16T15:24:05.561655+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:24:05.561655+0000","last_undegraded":"2024-01-16T15:24:05.561655+0000","last_fullsized":"2024-01-16T15:24:05.561655+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:27.727635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'50","reported_seq":70,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250374+0000","last_change":"2024-01-16T15:19:24.617249+0000","last_active":"2024-01-16T15:27:49.250374+0000","last_peered":"2024-01-16T15:27:49.250374+0000","last_clean":"2024-01-16T15:27:49.250374+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:27:49.250374+0000","last_undegraded":"2024-01-16T15:27:49.250374+0000","last_fullsized":"2024-01-16T15:27:49.250374+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:01.440713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.582905+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:23:55.582905+0000","last_peered":"2024-01-16T15:23:55.582905+0000","last_clean":"2024-01-16T15:23:55.582905+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:23:55.582905+0000","last_undegraded":"2024-01-16T15:23:55.582905+0000","last_fullsized":"2024-01-16T15:23:55.582905+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:27.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'61","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239584+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:27:49.239584+0000","last_peered":"2024-01-16T15:27:49.239584+0000","last_clean":"2024-01-16T15:27:49.239584+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:27:49.239584+0000","last_undegraded":"2024-01-16T15:27:49.239584+0000","last_fullsized":"2024-01-16T15:27:49.239584+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:20.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.560107+0000","last_change":"2024-01-16T15:19:23.622230+0000","last_active":"2024-01-16T15:23:55.560107+0000","last_peered":"2024-01-16T15:23:55.560107+0000","last_clean":"2024-01-16T15:23:55.560107+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:23:55.560107+0000","last_undegraded":"2024-01-16T15:23:55.560107+0000","last_fullsized":"2024-01-16T15:23:55.560107+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'77","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248743+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:27:49.248743+0000","last_peered":"2024-01-16T15:27:49.248743+0000","last_clean":"2024-01-16T15:27:49.248743+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:27:49.248743+0000","last_undegraded":"2024-01-16T15:27:49.248743+0000","last_fullsized":"2024-01-16T15:27:49.248743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:31.587040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.193814+0000","last_change":"2024-01-16T15:19:23.626903+0000","last_active":"2024-01-16T15:23:57.193814+0000","last_peered":"2024-01-16T15:23:57.193814+0000","last_clean":"2024-01-16T15:23:57.193814+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:23:57.193814+0000","last_undegraded":"2024-01-16T15:23:57.193814+0000","last_fullsized":"2024-01-16T15:23:57.193814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:58.751536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'66","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251742+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:27:49.251742+0000","last_peered":"2024-01-16T15:27:49.251742+0000","last_clean":"2024-01-16T15:27:49.251742+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:27:49.251742+0000","last_undegraded":"2024-01-16T15:27:49.251742+0000","last_fullsized":"2024-01-16T15:27:49.251742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:32.314064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.573325+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:23:55.573325+0000","last_peered":"2024-01-16T15:23:55.573325+0000","last_clean":"2024-01-16T15:23:55.573325+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:23:55.573325+0000","last_undegraded":"2024-01-16T15:23:55.573325+0000","last_fullsized":"2024-01-16T15:23:55.573325+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:00.825374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'62","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248439+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:27:49.248439+0000","last_peered":"2024-01-16T15:27:49.248439+0000","last_clean":"2024-01-16T15:27:49.248439+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:27:49.248439+0000","last_undegraded":"2024-01-16T15:27:49.248439+0000","last_fullsized":"2024-01-16T15:27:49.248439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:32.038856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'3","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.567924+0000","last_change":"2024-01-16T15:19:23.621285+0000","last_active":"2024-01-16T15:23:55.567924+0000","last_peered":"2024-01-16T15:23:55.567924+0000","last_clean":"2024-01-16T15:23:55.567924+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:23:55.567924+0000","last_undegraded":"2024-01-16T15:23:55.567924+0000","last_fullsized":"2024-01-16T15:23:55.567924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:44.070274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248670+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:27:49.248670+0000","last_peered":"2024-01-16T15:27:49.248670+0000","last_clean":"2024-01-16T15:27:49.248670+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:27:49.248670+0000","last_undegraded":"2024-01-16T15:27:49.248670+0000","last_fullsized":"2024-01-16T15:27:49.248670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:26.904243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.558296+0000","last_change":"2024-01-16T15:19:23.619619+0000","last_active":"2024-01-16T15:23:45.558296+0000","last_peered":"2024-01-16T15:23:45.558296+0000","last_clean":"2024-01-16T15:23:45.558296+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:23:45.558296+0000","last_undegraded":"2024-01-16T15:23:45.558296+0000","last_fullsized":"2024-01-16T15:23:45.558296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:15.166720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'66","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249594+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:27:49.249594+0000","last_peered":"2024-01-16T15:27:49.249594+0000","last_clean":"2024-01-16T15:27:49.249594+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:27:49.249594+0000","last_undegraded":"2024-01-16T15:27:49.249594+0000","last_fullsized":"2024-01-16T15:27:49.249594+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:58.872650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.565824+0000","last_change":"2024-01-16T15:19:23.620635+0000","last_active":"2024-01-16T15:23:45.565824+0000","last_peered":"2024-01-16T15:23:45.565824+0000","last_clean":"2024-01-16T15:23:45.565824+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:23:45.565824+0000","last_undegraded":"2024-01-16T15:23:45.565824+0000","last_fullsized":"2024-01-16T15:23:45.565824+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'63","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252548+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:27:49.252548+0000","last_peered":"2024-01-16T15:27:49.252548+0000","last_clean":"2024-01-16T15:27:49.252548+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:27:49.252548+0000","last_undegraded":"2024-01-16T15:27:49.252548+0000","last_fullsized":"2024-01-16T15:27:49.252548+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.580304+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:23:45.580304+0000","last_peered":"2024-01-16T15:23:45.580304+0000","last_clean":"2024-01-16T15:23:45.580304+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:23:45.580304+0000","last_undegraded":"2024-01-16T15:23:45.580304+0000","last_fullsized":"2024-01-16T15:23:45.580304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:48.621454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'70","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236778+0000","last_change":"2024-01-16T15:19:42.592735+0000","last_active":"2024-01-16T15:27:49.236778+0000","last_peered":"2024-01-16T15:27:49.236778+0000","last_clean":"2024-01-16T15:27:49.236778+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:27:49.236778+0000","last_undegraded":"2024-01-16T15:27:49.236778+0000","last_fullsized":"2024-01-16T15:27:49.236778+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:07.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.314110+0000","last_change":"2024-01-16T15:19:23.630170+0000","last_active":"2024-01-16T15:23:45.314110+0000","last_peered":"2024-01-16T15:23:45.314110+0000","last_clean":"2024-01-16T15:23:45.314110+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:23:45.314110+0000","last_undegraded":"2024-01-16T15:23:45.314110+0000","last_fullsized":"2024-01-16T15:23:45.314110+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:06.035275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256006+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:27:49.256006+0000","last_peered":"2024-01-16T15:27:49.256006+0000","last_clean":"2024-01-16T15:27:49.256006+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:27:49.256006+0000","last_undegraded":"2024-01-16T15:27:49.256006+0000","last_fullsized":"2024-01-16T15:27:49.256006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:58.148829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":48,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.257180+0000","last_change":"2024-01-16T15:19:23.623778+0000","last_active":"2024-01-16T15:27:44.257180+0000","last_peered":"2024-01-16T15:27:44.257180+0000","last_clean":"2024-01-16T15:27:44.257180+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:27:44.257180+0000","last_undegraded":"2024-01-16T15:27:44.257180+0000","last_fullsized":"2024-01-16T15:27:44.257180+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:53.783590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'69","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244950+0000","last_change":"2024-01-16T15:19:24.628361+0000","last_active":"2024-01-16T15:27:49.244950+0000","last_peered":"2024-01-16T15:27:49.244950+0000","last_clean":"2024-01-16T15:27:49.244950+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:27:49.244950+0000","last_undegraded":"2024-01-16T15:27:49.244950+0000","last_fullsized":"2024-01-16T15:27:49.244950+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:48.469423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.606971+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:25:40.606971+0000","last_peered":"2024-01-16T15:25:40.606971+0000","last_clean":"2024-01-16T15:25:40.606971+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:25:40.606971+0000","last_undegraded":"2024-01-16T15:25:40.606971+0000","last_fullsized":"2024-01-16T15:25:40.606971+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:40.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'53","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236803+0000","last_change":"2024-01-16T15:19:24.623591+0000","last_active":"2024-01-16T15:27:49.236803+0000","last_peered":"2024-01-16T15:27:49.236803+0000","last_clean":"2024-01-16T15:27:49.236803+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:27:49.236803+0000","last_undegraded":"2024-01-16T15:27:49.236803+0000","last_fullsized":"2024-01-16T15:27:49.236803+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:23:57.706980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.336436+0000","last_change":"2024-01-16T15:19:23.612616+0000","last_active":"2024-01-16T15:25:40.336436+0000","last_peered":"2024-01-16T15:25:40.336436+0000","last_clean":"2024-01-16T15:25:40.336436+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:25:40.336436+0000","last_undegraded":"2024-01-16T15:25:40.336436+0000","last_fullsized":"2024-01-16T15:25:40.336436+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:19.160736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'65","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242630+0000","last_change":"2024-01-16T15:19:24.626321+0000","last_active":"2024-01-16T15:27:49.242630+0000","last_peered":"2024-01-16T15:27:49.242630+0000","last_clean":"2024-01-16T15:27:49.242630+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:27:49.242630+0000","last_undegraded":"2024-01-16T15:27:49.242630+0000","last_fullsized":"2024-01-16T15:27:49.242630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.591466+0000","last_change":"2024-01-16T15:19:23.624071+0000","last_active":"2024-01-16T15:25:40.591466+0000","last_peered":"2024-01-16T15:25:40.591466+0000","last_clean":"2024-01-16T15:25:40.591466+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:25:40.591466+0000","last_undegraded":"2024-01-16T15:25:40.591466+0000","last_fullsized":"2024-01-16T15:25:40.591466+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:45.204823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'64","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236252+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:27:49.236252+0000","last_peered":"2024-01-16T15:27:49.236252+0000","last_clean":"2024-01-16T15:27:49.236252+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:27:49.236252+0000","last_undegraded":"2024-01-16T15:27:49.236252+0000","last_fullsized":"2024-01-16T15:27:49.236252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:41.760446+0000","last_change":"2024-01-16T15:19:23.619253+0000","last_active":"2024-01-16T15:25:41.760446+0000","last_peered":"2024-01-16T15:25:41.760446+0000","last_clean":"2024-01-16T15:25:41.760446+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:25:41.760446+0000","last_undegraded":"2024-01-16T15:25:41.760446+0000","last_fullsized":"2024-01-16T15:25:41.760446+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:16.781683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'73","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243648+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:27:49.243648+0000","last_peered":"2024-01-16T15:27:49.243648+0000","last_clean":"2024-01-16T15:27:49.243648+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:27:49.243648+0000","last_undegraded":"2024-01-16T15:27:49.243648+0000","last_fullsized":"2024-01-16T15:27:49.243648+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:49.637451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'31","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:43.237902+0000","last_change":"2024-01-16T15:19:23.629871+0000","last_active":"2024-01-16T15:25:43.237902+0000","last_peered":"2024-01-16T15:25:43.237902+0000","last_clean":"2024-01-16T15:25:43.237902+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:25:43.237902+0000","last_undegraded":"2024-01-16T15:25:43.237902+0000","last_fullsized":"2024-01-16T15:25:43.237902+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:05:17.872229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251399+0000","last_change":"2024-01-16T15:19:24.621286+0000","last_active":"2024-01-16T15:27:49.251399+0000","last_peered":"2024-01-16T15:27:49.251399+0000","last_clean":"2024-01-16T15:27:49.251399+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:27:49.251399+0000","last_undegraded":"2024-01-16T15:27:49.251399+0000","last_fullsized":"2024-01-16T15:27:49.251399+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:57.053368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.603477+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:25:25.603477+0000","last_peered":"2024-01-16T15:25:25.603477+0000","last_clean":"2024-01-16T15:25:25.603477+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:25:25.603477+0000","last_undegraded":"2024-01-16T15:25:25.603477+0000","last_fullsized":"2024-01-16T15:25:25.603477+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:01.099875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'54","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249675+0000","last_change":"2024-01-16T15:19:24.621294+0000","last_active":"2024-01-16T15:27:49.249675+0000","last_peered":"2024-01-16T15:27:49.249675+0000","last_clean":"2024-01-16T15:27:49.249675+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:27:49.249675+0000","last_undegraded":"2024-01-16T15:27:49.249675+0000","last_fullsized":"2024-01-16T15:27:49.249675+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.575031+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:24:05.575031+0000","last_peered":"2024-01-16T15:24:05.575031+0000","last_clean":"2024-01-16T15:24:05.575031+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:24:05.575031+0000","last_undegraded":"2024-01-16T15:24:05.575031+0000","last_fullsized":"2024-01-16T15:24:05.575031+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:59.676130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":261,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:50.938466+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:25:50.938466+0000","last_peered":"2024-01-16T15:25:50.938466+0000","last_clean":"2024-01-16T15:25:50.938466+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:25:50.938466+0000","last_undegraded":"2024-01-16T15:25:50.938466+0000","last_fullsized":"2024-01-16T15:25:50.938466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4358,"ondisk_log_size":4358,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509780,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.1559999999999999,"15min":1.25},"min":{"1min":0.69899999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5409999999999999,"5min":1.7749999999999999,"15min":19.224},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.212,"15min":1.3140000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.615,"5min":1.694,"15min":28.603999999999999},"last":1.4990000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.22,"15min":1.2250000000000001},"min":{"1min":0.90500000000000003,"5min":0.80800000000000005,"15min":0.374},"max":{"1min":1.488,"5min":1.837,"15min":3.7069999999999999},"last":1.641},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.3080000000000001,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.68899999999999995,"15min":0.68400000000000005},"max":{"1min":1.651,"5min":1.8819999999999999,"15min":33.225000000000001},"last":1.4019999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3380000000000001,"15min":1.4199999999999999},"min":{"1min":1.0529999999999999,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.752,"5min":1.7909999999999999,"15min":19.126000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3080000000000001,"15min":1.3120000000000001},"min":{"1min":1.008,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.694,"5min":1.718,"15min":2.9750000000000001},"last":1.3129999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.76500000000000001},"max":{"1min":1.526,"5min":1.73,"15min":3.7999999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.446,"15min":1.5580000000000001},"min":{"1min":0.97399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":19.268999999999998},"last":1.528}]},{"osd":4,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2290000000000001,"15min":1.264},"min":{"1min":0.88300000000000001,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.347,"15min":1.3919999999999999},"min":{"1min":1.1180000000000001,"5min":0.70299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6659999999999999,"5min":1.7330000000000001,"15min":9.7059999999999995},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":1.0389999999999999,"5min":0.86099999999999999,"15min":0.79400000000000004},"max":{"1min":1.74,"5min":1.9450000000000001,"15min":3.9129999999999998},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.5620000000000001},"min":{"1min":0.82699999999999996,"5min":0.77900000000000003,"15min":0.55100000000000005},"max":{"1min":1.8720000000000001,"5min":1.9630000000000001,"15min":19.036999999999999},"last":1.595}]},{"osd":7,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.4359999999999999,"15min":1.4410000000000001},"min":{"1min":1.198,"5min":0.74299999999999999,"15min":0.54800000000000004},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":4.2240000000000002},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.528,"15min":1.571},"min":{"1min":1.224,"5min":0.98999999999999999,"15min":0.65100000000000002},"max":{"1min":1.728,"5min":2.181,"15min":4.0709999999999997},"last":1.464}]}]},{"osd":4,"up_from":15,"seq":64424509782,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.296,"15min":1.2330000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.427,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.534},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1799999999999999,"15min":1.1859999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.0509999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.88700000000000001,"5min":0.73899999999999999,"15min":0.71899999999999997},"max":{"1min":1.6160000000000001,"5min":1.893,"15min":2.0510000000000002},"last":1.663},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.3149999999999999,"15min":1.302},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.6319999999999999,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3420000000000001,"15min":1.3320000000000001},"min":{"1min":0.79300000000000004,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.573,"5min":1.78,"15min":5.907},"last":1.226},{"interface":"front","average":{"1min":1.242,"5min":1.381,"15min":1.355},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.52600000000000002},"max":{"1min":1.6000000000000001,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":1.4330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4570000000000001,"15min":1.3939999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.76200000000000001},"max":{"1min":1.5289999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.4019999999999999,"15min":1.391},"min":{"1min":1.052,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.74,"5min":1.9670000000000001,"15min":2.0379999999999998},"last":1.3859999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.425,"15min":1.448},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6779999999999999,"5min":6.165,"15min":6.165},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4410000000000001,"15min":1.4770000000000001},"min":{"1min":0.95299999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8180000000000001,"5min":3.1469999999999998,"15min":5.3520000000000003},"last":1.4890000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.391,"15min":1.4219999999999999},"min":{"1min":1.101,"5min":0.90800000000000003,"15min":0.497},"max":{"1min":1.903,"5min":1.903,"15min":8.9600000000000009},"last":1.575},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5640000000000001,"15min":1.528},"min":{"1min":1.165,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.772,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.3300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.452,"15min":1.492},"min":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.89700000000000002},"max":{"1min":1.8500000000000001,"5min":3.3620000000000001,"15min":5.835},"last":1.369},{"interface":"front","average":{"1min":1.554,"5min":1.5469999999999999,"15min":1.581},"min":{"1min":1.238,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":2.0070000000000001,"5min":2.3679999999999999,"15min":8.0079999999999991},"last":1.724}]}]},{"osd":7,"up_from":14,"seq":60129542484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1891948,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92479892,"statfs":{"total":96636764160,"available":94699409408,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1659999999999999,"15min":1.2130000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.65400000000000003},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.1779999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.24,"5min":1.2210000000000001,"15min":1.248},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":12.461},"last":1.042}]},{"osd":1,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.89000000000000001,"5min":0.85099999999999998,"15min":0.48599999999999999},"max":{"1min":1.923,"5min":1.923,"15min":2.4249999999999998},"last":1.357},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2709999999999999,"15min":1.363},"min":{"1min":0.93400000000000005,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":1.8140000000000001,"5min":1.9079999999999999,"15min":14.848000000000001},"last":1.1599999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3069999999999999,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":3.726},"last":1.099},{"interface":"front","average":{"1min":1.286,"5min":1.3340000000000001,"15min":1.3959999999999999},"min":{"1min":0.88900000000000001,"5min":0.873,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.792,"15min":19.763999999999999},"last":1.5269999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3340000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80600000000000005},"max":{"1min":1.6539999999999999,"5min":1.8280000000000001,"15min":1.887},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.47,"15min":1.4650000000000001},"min":{"1min":1.113,"5min":1.0169999999999999,"15min":0.629},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":6.6029999999999998},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.345,"15min":1.347},"min":{"1min":0.98499999999999999,"5min":0.89100000000000001,"15min":0.54100000000000004},"max":{"1min":1.7090000000000001,"5min":1.7929999999999999,"15min":4.1390000000000002},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.401,"15min":1.413},"min":{"1min":1.0009999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":1.873,"5min":1.877,"15min":2.5190000000000001},"last":1.5920000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3340000000000001,"15min":1.3680000000000001},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":12.609999999999999},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5229999999999999},"min":{"1min":1.1719999999999999,"5min":1.046,"15min":0.66000000000000003},"max":{"1min":1.8879999999999999,"5min":1.909,"15min":3.4159999999999999},"last":1.486}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.099,"5min":1.0760000000000001,"15min":0.66000000000000003},"max":{"1min":2.0859999999999999,"5min":2.1429999999999998,"15min":2.653},"last":1.641},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.552},"min":{"1min":1.004,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":12.567},"last":1.7}]}]},{"osd":5,"up_from":14,"seq":60129542486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.208},"min":{"1min":0.83399999999999996,"5min":0.749,"15min":0.32900000000000001},"max":{"1min":1.7190000000000001,"5min":2.0939999999999999,"15min":6.8380000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1870000000000001,"15min":1.3200000000000001},"min":{"1min":1.0049999999999999,"5min":0.82999999999999996,"15min":0.57099999999999995},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":23.199999999999999},"last":1.2290000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.246},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.63400000000000001},"max":{"1min":1.637,"5min":1.698,"15min":8.0909999999999993},"last":0.996},{"interface":"front","average":{"1min":1.355,"5min":1.26,"15min":1.323},"min":{"1min":0.93600000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8049999999999999,"5min":1.851,"15min":17.062000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.409,"15min":1.415},"min":{"1min":1.069,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.6970000000000001,"5min":1.782,"15min":5.8810000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":1.131,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.6899999999999999,"5min":1.6970000000000001,"15min":5.4429999999999996},"last":1.022}]},{"osd":3,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.302,"15min":1.3420000000000001},"min":{"1min":0.94999999999999996,"5min":0.878,"15min":0.67500000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":7.984},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.385,"5min":1.4179999999999999,"15min":1.423},"min":{"1min":0.94599999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.7410000000000001,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":0.69099999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.323,"15min":1.387},"min":{"1min":0.878,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":15.975},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4079999999999999,"15min":1.413},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.54000000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":5.6349999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.329,"15min":1.3680000000000001},"min":{"1min":0.95999999999999996,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.8160000000000001,"5min":1.827,"15min":12.914},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.468,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":16.552},"last":1.165}]},{"osd":7,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4630000000000001,"15min":1.506},"min":{"1min":1.298,"5min":0.98099999999999998,"15min":0.64300000000000002},"max":{"1min":1.74,"5min":1.8540000000000001,"15min":13.034000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5740000000000001,"15min":1.5900000000000001},"min":{"1min":1.161,"5min":1.0429999999999999,"15min":0.70899999999999996},"max":{"1min":1.9159999999999999,"5min":2.073,"15min":8.0380000000000003},"last":0.95799999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542486,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":5.7400000000000002},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.1359999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40600000000000003},"max":{"1min":1.506,"5min":1.879,"15min":7.1429999999999998},"last":0.91100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.96899999999999997,"15min":1.0349999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.365,"5min":1.738,"15min":10.856999999999999},"last":1.224},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.1870000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4950000000000001,"5min":1.823,"15min":2.0089999999999999},"last":1.052}]},{"osd":2,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.3140000000000001,"15min":1.5209999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.946,"5min":1.946,"15min":39.265999999999998},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1559999999999999,"15min":1.2070000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.381,"5min":1.833,"15min":7.9160000000000004},"last":1.097}]},{"osd":4,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.337,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.70999999999999996},"max":{"1min":1.5980000000000001,"5min":1.9419999999999999,"15min":5.6890000000000001},"last":1.01},{"interface":"front","average":{"1min":1.329,"5min":1.464,"15min":1.4810000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8120000000000001,"5min":2.0920000000000001,"15min":2.3330000000000002},"last":1.889}]},{"osd":5,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3440000000000001,"15min":1.4670000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5669999999999999,"5min":1.827,"15min":14.003},"last":2.1339999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.5049999999999999,"15min":1.577},"min":{"1min":0.877,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":1.9359999999999999,"5min":2.008,"15min":6.0679999999999996},"last":1.8160000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.464,"15min":1.502},"min":{"1min":0.875,"5min":0.875,"15min":0.69799999999999995},"max":{"1min":1.8899999999999999,"5min":2.0139999999999998,"15min":7.0060000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.486,"15min":1.5580000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8140000000000001,"5min":1.982,"15min":15.114000000000001},"last":1.966}]},{"osd":7,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4950000000000001,"15min":1.6160000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80400000000000005},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":12.378},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5780000000000001,"15min":1.6220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9650000000000001,"5min":2.23,"15min":4.6029999999999998},"last":2.2589999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542485,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1059928,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93311912,"statfs":{"total":96636764160,"available":95551397888,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.90500000000000003,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.41199999999999998,"15min":0.32400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":1.833,"5min":1.913,"15min":7.9960000000000004},"last":0.97499999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0880000000000001,"15min":1.1040000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":1.7130000000000001,"5min":2.294,"15min":3.056},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.171,"5min":1.143,"15min":1.175},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.556,"5min":1.877,"15min":4.3819999999999997},"last":1.3819999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.206,"15min":1.2789999999999999},"min":{"1min":0.61199999999999999,"5min":0.55300000000000005,"15min":0.44700000000000001},"max":{"1min":1.704,"5min":1.883,"15min":18.838000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4950000000000001,"15min":1.552},"min":{"1min":0.66200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":2.0099999999999998,"15min":19.375},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.335,"15min":1.4390000000000001},"min":{"1min":0.80600000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.8240000000000001,"5min":2.242,"15min":23.225000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4730000000000001,"15min":1.494},"min":{"1min":1.038,"5min":0.75900000000000001,"15min":0.72699999999999998},"max":{"1min":1.883,"5min":1.883,"15min":6.359},"last":1.6779999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4399999999999999,"15min":1.5009999999999999},"min":{"1min":1.0800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.917,"5min":2.3170000000000002,"15min":20.157},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.597,"15min":1.6100000000000001},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.623},"max":{"1min":1.9330000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.0449999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5289999999999999,"15min":1.595},"min":{"1min":1.018,"5min":0.89800000000000002,"15min":0.70499999999999996},"max":{"1min":1.7330000000000001,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.101},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6160000000000001,"15min":1.7170000000000001},"min":{"1min":0.85799999999999998,"5min":0.79300000000000004,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.1419999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5429999999999999,"15min":1.6699999999999999},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.254},{"interface":"front","average":{"1min":1.587,"5min":1.569,"15min":1.5880000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":1.8999999999999999,"5min":2.0990000000000002,"15min":4.3419999999999996},"last":1.7569999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542487,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.093,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.28100000000000003},"max":{"1min":13.42,"5min":13.42,"15min":13.42},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.216,"5min":1.141,"15min":1.113},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":1.4970000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.03,"15min":1.0900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.649,"15min":3.6819999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.29,"15min":1.3320000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.39000000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":3.5600000000000001},"last":1.4610000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.26,"15min":1.2889999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46300000000000002},"max":{"1min":1.488,"5min":1.875,"15min":2.238},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.137,"15min":1.194},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":0.94299999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.72199999999999998},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.077}]},{"osd":5,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3939999999999999,"15min":1.431},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63700000000000001},"max":{"1min":1.6359999999999999,"5min":1.952,"15min":3.98},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4530000000000001,"15min":1.462},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.53300000000000003},"max":{"1min":1.669,"5min":1.893,"15min":3.3940000000000001},"last":1.385}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.56999999999999995},"max":{"1min":1.8660000000000001,"5min":2.016,"15min":16.48},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.508,"15min":1.532},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":0.65600000000000003},"max":{"1min":1.74,"5min":1.909,"15min":3.8180000000000001},"last":1.526}]},{"osd":7,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.573,"15min":1.617},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.48099999999999998},"max":{"1min":1.8480000000000001,"5min":2.0009999999999999,"15min":7.0499999999999998},"last":1.637},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.6339999999999999,"15min":1.647},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.81100000000000005},"max":{"1min":1.8640000000000001,"5min":2.3940000000000001,"15min":3.7839999999999998},"last":1.5700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542485,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":1.038},"min":{"1min":0.496,"5min":0.496,"15min":0.35199999999999998},"max":{"1min":1.427,"5min":1.665,"15min":15.776},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.151,"15min":1.171},"min":{"1min":0.67600000000000005,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":1.9019999999999999,"5min":2.7549999999999999,"15min":17.620999999999999},"last":1.119}]},{"osd":2,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.137,"15min":1.22},"min":{"1min":0.74099999999999999,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.671,"5min":1.788,"15min":15.997999999999999},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.23,"15min":1.1919999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":1.669,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.2490000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.129,"15min":1.179},"min":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.4279999999999999},"min":{"1min":0.86299999999999999,"5min":0.66600000000000004,"15min":0.47399999999999998},"max":{"1min":1.74,"5min":1.827,"15min":16.126000000000001},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.466,"15min":1.573},"min":{"1min":1.095,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.9470000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.675},{"interface":"front","average":{"1min":1.446,"5min":1.4430000000000001,"15min":1.419},"min":{"1min":1.0089999999999999,"5min":0.82299999999999995,"15min":0.71899999999999997},"max":{"1min":1.913,"5min":3.0099999999999998,"15min":3.0459999999999998},"last":1.395}]},{"osd":5,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.97099999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7869999999999999,"5min":2.157,"15min":5.5609999999999999},"last":1.351},{"interface":"front","average":{"1min":1.581,"5min":1.5529999999999999,"15min":1.639},"min":{"1min":1.2130000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0630000000000002,"5min":2.1989999999999998,"15min":16.501999999999999},"last":1.421}]},{"osd":6,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.137,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.891,"5min":2.0790000000000002,"15min":5.4509999999999996},"last":1.542},{"interface":"front","average":{"1min":1.649,"5min":1.6379999999999999,"15min":1.7190000000000001},"min":{"1min":0.98899999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":2.1269999999999998,"5min":2.351,"15min":16.395},"last":1.766}]},{"osd":7,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5800000000000001,"15min":1.595},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":2.3300000000000001,"5min":2.9510000000000001,"15min":9.6980000000000004},"last":1.484},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.589,"15min":1.5860000000000001},"min":{"1min":1.0860000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.954,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.633}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:02.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:02.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.559+0000 7fe71e800640 1 Processor -- start 2024-01-16T15:28:02.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.559+0000 7fe71e800640 1 -- start start 2024-01-16T15:28:02.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe71e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180b3ae0 0x7fe71815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe71e800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe7180cf290 0x7fe7180cf720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe71e800640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe718070c40 con 0x7fe7180ae5a0 2024-01-16T15:28:02.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe71e800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe718071450 con 0x7fe7180cf290 2024-01-16T15:28:02.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe71e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7180cfc60 con 0x7fe7180b3ae0 2024-01-16T15:28:02.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe717400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fe7180ae5a0 0x7fe71806cbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57588/0 (socket says 172.21.15.53:57588) 2024-01-16T15:28:02.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe717400640 1 -- 172.21.15.53:0/3104420497 learned_addr learned my addr 172.21.15.53:0/3104420497 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:02.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe717e00640 1 --2- 172.21.15.53:0/3104420497 >> v2:172.21.15.53:3300/0 conn(0x7fe7180cf290 0x7fe7180cf720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.562+0000 7fe70e800640 1 --2- 172.21.15.53:0/3104420497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180b3ae0 0x7fe71815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.561+0000 7fe716a00640 1 -- 172.21.15.53:0/3104420497 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148067497 0 0) 0x7fe718070c40 con 0x7fe7180ae5a0 2024-01-16T15:28:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.562+0000 7fe716a00640 1 -- 172.21.15.53:0/3104420497 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6f8003520 con 0x7fe7180ae5a0 2024-01-16T15:28:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.562+0000 7fe717e00640 1 -- 172.21.15.53:0/3104420497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180b3ae0 msgr2=0x7fe71815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.562+0000 7fe717e00640 1 --2- 172.21.15.53:0/3104420497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180b3ae0 0x7fe71815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.562+0000 7fe717e00640 1 -- 172.21.15.53:0/3104420497 >> v1:172.21.15.53:6789/0 conn(0x7fe7180ae5a0 legacy=0x7fe71806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.562+0000 7fe717e00640 1 -- 172.21.15.53:0/3104420497 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe71806c470 con 0x7fe7180cf290 2024-01-16T15:28:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.563+0000 7fe717e00640 1 --2- 172.21.15.53:0/3104420497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180cf290 0x7fe7180cf720 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fe7100097a0 tx=0x7fe71002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cfe93f4674da40a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe716a00640 1 -- 172.21.15.53:0/3104420497 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710030610 con 0x7fe7180cf290 2024-01-16T15:28:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe716a00640 1 -- 172.21.15.53:0/3104420497 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe710030770 con 0x7fe7180cf290 2024-01-16T15:28:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe716a00640 1 -- 172.21.15.53:0/3104420497 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe710030ae0 con 0x7fe7180cf290 2024-01-16T15:28:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe71e800640 1 -- 172.21.15.53:0/3104420497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180cf290 msgr2=0x7fe7180cf720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe71e800640 1 --2- 172.21.15.53:0/3104420497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180cf290 0x7fe7180cf720 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fe7100097a0 tx=0x7fe71002b960 comp rx=0 tx=0).stop 2024-01-16T15:28:02.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe71e800640 1 -- 172.21.15.53:0/3104420497 shutdown_connections 2024-01-16T15:28:02.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe71e800640 1 --2- 172.21.15.53:0/3104420497 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180cf290 0x7fe7180cf720 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe71e800640 1 --2- 172.21.15.53:0/3104420497 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180b3ae0 0x7fe71815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.564+0000 7fe71e800640 1 -- 172.21.15.53:0/3104420497 >> 172.21.15.53:0/3104420497 conn(0x7fe71806a800 msgr2=0x7fe71806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:02.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.565+0000 7fe71e800640 1 -- 172.21.15.53:0/3104420497 shutdown_connections 2024-01-16T15:28:02.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.565+0000 7fe71e800640 1 -- 172.21.15.53:0/3104420497 wait complete. 2024-01-16T15:28:02.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.566+0000 7fe71e800640 1 Processor -- start 2024-01-16T15:28:02.566 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.567+0000 7fe71e800640 1 -- start start 2024-01-16T15:28:02.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.567+0000 7fe71e800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180ae5a0 0x7fe71814ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.567+0000 7fe71e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 0x7fe718160350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe71e800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7180cf290 0x7fe718157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe71e800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7181505f0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe71e800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe718151d20 con 0x7fe7180cf290 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe71e800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe71815fa40 con 0x7fe7180ae5a0 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe70e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 0x7fe718160350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe70e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 0x7fe718160350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55506/0 (socket says 172.21.15.53:55506) 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe70e800640 1 -- 172.21.15.53:0/1139176410 learned_addr learned my addr 172.21.15.53:0/1139176410 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe717e00640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7180cf290 0x7fe718157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.568+0000 7fe717400640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180ae5a0 0x7fe71814ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70e800640 1 -- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180ae5a0 msgr2=0x7fe71814ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180ae5a0 0x7fe71814ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70e800640 1 -- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7180cf290 msgr2=0x7fe718157eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7180cf290 0x7fe718157eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70e800640 1 -- 172.21.15.53:0/1139176410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe710009400 con 0x7fe7180b3ae0 2024-01-16T15:28:02.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 0x7fe718160350 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fe70000d830 tx=0x7fe70000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:02.570 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70000b6c0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70000b820 con 0x7fe7180b3ae0 2024-01-16T15:28:02.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70000bbb0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.569+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe71814f1d0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.570+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe71815f1a0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.571+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7180675d0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.572+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe70001e460 con 0x7fe7180b3ae0 2024-01-16T15:28:02.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.573+0000 7fe70fe00640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6f006c550 0x7fe6f006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:02.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.573+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe7000891d0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.572 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.573+0000 7fe717400640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6f006c550 0x7fe6f006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:02.573 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.574+0000 7fe717400640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6f006c550 0x7fe6f006ea10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe708005c00 tx=0x7fe708009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:02.576 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.577+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fe7000557c0 con 0x7fe7180b3ae0 2024-01-16T15:28:02.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.767+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe718071170 con 0x7fe6f006c550 2024-01-16T15:28:02.789 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.790+0000 7fe70fe00640 1 -- 172.21.15.53:0/1139176410 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288968 (secure 0 0 0) 0x7fe70800feb0 con 0x7fe6f006c550 2024-01-16T15:28:02.793 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:02.794 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:28:02.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6f006c550 msgr2=0x7fe6f006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6f006c550 0x7fe6f006ea10 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7fe708005c00 tx=0x7fe708009040 comp rx=0 tx=0).stop 2024-01-16T15:28:02.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 msgr2=0x7fe718160350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:02.796 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 0x7fe718160350 secure :-1 s=READY pgs=2020 cs=0 l=1 rev1=1 crypto rx=0x7fe70000d830 tx=0x7fe70000dcf0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 shutdown_connections 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6f006c550 0x7fe6f006ea10 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe7180cf290 0x7fe718157eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe7180b3ae0 0x7fe718160350 unknown :-1 s=CLOSED pgs=2020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 --2- 172.21.15.53:0/1139176410 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe7180ae5a0 0x7fe71814ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 >> 172.21.15.53:0/1139176410 conn(0x7fe71806a800 msgr2=0x7fe71814fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 shutdown_connections 2024-01-16T15:28:02.797 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:02.796+0000 7fe71e800640 1 -- 172.21.15.53:0/1139176410 wait complete. 2024-01-16T15:28:02.816 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":953,"stamp":"2024-01-16T15:28:00.906556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4907,"ondisk_log_size":4907,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11125304,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":625011,"kb_avail":743849416,"statfs":{"total":773094113280,"available":761701801984,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":640011824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12773,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004945"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256079+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.256079+0000","last_peered":"2024-01-16T15:27:49.256079+0000","last_clean":"2024-01-16T15:27:49.256079+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.256079+0000","last_undegraded":"2024-01-16T15:27:49.256079+0000","last_fullsized":"2024-01-16T15:27:49.256079+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:08:35.875409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'73","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237782+0000","last_change":"2024-01-16T15:19:24.622288+0000","last_active":"2024-01-16T15:27:49.237782+0000","last_peered":"2024-01-16T15:27:49.237782+0000","last_clean":"2024-01-16T15:27:49.237782+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:27:49.237782+0000","last_undegraded":"2024-01-16T15:27:49.237782+0000","last_fullsized":"2024-01-16T15:27:49.237782+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:20.176733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251487+0000","last_change":"2024-01-16T15:19:24.618862+0000","last_active":"2024-01-16T15:27:49.251487+0000","last_peered":"2024-01-16T15:27:49.251487+0000","last_clean":"2024-01-16T15:27:49.251487+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:27:49.251487+0000","last_undegraded":"2024-01-16T15:27:49.251487+0000","last_fullsized":"2024-01-16T15:27:49.251487+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:02.115872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'56","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.231727+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:27:49.231727+0000","last_peered":"2024-01-16T15:27:49.231727+0000","last_clean":"2024-01-16T15:27:49.231727+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:27:49.231727+0000","last_undegraded":"2024-01-16T15:27:49.231727+0000","last_fullsized":"2024-01-16T15:27:49.231727+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:32.713991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'62","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.232748+0000","last_change":"2024-01-16T15:19:24.626093+0000","last_active":"2024-01-16T15:27:49.232748+0000","last_peered":"2024-01-16T15:27:49.232748+0000","last_clean":"2024-01-16T15:27:49.232748+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:27:49.232748+0000","last_undegraded":"2024-01-16T15:27:49.232748+0000","last_fullsized":"2024-01-16T15:27:49.232748+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:45.086271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'76","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239918+0000","last_change":"2024-01-16T15:19:24.625809+0000","last_active":"2024-01-16T15:27:49.239918+0000","last_peered":"2024-01-16T15:27:49.239918+0000","last_clean":"2024-01-16T15:27:49.239918+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:27:49.239918+0000","last_undegraded":"2024-01-16T15:27:49.239918+0000","last_fullsized":"2024-01-16T15:27:49.239918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:21.453533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'67","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.241483+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:27:49.241483+0000","last_peered":"2024-01-16T15:27:49.241483+0000","last_clean":"2024-01-16T15:27:49.241483+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:27:49.241483+0000","last_undegraded":"2024-01-16T15:27:49.241483+0000","last_fullsized":"2024-01-16T15:27:49.241483+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:15:57.152091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'58","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248998+0000","last_change":"2024-01-16T15:19:24.616757+0000","last_active":"2024-01-16T15:27:49.248998+0000","last_peered":"2024-01-16T15:27:49.248998+0000","last_clean":"2024-01-16T15:27:49.248998+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:27:49.248998+0000","last_undegraded":"2024-01-16T15:27:49.248998+0000","last_fullsized":"2024-01-16T15:27:49.248998+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:30.687663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.255855+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:27:49.255855+0000","last_peered":"2024-01-16T15:27:49.255855+0000","last_clean":"2024-01-16T15:27:49.255855+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:27:49.255855+0000","last_undegraded":"2024-01-16T15:27:49.255855+0000","last_fullsized":"2024-01-16T15:27:49.255855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:07.972777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257125+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:27:49.257125+0000","last_peered":"2024-01-16T15:27:49.257125+0000","last_clean":"2024-01-16T15:27:49.257125+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:27:49.257125+0000","last_undegraded":"2024-01-16T15:27:49.257125+0000","last_fullsized":"2024-01-16T15:27:49.257125+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:41.949199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'70","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.227737+0000","last_change":"2024-01-16T15:19:24.625137+0000","last_active":"2024-01-16T15:27:49.227737+0000","last_peered":"2024-01-16T15:27:49.227737+0000","last_clean":"2024-01-16T15:27:49.227737+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:27:49.227737+0000","last_undegraded":"2024-01-16T15:27:49.227737+0000","last_fullsized":"2024-01-16T15:27:49.227737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'64","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236182+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:27:49.236182+0000","last_peered":"2024-01-16T15:27:49.236182+0000","last_clean":"2024-01-16T15:27:49.236182+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:27:49.236182+0000","last_undegraded":"2024-01-16T15:27:49.236182+0000","last_fullsized":"2024-01-16T15:27:49.236182+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:27.512415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'65","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242745+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:27:49.242745+0000","last_peered":"2024-01-16T15:27:49.242745+0000","last_clean":"2024-01-16T15:27:49.242745+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:27:49.242745+0000","last_undegraded":"2024-01-16T15:27:49.242745+0000","last_fullsized":"2024-01-16T15:27:49.242745+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:08.270535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'72","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.245113+0000","last_change":"2024-01-16T15:19:24.624351+0000","last_active":"2024-01-16T15:27:49.245113+0000","last_peered":"2024-01-16T15:27:49.245113+0000","last_clean":"2024-01-16T15:27:49.245113+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:27:49.245113+0000","last_undegraded":"2024-01-16T15:27:49.245113+0000","last_fullsized":"2024-01-16T15:27:49.245113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:41.925967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'59","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246430+0000","last_change":"2024-01-16T15:19:24.613444+0000","last_active":"2024-01-16T15:27:49.246430+0000","last_peered":"2024-01-16T15:27:49.246430+0000","last_clean":"2024-01-16T15:27:49.246430+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:27:49.246430+0000","last_undegraded":"2024-01-16T15:27:49.246430+0000","last_fullsized":"2024-01-16T15:27:49.246430+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:59.930379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'74","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248683+0000","last_change":"2024-01-16T15:19:24.614694+0000","last_active":"2024-01-16T15:27:49.248683+0000","last_peered":"2024-01-16T15:27:49.248683+0000","last_clean":"2024-01-16T15:27:49.248683+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:27:49.248683+0000","last_undegraded":"2024-01-16T15:27:49.248683+0000","last_fullsized":"2024-01-16T15:27:49.248683+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:59.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'52","reported_seq":73,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.228316+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:27:49.228316+0000","last_peered":"2024-01-16T15:27:49.228316+0000","last_clean":"2024-01-16T15:27:49.228316+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:27:49.228316+0000","last_undegraded":"2024-01-16T15:27:49.228316+0000","last_fullsized":"2024-01-16T15:27:49.228316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.235929+0000","last_change":"2024-01-16T15:19:24.629022+0000","last_active":"2024-01-16T15:27:49.235929+0000","last_peered":"2024-01-16T15:27:49.235929+0000","last_clean":"2024-01-16T15:27:49.235929+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:27:49.235929+0000","last_undegraded":"2024-01-16T15:27:49.235929+0000","last_fullsized":"2024-01-16T15:27:49.235929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:22.444157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'74","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250216+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:27:49.250216+0000","last_peered":"2024-01-16T15:27:49.250216+0000","last_clean":"2024-01-16T15:27:49.250216+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:27:49.250216+0000","last_undegraded":"2024-01-16T15:27:49.250216+0000","last_fullsized":"2024-01-16T15:27:49.250216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:11.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247569+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:27:49.247569+0000","last_peered":"2024-01-16T15:27:49.247569+0000","last_clean":"2024-01-16T15:27:49.247569+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:27:49.247569+0000","last_undegraded":"2024-01-16T15:27:49.247569+0000","last_fullsized":"2024-01-16T15:27:49.247569+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'73","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248711+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:27:49.248711+0000","last_peered":"2024-01-16T15:27:49.248711+0000","last_clean":"2024-01-16T15:27:49.248711+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:27:49.248711+0000","last_undegraded":"2024-01-16T15:27:49.248711+0000","last_fullsized":"2024-01-16T15:27:49.248711+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'71","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233752+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:27:49.233752+0000","last_peered":"2024-01-16T15:27:49.233752+0000","last_clean":"2024-01-16T15:27:49.233752+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:27:49.233752+0000","last_undegraded":"2024-01-16T15:27:49.233752+0000","last_fullsized":"2024-01-16T15:27:49.233752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:26.924342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'63","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247674+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:27:49.247674+0000","last_peered":"2024-01-16T15:27:49.247674+0000","last_clean":"2024-01-16T15:27:49.247674+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:27:49.247674+0000","last_undegraded":"2024-01-16T15:27:49.247674+0000","last_fullsized":"2024-01-16T15:27:49.247674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:07.051272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'71","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249786+0000","last_change":"2024-01-16T15:19:24.626920+0000","last_active":"2024-01-16T15:27:49.249786+0000","last_peered":"2024-01-16T15:27:49.249786+0000","last_clean":"2024-01-16T15:27:49.249786+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:27:49.249786+0000","last_undegraded":"2024-01-16T15:27:49.249786+0000","last_fullsized":"2024-01-16T15:27:49.249786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'69","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233273+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:27:49.233273+0000","last_peered":"2024-01-16T15:27:49.233273+0000","last_clean":"2024-01-16T15:27:49.233273+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:27:49.233273+0000","last_undegraded":"2024-01-16T15:27:49.233273+0000","last_fullsized":"2024-01-16T15:27:49.233273+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251042+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:27:49.251042+0000","last_peered":"2024-01-16T15:27:49.251042+0000","last_clean":"2024-01-16T15:27:49.251042+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:27:49.251042+0000","last_undegraded":"2024-01-16T15:27:49.251042+0000","last_fullsized":"2024-01-16T15:27:49.251042+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:49.687478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'65","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.226338+0000","last_change":"2024-01-16T15:19:24.623489+0000","last_active":"2024-01-16T15:27:49.226338+0000","last_peered":"2024-01-16T15:27:49.226338+0000","last_clean":"2024-01-16T15:27:49.226338+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:27:49.226338+0000","last_undegraded":"2024-01-16T15:27:49.226338+0000","last_fullsized":"2024-01-16T15:27:49.226338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:11.064893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'50","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.241685+0000","last_change":"2024-01-16T15:19:42.218585+0000","last_active":"2024-01-16T15:27:49.241685+0000","last_peered":"2024-01-16T15:27:49.241685+0000","last_clean":"2024-01-16T15:27:49.241685+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:27:49.241685+0000","last_undegraded":"2024-01-16T15:27:49.241685+0000","last_fullsized":"2024-01-16T15:27:49.241685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:02.749154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'62","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248953+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:27:49.248953+0000","last_peered":"2024-01-16T15:27:49.248953+0000","last_clean":"2024-01-16T15:27:49.248953+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:27:49.248953+0000","last_undegraded":"2024-01-16T15:27:49.248953+0000","last_fullsized":"2024-01-16T15:27:49.248953+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:40.862803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'78","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246500+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:27:49.246500+0000","last_peered":"2024-01-16T15:27:49.246500+0000","last_clean":"2024-01-16T15:27:49.246500+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:27:49.246500+0000","last_undegraded":"2024-01-16T15:27:49.246500+0000","last_fullsized":"2024-01-16T15:27:49.246500+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:45.144417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'91","reported_seq":120,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252053+0000","last_change":"2024-01-16T15:19:24.629910+0000","last_active":"2024-01-16T15:27:49.252053+0000","last_peered":"2024-01-16T15:27:49.252053+0000","last_clean":"2024-01-16T15:27:49.252053+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:27:49.252053+0000","last_undegraded":"2024-01-16T15:27:49.252053+0000","last_fullsized":"2024-01-16T15:27:49.252053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:44.059899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'78","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246839+0000","last_change":"2024-01-16T15:19:24.633567+0000","last_active":"2024-01-16T15:27:49.246839+0000","last_peered":"2024-01-16T15:27:49.246839+0000","last_clean":"2024-01-16T15:27:49.246839+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:27:49.246839+0000","last_undegraded":"2024-01-16T15:27:49.246839+0000","last_fullsized":"2024-01-16T15:27:49.246839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:24.582502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'60","reported_seq":83,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.219133+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:27:49.219133+0000","last_peered":"2024-01-16T15:27:49.219133+0000","last_clean":"2024-01-16T15:27:49.219133+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:27:49.219133+0000","last_undegraded":"2024-01-16T15:27:49.219133+0000","last_fullsized":"2024-01-16T15:27:49.219133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:01.155551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":22,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.591516+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:25:40.591516+0000","last_peered":"2024-01-16T15:25:40.591516+0000","last_clean":"2024-01-16T15:25:40.591516+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:25:40.591516+0000","last_undegraded":"2024-01-16T15:25:40.591516+0000","last_fullsized":"2024-01-16T15:25:40.591516+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.038677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'67","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247582+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:27:49.247582+0000","last_peered":"2024-01-16T15:27:49.247582+0000","last_clean":"2024-01-16T15:27:49.247582+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:27:49.247582+0000","last_undegraded":"2024-01-16T15:27:49.247582+0000","last_fullsized":"2024-01-16T15:27:49.247582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:25.905229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.933509+0000","last_change":"2024-01-16T15:19:23.621310+0000","last_active":"2024-01-16T15:25:25.933509+0000","last_peered":"2024-01-16T15:25:25.933509+0000","last_clean":"2024-01-16T15:25:25.933509+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:25:25.933509+0000","last_undegraded":"2024-01-16T15:25:25.933509+0000","last_fullsized":"2024-01-16T15:25:25.933509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:19.958286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'66","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246028+0000","last_change":"2024-01-16T15:19:24.630161+0000","last_active":"2024-01-16T15:27:49.246028+0000","last_peered":"2024-01-16T15:27:49.246028+0000","last_clean":"2024-01-16T15:27:49.246028+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:27:49.246028+0000","last_undegraded":"2024-01-16T15:27:49.246028+0000","last_fullsized":"2024-01-16T15:27:49.246028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:54:32.432478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'3","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.565625+0000","last_change":"2024-01-16T15:19:23.630011+0000","last_active":"2024-01-16T15:24:25.565625+0000","last_peered":"2024-01-16T15:24:25.565625+0000","last_clean":"2024-01-16T15:24:25.565625+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:24:25.565625+0000","last_undegraded":"2024-01-16T15:24:25.565625+0000","last_fullsized":"2024-01-16T15:24:25.565625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:21.485125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'84","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250676+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:27:49.250676+0000","last_peered":"2024-01-16T15:27:49.250676+0000","last_clean":"2024-01-16T15:27:49.250676+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:27:49.250676+0000","last_undegraded":"2024-01-16T15:27:49.250676+0000","last_fullsized":"2024-01-16T15:27:49.250676+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:53.269911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.565587+0000","last_change":"2024-01-16T15:19:23.618597+0000","last_active":"2024-01-16T15:24:25.565587+0000","last_peered":"2024-01-16T15:24:25.565587+0000","last_clean":"2024-01-16T15:24:25.565587+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:24:25.565587+0000","last_undegraded":"2024-01-16T15:24:25.565587+0000","last_fullsized":"2024-01-16T15:24:25.565587+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:05.775312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'78","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250467+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:27:49.250467+0000","last_peered":"2024-01-16T15:27:49.250467+0000","last_clean":"2024-01-16T15:27:49.250467+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:27:49.250467+0000","last_undegraded":"2024-01-16T15:27:49.250467+0000","last_fullsized":"2024-01-16T15:27:49.250467+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:01.404745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.578781+0000","last_change":"2024-01-16T15:19:23.618024+0000","last_active":"2024-01-16T15:24:25.578781+0000","last_peered":"2024-01-16T15:24:25.578781+0000","last_clean":"2024-01-16T15:24:25.578781+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:24:25.578781+0000","last_undegraded":"2024-01-16T15:24:25.578781+0000","last_fullsized":"2024-01-16T15:24:25.578781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:50.476809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'82","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234990+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:27:49.234990+0000","last_peered":"2024-01-16T15:27:49.234990+0000","last_clean":"2024-01-16T15:27:49.234990+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:27:49.234990+0000","last_undegraded":"2024-01-16T15:27:49.234990+0000","last_fullsized":"2024-01-16T15:27:49.234990+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.321825+0000","last_change":"2024-01-16T15:19:23.622641+0000","last_active":"2024-01-16T15:24:25.321825+0000","last_peered":"2024-01-16T15:24:25.321825+0000","last_clean":"2024-01-16T15:24:25.321825+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:24:25.321825+0000","last_undegraded":"2024-01-16T15:24:25.321825+0000","last_fullsized":"2024-01-16T15:24:25.321825+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:39.846805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'80","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250148+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:27:49.250148+0000","last_peered":"2024-01-16T15:27:49.250148+0000","last_clean":"2024-01-16T15:27:49.250148+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:27:49.250148+0000","last_undegraded":"2024-01-16T15:27:49.250148+0000","last_fullsized":"2024-01-16T15:27:49.250148+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:19:48.939386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.744702+0000","last_change":"2024-01-16T15:19:23.620941+0000","last_active":"2024-01-16T15:24:16.744702+0000","last_peered":"2024-01-16T15:24:16.744702+0000","last_clean":"2024-01-16T15:24:16.744702+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:24:16.744702+0000","last_undegraded":"2024-01-16T15:24:16.744702+0000","last_fullsized":"2024-01-16T15:24:16.744702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:27.971375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'68","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234098+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:27:49.234098+0000","last_peered":"2024-01-16T15:27:49.234098+0000","last_clean":"2024-01-16T15:27:49.234098+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:27:49.234098+0000","last_undegraded":"2024-01-16T15:27:49.234098+0000","last_fullsized":"2024-01-16T15:27:49.234098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:53.297338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.587238+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:24:15.587238+0000","last_peered":"2024-01-16T15:24:15.587238+0000","last_clean":"2024-01-16T15:24:15.587238+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:24:15.587238+0000","last_undegraded":"2024-01-16T15:24:15.587238+0000","last_fullsized":"2024-01-16T15:24:15.587238+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:07.254538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'66","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242411+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:27:49.242411+0000","last_peered":"2024-01-16T15:27:49.242411+0000","last_clean":"2024-01-16T15:27:49.242411+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:27:49.242411+0000","last_undegraded":"2024-01-16T15:27:49.242411+0000","last_fullsized":"2024-01-16T15:27:49.242411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:40.365875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.563703+0000","last_change":"2024-01-16T15:19:23.621670+0000","last_active":"2024-01-16T15:24:15.563703+0000","last_peered":"2024-01-16T15:24:15.563703+0000","last_clean":"2024-01-16T15:24:15.563703+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:24:15.563703+0000","last_undegraded":"2024-01-16T15:24:15.563703+0000","last_fullsized":"2024-01-16T15:24:15.563703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:53.183004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'64","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237836+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:27:49.237836+0000","last_peered":"2024-01-16T15:27:49.237836+0000","last_clean":"2024-01-16T15:27:49.237836+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:27:49.237836+0000","last_undegraded":"2024-01-16T15:27:49.237836+0000","last_fullsized":"2024-01-16T15:27:49.237836+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:56.499150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.319813+0000","last_change":"2024-01-16T15:19:23.624332+0000","last_active":"2024-01-16T15:24:15.319813+0000","last_peered":"2024-01-16T15:24:15.319813+0000","last_clean":"2024-01-16T15:24:15.319813+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:24:15.319813+0000","last_undegraded":"2024-01-16T15:24:15.319813+0000","last_fullsized":"2024-01-16T15:24:15.319813+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:56.995452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'60","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243544+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:27:49.243544+0000","last_peered":"2024-01-16T15:27:49.243544+0000","last_clean":"2024-01-16T15:27:49.243544+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:27:49.243544+0000","last_undegraded":"2024-01-16T15:27:49.243544+0000","last_fullsized":"2024-01-16T15:27:49.243544+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:06.684073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'31","reported_seq":48,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:43.246740+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:25:43.246740+0000","last_peered":"2024-01-16T15:25:43.246740+0000","last_clean":"2024-01-16T15:25:43.246740+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:25:43.246740+0000","last_undegraded":"2024-01-16T15:25:43.246740+0000","last_fullsized":"2024-01-16T15:25:43.246740+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:48.585086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'69","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251340+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:27:49.251340+0000","last_peered":"2024-01-16T15:27:49.251340+0000","last_clean":"2024-01-16T15:27:49.251340+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:27:49.251340+0000","last_undegraded":"2024-01-16T15:27:49.251340+0000","last_fullsized":"2024-01-16T15:27:49.251340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:23.918834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'307","reported_seq":324,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.260197+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:27:59.260197+0000","last_peered":"2024-01-16T15:27:59.260197+0000","last_clean":"2024-01-16T15:27:59.260197+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:27:59.260197+0000","last_undegraded":"2024-01-16T15:27:59.260197+0000","last_fullsized":"2024-01-16T15:27:59.260197+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:52.253914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'63","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242837+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:27:49.242837+0000","last_peered":"2024-01-16T15:27:49.242837+0000","last_clean":"2024-01-16T15:27:49.242837+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:27:49.242837+0000","last_undegraded":"2024-01-16T15:27:49.242837+0000","last_fullsized":"2024-01-16T15:27:49.242837+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:37.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.569761+0000","last_change":"2024-01-16T15:19:23.617724+0000","last_active":"2024-01-16T15:24:05.569761+0000","last_peered":"2024-01-16T15:24:05.569761+0000","last_clean":"2024-01-16T15:24:05.569761+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:24:05.569761+0000","last_undegraded":"2024-01-16T15:24:05.569761+0000","last_fullsized":"2024-01-16T15:24:05.569761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:35.823425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'80","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249807+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:27:49.249807+0000","last_peered":"2024-01-16T15:27:49.249807+0000","last_clean":"2024-01-16T15:27:49.249807+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:27:49.249807+0000","last_undegraded":"2024-01-16T15:27:49.249807+0000","last_fullsized":"2024-01-16T15:27:49.249807+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.195535+0000","last_change":"2024-01-16T15:19:23.624683+0000","last_active":"2024-01-16T15:24:07.195535+0000","last_peered":"2024-01-16T15:24:07.195535+0000","last_clean":"2024-01-16T15:24:07.195535+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:24:07.195535+0000","last_undegraded":"2024-01-16T15:24:07.195535+0000","last_fullsized":"2024-01-16T15:24:07.195535+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:38.379463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'73","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244605+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:27:49.244605+0000","last_peered":"2024-01-16T15:27:49.244605+0000","last_clean":"2024-01-16T15:27:49.244605+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:27:49.244605+0000","last_undegraded":"2024-01-16T15:27:49.244605+0000","last_fullsized":"2024-01-16T15:27:49.244605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":25,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.561655+0000","last_change":"2024-01-16T15:19:23.625591+0000","last_active":"2024-01-16T15:24:05.561655+0000","last_peered":"2024-01-16T15:24:05.561655+0000","last_clean":"2024-01-16T15:24:05.561655+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:24:05.561655+0000","last_undegraded":"2024-01-16T15:24:05.561655+0000","last_fullsized":"2024-01-16T15:24:05.561655+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:27.727635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'50","reported_seq":70,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250374+0000","last_change":"2024-01-16T15:19:24.617249+0000","last_active":"2024-01-16T15:27:49.250374+0000","last_peered":"2024-01-16T15:27:49.250374+0000","last_clean":"2024-01-16T15:27:49.250374+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:27:49.250374+0000","last_undegraded":"2024-01-16T15:27:49.250374+0000","last_fullsized":"2024-01-16T15:27:49.250374+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:01.440713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.582905+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:23:55.582905+0000","last_peered":"2024-01-16T15:23:55.582905+0000","last_clean":"2024-01-16T15:23:55.582905+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:23:55.582905+0000","last_undegraded":"2024-01-16T15:23:55.582905+0000","last_fullsized":"2024-01-16T15:23:55.582905+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:27.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'61","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239584+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:27:49.239584+0000","last_peered":"2024-01-16T15:27:49.239584+0000","last_clean":"2024-01-16T15:27:49.239584+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:27:49.239584+0000","last_undegraded":"2024-01-16T15:27:49.239584+0000","last_fullsized":"2024-01-16T15:27:49.239584+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:20.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.560107+0000","last_change":"2024-01-16T15:19:23.622230+0000","last_active":"2024-01-16T15:23:55.560107+0000","last_peered":"2024-01-16T15:23:55.560107+0000","last_clean":"2024-01-16T15:23:55.560107+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:23:55.560107+0000","last_undegraded":"2024-01-16T15:23:55.560107+0000","last_fullsized":"2024-01-16T15:23:55.560107+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'77","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248743+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:27:49.248743+0000","last_peered":"2024-01-16T15:27:49.248743+0000","last_clean":"2024-01-16T15:27:49.248743+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:27:49.248743+0000","last_undegraded":"2024-01-16T15:27:49.248743+0000","last_fullsized":"2024-01-16T15:27:49.248743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:31.587040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.193814+0000","last_change":"2024-01-16T15:19:23.626903+0000","last_active":"2024-01-16T15:23:57.193814+0000","last_peered":"2024-01-16T15:23:57.193814+0000","last_clean":"2024-01-16T15:23:57.193814+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:23:57.193814+0000","last_undegraded":"2024-01-16T15:23:57.193814+0000","last_fullsized":"2024-01-16T15:23:57.193814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:58.751536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'66","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251742+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:27:49.251742+0000","last_peered":"2024-01-16T15:27:49.251742+0000","last_clean":"2024-01-16T15:27:49.251742+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:27:49.251742+0000","last_undegraded":"2024-01-16T15:27:49.251742+0000","last_fullsized":"2024-01-16T15:27:49.251742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:32.314064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.573325+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:23:55.573325+0000","last_peered":"2024-01-16T15:23:55.573325+0000","last_clean":"2024-01-16T15:23:55.573325+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:23:55.573325+0000","last_undegraded":"2024-01-16T15:23:55.573325+0000","last_fullsized":"2024-01-16T15:23:55.573325+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:00.825374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'62","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248439+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:27:49.248439+0000","last_peered":"2024-01-16T15:27:49.248439+0000","last_clean":"2024-01-16T15:27:49.248439+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:27:49.248439+0000","last_undegraded":"2024-01-16T15:27:49.248439+0000","last_fullsized":"2024-01-16T15:27:49.248439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:32.038856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'3","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.567924+0000","last_change":"2024-01-16T15:19:23.621285+0000","last_active":"2024-01-16T15:23:55.567924+0000","last_peered":"2024-01-16T15:23:55.567924+0000","last_clean":"2024-01-16T15:23:55.567924+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:23:55.567924+0000","last_undegraded":"2024-01-16T15:23:55.567924+0000","last_fullsized":"2024-01-16T15:23:55.567924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:44.070274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248670+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:27:49.248670+0000","last_peered":"2024-01-16T15:27:49.248670+0000","last_clean":"2024-01-16T15:27:49.248670+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:27:49.248670+0000","last_undegraded":"2024-01-16T15:27:49.248670+0000","last_fullsized":"2024-01-16T15:27:49.248670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:26.904243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.558296+0000","last_change":"2024-01-16T15:19:23.619619+0000","last_active":"2024-01-16T15:23:45.558296+0000","last_peered":"2024-01-16T15:23:45.558296+0000","last_clean":"2024-01-16T15:23:45.558296+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:23:45.558296+0000","last_undegraded":"2024-01-16T15:23:45.558296+0000","last_fullsized":"2024-01-16T15:23:45.558296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:15.166720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'66","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249594+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:27:49.249594+0000","last_peered":"2024-01-16T15:27:49.249594+0000","last_clean":"2024-01-16T15:27:49.249594+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:27:49.249594+0000","last_undegraded":"2024-01-16T15:27:49.249594+0000","last_fullsized":"2024-01-16T15:27:49.249594+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:58.872650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.565824+0000","last_change":"2024-01-16T15:19:23.620635+0000","last_active":"2024-01-16T15:23:45.565824+0000","last_peered":"2024-01-16T15:23:45.565824+0000","last_clean":"2024-01-16T15:23:45.565824+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:23:45.565824+0000","last_undegraded":"2024-01-16T15:23:45.565824+0000","last_fullsized":"2024-01-16T15:23:45.565824+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'63","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252548+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:27:49.252548+0000","last_peered":"2024-01-16T15:27:49.252548+0000","last_clean":"2024-01-16T15:27:49.252548+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:27:49.252548+0000","last_undegraded":"2024-01-16T15:27:49.252548+0000","last_fullsized":"2024-01-16T15:27:49.252548+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.580304+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:23:45.580304+0000","last_peered":"2024-01-16T15:23:45.580304+0000","last_clean":"2024-01-16T15:23:45.580304+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:23:45.580304+0000","last_undegraded":"2024-01-16T15:23:45.580304+0000","last_fullsized":"2024-01-16T15:23:45.580304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:48.621454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'70","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236778+0000","last_change":"2024-01-16T15:19:42.592735+0000","last_active":"2024-01-16T15:27:49.236778+0000","last_peered":"2024-01-16T15:27:49.236778+0000","last_clean":"2024-01-16T15:27:49.236778+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:27:49.236778+0000","last_undegraded":"2024-01-16T15:27:49.236778+0000","last_fullsized":"2024-01-16T15:27:49.236778+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:07.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.314110+0000","last_change":"2024-01-16T15:19:23.630170+0000","last_active":"2024-01-16T15:23:45.314110+0000","last_peered":"2024-01-16T15:23:45.314110+0000","last_clean":"2024-01-16T15:23:45.314110+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:23:45.314110+0000","last_undegraded":"2024-01-16T15:23:45.314110+0000","last_fullsized":"2024-01-16T15:23:45.314110+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:06.035275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256006+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:27:49.256006+0000","last_peered":"2024-01-16T15:27:49.256006+0000","last_clean":"2024-01-16T15:27:49.256006+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:27:49.256006+0000","last_undegraded":"2024-01-16T15:27:49.256006+0000","last_fullsized":"2024-01-16T15:27:49.256006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:58.148829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":48,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.257180+0000","last_change":"2024-01-16T15:19:23.623778+0000","last_active":"2024-01-16T15:27:44.257180+0000","last_peered":"2024-01-16T15:27:44.257180+0000","last_clean":"2024-01-16T15:27:44.257180+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:27:44.257180+0000","last_undegraded":"2024-01-16T15:27:44.257180+0000","last_fullsized":"2024-01-16T15:27:44.257180+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:53.783590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'69","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244950+0000","last_change":"2024-01-16T15:19:24.628361+0000","last_active":"2024-01-16T15:27:49.244950+0000","last_peered":"2024-01-16T15:27:49.244950+0000","last_clean":"2024-01-16T15:27:49.244950+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:27:49.244950+0000","last_undegraded":"2024-01-16T15:27:49.244950+0000","last_fullsized":"2024-01-16T15:27:49.244950+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:48.469423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.606971+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:25:40.606971+0000","last_peered":"2024-01-16T15:25:40.606971+0000","last_clean":"2024-01-16T15:25:40.606971+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:25:40.606971+0000","last_undegraded":"2024-01-16T15:25:40.606971+0000","last_fullsized":"2024-01-16T15:25:40.606971+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:40.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'53","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236803+0000","last_change":"2024-01-16T15:19:24.623591+0000","last_active":"2024-01-16T15:27:49.236803+0000","last_peered":"2024-01-16T15:27:49.236803+0000","last_clean":"2024-01-16T15:27:49.236803+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:27:49.236803+0000","last_undegraded":"2024-01-16T15:27:49.236803+0000","last_fullsized":"2024-01-16T15:27:49.236803+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:23:57.706980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.336436+0000","last_change":"2024-01-16T15:19:23.612616+0000","last_active":"2024-01-16T15:25:40.336436+0000","last_peered":"2024-01-16T15:25:40.336436+0000","last_clean":"2024-01-16T15:25:40.336436+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:25:40.336436+0000","last_undegraded":"2024-01-16T15:25:40.336436+0000","last_fullsized":"2024-01-16T15:25:40.336436+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:19.160736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'65","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242630+0000","last_change":"2024-01-16T15:19:24.626321+0000","last_active":"2024-01-16T15:27:49.242630+0000","last_peered":"2024-01-16T15:27:49.242630+0000","last_clean":"2024-01-16T15:27:49.242630+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:27:49.242630+0000","last_undegraded":"2024-01-16T15:27:49.242630+0000","last_fullsized":"2024-01-16T15:27:49.242630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.591466+0000","last_change":"2024-01-16T15:19:23.624071+0000","last_active":"2024-01-16T15:25:40.591466+0000","last_peered":"2024-01-16T15:25:40.591466+0000","last_clean":"2024-01-16T15:25:40.591466+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:25:40.591466+0000","last_undegraded":"2024-01-16T15:25:40.591466+0000","last_fullsized":"2024-01-16T15:25:40.591466+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:45.204823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'64","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236252+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:27:49.236252+0000","last_peered":"2024-01-16T15:27:49.236252+0000","last_clean":"2024-01-16T15:27:49.236252+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:27:49.236252+0000","last_undegraded":"2024-01-16T15:27:49.236252+0000","last_fullsized":"2024-01-16T15:27:49.236252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:41.760446+0000","last_change":"2024-01-16T15:19:23.619253+0000","last_active":"2024-01-16T15:25:41.760446+0000","last_peered":"2024-01-16T15:25:41.760446+0000","last_clean":"2024-01-16T15:25:41.760446+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:25:41.760446+0000","last_undegraded":"2024-01-16T15:25:41.760446+0000","last_fullsized":"2024-01-16T15:25:41.760446+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:16.781683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'73","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243648+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:27:49.243648+0000","last_peered":"2024-01-16T15:27:49.243648+0000","last_clean":"2024-01-16T15:27:49.243648+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:27:49.243648+0000","last_undegraded":"2024-01-16T15:27:49.243648+0000","last_fullsized":"2024-01-16T15:27:49.243648+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:49.637451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'31","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:43.237902+0000","last_change":"2024-01-16T15:19:23.629871+0000","last_active":"2024-01-16T15:25:43.237902+0000","last_peered":"2024-01-16T15:25:43.237902+0000","last_clean":"2024-01-16T15:25:43.237902+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:25:43.237902+0000","last_undegraded":"2024-01-16T15:25:43.237902+0000","last_fullsized":"2024-01-16T15:25:43.237902+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:05:17.872229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251399+0000","last_change":"2024-01-16T15:19:24.621286+0000","last_active":"2024-01-16T15:27:49.251399+0000","last_peered":"2024-01-16T15:27:49.251399+0000","last_clean":"2024-01-16T15:27:49.251399+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:27:49.251399+0000","last_undegraded":"2024-01-16T15:27:49.251399+0000","last_fullsized":"2024-01-16T15:27:49.251399+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:57.053368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.603477+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:25:25.603477+0000","last_peered":"2024-01-16T15:25:25.603477+0000","last_clean":"2024-01-16T15:25:25.603477+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:25:25.603477+0000","last_undegraded":"2024-01-16T15:25:25.603477+0000","last_fullsized":"2024-01-16T15:25:25.603477+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:01.099875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'54","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249675+0000","last_change":"2024-01-16T15:19:24.621294+0000","last_active":"2024-01-16T15:27:49.249675+0000","last_peered":"2024-01-16T15:27:49.249675+0000","last_clean":"2024-01-16T15:27:49.249675+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:27:49.249675+0000","last_undegraded":"2024-01-16T15:27:49.249675+0000","last_fullsized":"2024-01-16T15:27:49.249675+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.575031+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:24:05.575031+0000","last_peered":"2024-01-16T15:24:05.575031+0000","last_clean":"2024-01-16T15:24:05.575031+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:24:05.575031+0000","last_undegraded":"2024-01-16T15:24:05.575031+0000","last_fullsized":"2024-01-16T15:24:05.575031+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:59.676130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":261,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:50.938466+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:25:50.938466+0000","last_peered":"2024-01-16T15:25:50.938466+0000","last_clean":"2024-01-16T15:25:50.938466+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:25:50.938466+0000","last_undegraded":"2024-01-16T15:25:50.938466+0000","last_fullsized":"2024-01-16T15:25:50.938466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4358,"ondisk_log_size":4358,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509780,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.1559999999999999,"15min":1.25},"min":{"1min":0.69899999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5409999999999999,"5min":1.7749999999999999,"15min":19.224},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.212,"15min":1.3140000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.615,"5min":1.694,"15min":28.603999999999999},"last":1.4990000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.22,"15min":1.2250000000000001},"min":{"1min":0.90500000000000003,"5min":0.80800000000000005,"15min":0.374},"max":{"1min":1.488,"5min":1.837,"15min":3.7069999999999999},"last":1.641},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.3080000000000001,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.68899999999999995,"15min":0.68400000000000005},"max":{"1min":1.651,"5min":1.8819999999999999,"15min":33.225000000000001},"last":1.4019999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3380000000000001,"15min":1.4199999999999999},"min":{"1min":1.0529999999999999,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.752,"5min":1.7909999999999999,"15min":19.126000000000001},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3080000000000001,"15min":1.3120000000000001},"min":{"1min":1.008,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.694,"5min":1.718,"15min":2.9750000000000001},"last":1.3129999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.76500000000000001},"max":{"1min":1.526,"5min":1.73,"15min":3.7999999999999998},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.369,"5min":1.446,"15min":1.5580000000000001},"min":{"1min":0.97399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":19.268999999999998},"last":1.528}]},{"osd":4,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2290000000000001,"15min":1.264},"min":{"1min":0.88300000000000001,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.347,"15min":1.3919999999999999},"min":{"1min":1.1180000000000001,"5min":0.70299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6659999999999999,"5min":1.7330000000000001,"15min":9.7059999999999995},"last":1.0649999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":1.0389999999999999,"5min":0.86099999999999999,"15min":0.79400000000000004},"max":{"1min":1.74,"5min":1.9450000000000001,"15min":3.9129999999999998},"last":1.5620000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.5620000000000001},"min":{"1min":0.82699999999999996,"5min":0.77900000000000003,"15min":0.55100000000000005},"max":{"1min":1.8720000000000001,"5min":1.9630000000000001,"15min":19.036999999999999},"last":1.595}]},{"osd":7,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.4359999999999999,"15min":1.4410000000000001},"min":{"1min":1.198,"5min":0.74299999999999999,"15min":0.54800000000000004},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":4.2240000000000002},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.528,"15min":1.571},"min":{"1min":1.224,"5min":0.98999999999999999,"15min":0.65100000000000002},"max":{"1min":1.728,"5min":2.181,"15min":4.0709999999999997},"last":1.464}]}]},{"osd":4,"up_from":15,"seq":64424509782,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.296,"15min":1.2330000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.427,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.534},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1799999999999999,"15min":1.1859999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.0509999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.88700000000000001,"5min":0.73899999999999999,"15min":0.71899999999999997},"max":{"1min":1.6160000000000001,"5min":1.893,"15min":2.0510000000000002},"last":1.663},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.3149999999999999,"15min":1.302},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.6319999999999999,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.1499999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3420000000000001,"15min":1.3320000000000001},"min":{"1min":0.79300000000000004,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.573,"5min":1.78,"15min":5.907},"last":1.226},{"interface":"front","average":{"1min":1.242,"5min":1.381,"15min":1.355},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.52600000000000002},"max":{"1min":1.6000000000000001,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":1.4330000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4570000000000001,"15min":1.3939999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.76200000000000001},"max":{"1min":1.5289999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.4019999999999999,"15min":1.391},"min":{"1min":1.052,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.74,"5min":1.9670000000000001,"15min":2.0379999999999998},"last":1.3859999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.425,"15min":1.448},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6779999999999999,"5min":6.165,"15min":6.165},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4410000000000001,"15min":1.4770000000000001},"min":{"1min":0.95299999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8180000000000001,"5min":3.1469999999999998,"15min":5.3520000000000003},"last":1.4890000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.391,"15min":1.4219999999999999},"min":{"1min":1.101,"5min":0.90800000000000003,"15min":0.497},"max":{"1min":1.903,"5min":1.903,"15min":8.9600000000000009},"last":1.575},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5640000000000001,"15min":1.528},"min":{"1min":1.165,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.772,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.3300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.452,"15min":1.492},"min":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.89700000000000002},"max":{"1min":1.8500000000000001,"5min":3.3620000000000001,"15min":5.835},"last":1.369},{"interface":"front","average":{"1min":1.554,"5min":1.5469999999999999,"15min":1.581},"min":{"1min":1.238,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":2.0070000000000001,"5min":2.3679999999999999,"15min":8.0079999999999991},"last":1.724}]}]},{"osd":7,"up_from":14,"seq":60129542484,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1891948,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92479892,"statfs":{"total":96636764160,"available":94699409408,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1659999999999999,"15min":1.2130000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.65400000000000003},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.1779999999999999},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.24,"5min":1.2210000000000001,"15min":1.248},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":12.461},"last":1.042}]},{"osd":1,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.89000000000000001,"5min":0.85099999999999998,"15min":0.48599999999999999},"max":{"1min":1.923,"5min":1.923,"15min":2.4249999999999998},"last":1.357},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2709999999999999,"15min":1.363},"min":{"1min":0.93400000000000005,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":1.8140000000000001,"5min":1.9079999999999999,"15min":14.848000000000001},"last":1.1599999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3069999999999999,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":3.726},"last":1.099},{"interface":"front","average":{"1min":1.286,"5min":1.3340000000000001,"15min":1.3959999999999999},"min":{"1min":0.88900000000000001,"5min":0.873,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.792,"15min":19.763999999999999},"last":1.5269999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3340000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80600000000000005},"max":{"1min":1.6539999999999999,"5min":1.8280000000000001,"15min":1.887},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.47,"15min":1.4650000000000001},"min":{"1min":1.113,"5min":1.0169999999999999,"15min":0.629},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":6.6029999999999998},"last":1.4419999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.345,"15min":1.347},"min":{"1min":0.98499999999999999,"5min":0.89100000000000001,"15min":0.54100000000000004},"max":{"1min":1.7090000000000001,"5min":1.7929999999999999,"15min":4.1390000000000002},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.401,"15min":1.413},"min":{"1min":1.0009999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":1.873,"5min":1.877,"15min":2.5190000000000001},"last":1.5920000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3340000000000001,"15min":1.3680000000000001},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":12.609999999999999},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5229999999999999},"min":{"1min":1.1719999999999999,"5min":1.046,"15min":0.66000000000000003},"max":{"1min":1.8879999999999999,"5min":1.909,"15min":3.4159999999999999},"last":1.486}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.099,"5min":1.0760000000000001,"15min":0.66000000000000003},"max":{"1min":2.0859999999999999,"5min":2.1429999999999998,"15min":2.653},"last":1.641},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.552},"min":{"1min":1.004,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":12.567},"last":1.7}]}]},{"osd":5,"up_from":14,"seq":60129542486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.208},"min":{"1min":0.83399999999999996,"5min":0.749,"15min":0.32900000000000001},"max":{"1min":1.7190000000000001,"5min":2.0939999999999999,"15min":6.8380000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1870000000000001,"15min":1.3200000000000001},"min":{"1min":1.0049999999999999,"5min":0.82999999999999996,"15min":0.57099999999999995},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":23.199999999999999},"last":1.2290000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.246},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.63400000000000001},"max":{"1min":1.637,"5min":1.698,"15min":8.0909999999999993},"last":0.996},{"interface":"front","average":{"1min":1.355,"5min":1.26,"15min":1.323},"min":{"1min":0.93600000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8049999999999999,"5min":1.851,"15min":17.062000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.409,"15min":1.415},"min":{"1min":1.069,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.6970000000000001,"5min":1.782,"15min":5.8810000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":1.131,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.6899999999999999,"5min":1.6970000000000001,"15min":5.4429999999999996},"last":1.022}]},{"osd":3,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.302,"15min":1.3420000000000001},"min":{"1min":0.94999999999999996,"5min":0.878,"15min":0.67500000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":7.984},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.385,"5min":1.4179999999999999,"15min":1.423},"min":{"1min":0.94599999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.7410000000000001,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":0.69099999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.323,"15min":1.387},"min":{"1min":0.878,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":15.975},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4079999999999999,"15min":1.413},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.54000000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":5.6349999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.329,"15min":1.3680000000000001},"min":{"1min":0.95999999999999996,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.8160000000000001,"5min":1.827,"15min":12.914},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.468,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":16.552},"last":1.165}]},{"osd":7,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4630000000000001,"15min":1.506},"min":{"1min":1.298,"5min":0.98099999999999998,"15min":0.64300000000000002},"max":{"1min":1.74,"5min":1.8540000000000001,"15min":13.034000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5740000000000001,"15min":1.5900000000000001},"min":{"1min":1.161,"5min":1.0429999999999999,"15min":0.70899999999999996},"max":{"1min":1.9159999999999999,"5min":2.073,"15min":8.0380000000000003},"last":0.95799999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542486,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":5.7400000000000002},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.1359999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40600000000000003},"max":{"1min":1.506,"5min":1.879,"15min":7.1429999999999998},"last":0.91100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.96899999999999997,"15min":1.0349999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.365,"5min":1.738,"15min":10.856999999999999},"last":1.224},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.1870000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4950000000000001,"5min":1.823,"15min":2.0089999999999999},"last":1.052}]},{"osd":2,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.3140000000000001,"15min":1.5209999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.946,"5min":1.946,"15min":39.265999999999998},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1559999999999999,"15min":1.2070000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.381,"5min":1.833,"15min":7.9160000000000004},"last":1.097}]},{"osd":4,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.337,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.70999999999999996},"max":{"1min":1.5980000000000001,"5min":1.9419999999999999,"15min":5.6890000000000001},"last":1.01},{"interface":"front","average":{"1min":1.329,"5min":1.464,"15min":1.4810000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8120000000000001,"5min":2.0920000000000001,"15min":2.3330000000000002},"last":1.889}]},{"osd":5,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3440000000000001,"15min":1.4670000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5669999999999999,"5min":1.827,"15min":14.003},"last":2.1339999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.5049999999999999,"15min":1.577},"min":{"1min":0.877,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":1.9359999999999999,"5min":2.008,"15min":6.0679999999999996},"last":1.8160000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.464,"15min":1.502},"min":{"1min":0.875,"5min":0.875,"15min":0.69799999999999995},"max":{"1min":1.8899999999999999,"5min":2.0139999999999998,"15min":7.0060000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.486,"15min":1.5580000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8140000000000001,"5min":1.982,"15min":15.114000000000001},"last":1.966}]},{"osd":7,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4950000000000001,"15min":1.6160000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80400000000000005},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":12.378},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5780000000000001,"15min":1.6220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9650000000000001,"5min":2.23,"15min":4.6029999999999998},"last":2.2589999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542485,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1059928,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93311912,"statfs":{"total":96636764160,"available":95551397888,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.90500000000000003,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.41199999999999998,"15min":0.32400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":1.833,"5min":1.913,"15min":7.9960000000000004},"last":0.97499999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0880000000000001,"15min":1.1040000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":1.7130000000000001,"5min":2.294,"15min":3.056},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.171,"5min":1.143,"15min":1.175},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.556,"5min":1.877,"15min":4.3819999999999997},"last":1.3819999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.206,"15min":1.2789999999999999},"min":{"1min":0.61199999999999999,"5min":0.55300000000000005,"15min":0.44700000000000001},"max":{"1min":1.704,"5min":1.883,"15min":18.838000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4950000000000001,"15min":1.552},"min":{"1min":0.66200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":2.0099999999999998,"15min":19.375},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.335,"15min":1.4390000000000001},"min":{"1min":0.80600000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.8240000000000001,"5min":2.242,"15min":23.225000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4730000000000001,"15min":1.494},"min":{"1min":1.038,"5min":0.75900000000000001,"15min":0.72699999999999998},"max":{"1min":1.883,"5min":1.883,"15min":6.359},"last":1.6779999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4399999999999999,"15min":1.5009999999999999},"min":{"1min":1.0800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.917,"5min":2.3170000000000002,"15min":20.157},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.597,"15min":1.6100000000000001},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.623},"max":{"1min":1.9330000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.0449999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5289999999999999,"15min":1.595},"min":{"1min":1.018,"5min":0.89800000000000002,"15min":0.70499999999999996},"max":{"1min":1.7330000000000001,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.101},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6160000000000001,"15min":1.7170000000000001},"min":{"1min":0.85799999999999998,"5min":0.79300000000000004,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.1419999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5429999999999999,"15min":1.6699999999999999},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.254},{"interface":"front","average":{"1min":1.587,"5min":1.569,"15min":1.5880000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":1.8999999999999999,"5min":2.0990000000000002,"15min":4.3419999999999996},"last":1.7569999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542487,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.093,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.28100000000000003},"max":{"1min":13.42,"5min":13.42,"15min":13.42},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.216,"5min":1.141,"15min":1.113},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":1.4970000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.03,"15min":1.0900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.649,"15min":3.6819999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.29,"15min":1.3320000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.39000000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":3.5600000000000001},"last":1.4610000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.26,"15min":1.2889999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46300000000000002},"max":{"1min":1.488,"5min":1.875,"15min":2.238},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.137,"15min":1.194},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":0.94299999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.72199999999999998},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.077}]},{"osd":5,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3939999999999999,"15min":1.431},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63700000000000001},"max":{"1min":1.6359999999999999,"5min":1.952,"15min":3.98},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4530000000000001,"15min":1.462},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.53300000000000003},"max":{"1min":1.669,"5min":1.893,"15min":3.3940000000000001},"last":1.385}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.56999999999999995},"max":{"1min":1.8660000000000001,"5min":2.016,"15min":16.48},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.508,"15min":1.532},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":0.65600000000000003},"max":{"1min":1.74,"5min":1.909,"15min":3.8180000000000001},"last":1.526}]},{"osd":7,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.573,"15min":1.617},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.48099999999999998},"max":{"1min":1.8480000000000001,"5min":2.0009999999999999,"15min":7.0499999999999998},"last":1.637},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.6339999999999999,"15min":1.647},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.81100000000000005},"max":{"1min":1.8640000000000001,"5min":2.3940000000000001,"15min":3.7839999999999998},"last":1.5700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542485,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":1.038},"min":{"1min":0.496,"5min":0.496,"15min":0.35199999999999998},"max":{"1min":1.427,"5min":1.665,"15min":15.776},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.151,"15min":1.171},"min":{"1min":0.67600000000000005,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":1.9019999999999999,"5min":2.7549999999999999,"15min":17.620999999999999},"last":1.119}]},{"osd":2,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.137,"15min":1.22},"min":{"1min":0.74099999999999999,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.671,"5min":1.788,"15min":15.997999999999999},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.23,"15min":1.1919999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":1.669,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.2490000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.129,"15min":1.179},"min":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.4279999999999999},"min":{"1min":0.86299999999999999,"5min":0.66600000000000004,"15min":0.47399999999999998},"max":{"1min":1.74,"5min":1.827,"15min":16.126000000000001},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.466,"15min":1.573},"min":{"1min":1.095,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.9470000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.675},{"interface":"front","average":{"1min":1.446,"5min":1.4430000000000001,"15min":1.419},"min":{"1min":1.0089999999999999,"5min":0.82299999999999995,"15min":0.71899999999999997},"max":{"1min":1.913,"5min":3.0099999999999998,"15min":3.0459999999999998},"last":1.395}]},{"osd":5,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.97099999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7869999999999999,"5min":2.157,"15min":5.5609999999999999},"last":1.351},{"interface":"front","average":{"1min":1.581,"5min":1.5529999999999999,"15min":1.639},"min":{"1min":1.2130000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0630000000000002,"5min":2.1989999999999998,"15min":16.501999999999999},"last":1.421}]},{"osd":6,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.137,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.891,"5min":2.0790000000000002,"15min":5.4509999999999996},"last":1.542},{"interface":"front","average":{"1min":1.649,"5min":1.6379999999999999,"15min":1.7190000000000001},"min":{"1min":0.98899999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":2.1269999999999998,"5min":2.351,"15min":16.395},"last":1.766}]},{"osd":7,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5800000000000001,"15min":1.595},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":2.3300000000000001,"5min":2.9510000000000001,"15min":9.6980000000000004},"last":1.484},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.589,"15min":1.5860000000000001},"min":{"1min":1.0860000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.954,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.633}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:02.821 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-01-16T15:28:02.821 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:03.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.082+0000 7f3aed800640 1 Processor -- start 2024-01-16T15:28:03.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.082+0000 7f3aed800640 1 -- start start 2024-01-16T15:28:03.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3aed800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae815e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3aed800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f3ae80b3ae0 0x7f3ae806cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3aed800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ae806d120 0x7f3ae80cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3aed800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae8071450 con 0x7f3ae806d120 2024-01-16T15:28:03.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3aed800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ae8070c40 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3aed800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae806c470 con 0x7f3ae80ae5a0 2024-01-16T15:28:03.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3ae7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ae806d120 0x7f3ae80cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3ae7400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f3ae806d120 0x7f3ae80cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55520/0 (socket says 172.21.15.53:55520) 2024-01-16T15:28:03.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.084+0000 7f3ae7400640 1 -- 172.21.15.53:0/3519235169 learned_addr learned my addr 172.21.15.53:0/3519235169 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3adfe00640 1 --2- 172.21.15.53:0/3519235169 >> v2:172.21.15.146:3300/0 conn(0x7f3ae80b3ae0 0x7f3ae806cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3ae6a00640 1 --2- 172.21.15.53:0/3519235169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae815e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3ae7400640 1 -- 172.21.15.53:0/3519235169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 msgr2=0x7f3ae815e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3ae7400640 1 --2- 172.21.15.53:0/3519235169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae815e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3ae7400640 1 -- 172.21.15.53:0/3519235169 >> v2:172.21.15.146:3300/0 conn(0x7f3ae80b3ae0 msgr2=0x7f3ae806cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3ae7400640 1 --2- 172.21.15.53:0/3519235169 >> v2:172.21.15.146:3300/0 conn(0x7f3ae80b3ae0 0x7f3ae806cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.085+0000 7f3ae7400640 1 -- 172.21.15.53:0/3519235169 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae806c750 con 0x7f3ae806d120 2024-01-16T15:28:03.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.087+0000 7f3ae7400640 1 --2- 172.21.15.53:0/3519235169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae806d120 0x7f3ae80cf290 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80cfed0 tx=0x7f3ae002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b85259776e0291 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:03.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.087+0000 7f3ae6000640 1 -- 172.21.15.53:0/3519235169 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae002ca70 con 0x7f3ae806d120 2024-01-16T15:28:03.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.087+0000 7f3ae6000640 1 -- 172.21.15.53:0/3519235169 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae002cbd0 con 0x7f3ae806d120 2024-01-16T15:28:03.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.087+0000 7f3ae6000640 1 -- 172.21.15.53:0/3519235169 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae00343f0 con 0x7f3ae806d120 2024-01-16T15:28:03.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 -- 172.21.15.53:0/3519235169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae806d120 msgr2=0x7f3ae80cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 --2- 172.21.15.53:0/3519235169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae806d120 0x7f3ae80cf290 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7f3ae80cfed0 tx=0x7f3ae002bed0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 -- 172.21.15.53:0/3519235169 shutdown_connections 2024-01-16T15:28:03.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 --2- 172.21.15.53:0/3519235169 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae806d120 0x7f3ae80cf290 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 --2- 172.21.15.53:0/3519235169 >> v2:172.21.15.146:3300/0 conn(0x7f3ae80b3ae0 0x7f3ae806cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 --2- 172.21.15.53:0/3519235169 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae815e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 -- 172.21.15.53:0/3519235169 >> 172.21.15.53:0/3519235169 conn(0x7f3ae806a800 msgr2=0x7f3ae806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:03.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 -- 172.21.15.53:0/3519235169 shutdown_connections 2024-01-16T15:28:03.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.088+0000 7f3aed800640 1 -- 172.21.15.53:0/3519235169 wait complete. 2024-01-16T15:28:03.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.090+0000 7f3aed800640 1 Processor -- start 2024-01-16T15:28:03.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.090+0000 7f3aed800640 1 -- start start 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3aed800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae806d120 0x7f3ae80c47e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3aed800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae81f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3aed800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 0x7f3ae81f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3aed800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80c45b0 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3aed800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80c3ff0 con 0x7f3ae806d120 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3aed800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae80c42d0 con 0x7f3ae80ae5a0 2024-01-16T15:28:03.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 0x7f3ae81f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 0x7f3ae81f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55532/0 (socket says 172.21.15.53:55532) 2024-01-16T15:28:03.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 -- 172.21.15.53:0/517815362 learned_addr learned my addr 172.21.15.53:0/517815362 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:03.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 -- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 msgr2=0x7f3ae81f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae81f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 -- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae806d120 msgr2=0x7f3ae80c47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae806d120 0x7f3ae80c47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.091+0000 7f3ae7400640 1 -- 172.21.15.53:0/517815362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae0009b30 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.092+0000 7f3ae7400640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 0x7f3ae81f26e0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009f40 tx=0x7f3ae0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:03.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.092+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0009510 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.092+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae003e450 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.092+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae003e7e0 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.092+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae80cc3e0 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.092+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae8159100 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.093+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae80cc900 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.094+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f3ae003e940 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.095+0000 7f3adea00640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac006c550 0x7f3ac006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.095+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f3ae00ab610 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.096+0000 7f3ae6a00640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac006c550 0x7f3ac006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.097+0000 7f3ae6a00640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac006c550 0x7f3ac006ea10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009870 tx=0x7f3ad8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:03.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.100+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f3ae00e1e90 con 0x7f3ae80b3ae0 2024-01-16T15:28:03.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.289+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ae8070550 con 0x7f3ac006c550 2024-01-16T15:28:03.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.312+0000 7f3adea00640 1 -- 172.21.15.53:0/517815362 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288957 (secure 0 0 0) 0x7f3ad8009df0 con 0x7f3ac006c550 2024-01-16T15:28:03.314 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:03.317 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:28:03.318 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac006c550 msgr2=0x7f3ac006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac006c550 0x7f3ac006ea10 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8009870 tx=0x7f3ad8008040 comp rx=0 tx=0).stop 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 msgr2=0x7f3ae81f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 0x7f3ae81f26e0 secure :-1 s=READY pgs=2022 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009f40 tx=0x7f3ae0004a40 comp rx=0 tx=0).stop 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 shutdown_connections 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f3ac006c550 0x7f3ac006ea10 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f3ae80b3ae0 0x7f3ae81f26e0 unknown :-1 s=CLOSED pgs=2022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f3ae80ae5a0 0x7f3ae81f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.319 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 --2- 172.21.15.53:0/517815362 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f3ae806d120 0x7f3ae80c47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 >> 172.21.15.53:0/517815362 conn(0x7f3ae806a800 msgr2=0x7f3ae806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:03.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 shutdown_connections 2024-01-16T15:28:03.320 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.319+0000 7f3aed800640 1 -- 172.21.15.53:0/517815362 wait complete. 2024-01-16T15:28:03.339 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":954,"stamp":"2024-01-16T15:28:02.907428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4907,"ondisk_log_size":4907,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11125304,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":625011,"kb_avail":743849416,"statfs":{"total":773094113280,"available":761701801984,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":640011824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3083,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005140"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256079+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.256079+0000","last_peered":"2024-01-16T15:27:49.256079+0000","last_clean":"2024-01-16T15:27:49.256079+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.256079+0000","last_undegraded":"2024-01-16T15:27:49.256079+0000","last_fullsized":"2024-01-16T15:27:49.256079+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:08:35.875409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'73","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237782+0000","last_change":"2024-01-16T15:19:24.622288+0000","last_active":"2024-01-16T15:27:49.237782+0000","last_peered":"2024-01-16T15:27:49.237782+0000","last_clean":"2024-01-16T15:27:49.237782+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:27:49.237782+0000","last_undegraded":"2024-01-16T15:27:49.237782+0000","last_fullsized":"2024-01-16T15:27:49.237782+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:20.176733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251487+0000","last_change":"2024-01-16T15:19:24.618862+0000","last_active":"2024-01-16T15:27:49.251487+0000","last_peered":"2024-01-16T15:27:49.251487+0000","last_clean":"2024-01-16T15:27:49.251487+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:27:49.251487+0000","last_undegraded":"2024-01-16T15:27:49.251487+0000","last_fullsized":"2024-01-16T15:27:49.251487+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:02.115872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'56","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.231727+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:27:49.231727+0000","last_peered":"2024-01-16T15:27:49.231727+0000","last_clean":"2024-01-16T15:27:49.231727+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:27:49.231727+0000","last_undegraded":"2024-01-16T15:27:49.231727+0000","last_fullsized":"2024-01-16T15:27:49.231727+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:46:32.713991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'62","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.232748+0000","last_change":"2024-01-16T15:19:24.626093+0000","last_active":"2024-01-16T15:27:49.232748+0000","last_peered":"2024-01-16T15:27:49.232748+0000","last_clean":"2024-01-16T15:27:49.232748+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:27:49.232748+0000","last_undegraded":"2024-01-16T15:27:49.232748+0000","last_fullsized":"2024-01-16T15:27:49.232748+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:45.086271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'76","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239918+0000","last_change":"2024-01-16T15:19:24.625809+0000","last_active":"2024-01-16T15:27:49.239918+0000","last_peered":"2024-01-16T15:27:49.239918+0000","last_clean":"2024-01-16T15:27:49.239918+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:27:49.239918+0000","last_undegraded":"2024-01-16T15:27:49.239918+0000","last_fullsized":"2024-01-16T15:27:49.239918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:02:21.453533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'67","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.241483+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:27:49.241483+0000","last_peered":"2024-01-16T15:27:49.241483+0000","last_clean":"2024-01-16T15:27:49.241483+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:27:49.241483+0000","last_undegraded":"2024-01-16T15:27:49.241483+0000","last_fullsized":"2024-01-16T15:27:49.241483+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:15:57.152091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'58","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248998+0000","last_change":"2024-01-16T15:19:24.616757+0000","last_active":"2024-01-16T15:27:49.248998+0000","last_peered":"2024-01-16T15:27:49.248998+0000","last_clean":"2024-01-16T15:27:49.248998+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:27:49.248998+0000","last_undegraded":"2024-01-16T15:27:49.248998+0000","last_fullsized":"2024-01-16T15:27:49.248998+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:30.687663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.255855+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:27:49.255855+0000","last_peered":"2024-01-16T15:27:49.255855+0000","last_clean":"2024-01-16T15:27:49.255855+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:27:49.255855+0000","last_undegraded":"2024-01-16T15:27:49.255855+0000","last_fullsized":"2024-01-16T15:27:49.255855+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:52:07.972777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257125+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:27:49.257125+0000","last_peered":"2024-01-16T15:27:49.257125+0000","last_clean":"2024-01-16T15:27:49.257125+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:27:49.257125+0000","last_undegraded":"2024-01-16T15:27:49.257125+0000","last_fullsized":"2024-01-16T15:27:49.257125+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:08:41.949199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'70","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.227737+0000","last_change":"2024-01-16T15:19:24.625137+0000","last_active":"2024-01-16T15:27:49.227737+0000","last_peered":"2024-01-16T15:27:49.227737+0000","last_clean":"2024-01-16T15:27:49.227737+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:27:49.227737+0000","last_undegraded":"2024-01-16T15:27:49.227737+0000","last_fullsized":"2024-01-16T15:27:49.227737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'64","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236182+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:27:49.236182+0000","last_peered":"2024-01-16T15:27:49.236182+0000","last_clean":"2024-01-16T15:27:49.236182+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:27:49.236182+0000","last_undegraded":"2024-01-16T15:27:49.236182+0000","last_fullsized":"2024-01-16T15:27:49.236182+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:45:27.512415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'65","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242745+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:27:49.242745+0000","last_peered":"2024-01-16T15:27:49.242745+0000","last_clean":"2024-01-16T15:27:49.242745+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:27:49.242745+0000","last_undegraded":"2024-01-16T15:27:49.242745+0000","last_fullsized":"2024-01-16T15:27:49.242745+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:08.270535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'72","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.245113+0000","last_change":"2024-01-16T15:19:24.624351+0000","last_active":"2024-01-16T15:27:49.245113+0000","last_peered":"2024-01-16T15:27:49.245113+0000","last_clean":"2024-01-16T15:27:49.245113+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:27:49.245113+0000","last_undegraded":"2024-01-16T15:27:49.245113+0000","last_fullsized":"2024-01-16T15:27:49.245113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:41.925967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'59","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246430+0000","last_change":"2024-01-16T15:19:24.613444+0000","last_active":"2024-01-16T15:27:49.246430+0000","last_peered":"2024-01-16T15:27:49.246430+0000","last_clean":"2024-01-16T15:27:49.246430+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:27:49.246430+0000","last_undegraded":"2024-01-16T15:27:49.246430+0000","last_fullsized":"2024-01-16T15:27:49.246430+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:53:59.930379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'74","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248683+0000","last_change":"2024-01-16T15:19:24.614694+0000","last_active":"2024-01-16T15:27:49.248683+0000","last_peered":"2024-01-16T15:27:49.248683+0000","last_clean":"2024-01-16T15:27:49.248683+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:27:49.248683+0000","last_undegraded":"2024-01-16T15:27:49.248683+0000","last_fullsized":"2024-01-16T15:27:49.248683+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:05:59.656591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'52","reported_seq":73,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.228316+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:27:49.228316+0000","last_peered":"2024-01-16T15:27:49.228316+0000","last_clean":"2024-01-16T15:27:49.228316+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:27:49.228316+0000","last_undegraded":"2024-01-16T15:27:49.228316+0000","last_fullsized":"2024-01-16T15:27:49.228316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.235929+0000","last_change":"2024-01-16T15:19:24.629022+0000","last_active":"2024-01-16T15:27:49.235929+0000","last_peered":"2024-01-16T15:27:49.235929+0000","last_clean":"2024-01-16T15:27:49.235929+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:27:49.235929+0000","last_undegraded":"2024-01-16T15:27:49.235929+0000","last_fullsized":"2024-01-16T15:27:49.235929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:22.444157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'74","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250216+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:27:49.250216+0000","last_peered":"2024-01-16T15:27:49.250216+0000","last_clean":"2024-01-16T15:27:49.250216+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:27:49.250216+0000","last_undegraded":"2024-01-16T15:27:49.250216+0000","last_fullsized":"2024-01-16T15:27:49.250216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:11.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247569+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:27:49.247569+0000","last_peered":"2024-01-16T15:27:49.247569+0000","last_clean":"2024-01-16T15:27:49.247569+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:27:49.247569+0000","last_undegraded":"2024-01-16T15:27:49.247569+0000","last_fullsized":"2024-01-16T15:27:49.247569+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'73","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248711+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:27:49.248711+0000","last_peered":"2024-01-16T15:27:49.248711+0000","last_clean":"2024-01-16T15:27:49.248711+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:27:49.248711+0000","last_undegraded":"2024-01-16T15:27:49.248711+0000","last_fullsized":"2024-01-16T15:27:49.248711+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'71","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233752+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:27:49.233752+0000","last_peered":"2024-01-16T15:27:49.233752+0000","last_clean":"2024-01-16T15:27:49.233752+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:27:49.233752+0000","last_undegraded":"2024-01-16T15:27:49.233752+0000","last_fullsized":"2024-01-16T15:27:49.233752+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:03:26.924342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'63","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247674+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:27:49.247674+0000","last_peered":"2024-01-16T15:27:49.247674+0000","last_clean":"2024-01-16T15:27:49.247674+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:27:49.247674+0000","last_undegraded":"2024-01-16T15:27:49.247674+0000","last_fullsized":"2024-01-16T15:27:49.247674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:07.051272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'71","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249786+0000","last_change":"2024-01-16T15:19:24.626920+0000","last_active":"2024-01-16T15:27:49.249786+0000","last_peered":"2024-01-16T15:27:49.249786+0000","last_clean":"2024-01-16T15:27:49.249786+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:27:49.249786+0000","last_undegraded":"2024-01-16T15:27:49.249786+0000","last_fullsized":"2024-01-16T15:27:49.249786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'69","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233273+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:27:49.233273+0000","last_peered":"2024-01-16T15:27:49.233273+0000","last_clean":"2024-01-16T15:27:49.233273+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:27:49.233273+0000","last_undegraded":"2024-01-16T15:27:49.233273+0000","last_fullsized":"2024-01-16T15:27:49.233273+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251042+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:27:49.251042+0000","last_peered":"2024-01-16T15:27:49.251042+0000","last_clean":"2024-01-16T15:27:49.251042+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:27:49.251042+0000","last_undegraded":"2024-01-16T15:27:49.251042+0000","last_fullsized":"2024-01-16T15:27:49.251042+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:40:49.687478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'65","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.226338+0000","last_change":"2024-01-16T15:19:24.623489+0000","last_active":"2024-01-16T15:27:49.226338+0000","last_peered":"2024-01-16T15:27:49.226338+0000","last_clean":"2024-01-16T15:27:49.226338+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:27:49.226338+0000","last_undegraded":"2024-01-16T15:27:49.226338+0000","last_fullsized":"2024-01-16T15:27:49.226338+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:28:11.064893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'50","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.241685+0000","last_change":"2024-01-16T15:19:42.218585+0000","last_active":"2024-01-16T15:27:49.241685+0000","last_peered":"2024-01-16T15:27:49.241685+0000","last_clean":"2024-01-16T15:27:49.241685+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:27:49.241685+0000","last_undegraded":"2024-01-16T15:27:49.241685+0000","last_fullsized":"2024-01-16T15:27:49.241685+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:48:02.749154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'62","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248953+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:27:49.248953+0000","last_peered":"2024-01-16T15:27:49.248953+0000","last_clean":"2024-01-16T15:27:49.248953+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:27:49.248953+0000","last_undegraded":"2024-01-16T15:27:49.248953+0000","last_fullsized":"2024-01-16T15:27:49.248953+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:40.862803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'78","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246500+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:27:49.246500+0000","last_peered":"2024-01-16T15:27:49.246500+0000","last_clean":"2024-01-16T15:27:49.246500+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:27:49.246500+0000","last_undegraded":"2024-01-16T15:27:49.246500+0000","last_fullsized":"2024-01-16T15:27:49.246500+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:46:45.144417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'91","reported_seq":120,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252053+0000","last_change":"2024-01-16T15:19:24.629910+0000","last_active":"2024-01-16T15:27:49.252053+0000","last_peered":"2024-01-16T15:27:49.252053+0000","last_clean":"2024-01-16T15:27:49.252053+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:27:49.252053+0000","last_undegraded":"2024-01-16T15:27:49.252053+0000","last_fullsized":"2024-01-16T15:27:49.252053+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:44.059899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'78","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246839+0000","last_change":"2024-01-16T15:19:24.633567+0000","last_active":"2024-01-16T15:27:49.246839+0000","last_peered":"2024-01-16T15:27:49.246839+0000","last_clean":"2024-01-16T15:27:49.246839+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:27:49.246839+0000","last_undegraded":"2024-01-16T15:27:49.246839+0000","last_fullsized":"2024-01-16T15:27:49.246839+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:36:24.582502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'60","reported_seq":83,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.219133+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:27:49.219133+0000","last_peered":"2024-01-16T15:27:49.219133+0000","last_clean":"2024-01-16T15:27:49.219133+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:27:49.219133+0000","last_undegraded":"2024-01-16T15:27:49.219133+0000","last_fullsized":"2024-01-16T15:27:49.219133+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:54:01.155551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":22,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.591516+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:25:40.591516+0000","last_peered":"2024-01-16T15:25:40.591516+0000","last_clean":"2024-01-16T15:25:40.591516+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:25:40.591516+0000","last_undegraded":"2024-01-16T15:25:40.591516+0000","last_fullsized":"2024-01-16T15:25:40.591516+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:37:40.038677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'67","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247582+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:27:49.247582+0000","last_peered":"2024-01-16T15:27:49.247582+0000","last_clean":"2024-01-16T15:27:49.247582+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:27:49.247582+0000","last_undegraded":"2024-01-16T15:27:49.247582+0000","last_fullsized":"2024-01-16T15:27:49.247582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:25.905229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.933509+0000","last_change":"2024-01-16T15:19:23.621310+0000","last_active":"2024-01-16T15:25:25.933509+0000","last_peered":"2024-01-16T15:25:25.933509+0000","last_clean":"2024-01-16T15:25:25.933509+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:25:25.933509+0000","last_undegraded":"2024-01-16T15:25:25.933509+0000","last_fullsized":"2024-01-16T15:25:25.933509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:19.958286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'66","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246028+0000","last_change":"2024-01-16T15:19:24.630161+0000","last_active":"2024-01-16T15:27:49.246028+0000","last_peered":"2024-01-16T15:27:49.246028+0000","last_clean":"2024-01-16T15:27:49.246028+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:27:49.246028+0000","last_undegraded":"2024-01-16T15:27:49.246028+0000","last_fullsized":"2024-01-16T15:27:49.246028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:54:32.432478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'3","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.565625+0000","last_change":"2024-01-16T15:19:23.630011+0000","last_active":"2024-01-16T15:24:25.565625+0000","last_peered":"2024-01-16T15:24:25.565625+0000","last_clean":"2024-01-16T15:24:25.565625+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:24:25.565625+0000","last_undegraded":"2024-01-16T15:24:25.565625+0000","last_fullsized":"2024-01-16T15:24:25.565625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:16:21.485125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'84","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250676+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:27:49.250676+0000","last_peered":"2024-01-16T15:27:49.250676+0000","last_clean":"2024-01-16T15:27:49.250676+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:27:49.250676+0000","last_undegraded":"2024-01-16T15:27:49.250676+0000","last_fullsized":"2024-01-16T15:27:49.250676+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:53.269911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.565587+0000","last_change":"2024-01-16T15:19:23.618597+0000","last_active":"2024-01-16T15:24:25.565587+0000","last_peered":"2024-01-16T15:24:25.565587+0000","last_clean":"2024-01-16T15:24:25.565587+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:24:25.565587+0000","last_undegraded":"2024-01-16T15:24:25.565587+0000","last_fullsized":"2024-01-16T15:24:25.565587+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:05.775312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'78","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250467+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:27:49.250467+0000","last_peered":"2024-01-16T15:27:49.250467+0000","last_clean":"2024-01-16T15:27:49.250467+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:27:49.250467+0000","last_undegraded":"2024-01-16T15:27:49.250467+0000","last_fullsized":"2024-01-16T15:27:49.250467+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:28:01.404745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.578781+0000","last_change":"2024-01-16T15:19:23.618024+0000","last_active":"2024-01-16T15:24:25.578781+0000","last_peered":"2024-01-16T15:24:25.578781+0000","last_clean":"2024-01-16T15:24:25.578781+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:24:25.578781+0000","last_undegraded":"2024-01-16T15:24:25.578781+0000","last_fullsized":"2024-01-16T15:24:25.578781+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:32:50.476809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'82","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234990+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:27:49.234990+0000","last_peered":"2024-01-16T15:27:49.234990+0000","last_clean":"2024-01-16T15:27:49.234990+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:27:49.234990+0000","last_undegraded":"2024-01-16T15:27:49.234990+0000","last_fullsized":"2024-01-16T15:27:49.234990+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:25.321825+0000","last_change":"2024-01-16T15:19:23.622641+0000","last_active":"2024-01-16T15:24:25.321825+0000","last_peered":"2024-01-16T15:24:25.321825+0000","last_clean":"2024-01-16T15:24:25.321825+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:24:25.321825+0000","last_undegraded":"2024-01-16T15:24:25.321825+0000","last_fullsized":"2024-01-16T15:24:25.321825+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:39.846805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'80","reported_seq":107,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250148+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:27:49.250148+0000","last_peered":"2024-01-16T15:27:49.250148+0000","last_clean":"2024-01-16T15:27:49.250148+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:27:49.250148+0000","last_undegraded":"2024-01-16T15:27:49.250148+0000","last_fullsized":"2024-01-16T15:27:49.250148+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:19:48.939386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.744702+0000","last_change":"2024-01-16T15:19:23.620941+0000","last_active":"2024-01-16T15:24:16.744702+0000","last_peered":"2024-01-16T15:24:16.744702+0000","last_clean":"2024-01-16T15:24:16.744702+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:24:16.744702+0000","last_undegraded":"2024-01-16T15:24:16.744702+0000","last_fullsized":"2024-01-16T15:24:16.744702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:27.971375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'68","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234098+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:27:49.234098+0000","last_peered":"2024-01-16T15:27:49.234098+0000","last_clean":"2024-01-16T15:27:49.234098+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:27:49.234098+0000","last_undegraded":"2024-01-16T15:27:49.234098+0000","last_fullsized":"2024-01-16T15:27:49.234098+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:29:53.297338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.587238+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:24:15.587238+0000","last_peered":"2024-01-16T15:24:15.587238+0000","last_clean":"2024-01-16T15:24:15.587238+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:24:15.587238+0000","last_undegraded":"2024-01-16T15:24:15.587238+0000","last_fullsized":"2024-01-16T15:24:15.587238+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:07.254538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'66","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242411+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:27:49.242411+0000","last_peered":"2024-01-16T15:27:49.242411+0000","last_clean":"2024-01-16T15:27:49.242411+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:27:49.242411+0000","last_undegraded":"2024-01-16T15:27:49.242411+0000","last_fullsized":"2024-01-16T15:27:49.242411+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:40.365875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.563703+0000","last_change":"2024-01-16T15:19:23.621670+0000","last_active":"2024-01-16T15:24:15.563703+0000","last_peered":"2024-01-16T15:24:15.563703+0000","last_clean":"2024-01-16T15:24:15.563703+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:24:15.563703+0000","last_undegraded":"2024-01-16T15:24:15.563703+0000","last_fullsized":"2024-01-16T15:24:15.563703+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:53.183004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'64","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237836+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:27:49.237836+0000","last_peered":"2024-01-16T15:27:49.237836+0000","last_clean":"2024-01-16T15:27:49.237836+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:27:49.237836+0000","last_undegraded":"2024-01-16T15:27:49.237836+0000","last_fullsized":"2024-01-16T15:27:49.237836+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:56.499150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:15.319813+0000","last_change":"2024-01-16T15:19:23.624332+0000","last_active":"2024-01-16T15:24:15.319813+0000","last_peered":"2024-01-16T15:24:15.319813+0000","last_clean":"2024-01-16T15:24:15.319813+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:24:15.319813+0000","last_undegraded":"2024-01-16T15:24:15.319813+0000","last_fullsized":"2024-01-16T15:24:15.319813+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:23:56.995452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'60","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243544+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:27:49.243544+0000","last_peered":"2024-01-16T15:27:49.243544+0000","last_clean":"2024-01-16T15:27:49.243544+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:27:49.243544+0000","last_undegraded":"2024-01-16T15:27:49.243544+0000","last_fullsized":"2024-01-16T15:27:49.243544+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:11:06.684073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'31","reported_seq":48,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:43.246740+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:25:43.246740+0000","last_peered":"2024-01-16T15:25:43.246740+0000","last_clean":"2024-01-16T15:25:43.246740+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:25:43.246740+0000","last_undegraded":"2024-01-16T15:25:43.246740+0000","last_fullsized":"2024-01-16T15:25:43.246740+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:48.585086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'69","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251340+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:27:49.251340+0000","last_peered":"2024-01-16T15:27:49.251340+0000","last_clean":"2024-01-16T15:27:49.251340+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:27:49.251340+0000","last_undegraded":"2024-01-16T15:27:49.251340+0000","last_fullsized":"2024-01-16T15:27:49.251340+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:00:23.918834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'307","reported_seq":324,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.260197+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:27:59.260197+0000","last_peered":"2024-01-16T15:27:59.260197+0000","last_clean":"2024-01-16T15:27:59.260197+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:27:59.260197+0000","last_undegraded":"2024-01-16T15:27:59.260197+0000","last_fullsized":"2024-01-16T15:27:59.260197+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:52.253914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'63","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242837+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:27:49.242837+0000","last_peered":"2024-01-16T15:27:49.242837+0000","last_clean":"2024-01-16T15:27:49.242837+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:27:49.242837+0000","last_undegraded":"2024-01-16T15:27:49.242837+0000","last_fullsized":"2024-01-16T15:27:49.242837+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:24:37.710593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.569761+0000","last_change":"2024-01-16T15:19:23.617724+0000","last_active":"2024-01-16T15:24:05.569761+0000","last_peered":"2024-01-16T15:24:05.569761+0000","last_clean":"2024-01-16T15:24:05.569761+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:24:05.569761+0000","last_undegraded":"2024-01-16T15:24:05.569761+0000","last_fullsized":"2024-01-16T15:24:05.569761+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:50:35.823425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'80","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249807+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:27:49.249807+0000","last_peered":"2024-01-16T15:27:49.249807+0000","last_clean":"2024-01-16T15:27:49.249807+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:27:49.249807+0000","last_undegraded":"2024-01-16T15:27:49.249807+0000","last_fullsized":"2024-01-16T15:27:49.249807+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.195535+0000","last_change":"2024-01-16T15:19:23.624683+0000","last_active":"2024-01-16T15:24:07.195535+0000","last_peered":"2024-01-16T15:24:07.195535+0000","last_clean":"2024-01-16T15:24:07.195535+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:24:07.195535+0000","last_undegraded":"2024-01-16T15:24:07.195535+0000","last_fullsized":"2024-01-16T15:24:07.195535+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:38.379463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'73","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244605+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:27:49.244605+0000","last_peered":"2024-01-16T15:27:49.244605+0000","last_clean":"2024-01-16T15:27:49.244605+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:27:49.244605+0000","last_undegraded":"2024-01-16T15:27:49.244605+0000","last_fullsized":"2024-01-16T15:27:49.244605+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"109'8","reported_seq":25,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.561655+0000","last_change":"2024-01-16T15:19:23.625591+0000","last_active":"2024-01-16T15:24:05.561655+0000","last_peered":"2024-01-16T15:24:05.561655+0000","last_clean":"2024-01-16T15:24:05.561655+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:24:05.561655+0000","last_undegraded":"2024-01-16T15:24:05.561655+0000","last_fullsized":"2024-01-16T15:24:05.561655+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:27.727635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'50","reported_seq":70,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250374+0000","last_change":"2024-01-16T15:19:24.617249+0000","last_active":"2024-01-16T15:27:49.250374+0000","last_peered":"2024-01-16T15:27:49.250374+0000","last_clean":"2024-01-16T15:27:49.250374+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:27:49.250374+0000","last_undegraded":"2024-01-16T15:27:49.250374+0000","last_fullsized":"2024-01-16T15:27:49.250374+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:26:01.440713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.582905+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:23:55.582905+0000","last_peered":"2024-01-16T15:23:55.582905+0000","last_clean":"2024-01-16T15:23:55.582905+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:23:55.582905+0000","last_undegraded":"2024-01-16T15:23:55.582905+0000","last_fullsized":"2024-01-16T15:23:55.582905+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:27.435680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'61","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239584+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:27:49.239584+0000","last_peered":"2024-01-16T15:27:49.239584+0000","last_clean":"2024-01-16T15:27:49.239584+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:27:49.239584+0000","last_undegraded":"2024-01-16T15:27:49.239584+0000","last_fullsized":"2024-01-16T15:27:49.239584+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:08:20.084956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.560107+0000","last_change":"2024-01-16T15:19:23.622230+0000","last_active":"2024-01-16T15:23:55.560107+0000","last_peered":"2024-01-16T15:23:55.560107+0000","last_clean":"2024-01-16T15:23:55.560107+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:23:55.560107+0000","last_undegraded":"2024-01-16T15:23:55.560107+0000","last_fullsized":"2024-01-16T15:23:55.560107+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'77","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248743+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:27:49.248743+0000","last_peered":"2024-01-16T15:27:49.248743+0000","last_clean":"2024-01-16T15:27:49.248743+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:27:49.248743+0000","last_undegraded":"2024-01-16T15:27:49.248743+0000","last_fullsized":"2024-01-16T15:27:49.248743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:31.587040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.193814+0000","last_change":"2024-01-16T15:19:23.626903+0000","last_active":"2024-01-16T15:23:57.193814+0000","last_peered":"2024-01-16T15:23:57.193814+0000","last_clean":"2024-01-16T15:23:57.193814+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:23:57.193814+0000","last_undegraded":"2024-01-16T15:23:57.193814+0000","last_fullsized":"2024-01-16T15:23:57.193814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:58.751536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'66","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251742+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:27:49.251742+0000","last_peered":"2024-01-16T15:27:49.251742+0000","last_clean":"2024-01-16T15:27:49.251742+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:27:49.251742+0000","last_undegraded":"2024-01-16T15:27:49.251742+0000","last_fullsized":"2024-01-16T15:27:49.251742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:32.314064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.573325+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:23:55.573325+0000","last_peered":"2024-01-16T15:23:55.573325+0000","last_clean":"2024-01-16T15:23:55.573325+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:23:55.573325+0000","last_undegraded":"2024-01-16T15:23:55.573325+0000","last_fullsized":"2024-01-16T15:23:55.573325+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:30:00.825374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'62","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248439+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:27:49.248439+0000","last_peered":"2024-01-16T15:27:49.248439+0000","last_clean":"2024-01-16T15:27:49.248439+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:27:49.248439+0000","last_undegraded":"2024-01-16T15:27:49.248439+0000","last_fullsized":"2024-01-16T15:27:49.248439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:32.038856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'3","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:55.567924+0000","last_change":"2024-01-16T15:19:23.621285+0000","last_active":"2024-01-16T15:23:55.567924+0000","last_peered":"2024-01-16T15:23:55.567924+0000","last_clean":"2024-01-16T15:23:55.567924+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:23:55.567924+0000","last_undegraded":"2024-01-16T15:23:55.567924+0000","last_fullsized":"2024-01-16T15:23:55.567924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:14:44.070274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248670+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:27:49.248670+0000","last_peered":"2024-01-16T15:27:49.248670+0000","last_clean":"2024-01-16T15:27:49.248670+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:27:49.248670+0000","last_undegraded":"2024-01-16T15:27:49.248670+0000","last_fullsized":"2024-01-16T15:27:49.248670+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:17:26.904243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.558296+0000","last_change":"2024-01-16T15:19:23.619619+0000","last_active":"2024-01-16T15:23:45.558296+0000","last_peered":"2024-01-16T15:23:45.558296+0000","last_clean":"2024-01-16T15:23:45.558296+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:23:45.558296+0000","last_undegraded":"2024-01-16T15:23:45.558296+0000","last_fullsized":"2024-01-16T15:23:45.558296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:38:15.166720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'66","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249594+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:27:49.249594+0000","last_peered":"2024-01-16T15:27:49.249594+0000","last_clean":"2024-01-16T15:27:49.249594+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:27:49.249594+0000","last_undegraded":"2024-01-16T15:27:49.249594+0000","last_fullsized":"2024-01-16T15:27:49.249594+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:58.872650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.565824+0000","last_change":"2024-01-16T15:19:23.620635+0000","last_active":"2024-01-16T15:23:45.565824+0000","last_peered":"2024-01-16T15:23:45.565824+0000","last_clean":"2024-01-16T15:23:45.565824+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:23:45.565824+0000","last_undegraded":"2024-01-16T15:23:45.565824+0000","last_fullsized":"2024-01-16T15:23:45.565824+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:53:47.280904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'63","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252548+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:27:49.252548+0000","last_peered":"2024-01-16T15:27:49.252548+0000","last_clean":"2024-01-16T15:27:49.252548+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:27:49.252548+0000","last_undegraded":"2024-01-16T15:27:49.252548+0000","last_fullsized":"2024-01-16T15:27:49.252548+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.580304+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:23:45.580304+0000","last_peered":"2024-01-16T15:23:45.580304+0000","last_clean":"2024-01-16T15:23:45.580304+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:23:45.580304+0000","last_undegraded":"2024-01-16T15:23:45.580304+0000","last_fullsized":"2024-01-16T15:23:45.580304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:48.621454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'70","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236778+0000","last_change":"2024-01-16T15:19:42.592735+0000","last_active":"2024-01-16T15:27:49.236778+0000","last_peered":"2024-01-16T15:27:49.236778+0000","last_clean":"2024-01-16T15:27:49.236778+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:27:49.236778+0000","last_undegraded":"2024-01-16T15:27:49.236778+0000","last_fullsized":"2024-01-16T15:27:49.236778+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:07.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:45.314110+0000","last_change":"2024-01-16T15:19:23.630170+0000","last_active":"2024-01-16T15:23:45.314110+0000","last_peered":"2024-01-16T15:23:45.314110+0000","last_clean":"2024-01-16T15:23:45.314110+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:23:45.314110+0000","last_undegraded":"2024-01-16T15:23:45.314110+0000","last_fullsized":"2024-01-16T15:23:45.314110+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:06.035275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256006+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:27:49.256006+0000","last_peered":"2024-01-16T15:27:49.256006+0000","last_clean":"2024-01-16T15:27:49.256006+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:27:49.256006+0000","last_undegraded":"2024-01-16T15:27:49.256006+0000","last_fullsized":"2024-01-16T15:27:49.256006+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:37:58.148829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":48,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:44.257180+0000","last_change":"2024-01-16T15:19:23.623778+0000","last_active":"2024-01-16T15:27:44.257180+0000","last_peered":"2024-01-16T15:27:44.257180+0000","last_clean":"2024-01-16T15:27:44.257180+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:27:44.257180+0000","last_undegraded":"2024-01-16T15:27:44.257180+0000","last_fullsized":"2024-01-16T15:27:44.257180+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:53.783590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'69","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244950+0000","last_change":"2024-01-16T15:19:24.628361+0000","last_active":"2024-01-16T15:27:49.244950+0000","last_peered":"2024-01-16T15:27:49.244950+0000","last_clean":"2024-01-16T15:27:49.244950+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:27:49.244950+0000","last_undegraded":"2024-01-16T15:27:49.244950+0000","last_fullsized":"2024-01-16T15:27:49.244950+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:48.469423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.606971+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:25:40.606971+0000","last_peered":"2024-01-16T15:25:40.606971+0000","last_clean":"2024-01-16T15:25:40.606971+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:25:40.606971+0000","last_undegraded":"2024-01-16T15:25:40.606971+0000","last_fullsized":"2024-01-16T15:25:40.606971+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:40.909970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'53","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236803+0000","last_change":"2024-01-16T15:19:24.623591+0000","last_active":"2024-01-16T15:27:49.236803+0000","last_peered":"2024-01-16T15:27:49.236803+0000","last_clean":"2024-01-16T15:27:49.236803+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:27:49.236803+0000","last_undegraded":"2024-01-16T15:27:49.236803+0000","last_fullsized":"2024-01-16T15:27:49.236803+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:23:57.706980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.336436+0000","last_change":"2024-01-16T15:19:23.612616+0000","last_active":"2024-01-16T15:25:40.336436+0000","last_peered":"2024-01-16T15:25:40.336436+0000","last_clean":"2024-01-16T15:25:40.336436+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:25:40.336436+0000","last_undegraded":"2024-01-16T15:25:40.336436+0000","last_fullsized":"2024-01-16T15:25:40.336436+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:19.160736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'65","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242630+0000","last_change":"2024-01-16T15:19:24.626321+0000","last_active":"2024-01-16T15:27:49.242630+0000","last_peered":"2024-01-16T15:27:49.242630+0000","last_clean":"2024-01-16T15:27:49.242630+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:27:49.242630+0000","last_undegraded":"2024-01-16T15:27:49.242630+0000","last_fullsized":"2024-01-16T15:27:49.242630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:40.591466+0000","last_change":"2024-01-16T15:19:23.624071+0000","last_active":"2024-01-16T15:25:40.591466+0000","last_peered":"2024-01-16T15:25:40.591466+0000","last_clean":"2024-01-16T15:25:40.591466+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:25:40.591466+0000","last_undegraded":"2024-01-16T15:25:40.591466+0000","last_fullsized":"2024-01-16T15:25:40.591466+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:49:45.204823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'64","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236252+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:27:49.236252+0000","last_peered":"2024-01-16T15:27:49.236252+0000","last_clean":"2024-01-16T15:27:49.236252+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:27:49.236252+0000","last_undegraded":"2024-01-16T15:27:49.236252+0000","last_fullsized":"2024-01-16T15:27:49.236252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:41.760446+0000","last_change":"2024-01-16T15:19:23.619253+0000","last_active":"2024-01-16T15:25:41.760446+0000","last_peered":"2024-01-16T15:25:41.760446+0000","last_clean":"2024-01-16T15:25:41.760446+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:25:41.760446+0000","last_undegraded":"2024-01-16T15:25:41.760446+0000","last_fullsized":"2024-01-16T15:25:41.760446+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:16.781683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'73","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243648+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:27:49.243648+0000","last_peered":"2024-01-16T15:27:49.243648+0000","last_clean":"2024-01-16T15:27:49.243648+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:27:49.243648+0000","last_undegraded":"2024-01-16T15:27:49.243648+0000","last_fullsized":"2024-01-16T15:27:49.243648+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:49.637451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'31","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:43.237902+0000","last_change":"2024-01-16T15:19:23.629871+0000","last_active":"2024-01-16T15:25:43.237902+0000","last_peered":"2024-01-16T15:25:43.237902+0000","last_clean":"2024-01-16T15:25:43.237902+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:25:43.237902+0000","last_undegraded":"2024-01-16T15:25:43.237902+0000","last_fullsized":"2024-01-16T15:25:43.237902+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:05:17.872229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251399+0000","last_change":"2024-01-16T15:19:24.621286+0000","last_active":"2024-01-16T15:27:49.251399+0000","last_peered":"2024-01-16T15:27:49.251399+0000","last_clean":"2024-01-16T15:27:49.251399+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:27:49.251399+0000","last_undegraded":"2024-01-16T15:27:49.251399+0000","last_fullsized":"2024-01-16T15:27:49.251399+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:20:57.053368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.603477+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:25:25.603477+0000","last_peered":"2024-01-16T15:25:25.603477+0000","last_clean":"2024-01-16T15:25:25.603477+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:25:25.603477+0000","last_undegraded":"2024-01-16T15:25:25.603477+0000","last_fullsized":"2024-01-16T15:25:25.603477+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:01.099875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'54","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249675+0000","last_change":"2024-01-16T15:19:24.621294+0000","last_active":"2024-01-16T15:27:49.249675+0000","last_peered":"2024-01-16T15:27:49.249675+0000","last_clean":"2024-01-16T15:27:49.249675+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:27:49.249675+0000","last_undegraded":"2024-01-16T15:27:49.249675+0000","last_fullsized":"2024-01-16T15:27:49.249675+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:05.575031+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:24:05.575031+0000","last_peered":"2024-01-16T15:24:05.575031+0000","last_clean":"2024-01-16T15:24:05.575031+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:24:05.575031+0000","last_undegraded":"2024-01-16T15:24:05.575031+0000","last_fullsized":"2024-01-16T15:24:05.575031+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:10:59.676130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":261,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:50.938466+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:25:50.938466+0000","last_peered":"2024-01-16T15:25:50.938466+0000","last_clean":"2024-01-16T15:25:50.938466+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:25:50.938466+0000","last_undegraded":"2024-01-16T15:25:50.938466+0000","last_fullsized":"2024-01-16T15:25:50.938466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4358,"ondisk_log_size":4358,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":432,"ondisk_log_size":432,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509781,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.1559999999999999,"15min":1.25},"min":{"1min":0.69899999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5409999999999999,"5min":1.7749999999999999,"15min":19.224},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.212,"15min":1.3140000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.615,"5min":1.694,"15min":28.603999999999999},"last":0.93500000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.22,"15min":1.2250000000000001},"min":{"1min":0.90500000000000003,"5min":0.80800000000000005,"15min":0.374},"max":{"1min":1.488,"5min":1.837,"15min":3.7069999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.3080000000000001,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.68899999999999995,"15min":0.68400000000000005},"max":{"1min":1.651,"5min":1.8819999999999999,"15min":33.225000000000001},"last":1.093}]},{"osd":2,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3380000000000001,"15min":1.4199999999999999},"min":{"1min":1.0529999999999999,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.752,"5min":1.7909999999999999,"15min":19.126000000000001},"last":1.7549999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3080000000000001,"15min":1.3120000000000001},"min":{"1min":1.008,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.694,"5min":1.718,"15min":2.9750000000000001},"last":1.3500000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.76500000000000001},"max":{"1min":1.526,"5min":1.73,"15min":3.7999999999999998},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.446,"15min":1.5580000000000001},"min":{"1min":0.97399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":19.268999999999998},"last":1.972}]},{"osd":4,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2290000000000001,"15min":1.264},"min":{"1min":0.88300000000000001,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.573},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.347,"15min":1.3919999999999999},"min":{"1min":1.1180000000000001,"5min":0.70299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6659999999999999,"5min":1.7330000000000001,"15min":9.7059999999999995},"last":2.452}]},{"osd":5,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":1.0389999999999999,"5min":0.86099999999999999,"15min":0.79400000000000004},"max":{"1min":1.74,"5min":1.9450000000000001,"15min":3.9129999999999998},"last":2.8029999999999999},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.5620000000000001},"min":{"1min":0.82699999999999996,"5min":0.77900000000000003,"15min":0.55100000000000005},"max":{"1min":1.8720000000000001,"5min":1.9630000000000001,"15min":19.036999999999999},"last":2.9009999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.4359999999999999,"15min":1.4410000000000001},"min":{"1min":1.198,"5min":0.74299999999999999,"15min":0.54800000000000004},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":4.2240000000000002},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.528,"15min":1.571},"min":{"1min":1.224,"5min":0.98999999999999999,"15min":0.65100000000000002},"max":{"1min":1.728,"5min":2.181,"15min":4.0709999999999997},"last":2.3090000000000002}]}]},{"osd":4,"up_from":15,"seq":64424509783,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.296,"15min":1.2330000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.427,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.452},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1799999999999999,"15min":1.1859999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.4239999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.88700000000000001,"5min":0.73899999999999999,"15min":0.71899999999999997},"max":{"1min":1.6160000000000001,"5min":1.893,"15min":2.0510000000000002},"last":1.333},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.3149999999999999,"15min":1.302},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.6319999999999999,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.2889999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3420000000000001,"15min":1.3320000000000001},"min":{"1min":0.79300000000000004,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.573,"5min":1.78,"15min":5.907},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.242,"5min":1.381,"15min":1.355},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.52600000000000002},"max":{"1min":1.6000000000000001,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":1.653}]},{"osd":3,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4570000000000001,"15min":1.3939999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.76200000000000001},"max":{"1min":1.5289999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.3680000000000001},{"interface":"front","average":{"1min":1.391,"5min":1.4019999999999999,"15min":1.391},"min":{"1min":1.052,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.74,"5min":1.9670000000000001,"15min":2.0379999999999998},"last":1.397}]},{"osd":5,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.425,"15min":1.448},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6779999999999999,"5min":6.165,"15min":6.165},"last":1.544},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4410000000000001,"15min":1.4770000000000001},"min":{"1min":0.95299999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8180000000000001,"5min":3.1469999999999998,"15min":5.3520000000000003},"last":0.95399999999999996}]},{"osd":6,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.391,"15min":1.4219999999999999},"min":{"1min":1.101,"5min":0.90800000000000003,"15min":0.497},"max":{"1min":1.903,"5min":1.903,"15min":8.9600000000000009},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5640000000000001,"15min":1.528},"min":{"1min":1.165,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.772,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.603}]},{"osd":7,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.452,"15min":1.492},"min":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.89700000000000002},"max":{"1min":1.8500000000000001,"5min":3.3620000000000001,"15min":5.835},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.554,"5min":1.5469999999999999,"15min":1.581},"min":{"1min":1.238,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":2.0070000000000001,"5min":2.3679999999999999,"15min":8.0079999999999991},"last":1.0329999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542485,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1891948,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92479892,"statfs":{"total":96636764160,"available":94699409408,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1659999999999999,"15min":1.2130000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.65400000000000003},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.1779999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.24,"5min":1.2210000000000001,"15min":1.248},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":12.461},"last":1.163}]},{"osd":1,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.89000000000000001,"5min":0.85099999999999998,"15min":0.48599999999999999},"max":{"1min":1.923,"5min":1.923,"15min":2.4249999999999998},"last":1.091},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2709999999999999,"15min":1.363},"min":{"1min":0.93400000000000005,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":1.8140000000000001,"5min":1.9079999999999999,"15min":14.848000000000001},"last":1.2929999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3069999999999999,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":3.726},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.286,"5min":1.3340000000000001,"15min":1.3959999999999999},"min":{"1min":0.88900000000000001,"5min":0.873,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.792,"15min":19.763999999999999},"last":1.2569999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3340000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80600000000000005},"max":{"1min":1.6539999999999999,"5min":1.8280000000000001,"15min":1.887},"last":1.405},{"interface":"front","average":{"1min":1.456,"5min":1.47,"15min":1.4650000000000001},"min":{"1min":1.113,"5min":1.0169999999999999,"15min":0.629},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":6.6029999999999998},"last":1.359}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.345,"15min":1.347},"min":{"1min":0.98499999999999999,"5min":0.89100000000000001,"15min":0.54100000000000004},"max":{"1min":1.7090000000000001,"5min":1.7929999999999999,"15min":4.1390000000000002},"last":1.002},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.401,"15min":1.413},"min":{"1min":1.0009999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":1.873,"5min":1.877,"15min":2.5190000000000001},"last":1.454}]},{"osd":5,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3340000000000001,"15min":1.3680000000000001},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":12.609999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5229999999999999},"min":{"1min":1.1719999999999999,"5min":1.046,"15min":0.66000000000000003},"max":{"1min":1.8879999999999999,"5min":1.909,"15min":3.4159999999999999},"last":1.3129999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.099,"5min":1.0760000000000001,"15min":0.66000000000000003},"max":{"1min":2.0859999999999999,"5min":2.1429999999999998,"15min":2.653},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.552},"min":{"1min":1.004,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":12.567},"last":1.042}]}]},{"osd":5,"up_from":14,"seq":60129542486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.208},"min":{"1min":0.83399999999999996,"5min":0.749,"15min":0.32900000000000001},"max":{"1min":1.7190000000000001,"5min":2.0939999999999999,"15min":6.8380000000000001},"last":0.82199999999999995},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1870000000000001,"15min":1.3200000000000001},"min":{"1min":1.0049999999999999,"5min":0.82999999999999996,"15min":0.57099999999999995},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":23.199999999999999},"last":1.2290000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.246},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.63400000000000001},"max":{"1min":1.637,"5min":1.698,"15min":8.0909999999999993},"last":0.996},{"interface":"front","average":{"1min":1.355,"5min":1.26,"15min":1.323},"min":{"1min":0.93600000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8049999999999999,"5min":1.851,"15min":17.062000000000001},"last":0.76600000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.409,"15min":1.415},"min":{"1min":1.069,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.6970000000000001,"5min":1.782,"15min":5.8810000000000002},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":1.131,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.6899999999999999,"5min":1.6970000000000001,"15min":5.4429999999999996},"last":1.022}]},{"osd":3,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.302,"15min":1.3420000000000001},"min":{"1min":0.94999999999999996,"5min":0.878,"15min":0.67500000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":7.984},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.385,"5min":1.4179999999999999,"15min":1.423},"min":{"1min":0.94599999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.7410000000000001,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":0.69099999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.323,"15min":1.387},"min":{"1min":0.878,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":15.975},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4079999999999999,"15min":1.413},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.54000000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":5.6349999999999998},"last":0.59199999999999997}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.329,"15min":1.3680000000000001},"min":{"1min":0.95999999999999996,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.8160000000000001,"5min":1.827,"15min":12.914},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.468,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":16.552},"last":1.165}]},{"osd":7,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4630000000000001,"15min":1.506},"min":{"1min":1.298,"5min":0.98099999999999998,"15min":0.64300000000000002},"max":{"1min":1.74,"5min":1.8540000000000001,"15min":13.034000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5740000000000001,"15min":1.5900000000000001},"min":{"1min":1.161,"5min":1.0429999999999999,"15min":0.70899999999999996},"max":{"1min":1.9159999999999999,"5min":2.073,"15min":8.0380000000000003},"last":0.95799999999999996}]}]},{"osd":3,"up_from":14,"seq":60129542486,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":5.7400000000000002},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.1359999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40600000000000003},"max":{"1min":1.506,"5min":1.879,"15min":7.1429999999999998},"last":0.91100000000000003}]},{"osd":1,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.96899999999999997,"15min":1.0349999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.365,"5min":1.738,"15min":10.856999999999999},"last":1.224},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.1870000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4950000000000001,"5min":1.823,"15min":2.0089999999999999},"last":1.052}]},{"osd":2,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.3140000000000001,"15min":1.5209999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.946,"5min":1.946,"15min":39.265999999999998},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1559999999999999,"15min":1.2070000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.381,"5min":1.833,"15min":7.9160000000000004},"last":1.097}]},{"osd":4,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.337,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.70999999999999996},"max":{"1min":1.5980000000000001,"5min":1.9419999999999999,"15min":5.6890000000000001},"last":1.01},{"interface":"front","average":{"1min":1.329,"5min":1.464,"15min":1.4810000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8120000000000001,"5min":2.0920000000000001,"15min":2.3330000000000002},"last":1.889}]},{"osd":5,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3440000000000001,"15min":1.4670000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5669999999999999,"5min":1.827,"15min":14.003},"last":2.1339999999999999},{"interface":"front","average":{"1min":1.357,"5min":1.5049999999999999,"15min":1.577},"min":{"1min":0.877,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":1.9359999999999999,"5min":2.008,"15min":6.0679999999999996},"last":1.8160000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.464,"15min":1.502},"min":{"1min":0.875,"5min":0.875,"15min":0.69799999999999995},"max":{"1min":1.8899999999999999,"5min":2.0139999999999998,"15min":7.0060000000000002},"last":1.2709999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.486,"15min":1.5580000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8140000000000001,"5min":1.982,"15min":15.114000000000001},"last":1.966}]},{"osd":7,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4950000000000001,"15min":1.6160000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80400000000000005},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":12.378},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5780000000000001,"15min":1.6220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9650000000000001,"5min":2.23,"15min":4.6029999999999998},"last":2.2589999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542485,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1059928,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93311912,"statfs":{"total":96636764160,"available":95551397888,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.90500000000000003,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.41199999999999998,"15min":0.32400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.161,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":1.833,"5min":1.913,"15min":7.9960000000000004},"last":0.97499999999999998}]},{"osd":1,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0880000000000001,"15min":1.1040000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":1.7130000000000001,"5min":2.294,"15min":3.056},"last":0.82299999999999995},{"interface":"front","average":{"1min":1.171,"5min":1.143,"15min":1.175},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.556,"5min":1.877,"15min":4.3819999999999997},"last":1.3819999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.206,"15min":1.2789999999999999},"min":{"1min":0.61199999999999999,"5min":0.55300000000000005,"15min":0.44700000000000001},"max":{"1min":1.704,"5min":1.883,"15min":18.838000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4950000000000001,"15min":1.552},"min":{"1min":0.66200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":2.0099999999999998,"15min":19.375},"last":1.4930000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.335,"15min":1.4390000000000001},"min":{"1min":0.80600000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.8240000000000001,"5min":2.242,"15min":23.225000000000001},"last":0.89100000000000001},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4730000000000001,"15min":1.494},"min":{"1min":1.038,"5min":0.75900000000000001,"15min":0.72699999999999998},"max":{"1min":1.883,"5min":1.883,"15min":6.359},"last":1.6779999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4399999999999999,"15min":1.5009999999999999},"min":{"1min":1.0800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.917,"5min":2.3170000000000002,"15min":20.157},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.597,"15min":1.6100000000000001},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.623},"max":{"1min":1.9330000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.0449999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5289999999999999,"15min":1.595},"min":{"1min":1.018,"5min":0.89800000000000002,"15min":0.70499999999999996},"max":{"1min":1.7330000000000001,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.101},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6160000000000001,"15min":1.7170000000000001},"min":{"1min":0.85799999999999998,"5min":0.79300000000000004,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.1419999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5429999999999999,"15min":1.6699999999999999},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.254},{"interface":"front","average":{"1min":1.587,"5min":1.569,"15min":1.5880000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":1.8999999999999999,"5min":2.0990000000000002,"15min":4.3419999999999996},"last":1.7569999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542487,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.093,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.28100000000000003},"max":{"1min":13.42,"5min":13.42,"15min":13.42},"last":0.77000000000000002},{"interface":"front","average":{"1min":1.216,"5min":1.141,"15min":1.113},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":1.4970000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.03,"15min":1.0900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.649,"15min":3.6819999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.29,"15min":1.3320000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.39000000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":3.5600000000000001},"last":1.4610000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.26,"15min":1.2889999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46300000000000002},"max":{"1min":1.488,"5min":1.875,"15min":2.238},"last":1.2130000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.137,"15min":1.194},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":0.94299999999999995}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.72199999999999998},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.077}]},{"osd":5,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3939999999999999,"15min":1.431},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63700000000000001},"max":{"1min":1.6359999999999999,"5min":1.952,"15min":3.98},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4530000000000001,"15min":1.462},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.53300000000000003},"max":{"1min":1.669,"5min":1.893,"15min":3.3940000000000001},"last":1.385}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.56999999999999995},"max":{"1min":1.8660000000000001,"5min":2.016,"15min":16.48},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.508,"15min":1.532},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":0.65600000000000003},"max":{"1min":1.74,"5min":1.909,"15min":3.8180000000000001},"last":1.526}]},{"osd":7,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.573,"15min":1.617},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.48099999999999998},"max":{"1min":1.8480000000000001,"5min":2.0009999999999999,"15min":7.0499999999999998},"last":1.637},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.6339999999999999,"15min":1.647},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.81100000000000005},"max":{"1min":1.8640000000000001,"5min":2.3940000000000001,"15min":3.7839999999999998},"last":1.5700000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542485,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":1.038},"min":{"1min":0.496,"5min":0.496,"15min":0.35199999999999998},"max":{"1min":1.427,"5min":1.665,"15min":15.776},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.151,"15min":1.171},"min":{"1min":0.67600000000000005,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":1.9019999999999999,"5min":2.7549999999999999,"15min":17.620999999999999},"last":1.119}]},{"osd":2,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.137,"15min":1.22},"min":{"1min":0.74099999999999999,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.671,"5min":1.788,"15min":15.997999999999999},"last":1.5760000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.23,"15min":1.1919999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":1.669,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.2490000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.129,"15min":1.179},"min":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.4279999999999999},"min":{"1min":0.86299999999999999,"5min":0.66600000000000004,"15min":0.47399999999999998},"max":{"1min":1.74,"5min":1.827,"15min":16.126000000000001},"last":1.194}]},{"osd":4,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.466,"15min":1.573},"min":{"1min":1.095,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.9470000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.675},{"interface":"front","average":{"1min":1.446,"5min":1.4430000000000001,"15min":1.419},"min":{"1min":1.0089999999999999,"5min":0.82299999999999995,"15min":0.71899999999999997},"max":{"1min":1.913,"5min":3.0099999999999998,"15min":3.0459999999999998},"last":1.395}]},{"osd":5,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.97099999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7869999999999999,"5min":2.157,"15min":5.5609999999999999},"last":1.351},{"interface":"front","average":{"1min":1.581,"5min":1.5529999999999999,"15min":1.639},"min":{"1min":1.2130000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0630000000000002,"5min":2.1989999999999998,"15min":16.501999999999999},"last":1.421}]},{"osd":6,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.137,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.891,"5min":2.0790000000000002,"15min":5.4509999999999996},"last":1.542},{"interface":"front","average":{"1min":1.649,"5min":1.6379999999999999,"15min":1.7190000000000001},"min":{"1min":0.98899999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":2.1269999999999998,"5min":2.351,"15min":16.395},"last":1.766}]},{"osd":7,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5800000000000001,"15min":1.595},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":2.3300000000000001,"5min":2.9510000000000001,"15min":9.6980000000000004},"last":1.484},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.589,"15min":1.5860000000000001},"min":{"1min":1.0860000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.954,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.633}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:03.343 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-01-16T15:28:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.608+0000 7feabf200640 1 Processor -- start 2024-01-16T15:28:03.607 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.608+0000 7feabf200640 1 -- start start 2024-01-16T15:28:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.610+0000 7feabf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac00cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.610+0000 7feabf200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feac006d160 0x7feac006d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.610+0000 7feabf200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feac0070e50 con 0x7feac0143c40 2024-01-16T15:28:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.610+0000 7feabf200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feac00723c0 con 0x7feac006d160 2024-01-16T15:28:03.609 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.610+0000 7feabf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac0144020 con 0x7feac00cc500 2024-01-16T15:28:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feac006d160 0x7feac006d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabd400640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac00cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.610 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7feac006d160 0x7feac006d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3300/0 says I am v2:172.21.15.53:33626/0 (socket says 172.21.15.53:33626) 2024-01-16T15:28:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 -- 172.21.15.53:0/1506888032 learned_addr learned my addr 172.21.15.53:0/1506888032 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 -- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 msgr2=0x7feac00cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 --2- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac00cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 -- 172.21.15.53:0/1506888032 >> v1:172.21.15.146:6789/0 conn(0x7feac0143c40 legacy=0x7feac0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.611+0000 7feabe800640 1 -- 172.21.15.53:0/1506888032 --> v2:172.21.15.146:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac00ccea0 con 0x7feac006d160 2024-01-16T15:28:03.611 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.613+0000 7feabe800640 1 --2- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac006d160 0x7feac006d590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7feab8009870 tx=0x7feab802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f12c8bf3028196c3 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:03.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.613+0000 7feabca00640 1 -- 172.21.15.53:0/1506888032 <== mon.1 v2:172.21.15.146:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8031640 con 0x7feac006d160 2024-01-16T15:28:03.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.613+0000 7feabca00640 1 -- 172.21.15.53:0/1506888032 <== mon.1 v2:172.21.15.146:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab80317a0 con 0x7feac006d160 2024-01-16T15:28:03.612 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.613+0000 7feabca00640 1 -- 172.21.15.53:0/1506888032 <== mon.1 v2:172.21.15.146:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8031b30 con 0x7feac006d160 2024-01-16T15:28:03.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 -- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac006d160 msgr2=0x7feac006d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.613 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 --2- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac006d160 0x7feac006d590 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7feab8009870 tx=0x7feab802b8f0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 -- 172.21.15.53:0/1506888032 shutdown_connections 2024-01-16T15:28:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 --2- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac006d160 0x7feac006d590 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 --2- 172.21.15.53:0/1506888032 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac00cc960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 -- 172.21.15.53:0/1506888032 >> 172.21.15.53:0/1506888032 conn(0x7feac006a800 msgr2=0x7feac006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.614+0000 7feabf200640 1 -- 172.21.15.53:0/1506888032 shutdown_connections 2024-01-16T15:28:03.614 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.615+0000 7feabf200640 1 -- 172.21.15.53:0/1506888032 wait complete. 2024-01-16T15:28:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.616+0000 7feabf200640 1 Processor -- start 2024-01-16T15:28:03.615 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.617+0000 7feabf200640 1 -- start start 2024-01-16T15:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.617+0000 7feabf200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 0x7feac015a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.617+0000 7feabf200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac013e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabf200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac0143c40 0x7feac01e8b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabf200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac012d580 con 0x7feac006d160 2024-01-16T15:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabf200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac012d860 con 0x7feac0143c40 2024-01-16T15:28:03.617 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabf200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feac012db40 con 0x7feac00cc500 2024-01-16T15:28:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 0x7feac015a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabde00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 0x7feac015a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55536/0 (socket says 172.21.15.53:55536) 2024-01-16T15:28:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabde00640 1 -- 172.21.15.53:0/757471237 learned_addr learned my addr 172.21.15.53:0/757471237 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabd400640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac013e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.618 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.618+0000 7feabe800640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac0143c40 0x7feac01e8b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.619 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabde00640 1 -- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 msgr2=0x7feac013e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabde00640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac013e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabde00640 1 -- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac0143c40 msgr2=0x7feac01e8b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabde00640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac0143c40 0x7feac01e8b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabde00640 1 -- 172.21.15.53:0/757471237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80094d0 con 0x7feac006d160 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabd400640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac013e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabde00640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 0x7feac015a7d0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7feab000c730 tx=0x7feab000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0003b70 con 0x7feac006d160 2024-01-16T15:28:03.620 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0003cf0 con 0x7feac006d160 2024-01-16T15:28:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab0003ea0 con 0x7feac006d160 2024-01-16T15:28:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac012de20 con 0x7feac006d160 2024-01-16T15:28:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.619+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac015ad10 con 0x7feac006d160 2024-01-16T15:28:03.621 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.621+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac00cdf30 con 0x7feac006d160 2024-01-16T15:28:03.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.622+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7feab000aa90 con 0x7feac006d160 2024-01-16T15:28:03.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.622+0000 7feab6000640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea9806c550 0x7fea9806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:03.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.623+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7feab0088960 con 0x7feac006d160 2024-01-16T15:28:03.622 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.623+0000 7feabd400640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea9806c550 0x7fea9806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:03.623 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.624+0000 7feabd400640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea9806c550 0x7fea9806ea10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7feac00ccc20 tx=0x7feaa8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:03.626 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.627+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7feab0054f00 con 0x7feac006d160 2024-01-16T15:28:03.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.826+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feac0067450 con 0x7feac006d160 2024-01-16T15:28:03.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.829+0000 7feab6000640 1 -- 172.21.15.53:0/757471237 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16631 (secure 0 0 0) 0x7feab0058520 con 0x7feac006d160 2024-01-16T15:28:03.828 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:03.828 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":112,"fsid":"c2138e65-ef75-44e8-8327-90d7ebaa10cd","created":"2024-01-16T14:59:32.985603+0000","modified":"2024-01-16T15:19:42.134132+0000","last_up_change":"2024-01-16T14:59:45.770945+0000","last_in_change":"2024-01-16T14:59:36.723268+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-01-16T14:59:46.242799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2024-01-16T15:19:21.949075+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2024-01-16T15:19:23.088954+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bc153f72-2c5f-47f1-b716-49b22c3153a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6800","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6801","nonce":2075393413}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6802","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6803","nonce":2075393413}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6806","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6807","nonce":2075393413}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6804","nonce":2075393413},{"type":"v1","addr":"172.21.15.53:6805","nonce":2075393413}]},"public_addr":"172.21.15.53:6801/2075393413","cluster_addr":"172.21.15.53:6803/2075393413","heartbeat_back_addr":"172.21.15.53:6807/2075393413","heartbeat_front_addr":"172.21.15.53:6805/2075393413","state":["exists","up"]},{"osd":1,"uuid":"61465415-c9f2-4095-9f00-2721503d20f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6816","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6817","nonce":3560880694}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6818","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6819","nonce":3560880694}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6822","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6823","nonce":3560880694}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6820","nonce":3560880694},{"type":"v1","addr":"172.21.15.53:6821","nonce":3560880694}]},"public_addr":"172.21.15.53:6817/3560880694","cluster_addr":"172.21.15.53:6819/3560880694","heartbeat_back_addr":"172.21.15.53:6823/3560880694","heartbeat_front_addr":"172.21.15.53:6821/3560880694","state":["exists","up"]},{"osd":2,"uuid":"2b673652-9d47-429f-aa72-16eca4b02335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6824","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6825","nonce":16573798}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6826","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6827","nonce":16573798}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6830","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6831","nonce":16573798}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6828","nonce":16573798},{"type":"v1","addr":"172.21.15.53:6829","nonce":16573798}]},"public_addr":"172.21.15.53:6825/16573798","cluster_addr":"172.21.15.53:6827/16573798","heartbeat_back_addr":"172.21.15.53:6831/16573798","heartbeat_front_addr":"172.21.15.53:6829/16573798","state":["exists","up"]},{"osd":3,"uuid":"256f1c58-ed5e-4b01-9237-2b77f0bea192","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6808","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6809","nonce":2110883972}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6810","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6811","nonce":2110883972}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6814","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6815","nonce":2110883972}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6812","nonce":2110883972},{"type":"v1","addr":"172.21.15.53:6813","nonce":2110883972}]},"public_addr":"172.21.15.53:6809/2110883972","cluster_addr":"172.21.15.53:6811/2110883972","heartbeat_back_addr":"172.21.15.53:6815/2110883972","heartbeat_front_addr":"172.21.15.53:6813/2110883972","state":["exists","up"]},{"osd":4,"uuid":"003cd61b-e9e3-4139-8f3f-1348822ea61f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6808","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6810","nonce":981755469}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6812","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6813","nonce":981755469}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6818","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6820","nonce":981755469}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6815","nonce":981755469},{"type":"v1","addr":"172.21.15.146:6817","nonce":981755469}]},"public_addr":"172.21.15.146:6810/981755469","cluster_addr":"172.21.15.146:6813/981755469","heartbeat_back_addr":"172.21.15.146:6820/981755469","heartbeat_front_addr":"172.21.15.146:6817/981755469","state":["exists","up"]},{"osd":5,"uuid":"4a258471-1806-49be-91f6-98d1a0c01f95","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6824","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6825","nonce":2159577511}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6826","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6827","nonce":2159577511}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6830","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6831","nonce":2159577511}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6828","nonce":2159577511},{"type":"v1","addr":"172.21.15.146:6829","nonce":2159577511}]},"public_addr":"172.21.15.146:6825/2159577511","cluster_addr":"172.21.15.146:6827/2159577511","heartbeat_back_addr":"172.21.15.146:6831/2159577511","heartbeat_front_addr":"172.21.15.146:6829/2159577511","state":["exists","up"]},{"osd":6,"uuid":"fabb3d13-1a7f-4e95-8058-1ccc16400482","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6800","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6801","nonce":2492658325}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6802","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6803","nonce":2492658325}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6806","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6807","nonce":2492658325}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6804","nonce":2492658325},{"type":"v1","addr":"172.21.15.146:6805","nonce":2492658325}]},"public_addr":"172.21.15.146:6801/2492658325","cluster_addr":"172.21.15.146:6803/2492658325","heartbeat_back_addr":"172.21.15.146:6807/2492658325","heartbeat_front_addr":"172.21.15.146:6805/2492658325","state":["exists","up"]},{"osd":7,"uuid":"1cbcdcfb-fe12-4b0b-8ee8-d95b4fd4fed5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6809","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6811","nonce":902907051}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6814","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6816","nonce":902907051}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6822","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6823","nonce":902907051}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6819","nonce":902907051},{"type":"v1","addr":"172.21.15.146:6821","nonce":902907051}]},"public_addr":"172.21.15.146:6811/902907051","cluster_addr":"172.21.15.146:6816/902907051","heartbeat_back_addr":"172.21.15.146:6823/902907051","heartbeat_front_addr":"172.21.15.146:6821/902907051","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:41.958144+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.094279+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.083649+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.303196+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.308896+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.082269+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.197421+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-01-16T14:59:42.012427+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"17.6","mappings":[{"from":0,"to":2}]},{"pgid":"17.24","mappings":[{"from":0,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-01-16T15:28:03.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.831+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea9806c550 msgr2=0x7fea9806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.831+0000 7feabf200640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea9806c550 0x7fea9806ea10 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7feac00ccc20 tx=0x7feaa8008040 comp rx=0 tx=0).stop 2024-01-16T15:28:03.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.831+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 msgr2=0x7feac015a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.831+0000 7feabf200640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 0x7feac015a7d0 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7feab000c730 tx=0x7feab000cbf0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 shutdown_connections 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea9806c550 0x7fea9806ea10 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7feac0143c40 0x7feac01e8b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7feac00cc500 0x7feac013e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 --2- 172.21.15.53:0/757471237 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7feac006d160 0x7feac015a7d0 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:03.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 >> 172.21.15.53:0/757471237 conn(0x7feac006a800 msgr2=0x7feac0140060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:03.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 shutdown_connections 2024-01-16T15:28:03.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:03.832+0000 7feabf200640 1 -- 172.21.15.53:0/757471237 wait complete. 2024-01-16T15:28:04.850 INFO:tasks.ceph:Scrubbing osd.0 2024-01-16T15:28:04.850 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:05.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.082+0000 7f36e9200640 1 Processor -- start 2024-01-16T15:28:05.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.082+0000 7f36e9200640 1 -- start start 2024-01-16T15:28:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.084+0000 7f36e9200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f36e4161740 0x7f36e4168380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.084+0000 7f36e9200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40cf260 con 0x7f36e40d0560 2024-01-16T15:28:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.084+0000 7f36e9200640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e40cf4c0 con 0x7f36e40c9870 2024-01-16T15:28:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.084+0000 7f36e9200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e40d2ef0 con 0x7f36e4161740 2024-01-16T15:28:05.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.084+0000 7f36e1400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f36e40d0560 0x7f36e4161030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57594/0 (socket says 172.21.15.53:57594) 2024-01-16T15:28:05.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.084+0000 7f36e1400640 1 -- 172.21.15.53:0/1239847657 learned_addr learned my addr 172.21.15.53:0/1239847657 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e2800640 1 --2- 172.21.15.53:0/1239847657 >> v2:172.21.15.146:3300/0 conn(0x7f36e4161740 0x7f36e4168380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926485850 0 0) 0x7f36e40cf260 con 0x7f36e40d0560 2024-01-16T15:28:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c4003520 con 0x7f36e40d0560 2024-01-16T15:28:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1386476113 0 0) 0x7f36c4003520 con 0x7f36e40d0560 2024-01-16T15:28:05.086 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 >> v2:172.21.15.146:3300/0 conn(0x7f36e4161740 msgr2=0x7f36e4168380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 --2- 172.21.15.53:0/1239847657 >> v2:172.21.15.146:3300/0 conn(0x7f36e4161740 0x7f36e4168380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 >> v1:172.21.15.146:6789/0 conn(0x7f36e40c9870 legacy=0x7f36e40ceb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e40cf7a0 con 0x7f36e40d0560 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f36d40019f0 con 0x7f36e40d0560 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 >> v1:172.21.15.53:6789/0 conn(0x7f36e40d0560 legacy=0x7f36e4161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c40039c0 0x7f36c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40047c0 0x7f36c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c4005180 0x7f36c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40d2ef0 con 0x7f36c40039c0 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36c400c080 con 0x7f36c4005180 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c400c240 con 0x7f36c40047c0 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.085+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c40039c0 0x7f36c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e2800640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c4005180 0x7f36c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e1400640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40047c0 0x7f36c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e1e00640 1 -- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40047c0 msgr2=0x7f36c4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40047c0 0x7f36c4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e1e00640 1 -- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c4005180 msgr2=0x7f36c400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c4005180 0x7f36c400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e1e00640 1 -- 172.21.15.53:0/1239847657 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4003520 con 0x7f36c40039c0 2024-01-16T15:28:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e2800640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36c4005180 0x7f36c400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.086+0000 7f36e2800640 1 -- 172.21.15.53:0/1239847657 reap_dead start 2024-01-16T15:28:05.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.087+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c40039c0 0x7f36c4003de0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f36cc001720 tx=0x7f36c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edde48de818eb1f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.088+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0052d0 con 0x7f36c40039c0 2024-01-16T15:28:05.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.088+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc02e050 con 0x7f36c40039c0 2024-01-16T15:28:05.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.088+0000 7f36e0a00640 1 -- 172.21.15.53:0/1239847657 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc003830 con 0x7f36c40039c0 2024-01-16T15:28:05.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.088+0000 7f36e9200640 1 -- 172.21.15.53:0/1239847657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c40039c0 msgr2=0x7f36c4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.088+0000 7f36e9200640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c40039c0 0x7f36c4003de0 secure :-1 s=READY pgs=2025 cs=0 l=1 rev1=1 crypto rx=0x7f36cc001720 tx=0x7f36c40057f0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.089+0000 7f36e9200640 1 -- 172.21.15.53:0/1239847657 shutdown_connections 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.089+0000 7f36e9200640 1 --2- 172.21.15.53:0/1239847657 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36c40039c0 0x7f36c4003de0 unknown :-1 s=CLOSED pgs=2025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.089+0000 7f36e9200640 1 -- 172.21.15.53:0/1239847657 >> 172.21.15.53:0/1239847657 conn(0x7f36e409c560 msgr2=0x7f36e40d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.089+0000 7f36e9200640 1 -- 172.21.15.53:0/1239847657 shutdown_connections 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.089+0000 7f36e9200640 1 -- 172.21.15.53:0/1239847657 wait complete. 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.090+0000 7f36e9200640 1 Processor -- start 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.091+0000 7f36e9200640 1 -- start start 2024-01-16T15:28:05.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e9200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40039c0 0x7f36e409df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e9200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36e4150730 0x7f36e40a0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e9200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36e40c6d40 0x7f36e40c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e9200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e409e470 con 0x7f36e4150730 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e9200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40c7700 con 0x7f36e40c6d40 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e9200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e40a08b0 con 0x7f36c40039c0 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36e4150730 0x7f36e40a0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36e4150730 0x7f36e40a0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55566/0 (socket says 172.21.15.53:55566) 2024-01-16T15:28:05.094 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 -- 172.21.15.53:0/1732182033 learned_addr learned my addr 172.21.15.53:0/1732182033 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e2800640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36e40c6d40 0x7f36e40c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40039c0 0x7f36e409df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 -- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40039c0 msgr2=0x7f36e409df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40039c0 0x7f36e409df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 -- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36e40c6d40 msgr2=0x7f36e40c71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.092+0000 7f36e1400640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36e40c6d40 0x7f36e40c71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36e1400640 1 -- 172.21.15.53:0/1732182033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e4150bb0 con 0x7f36e4150730 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f36c40039c0 0x7f36e409df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:05.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36e2800640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f36e40c6d40 0x7f36e40c71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:05.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36e1400640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36e4150730 0x7f36e40a0370 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f36d4007a10 tx=0x7f36d400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d400ee50 con 0x7f36e4150730 2024-01-16T15:28:05.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d4013420 con 0x7f36e4150730 2024-01-16T15:28:05.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d40137d0 con 0x7f36e4150730 2024-01-16T15:28:05.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e40a0b30 con 0x7f36e4150730 2024-01-16T15:28:05.096 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.093+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e40b7190 con 0x7f36e4150730 2024-01-16T15:28:05.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.094+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f36e40cfd60 con 0x7f36e4150730 2024-01-16T15:28:05.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.096+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f36d4013930 con 0x7f36e4150730 2024-01-16T15:28:05.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36da000640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36bc06c580 0x7f36bc06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f36d4089340 con 0x7f36e4150730 2024-01-16T15:28:05.098 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36da000640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f36bc074840 0x7f36bc076ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 --> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f36bc077370 con 0x7f36bc074840 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36d4089f60 con 0x7f36e4150730 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36e2800640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f36bc074840 0x7f36bc076ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.097+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36bc06c580 0x7f36bc06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.098+0000 7f36e2800640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f36bc074840 0x7f36bc076ca0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.099+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== osd.0 v2:172.21.15.53:6800/2075393413 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f36bc077370 con 0x7f36bc074840 2024-01-16T15:28:05.099 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.099+0000 7f36e1e00640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36bc06c580 0x7f36bc06ea40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f36cc033000 tx=0x7f36cc009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.113+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 --> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f36e40c9c50 con 0x7f36bc074840 2024-01-16T15:28:05.112 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.113+0000 7f36da000640 1 -- 172.21.15.53:0/1732182033 <== osd.0 v2:172.21.15.53:6800/2075393413 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f36dc004040 con 0x7f36bc074840 2024-01-16T15:28:05.113 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:05.113 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:05.113 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:05.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f36bc074840 msgr2=0x7f36bc076ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:6800/2075393413,v1:172.21.15.53:6801/2075393413] conn(0x7f36bc074840 0x7f36bc076ca0 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36bc06c580 msgr2=0x7f36bc06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f36bc06c580 0x7f36bc06ea40 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f36cc033000 tx=0x7f36cc009030 comp rx=0 tx=0).stop 2024-01-16T15:28:05.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36e4150730 msgr2=0x7f36e40a0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.114 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 --2- 172.21.15.53:0/1732182033 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f36e4150730 0x7f36e40a0370 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7f36d4007a10 tx=0x7f36d400e530 comp rx=0 tx=0).stop 2024-01-16T15:28:05.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e2800640 1 -- 172.21.15.53:0/1732182033 reap_dead start 2024-01-16T15:28:05.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 shutdown_connections 2024-01-16T15:28:05.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 >> 172.21.15.53:0/1732182033 conn(0x7f36e409c560 msgr2=0x7f36e40ceb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:05.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 shutdown_connections 2024-01-16T15:28:05.115 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.114+0000 7f36e9200640 1 -- 172.21.15.53:0/1732182033 wait complete. 2024-01-16T15:28:05.129 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-01-16T15:28:05.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.350+0000 7f0ebd000640 1 Processor -- start 2024-01-16T15:28:05.350 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.351+0000 7f0ebd000640 1 -- start start 2024-01-16T15:28:05.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.352+0000 7f0ebd000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f0eb8071ce0 0x7f0eb80ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.351 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.352+0000 7f0ebd000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80b3490 con 0x7f0eb80adb00 2024-01-16T15:28:05.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.352+0000 7f0ebd000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eb80b3770 con 0x7f0eb8146850 2024-01-16T15:28:05.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.352+0000 7f0ebd000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eb80b3a50 con 0x7f0eb8071ce0 2024-01-16T15:28:05.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.353+0000 7f0eb7e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0eb80adb00 0x7f0eb806c470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57604/0 (socket says 172.21.15.53:57604) 2024-01-16T15:28:05.352 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.353+0000 7f0eb7e00640 1 -- 172.21.15.53:0/2848220951 learned_addr learned my addr 172.21.15.53:0/2848220951 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:05.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.353+0000 7f0eafe00640 1 --2- 172.21.15.53:0/2848220951 >> v2:172.21.15.146:3300/0 conn(0x7f0eb8071ce0 0x7f0eb80ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.353+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247058591 0 0) 0x7f0eb80b3490 con 0x7f0eb80adb00 2024-01-16T15:28:05.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.353+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e98003520 con 0x7f0eb80adb00 2024-01-16T15:28:05.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1729896038 0 0) 0x7f0e98003520 con 0x7f0eb80adb00 2024-01-16T15:28:05.354 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 >> v2:172.21.15.146:3300/0 conn(0x7f0eb8071ce0 msgr2=0x7f0eb80ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 --2- 172.21.15.53:0/2848220951 >> v2:172.21.15.146:3300/0 conn(0x7f0eb8071ce0 0x7f0eb80ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 >> v1:172.21.15.146:6789/0 conn(0x7f0eb8146850 legacy=0x7f0eb80715d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb80b3d30 con 0x7f0eb80adb00 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f0eb00019f0 con 0x7f0eb80adb00 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 >> v1:172.21.15.53:6789/0 conn(0x7f0eb80adb00 legacy=0x7f0eb806c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e98003970 0x7f0e98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e980047c0 0x7f0e98004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.355 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98005180 0x7f0e9800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb80b3a50 con 0x7f0e98003970 2024-01-16T15:28:05.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9800c020 con 0x7f0e98005180 2024-01-16T15:28:05.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9800c1e0 con 0x7f0e980047c0 2024-01-16T15:28:05.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb7400640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e98003970 0x7f0e98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eb7e00640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98005180 0x7f0e9800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.356 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.354+0000 7f0eafe00640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e980047c0 0x7f0e98004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.355+0000 7f0eb7400640 1 -- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e980047c0 msgr2=0x7f0e98004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.355+0000 7f0eb7400640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0e980047c0 0x7f0e98004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.355+0000 7f0eb7400640 1 -- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98005180 msgr2=0x7f0e9800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.355+0000 7f0eb7400640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98005180 0x7f0e9800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.355+0000 7f0eb7e00640 1 -- 172.21.15.53:0/2848220951 reap_dead start 2024-01-16T15:28:05.357 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.355+0000 7f0eb7400640 1 -- 172.21.15.53:0/2848220951 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e98003520 con 0x7f0e98003970 2024-01-16T15:28:05.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.356+0000 7f0eb7400640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e98003970 0x7f0e98003d90 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8001750 tx=0x7f0e980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a62d96395d31ac1 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.358 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.356+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea80052d0 con 0x7f0e98003970 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.356+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea802e050 con 0x7f0e98003970 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.356+0000 7f0eb6a00640 1 -- 172.21.15.53:0/2848220951 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea8003830 con 0x7f0e98003970 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 -- 172.21.15.53:0/2848220951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e98003970 msgr2=0x7f0e98003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e98003970 0x7f0e98003d90 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8001750 tx=0x7f0e980057f0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 -- 172.21.15.53:0/2848220951 shutdown_connections 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 --2- 172.21.15.53:0/2848220951 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0e98003970 0x7f0e98003d90 unknown :-1 s=CLOSED pgs=2028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 -- 172.21.15.53:0/2848220951 >> 172.21.15.53:0/2848220951 conn(0x7f0eb806a3e0 msgr2=0x7f0eb806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:05.359 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 -- 172.21.15.53:0/2848220951 shutdown_connections 2024-01-16T15:28:05.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.357+0000 7f0ebd000640 1 -- 172.21.15.53:0/2848220951 wait complete. 2024-01-16T15:28:05.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.359+0000 7f0ebd000640 1 Processor -- start 2024-01-16T15:28:05.360 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.359+0000 7f0ebd000640 1 -- start start 2024-01-16T15:28:05.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.360+0000 7f0ebd000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98003970 0x7f0eb80c45f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.360+0000 7f0ebd000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 0x7f0eb813a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.360+0000 7f0ebd000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb815a5e0 0x7f0eb81f2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.360+0000 7f0ebd000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb815c350 con 0x7f0eb815c7e0 2024-01-16T15:28:05.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.360+0000 7f0ebd000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb815ba20 con 0x7f0e98003970 2024-01-16T15:28:05.362 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.360+0000 7f0ebd000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eb815bd00 con 0x7f0eb815a5e0 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 0x7f0eb813a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 0x7f0eb813a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55594/0 (socket says 172.21.15.53:55594) 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 -- 172.21.15.53:0/3832237045 learned_addr learned my addr 172.21.15.53:0/3832237045 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eb7e00640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb815a5e0 0x7f0eb81f2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eb7400640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98003970 0x7f0eb80c45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 -- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb815a5e0 msgr2=0x7f0eb81f2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb815a5e0 0x7f0eb81f2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.363 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 -- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98003970 msgr2=0x7f0eb80c45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98003970 0x7f0eb80c45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 -- 172.21.15.53:0/3832237045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb815bfe0 con 0x7f0eb815c7e0 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eb7e00640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb815a5e0 0x7f0eb81f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.361+0000 7f0eafe00640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 0x7f0eb813a7f0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00114c0 tx=0x7f0ea0011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.362+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0015870 con 0x7f0eb815c7e0 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.362+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ea00159d0 con 0x7f0eb815c7e0 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.362+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0015d40 con 0x7f0eb815c7e0 2024-01-16T15:28:05.364 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.362+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eb815aa70 con 0x7f0eb815c7e0 2024-01-16T15:28:05.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.362+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eb80c4db0 con 0x7f0eb815c7e0 2024-01-16T15:28:05.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.363+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eb80b3070 con 0x7f0eb815c7e0 2024-01-16T15:28:05.365 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.364+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0ea0013400 con 0x7f0eb815c7e0 2024-01-16T15:28:05.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.365+0000 7f0eaf400640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e9006c580 0x7f0e9006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.366+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0ea0088e70 con 0x7f0eb815c7e0 2024-01-16T15:28:05.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.366+0000 7f0eb7400640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e9006c580 0x7f0e9006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.366 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.368+0000 7f0eb7400640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e9006c580 0x7f0e9006ea40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8033000 tx=0x7f0ea8009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.369 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.370+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0ea00553b0 con 0x7f0eb815c7e0 2024-01-16T15:28:05.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.563+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f0eb8145fc0 con 0x7f0e9006c580 2024-01-16T15:28:05.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.565+0000 7f0eaf400640 1 -- 172.21.15.53:0/3832237045 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0eb8145fc0 con 0x7f0e9006c580 2024-01-16T15:28:05.564 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 0 to deep-scrub 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.567+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e9006c580 msgr2=0x7f0e9006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e9006c580 0x7f0e9006ea40 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0ea8033000 tx=0x7f0ea8009030 comp rx=0 tx=0).stop 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 msgr2=0x7f0eb813a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 0x7f0eb813a7f0 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f0ea00114c0 tx=0x7f0ea0011980 comp rx=0 tx=0).stop 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 shutdown_connections 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0e9006c580 0x7f0e9006ea40 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0eb815a5e0 0x7f0eb81f2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0eb815c7e0 0x7f0eb813a7f0 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 --2- 172.21.15.53:0/3832237045 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0e98003970 0x7f0eb80c45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 >> 172.21.15.53:0/3832237045 conn(0x7f0eb806a3e0 msgr2=0x7f0eb81602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 shutdown_connections 2024-01-16T15:28:05.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.568+0000 7f0ebd000640 1 -- 172.21.15.53:0/3832237045 wait complete. 2024-01-16T15:28:05.585 INFO:tasks.ceph:Scrubbing osd.1 2024-01-16T15:28:05.585 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:05.803 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.804+0000 7f0736e00640 1 Processor -- start 2024-01-16T15:28:05.804 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.804+0000 7f0736e00640 1 -- start start 2024-01-16T15:28:05.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f0736e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f073013df80 0x7f073015bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f0736e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f073013e360 0x7f07300cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.805 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f0736e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073006c470 con 0x7f073015f9c0 2024-01-16T15:28:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f0736e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073006c750 con 0x7f073013df80 2024-01-16T15:28:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f0736e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073006ca30 con 0x7f073013e360 2024-01-16T15:28:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f072fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f073013df80 0x7f073015bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f072fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f073013df80 0x7f073015bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55596/0 (socket says 172.21.15.53:55596) 2024-01-16T15:28:05.806 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.806+0000 7f072fe00640 1 -- 172.21.15.53:0/3603839826 learned_addr learned my addr 172.21.15.53:0/3603839826 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.807+0000 7f0727e00640 1 --2- 172.21.15.53:0/3603839826 >> v2:172.21.15.146:3300/0 conn(0x7f073013e360 0x7f07300cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.807+0000 7f072fe00640 1 -- 172.21.15.53:0/3603839826 >> v2:172.21.15.146:3300/0 conn(0x7f073013e360 msgr2=0x7f07300cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.807+0000 7f072fe00640 1 --2- 172.21.15.53:0/3603839826 >> v2:172.21.15.146:3300/0 conn(0x7f073013e360 0x7f07300cf290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.807+0000 7f072fe00640 1 -- 172.21.15.53:0/3603839826 >> v1:172.21.15.146:6789/0 conn(0x7f073015f9c0 legacy=0x7f07301408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.807+0000 7f072fe00640 1 -- 172.21.15.53:0/3603839826 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073006cd10 con 0x7f073013df80 2024-01-16T15:28:05.807 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.808+0000 7f072fe00640 1 --2- 172.21.15.53:0/3603839826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013df80 0x7f073015bee0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f073006ded0 tx=0x7f072002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46e43eaeca0eb941 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.808 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.808+0000 7f072f400640 1 -- 172.21.15.53:0/3603839826 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720031870 con 0x7f073013df80 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.809+0000 7f072f400640 1 -- 172.21.15.53:0/3603839826 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07200319d0 con 0x7f073013df80 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.809+0000 7f072f400640 1 -- 172.21.15.53:0/3603839826 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720031d60 con 0x7f073013df80 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.809+0000 7f0736e00640 1 -- 172.21.15.53:0/3603839826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013df80 msgr2=0x7f073015bee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.809+0000 7f0736e00640 1 --2- 172.21.15.53:0/3603839826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013df80 0x7f073015bee0 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f073006ded0 tx=0x7f072002b910 comp rx=0 tx=0).stop 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.810+0000 7f0736e00640 1 -- 172.21.15.53:0/3603839826 shutdown_connections 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.810+0000 7f0736e00640 1 --2- 172.21.15.53:0/3603839826 >> v2:172.21.15.146:3300/0 conn(0x7f073013e360 0x7f07300cf290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.810+0000 7f0736e00640 1 --2- 172.21.15.53:0/3603839826 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013df80 0x7f073015bee0 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.810+0000 7f0736e00640 1 -- 172.21.15.53:0/3603839826 >> 172.21.15.53:0/3603839826 conn(0x7f073006a800 msgr2=0x7f073006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:05.809 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.810+0000 7f0736e00640 1 -- 172.21.15.53:0/3603839826 shutdown_connections 2024-01-16T15:28:05.810 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.810+0000 7f0736e00640 1 -- 172.21.15.53:0/3603839826 wait complete. 2024-01-16T15:28:05.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.812+0000 7f0736e00640 1 Processor -- start 2024-01-16T15:28:05.811 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.812+0000 7f0736e00640 1 -- start start 2024-01-16T15:28:05.812 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.813+0000 7f0736e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f073013df80 0x7f073015dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.813+0000 7f0736e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013e360 0x7f073014fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.813+0000 7f0736e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f073015f9c0 0x7f073014ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0736e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0730151150 con 0x7f073013e360 2024-01-16T15:28:05.813 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0736e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073015e1b0 con 0x7f073013df80 2024-01-16T15:28:05.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0736e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0730153500 con 0x7f073015f9c0 2024-01-16T15:28:05.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013e360 0x7f073014fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013e360 0x7f073014fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55604/0 (socket says 172.21.15.53:55604) 2024-01-16T15:28:05.814 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 -- 172.21.15.53:0/1395213289 learned_addr learned my addr 172.21.15.53:0/1395213289 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:05.815 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0734e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f073015f9c0 0x7f073014ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f072fe00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f073013df80 0x7f073015dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 -- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f073015f9c0 msgr2=0x7f073014ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f073015f9c0 0x7f073014ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 -- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f073013df80 msgr2=0x7f073015dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f073013df80 0x7f073015dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f0727e00640 1 -- 172.21.15.53:0/1395213289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200094d0 con 0x7f073013e360 2024-01-16T15:28:05.816 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.814+0000 7f072fe00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f073013df80 0x7f073015dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.815+0000 7f0727e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013e360 0x7f073014fad0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f071800d7d0 tx=0x7f071800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.815+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718018070 con 0x7f073013e360 2024-01-16T15:28:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.815+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0718004020 con 0x7f073013e360 2024-01-16T15:28:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.815+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07180041b0 con 0x7f073013e360 2024-01-16T15:28:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.815+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073014e0d0 con 0x7f073013e360 2024-01-16T15:28:05.817 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.816+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073014e690 con 0x7f073013e360 2024-01-16T15:28:05.818 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.817+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f073015eb10 con 0x7f073013e360 2024-01-16T15:28:05.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.818+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0718004310 con 0x7f073013e360 2024-01-16T15:28:05.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.819+0000 7f072cc00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f070806c550 0x7f070806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.819+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0718088a50 con 0x7f073013e360 2024-01-16T15:28:05.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.819+0000 7f072cc00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7f0708074810 0x7f0708076c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:05.819 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.819+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 --> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0708077340 con 0x7f0708074810 2024-01-16T15:28:05.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.819+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0718089660 con 0x7f073013e360 2024-01-16T15:28:05.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.819+0000 7f0734e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7f0708074810 0x7f0708076c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.820+0000 7f072fe00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f070806c550 0x7f070806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:05.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.820+0000 7f0734e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7f0708074810 0x7f0708076c70 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.820 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.821+0000 7f072fe00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f070806c550 0x7f070806ea10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0720036000 tx=0x7f072003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:05.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.821+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== osd.1 v2:172.21.15.53:6816/3560880694 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f0708077340 con 0x7f0708074810 2024-01-16T15:28:05.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.836+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 --> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f073015efd0 con 0x7f0708074810 2024-01-16T15:28:05.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.836+0000 7f072cc00640 1 -- 172.21.15.53:0/1395213289 <== osd.1 v2:172.21.15.53:6816/3560880694 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f073015efd0 con 0x7f0708074810 2024-01-16T15:28:05.835 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:05.836 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:05.836 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7f0708074810 msgr2=0x7f0708076c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:6816/3560880694,v1:172.21.15.53:6817/3560880694] conn(0x7f0708074810 0x7f0708076c70 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f070806c550 msgr2=0x7f070806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f070806c550 0x7f070806ea10 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0720036000 tx=0x7f072003c040 comp rx=0 tx=0).stop 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013e360 msgr2=0x7f073014fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 --2- 172.21.15.53:0/1395213289 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f073013e360 0x7f073014fad0 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7f071800d7d0 tx=0x7f071800dc90 comp rx=0 tx=0).stop 2024-01-16T15:28:05.837 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0734e00640 1 -- 172.21.15.53:0/1395213289 reap_dead start 2024-01-16T15:28:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 shutdown_connections 2024-01-16T15:28:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 >> 172.21.15.53:0/1395213289 conn(0x7f073006a800 msgr2=0x7f07300af420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 shutdown_connections 2024-01-16T15:28:05.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:05.837+0000 7f0736e00640 1 -- 172.21.15.53:0/1395213289 wait complete. 2024-01-16T15:28:05.855 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-01-16T15:28:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.080+0000 7f0fd4a00640 1 Processor -- start 2024-01-16T15:28:06.079 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.080+0000 7f0fd4a00640 1 -- start start 2024-01-16T15:28:06.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.081+0000 7f0fd4a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fd015e170 0x7f0fd00cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.081+0000 7f0fd4a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd00700f0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.080 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.081+0000 7f0fd4a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd00ce720 con 0x7f0fd013cda0 2024-01-16T15:28:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.081+0000 7f0fd4a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd00cea00 con 0x7f0fd015e170 2024-01-16T15:28:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcf200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f0fd00cdef0 0x7f0fd0070b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57612/0 (socket says 172.21.15.53:57612) 2024-01-16T15:28:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcde00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0fd015e170 0x7f0fd00cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.081 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcf200640 1 -- 172.21.15.53:0/2775956898 learned_addr learned my addr 172.21.15.53:0/2775956898 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcde00640 1 -- 172.21.15.53:0/2775956898 >> v1:172.21.15.146:6789/0 conn(0x7f0fd013cda0 legacy=0x7f0fd015da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcde00640 1 -- 172.21.15.53:0/2775956898 >> v1:172.21.15.53:6789/0 conn(0x7f0fd00cdef0 legacy=0x7f0fd0070b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcde00640 1 -- 172.21.15.53:0/2775956898 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd0070340 con 0x7f0fd015e170 2024-01-16T15:28:06.082 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.082+0000 7f0fcd400640 1 -- 172.21.15.53:0/2775956898 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452240791 0 0) 0x7f0fd00700f0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.083+0000 7f0fcde00640 1 --2- 172.21.15.53:0/2775956898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd015e170 0x7f0fd00cd9b0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f0fb80097a0 tx=0x7f0fb802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c03c56c271f0c42 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.084+0000 7f0fcd400640 1 -- 172.21.15.53:0/2775956898 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8031830 con 0x7f0fd015e170 2024-01-16T15:28:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.084+0000 7f0fcd400640 1 -- 172.21.15.53:0/2775956898 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8031990 con 0x7f0fd015e170 2024-01-16T15:28:06.083 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.084+0000 7f0fcd400640 1 -- 172.21.15.53:0/2775956898 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8031d20 con 0x7f0fd015e170 2024-01-16T15:28:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.084+0000 7f0fd4a00640 1 -- 172.21.15.53:0/2775956898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd015e170 msgr2=0x7f0fd00cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.084+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/2775956898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd015e170 0x7f0fd00cd9b0 secure :-1 s=READY pgs=2033 cs=0 l=1 rev1=1 crypto rx=0x7f0fb80097a0 tx=0x7f0fb802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.085+0000 7f0fd4a00640 1 -- 172.21.15.53:0/2775956898 shutdown_connections 2024-01-16T15:28:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.085+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/2775956898 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd015e170 0x7f0fd00cd9b0 unknown :-1 s=CLOSED pgs=2033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.084 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.085+0000 7f0fd4a00640 1 -- 172.21.15.53:0/2775956898 >> 172.21.15.53:0/2775956898 conn(0x7f0fd006a3e0 msgr2=0x7f0fd006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.085+0000 7f0fd4a00640 1 -- 172.21.15.53:0/2775956898 shutdown_connections 2024-01-16T15:28:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.085+0000 7f0fd4a00640 1 -- 172.21.15.53:0/2775956898 wait complete. 2024-01-16T15:28:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.086+0000 7f0fd4a00640 1 Processor -- start 2024-01-16T15:28:06.085 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.087+0000 7f0fd4a00640 1 -- start start 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.087+0000 7f0fd4a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 0x7f0fd013bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fd4a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd013cda0 0x7f0fd01416a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fd4a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd015e170 0x7f0fd013f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fd4a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd0141be0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fd4a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd013d580 con 0x7f0fd013cda0 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fd4a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd01f3470 con 0x7f0fd015e170 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 0x7f0fd013bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.087 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 0x7f0fd013bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55624/0 (socket says 172.21.15.53:55624) 2024-01-16T15:28:06.088 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 -- 172.21.15.53:0/4249301404 learned_addr learned my addr 172.21.15.53:0/4249301404 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 -- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd015e170 msgr2=0x7f0fd013f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd015e170 0x7f0fd013f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 -- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd013cda0 msgr2=0x7f0fd01416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd013cda0 0x7f0fd01416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.088+0000 7f0fce800640 1 -- 172.21.15.53:0/4249301404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8009400 con 0x7f0fd00cdef0 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.089+0000 7f0fce800640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 0x7f0fd013bb30 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c450 tx=0x7f0fc000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.089+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00052d0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.089 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.089+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc0005430 con 0x7f0fd00cdef0 2024-01-16T15:28:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.089+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd01f3750 con 0x7f0fd00cdef0 2024-01-16T15:28:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.089+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0012870 con 0x7f0fd00cdef0 2024-01-16T15:28:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.089+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd01f3d10 con 0x7f0fd00cdef0 2024-01-16T15:28:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.090+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd013d6f0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.090 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.091+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f0fc00129d0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.092+0000 7f0fc6a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa806c550 0x7f0fa806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.091 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.092+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f0fc00886b0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.092 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.093+0000 7f0fcde00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa806c550 0x7f0fa806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.093 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.094+0000 7f0fcde00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa806c550 0x7f0fa806ea10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8036000 tx=0x7f0fb8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.095 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.097+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0fd013d6f0 con 0x7f0fd00cdef0 2024-01-16T15:28:06.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.290+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0fd00b0960 con 0x7f0fa806c550 2024-01-16T15:28:06.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.292+0000 7f0fc6a00640 1 -- 172.21.15.53:0/4249301404 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0fd00b0960 con 0x7f0fa806c550 2024-01-16T15:28:06.291 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 1 to deep-scrub 2024-01-16T15:28:06.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa806c550 msgr2=0x7f0fa806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa806c550 0x7f0fa806ea10 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0fb8036000 tx=0x7f0fb8030000 comp rx=0 tx=0).stop 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 msgr2=0x7f0fd013bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 0x7f0fd013bb30 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000c450 tx=0x7f0fc000c910 comp rx=0 tx=0).stop 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 shutdown_connections 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f0fa806c550 0x7f0fa806ea10 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0fd015e170 0x7f0fd013f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f0fd013cda0 0x7f0fd01416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.294 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 --2- 172.21.15.53:0/4249301404 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0fd00cdef0 0x7f0fd013bb30 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.294+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 >> 172.21.15.53:0/4249301404 conn(0x7f0fd006a3e0 msgr2=0x7f0fd00b0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.295+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 shutdown_connections 2024-01-16T15:28:06.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.295+0000 7f0fd4a00640 1 -- 172.21.15.53:0/4249301404 wait complete. 2024-01-16T15:28:06.313 INFO:tasks.ceph:Scrubbing osd.2 2024-01-16T15:28:06.314 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:06.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.547+0000 7f558e000640 1 Processor -- start 2024-01-16T15:28:06.546 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.547+0000 7f558e000640 1 -- start start 2024-01-16T15:28:06.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.549+0000 7f558e000640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55880cc990 0x7f558806d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.549+0000 7f558e000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5588070e50 con 0x7f5588143c40 2024-01-16T15:28:06.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.549+0000 7f558e000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5588144020 con 0x7f55880cc500 2024-01-16T15:28:06.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.549+0000 7f558e000640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55880723c0 con 0x7f55880cc990 2024-01-16T15:28:06.548 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.549+0000 7f558cc00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f5588143c40 0x7f5588071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57620/0 (socket says 172.21.15.53:57620) 2024-01-16T15:28:06.549 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.549+0000 7f558cc00640 1 -- 172.21.15.53:0/824124729 learned_addr learned my addr 172.21.15.53:0/824124729 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:06.550 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f558d600640 1 --2- 172.21.15.53:0/824124729 >> v2:172.21.15.146:3300/0 conn(0x7f55880cc990 0x7f558806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3165645048 0 0) 0x7f5588070e50 con 0x7f5588143c40 2024-01-16T15:28:06.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5568003520 con 0x7f5588143c40 2024-01-16T15:28:06.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3660007899 0 0) 0x7f5568003520 con 0x7f5588143c40 2024-01-16T15:28:06.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 >> v2:172.21.15.146:3300/0 conn(0x7f55880cc990 msgr2=0x7f558806d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:06.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 --2- 172.21.15.53:0/824124729 >> v2:172.21.15.146:3300/0 conn(0x7f55880cc990 0x7f558806d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.551 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 >> v1:172.21.15.146:6789/0 conn(0x7f55880cc500 legacy=0x7f558806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5588071e00 con 0x7f5588143c40 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f5578001670 con 0x7f5588143c40 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 >> v1:172.21.15.53:6789/0 conn(0x7f5588143c40 legacy=0x7f5588071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5568003970 0x7f5568003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55680047c0 0x7f5568004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5568005180 0x7f556800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.552 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880723c0 con 0x7f5568003970 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556800c120 con 0x7f55680047c0 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.550+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556800c2e0 con 0x7f5568005180 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558cc00640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5568003970 0x7f5568003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f5587e00640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55680047c0 0x7f5568004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558d600640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5568005180 0x7f556800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558cc00640 1 -- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5568005180 msgr2=0x7f556800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558cc00640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5568005180 0x7f556800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.553 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558cc00640 1 -- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55680047c0 msgr2=0x7f5568004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:06.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558cc00640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55680047c0 0x7f5568004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558cc00640 1 -- 172.21.15.53:0/824124729 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5568003520 con 0x7f5568003970 2024-01-16T15:28:06.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558d600640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f5568005180 0x7f556800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:06.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f5587e00640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55680047c0 0x7f5568004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-01-16T15:28:06.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.551+0000 7f558d600640 1 -- 172.21.15.53:0/824124729 reap_dead start 2024-01-16T15:28:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.552+0000 7f558cc00640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5568003970 0x7f5568003d90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f557800c240 tx=0x7f55680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a96af989369ff792 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5578005ad0 con 0x7f5568003970 2024-01-16T15:28:06.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5578005c30 con 0x7f5568003970 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f5587400640 1 -- 172.21.15.53:0/824124729 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f557802ca00 con 0x7f5568003970 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f558e000640 1 -- 172.21.15.53:0/824124729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5568003970 msgr2=0x7f5568003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f558e000640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5568003970 0x7f5568003d90 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f557800c240 tx=0x7f55680057f0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f558e000640 1 -- 172.21.15.53:0/824124729 shutdown_connections 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f558e000640 1 --2- 172.21.15.53:0/824124729 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f5568003970 0x7f5568003d90 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.553+0000 7f558e000640 1 -- 172.21.15.53:0/824124729 >> 172.21.15.53:0/824124729 conn(0x7f558806a800 msgr2=0x7f558806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.554+0000 7f558e000640 1 -- 172.21.15.53:0/824124729 shutdown_connections 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.554+0000 7f558e000640 1 -- 172.21.15.53:0/824124729 wait complete. 2024-01-16T15:28:06.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.555+0000 7f558e000640 1 Processor -- start 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.556+0000 7f558e000640 1 -- start start 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558e000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5568003970 0x7f5588143120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55880a9fd0 0x7f558815a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558e000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f558813d390 0x7f55880ac380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558e000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f558815ad10 con 0x7f55880a9fd0 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558e000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55880a8f70 con 0x7f5568003970 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558e000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55880a9210 con 0x7f558813d390 2024-01-16T15:28:06.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55880a9fd0 0x7f558815a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55880a9fd0 0x7f558815a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55642/0 (socket says 172.21.15.53:55642) 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 -- 172.21.15.53:0/1189879691 learned_addr learned my addr 172.21.15.53:0/1189879691 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f558cc00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5568003970 0x7f5588143120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 -- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f558813d390 msgr2=0x7f55880ac380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f558813d390 0x7f55880ac380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 -- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5568003970 msgr2=0x7f5588143120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f5568003970 0x7f5588143120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.557+0000 7f5587e00640 1 -- 172.21.15.53:0/1189879691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55880a94f0 con 0x7f55880a9fd0 2024-01-16T15:28:06.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.558+0000 7f5587e00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55880a9fd0 0x7f558815a7d0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f5570000c00 tx=0x7f557000e870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.558+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55700052e0 con 0x7f55880a9fd0 2024-01-16T15:28:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.558+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5570005440 con 0x7f55880a9fd0 2024-01-16T15:28:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.558+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5570003810 con 0x7f55880a9fd0 2024-01-16T15:28:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.558+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55880a97d0 con 0x7f55880a9fd0 2024-01-16T15:28:06.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.558+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f558813dab0 con 0x7f55880a9fd0 2024-01-16T15:28:06.562 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.559+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f558806cc40 con 0x7f55880a9fd0 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.560+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f5570003970 con 0x7f55880a9fd0 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.561+0000 7f5584c00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f556006c550 0x7f556006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.562+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f5570088200 con 0x7f55880a9fd0 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.562+0000 7f5584c00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f5560074810 0x7f5560076c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.562+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 --> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5560077340 con 0x7f5560074810 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.562+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5570088a90 con 0x7f55880a9fd0 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.562+0000 7f558cc00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f556006c550 0x7f556006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.562+0000 7f558d600640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f5560074810 0x7f5560076c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.563+0000 7f558d600640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f5560074810 0x7f5560076c70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.563+0000 7f558cc00640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f556006c550 0x7f556006ea10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5578034000 tx=0x7f5578009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.564+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== osd.2 v2:172.21.15.53:6824/16573798 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f5560077340 con 0x7f5560074810 2024-01-16T15:28:06.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.578+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 --> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f558813dc40 con 0x7f5560074810 2024-01-16T15:28:06.577 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.578+0000 7f5584c00640 1 -- 172.21.15.53:0/1189879691 <== osd.2 v2:172.21.15.53:6824/16573798 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f558813dc40 con 0x7f5560074810 2024-01-16T15:28:06.578 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:06.578 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:06.578 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f5560074810 msgr2=0x7f5560076c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:6824/16573798,v1:172.21.15.53:6825/16573798] conn(0x7f5560074810 0x7f5560076c70 crc :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.579 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f556006c550 msgr2=0x7f556006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f556006c550 0x7f556006ea10 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f5578034000 tx=0x7f5578009030 comp rx=0 tx=0).stop 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55880a9fd0 msgr2=0x7f558815a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 --2- 172.21.15.53:0/1189879691 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55880a9fd0 0x7f558815a7d0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f5570000c00 tx=0x7f557000e870 comp rx=0 tx=0).stop 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558d600640 1 -- 172.21.15.53:0/1189879691 reap_dead start 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 shutdown_connections 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 >> 172.21.15.53:0/1189879691 conn(0x7f558806a800 msgr2=0x7f55880b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 shutdown_connections 2024-01-16T15:28:06.580 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.579+0000 7f558e000640 1 -- 172.21.15.53:0/1189879691 wait complete. 2024-01-16T15:28:06.598 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-01-16T15:28:06.821 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.822+0000 7fb5d2e00640 1 Processor -- start 2024-01-16T15:28:06.822 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.823+0000 7fb5d2e00640 1 -- start start 2024-01-16T15:28:06.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5d2e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb5cc0cf9a0 0x7fb5cc166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5d2e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc15c5f0 con 0x7fb5cc13e360 2024-01-16T15:28:06.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5d2e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5cc06c470 con 0x7fb5cc13df80 2024-01-16T15:28:06.823 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5d2e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5cc06c750 con 0x7fb5cc0cf9a0 2024-01-16T15:28:06.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5d0e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb5cc0cf9a0 0x7fb5cc166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5c3200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb5cc13e360 0x7fb5cc0cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57624/0 (socket says 172.21.15.53:57624) 2024-01-16T15:28:06.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.824+0000 7fb5c3200640 1 -- 172.21.15.53:0/3454111668 learned_addr learned my addr 172.21.15.53:0/3454111668 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:06.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.825+0000 7fb5d0e00640 1 -- 172.21.15.53:0/3454111668 >> v1:172.21.15.146:6789/0 conn(0x7fb5cc13df80 legacy=0x7fb5cc15bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.824 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.825+0000 7fb5d0e00640 1 -- 172.21.15.53:0/3454111668 >> v1:172.21.15.53:6789/0 conn(0x7fb5cc13e360 legacy=0x7fb5cc0cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.825+0000 7fb5d0e00640 1 -- 172.21.15.53:0/3454111668 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc06ca30 con 0x7fb5cc0cf9a0 2024-01-16T15:28:06.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.826+0000 7fb5d0e00640 1 --2- 172.21.15.53:0/3454111668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc0cf9a0 0x7fb5cc166030 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009d90 tx=0x7fb5c402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2075ce01aab32bc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5cb400640 1 -- 172.21.15.53:0/3454111668 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c402cc60 con 0x7fb5cc0cf9a0 2024-01-16T15:28:06.825 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5cb400640 1 -- 172.21.15.53:0/3454111668 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c402cdc0 con 0x7fb5cc0cf9a0 2024-01-16T15:28:06.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5cb400640 1 -- 172.21.15.53:0/3454111668 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c4034660 con 0x7fb5cc0cf9a0 2024-01-16T15:28:06.826 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 -- 172.21.15.53:0/3454111668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc0cf9a0 msgr2=0x7fb5cc166030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/3454111668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc0cf9a0 0x7fb5cc166030 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009d90 tx=0x7fb5c402beb0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 -- 172.21.15.53:0/3454111668 shutdown_connections 2024-01-16T15:28:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/3454111668 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc0cf9a0 0x7fb5cc166030 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 -- 172.21.15.53:0/3454111668 >> 172.21.15.53:0/3454111668 conn(0x7fb5cc06a800 msgr2=0x7fb5cc06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 -- 172.21.15.53:0/3454111668 shutdown_connections 2024-01-16T15:28:06.827 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.827+0000 7fb5d2e00640 1 -- 172.21.15.53:0/3454111668 wait complete. 2024-01-16T15:28:06.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.829+0000 7fb5d2e00640 1 Processor -- start 2024-01-16T15:28:06.828 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.829+0000 7fb5d2e00640 1 -- start start 2024-01-16T15:28:06.829 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.830+0000 7fb5d2e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5cc0cf9a0 0x7fb5cc1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.830+0000 7fb5d2e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5cc13df80 0x7fb5cc0afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.830+0000 7fb5d2e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 0x7fb5cc13d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.830+0000 7fb5d2e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc152e90 con 0x7fb5cc13e360 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.830+0000 7fb5d2e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc1f5190 con 0x7fb5cc13df80 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.830+0000 7fb5d2e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5cc1f5470 con 0x7fb5cc0cf9a0 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 0x7fb5cc13d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 0x7fb5cc13d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55662/0 (socket says 172.21.15.53:55662) 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 -- 172.21.15.53:0/1089904616 learned_addr learned my addr 172.21.15.53:0/1089904616 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:06.830 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5c3200640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5cc13df80 0x7fb5cc0afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.831 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5cbe00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5cc0cf9a0 0x7fb5cc1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 -- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5cc0cf9a0 msgr2=0x7fb5cc1f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5cc0cf9a0 0x7fb5cc1f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 -- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5cc13df80 msgr2=0x7fb5cc0afde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:06.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5cc13df80 0x7fb5cc0afde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:06.832 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.831+0000 7fb5d0e00640 1 -- 172.21.15.53:0/1089904616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c4009a70 con 0x7fb5cc13e360 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.832+0000 7fb5d0e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 0x7fb5cc13d180 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009d90 tx=0x7fb5c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.832+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c403e610 con 0x7fb5cc13e360 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.832+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c403e790 con 0x7fb5cc13e360 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.832+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c403eb40 con 0x7fb5cc13e360 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.832+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5cc1f5750 con 0x7fb5cc13e360 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.832+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5cc1f5d10 con 0x7fb5cc13e360 2024-01-16T15:28:06.833 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.833+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5cc15c410 con 0x7fb5cc13e360 2024-01-16T15:28:06.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.834+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb5c403b050 con 0x7fb5cc13e360 2024-01-16T15:28:06.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.835+0000 7fb5c8c00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a406c550 0x7fb5a406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:06.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.835+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb5c40aaf90 con 0x7fb5cc13e360 2024-01-16T15:28:06.834 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.836+0000 7fb5cbe00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a406c550 0x7fb5a406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:06.835 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.837+0000 7fb5cbe00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a406c550 0x7fb5a406ea10 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc007fd0 tx=0x7fb5bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:06.838 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:06.839+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fb5c4077500 con 0x7fb5cc13e360 2024-01-16T15:28:07.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.033+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5cc06cff0 con 0x7fb5a406c550 2024-01-16T15:28:07.034 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.034+0000 7fb5c8c00640 1 -- 172.21.15.53:0/1089904616 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5bc0050d0 con 0x7fb5a406c550 2024-01-16T15:28:07.035 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 2 to deep-scrub 2024-01-16T15:28:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a406c550 msgr2=0x7fb5a406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a406c550 0x7fb5a406ea10 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc007fd0 tx=0x7fb5bc009040 comp rx=0 tx=0).stop 2024-01-16T15:28:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 msgr2=0x7fb5cc13d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 0x7fb5cc13d180 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fb5c4009d90 tx=0x7fb5c4004a40 comp rx=0 tx=0).stop 2024-01-16T15:28:07.038 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 shutdown_connections 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb5a406c550 0x7fb5a406ea10 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb5cc13e360 0x7fb5cc13d180 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb5cc13df80 0x7fb5cc0afde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 --2- 172.21.15.53:0/1089904616 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb5cc0cf9a0 0x7fb5cc1f6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 >> 172.21.15.53:0/1089904616 conn(0x7fb5cc06a800 msgr2=0x7fb5cc0ce330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 shutdown_connections 2024-01-16T15:28:07.039 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.038+0000 7fb5d2e00640 1 -- 172.21.15.53:0/1089904616 wait complete. 2024-01-16T15:28:07.056 INFO:tasks.ceph:Scrubbing osd.3 2024-01-16T15:28:07.057 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:07.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.277+0000 7fe6efe00640 1 Processor -- start 2024-01-16T15:28:07.277 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.277+0000 7fe6efe00640 1 -- start start 2024-01-16T15:28:07.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f0143c40 0x7fe6f0071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6efe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe6f00717d0 0x7fe6f00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6efe00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f0070e50 con 0x7fe6f00cca40 2024-01-16T15:28:07.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6efe00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f00723c0 con 0x7fe6f00717d0 2024-01-16T15:28:07.280 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f00cce70 con 0x7fe6f0143c40 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6ee000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe6f00717d0 0x7fe6f00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6ee000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fe6f00717d0 0x7fe6f00cc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55666/0 (socket says 172.21.15.53:55666) 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6ee000640 1 -- 172.21.15.53:0/3252656900 learned_addr learned my addr 172.21.15.53:0/3252656900 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.279+0000 7fe6eea00640 1 --2- 172.21.15.53:0/3252656900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f0143c40 0x7fe6f0071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.280+0000 7fe6ee000640 1 -- 172.21.15.53:0/3252656900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f0143c40 msgr2=0x7fe6f0071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.280+0000 7fe6ee000640 1 --2- 172.21.15.53:0/3252656900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f0143c40 0x7fe6f0071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.280+0000 7fe6ee000640 1 -- 172.21.15.53:0/3252656900 >> v1:172.21.15.146:6789/0 conn(0x7fe6f00cca40 legacy=0x7fe6f015eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:07.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.280+0000 7fe6ee000640 1 -- 172.21.15.53:0/3252656900 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f006e170 con 0x7fe6f00717d0 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.281+0000 7fe6ee000640 1 --2- 172.21.15.53:0/3252656900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f00cc500 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009870 tx=0x7fe6d802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40cf178fb4ec870e server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.282+0000 7fe6ed600640 1 -- 172.21.15.53:0/3252656900 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80317c0 con 0x7fe6f00717d0 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.282+0000 7fe6ed600640 1 -- 172.21.15.53:0/3252656900 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8031920 con 0x7fe6f00717d0 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.282+0000 7fe6ed600640 1 -- 172.21.15.53:0/3252656900 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8031cb0 con 0x7fe6f00717d0 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.282+0000 7fe6efe00640 1 -- 172.21.15.53:0/3252656900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 msgr2=0x7fe6f00cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.282+0000 7fe6efe00640 1 --2- 172.21.15.53:0/3252656900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f00cc500 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8009870 tx=0x7fe6d802b860 comp rx=0 tx=0).stop 2024-01-16T15:28:07.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.283+0000 7fe6efe00640 1 -- 172.21.15.53:0/3252656900 shutdown_connections 2024-01-16T15:28:07.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.283+0000 7fe6efe00640 1 --2- 172.21.15.53:0/3252656900 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f00cc500 unknown :-1 s=CLOSED pgs=2041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.283+0000 7fe6efe00640 1 --2- 172.21.15.53:0/3252656900 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f0143c40 0x7fe6f0071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.283+0000 7fe6efe00640 1 -- 172.21.15.53:0/3252656900 >> 172.21.15.53:0/3252656900 conn(0x7fe6f006a800 msgr2=0x7fe6f006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:07.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.283+0000 7fe6efe00640 1 -- 172.21.15.53:0/3252656900 shutdown_connections 2024-01-16T15:28:07.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.283+0000 7fe6efe00640 1 -- 172.21.15.53:0/3252656900 wait complete. 2024-01-16T15:28:07.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.285+0000 7fe6efe00640 1 Processor -- start 2024-01-16T15:28:07.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.285+0000 7fe6efe00640 1 -- start start 2024-01-16T15:28:07.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f013d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.286 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f00cca40 0x7fe6f0138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6efe00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6f0143c40 0x7fe6f013ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6efe00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0139480 con 0x7fe6f00717d0 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6efe00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f013da10 con 0x7fe6f0143c40 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6efe00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f0158b90 con 0x7fe6f00cca40 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f013d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6eea00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f013d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55670/0 (socket says 172.21.15.53:55670) 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6eea00640 1 -- 172.21.15.53:0/3490006071 learned_addr learned my addr 172.21.15.53:0/3490006071 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6ee000640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f00cca40 0x7fe6f0138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6eea00640 1 -- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f00cca40 msgr2=0x7fe6f0138f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.286+0000 7fe6ef400640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6f0143c40 0x7fe6f013ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6eea00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f00cca40 0x7fe6f0138f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6eea00640 1 -- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6f0143c40 msgr2=0x7fe6f013ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6eea00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fe6f0143c40 0x7fe6f013ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6eea00640 1 -- 172.21.15.53:0/3490006071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d80094d0 con 0x7fe6f00717d0 2024-01-16T15:28:07.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6ee000640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fe6f00cca40 0x7fe6f0138f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:07.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6eea00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f013d3a0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000d5f0 tx=0x7fe6e000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00125f0 con 0x7fe6f00717d0 2024-01-16T15:28:07.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e0012750 con 0x7fe6f00717d0 2024-01-16T15:28:07.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0012b00 con 0x7fe6f00717d0 2024-01-16T15:28:07.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f0158e70 con 0x7fe6f00717d0 2024-01-16T15:28:07.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.287+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f0159430 con 0x7fe6f00717d0 2024-01-16T15:28:07.290 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.288+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6f00720e0 con 0x7fe6f00717d0 2024-01-16T15:28:07.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.290+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fe6e0012c60 con 0x7fe6f00717d0 2024-01-16T15:28:07.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.291+0000 7fe6e6a00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6c806c550 0x7fe6c806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.291+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fe6e0089000 con 0x7fe6f00717d0 2024-01-16T15:28:07.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.291+0000 7fe6e6a00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7fe6c8074810 0x7fe6c8076c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.291+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 --> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6c8077340 con 0x7fe6c8074810 2024-01-16T15:28:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.291+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6e0089c10 con 0x7fe6f00717d0 2024-01-16T15:28:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.292+0000 7fe6ef400640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7fe6c8074810 0x7fe6c8076c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.292+0000 7fe6ee000640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6c806c550 0x7fe6c806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.292+0000 7fe6ef400640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7fe6c8074810 0x7fe6c8076c70 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.293+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== osd.3 v2:172.21.15.53:6808/2110883972 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fe6c8077340 con 0x7fe6c8074810 2024-01-16T15:28:07.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.293+0000 7fe6ee000640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6c806c550 0x7fe6c806ea10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8036000 tx=0x7fe6d803d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.307 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.308+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 --> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe6f0071e00 con 0x7fe6c8074810 2024-01-16T15:28:07.308 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.308+0000 7fe6e6a00640 1 -- 172.21.15.53:0/3490006071 <== osd.3 v2:172.21.15.53:6808/2110883972 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fe6e8004070 con 0x7fe6c8074810 2024-01-16T15:28:07.308 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:07.309 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:07.309 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:07.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7fe6c8074810 msgr2=0x7fe6c8076c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:6808/2110883972,v1:172.21.15.53:6809/2110883972] conn(0x7fe6c8074810 0x7fe6c8076c70 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6c806c550 msgr2=0x7fe6c806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fe6c806c550 0x7fe6c806ea10 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7fe6d8036000 tx=0x7fe6d803d040 comp rx=0 tx=0).stop 2024-01-16T15:28:07.310 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 msgr2=0x7fe6f013d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 --2- 172.21.15.53:0/3490006071 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fe6f00717d0 0x7fe6f013d3a0 secure :-1 s=READY pgs=2042 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000d5f0 tx=0x7fe6e000dab0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6ef400640 1 -- 172.21.15.53:0/3490006071 reap_dead start 2024-01-16T15:28:07.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 shutdown_connections 2024-01-16T15:28:07.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 >> 172.21.15.53:0/3490006071 conn(0x7fe6f006a800 msgr2=0x7fe6f00af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:07.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 shutdown_connections 2024-01-16T15:28:07.311 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.309+0000 7fe6efe00640 1 -- 172.21.15.53:0/3490006071 wait complete. 2024-01-16T15:28:07.329 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-01-16T15:28:07.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.553+0000 7f5367e00640 1 Processor -- start 2024-01-16T15:28:07.554 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.554+0000 7f5367e00640 1 -- start start 2024-01-16T15:28:07.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.555+0000 7f5367e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f536813cda0 0x7f536815da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.555+0000 7f5367e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536815dfa0 0x7f53680cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.556 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.555+0000 7f5367e00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5368071b50 con 0x7f53680cdef0 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.555+0000 7f5367e00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53680700f0 con 0x7f536813cda0 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.555+0000 7f5367e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536815e3e0 con 0x7f536815dfa0 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f5366a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f536813cda0 0x7f536815da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f5366a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f536813cda0 0x7f536815da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55684/0 (socket says 172.21.15.53:55684) 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f5366a00640 1 -- 172.21.15.53:0/2969196528 learned_addr learned my addr 172.21.15.53:0/2969196528 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f535de00640 1 --2- 172.21.15.53:0/2969196528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536815dfa0 0x7f53680cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f5366a00640 1 -- 172.21.15.53:0/2969196528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536815dfa0 msgr2=0x7f53680cd9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:07.557 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f5366a00640 1 --2- 172.21.15.53:0/2969196528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536815dfa0 0x7f53680cd9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.556+0000 7f5366a00640 1 -- 172.21.15.53:0/2969196528 >> v1:172.21.15.146:6789/0 conn(0x7f53680cdef0 legacy=0x7f536815a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:07.558 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.557+0000 7f5366a00640 1 -- 172.21.15.53:0/2969196528 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53680ce720 con 0x7f536813cda0 2024-01-16T15:28:07.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.558+0000 7f5366a00640 1 --2- 172.21.15.53:0/2969196528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536813cda0 0x7f536815da60 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f5358009870 tx=0x7f535802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e559ea854f0396d6 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.558+0000 7f5366000640 1 -- 172.21.15.53:0/2969196528 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53580317c0 con 0x7f536813cda0 2024-01-16T15:28:07.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.558+0000 7f5366000640 1 -- 172.21.15.53:0/2969196528 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5358031920 con 0x7f536813cda0 2024-01-16T15:28:07.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.558+0000 7f5366000640 1 -- 172.21.15.53:0/2969196528 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358031cb0 con 0x7f536813cda0 2024-01-16T15:28:07.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 -- 172.21.15.53:0/2969196528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536813cda0 msgr2=0x7f536815da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.559 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 --2- 172.21.15.53:0/2969196528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536813cda0 0x7f536815da60 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f5358009870 tx=0x7f535802b860 comp rx=0 tx=0).stop 2024-01-16T15:28:07.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 -- 172.21.15.53:0/2969196528 shutdown_connections 2024-01-16T15:28:07.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 --2- 172.21.15.53:0/2969196528 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536815dfa0 0x7f53680cd9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 --2- 172.21.15.53:0/2969196528 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f536813cda0 0x7f536815da60 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 -- 172.21.15.53:0/2969196528 >> 172.21.15.53:0/2969196528 conn(0x7f536806a3e0 msgr2=0x7f536806a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:07.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 -- 172.21.15.53:0/2969196528 shutdown_connections 2024-01-16T15:28:07.560 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.559+0000 7f5367e00640 1 -- 172.21.15.53:0/2969196528 wait complete. 2024-01-16T15:28:07.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.561+0000 7f5367e00640 1 Processor -- start 2024-01-16T15:28:07.561 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.561+0000 7f5367e00640 1 -- start start 2024-01-16T15:28:07.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5367e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 0x7f5368152bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5367e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536813cda0 0x7f5368138b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5367e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f536815dfa0 0x7f5368153430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5367e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53681390d0 con 0x7f53680cdef0 2024-01-16T15:28:07.563 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5367e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5368151b20 con 0x7f536815dfa0 2024-01-16T15:28:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5367e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5368151e00 con 0x7f536813cda0 2024-01-16T15:28:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5366a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 0x7f5368152bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5366a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 0x7f5368152bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55700/0 (socket says 172.21.15.53:55700) 2024-01-16T15:28:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f5366a00640 1 -- 172.21.15.53:0/4102932321 learned_addr learned my addr 172.21.15.53:0/4102932321 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.562+0000 7f535de00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536813cda0 0x7f5368138b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.564 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5367400640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f536815dfa0 0x7f5368153430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5366a00640 1 -- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536813cda0 msgr2=0x7f5368138b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5366a00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536813cda0 0x7f5368138b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5366a00640 1 -- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f536815dfa0 msgr2=0x7f5368153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5366a00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f536815dfa0 0x7f5368153430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5366a00640 1 -- 172.21.15.53:0/4102932321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53580094d0 con 0x7f53680cdef0 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5367400640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f536815dfa0 0x7f5368153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:07.565 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f5366a00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 0x7f5368152bc0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f5358009870 tx=0x7f53580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.563+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358032860 con 0x7f53680cdef0 2024-01-16T15:28:07.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.564+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53580329c0 con 0x7f53680cdef0 2024-01-16T15:28:07.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.564+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5358032d70 con 0x7f53680cdef0 2024-01-16T15:28:07.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.564+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53681520e0 con 0x7f53680cdef0 2024-01-16T15:28:07.567 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.564+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53681526a0 con 0x7f53680cdef0 2024-01-16T15:28:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.565+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5368067450 con 0x7f53680cdef0 2024-01-16T15:28:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.566+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f535804d3f0 con 0x7f53680cdef0 2024-01-16T15:28:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.567+0000 7f535f400640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f534006c5a0 0x7f534006ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:07.568 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.567+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f53580aa700 con 0x7f53680cdef0 2024-01-16T15:28:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.568+0000 7f535de00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f534006c5a0 0x7f534006ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:07.569 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.569+0000 7f535de00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f534006c5a0 0x7f534006ea60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f53680b3b20 tx=0x7f53500039b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:07.571 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.572+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f5358076fc0 con 0x7f53680cdef0 2024-01-16T15:28:07.768 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.767+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5368070620 con 0x7f534006c5a0 2024-01-16T15:28:07.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.769+0000 7f535f400640 1 -- 172.21.15.53:0/4102932321 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f53500096f0 con 0x7f534006c5a0 2024-01-16T15:28:07.769 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 3 to deep-scrub 2024-01-16T15:28:07.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.771+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f534006c5a0 msgr2=0x7f534006ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.771+0000 7f5367e00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f534006c5a0 0x7f534006ea60 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f53680b3b20 tx=0x7f53500039b0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.771+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 msgr2=0x7f5368152bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:07.772 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.771+0000 7f5367e00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 0x7f5368152bc0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7f5358009870 tx=0x7f53580046c0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 shutdown_connections 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f534006c5a0 0x7f534006ea60 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f536815dfa0 0x7f5368153430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f536813cda0 0x7f5368138b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 --2- 172.21.15.53:0/4102932321 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f53680cdef0 0x7f5368152bc0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 >> 172.21.15.53:0/4102932321 conn(0x7f536806a3e0 msgr2=0x7f536806cd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:07.773 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 shutdown_connections 2024-01-16T15:28:07.774 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:07.772+0000 7f5367e00640 1 -- 172.21.15.53:0/4102932321 wait complete. 2024-01-16T15:28:07.793 INFO:tasks.ceph:Scrubbing osd.4 2024-01-16T15:28:07.793 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:08.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.009+0000 7f63cba00640 1 Processor -- start 2024-01-16T15:28:08.009 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.009+0000 7f63cba00640 1 -- start start 2024-01-16T15:28:08.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63cba00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63c415fa90 0x7f63c40cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63cba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 0x7f63c40d15b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.011 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63cba00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f63c40b48a0 0x7f63c40d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63cba00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c41613e0 con 0x7f63c415fa90 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63cba00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c4161630 con 0x7f63c40b48a0 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63cba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c4161910 con 0x7f63c40b4460 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63c415fa90 0x7f63c40cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f63c415fa90 0x7f63c40cde90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55704/0 (socket says 172.21.15.53:55704) 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 -- 172.21.15.53:0/980154799 learned_addr learned my addr 172.21.15.53:0/980154799 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:08.012 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 -- 172.21.15.53:0/980154799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 msgr2=0x7f63c40d15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:08.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 --2- 172.21.15.53:0/980154799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 0x7f63c40d15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 -- 172.21.15.53:0/980154799 >> v2:172.21.15.146:3300/0 conn(0x7f63c40b48a0 msgr2=0x7f63c40d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:08.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 --2- 172.21.15.53:0/980154799 >> v2:172.21.15.146:3300/0 conn(0x7f63c40b48a0 0x7f63c40d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.013 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.011+0000 7f63c9000640 1 -- 172.21.15.53:0/980154799 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4161bf0 con 0x7f63c415fa90 2024-01-16T15:28:08.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.013+0000 7f63c9000640 1 --2- 172.21.15.53:0/980154799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cde90 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f63c40d1350 tx=0x7f63b402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f62af63c246faffc server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.013+0000 7f63c3400640 1 -- 172.21.15.53:0/980154799 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4030860 con 0x7f63c415fa90 2024-01-16T15:28:08.014 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.013+0000 7f63c3400640 1 -- 172.21.15.53:0/980154799 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b40309c0 con 0x7f63c415fa90 2024-01-16T15:28:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.013+0000 7f63c3400640 1 -- 172.21.15.53:0/980154799 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4030d50 con 0x7f63c415fa90 2024-01-16T15:28:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 -- 172.21.15.53:0/980154799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 msgr2=0x7f63c40cde90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 --2- 172.21.15.53:0/980154799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cde90 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f63c40d1350 tx=0x7f63b402b9a0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 -- 172.21.15.53:0/980154799 shutdown_connections 2024-01-16T15:28:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 --2- 172.21.15.53:0/980154799 >> v2:172.21.15.146:3300/0 conn(0x7f63c40b48a0 0x7f63c40d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.015 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 --2- 172.21.15.53:0/980154799 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 0x7f63c40d15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 --2- 172.21.15.53:0/980154799 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cde90 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 -- 172.21.15.53:0/980154799 >> 172.21.15.53:0/980154799 conn(0x7f63c409c540 msgr2=0x7f63c40d3210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:08.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 -- 172.21.15.53:0/980154799 shutdown_connections 2024-01-16T15:28:08.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.014+0000 7f63cba00640 1 -- 172.21.15.53:0/980154799 wait complete. 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.016+0000 7f63cba00640 1 Processor -- start 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.016+0000 7f63cba00640 1 -- start start 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63cba00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 0x7f63c41f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63cba00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c40b48a0 0x7f63c40c6d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63cba00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63cba00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c41f7880 con 0x7f63c415fa90 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63cba00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40c73d0 con 0x7f63c40b48a0 2024-01-16T15:28:08.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63cba00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40c76b0 con 0x7f63c40b4460 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63c9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63c9a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cbd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55718/0 (socket says 172.21.15.53:55718) 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.017+0000 7f63c9a00640 1 -- 172.21.15.53:0/928973654 learned_addr learned my addr 172.21.15.53:0/928973654 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9000640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 0x7f63c41f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c3e00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c40b48a0 0x7f63c40c6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9a00640 1 -- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 msgr2=0x7f63c41f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9a00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f63c40b4460 0x7f63c41f63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9a00640 1 -- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c40b48a0 msgr2=0x7f63c40c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9a00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c40b48a0 0x7f63c40c6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9a00640 1 -- 172.21.15.53:0/928973654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b40094d0 con 0x7f63c415fa90 2024-01-16T15:28:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c3e00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f63c40b48a0 0x7f63c40c6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:08.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c9a00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cbd00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00b9e0 tx=0x7f63bc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc00cb20 con 0x7f63c415fa90 2024-01-16T15:28:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc00cca0 con 0x7f63c415fa90 2024-01-16T15:28:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc014680 con 0x7f63c415fa90 2024-01-16T15:28:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.018+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c40c7990 con 0x7f63c415fa90 2024-01-16T15:28:08.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.019+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c40c7f50 con 0x7f63c415fa90 2024-01-16T15:28:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.020+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f63c41621b0 con 0x7f63c415fa90 2024-01-16T15:28:08.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.021+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f63bc01d3b0 con 0x7f63c415fa90 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.022+0000 7f63c0c00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f639c06c5a0 0x7f639c06ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.022+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f63bc088b60 con 0x7f63c415fa90 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.022+0000 7f63c0c00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7f639c0748e0 0x7f639c076d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.022+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 --> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f639c077410 con 0x7f639c0748e0 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.022+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7f63bc089670 con 0x7f63c415fa90 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.023+0000 7f63c9000640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f639c06c5a0 0x7f639c06ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.023+0000 7f63c3e00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7f639c0748e0 0x7f639c076d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.024+0000 7f63c3e00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7f639c0748e0 0x7f639c076d40 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.025+0000 7f63c9000640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f639c06c5a0 0x7f639c06ea60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f63b4036000 tx=0x7f63b403b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.026 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.026+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== osd.4 v2:172.21.15.146:6808/981755469 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7f639c077410 con 0x7f639c0748e0 2024-01-16T15:28:08.041 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.041+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 --> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f63c40d2440 con 0x7f639c0748e0 2024-01-16T15:28:08.042 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.042+0000 7f63c0c00640 1 -- 172.21.15.53:0/928973654 <== osd.4 v2:172.21.15.146:6808/981755469 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f63c40d2440 con 0x7f639c0748e0 2024-01-16T15:28:08.042 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:08.042 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:08.043 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:08.043 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7f639c0748e0 msgr2=0x7f639c076d40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6808/981755469,v1:172.21.15.146:6810/981755469] conn(0x7f639c0748e0 0x7f639c076d40 crc :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f639c06c5a0 msgr2=0x7f639c06ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f639c06c5a0 0x7f639c06ea60 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f63b4036000 tx=0x7f63b403b040 comp rx=0 tx=0).stop 2024-01-16T15:28:08.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 msgr2=0x7f63c40cbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 --2- 172.21.15.53:0/928973654 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f63c415fa90 0x7f63c40cbd00 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00b9e0 tx=0x7f63bc00bea0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.044 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63c9a00640 1 -- 172.21.15.53:0/928973654 reap_dead start 2024-01-16T15:28:08.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 shutdown_connections 2024-01-16T15:28:08.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 >> 172.21.15.53:0/928973654 conn(0x7f63c409c540 msgr2=0x7f63c406b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:08.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 shutdown_connections 2024-01-16T15:28:08.045 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.043+0000 7f63cba00640 1 -- 172.21.15.53:0/928973654 wait complete. 2024-01-16T15:28:08.060 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-01-16T15:28:08.281 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.282+0000 7f71f8c00640 1 Processor -- start 2024-01-16T15:28:08.282 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.282+0000 7f71f8c00640 1 -- start start 2024-01-16T15:28:08.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.283+0000 7f71f8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 0x7f71f40b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.283+0000 7f71f8c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f71f40d0f40 0x7f71f40d13f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.283+0000 7f71f8c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71f40d0a10 con 0x7f71f4160920 2024-01-16T15:28:08.283 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.283+0000 7f71f8c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71f40cf080 con 0x7f71f40d0f40 2024-01-16T15:28:08.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.283+0000 7f71f8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40c6be0 con 0x7f71f40d0100 2024-01-16T15:28:08.284 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f3e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f71f40d0f40 0x7f71f40d13f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 0x7f71f40b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 0x7f71f40b3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.146:3301/0 says I am v2:172.21.15.53:56034/0 (socket says 172.21.15.53:56034) 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 -- 172.21.15.53:0/2804707622 learned_addr learned my addr 172.21.15.53:0/2804707622 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2000640 1 -- 172.21.15.53:0/2804707622 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354773140 0 0) 0x7f71f40d0a10 con 0x7f71f4160920 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 -- 172.21.15.53:0/2804707622 >> v2:172.21.15.146:3300/0 conn(0x7f71f40d0f40 msgr2=0x7f71f40d13f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 --2- 172.21.15.53:0/2804707622 >> v2:172.21.15.146:3300/0 conn(0x7f71f40d0f40 0x7f71f40d13f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 -- 172.21.15.53:0/2804707622 >> v1:172.21.15.146:6789/0 conn(0x7f71f4160920 legacy=0x7f71f40cf9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.285 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.284+0000 7f71f2a00640 1 -- 172.21.15.53:0/2804707622 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f40ce7e0 con 0x7f71f40d0100 2024-01-16T15:28:08.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.286+0000 7f71f2a00640 1 --2- 172.21.15.53:0/2804707622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 0x7f71f40b3f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009870 tx=0x7f71dc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afd4b0b7a9ebbd17 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.286+0000 7f71f2000640 1 -- 172.21.15.53:0/2804707622 <== mon.2 v2:172.21.15.146:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc030840 con 0x7f71f40d0100 2024-01-16T15:28:08.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.286+0000 7f71f2000640 1 -- 172.21.15.53:0/2804707622 <== mon.2 v2:172.21.15.146:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71dc0309a0 con 0x7f71f40d0100 2024-01-16T15:28:08.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.286+0000 7f71f2000640 1 -- 172.21.15.53:0/2804707622 <== mon.2 v2:172.21.15.146:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71dc030d30 con 0x7f71f40d0100 2024-01-16T15:28:08.287 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 -- 172.21.15.53:0/2804707622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 msgr2=0x7f71f40b3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 --2- 172.21.15.53:0/2804707622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 0x7f71f40b3f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f71dc009870 tx=0x7f71dc02b980 comp rx=0 tx=0).stop 2024-01-16T15:28:08.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 -- 172.21.15.53:0/2804707622 shutdown_connections 2024-01-16T15:28:08.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 --2- 172.21.15.53:0/2804707622 >> v2:172.21.15.146:3300/0 conn(0x7f71f40d0f40 0x7f71f40d13f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 --2- 172.21.15.53:0/2804707622 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0100 0x7f71f40b3f30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.288 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 -- 172.21.15.53:0/2804707622 >> 172.21.15.53:0/2804707622 conn(0x7f71f40d3090 msgr2=0x7f71f40d34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:08.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 -- 172.21.15.53:0/2804707622 shutdown_connections 2024-01-16T15:28:08.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.287+0000 7f71f8c00640 1 -- 172.21.15.53:0/2804707622 wait complete. 2024-01-16T15:28:08.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.289+0000 7f71f8c00640 1 Processor -- start 2024-01-16T15:28:08.289 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.289+0000 7f71f8c00640 1 -- start start 2024-01-16T15:28:08.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.290+0000 7f71f8c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 0x7f71f414f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.290+0000 7f71f8c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0f40 0x7f71f414e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.291 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.290+0000 7f71f8c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71f4160920 0x7f71f40cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.290+0000 7f71f8c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40b7190 con 0x7f71f40d0100 2024-01-16T15:28:08.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.290+0000 7f71f8c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71f414f730 con 0x7f71f4160920 2024-01-16T15:28:08.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.290+0000 7f71f8c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71f40b1df0 con 0x7f71f40d0f40 2024-01-16T15:28:08.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 0x7f71f414f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 0x7f71f414f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55726/0 (socket says 172.21.15.53:55726) 2024-01-16T15:28:08.292 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 -- 172.21.15.53:0/4022161326 learned_addr learned my addr 172.21.15.53:0/4022161326 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3e00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71f4160920 0x7f71f40cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f2a00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0f40 0x7f71f414e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 -- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0f40 msgr2=0x7f71f414e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0f40 0x7f71f414e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 -- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71f4160920 msgr2=0x7f71f40cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71f4160920 0x7f71f40cdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.291+0000 7f71f3400640 1 -- 172.21.15.53:0/4022161326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71dc0094d0 con 0x7f71f40d0100 2024-01-16T15:28:08.293 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.292+0000 7f71f3400640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 0x7f71f414f1f0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c4b0 tx=0x7f71e400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.292+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e4005480 con 0x7f71f40d0100 2024-01-16T15:28:08.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.292+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71e40035f0 con 0x7f71f40d0100 2024-01-16T15:28:08.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.292+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71f40b20d0 con 0x7f71f40d0100 2024-01-16T15:28:08.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.292+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71e40039e0 con 0x7f71f40d0100 2024-01-16T15:28:08.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.292+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71f40b2690 con 0x7f71f40d0100 2024-01-16T15:28:08.295 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.293+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71f4067450 con 0x7f71f40d0100 2024-01-16T15:28:08.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.295+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f71e4003b40 con 0x7f71f40d0100 2024-01-16T15:28:08.296 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.296+0000 7f71eb400640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f71cc06c550 0x7f71cc06ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.296+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f71e4088120 con 0x7f71f40d0100 2024-01-16T15:28:08.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.296+0000 7f71f2a00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f71cc06c550 0x7f71cc06ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.297 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.298+0000 7f71f2a00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f71cc06c550 0x7f71cc06ea10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f71dc036000 tx=0x7f71dc03b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.300 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.300+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f71e40546c0 con 0x7f71f40d0100 2024-01-16T15:28:08.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.493+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f71f4160d00 con 0x7f71cc06c550 2024-01-16T15:28:08.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.495+0000 7f71eb400640 1 -- 172.21.15.53:0/4022161326 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f71dc030b90 con 0x7f71cc06c550 2024-01-16T15:28:08.495 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 4 to deep-scrub 2024-01-16T15:28:08.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f71cc06c550 msgr2=0x7f71cc06ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f71cc06c550 0x7f71cc06ea10 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f71dc036000 tx=0x7f71dc03b040 comp rx=0 tx=0).stop 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 msgr2=0x7f71f414f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 0x7f71f414f1f0 secure :-1 s=READY pgs=2047 cs=0 l=1 rev1=1 crypto rx=0x7f71e400c4b0 tx=0x7f71e400c970 comp rx=0 tx=0).stop 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 shutdown_connections 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f71cc06c550 0x7f71cc06ea10 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f71f4160920 0x7f71f40cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f71f40d0f40 0x7f71f414e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 --2- 172.21.15.53:0/4022161326 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f71f40d0100 0x7f71f414f1f0 unknown :-1 s=CLOSED pgs=2047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 >> 172.21.15.53:0/4022161326 conn(0x7f71f40d3090 msgr2=0x7f71f406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:08.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.497+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 shutdown_connections 2024-01-16T15:28:08.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.498+0000 7f71f8c00640 1 -- 172.21.15.53:0/4022161326 wait complete. 2024-01-16T15:28:08.516 INFO:tasks.ceph:Scrubbing osd.5 2024-01-16T15:28:08.516 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:08.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.746+0000 7fb6ef200640 1 Processor -- start 2024-01-16T15:28:08.746 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.746+0000 7fb6ef200640 1 -- start start 2024-01-16T15:28:08.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ef200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb6f015af30 0x7fb6f015e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ef200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb6f01603c0 0x7fb6f006b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ef200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6f006bd80 con 0x7fb6f006b3f0 2024-01-16T15:28:08.748 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ef200640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f015e6a0 con 0x7fb6f01603c0 2024-01-16T15:28:08.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ef200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6f0071810 con 0x7fb6f015af30 2024-01-16T15:28:08.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ee800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fb6f01603c0 0x7fb6f006b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ed400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb6f006b3f0 0x7fb6f0160d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57640/0 (socket says 172.21.15.53:57640) 2024-01-16T15:28:08.749 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ed400640 1 -- 172.21.15.53:0/143129178 learned_addr learned my addr 172.21.15.53:0/143129178 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:08.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.748+0000 7fb6ede00640 1 --2- 172.21.15.53:0/143129178 >> v2:172.21.15.146:3300/0 conn(0x7fb6f015af30 0x7fb6f015e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.750 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.749+0000 7fb6eca00640 1 -- 172.21.15.53:0/143129178 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 516292578 0 0) 0x7fb6d8001110 con 0x7fb6f006b3f0 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.749+0000 7fb6ee800640 1 -- 172.21.15.53:0/143129178 >> v2:172.21.15.146:3300/0 conn(0x7fb6f015af30 msgr2=0x7fb6f015e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.749+0000 7fb6ee800640 1 --2- 172.21.15.53:0/143129178 >> v2:172.21.15.146:3300/0 conn(0x7fb6f015af30 0x7fb6f015e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.749+0000 7fb6ee800640 1 -- 172.21.15.53:0/143129178 >> v1:172.21.15.53:6789/0 conn(0x7fb6f006b3f0 legacy=0x7fb6f0160d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.749+0000 7fb6ee800640 1 -- 172.21.15.53:0/143129178 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6f0071af0 con 0x7fb6f01603c0 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.750+0000 7fb6ee800640 1 --2- 172.21.15.53:0/143129178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f01603c0 0x7fb6f006b840 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00702e0 tx=0x7fb6e802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00f2e5fe921eda8 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.750+0000 7fb6eca00640 1 -- 172.21.15.53:0/143129178 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8031830 con 0x7fb6f01603c0 2024-01-16T15:28:08.751 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.750+0000 7fb6eca00640 1 -- 172.21.15.53:0/143129178 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e8031990 con 0x7fb6f01603c0 2024-01-16T15:28:08.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.750+0000 7fb6eca00640 1 -- 172.21.15.53:0/143129178 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8031d20 con 0x7fb6f01603c0 2024-01-16T15:28:08.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 -- 172.21.15.53:0/143129178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f01603c0 msgr2=0x7fb6f006b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 --2- 172.21.15.53:0/143129178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f01603c0 0x7fb6f006b840 secure :-1 s=READY pgs=2049 cs=0 l=1 rev1=1 crypto rx=0x7fb6f00702e0 tx=0x7fb6e802b8d0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 -- 172.21.15.53:0/143129178 shutdown_connections 2024-01-16T15:28:08.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 --2- 172.21.15.53:0/143129178 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f01603c0 0x7fb6f006b840 unknown :-1 s=CLOSED pgs=2049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 --2- 172.21.15.53:0/143129178 >> v2:172.21.15.146:3300/0 conn(0x7fb6f015af30 0x7fb6f015e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 -- 172.21.15.53:0/143129178 >> 172.21.15.53:0/143129178 conn(0x7fb6f0070950 msgr2=0x7fb6f0070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:08.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 -- 172.21.15.53:0/143129178 shutdown_connections 2024-01-16T15:28:08.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.751+0000 7fb6ef200640 1 -- 172.21.15.53:0/143129178 wait complete. 2024-01-16T15:28:08.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.753+0000 7fb6ef200640 1 Processor -- start 2024-01-16T15:28:08.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.753+0000 7fb6ef200640 1 -- start start 2024-01-16T15:28:08.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ef200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f006b3f0 0x7fb6f0151fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ef200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6f015af30 0x7fb6f01560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ef200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6f01603c0 0x7fb6f0159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ef200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f0154d60 con 0x7fb6f006b3f0 2024-01-16T15:28:08.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ef200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f01f6860 con 0x7fb6f01603c0 2024-01-16T15:28:08.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ef200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6f01f6b40 con 0x7fb6f015af30 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ede00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f006b3f0 0x7fb6f0151fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ede00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f006b3f0 0x7fb6f0151fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55742/0 (socket says 172.21.15.53:55742) 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.754+0000 7fb6ede00640 1 -- 172.21.15.53:0/1483052545 learned_addr learned my addr 172.21.15.53:0/1483052545 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ee800640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6f01603c0 0x7fb6f0159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ed400640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6f015af30 0x7fb6f01560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ede00640 1 -- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6f015af30 msgr2=0x7fb6f01560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ede00640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb6f015af30 0x7fb6f01560c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ede00640 1 -- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6f01603c0 msgr2=0x7fb6f0159120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ede00640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6f01603c0 0x7fb6f0159120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ede00640 1 -- 172.21.15.53:0/1483052545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6e8009400 con 0x7fb6f006b3f0 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ee800640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb6f01603c0 0x7fb6f0159120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.755+0000 7fb6ede00640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f006b3f0 0x7fb6f0151fe0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0007e40 tx=0x7fb6e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.756+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0014620 con 0x7fb6f006b3f0 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.756+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6e000ee90 con 0x7fb6f006b3f0 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.756+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00149c0 con 0x7fb6f006b3f0 2024-01-16T15:28:08.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.756+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6f01f6e20 con 0x7fb6f006b3f0 2024-01-16T15:28:08.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.756+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6f01f6480 con 0x7fb6f006b3f0 2024-01-16T15:28:08.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.757+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb6f015ee90 con 0x7fb6f006b3f0 2024-01-16T15:28:08.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.758+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb6e0014b20 con 0x7fb6f006b3f0 2024-01-16T15:28:08.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.759+0000 7fb6e6000640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6c806c550 0x7fb6c806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.759+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb6e00894b0 con 0x7fb6f006b3f0 2024-01-16T15:28:08.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.759+0000 7fb6e6000640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7fb6c8074810 0x7fb6c8076c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:08.760 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.759+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 --> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb6c8077340 con 0x7fb6c8074810 2024-01-16T15:28:08.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.759+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6e008a150 con 0x7fb6f006b3f0 2024-01-16T15:28:08.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.760+0000 7fb6ed400640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6c806c550 0x7fb6c806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.760+0000 7fb6ee800640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7fb6c8074810 0x7fb6c8076c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:08.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.761+0000 7fb6ee800640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7fb6c8074810 0x7fb6c8076c70 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.761 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.761+0000 7fb6ed400640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6c806c550 0x7fb6c806ea10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0151f90 tx=0x7fb6d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:08.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.763+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== osd.5 v2:172.21.15.146:6824/2159577511 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb6c8077340 con 0x7fb6c8074810 2024-01-16T15:28:08.778 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.778+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 --> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb6f015ebb0 con 0x7fb6c8074810 2024-01-16T15:28:08.779 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.779+0000 7fb6e6000640 1 -- 172.21.15.53:0/1483052545 <== osd.5 v2:172.21.15.146:6824/2159577511 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb6e8041040 con 0x7fb6c8074810 2024-01-16T15:28:08.779 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:08.779 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:08.779 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:08.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.779+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7fb6c8074810 msgr2=0x7fb6c8076c70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.779+0000 7fb6ef200640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6824/2159577511,v1:172.21.15.146:6825/2159577511] conn(0x7fb6c8074810 0x7fb6c8076c70 crc :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.779+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6c806c550 msgr2=0x7fb6c806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.779+0000 7fb6ef200640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb6c806c550 0x7fb6c806ea10 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0151f90 tx=0x7fb6d8009040 comp rx=0 tx=0).stop 2024-01-16T15:28:08.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f006b3f0 msgr2=0x7fb6f0151fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:08.781 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ef200640 1 --2- 172.21.15.53:0/1483052545 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb6f006b3f0 0x7fb6f0151fe0 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7fb6e0007e40 tx=0x7fb6e000e4b0 comp rx=0 tx=0).stop 2024-01-16T15:28:08.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ee800640 1 -- 172.21.15.53:0/1483052545 reap_dead start 2024-01-16T15:28:08.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 shutdown_connections 2024-01-16T15:28:08.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 >> 172.21.15.53:0/1483052545 conn(0x7fb6f0070950 msgr2=0x7fb6f00733f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:08.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 shutdown_connections 2024-01-16T15:28:08.782 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:08.780+0000 7fb6ef200640 1 -- 172.21.15.53:0/1483052545 wait complete. 2024-01-16T15:28:08.797 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-01-16T15:28:09.016 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.016+0000 7fcb55c00640 1 Processor -- start 2024-01-16T15:28:09.017 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.017+0000 7fcb55c00640 1 -- start start 2024-01-16T15:28:09.018 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb55c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcb500cd240 0x7fcb500b3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb55c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb50099a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb55c00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5006cea0 con 0x7fcb50099fa0 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb55c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5009a430 con 0x7fcb500cd240 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb55c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5006cbc0 con 0x7fcb5006c6b0 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb4ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcb500cd240 0x7fcb500b3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb4ea00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fcb500cd240 0x7fcb500b3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55752/0 (socket says 172.21.15.53:55752) 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.018+0000 7fcb4ea00640 1 -- 172.21.15.53:0/3118917662 learned_addr learned my addr 172.21.15.53:0/3118917662 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:09.019 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.019+0000 7fcb4e000640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb50099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.019+0000 7fcb4ea00640 1 -- 172.21.15.53:0/3118917662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 msgr2=0x7fcb50099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.019+0000 7fcb4ea00640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb50099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.019+0000 7fcb4ea00640 1 -- 172.21.15.53:0/3118917662 >> v1:172.21.15.146:6789/0 conn(0x7fcb50099fa0 legacy=0x7fcb5006b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.019+0000 7fcb4ea00640 1 -- 172.21.15.53:0/3118917662 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5006f350 con 0x7fcb500cd240 2024-01-16T15:28:09.020 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.019+0000 7fcb4e000640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb50099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.020+0000 7fcb4ea00640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb500cd240 0x7fcb500b3820 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009e50 tx=0x7fcb4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227c24dffec1ef6c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.021 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.020+0000 7fcb4d600640 1 -- 172.21.15.53:0/3118917662 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4002ca80 con 0x7fcb500cd240 2024-01-16T15:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.020+0000 7fcb4d600640 1 -- 172.21.15.53:0/3118917662 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb4002cbe0 con 0x7fcb500cd240 2024-01-16T15:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.020+0000 7fcb4d600640 1 -- 172.21.15.53:0/3118917662 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb400343f0 con 0x7fcb500cd240 2024-01-16T15:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 -- 172.21.15.53:0/3118917662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb500cd240 msgr2=0x7fcb500b3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb500cd240 0x7fcb500b3820 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7fcb40009e50 tx=0x7fcb4002bee0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 -- 172.21.15.53:0/3118917662 shutdown_connections 2024-01-16T15:28:09.022 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb50099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 --2- 172.21.15.53:0/3118917662 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb500cd240 0x7fcb500b3820 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 -- 172.21.15.53:0/3118917662 >> 172.21.15.53:0/3118917662 conn(0x7fcb500b2d00 msgr2=0x7fcb500b3110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:09.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 -- 172.21.15.53:0/3118917662 shutdown_connections 2024-01-16T15:28:09.023 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.021+0000 7fcb55c00640 1 -- 172.21.15.53:0/3118917662 wait complete. 2024-01-16T15:28:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.023+0000 7fcb55c00640 1 Processor -- start 2024-01-16T15:28:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.023+0000 7fcb55c00640 1 -- start start 2024-01-16T15:28:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb55c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb500b6b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.024 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb55c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 0x7fcb50156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb55c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb500cd240 0x7fcb500c6d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb55c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500b7050 con 0x7fcb50099fa0 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb55c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500c4d40 con 0x7fcb500cd240 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb55c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb50156d40 con 0x7fcb5006c6b0 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb4e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 0x7fcb50156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb4e000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 0x7fcb50156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55754/0 (socket says 172.21.15.53:55754) 2024-01-16T15:28:09.025 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.024+0000 7fcb4e000640 1 -- 172.21.15.53:0/2924137890 learned_addr learned my addr 172.21.15.53:0/2924137890 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4f400640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb500cd240 0x7fcb500c6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.027 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4ea00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb500b6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4e000640 1 -- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 msgr2=0x7fcb500b6b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4e000640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb500b6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4e000640 1 -- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb500cd240 msgr2=0x7fcb500c6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4e000640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb500cd240 0x7fcb500c6d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4e000640 1 -- 172.21.15.53:0/2924137890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb40009b30 con 0x7fcb50099fa0 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4f400640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb500cd240 0x7fcb500c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb4e000640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 0x7fcb50156800 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c7f0 tx=0x7fcb3800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.028 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38007820 con 0x7fcb50099fa0 2024-01-16T15:28:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb38007980 con 0x7fcb50099fa0 2024-01-16T15:28:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb38007d10 con 0x7fcb50099fa0 2024-01-16T15:28:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.025+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb501593a0 con 0x7fcb50099fa0 2024-01-16T15:28:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.026+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb50159960 con 0x7fcb50099fa0 2024-01-16T15:28:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.027+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb500675d0 con 0x7fcb50099fa0 2024-01-16T15:28:09.029 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.028+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fcb380113f0 con 0x7fcb50099fa0 2024-01-16T15:28:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.029+0000 7fcb46a00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2806c550 0x7fcb2806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.029+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fcb38088af0 con 0x7fcb50099fa0 2024-01-16T15:28:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.030+0000 7fcb4ea00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2806c550 0x7fcb2806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.030 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.031+0000 7fcb4ea00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2806c550 0x7fcb2806ea10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcb40035000 tx=0x7fcb40030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.033 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.033+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fcb380553b0 con 0x7fcb50099fa0 2024-01-16T15:28:09.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.227+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7fcb500667d0 con 0x7fcb2806c550 2024-01-16T15:28:09.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.228+0000 7fcb46a00640 1 -- 172.21.15.53:0/2924137890 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcb500667d0 con 0x7fcb2806c550 2024-01-16T15:28:09.229 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 5 to deep-scrub 2024-01-16T15:28:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2806c550 msgr2=0x7fcb2806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2806c550 0x7fcb2806ea10 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fcb40035000 tx=0x7fcb40030000 comp rx=0 tx=0).stop 2024-01-16T15:28:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 msgr2=0x7fcb50156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 0x7fcb50156800 secure :-1 s=READY pgs=2052 cs=0 l=1 rev1=1 crypto rx=0x7fcb3800c7f0 tx=0x7fcb3800ccb0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 shutdown_connections 2024-01-16T15:28:09.232 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fcb2806c550 0x7fcb2806ea10 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fcb500cd240 0x7fcb500c6d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fcb50099fa0 0x7fcb50156800 unknown :-1 s=CLOSED pgs=2052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 --2- 172.21.15.53:0/2924137890 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fcb5006c6b0 0x7fcb500b6b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 >> 172.21.15.53:0/2924137890 conn(0x7fcb500b2d00 msgr2=0x7fcb501608a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:09.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 shutdown_connections 2024-01-16T15:28:09.233 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.231+0000 7fcb55c00640 1 -- 172.21.15.53:0/2924137890 wait complete. 2024-01-16T15:28:09.251 INFO:tasks.ceph:Scrubbing osd.6 2024-01-16T15:28:09.251 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:09.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.482+0000 7fb057e00640 1 Processor -- start 2024-01-16T15:28:09.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.482+0000 7fb057e00640 1 -- start start 2024-01-16T15:28:09.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb057e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 0x7fb0580cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb057e00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fb05806d160 0x7fb05806d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb057e00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb058070e50 con 0x7fb058143c40 2024-01-16T15:28:09.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb057e00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0580723c0 con 0x7fb05806d160 2024-01-16T15:28:09.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb057e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb058144020 con 0x7fb0580cc500 2024-01-16T15:28:09.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb056a00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7fb058143c40 0x7fb058071290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57654/0 (socket says 172.21.15.53:57654) 2024-01-16T15:28:09.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.484+0000 7fb056a00640 1 -- 172.21.15.53:0/1350250618 learned_addr learned my addr 172.21.15.53:0/1350250618 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:09.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb057400640 1 --2- 172.21.15.53:0/1350250618 >> v2:172.21.15.146:3300/0 conn(0x7fb05806d160 0x7fb05806d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb056000640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 0x7fb0580cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 404546660 0 0) 0x7fb058070e50 con 0x7fb058143c40 2024-01-16T15:28:09.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb038003520 con 0x7fb058143c40 2024-01-16T15:28:09.487 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 854547893 0 0) 0x7fb038003520 con 0x7fb058143c40 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 msgr2=0x7fb0580cc960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 0x7fb0580cc960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 >> v2:172.21.15.146:3300/0 conn(0x7fb05806d160 msgr2=0x7fb05806d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 --2- 172.21.15.53:0/1350250618 >> v2:172.21.15.146:3300/0 conn(0x7fb05806d160 0x7fb05806d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0580ccea0 con 0x7fb058143c40 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7fb048001670 con 0x7fb058143c40 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 >> v1:172.21.15.53:6789/0 conn(0x7fb058143c40 legacy=0x7fb058071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 0x7fb038003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.485+0000 7fb055600640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038003ef0 0x7fb0380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb055600640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb038004d00 0x7fb038005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580723c0 con 0x7fb038004d00 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb058144020 con 0x7fb038003ef0 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03800b9d0 con 0x7fb0580cc500 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb057400640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb038004d00 0x7fb038005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb056a00640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 0x7fb038003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb056000640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038003ef0 0x7fb0380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb057400640 1 -- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 msgr2=0x7fb038003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb057400640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0580cc500 0x7fb038003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb057400640 1 -- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038003ef0 msgr2=0x7fb0380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb057400640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038003ef0 0x7fb0380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.486+0000 7fb057400640 1 -- 172.21.15.53:0/1350250618 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb038003520 con 0x7fb038004d00 2024-01-16T15:28:09.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb057400640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb038004d00 0x7fb038005180 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e170 tx=0x7fb0380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=689a65593e7f8a44 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.493 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05002cc20 con 0x7fb038004d00 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb05000f040 con 0x7fb038004d00 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb055600640 1 -- 172.21.15.53:0/1350250618 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb050034440 con 0x7fb038004d00 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb057e00640 1 -- 172.21.15.53:0/1350250618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb038004d00 msgr2=0x7fb038005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb057e00640 1 --2- 172.21.15.53:0/1350250618 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb038004d00 0x7fb038005180 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7fb05000e170 tx=0x7fb0380057f0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.488+0000 7fb057400640 1 -- 172.21.15.53:0/1350250618 reap_dead start 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.489+0000 7fb057e00640 1 -- 172.21.15.53:0/1350250618 shutdown_connections 2024-01-16T15:28:09.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.489+0000 7fb057e00640 1 -- 172.21.15.53:0/1350250618 >> 172.21.15.53:0/1350250618 conn(0x7fb05806a800 msgr2=0x7fb05806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.489+0000 7fb057e00640 1 -- 172.21.15.53:0/1350250618 shutdown_connections 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.489+0000 7fb057e00640 1 -- 172.21.15.53:0/1350250618 wait complete. 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.490+0000 7fb057e00640 1 Processor -- start 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.491+0000 7fb057e00640 1 -- start start 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057e00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038004d00 0x7fb0580aa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb05815c510 0x7fb0580adcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057e00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0581598d0 0x7fb058159d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057e00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05815a290 con 0x7fb05815c510 2024-01-16T15:28:09.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057e00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb05813cba0 con 0x7fb038004d00 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057e00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0580a84c0 con 0x7fb0581598d0 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb05815c510 0x7fb0580adcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb05815c510 0x7fb0580adcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55768/0 (socket says 172.21.15.53:55768) 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 -- 172.21.15.53:0/2032108353 learned_addr learned my addr 172.21.15.53:0/2032108353 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb057400640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0581598d0 0x7fb058159d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056a00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038004d00 0x7fb0580aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 -- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0581598d0 msgr2=0x7fb058159d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fb0581598d0 0x7fb058159d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 -- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038004d00 msgr2=0x7fb0580aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038004d00 0x7fb0580aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056000640 1 -- 172.21.15.53:0/2032108353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0580aa5e0 con 0x7fb05815c510 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.492+0000 7fb056a00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fb038004d00 0x7fb0580aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.493+0000 7fb056000640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb05815c510 0x7fb0580adcc0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb040011540 tx=0x7fb040011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.493+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040015930 con 0x7fb05815c510 2024-01-16T15:28:09.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.493+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb040015a90 con 0x7fb05815c510 2024-01-16T15:28:09.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.493+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0580aa8c0 con 0x7fb05815c510 2024-01-16T15:28:09.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.493+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040015e60 con 0x7fb05815c510 2024-01-16T15:28:09.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.493+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0580a8760 con 0x7fb05815c510 2024-01-16T15:28:09.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.494+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb058067db0 con 0x7fb05815c510 2024-01-16T15:28:09.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.495+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fb04000f460 con 0x7fb05815c510 2024-01-16T15:28:09.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.496+0000 7fb04ea00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03006c580 0x7fb03006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.497+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fb040089070 con 0x7fb05815c510 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.497+0000 7fb04ea00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fb030074840 0x7fb030076ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.497+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 --> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb030077370 con 0x7fb030074840 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.497+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb040089880 con 0x7fb05815c510 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.497+0000 7fb056a00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03006c580 0x7fb03006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.497+0000 7fb057400640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fb030074840 0x7fb030076ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.499+0000 7fb057400640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fb030074840 0x7fb030076ca0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.499+0000 7fb056a00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03006c580 0x7fb03006ea40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb048005a10 tx=0x7fb048001560 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.501+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== osd.6 v2:172.21.15.146:6800/2492658325 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fb030077370 con 0x7fb030074840 2024-01-16T15:28:09.515 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.515+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 --> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fb0580aade0 con 0x7fb030074840 2024-01-16T15:28:09.517 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb04ea00640 1 -- 172.21.15.53:0/2032108353 <== osd.6 v2:172.21.15.146:6800/2492658325 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fb05002cb60 con 0x7fb030074840 2024-01-16T15:28:09.517 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:09.517 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:09.518 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fb030074840 msgr2=0x7fb030076ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb057e00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6800/2492658325,v1:172.21.15.146:6801/2492658325] conn(0x7fb030074840 0x7fb030076ca0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03006c580 msgr2=0x7fb03006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb057e00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fb03006c580 0x7fb03006ea40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb048005a10 tx=0x7fb048001560 comp rx=0 tx=0).stop 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb05815c510 msgr2=0x7fb0580adcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.517+0000 7fb057e00640 1 --2- 172.21.15.53:0/2032108353 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fb05815c510 0x7fb0580adcc0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fb040011540 tx=0x7fb040011a00 comp rx=0 tx=0).stop 2024-01-16T15:28:09.519 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.518+0000 7fb057400640 1 -- 172.21.15.53:0/2032108353 reap_dead start 2024-01-16T15:28:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.518+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 shutdown_connections 2024-01-16T15:28:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.518+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 >> 172.21.15.53:0/2032108353 conn(0x7fb05806a800 msgr2=0x7fb058141520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.518+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 shutdown_connections 2024-01-16T15:28:09.520 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.518+0000 7fb057e00640 1 -- 172.21.15.53:0/2032108353 wait complete. 2024-01-16T15:28:09.535 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-01-16T15:28:09.752 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.752+0000 7f1378c00640 1 Processor -- start 2024-01-16T15:28:09.753 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.752+0000 7f1378c00640 1 -- start start 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1378c00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f137406a3e0 0x7f13740ac8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1378c00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f13740d0b60 0x7f13740aede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1378c00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1374071de0 con 0x7f137406a7c0 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1378c00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13740af320 con 0x7f137406a3e0 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1378c00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1374072030 con 0x7f13740d0b60 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1372800640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f137406a7c0 0x7f137406f080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57662/0 (socket says 172.21.15.53:57662) 2024-01-16T15:28:09.754 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1371e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f137406a3e0 0x7f13740ac8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.755 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f1372800640 1 -- 172.21.15.53:0/2685778932 learned_addr learned my addr 172.21.15.53:0/2685778932 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:09.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.754+0000 7f136be00640 1 --2- 172.21.15.53:0/2685778932 >> v2:172.21.15.146:3300/0 conn(0x7f13740d0b60 0x7f13740aede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.755+0000 7f1371e00640 1 -- 172.21.15.53:0/2685778932 >> v2:172.21.15.146:3300/0 conn(0x7f13740d0b60 msgr2=0x7f13740aede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:09.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.755+0000 7f1371e00640 1 --2- 172.21.15.53:0/2685778932 >> v2:172.21.15.146:3300/0 conn(0x7f13740d0b60 0x7f13740aede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.756 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.755+0000 7f1371e00640 1 -- 172.21.15.53:0/2685778932 >> v1:172.21.15.53:6789/0 conn(0x7f137406a7c0 legacy=0x7f137406f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.755+0000 7f1371400640 1 -- 172.21.15.53:0/2685778932 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361188923 0 0) 0x7f1374071de0 con 0x7f137406a7c0 2024-01-16T15:28:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.755+0000 7f1371e00640 1 -- 172.21.15.53:0/2685778932 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1374072310 con 0x7f137406a3e0 2024-01-16T15:28:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.756+0000 7f1371e00640 1 --2- 172.21.15.53:0/2685778932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137406a3e0 0x7f13740ac8d0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f13740b6120 tx=0x7f136402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0d76e43bbb622c7 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.756+0000 7f1371400640 1 -- 172.21.15.53:0/2685778932 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364031830 con 0x7f137406a3e0 2024-01-16T15:28:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.756+0000 7f1371400640 1 -- 172.21.15.53:0/2685778932 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1364031990 con 0x7f137406a3e0 2024-01-16T15:28:09.757 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.756+0000 7f1371400640 1 -- 172.21.15.53:0/2685778932 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1364031d20 con 0x7f137406a3e0 2024-01-16T15:28:09.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 -- 172.21.15.53:0/2685778932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137406a3e0 msgr2=0x7f13740ac8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 --2- 172.21.15.53:0/2685778932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137406a3e0 0x7f13740ac8d0 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f13740b6120 tx=0x7f136402b8d0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 -- 172.21.15.53:0/2685778932 shutdown_connections 2024-01-16T15:28:09.758 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 --2- 172.21.15.53:0/2685778932 >> v2:172.21.15.146:3300/0 conn(0x7f13740d0b60 0x7f13740aede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 --2- 172.21.15.53:0/2685778932 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f137406a3e0 0x7f13740ac8d0 unknown :-1 s=CLOSED pgs=2057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 -- 172.21.15.53:0/2685778932 >> 172.21.15.53:0/2685778932 conn(0x7f13740cfef0 msgr2=0x7f13740d0300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:09.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 -- 172.21.15.53:0/2685778932 shutdown_connections 2024-01-16T15:28:09.759 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.757+0000 7f1378c00640 1 -- 172.21.15.53:0/2685778932 wait complete. 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.759+0000 7f1378c00640 1 Processor -- start 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.759+0000 7f1378c00640 1 -- start start 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1378c00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137406a3e0 0x7f137413eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1378c00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137406a7c0 0x7f137413dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1378c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 0x7f13741584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1378c00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13741580a0 con 0x7f13740d0b60 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1378c00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1374154130 con 0x7f137406a3e0 2024-01-16T15:28:09.762 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1378c00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1374157310 con 0x7f137406a7c0 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1372800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 0x7f13741584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1372800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 0x7f13741584e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55794/0 (socket says 172.21.15.53:55794) 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.760+0000 7f1372800640 1 -- 172.21.15.53:0/4228942873 learned_addr learned my addr 172.21.15.53:0/4228942873 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1371e00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137406a3e0 0x7f137413eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f136be00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137406a7c0 0x7f137413dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1372800640 1 -- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137406a7c0 msgr2=0x7f137413dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1372800640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137406a7c0 0x7f137413dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.763 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1372800640 1 -- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137406a3e0 msgr2=0x7f137413eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1372800640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137406a3e0 0x7f137413eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1372800640 1 -- 172.21.15.53:0/4228942873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1364009400 con 0x7f13740d0b60 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1371e00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137406a3e0 0x7f137413eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1372800640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 0x7f13741584e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f136c00c7e0 tx=0x7f136c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c00a640 con 0x7f13740d0b60 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f136c00a7c0 con 0x7f13740d0b60 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1374154380 con 0x7f13740d0b60 2024-01-16T15:28:09.764 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f136c00ab70 con 0x7f13740d0b60 2024-01-16T15:28:09.765 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.761+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f137413eff0 con 0x7f13740d0b60 2024-01-16T15:28:09.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.763+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13741581e0 con 0x7f13740d0b60 2024-01-16T15:28:09.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.764+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f136c00acd0 con 0x7f13740d0b60 2024-01-16T15:28:09.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.765+0000 7f136a000640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f134c06c580 0x7f134c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:09.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.765+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f136c0883d0 con 0x7f13740d0b60 2024-01-16T15:28:09.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.765+0000 7f1371e00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f134c06c580 0x7f134c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:09.766 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.766+0000 7f1371e00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f134c06c580 0x7f134c06ea40 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1364036000 tx=0x7f136403c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:09.769 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.769+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f136c054910 con 0x7f13740d0b60 2024-01-16T15:28:09.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.964+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f137406eb60 con 0x7f134c06c580 2024-01-16T15:28:09.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.966+0000 7f136a000640 1 -- 172.21.15.53:0/4228942873 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f1364031c80 con 0x7f134c06c580 2024-01-16T15:28:09.966 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 6 to deep-scrub 2024-01-16T15:28:09.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.968+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f134c06c580 msgr2=0x7f134c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.968+0000 7f1378c00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f134c06c580 0x7f134c06ea40 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f1364036000 tx=0x7f136403c040 comp rx=0 tx=0).stop 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.968+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 msgr2=0x7f13741584e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.968+0000 7f1378c00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 0x7f13741584e0 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f136c00c7e0 tx=0x7f136c00cca0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 shutdown_connections 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f134c06c580 0x7f134c06ea40 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f13740d0b60 0x7f13741584e0 unknown :-1 s=CLOSED pgs=2058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f137406a7c0 0x7f137413dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 --2- 172.21.15.53:0/4228942873 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f137406a3e0 0x7f137413eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:09.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 >> 172.21.15.53:0/4228942873 conn(0x7f13740cfef0 msgr2=0x7f13740b2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:09.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 shutdown_connections 2024-01-16T15:28:09.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:09.969+0000 7f1378c00640 1 -- 172.21.15.53:0/4228942873 wait complete. 2024-01-16T15:28:09.988 INFO:tasks.ceph:Scrubbing osd.7 2024-01-16T15:28:09.988 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-01-16T15:28:10.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.216+0000 7fd87d000640 1 Processor -- start 2024-01-16T15:28:10.216 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.216+0000 7fd87d000640 1 -- start start 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd87d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813df80 0x7fd87815bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd87d000640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd87813e360 0x7fd8780cf290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd87d000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd87806c470 con 0x7fd87815f9c0 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd87d000640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd87806c750 con 0x7fd87813e360 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd87d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd87806ca30 con 0x7fd87813df80 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd86d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd87813e360 0x7fd8780cf290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.218 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd86d200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fd87813e360 0x7fd8780cf290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55808/0 (socket says 172.21.15.53:55808) 2024-01-16T15:28:10.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.218+0000 7fd86d200640 1 -- 172.21.15.53:0/3752888252 learned_addr learned my addr 172.21.15.53:0/3752888252 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:10.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.219+0000 7fd875e00640 1 --2- 172.21.15.53:0/3752888252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813df80 0x7fd87815bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.219 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.219+0000 7fd86d200640 1 -- 172.21.15.53:0/3752888252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813df80 msgr2=0x7fd87815bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:10.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.219+0000 7fd86d200640 1 --2- 172.21.15.53:0/3752888252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813df80 0x7fd87815bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.219+0000 7fd86d200640 1 -- 172.21.15.53:0/3752888252 >> v1:172.21.15.146:6789/0 conn(0x7fd87815f9c0 legacy=0x7fd8781408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:10.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.219+0000 7fd86d200640 1 -- 172.21.15.53:0/3752888252 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87806cd10 con 0x7fd87813e360 2024-01-16T15:28:10.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.220+0000 7fd86d200640 1 --2- 172.21.15.53:0/3752888252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813e360 0x7fd8780cf290 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fd860009870 tx=0x7fd86002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72df990d1ff5a2d5 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.220 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.220+0000 7fd875400640 1 -- 172.21.15.53:0/3752888252 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860031640 con 0x7fd87813e360 2024-01-16T15:28:10.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd875400640 1 -- 172.21.15.53:0/3752888252 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8600317a0 con 0x7fd87813e360 2024-01-16T15:28:10.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd875400640 1 -- 172.21.15.53:0/3752888252 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd860031b30 con 0x7fd87813e360 2024-01-16T15:28:10.221 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 -- 172.21.15.53:0/3752888252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813e360 msgr2=0x7fd8780cf290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 --2- 172.21.15.53:0/3752888252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813e360 0x7fd8780cf290 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fd860009870 tx=0x7fd86002b8f0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 -- 172.21.15.53:0/3752888252 shutdown_connections 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 --2- 172.21.15.53:0/3752888252 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813e360 0x7fd8780cf290 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 --2- 172.21.15.53:0/3752888252 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813df80 0x7fd87815bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 -- 172.21.15.53:0/3752888252 >> 172.21.15.53:0/3752888252 conn(0x7fd87806a800 msgr2=0x7fd87806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.221+0000 7fd87d000640 1 -- 172.21.15.53:0/3752888252 shutdown_connections 2024-01-16T15:28:10.222 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.222+0000 7fd87d000640 1 -- 172.21.15.53:0/3752888252 wait complete. 2024-01-16T15:28:10.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.223+0000 7fd87d000640 1 Processor -- start 2024-01-16T15:28:10.223 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.223+0000 7fd87d000640 1 -- start start 2024-01-16T15:28:10.224 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.224+0000 7fd87d000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813df80 0x7fd878153f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.224+0000 7fd87d000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813e360 0x7fd8781f1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.224+0000 7fd87d000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd87815f9c0 0x7fd8781f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.224+0000 7fd87d000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878157730 con 0x7fd87813df80 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.224+0000 7fd87d000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd878156b20 con 0x7fd87815f9c0 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.224+0000 7fd87d000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd878156e00 con 0x7fd87813e360 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813df80 0x7fd878153f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813df80 0x7fd878153f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55810/0 (socket says 172.21.15.53:55810) 2024-01-16T15:28:10.225 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 -- 172.21.15.53:0/1581241687 learned_addr learned my addr 172.21.15.53:0/1581241687 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:10.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd86d200640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813e360 0x7fd8781f1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.226 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd876800640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd87815f9c0 0x7fd8781f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 -- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813e360 msgr2=0x7fd8781f1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fd87813e360 0x7fd8781f1f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 -- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd87815f9c0 msgr2=0x7fd8781f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd87815f9c0 0x7fd8781f24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 -- 172.21.15.53:0/1581241687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8600094d0 con 0x7fd87813df80 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd876800640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fd87815f9c0 0x7fd8781f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.225+0000 7fd875e00640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813df80 0x7fd878153f40 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fd86800d5f0 tx=0x7fd86800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.226+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8680125f0 con 0x7fd87813df80 2024-01-16T15:28:10.227 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.226+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd868012750 con 0x7fd87813df80 2024-01-16T15:28:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.226+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd868012b00 con 0x7fd87813df80 2024-01-16T15:28:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.226+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8781570e0 con 0x7fd87813df80 2024-01-16T15:28:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.226+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd878154480 con 0x7fd87813df80 2024-01-16T15:28:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.227+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd8780ce850 con 0x7fd87813df80 2024-01-16T15:28:10.228 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.228+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fd868012c60 con 0x7fd87813df80 2024-01-16T15:28:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.229+0000 7fd86ea00640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85006c580 0x7fd85006ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.229+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fd868088fc0 con 0x7fd87813df80 2024-01-16T15:28:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.230+0000 7fd86ea00640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7fd850074840 0x7fd850076ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.229 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.230+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 --> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd850077370 con 0x7fd850074840 2024-01-16T15:28:10.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.230+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_get_version_reply(handle=1 version=112) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd868089c60 con 0x7fd87813df80 2024-01-16T15:28:10.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.230+0000 7fd86d200640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85006c580 0x7fd85006ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.230 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.230+0000 7fd876800640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7fd850074840 0x7fd850076ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.231+0000 7fd86d200640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85006c580 0x7fd85006ea40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd860036000 tx=0x7fd86003c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.231 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.231+0000 7fd876800640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7fd850074840 0x7fd850076ca0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.234 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.234+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== osd.7 v2:172.21.15.146:6809/902907051 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26965 (crc 0 0 0) 0x7fd850077370 con 0x7fd850074840 2024-01-16T15:28:10.248 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.249+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 --> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd878156f70 con 0x7fd850074840 2024-01-16T15:28:10.250 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.250+0000 7fd86ea00640 1 -- 172.21.15.53:0/1581241687 <== osd.7 v2:172.21.15.146:6809/902907051 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fd878156f70 con 0x7fd850074840 2024-01-16T15:28:10.250 INFO:teuthology.orchestra.run.smithi053.stdout:{ 2024-01-16T15:28:10.250 INFO:teuthology.orchestra.run.smithi053.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-01-16T15:28:10.250 INFO:teuthology.orchestra.run.smithi053.stdout:} 2024-01-16T15:28:10.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.250+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7fd850074840 msgr2=0x7fd850076ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.250+0000 7fd87d000640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6809/902907051,v1:172.21.15.146:6811/902907051] conn(0x7fd850074840 0x7fd850076ca0 crc :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.250+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85006c580 msgr2=0x7fd85006ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.251 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fd85006c580 0x7fd85006ea40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fd860036000 tx=0x7fd86003c040 comp rx=0 tx=0).stop 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813df80 msgr2=0x7fd878153f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 --2- 172.21.15.53:0/1581241687 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fd87813df80 0x7fd878153f40 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fd86800d5f0 tx=0x7fd86800dab0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd876800640 1 -- 172.21.15.53:0/1581241687 reap_dead start 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 shutdown_connections 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 >> 172.21.15.53:0/1581241687 conn(0x7fd87806a800 msgr2=0x7fd87815d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 shutdown_connections 2024-01-16T15:28:10.252 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.251+0000 7fd87d000640 1 -- 172.21.15.53:0/1581241687 wait complete. 2024-01-16T15:28:10.272 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-01-16T15:28:10.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.492+0000 7fea75800640 1 Processor -- start 2024-01-16T15:28:10.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.492+0000 7fea75800640 1 -- start start 2024-01-16T15:28:10.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea75800640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fea70146850 0x7fea700715d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea75800640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7fea70071b10 0x7fea700ad5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea75800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7006d480 con 0x7fea700adb00 2024-01-16T15:28:10.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea75800640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea70071fa0 con 0x7fea70146850 2024-01-16T15:28:10.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea75800640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea700b3490 con 0x7fea70071b10 2024-01-16T15:28:10.494 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea6fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fea70146850 0x7fea700715d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea6fe00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fea70146850 0x7fea700715d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55822/0 (socket says 172.21.15.53:55822) 2024-01-16T15:28:10.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea6fe00640 1 -- 172.21.15.53:0/3550222142 learned_addr learned my addr 172.21.15.53:0/3550222142 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:10.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.494+0000 7fea67200640 1 --2- 172.21.15.53:0/3550222142 >> v2:172.21.15.146:3300/0 conn(0x7fea70071b10 0x7fea700ad5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.495+0000 7fea6fe00640 1 -- 172.21.15.53:0/3550222142 >> v2:172.21.15.146:3300/0 conn(0x7fea70071b10 msgr2=0x7fea700ad5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:10.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.495+0000 7fea6fe00640 1 --2- 172.21.15.53:0/3550222142 >> v2:172.21.15.146:3300/0 conn(0x7fea70071b10 0x7fea700ad5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.495+0000 7fea6fe00640 1 -- 172.21.15.53:0/3550222142 >> v1:172.21.15.146:6789/0 conn(0x7fea700adb00 legacy=0x7fea700700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:10.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.495+0000 7fea6fe00640 1 -- 172.21.15.53:0/3550222142 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700b3770 con 0x7fea70146850 2024-01-16T15:28:10.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.496+0000 7fea6fe00640 1 --2- 172.21.15.53:0/3550222142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70146850 0x7fea700715d0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fea70160cc0 tx=0x7fea6002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=913176de4825ac2 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.496+0000 7fea6f400640 1 -- 172.21.15.53:0/3550222142 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6002ca80 con 0x7fea70146850 2024-01-16T15:28:10.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.496+0000 7fea6f400640 1 -- 172.21.15.53:0/3550222142 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6002cbe0 con 0x7fea70146850 2024-01-16T15:28:10.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.496+0000 7fea6f400640 1 -- 172.21.15.53:0/3550222142 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea600343f0 con 0x7fea70146850 2024-01-16T15:28:10.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 -- 172.21.15.53:0/3550222142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70146850 msgr2=0x7fea700715d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 --2- 172.21.15.53:0/3550222142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70146850 0x7fea700715d0 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fea70160cc0 tx=0x7fea6002bee0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 -- 172.21.15.53:0/3550222142 shutdown_connections 2024-01-16T15:28:10.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 --2- 172.21.15.53:0/3550222142 >> v2:172.21.15.146:3300/0 conn(0x7fea70071b10 0x7fea700ad5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 --2- 172.21.15.53:0/3550222142 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70146850 0x7fea700715d0 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 -- 172.21.15.53:0/3550222142 >> 172.21.15.53:0/3550222142 conn(0x7fea7006a3e0 msgr2=0x7fea7006a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:10.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 -- 172.21.15.53:0/3550222142 shutdown_connections 2024-01-16T15:28:10.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.497+0000 7fea75800640 1 -- 172.21.15.53:0/3550222142 wait complete. 2024-01-16T15:28:10.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.499+0000 7fea75800640 1 Processor -- start 2024-01-16T15:28:10.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.499+0000 7fea75800640 1 -- start start 2024-01-16T15:28:10.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea75800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 0x7fea700abe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea75800640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea700adb00 0x7fea700700f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea75800640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea70146850 0x7fea70070630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea75800640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700a84c0 con 0x7fea70071b10 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea75800640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7013cd00 con 0x7fea700adb00 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea75800640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700a97b0 con 0x7fea70146850 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea6fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 0x7fea700abe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea6fe00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 0x7fea700abe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55828/0 (socket says 172.21.15.53:55828) 2024-01-16T15:28:10.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.500+0000 7fea6fe00640 1 -- 172.21.15.53:0/2487199387 learned_addr learned my addr 172.21.15.53:0/2487199387 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:10.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea6fe00640 1 -- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea70146850 msgr2=0x7fea70070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.502 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea6fe00640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea70146850 0x7fea70070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea6fe00640 1 -- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea700adb00 msgr2=0x7fea700700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea67200640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea700adb00 0x7fea700700f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea6fe00640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea700adb00 0x7fea700700f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea6fe00640 1 -- 172.21.15.53:0/2487199387 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60009b30 con 0x7fea70071b10 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea67200640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea700adb00 0x7fea700700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.501+0000 7fea6fe00640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 0x7fea700abe40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fea60036040 tx=0x7fea60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.502+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea60009510 con 0x7fea70071b10 2024-01-16T15:28:10.503 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.502+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea6003e450 con 0x7fea70071b10 2024-01-16T15:28:10.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.502+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea6003e7e0 con 0x7fea70071b10 2024-01-16T15:28:10.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.502+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea700a8710 con 0x7fea70071b10 2024-01-16T15:28:10.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.502+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea700a8cd0 con 0x7fea70071b10 2024-01-16T15:28:10.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.503+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea70067450 con 0x7fea70071b10 2024-01-16T15:28:10.504 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.504+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7fea6002cd50 con 0x7fea70071b10 2024-01-16T15:28:10.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.505+0000 7fea6cc00640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4806c550 0x7fea4806ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.505 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.505+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fea600ab3a0 con 0x7fea70071b10 2024-01-16T15:28:10.506 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.506+0000 7fea67200640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4806c550 0x7fea4806ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.507 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.507+0000 7fea67200640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4806c550 0x7fea4806ea10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fea58006fd0 tx=0x7fea58008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.510 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.510+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fea60077e30 con 0x7fea70071b10 2024-01-16T15:28:10.704 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.704+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fea70146c30 con 0x7fea4806c550 2024-01-16T15:28:10.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.706+0000 7fea6cc00640 1 -- 172.21.15.53:0/2487199387 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fea58009530 con 0x7fea4806c550 2024-01-16T15:28:10.706 INFO:teuthology.orchestra.run.smithi053.stderr:instructed osd(s) 7 to deep-scrub 2024-01-16T15:28:10.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.708+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4806c550 msgr2=0x7fea4806ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.708+0000 7fea75800640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4806c550 0x7fea4806ea10 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7fea58006fd0 tx=0x7fea58008040 comp rx=0 tx=0).stop 2024-01-16T15:28:10.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.708+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 msgr2=0x7fea700abe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.708+0000 7fea75800640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 0x7fea700abe40 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fea60036040 tx=0x7fea60004a40 comp rx=0 tx=0).stop 2024-01-16T15:28:10.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 shutdown_connections 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fea4806c550 0x7fea4806ea10 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fea70146850 0x7fea70070630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fea700adb00 0x7fea700700f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 --2- 172.21.15.53:0/2487199387 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fea70071b10 0x7fea700abe40 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 >> 172.21.15.53:0/2487199387 conn(0x7fea7006a3e0 msgr2=0x7fea700b0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 shutdown_connections 2024-01-16T15:28:10.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.709+0000 7fea75800640 1 -- 172.21.15.53:0/2487199387 wait complete. 2024-01-16T15:28:10.727 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:10.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.951+0000 7f2efd600640 1 Processor -- start 2024-01-16T15:28:10.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.951+0000 7f2efd600640 1 -- start start 2024-01-16T15:28:10.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2efd600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2ef8143c40 0x7f2ef8071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2efd600640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef8070e50 con 0x7f2ef80717d0 2024-01-16T15:28:10.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2efd600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ef8144020 con 0x7f2ef80ccc10 2024-01-16T15:28:10.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2efd600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ef80723c0 con 0x7f2ef8143c40 2024-01-16T15:28:10.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2ef7e00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f2ef8143c40 0x7f2ef8071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2ef7400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f2ef80717d0 0x7f2ef80cc500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:57670/0 (socket says 172.21.15.53:57670) 2024-01-16T15:28:10.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.953+0000 7f2ef7400640 1 -- 172.21.15.53:0/2114374174 learned_addr learned my addr 172.21.15.53:0/2114374174 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:10.954 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.954+0000 7f2ef7e00640 1 -- 172.21.15.53:0/2114374174 >> v1:172.21.15.146:6789/0 conn(0x7f2ef80ccc10 legacy=0x7f2ef806d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:10.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.954+0000 7f2ef7e00640 1 -- 172.21.15.53:0/2114374174 >> v1:172.21.15.53:6789/0 conn(0x7f2ef80717d0 legacy=0x7f2ef80cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.954+0000 7f2ef6a00640 1 -- 172.21.15.53:0/2114374174 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576804678 0 0) 0x7f2ef8070e50 con 0x7f2ef80717d0 2024-01-16T15:28:10.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.954+0000 7f2ef7e00640 1 -- 172.21.15.53:0/2114374174 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ef8071e00 con 0x7f2ef8143c40 2024-01-16T15:28:10.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.955+0000 7f2ef7e00640 1 --2- 172.21.15.53:0/2114374174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef8143c40 0x7f2ef8071290 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0af0970a4700e71 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.955 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2ef6a00640 1 -- 172.21.15.53:0/2114374174 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee802e070 con 0x7f2ef8143c40 2024-01-16T15:28:10.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2ef6a00640 1 -- 172.21.15.53:0/2114374174 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee80315a0 con 0x7f2ef8143c40 2024-01-16T15:28:10.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2ef6a00640 1 -- 172.21.15.53:0/2114374174 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8031930 con 0x7f2ef8143c40 2024-01-16T15:28:10.956 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 -- 172.21.15.53:0/2114374174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef8143c40 msgr2=0x7f2ef8071290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 --2- 172.21.15.53:0/2114374174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef8143c40 0x7f2ef8071290 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee802b840 comp rx=0 tx=0).stop 2024-01-16T15:28:10.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 -- 172.21.15.53:0/2114374174 shutdown_connections 2024-01-16T15:28:10.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 --2- 172.21.15.53:0/2114374174 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef8143c40 0x7f2ef8071290 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 -- 172.21.15.53:0/2114374174 >> 172.21.15.53:0/2114374174 conn(0x7f2ef806a800 msgr2=0x7f2ef806ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:10.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 -- 172.21.15.53:0/2114374174 shutdown_connections 2024-01-16T15:28:10.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.956+0000 7f2efd600640 1 -- 172.21.15.53:0/2114374174 wait complete. 2024-01-16T15:28:10.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.958+0000 7f2efd600640 1 Processor -- start 2024-01-16T15:28:10.958 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.958+0000 7f2efd600640 1 -- start start 2024-01-16T15:28:10.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.959+0000 7f2efd600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 0x7f2ef80b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.959+0000 7f2efd600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80ccc10 0x7f2ef813c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.959+0000 7f2efd600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef8143c40 0x7f2ef813c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.959+0000 7f2efd600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8140be0 con 0x7f2ef80717d0 2024-01-16T15:28:10.960 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.959+0000 7f2efd600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef8140ec0 con 0x7f2ef80ccc10 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.959+0000 7f2efd600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ef81411a0 con 0x7f2ef8143c40 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 0x7f2ef80b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 0x7f2ef80b31c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:55842/0 (socket says 172.21.15.53:55842) 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 -- 172.21.15.53:0/3692192872 learned_addr learned my addr 172.21.15.53:0/3692192872 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2efcc00640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef8143c40 0x7f2ef813c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7400640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80ccc10 0x7f2ef813c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 -- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef8143c40 msgr2=0x7f2ef813c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef8143c40 0x7f2ef813c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 -- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80ccc10 msgr2=0x7f2ef813c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:10.961 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80ccc10 0x7f2ef813c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:10.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7e00640 1 -- 172.21.15.53:0/3692192872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8009400 con 0x7f2ef80717d0 2024-01-16T15:28:10.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.960+0000 7f2ef7400640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80ccc10 0x7f2ef813c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:10.962 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.961+0000 7f2ef7e00640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 0x7f2ef80b31c0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.961+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8004230 con 0x7f2ef80717d0 2024-01-16T15:28:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.961+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee80043b0 con 0x7f2ef80717d0 2024-01-16T15:28:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.961+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8004520 con 0x7f2ef80717d0 2024-01-16T15:28:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.961+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ef8141480 con 0x7f2ef80717d0 2024-01-16T15:28:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.961+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ef8141c80 con 0x7f2ef80717d0 2024-01-16T15:28:10.963 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.962+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ef8067450 con 0x7f2ef80717d0 2024-01-16T15:28:10.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.963+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f2ee8032880 con 0x7f2ef80717d0 2024-01-16T15:28:10.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.964+0000 7f2eefe00640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed006c550 0x7f2ed006ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:10.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.964+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2ee80aad80 con 0x7f2ef80717d0 2024-01-16T15:28:10.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.965+0000 7f2ef7400640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed006c550 0x7f2ed006ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:10.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.966+0000 7f2ef7400640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed006c550 0x7f2ed006ea10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000f60 tx=0x7f2ee0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:10.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:10.969+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f2ee80772c0 con 0x7f2ef80717d0 2024-01-16T15:28:11.160 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.160+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ef806cc40 con 0x7f2ed006c550 2024-01-16T15:28:11.184 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.184+0000 7f2eefe00640 1 -- 172.21.15.53:0/3692192872 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286716 (secure 0 0 0) 0x7f2ee0001c70 con 0x7f2ed006c550 2024-01-16T15:28:11.187 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:11.189 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.190+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed006c550 msgr2=0x7f2ed006ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.190+0000 7f2efd600640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed006c550 0x7f2ed006ea10 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f2ee0000f60 tx=0x7f2ee0009040 comp rx=0 tx=0).stop 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.190+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 msgr2=0x7f2ef80b31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.190+0000 7f2efd600640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 0x7f2ef80b31c0 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097a0 tx=0x7f2ee80046c0 comp rx=0 tx=0).stop 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 shutdown_connections 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f2ed006c550 0x7f2ed006ea10 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f2ef8143c40 0x7f2ef813c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2ef80ccc10 0x7f2ef813c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:11.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 --2- 172.21.15.53:0/3692192872 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f2ef80717d0 0x7f2ef80b31c0 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 >> 172.21.15.53:0/3692192872 conn(0x7f2ef806a800 msgr2=0x7f2ef806abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 shutdown_connections 2024-01-16T15:28:11.192 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:11.191+0000 7f2efd600640 1 -- 172.21.15.53:0/3692192872 wait complete. 2024-01-16T15:28:11.211 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":958,"stamp":"2024-01-16T15:28:10.910479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4911,"ondisk_log_size":4911,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11125304,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":625011,"kb_avail":743849416,"statfs":{"total":773094113280,"available":761701801984,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":640011824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2942,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004635"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257209+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.257209+0000","last_peered":"2024-01-16T15:27:49.257209+0000","last_clean":"2024-01-16T15:27:49.257209+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.257209+0000","last_undegraded":"2024-01-16T15:27:49.257209+0000","last_fullsized":"2024-01-16T15:27:49.257209+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'73","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237782+0000","last_change":"2024-01-16T15:19:24.622288+0000","last_active":"2024-01-16T15:27:49.237782+0000","last_peered":"2024-01-16T15:27:49.237782+0000","last_clean":"2024-01-16T15:27:49.237782+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:27:49.237782+0000","last_undegraded":"2024-01-16T15:27:49.237782+0000","last_fullsized":"2024-01-16T15:27:49.237782+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:26:20.176733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251487+0000","last_change":"2024-01-16T15:19:24.618862+0000","last_active":"2024-01-16T15:27:49.251487+0000","last_peered":"2024-01-16T15:27:49.251487+0000","last_clean":"2024-01-16T15:27:49.251487+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:27:49.251487+0000","last_undegraded":"2024-01-16T15:27:49.251487+0000","last_fullsized":"2024-01-16T15:27:49.251487+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:00:02.115872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'56","reported_seq":80,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233027+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:27:49.233027+0000","last_peered":"2024-01-16T15:27:49.233027+0000","last_clean":"2024-01-16T15:27:49.233027+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:27:49.233027+0000","last_undegraded":"2024-01-16T15:27:49.233027+0000","last_fullsized":"2024-01-16T15:27:49.233027+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'62","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233733+0000","last_change":"2024-01-16T15:19:24.626093+0000","last_active":"2024-01-16T15:27:49.233733+0000","last_peered":"2024-01-16T15:27:49.233733+0000","last_clean":"2024-01-16T15:27:49.233733+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:27:49.233733+0000","last_undegraded":"2024-01-16T15:27:49.233733+0000","last_fullsized":"2024-01-16T15:27:49.233733+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'76","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.240041+0000","last_change":"2024-01-16T15:19:24.625809+0000","last_active":"2024-01-16T15:27:49.240041+0000","last_peered":"2024-01-16T15:27:49.240041+0000","last_clean":"2024-01-16T15:27:49.240041+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:27:49.240041+0000","last_undegraded":"2024-01-16T15:27:49.240041+0000","last_fullsized":"2024-01-16T15:27:49.240041+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'67","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243297+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:27:49.243297+0000","last_peered":"2024-01-16T15:27:49.243297+0000","last_clean":"2024-01-16T15:27:49.243297+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:27:49.243297+0000","last_undegraded":"2024-01-16T15:27:49.243297+0000","last_fullsized":"2024-01-16T15:27:49.243297+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'58","reported_seq":82,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248998+0000","last_change":"2024-01-16T15:19:24.616757+0000","last_active":"2024-01-16T15:27:49.248998+0000","last_peered":"2024-01-16T15:27:49.248998+0000","last_clean":"2024-01-16T15:27:49.248998+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:27:49.248998+0000","last_undegraded":"2024-01-16T15:27:49.248998+0000","last_fullsized":"2024-01-16T15:27:49.248998+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:25:30.687663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257379+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:27:49.257379+0000","last_peered":"2024-01-16T15:27:49.257379+0000","last_clean":"2024-01-16T15:27:49.257379+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:27:49.257379+0000","last_undegraded":"2024-01-16T15:27:49.257379+0000","last_fullsized":"2024-01-16T15:27:49.257379+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257401+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:27:49.257401+0000","last_peered":"2024-01-16T15:27:49.257401+0000","last_clean":"2024-01-16T15:27:49.257401+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:27:49.257401+0000","last_undegraded":"2024-01-16T15:27:49.257401+0000","last_fullsized":"2024-01-16T15:27:49.257401+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'70","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.227737+0000","last_change":"2024-01-16T15:19:24.625137+0000","last_active":"2024-01-16T15:27:49.227737+0000","last_peered":"2024-01-16T15:27:49.227737+0000","last_clean":"2024-01-16T15:27:49.227737+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:27:49.227737+0000","last_undegraded":"2024-01-16T15:27:49.227737+0000","last_fullsized":"2024-01-16T15:27:49.227737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:58:37.949290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'64","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236372+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:27:49.236372+0000","last_peered":"2024-01-16T15:27:49.236372+0000","last_clean":"2024-01-16T15:27:49.236372+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:27:49.236372+0000","last_undegraded":"2024-01-16T15:27:49.236372+0000","last_fullsized":"2024-01-16T15:27:49.236372+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'65","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242745+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:27:49.242745+0000","last_peered":"2024-01-16T15:27:49.242745+0000","last_clean":"2024-01-16T15:27:49.242745+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:27:49.242745+0000","last_undegraded":"2024-01-16T15:27:49.242745+0000","last_fullsized":"2024-01-16T15:27:49.242745+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:08.270535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'72","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.245113+0000","last_change":"2024-01-16T15:19:24.624351+0000","last_active":"2024-01-16T15:27:49.245113+0000","last_peered":"2024-01-16T15:27:49.245113+0000","last_clean":"2024-01-16T15:27:49.245113+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:27:49.245113+0000","last_undegraded":"2024-01-16T15:27:49.245113+0000","last_fullsized":"2024-01-16T15:27:49.245113+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:40:41.925967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'59","reported_seq":80,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246541+0000","last_change":"2024-01-16T15:19:24.613444+0000","last_active":"2024-01-16T15:27:49.246541+0000","last_peered":"2024-01-16T15:27:49.246541+0000","last_clean":"2024-01-16T15:27:49.246541+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:27:49.246541+0000","last_undegraded":"2024-01-16T15:27:49.246541+0000","last_fullsized":"2024-01-16T15:27:49.246541+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'74","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250236+0000","last_change":"2024-01-16T15:19:24.614694+0000","last_active":"2024-01-16T15:27:49.250236+0000","last_peered":"2024-01-16T15:27:49.250236+0000","last_clean":"2024-01-16T15:27:49.250236+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:27:49.250236+0000","last_undegraded":"2024-01-16T15:27:49.250236+0000","last_fullsized":"2024-01-16T15:27:49.250236+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'52","reported_seq":73,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.228316+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:27:49.228316+0000","last_peered":"2024-01-16T15:27:49.228316+0000","last_clean":"2024-01-16T15:27:49.228316+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:27:49.228316+0000","last_undegraded":"2024-01-16T15:27:49.228316+0000","last_fullsized":"2024-01-16T15:27:49.228316+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:36:48.890952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.235929+0000","last_change":"2024-01-16T15:19:24.629022+0000","last_active":"2024-01-16T15:27:49.235929+0000","last_peered":"2024-01-16T15:27:49.235929+0000","last_clean":"2024-01-16T15:27:49.235929+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:27:49.235929+0000","last_undegraded":"2024-01-16T15:27:49.235929+0000","last_fullsized":"2024-01-16T15:27:49.235929+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:27:22.444157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'74","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250216+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:27:49.250216+0000","last_peered":"2024-01-16T15:27:49.250216+0000","last_clean":"2024-01-16T15:27:49.250216+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:27:49.250216+0000","last_undegraded":"2024-01-16T15:27:49.250216+0000","last_fullsized":"2024-01-16T15:27:49.250216+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:47:11.257113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'70","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247569+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:27:49.247569+0000","last_peered":"2024-01-16T15:27:49.247569+0000","last_clean":"2024-01-16T15:27:49.247569+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:27:49.247569+0000","last_undegraded":"2024-01-16T15:27:49.247569+0000","last_fullsized":"2024-01-16T15:27:49.247569+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:23:44.308866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'73","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248711+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:27:49.248711+0000","last_peered":"2024-01-16T15:27:49.248711+0000","last_clean":"2024-01-16T15:27:49.248711+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:27:49.248711+0000","last_undegraded":"2024-01-16T15:27:49.248711+0000","last_fullsized":"2024-01-16T15:27:49.248711+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:31:35.142028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'71","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.235873+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:27:49.235873+0000","last_peered":"2024-01-16T15:27:49.235873+0000","last_clean":"2024-01-16T15:27:49.235873+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:27:49.235873+0000","last_undegraded":"2024-01-16T15:27:49.235873+0000","last_fullsized":"2024-01-16T15:27:49.235873+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'63","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247674+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:27:49.247674+0000","last_peered":"2024-01-16T15:27:49.247674+0000","last_clean":"2024-01-16T15:27:49.247674+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:27:49.247674+0000","last_undegraded":"2024-01-16T15:27:49.247674+0000","last_fullsized":"2024-01-16T15:27:49.247674+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:07.051272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'71","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.363290+0000","last_change":"2024-01-16T15:19:24.626920+0000","last_active":"2024-01-16T15:28:05.363290+0000","last_peered":"2024-01-16T15:28:05.363290+0000","last_clean":"2024-01-16T15:28:05.363290+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:28:05.363290+0000","last_undegraded":"2024-01-16T15:28:05.363290+0000","last_fullsized":"2024-01-16T15:28:05.363290+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'69","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233562+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:27:49.233562+0000","last_peered":"2024-01-16T15:27:49.233562+0000","last_clean":"2024-01-16T15:27:49.233562+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:27:49.233562+0000","last_undegraded":"2024-01-16T15:27:49.233562+0000","last_fullsized":"2024-01-16T15:27:49.233562+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:36:39.208617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566596+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:28:05.566596+0000","last_peered":"2024-01-16T15:28:05.566596+0000","last_clean":"2024-01-16T15:28:05.566596+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:28:05.566596+0000","last_undegraded":"2024-01-16T15:28:05.566596+0000","last_fullsized":"2024-01-16T15:28:05.566596+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'65","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.611088+0000","last_change":"2024-01-16T15:19:24.623489+0000","last_active":"2024-01-16T15:28:05.611088+0000","last_peered":"2024-01-16T15:28:05.611088+0000","last_clean":"2024-01-16T15:28:05.611088+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:28:05.611088+0000","last_undegraded":"2024-01-16T15:28:05.611088+0000","last_fullsized":"2024-01-16T15:28:05.611088+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'50","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.363355+0000","last_change":"2024-01-16T15:19:42.218585+0000","last_active":"2024-01-16T15:28:05.363355+0000","last_peered":"2024-01-16T15:28:05.363355+0000","last_clean":"2024-01-16T15:28:05.363355+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:28:05.363355+0000","last_undegraded":"2024-01-16T15:28:05.363355+0000","last_fullsized":"2024-01-16T15:28:05.363355+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'62","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248953+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:27:49.248953+0000","last_peered":"2024-01-16T15:27:49.248953+0000","last_clean":"2024-01-16T15:27:49.248953+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:27:49.248953+0000","last_undegraded":"2024-01-16T15:27:49.248953+0000","last_fullsized":"2024-01-16T15:27:49.248953+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:40.862803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'78","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246774+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:27:49.246774+0000","last_peered":"2024-01-16T15:27:49.246774+0000","last_clean":"2024-01-16T15:27:49.246774+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:27:49.246774+0000","last_undegraded":"2024-01-16T15:27:49.246774+0000","last_fullsized":"2024-01-16T15:27:49.246774+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'91","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.252230+0000","last_change":"2024-01-16T15:19:24.629910+0000","last_active":"2024-01-16T15:27:49.252230+0000","last_peered":"2024-01-16T15:27:49.252230+0000","last_clean":"2024-01-16T15:27:49.252230+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:27:49.252230+0000","last_undegraded":"2024-01-16T15:27:49.252230+0000","last_fullsized":"2024-01-16T15:27:49.252230+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'78","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246939+0000","last_change":"2024-01-16T15:19:24.633567+0000","last_active":"2024-01-16T15:27:49.246939+0000","last_peered":"2024-01-16T15:27:49.246939+0000","last_clean":"2024-01-16T15:27:49.246939+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:27:49.246939+0000","last_undegraded":"2024-01-16T15:27:49.246939+0000","last_fullsized":"2024-01-16T15:27:49.246939+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'60","reported_seq":84,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.231843+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:27:49.231843+0000","last_peered":"2024-01-16T15:27:49.231843+0000","last_clean":"2024-01-16T15:27:49.231843+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:27:49.231843+0000","last_undegraded":"2024-01-16T15:27:49.231843+0000","last_fullsized":"2024-01-16T15:27:49.231843+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":23,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.613766+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:27:45.613766+0000","last_peered":"2024-01-16T15:27:45.613766+0000","last_clean":"2024-01-16T15:27:45.613766+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:27:45.613766+0000","last_undegraded":"2024-01-16T15:27:45.613766+0000","last_fullsized":"2024-01-16T15:27:45.613766+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'67","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.247582+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:27:49.247582+0000","last_peered":"2024-01-16T15:27:49.247582+0000","last_clean":"2024-01-16T15:27:49.247582+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:27:49.247582+0000","last_undegraded":"2024-01-16T15:27:49.247582+0000","last_fullsized":"2024-01-16T15:27:49.247582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:33:25.905229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:25.933509+0000","last_change":"2024-01-16T15:19:23.621310+0000","last_active":"2024-01-16T15:25:25.933509+0000","last_peered":"2024-01-16T15:25:25.933509+0000","last_clean":"2024-01-16T15:25:25.933509+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:25:25.933509+0000","last_undegraded":"2024-01-16T15:25:25.933509+0000","last_fullsized":"2024-01-16T15:25:25.933509+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:34:19.958286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'66","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246144+0000","last_change":"2024-01-16T15:19:24.630161+0000","last_active":"2024-01-16T15:27:49.246144+0000","last_peered":"2024-01-16T15:27:49.246144+0000","last_clean":"2024-01-16T15:27:49.246144+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:27:49.246144+0000","last_undegraded":"2024-01-16T15:27:49.246144+0000","last_fullsized":"2024-01-16T15:27:49.246144+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'3","reported_seq":21,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:55.591888+0000","last_change":"2024-01-16T15:19:23.630011+0000","last_active":"2024-01-16T15:26:55.591888+0000","last_peered":"2024-01-16T15:26:55.591888+0000","last_clean":"2024-01-16T15:26:55.591888+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:26:55.591888+0000","last_undegraded":"2024-01-16T15:26:55.591888+0000","last_fullsized":"2024-01-16T15:26:55.591888+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'84","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250676+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:27:49.250676+0000","last_peered":"2024-01-16T15:27:49.250676+0000","last_clean":"2024-01-16T15:27:49.250676+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:27:49.250676+0000","last_undegraded":"2024-01-16T15:27:49.250676+0000","last_fullsized":"2024-01-16T15:27:49.250676+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:53.269911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"109'2","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:55.591861+0000","last_change":"2024-01-16T15:19:23.618597+0000","last_active":"2024-01-16T15:26:55.591861+0000","last_peered":"2024-01-16T15:26:55.591861+0000","last_clean":"2024-01-16T15:26:55.591861+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:26:55.591861+0000","last_undegraded":"2024-01-16T15:26:55.591861+0000","last_fullsized":"2024-01-16T15:26:55.591861+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'78","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250706+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:27:49.250706+0000","last_peered":"2024-01-16T15:27:49.250706+0000","last_clean":"2024-01-16T15:27:49.250706+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:27:49.250706+0000","last_undegraded":"2024-01-16T15:27:49.250706+0000","last_fullsized":"2024-01-16T15:27:49.250706+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:55.604663+0000","last_change":"2024-01-16T15:19:23.618024+0000","last_active":"2024-01-16T15:26:55.604663+0000","last_peered":"2024-01-16T15:26:55.604663+0000","last_clean":"2024-01-16T15:26:55.604663+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:26:55.604663+0000","last_undegraded":"2024-01-16T15:26:55.604663+0000","last_fullsized":"2024-01-16T15:26:55.604663+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'82","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234990+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:27:49.234990+0000","last_peered":"2024-01-16T15:27:49.234990+0000","last_clean":"2024-01-16T15:27:49.234990+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:27:49.234990+0000","last_undegraded":"2024-01-16T15:27:49.234990+0000","last_fullsized":"2024-01-16T15:27:49.234990+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:06:55.328226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:55.350264+0000","last_change":"2024-01-16T15:19:23.622641+0000","last_active":"2024-01-16T15:26:55.350264+0000","last_peered":"2024-01-16T15:26:55.350264+0000","last_clean":"2024-01-16T15:26:55.350264+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:26:55.350264+0000","last_undegraded":"2024-01-16T15:26:55.350264+0000","last_fullsized":"2024-01-16T15:26:55.350264+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'80","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250263+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:27:49.250263+0000","last_peered":"2024-01-16T15:27:49.250263+0000","last_clean":"2024-01-16T15:27:49.250263+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:27:49.250263+0000","last_undegraded":"2024-01-16T15:27:49.250263+0000","last_fullsized":"2024-01-16T15:27:49.250263+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:16.744702+0000","last_change":"2024-01-16T15:19:23.620941+0000","last_active":"2024-01-16T15:24:16.744702+0000","last_peered":"2024-01-16T15:24:16.744702+0000","last_clean":"2024-01-16T15:24:16.744702+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:24:16.744702+0000","last_undegraded":"2024-01-16T15:24:16.744702+0000","last_fullsized":"2024-01-16T15:24:16.744702+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:52:27.971375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'68","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234221+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:27:49.234221+0000","last_peered":"2024-01-16T15:27:49.234221+0000","last_clean":"2024-01-16T15:27:49.234221+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:27:49.234221+0000","last_undegraded":"2024-01-16T15:27:49.234221+0000","last_fullsized":"2024-01-16T15:27:49.234221+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:40.619278+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:26:40.619278+0000","last_peered":"2024-01-16T15:26:40.619278+0000","last_clean":"2024-01-16T15:26:40.619278+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:26:40.619278+0000","last_undegraded":"2024-01-16T15:26:40.619278+0000","last_fullsized":"2024-01-16T15:26:40.619278+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'66","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242520+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:27:49.242520+0000","last_peered":"2024-01-16T15:27:49.242520+0000","last_clean":"2024-01-16T15:27:49.242520+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:27:49.242520+0000","last_undegraded":"2024-01-16T15:27:49.242520+0000","last_fullsized":"2024-01-16T15:27:49.242520+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:40.589127+0000","last_change":"2024-01-16T15:19:23.621670+0000","last_active":"2024-01-16T15:26:40.589127+0000","last_peered":"2024-01-16T15:26:40.589127+0000","last_clean":"2024-01-16T15:26:40.589127+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:26:40.589127+0000","last_undegraded":"2024-01-16T15:26:40.589127+0000","last_fullsized":"2024-01-16T15:26:40.589127+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'64","reported_seq":85,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.237836+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:27:49.237836+0000","last_peered":"2024-01-16T15:27:49.237836+0000","last_clean":"2024-01-16T15:27:49.237836+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:27:49.237836+0000","last_undegraded":"2024-01-16T15:27:49.237836+0000","last_fullsized":"2024-01-16T15:27:49.237836+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:12:56.499150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:40.347273+0000","last_change":"2024-01-16T15:19:23.624332+0000","last_active":"2024-01-16T15:26:40.347273+0000","last_peered":"2024-01-16T15:26:40.347273+0000","last_clean":"2024-01-16T15:26:40.347273+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:26:40.347273+0000","last_undegraded":"2024-01-16T15:26:40.347273+0000","last_fullsized":"2024-01-16T15:26:40.347273+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'60","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244408+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:27:49.244408+0000","last_peered":"2024-01-16T15:27:49.244408+0000","last_clean":"2024-01-16T15:27:49.244408+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:27:49.244408+0000","last_undegraded":"2024-01-16T15:27:49.244408+0000","last_fullsized":"2024-01-16T15:27:49.244408+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'31","reported_seq":49,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:40.619414+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:26:40.619414+0000","last_peered":"2024-01-16T15:26:40.619414+0000","last_clean":"2024-01-16T15:26:40.619414+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:26:40.619414+0000","last_undegraded":"2024-01-16T15:26:40.619414+0000","last_fullsized":"2024-01-16T15:26:40.619414+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'69","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251406+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:27:49.251406+0000","last_peered":"2024-01-16T15:27:49.251406+0000","last_clean":"2024-01-16T15:27:49.251406+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:27:49.251406+0000","last_undegraded":"2024-01-16T15:27:49.251406+0000","last_fullsized":"2024-01-16T15:27:49.251406+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'307","reported_seq":325,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.261078+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:27:59.261078+0000","last_peered":"2024-01-16T15:27:59.261078+0000","last_clean":"2024-01-16T15:27:59.261078+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:27:59.261078+0000","last_undegraded":"2024-01-16T15:27:59.261078+0000","last_fullsized":"2024-01-16T15:27:59.261078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'63","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242943+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:27:49.242943+0000","last_peered":"2024-01-16T15:27:49.242943+0000","last_clean":"2024-01-16T15:27:49.242943+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:27:49.242943+0000","last_undegraded":"2024-01-16T15:27:49.242943+0000","last_fullsized":"2024-01-16T15:27:49.242943+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"109'2","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:25.594144+0000","last_change":"2024-01-16T15:19:23.617724+0000","last_active":"2024-01-16T15:26:25.594144+0000","last_peered":"2024-01-16T15:26:25.594144+0000","last_clean":"2024-01-16T15:26:25.594144+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:26:25.594144+0000","last_undegraded":"2024-01-16T15:26:25.594144+0000","last_fullsized":"2024-01-16T15:26:25.594144+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'80","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249986+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:27:49.249986+0000","last_peered":"2024-01-16T15:27:49.249986+0000","last_clean":"2024-01-16T15:27:49.249986+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:27:49.249986+0000","last_undegraded":"2024-01-16T15:27:49.249986+0000","last_fullsized":"2024-01-16T15:27:49.249986+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":17,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:24:07.195535+0000","last_change":"2024-01-16T15:19:23.624683+0000","last_active":"2024-01-16T15:24:07.195535+0000","last_peered":"2024-01-16T15:24:07.195535+0000","last_clean":"2024-01-16T15:24:07.195535+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:24:07.195535+0000","last_undegraded":"2024-01-16T15:24:07.195535+0000","last_fullsized":"2024-01-16T15:24:07.195535+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:26:38.379463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'73","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244737+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:27:49.244737+0000","last_peered":"2024-01-16T15:27:49.244737+0000","last_clean":"2024-01-16T15:27:49.244737+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:27:49.244737+0000","last_undegraded":"2024-01-16T15:27:49.244737+0000","last_fullsized":"2024-01-16T15:27:49.244737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'12","reported_seq":38,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.825417+0000","last_change":"2024-01-16T15:28:07.825453+0000","last_active":"2024-01-16T15:28:07.825417+0000","last_peered":"2024-01-16T15:28:07.825417+0000","last_clean":"2024-01-16T15:28:07.825417+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:28:07.825417+0000","last_undegraded":"2024-01-16T15:28:07.825417+0000","last_fullsized":"2024-01-16T15:28:07.825417+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.825335+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:20.349610+0000","scrub_duration":0.0056472520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'50","reported_seq":71,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250507+0000","last_change":"2024-01-16T15:19:24.617249+0000","last_active":"2024-01-16T15:27:49.250507+0000","last_peered":"2024-01-16T15:27:49.250507+0000","last_clean":"2024-01-16T15:27:49.250507+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:27:49.250507+0000","last_undegraded":"2024-01-16T15:27:49.250507+0000","last_fullsized":"2024-01-16T15:27:49.250507+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:10.613699+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:26:10.613699+0000","last_peered":"2024-01-16T15:26:10.613699+0000","last_clean":"2024-01-16T15:26:10.613699+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:26:10.613699+0000","last_undegraded":"2024-01-16T15:26:10.613699+0000","last_fullsized":"2024-01-16T15:26:10.613699+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'61","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.239704+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:27:49.239704+0000","last_peered":"2024-01-16T15:27:49.239704+0000","last_clean":"2024-01-16T15:27:49.239704+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:27:49.239704+0000","last_undegraded":"2024-01-16T15:27:49.239704+0000","last_fullsized":"2024-01-16T15:27:49.239704+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"109'2","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:10.584275+0000","last_change":"2024-01-16T15:19:23.622230+0000","last_active":"2024-01-16T15:26:10.584275+0000","last_peered":"2024-01-16T15:26:10.584275+0000","last_clean":"2024-01-16T15:26:10.584275+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:26:10.584275+0000","last_undegraded":"2024-01-16T15:26:10.584275+0000","last_fullsized":"2024-01-16T15:26:10.584275+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'77","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248743+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:27:49.248743+0000","last_peered":"2024-01-16T15:27:49.248743+0000","last_clean":"2024-01-16T15:27:49.248743+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:27:49.248743+0000","last_undegraded":"2024-01-16T15:27:49.248743+0000","last_fullsized":"2024-01-16T15:27:49.248743+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:49:31.587040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"109'1","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:23:57.193814+0000","last_change":"2024-01-16T15:19:23.626903+0000","last_active":"2024-01-16T15:23:57.193814+0000","last_peered":"2024-01-16T15:23:57.193814+0000","last_clean":"2024-01-16T15:23:57.193814+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:23:57.193814+0000","last_undegraded":"2024-01-16T15:23:57.193814+0000","last_fullsized":"2024-01-16T15:23:57.193814+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:07:58.751536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'66","reported_seq":94,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251742+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:27:49.251742+0000","last_peered":"2024-01-16T15:27:49.251742+0000","last_clean":"2024-01-16T15:27:49.251742+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:27:49.251742+0000","last_undegraded":"2024-01-16T15:27:49.251742+0000","last_fullsized":"2024-01-16T15:27:49.251742+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:35:32.314064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:10.596763+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:26:10.596763+0000","last_peered":"2024-01-16T15:26:10.596763+0000","last_clean":"2024-01-16T15:26:10.596763+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:26:10.596763+0000","last_undegraded":"2024-01-16T15:26:10.596763+0000","last_fullsized":"2024-01-16T15:26:10.596763+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'62","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249715+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:27:49.249715+0000","last_peered":"2024-01-16T15:27:49.249715+0000","last_clean":"2024-01-16T15:27:49.249715+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:27:49.249715+0000","last_undegraded":"2024-01-16T15:27:49.249715+0000","last_fullsized":"2024-01-16T15:27:49.249715+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:32.038856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'3","reported_seq":21,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:10.591820+0000","last_change":"2024-01-16T15:19:23.621285+0000","last_active":"2024-01-16T15:26:10.591820+0000","last_peered":"2024-01-16T15:26:10.591820+0000","last_clean":"2024-01-16T15:26:10.591820+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:26:10.591820+0000","last_undegraded":"2024-01-16T15:26:10.591820+0000","last_fullsized":"2024-01-16T15:26:10.591820+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249922+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:27:49.249922+0000","last_peered":"2024-01-16T15:27:49.249922+0000","last_clean":"2024-01-16T15:27:49.249922+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:27:49.249922+0000","last_undegraded":"2024-01-16T15:27:49.249922+0000","last_fullsized":"2024-01-16T15:27:49.249922+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.603487+0000","last_change":"2024-01-16T15:19:23.619619+0000","last_active":"2024-01-16T15:28:05.603487+0000","last_peered":"2024-01-16T15:28:05.603487+0000","last_clean":"2024-01-16T15:28:05.603487+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:28:05.603487+0000","last_undegraded":"2024-01-16T15:28:05.603487+0000","last_fullsized":"2024-01-16T15:28:05.603487+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'66","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249721+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:27:49.249721+0000","last_peered":"2024-01-16T15:27:49.249721+0000","last_clean":"2024-01-16T15:27:49.249721+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:27:49.249721+0000","last_undegraded":"2024-01-16T15:27:49.249721+0000","last_fullsized":"2024-01-16T15:27:49.249721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:01:58.872650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.611020+0000","last_change":"2024-01-16T15:19:23.620635+0000","last_active":"2024-01-16T15:28:05.611020+0000","last_peered":"2024-01-16T15:28:05.611020+0000","last_clean":"2024-01-16T15:28:05.611020+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:28:05.611020+0000","last_undegraded":"2024-01-16T15:28:05.611020+0000","last_fullsized":"2024-01-16T15:28:05.611020+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'63","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.363321+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:28:05.363321+0000","last_peered":"2024-01-16T15:28:05.363321+0000","last_clean":"2024-01-16T15:28:05.363321+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:28:05.363321+0000","last_undegraded":"2024-01-16T15:28:05.363321+0000","last_fullsized":"2024-01-16T15:28:05.363321+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566086+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:28:05.566086+0000","last_peered":"2024-01-16T15:28:05.566086+0000","last_clean":"2024-01-16T15:28:05.566086+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:28:05.566086+0000","last_undegraded":"2024-01-16T15:28:05.566086+0000","last_fullsized":"2024-01-16T15:28:05.566086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'70","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236890+0000","last_change":"2024-01-16T15:19:42.592735+0000","last_active":"2024-01-16T15:27:49.236890+0000","last_peered":"2024-01-16T15:27:49.236890+0000","last_clean":"2024-01-16T15:27:49.236890+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:27:49.236890+0000","last_undegraded":"2024-01-16T15:27:49.236890+0000","last_fullsized":"2024-01-16T15:27:49.236890+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:43:07.215380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"109'2","reported_seq":21,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.363384+0000","last_change":"2024-01-16T15:19:23.630170+0000","last_active":"2024-01-16T15:28:05.363384+0000","last_peered":"2024-01-16T15:28:05.363384+0000","last_clean":"2024-01-16T15:28:05.363384+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:28:05.363384+0000","last_undegraded":"2024-01-16T15:28:05.363384+0000","last_fullsized":"2024-01-16T15:28:05.363384+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566405+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:28:05.566405+0000","last_peered":"2024-01-16T15:28:05.566405+0000","last_clean":"2024-01-16T15:28:05.566405+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:28:05.566405+0000","last_undegraded":"2024-01-16T15:28:05.566405+0000","last_fullsized":"2024-01-16T15:28:05.566405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":49,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.255261+0000","last_change":"2024-01-16T15:19:23.623778+0000","last_active":"2024-01-16T15:27:49.255261+0000","last_peered":"2024-01-16T15:27:49.255261+0000","last_clean":"2024-01-16T15:27:49.255261+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:27:49.255261+0000","last_undegraded":"2024-01-16T15:27:49.255261+0000","last_fullsized":"2024-01-16T15:27:49.255261+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:33:53.783590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'69","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.245051+0000","last_change":"2024-01-16T15:19:24.628361+0000","last_active":"2024-01-16T15:27:49.245051+0000","last_peered":"2024-01-16T15:27:49.245051+0000","last_clean":"2024-01-16T15:27:49.245051+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:27:49.245051+0000","last_undegraded":"2024-01-16T15:27:49.245051+0000","last_fullsized":"2024-01-16T15:27:49.245051+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.632990+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:27:45.632990+0000","last_peered":"2024-01-16T15:27:45.632990+0000","last_clean":"2024-01-16T15:27:45.632990+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:27:45.632990+0000","last_undegraded":"2024-01-16T15:27:45.632990+0000","last_fullsized":"2024-01-16T15:27:45.632990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'53","reported_seq":76,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236803+0000","last_change":"2024-01-16T15:19:24.623591+0000","last_active":"2024-01-16T15:27:49.236803+0000","last_peered":"2024-01-16T15:27:49.236803+0000","last_clean":"2024-01-16T15:27:49.236803+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:27:49.236803+0000","last_undegraded":"2024-01-16T15:27:49.236803+0000","last_fullsized":"2024-01-16T15:27:49.236803+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:23:57.706980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.359987+0000","last_change":"2024-01-16T15:19:23.612616+0000","last_active":"2024-01-16T15:27:45.359987+0000","last_peered":"2024-01-16T15:27:45.359987+0000","last_clean":"2024-01-16T15:27:45.359987+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:27:45.359987+0000","last_undegraded":"2024-01-16T15:27:45.359987+0000","last_fullsized":"2024-01-16T15:27:45.359987+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'65","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242630+0000","last_change":"2024-01-16T15:19:24.626321+0000","last_active":"2024-01-16T15:27:49.242630+0000","last_peered":"2024-01-16T15:27:49.242630+0000","last_clean":"2024-01-16T15:27:49.242630+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:27:49.242630+0000","last_undegraded":"2024-01-16T15:27:49.242630+0000","last_fullsized":"2024-01-16T15:27:49.242630+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:10.387772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.613633+0000","last_change":"2024-01-16T15:19:23.624071+0000","last_active":"2024-01-16T15:27:45.613633+0000","last_peered":"2024-01-16T15:27:45.613633+0000","last_clean":"2024-01-16T15:27:45.613633+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:27:45.613633+0000","last_undegraded":"2024-01-16T15:27:45.613633+0000","last_fullsized":"2024-01-16T15:27:45.613633+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'64","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236252+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:27:49.236252+0000","last_peered":"2024-01-16T15:27:49.236252+0000","last_clean":"2024-01-16T15:27:49.236252+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:27:49.236252+0000","last_undegraded":"2024-01-16T15:27:49.236252+0000","last_fullsized":"2024-01-16T15:27:49.236252+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:33:08.966374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:41.760446+0000","last_change":"2024-01-16T15:19:23.619253+0000","last_active":"2024-01-16T15:25:41.760446+0000","last_peered":"2024-01-16T15:25:41.760446+0000","last_clean":"2024-01-16T15:25:41.760446+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:25:41.760446+0000","last_undegraded":"2024-01-16T15:25:41.760446+0000","last_fullsized":"2024-01-16T15:25:41.760446+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:19:16.781683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'73","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243648+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:27:49.243648+0000","last_peered":"2024-01-16T15:27:49.243648+0000","last_clean":"2024-01-16T15:27:49.243648+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:27:49.243648+0000","last_undegraded":"2024-01-16T15:27:49.243648+0000","last_fullsized":"2024-01-16T15:27:49.243648+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:54:49.637451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'31","reported_seq":80,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.599879+0000","last_change":"2024-01-16T15:19:23.629871+0000","last_active":"2024-01-16T15:27:45.599879+0000","last_peered":"2024-01-16T15:27:45.599879+0000","last_clean":"2024-01-16T15:27:45.599879+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:27:45.599879+0000","last_undegraded":"2024-01-16T15:27:45.599879+0000","last_fullsized":"2024-01-16T15:27:45.599879+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'65","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251468+0000","last_change":"2024-01-16T15:19:24.621286+0000","last_active":"2024-01-16T15:27:49.251468+0000","last_peered":"2024-01-16T15:27:49.251468+0000","last_clean":"2024-01-16T15:27:49.251468+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:27:49.251468+0000","last_undegraded":"2024-01-16T15:27:49.251468+0000","last_fullsized":"2024-01-16T15:27:49.251468+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:25.628146+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:27:25.628146+0000","last_peered":"2024-01-16T15:27:25.628146+0000","last_clean":"2024-01-16T15:27:25.628146+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:27:25.628146+0000","last_undegraded":"2024-01-16T15:27:25.628146+0000","last_fullsized":"2024-01-16T15:27:25.628146+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'54","reported_seq":79,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249912+0000","last_change":"2024-01-16T15:19:24.621294+0000","last_active":"2024-01-16T15:27:49.249912+0000","last_peered":"2024-01-16T15:27:49.249912+0000","last_clean":"2024-01-16T15:27:49.249912+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:27:49.249912+0000","last_undegraded":"2024-01-16T15:27:49.249912+0000","last_fullsized":"2024-01-16T15:27:49.249912+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":54,"log_dups_size":0,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:25.599209+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:26:25.599209+0000","last_peered":"2024-01-16T15:26:25.599209+0000","last_clean":"2024-01-16T15:26:25.599209+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:26:25.599209+0000","last_undegraded":"2024-01-16T15:26:25.599209+0000","last_fullsized":"2024-01-16T15:26:25.599209+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"18'117","reported_seq":261,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:25:50.938466+0000","last_change":"2024-01-16T14:59:48.055388+0000","last_active":"2024-01-16T15:25:50.938466+0000","last_peered":"2024-01-16T15:25:50.938466+0000","last_clean":"2024-01-16T15:25:50.938466+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:25:50.938466+0000","last_undegraded":"2024-01-16T15:25:50.938466+0000","last_fullsized":"2024-01-16T15:25:50.938466+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T14:59:46.775374+0000","last_clean_scrub_stamp":"2024-01-16T14:59:46.775374+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:09:21.761934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4358,"ondisk_log_size":4358,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":436,"ondisk_log_size":436,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509782,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.1559999999999999,"15min":1.25},"min":{"1min":0.69899999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5409999999999999,"5min":1.7749999999999999,"15min":19.224},"last":1.085},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.212,"15min":1.3140000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.615,"5min":1.694,"15min":28.603999999999999},"last":1.6140000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.22,"15min":1.2250000000000001},"min":{"1min":0.90500000000000003,"5min":0.80800000000000005,"15min":0.374},"max":{"1min":1.488,"5min":1.837,"15min":3.7069999999999999},"last":1.296},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.3080000000000001,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.68899999999999995,"15min":0.68400000000000005},"max":{"1min":1.651,"5min":1.8819999999999999,"15min":33.225000000000001},"last":1.339}]},{"osd":2,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3380000000000001,"15min":1.4199999999999999},"min":{"1min":1.0529999999999999,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.752,"5min":1.7909999999999999,"15min":19.126000000000001},"last":1.4339999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3080000000000001,"15min":1.3120000000000001},"min":{"1min":1.008,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.694,"5min":1.718,"15min":2.9750000000000001},"last":1.7190000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.76500000000000001},"max":{"1min":1.526,"5min":1.73,"15min":3.7999999999999998},"last":1.3979999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.446,"15min":1.5580000000000001},"min":{"1min":0.97399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":19.268999999999998},"last":1.25}]},{"osd":4,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2290000000000001,"15min":1.264},"min":{"1min":0.88300000000000001,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.794},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.347,"15min":1.3919999999999999},"min":{"1min":1.1180000000000001,"5min":0.70299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6659999999999999,"5min":1.7330000000000001,"15min":9.7059999999999995},"last":1.6930000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":1.0389999999999999,"5min":0.86099999999999999,"15min":0.79400000000000004},"max":{"1min":1.74,"5min":1.9450000000000001,"15min":3.9129999999999998},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.5620000000000001},"min":{"1min":0.82699999999999996,"5min":0.77900000000000003,"15min":0.55100000000000005},"max":{"1min":1.8720000000000001,"5min":1.9630000000000001,"15min":19.036999999999999},"last":1.5129999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.4359999999999999,"15min":1.4410000000000001},"min":{"1min":1.198,"5min":0.74299999999999999,"15min":0.54800000000000004},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":4.2240000000000002},"last":1.8580000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.528,"15min":1.571},"min":{"1min":1.224,"5min":0.98999999999999999,"15min":0.65100000000000002},"max":{"1min":1.728,"5min":2.181,"15min":4.0709999999999997},"last":1.9099999999999999}]}]},{"osd":4,"up_from":15,"seq":64424509784,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.296,"15min":1.2330000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.427,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.159},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1799999999999999,"15min":1.1859999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.032}]},{"osd":1,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.88700000000000001,"5min":0.73899999999999999,"15min":0.71899999999999997},"max":{"1min":1.6160000000000001,"5min":1.893,"15min":2.0510000000000002},"last":1.119},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.3149999999999999,"15min":1.302},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.6319999999999999,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.0649999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3420000000000001,"15min":1.3320000000000001},"min":{"1min":0.79300000000000004,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.573,"5min":1.78,"15min":5.907},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.242,"5min":1.381,"15min":1.355},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.52600000000000002},"max":{"1min":1.6000000000000001,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":0.83099999999999996}]},{"osd":3,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4570000000000001,"15min":1.3939999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.76200000000000001},"max":{"1min":1.5289999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":0.91300000000000003},{"interface":"front","average":{"1min":1.391,"5min":1.4019999999999999,"15min":1.391},"min":{"1min":1.052,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.74,"5min":1.9670000000000001,"15min":2.0379999999999998},"last":1.3660000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.425,"15min":1.448},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6779999999999999,"5min":6.165,"15min":6.165},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4410000000000001,"15min":1.4770000000000001},"min":{"1min":0.95299999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8180000000000001,"5min":3.1469999999999998,"15min":5.3520000000000003},"last":1.4510000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.391,"15min":1.4219999999999999},"min":{"1min":1.101,"5min":0.90800000000000003,"15min":0.497},"max":{"1min":1.903,"5min":1.903,"15min":8.9600000000000009},"last":1.5049999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5640000000000001,"15min":1.528},"min":{"1min":1.165,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.772,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.5429999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.452,"15min":1.492},"min":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.89700000000000002},"max":{"1min":1.8500000000000001,"5min":3.3620000000000001,"15min":5.835},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.5469999999999999,"15min":1.581},"min":{"1min":1.238,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":2.0070000000000001,"5min":2.3679999999999999,"15min":8.0079999999999991},"last":1.74}]}]},{"osd":7,"up_from":14,"seq":60129542486,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1891948,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92479892,"statfs":{"total":96636764160,"available":94699409408,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1659999999999999,"15min":1.2130000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.65400000000000003},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.1779999999999999},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.24,"5min":1.2210000000000001,"15min":1.248},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":12.461},"last":1.0960000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.89000000000000001,"5min":0.85099999999999998,"15min":0.48599999999999999},"max":{"1min":1.923,"5min":1.923,"15min":2.4249999999999998},"last":0.78400000000000003},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2709999999999999,"15min":1.363},"min":{"1min":0.93400000000000005,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":1.8140000000000001,"5min":1.9079999999999999,"15min":14.848000000000001},"last":1.3069999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3069999999999999,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":3.726},"last":0.82599999999999996},{"interface":"front","average":{"1min":1.286,"5min":1.3340000000000001,"15min":1.3959999999999999},"min":{"1min":0.88900000000000001,"5min":0.873,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.792,"15min":19.763999999999999},"last":0.92700000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3340000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80600000000000005},"max":{"1min":1.6539999999999999,"5min":1.8280000000000001,"15min":1.887},"last":0.999},{"interface":"front","average":{"1min":1.456,"5min":1.47,"15min":1.4650000000000001},"min":{"1min":1.113,"5min":1.0169999999999999,"15min":0.629},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":6.6029999999999998},"last":1.04}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.345,"15min":1.347},"min":{"1min":0.98499999999999999,"5min":0.89100000000000001,"15min":0.54100000000000004},"max":{"1min":1.7090000000000001,"5min":1.7929999999999999,"15min":4.1390000000000002},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.401,"15min":1.413},"min":{"1min":1.0009999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":1.873,"5min":1.877,"15min":2.5190000000000001},"last":1.149}]},{"osd":5,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3340000000000001,"15min":1.3680000000000001},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":12.609999999999999},"last":1.4039999999999999},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5229999999999999},"min":{"1min":1.1719999999999999,"5min":1.046,"15min":0.66000000000000003},"max":{"1min":1.8879999999999999,"5min":1.909,"15min":3.4159999999999999},"last":1.351}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.099,"5min":1.0760000000000001,"15min":0.66000000000000003},"max":{"1min":2.0859999999999999,"5min":2.1429999999999998,"15min":2.653},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.552},"min":{"1min":1.004,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":12.567},"last":1.6240000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542488,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.208},"min":{"1min":0.83399999999999996,"5min":0.749,"15min":0.32900000000000001},"max":{"1min":1.7190000000000001,"5min":2.0939999999999999,"15min":6.8380000000000001},"last":1.4199999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1870000000000001,"15min":1.3200000000000001},"min":{"1min":1.0049999999999999,"5min":0.82999999999999996,"15min":0.57099999999999995},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":23.199999999999999},"last":1.0269999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.246},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.63400000000000001},"max":{"1min":1.637,"5min":1.698,"15min":8.0909999999999993},"last":0.72399999999999998},{"interface":"front","average":{"1min":1.355,"5min":1.26,"15min":1.323},"min":{"1min":0.93600000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8049999999999999,"5min":1.851,"15min":17.062000000000001},"last":1.6080000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.409,"15min":1.415},"min":{"1min":1.069,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.6970000000000001,"5min":1.782,"15min":5.8810000000000002},"last":1.526},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":1.131,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.6899999999999999,"5min":1.6970000000000001,"15min":5.4429999999999996},"last":1.268}]},{"osd":3,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.302,"15min":1.3420000000000001},"min":{"1min":0.94999999999999996,"5min":0.878,"15min":0.67500000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":7.984},"last":0.94199999999999995},{"interface":"front","average":{"1min":1.385,"5min":1.4179999999999999,"15min":1.423},"min":{"1min":0.94599999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.7410000000000001,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":1.2250000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.323,"15min":1.387},"min":{"1min":0.878,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":15.975},"last":1.304},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4079999999999999,"15min":1.413},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.54000000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":5.6349999999999998},"last":1.6719999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.329,"15min":1.3680000000000001},"min":{"1min":0.95999999999999996,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.8160000000000001,"5min":1.827,"15min":12.914},"last":1.165},{"interface":"front","average":{"1min":1.536,"5min":1.468,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":16.552},"last":1.581}]},{"osd":7,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4630000000000001,"15min":1.506},"min":{"1min":1.298,"5min":0.98099999999999998,"15min":0.64300000000000002},"max":{"1min":1.74,"5min":1.8540000000000001,"15min":13.034000000000001},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5740000000000001,"15min":1.5900000000000001},"min":{"1min":1.161,"5min":1.0429999999999999,"15min":0.70899999999999996},"max":{"1min":1.9159999999999999,"5min":2.073,"15min":8.0380000000000003},"last":1.472}]}]},{"osd":3,"up_from":14,"seq":60129542488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":5.7400000000000002},"last":1.3},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.1359999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40600000000000003},"max":{"1min":1.506,"5min":1.879,"15min":7.1429999999999998},"last":1.391}]},{"osd":1,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.96899999999999997,"15min":1.0349999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.365,"5min":1.738,"15min":10.856999999999999},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.1870000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4950000000000001,"5min":1.823,"15min":2.0089999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.3140000000000001,"15min":1.5209999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.946,"5min":1.946,"15min":39.265999999999998},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1559999999999999,"15min":1.2070000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.381,"5min":1.833,"15min":7.9160000000000004},"last":1.0029999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.337,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.70999999999999996},"max":{"1min":1.5980000000000001,"5min":1.9419999999999999,"15min":5.6890000000000001},"last":1.169},{"interface":"front","average":{"1min":1.329,"5min":1.464,"15min":1.4810000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8120000000000001,"5min":2.0920000000000001,"15min":2.3330000000000002},"last":1.0900000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3440000000000001,"15min":1.4670000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5669999999999999,"5min":1.827,"15min":14.003},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.357,"5min":1.5049999999999999,"15min":1.577},"min":{"1min":0.877,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":1.9359999999999999,"5min":2.008,"15min":6.0679999999999996},"last":0.879}]},{"osd":6,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.464,"15min":1.502},"min":{"1min":0.875,"5min":0.875,"15min":0.69799999999999995},"max":{"1min":1.8899999999999999,"5min":2.0139999999999998,"15min":7.0060000000000002},"last":1.133},{"interface":"front","average":{"1min":1.381,"5min":1.486,"15min":1.5580000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8140000000000001,"5min":1.982,"15min":15.114000000000001},"last":1.5680000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4950000000000001,"15min":1.6160000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80400000000000005},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":12.378},"last":1.248},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5780000000000001,"15min":1.6220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9650000000000001,"5min":2.23,"15min":4.6029999999999998},"last":1.504}]}]},{"osd":2,"up_from":14,"seq":60129542487,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1059928,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93311912,"statfs":{"total":96636764160,"available":95551397888,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.90500000000000003,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.41199999999999998,"15min":0.32400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.161,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":1.833,"5min":1.913,"15min":7.9960000000000004},"last":0.63300000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0880000000000001,"15min":1.1040000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":1.7130000000000001,"5min":2.294,"15min":3.056},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.171,"5min":1.143,"15min":1.175},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.556,"5min":1.877,"15min":4.3819999999999997},"last":0.73199999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.206,"15min":1.2789999999999999},"min":{"1min":0.61199999999999999,"5min":0.55300000000000005,"15min":0.44700000000000001},"max":{"1min":1.704,"5min":1.883,"15min":18.838000000000001},"last":0.96299999999999997},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4950000000000001,"15min":1.552},"min":{"1min":0.66200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":2.0099999999999998,"15min":19.375},"last":0.98199999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.335,"15min":1.4390000000000001},"min":{"1min":0.80600000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.8240000000000001,"5min":2.242,"15min":23.225000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4730000000000001,"15min":1.494},"min":{"1min":1.038,"5min":0.75900000000000001,"15min":0.72699999999999998},"max":{"1min":1.883,"5min":1.883,"15min":6.359},"last":0.88800000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4399999999999999,"15min":1.5009999999999999},"min":{"1min":1.0800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.917,"5min":2.3170000000000002,"15min":20.157},"last":0.92200000000000004},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.597,"15min":1.6100000000000001},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.623},"max":{"1min":1.9330000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.083}]},{"osd":6,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5289999999999999,"15min":1.595},"min":{"1min":1.018,"5min":0.89800000000000002,"15min":0.70499999999999996},"max":{"1min":1.7330000000000001,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.012},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6160000000000001,"15min":1.7170000000000001},"min":{"1min":0.85799999999999998,"5min":0.79300000000000004,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.1579999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5429999999999999,"15min":1.6699999999999999},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.587,"5min":1.569,"15min":1.5880000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":1.8999999999999999,"5min":2.0990000000000002,"15min":4.3419999999999996},"last":1.2150000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542489,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.093,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.28100000000000003},"max":{"1min":13.42,"5min":13.42,"15min":13.42},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.216,"5min":1.141,"15min":1.113},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":0.745}]},{"osd":2,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.03,"15min":1.0900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.649,"15min":3.6819999999999999},"last":0.439},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.29,"15min":1.3320000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.39000000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":3.5600000000000001},"last":1.127}]},{"osd":3,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.26,"15min":1.2889999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46300000000000002},"max":{"1min":1.488,"5min":1.875,"15min":2.238},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.077,"5min":1.137,"15min":1.194},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":1.0229999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.095},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.72199999999999998},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.2490000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3939999999999999,"15min":1.431},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63700000000000001},"max":{"1min":1.6359999999999999,"5min":1.952,"15min":3.98},"last":1.1479999999999999},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4530000000000001,"15min":1.462},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.53300000000000003},"max":{"1min":1.669,"5min":1.893,"15min":3.3940000000000001},"last":1.226}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.56999999999999995},"max":{"1min":1.8660000000000001,"5min":2.016,"15min":16.48},"last":1.2949999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.508,"15min":1.532},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":0.65600000000000003},"max":{"1min":1.74,"5min":1.909,"15min":3.8180000000000001},"last":0.99399999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.573,"15min":1.617},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.48099999999999998},"max":{"1min":1.8480000000000001,"5min":2.0009999999999999,"15min":7.0499999999999998},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.6339999999999999,"15min":1.647},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.81100000000000005},"max":{"1min":1.8640000000000001,"5min":2.3940000000000001,"15min":3.7839999999999998},"last":1.1990000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542487,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":1.038},"min":{"1min":0.496,"5min":0.496,"15min":0.35199999999999998},"max":{"1min":1.427,"5min":1.665,"15min":15.776},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.151,"15min":1.171},"min":{"1min":0.67600000000000005,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":1.9019999999999999,"5min":2.7549999999999999,"15min":17.620999999999999},"last":0.78300000000000003}]},{"osd":2,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.137,"15min":1.22},"min":{"1min":0.74099999999999999,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.671,"5min":1.788,"15min":15.997999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.23,"15min":1.1919999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":1.669,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":0.91600000000000004}]},{"osd":3,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.129,"15min":1.179},"min":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":1.256},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.4279999999999999},"min":{"1min":0.86299999999999999,"5min":0.66600000000000004,"15min":0.47399999999999998},"max":{"1min":1.74,"5min":1.827,"15min":16.126000000000001},"last":1.546}]},{"osd":4,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.466,"15min":1.573},"min":{"1min":1.095,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.9470000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.665},{"interface":"front","average":{"1min":1.446,"5min":1.4430000000000001,"15min":1.419},"min":{"1min":1.0089999999999999,"5min":0.82299999999999995,"15min":0.71899999999999997},"max":{"1min":1.913,"5min":3.0099999999999998,"15min":3.0459999999999998},"last":1.488}]},{"osd":5,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.97099999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7869999999999999,"5min":2.157,"15min":5.5609999999999999},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.581,"5min":1.5529999999999999,"15min":1.639},"min":{"1min":1.2130000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0630000000000002,"5min":2.1989999999999998,"15min":16.501999999999999},"last":1.8799999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.137,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.891,"5min":2.0790000000000002,"15min":5.4509999999999996},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.649,"5min":1.6379999999999999,"15min":1.7190000000000001},"min":{"1min":0.98899999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":2.1269999999999998,"5min":2.351,"15min":16.395},"last":1.8009999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5800000000000001,"15min":1.595},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":2.3300000000000001,"5min":2.9510000000000001,"15min":9.6980000000000004},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.589,"15min":1.5860000000000001},"min":{"1min":1.0860000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.954,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.3080000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:11.217 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.218 INFO:tasks.ceph:pgid 17.3e last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.218 INFO:tasks.ceph:pgid 17.3d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.218 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.218 INFO:tasks.ceph:pgid 17.3b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.218 INFO:tasks.ceph:pgid 17.3a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.38 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.35 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.219 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.220 INFO:tasks.ceph:pgid 17.32 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.220 INFO:tasks.ceph:pgid 17.31 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.220 INFO:tasks.ceph:pgid 17.30 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.220 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.220 INFO:tasks.ceph:pgid 17.2e last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.220 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.28 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.221 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.222 INFO:tasks.ceph:pgid 17.25 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.222 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.222 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.222 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.222 INFO:tasks.ceph:pgid 17.21 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.222 INFO:tasks.ceph:pgid 17.20 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 16.1e last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 16.1d last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.223 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.224 INFO:tasks.ceph:pgid 16.1c last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.224 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.224 INFO:tasks.ceph:pgid 16.1b last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.224 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.224 INFO:tasks.ceph:pgid 16.1a last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.224 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 16.19 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 16.17 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.225 INFO:tasks.ceph:pgid 16.16 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 16.13 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.226 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 16.12 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 16.e last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.227 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 16.d last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 16.b last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.228 INFO:tasks.ceph:pgid 16.a last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 16.9 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 16.7 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.229 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 16.6 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 16.4 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.230 INFO:tasks.ceph:pgid 16.3 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 16.2 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 16.1 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.231 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.232 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-01-16T14:59:46.775374+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=14, tm_min=59, tm_sec=46, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:11.232 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:28:29.623 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:29.635 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:28:31.233 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:31.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.460+0000 7f1850a00640 1 Processor -- start 2024-01-16T15:28:31.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.460+0000 7f1850a00640 1 -- start start 2024-01-16T15:28:31.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f1850a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c143c40 0x7f184c071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:31.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f1850a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f184c0ccc10 0x7f184c06d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:31.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f1850a00640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184c070e50 con 0x7f184c0717d0 2024-01-16T15:28:31.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f1850a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f184c0723c0 con 0x7f184c0ccc10 2024-01-16T15:28:31.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f1850a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c144020 con 0x7f184c143c40 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f184b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f184c0ccc10 0x7f184c06d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f184b200640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f184c0ccc10 0x7f184c06d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33966/0 (socket says 172.21.15.53:33966) 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.462+0000 7f184b200640 1 -- 172.21.15.53:0/118664685 learned_addr learned my addr 172.21.15.53:0/118664685 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.463+0000 7f184a800640 1 --2- 172.21.15.53:0/118664685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c143c40 0x7f184c071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.463+0000 7f184b200640 1 -- 172.21.15.53:0/118664685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c143c40 msgr2=0x7f184c071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.463+0000 7f184b200640 1 --2- 172.21.15.53:0/118664685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c143c40 0x7f184c071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.463+0000 7f184b200640 1 -- 172.21.15.53:0/118664685 >> v1:172.21.15.146:6789/0 conn(0x7f184c0717d0 legacy=0x7f184c0cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:31.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.463+0000 7f184b200640 1 -- 172.21.15.53:0/118664685 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184c071e00 con 0x7f184c0ccc10 2024-01-16T15:28:31.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.464+0000 7f184b200640 1 --2- 172.21.15.53:0/118664685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0ccc10 0x7f184c06d160 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e50 tx=0x7f184402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721e59e804c9725d server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:31.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.464+0000 7f1849e00640 1 -- 172.21.15.53:0/118664685 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f184402c9e0 con 0x7f184c0ccc10 2024-01-16T15:28:31.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1849e00640 1 -- 172.21.15.53:0/118664685 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f184402cb40 con 0x7f184c0ccc10 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1849e00640 1 -- 172.21.15.53:0/118664685 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18440343f0 con 0x7f184c0ccc10 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1850a00640 1 -- 172.21.15.53:0/118664685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0ccc10 msgr2=0x7f184c06d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1850a00640 1 --2- 172.21.15.53:0/118664685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0ccc10 0x7f184c06d160 secure :-1 s=READY pgs=2066 cs=0 l=1 rev1=1 crypto rx=0x7f1844009e50 tx=0x7f184402be40 comp rx=0 tx=0).stop 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1850a00640 1 -- 172.21.15.53:0/118664685 shutdown_connections 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1850a00640 1 --2- 172.21.15.53:0/118664685 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0ccc10 0x7f184c06d160 unknown :-1 s=CLOSED pgs=2066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1850a00640 1 --2- 172.21.15.53:0/118664685 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c143c40 0x7f184c071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.465+0000 7f1850a00640 1 -- 172.21.15.53:0/118664685 >> 172.21.15.53:0/118664685 conn(0x7f184c06a800 msgr2=0x7f184c06ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:31.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.466+0000 7f1850a00640 1 -- 172.21.15.53:0/118664685 shutdown_connections 2024-01-16T15:28:31.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.466+0000 7f1850a00640 1 -- 172.21.15.53:0/118664685 wait complete. 2024-01-16T15:28:31.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.467+0000 7f1850a00640 1 Processor -- start 2024-01-16T15:28:31.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.467+0000 7f1850a00640 1 -- start start 2024-01-16T15:28:31.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.468+0000 7f1850a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 0x7f184c1e04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:31.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.468+0000 7f1850a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c0ccc10 0x7f184c13ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:31.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.468+0000 7f1850a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c143c40 0x7f184c13eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:31.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.468+0000 7f1850a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c1e0a30 con 0x7f184c0717d0 2024-01-16T15:28:31.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.468+0000 7f1850a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184c1e5740 con 0x7f184c143c40 2024-01-16T15:28:31.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.468+0000 7f1850a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184c1e5a20 con 0x7f184c0ccc10 2024-01-16T15:28:31.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 0x7f184c1e04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:31.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 0x7f184c1e04f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33976/0 (socket says 172.21.15.53:33976) 2024-01-16T15:28:31.469 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 -- 172.21.15.53:0/19252612 learned_addr learned my addr 172.21.15.53:0/19252612 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:31.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184b200640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c143c40 0x7f184c13eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 -- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c0ccc10 msgr2=0x7f184c13ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c0ccc10 0x7f184c13ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 -- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c143c40 msgr2=0x7f184c13eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c143c40 0x7f184c13eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.469+0000 7f184a800640 1 -- 172.21.15.53:0/19252612 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1844009b30 con 0x7f184c0717d0 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.470+0000 7f184a800640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 0x7f184c1e04f0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f183c00d5f0 tx=0x7f183c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:31.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.470+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c0125f0 con 0x7f184c0717d0 2024-01-16T15:28:31.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.470+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183c012750 con 0x7f184c0717d0 2024-01-16T15:28:31.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.470+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183c012b00 con 0x7f184c0717d0 2024-01-16T15:28:31.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.470+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184c1e5d00 con 0x7f184c0717d0 2024-01-16T15:28:31.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.470+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184c1e62c0 con 0x7f184c0717d0 2024-01-16T15:28:31.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.471+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184c067450 con 0x7f184c0717d0 2024-01-16T15:28:31.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.472+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f183c012c60 con 0x7f184c0717d0 2024-01-16T15:28:31.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.473+0000 7f1842a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f182406c550 0x7f182406ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:31.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.473+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f183c089080 con 0x7f184c0717d0 2024-01-16T15:28:31.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.474+0000 7f1843e00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f182406c550 0x7f182406ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:31.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.475+0000 7f1843e00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f182406c550 0x7f182406ea10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f184c06e020 tx=0x7f18340079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:31.477 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.478+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f183c0555c0 con 0x7f184c0717d0 2024-01-16T15:28:31.669 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.669+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f184c06cc40 con 0x7f182406c550 2024-01-16T15:28:31.691 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.692+0000 7f1842a00640 1 -- 172.21.15.53:0/19252612 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288044 (secure 0 0 0) 0x7f1834014400 con 0x7f182406c550 2024-01-16T15:28:31.695 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:31.697 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.697+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f182406c550 msgr2=0x7f182406ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.697+0000 7f1850a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f182406c550 0x7f182406ea10 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f184c06e020 tx=0x7f18340079b0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 msgr2=0x7f184c1e04f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 0x7f184c1e04f0 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7f183c00d5f0 tx=0x7f183c00dab0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 shutdown_connections 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f182406c550 0x7f182406ea10 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f184c143c40 0x7f184c13eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f184c0ccc10 0x7f184c13ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.699 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 --2- 172.21.15.53:0/19252612 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f184c0717d0 0x7f184c1e04f0 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:31.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 >> 172.21.15.53:0/19252612 conn(0x7f184c06a800 msgr2=0x7f184c144160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:31.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 shutdown_connections 2024-01-16T15:28:31.700 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:31.698+0000 7f1850a00640 1 -- 172.21.15.53:0/19252612 wait complete. 2024-01-16T15:28:31.717 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":968,"stamp":"2024-01-16T15:28:30.918958+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5368,"ondisk_log_size":5368,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11125304,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":625011,"kb_avail":743849416,"statfs":{"total":773094113280,"available":761701801984,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":640011824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005122"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257209+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.257209+0000","last_peered":"2024-01-16T15:27:49.257209+0000","last_clean":"2024-01-16T15:27:49.257209+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.257209+0000","last_undegraded":"2024-01-16T15:27:49.257209+0000","last_fullsized":"2024-01-16T15:27:49.257209+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'97","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.510978+0000","last_change":"2024-01-16T15:28:18.511076+0000","last_active":"2024-01-16T15:28:18.510978+0000","last_peered":"2024-01-16T15:28:18.510978+0000","last_clean":"2024-01-16T15:28:18.510978+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:28:18.510978+0000","last_undegraded":"2024-01-16T15:28:18.510978+0000","last_fullsized":"2024-01-16T15:28:18.510978+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'97","last_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_deep_scrub":"112'97","last_deep_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.510802+0000","objects_scrubbed":24,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:33.903169+0000","scrub_duration":0.27194245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'87","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:23.860238+0000","last_change":"2024-01-16T15:28:23.860329+0000","last_active":"2024-01-16T15:28:23.860238+0000","last_peered":"2024-01-16T15:28:23.860238+0000","last_clean":"2024-01-16T15:28:23.860238+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:28:23.860238+0000","last_undegraded":"2024-01-16T15:28:23.860238+0000","last_fullsized":"2024-01-16T15:28:23.860238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_clean_scrub_stamp":"2024-01-16T15:28:23.860048+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:49.716426+0000","scrub_duration":0.27335353899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'56","reported_seq":80,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233027+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:27:49.233027+0000","last_peered":"2024-01-16T15:27:49.233027+0000","last_clean":"2024-01-16T15:27:49.233027+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:27:49.233027+0000","last_undegraded":"2024-01-16T15:27:49.233027+0000","last_fullsized":"2024-01-16T15:27:49.233027+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'78","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.070128+0000","last_change":"2024-01-16T15:28:25.070176+0000","last_active":"2024-01-16T15:28:25.070128+0000","last_peered":"2024-01-16T15:28:25.070128+0000","last_clean":"2024-01-16T15:28:25.070128+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:28:25.070128+0000","last_undegraded":"2024-01-16T15:28:25.070128+0000","last_fullsized":"2024-01-16T15:28:25.070128+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'78","last_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_deep_scrub":"112'78","last_deep_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.070058+0000","objects_scrubbed":16,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:36.495682+0000","scrub_duration":0.150879281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'99","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.440173+0000","last_change":"2024-01-16T15:28:27.440281+0000","last_active":"2024-01-16T15:28:27.440173+0000","last_peered":"2024-01-16T15:28:27.440173+0000","last_clean":"2024-01-16T15:28:27.440173+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:28:27.440173+0000","last_undegraded":"2024-01-16T15:28:27.440173+0000","last_fullsized":"2024-01-16T15:28:27.440173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.439995+0000","objects_scrubbed":23,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:38.865619+0000","scrub_duration":0.25882440899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'67","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243297+0000","last_change":"2024-01-16T15:19:24.626631+0000","last_active":"2024-01-16T15:27:49.243297+0000","last_peered":"2024-01-16T15:27:49.243297+0000","last_clean":"2024-01-16T15:27:49.243297+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:27:49.243297+0000","last_undegraded":"2024-01-16T15:27:49.243297+0000","last_fullsized":"2024-01-16T15:27:49.243297+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'79","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.799942+0000","last_change":"2024-01-16T15:28:24.800039+0000","last_active":"2024-01-16T15:28:24.799942+0000","last_peered":"2024-01-16T15:28:24.799942+0000","last_clean":"2024-01-16T15:28:24.799942+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:28:24.799942+0000","last_undegraded":"2024-01-16T15:28:24.799942+0000","last_fullsized":"2024-01-16T15:28:24.799942+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.799761+0000","objects_scrubbed":21,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:58.738844+0000","scrub_duration":0.232299336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257379+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:27:49.257379+0000","last_peered":"2024-01-16T15:27:49.257379+0000","last_clean":"2024-01-16T15:27:49.257379+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:27:49.257379+0000","last_undegraded":"2024-01-16T15:27:49.257379+0000","last_fullsized":"2024-01-16T15:27:49.257379+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257401+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:27:49.257401+0000","last_peered":"2024-01-16T15:27:49.257401+0000","last_clean":"2024-01-16T15:27:49.257401+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:27:49.257401+0000","last_undegraded":"2024-01-16T15:27:49.257401+0000","last_fullsized":"2024-01-16T15:27:49.257401+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'93","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.541892+0000","last_change":"2024-01-16T15:28:26.541954+0000","last_active":"2024-01-16T15:28:26.541892+0000","last_peered":"2024-01-16T15:28:26.541892+0000","last_clean":"2024-01-16T15:28:26.541892+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:28:26.541892+0000","last_undegraded":"2024-01-16T15:28:26.541892+0000","last_fullsized":"2024-01-16T15:28:26.541892+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.541809+0000","objects_scrubbed":23,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:57.056238+0000","scrub_duration":0.29071314399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'64","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236372+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:27:49.236372+0000","last_peered":"2024-01-16T15:27:49.236372+0000","last_clean":"2024-01-16T15:27:49.236372+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:27:49.236372+0000","last_undegraded":"2024-01-16T15:27:49.236372+0000","last_fullsized":"2024-01-16T15:27:49.236372+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'65","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242820+0000","last_change":"2024-01-16T15:19:24.621994+0000","last_active":"2024-01-16T15:27:49.242820+0000","last_peered":"2024-01-16T15:27:49.242820+0000","last_clean":"2024-01-16T15:27:49.242820+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:27:49.242820+0000","last_undegraded":"2024-01-16T15:27:49.242820+0000","last_fullsized":"2024-01-16T15:27:49.242820+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'94","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.914292+0000","last_change":"2024-01-16T15:28:21.914459+0000","last_active":"2024-01-16T15:28:21.914292+0000","last_peered":"2024-01-16T15:28:21.914292+0000","last_clean":"2024-01-16T15:28:21.914292+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:28:21.914292+0000","last_undegraded":"2024-01-16T15:28:21.914292+0000","last_fullsized":"2024-01-16T15:28:21.914292+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.914080+0000","objects_scrubbed":22,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:00.078053+0000","scrub_duration":0.280246104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'71","reported_seq":100,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.408221+0000","last_change":"2024-01-16T15:28:24.408300+0000","last_active":"2024-01-16T15:28:24.408221+0000","last_peered":"2024-01-16T15:28:24.408221+0000","last_clean":"2024-01-16T15:28:24.408221+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:28:24.408221+0000","last_undegraded":"2024-01-16T15:28:24.408221+0000","last_fullsized":"2024-01-16T15:28:24.408221+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'71","last_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_deep_scrub":"112'71","last_deep_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.408030+0000","objects_scrubbed":12,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:51.797494+0000","scrub_duration":0.124741884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'99","reported_seq":128,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.255148+0000","last_change":"2024-01-16T15:28:18.255207+0000","last_active":"2024-01-16T15:28:18.255148+0000","last_peered":"2024-01-16T15:28:18.255148+0000","last_clean":"2024-01-16T15:28:18.255148+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:28:18.255148+0000","last_undegraded":"2024-01-16T15:28:18.255148+0000","last_fullsized":"2024-01-16T15:28:18.255148+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.254974+0000","objects_scrubbed":25,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:57.777992+0000","scrub_duration":0.28941331399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'52","reported_seq":74,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.228460+0000","last_change":"2024-01-16T15:19:24.626123+0000","last_active":"2024-01-16T15:27:49.228460+0000","last_peered":"2024-01-16T15:27:49.228460+0000","last_clean":"2024-01-16T15:27:49.228460+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:27:49.228460+0000","last_undegraded":"2024-01-16T15:27:49.228460+0000","last_fullsized":"2024-01-16T15:27:49.228460+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":52,"log_dups_size":0,"ondisk_log_size":52,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'108","reported_seq":151,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.741001+0000","last_change":"2024-01-16T15:28:26.741028+0000","last_active":"2024-01-16T15:28:26.741001+0000","last_peered":"2024-01-16T15:28:26.741001+0000","last_clean":"2024-01-16T15:28:26.741001+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:28:26.741001+0000","last_undegraded":"2024-01-16T15:28:26.741001+0000","last_fullsized":"2024-01-16T15:28:26.741001+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.740929+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:29.967191+0000","scrub_duration":0.29640463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'74","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.708135+0000","last_change":"2024-01-16T15:19:24.621920+0000","last_active":"2024-01-16T15:28:10.708135+0000","last_peered":"2024-01-16T15:28:10.708135+0000","last_clean":"2024-01-16T15:28:10.708135+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:28:10.708135+0000","last_undegraded":"2024-01-16T15:28:10.708135+0000","last_fullsized":"2024-01-16T15:28:10.708135+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'70","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.497021+0000","last_change":"2024-01-16T15:19:24.620463+0000","last_active":"2024-01-16T15:28:08.497021+0000","last_peered":"2024-01-16T15:28:08.497021+0000","last_clean":"2024-01-16T15:28:08.497021+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:28:08.497021+0000","last_undegraded":"2024-01-16T15:28:08.497021+0000","last_fullsized":"2024-01-16T15:28:08.497021+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'73","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.497052+0000","last_change":"2024-01-16T15:19:24.621554+0000","last_active":"2024-01-16T15:28:08.497052+0000","last_peered":"2024-01-16T15:28:08.497052+0000","last_clean":"2024-01-16T15:28:08.497052+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:28:08.497052+0000","last_undegraded":"2024-01-16T15:28:08.497052+0000","last_fullsized":"2024-01-16T15:28:08.497052+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'71","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.771329+0000","last_change":"2024-01-16T15:19:24.624730+0000","last_active":"2024-01-16T15:28:07.771329+0000","last_peered":"2024-01-16T15:28:07.771329+0000","last_clean":"2024-01-16T15:28:07.771329+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:28:07.771329+0000","last_undegraded":"2024-01-16T15:28:07.771329+0000","last_fullsized":"2024-01-16T15:28:07.771329+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'63","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:08.496946+0000","last_change":"2024-01-16T15:19:24.618409+0000","last_active":"2024-01-16T15:28:08.496946+0000","last_peered":"2024-01-16T15:28:08.496946+0000","last_clean":"2024-01-16T15:28:08.496946+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:28:08.496946+0000","last_undegraded":"2024-01-16T15:28:08.496946+0000","last_fullsized":"2024-01-16T15:28:08.496946+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'90","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.951640+0000","last_change":"2024-01-16T15:28:20.951722+0000","last_active":"2024-01-16T15:28:20.951640+0000","last_peered":"2024-01-16T15:28:20.951640+0000","last_clean":"2024-01-16T15:28:20.951640+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:28:20.951640+0000","last_undegraded":"2024-01-16T15:28:20.951640+0000","last_fullsized":"2024-01-16T15:28:20.951640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.951444+0000","objects_scrubbed":19,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:52.025141+0000","scrub_duration":0.20406614100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'69","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:06.788554+0000","last_change":"2024-01-16T15:19:24.626660+0000","last_active":"2024-01-16T15:28:06.788554+0000","last_peered":"2024-01-16T15:28:06.788554+0000","last_clean":"2024-01-16T15:28:06.788554+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:28:06.788554+0000","last_undegraded":"2024-01-16T15:28:06.788554+0000","last_fullsized":"2024-01-16T15:28:06.788554+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566596+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:28:05.566596+0000","last_peered":"2024-01-16T15:28:05.566596+0000","last_clean":"2024-01-16T15:28:05.566596+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:28:05.566596+0000","last_undegraded":"2024-01-16T15:28:05.566596+0000","last_fullsized":"2024-01-16T15:28:05.566596+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.302192+0000","last_change":"2024-01-16T15:28:28.302272+0000","last_active":"2024-01-16T15:28:28.302192+0000","last_peered":"2024-01-16T15:28:28.302192+0000","last_clean":"2024-01-16T15:28:28.302192+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:28:28.302192+0000","last_undegraded":"2024-01-16T15:28:28.302192+0000","last_fullsized":"2024-01-16T15:28:28.302192+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.302044+0000","objects_scrubbed":16,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:38.499053+0000","scrub_duration":0.149915824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'50","reported_seq":78,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.363355+0000","last_change":"2024-01-16T15:19:42.218585+0000","last_active":"2024-01-16T15:28:05.363355+0000","last_peered":"2024-01-16T15:28:05.363355+0000","last_clean":"2024-01-16T15:28:05.363355+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:28:05.363355+0000","last_undegraded":"2024-01-16T15:28:05.363355+0000","last_fullsized":"2024-01-16T15:28:05.363355+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'62","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249071+0000","last_change":"2024-01-16T15:19:24.619993+0000","last_active":"2024-01-16T15:27:49.249071+0000","last_peered":"2024-01-16T15:27:49.249071+0000","last_clean":"2024-01-16T15:27:49.249071+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:27:49.249071+0000","last_undegraded":"2024-01-16T15:27:49.249071+0000","last_fullsized":"2024-01-16T15:27:49.249071+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'78","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.246774+0000","last_change":"2024-01-16T15:19:24.632789+0000","last_active":"2024-01-16T15:27:49.246774+0000","last_peered":"2024-01-16T15:27:49.246774+0000","last_clean":"2024-01-16T15:27:49.246774+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:27:49.246774+0000","last_undegraded":"2024-01-16T15:27:49.246774+0000","last_fullsized":"2024-01-16T15:27:49.246774+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'117","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.343525+0000","last_change":"2024-01-16T15:28:29.343618+0000","last_active":"2024-01-16T15:28:29.343525+0000","last_peered":"2024-01-16T15:28:29.343525+0000","last_clean":"2024-01-16T15:28:29.343525+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:28:29.343525+0000","last_undegraded":"2024-01-16T15:28:29.343525+0000","last_fullsized":"2024-01-16T15:28:29.343525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'117","last_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_deep_scrub":"112'117","last_deep_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.343355+0000","objects_scrubbed":26,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:18.083505+0000","scrub_duration":0.31588118900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'102","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.453254+0000","last_change":"2024-01-16T15:28:19.453335+0000","last_active":"2024-01-16T15:28:19.453254+0000","last_peered":"2024-01-16T15:28:19.453254+0000","last_clean":"2024-01-16T15:28:19.453254+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:28:19.453254+0000","last_undegraded":"2024-01-16T15:28:19.453254+0000","last_fullsized":"2024-01-16T15:28:19.453254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'102","last_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_deep_scrub":"112'102","last_deep_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.453110+0000","objects_scrubbed":24,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.543525+0000","scrub_duration":0.24508891399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'60","reported_seq":84,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.231843+0000","last_change":"2024-01-16T15:19:24.623947+0000","last_active":"2024-01-16T15:27:49.231843+0000","last_peered":"2024-01-16T15:27:49.231843+0000","last_clean":"2024-01-16T15:27:49.231843+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:27:49.231843+0000","last_undegraded":"2024-01-16T15:27:49.231843+0000","last_fullsized":"2024-01-16T15:27:49.231843+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"109'4","reported_seq":23,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.613766+0000","last_change":"2024-01-16T15:19:23.620770+0000","last_active":"2024-01-16T15:27:45.613766+0000","last_peered":"2024-01-16T15:27:45.613766+0000","last_clean":"2024-01-16T15:27:45.613766+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:27:45.613766+0000","last_undegraded":"2024-01-16T15:27:45.613766+0000","last_fullsized":"2024-01-16T15:27:45.613766+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'67","reported_seq":93,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.248152+0000","last_change":"2024-01-16T15:19:24.628622+0000","last_active":"2024-01-16T15:27:49.248152+0000","last_peered":"2024-01-16T15:27:49.248152+0000","last_clean":"2024-01-16T15:27:49.248152+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:27:49.248152+0000","last_undegraded":"2024-01-16T15:27:49.248152+0000","last_fullsized":"2024-01-16T15:27:49.248152+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":67,"log_dups_size":0,"ondisk_log_size":67,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.589919+0000","last_change":"2024-01-16T15:28:20.589975+0000","last_active":"2024-01-16T15:28:20.589919+0000","last_peered":"2024-01-16T15:28:20.589919+0000","last_clean":"2024-01-16T15:28:20.589919+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:28:20.589919+0000","last_undegraded":"2024-01-16T15:28:20.589919+0000","last_fullsized":"2024-01-16T15:28:20.589919+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.589720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:37.660795+0000","scrub_duration":0.0035764669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'85","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.351967+0000","last_change":"2024-01-16T15:28:30.352108+0000","last_active":"2024-01-16T15:28:30.351967+0000","last_peered":"2024-01-16T15:28:30.351967+0000","last_clean":"2024-01-16T15:28:30.351967+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:28:30.351967+0000","last_undegraded":"2024-01-16T15:28:30.351967+0000","last_fullsized":"2024-01-16T15:28:30.351967+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.351810+0000","objects_scrubbed":19,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:22.513330+0000","scrub_duration":0.19453525299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'4","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.865026+0000","last_change":"2024-01-16T15:28:10.865070+0000","last_active":"2024-01-16T15:28:10.865026+0000","last_peered":"2024-01-16T15:28:10.865026+0000","last_clean":"2024-01-16T15:28:10.865026+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:28:10.865026+0000","last_undegraded":"2024-01-16T15:28:10.865026+0000","last_fullsized":"2024-01-16T15:28:10.865026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.864918+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:32.896445+0000","scrub_duration":0.0059496970000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'84","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.256703+0000","last_change":"2024-01-16T15:19:24.623360+0000","last_active":"2024-01-16T15:27:49.256703+0000","last_peered":"2024-01-16T15:27:49.256703+0000","last_clean":"2024-01-16T15:27:49.256703+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:27:49.256703+0000","last_undegraded":"2024-01-16T15:27:49.256703+0000","last_fullsized":"2024-01-16T15:27:49.256703+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'3","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:13.789441+0000","last_change":"2024-01-16T15:28:13.789502+0000","last_active":"2024-01-16T15:28:13.789441+0000","last_peered":"2024-01-16T15:28:13.789441+0000","last_clean":"2024-01-16T15:28:13.789441+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:28:13.789441+0000","last_undegraded":"2024-01-16T15:28:13.789441+0000","last_fullsized":"2024-01-16T15:28:13.789441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_clean_scrub_stamp":"2024-01-16T15:28:13.789255+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:19.022780+0000","scrub_duration":0.0072576510000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'78","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250706+0000","last_change":"2024-01-16T15:19:24.637578+0000","last_active":"2024-01-16T15:27:49.250706+0000","last_peered":"2024-01-16T15:27:49.250706+0000","last_clean":"2024-01-16T15:27:49.250706+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:27:49.250706+0000","last_undegraded":"2024-01-16T15:27:49.250706+0000","last_fullsized":"2024-01-16T15:27:49.250706+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.260333+0000","last_change":"2024-01-16T15:28:24.260389+0000","last_active":"2024-01-16T15:28:24.260333+0000","last_peered":"2024-01-16T15:28:24.260333+0000","last_clean":"2024-01-16T15:28:24.260333+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:28:24.260333+0000","last_undegraded":"2024-01-16T15:28:24.260333+0000","last_fullsized":"2024-01-16T15:28:24.260333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.260116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:22.819324+0000","scrub_duration":0.0042817660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'82","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.235102+0000","last_change":"2024-01-16T15:19:24.622129+0000","last_active":"2024-01-16T15:27:49.235102+0000","last_peered":"2024-01-16T15:27:49.235102+0000","last_clean":"2024-01-16T15:27:49.235102+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:27:49.235102+0000","last_undegraded":"2024-01-16T15:27:49.235102+0000","last_fullsized":"2024-01-16T15:27:49.235102+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.816242+0000","last_change":"2024-01-16T15:28:14.816296+0000","last_active":"2024-01-16T15:28:14.816242+0000","last_peered":"2024-01-16T15:28:14.816242+0000","last_clean":"2024-01-16T15:28:14.816242+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:28:14.816242+0000","last_undegraded":"2024-01-16T15:28:14.816242+0000","last_fullsized":"2024-01-16T15:28:14.816242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.816039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:58.559289+0000","scrub_duration":0.0043054030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'80","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250263+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:27:49.250263+0000","last_peered":"2024-01-16T15:27:49.250263+0000","last_clean":"2024-01-16T15:27:49.250263+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:27:49.250263+0000","last_undegraded":"2024-01-16T15:27:49.250263+0000","last_fullsized":"2024-01-16T15:27:49.250263+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.251244+0000","last_change":"2024-01-16T15:28:11.251291+0000","last_active":"2024-01-16T15:28:11.251244+0000","last_peered":"2024-01-16T15:28:11.251244+0000","last_clean":"2024-01-16T15:28:11.251244+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:28:11.251244+0000","last_undegraded":"2024-01-16T15:28:11.251244+0000","last_fullsized":"2024-01-16T15:28:11.251244+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.251026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:45.371008+0000","scrub_duration":0.0038716129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'68","reported_seq":91,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.234221+0000","last_change":"2024-01-16T15:19:24.626421+0000","last_active":"2024-01-16T15:27:49.234221+0000","last_peered":"2024-01-16T15:27:49.234221+0000","last_clean":"2024-01-16T15:27:49.234221+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:27:49.234221+0000","last_undegraded":"2024-01-16T15:27:49.234221+0000","last_fullsized":"2024-01-16T15:27:49.234221+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:40.619278+0000","last_change":"2024-01-16T15:19:23.626220+0000","last_active":"2024-01-16T15:26:40.619278+0000","last_peered":"2024-01-16T15:26:40.619278+0000","last_clean":"2024-01-16T15:26:40.619278+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:26:40.619278+0000","last_undegraded":"2024-01-16T15:26:40.619278+0000","last_fullsized":"2024-01-16T15:26:40.619278+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'66","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242520+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:27:49.242520+0000","last_peered":"2024-01-16T15:27:49.242520+0000","last_clean":"2024-01-16T15:27:49.242520+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:27:49.242520+0000","last_undegraded":"2024-01-16T15:27:49.242520+0000","last_fullsized":"2024-01-16T15:27:49.242520+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.843215+0000","last_change":"2024-01-16T15:28:11.843271+0000","last_active":"2024-01-16T15:28:11.843215+0000","last_peered":"2024-01-16T15:28:11.843215+0000","last_clean":"2024-01-16T15:28:11.843215+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:28:11.843215+0000","last_undegraded":"2024-01-16T15:28:11.843215+0000","last_fullsized":"2024-01-16T15:28:11.843215+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.843032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:01.354272+0000","scrub_duration":0.003520572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'64","reported_seq":86,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243319+0000","last_change":"2024-01-16T15:19:24.636450+0000","last_active":"2024-01-16T15:27:49.243319+0000","last_peered":"2024-01-16T15:27:49.243319+0000","last_clean":"2024-01-16T15:27:49.243319+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:27:49.243319+0000","last_undegraded":"2024-01-16T15:27:49.243319+0000","last_fullsized":"2024-01-16T15:27:49.243319+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.784304+0000","last_change":"2024-01-16T15:28:16.784389+0000","last_active":"2024-01-16T15:28:16.784304+0000","last_peered":"2024-01-16T15:28:16.784304+0000","last_clean":"2024-01-16T15:28:16.784304+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:28:16.784304+0000","last_undegraded":"2024-01-16T15:28:16.784304+0000","last_fullsized":"2024-01-16T15:28:16.784304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.784086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:32.176453+0000","scrub_duration":0.0040169810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'60","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244408+0000","last_change":"2024-01-16T15:19:24.628254+0000","last_active":"2024-01-16T15:27:49.244408+0000","last_peered":"2024-01-16T15:27:49.244408+0000","last_clean":"2024-01-16T15:27:49.244408+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:27:49.244408+0000","last_undegraded":"2024-01-16T15:27:49.244408+0000","last_fullsized":"2024-01-16T15:27:49.244408+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'31","reported_seq":49,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:40.619414+0000","last_change":"2024-01-16T15:19:23.620245+0000","last_active":"2024-01-16T15:26:40.619414+0000","last_peered":"2024-01-16T15:26:40.619414+0000","last_clean":"2024-01-16T15:26:40.619414+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:26:40.619414+0000","last_undegraded":"2024-01-16T15:26:40.619414+0000","last_fullsized":"2024-01-16T15:26:40.619414+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'69","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.251406+0000","last_change":"2024-01-16T15:19:24.630582+0000","last_active":"2024-01-16T15:27:49.251406+0000","last_peered":"2024-01-16T15:27:49.251406+0000","last_clean":"2024-01-16T15:27:49.251406+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:27:49.251406+0000","last_undegraded":"2024-01-16T15:27:49.251406+0000","last_fullsized":"2024-01-16T15:27:49.251406+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'307","reported_seq":325,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:59.261078+0000","last_change":"2024-01-16T15:19:23.624965+0000","last_active":"2024-01-16T15:27:59.261078+0000","last_peered":"2024-01-16T15:27:59.261078+0000","last_clean":"2024-01-16T15:27:59.261078+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:27:59.261078+0000","last_undegraded":"2024-01-16T15:27:59.261078+0000","last_fullsized":"2024-01-16T15:27:59.261078+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'63","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242943+0000","last_change":"2024-01-16T15:19:24.633681+0000","last_active":"2024-01-16T15:27:49.242943+0000","last_peered":"2024-01-16T15:27:49.242943+0000","last_clean":"2024-01-16T15:27:49.242943+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:27:49.242943+0000","last_undegraded":"2024-01-16T15:27:49.242943+0000","last_fullsized":"2024-01-16T15:27:49.242943+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'4","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.218924+0000","last_change":"2024-01-16T15:28:18.219036+0000","last_active":"2024-01-16T15:28:18.218924+0000","last_peered":"2024-01-16T15:28:18.218924+0000","last_clean":"2024-01-16T15:28:18.218924+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:28:18.218924+0000","last_undegraded":"2024-01-16T15:28:18.218924+0000","last_fullsized":"2024-01-16T15:28:18.218924+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.218660+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:48.176772+0000","scrub_duration":0.0091646809999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'80","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.249986+0000","last_change":"2024-01-16T15:19:24.630178+0000","last_active":"2024-01-16T15:27:49.249986+0000","last_peered":"2024-01-16T15:27:49.249986+0000","last_clean":"2024-01-16T15:27:49.249986+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:27:49.249986+0000","last_undegraded":"2024-01-16T15:27:49.249986+0000","last_fullsized":"2024-01-16T15:27:49.249986+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":26,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.450221+0000","last_change":"2024-01-16T15:28:16.450274+0000","last_active":"2024-01-16T15:28:16.450221+0000","last_peered":"2024-01-16T15:28:16.450221+0000","last_clean":"2024-01-16T15:28:16.450221+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:28:16.450221+0000","last_undegraded":"2024-01-16T15:28:16.450221+0000","last_fullsized":"2024-01-16T15:28:16.450221+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.449979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:39.537507+0000","scrub_duration":0.0036212340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'73","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.244737+0000","last_change":"2024-01-16T15:19:24.620659+0000","last_active":"2024-01-16T15:27:49.244737+0000","last_peered":"2024-01-16T15:27:49.244737+0000","last_clean":"2024-01-16T15:27:49.244737+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:27:49.244737+0000","last_undegraded":"2024-01-16T15:27:49.244737+0000","last_fullsized":"2024-01-16T15:27:49.244737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'12","reported_seq":38,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.825417+0000","last_change":"2024-01-16T15:28:07.825453+0000","last_active":"2024-01-16T15:28:07.825417+0000","last_peered":"2024-01-16T15:28:07.825417+0000","last_clean":"2024-01-16T15:28:07.825417+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:28:07.825417+0000","last_undegraded":"2024-01-16T15:28:07.825417+0000","last_fullsized":"2024-01-16T15:28:07.825417+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.825335+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:20.349610+0000","scrub_duration":0.0056472520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'62","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.989502+0000","last_change":"2024-01-16T15:28:21.989502+0000","last_active":"2024-01-16T15:28:21.989502+0000","last_peered":"2024-01-16T15:28:21.989502+0000","last_clean":"2024-01-16T15:28:21.989502+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:28:21.989502+0000","last_undegraded":"2024-01-16T15:28:21.989502+0000","last_fullsized":"2024-01-16T15:28:21.989502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'62","last_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_deep_scrub":"112'62","last_deep_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.989317+0000","objects_scrubbed":12,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:49.378780+0000","scrub_duration":0.125520883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566191+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:28:05.566191+0000","last_peered":"2024-01-16T15:28:05.566191+0000","last_clean":"2024-01-16T15:28:05.566191+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:28:05.566191+0000","last_undegraded":"2024-01-16T15:28:05.566191+0000","last_fullsized":"2024-01-16T15:28:05.566191+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'61","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:06.292926+0000","last_change":"2024-01-16T15:19:24.620858+0000","last_active":"2024-01-16T15:28:06.292926+0000","last_peered":"2024-01-16T15:28:06.292926+0000","last_clean":"2024-01-16T15:28:06.292926+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:28:06.292926+0000","last_undegraded":"2024-01-16T15:28:06.292926+0000","last_fullsized":"2024-01-16T15:28:06.292926+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.809350+0000","last_change":"2024-01-16T15:28:17.809350+0000","last_active":"2024-01-16T15:28:17.809350+0000","last_peered":"2024-01-16T15:28:17.809350+0000","last_clean":"2024-01-16T15:28:17.809350+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:28:17.809350+0000","last_undegraded":"2024-01-16T15:28:17.809350+0000","last_fullsized":"2024-01-16T15:28:17.809350+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.809101+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:47.767212+0000","scrub_duration":0.0070965819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'77","reported_seq":106,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.707945+0000","last_change":"2024-01-16T15:19:24.618661+0000","last_active":"2024-01-16T15:28:10.707945+0000","last_peered":"2024-01-16T15:28:10.707945+0000","last_clean":"2024-01-16T15:28:10.707945+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:28:10.707945+0000","last_undegraded":"2024-01-16T15:28:10.707945+0000","last_fullsized":"2024-01-16T15:28:10.707945+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'2","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.443869+0000","last_change":"2024-01-16T15:28:18.443869+0000","last_active":"2024-01-16T15:28:18.443869+0000","last_peered":"2024-01-16T15:28:18.443869+0000","last_clean":"2024-01-16T15:28:18.443869+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:28:18.443869+0000","last_undegraded":"2024-01-16T15:28:18.443869+0000","last_fullsized":"2024-01-16T15:28:18.443869+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.443623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:31.553582+0000","scrub_duration":0.0077739760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'66","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.707965+0000","last_change":"2024-01-16T15:19:24.625083+0000","last_active":"2024-01-16T15:28:10.707965+0000","last_peered":"2024-01-16T15:28:10.707965+0000","last_clean":"2024-01-16T15:28:10.707965+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:28:10.707965+0000","last_undegraded":"2024-01-16T15:28:10.707965+0000","last_fullsized":"2024-01-16T15:28:10.707965+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:09.230415+0000","last_change":"2024-01-16T15:19:23.625277+0000","last_active":"2024-01-16T15:28:09.230415+0000","last_peered":"2024-01-16T15:28:09.230415+0000","last_clean":"2024-01-16T15:28:09.230415+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:28:09.230415+0000","last_undegraded":"2024-01-16T15:28:09.230415+0000","last_fullsized":"2024-01-16T15:28:09.230415+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'62","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.965359+0000","last_change":"2024-01-16T15:19:24.615908+0000","last_active":"2024-01-16T15:28:05.965359+0000","last_peered":"2024-01-16T15:28:05.965359+0000","last_clean":"2024-01-16T15:28:05.965359+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:28:05.965359+0000","last_undegraded":"2024-01-16T15:28:05.965359+0000","last_fullsized":"2024-01-16T15:28:05.965359+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.174817+0000","last_change":"2024-01-16T15:28:17.174817+0000","last_active":"2024-01-16T15:28:17.174817+0000","last_peered":"2024-01-16T15:28:17.174817+0000","last_clean":"2024-01-16T15:28:17.174817+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:28:17.174817+0000","last_undegraded":"2024-01-16T15:28:17.174817+0000","last_fullsized":"2024-01-16T15:28:17.174817+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.174587+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:21.215097+0000","scrub_duration":0.0071591010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566466+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:28:05.566466+0000","last_peered":"2024-01-16T15:28:05.566466+0000","last_clean":"2024-01-16T15:28:05.566466+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:28:05.566466+0000","last_undegraded":"2024-01-16T15:28:05.566466+0000","last_fullsized":"2024-01-16T15:28:05.566466+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.833365+0000","last_change":"2024-01-16T15:28:19.833414+0000","last_active":"2024-01-16T15:28:19.833365+0000","last_peered":"2024-01-16T15:28:19.833365+0000","last_clean":"2024-01-16T15:28:19.833365+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:28:19.833365+0000","last_undegraded":"2024-01-16T15:28:19.833365+0000","last_fullsized":"2024-01-16T15:28:19.833365+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.923614+0000","scrub_duration":0.0036415929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'66","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.965217+0000","last_change":"2024-01-16T15:19:24.620664+0000","last_active":"2024-01-16T15:28:05.965217+0000","last_peered":"2024-01-16T15:28:05.965217+0000","last_clean":"2024-01-16T15:28:05.965217+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:28:05.965217+0000","last_undegraded":"2024-01-16T15:28:05.965217+0000","last_fullsized":"2024-01-16T15:28:05.965217+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.206909+0000","last_change":"2024-01-16T15:28:15.206975+0000","last_active":"2024-01-16T15:28:15.206909+0000","last_peered":"2024-01-16T15:28:15.206909+0000","last_clean":"2024-01-16T15:28:15.206909+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:28:15.206909+0000","last_undegraded":"2024-01-16T15:28:15.206909+0000","last_fullsized":"2024-01-16T15:28:15.206909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.206765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:27.403194+0000","scrub_duration":0.0041188539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'63","reported_seq":88,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.363321+0000","last_change":"2024-01-16T15:19:24.621367+0000","last_active":"2024-01-16T15:28:05.363321+0000","last_peered":"2024-01-16T15:28:05.363321+0000","last_clean":"2024-01-16T15:28:05.363321+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:28:05.363321+0000","last_undegraded":"2024-01-16T15:28:05.363321+0000","last_fullsized":"2024-01-16T15:28:05.363321+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566086+0000","last_change":"2024-01-16T15:19:23.623266+0000","last_active":"2024-01-16T15:28:05.566086+0000","last_peered":"2024-01-16T15:28:05.566086+0000","last_clean":"2024-01-16T15:28:05.566086+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:28:05.566086+0000","last_undegraded":"2024-01-16T15:28:05.566086+0000","last_fullsized":"2024-01-16T15:28:05.566086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'92","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:22.637191+0000","last_change":"2024-01-16T15:28:22.637284+0000","last_active":"2024-01-16T15:28:22.637191+0000","last_peered":"2024-01-16T15:28:22.637191+0000","last_clean":"2024-01-16T15:28:22.637191+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:28:22.637191+0000","last_undegraded":"2024-01-16T15:28:22.637191+0000","last_fullsized":"2024-01-16T15:28:22.637191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'92","last_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_deep_scrub":"112'92","last_deep_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_clean_scrub_stamp":"2024-01-16T15:28:22.636970+0000","objects_scrubbed":22,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:01.808019+0000","scrub_duration":0.23588124099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.749499+0000","last_change":"2024-01-16T15:28:17.749653+0000","last_active":"2024-01-16T15:28:17.749499+0000","last_peered":"2024-01-16T15:28:17.749499+0000","last_clean":"2024-01-16T15:28:17.749499+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:28:17.749499+0000","last_undegraded":"2024-01-16T15:28:17.749499+0000","last_fullsized":"2024-01-16T15:28:17.749499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.749294+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:03.835812+0000","scrub_duration":0.0071287670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566405+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:28:05.566405+0000","last_peered":"2024-01-16T15:28:05.566405+0000","last_clean":"2024-01-16T15:28:05.566405+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:28:05.566405+0000","last_undegraded":"2024-01-16T15:28:05.566405+0000","last_fullsized":"2024-01-16T15:28:05.566405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":58,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.479780+0000","last_change":"2024-01-16T15:28:17.479819+0000","last_active":"2024-01-16T15:28:17.479780+0000","last_peered":"2024-01-16T15:28:17.479780+0000","last_clean":"2024-01-16T15:28:17.479780+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:28:17.479780+0000","last_undegraded":"2024-01-16T15:28:17.479780+0000","last_fullsized":"2024-01-16T15:28:17.479780+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'31","last_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_deep_scrub":"112'31","last_deep_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.479610+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:48.802592+0000","scrub_duration":0.0060914480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593,"num_omap_keys":11,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'96","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.235685+0000","last_change":"2024-01-16T15:28:15.235729+0000","last_active":"2024-01-16T15:28:15.235685+0000","last_peered":"2024-01-16T15:28:15.235685+0000","last_clean":"2024-01-16T15:28:15.235685+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:28:15.235685+0000","last_undegraded":"2024-01-16T15:28:15.235685+0000","last_fullsized":"2024-01-16T15:28:15.235685+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.235576+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:39.275609+0000","scrub_duration":0.27566697899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:45.632990+0000","last_change":"2024-01-16T15:19:23.624633+0000","last_active":"2024-01-16T15:27:45.632990+0000","last_peered":"2024-01-16T15:27:45.632990+0000","last_clean":"2024-01-16T15:27:45.632990+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:27:45.632990+0000","last_undegraded":"2024-01-16T15:27:45.632990+0000","last_fullsized":"2024-01-16T15:27:45.632990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'66","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.564786+0000","last_change":"2024-01-16T15:28:20.564862+0000","last_active":"2024-01-16T15:28:20.564786+0000","last_peered":"2024-01-16T15:28:20.564786+0000","last_clean":"2024-01-16T15:28:20.564786+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:28:20.564786+0000","last_undegraded":"2024-01-16T15:28:20.564786+0000","last_fullsized":"2024-01-16T15:28:20.564786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'66","last_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_deep_scrub":"112'66","last_deep_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.564586+0000","objects_scrubbed":13,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:24.471110+0000","scrub_duration":0.12569713499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.884673+0000","last_change":"2024-01-16T15:28:10.884720+0000","last_active":"2024-01-16T15:28:10.884673+0000","last_peered":"2024-01-16T15:28:10.884673+0000","last_clean":"2024-01-16T15:28:10.884673+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:28:10.884673+0000","last_undegraded":"2024-01-16T15:28:10.884673+0000","last_fullsized":"2024-01-16T15:28:10.884673+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.884468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:06.507427+0000","scrub_duration":0.0032287930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.481473+0000","last_change":"2024-01-16T15:28:17.481559+0000","last_active":"2024-01-16T15:28:17.481473+0000","last_peered":"2024-01-16T15:28:17.481473+0000","last_clean":"2024-01-16T15:28:17.481473+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:28:17.481473+0000","last_undegraded":"2024-01-16T15:28:17.481473+0000","last_fullsized":"2024-01-16T15:28:17.481473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.481296+0000","objects_scrubbed":17,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:01.224547+0000","scrub_duration":0.21059523299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.256061+0000","last_change":"2024-01-16T15:28:18.256178+0000","last_active":"2024-01-16T15:28:18.256061+0000","last_peered":"2024-01-16T15:28:18.256061+0000","last_clean":"2024-01-16T15:28:18.256061+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:28:18.256061+0000","last_undegraded":"2024-01-16T15:28:18.256061+0000","last_fullsized":"2024-01-16T15:28:18.256061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.255843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:27.175270+0000","scrub_duration":0.0036640700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'64","reported_seq":90,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236381+0000","last_change":"2024-01-16T15:19:24.609380+0000","last_active":"2024-01-16T15:27:49.236381+0000","last_peered":"2024-01-16T15:27:49.236381+0000","last_clean":"2024-01-16T15:27:49.236381+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:27:49.236381+0000","last_undegraded":"2024-01-16T15:27:49.236381+0000","last_fullsized":"2024-01-16T15:27:49.236381+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.221495+0000","last_change":"2024-01-16T15:28:10.221583+0000","last_active":"2024-01-16T15:28:10.221495+0000","last_peered":"2024-01-16T15:28:10.221495+0000","last_clean":"2024-01-16T15:28:10.221495+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:28:10.221495+0000","last_undegraded":"2024-01-16T15:28:10.221495+0000","last_fullsized":"2024-01-16T15:28:10.221495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.221197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:27:58.612687+0000","scrub_duration":0.0041312190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'73","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.243721+0000","last_change":"2024-01-16T15:19:24.624959+0000","last_active":"2024-01-16T15:27:49.243721+0000","last_peered":"2024-01-16T15:27:49.243721+0000","last_clean":"2024-01-16T15:27:49.243721+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:27:49.243721+0000","last_undegraded":"2024-01-16T15:27:49.243721+0000","last_fullsized":"2024-01-16T15:27:49.243721+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'32","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.827087+0000","last_change":"2024-01-16T15:28:14.827144+0000","last_active":"2024-01-16T15:28:14.827087+0000","last_peered":"2024-01-16T15:28:14.827087+0000","last_clean":"2024-01-16T15:28:14.827087+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:28:14.827087+0000","last_undegraded":"2024-01-16T15:28:14.827087+0000","last_fullsized":"2024-01-16T15:28:14.827087+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'32","last_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_deep_scrub":"112'32","last_deep_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.826899+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:16:55.047969+0000","scrub_duration":0.0075779699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.118029+0000","last_change":"2024-01-16T15:28:25.118081+0000","last_active":"2024-01-16T15:28:25.118029+0000","last_peered":"2024-01-16T15:28:25.118029+0000","last_clean":"2024-01-16T15:28:25.118029+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:28:25.118029+0000","last_undegraded":"2024-01-16T15:28:25.118029+0000","last_fullsized":"2024-01-16T15:28:25.118029+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.117955+0000","objects_scrubbed":23,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:56.787030+0000","scrub_duration":0.17487287100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":19,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:25.628146+0000","last_change":"2024-01-16T15:19:23.623078+0000","last_active":"2024-01-16T15:27:25.628146+0000","last_peered":"2024-01-16T15:27:25.628146+0000","last_clean":"2024-01-16T15:27:25.628146+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:27:25.628146+0000","last_undegraded":"2024-01-16T15:27:25.628146+0000","last_fullsized":"2024-01-16T15:27:25.628146+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'68","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.884587+0000","last_change":"2024-01-16T15:28:19.884824+0000","last_active":"2024-01-16T15:28:19.884587+0000","last_peered":"2024-01-16T15:28:19.884587+0000","last_clean":"2024-01-16T15:28:19.884587+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:28:19.884587+0000","last_undegraded":"2024-01-16T15:28:19.884587+0000","last_fullsized":"2024-01-16T15:28:19.884587+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'68","last_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_deep_scrub":"112'68","last_deep_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.884357+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:10.532387+0000","scrub_duration":0.15341920100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":18,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:26:25.599209+0000","last_change":"2024-01-16T15:19:23.627817+0000","last_active":"2024-01-16T15:26:25.599209+0000","last_peered":"2024-01-16T15:26:25.599209+0000","last_clean":"2024-01-16T15:26:25.599209+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:26:25.599209+0000","last_undegraded":"2024-01-16T15:26:25.599209+0000","last_fullsized":"2024-01-16T15:26:25.599209+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"112'119","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.585009+0000","last_change":"2024-01-16T15:28:18.585089+0000","last_active":"2024-01-16T15:28:18.585009+0000","last_peered":"2024-01-16T15:28:18.585009+0000","last_clean":"2024-01-16T15:28:18.585009+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:28:18.585009+0000","last_undegraded":"2024-01-16T15:28:18.585009+0000","last_fullsized":"2024-01-16T15:28:18.585009+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'119","last_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_deep_scrub":"112'119","last_deep_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.584815+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:01.708690+0000","scrub_duration":0.06564681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4804,"ondisk_log_size":4804,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509786,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.0449999999999999,"5min":1.1559999999999999,"15min":1.25},"min":{"1min":0.69899999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.5409999999999999,"5min":1.7749999999999999,"15min":19.224},"last":1.415},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.212,"15min":1.3140000000000001},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.615,"5min":1.694,"15min":28.603999999999999},"last":0.96999999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":1.22,"15min":1.2250000000000001},"min":{"1min":0.90500000000000003,"5min":0.80800000000000005,"15min":0.374},"max":{"1min":1.488,"5min":1.837,"15min":3.7069999999999999},"last":1.143},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.3080000000000001,"15min":1.4970000000000001},"min":{"1min":0.93000000000000005,"5min":0.68899999999999995,"15min":0.68400000000000005},"max":{"1min":1.651,"5min":1.8819999999999999,"15min":33.225000000000001},"last":0.92300000000000004}]},{"osd":2,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.3380000000000001,"15min":1.4199999999999999},"min":{"1min":1.0529999999999999,"5min":0.77500000000000002,"15min":0.74299999999999999},"max":{"1min":1.752,"5min":1.7909999999999999,"15min":19.126000000000001},"last":1.091},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3080000000000001,"15min":1.3120000000000001},"min":{"1min":1.008,"5min":0.66700000000000004,"15min":0.66700000000000004},"max":{"1min":1.694,"5min":1.718,"15min":2.9750000000000001},"last":1.21}]},{"osd":3,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.26,"15min":1.3169999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.76500000000000001},"max":{"1min":1.526,"5min":1.73,"15min":3.7999999999999998},"last":1.2609999999999999},{"interface":"front","average":{"1min":1.369,"5min":1.446,"15min":1.5580000000000001},"min":{"1min":0.97399999999999998,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":19.268999999999998},"last":1.3009999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.2290000000000001,"15min":1.264},"min":{"1min":0.88300000000000001,"5min":0.60699999999999998,"15min":0.58799999999999997},"max":{"1min":1.8169999999999999,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.052},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.347,"15min":1.3919999999999999},"min":{"1min":1.1180000000000001,"5min":0.70299999999999996,"15min":0.58299999999999996},"max":{"1min":1.6659999999999999,"5min":1.7330000000000001,"15min":9.7059999999999995},"last":1.0109999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.5269999999999999,"15min":1.55},"min":{"1min":1.0389999999999999,"5min":0.86099999999999999,"15min":0.79400000000000004},"max":{"1min":1.74,"5min":1.9450000000000001,"15min":3.9129999999999998},"last":1.47},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.444,"15min":1.5620000000000001},"min":{"1min":0.82699999999999996,"5min":0.77900000000000003,"15min":0.55100000000000005},"max":{"1min":1.8720000000000001,"5min":1.9630000000000001,"15min":19.036999999999999},"last":1.502}]},{"osd":7,"last update":"Tue Jan 16 15:27:33 2024","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.4359999999999999,"15min":1.4410000000000001},"min":{"1min":1.198,"5min":0.74299999999999999,"15min":0.54800000000000004},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":4.2240000000000002},"last":1.329},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.528,"15min":1.571},"min":{"1min":1.224,"5min":0.98999999999999999,"15min":0.65100000000000002},"max":{"1min":1.728,"5min":2.181,"15min":4.0709999999999997},"last":1.3680000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509788,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.296,"15min":1.2330000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.427,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1799999999999999,"15min":1.1859999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.2190000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.88700000000000001,"5min":0.73899999999999999,"15min":0.71899999999999997},"max":{"1min":1.6160000000000001,"5min":1.893,"15min":2.0510000000000002},"last":1.095},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.3149999999999999,"15min":1.302},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.6319999999999999,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.304}]},{"osd":2,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3420000000000001,"15min":1.3320000000000001},"min":{"1min":0.79300000000000004,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.573,"5min":1.78,"15min":5.907},"last":1.353},{"interface":"front","average":{"1min":1.242,"5min":1.381,"15min":1.355},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.52600000000000002},"max":{"1min":1.6000000000000001,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":0.92800000000000005}]},{"osd":3,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4570000000000001,"15min":1.3939999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.76200000000000001},"max":{"1min":1.5289999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.385},{"interface":"front","average":{"1min":1.391,"5min":1.4019999999999999,"15min":1.391},"min":{"1min":1.052,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.74,"5min":1.9670000000000001,"15min":2.0379999999999998},"last":1.1830000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.425,"15min":1.448},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6779999999999999,"5min":6.165,"15min":6.165},"last":1.038},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4410000000000001,"15min":1.4770000000000001},"min":{"1min":0.95299999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8180000000000001,"5min":3.1469999999999998,"15min":5.3520000000000003},"last":1.597}]},{"osd":6,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.391,"15min":1.4219999999999999},"min":{"1min":1.101,"5min":0.90800000000000003,"15min":0.497},"max":{"1min":1.903,"5min":1.903,"15min":8.9600000000000009},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5640000000000001,"15min":1.528},"min":{"1min":1.165,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.772,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.4379999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.452,"15min":1.492},"min":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.89700000000000002},"max":{"1min":1.8500000000000001,"5min":3.3620000000000001,"15min":5.835},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.554,"5min":1.5469999999999999,"15min":1.581},"min":{"1min":1.238,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":2.0070000000000001,"5min":2.3679999999999999,"15min":8.0079999999999991},"last":1.556}]}]},{"osd":7,"up_from":14,"seq":60129542490,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1891948,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92479892,"statfs":{"total":96636764160,"available":94699409408,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":73071050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1879999999999999,"5min":1.1659999999999999,"15min":1.2130000000000001},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.65400000000000003},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":2.1779999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.24,"5min":1.2210000000000001,"15min":1.248},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":1.6180000000000001,"5min":1.7050000000000001,"15min":12.461},"last":1.2450000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2669999999999999,"15min":1.2669999999999999},"min":{"1min":0.89000000000000001,"5min":0.85099999999999998,"15min":0.48599999999999999},"max":{"1min":1.923,"5min":1.923,"15min":2.4249999999999998},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.3029999999999999,"5min":1.2709999999999999,"15min":1.363},"min":{"1min":0.93400000000000005,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":1.8140000000000001,"5min":1.9079999999999999,"15min":14.848000000000001},"last":1.119}]},{"osd":2,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3069999999999999,"15min":1.304},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":1.8280000000000001,"5min":1.8280000000000001,"15min":3.726},"last":1.498},{"interface":"front","average":{"1min":1.286,"5min":1.3340000000000001,"15min":1.3959999999999999},"min":{"1min":0.88900000000000001,"5min":0.873,"15min":0.54700000000000004},"max":{"1min":1.7130000000000001,"5min":1.792,"15min":19.763999999999999},"last":1.167}]},{"osd":3,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.321,"15min":1.3340000000000001},"min":{"1min":0.875,"5min":0.875,"15min":0.80600000000000005},"max":{"1min":1.6539999999999999,"5min":1.8280000000000001,"15min":1.887},"last":1.379},{"interface":"front","average":{"1min":1.456,"5min":1.47,"15min":1.4650000000000001},"min":{"1min":1.113,"5min":1.0169999999999999,"15min":0.629},"max":{"1min":1.9910000000000001,"5min":1.9910000000000001,"15min":6.6029999999999998},"last":1.591}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.345,"15min":1.347},"min":{"1min":0.98499999999999999,"5min":0.89100000000000001,"15min":0.54100000000000004},"max":{"1min":1.7090000000000001,"5min":1.7929999999999999,"15min":4.1390000000000002},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.401,"15min":1.413},"min":{"1min":1.0009999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":1.873,"5min":1.877,"15min":2.5190000000000001},"last":1.288}]},{"osd":5,"last update":"Tue Jan 16 15:27:36 2024","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.3340000000000001,"15min":1.3680000000000001},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":1.9490000000000001,"5min":1.9490000000000001,"15min":12.609999999999999},"last":1.351},{"interface":"front","average":{"1min":1.5209999999999999,"5min":1.5209999999999999,"15min":1.5229999999999999},"min":{"1min":1.1719999999999999,"5min":1.046,"15min":0.66000000000000003},"max":{"1min":1.8879999999999999,"5min":1.909,"15min":3.4159999999999999},"last":1.756}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.099,"5min":1.0760000000000001,"15min":0.66000000000000003},"max":{"1min":2.0859999999999999,"5min":2.1429999999999998,"15min":2.653},"last":1.6879999999999999},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.552},"min":{"1min":1.004,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":12.567},"last":1.55}]}]},{"osd":5,"up_from":14,"seq":60129542492,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1659999999999999,"5min":1.228,"15min":1.208},"min":{"1min":0.83399999999999996,"5min":0.749,"15min":0.32900000000000001},"max":{"1min":1.7190000000000001,"5min":2.0939999999999999,"15min":6.8380000000000001},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.1870000000000001,"15min":1.3200000000000001},"min":{"1min":1.0049999999999999,"5min":0.82999999999999996,"15min":0.57099999999999995},"max":{"1min":1.7649999999999999,"5min":1.7649999999999999,"15min":23.199999999999999},"last":0.93700000000000006}]},{"osd":1,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.226,"15min":1.246},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.63400000000000001},"max":{"1min":1.637,"5min":1.698,"15min":8.0909999999999993},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.355,"5min":1.26,"15min":1.323},"min":{"1min":0.93600000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.8049999999999999,"5min":1.851,"15min":17.062000000000001},"last":1.278}]},{"osd":2,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.409,"15min":1.415},"min":{"1min":1.069,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.6970000000000001,"5min":1.782,"15min":5.8810000000000002},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.3620000000000001,"5min":1.3500000000000001,"15min":1.389},"min":{"1min":1.131,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.6899999999999999,"5min":1.6970000000000001,"15min":5.4429999999999996},"last":1.2030000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2889999999999999,"5min":1.302,"15min":1.3420000000000001},"min":{"1min":0.94999999999999996,"5min":0.878,"15min":0.67500000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":7.984},"last":1.254},{"interface":"front","average":{"1min":1.385,"5min":1.4179999999999999,"15min":1.423},"min":{"1min":0.94599999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.7410000000000001,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":1.4950000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.323,"15min":1.387},"min":{"1min":0.878,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":15.975},"last":1.4299999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.4079999999999999,"15min":1.413},"min":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.54000000000000004},"max":{"1min":1.8480000000000001,"5min":1.8480000000000001,"15min":5.6349999999999998},"last":1.871}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.329,"15min":1.3680000000000001},"min":{"1min":0.95999999999999996,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.8160000000000001,"5min":1.827,"15min":12.914},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.536,"5min":1.468,"15min":1.5109999999999999},"min":{"1min":1.006,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":16.552},"last":1.4770000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.4970000000000001,"5min":1.4630000000000001,"15min":1.506},"min":{"1min":1.298,"5min":0.98099999999999998,"15min":0.64300000000000002},"max":{"1min":1.74,"5min":1.8540000000000001,"15min":13.034000000000001},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.5740000000000001,"15min":1.5900000000000001},"min":{"1min":1.161,"5min":1.0429999999999999,"15min":0.70899999999999996},"max":{"1min":1.9159999999999999,"5min":2.073,"15min":8.0380000000000003},"last":1.5369999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542492,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":5.7400000000000002},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.1359999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40600000000000003},"max":{"1min":1.506,"5min":1.879,"15min":7.1429999999999998},"last":1.206}]},{"osd":1,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.96899999999999997,"15min":1.0349999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.365,"5min":1.738,"15min":10.856999999999999},"last":0.91800000000000004},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.1870000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4950000000000001,"5min":1.823,"15min":2.0089999999999999},"last":0.95399999999999996}]},{"osd":2,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.3140000000000001,"15min":1.5209999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.946,"5min":1.946,"15min":39.265999999999998},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1559999999999999,"15min":1.2070000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.381,"5min":1.833,"15min":7.9160000000000004},"last":1.097}]},{"osd":4,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.337,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.70999999999999996},"max":{"1min":1.5980000000000001,"5min":1.9419999999999999,"15min":5.6890000000000001},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.329,"5min":1.464,"15min":1.4810000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8120000000000001,"5min":2.0920000000000001,"15min":2.3330000000000002},"last":1.4510000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3440000000000001,"15min":1.4670000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5669999999999999,"5min":1.827,"15min":14.003},"last":1.155},{"interface":"front","average":{"1min":1.357,"5min":1.5049999999999999,"15min":1.577},"min":{"1min":0.877,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":1.9359999999999999,"5min":2.008,"15min":6.0679999999999996},"last":1.3899999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.464,"15min":1.502},"min":{"1min":0.875,"5min":0.875,"15min":0.69799999999999995},"max":{"1min":1.8899999999999999,"5min":2.0139999999999998,"15min":7.0060000000000002},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.486,"15min":1.5580000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8140000000000001,"5min":1.982,"15min":15.114000000000001},"last":1.3500000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4950000000000001,"15min":1.6160000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80400000000000005},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":12.378},"last":1.587},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5780000000000001,"15min":1.6220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9650000000000001,"5min":2.23,"15min":4.6029999999999998},"last":1.6499999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542491,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1059928,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93311912,"statfs":{"total":96636764160,"available":95551397888,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":0.85399999999999998,"5min":0.90500000000000003,"15min":0.97899999999999998},"min":{"1min":0.46200000000000002,"5min":0.41199999999999998,"15min":0.32400000000000001},"max":{"1min":1.4770000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.69799999999999995},{"interface":"front","average":{"1min":1.161,"5min":1.1459999999999999,"15min":1.1319999999999999},"min":{"1min":0.49199999999999999,"5min":0.44400000000000001,"15min":0.40300000000000002},"max":{"1min":1.833,"5min":1.913,"15min":7.9960000000000004},"last":1.4550000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0880000000000001,"15min":1.1040000000000001},"min":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":1.7130000000000001,"5min":2.294,"15min":3.056},"last":2.145},{"interface":"front","average":{"1min":1.171,"5min":1.143,"15min":1.175},"min":{"1min":0.55300000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.556,"5min":1.877,"15min":4.3819999999999997},"last":2.1880000000000002}]},{"osd":3,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.206,"15min":1.2789999999999999},"min":{"1min":0.61199999999999999,"5min":0.55300000000000005,"15min":0.44700000000000001},"max":{"1min":1.704,"5min":1.883,"15min":18.838000000000001},"last":1.804},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4950000000000001,"15min":1.552},"min":{"1min":0.66200000000000003,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.714,"5min":2.0099999999999998,"15min":19.375},"last":2.1000000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.335,"15min":1.4390000000000001},"min":{"1min":0.80600000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.8240000000000001,"5min":2.242,"15min":23.225000000000001},"last":1.754},{"interface":"front","average":{"1min":1.4530000000000001,"5min":1.4730000000000001,"15min":1.494},"min":{"1min":1.038,"5min":0.75900000000000001,"15min":0.72699999999999998},"max":{"1min":1.883,"5min":1.883,"15min":6.359},"last":1.9850000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4399999999999999,"15min":1.5009999999999999},"min":{"1min":1.0800000000000001,"5min":0.68100000000000005,"15min":0.68100000000000005},"max":{"1min":1.917,"5min":2.3170000000000002,"15min":20.157},"last":1.1759999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.597,"15min":1.6100000000000001},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.623},"max":{"1min":1.9330000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.347}]},{"osd":6,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.5289999999999999,"15min":1.595},"min":{"1min":1.018,"5min":0.89800000000000002,"15min":0.70499999999999996},"max":{"1min":1.7330000000000001,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.6910000000000001},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.6160000000000001,"15min":1.7170000000000001},"min":{"1min":0.85799999999999998,"5min":0.79300000000000004,"15min":0.74099999999999999},"max":{"1min":1.827,"5min":2.4009999999999998,"15min":28.760999999999999},"last":2.0499999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:27:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.5429999999999999,"15min":1.6699999999999999},"min":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":2.4550000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.587,"5min":1.569,"15min":1.5880000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":1.8999999999999999,"5min":2.0990000000000002,"15min":4.3419999999999996},"last":1.8520000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542493,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.093,"15min":0.98799999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.28100000000000003},"max":{"1min":13.42,"5min":13.42,"15min":13.42},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.216,"5min":1.141,"15min":1.113},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.247},"max":{"1min":3.915,"5min":3.915,"15min":3.915},"last":1.1990000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":1.03,"15min":1.0900000000000001},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.37},"max":{"1min":1.5269999999999999,"5min":1.649,"15min":3.6819999999999999},"last":1.157},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.29,"15min":1.3320000000000001},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.39000000000000001},"max":{"1min":1.7090000000000001,"5min":1.7090000000000001,"15min":3.5600000000000001},"last":1.387}]},{"osd":3,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.26,"15min":1.2889999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.46300000000000002},"max":{"1min":1.488,"5min":1.875,"15min":2.238},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.077,"5min":1.137,"15min":1.194},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":1.7589999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":0.999}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.78400000000000003},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.579},{"interface":"front","average":{"1min":1.9199999999999999,"5min":1.569,"15min":1.5269999999999999},"min":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.72199999999999998},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.4870000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.3939999999999999,"15min":1.431},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.63700000000000001},"max":{"1min":1.6359999999999999,"5min":1.952,"15min":3.98},"last":1.095},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.4530000000000001,"15min":1.462},"min":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.53300000000000003},"max":{"1min":1.669,"5min":1.893,"15min":3.3940000000000001},"last":1.3340000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.4850000000000001,"15min":1.5700000000000001},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.56999999999999995},"max":{"1min":1.8660000000000001,"5min":2.016,"15min":16.48},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.508,"15min":1.532},"min":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":0.65600000000000003},"max":{"1min":1.74,"5min":1.909,"15min":3.8180000000000001},"last":1.7030000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.573,"15min":1.617},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.48099999999999998},"max":{"1min":1.8480000000000001,"5min":2.0009999999999999,"15min":7.0499999999999998},"last":1.7869999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.6339999999999999,"15min":1.647},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.81100000000000005},"max":{"1min":1.8640000000000001,"5min":2.3940000000000001,"15min":3.7839999999999998},"last":1.6379999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542491,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":0.93200000000000005,"5min":0.92400000000000004,"15min":1.038},"min":{"1min":0.496,"5min":0.496,"15min":0.35199999999999998},"max":{"1min":1.427,"5min":1.665,"15min":15.776},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.151,"15min":1.171},"min":{"1min":0.67600000000000005,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":1.9019999999999999,"5min":2.7549999999999999,"15min":17.620999999999999},"last":0.97799999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2330000000000001,"5min":1.137,"15min":1.22},"min":{"1min":0.74099999999999999,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.671,"5min":1.788,"15min":15.997999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":1.2549999999999999,"5min":1.23,"15min":1.1919999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":1.669,"5min":3.0649999999999999,"15min":3.0649999999999999},"last":1.119}]},{"osd":3,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.129,"15min":1.179},"min":{"1min":0.65000000000000002,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":1.218},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.341,"15min":1.4279999999999999},"min":{"1min":0.86299999999999999,"5min":0.66600000000000004,"15min":0.47399999999999998},"max":{"1min":1.74,"5min":1.827,"15min":16.126000000000001},"last":1.3939999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.466,"15min":1.573},"min":{"1min":1.095,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.9470000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.3280000000000001},{"interface":"front","average":{"1min":1.446,"5min":1.4430000000000001,"15min":1.419},"min":{"1min":1.0089999999999999,"5min":0.82299999999999995,"15min":0.71899999999999997},"max":{"1min":1.913,"5min":3.0099999999999998,"15min":3.0459999999999998},"last":1.353}]},{"osd":5,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4119999999999999,"15min":1.3979999999999999},"min":{"1min":0.97099999999999997,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.7869999999999999,"5min":2.157,"15min":5.5609999999999999},"last":1.631},{"interface":"front","average":{"1min":1.581,"5min":1.5529999999999999,"15min":1.639},"min":{"1min":1.2130000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0630000000000002,"5min":2.1989999999999998,"15min":16.501999999999999},"last":1.756}]},{"osd":6,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.137,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.891,"5min":2.0790000000000002,"15min":5.4509999999999996},"last":1.716},{"interface":"front","average":{"1min":1.649,"5min":1.6379999999999999,"15min":1.7190000000000001},"min":{"1min":0.98899999999999999,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":2.1269999999999998,"5min":2.351,"15min":16.395},"last":1.542}]},{"osd":7,"last update":"Tue Jan 16 15:27:34 2024","interfaces":[{"interface":"back","average":{"1min":1.633,"5min":1.5800000000000001,"15min":1.595},"min":{"1min":0.92600000000000005,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":2.3300000000000001,"5min":2.9510000000000001,"15min":9.6980000000000004},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.589,"15min":1.5860000000000001},"min":{"1min":1.0860000000000001,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.954,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.7769999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:31.723 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.724 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.724 INFO:tasks.ceph:pgid 17.39 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.724 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.724 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.724 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.724 INFO:tasks.ceph:pgid 17.33 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.725 INFO:tasks.ceph:pgid 17.2f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.725 INFO:tasks.ceph:pgid 17.2d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.725 INFO:tasks.ceph:pgid 17.2c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.725 INFO:tasks.ceph:pgid 17.2b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.725 INFO:tasks.ceph:pgid 17.2a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.725 INFO:tasks.ceph:pgid 17.29 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 17.27 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 17.24 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 17.23 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 17.22 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.726 INFO:tasks.ceph:pgid 16.1f last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.727 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.727 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.727 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.727 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.727 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.727 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.728 INFO:tasks.ceph:pgid 16.18 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.728 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.728 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.728 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.728 INFO:tasks.ceph:pgid 16.15 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.728 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.731 INFO:tasks.ceph:pgid 16.14 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.731 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.731 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.731 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.732 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.732 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.732 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.732 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.732 INFO:tasks.ceph:pgid 16.c last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.733 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.733 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.733 INFO:tasks.ceph:pgid 17.8 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.733 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.733 INFO:tasks.ceph:pgid 16.8 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.733 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.734 INFO:tasks.ceph:pgid 16.5 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.734 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.734 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.734 INFO:tasks.ceph:pgid 16.0 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.734 INFO:tasks.ceph:pgid 16.11 last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:31.734 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:28:51.736 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:28:51.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.965+0000 7f045ae00640 1 Processor -- start 2024-01-16T15:28:51.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.965+0000 7f045ae00640 1 -- start start 2024-01-16T15:28:51.966 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.967+0000 7f045ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f0454146770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:51.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.967+0000 7f045ae00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0454071f00 0x7f0454166cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:51.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.967+0000 7f045ae00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540b2e10 con 0x7f045406c7c0 2024-01-16T15:28:51.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.967+0000 7f045ae00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045406cc20 con 0x7f0454071f00 2024-01-16T15:28:51.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.967+0000 7f045ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f045406cec0 con 0x7f0454071410 2024-01-16T15:28:51.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f045a400640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7f0454071f00 0x7f0454166cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f0459000640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f045406c7c0 0x7f04540717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:34364/0 (socket says 172.21.15.53:34364) 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f0459000640 1 -- 172.21.15.53:0/201713333 learned_addr learned my addr 172.21.15.53:0/201713333 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f0459a00640 1 --2- 172.21.15.53:0/201713333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f0454146770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f045a400640 1 -- 172.21.15.53:0/201713333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 msgr2=0x7f0454146770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f045a400640 1 --2- 172.21.15.53:0/201713333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f0454146770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f045a400640 1 -- 172.21.15.53:0/201713333 >> v1:172.21.15.53:6789/0 conn(0x7f045406c7c0 legacy=0x7f04540717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:51.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.968+0000 7f045a400640 1 -- 172.21.15.53:0/201713333 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045406d1a0 con 0x7f0454071f00 2024-01-16T15:28:51.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.969+0000 7f045a400640 1 --2- 172.21.15.53:0/201713333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f0454166cf0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f044c009d90 tx=0x7f044c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c90c4d1969222b9 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f0453e00640 1 -- 172.21.15.53:0/201713333 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c02cc50 con 0x7f0454071f00 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f0453e00640 1 -- 172.21.15.53:0/201713333 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c02cdb0 con 0x7f0454071f00 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f0453e00640 1 -- 172.21.15.53:0/201713333 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c034660 con 0x7f0454071f00 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f045ae00640 1 -- 172.21.15.53:0/201713333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 msgr2=0x7f0454166cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f045ae00640 1 --2- 172.21.15.53:0/201713333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f0454166cf0 secure :-1 s=READY pgs=2069 cs=0 l=1 rev1=1 crypto rx=0x7f044c009d90 tx=0x7f044c02bea0 comp rx=0 tx=0).stop 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f045ae00640 1 -- 172.21.15.53:0/201713333 shutdown_connections 2024-01-16T15:28:51.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.970+0000 7f045ae00640 1 --2- 172.21.15.53:0/201713333 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f0454166cf0 unknown :-1 s=CLOSED pgs=2069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.971+0000 7f045ae00640 1 --2- 172.21.15.53:0/201713333 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f0454146770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.971+0000 7f045ae00640 1 -- 172.21.15.53:0/201713333 >> 172.21.15.53:0/201713333 conn(0x7f045406a800 msgr2=0x7f045406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.971+0000 7f045ae00640 1 -- 172.21.15.53:0/201713333 shutdown_connections 2024-01-16T15:28:51.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.971+0000 7f045ae00640 1 -- 172.21.15.53:0/201713333 wait complete. 2024-01-16T15:28:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.972+0000 7f045ae00640 1 Processor -- start 2024-01-16T15:28:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.973+0000 7f045ae00640 1 -- start start 2024-01-16T15:28:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045ae00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f045406c7c0 0x7f0454133090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045ae00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f04541316a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045ae00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f04540c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:51.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045ae00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0454131ff0 con 0x7f0454071f00 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045ae00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04541322d0 con 0x7f045406c7c0 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045ae00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04541325b0 con 0x7f0454071410 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f04540c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045a400640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f04540c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:33608/0 (socket says 172.21.15.53:33608) 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045a400640 1 -- 172.21.15.53:0/2375281141 learned_addr learned my addr 172.21.15.53:0/2375281141 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f0459000640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f04541316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045a400640 1 -- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 msgr2=0x7f04541316a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045a400640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f04541316a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:51.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.974+0000 7f045a400640 1 -- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f045406c7c0 msgr2=0x7f0454133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f045a400640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f045406c7c0 0x7f0454133090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f045a400640 1 -- 172.21.15.53:0/2375281141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044c009a70 con 0x7f0454071f00 2024-01-16T15:28:51.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f0459000640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f04541316a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:28:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f045a400640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f04540c7190 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f044c009d90 tx=0x7f044c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c040650 con 0x7f0454071f00 2024-01-16T15:28:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c0407d0 con 0x7f0454071f00 2024-01-16T15:28:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0454132890 con 0x7f0454071f00 2024-01-16T15:28:51.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c040b80 con 0x7f0454071f00 2024-01-16T15:28:51.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.975+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0454142de0 con 0x7f0454071f00 2024-01-16T15:28:51.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.977+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0454132db0 con 0x7f0454071f00 2024-01-16T15:28:51.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.978+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 52) v1 ==== 91222+0+0 (secure 0 0 0) 0x7f044c040ce0 con 0x7f0454071f00 2024-01-16T15:28:51.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.979+0000 7f0451600640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f042c06c580 0x7f042c06ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:28:51.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.979+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f044c0ab950 con 0x7f0454071f00 2024-01-16T15:28:51.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.979+0000 7f0459a00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f042c06c580 0x7f042c06ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:28:51.980 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.981+0000 7f0459a00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f042c06c580 0x7f042c06ea40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f045413f950 tx=0x7f0444008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:28:51.983 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:51.984+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f0454132db0 con 0x7f0454071f00 2024-01-16T15:28:52.176 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.176+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0454132440 con 0x7f042c06c580 2024-01-16T15:28:52.201 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.201+0000 7f0451600640 1 -- 172.21.15.53:0/2375281141 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+290424 (secure 0 0 0) 0x7f0454132440 con 0x7f042c06c580 2024-01-16T15:28:52.205 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:28:52.207 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:28:52.208 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f042c06c580 msgr2=0x7f042c06ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f042c06c580 0x7f042c06ea40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f045413f950 tx=0x7f0444008040 comp rx=0 tx=0).stop 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 msgr2=0x7f04540c7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f04540c7190 secure :-1 s=READY pgs=2070 cs=0 l=1 rev1=1 crypto rx=0x7f044c009d90 tx=0x7f044c004a40 comp rx=0 tx=0).stop 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 shutdown_connections 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f042c06c580 0x7f042c06ea40 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f0454071f00 0x7f04540c7190 unknown :-1 s=CLOSED pgs=2070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f0454071410 0x7f04541316a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:52.209 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 --2- 172.21.15.53:0/2375281141 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f045406c7c0 0x7f0454133090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:28:52.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 >> 172.21.15.53:0/2375281141 conn(0x7f045406a800 msgr2=0x7f04540b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:28:52.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.207+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 shutdown_connections 2024-01-16T15:28:52.210 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:28:52.208+0000 7f045ae00640 1 -- 172.21.15.53:0/2375281141 wait complete. 2024-01-16T15:28:52.226 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":978,"stamp":"2024-01-16T15:28:50.927775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6026,"ondisk_log_size":6026,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11133624,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":633331,"kb_avail":743841096,"statfs":{"total":773094113280,"available":761693282304,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":648531504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005083"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257209+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.257209+0000","last_peered":"2024-01-16T15:27:49.257209+0000","last_clean":"2024-01-16T15:27:49.257209+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.257209+0000","last_undegraded":"2024-01-16T15:27:49.257209+0000","last_fullsized":"2024-01-16T15:27:49.257209+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'97","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.510978+0000","last_change":"2024-01-16T15:28:18.511076+0000","last_active":"2024-01-16T15:28:18.510978+0000","last_peered":"2024-01-16T15:28:18.510978+0000","last_clean":"2024-01-16T15:28:18.510978+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:28:18.510978+0000","last_undegraded":"2024-01-16T15:28:18.510978+0000","last_fullsized":"2024-01-16T15:28:18.510978+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'97","last_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_deep_scrub":"112'97","last_deep_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.510802+0000","objects_scrubbed":24,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:33.903169+0000","scrub_duration":0.27194245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'87","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:23.860238+0000","last_change":"2024-01-16T15:28:23.860329+0000","last_active":"2024-01-16T15:28:23.860238+0000","last_peered":"2024-01-16T15:28:23.860238+0000","last_clean":"2024-01-16T15:28:23.860238+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:28:23.860238+0000","last_undegraded":"2024-01-16T15:28:23.860238+0000","last_fullsized":"2024-01-16T15:28:23.860238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_clean_scrub_stamp":"2024-01-16T15:28:23.860048+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:49.716426+0000","scrub_duration":0.27335353899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'56","reported_seq":80,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.233027+0000","last_change":"2024-01-16T15:19:24.635853+0000","last_active":"2024-01-16T15:27:49.233027+0000","last_peered":"2024-01-16T15:27:49.233027+0000","last_clean":"2024-01-16T15:27:49.233027+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:27:49.233027+0000","last_undegraded":"2024-01-16T15:27:49.233027+0000","last_fullsized":"2024-01-16T15:27:49.233027+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'78","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.070128+0000","last_change":"2024-01-16T15:28:25.070176+0000","last_active":"2024-01-16T15:28:25.070128+0000","last_peered":"2024-01-16T15:28:25.070128+0000","last_clean":"2024-01-16T15:28:25.070128+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:28:25.070128+0000","last_undegraded":"2024-01-16T15:28:25.070128+0000","last_fullsized":"2024-01-16T15:28:25.070128+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'78","last_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_deep_scrub":"112'78","last_deep_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.070058+0000","objects_scrubbed":16,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:36.495682+0000","scrub_duration":0.150879281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'99","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.440173+0000","last_change":"2024-01-16T15:28:27.440281+0000","last_active":"2024-01-16T15:28:27.440173+0000","last_peered":"2024-01-16T15:28:27.440173+0000","last_clean":"2024-01-16T15:28:27.440173+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:28:27.440173+0000","last_undegraded":"2024-01-16T15:28:27.440173+0000","last_fullsized":"2024-01-16T15:28:27.440173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.439995+0000","objects_scrubbed":23,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:38.865619+0000","scrub_duration":0.25882440899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'88","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:43.589370+0000","last_change":"2024-01-16T15:28:43.589452+0000","last_active":"2024-01-16T15:28:43.589370+0000","last_peered":"2024-01-16T15:28:43.589370+0000","last_clean":"2024-01-16T15:28:43.589370+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:28:43.589370+0000","last_undegraded":"2024-01-16T15:28:43.589370+0000","last_fullsized":"2024-01-16T15:28:43.589370+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:43.589215+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:43.589215+0000","last_clean_scrub_stamp":"2024-01-16T15:28:43.589215+0000","objects_scrubbed":21,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:23.823898+0000","scrub_duration":0.25936774699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'79","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.799942+0000","last_change":"2024-01-16T15:28:24.800039+0000","last_active":"2024-01-16T15:28:24.799942+0000","last_peered":"2024-01-16T15:28:24.799942+0000","last_clean":"2024-01-16T15:28:24.799942+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:28:24.799942+0000","last_undegraded":"2024-01-16T15:28:24.799942+0000","last_fullsized":"2024-01-16T15:28:24.799942+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.799761+0000","objects_scrubbed":21,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:58.738844+0000","scrub_duration":0.232299336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257379+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:27:49.257379+0000","last_peered":"2024-01-16T15:27:49.257379+0000","last_clean":"2024-01-16T15:27:49.257379+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:27:49.257379+0000","last_undegraded":"2024-01-16T15:27:49.257379+0000","last_fullsized":"2024-01-16T15:27:49.257379+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257401+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:27:49.257401+0000","last_peered":"2024-01-16T15:27:49.257401+0000","last_clean":"2024-01-16T15:27:49.257401+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:27:49.257401+0000","last_undegraded":"2024-01-16T15:27:49.257401+0000","last_fullsized":"2024-01-16T15:27:49.257401+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'93","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.541892+0000","last_change":"2024-01-16T15:28:26.541954+0000","last_active":"2024-01-16T15:28:26.541892+0000","last_peered":"2024-01-16T15:28:26.541892+0000","last_clean":"2024-01-16T15:28:26.541892+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:28:26.541892+0000","last_undegraded":"2024-01-16T15:28:26.541892+0000","last_fullsized":"2024-01-16T15:28:26.541892+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.541809+0000","objects_scrubbed":23,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:57.056238+0000","scrub_duration":0.29071314399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'64","reported_seq":97,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.236372+0000","last_change":"2024-01-16T15:19:24.623058+0000","last_active":"2024-01-16T15:27:49.236372+0000","last_peered":"2024-01-16T15:27:49.236372+0000","last_clean":"2024-01-16T15:27:49.236372+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:27:49.236372+0000","last_undegraded":"2024-01-16T15:27:49.236372+0000","last_fullsized":"2024-01-16T15:27:49.236372+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'87","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:36.975235+0000","last_change":"2024-01-16T15:28:36.975322+0000","last_active":"2024-01-16T15:28:36.975235+0000","last_peered":"2024-01-16T15:28:36.975235+0000","last_clean":"2024-01-16T15:28:36.975235+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:28:36.975235+0000","last_undegraded":"2024-01-16T15:28:36.975235+0000","last_fullsized":"2024-01-16T15:28:36.975235+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:36.974986+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:36.974986+0000","last_clean_scrub_stamp":"2024-01-16T15:28:36.974986+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:42.637017+0000","scrub_duration":0.26518550499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'94","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.914459+0000","last_change":"2024-01-16T15:28:21.914459+0000","last_active":"2024-01-16T15:28:21.914459+0000","last_peered":"2024-01-16T15:28:21.914459+0000","last_clean":"2024-01-16T15:28:21.914459+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:28:21.914459+0000","last_undegraded":"2024-01-16T15:28:21.914459+0000","last_fullsized":"2024-01-16T15:28:21.914459+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.914080+0000","objects_scrubbed":22,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:00.078053+0000","scrub_duration":0.280246104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'71","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.408300+0000","last_change":"2024-01-16T15:28:24.408300+0000","last_active":"2024-01-16T15:28:24.408300+0000","last_peered":"2024-01-16T15:28:24.408300+0000","last_clean":"2024-01-16T15:28:24.408300+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:28:24.408300+0000","last_undegraded":"2024-01-16T15:28:24.408300+0000","last_fullsized":"2024-01-16T15:28:24.408300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'71","last_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_deep_scrub":"112'71","last_deep_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.408030+0000","objects_scrubbed":12,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:51.797494+0000","scrub_duration":0.124741884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'99","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.255207+0000","last_change":"2024-01-16T15:28:18.255207+0000","last_active":"2024-01-16T15:28:18.255207+0000","last_peered":"2024-01-16T15:28:18.255207+0000","last_clean":"2024-01-16T15:28:18.255207+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:28:18.255207+0000","last_undegraded":"2024-01-16T15:28:18.255207+0000","last_fullsized":"2024-01-16T15:28:18.255207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.254974+0000","objects_scrubbed":25,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:57.777992+0000","scrub_duration":0.28941331399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'65","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.443280+0000","last_change":"2024-01-16T15:28:29.443280+0000","last_active":"2024-01-16T15:28:29.443280+0000","last_peered":"2024-01-16T15:28:29.443280+0000","last_clean":"2024-01-16T15:28:29.443280+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:28:29.443280+0000","last_undegraded":"2024-01-16T15:28:29.443280+0000","last_fullsized":"2024-01-16T15:28:29.443280+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'65","last_scrub_stamp":"2024-01-16T15:28:29.443041+0000","last_deep_scrub":"112'65","last_deep_scrub_stamp":"2024-01-16T15:28:29.443041+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.443041+0000","objects_scrubbed":13,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:23.604246+0000","scrub_duration":0.17569420299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'108","reported_seq":152,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.741028+0000","last_change":"2024-01-16T15:28:26.741028+0000","last_active":"2024-01-16T15:28:26.741028+0000","last_peered":"2024-01-16T15:28:26.741028+0000","last_clean":"2024-01-16T15:28:26.741028+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:28:26.741028+0000","last_undegraded":"2024-01-16T15:28:26.741028+0000","last_fullsized":"2024-01-16T15:28:26.741028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.740929+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:29.967191+0000","scrub_duration":0.29640463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'100","reported_seq":132,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.925516+0000","last_change":"2024-01-16T15:28:28.925578+0000","last_active":"2024-01-16T15:28:28.925516+0000","last_peered":"2024-01-16T15:28:28.925516+0000","last_clean":"2024-01-16T15:28:28.925516+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:28:28.925516+0000","last_undegraded":"2024-01-16T15:28:28.925516+0000","last_fullsized":"2024-01-16T15:28:28.925516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'100","last_scrub_stamp":"2024-01-16T15:28:28.925340+0000","last_deep_scrub":"112'100","last_deep_scrub_stamp":"2024-01-16T15:28:28.925340+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.925340+0000","objects_scrubbed":26,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:24:04.310360+0000","scrub_duration":0.30850802500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'90","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.497303+0000","last_change":"2024-01-16T15:28:28.497397+0000","last_active":"2024-01-16T15:28:28.497303+0000","last_peered":"2024-01-16T15:28:28.497303+0000","last_clean":"2024-01-16T15:28:28.497303+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:28:28.497303+0000","last_undegraded":"2024-01-16T15:28:28.497303+0000","last_fullsized":"2024-01-16T15:28:28.497303+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:28.497088+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:28.497088+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.497088+0000","objects_scrubbed":20,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:34.899596+0000","scrub_duration":0.25302828599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'95","reported_seq":134,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.497418+0000","last_change":"2024-01-16T15:28:32.497568+0000","last_active":"2024-01-16T15:28:32.497418+0000","last_peered":"2024-01-16T15:28:32.497418+0000","last_clean":"2024-01-16T15:28:32.497418+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:28:32.497418+0000","last_undegraded":"2024-01-16T15:28:32.497418+0000","last_fullsized":"2024-01-16T15:28:32.497418+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'95","last_scrub_stamp":"2024-01-16T15:28:32.497207+0000","last_deep_scrub":"112'95","last_deep_scrub_stamp":"2024-01-16T15:28:32.497207+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.497207+0000","objects_scrubbed":22,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:13.297614+0000","scrub_duration":0.23998999500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'94","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:35.999324+0000","last_change":"2024-01-16T15:28:35.999421+0000","last_active":"2024-01-16T15:28:35.999324+0000","last_peered":"2024-01-16T15:28:35.999324+0000","last_clean":"2024-01-16T15:28:35.999324+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:28:35.999324+0000","last_undegraded":"2024-01-16T15:28:35.999324+0000","last_fullsized":"2024-01-16T15:28:35.999324+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:35.999128+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:35.999128+0000","last_clean_scrub_stamp":"2024-01-16T15:28:35.999128+0000","objects_scrubbed":23,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:18.663684+0000","scrub_duration":0.30044927999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'81","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.481673+0000","last_change":"2024-01-16T15:28:33.481747+0000","last_active":"2024-01-16T15:28:33.481673+0000","last_peered":"2024-01-16T15:28:33.481673+0000","last_clean":"2024-01-16T15:28:33.481673+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:28:33.481673+0000","last_undegraded":"2024-01-16T15:28:33.481673+0000","last_fullsized":"2024-01-16T15:28:33.481673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:33.481512+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:33.481512+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.481512+0000","objects_scrubbed":18,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:34.932260+0000","scrub_duration":0.20372825899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'90","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.951640+0000","last_change":"2024-01-16T15:28:20.951722+0000","last_active":"2024-01-16T15:28:20.951640+0000","last_peered":"2024-01-16T15:28:20.951640+0000","last_clean":"2024-01-16T15:28:20.951640+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:28:20.951640+0000","last_undegraded":"2024-01-16T15:28:20.951640+0000","last_fullsized":"2024-01-16T15:28:20.951640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.951444+0000","objects_scrubbed":19,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:52.025141+0000","scrub_duration":0.20406614100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'89","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.496415+0000","last_change":"2024-01-16T15:28:27.496503+0000","last_active":"2024-01-16T15:28:27.496415+0000","last_peered":"2024-01-16T15:28:27.496415+0000","last_clean":"2024-01-16T15:28:27.496415+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:28:27.496415+0000","last_undegraded":"2024-01-16T15:28:27.496415+0000","last_fullsized":"2024-01-16T15:28:27.496415+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:27.496240+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:27.496240+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.496240+0000","objects_scrubbed":20,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:37.199022+0000","scrub_duration":0.25531969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566596+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:28:05.566596+0000","last_peered":"2024-01-16T15:28:05.566596+0000","last_clean":"2024-01-16T15:28:05.566596+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:28:05.566596+0000","last_undegraded":"2024-01-16T15:28:05.566596+0000","last_fullsized":"2024-01-16T15:28:05.566596+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.302192+0000","last_change":"2024-01-16T15:28:28.302272+0000","last_active":"2024-01-16T15:28:28.302192+0000","last_peered":"2024-01-16T15:28:28.302192+0000","last_clean":"2024-01-16T15:28:28.302192+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:28:28.302192+0000","last_undegraded":"2024-01-16T15:28:28.302192+0000","last_fullsized":"2024-01-16T15:28:28.302192+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.302044+0000","objects_scrubbed":16,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:38.499053+0000","scrub_duration":0.149915824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'63","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:44.708897+0000","last_change":"2024-01-16T15:28:44.708973+0000","last_active":"2024-01-16T15:28:44.708897+0000","last_peered":"2024-01-16T15:28:44.708897+0000","last_clean":"2024-01-16T15:28:44.708897+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:28:44.708897+0000","last_undegraded":"2024-01-16T15:28:44.708897+0000","last_fullsized":"2024-01-16T15:28:44.708897+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'63","last_scrub_stamp":"2024-01-16T15:28:44.708715+0000","last_deep_scrub":"112'63","last_deep_scrub_stamp":"2024-01-16T15:28:44.708715+0000","last_clean_scrub_stamp":"2024-01-16T15:28:44.708715+0000","objects_scrubbed":13,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:25.059080+0000","scrub_duration":0.14963963599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'81","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.316071+0000","last_change":"2024-01-16T15:28:40.316137+0000","last_active":"2024-01-16T15:28:40.316071+0000","last_peered":"2024-01-16T15:28:40.316071+0000","last_clean":"2024-01-16T15:28:40.316071+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:28:40.316071+0000","last_undegraded":"2024-01-16T15:28:40.316071+0000","last_fullsized":"2024-01-16T15:28:40.316071+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:40.315969+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:40.315969+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.315969+0000","objects_scrubbed":19,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:11.371995+0000","scrub_duration":0.23402421100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'95","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.537448+0000","last_change":"2024-01-16T15:28:42.537543+0000","last_active":"2024-01-16T15:28:42.537448+0000","last_peered":"2024-01-16T15:28:42.537448+0000","last_clean":"2024-01-16T15:28:42.537448+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:28:42.537448+0000","last_undegraded":"2024-01-16T15:28:42.537448+0000","last_fullsized":"2024-01-16T15:28:42.537448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'95","last_scrub_stamp":"2024-01-16T15:28:42.537268+0000","last_deep_scrub":"112'95","last_deep_scrub_stamp":"2024-01-16T15:28:42.537268+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.537268+0000","objects_scrubbed":17,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:49.101480+0000","scrub_duration":0.19056937099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'117","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.343525+0000","last_change":"2024-01-16T15:28:29.343618+0000","last_active":"2024-01-16T15:28:29.343525+0000","last_peered":"2024-01-16T15:28:29.343525+0000","last_clean":"2024-01-16T15:28:29.343525+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:28:29.343525+0000","last_undegraded":"2024-01-16T15:28:29.343525+0000","last_fullsized":"2024-01-16T15:28:29.343525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'117","last_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_deep_scrub":"112'117","last_deep_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.343355+0000","objects_scrubbed":26,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:18.083505+0000","scrub_duration":0.31588118900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'102","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.453254+0000","last_change":"2024-01-16T15:28:19.453335+0000","last_active":"2024-01-16T15:28:19.453254+0000","last_peered":"2024-01-16T15:28:19.453254+0000","last_clean":"2024-01-16T15:28:19.453254+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:28:19.453254+0000","last_undegraded":"2024-01-16T15:28:19.453254+0000","last_fullsized":"2024-01-16T15:28:19.453254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'102","last_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_deep_scrub":"112'102","last_deep_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.453110+0000","objects_scrubbed":24,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.543525+0000","scrub_duration":0.24508891399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'81","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.343081+0000","last_change":"2024-01-16T15:28:33.343229+0000","last_active":"2024-01-16T15:28:33.343081+0000","last_peered":"2024-01-16T15:28:33.343081+0000","last_clean":"2024-01-16T15:28:33.343081+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:28:33.343081+0000","last_undegraded":"2024-01-16T15:28:33.343081+0000","last_fullsized":"2024-01-16T15:28:33.343081+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:33.342865+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:33.342865+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.342865+0000","objects_scrubbed":21,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:09.821872+0000","scrub_duration":0.226300789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'6","reported_seq":33,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:41.310657+0000","last_change":"2024-01-16T15:28:41.310726+0000","last_active":"2024-01-16T15:28:41.310657+0000","last_peered":"2024-01-16T15:28:41.310657+0000","last_clean":"2024-01-16T15:28:41.310657+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:28:41.310657+0000","last_undegraded":"2024-01-16T15:28:41.310657+0000","last_fullsized":"2024-01-16T15:28:41.310657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'6","last_scrub_stamp":"2024-01-16T15:28:41.310472+0000","last_deep_scrub":"112'6","last_deep_scrub_stamp":"2024-01-16T15:28:41.310472+0000","last_clean_scrub_stamp":"2024-01-16T15:28:41.310472+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:08.138176+0000","scrub_duration":0.0084308639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.835785+0000","last_change":"2024-01-16T15:28:32.835862+0000","last_active":"2024-01-16T15:28:32.835785+0000","last_peered":"2024-01-16T15:28:32.835785+0000","last_clean":"2024-01-16T15:28:32.835785+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:28:32.835785+0000","last_undegraded":"2024-01-16T15:28:32.835785+0000","last_fullsized":"2024-01-16T15:28:32.835785+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:32.835644+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:32.835644+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.835644+0000","objects_scrubbed":21,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:16:19.714227+0000","scrub_duration":0.264673932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.589919+0000","last_change":"2024-01-16T15:28:20.589975+0000","last_active":"2024-01-16T15:28:20.589919+0000","last_peered":"2024-01-16T15:28:20.589919+0000","last_clean":"2024-01-16T15:28:20.589919+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:28:20.589919+0000","last_undegraded":"2024-01-16T15:28:20.589919+0000","last_fullsized":"2024-01-16T15:28:20.589919+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.589720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:37.660795+0000","scrub_duration":0.0035764669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'85","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.351967+0000","last_change":"2024-01-16T15:28:30.352108+0000","last_active":"2024-01-16T15:28:30.351967+0000","last_peered":"2024-01-16T15:28:30.351967+0000","last_clean":"2024-01-16T15:28:30.351967+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:28:30.351967+0000","last_undegraded":"2024-01-16T15:28:30.351967+0000","last_fullsized":"2024-01-16T15:28:30.351967+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.351810+0000","objects_scrubbed":19,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:22.513330+0000","scrub_duration":0.19453525299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'4","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.865026+0000","last_change":"2024-01-16T15:28:10.865070+0000","last_active":"2024-01-16T15:28:10.865026+0000","last_peered":"2024-01-16T15:28:10.865026+0000","last_clean":"2024-01-16T15:28:10.865026+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:28:10.865026+0000","last_undegraded":"2024-01-16T15:28:10.865026+0000","last_fullsized":"2024-01-16T15:28:10.865026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.864918+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:32.896445+0000","scrub_duration":0.0059496970000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'109","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.975751+0000","last_change":"2024-01-16T15:28:30.975803+0000","last_active":"2024-01-16T15:28:30.975751+0000","last_peered":"2024-01-16T15:28:30.975751+0000","last_clean":"2024-01-16T15:28:30.975751+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:28:30.975751+0000","last_undegraded":"2024-01-16T15:28:30.975751+0000","last_fullsized":"2024-01-16T15:28:30.975751+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'109","last_scrub_stamp":"2024-01-16T15:28:30.975585+0000","last_deep_scrub":"112'109","last_deep_scrub_stamp":"2024-01-16T15:28:30.975585+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.975585+0000","objects_scrubbed":25,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:50.616075+0000","scrub_duration":0.273993918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'3","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:13.789441+0000","last_change":"2024-01-16T15:28:13.789502+0000","last_active":"2024-01-16T15:28:13.789441+0000","last_peered":"2024-01-16T15:28:13.789441+0000","last_clean":"2024-01-16T15:28:13.789441+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:28:13.789441+0000","last_undegraded":"2024-01-16T15:28:13.789441+0000","last_fullsized":"2024-01-16T15:28:13.789441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_clean_scrub_stamp":"2024-01-16T15:28:13.789255+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:19.022780+0000","scrub_duration":0.0072576510000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'94","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:31.165009+0000","last_change":"2024-01-16T15:28:31.165088+0000","last_active":"2024-01-16T15:28:31.165009+0000","last_peered":"2024-01-16T15:28:31.165009+0000","last_clean":"2024-01-16T15:28:31.165009+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:28:31.165009+0000","last_undegraded":"2024-01-16T15:28:31.165009+0000","last_fullsized":"2024-01-16T15:28:31.165009+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:31.164841+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:31.164841+0000","last_clean_scrub_stamp":"2024-01-16T15:28:31.164841+0000","objects_scrubbed":16,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:05.015647+0000","scrub_duration":0.15844903499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.260333+0000","last_change":"2024-01-16T15:28:24.260389+0000","last_active":"2024-01-16T15:28:24.260333+0000","last_peered":"2024-01-16T15:28:24.260333+0000","last_clean":"2024-01-16T15:28:24.260333+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:28:24.260333+0000","last_undegraded":"2024-01-16T15:28:24.260333+0000","last_fullsized":"2024-01-16T15:28:24.260333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.260116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:22.819324+0000","scrub_duration":0.0042817660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'108","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:39.416380+0000","last_change":"2024-01-16T15:28:39.416442+0000","last_active":"2024-01-16T15:28:39.416380+0000","last_peered":"2024-01-16T15:28:39.416380+0000","last_clean":"2024-01-16T15:28:39.416380+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:28:39.416380+0000","last_undegraded":"2024-01-16T15:28:39.416380+0000","last_fullsized":"2024-01-16T15:28:39.416380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:39.416188+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:39.416188+0000","last_clean_scrub_stamp":"2024-01-16T15:28:39.416188+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:12.499311+0000","scrub_duration":0.29096126300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.816242+0000","last_change":"2024-01-16T15:28:14.816296+0000","last_active":"2024-01-16T15:28:14.816242+0000","last_peered":"2024-01-16T15:28:14.816242+0000","last_clean":"2024-01-16T15:28:14.816242+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:28:14.816242+0000","last_undegraded":"2024-01-16T15:28:14.816242+0000","last_fullsized":"2024-01-16T15:28:14.816242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.816039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:58.559289+0000","scrub_duration":0.0043054030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'80","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.250263+0000","last_change":"2024-01-16T15:19:24.621722+0000","last_active":"2024-01-16T15:27:49.250263+0000","last_peered":"2024-01-16T15:27:49.250263+0000","last_clean":"2024-01-16T15:27:49.250263+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:27:49.250263+0000","last_undegraded":"2024-01-16T15:27:49.250263+0000","last_fullsized":"2024-01-16T15:27:49.250263+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":26,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.251244+0000","last_change":"2024-01-16T15:28:11.251291+0000","last_active":"2024-01-16T15:28:11.251244+0000","last_peered":"2024-01-16T15:28:11.251244+0000","last_clean":"2024-01-16T15:28:11.251244+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:28:11.251244+0000","last_undegraded":"2024-01-16T15:28:11.251244+0000","last_fullsized":"2024-01-16T15:28:11.251244+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.251026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:45.371008+0000","scrub_duration":0.0038716129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'86","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:44.544342+0000","last_change":"2024-01-16T15:28:44.544412+0000","last_active":"2024-01-16T15:28:44.544342+0000","last_peered":"2024-01-16T15:28:44.544342+0000","last_clean":"2024-01-16T15:28:44.544342+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:28:44.544342+0000","last_undegraded":"2024-01-16T15:28:44.544342+0000","last_fullsized":"2024-01-16T15:28:44.544342+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'86","last_scrub_stamp":"2024-01-16T15:28:44.544211+0000","last_deep_scrub":"112'86","last_deep_scrub_stamp":"2024-01-16T15:28:44.544211+0000","last_clean_scrub_stamp":"2024-01-16T15:28:44.544211+0000","objects_scrubbed":18,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:52.626917+0000","scrub_duration":0.22887946100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":26,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.060990+0000","last_change":"2024-01-16T15:28:42.061040+0000","last_active":"2024-01-16T15:28:42.060990+0000","last_peered":"2024-01-16T15:28:42.060990+0000","last_clean":"2024-01-16T15:28:42.060990+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:28:42.060990+0000","last_undegraded":"2024-01-16T15:28:42.060990+0000","last_fullsized":"2024-01-16T15:28:42.060990+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:42.060723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:42.060723+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.060723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:46.204134+0000","scrub_duration":0.0036082470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'66","reported_seq":87,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.242520+0000","last_change":"2024-01-16T15:19:24.633286+0000","last_active":"2024-01-16T15:27:49.242520+0000","last_peered":"2024-01-16T15:27:49.242520+0000","last_clean":"2024-01-16T15:27:49.242520+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:27:49.242520+0000","last_undegraded":"2024-01-16T15:27:49.242520+0000","last_fullsized":"2024-01-16T15:27:49.242520+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.843215+0000","last_change":"2024-01-16T15:28:11.843271+0000","last_active":"2024-01-16T15:28:11.843215+0000","last_peered":"2024-01-16T15:28:11.843215+0000","last_clean":"2024-01-16T15:28:11.843215+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:28:11.843215+0000","last_undegraded":"2024-01-16T15:28:11.843215+0000","last_fullsized":"2024-01-16T15:28:11.843215+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.843032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:01.354272+0000","scrub_duration":0.003520572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'82","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.726716+0000","last_change":"2024-01-16T15:28:29.726792+0000","last_active":"2024-01-16T15:28:29.726716+0000","last_peered":"2024-01-16T15:28:29.726716+0000","last_clean":"2024-01-16T15:28:29.726716+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:28:29.726716+0000","last_undegraded":"2024-01-16T15:28:29.726716+0000","last_fullsized":"2024-01-16T15:28:29.726716+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:29.726602+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:29.726602+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.726602+0000","objects_scrubbed":18,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:25.349562+0000","scrub_duration":0.187189361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":26,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.784304+0000","last_change":"2024-01-16T15:28:16.784389+0000","last_active":"2024-01-16T15:28:16.784304+0000","last_peered":"2024-01-16T15:28:16.784304+0000","last_clean":"2024-01-16T15:28:16.784304+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:28:16.784304+0000","last_undegraded":"2024-01-16T15:28:16.784304+0000","last_fullsized":"2024-01-16T15:28:16.784304+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.784086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:32.176453+0000","scrub_duration":0.0040169810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'79","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.831200+0000","last_change":"2024-01-16T15:28:42.831200+0000","last_active":"2024-01-16T15:28:42.831200+0000","last_peered":"2024-01-16T15:28:42.831200+0000","last_clean":"2024-01-16T15:28:42.831200+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:28:42.831200+0000","last_undegraded":"2024-01-16T15:28:42.831200+0000","last_fullsized":"2024-01-16T15:28:42.831200+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:42.830943+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:42.830943+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.830943+0000","objects_scrubbed":19,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:48.006952+0000","scrub_duration":0.21546314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'33","reported_seq":59,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:50.170194+0000","last_change":"2024-01-16T15:28:50.170253+0000","last_active":"2024-01-16T15:28:50.170194+0000","last_peered":"2024-01-16T15:28:50.170194+0000","last_clean":"2024-01-16T15:28:50.170194+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:28:50.170194+0000","last_undegraded":"2024-01-16T15:28:50.170194+0000","last_fullsized":"2024-01-16T15:28:50.170194+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'33","last_scrub_stamp":"2024-01-16T15:28:50.169974+0000","last_deep_scrub":"112'33","last_deep_scrub_stamp":"2024-01-16T15:28:50.169974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:50.169974+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:02.553184+0000","scrub_duration":0.008481901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'93","reported_seq":128,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:45.595567+0000","last_change":"2024-01-16T15:28:45.595636+0000","last_active":"2024-01-16T15:28:45.595567+0000","last_peered":"2024-01-16T15:28:45.595567+0000","last_clean":"2024-01-16T15:28:45.595567+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:28:45.595567+0000","last_undegraded":"2024-01-16T15:28:45.595567+0000","last_fullsized":"2024-01-16T15:28:45.595567+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:45.595459+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:45.595459+0000","last_clean_scrub_stamp":"2024-01-16T15:28:45.595459+0000","objects_scrubbed":24,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:41:29.813183+0000","scrub_duration":0.29598812099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'311","reported_seq":338,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:48.165089+0000","last_change":"2024-01-16T15:28:48.165251+0000","last_active":"2024-01-16T15:28:48.165089+0000","last_peered":"2024-01-16T15:28:48.165089+0000","last_clean":"2024-01-16T15:28:48.165089+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:28:48.165089+0000","last_undegraded":"2024-01-16T15:28:48.165089+0000","last_fullsized":"2024-01-16T15:28:48.165089+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'311","last_scrub_stamp":"2024-01-16T15:28:48.164882+0000","last_deep_scrub":"112'311","last_deep_scrub_stamp":"2024-01-16T15:28:48.164882+0000","last_clean_scrub_stamp":"2024-01-16T15:28:48.164882+0000","objects_scrubbed":4,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:23.660630+0000","scrub_duration":0.020197336999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'79","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.880948+0000","last_change":"2024-01-16T15:28:33.880948+0000","last_active":"2024-01-16T15:28:33.880948+0000","last_peered":"2024-01-16T15:28:33.880948+0000","last_clean":"2024-01-16T15:28:33.880948+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:28:33.880948+0000","last_undegraded":"2024-01-16T15:28:33.880948+0000","last_fullsized":"2024-01-16T15:28:33.880948+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:33.880692+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:33.880692+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.880692+0000","objects_scrubbed":16,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:30.122032+0000","scrub_duration":0.177224733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.219036+0000","last_change":"2024-01-16T15:28:18.219036+0000","last_active":"2024-01-16T15:28:18.219036+0000","last_peered":"2024-01-16T15:28:18.219036+0000","last_clean":"2024-01-16T15:28:18.219036+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:28:18.219036+0000","last_undegraded":"2024-01-16T15:28:18.219036+0000","last_fullsized":"2024-01-16T15:28:18.219036+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.218660+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:48.176772+0000","scrub_duration":0.0091646809999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'98","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:36.911299+0000","last_change":"2024-01-16T15:28:36.911299+0000","last_active":"2024-01-16T15:28:36.911299+0000","last_peered":"2024-01-16T15:28:36.911299+0000","last_clean":"2024-01-16T15:28:36.911299+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:28:36.911299+0000","last_undegraded":"2024-01-16T15:28:36.911299+0000","last_fullsized":"2024-01-16T15:28:36.911299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'98","last_scrub_stamp":"2024-01-16T15:28:36.911039+0000","last_deep_scrub":"112'98","last_deep_scrub_stamp":"2024-01-16T15:28:36.911039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:36.911039+0000","objects_scrubbed":18,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:57.499710+0000","scrub_duration":0.20160798599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.450274+0000","last_change":"2024-01-16T15:28:16.450274+0000","last_active":"2024-01-16T15:28:16.450274+0000","last_peered":"2024-01-16T15:28:16.450274+0000","last_clean":"2024-01-16T15:28:16.450274+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:28:16.450274+0000","last_undegraded":"2024-01-16T15:28:16.450274+0000","last_fullsized":"2024-01-16T15:28:16.450274+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.449979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:39.537507+0000","scrub_duration":0.0036212340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'96","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.977310+0000","last_change":"2024-01-16T15:28:32.977310+0000","last_active":"2024-01-16T15:28:32.977310+0000","last_peered":"2024-01-16T15:28:32.977310+0000","last_clean":"2024-01-16T15:28:32.977310+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:28:32.977310+0000","last_undegraded":"2024-01-16T15:28:32.977310+0000","last_fullsized":"2024-01-16T15:28:32.977310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:32.977043+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:32.977043+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.977043+0000","objects_scrubbed":23,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:27:22.982613+0000","scrub_duration":0.25583332399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'12","reported_seq":39,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.825453+0000","last_change":"2024-01-16T15:28:07.825453+0000","last_active":"2024-01-16T15:28:07.825453+0000","last_peered":"2024-01-16T15:28:07.825453+0000","last_clean":"2024-01-16T15:28:07.825453+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:28:07.825453+0000","last_undegraded":"2024-01-16T15:28:07.825453+0000","last_fullsized":"2024-01-16T15:28:07.825453+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.825335+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:20.349610+0000","scrub_duration":0.0056472520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'62","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.989502+0000","last_change":"2024-01-16T15:28:21.989502+0000","last_active":"2024-01-16T15:28:21.989502+0000","last_peered":"2024-01-16T15:28:21.989502+0000","last_clean":"2024-01-16T15:28:21.989502+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:28:21.989502+0000","last_undegraded":"2024-01-16T15:28:21.989502+0000","last_fullsized":"2024-01-16T15:28:21.989502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'62","last_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_deep_scrub":"112'62","last_deep_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.989317+0000","objects_scrubbed":12,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:49.378780+0000","scrub_duration":0.125520883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"109'1","reported_seq":20,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566191+0000","last_change":"2024-01-16T15:19:23.627441+0000","last_active":"2024-01-16T15:28:05.566191+0000","last_peered":"2024-01-16T15:28:05.566191+0000","last_clean":"2024-01-16T15:28:05.566191+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:28:05.566191+0000","last_undegraded":"2024-01-16T15:28:05.566191+0000","last_fullsized":"2024-01-16T15:28:05.566191+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:22.561508+0000","last_clean_scrub_stamp":"2024-01-16T15:19:22.561508+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'81","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:34.355627+0000","last_change":"2024-01-16T15:28:34.355708+0000","last_active":"2024-01-16T15:28:34.355627+0000","last_peered":"2024-01-16T15:28:34.355627+0000","last_clean":"2024-01-16T15:28:34.355627+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:28:34.355627+0000","last_undegraded":"2024-01-16T15:28:34.355627+0000","last_fullsized":"2024-01-16T15:28:34.355627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:34.355450+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:34.355450+0000","last_clean_scrub_stamp":"2024-01-16T15:28:34.355450+0000","objects_scrubbed":20,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:13.088452+0000","scrub_duration":0.20144817200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.809350+0000","last_change":"2024-01-16T15:28:17.809350+0000","last_active":"2024-01-16T15:28:17.809350+0000","last_peered":"2024-01-16T15:28:17.809350+0000","last_clean":"2024-01-16T15:28:17.809350+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:28:17.809350+0000","last_undegraded":"2024-01-16T15:28:17.809350+0000","last_fullsized":"2024-01-16T15:28:17.809350+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.809101+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:47.767212+0000","scrub_duration":0.0070965819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'100","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.997559+0000","last_change":"2024-01-16T15:28:40.997649+0000","last_active":"2024-01-16T15:28:40.997559+0000","last_peered":"2024-01-16T15:28:40.997559+0000","last_clean":"2024-01-16T15:28:40.997559+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:28:40.997559+0000","last_undegraded":"2024-01-16T15:28:40.997559+0000","last_fullsized":"2024-01-16T15:28:40.997559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'100","last_scrub_stamp":"2024-01-16T15:28:40.997402+0000","last_deep_scrub":"112'100","last_deep_scrub_stamp":"2024-01-16T15:28:40.997402+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.997402+0000","objects_scrubbed":23,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:49.477973+0000","scrub_duration":0.25693853799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'2","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.443869+0000","last_change":"2024-01-16T15:28:18.443869+0000","last_active":"2024-01-16T15:28:18.443869+0000","last_peered":"2024-01-16T15:28:18.443869+0000","last_clean":"2024-01-16T15:28:18.443869+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:28:18.443869+0000","last_undegraded":"2024-01-16T15:28:18.443869+0000","last_fullsized":"2024-01-16T15:28:18.443869+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.443623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:31.553582+0000","scrub_duration":0.0077739760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'89","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:34.943163+0000","last_change":"2024-01-16T15:28:34.943251+0000","last_active":"2024-01-16T15:28:34.943163+0000","last_peered":"2024-01-16T15:28:34.943163+0000","last_clean":"2024-01-16T15:28:34.943163+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:28:34.943163+0000","last_undegraded":"2024-01-16T15:28:34.943163+0000","last_fullsized":"2024-01-16T15:28:34.943163+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:34.942995+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:34.942995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:34.942995+0000","objects_scrubbed":23,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:58.983028+0000","scrub_duration":0.28306326100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.220561+0000","last_change":"2024-01-16T15:28:32.220610+0000","last_active":"2024-01-16T15:28:32.220561+0000","last_peered":"2024-01-16T15:28:32.220561+0000","last_clean":"2024-01-16T15:28:32.220561+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:28:32.220561+0000","last_undegraded":"2024-01-16T15:28:32.220561+0000","last_fullsized":"2024-01-16T15:28:32.220561+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:32.220364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:32.220364+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.220364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:02.571057+0000","scrub_duration":0.0041910580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'89","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.838225+0000","last_change":"2024-01-16T15:28:26.838277+0000","last_active":"2024-01-16T15:28:26.838225+0000","last_peered":"2024-01-16T15:28:26.838225+0000","last_clean":"2024-01-16T15:28:26.838225+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:28:26.838225+0000","last_undegraded":"2024-01-16T15:28:26.838225+0000","last_fullsized":"2024-01-16T15:28:26.838225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:26.838060+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:26.838060+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.838060+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:31.159826+0000","scrub_duration":0.27286994599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.174817+0000","last_change":"2024-01-16T15:28:17.174817+0000","last_active":"2024-01-16T15:28:17.174817+0000","last_peered":"2024-01-16T15:28:17.174817+0000","last_clean":"2024-01-16T15:28:17.174817+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:28:17.174817+0000","last_undegraded":"2024-01-16T15:28:17.174817+0000","last_fullsized":"2024-01-16T15:28:17.174817+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.174587+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:21.215097+0000","scrub_duration":0.0071591010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566466+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:28:05.566466+0000","last_peered":"2024-01-16T15:28:05.566466+0000","last_clean":"2024-01-16T15:28:05.566466+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:28:05.566466+0000","last_undegraded":"2024-01-16T15:28:05.566466+0000","last_fullsized":"2024-01-16T15:28:05.566466+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.833365+0000","last_change":"2024-01-16T15:28:19.833414+0000","last_active":"2024-01-16T15:28:19.833365+0000","last_peered":"2024-01-16T15:28:19.833365+0000","last_clean":"2024-01-16T15:28:19.833365+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:28:19.833365+0000","last_undegraded":"2024-01-16T15:28:19.833365+0000","last_fullsized":"2024-01-16T15:28:19.833365+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.923614+0000","scrub_duration":0.0036415929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'84","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.813828+0000","last_change":"2024-01-16T15:28:29.813905+0000","last_active":"2024-01-16T15:28:29.813828+0000","last_peered":"2024-01-16T15:28:29.813828+0000","last_clean":"2024-01-16T15:28:29.813828+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:28:29.813828+0000","last_undegraded":"2024-01-16T15:28:29.813828+0000","last_fullsized":"2024-01-16T15:28:29.813828+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'84","last_scrub_stamp":"2024-01-16T15:28:29.813679+0000","last_deep_scrub":"112'84","last_deep_scrub_stamp":"2024-01-16T15:28:29.813679+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.813679+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:02.832647+0000","scrub_duration":0.15799079199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.206909+0000","last_change":"2024-01-16T15:28:15.206975+0000","last_active":"2024-01-16T15:28:15.206909+0000","last_peered":"2024-01-16T15:28:15.206909+0000","last_clean":"2024-01-16T15:28:15.206909+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:28:15.206909+0000","last_undegraded":"2024-01-16T15:28:15.206909+0000","last_fullsized":"2024-01-16T15:28:15.206909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.206765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:27.403194+0000","scrub_duration":0.0041188539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.874708+0000","last_change":"2024-01-16T15:28:30.874778+0000","last_active":"2024-01-16T15:28:30.874708+0000","last_peered":"2024-01-16T15:28:30.874708+0000","last_clean":"2024-01-16T15:28:30.874708+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:28:30.874708+0000","last_undegraded":"2024-01-16T15:28:30.874708+0000","last_fullsized":"2024-01-16T15:28:30.874708+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:30.874583+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:30.874583+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.874583+0000","objects_scrubbed":18,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:07.770194+0000","scrub_duration":0.163030705,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.044284+0000","last_change":"2024-01-16T15:28:40.044333+0000","last_active":"2024-01-16T15:28:40.044284+0000","last_peered":"2024-01-16T15:28:40.044284+0000","last_clean":"2024-01-16T15:28:40.044284+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:28:40.044284+0000","last_undegraded":"2024-01-16T15:28:40.044284+0000","last_fullsized":"2024-01-16T15:28:40.044284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:40.044093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:40.044093+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.044093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:49.863547+0000","scrub_duration":0.0043197859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'92","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:22.637191+0000","last_change":"2024-01-16T15:28:22.637284+0000","last_active":"2024-01-16T15:28:22.637191+0000","last_peered":"2024-01-16T15:28:22.637191+0000","last_clean":"2024-01-16T15:28:22.637191+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:28:22.637191+0000","last_undegraded":"2024-01-16T15:28:22.637191+0000","last_fullsized":"2024-01-16T15:28:22.637191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'92","last_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_deep_scrub":"112'92","last_deep_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_clean_scrub_stamp":"2024-01-16T15:28:22.636970+0000","objects_scrubbed":22,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:01.808019+0000","scrub_duration":0.23588124099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.749499+0000","last_change":"2024-01-16T15:28:17.749653+0000","last_active":"2024-01-16T15:28:17.749499+0000","last_peered":"2024-01-16T15:28:17.749499+0000","last_clean":"2024-01-16T15:28:17.749499+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:28:17.749499+0000","last_undegraded":"2024-01-16T15:28:17.749499+0000","last_fullsized":"2024-01-16T15:28:17.749499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.749294+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:03.835812+0000","scrub_duration":0.0071287670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566405+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:28:05.566405+0000","last_peered":"2024-01-16T15:28:05.566405+0000","last_clean":"2024-01-16T15:28:05.566405+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:28:05.566405+0000","last_undegraded":"2024-01-16T15:28:05.566405+0000","last_fullsized":"2024-01-16T15:28:05.566405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":58,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.479780+0000","last_change":"2024-01-16T15:28:17.479819+0000","last_active":"2024-01-16T15:28:17.479780+0000","last_peered":"2024-01-16T15:28:17.479780+0000","last_clean":"2024-01-16T15:28:17.479780+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:28:17.479780+0000","last_undegraded":"2024-01-16T15:28:17.479780+0000","last_fullsized":"2024-01-16T15:28:17.479780+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'31","last_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_deep_scrub":"112'31","last_deep_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.479610+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:48.802592+0000","scrub_duration":0.0060914480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593,"num_omap_keys":11,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'96","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.235685+0000","last_change":"2024-01-16T15:28:15.235729+0000","last_active":"2024-01-16T15:28:15.235685+0000","last_peered":"2024-01-16T15:28:15.235685+0000","last_clean":"2024-01-16T15:28:15.235685+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:28:15.235685+0000","last_undegraded":"2024-01-16T15:28:15.235685+0000","last_fullsized":"2024-01-16T15:28:15.235685+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.235576+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:39.275609+0000","scrub_duration":0.27566697899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:47.140086+0000","last_change":"2024-01-16T15:28:47.140200+0000","last_active":"2024-01-16T15:28:47.140086+0000","last_peered":"2024-01-16T15:28:47.140086+0000","last_clean":"2024-01-16T15:28:47.140086+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:28:47.140086+0000","last_undegraded":"2024-01-16T15:28:47.140086+0000","last_fullsized":"2024-01-16T15:28:47.140086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:47.139873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:47.139873+0000","last_clean_scrub_stamp":"2024-01-16T15:28:47.139873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:12.869817+0000","scrub_duration":0.004451694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'66","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.564786+0000","last_change":"2024-01-16T15:28:20.564862+0000","last_active":"2024-01-16T15:28:20.564786+0000","last_peered":"2024-01-16T15:28:20.564786+0000","last_clean":"2024-01-16T15:28:20.564786+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:28:20.564786+0000","last_undegraded":"2024-01-16T15:28:20.564786+0000","last_fullsized":"2024-01-16T15:28:20.564786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'66","last_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_deep_scrub":"112'66","last_deep_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.564586+0000","objects_scrubbed":13,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:24.471110+0000","scrub_duration":0.12569713499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.884673+0000","last_change":"2024-01-16T15:28:10.884720+0000","last_active":"2024-01-16T15:28:10.884673+0000","last_peered":"2024-01-16T15:28:10.884673+0000","last_clean":"2024-01-16T15:28:10.884673+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:28:10.884673+0000","last_undegraded":"2024-01-16T15:28:10.884673+0000","last_fullsized":"2024-01-16T15:28:10.884673+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.884468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:06.507427+0000","scrub_duration":0.0032287930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.481473+0000","last_change":"2024-01-16T15:28:17.481559+0000","last_active":"2024-01-16T15:28:17.481473+0000","last_peered":"2024-01-16T15:28:17.481473+0000","last_clean":"2024-01-16T15:28:17.481473+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:28:17.481473+0000","last_undegraded":"2024-01-16T15:28:17.481473+0000","last_fullsized":"2024-01-16T15:28:17.481473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.481296+0000","objects_scrubbed":17,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:01.224547+0000","scrub_duration":0.21059523299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.256061+0000","last_change":"2024-01-16T15:28:18.256178+0000","last_active":"2024-01-16T15:28:18.256061+0000","last_peered":"2024-01-16T15:28:18.256061+0000","last_clean":"2024-01-16T15:28:18.256061+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:28:18.256061+0000","last_undegraded":"2024-01-16T15:28:18.256061+0000","last_fullsized":"2024-01-16T15:28:18.256061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.255843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:27.175270+0000","scrub_duration":0.0036640700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'85","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:37.411650+0000","last_change":"2024-01-16T15:28:37.411773+0000","last_active":"2024-01-16T15:28:37.411650+0000","last_peered":"2024-01-16T15:28:37.411650+0000","last_clean":"2024-01-16T15:28:37.411650+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:28:37.411650+0000","last_undegraded":"2024-01-16T15:28:37.411650+0000","last_fullsized":"2024-01-16T15:28:37.411650+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:37.411457+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:37.411457+0000","last_clean_scrub_stamp":"2024-01-16T15:28:37.411457+0000","objects_scrubbed":21,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:09.422228+0000","scrub_duration":0.22994655999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.221495+0000","last_change":"2024-01-16T15:28:10.221583+0000","last_active":"2024-01-16T15:28:10.221495+0000","last_peered":"2024-01-16T15:28:10.221495+0000","last_clean":"2024-01-16T15:28:10.221495+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:28:10.221495+0000","last_undegraded":"2024-01-16T15:28:10.221495+0000","last_fullsized":"2024-01-16T15:28:10.221495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.221197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:27:58.612687+0000","scrub_duration":0.0041312190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'94","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:35.980220+0000","last_change":"2024-01-16T15:28:35.980283+0000","last_active":"2024-01-16T15:28:35.980220+0000","last_peered":"2024-01-16T15:28:35.980220+0000","last_clean":"2024-01-16T15:28:35.980220+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:28:35.980220+0000","last_undegraded":"2024-01-16T15:28:35.980220+0000","last_fullsized":"2024-01-16T15:28:35.980220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:35.980111+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:35.980111+0000","last_clean_scrub_stamp":"2024-01-16T15:28:35.980111+0000","objects_scrubbed":21,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:55:32.046445+0000","scrub_duration":0.287987259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'32","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.827087+0000","last_change":"2024-01-16T15:28:14.827144+0000","last_active":"2024-01-16T15:28:14.827087+0000","last_peered":"2024-01-16T15:28:14.827087+0000","last_clean":"2024-01-16T15:28:14.827087+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:28:14.827087+0000","last_undegraded":"2024-01-16T15:28:14.827087+0000","last_fullsized":"2024-01-16T15:28:14.827087+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'32","last_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_deep_scrub":"112'32","last_deep_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.826899+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:16:55.047969+0000","scrub_duration":0.0075779699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.118029+0000","last_change":"2024-01-16T15:28:25.118081+0000","last_active":"2024-01-16T15:28:25.118029+0000","last_peered":"2024-01-16T15:28:25.118029+0000","last_clean":"2024-01-16T15:28:25.118029+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:28:25.118029+0000","last_undegraded":"2024-01-16T15:28:25.118029+0000","last_fullsized":"2024-01-16T15:28:25.118029+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.117955+0000","objects_scrubbed":23,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:56.787030+0000","scrub_duration":0.17487287100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:37.058158+0000","last_change":"2024-01-16T15:28:37.058260+0000","last_active":"2024-01-16T15:28:37.058158+0000","last_peered":"2024-01-16T15:28:37.058158+0000","last_clean":"2024-01-16T15:28:37.058158+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:28:37.058158+0000","last_undegraded":"2024-01-16T15:28:37.058158+0000","last_fullsized":"2024-01-16T15:28:37.058158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:37.057940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:37.057940+0000","last_clean_scrub_stamp":"2024-01-16T15:28:37.057940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:28.720608+0000","scrub_duration":0.0036757579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'68","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.884824+0000","last_change":"2024-01-16T15:28:19.884824+0000","last_active":"2024-01-16T15:28:19.884824+0000","last_peered":"2024-01-16T15:28:19.884824+0000","last_clean":"2024-01-16T15:28:19.884824+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:28:19.884824+0000","last_undegraded":"2024-01-16T15:28:19.884824+0000","last_fullsized":"2024-01-16T15:28:19.884824+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'68","last_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_deep_scrub":"112'68","last_deep_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.884357+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:10.532387+0000","scrub_duration":0.15341920100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.314517+0000","last_change":"2024-01-16T15:28:40.314517+0000","last_active":"2024-01-16T15:28:40.314517+0000","last_peered":"2024-01-16T15:28:40.314517+0000","last_clean":"2024-01-16T15:28:40.314517+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:28:40.314517+0000","last_undegraded":"2024-01-16T15:28:40.314517+0000","last_fullsized":"2024-01-16T15:28:40.314517+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:40.314218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:40.314218+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.314218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:28.132750+0000","scrub_duration":0.0055196840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"112'119","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.585009+0000","last_change":"2024-01-16T15:28:18.585089+0000","last_active":"2024-01-16T15:28:18.585009+0000","last_peered":"2024-01-16T15:28:18.585009+0000","last_clean":"2024-01-16T15:28:18.585009+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:28:18.585009+0000","last_undegraded":"2024-01-16T15:28:18.585009+0000","last_fullsized":"2024-01-16T15:28:18.585009+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'119","last_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_deep_scrub":"112'119","last_deep_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.584815+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:01.708690+0000","scrub_duration":0.06564681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5454,"ondisk_log_size":5454,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":453,"ondisk_log_size":453,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509790,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1439999999999999,"15min":1.2210000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.593,"5min":1.7749999999999999,"15min":19.224},"last":1.2929999999999999},{"interface":"front","average":{"1min":1.238,"5min":1.2250000000000001,"15min":1.2070000000000001},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.6140000000000001,"5min":1.694,"15min":5.085},"last":1.004}]},{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.194,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.374},"max":{"1min":1.641,"5min":1.837,"15min":3.7069999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.204,"5min":1.2909999999999999,"15min":1.3640000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.536,"5min":1.8819999999999999,"15min":19.178000000000001},"last":1.7829999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3080000000000001,"15min":1.383},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":19.126000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.3200000000000001,"15min":1.3160000000000001},"min":{"1min":1.0669999999999999,"5min":0.88700000000000001,"15min":0.66700000000000004},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":2.9750000000000001},"last":1.2210000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2689999999999999,"15min":1.3200000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.76800000000000002},"max":{"1min":1.6599999999999999,"5min":1.73,"15min":3.7999999999999998},"last":1.71},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4390000000000001,"15min":1.5269999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.63500000000000001},"max":{"1min":1.972,"5min":1.972,"15min":19.268999999999998},"last":1.6639999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2470000000000001,"15min":1.264},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.58799999999999997},"max":{"1min":1.794,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.363},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.369,"15min":1.397},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.58299999999999996},"max":{"1min":2.452,"5min":2.452,"15min":9.7059999999999995},"last":1.524}]},{"osd":5,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5329999999999999,"15min":1.548},"min":{"1min":0.99099999999999999,"5min":0.97599999999999998,"15min":0.79400000000000004},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":3.9129999999999998},"last":1.417},{"interface":"front","average":{"1min":1.514,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.86799999999999999,"5min":0.82699999999999996,"15min":0.55100000000000005},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":19.036999999999999},"last":1.4610000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.456,"15min":1.444},"min":{"1min":1.0069999999999999,"5min":0.88500000000000001,"15min":0.54800000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":4.2240000000000002},"last":1.127},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5409999999999999,"15min":1.5720000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.65100000000000002},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":4.0709999999999997},"last":1.599}]}]},{"osd":4,"up_from":15,"seq":64424509792,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.296,"15min":1.2330000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.427,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.755},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1799999999999999,"15min":1.1859999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.65600000000000003},"max":{"1min":1.5069999999999999,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.0960000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.2170000000000001,"5min":1.2509999999999999,"15min":1.222},"min":{"1min":0.88700000000000001,"5min":0.73899999999999999,"15min":0.71899999999999997},"max":{"1min":1.6160000000000001,"5min":1.893,"15min":2.0510000000000002},"last":1.2230000000000001},{"interface":"front","average":{"1min":1.1559999999999999,"5min":1.3149999999999999,"15min":1.302},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.6319999999999999,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.88300000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.3420000000000001,"15min":1.3320000000000001},"min":{"1min":0.79300000000000004,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.573,"5min":1.78,"15min":5.907},"last":1.464},{"interface":"front","average":{"1min":1.242,"5min":1.381,"15min":1.355},"min":{"1min":0.86799999999999999,"5min":0.85899999999999999,"15min":0.52600000000000002},"max":{"1min":1.6000000000000001,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":1.161}]},{"osd":3,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.4570000000000001,"15min":1.3939999999999999},"min":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.76200000000000001},"max":{"1min":1.5289999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.4019999999999999,"15min":1.391},"min":{"1min":1.052,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.74,"5min":1.9670000000000001,"15min":2.0379999999999998},"last":1.0620000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.333,"5min":1.425,"15min":1.448},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6779999999999999,"5min":6.165,"15min":6.165},"last":1.411},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4410000000000001,"15min":1.4770000000000001},"min":{"1min":0.95299999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8180000000000001,"5min":3.1469999999999998,"15min":5.3520000000000003},"last":1.661}]},{"osd":6,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.482,"5min":1.391,"15min":1.4219999999999999},"min":{"1min":1.101,"5min":0.90800000000000003,"15min":0.497},"max":{"1min":1.903,"5min":1.903,"15min":8.9600000000000009},"last":1.335},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.5640000000000001,"15min":1.528},"min":{"1min":1.165,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.772,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.601}]},{"osd":7,"last update":"Tue Jan 16 15:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.427,"5min":1.452,"15min":1.492},"min":{"1min":1.0469999999999999,"5min":1.0449999999999999,"15min":0.89700000000000002},"max":{"1min":1.8500000000000001,"5min":3.3620000000000001,"15min":5.835},"last":1.284},{"interface":"front","average":{"1min":1.554,"5min":1.5469999999999999,"15min":1.581},"min":{"1min":1.238,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":2.0070000000000001,"5min":2.3679999999999999,"15min":8.0079999999999991},"last":1.5609999999999999}]}]},{"osd":7,"up_from":14,"seq":60129542494,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1896108,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92475732,"statfs":{"total":96636764160,"available":94695149568,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0419999999999998,"5min":1.339,"15min":1.2649999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.65400000000000003},"max":{"1min":16.832999999999998,"5min":16.832999999999998,"15min":16.832999999999998},"last":1.1259999999999999},{"interface":"front","average":{"1min":2.4359999999999999,"5min":1.456,"15min":1.304},"min":{"1min":0.86199999999999999,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":21.283000000000001,"5min":21.283000000000001,"15min":21.283000000000001},"last":0.89600000000000002}]},{"osd":1,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.254,"15min":1.272},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.61799999999999999},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.2090000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":1.47,"15min":1.395},"min":{"1min":0.90700000000000003,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":16.350000000000001,"5min":16.350000000000001,"15min":16.350000000000001},"last":1.24}]},{"osd":2,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.5049999999999999,"15min":1.3700000000000001},"min":{"1min":0.82599999999999996,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":16.960999999999999,"5min":16.960999999999999,"15min":16.960999999999999},"last":1.51},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3280000000000001,"15min":1.409},"min":{"1min":0.92700000000000005,"5min":0.873,"15min":0.69299999999999995},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":19.763999999999999},"last":1.302}]},{"osd":3,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":1.6339999999999999,"15min":1.4450000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.80600000000000005},"max":{"1min":27.920999999999999,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.3839999999999999},{"interface":"front","average":{"1min":2.4430000000000001,"5min":1.661,"15min":1.538},"min":{"1min":1.04,"5min":1.0169999999999999,"15min":0.76500000000000001},"max":{"1min":17.709,"5min":17.709,"15min":17.709},"last":1.4410000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.345,"15min":1.347},"min":{"1min":0.98499999999999999,"5min":0.89100000000000001,"15min":0.54100000000000004},"max":{"1min":1.7090000000000001,"5min":1.7929999999999999,"15min":4.1390000000000002},"last":1.02},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.401,"15min":1.413},"min":{"1min":1.0009999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":1.873,"5min":1.877,"15min":2.5190000000000001},"last":1.0600000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.425,"15min":1.361},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":7.641,"5min":7.641,"15min":7.641},"last":1.3400000000000001},{"interface":"front","average":{"1min":2.4780000000000002,"5min":1.718,"15min":1.585},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":0.66000000000000003},"max":{"1min":16.911999999999999,"5min":16.911999999999999,"15min":16.911999999999999},"last":1.266}]},{"osd":6,"last update":"Tue Jan 16 15:27:45 2024","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.593,"15min":1.5680000000000001},"min":{"1min":1.099,"5min":1.0760000000000001,"15min":0.66000000000000003},"max":{"1min":2.0859999999999999,"5min":2.1429999999999998,"15min":2.653},"last":1.556},{"interface":"front","average":{"1min":1.528,"5min":1.522,"15min":1.552},"min":{"1min":1.004,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":12.567},"last":1.169}]}]},{"osd":5,"up_from":14,"seq":60129542496,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.1830000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.32900000000000001},"max":{"1min":1.7070000000000001,"5min":2.0939999999999999,"15min":6.0720000000000001},"last":1.1399999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.1759999999999999,"15min":1.3160000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.57099999999999995},"max":{"1min":1.302,"5min":1.7649999999999999,"15min":23.199999999999999},"last":1.341}]},{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.21,"15min":1.224},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.63400000000000001},"max":{"1min":1.736,"5min":1.736,"15min":4.2450000000000001},"last":1.198},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2649999999999999,"15min":1.327},"min":{"1min":0.76600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.786,"5min":1.851,"15min":17.062000000000001},"last":1.282}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.367,"15min":1.403},"min":{"1min":0.65300000000000002,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":5.8810000000000002},"last":1.583},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.96599999999999997,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.921,"5min":1.921,"15min":5.4429999999999996},"last":1.244}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.323,"15min":1.3240000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.67500000000000004},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.4769999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.363,"5min":1.397,"15min":1.4179999999999999},"min":{"1min":0.69099999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.742,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":1.393}]},{"osd":4,"last update":"Tue Jan 16 15:28:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.363,"15min":1.3939999999999999},"min":{"1min":0.88300000000000001,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":2.2610000000000001,"15min":15.975},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.417,"15min":1.4179999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.54000000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":5.6349999999999998},"last":1.482}]},{"osd":6,"last update":"Tue Jan 16 15:28:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.335,"15min":1.369},"min":{"1min":0.96999999999999997,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":12.914},"last":1.01},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4950000000000001,"15min":1.5209999999999999},"min":{"1min":1.0780000000000001,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":1.9990000000000001,"5min":2.4660000000000002,"15min":16.552},"last":1.5369999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.4750000000000001,"15min":1.51},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.64300000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":13.034000000000001},"last":1.177},{"interface":"front","average":{"1min":1.669,"5min":1.5900000000000001,"15min":1.5760000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.70899999999999996},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":1.7050000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542496,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.97199999999999998,"15min":0.97599999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.315},"max":{"1min":1.6910000000000001,"5min":1.6910000000000001,"15min":5.7400000000000002},"last":1.175},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.1259999999999999,"15min":1.1359999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.40600000000000003},"max":{"1min":1.506,"5min":1.879,"15min":7.1429999999999998},"last":0.98999999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.96899999999999997,"15min":1.0349999999999999},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.365,"5min":1.738,"15min":10.856999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.075,"5min":1.198,"15min":1.1870000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.4950000000000001,"5min":1.823,"15min":2.0089999999999999},"last":1.3640000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.1779999999999999,"5min":1.3140000000000001,"15min":1.5209999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.946,"5min":1.946,"15min":39.265999999999998},"last":1.641},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.1559999999999999,"15min":1.2070000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.381,"5min":1.833,"15min":7.9160000000000004},"last":0.88}]},{"osd":4,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.337,"15min":1.3680000000000001},"min":{"1min":0.748,"5min":0.748,"15min":0.70999999999999996},"max":{"1min":1.5980000000000001,"5min":1.9419999999999999,"15min":5.6890000000000001},"last":1.411},{"interface":"front","average":{"1min":1.329,"5min":1.464,"15min":1.4810000000000001},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8120000000000001,"5min":2.0920000000000001,"15min":2.3330000000000002},"last":1.524}]},{"osd":5,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.3440000000000001,"15min":1.4670000000000001},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.5669999999999999,"5min":1.827,"15min":14.003},"last":1.097},{"interface":"front","average":{"1min":1.357,"5min":1.5049999999999999,"15min":1.577},"min":{"1min":0.877,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":1.9359999999999999,"5min":2.008,"15min":6.0679999999999996},"last":1.756}]},{"osd":6,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.464,"15min":1.502},"min":{"1min":0.875,"5min":0.875,"15min":0.69799999999999995},"max":{"1min":1.8899999999999999,"5min":2.0139999999999998,"15min":7.0060000000000002},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.486,"15min":1.5580000000000001},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.8140000000000001,"5min":1.982,"15min":15.114000000000001},"last":1.3300000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:27:51 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4950000000000001,"15min":1.6160000000000001},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.80400000000000005},"max":{"1min":2.1349999999999998,"5min":2.1349999999999998,"15min":12.378},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.5780000000000001,"15min":1.6220000000000001},"min":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":1.9650000000000001,"5min":2.23,"15min":4.6029999999999998},"last":1.4690000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542495,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1064088,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93307752,"statfs":{"total":96636764160,"available":95547138048,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91300000000000003,"15min":0.97699999999999998},"min":{"1min":0.626,"5min":0.46200000000000002,"15min":0.32400000000000001},"max":{"1min":1.3720000000000001,"5min":1.4770000000000001,"15min":17.77},"last":1.431},{"interface":"front","average":{"1min":1.034,"5min":1.139,"15min":1.121},"min":{"1min":0.63300000000000001,"5min":0.49199999999999999,"15min":0.40300000000000002},"max":{"1min":1.708,"5min":1.913,"15min":7.9960000000000004},"last":1.3720000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1120000000000001,"15min":1.1020000000000001},"min":{"1min":0.66800000000000004,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":2.145,"5min":2.294,"15min":2.294},"last":1.617},{"interface":"front","average":{"1min":1.2,"5min":1.1799999999999999,"15min":1.1819999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48399999999999999},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":4.3819999999999997},"last":0.97999999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.171,"15min":1.2789999999999999},"min":{"1min":0.63700000000000001,"5min":0.61199999999999999,"15min":0.44700000000000001},"max":{"1min":1.804,"5min":1.883,"15min":18.838000000000001},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4830000000000001,"15min":1.554},"min":{"1min":0.79300000000000004,"5min":0.66200000000000003,"15min":0.58099999999999996},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":19.375},"last":1.577}]},{"osd":4,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.327,"15min":1.429},"min":{"1min":0.80800000000000005,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.754,"5min":2.242,"15min":23.225000000000001},"last":1.738},{"interface":"front","average":{"1min":1.365,"5min":1.466,"15min":1.484},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.72699999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":6.359},"last":1.4890000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4390000000000001,"15min":1.494},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.68100000000000005},"max":{"1min":2.165,"5min":2.3170000000000002,"15min":20.157},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.5589999999999999,"15min":1.593},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.522}]},{"osd":6,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5069999999999999,"15min":1.5900000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.83599999999999997},"max":{"1min":2.0579999999999998,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.7709999999999999},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.595,"15min":1.7130000000000001},"min":{"1min":0.99199999999999999,"5min":0.85799999999999998,"15min":0.76200000000000001},"max":{"1min":2.238,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.827}]},{"osd":7,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5329999999999999,"15min":1.6579999999999999},"min":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":1.9930000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":2.3889999999999998},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.583,"15min":1.587},"min":{"1min":1.1259999999999999,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":4.3419999999999996},"last":1.663}]}]},{"osd":1,"up_from":14,"seq":60129542497,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0660000000000001,"15min":0.98699999999999999},"min":{"1min":0.71599999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.296,"5min":13.42,"15min":13.42},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1359999999999999,"15min":1.123},"min":{"1min":0.50900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.6339999999999999,"5min":3.915,"15min":3.915},"last":1.141}]},{"osd":2,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0269999999999999,"15min":1.0900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.53,"5min":1.53,"15min":3.6819999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":1.284,"5min":1.2729999999999999,"15min":1.335},"min":{"1min":0.64100000000000001,"5min":0.46999999999999997,"15min":0.40100000000000002},"max":{"1min":1.736,"5min":1.736,"15min":3.5600000000000001},"last":1.5780000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6950000000000001,"5min":1.875,"15min":2.238},"last":1.6930000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.127,"15min":1.1919999999999999},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.7509999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":0.83999999999999997}]},{"osd":4,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2549999999999999,"15min":1.327},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6739999999999999,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.417,"5min":1.544,"15min":1.528},"min":{"1min":1.077,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.679,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.637}]},{"osd":5,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.419,"15min":1.4450000000000001},"min":{"1min":1.095,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.98},"last":1.732},{"interface":"front","average":{"1min":1.327,"5min":1.417,"15min":1.4530000000000001},"min":{"1min":0.97199999999999998,"5min":0.92900000000000005,"15min":0.84499999999999997},"max":{"1min":1.78,"5min":1.893,"15min":3.3940000000000001},"last":1.212}]},{"osd":6,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.5089999999999999,"15min":1.583},"min":{"1min":1.2549999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":16.48},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4930000000000001,"15min":1.536},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.90000000000000002},"max":{"1min":1.978,"5min":1.978,"15min":3.8180000000000001},"last":1.0900000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.5389999999999999,"15min":1.613},"min":{"1min":0.94199999999999995,"5min":0.92200000000000004,"15min":0.68899999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":7.0499999999999998},"last":1.345},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.607,"15min":1.6499999999999999},"min":{"1min":1.0920000000000001,"5min":0.91700000000000004,"15min":0.89900000000000002},"max":{"1min":1.931,"5min":2.258,"15min":3.7839999999999998},"last":1.9419999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542495,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.93500000000000005,"15min":1.0449999999999999},"min":{"1min":0.69199999999999995,"5min":0.496,"15min":0.376},"max":{"1min":1.5509999999999999,"5min":1.665,"15min":15.776},"last":0.875},{"interface":"front","average":{"1min":1.131,"5min":1.1559999999999999,"15min":1.1739999999999999},"min":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":17.620999999999999},"last":0.80800000000000005}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.135,"15min":1.226},"min":{"1min":0.82899999999999996,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.629,"5min":1.671,"15min":15.997999999999999},"last":1.357},{"interface":"front","average":{"1min":1.246,"5min":1.2430000000000001,"15min":1.2030000000000001},"min":{"1min":0.89200000000000002,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.3049999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1080000000000001,"15min":1.179},"min":{"1min":0.77900000000000003,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.7190000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.345,"15min":1.4339999999999999},"min":{"1min":0.86599999999999999,"5min":0.66600000000000004,"15min":0.51200000000000001},"max":{"1min":1.6459999999999999,"5min":1.827,"15min":16.126000000000001},"last":1.512}]},{"osd":4,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4750000000000001,"15min":1.5740000000000001},"min":{"1min":1.0389999999999999,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.8640000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.46,"15min":1.4279999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":3.5219999999999998,"5min":3.5219999999999998,"15min":3.5219999999999998},"last":1.4690000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.736,"5min":2.157,"15min":5.5609999999999999},"last":1.337},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.5609999999999999,"15min":1.649},"min":{"1min":1.157,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":16.501999999999999},"last":1.629}]},{"osd":6,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4390000000000001,"15min":1.4510000000000001},"min":{"1min":1.0569999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.7370000000000001,"5min":2.0499999999999998,"15min":5.4509999999999996},"last":1.266},{"interface":"front","average":{"1min":1.601,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":1.2470000000000001,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":1.8979999999999999,"5min":2.351,"15min":16.395},"last":1.724}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.591,"15min":1.5780000000000001},"min":{"1min":1.22,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":3.423,"5min":3.423,"15min":3.8239999999999998},"last":1.571},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.579,"15min":1.5860000000000001},"min":{"1min":1.1419999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.0600000000000001,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.4339999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:28:52.231 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.232 INFO:tasks.ceph:pgid 17.3c last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.232 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.232 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.232 INFO:tasks.ceph:pgid 17.34 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.233 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.233 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.233 INFO:tasks.ceph:pgid 17.16 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.234 INFO:tasks.ceph:pgid 16.f last_scrub_stamp 2024-01-16T15:19:22.561508+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=22, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.234 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.234 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:28:52.235 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:29:00.004 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:00.012 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:12.236 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:29:12.457 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.457+0000 7f24a5000640 1 Processor -- start 2024-01-16T15:29:12.458 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.457+0000 7f24a5000640 1 -- start start 2024-01-16T15:29:12.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.458+0000 7f24a5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 0x7f24a0166030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.458+0000 7f24a5000640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a015c5f0 con 0x7f24a013e360 2024-01-16T15:29:12.459 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f24a5000640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a006c470 con 0x7f24a013df80 2024-01-16T15:29:12.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f24a5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a006c750 con 0x7f24a00cf9a0 2024-01-16T15:29:12.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f2495200640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f24a013e360 0x7f24a00cf290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:39950/0 (socket says 172.21.15.53:39950) 2024-01-16T15:29:12.460 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f2495200640 1 -- 172.21.15.53:0/4261724264 learned_addr learned my addr 172.21.15.53:0/4261724264 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:29:12.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f249e800640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 0x7f24a0166030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289982485 0 0) 0x7f24a015c5f0 con 0x7f24a013e360 2024-01-16T15:29:12.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.459+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2480003520 con 0x7f24a013e360 2024-01-16T15:29:12.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1919525896 0 0) 0x7f2480003520 con 0x7f24a013e360 2024-01-16T15:29:12.462 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 msgr2=0x7f24a0166030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 0x7f24a0166030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 >> v1:172.21.15.146:6789/0 conn(0x7f24a013df80 legacy=0x7f24a015bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a006ca30 con 0x7f24a013e360 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f2488001670 con 0x7f24a013e360 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 >> v1:172.21.15.53:6789/0 conn(0x7f24a013e360 legacy=0x7f24a00cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24800039a0 0x7f2480003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 0x7f24800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2480004d00 0x7f2480005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.463 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a006c750 con 0x7f24800039a0 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f248000ba90 con 0x7f2480004d00 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f248000bc50 con 0x7f24a00cf9a0 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.460+0000 7f249de00640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24800039a0 0x7f2480003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f2495200640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 0x7f24800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f249e800640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2480004d00 0x7f2480005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f249de00640 1 -- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 msgr2=0x7f24800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f249de00640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00cf9a0 0x7f24800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.464 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f249de00640 1 -- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2480004d00 msgr2=0x7f2480005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:12.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f249de00640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f2480004d00 0x7f2480005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.465 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.461+0000 7f249de00640 1 -- 172.21.15.53:0/4261724264 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2480003520 con 0x7f24800039a0 2024-01-16T15:29:12.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.462+0000 7f249de00640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24800039a0 0x7f2480003dc0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2490001e20 tx=0x7f24800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ada8fca92c26432c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:12.466 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.462+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490003b70 con 0x7f24800039a0 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2490003cd0 con 0x7f24800039a0 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f249d400640 1 -- 172.21.15.53:0/4261724264 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2490005310 con 0x7f24800039a0 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f24a5000640 1 -- 172.21.15.53:0/4261724264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24800039a0 msgr2=0x7f2480003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f24a5000640 1 --2- 172.21.15.53:0/4261724264 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24800039a0 0x7f2480003dc0 secure :-1 s=READY pgs=2072 cs=0 l=1 rev1=1 crypto rx=0x7f2490001e20 tx=0x7f24800057f0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f249e800640 1 -- 172.21.15.53:0/4261724264 reap_dead start 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f24a5000640 1 -- 172.21.15.53:0/4261724264 shutdown_connections 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f24a5000640 1 -- 172.21.15.53:0/4261724264 >> 172.21.15.53:0/4261724264 conn(0x7f24a006a800 msgr2=0x7f24a006ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f24a5000640 1 -- 172.21.15.53:0/4261724264 shutdown_connections 2024-01-16T15:29:12.467 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.463+0000 7f24a5000640 1 -- 172.21.15.53:0/4261724264 wait complete. 2024-01-16T15:29:12.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.465+0000 7f24a5000640 1 Processor -- start 2024-01-16T15:29:12.468 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.465+0000 7f24a5000640 1 -- start start 2024-01-16T15:29:12.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.466+0000 7f24a5000640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24800039a0 0x7f24a013cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.466+0000 7f24a5000640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00afc90 0x7f24a00b0110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.466+0000 7f24a5000640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 0x7f24a01491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.466+0000 7f24a5000640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a014a710 con 0x7f24a0148d60 2024-01-16T15:29:12.470 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.466+0000 7f24a5000640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a00b0650 con 0x7f24800039a0 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.466+0000 7f24a5000640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24a013d040 con 0x7f24a00afc90 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 0x7f24a01491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 0x7f24a01491e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:44416/0 (socket says 172.21.15.53:44416) 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 -- 172.21.15.53:0/959509628 learned_addr learned my addr 172.21.15.53:0/959509628 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249de00640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24800039a0 0x7f24a013cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f2495200640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00afc90 0x7f24a00b0110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 -- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00afc90 msgr2=0x7f24a00b0110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.471 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00afc90 0x7f24a00b0110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 -- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24800039a0 msgr2=0x7f24a013cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24800039a0 0x7f24a013cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 -- 172.21.15.53:0/959509628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a0149a90 con 0x7f24a0148d60 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f2495200640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00afc90 0x7f24a00b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.467+0000 7f249e800640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 0x7f24a01491e0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cfea0 tx=0x7f2498011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.468+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2498004440 con 0x7f24a0148d60 2024-01-16T15:29:12.472 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.468+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2498003610 con 0x7f24a0148d60 2024-01-16T15:29:12.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.468+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24980039c0 con 0x7f24a0148d60 2024-01-16T15:29:12.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.468+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a0149d70 con 0x7f24a0148d60 2024-01-16T15:29:12.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.468+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a014a330 con 0x7f24a0148d60 2024-01-16T15:29:12.473 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.469+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a014a850 con 0x7f24a0148d60 2024-01-16T15:29:12.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.470+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f2498003b20 con 0x7f24a0148d60 2024-01-16T15:29:12.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.471+0000 7f2496a00640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f247806c4c0 0x7f247806e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:12.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.471+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f2498088b10 con 0x7f24a0148d60 2024-01-16T15:29:12.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.472+0000 7f249de00640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f247806c4c0 0x7f247806e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:12.474 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.473+0000 7f249de00640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f247806c4c0 0x7f247806e980 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2490034000 tx=0x7f2490009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:12.476 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.476+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f24a014a850 con 0x7f24a0148d60 2024-01-16T15:29:12.675 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.674+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24a006cff0 con 0x7f247806c4c0 2024-01-16T15:29:12.698 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.698+0000 7f2496a00640 1 -- 172.21.15.53:0/959509628 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291028 (secure 0 0 0) 0x7f249003f040 con 0x7f247806c4c0 2024-01-16T15:29:12.702 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:29:12.704 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.704+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f247806c4c0 msgr2=0x7f247806e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.704+0000 7f24a5000640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f247806c4c0 0x7f247806e980 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f2490034000 tx=0x7f2490009030 comp rx=0 tx=0).stop 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.704+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 msgr2=0x7f24a01491e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.704+0000 7f24a5000640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 0x7f24a01491e0 secure :-1 s=READY pgs=2073 cs=0 l=1 rev1=1 crypto rx=0x7f24a00cfea0 tx=0x7f2498011920 comp rx=0 tx=0).stop 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 shutdown_connections 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f247806c4c0 0x7f247806e980 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f24a0148d60 0x7f24a01491e0 unknown :-1 s=CLOSED pgs=2073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f24a00afc90 0x7f24a00b0110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 --2- 172.21.15.53:0/959509628 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f24800039a0 0x7f24a013cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 >> 172.21.15.53:0/959509628 conn(0x7f24a006a800 msgr2=0x7f24a00ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 shutdown_connections 2024-01-16T15:29:12.707 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:12.705+0000 7f24a5000640 1 -- 172.21.15.53:0/959509628 wait complete. 2024-01-16T15:29:12.725 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":988,"stamp":"2024-01-16T15:29:10.935970+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370502932,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1978,"num_write_kb":5246745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6099,"ondisk_log_size":6099,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11133624,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":633331,"kb_avail":743841096,"statfs":{"total":773094113280,"available":761693282304,"internally_reserved":0,"allocated":10752221184,"data_stored":10745813792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":648531504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004539"},"pg_stats":[{"pgid":"17.3f","version":"112'71","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:27:49.257209+0000","last_change":"2024-01-16T15:19:24.628397+0000","last_active":"2024-01-16T15:27:49.257209+0000","last_peered":"2024-01-16T15:27:49.257209+0000","last_clean":"2024-01-16T15:27:49.257209+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:27:49.257209+0000","last_undegraded":"2024-01-16T15:27:49.257209+0000","last_fullsized":"2024-01-16T15:27:49.257209+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'97","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.510978+0000","last_change":"2024-01-16T15:28:18.511076+0000","last_active":"2024-01-16T15:28:18.510978+0000","last_peered":"2024-01-16T15:28:18.510978+0000","last_clean":"2024-01-16T15:28:18.510978+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:28:18.510978+0000","last_undegraded":"2024-01-16T15:28:18.510978+0000","last_fullsized":"2024-01-16T15:28:18.510978+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'97","last_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_deep_scrub":"112'97","last_deep_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.510802+0000","objects_scrubbed":24,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:33.903169+0000","scrub_duration":0.27194245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'87","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:23.860238+0000","last_change":"2024-01-16T15:28:23.860329+0000","last_active":"2024-01-16T15:28:23.860238+0000","last_peered":"2024-01-16T15:28:23.860238+0000","last_clean":"2024-01-16T15:28:23.860238+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:28:23.860238+0000","last_undegraded":"2024-01-16T15:28:23.860238+0000","last_fullsized":"2024-01-16T15:28:23.860238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_clean_scrub_stamp":"2024-01-16T15:28:23.860048+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:49.716426+0000","scrub_duration":0.27335353899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'71","reported_seq":103,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:10.256439+0000","last_change":"2024-01-16T15:29:10.256523+0000","last_active":"2024-01-16T15:29:10.256439+0000","last_peered":"2024-01-16T15:29:10.256439+0000","last_clean":"2024-01-16T15:29:10.256439+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:29:10.256439+0000","last_undegraded":"2024-01-16T15:29:10.256439+0000","last_fullsized":"2024-01-16T15:29:10.256439+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'71","last_scrub_stamp":"2024-01-16T15:29:10.256271+0000","last_deep_scrub":"112'71","last_deep_scrub_stamp":"2024-01-16T15:29:10.256271+0000","last_clean_scrub_stamp":"2024-01-16T15:29:10.256271+0000","objects_scrubbed":15,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:51.152173+0000","scrub_duration":0.17038672199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'78","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.070128+0000","last_change":"2024-01-16T15:28:25.070176+0000","last_active":"2024-01-16T15:28:25.070128+0000","last_peered":"2024-01-16T15:28:25.070128+0000","last_clean":"2024-01-16T15:28:25.070128+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:28:25.070128+0000","last_undegraded":"2024-01-16T15:28:25.070128+0000","last_fullsized":"2024-01-16T15:28:25.070128+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'78","last_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_deep_scrub":"112'78","last_deep_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.070058+0000","objects_scrubbed":16,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:36.495682+0000","scrub_duration":0.150879281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'99","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.440173+0000","last_change":"2024-01-16T15:28:27.440281+0000","last_active":"2024-01-16T15:28:27.440173+0000","last_peered":"2024-01-16T15:28:27.440173+0000","last_clean":"2024-01-16T15:28:27.440173+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:28:27.440173+0000","last_undegraded":"2024-01-16T15:28:27.440173+0000","last_fullsized":"2024-01-16T15:28:27.440173+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.439995+0000","objects_scrubbed":23,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:38.865619+0000","scrub_duration":0.25882440899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'88","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:43.589370+0000","last_change":"2024-01-16T15:28:43.589452+0000","last_active":"2024-01-16T15:28:43.589370+0000","last_peered":"2024-01-16T15:28:43.589370+0000","last_clean":"2024-01-16T15:28:43.589370+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:28:43.589370+0000","last_undegraded":"2024-01-16T15:28:43.589370+0000","last_fullsized":"2024-01-16T15:28:43.589370+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:43.589215+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:43.589215+0000","last_clean_scrub_stamp":"2024-01-16T15:28:43.589215+0000","objects_scrubbed":21,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:23.823898+0000","scrub_duration":0.25936774699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'79","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.799942+0000","last_change":"2024-01-16T15:28:24.800039+0000","last_active":"2024-01-16T15:28:24.799942+0000","last_peered":"2024-01-16T15:28:24.799942+0000","last_clean":"2024-01-16T15:28:24.799942+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:28:24.799942+0000","last_undegraded":"2024-01-16T15:28:24.799942+0000","last_fullsized":"2024-01-16T15:28:24.799942+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.799761+0000","objects_scrubbed":21,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:58.738844+0000","scrub_duration":0.232299336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'70","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566737+0000","last_change":"2024-01-16T15:19:24.637847+0000","last_active":"2024-01-16T15:28:05.566737+0000","last_peered":"2024-01-16T15:28:05.566737+0000","last_clean":"2024-01-16T15:28:05.566737+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:28:05.566737+0000","last_undegraded":"2024-01-16T15:28:05.566737+0000","last_fullsized":"2024-01-16T15:28:05.566737+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'70","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566710+0000","last_change":"2024-01-16T15:19:24.627702+0000","last_active":"2024-01-16T15:28:05.566710+0000","last_peered":"2024-01-16T15:28:05.566710+0000","last_clean":"2024-01-16T15:28:05.566710+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:28:05.566710+0000","last_undegraded":"2024-01-16T15:28:05.566710+0000","last_fullsized":"2024-01-16T15:28:05.566710+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":70,"log_dups_size":0,"ondisk_log_size":70,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'93","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.541954+0000","last_change":"2024-01-16T15:28:26.541954+0000","last_active":"2024-01-16T15:28:26.541954+0000","last_peered":"2024-01-16T15:28:26.541954+0000","last_clean":"2024-01-16T15:28:26.541954+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:28:26.541954+0000","last_undegraded":"2024-01-16T15:28:26.541954+0000","last_fullsized":"2024-01-16T15:28:26.541954+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.541809+0000","objects_scrubbed":23,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:57.056238+0000","scrub_duration":0.29071314399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'88","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:08.322242+0000","last_change":"2024-01-16T15:29:08.322329+0000","last_active":"2024-01-16T15:29:08.322242+0000","last_peered":"2024-01-16T15:29:08.322242+0000","last_clean":"2024-01-16T15:29:08.322242+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:29:08.322242+0000","last_undegraded":"2024-01-16T15:29:08.322242+0000","last_fullsized":"2024-01-16T15:29:08.322242+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:29:08.322077+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:29:08.322077+0000","last_clean_scrub_stamp":"2024-01-16T15:29:08.322077+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:17.381831+0000","scrub_duration":0.24970814399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'87","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:36.975322+0000","last_change":"2024-01-16T15:28:36.975322+0000","last_active":"2024-01-16T15:28:36.975322+0000","last_peered":"2024-01-16T15:28:36.975322+0000","last_clean":"2024-01-16T15:28:36.975322+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:28:36.975322+0000","last_undegraded":"2024-01-16T15:28:36.975322+0000","last_fullsized":"2024-01-16T15:28:36.975322+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:36.974986+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:36.974986+0000","last_clean_scrub_stamp":"2024-01-16T15:28:36.974986+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:42.637017+0000","scrub_duration":0.26518550499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'94","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.914459+0000","last_change":"2024-01-16T15:28:21.914459+0000","last_active":"2024-01-16T15:28:21.914459+0000","last_peered":"2024-01-16T15:28:21.914459+0000","last_clean":"2024-01-16T15:28:21.914459+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:28:21.914459+0000","last_undegraded":"2024-01-16T15:28:21.914459+0000","last_fullsized":"2024-01-16T15:28:21.914459+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.914080+0000","objects_scrubbed":22,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:00.078053+0000","scrub_duration":0.280246104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'71","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.408300+0000","last_change":"2024-01-16T15:28:24.408300+0000","last_active":"2024-01-16T15:28:24.408300+0000","last_peered":"2024-01-16T15:28:24.408300+0000","last_clean":"2024-01-16T15:28:24.408300+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:28:24.408300+0000","last_undegraded":"2024-01-16T15:28:24.408300+0000","last_fullsized":"2024-01-16T15:28:24.408300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'71","last_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_deep_scrub":"112'71","last_deep_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.408030+0000","objects_scrubbed":12,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:51.797494+0000","scrub_duration":0.124741884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'99","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.255207+0000","last_change":"2024-01-16T15:28:18.255207+0000","last_active":"2024-01-16T15:28:18.255207+0000","last_peered":"2024-01-16T15:28:18.255207+0000","last_clean":"2024-01-16T15:28:18.255207+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:28:18.255207+0000","last_undegraded":"2024-01-16T15:28:18.255207+0000","last_fullsized":"2024-01-16T15:28:18.255207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.254974+0000","objects_scrubbed":25,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:57.777992+0000","scrub_duration":0.28941331399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'65","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.443280+0000","last_change":"2024-01-16T15:28:29.443280+0000","last_active":"2024-01-16T15:28:29.443280+0000","last_peered":"2024-01-16T15:28:29.443280+0000","last_clean":"2024-01-16T15:28:29.443280+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:28:29.443280+0000","last_undegraded":"2024-01-16T15:28:29.443280+0000","last_fullsized":"2024-01-16T15:28:29.443280+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'65","last_scrub_stamp":"2024-01-16T15:28:29.443041+0000","last_deep_scrub":"112'65","last_deep_scrub_stamp":"2024-01-16T15:28:29.443041+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.443041+0000","objects_scrubbed":13,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:23.604246+0000","scrub_duration":0.17569420299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'108","reported_seq":152,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.741028+0000","last_change":"2024-01-16T15:28:26.741028+0000","last_active":"2024-01-16T15:28:26.741028+0000","last_peered":"2024-01-16T15:28:26.741028+0000","last_clean":"2024-01-16T15:28:26.741028+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:28:26.741028+0000","last_undegraded":"2024-01-16T15:28:26.741028+0000","last_fullsized":"2024-01-16T15:28:26.741028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.740929+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:29.967191+0000","scrub_duration":0.29640463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'100","reported_seq":132,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.925516+0000","last_change":"2024-01-16T15:28:28.925578+0000","last_active":"2024-01-16T15:28:28.925516+0000","last_peered":"2024-01-16T15:28:28.925516+0000","last_clean":"2024-01-16T15:28:28.925516+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:28:28.925516+0000","last_undegraded":"2024-01-16T15:28:28.925516+0000","last_fullsized":"2024-01-16T15:28:28.925516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'100","last_scrub_stamp":"2024-01-16T15:28:28.925340+0000","last_deep_scrub":"112'100","last_deep_scrub_stamp":"2024-01-16T15:28:28.925340+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.925340+0000","objects_scrubbed":26,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:24:04.310360+0000","scrub_duration":0.30850802500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'90","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.497303+0000","last_change":"2024-01-16T15:28:28.497397+0000","last_active":"2024-01-16T15:28:28.497303+0000","last_peered":"2024-01-16T15:28:28.497303+0000","last_clean":"2024-01-16T15:28:28.497303+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:28:28.497303+0000","last_undegraded":"2024-01-16T15:28:28.497303+0000","last_fullsized":"2024-01-16T15:28:28.497303+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:28.497088+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:28.497088+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.497088+0000","objects_scrubbed":20,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:34.899596+0000","scrub_duration":0.25302828599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'95","reported_seq":134,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.497418+0000","last_change":"2024-01-16T15:28:32.497568+0000","last_active":"2024-01-16T15:28:32.497418+0000","last_peered":"2024-01-16T15:28:32.497418+0000","last_clean":"2024-01-16T15:28:32.497418+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:28:32.497418+0000","last_undegraded":"2024-01-16T15:28:32.497418+0000","last_fullsized":"2024-01-16T15:28:32.497418+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'95","last_scrub_stamp":"2024-01-16T15:28:32.497207+0000","last_deep_scrub":"112'95","last_deep_scrub_stamp":"2024-01-16T15:28:32.497207+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.497207+0000","objects_scrubbed":22,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:13.297614+0000","scrub_duration":0.23998999500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'94","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:35.999324+0000","last_change":"2024-01-16T15:28:35.999421+0000","last_active":"2024-01-16T15:28:35.999324+0000","last_peered":"2024-01-16T15:28:35.999324+0000","last_clean":"2024-01-16T15:28:35.999324+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:28:35.999324+0000","last_undegraded":"2024-01-16T15:28:35.999324+0000","last_fullsized":"2024-01-16T15:28:35.999324+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:35.999128+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:35.999128+0000","last_clean_scrub_stamp":"2024-01-16T15:28:35.999128+0000","objects_scrubbed":23,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:18.663684+0000","scrub_duration":0.30044927999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'81","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.481673+0000","last_change":"2024-01-16T15:28:33.481747+0000","last_active":"2024-01-16T15:28:33.481673+0000","last_peered":"2024-01-16T15:28:33.481673+0000","last_clean":"2024-01-16T15:28:33.481673+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:28:33.481673+0000","last_undegraded":"2024-01-16T15:28:33.481673+0000","last_fullsized":"2024-01-16T15:28:33.481673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:33.481512+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:33.481512+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.481512+0000","objects_scrubbed":18,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:34.932260+0000","scrub_duration":0.20372825899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'90","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.951640+0000","last_change":"2024-01-16T15:28:20.951722+0000","last_active":"2024-01-16T15:28:20.951640+0000","last_peered":"2024-01-16T15:28:20.951640+0000","last_clean":"2024-01-16T15:28:20.951640+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:28:20.951640+0000","last_undegraded":"2024-01-16T15:28:20.951640+0000","last_fullsized":"2024-01-16T15:28:20.951640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.951444+0000","objects_scrubbed":19,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:52.025141+0000","scrub_duration":0.20406614100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'89","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.496415+0000","last_change":"2024-01-16T15:28:27.496503+0000","last_active":"2024-01-16T15:28:27.496415+0000","last_peered":"2024-01-16T15:28:27.496415+0000","last_clean":"2024-01-16T15:28:27.496415+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:28:27.496415+0000","last_undegraded":"2024-01-16T15:28:27.496415+0000","last_fullsized":"2024-01-16T15:28:27.496415+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:27.496240+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:27.496240+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.496240+0000","objects_scrubbed":20,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:37.199022+0000","scrub_duration":0.25531969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'82","reported_seq":108,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566596+0000","last_change":"2024-01-16T15:19:24.631821+0000","last_active":"2024-01-16T15:28:05.566596+0000","last_peered":"2024-01-16T15:28:05.566596+0000","last_clean":"2024-01-16T15:28:05.566596+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:28:05.566596+0000","last_undegraded":"2024-01-16T15:28:05.566596+0000","last_fullsized":"2024-01-16T15:28:05.566596+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.302192+0000","last_change":"2024-01-16T15:28:28.302272+0000","last_active":"2024-01-16T15:28:28.302192+0000","last_peered":"2024-01-16T15:28:28.302192+0000","last_clean":"2024-01-16T15:28:28.302192+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:28:28.302192+0000","last_undegraded":"2024-01-16T15:28:28.302192+0000","last_fullsized":"2024-01-16T15:28:28.302192+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.302044+0000","objects_scrubbed":16,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:38.499053+0000","scrub_duration":0.149915824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'63","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:44.708897+0000","last_change":"2024-01-16T15:28:44.708973+0000","last_active":"2024-01-16T15:28:44.708897+0000","last_peered":"2024-01-16T15:28:44.708897+0000","last_clean":"2024-01-16T15:28:44.708897+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:28:44.708897+0000","last_undegraded":"2024-01-16T15:28:44.708897+0000","last_fullsized":"2024-01-16T15:28:44.708897+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'63","last_scrub_stamp":"2024-01-16T15:28:44.708715+0000","last_deep_scrub":"112'63","last_deep_scrub_stamp":"2024-01-16T15:28:44.708715+0000","last_clean_scrub_stamp":"2024-01-16T15:28:44.708715+0000","objects_scrubbed":13,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:25.059080+0000","scrub_duration":0.14963963599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'81","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.316071+0000","last_change":"2024-01-16T15:28:40.316137+0000","last_active":"2024-01-16T15:28:40.316071+0000","last_peered":"2024-01-16T15:28:40.316071+0000","last_clean":"2024-01-16T15:28:40.316071+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:28:40.316071+0000","last_undegraded":"2024-01-16T15:28:40.316071+0000","last_fullsized":"2024-01-16T15:28:40.316071+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:40.315969+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:40.315969+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.315969+0000","objects_scrubbed":19,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:11.371995+0000","scrub_duration":0.23402421100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'95","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.537448+0000","last_change":"2024-01-16T15:28:42.537543+0000","last_active":"2024-01-16T15:28:42.537448+0000","last_peered":"2024-01-16T15:28:42.537448+0000","last_clean":"2024-01-16T15:28:42.537448+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:28:42.537448+0000","last_undegraded":"2024-01-16T15:28:42.537448+0000","last_fullsized":"2024-01-16T15:28:42.537448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'95","last_scrub_stamp":"2024-01-16T15:28:42.537268+0000","last_deep_scrub":"112'95","last_deep_scrub_stamp":"2024-01-16T15:28:42.537268+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.537268+0000","objects_scrubbed":17,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:49.101480+0000","scrub_duration":0.19056937099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'117","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.343525+0000","last_change":"2024-01-16T15:28:29.343618+0000","last_active":"2024-01-16T15:28:29.343525+0000","last_peered":"2024-01-16T15:28:29.343525+0000","last_clean":"2024-01-16T15:28:29.343525+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:28:29.343525+0000","last_undegraded":"2024-01-16T15:28:29.343525+0000","last_fullsized":"2024-01-16T15:28:29.343525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'117","last_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_deep_scrub":"112'117","last_deep_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.343355+0000","objects_scrubbed":26,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:18.083505+0000","scrub_duration":0.31588118900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'102","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.453254+0000","last_change":"2024-01-16T15:28:19.453335+0000","last_active":"2024-01-16T15:28:19.453254+0000","last_peered":"2024-01-16T15:28:19.453254+0000","last_clean":"2024-01-16T15:28:19.453254+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:28:19.453254+0000","last_undegraded":"2024-01-16T15:28:19.453254+0000","last_fullsized":"2024-01-16T15:28:19.453254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'102","last_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_deep_scrub":"112'102","last_deep_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.453110+0000","objects_scrubbed":24,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.543525+0000","scrub_duration":0.24508891399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'81","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.343081+0000","last_change":"2024-01-16T15:28:33.343229+0000","last_active":"2024-01-16T15:28:33.343081+0000","last_peered":"2024-01-16T15:28:33.343081+0000","last_clean":"2024-01-16T15:28:33.343081+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:28:33.343081+0000","last_undegraded":"2024-01-16T15:28:33.343081+0000","last_fullsized":"2024-01-16T15:28:33.343081+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:33.342865+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:33.342865+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.342865+0000","objects_scrubbed":21,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:09.821872+0000","scrub_duration":0.226300789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'6","reported_seq":33,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:41.310657+0000","last_change":"2024-01-16T15:28:41.310726+0000","last_active":"2024-01-16T15:28:41.310657+0000","last_peered":"2024-01-16T15:28:41.310657+0000","last_clean":"2024-01-16T15:28:41.310657+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:28:41.310657+0000","last_undegraded":"2024-01-16T15:28:41.310657+0000","last_fullsized":"2024-01-16T15:28:41.310657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'6","last_scrub_stamp":"2024-01-16T15:28:41.310472+0000","last_deep_scrub":"112'6","last_deep_scrub_stamp":"2024-01-16T15:28:41.310472+0000","last_clean_scrub_stamp":"2024-01-16T15:28:41.310472+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:08.138176+0000","scrub_duration":0.0084308639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.835785+0000","last_change":"2024-01-16T15:28:32.835862+0000","last_active":"2024-01-16T15:28:32.835785+0000","last_peered":"2024-01-16T15:28:32.835785+0000","last_clean":"2024-01-16T15:28:32.835785+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:28:32.835785+0000","last_undegraded":"2024-01-16T15:28:32.835785+0000","last_fullsized":"2024-01-16T15:28:32.835785+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:32.835644+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:32.835644+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.835644+0000","objects_scrubbed":21,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:16:19.714227+0000","scrub_duration":0.264673932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.589919+0000","last_change":"2024-01-16T15:28:20.589975+0000","last_active":"2024-01-16T15:28:20.589919+0000","last_peered":"2024-01-16T15:28:20.589919+0000","last_clean":"2024-01-16T15:28:20.589919+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:28:20.589919+0000","last_undegraded":"2024-01-16T15:28:20.589919+0000","last_fullsized":"2024-01-16T15:28:20.589919+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.589720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:37.660795+0000","scrub_duration":0.0035764669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'85","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.351967+0000","last_change":"2024-01-16T15:28:30.352108+0000","last_active":"2024-01-16T15:28:30.351967+0000","last_peered":"2024-01-16T15:28:30.351967+0000","last_clean":"2024-01-16T15:28:30.351967+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:28:30.351967+0000","last_undegraded":"2024-01-16T15:28:30.351967+0000","last_fullsized":"2024-01-16T15:28:30.351967+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.351810+0000","objects_scrubbed":19,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:22.513330+0000","scrub_duration":0.19453525299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'4","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.865026+0000","last_change":"2024-01-16T15:28:10.865070+0000","last_active":"2024-01-16T15:28:10.865026+0000","last_peered":"2024-01-16T15:28:10.865026+0000","last_clean":"2024-01-16T15:28:10.865026+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:28:10.865026+0000","last_undegraded":"2024-01-16T15:28:10.865026+0000","last_fullsized":"2024-01-16T15:28:10.865026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.864918+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:32.896445+0000","scrub_duration":0.0059496970000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'109","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.975751+0000","last_change":"2024-01-16T15:28:30.975803+0000","last_active":"2024-01-16T15:28:30.975751+0000","last_peered":"2024-01-16T15:28:30.975751+0000","last_clean":"2024-01-16T15:28:30.975751+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:28:30.975751+0000","last_undegraded":"2024-01-16T15:28:30.975751+0000","last_fullsized":"2024-01-16T15:28:30.975751+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'109","last_scrub_stamp":"2024-01-16T15:28:30.975585+0000","last_deep_scrub":"112'109","last_deep_scrub_stamp":"2024-01-16T15:28:30.975585+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.975585+0000","objects_scrubbed":25,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:50.616075+0000","scrub_duration":0.273993918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'3","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:13.789441+0000","last_change":"2024-01-16T15:28:13.789502+0000","last_active":"2024-01-16T15:28:13.789441+0000","last_peered":"2024-01-16T15:28:13.789441+0000","last_clean":"2024-01-16T15:28:13.789441+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:28:13.789441+0000","last_undegraded":"2024-01-16T15:28:13.789441+0000","last_fullsized":"2024-01-16T15:28:13.789441+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_clean_scrub_stamp":"2024-01-16T15:28:13.789255+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:19.022780+0000","scrub_duration":0.0072576510000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'94","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:31.165009+0000","last_change":"2024-01-16T15:28:31.165088+0000","last_active":"2024-01-16T15:28:31.165009+0000","last_peered":"2024-01-16T15:28:31.165009+0000","last_clean":"2024-01-16T15:28:31.165009+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:28:31.165009+0000","last_undegraded":"2024-01-16T15:28:31.165009+0000","last_fullsized":"2024-01-16T15:28:31.165009+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:31.164841+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:31.164841+0000","last_clean_scrub_stamp":"2024-01-16T15:28:31.164841+0000","objects_scrubbed":16,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:05.015647+0000","scrub_duration":0.15844903499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.260333+0000","last_change":"2024-01-16T15:28:24.260389+0000","last_active":"2024-01-16T15:28:24.260333+0000","last_peered":"2024-01-16T15:28:24.260333+0000","last_clean":"2024-01-16T15:28:24.260333+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:28:24.260333+0000","last_undegraded":"2024-01-16T15:28:24.260333+0000","last_fullsized":"2024-01-16T15:28:24.260333+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.260116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:22.819324+0000","scrub_duration":0.0042817660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'108","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:39.416380+0000","last_change":"2024-01-16T15:28:39.416442+0000","last_active":"2024-01-16T15:28:39.416380+0000","last_peered":"2024-01-16T15:28:39.416380+0000","last_clean":"2024-01-16T15:28:39.416380+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:28:39.416380+0000","last_undegraded":"2024-01-16T15:28:39.416380+0000","last_fullsized":"2024-01-16T15:28:39.416380+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:39.416188+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:39.416188+0000","last_clean_scrub_stamp":"2024-01-16T15:28:39.416188+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:12.499311+0000","scrub_duration":0.29096126300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.816242+0000","last_change":"2024-01-16T15:28:14.816296+0000","last_active":"2024-01-16T15:28:14.816242+0000","last_peered":"2024-01-16T15:28:14.816242+0000","last_clean":"2024-01-16T15:28:14.816242+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:28:14.816242+0000","last_undegraded":"2024-01-16T15:28:14.816242+0000","last_fullsized":"2024-01-16T15:28:14.816242+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.816039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:58.559289+0000","scrub_duration":0.0043054030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'99","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:02.298030+0000","last_change":"2024-01-16T15:29:02.298030+0000","last_active":"2024-01-16T15:29:02.298030+0000","last_peered":"2024-01-16T15:29:02.298030+0000","last_clean":"2024-01-16T15:29:02.298030+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:29:02.298030+0000","last_undegraded":"2024-01-16T15:29:02.298030+0000","last_fullsized":"2024-01-16T15:29:02.298030+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:29:02.297776+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:29:02.297776+0000","last_clean_scrub_stamp":"2024-01-16T15:29:02.297776+0000","objects_scrubbed":19,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:22.958112+0000","scrub_duration":0.19285666200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.251291+0000","last_change":"2024-01-16T15:28:11.251291+0000","last_active":"2024-01-16T15:28:11.251291+0000","last_peered":"2024-01-16T15:28:11.251291+0000","last_clean":"2024-01-16T15:28:11.251291+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:28:11.251291+0000","last_undegraded":"2024-01-16T15:28:11.251291+0000","last_fullsized":"2024-01-16T15:28:11.251291+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.251026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:45.371008+0000","scrub_duration":0.0038716129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'86","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:44.544412+0000","last_change":"2024-01-16T15:28:44.544412+0000","last_active":"2024-01-16T15:28:44.544412+0000","last_peered":"2024-01-16T15:28:44.544412+0000","last_clean":"2024-01-16T15:28:44.544412+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:28:44.544412+0000","last_undegraded":"2024-01-16T15:28:44.544412+0000","last_fullsized":"2024-01-16T15:28:44.544412+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'86","last_scrub_stamp":"2024-01-16T15:28:44.544211+0000","last_deep_scrub":"112'86","last_deep_scrub_stamp":"2024-01-16T15:28:44.544211+0000","last_clean_scrub_stamp":"2024-01-16T15:28:44.544211+0000","objects_scrubbed":18,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:52.626917+0000","scrub_duration":0.22887946100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.061040+0000","last_change":"2024-01-16T15:28:42.061040+0000","last_active":"2024-01-16T15:28:42.061040+0000","last_peered":"2024-01-16T15:28:42.061040+0000","last_clean":"2024-01-16T15:28:42.061040+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:28:42.061040+0000","last_undegraded":"2024-01-16T15:28:42.061040+0000","last_fullsized":"2024-01-16T15:28:42.061040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:42.060723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:42.060723+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.060723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:46.204134+0000","scrub_duration":0.0036082470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'80","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:52.478582+0000","last_change":"2024-01-16T15:28:52.478582+0000","last_active":"2024-01-16T15:28:52.478582+0000","last_peered":"2024-01-16T15:28:52.478582+0000","last_clean":"2024-01-16T15:28:52.478582+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:28:52.478582+0000","last_undegraded":"2024-01-16T15:28:52.478582+0000","last_fullsized":"2024-01-16T15:28:52.478582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'80","last_scrub_stamp":"2024-01-16T15:28:52.478341+0000","last_deep_scrub":"112'80","last_deep_scrub_stamp":"2024-01-16T15:28:52.478341+0000","last_clean_scrub_stamp":"2024-01-16T15:28:52.478341+0000","objects_scrubbed":14,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:58.318697+0000","scrub_duration":0.152373551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.843271+0000","last_change":"2024-01-16T15:28:11.843271+0000","last_active":"2024-01-16T15:28:11.843271+0000","last_peered":"2024-01-16T15:28:11.843271+0000","last_clean":"2024-01-16T15:28:11.843271+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:28:11.843271+0000","last_undegraded":"2024-01-16T15:28:11.843271+0000","last_fullsized":"2024-01-16T15:28:11.843271+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.843032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:01.354272+0000","scrub_duration":0.003520572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'82","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.726792+0000","last_change":"2024-01-16T15:28:29.726792+0000","last_active":"2024-01-16T15:28:29.726792+0000","last_peered":"2024-01-16T15:28:29.726792+0000","last_clean":"2024-01-16T15:28:29.726792+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:28:29.726792+0000","last_undegraded":"2024-01-16T15:28:29.726792+0000","last_fullsized":"2024-01-16T15:28:29.726792+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:29.726602+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:29.726602+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.726602+0000","objects_scrubbed":18,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:25.349562+0000","scrub_duration":0.187189361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.784389+0000","last_change":"2024-01-16T15:28:16.784389+0000","last_active":"2024-01-16T15:28:16.784389+0000","last_peered":"2024-01-16T15:28:16.784389+0000","last_clean":"2024-01-16T15:28:16.784389+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:28:16.784389+0000","last_undegraded":"2024-01-16T15:28:16.784389+0000","last_fullsized":"2024-01-16T15:28:16.784389+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.784086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:32.176453+0000","scrub_duration":0.0040169810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'79","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.831200+0000","last_change":"2024-01-16T15:28:42.831200+0000","last_active":"2024-01-16T15:28:42.831200+0000","last_peered":"2024-01-16T15:28:42.831200+0000","last_clean":"2024-01-16T15:28:42.831200+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:28:42.831200+0000","last_undegraded":"2024-01-16T15:28:42.831200+0000","last_fullsized":"2024-01-16T15:28:42.831200+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:42.830943+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:42.830943+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.830943+0000","objects_scrubbed":19,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:48.006952+0000","scrub_duration":0.21546314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'33","reported_seq":60,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:50.170253+0000","last_change":"2024-01-16T15:28:50.170253+0000","last_active":"2024-01-16T15:28:50.170253+0000","last_peered":"2024-01-16T15:28:50.170253+0000","last_clean":"2024-01-16T15:28:50.170253+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:28:50.170253+0000","last_undegraded":"2024-01-16T15:28:50.170253+0000","last_fullsized":"2024-01-16T15:28:50.170253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'33","last_scrub_stamp":"2024-01-16T15:28:50.169974+0000","last_deep_scrub":"112'33","last_deep_scrub_stamp":"2024-01-16T15:28:50.169974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:50.169974+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:02.553184+0000","scrub_duration":0.008481901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'93","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:45.595636+0000","last_change":"2024-01-16T15:28:45.595636+0000","last_active":"2024-01-16T15:28:45.595636+0000","last_peered":"2024-01-16T15:28:45.595636+0000","last_clean":"2024-01-16T15:28:45.595636+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:28:45.595636+0000","last_undegraded":"2024-01-16T15:28:45.595636+0000","last_fullsized":"2024-01-16T15:28:45.595636+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:45.595459+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:45.595459+0000","last_clean_scrub_stamp":"2024-01-16T15:28:45.595459+0000","objects_scrubbed":24,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:41:29.813183+0000","scrub_duration":0.29598812099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'311","reported_seq":338,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:48.165089+0000","last_change":"2024-01-16T15:28:48.165251+0000","last_active":"2024-01-16T15:28:48.165089+0000","last_peered":"2024-01-16T15:28:48.165089+0000","last_clean":"2024-01-16T15:28:48.165089+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:28:48.165089+0000","last_undegraded":"2024-01-16T15:28:48.165089+0000","last_fullsized":"2024-01-16T15:28:48.165089+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'311","last_scrub_stamp":"2024-01-16T15:28:48.164882+0000","last_deep_scrub":"112'311","last_deep_scrub_stamp":"2024-01-16T15:28:48.164882+0000","last_clean_scrub_stamp":"2024-01-16T15:28:48.164882+0000","objects_scrubbed":4,"log_size":311,"log_dups_size":0,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:23.660630+0000","scrub_duration":0.020197336999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196841,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":306,"num_write_kb":1324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'79","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.880948+0000","last_change":"2024-01-16T15:28:33.880948+0000","last_active":"2024-01-16T15:28:33.880948+0000","last_peered":"2024-01-16T15:28:33.880948+0000","last_clean":"2024-01-16T15:28:33.880948+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:28:33.880948+0000","last_undegraded":"2024-01-16T15:28:33.880948+0000","last_fullsized":"2024-01-16T15:28:33.880948+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:33.880692+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:33.880692+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.880692+0000","objects_scrubbed":16,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:30.122032+0000","scrub_duration":0.177224733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.219036+0000","last_change":"2024-01-16T15:28:18.219036+0000","last_active":"2024-01-16T15:28:18.219036+0000","last_peered":"2024-01-16T15:28:18.219036+0000","last_clean":"2024-01-16T15:28:18.219036+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:28:18.219036+0000","last_undegraded":"2024-01-16T15:28:18.219036+0000","last_fullsized":"2024-01-16T15:28:18.219036+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.218660+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:48.176772+0000","scrub_duration":0.0091646809999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'98","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:36.911299+0000","last_change":"2024-01-16T15:28:36.911299+0000","last_active":"2024-01-16T15:28:36.911299+0000","last_peered":"2024-01-16T15:28:36.911299+0000","last_clean":"2024-01-16T15:28:36.911299+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:28:36.911299+0000","last_undegraded":"2024-01-16T15:28:36.911299+0000","last_fullsized":"2024-01-16T15:28:36.911299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'98","last_scrub_stamp":"2024-01-16T15:28:36.911039+0000","last_deep_scrub":"112'98","last_deep_scrub_stamp":"2024-01-16T15:28:36.911039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:36.911039+0000","objects_scrubbed":18,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:57.499710+0000","scrub_duration":0.20160798599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.450274+0000","last_change":"2024-01-16T15:28:16.450274+0000","last_active":"2024-01-16T15:28:16.450274+0000","last_peered":"2024-01-16T15:28:16.450274+0000","last_clean":"2024-01-16T15:28:16.450274+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:28:16.450274+0000","last_undegraded":"2024-01-16T15:28:16.450274+0000","last_fullsized":"2024-01-16T15:28:16.450274+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.449979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:39.537507+0000","scrub_duration":0.0036212340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'96","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.977310+0000","last_change":"2024-01-16T15:28:32.977310+0000","last_active":"2024-01-16T15:28:32.977310+0000","last_peered":"2024-01-16T15:28:32.977310+0000","last_clean":"2024-01-16T15:28:32.977310+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:28:32.977310+0000","last_undegraded":"2024-01-16T15:28:32.977310+0000","last_fullsized":"2024-01-16T15:28:32.977310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:32.977043+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:32.977043+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.977043+0000","objects_scrubbed":23,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:27:22.982613+0000","scrub_duration":0.25583332399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'12","reported_seq":39,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.825453+0000","last_change":"2024-01-16T15:28:07.825453+0000","last_active":"2024-01-16T15:28:07.825453+0000","last_peered":"2024-01-16T15:28:07.825453+0000","last_clean":"2024-01-16T15:28:07.825453+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:28:07.825453+0000","last_undegraded":"2024-01-16T15:28:07.825453+0000","last_fullsized":"2024-01-16T15:28:07.825453+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.825335+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:20.349610+0000","scrub_duration":0.0056472520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'62","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.989502+0000","last_change":"2024-01-16T15:28:21.989502+0000","last_active":"2024-01-16T15:28:21.989502+0000","last_peered":"2024-01-16T15:28:21.989502+0000","last_clean":"2024-01-16T15:28:21.989502+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:28:21.989502+0000","last_undegraded":"2024-01-16T15:28:21.989502+0000","last_fullsized":"2024-01-16T15:28:21.989502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'62","last_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_deep_scrub":"112'62","last_deep_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.989317+0000","objects_scrubbed":12,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:49.378780+0000","scrub_duration":0.125520883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'2","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:54.163157+0000","last_change":"2024-01-16T15:28:54.163222+0000","last_active":"2024-01-16T15:28:54.163157+0000","last_peered":"2024-01-16T15:28:54.163157+0000","last_clean":"2024-01-16T15:28:54.163157+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:28:54.163157+0000","last_undegraded":"2024-01-16T15:28:54.163157+0000","last_fullsized":"2024-01-16T15:28:54.163157+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-01-16T15:28:54.162972+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-01-16T15:28:54.162972+0000","last_clean_scrub_stamp":"2024-01-16T15:28:54.162972+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:08.962695+0000","scrub_duration":0.0090223179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'81","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:34.355627+0000","last_change":"2024-01-16T15:28:34.355708+0000","last_active":"2024-01-16T15:28:34.355627+0000","last_peered":"2024-01-16T15:28:34.355627+0000","last_clean":"2024-01-16T15:28:34.355627+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:28:34.355627+0000","last_undegraded":"2024-01-16T15:28:34.355627+0000","last_fullsized":"2024-01-16T15:28:34.355627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:34.355450+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:34.355450+0000","last_clean_scrub_stamp":"2024-01-16T15:28:34.355450+0000","objects_scrubbed":20,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:13.088452+0000","scrub_duration":0.20144817200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.809350+0000","last_change":"2024-01-16T15:28:17.809350+0000","last_active":"2024-01-16T15:28:17.809350+0000","last_peered":"2024-01-16T15:28:17.809350+0000","last_clean":"2024-01-16T15:28:17.809350+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:28:17.809350+0000","last_undegraded":"2024-01-16T15:28:17.809350+0000","last_fullsized":"2024-01-16T15:28:17.809350+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.809101+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:47.767212+0000","scrub_duration":0.0070965819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'100","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.997559+0000","last_change":"2024-01-16T15:28:40.997649+0000","last_active":"2024-01-16T15:28:40.997559+0000","last_peered":"2024-01-16T15:28:40.997559+0000","last_clean":"2024-01-16T15:28:40.997559+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:28:40.997559+0000","last_undegraded":"2024-01-16T15:28:40.997559+0000","last_fullsized":"2024-01-16T15:28:40.997559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'100","last_scrub_stamp":"2024-01-16T15:28:40.997402+0000","last_deep_scrub":"112'100","last_deep_scrub_stamp":"2024-01-16T15:28:40.997402+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.997402+0000","objects_scrubbed":23,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:49.477973+0000","scrub_duration":0.25693853799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'2","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.443869+0000","last_change":"2024-01-16T15:28:18.443869+0000","last_active":"2024-01-16T15:28:18.443869+0000","last_peered":"2024-01-16T15:28:18.443869+0000","last_clean":"2024-01-16T15:28:18.443869+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:28:18.443869+0000","last_undegraded":"2024-01-16T15:28:18.443869+0000","last_fullsized":"2024-01-16T15:28:18.443869+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.443623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:31.553582+0000","scrub_duration":0.0077739760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'89","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:34.943163+0000","last_change":"2024-01-16T15:28:34.943251+0000","last_active":"2024-01-16T15:28:34.943163+0000","last_peered":"2024-01-16T15:28:34.943163+0000","last_clean":"2024-01-16T15:28:34.943163+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:28:34.943163+0000","last_undegraded":"2024-01-16T15:28:34.943163+0000","last_fullsized":"2024-01-16T15:28:34.943163+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:34.942995+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:34.942995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:34.942995+0000","objects_scrubbed":23,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:58.983028+0000","scrub_duration":0.28306326100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.220561+0000","last_change":"2024-01-16T15:28:32.220610+0000","last_active":"2024-01-16T15:28:32.220561+0000","last_peered":"2024-01-16T15:28:32.220561+0000","last_clean":"2024-01-16T15:28:32.220561+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:28:32.220561+0000","last_undegraded":"2024-01-16T15:28:32.220561+0000","last_fullsized":"2024-01-16T15:28:32.220561+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:32.220364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:32.220364+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.220364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:02.571057+0000","scrub_duration":0.0041910580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'89","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.838225+0000","last_change":"2024-01-16T15:28:26.838277+0000","last_active":"2024-01-16T15:28:26.838225+0000","last_peered":"2024-01-16T15:28:26.838225+0000","last_clean":"2024-01-16T15:28:26.838225+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:28:26.838225+0000","last_undegraded":"2024-01-16T15:28:26.838225+0000","last_fullsized":"2024-01-16T15:28:26.838225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:26.838060+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:26.838060+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.838060+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:31.159826+0000","scrub_duration":0.27286994599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.174817+0000","last_change":"2024-01-16T15:28:17.174817+0000","last_active":"2024-01-16T15:28:17.174817+0000","last_peered":"2024-01-16T15:28:17.174817+0000","last_clean":"2024-01-16T15:28:17.174817+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:28:17.174817+0000","last_undegraded":"2024-01-16T15:28:17.174817+0000","last_fullsized":"2024-01-16T15:28:17.174817+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.174587+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:21.215097+0000","scrub_duration":0.0071591010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'73","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566466+0000","last_change":"2024-01-16T15:19:24.624650+0000","last_active":"2024-01-16T15:28:05.566466+0000","last_peered":"2024-01-16T15:28:05.566466+0000","last_clean":"2024-01-16T15:28:05.566466+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:28:05.566466+0000","last_undegraded":"2024-01-16T15:28:05.566466+0000","last_fullsized":"2024-01-16T15:28:05.566466+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.833365+0000","last_change":"2024-01-16T15:28:19.833414+0000","last_active":"2024-01-16T15:28:19.833365+0000","last_peered":"2024-01-16T15:28:19.833365+0000","last_clean":"2024-01-16T15:28:19.833365+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:28:19.833365+0000","last_undegraded":"2024-01-16T15:28:19.833365+0000","last_fullsized":"2024-01-16T15:28:19.833365+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.923614+0000","scrub_duration":0.0036415929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'84","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.813828+0000","last_change":"2024-01-16T15:28:29.813905+0000","last_active":"2024-01-16T15:28:29.813828+0000","last_peered":"2024-01-16T15:28:29.813828+0000","last_clean":"2024-01-16T15:28:29.813828+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:28:29.813828+0000","last_undegraded":"2024-01-16T15:28:29.813828+0000","last_fullsized":"2024-01-16T15:28:29.813828+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'84","last_scrub_stamp":"2024-01-16T15:28:29.813679+0000","last_deep_scrub":"112'84","last_deep_scrub_stamp":"2024-01-16T15:28:29.813679+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.813679+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:02.832647+0000","scrub_duration":0.15799079199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.206909+0000","last_change":"2024-01-16T15:28:15.206975+0000","last_active":"2024-01-16T15:28:15.206909+0000","last_peered":"2024-01-16T15:28:15.206909+0000","last_clean":"2024-01-16T15:28:15.206909+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:28:15.206909+0000","last_undegraded":"2024-01-16T15:28:15.206909+0000","last_fullsized":"2024-01-16T15:28:15.206909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.206765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:27.403194+0000","scrub_duration":0.0041188539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.874708+0000","last_change":"2024-01-16T15:28:30.874778+0000","last_active":"2024-01-16T15:28:30.874708+0000","last_peered":"2024-01-16T15:28:30.874708+0000","last_clean":"2024-01-16T15:28:30.874708+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:28:30.874708+0000","last_undegraded":"2024-01-16T15:28:30.874708+0000","last_fullsized":"2024-01-16T15:28:30.874708+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:30.874583+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:30.874583+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.874583+0000","objects_scrubbed":18,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:07.770194+0000","scrub_duration":0.163030705,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.044284+0000","last_change":"2024-01-16T15:28:40.044333+0000","last_active":"2024-01-16T15:28:40.044284+0000","last_peered":"2024-01-16T15:28:40.044284+0000","last_clean":"2024-01-16T15:28:40.044284+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:28:40.044284+0000","last_undegraded":"2024-01-16T15:28:40.044284+0000","last_fullsized":"2024-01-16T15:28:40.044284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:40.044093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:40.044093+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.044093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:49.863547+0000","scrub_duration":0.0043197859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'92","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:22.637191+0000","last_change":"2024-01-16T15:28:22.637284+0000","last_active":"2024-01-16T15:28:22.637191+0000","last_peered":"2024-01-16T15:28:22.637191+0000","last_clean":"2024-01-16T15:28:22.637191+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:28:22.637191+0000","last_undegraded":"2024-01-16T15:28:22.637191+0000","last_fullsized":"2024-01-16T15:28:22.637191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'92","last_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_deep_scrub":"112'92","last_deep_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_clean_scrub_stamp":"2024-01-16T15:28:22.636970+0000","objects_scrubbed":22,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:01.808019+0000","scrub_duration":0.23588124099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.749499+0000","last_change":"2024-01-16T15:28:17.749653+0000","last_active":"2024-01-16T15:28:17.749499+0000","last_peered":"2024-01-16T15:28:17.749499+0000","last_clean":"2024-01-16T15:28:17.749499+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:28:17.749499+0000","last_undegraded":"2024-01-16T15:28:17.749499+0000","last_fullsized":"2024-01-16T15:28:17.749499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.749294+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:03.835812+0000","scrub_duration":0.0071287670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'64","reported_seq":95,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:05.566405+0000","last_change":"2024-01-16T15:19:24.638377+0000","last_active":"2024-01-16T15:28:05.566405+0000","last_peered":"2024-01-16T15:28:05.566405+0000","last_clean":"2024-01-16T15:28:05.566405+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:28:05.566405+0000","last_undegraded":"2024-01-16T15:28:05.566405+0000","last_fullsized":"2024-01-16T15:28:05.566405+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:19:23.577505+0000","last_clean_scrub_stamp":"2024-01-16T15:19:23.577505+0000","objects_scrubbed":0,"log_size":64,"log_dups_size":0,"ondisk_log_size":64,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":58,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.479780+0000","last_change":"2024-01-16T15:28:17.479819+0000","last_active":"2024-01-16T15:28:17.479780+0000","last_peered":"2024-01-16T15:28:17.479780+0000","last_clean":"2024-01-16T15:28:17.479780+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:28:17.479780+0000","last_undegraded":"2024-01-16T15:28:17.479780+0000","last_fullsized":"2024-01-16T15:28:17.479780+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'31","last_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_deep_scrub":"112'31","last_deep_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.479610+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:48.802592+0000","scrub_duration":0.0060914480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593,"num_omap_keys":11,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'96","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.235685+0000","last_change":"2024-01-16T15:28:15.235729+0000","last_active":"2024-01-16T15:28:15.235685+0000","last_peered":"2024-01-16T15:28:15.235685+0000","last_clean":"2024-01-16T15:28:15.235685+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:28:15.235685+0000","last_undegraded":"2024-01-16T15:28:15.235685+0000","last_fullsized":"2024-01-16T15:28:15.235685+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.235576+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:39.275609+0000","scrub_duration":0.27566697899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:47.140086+0000","last_change":"2024-01-16T15:28:47.140200+0000","last_active":"2024-01-16T15:28:47.140086+0000","last_peered":"2024-01-16T15:28:47.140086+0000","last_clean":"2024-01-16T15:28:47.140086+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:28:47.140086+0000","last_undegraded":"2024-01-16T15:28:47.140086+0000","last_fullsized":"2024-01-16T15:28:47.140086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:47.139873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:47.139873+0000","last_clean_scrub_stamp":"2024-01-16T15:28:47.139873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:12.869817+0000","scrub_duration":0.004451694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'66","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.564786+0000","last_change":"2024-01-16T15:28:20.564862+0000","last_active":"2024-01-16T15:28:20.564786+0000","last_peered":"2024-01-16T15:28:20.564786+0000","last_clean":"2024-01-16T15:28:20.564786+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:28:20.564786+0000","last_undegraded":"2024-01-16T15:28:20.564786+0000","last_fullsized":"2024-01-16T15:28:20.564786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'66","last_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_deep_scrub":"112'66","last_deep_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.564586+0000","objects_scrubbed":13,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:24.471110+0000","scrub_duration":0.12569713499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.884673+0000","last_change":"2024-01-16T15:28:10.884720+0000","last_active":"2024-01-16T15:28:10.884673+0000","last_peered":"2024-01-16T15:28:10.884673+0000","last_clean":"2024-01-16T15:28:10.884673+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:28:10.884673+0000","last_undegraded":"2024-01-16T15:28:10.884673+0000","last_fullsized":"2024-01-16T15:28:10.884673+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.884468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:06.507427+0000","scrub_duration":0.0032287930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.481473+0000","last_change":"2024-01-16T15:28:17.481559+0000","last_active":"2024-01-16T15:28:17.481473+0000","last_peered":"2024-01-16T15:28:17.481473+0000","last_clean":"2024-01-16T15:28:17.481473+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:28:17.481473+0000","last_undegraded":"2024-01-16T15:28:17.481473+0000","last_fullsized":"2024-01-16T15:28:17.481473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.481296+0000","objects_scrubbed":17,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:01.224547+0000","scrub_duration":0.21059523299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.256061+0000","last_change":"2024-01-16T15:28:18.256178+0000","last_active":"2024-01-16T15:28:18.256061+0000","last_peered":"2024-01-16T15:28:18.256061+0000","last_clean":"2024-01-16T15:28:18.256061+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:28:18.256061+0000","last_undegraded":"2024-01-16T15:28:18.256061+0000","last_fullsized":"2024-01-16T15:28:18.256061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.255843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:27.175270+0000","scrub_duration":0.0036640700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'85","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:37.411650+0000","last_change":"2024-01-16T15:28:37.411773+0000","last_active":"2024-01-16T15:28:37.411650+0000","last_peered":"2024-01-16T15:28:37.411650+0000","last_clean":"2024-01-16T15:28:37.411650+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:28:37.411650+0000","last_undegraded":"2024-01-16T15:28:37.411650+0000","last_fullsized":"2024-01-16T15:28:37.411650+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:37.411457+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:37.411457+0000","last_clean_scrub_stamp":"2024-01-16T15:28:37.411457+0000","objects_scrubbed":21,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:09.422228+0000","scrub_duration":0.22994655999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.221495+0000","last_change":"2024-01-16T15:28:10.221583+0000","last_active":"2024-01-16T15:28:10.221495+0000","last_peered":"2024-01-16T15:28:10.221495+0000","last_clean":"2024-01-16T15:28:10.221495+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:28:10.221495+0000","last_undegraded":"2024-01-16T15:28:10.221495+0000","last_fullsized":"2024-01-16T15:28:10.221495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.221197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:27:58.612687+0000","scrub_duration":0.0041312190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'94","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:35.980220+0000","last_change":"2024-01-16T15:28:35.980283+0000","last_active":"2024-01-16T15:28:35.980220+0000","last_peered":"2024-01-16T15:28:35.980220+0000","last_clean":"2024-01-16T15:28:35.980220+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:28:35.980220+0000","last_undegraded":"2024-01-16T15:28:35.980220+0000","last_fullsized":"2024-01-16T15:28:35.980220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:35.980111+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:35.980111+0000","last_clean_scrub_stamp":"2024-01-16T15:28:35.980111+0000","objects_scrubbed":21,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:55:32.046445+0000","scrub_duration":0.287987259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'32","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.827087+0000","last_change":"2024-01-16T15:28:14.827144+0000","last_active":"2024-01-16T15:28:14.827087+0000","last_peered":"2024-01-16T15:28:14.827087+0000","last_clean":"2024-01-16T15:28:14.827087+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:28:14.827087+0000","last_undegraded":"2024-01-16T15:28:14.827087+0000","last_fullsized":"2024-01-16T15:28:14.827087+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'32","last_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_deep_scrub":"112'32","last_deep_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.826899+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:16:55.047969+0000","scrub_duration":0.0075779699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.118029+0000","last_change":"2024-01-16T15:28:25.118081+0000","last_active":"2024-01-16T15:28:25.118029+0000","last_peered":"2024-01-16T15:28:25.118029+0000","last_clean":"2024-01-16T15:28:25.118029+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:28:25.118029+0000","last_undegraded":"2024-01-16T15:28:25.118029+0000","last_fullsized":"2024-01-16T15:28:25.118029+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.117955+0000","objects_scrubbed":23,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:56.787030+0000","scrub_duration":0.17487287100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:37.058158+0000","last_change":"2024-01-16T15:28:37.058260+0000","last_active":"2024-01-16T15:28:37.058158+0000","last_peered":"2024-01-16T15:28:37.058158+0000","last_clean":"2024-01-16T15:28:37.058158+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:28:37.058158+0000","last_undegraded":"2024-01-16T15:28:37.058158+0000","last_fullsized":"2024-01-16T15:28:37.058158+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:37.057940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:37.057940+0000","last_clean_scrub_stamp":"2024-01-16T15:28:37.057940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:28.720608+0000","scrub_duration":0.0036757579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'68","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.884824+0000","last_change":"2024-01-16T15:28:19.884824+0000","last_active":"2024-01-16T15:28:19.884824+0000","last_peered":"2024-01-16T15:28:19.884824+0000","last_clean":"2024-01-16T15:28:19.884824+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:28:19.884824+0000","last_undegraded":"2024-01-16T15:28:19.884824+0000","last_fullsized":"2024-01-16T15:28:19.884824+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'68","last_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_deep_scrub":"112'68","last_deep_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.884357+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:10.532387+0000","scrub_duration":0.15341920100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.314517+0000","last_change":"2024-01-16T15:28:40.314517+0000","last_active":"2024-01-16T15:28:40.314517+0000","last_peered":"2024-01-16T15:28:40.314517+0000","last_clean":"2024-01-16T15:28:40.314517+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:28:40.314517+0000","last_undegraded":"2024-01-16T15:28:40.314517+0000","last_fullsized":"2024-01-16T15:28:40.314517+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:40.314218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:40.314218+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.314218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:28.132750+0000","scrub_duration":0.0055196840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"112'119","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.585009+0000","last_change":"2024-01-16T15:28:18.585089+0000","last_active":"2024-01-16T15:28:18.585009+0000","last_peered":"2024-01-16T15:28:18.585009+0000","last_clean":"2024-01-16T15:28:18.585009+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:28:18.585009+0000","last_undegraded":"2024-01-16T15:28:18.585009+0000","last_fullsized":"2024-01-16T15:28:18.585009+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'119","last_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_deep_scrub":"112'119","last_deep_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.584815+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:01.708690+0000","scrub_duration":0.06564681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5526,"ondisk_log_size":5526,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201384,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":469,"num_write_kb":1410,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2402768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":454,"ondisk_log_size":454,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509794,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1439999999999999,"15min":1.2210000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.593,"5min":1.7749999999999999,"15min":19.224},"last":1.25},{"interface":"front","average":{"1min":1.238,"5min":1.2250000000000001,"15min":1.2070000000000001},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.6140000000000001,"5min":1.694,"15min":5.085},"last":1.0009999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.194,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.374},"max":{"1min":1.641,"5min":1.837,"15min":3.7069999999999999},"last":1.3129999999999999},{"interface":"front","average":{"1min":1.204,"5min":1.2909999999999999,"15min":1.3640000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.536,"5min":1.8819999999999999,"15min":19.178000000000001},"last":1.0720000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3080000000000001,"15min":1.383},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":19.126000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.3200000000000001,"15min":1.3160000000000001},"min":{"1min":1.0669999999999999,"5min":0.88700000000000001,"15min":0.66700000000000004},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":2.9750000000000001},"last":1.4490000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2689999999999999,"15min":1.3200000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.76800000000000002},"max":{"1min":1.6599999999999999,"5min":1.73,"15min":3.7999999999999998},"last":1.141},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4390000000000001,"15min":1.5269999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.63500000000000001},"max":{"1min":1.972,"5min":1.972,"15min":19.268999999999998},"last":1.6819999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2470000000000001,"15min":1.264},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.58799999999999997},"max":{"1min":1.794,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.556},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.369,"15min":1.397},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.58299999999999996},"max":{"1min":2.452,"5min":2.452,"15min":9.7059999999999995},"last":1.421}]},{"osd":5,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5329999999999999,"15min":1.548},"min":{"1min":0.99099999999999999,"5min":0.97599999999999998,"15min":0.79400000000000004},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":3.9129999999999998},"last":1.524},{"interface":"front","average":{"1min":1.514,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.86799999999999999,"5min":0.82699999999999996,"15min":0.55100000000000005},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":19.036999999999999},"last":1.8720000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.456,"15min":1.444},"min":{"1min":1.0069999999999999,"5min":0.88500000000000001,"15min":0.54800000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":4.2240000000000002},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5409999999999999,"15min":1.5720000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.65100000000000002},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":4.0709999999999997},"last":1.6040000000000001}]}]},{"osd":4,"up_from":15,"seq":64424509796,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.3029999999999999,"15min":1.23},"min":{"1min":0.755,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.5900000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.177,"5min":1.194,"15min":1.1879999999999999},"min":{"1min":0.93400000000000005,"5min":0.76100000000000001,"15min":0.69799999999999995},"max":{"1min":1.4930000000000001,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.3580000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.238,"15min":1.2190000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.71899999999999997},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.0510000000000002},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.2909999999999999},"min":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.752,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":0.97799999999999998}]},{"osd":2,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3240000000000001,"15min":1.3160000000000001},"min":{"1min":0.97199999999999998,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.6339999999999999,"5min":1.78,"15min":1.843},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.341,"15min":1.349},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.52600000000000002},"max":{"1min":1.653,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":1.407}]},{"osd":3,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.76200000000000001},"max":{"1min":1.8879999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.377,"5min":1.3839999999999999,"15min":1.391},"min":{"1min":1.056,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.71,"5min":1.74,"15min":2.0299999999999998},"last":1.25}]},{"osd":5,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.427,"15min":1.429},"min":{"1min":0.95199999999999996,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6839999999999999,"5min":6.165,"15min":6.165},"last":1.296},{"interface":"front","average":{"1min":1.502,"5min":1.4450000000000001,"15min":1.474},"min":{"1min":0.95399999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8129999999999999,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.623}]},{"osd":6,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.365,"15min":1.4259999999999999},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.497},"max":{"1min":1.716,"5min":1.903,"15min":8.9600000000000009},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.54,"15min":1.524},"min":{"1min":1.1579999999999999,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.887,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":1.5189999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4339999999999999,"15min":1.4790000000000001},"min":{"1min":1.179,"5min":1.0469999999999999,"15min":0.89700000000000002},"max":{"1min":1.8080000000000001,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":1.7170000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.518,"15min":1.5600000000000001},"min":{"1min":1.0329999999999999,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":1.923,"5min":2.0259999999999998,"15min":3.8540000000000001},"last":1.452}]}]},{"osd":7,"up_from":14,"seq":60129542498,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1896108,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92475732,"statfs":{"total":96636764160,"available":94695149568,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0419999999999998,"5min":1.339,"15min":1.2649999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.65400000000000003},"max":{"1min":16.832999999999998,"5min":16.832999999999998,"15min":16.832999999999998},"last":1.3080000000000001},{"interface":"front","average":{"1min":2.4359999999999999,"5min":1.456,"15min":1.304},"min":{"1min":0.86199999999999999,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":21.283000000000001,"5min":21.283000000000001,"15min":21.283000000000001},"last":1.45}]},{"osd":1,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.254,"15min":1.272},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.61799999999999999},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.6719999999999999},{"interface":"front","average":{"1min":2.2269999999999999,"5min":1.47,"15min":1.395},"min":{"1min":0.90700000000000003,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":16.350000000000001,"5min":16.350000000000001,"15min":16.350000000000001},"last":1.0629999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.5049999999999999,"15min":1.3700000000000001},"min":{"1min":0.82599999999999996,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":16.960999999999999,"5min":16.960999999999999,"15min":16.960999999999999},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3280000000000001,"15min":1.409},"min":{"1min":0.92700000000000005,"5min":0.873,"15min":0.69299999999999995},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":19.763999999999999},"last":1.121}]},{"osd":3,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":1.6339999999999999,"15min":1.4450000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.80600000000000005},"max":{"1min":27.920999999999999,"5min":27.920999999999999,"15min":27.920999999999999},"last":1.6399999999999999},{"interface":"front","average":{"1min":2.4430000000000001,"5min":1.661,"15min":1.538},"min":{"1min":1.04,"5min":1.0169999999999999,"15min":0.76500000000000001},"max":{"1min":17.709,"5min":17.709,"15min":17.709},"last":1.5549999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":2.1779999999999999,"5min":1.5109999999999999,"15min":1.391},"min":{"1min":0.753,"5min":0.753,"15min":0.54100000000000004},"max":{"1min":17.006,"5min":17.006,"15min":17.006},"last":0.98799999999999999},{"interface":"front","average":{"1min":2.1819999999999999,"5min":1.5620000000000001,"15min":1.4710000000000001},"min":{"1min":1.0569999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":17.658000000000001,"5min":17.658000000000001,"15min":17.658000000000001},"last":1.4059999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.425,"15min":1.361},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":7.641,"5min":7.641,"15min":7.641},"last":0.95399999999999996},{"interface":"front","average":{"1min":2.4780000000000002,"5min":1.718,"15min":1.585},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":0.66000000000000003},"max":{"1min":16.911999999999999,"5min":16.911999999999999,"15min":16.911999999999999},"last":1.5820000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.7390000000000001,"15min":1.625},"min":{"1min":1.2370000000000001,"5min":1.0760000000000001,"15min":0.67100000000000004},"max":{"1min":16.510000000000002,"5min":16.510000000000002,"15min":16.510000000000002},"last":1.7669999999999999},{"interface":"front","average":{"1min":2.3170000000000002,"5min":1.681,"15min":1.575},"min":{"1min":1.0289999999999999,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":16.405999999999999,"5min":16.405999999999999,"15min":16.405999999999999},"last":1.355}]}]},{"osd":5,"up_from":14,"seq":60129542500,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.1830000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.32900000000000001},"max":{"1min":1.7070000000000001,"5min":2.0939999999999999,"15min":6.0720000000000001},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.123,"5min":1.1759999999999999,"15min":1.3160000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.57099999999999995},"max":{"1min":1.302,"5min":1.7649999999999999,"15min":23.199999999999999},"last":1.4179999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.21,"15min":1.224},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.63400000000000001},"max":{"1min":1.736,"5min":1.736,"15min":4.2450000000000001},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2649999999999999,"15min":1.327},"min":{"1min":0.76600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.786,"5min":1.851,"15min":17.062000000000001},"last":1.2270000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.367,"15min":1.403},"min":{"1min":0.65300000000000002,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":5.8810000000000002},"last":1.6379999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.96599999999999997,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.921,"5min":1.921,"15min":5.4429999999999996},"last":1.591}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.323,"15min":1.3240000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.67500000000000004},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.4769999999999999},"last":1.256},{"interface":"front","average":{"1min":1.363,"5min":1.397,"15min":1.4179999999999999},"min":{"1min":0.69099999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.742,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":1.323}]},{"osd":4,"last update":"Tue Jan 16 15:28:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.363,"15min":1.3939999999999999},"min":{"1min":0.88300000000000001,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":2.2610000000000001,"15min":15.975},"last":1.173},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.417,"15min":1.4179999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.54000000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":5.6349999999999998},"last":1.119}]},{"osd":6,"last update":"Tue Jan 16 15:28:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.335,"15min":1.369},"min":{"1min":0.96999999999999997,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":12.914},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4950000000000001,"15min":1.5209999999999999},"min":{"1min":1.0780000000000001,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":1.9990000000000001,"5min":2.4660000000000002,"15min":16.552},"last":1.3720000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.4750000000000001,"15min":1.51},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.64300000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":13.034000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.669,"5min":1.5900000000000001,"15min":1.5760000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.70899999999999996},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":1.5469999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542500,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95799999999999996,"15min":0.95699999999999996},"min":{"1min":0.56599999999999995,"5min":0.373,"15min":0.315},"max":{"1min":1.3,"5min":1.6910000000000001,"15min":2.02},"last":0.79000000000000004},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.125,"15min":1.147},"min":{"1min":0.79000000000000004,"5min":0.48299999999999998,"15min":0.42799999999999999},"max":{"1min":1.5720000000000001,"5min":1.879,"15min":7.1429999999999998},"last":0.93600000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.98899999999999999,"15min":1.044},"min":{"1min":0.77500000000000002,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.4990000000000001,"5min":1.5640000000000001,"15min":10.856999999999999},"last":0.85099999999999998},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2050000000000001,"15min":1.194},"min":{"1min":0.66400000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.5589999999999999,"5min":1.823,"15min":2.0089999999999999},"last":1.1659999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.306,"15min":1.387},"min":{"1min":0.65100000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.641,"5min":1.946,"15min":10.045999999999999},"last":1.349},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1599999999999999,"15min":1.21},"min":{"1min":0.627,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.97,"5min":1.97,"15min":7.9160000000000004},"last":1.4670000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.343,"15min":1.3620000000000001},"min":{"1min":1.01,"5min":0.748,"15min":0.73799999999999999},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":4.5570000000000004},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.476,"15min":1.484},"min":{"1min":1.0900000000000001,"5min":0.749,"15min":0.749},"max":{"1min":1.946,"5min":2.0920000000000001,"15min":2.165},"last":1.619}]},{"osd":5,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.3759999999999999,"15min":1.458},"min":{"1min":1.097,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.3660000000000001,"5min":2.3660000000000001,"15min":14.003},"last":1.266},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5109999999999999,"15min":1.546},"min":{"1min":0.879,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.2970000000000002},"last":1.5029999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.4610000000000001,"15min":1.5009999999999999},"min":{"1min":0.96199999999999997,"5min":0.875,"15min":0.85299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":7.0060000000000002},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.512,"15min":1.5740000000000001},"min":{"1min":1.252,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":15.114000000000001},"last":1.4390000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.494,"15min":1.6220000000000001},"min":{"1min":1.248,"5min":0.85699999999999998,"15min":0.81899999999999995},"max":{"1min":2.1139999999999999,"5min":2.1349999999999998,"15min":12.378},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.806,"5min":1.617,"15min":1.6339999999999999},"min":{"1min":1.4690000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.48,"5min":2.48,"15min":4.6029999999999998},"last":1.5589999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542499,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1064088,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93307752,"statfs":{"total":96636764160,"available":95547138048,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91300000000000003,"15min":0.97699999999999998},"min":{"1min":0.626,"5min":0.46200000000000002,"15min":0.32400000000000001},"max":{"1min":1.3720000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.73299999999999998},{"interface":"front","average":{"1min":1.034,"5min":1.139,"15min":1.121},"min":{"1min":0.63300000000000001,"5min":0.49199999999999999,"15min":0.40300000000000002},"max":{"1min":1.708,"5min":1.913,"15min":7.9960000000000004},"last":0.92700000000000005}]},{"osd":1,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1120000000000001,"15min":1.1020000000000001},"min":{"1min":0.66800000000000004,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":2.145,"5min":2.294,"15min":2.294},"last":1.498},{"interface":"front","average":{"1min":1.2,"5min":1.1799999999999999,"15min":1.1819999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48399999999999999},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":4.3819999999999997},"last":1.232}]},{"osd":3,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.171,"15min":1.2789999999999999},"min":{"1min":0.63700000000000001,"5min":0.61199999999999999,"15min":0.44700000000000001},"max":{"1min":1.804,"5min":1.883,"15min":18.838000000000001},"last":1.7250000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4830000000000001,"15min":1.554},"min":{"1min":0.79300000000000004,"5min":0.66200000000000003,"15min":0.58099999999999996},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":19.375},"last":1.5549999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.327,"15min":1.429},"min":{"1min":0.80800000000000005,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.754,"5min":2.242,"15min":23.225000000000001},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.365,"5min":1.466,"15min":1.484},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.72699999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":6.359},"last":1.45}]},{"osd":5,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4390000000000001,"15min":1.494},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.68100000000000005},"max":{"1min":2.165,"5min":2.3170000000000002,"15min":20.157},"last":1.3999999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.5589999999999999,"15min":1.593},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.177}]},{"osd":6,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5069999999999999,"15min":1.5900000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.83599999999999997},"max":{"1min":2.0579999999999998,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.302},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.595,"15min":1.7130000000000001},"min":{"1min":0.99199999999999999,"5min":0.85799999999999998,"15min":0.76200000000000001},"max":{"1min":2.238,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.6040000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5329999999999999,"15min":1.6579999999999999},"min":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":1.9930000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.8340000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.583,"15min":1.587},"min":{"1min":1.1259999999999999,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":4.3419999999999996},"last":1.3600000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542501,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0660000000000001,"15min":0.98699999999999999},"min":{"1min":0.71599999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.296,"5min":13.42,"15min":13.42},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1359999999999999,"15min":1.123},"min":{"1min":0.50900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.6339999999999999,"5min":3.915,"15min":3.915},"last":1.2829999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0269999999999999,"15min":1.0900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.53,"5min":1.53,"15min":3.6819999999999999},"last":0.873},{"interface":"front","average":{"1min":1.284,"5min":1.2729999999999999,"15min":1.335},"min":{"1min":0.64100000000000001,"5min":0.46999999999999997,"15min":0.40100000000000002},"max":{"1min":1.736,"5min":1.736,"15min":3.5600000000000001},"last":1.415}]},{"osd":3,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6950000000000001,"5min":1.875,"15min":2.238},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.133,"5min":1.127,"15min":1.1919999999999999},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.7509999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":1.4750000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2549999999999999,"15min":1.327},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6739999999999999,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.417,"5min":1.544,"15min":1.528},"min":{"1min":1.077,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.679,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.5629999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.419,"15min":1.4450000000000001},"min":{"1min":1.095,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.98},"last":2.2490000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.417,"15min":1.4530000000000001},"min":{"1min":0.97199999999999998,"5min":0.92900000000000005,"15min":0.84499999999999997},"max":{"1min":1.78,"5min":1.893,"15min":3.3940000000000001},"last":1.605}]},{"osd":6,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.5089999999999999,"15min":1.583},"min":{"1min":1.2549999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":16.48},"last":2.1080000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4930000000000001,"15min":1.536},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.90000000000000002},"max":{"1min":1.978,"5min":1.978,"15min":3.8180000000000001},"last":1.7130000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.5389999999999999,"15min":1.613},"min":{"1min":0.94199999999999995,"5min":0.92200000000000004,"15min":0.68899999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":7.0499999999999998},"last":1.8049999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.607,"15min":1.6499999999999999},"min":{"1min":1.0920000000000001,"5min":0.91700000000000004,"15min":0.89900000000000002},"max":{"1min":1.931,"5min":2.258,"15min":3.7839999999999998},"last":1.6319999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542499,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.93500000000000005,"15min":1.0449999999999999},"min":{"1min":0.69199999999999995,"5min":0.496,"15min":0.376},"max":{"1min":1.5509999999999999,"5min":1.665,"15min":15.776},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.1559999999999999,"15min":1.1739999999999999},"min":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":17.620999999999999},"last":0.85899999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.135,"15min":1.226},"min":{"1min":0.82899999999999996,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.629,"5min":1.671,"15min":15.997999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.246,"5min":1.2430000000000001,"15min":1.2030000000000001},"min":{"1min":0.89200000000000002,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.155}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1080000000000001,"15min":1.179},"min":{"1min":0.77900000000000003,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.7190000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":1.0029999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.345,"15min":1.4339999999999999},"min":{"1min":0.86599999999999999,"5min":0.66600000000000004,"15min":0.51200000000000001},"max":{"1min":1.6459999999999999,"5min":1.827,"15min":16.126000000000001},"last":1.552}]},{"osd":4,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4750000000000001,"15min":1.5740000000000001},"min":{"1min":1.0389999999999999,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.8640000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.508,"5min":1.46,"15min":1.4279999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":3.5219999999999998,"5min":3.5219999999999998,"15min":3.5219999999999998},"last":1.5129999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.736,"5min":2.157,"15min":5.5609999999999999},"last":1.7110000000000001},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.5609999999999999,"15min":1.649},"min":{"1min":1.157,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":16.501999999999999},"last":1.2949999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4390000000000001,"15min":1.4510000000000001},"min":{"1min":1.0569999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.7370000000000001,"5min":2.0499999999999998,"15min":5.4509999999999996},"last":1.7809999999999999},{"interface":"front","average":{"1min":1.601,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":1.2470000000000001,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":1.8979999999999999,"5min":2.351,"15min":16.395},"last":1.4590000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.591,"15min":1.5780000000000001},"min":{"1min":1.22,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":3.423,"5min":3.423,"15min":3.8239999999999998},"last":1.6799999999999999},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.579,"15min":1.5860000000000001},"min":{"1min":1.1419999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.0600000000000001,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.8380000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1196953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1196911,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:29:12.732 INFO:tasks.ceph:pgid 17.3f last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:29:12.733 INFO:tasks.ceph:pgid 17.37 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:29:12.733 INFO:tasks.ceph:pgid 17.36 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:29:12.733 INFO:tasks.ceph:pgid 17.26 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:29:12.734 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:29:12.734 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-01-16T15:19:23.577505+0000 time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=19, tm_sec=23, tm_wday=1, tm_yday=16, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=1, tm_mday=16, tm_hour=15, tm_min=28, tm_sec=3, tm_wday=1, tm_yday=16, tm_isdst=0) 2024-01-16T15:29:12.734 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-01-16T15:29:30.143 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:30.151 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:29:32.735 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-01-16T15:29:32.964 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.965+0000 7ff6c8a00640 1 Processor -- start 2024-01-16T15:29:32.965 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.965+0000 7ff6c8a00640 1 -- start start 2024-01-16T15:29:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.966+0000 7ff6c8a00640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7ff6c4143c40 0x7ff6c4071290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.966+0000 7ff6c8a00640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7ff6c40717d0 0x7ff6c40cc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:32.967 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c8a00640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6c4070e50 con 0x7ff6c40cca40 2024-01-16T15:29:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c8a00640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c40723c0 con 0x7ff6c40717d0 2024-01-16T15:29:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c8a00640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6c40cce40 con 0x7ff6c4143c40 2024-01-16T15:29:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c3e00640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7ff6c40cca40 0x7ff6c415eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59036/0 (socket says 172.21.15.53:59036) 2024-01-16T15:29:32.968 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c3e00640 1 -- 172.21.15.53:0/3854296904 learned_addr learned my addr 172.21.15.53:0/3854296904 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6ba800640 1 --2- 172.21.15.53:0/3854296904 >> v2:172.21.15.53:3300/0 conn(0x7ff6c40717d0 0x7ff6c40cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c3400640 1 --2- 172.21.15.53:0/3854296904 >> v2:172.21.15.146:3300/0 conn(0x7ff6c4143c40 0x7ff6c4071290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c2a00640 1 -- 172.21.15.53:0/3854296904 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400892762 0 0) 0x7ff6c4070e50 con 0x7ff6c40cca40 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.967+0000 7ff6c2a00640 1 -- 172.21.15.53:0/3854296904 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6a4003520 con 0x7ff6c40cca40 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.968+0000 7ff6ba800640 1 -- 172.21.15.53:0/3854296904 >> v2:172.21.15.146:3300/0 conn(0x7ff6c4143c40 msgr2=0x7ff6c4071290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.968+0000 7ff6ba800640 1 --2- 172.21.15.53:0/3854296904 >> v2:172.21.15.146:3300/0 conn(0x7ff6c4143c40 0x7ff6c4071290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:32.969 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.968+0000 7ff6ba800640 1 -- 172.21.15.53:0/3854296904 >> v1:172.21.15.53:6789/0 conn(0x7ff6c40cca40 legacy=0x7ff6c415eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:32.970 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.968+0000 7ff6ba800640 1 -- 172.21.15.53:0/3854296904 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6c406e170 con 0x7ff6c40717d0 2024-01-16T15:29:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.969+0000 7ff6ba800640 1 --2- 172.21.15.53:0/3854296904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40717d0 0x7ff6c40cc500 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7ff6c406e410 tx=0x7ff6ac02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d29373b352d7b4e0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.969+0000 7ff6c2a00640 1 -- 172.21.15.53:0/3854296904 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac02e070 con 0x7ff6c40717d0 2024-01-16T15:29:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.969+0000 7ff6c2a00640 1 -- 172.21.15.53:0/3854296904 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac0315a0 con 0x7ff6c40717d0 2024-01-16T15:29:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.969+0000 7ff6c2a00640 1 -- 172.21.15.53:0/3854296904 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac031930 con 0x7ff6c40717d0 2024-01-16T15:29:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 -- 172.21.15.53:0/3854296904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40717d0 msgr2=0x7ff6c40cc500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:32.971 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/3854296904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40717d0 0x7ff6c40cc500 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7ff6c406e410 tx=0x7ff6ac02b840 comp rx=0 tx=0).stop 2024-01-16T15:29:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 -- 172.21.15.53:0/3854296904 shutdown_connections 2024-01-16T15:29:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/3854296904 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40717d0 0x7ff6c40cc500 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/3854296904 >> v2:172.21.15.146:3300/0 conn(0x7ff6c4143c40 0x7ff6c4071290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 -- 172.21.15.53:0/3854296904 >> 172.21.15.53:0/3854296904 conn(0x7ff6c406a800 msgr2=0x7ff6c406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 -- 172.21.15.53:0/3854296904 shutdown_connections 2024-01-16T15:29:32.972 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.970+0000 7ff6c8a00640 1 -- 172.21.15.53:0/3854296904 wait complete. 2024-01-16T15:29:32.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.972+0000 7ff6c8a00640 1 Processor -- start 2024-01-16T15:29:32.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.972+0000 7ff6c8a00640 1 -- start start 2024-01-16T15:29:32.974 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c8a00640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c40717d0 0x7ff6c40a84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c8a00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 0x7ff6c40a9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c8a00640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c4143c40 0x7ff6c4137f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c8a00640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c41385e0 con 0x7ff6c40cca40 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c8a00640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c41388c0 con 0x7ff6c40717d0 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c8a00640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c4138ba0 con 0x7ff6c4143c40 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 0x7ff6c40a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 0x7ff6c40a9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59622/0 (socket says 172.21.15.53:59622) 2024-01-16T15:29:32.975 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 -- 172.21.15.53:0/1916744903 learned_addr learned my addr 172.21.15.53:0/1916744903 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6c3400640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c40717d0 0x7ff6c40a84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 -- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c4143c40 msgr2=0x7ff6c4137f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c4143c40 0x7ff6c4137f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 -- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c40717d0 msgr2=0x7ff6c40a84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c40717d0 0x7ff6c40a84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.973+0000 7ff6ba800640 1 -- 172.21.15.53:0/1916744903 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6ac009400 con 0x7ff6c40cca40 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.974+0000 7ff6ba800640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 0x7ff6c40a9590 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac009890 tx=0x7ff6ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.974+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac032840 con 0x7ff6c40cca40 2024-01-16T15:29:32.976 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.974+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ac031f00 con 0x7ff6c40cca40 2024-01-16T15:29:32.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.974+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ac032be0 con 0x7ff6c40cca40 2024-01-16T15:29:32.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.974+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c4138e80 con 0x7ff6c40cca40 2024-01-16T15:29:32.977 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.974+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c413ef50 con 0x7ff6c40cca40 2024-01-16T15:29:32.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.976+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c4139620 con 0x7ff6c40cca40 2024-01-16T15:29:32.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.976+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7ff6ac032d40 con 0x7ff6c40cca40 2024-01-16T15:29:32.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.977+0000 7ff6bbe00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff69c06c4f0 0x7ff69c06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:32.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.977+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7ff6ac0aa9f0 con 0x7ff6c40cca40 2024-01-16T15:29:32.978 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.978+0000 7ff6c3400640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff69c06c4f0 0x7ff69c06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:32.979 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.979+0000 7ff6c3400640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff69c06c4f0 0x7ff69c06e9b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4006fd0 tx=0x7ff6b4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:32.982 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:32.983+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7ff6ac0e0e90 con 0x7ff6c40cca40 2024-01-16T15:29:33.175 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.175+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 --> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff6c4138a30 con 0x7ff69c06c4f0 2024-01-16T15:29:33.197 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.198+0000 7ff6bbe00640 1 -- 172.21.15.53:0/1916744903 <== mgr.4099 v2:172.21.15.146:6832/2200557793 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+291167 (secure 0 0 0) 0x7ff6c4138a30 con 0x7ff69c06c4f0 2024-01-16T15:29:33.201 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:29:33.203 INFO:teuthology.orchestra.run.smithi053.stderr:dumped all 2024-01-16T15:29:33.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff69c06c4f0 msgr2=0x7ff69c06e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff69c06c4f0 0x7ff69c06e9b0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4006fd0 tx=0x7ff6b4008040 comp rx=0 tx=0).stop 2024-01-16T15:29:33.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 msgr2=0x7ff6c40a9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 0x7ff6c40a9590 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7ff6ac009890 tx=0x7ff6ac0046c0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.205 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 shutdown_connections 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7ff69c06c4f0 0x7ff69c06e9b0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7ff6c4143c40 0x7ff6c4137f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7ff6c40cca40 0x7ff6c40a9590 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 --2- 172.21.15.53:0/1916744903 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7ff6c40717d0 0x7ff6c40a84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 >> 172.21.15.53:0/1916744903 conn(0x7ff6c406a800 msgr2=0x7ff6c406abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.205+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 shutdown_connections 2024-01-16T15:29:33.206 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.206+0000 7ff6c8a00640 1 -- 172.21.15.53:0/1916744903 wait complete. 2024-01-16T15:29:33.226 INFO:teuthology.orchestra.run.smithi053.stdout:{"pg_ready":true,"pg_map":{"version":999,"stamp":"2024-01-16T15:29:32.944761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5370503073,"num_objects":1315,"num_object_clones":1,"num_object_copies":2630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1315,"num_whiteouts":1,"num_read":689,"num_read_kb":2097313,"num_write":1980,"num_write_kb":5246747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6223,"ondisk_log_size":6223,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":11133624,"kb_used_data":10500216,"kb_used_omap":12,"kb_used_meta":633331,"kb_avail":743841096,"statfs":{"total":773094113280,"available":761693282304,"internally_reserved":0,"allocated":10752221184,"data_stored":10745814074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":648531504},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004870"},"pg_stats":[{"pgid":"17.3f","version":"112'91","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:24.240373+0000","last_change":"2024-01-16T15:29:24.240455+0000","last_active":"2024-01-16T15:29:24.240373+0000","last_peered":"2024-01-16T15:29:24.240373+0000","last_clean":"2024-01-16T15:29:24.240373+0000","last_became_active":"2024-01-16T15:19:24.627894+0000","last_became_peered":"2024-01-16T15:19:24.627894+0000","last_unstale":"2024-01-16T15:29:24.240373+0000","last_undegraded":"2024-01-16T15:29:24.240373+0000","last_fullsized":"2024-01-16T15:29:24.240373+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'91","last_scrub_stamp":"2024-01-16T15:29:24.240195+0000","last_deep_scrub":"112'91","last_deep_scrub_stamp":"2024-01-16T15:29:24.240195+0000","last_clean_scrub_stamp":"2024-01-16T15:29:24.240195+0000","objects_scrubbed":20,"log_size":91,"log_dups_size":0,"ondisk_log_size":91,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:28:46.028521+0000","scrub_duration":0.22543665099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"112'97","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.510978+0000","last_change":"2024-01-16T15:28:18.511076+0000","last_active":"2024-01-16T15:28:18.510978+0000","last_peered":"2024-01-16T15:28:18.510978+0000","last_clean":"2024-01-16T15:28:18.510978+0000","last_became_active":"2024-01-16T15:19:24.621990+0000","last_became_peered":"2024-01-16T15:19:24.621990+0000","last_unstale":"2024-01-16T15:28:18.510978+0000","last_undegraded":"2024-01-16T15:28:18.510978+0000","last_fullsized":"2024-01-16T15:28:18.510978+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'97","last_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_deep_scrub":"112'97","last_deep_scrub_stamp":"2024-01-16T15:28:18.510802+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.510802+0000","objects_scrubbed":24,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:33.903169+0000","scrub_duration":0.27194245,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":98305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"112'87","reported_seq":121,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:23.860238+0000","last_change":"2024-01-16T15:28:23.860329+0000","last_active":"2024-01-16T15:28:23.860238+0000","last_peered":"2024-01-16T15:28:23.860238+0000","last_clean":"2024-01-16T15:28:23.860238+0000","last_became_active":"2024-01-16T15:19:24.615755+0000","last_became_peered":"2024-01-16T15:19:24.615755+0000","last_unstale":"2024-01-16T15:28:23.860238+0000","last_undegraded":"2024-01-16T15:28:23.860238+0000","last_fullsized":"2024-01-16T15:28:23.860238+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:23.860048+0000","last_clean_scrub_stamp":"2024-01-16T15:28:23.860048+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:22:49.716426+0000","scrub_duration":0.27335353899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":26,"num_write_kb":90114,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"112'71","reported_seq":104,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:10.256523+0000","last_change":"2024-01-16T15:29:10.256523+0000","last_active":"2024-01-16T15:29:10.256523+0000","last_peered":"2024-01-16T15:29:10.256523+0000","last_clean":"2024-01-16T15:29:10.256523+0000","last_became_active":"2024-01-16T15:19:24.634664+0000","last_became_peered":"2024-01-16T15:19:24.634664+0000","last_unstale":"2024-01-16T15:29:10.256523+0000","last_undegraded":"2024-01-16T15:29:10.256523+0000","last_fullsized":"2024-01-16T15:29:10.256523+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'71","last_scrub_stamp":"2024-01-16T15:29:10.256271+0000","last_deep_scrub":"112'71","last_deep_scrub_stamp":"2024-01-16T15:29:10.256271+0000","last_clean_scrub_stamp":"2024-01-16T15:29:10.256271+0000","objects_scrubbed":15,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:04:51.152173+0000","scrub_duration":0.17038672199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62914560,"num_objects":15,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":15,"num_write_kb":61440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"112'78","reported_seq":111,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.070176+0000","last_change":"2024-01-16T15:28:25.070176+0000","last_active":"2024-01-16T15:28:25.070176+0000","last_peered":"2024-01-16T15:28:25.070176+0000","last_clean":"2024-01-16T15:28:25.070176+0000","last_became_active":"2024-01-16T15:19:24.625825+0000","last_became_peered":"2024-01-16T15:19:24.625825+0000","last_unstale":"2024-01-16T15:28:25.070176+0000","last_undegraded":"2024-01-16T15:28:25.070176+0000","last_fullsized":"2024-01-16T15:28:25.070176+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'78","last_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_deep_scrub":"112'78","last_deep_scrub_stamp":"2024-01-16T15:28:25.070058+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.070058+0000","objects_scrubbed":16,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:36.495682+0000","scrub_duration":0.150879281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"112'99","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.440281+0000","last_change":"2024-01-16T15:28:27.440281+0000","last_active":"2024-01-16T15:28:27.440281+0000","last_peered":"2024-01-16T15:28:27.440281+0000","last_clean":"2024-01-16T15:28:27.440281+0000","last_became_active":"2024-01-16T15:19:24.625506+0000","last_became_peered":"2024-01-16T15:19:24.625506+0000","last_unstale":"2024-01-16T15:28:27.440281+0000","last_undegraded":"2024-01-16T15:28:27.440281+0000","last_fullsized":"2024-01-16T15:28:27.440281+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:27.439995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.439995+0000","objects_scrubbed":23,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:38:38.865619+0000","scrub_duration":0.25882440899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"112'88","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:43.589452+0000","last_change":"2024-01-16T15:28:43.589452+0000","last_active":"2024-01-16T15:28:43.589452+0000","last_peered":"2024-01-16T15:28:43.589452+0000","last_clean":"2024-01-16T15:28:43.589452+0000","last_became_active":"2024-01-16T15:19:24.625528+0000","last_became_peered":"2024-01-16T15:19:24.625528+0000","last_unstale":"2024-01-16T15:28:43.589452+0000","last_undegraded":"2024-01-16T15:28:43.589452+0000","last_fullsized":"2024-01-16T15:28:43.589452+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:43.589215+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:43.589215+0000","last_clean_scrub_stamp":"2024-01-16T15:28:43.589215+0000","objects_scrubbed":21,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:51:23.823898+0000","scrub_duration":0.25936774699999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"112'79","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.800039+0000","last_change":"2024-01-16T15:28:24.800039+0000","last_active":"2024-01-16T15:28:24.800039+0000","last_peered":"2024-01-16T15:28:24.800039+0000","last_clean":"2024-01-16T15:28:24.800039+0000","last_became_active":"2024-01-16T15:19:24.616472+0000","last_became_peered":"2024-01-16T15:19:24.616472+0000","last_unstale":"2024-01-16T15:28:24.800039+0000","last_undegraded":"2024-01-16T15:28:24.800039+0000","last_fullsized":"2024-01-16T15:28:24.800039+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:24.799761+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.799761+0000","objects_scrubbed":21,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:31:58.738844+0000","scrub_duration":0.232299336,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"112'83","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:25.145766+0000","last_change":"2024-01-16T15:29:25.145852+0000","last_active":"2024-01-16T15:29:25.145766+0000","last_peered":"2024-01-16T15:29:25.145766+0000","last_clean":"2024-01-16T15:29:25.145766+0000","last_became_active":"2024-01-16T15:19:24.636933+0000","last_became_peered":"2024-01-16T15:19:24.636933+0000","last_unstale":"2024-01-16T15:29:25.145766+0000","last_undegraded":"2024-01-16T15:29:25.145766+0000","last_fullsized":"2024-01-16T15:29:25.145766+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'83","last_scrub_stamp":"2024-01-16T15:29:25.145647+0000","last_deep_scrub":"112'83","last_deep_scrub_stamp":"2024-01-16T15:29:25.145647+0000","last_clean_scrub_stamp":"2024-01-16T15:29:25.145647+0000","objects_scrubbed":13,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:36:19.858918+0000","scrub_duration":0.13017868499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331909,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":13,"num_write_kb":49153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"112'90","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:18.313754+0000","last_change":"2024-01-16T15:29:18.313831+0000","last_active":"2024-01-16T15:29:18.313754+0000","last_peered":"2024-01-16T15:29:18.313754+0000","last_clean":"2024-01-16T15:29:18.313754+0000","last_became_active":"2024-01-16T15:19:24.627390+0000","last_became_peered":"2024-01-16T15:19:24.627390+0000","last_unstale":"2024-01-16T15:29:18.313754+0000","last_undegraded":"2024-01-16T15:29:18.313754+0000","last_fullsized":"2024-01-16T15:29:18.313754+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:29:18.313626+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:29:18.313626+0000","last_clean_scrub_stamp":"2024-01-16T15:29:18.313626+0000","objects_scrubbed":20,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:48:58.055152+0000","scrub_duration":0.208244507,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":40960,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"112'93","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.541954+0000","last_change":"2024-01-16T15:28:26.541954+0000","last_active":"2024-01-16T15:28:26.541954+0000","last_peered":"2024-01-16T15:28:26.541954+0000","last_clean":"2024-01-16T15:28:26.541954+0000","last_became_active":"2024-01-16T15:19:24.624847+0000","last_became_peered":"2024-01-16T15:19:24.624847+0000","last_unstale":"2024-01-16T15:28:26.541954+0000","last_undegraded":"2024-01-16T15:28:26.541954+0000","last_fullsized":"2024-01-16T15:28:26.541954+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:26.541809+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.541809+0000","objects_scrubbed":23,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:28:57.056238+0000","scrub_duration":0.29071314399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"112'88","reported_seq":130,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:08.322242+0000","last_change":"2024-01-16T15:29:08.322329+0000","last_active":"2024-01-16T15:29:08.322242+0000","last_peered":"2024-01-16T15:29:08.322242+0000","last_clean":"2024-01-16T15:29:08.322242+0000","last_became_active":"2024-01-16T15:19:24.622786+0000","last_became_peered":"2024-01-16T15:19:24.622786+0000","last_unstale":"2024-01-16T15:29:08.322242+0000","last_undegraded":"2024-01-16T15:29:08.322242+0000","last_fullsized":"2024-01-16T15:29:08.322242+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:29:08.322077+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:29:08.322077+0000","last_clean_scrub_stamp":"2024-01-16T15:29:08.322077+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:13:17.381831+0000","scrub_duration":0.24970814399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":16,"num_read_kb":61441,"num_write":28,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"112'87","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:36.975322+0000","last_change":"2024-01-16T15:28:36.975322+0000","last_active":"2024-01-16T15:28:36.975322+0000","last_peered":"2024-01-16T15:28:36.975322+0000","last_clean":"2024-01-16T15:28:36.975322+0000","last_became_active":"2024-01-16T15:19:24.618301+0000","last_became_peered":"2024-01-16T15:19:24.618301+0000","last_unstale":"2024-01-16T15:28:36.975322+0000","last_undegraded":"2024-01-16T15:28:36.975322+0000","last_fullsized":"2024-01-16T15:28:36.975322+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'87","last_scrub_stamp":"2024-01-16T15:28:36.974986+0000","last_deep_scrub":"112'87","last_deep_scrub_stamp":"2024-01-16T15:28:36.974986+0000","last_clean_scrub_stamp":"2024-01-16T15:28:36.974986+0000","objects_scrubbed":22,"log_size":87,"log_dups_size":0,"ondisk_log_size":87,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:52:42.637017+0000","scrub_duration":0.26518550499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"112'94","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.914459+0000","last_change":"2024-01-16T15:28:21.914459+0000","last_active":"2024-01-16T15:28:21.914459+0000","last_peered":"2024-01-16T15:28:21.914459+0000","last_clean":"2024-01-16T15:28:21.914459+0000","last_became_active":"2024-01-16T15:19:24.624030+0000","last_became_peered":"2024-01-16T15:19:24.624030+0000","last_unstale":"2024-01-16T15:28:21.914459+0000","last_undegraded":"2024-01-16T15:28:21.914459+0000","last_fullsized":"2024-01-16T15:28:21.914459+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:21.914080+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.914080+0000","objects_scrubbed":22,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:39:00.078053+0000","scrub_duration":0.280246104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"112'71","reported_seq":101,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.408300+0000","last_change":"2024-01-16T15:28:24.408300+0000","last_active":"2024-01-16T15:28:24.408300+0000","last_peered":"2024-01-16T15:28:24.408300+0000","last_clean":"2024-01-16T15:28:24.408300+0000","last_became_active":"2024-01-16T15:19:24.612484+0000","last_became_peered":"2024-01-16T15:19:24.612484+0000","last_unstale":"2024-01-16T15:28:24.408300+0000","last_undegraded":"2024-01-16T15:28:24.408300+0000","last_fullsized":"2024-01-16T15:28:24.408300+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'71","last_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_deep_scrub":"112'71","last_deep_scrub_stamp":"2024-01-16T15:28:24.408030+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.408030+0000","objects_scrubbed":12,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:51.797494+0000","scrub_duration":0.124741884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"112'99","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.255207+0000","last_change":"2024-01-16T15:28:18.255207+0000","last_active":"2024-01-16T15:28:18.255207+0000","last_peered":"2024-01-16T15:28:18.255207+0000","last_clean":"2024-01-16T15:28:18.255207+0000","last_became_active":"2024-01-16T15:19:24.614377+0000","last_became_peered":"2024-01-16T15:19:24.614377+0000","last_unstale":"2024-01-16T15:28:18.255207+0000","last_undegraded":"2024-01-16T15:28:18.255207+0000","last_fullsized":"2024-01-16T15:28:18.255207+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:28:18.254974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.254974+0000","objects_scrubbed":25,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:57.777992+0000","scrub_duration":0.28941331399999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"112'65","reported_seq":96,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.443280+0000","last_change":"2024-01-16T15:28:29.443280+0000","last_active":"2024-01-16T15:28:29.443280+0000","last_peered":"2024-01-16T15:28:29.443280+0000","last_clean":"2024-01-16T15:28:29.443280+0000","last_became_active":"2024-01-16T15:19:24.624940+0000","last_became_peered":"2024-01-16T15:19:24.624940+0000","last_unstale":"2024-01-16T15:28:29.443280+0000","last_undegraded":"2024-01-16T15:28:29.443280+0000","last_fullsized":"2024-01-16T15:28:29.443280+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'65","last_scrub_stamp":"2024-01-16T15:28:29.443041+0000","last_deep_scrub":"112'65","last_deep_scrub_stamp":"2024-01-16T15:28:29.443041+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.443041+0000","objects_scrubbed":13,"log_size":65,"log_dups_size":0,"ondisk_log_size":65,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:59:23.604246+0000","scrub_duration":0.17569420299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"112'108","reported_seq":152,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.741028+0000","last_change":"2024-01-16T15:28:26.741028+0000","last_active":"2024-01-16T15:28:26.741028+0000","last_peered":"2024-01-16T15:28:26.741028+0000","last_clean":"2024-01-16T15:28:26.741028+0000","last_became_active":"2024-01-16T15:19:24.628639+0000","last_became_peered":"2024-01-16T15:19:24.628639+0000","last_unstale":"2024-01-16T15:28:26.741028+0000","last_undegraded":"2024-01-16T15:28:26.741028+0000","last_fullsized":"2024-01-16T15:28:26.741028+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:26.740929+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.740929+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:24:29.967191+0000","scrub_duration":0.29640463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":17,"num_read_kb":53252,"num_write":30,"num_write_kb":106498,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"112'100","reported_seq":132,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.925516+0000","last_change":"2024-01-16T15:28:28.925578+0000","last_active":"2024-01-16T15:28:28.925516+0000","last_peered":"2024-01-16T15:28:28.925516+0000","last_clean":"2024-01-16T15:28:28.925516+0000","last_became_active":"2024-01-16T15:19:24.618512+0000","last_became_peered":"2024-01-16T15:19:24.618512+0000","last_unstale":"2024-01-16T15:28:28.925516+0000","last_undegraded":"2024-01-16T15:28:28.925516+0000","last_fullsized":"2024-01-16T15:28:28.925516+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'100","last_scrub_stamp":"2024-01-16T15:28:28.925340+0000","last_deep_scrub":"112'100","last_deep_scrub_stamp":"2024-01-16T15:28:28.925340+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.925340+0000","objects_scrubbed":26,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:24:04.310360+0000","scrub_duration":0.30850802500000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"112'90","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.497303+0000","last_change":"2024-01-16T15:28:28.497397+0000","last_active":"2024-01-16T15:28:28.497303+0000","last_peered":"2024-01-16T15:28:28.497303+0000","last_clean":"2024-01-16T15:28:28.497303+0000","last_became_active":"2024-01-16T15:19:24.620129+0000","last_became_peered":"2024-01-16T15:19:24.620129+0000","last_unstale":"2024-01-16T15:28:28.497303+0000","last_undegraded":"2024-01-16T15:28:28.497303+0000","last_fullsized":"2024-01-16T15:28:28.497303+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:28.497088+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:28.497088+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.497088+0000","objects_scrubbed":20,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:04:34.899596+0000","scrub_duration":0.25302828599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"112'95","reported_seq":134,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.497418+0000","last_change":"2024-01-16T15:28:32.497568+0000","last_active":"2024-01-16T15:28:32.497418+0000","last_peered":"2024-01-16T15:28:32.497418+0000","last_clean":"2024-01-16T15:28:32.497418+0000","last_became_active":"2024-01-16T15:19:24.621251+0000","last_became_peered":"2024-01-16T15:19:24.621251+0000","last_unstale":"2024-01-16T15:28:32.497418+0000","last_undegraded":"2024-01-16T15:28:32.497418+0000","last_fullsized":"2024-01-16T15:28:32.497418+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'95","last_scrub_stamp":"2024-01-16T15:28:32.497207+0000","last_deep_scrub":"112'95","last_deep_scrub_stamp":"2024-01-16T15:28:32.497207+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.497207+0000","objects_scrubbed":22,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:14:13.297614+0000","scrub_duration":0.23998999500000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":24,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"112'94","reported_seq":126,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:35.999324+0000","last_change":"2024-01-16T15:28:35.999421+0000","last_active":"2024-01-16T15:28:35.999324+0000","last_peered":"2024-01-16T15:28:35.999324+0000","last_clean":"2024-01-16T15:28:35.999324+0000","last_became_active":"2024-01-16T15:19:24.623910+0000","last_became_peered":"2024-01-16T15:19:24.623910+0000","last_unstale":"2024-01-16T15:28:35.999324+0000","last_undegraded":"2024-01-16T15:28:35.999324+0000","last_fullsized":"2024-01-16T15:28:35.999324+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:35.999128+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:35.999128+0000","last_clean_scrub_stamp":"2024-01-16T15:28:35.999128+0000","objects_scrubbed":23,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:29:18.663684+0000","scrub_duration":0.30044927999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":25,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"112'81","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.481673+0000","last_change":"2024-01-16T15:28:33.481747+0000","last_active":"2024-01-16T15:28:33.481673+0000","last_peered":"2024-01-16T15:28:33.481673+0000","last_clean":"2024-01-16T15:28:33.481673+0000","last_became_active":"2024-01-16T15:19:24.614471+0000","last_became_peered":"2024-01-16T15:19:24.614471+0000","last_unstale":"2024-01-16T15:28:33.481673+0000","last_undegraded":"2024-01-16T15:28:33.481673+0000","last_fullsized":"2024-01-16T15:28:33.481673+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:33.481512+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:33.481512+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.481512+0000","objects_scrubbed":18,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:10:34.932260+0000","scrub_duration":0.20372825899999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"112'90","reported_seq":123,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.951640+0000","last_change":"2024-01-16T15:28:20.951722+0000","last_active":"2024-01-16T15:28:20.951640+0000","last_peered":"2024-01-16T15:28:20.951640+0000","last_clean":"2024-01-16T15:28:20.951640+0000","last_became_active":"2024-01-16T15:19:24.626609+0000","last_became_peered":"2024-01-16T15:19:24.626609+0000","last_unstale":"2024-01-16T15:28:20.951640+0000","last_undegraded":"2024-01-16T15:28:20.951640+0000","last_fullsized":"2024-01-16T15:28:20.951640+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'90","last_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_deep_scrub":"112'90","last_deep_scrub_stamp":"2024-01-16T15:28:20.951444+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.951444+0000","objects_scrubbed":19,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:27:52.025141+0000","scrub_duration":0.20406614100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"112'89","reported_seq":125,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:27.496415+0000","last_change":"2024-01-16T15:28:27.496503+0000","last_active":"2024-01-16T15:28:27.496415+0000","last_peered":"2024-01-16T15:28:27.496415+0000","last_clean":"2024-01-16T15:28:27.496415+0000","last_became_active":"2024-01-16T15:19:24.626392+0000","last_became_peered":"2024-01-16T15:19:24.626392+0000","last_unstale":"2024-01-16T15:28:27.496415+0000","last_undegraded":"2024-01-16T15:28:27.496415+0000","last_fullsized":"2024-01-16T15:28:27.496415+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:27.496240+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:27.496240+0000","last_clean_scrub_stamp":"2024-01-16T15:28:27.496240+0000","objects_scrubbed":20,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:27:37.199022+0000","scrub_duration":0.25531969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":22,"num_write_kb":81921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"112'107","reported_seq":142,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:14.336760+0000","last_change":"2024-01-16T15:29:14.336822+0000","last_active":"2024-01-16T15:29:14.336760+0000","last_peered":"2024-01-16T15:29:14.336760+0000","last_clean":"2024-01-16T15:29:14.336760+0000","last_became_active":"2024-01-16T15:19:24.630870+0000","last_became_peered":"2024-01-16T15:19:24.630870+0000","last_unstale":"2024-01-16T15:29:14.336760+0000","last_undegraded":"2024-01-16T15:29:14.336760+0000","last_fullsized":"2024-01-16T15:29:14.336760+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'107","last_scrub_stamp":"2024-01-16T15:29:14.336595+0000","last_deep_scrub":"112'107","last_deep_scrub_stamp":"2024-01-16T15:29:14.336595+0000","last_clean_scrub_stamp":"2024-01-16T15:29:14.336595+0000","objects_scrubbed":25,"log_size":107,"log_dups_size":0,"ondisk_log_size":107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:43:49.934691+0000","scrub_duration":0.27487290800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":104857600,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":25,"num_write_kb":102400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:28.302192+0000","last_change":"2024-01-16T15:28:28.302272+0000","last_active":"2024-01-16T15:28:28.302192+0000","last_peered":"2024-01-16T15:28:28.302192+0000","last_clean":"2024-01-16T15:28:28.302192+0000","last_became_active":"2024-01-16T15:19:24.622264+0000","last_became_peered":"2024-01-16T15:19:24.622264+0000","last_unstale":"2024-01-16T15:28:28.302192+0000","last_undegraded":"2024-01-16T15:28:28.302192+0000","last_fullsized":"2024-01-16T15:28:28.302192+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:28.302044+0000","last_clean_scrub_stamp":"2024-01-16T15:28:28.302044+0000","objects_scrubbed":16,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:39:38.499053+0000","scrub_duration":0.149915824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"112'63","reported_seq":99,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:44.708897+0000","last_change":"2024-01-16T15:28:44.708973+0000","last_active":"2024-01-16T15:28:44.708897+0000","last_peered":"2024-01-16T15:28:44.708897+0000","last_clean":"2024-01-16T15:28:44.708897+0000","last_became_active":"2024-01-16T15:19:42.217848+0000","last_became_peered":"2024-01-16T15:19:42.217848+0000","last_unstale":"2024-01-16T15:28:44.708897+0000","last_undegraded":"2024-01-16T15:28:44.708897+0000","last_fullsized":"2024-01-16T15:28:44.708897+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'63","last_scrub_stamp":"2024-01-16T15:28:44.708715+0000","last_deep_scrub":"112'63","last_deep_scrub_stamp":"2024-01-16T15:28:44.708715+0000","last_clean_scrub_stamp":"2024-01-16T15:28:44.708715+0000","objects_scrubbed":13,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:16:25.059080+0000","scrub_duration":0.14963963599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":13,"num_object_clones":0,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":3,"num_read_kb":12288,"num_write":13,"num_write_kb":53248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"112'81","reported_seq":116,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.316071+0000","last_change":"2024-01-16T15:28:40.316137+0000","last_active":"2024-01-16T15:28:40.316071+0000","last_peered":"2024-01-16T15:28:40.316071+0000","last_clean":"2024-01-16T15:28:40.316071+0000","last_became_active":"2024-01-16T15:19:24.619704+0000","last_became_peered":"2024-01-16T15:19:24.619704+0000","last_unstale":"2024-01-16T15:28:40.316071+0000","last_undegraded":"2024-01-16T15:28:40.316071+0000","last_fullsized":"2024-01-16T15:28:40.316071+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:40.315969+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:40.315969+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.315969+0000","objects_scrubbed":19,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:29:11.371995+0000","scrub_duration":0.23402421100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":9,"num_read_kb":36864,"num_write":19,"num_write_kb":77824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"112'95","reported_seq":133,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.537448+0000","last_change":"2024-01-16T15:28:42.537543+0000","last_active":"2024-01-16T15:28:42.537448+0000","last_peered":"2024-01-16T15:28:42.537448+0000","last_clean":"2024-01-16T15:28:42.537448+0000","last_became_active":"2024-01-16T15:19:24.632508+0000","last_became_peered":"2024-01-16T15:19:24.632508+0000","last_unstale":"2024-01-16T15:28:42.537448+0000","last_undegraded":"2024-01-16T15:28:42.537448+0000","last_fullsized":"2024-01-16T15:28:42.537448+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'95","last_scrub_stamp":"2024-01-16T15:28:42.537268+0000","last_deep_scrub":"112'95","last_deep_scrub_stamp":"2024-01-16T15:28:42.537268+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.537268+0000","objects_scrubbed":17,"log_size":95,"log_dups_size":0,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:54:49.101480+0000","scrub_duration":0.19056937099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":12,"num_read_kb":32772,"num_write":25,"num_write_kb":69636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"112'117","reported_seq":156,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.343525+0000","last_change":"2024-01-16T15:28:29.343618+0000","last_active":"2024-01-16T15:28:29.343525+0000","last_peered":"2024-01-16T15:28:29.343525+0000","last_clean":"2024-01-16T15:28:29.343525+0000","last_became_active":"2024-01-16T15:19:24.629607+0000","last_became_peered":"2024-01-16T15:19:24.629607+0000","last_unstale":"2024-01-16T15:28:29.343525+0000","last_undegraded":"2024-01-16T15:28:29.343525+0000","last_fullsized":"2024-01-16T15:28:29.343525+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'117","last_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_deep_scrub":"112'117","last_deep_scrub_stamp":"2024-01-16T15:28:29.343355+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.343355+0000","objects_scrubbed":26,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:55:18.083505+0000","scrub_duration":0.31588118900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"112'102","reported_seq":135,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.453254+0000","last_change":"2024-01-16T15:28:19.453335+0000","last_active":"2024-01-16T15:28:19.453254+0000","last_peered":"2024-01-16T15:28:19.453254+0000","last_clean":"2024-01-16T15:28:19.453254+0000","last_became_active":"2024-01-16T15:19:24.633246+0000","last_became_peered":"2024-01-16T15:19:24.633246+0000","last_unstale":"2024-01-16T15:28:19.453254+0000","last_undegraded":"2024-01-16T15:28:19.453254+0000","last_fullsized":"2024-01-16T15:28:19.453254+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'102","last_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_deep_scrub":"112'102","last_deep_scrub_stamp":"2024-01-16T15:28:19.453110+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.453110+0000","objects_scrubbed":24,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.543525+0000","scrub_duration":0.24508891399999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663296,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":24,"num_write_kb":98304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.343229+0000","last_change":"2024-01-16T15:28:33.343229+0000","last_active":"2024-01-16T15:28:33.343229+0000","last_peered":"2024-01-16T15:28:33.343229+0000","last_clean":"2024-01-16T15:28:33.343229+0000","last_became_active":"2024-01-16T15:19:24.623690+0000","last_became_peered":"2024-01-16T15:19:24.623690+0000","last_unstale":"2024-01-16T15:28:33.343229+0000","last_undegraded":"2024-01-16T15:28:33.343229+0000","last_fullsized":"2024-01-16T15:28:33.343229+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:33.342865+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:33.342865+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.342865+0000","objects_scrubbed":21,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:48:09.821872+0000","scrub_duration":0.226300789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"112'6","reported_seq":33,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:41.310657+0000","last_change":"2024-01-16T15:28:41.310726+0000","last_active":"2024-01-16T15:28:41.310657+0000","last_peered":"2024-01-16T15:28:41.310657+0000","last_clean":"2024-01-16T15:28:41.310657+0000","last_became_active":"2024-01-16T15:19:23.620476+0000","last_became_peered":"2024-01-16T15:19:23.620476+0000","last_unstale":"2024-01-16T15:28:41.310657+0000","last_undegraded":"2024-01-16T15:28:41.310657+0000","last_fullsized":"2024-01-16T15:28:41.310657+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'6","last_scrub_stamp":"2024-01-16T15:28:41.310472+0000","last_deep_scrub":"112'6","last_deep_scrub_stamp":"2024-01-16T15:28:41.310472+0000","last_clean_scrub_stamp":"2024-01-16T15:28:41.310472+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:55:08.138176+0000","scrub_duration":0.0084308639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.835785+0000","last_change":"2024-01-16T15:28:32.835862+0000","last_active":"2024-01-16T15:28:32.835785+0000","last_peered":"2024-01-16T15:28:32.835785+0000","last_clean":"2024-01-16T15:28:32.835785+0000","last_became_active":"2024-01-16T15:19:24.628120+0000","last_became_peered":"2024-01-16T15:19:24.628120+0000","last_unstale":"2024-01-16T15:28:32.835785+0000","last_undegraded":"2024-01-16T15:28:32.835785+0000","last_fullsized":"2024-01-16T15:28:32.835785+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:32.835644+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:32.835644+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.835644+0000","objects_scrubbed":21,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:16:19.714227+0000","scrub_duration":0.264673932,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":86017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"109'1","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.589975+0000","last_change":"2024-01-16T15:28:20.589975+0000","last_active":"2024-01-16T15:28:20.589975+0000","last_peered":"2024-01-16T15:28:20.589975+0000","last_clean":"2024-01-16T15:28:20.589975+0000","last_became_active":"2024-01-16T15:19:23.620787+0000","last_became_peered":"2024-01-16T15:19:23.620787+0000","last_unstale":"2024-01-16T15:28:20.589975+0000","last_undegraded":"2024-01-16T15:28:20.589975+0000","last_fullsized":"2024-01-16T15:28:20.589975+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:20.589720+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.589720+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:55:37.660795+0000","scrub_duration":0.0035764669999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"112'85","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.352108+0000","last_change":"2024-01-16T15:28:30.352108+0000","last_active":"2024-01-16T15:28:30.352108+0000","last_peered":"2024-01-16T15:28:30.352108+0000","last_clean":"2024-01-16T15:28:30.352108+0000","last_became_active":"2024-01-16T15:19:24.629889+0000","last_became_peered":"2024-01-16T15:19:24.629889+0000","last_unstale":"2024-01-16T15:28:30.352108+0000","last_undegraded":"2024-01-16T15:28:30.352108+0000","last_fullsized":"2024-01-16T15:28:30.352108+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:30.351810+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.351810+0000","objects_scrubbed":19,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:26:22.513330+0000","scrub_duration":0.19453525299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":13,"num_read_kb":45058,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.865070+0000","last_change":"2024-01-16T15:28:10.865070+0000","last_active":"2024-01-16T15:28:10.865070+0000","last_peered":"2024-01-16T15:28:10.865070+0000","last_clean":"2024-01-16T15:28:10.865070+0000","last_became_active":"2024-01-16T15:19:23.629735+0000","last_became_peered":"2024-01-16T15:19:23.629735+0000","last_unstale":"2024-01-16T15:28:10.865070+0000","last_undegraded":"2024-01-16T15:28:10.865070+0000","last_fullsized":"2024-01-16T15:28:10.865070+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:10.864918+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.864918+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:51:32.896445+0000","scrub_duration":0.0059496970000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"112'109","reported_seq":147,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.975751+0000","last_change":"2024-01-16T15:28:30.975803+0000","last_active":"2024-01-16T15:28:30.975751+0000","last_peered":"2024-01-16T15:28:30.975751+0000","last_clean":"2024-01-16T15:28:30.975751+0000","last_became_active":"2024-01-16T15:19:24.618931+0000","last_became_peered":"2024-01-16T15:19:24.618931+0000","last_unstale":"2024-01-16T15:28:30.975751+0000","last_undegraded":"2024-01-16T15:28:30.975751+0000","last_fullsized":"2024-01-16T15:28:30.975751+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'109","last_scrub_stamp":"2024-01-16T15:28:30.975585+0000","last_deep_scrub":"112'109","last_deep_scrub_stamp":"2024-01-16T15:28:30.975585+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.975585+0000","objects_scrubbed":25,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:41:50.616075+0000","scrub_duration":0.273993918,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100663557,"num_objects":25,"num_object_clones":0,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":27,"num_write_kb":98306,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:13.789502+0000","last_change":"2024-01-16T15:28:13.789502+0000","last_active":"2024-01-16T15:28:13.789502+0000","last_peered":"2024-01-16T15:28:13.789502+0000","last_clean":"2024-01-16T15:28:13.789502+0000","last_became_active":"2024-01-16T15:19:23.618217+0000","last_became_peered":"2024-01-16T15:19:23.618217+0000","last_unstale":"2024-01-16T15:28:13.789502+0000","last_undegraded":"2024-01-16T15:28:13.789502+0000","last_fullsized":"2024-01-16T15:28:13.789502+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:13.789255+0000","last_clean_scrub_stamp":"2024-01-16T15:28:13.789255+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:28:19.022780+0000","scrub_duration":0.0072576510000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"112'94","reported_seq":128,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:31.165088+0000","last_change":"2024-01-16T15:28:31.165088+0000","last_active":"2024-01-16T15:28:31.165088+0000","last_peered":"2024-01-16T15:28:31.165088+0000","last_clean":"2024-01-16T15:28:31.165088+0000","last_became_active":"2024-01-16T15:19:24.637270+0000","last_became_peered":"2024-01-16T15:19:24.637270+0000","last_unstale":"2024-01-16T15:28:31.165088+0000","last_undegraded":"2024-01-16T15:28:31.165088+0000","last_fullsized":"2024-01-16T15:28:31.165088+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:31.164841+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:31.164841+0000","last_clean_scrub_stamp":"2024-01-16T15:28:31.164841+0000","objects_scrubbed":16,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:23:05.015647+0000","scrub_duration":0.15844903499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":16,"num_write_kb":65536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"109'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:24.260389+0000","last_change":"2024-01-16T15:28:24.260389+0000","last_active":"2024-01-16T15:28:24.260389+0000","last_peered":"2024-01-16T15:28:24.260389+0000","last_clean":"2024-01-16T15:28:24.260389+0000","last_became_active":"2024-01-16T15:19:23.617730+0000","last_became_peered":"2024-01-16T15:19:23.617730+0000","last_unstale":"2024-01-16T15:28:24.260389+0000","last_undegraded":"2024-01-16T15:28:24.260389+0000","last_fullsized":"2024-01-16T15:28:24.260389+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:24.260116+0000","last_clean_scrub_stamp":"2024-01-16T15:28:24.260116+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:56:22.819324+0000","scrub_duration":0.0042817660000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"112'108","reported_seq":148,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:39.416442+0000","last_change":"2024-01-16T15:28:39.416442+0000","last_active":"2024-01-16T15:28:39.416442+0000","last_peered":"2024-01-16T15:28:39.416442+0000","last_clean":"2024-01-16T15:28:39.416442+0000","last_became_active":"2024-01-16T15:19:24.621834+0000","last_became_peered":"2024-01-16T15:19:24.621834+0000","last_unstale":"2024-01-16T15:28:39.416442+0000","last_undegraded":"2024-01-16T15:28:39.416442+0000","last_fullsized":"2024-01-16T15:28:39.416442+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'108","last_scrub_stamp":"2024-01-16T15:28:39.416188+0000","last_deep_scrub":"112'108","last_deep_scrub_stamp":"2024-01-16T15:28:39.416188+0000","last_clean_scrub_stamp":"2024-01-16T15:28:39.416188+0000","objects_scrubbed":26,"log_size":108,"log_dups_size":0,"ondisk_log_size":108,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:50:12.499311+0000","scrub_duration":0.29096126300000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109051904,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":13,"num_read_kb":53248,"num_write":26,"num_write_kb":106496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"109'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.816296+0000","last_change":"2024-01-16T15:28:14.816296+0000","last_active":"2024-01-16T15:28:14.816296+0000","last_peered":"2024-01-16T15:28:14.816296+0000","last_clean":"2024-01-16T15:28:14.816296+0000","last_became_active":"2024-01-16T15:19:23.622361+0000","last_became_peered":"2024-01-16T15:19:23.622361+0000","last_unstale":"2024-01-16T15:28:14.816296+0000","last_undegraded":"2024-01-16T15:28:14.816296+0000","last_fullsized":"2024-01-16T15:28:14.816296+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'1","last_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_deep_scrub":"109'1","last_deep_scrub_stamp":"2024-01-16T15:28:14.816039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.816039+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:09:58.559289+0000","scrub_duration":0.0043054030000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"112'99","reported_seq":136,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:02.298030+0000","last_change":"2024-01-16T15:29:02.298030+0000","last_active":"2024-01-16T15:29:02.298030+0000","last_peered":"2024-01-16T15:29:02.298030+0000","last_clean":"2024-01-16T15:29:02.298030+0000","last_became_active":"2024-01-16T15:19:24.621422+0000","last_became_peered":"2024-01-16T15:19:24.621422+0000","last_unstale":"2024-01-16T15:29:02.298030+0000","last_undegraded":"2024-01-16T15:29:02.298030+0000","last_fullsized":"2024-01-16T15:29:02.298030+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'99","last_scrub_stamp":"2024-01-16T15:29:02.297776+0000","last_deep_scrub":"112'99","last_deep_scrub_stamp":"2024-01-16T15:29:02.297776+0000","last_clean_scrub_stamp":"2024-01-16T15:29:02.297776+0000","objects_scrubbed":19,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:22.958112+0000","scrub_duration":0.19285666200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":11,"num_read_kb":36866,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.251291+0000","last_change":"2024-01-16T15:28:11.251291+0000","last_active":"2024-01-16T15:28:11.251291+0000","last_peered":"2024-01-16T15:28:11.251291+0000","last_clean":"2024-01-16T15:28:11.251291+0000","last_became_active":"2024-01-16T15:19:23.620197+0000","last_became_peered":"2024-01-16T15:19:23.620197+0000","last_unstale":"2024-01-16T15:28:11.251291+0000","last_undegraded":"2024-01-16T15:28:11.251291+0000","last_fullsized":"2024-01-16T15:28:11.251291+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:11.251026+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.251026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:41:45.371008+0000","scrub_duration":0.0038716129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"112'86","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:44.544412+0000","last_change":"2024-01-16T15:28:44.544412+0000","last_active":"2024-01-16T15:28:44.544412+0000","last_peered":"2024-01-16T15:28:44.544412+0000","last_clean":"2024-01-16T15:28:44.544412+0000","last_became_active":"2024-01-16T15:19:24.626168+0000","last_became_peered":"2024-01-16T15:19:24.626168+0000","last_unstale":"2024-01-16T15:28:44.544412+0000","last_undegraded":"2024-01-16T15:28:44.544412+0000","last_fullsized":"2024-01-16T15:28:44.544412+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'86","last_scrub_stamp":"2024-01-16T15:28:44.544211+0000","last_deep_scrub":"112'86","last_deep_scrub_stamp":"2024-01-16T15:28:44.544211+0000","last_clean_scrub_stamp":"2024-01-16T15:28:44.544211+0000","objects_scrubbed":18,"log_size":86,"log_dups_size":0,"ondisk_log_size":86,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:37:52.626917+0000","scrub_duration":0.22887946100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":6,"num_read_kb":24576,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.061040+0000","last_change":"2024-01-16T15:28:42.061040+0000","last_active":"2024-01-16T15:28:42.061040+0000","last_peered":"2024-01-16T15:28:42.061040+0000","last_clean":"2024-01-16T15:28:42.061040+0000","last_became_active":"2024-01-16T15:19:23.625824+0000","last_became_peered":"2024-01-16T15:19:23.625824+0000","last_unstale":"2024-01-16T15:28:42.061040+0000","last_undegraded":"2024-01-16T15:28:42.061040+0000","last_fullsized":"2024-01-16T15:28:42.061040+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:42.060723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:42.060723+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.060723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T22:33:46.204134+0000","scrub_duration":0.0036082470000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"112'80","reported_seq":110,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:52.478582+0000","last_change":"2024-01-16T15:28:52.478582+0000","last_active":"2024-01-16T15:28:52.478582+0000","last_peered":"2024-01-16T15:28:52.478582+0000","last_clean":"2024-01-16T15:28:52.478582+0000","last_became_active":"2024-01-16T15:19:24.632894+0000","last_became_peered":"2024-01-16T15:19:24.632894+0000","last_unstale":"2024-01-16T15:28:52.478582+0000","last_undegraded":"2024-01-16T15:28:52.478582+0000","last_fullsized":"2024-01-16T15:28:52.478582+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'80","last_scrub_stamp":"2024-01-16T15:28:52.478341+0000","last_deep_scrub":"112'80","last_deep_scrub_stamp":"2024-01-16T15:28:52.478341+0000","last_clean_scrub_stamp":"2024-01-16T15:28:52.478341+0000","objects_scrubbed":14,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:02:58.318697+0000","scrub_duration":0.152373551,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"112'1","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:11.843271+0000","last_change":"2024-01-16T15:28:11.843271+0000","last_active":"2024-01-16T15:28:11.843271+0000","last_peered":"2024-01-16T15:28:11.843271+0000","last_clean":"2024-01-16T15:28:11.843271+0000","last_became_active":"2024-01-16T15:19:23.619355+0000","last_became_peered":"2024-01-16T15:19:23.619355+0000","last_unstale":"2024-01-16T15:28:11.843271+0000","last_undegraded":"2024-01-16T15:28:11.843271+0000","last_fullsized":"2024-01-16T15:28:11.843271+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'1","last_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_deep_scrub":"112'1","last_deep_scrub_stamp":"2024-01-16T15:28:11.843032+0000","last_clean_scrub_stamp":"2024-01-16T15:28:11.843032+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:38:01.354272+0000","scrub_duration":0.003520572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"112'82","reported_seq":113,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.726792+0000","last_change":"2024-01-16T15:28:29.726792+0000","last_active":"2024-01-16T15:28:29.726792+0000","last_peered":"2024-01-16T15:28:29.726792+0000","last_clean":"2024-01-16T15:28:29.726792+0000","last_became_active":"2024-01-16T15:19:24.636162+0000","last_became_peered":"2024-01-16T15:19:24.636162+0000","last_unstale":"2024-01-16T15:28:29.726792+0000","last_undegraded":"2024-01-16T15:28:29.726792+0000","last_fullsized":"2024-01-16T15:28:29.726792+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:29.726602+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:29.726602+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.726602+0000","objects_scrubbed":18,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:25.349562+0000","scrub_duration":0.187189361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":20,"num_write_kb":73729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.784389+0000","last_change":"2024-01-16T15:28:16.784389+0000","last_active":"2024-01-16T15:28:16.784389+0000","last_peered":"2024-01-16T15:28:16.784389+0000","last_clean":"2024-01-16T15:28:16.784389+0000","last_became_active":"2024-01-16T15:19:23.624040+0000","last_became_peered":"2024-01-16T15:19:23.624040+0000","last_unstale":"2024-01-16T15:28:16.784389+0000","last_undegraded":"2024-01-16T15:28:16.784389+0000","last_fullsized":"2024-01-16T15:28:16.784389+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.784086+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.784086+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:37:32.176453+0000","scrub_duration":0.0040169810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"112'79","reported_seq":117,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:42.831200+0000","last_change":"2024-01-16T15:28:42.831200+0000","last_active":"2024-01-16T15:28:42.831200+0000","last_peered":"2024-01-16T15:28:42.831200+0000","last_clean":"2024-01-16T15:28:42.831200+0000","last_became_active":"2024-01-16T15:19:24.627935+0000","last_became_peered":"2024-01-16T15:19:24.627935+0000","last_unstale":"2024-01-16T15:28:42.831200+0000","last_undegraded":"2024-01-16T15:28:42.831200+0000","last_fullsized":"2024-01-16T15:28:42.831200+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:42.830943+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:42.830943+0000","last_clean_scrub_stamp":"2024-01-16T15:28:42.830943+0000","objects_scrubbed":19,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:42:48.006952+0000","scrub_duration":0.21546314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79691776,"num_objects":19,"num_object_clones":0,"num_object_copies":38,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":12,"num_read_kb":45057,"num_write":21,"num_write_kb":77825,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"112'33","reported_seq":60,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:50.170253+0000","last_change":"2024-01-16T15:28:50.170253+0000","last_active":"2024-01-16T15:28:50.170253+0000","last_peered":"2024-01-16T15:28:50.170253+0000","last_clean":"2024-01-16T15:28:50.170253+0000","last_became_active":"2024-01-16T15:19:23.619753+0000","last_became_peered":"2024-01-16T15:19:23.619753+0000","last_unstale":"2024-01-16T15:28:50.170253+0000","last_undegraded":"2024-01-16T15:28:50.170253+0000","last_fullsized":"2024-01-16T15:28:50.170253+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'33","last_scrub_stamp":"2024-01-16T15:28:50.169974+0000","last_deep_scrub":"112'33","last_deep_scrub_stamp":"2024-01-16T15:28:50.169974+0000","last_clean_scrub_stamp":"2024-01-16T15:28:50.169974+0000","objects_scrubbed":2,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:08:02.553184+0000","scrub_duration":0.008481901,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"112'93","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:45.595636+0000","last_change":"2024-01-16T15:28:45.595636+0000","last_active":"2024-01-16T15:28:45.595636+0000","last_peered":"2024-01-16T15:28:45.595636+0000","last_clean":"2024-01-16T15:28:45.595636+0000","last_became_active":"2024-01-16T15:19:24.630295+0000","last_became_peered":"2024-01-16T15:19:24.630295+0000","last_unstale":"2024-01-16T15:28:45.595636+0000","last_undegraded":"2024-01-16T15:28:45.595636+0000","last_fullsized":"2024-01-16T15:28:45.595636+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:28:45.595459+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:28:45.595459+0000","last_clean_scrub_stamp":"2024-01-16T15:28:45.595459+0000","objects_scrubbed":24,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:41:29.813183+0000","scrub_duration":0.29598812099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469184,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":10,"num_read_kb":32770,"num_write":26,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"112'313","reported_seq":340,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:10.900823+0000","last_change":"2024-01-16T15:28:48.165251+0000","last_active":"2024-01-16T15:29:10.900823+0000","last_peered":"2024-01-16T15:29:10.900823+0000","last_clean":"2024-01-16T15:29:10.900823+0000","last_became_active":"2024-01-16T15:19:23.624648+0000","last_became_peered":"2024-01-16T15:19:23.624648+0000","last_unstale":"2024-01-16T15:29:10.900823+0000","last_undegraded":"2024-01-16T15:29:10.900823+0000","last_fullsized":"2024-01-16T15:29:10.900823+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'311","last_scrub_stamp":"2024-01-16T15:28:48.164882+0000","last_deep_scrub":"112'311","last_deep_scrub_stamp":"2024-01-16T15:28:48.164882+0000","last_clean_scrub_stamp":"2024-01-16T15:28:48.164882+0000","objects_scrubbed":4,"log_size":313,"log_dups_size":0,"ondisk_log_size":313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:43:23.660630+0000","scrub_duration":0.020197336999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1196982,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":308,"num_write_kb":1326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"112'79","reported_seq":112,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:33.880948+0000","last_change":"2024-01-16T15:28:33.880948+0000","last_active":"2024-01-16T15:28:33.880948+0000","last_peered":"2024-01-16T15:28:33.880948+0000","last_clean":"2024-01-16T15:28:33.880948+0000","last_became_active":"2024-01-16T15:19:24.633321+0000","last_became_peered":"2024-01-16T15:19:24.633321+0000","last_unstale":"2024-01-16T15:28:33.880948+0000","last_undegraded":"2024-01-16T15:28:33.880948+0000","last_fullsized":"2024-01-16T15:28:33.880948+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'79","last_scrub_stamp":"2024-01-16T15:28:33.880692+0000","last_deep_scrub":"112'79","last_deep_scrub_stamp":"2024-01-16T15:28:33.880692+0000","last_clean_scrub_stamp":"2024-01-16T15:28:33.880692+0000","objects_scrubbed":16,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:44:30.122032+0000","scrub_duration":0.177224733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67108864,"num_objects":16,"num_object_clones":0,"num_object_copies":32,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":7,"num_read_kb":24577,"num_write":18,"num_write_kb":65537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.219036+0000","last_change":"2024-01-16T15:28:18.219036+0000","last_active":"2024-01-16T15:28:18.219036+0000","last_peered":"2024-01-16T15:28:18.219036+0000","last_clean":"2024-01-16T15:28:18.219036+0000","last_became_active":"2024-01-16T15:19:23.616945+0000","last_became_peered":"2024-01-16T15:19:23.616945+0000","last_unstale":"2024-01-16T15:28:18.219036+0000","last_undegraded":"2024-01-16T15:28:18.219036+0000","last_fullsized":"2024-01-16T15:28:18.219036+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:18.218660+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.218660+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:48.176772+0000","scrub_duration":0.0091646809999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"112'98","reported_seq":131,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:36.911299+0000","last_change":"2024-01-16T15:28:36.911299+0000","last_active":"2024-01-16T15:28:36.911299+0000","last_peered":"2024-01-16T15:28:36.911299+0000","last_clean":"2024-01-16T15:28:36.911299+0000","last_became_active":"2024-01-16T15:19:24.629917+0000","last_became_peered":"2024-01-16T15:19:24.629917+0000","last_unstale":"2024-01-16T15:28:36.911299+0000","last_undegraded":"2024-01-16T15:28:36.911299+0000","last_fullsized":"2024-01-16T15:28:36.911299+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'98","last_scrub_stamp":"2024-01-16T15:28:36.911039+0000","last_deep_scrub":"112'98","last_deep_scrub_stamp":"2024-01-16T15:28:36.911039+0000","last_clean_scrub_stamp":"2024-01-16T15:28:36.911039+0000","objects_scrubbed":18,"log_size":98,"log_dups_size":0,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:47:57.499710+0000","scrub_duration":0.20160798599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":20482,"num_write":20,"num_write_kb":69634,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:16.450274+0000","last_change":"2024-01-16T15:28:16.450274+0000","last_active":"2024-01-16T15:28:16.450274+0000","last_peered":"2024-01-16T15:28:16.450274+0000","last_clean":"2024-01-16T15:28:16.450274+0000","last_became_active":"2024-01-16T15:19:23.624314+0000","last_became_peered":"2024-01-16T15:19:23.624314+0000","last_unstale":"2024-01-16T15:28:16.450274+0000","last_undegraded":"2024-01-16T15:28:16.450274+0000","last_fullsized":"2024-01-16T15:28:16.450274+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:16.449979+0000","last_clean_scrub_stamp":"2024-01-16T15:28:16.449979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:59:39.537507+0000","scrub_duration":0.0036212340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"112'96","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.977310+0000","last_change":"2024-01-16T15:28:32.977310+0000","last_active":"2024-01-16T15:28:32.977310+0000","last_peered":"2024-01-16T15:28:32.977310+0000","last_clean":"2024-01-16T15:28:32.977310+0000","last_became_active":"2024-01-16T15:19:24.618960+0000","last_became_peered":"2024-01-16T15:19:24.618960+0000","last_unstale":"2024-01-16T15:28:32.977310+0000","last_undegraded":"2024-01-16T15:28:32.977310+0000","last_fullsized":"2024-01-16T15:28:32.977310+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:32.977043+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:32.977043+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.977043+0000","objects_scrubbed":23,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:27:22.982613+0000","scrub_duration":0.25583332399999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274949,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":23,"num_write_kb":90113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"112'12","reported_seq":39,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:07.825453+0000","last_change":"2024-01-16T15:28:07.825453+0000","last_active":"2024-01-16T15:28:07.825453+0000","last_peered":"2024-01-16T15:28:07.825453+0000","last_clean":"2024-01-16T15:28:07.825453+0000","last_became_active":"2024-01-16T15:19:23.625168+0000","last_became_peered":"2024-01-16T15:19:23.625168+0000","last_unstale":"2024-01-16T15:28:07.825453+0000","last_undegraded":"2024-01-16T15:28:07.825453+0000","last_fullsized":"2024-01-16T15:28:07.825453+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'12","last_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_deep_scrub":"112'12","last_deep_scrub_stamp":"2024-01-16T15:28:07.825335+0000","last_clean_scrub_stamp":"2024-01-16T15:28:07.825335+0000","objects_scrubbed":4,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:19:20.349610+0000","scrub_duration":0.0056472520000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"112'62","reported_seq":92,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:21.989502+0000","last_change":"2024-01-16T15:28:21.989502+0000","last_active":"2024-01-16T15:28:21.989502+0000","last_peered":"2024-01-16T15:28:21.989502+0000","last_clean":"2024-01-16T15:28:21.989502+0000","last_became_active":"2024-01-16T15:19:24.616969+0000","last_became_peered":"2024-01-16T15:19:24.616969+0000","last_unstale":"2024-01-16T15:28:21.989502+0000","last_undegraded":"2024-01-16T15:28:21.989502+0000","last_fullsized":"2024-01-16T15:28:21.989502+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'62","last_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_deep_scrub":"112'62","last_deep_scrub_stamp":"2024-01-16T15:28:21.989317+0000","last_clean_scrub_stamp":"2024-01-16T15:28:21.989317+0000","objects_scrubbed":12,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:16:49.378780+0000","scrub_duration":0.125520883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50331648,"num_objects":12,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":12,"num_write_kb":49152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"112'2","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:54.163157+0000","last_change":"2024-01-16T15:28:54.163222+0000","last_active":"2024-01-16T15:28:54.163157+0000","last_peered":"2024-01-16T15:28:54.163157+0000","last_clean":"2024-01-16T15:28:54.163157+0000","last_became_active":"2024-01-16T15:19:23.626654+0000","last_became_peered":"2024-01-16T15:19:23.626654+0000","last_unstale":"2024-01-16T15:28:54.163157+0000","last_undegraded":"2024-01-16T15:28:54.163157+0000","last_fullsized":"2024-01-16T15:28:54.163157+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-01-16T15:28:54.162972+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-01-16T15:28:54.162972+0000","last_clean_scrub_stamp":"2024-01-16T15:28:54.162972+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:52:08.962695+0000","scrub_duration":0.0090223179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"112'81","reported_seq":118,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:34.355627+0000","last_change":"2024-01-16T15:28:34.355708+0000","last_active":"2024-01-16T15:28:34.355627+0000","last_peered":"2024-01-16T15:28:34.355627+0000","last_clean":"2024-01-16T15:28:34.355627+0000","last_became_active":"2024-01-16T15:19:24.620547+0000","last_became_peered":"2024-01-16T15:19:24.620547+0000","last_unstale":"2024-01-16T15:28:34.355627+0000","last_undegraded":"2024-01-16T15:28:34.355627+0000","last_fullsized":"2024-01-16T15:28:34.355627+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:34.355450+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:34.355450+0000","last_clean_scrub_stamp":"2024-01-16T15:28:34.355450+0000","objects_scrubbed":20,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:17:13.088452+0000","scrub_duration":0.20144817200000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":11,"num_read_kb":45056,"num_write":20,"num_write_kb":81920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.809350+0000","last_change":"2024-01-16T15:28:17.809350+0000","last_active":"2024-01-16T15:28:17.809350+0000","last_peered":"2024-01-16T15:28:17.809350+0000","last_clean":"2024-01-16T15:28:17.809350+0000","last_became_active":"2024-01-16T15:19:23.621944+0000","last_became_peered":"2024-01-16T15:19:23.621944+0000","last_unstale":"2024-01-16T15:28:17.809350+0000","last_undegraded":"2024-01-16T15:28:17.809350+0000","last_fullsized":"2024-01-16T15:28:17.809350+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.809101+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.809101+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T21:15:47.767212+0000","scrub_duration":0.0070965819999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"112'100","reported_seq":138,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.997559+0000","last_change":"2024-01-16T15:28:40.997649+0000","last_active":"2024-01-16T15:28:40.997559+0000","last_peered":"2024-01-16T15:28:40.997559+0000","last_clean":"2024-01-16T15:28:40.997559+0000","last_became_active":"2024-01-16T15:19:24.618381+0000","last_became_peered":"2024-01-16T15:19:24.618381+0000","last_unstale":"2024-01-16T15:28:40.997559+0000","last_undegraded":"2024-01-16T15:28:40.997559+0000","last_fullsized":"2024-01-16T15:28:40.997559+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'100","last_scrub_stamp":"2024-01-16T15:28:40.997402+0000","last_deep_scrub":"112'100","last_deep_scrub_stamp":"2024-01-16T15:28:40.997402+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.997402+0000","objects_scrubbed":23,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:15:49.477973+0000","scrub_duration":0.25693853799999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":11,"num_read_kb":28676,"num_write":27,"num_write_kb":94210,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"112'2","reported_seq":29,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.443869+0000","last_change":"2024-01-16T15:28:18.443869+0000","last_active":"2024-01-16T15:28:18.443869+0000","last_peered":"2024-01-16T15:28:18.443869+0000","last_clean":"2024-01-16T15:28:18.443869+0000","last_became_active":"2024-01-16T15:19:23.626629+0000","last_became_peered":"2024-01-16T15:19:23.626629+0000","last_unstale":"2024-01-16T15:28:18.443869+0000","last_undegraded":"2024-01-16T15:28:18.443869+0000","last_fullsized":"2024-01-16T15:28:18.443869+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'2","last_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_deep_scrub":"112'2","last_deep_scrub_stamp":"2024-01-16T15:28:18.443623+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.443623+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:54:31.553582+0000","scrub_duration":0.0077739760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"112'89","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:34.943163+0000","last_change":"2024-01-16T15:28:34.943251+0000","last_active":"2024-01-16T15:28:34.943163+0000","last_peered":"2024-01-16T15:28:34.943163+0000","last_clean":"2024-01-16T15:28:34.943163+0000","last_became_active":"2024-01-16T15:19:24.624056+0000","last_became_peered":"2024-01-16T15:19:24.624056+0000","last_unstale":"2024-01-16T15:28:34.943163+0000","last_undegraded":"2024-01-16T15:28:34.943163+0000","last_fullsized":"2024-01-16T15:28:34.943163+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:34.942995+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:34.942995+0000","last_clean_scrub_stamp":"2024-01-16T15:28:34.942995+0000","objects_scrubbed":23,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:58.983028+0000","scrub_duration":0.28306326100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":12,"num_read_kb":49152,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:32.220561+0000","last_change":"2024-01-16T15:28:32.220610+0000","last_active":"2024-01-16T15:28:32.220561+0000","last_peered":"2024-01-16T15:28:32.220561+0000","last_clean":"2024-01-16T15:28:32.220561+0000","last_became_active":"2024-01-16T15:19:23.624695+0000","last_became_peered":"2024-01-16T15:19:23.624695+0000","last_unstale":"2024-01-16T15:28:32.220561+0000","last_undegraded":"2024-01-16T15:28:32.220561+0000","last_fullsized":"2024-01-16T15:28:32.220561+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:32.220364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:32.220364+0000","last_clean_scrub_stamp":"2024-01-16T15:28:32.220364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:02:02.571057+0000","scrub_duration":0.0041910580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"112'89","reported_seq":124,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:26.838225+0000","last_change":"2024-01-16T15:28:26.838277+0000","last_active":"2024-01-16T15:28:26.838225+0000","last_peered":"2024-01-16T15:28:26.838225+0000","last_clean":"2024-01-16T15:28:26.838225+0000","last_became_active":"2024-01-16T15:19:24.615561+0000","last_became_peered":"2024-01-16T15:19:24.615561+0000","last_unstale":"2024-01-16T15:28:26.838225+0000","last_undegraded":"2024-01-16T15:28:26.838225+0000","last_fullsized":"2024-01-16T15:28:26.838225+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'89","last_scrub_stamp":"2024-01-16T15:28:26.838060+0000","last_deep_scrub":"112'89","last_deep_scrub_stamp":"2024-01-16T15:28:26.838060+0000","last_clean_scrub_stamp":"2024-01-16T15:28:26.838060+0000","objects_scrubbed":27,"log_size":89,"log_dups_size":0,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:52:31.159826+0000","scrub_duration":0.27286994599999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"112'4","reported_seq":31,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.174817+0000","last_change":"2024-01-16T15:28:17.174817+0000","last_active":"2024-01-16T15:28:17.174817+0000","last_peered":"2024-01-16T15:28:17.174817+0000","last_clean":"2024-01-16T15:28:17.174817+0000","last_became_active":"2024-01-16T15:19:23.620065+0000","last_became_peered":"2024-01-16T15:19:23.620065+0000","last_unstale":"2024-01-16T15:28:17.174817+0000","last_undegraded":"2024-01-16T15:28:17.174817+0000","last_fullsized":"2024-01-16T15:28:17.174817+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'4","last_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_deep_scrub":"112'4","last_deep_scrub_stamp":"2024-01-16T15:28:17.174587+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.174587+0000","objects_scrubbed":1,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:33:21.215097+0000","scrub_duration":0.0071591010000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"112'93","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:12.299910+0000","last_change":"2024-01-16T15:29:12.299964+0000","last_active":"2024-01-16T15:29:12.299910+0000","last_peered":"2024-01-16T15:29:12.299910+0000","last_clean":"2024-01-16T15:29:12.299910+0000","last_became_active":"2024-01-16T15:19:24.624349+0000","last_became_peered":"2024-01-16T15:19:24.624349+0000","last_unstale":"2024-01-16T15:29:12.299910+0000","last_undegraded":"2024-01-16T15:29:12.299910+0000","last_fullsized":"2024-01-16T15:29:12.299910+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'93","last_scrub_stamp":"2024-01-16T15:29:12.299833+0000","last_deep_scrub":"112'93","last_deep_scrub_stamp":"2024-01-16T15:29:12.299833+0000","last_clean_scrub_stamp":"2024-01-16T15:29:12.299833+0000","objects_scrubbed":20,"log_size":93,"log_dups_size":0,"ondisk_log_size":93,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:18:44.858402+0000","scrub_duration":0.21959893999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":20,"num_object_clones":0,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":8,"num_read_kb":28673,"num_write":24,"num_write_kb":81922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.833365+0000","last_change":"2024-01-16T15:28:19.833414+0000","last_active":"2024-01-16T15:28:19.833365+0000","last_peered":"2024-01-16T15:28:19.833365+0000","last_clean":"2024-01-16T15:28:19.833365+0000","last_became_active":"2024-01-16T15:19:23.619319+0000","last_became_peered":"2024-01-16T15:19:23.619319+0000","last_unstale":"2024-01-16T15:28:19.833365+0000","last_undegraded":"2024-01-16T15:28:19.833365+0000","last_fullsized":"2024-01-16T15:28:19.833365+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:19.833199+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.833199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:44:00.923614+0000","scrub_duration":0.0036415929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"112'84","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:29.813828+0000","last_change":"2024-01-16T15:28:29.813905+0000","last_active":"2024-01-16T15:28:29.813828+0000","last_peered":"2024-01-16T15:28:29.813828+0000","last_clean":"2024-01-16T15:28:29.813828+0000","last_became_active":"2024-01-16T15:19:24.620304+0000","last_became_peered":"2024-01-16T15:19:24.620304+0000","last_unstale":"2024-01-16T15:28:29.813828+0000","last_undegraded":"2024-01-16T15:28:29.813828+0000","last_fullsized":"2024-01-16T15:28:29.813828+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'84","last_scrub_stamp":"2024-01-16T15:28:29.813679+0000","last_deep_scrub":"112'84","last_deep_scrub_stamp":"2024-01-16T15:28:29.813679+0000","last_clean_scrub_stamp":"2024-01-16T15:28:29.813679+0000","objects_scrubbed":18,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:00:02.832647+0000","scrub_duration":0.15799079199999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75497472,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":5,"num_read_kb":20480,"num_write":18,"num_write_kb":73728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.206909+0000","last_change":"2024-01-16T15:28:15.206975+0000","last_active":"2024-01-16T15:28:15.206909+0000","last_peered":"2024-01-16T15:28:15.206909+0000","last_clean":"2024-01-16T15:28:15.206909+0000","last_became_active":"2024-01-16T15:19:23.620064+0000","last_became_peered":"2024-01-16T15:19:23.620064+0000","last_unstale":"2024-01-16T15:28:15.206909+0000","last_undegraded":"2024-01-16T15:28:15.206909+0000","last_fullsized":"2024-01-16T15:28:15.206909+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:15.206765+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.206765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:07:27.403194+0000","scrub_duration":0.0041188539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"112'81","reported_seq":114,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:30.874708+0000","last_change":"2024-01-16T15:28:30.874778+0000","last_active":"2024-01-16T15:28:30.874708+0000","last_peered":"2024-01-16T15:28:30.874708+0000","last_clean":"2024-01-16T15:28:30.874708+0000","last_became_active":"2024-01-16T15:19:24.621034+0000","last_became_peered":"2024-01-16T15:19:24.621034+0000","last_unstale":"2024-01-16T15:28:30.874708+0000","last_undegraded":"2024-01-16T15:28:30.874708+0000","last_fullsized":"2024-01-16T15:28:30.874708+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'81","last_scrub_stamp":"2024-01-16T15:28:30.874583+0000","last_deep_scrub":"112'81","last_deep_scrub_stamp":"2024-01-16T15:28:30.874583+0000","last_clean_scrub_stamp":"2024-01-16T15:28:30.874583+0000","objects_scrubbed":18,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:01:07.770194+0000","scrub_duration":0.163030705,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303429,"num_objects":18,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":18,"num_write_kb":69633,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.044284+0000","last_change":"2024-01-16T15:28:40.044333+0000","last_active":"2024-01-16T15:28:40.044284+0000","last_peered":"2024-01-16T15:28:40.044284+0000","last_clean":"2024-01-16T15:28:40.044284+0000","last_became_active":"2024-01-16T15:19:23.622798+0000","last_became_peered":"2024-01-16T15:19:23.622798+0000","last_unstale":"2024-01-16T15:28:40.044284+0000","last_undegraded":"2024-01-16T15:28:40.044284+0000","last_fullsized":"2024-01-16T15:28:40.044284+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:40.044093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:40.044093+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.044093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T15:58:49.863547+0000","scrub_duration":0.0043197859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"112'92","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:22.637191+0000","last_change":"2024-01-16T15:28:22.637284+0000","last_active":"2024-01-16T15:28:22.637191+0000","last_peered":"2024-01-16T15:28:22.637191+0000","last_clean":"2024-01-16T15:28:22.637191+0000","last_became_active":"2024-01-16T15:19:42.592017+0000","last_became_peered":"2024-01-16T15:19:42.592017+0000","last_unstale":"2024-01-16T15:28:22.637191+0000","last_undegraded":"2024-01-16T15:28:22.637191+0000","last_fullsized":"2024-01-16T15:28:22.637191+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'92","last_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_deep_scrub":"112'92","last_deep_scrub_stamp":"2024-01-16T15:28:22.636970+0000","last_clean_scrub_stamp":"2024-01-16T15:28:22.636970+0000","objects_scrubbed":22,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T02:30:01.808019+0000","scrub_duration":0.23588124099999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92274688,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":4,"num_read_kb":16384,"num_write":22,"num_write_kb":90112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"112'3","reported_seq":30,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.749499+0000","last_change":"2024-01-16T15:28:17.749653+0000","last_active":"2024-01-16T15:28:17.749499+0000","last_peered":"2024-01-16T15:28:17.749499+0000","last_clean":"2024-01-16T15:28:17.749499+0000","last_became_active":"2024-01-16T15:19:23.629365+0000","last_became_peered":"2024-01-16T15:19:23.629365+0000","last_unstale":"2024-01-16T15:28:17.749499+0000","last_undegraded":"2024-01-16T15:28:17.749499+0000","last_fullsized":"2024-01-16T15:28:17.749499+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'3","last_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_deep_scrub":"112'3","last_deep_scrub_stamp":"2024-01-16T15:28:17.749294+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.749294+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:57:03.835812+0000","scrub_duration":0.0071287670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"112'88","reported_seq":127,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:29:20.314956+0000","last_change":"2024-01-16T15:29:20.315059+0000","last_active":"2024-01-16T15:29:20.314956+0000","last_peered":"2024-01-16T15:29:20.314956+0000","last_clean":"2024-01-16T15:29:20.314956+0000","last_became_active":"2024-01-16T15:19:24.637429+0000","last_became_peered":"2024-01-16T15:19:24.637429+0000","last_unstale":"2024-01-16T15:29:20.314956+0000","last_undegraded":"2024-01-16T15:29:20.314956+0000","last_fullsized":"2024-01-16T15:29:20.314956+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:29:20.314764+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:29:20.314764+0000","last_clean_scrub_stamp":"2024-01-16T15:29:20.314764+0000","objects_scrubbed":24,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T03:02:29.959467+0000","scrub_duration":0.26989558800000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96469172,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":13,"num_read_kb":49153,"num_write":24,"num_write_kb":94209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"112'31","reported_seq":58,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.479780+0000","last_change":"2024-01-16T15:28:17.479819+0000","last_active":"2024-01-16T15:28:17.479780+0000","last_peered":"2024-01-16T15:28:17.479780+0000","last_clean":"2024-01-16T15:28:17.479780+0000","last_became_active":"2024-01-16T15:19:23.623455+0000","last_became_peered":"2024-01-16T15:19:23.623455+0000","last_unstale":"2024-01-16T15:28:17.479780+0000","last_undegraded":"2024-01-16T15:28:17.479780+0000","last_fullsized":"2024-01-16T15:28:17.479780+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'31","last_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_deep_scrub":"112'31","last_deep_scrub_stamp":"2024-01-16T15:28:17.479610+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.479610+0000","objects_scrubbed":1,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T00:16:48.802592+0000","scrub_duration":0.0060914480000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":67,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":593,"num_omap_keys":11,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"112'96","reported_seq":137,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:15.235685+0000","last_change":"2024-01-16T15:28:15.235729+0000","last_active":"2024-01-16T15:28:15.235685+0000","last_peered":"2024-01-16T15:28:15.235685+0000","last_clean":"2024-01-16T15:28:15.235685+0000","last_became_active":"2024-01-16T15:19:24.626470+0000","last_became_peered":"2024-01-16T15:19:24.626470+0000","last_unstale":"2024-01-16T15:28:15.235685+0000","last_undegraded":"2024-01-16T15:28:15.235685+0000","last_fullsized":"2024-01-16T15:28:15.235685+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'96","last_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_deep_scrub":"112'96","last_deep_scrub_stamp":"2024-01-16T15:28:15.235576+0000","last_clean_scrub_stamp":"2024-01-16T15:28:15.235576+0000","objects_scrubbed":27,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:43:39.275609+0000","scrub_duration":0.27566697899999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113246208,"num_objects":27,"num_object_clones":0,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":0,"num_read":14,"num_read_kb":57344,"num_write":27,"num_write_kb":110592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:47.140086+0000","last_change":"2024-01-16T15:28:47.140200+0000","last_active":"2024-01-16T15:28:47.140086+0000","last_peered":"2024-01-16T15:28:47.140086+0000","last_clean":"2024-01-16T15:28:47.140086+0000","last_became_active":"2024-01-16T15:19:23.624352+0000","last_became_peered":"2024-01-16T15:19:23.624352+0000","last_unstale":"2024-01-16T15:28:47.140086+0000","last_undegraded":"2024-01-16T15:28:47.140086+0000","last_fullsized":"2024-01-16T15:28:47.140086+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:47.139873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:47.139873+0000","last_clean_scrub_stamp":"2024-01-16T15:28:47.139873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:56:12.869817+0000","scrub_duration":0.004451694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"112'66","reported_seq":98,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:20.564786+0000","last_change":"2024-01-16T15:28:20.564862+0000","last_active":"2024-01-16T15:28:20.564786+0000","last_peered":"2024-01-16T15:28:20.564786+0000","last_clean":"2024-01-16T15:28:20.564786+0000","last_became_active":"2024-01-16T15:19:24.623264+0000","last_became_peered":"2024-01-16T15:19:24.623264+0000","last_unstale":"2024-01-16T15:28:20.564786+0000","last_undegraded":"2024-01-16T15:28:20.564786+0000","last_fullsized":"2024-01-16T15:28:20.564786+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'66","last_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_deep_scrub":"112'66","last_deep_scrub_stamp":"2024-01-16T15:28:20.564586+0000","last_clean_scrub_stamp":"2024-01-16T15:28:20.564586+0000","objects_scrubbed":13,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:49:24.471110+0000","scrub_duration":0.12569713499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46137466,"num_objects":13,"num_object_clones":1,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":7,"num_read_kb":28672,"num_write":13,"num_write_kb":45057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.884673+0000","last_change":"2024-01-16T15:28:10.884720+0000","last_active":"2024-01-16T15:28:10.884673+0000","last_peered":"2024-01-16T15:28:10.884673+0000","last_clean":"2024-01-16T15:28:10.884673+0000","last_became_active":"2024-01-16T15:19:23.612274+0000","last_became_peered":"2024-01-16T15:19:23.612274+0000","last_unstale":"2024-01-16T15:28:10.884673+0000","last_undegraded":"2024-01-16T15:28:10.884673+0000","last_fullsized":"2024-01-16T15:28:10.884673+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.884468+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.884468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T18:40:06.507427+0000","scrub_duration":0.0032287930000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"112'82","reported_seq":115,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:17.481473+0000","last_change":"2024-01-16T15:28:17.481559+0000","last_active":"2024-01-16T15:28:17.481473+0000","last_peered":"2024-01-16T15:28:17.481473+0000","last_clean":"2024-01-16T15:28:17.481473+0000","last_became_active":"2024-01-16T15:19:24.626009+0000","last_became_peered":"2024-01-16T15:19:24.626009+0000","last_unstale":"2024-01-16T15:28:17.481473+0000","last_undegraded":"2024-01-16T15:28:17.481473+0000","last_fullsized":"2024-01-16T15:28:17.481473+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'82","last_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_deep_scrub":"112'82","last_deep_scrub_stamp":"2024-01-16T15:28:17.481296+0000","last_clean_scrub_stamp":"2024-01-16T15:28:17.481296+0000","objects_scrubbed":17,"log_size":82,"log_dups_size":0,"ondisk_log_size":82,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:10:01.224547+0000","scrub_duration":0.21059523299999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71303168,"num_objects":17,"num_object_clones":0,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":7,"num_read_kb":28672,"num_write":17,"num_write_kb":69632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.256061+0000","last_change":"2024-01-16T15:28:18.256178+0000","last_active":"2024-01-16T15:28:18.256061+0000","last_peered":"2024-01-16T15:28:18.256061+0000","last_clean":"2024-01-16T15:28:18.256061+0000","last_became_active":"2024-01-16T15:19:23.623304+0000","last_became_peered":"2024-01-16T15:19:23.623304+0000","last_unstale":"2024-01-16T15:28:18.256061+0000","last_undegraded":"2024-01-16T15:28:18.256061+0000","last_fullsized":"2024-01-16T15:28:18.256061+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:18.255843+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.255843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:37:27.175270+0000","scrub_duration":0.0036640700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"112'85","reported_seq":119,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:37.411650+0000","last_change":"2024-01-16T15:28:37.411773+0000","last_active":"2024-01-16T15:28:37.411650+0000","last_peered":"2024-01-16T15:28:37.411650+0000","last_clean":"2024-01-16T15:28:37.411650+0000","last_became_active":"2024-01-16T15:19:24.608654+0000","last_became_peered":"2024-01-16T15:19:24.608654+0000","last_unstale":"2024-01-16T15:28:37.411650+0000","last_undegraded":"2024-01-16T15:28:37.411650+0000","last_fullsized":"2024-01-16T15:28:37.411650+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'85","last_scrub_stamp":"2024-01-16T15:28:37.411457+0000","last_deep_scrub":"112'85","last_deep_scrub_stamp":"2024-01-16T15:28:37.411457+0000","last_clean_scrub_stamp":"2024-01-16T15:28:37.411457+0000","objects_scrubbed":21,"log_size":85,"log_dups_size":0,"ondisk_log_size":85,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T19:04:09.422228+0000","scrub_duration":0.22994655999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:10.221495+0000","last_change":"2024-01-16T15:28:10.221583+0000","last_active":"2024-01-16T15:28:10.221495+0000","last_peered":"2024-01-16T15:28:10.221495+0000","last_clean":"2024-01-16T15:28:10.221495+0000","last_became_active":"2024-01-16T15:19:23.618880+0000","last_became_peered":"2024-01-16T15:19:23.618880+0000","last_unstale":"2024-01-16T15:28:10.221495+0000","last_undegraded":"2024-01-16T15:28:10.221495+0000","last_fullsized":"2024-01-16T15:28:10.221495+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:10.221197+0000","last_clean_scrub_stamp":"2024-01-16T15:28:10.221197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T17:27:58.612687+0000","scrub_duration":0.0041312190000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"112'94","reported_seq":129,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:35.980220+0000","last_change":"2024-01-16T15:28:35.980283+0000","last_active":"2024-01-16T15:28:35.980220+0000","last_peered":"2024-01-16T15:28:35.980220+0000","last_clean":"2024-01-16T15:28:35.980220+0000","last_became_active":"2024-01-16T15:19:24.624644+0000","last_became_peered":"2024-01-16T15:19:24.624644+0000","last_unstale":"2024-01-16T15:28:35.980220+0000","last_undegraded":"2024-01-16T15:28:35.980220+0000","last_fullsized":"2024-01-16T15:28:35.980220+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'94","last_scrub_stamp":"2024-01-16T15:28:35.980111+0000","last_deep_scrub":"112'94","last_deep_scrub_stamp":"2024-01-16T15:28:35.980111+0000","last_clean_scrub_stamp":"2024-01-16T15:28:35.980111+0000","objects_scrubbed":21,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-18T01:55:32.046445+0000","scrub_duration":0.287987259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88080384,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":21,"num_write_kb":86016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"112'32","reported_seq":89,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:14.827087+0000","last_change":"2024-01-16T15:28:14.827144+0000","last_active":"2024-01-16T15:28:14.827087+0000","last_peered":"2024-01-16T15:28:14.827087+0000","last_clean":"2024-01-16T15:28:14.827087+0000","last_became_active":"2024-01-16T15:19:23.628815+0000","last_became_peered":"2024-01-16T15:19:23.628815+0000","last_unstale":"2024-01-16T15:28:14.827087+0000","last_undegraded":"2024-01-16T15:28:14.827087+0000","last_fullsized":"2024-01-16T15:28:14.827087+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'32","last_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_deep_scrub":"112'32","last_deep_scrub_stamp":"2024-01-16T15:28:14.826899+0000","last_clean_scrub_stamp":"2024-01-16T15:28:14.826899+0000","objects_scrubbed":1,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:16:55.047969+0000","scrub_duration":0.0075779699999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3745,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":30,"num_write_kb":30,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"112'88","reported_seq":122,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:25.118029+0000","last_change":"2024-01-16T15:28:25.118081+0000","last_active":"2024-01-16T15:28:25.118029+0000","last_peered":"2024-01-16T15:28:25.118029+0000","last_clean":"2024-01-16T15:28:25.118029+0000","last_became_active":"2024-01-16T15:19:24.619917+0000","last_became_peered":"2024-01-16T15:19:24.619917+0000","last_unstale":"2024-01-16T15:28:25.118029+0000","last_undegraded":"2024-01-16T15:28:25.118029+0000","last_fullsized":"2024-01-16T15:28:25.118029+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'88","last_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_deep_scrub":"112'88","last_deep_scrub_stamp":"2024-01-16T15:28:25.117955+0000","last_clean_scrub_stamp":"2024-01-16T15:28:25.117955+0000","objects_scrubbed":23,"log_size":88,"log_dups_size":0,"ondisk_log_size":88,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:27:56.787030+0000","scrub_duration":0.17487287100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":96468992,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":23,"num_write_kb":94208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":28,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:37.058260+0000","last_change":"2024-01-16T15:28:37.058260+0000","last_active":"2024-01-16T15:28:37.058260+0000","last_peered":"2024-01-16T15:28:37.058260+0000","last_clean":"2024-01-16T15:28:37.058260+0000","last_became_active":"2024-01-16T15:19:23.619982+0000","last_became_peered":"2024-01-16T15:19:23.619982+0000","last_unstale":"2024-01-16T15:28:37.058260+0000","last_undegraded":"2024-01-16T15:28:37.058260+0000","last_fullsized":"2024-01-16T15:28:37.058260+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:37.057940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:37.057940+0000","last_clean_scrub_stamp":"2024-01-16T15:28:37.057940+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T16:42:28.720608+0000","scrub_duration":0.0036757579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"112'68","reported_seq":102,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:19.884824+0000","last_change":"2024-01-16T15:28:19.884824+0000","last_active":"2024-01-16T15:28:19.884824+0000","last_peered":"2024-01-16T15:28:19.884824+0000","last_clean":"2024-01-16T15:28:19.884824+0000","last_became_active":"2024-01-16T15:19:24.620937+0000","last_became_peered":"2024-01-16T15:19:24.620937+0000","last_unstale":"2024-01-16T15:28:19.884824+0000","last_undegraded":"2024-01-16T15:28:19.884824+0000","last_fullsized":"2024-01-16T15:28:19.884824+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'68","last_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_deep_scrub":"112'68","last_deep_scrub_stamp":"2024-01-16T15:28:19.884357+0000","last_clean_scrub_stamp":"2024-01-16T15:28:19.884357+0000","objects_scrubbed":14,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T23:58:10.532387+0000","scrub_duration":0.15341920100000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58720256,"num_objects":14,"num_object_clones":0,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":8,"num_read_kb":32768,"num_write":14,"num_write_kb":57344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":27,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:40.314517+0000","last_change":"2024-01-16T15:28:40.314517+0000","last_active":"2024-01-16T15:28:40.314517+0000","last_peered":"2024-01-16T15:28:40.314517+0000","last_clean":"2024-01-16T15:28:40.314517+0000","last_became_active":"2024-01-16T15:19:23.627565+0000","last_became_peered":"2024-01-16T15:19:23.627565+0000","last_unstale":"2024-01-16T15:28:40.314517+0000","last_undegraded":"2024-01-16T15:28:40.314517+0000","last_fullsized":"2024-01-16T15:28:40.314517+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-01-16T15:28:40.314218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-01-16T15:28:40.314218+0000","last_clean_scrub_stamp":"2024-01-16T15:28:40.314218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:40:28.132750+0000","scrub_duration":0.0055196840000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"112'119","reported_seq":272,"reported_epoch":112,"state":"active+clean","last_fresh":"2024-01-16T15:28:18.585009+0000","last_change":"2024-01-16T15:28:18.585089+0000","last_active":"2024-01-16T15:28:18.585009+0000","last_peered":"2024-01-16T15:28:18.585009+0000","last_clean":"2024-01-16T15:28:18.585009+0000","last_became_active":"2024-01-16T14:59:48.052905+0000","last_became_peered":"2024-01-16T14:59:48.052905+0000","last_unstale":"2024-01-16T15:28:18.585009+0000","last_undegraded":"2024-01-16T15:28:18.585009+0000","last_fullsized":"2024-01-16T15:28:18.585009+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'119","last_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_deep_scrub":"112'119","last_deep_scrub_stamp":"2024-01-16T15:28:18.584815+0000","last_clean_scrub_stamp":"2024-01-16T15:28:18.584815+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-01-17T20:44:01.708690+0000","scrub_duration":0.06564681,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":5368711180,"num_objects":1290,"num_object_clones":1,"num_object_copies":2580,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1290,"num_whiteouts":1,"num_read":543,"num_read_kb":2097183,"num_write":1350,"num_write_kb":5242919,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10737491968,"data_stored":10737422360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5648,"ondisk_log_size":5648,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":1201525,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":30,"num_read_kb":30,"num_write":471,"num_write_kb":1412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5593,"num_omap_keys":21,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2465792,"data_stored":2403050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":15,"seq":64424509799,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1215512,"kb_used_data":1144144,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":93156328,"statfs":{"total":96636764160,"available":95392079872,"internally_reserved":0,"allocated":1171603456,"data_stored":1170811999,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73071046},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.1439999999999999,"15min":1.2210000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.593,"5min":1.7749999999999999,"15min":19.224},"last":1.472},{"interface":"front","average":{"1min":1.238,"5min":1.2250000000000001,"15min":1.2070000000000001},"min":{"1min":0.873,"5min":0.73999999999999999,"15min":0.57399999999999995},"max":{"1min":1.6140000000000001,"5min":1.694,"15min":5.085},"last":0.96599999999999997}]},{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.194,"15min":1.218},"min":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.374},"max":{"1min":1.641,"5min":1.837,"15min":3.7069999999999999},"last":1.254},{"interface":"front","average":{"1min":1.204,"5min":1.2909999999999999,"15min":1.3640000000000001},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.536,"5min":1.8819999999999999,"15min":19.178000000000001},"last":1.218}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.224,"5min":1.3080000000000001,"15min":1.383},"min":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"max":{"1min":1.8919999999999999,"5min":1.8919999999999999,"15min":19.126000000000001},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.325,"5min":1.3200000000000001,"15min":1.3160000000000001},"min":{"1min":1.0669999999999999,"5min":0.88700000000000001,"15min":0.66700000000000004},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":2.9750000000000001},"last":1.4199999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.2689999999999999,"15min":1.3200000000000001},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.76800000000000002},"max":{"1min":1.6599999999999999,"5min":1.73,"15min":3.7999999999999998},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.4390000000000001,"15min":1.5269999999999999},"min":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.63500000000000001},"max":{"1min":1.972,"5min":1.972,"15min":19.268999999999998},"last":1.583}]},{"osd":4,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.2470000000000001,"15min":1.264},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.58799999999999997},"max":{"1min":1.794,"5min":1.8169999999999999,"15min":2.2519999999999998},"last":1.298},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.369,"15min":1.397},"min":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.58299999999999996},"max":{"1min":2.452,"5min":2.452,"15min":9.7059999999999995},"last":1.032}]},{"osd":5,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.5329999999999999,"15min":1.548},"min":{"1min":0.99099999999999999,"5min":0.97599999999999998,"15min":0.79400000000000004},"max":{"1min":2.8029999999999999,"5min":2.8029999999999999,"15min":3.9129999999999998},"last":1.458},{"interface":"front","average":{"1min":1.514,"5min":1.4570000000000001,"15min":1.532},"min":{"1min":0.86799999999999999,"5min":0.82699999999999996,"15min":0.55100000000000005},"max":{"1min":2.9009999999999998,"5min":2.9009999999999998,"15min":19.036999999999999},"last":1.734}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.456,"15min":1.444},"min":{"1min":1.0069999999999999,"5min":0.88500000000000001,"15min":0.54800000000000004},"max":{"1min":1.8580000000000001,"5min":1.8580000000000001,"15min":4.2240000000000002},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5409999999999999,"15min":1.5720000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.65100000000000002},"max":{"1min":2.3090000000000002,"5min":2.3090000000000002,"15min":4.0709999999999997},"last":1.381}]}]},{"osd":4,"up_from":15,"seq":64424509801,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1612828,"kb_used_data":1541460,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":92759012,"statfs":{"total":96636764160,"available":94985228288,"internally_reserved":0,"allocated":1578455040,"data_stored":1577659997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73071045},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.194,"5min":1.3029999999999999,"15min":1.23},"min":{"1min":0.755,"5min":0.65700000000000003,"15min":0.65000000000000002},"max":{"1min":1.5900000000000001,"5min":6.5960000000000001,"15min":6.5960000000000001},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.194,"15min":1.1879999999999999},"min":{"1min":0.93400000000000005,"5min":0.76100000000000001,"15min":0.69799999999999995},"max":{"1min":1.4930000000000001,"5min":3.1030000000000002,"15min":3.1030000000000002},"last":1.1180000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.238,"15min":1.2190000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.71899999999999997},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.0510000000000002},"last":1.415},{"interface":"front","average":{"1min":1.23,"5min":1.28,"15min":1.2909999999999999},"min":{"1min":0.72499999999999998,"5min":0.72299999999999998,"15min":0.59799999999999998},"max":{"1min":1.752,"5min":6.0380000000000003,"15min":6.0380000000000003},"last":1.8500000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2609999999999999,"5min":1.3240000000000001,"15min":1.3160000000000001},"min":{"1min":0.97199999999999998,"5min":0.69499999999999995,"15min":0.45600000000000002},"max":{"1min":1.6339999999999999,"5min":1.78,"15min":1.843},"last":1.6870000000000001},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.341,"15min":1.349},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.52600000000000002},"max":{"1min":1.653,"5min":6.1100000000000003,"15min":6.1100000000000003},"last":1.738}]},{"osd":3,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.4419999999999999,"15min":1.395},"min":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.76200000000000001},"max":{"1min":1.8879999999999999,"5min":12.574999999999999,"15min":12.574999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.377,"5min":1.3839999999999999,"15min":1.391},"min":{"1min":1.056,"5min":0.97899999999999998,"15min":0.78500000000000003},"max":{"1min":1.71,"5min":1.74,"15min":2.0299999999999998},"last":1.7869999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":1.427,"15min":1.429},"min":{"1min":0.95199999999999996,"5min":0.81499999999999995,"15min":0.65900000000000003},"max":{"1min":1.6839999999999999,"5min":6.165,"15min":6.165},"last":1.944},{"interface":"front","average":{"1min":1.502,"5min":1.4450000000000001,"15min":1.474},"min":{"1min":0.95399999999999996,"5min":0.79000000000000004,"15min":0.78900000000000003},"max":{"1min":1.8129999999999999,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.379}]},{"osd":6,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.365,"15min":1.4259999999999999},"min":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.497},"max":{"1min":1.716,"5min":1.903,"15min":8.9600000000000009},"last":1.879},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.54,"15min":1.524},"min":{"1min":1.1579999999999999,"5min":1.0329999999999999,"15min":0.56100000000000005},"max":{"1min":1.887,"5min":6.2300000000000004,"15min":6.2300000000000004},"last":2.0459999999999998}]},{"osd":7,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4339999999999999,"15min":1.4790000000000001},"min":{"1min":1.179,"5min":1.0469999999999999,"15min":0.89700000000000002},"max":{"1min":1.8080000000000001,"5min":3.3620000000000001,"15min":3.3620000000000001},"last":1.4830000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.518,"15min":1.5600000000000001},"min":{"1min":1.0329999999999999,"5min":0.93999999999999995,"15min":0.69599999999999995},"max":{"1min":1.923,"5min":2.0259999999999998,"15min":3.8540000000000001},"last":1.992}]}]},{"osd":7,"up_from":14,"seq":60129542503,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1896108,"kb_used_data":1820580,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92475732,"statfs":{"total":96636764160,"available":94695149568,"internally_reserved":0,"allocated":1864273920,"data_stored":1863463926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77330890},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0419999999999998,"5min":1.339,"15min":1.2649999999999999},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.65400000000000003},"max":{"1min":16.832999999999998,"5min":16.832999999999998,"15min":16.832999999999998},"last":1.415},{"interface":"front","average":{"1min":2.4359999999999999,"5min":1.456,"15min":1.304},"min":{"1min":0.86199999999999999,"5min":0.78700000000000003,"15min":0.67900000000000005},"max":{"1min":21.283000000000001,"5min":21.283000000000001,"15min":21.283000000000001},"last":1.3859999999999999}]},{"osd":1,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.254,"15min":1.272},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.61799999999999999},"max":{"1min":3.056,"5min":3.056,"15min":3.056},"last":1.4910000000000001},{"interface":"front","average":{"1min":2.2269999999999999,"5min":1.47,"15min":1.395},"min":{"1min":0.90700000000000003,"5min":0.83299999999999996,"15min":0.745},"max":{"1min":16.350000000000001,"5min":16.350000000000001,"15min":16.350000000000001},"last":1.2130000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.2610000000000001,"5min":1.5049999999999999,"15min":1.3700000000000001},"min":{"1min":0.82599999999999996,"5min":0.78600000000000003,"15min":0.57199999999999995},"max":{"1min":16.960999999999999,"5min":16.960999999999999,"15min":16.960999999999999},"last":0.88100000000000001},{"interface":"front","average":{"1min":1.4159999999999999,"5min":1.3280000000000001,"15min":1.409},"min":{"1min":0.92700000000000005,"5min":0.873,"15min":0.69299999999999995},"max":{"1min":2.9220000000000002,"5min":2.9220000000000002,"15min":19.763999999999999},"last":1.0429999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":2.9340000000000002,"5min":1.6339999999999999,"15min":1.4450000000000001},"min":{"1min":0.874,"5min":0.874,"15min":0.80600000000000005},"max":{"1min":27.920999999999999,"5min":27.920999999999999,"15min":27.920999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":2.4430000000000001,"5min":1.661,"15min":1.538},"min":{"1min":1.04,"5min":1.0169999999999999,"15min":0.76500000000000001},"max":{"1min":17.709,"5min":17.709,"15min":17.709},"last":1.254}]},{"osd":4,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":2.1779999999999999,"5min":1.5109999999999999,"15min":1.391},"min":{"1min":0.753,"5min":0.753,"15min":0.54100000000000004},"max":{"1min":17.006,"5min":17.006,"15min":17.006},"last":1.0940000000000001},{"interface":"front","average":{"1min":2.1819999999999999,"5min":1.5620000000000001,"15min":1.4710000000000001},"min":{"1min":1.0569999999999999,"5min":0.92500000000000004,"15min":0.57199999999999995},"max":{"1min":17.658000000000001,"5min":17.658000000000001,"15min":17.658000000000001},"last":1.3460000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:38 2024","interfaces":[{"interface":"back","average":{"1min":1.7110000000000001,"5min":1.425,"15min":1.361},"min":{"1min":0.90000000000000002,"5min":0.88900000000000001,"15min":0.59999999999999998},"max":{"1min":7.641,"5min":7.641,"15min":7.641},"last":1.554},{"interface":"front","average":{"1min":2.4780000000000002,"5min":1.718,"15min":1.585},"min":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":0.66000000000000003},"max":{"1min":16.911999999999999,"5min":16.911999999999999,"15min":16.911999999999999},"last":1.3}]},{"osd":6,"last update":"Tue Jan 16 15:28:46 2024","interfaces":[{"interface":"back","average":{"1min":2.3450000000000002,"5min":1.7390000000000001,"15min":1.625},"min":{"1min":1.2370000000000001,"5min":1.0760000000000001,"15min":0.67100000000000004},"max":{"1min":16.510000000000002,"5min":16.510000000000002,"15min":16.510000000000002},"last":1.1599999999999999},{"interface":"front","average":{"1min":2.3170000000000002,"5min":1.681,"15min":1.575},"min":{"1min":1.0289999999999999,"5min":1.004,"15min":0.59099999999999997},"max":{"1min":16.405999999999999,"5min":16.405999999999999,"15min":16.405999999999999},"last":1.4490000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542504,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":936860,"kb_used_data":873812,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":93434980,"statfs":{"total":96636764160,"available":95677419520,"internally_reserved":0,"allocated":894783488,"data_stored":893988539,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":64551365},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.1839999999999999,"15min":1.1830000000000001},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.32900000000000001},"max":{"1min":1.7070000000000001,"5min":2.0939999999999999,"15min":6.0720000000000001},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.123,"5min":1.1759999999999999,"15min":1.3160000000000001},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.57099999999999995},"max":{"1min":1.302,"5min":1.7649999999999999,"15min":23.199999999999999},"last":0.94399999999999995}]},{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.181,"5min":1.21,"15min":1.224},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.63400000000000001},"max":{"1min":1.736,"5min":1.736,"15min":4.2450000000000001},"last":1.6890000000000001},{"interface":"front","average":{"1min":1.2789999999999999,"5min":1.2649999999999999,"15min":1.327},"min":{"1min":0.76600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.786,"5min":1.851,"15min":17.062000000000001},"last":1.131}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.367,"15min":1.403},"min":{"1min":0.65300000000000002,"5min":0.53800000000000003,"15min":0.42499999999999999},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":5.8810000000000002},"last":1.339},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.361,"15min":1.387},"min":{"1min":0.96599999999999997,"5min":0.86099999999999999,"15min":0.60799999999999998},"max":{"1min":1.921,"5min":1.921,"15min":5.4429999999999996},"last":1.4430000000000001}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.323,"15min":1.3240000000000001},"min":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.67500000000000004},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.4769999999999999},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.363,"5min":1.397,"15min":1.4179999999999999},"min":{"1min":0.69099999999999995,"5min":0.63900000000000001,"15min":0.47399999999999998},"max":{"1min":1.742,"5min":1.7869999999999999,"15min":5.9489999999999998},"last":1.2350000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.363,"15min":1.3939999999999999},"min":{"1min":0.88300000000000001,"5min":0.82699999999999996,"15min":0.56399999999999995},"max":{"1min":1.736,"5min":2.2610000000000001,"15min":15.975},"last":1.494},{"interface":"front","average":{"1min":1.4199999999999999,"5min":1.417,"15min":1.4179999999999999},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.54000000000000004},"max":{"1min":1.9610000000000001,"5min":1.9610000000000001,"15min":5.6349999999999998},"last":1.53}]},{"osd":6,"last update":"Tue Jan 16 15:28:47 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.335,"15min":1.369},"min":{"1min":0.96999999999999997,"5min":0.94299999999999995,"15min":0.44800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":12.914},"last":1.286},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4950000000000001,"15min":1.5209999999999999},"min":{"1min":1.0780000000000001,"5min":1.006,"15min":0.65800000000000003},"max":{"1min":1.9990000000000001,"5min":2.4660000000000002,"15min":16.552},"last":1.3759999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.556,"5min":1.4750000000000001,"15min":1.51},"min":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.64300000000000002},"max":{"1min":2.2109999999999999,"5min":2.2109999999999999,"15min":13.034000000000001},"last":1.6000000000000001},{"interface":"front","average":{"1min":1.669,"5min":1.5900000000000001,"15min":1.5760000000000001},"min":{"1min":0.95799999999999996,"5min":0.95799999999999996,"15min":0.70899999999999996},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.6549999999999998},"last":1.764}]}]},{"osd":3,"up_from":14,"seq":60129542504,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1438720,"kb_used_data":1317368,"kb_used_omap":1,"kb_used_meta":121342,"kb_avail":92933120,"statfs":{"total":96636764160,"available":95163514880,"internally_reserved":0,"allocated":1348984832,"data_stored":1348170671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":124254663},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.95799999999999996,"15min":0.95699999999999996},"min":{"1min":0.56599999999999995,"5min":0.373,"15min":0.315},"max":{"1min":1.3,"5min":1.6910000000000001,"15min":2.02},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.125,"15min":1.147},"min":{"1min":0.79000000000000004,"5min":0.48299999999999998,"15min":0.42799999999999999},"max":{"1min":1.5720000000000001,"5min":1.879,"15min":7.1429999999999998},"last":1.097}]},{"osd":1,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":0.98899999999999999,"15min":1.044},"min":{"1min":0.77500000000000002,"5min":0.40799999999999997,"15min":0.33200000000000002},"max":{"1min":1.4990000000000001,"5min":1.5640000000000001,"15min":10.856999999999999},"last":0.872},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2050000000000001,"15min":1.194},"min":{"1min":0.66400000000000003,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.5589999999999999,"5min":1.823,"15min":2.0089999999999999},"last":1.1539999999999999}]},{"osd":2,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.29,"5min":1.306,"15min":1.387},"min":{"1min":0.65100000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.641,"5min":1.946,"15min":10.045999999999999},"last":1.044},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1599999999999999,"15min":1.21},"min":{"1min":0.627,"5min":0.54000000000000004,"15min":0.44800000000000001},"max":{"1min":1.97,"5min":1.97,"15min":7.9160000000000004},"last":1.4339999999999999}]},{"osd":4,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.343,"15min":1.3620000000000001},"min":{"1min":1.01,"5min":0.748,"15min":0.73799999999999999},"max":{"1min":2.4279999999999999,"5min":2.4279999999999999,"15min":4.5570000000000004},"last":1.01},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.476,"15min":1.484},"min":{"1min":1.0900000000000001,"5min":0.749,"15min":0.749},"max":{"1min":1.946,"5min":2.0920000000000001,"15min":2.165},"last":1.3280000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5509999999999999,"5min":1.3759999999999999,"15min":1.458},"min":{"1min":1.097,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":2.3660000000000001,"5min":2.3660000000000001,"15min":14.003},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5109999999999999,"15min":1.546},"min":{"1min":0.879,"5min":0.877,"15min":0.70199999999999996},"max":{"1min":2.0630000000000002,"5min":2.0630000000000002,"15min":2.2970000000000002},"last":1.5129999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.4610000000000001,"15min":1.5009999999999999},"min":{"1min":0.96199999999999997,"5min":0.875,"15min":0.85299999999999998},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":7.0060000000000002},"last":1.272},{"interface":"front","average":{"1min":1.627,"5min":1.512,"15min":1.5740000000000001},"min":{"1min":1.252,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.2719999999999998,"5min":2.2719999999999998,"15min":15.114000000000001},"last":1.4750000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:52 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.494,"15min":1.6220000000000001},"min":{"1min":1.248,"5min":0.85699999999999998,"15min":0.81899999999999995},"max":{"1min":2.1139999999999999,"5min":2.1349999999999998,"15min":12.378},"last":1.546},{"interface":"front","average":{"1min":1.806,"5min":1.617,"15min":1.6339999999999999},"min":{"1min":1.4690000000000001,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.48,"5min":2.48,"15min":4.6029999999999998},"last":1.3879999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542503,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1064088,"kb_used_data":984400,"kb_used_omap":1,"kb_used_meta":79678,"kb_avail":93307752,"statfs":{"total":96636764160,"available":95547138048,"internally_reserved":0,"allocated":1008025600,"data_stored":1007237957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":81590725},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91300000000000003,"15min":0.97699999999999998},"min":{"1min":0.626,"5min":0.46200000000000002,"15min":0.32400000000000001},"max":{"1min":1.3720000000000001,"5min":1.4770000000000001,"15min":17.77},"last":0.74299999999999999},{"interface":"front","average":{"1min":1.034,"5min":1.139,"15min":1.121},"min":{"1min":0.63300000000000001,"5min":0.49199999999999999,"15min":0.40300000000000002},"max":{"1min":1.708,"5min":1.913,"15min":7.9960000000000004},"last":1.1040000000000001}]},{"osd":1,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1120000000000001,"15min":1.1020000000000001},"min":{"1min":0.66800000000000004,"5min":0.52600000000000002,"15min":0.39400000000000002},"max":{"1min":2.145,"5min":2.294,"15min":2.294},"last":1.0369999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.1799999999999999,"15min":1.1819999999999999},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.48399999999999999},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":4.3819999999999997},"last":0.97999999999999998}]},{"osd":3,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.171,"15min":1.2789999999999999},"min":{"1min":0.63700000000000001,"5min":0.61199999999999999,"15min":0.44700000000000001},"max":{"1min":1.804,"5min":1.883,"15min":18.838000000000001},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4830000000000001,"15min":1.554},"min":{"1min":0.79300000000000004,"5min":0.66200000000000003,"15min":0.58099999999999996},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":19.375},"last":1.569}]},{"osd":4,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.246,"5min":1.327,"15min":1.429},"min":{"1min":0.80800000000000005,"5min":0.80600000000000005,"15min":0.71799999999999997},"max":{"1min":1.754,"5min":2.242,"15min":23.225000000000001},"last":1.159},{"interface":"front","average":{"1min":1.365,"5min":1.466,"15min":1.484},"min":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.72699999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":6.359},"last":1.292}]},{"osd":5,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.4390000000000001,"15min":1.494},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.68100000000000005},"max":{"1min":2.165,"5min":2.3170000000000002,"15min":20.157},"last":1.427},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.5589999999999999,"15min":1.593},"min":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.746},"max":{"1min":1.9430000000000001,"5min":13.279999999999999,"15min":25.004000000000001},"last":1.3240000000000001}]},{"osd":6,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.5069999999999999,"15min":1.5900000000000001},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.83599999999999997},"max":{"1min":2.0579999999999998,"5min":2.3639999999999999,"15min":21.030000000000001},"last":1.379},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.595,"15min":1.7130000000000001},"min":{"1min":0.99199999999999999,"5min":0.85799999999999998,"15min":0.76200000000000001},"max":{"1min":2.238,"5min":2.4009999999999998,"15min":28.760999999999999},"last":1.484}]},{"osd":7,"last update":"Tue Jan 16 15:28:31 2024","interfaces":[{"interface":"back","average":{"1min":1.49,"5min":1.5329999999999999,"15min":1.6579999999999999},"min":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":0.82099999999999995},"max":{"1min":1.9930000000000001,"5min":2.4550000000000001,"15min":19.757000000000001},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.583,"15min":1.587},"min":{"1min":1.1259999999999999,"5min":0.83399999999999996,"15min":0.78600000000000003},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":4.3419999999999996},"last":1.5029999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542505,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1203292,"kb_used_data":1127764,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":93168548,"statfs":{"total":96636764160,"available":95404593152,"internally_reserved":0,"allocated":1154830336,"data_stored":1154038460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":1.0660000000000001,"15min":0.98699999999999999},"min":{"1min":0.71599999999999997,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.296,"5min":13.42,"15min":13.42},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.1359999999999999,"15min":1.123},"min":{"1min":0.50900000000000001,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.6339999999999999,"5min":3.915,"15min":3.915},"last":1.3220000000000001}]},{"osd":2,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":1.0269999999999999,"15min":1.0900000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":1.53,"5min":1.53,"15min":3.6819999999999999},"last":1.24},{"interface":"front","average":{"1min":1.284,"5min":1.2729999999999999,"15min":1.335},"min":{"1min":0.64100000000000001,"5min":0.46999999999999997,"15min":0.40100000000000002},"max":{"1min":1.736,"5min":1.736,"15min":3.5600000000000001},"last":1.0369999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.2589999999999999,"15min":1.298},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.6950000000000001,"5min":1.875,"15min":2.238},"last":1.173},{"interface":"front","average":{"1min":1.133,"5min":1.127,"15min":1.1919999999999999},"min":{"1min":0.46600000000000003,"5min":0.436,"15min":0.436},"max":{"1min":1.7509999999999999,"5min":1.7589999999999999,"15min":2.9750000000000001},"last":0.97599999999999998}]},{"osd":4,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2549999999999999,"15min":1.327},"min":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.6739999999999999,"5min":1.8120000000000001,"15min":3.3420000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.544,"15min":1.528},"min":{"1min":1.077,"5min":0.78300000000000003,"15min":0.78300000000000003},"max":{"1min":1.679,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":1.4390000000000001}]},{"osd":5,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.4850000000000001,"5min":1.419,"15min":1.4450000000000001},"min":{"1min":1.095,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":3.98},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.417,"15min":1.4530000000000001},"min":{"1min":0.97199999999999998,"5min":0.92900000000000005,"15min":0.84499999999999997},"max":{"1min":1.78,"5min":1.893,"15min":3.3940000000000001},"last":1.371}]},{"osd":6,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.6140000000000001,"5min":1.5089999999999999,"15min":1.583},"min":{"1min":1.2549999999999999,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":2.1589999999999998,"5min":2.1589999999999998,"15min":16.48},"last":1.409},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4930000000000001,"15min":1.536},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.90000000000000002},"max":{"1min":1.978,"5min":1.978,"15min":3.8180000000000001},"last":1.5009999999999999}]},{"osd":7,"last update":"Tue Jan 16 15:28:45 2024","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.5389999999999999,"15min":1.613},"min":{"1min":0.94199999999999995,"5min":0.92200000000000004,"15min":0.68899999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":7.0499999999999998},"last":1.732},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.607,"15min":1.6499999999999999},"min":{"1min":1.0920000000000001,"5min":0.91700000000000004,"15min":0.89900000000000002},"max":{"1min":1.931,"5min":2.258,"15min":3.7839999999999998},"last":1.5780000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542503,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1766216,"kb_used_data":1690688,"kb_used_omap":1,"kb_used_meta":75518,"kb_avail":92605624,"statfs":{"total":96636764160,"available":94828158976,"internally_reserved":0,"allocated":1731264512,"data_stored":1730442525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77330885},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":0.93500000000000005,"15min":1.0449999999999999},"min":{"1min":0.69199999999999995,"5min":0.496,"15min":0.376},"max":{"1min":1.5509999999999999,"5min":1.665,"15min":15.776},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.131,"5min":1.1559999999999999,"15min":1.1739999999999999},"min":{"1min":0.68799999999999994,"5min":0.66400000000000003,"15min":0.38500000000000001},"max":{"1min":3.2599999999999998,"5min":3.2599999999999998,"15min":17.620999999999999},"last":1.278}]},{"osd":2,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1299999999999999,"5min":1.135,"15min":1.226},"min":{"1min":0.82899999999999996,"5min":0.57399999999999995,"15min":0.42399999999999999},"max":{"1min":1.629,"5min":1.671,"15min":15.997999999999999},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.2430000000000001,"15min":1.2030000000000001},"min":{"1min":0.89200000000000002,"5min":0.60299999999999998,"15min":0.45000000000000001},"max":{"1min":3.4769999999999999,"5min":3.4769999999999999,"15min":3.4769999999999999},"last":1.0269999999999999}]},{"osd":3,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.1080000000000001,"15min":1.179},"min":{"1min":0.77900000000000003,"5min":0.60399999999999998,"15min":0.40000000000000002},"max":{"1min":1.7190000000000001,"5min":1.8029999999999999,"15min":12.029999999999999},"last":1.323},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.345,"15min":1.4339999999999999},"min":{"1min":0.86599999999999999,"5min":0.66600000000000004,"15min":0.51200000000000001},"max":{"1min":1.6459999999999999,"5min":1.827,"15min":16.126000000000001},"last":1.5660000000000001}]},{"osd":4,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4750000000000001,"15min":1.5740000000000001},"min":{"1min":1.0389999999999999,"5min":0.78900000000000003,"15min":0.78800000000000003},"max":{"1min":1.8640000000000001,"5min":2.0899999999999999,"15min":16.283000000000001},"last":1.3759999999999999},{"interface":"front","average":{"1min":1.508,"5min":1.46,"15min":1.4279999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":3.5219999999999998,"5min":3.5219999999999998,"15min":3.5219999999999998},"last":1.1759999999999999}]},{"osd":5,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.379,"15min":1.3979999999999999},"min":{"1min":0.77800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.736,"5min":2.157,"15min":5.5609999999999999},"last":1.625},{"interface":"front","average":{"1min":1.6120000000000001,"5min":1.5609999999999999,"15min":1.649},"min":{"1min":1.157,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.3159999999999998,"5min":2.3159999999999998,"15min":16.501999999999999},"last":1.7989999999999999}]},{"osd":6,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.4390000000000001,"15min":1.4510000000000001},"min":{"1min":1.0569999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"max":{"1min":1.7370000000000001,"5min":2.0499999999999998,"15min":5.4509999999999996},"last":1.512},{"interface":"front","average":{"1min":1.601,"5min":1.6259999999999999,"15min":1.722},"min":{"1min":1.2470000000000001,"5min":0.77200000000000002,"15min":0.76600000000000001},"max":{"1min":1.8979999999999999,"5min":2.351,"15min":16.395},"last":1.7230000000000001}]},{"osd":7,"last update":"Tue Jan 16 15:28:34 2024","interfaces":[{"interface":"back","average":{"1min":1.5900000000000001,"5min":1.591,"15min":1.5780000000000001},"min":{"1min":1.22,"5min":0.88100000000000001,"15min":0.76800000000000002},"max":{"1min":3.423,"5min":3.423,"15min":3.8239999999999998},"last":1.444},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.579,"15min":1.5860000000000001},"min":{"1min":1.1419999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.0600000000000001,"5min":2.4409999999999998,"15min":5.5060000000000002},"last":1.6870000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1216512,"data_stored":1197094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":3745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1212416,"data_stored":1197052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1728069632,"data_stored":1728054072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1153433600,"data_stored":1153433600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1006637056,"data_stored":1006633221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1346387968,"data_stored":1346372628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1577070592,"data_stored":1577059006,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":893390848,"data_stored":893386944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1170214912,"data_stored":1170210938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1862287360,"data_stored":1862271951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-01-16T15:29:33.233 DEBUG:teuthology.orchestra.run.smithi053:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-01-16T15:29:33.483 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.483+0000 7f55d7200640 1 Processor -- start 2024-01-16T15:29:33.484 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.484+0000 7f55d7200640 1 -- start start 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 0x7f55d80d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d7200640 1 --2- >> v2:172.21.15.146:3300/0 conn(0x7f55d80b41e0 0x7f55d80cfde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d7200640 1 -- --> v1:172.21.15.53:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d80d0320 con 0x7f55d80cf9a0 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d7200640 1 -- --> v2:172.21.15.146:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d80b2730 con 0x7f55d80b41e0 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80d1ec0 con 0x7f55d815d4b0 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d5400640 1 --1- >> v1:172.21.15.53:6789/0 conn(0x7f55d80cf9a0 0x7f55d80cddf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.53:6789/0 says I am v1:172.21.15.53:59042/0 (socket says 172.21.15.53:59042) 2024-01-16T15:29:33.485 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.485+0000 7f55d5400640 1 -- 172.21.15.53:0/2868520627 learned_addr learned my addr 172.21.15.53:0/2868520627 (peer_addr_for_me v1:172.21.15.53:0/0) 2024-01-16T15:29:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 <== mon.0 v1:172.21.15.53:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532828564 0 0) 0x7f55d80d0320 con 0x7f55d80cf9a0 2024-01-16T15:29:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 --> v1:172.21.15.53:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55b8003520 con 0x7f55d80cf9a0 2024-01-16T15:29:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d5e00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 0x7f55d80d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d6800640 1 --2- 172.21.15.53:0/2868520627 >> v2:172.21.15.146:3300/0 conn(0x7f55d80b41e0 0x7f55d80cfde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.488 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 <== mon.0 v1:172.21.15.53:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2376598365 0 0) 0x7f55b8003520 con 0x7f55d80cf9a0 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 msgr2=0x7f55d80d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 0x7f55d80d1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 >> v2:172.21.15.146:3300/0 conn(0x7f55d80b41e0 msgr2=0x7f55d80cfde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 --2- 172.21.15.53:0/2868520627 >> v2:172.21.15.146:3300/0 conn(0x7f55d80b41e0 0x7f55d80cfde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 --> v1:172.21.15.53:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d80b29d0 con 0x7f55d80cf9a0 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 <== mon.0 v1:172.21.15.53:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1103037742 0 0) 0x7f55c80019f0 con 0x7f55d80cf9a0 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 >> v1:172.21.15.53:6789/0 conn(0x7f55d80cf9a0 legacy=0x7f55d80cddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.489 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55b8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 0x7f55b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55b8004d00 0x7f55b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b2730 con 0x7f55b8003970 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80d1ec0 con 0x7f55b8004d00 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55b800ba60 con 0x7f55d815d4b0 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.486+0000 7f55d5e00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55b8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.490 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d5400640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 0x7f55b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d5e00640 1 -- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 msgr2=0x7f55b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d5e00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815d4b0 0x7f55b80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d5e00640 1 -- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55b8004d00 msgr2=0x7f55b8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d6800640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55b8004d00 0x7f55b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d5e00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55b8004d00 0x7f55b8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.487+0000 7f55d5e00640 1 -- 172.21.15.53:0/2868520627 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55b8003520 con 0x7f55b8003970 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.488+0000 7f55d5e00640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55b8003d90 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f55c000ba30 tx=0x7f55b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea83012cd37bdf0c server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:33.491 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.488+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0032650 con 0x7f55b8003970 2024-01-16T15:29:33.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.488+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c00327b0 con 0x7f55b8003970 2024-01-16T15:29:33.492 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.488+0000 7f55d4a00640 1 -- 172.21.15.53:0/2868520627 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c0032b80 con 0x7f55b8003970 2024-01-16T15:29:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d7200640 1 -- 172.21.15.53:0/2868520627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 msgr2=0x7f55b8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d7200640 1 --2- 172.21.15.53:0/2868520627 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55b8003d90 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f55c000ba30 tx=0x7f55b80057f0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d6800640 1 -- 172.21.15.53:0/2868520627 reap_dead start 2024-01-16T15:29:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d7200640 1 -- 172.21.15.53:0/2868520627 shutdown_connections 2024-01-16T15:29:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d7200640 1 -- 172.21.15.53:0/2868520627 >> 172.21.15.53:0/2868520627 conn(0x7f55d80d3000 msgr2=0x7f55d80d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:33.495 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d7200640 1 -- 172.21.15.53:0/2868520627 shutdown_connections 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.489+0000 7f55d7200640 1 -- 172.21.15.53:0/2868520627 wait complete. 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.491+0000 7f55d7200640 1 Processor -- start 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.491+0000 7f55d7200640 1 -- start start 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d7200640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55d8159c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d7200640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815a380 0x7f55d809cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d7200640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55d809e430 0x7f55d809e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d7200640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d815a190 con 0x7f55b8003970 2024-01-16T15:29:33.496 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d7200640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d809edf0 con 0x7f55d809e430 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d7200640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d809d5c0 con 0x7f55d815a380 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55d8159c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d5e00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55d8159c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59648/0 (socket says 172.21.15.53:59648) 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.492+0000 7f55d5e00640 1 -- 172.21.15.53:0/592222711 learned_addr learned my addr 172.21.15.53:0/592222711 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d6800640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55d809e430 0x7f55d809e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5e00640 1 -- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815a380 msgr2=0x7f55d809cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5400640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815a380 0x7f55d809cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.497 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5e00640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815a380 0x7f55d809cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5e00640 1 -- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55d809e430 msgr2=0x7f55d809e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5e00640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55d809e430 0x7f55d809e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5e00640 1 -- 172.21.15.53:0/592222711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d815da10 con 0x7f55b8003970 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5400640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815a380 0x7f55d809cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d6800640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55d809e430 0x7f55d809e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d5e00640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55d8159c50 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f55c000b950 tx=0x7f55c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c003b620 con 0x7f55b8003970 2024-01-16T15:29:33.498 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55c003b780 con 0x7f55b8003970 2024-01-16T15:29:33.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55c003bb30 con 0x7f55b8003970 2024-01-16T15:29:33.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.493+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d809d8a0 con 0x7f55b8003970 2024-01-16T15:29:33.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.494+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d809de60 con 0x7f55b8003970 2024-01-16T15:29:33.499 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.495+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d8067450 con 0x7f55b8003970 2024-01-16T15:29:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.496+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7f55c002e070 con 0x7f55b8003970 2024-01-16T15:29:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.497+0000 7f55ce000640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f55b006c4f0 0x7f55b006e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.497+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7f55c00aad70 con 0x7f55b8003970 2024-01-16T15:29:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.498+0000 7f55d5400640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f55b006c4f0 0x7f55b006e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.500 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.499+0000 7f55d5400640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f55b006c4f0 0x7f55b006e9b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f55c8000f80 tx=0x7f55c8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:33.501 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.502+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7f55c0077350 con 0x7f55b8003970 2024-01-16T15:29:33.696 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.696+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f55d80afd70 con 0x7f55b8003970 2024-01-16T15:29:33.703 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.704+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v45)=0 v45) v1 ==== 123+0+0 (secure 0 0 0) 0x7f55c0036070 con 0x7f55b8003970 2024-01-16T15:29:33.706 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.707+0000 7f55ce000640 1 -- 172.21.15.53:0/592222711 <== mon.0 v2:172.21.15.53:3300/0 8 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f55c0077350 con 0x7f55b8003970 2024-01-16T15:29:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f55b006c4f0 msgr2=0x7f55b006e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.708 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f55b006c4f0 0x7f55b006e9b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f55c8000f80 tx=0x7f55c8009040 comp rx=0 tx=0).stop 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 msgr2=0x7f55d8159c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55d8159c50 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f55c000b950 tx=0x7f55c0004730 comp rx=0 tx=0).stop 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 shutdown_connections 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7f55b006c4f0 0x7f55b006e9b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7f55d809e430 0x7f55d809e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7f55d815a380 0x7f55d809cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 --2- 172.21.15.53:0/592222711 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7f55b8003970 0x7f55d8159c50 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.709 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 >> 172.21.15.53:0/592222711 conn(0x7f55d80d3000 msgr2=0x7f55d80b1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:33.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 shutdown_connections 2024-01-16T15:29:33.710 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.708+0000 7f55d7200640 1 -- 172.21.15.53:0/592222711 wait complete. 2024-01-16T15:29:33.728 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-01-16T15:29:33.728 DEBUG:teuthology.orchestra.run.smithi053:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-01-16T15:29:33.744 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:29:33.744+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:29:33.745 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:29:33.744+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:29:33.745 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:29:33.744+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:29:33.745 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:29:33.744+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:29:33.745 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-01-16T15:29:33.744+0000 7f613b400640 -1 client.0 error registering admin socket command: (17) File exists 2024-01-16T15:29:33.938 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.939+0000 7fa45c600640 1 Processor -- start 2024-01-16T15:29:33.939 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.939+0000 7fa45c600640 1 -- start start 2024-01-16T15:29:33.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.940+0000 7fa45c600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413df80 0x7fa45415bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.940+0000 7fa45c600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa4540cf9a0 0x7fa4540cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45c600640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa45406c470 con 0x7fa45413e360 2024-01-16T15:29:33.941 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45c600640 1 -- --> v2:172.21.15.53:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45406c750 con 0x7fa4540cf9a0 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45c600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45406ca30 con 0x7fa45413df80 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa4540cf9a0 0x7fa4540cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 --2- >> v2:172.21.15.53:3300/0 conn(0x7fa4540cf9a0 0x7fa4540cd840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59658/0 (socket says 172.21.15.53:59658) 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 -- 172.21.15.53:0/1968399666 learned_addr learned my addr 172.21.15.53:0/1968399666 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa459c00640 1 --2- 172.21.15.53:0/1968399666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413df80 0x7fa45415bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 -- 172.21.15.53:0/1968399666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413df80 msgr2=0x7fa45415bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 --2- 172.21.15.53:0/1968399666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413df80 0x7fa45415bee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 -- 172.21.15.53:0/1968399666 >> v1:172.21.15.146:6789/0 conn(0x7fa45413e360 legacy=0x7fa4540cf290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-01-16T15:29:33.942 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.941+0000 7fa45a600640 1 -- 172.21.15.53:0/1968399666 --> v2:172.21.15.53:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45406cd10 con 0x7fa4540cf9a0 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.943+0000 7fa45a600640 1 --2- 172.21.15.53:0/1968399666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4540cd840 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009e50 tx=0x7fa44c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799c2abb1103d00a server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.943+0000 7fa453e00640 1 -- 172.21.15.53:0/1968399666 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c02ca70 con 0x7fa4540cf9a0 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.943+0000 7fa453e00640 1 -- 172.21.15.53:0/1968399666 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa44c02cbd0 con 0x7fa4540cf9a0 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.943+0000 7fa453e00640 1 -- 172.21.15.53:0/1968399666 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c0343f0 con 0x7fa4540cf9a0 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 -- 172.21.15.53:0/1968399666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 msgr2=0x7fa4540cd840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 --2- 172.21.15.53:0/1968399666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4540cd840 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fa44c009e50 tx=0x7fa44c02bed0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 -- 172.21.15.53:0/1968399666 shutdown_connections 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 --2- 172.21.15.53:0/1968399666 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4540cd840 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.944 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 --2- 172.21.15.53:0/1968399666 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413df80 0x7fa45415bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 -- 172.21.15.53:0/1968399666 >> 172.21.15.53:0/1968399666 conn(0x7fa45406a800 msgr2=0x7fa45406ac10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:33.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 -- 172.21.15.53:0/1968399666 shutdown_connections 2024-01-16T15:29:33.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.944+0000 7fa45c600640 1 -- 172.21.15.53:0/1968399666 wait complete. 2024-01-16T15:29:33.945 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.946+0000 7fa45c600640 1 Processor -- start 2024-01-16T15:29:33.946 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.946+0000 7fa45c600640 1 -- start start 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa45c600640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4541f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa45c600640 1 --2- >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45413df80 0x7fa4541f3860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa45c600640 1 --2- >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413e360 0x7fa45415d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa45c600640 1 -- --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4541f3da0 con 0x7fa4540cf9a0 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa45c600640 1 -- --> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4541f7300 con 0x7fa45413df80 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa45c600640 1 -- --> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4541f6a60 con 0x7fa45413e360 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa459c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4541f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.947 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa459c00640 1 --2- >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4541f63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.53:3300/0 says I am v2:172.21.15.53:59662/0 (socket says 172.21.15.53:59662) 2024-01-16T15:29:33.948 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.947+0000 7fa459c00640 1 -- 172.21.15.53:0/3274627479 learned_addr learned my addr 172.21.15.53:0/3274627479 (peer_addr_for_me v2:172.21.15.53:0/0) 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa45a600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413e360 0x7fa45415d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459200640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45413df80 0x7fa4541f3860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459c00640 1 -- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413e360 msgr2=0x7fa45415d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459c00640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413e360 0x7fa45415d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459c00640 1 -- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45413df80 msgr2=0x7fa4541f3860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459c00640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45413df80 0x7fa4541f3860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:33.949 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459c00640 1 -- 172.21.15.53:0/3274627479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c009b30 con 0x7fa4540cf9a0 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa45a600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413e360 0x7fa45415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459200640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45413df80 0x7fa4541f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa459c00640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4541f63d0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c820 tx=0x7fa44400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444004220 con 0x7fa4540cf9a0 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fa4440043a0 con 0x7fa4540cf9a0 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa444007a90 con 0x7fa4540cf9a0 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4541f6d40 con 0x7fa4540cf9a0 2024-01-16T15:29:33.950 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.948+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4540a8660 con 0x7fa4540cf9a0 2024-01-16T15:29:33.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.950+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4540a84c0 con 0x7fa4540cf9a0 2024-01-16T15:29:33.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.951+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 4 ==== mgrmap(e 53) v1 ==== 91171+0+0 (secure 0 0 0) 0x7fa444007bf0 con 0x7fa4540cf9a0 2024-01-16T15:29:33.951 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.952+0000 7fa451600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa42c06c4c0 0x7fa42c06e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-01-16T15:29:33.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.952+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 5795+0+0 (secure 0 0 0) 0x7fa444088960 con 0x7fa4540cf9a0 2024-01-16T15:29:33.952 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.953+0000 7fa459200640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa42c06c4c0 0x7fa42c06e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-01-16T15:29:33.953 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.954+0000 7fa459200640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa42c06c4c0 0x7fa42c06e980 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa45415f2c0 tx=0x7fa43c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-16T15:29:33.957 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:33.957+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+179825 (secure 0 0 0) 0x7fa4540a84c0 con 0x7fa4540cf9a0 2024-01-16T15:29:34.190 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.190+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 --> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4540a84c0 con 0x7fa4540cf9a0 2024-01-16T15:29:34.191 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.191+0000 7fa451600640 1 -- 172.21.15.53:0/3274627479 <== mon.0 v2:172.21.15.53:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fa4540a84c0 con 0x7fa4540cf9a0 2024-01-16T15:29:34.191 INFO:teuthology.orchestra.run.smithi053.stdout: 2024-01-16T15:29:34.192 INFO:teuthology.orchestra.run.smithi053.stdout:{"epoch":75,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6599,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6837/1519219735","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6835","nonce":1519219735},{"type":"v1","addr":"172.21.15.146:6837","nonce":1519219735}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":7372,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.146:6836/2058874467","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.146:6834","nonce":2058874467},{"type":"v1","addr":"172.21.15.146:6836","nonce":2058874467}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68},{"gid":7381,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.53:6835/3227938262","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6832","nonce":3227938262},{"type":"v1","addr":"172.21.15.53:6835","nonce":3227938262}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-01-16T15:19:24.074687+0000","modified":"2024-01-16T15:19:27.148085+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9054},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9054":{"gid":9054,"name":"c","rank":0,"incarnation":70,"state":"up:active","state_seq":3,"addr":"172.21.15.53:6834/1996992482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.53:6833","nonce":1996992482},{"type":"v1","addr":"172.21.15.53:6834","nonce":1996992482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-01-16T15:29:34.192 INFO:teuthology.orchestra.run.smithi053.stderr:dumped fsmap epoch 75 2024-01-16T15:29:34.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa42c06c4c0 msgr2=0x7fa42c06e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:34.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa42c06c4c0 0x7fa42c06e980 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fa45415f2c0 tx=0x7fa43c009040 comp rx=0 tx=0).stop 2024-01-16T15:29:34.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 msgr2=0x7fa4541f63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-16T15:29:34.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4541f63d0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fa44400c820 tx=0x7fa44400cce0 comp rx=0 tx=0).stop 2024-01-16T15:29:34.194 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 shutdown_connections 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:6832/2200557793,v1:172.21.15.146:6833/2200557793] conn(0x7fa42c06c4c0 0x7fa42c06e980 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3301/0,v1:172.21.15.146:6790/0] conn(0x7fa45413e360 0x7fa45415d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.146:3300/0,v1:172.21.15.146:6789/0] conn(0x7fa45413df80 0x7fa4541f3860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 --2- 172.21.15.53:0/3274627479 >> [v2:172.21.15.53:3300/0,v1:172.21.15.53:6789/0] conn(0x7fa4540cf9a0 0x7fa4541f63d0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 >> 172.21.15.53:0/3274627479 conn(0x7fa45406a800 msgr2=0x7fa454153fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 shutdown_connections 2024-01-16T15:29:34.195 INFO:teuthology.orchestra.run.smithi053.stderr:2024-01-16T15:29:34.194+0000 7fa45c600640 1 -- 172.21.15.53:0/3274627479 wait complete. 2024-01-16T15:29:34.214 DEBUG:tasks.cephfs.filesystem:already dead... 2024-01-16T15:29:34.215 INFO:teuthology.misc:Shutting down mds daemons... 2024-01-16T15:29:34.215 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-01-16T15:29:34.215 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:34.231 INFO:tasks.ceph.mds.a:Stopped 2024-01-16T15:29:34.231 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-01-16T15:29:34.231 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:34.313 INFO:tasks.ceph.mds.c:Stopped 2024-01-16T15:29:34.313 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-01-16T15:29:34.313 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:34.332 INFO:tasks.ceph.mds.b:Stopped 2024-01-16T15:29:34.333 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-01-16T15:29:34.333 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:34.404 INFO:tasks.ceph.mds.d:Stopped 2024-01-16T15:29:34.404 INFO:teuthology.misc:Shutting down osd daemons... 2024-01-16T15:29:34.404 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-01-16T15:29:34.404 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:36.266 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-01-16T15:29:40.506 INFO:tasks.ceph.osd.0:Stopped 2024-01-16T15:29:40.507 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-01-16T15:29:40.507 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:40.576 INFO:tasks.ceph.osd.1:Stopped 2024-01-16T15:29:40.576 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-01-16T15:29:40.576 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:42.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-01-16T15:29:42.274 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-01-16T15:29:46.678 INFO:tasks.ceph.osd.2:Stopped 2024-01-16T15:29:46.679 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-01-16T15:29:46.679 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:48.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-01-16T15:29:48.181 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-01-16T15:29:52.780 INFO:tasks.ceph.osd.3:Stopped 2024-01-16T15:29:52.781 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-01-16T15:29:52.781 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:52.849 INFO:tasks.ceph.osd.4:Stopped 2024-01-16T15:29:52.849 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-01-16T15:29:52.849 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:53.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-01-16T15:29:53.887 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-01-16T15:29:58.951 INFO:tasks.ceph.osd.5:Stopped 2024-01-16T15:29:58.951 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-01-16T15:29:58.951 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:59.031 INFO:tasks.ceph.osd.6:Stopped 2024-01-16T15:29:59.031 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-01-16T15:29:59.031 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:59.105 INFO:tasks.ceph.osd.7:Stopped 2024-01-16T15:29:59.105 INFO:teuthology.misc:Shutting down mgr daemons... 2024-01-16T15:29:59.105 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-01-16T15:29:59.105 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:29:59.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~6s 2024-01-16T15:29:59.492 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~0s 2024-01-16T15:30:00.415 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:30:00.424 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:30:04.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y is failed for ~6s 2024-01-16T15:30:04.895 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-01-16T15:30:05.208 INFO:tasks.ceph.mgr.y:Stopped 2024-01-16T15:30:05.208 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-01-16T15:30:05.208 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:30:10.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x is failed for ~0s 2024-01-16T15:30:10.198 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.y has been restored 2024-01-16T15:30:11.310 INFO:tasks.ceph.mgr.x:Stopped 2024-01-16T15:30:11.310 INFO:teuthology.misc:Shutting down mon daemons... 2024-01-16T15:30:11.311 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-01-16T15:30:11.311 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:30:11.371 INFO:tasks.ceph.mon.a:Stopped 2024-01-16T15:30:11.371 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-01-16T15:30:11.371 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:30:11.421 INFO:tasks.ceph.mon.b:Stopped 2024-01-16T15:30:11.421 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-01-16T15:30:11.421 INFO:teuthology.orchestra.run:waiting for 300 2024-01-16T15:30:11.448 INFO:tasks.ceph.mon.c:Stopped 2024-01-16T15:30:11.448 INFO:tasks.ceph:Checking cluster log for badness... 2024-01-16T15:30:11.448 DEBUG:teuthology.orchestra.run.smithi053:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2024-01-16T15:30:11.562 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:30:11.562 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-01-16T15:30:11.841 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:30:11.841 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-01-16T15:30:12.027 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:30:12.027 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-01-16T15:30:12.195 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:30:12.195 DEBUG:teuthology.orchestra.run.smithi053:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-01-16T15:30:12.383 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:30:12.383 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-01-16T15:30:12.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:30:12.667 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-01-16T15:30:12.789 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:30:12.789 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-01-16T15:30:12.960 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:30:12.960 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-01-16T15:30:13.139 INFO:tasks.ceph:Archiving mon data... 2024-01-16T15:30:13.141 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/data/mon.a.tgz 2024-01-16T15:30:13.141 DEBUG:teuthology.orchestra.run.smithi053:> mktemp 2024-01-16T15:30:13.202 INFO:teuthology.orchestra.run.smithi053.stdout:/tmp/tmp.mGQA1SXxba 2024-01-16T15:30:13.202 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mGQA1SXxba 2024-01-16T15:30:13.559 DEBUG:teuthology.orchestra.run.smithi053:> sudo chmod 0666 /tmp/tmp.mGQA1SXxba 2024-01-16T15:30:13.719 DEBUG:teuthology.orchestra.remote:smithi053:/tmp/tmp.mGQA1SXxba is 1MB 2024-01-16T15:30:13.846 DEBUG:teuthology.orchestra.run.smithi053:> rm -fr /tmp/tmp.mGQA1SXxba 2024-01-16T15:30:13.904 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/data/mon.b.tgz 2024-01-16T15:30:13.905 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-01-16T15:30:13.962 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.D2u7I2cnc3 2024-01-16T15:30:13.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.D2u7I2cnc3 2024-01-16T15:30:14.303 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.D2u7I2cnc3 2024-01-16T15:30:14.452 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.D2u7I2cnc3 is 1MB 2024-01-16T15:30:14.573 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.D2u7I2cnc3 2024-01-16T15:30:14.631 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/data/mon.c.tgz 2024-01-16T15:30:14.631 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-01-16T15:30:14.685 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.plLPsG3afF 2024-01-16T15:30:14.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.plLPsG3afF 2024-01-16T15:30:15.017 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.plLPsG3afF 2024-01-16T15:30:15.169 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.plLPsG3afF is 1MB 2024-01-16T15:30:15.199 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mgr.x has been restored 2024-01-16T15:30:15.290 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.plLPsG3afF 2024-01-16T15:30:15.347 INFO:tasks.ceph:Cleaning ceph cluster... 2024-01-16T15:30:15.347 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:30:15.355 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-01-16T15:30:15.511 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:30:15.568 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-01-16T15:30:15.569 INFO:tasks.ceph:Archiving crash dumps... 2024-01-16T15:30:15.571 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/remote/smithi053/crash 2024-01-16T15:30:15.571 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:30:15.677 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/remote/smithi146/crash 2024-01-16T15:30:15.678 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-01-16T15:30:15.771 INFO:tasks.ceph:Shutting down logrotate 2024-01-16T15:30:15.771 DEBUG:teuthology.orchestra.run.smithi053:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:30:15.779 DEBUG:teuthology.orchestra.run.smithi146:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-01-16T15:30:15.912 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:30:16.018 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-01-16T15:30:16.116 INFO:tasks.ceph:Compressing logs... 2024-01-16T15:30:16.116 DEBUG:teuthology.orchestra.run.smithi053:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:30:16.124 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-16T15:30:16.214 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-01-16T15:30:16.215 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.47621.log 2024-01-16T15:30:16.216 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44911.log 2024-01-16T15:30:16.217 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67082.log 2024-01-16T15:30:16.218 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.1.47621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.41697.log 2024-01-16T15:30:16.218 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52612.log 2024-01-16T15:30:16.219 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.1.44911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36339.log 2024-01-16T15:30:16.220 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48152.log 2024-01-16T15:30:16.220 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67082.log.gz 2024-01-16T15:30:16.221 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.3.41697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44041.log 2024-01-16T15:30:16.221 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52612.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52612.log.gz 2024-01-16T15:30:16.221 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70275.log 2024-01-16T15:30:16.223 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2024-01-16T15:30:16.223 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48152.log.gz 2024-01-16T15:30:16.223 INFO:teuthology.orchestra.run.smithi146.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.3.41697.log.gz 2024-01-16T15:30:16.223 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36339.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:30:16.224 INFO:teuthology.orchestra.run.smithi146.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36339.log.gz 2024-01-16T15:30:16.224 INFO:teuthology.orchestra.run.smithi146.stderr:gzip/var/log/ceph/ceph-client.1.44041.log: -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-01-16T15:30:16.225 INFO:teuthology.orchestra.run.smithi146.stderr: 96.1% -- replaced with /var/log/ceph/ceph-client.1.44911.log.gz 2024-01-16T15:30:16.225 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.42465.log 2024-01-16T15:30:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50463.log 2024-01-16T15:30:16.226 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70275.log.gz 2024-01-16T15:30:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67815.log 2024-01-16T15:30:16.226 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2024-01-16T15:30:16.226 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70569.log 2024-01-16T15:30:16.227 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.50463.log.gz 2024-01-16T15:30:16.228 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67815.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67815.log.gz 2024-01-16T15:30:16.228 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59043.log 2024-01-16T15:30:16.228 INFO:teuthology.orchestra.run.smithi146.stderr: 96.6% -- replaced with /var/log/ceph/ceph-client.1.47621.log.gz 2024-01-16T15:30:16.229 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-01-16T15:30:16.229 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.1.42465.log: 95.1% -- replaced with /var/log/ceph/ceph-client.1.44041.log.gz 2024-01-16T15:30:16.229 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.46780.log 2024-01-16T15:30:16.230 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36219.log 2024-01-16T15:30:16.230 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.1.42465.log.gz 2024-01-16T15:30:16.231 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58959.log 2024-01-16T15:30:16.231 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70569.log.gz 2024-01-16T15:30:16.231 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46095.log 2024-01-16T15:30:16.231 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.59043.log.gz 2024-01-16T15:30:16.232 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46950.log 2024-01-16T15:30:16.232 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58959.log.gz 2024-01-16T15:30:16.234 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45891.log 2024-01-16T15:30:16.234 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46095.log.gz 2024-01-16T15:30:16.234 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46263.log 2024-01-16T15:30:16.234 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46950.log.gz 2024-01-16T15:30:16.234 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.1.46780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.41177.log 2024-01-16T15:30:16.234 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:30:16.235 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36219.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36219.log.gz 2024-01-16T15:30:16.235 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-01-16T15:30:16.235 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-01-16T15:30:16.236 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48026.log 2024-01-16T15:30:16.236 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45891.log.gz 2024-01-16T15:30:16.236 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56051.log 2024-01-16T15:30:16.236 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46263.log.gz 2024-01-16T15:30:16.237 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.1.41177.log: /var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36099.log 2024-01-16T15:30:16.237 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.1.41177.log.gz 2024-01-16T15:30:16.238 INFO:teuthology.orchestra.run.smithi146.stderr: 95.7% -- replaced with /var/log/ceph/ceph-client.1.46780.log.gz 2024-01-16T15:30:16.238 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36059.log 2024-01-16T15:30:16.238 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48026.log: gzip -5 88.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.48026.log.gz -- 2024-01-16T15:30:16.238 INFO:teuthology.orchestra.run.smithi053.stderr: /var/log/ceph/ceph-client.admin.67549.log 2024-01-16T15:30:16.239 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36099.log: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36139.log 2024-01-16T15:30:16.239 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36099.log.gz 2024-01-16T15:30:16.240 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51532.log 2024-01-16T15:30:16.240 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56051.log.gz 2024-01-16T15:30:16.240 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67549.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67549.log.gz 2024-01-16T15:30:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43956.log 2024-01-16T15:30:16.241 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51532.log: gzip -5 --verbose 88.5% -- /var/log/ceph/ceph-client.admin.62435.log -- replaced with /var/log/ceph/ceph-client.admin.51532.log.gz 2024-01-16T15:30:16.242 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.242 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36059.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-01-16T15:30:16.242 INFO:teuthology.orchestra.run.smithi146.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36059.log.gz 2024-01-16T15:30:16.242 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.45986.log 2024-01-16T15:30:16.243 INFO:teuthology.orchestra.run.smithi146.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.36139.log.gz 2024-01-16T15:30:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40850.log 2024-01-16T15:30:16.244 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43956.log.gz 2024-01-16T15:30:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76837.log 2024-01-16T15:30:16.244 INFO:teuthology.orchestra.run.smithi053.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.62435.log.gz 2024-01-16T15:30:16.244 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68109.log 2024-01-16T15:30:16.245 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-client.1.45986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36179.log 2024-01-16T15:30:16.246 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.40850.log.gz 2024-01-16T15:30:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76837.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.76837.log.gz 2024-01-16T15:30:16.246 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74892.log 2024-01-16T15:30:16.248 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68109.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-01-16T15:30:16.248 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68109.log.gz 2024-01-16T15:30:16.248 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-01-16T15:30:16.248 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.74892.log.gz 2024-01-16T15:30:16.249 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36179.log: 93.8% -- replaced with /var/log/ceph/ceph-client.1.45986.log.gz 2024-01-16T15:30:16.249 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:30:16.250 INFO:teuthology.orchestra.run.smithi146.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.36179.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-01-16T15:30:16.250 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-01-16T15:30:16.251 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40259.log 2024-01-16T15:30:16.251 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51140.log 2024-01-16T15:30:16.251 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2024-01-16T15:30:16.251 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36004.log 2024-01-16T15:30:16.252 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40259.log.gz 2024-01-16T15:30:16.253 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51140.log: gzip -5 --verbose -- 88.0% /var/log/ceph/ceph-client.admin.71698.log -- replaced with /var/log/ceph/ceph-client.admin.51140.log.gz 2024-01-16T15:30:16.253 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.253 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.43192.log 2024-01-16T15:30:16.254 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71174.log 2024-01-16T15:30:16.254 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36004.log.gz 2024-01-16T15:30:16.254 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49925.log 2024-01-16T15:30:16.255 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.71698.log.gz 2024-01-16T15:30:16.256 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.1.43192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36259.log 2024-01-16T15:30:16.256 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57311.log 2024-01-16T15:30:16.257 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71174.log.gz 2024-01-16T15:30:16.257 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49925.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49925.log.gz -5 2024-01-16T15:30:16.257 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.54974.log 2024-01-16T15:30:16.258 INFO:teuthology.orchestra.run.smithi146.stderr: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:30:16.258 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70994.log 2024-01-16T15:30:16.258 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57311.log.gz 2024-01-16T15:30:16.259 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72264.log 2024-01-16T15:30:16.260 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54974.log.gz 2024-01-16T15:30:16.260 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-01-16T15:30:16.260 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36299.log 2024-01-16T15:30:16.261 INFO:teuthology.orchestra.run.smithi146.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36259.log.gz 2024-01-16T15:30:16.261 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58260.log 2024-01-16T15:30:16.261 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.70994.log.gz 2024-01-16T15:30:16.262 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72264.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72264.log.gz 2024-01-16T15:30:16.262 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59265.log 2024-01-16T15:30:16.263 INFO:teuthology.orchestra.run.smithi146.stderr: 91.9% 94.5% -- replaced with /var/log/ceph/ceph-client.1.43192.log.gz 2024-01-16T15:30:16.263 INFO:teuthology.orchestra.run.smithi146.stderr: -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-01-16T15:30:16.263 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-01-16T15:30:16.264 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.36299.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36299.log.gz 2024-01-16T15:30:16.265 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54764.log 2024-01-16T15:30:16.265 INFO:teuthology.orchestra.run.smithi053.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.58260.log.gz 2024-01-16T15:30:16.265 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51056.log 2024-01-16T15:30:16.265 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59265.log.gz 2024-01-16T15:30:16.266 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54764.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54764.log.gz 2024-01-16T15:30:16.266 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47616.log 2024-01-16T15:30:16.267 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79177.log 2024-01-16T15:30:16.267 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51056.log.gz 2024-01-16T15:30:16.268 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63420.log 2024-01-16T15:30:16.268 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47616.log.gz 2024-01-16T15:30:16.269 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79177.log: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.79177.log.gz 2024-01-16T15:30:16.269 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73978.log 2024-01-16T15:30:16.270 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66533.log 2024-01-16T15:30:16.270 INFO:teuthology.orchestra.run.smithi053.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63420.log.gz 2024-01-16T15:30:16.271 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60453.log 2024-01-16T15:30:16.271 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73978.log.gz 2024-01-16T15:30:16.272 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66533.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.66533.log.gz 2024-01-16T15:30:16.273 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44334.log 2024-01-16T15:30:16.274 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79259.log 2024-01-16T15:30:16.274 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.60453.log.gz 2024-01-16T15:30:16.275 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60537.log 2024-01-16T15:30:16.275 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44334.log.gz 2024-01-16T15:30:16.276 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79259.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79259.log.gz 2024-01-16T15:30:16.276 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56948.log 2024-01-16T15:30:16.277 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51700.log 2024-01-16T15:30:16.277 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60537.log.gz 2024-01-16T15:30:16.278 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56948.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.56948.log.gz -5 2024-01-16T15:30:16.279 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51098.log 2024-01-16T15:30:16.280 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50715.log 2024-01-16T15:30:16.280 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51700.log.gz 2024-01-16T15:30:16.281 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41898.log 2024-01-16T15:30:16.281 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51098.log.gz 2024-01-16T15:30:16.282 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50715.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50715.log.gz 2024-01-16T15:30:16.282 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59604.log 2024-01-16T15:30:16.283 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76120.log 2024-01-16T15:30:16.284 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41898.log.gz 2024-01-16T15:30:16.284 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52738.log 2024-01-16T15:30:16.285 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59604.log.gz 2024-01-16T15:30:16.286 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76120.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76120.log.gz 2024-01-16T15:30:16.286 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74425.log 2024-01-16T15:30:16.287 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45933.log 2024-01-16T15:30:16.287 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.52738.log.gz 2024-01-16T15:30:16.288 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44040.log 2024-01-16T15:30:16.288 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.74425.log.gz 2024-01-16T15:30:16.289 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45933.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45933.log.gz 2024-01-16T15:30:16.289 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43914.log 2024-01-16T15:30:16.290 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73443.log 2024-01-16T15:30:16.291 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44040.log.gz 2024-01-16T15:30:16.291 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80073.log 2024-01-16T15:30:16.292 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43914.log.gz 2024-01-16T15:30:16.293 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73443.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.73443.log.gz 2024-01-16T15:30:16.293 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53396.log 2024-01-16T15:30:16.294 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42066.log 2024-01-16T15:30:16.294 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.80073.log.gz 2024-01-16T15:30:16.295 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42276.log 2024-01-16T15:30:16.295 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53396.log.gz 2024-01-16T15:30:16.296 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42066.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.42066.log.gz 2024-01-16T15:30:16.296 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41194.log 2024-01-16T15:30:16.297 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75156.log 2024-01-16T15:30:16.298 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42276.log.gz 2024-01-16T15:30:16.298 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73569.log 2024-01-16T15:30:16.299 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75156.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41194.log.gz 2024-01-16T15:30:16.300 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.75156.log.gz /var/log/ceph/ceph-client.admin.54345.log 2024-01-16T15:30:16.300 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.301 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58133.log 2024-01-16T15:30:16.301 INFO:teuthology.orchestra.run.smithi053.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.73569.log.gz 2024-01-16T15:30:16.302 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53671.log 2024-01-16T15:30:16.302 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54345.log.gz 2024-01-16T15:30:16.304 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78585.log 2024-01-16T15:30:16.304 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58133.log.gz 2024-01-16T15:30:16.305 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62999.log 2024-01-16T15:30:16.305 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53671.log.gz 2024-01-16T15:30:16.306 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77565.log 2024-01-16T15:30:16.306 INFO:teuthology.orchestra.run.smithi053.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.78585.log.gz 2024-01-16T15:30:16.307 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62999.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62999.log.gz 2024-01-16T15:30:16.307 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41526.log 2024-01-16T15:30:16.308 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59646.log 2024-01-16T15:30:16.308 INFO:teuthology.orchestra.run.smithi053.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.77565.log.gz 2024-01-16T15:30:16.309 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41526.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41526.log.gz 2024-01-16T15:30:16.310 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45153.log 2024-01-16T15:30:16.311 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53714.log 2024-01-16T15:30:16.311 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59646.log.gz 2024-01-16T15:30:16.312 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45153.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45153.log.gz 2024-01-16T15:30:16.312 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72997.log 2024-01-16T15:30:16.314 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.43406.log 2024-01-16T15:30:16.314 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53714.log.gz 2024-01-16T15:30:16.315 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78667.log 2024-01-16T15:30:16.315 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72997.log.gz 2024-01-16T15:30:16.316 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.2.43406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64011.log 2024-01-16T15:30:16.316 INFO:teuthology.orchestra.run.smithi053.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.2.43406.log.gz 2024-01-16T15:30:16.317 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78667.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78667.log.gz 2024-01-16T15:30:16.318 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65723.log 2024-01-16T15:30:16.319 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76963.log 2024-01-16T15:30:16.319 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65723.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64011.log.gz 2024-01-16T15:30:16.320 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65723.log.gz 2024-01-16T15:30:16.320 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76795.log 2024-01-16T15:30:16.321 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46053.log 2024-01-16T15:30:16.322 INFO:teuthology.orchestra.run.smithi053.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.76963.log.gz 2024-01-16T15:30:16.322 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76795.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.76795.log.gz 2024-01-16T15:30:16.323 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69021.log 2024-01-16T15:30:16.324 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52085.log 2024-01-16T15:30:16.325 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46053.log.gz 2024-01-16T15:30:16.326 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69021.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.69021.log.gz 2024-01-16T15:30:16.326 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44949.log 2024-01-16T15:30:16.327 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.52085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81528.log 2024-01-16T15:30:16.328 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78831.log 2024-01-16T15:30:16.328 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44949.log.gz 2024-01-16T15:30:16.330 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68937.log 2024-01-16T15:30:16.330 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.81528.log.gz 2024-01-16T15:30:16.331 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78831.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.78831.log.gz 2024-01-16T15:30:16.332 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74934.log 2024-01-16T15:30:16.334 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68937.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.68937.log.gz 2024-01-16T15:30:16.334 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49431.log 2024-01-16T15:30:16.335 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74934.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74934.log.gz 2024-01-16T15:30:16.335 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67040.log 2024-01-16T15:30:16.336 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57437.log 2024-01-16T15:30:16.337 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49431.log.gz 2024-01-16T15:30:16.338 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67040.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67040.log.gz 2024-01-16T15:30:16.338 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80730.log 2024-01-16T15:30:16.339 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49799.log 2024-01-16T15:30:16.340 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57437.log.gz 2024-01-16T15:30:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80730.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.80730.log.gz 2024-01-16T15:30:16.341 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81692.log 2024-01-16T15:30:16.343 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47754.log 2024-01-16T15:30:16.343 INFO:teuthology.orchestra.run.smithi053.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.49799.log.gz 2024-01-16T15:30:16.345 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81692.log: gzip 88.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.81692.log.gz 2024-01-16T15:30:16.345 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.57920.log 2024-01-16T15:30:16.346 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71038.log 2024-01-16T15:30:16.346 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47754.log.gz 2024-01-16T15:30:16.348 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57920.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57920.log.gz 2024-01-16T15:30:16.348 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47858.log 2024-01-16T15:30:16.349 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79565.log 2024-01-16T15:30:16.350 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.71038.log.gz 2024-01-16T15:30:16.351 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47858.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47858.log.gz 2024-01-16T15:30:16.351 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62477.log 2024-01-16T15:30:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79565.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.79565.log.gz 2024-01-16T15:30:16.353 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63083.log 2024-01-16T15:30:16.354 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46487.log 2024-01-16T15:30:16.354 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.62477.log.gz 2024-01-16T15:30:16.355 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62561.log 2024-01-16T15:30:16.355 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63083.log.gz 2024-01-16T15:30:16.357 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46179.log 88.5% 2024-01-16T15:30:16.357 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46487.log.gz 2024-01-16T15:30:16.358 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64197.log 2024-01-16T15:30:16.358 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62561.log.gz 2024-01-16T15:30:16.360 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65387.log 2024-01-16T15:30:16.360 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46179.log.gz 2024-01-16T15:30:16.361 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65303.log 2024-01-16T15:30:16.362 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64197.log.gz 2024-01-16T15:30:16.363 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41515.log 2024-01-16T15:30:16.363 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.65387.log.gz 2024-01-16T15:30:16.364 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51362.log 2024-01-16T15:30:16.364 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.65303.log.gz 2024-01-16T15:30:16.366 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41515.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.41515.log.gz 2024-01-16T15:30:16.366 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.73527.log 2024-01-16T15:30:16.367 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65471.log 2024-01-16T15:30:16.368 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51362.log.gz 2024-01-16T15:30:16.369 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73527.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.73527.log.gz 2024-01-16T15:30:16.369 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47195.log 2024-01-16T15:30:16.370 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59856.log 2024-01-16T15:30:16.371 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65471.log.gz 2024-01-16T15:30:16.372 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81276.log 2024-01-16T15:30:16.372 INFO:teuthology.orchestra.run.smithi053.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47195.log.gz 2024-01-16T15:30:16.373 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40512.log 2024-01-16T15:30:16.374 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59856.log.gz 2024-01-16T15:30:16.375 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70824.log 2024-01-16T15:30:16.375 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81276.log.gz 2024-01-16T15:30:16.376 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61659.log 2024-01-16T15:30:16.377 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.40512.log.gz 2024-01-16T15:30:16.378 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70824.log: gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.70824.log.gz 2024-01-16T15:30:16.378 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.78157.log 2024-01-16T15:30:16.379 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61617.log 2024-01-16T15:30:16.379 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61659.log.gz 2024-01-16T15:30:16.380 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73384.log 2024-01-16T15:30:16.381 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78157.log.gz 2024-01-16T15:30:16.382 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54647.log 2024-01-16T15:30:16.382 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61617.log.gz 2024-01-16T15:30:16.383 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46698.log 2024-01-16T15:30:16.383 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.73384.log.gz 2024-01-16T15:30:16.385 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67422.log 2024-01-16T15:30:16.385 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54647.log.gz 2024-01-16T15:30:16.386 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57877.log 2024-01-16T15:30:16.386 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46698.log.gz 2024-01-16T15:30:16.387 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67422.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.67422.log.gz 2024-01-16T15:30:16.387 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49344.log 2024-01-16T15:30:16.388 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41525.log 2024-01-16T15:30:16.389 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57877.log.gz 2024-01-16T15:30:16.390 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77770.log 2024-01-16T15:30:16.390 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.49344.log.gz 2024-01-16T15:30:16.391 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74341.log 2024-01-16T15:30:16.391 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41525.log.gz 2024-01-16T15:30:16.393 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77770.log: gzip -5 --verbose -- 87.6% /var/log/ceph/ceph-client.admin.79947.log -- replaced with /var/log/ceph/ceph-client.admin.77770.log.gz 2024-01-16T15:30:16.393 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.394 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49841.log 2024-01-16T15:30:16.394 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74341.log.gz 2024-01-16T15:30:16.395 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79947.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79947.log.gz 2024-01-16T15:30:16.395 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.78790.log 2024-01-16T15:30:16.396 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79341.log 2024-01-16T15:30:16.396 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49841.log.gz 2024-01-16T15:30:16.398 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78790.log: gzip -5 --verbose -- 86.5% /var/log/ceph/ceph-client.admin.69241.log -- replaced with /var/log/ceph/ceph-client.admin.78790.log.gz 2024-01-16T15:30:16.398 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.399 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49580.log 2024-01-16T15:30:16.399 INFO:teuthology.orchestra.run.smithi053.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.79341.log.gz 2024-01-16T15:30:16.400 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69241.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69241.log.gz 2024-01-16T15:30:16.400 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47900.log 2024-01-16T15:30:16.401 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53543.log 2024-01-16T15:30:16.402 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49580.log.gz 2024-01-16T15:30:16.403 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47900.log: gzip -5 --verbose 88.6% -- -- replaced with /var/log/ceph/ceph-client.admin.47900.log.gz /var/log/ceph/ceph-client.admin.75386.log 2024-01-16T15:30:16.403 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.404 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47034.log 2024-01-16T15:30:16.404 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53543.log.gz 2024-01-16T15:30:16.406 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62039.log 2024-01-16T15:30:16.406 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.75386.log.gz 2024-01-16T15:30:16.407 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55275.log 2024-01-16T15:30:16.407 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47034.log.gz 2024-01-16T15:30:16.408 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63294.log 2024-01-16T15:30:16.408 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.62039.log.gz 2024-01-16T15:30:16.409 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63251.log 2024-01-16T15:30:16.410 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55275.log.gz 2024-01-16T15:30:16.411 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63294.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63294.log.gz 2024-01-16T15:30:16.411 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51406.log 2024-01-16T15:30:16.412 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78503.log 2024-01-16T15:30:16.412 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63251.log.gz 2024-01-16T15:30:16.413 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61491.log 2024-01-16T15:30:16.413 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51406.log.gz 2024-01-16T15:30:16.414 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56093.log 2024-01-16T15:30:16.415 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78503.log.gz 2024-01-16T15:30:16.416 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61491.log: gzip 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61491.log.gz -5 2024-01-16T15:30:16.416 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.64655.log 2024-01-16T15:30:16.417 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75302.log 2024-01-16T15:30:16.418 INFO:teuthology.orchestra.run.smithi053.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.56093.log.gz 2024-01-16T15:30:16.419 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47660.log 2024-01-16T15:30:16.419 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64655.log.gz 2024-01-16T15:30:16.420 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44208.log 2024-01-16T15:30:16.420 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75302.log.gz 2024-01-16T15:30:16.421 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67591.log 2024-01-16T15:30:16.421 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47660.log.gz 2024-01-16T15:30:16.422 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51742.log 2024-01-16T15:30:16.423 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44208.log.gz 2024-01-16T15:30:16.424 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52856.log 2024-01-16T15:30:16.424 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.67591.log.gz 2024-01-16T15:30:16.425 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63462.log 2024-01-16T15:30:16.425 INFO:teuthology.orchestra.run.smithi053.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51742.log.gz 2024-01-16T15:30:16.427 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54932.log 2024-01-16T15:30:16.427 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.52856.log.gz 2024-01-16T15:30:16.428 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63462.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63462.log.gz 2024-01-16T15:30:16.428 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44991.log 2024-01-16T15:30:16.429 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55925.log 2024-01-16T15:30:16.429 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54932.log.gz 2024-01-16T15:30:16.431 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69300.log 2024-01-16T15:30:16.431 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44991.log.gz 2024-01-16T15:30:16.432 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54485.log 2024-01-16T15:30:16.432 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.55925.log.gz 2024-01-16T15:30:16.433 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79805.log 2024-01-16T15:30:16.433 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69300.log.gz 2024-01-16T15:30:16.434 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78890.log 2024-01-16T15:30:16.435 INFO:teuthology.orchestra.run.smithi053.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54485.log.gz 2024-01-16T15:30:16.436 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77606.log 2024-01-16T15:30:16.436 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.79805.log.gz 2024-01-16T15:30:16.437 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57185.log 2024-01-16T15:30:16.437 INFO:teuthology.orchestra.run.smithi053.stderr: 84.9% -- replaced with /var/log/ceph/ceph-client.admin.78890.log.gz 2024-01-16T15:30:16.438 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78034.log 2024-01-16T15:30:16.439 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.77606.log.gz 2024-01-16T15:30:16.440 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57185.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42108.log 2024-01-16T15:30:16.440 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57185.log.gz 2024-01-16T15:30:16.441 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78034.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.78034.log.gz 2024-01-16T15:30:16.441 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43746.log 2024-01-16T15:30:16.442 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69912.log 2024-01-16T15:30:16.442 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42108.log.gz 2024-01-16T15:30:16.444 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70744.log 89.3% 2024-01-16T15:30:16.444 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.43746.log.gz 2024-01-16T15:30:16.445 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74140.log 2024-01-16T15:30:16.445 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69912.log.gz 2024-01-16T15:30:16.446 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70744.log: gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.70744.log.gz -5 2024-01-16T15:30:16.446 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40554.log 2024-01-16T15:30:16.447 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40087.log 2024-01-16T15:30:16.448 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74140.log.gz 2024-01-16T15:30:16.449 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49218.log 2024-01-16T15:30:16.449 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40554.log.gz 2024-01-16T15:30:16.450 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60579.log 2024-01-16T15:30:16.450 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40087.log.gz 2024-01-16T15:30:16.452 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50589.log 2024-01-16T15:30:16.452 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49218.log.gz 2024-01-16T15:30:16.453 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62165.log 2024-01-16T15:30:16.453 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60579.log.gz 2024-01-16T15:30:16.454 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50589.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50589.log.gz 2024-01-16T15:30:16.454 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49757.log 2024-01-16T15:30:16.456 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72348.log 2024-01-16T15:30:16.456 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.62165.log.gz 2024-01-16T15:30:16.457 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63867.log 2024-01-16T15:30:16.457 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49757.log.gz 2024-01-16T15:30:16.458 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78708.log 2024-01-16T15:30:16.458 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72348.log.gz 2024-01-16T15:30:16.459 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63867.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73081.log 2024-01-16T15:30:16.459 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.63867.log.gz 2024-01-16T15:30:16.460 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72306.log 2024-01-16T15:30:16.461 INFO:teuthology.orchestra.run.smithi053.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.78708.log.gz 2024-01-16T15:30:16.461 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73081.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.73081.log.gz 2024-01-16T15:30:16.462 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40846.log 2024-01-16T15:30:16.463 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81066.log 2024-01-16T15:30:16.463 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72306.log.gz 2024-01-16T15:30:16.464 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40846.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80251.log 2024-01-16T15:30:16.464 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40846.log.gz 2024-01-16T15:30:16.465 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81066.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72736.log 2024-01-16T15:30:16.465 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.81066.log.gz 2024-01-16T15:30:16.466 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44460.log 2024-01-16T15:30:16.466 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.80251.log.gz 2024-01-16T15:30:16.468 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61879.log 2024-01-16T15:30:16.468 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72736.log.gz 2024-01-16T15:30:16.469 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78421.log 2024-01-16T15:30:16.469 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.44460.log.gz 2024-01-16T15:30:16.470 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67941.log 2024-01-16T15:30:16.470 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61879.log.gz 2024-01-16T15:30:16.471 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40862.log 2024-01-16T15:30:16.471 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.78421.log.gz 2024-01-16T15:30:16.472 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58347.log 2024-01-16T15:30:16.472 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.67941.log.gz 2024-01-16T15:30:16.473 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74467.log 2024-01-16T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40862.log.gz 2024-01-16T15:30:16.474 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74257.log 2024-01-16T15:30:16.475 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58347.log.gz 2024-01-16T15:30:16.476 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68025.log 2024-01-16T15:30:16.476 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74467.log.gz 2024-01-16T15:30:16.477 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69713.log 2024-01-16T15:30:16.477 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.74257.log.gz 2024-01-16T15:30:16.478 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59307.log 2024-01-16T15:30:16.478 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68025.log.gz 2024-01-16T15:30:16.479 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76372.log 2024-01-16T15:30:16.479 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69713.log.gz 2024-01-16T15:30:16.480 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59307.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.59307.log.gz /var/log/ceph/ceph-client.admin.47448.log 2024-01-16T15:30:16.480 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.482 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81192.log 2024-01-16T15:30:16.482 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76372.log.gz 2024-01-16T15:30:16.483 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67983.log 2024-01-16T15:30:16.483 INFO:teuthology.orchestra.run.smithi053.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47448.log.gz 2024-01-16T15:30:16.484 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52308.log 2024-01-16T15:30:16.484 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81192.log.gz 2024-01-16T15:30:16.485 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46137.log 2024-01-16T15:30:16.485 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.67983.log.gz 2024-01-16T15:30:16.486 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69384.log 2024-01-16T15:30:16.487 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52308.log.gz 2024-01-16T15:30:16.488 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47322.log 2024-01-16T15:30:16.488 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.46137.log.gz 2024-01-16T15:30:16.489 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45111.log 2024-01-16T15:30:16.489 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.69384.log.gz 2024-01-16T15:30:16.490 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47322.log: gzip 88.8% -5 -- replaced with /var/log/ceph/ceph-client.admin.47322.log.gz --verbose 2024-01-16T15:30:16.490 INFO:teuthology.orchestra.run.smithi053.stderr: -- /var/log/ceph/ceph-client.admin.70116.log 2024-01-16T15:30:16.491 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54202.log 2024-01-16T15:30:16.492 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45111.log.gz 2024-01-16T15:30:16.493 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40861.log 2024-01-16T15:30:16.493 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70116.log.gz 2024-01-16T15:30:16.494 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47532.log 2024-01-16T15:30:16.494 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54202.log.gz 2024-01-16T15:30:16.495 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65765.log 2024-01-16T15:30:16.495 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40861.log.gz 2024-01-16T15:30:16.496 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59186.log 2024-01-16T15:30:16.497 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.47532.log.gz 2024-01-16T15:30:16.498 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60035.log 2024-01-16T15:30:16.498 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.65765.log.gz 2024-01-16T15:30:16.499 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79679.log 2024-01-16T15:30:16.499 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59186.log.gz 2024-01-16T15:30:16.500 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57269.log 2024-01-16T15:30:16.500 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60035.log.gz 2024-01-16T15:30:16.501 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71404.log 2024-01-16T15:30:16.501 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79679.log.gz 2024-01-16T15:30:16.502 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58749.log 2024-01-16T15:30:16.502 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57269.log.gz 2024-01-16T15:30:16.503 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64444.log 2024-01-16T15:30:16.504 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.71404.log.gz 2024-01-16T15:30:16.505 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54605.log 2024-01-16T15:30:16.505 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58749.log.gz 2024-01-16T15:30:16.506 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76330.log 2024-01-16T15:30:16.507 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64444.log.gz 2024-01-16T15:30:16.508 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48760.log 2024-01-16T15:30:16.508 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.54605.log.gz 2024-01-16T15:30:16.509 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58833.log 2024-01-16T15:30:16.509 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76330.log.gz 2024-01-16T15:30:16.510 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54848.log 2024-01-16T15:30:16.510 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48760.log.gz 2024-01-16T15:30:16.512 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58833.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.58833.log.gz 2024-01-16T15:30:16.512 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77325.log 2024-01-16T15:30:16.513 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48844.log 2024-01-16T15:30:16.513 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.54848.log.gz 2024-01-16T15:30:16.514 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77325.log: 85.1% -- replaced with /var/log/ceph/ceph-client.admin.77325.log.gz 2024-01-16T15:30:16.514 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78931.log 2024-01-16T15:30:16.516 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66143.log 2024-01-16T15:30:16.516 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48844.log.gz 2024-01-16T15:30:16.517 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65513.log 2024-01-16T15:30:16.517 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.78931.log.gz 2024-01-16T15:30:16.518 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64967.log 2024-01-16T15:30:16.518 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66143.log.gz 2024-01-16T15:30:16.519 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65513.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.65513.log.gz 2024-01-16T15:30:16.519 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.45363.log 2024-01-16T15:30:16.521 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69954.log 2024-01-16T15:30:16.521 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.64967.log.gz 2024-01-16T15:30:16.522 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45363.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77264.log 2024-01-16T15:30:16.522 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45363.log.gz 2024-01-16T15:30:16.523 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49477.log 2024-01-16T15:30:16.523 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.69954.log.gz 2024-01-16T15:30:16.524 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77264.log: gzip 88.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.77264.log.gz --verbose 2024-01-16T15:30:16.524 INFO:teuthology.orchestra.run.smithi053.stderr: -- /var/log/ceph/ceph-client.admin.55441.log 2024-01-16T15:30:16.526 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40860.log 2024-01-16T15:30:16.526 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49477.log.gz 2024-01-16T15:30:16.527 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66998.log 2024-01-16T15:30:16.527 INFO:teuthology.orchestra.run.smithi053.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.55441.log.gz 2024-01-16T15:30:16.528 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64883.log 2024-01-16T15:30:16.529 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40860.log.gz 2024-01-16T15:30:16.530 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58217.log 2024-01-16T15:30:16.530 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.66998.log.gz 2024-01-16T15:30:16.531 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66395.log 2024-01-16T15:30:16.531 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.64883.log.gz 2024-01-16T15:30:16.532 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49260.log 2024-01-16T15:30:16.532 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.58217.log.gz 2024-01-16T15:30:16.534 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69545.log 2024-01-16T15:30:16.534 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.66395.log.gz 2024-01-16T15:30:16.535 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74098.log 2024-01-16T15:30:16.535 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49260.log.gz 2024-01-16T15:30:16.536 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69545.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.69545.log.gz /var/log/ceph/ceph-client.admin.56619.log 2024-01-16T15:30:16.536 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.537 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53486.log 2024-01-16T15:30:16.537 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74098.log.gz 2024-01-16T15:30:16.539 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56619.log: gzip -5 88.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.56619.log.gz -- 2024-01-16T15:30:16.539 INFO:teuthology.orchestra.run.smithi053.stderr: /var/log/ceph/ceph-client.admin.64341.log 2024-01-16T15:30:16.540 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53299.log 2024-01-16T15:30:16.540 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53486.log.gz 2024-01-16T15:30:16.541 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64341.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56703.log 2024-01-16T15:30:16.541 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64341.log.gz 2024-01-16T15:30:16.542 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60789.log 2024-01-16T15:30:16.543 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53299.log.gz 2024-01-16T15:30:16.544 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56703.log: gzip -5 --verbose 88.5% -- -- replaced with /var/log/ceph/ceph-client.admin.56703.log.gz /var/log/ceph/ceph-client.admin.62081.log 2024-01-16T15:30:16.544 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.545 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45707.log 2024-01-16T15:30:16.545 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60789.log.gz 2024-01-16T15:30:16.547 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62081.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77439.log 2024-01-16T15:30:16.547 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62081.log.gz 2024-01-16T15:30:16.548 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67635.log 2024-01-16T15:30:16.548 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45707.log.gz 2024-01-16T15:30:16.549 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78462.log 2024-01-16T15:30:16.549 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.77439.log.gz 2024-01-16T15:30:16.550 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53126.log 2024-01-16T15:30:16.550 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.67635.log.gz 2024-01-16T15:30:16.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78462.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.78462.log.gz 2024-01-16T15:30:16.552 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66101.log 2024-01-16T15:30:16.553 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51448.log 2024-01-16T15:30:16.553 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.53126.log.gz 2024-01-16T15:30:16.554 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66101.log: gzip 87.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.66101.log.gz --verbose 2024-01-16T15:30:16.554 INFO:teuthology.orchestra.run.smithi053.stderr: -- /var/log/ceph/ceph-client.admin.71614.log 2024-01-16T15:30:16.555 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.42894.log 2024-01-16T15:30:16.556 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51448.log.gz 2024-01-16T15:30:16.557 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71614.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29648.log 2024-01-16T15:30:16.557 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71614.log.gz 2024-01-16T15:30:16.558 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.42894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80940.log 2024-01-16T15:30:16.559 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.tmp-client.admin.29648.log: 91.4% -- replaced with /var/log/ceph/ceph-client.0.42894.log.gz 2024-01-16T15:30:16.560 INFO:teuthology.orchestra.run.smithi053.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29648.log.gz 2024-01-16T15:30:16.560 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79989.log 2024-01-16T15:30:16.561 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80940.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80940.log.gz 2024-01-16T15:30:16.561 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61837.log 2024-01-16T15:30:16.562 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70401.log 2024-01-16T15:30:16.562 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.79989.log.gz 2024-01-16T15:30:16.564 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73796.log 2024-01-16T15:30:16.564 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.61837.log.gz 2024-01-16T15:30:16.565 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49883.log 2024-01-16T15:30:16.565 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70401.log.gz 2024-01-16T15:30:16.566 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67899.log 2024-01-16T15:30:16.567 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.73796.log.gz 2024-01-16T15:30:16.568 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74299.log 2024-01-16T15:30:16.568 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49883.log.gz 2024-01-16T15:30:16.569 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57689.log 2024-01-16T15:30:16.569 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.67899.log.gz 2024-01-16T15:30:16.570 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36132.log 2024-01-16T15:30:16.570 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74299.log.gz 2024-01-16T15:30:16.572 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44166.log 2024-01-16T15:30:16.572 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57689.log.gz 2024-01-16T15:30:16.573 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.36132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2024-01-16T15:30:16.573 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36132.log.gz 2024-01-16T15:30:16.574 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57834.log 2024-01-16T15:30:16.574 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44166.log.gz 2024-01-16T15:30:16.575 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73198.log 2024-01-16T15:30:16.576 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz 2024-01-16T15:30:16.577 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74976.log 2024-01-16T15:30:16.577 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57834.log.gz 2024-01-16T15:30:16.578 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77993.log 2024-01-16T15:30:16.578 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.73198.log.gz 2024-01-16T15:30:16.579 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74976.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74976.log.gzgzip 2024-01-16T15:30:16.580 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.79300.log 2024-01-16T15:30:16.581 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72222.log 2024-01-16T15:30:16.581 INFO:teuthology.orchestra.run.smithi053.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.77993.log.gz 2024-01-16T15:30:16.582 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59730.log 2024-01-16T15:30:16.582 INFO:teuthology.orchestra.run.smithi053.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.79300.log.gz 2024-01-16T15:30:16.583 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47942.log 2024-01-16T15:30:16.584 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.72222.log.gz 2024-01-16T15:30:16.585 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67261.log 2024-01-16T15:30:16.585 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59730.log.gz 2024-01-16T15:30:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74646.log 2024-01-16T15:30:16.587 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47942.log.gz 2024-01-16T15:30:16.587 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67261.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-01-16T15:30:16.588 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67261.log.gz 2024-01-16T15:30:16.589 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74646.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74646.log.gz 2024-01-16T15:30:16.589 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59001.log 2024-01-16T15:30:16.590 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41475.log 2024-01-16T15:30:16.591 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58006.log 2024-01-16T15:30:16.592 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59001.log.gz 2024-01-16T15:30:16.593 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41475.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41475.log.gzgzip 2024-01-16T15:30:16.593 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.72913.log 2024-01-16T15:30:16.595 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58006.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.58006.log.gz 2024-01-16T15:30:16.595 INFO:teuthology.orchestra.run.smithi053.stderr: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-01-16T15:30:16.595 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54890.log 2024-01-16T15:30:16.596 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62351.log 2024-01-16T15:30:16.596 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.72913.log.gz 2024-01-16T15:30:16.598 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54890.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67464.log 2024-01-16T15:30:16.598 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54890.log.gz 2024-01-16T15:30:16.599 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80031.log 2024-01-16T15:30:16.599 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62351.log.gz 2024-01-16T15:30:16.600 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64113.log 2024-01-16T15:30:16.600 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.67464.log.gz 2024-01-16T15:30:16.601 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76921.log 2024-01-16T15:30:16.601 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.80031.log.gz 2024-01-16T15:30:16.602 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47490.log 2024-01-16T15:30:16.603 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.64113.log.gz 2024-01-16T15:30:16.604 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71530.log 2024-01-16T15:30:16.604 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76921.log.gz 2024-01-16T15:30:16.605 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64487.log 2024-01-16T15:30:16.605 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47490.log.gz 2024-01-16T15:30:16.606 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51658.log 2024-01-16T15:30:16.606 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.71530.log.gz 2024-01-16T15:30:16.607 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81444.log 2024-01-16T15:30:16.608 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.64487.log.gz 2024-01-16T15:30:16.608 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45465.log 2024-01-16T15:30:16.609 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51658.log.gz 2024-01-16T15:30:16.610 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.75897.log 88.4% 2024-01-16T15:30:16.610 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.81444.log.gz 2024-01-16T15:30:16.611 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70233.log 2024-01-16T15:30:16.611 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45465.log.gz 2024-01-16T15:30:16.612 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.75897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73894.log 2024-01-16T15:30:16.614 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41123.log 2024-01-16T15:30:16.614 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70233.log.gz 2024-01-16T15:30:16.615 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73894.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49967.log 2024-01-16T15:30:16.616 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73894.log.gz 2024-01-16T15:30:16.616 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80898.log 2024-01-16T15:30:16.617 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49967.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41123.log.gz 2024-01-16T15:30:16.618 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53908.log 2024-01-16T15:30:16.618 INFO:teuthology.orchestra.run.smithi053.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.49967.log.gz 2024-01-16T15:30:16.620 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72474.log 2024-01-16T15:30:16.620 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80898.log.gz 2024-01-16T15:30:16.621 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77222.log 2024-01-16T15:30:16.621 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53908.log: /var/log/ceph/ceph-client.admin.72474.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53908.log.gz 2024-01-16T15:30:16.622 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.72474.log.gz 2024-01-16T15:30:16.622 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55569.log 2024-01-16T15:30:16.623 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77222.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.77222.log.gz 2024-01-16T15:30:16.624 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71446.log 2024-01-16T15:30:16.625 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39837.log 2024-01-16T15:30:16.625 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71446.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55569.log.gz 2024-01-16T15:30:16.626 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.71446.log.gz 2024-01-16T15:30:16.627 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44865.log 2024-01-16T15:30:16.627 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39837.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39837.log.gz 2024-01-16T15:30:16.628 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44639.log 2024-01-16T15:30:16.629 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-01-16T15:30:16.630 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44639.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44865.log.gz 2024-01-16T15:30:16.630 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.44639.log.gz 2024-01-16T15:30:16.631 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46656.log 2024-01-16T15:30:16.631 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41544.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-01-16T15:30:16.632 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74383.log 2024-01-16T15:30:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56009.log 2024-01-16T15:30:16.634 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74383.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.46656.log.gz 2024-01-16T15:30:16.635 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.74383.log.gz 2024-01-16T15:30:16.635 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65219.log 2024-01-16T15:30:16.636 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56009.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56009.log.gz 2024-01-16T15:30:16.636 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74215.log 2024-01-16T15:30:16.637 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65219.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40045.log 2024-01-16T15:30:16.638 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74215.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65219.log.gz 2024-01-16T15:30:16.639 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.74215.log.gz 2024-01-16T15:30:16.639 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57749.log 2024-01-16T15:30:16.639 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40045.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40045.log.gz 2024-01-16T15:30:16.640 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57792.log 2024-01-16T15:30:16.641 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63041.log 2024-01-16T15:30:16.642 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57749.log.gz 2024-01-16T15:30:16.643 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81234.log 2024-01-16T15:30:16.643 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57792.log.gz 2024-01-16T15:30:16.644 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63041.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.63041.log.gz 2024-01-16T15:30:16.644 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73240.log 2024-01-16T15:30:16.645 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54160.log/var/log/ceph/ceph-client.admin.81234.log: 2024-01-16T15:30:16.646 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73240.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81234.log.gz 2024-01-16T15:30:16.646 INFO:teuthology.orchestra.run.smithi053.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.73240.log.gz 2024-01-16T15:30:16.647 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55058.log 2024-01-16T15:30:16.647 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54160.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54160.log.gz 2024-01-16T15:30:16.648 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.68494.log 2024-01-16T15:30:16.649 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81024.log 2024-01-16T15:30:16.650 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.55058.log.gz 2024-01-16T15:30:16.650 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.68494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67341.log 2024-01-16T15:30:16.652 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67679.log 2024-01-16T15:30:16.652 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.81024.log.gz 2024-01-16T15:30:16.653 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67341.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.67341.log.gz 2024-01-16T15:30:16.654 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45975.log 2024-01-16T15:30:16.655 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67679.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.67679.log.gz 2024-01-16T15:30:16.655 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55883.log 2024-01-16T15:30:16.657 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64155.log 2024-01-16T15:30:16.657 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.45975.log.gz 2024-01-16T15:30:16.658 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62393.log 2024-01-16T15:30:16.658 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55883.log.gz 2024-01-16T15:30:16.660 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59814.log 2024-01-16T15:30:16.660 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64155.log.gz 2024-01-16T15:30:16.661 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47984.log 2024-01-16T15:30:16.661 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62393.log.gz 2024-01-16T15:30:16.662 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80167.log 2024-01-16T15:30:16.662 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59814.log.gz 2024-01-16T15:30:16.664 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47984.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2024-01-16T15:30:16.664 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47984.log.gz 2024-01-16T15:30:16.665 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35748.log 2024-01-16T15:30:16.665 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.80167.log.gz 2024-01-16T15:30:16.666 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65555.log 2024-01-16T15:30:16.666 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2024-01-16T15:30:16.668 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.35748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57032.log 2024-01-16T15:30:16.668 INFO:teuthology.orchestra.run.smithi053.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35748.log.gz 2024-01-16T15:30:16.669 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49520.log 2024-01-16T15:30:16.669 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65555.log.gz 2024-01-16T15:30:16.670 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43872.log 2024-01-16T15:30:16.671 INFO:teuthology.orchestra.run.smithi053.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.57032.log.gz 2024-01-16T15:30:16.672 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49520.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76711.log 2024-01-16T15:30:16.672 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49520.log.gz 2024-01-16T15:30:16.673 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39960.log 2024-01-16T15:30:16.673 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.43872.log.gz 2024-01-16T15:30:16.675 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40301.log 2024-01-16T15:30:16.675 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76711.log.gz 2024-01-16T15:30:16.676 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39960.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48260.log 2024-01-16T15:30:16.677 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39960.log.gz 2024-01-16T15:30:16.677 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65597.log 2024-01-16T15:30:16.677 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40301.log.gz 2024-01-16T15:30:16.679 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53950.log 2024-01-16T15:30:16.679 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48260.log.gz 2024-01-16T15:30:16.680 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51182.log 2024-01-16T15:30:16.681 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.65597.log.gz 2024-01-16T15:30:16.681 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50208.log 2024-01-16T15:30:16.682 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53950.log.gz 2024-01-16T15:30:16.683 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55799.log 2024-01-16T15:30:16.683 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51182.log.gz 2024-01-16T15:30:16.684 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76669.log 2024-01-16T15:30:16.685 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50208.log.gz 2024-01-16T15:30:16.685 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57479.log 2024-01-16T15:30:16.686 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55799.log.gz 2024-01-16T15:30:16.687 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52982.log 2024-01-16T15:30:16.687 INFO:teuthology.orchestra.run.smithi053.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.76669.log.gz 2024-01-16T15:30:16.689 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53350.log 2024-01-16T15:30:16.689 INFO:teuthology.orchestra.run.smithi053.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.57479.log.gz 2024-01-16T15:30:16.690 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65807.log 2024-01-16T15:30:16.690 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.52982.log.gz 2024-01-16T15:30:16.691 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64571.log 2024-01-16T15:30:16.691 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53350.log.gz 2024-01-16T15:30:16.693 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55525.log 2024-01-16T15:30:16.693 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65807.log.gz 2024-01-16T15:30:16.694 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50026.log 2024-01-16T15:30:16.694 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64571.log.gz 2024-01-16T15:30:16.695 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2024-01-16T15:30:16.696 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55525.log.gz 2024-01-16T15:30:16.697 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50026.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50505.log 2024-01-16T15:30:16.697 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50026.log.gz 2024-01-16T15:30:16.698 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41523.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41417.log 2024-01-16T15:30:16.698 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2024-01-16T15:30:16.699 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64697.log 2024-01-16T15:30:16.700 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.50505.log.gz 2024-01-16T15:30:16.701 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70443.log 2024-01-16T15:30:16.701 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41417.log.gz 2024-01-16T15:30:16.702 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63336.log 2024-01-16T15:30:16.702 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.64697.log.gz 2024-01-16T15:30:16.704 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63209.log 2024-01-16T15:30:16.704 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.70443.log.gz 2024-01-16T15:30:16.705 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65177.log 2024-01-16T15:30:16.705 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.63336.log.gz 2024-01-16T15:30:16.706 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66854.log 2024-01-16T15:30:16.706 INFO:teuthology.orchestra.run.smithi053.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63209.log.gz 2024-01-16T15:30:16.708 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58410.log 2024-01-16T15:30:16.708 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.65177.log.gz 2024-01-16T15:30:16.709 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51808.log 2024-01-16T15:30:16.710 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66854.log.gz 2024-01-16T15:30:16.710 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57074.log 2024-01-16T15:30:16.711 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58410.log.gz 2024-01-16T15:30:16.712 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77005.log 2024-01-16T15:30:16.712 INFO:teuthology.orchestra.run.smithi053.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51808.log.gz 2024-01-16T15:30:16.714 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51014.log 2024-01-16T15:30:16.714 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57074.log.gz 2024-01-16T15:30:16.715 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65345.log 2024-01-16T15:30:16.715 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.77005.log.gz 2024-01-16T15:30:16.716 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55841.log 2024-01-16T15:30:16.716 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.51014.log.gz 2024-01-16T15:30:16.718 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52814.log 2024-01-16T15:30:16.718 INFO:teuthology.orchestra.run.smithi053.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65345.log.gz 2024-01-16T15:30:16.719 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75620.log 2024-01-16T15:30:16.719 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55841.log.gz 2024-01-16T15:30:16.720 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57563.log 2024-01-16T15:30:16.720 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52814.log.gz 2024-01-16T15:30:16.722 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43552.log 2024-01-16T15:30:16.722 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75620.log.gz 2024-01-16T15:30:16.723 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44723.log 2024-01-16T15:30:16.723 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.57563.log.gz 2024-01-16T15:30:16.725 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60915.log 2024-01-16T15:30:16.725 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43552.log.gz 2024-01-16T15:30:16.726 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76879.log 2024-01-16T15:30:16.726 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44723.log.gz 2024-01-16T15:30:16.727 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42024.log 2024-01-16T15:30:16.728 INFO:teuthology.orchestra.run.smithi053.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.60915.log.gz 2024-01-16T15:30:16.729 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54034.log 2024-01-16T15:30:16.729 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.76879.log.gz 2024-01-16T15:30:16.730 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42024.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52898.log 2024-01-16T15:30:16.730 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42024.log.gz 2024-01-16T15:30:16.732 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48886.log 2024-01-16T15:30:16.732 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54034.log.gz 2024-01-16T15:30:16.733 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59772.log 2024-01-16T15:30:16.733 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52898.log.gz 2024-01-16T15:30:16.734 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58453.log 2024-01-16T15:30:16.734 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.48886.log.gz 2024-01-16T15:30:16.735 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76593.log 2024-01-16T15:30:16.736 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59772.log.gz 2024-01-16T15:30:16.737 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2024-01-16T15:30:16.737 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.58453.log.gz 2024-01-16T15:30:16.738 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80503.log 2024-01-16T15:30:16.738 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.76593.log.gz 2024-01-16T15:30:16.740 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55398.log 2024-01-16T15:30:16.740 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2024-01-16T15:30:16.741 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77688.log 2024-01-16T15:30:16.741 INFO:teuthology.orchestra.run.smithi053.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.80503.log.gz 2024-01-16T15:30:16.742 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40428.log 2024-01-16T15:30:16.742 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55398.log.gz 2024-01-16T15:30:16.744 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43704.log 2024-01-16T15:30:16.744 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.77688.log.gz 2024-01-16T15:30:16.745 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64383.log 2024-01-16T15:30:16.745 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.40428.log.gz 2024-01-16T15:30:16.746 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67729.log 2024-01-16T15:30:16.747 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.43704.log.gz 2024-01-16T15:30:16.748 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64383.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62309.log 2024-01-16T15:30:16.748 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64383.log.gz 2024-01-16T15:30:16.749 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46824.log 2024-01-16T15:30:16.749 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.67729.log.gz 2024-01-16T15:30:16.751 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65891.log 2024-01-16T15:30:16.751 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62309.log.gz 2024-01-16T15:30:16.752 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46824.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46614.log 2024-01-16T15:30:16.752 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46824.log.gz 2024-01-16T15:30:16.753 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58305.log 2024-01-16T15:30:16.753 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65891.log.gz 2024-01-16T15:30:16.755 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46614.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-01-16T15:30:16.755 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46614.log.gz 2024-01-16T15:30:16.756 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58090.log 2024-01-16T15:30:16.756 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.58305.log.gz 2024-01-16T15:30:16.757 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53084.log 2024-01-16T15:30:16.759 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71320.log 2024-01-16T15:30:16.759 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58090.log.gz 2024-01-16T15:30:16.760 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43510.log 2024-01-16T15:30:16.760 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53084.log.gz 2024-01-16T15:30:16.761 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75470.log 2024-01-16T15:30:16.762 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.71320.log.gz 2024-01-16T15:30:16.763 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46347.log 2024-01-16T15:30:16.763 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43510.log.gz 2024-01-16T15:30:16.764 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70074.log 2024-01-16T15:30:16.764 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.75470.log.gz 2024-01-16T15:30:16.765 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45849.log 2024-01-16T15:30:16.766 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46347.log.gz 2024-01-16T15:30:16.767 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46782.log 2024-01-16T15:30:16.767 INFO:teuthology.orchestra.run.smithi053.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.70074.log.gz 2024-01-16T15:30:16.768 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62123.log 2024-01-16T15:30:16.768 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45849.log.gz 2024-01-16T15:30:16.770 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44765.log 2024-01-16T15:30:16.770 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46782.log.gz 2024-01-16T15:30:16.771 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79054.log 2024-01-16T15:30:16.771 INFO:teuthology.orchestra.run.smithi053.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.62123.log.gz 2024-01-16T15:30:16.772 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44765.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42234.log 2024-01-16T15:30:16.773 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.44765.log.gz 2024-01-16T15:30:16.774 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49386.log 2024-01-16T15:30:16.774 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79054.log.gz 2024-01-16T15:30:16.775 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44907.log 2024-01-16T15:30:16.776 INFO:teuthology.orchestra.run.smithi053.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42234.log.gz 2024-01-16T15:30:16.776 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49386.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49386.log.gz 2024-01-16T15:30:16.778 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44907.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44907.log.gz 2024-01-16T15:30:16.778 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60831.log 2024-01-16T15:30:16.779 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79763.log 2024-01-16T15:30:16.781 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53587.log 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60831.log.gz 2024-01-16T15:30:16.781 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:16.782 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79763.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.79763.log.gz 2024-01-16T15:30:16.783 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64299.log 2024-01-16T15:30:16.784 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47280.log 2024-01-16T15:30:16.784 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53587.log.gz 2024-01-16T15:30:16.785 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78626.log 2024-01-16T15:30:16.786 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.64299.log.gz 2024-01-16T15:30:16.787 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47280.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47280.log.gz 2024-01-16T15:30:16.787 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55016.log 2024-01-16T15:30:16.788 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78626.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.78626.log.gz 2024-01-16T15:30:16.788 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-01-16T15:30:16.790 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68727.log 2024-01-16T15:30:16.790 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55016.log.gz 2024-01-16T15:30:16.791 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40173.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-01-16T15:30:16.792 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.68727.log: /var/log/ceph/ceph-client.admin.53169.log 2024-01-16T15:30:16.793 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.68727.log.gz 2024-01-16T15:30:16.793 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53846.log 2024-01-16T15:30:16.795 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47114.log 2024-01-16T15:30:16.795 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.53169.log.gz 2024-01-16T15:30:16.796 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53846.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53846.log.gz 2024-01-16T15:30:16.797 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60115.log 2024-01-16T15:30:16.797 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47114.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47114.log.gz 2024-01-16T15:30:16.798 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73039.log 2024-01-16T15:30:16.800 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77870.log 2024-01-16T15:30:16.800 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60115.log.gz 2024-01-16T15:30:16.801 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61533.log 2024-01-16T15:30:16.801 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.73039.log.gz 2024-01-16T15:30:16.802 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54806.log 2024-01-16T15:30:16.802 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77870.log.gz 2024-01-16T15:30:16.804 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46908.log 2024-01-16T15:30:16.804 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61533.log.gz 2024-01-16T15:30:16.805 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68895.log 2024-01-16T15:30:16.805 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54806.log.gz 2024-01-16T15:30:16.806 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46908.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46908.log.gz 2024-01-16T15:30:16.807 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77811.log 2024-01-16T15:30:16.808 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68895.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65261.log 2024-01-16T15:30:16.808 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68895.log.gz 2024-01-16T15:30:16.809 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66227.log 2024-01-16T15:30:16.809 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.77811.log.gz 2024-01-16T15:30:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65261.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65261.log.gz 2024-01-16T15:30:16.811 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-client.admin.66227.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50757.log 2024-01-16T15:30:16.812 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66227.log.gz 2024-01-16T15:30:16.813 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70952.log 2024-01-16T15:30:16.814 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60281.log 2024-01-16T15:30:16.814 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50757.log.gz 2024-01-16T15:30:16.816 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70952.log: gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.70952.log.gz -5 2024-01-16T15:30:16.816 INFO:teuthology.orchestra.run.smithi053.stderr: --verbose -- /var/log/ceph/ceph-client.admin.60323.log 2024-01-16T15:30:16.817 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55483.log 2024-01-16T15:30:16.817 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60281.log.gz 2024-01-16T15:30:16.819 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52696.log 2024-01-16T15:30:16.819 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60323.log.gz 2024-01-16T15:30:16.820 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69814.log 2024-01-16T15:30:16.820 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55483.log.gz 2024-01-16T15:30:16.821 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61997.log 2024-01-16T15:30:16.821 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52696.log.gz 2024-01-16T15:30:16.823 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41730.log 2024-01-16T15:30:16.823 INFO:teuthology.orchestra.run.smithi053.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.69814.log.gz 2024-01-16T15:30:16.824 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50286.log 2024-01-16T15:30:16.824 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.61997.log.gz 2024-01-16T15:30:16.825 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41730.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41730.log.gz 2024-01-16T15:30:16.826 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64529.log 2024-01-16T15:30:16.827 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50286.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50286.log.gz 2024-01-16T15:30:16.827 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42462.log 2024-01-16T15:30:16.828 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71218.log 2024-01-16T15:30:16.829 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.64529.log.gz 2024-01-16T15:30:16.829 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42462.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42462.log.gz 2024-01-16T15:30:16.830 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63927.log 2024-01-16T15:30:16.831 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63504.log 2024-01-16T15:30:16.832 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71218.log.gz 2024-01-16T15:30:16.833 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-01-16T15:30:16.833 INFO:teuthology.orchestra.run.smithi053.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63927.log.gz 2024-01-16T15:30:16.834 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48194.log 2024-01-16T15:30:16.834 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.63504.log.gz 2024-01-16T15:30:16.835 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40003.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-01-16T15:30:16.836 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57395.log 2024-01-16T15:30:16.837 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76204.log 2024-01-16T15:30:16.837 INFO:teuthology.orchestra.run.smithi053.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.48194.log.gz 2024-01-16T15:30:16.839 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64613.log 2024-01-16T15:30:16.839 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57395.log.gz 2024-01-16T15:30:16.840 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44376.log 2024-01-16T15:30:16.840 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76204.log.gz 2024-01-16T15:30:16.842 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64613.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67857.log 2024-01-16T15:30:16.842 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.64613.log.gz 2024-01-16T15:30:16.843 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79482.log 2024-01-16T15:30:16.843 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44376.log.gz 2024-01-16T15:30:16.844 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63378.log 2024-01-16T15:30:16.844 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.67857.log.gz 2024-01-16T15:30:16.846 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42150.log 2024-01-16T15:30:16.846 INFO:teuthology.orchestra.run.smithi053.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.79482.log.gz 2024-01-16T15:30:16.847 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66454.log 2024-01-16T15:30:16.847 INFO:teuthology.orchestra.run.smithi053.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.63378.log.gz 2024-01-16T15:30:16.848 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64740.log 2024-01-16T15:30:16.849 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42150.log.gz 2024-01-16T15:30:16.850 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66454.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66454.log.gz 2024-01-16T15:30:16.850 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78239.log 2024-01-16T15:30:16.851 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64740.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64740.log.gz 2024-01-16T15:30:16.851 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68853.log 2024-01-16T15:30:16.853 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78544.log 2024-01-16T15:30:16.853 INFO:teuthology.orchestra.run.smithi053.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.78239.log.gz 2024-01-16T15:30:16.854 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72778.log 2024-01-16T15:30:16.854 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.68853.log.gz 2024-01-16T15:30:16.856 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58580.log 2024-01-16T15:30:16.856 INFO:teuthology.orchestra.run.smithi053.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.78544.log.gz 2024-01-16T15:30:16.857 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72778.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.72778.log.gz 2024-01-16T15:30:16.858 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58580.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58580.log.gz 2024-01-16T15:30:16.859 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65429.log 2024-01-16T15:30:16.860 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62225.log 2024-01-16T15:30:16.861 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71572.log 2024-01-16T15:30:16.862 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65429.log.gz 2024-01-16T15:30:16.863 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62225.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.62225.log.gz 2024-01-16T15:30:16.863 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69444.log 2024-01-16T15:30:16.864 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71572.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.71572.log.gz 2024-01-16T15:30:16.865 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79136.log 2024-01-16T15:30:16.866 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79847.log 2024-01-16T15:30:16.866 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69444.log.gz 2024-01-16T15:30:16.868 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79136.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.79136.log.gz 2024-01-16T15:30:16.869 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/ceph-client.admin.79847.log: --verbose -- /var/log/ceph/ceph-client.admin.59469.log 2024-01-16T15:30:16.869 INFO:teuthology.orchestra.run.smithi053.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.79847.log.gz 2024-01-16T15:30:16.870 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75428.log 2024-01-16T15:30:16.871 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45279.log 2024-01-16T15:30:16.871 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59469.log.gz 2024-01-16T15:30:16.873 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75428.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.75428.log.gz 2024-01-16T15:30:16.873 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69755.log 2024-01-16T15:30:16.875 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66575.log 2024-01-16T15:30:16.875 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45279.log.gz 2024-01-16T15:30:16.876 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78380.log 2024-01-16T15:30:16.876 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.69755.log.gz 2024-01-16T15:30:16.877 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81630.log 2024-01-16T15:30:16.878 INFO:teuthology.orchestra.run.smithi053.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.66575.log.gz 2024-01-16T15:30:16.879 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58538.log 2024-01-16T15:30:16.879 INFO:teuthology.orchestra.run.smithi053.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.78380.log.gz 2024-01-16T15:30:16.880 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54689.log 2024-01-16T15:30:16.880 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.81630.log.gz 2024-01-16T15:30:16.882 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54244.log 2024-01-16T15:30:16.882 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58538.log.gz 2024-01-16T15:30:16.883 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54689.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54689.log.gz 2024-01-16T15:30:16.883 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75260.log 2024-01-16T15:30:16.884 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54244.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46431.log 2024-01-16T15:30:16.885 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.54244.log.gz 2024-01-16T15:30:16.886 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62729.log 2024-01-16T15:30:16.886 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75260.log.gz 2024-01-16T15:30:16.887 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46221.log 2024-01-16T15:30:16.887 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46431.log.gz 2024-01-16T15:30:16.889 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63969.log 2024-01-16T15:30:16.889 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62729.log.gz 2024-01-16T15:30:16.890 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46221.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46221.log.gz 2024-01-16T15:30:16.891 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54076.log 2024-01-16T15:30:16.891 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63969.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.63969.log.gz 2024-01-16T15:30:16.892 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63670.log 2024-01-16T15:30:16.893 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54076.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77065.log 2024-01-16T15:30:16.893 INFO:teuthology.orchestra.run.smithi053.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.54076.log.gz 2024-01-16T15:30:16.894 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59349.log 2024-01-16T15:30:16.895 INFO:teuthology.orchestra.run.smithi053.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.63670.log.gz 2024-01-16T15:30:16.896 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66905.log 2024-01-16T15:30:16.896 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.77065.log.gz 2024-01-16T15:30:16.897 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59349.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59349.log.gz 2024-01-16T15:30:16.898 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66905.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.66905.log.gz 2024-01-16T15:30:16.899 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.c.log: 92.0% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-01-16T15:30:16.899 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60621.log 2024-01-16T15:30:16.900 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44502.log 2024-01-16T15:30:16.902 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55655.log 2024-01-16T15:30:16.902 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60621.log.gz 2024-01-16T15:30:16.903 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44502.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44502.log.gz 2024-01-16T15:30:16.903 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72871.log 2024-01-16T15:30:16.904 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44124.log 2024-01-16T15:30:16.905 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55655.log.gz 2024-01-16T15:30:16.906 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61575.log 2024-01-16T15:30:16.906 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.72871.log.gz 2024-01-16T15:30:16.908 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44124.log: 89.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81486.log 2024-01-16T15:30:16.908 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44124.log.gz 2024-01-16T15:30:16.909 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61575.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.61575.log.gz 2024-01-16T15:30:16.909 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61921.log 2024-01-16T15:30:16.910 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65849.log 2024-01-16T15:30:16.911 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.81486.log.gz 2024-01-16T15:30:16.912 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61449.log 2024-01-16T15:30:16.912 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.61921.log.gz 2024-01-16T15:30:16.913 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44807.log 2024-01-16T15:30:16.913 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65849.log.gz 2024-01-16T15:30:16.915 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.61449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63825.log 2024-01-16T15:30:16.915 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.61449.log.gz 2024-01-16T15:30:16.916 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44807.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44807.log.gz 2024-01-16T15:30:16.917 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65051.log 2024-01-16T15:30:16.917 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63825.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.63825.log.gz 2024-01-16T15:30:16.918 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-01-16T15:30:16.919 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73653.log 2024-01-16T15:30:16.919 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.65051.log.gz 2024-01-16T15:30:16.921 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70359.log 2024-01-16T15:30:16.923 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73653.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.73653.log.gz 2024-01-16T15:30:16.923 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53253.log 2024-01-16T15:30:16.924 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70359.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.70359.log.gz 2024-01-16T15:30:16.925 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79905.log 2024-01-16T15:30:16.926 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53253.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.53253.log.gz 2024-01-16T15:30:16.927 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78749.log 2024-01-16T15:30:16.927 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79905.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79905.log.gz 2024-01-16T15:30:16.928 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40857.log 2024-01-16T15:30:16.929 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78749.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.78749.log.gz 2024-01-16T15:30:16.930 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55697.log 2024-01-16T15:30:16.931 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40857.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.40857.log.gz 2024-01-16T15:30:16.932 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50631.log 2024-01-16T15:30:16.933 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55697.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55697.log.gz 2024-01-16T15:30:16.934 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65135.log 2024-01-16T15:30:16.935 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50631.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50631.log.gz 2024-01-16T15:30:16.936 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63546.log 2024-01-16T15:30:16.936 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65135.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.65135.log.gz 2024-01-16T15:30:16.937 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45665.log 2024-01-16T15:30:16.939 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63546.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63546.log.gz 2024-01-16T15:30:16.940 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75020.log 2024-01-16T15:30:16.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45665.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45665.log.gz 2024-01-16T15:30:16.941 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75554.log 2024-01-16T15:30:16.943 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75020.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.75020.log.gz 2024-01-16T15:30:16.943 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41195.log 2024-01-16T15:30:16.944 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75554.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.75554.log.gz 2024-01-16T15:30:16.945 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47574.log 2024-01-16T15:30:16.947 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41195.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41195.log.gz 2024-01-16T15:30:16.947 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.47574.log: -- /var/log/ceph/ceph-client.admin.69996.log 2024-01-16T15:30:16.948 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47574.log.gz 2024-01-16T15:30:16.949 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66059.log 2024-01-16T15:30:16.950 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69996.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69996.log.gz 2024-01-16T15:30:16.951 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40217.log 2024-01-16T15:30:16.951 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66059.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.66059.log.gz 2024-01-16T15:30:16.952 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46866.log 2024-01-16T15:30:16.954 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40217.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.40217.log.gz 2024-01-16T15:30:16.955 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74509.log/var/log/ceph/ceph-client.admin.46866.log: 2024-01-16T15:30:16.955 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46866.log.gz 2024-01-16T15:30:16.956 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78198.log 2024-01-16T15:30:16.958 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74509.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.74509.log.gz 2024-01-16T15:30:16.958 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.78198.log: /var/log/ceph/ceph-client.admin.46572.log 2024-01-16T15:30:16.959 INFO:teuthology.orchestra.run.smithi053.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.78198.log.gz 2024-01-16T15:30:16.960 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49715.log 2024-01-16T15:30:16.961 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46572.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46572.log.gz 2024-01-16T15:30:16.963 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.49715.log: -- /var/log/ceph/ceph-client.admin.50852.log 2024-01-16T15:30:16.963 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49715.log.gz 2024-01-16T15:30:16.964 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40470.log 2024-01-16T15:30:16.965 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50852.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50852.log.gz 2024-01-16T15:30:16.966 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76753.log 2024-01-16T15:30:16.967 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40470.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40470.log.gz 2024-01-16T15:30:16.968 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59085.log 2024-01-16T15:30:16.970 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76753.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.76753.log.gz 2024-01-16T15:30:16.970 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81834.log 2024-01-16T15:30:16.970 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59085.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59085.log.gz 2024-01-16T15:30:16.971 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62813.log 2024-01-16T15:30:16.973 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81834.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.81834.log.gz 2024-01-16T15:30:16.973 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79721.log/var/log/ceph/ceph-client.admin.62813.log: 2024-01-16T15:30:16.974 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62813.log.gz 2024-01-16T15:30:16.975 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77952.log 2024-01-16T15:30:16.976 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79721.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.79721.log.gz 2024-01-16T15:30:16.978 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/ceph-client.admin.77952.log: --verbose -- /var/log/ceph/ceph-client.admin.49671.log 2024-01-16T15:30:16.978 INFO:teuthology.orchestra.run.smithi053.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.77952.log.gz 2024-01-16T15:30:16.979 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41856.log 2024-01-16T15:30:16.980 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49671.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.49671.log.gz 2024-01-16T15:30:16.981 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75344.log 2024-01-16T15:30:16.982 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41856.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.41856.log.gz 2024-01-16T15:30:16.983 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60495.log 2024-01-16T15:30:16.984 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75344.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.75344.log.gz 2024-01-16T15:30:16.985 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44544.log 2024-01-16T15:30:16.986 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60495.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.60495.log.gz 2024-01-16T15:30:16.987 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64799.log 2024-01-16T15:30:16.987 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44544.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.44544.log.gz 2024-01-16T15:30:16.988 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80461.log 2024-01-16T15:30:16.989 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64799.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.64799.log.gz 2024-01-16T15:30:16.990 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78116.log 2024-01-16T15:30:16.991 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80461.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.80461.log.gz 2024-01-16T15:30:16.993 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45807.log 2024-01-16T15:30:16.993 INFO:teuthology.orchestra.run.smithi053.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.78116.log.gz 2024-01-16T15:30:16.994 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62897.log 2024-01-16T15:30:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45807.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45807.log.gz 2024-01-16T15:30:16.996 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.62897.log: /var/log/ceph/ceph-client.admin.58665.log 2024-01-16T15:30:16.997 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.62897.log.gz 2024-01-16T15:30:16.998 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43662.log 2024-01-16T15:30:17.000 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58665.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58665.log.gz 2024-01-16T15:30:17.000 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/ceph-client.admin.43662.log: --verbose -- /var/log/ceph/ceph-client.admin.41940.log 2024-01-16T15:30:17.001 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43662.log.gz 2024-01-16T15:30:17.002 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45321.log 2024-01-16T15:30:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41940.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41940.log.gz 2024-01-16T15:30:17.004 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65975.log/var/log/ceph/ceph-client.admin.45321.log: 2024-01-16T15:30:17.004 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45321.log.gz 2024-01-16T15:30:17.005 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67124.log 2024-01-16T15:30:17.007 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65975.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.65975.log.gz 2024-01-16T15:30:17.008 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/ceph-client.admin.67124.log: --verbose -- /var/log/ceph/ceph-client.admin.68217.log 2024-01-16T15:30:17.008 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67124.log.gz 2024-01-16T15:30:17.009 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66779.log 2024-01-16T15:30:17.011 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68217.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.68217.log.gz 2024-01-16T15:30:17.012 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.66779.log: /var/log/ceph/ceph-client.admin.45581.log 2024-01-16T15:30:17.012 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.66779.log.gz 2024-01-16T15:30:17.013 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53042.log 2024-01-16T15:30:17.014 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45581.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45581.log.gz 2024-01-16T15:30:17.015 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58875.log 2024-01-16T15:30:17.016 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53042.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.53042.log.gz 2024-01-16T15:30:17.017 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60409.log 2024-01-16T15:30:17.018 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58875.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.58875.log.gz 2024-01-16T15:30:17.019 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81588.log 2024-01-16T15:30:17.020 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60409.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.60409.log.gz 2024-01-16T15:30:17.020 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56159.log 2024-01-16T15:30:17.022 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81588.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.81588.log.gz 2024-01-16T15:30:17.023 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5/var/log/ceph/ceph-client.admin.56159.log: --verbose -- /var/log/ceph/ceph-client.admin.55100.log 2024-01-16T15:30:17.023 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.56159.log.gz 2024-01-16T15:30:17.024 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75200.log 2024-01-16T15:30:17.025 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55100.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55100.log.gz 2024-01-16T15:30:17.026 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44250.log 2024-01-16T15:30:17.027 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75200.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.75200.log.gz 2024-01-16T15:30:17.028 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40596.log 2024-01-16T15:30:17.029 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44250.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44250.log.gz 2024-01-16T15:30:17.031 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.40596.log: -- /var/log/ceph/ceph-client.admin.72558.log 2024-01-16T15:30:17.031 INFO:teuthology.orchestra.run.smithi053.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40596.log.gz 2024-01-16T15:30:17.032 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80419.log 2024-01-16T15:30:17.033 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72558.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.72558.log.gz 2024-01-16T15:30:17.034 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-01-16T15:30:17.035 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80419.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80419.log.gz 2024-01-16T15:30:17.035 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52654.log 2024-01-16T15:30:17.037 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41190.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-01-16T15:30:17.038 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56661.log/var/log/ceph/ceph-client.admin.52654.log: 2024-01-16T15:30:17.038 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52654.log.gz 2024-01-16T15:30:17.039 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75512.log 2024-01-16T15:30:17.041 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56661.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56661.log.gz 2024-01-16T15:30:17.042 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-client.admin.75512.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51490.log 2024-01-16T15:30:17.042 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.75512.log.gz 2024-01-16T15:30:17.043 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80545.log 2024-01-16T15:30:17.044 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51490.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51490.log.gz 2024-01-16T15:30:17.045 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48068.log/var/log/ceph/ceph-client.admin.80545.log: 2024-01-16T15:30:17.046 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80545.log.gz 2024-01-16T15:30:17.047 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44418.log 2024-01-16T15:30:17.048 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48068.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48068.log.gz 2024-01-16T15:30:17.049 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68067.log 2024-01-16T15:30:17.050 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44418.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44418.log.gz 2024-01-16T15:30:17.051 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65639.log 2024-01-16T15:30:17.052 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68067.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.68067.log.gz 2024-01-16T15:30:17.053 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48110.log 2024-01-16T15:30:17.053 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65639.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.65639.log.gz 2024-01-16T15:30:17.054 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80293.log 2024-01-16T15:30:17.056 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48110.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.48110.log.gz 2024-01-16T15:30:17.057 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.80293.log: /var/log/ceph/ceph-client.admin.70910.log 2024-01-16T15:30:17.057 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.80293.log.gz 2024-01-16T15:30:17.058 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77729.log 2024-01-16T15:30:17.059 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70910.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.70910.log.gz 2024-01-16T15:30:17.060 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80209.log/var/log/ceph/ceph-client.admin.77729.log: 2024-01-16T15:30:17.061 INFO:teuthology.orchestra.run.smithi053.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.77729.log.gz 2024-01-16T15:30:17.062 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68769.log 2024-01-16T15:30:17.063 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80209.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.80209.log.gz 2024-01-16T15:30:17.064 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.68769.log: -- /var/log/ceph/ceph-client.admin.69587.log 2024-01-16T15:30:17.065 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.68769.log.gz 2024-01-16T15:30:17.066 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49302.log 2024-01-16T15:30:17.068 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69587.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69587.log.gz 2024-01-16T15:30:17.068 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81752.log 2024-01-16T15:30:17.069 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49302.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49302.log.gz 2024-01-16T15:30:17.069 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80982.log 2024-01-16T15:30:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81752.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.81752.log.gz 2024-01-16T15:30:17.071 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51616.log 2024-01-16T15:30:17.072 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80982.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.80982.log.gz 2024-01-16T15:30:17.073 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76078.log 2024-01-16T15:30:17.074 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51616.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51616.log.gz 2024-01-16T15:30:17.075 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53211.log 2024-01-16T15:30:17.076 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76078.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.76078.log.gz 2024-01-16T15:30:17.077 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.48537.log 2024-01-16T15:30:17.079 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53211.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53211.log.gz 2024-01-16T15:30:17.079 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40638.log/var/log/ceph/ceph-client.0.48537.log: 2024-01-16T15:30:17.082 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70158.log 2024-01-16T15:30:17.083 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40638.log.gz 2024-01-16T15:30:17.084 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73324.log 2024-01-16T15:30:17.087 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70158.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.70158.log.gz 2024-01-16T15:30:17.089 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73485.log 2024-01-16T15:30:17.090 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.73324.log.gz 2024-01-16T15:30:17.092 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59940.log 2024-01-16T15:30:17.094 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73485.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.73485.log.gz 2024-01-16T15:30:17.097 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71740.log 2024-01-16T15:30:17.099 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59940.log.gz 2024-01-16T15:30:17.099 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68151.log 2024-01-16T15:30:17.101 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71740.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.71740.log.gz 2024-01-16T15:30:17.103 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78321.log 2024-01-16T15:30:17.106 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68151.log: 90.9%/var/log/ceph/ceph-client.admin.78321.log: -- replaced with /var/log/ceph/ceph-client.admin.68151.log.gz 2024-01-16T15:30:17.107 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46305.log 2024-01-16T15:30:17.108 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41191.log 2024-01-16T15:30:17.108 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78321.log.gz 2024-01-16T15:30:17.110 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46305.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46305.log.gz 2024-01-16T15:30:17.113 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41191.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41191.log.gz 2024-01-16T15:30:17.114 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58048.log 2024-01-16T15:30:17.116 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45033.log 2024-01-16T15:30:17.118 INFO:teuthology.orchestra.run.smithi053.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.58048.log.gz 2024-01-16T15:30:17.119 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77180.log 2024-01-16T15:30:17.124 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45033.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-01-16T15:30:17.124 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.45033.log.gz 2024-01-16T15:30:17.125 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77180.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.77180.log.gz 2024-01-16T15:30:17.127 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59427.log 2024-01-16T15:30:17.132 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59427.log: /var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62645.log 2024-01-16T15:30:17.133 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53803.log 2024-01-16T15:30:17.135 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59427.log.gz 2024-01-16T15:30:17.139 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62645.log: /var/log/ceph/ceph-client.admin.53803.log: 87.3% 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53803.log.gz 2024-01-16T15:30:17.139 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.62645.log.gz 2024-01-16T15:30:17.140 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59127.log 2024-01-16T15:30:17.144 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57964.log 2024-01-16T15:30:17.146 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59127.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.59127.log.gz 2024-01-16T15:30:17.148 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56787.log 2024-01-16T15:30:17.149 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57964.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57964.log.gz 2024-01-16T15:30:17.153 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56787.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56787.log.gz 2024-01-16T15:30:17.155 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53440.log 2024-01-16T15:30:17.158 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52224.log 2024-01-16T15:30:17.162 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73695.log 2024-01-16T15:30:17.164 INFO:teuthology.orchestra.run.smithi053.stderr: 87.1%/var/log/ceph/ceph-client.admin.52224.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52224.log.gz 2024-01-16T15:30:17.164 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53440.log.gz 2024-01-16T15:30:17.165 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49057.log 2024-01-16T15:30:17.167 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40680.log 2024-01-16T15:30:17.169 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2%/var/log/ceph/ceph-client.admin.49057.log: -- replaced with /var/log/ceph/ceph-client.admin.73695.log.gz 2024-01-16T15:30:17.170 INFO:teuthology.orchestra.run.smithi053.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49057.log.gz 2024-01-16T15:30:17.172 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70664.log 2024-01-16T15:30:17.173 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40680.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40680.log.gz 2024-01-16T15:30:17.176 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70664.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70664.log.gz 2024-01-16T15:30:17.176 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55195.log 2024-01-16T15:30:17.179 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50547.log 2024-01-16T15:30:17.180 INFO:teuthology.orchestra.run.smithi053.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.55195.log.gz 2024-01-16T15:30:17.185 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50547.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.50547.log.gz 2024-01-16T15:30:17.185 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71488.log 2024-01-16T15:30:17.186 INFO:teuthology.orchestra.run.smithi053.stderr: 94.6% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-01-16T15:30:17.187 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40845.log 2024-01-16T15:30:17.189 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71488.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71488.log.gz 2024-01-16T15:30:17.190 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44082.log 2024-01-16T15:30:17.192 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66956.log 2024-01-16T15:30:17.194 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44082.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44082.log.gz 2024-01-16T15:30:17.194 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40845.log.gz 2024-01-16T15:30:17.198 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70527.log 2024-01-16T15:30:17.198 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.66956.log.gz 2024-01-16T15:30:17.199 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81792.log 2024-01-16T15:30:17.201 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80335.log 2024-01-16T15:30:17.203 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2%/var/log/ceph/ceph-client.admin.81792.log: -- replaced with /var/log/ceph/ceph-client.admin.70527.log.gz 2024-01-16T15:30:17.204 INFO:teuthology.orchestra.run.smithi053.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.81792.log.gz 2024-01-16T15:30:17.206 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64239.log 2024-01-16T15:30:17.208 INFO:teuthology.orchestra.run.smithi146.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-01-16T15:30:17.209 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60365.log 2024-01-16T15:30:17.210 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64239.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.64239.log.gz 2024-01-16T15:30:17.210 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.80335.log.gz 2024-01-16T15:30:17.213 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60365.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60365.log.gz 2024-01-16T15:30:17.214 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45195.log 2024-01-16T15:30:17.217 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57353.log 2024-01-16T15:30:17.220 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57353.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.45195.log.gz 2024-01-16T15:30:17.221 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72516.log 2024-01-16T15:30:17.222 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80377.log 2024-01-16T15:30:17.225 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4%/var/log/ceph/ceph-client.admin.72516.log: -- replaced with /var/log/ceph/ceph-client.admin.57353.log.gz 2024-01-16T15:30:17.229 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9%/var/log/ceph/ceph-client.admin.80377.log: 88.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48718.log 2024-01-16T15:30:17.232 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48718.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48718.log.gz 2024-01-16T15:30:17.233 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.80377.log.gz -- /var/log/ceph/ceph-client.admin.57605.log 2024-01-16T15:30:17.233 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:17.233 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.72516.log.gz 2024-01-16T15:30:17.236 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56436.log 2024-01-16T15:30:17.240 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.56436.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57605.log.gz 2024-01-16T15:30:17.240 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62855.log 2024-01-16T15:30:17.244 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79013.log 2024-01-16T15:30:17.248 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.62855.log.gz 2024-01-16T15:30:17.250 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79013.log: 87.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58791.log 2024-01-16T15:30:17.251 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.79013.log.gz 2024-01-16T15:30:17.254 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56745.log 2024-01-16T15:30:17.255 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58791.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.58791.log.gz 2024-01-16T15:30:17.260 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79400.log 2024-01-16T15:30:17.261 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56745.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56745.log.gz 2024-01-16T15:30:17.262 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47796.log 2024-01-16T15:30:17.263 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79400.log: 85.0% -- replaced with /var/log/ceph/ceph-client.admin.79400.log.gz 2024-01-16T15:30:17.269 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59511.log 2024-01-16T15:30:17.270 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47796.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47796.log.gz 2024-01-16T15:30:17.272 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.61266.log 2024-01-16T15:30:17.277 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55739.log 2024-01-16T15:30:17.277 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59511.log.gz 2024-01-16T15:30:17.281 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.61266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60705.log 2024-01-16T15:30:17.283 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55739.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55739.log.gz 2024-01-16T15:30:17.286 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41982.log 2024-01-16T15:30:17.288 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60705.log.gz 2024-01-16T15:30:17.290 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70317.log 2024-01-16T15:30:17.291 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41982.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.41982.log.gz 2024-01-16T15:30:17.293 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45423.log 2024-01-16T15:30:17.295 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70317.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70317.log.gz 2024-01-16T15:30:17.298 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64841.log 2024-01-16T15:30:17.300 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45423.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45423.log.gz 2024-01-16T15:30:17.304 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73611.log 2024-01-16T15:30:17.305 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64841.log.gz 2024-01-16T15:30:17.307 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43830.log 2024-01-16T15:30:17.313 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73611.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.73611.log.gz/var/log/ceph/ceph-client.admin.43830.log: 2024-01-16T15:30:17.313 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.65933.log 2024-01-16T15:30:17.313 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43830.log.gz 2024-01-16T15:30:17.315 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51574.log 2024-01-16T15:30:17.317 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65933.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.65933.log.gz 2024-01-16T15:30:17.318 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64053.log 2024-01-16T15:30:17.320 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51574.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51574.log.gz 2024-01-16T15:30:17.323 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.64053.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.64053.log.gz 2024-01-16T15:30:17.324 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55356.log 2024-01-16T15:30:17.329 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80814.log 2024-01-16T15:30:17.331 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55356.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55356.log.gz 2024-01-16T15:30:17.332 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62519.log 2024-01-16T15:30:17.334 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80814.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.80814.log.gz 2024-01-16T15:30:17.334 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-01-16T15:30:17.336 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62519.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.62519.log.gz 2024-01-16T15:30:17.336 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68811.log 2024-01-16T15:30:17.340 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80688.log 2024-01-16T15:30:17.344 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68811.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.68811.log.gz 2024-01-16T15:30:17.345 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74848.log 2024-01-16T15:30:17.347 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80688.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.80688.log.gz 2024-01-16T15:30:17.348 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77523.log 2024-01-16T15:30:17.350 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74848.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74848.log.gz 2024-01-16T15:30:17.351 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71132.log 2024-01-16T15:30:17.353 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77523.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.77523.log.gz 2024-01-16T15:30:17.356 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71132.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.71132.log.gz 2024-01-16T15:30:17.356 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66353.log 2024-01-16T15:30:17.358 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62771.log 2024-01-16T15:30:17.360 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66353.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.66353.log.gz 2024-01-16T15:30:17.366 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62771.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.62771.log.gz 2024-01-16T15:30:17.366 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77481.log 2024-01-16T15:30:17.371 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47364.log 2024-01-16T15:30:17.372 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77481.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.77481.log.gz 2024-01-16T15:30:17.375 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66311.log 2024-01-16T15:30:17.376 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47364.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47364.log.gz 2024-01-16T15:30:17.379 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63588.log 2024-01-16T15:30:17.380 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66311.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.66311.log.gz 2024-01-16T15:30:17.383 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79218.log 2024-01-16T15:30:17.387 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71082.log 2024-01-16T15:30:17.387 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.63588.log.gz 2024-01-16T15:30:17.392 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79218.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.79218.log.gz 2024-01-16T15:30:17.392 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51226.log 2024-01-16T15:30:17.392 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.71082.log.gz 2024-01-16T15:30:17.397 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46992.log 2024-01-16T15:30:17.398 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51226.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51226.log.gz 2024-01-16T15:30:17.402 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-client.admin.46992.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50166.log 2024-01-16T15:30:17.405 INFO:teuthology.orchestra.run.smithi053.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75064.log 2024-01-16T15:30:17.405 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46992.log.gz 2024-01-16T15:30:17.410 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50166.log: /var/log/ceph/ceph-client.admin.75064.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50166.log.gz 2024-01-16T15:30:17.410 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67166.log 2024-01-16T15:30:17.411 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.75064.log.gz 2024-01-16T15:30:17.413 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48802.log 2024-01-16T15:30:17.414 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67166.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.67166.log.gz 2024-01-16T15:30:17.418 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52392.log 2024-01-16T15:30:17.419 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48802.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.48802.log.gz 2024-01-16T15:30:17.422 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73282.log 2024-01-16T15:30:17.423 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52392.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52392.log.gz 2024-01-16T15:30:17.425 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52266.log 2024-01-16T15:30:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73282.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.73282.log.gz 2024-01-16T15:30:17.430 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52266.log: -- /var/log/ceph/ceph-client.admin.45623.log 2024-01-16T15:30:17.431 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52266.log.gz 2024-01-16T15:30:17.437 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-client.admin.45623.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.45623.log.gz 2024-01-16T15:30:17.437 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.68979.log 2024-01-16T15:30:17.440 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56990.log 2024-01-16T15:30:17.441 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68979.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.68979.log.gz 2024-01-16T15:30:17.443 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53629.log 2024-01-16T15:30:17.444 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56990.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.56990.log.gz 2024-01-16T15:30:17.444 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40386.log 2024-01-16T15:30:17.448 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40386.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40386.log.gz 2024-01-16T15:30:17.449 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53629.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.53629.log.gz 2024-01-16T15:30:17.449 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69671.log 2024-01-16T15:30:17.452 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60747.log 2024-01-16T15:30:17.454 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69671.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.69671.log.gz 2024-01-16T15:30:17.455 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55613.log 2024-01-16T15:30:17.457 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72432.log 2024-01-16T15:30:17.459 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55613.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.55613.log.gz 2024-01-16T15:30:17.460 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72432.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60747.log.gz 2024-01-16T15:30:17.460 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.72432.log.gz 2024-01-16T15:30:17.461 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.52085.log.gz 2024-01-16T15:30:17.462 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51320.log 2024-01-16T15:30:17.464 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54118.log 2024-01-16T15:30:17.468 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51320.log: /var/log/ceph/ceph-client.admin.54118.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.51320.log.gz 2024-01-16T15:30:17.468 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66269.log 2024-01-16T15:30:17.471 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66269.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54118.log.gz 2024-01-16T15:30:17.472 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-01-16T15:30:17.474 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52350.log 2024-01-16T15:30:17.476 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.66269.log.gz 2024-01-16T15:30:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52350.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52350.log.gz 2024-01-16T15:30:17.479 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2024-01-16T15:30:17.484 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55967.log 2024-01-16T15:30:17.484 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2024-01-16T15:30:17.485 INFO:teuthology.orchestra.run.smithi053.stderr: 86.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-01-16T15:30:17.486 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46389.log 2024-01-16T15:30:17.491 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.55967.log: /var/log/ceph/ceph-client.admin.46389.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78972.log 2024-01-16T15:30:17.491 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55967.log.gz 2024-01-16T15:30:17.492 INFO:teuthology.orchestra.run.smithi053.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.46389.log.gz 2024-01-16T15:30:17.494 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42360.log 2024-01-16T15:30:17.498 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78972.log: /var/log/ceph/ceph-client.admin.42360.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66017.log 2024-01-16T15:30:17.498 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.78972.log.gz 2024-01-16T15:30:17.501 INFO:teuthology.orchestra.run.smithi053.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42360.log.gz 2024-01-16T15:30:17.503 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-client.admin.66017.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.79606.log 2024-01-16T15:30:17.506 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79606.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.66017.log.gz 2024-01-16T15:30:17.506 INFO:teuthology.orchestra.run.smithi053.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43998.log 2024-01-16T15:30:17.508 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79095.log 2024-01-16T15:30:17.510 INFO:teuthology.orchestra.run.smithi053.stderr: 87.7%/var/log/ceph/ceph-client.admin.43998.log: -- replaced with /var/log/ceph/ceph-client.admin.79606.log.gz 2024-01-16T15:30:17.511 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43998.log.gz 2024-01-16T15:30:17.513 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57647.log 2024-01-16T15:30:17.516 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57647.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79095.log.gz 2024-01-16T15:30:17.517 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59688.log 2024-01-16T15:30:17.519 INFO:teuthology.orchestra.run.smithi053.stderr:gzip 88.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.39741.log 2024-01-16T15:30:17.520 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59688.log: -- replaced with /var/log/ceph/ceph-client.admin.57647.log.gz 2024-01-16T15:30:17.521 INFO:teuthology.orchestra.run.smithi053.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59688.log.gz 2024-01-16T15:30:17.523 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-01-16T15:30:17.527 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.39741.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71656.log 2024-01-16T15:30:17.529 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39741.log.gz 2024-01-16T15:30:17.532 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71656.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.71656.log.gz 2024-01-16T15:30:17.532 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2024-01-16T15:30:17.535 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81318.log 2024-01-16T15:30:17.536 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54527.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gz 2024-01-16T15:30:17.539 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48676.log 2024-01-16T15:30:17.540 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81318.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.81318.log.gz 2024-01-16T15:30:17.544 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.48676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41197.log 2024-01-16T15:30:17.545 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48676.log.gz 2024-01-16T15:30:17.551 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50932.log 2024-01-16T15:30:17.551 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41197.log.gz 2024-01-16T15:30:17.554 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39642.log 2024-01-16T15:30:17.555 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50932.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50932.log.gz 2024-01-16T15:30:17.559 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39642.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78280.log 2024-01-16T15:30:17.559 INFO:teuthology.orchestra.run.smithi053.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.39642.log.gz 2024-01-16T15:30:17.562 INFO:teuthology.orchestra.run.smithi053.stderr:gzip/var/log/ceph/ceph-client.admin.78280.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.66185.log 2024-01-16T15:30:17.563 INFO:teuthology.orchestra.run.smithi053.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.78280.log.gz 2024-01-16T15:30:17.566 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-01-16T15:30:17.567 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66185.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66185.log.gz 2024-01-16T15:30:17.571 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74726.log 2024-01-16T15:30:17.576 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62957.log 2024-01-16T15:30:17.578 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74726.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.74726.log.gz 2024-01-16T15:30:17.582 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62957.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.62957.log.gz 2024-01-16T15:30:17.582 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76036.log 2024-01-16T15:30:17.587 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78075.log 2024-01-16T15:30:17.589 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76036.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76036.log.gz 2024-01-16T15:30:17.592 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42192.log 2024-01-16T15:30:17.593 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.78075.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.78075.log.gz 2024-01-16T15:30:17.596 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67771.log 2024-01-16T15:30:17.597 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42192.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.42192.log.gz 2024-01-16T15:30:17.604 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67771.log: 88.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46740.log 2024-01-16T15:30:17.604 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.67771.log.gz 2024-01-16T15:30:17.606 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62603.log 2024-01-16T15:30:17.610 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46740.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.46740.log.gz 2024-01-16T15:30:17.610 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52434.log 2024-01-16T15:30:17.612 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62603.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.62603.log.gz 2024-01-16T15:30:17.612 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69486.log 2024-01-16T15:30:17.616 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52434.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52434.log.gz 2024-01-16T15:30:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69486.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69486.log.gz 2024-01-16T15:30:17.619 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63167.log 2024-01-16T15:30:17.622 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54443.log 2024-01-16T15:30:17.623 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63167.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.63167.log.gz 2024-01-16T15:30:17.624 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69342.log 2024-01-16T15:30:17.625 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54443.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54443.log.gz 2024-01-16T15:30:17.628 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60238.log 2024-01-16T15:30:17.629 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69342.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.69342.log.gz 2024-01-16T15:30:17.631 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76162.log 2024-01-16T15:30:17.632 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60238.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60238.log.gz 2024-01-16T15:30:17.636 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76162.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76162.log.gz 2024-01-16T15:30:17.636 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42998.log 2024-01-16T15:30:17.639 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65009.log 2024-01-16T15:30:17.643 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42998.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42998.log.gz 2024-01-16T15:30:17.646 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65009.log: 89.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74056.log 2024-01-16T15:30:17.646 INFO:teuthology.orchestra.run.smithi053.stderr: -- replaced with /var/log/ceph/ceph-client.admin.65009.log.gz 2024-01-16T15:30:17.650 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71278.log 2024-01-16T15:30:17.651 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74056.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74056.log.gz 2024-01-16T15:30:17.653 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50124.log 2024-01-16T15:30:17.654 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71278.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.71278.log.gz 2024-01-16T15:30:17.656 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81360.log 2024-01-16T15:30:17.657 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50124.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50124.log.gz 2024-01-16T15:30:17.661 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44292.log 2024-01-16T15:30:17.662 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81360.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.81360.log.gz 2024-01-16T15:30:17.667 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44292.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.44292.log.gz 2024-01-16T15:30:17.668 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69199.log 2024-01-16T15:30:17.670 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.72083.log 2024-01-16T15:30:17.674 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47704.log 2024-01-16T15:30:17.676 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.0.72083.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.69199.log.gz 2024-01-16T15:30:17.678 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47704.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.47704.log.gz 2024-01-16T15:30:17.681 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57521.log 2024-01-16T15:30:17.683 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65093.log 2024-01-16T15:30:17.685 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50388.log 2024-01-16T15:30:17.685 INFO:teuthology.orchestra.run.smithi053.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57521.log.gz 2024-01-16T15:30:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65093.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.65093.log.gz 2024-01-16T15:30:17.687 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44681.log 2024-01-16T15:30:17.689 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60981.log 2024-01-16T15:30:17.691 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50388.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50388.log.gz 2024-01-16T15:30:17.695 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.44681.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44681.log.gz 2024-01-16T15:30:17.696 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63125.log 2024-01-16T15:30:17.696 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60981.log.gz 2024-01-16T15:30:17.698 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81150.log 2024-01-16T15:30:17.700 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63125.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63125.log.gz 2024-01-16T15:30:17.702 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76288.log 2024-01-16T15:30:17.704 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81150.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.81150.log.gz 2024-01-16T15:30:17.707 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-01-16T15:30:17.708 INFO:teuthology.orchestra.run.smithi053.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.76288.log.gz 2024-01-16T15:30:17.710 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43788.log 2024-01-16T15:30:17.712 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41772.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-01-16T15:30:17.715 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.43788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-01-16T15:30:17.716 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43788.log.gz 2024-01-16T15:30:17.718 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67507.log 2024-01-16T15:30:17.720 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.40129.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-01-16T15:30:17.723 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.67507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50673.log 2024-01-16T15:30:17.724 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.67507.log.gz 2024-01-16T15:30:17.726 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60663.log 2024-01-16T15:30:17.728 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50673.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50673.log.gz 2024-01-16T15:30:17.731 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60663.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57227.log 2024-01-16T15:30:17.732 INFO:teuthology.orchestra.run.smithi053.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60663.log.gz 2024-01-16T15:30:17.734 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52940.log 2024-01-16T15:30:17.736 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.57227.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57227.log.gz 2024-01-16T15:30:17.739 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.52940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80856.log 2024-01-16T15:30:17.740 INFO:teuthology.orchestra.run.smithi053.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52940.log.gz 2024-01-16T15:30:17.742 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49622.log 2024-01-16T15:30:17.744 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80856.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80856.log.gz 2024-01-16T15:30:17.748 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49622.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.49622.log.gz 2024-01-16T15:30:17.748 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62687.log 2024-01-16T15:30:17.750 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58496.log 2024-01-16T15:30:17.753 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62687.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.62687.log.gz 2024-01-16T15:30:17.754 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2024-01-16T15:30:17.756 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58496.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.58496.log.gz 2024-01-16T15:30:17.758 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79524.log 2024-01-16T15:30:17.760 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45507.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2024-01-16T15:30:17.764 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79524.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.79524.log.gz 2024-01-16T15:30:17.765 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63711.log 2024-01-16T15:30:17.769 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58917.log 2024-01-16T15:30:17.771 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63711.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.63711.log.gz 2024-01-16T15:30:17.773 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49141.log 2024-01-16T15:30:17.775 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58917.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.58917.log.gz 2024-01-16T15:30:17.777 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39918.log 2024-01-16T15:30:17.779 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.49141.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49141.log.gz 2024-01-16T15:30:17.781 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72390.log 2024-01-16T15:30:17.783 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.39918.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.39918.log.gz 2024-01-16T15:30:17.785 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46530.log 2024-01-16T15:30:17.787 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72390.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.72390.log.gz 2024-01-16T15:30:17.791 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.46530.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46530.log.gz 2024-01-16T15:30:17.792 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73737.log 2024-01-16T15:30:17.797 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41198.log 2024-01-16T15:30:17.799 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73737.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.73737.log.gz 2024-01-16T15:30:17.800 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.72955.log 2024-01-16T15:30:17.802 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41198.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41198.log.gz 2024-01-16T15:30:17.802 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59898.log 2024-01-16T15:30:17.804 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.72955.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.72955.log.gz 2024-01-16T15:30:17.806 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53761.log 2024-01-16T15:30:17.808 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.59898.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59898.log.gz 2024-01-16T15:30:17.810 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45749.log 2024-01-16T15:30:17.812 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53761.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.53761.log.gz 2024-01-16T15:30:17.815 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.45749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81108.log 2024-01-16T15:30:17.816 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45749.log.gz 2024-01-16T15:30:17.818 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51270.log 2024-01-16T15:30:17.820 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81108.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.81108.log.gz 2024-01-16T15:30:17.823 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.51270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79441.log 2024-01-16T15:30:17.824 INFO:teuthology.orchestra.run.smithi053.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51270.log.gz 2024-01-16T15:30:17.827 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.81402.log 2024-01-16T15:30:17.829 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.79441.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.79441.log.gz 2024-01-16T15:30:17.831 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70866.log 2024-01-16T15:30:17.833 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.81402.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.81402.log.gz 2024-01-16T15:30:17.833 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56577.log 2024-01-16T15:30:17.835 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70866.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.70866.log.gz 2024-01-16T15:30:17.836 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60196.log 2024-01-16T15:30:17.838 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.56577.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56577.log.gz 2024-01-16T15:30:17.841 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58175.log 2024-01-16T15:30:17.843 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60196.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.60196.log.gz 2024-01-16T15:30:17.845 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58175.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58175.log.gz 2024-01-16T15:30:17.846 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60873.log 2024-01-16T15:30:17.848 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76533.log 2024-01-16T15:30:17.850 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.60873.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60873.log.gz 2024-01-16T15:30:17.852 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74551.log 2024-01-16T15:30:17.854 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76533.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.76533.log.gz 2024-01-16T15:30:17.855 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69629.log 2024-01-16T15:30:17.858 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74551.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.74551.log.gz 2024-01-16T15:30:17.860 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62267.log 2024-01-16T15:30:17.862 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.69629.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.69629.log.gz 2024-01-16T15:30:17.864 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63782.log 2024-01-16T15:30:17.866 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.62267.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.62267.log.gz 2024-01-16T15:30:17.870 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63782.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.63782.log.gz 2024-01-16T15:30:17.870 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64925.log 2024-01-16T15:30:17.872 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47237.log 2024-01-16T15:30:17.876 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66737.log 2024-01-16T15:30:17.878 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47237.log: /var/log/ceph/ceph-client.admin.64925.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47237.log.gz 2024-01-16T15:30:17.878 INFO:teuthology.orchestra.run.smithi053.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.64925.log.gz 2024-01-16T15:30:17.879 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66737.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.66737.log.gz 2024-01-16T15:30:17.882 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63629.log 2024-01-16T15:30:17.886 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66695.log 2024-01-16T15:30:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.63629.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.63629.log.gz 2024-01-16T15:30:17.890 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42318.log 2024-01-16T15:30:17.892 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66695.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.66695.log.gz 2024-01-16T15:30:17.894 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71806.log 2024-01-16T15:30:17.896 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.42318.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42318.log.gz 2024-01-16T15:30:17.898 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77647.log 2024-01-16T15:30:17.900 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71806.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.71806.log.gz 2024-01-16T15:30:17.903 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74806.log 2024-01-16T15:30:17.904 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77647.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.77647.log.gz 2024-01-16T15:30:17.908 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.74806.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.74806.log.gz 2024-01-16T15:30:17.910 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54286.log 2024-01-16T15:30:17.911 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41133.log 2024-01-16T15:30:17.913 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.54286.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54286.log.gz 2024-01-16T15:30:17.913 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76246.log 2024-01-16T15:30:17.915 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.41133.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.41133.log.gz 2024-01-16T15:30:17.917 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47406.log 2024-01-16T15:30:17.919 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.76246.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.76246.log.gz 2024-01-16T15:30:17.919 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-01-16T15:30:17.921 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.47406.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47406.log.gz 2024-01-16T15:30:17.922 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.80772.log 2024-01-16T15:30:17.929 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65681.log 2024-01-16T15:30:17.932 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.80772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58623.log 2024-01-16T15:30:17.932 INFO:teuthology.orchestra.run.smithi053.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.80772.log.gz 2024-01-16T15:30:17.934 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.65681.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.65681.log.gz 2024-01-16T15:30:17.936 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50328.log 2024-01-16T15:30:17.937 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58623.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68685.log 2024-01-16T15:30:17.938 INFO:teuthology.orchestra.run.smithi053.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.58623.log.gz 2024-01-16T15:30:17.940 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.50328.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50328.log.gz 2024-01-16T15:30:17.940 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77911.log 2024-01-16T15:30:17.942 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.71362.log 2024-01-16T15:30:17.944 INFO:teuthology.orchestra.run.smithi053.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-01-16T15:30:17.946 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.68494.log.gz 2024-01-16T15:30:17.948 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.68685.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.68685.log.gz 2024-01-16T15:30:17.950 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.71362.log: 88.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58707.log 2024-01-16T15:30:17.950 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.77911.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.77911.log.gz -- replaced with /var/log/ceph/ceph-client.admin.71362.log.gz 2024-01-16T15:30:17.951 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:17.953 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73936.log 2024-01-16T15:30:17.957 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.58707.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.58707.log.gz 2024-01-16T15:30:17.960 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.73936.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.73936.log.gz 2024-01-16T15:30:17.962 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53992.log 2024-01-16T15:30:17.963 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.70485.log 2024-01-16T15:30:17.965 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.53992.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53992.log.gz 2024-01-16T15:30:17.968 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.75114.log 2024-01-16T15:30:17.970 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.70485.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.70485.log.gz 2024-01-16T15:30:17.972 INFO:teuthology.orchestra.run.smithi053.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66617.log 2024-01-16T15:30:17.974 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.75114.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.75114.log.gz 2024-01-16T15:30:17.976 INFO:teuthology.orchestra.run.smithi053.stderr:/var/log/ceph/ceph-client.admin.66617.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.66617.log.gz 2024-01-16T15:30:18.002 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.75897.log.gz 2024-01-16T15:30:18.266 INFO:teuthology.orchestra.run.smithi053.stderr: 90.1% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-01-16T15:30:18.491 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.48537.log.gz 2024-01-16T15:30:18.889 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.61266.log.gz 2024-01-16T15:30:19.119 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.56436.log.gz 2024-01-16T15:30:19.231 INFO:teuthology.orchestra.run.smithi053.stderr: 96.2% -- replaced with /var/log/ceph/ceph-client.0.72083.log.gz 2024-01-16T15:30:19.288 INFO:teuthology.orchestra.run.smithi146.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-01-16T15:30:19.643 INFO:teuthology.orchestra.run.smithi146.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-01-16T15:30:19.890 INFO:teuthology.orchestra.run.smithi146.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-01-16T15:30:19.903 INFO:teuthology.orchestra.run.smithi146.stderr: 93.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-01-16T15:30:20.262 INFO:teuthology.orchestra.run.smithi053.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-01-16T15:30:20.395 INFO:teuthology.orchestra.run.smithi146.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-01-16T15:30:20.408 INFO:teuthology.orchestra.run.smithi146.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-01-16T15:30:20.413 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-01-16T15:30:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m4.239s 2024-01-16T15:30:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m22.120s 2024-01-16T15:30:20.413 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m1.809s 2024-01-16T15:30:20.422 INFO:teuthology.orchestra.run.smithi053.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-01-16T15:30:21.217 INFO:teuthology.orchestra.run.smithi053.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-01-16T15:30:21.224 INFO:teuthology.orchestra.run.smithi053.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-01-16T15:30:21.385 INFO:teuthology.orchestra.run.smithi053.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-01-16T15:30:21.389 INFO:teuthology.orchestra.run.smithi053.stderr: 2024-01-16T15:30:21.389 INFO:teuthology.orchestra.run.smithi053.stderr:real 0m5.216s 2024-01-16T15:30:21.389 INFO:teuthology.orchestra.run.smithi053.stderr:user 0m25.512s 2024-01-16T15:30:21.389 INFO:teuthology.orchestra.run.smithi053.stderr:sys 0m4.474s 2024-01-16T15:30:21.390 INFO:tasks.ceph:Archiving logs... 2024-01-16T15:30:21.391 DEBUG:teuthology.misc:Transferring archived files from smithi053:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/remote/smithi053/log 2024-01-16T15:30:21.392 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:30:24.583 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/remote/smithi146/log 2024-01-16T15:30:24.584 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-16T15:30:27.039 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-16T15:30:27.050 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-16T15:30:27.051 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:30:27.059 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-16T15:30:27.219 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-16T15:30:27.227 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-16T15:30:27.228 DEBUG:teuthology.orchestra.run.smithi053:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:30:27.236 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-16T15:30:27.287 INFO:teuthology.orchestra.run.smithi053.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:30:27.290 INFO:teuthology.orchestra.run.smithi146.stderr:bash: line 1: ntpq: command not found 2024-01-16T15:30:27.298 INFO:teuthology.orchestra.run.smithi053.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:30:27.298 INFO:teuthology.orchestra.run.smithi053.stdout:=============================================================================== 2024-01-16T15:30:27.298 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +2794us[+2794us] +/- 110ms 2024-01-16T15:30:27.298 INFO:teuthology.orchestra.run.smithi053.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 +118us[ +123us] +/- 83ms 2024-01-16T15:30:27.298 INFO:teuthology.orchestra.run.smithi053.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -2011us[-2011us] +/- 78ms 2024-01-16T15:30:27.298 INFO:teuthology.orchestra.run.smithi053.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:30:27.299 INFO:teuthology.orchestra.run.smithi146.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-16T15:30:27.300 INFO:teuthology.orchestra.run.smithi146.stdout:=============================================================================== 2024-01-16T15:30:27.300 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 14 +2761us[+2711us] +/- 110ms 2024-01-16T15:30:27.300 INFO:teuthology.orchestra.run.smithi146.stdout:^* hv02.front.sepia.ceph.com 3 6 377 12 +39us[ -11us] +/- 83ms 2024-01-16T15:30:27.300 INFO:teuthology.orchestra.run.smithi146.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 11 -2093us[-2093us] +/- 78ms 2024-01-16T15:30:27.300 INFO:teuthology.orchestra.run.smithi146.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-01-16T15:30:27.301 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-16T15:30:27.313 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-16T15:30:27.314 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-16T15:30:27.328 DEBUG:teuthology.orchestra.run.smithi053:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:30:27.588 DEBUG:teuthology.orchestra.run.smithi146:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-16T15:30:27.803 DEBUG:teuthology.orchestra.run.smithi053:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:30:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:30:27.915 DEBUG:teuthology.orchestra.run.smithi146:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-16T15:30:28.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:30:28.016 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-16T15:30:28.026 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-16T15:30:28.040 INFO:teuthology.task.internal:Duration was 2373.036129 seconds 2024-01-16T15:30:28.041 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-16T15:30:28.055 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-16T15:30:28.056 DEBUG:teuthology.orchestra.run.smithi053:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:30:28.063 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-16T15:30:28.225 INFO:teuthology.orchestra.run.smithi146.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:30:28.230 INFO:teuthology.orchestra.run.smithi053.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-16T15:30:28.588 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-16T15:30:28.589 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi053.front.sepia.ceph.com 2024-01-16T15:30:28.589 DEBUG:teuthology.orchestra.run.smithi053:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:30:28.657 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-01-16T15:30:28.658 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-16T15:30:28.733 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-16T15:30:28.733 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:30:28.741 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-16T15:30:28.833 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-16T15:30:28.833 DEBUG:teuthology.orchestra.run.smithi053:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:30:28.839 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-16T15:30:29.067 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-16T15:30:29.077 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-16T15:30:29.077 DEBUG:teuthology.orchestra.run.smithi053:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:30:29.185 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-16T15:30:29.288 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-16T15:30:29.297 DEBUG:teuthology.orchestra.run.smithi053:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:30:29.305 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:30:29.407 INFO:teuthology.orchestra.run.smithi053.stdout:kernel.core_pattern = core 2024-01-16T15:30:29.412 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-01-16T15:30:29.465 DEBUG:teuthology.orchestra.run.smithi053:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:30:29.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:30:29.515 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-16T15:30:29.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-16T15:30:29.569 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-16T15:30:29.579 INFO:teuthology.task.internal:Transferring archived files... 2024-01-16T15:30:29.580 DEBUG:teuthology.misc:Transferring archived files from smithi053:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/remote/smithi053 2024-01-16T15:30:29.580 DEBUG:teuthology.orchestra.run.smithi053:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:30:29.694 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206/remote/smithi146 2024-01-16T15:30:29.694 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-16T15:30:29.796 INFO:teuthology.task.internal:Removing archive directory... 2024-01-16T15:30:29.796 DEBUG:teuthology.orchestra.run.smithi053:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:30:29.805 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-16T15:30:29.859 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-16T15:30:29.872 INFO:teuthology.task.internal:Not uploading archives. 2024-01-16T15:30:29.872 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-16T15:30:29.893 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-16T15:30:29.893 DEBUG:teuthology.orchestra.run.smithi053:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:30:29.901 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-16T15:30:29.953 INFO:teuthology.orchestra.run.smithi053.stdout: 395190 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:30 /home/ubuntu/cephtest 2024-01-16T15:30:29.958 INFO:teuthology.orchestra.run.smithi146.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 16 15:30 /home/ubuntu/cephtest 2024-01-16T15:30:29.963 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-16T15:30:29.973 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-16T15:30:30.001 INFO:teuthology.nuke:Checking targets against current locks 2024-01-16T15:30:30.027 DEBUG:teuthology.nuke:shortname: smithi053 2024-01-16T15:30:30.027 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:30:30.052 DEBUG:teuthology.nuke:shortname: smithi146 2024-01-16T15:30:30.052 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-16T15:30:30.055 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi053.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:39:43.438575', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM0JQjhuNggf5ic6Z+v+ATyjbZxtPtAlHaYJNWISkWvf8Z0gynhR96Yu8TtxOs6iLtho5IdeDI+TpDb1zD7vs7E='} 2024-01-16T15:30:30.081 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-01-16_14:39:36-fs:volumes-rishabh-clone-stats-testing-default-smithi/7518206', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-16 14:39:43.436837', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPgRJ4UNIuRtkJXGmoH7Fyp95mWSrJnXr95VQfYW7AHroNiCVmwc+foJoouNVGIELbaIXhrMCms6FZ2ILBwlnIQ='} 2024-01-16T15:30:30.111 INFO:teuthology.orchestra.console.smithi053:Power off 2024-01-16T15:30:30.112 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:30:30.136 DEBUG:teuthology.orchestra.console.smithi053:power off output: Chassis Power Control: Down/Off 2024-01-16T15:30:30.137 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:30.152 INFO:teuthology.orchestra.console.smithi146:Power off 2024-01-16T15:30:30.152 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-16T15:30:30.161 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-01-16T15:30:30.173 DEBUG:teuthology.orchestra.console.smithi146:power off output: Chassis Power Control: Down/Off 2024-01-16T15:30:30.173 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:30.208 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-01-16T15:30:34.162 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:34.209 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:34.273 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is on 2024-01-16T15:30:34.318 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-01-16T15:30:38.277 DEBUG:teuthology.orchestra.console.smithi053:pexpect command: ipmitool -H smithi053.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:38.319 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:38.387 DEBUG:teuthology.orchestra.console.smithi053:check power output: Chassis Power is off 2024-01-16T15:30:38.429 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is on 2024-01-16T15:30:38.488 INFO:teuthology.orchestra.console.smithi053:Power off completed 2024-01-16T15:30:42.432 DEBUG:teuthology.orchestra.console.smithi146:pexpect command: ipmitool -H smithi146.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-16T15:30:42.542 DEBUG:teuthology.orchestra.console.smithi146:check power output: Chassis Power is off 2024-01-16T15:30:42.643 INFO:teuthology.orchestra.console.smithi146:Power off completed 2024-01-16T15:30:42.748 INFO:teuthology.run:Summary data: description: fs:volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/clone-progress}} duration: 2373.0361289978027 failure_reason: 'Test failure: test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress), test_when_7_clones_are_cancelled (tasks.cephfs.test_volumes.TestCloneProgress)' flavor: default mon.a-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 mon.b-kernel-sha1: 2668949e0971849893a3fb7f177ce9032fc65780 owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6b7e28a498e34ae7baf49d210efb213d status: fail success: false 2024-01-16T15:30:42.749 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-16T15:30:42.835 INFO:teuthology.run:FAIL